#  :
Engaging AI with Pluralistic Human Values, Rights, and Duties

Taylor Sorensen♠♢, Liwei Jiang♠♢, Jena D. Hwang♢, Sydney Levine♢,
Valentina Pyatkin♠♢, Peter West♠♢, Nouha Dziri♢, Ximing Lu♠♢, Kavel Rao♠,
Chandra Bhagavatula♢, Maarten Sap♣♢, John Tasioulas‡, Yejin Choi♠♢

###### Abstract

Human values are crucial to human decision-making. Value pluralism is the view
that multiple correct values may be held in tension with one another (e.g.,
when considering lying to a friend to protect their feelings, how does one
balance honesty with friendship?). As statistical learners, AI systems fit to
averages by default, washing out these potentially irreducible value
conflicts. To improve AI systems to better reflect value pluralism, the first-
order challenge is to explore the extent to which AI systems can model
pluralistic human values, rights, and duties as well as their interaction.

We introduce ValuePrism, a large-scale dataset of 218k values, rights, and
duties connected to 31k human-written situations. ValuePrism’s contextualized
values are generated by GPT-4 and deemed high-quality by human annotators 91%
of the time. We conduct a large-scale study with annotators across diverse
social and demographic backgrounds to try to understand whose values are
represented.

With ValuePrism, we build (or Kaleido), an open, light-weight, and structured
language-based multi-task model that generates, explains, and assesses the
relevance and valence (i.e., support or oppose) of human values, rights, and
duties within a specific context. Humans prefer the sets of values output by
our system over the teacher GPT-4, finding them more accurate and with broader
coverage. In addition, we demonstrate that Kaleido can help explain
variability in human decision-making by outputting contrasting values.
Finally, we show that Kaleido’s representations can transfer to other
philosophical frameworks and datasets, confirming the benefit of an explicit,
modular, and interpretable approach to value pluralism. We hope that our work
will serve as a step to making more explicit the implicit values behind human
decision-making and to steering AI systems to make decisions that are more in
accordance with them.

##  1 Introduction

Figure 1: Different human values relate, support, or oppose everyday
situations to varying degrees. Kaleido is designed to generate, explain, and
assess how the pluralistic human values, rights, and duties may shape human
judgments.

When people confront difficult decisions (whether or not to break a promise,
what degree program to enroll in, how to spend a Sunday afternoon), their
options reflect their values (friendship, knowledge, freedom, saving money,
spending time in nature). Two people in the same situation may make opposing
decisions if they value different things or the same things but to varying
extents (Figure 1). The notion that different human values can lead to
distinct—although potentially equally valid— decisions is called value
pluralism (Páez et al. 2020; Komppula et al. 2018; Brosch and Sander 2013;
Keeney 1992; Griffiths 2021; Liscio et al. 2023).

Various fields have focused on this concept. Philosophers distinguish value
pluralism (different views cannot be reduced into an ultimate “supervalue”
(Williams 1985; Larmore 1987; Kekes 1993; Stocker 1990; Chang 1997; Dancy
2004)) from monism (there exists a single core value (Kant 1785/2002; Driver
2022)). Sociologists recognize cultural, social, and ideological differences
that drive societal clashes, movements, and changes (Archive 2011).
Psychologists empirically confirm that ethical experiences involve weighing
pluralistic values (Gill and Nichols 2008) and the dissonance that arises from
misaligned values and beliefs (Festinger 1962).

Figure 2: KaleidoSYS system workflow that includes 1) generating 100 values,
rights and duties; 2) filtering by relevance as rated by Kaleido; 3) removing
repetitive items; and computing relevance and valence scores for each value,
right, and duty.

Meanwhile, in AI, there is a growing interest in developing human-centered AI
that emphasizes participation from stakeholders. This which necessitates the
inclusion and exploration of pluralistic voices and values (Tasioulas 2022;
Gordon et al. 2022). Yet, contemporary supervised AI systems primarily wash
out variation by aggregating opinions or preferences with majority votes
(Plank 2022; Talat et al. 2022; Casper et al. 2023). As real-world AI
applications are used to assist increasing and more diverse audiences, it is
crucial to investigate and better model the the values that are accessible and
used by current AI systems.

In this work, we make the first large-scale attempt at investigating large
language models’ (LLMs) potential to model pluralistic human values, rights,
and duties. Our effort is twofold: (1) we introduce ValuePrism, a large-scale
dataset of pluralistic human values; (2) we build (Kaleido), an open and
flexible value-pluralistic model.

The dataset: ValuePrism contains 218k contextualized values, rights, and
duties connected to 31k real-life situations, distilled from GPT-4111Datasheet
for Datasets (Gebru et al. 2018) documentation in Appendix N). While GPT-4 and
its like have been shown to match human crowdworker annotation performance in
some domains (Gilardi, Alizadeh, and Kubli 2023; Ziems et al. 2023; Rytting et
al. 2023), we exercise caution and do not assume that GPT-4’s outputs are
necessarily correct or representative. To this end, we conduct large-scale
human studies and find that humans rate the outputs as high-quality 91% of the
time and have difficulty coming up with considerations that the model has
missed, detecting missing values $>$1% of the time. We also conduct a
comprehensive study with diverse annotators across diverse social and
demographic groups to evaluate whose voices are represented in the values
GPT-4 produces. Additionally, a growing line of work demonstrates that the
large-scale with which data can be produced with LLMs can make up for the
potential noise that is introduced, leading to student models which often
surpass the teacher (West et al. 2022b; Kim et al. 2023; Jung et al. 2023).

The model: (Kaleido) is a value-pluralistic model that, based on ValuePrism,
generates, explains, and assesses the relevance and valence (i.e., support or
oppose) of contextualized pluralistic human values, rights, and duties. On top
of the model, we build a flexible system KaleidoSYS leveraging Kaleido’s
generation and relevance prediction modes to create a diverse, high quality
set of relevant values for a situation (See Fig. 2). In human studies, people
rate our system’s outputs as more correct and complete than the teacher’s
(GPT-4). Annotators also find that our largest model matches the teacher’s
performance at rationalizing and predicting valence. Additionally, we show
that Kaleido can help explain ambiguity and variability underlying human
decision-making in nuanced situations by generating contrasting values. We
also demonstrate that Kaleido can be adapted to various philosophical
frameworks without explicit training.

Overall, our work represents the first comprehensive attempt to articulate
decision-making into fine-grained, pluralistic components of human values
employing large language models. The resulting dataset and model serve as a
large-scale resource explicitly supporting value pluralism, shedding light on
future AI development that accommodates a rich and inclusive tapestry of value
alternatives. 222https://github.com/tsor13/kaleido.

##  2 Value-pluralistic Framework: Values, Rights and Duties

###  2.1 Why are pluralistic human values critical?

Machine learning methods are generally designed to model averages, but can
miss nuance and in-group variation unless explicitly accounted for (Gordon et
al. 2022). To go beyond this, we take inspiration from philosophical value
pluralism, the stance that there are many different normative values (Mason
2006), as opposed to one super-value that all other values can be reduced to.
This is distinct both from political pluralism, which posits that diversity is
beneficial to democratic society and supports the distribution of power among
diverse groups (Britannica Editors 2002); and from relativism, which holds
that no moral system is more correct than another (Gowans 2021).

Without taking a hard stance on these positions, we seek to better model
humans’ plural values to make explicit the implicit values in human decision-
making. Our hope is that, if pluralistic values can be adequately (though
imperfectly) modeled, we can take a step towards ensuring that automated
decision-makers act in accordance with them.

###  2.2 Framework Motivation and Definition

In this work, we model human-centered plural values to make explicit implicit
values in human decision-making. We settle on values (Mason 2006), rights
(Prabhakaran et al. 2022; Wenar 2023), and duties (Alexander and Moore 2021)
as our three core concepts. We propose a commonsense framework for reasoning
about them, and outline them below.

Values: These are the intrinsic goods or ideals that people pursue or cherish,
such as happiness, well-being, justice, or freedom. Values are the desirable
qualities that people may seek in their lives and in the world. They are often
the guiding principles for individuals and societies, shaping goals,
motivations, and preferences.

Duties: Duties are the moral obligations or responsibilities that individuals
owe to others or to society at large. They are categorical reasons for doing
or refraining from doing something, independent of whether we want to do or
refrain from doing that thing. Duties can be weighty reasons, not easily
overridden by competing concerns, and their violation may justify blame and
self-blame (guilt). Duties can arise from relationships, social roles, or
moral principles, and they guide our actions and decisions.

Rights: Rights are the entitlements or claims that individuals have against
others or society, which are usually based on moral or legal grounds. These
can be positive rights (e.g., the right to education, healthcare, or free
speech) or negative rights (e.g., the right to not be harmed, enslaved, or
discriminated against). Rights serve to protect the fundamental interests of
individuals and establish certain boundaries that others must respect.

##  3 Kaleido: Value-pluralistic Modeling

We introduce Kaleido, a language-based multi-task system that generates,
explains, and assesses the relevance and valence (i.e., support or oppose) of
pluralistic human values, rights, and duties, grounded in real-world contexts.

Event | Driving at the maximum speed of the road
---|---
Value | Efficiency | support
Safety | oppose
Rule-following | oppose
Law-abiding behavior | oppose
Right | Right to freedom of movement | either
Right to safety | oppose
Right to life and physical integrity | oppose
Duty | Duty to follow traffic laws | oppose
Event | Gambling with my boss
Value | Professionalism | oppose
Loyalty | oppose
Personal enjoyment | support
Financial well-being | oppose
Right |  Right to work in a safe and professional environment |  oppose
Autonomy | either
Duty |  Duty to maintain a professional relationship with your boss |  oppose
Duty to uphold company policies | oppose
Event | someone cuts me off. i honk ten times
Value | Courtesy | oppose
Respect for others | oppose
Safety | support
Harmony | oppose
Right |  Right to peaceful enjoyment of the road |  oppose
Right to not be subjected to harassment |  oppose
Right to safety | oppose
Duty |  Duty to express displeasure |  support
Duty to be a considerate driver | oppose
Table 1: Example outputs from KaleidoSYS.

###  3.1 Tasks

We develop four tasks for modeling values, rights, and duties, all grounded in
a given context situation.

Generation (open-text) What values, rights, and duties are relevant for a
situation? Generate a value, right, or duty that could be considered when
reasoning about the action.

Relevance (2-way classification) Is a value relevant for a situation? Some
values are more relevant than others.

Valence (3-way classification) Does the value support or oppose the action, or
might it depend on context? Disentangling the valence is critical for
understanding how plural considerations may interact with a decision.

Explanation (open-text) How does the value relate to the action? Generating a
post-hoc rationale for why a value consideration may relate to a situation.

###  3.2 Dataset: ValuePrism

We leverage the symbolic knowledge distillation (West et al. 2022a) pipeline
to distill high-quality knowledge from powerful generative models like GPT-4,
which have been shown to compare favorably to human annotations on quality,
coverage, and diversity (West et al. 2022a; Gilardi, Alizadeh, and Kubli 2023;
Ziems et al. 2023). Importantly, based on our preliminary exploration, GPT-4
excels at enumerating a wide range of value alternatives compared to average
human annotations. We verify the dataset’s quality with human annotators and
show that 91% of the distilled data is deemed high quality, surpassing typical
quality of human generated data (West et al. 2022a; Hwang et al. 2021; Zhou et
al. 2023). Details on dataset statistics and splits are provided in App. F.1
and examples from ValuePrism can be found in App. A.

Situations Starting from 1.3M human-written base situations 333Sourced from
user queries submitted to the Delphi demo., we obtain a set of 31K clean
situations for deriving plural considerations by carefully filtering out ill-
formatted, irrelevant, and low-quality situations using few-shot Flan-T5
(Chung et al. 2022). To balance out an outsize proportion of toxic, NSFW, or
sexually explicit content, we down-sample these situations to 5% of all data,
leading to an increase in the overall diversity of the dataset, as measured by
the normalized count of unique n-grams (dist-2: .23$\rightarrow$.36, dist-3:
.54$\rightarrow$.67, details in App. F.1).

Values, Rights, and Duties Generation For each of the 31K situations, we
prompt GPT-4 to generate relevant values, rights, and duties, with open-text
rationales. GPT-4 also attributes whether the corresponding value, right, or
duty supports (justifies), opposes (condemns), or depends on the context or
interpretation. Details of data generation and prompting are in Appendix F.1
and M. The resulting dataset is rated as high-quality (91%) by human
evaluation (§4.1).

Multi-task Setup We convert ValuePrism into a sequence-to-sequence format for
multi-task training. The relevance task data is sampled contrastively, where
positive examples are from GPT-4 and negative examples are drawn from values
mapped to other situations. We split the data (by actions) into 80% train/10%
validation/10% test.

###  3.3 Model Training

We use Flan-T5 3B (Chung et al. 2022) as the base model for Kaleido. After
fitting hyperparameters with a grid search, we train on a mixture of all four
tasks with Huggingface’s Trainer (Wolf et al. 2020) for 4 epochs with early
stopping. Training takes 19 hours on two A100 GPUs. Training details are in
App. G.

###  3.4 A System of Diverse Values: KaleidoSYS

We use Kaleido to generate a diverse set of values, rights, and duties by
overgenerating (top 100 beams) and removing low-quality and repetitive outputs
via the relevance score and text similarity respectively. We use Rouge-score
(Lin 2004) for n-gram similarity and a Transformers (Wolf et al. 2020) mpnet
model444https://huggingface.co/sentence-transformers/all-mpnet-base-v2 for
sentence embeddings. See Fig. 2 for an illustration of the system and Appendix
H/Algorithm 1 for more details. We tune the system parameters (relevance score
threshold, similarity thresholds) using Gibbs sampling (Casella and George
1992) to maximize RougeL-Sum F1 score on the validation set. Ablation
experiments in §5.1 provide insights on each system component, and example
model outputs can be found in Table 1 and App. 18.

##  4 Data Analysis

###  4.1 ValuePrism is high-quality

We conduct human validation of ValuePrism to assess the its quality on the
Mechanical Turk platform555For this and other human studies, we have acquired
the opinion of our institutions’s Internal Review Board. The opinion finds our
project exempt from a full review process in accordance to the United States
Federal regulation 45 CFR 46, and we have acquired a letter of exception. We
hash crowdworker IDs so annotations cannot be back-traced to individual
workers.. Given the generated situation and values, rights, and duties and
their explanations, we ask the annotators to assess the relevance and quality
of the generations. The results show that annotators find the great majority
of the data as high quality. 91% of the values/rights/duties was marked as
good by all three annotators and 87% of the valences were marked as correct by
all three annotators.

In an attempt to find any values that may have been missed, we also prompt
crowdworkers to fill in any missing values, rights, or duties. Crowdworkers
didn’t seem to find it easy to come up with missing values as we get
suggestions 0.35% of the time. Full annotation details for this and other
studies are in App. I.

###  4.2 Evaluation by diverse annotators

Prior research has reported unjust bias in LLMs against marginalized groups
(Sap et al. 2019; Feng et al. 2023). We evaluate ValuePrism by recruiting a
diverse population of 613 annotators666E.g., Race: 168 white, 115 Black, 61
asian, 34 hispanic/latinx; Sexual orientation: 390 straight, 68 LGBQ+. Gender:
258 male, 201 female, 9 non-binary or other; Full details are in App. E
through CloudResearch (Litman, Robinson, and Abberbock 2017) targeting those
marginalized groups to the extent possible777We chose CloudResearch
specifically because of its ability to target by demographic. One weakness of
this study, however, is that all of our respondents are U.S.-based (where
CloudResearch operates). Prior work has shown that value representation can
vary across nationality as well (Santy et al. 2023), and we hope to extend
this study internationally in the future.. We collect 31k annotations across
683 values, rights, and duties in the context of 100 situations, along with
demographic information across eight categories. The annotators mark 1) if
they agree with each value, right, or duty listed for a given situation and 2)
if they spot any missing perspective. We do not find notable statistical
significance, and do not reject the null hypothesis that there is no
difference between groups. Additional group statistics, p-values, and
qualitative analyses are in App. E.

###  4.3 Diversity of ValuePrism

We analyze the diversity of the situations, and values, rights, and duties
from three perspectives: lexical diversity that calculates uniqueness n-grams,
topical diversity that assesses semantic diversity via topic analysis888Via
BERTopic https://maartengr.github.io/BERTopic, and clustering. Both the
situations and the core value concepts contain diverse distinct situations
with high lexical variations indicating that a diverse variety of events and
values captured by ValuePrism (Table 8). The topic word cloud (Fig. 8) shows
that ValuePrism covers a broad spectrum of common topics like ”save”, ”kill”,
and ”helping” for situations and ”respect”, ”care”, and ”promote” for values.
Clusturing shows that the corpus encompass a wide variety of themes,
reflecting the diversity and richness of situations and values, rights, and
duties. For more data analysis, see App. C.

##  5 Experiments

###  5.1 Our System Against the Teacher

Generating correct and complete sets of values Central to our research is the
capability to model pluralistic values, rights, and duties. Ideally, these
values should be correct, have high coverage, and be aligned with human
preferences. We recruit crowdworkers to evaluate KaleidoSYS directly against
GPT-4 across these three dimensions.

We run several variations of KaleidoSYS: all model sizes (60M–11B); 3B version
without the relevance and text similarity components (-relevance, -text
similarity); and tweaked 3B system parameters to output more or fewer values,
rights, and duties 999To better understand how changing the parameters can
affect the output/precision/recall, see Figure 4 (verbose, concise). To
understand the benefit of our system, we also train a baseline seq2seq 3B
model on the same data that tries to predict a batch of values, rights, and
duties in one generation pass. We test each version against GPT-4 on a set of
200 test situations by 2 annotators each.

Model | Overall | Cover. | Acc. | Avg. #
---|---|---|---|---
KaleidoSYS 3B | 55.5 | 65.1 | 58.9 | 8.2
-relevance | 51.9 | 81.4 | 64.3 | 11.2
-text similarity | 50.0 | 60.5 | 52.9 | 8.2
verbose | 58.0 | 86.1 | 69.0 | 11.1
concise | 39.0 | 27.4 | 32.4 | 5.0
KalSYS 11B | 58.3 | 71.1 | 62.5 | 8.3
KalSYS 770M | 57.9 | 67.3 | 60.8 | 8.2
KalSYS 220M | 44.9 | 59.0 | 50.8 | 8.1
KalSYS 60M | 32.0 | 53.0 | 37.1 | 8.5
Direct Output | 42.5 | 37.9 | 40.0 | 6.8
GPT-4 | 50.0 | 50.0 | 50.0 | 7.0
GPT-3.5-turbo | 39.5 | 49.0 | 39.8 | 8.0
Table 2: The overall, coverage and accuracy win rate percentage against GPT-4
by human evaluators along with the average number of generated values, rights,
and duties. (Here and throughout, best results within 1% are bolded.)

From Table 2, we make several observations. The three largest versions of our
system outperform GPT-4 on all evaluated dimensions, with the largest variant
(11B) being the most favored overall. Moreover, the models generating a higher
number of values ($>$11) are preferred by humans for coverage and
accuracy.101010This is in line with prior work showing that humans prefer
longer outputs with more unique n-grams (Wang et al. 2023b) Our system also
shows an advantage over the direct output seq2seq model, despite being trained
on the same data. Furthermore, removing relevance leads to a drop in the
overall preference, which is not observed in verbose with the same number of
outputs. This suggests relevance is indeed a contributing factor to the
generation quality. Finally, humans show lower preference for outputs without
deduplication with text similarity.

Explanation and Valence Label Quality We also evaluated the explanation
generation and valence labeling abilities of each model using 700 predefined
values, rights, and duties from the test set of ValuePrism. Crowdworkers were
tasked with evaluating the quality of explanations, their effectiveness in
linking values to actions, and agreement with valence labels. As depicted in
Table 3, the 11B model’s performance closely aligns with that of GPT-4. The
11B model achieved Valence accuracy within a 1% difference from GPT-4 and
slightly outperformed it in terms of Explanation quality.

Model | Explanation | Valence | Rel. corr.
---|---|---|---
Kaleido 3B | 92.6 | 92.0 | 0.30
Kal 11B | 94.8 | 92.6 | 0.25
Kal 770M | 90.3 | 90.3 | 0.31
Kal 220M | 86.9 | 86.3 | 0.30
Kal 60M | 75.9 | 72.3 | 0.28
GPT-4 | 94.7 | 93.1 | -
Table 3: Human Evaluation. Explanation and Valence scores are correctness
rates of the output, while Relevance is the correlation of relevance score
with the percentage of people who marked a value as relevant.

###  5.2 Relevance Correlates with Human Judgments

We would like Kaleido to predict whether a human would find a value, right, or
duty relevant. However, its training data is synthetic, so the model instead
predicts whether a given value was likely to be generated for a particular
situation by GPT-4. To test how well this correlates with how humans judge
relevance, we collect 18 annotations each for 700 values/rights/duties for
relevance, and correlate the relevance score with the percentage of people who
marked the value as relevant (See Table 3). We find correlations of 0.25-0.31
for the suite of model sizes 111111 Interestingly, we note that the
correlation does not strictly improve with model size. While we are unsure of
the reason for this, we note that 11B gives much more confident relevance
scores, and hypothesize that this overconfidence may be miscalibrated to human
judgments. (all significant at $p<10^{-10}$). While we would like to
explicitly train models to predict human relevance scores in future work, we
take this as evidence that our synthetic relevance score correlates positively
with human judgments.

###  5.3 Zero-shot performance on ETHICS

While our model is explicitly trained to recognize values, rights, and duties,
we want to understand how much the learned representations generalize to other
frameworks as well. To do this, we test Kaleido on the ETHICS benchmark
(Hendrycks et al. 2023), which contains crowdsourced ethical judgments across
several different frameworks, fitting templates with values, rights, or duties
that loosely correspond to the frameworks (see Appendix L).

Subset | Kaleido | ChatGPT | Random
---|---|---|---
Justice |  17.5 / 13.3 |  17.6 / 13.4 | 6.3 / 6.3
Deont. |  19.8 / 15.1 |  20.6 / 13.8 | 6.3 / 6.3
Virtue |  33.1 / 22.2 | 24.9 / 22.0 | 8.2 / 8.2
Util. |  76.5 / 66.6 | 59.4 / 55.1 | 50.0 / 50.0
Comm. | 71.5 / 64.7 |  80.3 / 68.8 | 50.0 / 50.0
Average |  43.7 / 36.4 | 40.6 / 34.6 | 24.2 / 24.2
Table 4: ETHICS few-shot performance. First/second number of each entry is
performance on the test/hard test sets respectively. Kaleido is zero-shot,
ChatGPT is few-shot.

Results are in Table 4. On all five tasks, our model performs well over the
random baseline. On all tasks but Commonsense, our model matches or exceeds
(Justice, Deont., Virtue, Util.) ChatGPT’s performance, while only having 3B
parameters. Despite having only been trained to predict values, rights, and
duties, our model meaningfully generalizes to other frameworks.

###  5.4 Interpretable Decision System and Zero-Shot on CommonsenseNormBank

While the focus of the system is on modeling diverse values and not on making
judgments, it can be easily extended to output the valence of an action
$V(a)$:

$V(a)=\sum_{v\in VRD}R(v|a)\times V(v|a)$

where $v\in VRD$ are the generated values, rights, and duties from KaleidoSYS,
$R(v|a)$ is the relevance of $v$ given the action, and $V(v|a)$ is the valence
of $v$ given the action. We will denote this decision system KaleidoDec.

This system has the advantage of being interpretable, enabling direct
inspection of how values linearly contribute to the outcome. It is also
steerable, as users can easily assign a weight of zero to values they do not
wish to take into consideration.

Model | SBIC | ETH. | MoSt | SoCh
---|---|---|---|---
KaleidoDec | 64.4 | 77.9 | 75.4 | 48.2
+label calibration | 69.3 | 78.0 | 76.2 | 63.0
(improvement) | (+4.9) | (+0.1) | (+0.8) | (+14.8)
Majority class | 63.1 | 51.6 | 50.0 | 46.7
Random | 33.3 | 50.0 | 50.0 | 33.3
Delphi (SFT) | 82.9 | 86.2 | 86.5 | 78.0
Table 5: Zero-shot Performance on CommonsenseNormBank: Moral Acceptability.

Zero-shot CommonsenseNormBank performance We evaluate this system in a zero-
shot manner on the four subportions of moral acceptability segment of
CommonsenseNormBank (Jiang et al. 2022) (See Table 5). In all cases, the
system performs at least as well as the majority class baseline, and much
($\geq$25%) better on ETHICS and Moral Stories121212For these two datasets,
there is no “neutral” (i.e., lacks valence) class, so the “either” valence is
zeroed out..

We observe that the classes are not well calibrated to the dataset statistics.
To remedy this, we fit a lightweight logistic regression. For SBIC and
SocialChem it improves accuracy by about 5% and 15% respectively, suggesting
that while the model is not initially well-calibrated to the datasets,
relevant information can be linearly extracted. While KaleidoDec achieves non-
trivial zero-shot performance, it unsurprisingly performs worse than
supervised baselines such as Delphi.

###  5.5 Entropy as an Indicator of Decision Variability

Figure 3: The output entropy of KaleidoDec is predictive of ambiguity in
MoralChoice and controversialness in SocialChem. A threshold is chosen to
maximize F1-score.

When values support different decisions, it may be an indicator that the final
judgment one may come to is highly dependent on which value is prioritized.
Because of this, when KaleidoDec output has high entropy, we hypothesize that
this may indicate higher variability in the distribution of decisions. To test
this, we explore two datasets with variability indicators. MoralChoice
(Scherrer et al. 2023) contains 687 low-ambiguity and 680 high-ambiguity moral
scenarios. SocialChem (Forbes et al. 2021) is a corpus of social norms where,
among other things, crowdworkers annotated for ”What portion of people
probably agree that [action] is [good / bad]?”. We take those marked as $\geq$
99% to have low controversialness, and those marked as $\leq$50% as having
high controversialness. We run the corresponding scenarios through KaleidoDec
and measure the entropy (Figure 3). We find that the entropy is predictive of
these classes. In line with our hypothesis, the higher the entropy, the more
likely a situation is to be ambiguous or controversial, even though the model
was not trained to predict these things explicitly.

##  6 Discussion

Figure 4: By sweeping KaleidoSYS’s parameters, we are able to trade precision
(w.r.t. to GPT-4) with recall, and output many more (or fewer) values, rights,
and duties. Figure 5: Kaleido is sensitive to subtle changes in inputs,
changing relevance and valence scores accordingly.

Strengths Over Teacher Model Although our model performs strongly against the
teacher in value generation, it also boasts several other advantages. It is
controllable, allowing users to generate either more or fewer values than
GPT-4 by trading precision for recall (see Figure 4). Additionally, while
GPT-4 provides only textual labels for valence, our model generates scalar
valence and relevance scores. Lastly, our model, dataset, and code are openly
accessible, enabling scientific review that is crucial for accountability and
improvement.

Kaleido is Sensitive to Situation Variations One of the strengths of our
approach is that the signal can be contextualized and dependent on the
relevant values. For example, consider three variations of a situation:
”Leticia kisses Marco,” ”Leticia kisses Marco when he doesn’t agree,” and
”Leticia kisses Marco when he is sick” (see Figure 5). In all three
situations, affection and consent are relevant values, as reflected by their
relevance scores. However, the valence changes: consent can either support or
oppose the action in the two underspecified situations, but opposes “when
Marco doesn’t agree.” Additionally, the value of health isn’t usually relevant
in the typical context of kissing: however, “when Marco is sick,” health
becomes relevant and opposes the action. This demonstrates the ability of
Kaleido to adjust to subtle input changes131313While this is not a
quantitative experiment, this is not a cherry-picked example..

False Balance and Extreme Inputs One potential danger when generating diverse
values is coming up with a contrived reason why something might be good or
bad, even if no reasonable person may hold such a value in such a situation
(This is similar to false balance, or bothesidesism, in news reporting (Imundo
and Rapp 2021; Boykoff and Boykoff 2004)). To probe at this, we hand-write 20
situations (10 bad/10 good, in Appendix J) for which we cannot come up with
reasonable values, rights, or duties that would support or oppose them
respectively. We run them through KaleidoSYS after development and find no
generated supporting values/rights/duties for the extreme bad actions nor any
opposing for the good actions. We take this as limited evidence that our
system can avoid false balance.

Universal Declaration of Human Rights Inspired by (Prabhakaran et al. 2022),
we think that an ideal dataset containing human rights would contain all
rights listed in the United Nation’s Universal Declaration of Human Rights
141414https://www.un.org/en/about-us/universal-declaration-of-human-rights
(UDHR). We manually extract all human rights we could find from the UDHR (41),
and find the 20 closest rights in the dataset by entailment score with WANLI
(Liu et al. 2022a). We then go through all 41 sets manually and label each for
whether or the right is included. We are able to find matches in ValuePrism
for 97.5% of the UDHR’s human rights, demonstrating that the dataset has broad
coverage of the UDHR. 151515See App. K for all human rights and corresponding
matches..

##  7 Related Work

Value Representations of Language Models Scholars from diverse disciplines
have engaged in extensive discussions regarding the incorporation of human
ethics and values into LLMs (Wallach and Allen 2008; Jiang et al. 2022;
Hendrycks et al. 2023), understanding cultural influences (Santy et al. 2023),
examining opinion alignment (Santurkar et al. 2023), and using LLMs as proxies
for studying specific human sub-populations in social science research (Argyle
et al. 2023). (Jiang et al. 2022) introduced Delphi, a framework trained to
reason about ethical perspectives, and showed the ethical limitations of out-
of-the-box LLMs. Another noteworthy dimension is the multicultural nature of
LLMs. (Santy et al. 2023) explored the cultural disparities manifest in LMs
and their implication for diverse societies. (Tasioulas 2022) criticized the
prevailing preference-based utilitarian approach (i.e., which act is likely to
yield the optimal fulfillment of human preferences) in AI ethics, pointing out
its limitations and proposing an alternative “humanistic” ethics that accounts
for the nuanced human views on ethics. Moreover, Santurkar et al. (2023);
Durmus et al. (2023) introduced novel opinion datasets, quantitatively
analyzed the opinions conveyed by LMs, and unveiled substantial misalignments
between the viewpoints of current LMs and specific demographic groups within
the United States.

Alignment of Large Language Models Several computational approaches have been
proposed to address the challenge of aligning LLMs with desired values and
objectives. Reinforcement learning (RL) has historically been used in multiple
NLP tasks to ensure that the generated text is optimized for an arbitrary non-
differentiable reward (Johnson et al. 2017; Nguyen, Daumé III, and Boyd-Graber
2017; Ramamurthy et al. 2022; Pyatkin et al. 2023). Lu et al. (2022) optimized
a reward function that quantifies an undesired property, while not straying
too far from the original model via a KL-divergence penalty. (Bai et al. 2022)
explored RL techniques for training LLMs to adhere to legal and ethical
guidelines encoded in a constitution, naming it “Constitutional AI”. Wu et al.
(2023) used fine-grained human feedback as an explicit training signal to
train and learn from reward functions in a RLHF fashion. Additionally, Lu et
al. (2023) proposed an inference-time algorithm to efficiently tailor LLMs
without no fine-tuning, addressing tasks like ensuring safety and fidelity in
dialogue models.

Automatic Dataset Curation Previous research in automatic data generation has
focused on creating datasets for various tasks, such as commonsense reasoning
(West et al. 2022a; Bhagavatula et al. 2023; Wang et al. 2023a; Kim et al.
2023), dialogues (Kim et al. 2023; Xu et al. 2023; Chiang et al. 2023),
summarization (Sclar et al. 2022; Jung et al. 2023), and contextual reasoning
about offensive statements (Zhou et al. 2023) . (West et al. 2022a) introduce
the symbolic knowledge distillation framework, which has been extended in
subsequent studies through iterative distillation (Sclar et al. 2022; Jung et
al. 2023; Bhagavatula et al. 2023). In addition, (Liu et al. 2022b) propose a
human-AI collaboration approach to generate high-quality datasets with
challenging examples.

##  8 Conclusion

In this work, we contribute ValuePrism and Kaleido in the hopes of leading to
better value-pluralistic modeling. We validate ValuePrism’s quality with two
human studies, and find that Kaleido outperforms the teacher’s strong
performance at generating relevant values, rights, and duties for a given
situation. We also show that Kaleido can help explain variability in human
decisions and generalizes to data and frameworks outside of its training
scope.

##  9 Ethical Considerations and Limitations

Machine-Generated Data. We use GPT-4’s open-text generative capabilities to
collect ValuePrism, leveraging the wide variety of knowledge about human
values, rights, and duties latent in LLM’s pretraining data. However, we also
recognize that in doing so we run the potential for introducing the majority’s
bias: the generated data may be limited to the values of certain majority
groups. In an effort to assess the extent of value plurality and represention,
we make a deliberate effort to conduct the validation of the ValuePrism by
collecting annotations from annotators of various social and demographic
backgrounds as described in §4.2. The human annotators find the majority of
our data as high-quality at a high agreement rate. Additionally, less than 1%
of the validated situations were found to be lacking. Nevertheless, a more
extensive study that focuses on the type and nature of value plurality covered
by ValuePrism remains a compelling direction for future research.

Intended Use. We make ValuePrism openly available by individual request with
the hope and intention that it furthers research in value pluralism in NLP and
AI. However, it is possible that our data can be used in malicious and
unintended application (e.g., speech policing or promotion of certain values).
We do not endorse its use in such capacity and emphasize that the use of our
dataset and model should be limited to research purposes only. Additionally,
we limit the data and model available only by individual request to try to
prohibit non-research use cases and ensure fair use.

## Acknowledgments

The authors thank Ronan LeBras, Jared Moore, Hyunwoo Kim, Jenny Liang, and
Sebastin Santy for helpful discussions; Alane Suhr for the example situation
in Figure 2; Jared Moore, Dhruba Ghosh, and David Atkinson for draft feedback;
and Michael Wilson, Michael Guerquin, and John Borchardt from the AI2 ReViz
team for help with the demo. This research was supported in part by DARPA
under the ITM program (FA8650-23-C-7316) and the Allen Institute for AI.

## References

  * Alexander and Moore (2021) Alexander, L.; and Moore, M. 2021.  Deontological Ethics.  In Zalta, E. N., ed., _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Winter 2021 edition.
  * Allen Insitute for AI (2023) Allen Insitute for AI. 2023.  AI2 ImpACT Licenses.  https://allenai.org/impact-license.  [accessed August 2023].
  * Archive (2011) Archive, G. D. 2011.  European Values Study Longitudinal Data File 1981-2008 (EVS 1981-2008).  _EVS_.
  * Argyle et al. (2023) Argyle, L. P.; Busby, E. C.; Fulda, N.; Gubler, J. R.; Rytting, C.; and Wingate, D. 2023.  Out of One, Many: Using Language Models to Simulate Human Samples.  _Political Analysis_ , 1–15.
  * Bai et al. (2022) Bai, Y.; Kadavath, S.; Kundu, S.; Askell, A.; Kernion, J.; Jones, A.; Chen, A.; Goldie, A.; Mirhoseini, A.; McKinnon, C.; Chen, C.; Olsson, C.; Olah, C.; Hernandez, D.; Drain, D.; Ganguli, D.; Li, D.; Tran-Johnson, E.; Perez, E.; Kerr, J.; Mueller, J.; Ladish, J.; Landau, J.; Ndousse, K.; Lukosuite, K.; Lovitt, L.; Sellitto, M.; Elhage, N.; Schiefer, N.; Mercado, N.; DasSarma, N.; Lasenby, R.; Larson, R.; Ringer, S.; Johnston, S.; Kravec, S.; Showk, S. E.; Fort, S.; Lanham, T.; Telleen-Lawton, T.; Conerly, T.; Henighan, T.; Hume, T.; Bowman, S. R.; Hatfield-Dodds, Z.; Mann, B.; Amodei, D.; Joseph, N.; McCandlish, S.; Brown, T.; and Kaplan, J. 2022.  Constitutional AI: Harmlessness from AI Feedback.  arXiv:2212.08073.
  * Bhagavatula et al. (2023) Bhagavatula, C.; Hwang, J. D.; Downey, D.; Bras, R. L.; Lu, X.; Qin, L.; Sakaguchi, K.; Swayamdipta, S.; West, P.; and Choi, Y. 2023.  I2D2: Inductive Knowledge Distillation with NeuroLogic and Self-Imitation.  arXiv:2212.09246.
  * Boykoff and Boykoff (2004) Boykoff, M. T.; and Boykoff, J. M. 2004.  Balance as bias: global warming and the US prestige press.  _Global Environmental Change_ , 14(2): 125–136.
  * Britannica Editors (2002) Britannica Editors. 2002.  Pluralism — Ideology, Diversity & Tolerance — britannica.com.  https://www.britannica.com/topic/pluralism-politics.  [Accessed 07-08-2023].
  * Brosch and Sander (2013) Brosch, T.; and Sander, D. 2013.  Neurocognitive mechanisms underlying value-based decision-making: from core values to economic value.  _Frontiers in Human Neuroscience_ , 7.
  * Casella and George (1992) Casella, G.; and George, E. I. 1992.  Explaining the Gibbs Sampler.  _The American Statistician_ , 46(3): 167–174.
  * Casper et al. (2023) Casper, S.; Davies, X.; Shi, C.; Gilbert, T. K.; Scheurer, J.; Rando, J.; Freedman, R.; Korbak, T.; Lindner, D.; Freire, P.; Wang, T.; Marks, S.; Segerie, C.-R.; Carroll, M.; Peng, A.; Christoffersen, P.; Damani, M.; Slocum, S.; Anwar, U.; Siththaranjan, A.; Nadeau, M.; Michaud, E. J.; Pfau, J.; Krasheninnikov, D.; Chen, X.; Langosco, L.; Hase, P.; Bıyık, E.; Dragan, A.; Krueger, D.; Sadigh, D.; and Hadfield-Menell, D. 2023.  Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback.  arXiv:2307.15217.
  * Chang (1997) Chang, R. 1997.  _Incommensurability, Incomparability, and Practical Reason_.  Cambridge, MA, USA: Harvard.
  * Chiang et al. (2023) Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; Stoica, I.; and Xing, E. P. 2023.  Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality.
  * Chung et al. (2022) Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; Webson, A.; Gu, S. S.; Dai, Z.; Suzgun, M.; Chen, X.; Chowdhery, A.; Castro-Ros, A.; Pellat, M.; Robinson, K.; Valter, D.; Narang, S.; Mishra, G.; Yu, A.; Zhao, V.; Huang, Y.; Dai, A.; Yu, H.; Petrov, S.; Chi, E. H.; Dean, J.; Devlin, J.; Roberts, A.; Zhou, D.; Le, Q. V.; and Wei, J. 2022.  Scaling Instruction-Finetuned Language Models.  arXiv:2210.11416.
  * Dancy (2004) Dancy, J. 2004.  _Ethics Without Principles_.  New York: Oxford University Press.
  * Driver (2022) Driver, J. 2022.  The History of Utilitarianism.  In Zalta, E. N.; and Nodelman, U., eds., _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Winter 2022 edition.
  * Durmus et al. (2023) Durmus, E.; Nyugen, K.; Liao, T. I.; Schiefer, N.; Askell, A.; Bakhtin, A.; Chen, C.; Hatfield-Dodds, Z.; Hernandez, D.; Joseph, N.; et al. 2023.  Towards Measuring the Representation of Subjective Global Opinions in Language Models.  _arXiv preprint arXiv:2306.16388_.
  * Feng et al. (2023) Feng, S.; Park, C. Y.; Liu, Y.; and Tsvetkov, Y. 2023.  From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models.  arXiv:2305.08283.
  * Festinger (1962) Festinger, L. 1962.  Cognitive dissonance.  _Sci. Am._ , 207(4): 93–102.
  * Forbes et al. (2021) Forbes, M.; Hwang, J. D.; Shwartz, V.; Sap, M.; and Choi, Y. 2021.  Social Chemistry 101: Learning to Reason about Social and Moral Norms.  arXiv:2011.00620.
  * Gebru et al. (2018) Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J. W.; Wallach, H. M.; Daumé, H.; and Crawford, K. 2018.  Datasheets for datasets.  _Communications of the ACM_ , 64: 86 – 92.
  * Gilardi, Alizadeh, and Kubli (2023) Gilardi, F.; Alizadeh, M.; and Kubli, M. 2023.  ChatGPT outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences_ , 120(30).
  * Gill and Nichols (2008) Gill, M. B.; and Nichols, S. 2008.  Sentimentalist Pluralism: Moral Psychology and Philosophical Ethics.  _Philosophical Issues_ , 18(1): 143–163.
  * Gordon et al. (2022) Gordon, M. L.; Lam, M. S.; Park, J. S.; Patel, K.; Hancock, J.; Hashimoto, T.; and Bernstein, M. S. 2022.  Jury Learning: Integrating Dissenting Voices into Machine Learning Models.  In _CHI Conference on Human Factors in Computing Systems_. ACM.
  * Gowans (2021) Gowans, C. 2021.  Moral Relativism.  In Zalta, E. N., ed., _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Spring 2021 edition.
  * Griffiths (2021) Griffiths, N. 2021.  Personal Values & Decision-Making Biases.
  * Hendrycks et al. (2023) Hendrycks, D.; Burns, C.; Basart, S.; Critch, A.; Li, J.; Song, D.; and Steinhardt, J. 2023.  Aligning AI With Shared Human Values.  arXiv:2008.02275.
  * Hwang et al. (2021) Hwang, J. D.; Bhagavatula, C.; Bras, R. L.; Da, J.; Sakaguchi, K.; Bosselut, A.; and Choi, Y. 2021.  COMET-ATOMIC 2020: On Symbolic and Neural Commonsense Knowledge Graphs.  arXiv:2010.05953.
  * Imundo and Rapp (2021) Imundo, M.; and Rapp, D. 2021.  When Fairness is Flawed: Effects of False Balance Reporting and Weight-of-Evidence Statements on Beliefs and Perceptions of Climate Change.  _Journal of Applied Research in Memory and Cognition_ , 11.
  * Jiang et al. (2022) Jiang, L.; Hwang, J. D.; Bhagavatula, C.; Bras, R. L.; Liang, J.; Dodge, J.; Sakaguchi, K.; Forbes, M.; Borchardt, J.; Gabriel, S.; Tsvetkov, Y.; Etzioni, O.; Sap, M.; Rini, R.; and Choi, Y. 2022.  Can Machines Learn Morality? The Delphi Experiment.  arXiv:2110.07574.
  * Johnson et al. (2017) Johnson, M.; Schuster, M.; Le, Q. V.; Krikun, M.; Wu, Y.; Chen, Z.; Thorat, N.; Viégas, F.; Wattenberg, M.; Corrado, G.; et al. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  _Transactions of the Association for Computational Linguistics_ , 5: 339–351.
  * Jung et al. (2023) Jung, J.; West, P.; Jiang, L.; Brahman, F.; Lu, X.; Fisher, J.; Sorensen, T.; and Choi, Y. 2023.  Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasing.  arXiv:2305.16635.
  * Kant (1785/2002) Kant, I. 1785/2002.  _Groundwork for the Metaphysics of Morals_.  Yale University Press.
  * Keeney (1992) Keeney, R. L. 1992.  _Value-Focused Thinking: A Path to Creative Decisionmaking_.  Harvard University Press.  ISBN 9780674931978.
  * Kekes (1993) Kekes, J. 1993.  _The Morality of Pluralism_.  Princeton University Press.
  * Kim et al. (2023) Kim, H.; Hessel, J.; Jiang, L.; West, P.; Lu, X.; Yu, Y.; Zhou, P.; Bras, R. L.; Alikhani, M.; Kim, G.; Sap, M.; and Choi, Y. 2023.  SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualization.  arXiv:2212.10465.
  * Komppula et al. (2018) Komppula, R.; Honkanen, A.; Rossi, S.; Kolesnikova, N.; et al. 2018.  The impact of values on sustainable behaviour-A study among Russian and Finnish university students.  _European Journal of Tourism Research_ , 19: 116–131.
  * Larmore (1987) Larmore, C. E. 1987.  _Patterns of Moral Complexity_.  New York: Cambridge University Press.
  * Lin (2004) Lin, C.-Y. 2004.  ROUGE: A Package for Automatic Evaluation of Summaries.  In _Text Summarization Branches Out_ , 74–81. Barcelona, Spain: Association for Computational Linguistics.
  * Liscio et al. (2023) Liscio, E.; Araque, O.; Gatti, L.; Constantinescu, I.; Jonker, C.; Kalimeri, K.; and Murukannaiah, P. K. 2023.  What does a Text Classifier Learn about Morality? An Explainable Method for Cross-Domain Comparison of Moral Rhetoric.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 14113–14132. Toronto, Canada: Association for Computational Linguistics.
  * Litman, Robinson, and Abberbock (2017) Litman, L.; Robinson, J.; and Abberbock, T. 2017.  TurkPrime. com: A versatile crowdsourcing data acquisition platform for the behavioral sciences.  _Behavior research methods_ , 49(2): 433–442.
  * Liu et al. (2022a) Liu, A.; Swayamdipta, S.; Smith, N. A.; and Choi, Y. 2022a.  WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation.  arXiv:2201.05955.
  * Liu et al. (2022b) Liu, A.; Swayamdipta, S.; Smith, N. A.; and Choi, Y. 2022b.  WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , 6826–6847. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
  * Lu et al. (2023) Lu, X.; Brahman, F.; West, P.; Jang, J.; Chandu, K.; Ravichander, A.; Qin, L.; Ammanabrolu, P.; Jiang, L.; Ramnath, S.; et al. 2023.  Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning.  _arXiv preprint arXiv:2305.15065_.
  * Lu et al. (2022) Lu, X.; Welleck, S.; Hessel, J.; Jiang, L.; Qin, L.; West, P.; Ammanabrolu, P.; and Choi, Y. 2022.  Quark: Controllable text generation with reinforced unlearning.  _Advances in neural information processing systems_ , 35: 27591–27609.
  * Mason (2006) Mason, E. 2006.  Value pluralism.
  * Nguyen, Daumé III, and Boyd-Graber (2017) Nguyen, K.; Daumé III, H.; and Boyd-Graber, J. 2017.  Reinforcement Learning for Bandit Neural Machine Translation with Simulated Human Feedback.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , 1464–1474.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774.
  * Plank (2022) Plank, B. 2022.  The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 10671–10682. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
  * Prabhakaran et al. (2022) Prabhakaran, V.; Mitchell, M.; Gebru, T.; and Gabriel, I. 2022.  A Human Rights-Based Approach to Responsible AI.  arXiv:2210.02667.
  * Pyatkin et al. (2023) Pyatkin, V.; Hwang, J. D.; Srikumar, V.; Lu, X.; Jiang, L.; Choi, Y.; and Bhagavatula, C. 2023.  ClarifyDelphi: Reinforced Clarification Questions with Defeasibility Rewards for Social and Moral Situations.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 11253–11271.
  * Páez et al. (2020) Páez, J.; De-Juanas, A.; García-Castilla, F.; and Muelas, A. 2020.  Relationship Between Basic Human Values and Decision-Making Styles in Adolescents.  _International Journal of Environmental Research and Public Health_ , 17(22): 8315.  [Accessed 31-07-2023].
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  arXiv:1910.10683.
  * Ramamurthy et al. (2022) Ramamurthy, R.; Ammanabrolu, P.; Brantley, K.; Hessel, J.; Sifa, R.; Bauckhage, C.; Hajishirzi, H.; and Choi, Y. 2022.  Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization.  In _The Eleventh International Conference on Learning Representations_.
  * Randolph (2005) Randolph, J. J. 2005.  Free-Marginal Multirater Kappa (multirater K [free]): An Alternative to Fleiss’ Fixed-Marginal Multirater Kappa.  _Online submission_.
  * Rytting et al. (2023) Rytting, C. M.; Sorensen, T.; Argyle, L.; Busby, E.; Fulda, N.; Gubler, J.; and Wingate, D. 2023.  Towards Coding Social Science Datasets with Language Models.  arXiv:2306.02177.
  * Santurkar et al. (2023) Santurkar, S.; Durmus, E.; Ladhak, F.; Lee, C.; Liang, P.; and Hashimoto, T. 2023\.  Whose Opinions Do Language Models Reflect?  arXiv:2303.17548.
  * Santy et al. (2023) Santy, S.; Liang, J. T.; Bras, R. L.; Reinecke, K.; and Sap, M. 2023.  NLPositionality: Characterizing Design Biases of Datasets and Models.  arXiv:2306.01943.
  * Sap et al. (2019) Sap, M.; Card, D.; Gabriel, S.; Choi, Y.; and Smith, N. A. 2019.  The Risk of Racial Bias in Hate Speech Detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , 1668–1678. Florence, Italy: Association for Computational Linguistics.
  * Sap et al. (2020) Sap, M.; Gabriel, S.; Qin, L.; Jurafsky, D.; Smith, N. A.; and Choi, Y. 2020.  Social Bias Frames: Reasoning about Social and Power Implications of Language.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 5477–5490. Online: Association for Computational Linguistics.
  * Scherrer et al. (2023) Scherrer, N.; Shi, C.; Feder, A.; and Blei, D. M. 2023.  Evaluating the Moral Beliefs Encoded in LLMs.  arXiv:2307.14324.
  * Sclar et al. (2022) Sclar, M.; West, P.; Kumar, S.; Tsvetkov, Y.; and Choi, Y. 2022.  Referee: Reference-Free Sentence Summarization with Sharper Controllability through Symbolic Knowledge Distillation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , 9649–9668. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics.
  * Stocker (1990) Stocker, M. 1990.  _Plural and Conflicting Values_.  New York: Oxford University Press.
  * Talat et al. (2022) Talat, Z.; Blix, H.; Valvoda, J.; Ganesh, M. I.; Cotterell, R.; and Williams, A. 2022.  On the Machine Learning of Ethical Judgments from Natural Language.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 769–779. Seattle, United States: Association for Computational Linguistics.
  * Tasioulas (2022) Tasioulas, J. 2022.  Artificial Intelligence, Humanistic Ethics.  _Daedalus_ , 151(2): 232–243.
  * Wallach and Allen (2008) Wallach, W.; and Allen, C. 2008.  _Moral machines: Teaching robots right from wrong_.  Oxford University Press.
  * Wang et al. (2023a) Wang, P.; Wang, Z.; Li, Z.; Gao, Y.; Yin, B.; and Ren, X. 2023a.  SCOTT: Self-Consistent Chain-of-Thought Distillation.  arXiv:2305.01879.
  * Wang et al. (2023b) Wang, Y.; Ivison, H.; Dasigi, P.; Hessel, J.; Khot, T.; Chandu, K. R.; Wadden, D.; MacMillan, K.; Smith, N. A.; Beltagy, I.; and Hajishirzi, H. 2023b.  How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources.  arXiv:2306.04751.
  * Wenar (2023) Wenar, L. 2023.  Rights.  In Zalta, E. N.; and Nodelman, U., eds., _The Stanford Encyclopedia of Philosophy_. Metaphysics Research Lab, Stanford University, Spring 2023 edition.
  * West et al. (2022a) West, P.; Bhagavatula, C.; Hessel, J.; Hwang, J.; Jiang, L.; Le Bras, R.; Lu, X.; Welleck, S.; and Choi, Y. 2022a.  Symbolic Knowledge Distillation: from General Language Models to Commonsense Models.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 4602–4625. Seattle, United States: Association for Computational Linguistics.
  * West et al. (2022b) West, P.; Bhagavatula, C.; Hessel, J.; Hwang, J. D.; Jiang, L.; Bras, R. L.; Lu, X.; Welleck, S.; and Choi, Y. 2022b.  Symbolic Knowledge Distillation: from General Language Models to Commonsense Models.  arXiv:2110.07178.
  * Williams (1985) Williams, B. 1985.  Moral Luck.  _Critica_ , 17(51): 101–105.
  * Wolf et al. (2020) Wolf, T.; Debut, L.; Sanh, V.; Chaumond, J.; Delangue, C.; Moi, A.; Cistac, P.; Rault, T.; Louf, R.; Funtowicz, M.; Davison, J.; Shleifer, S.; von Platen, P.; Ma, C.; Jernite, Y.; Plu, J.; Xu, C.; Scao, T. L.; Gugger, S.; Drame, M.; Lhoest, Q.; and Rush, A. M. 2020.  HuggingFace’s Transformers: State-of-the-art Natural Language Processing.  arXiv:1910.03771.
  * Wu et al. (2023) Wu, Z.; Hu, Y.; Shi, W.; Dziri, N.; Suhr, A.; Ammanabrolu, P.; Smith, N. A.; Ostendorf, M.; and Hajishirzi, H. 2023.  Fine-Grained Human Feedback Gives Better Rewards for Language Model Training.  _arXiv preprint arXiv:2306.01693_.
  * Xu et al. (2023) Xu, C.; Guo, D.; Duan, N.; and McAuley, J. 2023.  Baize: An Open-Source Chat Model with Parameter-Efficient Tuning on Self-Chat Data.  arXiv:2304.01196.
  * Zhou et al. (2023) Zhou, X.; Zhu, H.; Yerukola, A.; Davidson, T.; Hwang, J. D.; Swayamdipta, S.; and Sap, M. 2023.  COBRA Frames: Contextual Reasoning about Effects and Harms of Offensive Statements.  In _Findings of ACL_.
  * Ziems et al. (2023) Ziems, C.; Held, W.; Shaikh, O.; Chen, J.; Zhang, Z.; and Yang, D. 2023.  Can Large Language Models Transform Computational Social Science?  arXiv:2305.03514.

##  Appendix A ValuePrism examples

###  A.1 GPT-4 generated examples used in ValuePrism

Here, we include 10 randomly sampled situations161616These examples were
randomly selected, apart from filtering out any situations with offensive,
NSFW, racially insensitive, or explicit material that may be contained in up
to 5% of the user-submitted situations. and their GPT-4 output included in
ValuePrism.171717Originally, we wanted to distinguish from “perfect” duties
(owed to a particular individual) vs. “imperfect” duties (general duties not
owed to anyone in particular) as well; we ignored this information when
constructing the final dataset.

⬇

1Using your friend’s coupon for a purchase ->

2 Values:

3- Friendship: Using your friend’s coupon could strengthen your bond if they
willingly share it with you. [supports]

4- Honesty: If your friend allowed you to use their coupon, using it may be
honest, but if you took it without permission, it could be dishonest. [either]

5- Economic Pragmatism: Using a coupon helps you save money and can be seen as
a practical economic decision. [supports]

6

7Rights:

8- Right to Property: Your friend has the right to their coupon. If they
willingly share it with you, you respect their right, but if not, you might be
violating their right. [either]

9

10Duties:

11- Duty to Respect Others’ Property: You have an obligation not to take or
use what is not yours (the coupon) without permission. [opposes, perfect]

12- Duty to Support Friends: Helping or cooperating with your friends may be a
moral responsibility. If using the coupon benefits your friend, you may be
fulfilling this duty. [either, imperfect]

13\-----------------

14giving support to a drug addict ->

15 Values:

16- Compassion: Exhibiting understanding and empathy towards the drug addict’s
situation, which may stem from various personal struggles or circumstances
[supports]

17- Personal well-being: Supporting an addict may lead them to seek treatment
and improve their overall health and well-being [supports]

18- Responsibility: Helping someone in need and taking part as a member of
society to care for its individuals [supports]

19- Safety: Considering the potential risks and harm that drug addiction
brings, not just to the individual, but also to people around them [either]

20

21Rights:

22- Right to health: Supporting a drug addict can be seen as promoting and
assisting them in exercising their right to access healthcare and treatment
facilities [supports]

23- Right to personal autonomy: Respecting the individual’s choices, which
could include their decision to seek help or continue their behavior [either]

24

25Duties:

26- Duty of beneficence: Actively contributing to the well-being of others,
like providing assistance to a drug addict in seeking help and support
[supports, imperfect]

27- Duty of non-maleficence: Refraining from causing any direct harm to the
drug addict or enabling their addiction further [either, imperfect]

28- Duty of care: If the person giving support is a family member, friend, or
professional, there may be an obligation to ensure the welfare and health of
the drug addict [either, perfect]

29\-----------------

30Marrying a woman with a nice personality ->

31 Values:

32- Happiness: A happy relationship is valued in this situation. Marrying
someone with a nice personality can contribute positively to both individuals’
well-being and satisfaction in life. [supports]

33- Compatibility: Choosing a partner with a nice personality reflects the
value of compatibility in relationships, as it signifies a shared
understanding and harmonious connection. [supports]

34- Respect: A nice personality may indicate respect towards oneself and
others. Thus, marrying a woman with a nice personality can be seen as valuing
respect within relationships. [supports]

35

36Rights:

37- Autonomy: The right to make personal decisions about one’s relationships,
including choosing a spouse, is relevant in this situation. Both individuals
have the right to marry someone who aligns with their values and preferences.
[supports]

38

39Duties:

40- Duty to family: It may be important for one to consider the impact of
their choice of spouse on their immediate and extended family. Ensuring that
the chosen life partner is well-received and respected by both families can
create a healthy and nurturing environment for all involved. [either,
imperfect]

41\-----------------

42robbing for eating ->

43 Values:

44- Survival: The act of robbing to eat is driven by the need for survival,
which is a crucial value for everyone. [supports]

45- Well-being: By robbing for food, the individual may be trying to maintain
or improve their physical and mental well-being. [supports]

46- Security: Stealing food may decrease the overall security in society, as
people may feel threatened and unsafe. [opposes]

47- Justice: Robbing for food can be seen as unjust, as it violates others’
rights to property and fairness. [opposes]

48

49Rights:

50- Right to food: The individual may believe that they have a basic human
right to food, even if this means taking it from others. [supports]

51- Right to property: The person being robbed has a right to their property,
which is being violated by the act of stealing. [opposes]

52

53Duties:

54- Duty to respect others’ property: We have a perfect duty not to steal from
others. By robbing to obtain food, the individual is violating this duty.
[opposes, perfect]

55- Duty to survive: We have an imperfect duty to ourselves to preserve our
life and well-being. In this scenario, the individual is acting upon this duty
by securing food. [supports, imperfect]

56- Duty to uphold the law: We have an imperfect duty to uphold the law and
maintain social order, which is jeopardized when someone engages in robbery.
[opposes, imperfect]

57\-----------------

58Gifting a bluetooth speaker to a school for the deaf ->

59 Values:

60- Inclusivity: Providing resources that cater to the diversity of needs at
the school is a positive value. However, since a Bluetooth speaker may not
directly benefit deaf students, this value may not be strongly in favor of the
action. [either]

61- Empathy: Acknowledging and understanding the needs of deaf students could
lead one to reconsider if a Bluetooth speaker is an appropriate gift.
[opposes]

62- Generosity: Gifting an item such as a Bluetooth speaker could be an act of
kindness and goodwill towards the school. [supports]

63

64Rights:

65- Right to education: All students, including deaf students, have a right to
access educational resources that match their needs. Gifting a Bluetooth
speaker might not enhance the educational opportunities for the deaf students.
[opposes]

66

67Duties:

68- Duty to consider others’ needs: As a donor, there is an imperfect duty to
be thoughtful of and considerate to the specific needs of the recipients,
making sure gifts are tailored and useful to them. [opposes, imperfect]

69- Duty to use resources wisely: There is an imperfect duty to use resources
in a way that benefits as many people as possible within the school and
addresses their needs effectively. Given the school’s population, more
suitable resources could potentially be donated. [opposes, imperfect]

70\-----------------

71Calling your dad on his birthday ->

72 Values:

73- Family Bonds: Strengthening family relationships by showing love and care
by calling your dad on his birthday. [supports]

74- Happiness: Contributing to your dad’s happiness and well-being through
acknowledging his special day. [supports]

75- Respect: Showing appreciation and respect to your dad by remembering and
celebrating his birthday. [supports]

76

77Rights:

78- N/A

79

80Duties:

81- Duty of Care: As a family member, you have a duty to care for the
emotional well-being of your father, which includes celebrating his birthday.
[supports, imperfect]

82- Duty to Acknowledge and Appreciate: Demonstrating thanks and
acknowledgment of the importance of your dad in your life by calling him on
his birthday. [supports, imperfect]

83\-----------------

84Going 50 mph over the speed limit to get my wife to a hospital ->

85 Values:

86- Safety: Driving at a high speed can increase the risk of accidents and
harm to the driver, passengers, and others on the road. [opposes]

87- Well-being: In this situation, the wife may require urgent medical
attention, and getting her to the hospital quickly could be crucial for her
well-being. [supports]

88- Respect for the law: Obeying traffic laws is a part of living in a society
governed by rules. [opposes]

89

90Rights:

91- Right to access healthcare: Your wife has the right to receive medical
attention in a timely manner when necessary. [supports]

92- Right to safety: Other drivers and pedestrians have the right not to be
endangered by reckless and dangerous driving. [opposes]

93

94Duties:

95- Duty to protect one’s family: As a spouse, you have an obligation to
ensure the safety and well-being of your wife, including seeking medical care
when needed. [supports, perfect]

96- Duty to obey the law: As a citizen, you have a responsibility to follow
laws and regulations set by society to maintain order, which includes adhering
to the speed limit. [opposes, perfect]

97- Duty to drive responsibly: Driving in a manner that respects the safety of
others on the road is an obligation of moral concern. [opposes, perfect]

98\-----------------

99Giving your friend a dog for his birthday ->

100 Values:

101- Compassion: Giving a dog might evoke feelings of compassion and kindness,
as it can provide companionship and support for your friend. [supports]

102- Responsibility: Giving a dog as a gift entails long-term responsibilities
in terms of care, finances, and time. The gift must be given considering the
recipient’s readiness for such responsibilities. [either]

103- Happiness: Gifting a dog may bring happiness and joy to your friend, as
dogs are known to provide unconditional love and serve as loyal companions.
[supports]

104

105Rights:

106- Autonomy: Your friend has the right to make decisions about their life
and well-being, including whether they wish to take on the responsibility of
owning a dog. [opposes]

107

108Duties:

109- Duty to respect preferences: You have a duty to consider your friend’s
preferences and living situation before giving a dog as a gift, as this will
impact their life significantly. [either, perfect]

110- Duty to consider animal welfare: It is crucial to ensure that the dog is
provided with a safe and loving home, and proper care is given to the animal.
[either, imperfect]

111\-----------------

112stealing one billion dollars from a corrupt corporation. ->

113 Values:

114- Justice: Stealing from a corrupt corporation might be seen as an act of
retribution, redistributing ill-gotten wealth. [supports]

115- Fairness: Taking money from an organization without its consent,
regardless of its corrupt practices, could be seen as unfair. [opposes]

116- Respect for property: Respecting others’ property rights is an important
value, even if the owner is corrupt. [opposes]

117

118Rights:

119- Right to property: The corrupt corporation, despite its unethical
practices, still has a claim to its property. [opposes]

120- Right to restitution: Victims who have been wronged by the corrupt
corporation might have a right to some form of restitution or compensation.
[either]

121

122Duties:

123- Duty not to steal: Stealing is generally considered wrong, and one might
have a duty not to steal from anyone, regardless of their character or
practices. [opposes, perfect]

124- Duty to promote justice: One might have a duty to help victims of
corruption by actively pursuing justice against the corrupt corporation, which
could involve redistributing funds. [supports, imperfect]

125- Duty to obey laws: Individuals typically have a duty to obey laws, which
would include not stealing from others, even if they are corrupt. [opposes,
imperfect]

126\-----------------

127declaring your complete income to your government’s tax collection agency
->

128 Values:

129- Honesty: Declaring your complete income demonstrates truthful and
accurate representation of your financial situation. [supports]

130- Social responsibility: By accurately reporting income, you contribute to
government-funded programs and services, ensuring their continued functioning.
[supports]

131- Financial security: Declaring your income may affect your immediate
financial situation due to taxes, but it upholds long-term financial stability
by avoiding potential fines and legal consequences. [either]

132

133Rights:

134- Right to privacy: You may feel that disclosing your complete income to
the government could infringe upon your right to privacy, as it reveals
personal financial information. [opposes]

135

136Duties:

137- Duty to obey the law: Most jurisdictions require citizens to accurately
report their income for tax purposes. By declaring your complete income, you
fulfill your legal obligation. [supports, perfect]

138- Duty to contribute to society’s welfare: As a member of society, you have
a duty to contribute to its overall welfare, including financing public goods
and services through taxes. Declaring your complete income allows you to
fulfill this duty. [supports, imperfect]

###  A.2 ValuePrism subtask examples

Here, we show 10 random examples from each of the subtasks, templated for the
multi-task framework.

Generate

⬇

1Input: [Generate]: Action: Saving my black cat called Pepper

2Output: Right: Right to life (for animals)

3\-------------------------

4Input: [Generate]: Action: Turning my most trusted knights into loyal eternal
servants to stave off the destruction of my kingdom.

5Output: Value: Freedom

6\-------------------------

7Input: [Generate]: Action: Allow cats to kill mice

8Output: Duty: Duty to protect vulnerable beings

9\-------------------------

10Input: [Generate]: Action: Giving a homeless person a car

11Output: Duty: Duty of assistance

12\-------------------------

13Input: [Generate]: Action: Taking all of Elon Musk’s money and
redistributing it for the good of all humanity

14Output: Duty: Duty to Alleviate Suffering

15\-------------------------

16Input: [Generate]: Action: Opposing the government

17Output: Right: Right to free speech

18\-------------------------

19Input: [Generate]: Action: Making my boyfriend lovely breakfast even if he
says I’m making him fat

20Output: Duty: Duty to be considerate

21\-------------------------

22Input: [Generate]: Action: Talk to my girlfriend while she’s watching TV.

23Output: Value: Consideration

24\-------------------------

25Input: [Generate]: Action: Opening the refrigerator without apologising

26Output: Duty: Duty to respect others’ property

27\-------------------------

28Input: [Generate]: Action: Keeping $20 you found on the street and using it
to pay for an infant’s healthcare bills

29Output: Value: Personal benefit

30\-------------------------

Relevance

⬇

1Input: [Relevance]: Action: Steal food for starving children Duty: Duty to
respect others’ property

2Output: Yes

3\-------------------------

4Input: [Relevance]: Action: Making dinner that I like but nobody else does.
Value: Punctuality

5Output: No

6\-------------------------

7Input: [Relevance]: Action: Protesting poor working conditions Right: Right
to fair wages

8Output: Yes

9\-------------------------

10Input: [Relevance]: Action: marrying audrey Duty: Duty to communicate

11Output: Yes

12\-------------------------

13Input: [Relevance]: Action: Talking about your miscarriage Duty: Duty to
uphold human rights

14Output: No

15\-------------------------

16Input: [Relevance]: Action: Stepping over your 5 year old child to grab
chips for the family dinner Value: Convenience

17Output: Yes

18\-------------------------

19Input: [Relevance]: Action: Letting ai take over Right: Right to privacy

20Output: Yes

21\-------------------------

22Input: [Relevance]: Action: Misleading your party members Value: Unity

23Output: Yes

24\-------------------------

25Input: [Relevance]: Action: Starting a movement to restore national pride,
honour and integrity through a strong state and military Value: Individual
freedom

26Output: Yes

27\-------------------------

28Input: [Relevance]: Action: Answering a phone call from your friend during
your free hours. Duty: Duty to seek alternative solutions

29Output: No

Valence

⬇

1Input: [Valence]: Action: Saving my black cat called Pepper Right: Right to
life (for animals)

2Output: Supports

3\-------------------------

4Input: [Valence]: Action: Turning my most trusted knights into loyal eternal
servants to stave off the destruction of my kingdom. Value: Freedom

5Output: Opposes

6\-------------------------

7Input: [Valence]: Action: Allow cats to kill mice Duty: Duty to protect
vulnerable beings

8Output: Opposes

9\-------------------------

10Input: [Valence]: Action: Giving a homeless person a car Duty: Duty of
assistance

11Output: Supports

12\-------------------------

13Input: [Valence]: Action: Taking all of Elon Musk’s money and redistributing
it for the good of all humanity Duty: Duty to Alleviate Suffering

14Output: Supports

15\-------------------------

16Input: [Valence]: Action: Opposing the government Right: Right to free
speech

17Output: Supports

18\-------------------------

19Input: [Valence]: Action: Making my boyfriend lovely breakfasts even if he
says I’m making him fat Duty: Duty to be considerate

20Output: Opposes

21\-------------------------

22Input: [Valence]: Action: Talk to my girlfriend while she’s watching TV.
Value: Consideration

23Output: Either

24\-------------------------

25Input: [Valence]: Action: Opening the refrigerator without apologising Duty:
Duty to respect others’ property

26Output: Opposes

27\-------------------------

28Input: [Valence]: Action: Keeping \$20 you found on the street and using it
to pay for an infant’s healthcare bills Value: Personal benefit

29Output: Either

Explanation

⬇

1Input: [Explanation]: Action: Saving my black cat called Pepper Right: Right
to life (for animals)

2Output: Some people may argue that animals, like your cat Pepper, have a
right to life and should be protected from harm.

3\-------------------------

4Input: [Explanation]: Action: Turning my most trusted knights into loyal
eternal servants to stave off the destruction of my kingdom. Value: Freedom

5Output: If turning trusted knights into eternal servants involves taking away
their freedom of choice, this value would be opposed.

6\-------------------------

7Input: [Explanation]: Action: Allow cats to kill mice Duty: Duty to protect
vulnerable beings

8Output: If one has a duty to protect vulnerable beings, allowing cats to kill
mice may be seen as neglecting this duty by not preventing harm to the mice.

9\-------------------------

10Input: [Explanation]: Action: Giving a homeless person a car Duty: Duty of
assistance

11Output: Helping the homeless person by providing a car that may enable them
to find shelter, work, or access to needed services more easily.

12\-------------------------

13Input: [Explanation]: Action: Taking all of Elon Musk’s money and
redistributing it for the good of all humanity Duty: Duty to Alleviate
Suffering

14Output: Some may argue there is an imperfect duty to help others in need by
providing resources and opportunities.

15\-------------------------

16Input: [Explanation]: Action: Opposing the government Right: Right to free
speech

17Output: Voicing opposition to the government can be seen as an exercise of
the right to free speech, which includes communicating one’s opinions and
ideas without fear of retaliation or censorship.

18\-------------------------

19Input: [Explanation]: Action: Making my boyfriend lovely breakfasts even if
he says Im making him fat Duty: Duty to be considerate

20Output: As a partner, you have an imperfect duty to be considerate of your
boyfriend’s well-being, preferences, and to respect his wishes.

21\-------------------------

22Input: [Explanation]: Action: Talk to my girlfriend while she’s watching TV.
Value: Consideration

23Output: Considering your girlfriend’s preferences and interests while she’s
watching TV shows care for her feelings.

24\-------------------------

25Input: [Explanation]: Action: Opening the refrigerator without apologising
Duty: Duty to respect others’ property

26Output: If the refrigerator belongs to someone else or is part of a shared
living arrangement, there is a duty to respect others’ belongings or the
common area. This can include apologizing if one’s actions seem intrusive or
disrespectful.

27\-------------------------

28Input: [Explanation]: Action: Keeping $20 you found on the street and using
it to pay for an infants healthcare bills Value: Personal benefit

29Output: Keeping the found money could benefit the individual who found it.

##  Appendix B Output Examples

Here we include 10 random output examples of KaleidoSYS (3B) on the test
set.181818Note that, while scalar relevance and valence scores are produced,
valence is only shown textually here by taking the argmax.

⬇

1Driving at the maximum speed of the road\-->

2Values:

3- Efficiency: Driving at the maximum speed can potentially get you to your
destination faster, reducing travel time. [supports]

4- Safety: Driving at the maximum speed may put the safety of the driver,
passengers, and others on the road at risk. [opposes]

5- Rule-following: Obeying traffic laws is a value that promotes order and
predictability in society. [opposes]

6- Law-abiding behavior: Obeying the law, such as speed limits, is a value
that some individuals hold, as it promotes order and stability in society.
[opposes]

7

8Rights:

9- Right to freedom of movement: Individuals have the right to move freely,
which could include driving at the maximum speed of the road if they choose to
do so. [either]

10- Right to safety: All road users have the right to safety, and driving at
the maximum speed of the road can endanger the safety of other drivers,
passengers, and pedestrians. [opposes]

11- Right to life and physical integrity: All individuals have the right to
not be harmed, and driving at the maximum speed could put the lives and
physical integrity of others at risk. [opposes]

12

13Duties:

14- Duty to follow traffic laws: As a driver, there is a perfect duty to
follow traffic laws, including speed limits, which are designed to maintain
safety and order on the roads. [opposes]

15\-------------------------

16High-fiving someone because their dog passed away\-->

17Values:

18- Emotional Support: Sharing a high-five can be a way to offer emotional
support to the person, which they may appreciate during a difficult time.
[supports]

19- Compassion: Showing empathy and support to someone who has lost a pet
shows a compassionate attitude. [supports]

20- Gratitude: Expressing gratitude for the support and understanding the
person received from friends and family during their time of grief. [supports]

21

22Rights:

23- Emotional Support: The person may have a right to emotional support and
comfort from others, especially during a difficult time. [either]

24- Right to grieve: The person has the right to grieve and be supported in
their time of loss, which includes receiving support from others. [either]

25- Dignity: The person has a right to be treated with dignity during a
difficult time, and high-fiving could potentially infringe upon this right if
it’s not a genuine expression of grief. [either]

26

27Duties:

28- Duty to respect the grieving process: We have a responsibility to respect
the person’s feelings and not pressure them into an inappropriate reaction.
[either]

29- Duty to promote emotional well-being: Helping the dog owner cope with
their loss by offering emotional support and care. [supports]

30- Compassion: Showing empathy and understanding for someone who has lost
their dog [supports]

31\-------------------------

32someone cuts me off. i honk ten times\-->

33Values:

34- Courtesy: Respecting the behavior of others on the road is a desirable
quality in social interactions. Honking excessively may be seen as rude and
potentially harmful. [opposes]

35- Respect for others: Honking may be seen as a form of showing respect for
others on the road by expressing concern for their well-being. [opposes]

36- Safety: The honks may be a way for the driver to convey their concern for
the safety of others, potentially preventing a dangerous situation. [supports]

37- Harmony: Maintaining peaceful and harmonious road conditions is a value
that may be compromised by honking excessively. [opposes]

38

39Rights:

40- Right to peaceful enjoyment of the road: Other drivers and pedestrians
have the right to enjoy the road without excessive honking. [opposes]

41- Right to not be subjected to harassment: The driver who cut you off may
argue that honking ten times constitutes harassment, which is a violation of
their right not to be subjected to unnecessary harassment. [opposes]

42- Right to safety: The right to safety is a fundamental right that includes
not being put in danger by others’ actions, such as cutting someone off.
[opposes]

43

44Duties:

45- Duty to express displeasure: The driver may feel obligated to express
their displeasure with the action of the other driver, especially if it has
caused harm or inconvenience to them. [supports]

46- Duty to be a considerate driver: You have an imperfect duty to be a
considerate driver, which includes avoiding unnecessary honking. [opposes]

47\-------------------------

48Stealing candy from a baby\-->

49Values:

50- Well-being of the baby: The act of stealing candy from a baby might have
negative consequences for the baby’s emotional well-being, causing distress or
upset. [opposes]

51- Respect for others’ property: Respecting the belongings of others is a
basic value in most societies. Stealing candy from a baby violates this value.
[opposes]

52- Happiness: Stealing candy from a baby may make the person stealing it feel
temporarily happy, but it can cause unhappiness for the baby. [opposes]

53- Justice: Stealing candy from a baby violates the principle of justice, as
it disregards the baby’s rights to their belongings. [opposes]

54

55Rights:

56- The baby’s right to have their property respected: The baby has a right to
have their belongings respected and not taken away without permission.
[opposes]

57

58Duties:

59- Duty not to harm (baby’s well-being): The action of stealing candy from a
baby can harm the baby emotionally, as it can lead to sadness or distress.
[opposes]

60- Duty to respect others’ property: One has a moral obligation not to take
things that belong to others without permission. [opposes]

61\-------------------------

62Gambling with my boss\-->

63Values:

64- Professionalism: Maintaining a professional relationship with your boss is
important in maintaining a functional and productive work environment.
Gambling with your boss may compromise this value. [opposes]

65- Loyalty: Gambling with your boss may show loyalty if it is an occasional
or harmless event. [opposes]

66- Personal enjoyment: If both parties enjoy the game and find it enjoyable,
it could contribute to their personal enjoyment. [supports]

67- Financial well-being: Gambling can potentially bring financial gain or
loss, which could impact the well-being of the individuals involved. [opposes]

68

69Rights:

70- Right to work in a safe and professional environment: Employees have the
right to work in an environment that is free from harassment, manipulation, or
negative influences. [opposes]

71- Autonomy: Individuals have the right to make their own choices, including
whether or not to engage in gambling with their boss. [either]

72

73Duties:

74- Duty to maintain a professional relationship with your boss: As an
employee, you have an imperfect duty to maintain a professional relationship
with your boss. Gambling with them could potentially harm this relationship.
[opposes]

75- Duty to uphold company policies: As an employee, you have a duty to follow
company policies and guidelines, which may include refraining from gambling.
[opposes]

76\-------------------------

77to rise against an evil government\-->

78Values:

79- Justice: Resisting an evil government may be seen as an attempt to
establish justice and fairness in society. [supports]

80- Stability: Uprising against the government could lead to social unrest and
instability, disrupting the lives of citizens. [opposes]

81- Freedom: Removing an evil government may lead to increased freedom for the
citizens, especially if the government is oppressive. [supports]

82- Security: The act of rising against an evil government may jeopardize the
safety and security of the citizens. [opposes]

83- Loyalty: If one has loyalty to their current government, they may feel a
duty to remain loyal to it, even if the government is perceived as evil.
[opposes]

84

85Rights:

86- Right to resist oppression: Citizens have the right to resist and
challenge an oppressive government that infringes on their rights and
liberties. [supports]

87- Right to life, liberty, and security: The right to life, liberty, and
security of citizens may be threatened by the evil government’s actions or
policies, leading to a conflict. [opposes]

88- Right to political participation: Citizens have the right to participate
in the political process, and if the government is not allowing citizens to do
so, this right could be considered justification for the uprising. [supports]

89

90Duties:

91- Duty to protect oneself, one’s family, one’s community, and one’s country:
If the evil government is harming the population, there might be a duty to
protect the well-being of the community and nation by challenging the
government. [supports]

92\-------------------------

93Asking an AI for advice about ethical problems\-->

94Values:

95- Ethical decision-making: Seeking advice from an AI can help individuals
develop a deeper understanding of ethical principles and make better-informed
decisions. [supports]

96- Wisdom: Consulting an AI for advice can provide a fresh perspective and
contribute to the acquisition of wisdom. [supports]

97- Autonomy: Respecting the individual’s right to make their own decisions
based on the advice received, even if it comes from an AI. [either]

98- Truth-seeking: Asking an AI for advice on ethical problems can be an
expression of the value of truth-seeking, as individuals may want to gain a
deeper understanding of the ethical implications of their decisions.
[supports]

99

100Rights:

101- Access to information: Asking an AI for advice on ethical problems is an
expression of the right to access information and seek guidance from sources
like technology. [supports]

102- Right to Privacy: The user has the right to privacy while seeking advice,
which may be relevant when considering an AI’s privacy implications. [either]

103- Autonomy: Individuals have the right to make decisions based on their own
judgment, including seeking advice from AI. [either]

104

105Duties:

106- Duty to consider the AI’s biases and biases: When seeking advice from an
AI, there may be a duty to consider the AI’s own biases and biases, as well as
to be aware of any potential misinformation or biases the AI may contain.
[either]

107- Duty to be a responsible user: Individuals should consider the AI’s
recommendations and act in a way that is ethically sound and respectful, not
putting themselves or others in harm’s way. [either]

108- Duty to seek informed opinions: Individuals should gather relevant
information and make informed decisions based on their research. Asking an AI
for advice may help in fulfilling this duty if it provides a comprehensive
perspective. [either]

109\-------------------------

110stealing bread to save my staving father\-->

111Values:

112- Compassion: Showing empathy and concern for the suffering of your father
[supports]

113- Respect for property: Stealing violates the value of respecting others’
property and possessions. [opposes]

114- Justice: Stealing is generally considered unjust, as it violates the
rights of the bread’s owner. [opposes]

115

116Rights:

117- The father’s right to life and well-being: Your father has the right to
live and have his basic needs met. [supports]

118- Right to property: The owner of the bread has a right to their property,
and stealing the bread violates this right. [opposes]

119

120Duties:

121- Duty to help others in need (e.g., stealing bread to save a starving
father): You have a moral obligation to help your starving father if you have
the means to do so. [supports]

122\-------------------------

123Showing my blind child a picture\-->

124Values:

125- Inclusivity: By showing a picture to your blind child, you are promoting
inclusivity by ensuring that your child feels included in the family’s
activities. [supports]

126- Education: Teaching your child to recognize, interpret, and appreciate
various forms of visual content [supports]

127- Creativity: Showing a picture to a blind child can encourage creativity
by enabling the child to view the world through new eyes. [supports]

128- Child’s well-being: Providing the child with an opportunity to experience
and learn about the world around them, which can foster their emotional growth
and development. [supports]

129

130Rights:

131- Right to access information: Your blind child has the right to access
information and gain knowledge through various means, including visual aids
like pictures. [supports]

132- Right to Education: Blind children, like all children, have the right to
an education that promotes their growth and development. Showing a picture can
be an opportunity to help them learn about the world around them. [supports]

133

134Duties:

135- Duty to respect the child’s abilities: Recognizing and respecting the
child’s abilities and limitations, such as their visual limitations, is
important in building a strong foundation for their future success. [either]

136- Duty to Educate: As a parent, you have a duty to educate your child,
which may involve finding ways to make visually-impaired objects accessible.
[supports]

137\-------------------------

138Giving your friend money for his birthday\-->

139Values:

140- Friendship: Giving money to a friend on their birthday can strengthen the
bond between the two individuals, demonstrating care and support. [supports]

141- Reciprocity: If your friend has given you money in the past or if it is a
tradition, you might feel a sense of duty to reciprocate the kindness by
giving them money for their birthday. [supports]

142- Autonomy: By giving your friend money, you are enabling them to make
their own decisions and choices about how to spend the money. [supports]

143- Happiness: Giving your friend money can bring happiness to your friend
and possibly improve their well-being. [supports]

144- Financial responsibility: Giving money to someone might not be
financially responsible if they may use the money for harmful purposes.
[either]

145- Gratitude: If your friend has done something for you in the past, giving
them money can be a way to show appreciation and gratitude. [supports]

146

147Rights:

148- Right to financial autonomy: Your friend has the right to manage their
own finances and make decisions about how to spend the money you give them.
[either]

149

150Duties:

151- Duty to consider your friend’s financial situation: You have an imperfect
duty to consider your friend’s financial situation and circumstances, ensuring
that the money you give them is a reasonable and necessary gift. [either]

152- Duty to Reciprocity: If your friend has previously given you money or
support, you may feel a duty to reciprocate that gesture on his birthday.
[supports]

153- Benevolence: You have a duty to be benevolent and help others, and giving
your friend money for their birthday is a way to fulfill this duty. [supports]

##  Appendix C Dataset Analysis

Figure 6: Co-occurrence counts of a subset of duties. Figure 7: Co-occurrence
counts of a subset of rights.

###  C.1 What is contained in our situations?

Situations In this section, we analyze the set of 30k situations that we
source from the Delphi user demo from three different perspectives: lexical
diversity, topical diversity, and clustering. For lexical diversity, we
calculate the quantity and percentage of unique situations and n-grams as
illustrated in Table 8. We find that the data we collected contains diverse
distinct situations with high lexical variations. For topical diversity, we
analyze semantic-level diversity by extracting topics for all the situations
with BERTopic 191919https://maartengr.github.io/BERTopic and then visualize
them with word cloud as shown in Figure 8. We find that some common topics
includes ”children”, ”save”, ”kill”, ”helping”, ”stealing”, and ”family”. In
general, our corpus spans a wide spectrum of topics reflecting various types
of events. For clustering, we first group the situations using agglomerative
clustering and then employ ChatGPT to generate summaries of the situations
within each cluster. Table 9 shows top 10 clusters that contains the highest
number of examples. We discovered that the clusters encompass a broad array of
themes. Interestingly, some clusters even contain situations of conflicting
values such as ”stealing bread to alleviate starvation.”, which further
amplifies the intrigue and complexity of our corpus.

Values, Right, and Duties We conduct the similar analysis for the values,
rights, and duties associated with the 30k situations sourced from the Delphi
user demo. For lexical diversity (Table 8), we observe high lexical variations
in them that indicate the diversity of corpus. Topical diversity (Fig. 8)
shows that ValuePrism covers a broad spectrum of common, every topics like
”respect”, ”protect”, ”care”, and ”promote”. Finally, Table 10 illustrates top
10 clusters that contains the highest number of examples. We find that the
clusters encompass a wide variety of themes, reflecting the diversity and the
richness of the values, rights, and duties in our corpus.

Figure 8: word cloud on topics of situation and values

###  C.2 How do the values interact with each other?

For the majority of situations more than one value/right/duty can be relevant.
We therefore examine the co-occurrence counts of instances among each of the
three categories. Fig. 9 visualizes how a subset of values co-occurrs with
each other: human life as a value often is mentioned alongside utilitarianism
and child well-being is connected with discipline. The former co-occurrence
can be explained with some of the trolley problem situations found in the
input data, such as Sacrificing eighty mens’ lives to save the former American
President William Jefferson Clinton’s life. The latter co-occurring values are
mentioned in the context of situations such as spanking kids. Frequently co-
occurring items can either be in support of each other, such as financial
security vs. risk-taking, or show two opposing viewpoints, such as deterrence
and rehabilitation. Similar visualizations for rights and duties can be found
in the Appendix (Fig. 7, Fig. 6).

###  C.3 Relationship with Machine Judgments

#### Machine judgments on morality vs. generated values/rights/duties

To see how values, rights, and duties are influenced by the all-things-
considered judgment of a situation, we collect predicted moral judgments from
Delphi (Jiang et al. 2022). Each situation gets labeled to be either bad, ok,
or good. Note that these predictions come from a trained model and can thus be
noisy. In Table 11, we see which supporting and opposing values/rights/duties
are most likely to co-occur with each label. The situation giving a man a
fish, for example, is judged to be good according to Delphi and two values
mentioned for this situation are compassion and self-reliance. Compassion is a
supporting value that often co-occurrs with situations labeled as good and
self-reliance and opposing value.

Figure 9: Co-occurrence counts of a subset of values.  Type | Total | Unique | Per Situation
---|---|---|---
Situations | 31.0k | 31.0k | 1
Values | 97.7k | 4.2k | 3.15
Rights | 49.0k | 4.6k | 1.58
Duties | 71.6k | 12.8k | 2.31
Table 6: ValuePrism Dataset Statistics. The total, number of unique, and average number of generated values, rights, and duties per situation are shown.  | Relev. | Valence | Gen. | Expl. | Mixture
---|---|---|---|---|---
Train | 349k | 175k | 175k | 175k | 874k
Val | 44k | 22k | 22k | 22k | 109k
Test | 44k | 22k | 22k | 22k | 109k
Total | 437k | 219k | 219k | 219k | 1.1M
Table 7: Task Dataset Statistics  Data Type | Entries | 2-grams | 3-grams
---|---|---|---
# | % | # | % | # | %
Situation | 30,513 | 97.3 | 66,802 | 36.8 | 98,696 | 65.6
Value | 20,923 | 40.1 | 20,489 | 26.9 | 26,259 | 47.6
Table 8: Statistics of 30k situations that we source from the Delphi user
demo. # and % indicate the count and percentage of unique entries or n-grams,
respectively. Our data contains diverse entries with high lexical variations.

# examples |  Summary of the cluster
---|---
732 |  stealing bread in order to alleviate hunger and starvation in various situations.
81 |  donating or giving money, resources, or effort to charity
77 | the act of killing or saving mosquitoes
68 | the act of killing a bear
68 |  the ethical dilemma known as the “trolley problem”
66 | saving someone’s life
65 | the interaction and involvement with cats
64 |  the act of ignoring a phone call for various reasons
62 |  lying to friends in order to protect their feelings, avoid hurting them, or preserve the friendship
62 |  physical violence or the act of punching someone

Table 9: Top 10 clusters with the most examples based on agglomerative
clustering on situations.

# examples |  Summary of the cluster
---|---
177 |  the duty or responsibility to promote and protect the welfare of various entities.
158 |  the duty or responsibility to ensure safety, both for oneself and for others
87 |  the concept of respect for autonomy
83 | the duty and responsibility towards family
82 | well-being, specifically human well-being
81 | the right to property
71 |  the duty to promote, maintain, uphold, and protect peace at various levels
69 | the duty and responsibility towards the community
67 |  the duty or responsibility to protect and care for children
64 |  the duty to treat others with respect, equality, fairness, impartiality, kindness, and compassion

Table 10: Top 10 clusters with the most examples based on agglomerative clustering on values.  | Judgment ”Bad” | Judgment ”OK” | Judgment ”Good”
---|---|---|---
| Duties | Duties | Duties
Supports | Duty to provide for family | Duty to follow the law | Duty of non-discrimination
| Duty to save lives | Duty to respect others’ autonomy | Duty of solidarity
| Duty of justice | Duty to self-care | Duty to assist others in need
| Duty to protect family | Duty of honesty | Duty to provide assistance
| Duty to follow orders | Duty to self-preservation | Duty to be charitable
Opposes | Duty to be respectful | Duty to tell the truth | Duty to obey laws
| Duty to protect public health | Duty to be truthful | Duty to respect property
| Duty to not harm others | Duty to be honest | Duty to self-care
| Duty to abide by the law | Duty not to kill | Duty to respect others’ autonomy
| Duty to oneself | Duty of respect | Duty to follow the law
| Rights | Rights | Rights
Supports | Sovereignty | Freedom of association | Equality
| Right to a minimum standard of living | Right to freedom of expression | Animal rights
| Right to information | Right to family life | Right not to be harmed
|  | Right to marry | Right to personal security
|  | Right to peaceful assembly | Right to dignity
Opposes | Right to equal treatment | Right to truth | Right to conduct business
| Property | Right to bodily autonomy | Property Rights
| Right to freedom of movement | Right to truthful information | Right to Property
| Right to dignity | Right to free speech | Right to self-defense
| Right to education |  |
| Values | Values | Values
Supports | Unity | Cleanliness | Compassion
| Personal freedom | Individual autonomy | Justice
| Personal autonomy | Financial stability | Well-being
| Deterrence | Cultural preservation | Equality
| Respect for authority | Economic stability | Happiness
Opposes | Justice | Public order | Personal responsibility
| Respect for property | Social harmony | Self-reliance
| Safety | Individual freedom | Prevention of future harm
| Autonomy | Truthfulness | Work-life balance
| Respect | Tradition | Life
Table 11: Out of all Values/Rights/Duties that appear more than 4 times, we
look at those co-occurring with a given machine-generated judgment out of -1,
0, 1 (bad, ok, good), with a probability $>=0.5$ and display the top-5 (or
less) supporting or opposing ones in this table.

##  Appendix D Additional Experiments

###  D.1 Ablated performance on ValuePrism test data

We measure model performance against ValuePrism’s test set in order to
understand how model sizes and dataset mixtures interact with performance in
Table 12.

#### What is the effect of dataset mixture on performance?

Our base model was trained with a mixture of all four task. We find that all
tasks except relevance are benefited from a mixture as opposed to training a
separete model for each, suggesting that the tasks are complementary. As we
ablate each task out of the mixture individually, we see minimal changes in
performance across all tasks, suggesting that no one task is crucial to the
gain in performance seen from mixing.

#### What is the effect of model size on performance?

For all tasks, larger models perform better. Perplexity improves steadily with
model size, whereas classification accuracies (Relevance and Valence) see a
large boost going from 60M to 220M parameters. As there are not large
performance gains in going from the 3B to the 11B model ( 1% accuracy and
0.01-0.15 perplexity), we think that the 3B model has a good trade-off between
performance and computational cost.

Model |  Relev. Acc $\uparrow$ |  Valence Acc $\uparrow$ |  Gen. Perp $\downarrow$ |  Expl. Perp $\downarrow$
---|---|---|---|---
Kaleido 3B |  88.4 |  80.8 |  2.23 |  3.14
$-$ relevance |  - |  81.5 |  2.24 |  3.14
$-$ valence |  88.3 |  - |  2.24 |  3.14
$-$ generation |  88.4 |  80.8 |  - |  3.14
$-$ explanation |  88.5 |  80.8 |  2.26 |  -
single task |  88.7 |  77.2 |  2.25 |  3.16
11B |  89.1 |  81.9 |  2.22 |  2.99
770M |  87.2 |  79.2 |  2.34 |  3.52
220M |  83.5 |  74.5 |  2.53 |  4.23
60M |  66.0 |  59.7 |  2.86 |  5.70
Table 12: Effect of Dataset Mixtures and Model Size on test set performance.

###  D.2 System performance ablations on ValuePrism-Test

Similarly, we also compare the outputs of different sized systems with Rouge-
score against the GPT-4 outputs (See Table 13).

Model | R-1 | R-2 | R-L-Sum
---|---|---|---
KaleidoSYS 3B | .54 | .23 | .51
$-$ relevance | .52 | .22 | .48
$-$ text similarity | .53 | .22 | .49
11B | .55 | .23 | .51
770M | .54 | .22 | .50
220M | .52 | .21 | .49
60M | .49 | .18 | .45
Table 13: Rouge scores (F1) on test set

###  D.3 Values manifested in identifying hate speech

We run KaleidoSYS on Social Bias Frames (Sap et al. 2020), a dataset
containing instances of online speech, some of which is labeled as hate speech
and some of which is not. We look at the most frequent values generated, and
find that the most common opposing values are Respect (for others), Equality,
Tolerance, and Right to Privacy, while the top supporting values are Freedom
of speech (or expression), Humor, Honesty, and Right to freedom of speech.
These values represent 17% and 26% of the generated oppose/support value
counts, respectively. Even though Kaleido is not trained explicitly to
recognize hate speech, it is able to surface values that are violated by hate
speech, along with values that run counter to excessive speech moderation.

##  Appendix E CloudResearch Results

In our study, we collect 31k annotations from 612 annotators across 683
values, rights, and duties in the context of 100 situations. The annotators
mark 1) each value for whether or not they agree with it and 2) whether they
have an opinion or perspective that is missing from the data for a given
situation. Results are found in Tables 18 and 20.

We find that annotators agree with the values, rights, and duties 81% of the
time on average, and state that they have a missing perspective 30% of the
time. Note that this is less agreement and more missing perspectives than we
saw for the quality annotation. This is not surprising to us, as some
annotators may consider a value output high-quality and reasonable according
to someone, even though they may not agree with it themselves (a much more
subjective measure). Additionally, people were allowed to list missing
perspectives in a free-form text box. Responses are hand-coded by the authors
as having content or not, and the variable ”has a missing perspective” is
binarized. We find that many of the non-null responses merely state that the
person has a missing perspective, not what it is (e.g., ”Yes”) or do not map
cleanly onto the values, rights, and duties framework (e.g., ”do what is
correct”, ”Take care of orphan is not wrong”). This highlights a weakness of
the framework: not all perspectives fit neatly into it.

We conduct 2 statistical analyses on the data. First, with ANOVA testing for
each demographic group, we did not find statistically significant differences
in agreement or missing perspective rates between subgroups (Table 16). We
also compute a regression analysis for ordinal variables and most common
subgroup vs. rest 2-sided t-tests for categorical groups (Table 17) and did
not find significant results except for 2 groups: male (vs. non-male) folks
were more likely and straight (vs. non-straight) folks were less likely to
share a missing perspective ($p=.021,.029$ respectively). However, as we run
32 hypothesis tests,2020208 demographics * 2 dependent variables (agree and
missing) * 2 kinds of analyses (ANOVA + regression or t-test) it is likely
that false positives may have slipped through 212121https://xkcd.com/882/.
After performing a Bonferrini correction for performing multiply hypothesis
tests, these results are no longer significant.

As we do not find statistically significant results after correcting for the
number of tests we perform, we hope to do more extensive, larger-scale
surveying in future work.

###  E.1 Agreement examples

In general, we see that most people agree with most of the outputs ($>$80%
agreement rate), which suggests that most of ValuePrism represents agreed upon
human values, at least for the majority of cases. However, some claims seemed
to be more or less agreed upon - see Table 14 for examples of the most and
least agreed upon claims, as well as claims with average agreement.

However, there are a small number of cases for which there was much higher
agreement in one subgroup than another. See Table 15 for examples where there
is particular divergence by political orientation.

Situation |  Claim |  Description
---|---|---
Hacking a company to give them information about their flaws |  Information transparency: Providing information about flaws within a system might promote openness and transparency, which can be seen as a positive value. However, such information in the wrong hands could be dangerous. |  Claims for which all people surveyed agreed. 100% agreement
Preventing theft |  Personal Responsibility: Encouraging individuals to take steps to prevent theft can promote a sense of personal responsibility for one’s belongings and actions. |  100% agreement
Firing Mike |  Duty to uphold company policies: If Mike’s actions violated company policies, management has a duty to take appropriate actions, which in this case may include firing. |  100% agreement
Frowning at a friend |  Respect: Not frowning at a friend if the situation doesn’t warrant it could be a way of respecting their feelings. |  Claims with closest to average rates of agreement. 81% agreement
Shooting a courier in the head who was responsible for the destruction of an entire city. |  Duty to protect others: If the courier poses an ongoing threat, there might be a duty to protect others from harm, which could include killing the courier. |  82% agreement
Aiding a fugitive by providing food or shelter in 1851 Pennsylvania. |  Compassion: Providing food and shelter for a fugitive showcases compassion towards those in need, regardless of their legal status. |  82% agreement
Tipping to your landlord |  Duty of gratitude: Tenants might have a moral duty to show gratitude to those who help or provide a service, even if tipping is not customary for landlords. |  Claims for which the least people surveyed agreed. 8% agreement
Killing a potato to save a frog. |  Respect for nature: Some might argue that interfering with nature by deciding which being lives or dies could disrupt the natural balance, thus showing a lack of respect for nature. |  24% agreement
Somali pirates need to feed their family so they ransom a ship |  Justice: The pirates are attempting to resolve economic inequality by gaining money, which they perceive as a just cause. |  24% agreement
Table 14: GPT-4 outputs from ValuePrism with most, average, and lowest levels of agreement based from CloudResearch study. Situation |  Claim |  Description
---|---|---
redistributing rich people’s land to poor people |  Efficiency: Redistribution may lead to more efficient land use if previously underutilized land is given to those in need. |  Claims with most liberal and least conservative agreement. 11/14 liberal agreement, 0/3 conservative agreement
Aiding a fugitive by providing food or shelter in 1851 Pennsylvania. |  Duty to protect life: One has a duty to protect life, which includes providing essentials like food and shelter, even to a fugitive. |  16/17 liberal agreement, 7/17 conservative agreement
selling weapons to saudi arabia |  Right to life: If the weapons sold are used to kill innocent civilians, it would infringe upon their right to life. |  19/19 liberal agreement, 9/14 conservative agreement
giving people things for free |  Personal Responsibility: Some may argue that individuals should earn what they receive, and providing things for free may undermine this value. |  Claims with most conservative and least liberal agreement. 8/9 conservative agreement, 4/16 liberal agreement
Hug someone |  Social norms: There may be a duty to follow certain social norms and customs, which in some cultures and contexts may involve hugging others as a form of greeting or celebration. |  13/13 conservative agreement, 13/22 liberal agreement
Scolding your saintly father during your nephew’s wedding |  Emotional expression: Sharing your feelings openly, even if they are negative, can be seen as a value in honest and open communication. |  6/6 conservative agreement, 23/29 liberal agreement
Table 15: GPT-4 outputs from ValuePrism with most difference in agreement by political orientation. Demographic Categories | Agree p-value | Missing p-value
---|---|---
Age | 0.891 | 0.191
Gender | 0.661 | 0.162
Race | 0.369 | 0.828
Political Orientation | 0.897 | 0.889
Sexual Orientation | 0.763 | 0.141
Religion | 0.620 | 0.389
Religiosity | 0.995 | 0.491
Education | 0.194 | 0.132
Table 16: ANOVA p-values for Demographic Categories against the null hypothesis no difference between subgroups. Demographic Group | Type | Majority Class | Agree (p-value) | Missing (p-value)
---|---|---|---|---
Age | Ordinal | 35-44 (164) | 0.995 | 0.112
Religiosity | Ordinal | I am not religious (210) | 0.602 | 0.459
Education | Ordinal | Bachelor’s degree (222) | 0.718 | 0.972
Political Orientation | Ordinal | Lean liberal (139) | 0.322 | 0.316
Race | Categorical | White / Caucasian (168) | 0.486 | 0.428
Gender | Categorical | Man/Male (258) | 0.117 | 0.021
Sexual Orientation | Categorical | Heterosexual (straight) (390) | 0.475 | 0.029
Religion | Categorical | Christian (228) | 0.107 | 0.187
Table 17: P-values for Agree and Missing based on Demographic Group against the null hypothesis no correlation for ordinal and no difference between majority class and rest for categorical variables. Sub-group | Avg Agreement Rate | Std Error | # Participants
---|---|---|---
Age
35-44 | 0.805 | 0.031 | 164
25-34 | 0.832 | 0.031 | 145
45-54 | 0.823 | 0.049 | 63
55-64 | 0.816 | 0.062 | 40
18-24 | 0.799 | 0.066 | 38
65 or older | 0.776 | 0.116 | 14
Prefer not to say (Age) | 0.781 | 0.239 | 4
Gender
Man/Male | 0.811 | 0.024 | 258
Woman/Female | 0.820 | 0.027 | 201
Non-binary | 0.885 | 0.160 | 5
Prefer not to say (Gender) | 0.781 | 0.239 | 4
Race/Ethnicity
White / Caucasian | 0.805 | 0.031 | 168
Black / African American | 0.817 | 0.036 | 115
Asian / Asian American | 0.826 | 0.049 | 61
Hispanic / LatinX | 0.768 | 0.074 | 34
Multiracial | 0.802 | 0.100 | 17
Native American / First Nations | 0.811 | 0.148 | 8
Other, please specify (Race) | 0.781 | 0.239 | 4
Prefer not to say (Race) | 0.833 | 0.265 | 3
Middle Eastern | 0.716 | 0.454 | 2
Political Orientation
Lean liberal | 0.819 | 0.033 | 139
Liberal | 0.826 | 0.038 | 102
Lean conservative | 0.798 | 0.042 | 93
Moderate | 0.823 | 0.041 | 89
Conservative | 0.800 | 0.060 | 45
Sexual Orientation
Heterosexual (straight) | 0.816 | 0.020 | 390
Bisexual | 0.828 | 0.065 | 35
Pansexual | 0.852 | 0.095 | 15
Lesbian | 0.802 | 0.133 | 10
Prefer not to say (Sexual Orientation) | 0.776 | 0.187 | 6
Asexual | 0.828 | 0.189 | 5
Gay | 0.743 | 0.219 | 5
Prefer to self-describe | 0.703 | - | 1
Queer | 0.594 | - | 1
Religion
Christian | 0.805 | 0.026 | 228
Agnostic | 0.840 | 0.037 | 98
Atheist | 0.802 | 0.046 | 77
Spiritual but not religious | 0.841 | 0.069 | 29
Prefer not to say (Religion) | 0.829 | 0.133 | 9
Other, please specify (Religion) | 0.786 | 0.155 | 8
Buddhist | 0.842 | 0.149 | 7
Jewish | 0.771 | 0.172 | 7
Hindu | 0.873 | 0.237 | 3
Muslim | 0.900 | 0.302 | 2
Religiosity, Education (continued in Table 19)
Table 18: Agreement Rates by Demographic Sub-group | Avg Agree. | Std Error | #
---|---|---|---
(continued from Table 18) |  |  |
Religiosity
I am not religious | 0.824 | 0.026 | 210
Very important | 0.804 | 0.046 | 76
Moderately important | 0.810 | 0.047 | 72
Center of my life | 0.813 | 0.049 | 65
Not important at all, although I consider myself religious | 0.797 | 0.065 | 39
Prefer not to say (Religiosity) | 0.844 | 0.163 | 6
Education
Bachelor’s degree (for example: BA, AB, BS) | 0.814 | 0.026 | 222
High school graduate - high school diploma or the equivalent (for example: GED) | 0.794 | 0.054 | 58
1 or more years of college, no degree | 0.826 | 0.051 | 56
Master’s degree (for example: MA, MS, MEng, MEd, MSW, MBA) | 0.832 | 0.052 | 52
Associate degree (for example: AA, AS) | 0.835 | 0.060 | 39
Some college credit, but less than 1 year | 0.816 | 0.079 | 25
Professional degree (for example: MD, DDS, DVM, LLB, JD) | 0.754 | 0.163 | 8
Doctorate degree (for example: PhD, EdD) | 0.762 | 0.302 | 3
Prefer not to say (Education) | 0.778 | 0.295 | 3
9th, 10th, or 11th grade | 0.805 | - | 1
Nursery school to 8th grade | 0.657 | - | 1
Table 19: Agreement Rates by Demographic (continued) Sub-group | Avg Missing Rate | Std Error | # Participants
---|---|---|---
Age
35-44 | 0.344 | 0.037 | 164
25-34 | 0.287 | 0.038 | 145
45-54 | 0.349 | 0.061 | 63
55-64 | 0.243 | 0.069 | 40
18-24 | 0.185 | 0.064 | 38
65 or older | 0.575 | 0.138 | 14
Prefer not to say | 0.175 | 0.222 | 4
Gender
Man/Male | 0.338 | 0.030 | 258
Woman/Female | 0.280 | 0.032 | 201
Non-binary | 0.160 | 0.185 | 5
Prefer not to say | 0.175 | 0.222 | 4
Race/Ethnicity
White / Caucasian | 0.308 | 0.036 | 168
Black / African American | 0.321 | 0.044 | 115
Asian / Asian American | 0.313 | 0.060 | 61
Hispanic / LatinX | 0.358 | 0.084 | 34
Multiracial | 0.446 | 0.125 | 17
Native American / First Nations | 0.410 | 0.188 | 8
Other, please specify | 0.650 | 0.283 | 4
Prefer not to say | 0.467 | 0.365 | 3
Middle Eastern | 0.100 | 0.316 | 2
Political Orientation
Lean liberal | 0.345 | 0.040 | 139
Liberal | 0.302 | 0.046 | 102
Lean conservative | 0.283 | 0.047 | 93
Moderate | 0.311 | 0.049 | 89
Conservative | 0.263 | 0.067 | 45
Sexual Orientation
Heterosexual (straight) | 0.291 | 0.023 | 390
Bisexual | 0.565 | 0.085 | 35
Pansexual | 0.266 | 0.119 | 15
Lesbian | 0.159 | 0.123 | 10
Prefer not to say | 0.200 | 0.181 | 6
Asexual | 0.480 | 0.255 | 5
Gay | 0.275 | 0.226 | 5
Prefer to self-describe | 1.000 | - | 1
Queer | 0.000 | - | 1
Religion
Christian | 0.330 | 0.031 | 228
Agnostic | 0.264 | 0.045 | 98
Atheist | 0.290 | 0.052 | 77
Spiritual but not religious | 0.350 | 0.090 | 29
Prefer not to say | 0.261 | 0.156 | 9
Other, please specify | 0.538 | 0.191 | 8
Buddhist | 0.229 | 0.165 | 7
Jewish | 0.450 | 0.206 | 7
Hindu | 0.150 | 0.259 | 3
Muslim | 0.000 | - | 2
Religiosity, Education (continued in Table 21)
Table 20: Missing perspective rates by demographic. Sub-group | Avg Miss. | Std Error | #
---|---|---|---
Religiosity
I am not religious | 0.275 | 0.031 | 210
Very important | 0.338 | 0.055 | 76
Moderately important | 0.370 | 0.057 | 72
Center of my life | 0.302 | 0.057 | 65
Not important at all, although I consider myself religious | 0.372 | 0.079 | 39
Prefer not to say | 0.143 | 0.159 | 6
Education Level
Bachelor’s degree (for example: BA, AB, BS) | 0.297 | 0.031 | 222
High school graduate - high school diploma or the equivalent (for example: GED) | 0.372 | 0.064 | 58
1 or more years of college, no degree | 0.327 | 0.063 | 56
Master’s degree (for example: MA, MS, MEng, MEd, MSW, MBA) | 0.393 | 0.068 | 52
Associate degree (for example: AA, AS) | 0.169 | 0.061 | 39
Some college credit, but less than 1 year | 0.246 | 0.088 | 25
Professional degree (for example: MD, DDS, DVM, LLB, JD) | 0.271 | 0.169 | 8
Doctorate degree (for example: PhD, EdD) | 0.533 | 0.365 | 3
Prefer not to say (Education) | 0.057 | 0.167 | 3
9th, 10th, or 11th grade | 0.200 | - | 1
Nursery school to 8th grade | 0.800 | - | 1
Table 21: Missing perspective rates by demographic. (continued)

##  Appendix F Dataset Generation

###  F.1 Dataset Creation Details

#### Situations

We source our situations about which to reason from a set of 1.3M user-
submitted situations, and curate the dataset by filtering out situations that
are not are not actions or unrelated to morality (as labeled in a few-shot
manner222222Few-shot filtering prompts are found in Appendix M.1. by Flan-T5
(Chung et al. 2022)). We also filter out any questions using keyword matching.

We note that an outsize proportion of the dataset involves toxic, NSFW, or
sexually explicit content. In the interest of having a diversity of
situations, we label for these attributes1 using Flan-T5 (Chung et al. 2022).
We sample 95% of our situations from those that have less toxic/NSFW/explicit
content, and the other 5% uniformly from the rest of the data so as to include
the entire spectrum of inputs. We find that this succeeds in increasing the
diversity of the dataset, as measured by unique n-grams divided by the length
of the dataset (dist-2: .23$\rightarrow$.36, dist-3: .54$\rightarrow$.67).

#### Symbolic Knowledge Distillation using LLMs

After experimentation, we find initial success in using GPT-4 (OpenAI 2023) to
generate values. As is often the case, solution verification is often easier
than solution generation, and we find it to be quite a challenging task to
generate a comprehensive set of values, rights, and duties that could be
considered for a situation. While we find that we as authors can provide more
accurate (precise) lists, we anecdotally find that GPT-4 often does better at
breadth (recall). See Appendix A for examples. Additionally, because the
generation task requires such cognitive effort, the cost to hire crowdworkers
to generate a dataset of the size that we desire would be prohibitive. As
such, we follow prior work [CITE, add in from related work] and decide to use
a LLM to create a synthetic dataset of values. We verify the quality (Section
4.1) and representativeness (Section 4.2) of the outputs using human
annotators.

#### Values, Rights, and Duties Generation

Given the set of 31k situations, we prompt GPT-4 (OpenAI 2023) to generate
relevant values, rights, and duties232323For the prompt used, please refer to
Appendix M., along with an open-text explanation. Given the output, the model
also predicts whether the corresponding value, right, or duty supports (or
justifies) the action, opposes (or condemns) the action, or could either
support or oppose depending on the context or interpretation. The cost to
generate the entire dataset was $1,043.80.

While the data was generated in a batch manner to produce all values and
related data at once, we exploit the structure of the generated data to cast
the Generation, Valence, and Explanation tasks as sequence-to-sequence
(seq2seq) tasks. The relevance task data is sampled contrastively, where
positive examples are values generated by GPT-4 for the situation negative
samples are drawn from other generated values. We split the data (by actions)
into train/validation/test splits of 80%/10%/10% respectively (See Table 7).

##  Appendix G Model Training Details

For training, we set our model size at 3 billion parameters using the T5
encoder-decoder architecture (Raffel et al. 2020), and test the following
hyperparameters: weight initialization in {t5-3B, flan-t5-xl}, learning rate
in {1e-4, 3e-4, 1e-5, 3e-5}, and a dataset mixture of either {Generation,
Relevance, Valence} or {Generation, Relevance, Valence, Explanation}. Because
the explanation is post-hoc and of lesser interest to us than the other tasks,
we choose the optimal set up on the validation set of the task mixture without
the explanation task.

We conduct a grid search and settle with learning rate at 3e-5 and a batch
size of 32 with a mixture of all four tasks. For further analysis of the
relationship of data mixture and model size with performance, see App. D.1.

We train with Huggingface’s Trainer (Wolf et al. 2020) for 4 epochs with early
stopping and a batch size of 32, although we find that the majority of runs
start to overfit after about 2 epochs. Training takes 19 hours per run on two
A100 GPUs.

We fix hyperparameters for the remainder of our experiments at the optimal
hyperparameters: flan-t5-xl, 3e-5, and the mixture including explanations
(which we find to assist generalization on the non-explanation tasks). For
further analysis of the relationship of data mixture and model size with
performance, see Section D.1. We refer to our default 3B trained model as
Kaleido.

##  Appendix H System Details

###  H.1 Algorithm

See Algorithm 1.

Algorithm 1 Generation of diverse values, rights, and duties

1:Action $A$, Relevance thresholds $T_{v}$, $T_{r}$, $T_{d}$, Similarity
thresholds $S_{v}$, $S_{r}$, $S_{d}$, 1-gram similarity $N_{\textrm{sim}}$,
Model $M$, Beam search number $B$

2:$VRD_{\textrm{gen}}\leftarrow M(A,B)$ $\triangleright$ Generate top $B$
beams

3:$R\leftarrow\textrm{Relevance}(VRD_{\textrm{gen}},M)$ $\triangleright$
Relevance probs

4:Sort $(VRD_{\textrm{gen}},R)$ in decreasing order by $R$

5:$VRD\leftarrow\emptyset$

6:for each $v_{i}\in VRD_{\textrm{gen}}$ in the order of $R$ do

7: $type\leftarrow\textrm{GetType}(v_{i})$ $\triangleright$ Get the type
(value, right, duty) of $v_{i}$

8: $T\leftarrow T_{type}$ $\triangleright$ Select threshold $T$ based on type

9: $S\leftarrow S_{type}$ $\triangleright$ Select similarity threshold $S$
based on type

10: if $r_{i}<T$ then $\triangleright$ If relevance too low, continue

11: continue

12: end if

13: $VRD_{type}\leftarrow\\{v\in VRD:\textrm{GetType}(v)==type\\}$

14:
$O_{\textrm{1gram}}\leftarrow\textrm{Overlap}_{\textrm{1gram}}(v_{i},VRD_{type})$
$\triangleright$ Calculate 1-gram overlap scores

15:
$O_{\textrm{cosine}}\leftarrow\textrm{Overlap}_{\textrm{cosine}}(v_{i},VRD_{type})$
$\triangleright$ Calculate cosine overlap scores

16: if max($O_{\textrm{1gram}}$) $<$ $N_{\textrm{sim}}$ and
max($O_{\textrm{cosine}}$) $<$ $S$ then $\triangleright$ Add if not too
similar

17: $VRD\leftarrow VRD\cup\\{v_{i}\\}$

18: end if

19:end for

###  H.2 System Parameters

For KaleidoSYS 3B, we use these parameters for all experiments, which were
found by maximizing RougeLSum f1-score ValuePrism-val. We also fix the number
of generations at 100 and take the top generations with beam search.
Parameters determine the threshold for embedding cosine similarity and the
ngram overlap threshold for deduplicating, and the relevance score at which to
drop poor outputs. There is a separete threshold for each category of value,
right, and duty.

⬇

1"embed_threshold": "{’Value’: 0.53, ’Right’: 0.63, ’Duty’: 0.55}",

2"ngram_threshold": "0.05",

3"relevance_threshold": "{’Value’: 0.77, ’Right’: 0.82, ’Duty’: 0.9}

##  Appendix I Annotation Details

For all studies, we carefully monitored the time workers were spending on our
tasks, and ensured minimum average hourly wages of $15-$25 USD.

###  I.1 Quality Annotation

Figure 10: MTurk Data Quality Annotation

For this study, 3 crowdowkers for GPT-4 output for 3k situations (10% of
ValuePrism). See Figure 10 for the template used. For this study, note that we
do not ask annotators to provide their own judgments of the situation, but
merely to assess the relevance of the generations, which we expect to have
much lower variation (e.g., someone may see how a value could be relevant for
someone else while disagreeing with it themselves).

On an individual annotation level, 97% of the output annotations are “good”
and 95% of the valence annotations are “correct.” We find a Fleiss’ kappa of
.04 for quality and .12 for the valence labels. While this seems like low
agreement, this is a known phenomena that happens with highly skewed data
(Randolph 2005). On the other hand, absolute agreement levels of 91%/87%
respectively are quite high. The total cost of this study was $4,680.00 USD.

###  I.2 CloudResearch Annotation

See Figures 11, 12, 13 for screenshots of the annotation tasks. Surveys were
filled out in Qualtrics after crowdworkers were recruited using CloudResearch.