# Harnessing Collective Intelligence Under a Lack of Cultural Consensus

Necdet Gürkan
The School of Business
Stevens Institute of Technology
Hoboken, New Jersey
<EMAIL_ADDRESS>
&Jordan W. Suchow
The School of Business
Stevens Institute of Technology
Hoboken, New Jersey
<EMAIL_ADDRESS>

###### Abstract

Harnessing collective intelligence to drive effective decision-making and
collaboration benefits from the ability to detect and characterize
heterogeneity in consensus beliefs. This is particularly true in domains such
as technology acceptance or leadership perception, where a consensus defines
an intersubjective truth, leading to the possibility of multiple “ground
truths” when subsets of respondents sustain mutually incompatible consensuses.
Cultural Consensus Theory (cct) provides a statistical framework for detecting
and characterizing these divergent consensus beliefs. However, it is
unworkable in modern applications because it lacks the ability to generalize
across even highly similar beliefs, is ineffective with sparse data, and can
leverage neither external knowledge bases nor learned machine representations.
Here, we overcome these limitations through Infinite Deep Latent Construct
Cultural Consensus Theory (idlc-cct), a nonparametric Bayesian model that
extends cct with a latent construct that maps between pretrained deep neural
network embeddings of entities and the consensus beliefs regarding those
entities among one or more subsets of respondents. We validate the method
across domains including perceptions of risk sources, food healthiness,
leadership, first impressions, and humor. We find that idlc-cct better
predicts the degree of consensus, generalizes well to out-of-sample entities,
and is effective even with sparse data. To improve scalability, we introduce
an efficient hard-clustering variant of the idlc-cct using an algorithm
derived from a small-variance asymptotic analysis of the model. The idlc-cct,
therefore, provides a workable computational foundation for harnessing
collective intelligence under a lack of cultural consensus and may potentially
form the basis of consensus-aware information technologies.

_K_ eywords collective intelligence, consensus beliefs, bayesian modelling,
cultural consensus theory

##  1 Introduction

Collective intelligence refers to decision-making processes that draw on the
collective opinions of multiple individuals. This often results in higher-
quality decisions compared to those made by a single individual, a phenomenon
commonly termed the “wisdom of the crowd” [1, 2, 3]. A multiplicity of methods
can serve as information-pooling mechanisms to derive collective intelligence,
ranging from structured communication techniques to prediction markets and
algorithmic aggregation methods. Technologies that implement these methods and
facilitate the sharing of opinions and knowledge have further enhanced
collective intelligence, leading to innovative approaches and methodologies
for problem-solving [4]. This integration is readily apparent in practical
applications; for instance, businesses now routinely leverage crowdsourcing
platforms to perform tasks such as creative design, ideation, and prediction
[5, 6, 7].

From their origins in taking the median response of the group [8], traditional
aggregation and belief-merging methods have been premised on the notion that
each respondent provides a partial view of an objective truth (e.g., the
weight of the ox) or of a single intersubjective consensus truth (e.g., who is
the group’s leader), while acknowledging that individuals may have unequal
domain knowledge, vigilance, or bias that causes their responses to vary [9,
10, 11]. The effectiveness of aggregation methods for deriving collective
intelligence then hinges on their ability to parcel out these sources of
signal, noise, and bias in the judgments respondents make, with more effective
aggregation mechanisms coming closer to that objective or intersubjective
truth.

However, these methods are often applied in domains where there is a lack of
cultural consensus, with different subsets of people forming conflicting
culturally constructed beliefs. For example, in one subset, a leader may be
perceived as someone who is authoritative and decisive, while in another
subset, leadership may be associated with collaborative and inclusive
decision-making [12]. One subset may view face tattoos as taboo, whereas
another may find them to be stylish and meaningful expressions of individual
or group identity [13]. A subset may avoid certain technologies because of
privacy concerns, while another views them as vital for improving the quality
of life [14]. It quickly becomes evident that many concepts are socially
learned and deeply influenced by respondents’ personal and cultural beliefs,
leading to the possibility of multiple incommensurate views [15].

Cultural Consensus Theory (cct) provides a statistical framework for
information pooling in domains where there may be a lack of cultural
consensus, enabling those who use it to infer the beliefs and attitudes that
influence social practices and the degree to which respondents know or express
those beliefs [16]. Consensus models based on cct then provide an opportunity
to simultaneously study both individual and group-level differences by
examining the extent to which a respondent conforms to the consensus within
one or more subsets, and facilitating the representation of how people differ
in terms of their level of knowledge and response biases. Researchers have
applied the cct framework to find a practical and concise definition of
beliefs that are accepted by a group sharing common knowledge. Applications
are found in a wide variety of domains, including cognitive evaluation [17],
organizational culture [18], and online communities [19].

However, cct has several limitations that preclude its use as a computational
foundation for harnessing collective intelligence in modern applications.
First, it treats each question–answer pair independently, which prevents
generalization across questions. This hinders the model’s performance in the
sparse data regime and leads to a cold-start problem when the model cannot
generalize to new questions, even those highly similar to questions previously
asked. Therefore, the number of questions required to characterize a culture’s
consensus beliefs scales linearly with the number of culturally held beliefs.
Finally, cct can leverage neither existing structured knowledge bases nor
pretrained learned representations that could provide relevant information
about known entities and their relations. These constraints hinder the
application of cct in more complex or dynamic cultural contexts where the
interrelatedness of beliefs and the availability of preexisting knowledge
could play a pivotal role.

These issues find solution in modern machine-learning techniques that enable
one to make predictions about out-of-sample items by learning a model over
latent representations. Machine learning algorithms can analyze vast
quantities of data from various modalities, including text, images, and audio,
and identify patterns and relationships that generalize beyond the training
data [20, 21, 22]. Leveraging these powerful techniques, it is now possible to
create vector-feature representations of words, sentences, visual scenes, and
images of objects. These high-dimensional representations at times approximate
human mental representations [23]. Although they are not comprehensive
theories of human cognition, vector representations of various real-world
objects and concepts have been used as inputs to linear models that can
predict individual and aggregate evaluations on a wide range of topics,
including perceived risk [24], first impressions based on faces [25, 26],
perceptions of leadership [27], and evaluation of creative writing [28].

Even so, machine-learning methods are not immune to the very same limitations
that beset traditional aggregation methods, and they similarly fail in domains
that lack a clear cultural consensus. For instance, a computer vision
algorithm might be used to predict a label from an image without recognizing
that different subsets of respondents have distinct normative responses
regarding which label ought to be applied. Or, a natural-language processing
algorithm might be used to predict whether user-generated content is
consistent with the norms of an online community, without recognizing that
different subsets of respondents have distinct normative responses regarding
what content is deemed acceptable. This is particularly relevant in domains
where a consensus among respondents defines an intersubjective truth, leading
to the possibility of multiple “ground truths” when subsets of respondents
sustain mutually incompatible consensuses.

To combine the strengths of cct and machine-learning methods while addressing
their respective limitations, we propose the idlc-cct, an extension to the cct
that (1) allows culturally held beliefs to take the form of a deep latent
construct: a fine-tuned deep neural network that maps the features of a
concept or entity to the consensus response among a subset of respondents, and
(2) draws these deep latent constructs from a Dirichlet Process using the
stick-breaking construction. The approach therefore aligns pretrained machine
representations to both group- and individual-level judgments, effectively
capturing variations in belief processes and behaviors across them under a
lack of cultural consensus. We evaluate the idlc-cct on people’s judgments of
various phenomena, including risk sources, leadership effectiveness, first
impressions of faces, and humor. This refined technique has broad scientific
and practical applicability in domains where (1) social scientists and
organizations currently study group-level variation but do not leverage
machine-learning methods to gain insight, and (2) computer scientists and
behavioral scientists apply machine-learning methods to harness collective
intelligence without considering conflicting culturally constructed beliefs.

The plan for the paper is as follows. In Section 2, we introduce advancements
in deep learning and applications of word, sentence, and image embeddings.
Section 3 reviews the cct and provides a mathematical formalism for continuous
responses that serves as the foundation for our extended model. Following
that, in Section 4 we introduce our extended deep latent-construct cct model,
the idlc-cct. Sections 5 and 6 describe the datasets and methods used to
validate the model. Finally, in Sections 7 and 8 we present and discuss the
results obtained from fitting the model to data across the several domains.

##  2 Machine representations

Recent advancements in deep learning have given rise to expressive high-
dimensional vector representations, known as embeddings [23]. These embeddings
capture the relationships, patterns, and similarities of entities such as
words or objects. Deep learning involves training large multi-layered neural
networks, consisting of input, hidden, and output layers [23]. The input layer
receives data (e.g., image or text), while hidden layers transform the data
into intermediate representations, and the output layer generates a response
(e.g., label or action). These intermediate representations, or embeddings,
effectively capture expressive features of high-dimensional data points,
allowing for successful mapping to responses. Embeddings are versatile inputs
for a variety of predictive and explanatory models [29, 30, 20, 22, 31].
Researchers employ different deep-learning techniques to generate these
embeddings based on the specific application [32]. Subsequent sections will
discuss word, sentence, and image embeddings in detail.

###  2.1 Sentence and Word Representations

The field of distributional semantics concentrates on automatically deriving
embeddings for real-world concepts from large-scale natural language data
[33]. These models are based on the understanding that a significant portion
of human knowledge is embedded in word co-occurrence patterns [34]. By
leveraging these patterns, researchers can create word embeddings as vectors
in high-dimensional semantic spaces or as collections of probabilistic
semantic topics [35, 36, 37, 38]. The resulting vectors show that closely
associated words, which are often discussed and referenced in similar
contexts, have similar representations and are therefore closer to each other
in the semantic space.

In most use cases, “pre-trained” distributional semantic representations for
concepts serve as inputs for more sophisticated models. These models are then
“fine-tuned” using participant data, allowing them to approximate the
structured knowledge inherent in participants’ responses. Applications include
predicting the perceived riskiness of various hazards [24], evaluating
perceived leadership [27], modeling stereotypes and prejudices that bias
social judgments [39], and extracting team cognitive diversity [31]. In
addition, recent deep learning advancements can generate vector
representations that capture key aspects of sentence meaning [40, 41, 42],
which can then be fine-tuned for downstream tasks using secondary machine
learning models (see [43] for an intuitive explanation). Researchers have
applied these methods to study people’s semantic cognition, such as
commonsense [44], relation knowledge [45], similarity judgements [46], and
individual’s propensity ratings [47].

###  2.2 Image representations

Deep neural networks have demonstrated the ability to match or surpass human
performance on benchmarks for image classification [48], detection [49], and
recognition [50]. Training these networks on large-scale database of images
enables them to learn versatile feature sets that generalize effectively to
real-world settings. The representations discovered by deep neural networks
can be used in models of human behavior for perceptual tasks, such as
predicting the memorability of objects in images [51] and predicting human
typicality judgments [52]. While their predictive power is evidence of
relevance to human judgments, these models fail to fully capture the structure
of human psychological representations [53]. Comparison of these two
representations is challenging because human psychological representations
cannot be directly observed. Researchers thus have combined representations
obtained from machines with psychological models to examine the correspondence
between the two [54] and bring them into closer alignment [55, 25].

##  3 Cultural Consensus Theory

The underlying principle of consensus-based models is that, for many tasks, a
group’s central tendency often yields a precise outcome. The collective
response can serve as a substitute for the actual answer when assessing
individual group members—those whose judgments are closer to the group’s
central tendency (across multiple questions) are presumed to possess greater
knowledge. Consequently, consensus-based models can be employed to estimate an
individual’s level of knowledgeability when there is no available ground
truth. Cultural Consensus Theory (cct), developed by [16], is a theory and
method that outlines the conditions under which agreement among individuals
can indicate knowledge or accuracy. Many folk epistemological systems are
based on this very relationship between consensus and truth; notable examples
include the jury system, decentralized content moderation in online
communities, and numerical ratings on review aggregation sites. The principle
is even reflected in expressions such as “50,000,000 Elvis fans can’t be
wrong.”

Researchers employing cct strive to measure the consensus from respondents’
individual responses in cases where the researchers do not know the consensus
ahead of time, nor which respondents have more or less knowledge. Indeed, the
problem solved by cct is akin to determining the answer key to a test given to
respondents while simultaneously grading those respondents with respect to the
answer key [56, 57]. Additionally, like related Item Response Theory methods
[58], cct measures the difficulty of the questions. In cct, this is
accomplished with cognitive-based response process models, with consensus
answers and cognitive characteristics of the respondents estimated
endogenously. Thus, cct is useful to researchers in situations where (a) the
consensus knowledge of the respondents is unknown to the researcher ahead of
time, (b) the researcher has access to a limited number of respondents who may
or may not have equal access to this shared cultural knowledge, (c) the
researcher can construct a relevant questionnaire but does not know which
questions are more or less difficult, and (d) the researcher does not know
much a priori about the characteristics of the respondents.

The cct framework additionally takes into account each respondent’s response
biases, which are the shift and scale parameters of the function that maps
their latent appraisal to a location on the response scale. An example of a
scale bias is a tendency to mark most values either at the outer ends or in
the middle section of the scale. An example of a shift bias is a tendency to
mark values more frequently on one side of the scale. These parameters derive
from a function often used to scale bias in probability estimation in the
range (0, 1), called the Linear in Log Odds function [59]. Thus, the scale
bias parameter estimates the shrinkage or expansion of the latent appraisals,
while the shift bias parameter estimates a shift to the left or right.

The first cct model, the General Condorcet Model (GCM), was developed for
binary data (true/false responses) and assumes that the consensus truth of
each item is also a binary value [16]. The GCM has been widely used in the
social and behavioral sciences [60]. [61] introduced an alternate assumption
to the GCM to extend it to continuous truths. An extensive cct model for
ordinal data was developed using a Gaussian appraisal model [62]. In addition,
cct models for continuous response data were developed to estimate and detect
cultural consensuses, respondent knowledge, response biases, and item
difficulty from continuous data [63, 56].

Here, we describe the Continuous Response Model (CRM), developed by [63] and
that allows for multiple consensus truths, which serves as the basis for our
extension to the model.

###  3.1 Continuous Cultural Consensus Theory

As a starting point, consider the Continuous Response Model (crm) [63], a
cultural consensus model for continuous data derived from observations of the
random response profile matrix $\textbf{X}_{ik}=(\emph{X}_{ik})_{N\times M}$
for $N$ respondents and $M$ items, where each respondent’s response falls
within $(0,1)$ or a finite range that permits a linear transformation to
$(0,1)$. The crm links the random response variables in (0, 1) to the real
line through the logit transform $\emph{X}^{*}=\textrm{logit}(\emph{X}_{ik})$.
Therefore, each item has a consensus value in $(-\infty,\infty)$.

The crm is formalized and further explained by the following axioms:

Axiom 1 (_Cultural truths_). There is a collection of of $\emph{V}\geq 1$
latent cultural truths, $\\{T_{1},...,T_{v},...,T_{V}\\}$, where $T_{V}\in\
\prod_{k=1}^{M}(-\infty,\infty)\ $. Each participant _i_ responds according to
only one cultural truth (corresponding consensus locations), as
$T_{\Omega_{i}}$, where $\Omega_{i}\in\\{1,....,V\\}$, and parameter
$\Omega=(\Omega_{i})_{1\times N}$ denotes the cultural membership for each
respondent.

Axiom 2 (_Latent Appraisals_). It is assumed that each participant draws a
latent appraisal, $Y_{ik}$, of each $T_{\Omega_{ik}}$, in which
$Y_{ik}=T_{\Omega_{i}k}+\epsilon_{ik}$, The $\epsilon_{ik}$ error variables
are distributed normal with mean 0 and standard deviation $\sigma_{ik}$.

Axiom 3 (_Conditional Independence)_. The $\epsilon_{ik}$ are mutually
stochastically independent.

Axiom 4 (_Precision._). There are cultural competency parameters
$\textbf{E}=(E_{i})_{1\times N}$ with all $E_{i}>0$, and item difficulty
parameters specific to each cultural truth
$\Lambda=(\lambda_{k})_{1\times\emph{M}}$, $\lambda_{k}>0$ such that

$\sigma_{ik}=\lambda_{k}/\emph{E}_{i}.$ (1)

If all item difficulties are equal, then each $\lambda_{k}$ is set to 1.

Axiom 5 (_Response Bias_). There are two respondent bias parameters that act
on each respondent’s latent appraisals to arrive at the observed responses
$\emph{X}_{ik}$. These include a scaling bias,
$\textbf{A}=(\emph{a}_{i})_{1\times N},\emph{a}_{i}>0$; and shifting bias
$\textbf{B}=(\emph{b}_{i})_{1\times N},-\infty<\emph{b}_{i}<\infty$, such that

$X^{*}_{ik}=a_{i}Y_{ik}+b_{i}.$ (2)

These axioms are developed to model the continuous responses of respondents
that differ in cultural competency, $\emph{E}_{i}$, and response biases,
$\emph{a}_{i}$ and $\emph{b}_{i}$, to items that have different shared latent
truth values. The respondents have a latent appraisal with a mean at the
item’s consensus location, plus some error, which depends on their competence
and the item difficulty. Axiom 1 locates the item truth values in the
continuum. Axiom 2 defines the appraisal error is normally distributed with
mean zero. Axiom 3 sets the appraisals to be conditionally independent given
the respondents’ cultural truth and the error standard deviations. Axiom 4
specifies the standard appraisal error that depends on the respondent’s
competence and item difficulty. Axiom 5 sets each respondent’s response shift
and scale biases.

##  4 Deep Latent-Construct CCT

cct represents the structure of culturally held beliefs as a lookup table,
where keys correspond to questions and values to answers. In this
representation, questions and answers lack defined internal structures and are
connected only through correlations across respondents’ answers. However, this
formulation comes with several limitations. First, it treats each question-
answer pair independently, preventing information from one question from
informing our understanding of others. Second, the number of questions
required to characterize a culture scales linearly with the number of
culturally held beliefs. Finally, there is no way to benefit from existing
structured knowledge bases that provide information about known entities and
their relations.

We begin by extending cct to provide a more sophisticated representation of
culturally held beliefs than a mere lookup table. In this extension, we define
these beliefs as a latent construct, a function that maps a question to a
consensus answer through an intermediate representation [64]. We consider
latent constructs that are structured as deep neural networks fine-tuned using
a linear readout layer that is specific to a particular culture. Thus the
latent construct as it applies to a particular culture is a combination of
pre-trained embeddings and the regression weights associated with that
culture.

Formally, then, we introduce two elements to the model described in Section
3.1: (1) pre-trained embeddings $\phi_{k}$ that represent a featurization of
the entity $k$ under question, and (2) the culture-specific regression weights
$\omega_{\Omega_{i}}$. The relationship between the regression weights of the
latent construct and the embeddings is encapsulated in the embeddings is
represented through the regression equation

$\displaystyle\begin{split}T_{vk}=\phi_{k}\omega_{\Omega_{i}}^{T}\\\
Y_{ik}=T_{vk}+\epsilon_{ik},\end{split}$ (3)

where $\epsilon_{ik}$ is the error variable in Axiom 4 (Eq. 1). We then
replace the consensus location described in Axiom 1 with a function that takes
as input the embeddings and corresponding weights for each feature and outputs
the consensus.

When fitting the latent construct, we use Bayesian Ridge regression to
regularize its weights. The prior for the coefficients, $\omega_{\Omega_{i}}$,
is given by a spherical Gaussian:

$p(\omega_{\Omega_{i}}\mid\zeta)=\text{Normal}(\omega_{\Omega_{i}}\mid
0,\zeta^{-1}\textbf{I}_{p}),$ (4)

with the prior over $\zeta$ assumed to be Gamma distributed, the conjugate
prior for the precision of the Gaussian.

###  4.1 Infinite Deep-Latent Construct CCT

In formulations that allow for the possibility of multiple cultures, cct
analyzes eigenvalues obtained from the cross-participant correlation matrix to
determine the number of cultures present [63]. In the context of our Bayesian
formulation of deep latent construct CCT, we employ a Bayesian non-parametric
technique in which cultures (and their associated latent constructs) are drawn
from a Dirichlet Process (DP) by way of the stick-breaking construction [65].
The Dirichlet Process serves as a prior over discrete distributions and is
particularly useful for mixture models, where the number of components is not
known a priori and can grow with the data. The DP’s attribute of being an
infinite discrete prior has led to it being widely applied not only to mixture
models [66, 67], but also to psychometric models as a prior over
probabilities, facilitating flexible modeling of individual differences and
underlying traits [68, 69]. Using this method in the context of CCT provides
an end-to-end probabilistic approach to learning the number of cultures needed
to account for the observed data that simultaneously learns the culture
assignments, latent constructs, and individual-level parameters. These
parameters can be marginalized over the discrete cultural membership
indicators $\boldsymbol{z}$ by using an efficient posterior inference
algorithm (e.g., adi, nuts, hmc) for learning the joint posterior of the
remaining model parameters.

Although the model assumes an unbounded number of cultures in theory, in
practice, for a given set of respondents, only a finite number of cultures
will have at least one respondent assigned to them. To make the model more
computationally feasible and efficient, we truncate the Dirichlet Process by
selecting an upper bound that caps the possible number of cultures. This
approach strikes a balance between enabling the model to account for a variety
of cultures while keeping resource consumption, such as processing time and
memory, in check.

###  4.2 Hierarchical specification of the extended CCT

In this section, we provide a hierarchical specification of the generative
model that underpins Infinite Deep Latent Construct (idlc-cct), where
population distributions are specified for the parameters using
hyperparameters [70]. These hyperparameters are estimated from their own
distributions and can represent the central tendency across items or
respondents, which may be unique to each dataset. The hierarchical structure
of the generative model is as follows:

$\displaystyle\omega_{\Omega_{i}}\sim\text{Normal}(0,\zeta^{-1})$ Coefficient
weights $\displaystyle T_{vk}=\phi_{k}\omega_{\Omega_{i}}^{T}$ Latent
construct item location
$\displaystyle\text{log}(\lambda_{vk})\sim\text{Normal}(\mu_{vk},\tau_{vk})$
Item difficulty
$\displaystyle\text{log}(E_{i})\sim\text{Normal}(\alpha_{E_{\Omega_{i}}},\kappa_{E_{\Omega_{i}}})$
Respondent competency
$\displaystyle\text{log}(a_{i})\sim\text{Normal}(\mu_{a_{i}},\tau_{a_{i}})$
Respondent scaling bias $\displaystyle
b_{i}\sim\text{Normal}(\mu_{b_{i}},\tau_{b_{{}_{i}}})$ Respondent shifting
bias $\displaystyle\Omega_{i}\sim\text{Categorical}(\pi)$ Group membership
$\displaystyle\pi\sim\text{stickbreaking}(\beta)$ Pr. of group membership
$\displaystyle\beta\sim\text{Beta}(1,\alpha)$ Group sparsity

The item consensus $T_{vk}$ is derived by applying Bayesian Ridge Regression
to the pretrained embeddings. The location of the item consensus is calculated
by taking the dot product of the relevant two row matrices. The other model
parameters, $E_{i}$, $\lambda_{i}$, and $a_{i}$, which are each located on the
positive half-line, are log-transformed to the real line and also assumed to
be sampled from a normal population-level distribution. The shift bias,
$b_{i}$, is located on the real line, paramaterized with a mean and precision
(inverse variance). Note that the respondents’ competence parameter remains
singly indexed by _i_ , through an indexing technique in which their
distribution is specified by their group membership $\Omega_{i}$. Culture
assignments are derived via a stick-breaking prior, and this allows for
varying probabilities of being in any of _V_ groups; note that _V_ is unknown
priori and needs to be estimated from observed data. The sparsity of cultures,
$\beta$, is sampled from a beta distribution. The variables $\Omega_{i}$ and
$\pi$ are removed for the single-truth variant of our model.

##  5 Method

We applied the idlc-cct and dlc-cct to a diverse range of datasets that
included judgments of the healthiness of food, humor, leadership
effectiveness, social attributes of faces, and risk. These datasets were
chosen because they include respondents’ individual responses to questions
(items) that are related to their shared knowledge or beliefs, and because the
nature of the domain is such that a consensus contributes to an
intersubjective truth. For our idlc-cct model, we rescaled these ratings to
fall within the range of 0 to 1. Excluded were datasets with only aggregate
responses, or where the nature of the domain is fully objective or subjective.

Models were implemented in NumPyro [71] with the jax backend [72]. The model
components were integrated into a single likelihood function and a set of
prior distributions. Inference was performed using a Gibbs Sampler [73]
combined with the No-U-Turn Sampler (nuts) [74], a standard Markov Chain Monte
Carlo sampling algorithm, as implemented in NumPyro. We used two chains with
10,000 warm-up samples and 10,000 draw samples, thereby obtaining 20,000
posterior samples for each model. We ensured that the posterior had converged
by ensuring there were no divergent transitions and monitoring chain
diagnostics such as $R_{\textrm{hat}}$. The code for the models and datasets
are available on GitHub at [Redacted for blind submission].

In the next section, we elucidate the phenomena associated with each dataset,
along with a thorough description of the datasets themselves.

##  6 Concept and Data

###  6.1 Sense of humor

Everyone appreciates humor. In the workplace, it offers benefits such as team
bonding, employee motivation, idea generation, and frustration diffusion
through venting [75, 76]. However, it is important to recognize that humor can
also have downsides, such as causing distractions, harming credibility, or
offending others in diverse work environments [77]. The differences in how
humor is enacted and understood arise from individual differences, such as
gender, beliefs, and mental state [78]. Therefore, organizations may find
themselves in a complex position, responsible for considering individual
differences in perception of humor to avoid negative consequences in the
workplace [79].

Another stream of research focuses on the use of humor in marketing materials,
such as commercial and print ads, as well as entertainment products like TV
shows and web content. A common belief is that making someone laugh can lead
to a purchase. Advertising and marketing research, indeed, has demonstrated
that humor can enhance ad likability, partly due to its ability to evoke
positive emotions [80]. Humorous ads positively impact viewers’ moods and
elicit more positive feelings than non-humorous ads [81]. These positive
emotions may lead to a favorable attitude towards the ad [82, 83], increased
liking of the source [84], and stronger persuasion [85]. While studies
indicate the positive impacts of humor in marketing and advertising, the
effectiveness of humor depends on the targeted audience’s idiosyncratic
features [86, 87]. Hence, incorporating humor tailored to the target audience
in marketing and advertising could be vital for companies to positively
influence consumers and avoid negative consequences [88].

We employed the idlc-cct on the Jester dataset [89], commonly used in
recommendation system tasks. The dataset consists of 140 jokes rated by 59,132
internet users on a scale of -10 (extremely unfunny) to 10 (extremely funny).
To avoid computational intensity that could be caused by the high number of
respondents, we selected the ratings of a random subset of 5,000 users from
the dataset as input for our model. Rating distribution across items is sparse
due to the disparity in the number of ratings made for different items.
Ratings of 112 jokes were used for training and the remaining 28 were held-out
as a validation set, an 80-20 split.

We obtained embeddings for each joke from RoBERTa, a pre-trained language
model [42] that extracts features from textual data and generates a latent
vector in a 768-dimensional space. These latent vectors serve as pre-trained
embeddings in the model.

###  6.2 First impression of faces

Though warned not to judge a book by its cover, people nonetheless attribute a
wide variety of traits (e.g., trustworthiness, dominance, or smartness) to
strangers based on their facial appearance. These judgments are formed quickly
and have considerable impact on human behavior in domains as diverse as loan
approval [90], politics [91], law [92], business [93], and social decisions
[94, 93]. For example, [93] showed that people form impressions of aspiring
leaders from their faces, which in turn predict their success in reaching
prestigious leadership positions.

Some trait judgments have high levels of inter-rater agreement within
particular cultures and are shared by observers in multiple cultures [95, 96].
Such agreement among the groups is a so-called consensus impression that is
shared by many individuals or communities. For example, many individuals in
the U.K. and the U.S. attribute naivety and trustworthiness to faces with
large eyes and round features [96]. The same participants tend to judge short,
squat faces to be more aggressive than faces that are tall and thin [97]. On
the other hand, face-trait judgments are found to be idiosyncratic and differ
between individuals living within the same culture [95]. It suggests that
one’s local environment that is surrounded around family and community likely
drives the observed individual differences. Similarly, [98] showed that
perceivers learn conceptual knowledge of how traits correlate that shape trait
inferences across faces, personal knowledge, and stereotypes in one’s
environment. These results show the importance of individual differences in
first impression formation relative to cultural variability.

We applied the iDLC-cct to a large dataset of people’s first impressions of
faces [25]. The dataset contains over 1 million judgments of 34 trait
inferences for 1,000 face images. Each face is rated by 30 unique participants
for each trait. In this dataset, the face images are generated using a
synthetic photorealistic image generator, StyleGAN2 [99]. The generator
network component of StyleGAN2 models the distribution of face images
conditioned on a 512-dimensional, unit-variance, multivariate normal latent
variable. This vector is the pretrained embedding used for our modeling. For
each trait, we used the ratings of 800 faces for training and the remainder
(200 faces) for validation, an 80–20 split.

###  6.3 Risk perception

Research on risk perception investigates people’s views when asked to describe
and assess potentially dangerous activities and technologies. Such research
supports risk analysis and decision-making in society, such as enhancing
techniques for obtaining risk-related opinions, predicting public reactions to
hazards, and refining the exchange of risk information among the public,
technical experts, and policymakers [100, 101]. Risk is fundamentally a
subjective notion [100], and discerning how individuals perceive risk is
essential for understanding the relationship between risk and decision-making
processes at the individual, group, and organizational levels [100, 102].
Various factors have been found to explain how people perceive a hazard, such
as the risk’s characteristics [103], perceived benefits [104], knowledge
[105], personal norms [106], and affective associations [107]. These factors
could be unobserved variables that should be considered when examining the
public’s perception of a hazard.

We applied the iDLC-cct to people’s risk perceptions of technological hazards,
activities, and participant-generated risk sources [24]. The dataset on risk
perception of technological hazards consists of 125 technologies of varying
risk levels. The items in this dataset were based on Slovic’s experiment
[100]. The dataset on risk perception of daily activities consists of 125
activities of varying risk levels. The items in this dataset were also based
on Slovic’s experiment [100]. The dataset on participant-generated risk
sources consists of the 200 risk sources most frequently listed by
participants, without any category limitations. We used to word2vec [37] word
representations to obtain an embedding for each risk source, which is in a
300-dimensional space. This vector is the pretrained embedding used for our
modeling. For risk sources of technologies and activities, we used the ratings
of 100 sources for training and the remainder (25 sources) for validation, an
80–20 split. For participant-generated risk sources, we used the ratings of
160 risk sources for training and the remainder (40 sources) for validation,
an 80–20 split.

###  6.4 Leadership perception

Identifying the factors that predict leader selection is crucial, as a
leader’s influence on their organization’s success impacts the well-being of
its members and those affected by the organization’s output [108]. As a
result, organizations and their members should be highly motivated to
recognize and select effective leaders within their domain, ideally based on
objective indicators of leadership quality. Leadership researchers have
emphasized the importance of studying how followers perceive their leaders
[109, 12, 110]. Although researchers’ opinions may vary regarding the
characteristics of effective leaders, individuals often possess implicit
leadership theories that define the cognitive structures and criteria
distinguishing leaders from non-leaders, as well as effective leaders from
ineffective ones [111]. While skilled leaders may not always be acknowledged
as such, the perception of leadership ability is essential for an individual’s
success in a leadership role [112, 91].

We applied the iDLC-cct model to a dataset containing people’s judgments of
leadership effectiveness [27]. This dataset includes assessments from 210
participants regarding 293 individuals, consisting of both leaders and non-
leaders. Participants rated leadership effectiveness on a scale of 0
(extremely ineffective) to 100 (extremely effective). We utilized word2vec
word representations [37] to generate an embedding for each individual,
projecting the word vectors into a 300-dimensional space. This vector is the
pretrained embedding used for our modeling. We used the ratings of 235
individuals for training and the remainder (58 individuals) for validation, an
80–20 split.

###  6.5 Food healthiness

People hold diverse opinions regarding what constitutes a healthy diet [113,
114]. These opinions are shaped by various factors, including personal
beliefs, associations, attitudes, and knowledge regarding the healthiness of
food [115]. Due to the complex interplay of psychological, cognitive, and
social factors influencing consumer choices, there is no definitive answer to
what makes food appear healthy [116]. Even experts cannot reach a consensus on
the definition of a healthy diet [117]. Consequently, understanding people’s
health beliefs and perceptions of healthy food is crucial for creating
culturally and socially appropriate behavior change interventions [118].

Choosing healthy food is a complex behavior that involves cultural (e.g.,
customs, social norms), psychological (e.g., body image), and social factors
(e.g., price availability, ethical concerns) [119]. Prior research on healthy
food perceptions across diverse groups has highlighted the unique ways
individuals define healthy food [120]. Cross-cultural studies indicate that
some cultures share similar views on healthy food choices and nutritional
intake [121], while others differ [120]. Factors such as gender and education
level also impact judgments about the healthiness of food [122]. Beyond
individual characteristics, front-of-package labeling is considered to have a
significant influence on people’s evaluations of food healthiness [123].
However, judgments based on front-of-package labeling are also affected by
individual features [124].

We applied the iDLC-cct model to a dataset of laypeople and experts’ judgments
of food healthiness [125]. The lay people dataset contains judgments of 149
lay and 19 expert participants of a diverse set of 172 foods. Participants
were asked to judge the healthiness of food on a scale ranging from -100
(extremely unhealthy) to +100 (extremely healthy). We used word2vec word
representations [37] to obtain an embedding for each food, projecting the word
vectors into a 300-dimensional space. This vector is the pretrained embedding
used for our modeling. We used the ratings of 235 food items for training and
the remainder (34 food items) for validation, an 80–20 split.

##  7 Results

Across all the datasets, we consistently found that the dlc-cct outperformed
the original cct model, showing a reduction in the average RMSE by 0.12 and an
increase in $R^{2}$ by 0.22 (Table 1). Because the original cct lacks a
mechanism by which to meaningfully generalize across items based on their
content, it relies only on the posterior mean of the consensus parameter’s
prior when predicting held-out items. In contrast, as indicated in Table 1,
dlc-cct significantly outperforms the original cct by generalizing to held-out
items, demonstrating the benefits of integrating a deep latent construct into
cultural consensus analysis.

Likewise, we found that the idlc-cct (with its multiple consensuses)
outperforms the dlc-cct (with only a single consensus). The dlc-cct yielded
average RMSE and $R^{2}$ values of 0.22 and 0.33, respectively. In comparison,
the idlc-cct performed better, reflected in an average RMSE of 0.20 and an
$R^{2}$ of 0.41. This represents a reduction in RMSE by 0.02 and an increase
in $R^{2}$ by 0.08. Comparison between the two reveals the advantages of
incorporating heterogeneity in consensus beliefs across subsets of
respondents.

Table 1:  Model comparison.

|  | CCT |  | DLC-CCT |  | iDLC-CCT |  |  |
---|---|---|---|---|---|---|---|---|---
|  | $R^{2}$ | RMSE | $R^{2}$ | RMSE | $R^{2}$ | RMSE | Culture entropy | No. cultures
Leadership | Leadership | .08 | 0.29 | .31 | .24 | .35 | .24 | 2.23 | 7
Humor | Humor perception | .00 | .30 | .27 | .22 | .27 | .22 | 2.69 | 12
Healthy food | Lay people | .00 | .30 | .52 | .20 | .56 | .19 | .40 | 2
| Experts | .00 | .31 | .43 | .21 | .43 | .21 | .74 | 2
Risk sources | Technology hazards | .00 | .32 | .33 | .26 | .36 | .24 | .51 | 4
| Daily activities | .04 | .29 | .26 | .26 | .32 | .24 | .67 | 2
| Participant-generated risks | .00 | .31 | .47 | .23 | .50 | .22 | 1.44 | 3
Impressions of faces | Dominant | .00 | .38 | .33 | .22 | .41 | .20 | 1.46 | 7
| Trustworthy | .00 | .40 | .29 | .20 | .34 | .19 | 2.11 | 8
| Smart | .00 | .33 | .29 | .18 | .36 | .17 | 2.17 | 5
| Attractive | .00 | .39 | .41 | .19 | .45 | .18 | 2.29 | 6
| Outgoing | .00 | .38 | .36 | .18 | .46 | .17 | 1.97 | 5
| Age | .00 | .26 | .40 | .12 | .42 | .12 | .03 | 2
| Fat | .00 | .30 | .37 | .15 | .44 | .13 | 1.56 | 5
| Familiar | .00 | .35 | .47 | .20 | .48 | .20 | 0.59 | 3
| Gender | .00 | .43 | .58 | .22 | .67 | .19 | 1.37 | 5
| Typical | .00 | .41 | .27 | .20 | .30 | .19 | 2.55 | 8
| Happy | .00 | .35 | .51 | .16 | .60 | .15 | 1.33 | 4
| Dorky | .00 | .42 | .19 | .25 | .30 | .23 | 2.08 | 6
| Long hair | .00 | .35 | .26 | .21 | .55 | .17 | 1.44 | 7
| Skin color | .00 | .28 | .59 | .13 | .66 | .12 | 0.83 | 5
| Smug | .00 | .35 | .27 | .24 | .41 | .22 | 2.21 | 7
| Groomed | .00 | .42 | .28 | .20 | .36 | .19 | 1.54 | 5
| Cute | .00 | .41 | .47 | .20 | .53 | .19 | 1.89 | 5
| Alert | .00 | .40 | .26 | .20 | .36 | .18 | 2.16 | 6
| Hair color | .00 | .44 | .60 | .18 | .64 | .17 | .48 | 5
| Privileged | .00 | .35 | .31 | .19 | .39 | .17 | 2.07 | 5
| Liberal | .00 | .26 | .16 | .23 | .20 | .22 | 1.79 | 5
| Asian | .00 | .38 | .19 | .28 | .39 | .27 | .15 | 6
| Middle-Eastern | .00 | .39 | .30 | .25 | .34 | .24 | 1.03 | 6
| Hispanic | .00 | .40 | .16 | .29 | .17 | .29 | .38 | 4
| Polynesian | .00 | .37 | .29 | .25 | .37 | .24 | 1.49 | 5
| Native-American | .00 | .33 | .26 | .25 | .33 | .23 | .89 | 4
| Black | .00 | .45 | .25 | .19 | .43 | .17 | .24 | 5
| White | .00 | .52 | .38 | .28 | .60 | .23 | .86 | 4
| Looks like me | .00 | .29 | .35 | .19 | .49 | .17 | 1.57 | 3
| Electable | .00 | .39 | .18 | .26 | .24 | .25 | .77 | 3
| Gay | .00 | .40 | .07 | .25 | .09 | .25 | 1.77 | 4
| Religious | .00 | .42 | .14 | .22 | .22 | .20 | 2.21 | 6
| Outdoors | .00 | .38 | .44 | .27 | .47 | .27 | 1.00 | 3

  * •

_Notes._ Performance ($R^{2}$) and predictive accuracy (RMSE) of single-truth
(dlc-cct) and multi-truth (idlc-cct) models. No. of cultures and entropy of
cluster assignment shown for idlc-cct.

The sixth column of Table 1 presents the posterior mode of the number of
instantiated cultures — those with at least one respondent assigned to them.
Although this count provides insights into the magnitude of heterogeneity in
cultural consensus, it does not convey the distribution’s uniformity (or lack
thereof) concerning cultural assignments. An entropy-based metric fills this
gap by estimating the uncertainty in the assignment of a randomly selected
respondent under the modal posterior cultural distribution. Lower entropy
occurs when there are a few dominant clusters, while higher entropy occurs
when there is a more balanced distribution of cultures. The posterior over
models parameters for the idlc-cct featured an average cultural entropy of
1.39 and an average cultural count of 4.82 instantiated cultures (i.e., those
with at least one respondent assigned to them in the modal posterior
assignment), demonstrating both a multiplicity of consensus as well as a non-
uniform distribution of assigned respondents per culture. Crucially, as
highlighted in the sixth column of Table 1, despite varying in the extent of
heterogeneity, none of the 40 datasets was best fit by the single-culture
model — all displayed heterogeneity in consensus beliefs.

Figure 1: Individual response biases plotted against each other for each
participant.

To ascertain if there was a dominant culture within a dataset, we measured the
proportion of datasets that assigned at least 50% of respondents to a single
culture — these datasets represent a “majority” culture. Out of the 40
datasets, 23 exhibited a majority culture. The datasets with the weakest
majority (less than 66%) included participant-generated-risks, long hair,
groomed, alert, liberal, electable, outdoors, fat, and happy. On the other
hand, those with the strongest majority (more than 66%) were technology
hazards, daily activities, lay and expert opinions on healthy food, age,
familiarity, skin color, hair color, and ethnic backgrounds such as Asian,
Middle-Eastern, Hispanic, Native-American, and White. A comparison of the
first and third columns of Table 1 reveals that the idlc-cct improved the fit
for 38 out of the 40 concepts evaluated. The predictive accuracy showed an
improvement for 32 of the 40 concepts, as evidenced by the comparison between
the second and fourth columns in Table 1. These enhancements are more
pronounced for the datasets with significant heterogeneity in consensus
beliefs among participants because the single-truth model overlooks such
variation by assuming that all respondents operate under a single consensus
view.

Figure 1 illustrates respondents’ response biases. Understanding these biases
helps to account for individual differences in responses that cannot be
attributed to cultural competency, nor culture assignment. By definition, the
average shift bias was 0.0 and the average log scale bias was 0.0. Across
participants, the standard deviation of the shift bias was 0.65, while for
scale bias, it was 0.08. Many datasets exhibit roughly similar variations in
shift and scale biases, but some show notable differences. For instance, in
the first impression dataset, the “age” trait has little variation in shift
bias (_SD_ = 0.10), whereas the religious trait shows opposite pattern, with
little variation in scale bias (_SD_ = 0.02) but considerable variation in
shift bias (_SD_ = 1.43).

###  7.1 Model Ablation Experiments

Understanding the contribution and significance of different components within
idlc-cct is important for gaining insights into the model’s behavior and
understanding where the observed heterogeneity in consensus views arise. In
this section, we focus on model ablation experiments, a systematic approach
aimed at evaluating the impact of individual components or features on the
model’s overall performance [126]. By selectively removing or neutralizing
certain elements of the model, we can isolate the effect that each component
has on the outcome. This is instrumental in identifying which aspects of the
model are critical to its predictive power, and which may be redundant or even
detrimental. Such experiments are especially useful for model interpretation,
fine-tuning, and guiding future developments. We used the leadership
perception dataset for the parameter ablation experiments. The ablation
experiments for other datasets can be found in the Appendix.

Model | $R^{2}$ | RMSE | Cultural Entropy | No. Cultures
---|---|---|---|---
idlc-cct | .35 | .24 | 2.23 | 6
dlc-cct | .31 | .24 | 0.0 | 1
idlc | .22 | .26 | 4.00 | 8
cct | .08 | .29 | 1.07 | 5
Bayesian Ridge Regression | .13 | .28 | .00 | 1
Ablation shift | .25 | .27 | .84 | 2
Ablation scale | .31 | .25 | .00 | 1
Ablation competence | .23 | .26 | 2.60 | 10
Ablation item difficulty | .24 | .25 | 3.50 | 8
Table 2:  iDLC-CCT Model Ablation.

In our first ablation experiment, we examined the significance and
contribution of the deep latent constructs to the model’s performance. To
achieve this, we randomly shuffled the embedding feature values across items,
disrupting their structure while preserving their marginal distribution. This
process eliminates any meaningful relationships that the features might have
captured. The idlc-cct performed poorly when the deep features’ structure was
disrupted ($R^{2}$ = 0.0). The results indicate that the content of the pre-
trained embeddings is critical for the model’s performance.

To examine the significance of each individual and item-level parameters on
the predictive performances of idlc-cct, we conducted parameter ablation
experiments. Table 2 presents the results of these ablation experiments,
detailing which parameters were held constant to assess their contribution to
prediction accuracy. The results reveal that each parameter in i-dlc-cct
enhances predictive accuracy. Bayesian Ridge Regression yields the poorest
performance, whereas idlc-cct, when incorporating all the model components,
emerges as the most effective. Setting the scale bias to a constant had the
least impact on predictive power, and interestingly, it caused the model to
output a single consensus belief, indicating a singular cultural cluster. The
scale parameter specifically accounts for magnitude preferences, such as a
tendency to mark most values at the outer ends or the middle section of the
scale. Therefore, taking into account respondents’ magnitude preferences may
be instrumental in discerning group-level differences.

###  7.2 Robustness to Sparse Sampling

Robustness to sparse sampling refers to a model’s ability to sustain
performance when faced with limited data. This analysis is especially valuable
in situations where data collection and labeling are resource-intensive or
challenging. To assess the robustness of our models against sparse sampling,
we trained them using incremental data quantities of 5%, starting from 10% and
increasing up to 80% of the total data. In the analyses presented here, we
considered the leadership perception dataset. The robustness to sparse
sampling experiments for other datasets can be found in the Appendix. Fig. 2
illustrates our robustness analysis, revealing that both the idlc-cct and dlc-
cct models display considerable resilience to sparse sampling. Impressively,
even with a significantly reduced training set, constituting merely 10% of the
items, both models could still achieve some level of generalizability (i.e.,
idlc-cct: $R^{2}$ = .17; dlc-cct: $R^{2}$ = .15). As the quantity of training
data increased, the models’ performance also improved incrementally. Notably,
the models reached parity with the performance achieved with 80% of the
training data while requiring only 65% of the data to do so.

Figure 2: Robustness analysis of the iDLC-CCT and DLC-CCT models under sparse
sampling conditions. The models were trained with incrementally increasing
quantities of data (from 10% to 80%) and tested for performance ($R^{2}$).

Compared to idlc-cct and dlc-cct, the Bayesian Ridge Regression model did not
demonstrate a reasonable level of performance under these conditions. This
suggests that considering individual and group-level differences is a
significant factor in harnessing collective intelligence effectively. Our
findings emphasize the value of utilizing models such as idlc-cct and dlc-cct,
which are designed to account for these variances, especially in scenarios
where data might be sparse or difficult to gather. The original cct
demonstrates a slight improvement in its predictions when there is a larger
training dataset because it can better estimate the posterior of the consensus
parameter’s prior with more data when predicting held-out items.

###  7.3 The Discrepancy Between Demographic Features vs. Identified Cultures

Researchers have used demographic features to explain variations in people’s
perceptions of concepts and entities [127]. While some studies suggest these
features can be indicative, others assert they cannot be solely relied upon
[128]. Although demographic features can sometimes explain these variations,
they are not always available for organizations and researchers to incorporate
into their computational models. Moreover, in the interconnected society we
live in, people’s perceptions are influenced by a complex network of factors
that extend beyond basic demographic attributes [129]. A significant part of
this complexity is contributed by the easily accessible and diverse array of
viewpoints on concepts found on social media, online platforms, and
professional and social networks [130]. These platforms not only allow
individuals to form unique understandings but also expose them to a variety of
perspectives [131].

We continued to use the leadership perception dataset to investigate whether
the cultures identified by the idlc-cct align with demographic features. Fig.
3 shows that demographic features alone do not provide clear insight into the
hetereogeneity in perceptions of leadership that are captured by the idlc-cct.
This insight illuminates the multifaceted and nuanced nature of leadership
perceptions in our interconnected society, highlighting that these perceptions
are not just products of individual demographic attributes, but also of the
shared knowledge and perspectives within our interconnected social fabric.

Figure 3: Demographic Distribution of Cultural Assignments

##  8 Discussion

In this paper, we developed idlc-cct, an extension of Cultural Consensus
Theory that integrates a latent construct to map between pre-trained
embeddings of an entity and the consensus belief among one or more respondent
subsets concerning those entities. The model effectively aggregates beliefs
from individuals, including experts and non-experts, to estimate consensus
while identifying idiosyncratic and group-level differences in cultural
constructs. By incorporating features from deep neural networks into cct, we
can estimate cultural consensus for any entity using pre-trained networks or
other available embeddings. We argue that idlc-cct is a robust foundation for
assessing group consensus levels by leveraging the underlying structure and
inter-relatedness of beliefs and a foundation for consensus-aware
technologies. Our findings reveal that considering group-level consensus
variations enhances predictive accuracy and effectively harnesses collective
intelligence for decision-making and collaboration within organizations.

In the subsections that follow, we discuss the scalability of the idlc-cct
model (and multiple-consensus models more generally), describe its possible
applications to consensus building and as a computational foundation for
consensus-aware information technologies, and consider variants of the model
that relax some of its core assumptions.

###  8.1 The Small-Variance Asymptotic Approximation

An ongoing challenge in machine learning involves creating algorithms that can
scale to extremely large datasets. Although probabilistic methods — and
Bayesian models in particular — offer flexibility, the absence of scalable
inference techniques may hinder their effectiveness with certain data. For
instance, in the context of clustering, $k$-means algorithms are frequently
preferred in large-scale scenarios over probabilistic methods like Gaussian
mixtures or Dirichlet process (DP) mixtures [132]. This is because $k$-means
is relatively easy to implement and can manage larger datasets.

Applying an asymptotic analysis to the variance or covariance of distributions
within a model has been used to create connections between probabilistic and
non-probabilistic models. Examples of this include forming connections between
probabilistic and standard PCA by allowing the covariance of the data
likelihood in probabilistic PCA to approach zero [133, 134]. Similarly, the
$k$-means algorithm can be obtained as a limit of the EM algorithm when the
covariances of the Gaussians associated with each cluster decrease towards
zero [135]. Not only do small-variance asymptotics offer a conceptual link
between different approaches, they can also provide practical alternatives to
probabilistic models when dealing with large datasets, as non-probabilistic
models often have better scalability. While still emerging, the use of such
techniques to derive scalable algorithms from complex probabilistic models
offers a promising direction for the development of scalable learning
algorithms.

A simple procedure (Algorithm 1) enables the conversion of any existing
single-culture cct to an icct via an algorithm derived from a small-variance
asymptotic analysis of the Dirichlet Process [136]. One of the primary
advantages of the conversion is that it opens the door to novel scalable
algorithms for multiple cultures.

We begin with a description of the small-variance asymptotic approximation to
the Infinite Gaussian Mixture Model, which alternates through three stages:
cluster assignment, cluster updating, and cluster instantiation [135]. In
cluster assignment, each point is assigned to the cluster that is nearest by
Euclidean distance. Initially, all points belong to the same (and only)
cluster. In the cluster updating phase, the cluster center is adjusted to the
mean of the points assigned to it. During cluster instantiation, the algorithm
identifies the point farthest from the center of its assigned cluster, and if
that distance exceeds $\lambda$, it establishes a new cluster. The center of
the new cluster is positioned at that point, which is then assigned to it. The
algorithm repeats until no point is further than $\lambda$ from its cluster
center. The parameter $\lambda$ thus acts as a concentration parameter, which,
along with the complexity of the data, will determine the number of clusters
that eventually appear, much like the concentration hyperparameter of a
Dirichlet process that controls the number of clusters in an infinite Gaussian
mixture model.

Algorithm 1 Small-Variance Asymptotic Approximation to the Infinite Cultural
Consensus Model

1:Input: Individuals $x_{1},\dots,x_{n}$, $\lambda>0$

2:Initialization: $C=\emptyset$, dlc-cct

3:while $\forall c\in C,\forall x_{i}\in c:CC(x_{i},c)<\lambda$ do

4: $\triangleright$ Culture Assignment

5: for each $x_{i}$ do

6:
$\text{CC}(x_{i},C)={\text{CC}(x_{i},c_{1}),\text{CC}(x_{i},c_{2}),...,\text{CC}(x_{i},c_{n}))}$

7: $c_{\text{max}}=\arg\max_{c}CC(x_{i},c)$

8: if $CC(x_{i},c_{\text{max}})\geq\lambda$ then

9: Assign $x_{i}$ to $c_{\text{max}}$

10: else

11:
$c_{new}={\text{logit}(x_{i1}),\text{logit}(x_{i2}),...,\text{logit}(x_{im})}$

12: Assign $x_{i}$ to $c_{\text{new}}$

13: Add $c_{\text{new}}$ to $C$

14: end if

15: end for

16: $\triangleright$ Culture Updating

17: for each $c$ do

18: $C_{c}=\textsc{dlc-cct}(X_{c})$

19: end for

20: $\triangleright$ Culture Instantiation

21: for each $c$ do

22: $x_{\text{min}}=\arg\min_{x_{i}\in c}\text{CC}(x_{i},c)$

23: if $\text{CC}(x_{\text{min}},c)<\lambda$ then

24: $c_{\text{new}}=\text{logit}(x_{\text{min}})$

25: Assign $x_{\text{min}}$ to $c_{\text{new}}$

26: Add $c_{\text{new}}$ to $C$

27: end if

28: end for

29:end while

30:Output: $C$

By analogy, the small-variance asymptotic approximation to the Infinite
Cultural Consensus Model likewise proceeds through three steps: culture
assignment, culture updating, and culture instantiation. In culture
assignment, each respondent is assigned to the culture for which they exhibit
the highest cultural competence. Since the continuous version of cultural
consensus theory models ratings as a logistic transformation of the latent
appraisal, the assigned culture will be the one whose cultural consensus
values correlate highest with the participant’s logit-transformed responses.
Initially, all respondents belong to the same (and only) culture. In culture
updating, consensus values for each culture are determined by fitting a
single-truth cultural consensus model to the data from respondents assigned to
that culture. And during culture instantiation, the algorithm examines the
participant with the lowest cultural competence for their assigned culture. If
that respondent’s cultural competence is less than $\lambda$, a new culture is
created with consensus values set to the logit-transformed responses of the
respondent, who is then assigned to it.

The small-variance asymptotic approximation facilitates the conversion of
existing single-truth cultural consensus models to the infinite cultural
consensus model. There are two preconditions. First, the model must provide a
means to compute the cultural consensus, which is precisely the function of
existing single-truth cultural consensus methods and thus presents no issue.
Second, the model must enable one to identify the culture for which a
respondent exhibits the highest cultural competence. The readiness of this
capability under existing methods is varied. As previously discussed, for the
continuous model, it involves calculating the linear correlation between the
respondent’s logit-transformed responses and the consensus for each culture,
then identifying the culture with the highest correlation. When these
conditions are met, the single-truth consensus models can be scaled to an
efficient multiple-truth consensus model.

###  8.2 Applications and Considerations

Next, we consider potential applications of the idlc-cct. One such application
is as a computational foundation for consensus-aware information technologies.
Our findings demonstrate the potential of methods to detect and characterize
heterogenity in consensus beliefs. Building on these methods, one could
integrate idlc-cct into information systems to enable interventions that help
to detect and understand a lack of consensus within a purported group or to
build consensus among the members of a team to foster improved decision-making
and collaboration. Beyond detecting and characterizing heterogeneity in
consensus beliefs, idlc-cct also provides insights at both the question and
individual levels, deepening our comprehension of the domain of interest. For
instance, by analyzing variations in question difficulty across cultures, a
consenus-aware information technology could pinpoint questions that some
subsets of respondents may find challenging to agree upon or identify
controversial topics unique to specific cultural contexts. Such insights could
assist in tailoring interventions and strategies to promote collaboration and
consensus-building. By recognizing diverse opinions, a consensus-aware
algorithm can generate proposals that effectively represent various
perspectives, promoting a more inclusive and equitable approach to consensus
formation. It paves the way for information technology applications that
either harness collective intelligence under a lack of cross-cultural
consensus.

Another promising application of cct is its integration into AI systems to
enhance their interoperability. By leveraging cct’s unique capability to
interpret shared beliefs within cultural contexts, we can equip AI systems
with a more culturally nuanced understanding of consensus beliefs. This
integration could significantly enhance these models’ generalizability. The
integration of cct could also increase the transparency of AI systems by
rendering their decision-making processes more intelligible; decisions could
be attributed to culturally shared beliefs or norms. Furthermore, applying cct
could foster a greater level of trust in AI systems among users. By ensuring
that the systems’ decision-making processes align more closely with shared
beliefs, we can develop systems that are ethically sound and culturally
sensitive. Even so, the successful integration of cct into AI systems will
require carefully designed methods to translate a consensus derived from cct
into workable AI systems.

The idlc-cct can also be used to support the consensus-building process by
surfacing the causes of disagreement between respondents, whether those are
individual-level parameters such as shift and scale biases or competency, or
culture-level parameters such as the consensus. However, using idlc-cct for
consensus-building may be challenging under adversarial behavior. Consider,
for example, voting mechanisms or other consensus-building methods, where
knowledge of the aggregation scheme could potentially allow individuals or
groups to strategically manipulate the system by skewing their responses,
misreporting their preferences, or coordinating their actions with others to
drive the consensus toward a desired value [137]. This is sometimes known as
“gaming the system” [137]. It is an open question as to the extent to which
idlc-cct is robust to gaming and the most effect means of reducing the risk of
such strategic behavior to ensure that each individual’s input is a genuine
reflection of their own beliefs, views, or preferences, rather than a
strategic attempt to influence the final outcome. Doing so can lead to more
robust and representative consensus outcomes and more effect consensus-
building tools. In summary, while a respondent’s naivety about the aggregation
scheme may help ensure the genuineness and objectivity of the consensus
process, it also implies the need for the design of robust, resistant-to-
manipulation aggregation schemes, and vigilance towards potential strategic
behaviors.

The iidlc-cct can be extended in various ways. First, building on the insights
derived from the iidl-cct approach, researchers and organizations can further
explore the potential of multimodal fusion models in capturing and modeling
cultural consensus. People perceive the world by simultaneously processing and
merging high-dimensional inputs from multiple modalities, including vision and
semantic meaning [138]. With the advancements in deep learning and computing
power, multimodal fusion models have gained popularity for their ability to
combine different modalities in a way similar to human perception, leading to
more accurate predictions of human judgments [139]. By integrating these
multimodal models within the idlc-cct framework, future research can combine
information from various sources to achieve better alignment between machine
representations and cultural consensus, further enhancing our understanding of
the complex interplay between culture and perception.

Finally, we note that in the idlc-cct model, we rely on the Dirichlet Process,
which assumes the existence of an infinite number of potential cultures, with
each individual belonging to one specific culture. While this simplification
facilitates mathematical modeling and computational tractability, it may not
accurately represent the multifaceted, overlapping, and fluid nature of
cultural identities in reality. Real-world cultures are seldom mutually
exclusive, and individuals often navigate multiple cultural spaces
simultaneously. They may adhere to one cultural norm in a certain context and
another in a different setting. Additionally, cultural affiliations are not
static; they can evolve, intersect, and change over time based on personal
experiences, societal influences, migration, and many other factors. Further,
the distribution over the size of clusters under the DP has exponential tails,
which may or may not be a reasonable assumption regarding how respondents are
spread among cultures. Extensions of the present work to the Pitman-Yor
process could relax this assumption by allowing for non-exponential tail
behavior, such as power-law tails. Therefore, while the Dirichlet Process
provides a practical method for dealing with an unknown number of cultures, it
is essential to understand its limitations. Future studies can explore more
intricate models that better capture the dynamic and overlapping nature of
cultural identities while relaxing some of the assumptions made about cluster
assignments.

###  8.3 Conclusion

In conclusion, we presented the idlc-cct, which allows culturally held beliefs
to be transformed into fine-tuned machine representations. These
representations map features of a concept or entity to the consensus response
among a subset of respondents using the Dirichlet Process. This method
integrates the strengths of the cultural consensus model with machine learning
techniques, overcoming their respective limitations. It has demonstrated both
predictive and explanatory benefits. The idlc-cct method offers both
scientific and practical applicability, allowing researchers to study group-
level variation while benefiting from the latest advances in machine learning.
The advantages of incorporating heterogeneity in consensus beliefs are
evident, and idlc-cct proves invaluable for analyzing and devising strategies
that promote consensus-building. Through the aggregation of information, it
advances collective intelligence. The integration of idlc-cct into
technological systems can enhance our understanding of consensus beliefs,
deepening our knowledge of the relevant domain.

## References

  * [1] Thomas S Wallsten and David V Budescu.  State of the art—encoding subjective probabilities: A psychological and psychometric review.  Management Science, 29(2):151–173, 1983.
  * [2] James Surowiecki.  The wisdom of crowds.  Anchor, 2005.
  * [3] Jeff Howe et al.  The rise of crowdsourcing.  Wired magazine, 14(6):1–4, 2006.
  * [4] Thomas W Malone, Robert Laubacher, and Chrysanthos Dellarocas.  The collective intelligence genome.  MIT Sloan Management Review, 2010.
  * [5] Eric Bonabeau.  Decisions 2.0: The power of collective intelligence.  MIT Sloan Management Review, 50(2):45, 2009.
  * [6] Barry L Bayus.  Crowdsourcing new product ideas over time: An analysis of the dell ideastorm community.  Management Science, 59(1):226–244, 2013.
  * [7] Zhi Da and Xing Huang.  Harnessing the wisdom of crowds.  Management Science, 66(5):1847–1867, 2020.
  * [8] Francis Galton.  Vox populi.  Nature, 75(1949):450–451, 1907.
  * [9] Alexander Philip Dawid and Allan M Skene.  Maximum likelihood estimation of observer error-rates using the em algorithm.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28, 1979.
  * [10] Asa B Palley and Ville A Satopää.  Boosting the wisdom of crowds within a single judgment problem: Weighted averaging based on peer predictions.  Management Science, 2023.
  * [11] David V Budescu and Eva Chen.  Identifying expertise to extract the wisdom of crowds.  Management Science, 61(2):267–280, 2015.
  * [12] Steffen R Giessner and Daan van Knippenberg.  “license to fail”: Goal definition, leader group prototypicality, and perceptions of leadership effectiveness after leader failure.  Organizational Behavior and Human Decision Processes, 105(1):14–35, 2008.
  * [13] Kristin A Broussard and Helen C Harton.  Tattoo or taboo? tattoo stigma and negative attitudes toward tattooed individuals.  The Journal of Social Psychology, 158(5):521–540, 2018.
  * [14] Xiaojun Lai and Pei-Luen Patrick Rau.  Has facial recognition technology been misused? a public perception model of facial recognition scenarios.  Computers in Human Behavior, 124:106894, 2021.
  * [15] Peter Berger and Thomas Luckmann.  The social construction of reality.  In Wired Magazine. Routledge, 1966.
  * [16] A Kimball Romney, Susan C Weller, and William H Batchelder.  Culture as consensus: A theory of culture and informant accuracy.  American Anthropologist, 88(2):313–338, 1986.
  * [17] Saeideh Heshmati, Zita Oravecz, Sarah Pressman, William H Batchelder, Chelsea Muth, and Joachim Vandekerckhove.  What does it mean to feel loved: Cultural consensus and individual differences in felt love.  Journal of Social and Personal Relationships, 36(1):214–243, 2019.
  * [18] Tiffany Rinne and John Fairweather.  A mixed methods approach: Using cultural modeling and consensus analysis to better understand new zealand’s international innovation performance.  Journal of Mixed Methods Research, 6(3):166–183, 2012.
  * [19] Necdet Gurkan and Jordan W Suchow.  Learning and enforcing a cultural consensus in online communities.  In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 44, 2022.
  * [20] Kai Yang, Raymond YK Lau, and Ahmed Abbasi.  Getting personal: A deep learning artifact for text-based measurement of personality.  Information Systems Research, 2022.
  * [21] Rong Liu, Feng Mai, Zhe Shan, and Ying Wu.  Predicting shareholder litigation on insider trading from financial text: An interpretable deep learning approach.  Information & Management, 57(8):103387, 2020.
  * [22] Luyang Chen, Markus Pelger, and Jason Zhu.  Deep learning in asset pricing.  Management Science, 2023.
  * [23] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  Nature, 521(7553):436–444, 2015.
  * [24] Sudeep Bhatia.  Predicting risk perception: New insights from data science.  Management Science, 65(8):3800–3823, 2019.
  * [25] Joshua C Peterson, Stefan Uddenberg, Thomas L Griffiths, Alexander Todorov, and Jordan W Suchow.  Deep models of superficial face judgments.  Proceedings of the National Academy of Sciences, 119(17):e2115228119, 2022.
  * [26] Necdet Gurkan and Jordan Suchow.  Cultural alignment of machine-vision representations.  In SVRHM 2022 Workshop@ NeurIPS, 2022.
  * [27] Sudeep Bhatia, Christopher Y Olivola, Nazlı Bhatia, and Amnah Ameen.  Predicting leadership perception with large-scale natural language data.  The Leadership Quarterly, 33(5):101535, 2022.
  * [28] Dan R Johnson, James C Kaufman, Brendan S Baker, John D Patterson, Baptiste Barbot, Adam E Green, Janet van Hell, Evan Kennedy, Grace F Sullivan, Christa L Taylor, et al.  Divergent semantic integration (dsi): Extracting creativity from narratives with distributional semantic modeling.  Behavior Research Methods, pages 1–34, 2022.
  * [29] Sebastian Gabel and Artem Timoshenko.  Product choice with large assortments: A scalable deep-learning model.  Management Science, 68(3):1808–1827, 2022.
  * [30] Jorge Guzman and Aishen Li.  Measuring founding strategy.  Management Science, 69(1):101–118, 2023.
  * [31] Katharina Lix, Amir Goldberg, Sameer B Srivastava, and Melissa A Valentine.  Aligning differences: Discursive diversity and team performance.  Management Science, 68(11):8430–8448, 2022.
  * [32] Mudasir A Ganaie, Minghui Hu, AK Malik, M Tanveer, and PN Suganthan.  Ensemble deep learning: A review.  Engineering Applications of Artificial Intelligence, 115:105151, 2022.
  * [33] Gemma Boleda.  Distributional semantics and linguistic theory.  Annual Review of Linguistics, 6:213–234, 2020.
  * [34] Sudeep Bhatia, Russell Richie, and Wanling Zou.  Distributed semantic representations for modeling human judgment.  Current Opinion in Behavioral Sciences, 29:31–36, 2019.
  * [35] Thomas L Griffiths, Mark Steyvers, and Joshua B Tenenbaum.  Topics in semantic representation.  Psychological Review, 114(2):211, 2007.
  * [36] Michael N Jones and Douglas JK Mewhort.  Representing word meaning and order information in a composite holographic lexicon.  Psychological Review, 114(1):1, 2007.
  * [37] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  Advances in Neural Information Processing Systems, 26, 2013.
  * [38] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, 2014.
  * [39] Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan.  Semantics derived automatically from language corpora contain human-like biases.  Science, 356(6334):183–186, 2017.
  * [40] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in Neural Information Processing Systems, 30, 2017.
  * [42] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [43] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al.  Transformers: State-of-the-art natural language processing.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, 2020.
  * [44] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi.  Comet: Commonsense transformers for automatic knowledge graph construction.  arXiv preprint arXiv:1906.05317, 2019.
  * [45] Zied Bouraoui, Jose Camacho-Collados, and Steven Schockaert.  Inducing relational knowledge from bert.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 7456–7463, 2020.
  * [46] Raja Marjieh, Ilia Sucholutsky, Theodore R Sumers, Nori Jacoby, and Thomas L Griffiths.  Predicting human similarity judgments using large language models.  arXiv preprint arXiv:2202.04728, 2022.
  * [47] Malhar Singh, Russell Richie, and Sudeep Bhatia.  Representing and predicting everyday behavior.  Computational Brain & Behavior, 5(1):1–21, 2022.
  * [48] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1026–1034, 2015.
  * [49] Dezső Ribli, Anna Horváth, Zsuzsa Unger, Péter Pollner, and István Csabai.  Detecting and classifying lesions in mammograms with deep learning.  Scientific reports, 8(1):4165, 2018.
  * [50] Chaochao Lu and Xiaoou Tang.  Surpassing human-level face verification performance on lfw with gaussianface.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015.
  * [51] Aditya Khosla, Akhil S Raju, Antonio Torralba, and Aude Oliva.  Understanding and predicting image memorability at a large scale.  In Proceedings of the IEEE international Conference on Computer Vision, pages 2390–2398, 2015.
  * [52] Brenden M. Lake, Wojciech Zaremba, Rob Fergus, and Todd M. Gureckis.  Deep neural networks predict category typicality ratings for images.  Cognitive Science, 2015.
  * [53] Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C Frank, James J DiCarlo, and Daniel LK Yamins.  Unsupervised neural network models of the ventral visual stream.  Proceedings of the National Academy of Sciences, 118(3):e2014196118, 2021.
  * [54] Joshua C Peterson, Joshua T Abbott, and Thomas L Griffiths.  Adapting deep network features to capture psychological representations.  arXiv preprint arXiv:1608.02164, 2016.
  * [55] Ruairidh M Battleday, Joshua C Peterson, and Thomas L Griffiths.  Capturing human categorization of natural images by combining deep networks and cognitive models.  Nature Communications, 11(1):5418, 2020.
  * [56] William H Batchelder and A Kimball Romney.  Test theory without an answer key.  Psychometrika, 53(1):71–92, 1988.
  * [57] Zita Oravecz, Royce Anders, and William H Batchelder.  Hierarchical bayesian modeling for test theory without an answer key.  Psychometrika, 80:341–364, 2015.
  * [58] Susan E Embretson and Steven P Reise.  Item response theory.  Psychology Press, 2013.
  * [59] Craig R Fox and Amos Tversky.  Ambiguity aversion and comparative ignorance.  The Quarterly Journal of Economics, 110(3):585–603, 1995.
  * [60] Susan C Weller.  Cultural consensus theory: Applications and frequently asked questions.  Field Methods, 19(4):339–368, 2007.
  * [61] William H Batchelder and Royce Anders.  Cultural consensus theory: Comparing different concepts of cultural truth.  Journal of Mathematical Psychology, 56(5):316–332, 2012.
  * [62] Royce Anders and William H Batchelder.  Cultural consensus theory for the ordinal data case.  Psychometrika, 80(1):151–181, 2015.
  * [63] Royce Anders, Zita Oravecz, and William H Batchelder.  Cultural consensus theory for continuous responses: A latent appraisal model for information pooling.  Journal of Mathematical Psychology, 61:1–13, 2014.
  * [64] Don van den Bergh, Stefan Bogaerts, Marinus Spreen, Rob Flohr, Joachim Vandekerckhove, William H Batchelder, and Eric-Jan Wagenmakers.  Cultural consensus theory for the evaluation of patients’ mental health scores in forensic psychiatric hospitals.  Journal of Mathematical Psychology, 98:102383, 2020.
  * [65] Hemant Ishwaran and Lancelot F James.  Gibbs sampling methods for stick-breaking priors.  Journal of the American statistical Association, 96(453):161–173, 2001.
  * [66] Thomas Griffiths, Michael Jordan, Joshua Tenenbaum, and David Blei.  Hierarchical topic models and the nested chinese restaurant process.  Advances in Neural Information Processing Systems, 16, 2003.
  * [67] Radford M Neal.  Markov chain sampling methods for dirichlet process mixture models.  Journal of Computational and Graphical Statistics, 9(2):249–265, 2000.
  * [68] Kristin A Duncan and Steven N MacEachern.  Nonparametric bayesian modelling for item response.  Statistical Modelling, 8(1):41–66, 2008.
  * [69] Kei Miyazaki and Takahiro Hoshino.  A bayesian semiparametric item response model with dirichlet process priors.  Psychometrika, 74(3):375–393, 2009.
  * [70] Michael D Lee.  How cognitive modeling can benefit from hierarchical bayesian models.  Journal of Mathematical Psychology, 55(1):1–7, 2011.
  * [71] Du Phan, Neeraj Pradhan, and Martin Jankowiak.  Composable effects for flexible and accelerated probabilistic programming in NumPyro.  arXiv preprint arXiv:1912.11554, 2019.
  * [72] James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, and Skye Wanderman-Milne.  JAX: composable transformations of Python+ NumPy programs, 2018.  URL http://github. com/google/jax, 4:16, 2020.
  * [73] Jun S Liu.  Peskun’s theorem and a modified discrete-state gibbs sampler.  Biometrika, 83(3), 1996.
  * [74] Matthew D Hoffman, Andrew Gelman, et al.  The no-u-turn sampler: adaptively setting path lengths in hamiltonian monte carlo.  J. Mach. Learn. Res., 15(1):1593–1623, 2014.
  * [75] Janet Holmes.  Making humour work: Creativity on the job.  Applied linguistics, 28(4):518–537, 2007.
  * [76] Mildred Golden Pryor, Lisa Pryor Singleton, Sonia Taneja, and John H Humphreys.  Workplace fun and its correlates: A conceptual inquiry.  International Journal of Management, 27(2):294, 2010.
  * [77] Wayne H Decker and Denise M Rotondo.  Relationships among gender, type of humor, and perceived leader effectiveness.  Journal of Managerial Issues, pages 450–465, 2001.
  * [78] W Jack Duncan, Larry R Smeltzer, and Terry L Leap.  Humor and work: Applications of joking behavior to management.  Journal of Management, 16(2):255–278, 1990.
  * [79] Jim Lyttle.  The judicious use and management of humor in the workplace.  Business Horizons, 50(3):239–245, 2007.
  * [80] Thomas W Cline, Moses B Altsech, and James J Kellaris.  When does humor enhance or inhibit ad responses?-the moderating role of the need for humor.  Journal of Advertising, 32(3):31–45, 2003.
  * [81] Thomas J Madden, Chris T Allen, and Jacquelyn L Twible.  Attitude toward the ad: An assessment of diverse measurement indices under different processing “sets”.  Journal of Marketing Research, 25(3):242–252, 1988.
  * [82] Dana L Alden, Ashesh Mukherjee, and Wayne D Hoyer.  The effects of incongruity, surprise and positive moderators on perceived humor in television advertising.  Journal of Advertising, 29(2):1–15, 2000.
  * [83] Martin Eisend.  How humor in advertising works: A meta-analytic test of alternative models.  Marketing Letters, 22:115–132, 2011.
  * [84] Yong Zhang.  Responses to humorous advertising: The moderating effect of need for cognition.  Journal of Advertising, 25(1):15–32, 1996.
  * [85] Nathan Walter, Michael J Cody, Larry Zhiming Xu, and Sheila T Murphy.  A priest, a rabbi, and a minister walk into a bar: A meta-analysis of humor effects on persuasion.  Human Communication Research, 44(4):343–373, 2018.
  * [86] Sergio Barta, Daniel Belanche, Ana Fernández, and Marta Flavián.  Influencer marketing on tiktok: The effectiveness of humor and followers’ hedonic experience.  Journal of Retailing and Consumer Services, 70:103149, 2023.
  * [87] Yong Zhang.  The effect of humor in advertising: An individual-difference perspective.  Psychology & Marketing, 13(6):531–545, 1996.
  * [88] Fred K Beard.  Advertising and audience offense: The role of intentional humor.  Journal of Marketing Communications, 14(1):1–17, 2008.
  * [89] Ken Goldberg, Theresa Roeder, Dhruv Gupta, and Chris Perkins.  Eigentaste: A constant time collaborative filtering algorithm.  Information Retrieval, 4:133–151, 2001.
  * [90] Zeyang Chen, Yu-Jane Liu, Juanjuan Meng, and Zeng Wang.  What’s in a face? an experiment on facial information and loan-approval decision.  Management Science, 2022.
  * [91] Christopher Y Olivola and Alexander Todorov.  Elected in 100 milliseconds: Appearance-based trait inferences and voting.  Journal of Nonverbal Behavior, 34:83–110, 2010.
  * [92] Heather D Flowe and Joyce E Humphries.  An examination of criminal face bias in a random sample of police lineups.  Applied Cognitive Psychology, 25(2):265–273, 2011.
  * [93] Christopher Y Olivola, Dawn L Eubanks, and Jeffrey B Lovelace.  The many (distinctive) faces of leadership: Inferring leadership domain from facial appearance.  The Leadership Quarterly, 25(5):817–834, 2014.
  * [94] Ulrich Mueller and Allan Mazur.  Facial dominance of west point cadets as a predictor of later military rank.  Social Forces, 74(3):823–850, 1996.
  * [95] Clare AM Sutherland, Xizi Liu, Lingshan Zhang, Yingtung Chu, Julian A Oldmeadow, and Andrew W Young.  Facial first impressions across culture: Data-driven modeling of chinese and british perceivers’ unconstrained facial impressions.  Personality and Social Psychology bulletin, 44(4):521–537, 2018.
  * [96] Leslie Zebrowitz McArthur and Diane S Berry.  Cross-cultural agreement in perceptions of babyfaced adults.  Journal of Cross-Cultural Psychology, 18(2):165–192, 1987.
  * [97] Shawn N Geniole, Danielle S Molnar, Justin M Carré, and Cheryl M McCormick.  The facial width-to-height ratio shares stronger links with judgments of aggression than with judgments of trustworthiness.  Journal of Experimental Psychology: Human Perception and Performance, 40(4):1526, 2014.
  * [98] Ryan M Stolier, Eric Hehman, and Jonathan B Freeman.  Trait knowledge forms a common structure across social cognition.  Nature Human Behaviour, 4(4):361–371, 2020.
  * [99] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8110–8119, 2020.
  * [100] Paul Slovic.  Perception of risk.  Science, 236(4799):280–285, 1987.
  * [101] Daniel Kahneman and Amos Tversky.  Prospect theory: An analysis of decision under risk.  Econometrica, 47(2):363–391, 1979.
  * [102] Paul Slovic and Elke U Weber.  Perception of risk posed by extreme events.  Regulation of Toxic Substances and Hazardous Waste (2nd edition)(Applegate, Gabba, Laitos, and Sachs, Editors), Foundation Press, Forthcoming, 2013.
  * [103] Baruch Fischhoff, Paul Slovic, Sarah Lichtenstein, Stephen Read, and Barbara Combs.  How safe is safe enough? a psychometric study of attitudes towards technological risks and benefits.  Policy Sciences, 9:127–152, 1978.
  * [104] Laressa L Wolfenbarger and Paul R Phifer.  The ecological risks and benefits of genetically engineered plants.  Science, 290(5499):2088–2093, 2000.
  * [105] Geoffrey Evans and John Durant.  The relationship between knowledge and attitudes in the public understanding of science in britain.  Public Understanding of Science, 4(1):57, 1995.
  * [106] Judith IM De Groot and Linda Steg.  Morality and nuclear energy: Perceptions of risks and benefits, personal norms, and willingness to take action related to nuclear energy.  Risk Analysis: An International Journal, 30(9):1363–1373, 2010.
  * [107] Melissa L Finucane, Ali Alhakami, Paul Slovic, and Stephen M Johnson.  The affect heuristic in judgments of risks and benefits.  Journal of Behavioral Decision Making, 13(1):1–17, 2000.
  * [108] Arthur G Jago.  Leadership: Perspectives in theory and research.  Management science, 28(3):315–336, 1982.
  * [109] S Bartholomew Craig and Sigrid B Gustafson.  Perceived leader integrity scale: An instrument for assessing employee perceptions of leader integrity.  The Leadership Quarterly, 9(2):127–145, 1998.
  * [110] James S Phillips and Robert G Lord.  Causal attributions and perceptions of leadership.  Organizational Behavior and Human performance, 28(2):143–163, 1981.
  * [111] Robert G Lord, Roseanne J Foti, and Christy L De Vader.  A test of leadership categorization theory: Internal structure, information processing, and leadership perceptions.  Organizational Behavior and Human Performance, 34(3):343–378, 1984.
  * [112] Andrew Carnes, Jeffery D Houghton, and Christopher N Ellison.  What matters most in leader selection? the role of personality and implicit leadership theories.  Leadership & Organization Development Journal, 36(4):360–379, 2015.
  * [113] Simon Capewell and Martin O’Flaherty.  Rapid mortality falls after risk-factor changes in populations.  The Lancet, 378(9793):752–753, 2011.
  * [114] Necdet Gurkan and Jordan W Suchow.  Predicting judgments of food healthiness with deep latent-construct cultural consensus theory.  In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45, 2023.
  * [115] Ellen Messer.  Anthropological perspectives on diet.  Annual Review of Anthropology, 13(1):205–249, 1984.
  * [116] Tim Lobstein and Stephen Davies.  Defining and labelling ‘healthy’and ‘unhealthy’food.  Public Health Nutrition, 12(3):331–340, 2009.
  * [117] Kevin Quealy and Margot Sanger-Katz.  Is sushi ‘healthy’? what about granola? where americans and nutritionists disagree.  The New York Times, 5, 2016.
  * [118] Manuel Barrera Jr, Felipe G Castro, Lisa A Strycker, and Deborah J Toobert.  Cultural adaptations of behavioral health interventions: a progress report.  Journal of Consulting and Clinical Psychology, 81(2):196, 2013.
  * [119] Sharon M Pearcey and Ginny Q Zhan.  A comparative study of american and chinese college students’ motives for food choice.  Appetite, 123:325–333, 2018.
  * [120] Jinan C Banna, Betsy Gilliland, Margaret Keefe, and Dongping Zheng.  Cross-cultural comparison of perspectives on healthy eating among chinese and american undergraduate students.  BMC Public Health, 16(1):1–12, 2016.
  * [121] Rie Akamatsu, Yuko Maeda, Akihito Hagihara, and Taro Shirakawa.  Interpretations and attitudes toward healthy eating among japanese workers.  Appetite, 44(1):123–129, 2005.
  * [122] Gavin Turrell and Anne M Kavanagh.  Socio-economic pathways to diet: modelling the association between socio-economic position and food purchasing behaviour.  Public Health Nutrition, 9(3):375–383, 2006.
  * [123] Jacob L Orquin.  A brunswik lens model of consumer health judgments of packaged foods.  Journal of Consumer Behaviour, 13(4):270–281, 2014.
  * [124] Jonathon P Schuldt.  Does green mean healthy? nutrition label color affects perceptions of healthfulness.  Health Communication, 28(8):814–821, 2013.
  * [125] Natasha Gandhi, Wanling Zou, Caroline Meyer, Sudeep Bhatia, and Lukasz Walasek.  Computational methods for predicting and understanding food judgment.  Psychological Science, 33(4):579–594, 2022.
  * [126] Yann LeCun, John Denker, and Sara Solla.  Optimal brain damage.  Advances in Neural Information Processing Systems, 2, 1989.
  * [127] Marc Oliver Rieger, Mei Wang, and Thorsten Hens.  Risk preferences around the world.  Management Science, 61(3):637–648, 2015.
  * [128] Clare AM Sutherland, Nichola S Burton, Jeremy B Wilmer, Gabriëlla AM Blokland, Laura Germine, Romina Palermo, Jemma R Collova, and Gillian Rhodes.  Individual differences in trust evaluations are shaped mostly by environments, not genes.  Proceedings of the National Academy of Sciences, 117(19):10218–10224, 2020.
  * [129] Anne S Tsui and Charles A O’reilly III.  Beyond simple demographic effects: The importance of relational demography in superior-subordinate dyads.  Academy of management journal, 32(2):402–423, 1989.
  * [130] Elizabeth Dubois and Grant Blank.  The echo chamber is overstated: the moderating effect of political interest and diverse media.  Information, communication & society, 21(5):729–745, 2018.
  * [131] Matteo Cinelli, Gianmarco De Francisci Morales, Alessandro Galeazzi, Walter Quattrociocchi, and Michele Starnini.  The echo chamber effect on social media.  Proceedings of the National Academy of Sciences, 118(9):e2023301118, 2021.
  * [132] Yee Whye Teh et al.  Dirichlet process.  Encyclopedia of machine learning, 1063:280–287, 2010.
  * [133] Michael E Tipping and Christopher M Bishop.  Probabilistic principal component analysis.  Journal of the Royal Statistical Society: Series B (Statistical Methodology), 61(3):611–622, 1999.
  * [134] Sam Roweis.  Em algorithms for pca and spca.  Advances in Neural Information Processing Systems, 10, 1997.
  * [135] Brian Kulis and Michael I Jordan.  Revisiting k-means: New algorithms via bayesian nonparametrics.  arXiv preprint arXiv:1111.0352, 2011.
  * [136] Ke Jiang, Brian Kulis, and Michael Jordan.  Small-variance asymptotics for exponential family dirichlet process mixture models.  Advances in Neural Information Processing Systems, 25, 2012.
  * [137] Klaus Fiedler.  Explaining and simulating judgment biases as an aggregation phenomenon in probabilistic, multiple-cue environments.  Psychological Review, 103(1):193, 1996.
  * [138] Linda Smith and Michael Gasser.  The development of embodied cognition: Six lessons from babies.  Artificial Life, 11(1-2):13–29, 2005.
  * [139] Dhanesh Ramachandram and Graham W Taylor.  Deep multimodal learning: A survey on recent advances and trends.  IEEE Signal Processing Magazine, 34(6):96–108, 2017.