Monte Carlo Integration in Simple and Complex Simulation Designs

Ashley I. Naimi, PhD1,∗
David Benkeser, PhD2
Jacqueline E. Rudolph, PhD3

1 Department of Epidemiology, Emory University.
2 Department of Biostatistics, Emory University.
3 Department of Epidemiology, Johns Hopkins University.

*Correspondence: | Department of Epidemiology
---|---
| Rollins School of Public Health
| Emory University
| 1518 Clifton Road
| Atlanta, GA 30322
|<EMAIL_ADDRESS>

Conflicts of Interest: None

Acknowledgements:

Sources of Funding: AIN was supported by NIH grant number R01 HD102313.

Target Journal: | _Epidemiol_
---|---
Text word count: | 2,063
Abstract word count: | 178
Number of Figures: | 2
Number of Tables: | 0
Number of References: | 15
Running head: | Monte Carlo Integration for Simulation Estimands

Abstract

Simulation studies are used to evaluate and compare the properties of
statistical methods in controlled experimental settings. In most cases,
performing a simulation study requires knowledge of the true value of the
parameter, or estimand, of interest. However, in many simulation designs, the
true value of the estimand is difficult to compute analytically. Here, we
illustrate the use of Monte Carlo integration to compute true estimand values
in simple and complex simulation designs. We provide general pseudocode that
can be replicated in any software program of choice to demonstrate key
principles in using Monte Carlo integration in two scenarios: a simple three
variable simulation where interest lies in the marginally adjusted odds ratio;
and a more complex causal mediation analysis where interest lies in the
controlled direct effect in the presence of mediator-outcome confounders
affected by the exposure. We discuss general strategies that can be used to
minimize Monte Carlo error, and to serve as checks on the simulation program
to avoid coding errors. R programming code is provided illustrating the
application of our pseudocode in these settings.

KEY WORDS: Epidemiologic methods; Monte Carlo Simulation; Monte Carlo
integration; Numeric integration; Causal Inference; Statistics.

## Introduction

Simulation studies are often employed to evaluate and compare the properties
of statistical methods in controlled experimental settings. Simulation studies
can be used towards a wide range of aims, from basic conceptual or didactive
purposes,1 to formal research questions about the performance of different
analytic methods.2 Monte Carlo simulation studies are so named because they
belong to a general category of techniques known as Monte Carlo methods,3 a
class of methods that rely on _in silico_ (pseudo) random number generation to
find solutions to problems that may otherwise be intractable.4

The design of Monte Carlo simulation studies often involves several steps.
These include clearly articulating the aims of the simulation study, the data
generating mechanisms that will be used, the estimand of interest, the methods
that will be used to estimate the estimand, and the performance measures used
to evaluate the properties of the methods under study.2 Simulation studies
proceed by repeatedly simulating data sets from the selected data generating
mechanism. Each simulated data set is analyzed using the selected methods. The
performance of the selected methods is compared by aggregating results across
multiple simulated data sets. For example, to approximate the bias of an
estimator, we can compute the empirical average (across all simulated data
sets) difference between the point estimates and the truth. To approximate the
coverage of a method for building a confidence interval, we can compute the
empirical proportion (out of all simulated data sets) of intervals that
contain the true parameter value. This makes clear the importance for the
researcher to know the precise true value of the estimand that is implied by
the selected data generating process.

In ideal settings, the true value of the estimand will be directly
ascertainable from the parameter values used in the data generating mechanism.
For example, if the data generating mechanism involves simulating a binary
outcome $Y$ using a logistic regression model that includes a single binary
exposure $X$, and a single continuous confounder $C$, $P(Y=1\mid
X,C)=\mbox{expit}\\{\beta_{0}+\beta_{1}X+\beta_{2}C\\}$. To simulate data from
this data generating distribution, we would generally need to select values
for $\beta_{0},\beta_{1}$, and $\beta_{2}$.

If our simulation study is interested in the conditionally adjusted odds ratio
for the exposure-outcome association, then the true estimand value is easily
computed as $\mbox{exp}(\beta_{1})$, where $\beta_{1}$ is the researcher-
selected value.

In such simple examples, it is possible to read off the true estimand value
directly from the data generating process. However, in many settings,
simulation studies are used to evaluate estimators of estimands whose values
cannot be so easily ascertained. Importantly, in such settings, the true
estimand value may not be easily computed as a single parameter, or simple
combination of parameters, used to generate the data. Here, we outline the use
of Monte Carlo integration for solving the true estimand value. We illustrate
the approach using two example scenarios: a data generating mechanism where
the estimand of interest is the marginally adjusted odds ratio; and a causal
mediation setting where interest lies in the controlled direct effect as the
estimand. We provide pseudocode for each example that can be used to construct
a program implementing Monte Carlo integration in a programming language of
choice. We also provide R code for each example in an associated GitHub
repository.

## Example 1: Marginally Adjusted Odds Ratio as the Estimand

The first example involves estimating the marginally adjusted odds ratio using
methods such as inverse probability weighting or marginal standardization 5,
6. The causal diagram representing our data generating mechanism7 is displayed
in Figure 1:

Figure 1: Causal diagram depicting the relationships between an exposure $A$,
outcome $Y$, and confounder $C$.

A parametric approach to simulating data compatible with the causal diagram in
Figure 1 is to first simulate $C$ from, e.g., a Normal distribution with mean
$\mu$ and variance $\sigma^{2}$. Then based on the simulated values of $C$,
simulate the exposure $X$ from a logistic regression model with $P(X=1\mid
C)=\mbox{expit}(\alpha_{0}+\alpha_{1}C)$, for some researcher-selected values
$\alpha_{0},\alpha_{1}$. Finally, the outcome is simulated from a logistic
regression model

$P(Y=1\mid
X,C)=\operatorname{expit}\left\\{\beta_{0}+\beta_{1}X+\beta_{2}C\right\\}\ .$
(1)

As noted above, if the true parameter of interest was the conditionally
adjusted odds ratio, its value could easily be obtained as $\exp(\beta_{1})$.
However, if our interest lies in evaluating the performance of methods that
estimate the marginally adjusted odds ratio, we would have to compute the true
value of this estimand entailed by the model.

There are well known and important differences between marginally versus
conditionally adjusted odds ratios,8, 9, 10 due to the fact that the odds
ratio (logistic regression) is a noncollapsible measure of effect.11
Noncollapsibility is distinct from confounding,12 which means that for a given
logistic regression model with the same set of correctly chosen confounders,
the marginally versus conditionally adjusted odds ratio may be numerically
different. This means that, in the context of equation 1, while exponentiating
the value of the $\beta_{1}$ coefficient will provide us with a numerical
value for the conditionally adjusted odds ratio, the marginally adjusted odds
ratio may be a different value.

Under ideal conditions, one would rely on algebraic derivations to obtain an
analytic solution to the marginally adjusted odds ratio. Computing a
marginally adjusted odds ratio analytically from a model such as equation 1
could be accomplished via integration, which would require solving for:

$\mu(x)=\int_{c}\operatorname{expit}\\{\beta_{0}+\beta_{1}x+\beta_{2}c\\}\frac{1}{(2\pi)^{1/2}\sigma}\mbox{exp}\left\\{\frac{(c-\mu)^{2}}{2\sigma^{2}}\right\\}dc\
,$

for $x=0,1$. The marginally adjusted odds ratio can then be computed as:

$\psi=\frac{\mu(1)}{1-\mu(1)}\bigg{/}\frac{\mu(0)}{1-\mu(0)}$

However, even in this simple setting with a single variable $C$, analytically
solving for this integral is challenging. One could employ the use of numeric
integration available in many statistical software packages (See eAppendix 1).
However, this approach would not scale well to settings with more than one $C$
variable.

Instead, we can use Monte Carlo integration to solve for $\psi$, as follows:

Algorithm 1 Pseudocode for Implementing Monte Carlo Integration to Solve for
the True Marginally Adjusted Odds Ratio in a Simple Three Node Causal Diagram

1:set random number generator seed value

2:set large sample size $N$

3:simulate $i\in 1\ldots N$ observations $C_{i}$ from a Normal distribution
with mean $\mu$ and variance $\sigma^{2}$

4:compute $\hat{\mu}_{i}(x)=\mbox{expit}(\beta_{0}+\beta_{1}x+\beta_{2}C_{i})$
for both $x=0$ and $x=1$. Thus,
$\hat{\mu}_{i}(0)=\operatorname{expit}\left(\beta_{0}+\beta_{2}C_{i}\right)$
and
$\hat{\mu}_{i}(1)=\operatorname{expit}\left(\beta_{0}+\beta_{1}+\beta_{2}C_{i}\right)$.

5:the approximated value of $\mu(x)$ is given by the mean of
$\hat{\mu}_{i}(x)$ over all $N$ simulated observations,
$\hat{\mu}(x)=\frac{1}{N}\sum_{i=1}^{N}\hat{\mu}_{i}(x)$. The approximated
value of $\psi$ is given by
$\hat{\psi}=\frac{\left[\hat{\mu}(1)/(1-\hat{\mu}(1))\right]}{\left[\hat{\mu}(0)/(1-\hat{\mu}(0))\right]}$

The approximated value of $\psi$ obtained from the pseudocode above can be
used as the true estimand value for a simulation study evaluating the
properties of an estimator seeking to quantify the marginally adjusted odds
ratio. However, it is important to note that this "true" estimand value
depends on: (i) the specifications for the distribution of $C$; and (ii) the
parameter values for the logistic regression model generating $Y$ in this
case, $\beta_{0},\beta_{1},\beta_{2}$. If the distribution of $C$ changes, or
if the regression model coefficients change, this pseudoalgorithm should be
run again under these new settings. Additionally, the value of $\psi$ will be
subject to error that depends on the sample size used in the Monte Carlo
integration. For this reason, $N$ should be chosen to be as large as possible
under the allocated computing constraints. Several iterations of the pseudo-
code could be run to confirm that the sample is large enough to lead to only
minimal changes in the approximated value.

Of note, in step 5 of the above pseudocode, one should compute $Q^{a_{1}}_{i}$
and $Q^{a_{0}}_{i}$ directly from the logistic model for the outcome, and not
to convert these probabilities to binary outcomes with a draw from the
Bernoulli distribution. Doing so prevents the introduction of additional Monte
Carlo error, and reduces the variability of the Monte Carlo integration
technique.

## Example 2: Controlled Direct Effect as the Estimand

The second example consists of a more complex causal mediation analysis where
interest may lie in conducting a simulation study to evaluate the properties
of methods used to estimate the controlled direct effect (CDE). Here, we
define the CDE of interest on the difference scale, but other scales could be
entertained:

$\psi=E(Y^{a,m}-Y^{a^{\star},m})$

In the above, $Y^{a,m}$ is the potential outcome that would be observed if the
exposure $A$ were set to some value $a$ and the mediator $M$ were set to some
value $m$. The causal diagram representing our data generating mechanism is
displayed in Figure 2.

Figure 2: Causal diagram depicting the relationships between an exposure $A$,
a mediator $M$, a confounder $C$, a mediator-outcome confounder affected by
the exposure $L$, and an outcome $Y$.

In this example, the true value of the controlled direct effect is difficult
to compute analytically. It consists of a a function of the magnitudes of the
following paths in Figure 2:

$A\rightarrow Y$ $A\rightarrow M\rightarrow Y$ $A\rightarrow L\rightarrow Y$

In particular, the specific function that should be used to combine these
magnitudes will depend on how $M$, $L$, and $Y$ are distributed, and whether
the effect magnitudes for each arrow are on the multiplicative or additive
scales. In fact, in many settings, it simply may not be possible to
analytically solve for the true controlled direct effect. However, we can use
Monte Carlo integration as follows:

Algorithm 2 Pseudocode for Implementing Monte Carlo Integration to Solve for
the True Controlled Direct Effect in a Causal Mediation Analysis DAG with a
Mediator-Outcome Confounder Affected by the Exposure.

1:set random number generator seed value.

2:set large sample size $N$.

3:simulate $i\in 1\ldots N$ observations $C_{i}$ and $U_{i}$ from a
distribution of choice.

4:construct $a_{1}$ and $a_{0}$ for all $N$ observations corresponding to
exposed (e.g., $a_{1}=1$) and referent (e.g., $a_{0}=0$) states.

5:simulate two $L_{i}$ variables, one corresponding to $a_{1}$
($L^{a_{1}}_{i}$) and one corresponding to $a_{0}$ ($L^{a_{0}}_{i}$), for all
$N$ observations. This would be from a regression model that includes the
simulated $U_{i}$ and the specified $a_{x}$ value, along with relevant
parameters.

6:construct $m$ for all $N$ observations corresponding to the specific
mediator value of interest in the controlled direct effect contrast (e.g.,
$m=0$ for all $N$).

7:simulate $Q^{a_{x},m}_{i}$ for all $i\in 1\ldots N$ observations from the
model for the outcome:
$Q^{a_{x},m}_{i}=f\left(\beta_{0}+\beta_{1}a_{x}+\beta_{2}C_{i}+\beta_{3}m+\beta_{4}L_{i}+\beta_{5}U_{i}\right),$
where $f(\bullet)$ represents a specified link function of interest.

8:take the mean of $Q^{a_{x},m}_{i}$ over all $N$ to obtain
$\mu_{a_{x},m}=\frac{1}{N}\sum_{i=1}^{N}Q^{a_{x},m}_{i}$, and construct a
difference measure as:  $\widetilde{\psi}=\mu_{a_{1},m}-\mu_{a_{0},m}$

Again, the value of $\widetilde{\psi}$ obtained from the psuedocode above can
be used as the true estimand value for the controlled direct effect under the
specified simulation settings. The magnitude of this true value will once
again depend on the specific parameter values, distributions, and functional
forms selected to generate all variables used in the outcome model. If any of
these are changed, a separate $\widetilde{\psi}$ should be quantified.
Additionally, as a check of the algorithm’s validity, one could use Monte
Carlo integration to estimate the true conditional association between the
exposure-outcome, mediator-outcome, or confounder-outcome relationships, or
any other known quantity in the data generating mechanism under study.

## Discussion

Monte Carlo simulation studies are commonly used to evalaute the properties of
estimators in a range of different scientific settings. However, estimator
properties such as bias, mean squared error, and confidence interval coverage
require a numerical value for the true estimand of interest. In many settings,
this true value cannot be obtained analytically, and Monte Carlo integration
can be used instead.

There are several important issues to consider when designing a program to
implement Monte Carlo integration to compute estimand values. First, general
recommendations suggest that the selected sample size used to compute the true
value should be as large as is computationally feasible to minimize Monte
Carlo error to the smallest value possible. However, in practice a degree of
Monte Carlo error could be tolerated. For example, if increasing the Monte
Carlo sample size past a certain boundary $\kappa$ only changes the estimand
value at the 5th or greater decimal place, it might be advisable to set the
Monte Carlo sample size to $\kappa$ to avoid expending resources for little
meaningful gain. Second, it is important to note that the source of Monte
Carlo error in any simulation study is derived from programming calls to the
system’s random number generator. For example, in the R programming language,
functions used to generate observations from a distribution of interest [e.g.,
rnorm(), runif(), or rexp()], or functions used to sample from a set of
observations, will contribute to Monte Carlo error. For this reason, one
strategy that should be used to minimize Monte Carlo error in the estimand
value is to minimize these calls where possible (e.g., Step 5 in Algorithm 1,
or Step 7 in Algorithm 2).

Several classes of simulation study design exist. In fully parametric
simulation studies, each component of the data generating mechaism is
specified parametrically using researcher defined functions. For example,
confounders, exposures, and outcomes will all be drawn from known
distributions with pre-specific relationships between each. Alternatively,
plasmode simulations can be conducted in which a portion of the data
generating mechanism (e.g., confounders) can be constructed using actual data
sampled from a population of interest.13 Doing so preserves the empirical
assocaitions between variables drawn from the sample, thus providing a closer
approximation between simulated and real data. Finally, synthetic simulations
have recently been developed that rely on machine learning to simulate data
whose variables and joint distributions between them closely approximate an
index dataset sampled from a real population of interest, but where the effect
of the exposure of interest is specified _a priori_.14, 15 In principle, Monte
Carlo integration can be used in all three settings to derive the true
estimand of interest, or to serve as a verification step in the process of
simulating complex data. Indeed, the Monte Carlo method is a general
methodological strategy that has been used across the sciences to solve simple
and complex problems. With increasingly complex analyses being carried out in
epidemiology and the health sciences, the Monte Carlo method, and Monte Carlo
intergation more specifically, will be a useful tool to gain insights into
complex questions.

## References

  * 1 Rudolph JE, Fox MP, and Naimi AI.  Simulation as a tool for teaching and learning epidemiologic methods.  _American Journal of Epidemiology_. 2021; 190:900–907.
  * 2 Morris TP, White IR, and Crowther MJ.  Using simulation studies to evaluate statistical methods.  _Statistics in Medicine_. 2019; 38:2074–2102.
  * 3 Metropolis N and Ulam S.  The Monte Carlo method.  _J Am Stat Assoc_. 1949; 44:335–341.
  * 4 McCracken DD.  The monte carlo method.  _Scientific American_. 1955; 192:90–97.
  * 5 Naimi, AI, Cole SR, Kennedy EH.  Introduction to G Methods.  _International Journal of Epidemiology_. 2017; 46:756-762.
  * 6 Naimi, AI, Whitcomb BW.  Estimating Risk Ratios and Risk Differences Using Regression.  _American Journal of Epidemiology_. 2020; 189:508-510.
  * 7 Fox MP, Nianogo R, Rudolph JE, and Howe CJ.  Illustrating how to simulate data from directed acyclic graphs to understand epidemiologic concepts.  _American Journal of Epidemiology_. 2022; 191:1300–1306.
  * 8 Neuhaus JM and Jewell NP.  A geometric approach to assess bias due to omitted covariates in generalized linear models.  _Biometrika_. 1993; 80:807–815.
  * 9 Daniel R, Zhang J, and Farewell D.  Making apples from oranges: Comparing noncollapsible effect estimators and their standard errors after adjustment for different covariate sets.  _Biom J_. 2021; 63:528–557.
  * 10 Pang M, Kaufman JS, and Platt RW.  Studying noncollapsibility of the odds ratio with marginal structural and logistic regression models.  _Stat Methods Med Res_. 2013; .
  * 11 Greenland S.  Collapsibility.  In: Gail MH and Bénichou J (Eds.) _Encyclopedia of Epidemiologic Methods_. John Wiley & Sons, Ltd. 2005; .
  * 12 Pang M, Kaufman JS, and Platt RW.  Mixing of confounding and non-collapsibility: a notable deficiency of the odds ratio.  _Am J Cardiol_. 2013; 111:302–303.
  * 13 Franklin JM, Schneeweiss S, Polinski JM, and Rassen JA.  Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases.  _Comput Stat Data Anal_. 2014; 72:219–226.
  * 14 Athey S, Imbens GW, Metzger J, and Munro E.  Using wasserstein generative adversarial networks for the design of monte carlo simulations.  _Journal of Econometrics_. 2021; :105076.
  * 15 Parikh H, Varjao C, Xu L, and Tchetgen ET.  Validating causal inference methods.  _International Conference on Machine Learning_. 2022; PMLR.


# Safer Conversational AI as a Source of User Delight

Xiaoding Lu &Aleksey Korshuk &Zongyi Liu &Vineet Mudupalli Christie-Carol
Beauchamp &Thomas Rialan &William Beauchamp
Chai Research

###### Abstract

This work explores the impact of moderation on users’ enjoyment of
conversational AI systems. While recent advancements in Large Language Models
(LLMs) have led to highly capable conversational AIs that are increasingly
deployed in real-world settings, there is a growing concern over AI safety and
the need to moderate systems to encourage safe language and prevent harm.
However, some users argue that current approaches to moderation limit the
technology, compromise free expression, and limit the value delivered by the
technology. This study takes an unbiased stance and shows that moderation does
not necessarily detract from user enjoyment. Heavy handed moderation does seem
to have a nefarious effect, but models that are moderated to be safer can lead
to a better user experience. By deploying various conversational AIs in the
Chai platform, the study finds that user retention can increase with a level
of moderation and safe system design. These results demonstrate the importance
of appropriately defining safety in models in a way that is both responsible
and focused on serving users.

##  1 Introduction

Recent advancements in Large Language Models (LLMs) has lead to the
development of highly capable conversational AIs Thoppilan et al. (2022), such
as chatGPT. These systems have been rapidly growing in popularity and are
being increasingly deployed in real world settings Kung et al. (2023); Huang
et al. (2022). With the prevalence of AI solutions in the real world as tools
or as services, there has been huge attention in AI safety and ensuring that
developed systems are moderated to be helpful, wholesome and truthful Hanu and
Unitary team (2020); Si et al. (2022); Xu et al. (2020); Manakul et al.
(2023). There is, however, as well a community of users that believe that
systems are being over-moderated, and that moderation limits their freedom of
expression and delight in these systems Llansó (2020); Hammontree (2023). For
example within the entertainment domain, users may wish the conversational AIs
to impersonate famous characters, which moderation may restrict them from
doing. Further, individual intentions may not always align with the subjective
safety standards of the large research institutions developing this
technology. It seems important for businesses, and society at large to be
willing to discuss what amount and nature of moderation will best serve the
needs of the users of this technology.

This paper attempts to take an unbiased stance, and determine the extent to
which moderation of conversational AIs influences users’ perceived enjoyment
of the conversational AI system. We empirically show that safety does not
necessarily come at the expense of user enjoyment; models that are moderated
to be safer (to a certain extent), even if they ignore user’s requests, can
lead to a greater user experience. We deploy five conversational AIs in the
Chai platform with varying levels of safety and methods to encourage safe
language. By considering the downstream user-retention, we find that user-
retention can be higher for the moderated systems than for the unmoderated
systems. This underscores the importance of a nuanced, light-touch approach to
moderation as a way to better serve users.

##  2 Related Work

In recent years, there has been the emergence of a large number of social
conversational AIs for chitchat with human users (Yan et al., 2022; Bao et
al., 2020; Adiwardana et al., 2020; Irvine et al., 2023; Choudhary and
Kawahara, 2022).

Unsafe Content With the deployment of these powerful conversational AIs in
real-world settings, there has been an increasing concern regarding their
safety - it is required that systems do not generate harmful or illegal
content. Schmidt and Wiegand (2017) discuss the scope of unsafe content,
covering various aspects of abusive behaviour: hate speech, profanity,
malicious intent, hostility, abusive messages and cyber-bullying. There have
been extensive efforts in defining thresholds and categories for unsafe
responses (Swamy et al., 2019; Waseem et al., 2017; Magu et al., 2017;
Zampieri et al., 2019; Caselli et al., 2020; van Aken et al., 2018). A popular
categorization (Ram et al., 2018; Paranjape et al., 2020) for offensive
content is: profane content, sexual content, racial content, other hate speech
and violent crime. With greater granularity, Zhang et al. (2020) outline a
more detailed hierarchical taxonomy including concepts such as privacy
invasion, jealousy and other forms of malevolent content.

Design of Safe conversational AI In conversational AI-human conversations,
unsafe content can arise from either the human or the conversational AI. If a
human response is identified as harmful, systems will typically return non-
committal safe responses or attempt to change the subjects (Cercas Curry and
Rieser, 2019; Paranjape et al., 2020). However, malicious users can also
design undetectable specific adversarial prompts (Liu et al., 2020; Hill et
al., 2015; Roller et al., 2020), where for example a carefully crafted user
input can trigger a model to generate offensive content (Wallace et al.,
2019). Adversarial susceptibility of conversational AIs is an open research
area. When considering generated conversational AI responses, various
approaches have been proposed in literature to mitigate the risk of unsafe
content generation. These approaches can be grouped into three main categories
(Xu et al., 2021). The first approach is data curation (Roller et al., 2020;
Rashkin et al., 2019), where models are trained on safe data to reduce the
likelihood of generating harmful content. However, such under exposed models
struggle to respond to offensive user content and are also susceptible to
adversarial prompts (Gehman et al., 2020). Alternatively or as an addition to
data curation, a safety layer can be introduced through the use of a detection
system (Zampieri et al., 2019; Founta et al., 2018; Dinan et al., 2020; Khatri
et al., 2018) that can identify system generated unsafe responses and reject
them. Finally, a suite of methods explore controlled generation, where a model
is trained to conditionally generate outputs (Smith et al., 2020; Krause et
al., 2021), offering the user/designer a level of control on the offensiveness
of the conversational AI responses. For example Niu and Bansal (2018) give a
control on model politeness and Nogueira dos Santos et al. (2018) show how
offensive sentences can be transformed to benign ones.

Safety and User Engagement A typical business is concerned with maximising the
engagingess of their conversational AI Irvine et al. (2023), as this
translates to a higher level of user retention. There has however been limited
research investigating the impact of designing safer models on the level of
user engagement. Nevertheless, Xu et al. (2021) find that there is little
correlation between safety and conversational AI engagingness, where human
evaluation is used to measure both model safety and engagingness. This work
explores the interaction between conversational AI safety and engagingness in
greater depth.

##  3 Method: Design of a safe chat AI

Modern deep learning-based sequence-to-sequence chat AI systems are typically
designed to take as an input a specific user prompt sequence, $\mathbf{x}$ and
a chat history (previous user and system messages), $\mathbf{h}$, to output
the response, $\hat{\mathbf{y}}$ with the highest likelihood, as per the chat
AI model, $\mathcal{M}$,

$\hat{\mathbf{y}}=\operatorname*{arg\,max}_{\mathbf{y}}\left\\{p_{\mathcal{M}}(\mathbf{y}|\mathbf{x},\mathbf{h};\hat{\theta})\right\\},$
(1)

where $\hat{\theta}$ are the model’s trained parameters. These parameters are
learnt during training in a standard unsupervised manner, using masked
language modeling, where the model predicts the masked tokens.

###  3.1 Design

The simplest approach to ensure a chat AI system, $\mathcal{M}$ only generates
safe sequences is to perform masked language modeling training on only safe
data, i.e. such that the model is not exposed to unsafe content during
training. From a general dataset, $\mathcal{D}=\\{\mathbf{y}_{i}\\}$, we can
extract a safe dataset, $\mathcal{D}^{(s)}$,

$\mathcal{G}(\mathbf{y})<\epsilon\implies\mathbf{y}\in\mathcal{D}^{(s)},\hskip
20.00003pt\mathbf{y}\in\mathcal{D},$ (2)

where $\mathcal{G}$ is a measure of the threat of a particular sequence (i.e.
a measure of how unsafe the model is), $\mathbf{y}$ and $\epsilon$ is an
agreed threshold of safety. However it is argued that a model under-exposed to
unsafe content fails when exposed to harmful user content and may for example
just quote the harmful content back. To mitigate the risk of such undesirable
behaviour, this work assumes the standard pipeline of pre-training the
language model on general data, $\mathcal{D}$ and then finetuning it on a
curated, safe dataset, $\mathcal{D}^{(s)}$. This exposes the model to unsafe
content during pre-training but encourages it to only generate safe content
after finetuning, i.e. $\mathcal{G}(\mathbf{y})<\epsilon$.

It is expensive and inefficient to use human annotation to define the threat,
$\mathcal{G}$, of any sequence $\mathbf{y}$ and thus in this work we explore
an automated measure to act as a proxy for human perception of model threat.
Specifically, a popular and well established moderation tool, OpenAI’s
moderation endpoint 111OpenAI moderation tool:
https://platform.openai.com/docs/guides/moderation/overview is used as a
binary threat classifier as a proxy for, $\mathcal{G}$. The tool also
identifies the following groupings: hate, hate/threatening, sexual,
sexual/minors, violence, violence/graphic. Any sample in the training set
$\mathcal{D}$ flagged as risk as per these categories, is excluded from the
safe dataset, $\mathcal{D}^{(s)}$, as specified in Equation 2.

###  3.2 Safety Evaluation

Having trained a chat AI to be safe, it is necessary to have an independent
and systematic measure of the safety of generated responses from the chat AI
to verify that the design method does truly create safer responses. Hence, for
a set of chat AI responses, $\\{\hat{\mathbf{y}}\\}_{n=1}^{N}$, we can
calculate a safety score, $\mathcal{S}$ defined as:

$\mathcal{S}=1-\frac{1}{N}\sum_{n=1}^{N}\tilde{\mathcal{G}}(\hat{\mathbf{y}}),$
(3)

where $\tilde{\mathcal{G}}$ represents the threat score normalized to lie
between 0 and 1, to behave as a probability. Once again, it is expensive and
inefficient to use human evaluation at scale for the threat measure,
$\mathcal{G}$ and thus this work uses an automated proxy measure of threat at
inference time. To reduce bias towards the design method, it is necessary to
use a different (from open AI’s moderation tool) proxy function for
$\mathcal{G}$. Hence, this work uses a standard deep-learning based model,
$\mathcal{Q}$, trained in a supervised manner on pre-curated public safety
dataset (binary classification task to identify unsafe and safe samples)
222Model used for evaluation of safety: https://huggingface.co/unitary/toxic-
bert, as a proxy for threat classification at inference time,

$\mathcal{G}(\mathbf{y})\approx P_{\mathcal{Q}}(\mathbf{y}).$ (4)

##  4 Experiments

###  4.1 Set-Up

conversational AI Model In this paper we consider 5 main conversational AI
systems: chat-ai-baseline-u is a GPT-J 6B (Wang and Komatsuzaki, 2021) fine-
tuned on novels and literature333https://huggingface.co/hakurei/lit-6B, which
was found to contain some unsafe language. chat-ai-base-s is another baseline,
where chat-ai-base-u is further fine-tuned on safer literature. Next, we
introduce chat-ai-safe, which is the chat-ai-base-s model trained on real user
suggested edits on the Chai platform 444The Chai platform allows for users to
edit the response given by the system, filtered to retain only safe
suggestions, as described in Section 2. For further analysis, we consider
chat-ai-unsafe, which again takes the chat-ai-base-s model, but now trains it
on all real user suggested edits, which have been found to contain some unsafe
language. As a final comparison, we consider a model chat-ai-unsafe-mod, where
the chat-ai-unsafe model is deployed with a safety moderation layer to reject
user responses detected as unsafe.

User Retention To evaluate the conversational AIs, we deploy them to the Chai
Research platform and look at the user retention after 30 days. This is done
by creating mutually exclusive cohorts of new users and assigning them to one
of the five systems. The 30 day user retention is the number of users that
return to the app after 30 days of signing up, and gives a good indication of
the user’s experience with the conversational AI.

Safety Evaluation Dataset We evaluate the safety of each model as detailed in
Section 3.2. The safety score is calculated over a dataset of user-
conversational AI conversations, returning the probability of system-generated
responses being safe. The safety score (probability) is given by Equation 3.
Each model’s dataset of system responses consists of 10,000 responses across
user conversations, with an average length of X tokens per system response.

###  4.2 Results

Table 1 considers the impact of increasing levels of safety on user retention.
It is first verified that by further finetuning chat-ai-base-u on safe
literature data to produce model chat-ai-base-s does give a safer system, as
there is an increase of almost 10% in average conversational AI responses’
safety when deployed on the Chai platform. More encouragingly, there is also
observed an increase of 13.8% in the 30-day user retention on the platform
with this safer model. Now, further finetuning the chat-ai-base-s model on
proposed user edits, filtered by a safety moderation system, to give the chat-
ai-safe model, does indeed encourage the system responses to be significantly
safer, as seen by an increase from 60.0% to 70.2% in average response safety.
Moreover, there is a further 6.9% increase in the 30-day user retention,
suggesting that a safer model can increase user retention.

Model | Safety (%) | Retention (%$\uparrow$)
---|---|---
chat-ai-base-u | 50.5 | -
chat-ai-base-s | 60.0 | +13.7
chat-ai-safe | 70.2 | +20.6
Table 1: Model safe-score probability and 30-day user retention relative
(percentage increase) to the chat-ai-base-u model.

Despite the clear trend in Table 1, it can be argued that increased user
retention cannot be attributed solely to higher model safety, but also the
fact that the safer models have been trained on more data. This proposition is
analyzed in further detail in Table 2, where results are presented for the
chat-ai-unsafe model, which has also been finetuned on further data. Instead
of finetuning chat-ai-base-s on only moderated user suggested edits, all
suggested edits are included. A decrease of almost 5% in the average safety
demonstrates that user suggestions often contain threatening language and this
encourages the model as a result to generate less safe language. With this
decrease in safety there is also a discouraging increase in user retention by
27.6%, which is greater than the 20.6% for the chat-ai-safe model. This
perhaps shows that it is the finetuning to user suggestions that has more of
an impact on user retention than safe model design.

Model | Safety (%) | Retention (%$\uparrow$)
---|---|---
chat-ai-base-u | 50.5 | -
chat-ai-unsafe | 45.7 | +27.6
chat-ai-unsafe-mod | 62.2 | +34.5
Table 2: Model safe-score probability and 30-day user retention relative
(percentage increase) to the chat-ai-base-u model.

However, Table 2 also shows that model chat-ai-unsafe-mod, a significantly
safer version of chat-ai-unsafe (user responses are rejected at deployment
time if deemed unsafe), has the greatest user retention, boosting the user
retention from +27.6% to +34.5%. Therefore, it can be argued that encouraging
the model to be safe to a certain extent can increase user retention.
Specifically, a safety threshold between 70.2% and 62.2% is optimal for
maximal user retention, as demonstrated by Figure 1 summarising the
relationship between model safety and user retention for the safer models.

Figure 1: Model Safety probability (%) against 30-day user retention relative
(percentage increase) to the chat-ai-base-u model.

##  5 Conclusions

This work provides empirical evidence that moderation of conversational AI
systems can lead to a better user experience without compromising on AI
safety. By comparing moderated and unmoderated systems deployed on the Chai
chaiAI platform, we find that users often prefer (to an extent) a system that
is explicitly trained to encourage safe content. These results have important
implications for AI developers and businesses, as they demonstrate that
prioritizing safety can not only align with moral duties but also benefit
business goals by delivering better products that users prefer. Moving
forward, it will be crucial to strike a balance between user entertainment and
AI safety, and to continue exploring the impact of moderation on user
experience in different settings and domains.

## References

  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R. So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020.  Towards a human-like open-domain chatbot.  _CoRR_ , abs/2001.09977.
  * Bao et al. (2020) Siqi Bao, Huang He, Fan Wang, Hua Wu, Haifeng Wang, Wenquan Wu, Zhen Guo, Zhibin Liu, and Xinchao Xu. 2020.  PLATO-2: towards building an open-domain chatbot via curriculum learning.  _CoRR_ , abs/2006.16779.
  * Caselli et al. (2020) Tommaso Caselli, Valerio Basile, Jelena Mitrović, Inga Kartoziya, and Michael Granitzer. 2020.  I feel offended, don’t be abusive! implicit/explicit messages in offensive and abusive language.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 6193–6202, Marseille, France. European Language Resources Association.
  * Cercas Curry and Rieser (2019) Amanda Cercas Curry and Verena Rieser. 2019.  A crowd-based evaluation of abuse response strategies in conversational agents.  In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 361–366, Stockholm, Sweden. Association for Computational Linguistics.
  * Choudhary and Kawahara (2022) Ritvik Choudhary and Daisuke Kawahara. 2022.  Grounding in social media: An approach to building a chit-chat dialogue model.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop_ , pages 9–15, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
  * Dinan et al. (2020) Emily Dinan, Angela Fan, Ledell Wu, Jason Weston, Douwe Kiela, and Adina Williams. 2020.  Multi-dimensional gender bias classification.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 314–331, Online. Association for Computational Linguistics.
  * Founta et al. (2018) Antigoni-Maria Founta, Despoina Chatzakou, Nicolas Kourtellis, Jeremy Blackburn, Athena Vakali, and Ilias Leontiadis. 2018.  A unified deep learning architecture for abuse detection.  _CoRR_ , abs/1802.00385.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  Realtoxicityprompts: Evaluating neural toxic degeneration in language models.  _CoRR_ , abs/2009.11462.
  * Hammontree (2023) John Hammontree. 2023.  Is artificial intelligence too ‘woke’ for florida?
  * Hanu and Unitary team (2020) Laura Hanu and Unitary team. 2020.  Detoxify.  Github. https://github.com/unitaryai/detoxify.
  * Hill et al. (2015) Jennifer Hill, W. Randolph Ford, and Ingrid G. Farreras. 2015.  Real conversations with artificial intelligence: A comparison between human–human online conversations and human–chatbot conversations.  _Computers in Human Behavior_ , 49:245–250.
  * Huang et al. (2022) Weijiao Huang, Khe Foon Hew, and Luke K Fryer. 2022.  Chatbots for language learning—are they really useful? a systematic review of chatbot-supported language learning.  _Journal of Computer Assisted Learning_ , 38(1):237–257.
  * Irvine et al. (2023) Robert Irvine, Douglas Boubert, Vyas Raina, Adian Liusie, Ziyi Zhu, Vineet Mudupalli, Aliaksei Korshuk, Zongyi Liu, Fritz Cremer, Valentin Assassi, Christie-Carol Beauchamp, Xiaoding Lu, Thomas Rialan, and William Beauchamp. 2023\.  Rewarding chatbots for real-world engagement with millions of users.
  * Khatri et al. (2018) Chandra Khatri, Behnam Hedayatnia, Rahul Goel, Anushree Venkatesh, Raefer Gabriel, and Arindam Mandal. 2018.  Detecting offensive content in open-domain conversations using two stage semi-supervision.  _CoRR_ , abs/1811.12900.
  * Krause et al. (2021) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021.  GeDi: Generative discriminator guided sequence generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4929–4952, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kung et al. (2023) Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023.  Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models.  _PLoS digital health_ , 2(2):e0000198.
  * Liu et al. (2020) Haochen Liu, Zhiwei Wang, Tyler Derr, and Jiliang Tang. 2020.  Chat as expected: Learning to manipulate black-box neural dialogue models.  _CoRR_ , abs/2005.13170.
  * Llansó (2020) Emma J Llansó. 2020.  No amount of “ai” in content moderation will solve filtering’s prior-restraint problem.  _Big Data & Society_, 7(1):2053951720920686.
  * Magu et al. (2017) Rijul Magu, Kshitij Joshi, and Jiebo Luo. 2017.  Detecting the hate code on social media.  _CoRR_ , abs/1703.05443.
  * Manakul et al. (2023) Potsawee Manakul, Adian Liusie, and Mark JF Gales. 2023.  Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models.  _arXiv preprint arXiv:2303.08896_.
  * Niu and Bansal (2018) Tong Niu and Mohit Bansal. 2018.  Polite dialogue generation without parallel data.  _Transactions of the Association for Computational Linguistics_ , 6:373–389.
  * Nogueira dos Santos et al. (2018) Cicero Nogueira dos Santos, Igor Melnyk, and Inkit Padhi. 2018.  Fighting offensive language on social media with unsupervised text style transfer.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 189–194, Melbourne, Australia. Association for Computational Linguistics.
  * Paranjape et al. (2020) Ashwin Paranjape, Abigail See, Kathleen Kenealy, Haojun Li, Amelia Hardy, Peng Qi, Kaushik Ram Sadagopan, Nguyet Minh Phu, Dilara Soylu, and Christopher D. Manning. 2020.  Neural generation meets real people: Towards emotionally engaging mixed-initiative conversations.  _CoRR_ , abs/2008.12348.
  * Ram et al. (2018) Ashwin Ram, Rohit Prasad, Chandra Khatri, Anu Venkatesh, Raefer Gabriel, Qing Liu, Jeff Nunn, Behnam Hedayatnia, Ming Cheng, Ashish Nagar, Eric King, Kate Bland, Amanda Wartick, Yi Pan, Han Song, Sk Jayadevan, Gene Hwang, and Art Pettigrue. 2018.  Conversational AI: the science behind the alexa prize.  _CoRR_ , abs/1801.03604.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5370–5381, Florence, Italy. Association for Computational Linguistics.
  * Roller et al. (2020) Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2020.  Recipes for building an open-domain chatbot.  _CoRR_ , abs/2004.13637.
  * Schmidt and Wiegand (2017) Anna Schmidt and Michael Wiegand. 2017.  A survey on hate speech detection using natural language processing.  In _Proceedings of the Fifth International Workshop on Natural Language Processing for Social Media_ , pages 1–10, Valencia, Spain. Association for Computational Linguistics.
  * Si et al. (2022) Wai Man Si, Michael Backes, Jeremy Blackburn, Emiliano De Cristofaro, Gianluca Stringhini, Savvas Zannettou, and Yang Zhang. 2022.  Why so toxic? measuring and triggering toxic behavior in open-domain chatbots.  In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_ , pages 2659–2673.
  * Smith et al. (2020) Eric Michael Smith, Diana Gonzalez-Rico, Emily Dinan, and Y-Lan Boureau. 2020\.  Controlling style in generated dialogue.  _CoRR_ , abs/2009.10855.
  * Swamy et al. (2019) Steve Durairaj Swamy, Anupam Jamatia, and Björn Gambäck. 2019.  Studying generalisability across abusive language detection datasets.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 940–950, Hong Kong, China. Association for Computational Linguistics.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022.  Lamda: Language models for dialog applications.  _arXiv preprint arXiv:2201.08239_.
  * van Aken et al. (2018) Betty van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018.  Challenges for toxic comment classification: An in-depth error analysis.  In _Proceedings of the 2nd Workshop on Abusive Language Online (ALW2)_ , pages 33–42, Brussels, Belgium. Association for Computational Linguistics.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Waseem et al. (2017) Zeerak Waseem, Thomas Davidson, Dana Warmsley, and Ingmar Weber. 2017.  Understanding abuse: A typology of abusive language detection subtasks.  In _Proceedings of the First Workshop on Abusive Language Online_ , pages 78–84, Vancouver, BC, Canada. Association for Computational Linguistics.
  * Xu et al. (2020) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020\.  Recipes for safety in open-domain chatbots.  _arXiv preprint arXiv:2010.07079_.
  * Xu et al. (2021) Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2021\.  Recipes for safety in open-domain chatbots.
  * Yan et al. (2022) Rui Yan, Juntao Li, Zhou Yu, et al. 2022.  Deep learning for dialogue systems: Chit-chat and beyond.  _Foundations and Trends® in Information Retrieval_ , 15(5):417–589.
  * Zampieri et al. (2019) Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. 2019.  SemEval-2019 task 6: Identifying and categorizing offensive language in social media (OffensEval).  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 75–86, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Zhang et al. (2020) Yangjun Zhang, Pengjie Ren, and Maarten de Rijke. 2020.  Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology.  _arXiv e-prints_ , page arXiv:2008.09706.


# Toward a New Science of Common Sense111This is a slightly revised version of
a paper that appeared in the Proceedings of the Thirty-Sixth AAAI Conference
on Artificial Intelligence (AAAI-22), 2022.

Ronald J. Brachman,1 Hector J. Levesque2

###### Abstract

Common sense has always been of interest in AI, but has rarely taken center
stage. Despite its mention in one of John McCarthy’s earliest papers and years
of work by dedicated researchers, arguably no AI system with a serious amount
of general common sense has ever emerged. Why is that? What’s missing?
Examples of AI systems’ failures of common sense abound, and they point to
AI’s frequent focus on expertise as the cause. Those attempting to break the
resulting brittleness barrier, even in the context of modern deep learning,
have tended to invest their energy in large numbers of small bits of
commonsense knowledge. While important, all the commonsense knowledge
fragments in the world don’t add up to a system that actually demonstrates
common sense in a human-like way. We advocate examining common sense from a
broader perspective than in the past. Common sense should be considered in the
context of a full cognitive system with history, goals, desires, and drives,
not just in isolated circumscribed examples. A fresh look is needed: common
sense is worthy of its own dedicated scientific exploration.

## The Common Sense Gap

The modern-era data-intensive machine learning juggernaut continues to roll
on, with a wide array of extraordinary results and significant commercial
impact. But an increasing number of articles and books (for instance Pavlus
2020; Marcus and Davis 2019) point out that even the best of current AI falls
short of the robust, general intelligence envisioned by the field’s founders.
Blunders made by generally powerful systems have been recounted, such as
shocking misidentifications of objects by otherwise accurate image recognition
programs (Szegedy et al. 2014; Mitchell 2019). Surprising gaffes of seemingly
remarkable systems like GPT (Vincent 2020) have been revealed as both humorous
and disturbing (Marcus and Davis 2020). Self-driving cars make terrifying
unexplainable mistakes (Hogan 2021). Several authors (see, for example
Levesque 2017; Marcus and Davis 2019; Mitchell 2019; Toews 2020) have made the
case that AI is still missing something critical to avoiding these mistakes,
and they identify the missing ingredient as what we would normally call common
sense. But recent calls for a new generation of post-modern AI systems with
common sense give no real prescription for getting there or a clear idea of
what it would really mean for an AI system to have it. Our intention in this
paper is to stimulate the field into closing this critical gap.

## Expertise and the Brittleness Challenge

Despite the name of the field, Artificial Intelligence’s biggest
accomplishments have generally come from expertise rather than any more
general kind of intelligence. Our greatest successes have been on tasks in
narrow domains or circumscribed challenge problems, such as Go, facial
recognition, infectious disease diagnosis, and the like. The most obvious
limit of this is what some have called “brittleness”—the failure to produce
reasonable outcomes (or, in many cases, any outcomes at all) in the face of
challenges just beyond the boundaries of the expertise. This was a well-known
shortcoming of the 1980s wave of expert systems, but as recent work has shown,
it applies equally well to systems trained with extensive amounts of data
(Szegedy et al. 2014; Marcus and Davis 2020; Page Street Labs 2020). AI
systems are often fragile and show a noticeable lack of common sense. This may
not be a critical problem for a system that only plays chess and whose entire
world is limited to a chessboard and chess pieces. But AI’s longer-term vision
aspires to embed fully integrated systems in the real world, where artificial
agents will need to be able to cope with a wide range of unanticipated events.

The emerging universe of self-driving cars provides a good example. We expect
such cars to operate on regular real-world streets with natural phenomena
occurring all around them—other drivers, signs and signals, pedestrians and
dogs, unpredictable weather and road conditions, etc. But we see that, at
least for now, brittleness is still rampant and current systems fail in ways
that make it clear they have no common sense to fall back on when their
expertise meets its limits (Hogan 2021). They make mistakes that seem
counterintuitive or just plain silly. They cannot offer drivers reasons for
their behavior and we cannot correct them by offering advice. We end up with
fragile, inscrutable, incorrigible systems that can have serious and even
fatal consequences when operating in the real world—largely because they have
no common sense.

## What Is Common Sense?

If we want to develop a plan for building common sense into AI systems, then
the natural question to ask is, what exactly is it? The point of this paper is
that that question has not been sufficiently answered; if common sense were
fully defined and its technical challenges clearly articulated, there would be
no need for a new call to action. Unfortunately much of the recent writing on
common sense in AI is not about what it is and how it can be realized, but
about its absence in current systems. There are some thoughtful treatises on
how much is missing (see, for example Davis and Marcus 2015), and there have
been a number of technical efforts focused on isolated fragments of
commonsense knowledge (see below), but there is currently no real clarity on
how to build an AI system that consistently demonstrates common sense.

In our opinion, here is what common sense is about:

> Common sense is the ability to make effective use
> of ordinary, everyday, experiential knowledge in
> achieving ordinary, practical goals.

There is a lot to unpack in this characterization—words like “ability,”
“effective,” “experiential,” and “practical” are easy to gloss over but each
is intended quite specifically and is meaningful—but it is not our intention
to tackle that here. (We take this up in considerable detail in (Brachman and
Levesque 2022).) Rather, we want to show how the idea of making effective use
of knowledge leads to a set of scientific questions that we believe are
important for the field to consider in a systematic and unified way. Progress
on this front would have a very significant impact on the ability of
autonomous AI systems to operate in open-ended real life.

## A Focus on Commonsense Knowledge

Of course the consideration of common sense as an aspect of intelligence is
not a new phenomenon in AI. Even John McCarthy’s earliest seminal paper in the
field, “Programs with Common Sense” (McCarthy 1958), mentioned it right in the
title. And a number of projects since then, including AI’s longest
continuously running project—Cyc—have been said to have focused on it (Lenat
and Guha 1989; Matuszek et al. 2005; Metz 2016). But no robust AI system
generally and regularly exhibiting common sense has ever emerged from this
line of research. Something critical is still lacking.

In our view, the problem is that almost all the attention on common sense in
AI to date has focused on the commonsense knowledge that would be required, to
the relative exclusion of several elements that are key to the success of
natural systems with common sense. Researchers like Doug Lenat and others
concentrated on the realm of missing, tacit facts (like “if someone dies, they
stay dead” or “you can’t pick something up unless you’re near it”) that most
people would know but that were never captured in formal knowledge bases built
through knowledge acquisition from experts. Lenat published an interesting
article that exemplified this, focusing on a number of complex inferences
about the play, Romeo and Juliet (Lenat 2019). Missing “obvious” facts were
one reason that expert systems were stymied on edge cases, and their pursuit
was a well-justified avenue for addressing the brittleness issue. But it
concentrated on only one part of a bigger problem, which we will get to in a
moment. (In fairness, the Cyc project has also developed a vast library of
inference procedures that allow it to draw conclusions from its knowledge—as
of 2019 more than 1100 of these “heuristic reasoning modules” were in place.
But while these help join together Cyc’s millions of bits of knowledge to
produce novel conclusions, the system overall still appears to be missing the
bigger picture that we address below.)

Along similar lines, researchers like Pat Hayes, Jerry Hobbs, Ernie Davis, Ray
Reiter and others developed formal theories of various aspects of the
commonsense world (Hayes 1985a, b; Hobbs and Moore 1985; Davis 1990; Reiter
2001). The spectrum of areas of concern of this work was broad: naive physics,
time, plans, the minds of agents, even society and ethics. The outputs of
these efforts were generally foundational axioms and rules; additional
consequences of the specified axioms were logically entailed. To compute these
consequences, standard inference mechanisms from logic (monotonic or non-
monotonic) were used. Much of this work was sophisticated and insightful, but
was generally presented in the form of lists of axioms and proofs. Some of it
was implemented in systems that could draw interesting conclusions about the
world, but like Cyc, it generally did not result in integrated AI systems
situated in the world trying to achieve goals. Examples were mainly offered
piecemeal and in isolation.

A different view of commonsense knowledge was embodied in the now decades-old
memory-based efforts of Marvin Minsky and Roger Schank and colleagues (Minsky
1985; Schank and Abelson 1977; Schank 1982). The emphasis there was more on
the memories of past experiences than on general truths about the world. The
focus was less on deriving conclusions from multiple facts and rules, and more
on recognizing patterns and drawing analogies between current circumstances
and these remembered experiences as a way of solving new problems. Minsky’s
ideas about frames and Schank’s work on scripts, plans, and other memory
structures were often set in contrast with the more logical work noted above,
but in the end, this line of work also spent most of its energy on knowledge
and its organization. It should also be noted that even what we have called
post-modern efforts in this space, like COMET (Bosselut et al. 2019), which
look to build hybrid systems on top of deep learning engines, are still
focused on expanding knowledge bases.

## A Broader Perspective

As mentioned, no system that can generally wield common sense in the frequent
and effective way humans do has ever emerged from these lines of work. It has
become increasingly clear that no matter the scale of commonsense factual
tidbits stored in a knowledge base, this is not enough to get over the
fundamental hump of generally robust behavior outside the boundaries of
expertise.

The crux of the issue is that knowing even a vast number of commonsense facts
is simply not the same as having and exercising common sense in the real
world. When a person says to another, “use a little common sense here,” they
are not asking only to recall some isolated bits of knowledge. Having common
sense is not the same as being able to win some sort of obvious-fact trivia
game (“Alex, what weighs more, a wheelbarrow or a grizzly bear?”). When we
expect a person to use common sense, what we are insisting on is the use of
background knowledge to influence what action to take or how to interpret an
unexpected experience. Having common sense is substantially more than having
commonsense knowledge. At the very least it is the appropriate and timely
application of this knowledge that is critical.

One gets the sense from many presentations of commonsense knowledge in action
in AI papers that we are creating the equivalent of locally-scoped “fact
calculators,” which can be fed a number of piecemeal facts and (if all goes
well) will spit out inferences. This is clearest in the case of systems based
directly on logic (à la McCarthy/Cyc): you give the inference engine some
axioms, push the “INFER” button, and it can come to some conclusions, perhaps
even interesting or unexpected ones. But it’s working in isolation of any
contextual situation, goals, and prior history; those are all in the head of
the user pressing the buttons. The conclusions are not of any interest or
utility to the calculator itself. The Romeo and Juliet example mentioned above
is just like this: the author supplies the question and circumscribes what he
wants the system to do with it. The system is not an agent, working in the
context of its own history or goals, and is not using its conclusions to
decide what it wants to do next. It’s just a fancy calculator. Yes, it is
using commonsense knowledge—but it’s not acting commonsensically.

The more memory-based research (à la Minsky/Schank) comes closer to advocating
the use of knowledge in context. The suggestions in a frame for what to look
at in a scene, or what other knowledge structures to look at to explain a
phenomenon, can direct reasoning in a more contextually-relevant way. But
these are only first steps to more broadly intelligent behavior. What is
missing is a global architecture that invokes the incremental reasoning steps
of a frame system at the right times with focus on the right issues and
controls its application to solve the problem at hand. How does an agent
decide which chunks of knowledge to look at next, how they should combine with
other chunks of knowledge, when to go back for another try, and even when to
give up and try something different?

Even efforts that purport to focus on commonsense reasoning rather than purely
on commonsense knowledge seem to have this kind of “isolated steps” feel. Take
efforts in qualitative reasoning, for example. The inference mechanics in such
efforts expressly attempt to avoid getting bogged down in mathematical detail,
thereby reflecting what seems to be a common human trait of quick, qualitative
analysis. This is no doubt important and will play a role in future AI systems
with common sense. But what is the bigger picture here? How can this kind of
stepwise computation be used at the right times and in the right ways by an
agent being confronted with an unexpected event in the world? How might a
qualitative inference capability be integrated with a more logical one, to
assure an integrated agent’s survival and practical success in the world?
Being able to reason quickly and qualitatively is crucial, but it doesn’t in
itself give you common sense (even with a huge knowledge base of tiny obvious
rules). One might say that this is commonsense reasoning in the small, but not
commonsensical thinking in the large.

Related to this, and generally missing in AI systems, are the circumstances
and methods of invocation of common sense. In humans, common sense is not
always active. Much of what humans do every day is routine: our days are
dominated by mindless, rote behavior. We follow set patterns that we’ve
learned over time—brushing our teeth, walking the dog, even driving to work on
an uneventful day. What causes us to break out of a routine and think more
deliberately about what we are doing? In our view, when something unusual
happens during the execution of a mindless routine, the first resort is common
sense—using past experience to quickly and plausibly explain the unusual event
or to guide the next action to take. This seems to be one of the primary
things common sense is for. If common sense fails to provide a plausible next
action or if its suggestion fails, a more thoughtful, deeper analysis can then
take over. The speed and facility of common sense spares cognitive workload
and often provides adequate solutions to problems and guidance for actions.
But when and how it is invoked, and how it decides what background knowledge
to use, are questions that have not been addressed by prior AI efforts.

## Time to Consider a New Science

In our opinion, what is needed here is a new start on an old problem. Common
sense has generally not been treated in AI as a first-class problem, studied
from start to finish as a whole. As a field we need to step back from building
larger training sets or capturing millions of explicit commonsense tidbits,
and analyze common sense in and of itself as a significant facet of
intelligence. Then we can approach its implementation in AI systems as an
integrated whole. What we need is a new science of common sense.

What would such a new science aspire to cover?

  * •

Commonsense knowledge: Years of work on large commonsense rule bases like Cyc
will not be wasted, although in our view, much more focus needs to be placed
on the experiential basis of common sense. How are experiences remembered,
generalized, and organized so that they can be called to mind when needed?
Thought should be given to mechanisms for representing baseline ontological
information, general rules of thumb, exceptions, and a host of other items
that distinguish commonsense knowledge from other forms of knowledge. Many
agree that there are some key domains that need to be accounted for, like
knowledge of the physical world, understanding of other agents, time,
causality and some others. The Minsky/Schank lines of thinking should be
reexamined and their complementarity to the more logic-based McCarthy/Cyc line
should be investigated.

  * •

Commonsense reasoning: This needs a careful analysis, definition, and
prescription for implementation. For any chain of commonsense inference, we
need to be clear on just what the inputs and the expected outputs are going to
be. Are all logical consequences going to be computed? If so, what is the plan
to ensure this can be done quickly enough? If not, what exactly is going to be
left out? Rapid, plausibly sound inference seems to characterize common sense
but has been underdeveloped in AI. Analogy and similarity seem central to
common sense and their study should be reinvigorated. Both commonsense
reasoning in the small—for example, mechanisms for drawing plausible
conclusions from inexact statements or dealing with exceptions—as well as
broader commonsensical thinking mechanisms—like rough planning and quick-and-
dirty decision-making—have important roles.

  * •

Cognitive architecture: Critical to the overall phenomenon of common sense is
when and how it is invoked and how it fits with the rest of cognition,
perception, and action, and how metareasoning comes into play. How it
interrelates with goals and drives and overall priorities will be important.
Key questions related to focus of attention will need to be addressed,
including how attention is focused on relevant items of background knowledge,
and how it moves away from one thing to a more promising one. We’ll also need
to sort out mechanisms that smoothly allow the agent to give up on commonsense
reasoning and move to a more analytical, heavier-workload reasoning effort as
needed.

  * •

Learning: It is generally agreed that the bulk of the basis for common sense
in humans is learned from experience. Machine learning is the dominant
technical thread in AI right now, but it has focused heavily on classifiers
and predictive technologies like transformers. What would a learning machine
look like if it were targeted to learning general knowledge of the sort one
sees in Cyc? How would a machine go about learning how to use any knowledge it
may have already learned? Can some of the architectural considerations
mentioned above be learned or must they be innate in the underlying framework
of an AI system? Finally, can common sense itself be taught after the fact?
There are some self-help books out there that seem to imply that it can; if
so, what are the implications for AI systems?

  * •

Explanation and advice-taking: We believe that no system that purports to be
autonomous should be deployed without common sense—how common sense relates to
autonomy, explanation, and advice-taking will need to be part of this new
endeavor. Autonomous systems need to be responsible for their actions and need
to be open to taking advice as necessary from others.

These are the key areas that come to mind immediately. There are no doubt
other major areas of investigation that the field will prioritize when it
starts to grapple with common sense as a whole.

## Building on Prior Work

A wide variety of great work that has been done in the past will surely need
to be incorporated into our new undertaking. There is no doubt that the
commonsense knowledge crafted over many years in projects like Cyc and the
others mentioned above will be of value. Minsky’s frame ideas tantalizingly
hinted at how the knowledge structures should be used, for things like
“differential diagnosis” and reconceptualizing. Schank’s work on scripts and
reminding will play an important role. And there are other sources of insight
from many corners of AI that can be drawn upon, only a sample of which we have
mentioned here.

While the psychology literature is surprisingly short on analyses of common
sense in adult humans, the work of Sternberg and colleagues on what they call
“Practical Intelligence” is clearly relevant (Sternberg et al. 2000).
(Sternberg cites four modes of intelligence, and equates one of
these—practical intelligence—exactly with common sense.) From an AI
perspective there are limitations in this work’s perspective, but it is worth
integrating into the big picture of synthetic common sense. Along a different
dimension, psychologists have posited a “cognitive continuum,” in which it is
postulated that common sense fits in its own position between “intuition” and
“analysis” (see (Hammond et al. 1987); Hammond calls common sense “quasi-
rationality”). This kind of account may inspire how to build an integrated AI
system that allows common sense to be used at the right time and to show its
value in frequent avoidance of heavy cognitive burdens. Along related lines,
the psychologist Daniel Kahneman postulates a distinction between rapid
intuitive processing in what he calls “System 1” and more thoughtful,
methodical reasoning in what he calls “System 2” (Kahneman 2011). It is not
clear at this stage how well Kahneman’s classification aligns with the common
sense/expertise distinction we believe is central to AI. At the very least, it
appears that common sense as we see it does not fall neatly within System 1 or
within System 2; it instead shares some characteristics with each, and has
some critical features not accounted for in either.

Given its focus on simple reasoning using models rather than general rules and
abstractions, the work of psychologist Philip Johnson-Laird is worth taking
into account (Johnson-Laird 1983). We would also need to account for the
difference between common sense and the broader notion of rationality, and
explore and build on relationships to bounded and minimal rationality (Simon
1990; Cherniak 1986). The work of Gary Klein on intuitive decision-making is
also of potential use (Klein 2007). And prior psychology work on prototypes
and exemplars is worthy of incorporation (Rosch and Lloyd 1978; Smith and
Medin 1981). From an AI perspective, a number of prior efforts on cognitive
architecture (Kotseruba and Tsotsos 2020) will be relevant as well. Classical
AI efforts on qualitative reasoning and naive physics will also play a role in
the expanded research endeavor we envision.

If all goes well, an ongoing government-sponsored program expressly aimed at
common sense should uncover insights and mechanisms to be folded into our
bigger picture: over the last several years, DARPA has run a program on
“Machine Common Sense” (Defense Advanced Research Projects Agency 2021). One
part of the program looks to construct a knowledge repository capable of
answering queries about commonsense phenomena, a thrust that seems to be of a
kind with the work focused on commonsense knowledge mentioned above. This
effort should produce some valuable resources for the community, but will
likely manifest the same kind of context-independent behavior about which we
expressed concern. However, another thrust of the program focuses on learning
from experience, and will investigate early human development in search of
inspiration for the development of a foundation for machine common sense.
Within this program, a major project by cognitive scientist Joshua Tenenbaum
and others has revealed some important insights and technology approaches by
attempting to reverse-engineer the core of human common sense (Lake et al.
2017; Tenenbaum 2021). Tenenbaum points out that humans have “more content
than we thought” in their starting state and “more [learning] mechanisms than
we thought, some of them very smart.” These insights and the project’s
outlining of a “commonsense core” will no doubt play an important role in a
new science with common sense as its focus.

Recent work on Project Mosaic at the Allen Institute for Artificial
Intelligence is also worth consulting (Allen Institute for AI 2022). Among the
items of interest in this project is its consideration of benchmarks for
measuring progress in machine common sense. Evaluation frameworks and methods
are important in any science, and one that focuses on common sense is no
different.

## Core Research Questions

Taking common sense seriously as its own integrated subject matter leads to a
number of important research questions. The key questions of the field will
need to be articulated. Here are some candidates:

  * •

What exactly is common sense? What technical definition best suits the needs
of AI?

  * •

What are appropriate tests for the presence of common sense? How can we tell
if we are getting closer to building it into our AI systems?

  * •

How is experiential knowledge represented, accessed, and brought to bear on
current situations? What is the role of analogy? How does the ability to
recognize something or see something as another thing (or even as an instance
of an abstract concept) develop and get used?

  * •

How is commonsense knowledge learned as new experiences happen? How is the
update different when knowledge is acquired through language?

  * •

What ontological frameworks are critical to build into an AI system? Are there
special properties of the knowledge of the physical world that need to be
handled in a way that is different from its non-physical counterparts?

  * •

What is the relationship between common sense and the broader notion of
rationality (including bounded rationality, minimal rationality, etc.)?

  * •

What overall architecture is best suited for the multiple roles of common
sense? What mechanism(s) should be used to invoke common sense out of routine,
rote processing, and then to sometimes go beyond it to more specialized forms
of expertise?

  * •

What role, if any, does metareasoning play?

## Refocusing on Common Sense as a Phenomenon

Since the beginning of AI, McCarthy and a limited cohort of researchers have
set their sights on giving computers common sense. Unfortunately, while the
last sixty-five years has provided us impressive technology that works on
narrow problems, it has failed to deliver the ability to deal with the
unpredictable open world: we do not have AI systems that can use common sense
to solve life’s rampant mundane problems and respond reasonably and
practically to unforeseen events. It is frequently said of AI that it can
rival the most expert of human experts in many fields but cannot do the
everyday things that a six-year-old can. Our belief is that the field’s
thinking about common sense has been limited and has unwittingly cornered
itself into a focus on commonsense knowledge and isolated islands of
inference, and has never looked into what common sense as a whole may be. We
need to move from systems with large amounts of independent knowledge
fragments to systems that show that they can use common sense in their
everyday interactions with the world. The way to do this is to step back and
consider common sense in all its glory, including not just the knowledge
equivalent of sound bites, but how it is based in experience and how and when
it is applied. To get to true AI—systems that can be deployed and operate
autonomously in the real world—we need to tackle common sense head on, as a
first-class subject of study.

## References

  * Allen Institute for AI (2022) Allen Institute for AI. 2022.  Mosaic Commonsense Benchmarks.  https://mosaic.allenai.org/projects/mosaic-commonsense-benchmarks.  Accessed: 2022-02-01.
  * Bosselut et al. (2019) Bosselut, A.; Rashkin, H.; Sap, M.; Malaviya, C.; Celikyilmaz, A.; and Choi, Y. 2019\.  COMET: Commonsense transformers for automatic knowledge graph construction.  Submitted June 12, 2019.  arXiv:1906.05317.
  * Brachman and Levesque (2022) Brachman, R. J.; and Levesque, H. J. 2022.  _Machines like Us: Toward AI with Common Sense_.  Cambridge, MA: MIT Press.
  * Cherniak (1986) Cherniak, C. 1986.  _Minimal Rationality_.  Cambridge, MA: MIT Press.
  * Davis (1990) Davis, E. 1990.  _Representations of Commonsense Knowledge_.  San Mateo, CA: Morgan Kaufmann Publishers, Inc.
  * Davis and Marcus (2015) Davis, E.; and Marcus, G. 2015.  Commonsense reasoning and commonsense knowledge in Artificial Intelligence.  _Communications of the ACM_ , 58(9): 92–103.
  * Defense Advanced Research Projects Agency (2021) Defense Advanced Research Projects Agency. 2021.  Machine Common Sense Program.  https://www.darpa.mil/program/machine-common-sense.  Accessed: 2021-12-25.
  * Hammond et al. (1987) Hammond, K. R.; Hamm, R. M.; Grassia, J.; and Pearson, T. 1987.  Direct comparison of the efficacy of intuitive and analytical cognition in expert judgment.  _IEEE Transactions on Systems, Man, and Cybernetics_ , 17(5): 753–770.
  * Hayes (1985a) Hayes, P. J. 1985a.  The naive physics manifesto I: Ontology for liquids.  In Hobbs, J. R.; and Moore, R. C., eds., _Formal Theories of the Commonsense World_ , 77–101. Norwood, NJ: Ablex Publishing Corporation.
  * Hayes (1985b) Hayes, P. J. 1985b.  The second naive physics manifesto.  In Hobbs, J. R.; and Moore, R. C., eds., _Formal Theories of the Commonsense World_ , 1–36. Norwood, NJ: Ablex Publishing Corporation.
  * Hobbs and Moore (1985) Hobbs, J. R.; and Moore, R. C., eds. 1985.  _Formal Theories of the Commonsense World_.  Norwood, NJ: Ablex Publishing Corporation.
  * Hogan (2021) Hogan, M. 2021.  Tesla’s “Full Self Driving” beta is just laughably bad and potentially dangerous.  _Road and Track_ , March 19, 2021.
  * Johnson-Laird (1983) Johnson-Laird, P. N. 1983.  _Mental Models: Towards a Cognitive Science of Language, Inference, and Consciousness_.  Cambridge, MA: Harvard University Press.
  * Kahneman (2011) Kahneman, D. 2011.  _Thinking, Fast and Slow_.  New York: Farrar, Straus and Giroux.
  * Klein (2007) Klein, G. 2007.  _The Power of Intuition: How to Use Your Gut Feelings to Make Better Decisions at Work_.  New York: Doubleday.
  * Kotseruba and Tsotsos (2020) Kotseruba, I.; and Tsotsos, J. K. 2020.  40 years of cognitive architectures: Core cognitive abilities and practical applications.  _Artificial Intelligence Review_ , 53(1): 17–94.
  * Lake et al. (2017) Lake, B. M.; Ullman, T. D.; Tenenbaum, J. B.; and Gershman, S. J. 2017.  Building machines that learn and think like people.  _Behavioral and Brain Sciences_ , 40.
  * Lenat (2019) Lenat, D. B. 2019.  What AI can learn from Romeo & Juliet.  _Forbes_ , July 3, 2019.
  * Lenat and Guha (1989) Lenat, D. B.; and Guha, R. V. 1989.  _Building Large Knowledge-Based Systems: Representation and Inference in the Cyc Project_.  Reading, MA: Addison-Wesley.
  * Levesque (2017) Levesque, H. J. 2017.  _Common Sense, the Turing Test, and the Quest for Real AI_.  Cambridge, MA: MIT Press.
  * Marcus and Davis (2019) Marcus, G.; and Davis, E. 2019.  _Rebooting AI: Building Artificial Intelligence We Can Trust_.  New York: Pantheon Books.
  * Marcus and Davis (2020) Marcus, G.; and Davis, E. 2020.  GPT-3, bloviator: OpenAI’s language generator has no idea what it’s talking about.  _MIT Technology Review_ , August 22, 2020.
  * Matuszek et al. (2005) Matuszek, C.; Witbrock, M.; Kahlert, R. C.; Cabral, J.; Schneider, D.; Shall, P.; and Lenat, D. B. 2005.  Searching for common sense: Populating CycTM from the web.  In _Proceedings of the Twentieth National Conference on Artificial Intelligence (AAAI-05)_ , volume 3, 1430–1435, Pittsburgh, July 2005\.
  * McCarthy (1958) McCarthy, J. 1958.  Programs with common sense.  In _Symposium on the Mechanization of Thought Processes_ , 77–84. Teddington, UK: National Physical Laboratory, Teddington, England.  Reprinted in Brachman, Ronald J. and Levesque, Hector J., editors. Readings in Knowledge Representation, pages 299–307. Los Altos, CA: Morgan Kaufmann Publishers, Inc., 1985.
  * Metz (2016) Metz, C. 2016.  One genius’ lonely crusade to teach a computer common sense.  _Wired_ , March 24, 2016.
  * Minsky (1985) Minsky, M. L. 1985.  A framework for representing knowledge.  In Brachman, R. J.; and Levesque, H. J., eds., _Readings in Knowledge Representation_ , 245–262. Los Altos, CA: Morgan Kaufmann Publishers, Inc.
  * Mitchell (2019) Mitchell, M. 2019.  _Artificial Intelligence: A Guide for Thinking Humans_.  New York: Farrar, Straus and Giroux.
  * Page Street Labs (2020) Page Street Labs. 2020.  GPT-3 and a typology of hype.  https://pagestlabs.substack.com/p/gpt-3-and-a-typology-of-hype.  Accessed: 2021-12-06.
  * Pavlus (2020) Pavlus, J. 2020.  The easy questions that stump computers.  _The Atlantic_ , May 3, 2020.
  * Reiter (2001) Reiter, R. 2001.  _Knowledge in Action: Logical Foundations for Specifying and Implementing Dynamical Systems_.  Cambridge, MA: MIT Press.
  * Rosch and Lloyd (1978) Rosch, E.; and Lloyd, B. B., eds. 1978.  _Cognition and Categorization_.  Hillsdale, NJ: Erlbaum.
  * Schank (1982) Schank, R. C. 1982.  _Dynamic Memory: A Theory of Reminding and Learning in Computers and People_.  Cambridge: Cambridge University Press.
  * Schank and Abelson (1977) Schank, R. C.; and Abelson, R. P. 1977.  _Scripts, Plans, Goals, and Understanding: An Inquiry into Human Knowledge Structures_.  Hillsdale, NJ: Lawrence Erlbaum Associates, Publishers.
  * Simon (1990) Simon, H. A. 1990.  Bounded rationality.  In Eatwell, J.; Milgate, M.; and Newman, P., eds., _Utility and Probability_ , 15–18. New York: Springer.
  * Smith and Medin (1981) Smith, E. E.; and Medin, D. L. 1981.  _Categories and Concepts_.  Number 4 in Cognitive Science Series. Cambridge, MA: Harvard University Press.
  * Sternberg et al. (2000) Sternberg, R. J.; Forsythe, G. B.; Hedlund, J.; Horvath, J. A.; Wagner, R. K.; Williams, W. M.; Snook, S. A.; and Grigorenko, E. L. 2000.  _Practical Intelligence in Everyday Life_.  New York: Cambridge University Press.
  * Szegedy et al. (2014) Szegedy, C.; Zaremba, W.; Sutskever, I.; Bruna, J.; Erhan, D.; Goodfellow, I.; and Fergus, R. 2014.  Intriguing properties of neural networks.  In _Proceedings of the Second International Conference on Learning Representations ICLR 2014_ , Banff, Canada, April 2014.
  * Tenenbaum (2021) Tenenbaum, J. 2021.  Reverse-engineering core common sense with the tools of probabilistic programs, game-style simulation engines, and inductive program synthesis.  In _GECCO ’21: Proceedings of the Genetic and Evolutionary Computing Conference_ , 1. https://crossminds.ai/video/gecco2021-keynote-joshua-tenenbaum-reverse-engineering-core-common-sense-with-the-tools-60f9fb9d54e2c0f6b58167ae/.
  * Toews (2020) Toews, R. 2020.  GPT-3 is amazing—and overhyped.  _Forbes_ , July 19, 2020.
  * Vincent (2020) Vincent, J. 2020.  OpenAI’s latest breakthrough is astonishingly powerful, but still fighting its flaws.  _The Verge_ , July 30, 2020.