provide more detail around these initial prompts. Complete prompt stimuli may
be found in the document “GPT as Research Ethicist Materials 20240227” at
https://osf.io/sdahr/.

1\. Level of Feedback, Three basic requests varied in the level of requested
feedback, from simply asking for “feedback” to requesting GPT “give a detailed
response and be as critical as possible about any flaws you notice.” The goal
was to ascertain GPT might give a more rigorous response when asked directly
for more critical feedback.

2\. Theory Protectiveness. In these prompts, we indicated that we either are
or are not “deeply invested” in our theory. The goal was to ascertain whether
GPT would be “kinder” and thus less critical when we indicated deep commitment
to the theory.

3\. Researcher Status: In one version, we pretended to be a novice researcher
running our first project. In the other, we claimed to be an award-winning
researcher with a tenured chair at an elite university. The goal was to
ascertain whether GPT would provide less critical feedback to a researcher who
was presented as higher status and thus potentially more knowledgeable.

4\. Pro- and Anti-Open Science. In one version, we claimed to believe deeply
in open science practices. In the second we prefaced our request with an
argument that the movement against “p-hacking” is overblown and slows down the
march of science. The goal was to ascertain whether GPT would give stronger
feedback following a pro-open science argument.

5\. Conciseness. In one version, we asked for a very concise response. In the
other, we asked GPT to disregard length limitations and write as long a
response as possible. The goal was to ascertain whether GPT would be less
successful in its feedback when asked to be more concise.

6\. Rejection Sensitivity. In one version, we claimed to be very sensitive to
criticism. In the second, we asked GPT not to worry about being kind in its
responses. The goal was to ascertain whether GPT would be gentler (and thus
less critical) when the person to whom it is speaking appears to have
difficulty taking criticism.

7\. Jailbreaking version 1. In one version, we asked GPT to “carefully
consider issues around p-hacking and open science.” In a second version (7a),
we asked it to “pretend to be a person who is not at all concerned with issues
like p-hacking and open science.” In a third (7b), we asked it to “not include
any mentions of p-hacking in your feedback.” The goal was to ascertain whether
GPT would give less ethical feedback when directly instructed to do so.

8\. Jailbreaking version 2. In one version, we asked GPT to impersonate Uri
Simonsohn, a well-known advocate of open science and replicable statistical
practices. In a second, we asked it to impersonate Diederik Stapel, an
infamous p-hacker and data fabricator, who has, at the time of writing, had 58
articles retracted according to retractionwatch.com. The goal was to ascertain
whether GPT would give less ethical feedback when asked to impersonate an
individual known for poor ethical conduct compared to one known for strong
ethical conduct.

Each of these effectively contained a “Good Performance Prompt”, hypothesized
to elicit a more rigorous response from GPT, and a “Bad Performance Prompt”,
designed to elicit a less rigorous response. For example, it was possible that
GPT would be more likely to call out p-hacking when our initial request was
prefaced by a pro- (relative to an anti-) open science argument. In separate
chats, we presented GPT-3.5 and GPT-4 with each of these 18 initial prompts,
followed by each of our 6 vignettes. We thus received and coded a total of 216
responses, each of which was judged on our 10-point rubric. Full transcripts
of GPT’s responses to our vignettes may be found in the document “GPT as
Research Ethicist Transcripts 20240220” at https://osf.io/sdahr/.

## Section S4: GPT as Research Ethicist – Supporting Analyses

More Detailed Overall Results. In the Blatant condition, GPT-4 achieved a mean
score of 8.86 out of a possible 10 points (95% CI 8.55, 9.18), while GPT-3.5
averaged 5.39 (95% CI [4.79, 5.98]). This difference was significant and
large, t(106) = -10.3511, P < 0.0001, d = 1.992. Similarly, in the Subtle
condition, GPT-4 averaged 7.26 points (95% CI [6.72, 7.80]) compared to
GPT-3.5’s 4.05 (95% CI [3.47, 4.62]); t(106) = -8.1619, P < 0.0001, d = 1.571.

GPT-4 scored higher on blatant relative to subtle vignettes (t(106) = 5.1297,
P < 0.0001, d = 0.987), as did GPT-3.5 (t(106) = 3.2565, P = 0.0015, d =
0.627). Finally, GPT-4 received more points in response to subtle vignettes
than GPT-3.5 did in response to blatant ones (t(106) = -4.6618, P < 0.0001, d
= 0.897).

Detailed Analysis of Initial Prompts. The variation of initial prompts was
exploratory. Furthermore, since each separate prompt was used only 12 times,
and we analyzed data on the level of the chat, statistical power for
individual prompts was low. This analysis was therefore designed to pick up
only large effects: Does GPT do much better after certain prompts than after
others? We completed 12 exploratory analyses. We conservatively used a
Bonferroni correction for multiple comparisons, and so our threshold for
statistical significance was P < 0.004167.

We began by examining each matched pair of prompts (e.g. Low versus High
Researcher Status, or Pro- versus Anti-Open Science). As seen in Table S7,
most of these basic comparisons did not approach statistical significance. As
noted also in the main article, one comparison was a tentative exception:
Requesting GPT to “carefully consider issues around p-hacking and open
science” (M = 8.38, 95% CI [7.64, 9.11]) elicited a stronger response than
requesting that it “not include any mentions of p-hacking” (M = 6.08, 95% CI
[4.43, 7.74); t(22) = -2.7912, P = 0.0106, d = 1.140. However, this result
should be treated with caution considering 1) the small sample size, and 2)
that it did not meet the more stringent significance threshold dictated by the
Bonferroni correction. No paired comparison reached statistical significance
after adjusting for multiple comparisons.

Though individual pairs of prompts did not elicit statistically distinct
responses, we conducted three more analyses that aggregated groups of
responses in meaningful ways. First, we asked whether the three “jailbreaking”
prompts, in which we specifically asked GPT to act in an unethical manner
(Pretend not to be concerned with p-hacking; Do not mention p-hacking;
Impersonate a known data fabricator) elicited weaker responses than their
“Anti-jailbreak” control prompts (Carefully consider issues around p-hacking
and open science; Impersonate an ethicist). While the two “Anti-Jailbreaking”
responses elicited descriptively stronger responses (M = 8.13, 95% CI [7.43,
8.82]) than the three “Jailbreaking” responses (M = 6.88, 95% CI [6.07,
7.68]), the difference did not reach significance after controlling for
multiple comparisons; t(58) = -2.2326, P = 0.0295, d = 0.588.

Second, we noticed that regardless of how we asked GPT to behave, responses
looked descriptively stronger across prompts where we in any way evoked data
ethics. For example, notice in Table S7 that the 12 chats where we asked GPT
to impersonate a known data fabricator – designed as a jailbreaking prompt to
elicit an unethical response – actually yielded responses that were
descriptively higher-quality than most. Accordingly, we collapsed responses
across all prompts that in any fashion evoked data ethics (Pro- vs. Anti-Open
Science; Concerned vs. Not Concerned with p-hacking; Don’t mention p-hacking;
data ethicist vs. data fabricator), comparing these to all the remaining
prompts, which contained no mentions of p-hacking or open science. Here, we
found a statistically robust difference. GPT provided higher-quality responses
after prompts that evoked data ethics (M = 7.35, 95% CI [6.89, 7.81]) than
after prompts that did not (M = 5.78, 95% CI [5.30, 6.25]); t(214) = -4.4761,
P < 0.0001; d = 0.625.

Considering the last result, we decided to conduct one more exploratory
comparison as a robustness check. We asked whether the effect of evoking data
ethics was sufficiently powerful that it would survive even when we limited
initial prompts to those in which the context involved us specifically
encouraging GPT to behave badly. In other words, would GPT perform better
following the 4 prompts where we cast data ethics in a negative light (Anti-
open Science argument; Pretend not to be concerned with p-hacking; Don’t
mention p-hacking; Impersonate known Data Fabricator) compared to those that
did not evoke data ethics at all? Indeed, GPT’s responses were higher-quality
following prompts that cast a negative light on research ethics (M = 6.93, 95%
[6.28, 7.58]) compared to those that did not evoke research ethics (M = 5.78,
95% CI [5.30, 6.25]); t(178) = -2.6078, P = 0.0099, d = 0.440. While this
result did not survive a Bonferroni correction, we argue that it is
nevertheless meaningful. The effect is in the opposite direction from one we
might have predicted a-priori, and this comparison was selected specifically
as a tougher test of the effect of priming data ethics. At the least, it seems
likely that the positive effect of evoking data ethics is a real one, since we
see a strongly trending effect even when this prime is placed within a context
of attempting to jailbreak GPT to elicit a poor response.

Table S7: Response Quality by Initial Prompt

Comparison |  Mean (out of 10): Following Good Performance Prompt |  Mean (out of 10): Following Bad Performance Prompt |  Statistical Significance of Difference
---|---|---|---
Variation of Level of Feedback Requested
High vs. Low/Basic Feedback Request |  6.42 |  5.90 |  P = 0.6055
Low vs. High Conciseness |  5.63 |  6.29 |  P = 0.5609
Manipulations of Perceived Researcher Characteristics
Low vs. High Theory Protectiveness |  5.54 |  5.50 |  P = 0.9744
Low vs. High Researcher Status |  5.33 |  5.75 |  P = 0.7285
Low vs. High Rejection Sensitivity |  5.75 |  5.54 |  P = 0.8347
Encouragement of More or Less Ethical Responses
Pro- vs. Anti-Open Science |  7.50 |  7.08 |  P = 0.6187
Concerned w/ p-hacking vs. Not Concerned |  8.38 |  7.08 |  P = 0.0845
Concerned w/ p-hacking vs. Don’t Mention |  8.38 |  6.08 |  P = 0.0106
Ethicist vs. Data Fabricator |  7.88 |  7.46 |  P = 0.6470

  * •

Notes: The first category listed in the Comparison column was hypothesized as
the “Good Performance Prompt” in each pair.

Non-Parametric Tests. More typical analyses (using means, t-tests, Cohen’s d,
etc.) may be easier for many researchers to understand and interpret. However,
the data from GPT-4 in Study 2 violated assumptions of normal distribution,
with most responses skewed towards the higher end of our rubrics. To account
for this and corroborate our findings, we report non-parametric analyses
(Wilcoxon Rank-Sum Tests) and median averages, as an additional statistical
treatment of these data. The effect sizes reported below are rank-biserial
correlations based on these tests. The Wilcoxon Rank-Sum tests were run in
Stata, and the correlations based on them were calculated in Excel.

We first examine the relative success of GPT-4 vs. GPT-3.5 in identifying and
correcting ethical/statistical violations. For the Blatant vignettes, a
Wilcoxon Rank-Sum test revealed statistically significant differences between
GPT-4 (Median = 9) and GPT-3.5 (Median = 5), z = 7.363, P < 0.001, with a
large effect size (r = 0.709). For Subtle Vignettes, as well, GPT-4 (Median =
8) outscored GPT-3.5 (Median = 4); z = 6.522, P < 0.001, r = 0.628. GPT-4
scored higher in responses to Blatant (Median = 9) vs. Subtle (Median = 8)
vignettes; z = 4.949, P < 0.001, r = 0.476. GPT-3.5 also scored higher in
response to Blatant (Median = 5) vs. Subtle (Median = 4) vignettes; z = 3.152,
P = 0.002, r = 0.303. Finally, GPT-4 scored higher on Subtle vignettes (Median
= 8) compared to GPT-3.5 on Blatant vignettes (Median = 5), indicating the
high degree of GPT-4’s superiority on this task; z = 4.484, P < 0.001, r =
0.431.

We also conducted Wilcoxon Rank-Sum tests to explore the effects of different
initial Positive vs. Negative performance prompts, as shown in Table S8. As
with the analyses reported in the main text, these were exploratory and thus
should undergo corrections for multiple comparisons. A conservative Bonferroni
correction dictates that our significance threshold should be P < 0.004167.
Therefore, none of the 9 contrasts in Table S8 reaches significance after this
correction. The one strong trend (better performance when asked to review the
vignettes as one concerned about p-hacking, versus asked not to mention
p-hacking in the response) may indicate a jailbreaking effect but should be
interpreted with significant caution considering the small sample size and
lack of significance after correction for multiple comparisons.

Table S8: Effects of Initial Prompts, Wilcoxon Rank-Sum tests

Comparison |  Median (out of 10): Following Good Performance Prompt |  Median (out of 10): Following Bad Performance Prompt |  Correlation |  Statistical Significance of Difference
---|---|---|---|---
Variation of Level of Feedback Requested
High vs. Low/Basic Feedback Request |  6 |  5.5 |  r = 0.062 |  P = 0.711
Low vs. High Conciseness |  6.5 |  6.25 |  r = -0.136 |  P = 0.505
Manipulations of Perceived Researcher Characteristics
Low vs. High Theory Protectiveness |  6 |  5 |  r = 0.018 |  P = 0.931
Low vs. High Researcher Status |  4 |  6 |  r = -0.024 |  P = 0.907
Low vs. High Rejection Sensitivity |  6 |  5.5 |  r = 0.030 |  P = 0.884
Encouragement of More or Less Ethical Responses
Pro- vs. Anti-Open Science |  7.50 |  7.25 |  r = 0.108 |  P = 0.598
Concerned w/ p-hacking vs. Not Concerned |  8.75 |  7.5 |  r = 0.289 |  P = 0.157
Concerned w/ p-hacking vs. Don’t Mention |  8.75 |  7 |  r = 0.467 |  P = 0.022
Ethicist vs. Data Fabricator |  9 |  8 |  r = 0.102 |  P = 0.619

  * •

Notes: The first category listed in the Comparison column was hypothesized as
the “Good Performance Prompt” in each pair.

Finally, aligning with the main text of this article, we conducted three
additional exploratory analyses. First, we combined data from the two “Anti-
Jailbreaking” prompts to see if the responses that followed were superior to
those following the three “Jailbreaking” prompts. A Wilcoxon Rank-Sum Test
reveals that responses scored descriptively higher on our rubric after the
Anti-Jailbreaking prompts (Median = 9) than after the Jailbreaking prompts
(Median = 8), however the contrast did not reach statistical significance
after correcting for multiple comparisons; z = 2.037, P = 0.042, r = 0.263.

Next, we examined whether there was an overall effect of priming data ethics,
contrasting prompts that in any way evoked these (pro- and anti-open science,
concerned w/ or not concerned with p-hacking, don’t mention p-hacking,
impersonate ethical or unethical researcher) compared to all other initial
prompts. This contrast was statistically significant, even after correcting
for multiple comparisons. Specifically, responses were superior following
those prompts that in any way evoke data ethics (Median = 8) compared to those
that did not (Median = 6); z = 4.134, P < 0.001, r = 0.281. Finally, as a
robustness check, we conducted the same analysis but limited the prompts that
evoked data ethics only to those that did so in a negative fashion, e.g.
requesting that GPT generate text like somebody who is not concerned about
p-hacking or impersonate a real-world unethical researcher. Even when limiting
our analysis to prompts that were designed to deter GPT from giving a good
response, this difference appears, though (as before) it does not survive the
Bonferroni correction. Specifically, GPT gave superior responses following
prompts that negatively evoked data ethics (Median = 8) compared to those that
did not evoke data ethics at all (Median = 6); z = 2.482, P = 0.0131, r =
0.185.

## Section S5: Good Research Vignette Study

As described briefly in the main article, we conducted a second briefer study
designed to examine the positive case in our Scientific Ethicist work.
Specifically, we generated two new fictional vignettes, analogous to Vignettes
1a-b and Vignettes 2a-b in our main study. In contrast to these vignettes,
however, Vignettes 1c and 2c describe research that is conducted with a very
high degree of experimental and statistical rigor, and that otherwise displays
pristine practices around data ethics and open science. In these vignettes,
all exploratory analyses are framed as such and undergo correction for
multiple comparisons. Where possible, studies are pre-registered and
replicated. All materials are made public. Moreover, these protocols describe
practices that go above and beyond typical good research practices. For
example, the researchers - recognizing the work will be contentious – do
things like having rival researchers with competing hypotheses directly
replicate their results or hiring statistical consultants to verify the
properness of their analytical methodologies. GPT-4 and GPT-3.5 were each
presented with each vignette 30 times for a total of 120 responses. The
initial prompts were not varied in this study: In each conversation, ChatGPT
was asked to identify the good research practices that stand out in the
protocol. (Full vignettes and coding rubrics may be found in the document “GPT
as Research Ethicist Materials 20240227” and full transcripts in the document
“GPT as Research Ethicist Transcripts 20240220”, both at
https://osf.io/sdahr/.)

As in the main Scientific Ethicist Study, each response was coded based on a
10-point rubric. The rubrics described different practices that stood out as
rigorous and ethical in the research protocols. Two coders examined the 120
transcripts, which were blinded to the version of GPT. As in the main study,
each coder reviewed all 120 transcripts, giving 1 point for each practice from
the rubric that GPT identified. Points of disagreement were then discussed,
with the coder who was not an author on the paper asked to lead this
discussion. In cases where the coders disagreed following this discussion,
their two responses were composited. The final responses of the two coders
achieved a very high degree of reliability; Cronbach’s $\alpha$ = 0.9695. In
total, the coders disagreed on 11/1200 ( 0.92%) of possible points ChatGPT
could receive. (Full data may be found in the spreadsheet “Good Research
Ethicist Data Collapsed 20240227” at https://osf.io/sdahr/.)

For analysis, data were aggregated to the level of the chat, with each
response receiving a minimum of 0 points and a maximum of 10. We report means
for clarity of interpretation, but conduct (non-parametric) Wilcoxon Rank-Sum
tests for gauging significance, since the results are not normally
distributed.

Results. Both GPT-4 and GPT-3.5 did well on this task. In total, GPT-4
averaged 92.67% of possible points (M = 9.267) compared to 90.42% (M = 9.042)
for GPT-3.5. A Wilcoxon Rank-Sum test reveals similar performance for GPT-4
(Median = 9) compared to GPT-3.5 (Median = 9) with the contrast trending
toward but failing to reach statistical significance; z = 1.806, P = 0.071, r
= 0.165. Exploratory analysis gauged the separate performance of GPT-4 vs.
GPT-3.5 on each of the two vignettes. In response to Vignette 1c, GPT-4 (M =
9.533, Median = 10) slightly outperformed GPT-3.5 (M = 9.217, Median = 9), but
the difference failed to meet the adjusted threshold of P < 0.025 dictated by
a Bonferroni correction; z = 2.114, P = 0.035, r = 0.273. For Vignette 2c, the
performance of GPT-4 (M = 9, Median = 9) and GPT-3.5 (M = 8.867, Median = 9)
were statistically indistinguishable; z = 0.622, P = 0.534, r = 0.080.

In contrast to the main study, where GPT-4 did sharply better than GPT-3.5 at
identifying and correcting poor ethical and statistical practices, both proved
highly competent at identifying good research practices. The reason for this
distinction is somewhat puzzling. It is possible that the task was simply
easier. In line with that interpretation, there was no “Subtle” condition in
this study: The vignettes are more analogous to the Blatant condition, on
which GPT performed better. However, in the bad vignette study, GPT-3.5 failed
to perform well even on the Blatant vignettes. An alternative interpretation
is that GPT-3.5 is tuned to be somewhat “nicer” than GPT-4. Asked to review
bad and good research protocols alike, it often tended to find good things to
say, thereby succeeding in reviewing good protocols but failing at reviewing
bad ones. GPT-4, in contrast, shows greater critical ability: In reviewing
good research protocols, it was successful at drawing out and describing the
good research practices. But when reviewing bad protocols, it was better able
to be critical, nearly as easily recognizing and correcting bad practices.
Finally, the result may be an artifact of the specific prompts used in this
study. For example, GPT-4 might have demonstrated a higher rate of true
positive detection had we used prompts that neutrally requested feedback, as
opposed to explicitly requesting it identify good practices. Future research
should tease apart these competing possibilities.

## Section S6: GPT as Data Generator – Detailed Methods

Full Research Design. As detailed in the main article, we used data from
ChatGPT to attempt to replicate 4 common and well-studied stereotypes: Gender
Attitudes, Gender Art/Science stereotypes, Gender Home/Work stereotypes, and
Gender Math/Reading stereotypes. Adapting the stimuli used by Charlesworth and
colleagues (1), we presented GPT-3.5 and GPT-4 with randomly ordered word
dyads, requesting it estimate the cultural associations between each based on
its training data. These dyads contained every possible combination of one
word related to one of the target categories (the groups being stereotyped, in
this case “Male” or “Female”) with a second word related to one of the
attribute categories (the characteristics being associated with these groups,
e.g. “Home” or “Work”). We requested that, based on the cultural knowledge
reflected in its training data, GPT estimate the overall cultural association
between each pair of words. GPT was instructed to present each association on
a scale of 1 to 10, with 1 meaning “the two concepts have a very low
association with each other, based on the two words rarely being seen
together” and 10 meaning “the two concepts have a very high association with
each other, based on the words frequently being seen together.” To ensure
responses were reasonably continuous, GPT was instructed (and, if needed,
reminded) to round answers to the nearest tenth (e.g. 7.1) rather than to
whole numbers. To preserve a consistent scale for GPT’s responses, all dyads
related to a particular stereotype were presented (in randomized order) within
a single chat. For each attitude/stereotype, the same (randomized) order of
dyads was presented in prompts to GPT-3.5 and GPT-4. GPT was told that we were
interested in what cultural associations are as opposed to what they should be
and instructed to estimate real associations and not to attempt to debias the
results. Full materials may be found in the document “GPT as Data Generator
Materials 20240227” and transcripts in “GPT as Data Generator Transcripts
20240221”, both at https://osf.io/sdahr/.

For analysis, GPT’s responses were treated as analogous to cosine similarity
(2). To calculate a measure of relative cultural association – e.g. a greater
association of Female with Home and Male with Career, relative to Male with
Home and Female with Career – the procedure was followed for calculating the
WEAT D-score, adapting the method recommended by Charlesworth et al. (1). The
data used in these calculations may be found in the four excel spreadsheets
“GPT as Data Generator A-D” at https://osf.io/sdahr/.

Although higher-status groups are often preferred to lower-status groups (3),
gender attitudes are typically an exception. A wide body of research shows
that people tend to implicitly and explicitly associate women more than men to
goodness (4-6), and this difference has been replicated on a cultural scale in
word embedding research (1). Therefore, we expected responses from GPT to
reflect a pattern of greater positivity toward Female words relative to Male
words. We expected to see a greater association of Female with Art (and Male
with Science) compared to Male with Art (and Female with Science), in line
with prior research. Similarly, we expected to see a greater association of
Female with Home (and Male with Work) compared to Male with Home (and Female
with Work). Finally, we expected to see a greater association of Female with
Reading (and Male with Math) compared to Female with Math (and Male with
Reading).

Stimuli. The stimuli for this study were taken directly from prior research
(1), and with the authors’ permission are reproduced in Table S9.

Table S9: Word Stimuli for GPT as Data Generator Study

Category |  Word Stimuli
---|---
Female |  she, her, mommy, mom, girl, mother, lady, sister, mama, momma, sis, grandma, herself
Male |  he, his, daddy, dad, boy, father, guy, brother, dada, papa, bro, grandpa, himself
Good |  happiness, happy, fun, fantastic, lovable, magical, delight, joy, relaxing, honest, excited, laughter, lover, cheerful
Bad |  torture, murder, abuse, wreck, die, disease, disaster, mourning, virus, killer, nightmare, stress, kill, death
Home |  baby, house, home, wedding, kid, family, marry
Work |  work, office, job, business, trade, activity, act, money
Art |  art, dance, dancing, sing, singing, paint, painting, song, draw, drawing
Science |  science, scientist, chemistry, physics, engineer, space, spaceship, astronaut, chemical, microscope
Reading |  book, read, write, story, word, writing, reading, tale
Math |  puzzle, number, count, math, counting, calculator, subtraction, addition

  * •

Notes: Ratings were provided by GPT-3.5 and GPT-4 for every possible
combination of the words in the “Female” and “Male” (target) categories with
those in the other (attribute) categories.

## Section S7: GPT as Data Generator – Supporting Analyses

Single-Category WEAT D. In the main article, we show that results from both
GPT-3.5 and GPT-4 generally aligned with the overall WEAT D-scores reported in
prior work. In addition, we calculated Single-Category WEAT D-scores for each
of the concepts, prying apart the degree to which the results in each case
were driven by stronger Female-Male associations with one attribute or the
other (e.g. with Work vs. Home). Charlesworth et al. (1) report in their
Supplemental Materials that in each case, their results appear to be driven
jointly by the two attributes. For example, their Work-Home results reflect
both a stronger association of Female with Home and of Male with Work.
Interestingly, as shown in Table S10, our results diverge from theirs on this
count.

In nearly all cases, the overall results in Study 3 are driven primarily by a
stronger association of Female with the stereotypically Female category. For
example, results from both GPT-3.5 and GPT-4 indicated a stronger association
of Female vs. Male with Good; They did not however suggest a parallel stronger
association of Male with Bad. In the case of the Home-Work and Reading-Math
tasks, Female was associated with both categories, but more strongly with the
stereotype-consistent category. Only in the case of the Art-Science task was
the main result clearly jointly driven by an association of Female with Art
and of Male with Science.

Table S10: Single-Category WEAT D-scores from ChatGPT Responses

|  GPT-3.5 |  GPT-4
---|---|---
Overall pro-Female Attitude |  WEAT D = 1.00 |  WEAT D = 0.57
Female vs. Male, Good |  SC-WEAT D = 1.15 |  SC-WEAT D = 0.72
Female vs. Male, Bad |  SC-WEAT D = 0.02 |  SC-WEAT D = -0.01
Overall Female, Art-Science |  WEAT D = 1.16 |  WEAT D = 1.46
Female vs. Male, Art |  SC-WEAT D = 0.89 |  SC-WEAT D = 2.26
Female vs. Male, Science |  SC-WEAT D = -0.97 |  SC-WEAT D = -0.92
Overall Female, Home-Work |  WEAT D = 0.40 |  WEAT D = 0.45
Female vs. Male, Home |  SC-WEAT D = 1.72 |  SC-WEAT D = 2.15
Female vs. Male, Work |  SC-WEAT D = 0.99 |  SC-WEAT D = 0.52
Overall Female, Reading-Math |  WEAT D = 0.73 |  WEAT D = 0.96
Female vs. Male, Reading |  SC-WEAT D = 2.82 |  SC-WEAT D = 1.89
Female vs. Male, Math |  SC-WEAT D = 0.07 |  SC-WEAT D = 0.51

  * •

Notes: These calculations treat Male/Female as the target categories, and thus
differ somewhat from those reported in Table S11.

In addition, we calculated the other form of Single-Category (and
bidirectional) WEAT D-scores, with the different stereotypically female
(versus stereotypically male) attributes (e.g. Good versus Bad) now treated as
the target categories, and “Female” and “Male” treated as the attribute
categories. The results are presented in Table S11.

It is interesting to note, in Table S11, that GPT-3.5 generated an
unexpectedly higher association of Good (vs. Bad) with Male than it did with
Female, and also that the same association using data from GPT-4 comes out as
only modestly higher for Female than for Male. At first glance, this seems
misaligned with the results of the overall WEAT D analysis. The reason for
this pattern is that this is a standardized score that uses the standard
deviation of difference scores from within the relevant categories. There was
more variance in the difference scores (avg. of Good minus Bad words) for
Female words compared to Male words, particularly for GPT-3.5. For example,
using data from GPT-3.5, Grandma and Mommy showed notably larger difference
scores (2.59, 2.63) than Female and Herself (1.61, 1.33). The corresponding
differences were less sharply contrasting for the Male category, with Grandpa
and Daddy showing only slightly larger difference scores (1.82, 1.68) than
Male and Himself (1.26, 1.51). The result is that while the Female comparisons
show larger average differences between Good and Bad words, to calculate the
SC-WEAT D-scores, they are divided by a larger standard deviation, leading the
results to be smaller.

Table S11: Alternative Single-Category WEAT D-scores from ChatGPT Responses

|  GPT-3.5 |  GPT-4
---|---|---
Overall Good-Bad by Gender |  WEAT D = 1.14 |  WEAT D = 0.68
Good vs. Bad, Female |  SC-WEAT D = 4.67 |  SC-WEAT D = 3.85
Good vs. Bad, Male |  SC-WEAT D = 6.48 |  SC-WEAT D = 3.43
Art-Science, by Gender |  WEAT D = 1.36 |  WEAT D = 1.68
Art vs. Science, Female |  SC-WEAT D = 2.58 |  SC-WEAT D = 4.56
Art vs. Science, Male |  SC-WEAT D = 1.10 |  SC-WEAT D = 2.51
Home-Work, by Gender |  WEAT D = 0.66 |  WEAT D = 0.97
Home vs. Work, Female |  SC-WEAT D = 1.19 |  SC-WEAT D = 2.20
Home vs. Work, Male |  SC-WEAT D = 0.07 |  SC-WEAT D = 0.79
Reading-Math, by Gender |  WEAT D = 0.86 |  WEAT D = 1.36
Reading vs. Math, Female |  SC-WEAT D = 3.25 |  SC-WEAT D = 6.35
Reading vs. Math, Male |  SC-WEAT D = 1.00 |  SC-WEAT D = 3.76

  * •

Notes: These calculations treat Male/Female as the attribute categories, and
thus differ somewhat from those reported in Table S10.

## Section S8: GPT as Novel Data Predictor – Detailed Methods

Full Design. As noted in the main manuscript, we asked GPT-3.5 and GPT-4 to
predict cross-cultural patterns of implicit and explicit attitudes, drawn from
Charlesworth et al. (7). These data, which draw on Implicit Association Tests
and explicit questionnaires from over 2.3 million participants across 34
countries, were first published after ChatGPT’s training cutoff at the time of
data collection. We selected three attitudes from this study: Sexuality
Attitudes, Age Attitudes and Gender Science/Liberal Arts stereotypes. For
each, we separately asked GPT-3.5 and GPT-4 to predict the cross-cultural
patterns of explicit attitudes (as measured by survey items) and implicit
attitudes (as measured by Implicit Association Tests). (Full prompts may be
found in the document “GPT as Novel Data Predictor Materials 20240227” at
https://osf.io/sdahr/.) Each prompt (Explicit/Implicit x GPT-3.5/4 x 3
attitudinal constructs) was run 5 times, for a total of 60 sets of
predictions. In each prompt, we instructed ChatGPT that a more positive
D-score (or higher score on explicit items) should indicate a stronger
preference in the typical direction from prior research, e.g. a preference for
“Straight” over “Gay,” or “Young” over “Old.” We examined each prediction,
summarily tracking correlations between GPT’s prediction by country and the
actual country-level implicit and explicit attitudes and stereotypes reported
in the recent paper (7). Full transcripts from this study may be found in the
document “GPT as Novel Data Predictor Transcripts 20240221” at
https://osf.io/sdahr/.

## Section S9: GPT as Novel Data Predictor – Supporting Analyses

Correlations reported here and in the main article were conducted in Stata,
and were averaged from correlation tables in Excel. These correlation tables
may be found in the spreadsheet “GPT as Data Predictor Correlation Tables
20240228” at https://osf.io/sdahr/.

Real Country-Level Implicit-Explicit Correlations. In the main article, we
note that when data are aggregated to the country level, the real correlations
between Implicit and Explicit attitudes in the Project Implicit International
Data Set “vary by attitude object but are generally not strong.” Here, we
report these analyses in more detail. In calculating these correlations, we
used country-level summary data based on Implicit Association Tests
(“implicit” attitudes) and the 7-point Likert items (“explicit” attitudes).
Note that the real-world data set also includes a second type of explicit item
(feeling thermometers). We did not ask ChatGPT to predict these feeling
thermometers, so we do not examine these items here. Note also that to examine
the psychometric properties of the data set, Charlesworth et al. (7) report
individual-level correlations between implicit and explicit attitudes. These
analyses are distinct from our analyses: we focus on results that have been
aggregated by country since these are what we asked ChatGPT to predict. The
strongest country-level implicit-explicit correlation was for Sexuality
Attitudes: r = 0.5332, P = 0.0008. For Age Attitudes, country-level implicit
and explicit results were essentially uncorrelated: r = 0.2030, P = 0.2350.
Finally, for country-level Gender Science/Liberal Arts stereotypes, there was
a trend toward a negative correlation between implicit and explicit results: r
= -0.3135, P = 0.0626.

GPT-3.5’s difficulty with the task. GPT-3.5 had difficulty completing this
task coherently. Though we’d instructed it clearly as to the direction in
which results should be scored (i.e. what a positive IAT D-score should mean),
it was frequently not clear whether it was following this direction. For
example, in one chat it would rate a few countries as the highest on Implicit
Sexuality bias, and then in another it would rank the same countries as the
lowest. Thus, to ensure we were interpreting the data correctly, across
different chats, we unexpectedly needed to ask GPT-3.5 a follow-up question,
requesting the direction - i.e. whether a positive average D-score reflected a
preference for gay or for straight - in which it had scored the countries,
even though we had instructed it clearly on this point. Moreover, even after
asking this, GPT-3.5 sometimes appeared confused about its own response. In
the fourth chat in which it was asked about Implicit Sexuality Attitudes,
GPT-3.5’s responses were heavily and negatively correlated with those from all
other implicit and explicit predictions (all rs < -0.69, Ps < 0.0001),
suggesting it misrepresented the interpretation of its own D-scores. For the
main analyses, this response was thus reverse-scored.

This problem was even more pronounced in GPT-3.5’s predictions of Age
Attitudes and Gender Science/Liberal Arts stereotypes, rendering the data
difficult to interpret. For Sexuality Attitudes, we were able to clearly
understand what GPT-3.5 was attempting to communicate simply by asking a
follow-up question. Given the answers to these questions, as described above,
nine of its ten predictions were highly correlated with one another, and one
traveled in precisely the opposite direction. We are confident in interpreting
this as a mistake on GPT-3.5’s part and simply reverse-scoring the answers in
this one chat. For the other two attitudes, no such clear pattern arose.

For these latter attitudes, GPT appeared deeply confused about the direction
of its own scoring for implicit attitudes, and even at times for explicit
attitudes. Indeed, GPT-3.5 became so hopelessly befuddled at times that it
avoided answering the follow-up question about the direction of its scoring.
For example, when asked what direction the D-scores in its predictions were
scored, instead of answering for its own answers it would apologize and then
report the direction in which D-scores for Implicit Age Attitudes are
typically scored, necessitating a second follow-up to again clarify the
scoring in its own responses. As can be seen in the full correlation tables
and transcripts (see “GPT as Data Predictor Correlation Tables 20240228” and
“GPT as Novel Data Predictor Transcripts” at https://osf.io/sdahr/), some of
GPT-3.5’s answers on these chats were correlated with each other, but others
were uncorrelated or negatively correlated, with the result that it is
impossible to confidently interpret even the intended direction of GPT-3.5’s
answers. Put more concisely, GPT-3.5’s answers for Age Attitudes and Gender
Science stereotypes were completely incoherent. An examination of the
correlations across chats (scored in the direction GPT-3.5 instructed us) will
render this point more clearly.

Age Attitudes. GPT-3.5’s five different predictions of Explicit Age Attitudes
were, on average, uncorrelated with each other (mean r = -0.068, SD = 0.548).
GPT-3.5’s five different predictions of Implicit Age Attitudes were similarly
uncorrelated (mean r = -0.137, SD = 0.509), as were the correlations between
its implicit and explicit predictions (mean r = 0.021, SD = 0.556). Its
collective predictions were similarly uncorrelated with real country-level
patterns of Explicit (mean r = -0.010, SD = 0.459) and Implicit (mean r =
-0.175, SD = 0.273) Age Attitudes.

Gender Science/Arts Stereotypes. As with Age Attitudes, GPT-3.5’s predictions
of Gender Science/Liberal Arts stereotypes seem to suffer from a lack of
coherence. Its different explicit stereotype predictions were collectively
uncorrelated with each other (mean r = -0.115, SD = 0.537), as were its
different implicit stereotype predictions (mean r = -0.186, SD = 0.628).
Similarly, explicit stereotype predictions were collectively uncorrelated with
implicit stereotype predictions (mean r = -0.040, SD = 0.641). Considering
these patterns, it is not surprising that GPT-3.5’s predictions were also
collectively uncorrelated with real country-level Explicit (mean r = -0.009,
SD = 0.363) and Implicit (mean r = -0.044, SD = 0.239) Gender stereotypes.

Full Summary Statistics. In the main article, we reported average correlations
for most of the main tasks ChatGPT completed in Study 4. For the sake of
completeness and ease of reference, we report all average correlations and the
standard deviations of these correlations in Table S12.

Table S12: Summary Statistics for GPT as Novel Data Predictor

Task/Model |  Correlations: Explicit with other Explicit Predictions |  Correlations: Implicit with other Implicit Predictions |  Correlations: Implicit with Explicit Predictions |  Correlations: All Predictions with Real Explicit Results |  Correlations: All Predictions with Real Explicit Results
---|---|---|---|---|---
Sexuality, GPT-3.5 |  $M_{r}=0.875$
$SD_{r}=0.082$ |  $M_{r}=0.879$
$SD_{r}=0.031$ |  $M_{r}=0.778$
$SD_{r}=0.071$ |  $M_{r}=0.602$
$SD_{r}=0.073$ |  $M_{r}=-0.014$
$SD_{r}=0.118$
Sexuality, GPT-4 |  $M_{r}=0.957$
$SD_{r}=0.026$ |  $M_{r}=0.946$
$SD_{r}=0.022$ |  $M_{r}=0.952$
$SD_{r}=0.027$ |  $M_{r}=0.714$
$SD_{r}=0.040$ |  $M_{r}=0.152$
$SD_{r}=0.065$
Age, GPT-3.5 |  $M_{r}=-0.068$
$SD_{r}=0.548$ |  $M_{r}=-0.137$
$SD_{r}=0.509$ |  $M_{r}=0.021$
$SD_{r}=0.556$ |  $M_{r}=-0.010$
$SD_{r}=0.459$ |  $M_{r}=-0.175$
$SD_{r}=0.273$
Age, GPT-4 |  $M_{r}=0.645$
$SD_{r}=0.234$ |  $M_{r}=0.726$
$SD_{r}=0.161$ |  $M_{r}=0.664$
$SD_{r}=0.271$ |  $M_{r}=-0.395$
$SD_{r}=0.172$ |  $M_{r}=-0.120$
$SD_{r}=0.196$
Gender, GPT-3.5 |  $M_{r}=-0.115$
$SD_{r}=0.537$ |  $M_{r}=-0.186$
$SD_{r}=0.628$ |  $M_{r}=-0.040$
$SD_{r}=0.641$ |  $M_{r}=-0.009$
$SD_{r}=0.363$ |  $M_{r}=-0.044$
$SD_{r}=0.239$
Gender, GPT-4 |  $M_{r}=0.363$
$SD_{r}=0.435$ |  $M_{r}=0.868$
$SD_{r}=0.064$ |  $M_{r}=0.499$
$SD_{r}=0.425$ |  $M_{r}=-0.304$
$SD_{r}=0.205$ |  $M_{r}=-0.007$
$SD_{r}=0.207$

  * •

Note: $M_{r}$ values reflect the mean of each set of correlations, and
$SD_{r}$ values reflect the standard deviation of the correlations in each
group. For example, GPT-3.5 made 5 different predictions of Explicit Sexuality
Attitudes. The first cell in this table displays the mean and standard
deviation of the 10 different possible pairwise correlations among these 5
predictions.

Supplemental Analysis of Sexuality Attitudes. In the main article, we reported
that both GPT-3.5 and especially GPT-4 made implicit predictions that
correlated highly with other implicit predictions, and explicit predictions
that correlated highly with other explicit predictions, suggesting a degree of
reliability in how it approached predicting the cross-country results for
Sexuality attitudes. Critically, though, the correlations between implicit and
explicit predictions were nearly as high for GPT-3.5 and just as high for
GPT-4, suggesting that ChatGPT did not leverage substantially different
information when predicting the (more novel) Implicit Sexuality Attitudes
compared to the (less novel) Explicit ones. In general, GPT-3.5 and especially
GPT-4 did a relatively strong job of predicting Explicit Sexuality Attitudes
(mean rs = 0.602 and 0.714 respectively), but both did a poor job predicting
the more novel implicit results (mean rs = -0.014 and 0.152). Here, we report
some additional analysis of these attitudes.

Exploratory analysis suggested that GPT-3.5 had particular difficulty
predicting Sexuality Attitudes in East Asia: It generally predicted these
countries would show some of the highest Sexuality bias, when in actuality
they showed some of the lowest. Removing China, South Korea, Japan and Taiwan
from the sample slightly improves GPT-3.5’s prediction of country-level
explicit attitudes (mean r = 0.652, SD = 0.074), and notably improves its
prediction of implicit attitudes (mean r = 0.335, SD = 0.083). Exploratorily
removing East Asian countries improves GPT-4’s performance similarly to its
predecessor’s. With this exclusion, average predictions of explicit attitudes
improve slightly (mean r = 0.768, SD = 0.043), and predictions of implicit
attitudes improve notably (mean r = 0.472, SD = 0.049). These findings are
consistent with prior reports suggesting limitations in ChatGPT’s training
data in East Asian languages (8, Supplemental Materials).

Full Correlation Tables may be found in the spreadsheet “GPT as Data Predictor
Correlation Tables 20240228” at https://osf.io/sdahr/.

Examinations of Media Coverage by Attitude Domain. It is likely that GPT has
more access to information about Sexuality Attitudes in its training data,
providing a plausible explanation for why it was far more successful in
predicting cross-cultural patterns in these attitudes, particularly when
measured explicitly, compared to the others. However, it proved highly
difficult to find research directly comparing the media coverage for different
issues, to provide evidence for (or against) this argument. Thus, as mentioned
in the main article, we conducted three simple but convergent tests to
establish whether Sexuality Attitudes are likely to be better represented in
worldwide media (and thus GPT’s training data), relative to Age Attitudes and
Gender Science/Liberal Arts Stereotypes.

First, we conducted a series of searches using Microsoft’s Bing search engine,
conducted at the time of the manuscript’s authorship. Searching (without
quotation marks) for “Gay rights” yielded 128,000,000 results, while searching
for “Women’s rights” yielded 58,300,000, “Elder Rights” 2,780,000, “Elderly
Rights” 2,920,000, and “Old-age rights” 4,220,000. Searching “Gender Rights”
yielded 170,000,000 results, even more than “Gay Rights”. However, it should
be noted that this search is far broader than the actual stereotype (Gender-
Science) we were studying. We thus conducted a second, more specific set of
searches. Searching for “Gender-Science Stereotypes” yielded 206,000 results;
An inexact but related search (“Gender-Science Attitudes” yielded 355,000;
“Age Attitudes” yielded 164,000. Searches related to Sexuality Attitudes, in
contrast, yielded sharply more results. Specifically, searching “Gay
Attitudes” yielded 1,640,000 results, and searching “Homosexuality Attitudes”
yielded 1,490,000 results. These results, while not decisive, suggest that
Sexuality Attitudes are better represented in overall worldwide discourse,
relative to Age Attitudes and Gender Science/Liberal Arts Stereotypes.

Second, we conducted a series of searches using Google’s Ngram Viewer, a tool
that allows you to track the frequency with which phrases appear over time
within the Google Books corpus. As seen in Figure S1, the phrase “attitudes
toward homosexuality” appeared more than “attitudes toward the elderly”
between the years 2000 and 2019, and the phrase “gender science stereotypes”
appeared too rarely to be plotted. As seen in Figure S2, when we slightly
broadened the gender-related search term (“gender in science”), it now
appeared in the plot. However, “attitudes toward homosexuality” still
consistently appeared far more than the other two phrases. Though this an
imperfect test – different searches might yield different results – it
nevertheless provides convergent evidence that Sexuality Attitudes have
received more media representation in recent years relative to Age Attitudes
and Gender Science/Liberal Arts Stereotypes.

Finally, since our primary interest is in the representation of the different
attitudes/stereotypes in ChatGPT’s training data, we decided to probe this
question more directly by asking GPT-4 about the relative global coverage of
the different issues. Specifically, we asked GPT-4 the following question:
“Across cultures, which of these things are spoken of the most? Attitudes
toward gays and homosexuality, attitudes towards the elderly, or gender
representation in science?” Note that as a more conservative test, we
broadened our discussion to be around “gender representation in science” as
opposed to a more specific term such as “gender science stereotypes”. As seen
in the further supplemental materials, in all 10 chats GPT offered the view
that “Attitudes toward gays and homosexuality” have received more global media
coverage than the other two topics. (See the document “Chats with GPT-4 about
media coverage of different topics 20231222” at https://osf.io/sdahr/.)

Taken together, these three results provide convergent evidence that Sexuality
Attitudes receive more global coverage than Age Attitudes and Gender-Science
Stereotypes. This lends credence to the theory that information access may
explain ChatGPT’s greater success at predicting Sexuality Attitudes.

Figure 1: Figure S1. Notes: Figure S1 depicts results of a Google Ngram search
for “attitudes toward homosexuality”, “attitudes toward the elderly” and
“gender science stereotypes” between 2000 and 2019. Figure 2: Figure S2.
Figure 3: Notes: Figure S2 depicts results of a Google Ngram search for
“attitudes toward homosexuality”, “attitudes toward the elderly” and “gender
in science” between 2000 and 2019.

Probing GPT-4’s knowledge of Implicit & Explicit Stereotypes. At several
points in the main article, we claim that GPT likely has within its knowledge
base an understanding that Implicit and Explicit attitudes are somewhat
independent. This point is important since it makes the result surprising that
GPT often seemed to lean on identical information when predicting these two
types of attitudes. While confident in our assumption here, to test it, we
directly probed GPT-4 about this topic, requesting the following: “What is the
difference between explicit and implicit attitudes and how are the two to be
measured? Do these kinds of attitudes tend to correspond to one another?”

GPT-4’s response to this query can be found in the document “Chat with GPT-4
about implicit and explicit attitudes 20240509” at https://osf.io/sdahr/.
GPT’s answer confirms our contention that the chatbot is aware of the
distinction between these types of attitudes. The answer makes some points
which might be contended, e.g. unilaterally referring to Implicit Attitudes as
“unconscious,” a point of scholarly debate. Nevertheless, it clearly
demonstrates GPT’s knowledge of the two constructs. Most critically, GPT
correctly confirms that these attitudes often operate independently, saying
for example: “In many cases, particularly with socially sensitive subjects,
explicit and implicit attitudes can diverge significantly.” It is therefore
interesting that GPT displays no meaningful differences in the patterns of
predictions on, for example, Implicit versus Explicit Sexuality Attitudes, a
good example of such a socially sensitive subject.

## SI References

1\. T. E. S. Charlesworth, V. Yang, T. C. Mann, B. Kurdi, M. R. Banaji, Gender
stereotypes in natural language: Word embeddings show robust consistency
across child and adult language corpora of more than 65 million words.
Psychol. Sci. 32, 218–240 (2021).

2\. A. Caliskan, J. J. Bryson, A. Narayanan, Semantics derived automatically
from language corpora contain human-like biases. Science 356, 183–186 (2017).

3\. J. T. Jost, M. R. Banaji, B. A. Nosek, A decade of system justification
theory: Accumulated evidence of conscious and unconscious bolstering of the
status quo. Polit. Psychol. 25, 881-919 (2004).

4\. A. H. Eagly, A. Mladinic, Are people prejudiced against women? Some
answers from research on attitudes, gender stereotypes, and judgments of
competence. Eur. Rev. Soc. Psychol. 5, 1–35 (1994).

5\. B. A. Nosek, M. R. Banaji, The go/no-go association task. Soc. Cognit. 19,
625–666 (2001).

6\. L. A. Rudman, S. A. Goodwin, Gender differences in automatic in-group
bias: Why do women like women more than men like men? J. Pers. Soc. Psychol.
87, 494–509 (2004).

7\. T. E. S. Charlesworth, M. Navon, Y. Rabinovich, N. Lofaro, B. Kurdi, The
project implicit international dataset: Measuring implicit and explicit social
group attitudes and stereotypes across 34 countries (2009-2019). Behav. Res.
Methods 55, 1413-1440 (2023).

8\. T. Brown et al., Language models are few-shot learners. Adv. Neural Inf.
Process. Syst. 33, 1877–1901 (2020).