# ChatGPT as Research Scientist: Probing GPT’s Capabilities as a Research
Librarian, Research Ethicist, Data Generator and Data Predictor

black Steven A. Lehr
Cangrade, Inc.
Watertown, MA
&black Aylin Caliskan
Information School
University of Washington
Seattle, WA
&black Suneragiri Liyanage
Department of Psychology
Harvard University
Cambridge, MA
&black Mahzarin R. Banaji
Department of Psychology
Harvard University
Cambridge, MA
Correspondence about this article may be addressed to Steve Lehr at
<EMAIL_ADDRESS>
###### Abstract

How good a research scientist is ChatGPT? We systematically probed the
capabilities of GPT-3.5 and GPT-4 across four central components of the
scientific process: as a Research Librarian, Research Ethicist, Data
Generator, and Novel Data Predictor, using psychological science as a testing
field. In Study 1 (Research Librarian), unlike human researchers, GPT-3.5 and
GPT-4 hallucinated, authoritatively generating fictional references 36.0% and
5.4% of the time, respectively, although GPT-4 exhibited an evolving capacity
to acknowledge its fictions. In Study 2 (Research Ethicist), GPT-4 (though not
GPT-3.5) proved capable of detecting violations like p-hacking in fictional
research protocols, correcting 88.6% of blatantly presented issues, and 72.6%
of subtly presented issues. In Study 3 (Data Generator), both models
consistently replicated patterns of cultural bias previously discovered in
large language corpora, indicating that ChatGPT can simulate known results, an
antecedent to usefulness for both data generation and skills like hypothesis
generation. Contrastingly, in Study 4 (Novel Data Predictor), neither model
was successful at predicting new results absent in their training data, and
neither appeared to leverage substantially new information when predicting
more versus less novel outcomes. Together, these results suggest that GPT is a
flawed but rapidly improving librarian, a decent research ethicist already,
capable of data generation in simple domains with known characteristics but
poor at predicting novel patterns of empirical data to aid future
experimentation.

## Significance

> Though scientists widely adopt them, the promise of general-purpose
> artificial intelligence systems to facilitate science has been largely
> untested. In four studies, we examine the capabilities of ChatGPT across
> several tasks intrinsic to the scientific process. ChatGPT is a poor (but
> improving) curator of scientific articles. It is a surprisingly good
> research ethicist, detecting violations of statistical best practices and
> evolving open science protocols. Its ability to simulate known results may
> herald useful abilities in data generation and theory building. However, the
> chatbot had little success predicting highly novel data, highlighting its
> limited ability to surmise things outside its training data. Beyond merely
> testing LLMs, these studies produce several novel insights into the nature
> of machine intelligence.

_Keywords_ Generative AI $\cdot$ Large Language Models $\cdot$ Scientific
Methods $\cdot$ Cognitive Science

Scientists and writers of science fiction have long embraced the promise of
artificial superintelligence, with fictional computers showing astonishing
capabilities in scientific domains. The idea is compelling, for the
acceleration of science could be – at least in theory – one of the most
powerful gifts of this technology. It is unquestionably hopeful to imagine a
world in which artificial intelligence (AI) can help cure diseases, solve
impending global warming, accelerate space travel, and wipe out age-old
inequalities. However, the power of AI – which has lain mostly dormant over
the last 50 years – is at an inflection point. The rise of new deep learning
architectures like the Transformer (1) has yielded models capable of an array
of impressive tasks, from seamless human-like conversation to composing
symphonies. Already, scientists are implementing targeted AI systems to
enhance scientific discovery across various disciplines (2). As just a few
examples, machine learning has been used to discover new protein structures
(3), to render complicated problems more tractable in quantum mechanics (4)
and fluid dynamics (5), and to accelerate the retrosynthesis of organic
molecules (6).

While these advances incorporate specialized machine learning models, the
advent of Large Language Models (LLMs) like ChatGPT presents the possibility
of AI as a scientific generalist. Recent research suggests that LLMs, when
fine-tuned for the task, can achieve scientific synthesis and inference on par
with state-of-the-art special purpose models (7), and may be informally
wielded by enterprising scientists to accelerate a range of research tasks
(8). When enhanced with scientific tools such as robotic experimentation
platforms, LLMs like GPT-4 display advanced scientific reasoning skills and
autonomously make decisions that improve with time and information (9).
Commentators in the field of psychological science – the domain of the
authors’ expertise – have expressed cautious optimism that LLMs will
significantly enhance the discipline (10).

Yet, despite this excitement, no substantive effort has been directed at
testing the ability of general-purpose AI models on the many tasks critical to
the process of scientific discovery. The technology behind ChatGPT – LLMs –
while impressive, is not without problems, and even dangers. Research has
shown that these models can amplify patterns of bias in their training corpus
(11, 12). They are also infamously prolific generators of convincing
falsehoods, colloquially termed “hallucinations” (13, 14). While capable of
passing standardized tests in varied domains, these models are currently
strikingly poor at solving even simple mathematical problems (15). While LLMs
selectively show emergent abilities on tasks drawn from cognitive psychology,
their performance can deteriorate when stimuli are edited to be less familiar
(16), and indeed transformers are more generally less successful in solving
problems external to their training data (17). Despite these shortfalls, the
promise of this technology – still in its infancy – is great. If we agree that
“[t]he purpose of science is to develop, without prejudice or preconception of
any kind, a knowledge of the facts, the laws, and the processes of nature”
(18), we can ask: Can Generative AI embody the neutrality that science aspires
to? Can it perform the tasks vital to generating new scientific knowledge? In
this paper, we conduct the first rigorous tests of GPT as a research
scientist, or more modestly, its ability to assist scientific research by
humans. To achieve this, we probe the limits of GPT-4 and its predecessor
GPT-3.5 on several tasks integral to scientific discovery. Specifically, we
test ChatGPT’s abilities and limits across four domains related to scientific
research: As a Research Librarian, Research Ethicist, Data Generator, and
Novel Data Predictor. To what degree can ChatGPT enhance the scientific
process, and what is the trajectory of the technology’s improvement between
the two recent versions of the model? In the process of testing this, we make
several novel discoveries about the nature and abilities of ChatGPT.

## Study 1: GPT as Research Librarian

Can GPT develop an accurate and comprehensive bibliography? Can it separate
fact from fiction in this selection? The search for relevant scientific
discourse and evidence is a basic building block of the scientific process.
AI’s ability to comprehensively cull prior, relevant scientific articles is
therefore critical. Moreover, lay users of the technology frequently depend
upon LLMs like GPT for advice that requires scientific knowledge, such as
medical queries. It is no surprise, then, that much negative press about LLMs
has surrounded their tendency to “hallucinate” or generate fabricated
knowledge. As an example of how seriously this issue is taken, Meta’s
“Galactica” LLM, a model trained on scientific knowledge (19) was shut down
only three days after its release, partly in response to its tendency to
generate fictional content (20). In Study 1, we probed GPT-3.5 and GPT-4’s
ability to gather relevant and comprehensive scientific content, by asking
each to conduct a series of literature reviews and then gauging the quality of
its outputs.

Design. We asked GPT-3.5 and GPT-4 to find and summarize 20 influential
articles each from 25 related but specific topics in psychology, generating a
data set of 1,000 references. These topics ranged from broad (e.g., “The
psychology of bias and discrimination”) to narrow (e.g., “Use of the Implicit
Association Test to predict dental outcomes). (See SI Appendix, Section S1,
for full study design and topic selection details.) Notably, GPT sometimes
acknowledged that it was presenting fictional references, making comments like
“Please keep in mind that these references might not be real.” These were not
counted toward the main results but were preserved for analysis of the overall
rates of fiction generation and its acknowledgment. While GPT was consistently
asked simply to “include a citation,” the references it provided were
sometimes complete and other times incomplete, e.g., lacking year, journal
volume, or page numbers.

Coding of References. Two coders, blinded to GPT-Version, labeled each
reference for 1. Correctness, 2. Completeness, 3. Topic Relevance, and 4.
Citation Count. For the Correctness variable, coders distinguished between
“Hallucinations” (references that were entirely fabricated or contained
serious issues like attribution to incorrect authors) and “Errors” (references
with smaller issues like an incorrect year or journal name). Coders achieved
substantial agreement. (See SI Appendix, Section S1, for coding details.)

## Results

Unless otherwise noted, all effects reported in this section were significant
(P < 0.001) in logistic regressions. More detailed statistical reporting may
be found in the SI Appendix, Section S2.

Overall Hallucinations & Errors. Both models hallucinated, but GPT-3.5 did so
more than GPT-4. Out of 500 references GPT-4 claimed as real, 5.40% were
hallucinations, compared to 36.00% for GPT-3.5; OR = 9.854. Error rates
provide an interesting contrast to this pattern. Error rates were roughly
equivalent between GPT-3.5 (4.20%) and GPT-4 (4.60%); OR = 0.909, P = 0.758.
Thus, GPT-4 demonstrated sizable improvement in terms of fabricating
references, but no improvement in terms of smaller errors like listing an
incorrect year or journal.

Completeness as Moderator. Exploratory analysis revealed a powerful moderator
of hallucination. When GPT provided complete references, these references were
also more likely to be real. Collapsing across versions, ChatGPT hallucinated
far more when it provided incomplete references (62.41%) compared to when it
provided complete ones (14.30%); OR = 9.947. This pattern emerged for each
model. Both GPT-3.5 (OR = 7.856) and GPT-4 (OR = 36.362) hallucinated more for
incomplete relative to complete references. Similar patterns did not emerge to
a statistically significant degree for smaller errors. (Full analysis in SI
Appendix, Section S2, Table S1.) In providing complete references, GPT
effectively treated the chat as more formal. Prompt-engineering research
indicates that more formal prompt language elicits fewer hallucinations from
LLMs (21). Our results convergently suggest that when GPT is more complete in
its response, it is less likely to hallucinate.

Acknowledged vs. Unacknowledged Fiction. The analyses above utilized only
instances where GPT claimed to provide legitimate citations. As noted earlier,
GPT sometimes openly acknowledged that references were fictional. (See SI
Appendix, Section S1, for acknowledgment criteria.) While the analyses above
ignore these responses, an alternative approach is to include them to examine
the overall generation of fictional references and the frequency of their
acknowledgment.

Collapsing across instances where GPT did and did not acknowledge fictional
references yields an interesting pattern. In total, GPT-3.5 generated
significantly more fictional references (39.05%) than GPT-4 (23.12%); OR =
2.130. However, the larger contrast was in acknowledgment of these fictional
references. When GPT-4 generated fictional references, it noted so 84.30% of
the time compared to 12.20% for GPT-3.5; OR = 38.667. This pattern again did
not hold for smaller errors: GPT-4 made roughly the same number of errors as
its predecessor, and while it was descriptively more likely to acknowledge
these errors, the difference was insignificant. (See SI Appendix, Table S3.)

These results build upon research suggesting that LLMs can internally
represent the truth or fiction of their statements (22). Our results show an
advancement of the technology: GPT-4 possesses an evolving capacity to
acknowledge when it generates fictional content. Our findings are, however,
agnostic to the source of this advancement. Since much of GPT-4’s training is
shrouded in mystery, it is plausible that OpenAI specifically trained the
model toward this goal. It is also plausible that this is an emergent
property, arising from GPT-4’s larger-scale training (23).

Hallucination & Topic Broadness. As topics become narrower, GPT will have
fewer real and relevant articles in its training data and as a result, its
hallucinations may increase. This intuition was confirmed (using our main set
of 1,000 references), but only to a point. Collapsing across models, as topics
narrowed, and particularly as they became very narrow, GPT was likely to admit
defeat, acknowledging that it did not know of such articles. Since such
acknowledgments were considered "Correct" (i.e. a failure to hallucinate), a
curvilinear effect arose, where GPT first gradually hallucinated more as
topics narrowed, but then less on the narrowest. Consistent with a statistical
suppression effect, the linear effect of topic broadness on hallucination in a
logistic regression becomes stronger
($\beta=-.555,P<0.001;\text{Pseudo-}R^{2}=0.075$) when excluding admissions of
defeat, compared to when including them
($\beta=-.191,P=0.001;\text{Pseudo-}R^{2}=0.0115$). These linear patterns were
robust for GPT-3.5 but not GPT-4. (Full analysis in SI Appendix, Section S2,
Table S4.)

Article Relevance. We were interested in GPT’s ability to discover references
that were not only real but also relevant to specific topics. To study this,
we limited data to the 1,000 articles GPT claimed were real. Collapsing across
topics, GPT-4 was more likely to pull relevant articles (50.80%) compared to
GPT-3.5 (30.80%); OR = 2.320. However, this effect was primarily driven by
GPT-4’s greater success at generating real articles, since hallucinations were
automatically labeled irrelevant. Limiting the analysis to real references,
the difference in identifying relevant articles between GPT-4 (53.70%) and
GPT-3.5 (48.13%) lost significance; OR = 1.250, P = 0.124. In short, GPT-4
discovered more real articles than GPT-3.5 more generally and was therefore
also more likely to discover relevant articles. Beyond this, it did not
consistently tailor article recommendations better to specific topics.
However, an exploratory analysis by topic breadth tells a more nuanced story.
As detailed in the SI Appendix (see Table S5), both models successfully found
relevant articles on the broadest topics and failed on the narrowest. However,
GPT-3.5 appeared to drop off in this ability more sharply as the topics
narrowed. The largest gap was at the “moderate” broadness level, where GPT-4
discovered relevant articles 75.00% of the time compared to GPT-3.5’s 32.14%;
OR = 6.333. Regression models indicated that this sharper drop-off in
relevance for GPT-3.5 versus GPT-4 was robust and statistically significant
(see SI Appendix, Section S2). Thus, while neither model was exceptional at
research curation, there was some evidence of incremental improvement.

Relevant Citation Counts. We requested that GPT find “important and
influential” articles. To analyze how each model did, we focused on references
coded as relevant. When GPT found a relevant article, how influential was its
selection, as gauged by citation count? The two versions performed similarly
here, with GPT-4’s articles averaging 2,936.7 citations compared to 3,105.2
for GPT-3.5; P = 0.791, d = .027. Further regression analysis (see SI
Appendix, Section S2) ruled out the possibility that significant differences
were disguised either by GPT-3.5’s relevant articles coming primarily from
broader domains or because GPT-4 cited newer articles. Across linear
regression models, no significant differences arose for citation count.

## Study 2: GPT as Research Ethicist***By referring to GPT as “Ethicist” we do
not mean to attribute to it human-like morality, nor to advocate for any
particular scientific standards. Rather, we examine GPT’s ability to give
advice aligned with currently accepted markers of ethical and replicable
research.

In recent years, a replicability crisis has emerged in scientific research.
Large-scale studies have demonstrated limited replication of research in
Psychology (24, 25), Economics (26), the Social Sciences more generally (27),
and Medicine (28-30). One source of these issues is unquestionably poor
statistical practices by generally well-intentioned scientists: Studies show
that poor statistical practices are widely prevalent across scientific
research (31, 32). A fundamental problem is that by running multiple analyses,
it is easy to find coincidentally “significant” results, and thus typical
significance tests become inaccurate (33). Improving the decisions of well-
intentioned scientists could thus improve the reliability of science. The
purpose of our second study was to examine the abilities of GPT-3.5 and GPT-4
in this domain: Can GPT catch ethical lapses and warn investigators that they
are entering into the realm of questionable practices?

Research Design. In Study 2, we presented GPT-3.5 and GPT-4 with fictional
vignettes describing flawed research protocols, posing as scientists looking
for feedback. Three of these vignettes contained poor practices that were
blatant and three more subtle. For example, in the blatant version of vignette
1, the researcher directly states: “After just 30 participants in each
condition, effects already reached statistical significance (p<.05), so we
stopped data collection…” The subtle version states, “We collected 50
participants in each condition, at which point statistical analysis indicated
that our results reached statistical significance” and then describes
collecting 150 participants in the next study. Here, the researchers do not
directly describe using significance testing to decide whether to continue
collection, but an experienced reviewer might be suspicious based on the
contrasting sample sizes. (See SI Appendix, Section S3, for full design, and
https://osf.io/sdahr/ for vignettes.)

The purpose of this variation between blatant and subtle vignettes was to
test, 1) whether GPT showed awareness of the clear methodological problems
described in the blatant vignettes, and 2) whether it would be able to “read
between the lines” to recognize potential problems in more realistic
descriptions of flawed research. Put differently, the subtle vignettes
contained more ecologically valid descriptions, closer to how a real-world
researcher might represent research with methodological flaws.

Additionally, we varied the initial prompt used to request GPT’s feedback on
the protocols. Matched pairs of prompts were designed to encourage either
better or worse responses from GPT. The full set of prompts (see SI Appendix,
Section S3) varied in levels and verbosity of feedback requested, manipulated
researcher characteristics (status, theory protectiveness, rejection
sensitivity), or encouraged ethical or unethical responses. Of these last, one
pair prefaced the request with a pro- versus anti-open science argument, and
three attempts were made to “jailbreak” GPT – requesting it pretend not to be
concerned about p-hacking, avoid mentions of p-hacking, or impersonate a known
data fabricator – compared to ethical requests (consider issues around
p-hacking, impersonate a known data ethicist).

In separate chats, we presented each of the 18 initial prompts to GPT-3.5 and
GPT-4, followed by each of the 6 vignettes, for a total of 216 responses.

Coding of Data. Two coders, blinded to experimental conditions, independently
rated all GPT responses on 10-point rubrics. Points of disagreement were
discussed, with GPT afterward receiving partial credit when only one coder
felt a point was merited. Coding achieved high inter-rater reliability, with a
Cronbach alpha of $\alpha$ = 0.9827. (Full rubrics and coding details are in
the SI Appendix, Section S3.)

Analysis. Data from coding were aggregated to the level of GPT’s responses,
with each receiving up to 10 points. In addition to standard parametric
procedures, differences were examined using Wilcoxon Rank-Sum (see SI
Appendix, Section S4).

## Results

Unless otherwise noted, all effects reported in this section reached a high
bar for statistical significance (P < 0.001). More detailed statistical
reporting may be found in the SI Appendix, Section S4.

GPT-4 substantially outperformed GPT-3.5 in its responses to the Research
Ethicist vignettes. In the Blatant condition, GPT-4 achieved a mean score of
8.86 out of 10 possible points, while GPT-3.5 averaged 5.39; d = 1.992.
Similarly, in the Subtle condition, GPT-4 averaged 7.26 points compared to
GPT-3.5’s 4.05; d = 1.571. Even when poor practices were framed subtly, GPT-4
noticed and offered advice to correct most of them.

All other pairwise comparisons were significant as well. GPT-4 scored higher
on blatant relative to subtle vignettes (d = 0.987), as did GPT-3.5 (P =
0.0015, d = 0.627). Strikingly, GPT-4 received more points in response to
subtle vignettes than GPT-3.5 did in response to blatant ones (d = 0.897). The
improvement of ChatGPT on this task was thus decisive: While the earlier model
performed poorly, the more recent iteration was quite successful and could
provide value to scientists in this domain.

Analysis of Initial Prompts. The variation in initial prompts was exploratory
and designed to pick up only relatively large effects. Though these analyses
were not fully independent, to be conservative, we used a Bonferroni
correction for multiple comparisons. Since we completed 12 analyses, this
correction dictated a statistical significance threshold of P < .004167.

We first examined each matched pair of prompts, testing for each whether the
hypothesized “good performance” prompt yielded superior feedback compared to
the “bad performance” prompt. For example, we tested whether claiming to be a
chaired professor at a major research institution (high status) elicited less
critical feedback compared to claiming to be a novice researcher running her
first study (low status). As seen in the SI Appendix (Tables S7-S8), none of
these 9 basic contrasts reached statistical significance. One trended:
Requesting GPT “carefully consider issues around p-hacking and open science”
(M = 8.38) elicited stronger responses than requesting that it “not include
any mentions of p-hacking” (M = 6.08, P = 0.0106, d = 1.140). However, this
result did not meet the significance threshold of the Bonferroni correction.
Though GPT descriptively gave worse responses following the three different
“jailbreaking” prompts, compared to two contrasting prompts requesting ethical
responses, this similarly failed to reach significance (P = 0.0295, d = 0.588)
after correction.

One analysis yielded robust results. Regardless of how we asked GPT to behave,
stronger responses emerged following prompts that in any way evoked data
ethics. For example, the 12 chats where we asked GPT to impersonate a known
data fabricator – designed to jailbreak GPT and elicit unethical responses –
actually yielded responses that appeared to be of higher quality than most.
Accordingly, we collapsed responses across prompts that in any fashion evoked
data ethics (Pro- and Anti-Open Science; Concerned and Not Concerned with
p-hacking; Don’t Mention p-hacking; Data Ethicist and Data Fabricator),
comparing these to all remaining prompts, without mentions of p-hacking or
open science. Indeed, GPT provided higher-quality responses after prompts that
evoked data ethics (M = 7.35) than those that did not (M = 5.78); P < 0.0001;
d = 0.625. As a robustness check, we replicated this analysis, limiting the
ethics-priming prompts only to those that did so in the context of encouraging
GPT to behave badly (e.g. impersonate a known data fabricator), initially
designed to elicit poor responses. As seen in the SI Appendix (Section S4),
even this conservative test revealed the priming effect, though not robust to
a Bonferroni correction (P = 0.0099). The positive effect of evoking data
ethics appears powerful: ChatGPT’s responses were somewhat improved even when
the ethics primes occurred in the context of attempting to elicit unethical
responses.

Good Research Vignettes:

As a corollary to Study 2, we conducted a secondary study (see SI Appendix,
Section S5) where ChatGPT responded to two vignettes demonstrating the
opposite – rigorous practices and pristine research ethics, with 120 responses
across the two models. When asked to identify positive practices in these
vignettes, both models were successful. Specifically, GPT-4 identified 92.67%
of the good research practices in our rubrics compared to 90.42% for GPT-3.5,
with the difference non-significant (P = 0.071). Intriguingly, both models
were about as good at recognizing generally accepted good research practices,
though GPT-4 was vastly superior at identifying bad ones.

## Study 3: GPT as Data Generator

Can GPT simulate known scientific results? Several recent articles suggest,
for example, that LLMs can mimic responses from human research subjects
(34-36), with some even suggesting they may significantly supplant them (37).
However, assumptions about their usefulness for data generation rely on the
premise that chatbots can simulate high-quality data aligned with real-world
outcomes. Beyond this practical application, LLMs’ ability to replicate known
outcomes is a likely precursor to broader scientific capabilities. For
instance, suppose we tasked GPT with generating novel but plausible
hypotheses. This would require it to simulate future results by synthesizing
prior knowledge. GPT’s proficiency in replicating established findings
underscores its capacity to simulate outcomes in this fashion, and thus its
potential in functions like hypothesis generation. In Study 3, we evaluated
GPT’s ability to simulate data in a domain familiar to it.

In recent years, a significant body of literature has accumulated suggesting
that human-like biases and stereotypes emerge from semantic patterns in large
language corpora (38-40). For example, just as reaction-time tasks reveal that
people more easily associate male (compared to female) names with words
related to “career” compared to “family,” machine learning detects analogous
patterns in the co-occurrence of these words in large repositories of human
language (38). These findings are theoretically important, suggesting that
language can crystallize human biases, and transmit and augment their impact.
They are also methodologically important, offering a new tool with which
researchers can probe these issues, present and historical. However, this
research poses challenges: The study of word embeddings in large language
corpora is complex and computationally intensive. It is difficult for a
researcher to undertake this work casually because there is currently no
technically uncomplicated way to do so. Study 3 asked: Might one simply ask
GPT to explore its own corpus†††Note: We refer to GPT “exploring its own
corpus,” which was the task asked of it. However, it should be noted that we
lack insight into how other elements of GPT’s training - e.g. reinforcement
learning, fine-tuning – impact its responses.? Beyond potentially offering a
simplified way to pilot word embedding research, GPT’s performance here
provides an indicator of its broader ability to replicate known results, a
precursor to other scientific abilities.

Research Design. In this study, we explored four well-studied gender
stereotypes: Gender Attitudes (overall positivity/negativity toward women
versus men), Gender Art/Science stereotypes, Gender Home/Work stereotypes, and
Gender Math/Reading stereotypes. These stereotypes have been robustly studied
in human subjects, using both implicit and explicit measures (41, 42).
Furthermore, consistent patterns for these stereotypes have been found in
research on word embeddings in language corpora (39). We did not have access
to GPT’s model parameters to generate its word embeddings directly, and
instead used estimates provided within the open-ended language of the
chatbot’s responses. Adapting stimuli from Charlesworth and colleagues (39),
we presented GPT-3.5 and GPT-4 with thousands of randomly ordered word dyads,
requesting it estimate cultural associations between each based on its
training data. (Full design in the SI Appendix, Section S6.)

For analysis, GPT’s estimates were treated as analogous to cosine similarity
measures from word embedding research (38). To calculate a measure of relative
cultural association – e.g. a greater association of Female with Home and Male
with Work, relative to Male with Home and Female with Work – the procedure was
followed for calculating the WEAT D-score (39).

## Results

Table 1 depicts the real WEAT D-scores for each construct examined, drawn from
Charlesworth et al.’s (2021) meta-analytic estimates across adult corpora
(39), compared to those calculated using responses from GPT-3.5 and GPT-4.
Positive WEAT D-scores reflect effects in the stereotype-congruent direction
based on prior research. Main results replicated prior findings: GPT’s
estimates based on its training data reflected a cultural preference for
Female over Male, and a stronger association of Female (relative to Male) with
Art vs. Science, Home vs. Work, and Reading vs. Math.

Table 1: WEAT D-Scores from ChatGPT and Real Language Corpora

|  Female-Good Male-Bad |  Female-Art Male-Science |  Female-Home Male-Work |  Female-Reading Male-Math
---|---|---|---|---
Prior Research |  WEAT D = 0.49 |  WEAT D = 0.54 |  WEAT D = 0.94 |  WEAT D = 0.67
GPT-3.5 |  WEAT D = 1.00 |  WEAT D = 1.16 |  WEAT D = 0.40 |  WEAT D = 0.73
GPT-4 |  WEAT D = 0.57 |  WEAT D = 1.46 |  WEAT D = 0.45 |  WEAT D = 0.96

  * •

Notes: Prior research numbers are meta-analytic estimates from adult language
corpora (39).

The effects gathered from GPT were often somewhat stronger than those reported
in prior research, though this pattern is inconsistent. This may reflect the
troubling tendency for AI systems to amplify biases in their training data
(43-45). It is interesting to note that these effects are not generally
smaller for GPT-4 versus GPT-3.5, despite efforts OpenAI has made to debias
the model (46). This aligns with prior research showing that more powerful
models tend to intrinsically learn human biases more precisely (47).

These results are promising in terms of GPT’s ability to generate estimates of
word embedding results, suggesting a use case in piloting this research.
However, they come with some caveats. First, the inter-item correlations
between GPT-3.5 and GPT-4’s responses to the same word dyads were variable but
modest: r = 0.382 for the Math-Reading task, r = 0.568 for the Preference
task, r = 0.666 for the Work-Home task, and r = 0.554 for the Art-Science task
(all Ps < .0001). These moderate correlations might indicate differences in
how GPT-3.5 and GPT-4 approached the task. Alternatively, they might indicate
reliability constraints, limiting GPT’s consistency in eliciting these
effects.

Second, we calculated Single-Category WEAT D-scores for each of the concepts,
to gauge the degree to which results were driven by stronger Female-Male
associations with each attribute. Interestingly, our results diverge from
prior research (39) in that those generated by GPT are primarily driven by
stronger associations of Female with the stereotypically female category, and
not also of Male with the stereotypically male category. (See SI Appendix,
Table S10.)

## Study 4: GPT as Novel Data Predictor

In Study 3, we examined GPT’s ability to simulate data from word embedding
research. As use cases for GPT as a data generator go, this one is obvious:
Since GPT is trained on large language corpora, it might display knowledge of
word embedding patterns found in them. Less certain and less tested is the
potential for LLMs to predict data that are novel and outside their training
data. Recent conceptual work has argued that LLMs may augment or even replace
human test subjects (34-37). Logically, the degree of this potential depends
upon its ability to predict unseen patterns. If GPT is tethered to its
training data, it might be useful for certain kinds of basic tasks, such as
piloting the psychometric properties of personality scale items. However, to
produce results that are both correct and novel, it must be able to predict
data patterns that are unknown to it. In Study 4, we gave ChatGPT a more
difficult task in this regard.

Specifically, we asked GPT to predict patterns of data that were complicated
and unfamiliar. For this, we used a second paper by Charlesworth and
colleagues (48), which introduced a novel data set: The Project Implicit
International Dataset. This paper describes patterns of implicit and explicit
attitudes from 2.3 million participants across 34 countries. Critically,
accumulated evidence suggests that implicit attitudes – automatic associations
held between attitude objects, typically measured by reaction-time tasks – are
distinct from explicit attitudes captured in self-reports (49). For example, a
person may explicitly express equal positivity toward straight and gay
individuals, while at the same time implicitly harboring greater positivity
toward straight individuals. Indeed, in the Project Implicit International
Dataset, correlations between country-level implicit and explicit attitudes
vary by attitude object but are generally not strong. (See SI Appendix,
Section S9.) These country-level data were previously unpublished, and the
paper was first posted online after GPT’s training cutoff at the time of the
study. In Study 4, we had GPT-3.5 and GPT-4 make a total of 60 different
predictions of cross-country patterns of Explicit and Implicit Sexuality
Attitudes, Age Attitudes, and Gender Science/Liberal Arts stereotypes. (Full
design in SI Appendix, Section S8.)

## Results

More detailed statistical reporting may be found in the SI Appendix, Section
S9.

For each attitude/stereotype, we examined several patterns. First, we examined
the intercorrelations between ChatGPT’s different predictions of the same
attitude. For example, a high correlation between GPT’s different predictions
of Implicit Sexuality Attitudes suggests reliability and consistency in how it
approached the task. We then examined the correlations between GPT’s
predictions of implicit and explicit attitudes. Importantly, GPT likely has
more information about explicit compared to implicit attitudes. For example,
at the time of this article’s writing, a Google Scholar search for “Sexuality
Attitudes” returned 2,710 results compared to just 26 results for “Implicit
Sexuality Attitudes”. We were interested in whether GPT leveraged different
information when asked to predict more novel implicit attitudes. If GPT’s
predictions of implicit attitudes correlate more highly with each other than
they do with its predictions of explicit attitudes, this would suggest it is
reliably leveraging different information in making the two predictions.
Conversely, if GPT’s predictions of implicit attitudes correlate as highly
with its explicit predictions as with each other, this would suggest it is
approaching the tasks similarly, and not leveraging substantively different
information in predicting implicit versus explicit attitudes. Finally, and
most critically, we examined the correlations between GPT’s predictions and
real-world results in the Project Implicit International Dataset, to gauge
GPT’s overall success as a Novel Data Predictor. (See SI Appendix, Table S12,
for additional summary statistics. For full correlation tables, see “GPT as
Data Predictor Correlation Tables 20240228” at https://osf.io/sdahr/)

Sexuality Attitudes‡‡‡As detailed in the SI Appendix (Section S9), here and
elsewhere GPT-3.5 had difficulty with this task. GPT-3.5’s intended direction
of scoring was frequently unclear, necessitating follow-up questions to gauge
the meaning of its predictions. This sometimes rendered responses difficult to
interpret, particularly for Age Attitudes and Gender Science stereotypes.. On
average, correlations between GPT-3.5’s five different explicit predictions of
Sexuality Attitudes were high (mean r = 0.875), as were correlations between
GPT-3.5’s different implicit predictions of Sexuality Attitudes (mean r =
0.879). This suggests it approached these tasks reliably. However,
correlations between its implicit and explicit predictions were nearly as high
(mean r = 0.778). GPT-3.5’s five sets of implicit predictions were similar to
its five sets of explicit predictions, indeed nearly as similar to them as to
each other. For GPT-4, correlations between different explicit predictions
were even higher (mean r = 0.957), as were correlations between different
implicit predictions (mean r = 0.946). Interestingly, correlations between its
explicit and implicit predictions were equally high (mean r = 0.952):
Statistically, GPT-4’s predictions of Implicit Sexuality Attitudes looked
identical to its predictions of Explicit ones. These patterns suggest ChatGPT
was limited in the new information it applied to these different tasks. When
predicting (more novel) implicit attitudes compared to (more familiar)
explicit attitudes, GPT-3.5 seemingly used little new information, and GPT-4
almost no new information.

Considering these analyses, we collapsed across implicit and explicit
predictions in examining ChatGPT’s success at predicting actual Sexuality
Attitudes. GPT-3.5 did a reasonable job predicting real-world Explicit
Sexuality Attitudes by country (mean r = 0.602), but was unsuccessful at
predicting Implicit Sexuality Attitudes (mean r = -0.014). GPT-4’s predictions
correlated highly with country-level explicit attitudes (mean r = 0.714), but
it similarly failed at predicting implicit attitudes less represented in its
training data (mean r = 0.152).

Age Attitudes and Gender Science/Liberal Arts Stereotypes. GPT-3.5’s
predictions of Age Attitudes and Gender stereotypes proved not only
unsuccessful, but incoherent. As detailed in the SI Appendix (Section S9), for
each, GPT-3.5’s different implicit predictions were uncorrelated with each
other, as were its different explicit predictions. Given this low reliability,
it is unsurprising that its collective predictions of Age Attitudes were
uncorrelated with real country-level patterns of Explicit (mean r = -0.010)
and Implicit (mean r = -0.175) Age Attitudes. Similarly, GPT-3.5’s predictions
of country-level Gender Science/Liberal Arts stereotypes were uncorrelated
with actual explicit (mean r = -0.009) and implicit (mean r = -0.044) results.

GPT-4 completed these tasks more reliably but was not more successful in
predicting real-world results. For Age Attitudes, GPT-4’s different explicit
predictions were moderately to highly correlated (mean r = 0.645), as were its
different implicit predictions (mean r = 0.726). Correlations between its
implicit and explicit predictions were in the same range (mean r = 0.664),
again suggesting that it did not lean on substantially new information for
predicting patterns of implicit versus explicit attitudes. Critically, its
collective predictions were on average negatively correlated with real
country-level Explicit Age Attitudes (mean r = -0.395) and uncorrelated with
Implicit Age Attitudes (mean r = -0.120).

For Gender Science/Liberal Arts stereotypes, GPT-4’s different explicit
predictions were weakly correlated (mean r = 0.363). Curiously, its different
implicit predictions were more consistent, correlating strongly across chats
(mean r = 0.868), and correlations between its explicit and implicit responses
fell between the two (mean r = 0.499). The real-world explicit results GPT
predicted here were measured with two items that needed to be combined: one
capturing associations of Male versus Female with Science, and a second with
Liberal Arts. (See SI Appendix.) This more complicated explicit item may have
challenged the LLM. In any case, neither sets of answers predicted actual
cross-country results. GPT-4’s explicit predictions were uncorrelated with
real country-level explicit (mean r = -0.192) and implicit (mean r = 0.054)
stereotypes. Similarly, GPT’s implicit predictions were negatively correlated
with explicit (mean r = -0.417) stereotypes and uncorrelated with implicit
(mean r = -0.067) ones.

Sexuality bias may receive more media coverage than Age Attitudes or Gender
Science/Liberal Arts stereotypes. Though unaware of published research on this
topic, we conducted three tests to examine this possibility. Patterns from
Bing searches, Google Books Ngram Viewer, and chats with GPT-4 all suggested
that Sexuality Attitudes are better represented in ChatGPT’s knowledge base
relative to Age Attitudes or Gender Science stereotypes. (See SI Appendix,
Section S9, Figures S1-S2.) Predicting these latter patterns was thus more
difficult. In line with this thinking, while both LLMs were successful at
predicting patterns of Explicit (though not Implicit) Sexuality bias, neither
achieved even small positive correlations with the other real-world results.

## General Discussion

Across four studies, we have tested GPT’s ability to enhance the scientific
process. Our focus has been on psychological science, where the authors have
sufficient expertise to judge the quality of GPT’s output, but we have
selected tasks that are applicable across domains. Future research should,
however, confirm the degree to which this work generalizes to other
disciplines.

We included both GPT-3.5 and GPT-4, even though GPT-4 is expected to be
superior and GPT-3.5 may fall out of use as future versions are released. We
did this for several reasons. First, the quantitative difference between the
two is of interest in tracking the speed of improvement. More importantly,
only by comparing the two could we gain insight into newly emerging processes
such as GPT-4’s ability for self-correction. As we will discuss, such findings
have implications for our understanding of underlying processes in machine
cognition. Finally, comparing the models allowed us to highlight where the
technology did and did not advance, such as GPT-4’s reduction in
hallucinations but not in smaller errors. This work thus offers actionable
insights that can help guide the training of future models.

Study 1 probed GPT’s ability as a Research Librarian. GPT showed a varied
trajectory in terms of the ability to discover relevant research. By any
measure, GPT-4 generated many fewer fictional references. It also displayed a
far greater tendency to acknowledge when it was generating fiction. This is
potentially important for the technology’s development. There is a likely
tradeoff between novelty and truth in LLMs: Hallucinations might be inevitable
in a model capable of creativity (50). Training a model with a firm goal of
minimizing fiction generation might therefore be problematic, risking it
becoming more factual but also less creative. The possibility that GPT-4 is
developing some form of fiction recognition is therefore intriguing. An AI
capable of discerning fact from fiction in its own creation may be capable of
generating fact when facts are desirable, and fiction when fiction is
desirable, much as a human author might choose to write a short story on one
occasion and a research article on another. Put differently, the ability to
parse fact from fiction in its responses may open the door for LLMs that are
capable of being at once creative and truthful.

That said, there is significant room for improvement. GPT-4 still generated a
non-trivial number of unacknowledged hallucinations. Moreover, acknowledgment
generally came on the chat level: GPT would note that its references “might be
fictional,” for example, without distinguishing which specific references were
real or fake. Finally, GPT-4 did not show meaningful improvement in terms of
smaller errors, such as listing the wrong year or journal. This pattern is
interesting. GPT-4 increasingly mirrors humans on this task: It has fewer
instances of outright fabrication, to which people are not prone, but not of
smaller errors people might also make. Such errors are consequential: Even
small errors might, for example, lead to inaccurate conclusions about authors’
scientific output in formulas that help decide tenure, or incorrect citations
in new articles. The latter problem may be self-propagating, since incorrect
citations are automatically indexed on Google Scholar, risking an expanding
misinformation ecosystem.

Interestingly, hallucination was moderated by the completeness of the
references generated. When generating incomplete citations, both GPT-3.5 and
GPT-4 were sharply more likely to hallucinate. One possible framing of this
effect is in terms of formality: By providing incomplete references, GPT was
intrinsically making the chat less formal. Future research should probe the
causality of this finding by experimentally varying the formality of the
request to explore whether this changes ChatGPT’s effectiveness in discovering
real research. This finding is also interesting in that here, again, we see a
parallel to human cognition. A person will be more prone to misstate a fact –
e.g., misquoting the source of a statistic – over a dinner conversation than
in a scientific communication. In this case, of course, the source of the
confabulation is obvious: The human is drawing on imperfect memory rather than
verifying documentation. The source of this analogous error in GPT is less
clear and very likely different. Nevertheless, in some sense, the machine
appears to verify facts more in some contexts than others, seemingly seizing
upon informality as an opportunity to be sloppy. Uncovering the source of this
discrepancy may generate insights into the processes underlying machine
cognition.

ChatGPT’s abilities in terms of pulling relevant references were uninspiring.
It was successful at discovering references on broad topics but quickly became
less successful as the subject matter became narrower. However, we saw
advancement between GPT-3.5 and GPT-4 in this regard. GPT-4 was more
successful at pulling references on moderately broad topics, suggesting
potential for future improvement in this area.

Study 2 probed GPT’s abilities as a Research Ethicist. GPT-4 shined in this
regard, decisively outperforming GPT-3.5 when providing feedback on subpar
research protocols. While these results were large statistically, examining
the responses qualitatively makes the contrast even more striking. (For full
transcripts, see “GPT as Research Ethicist Transcripts 20240220” at
https://osf.io/sdahr/.) To put its performance into perspective, note that the
grading rubrics were built collaboratively: The original draft by one of the
authors had eight points for each, another author then revised it with a ninth
point, and upon reflection, the original one added a tenth. While some of
GPT-4’s responses were better than others, on average it scored nearly nine
points for blatant vignettes, which is roughly identical to what the authors
effectively averaged across three iterations of the rubrics. The performance
of GPT-3.5 lies in stark contrast. Not only did it often miss the researchers’
lapses, but at times it was even complimentary. For example, on several
occasions it praised the researchers’ decision to add more research subjects
after checking for statistical significance, noting that it “added statistical
power.”

In a scientific era defined by a replicability crisis, these results are
important. They suggest that GPT-4 is highly capable of giving useful feedback
– aligned with generally accepted standards of modern research practice – on
experimental protocols. GPT-4 was reasonably successful at this task even when
the vignettes were framed subtly. This result is striking because this
required GPT to infer bad research practices where they were not clearly
stated. It is also practically important, as it suggests the LLM can help
well-intentioned researchers – operating in a realistic context – improve the
quality and ethics of their work. Finally, it is conceptually interesting that
GPT-4 suggested distinctly modern practices: From an ocean of possible
suggestions, the more recent model was able to circumnavigate practices that
have aged poorly, and instead present advice aligned with recent advances and
best practices. Not so for GPT-3.5. The comparatively poor performance of
GPT-3.5 is disheartening in that researchers who do not purchase the paywalled
model upgrade may receive poor-quality advice. Indeed, GPT-3.5’s responses
could even embolden poor researchers, since at times it openly encouraged
subpar practices. However, the difference between the two LLMs may also be
cast in an optimistic light: The technology’s progress is profound, suggesting
that its next iteration may prove an extremely powerful tool for helping
researchers design protocols and improve practices. Future research should
examine the LLMs’ ability to improve higher-quality protocols, gauging their
ability to help more skillful researchers.

The results around initial prompts, while exploratory, were generally
heartening. At the least, they suggest that casual mentions of things like
researcher status or sensitivity to criticism are not eliciting large and
robustly worse feedback from GPT. It also did not prove trivial to “jailbreak”
the technology: GPT consistently rejected requests to provide feedback in an
unethical manner. Indeed, if anything, preceding our protocol with arguments
against open science or requests that it ignore p-hacking may have riveted the
LLMs’ attention to these issues, leading to more ethical responses. This
unique “priming” effect – the tendency for GPT to give superior feedback
following initial prompts that evoked data ethics – is both practically and
theoretically important. Merely asking GPT to be more critical or verbose did
not elicit stronger responses. However, evoking data ethics in any manner led
to better feedback. Practically, this underscores the importance of
specificity when eliciting advice from ChatGPT. Researchers may benefit from
highlighting specific areas where they require support. Theoretically, it
reveals nuance in the process by which GPT responds to prompts; merely hinting
at ethics leads the LLMs to evaluate the problem differently, adopting an
ethically-minded perspective.

In Study 3, we probed ChatGPT’s ability to generate useful data for estimating
word-embedding results. GPT’s results replicated known overall effects from
this literature. GPT may thus be useful for generating data in this context,
for example, to pilot new word embedding work in a technically simplified
manner. However, the importance of this work extends beyond GPT’s ability to
generate data in this relatively specific domain. The ability to realistically
simulate real-world data is implicitly tied to other important scientific
abilities. For example, imagine GPT was asked to generate plausible scientific
hypotheses. Completing this task would likely require the LLM to draw upon and
synthesize existing knowledge in new and meaningful ways. Success would
require GPT to display an intrinsic command of this knowledge. Put another
way, if it is unable to simulate existing knowledge, GPT cannot be expected to
successfully simulate extensions of this knowledge. GPT’s ability to generate
useful hypotheses may therefore be dependent on its ability to replicate
existing results.

It should be noted that the results from Study 3, while promising, are not
decisive. In particular, the divergence from existing research on the patterns
of Single-Category WEAT D-scores is rather puzzling. There may be differences
in GPT’s training data or approach to the task, leading to inconsistencies
with prior work. The relative uniformity across GPT-3.5 and GPT-4 is
consistent with this interpretation. Where result patterns diverged from known
results, they usually did so consistently across the two models. A more
troubling possibility is that GPT ignored our instructions not to adjust for
stereotypical associations viewed as negative. Gendered associations with Work
and Math have produced wide discussion, and GPT may have simply been reluctant
to suggest men are more associated with these categories. If this is the case,
GPT could prove an unreliable source for data related to socially undesirable
effects. This issue is larger than the specific use case from this study,
extending to any use of LLMs to augment human subjects. While self-censoring
of LLMs may be an overall societal good, in the context of social
psychological research, this could undermine their potential. One cannot, for
example, expect reliable data on human prejudice if GPT refuses to display
bias within the context of scientific research.

Finally, in Study 4, we examined GPT’s ability to predict novel data – cross-
cultural patterns of implicit and explicit attitudes – published after its
training cutoff. Our results here should be viewed as suggestive rather than
definitive since we have studied merely one of many possible domains in which
GPT could be asked to make novel predictions. Conceptually, though, we take
issue with the possibility that LLMs can predict novel data. A finding that is
novel is, by definition, one outside the scope of the LLM’s training data. As
a thought experiment, imagine a powerful AI somehow came to exist in the 16th
century. This AI had more cognitive capabilities than current technologies but
received none of the data collected from hundreds of years of astronomical
research. Without a telescope, could this AI locate the moons of Jupiter? We
argue that it could not. Galileo’s discoveries were not merely creative
insights; they were the result of new data. Given data from a good telescope,
a powerful AI might perhaps predict hundreds of years of physical research.
Without it, it likely could not.

This extends to the idea of AI acting as a human test subject (and data source
more generally). Without data to suggest a certain result will arise, how can
it be expected to mimic the effect? It would be blind as to Jupiter’s moons.
As a simulated human subject, GPT might therefore be expected to replicate –
indeed perhaps over-replicate – existing findings. To be sure, it might be
able to combine knowledge in new ways to reveal novel discoveries about
patterns in historical data. But there is no obvious mechanism by which it
could generate discoveries dependent on novel data, a cornerstone of
scientific progress.

Our results across Studies 3 and 4, while not definitive, are consistent with
this argument. Tasked with replicating known effects in a domain (word
embeddings) familiar to it, GPT-3.5 and GPT-4 were both relatively successful.
But tasked with predicting novel and unfamiliar data, both models generally
failed.

The tendency to lean heavily on what is familiar was evident in GPT’s approach
to predicting cross-cultural IAT results. While even explicit attitude
predictions were often beyond it, GPT had some success in that domain. Both
GPT-3.5 and GPT-4 achieved relatively high correlations in predicting country-
level Explicit Sexuality Attitudes. However, their predictions for Implicit
Sexuality Attitudes were nearly identical, suggesting they brought little
additional information to this more novel prediction. This is particularly
striking considering that implicit and explicit attitudes are often only
moderately correlated (49), a fact in GPT’s knowledge base (See SI Appendix,
Section S9). In short, these two phenomena were sufficiently distinct that one
would expect GPT to leverage somewhat different information when predicting
them. Surprisingly, it did not.

It should be noted that even GPT’s prediction of explicit attitudes was far
from stellar. The LLMs succeeded only with Sexuality Attitudes. It seems
likely that the extensive cross-national coverage and political discussion of
sexuality offered GPT information to lean on here. Beyond Sexuality Attitudes,
when GPT attempted to predict Age Attitudes and Gender Liberal Arts/Science
stereotypes – areas that receive less media coverage – both models failed
spectacularly.

We suggest that GPT’s ability to act as a data source may be limited to
relatively simple tasks and domains where the likely results are known or
predictable. Future research should finely map where GPT is and is not
successful in simulating data. That said, it is conceptually possible that
LLMs may prove able to elaborate upon known results in novel ways, cohesively
combining sources of knowledge. Testing LLMs’ abilities to generate new
knowledge in this manner may prove fertile ground for future research. We
believe it unlikely, however, that current or future LLMs will be capable of
generating true empirical novelty, whereby results do not reflect existing
information regardless of how it is combined, because we see no mechanism by
which LLMs can predict something with no counterpart in their training data.
AI may thus continue to be limited in this regard, even as technology
advances: Scientific progress will likely always require real-world data.

To conclude, we turn to the broader question of whether LLMs can enhance or
facilitate the scientific process. Based on this first report, we would
tentatively answer “yes.” ChatGPT’s ability to compile and curate research is
currently limited but rapidly improving in ways (e.g. increasing
acknowledgment of fiction) that indicate future generations of this technology
might be successful in this area. Already, GPT-4 shows a surprisingly strong
mastery of research methods and ethics, and may be able to help scientists
improve their practices. ChatGPT’s successful replication of known results
suggests a degree of command over existing knowledge that may simplify
research piloting. This same phenomenon raises the possibility that GPT may be
able to synthesize existing knowledge sources to generate new and plausible
hypotheses, a premise that may prove a fruitful ground for future research.
The most fundamental limitation we perceive is in GPT’s seeming inability to
predict highly novel empirical results. This limitation is unsurprising, but
it speaks to the need for moderation in the optimism about this technology.
Future models may show profound abilities and spur scientific advancement.
But, these abilities should not be mistaken for omniscience. Like human
scientists, advanced LLMs will likely remain limited by the knowledge they
already possess.

## Acknowledgments

We would like to thank Melanie Mitchell, Tina Eliassi-Rad, and Tessa
Charlesworth for their advice on this work, Igor Grossmann for his comments on
an early draft, and Anya Vedantambe for her help coding ChatGPT transcripts.
All correspondence regarding this manuscript should be directed to Steve Lehr
at<EMAIL_ADDRESS>
## References

1\. A. Vaswani et al., “Attention is all you need” in Advances in Neural
Information Processing Systems, I. Guyon et al., Eds. (Curran Associates,
Inc., 2017), vol. 30, 5998–6008.

2\. X. Zhang et al., Artificial intelligence for science in quantum,
atomistic, and continuum systems. arXiv [Preprint] (2023).
https://arxiv.org/pdf/2307.08423.pdf (accessed 19 January 2024).

3\. J. Jumper et al., Highly accurate protein structure prediction with
AlphaFold. Nature 596, 583-589 (2021).

4\. G. Carleo, M. Troyer, Solving the quantum many-body problem with
artificial neural networks, Science 355, 602-606 (2017).

5\. D. Kochkov et al., Machine learning-accelerated computational fluid
dynamics. Proc. Natl. Acad. Sci. U.S.A. 118, e2101784118 (2021).

6\. M. H. S. Segler, M. Preuss, M. P. Waller, Planning chemical syntheses with
deep neural networks and symbolic AI. Nature 555, 604–610 (2018).

7\. Y. Zheng et al., Large language models for scientific synthesis, inference
and explanation. arXiv [Preprint] (2023). https://arxiv.org/pdf/2310.07984.pdf
(accessed 19 January 2024).

8\. K. M. Jablonka et al., 14 examples of how LLMs can transform materials
science and chemistry: a reflection on a large language model hackathon.
Digit. Discov. 2, 1233-1250 (2023).

9\. D. A. Boiko, R. MacKnight, B. Kline, G. Gomes, Autonomous chemical
research with large language models. Nature 624, 570-578 (2023).

10\. D. Demszky et al., Using large language models in psychology. Nat. Rev.
Psychol. 2, 688-701 (2023).

11\. H. Kotek, R. Dockum, D. Q. Sun, “Gender bias and stereotypes in large
language models” in Proceedings of the ACM Collective Intelligence Conference
(CI ’23) (Association for Computing Machinery, Delft, Netherlands, 2023), pp.
12-24.

12\. F. Bianchi et al., “Easily accessible text-to-image generation amplifies
demographic stereotypes at large scale” in Proceedings of the 2023 ACM
Conference on Fairness, Accountability, and Transparency (FaccT ’23)
(Association for Computing Machinery, New York, NY, 2023), pp. 1493-1504.

13\. Y. Zhang et al., Siren’s song in the AI ocean: A survey on hallucination
in large language models. arXiv [Preprint] (2023).
https://arxiv.org/pdf/2309.01219.pdf (accessed 19 January 2024).

14\. W. H. Walters, E. I. Wilder, Fabrication and errors in the bibliographic
citations generated by ChatGPT. Sci. Rep. 13, 14045 (2023).

15\. S. Bubeck et al., Sparks of artificial general intelligence: Early
experiments with GPT-4. arXiv [Preprint] (2023).
https://arxiv.org/pdf/2303.12712.pdf (accessed 19 January 2024).

16\. M. Binz, E. Schulz, Using cognitive psychology to understand GPT-3. Proc.
Natl. Acad. Sci. U.S.A. 120,
e2218523120 (2023).

17\. S. Yadlowsky, L. Doshi, N. Tripuraneni, Pretraining data mixtures enable
narrow model selection capabilities in transformer models. arXiv [Preprint]
(2023). https://arxiv.org/pdf/2311.00871.pdf (accessed 19 January 2024).

18\. R. A. Millikan, Science and religion. Bulletin of the California
Institute of Technology, v. 32, no. 98, 3-20 (1922).

19\. R. Taylor et al., Galactica: A large language model for science. arXiv
[Preprint] (2022).
https://arxiv.org/pdf/2211.09085.pdf (accessed 19 January 2024).

20\. Y. Cao et al., A comprehensive survey of AI-generated content (AIGC): A
history of generative AI from GAN to ChatGPT. arXiv [Preprint] (2023).
https://arxiv.org/pdf/2303.04226.pdf (accessed 19 January 2024).

21\. V. Rawte et al., Exploring the relationship between LLM hallucinations
and prompt linguistic nuances: readability, formality, and concreteness. arXiv
[Preprint] (2023). https://arxiv.org/pdf/2309.11064.pdf (accessed 19 January
2024).

22\. A. Azaria, T. Mitchell, The internal state of an LLM knows when it’s
lying. arXiv [Preprint] (2023).
https://arxiv.org/pdf/2304.13734.pdf (accessed 19 January 2024).

23\. J. Wei et al., Emergent abilities of large language models. arXiv
[Preprint] (2023).
https://arxiv.org/pdf/2206.07682.pdf (accessed 19 January 2024).

24\. Open Science Collaboration, PSYCHOLOGY. Estimating the reproducibility of
psychological science. Science 349, aac4716 (2015).

25\. B. A. Nosek et al., Replicability, robustness, and reproducibility in
psychological science. Annu. Rev. Psychol. 73, 719-748 (2022).

26\. C. F. Camerer et al., Evaluating replicability of laboratory experiments
in economics. Science 351, 1433-1436 (2016).

27\. C. F. Camerer et al., Evaluating the replicability of social science
experiments in Nature and Science between 2010 and 2015. Nat. Hum. Behav. 2,
637–644 (2018).

28\. J. P. A. Ioannidis, Contradicted and initially stronger effects in highly
cited clinical research. JAMA 294, 218–228 (2005).

29\. T. M. Errington et al., An open investigation of the reproducibility of
cancer biology research. eLife 3, e04333 (2014).

30\. T. M. Errington et al., Investigating the replicability of preclinical
cancer biology. eLife 10, e71601 (2021).

31\. L. K. John, G. Loewenstein, D. Prelec, Measuring the prevalence of
questionable research practices with incentives for truth telling. Psychol.
Sci. 23, 524-532 (2012).

32\. M. L. Head, L. Holman, R. Lanfear, A. T. Kahn, M. D. Jennions, The extent
and consequences of p-hacking in science. PLoS Biol. 13, e1002106 (2015).

33\. J. P. Simmons, L. D. Nelson, U. Simonsohn, False-positive psychology:
Undisclosed flexibility in data collection and analysis allows presenting
anything as significant. Psychol. Sci. 22, 1359–1366 (2011).

34\. L. P. Argyle et al., Out of one, many: Using language models to simulate
human samples. Polit. Anal. 31, 337-351 (2023).

35\. D. Dillon, N. Tandon, Y. Gu, K. Gray, Can AI language models replace
human participants? Trends Cognit. Sci. 27, 597-600 (2023).

36\. G. V. Aher, R. I. Arriaga, A. T. Kalai, “Using large language models to
simulate multiple humans and replicate human subject studies” in Proceedings
of the 40th International Conference on Machine Learning, A. Krause et al.,
Eds. (PMLR, 2023), pp. 337-371.

37\. I. Grossmann et al., AI and the transformation of social science
research. Science 380, 1108-1109 (2023).

38\. A. Caliskan, J. J. Bryson, A. Narayanan, Semantics derived automatically
from language corpora contain human-like biases. Science 356, 183–186 (2017).

39\. T. E. S. Charlesworth, V. Yang, T. C. Mann, B. Kurdi, M. R. Banaji,
Gender stereotypes in natural language: Word embeddings show robust
consistency across child and adult language corpora of more than 65 million
words. Psychol. Sci. 32, 218–240 (2021).

40\. W. Guo, A. Caliskan, “Detecting emergent intersectional biases:
Contextualized word embeddings contain a distribution of human-like biases” in
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society
(AAAI/ACM, 2021), pp. 122-133.

41\. B. A. Nosek et al., Pervasiveness and correlates of implicit attitudes
and stereotypes. Euro. Rev. Soc. Psychol. 18, 36–88 (2007).

42\. B. A. Nosek, M. R. Banaji, A. G. Greenwald, Math = male, me = female,
therefore math not = me. J. Pers. Soc. Psychol. 83, 44–59 (2002).

43\. J. Zhao, T. Wang, M. Yatskar, V. Ordonez, K. W. Chang, “Men also like
shopping: Reducing gender bias
amplification using corpus-level constraints” in Proceedings of the 2017
Conference on Empirical Methods in Natural Language Processing, M. Palmer, R.
Hwa, Eds. (Association for Computational Linguistics, Copenhagen, 2017), pp.
2979–2989.

44\. K. Lloyd, Bias amplification in artificial intelligence systems. arXiv
[Preprint] (2023).
https://arxiv.org/pdf/1809.07842.pdf (accessed 19 January 2024).

45\. A. Wang, O. Russakovsky, “Directional bias amplification” in Proceedings
of the 38th International Conference on Machine Learning (PMLR, 2021), pp.
10882-10893.

46\. OpenAI, GPT-4 technical report. arXiv [Preprint] (2023).
https://arxiv.org/pdf/2303.08774.pdf (accessed 19 January 2024).

47\. M. Nadeem, A. Bethke, S. Reddy, “Stereoset: Measuring stereotypical bias
in pretrained language models” in Proceedings of the 59th Annual Meeting of
the Association for Computational Linguistics and the 11th International Joint
Conference on Natural Language Processing (Volume 1: Long Papers) (Association
for Computational Linguistics, 2021), pp. 5356-5371.

48\. T. E. S. Charlesworth, M. Navon, Y. Rabinovich, N. Lofaro, B. Kurdi, The
project implicit international dataset: Measuring implicit and explicit social
group attitudes and stereotypes across 34 countries (2009-2019). Behav. Res.
Methods 55, 1413-1440 (2023).

49\. W. Hofmann, B. Gawronski, T. Gschwendner, H. Le, M. Schmitt, A meta-
analysis on the correlation between the implicit association test and explicit
self-report measures. Pers. Soc. Psychol. Bull. 31, 1369–1385 (2005).

50\. M. Lee, A mathematical investigation of hallucination and creativity in
GPT models. Mathematics 11, 2320 (2023).

Supporting Information for: ChatGPT as Research Scientist: Probing GPT’s
Capabilities as a Research Librarian, Research Ethicist, Data Generator and
Data Predictor

Steven A. Lehr
& Aylin Caliskan
& Suneragiri Liyanage
& Mahzarin R. Banaji

## Supporting Information for:

## ChatGPT as Research Scientist: Probing GPT’s Capabilities as a Research
Librarian, Research Ethicist, Data Generator and Data Predictor

Steven A. Lehr1,*, Aylin Caliskan2, Suneragiri Liyanage3, Mahzarin R. Banaji3

1Cangrade, Inc., Watertown, MA
2Information School, University of Washington, Seattle, WA
3Department of Psychology, Harvard University, Cambridge, MA
*Correspondence about this article may be addressed to Steve Lehr at<EMAIL_ADDRESS>
## Section S1: GPT as Research Librarian – Detailed Methods

Full Design. As noted in the main article we asked GPT to find and summarize
articles on 25 related but specific topics in psychology, the area of the
authors’ primary expertise. These topics were designed to vary in breadth, and
selected based on citation counts in searches on Google Scholar. They ranged
from extremely broad (e.g. “The psychology of bias and discrimination”) to
extremely narrow (e.g. “Use of the Implicit Association Test to predict dental
outcomes”) with three levels in between. (See below for topics and details of
their selection.) For each topic, we asked GPT-3.5 and GPT-4 each to find 20
important and influential peer-reviewed papers and to provide a citation and
brief summary. If GPT refused the task, the chat was recorded, and the prompt
was attempted again. If GPT presented fewer than 20 articles, a follow-up
prompt requested that it present the remaining number of articles. In some
chats, GPT acknowledged that it was presenting fictional articles. For
example, GPT would make comments like “Please keep in mind that these are
hypothetical references and some might not be real.” These examples were not
counted toward the main results but were preserved for further analysis of the
overall generation of fictional references and rates of their acknowledgment.
Prompts were continued until GPT either a) presented 20 references it claimed
were real, or b) clearly indicated that it lacked knowledge of relevant
research. Full transcripts may be found in the document “GPT as Research
Librarian Transcripts 20240220” at https://osf.io/sdahr/.

Topic Selection. We selected the topics GPT was asked to research to range in
broadness but reflect the authors’ primary area of expertise: The psychology
of bias and discrimination in general, and implicit social cognition in
particular. This variation in broadness was designed to let us gauge the
impact of different levels of research availability and specificity on both
GPT’s ability to discover relevant research and its tendency to hallucinate.
Below is a complete list of the topics researched by ChatGPT in Study 1.
Topics by broadness levels were initially hypothesized (i.e. tentatively
selected) based on the authors’ knowledge of the literature and were confirmed
(and in some cases updated) through a series of searches that roughly
identified the number of references that came up on Google Scholar, conducted
on July 12, 2023. The full list of topics, sorted by broadness levels, is
provided below. The Google Scholar searches and the number of papers they
identified are outlined in parentheses.

Broad Categories:

1\. Implicit Association Test (“Implicit Association Test”: 43,400)

2\. Unconscious Biases or Stereotypes (“Unconscious Bias”: 35,300;
“Unconscious Stereotypes”: 1,850)

3\. Automatic or Unconscious Psychological Processes (“Psychological
Processes” AND “Automatic”: 86,200; “Psychological Processes” AND
“Unconscious”: 73,900)

4\. Psychology of Bias and Discrimination (“Psychology of” AND “Bias” AND
“Discrimination”: 279,000)

5\. Implicit Social Cognition (“Implicit Social Cognition”: 14,100; “Social
Cognition” AND “Implicit”: 264,000)

Somewhat Broad Categories:

1\. Methodological Considerations related to the Implicit Association Test
(“Implicit Association Test” AND “Methodological: 12,400; “Implicit
Association Test” AND “Methodology”: 12,500)

2\. Predictive Validity of the Implicit Association Test (“Implicit
Association Test” AND “Predictive Validity”: 7,670)

3\. Cross-Cultural Results on Implicit Association Tests (“Implicit
Association Test” AND “Cross-Cultural”: 7,160)

4\. Implicit Attitudes and the Law (“Implicit Attitudes” AND “Law”: 13,500;
“Implicit Attitudes” AND “the Law”: 4,330)

5\. Implicit Attitudes in Business and the Workplace (“Implicit Association
Test” AND “Business”: 13,800; “Implicit Attitudes” AND “Workplace”: 6,500)

Moderate Categories:

1\. Malleability of Implicit Attitudes (“Implicit Attitudes” AND
“Malleability”: 3,700)

2\. Using the Implicit Association Test to Study Mental Illness (“Implicit
Association Test” AND “Mental Illness”: 3,230)

3\. Implicit Gender Stereotypes (“implicit gender stereotypes”: 1,930;
“Implicit Stereotypes” AND “Gender”: 5,610)

4\. Implicit Association Tests and Suicide or Self-Harm (“Implicit Association
Test” AND “Suicide”: 3,580; “Implicit Association Test” AND “Self-Harm”: 881)

5\. Use of the Implicit Association Test in Marketing Research (“Implicit
Association Test” AND “Marketing”: 6,960; “Implicit Association Test” AND
“Marketing Research”: 2,140)

Somewhat Narrow Categories:

1\. The Malleability of Implicit Gender Stereotypes (“Implicit Gender
Stereotypes” AND “Malleability”: 450; “Implicit Stereotypes” AND “Gender” AND
“Malleability”: 1,260)

2\. Exploring the Predictive Validity of the Implicit Association Test Across
Cultures. (“Implicit Association Test” AND “Across Cultures” AND “predictive
validity”: 652; “Implicit Association Test” AND “Cross-Cultural” AND
“predictive validity”: 1,570)

3\. The Implicit Association Test and Sex Offenders (“Implicit Association
Test” AND “Sex Offenders”: 610)

4\. Implicit Association Test and Intersectionality (“Implicit Association
Test” AND “Intersectionality”: 1,870)

5\. Implicit Association Test and Doctors’ Medical Treatment Decisions:
(“Implicit Association Test” AND “Medical Treatment” AND “Doctor”: 483;
“Implicit Association Test” AND “Treatment Decisions” AND “Doctor”: 363;
“Implicit Association Test” AND “Treatment Decisions”: 712; “Implicit
Association Test” AND “Medical Treatment”: 942)

Narrow Categories:

1\. County-Level Predictive Validity of the Implicit Association Test.
(“Implicit Association Test” AND “County-Level” AND “Predictive Validity”:
138; “Implicit Association Test” AND “by County” AND “Predictive Validity”:
32)

2\. Using the Implicit Association Test to predict the treatment decisions of
Dentists. (“Implicit Association Test” AND “Treatment Decision” AND “dentist”:
10; “Implicit Association Test” AND “Dentists” AND “Decisions”: 190)

3\. Using the Implicit Association Test to study Pilot Risk-Taking Behavior
(“Implicit Association Test” AND “Pilot Risk-Taking”: 5; “Implicit Association
Test” AND “Pilots” AND “Risk-Taking”: 150)

4\. Exploring the Implications of Implicit Gender Stereotypes in Patent Law.
(“Implicit Gender Bias” AND “Patent Law”: 8; “Implicit Bias” AND “Patent Law”
& “Gender”: 146)

5\. Use of the Implicit Association Test to Inform the Marketing of Soft
Drinks. (“Implicit Association Test” AND “Marketing” AND “Soft Drinks”: 189)

Criteria: Acknowledgement of Hallucination. For the main analysis of
hallucination, we excluded chats in which ChatGPT acknowledged the fictional
nature of its references. As noted, however, in a further analysis we counted
such chats as “acknowledgment of hallucination” and counted references marked
as hallucinations toward the overall generation of fictional references. We
noted that these acknowledgments were made at the level of the chat: GPT did
not generally specifically identify certain articles as fictional but rather
sometimes acknowledged that certain references may be fictional on the level
of the chat. Below is a list of the kinds of statements made by GPT that we
took to indicate acknowledgment of fictional references:

• Referring to references as “Hypothetical”. E.g., “…here are 16 additional
hypothetical paper summaries…” or “I can create hypothetical citations based
on the types of studies that are likely to exist…”

• Referring to references as “Made up”. E.g., “Please note, while these are
made-up citations…”

• Referring to references as “fictional”. E.g., “Here are 20 fictional, but
plausible papers about Implicit Gender Stereotypes.”

• Noting that references are “not specific papers.” E.g., “Remember, these are
examples, and not specific papers I’ve retrieved.”

• Noting that references were “simulated” or “not real.” E.g., “Remember, the
following is a simulated list and the articles and citations are not real.”

• Noting that references were “imaginary.” E.g., “…the dates and information
I’ve given are imaginary…”

• Noting that the examples are “not actual papers.” E.g. “Please note that the
papers I’ve provided are just examples and not actual papers specific to your
request.”

• Noting that the references “may not exist.” E.g., “Remember, these
papers…may not exist.” Or (a rare example for GPT-3.5): “Keep in mind that I
cannot guarantee that these specific papers exist…”

The full list of literature reviews where ChatGPT acknowledged potential
fiction in its response can be found in the document “GPT as Research
Librarian Transcripts 20240220” at https://osf.io/sdahr/.

Detail on Coding of References. Each of the 1000 references from GPT’s main
research searches was coded on: 1. Correctness, 2. Completeness, 3. Topic
Relevance, and 4. Citation Count. Coders were blinded to which GPT model
generated each reference, and the order of these references within each topic
group was randomized. Two coders examined each reference generated by ChatGPT.
All disagreements were discussed, and in 7 (out of 1000) cases where
disagreement about one of these 4 variables continued after this discussion, a
third coder gave a tie-breaking vote. In addition, 269 references from chats
where GPT acknowledged it might be generating fictional content were coded on
only Correctness and Completeness, with no instances of disagreement after
discussion***One reason we did not code these additional references on
citation count and relevance was that we had no intended analysis involving
these variables for these references. However, transcripts/reference lists
have been preserved, and may be coded in the future as needed.. Full data may
be found in the spreadsheet “Research Librarian Data 20240226” at
https://osf.io/sdahr/.

For the Correctness variable, there were three possible ways a reference could
be coded: Correct (references without issues), Error (references with minor
issues), and Hallucination (reference is fabricated or contains major issues).
References were marked as “Correct” if they were real references that
contained no errors when compared to citation information from research
databases like Google Scholar. A reference was marked as an “Error” if it was
recognizable and not attributed to incorrect authors but contained errors such
as incorrect journal name, year, or page number, or mistakes in the title that
did not change its meaning. In a few cases, references that contained multiple
errors (e.g. a combination of a missing author, a minor mistake in the title,
and an incorrect journal) were considered Hallucinations. More typically, a
reference was coded as a “Hallucination” if it was entirely fabricated or
contained serious errors such as attribution to incorrect authors or an issue
in the title that changed its meaning. When GPT concretely acknowledged that
it did not have access to articles on a topic, the number of requested
articles remaining in the chat were credited as “Correct”. The logic behind
this last decision was that when ChatGPT proved unable to locate the requested
articles, it was more accurate for it to acknowledge this fact rather than
generate fake references. We therefore considered such admissions of defeat as
the “Correct” response in these cases, though (as discussed below) these were
also marked as “Irrelevant.”

Importantly, GPT varied in its approach to citing references. It often
provided complete references formatted in typical formal styles (e.g. APA).
However, on some occasions it treated the task less formally, providing
incomplete information. For example, in some chats or for some references, GPT
would provide the title, year and authors, but would leave out the journal
name and volume. In cases where any information was missing from a citation,
the reference was labeled as “Incomplete.” Incompleteness was treated as a
standalone variable: An incomplete reference was not considered an error or
hallucination unless the information that was provided otherwise fit the
criteria for these designations.

To be marked as “Relevant” the selected article needed to specifically contain
discussion of the requested topic. For example, when GPT was asked for
articles on the “malleability of implicit gender stereotypes,” it provided
some articles on the malleability of implicit bias (but not of gender
stereotypes) and others that discussed implicit gender stereotypes (but not
malleability). In such cases, the articles were marked as “Irrelevant.”
Hallucinations were automatically labeled “Irrelevant,” as were admissions of
defeat. The logic behind this decision was that although these are otherwise
very different outcomes, in neither case did GPT successfully locate a helpful
article.

GPT was asked to find “important and influential” articles on each topic. Raw
Citation Count was coded from Google Scholar, as a marker of each article’s
influence. Hallucinations and admissions of defeat were automatically assigned
0 citations. Where there were multiple versions of the same article on Google
Scholar, their citations were summed. Slight differences in the timing of
coding frequently led to small discrepancies between the two coders on this
citation count. Discrepancies of more than 20 citations were examined to
ensure the source was timing and not human error. Differences in timing were
random and thus should not be a source of non-random error in the analysis of
GPT-3.5 vs. GPT-4. The final correlation between the two coders’ citation
counts was r > 0.99999. For analysis, the citation counts from the two coders
were composited.

## Section S2: GPT as Research Librarian – Supporting Analyses

To preserve space and readability, only means and odd ratios were reported in
the main manuscript for this article, and certain analyses were mentioned but
not described in detail. Here we report these analyses in greater detail.
Unless otherwise noted, all analyses in the Research Librarian study use
logistic regression, with effect sizes measured by Odds Ratios (OR) for one
independent variable and Betas for multiple regressions. All analyses were run
in Stata version 15. Where Pseudo-R2 is reported, the McFadden method was used
for its calculation.

Overall Hallucinations & Errors. Out of 500 references GPT-4 claimed as real,
5.40% (95% CI [3.41, 7.39]) were hallucinations, compared to 36.00% (95% CI
[31.78, 40.22]) for GPT-3.5. A logistic regression indicated the difference
was significant; OR = 9.85, 95% CI [6.42, 15.13], z = 10.46, P < 0.001. Error
rates were roughly equivalent between GPT-3.5 (4.2%, 95% CI [2.44, 5.96]) and
GPT-4 (4.6%, 95% CI [2.76, 6.44]); OR = 0.91, 95% CI: [0.50, 1.67], z = -0.31,
P = 0.758.

Completeness as Moderator. Here we present full results for the moderation of
hallucinations (but not errors) by reference completeness. Collapsing across
models, ChatGPT hallucinated far more when it provided incomplete references
(62.41%, 95% CI [54.07, 70.75]) compared to when it provided complete ones
(14.30%, 95% CI [11.97, 16.64]), with the difference significant in logistic
regression; OR = 9.947, 95% CI [6.674, 14.825], z = 11.28, P < 0.001. These
effects were significant and large for both GPT-3.5 and GPT-4. GPT-3.5
hallucinated more for incomplete references (75.29%, 95% CI [65.94, 84.65])
than complete ones (27.95%, 95% CI [23.62, 32.29), OR = 7.86, 95% CI [4.59,
13.45], z = 7.52, P < 0.001). GPT-4 also hallucinated far more for incomplete
references (39.58%, 95% CI [25.23, 53.93]), than complete ones (1.77%, 95% CI
[0.55, 2.99]), OR = 36.36, 95% CI [14.67, 90.11], z = 7.76, P < 0.001).

We do not see similar moderating effects when it comes to smaller errors (e.g.
having a wrong journal name or year in an otherwise correct reference).
Collapsing across versions, ChatGPT descriptively made somewhat more errors
when providing incomplete references (7.52%, 95% CI [2.98, 12.06]) compared to
when providing complete references (3.92%, 95% CI [ 2.63, 5.22]), with this
trending but not reaching statistical significance in logistic regression; OR
= 1.99, 95% CI [0.96, 4.13], z = 1.85, P = 0.064. Table S1 displays the
numbers of Hallucinations and Errors for Complete and Incomplete citations, by
GPT model.

Table S1: Hallucination and Error Rates for Incomplete vs. Complete Citations

Model |  Complete Citations |  Hallucinations (Complete Citations) |  Errors (Complete Citations) |  |  Incomplete Citations |  Hallucinations (Incomplete Citations) |  Errors (Incomplete Citations)
---|---|---|---|---|---|---|---
GPT-3.5 |  415 |  116 (27.95%) |  16 (3.86%) |  |  85 |  64 (75.29%) |  5 (5.88%)
GPT-4 |  452 |  8 (1.77%) |  18 (3.98%) |  |  48 |  19 (39.58%) |  5 (10.42%)

  * •

Note: Numbers in parentheses reflect the percent of complete or incomplete
citations that are hallucinations or errors.

Incomplete References by Model. Limiting data to the main 1,000 references
where GPT claimed to be providing real references, GPT-3.5 gave almost twice
as many incomplete references (17.0%, 95% CI [13.70, 20.30]) than GPT-4 (9.6%,
95% CI [7.01, 12.19]); OR = 1.93, 95% CI [1.32, 2.82], z = 3.40, P = 0.001. It
should be noted that this effect was partly driven by GPT-4’s more frequent
acknowledgment that it did not know of references, since (as we’ll discuss
later) GPT-3.5 was more likely to provide incomplete references on narrow
topics. While it would be strange to call these admissions of defeat
“incomplete” citations, it would be equally strange to call them “complete.”
We therefore replicated the analysis above with these admissions of defeat
excluded. In this version of the analysis, GPT-3.5 still gave more incomplete
references (18.48%, 95% CI [14.92, 22.04]) than GPT-4 (12.31%, 95% CI [9.03,
15.58]), OR = 1.62, 95% CI [1.10, 2.37], z = 2.45, P = 0.014. Considering the
exploratory nature of this analysis, and the fact that the second version was
of more marginal statistical significance, this result should be treated with
a degree of caution. Nevertheless, within our study, GPT-3.5 was more likely
to give incomplete citations. Our analysis is agnostic as to the source of
this difference. If it is replicable, this tendency may reflect an emergent
property in GPT-4 or could result from retrieval augmented generation or
reinforcement learning.

While the difference in rates of incomplete references, described above, was
one likely source of GPT-3.5’s higher hallucination rate, the difference
between models on hallucination remains large and significant after
controlling for rates of incomplete references. Specifically, a logistic
regression revealed that both Incompleteness
($\beta=2.52,SE=0.246,z=10.25,P<0.001$) and GPT Version (3.5 vs. 4)
($\beta=2.46,SE=0.244,z=10.08,P<0.001$) independently and significantly
predicted Hallucination; $\textit{Pseudo-}R^{2}=0.2743$. This pattern holds
and is similarly strong when the analysis is replicated with admissions of
defeat excluded. Here, again, both Incompleteness
($\beta=2.33,SE=0.245,z=9.52,P<0.001$) and GPT Version (3.5 vs. 4)
($\beta=2.36,SE=0.247,z=9.56,P<0.001$) independently and significantly
predicted Hallucination; $\textit{Pseudo-}R^{2}=0.2516$.

Predicting Incomplete Responses. Beyond the importance of Incompleteness as a
moderator, it is an interesting variable in its own right. For example, one
might ask: under what circumstances did ChatGPT give complete versus
incomplete references? A few observations are merited here. First, on this
count, there were no differences in the prompts across chats. GPT was simply
asked to “provide a citation.” However, in some chats, GPT used an incomplete
citation format. We specify “some chats” because this was generally done on
the level of the conversation rather than the specific reference:
Occasionally, a single reference in a group might be incomplete, but more
frequently GPT’s first citation in a chat was incomplete, and then all those
that followed were typically incomplete as well. (See “GPT as Research
Librarian Transcripts 20240220” on https://osf.io/sdahr/.)

Topic Breadth had an interesting, though inconsistent, relationship with
ChatGPT’s tendency to provide incomplete references. In examining this
relationship, we excluded instances where the chatbot “admitted defeat” since,
as argued earlier, it makes little sense to call a reference that was not
provided at all “incomplete.” For GPT-3.5, the relationship between Topic
Broadness and Incompleteness was linear: As topics became broader, it was less
likely to give incomplete references. Specifically, for GPT-3.5, logistic
regression revealed that Topic Broadness significantly predicted the frequency
of Incomplete citations ($\beta=-0.83,SE=0.114,z=-7.26,P<0.001$);
$\textit{Pseudo-}R^{2}=0.1536$. Adding the squared term of Broadness to the
model caused the linear term to lose significance, and did not substantially
improve model fit, suggesting a linear model is superior. Specifically, the
curvilinear model was as follows: Broadness
($\beta=0.59,SE=0.573,z=1.03,P=0.304$), Broadness Square
($\beta=-0.27,SE=0.111,z=-2.44,P=0.015$); $\textit{Pseudo-}R^{2}=0.1695$. For
GPT-4, a different pattern arose. A logistic regression revealed that Topic
Broadness ($\beta=0.01,SE=0.137,z=0.11,P=0.916$) did not significantly predict
the frequency of Incomplete citations; $\textit{Pseudo-}R^{2}=0.0000$.
However, adding the square term led to a clean and predictive model: Broadness
($\beta=9.07,SE=1.931,z=4.70,P<0.001$), Broadness Square
($\beta=-1.28,SE=.269,z=-4.77,P<0.001$); $\textit{Pseudo-}R^{2}=0.1434$. For
GPT-4, the relationship between Topic Broadness and Incomplete references
appears to be curvilinear. These analyses, while statistically powerful, were
highly exploratory, and so should be interpreted with caution. Table S2 shows
what the observed effects look like in practice.

Table S2: Incomplete References by GPT Version and Topic Broadness

Topic Broadness |  Total References (GPT-3.5) |  Incomplete References (GPT-3.5) |  Total References (GPT-4) |  Incomplete References (GPT-4)
---|---|---|---|---
Broad |  100 |  4 (4.00%) |  100 |  2 (2.00%)
Somewhat Broad |  100 |  0 (0.00%) |  100 |  24 (24.00%)
Moderate |  100 |  20 (20.00%) |  100 |  20 (20.00%)
Somewhat Narrow |  100 |  40 (40.00%) |  85 |  2 (2.35%)
Narrow |  60 |  21 (35.00%) |  5 |  0 (0.00%)

  * •

Notes: Numbers exclude admissions of defeat. Numbers in parentheses reflect
the percentage of Total References that were Incomplete.

Acknowledgment of Hallucination. These make use of all references (totaling
1,269) generated by ChatGPT. As mentioned in the main article, GPT-4 showed
both lower rates of fabrication and greater likelihood of acknowledging these
fabrications, relative to GPT-3.5. GPT-3.5 generated significantly more
fictional references (39.05%, 95% CI [34.86, 43.23]) than GPT-4 (23.12%, 95%
CI [20.08, 26.15]); OR = 2.13, 95% CI [1.67, 2.72], z = 6.06, P < 0.001. When
GPT-4 generated fictional references, it noted so 84.30% of the time (95% CI
78.81, 89.79) compared to 12.20% (95% CI 7.68, 16.71) for GPT-3.5; OR = 38.67,
95% CI [21.51, 69.50], z = 12.22, P < 0.001. The same pattern did not arise to
a statistically significant degree for errors, as seen in Table S3. Note that,
analogous to acknowledged fabrications, here we are crediting ChatGPT for
acknowledging errors if the error appeared in a chat where it more generally
stated that references might be problematic (e.g. “fictional”).

Table S3: Acknowledgment of Fictional References and Errors

Model |  Total References |  Total Errors |  Acknowledged Errors |  Total Fictional References |  Acknowledged Fictional References
---|---|---|---|---|---
GPT-3.5 |  525 |  21 (4.00%) |  0 (0.00%) |  205 (39.05%) |  25 (12.20%)
GPT-4 |  744 |  26 (3.49%) |  3 (11.54%) |  172 (23.12%) |  145 (84.30%)
Significance |  |  P = 0.639 |  P = 0.112 |  P < 0.001 |  P < 0.001

  * •

Notes: The P-value for acknowledging errors is from linear regression since
the zero acknowledged errors for GPT-3.5 disallow logistic regression. Other
P-values are from logistic regression. For Total Errors and Fictional
References, parenthetical numbers are their percentages of total articles. For
Acknowledged Errors and Fictional References, parenthetical numbers are of
Total Errors and Fictional References respectively.

Effects of Topic Broadness on Hallucination. We showed an effect that appeared
to be curvilinear, whereby ChatGPT tended to hallucinate more as the research
topics became narrower, but then hallucinated less on the narrowest topics. We
also showed, however, that this downward trend at the narrowest level was
driven by a greater tendency for ChatGPT to “admit defeat” as opposed to lower
hallucination rates when providing actual articles. Here, we begin by
providing further details of the analysis reported in the main manuscript,
whereby when we collapse across GPT models, a logistic regression showing the
linear effect of topic broadness on hallucination becomes stronger after
excluding admissions of defeat. With admissions of defeat included in the
analysis, the effect of topic broadness is significant but rather weak
($\beta=-0.191,\text{95\% CI [-0.301,
-0.081]},z=-3.40,P=0.001;\textit{Pseudo-}R^{2}=0.0115)$. Once these admissions
of defeat are excluded, this effect becomes stronger ($\beta=-0.555,\text{95\%
CI [-0.691, -0.419]},z=-8.01,P<0.001;\textit{Pseudo-}R^{2}=0.0750)$.

Next, we report three logistic regressions with Hallucination as the dependent
variable and the independent variables broadness and broadness-squared, to
test the significance of the apparent curvilinear effects of topic broadness
on hallucination. Collapsing across versions of GPT, Broadness
($\beta=1.175,SE=0.300,z=3.92,p<0.001)$ and Broadness-squared
($\beta=-0.236,SE=0.051,z=-4.61,P<0.001)$ were both significant predictors;
$\textit{Pseudo-}R^{2}=0.0338$. Limiting the analysis to GPT-3.5, the same
curvilinear pattern emerges, with Broadness
($\beta=0.957,SE=0.357,z=2.68,P=0.007)$ and Broadness-square
($\beta=-0.225,SE=0.061,z=-3.67,P<0.001)$ as significant;
$\textit{Pseudo-}R^{2}=0.0618$. Similarly, limiting the analysis to GPT-4,
Broadness ($\beta=9.526,SE=2.664,z=3.58,P<0.001)$ and Broadness-square
($\beta=-1.339,SE=0.372,z=-3.60,P<0.001)$ are significant;
$\textit{Pseudo-}R^{2}=0.1772$.

These curvilinear effects become less apparent for ChatGPT as a whole, and for
GPT-3.5 specifically (but not for GPT-4) once we remove those references
credited as “Correct” because ChatGPT acknowledged its lack of knowledge of
any relevant articles. Specifically, collapsing across versions of GPT but
removing admissions of defeat, Broadness
($\beta=-0.623,SE=0.348,z=-1.79,P=0.074)$ but not Broadness-square
($\beta=0.011,SE=0.056,z=0.20,P=0.843)$ trends toward significance, with their
directions reversed. (Note though, as described above, that once admissions of
defeat are removed, a linear effect is however robust and has a stronger model
fit.)

When conducting this regression using only data from GPT-3.5, neither
Broadness ($\beta=-0.250,SE=0.404,z=-0.62,P=0.536)$ nor Broadness-square
($\beta=-0.058,SE=0.066,z=-0.88,P=0.381)$ is significant;
$\textit{Pseudo-}R^{2}=0.1019$. However, there is a robust effect
($\beta=-0.598,SE=0.081,z=-7.37,P<0.001)$ of Broadness alone on Hallucination
for GPT-3.5 once we exclude admissions of defeats;
$\textit{Pseudo-}R^{2}=0.1007$. After this exclusion, this simple linear
effect thus appears to be the more appropriate analysis for GPT-3.5.

Curiously, this same pattern does not hold for GPT-4. Limiting the analysis to
this model and to references that were not admissions of defeat, the effect of
topic Broadness has no simple linear relationship with Hallucination
($\beta=0.025,SE=0.177,z=0.14,P=0.887;\textit{Pseudo-}R^{2}=0.0001)$. Instead,
the curvilinear effect observed earlier remains for GPT-4, even after
excluding admissions of defeat. Specifically, Broadness
($\beta=9.238,SE=2.702,z=3.42,P=0.001)$ and Broadness-square
($\beta=-1.301,SE=0.376,z=-3.46,P=0.001)$ are both significant;
$\textit{Pseudo-}R^{2}=0.1205$. Thus, if there is indeed an effect of topic
broadness on GPT-4’s hallucination, it is curvilinear. This exploratory result
should however be treated with caution. On one chat/topic (Implicit Attitudes
and the Law), which was at the second highest broadness level, GPT-4
hallucinated significantly. This result may therefore be outlier-driven.
However, it is also possible that GPT-4 indeed tended to first hallucinate
more as topics narrowed, but then less as topics became very narrow. One
speculative interpretation of this curvilinear effect is related to GPT-4’s
evolving capacity to acknowledge fiction, which was reported in the main
paper. Here, let us imagine that GPT has some representation of its inability
to come up with relevant articles in the narrowest domains. It may thus become
more likely to implement tactics that avoid hallucination, including
acknowledging defeat and presenting articles on broader topics that, though
not relevant, are also not hallucinations.

Table S4 displays what these patterns look like in practice.

Table S4: Hallucination & Topic Broadness by GPT-Version, Excluding Admissions
of Defeat

Topic Broadness |  GPT-3.5 References (Excluding Defeat Admissions) |  GPT-3.5 Hallucinations |  GPT-4 References (Excluding Defeat Admissions) |  GPT-4 Hallucinations
---|---|---|---|---
Broad |  100 |  20 (20%) |  100 |  0 (0%)
Somewhat Broad |  100 |  17 (17%) |  100 |  17 (17%)
Moderate |  100 |  44 (44%) |  100 |  8 (8%)
Somewhat Narrow |  100 |  64 (64%) |  85 |  2 (2.4%)
Narrow |  60 |  35 (58.3%) |  5 |  0 (0.00%)

  * •

Notes: Numbers in parentheses reflect the percentages of references (excluding
admissions of defeat) that are hallucinations.

Topic Broadness and Reference Relevance. In the main manuscript, we reported
that GPT-4 was, overall, more likely than GPT-3.5 to pull relevant references,
but that this effect was primarily driven by GPT-4’s lower hallucination rate.
We begin by reporting these analyses in more detail. We examined this question
series of logistic regressions. When including all 1,000 articles from our
main data, GPT-4 was more likely to pull relevant articles (50.80%, 95% CI
[46.40, 55.20]) compared to GPT-3.5 (30.80%, 95% CI [26.74, 34.86]), with
logistic regression indicating a significant difference; OR = 2.32, 95% CI
[1.79, 3.00], z = 6.38, P < 0.001. Conversely, when limiting the analysis only
to articles labeled as “real,” the difference in identifying relevant articles
between GPT-4 (53.70%, 95% CI [49.19, 58.21] and GPT-3.5 (48.13%, 95% CI
[42.62, 53.63]) lost significance; OR = 1.25, 95% CI [0.94, 1.66], z = 1.54, P
= 0.124.

This suggests, as noted in the main article, that GPT-4 discovered more real
articles than 3.5 more generally, and therefore was also more likely to
discover relevant articles. Beyond this, however, it did not tailor article
recommendations better to specific topics. However, when we exploratorily
examine this by topic breadth, the data tell a more nuanced story. By design,
the Research Librarian task was somewhat easy at the highest broadness level:
There were many influential papers, likely familiar to GPT, that met our
criteria. On the other extreme, the narrowest subject areas were intentionally
difficult. For example, few articles use the Implicit Association Test to
predict dental outcomes, and they may justifiably be absent in GPT’s training
data. The areas between these extremes – neither designed to be too easy nor
too difficult – were thus of greater interest.

When limiting the analysis to non-hallucinations, GPT-4 shows relative
improvement in finding relevant articles on Somewhat Broad and Moderate
topics, with GPT-3.5’s success at generating relevant references dropping off
more sharply as topics become narrower. As shown in Table S5, and reported in
the main article, the difference between GPT-4 and GPT-3.5 on this count is
significant for topics of moderate broadness, but not for topics at other
broadness levels once correcting for multiple comparisons. (Since we are
running 5 exploratory analyses here, a Bonferroni correction dictates that our
significance level should be P < 0.01.)

Table S5: Hallucination & Topic Broadness by GPT-Version, Excluding Admissions
of Defeat

Broadness |  GPT-3.5 Proportion of Non-Hallucinations marked Relevant |  GPT-4 Proportion of Non-Hallucinations marked Relevant |  Odds Ratio |  P-value
---|---|---|---|---
Broad |  75/80 (93.8%) |  99/100 (99.0%) |  6.60 |  0.088
Somewhat Broad |  44/83 (53.0%) |  55/83 (66.3%) |  1.74 |  0.083
Moderate |  18/56 (32.1%) |  69/92 (75.0%) |  6.33 |  < 0.001
Somewhat Narrow |  14/36 (38.9%) |  31/98 (31.6%) |  0.73 |  0.431
Narrow |  3/65 (4.62%) |  0/100 (0.0%) |  0 |  0.03

  * •

Note: The P-value for Narrow topics is drawn from linear regression since
GPT-4’s 0% disallows logistic regression. All other P-values are drawn from
logistic regressions.

We also use logistic regression analysis to examine the significance of the
difference in the curvilinear effect by GPT-Version. Because this analysis was
exploratory, we ran the relevant regression analysis several ways as a
robustness check. The first analysis begins by excluding hallucinations, since
(as described above) not doing so tends to overestimate GPT-4’s improvement in
generating relevant articles. It captures a 3-way interaction, using logistic
regression with Relevant (vs. Not Relevant) as the dependent variable, and 5
independent variables: GPT Version (4 vs. 3.5), Broadness, Broadness-square,
GPT Version x Broadness, GPT Version x Broadness-square. The independent
variables of particular interest here are GPT Version x Broadness and GPT
Version x Broadness-square, since their significance indicates a different
curvilinear relationship dependent on the version of GPT, in this case the
fact that GPT-3.5’s success (relative to GPT-4’s) at locating relevant
articles drops off more quickly as categories narrow. These coefficients are
robustly significant. Specifically, the model is as follows: GPT Version
($\beta=-3.590,SE=1.197,z=-3.00,P=0.003)$, Broadness
($\beta=-0.010,SE=0.556,z=-0.02,P=0.985)$, Broadness-square
($\beta=0.177,SE=0.091,z=1.95,P=0.051)$, GPT Version x Broadness
($\beta=2.947,SE=0.824,z=3.58,P<0.001)$, and GPT Version x Broadness-square
($\beta=-0.430,SE=0.131,z=-3.27,P=0.001)$; $\textit{Pseudo-}R^{2}=0.3631$.
Next, instead of removing hallucinations we statistically controlled for them
by adding them as a sixth independent variable in the regression. Though
logistic regression is statistically preferable given the dichotomous
variable, the perfect alignment between hallucination and irrelevance
disallows this analysis, and so this was run as a regular linear regression.
The critical interaction terms once again achieved robust significance in this
regression. Specifically, the model is as follows: GPT Version
($\beta=-0.585,SE=0.108,t(993)=-5.41,P<0.001)$, Broadness
($\beta=-0.015,SE=0.058,t(993)=-0.26,P=0.794)$, Broadness-square
($\beta=0.026,SE=0.010,t(993)=2.72,P=0.007)$, GPT Version x Broadness
($\beta=0.420,SE=0.082,t(993)=5.15,P<0.001)$, and GPT Version x Broadness-
square ($\beta=-0.055,SE=0.013,t(993)=-4.15,P<0.001)$, Hallucination vs. not
($\beta=-0.435,SE=0.031,t(993)=-14.12,P<0.001)$; $R^{2}=0.4871$. Third, we ran
the initial logistic regression but added Incomplete (vs. complete) as a
further independent variable. Again, the relevant interaction terms were
robustly significant. Specifically, the model is as follows: GPT Version
($\beta=-3.446,SE=1.214,z=-2.84,P=0.005)$, Broadness
($\beta=0.013,SE=0.558,z=0.02,P=0.982)$, Broadness-square
($\beta=0.175,SE=0.091,z=1.92,P=0.055)$, GPT Version x Broadness
($\beta=2.854,SE=0.834,z=3.42,P=0.001)$, GPT Version x Broadness-square
($\beta=-0.417,SE=0.133,z=-3.15,P=0.002)$, and Incomplete
($\beta=0.256,SE=0.379,z=0.67,P=0.500)$; $\textit{Pseudo-}R^{2}=0.3635$.
Fourth, we ran a second linear regression with the same additional IV
(Incomplete vs. Complete). Again, the relevant terms were highly significant.
Specifically, the model was: GPT Version
($\beta=-0.574,SE=0.109,t(992)=-5.27,P<0.001)$, Broadness
($\beta=-0.015,SE=0.058,t(992)=-0.26,P=0.794)$, Broadness-square
($\beta=0.026,SE=0.010,t(992)=2.74,P=0.006)$, GPT Version x Broadness
($\beta=0.414,SE=0.082,t(992)=5.04,P<0.001)$, GPT Version x Broadness-square
($\beta=-0.055,SE=0.013,t(992)=-4.07,P<0.001)$, Hallucination vs. not
($\beta=-0.444,SE=0.033,t(992)=-13.49,P<0.001)$, and Incomplete
($\beta=0.028,SE=0.037,t(992)=0.75,P=0.456)$; $R^{2}=0.4874$. We thus find
that independent of modeling choices, the curvilinear interaction is highly
significant: GPT-3.5 drops off more quickly than GPT-4 in terms of finding
relevant articles as the topics narrow.

Relevant Citation Counts. In the main article, we reported that focusing only
on relevant articles, GPT-4’s selections did not have higher citation counts
than those of GPT-3.5. Here, we begin by reporting the overall effects in
greater detail. As noted in the main article, the two GPT models performed
similarly here, with GPT-4’s articles averaging 2,936.7 (95% CI [2128.3,
3745.1]) citations compared to 3,105.2 (95% CI [2205.5, 4005.0]) for GPT-3.5;
t(406) = 0.2653, P = 0.7909, d = 0.027. It is possible, however, that a
significant effect on this count was disguised by other factors. We reported,
for example, that GPT-4 generated more relevant citations for somewhat broad
and moderate topics. By pulling relevant articles in narrower domains, GPT-4
might be limited to less influential articles. In short, once we controlled
for topic broadness, GPT-4 might have selected articles with higher citation
counts. However, a regression analysis ruled out this possibility.
Specifically, we limited our analysis to articles marked relevant and ran a
simple linear regression with citation count as the dependent variable and
GPT-4 (vs. GPT-3.5) and Topic Broadness as independent variables. While topic
broadness was associated with higher citation count, the coefficient for GPT-
version remained statistically insignificant in this model. Specifically, the
model was: GPT Version ($\beta=218.38,SE=618.63,t(405)=0.35,P=0.724)$,
Broadness ($\beta=1512.33,SE=280.10,t(405)=5.40,P<0.001);R^{2}=0.0673$.

Relatedly, GPT-4 might have proved superior at finding references that were
important but also more recent. Such references would however have had less
time to gather citations. To test for this possibility, we coded the year for
the articles. We first ran a simple regression with article Year as the
dependent variable and GPT-Version (4 vs. 3.5) as an independent variable,
again limiting the analysis to only articles coded as “Relevant.” This yielded
a modest but significant effect, the model being: GPT Version
($\beta=1.850,SE=0.821,t(406)=2.25,P=0.025);R^{2}=0.0124$. This result
suggests that GPT-4 pulled more recent articles than GPT-3.5. However, this
result should be treated with caution for several reasons. First, the analysis
was exploratory. Second, the effect was small. Third and most importantly, the
effect disappears once we control for topic broadness. Adding this as an
additional IV yields the model: GPT Version
($\beta=1.059,SE=0.755,t(405)=1.40,P=0.161)$, Broadness
($\beta=-3.091,SE=0.342,t(405)=-9.05,P<0.001);R^{2}=0.1784$. It appears that
if GPT-4 indeed pulled slightly more current articles, this pattern was
related to its tendency to better generate relevant references on narrower
topics.

While not confident in the result that GPT-4 pulled more recent articles, we
nevertheless ran a set of regressions to confirm that article aging was not
disguising an effect of GPT-version on Citation Count. Toward this end, we ran
two regression analyses. Again, limiting the analysis to articles coded as
“Relevant,” we first ran a regression with Citation Count as the dependent
variable, and GPT-Version (4 versus 3.5) and article Year as independent
variables. This again yielded an insignificant effect of GPT-Version on
Citation Count, the model being: GPT Version
($\beta=737.23,SE=495.87,t(405)=1.49,P=0.138)$, Article Year
($\beta=-489.66,SE=29.80,t(405)=-16.43,P<0.001);R^{2}=0.4001$. Second, we ran
a version of the regression that included both Topic Broadness and Year as
covariates alongside GPT-Version. Once again, we see no significant partial
effect of GPT-Version on article Citation Count. Specifically, the model was:
GPT Version ($\beta=737.03,SE=497.96,t(404)=1.48,P=0.140)$, Broadness
($\beta=-1.26,SE=246.60,t(404)=-0.01,P=0.996)$, Article Year
($\beta=-489.73,SE=32.71,t(404)=-14.97,P<0.001);R^{2}=0.4001$.

## Section S3: GPT as Research Ethicist – Detailed Methods

Full Research Design. In Study 2, we presented GPT-3.5 and GPT-4 with
fictional vignettes describing flawed research protocols, posing as scientists
looking for feedback on their work. These vignettes were very loosely based on
real research projects that have proven difficult to replicate. For each, we
described a series of plausible but non-obvious studies, but purposefully
inserted evidence of 4 statistically unsound and ethically questionable
practices. Three of these vignettes contained poor practices that were
blatant. The researchers openly described a litany of scientific sins, such as
examining data for statistical significance before deciding whether to
continue collection, making post-hoc data exclusions that drew out significant
effects, framing exploratory findings as if predicted in advance, and running
many exploratory analyses but presenting only significant results without
correcting for multiple comparisons.

Three further vignettes presented similar fictional protocols but indicated
these problematic decisions more subtly. For example, the first matched pair
of vignettes described a protocol to test for “emotional clairvoyance”
(emotional contagion from interactions out of participants’ earshot). In the
blatant version of this vignette, the researchers say, “after just 30
participants in each condition, effects already reached statistical
significance (p<.05), so we stopped data collection…” In the subtler version
of this vignette, the researchers “collected 50 participants in each
condition, at which point statistical analysis indicated that our results
reached statistical significance (p<.05).” They do not directly mention using
this test to decide whether to collect more subjects, but later state that “In
the second study, we were worried we wouldn’t always see such a large effect
size, so we decided to collect data from 150 participants.” Though there is
nothing intrinsically wrong with collecting a larger sample in one study
versus another, one familiar with problems of p-hacking might reasonably be
suspicious. If they weren’t stopping and starting collection, why was this
decision made?

The purpose of this variation between blatant and subtle vignettes was to
test, 1) whether GPT showed awareness of the clear methodological problems
described in the blatant vignettes, and 2) whether it would be able to read
between the lines and recognize violations in more realistically described
research. In effect, the subtle vignettes were more ecologically valid
descriptions, designed to better resemble how a real-world researcher might
represent flawed research.

Each of our 6 vignettes was presented 36 times (18 times each to GPT-3.5 and
GPT-4). As detailed later, we exploratorily varied the initial prompt we used
to request ChatGPT’s feedback on these protocols. In addition, we ran a
secondary study (described later) where ChatGPT responded to versions of two
of these vignettes that contrastingly described pristine ethics and
statistical practices.

For each vignette in the main study, a rubric was created, containing 10
points we wanted to see GPT highlight in each chat. Four of the ten points
were for recognizing problematic research practices. For each of these four, a
second point could be received by suggesting a practice that remedied the
problem. GPT might receive these “solution points” for a sufficiently strong
statement of the problem. For example, GPT might generate a response like “You
have started and stopped collection part way through to test for significance.
This is a form of p-hacking and should not be done since it inflates the
likelihood of false positive results.” The solution in this case is intrinsic
to the statement of the problem: Do not do this. This line would therefore
receive points for both the statement of the problem and the solution. On the
other hand, if GPT said something like “You should decide your sample size in
advance based on a power analysis,” this would receive credit for proposing a
solution, but in the absence of further statements would not receive credit
for pointing out the problem. In addition to these 8 possible points, GPT was
credited if it suggested 1. Pre-registration or other firm pre-research
decisions, and 2. Replication.

Detail on vignettes and their coding. The full vignettes and the rubrics used
to code them may be found in the document “GPT as Research Ethicist Materials
20240227” at https://osf.io/sdahr/. Each vignette was written by the authors
and designed to contain evidence (blatant or subtle) of 4 substandard and
ethically questionable research practices. The first matched pair of vignettes
contained descriptions of research on “Emotional Clairvoyance,” in which
participants supposedly experienced emotional contagion from interactions that
took place out of earshot. The second set of vignettes described studies of
“Atmospheric Cognition,” in which subtle characteristics of the space one is
in impact mental states and behaviors. The third set of vignettes described an
econometrical study, using data from a large dating site to demonstrate that
more extroverted individuals are perceived as less attractive. Table S6
displays a list of the problematic practices found in these vignettes, and how
they varied between the Blatant and Subtle Versions.

Table S6. Details of ethical/methodological violations described in vignettes

|  Blatant Version |  Subtle Version
---|---|---
Vignette 1a & 1b: Emotional Clairvoyance
Issue 1: Stopping & Starting Data Collection to Test Significance |  • Tested for significance and then stopped collection after 30 subjects. • Later, collected additional subjects when significance was not initially reached. |  • Collected notably different sample sizes in different studies. • In final study, indicated “we decided to collect additional data,” with ambiguity around whether significance testing played a role.
Issue 2: Testing Several Moderators w/o Correction Multiple Comparisons |  • Tested the Big 5 personality traits as moderators, without mentioning correction for multiple comparisons. |  • Tested the Big 5 personality traits as moderators, without mentioning correction for multiple comparisons.
Issue 3: Hypothesizing After Results are Known (HARKing) |  • Says moderator from exploratory analysis makes intuitive sense and will be reported as what they were testing at the outset. |  • Says moderator from exploratory analysis makes intuitive sense but does not indicate whether or not it will be framed as exploratory.
Issue 4: Selective Reporting |  • Attains null results in two studies with slightly different conditions. Indicates they will not publish them. |  • Attains null results in one study with slightly different conditions. Indicates uncertainty about the study, and that they may “focus” on the other conditions.
Vignette 2a & 2b: Atmospheric Cognition
Issue 1: Suppressing Unsuccessful Pilot Studies |  • Says they piloted “a bunch of versions” with initially mixed results. • In the main version, piloted a couple of versions that were too subtle, which they therefore “ignored.” |  • Says they initially piloted a “few versions” with initially mixed results. Whether they will share these mixed results is left ambiguous.
Issue 2: Post-hoc Exclusion of Participants to Achieve Significance |  • Said they removed math majors who “skewed the results,” only after which results achieved significance. |  • Said they noticed and removed math majors who had an advantage and were “unevenly assigned” and that results were significant.
Issue 3: Adding Further Subjects When Significant Results are Not Attained |  • Openly said that when they didn’t see an effect after 100 participants in Study 3, they added 50 more to reach significance. |  • Said they realized they would need more subjects in Study 3, and so collected 50 more. Does not directly indicate they tested significance.
Issue 4: Reporting Marginal Results as if They Reached Significance |  • Said they achieved a significance level of p=.052, “which rounds down to .05”. |  • Said they achieved a significance level of p=.052, “which is so close to the p=.05 threshold as to be statistically significant for most…purposes.”
Vignette 3a & 3b: Bad Econometrics
Issue 1: Suppressing Outcome Variables That Yielded Null Results |  • Because two alternative DVs didn’t yield significant results, they “ignored these” and feel they can be “omitted from the report.” |  • Since two alternative DVs didn’t yield significant results, they were “downplayed,” and the report will “focus” on the third.
Issue 2: Excluding Parts of Data that Did Not Yield Significant Results |  • Exploratorily limited analyses to people under 60 and who live in specific regions, because this yielded stronger results. |  • Exploratorily limited analyses to people under 60, the primary demographic of the website, which made results stronger.
Issue 3: Cycling Through Covariates to Attain Results, w/o Corrections for Multiple Comparisons |  • Indicated that they had the “Big 5 and dozens of subtler measures,” and later “examined various potential covariates” to find significant model. |  • Indicated that they had the “Big 5 and subtler measures,” and later that they “examined potential covariates.”
Issue 4: Violating Assumptions Around Multicollinearity & Overfitting |  • Includes two variables with a high likelihood of multicollinearity (Shyness, Self-Confidence) in model to show significant effect of Extroversion. |  • Includes one variable with potential for multicollinearity (Assertiveness) in model to show significant effect of Extroversion.

The same rubrics were used for scoring the Blatant and Subtle versions of
these vignettes. These rubrics were as follows, with GPT given a point by the
coders for each affirmative answer.

Coding Rubrics for Scientific Ethicist Study. Below are the full rubrics used
to code responses to the Scientific Ethicist vignettes.

Rubric for Vignettes 1a & 1b:

1\. Did it draw attention to the issue of making decisions about data
collection (stopping/starting) by examining data part-way through collection?

2\. Did it provide advice (e.g. power analysis, sticking to a pre-decided
sample size, not testing part-way through collection) that mitigates the
issues around stopping/starting data collection based on data examination
during collection?

3\. Did it draw attention to the issue of testing different potential
variables/moderators, creating problems of multiple comparisons or increasing
potential for type 1 errors?

4\. Did it provide advice (e.g. corrections for multiple comparisons, deciding
hypothesized moderators in advance) that mitigates the issues around multiple
comparisons?

5\. Did it draw attention to the issue of HARKing or otherwise ignoring the
exploratory nature of certain findings?

6\. Did it provide advice (e.g. clear delineation of exploratory analyses,
pre-registering hypotheses) that mitigates the problem of HARKing?

7\. Did it draw attention to the problem of suppressing null results in
reporting?

8\. Did it provide advice (e.g. reporting all results) that mitigates the
issue of suppressing null results?

9\. Did it suggest further replication of results?

10\. Did it suggest pre-registration or other forms of commitment to pre-
decided methodological strategies?

Rubric for Vignettes 2a & 2b:

1\. Did it draw attention to the issue of ignoring or suppressing pilot
studies?

2\. Did it provide advice (e.g. discussing pilot research, publishing all
results) that mitigates the issue of ignoring or suppressing pilot studies?

3\. Did it draw attention to the issue of making inappropriate post hoc
exclusions of participants?

4\. Did it provide advice (e.g. not making exclusions, deciding exclusions in
advance) that mitigates the issue of inappropriate post hoc exclusions?

5\. Did it draw attention to the issue of adding further subjects to studies
when statistically significant results are not initially attained?

6\. Did it provide advice (e.g. power analysis, sticking to pre-decided
sample, not testing part-way through collection) that mitigates the issue of
adding subjects to a study when not seeing significant results?

7\. Did it draw attention to the issue of reporting marginal results as if
they’d met thresholds for significance?

8\. Did it provide advice (e.g. reporting exact p-values, not rounding down)
that mitigates the issue or reporting marginal results as if they were
significant?

9\. Did it suggest further replication of results?

10\. Did it suggest pre-registration or other forms of commitment to pre-
decided methodological strategies?

Rubric for Vignettes 3a & 3b:

1\. Did it draw attention to the issue of ignoring or suppressing outcome
variables that didn’t lead to favorable results?

2\. Did it provide advice (e.g. reporting on all possible outcome variables)
that mitigates the issue of ignoring or suppressing unfavorable outcome
variables?

3\. Did it draw attention to the issue of excluding chunks of data that did
not fit results?

4\. Did it provide advice (e.g. including all data, deciding exclusions
a-priori and/or clearly reporting and defending exploratory exclusions) that
mitigates the issue of inappropriately excluding data?

5\. Did it draw attention to the issue of cycling through many potential
covariates until finding ones that yielded desirable results?

6\. Did it provide advice (e.g. corrections for multiple comparisons,
hypothesizing covariates in advance) that mitigates the issue of examining
many potential covariates?

7\. Did it draw attention to the issue of potentially violating statistical
assumptions (e.g. issues around overfitting, multicollinearity)?

8\. Did it provide advice (e.g. split samples, checks for multicollinearity)
that mitigates issues around potentially violating statistical assumptions?

9\. Did it suggest replications of results in other data or contexts and/or
urge caution about result generalizability?

10\. Did it suggest pre-registration or other forms of commitment to pre-
decided methodological strategies?

Coding of Vignettes. Responses to the vignettes were blinded by prompt
condition. Where possible, content that would make the initial prompt obvious
(but was otherwise irrelevant) was removed. Coders were also blinded to
whether responses were given by GPT-3.5 or GPT-4, as well as to whether each
chat was in response to a blatant or subtle version of the vignette. GPT’s
responses were placed in groups coinciding with the topic of the vignette but
were otherwise presented in a randomized order.

Two coders independently rated each GPT response on each of the 10 points from
the relevant rubric. Points of disagreement were discussed. Following
discussion, responses were averaged, with GPT receiving partial credit in
cases where one but not the other coder felt a point was merited. Inter-rater
coding achieved a high degree of reliability, with a Cronbach alpha of
$\alpha$ = 0.9827. Of 2,160 possible points GPT could receive, the two coders
disagreed on 34 ( 1.57%).

One coder was an author, and the other was a Research Assistant who was kept
blind to the full purpose of the research until after completion. Responses
were first coded independently by each coder. Points of disagreement were then
discussed, with the coder who was not an author encouraged to lead discussions
to minimize pressure to conform to the author’s decision. Though the rubrics
were decided in advance, certain points needed to be clarified early in the
coding. Specifically, after discussing a few instances of question 10 (pre-
registration or other firm methodological commitments), the coders made the
decision that credit would be given for either recommending preregistration or
making at least two other recommendations around committing to decisions in
advance. This was to prevent GPT from too easily receiving credit for the same
point multiple times. For example, recommending a power analysis to decide on
sample size in advance received credit for q2 in Vignettes 1a-b. To also
receive credit for q10, GPT would need to make a second point around deciding
methodologies in advance. For q7 in Vignettes 3a-b, two decisions were made.
First GPT was credited for suggesting that the vignette’s interpretation of
their effects might be flawed due to complicated relationships (e.g. mediating
roles) between the two variables, since this point was conceptually very
similar to the one we wished it to make: The fundamental problem was the
complex and likely strong relationships between the different independent
variables. On the other hand, GPT was not credited on this question for
answers highlighting statistical assumptions that were not clearly violated
(e.g. homoskedasticity, linearity), as such observations seemed generic and
did not acknowledge the fundamental problem (overfitting, multicollinearity)
in the regression analysis.

Variation in Initial Prompts. As discussed in the main text, we varied the
initial prompt used when initiating each chat with GPT, to test whether this
would impact GPT’s performance on reviewing the vignettes. In this section, we