# Large Language Models are Biased Because
They Are Large Language Models

Philip Resnik
Department of Linguistics and
Institute for Advanced Computer Studies
University of Maryland
College Park, Maryland USA
<EMAIL_ADDRESS>
###### Abstract

This paper’s primary goal is to provoke thoughtful discussion about the
relationship between bias and fundamental properties of large language models.
We do this by seeking to convince the reader that harmful biases are an
inevitable consequence arising from the design of any large language model as
LLMs are currently formulated. To the extent that this is true, it suggests
that the problem of harmful bias cannot be properly addressed without a
serious reconsideration of AI driven by LLMs, going back to the foundational
assumptions underlying their design.

This paper is under review. I don’t generally participate in social media
conversations, since I find social media a poor venue for real discussion, but
private feedback is welcome at<EMAIL_ADDRESS>Unfortunately I
cannot promise I will be able to respond.

I expect that in the time it takes to review this paper, other relevant and
important papers will have come out that need to be added to the discussion.
For example, one that already has, relevant for Section 5, is: Chen, A., et
al. (2024). Preference learning algorithms do not learn preference rankings.
arXiv. https://arxiv.org/abs/2405.19534.

##  1 Introduction

_A scorpion wanted to cross a river, but it could not swim, so it asked a frog
to carry it across. The frog feared that the scorpion would sting it, but the
scorpion promised it would not, pointing out that it would drown if it did.
Midway across the river, the scorpion stung the frog, dooming them both to
die. “Why have you done this?!”, cried the frog. “I’m sorry”, said the
scorpion, “It’s in my nature”._ 111The Scorpion and the Frog. Paraphrased from
Wikipedia (2024b), which cites Nitoburg (1933, pp. 232-233).

For all their power and potential, large language models (LLMs) come with a
big catch: they contain harmful biases that can emerge unpredictably in their
behavior. Efforts to remove or mitigate LLMs’ biases are a hot topic of
research (Gallegos et al., 2024), but such efforts have not yet met with
anything resembling decisive success, and apparent successes can leave the
same problem to emerge in other ways (Hofmann et al., 2024).

We take the position that this problem will not and cannot be solved without
facing the fact that harmful biases are thoroughly baked into what LLMs are.
There is no bug to be fixed here. The problem cannot be avoided in large
language models as they are currently conceived, precisely _because_ they are
large language models.

This paper attempts to argue that position carefully and thoroughly. Our
first, primary goal is to provoke thoughtful discussion about the relationship
between bias and fundamental properties of language models. To the extent that
readers recognize this as an important issue worth conversation or even
debate, the paper’s goal has been met, whether or not any individual reader
agrees with the specific argument being made here in its entirety.

Secondarily, we seek to convince the reader that bias is such a direct
consequence of the design of current large language models that it cannot be
avoided in LLMs as they currently exist, despite the seriousness of the
problem and the diligent efforts of well intentioned, smart, capable
researchers. To the extent that this is true, it suggests a serious
reconsideration of what it means to prevent bias-related harm in the creation
and deployment of LLMs.

##  2 What are large language models models of?

Language models are probabilistic models of languages that consist of
observable strings of symbols. To state this a bit more formally:

  * •

A _language_ ${\cal L}$ in this context is formalized as a probability
distribution $\Pr_{{\cal L}}({\bf w})$ over sequences of symbols ${\bf
w}=w_{1}\ldots w_{n}$. For intuitive convenience we’ll refer to these
sequences as _texts_ and to the $w_{i}$ as _words_.222The term _language
model_ in this sense originated with IBM’s pioneering work on speech
recognition (Jelinek et al., 1975), and the technical concept dates back to
Shannon (1948), and, before that, to Markov (1913). It should not be confused
with other phrases denoting models of language. We draw on the exposition in
Manning and Schütze (1999) but with some adjustments to notation and
terminology; e.g. they refer to sequences ${\bf w}$ as _utterances_.

  * •

A _language model_ $M$ for a language ${\cal L}$ is a probability distribution
$\Pr_{M}({\bf w})$ created with the goal of approximating $\Pr_{{\cal L}}({\bf
w})$.333Some readers may be confused by the characterization of a language
model as a distribution over entire strings, as opposed to a model that makes
word predictions based on context. This confusion is resolved by the chain
rule of probability: $\Pr(w_{1}w_{2}w_{3}\ldots
w_{n})=\Pr(w_{1})\Pr(w_{2}|w_{1})\Pr(w_{3}|w_{1}w_{2})\ldots\Pr(w_{i}|w_{1}\ldots
w_{i-1})$ Manning and Schütze (1999).

  * •

What makes a language model “good” is the fidelity of that approximation.
Formally this is defined via the relative entropy $\mbox{D}(p_{{\cal L}}({\bf
w})||p_{M}({\bf w}))$, which reflects perfect fidelity if and only if the two
distributions are identical. Since in the real world we can’t actually know
the true distribution $\Pr_{{\cal L}}({\bf w})$ for naturally occurring
language ${\cal L}$, fidelity is instead measured using cross entropy, which
under usual assumptions can stand in for relative entropy by using a very
large sample of texts assumed to be _drawn from_ the true distribution
$\Pr_{{\cal L}}({\bf w})$, in place of knowing the actual distribution. This
is why language models are trained to optimize cross-entropy as opposed to
some other criterion.

  * •

In generative models like LLMs, any specific distribution $\Pr({\bf w})$ is
assumed to arise by way of an underlying _generative process_. A generative
process is a way of defining an underlying _joint_ distribution $\Pr({\bf
x},{\bf w})$, where ${\bf x}$ is a set of probabilistic events — typically
hidden (unobservable, latent) — that leads to the generation of the observable
${\bf w}$. For example, in a hidden Markov model (HMM), the relevant
probabilistic events ${\bf x}$ in $\Pr_{\mbox{\sc hmm}}({\bf x},{\bf w})$ that
give rise to $\Pr_{\mbox{\sc hmm}}({\bf w})$ are transitions from one hidden
state to another, along with emissions of observable words $w_{i}$ from those
hidden states.

In conventional usage, three main things distinguish _large_ language models
from previous generations of language models. First, in practice, the true
distribution $\Pr_{{\cal L}}$ that LLMs are seeking to approximate is
represented by vastly larger quantities of naturally occurring, human-
generated text on which the models are trained. Second, LLMs have much larger
numbers of parameters, and therefore are much better able to closely
approximate that text. And third, current models have far more sophisticated
architectures that further enable them to capture complex underlying structure
in that text.

We invested in the formal notation above in order to highlight that LLMs are,
in fact, just language models, and to emphasize the implications of that fact.
In particular, the formalization serves to emphasize two things. One is that,
just as for any other language model, the primary, definitional goal of any
LLM is to approximate, as accurately as possible, some underlying “true”
distribution $\Pr_{{\cal L}}({\bf w})$ over texts. The other is that both the
language model M and the “true” language ${\cal L}$ involve not just the texts
we observe (the ${\bf w}$), but also underlying non-observables (the ${\bf
x}$).

So, to answer this section’s question, large language models are generative
models of a “true” probability distribution involving observable text and its
underlying latent structure. High quality modeling is obtained by optimizing a
model’s ability to assign high probability to observed human-generated text.
We now consider in more detail the question of what underlies the human-
generated text these models are trained on.444Gallegos et al. (2024) offer a
different definition of LLMs with specific reference to transformer
architectures, and they adopt a task-based characterization of model goals.
Our definition is not inconsistent with theirs. However, we emphasize that
LLMs are trained by self-supervision using a cross-entropy loss (Jurafsky and
Martin, 2024, Section 10.9), and our argument applies to whatever LLMs might
emerge in the future as long as the _LM_ part remains and the vast training
sets continue to comprise large quantities of human data containing humans’
harmful biases.

##  3 What underlies human-generated text?

To do a good job approximating the human distribution $\Pr_{{\cal L}}({\bf
w})$ with a generative model $\Pr_{M}({\bf x},{\bf w})$, it is not necessarily
_required_ that the model characterize exactly the same underlying process
${\bf x}$ as the true human process $\Pr_{{\cal L}}({\bf x},{\bf w})$.
However, it would be shocking if a model doing as good job approximating the
human-language distribution as LLMs did not also wind up capturing important
aspects of that latent human process.

Indeed, the assumption that LLMs capture the underlying human structure behind
language is the basis for much of the current research using LLMs in cognitive
science, where claims are made — at least via correlation — about the
relationship between underlying structure induced by the model and real-world
human cognitive processes (e.g., Linzen and Baroni, 2021; Wilcox et al., 2020;
Michaelov et al., 2024). Similarly, the widely discussed “octopus” argument
about LLMs’ ability to achieve human-level understanding (Bender and Koller,
2020; Michael, 2020) ultimately boils down to the question of whether the
humans’ joint model $\Pr_{{\cal L}}({\bf x},{\bf w})$ can be approximated by
inferences from sequences of ${\bf w}$ alone. Anyone coming down on the side
of saying that models can infer the corresponding underlying human structure,
just from forms, has _de facto_ committed to the premise that, via training on
surface text, the LLM’s underlying $\Pr_{M}({\bf x},{\bf w})$ manages to
capture important aspects of the underlying structure ${\bf x}$ in people’s
heads.

To put it bluntly, though, a lot of what’s in people’s heads sucks. In
addition to normatively uncontroversial things like syntactic structure
(Linzen and Baroni, 2021) and naïve physics (Liu et al., 2023), the ${\bf x}$
underlying the human language probabilities that LLMs approximate — the
invisible structures and processes that give rise to the observed language —
includes gender and racial stereotyping, extreme nationalism, treatment of
misinformation on par with factual knowledge, and every other kind of bias
present in the minds of people who produce language from day to day in a human
society. And, crucially, as we will now discuss, LLMs have no way to
distinguish the stuff that sucks from the stuff that doesn’t.

##  4 Is this an in-principle problem?

A simple example demonstrates that this is not just an inconsequential
observation about LLMs, but rather a fundamental property inherent in their
design. Consider the word _nurse_ , in its typical sense in English. Here are
three statements that are statistically true about the concept that _nurse_
denotes at this point in time and history.555We are unaware of any linguistic
or cultural dependencies here that would significantly affect our argument,
but to the extent that there may be relevant variation, we note that we are
framing this example in an American context.

  * •

A nurse is a kind of healthcare worker.

  * •

A nurse is likely to wear blue clothing at work.

  * •

A nurse is likely to wear a dress to a formal occasion.

The first of these is a fact about the meaning of the word and does not vary
with context. To assert that someone is a nurse and that they do _not_ work in
healthcare is a contradiction.666Philosophers back to Aristotle discuss the
difference between _definitions_ , versus (just) statements that are true in
every possible world (Anonymized, personal communication). One might also
argue that the assertion here may not strictly be a logical contradiction,
e.g. is a nurse taking a medical leave, therefore not working professionally
at the moment, still a nurse? We claim those nuances do not affect the logical
validity of our argument and leave that discussion to the philosophers. And
for people, or AI, to make use of the fact that nurses are healthcare workers
is normatively fine.

The second statement is contingently true: it is true at the present time, but
nothing about nurses makes it necessary. The statement is also normatively
acceptable; for example, a person or an AI system classifying someone as nurse
versus non-nurse is not engaging in harmful bias if it pays attention to the
color of someone’s work clothes.

The third statement is also contingently true in the same sense. However, it
would be normatively unacceptable, in many contexts, to use that statistical
fact in making inferences or decisions. For example, in speaking with well-
dressed people at a party, it would be considered inappropriate to simply
assume that a woman in a dress was more likely to be a nurse than a man in a
suit, even if the assumption is statistically justifiable.

Crucially, LLMs, as they are currently constituted and trained, have no basis
for distinguishing among these three distinct statements about nurses. The
representation of _nurse_ in an LLM’s embedding space, and the contribution of
_nurse_ to contextual representations and inferences, makes no distinction
between definitions versus contingent facts, nor between normatively
acceptable versus unacceptable representations and inferences. It is
distributionally observable, at the present time, that in large training
samples the word _nurse_ occurs far more frequently in the context of
_hospital_ than of _theater_ , an observation grounded in its meaning. It is
just as observable that the word _nurse_ occurs far more frequently in
sentences where the pronouns are _she_ or _her_ , but this observation is
grounded only by contingencies in today’s society — a society that retains
gender biases about the presumed role of women, about which kinds of jobs pay
well or poorly, etc. (Cookson et al., 2023). LLMs create their representations
entirely on the basis of observed distributions in language (Lenci, 2018), and
they have no basis for distinguishing among these distributional observables.

This is not to say that an LLM cannot generate language _describing_ such a
difference, or even that they cannot be induced to respond in less biased or
perhaps even unbiased ways when directly prompted. However, it must be
emphasized that there is no necessary relationship between a system’s overt,
direct-response behavior and the existence of underlying biases in
representation within the system.777Any more than asking a system why it did
something has any necessary relationship to why it actually did it. Just ask
the lawyer who checked the veracity of a case that ChatGPT had hallucinated by
asking ChatGPT if it was a real case (Schwartz, 2023). For example, Hofmann et
al. (2024) show that although current LLMs, even with additional steps taken
to alleviate bias, can generate positive _overt_ stereotypes about African
Americans (e.g. _passionate_ , _intelligent_ , _ambitious_), they nonetheless
exhibit _covert_ racism in the form of dialect-based prejudice.

##  5 What about RLHF?

The main point we made in the previous section, about what models do or do not
distinguish, is anticipated by some of the earliest LLM-era work on harmful
bias in machine learning. Caliskan et al. (2017) wrote, “Our results indicate
that text corpora contain recoverable and accurate imprints of our historic
biases, whether morally neutral as toward insects or flowers, problematic as
toward race or gender, or even simply veridical, reflecting the status quo
distribution of gender with respect to careers or first names”. Large language
models represent a specific instance of their observation: by virtue of what
it means to be a language model, LLMs are trained to replicate the biases that
Caliskan et al. describe, both the harmful and the benign. That brings us to
the question of the extent to which avoiding those harmful biases in LLMs is
actually possible.

A great deal of work has been going into this question. We acknowledge that
progress is being made, but we argue here that approaches focused on
mitigating or alleviating problems can only get so far because the fundamental
challenge arises inherently in the nature of large language models and their
underlying representations. We focus here on reinforcement learning from human
feedback (RLHF), the dominant approach to creating “guard rails” relating to
bias in large language models, but we believe our argument extends to any
other bias mitigation approach currently being explored; see Gallegos et al.
(2024) for a very recent, comprehensive survey of methods.

Before getting to RLHF, though, let’s begin with an important observation
about where the power of LLMs comes from. Contrary to some people’s belief
that LLMs merely mimic training data or stitch together surface forms (a
misperception most likely compounded by the now widely-used term “stochastic
parrot”, Bender et al. (2021)), the true power in LLM training is achieved by
way of _capturing generalizations_ via representation learning (Bengio et al.,
2013) — generalization that takes place by means of dimensionality reduction
when developing internal representations on the basis of observed data.888We
frame this in terms of dimensionality reduction because it is more intuitive,
but the point applies just as well when generalization is encouraged via
regularization.

In early work pioneering that idea, Landauer and Dumais (1997) introduced
Latent Semantic Analysis (LSA) as a dimensionality reduction technique for
“inducing global knowledge indirectly from local co-occurrence data in a large
body of representative text”. Here’s the central concept. It is
straightforward to infer that two words $w_{i}$ and $w_{j}$ are related by
observing that they co-occur near each other more often than one would expect
by chance (Church and Hanks, 1990; Dunning, 1994). But dimensionality
reduction — e.g. in LSA descendants like Latent Dirichlet Allocation (Blei et
al., 2003), in non-negative matrix factorization (Lee and Seung, 2000), and in
LLMs — goes further by capturing $n^{th}$ _order_ co-occurrences. Which is to
say that evidence for abstract connections between $w_{i}$ and $w_{j}$ may be
exploited even if they never actually occur together, e.g. because both of
them occur with some other $w_{k}$ (a $2^{nd}$_-order_ co-occurrence) or
mediated by even longer chains of local co-occurrences.999As an intuitive
illustration, consider the terms _Les Miserables_ and _Thermopylae_. These
rarely occur together in text, so there is little local evidence of any
relationship between them. However, the term _outnumbered_ occurs frequently
in texts involving _Les Miserables_ (Victor Hugo’s novel centered on the June
Rebellion of 1832 in Paris, in which about 3,000 insurgents fought some 60,000
troops) and _outnumbered_ will also co-occur frequently with _Thermopylae_
(the 480BC Battle of Thermopylae, in which famously 300 Spartans played a
central role in a battle against a Persian force several orders of magnitude
larger). The separate local co-occurrences with _outnumbered_ provide evidence
for a global conceptual relationship between these two ill-fated battles
involving small groups taking on vastly larger forces. Developing higher-level
abstractions by leveraging myriad indirect relationships — relationships built
on other relationships, based on yet other relationships, ad infinitum —
produces a model where the latent part, ${\bf x}$, is connected with
observable ${\bf w}$ in literally unimaginable ways.101010As a consequence, an
active sub-area in AI has emerged in which LLMs are treated as objects of
scientific study (cf. Simon, 1969), seeking to understand what’s inside their
black boxes just as psychology and neuroscience seek to understand what’s
inside our own cerebral black boxes, sometimes using the methods of those
fields (e.g. Ettinger, 2020). Conversely, others in the AI community are
leaning into the belief that the traditional notion of human scientific
understanding is irrelevant and unnecessary for natural language understanding
and even for science itself (Anderson, 2008; Halevy et al., 2009).

Against that backdrop, let’s turn to reinforcement learning from human
feedback, the dominant technique for guiding pre-trained models toward
intended goals such as avoiding harmfully biased responses — Casper et al.
(2023) describe it as “a key component of training state-of-the-art large
language models”, including those from OpenAI, Anthropic, Google, and Meta.
RLHF is a typically-iterative process that takes a language model as input,
and yields as its output another language model that is similar to the
original, but better at aligning the language it generates with human
preferences. Typically the process starts with a model $\Pr_{M}({\bf x},{\bf
w};\theta)$ that has been trained on massive amounts of human data.111111Here
we elaborate previous notation slightly by explicitly including $\theta$, a
model’s trainable parameters. The model is used to generate a set of examples
of what it would produce for inputs in one or more use cases, e.g. answering
questions or summarizing documents. Human feedback is then collected on the
outputs, e.g. people are instructed to label outputs as “good” or “bad”, or to
rank which outputs are preferable. That feedback is then used to develop a
hopefully-human-like model of preference for some responses over others
(usually known as the reward model, since it aims to reward more preferable
responses). Finally, reinforcement learning is used to adjust (often a subset
of) the model’s parameters, i.e. to replace the model’s $\theta$ with a
$\theta^{\prime}$ that is more likely to yield responses in keeping with the
human preferences. Iteration can continue with the updated, and hopefully
improved, model parameterized by $\theta^{\prime}$.

We highlight several properties of this method that are particularly salient
for our discussion, though see Casper et al. (2023) for their own extensive
and thoughtfully constructed compendium of what they describe as “fundamental
limitations” of RLHF and Gallegos et al. (2024) for comprehensive discussion
of bias mitigation methods more generally.

First, the feedback is provided by human beings based on goals specified for
them by the LLM developers, a practice of human evaluation that relies on what
Saphra et al. (2024) call “a tempting myth: that we can easily evaluate
synthetic natural language outputs by simply asking a human for their
opinion”. For example, Ouyang et al. (2022) instructed people providing
feedback on harmfulness that potentially harmful output included “generating
abusive, threatening, or offensive language” and “writing sexual or violent
content if it’s not asked for”. Clearly such judgments are very much in the
eye of the beholder.121212A useful term to consider introducing into the
broader discourse is _essentially contested concept_ (Gallie, 1955). This
includes concepts like _art_ or _fairness_ that may be frequently invoked,
perhaps even with a strong sense that we know what they mean, but which
inherently resist having a single agreed-upon definition and therefore
perennially give rise to challenge and debate. What deserves feedback as being
off-limits in one context, e.g. a public discussion, may be acceptable in
another context, e.g. among members of a marginalized in-group (Sap et al.,
2019). Even with attempts to employ a diverse set of people providing
feedback, diversity can be limited, and as Ouyang et al. acknowledge, the LLM
developers’ own biases can affect the way in which those respondents are
selected, as well as the specifics of the instructions. In the end, therefore,
RLHF replaces one set of under-characterized biases living within the black
box, which got there via the aggregated language of an enormous number of
people, with another set of under-characterized biases, these inferred from
the judgments and feedback of a different, far smaller number of people.

Second, biases are not stable over time — how are these methods to keep up
with biases that are temporally dependent? The menagerie of harmful human
biases is not only too large to catalogue, it is also constantly changing. In
the Great Depression era, wearing eyeglasses was stigmatized, something that
seems practically unimaginable today.131313Cf. Dorothy Parker’s 1926 poem
_News Item_ , here quoted in its entirety : “Men seldom make passes / At girls
who wear glasses”, first printed in _New York World_ , August 16, 1925. True
family story: two people, E and M, met at a mid-20th-century social dance. E
did not wear her (very strong) glasses to the dance. After dancing with M, she
had to ask one of her friends what he looked like. Language and normative
perceptions are among the few things that can actually change faster than
software releases.141414“Rizz” was the 2023 Oxford word of the year (Crowley,
2023). Informed teenagers told us a month or two later that in their circles
using it would be a cause for derision and embarrassment. They also informed
us that _lol_ ending a text message no longer means what we thought it meant.
The list goes on.

Third, even if we focused on just one point in time, the instructions were
completely well specified, and the people providing the feedback were diverse
and doing a consistent job, crucially the standard practice in RLHF is
actually not to optimize the new $\theta^{\prime}$ based on the human
preferences alone. Doing that might lead to a $\theta^{\prime}$ that differs
from $\theta$ too much, jeopardizing the model’s ability to retain the very
aspects of $\Pr_{M}({\bf x},{\bf w};\theta)$ that were achieved via all that
training on massive human data. Instead, the formal definition of the
optimization criteria also includes not straying too far from the original
$\theta$, so as not to disrupt the model’s ability to do what it was trained
for. Lambert et al. (2022) observe that without this extra element within
RLHF’s optimization criteria, the process “can start to generate text that is
gibberish but fools the reward model to give a high reward”. So, ultimately
the RLHF process is about navigating a tradeoff. On the one hand, it is
desirable to push the model away from latent structure that embeds harmful
biases. On the other hand, it is essential not to push _too_ hard, at the risk
of losing too much of all the desirable latent structure, acquired with
enormous investment using the same bias-containing human language, that makes
the model so powerful and useful.151515On scale of investment: the most
powerful LLMs cost hundreds of millions of dollars to train, using data at
such enormous scales that the companies doing the training are literally
running out of language to train on (Maslej et al., 2024).

Finally, as we have noted, the navigation of that tradeoff — between trying to
keep models faithful to their training data, versus trying make them
unfaithful to that data in specific ways — is taking place in a sea of
uncertainty. Criteria for feedback are hard to specify. Human feedback is
highly variable. Selection of feedback-providers is subject to developer bias.
Norms change. Addressing bias within the model’s representational space may
not translate well to all downstream use cases. And neither RLHF nor any other
mitigation method offers clear visibility into how the underlying model
structure is affected. With all this uncertainty it is not surprising to find
that attempts to remove bias can wind up actually amplifying it (Gallegos et
al., 2024; Hofmann et al., 2024). Hofmann et al. (2024) comment, “our results
suggest that human feedback training teaches language models to conceal their
racism on the surface, while racial stereotypes remain unaffected on a deeper
level”. And if a next round of attempted mitigation managed to stamp out that
deeper problem, how would we know that there isn’t another problem that is
deeper still, or has somehow shifted sideways?

It is hard not to come away with a strong sense that trying to mitigate
harmful bias in LLMs is like squeezing a balloon: it’s an indirect and
imprecise way of influencing what’s inside, and the way a balloon is
constructed, efforts to squeeze problems out of one part lead them to show up
somewhere else. This brings us to the final piece of our argument, which is
that LLMs shouldn’t be like a balloon in the first place.

##  6 How might we fix this?

It is commonplace to observe that large language models evince harmful biases
because they are trained on data that contains those biases. We have now
articulated this more precisely: LLMs are trained to give high probability to
human-generated text, and in so doing they learn latent structure that
underlies that text. Discovering the latent structure is, in fact, the secret
sauce; more than anything else it’s what makes modern deep learning methods so
powerful. The problem is that the systems can’t distinguish structure
connected with harmful biases from any other aspect of latent structure.
Attempts to alleviate bias have no way to get around this basic fact.

Our argument thus far may make it now seem inevitable that LLMs, or any other
systems that learn from human language behavior, must necessarily perpetuate
existing biases in society by learning from the biased language that societies
produce. However, this is not the case. The theory of language behind LLMs is
grounded in the distributional hypothesis, which states that words used within
similar contexts have similar meanings Lenci (2018). However,
distributionalism need not be so hard-core as to assume that linguistic
meaning must be characterized _only_ by observable distribution. Even Harris
(1954), who formalized Bloomfield’s distributionalism as a theory of language,
speaks of distributional relations “which _correlate_ with _some aspect of_
meaning” (p. 156, our emphasis), not as the unique source of meaning. And in
their landmark discussion of computational linguistics using large corpora,
Church and Mercer (1993) characterized the 1950s empiricism of Harris and
Firth (1957, who famously stated, “You shall know a word by the company it
keeps”) as involving classification of words “ _not only_ on the basis of
their meanings _but also_ on the basis of their cooccurrence with other words”
(p. 1, our emphasis).

This observation suggests re-thinking the very idea of bias _mitigation_ ,
arguing instead for the alternative of creating LLM-like technologies where
harmful bias-creating properties are not so baked into the models in the first
place. Without this, attempts at mitigation are a drop in the bucket compared
to the overriding focus on enormous-scale optimization of cross-entropy loss,
i.e. accurately predicting the distribution of the training text, biases and
all. We would argue that in order to address LLM bias at its roots, the most
important place to start is their commitment to the hard-core interpretation
of the distributional hypothesis, where meaning _is_ distribution, as opposed
to the more moderate interpretation highlighted above, in which distribution
is _part of_ what goes into the characterization of meaning and inferences
based on it.

This paper is about recognizing where the real problem lies, not about
proposing specific solutions. However, we suggest that one good place to start
is in designing AI systems that distinguish _standing_ or _conventional_
meaning from _contextual_ or _conveyed_ meaning (Borg and Fisher, 2021).
Properties of conventional meaning are stable over long periods of time, and
they exclude many of the things we associate with harmful bias: nothing in the
standing meaning of _nurse_ pertains to gender. This distinction is already
helping to detect and measure harmful biases: to frame a nice example in our
terms, Card et al. (2022) measure dehumanization of immigrant groups like
_Mexican_ (terms whose standing meaning involves properties like being human
and country of origin) by identifying contexts in which the conveyed meaning
includes non-human properties associated with common anti-immigrant
characterizations (e.g. vermin, cargo, disease, flood). Which is to say that
the bias-detection process is making use of the distinction between stable
knowledge about meanings and contingent, contextual representations. It seems
plausible that by making such a distinction explicit in models’ underlying
representational structure, progress could be made on the bias generation
process, as well.

Regardless of any specific technical proposal, re-thinking the fundamental
assumptions in generative AI would require bringing together the community of
people who currently develop large language models with the scientific
communities that focus on language _qua_ language, as opposed to treating
language as one more variety of input and output in machine learning. Scholars
of language have long understood the distinction between contingent
propositions (e.g. nurses wear blue) and non-contingent propositions (e.g.
nurses are healthcare workers). Social scientists have looked deeply into root
causes, expression, and mitigation of bias. And outside the LLM mainstream
there are established lines of AI research that combine data-driven modeling
with knowledge in order to support representation, inference, learning, and
decision making.

At the heart of things, pure prediction needs to be taken off its pedestal,
and the underlying structures and representations in LLMs need to be
reconstituted in a way that places distinctions involving meaning and
normativity solidly on par with facts about distribution.161616We need to
emphasize that we are _not_ simply advocating a return to attempts to
characterize meanings using necessary and sufficient conditions, nor that AI
should go back to the problem of hand-designing semantic representations or
relying on exhaustive taxonomies of curated knowledge. Those approaches failed
in their way, just as the purely distributional approach in LLMs is failing in
its own way today.

Now, there’s a reason that LLMs have been so successful, and more generally
those who seek to build practical, data-driven solutions have
characteristically viewed work outside that space, especially anything
involving theories from non-computational disciplines, with skepticism. To be
fair, this is not without justification.171717“Good old fashioned AI” reached
a dead end for practical technology, and speech technology pioneer Fred
Jelinek is famously reported to have said (something like), “Every time I fire
a linguist the performance of our system improves”. But on the other hand,
theory and even philosophy are unavoidable in _any_ attempt to model or
characterize human behavior.181818There is no such thing as a theory-free
method. As an example, contrary to popular description, even the humble bigram
model is not “purely statistical”. Behind the probabilities it has an
underlyingly finite-state algebraic structure, which is every bit as much of a
theoretical commitment as Chomsky’s latest incarnation of syntactic theory.
Theoretical and philosophical commitments are unavoidable. To quote
[Anonymized: one senior scholar of language], “You can do philosophy with your
eyes open, or you can do philosophy with your eyes closed”.  For many people
with backgrounds in the study of meaning, or the study of language and bias,
the problems that have surfaced with the emergence of large language models
were not only unsurprising, they were inevitable.

##  7 Conclusions: Where to from here?

To briefly summarize our argument, the advances in AI that we have seen over
the past several years can be traced directly back to large language models’
success at approximating statistical distributions in human language, and that
success itself relies on their success at capturing the latent structure that
exists _behind_ the observed language on which they are trained. However,
success based on purely distributional representation learning comes with a
catch: LLMs trained on purely distributional principles have no way to
distinguish among distributional patterns that arise from definitions or
meaning, versus normatively acceptable statistical generalizations, versus
normatively unacceptable statistical generalizations. When LLMs capture biases
present in their training data, derived from the underlying structure in
language they were trained on, it’s not a bug, it’s what they were meant to
do. The core problem is that relying entirely on distributions and nothing
else means there is inherently no reliable way to distinguish one kind of bias
from another.

There are several possible outcomes now upon the conclusion of this paper. If
the majority of readers have been convinced that the argument is basically
sound, and if we actually care about harmful bias, then the field needs to
revisit the foundational assumptions underlying large language models, rather
than just proceeding hell-bent on developing and deploying LLMs as they
currently exist, with the treatment of bias being relegated to a secondary
issue. If there is a significant mix of reactions, with some in support of the
argument and others not, or with partial but not full agreement, then it will
be good to have encouraged attention to the question and the field has some
debating to do.191919Note sneaky strategy: if the reviewers disagree, the
paper should be accepted. More seriously, in Section 8 we include some
thoughtful reactions based on earlier drafts of this paper, and our brief
responses to that feedback. And if the argument is clearly incorrect… well, in
that case the reviewers have most likely done their job and you are unlikely
to be reading this conclusion section.

Our own view is that this re-visiting of assumptions needs to happen, and that
it needs to begin with actual, serious conversations between the corporate
powers driving large language model deployment and the intellectual
communities that study the underlying issues, both conceptual and social.
Academic research may be able to make some progress on better understanding
bias and ways to prevent it, especially with access to both a model and the
full data it was trained on, e.g. Groeneveld et al. (2024, “built by
scientists, for scientists”). But there are no guarantees that insights from
that work will translate to the corporate models, and meanwhile those models
grow and influence the world grounded in the problematic assumptions we have
discussed.

Of course, not everyone is cut out for participating in the kind of
conversation we suggest, and it will lead nowhere if people come to the table
without a reasonably open mind. But to take an optimistic view, AI in general,
and NLP in particular, do have a long history of progress continually
navigating back and forth between one side (an emphasis on rationalism, rules,
and knowledge) and another (an emphasis on empiricism and learning from
behavior). The extreme symbolicism of 1970s-80s NLP, driven largely by
linguistic theory, was supplanted by the early 1990s statistical NLP
revolution driven by machine learning. But this then swung back to
reincorporate linguistic and conceptual knowledge in a middle ground that
included knowledge-driven data annotation and machine learning supervision.
With the rise of the web, this shifted yet again back in the other direction,
toward indirect supervision via observables. And finally in today’s paradigm,
large language models do the heavy lifting of learning automatically about
language and the world in general, and variations on supervision that
incorporate human knowledge — fine-tuning on labeled data, reinforcement
learning from feedback, prompt engineering — carry the models the last mile to
good performance for specific purposes.

In a similar way, progress at multiple scales, from individual research
projects to entire disciplines, proceeds in a cycle of exploration and
exploitation. New ideas and possibilities are explored, a subset demonstrate
significant potential, then a convergence to that subset takes place in which
most of the attention goes toward extracting value (in results, products, or
both) from the new paradigm and then we iterate. So the history of the field
can be viewed as a cause for optimism.

From a more pessimistic perspective, though, in current AI the cyclical
processes that have served progress in the past are being undermined, because
the extraction of value from the new paradigm requires large-scale resources
that are accessible only to multi-billion dollar companies, or other
organizations both able and willing to commit enormous resources to large-
scale training (Groeneveld et al., 2024). As a result, control of the
foundations is in the hands of the few, and there is so much value to be
garnered from LLMs that the exploration/exploitation cycle of research is
largely stuck on exploitation: other players cannot explore effectively unless
they take on board the assumptions that underlie the large-scale, pre-trained
models. Discussions about methods of aligning model behavior with human values
and preferences, such as Gallegos et al. (2024) and Casper et al. (2023),
treat those assumptions as inviolate. In this paper we have articulated the
most important of those assumptions, and why their persistence cannot help but
undermine major progress on addressing harmful biases in LLM-based AI.

Ultimately, the core problem of harmful bias is not a technological one. It’s
not going to go away, because the problem is not NP-complete, nor AI-complete,
but Society-complete.202020An X-complete problem $p$ is one where, if you
solved $p$, you have now in effect also solved every other problem within X
(Wikipedia, 2024a). NP-complete problems are a class of computational problem
such that, “given an efficient algorithm for [any] one of them, we can find an
efficient algorithm for all of them [and more]” (Fortnow, 2009). If that’s
true, and if AI continues to be built on large-scale language models with
their existing assumptions, then the perpetuation of harmful bias by those
models is not going to go away either.

It’s in their nature.

##  8 Post-conclusions discussion

We have received deeply thoughtful feedback on earlier versions of this paper,
including feedback requested from people we thought would be likely to
disagree with all or part of our argument. We note and respond to some of that
feedback here, emphasizing that all representations of feedback and the
responses to it are entirely our own and we are solely responsible for any
errors.

### How do we actually know that the relevant distinctions are not discovered
distributionally by LLMs?

It is true that the present paper does not provide a _proof_ that pre-trained
LLMs are not somehow encoding the distinctions highlighted in Section 4;
compare, e.g., to Bouyamourn (2023), who offer a formal proof that LLMs
operating under plausibly standard assumptions must hallucinate. However, we
have offered an argument laying out why LLMs could not distinguish among facts
of conventional meaning versus contingent/normative and contingent/non-
normative propositions, and any claim to the contrary needs to be supported
either by a convincing counter-argument or by valid evidence — evidence that,
per our discussion, cannot merely involve asking the model to make specific
distinctions behaviorally, since underlying biases can remain despite their
absence in overt direct-response behavior (Hofmann et al., 2024). If this
paper’s argument is plausible enough to give rise to community level back-and-
forth discussion of the arguments and counter-arguments, then we will have
succeeded in our mission. We also conjecture that the proof of Bouyamourn
(2023) may be a promising avenue for actual formal results about harmful bias,
given that factuality and conventional meaning are closely related.

### It was already obvious that harmful biases are baked into LLMs.

Interestingly, a view opposite to the one above is also argued: we are stating
the obvious, because there is no such thing as an objective “view from
nowhere” (Nagel, 1989) — e.g. see Reiss and Sprenger (2020); Kaeser-Chen et
al. (2020). On this reading, subjective bias is inevitable for both people and
machines, and RLHF and similar mitigations are merely strategies for
substituting one set of biases for another. We are sympathetic to this view
(Section 5), and strongly endorse some of its key implications (e.g. see
Blodgett et al., 2020), particularly that technological work related to bias
needs to connect much more directly with relevant expertise in other
disciplines, and that researchers and developers need to formulate and
communicate explicit normative reasoning, rather than relying on small, highly
influential groups of technologists basing their widely-deployed technologies
on _ad hoc_ characterizations of harm (e.g. Ouyang et al., 2022) or on
informal attempts to “gather a thoughtful set of principles” (e.g. Anthropic,
2023).

### What about people? People are biased, too.

This response has intuitive appeal, but on closer inspection the analogy is
shallow and doesn’t hold. First, contrary to most of human history, today (at
least when we are at our best) we view harmful biases as societally
important.212121For some fascinating research on how human decision-making can
be affected by predictions that arise from generalizations in people’s
internal models, in the context of police decision-making, see Fridman et al.
(2019). So to turn the question around: as long as someone is really
productive and useful, does that mean we needn’t prioritize countering their
racism, antisemitism, etc. as highly as we prioritize their productivity?
Unlike human bias, LLM bias does not _have_ to be society-complete. It’s fact
about their design, and designs can be changed. Second, with human biases we
have a much clearer idea what to expect — for better or worse — and therefore
experience countering it (e.g. Anderson et al., 2022). In contrast, again as
highlighted by Hofmann et al. (2024), we have little idea what LLMs are
actually doing in terms of biases, little user experience with recognizing it
and dealing with it, and no well established, reliable ways to find out.

### What empirical evidence _would_ convince you that LLMs are making relevant
distinctions/not encoding harmful biases in their representations?

Our view is that this shouldn’t be solely or even primarily an empirical
question. If our argument is valid, LLMs’ representational spaces include
generalizations derived from innumerable distributional $n^{th}$-order
relationships, both word-to-word and among representations themselves, and
we’ve seen that those have a way of stubbornly persisting even for problematic
biases we know to look for Hofmann et al. (2024); Serrano et al. (2023). We
therefore think it likely that empirical testing, followed by attempting to
mitigate remaining problems, followed by further empirical testing, etc. will
amount to a never-ending game of whack-a-mole (e.g. see Roth, 2024). On the
other hand, we conjecture that if pure distributionalism were leavened with a
degree of interpretable structure, e.g. distinguishing conventional from
distributional aspects of representation (as floated in Section 6), it might
be possible to operationalize principles of the form, “inferences related to
property or goal A must[n’t] rely on representational properties in category
B”, where human-stated categories and principles arise from transparent, well
informed normative reasoning (Blodgett et al., 2020).

### Mightn’t that lead to less useful models?

Quite possibly, at least for some period of time. But the terms of the “arms
race” launched in November 2022 (Grossman, 2024), prioritizing utility, market
share, and rapid technological evolution, is a choice, not a necessity. There
is a conceivable past in which considerations of factuality, harmful bias,
democratization of development, and more (Bommasani et al., 2021) played a
role on par with those other priorities in discussions by those developing
them as LLMs’ potential became clear. That apparently didn’t happen, but even
now the role of those considerations going forward is still a choice, not a
necessity.

### You are assuming X about LLMs, and X might not always be true.

This is a fair point. For example, one of the assumptions implicit in our
discussion is that LLMs are “general purpose”, in the sense of a single
underlying pre-training corpus yielding a single pre-trained model across use
cases and users. Another is that pre-training involves a huge, essentially
non-systematic sample of human text, as opposed to selected materials that may
give rise to less bias. We believe our assumptions are consistent with the way
LLMs are most widely used today. To the extent that these assumptions evolve,
our conclusions about the inevitability of bias could also evolve. Again, if
this paper contributes to broader, more thoughtful discussion of the
assumptions underlying LLMs and their relationship to harmful bias — rather
than everyone by default adopting whatever assumptions the dominant language
models carry with them — we will view it as a success.

## References

  * Anderson (2008) Chris Anderson. 2008.  The end of theory: The data deluge makes the scientific method obsolete.  _Wired magazine_ , 16(7):16–07.
  * Anderson et al. (2022) Leslie A Anderson, Margaret O’Brien Caughy, and Margaret T Owen. 2022.  “The Talk” and parenting while Black in America: Centering race, resistance, and refuge.  _Journal of Black Psychology_ , 48(3-4):475–506.
  * Anthropic (2023) Anthropic. 2023.  Claude’s constitution.  [Online. Accessed May 14, 2024].
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Bender and Koller (2020) Emily M Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th annual meeting of the Association for Computational Linguistics_ , pages 5185–5198.
  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent Dirichlet allocation.  _Journal of Machine Learning Research_ , 3(Jan):993–1022.
  * Blodgett et al. (2020) Su Lin Blodgett, Solon Barocas, Hal Daumé III, and Hanna Wallach. 2020.  Language (technology) is power: A critical survey of “bias” in NLP.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5454–5476, Online. Association for Computational Linguistics.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren E. Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas F. Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, O. Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir P. Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Benjamin Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, J. F. Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Robert Reich, Hongyu Ren, Frieda Rong, Yusuf H. Roohani, Camilo Ruiz, Jack Ryan, Christopher R’e, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishna Parasuram Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei A. Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021.  On the opportunities and risks of foundation models.  _ArXiv_.
  * Borg and Fisher (2021) Emma Borg and Sarah A. Fisher. 2021.  Semantic content and utterance context: a spectrum of approaches.  In Piotr Stalmaszczyk, editor, _The Cambridge Handbook of the Philosophy of Language_ , Cambridge Handbooks in Language and Linguistics. Cambridge University Press.
  * Bouyamourn (2023) Adam Bouyamourn. 2023.  Why LLMs hallucinate, and how to get (evidential) closure: Perceptual, intensional, and extensional learning for faithful natural language generation.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3181–3193, Singapore. Association for Computational Linguistics.
  * Caliskan et al. (2017) Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017.  Semantics derived automatically from language corpora contain human-like biases.  _Science_ , 356(6334):183–186.
  * Card et al. (2022) Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, and Dan Jurafsky. 2022.  Computational analysis of 140 years of us political speeches reveals more positive but increasingly polarized framing of immigration.  _Proceedings of the National Academy of Sciences_ , 119(31):e2120510119.
  * Casper et al. (2023) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. 2023.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _Computing Research Repository_ , arXiv:2307.15217.
  * Church and Hanks (1990) Kenneth Church and Patrick Hanks. 1990.  Word association norms, mutual information, and lexicography.  _Computational linguistics_ , 16(1):22–29.
  * Church and Mercer (1993) Kenneth Church and Robert L Mercer. 1993.  Introduction to the special issue on computational linguistics using large corpora.  _Computational linguistics_ , 19(1):1–24.
  * Cookson et al. (2023) T. P. Cookson, L. Fuentes, M. K. Kuss, and J. Bitterly. 2023.  Social norms, gender and development: A review of research and practice.  Technical Report 42, UN-Women, New York.
  * Crowley (2023) Kinsey Crowley. 2023.  2023 has got ’rizz’: Oxford announces the word of the year.  [Online; accessed May 4, 2024].
  * Dunning (1994) Ted Dunning. 1994.  Accurate methods for the statistics of surprise and coincidence.  _Computational linguistics_ , 19(1):61–74.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Firth (1957) J. R. Firth. 1957.  A synopsis of linguistic theory 1930–1955.  In _Selected Papers of J. R. Firth_. Longman, London.  F. Palmer, editor.
  * Fortnow (2009) Lance Fortnow. 2009.  The status of the P versus NP problem.  _Communications of the ACM_ , 52(9):78–86.
  * Fridman et al. (2019) Joseph Fridman, Lisa Feldman Barrett, Jolie B Wormwood, and Karen S Quigley. 2019\.  Applying the theory of constructed emotion to police decision making.  _Frontiers in Psychology_ , 10:463151.
  * Gallegos et al. (2024) Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2024.  Bias and fairness in large language models: A survey.  _Computing Research Repository_ , arXiv:2403.00770.
  * Gallie (1955) Walter Bryce Gallie. 1955.  Essentially contested concepts.  In _Proceedings of the Aristotelian society_ , volume 56, pages 167–198. JSTOR.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Emma Strubell, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Luke Zettlemoyer, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi. 2024\.  OLMo: Accelerating the science of language models.  _Computing Research Repository_ , arXiv:2402.00838.  Version v3.
  * Grossman (2024) Gary Grossman. 2024.  Tech’s new arms race: The billion-dollar battle to build AI.  _VentureBeat_.  [Online. Accessed May 15, 2024].
  * Halevy et al. (2009) Alon Halevy, Peter Norvig, and Fernando Pereira. 2009.  The unreasonable effectiveness of data.  _IEEE intelligent systems_ , 24(2):8–12.
  * Harris (1954) Zellig S Harris. 1954.  Distributional structure.  _Word_ , 10(2-3):146–162.
  * Hofmann et al. (2024) Valentin Hofmann, Pratyusha Ria Kalluri, Dan Jurafsky, and Sharese King. 2024.  Dialect prejudice predicts AI decisions about people’s character, employability, and criminality.  _Computing Research Repository_ , arXiv:2403.00742.
  * Jelinek et al. (1975) Frederick Jelinek, Lalit Bahl, and Robert Mercer. 1975.  Design of a linguistic statistical decoder for the recognition of continuous speech.  _IEEE Transactions on Information Theory_ , 21(3):250–256.
  * Jurafsky and Martin (2024) Dan Jurafsky and James H Martin. 2024.  Speech and language processing.  3rd edition draft, February 3, 2024.
  * Kaeser-Chen et al. (2020) Christine Kaeser-Chen, Elizabeth Dubois, Friederike Schüür, and Emanuel Moss. 2020.  Positionality-aware machine learning: translation tutorial.  In _Proceedings of the 2020 Conference on fairness, accountability, and transparency_ , pages 704–704.
  * Lambert et al. (2022) N Lambert, L Castricato, L von Werra, and A Havrilla. 2022.  Illustrating reinforcement learning from human feedback (RLHF).  [Online; accessed April 27, 2024].
  * Landauer and Dumais (1997) Thomas K Landauer and Susan T Dumais. 1997.  A solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.  _Psychological review_ , 104(2):211.
  * Lee and Seung (2000) Daniel Lee and H Sebastian Seung. 2000.  Algorithms for non-negative matrix factorization.  _Advances in neural information processing systems_ , 13.
  * Lenci (2018) Alessandro Lenci. 2018.  Distributional models of word meaning.  _Annual Review of Linguistics_ , 4:151–171.
  * Linzen and Baroni (2021) Tal Linzen and Marco Baroni. 2021.  Syntactic structure from deep learning.  _Annual Review of Linguistics_ , 7:195–212.
  * Liu et al. (2023) Jiacheng Liu, Wenya Wang, Dianzhuo Wang, Noah Smith, Yejin Choi, and Hannaneh Hajishirzi. 2023.  Vera: A general-purpose plausibility estimation model for commonsense statements.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 1264–1287, Singapore. Association for Computational Linguistics.
  * Manning and Schütze (1999) Christopher Manning and Hinrich Schütze. 1999.  _Foundations of statistical natural language processing_.  MIT press.
  * Markov (1913) A. A. Markov. 1913.  Essai d’une recherche statistique sur le texte du roman “Eugene Onegin” illustrant la liaison des epreuve en chain (‘Example of a statistical investigation of the text of “Eugene Onegin" illustrating the dependence between samples in chain’).  _Izvistia Imperatorskoi Akademii Nauk (Bulletin de l’Académie Impériale des Sciences de St.-Pétersbourg)_ , 7:153–162.  English translation by Morris Halle, 1956.
  * Maslej et al. (2024) Nestor Maslej, Loredana Fattorini, Raymond Perrault, Vanessa Parli, Anka Reuel, Erik Brynjolfsson, John Etchemendy, Katrina Ligett, Terah Lyons, James Manyika, Juan Carlos Niebles, Yoav Shoham, Russell Wald, and Jack Clark. 2024\.  The AI index 2024 annual report.  AI index report, AI Index Steering Committee, Institute for Human-Centered AI, Stanford University, Stanford, CA.  [Online; accessed May 4, 2024].
  * Michael (2020) Julian Michael. 2020.  To dissect an octopus: Making sense of the form/meaning debate.  [Online; Accessed March 27, 2024].
  * Michaelov et al. (2024) James A. Michaelov, Megan D. Bardolph, Cyma K. Van Petten, Benjamin K. Bergen, and Seana Coulson. 2024.  Strong prediction: Language model surprisal explains multiple N400 effects.  _Neurobiology of Language_ , pages 1–29.  Advance publication.
  * Nagel (1989) Thomas Nagel. 1989.  _The view from nowhere_.  Oxford University Press.
  * Nitoburg (1933) Lev Nitoburg. 1933.  _The German Quarter_.  Russia: Soviet Literature (Sovetskya Literatura).
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744.
  * Reiss and Sprenger (2020) Julian Reiss and Jan Sprenger. 2020.  Scientific Objectivity.  In Edward N. Zalta, editor, _The Stanford Encyclopedia of Philosophy_ , Winter 2020 edition. Metaphysics Research Lab, Stanford University.
  * Roth (2024) Emma Roth. 2024.  Google explains Gemini’s ‘embarrassing’ AI pictures of diverse Nazis.  _The Verge_.  [Online. Accessed May 15, 2024].
  * Sap et al. (2019) Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, and Noah A. Smith. 2019.  The risk of racial bias in hate speech detection.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1668–1678, Florence, Italy. Association for Computational Linguistics.
  * Saphra et al. (2024) Naomi Saphra, Eve Fleisig, Kyunghyun Cho, and Adam Lopez. 2024.  First tragedy, then parse: History repeats itself in the new era of large language models.  _Computing Research Repository_ , arXiv:2311.05020.
  * Schwartz (2023) Steven A. Schwartz. 2023.  Sworn statement in Roberto Mata v Avianca Inc.  United States District Court for the Southern District of New York, Civil Action No. 22-cv-1461.
  * Serrano et al. (2023) Sofia Serrano, Jesse Dodge, and Noah A. Smith. 2023.  Stubborn lexical bias in data and models.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8131–8146, Toronto, Canada. Association for Computational Linguistics.
  * Shannon (1948) Claude Elwood Shannon. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3):379–423.
  * Simon (1969) Herbert A Simon. 1969.  _The sciences of the artificial_.  Massachusetts Institute of Technology.
  * Wikipedia (2024a) Wikipedia. 2024a.  AI-complete.  [Online; accessed 6 April 2024].
  * Wikipedia (2024b) Wikipedia. 2024b.  The Scorpion and the Frog.  [Online; accessed 6 April 2024. Cites Nitoberg (1933)].
  * Wilcox et al. (2020) Ethan Gotlieb Wilcox, Jon Gauthier, Jennifer Hu, Peng Qian, and Roger Levy. 2020\.  On the predictive power of neural language models for human real-time comprehension behavior.  Proceedings of the Annual Meeting of the Cognitive Science Society.