# Increased Compute Efficiency and the Diffusion of AI Capabilities

Konstantin Pilz
Georgetown University&Lennart Heim∗
Centre for the Governance of AI &Nicholas Brown Equal contribution.
Correspondence to<EMAIL_ADDRESS>and<EMAIL_ADDRESS>
###### Abstract

Training advanced AI models requires large investments in computational
resources, or compute. Yet, as hardware innovation reduces the price of
compute and algorithmic advances make its use more efficient, the cost of
training an AI model to a given performance falls over time. To analyze this
phenomenon, we introduce compute (investment) efficiency, which relates
training compute investment to the resulting AI model performance. We then
present a conceptual model of increases in compute efficiency and assess the
social and governance implications. We find that while an access effect
increases the number of actors who can train models to a given performance
over time, a performance effect simultaneously increases the performance
available to every actor – potentially enabling large compute investors to
pioneer new capabilities and maintain a performance advantage even as
capabilities diffuse. The market effects are multifaceted: while a relative
performance advantage might grant outsized benefits in zero-sum competition,
performance ceilings might reduce leaders’ advantage. Nonetheless, we find
that if the most severe risks arise from the most advanced models, large
compute investors warrant particular scrutiny since they discover potentially
dangerous capabilities first. Consequently, governments should require large
compute investors to warn them about dangerous capabilities, thereby enabling
timely preparation and potentially using their superior model performance and
compute access for defensive measures. In cases of extreme risks, especially
offense-dominant capabilities, the government might need to actively restrict
the proliferation entirely.

## Executive Summary

The cost of training an AI model to a given level of performance falls over
time. In 2017, training a classifier to 93% accuracy on ImageNet cost over
$1,000; in 2021, it cost only $5 – a reduction of over 99%. What are the
drivers of this trend and what does it imply for the diffusion of AI
capabilities?

Two factors reduce the compute cost of developing AI models:

  1. a)

Improvements in hardware price performance – as predicted by Moore’s Law –
increase the computational performance available per dollar over time, often
measured in FLOP/s per $. Between 2006 and 2021, the price performance of AI
accelerators has doubled approximately every two years.

  2. b)

Due to improvements in algorithmic efficiency, training an AI model to a given
level of performance requires fewer computational operations over time. For
example, between 2012 and 2022, advances in image recognition algorithms
halved the compute required for achieving 93% classification accuracy on the
ImageNet dataset every nine months.

We introduce the concept of compute investment efficiency – abbreviated to
_compute efficiency_ – to capture the combined impact of these factors (Figure
1). Compute investment efficiency determines the _AI model performance_ 111In
the context of compute efficiency, _AI model performance_ refers to a directly
measurable model property such as test loss for a language model or cumulative
reward for an RL agent. available with a given level of investment in
_training compute_ , provided sufficient training data (Section 1).

Figure 1: Compute (investment) efficiency is the relationship between training
compute investment and AI model performance.

Increasing compute efficiency has two main effects (Section 3):

  * •

An access effect: over time, access to a given level of performance requires
less compute investment (Figure 2, red).

  * •

A performance effect: over time, a given level of compute investment enables
increased performance (Figure 2, blue).

Figure 2: Compute efficiency improves between time $t=0$ and $t=1$, causing an
access effect (red) and a performance effect (blue). Figures are merely
conceptual and do not assert specific claims regarding the slopes of the
curves.

If actors have equal access to compute efficiency improvements, these effects
have the following consequences (Section 4.1):222For simplicity, our model
assumes that all actors have equal access to compute efficiency improvements.
For a discussion of these assumptions, see Section 5.

  * •

Large compute investors discover novel capabilities first. The performance
effect lets actors achieve increased performance with a fixed compute
investment. Since large compute investors achieve the highest performance
levels, they are the first to discover new model capabilities333With AI model
capabilities, we refer to concrete tasks AI models accomplish in the real
world, whereas we define a model’s performance by technical metrics, such as
test loss. For instance, a language model that achieves a certain performance
on next-word prediction may gain the capability to solve coding problems. that
allow novel use cases.

  * •

Capabilities diffuse over time. Due to the access effect, the investment
required to reach a given performance level decreases over time, giving an
increased number of actors access to capabilities previously restricted to
large compute investors.

  * •

Large compute investors maintain a performance advantage even as capabilities
diffuse. Compute efficiency improvements that expand access to smaller actors
simultaneously increase the performance available to large compute investors.
Absent a ceiling on absolute performance, those actors continue to demonstrate
the highest level of performance over time.

Apart from these consequences, additional effects influence the competitive
advantage of different actors (Section 4.2):

  * •

Performance ceilings dampen the performance effect, reducing leaders’
advantage. Many AI applications have a ceiling on technical performance or
real-world usefulness. For instance, handwritten digit classifiers have
achieved above 99% accuracy since the early 2000s, so further progress is
insignificant. As leaders approach the ceiling, performance only marginally
increases with improved compute efficiency, allowing smaller actors to catch
up.

  * •

Winner-takes-all effects may allow leaders to entrench their lead even as they
lose their performance advantage. By initially developing the best-performing
models, large compute investors may accrue a number of advantages unrelated to
performance, such as network effects and economies of scale.

  * •

In zero-sum competition, a relative performance advantage may grant outsized
benefits. If AI models directly compete, the developer of the leading model
may reap disproportionate benefits even if their absolute performance
advantage is small. Such disproportionate rewards occur in games such as chess
but likely also apply to AI models used in trading, law, or entertainment.

Implications for Dangerous Capabilities (Section 4.3)

  * •

Large compute investors most likely encounter dangerous capabilities first.
Future AI models could show dangerous capabilities, such as exploiting
cybersecurity vulnerabilities, bioweapon design, or evading human control.
Since these capabilities require high levels of performance, large compute
investors most likely encounter them first.

  * •

Dangerous capabilities proliferate over time. As efficiency improvements
increase the performance available with a given compute investment, more
actors become able to train and develop models with potentially dangerous
capabilities, complicating coordination and, therefore, increasing the chance
of accidents and misuse.

  * •

Deploying leading AI models defensively could strengthen resilience against
misuse and accidents. More speculatively, large compute investors maintain a
performance advantage and could use their more capable models to counteract
harm caused by weaker models. In addition to this performance edge, large
compute investors also possess a larger budget for compute resources, which
enables them to deploy a greater number of model copies or to operate them
with enhanced efficiency and speed. For example, cybersecurity tools powered
by advanced models could fix vulnerabilities before weaker models can exploit
them. However, some domains, such as biotechnology, may greatly favor the
offense, making it difficult to defend against dangerous deployments even with
superior models.

Adjusting our model assumption to the real world further emphasizes the
outsized role of large compute investors (Section 5):

Figure 3: Compute investment scaling increases the performance lead of large
compute investors over time. The dashed arrows represent performance
attainable without investment scaling.

  * •

Leaders increase their performance advantage through increasing investments
and proprietary algorithmic advancements. Large compute investors have
historically increased their compute investment significantly faster than
others, widening the investment gap to smaller actors. Additionally, the
proprietary development of algorithmic and hardware enhancements might further
widen this divide, consolidating their competitive advantage.

  * •

Leaders can release their model parameters and, therefore, allow others to
overcome compute investment barriers. While product integrations and
structured access protocols allow for continuous moderation, releasing model
parameters causes irreversible proliferation, foregoing any control.

Key Conclusions (Section 6)

  * •

Oversight of large compute investors helps to address the most severe risks.
If the most severe risks from AI development originate from the most capable
models and their large-scale deployment, then regulating large-scale compute
users can address the most severe risks. For instance, governments could
mandate capability evaluation and risk assessment procedures for training
large models, monitor large-scale deployments, and quickly intervene when
models cause harm.

  * •

Large compute investors should help mitigate risks from the proliferation of
advanced capabilities. The effectiveness of societal measures to mitigate harm
from proliferation hinges on the time that passes between large compute
investors disclosing harmful capabilities and their proliferation to malicious
or irresponsible actors. To extend this critical period, governments can
implement information-sharing frameworks with large compute investors and
thoroughly evaluate the risks posed by capability proliferation. Additionally,
leaders should invest in and provide defensive solutions that can be widely
adopted before offensive capabilities proliferate.

  * •

Governments should respond early to offense-dominant capabilities. In the
future, AI models of a given performance could develop heavily offense-
dominant capabilities or become inherently uncontrollable. Governments should
closely monitor the emergence of such capabilities and preemptively develop
mechanisms that could restrict their proliferation.

###### Contents

  1. Executive Summary
  2. 1 Introduction
    1. 1.1 Hardware Price Performance
    2. 1.2 Algorithmic Efficiency
    3. 1.3 Compute Investment Efficiency
  3. 2 Formal Model
  4. 3 Effects of Compute Efficiency Increases
    1. 3.1 The Access Effect
    2. 3.2 The Performance Effect
  5. 4 Consequences
    1. 4.1 Consequences of the Effects for Different Actors
    2. 4.2 Implications for Competitive Advantage
      1. Capability Diffusion Enables Innovation
      2. Threshold Effects
      3. Performance Ceilings
    3. Zero-Sum Competition
    4. 4.3 Implications for Dangerous Capabilities
      1. Large Compute Investors Encounter Dangerous Capabilities First
      2. Dangerous Capabilities Proliferate Over Time
      3. Using Advanced Models for Defense
      4. Importance of Offense-Defense Balance
  6. 5 Additional Factors Influencing the Distribution of Capabilities
    1. 5.1 Compute Scaling
    2. 5.2 Unequal Distribution of Compute Efficiency Improvements
    3. 5.3 Alternative Means of Diffusion: APIs and Release of Model Parameters
  7. 6 Discussion
    1. 6.1 Since Dangerous Capabilities First Originate from Large Compute Investors, They Should Face Particular Scrutiny
    2. 6.2 As Capabilities Proliferate Widely, Addressing the Harms They Cause Becomes Increasingly Challenging
      1. Using Advanced Models for Defense
    3. Limiting Development and Diffusion of Dangerous Capabilities
  8. 7 Conclusion
  9. Acknowledgements

##  1 Introduction

Over the past decade, computational resources (compute) emerged as a major
driver of rapid advances in the field of artificial intelligence (AI). The
amount of computational operations used in training the largest AI models has
doubled approximately every six months since 2010 (Sevilla et al., 2022).
Increasing training compute delivered numerous scientific breakthroughs and
enabled qualitatively new applications, such as high-quality image generators,
coding assistants, and conversational chatbots (Yu et al., 2022; Chen et al.,
2021; OpenAI, 2023). Researchers have discovered empirical scaling laws that
formalize the relationship between increased training compute and improved
model performance across numerous domains.444Scaling laws have been found in a
variety domains, such as natural language processing (Kaplan et al., 2020;
Hoffmann et al., 2022) and other specialized domains like protein structure
prediction, acoustic modeling, and recommender models (Chen et al., 2023;
Droppo and Elibol, 2021; Ardalani et al., 2022). These scaling laws and the
current pace of training compute growth strongly indicate that compute will
continue to drive increased model performance for the foreseeable future
(Sevilla et al., 2022; Villalobos, 2023).555Our model assumes that compute is
a main determinant of an AI model’s capabilities. However, note that, in
reality, training AI models on large amounts of compute is a sophisticated
engineering challenge that requires talented researchers. Thus, simply having
access to large amounts of compute does not guarantee a powerful AI model.

### Falling Training Cost

As compute budgets have surged, the cost of training an AI model to a given
performance has fallen continuously. In 2017, training an image classifier to
93% classification accuracy on the ImageNet dataset cost over $1,000; by 2021,
it cost only $5 – a reduction of over 99% (Zhang et al., 2022, p. 97). In
2020, OpenAI’s GPT-3 cost at least $4.6 million in cloud compute to train (Li,
2020); two years later, the company Mosaic claimed to achieve the same
performance for a tenth of the price (Venigalla and Li, 2022).

The cost of training falls due to two primary factors: (a) improvements in the
price performance of AI hardware and (b) improvements in the efficiency of AI
algorithms.

###  1.1 Hardware Price Performance

Hardware price performance determines the quantity of computational resources
available for a given investment. Innovations in hardware manufacturing and
design increased price performance over time. For example, consumer prices for
Intel’s processors fell by 42% annually from 2004 to 2013, while their
performance improved by 29% each year; both trends increased the computational
performance available per dollar (Byrne et al., 2017). In the context of AI
accelerators, Hobbhahn et al. (2023) found that GPUs used in machine learning
doubled in price performance approximately every two years. In addition, the
introduction of hardware-supported lower-precision number formats further
increased price performance by up to an order of magnitude (Hobbhahn et al.,
2023).

These trends are driven in part by a continued increase in the transistor
density of chips, popularly known as Moore’s Law (Roser et al., 2023), and by
specialized chip design (Hobbhahn et al., 2023). Beyond nominal chip
performance, improvements in other factors like power and cooling can also
lower the cost of operation and therefore improve price performance (Katal et
al., 2023).

###  1.2 Algorithmic Efficiency

Falling AI training costs stem not just from more compute being available per
dollar but also from ongoing improvements in algorithmic efficiency.
Algorithmic efficiency determines the computational resources required to
execute a given algorithm. In the context of AI, this identifies the compute
budget required to achieve a given level of model performance.

Due to innovations across the AI stack, algorithmic efficiency has improved
dramatically over the past decade. For example, Hernandez and Brown (2020)
found that over the period of 2012 to 2019, improvements in image
classification algorithms had led to a 97.7% reduction in the compute required
to train a classifier to the performance of AlexNet. Erdil and Besiroglu
(2023) extended the analysis to 2022 and found a faster rate of algorithmic
improvement in image classification, in which compute requirements halve every
nine months.666Rather than smoothly doubling every nine months, algorithmic
improvements usually occur in disjunct innovations. For instance, in image
classifiers, those innovations included sparsity, batch normalization, and
residual connections (Hernandez and Brown, 2020). Although both of these
studies are limited to image classification algorithms, algorithmic
improvements have had impact in many other subfields. For example, forthcoming
work by Besiroglu et al. analyzes the rate of algorithmic progress in large
language models (LLMs) and finds an average halving of training compute
requirement every 10 to 25 months.777Note that both studies focus on
improvements in model architecture and thus overlook improvements in lower
levels of the software stack, such as compiler optimization, which allow more
efficient utilization of the underlying hardware.

###  1.3 Compute Investment Efficiency

Figure 4: Hardware price performance is the conversion function between the
training compute investment in dollars and the training compute budget in
operations. Algorithmic efficiency is the subsequent conversion function
between the training compute budget and the performance of the resulting AI
model. Compute investment efficiency combines hardware price performance and
algorithmic efficiency, relatingf training compute investment to the
performance of the resulting model.

We introduce the concept of _compute investment efficiency,_ or _compute
efficiency_ for short, to refer to the relationship between the monetary
spending on training compute and the performance of the resulting AI model
(Figure 4).888The model presented in this paper and many of its implications
draw heavily from Tucker et al. (2020) “Social and Governance Implications of
Improved Data Efficiency”. We translate the concepts and implications
discussed in the context of increased data efficiency to the realm of compute
efficiency. Tracking compute efficiency over time captures advances in both
the hardware and software used for AI training, including those advancements
that depend on a mixture of hardware and software breakthroughs.999For
instance, (Hobbhahn et al., 2023) identify the recent shift from FP32
precision to tensor-FP16/8 as one of the primary drivers of AI hardware
improvements.

In the following sections, we first build a formal model of improvements to
compute investment efficiency and their impacts on various actors. Next, we
evaluate the model’s implications for competitive advantage and for the risks
of dangerous capabilities. Finally, we provide a short discussion of relevant
insights for policymakers and any others seeking to understand or respond to
increased compute efficiency.

##  2 Formal Model

This section formalizes our model assumptions and defines key terms. In the
context of compute efficiency, the paper distinguishes an AI model’s
performance from the resulting capabilities:

  1. 1.

_Performance_ refers to a hard metric, such as the loss or the accuracy on a
test dataset.

  2. 2.

_Capabilities_ refer to a more qualitative metric, such as the problems an AI
model can solve in the real world. They are, therefore, more difficult to
quantify.

For illustration, a language model that achieves a certain _performance_ on
next-word prediction may gain the _capability_ to solve coding problems. While
the performance of models increases continuously, capabilities may emerge
abruptly and are more difficult to predict (Ganguli et al., 2022).

Note that in our context, performance explicitly refers to direct performance
measures rather than the model’s performance on benchmarks. Depending on their
nature, benchmarks can capture either the performance of a model or its
capabilities. For instance, the ImageNet benchmark (Deng et al., 2009)
directly evaluates the models’ ability to classify images, the task they were
explicitly trained for. On the other hand, diverse benchmarks such as Massive
Multitask Language Understanding (MMLU) (Hendrycks et al., 2021) primarily
evaluate models’ usefulness on real-world tasks such as answering scientific
exams and legal questions.

### Algorithmic Efficiency

The performance $p$ achievable with a compute budget $b$ depends on the
algorithmic efficiency $f_{a}$ (assuming sufficient access to training data).

$p=f_{a}(b),$

where:

$\displaystyle p$ $\displaystyle=\text{Performance (accuracy of the AI
model)}$ $\displaystyle b$ $\displaystyle=\text{Compute budget (in
operations)}$ $\displaystyle f_{a}$ $\displaystyle=\text{Algorithmic
efficiency}$

Algorithmic efficiency improves over time. Hence the same compute budget
yields improved performance:

$f_{a}^{t+1}(b)>f_{a}^{t}(b),$

where:

$\displaystyle t$ $\displaystyle=\text{time}$

### Hardware Price Performance

Meanwhile, the compute budget $b$ available for training depends on the
compute investment $i$ in dollars and the hardware price performance $f_{h}$
that converts the compute investment into the compute budget.

$b=f_{h}(i),$

where:

$\displaystyle b$ $\displaystyle=\text{Compute budget (in operations)}$
$\displaystyle i$ $\displaystyle=\text{Compute investment (in $\$$)}$
$\displaystyle f_{h}$ $\displaystyle=\text{Hardware price performance (in
operations/$\$$)}$

As hardware price performance improves over time, the same compute investment
yields a larger compute budget:

$f_{h}^{t+1}(i)>f_{h}^{t}(i)$

In other words, compute gets cheaper over time.

### Compute Investment Efficiency

Compute investment efficiency $f_{c}$ combines hardware price performance
$f_{h}$ and algorithmic efficiency $f_{a}$ into one term. It thus determines
the performance obtainable with a given financial investment:

$f_{c}(i)=f_{a}\left(f_{h}(i)\right),$

where:

$\displaystyle f_{c}$ $\displaystyle=\text{Compute investment efficiency}$
$\displaystyle i$ $\displaystyle=\text{Compute investment (in $\$$)}$
$\displaystyle f_{h}$ $\displaystyle=\text{Hardware price performance (in
operations / $\$$)}$ $\displaystyle f_{a}$ $\displaystyle=\text{Algorithmic
efficiency}$

This simplifies the relationship between performance and investment to:

$p=f_{c}(i)$

where:

$\displaystyle p$ $\displaystyle=\text{Performance}$ $\displaystyle f_{c}$
$\displaystyle=\text{Compute investment efficiency}$ $\displaystyle i$
$\displaystyle=\text{Compute investment (in $\$$)}$

Since both $f_{a}$ and $f_{h}$ increase over time, $f_{c}$ similarly increases
over time.

$f_{c}^{t+1}(i)>f_{c}^{t}(i)$

In other words, the performance attainable with a given compute investment
increases over time.

##  3 Effects of Compute Efficiency Increases

Similar to the effects of improved data efficiency in Tucker et al. (2020), we
model the impact of increased compute efficiency through two effects: an
access effect and a performance effect. We then extrapolate these effects to
assess how capabilities emerge and proliferate over time.

###  3.1 The Access Effect

Figure 5: The access effect. Between $t=0$ and $t=1$, compute efficiency
reduces the compute investment required to achieve a given level of
performance. Note that throughout the paper, the presented figures are merely
conceptual and do not assert specific claims regarding the slopes of the
curves.

Compute efficiency improvements decrease the investment required to obtain a
given level of performance, which can be visualized as a leftward shift on the
performance vs. investment graph (Figure 5). Compute efficiency improvements,
thus let an increasing number of actors access models of a given performance.

###  3.2 The Performance Effect

Figure 6: The performance effect. As compute efficiency improves between time
$t=0$ and $t=1$, it increases the performance available with a given level of
compute investment.

At the same time, compute efficiency improvements increase the performance of
a given level of compute investment, which can be visualized as an upward
shift in the performance vs. investment graph (Figure 6). Compute efficiency
improvements thus grant any actor access to increased performance.

##  4 Consequences

The performance and access effects bring about a range of consequences, which
we explore in this section. First, we illustrate the direct effects of our
model through an example with three types of actors. We then investigate what
consequences our model suggests for competitive advantage, and finally outline
the implications for dangerous capabilities.

###  4.1 Consequences of the Effects for Different Actors

To illustrate the impact of increased compute efficiency, we consider a
scenario with three groups of actors:

  1. 1)

Large compute investors, who are able to spend large amounts on training
compute in order to advance the frontier of AI capabilities. Today, this group
includes frontier AI developers like OpenAI, Anthropic, or Google DeepMind; in
the future, it might include governments that decide to prioritize AI
development.

  2. 2)

Secondary actors, who are able to make significant investments in training
compute but who do _not_ attempt to advance the frontier. Examples include
large companies that integrate some AI-based tools into their products and
small companies developing AI models on a limited budget.

  3. 3)

Compute-limited actors, such as individuals, small collaborative projects, or
academic researchers, who can only access relatively inexpensive hardware.

For simplicity, we make the following assumptions about these groups (each of
which will be relaxed in Section 5):

  1. 1.

Actors maintain a constant level of compute investment over time.

  2. 2.

All actors benefit equally from compute efficiency improvements.

  3. 3.

Actors only have access to models they train themselves.

We further assume that a model attains some relevant “Capability $X$” when it
surpasses a given level of performance on the task. We now extrapolate
repeated efficiency improvements and their effect on access to “Capability
$X$”.

Figure 7: Diffusion of Capability $X$ over time among three illustrative
actors with varying levels of compute investment. At time $t=0$, no actor has
access to Capability $X$ with the performance enabled by their current level
of compute investment. The large compute investor first achieves sufficient
performance to attain Capability $X$ as compute efficiency increases at time
$t=1$. While performance also increases for the two smaller actors, it is
still insufficient to cross the performance threshold. At time $t=2$, further
compute efficiency improvements also enable the secondary actor with its
moderate compute budget to access Capability $X$. Finally, after ongoing
improvements at time $t=3$, even the individual can afford the compute
required for Capability $X$. Meanwhile, increased compute efficiency already
grants the secondary actor and the large compute investor access to a
substantially higher level of performance.

This extrapolation suggests the following insights:

##### 1\. Large compute investors are the first to discover novel
capabilities.

As the performance effect increases the performance attainable to every actor,
large compute investors are the first to discover novel capabilities (Figure
7, $t=1$). For example, OpenAI’s GPT-4 – according to estimates the most
training-compute-intense model to date (Epoch, 2023) – showed previously
unknown capabilities in mathematical reasoning, coding, and psychology
(OpenAI, 2023; Bubeck et al., 2023).

##### 2\. Access to capabilities expands over time.

As compute efficiency continues to improve, the investment required to achieve
a given level of performance decreases. Over time, this lets increasingly
compute-limited actors access advanced capabilities (Figure 7, $t=2$).
Eventually, even actors with very limited compute investments, such as
research collectives and labs or individuals, can access capabilities
previously restricted to large investors (Figure 7, $t=3$).

3\. Large compute investors can maintain a performance advantage as access
expands. This is because the same efficiency increase that lets a smaller
investor access the previous state-of-the-art often101010This rule can break
down if there is a ceiling on technical performance (see “Performance
Ceilings” below). increases the current state-of-the-art as well (Figure 7,
$t=3$).111111However, diminishing returns cause improvements for large compute
investors to be smaller than those for compute-limited actors. Depending on
how quickly returns diminish, this could cause similar effects to a
performance ceiling, as discussed in the next section. Assuming it is possible
to derive additional value from increased performance, large investors will
maintain an advantage. We explore this topic in more detail in the following
sections.

###  4.2 Implications for Competitive Advantage

Our model indicates several impacts on the competitive advantage of different
actors in the AI market.121212Note that our model is highly simplified and
merely conceptual and thus has limited predictive power for the market impacts
of compute efficiency improvements.

#### Capability Diffusion Enables Innovation

Due to the access effect of improved compute efficiency, a broader range of
actors can now potentially train customized AI models, resulting in a higher
variety of products. However, our model suggests only a limited impact on
competition, given that large compute investors maintain an absolute
performance advantage and thus continue to develop the most advanced models.
Hence, smaller companies likely focus on niche applications where they face
limited competition from leaders.

For instance, various companies currently develop smaller language models that
perform significantly below OpenAI’s GPT-4 on general benchmarks (Papers with
Code, 2023) but outperform it in specialized niches, such as finance (Wu et
al., 2023), coding assistance (AWS, 2023b; Chen et al., 2021), or role-playing
for entertainment (Character.AI, 2023).

Eventually, efficiency gains enable even compute-limited actors like
individuals and academic labs to deploy more capable AI models. Because these
groups have a broader range of interests than commercial entities do, this
diffusion could represent a qualitative shift in AI applications that enables
a much wider variety of use cases (Besiroglu et al., forthcoming; Ahmed and
Wahed 2020).

#### Threshold Effects

AI models sometimes show sudden improvements in capabilities even when compute
investment increases steadily. Such performance thresholds often occur in
large language models, such as when tested on three-digit addition or general
benchmarks like MMLU (Ganguli et al., 2022). Sometimes, performance thresholds
seemingly occur despite the model’s accuracy continuing to increase at a
steady rate, prompting an ongoing discussion about whether some “emergent
capabilities” are impossible to predict ahead of time (Wei et al., 2022;
Schaeffer et al., 2023; Wei, 2023).

Performance thresholds play a vital role in the usefulness of a model. For
example, language models trained with small amounts of compute struggle to
generate coherent sentences; with more compute, they eventually cross the
performance threshold of producing text of sufficient quality to power
chatbots and other applications.

Most performance thresholds are gradual. In the previous example, the
capability to generate coherent text is not a discrete jump but rather a rapid
but continuous improvement. However, discrete performance thresholds occur if
real-world applications require a predefined level of capabilities. For
instance, regulations may require the error rate of an image classifier used
in autonomous vehicles to be below a certain limit before it becomes
acceptable to deploy it in traffic.

Since large compute investors can access the highest level of performance,
they are the first to cross performance thresholds, allowing them to dominate
the market until other actors follow, as illustrated in Figure 7.

#### Performance Ceilings

As developers scale their training compute investment, they may eventually
encounter a region where further compute investment only marginally (if at
all) increases a model’s usefulness for real-world tasks. For example, the
problem of identifying handwritten digits was effectively solved in the early
2000s, with classifiers reaching over 99% accuracy on the MNIST dataset (Liu
et al., 2003). Similarly, face recognition technology has already achieved
impressively low error rates, down to 1:1000 on the VISA dataset (NIST, 2023).
Such performance ceilings can stem from either technical limitations or real-
world constraints.

Figure 8: Performance ceilings dampen the performance effect for large compute
investors, so they only make marginal progress. Meanwhile, the access effect
continues to grant smaller actors access to higher performance until they also
approach the ceiling.

#### Technical Performance Ceilings

Hestness et al. (2017) propose that, with sufficient training compute, an AI
model may encounter an irreducible error region where its accuracy can no
longer increase because the fixed training set and model architecture have
extracted all possible information relevant to the task. However, they only
find this irreducible error in a highly simplified toy experiment and suggest
that most AI applications are still far away from the irreducible error
region. Furthermore, in some areas, such as natural language processing,
humans continuously generate new data, meaning that the model may practically
never approach an irreducible error region.

#### Practical Performance Ceilings

Performance ceilings also occur when performance continues to increase but no
longer contributes to advances in real-world capabilities. For instance, for a
voice-controlled application designed to recognize only a fixed set of
commands, improving the underlying model’s accuracy on less frequent words
will not enhance its real-world functionality. Similarly, a low-performance
language model used in a content filter may detect most harmful phrases
already, so further advances improve its usefulness only marginally.

#### Implications

A performance ceiling presents an upper bound for an AI model’s capabilities
and thus dampens the performance effect, reducing the benefits leaders get
from improved compute efficiency. Since the access effect remains unchanged,
compute efficiency improvements allow smaller actors to move closer to the
leaders who are stuck at the performance ceiling (Figure 8). The performance
ceiling, therefore, distributes capabilities evenly, and large compute
investors slowly lose their performance lead.

However, under some circumstances, leaders can entrench their initial
advantage and continue to dominate the field even as their performance
advantage diminishes. Such _winner-take-all effects_ occur when large compute
investors integrate AI models into their ecosystems and thus create lock-in
effects.131313For instance, Google currently integrates its chatbot Bard into
its search engine and the Google workspace (Google, 2023). Such integration
may increase customers’ switching costs, so they may continue using Google’s
AI model, even if competitors offer similar-performing models. For instance,
large compute investors may further use their early profits to build economies
of scale or apply exclusionary practices against later competitors (Kuchinke
and Vidal, 2016).

### Diminishing Returns

Even without an apparent performance ceiling, AI models usually show reducing
improvements from increasing compute investment or compute efficiency (Kaplan
et al., 2020). If returns diminish quickly, the field may encounter an effect
similar to a performance ceiling, where the leaders’ advantage reduces with
increasing compute efficiency.

#### Zero-Sum Competition

In certain areas, a relative performance advantage compared to competitors can
lead to outsized benefits, even if the absolute performance difference is
small. For instance, in chess or go, a somewhat better AI model will
consistently win against its opponent, even if both models achieve a high
performance in absolute terms.

Zero-sum competition141414Note that while leaders in these markets capture
outsized benefits, they may not constitute zero-sum games in an economic
context. The overall market can still expand due to rising model capabilities,
effectively resulting in a positive-sum game where increased competition
drives market growth. that strongly favors the leader also occurs in the real
world, such as in algorithmic trading, where a marginal speed advantage allows
a model to exploit tiny market inefficiencies before competing models can
(Biais et al., 2015). Further areas of direct competition between future AI
models may include the legal and the entertainment sectors, both of which
reward the strongest competitor with outsized benefits. Zero-sum competition
allows a leader to continue dominating a market, even if they only have a
small absolute performance advantage. This implies that they may be able to
sustain their leadership even as they approach a performance ceiling.

###  4.3 Implications for Dangerous Capabilities

Today’s AI models can already be used to cause harm, for instance, by
producing racist, sexist, or otherwise discriminating content or by aiding the
suppression of minorities or political views (McGregor, 2021; Weidinger et
al., 2021; Creemers, 2018). As models become more performant, they may develop
capabilities that pose acute risks to safety and security, such as by
automating cyberattacks (Amodei, 2023), aiding in the design of bioweapons
(Urbina et al., 2022; Nelson and Rose, 2023), generating targeted
disinformation (Sedova et al., 2021), or irreversibly evading human control
(Carlsmith, 2022; Ngo et al., 2023). Although some dangerous capabilities may
arise in some highly specialized models that require only small amounts of
compute, the majority are likely to first appear in compute-intensive general-
purpose models (Anderljung et al., 2023).151515Not all dangerous capabilities
are concentrated in models requiring large amounts of compute. For instance,
specialized models that predict host-pathogen interaction for viral proteins
could significantly aid bioweapon development but may not require large
amounts of training compute. For instance, AlphaFold, an advanced protein
predictor, required only $10^{20}$ FLOP, three orders of magnitude less than
OpenAI’s GPT-3 developed in the same year (Senior et al., 2020; Brown et al.,
2020; Epoch, 2022). Biological design tools similar to AlphaFold could play a
crucial role in the development of bioweapons (Sandbrink, 2023).

#### Large Compute Investors Encounter Dangerous Capabilities First

As discussed, we expect that novel capabilities are generally encountered
first by large compute investors. This rule is likely to hold equally for
benign and dangerous capabilities. However, not all such capabilities may be
immediately evident. Without rigorous testing, developers might overlook
inherent risks in their models (Shevlane et al., 2023). Such risks may become
apparent only after being made broadly available, including to potential
malicious actors (Anderljung et al., 2023).

Figure 9: Example of three actors discovering novel dangerous capabilities $X$
and $Y$. At $t=1$, the large compute investor is the first to encounter
dangerous capability $X$. After further compute efficiency improvements at
$t=2$, the secondary actor (such as a small startup or a large company that
only secondarily invests in AI development also encounters dangerous
capability $X$. However, by that time, the large compute investor has already
encountered an even more dangerous capability $Y$.

#### Dangerous Capabilities Proliferate Over Time

When a dangerous capability first emerges, access is likely limited to large
compute investors. Over time, however, compute efficiency improvements will
give more and more actors access to that capability. This complicates
coordination and oversight, making it potentially difficult to entirely avoid
harmful deployment.

#### Using Advanced Models for Defense

One way to mitigate risks from the proliferation of dangerous capabilities is
to invest in defensive measures. For example, leading actors can help limit
the risks posed by capability diffusion through strengthening resilience by
leveraging their more advanced models. In particular, our model suggests that
large compute investors may be able to use their ongoing performance advantage
to detect and address threats posed by irresponsible or malicious actors
(Anderljung et al., 2023, p. 25). For instance, large compute investors may
offer cybersecurity tools for automatic threat detection and response that
defend against attacks by less capable models developed by rogue actors (Lohn
and Jackson, 2022). Similarly, automated detection of disinformation could
limit the impact of AI on epistemic security (Demartini et al., 2020). Even as
a model approaches a performance ceiling, large compute investors may be able
to provide effective defensive measures by leveraging their superior quantity
of inference compute (e.g., by deploying more and/or faster model
instances161616A model instance refers to a single copy of a particular AI
model. Once a model is trained, the developer can run many such instances
using the available supply of compute.).

#### Importance of Offense-Defense Balance

Nonetheless, some applications of AI capabilities may inherently favor
offensive use, making it difficult to defend against them even with advantages
in performance or scale of deployment. (Garfinkel and Dafoe, 2019) reviewed
how the balance between offense and defense scales with increased investment
across a number of military scenarios and found significant differences
between them. For example, higher spending favors the defender in ground
warfare and drone swarms but the offender in missile attacks, indicating that
the offense-defense balance may differ considerably between domains. Another
relevant example comes from biology: even a very capable toxicology predictor
may not easily find cures for poisonous agents developed by a somewhat weaker
model.

The offense-defense balance of advanced capabilities may be particularly
decisive when a field approaches a performance ceiling. In this case, there
may be little or no performance advantage to count on; therefore, the
feasibility of defending against irresponsible or malicious use depends on
whether the proliferating capability fundamentally favors offense or defense.

Even without a ceiling, model proliferation in some particularly offense-
dominant domains, such as biotechnology, may pose an unacceptable risk. This
is particularly concerning given the unpredictable nature of AI progress makes
it possible that future technologies could cause harm on a large scale
(Bostrom, 2019).

##  5 Additional Factors Influencing the Distribution of Capabilities

Given its simplicity, our model faces some limitations, which we will discuss
in this section.

###  5.1 Compute Scaling

Figure 10: Illustration of the effect of compute scaling. The ability of large
compute investors to rapidly scale their investment greatly increases the
performance gap between them and other actors, who only have limited capacity
to scale. The dotted arrows represent the performance each actor would achieve
before scaling their investment. The solid arrows represent the actual
performance achieved due to increased investment.

So far, our model has assumed that compute investment stays constant over
time. However, in reality, AI companies have massively increased their
investments, which rose thirtyfold between 2013 and 2021 (Roser, 2023). This
exponential increase in investment is primarily concentrated in large compute
investors, allowing them to double their training compute investment every two
years (Cottier, 2023). For example, training GPT-3 cost OpenAI about $4.6
million in 2020 (Li, 2020), while GPT-4 reportedly cost up to $100 million in
2023 (Knight, 2023).

Although scaling compute presents a significant engineering
challenge,171717For instance, parallelizing AI training across larger and
larger clusters makes it hard to avoid bottlenecks that reduce utilization.
success grants large compute investors an additional performance advantage and
thus increases their lead over less well-resourced actors (Figure 10).
Besiroglu et al. (forthcoming) identify the widening gap between the compute
investment of industry and academia as a compute divide (Our World in Data,
2023). Although large compute investors can only maintain exponential scaling
for a limited time (Heim, 2023a; Lohn and Musser, 2022), increasing investment
will continue to extend their lead for at least the near future.

### Data as a Bottleneck

The increasing training compute budget stems from training models on more
data. However, there is a potential bottleneck if one were to run out of data.
While the availability of high-quality data is a crucial determinant for
performance in many specialized domains (Budach et al., 2022), it does not
currently limit scaling training compute for general-purpose models that rely
on non-specialized data (Patel, 2023). Yet, if current data scaling continues
with these increased compute training budgets, it could exhaust high-quality
language data by 2026 (Villalobos et al., 2022). A concentration of high-
quality data within a specific actor could considerably shift the competitive
advantage, reducing the effects of compute efficiency. However, algorithmic
advances increase not only compute efficiency, but also data efficiency – the
amount of data required to achieve a given level of performance (Tucker et
al., 2020).

###  5.2 Unequal Distribution of Compute Efficiency Improvements

Our model has assumed that all actors benefit from a compute efficiency
improvement. In reality, compute efficiency improvements are often distributed
unequally. One source of inequality is access to innovation driven by the
workforce. Large compute investors can afford large teams and attract top
researchers, often allowing them to discover algorithmic advances first. Given
the importance of such advances to their profitability, frontier AI developers
are increasingly reluctant to publish those advances, given their relevance to
competitive advantage and their potential misuse. For example, in 2019, OpenAI
published details on the architecture, data, and algorithms used for GPT-2,
whereas in 2023, GPT-4’s publication included almost no such information
(Radford et al., 2019; OpenAI, 2023). As a result, algorithmic advances are
increasingly concentrated at frontier AI developers, diffusing only through
leaks or parallel innovation.181818Over the last year, we observe that
publication norms have entered a new phase. Frontier AI developers are
reluctant to share even basic details of their models, such as architecture
and compute used, making it challenging to estimate their internal compute
efficiency. This further reduces the diffusion of compute efficiency
improvements, which now primarily spread through employees switching
companies.

Additionally, several frontier AI companies or partners – such as Google,
Meta, and Amazon (Jouppi et al., 2023; Janardhan, 2023; AWS, 2023a) – develop
their own hardware. While some of these companies offer external access to
their custom hardware through cloud services, internal teams likely incur
lower costs and benefit from early access. Custom hardware, therefore,
potentially confers a unique price performance advantage, further increasing
the performance gap between frontier AI labs and smaller
actors.191919Potentially, different frontier AI developers will design their
own architectures, including optimized hardware, making it increasingly
difficult to assess and compare their models. 202020Should AI models become
sufficiently capable to significantly enhance R&D in hardware or algorithms,
the performance advantage of large compute investors could allow them to
deploy more and more capable automated workers that might further enhance the
compute efficiency available to them.

###  5.3 Alternative Means of Diffusion: APIs and Release of Model Parameters

Our model has assumed that actors can only access capabilities of models they
can train themselves. However, this perspective misses other avenues through
which compute-limited actors can gain access to state-of-the-art AI models.

### API and Product Integrations

Application programming interfaces (APIs) or AI models integrated into
products offer compute-limited actors access to otherwise inaccessible
advanced capabilities. Typically, users can only query such models without
being able to make changes to the model, but some APIs also offer fine-tuning
functionality, thus allowing users to partly retrain custom models (Peng et
al., 2023). In this access pattern, the model developer is able to maintain
control over the interaction, such as by charging for use and implementing
safeguards to prevent misuse and accidents.

### Public Release of Model Parameters

Some major developers with significant compute investment have publicly
released the parameters of their advanced models. For example, Meta released
the parameters of Llama 2 for both research and commercial use in July 2023
(Meta, 2023). This access pattern grants third parties unrestricted access to
the model, foregoing the ability to implement user guidelines or charge fees
for usage later. While this allows independent researchers to scrutinize the
models produced by large compute investors, it also significantly increases
the risk of misuse by irresponsible actors. Seger et al. (2023) suggest that
structured access protocols may allow independent scrutiny while limiting the
risks from model parameters release.

### Industrial Espionage

Sophisticated actors like competitors, nation-states, or major terrorist
groups may try to infiltrate large compute investors and steal the parameters
of highly capable models. This poses risks not only to the developer’s
business prospects but also to society if stolen models are used for malicious
purposes (Nevo et al., 2023; Heim and Ladish, 2022).

##  6 Discussion

In this section, we discuss the broader implications of our findings, with a
focus on the governance and societal implications of risks. We argue that the
emergence of dangerous capabilities most likely occurs at frontier AI models
developed by frontier AI developers with access to substantial compute
resources. Thus, these frontier AI developers should be subject to increased
scrutiny and oversight. Furthermore, as advanced AI capabilities diffuse, the
challenge of mitigating these risks grows in complexity. We propose
recommendations for effective oversight and risk management.

###  6.1 Since Dangerous Capabilities First Originate from Large Compute
Investors, They Should Face Particular Scrutiny

Many of the most dangerous capabilities, such as hacking or social
manipulation, likely first arise in the highest-performing general-purpose AI
models. Such advanced AI models are concentrated at a small number of
developers who can afford large and rapidly increasing compute investments and
who often create their own improvements in algorithms and hardware, giving
them exclusive access to compute efficiency advances.

In choosing to train the most capable models, large compute investors have a
special responsibility to identify novel capabilities and the risks they pose.
To reliably detect potentially dangerous capabilities of their models,
developers can design and apply evaluations that test whether models are
capable of causing harm (Liang et al., 2023; Shevlane et al., 2023; Kinniment
et al., 2023). Furthermore, developers can employ risk assessment procedures
that review more complex risks that may not directly arise from a model’s
capabilities (Novelli et al., 2023; Koessler and Schuett, 2023).

While some frontier AI developers actively try to develop practices for
preventing harm caused by their AI models (Anthropic, 2023a), incentives for
industry-directed initiatives may not always align with addressing the most
severe risks. Therefore, governments should intensely scrutinize the decisions
and methodologies AI developers use, particularly targeting well-resourced
labs that concentrate on creating highly capable foundation models (Anderljung
et al., 2023).

### Overseeing Large-Scale Compute Infrastructure Might Address the Most
Severe Risks

Assuming that the worst risks stem from the most capable AI models and that
these models’ performance directly results from the size of their training
compute budget, then overseeing access to large-scale AI compute clusters may
present an effective way of regulation (Heim 2023b; Brundage et al.,
_forthcoming_). Moreover, the amount of compute available determines not only
the trained model’s capabilities but also the number of model instances the
developer can deploy. Controlling access to large-scale compute resources
could thus allow governments not only to monitor the most advanced
capabilities but also to quickly address harms caused by the large-scale
deployment of potentially dangerous models (O’Brien et al., 2023).

Currently, a large fraction of all AI compute likely belongs to a small number
of cloud providers that offer their resources as a service (Richter, 2023).
These companies have already occasionally restricted harmful services, such as
AWS banning the far-right social media site Parler from using its
infrastructure after the January 6 attack on the US Capitol (Fitzpatrick,
2021). To address proliferation risks, large cloud providers could introduce
more extensive Know Your Customer (KYC) regimes, monitor suspicious activity,
and develop methods to quickly shut down AI models that currently cause harm
(Egan and Heim, 2023; O’Brien et al., 2023). Besides avoiding harm, deployment
oversight could also become a tool for attributing harms caused and holding
actors liable (Anderljung and Hazell, 2023; Buiten et al., 2023).

###  6.2 As Capabilities Proliferate Widely, Addressing the Harms They Cause
Becomes Increasingly Challenging

Over time, the access effect causes an increasing number of actors to attain
dangerous capabilities previously limited to large compute investors. This
expansion complicates efforts to mitigate risks, as it becomes increasingly
complex to coordinate and infeasible to oversee every model with dangerous
capabilities.

Frontier AI developers play a key role in identifying and mitigating risks
associated with the proliferation of advanced model capabilities. Given their
knowledge advantage from developing the most capable models and their research
capacity due to designated safety and ethics teams, these labs are in a strong
position to investigate proliferation risks. To allow society to prepare for
the impact of proliferation, governments may mandate frontier AI developers to
share information about their advanced models publicly or with selected
government organizations (Mulani and Whittlestone, 2023; Department for
Science, Innovation and Technology, 2023). With insights into possible risks,
governments could then evaluate various measures aimed at either preventing
model proliferation or addressing its risks effectively.

Besides limiting the harmful use of proliferated models, governments may also
increase oversight of crucial inputs that allow malicious actors to cause harm
based on knowledge provided by these models (Anderljung and Hazell, 2023). For
instance, controls on laboratory equipment and DNA synthesis screening could
prevent malicious actors from creating dangerous pathogens even if the
knowledge required is widely available (The White House, 2023; DiEuliis et
al., 2017).

#### Using Advanced Models for Defense

While the access effect of compute efficiency improvements causes
proliferation, large compute investors continue to benefit from the
performance effect, allowing them to maintain a performance advantage over
proliferating models. This performance advantage potentially enables defensive
deployments of AI models to counteract risks caused by proliferation. However,
the feasibility of defending depends on several crucial factors, such as the
offense-defense balance, the gap between leaders’ and proliferated models, and
the regulatory environment.

#### Offense-Defense Balance

As discussed in Section 4.3, the offense-defense balance may vary greatly
between different areas. Should proliferating AI capabilities significantly
favor the offense, it may become infeasible to use other models for defense,
even if the latter are more advanced and deployed on a larger scale. To assess
the feasibility of defensive solutions ahead of time, regulators may need to
continuously monitor current capabilities and conduct experiments. However,
assessing and experimenting with advanced models will likely require frontier
AI developers to securely share their advanced models with trusted
researchers, potentially enabled by secure structured access protocols (Seger
et al., 2023; Bucknall and Trager, 2023).

#### Proportion of Leaders’ Performance Advantage

Besides the offense-defense balance, the effectiveness of using advanced AI
models for defense also depends on the performance gap between the defensive
models and the proliferated capabilities. Large compute investors can leverage
their considerable compute budgets and exclusive access to algorithmic and
hardware improvements to maintain a substantial lead over other actors,
allowing them to develop more capable defensive models.

The performance gap between offense and defense is largest when defending
against compute-limited actors like individuals, but well-resourced actors
like authoritarian states or irresponsible companies could train AI models
much closer in performance to the frontier. As capabilities approach
performance ceilings or diminishing returns set in, the gap shrinks, limiting
defense feasibility. However, even near a ceiling, leaders can potentially
leverage their large compute investment and access to compute in order to
deploy a large number of model instances for defense and thus maintain a
substantial advantage over less well-resourced actors using proliferated
models.

#### Regulatory Environment

Beyond just technical feasibility, the effectiveness of defensive AI
strategies may also depend on regulatory frameworks. Large compute investors
may not develop defensive measures by themselves, so governments may need to
actively contract them or initiate collaborative projects. Moreover, certain
defense approaches may require explicit regulatory permission in order to be
active in sensitive domains. For example, cybersecurity defenses might need
direct access to sensitive networks and permission to make real-time decisions
autonomously without human oversight – which brings new risks. Similarly,
protecting individuals from social manipulation through AI may require
monitoring private communications, raising significant privacy concerns.
Governments, therefore, may need to develop and enforce standards that
preserve privacy in these defensive models while still giving them sufficient
permissions to ensure their effectiveness.

### Limiting Development and Diffusion of Dangerous Capabilities

If AI models strongly favor offensive applications or are an inherent threat
due to uncontrollability, or if defensive solutions are too invasive,
governments may have to limit the diffusion of such capabilities in the first
place. Given the difficulty in accurately predicting these factors beforehand,
it would be prudent to establish precautionary mechanisms to manage potential
risks.

#### Misaligned AI Models

So far, we primarily discussed how to address the misuse of advanced AI
models. However, an increasing number of AI researchers emphasize the
possibility that future, highly capable general AI models could pursue
autonomous goals, thus posing an inherent threat (Carlsmith, 2022; Ngo et al.,
2023; Russell, 2019). They argue it will be difficult to ensure such models
are safe and controllable, a challenge described as the alignment problem.
Some frontier AI labs have already established designated research teams
focused on developing solutions to the safety of autonomous AI models with
near or above human-level capabilities (Leike and Sutskever, 2023; Anthropic,
2023b). The difficulty of solving the alignment problem remains widely
debated, and some leading experts recently voiced concerns that failing to do
so could cause human extinction (Center for AI Safety, 2023).

First, if advanced AI models are difficult to control, it could be hard to
leverage them for defense. Additionally, the gap between frontier AI
developers and others could narrow if frontier AI developers prioritize safety
over capability advances. Even if frontier AI developers accurately assess
that a model is uncontrollable and avoid deploying it, the proliferation of
the capabilities to create such models greatly increases the risk that someone
eventually will. This proliferation of misaligned models could have
catastrophic consequences.

#### Structured Access to AI Models Balances Scrutiny and Proliferation Risks

The period between the first discovery of a dangerous capability and its
proliferation to malicious actors is crucial for developing societal
resilience through regulation or defensive solutions. Hence, developers should
be cautious about hastening the diffusion of advanced AI models. In
particular, providing the parameters of advanced AI models causes the
immediate, irreversible proliferation of the model’s capabilities. Rather than
allowing anyone unrestricted access to advanced models, frontier AI developers
should introduce structured access procedures to provide their model
parameters only to responsible researchers (Seger et al., 2023; Solaiman,
2023; Shevlane, 2022; Bucknall and Trager, 2023).

#### Coordinated Pausing

Should AI capabilities reach an unacceptable level of danger or models become
too difficult to align, developers may have to halt further progress until
they can appropriately mitigate these risks. Alaga and Schuett (2023) suggest
that frontier AI developers could collectively pause the development of highly
capable models when they encounter capabilities deemed too dangerous. This
pause, however, is likely only effective when few actors have access to such
capabilities. If an extended pause is needed, the access effect will gradually
enable more actors to develop dangerous models. In this scenario, preventing a
growing number of entities from pursuing such development may require global
coordination and strict oversight of compute resources. Yet, even with these
measures, fully controlling the spread and use of these technologies may
remain an elusive goal.

##  7 Conclusion

We found that increased compute efficiency results both in an access effect –
making a given capability more accessible – and a performance effect –
enabling a higher performance level for a given compute investment. The
impacts of the two effects depend on how performance converts to usefulness in
real-world tasks. Threshold effects advantage large compute investors, who are
the first to discover qualitatively new capabilities. Meanwhile, performance
ceilings reduce the gap between leading and lagging actors, whereas, in direct
competition, even a marginal advantage allows outsized benefits for the
leaders.

Our model suggests that large compute investors are at the frontier of
capabilities and, therefore, potentially the first to discover dangerous
capabilities. To adequately address these capabilities, large compute
investors should implement extensive capability evaluation and risk assessment
procedures. To ensure responsible behavior, governments should increase
oversight of the compute resources required for frontier model development.

As compute efficiency increases, dangerous capabilities proliferate to an
increasing number of actors. The effectiveness of societal measures to
mitigate harm from this proliferation hinges on the duration between frontier
AI labs disclosing these capabilities and their proliferation to malicious or
irresponsible actors. To extend this critical period, governments should
implement information-sharing frameworks with leading AI labs and thoroughly
evaluate the risks of proliferation. These evaluations may necessitate access
to advanced models by governments or independent bodies.

Once informed about dangerous capabilities, governments should begin
increasing societal resilience against them. This can involve restricting
certain inputs required to cause harm, such as laboratory equipment or
compounds needed to develop dangerous pathogens. Furthermore, governments can
contract or coordinate with leading AI developers to use their advanced models
in defensive solutions that address risks caused by proliferation. However,
the feasibility of AI models used for defense critically depends on the
performance difference between the defending model and the proliferated one,
as well as the fundamental offense-defense balance in the field.

Although increasing compute efficiency makes AI capabilities more widely
available over time, regulating access to large-scale compute clusters can
still increase oversight. Specifically, monitoring the largest compute
clusters allows monitoring the most extensive training runs, which likely
produce the most advanced and potentially dangerous AI capabilities.

However, both compute oversight and defensive solutions may be inadequate
should sufficiently dangerous capabilities arise, such as AI models enabling
individuals to engineer dangerous pathogens or if AI models of a given
performance are inherently uncontrollable. Governments may need to
preemptively develop mechanisms that could restrict their development and
proliferation, such as moratoria on specific research areas to avoid large-
scale harm.

## Acknowledgements

We thank Aaron Tucker, Markus Anderljung, and Allan Dafoe for their paper
“Social and Governance Implications of Improved Data Efficiency” (Tucker et
al., 2020) which inspired this paper and first introduced many of the ideas
presented here. We would like to express our thanks to the people who have
offered feedback and input on the ideas in this paper, including Markus
Anderljung, Ben Garfinkel, Ben Harack, and many other GovAI team members. We
thank Wes Cowley for copy editing and José Medina for formatting assistance.

## References

  * Ahmed and Wahed [2020] Nur Ahmed and Muntasir Wahed.  The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research, October 2020.  URL http://arxiv.org/abs/2010.15581.  Issue: arXiv:2010.15581 arXiv:2010.15581 [cs].
  * Alaga and Schuett [2023] Jide Alaga and Jonas Schuett.  Coordinated Pausing: An Evaluation-Based Coordination Scheme for Frontier AI Developers | GovAI.  Research Paper, Centre for the Governance of AI, November 2023.  URL https://www.governance.ai/research-paper/coordinated-pausing-evaluation-based-scheme.
  * Amodei [2023] Dario Amodei.  Written Testimony of Dario Amodei, Ph.D. Co-Founder and CEO, Anthropic, July 2023.  URL https://www.judiciary.senate.gov/imo/media/doc/2023-07-26_-_testimony_-_amodei.pdf.  Subcommittee on Privacy, Technology, and the Law, United States Senate.
  * Anderljung and Hazell [2023] Markus Anderljung and Julian Hazell.  Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted?, March 2023.  URL http://arxiv.org/abs/2303.09377.  Issue: arXiv:2303.09377 arXiv:2303.09377 [cs].
  * Anderljung et al. [2023] Markus Anderljung, Joslyn Barnhart, Anton Korinek, Jade Leung, Cullen O’Keefe, Jess Whittlestone, Shahar Avin, Miles Brundage, Justin Bullock, Duncan Cass-Beggs, Ben Chang, Tantum Collins, Tim Fist, Gillian Hadfield, Alan Hayes, Lewis Ho, Sara Hooker, Eric Horvitz, Noam Kolt, Jonas Schuett, Yonadav Shavit, Divya Siddarth, Robert Trager, and Kevin Wolf.  Frontier AI Regulation: Managing Emerging Risks to Public Safety, November 2023.  URL http://arxiv.org/abs/2307.03718.  Issue: arXiv:2307.03718 arXiv:2307.03718 [cs].
  * Anthropic [2023a] Anthropic.  Anthropic’s Responsible Scaling Policy, September 2023a.  URL https://www.anthropic.com/index/anthropics-responsible-scaling-policy.
  * Anthropic [2023b] Anthropic.  Core Views on AI Safety: When, Why, What, and How, March 2023b.  URL https://www.anthropic.com/index/core-views-on-ai-safety.
  * Ardalani et al. [2022] Newsha Ardalani, Carole-Jean Wu, Zeliang Chen, Bhargav Bhushanam, and Adnan Aziz.  Understanding Scaling Laws for Recommendation Models, August 2022.  URL http://arxiv.org/abs/2208.08489.  Issue: arXiv:2208.08489 arXiv:2208.08489 [cs].
  * AWS [2023a] AWS.  Ai Accelerator - AWS Trainium - AWS, November 2023a.  URL https://aws.amazon.com/machine-learning/trainium/.
  * AWS [2023b] AWS.  AI Code Generator – Amazon CodeWhisperer – AWS, November 2023b.  URL https://aws.amazon.com/codewhisperer/.
  * Biais et al. [2015] Bruno Biais, Thierry Foucault, and Sophie Moinas.  Equilibrium fast trading.  _Journal of Financial Economics_ , 116(2):292–313, May 2015.  ISSN 0304-405X.  doi: 10.1016/j.jfineco.2015.03.004.  URL https://www.sciencedirect.com/science/article/pii/S0304405X15000288.  Number: 2.
  * Bostrom [2019] Nick Bostrom.  The Vulnerable World Hypothesis.  _Global Policy_ , 10(4):455–476, November 2019.  ISSN 1758-5880, 1758-5899.  doi: 10.1111/1758-5899.12718.  URL https://onlinelibrary.wiley.com/doi/10.1111/1758-5899.12718.  Number: 4.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language Models are Few-Shot Learners, July 2020.  URL http://arxiv.org/abs/2005.14165.  Issue: arXiv:2005.14165 arXiv:2005.14165 [cs].
  * Bubeck et al. [2023] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang.  Sparks of Artificial General Intelligence: Early experiments with GPT-4, April 2023.  URL http://arxiv.org/abs/2303.12712.  Issue: arXiv:2303.12712 arXiv:2303.12712 [cs].
  * Bucknall and Trager [2023] Benjamin S. Bucknall and Robert F. Trager.  Structured access for third-party research on frontier AI models: Investigating researchers’ model access requirements.  Whitepaper, Oxford Martin AI Governance Initiative, Centre for the Governance of AI, October 2023.  URL https://www.oxfordmartin.ox.ac.uk/publications/structured-access-for-third-party-research-on-frontier-ai-models-investigating-researchers-model-access-requirements/.
  * Budach et al. [2022] Lukas Budach, Moritz Feuerpfeil, Nina Ihde, Andrea Nathansen, Nele Noack, Hendrik Patzlaff, Felix Naumann, and Hazar Harmouch.  The Effects of Data Quality on Machine Learning Performance, November 2022.  URL http://arxiv.org/abs/2207.14529.  Issue: arXiv:2207.14529 arXiv:2207.14529 [cs].
  * Buiten et al. [2023] Miriam Buiten, Alexandre de Streel, and Martin Peitz.  The law and economics of AI liability.  _Computer Law & Security Review_, 48:105794, April 2023.  ISSN 0267-3649.  doi: 10.1016/j.clsr.2023.105794.  URL https://www.sciencedirect.com/science/article/pii/S0267364923000055.
  * Byrne et al. [2017] David M. Byrne, Stephen D. Oliner, and Daniel E. Sichel.  How Fast are Semiconductor Prices Falling?  _Review of Income and Wealth_ , 64(3):679–702, April 2017.  ISSN 1475-4991.  doi: 10.1111/roiw.12308.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/roiw.12308.  Number: 3 _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/roiw.12308.
  * Carlsmith [2022] Joseph Carlsmith.  Is Power-Seeking AI an Existential Risk?, June 2022.  URL http://arxiv.org/abs/2206.13353.  Issue: arXiv:2206.13353 arXiv:2206.13353 [cs].
  * Center for AI Safety [2023] Center for AI Safety.  Statement on AI Risk, 2023.  URL https://www.safe.ai/statement-on-ai-risk.
  * Character.AI [2023] Character.AI.  character.ai, November 2023.  URL https://beta.character.ai/help.
  * Chen et al. [2023] Bo Chen, Xingyi Cheng, Yangli-ao Geng, Shen Li, Xin Zeng, Boyan Wang, Jing Gong, Chiming Liu, Aohan Zeng, Yuxiao Dong, Jie Tang, and Le Song.  xTrimoPGLM: Unified 100B-Scale Pre-trained Transformer for Deciphering the Language of Protein, July 2023.  URL https://www.biorxiv.org/content/10.1101/2023.07.05.547496v1.  Pages: 2023.07.05.547496 Section: New Results.
  * Chen et al. [2021] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba.  Evaluating Large Language Models Trained on Code, July 2021.  URL http://arxiv.org/abs/2107.03374.  Issue: arXiv:2107.03374 arXiv:2107.03374 [cs].
  * Cottier [2023] Ben Cottier.  Trends in the Dollar Training Cost of Machine Learning Systems, January 2023.  URL https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems.
  * Creemers [2018] Rogier Creemers.  China’s Social Credit System: An Evolving Practice of Control, May 2018.  URL https://papers.ssrn.com/abstract=3175792.  Issue: 3175792.
  * Demartini et al. [2020] Gianluca Demartini, Stefano Mizzaro, and Damiano Spina.  Human-in-the-loop Artificial Intelligence for Fighting Online Misinformation: Challenges and Opportunities.  In _Bulletin of the IEEE Computer Society Technical Committee on Data Engineering_ , volume 43. IEEE Computer Society, September 2020.  URL https://www.damianospina.com/publication/demartini-2020-human/demartini-2020-human.pdf.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  ImageNet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, June 2009.  doi: 10.1109/CVPR.2009.5206848.  URL https://ieeexplore.ieee.org/document/5206848.  ISSN: 1063-6919.
  * Department for Science, Innovation and Technology [2023] Department for Science, Innovation and Technology.  Emerging processes for frontier AI safety.  Policy Paper, UK Government, October 2023.  URL https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safety.
  * DiEuliis et al. [2017] Diane DiEuliis, Sarah R. Carter, and Gigi Kwik Gronvall.  Options for Synthetic DNA Order Screening, Revisited.  _mSphere_ , 2(4):10.1128/msphere.00319–17, August 2017.  doi: 10.1128/msphere.00319-17.  URL https://journals.asm.org/doi/full/10.1128/msphere.00319-17.  Number: 4 Publisher: American Society for Microbiology.
  * Droppo and Elibol [2021] Jasha Droppo and Oguz Elibol.  Scaling Laws for Acoustic Models, June 2021.  URL http://arxiv.org/abs/2106.09488.  Issue: arXiv:2106.09488 arXiv:2106.09488 [cs, eess].
  * Egan and Heim [2023] Janet Egan and Lennart Heim.  Oversight for Frontier AI through a Know-Your-Customer Scheme for Compute Providers, October 2023.  URL http://arxiv.org/abs/2310.13625.  Issue: arXiv:2310.13625 arXiv:2310.13625 [cs].
  * Epoch [2022] Epoch.  Parameter, compute and data trends in machine learning, 2022.  URL https://epochai.org/data/pcd.  tex.copyright: CC-BY.
  * Epoch [2023] Epoch.  AI Trends, April 2023.  URL https://epochai.org/trends.
  * Erdil and Besiroglu [2023] Ege Erdil and Tamay Besiroglu.  Algorithmic progress in computer vision, August 2023.  URL http://arxiv.org/abs/2212.05153.  Issue: arXiv:2212.05153 arXiv:2212.05153 [cs].
  * Fitzpatrick [2021] Alex Fitzpatrick.  Why Amazon’s Move to Drop Parler Is a Big Deal for the Future of the Internet.  _Time_ , January 2021.  URL https://time.com/5929888/amazon-parler-aws/.
  * Ganguli et al. [2022] Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova DasSarma, Tom Henighan, Andy Jones, Nicholas Joseph, Jackson Kernion, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Scott Johnston, Shauna Kravec, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Dario Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Chris Olah, and Jack Clark.  Predictability and Surprise in Large Generative Models.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , pages 1747–1764, June 2022.  doi: 10.1145/3531146.3533229.  URL http://arxiv.org/abs/2202.07785.  arXiv:2202.07785 [cs].
  * Garfinkel and Dafoe [2019] Ben Garfinkel and Allan Dafoe.  How does the offense-defense balance scale?  _Journal of Strategic Studies_ , 42(6):736–763, September 2019.  ISSN 0140-2390.  doi: 10.1080/01402390.2019.1631810.  URL https://doi.org/10.1080/01402390.2019.1631810.  Number: 6 Publisher: Routledge _eprint: https://doi.org/10.1080/01402390.2019.1631810.
  * Google [2023] Google.  Bard can now connect to your Google apps and services, September 2023.  URL https://blog.google/products/bard/google-bard-new-features-update-sept-2023/.
  * Heim [2023a] Lennart Heim.  This can’t go on(?) - AI Training Compute Costs, June 2023a.  URL https://blog.heim.xyz/this-cant-go-on-compute-training-costs/.
  * Heim [2023b] Lennart Heim.  Video and Transcript of Presentation on Introduction to Compute Governance, May 2023b.  URL https://blog.heim.xyz/presentation-on-introduction-to-compute-governance/.
  * Heim and Ladish [2022] Lennart Heim and Jeffrey Ladish.  Information security considerations for AI and the long term future, May 2022.  URL https://blog.heim.xyz/information-security-considerations-for-ai/.
  * Hendrycks et al. [2021] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring Massive Multitask Language Understanding, January 2021.  URL http://arxiv.org/abs/2009.03300.  Issue: arXiv:2009.03300 arXiv:2009.03300 [cs].
  * Hernandez and Brown [2020] Danny Hernandez and Tom B. Brown.  Measuring the Algorithmic Efficiency of Neural Networks, May 2020.  URL http://arxiv.org/abs/2005.04305.  Issue: arXiv:2005.04305 arXiv:2005.04305 [cs, stat].
  * Hestness et al. [2017] Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianinejad, Md Mostofa Ali Patwary, Yang Yang, and Yanqi Zhou.  Deep Learning Scaling is Predictable, Empirically, December 2017.  URL http://arxiv.org/abs/1712.00409.  Issue: arXiv:1712.00409 arXiv:1712.00409 [cs, stat].
  * Hobbhahn et al. [2023] Marius Hobbhahn, Lennart Heim, and Gökçe Aydos.  Trends in Machine Learning Hardware, November 2023.  URL https://epochai.org/blog/trends-in-machine-learning-hardware.
  * Hoffmann et al. [2022] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training Compute-Optimal Large Language Models, March 2022.  URL http://arxiv.org/abs/2203.15556.  Issue: arXiv:2203.15556 arXiv:2203.15556 [cs].
  * Janardhan [2023] Santosh Janardhan.  Reimagining Our Infrastructure for the AI Age, May 2023.  URL https://about.fb.com/news/2023/05/metas-infrastructure-for-ai/.
  * Jouppi et al. [2023] Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson.  TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings, April 2023.  URL http://arxiv.org/abs/2304.01433.  Issue: arXiv:2304.01433 arXiv:2304.01433 [cs].
  * Kaplan et al. [2020] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei.  Scaling Laws for Neural Language Models, January 2020.  URL http://arxiv.org/abs/2001.08361.  Issue: arXiv:2001.08361 arXiv:2001.08361 [cs, stat].
  * Katal et al. [2023] Avita Katal, Susheela Dahiya, and Tanupriya Choudhury.  Energy efficiency in cloud computing data centers: a survey on software technologies.  _Cluster Computing_ , 26(3):1845–1875, June 2023.  ISSN 1573-7543.  doi: 10.1007/s10586-022-03713-0.  URL https://doi.org/10.1007/s10586-022-03713-0.  Number: 3.
  * Kinniment et al. [2023] Megan Kinniment, Lucas Jun Koba Sato, Haoxing Du, Brian Goodrich, Max Hasin, Lawrence Chan, Luke Harold Miles, Tao R Lin, Hjalmar Wijk, Joel Burget, Aaron Ho, Elizabeth Barnes, and Paul Christiano.  Evaluating Language-Model Agents on Realistic Autonomous Tasks.  Research paper, Alignment Research Center, 2023.
  * Knight [2023] Will Knight.  OpenAI’s CEO Says the Age of Giant AI Models Is Already Over.  _Wired_ , November 2023.  ISSN 1059-1028.  URL https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/.  Section: tags.
  * Koessler and Schuett [2023] Leonie Koessler and Jonas Schuett.  Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industries, July 2023.  URL http://arxiv.org/abs/2307.08823.  Issue: arXiv:2307.08823 arXiv:2307.08823 [cs].
  * Kuchinke and Vidal [2016] Björn A. Kuchinke and Miguel Vidal.  Exclusionary strategies and the rise of winner-takes-it-all markets on the Internet.  _Telecommunications Policy_ , 40(6):582–592, June 2016.  ISSN 0308-5961.  doi: 10.1016/j.telpol.2016.02.009.  URL https://www.sciencedirect.com/science/article/pii/S0308596116000549.  Number: 6.
  * Leike and Sutskever [2023] Jan Leike and Ilya Sutskever.  Introducing Superalignment, July 2023.  URL https://openai.com/blog/introducing-superalignment.
  * Li [2020] Chuan Li.  OpenAI’s GPT-3 Language Model: A Technical Overview, June 2020.  URL https://lambdalabs.com/blog/demystifying-gpt-3.
  * Liang et al. [2023] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.  Holistic Evaluation of Language Models, October 2023.  URL http://arxiv.org/abs/2211.09110.  Issue: arXiv:2211.09110 arXiv:2211.09110 [cs].
  * Liu et al. [2003] Cheng-Lin Liu, Kazuki Nakashima, Hiroshi Sako, and Hiromichi Fujisawa.  Handwritten digit recognition: benchmarking of state-of-the-art techniques.  _Pattern Recognition_ , 36(10):2271–2285, October 2003.  ISSN 0031-3203.  doi: 10.1016/S0031-3203(03)00085-2.  URL https://www.sciencedirect.com/science/article/pii/S0031320303000852.  Number: 10.
  * Lohn and Jackson [2022] Andrew Lohn and Krystal Jackson.  Will AI Make Cyber Swords or Shields?, August 2022.  URL https://cset.georgetown.edu/publication/will-ai-make-cyber-swords-or-shields/.
  * Lohn and Musser [2022] Andrew Lohn and Micah Musser.  AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?  Technical report, Center for Security and Emerging Technology, January 2022.  URL https://cset.georgetown.edu/publication/ai-and-compute/.
  * McGregor [2021] Sean McGregor.  Preventing Repeated Real World AI Failures by Cataloging Incidents: The AI Incident Database.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 35(17):15458–15463, May 2021.  ISSN 2374-3468.  doi: 10.1609/aaai.v35i17.17817.  URL https://ojs.aaai.org/index.php/AAAI/article/view/17817.  Number: 17.
  * Meta [2023] Meta.  Introducing Llama 2, 2023.  URL https://ai.meta.com/blog/llama-2/.
  * Mulani and Whittlestone [2023] Nikhil Mulani and Jess Whittlestone.  Proposing a Foundation Model Information-Sharing Regime for the UK | GovAI Blog.  Research Post, Centre for the Governance of AI, June 2023.  URL https://www.governance.ai/post/proposing-a-foundation-model-information-sharing-regime-for-the-uk#.
  * Nelson and Rose [2023] Cassidy Nelson and Sophie Rose.  Report launch: examining risks at the intersection of AI and bio.  Research Report, Centre for Long-Term Resilience, October 2023.  URL https://www.longtermresilience.org/post/report-launch-examining-risks-at-the-intersection-of-ai-and-bio.
  * Nevo et al. [2023] Sella Nevo, Dan Lahav, Ajay Karpur, Jeff Alstott, and Jason Matheny.  Securing Artificial Intelligence Model Weights: Interim Report.  Technical report, RAND Corporation, October 2023.  URL https://www.rand.org/pubs/working_papers/WRA2849-1.html.
  * Ngo et al. [2023] Richard Ngo, Lawrence Chan, and Sören Mindermann.  The alignment problem from a deep learning perspective, September 2023.  URL http://arxiv.org/abs/2209.00626.  Issue: arXiv:2209.00626 arXiv:2209.00626 [cs].
  * NIST [2023] NIST.  Face Recognition Technology Evaluation (FRTE) 1:1 Verification, November 2023.  URL https://pages.nist.gov/frvt/html/frvt11.html.
  * Novelli et al. [2023] Claudio Novelli, Federico Casolari, Antonino Rotolo, Mariarosaria Taddeo, and Luciano Floridi.  Taking AI risks seriously: a new assessment model for the AI Act.  _AI & SOCIETY_, July 2023.  ISSN 1435-5655.  doi: 10.1007/s00146-023-01723-z.  URL https://doi.org/10.1007/s00146-023-01723-z.
  * O’Brien et al. [2023] Joe O’Brien, Shaun Ee, and Zoe Williams.  Deployment corrections: An incident response framework for frontier AI models.  Research Report, Institute for AI Policy and Strategy, September 2023.  URL https://www.iaps.ai/research/deployment-corrections.
  * OpenAI [2023] OpenAI.  GPT-4, March 2023.  URL https://openai.com/research/gpt-4.
  * OpenAI [2023] OpenAI.  GPT-4 Technical Report, March 2023.  URL http://arxiv.org/abs/2303.08774.  Issue: arXiv:2303.08774 arXiv:2303.08774 [cs].
  * Our World in Data [2023] Our World in Data.  Computation used to train notable AI systems, by affiliation of researchers, November 2023.  URL https://ourworldindata.org/grapher/artificial-intelligence-training-computation-by-researcher-affiliation.
  * Papers with Code [2023] Papers with Code.  Papers with Code - MMLU Benchmark (Multi-task Language Understanding), November 2023.  URL https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu.
  * Patel [2023] Dwarkesh Patel.  Dario Amodei (Anthropic CEO) - Scaling, Alignment, & AI Progress, August 2023.  URL https://www.dwarkeshpatel.com/p/dario-amodei.  Dwarkesh Podcast.
  * Peng et al. [2023] Andrew Peng, Michael Wu, John Allard, Logan Kilpatrick, and Steven Heidel.  GPT-3.5 Turbo fine-tuning and API updates, August 2023.  URL https://openai.com/blog/gpt-3-5-turbo-fine-tuning-and-api-updates.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language Models are Unsupervised Multitask Learners, 2019.  URL https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf.
  * Richter [2023] Felix Richter.  Amazon Maintains Lead in the Cloud Market, August 2023.  URL https://www.statista.com/chart/18819/worldwide-market-share-of-leading-cloud-infrastructure-service-providers.
  * Roser [2023] Max Roser.  Artificial intelligence has advanced despite having few resources dedicated to its development – now investments have increased substantially.  _Our World in Data_ , October 2023.  URL https://ourworldindata.org/ai-investments.
  * Roser et al. [2023] Max Roser, Hannah Ritchie, and Edouard Mathieu.  What is Moore’s Law?, March 2023.  URL https://ourworldindata.org/moores-law.
  * Russell [2019] Stuart Russell.  _Human Compatible: Artificial Intelligence and the Problem of Control_.  Penguin, October 2019.  ISBN 978-0-525-55862-0.
  * Sandbrink [2023] Jonas B. Sandbrink.  Artificial intelligence and biological misuse: Differentiating risks of language models and biological design tools, October 2023.  URL http://arxiv.org/abs/2306.13952.  Issue: arXiv:2306.13952 arXiv:2306.13952 [cs].
  * Schaeffer et al. [2023] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo.  Are Emergent Abilities of Large Language Models a Mirage?, May 2023.  URL http://arxiv.org/abs/2304.15004.  Issue: arXiv:2304.15004 arXiv:2304.15004 [cs].
  * Sedova et al. [2021] Katerina Sedova, Christine McNeill, Aurora Johnson, Aditi Joshi, and Ido Wulkan.  AI and the Future of Disinformation Campaigns.  CSET Policy Brief, Centre for Security and Emerging Technology, December 2021.  URL https://cset.georgetown.edu/wp-content/uploads/CSET-AI-and-the-Future-of-Disinformation-Campaigns-Part-2.pdf.
  * Seger et al. [2023] Elizabeth Seger, Noemi Dreksler, Richard Moulange, Emily Dardaman, Jonas Schuett, K. Wei, Christoph Winter, Mackenzie Arnold, Seán Ó hÉigeartaigh, Anton Korinek, Markus Anderljung, Ben Bucknall, Alan Chan, Eoghan Stafford, Leonie Koessler, Aviv Ovadya, Ben Garfinkel, Emma Bluemke, Michael Aird, Patrick Levermore, Julian Hazell, and Abhishek Gupta.  Open-Sourcing Highly Capable Foundation Models.  Research paper, Centre for the Governance of AI, September 2023.  URL https://www.governance.ai/research-paper/open-sourcing-highly-capable-foundation-models.
  * Senior et al. [2020] Andrew W. Senior, Richard Evans, John Jumper, James Kirkpatrick, Laurent Sifre, Tim Green, Chongli Qin, Augustin Žídek, Alexander W. R. Nelson, Alex Bridgland, Hugo Penedones, Stig Petersen, Karen Simonyan, Steve Crossan, Pushmeet Kohli, David T. Jones, David Silver, Koray Kavukcuoglu, and Demis Hassabis.  Improved protein structure prediction using potentials from deep learning.  _Nature_ , 577(7792):706–710, January 2020.  ISSN 1476-4687.  doi: 10.1038/s41586-019-1923-7.  URL https://www.nature.com/articles/s41586-019-1923-7.  Number: 7792 Publisher: Nature Publishing Group.
  * Sevilla et al. [2022] Jaime Sevilla, Lennart Heim, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, and Pablo Villalobos.  Compute Trends Across Three Eras of Machine Learning, March 2022.  URL http://arxiv.org/abs/2202.05924.  Issue: arXiv:2202.05924 arXiv:2202.05924 [cs].
  * Shevlane [2022] Toby Shevlane.  Structured access: an emerging paradigm for safe AI deployment, April 2022.  URL http://arxiv.org/abs/2201.05159.  Issue: arXiv:2201.05159 arXiv:2201.05159 [cs].
  * Shevlane et al. [2023] Toby Shevlane, Sebastian Farquhar, Ben Garfinkel, Mary Phuong, Jess Whittlestone, Jade Leung, Daniel Kokotajlo, Nahema Marchal, Markus Anderljung, Noam Kolt, Lewis Ho, Divya Siddarth, Shahar Avin, Will Hawkins, Been Kim, Iason Gabriel, Vijay Bolina, Jack Clark, Yoshua Bengio, Paul Christiano, and Allan Dafoe.  Model evaluation for extreme risks, September 2023.  URL http://arxiv.org/abs/2305.15324.  Issue: arXiv:2305.15324 arXiv:2305.15324 [cs].
  * Solaiman [2023] Irene Solaiman.  The Gradient of Generative AI Release: Methods and Considerations, February 2023.  URL http://arxiv.org/abs/2302.04844.  Issue: arXiv:2302.04844 arXiv:2302.04844 [cs].
  * The White House [2023] The White House.  Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence, October 2023.  URL https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/.
  * Tucker et al. [2020] Aaron D. Tucker, Markus Anderljung, and Allan Dafoe.  Social and Governance Implications of Improved Data Efficiency.  In _Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society_ , pages 378–384, February 2020.  doi: 10.1145/3375627.3375863.  URL http://arxiv.org/abs/2001.05068.  arXiv:2001.05068 [cs].
  * Urbina et al. [2022] Fabio Urbina, Filippa Lentzos, Cédric Invernizzi, and Sean Ekins.  Dual use of artificial-intelligence-powered drug discovery.  _Nature Machine Intelligence_ , 4(3):189–191, March 2022.  ISSN 2522-5839.  doi: 10.1038/s42256-022-00465-9.  URL https://www.nature.com/articles/s42256-022-00465-9.  Number: 3 Publisher: Nature Publishing Group.
  * Venigalla and Li [2022] Abhinav Venigalla and Linden Li.  Mosaic LLMs (Part 2): GPT-3 quality for <$500k, September 2022.  URL https://www.mosaicml.com/blog/gpt-3-quality-for-500k.
  * Villalobos [2023] Pablo Villalobos.  Scaling Laws Literature Review, January 2023.  URL https://epochai.org/blog/scaling-laws-literature-review.
  * Villalobos et al. [2022] Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn, and Anson Ho.  Will we run out of data? An analysis of the limits of scaling datasets in Machine Learning, October 2022.  URL http://arxiv.org/abs/2211.04325.  Issue: arXiv:2211.04325 arXiv:2211.04325 [cs].
  * Wei [2023] Jason Wei.  Common arguments regarding emergent abilities, May 2023.  URL https://www.jasonwei.net/blog/common-arguments-regarding-emergent-abilities.
  * Wei et al. [2022] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus.  Emergent Abilities of Large Language Models, October 2022.  URL http://arxiv.org/abs/2206.07682.  Issue: arXiv:2206.07682 arXiv:2206.07682 [cs].
  * Weidinger et al. [2021] Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel.  Ethical and social risks of harm from Language Models, December 2021.  URL http://arxiv.org/abs/2112.04359.  Issue: arXiv:2112.04359 arXiv:2112.04359 [cs].
  * Wu et al. [2023] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann.  BloombergGPT: A Large Language Model for Finance, May 2023.  URL http://arxiv.org/abs/2303.17564.  Issue: arXiv:2303.17564 arXiv:2303.17564 [cs, q-fin] version: 2.
  * Yu et al. [2022] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu.  Scaling Autoregressive Models for Content-Rich Text-to-Image Generation, June 2022.  URL http://arxiv.org/abs/2206.10789.  Issue: arXiv:2206.10789 arXiv:2206.10789 [cs].
  * Zhang et al. [2022] Daniel Zhang, Nestor Maslej, Erik Brynjolfsson, John Etchemendy, Terah Lyons, James Manyika, Helen Ngo, Juan Carlos Niebles, Michael Sellitto, Ellie Sakhaee, Yoav Shoham, Jack Clark, and Raymond Perrault.  The AI Index 2022 Annual Report.  Technical report, AI Index Steering Committee, Stanford Institute for Human-Centered AI, Stanford University, March 2022.  URL https://aiindex.stanford.edu/wp-content/uploads/2022/03/2022-AI-Index-Report_Master.pdf.