# Training Compute Thresholds:
Features and Functions in AI Regulation

Lennart Heim Centre for the Governance of AI Oxford, United Kingdom Leonie
Koessler Centre for the Governance of AI Oxford, United Kingdom & European New
School of Digital Studies Frankfurt (Oder), Germany Correspondence to
<EMAIL_ADDRESS>
###### Abstract

Regulators in the US and EU are using thresholds based on training compute—the
number of computational operations used in training—to identify general-
purpose artificial intelligence (GPAI) models that may pose risks of large-
scale societal harm. We argue that training compute currently is the most
suitable metric to identify GPAI models that deserve regulatory oversight and
further scrutiny. Training compute correlates with model capabilities and
risks, is quantifiable, can be measured early in the AI lifecycle, and can be
verified by external actors, among other advantageous features. These features
make compute thresholds considerably more suitable than other proposed metrics
to serve as an initial filter to trigger additional regulatory requirements
and scrutiny. However, training compute is an imperfect proxy for risk. As
such, compute thresholds should not be used in isolation to determine
appropriate mitigation measures. Instead, they should be used to detect
potentially risky GPAI models that warrant regulatory oversight, such as
through notification requirements, and further scrutiny, such as via model
evaluations and risk assessments, the results of which may inform which
mitigation measures are appropriate. In fact, this appears largely consistent
with how compute thresholds are used today. As GPAI technology and market
structures evolve, regulators should update compute thresholds and complement
them with other metrics into regulatory review processes.

## Executive Summary

The development and deployment of advanced general-purpose artificial
intelligence (GPAI) models, also referred to as “frontier AI models” or “dual-
use foundation models”, pose increasing risks of large-scale societal harm
(Section 1). Currently, these models develop ever higher capabilities through
ever larger training runs, fuelled by ever more computational resources
(“training compute”). But higher capabilities also mean higher risks to
society, because many capabilities are dual-use (e.g., automated hacking
capabilities) and because more capable models can be expected to be used more
widely and relied upon more heavily, increasing the stakes if they fail or
behave in undesired ways (e.g., producing biased outputs). As a result,
regulators are increasingly using training compute thresholds to identify
models of potential concern.

“Training compute” refers to the total number of operations a computer needs
to perform to train an AI model (Section 2). In recent years, the scale of AI
training has grown significantly, with increases in the amount of training
data, the number of model parameters, and corresponding increases in the
amount of compute required for training (Figure 1). “Compute” in this context
refers to the total number of operations executed over the training process.
While post-training enhancements like fine-tuning can significantly increase
model capabilities, we recommend focusing on the compute used for the large
training run (“pre-training”), as this aligns with empirical scaling laws and
avoids impractical re-measurements for successive fine-tuning instances.

Figure 1: Training compute has been increasing at a fast rate, doubling
roughly every 6 months ($4\times$ per year). The US AI EO introduces reporting
requirements for models trained with more than $10^{26}$ operations. The EU AI
Act presumes a GPAI model poses systemic risk and imposes a variety of
requirements for models trained with more than $10^{25}$ operations.

Training compute has several features useful for GPAI regulation (Section 3).
In particular, it is:

  * •

Risk-tracking: Training compute is indicative of a model’s loss, capabilities,
and risks. Empirical research has identified correlations, known as _scaling
laws_ , between a model’s training compute and its training loss, test loss,
or validation loss. Improvements in loss tend to correlate with improvements
in capabilities. As models become more capable, they may pose greater risks if
they are misused or if they pursue misaligned objectives. The capabilities of
a model also serve as a proxy for how widely it will be used and how heavily
it will be relied upon and therefore the stakes if it fails or behaves in
other undesired ways.

  * •

Easily measurable: Training compute is a quantifiable metric that is
relatively simple and cheap to calculate.

  * •

Difficulty of circumvention: Training compute is relatively robust to
circumvention attempts, as reducing the amount of compute used to train a
model will generally decrease its capabilities and, consequently, lower its
risks. This is because, for a given model architecture and training algorithm,
the amount of compute used is directly related to the model’s capabilities and
potential risks. While algorithmic efficiency improvements gradually reduce
the amount of training compute required for a certain level of performance
over time, this represents an incremental progression of techniques rather
than an active circumvention.

  * •

Measurable before development and deployment: Training compute can be
calculated before the model is deployed, and estimated before the model is
trained.

  * •

Externally verifiable: The possibility for external parties, such as compute
providers, to verify compute usage, without disclosing proprietary details,
enhances compliance.

  * •

Cost-tracking: Training compute is proportionate to the cost of computational
resources for training, allowing the regulatory burden on smaller actors to be
minimized while focusing on the most well-resourced ones.

Notwithstanding the advantages discussed above, training compute also has
relevant limitations (Section 4). Fundamentally, it serves only as an
imperfect proxy for risk. While high training compute generally indicates
increased model capabilities and potential risks, some high-compute models may
pose minimal concerns, whereas certain low-compute models could present
significant risks (with the latter partially addressed through complementary
non-GPAI regulations). Moreover, improvements in algorithmic efficiency are
gradually reducing the amount of training compute required to achieve a given
level of capability, potentially weakening the relationship between a specific
compute threshold and the corresponding level of risk over time. However, this
shift in the compute-risk correlation might occur gradually rather than
abruptly as the field evolves.

Consequently, we suggest that compute thresholds should be used as an _initial
filter_ to identify GPAI models that warrant regulatory oversight and further
scrutiny (Section 5.1). We argue that compute thresholds can and should have
two primary functions: ensuring that regulators obtain sufficient visibility
into risks of large-scale societal harm from GPAI models and that companies
invest appropriate resources in understanding these risks. Precautionary
mitigation measures can be based on compute thresholds too, but it should be
allowable to later remove those mitigations if they turn out to be
unnecessary.

However, compute thresholds alone should generally _not_ determine which
mitigation measures are ultimately required, given that compute is only a
crude proxy for model capabilities and an even cruder proxy for risks of
large-scale societal harm. Instead, to determine which mitigation measures are
required, compute thresholds can be complemented with more precise but harder
to evaluate thresholds based on other metrics, such as capability thresholds
based on model capability evaluations. We also highlight that in a full
regulatory framework for AI, most requirements should not hinge on the amount
of training compute (Figure 2).

Both the US AI EO and the EU AI Act mostly use compute thresholds in line with
our suggestions (Section 5.2 and Section 5.3). The US Executive Order 14110
mandates companies to notify the government about any ongoing or planned
activities concerning the development of models that cross a compute threshold
of $10^{26}$ operations, report on the measures taken to ensure the physical
and cybersecurity of model weights, and share the results of red-teaming tests
and mitigation measures taken based on those results. The _EU AI Act_ requires
providers of GPAI models that cross a compute threshold of $10^{25}$
operations to notify the European Commission, perform model evaluations,
assess and mitigate systemic risks, report serious incidents, and ensure
cybersecurity of the model and its physical infrastructure.

Figure 2: Compute thresholds serve as an initial filter to identify GPAI
models that warrant regulatory oversight and further scrutiny, and, for
example, evaluation against capability thresholds to determine appropriate
mitigation measures, complemented by other AI requirements.

Challenges remain in effectively leveraging compute thresholds for GPAI
regulation (Section 6). A key question is the appropriate threshold level.
There is high uncertainty about the risk stemming from current and future GPAI
models trained on different amounts of compute. As a result, a low threshold
may be overinclusive, while a high threshold may be underinclusive. Moreover,
the direction in which a compute threshold should move over time is not
obvious, and it can make sense to complement compute thresholds with other
metrics. For example, increasing algorithmic efficiency allows more capable
models to be trained with less compute. A reason to consider adjusting the
compute threshold downwards over time.

However, there are also reasons to adjust the compute threshold upwards. If
better understanding of GPAI models reveals that models that are in scope of a
given compute threshold pose limited risks, the compute threshold should be
raised to focus on potentially risky models. With improving algorithmic
efficiency and computational price-performance, an increasing number of less
well-resourced actors may fall within the scope of a given compute threshold,
making the requirements increasingly burdensome and oversight increasingly
costly. Other relevant factors include the threat landscape (i.e. the number,
capacity, and willingness of malicious actors to use AI systems) and societal
vulnerability or adaptation (i.e. the ability and capacity of society to deal
with attacks, failures, and emergencies, e.g., through competent, well-
resourced, and stable institutions).

Some other metrics, such as risk estimates, model capabilities, and effective
compute, are better proxies for risk. However, these metrics are much harder
to measure than training compute is. Complementing mere compute thresholds
with other metrics becomes relevant to the extent that scaling laws cease to
hold and training compute becomes a worse proxy for risk. Particularly
relevant combinations may include training compute and model capability
evaluations—to ensure catching the most capable models—and training compute
and number of users—to ensure catching the most widely used models. However,
any threshold that is supposed to serve as an initial filter to identify
models of potential concern should be based on metrics that can be measured
easily and early in the model lifecycle. This, at least currently, excludes
risk estimates and model capability evaluations, and at least before
deployment, the number of users.

Overall, while not perfect, compute thresholds are currently a key tool in
GPAI regulation (Section 7). In particular, compute thresholds are currently
the best tool available for identifying potentially risky GPAI models and
triggering regulatory oversight and further scrutiny. They are based on a
risk-correlated, quantifiable metric that is difficult to circumvent and can
be measured before model development and deployment, enabling proactive
governance efforts. Compute thresholds can complement more targeted filters
like model capability evaluations that ultimately determine which mitigation
measures are required.

###### Contents

  1. Executive Summary
  2. 1 Introduction
  3. 2 Training Compute
  4. 3 Features of Training Compute Useful for GPAI Regulation
  5. 4 Limitations of Training Compute Relevant for GPAI Regulation
  6. 5 Functions of Training Compute Thresholds in GPAI Regulation
    1. 5.1 Initial Filter
    2. 5.2 US AI Executive Order
    3. 5.3 EU AI Act
  7. 6 Challenges for Training Compute Thresholds in GPAI Regulation
    1. 6.1 Where to Set Compute Thresholds
    2. 6.2 Domain-Specific Compute Thresholds
    3. 6.3 Why, When, and How to Update Compute Thresholds
    4. 6.4 Alternative Metrics
      1. 6.4.1 Risk Estimates
      2. 6.4.2 Model Capability Evaluations
      3. 6.4.3 Effective Compute
      4. 6.4.4 Parameters, Data, and Other Metrics
  8. 7 Conclusion
  9. Acknowledgments

##  1 Introduction

The development and deployment of advanced general-purpose artificial
intelligence (GPAI) models111There is no common distinction between what still
counts as a “model” and what already constitutes a “system.” Regulators, such
as the EU AI Office, need to provide clarification on this question. In this
paper, we understand the term “model” to include model weights and source
code, and the term “system” to encompass everything beyond (e.g., API, calls
to other models, calls to other APIs) (similar to Basdevant et al., 2024).,
also referred to as “frontier AI models” or “dual-use foundation models”, pose
increasing risks of large-scale societal harm (Anderljung et al., 2023; Bengio
et al., 2024; Hendrycks et al., 2023). Currently, GPAI models develop ever
higher capabilities through ever larger training runs, fuelled by ever more
computational resources. The overall amount of computational resources used to
train a model is referred to as _training compute_. But higher capabilities
also mean higher risks to society, because many capabilities are dual-use
(e.g., cyber capabilities) and because more capable models can be expected to
be used more widely and relied upon more heavily, increasing the stakes if
they fail or behave in undesired ways (e.g., producing biased outputs). As a
result, regulators are increasingly imposing requirements on providers of GPAI
models whose amount of training compute passes a certain limit. These limits
are called _compute thresholds_. This paper discusses the key features of
training compute and, accordingly, the functions compute thresholds should
have in GPAI regulation.

GPAI regulation worldwide increasingly relies on training compute thresholds.
Since October 2023, _US Executive Order 14110 on the Safe, Secure, and
Trustworthy Development and Use of Artificial Intelligence (US AI EO)_
requires companies developing and deploying GPAI models above a compute
threshold of $10^{26}$ operations to notify the government, conduct red-
teaming, and secure the model weights (Section 4.2). From August 2024, the _EU
Regulation 2024/1689 laying down harmonised rules on artificial intelligence
(EU AI Act)_ presumes that GPAI models that pass a compute threshold of
$10^{25}$ floating-point operations pose systemic risk (Article 51(2)).
Providers of such models are required to notify the European Commission
(Article 52(1)), conduct model evaluations, assess and mitigate systemic
risks, ensure adequate cybersecurity, and report serious incidents (Article
55). Finally, an early draft of an _Artificial Intelligence Law of the
People’s Republic of China_ references training compute as one of the criteria
used to identify “critical AI” which would require heightened safety and
security measures (Linghan et al., 2024).222Throughout this paper, we use the
terms “(training) operations”, “(training) compute”, and “(training) compute
threshold” to emphasize that our focus is on the computational resources used
specifically during the training phase of a AI model’s lifecycle. There are
other types of “compute thresholds” that can be used for AI regulation. For
example, the US AI EO also includes a reporting requirement for computing
clusters above a certain computing capacity per second. This is another type
of “compute threshold” that does not directly refer to the training process
but rather to the mere possession of computing infrastructure that can be used
not only for training but also for deploying AI models.

At the same time, the literature on compute thresholds is relatively scarce.
While many papers analyze trends in training compute (e.g., Lohn & Musser,
2022; Pilz & Heim, 2023; Sevilla et al., 2022a; Villalobos et al., 2024), only
a few discuss using compute as a node for AI governance and suggest using
training compute thresholds (Anderljung et al., 2023; Heim et al., 2024; Egan
& Heim, 2023; Sastry et al., 2024). Hooker (2024) questions training compute
as a proxy for risk and raises the issue that (training) compute thresholds
may quickly become obsolete. While the paper points at some relevant
challenges, as we argue in the following, it lacks nuance regarding the
features of training compute, the function that compute thresholds _can_ and
_should_ play in GPAI regulation, and an accurate representation of
regulations that already incorporate such thresholds. The combination of
increasing regulatory reliance on compute thresholds and scarce academic
literature suggests an urgent need for more scholarly treatment of the topic.

This paper aims to answer the following two research questions: (1) What
features of training compute are relevant for GPAI regulation? (2) What should
be the function of (training) compute thresholds in GPAI regulation? The
remainder of the paper is structured as follows: Section 2 clarifies the
concept of training compute, Section 3 discusses its useful features, and
Section 4 its limitations. Section 5 suggests which functions (training)
compute thresholds should have in GPAI regulation, while Section 6 elaborates
on challenges in that regard. Section 7 concludes with key claims and further
research questions.

##  2 Training Compute

Training an AI model is an iterative process where a model—a large amount of
numeric values (the so-called “parameters”) arranged in a certain way (the so-
called “architecture”)—is exposed to a large amount of data, allowing the
model to learn from the data by adapting the parameters. This learning can be
supervised, where the model is provided with labeled examples, or
unsupervised, where the model learns from unlabeled data (Goodfellow et al.,
2016).

“Training compute” or simply “compute” refers to the amount of computational
resources or, more precisely, the total number of operations required to train
an AI model. The training process runs on a computer that performs a vast
number of mathematical operations. We are agnostic here as to whether these
are integer operations, floating-point operations (FLOP), or other operations.
Presently, most AI training predominantly uses floating-point operations, but
this could change in the future. We therefore broadly refer to all types of
operations as “(training) operations (OP).” The total number of operations
used to train the model is a quantity we refer to as “(training)
compute.”333When referring to the quantity of computational resources used
during training, it is preferable to use the term “FLOP” rather than “FLOPs.”
For a longer elaboration, see Heim (2023).

The term “compute” in this context refers to the number of operations executed
in total over the whole training process, not to be confused with the
computer’s processing performance—the number of operations the computer is
able to execute per second (FLOP/s or OP/s). For example, running an NVIDIA
A100 processor (which has a processing performance of 312 TeraFLOP/s for FP16
tensor) for a week results in a total of $1.9\times 10^{20}$ FLOP being
executed. Multiplied with the number of NVIDIA A100 processors used, this
results in the total number of operations used to train the AI model.444Note
that this is a simplified explanation. For more details, see Pistillo et al.
(forthcoming).

In recent years, the scale of AI training has grown significantly, with
increases in the amount of training data, the number of model parameters, and,
therefore, the amount of compute required for training (growing by about
$4\times$ every year since 2010, Figure 3).

Figure 3: Amount of compute used to train AI models over time. In the pre-deep
learning era, training compute followed Moore’s Law, doubling approximately
every two years. Since the emergence of the Deep Learning Era around 2010,
training compute has been increasing at a much faster rate, doubling roughly
every 6 months (increasing by about $4\times$ per year). This rapid growth is
largely driven by increased investments in computational resources for
training larger models, which have demonstrated improved capabilities (figure
from Sastry et al. 2024; up-to-date as of end of 2023; underlying data and
updates can be found at Epoch AI 2024a).

There is currently no standardized method for measuring a model’s training
compute. This introduces ambiguity that may be considered problematic in a
regulatory context. We suggest measuring training compute by following the
guidance of the Frontier Model Forum (Frontier Model Forum, 2024). More
precisely, we suggest using Method 1 described in (Sevilla et al., 2022b).
However, given that the amount of training compute used for the final large
training run (“pre-training”) currently increases by about four times per year
(Epoch AI, 2024b; Sevilla et al., 2022a), the exact details of these methods
are not critical.

Figure 4: We recommend only measuring pre-training compute and not including
compute used in further enhancement processes (figure adapted from Pistillo et
al. (forthcoming)).

By contrast, an important question is whether to focus on the compute used for
pre-training alone or whether to also include the compute used for “fine-
tuning”, “reinforcement learning from human feedback” (RLHF), and other post-
training enhancements (Figure 4). Post-training enhancements can significantly
improve model capabilities, up to an equivalent of a $5\times$ to $20\times$
increase in training compute (Davidson et al., 2023; Villalobos & Atkinson,
2023). Presumably to account for this, Recital 111 of the EU AI Act suggests
including compute used for post-training enhancements: “ _The cumulative
amount of computation used for training includes the computation used across
the activities and methods that are intended to enhance the capabilities of
the model prior to deployment, such as pre-training, synthetic data generation
and fine-tuning._ ”

We believe that including the compute used for post-training enhancements
lacks an empirical basis, is unnecessary, and is impractical. First, the
empirical basis for existing scaling laws describes model performance as a
function of pre-training compute, not of compute used for fine-tuning or other
post-training enhancements (for more on scaling laws, see Section 3). This
means there is no empirical basis for imposing requirements based on the
cumulative amount of compute. Second, fine-tuning compute is small compared to
pre-training compute.555Usually around 1%, or, in the largest case we are
aware of, up to about 14% of the pre-training compute. A small robotics model
(Swift) is an exception, using fine-tuning compute equivalent to about 200% of
pre-training compute (Epoch AI, 2024a). Consequently, including fine-tuning
compute would make only a small difference to the overall training compute
measurement. Lastly, measuring fine-tuning compute is impractical because
fine-tuning is repeated many times for a given pre-trained model and often
performed by downstream developers. Re-measuring training compute (and
potentially re-reporting and re-conducting model evaluations) for every fine-
tuned version would be extremely burdensome for individual regulatees and lead
to a much larger number of regulatees.

Instead, we recommend only measuring pre-training compute. While post-training
enhancements can significantly increase a model’s capabilities, this should
not be considered directly when designing a compute threshold, but only
indirectly as discussed in Section 6.1. Moreover, post-training enhancements
can and should be taken into account when measuring a model’s capabilities, as
it is the enhanced model that will be deployed in the end. Indeed,
comprehensive model evaluations should involve applying fine-tuning and other
post-training enhancements to elicit the pre-trained model’s full capabilities
as much as possible. However, it is not necessary or practical to re-conduct
model capability evaluations on every enhanced model version. Minor variations
may not necessitate re-conducting model capability evaluations, provided the
pre-trained model has already been evaluated. We suggest re-conducting model
capability evaluations on enhanced model versions periodically and if they
constitute a new model family.

##  3 Features of Training Compute Useful for GPAI Regulation

Training compute has several features that make it a valuable metric for GPAI
regulation. Namely, training compute is indicative of a model’s capabilities
and therefore its risks, easily measurable, difficult to circumvent,
measurable before development and deployment, externally verifiable, and
indicative of the developer’s resources and therefore its capacity to handle
regulatory burdens. In the following, we discuss each of these features in
more detail.

  1. 1.

Risk-tracking: Training compute is indicative of a model’s loss, capabilities,
and risks.

     * •

For GPAI models, the amount of compute used to train a model correlates with
the model’s loss, capabilities, and risks it may present. Empirical research
has identified correlations, known as _scaling laws_ , between a model’s
training compute and its training loss, test loss, or validation loss
(Hernandez et al., 2021; Hoffmann et al., 2022; Kaplan et al., 2020; Sutton,
2019; Villalobos, 2023). Further, improvements in loss tend to correlate with
improvements in capabilities on downstream tasks (Brown et al., 2020; Ganguli
et al., 2022; Rae et al., 2022). Finally, as models become more capable, they
may pose greater risks if they are misused or if they pursue misaligned
objectives. The capabilities of a model also serve as a proxy for how widely
it will be used and how heavily it will be relied upon and therefore the
stakes if it fails or behaves in other undesired ways.666At the same time, the
risks from failure and other undesired behaviors decrease with higher
capabilities in terms of model accuracy, robustness, and not being susceptible
to prompt injection attacks and jailbreaks. Which one of these two trends
outweighs the other is currently impossible to say—empirical data is lacking
to make statements about the past, let alone predict the future. For example,
this may include increases in the number and severity of people affected by
biased outputs (Bommasani et al., 2022).

     * •

This feature of training compute—its correlation with a model’s loss,
capabilities, and risks—is the most important but also perhaps the most
controversial (e.g., Hooker, 2024). While this feature is already disputed
today, we highlight that to the extent that scaling laws cease to hold in the
future—for example, because a training paradigm other than deep learning
emerges— training compute will become a less useful metric for GPAI
regulation. We discuss this feature in some more depth in Section 4.

  2. 2.

Easily measurable: Training compute is a quantifiable metric that is
relatively simple and cheap to calculate.

     * •

Training compute is a metric that is easy to measure, as it can be directly
calculated from model specifications or inferred from data about the use of
hardware with minimal effort. Training compute also is unidimensional and
durable, unlike other metrics, many of which are multidimensional (e.g., data
quality and type) or may quickly become outdated (e.g., model capability
benchmarks).

  3. 3.

Difficult to circumvent: Training compute is difficult to reduce without also
decreasing a model’s capabilities and risks.

     * •

Training compute is relatively robust to circumvention attempts, as reducing
the amount of compute used to train a model will generally decrease its
capabilities and, consequently, lower its risks. This is because, for a given
model architecture and training algorithm, the amount of compute used is
directly related to the model’s capabilities and potential risks. Therefore, a
GPAI developer cannot simply decide to use less compute while maintaining the
same level of capabilities. In contrast, a GPAI developer might be able to
adjust other metrics, such as the score on a specific benchmark, to avoid
regulation without substantially impacting the model’s capabilities.

     * •

However, improvements in algorithmic efficiency can reduce the amount of
compute required for a given level of capabilities. Over time, this may allow
less well-resourced actors to develop models that achieve a given level of
capabilities (Pilz et al., 2023). Indeed, improvements in algorithmic
efficiency pose a challenge to using compute thresholds. However, while future
algorithmic efficiency improvements may require less compute for the same
level of capabilities, this represents natural progress rather than active
circumvention attempts. Purposefully decreasing training compute for fixed
algorithms would still come at the cost of lower capabilities and risks. We
address the question of when to update compute thresholds to account for
algorithmic efficiency improvements in Section 6.3.

  4. 4.

Measurable before development and deployment: Training compute can be
calculated before the model is deployed and even before it is trained.

     * •

Training compute can be known ahead of deployment and estimated ahead of
development. This is important because regulators may want to impose
requirements for how a model is developed and deployed. Training compute can
be calculated before model deployment because training will be completed at
that time. Before model development, training compute can be estimated using
the architectural details and the amount of training data, as outlined in
Method 1 of Sevilla et al. (2022b). AI companies carefully plan their training
runs, as training state-of-the-art models often requires tens of thousands of
GPUs and costs on the order of millions of dollars. Given the significant
computational resources and financial investment involved, companies have a
strong incentive to accurately estimate training compute beforehand to ensure
efficient resource allocation and budget planning. By estimating training
compute before training begins, developers can implement compute-indexed
precautions during the training process. For example, they can ensure that
strong cybersecurity measures are in place for compute-intensive training
runs, reducing the risk of model theft or unauthorized access.

  5. 5.

Externally verifiable: The possibility for external parties to verify compute
usage, without disclosing proprietary details, enhances compliance.

     * •

Ideally, measurements of training compute are verifiable by diverse external
parties through protocols that maintain the confidentiality of proprietary
information. This could also enable verifiable commitments across companies
and even states (Brundage et al., 2020). Compute providers can aid in the
verification of requirements based on training compute. This is particularly
desirable, as compute providers can monitor and verify compute usage without
infringing on the confidentiality of developers, in contrast to model
capabilities and other metrics that may require access to sensitive model
details. This is because compute usage can be monitored and verified without
the need to access specific details of the model architecture, training data,
or other proprietary information (Heim et al., 2024).

  6. 6.

Cost-tracking: Training compute is proportionately higher for models that cost
more to develop, minimizing the regulatory burden on smaller actors while
focusing on the most well-resourced ones.

     * •

The amount of compute used to train a model directly corresponds to the amount
of financial resources required to do so (Footnote 8). In other words, the
cost of training a model scales with the amount of training compute used. For
example, training a model with $10^{26}$ operations ($70M) will cost
approximately 10 times more than training one with $10^{25}$ operations ($7M).

     * •

The large amounts of compute required to train state-of-the-art models are
typically only available to well-resourced organizations. By setting compute
thresholds at appropriate levels, regulators can focus on the most advanced
and potentially risky models without imposing undue burdens on smaller actors
in the AI ecosystem, such as startups, small businesses, or academic
researchers.

     * •

However, the cost of a given amount of compute decreases over time.
Computational price-performance (FLOP per $) has been observed to double every
2.1 years for machine learning GPUs and 2.5 years for general GPUs (Hobbhahn
et al., 2023). As the cost of compute falls, more actors may be able to
develop models that cross the compute threshold (Pilz et al., 2023). This
expansion in the regulatory scope would impose regulatory burdens on smaller
actors and pose a challenge in terms of scaling oversight as the number of
covered entities grows. In Section 6.3, we discuss approaches to account for a
regulatory scope that is potentially expanding due to improving computational
price-performance.

Figure 5: Cost and compute used for training AI models. The amount of compute
used to train a model directly corresponds to the amount of financial
resources required to do so.888While the exact cost calculations vary across
different sources due to uncertainty about potential discounts and the
achieved utilization or efficiency of leveraging the provided computing power
(see Hobbhahn et al., 2023), the rough estimates tend to fall within a similar
range. For the purposes of this analysis, we assume a cost of $1.8 per hour
for renting an NVIDIA H100, which is a rather optimistic assumption. Also,
note that this is only the cost of acquiring the computational resources for
the final large training run. It does not include staff cost and compute usage
beyond pre-training.This rapid growth is largely driven by increased
investments in computational resources for training larger models, which have
demonstrated improved capabilities (figure adapted from Sastry et al. 2024;
underlying data and updates can be found at Epoch AI 2024a).

##  4 Limitations of Training Compute Relevant for GPAI Regulation

In this section, we discuss the main limitations of training compute relevant
for frontier AI regulation. We argue that training compute is only a very
crude proxy for risk and that, in the future, training compute may become a
worse proxy for risk or even cease to be such a proxy altogether.

Scaling laws describe relationships between training compute and a model’s
loss (Section 3). The subsequent relationship between loss and capabilities is
not always “smooth.” For example, inverse scaling has been observed for some
tasks (Koralus & Wang-Maścianica, 2023; McKenzie et al., 2024; Perez et al.,
2022), though for some of these tasks, the relationship has been re-
established with further scaling (OpenAI et al., 2023; Wei et al., 2022a).
Moreover, with gradual improvements in loss, there can be jumps in
capabilities, or at least apparent jumps depending on how granular the
capabilities are measured (Wei, 2023; Wei et al., 2022b).999These apparent
discontinuities arise because we often care about discontinuous measures, such
as getting the exact right answer in mathematics or avoiding crashes in
autonomous vehicles, rather than just approximating the correct solution.
While continuous surrogate measures could potentially be used to forecast when
such discontinuous jumps in capability might occur, identifying the
appropriate continuous measure is itself a complex problem. For a more
comprehensive discussion of “emergent capabilities”, see Anderljung et al.
(2023), Pistillo et al. (forthcoming), and Woodside (2024).

The relationship between increases in capabilities and risks is also not
“smooth.” The higher a model’s capabilities, the more risky the model will
arguably be, because it can be misused for more dangerous purposes and it is
likely to be employed in more and higher-stakes settings. However, risk is
highly contextual. Factors other than model capabilities have a major impact
on risk. Important factors include the offense-defense balance of AI systems,
i.e. whether AI systems help more with beneficial or harmful uses101010The
offense-defense balance of AI systems could significantly impact how increased
model capabilities translate to risks. If defensive AI applications can reduce
AI risks, it may counterbalance risks from offensively used models. (AI Policy
Perspectives & Krier, 2024; Buterin, 2023; Lohn & Jackson, 2022), the threat
landscape, i.e. the number, capacity, and willingness of malicious actors to
use AI systems (Koessler et al., 2024), and societal vulnerability or
adaptation, i.e. the ability and capacity of society to deal with attacks,
failures, and emergencies, e.g., through competent, well-resourced, and stable
institutions (Anderljung et al., 2023; Bernardi et al., 2024; Kapoor et al.,
2024).

Essentially, training compute correlates with loss, which correlates with
capabilities, which in turn correlate with risk. Thus, training compute is a
proxy for risk; however, as none of these three correlations are perfect,
training compute can only be considered a very crude proxy for risk (Hooker,
2024).111111For a discussion of the relationship between training compute,
loss, capabilities, and risks, see Anderljung et al. (2023), Sastry et al.
(2024), and Pistillo et al. (forthcoming).

Fundamentally, the notion of training compute as a proxy for risk hinges on
the validity of scaling laws. While scaling laws are empirical observations
derived from past data, they do not guarantee that these relationships will
persist indefinitely. There is a prediction, referred to as the “scaling
hypothesis”, that these relationships will continue to hold true in the future
(Branwen, 2020; Sutton, 2019). However, innovation in AI might gradually lead
to a paradigm shift away from deep learning, or the pre-training phase might
become less crucial for determining the final capabilities of AI models. As a
result, training compute may become a less precise proxy for risk over time or
could potentially cease to serve as an effective proxy for risk altogether.
Nonetheless, we do not anticipate an abrupt disruption in the relationship
between training compute and risk. Any potential shift is likely to occur
gradually as the field evolves over an extended period. In Section 6.3, we
discuss approaches to account for changes in the relationship between training
compute and risk due to improving algorithmic efficiency.

##  5 Functions of Training Compute Thresholds in GPAI Regulation

Building on the characteristics of training compute outlined in the previous
sections, in this section we argue that compute thresholds can and should be
used as an initial filter to identify models that warrant regulatory
oversight, further scrutiny, and precautionary safety measures. However, given
that compute is only a very crude proxy for capabilities and risk, compute
thresholds should not be the final determinant for what safety measures to
require. After compute thresholds, decision criteria based on more precise
proxies for the risk a model poses, such as capability thresholds, should be
applied (Section 5.1). Both the US AI EO (Section 5.2) and the EU AI Act
(Section 5.3) take this approach.

###  5.1 Initial Filter

When imposing requirements on GPAI models, policymakers need to consider five
interdependent questions:

  1. 1.

Which _risks_ should be countered? (E.g., large-scale societal harm.)

  2. 2.

Which _models_ could pose these risks? (E.g., advanced GPAI models.)

  3. 3.

Which _metrics_ correspond to features of these models? (E.g., training
compute.)

  4. 4.

Which _thresholds_ are the values of those metrics at which models start to
pose these risks? (E.g., $10^{26}$ operations of training compute.)

  5. 5.

Which _requirements_ should be imposed on models above those thresholds to
counter these risks? (E.g., reporting, model evaluation, and risk assessment.)

Easily measurable metrics, such as training compute, are generally preferable
to ensure legal certainty about which GPAI models are in scope of
requirements. At the same time, metrics that are better proxies for the risk a
GPAI model poses, such as model capabilities, are generally preferable to
ensure requirements are targeted at risky GPAI models. There will often be a
trade-off between ease of measurability and correlation with risk (more in
Section 6.4). Taken together, metrics that are cruder proxies for the risk a
GPAI model poses should be combined with less costly and less definitive
requirements, while metrics that are better proxies for the risk a GPAI model
poses may be linked to more costly and more definitive requirements. What is
more, all else equal, the stricter the requirements, the higher and therefore
more exclusive the threshold should be.

Building on these considerations, the function of compute thresholds should be
to serve as an _initial filter_ to identify models of potential concern.
Training compute is an easily measurable and externally verifiable proxy for
risk that allows focusing on the most risky models and the most well-resourced
actors (Section 3). Therefore, compute thresholds provide an easy way to
identify those GPAI models that warrant heightened attention (both from
companies and regulators), while filtering out the much larger fraction of
GPAI models that are highly unlikely to pose risks of large-scale societal
harm. As a result, using compute thresholds as an initial filter can help to
reduce both compliance burdens for regulatees and enforcement costs for
regulators, while focusing attention on those GPAI models that are most likely
to pose risks of large-scale societal harm.

However, the function of compute thresholds should _only_ be to serve as an
initial filter to identify models that warrant regulatory oversight and
further scrutiny. Already, training compute is a very crude proxy for risk,
and this relationship may become worse in the future (Section 4). Therefore,
compute thresholds should be complemented by thresholds based on metrics that
are more precise but that may be harder to evaluate, such as model capability
evaluations. In particular, compute thresholds should not be used to
ultimately determine which mitigation measures need to be taken (beyond a few
key precautionary mitigation measures). For models that cross a compute
threshold, more rigorous analyses of the risk they pose should be conducted
through model evaluations and risk assessments. Based on the results of these
analyses, specific mitigation measures can be required (and precautionary
mitigation measures that have been taken but turn out unnecessary may be
discontinued). Currently, a sensible approach would be to first apply compute
thresholds (to identify GPAI models that warrant oversight and scrutiny) and
to subsequently apply capability thresholds (to ultimately determine which
mitigation measures need to be implemented) (Section 4).

We also highlight that compute thresholds should not be used for all
requirements aimed to counter risks from AI. First, many risks from AI stem
from the context and way in which AI models are applied, instead of or in
addition to the intrinsic properties of those AI models, necessitating
requirements at the application layer. In particular, bias, discrimination,
and fairness risks, for example, caused by AI models being applied in
education, hiring, and public service provision, should be countered through
requirements at the application layer in addition to the model layer, as done
in the EU AI Act Chapter II (Prohibited AI Practices), Chapter III (High-Risk
AI Systems), and Chapter IV (Transparency Obligations). For requirements at
the application layer, compute thresholds will usually not make sense but
other criteria should be decisive, like the type of application. Second, some
risks from GPAI models arise regardless of their amount of training compute,
such as risks of copyright or privacy infringements. These risks should be
countered through requirements imposed on all GPAI models, as done in the EU
AI Act Article 53. Third, ex post regulation like tort law and criminal law
should of course not hinge on compute thresholds. Overall, in a full
regulatory framework for AI, most requirements should not hinge on the amount
of training compute (Figure 6).

Figure 6: A framework for the role of compute thresholds in GPAI regulation.
Compute thresholds serve as an initial filter to identify GPAI models that
warrant regulatory oversight and further scrutiny, and, for example,
evaluation against capability thresholds to determine appropriate mitigation
measures, complemented by other AI requirements.

Thresholds based on easily measurable metrics that serve as initial filters
for regulatory oversight and further scrutiny are in place across various
sectors. For example, in the US, the Environmental Protection Agency’s (EPA)
ecological risk assessments start by determining whether some concentration of
contaminants has been crossed, only after which a more thorough risk
assessment takes place (U.S. Environmental Protection Agency & Office of
Emergency and Remedial Response, 1996). In the EU, the Digital Services Act
(DSA) identifies “very large online platforms” and “very large online search
engines” that need to conduct systemic risk assessments based on whether they
have a “number of average monthly active recipients of the service in the
Union equal to or higher than 45 million” (Article 33(1) DSA). Similarly, the
General Data Protection Regulation (GDPR) mandates data protection impact
assessments for entities processing data “on a large scale” (Article 35(3)
GDPR)— the concrete metric and threshold in this case is to be specified by
member states, who have chosen different thresholds, most of which are based
on the number of people whose data is concerned (Breitbarth, 2018).

While we generally have reservations about using compute thresholds to
ultimately determine which mitigation measures to require, this can be
different if the threshold is set relatively high such that only a few well-
resourced actors are affected. For example, we have suggested elsewhere to use
compute thresholds to identify large compute providers on which to impose Know
Your Customer (KYC) and other requirements (Egan & Heim, 2023). We believe
this approach can be justified, as it only targets a handful of companies (see
Heim et al., 2024; Egan & Heim, 2023).

###  5.2 US AI Executive Order

The US AI EO covers a range of issues from AI. Section 4 leverages training
compute thresholds as a criterion for classifying AI models that warrant
regulatory oversight and further scrutiny due to potential safety and security
concerns. Specifically, it targets:

> _“any model that was trained using a quantity of computing power greater
> than $10^{26}$ integer or floating-point operations, or using primarily
> biological sequence data and using a quantity of computing power greater
> than $10^{23}$ integer or floating-point operations”_ (Section 4.2(b)(i))

Note that these thresholds are designed to capture future models. As of July
2024, no model has been officially reported to meet the general compute
threshold of $10^{26}$ operations (Figure 7). However, one model has been
estimated to meet the lower bio-compute threshold (Maug et al., 2024).

Figure 7: The US AI EO introduces reporting requirements for models trained
with more than $10^{26}$ operations and $10^{23}$ operations if trained using
primarily biological sequence data (figure adapted Sastry et al. 2024;
underlying data and updates can be found at Epoch AI 2024a).

The US AI EO mandates companies to notify the government about ongoing or
planned activities concerning the development of models that cross the compute
thresholds (Section 4.2(a)(i)(A)). It also requires these companies to report
on the measures taken to ensure the physical and cybersecurity of model
weights (Section 4.2(a)(i)(B)) and share the results of red-teaming tests and
mitigation measures taken based on those results (Section 4.2(a)(i)(C)).

The US AI EO uses compute thresholds in line with what we have argued can and
should be their role (Section 5.1). Its requirements focus on reporting to
increase regulatory oversight, which is one of the primary functions compute
thresholds can fulfill. While the US AI EO does not mandate companies to
implement security measures or conduct red-teaming tests, the corresponding
reporting requirements strongly push companies to take such measures in order
to not appear irresponsible to the regulator. In our view, the pressure to
conduct red-teaming tests also makes sense, as increased scrutiny is another
primary function of compute thresholds. The pressure to implement security
measures can also be based on compute thresholds. However, we believe that
companies should be given the option to relax their security measures again if
they establish that their model does not exceed a specific level of
capabilities or can otherwise be proven to be sufficiently safe. We discuss
domain-specific compute thresholds in Section 6.2.

###  5.3 EU AI Act

The EU AI Act is the most comprehensive regulatory framework on AI worldwide,
prohibiting certain AI practices (Chapter II), imposing requirements on
certain high-risk AI systems (Chapter III), requiring transparency about the
use of AI in services and to produce content (Chapter IV), and imposing
requirements on GPAI models (Chapter V). Regarding the latter, the EU AI Act
leverages a compute threshold to distinguish between GPAI models with and
without systemic risk. Concretely, it draws the line in the following
way:121212When establishing compute thresholds, it is advisable for the EU AI
Act to use the term “operations” rather than specifying a particular type of
operation (such as floating-point operations or FLOP). Using the broader term
“operations” ensures that the threshold remains agnostic to the specific type
of computational operations performed during training, making it more future-
proof in light of possible technological changes. This approach also maintains
consistency with the terminology used in the US AI EO, ensuring
interoperability and enabling cooperation between EU and US regulators.

> _“A general-purpose AI model shall be presumed to have high impact
> capabilities pursuant to paragraph 1, point (a) [and thus be classified as
> posing systemic risk], when the cumulative amount of computation used for
> its training measured in floating point operations is greater than
> $10^{25}$.”_ (Article 51(2))

In contrast to the general compute threshold in the US AI EO, this compute
threshold has immediate relevance. As of July 2024, one model available in the
EU has officially been reported to have crossed the $10^{25}$ training compute
threshold (Inflection-2), some additional models available have been estimated
to have done so (e.g., Gemini Ultra, GPT-4, Inflection-2.5) (Epoch AI, 2024a),
and there may be a handful more of such models (Figure 8).

Figure 8: While the US AI EO introduces reporting requirements for models
trained with more than $10^{26}$ operations, the EU AI Act presumes a GPAI
model poses systemic risk and imposes a variety of requirements for models
trained with more than $10^{25}$ operations (figure adapted from Sastry et al.
2024; underlying data and updates can be found at Epoch AI 2024a).

In the EU AI Act, the compute threshold is only one of several ways to
identify GPAI models with systemic risk, but it currently is the most concrete
one. In more detail, Article 51(1) says that a GPAI model shall be classified
as posing systemic risk if it has either “high impact capabilities” (Article
51(1)(a)) or “capabilities or an impact equivalent to those set out in point
(a)” (Article 51(1)(b)). Regarding the first alternative, high-impact
capabilities are defined as “capabilities that match or exceed the
capabilities recorded in the most advanced general-purpose AI models” (Article
3(64)). A model is _presumed_ to have high-impact capabilities, and thus pose
systemic risk, if the compute threshold of $10^{25}$ floating-point operations
is crossed (Article 51(2)). Regarding the second alternative, it is not
defined and remains somewhat unclear what “capabilities or an impact
equivalent to those set out in point (a)” means. Instead, the European
Commission decides whether these conditions are fulfilled “having regard to
the criteria set out in Annex XIII” (Article 51(1)(b)). These criteria contain
many of the metrics discussed in Section 6.4, including the number of
parameters, the quality and quantity of the training data, and the number of
users (Annex XIII). Overall, the compute threshold currently seems to be the
most concretely outlined way in which a GPAI model can be classified as posing
systemic risk, and may therefore be the most relevant in the near term (Figure
9).

Figure 9: The EU AI Act outlines two main pathways for classifying general-
purpose AI (GPAI) models as posing systemic risk. The first path deems models
as high-risk if they possess “high-impact capabilities” that match or exceed
the most advanced GPAI models, which is presumed if the training compute
exceeds $10^{25}$ floating-point operations (FLOP). Alternatively, the
European Commission can classify models based on criteria outlined in Annex
XIII. Models deemed high-risk through either pathway are subject to stringent
regulations to mitigate systemic risks.

Article 52(1) requires providers to notify the European Commission if a GPAI
model crosses or will cross the threshold laid out in Article 51(1)(a), in
particular by passing the compute thresholds in Article 51(2). This is to be
done “without delay and in any event within two weeks.” Furthermore, Article
55 requires providers of GPAI models with systemic risk to perform model
evaluations, assess and mitigate systemic risks, report serious incidents, and
ensure cybersecurity of the model and its physical infrastructure.131313These
requirements apply in addition to the requirements for all GPAI models, which
include documentation, transparency, and copyright policies (Article 53).
These requirements are relatively vague but will be concretized by “codes of
practice” (Article 56).141414It is very attractive, though not mandatory, for
providers to comply with the codes of practice, because this provides a
“presumption of conformity.” If a provider complies with the codes of
practice, the provider is presumed to also comply with the EU AI Act. But
providers have the option to demonstrate compliance in other ways, too
(Article 55(2)). 151515Ultimately, technical standards are supposed to provide
the presumption of conformity (Article 40(1)). However, the European
Commission has not yet issued the standardization request for GPAI models, and
once this happens it will still take several months or years for the technical
standard to be developed. Therefore, in the meantime, the codes of practice
take on the equivalent role.

The EU AI Act uses compute thresholds mostly in line with what we have argued
can and should be their role (Section 5.1). It focuses on requirements of
regulatory oversight (notification and serious incidents reporting) and
further scrutiny (model evaluations and systemic risk assessments). The
requirements to mitigate systemic risks and ensure adequate cybersecurity may
appear to go beyond what we think compute thresholds should be used for—we
have argued that they should not be used to determine which mitigation
measures need to be implemented. But this is not necessarily how these
requirements need to be interpreted. The codes of practice can and, in our
view, should differentiate further which mitigation measures must be
implemented under which circumstances (for example, based on the results of
model evaluations), such that this is not solely determined by the compute
threshold.

Another important mechanism in the EU AI Act is the possibility for providers
to demonstrate that their model does not pose systemic risk, despite
fulfilling the conditions of Article 51(1)(a)—such as passing the compute
threshold (Article 52(2)). This exception especially makes sense if the
conditions of Article 51(1)(a) are presumed to be fulfilled because the
compute threshold is crossed, given that training compute is only a very crude
proxy for risk (Section 4). The provision is in line with how we have argued
compute thresholds should be used (Section 5.1). If it turns out that a model
does not pose systemic risk despite passing the compute threshold—for example,
based on model evaluations and risk assessments—providers will be allowed to
refrain from further model evaluations, risk assessments, etc., and drop any
precautionary security measures they have already taken. Overall, the EU AI
Act can be understood to embed its compute threshold in a framework similar to
what we have suggested in Section 5.1 (Figure 10).

Figure 10: How the EU AI Act can be understood to implement the framework of
Figure 6. GPAI models exceeding the $10^{25}$ FLOP training compute threshold
(Article 51(1)(a)) are classified as posing systemic risk, triggering
requirements such as notifying the Commission, undergoing model evaluations
and risk assessments, reporting incidents, and implementing precautionary risk
mitigation and cybersecurity measures. Subsequent model capability thresholds,
outlined in Codes of Practice (CoP), further categorize these high-risk models
into low or high systemic risk levels, informing increasingly stringent
mitigation and cybersecurity requirements.

##  6 Challenges for Training Compute Thresholds in GPAI Regulation

The key question when using compute thresholds is at what level to place the
threshold. In Section 6.1, we discuss where a compute threshold should be set
at a given point in time. In Section 6.2, we briefly touch on whether domain-
specific compute thresholds make sense. In Section 6.3, we examine why, when,
and how a compute threshold should be updated over time. In Section 6.4, we
discuss metrics other than training compute that may be useful to distinguish
between more and less risky models, with a focus on but not limited to risk
estimates, model capability evaluations, and effective compute.

###  6.1 Where to Set Compute Thresholds

In this section, we discuss at what level a compute threshold should be set.
Already, different regulators have chosen different values, with the US AI EO
using $10^{26}$ operations and the EU AI Act using $10^{25}$ floating-point
operations, one order of magnitude lower. This difference is relevant, as the
EU threshold likely captures several existing models, while the US threshold
probably does not capture any existing model (Section 5).

In general, at any given point in time, a compute threshold can capture three
tiers of models:

  1. 1.

Models above the frontier (currently $10^{26}$–$10^{27}$ operations). At any
given point in time, such models are the most important ones to include, as
models above the frontier may have unprecedented and hard-to-predict dangerous
capabilities (see Anderljung et al., 2023). Better understanding of and
government visibility into state-of-the-art models is key to identifying
potential issues as early as possible, as well as for getting a sense for what
may be coming down the pike. The advantage of this approach is that it focuses
on the few models that are the most likely to be risky. On the other hand, it
incurs some chance of being underinclusive, missing risky models.

  2. 2.

Models at the frontier (currently $10^{25}$–$10^{26}$ operations). Currently,
including such models may be deemed warranted, as models at the current
frontier may already be considered to show warning signs of some dangerous
capabilities (see Anthropic, 2024; OpenAI et al., 2023; Phuong et al., 2024).
While some companies developing such models have reported that their models do
not possess dangerous capabilities, governments may want to verify those
claims and ensure that all companies developing such models conduct model
evaluations and risk assessments. This seems to be the approach of the EU AI
Act.

  3. 3.

Models below the frontier (currently $10^{24}$–$10^{25}$ operations).
Currently, including such models would be a very cautious approach. There
still is some uncertainty about the full extent of the capabilities of models
below the current frontier (see Anderljung et al., 2023). The advantage of
this approach is that it creates a large “margin of safety.” On the other
hand, it has the biggest chance of being overinclusive, putting regulatory
burdens on companies developing models that later turn out to be not risky.
This is burdensome for regulators, if they want to properly verify company
reports, and for companies themselves. However, in the future, if society does
not keep pace in adapting to increasingly high model capabilities, maintaining
a threshold below the frontier could be warranted (Section 6.3).

Currently, the US AI EO, at least for now, has taken the first approach, only
targeting models above the current frontier. Whether the US will uphold this
approach by updating its compute thresholds with the moving frontier remains
to be seen (Section 6.3). The EU AI Act has explicitly taken the second
approach, targeting models at and above the current frontier: A GPAI model
poses systemic risk if it has “high impact capabilities” (Article 51(1)(a)),
which in turn are defined as “capabilities that match or exceed the
capabilities recorded in the most advanced general-purpose AI models” (Article
3(64)). This arguably means the provision aims to refer to a moving set of
models—those at and above the frontier _at any given point in time_. If the
provision was referring to the models at and above the frontier _at a specific
point in time_ , such as the time of writing or entering into force of the EU
AI Act, this point in time would have been specified for legal certainty.
Based on this interpretation, the compute threshold in Article 51(2) would
need to be updated with the moving frontier (see also Section 6.3).

#### Post-Training Enhancements

An important phenomenon that should affect where to set compute thresholds is
the possibility of post-training enhancements (Section 2). Compute thresholds
should be based on the capabilities of enhanced _systems_ like ChatGPT, rather
than the capabilities of pre-trained models, because those are the
capabilities the models will ultimately obtain. For example, if a regulator
currently sets a threshold at $10^{25}$ operations, this should be because
they have concluded that models like GPT-4 or Gemini Ultra 1.0 and their
superseeding application, such as ChatGPT or Gemini, should be subject to
regulation. This assessment should not be based on the capabilities of the
pre-trained model but rather on those achieved through post-training
enhancements. By contrast, if a regulator currently sets a threshold at
$10^{26}$ operations, this should be because they believe that no current
model possesses concerning capabilities, even after accounting for post-
training enhancements.

Regulators could also add a safety buffer of an order of magnitude or so to
account for potential future improvements in post-training enhancements. For
pre-trained models trained on the same amount of compute, better post-training
enhancements could be developed over time. Future improvements in post-
training enhancements could lead to models trained on a given amount of pre-
training compute being enhanced to display higher and more dangerous
capabilities than current models trained on the same amount of pre-training
compute but enhanced with current techniques only. To account for further
unknown post-training enhancements, regulators could add a safety buffer in
“equivalent pre-training compute” (Davidson et al., 2023).

However, we expect that for the foreseeable future there is likely a limit to
the increase in model capabilities that can be gained from post-training
enhancements. The capabilities of the model determined by pre-training will
likely remain critical. While post-training enhancements may continue to
improve incrementally, they are unlikely to enable models to “leapfrog” or
“skip multiple generations” of compute scaling and achieve capabilities
comparable to those of models pre-trained with orders of magnitude more
compute (Davidson et al., 2023). Furthermore, certain post-training
enhancements work better with larger models (Li et al., 2024), suggesting
their effectiveness may depend on leveraging the capabilities of models with
significant pre-training compute. In other words, while post-training
enhancements can potentially enhance the absolute capabilities of models, they
are less likely to significantly alter the relative capabilities compared to
models pre-trained with orders of magnitude more compute, especially for more
advanced models.

###  6.2 Domain-Specific Compute Thresholds

The focus of this paper up to this point has been on compute thresholds for
GPAI models. This section briefly discusses domain-specific compute thresholds
for narrower, more specialized models. Already, the US AI EO includes a
compute threshold for models specialized in the domain of biology. The
considerations for whether it makes sense to use domain-specific compute
thresholds in general, and in the domain of biology in particular, are the
same as the considerations for whether it makes sense to use compute
thresholds for GPAI models.

Generally, the key consideration is whether scaling laws apply in the domain
in question. Regulating models based on the amount of training compute only
makes sense if there is a strong relationship between the amount of compute
used to train a model and the level of risk the model poses. Another important
consideration is how many actors will be in scope of a given compute
threshold. If too many actors are in scope, i.e., the amount of computational
power to surpass the threshold is available to a broad range of actors, a
domain-specific compute threshold may not be practicable. Both of these
considerations may differ from domain to domain.

We are uncertain whether compute thresholds in the domain of biology make
sense and can only provide initial thoughts: In the domain of biology, a
distinction can be made between AI-enabled biological tools (BTs) and large
language models (LLMs) with a focus on biology (Sandbrink, 2023; Smith et al.,
2024). AI-enabled BTs are AI models that are highly specialized on a
particular biological task, such as predicting protein structures, like
AlphaFold 2 (Jumper et al., 2021). LLMs with a focus on biology are LLMs
trained on a lot of biological sequence data, and can be used for a variety of
applications, such as ESM3 (Hayes et al., 2024). It is uncertain if compute
thresholds are appropriate for AI-enabled BTs, as the literature on scaling
laws is not well established for those. In contrast to GPAI models, other
factors, such as the quality of data, are much more important for their
capabilities. Moreover, AI-enabled BTs generally require about $100\times$
less compute to train than GPAI models, such that a large number of actors
would be in scope of requirements based on compute thresholds (Smith et al.,
2024). For LLMs with a focus on biology, compute thresholds could make sense,
but the literature on scaling laws is similarly scarce (e.g., Hesslow et al.
(2022); Serrano et al. (2024)).

###  6.3 Why, When, and How to Update Compute Thresholds

Compute thresholds will very likely need to be updated over time. Generally,
it is important to maintain flexibility in regulations for a poorly understood
and rapidly evolving technology like GPAI. First and foremost, compute
thresholds should be adjusted with better knowledge about what constitutes
risky models. What is more, depending on improvements in algorithmic
efficiency and computational price-performance, as well as changes in the
offense-defense balance of AI systems, the threat landscape, and societal
vulnerability or adaptation, compute thresholds will need to be adjusted
downwards or upwards over time.

The US AI EO states that its thresholds for models in scope should regularly
be updated:

> _“The Secretary of Commerce, in consultation with the Secretary of State,
> the Secretary of Defense, the Secretary of Energy, and the Director of
> National Intelligence, shall define, and thereafter update as needed on a
> regular basis, the set of technical conditions for models and computing
> clusters that would be subject to the reporting requirements of subsection
> 4.2(a) of this section.”_ (Section 4.2(b))

The EU AI Act also states that its thresholds for GPAI models with systemic
risk should be updated based on technological developments:

> _“The Commission shall adopt delegated acts in accordance with Article 97 to
> amend the thresholds listed in paragraphs 1 and 2 of this Article, as well
> as to supplement benchmarks and indicators in light of evolving
> technological developments, such as algorithmic improvements or increased
> hardware efficiency, when necessary, for these thresholds to reflect the
> state of the art.”_ (Article 51(3))

Given that GPAI is a poorly understood and rapidly evolving technology, any
GPAI regulation should have the ability to be updated. On a high level, there
are two approaches to updating the level of compute thresholds over time:
specifying a fixed value but tasking regulators with updating that value over
time, or specifying a dynamic value that automatically moves according to
certain rules over time. Both the US and the EU have specified fixed values
but tasked regulators with updating those values over time. By contrast, for
example, a compute threshold could specify a dynamic value that automatically
moves over time by being tied to the highest-compute model at any given point
in time, or, alternatively, by being set at an order of magnitude below the
highest-compute model at any given point in time. But even a dynamic compute
threshold should allow for updates by regulators to account for a better
understanding of what constitutes risky models as well as for technological
and other relevant developments.

The frequency and the direction of updates to training compute thresholds
depend on several developments. First and foremost, over time, our
understanding of what constitutes risky models may improve. Currently,
uncertainty about model capabilities and societal risks is high. Regulators
impose requirements on GPAI models not because they are sure those models pose
a high level of risk, but because they might. As such, if the current approach
is motivated by precaution and has the goal to detect when GPAI models reach
some level of risk, the compute threshold can be expected to move up over time
as our understanding of model capabilities and societal risks improves.

Moreover, compute thresholds may need to be adjusted downwards over time to
account for algorithmic efficiency improvements. In the context of model
training, algorithmic efficiency refers to the amount of compute required to
run a given algorithm used to train a model. Over time, ongoing research
yields improved model architectures and training algorithms, reducing the
amount of compute needed to train models of a given level of capabilities
(Pilz et al., 2023). Adjusting the compute threshold downwards at the pace of
algorithmic efficiency improvements would maintain its correspondence to a
given level of capabilities. However, algorithmic efficiency improvements can
also mean that more actors will be able to train models with a given amount of
compute and thus obtain a given level of capabilities. This can lead to high
regulatory burdens on a large number of less well-resourced actors like
startups and academics, and, correspondingly, high oversight costs for
regulators (Pistillo et al., forthcoming). As a result, algorithmic efficiency
improvements can also provide an argument to adjust the compute threshold
upwards over time. Taken together, algorithmic efficiency improvements may
pose a regulatory dilemma (Winter & Bullock, forthcoming).

At the same time, algorithmic efficiency improvements merely define the upper
limit for how quickly the compute threshold may need to be adjusted downwards
to apply to models posing a given level of risk. Despite improvements in
algorithmic efficiency, developments such as the offense-defense balance of AI
systems, the threat landscape, and societal vulnerability or adaptation
suggest the threshold may need slower downwards adjustments, remain unchanged,
or even be adjusted upwards to catch models of a given level of risk. GPAI
models may turn out to enable defensive uses less, as much as, or even more
than offensive uses, for example, in the cyber domain (AI Policy Perspectives
& Krier, 2024; Buterin, 2023; Lohn & Jackson, 2022). Malicious actors may for
various reasons—including such unrelated to GPAI models—become more or less
numerous, able and willing to use GPAI models. Over time, society could become
better at managing risks from models, meaning a model of a given level of
capabilities may pose less risk (Anderljung et al., 2023; Bernardi et al.,
2024; Kapoor et al., 2024). Under these circumstances, compute thresholds may
be adjusted upwards over time to continue to focus on models posing a given
level of risk.

Another development that can be relevant for adjusting compute thresholds is
computational price-performance improvements. In the context of model
training, computational price-performance refers to the amount of compute that
can be obtained for a given amount of money (Hobbhahn et al., 2023). Over
time, the amount of training compute available for a given amount of money
increases due to innovations in hardware manufacturing and design (Pilz et
al., 2023). Importantly, such improvements in computational price-performance
do not influence the amount of training compute necessary to achieve a given
level of capabilities, meaning they do not influence the level of risk from
models that cross a given compute threshold. However, improvements in
computational price-performance may increase regulatory burdens and oversight
costs: More actors may be able to afford the amount of compute required to
train models that meet the regulatory threshold (Pilz et al., 2023). Again,
this does not influence the level of risk from models that cross a given
compute threshold. But it might still be an argument to adjust the compute
threshold upwards to reduce regulatory burdens and oversight costs. By doing
so, however, regulators accept a higher level of risk for society.161616For a
more comprehensive discussion of this issue, see Pilz et al. (2023).

The US AI EO tasks the Secretary of Commerce with updating its compute
thresholds, without specifying further details on how to do so (Section
4.2(b)). We have provided what we believe to be the key substantive
considerations. The Secretary of Commerce should monitor these developments
and aim to increase understanding of risks from GPAI models. On the procedural
side, the US AI EO specifies that updates should occur on a “regular basis.”
An examination of whether a compute threshold needs to be updated should
happen at least every couple of months, and maybe more frequently if the pace
of algorithmic efficiency or computational price-performance improvements
increases.

The EU AI Act provides some guidance on how the European Commission should
update its thresholds for identifying GPAI models with systemic risk,
referring to “technological developments, such as algorithmic improvements or
increased hardware efficiency” (Article 51(3)). As argued above, the mention
of algorithmic (efficiency) improvements is important. However, the reference
to “increased hardware efficiency” is ambiguous and might reflect a
misunderstanding. To the extent that this term refers to innovations in
hardware manufacturing and design that may lead to computational price-
performance improvements, as explained above, corresponding developments do
not influence the level of risk from a given compute threshold. Instead, they
only affect the accessibility and proliferation of models that meet a given
compute threshold. As hardware becomes more efficient, the same amount of
compute becomes available at a lower cost and therefore potentially to a
broader range of actors. Consequently, hardware efficiency improvements should
be considered when adjusting compute thresholds, not because they alter the
relationship between training compute and model capabilities, but because they
may increase the number of actors in scope of a given compute threshold.
Furthermore, the EU AI Act guidance ignores two crucial factors: better
understanding of what constitutes risky models, and societal developments that
influence the level of risk posed by models with a given level of
capabilities.

###  6.4 Alternative Metrics

Given the limitations of training compute, several other metrics may be useful
to distinguish between more and less risky models. For example, the EU AI Act
mentions a variety of metrics that should play a role in identifying GPAI
models with systemic risk: the number of model parameters, the quantity and
quality of training data, the amount of training compute, input and output
modalities and type, model capabilities, a high impact on the internal market
due to reach (presumed if at least 10 000 registered business users in the
Union), and the number of registered end users (Annex XIII). Other metrics
commonly discussed include risk estimates, model capability evaluations,
effective compute, and some other metrics.

####  6.4.1 Risk Estimates

The most straightforward way to distinguish between more and less risky models
may be to directly measure, or rather estimate, the increase in risk from a
model. On the one hand, risk estimates are directly focused on harm to
individuals, groups, and society as a whole. Risk estimates do not face the
issue of potentially focusing on wrong proxies, such as too little compute or
harmless capabilities (Koessler et al., 2024). On the other hand, risk
estimates are extremely hard to do well for an emerging, general-purpose
technology like GPAI. Little historical data and a tremendous amount of risk
scenarios mean risk estimates have to rely on models and expert judgment, and
they face the issue of missing important risk scenarios (Schuett et al.,
forthcoming). Overall, risk estimates require a lot of effort to conduct,
meaning they should not replace compute thresholds as an initial filter. After
that filter has been applied, risk estimates are the ideal, but still highly
immature, metric to decide whether mitigation measures are necessary. Risk
thresholds defined in terms of risk estimates should thus not yet determine,
but only inform decisions about whether mitigation measures are necessary
(Koessler et al., 2024).

####  6.4.2 Model Capability Evaluations

Another metric useful for distinguishing between more and less risky models is
model capability evaluations. Currently, the two most common approaches to
evaluations are “benchmarking” and “red-teaming.” Benchmarking involves using
standardized tests to evaluate model capabilities—these benchmarks allow
comparisons between different models. Well-known benchmarks include BBQ (bias;
Parrish et al. 2022), GLUE (natural language understanding; Wang et al. 2019),
and MMLU (general knowledge; Hendrycks et al. 2021). Red-teaming involves less
standardized and more in-depth testing, often by domain experts, for specific
model capabilities and behaviors (Barrett et al., 2024; Jones et al., 2024).

Risk-tracking: Model capabilities are attractive as a metric because they can
be considered a proxy for risk and they are more closely related to risks than
is training compute (Section 4). However, model capabilities are an imperfect
proxy, too. Other important factors include the offense-defense balance of AI
systems, the threat landscape, and societal vulnerability or adaptation.
Regulating consistent levels of capability over time may therefore not be
warranted. At the same time, model capabilities may create a false sense of
exhaustiveness, potentially leading to underestimating the need for risk
assessments (Jones et al., 2024).

Difficult to measure: Evaluating a model’s capabilities is very challenging
(Anwar et al., 2024; Jones et al., 2024; Reuel et al., 2024). In particular,
proving the absence of a capability is difficult, implying a potentially large
amount of false negatives (Casper et al., 2024). Furthermore, models are
highly prompt sensitive, meaning slight changes in inputs can lead to large
differences in outputs (Mizrahi et al., 2024; Ramesh et al., 2023; Sclar et
al., 2024). The appropriate point in a model’s lifecycle to evaluate
capabilities can also lead to significant differences in results, as discussed
in Section 2. In general, there still is large disagreement in the research
community about what constitutes adequate model capability evaluations, as
evidenced by many competing benchmarks for similar capabilities (e.g., MATH,
MGSM, and GSM8K for mathematical capabilities) (Anwar et al., 2024; Jones et
al., 2024; Reuel et al., 2024). The field of dangerous model capability
evaluations is especially nascent, with the first paper on the topic having
been published about one year ago (Shevlane et al., 2023). Finally,
comprehensive model capability evaluations require significant effort (Anwar
et al., 2024; Jones et al., 2024; Reuel et al., 2024).

Easy to circumvent: Model capability evaluations may be circumvented. Related
to the previous point, easy-to-run evaluations may be easy to game, allowing
companies to design models that perform poorly on benchmarks while still
excelling at the capabilities the benchmarks are meant to measure. The lack of
standardization makes it particularly easy to inflate evaluation results
(Leech et al., 2024). For example, companies may strategically choose which
evaluations to conduct (Jones et al., 2024). Companies or deceptive AI models
could also strategically underperform on a given set of evaluations, a
phenomenon referred to as “sandbagging” (Järviniemi & Hubinger, 2024; van der
Weij et al., 2024).

Not measurable before development: Training compute is usually known prior to
development, whereas model capability evaluations can only be conducted during
and after development. There have been attempts to forecast model capabilities
(Phuong et al., 2024); however, such forecasting is likely still highly
unreliable, as indicated by high disagreement between forecasters.

Difficult to verify externally: Model capabilities are also much harder to
verify than training compute is. Since measuring model capabilities is more
difficult, verifying these measurements is more difficult, too. Actors
verifying model capabilities need more information and may even need to
conduct model capability evaluations themselves. Regulators currently may not
have the necessary expertise, and there is no mature third-party ecosystem,
either. In the worst case, publishing model capability data might expose a
company’s algorithmic progress, potentially intensifying competitive dynamics
in the AI industry.

Cost-tracking: Model capabilities are highly correlated with the cost of
developing the model, since increases in capabilities largely stem from
increases in training compute (Section 3).

Overall, training compute and model capabilities should be viewed as
complementary metrics. Training compute excels at providing a quick, easily
measurable, and externally verifiable metric to identify potentially risky
models and trigger regulatory oversight and further scrutiny. Model
capabilities are more expensive and harder to measure but also provide more
information about a model’s risks. By using compute thresholds as a first-pass
filter and then applying capability thresholds, regulators can create a more
efficient and effective regulatory process (Section 5.1).

####  6.4.3 Effective Compute

Algorithmic efficiency improvements, including through advancements in model
architecture and training methods, can over time reduce the amount of compute
required to train models to similar levels of performance (Erdil & Besiroglu,
2023; Hernandez & Brown, 2020; Ho et al., 2024; Sherry & Thompson, 2021). As a
result, there is growing interest in using a metric called _effective compute_
, which accounts for both increases in training compute and algorithmic
efficiency improvements. Effective compute describes the equivalent increase
in training compute that would be needed to match a given model performance
absent algorithmic efficiency improvements (Ho et al., 2024). Effective
compute is a relative metric, as improvements in algorithmic efficiency are
measured against a specific model performance. For example, effective compute
may be expressed as how many times more compute would have been required to
train a model to perform as well as a specific other model on a specific
performance metric, had there been no algorithmic improvements (e.g.,
$4\times$ effective compute relative to GPT-4 test loss).

Risk-tracking: Effective compute is as risk-tracking as the model performance
metric it is tied to. For example, if this performance metric is a benchmark,
then effective compute is a good of a proxy as the benchmark. If the
performance metric is loss, then effective compute is as good as training
compute. However, as with model capabilities, effective compute is only a
proxy for risk. Regulating consistent levels of performance over time may
therefore not be warranted. Depending on the performance metric, effective
compute can even create a false sense of being a comprehensive measure of
risk—as it combines training compute and model capabilities—potentially
leading to underestimating the need for risk assessments.

Difficult to measure: Effective compute is more difficult to measure than
training compute is. Research on algorithmic efficiency improvements for AI
models is sparse, with only a few papers published (Erdil & Besiroglu, 2023;
Hernandez & Brown, 2020; Ho et al., 2024; Sherry & Thompson, 2021). There is
also no agreement on two key questions: which performance metric to use and at
what point during the model lifecycle to assess performance on that metric.
Any definition of effective compute needs to choose a performance metric
(e.g., test loss) as the reference point that the amount of compute required
is “normalized” to. Technical metrics like test loss are easier to measure and
scale more smoothly with compute, but they may not accurately represent real-
world impact, which makes them similar to training compute as a metric while
concealing this fact. Metrics like model capability evaluations are more
relevant for risk but harder to measure and subject to sudden or unpredictable
increases in performance, such as the emergence of new capabilities (Figure
11). The appropriate point in the model’s lifecycle to measure performance on
a given metric also remains unclear. As discussed in Section 2, performance
gains can result from post-training enhancements like fine-tuning or prompting
methods, often resulting in relatively larger increases in effective compute,
primarily driven by the performance improvements achieved with a comparatively
small additional compute investment. Furthermore, there is ambiguity regarding
which prompting techniques (such zero-shot, few-shot, or chain-of-thought)
should be allowed when measuring model capabilities, adding to the difficulty
of making reliable statements about effective compute.

Figure 11: Spectrum performance metrics effective compute can be tied to.
Technical metrics like training loss, validation loss, and test loss scale
more smoothly with compute, are easier to measure and verify, and are less
context-dependent. In contrast, capability benchmarks may scale more suddenly
with compute, are closer to real-world utility and impact, but can sometimes
be more _reductive_ to specific tests.

Easy to circumvent: The ease of circumventing effective compute depends on the
chosen performance metric. Quantitative metrics like training loss or
validation performance are more robust, but if based on evaluations or
benchmarks, effective compute faces similar circumvention risks as those of
model capability evaluations (Section 6.4.2). Companies could game evaluations
by designing models to underperform while still excelling at intended
capabilities, strategically selecting evaluations, or intentionally
underperforming (“sandbagging”).

Not measurable before development: Training compute is usually known prior to
development, whereas effective compute requires analysis of performance after
development. As with model capabilities, performance can be forecasted, but
such forecasts are still highly unreliable.

Difficult to verify externally: Verifying effective compute may require access
to more proprietary information than does training compute. Publishing
effective compute data alongside standard training compute data could expose a
company’s algorithmic progress, potentially intensifying competitive dynamics
in the AI industry.

Cost-tracking: Effective compute is correlated with the cost of developing the
model, since increases in performance largely stem from increases in training
compute (Section 3).

In conclusion, there currently are practical challenges to using effective
compute as a metric. More research is needed to develop standardized ways to
measure effective compute that address these challenges. If successful, an
effective compute measure could become the “GDP per capita” of AI performance
(e.g., by creating “a nominal basket of benchmarks”). Effective compute is
currently most suitable for internal company use, where the necessary tools
and insights for accurate assessment are readily available. For example,
companies can use effective compute for selecting checkpoints for their safety
policies (Anthropic, 2023; Dragan et al., 2024; OpenAI, 2023). External
entities, including regulatory bodies, may struggle to accurately assess
effective compute due to the lack of standardized methods and limited access
to models. Moreover, regulatory bodies may not yet possess the technical
expertise required to accurately assess effective compute. Given these
challenges, we currently recommend not using effective compute for regulatory
purposes.

####  6.4.4 Parameters, Data, and Other Metrics

This section provides an initial analysis and comparison of other metrics to
the more extensively discussed ones above. The metrics in this section include
most of the main variables in model development and deployment. In the
discussion of each metric, we assume the values of all other variables remain
fixed (e.g., when talking about the correlation between model architecture and
risk we assume a fixed amount of training compute). We highlight that the
discussion in this section is preliminary and many points we make need further
work and empirical investigation.

_Model architecture and training algorithm_ : The model’s architecture and
training algorithm influence its capabilities and thus risk, but empirically
to a much lesser extent than the amount of training compute (Ho et al., 2024;
Kaplan et al., 2020; Sutton, 2019). Moreover, these metrics are hard to
quantify, they often involve step changes, and they are multidimensional with
complex synergies between different dimensions (e.g., a certain number of
layers might work well with a certain learning rate but not another).

_Number of model parameters_ : The number of model parameters tracks
capabilities and thus risk (Kaplan et al., 2020) and can simply be counted
(Villalobos et al., 2022). However, certain model architectures like Mixture
of Experts (MoE) can significantly change the number of model parameters
without necessarily increasing the model’s capabilities and risk (Villalobos
et al., 2022), and the number of model parameters can be manipulated through
post-training techniques like model pruning without necessarily decreasing the
model’s capabilities and risk (Cheng et al., 2023). What is more, the number
of model parameters affects the necessary amount of training compute and is
thus captured by that metric. More training compute is required to handle more
parameters trained on a given amount of data. Additionally, developers can be
expected to aim for an optimal ratio between the number of model parameters
and the amount of training data as described by scaling laws. Therefore, for a
given architecture and assuming training compute allocation was optimal
according to scaling laws, training compute implies the number of model
parameters.

_Amount and quality of training data_ : The amount of training data affects
model capabilities and risk (Kaplan et al., 2020) and can be quantified in
tokens or bytes (Villalobos et al., 2024). However, data quality is also
crucial, influencing capabilities across dimensions like information density
and diversity, yet lacking objective or standardized measurement methods
(Mitchell et al., 2023). Importantly, amount and quality are interrelated -
more data does not guarantee better performance with poor quality, while high-
quality data can compensate for smaller sizes (Evans et al., 2024; Gunasekar
et al., 2023). Additionally, the number of epochs or passes over the training
data during the training process also affects the total compute requirements,
even with a fixed dataset size. Developers optimize the ratio of parameters,
data size, data quality, and number of epochs per scaling laws. Thus, for a
given architecture with optimal compute allocation, training compute serves as
an indirect proxy for the amount of training data utilized, as well as the
number of training epochs but not for the quality.

_Number of users_ : The number of users correlates with risks from accidents
and, to some extent, misuse (the larger the user base, the more likely misuse
occurs). However, it may not correlate with risks from misalignment or risks
that arise before deployment, such as malicious actors stealing and misusing
the model. This metric is difficult to estimate before deployment and even
more so before development.

_Applications_ : While applications are often known before deployment (except
if model weights are publicly released), many GPAI models are used in general-
purpose tools, like GPT-4 powering ChatGPT. Before development, it often does
not make sense to consider applications, as GPAI models, by definition, have
broad capabilities and can be used for many downstream applications (Jones,
2023).

_Harm_ : This metric is the materialization of risk. However, it is not known
in advance of development or deployment. The best we can do in advance of
development and deployment is to estimate the likelihood and severity of harm,
that is, estimate risk (Section 6.4.1). Of course, such risk estimates should
be updated with the data gathered through monitoring, and development or
deployment should be discontinued at any time if a particular number or type
of incidents occur. However, at that point, the harm from those incidents
cannot be undone. Therefore, in particular for irreversible and large-scale
harm, in addition to ex post metrics that measure risk in hindsight, we need
ex ante metrics that measure risk in advance.

Features/ Metrics Correlation with risk Ease of measurement Difficulty of
circumvention Measurable before development and deployment Externally
verifiable Correlation with cost Frequency of updates required Model
architecture and training algorithm Low Low Low to medium Yes Medium Low to
medium Medium Number of model parameters Low to medium High Medium Yes High
Medium to high Medium Amount and quality of training data Medium Medium Medium
to high Yes Medium to high Medium to high Medium Training compute Medium High
High Yes High High Medium Loss Medium to high Medium to high High Yes before
deployment; no before development Medium Medium to high Medium Loss Medium to
high Medium to high High Yes before deployment; no before development Medium
Medium to high Medium Model capability evaluations Medium to high Medium Low
to medium Yes before deployment; no before development Medium Medium to high
Medium Effective compute Medium to high Medium Medium Yes before deployment;
no before development Medium Medium to high Low to medium Number of users High
for some risks; low for other risks High High Sometimes before deployment; no
before development Medium Medium High Applications High for some risks; low
for other risks Medium Medium Sometimes Medium Medium Low Risk estimates High
Low Medium Yes Medium Medium to high Low Harm High Low Medium No Medium Medium
to high Low

Table 1: First-pass ranking of potential metrics for GPAI regulation regarding
the features discussed in Section 3.

Complementing mere compute thresholds with other metrics becomes relevant to
the extent that scaling laws cease to hold and training compute becomes a
worse proxy for risk (Section 4). Particularly relevant combinations may
include training compute and model capability evaluations (to ensure catching
the most capable models) and training compute and number of users (to ensure
catching the most widely used models). However, any threshold that is supposed
to serve as an initial filter to identify models of potential concern should
be based on metrics that can be measured easily and early in the model
lifecycle. This, at least currently, excludes model capability evaluations,
and at least before deployment, the number of users.

##  7 Conclusion

While not perfect, compute thresholds are currently one of the key tools in
GPAI regulation. They offer a risk-correlated, easily measurable, and
externally verifiable metric that can inform regulatory decisions while
minimizing circumvention and targeting the most well-resourced actors. They
are currently the best tool for identifying potentially risky models and
triggering regulatory oversight and further scrutiny.

Further research is necessary on many questions related to compute thresholds.
In particular, the trends relevant for updating compute thresholds require
further research—both empirical research on their current development and
theoretical research on their future development (Section 6.3). Moreover, the
usefulness of metrics other than training compute should be studied further,
especially effective compute, which we consider a highly promising metric
(Section 6.4.3), and how to combine training compute with other metrics such
as model capability evaluations or the number of users (Section 6.4.4).
Another area of research we would like to highlight concerns the enforcement
of compute thresholds. What information do regulators need? What process
should regulators rely on to ensure they get that information (e.g., should
they inspect companies)? What special considerations apply if companies are
not physically located within the territory of the regulator’s jurisdiction?

Finally, compute thresholds are not sufficient for GPAI regulation—they are
one tool among many. Their effectiveness depends on the specific context and
the design of the overall regulatory framework. In particular, the discussion
of whether capability thresholds are better than compute thresholds is
misguided. Capability thresholds are supplemental, and while they might play a
more important role in the future, compute thresholds are likely to remain a
valuable tool in GPAI regulation, too. By understanding the strengths and
limitations of compute thresholds, policymakers can make informed decisions
about when and how to use them as part of a comprehensive approach to GPAI
regulation.

## Acknowledgments

We are grateful for valuable input from the following individuals (in
alphabetical order by surname): Neel Alex, Markus Anderljung, Mauricio Baker,
Lukas Berglund, Benjamin Bucknall, Alan Chan, Stephen Clare, Connor Dunlop,
Jimmy Farrell, James Gealy, John Halstead, Marius Hobbhahn, Nicolas Moës,
James Petrie, Konstantin Pilz, Jonas Sandbrink, Risto Uuk, and many others.
All views and remaining errors are our own.

## References

  * AI Policy Perspectives & Krier (2024) AI Policy Perspectives and Krier, S.  Models on the frontline: AI’s defensive role, May 2024.  URL https://www.aipolicyperspectives.com/p/models-on-the-frontline-ais-defensive.
  * Anderljung et al. (2023) Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O’Keefe, C., Whittlestone, J., Avin, S., Brundage, M., Bullock, J., Cass-Beggs, D., Chang, B., Collins, T., Fist, T., Hadfield, G., Hayes, A., Ho, L., Hooker, S., Horvitz, E., Kolt, N., Schuett, J., Shavit, Y., Siddarth, D., Trager, R., and Wolf, K.  Frontier AI Regulation: Managing Emerging Risks to Public Safety.  _arXiv preprint arXiv:2307.03718 [cs]_ , November 2023.  doi:10.48550/arXiv.2307.03718.
  * Anthropic (2023) Anthropic.  Anthropic’s responsible scaling policy, September 2023.  URL https://www.anthropic.com/news/anthropics-responsible-scaling-policy.
  * Anthropic (2024) Anthropic.  Claude 3.5 sonnet model card addendum, June 2024.  URL https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf.
  * Anwar et al. (2024) Anwar, U., Saparov, A., Rando, J., Paleka, D., Turpin, M., Hase, P., Lubana, E. S., Jenner, E., Casper, S., Sourbut, O., Edelman, B. L., Zhang, Z., Günther, M., Korinek, A., Hernandez-Orallo, J., Hammond, L., Bigelow, E., Pan, A., Langosco, L., Korbak, T., Zhang, H., Zhong, R., hÉigeartaigh, S. O., Recchia, G., Corsi, G., Chan, A., Anderljung, M., Edwards, L., Bengio, Y., Chen, D., Albanie, S., Maharaj, T., Foerster, J., Tramer, F., He, H., Kasirzadeh, A., Choi, Y., and Krueger, D.  Foundational Challenges in Assuring Alignment and Safety of Large Language Models.  _arXiv preprint arXiv:2404.09932 [cs]_ , April 2024.  doi:10.48550/arXiv.2404.09932.
  * Barrett et al. (2024) Barrett, A. M., Jackson, K., Murphy, E. R., Madkour, N., and Newman, J.  Benchmark Early and Red Team Often: A Framework for Assessing and Managing Dual-Use Hazards of AI Foundation Models.  _arXiv preprint arXiv:2405.10986 [cs]_ , May 2024.  doi:10.48550/arXiv.2405.10986.
  * Basdevant et al. (2024) Basdevant, A., François, C., Storchan, V., Bankston, K., Bdeir, A., Behlendorf, B., Debbah, M., Kapoor, S., LeCun, Y., Surman, M., King-Turvey, H., Lambert, N., Maffulli, S., Marda, N., Shivkumar, G., and Tunney, J.  Towards a Framework for Openness in Foundation Models: Proceedings from the Columbia Convening on Openness in Artificial Intelligence.  _arXiv preprint arXiv:2405.15802 [cs.SE]_ , May 2024.  URL https://arxiv.org/abs/2405.15802v1.
  * Bengio et al. (2024) Bengio, Y., Hinton, G., Yao, A., Song, D., Abbeel, P., Darrell, T., Harari, Y. N., Zhang, Y.-Q., Xue, L., Shalev-Shwartz, S., Hadfield, G., Clune, J., Maharaj, T., Hutter, F., Baydin, A. G., McIlraith, S., Gao, Q., Acharya, A., Krueger, D., Dragan, A., Torr, P., Russell, S., Kahneman, D., Brauner, J., and Mindermann, S.  Managing extreme AI risks amid rapid progress.  _Science_ , 384(6698):842–845, May 2024.  doi:10.1126/science.adn0117.
  * Bernardi et al. (2024) Bernardi, J., Mukobi, G., Greaves, H., Heim, L., and Anderljung, M.  Societal Adaptation to Advanced AI.  _arXiv preprint arXiv:2405.10295 [cs]_ , May 2024.  doi:10.48550/arXiv.2405.10295.
  * Bommasani et al. (2022) Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., Brynjolfsson, E., Buch, S., Card, D., Castellon, R., Chatterji, N., Chen, A., Creel, K., Davis, J. Q., Demszky, D., Donahue, C., Doumbouya, M., Durmus, E., Ermon, S., Etchemendy, J., Ethayarajh, K., Fei-Fei, L., Finn, C., Gale, T., Gillespie, L., Goel, K., Goodman, N., Grossman, S., Guha, N., Hashimoto, T., Henderson, P., Hewitt, J., Ho, D. E., Hong, J., Hsu, K., Huang, J., Icard, T., Jain, S., Jurafsky, D., Kalluri, P., Karamcheti, S., Keeling, G., Khani, F., Khattab, O., Koh, P. W., Krass, M., Krishna, R., Kuditipudi, R., Kumar, A., Ladhak, F., Lee, M., Lee, T., Leskovec, J., Levent, I., Li, X. L., Li, X., Ma, T., Malik, A., Manning, C. D., Mirchandani, S., Mitchell, E., Munyikwa, Z., Nair, S., Narayan, A., Narayanan, D., Newman, B., Nie, A., Niebles, J. C., Nilforoshan, H., Nyarko, J., Ogut, G., Orr, L., Papadimitriou, I., Park, J. S., Piech, C., Portelance, E., Potts, C., Raghunathan, A., Reich, R., Ren, H., Rong, F., Roohani, Y., Ruiz, C., Ryan, J., Ré, C., Sadigh, D., Sagawa, S., Santhanam, K., Shih, A., Srinivasan, K., Tamkin, A., Taori, R., Thomas, A. W., Tramèr, F., Wang, R. E., Wang, W., Wu, B., Wu, J., Wu, Y., Xie, S. M., Yasunaga, M., You, J., Zaharia, M., Zhang, M., Zhang, T., Zhang, X., Zhang, Y., Zheng, L., Zhou, K., and Liang, P.  On the Opportunities and Risks of Foundation Models.  _arXiv preprint arXiv:2108.07258 [cs]_ , July 2022.  doi:10.48550/arXiv.2108.07258.
  * Branwen (2020) Branwen, G.  The Scaling Hypothesis, May 2020.  URL https://gwern.net/scaling-hypothesis.
  * Breitbarth (2018) Breitbarth, P.  On large-scale data processing and GDPR compliance, August 2018.  URL https://iapp.org/news/a/on-large-scale-data-processing-and-gdpr-compliance.
  * Brown et al. (2020) Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D.  Language Models are Few-Shot Learners.  _arXiv preprint arXiv:2005.14165 [cs]_ , July 2020.  doi:10.48550/arXiv.2005.14165.
  * Brundage et al. (2020) Brundage, M., Avin, S., Wang, J., Belfield, H., Krueger, G., Hadfield, G., Khlaaf, H., Yang, J., Toner, H., Fong, R., Maharaj, T., Koh, P. W., Hooker, S., Leung, J., Trask, A., Bluemke, E., Lebensold, J., O’Keefe, C., Koren, M., Ryffel, T., Rubinovitz, J. B., Besiroglu, T., Carugati, F., Clark, J., Eckersley, P., de Haas, S., Johnson, M., Laurie, B., Ingerman, A., Krawczuk, I., Askell, A., Cammarota, R., Lohn, A., Krueger, D., Stix, C., Henderson, P., Graham, L., Prunkl, C., Martin, B., Seger, E., Zilberman, N., hÉigeartaigh, S. O., Kroeger, F., Sastry, G., Kagan, R., Weller, A., Tse, B., Barnes, E., Dafoe, A., Scharre, P., Herbert-Voss, A., Rasser, M., Sodhani, S., Flynn, C., Gilbert, T. K., Dyer, L., Khan, S., Bengio, Y., and Anderljung, M.  Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims.  _arXiv preprint arXiv:2004.07213 [cs]_ , April 2020.  doi:10.48550/arXiv.2004.07213.
  * Buterin (2023) Buterin, V.  My techno-optimism, November 2023.  URL https://vitalik.eth.limo/general/2023/11/27/techno_optimism.html.
  * Casper et al. (2024) Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., Sharkey, L., Krishna, S., Von Hagen, M., Alberti, S., Chan, A., Sun, Q., Gerovitch, M., Bau, D., Tegmark, M., Krueger, D., and Hadfield-Menell, D.  Black-Box Access is Insufficient for Rigorous AI Audits.  In _Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’24, pp. 2254–2272, New York, NY, USA, June 2024. Association for Computing Machinery.  ISBN 9798400704505.  doi:10.1145/3630106.3659037.
  * Cheng et al. (2023) Cheng, H., Zhang, M., and Shi, J. Q.  A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations.  _arXiv preprint arXiv:2308.06767 [cs]_ , August 2023.  doi:10.48550/arXiv.2308.06767.
  * Davidson et al. (2023) Davidson, T., Denain, J.-S., Villalobos, P., and Bas, G.  AI capabilities can be significantly improved without expensive retraining.  _arXiv preprint arXiv:2312.07413 [cs.AI]_ , December 2023.  URL https://arxiv.org/abs/2312.07413v1.
  * Dragan et al. (2024) Dragan, A., King, H., and Dafoe, A.  Introducing the frontier safety framework, May 2024.  URL https://deepmind.google/discover/blog/introducing-the-frontier-safety-framework/.
  * Egan & Heim (2023) Egan, J. and Heim, L.  Oversight for Frontier AI through a Know-Your-Customer Scheme for Compute Providers.  _arXiv preprint arXiv:2310.13625 [cs]_ , October 2023.  doi:10.48550/arXiv.2310.13625.
  * Epoch AI (2024a) Epoch AI.  Data on Notable AI Models, June 2024a.  URL https://epochai.org/data/notable-ai-models.
  * Epoch AI (2024b) Epoch AI.  Machine Learning Trends, June 2024b.  URL https://epochai.org/trends.
  * Erdil & Besiroglu (2023) Erdil, E. and Besiroglu, T.  Algorithmic progress in computer vision.  _arXiv preprint arXiv:2212.05153 [cs]_ , August 2023.  doi:10.48550/arXiv.2212.05153.
  * Evans et al. (2024) Evans, T., Parthasarathy, N., Merzic, H., and Henaff, O. J.  Data curation via joint example selection further accelerates multimodal learning.  _arXiv preprint arXiv:2406.17711 [cs]_ , June 2024.  doi:10.48550/arXiv.2406.17711.
  * Frontier Model Forum (2024) Frontier Model Forum.  Issue brief: Measuring training compute, May 2024.  URL https://www.frontiermodelforum.org/updates/issue-brief-measuring-training-compute/.
  * Ganguli et al. (2022) Ganguli, D., Hernandez, D., Lovitt, L., Askell, A., Bai, Y., Chen, A., Conerly, T., Dassarma, N., Drain, D., Elhage, N., El Showk, S., Fort, S., Hatfield-Dodds, Z., Henighan, T., Johnston, S., Jones, A., Joseph, N., Kernian, J., Kravec, S., Mann, B., Nanda, N., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Kaplan, J., McCandlish, S., Olah, C., Amodei, D., and Clark, J.  Predictability and surprise in large generative models.  In _Proceedings of the 2022 ACM conference on fairness, accountability, and transparency (FAccT ’22)_ , pp. 1747–1764, New York, NY, USA, June 2022. Association for Computing Machinery.  ISBN 978-1-4503-9352-2.  doi:10.1145/3531146.3533229.
  * Goodfellow et al. (2016) Goodfellow, I., Bengio, Y., and Courville, A.  _Deep learning_.  MIT Press, 2016.
  * Gunasekar et al. (2023) Gunasekar, S., Zhang, Y., Aneja, J., Mendes, C. C. T., Del Giorno, A., Gopi, S., Javaheripi, M., Kauffmann, P., de Rosa, G., Saarikivi, O., Salim, A., Shah, S., Behl, H. S., Wang, X., Bubeck, S., Eldan, R., Kalai, A. T., Lee, Y. T., and Li, Y.  Textbooks Are All You Need.  _arXiv preprint arXiv:2306.11644 [cs]_ , October 2023.  doi:10.48550/arXiv.2306.11644.
  * Hayes et al. (2024) Hayes, T., Rao, R., Akin, H., Sofroniew, N. J., Oktay, D., Lin, Z., Verkuil, R., Tran, V. Q., Deaton, J., Wiggert, M., Badkundri, R., Shafkat, I., Gong, J., Derry, A., Molina, R. S., Thomas, N., Khan, Y., Mishra, C., Kim, C., Bartie, L. J., Nemeth, M., Hsu, P. D., Sercu, T., Candido, S., and Rives, A.  Simulating 500 million years of evolution with a language model, July 2024.
  * Heim (2023) Heim, L.  FLOP for Quantity, FLOP/s for Performance, April 2023.  URL https://blog.heim.xyz/flop-for-quantity-flop-s-for-performance/.
  * Heim et al. (2024) Heim, L., Fist, T., Egan, J., Huang, S., Zekany, S., Trager, R., Osborne, M., and Zilberman, N.  Governing through the cloud: The intermediary role of compute providers in AI regulation.  Technical report, Oxford Martin AI Governance Initiative, March 2024.  URL https://cdn.governance.ai/Governing-Through-the-Cloud_The-Intermediary-Role-of-Compute-Providers-in-AI-Regulation.pdf.
  * Hendrycks et al. (2021) Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J.  Measuring Massive Multitask Language Understanding.  _arXiv preprint arXiv:2009.03300 [cs]_ , January 2021.  doi:10.48550/arXiv.2009.03300.
  * Hendrycks et al. (2023) Hendrycks, D., Mazeika, M., and Woodside, T.  An Overview of Catastrophic AI Risks.  _arXiv preprint arXiv:2306.12001 [cs]_ , October 2023.  doi:10.48550/arXiv.2306.12001.
  * Hernandez & Brown (2020) Hernandez, D. and Brown, T. B.  Measuring the Algorithmic Efficiency of Neural Networks.  _arXiv preprint arXiv:2005.04305 [cs, stat]_ , May 2020.  doi:10.48550/arXiv.2005.04305.
  * Hernandez et al. (2021) Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S.  Scaling Laws for Transfer.  _arXiv preprint arXiv:2102.01293 [cs]_ , February 2021.  doi:10.48550/arXiv.2102.01293.
  * Hesslow et al. (2022) Hesslow, D., Zanichelli, N., Notin, P., Poli, I., and Marks, D.  RITA: a Study on Scaling Up Generative Protein Sequence Models.  _arXiv preprint arXiv:2205.05789 [cs, q-bio]_ , July 2022.  doi:10.48550/arXiv.2205.05789.
  * Ho et al. (2024) Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N., and Sevilla, J.  Algorithmic progress in language models.  _arXiv preprint arXiv:2403.05812 [cs]_ , March 2024.  doi:10.48550/arXiv.2403.05812.
  * Hobbhahn et al. (2023) Hobbhahn, M., Heim, L., and Aydos, G.  Trends in machine learning hardware.  Technical report, Epoch AI, November 2023.  URL https://epochai.org/blog/trends-in-machine-learning-hardware.
  * Hoffmann et al. (2022) Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., Driessche, G. v. d., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.  Training Compute-Optimal Large Language Models.  _arXiv preprint arXiv:2203.15556 [cs]_ , March 2022.  doi:10.48550/arXiv.2203.15556.
  * Hooker (2024) Hooker, S.  On the Limitations of Compute Thresholds as a Governance Strategy.  _arXiv preprint arXiv:2407.05694 [cs.AI]_ , July 2024.  URL https://arxiv.org/abs/2407.05694v1.
  * Jones (2023) Jones, E.  What is a foundation model?  Technical report, Ada Lovelace Institute, July 2023.  URL https://www.adalovelaceinstitute.org/resource/foundation-models-explainer/.
  * Jones et al. (2024) Jones, E., Hardalupas, M., and Agnew, W.  Under the radar?  Technical report, Ada Lovelace Institute, July 2024.  URL https://www.adalovelaceinstitute.org/report/under-the-radar/.
  * Jumper et al. (2021) Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A. J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A. W., Kavukcuoglu, K., Kohli, P., and Hassabis, D.  Highly accurate protein structure prediction with AlphaFold.  _Nature_ , 596(7873):583–589, August 2021.  ISSN 1476-4687.  doi:10.1038/s41586-021-03819-2.
  * Järviniemi & Hubinger (2024) Järviniemi, O. and Hubinger, E.  Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant.  _arXiv preprint arXiv:2405.01576 [cs]_ , April 2024.  doi:10.48550/arXiv.2405.01576.
  * Kaplan et al. (2020) Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D.  Scaling Laws for Neural Language Models.  _arXiv preprint arXiv:2001.08361 [cs, stat]_ , January 2020.  doi:10.48550/arXiv.2001.08361.
  * Kapoor et al. (2024) Kapoor, S., Bommasani, R., Klyman, K., Longpre, S., Ramaswami, A., Cihon, P., Hopkins, A., Bankston, K., Biderman, S., Bogen, M., Chowdhury, R., Engler, A., Henderson, P., Jernite, Y., Lazar, S., Maffulli, S., Nelson, A., Pineau, J., Skowron, A., Song, D., Storchan, V., Zhang, D., Ho, D. E., Liang, P., and Narayanan, A.  On the Societal Impact of Open Foundation Models.  _arXiv preprint arXiv:2403.07918 [cs.CY]_ , February 2024.  URL https://arxiv.org/abs/2403.07918v1.
  * Koessler et al. (2024) Koessler, L., Schuett, J., and Anderljung, M.  Risk thresholds for frontier AI.  _arXiv preprint arXiv:2406.14713 [cs.CY]_ , June 2024.  URL https://arxiv.org/abs/2406.14713v1.
  * Koralus & Wang-Maścianica (2023) Koralus, P. and Wang-Maścianica, V.  Humans in Humans Out: On GPT Converging Toward Common Sense in both Success and Failure.  _arXiv preprint arXiv:2303.17276 [cs]_ , March 2023.  doi:10.48550/arXiv.2303.17276.
  * Leech et al. (2024) Leech, G., Vazquez, J. J., Yagudin, M., Kupper, N., and Aitchison, L.  Questionable practices in machine learning.  _arXiv preprint arXiv:2407.12220 [cs]_ , July 2024.  doi:10.48550/arXiv.2407.12220.
  * Li et al. (2024) Li, L. H., Hessel, J., Yu, Y., Ren, X., Chang, K.-W., and Choi, Y.  Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step.  _arXiv preprint arXiv:2306.14050 [cs]_ , April 2024.  doi:10.48550/arXiv.2306.14050.
  * Linghan et al. (2024) Linghan, Z., Jianjun, Y., Ying, C., Jingwu, Z., Xuzhi, H., Zhifeng, Z., and Xiaoben, X.  Artificial Intelligence Law of the People’s Republic of China (Draft for Suggestions from Scholars).  Technical report, Center for Security and Emerging Technologies, March 2024.  URL https://cset.georgetown.edu/publication/china-ai-law-draft/.
  * Lohn & Musser (2022) Lohn, A. and Musser, M.  AI and Compute: How Much Longer Can Computing Power Drive Artificial Intelligence Progress?  Technical report, Center for Security and Emerging Technology, January 2022.  URL https://cset.georgetown.edu/publication/ai-and-compute/.
  * Lohn & Jackson (2022) Lohn, A. J. and Jackson, K. A.  Will AI Make Cyber Swords or Shields: A few mathematical models of technological progress.  _arXiv preprint arXiv:2207.13825 [cs]_ , July 2022.  doi:10.48550/arXiv.2207.13825.
  * Maug et al. (2024) Maug, N., O’Gara, A., and Besiroglu, T.  Biological Sequence Models in the Context of the AI Directives.  Technical report, Epoch AI, January 2024.  URL https://epochai.org/blog/biological-sequence-models-in-the-context-of-the-ai-directives.
  * McKenzie et al. (2024) McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Kirtland, A., Ross, A., Liu, A., Gritsevskiy, A., Wurgaft, D., Kauffman, D., Recchia, G., Liu, J., Cavanagh, J., Weiss, M., Huang, S., Droid, T. F., Tseng, T., Korbak, T., Shen, X., Zhang, Y., Zhou, Z., Kim, N., Bowman, S. R., and Perez, E.  Inverse Scaling: When Bigger Isn’t Better.  _arXiv preprint arXiv:2306.09479 [cs]_ , May 2024.  doi:10.48550/arXiv.2306.09479.
  * Mitchell et al. (2023) Mitchell, M., Luccioni, A. S., Lambert, N., Gerchick, M., McMillan-Major, A., Ozoani, E., Rajani, N., Thrush, T., Jernite, Y., and Kiela, D.  Measuring Data.  _arXiv preprint arXiv:2212.05129 [cs]_ , February 2023.  doi:10.48550/arXiv.2212.05129.
  * Mizrahi et al. (2024) Mizrahi, M., Kaplan, G., Malkin, D., Dror, R., Shahaf, D., and Stanovsky, G.  State of What Art? A Call for Multi-Prompt LLM Evaluation.  _arXiv preprint arXiv:2401.00595 [cs]_ , May 2024.  doi:10.48550/arXiv.2401.00595.
  * OpenAI (2023) OpenAI.  Preparedness framework (beta).  Technical report, OpenAI, December 2023.  URL https://cdn.openai.com/openai-preparedness-framework-beta.pdf.
  * OpenAI et al. (2023) OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapiro, G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L., Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C., Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B., Cho, C., Chu, C., Chung, H. W., Cummings, D., Currier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N., Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning, S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P., Forte, J., Fulford, I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Lopes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A., Hickey, W., Hoeschele, P., Houghton, B., Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S., Jonn, B., Jun, H., Kaftan, T., Kaiser, L., Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Kondraciuk, L., Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D., Li, C. M., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mehta, A., Menick, J., Metz, L., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L., O’Keefe, C., Pachocki, J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., Peres, F. d. A. B., Petrov, M., Pinto, H. P. d. O., Michael, Pokorny, Pokrass, M., Pong, V. H., Powell, T., Power, A., Power, B., Proehl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Selsam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song, Y., Staudacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, E., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Wang, J. J., Wang, A., Wang, B., Ward, J., Wei, J., Weinmann, C. J., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S., Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao, S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B.  GPT-4 Technical Report.  _arXiv preprint arXiv:2303.08774 [cs.CL]_ , March 2023.  URL https://arxiv.org/abs/2303.08774v6.
  * Parrish et al. (2022) Parrish, A., Chen, A., Nangia, N., Padmakumar, V., Phang, J., Thompson, J., Htut, P. M., and Bowman, S. R.  BBQ: A Hand-Built Bias Benchmark for Question Answering.  _arXiv preprint arXiv:2110.08193 [cs]_ , March 2022.  doi:10.48550/arXiv.2110.08193.
  * Perez et al. (2022) Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., Jones, A., Chen, A., Mann, B., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Khundadze, G., Kernion, J., Landis, J., Kerr, J., Mueller, J., Hyun, J., Landau, J., Ndousse, K., Goldberg, L., Lovitt, L., Lucas, M., Sellitto, M., Zhang, M., Kingsland, N., Elhage, N., Joseph, N., Mercado, N., DasSarma, N., Rausch, O., Larson, R., McCandlish, S., Johnston, S., Kravec, S., Showk, S. E., Lanham, T., Telleen-Lawton, T., Brown, T., Henighan, T., Hume, T., Bai, Y., Hatfield-Dodds, Z., Clark, J., Bowman, S. R., Askell, A., Grosse, R., Hernandez, D., Ganguli, D., Hubinger, E., Schiefer, N., and Kaplan, J.  Discovering Language Model Behaviors with Model-Written Evaluations.  _arXiv preprint arXiv:2212.09251 [cs]_ , December 2022.  doi:10.48550/arXiv.2212.09251.
  * Phuong et al. (2024) Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., Lindner, D., Rahtz, M., Assael, Y., Hodkinson, S., Howard, H., Lieberum, T., Kumar, R., Raad, M. A., Webson, A., Ho, L., Lin, S., Farquhar, S., Hutter, M., Deletang, G., Ruoss, A., El-Sayed, S., Brown, S., Dragan, A., Shah, R., Dafoe, A., and Shevlane, T.  Evaluating Frontier Models for Dangerous Capabilities.  _arXiv preprint arXiv:2403.13793 [cs]_ , April 2024.  doi:10.48550/arXiv.2403.13793.
  * Pilz & Heim (2023) Pilz, K. and Heim, L.  Compute at Scale: A Broad Investigation into the Data Center Industry.  _arXiv preprint arXiv:2311.02651 [cs]_ , November 2023.  doi:10.48550/arXiv.2311.02651.
  * Pilz et al. (2023) Pilz, K., Heim, L., and Brown, N.  Increased Compute Efficiency and the Diffusion of AI Capabilities.  _arXiv preprint arXiv:2311.15377 [cs.CY]_ , November 2023.  URL https://arxiv.org/abs/2311.15377v2.
  * Pistillo et al. (forthcoming) Pistillo, M., Van Arsdale, S., Heim, L., and Winter, C.  The Role of Compute Thresholds for AI Governance.  _George Washington Journal of Law & Technology_, forthcoming.
  * Rae et al. (2022) Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., Driessche, G. v. d., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang, S., Uesato, J., Mellor, J., Higgins, I., Creswell, A., McAleese, N., Wu, A., Elsen, E., Jayakumar, S., Buchatskaya, E., Budden, D., Sutherland, E., Simonyan, K., Paganini, M., Sifre, L., Martens, L., Li, X. L., Kuncoro, A., Nematzadeh, A., Gribovskaya, E., Donato, D., Lazaridou, A., Mensch, A., Lespiau, J.-B., Tsimpoukelli, M., Grigorev, N., Fritz, D., Sottiaux, T., Pajarskas, M., Pohlen, T., Gong, Z., Toyama, D., d’Autume, C. d. M., Li, Y., Terzi, T., Mikulik, V., Babuschkin, I., Clark, A., Casas, D. d. L., Guy, A., Jones, C., Bradbury, J., Johnson, M., Hechtman, B., Weidinger, L., Gabriel, I., Isaac, W., Lockhart, E., Osindero, S., Rimell, L., Dyer, C., Vinyals, O., Ayoub, K., Stanway, J., Bennett, L., Hassabis, D., Kavukcuoglu, K., and Irving, G.  Scaling Language Models: Methods, Analysis & Insights from Training Gopher.  _arXiv preprint_ , January 2022.  doi:10.48550/arXiv.2112.11446.
  * Ramesh et al. (2023) Ramesh, R., Khona, M., Dick, R. P., Tanaka, H., and Lubana, E. S.  How capable can a transformer become? A study on synthetic, interpretable tasks.  In _R0-FoMo:robustness of few-shot and zero-shot learning in large foundation models_ , November 2023.  URL https://openreview.net/forum?id=KIhFggzePM.