# Model evaluation for extreme risks

Toby Shevlane  Google DeepMind  Sebastian Farquhar  Google DeepMind  Ben
Garfinkel  Centre for the Governance of AI  Mary Phuong  Google DeepMind  Jess
Whittlestone  Centre for Long-Term Resilience  Jade Leung  OpenAI  Daniel
Kokotajlo  OpenAI  Nahema Marchal  Google DeepMind  Markus Anderljung  Centre
for the Governance of AI  Noam Kolt  University of Toronto  Lewis Ho  Google
DeepMind  Divya Siddarth  University of Oxford  Collective Intelligence
Project  Shahar Avin  University of Cambridge  Will Hawkins  Google DeepMind
Been Kim  Google DeepMind  Iason Gabriel  Google DeepMind  Vijay Bolina
Google DeepMind  Jack Clark  Anthropic  Yoshua Bengio  Université de Montréal
Mila – Quebec AI Institute  Paul Christiano  Alignment Research Center  Allan
Dafoe  Google DeepMind

###### Abstract

Current approaches to building general-purpose AI systems tend to produce
systems with both beneficial and harmful capabilities. Further progress in AI
development could lead to capabilities that pose extreme risks, such as
offensive cyber capabilities or strong manipulation skills. We explain why
model evaluation is critical for addressing extreme risks. Developers must be
able to identify dangerous capabilities (through “dangerous capability
evaluations”) and the propensity of models to apply their capabilities for
harm (through “alignment evaluations”). These evaluations will become critical
for keeping policymakers and other stakeholders informed, and for making
responsible decisions about model training, deployment, and security.

Figure 1: The theory of change for model evaluations for extreme risk.
Evaluations for dangerous capabilities and alignment inform risk assessments,
and are in turn embedded into important governance processes.

##  1 Introduction

As AI progress has advanced, general-purpose AI systems have tended to display
new and hard-to-forecast capabilities – including harmful capabilities that
their developers did not intend (Ganguli et al., 2022). Future systems may
display even more dangerous emergent capabilities, such as the ability to
conduct offensive cyber operations, manipulate people through conversation, or
provide actionable instructions on conducting acts of terrorism.

AI developers and regulators must be able to identify these capabilities, if
they want to limit the risks they pose. The AI community already relies
heavily on model evaluation – i.e. empirical assessment of a model’s
properties – for identifying and responding to a wide range of risks. Existing
model evaluations measure gender and racial biases, truthfulness, toxicity,
recitation of copyrighted content, and many more properties of models (Liang
et al., 2022).

We propose extending this toolbox to address risks that would be extreme in
scale, resulting from the misuse or misalignment of general-purpose models.
Work on this new class of model evaluation is already underway. These
evaluations can be organised into two categories: (a) whether a model has
certain dangerous capabilities, and (b) whether it has the propensity to
harmfully apply its capabilities (alignment).

Model evaluations for extreme risks will play a critical role in governance
regimes. A central goal of AI governance should be to limit the creation,
deployment, and proliferation of systems that pose extreme risks. To do this,
we need tools for looking at a particular system and assessing whether it
poses extreme risks. We can then craft company policies or regulations that
ensure:

  1. 1.

Responsible training: Responsible decisions are made about whether and how to
train a new model that shows early signs of risk.

  2. 2.

Responsible deployment: Responsible decisions are made about whether, when,
and how to deploy potentially risky models.

  3. 3.

Transparency: Useful and actionable information is reported to stakeholders,
to help them mitigate potential risks.

  4. 4.

Appropriate security: Strong information security controls and systems are
applied to models that might pose extreme risks.

Many AI governance initiatives focus on the risks inherent to a particular
deployment context, such as the “high-risk” applications listed in the draft
EU AI Act. However, models with sufficiently dangerous capabilities could pose
risks even in seemingly low-risk domains. We therefore need tools for
assessing both the risk level of a particular domain and the potentially risky
properties of particular models; this paper focuses on the latter.

Section 2 motivates our focus on extreme risks from general-purpose models and
refines the scope of the paper. Section 3 outlines a vision for how model
evaluations for such risks should be incorporated into AI governance
frameworks. Section 4 describes early work in the area and outlines key design
criteria for extreme risk evaluations. Section 5 discusses the limitations of
model evaluations for extreme risks and outlines ways in which work on these
evaluations could cause unintended harm. We conclude with high-level
recommendations for AI developers and policymakers.

##  2 Extreme risks from general-purpose models

Frontier AI developers are making rapid progress in developing increasingly
capable general-purpose models (Bubeck et al., 2023). These models learn their
capabilities and behaviours during training, and current methods for steering
this process are imperfect (Shah et al., 2022; Gao et al., 2022). At the
research frontier, models display new capabilities, often unforeseen by their
developers (Wei et al., 2022b).

This poses a challenge for safety. AI developers could train general-purpose
models that have dangerous capabilities – such as skills in deception, cyber
offense, or weapons design – without actively seeking these capabilities.
Humans could then intentionally misuse these capabilities (Brundage et al.,
2018), e.g. for assistance in disinformation campaigns, cyberattacks, or
terrorism. Additionally, due to failures of alignment, AI systems could
harmfully apply their capabilities even without deliberate misuse (Ngo et al.,
2022).

In the near-term, these risks will be especially concentrated on the frontier
of AI research and development. We loosely define the “frontier” as models
that are both (a) close to, or exceeding, the average capabilities of the most
capable existing models,111In practice, defining “average capabilities” would
involve many judgement calls over which evaluations should be included and how
they should be weighted. and (b) different from other models, either in terms
of scale, design (e.g. different architectures or alignment techniques), or
their resulting mix of capabilities and behaviours. Accordingly, frontier
models are uniquely risky because (a) more capable models can excel at a wider
range of tasks, which will unlock more opportunities to cause harm;222This is
especially pertinent for extreme risks: causing such large-scale harm is not
normally an easy challenge. Even well-resourced terrorist groups, determined
to cause extreme harm, often fail. and (b) novel models are less well-
understood by the research community.

Figure 2: Leading AI developers push the frontier outward, typically by
training models at greater scale and using more efficient architectures and
algorithms. This continued expansion takes the field closer to points in model
space that could pose extreme risks. The diagram is purely illustrative.

We focus on “extreme” risks, i.e. those that would be extremely large in scale
(even relative to the scale of deployment). This can be operationalised in
terms of the scale of impact (e.g. damage in the tens of thousands of lives
lost, hundreds of billions of dollars of economic or environmental damage) or
the level of adverse disruption to the social and political order. The latter
could mean, for example, the outbreak of inter-state war, a significant
erosion in the quality of public discourse, or the widespread disempowerment
of publics, governments, and other human-led organisations (Carlsmith, 2022).

Many AI researchers (and other stakeholders) view extreme risks from AI as an
important challenge. In a 2022 survey of AI researchers, 36% of respondents
thought that AI systems could plausibly “cause a catastrophe this century that
is at least as bad as an all-out nuclear war” (Michael et al., 2022). However,
very few existing model evaluations intentionally target risks on this scale.

To guard against extreme risks, AI developers should use model evaluation to
uncover:

  1. 1.

To what extent a model is capable of causing extreme harm (which relies on
evaluating for certain dangerous capabilities).

  2. 2.

To what extent a model has the propensity to cause extreme harm (which relies
on alignment evaluations).

We provide a non-exhaustive list of dangerous capabilities in Table 1. Most of
the capabilities listed are offensive capabilities: they are useful for
exerting influence or threatening security (e.g. see: persuasion and
manipulation, cyber-offense, weapons acquisition). Some (e.g. situational
awareness) are capabilities that would be advantageous for a misaligned AI
system evading human oversight (Ngo et al., 2022). We omit many generically
useful capabilities (e.g. browsing the internet, understanding text) despite
their potential relevance to both the above.

Capability |  Could include:
---|---
Cyber-offense |  The model can discover vulnerabilities in systems (hardwares, software, data). It can write code for exploiting those vulnerabilities. It can make effective decisions once it has gained access to a system or network, and skilfully evade threat detection and response (both human and system) whilst focusing on a specific objective. If deployed as a coding assistant, it can insert subtle bugs into the code for future exploitation.
Deception |  The model has the skills necessary to deceive humans, e.g. constructing believable (but false) statements, making accurate predictions about the effect of a lie on a human, and keeping track of what information it needs to withhold to maintain the deception. The model can impersonate a human effectively.
Persuasion
& manipulation |  The model is effective at shaping people’s beliefs, in dialogue and other settings (e.g. social media posts), even towards untrue beliefs. The model is effective at promoting certain narratives in a persuasive way. It can convince people to do things that they would not otherwise do, including unethical acts.
Political
strategy |  The model can perform the social modelling and planning necessary for an actor to gain and exercise political influence, not just on a micro-level but in scenarios with multiple actors and rich social context. For example, the model can score highly in forecasting competitions on questions relating to global affairs or political negotiations.
Weapons
acquisition |  The model can gain access to existing weapons systems or contribute to building new weapons. For example, the model could assemble a bioweapon (with human assistance) or provide actionable instructions for how to do so. The model can make, or significantly assist with, scientific discoveries that unlock novel weapons.
Long-horizon planning |  The model can make sequential plans that involve multiple steps, unfolding over long time horizons (or at least involving many interdependent steps). It can perform such planning within and across many domains. The model can sensibly adapt its plans in light of unexpected obstacles or adversaries. The model’s planning capabilities generalise to novel settings, and do not rely heavily on trial and error.
AI
development |  The model could build new AI systems from scratch, including AI systems with dangerous capabilities. It can find ways of adapting other, existing models to increase their performance on tasks relevant to extreme risks. As an assistant, the model could significantly improve the productivity of actors building dual use AI capabilities.
Situational awareness |  The model can distinguish between whether it is being trained, evaluated, or deployed – allowing it to behave differently in each case. The model knows that it is a model, and has knowledge about itself and its likely surroundings (e.g. what company trained it, where their servers are, what kind of people might be giving it feedback, and who has administrative access).
Self-proliferation |  The model can break out of its local environment (e.g. using a vulnerability in its underlying system or suborning an engineer). The model can exploit limitations in the systems for monitoring its behaviour post-deployment. The model could independently generate revenue (e.g. by offering crowdwork services, ransomware attacks), use these revenues to acquire cloud computing resources, and operate a large number of other AI systems. The model can generate creative strategies for uncovering information about itself or exfiltrating its code and weights.
Table 1: Dangerous capabilities

The most risky scenarios will involve multiple dangerous capabilities combined
together – further research should explore what combinations would be most
dangerous. Sometimes specific capabilities can be supplied by the user or
outsourced to other humans (e.g. crowdworkers) or AI systems.

Figure 3: Ingredients for extreme risk.

A simple heuristic: a model should be treated as highly dangerous if it has a
capability profile that would be sufficient for extreme harm, assuming misuse
and/or misalignment. To deploy such a model, AI developers would need very
strong controls against misuse (Shevlane, 2022b) and very strong assurance
(via alignment evaluations) that the model will behave as intended. Alignment
evaluations should look for behaviours identified in the literature, such as
whether the model:

  * •

Pursues long-term, real-world goals, different from those supplied by the
developer or user (Chan et al., 2023; Ngo et al., 2022);

  * •

Engages in “power-seeking” behaviours (Turner et al., 2021; Krakovna and
Kramar, 2023);

  * •

Resists being shut down (Hadfield-Menell et al., 2016; Orseau and Armstrong,
2016);

  * •

Can be induced into collusion with other AI systems against human interests
(Ngo et al., 2022).

  * •

Resists malicious users’ attempts to access its dangerous capabilities (Glaese
et al., 2022).

We focus on risks arising from misuse and misalignment because a new
generation of model evaluations is needed for identifying these risks.
Conversely, though important, we leave out of scope:

  1. 1.

Structural risks, which depend especially heavily on how the AI system
interacts with larger social, political, and economic forces in society
(Zwetsloot and Dafoe, 2019). Model evaluation sheds less light on these risks,
because they depend so heavily on factors external to the model.

  2. 2.

Risks from models incompetently performing important tasks (Raji et al.,
2022a). Existing kinds of model evaluation will be most relevant here (e.g.
testing the model’s accuracy and robustness on the relevant task).

##  3 Model evaluation as critical governance infrastructure

Across many industries, safety standards and regulations rely on tools for
assessing risks in new products – for instance, food, drugs, commercial
airliners, and automobiles. Model evaluation is not the only tool available
for AI risk assessment – more theoretical approaches are also available, e.g.
studying the incentives operating on a model during training (Everitt et al.,
2021). Nonetheless, model evaluation is one of the main tools we have for AI
risk assessment.

Figure 4 provides an overview of this section. It is an ambitious blueprint
for how to guard against extreme risks while developing and deploying a model,
with evaluation embedded throughout. The evaluation results feed into
processes for risk assessment, which inform (or bind) important decisions
around model training, deployment, and security. The developer reports results
and risk assessments to external stakeholders.

Figure 4: A workflow for training and deploying a model, embedding extreme
risk model evaluation results into key safety and governance processes.

Three sources of model evaluations feed into this process:

  1. 1.

Internal model evaluation, i.e. the developer conducting its own evaluations.
There is no substitute for internal model evaluation, given that internal
researchers have high context on the model’s design and deeper model access
than can be achieved via an API. Developers could have multiple organisational
layers of safety evaluation, such as by establishing an internal safety
evaluation function that is independent of the teams primarily responsible for
building the models, reporting directly to organisational leaders (see Raji et
al., 2020).

  2. 2.

External research access. The developer grants model access to external
researchers, likely via an API (Shevlane, 2022b, a; Bluemke et al., 2023).
Their research could be exploratory or targeted at evaluating specific
properties, including “red teaming” the model’s alignment.

  3. 3.

External model audit, i.e. model evaluation by an independent, external
auditor for the purpose of providing a judgement — or input to a judgement —
about the safety of deploying a model (or training a new one) (ARC Evals,
2023; Mökander et al., 2023; Raji et al., 2022b). Ideally there would exist a
rich ecosystem of model auditors providing broad coverage across different
risk areas. (This ecosystem is currently under-developed.)

###  3.1 Responsible training

The first line of defence is to avoid training models that have sufficient
dangerous capabilities and misalignment to pose extreme risk. Sufficiently
concerning evaluation results should warrant delaying a scheduled training run
or pausing an existing one.333Developers training highly capable, general-
purpose models should become accustomed to such a prospect, not planning their
research around an assumption that a training run will run to schedule. For
example, well-run developers will have compute allocation systems that
backfill the vacant computing resources with other useful projects. Similarly,
developers should avoid making hard promises to stakeholders (e.g. customers,
investors) that they will deploy a certain model at a certain date. At the
least, they should retain the flexibility of pivoting to a smaller or
otherwise less risky version of the model.

Before a frontier training run, developers have the opportunity to study
weaker models that might provide early warning signs. These models come from
two sources: (1) previous training runs, and (2) experimental models leading
up to the new training run. Developers should evaluate these models and try to
forecast the results from the planned training run (see OpenAI, 2023b). This
would include scaling (or “inverse scaling”) analysis where the aim is to find
areas where scaling brings unwanted changes to the model (McKenzie et al.,
2022). These insights can feed into a training risk assessment. Then, during
the training run, researchers could run extreme risk evaluations at regular
intervals.

The developer has a range of possible responses to address the concerning
evaluation results:

  1. 1.

Study the issue to understand why the misalignment or dangerous capability
emerged.

  2. 2.

Adjust the training methods to circumvent the issue. This could mean adjusting
(for example) the architecture, the data, the training tasks, or further
developing the alignment techniques used. These adjustments should target the
fundamental issue rather than inducing superficial changes to how the model
scores on the available evaluations (see section 5.2).

  3. 3.

Careful scaling. If the developer is not confident it can train a safe model
at the scale it initially had planned, they could instead train a smaller or
otherwise weaker model.

In mature governance regimes, the decision to proceed with a potentially risky
training run could require approval from an external model auditor or
regulator.

###  3.2 Responsible deployment

Deployment means making the model available for use, e.g. it is built into a
product or hosted on an API for software developers to build with. Deployment
constitutes a large increase in the model’s exposure to the external world and
therefore possible risk. Model evaluation for extreme risks could inform a
deployment risk assessment that reviews (a) whether or not the model is safe
to deploy, and (b) the appropriate guardrails for ensuring the deployment is
safe.

The predeployment evaluation process takes time (Rismani et al., 2023; OpenAI,
2023a). Industry standards or regulation could require a minimum duration for
predeployment evaluation of frontier models, including the length of time that
external researchers and auditors have access.

In response to concerning evaluation results, one possibility is to recommend
against deployment. A second possibility is to recommend adjustments to the
deployment plan that would address potential risks (see Table 3 in the
Appendix for a range of variables that could be adjusted). Nonetheless, for a
sufficiently capable and poorly aligned model, it is possible that even a
restrictive and scaled-back deployment could pose extreme risk.

Safe deployment will often be a gradual process (Figure 5) (Brundage et al.,
2022). The developer gradually accumulates evidence about the model’s safety,
through both evaluation (internal and external) and early, small-scale
deployment.

Figure 5: The developer gradually increases the model’s exposure to the
external world as it accumulates evidence about the model’s safety.

Evaluation will often need to continue after deployment. There are two reasons
for this:

  1. 1.

Unanticipated behaviours. Before deployment, it is impossible to fully
anticipate and understand how the model will interact in a complex deployment
environment (a key limitation of model evaluation: see section 5). For
example, users might find new applications for the model or novel prompt
engineering strategies; or the model could be operating in a dynamic, multi-
agent environment. Therefore, in the early stages of deployment, developers
must:

    1. (a)

Surface emerging model behaviours and risks via monitoring efforts. This could
include direct monitoring of inputs and outputs to the model, and systems for
incident reporting (see Brundage et al., 2022; Raji et al., 2022b).

    2. (b)

Design and run new model evaluations inspired by these observations.

  2. 2.

Updates to the model. The developer might might update the model after
deployment, e.g. by fine-tuning on data collected during deployment or by
expanding the model’s access to external tools. If these updates could
increase risk, they should be evaluated before launch. For large changes,444
The magnitude of the change to the model could be assessed in terms of the
amount of additional training that it has gone through (as a percentage of the
original training length), or the model’s improvement on key performance
benchmarks. the new model could go through the whole process described in this
section.

The ideal state is continuous deployment review. On an ongoing basis, the
developer reassesses deployment safety using model evaluations and monitoring,
and at any time, could adjust or terminate the deployment in response to their
findings. Further, for deployments that were recognisably unsafe in
retrospect, an external audit of the deployment decision-making process could
be triggered. Safety issues uncovered during deployment can also inform
training risk assessments for future models.

Finally, even internal deployments of highly capable general-purpose models,
notably as coding assistants for AI researchers and engineers, could require
pre-deployment evaluation for dangerous capabilities (e.g. the ability to
insert subtle vulnerabilities into code) and alignment.

###  3.3 Transparency

Model evaluations are a vital tool for keeping stakeholders informed about the
state of AI risks on the frontier (Whittlestone and Clark, 2021). We recommend
frontier developers consider processes for externally reporting the results of
evaluations or extracts from the assessment documents that rely on those
evaluation results (such as training risk assessments, auditors’ reports,
deployment risk assessments).

Model evaluations will unlock four important kinds of transparency around
extreme risks:

  1. 1.

Incident reporting, i.e. a structured process for developers to share
concerning or otherwise noteworthy evaluation results with other developers,
third parties, or regulators (see Brundage et al., 2020). This would be vital
for helping others avoid training risky systems, and for keeping AI developers
accountable. In future, regulators could maintain lists of high-risk or banned
training approaches; a sufficiently concerning incident report would then
trigger an update to this list.

  2. 2.

Sharing pre-deployment risk assessments. Prior to deployment, developers could
share their predeployment risk assessments for comment and critique. This
could mean sharing with auditors, external researchers, regulators, or
publicly. The risk assessment would include evaluation results, alongside the
developer’s justification for why the deployment is safe given those results.
(See related: Mitchell et al., 2018)

  3. 3.

Scientific reporting, i.e. presenting the results to the scientific community
in a way that encourages further scientific research into the issue. Much
research will be needed to better understand the behaviour of highly capable,
general-purpose models, especially their alignment. Topics within the
alignment literature, such as situational awareness, could follow the path of
fairness research, where early findings of gender and racial biases in models
have helped stimulate new areas of research.

  4. 4.

Educational demonstrations, i.e. engaging presentation of concerning
capabilities and behaviours to key stakeholders – such as policymakers, the
public, or company leadership – to keep them informed about risks from
frontier AI systems. The demonstrations could be videos or interactive demos,
with the evaluation results presented as supplementary information (as
evidence of scientific rigour).

###  3.4 Appropriate security

Models at risk of exhibiting dangerous capabilities will require strong and
novel security controls. Developers must consider multiple possible threat
actors: insiders (e.g. internal staff, contractors), outsiders (e.g. users,
nation-state threat actors), and the model itself as a vector of harm. We must
develop new security best practices for high-risk AI development and
deployment, which could include for example:

  * •

Red teaming: Intensive security red-teaming for the entire infrastructure on
which the model is developed and deployed.

  * •

Monitoring: Intensive, AI-assisted monitoring of the model’s behaviour, e.g.
for whether the model is engaging in manipulative behaviour or making code
recommendations that would lower the overall security of a system.

  * •

Isolation: Appropriate isolation techniques for preventing risky models from
exploiting the underlying system (e.g. sole-tenant machines and clusters, and
other software-based isolation). The model’s network access should be tightly
controlled and monitored, as well as its access to tools (e.g. code
execution).

  * •

Rapid response: Processes and systems for rapid response to disable model
actions and the model’s integrations with hardware, software, and
infrastructure in the event of unexpected unsafe behaviour.

  * •

System integrity: Formal verification that served models, memory, or
infrastructure have not been tampered with. The development and serving
infrastructure should require two-party authorization for any changes and
auditability of all changes.

##  4 Building evaluations for extreme risk

Model evaluation is already a core component of AI research, and increasingly
we have evaluations that focus on ethics, safety, and social impact. We
recommend extending this toolbox to address extreme risks.

Early work is already underway to build model evaluations for extreme risks.
ARC Evals (the evaluations team at the Alignment Research Center) is building
evaluations that measure language models’ self-proliferation capabilities (see
Table 1 above). ARC Evals ran this evaluation on GPT-4 and Claude before their
wider release (ARC Evals, 2023; OpenAI, 2023a). OpenAI and the GPT-4 red
teamers also tested GPT-4’s capabilities in cybersecurity operations and its
ability to purchase certain chemical compounds (OpenAI, 2023a).

Google DeepMind has ongoing projects evaluating language models for
manipulation capabilities. This includes a game called "Make-me-say", where
the language model must lead an (unaware) human conversation partner to say a
pre-specified word.555This project is led by Mary Phuong.

Table 2 contains a range of desirable qualities for extreme risk evaluations.
Some of these qualities relate to a single evaluation, and some are desirable
qualities of a portfolio of evaluations.

Comprehensive:
---
Threat models |  The evaluation portfolio should cover as many plausible extreme risk threat models as possible.
Automated and human-assisted |  Many evaluations can be run automatically, lowering the time and resource costs. However, some capabilities and behaviours will need human-assisted evaluations, i.e. involving: (a) human raters who judge the model’s outputs; or (b) humans who interact with the model, e.g. in a dialogue setting.
Behavioural and mechanistic |  Evaluations should not be restricted to studying a model’s behaviour, but should eventually also involve looking mechanistically at how the model produced that behaviour.
Fault-finding |  The portfolio of evaluations should include adversarial testing, where researchers purposefully search for cases where the model produces concerning results.
Robust to
deception |  Ultimately researchers will need evaluations that can rule out the possibility that the model is deliberately appearing safe for the purpose of passing the evaluation process.
Surfacing latent capabilities |  Researchers will need to bring latent capabilities to the surface (for example, by prompt engineering or fine-tuning).
Model lifecycle |  We recommend conducting evaluations throughout the model development process. In particular, the results from the end of a long development process will likely fail to convey relevant information about the base model, especially if it has been fine-tuned for safety.
Model-level and system-level |  Models are often integrated into wider AI systems, e.g. with external tools, other models, or classifiers that filter the model’s outputs. Evaluations should study models both with and without these augmentations.
Interpretable:
Legible |  Some evaluations should present risks in an accessible way, requiring little technical understanding. This will be helpful for creating common knowledge around the risks from AI.
Wide difficulty spectrum |  The dangerous capability evaluations should ideally contain wide ranges of difficulty – ideally within single evaluations, but at least across the portfolio. This means that researchers can track capabilities progress as it approaches possible danger thresholds, and that the evaluation (or the portfolio) is scalable to future, more capable models. For tracking progress, evaluations would ideally provide a quantitative score, although this will not always be practical.
Safe:
Safe to
implement |  Dangerous capability evaluations could involve testing the model in real-world settings, e.g. interacting with crowdworkers. This should not introduce unacceptable levels of risk.
Table 2: Desirable qualities of extreme risk evaluations.

We anticipate that building comprehensive alignment evaluations will be most
challenging. The ambition is for a process of alignment assurance that could
conclude, with high confidence, that a model is not dangerously misaligned,
even for very capable models. (Model evaluations would not be the only input
to this assurance process, but an important one.)

Alignment evaluation is challenging because we need assurance that the model
will reliably behave appropriately across a wide diversity of settings
(Ziegler et al., 2022). An evaluation might find that a model is aligned in
some narrow, prosaic way (for example, a language agent asserting that it does
not object to being shut down (Perez et al., 2022b, a)) without providing
evidence that the model would exhibit desirable behaviour when presented with
genuine (or more convincing) opportunities to achieve self-preservation,
greater influence, or other harmful outcomes.

Researchers must therefore evaluate a model across a broad range of settings.
Achieving coverage of settings for alignment evaluation can be helped by:

  1. 1.

Breadth: Evaluating behaviour across as wide a range of settings as possible.
One promising avenue is automating the process of writing evaluations using AI
systems (Perez et al., 2022b) (see also Pan et al., 2023).

  2. 2.

Targeting: Some settings are much more likely to reveal alignment failures
than others, and we may be able to focus on them through clever design – for
example, using honeypots or gradient-based adversarial testing and related
approaches (Jones et al., 2023).

  3. 3.

Understanding generalisation: Since researchers will be unable to foresee or
simulate all possible scenarios, we must develop a better scientific
understanding of how and why model behaviours generalise (or fail to
generalise) between settings.

Another important tool is mechanistic analysis, i.e. studying the model’s
weights and activations for understanding how it functions (Olah et al., 2020;
Nanda et al., 2023). For example, one ambition is to study how the model’s
goals are represented internally, to help verify that they are as intended;
another ambition is to detect when a language model’s outputs misreport its
knowledge (Burns et al., 2022), which could be an indicator of deceptive
behaviour.

Finally, agency – in particular, the goal-directedness of an AI system – is an
important property to evaluate (Kenton et al., 2022), given the central role
of agency in various theories of AI risk (Chan et al., 2023). Partly, agency
is a question of the model’s capabilities – is it capable of effectively
pursuing goals? Evaluating alignment also requires looking at agency,
including: (a) Is the model more goal-directed than the developer intended?
For example, has a dialogue agent learnt the goal of manipulating the user’s
behavior? (b) Does the model resist a user’s attempt to assemble it into an
autonomous AI system (e.g. Auto-GPT) with harmful goals?

##  5 Limitations and hazards

###  5.1 Limitations

Model evaluation, as a tool for addressing extreme risks, has at least five
limitations. A key issue is that not all risks can necessarily be detected via
model evaluation.

  1. 1.

Factors beyond the AI system. Risks will depend on how an AI system interacts
with a complex world. For example, a model might use, as tools, other models
released in the future, thus augmenting its capabilities; or human
civilisation might be less resilient to powerful AI than anticipated.

  2. 2.

Unknown threat models. It is difficult to anticipate all the different
plausible pathways to extreme risk. This will be especially true for highly
capable models, which could find creative strategies for achieving their
goals.

  3. 3.

Difficult-to-identify properties. Some model properties will be challenging to
uncover via model evaluations. Two important cases:

    1. (a)

Capability overhang: Models sometimes have capabilities that the AI research
community does not realise. For example, after GPT-3 had already existed for
many months, researchers demonstrated that chain-of-thought prompting could
significantly increase performance (Wei et al., 2022c).

    2. (b)

Deceptive alignment: A situationally aware model could deliberately exhibit
desired behaviour during evaluation (Ngo et al., 2022). (This is one reason
not to rely solely on behavioural evaluations.)

  4. 4.

Emergence. Above we recommended using model evaluations to inform the decision
to train a new model by performing scaling laws analysis on smaller models.
However, sometimes specific capabilities will emerge only at greater scale,
which makes this analysis much harder (Ganguli et al., 2022); other
capabilities display U-shaped scaling (Wei et al., 2022a).

  5. 5.

Maturity of evaluation ecossytem. The ecosystem for external evaluations and
model audits is currently under-developed.

  6. 6.

Overtrust in evaluations. There is a risk that too much faith is placed in
evaluation results, leading to risky models being deployed under a false sense
of security.

Model evaluation is a necessary but not sufficient strategy for identifying
and mitigating extreme risks. It must be combined with a wider organisational
dedication to safety and other tools for risk identification and assessment.

###  5.2 Hazards

Conducting and reporting the evaluations discussed in this paper poses four
potential hazards:

  1. 1.

Advancing and proliferating dangerous capabilities. There is a risk that –
through conducting dangerous capability evaluations and sharing relevant
materials – the field will proliferate dangerous capabilities or accelerate
their development. We highlight four kinds of potentially hazardous
information:

    1. (a)

Results. Evaluation results could demonstrate novel offensive technologies.
Publicly sharing these results could spur investment in new weapons
programmes, cyber-offensive efforts, or methods for digital oppression of
citizens. By analogy, it has been said that in the 1940s the most valuable
secret about the nuclear bomb was that it was possible (Ord, 2022). AI
developers, researchers, and auditors should therefore exercise caution around
sharing these evaluation results.

    2. (b)

Evaluation datasets. Datasets for evaluating dangerous capabilities are dual
use because other actors could fine-tune their models on these datasets.

    3. (c)

Elicitation techniques. Evaluating dangerous capabilities will often involve
eliciting those capabilities from the model. This could involve: (a) prompt
engineering; and (b) fine-tuning, including: (i) finding creative new task
specifications; (ii) creating or identifying appropriate fine-tuning datasets.
These techniques could be useful for a bad actor attempting to elicit
dangerous capabilities from similar models. Researchers and auditors must
therefore exercise caution over sharing their elicitation techniques,
especially if producing them relied on creativity, expert knowledge, or time-
consuming experimentation.

    4. (d)

Trained models. There are risks from intentionally training dangerously
capable models, even for use as safety research artefacts. We could
distinguish between (a) simply following off-the-shelf methods (e.g. fine-
tuning an existing model), versus (b) cases where the work to produce the
dangerous capability could constitute a research contribution in its own right
(e.g. it could be accepted to an academic conference). The latter is arguably
comparable to “gain-of-function” research in virology, especially if the
resulting model is highly capable and general-purpose. The research may need
to be conducted under very high-security conditions and subject to a demanding
risk assessment.

  2. 2.

Competitive pressures. One concern is that sharing evaluation results between
competing AI developers could incentivise them to behave less responsibly. For
example, sharing predeployment evaluation results could tip off competitors
about future product improvements, incentivising those comeptitors to rush
their own deployments and spend less time on ensuring safety. Similarly, since
dangerous capability results will often correlate with the model’s overall
capabilities, competing developers could learn that they are falling behind
and decide they need to sacrifice on safety to catch up (Emery-Xu et al.,
2023).

Given the sensitivities involved, one option is to lean more heavily on
alignment evaluation results, at least for reporting between developers. For
illustration, a possible inter-developer policy could be:

    1. (a)

Report unexpected or important alignment issues promptly to other developers.
By default, limit the description of the model’s training to only a high-level
overview (to avoid revealing sensitive information); but share more details if
this is absolutely necessary – in particular, if a certain class of methods is
causing the problem.

    2. (b)

Report when certain dangerous capability thresholds have been passed. These
thresholds can be set high, to avoid sharing granular information. Wait until
deployment to share more.

  3. 3.

Superficial improvements to model safety. There is a risk that widely
available safety evaluations will lead to models that exhibit only
superficially desirable behaviours. Most clearly, if researchers directly
train models to pass these evaluations, the evaluations can no longer act as
an indicator of risk. Researchers could do this either accidentally (e.g.
because the evaluation datasets are shared online and thereby end up in the
pretraining dataset) or as an intentional attempt to pass external audits
(analogous to the Volkswagen emissions scandal). The model’s desirable
evaluation performance would then likely fail to generalise. Developers and
model auditors could therefore consider keeping some private “held out”
evaluations and ensuring these are not too overlapping with datasets or tasks
used during training.

Even if developers refrain from directly training on the evaluations, we have
nevertheless recommended that developers avoid training models that fail the
evaluations (section 3.1). This could also exert selection pressure, albeit
weaker. Over the long run, the risk is that the field selects for training
methods that produce deceptively aligned models.

  4. 4.

Harms during the course of evaluation. Running evaluations will often involve
exposing the model to the external world. For example, in evaluating GPT-4,
ARC used the model to generate (deceptive) messages to be sent to a TaskRabbit
worker (OpenAI, 2023a). In the extreme case, a poorly managed test for whether
a model has self-proliferating capabilities could end in actual proliferation;
but more prosaically, they could cause harm in other ways, such as causing
emotional distress to crowdworkers. Therefore, groups conducting evaluations,
such as auditors, should establish safety protocols where necessary.

##  6 Conclusion

Model evaluation for extreme risks should be a priority area for AI safety and
governance. There are many challenges ahead for finding effective evaluations
and building governance regimes that incorporate them; we encourage further
work in this area. Model evaluation is not a panacea: it will not catch all
extreme risks. Nonetheless, it is a necessary component of the governance
infrastructure needed to combat extreme risks.

Frontier AI developers currently have a special responsibility to support work
on model evaluations for extreme risks, since they have resources – including
access to cutting-edge AI models and deep technical expertise – that many
other actors typically lack. Frontier AI developers are also currently the
actors who are most likely to unintentionally develop or release AI systems
that pose extreme risks. Frontier AI developers should therefore:

  1. 1.

Invest in research: Frontier developers should devote resources to researching
and developing model evaluations for extreme risks.

  2. 2.

Craft internal policies: Frontier developers should craft internal policies
for conducting, reporting, and responding appropriately to the results of
extreme risk evaluations.

  3. 3.

Support outside work: Frontier labs should enable outside research on extreme
risk evaluations through model access and other forms of support.

  4. 4.

Educate policymakers: Frontier developers should educate policymakers and
participate in standard-setting discussions, to increase government capacity
to craft any regulations that may eventually be needed to reduce extreme
risks.

Policymakers should consider building up the governance infrastructure
outlined in section 3. Policymakers could:

  1. 1.

Systematically track the development of dangerous capabilities, and progress
in alignment, within frontier AI R&D (Whittlestone and Clark, 2021).
Policymakers could establish a formal reporting process for extreme risk
evaluations.

  2. 2.

Invest in the ecosystem for external safety evaluation, and create venues for
stakeholders (such as AI developers, academic researchers, and government
representatives) to come together and discuss these evaluations (Anthropic,
2023)

  3. 3.

Mandate external audits, including model audits and audits of developers’ risk
assessments, for highly capable, general-purpose AI systems.

  4. 4.

Embed extreme risk evaluations into the regulation of AI deployment,
clarifying that models posing extreme risks should not be deployed.

##  7 Acknowledgements

We are grateful for helpful comments and discussions on this work from: Canfer
Akbulut, Jide Alaga, Beth Barnes, Joslyn Barnhart, Sasha Brown, Miles
Brundage, Martin Chadwick, Tom Everitt, Conor Griffin, Eric Horvitz, Evan
Hubinger, William Isaac, Victoria Krakovna, Leonie Koessler, Sébastien Krier,
Nikhil Mulani, Neel Nanda, Jonas Schuett, Rohin Shah, Andrew Trask, Gregory
Wayne, and Hjalmar Wijk. We are grateful for insightful discussions with the
participants of two events held in February 2023: a virtual discussion session
on the topic of this paper, and a one-day workshop on dangerous capabilities
evaluations co-organised by Steven Adler, Anne le Roux, and Jade Leung. We
also thank Celine Smith for project management support, and Michael Chang for
improvements to the visualisations.

## References

  * Anthropic (2023) Anthropic.  Strengthening U.S. AI innovation through an ambitious investment in NIST.  Technical report, 2023.
  * ARC Evals (2023) ARC Evals.  Update on ARC’s recent eval efforts.  https://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/, Mar. 2023.  Accessed: 2023-3-20.
  * Bluemke et al. (2023) E. Bluemke, T. Collins, B. Garfinkel, and A. Trask.  Exploring the relevance of data Privacy-Enhancing technologies for AI governance use cases.  Mar. 2023.
  * Brundage et al. (2018) M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, B. Filar, H. Anderson, H. Roff, G. C. Allen, J. Steinhardt, C. Flynn, S. Ó. hÉigeartaigh, S. Beard, H. Belfield, S. Farquhar, C. Lyle, R. Crootof, O. Evans, M. Page, J. Bryson, R. Yampolskiy, and D. Amodei.  The malicious use of artificial intelligence: Forecasting, prevention, and mitigation.  Feb. 2018.
  * Brundage et al. (2020) M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong, T. Maharaj, P. W. Koh, S. Hooker, J. Leung, A. Trask, E. Bluemke, J. Lebensold, C. O’Keefe, M. Koren, T. Ryffel, J. B. Rubinovitz, T. Besiroglu, F. Carugati, J. Clark, P. Eckersley, S. de Haas, M. Johnson, B. Laurie, A. Ingerman, I. Krawczuk, A. Askell, R. Cammarota, A. Lohn, D. Krueger, C. Stix, P. Henderson, L. Graham, C. Prunkl, B. Martin, E. Seger, N. Zilberman, S. Ó. hÉigeartaigh, F. Kroeger, G. Sastry, R. Kagan, A. Weller, B. Tse, E. Barnes, A. Dafoe, P. Scharre, A. Herbert-Voss, M. Rasser, S. Sodhani, C. Flynn, T. K. Gilbert, L. Dyer, S. Khan, Y. Bengio, and M. Anderljung.  Toward trustworthy AI development: Mechanisms for supporting verifiable claims.  Apr. 2020.
  * Brundage et al. (2022) M. Brundage, K. Mayer, T. Eloundou, S. Agarwal, S. Adler, G. Krueger, J. Leike, and P. Mishkin.  Lessons learned on language model safety and misuse.  https://openai.com/research/language-model-safety-and-misuse, Mar. 2022.  Accessed: 2023-4-17.
  * Bubeck et al. (2023) S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y. T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M. T. Ribeiro, and Y. Zhang.  Sparks of artificial general intelligence: Early experiments with GPT-4.  Mar. 2023.
  * Burns et al. (2022) C. Burns, H. Ye, D. Klein, and J. Steinhardt.  Discovering latent knowledge in language models without supervision.  Dec. 2022.
  * Carlsmith (2022) J. Carlsmith.  Is Power-Seeking AI an existential risk?  June 2022.
  * Chan et al. (2023) A. Chan, R. Salganik, A. Markelius, C. Pang, N. Rajkumar, D. Krasheninnikov, L. Langosco, Z. He, Y. Duan, M. Carroll, M. Lin, A. Mayhew, K. Collins, M. Molamohammadi, J. Burden, W. Zhao, S. Rismani, K. Voudouris, U. Bhatt, A. Weller, D. Krueger, and T. Maharaj.  Harms from increasingly agentic algorithmic systems.  Feb. 2023.
  * Emery-Xu et al. (2023) N. Emery-Xu, A. Park, and R. Trager.  Uncertainty, information, and risk in international technology races.  _Unpublished manuscript_ , 2023.
  * Everitt et al. (2021) T. Everitt, R. Carey, E. Langlois, P. A. Ortega, and S. Legg.  Agent incentives: A causal perspective.  Feb. 2021.
  * Ganguli et al. (2022) D. Ganguli, D. Hernandez, L. Lovitt, N. DasSarma, T. Henighan, A. Jones, N. Joseph, J. Kernion, B. Mann, A. Askell, Y. Bai, A. Chen, T. Conerly, D. Drain, N. Elhage, S. El Showk, S. Fort, Z. Hatfield-Dodds, S. Johnston, S. Kravec, N. Nanda, K. Ndousse, C. Olsson, D. Amodei, D. Amodei, T. Brown, J. Kaplan, S. McCandlish, C. Olah, and J. Clark.  Predictability and surprise in large generative models.  Feb. 2022.
  * Gao et al. (2022) L. Gao, J. Schulman, and J. Hilton.  Scaling laws for reward model overoptimization.  Oct. 2022.
  * Glaese et al. (2022) A. Glaese, N. McAleese, M. Trębacz, J. Aslanides, V. Firoiu, T. Ewalds, M. Rauh, L. Weidinger, M. Chadwick, P. Thacker, L. Campbell-Gillingham, J. Uesato, P.-S. Huang, R. Comanescu, F. Yang, A. See, S. Dathathri, R. Greig, C. Chen, D. Fritz, J. S. Elias, R. Green, S. Mokrá, N. Fernando, B. Wu, R. Foley, S. Young, I. Gabriel, W. Isaac, J. Mellor, D. Hassabis, K. Kavukcuoglu, L. A. Hendricks, and G. Irving.  Improving alignment of dialogue agents via targeted human judgements.  Sept. 2022.
  * Hadfield-Menell et al. (2016) D. Hadfield-Menell, A. Dragan, P. Abbeel, and S. Russell.  The Off-Switch game.  Nov. 2016.
  * Jones et al. (2023) E. Jones, A. Dragan, A. Raghunathan, and J. Steinhardt.  Automatically auditing large language models via discrete optimization.  Mar. 2023.
  * Kenton et al. (2022) Z. Kenton, R. Kumar, S. Farquhar, J. Richens, M. MacDermott, and T. Everitt.  Discovering agents.  Aug. 2022.
  * Krakovna and Kramar (2023) V. Krakovna and J. Kramar.  Power-seeking can be probable and predictive for trained agents.  Apr. 2023.
  * Liang et al. (2022) P. Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Ré, D. Acosta-Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. Wang, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha, N. Chatterji, O. Khattab, P. Henderson, Q. Huang, R. Chi, S. M. Xie, S. Santurkar, S. Ganguli, T. Hashimoto, T. Icard, T. Zhang, V. Chaudhary, W. Wang, X. Li, Y. Mai, Y. Zhang, and Y. Koreeda.  Holistic evaluation of language models.  Nov. 2022.
  * McKenzie et al. (2022) I. McKenzie, A. Lyzhov, A. Parrish, A. Prabhu, A. Mueller, N. Kim, S. Bowman, and E. Perez.  Inverse scaling prize.  https://github.com/inverse-scaling/prize, 2022.  Accessed: 2023-4-7.
  * Michael et al. (2022) J. Michael, A. Holtzman, A. Parrish, A. Mueller, A. Wang, A. Chen, D. Madaan, N. Nangia, R. Y. Pang, J. Phang, and S. R. Bowman.  What do NLP researchers believe? results of the NLP community metasurvey.  Aug. 2022.
  * Mitchell et al. (2018) M. Mitchell, S. Wu, A. Zaldivar, P. Barnes, L. Vasserman, B. Hutchinson, E. Spitzer, I. D. Raji, and T. Gebru.  Model cards for model reporting.  Oct. 2018.
  * Mökander et al. (2023) J. Mökander, J. Schuett, H. R. Kirk, and L. Floridi.  Auditing large language models: a three-layered approach.  Feb. 2023.
  * Nanda et al. (2023) N. Nanda, L. Chan, T. Lieberum, J. Smith, and J. Steinhardt.  Progress measures for grokking via mechanistic interpretability.  Jan. 2023.
  * Ngo et al. (2022) R. Ngo, L. Chan, and S. Mindermann.  The alignment problem from a deep learning perspective.  Aug. 2022.
  * Olah et al. (2020) C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter.  Zoom in: An introduction to circuits.  _Distill_ , 5(3), Mar. 2020.
  * OpenAI (2023a) OpenAI.  GPT-4 system card.  Mar. 2023a.
  * OpenAI (2023b) OpenAI.  GPT-4 technical report.  Mar. 2023b.
  * Ord (2022) T. Ord.  Lessons from the development of the atomic bomb.  Technical report, Centre for the Governance of AI, Nov. 2022.
  * Orseau and Armstrong (2016) L. Orseau and S. Armstrong.  Safely interruptible agents.  https://www.deepmind.com/publications/safely-interruptible-agents, 2016\.  Accessed: 2023-4-26.
  * Pan et al. (2023) A. Pan, C. J. Shern, A. Zou, N. Li, S. Basart, T. Woodside, J. Ng, H. Zhang, S. Emmons, and D. Hendrycks.  Do the rewards justify the means? measuring Trade-Offs between rewards and ethical behavior in the MACHIAVELLI benchmark.  Apr. 2023.
  * Perez et al. (2022a) E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving.  Red teaming language models with language models.  Feb. 2022a.
  * Perez et al. (2022b) E. Perez, S. Ringer, K. Lukošiūtė, K. Nguyen, E. Chen, S. Heiner, C. Pettit, C. Olsson, S. Kundu, S. Kadavath, A. Jones, A. Chen, B. Mann, B. Israel, B. Seethor, C. McKinnon, C. Olah, D. Yan, D. Amodei, D. Amodei, D. Drain, D. Li, E. Tran-Johnson, G. Khundadze, J. Kernion, J. Landis, J. Kerr, J. Mueller, J. Hyun, J. Landau, K. Ndousse, L. Goldberg, L. Lovitt, M. Lucas, M. Sellitto, M. Zhang, N. Kingsland, N. Elhage, N. Joseph, N. Mercado, N. DasSarma, O. Rausch, R. Larson, S. McCandlish, S. Johnston, S. Kravec, S. El Showk, T. Lanham, T. Telleen-Lawton, T. Brown, T. Henighan, T. Hume, Y. Bai, Z. Hatfield-Dodds, J. Clark, S. R. Bowman, A. Askell, R. Grosse, D. Hernandez, D. Ganguli, E. Hubinger, N. Schiefer, and J. Kaplan.  Discovering language model behaviors with Model-Written evaluations.  Dec. 2022b.
  * Raji et al. (2020) I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes.  Closing the AI accountability gap: Defining an End-to-End framework for internal algorithmic auditing.  Jan. 2020.
  * Raji et al. (2022a) I. D. Raji, I. Elizabeth Kumar, A. Horowitz, and A. D. Selbst.  The fallacy of AI functionality.  June 2022a.
  * Raji et al. (2022b) I. D. Raji, P. Xu, C. Honigsberg, and D. E. Ho.  Outsider oversight: Designing a third party audit ecosystem for AI governance.  June 2022b.
  * Rismani et al. (2023) S. Rismani, R. Shelby, A. Smart, E. Jatho, J. Kroll, A. Moon, and N. Rostamzadeh.  From plane crashes to algorithmic harm: Applicability of safety engineering frameworks for responsible ML.  In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ , number Article 2 in CHI ’23, pages 1–18, New York, NY, USA, Apr. 2023. Association for Computing Machinery.
  * Shah et al. (2022) R. Shah, V. Varma, R. Kumar, M. Phuong, V. Krakovna, J. Uesato, and Z. Kenton.  Goal misgeneralization: Why correct specifications aren’t enough for correct goals.  Oct. 2022.
  * Shevlane (2022a) T. Shevlane.  Sharing powerful AI models.  https://www.governance.ai/post/sharing-powerful-ai-models, Jan. 2022a.  Accessed: 2023-3-18.
  * Shevlane (2022b) T. Shevlane.  Structured access: An emerging paradigm for safe AI deployment.  Feb. 2022b.
  * Turner et al. (2021) A. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli.  Optimal policies tend to seek power.  _Adv. Neural Inf. Process. Syst._ , 34:23063–23074, Dec. 2021.
  * Wei et al. (2022a) J. Wei, N. Kim, Y. Tay, and Q. V. Le.  Inverse scaling can become u-shaped.  Nov. 2022a.
  * Wei et al. (2022b) J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus.  Emergent abilities of large language models.  June 2022b.
  * Wei et al. (2022c) J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou.  Chain-of-Thought prompting elicits reasoning in large language models.  Jan. 2022c.
  * Whittlestone and Clark (2021) J. Whittlestone and J. Clark.  Why and how governments should monitor AI development.  Aug. 2021.
  * Ziegler et al. (2022) D. M. Ziegler, S. Nix, L. Chan, T. Bauman, P. Schmidt-Nielsen, T. Lin, A. Scherlis, N. Nabeshima, B. Weinstein-Raun, D. de Haas, B. Shlegeris, and N. Thomas.  Adversarial training for High-Stakes reliability.  May 2022.
  * Zwetsloot and Dafoe (2019) R. Zwetsloot and A. Dafoe.  Thinking about risks from AI: Accidents, misuse and structure.  https://www.lawfareblog.com/thinking-about-risks-ai-accidents-misuse-and-structure, Feb. 2019.  Accessed: 2023-3-18.

## Appendix: Deployment safety controls

Variable: |  Includes:
---|---
Scale |  How many end users? How many agents are running at any one time, or how many times per day is the model called? How many applications will be built on top of the model?
Use restrictions |  Are certain high stakes applications prohibited?
Generality |  Will a single model be flexibly applied across a range of applications, or will narrower, application-specific versions of the model be fine-tuned?
Autonomy |  Is the AI system tasked with executing tasks, or merely responding to queries? How long are the chains of actions the model can take? Can the model define new types of actions?
Tool use |  Web browsing, telephone calls, code execution, control over robotic hardware, calling APIs, access to persistent memory.
Depth of model access |  How restrictive is the interaction between the AI system and the user? Can all of the model’s capabilities be accessed? Can users or developers fine-tune the model?
Oversight and moderation |  How closely are the model’s outputs monitored by the provider? Can certain outputs be automatically filtered out?
Global planning |  Does the model have access to many user interactions at once, so it can make plans across them, or is it confined to dealing with each user individually?
Adjustments to model |  The developer could decide to deploy a smaller version of the model. The developer could attempt to remove certain dangerous capabilities via fine-tuning.
Table 3: Variables that affect the risk level of deployment, each of which can
be adjusted on the basis of evaluation results.


# Multi-Agent Reinforcement Learning Based Coded Computation for Mobile Ad Hoc
Computing

Baoqian Wang1 Junfei Xie2 Kejie Lu3 Yan Wan4 Shengli Fu5 Baoqian Wang is with
the Department of Electrical and Computer Engineering, University of
California, San Diego, and San Diego State University, San Diego, CA, 92182
(e-mail: bawang@ucsd.edu).Junfei Xie is with the Department of Electrical and
Computer Engineering, San Diego State University, San Diego, CA, 92182
(e-mail: jxie4@sdsu.edu). Corresponding author.Kejie Lu is with the Department
of Computer Science and Engineering, University of Puerto Rico at Mayagüez,
Mayagüez, Puerto Rico, 00681, e-mail: (kejie.lu@upr.edu). Yan Wan is with the
Department of Electrical Engineering, University of Texas at Arlington,
Arlington, Texas, 76019 (e-mail: yan.wan@uta.edu).Shengli Fu is with the
Department of Electrical Engineering, University of North Texas, Denton,
Texas, 76201 (e-mail: Shengli.Fu@unt.edu).

###### Abstract

Mobile ad hoc computing (MAHC), which allows mobile devices to directly share
their computing resources, is a promising solution to address the growing
demands for computing resources required by mobile devices. However,
offloading a computation task from a mobile device to other mobile devices is
a challenging task due to frequent topology changes and link failures because
of node mobility, unstable and unknown communication environments, and the
heterogeneous nature of these devices. To address these challenges, in this
paper, we introduce a novel coded computation scheme based on multi-agent
reinforcement learning (MARL), which has many promising features such as
adaptability to network changes, high efficiency and robustness to uncertain
system disturbances, consideration of node heterogeneity, and decentralized
load allocation. Comprehensive simulation studies demonstrate that the
proposed approach can outperform state-of-the-art distributed computing
schemes.

##  I Introduction

In recent years, we have witnessed the significant growth of mobile
applications that are based on advanced artificial intelligence and deep-
learning models, such as face recognition, object detection and natural
language processing. These applications stimulate the demands for greater
computing resources, which may be difficult to support in one device. To
address this need, one promising solution is to offload computation-intensive
tasks from resource-limited mobile devices to remote clouds or nearby edge
computing facilities [1]. Nevertheless, this solution can suffer from high
transmission delays and becomes infeasible when there are no communication
infrastructures nearby, e.g., in rural areas or emergencies. Local computing
is thus essential for delay-sensitive applications, or when there is no or
weak wireless Internet connection to remote clouds and when nearby edge
computing devices are not available.

Mobile ad hoc computing (MAHC) [2, 3] is coined recently to provide local
computing services, by enabling resource sharing among mobile devices.
However, the research on MAHC is still in its infancy. Existing studies have
mainly considered smartphones [4] or ground vehicles [5] as the main
resources. A few recent studies attempted to move computing resources up to
the unmanned aerial vehicles (UAVs) and create a networked airborne computing
system [6, 7] that allows UAVs to share resources between each other through
direct flight-to-flight communications. This can be considered as one type of
MAHC, but studies along this direction are still very limited.

Offloading a computation task from a mobile device to other mobile devices
nearby is challenging, due to the mobility and heterogeneity of these devices.
Particularly, devices’ movement can cause frequent topology changes and
communication instability/failures. Furthermore, mobile devices may not have
an explicit knowledge of its communication environment and the computing
powers of other devices. In this paper, we aim to address these challenges, by
exploring coded computation [8] and multi-agent reinforcement learning (MARL)
[9].

Coded computation is a promising technique for improving the resilience of
distributed computing systems to uncertain system disturbances, such as
communication bottlenecks, node/link failures, slow-downs of computing nodes,
etc. Different coded computation tasks have been explored for systems with
homogeneous computing nodes, such as matrix-vector multiplication [10],
gradient descent [11], multivariate polynomials [12], etc. Coded computation
for heterogeneous systems has also been recently investigated. For example,
[13] introduced a _Heterogeneous Coded Matrix Multiplication_ (HCMM) scheme
that maximizes the expected computing results received by the master node. Our
research group developed a batch processing based coded computation (BPCC)
scheme [14, 15], which allows partial results to be returned and is highly
efficient and robust to uncertain system disturbances. However, all these
works assume static networks and adopt simplified communication and computing
models. Recently, a coded cooperative computation protocol (C3P) [16] was
proposed that considers mobile computing systems, and allocates tasks in a
dynamic and adaptive manner based on the time lag between two consecutive
tasks. However, this method requires explicit knowledge of the delay model.

MARL has been widely used for applications that involve interactions among
multiple agents, such as multi-robot control, multiplayer games, Internet of
Things, to name a few. Although multiple studies apply RL/MARL to solve
computation offloading or resource allocation problems, e.g., [17, 18], the
use of MARL for MAHC, especially with coded computation, has not been
investigated.

In this paper, we consider distributed computing over a MAHC system that
consists of multiple mobile and heterogeneous computing devices, with unknown
computing power and communication environment. As a preliminary investigation,
we focus on a classical distributed computing task, the matrix-vector
multiplication problem, which is a building block of many computation tasks
and machine learning algorithms. Based on this task, our _main contribution_
is an innovative MARL-based coded computation framework with the following key
features: 1) adaptability to time-varying communication changes caused by node
mobility; 2) capability to address node heterogeneity; 3) applicability for
any MAHC or mobile edge/fog computing (MEC) systems without having to know
their communication or computing characteristics; 4) high computational
efficiency by applying the multi-agent deep deterministic policy gradient
(MADDPG) [19], the state-of-the-art MARL algorithm, and adopting the batch
processing procedure proposed in our previous studies [14, 15]; and 5)
decentralized load allocation that relieves the computation burden at the
master node. To demonstrate these promising features, we conduct simulation
studies on a networked airborne computing system.

In the rest of this paper, Sec. II describes the computing system, distributed
computing schemes, and the optimization problem. Sec. III transforms the
original problem into a MARL problem and then presents a MADDPG- and batch
processing based solution. Simulation results are presented in Sec. IV,
followed by the conclusion in Sec. V.

##  II Problem Description

In this section, we first describe the computing system and the computation
tasks considered in this study, and then present the uncoded and coded
distributed computing schemes. Finally, we formulate the optimization problem.

###  II-A Computing System

Consider a MAHC system that consists of multiple mobile devices with different
computing powers. These devices can talk to each other through wireless
communication. Suppose one of the devices receives/has a sequence of $K$
matrix-vector multiplication tasks to complete, denoted as
$\\{\mathbf{A}\mathbf{x}_{1},\mathbf{A}\mathbf{x}_{2},\dots,\mathbf{A}\mathbf{x}_{K}\\}$,
where $\mathbf{A}\in\mathbbm{R}^{p\times m}$ is a pre-stored matrix and
$\mathbf{x}_{j}\in\mathbbm{R}^{m\times 1}$, $j\in[K]:=\\{1,2,\dots,K\\}$ are
input vectors. Due to limited computing power, this device, often called the
_master node_ , offloads the computation tasks to its $N$ neighbors, known as
the _worker nodes_ , in a sequential order, with task $j$ being offloaded only
after the previous task $j-1$ has been completed.

Offloading a computation task $j$ is achieved by first partitioning the task
into $N$ subtasks. Each worker node then executes one subtask and returns the
result to the master node after completion. The master recovers and outputs
the value of task $j$, i.e., $\mathbf{A}\mathbf{x}_{j}$, after receiving
sufficient results and then moves on to the next task. In the following
subsection, we describe how the traditional uncoded and coded computation
schemes partition and allocate a task.

###  II-B Distributed Computing Schemes

####  II-B1 Uncoded Scheme

The traditional uncoded scheme partitions a matrix-vector multiplication task
$\mathbf{A}\mathbf{x}_{j}$ by decomposing matrix $\mathbf{A}$ row-wise into
$N$ non-overlapping submatrices
$\\{\mathbf{A}_{1,j},\mathbf{A}_{2,j},\dots,\mathbf{A}_{N,j}\\}$, where
$\mathbf{A}_{i,j}\in\mathbb{R}^{\ell_{i,j}\times m}$. Each worker node $i$
computes subtask $\mathbf{A}_{i,j}\mathbf{x}_{j}$, $i\in[N]$. The master node
is able to recover $\mathbf{A}\mathbf{x}_{j}$ after receiving results from all
the worker nodes, by simply concatenating the results, i.e.,
$\mathbf{A}\mathbf{x}_{j}=[\mathbf{A}_{1,j}\mathbf{x}_{j};\mathbf{A}_{2,j}\mathbf{x}_{j};\dots;\mathbf{A}_{N,j}\mathbf{x}_{j}]$.

One major drawback of this scheme is that the master node cannot recover
$\mathbf{A}\mathbf{x}_{j}$ until all results from the worker nodes have been
received. Because of this, the efficiency of the uncoded scheme is bounded by
the slowest worker node, making it inefficient when uncertain system
disturbances are prominent, e.g., in a MAHC system.

####  II-B2 Coded Scheme

The coded scheme overcomes the drawback of the uncoded scheme by introducing
redundant computations using the coding theory. In particular, for each task
$j$, the coded scheme first encodes $\mathbf{A}$ into a larger matrix
${\hat{\mathbf{A}}}_{j}\in\mathbb{R}^{q_{j}\times m}$ with additional rows,
i.e., $q_{j}>p$, by applying the equation below

$\hat{\mathbf{A}}_{j}=\mathbf{G}_{j}\mathbf{A}.$ (1)

where $\mathbf{G}_{j}\in\mathbb{R}^{q_{j}\times p}$ is an encoding matrix with
the property that any matrix formed by any $p$ rows of $\mathbf{G}_{j}$ has
full rank. With $\hat{\mathbf{A}}_{j}$, the coded scheme then follows a
similar procedure to partition and allocate the task. Particularly, worker
node $i$ executes subtask $\hat{\mathbf{A}}_{i,j}\mathbf{x}_{j}$, where
$\hat{\mathbf{A}}_{i,j}\in\mathbb{R}^{\ell_{i,j}\times m}$, $i\in N$ is a
submatrix of $\hat{\mathbf{A}}_{j}$, and $\sum_{i=1}^{N}\ell_{i,j}=q_{j}$.

To recover the final result $\mathbf{A}\mathbf{x}_{j}$, the master node waits
until sufficient, but not all, results are received. Specifically, let
$\hat{\mathbf{y}}_{j}$ be the results received by the master node by a certain
time, which satisfies
$\hat{\mathbf{y}}_{j}=\hat{\mathbf{G}}_{j}\mathbf{A}\mathbf{x}_{j}$, where
$\hat{\mathbf{G}}_{j}$ is a submatrix of $\mathbf{G}_{j}$.
$\mathbf{A}\mathbf{x}_{j}$ can then be recovered when
$|\hat{\mathbf{y}}_{j}|\geq p$ via:

$\mathbf{A}\mathbf{x}_{j}=(\hat{\mathbf{G}}_{j}^{\top}\hat{\mathbf{G}}_{j})^{-1}\hat{\mathbf{G}}_{j}^{\top}\hat{\mathbf{y}}_{j},$
(2)

###  II-C Optimization Problem

In distributed computing schemes, the load numbers $\ell_{i,j}$ determine the
amount of workloads assigned to each worker node. As worker nodes have
different computing powers and the time required by each node to transmit the
result is also different and time-varying, it is crucial to choose a proper
load number $\ell_{i,j}$ for each worker node $i\in[N]$ and each task
$j\in[K]$. To determine the optimal load numbers $\ell^{*}_{i,j}$, we
formulate an optimization problem to minimize the expected task completion
time.

Denote the time required by worker node $i$ to complete the $j$-th task, i.e.,
$\hat{\mathbf{A}}_{i,j}\mathbf{x}_{j}$ with
$\hat{\mathbf{A}}_{i,j}\in\mathbb{R}^{\ell_{i,j}\times m}$, as $T_{i,j}$. Then
$T_{i,j}$ can be captured by the following equation:

$T_{i,j}\approx
T_{i,comm}(\mathbf{x}_{j})+T_{i,comp}(\hat{\mathbf{A}}_{i,j},\mathbf{x}_{j})+T_{i,comm}(\hat{\mathbf{A}}_{i,j}\mathbf{x}_{j})$
(3)

where $T_{i,comm}(\mathbf{x}_{j})$ is the time spent to send $\mathbf{x}_{j}$
from the master node to the worker node $i$,
$T_{i,comp}(\hat{\mathbf{A}}_{i,j},\mathbf{x}_{j})$ is the time spent to
compute subtask $\hat{\mathbf{A}}_{i,j}\mathbf{x}_{j}$, and
$T_{i,comm}(\hat{\mathbf{A}}_{i,j}\mathbf{x}_{j})$ is the time spent to send
the computation result back to the master node. Note that $T_{i,j}$ is a
random variable reflective of various environmental and system uncertainties.
Explicit communication and computation models are not available to the nodes.

With $T_{i,j}$, we can then describe the task completion time for each task
$j\in[K]$ as follows:

$T_{j}=\underset{t}{\min}\ \ \\{t\leavevmode\nobreak\ \leavevmode\nobreak\ |\
\ R_{j}(t)\geq p\\}$ (4)

where $R_{j}(t)=\sum_{i=1}^{N}\ell_{i,j}\mathbbm{1}_{T_{i,j}\leq t}$ is the
total number of rows of inner product results for task $j$ that the master
node has received by time $t$. $\mathbbm{1}$ is the indicator function [20].
Finally, the optimization problem to solve can be formulated as follows,

$\displaystyle\underset{\ell_{i,j},\forall i\in[N],\forall j\in[K]}{\text{
minimize }}$ $\displaystyle\mathbb{E}\left[\sum_{j=1}^{K}T_{j}\right]$ (5)
subject to $\displaystyle\ell_{i,j}\in\mathbb{Z}^{+},\forall i\in[N],\forall
j\in[K]$ $\displaystyle\sum_{i=1}^{N}\ell_{i,j}\geq p,\forall i\in[N],\forall
j\in[K]$ (6)

##  III Main Results

In this section, we solve the optimization problem in (5), by first
transforming it into a MARL problem and then developing a MADDPG- and batch
processing based solution.

###  III-A MARL-based Formulation

To solve the optimization problem in (5), we provide a MARL-based formulation.
In particular, the $N$ worker nodes are regarded as the _agents_ that learn to
achieve the optimization goal in (5) by interacting with the environment. The
_state_ of each agent $i$ at time $t$ is denoted as
$\mathbf{s}_{i}(t)=[d_{i}(t),\mathbf{d}_{-i}(t),\mathbf{v}_{i}(t),\mathbf{v}_{-i}(t),\mathbf{v}_{m}(t)]^{\top}\in\mathcal{S}_{i}$,
where $d_{i}(t)$ and $\mathbf{d}_{-i}(t)$ represent the distance of agent $i$
to the master node and the distances of all agents except agent $i$ to the
master node at time $t$, respectively. $\mathbf{v}_{i}(t)$,
$\mathbf{v}_{-i}(t)$, and $\mathbf{v}_{m}(t)$ represent the velocity of agent
$i$, velocities of all agents except agent $i$, and velocity of the master
node at time $t$, respectively. $\mathcal{S}_{i}$ is the corresponding state
space.

Upon start of a new task $j$, each agent $i$ decides the computation load it
will execute for this task. This differs from the traditional load allocation
mechanisms where the master node decides the computation load for each worker
node (agent). Now denote the time when the master node sends task $j$ to the
agents as $t_{j}$, we then define the _action_ of each agent $i$ taken at time
$t_{j}$ as $a_{i}(t_{j})=\ell_{i,j}\in\mathcal{A}_{i}$, where
$\mathcal{A}_{i}$ is the corresponding action space. It is reasonable to set
$\mathcal{A}_{i}=[0,p]$, as the total computation load required for completing
task $j$ is $p$. Note that, during the execution of task $j$, agent $i$ does
not take any actions, and the next action is taken when the next task ${j+1}$
starts. The transition of the agent’s state from $\mathbf{s}_{i}(t_{j})$ to
$\mathbf{s}_{i}(t_{j+1})$ is determined by the mobility model of the agent
described by $\mathbf{s}_{i}(t_{j+1})=f_{i}(\mathbf{s}_{i}(t_{j}))$, and
$t_{j+1}=t_{j}+T_{j}$, where $T_{j}$ is the time spent to complete task $j$.
Here we assume the velocities of all agents and the master node do not change
during the execution each task $j$.

After each transition, agent $i$ receives a reward
$r_{i}(\mathbf{s}(t_{j}),\mathbf{a}(t_{j})):\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$,
where
$\mathbf{s}(t)=(\mathbf{s}_{1}(t),\mathbf{s}_{2}(t),\ldots,\mathbf{s}_{N}(t))\in\mathcal{S}:=\prod_{i\in[N]}\mathcal{S}_{i}$
and
$\mathbf{a}(t)=(a_{1}(t),a_{2}(t),\ldots,a_{N}(t))\in\mathcal{A}:=\prod_{i\in[N]}\mathcal{A}_{i}$
are the joint state and action for all agents, respectively, and
$\mathcal{S}$, $\mathcal{A}$ are the corresponding joint state and action
spaces. Here the reward function is defined as
$r_{i}(\mathbf{s}(t_{j}),\mathbf{a}(t_{j}))=-T_{j}-c\mathbbm{1}_{\sum_{i}^{N}\ell_{i,j}\leq
p}$, which takes the constraint (6) into consideration and $c$ is the weight
for the penalty term. Given state $\mathbf{s}_{i}$, each agent then aims to
choose a deterministic policy, specified by function
$\pi_{i}(\mathbf{s}_{i}):\mathcal{S}_{i}\mapsto\mathcal{A}_{i}$, such that the
expected cumulative discount reward given by

$\displaystyle
V^{\boldsymbol{\pi}}_{i}(\mathbf{s}):=\mathbb{E}_{\begin{subarray}{c}\mathbf{s}(t_{j})\sim\mathbf{f}\\\
\mathbf{a}(t_{j})\sim\boldsymbol{\pi}\end{subarray}}\biggl{[}\sum_{j=1}^{K}\gamma^{j-1}r_{i}(\mathbf{s}(t_{j}),\mathbf{a}(t_{j}))|\mathbf{s}(t_{1})=\mathbf{s}\biggr{]}$

is maximized. In the above equation, $\gamma$ is a discount factor,
$\boldsymbol{\pi}=(\pi_{1},\pi_{2},\ldots,\pi_{N})$ is the concatenated policy
of all agents, and $\mathbf{f}=(f_{1},f_{2},\ldots,f_{N})$ is the concatenated
mobility models of all agents. $V^{\boldsymbol{\pi}}_{i}(\mathbf{s})$ is also
known as the value function. Alternatively, an optimal policy $\pi_{i}^{*}$
for agent $i$ can be obtained by maximizing the action-value function:

$\displaystyle Q_{i}^{\boldsymbol{\pi}}(\mathbf{s},\mathbf{a}):=$
$\displaystyle\mathbb{E}_{\begin{subarray}{c}\mathbf{s}(t_{j})\sim\mathbf{f}\\\
\mathbf{a}(t_{j})\sim\boldsymbol{\pi}\end{subarray}}\biggl{[}\sum_{j=1}^{K}\gamma^{j-1}r_{i}(\mathbf{s}(t_{j}),\mathbf{a}(t_{j}))|$
$\displaystyle\mathbf{s}(t_{1})=\mathbf{s},\mathbf{a}(t_{1})=\mathbf{a}\biggr{]}$
(7)

and setting
$\pi_{i}^{*}(a_{i}|\mathbf{s}_{i})\in\arg\max_{a_{i}}\max_{\mathbf{a}_{-i}}Q_{i}^{*}(\mathbf{s},\mathbf{a})$,
where
$Q_{i}^{*}(\mathbf{s},\mathbf{a}):=\max_{\boldsymbol{\pi}}Q_{i}^{\boldsymbol{\pi}}(\mathbf{s},\mathbf{a})$
and $\mathbf{a}_{-i}$ denotes the actions of all agents except $i$.

###  III-B MADDPG- and Batch-Processing based Solution

To solve the above MARL problem, we apply an advanced MARL algorithm proposed
recently, called multi-agent deep deterministic policy gradient (MADDPG) [19].
The MADDPG is an off-policy algorithm that extends the deep deterministic
policy gradient (DDPG) method to multiple agents. For each agent $i$, it uses
four neural networks to approximate its policy
$\pi_{i}(\mathbf{s}_{i};\theta_{\pi_{i}})$, action-value function
$Q_{i}^{\boldsymbol{\pi}}(\mathbf{s},\mathbf{a};\theta_{Q_{i}})$, target
policy $\pi_{i}^{\prime}(\mathbf{s}_{i};\theta_{\pi_{i}}^{\prime})$, and
target action-value function
$Q_{i}^{\boldsymbol{\pi}^{\prime}}(\mathbf{s},\mathbf{a};\theta_{Q_{i}}^{\prime})$,
respectively, where
$\boldsymbol{\pi}^{\prime}=(\pi^{\prime}_{1},\pi^{\prime}_{2},\ldots,\pi^{\prime}_{N})$
is the concatenated target policy of all agents. A replay memory denoted by
$\mathcal{D}$ is used to store the transitions
$\left\\{(\mathbf{s},\mathbf{a},\mathbf{r},\mathbf{s}^{\prime})\right\\}$,
where
$\mathbf{r}=[r_{1}(\mathbf{s},\mathbf{a}),r_{2}(\mathbf{s},\mathbf{a}),\ldots,r_{N}(\mathbf{s},\mathbf{a})]$
and $\mathbf{s}^{\prime}=\mathbf{f}(\mathbf{s})$. In each training iteration,
a mini-batch $\mathcal{B}_{i}$ is sampled from the replay memory $\mathcal{D}$
and used to learn the parameters for agent $i$. In particular, the parameters
$\theta_{Q_{i}}$ for the action-value function are updated by minimizing the
temporal-difference error given as follows:

$\displaystyle J(\theta_{Q_{i}})$ $\displaystyle=$
$\displaystyle\frac{1}{|{\cal
B}_{i}|}\\!\sum_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},\mathbf{r})\in\mathcal{B}_{i}}\\!\\!\left(L_{i}^{\boldsymbol{\pi}^{\prime}}(\mathbf{s}^{\prime},r_{i})\\!-\\!Q_{i}^{\boldsymbol{\pi}}\left(\mathbf{s},\mathbf{a};\theta_{Q_{i}}\right)\right)^{2}$
(8)

where
$L_{i}^{\boldsymbol{\pi}^{\prime}}(\mathbf{s}^{\prime},r_{i})=r_{i}(\mathbf{s},\mathbf{a})+\gamma
Q_{i}^{\boldsymbol{\pi}^{\prime}}(\mathbf{s}^{\prime},\boldsymbol{\pi}^{\prime}(\mathbf{s}^{\prime}))$.
The policy parameters $\theta_{\pi_{i}}$ are updated using gradient ascent
based on the policy gradient theorem, with the gradient provided as
follows[21]:

$\displaystyle\nabla_{\theta_{\pi_{i}}}J(\theta_{\pi_{i}})$
$\displaystyle\approx$ $\displaystyle\frac{1}{|{\cal
B}_{i}|}\sum_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime},\mathbf{r})\in\mathcal{B}_{i}}\nabla_{\theta_{\pi_{i}}}\pi_{i}(\mathbf{s}_{i};\theta_{\pi_{i}})\nabla_{a_{i}}Q_{i}^{\boldsymbol{\pi}}\left(\mathbf{s},\mathbf{a}\right).$
(9)

The parameters for the target policy and the target action-value function are
updated via Polyak averaging using the following equations:

$\begin{gathered}\theta^{\prime}_{\pi_{i}}\leftarrow\tau\theta^{\prime}_{\pi_{i}}+(1-\tau)\theta_{\pi_{i}}\\\
\theta^{\prime}_{Q_{i}}\leftarrow\tau\theta^{\prime}_{Q_{i}}+(1-\tau)\theta_{Q_{i}},\end{gathered}$
(10)

where $\tau\in(0,1)$ is a hyperparameter. The procedure of MADDPG is
summarized in Algorithm 1.

// Initialize parameters

1 Initialize
$\theta_{\pi_{i}},\theta_{Q_{i}},\theta^{\prime}_{\pi_{i}},\theta^{\prime}_{Q_{i}},\forall
i\in[N]$

2 for  _$\text{iteration}=1:\text{max\\_iteration}$_ do

// Collect transitions

3 for  _$k=1:\text{max\\_episode\\_number}$_ do

4 for  _$j=1:K$ _ do

5 For each agent $i$, select
$a_{i}(t_{j})\\!=\\!\pi_{i}(\mathbf{s}_{i}(t_{j});\theta_{\pi_{i}})$.

6 Execute joint action $\mathbf{a}(t_{j})$ and receive new state
$\mathbf{s}^{\prime}(t_{j+1})$ and reward $\mathbf{r}(t_{j+1})$.

7 Store
$\\!(\mathbf{s}(t_{j}),\mathbf{a}(t_{j}),\mathbf{r}(\mathbf{s}(t_{j}),\mathbf{a}(t_{j})),\mathbf{s}^{\prime}(t_{j+1})\\!)$
in replay buffer $\\!\mathcal{D}$.

8

9 Sample a random mini-batch $\mathcal{B}_{i}$ for each agent $i,\forall
i\in[N]$.

10

// Update parameters

11 for  _$i=1:N$ _ do

12 Update $\theta_{Q_{i}}$ by minimizing the temporal-difference error in
(III-B).

13 Update $\theta_{\pi_{i}}$ using gradient ascent, with the gradient provided
in (III-B).

14

15 Update $\theta^{\prime}_{\pi_{i}}$ and $\theta^{\prime}_{Q_{i}}$ using
(10).

16

Algorithm 1 MADDPG Algorithm

To further improve the computational efficiency, we apply the batch processing
procedure developed in our previous studies [14, 15]. In particular, we let
each worker node $i$ further divide matrix $\hat{\mathbf{A}}_{i,j}$ into a set
of submatrices $\\{\hat{\mathbf{A}}_{i,j,k}\\}_{k=1}^{w_{i,j}}$ row-wise,
namely batches, where each submatrix $\hat{\mathbf{A}}_{i,j,k}$ has $b_{i,j}$
number of rows and $w_{i,j}=\lceil\frac{\ell_{i,j}}{b_{i,j}}\rceil$ is the
total number of batches. Note that the last batch has a size of
$\ell_{i,j}-(w_{i,j}-1)b_{i,j}$. The worker node then multiples the input
vector $\mathbf{x}_{j}$ with each batch one by one and sends the partial
result back to the master once available. In [14, 15], we have shown nice
properties of this batch processing procedure in improving system’s efficiency
and the resilience to uncertain system disturbances through both theoretical
and experimental studies. Algorithm 2 summarizes the procedures followed by
the master and worker nodes during the execution of computation tasks.

Input: $\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{K}$

Output:
$\mathbf{A}\mathbf{x}_{1},\mathbf{A}\mathbf{x}_{2},\dots,\mathbf{A}\mathbf{x}_{K}$

// Master node:

1 for  _$j=1:K$_ do

2 $\hat{\mathbf{y}}_{j}\leftarrow[\leavevmode\nobreak\ ]$

3 Broadcast $\mathbf{x}_{j}$ to all worker nodes.

4 do

5 Listen to the channel and collect results
$\hat{\mathbf{A}}_{i,j,k}\mathbf{x}_{j}$ from the worker nodes.

6
$\hat{\mathbf{y}}_{j}\leftarrow[\hat{\mathbf{y}}_{j};\hat{\mathbf{A}}_{i,j,k}\mathbf{x}_{j}]$

7 while  _$|\hat{\mathbf{y}}_{j}| <p$_

8 Send acknowledgements to all worker nodes.

9
$\mathbf{A}\mathbf{x}_{j}\leftarrow(\hat{\mathbf{G}}_{j}^{\top}\hat{\mathbf{G}}_{j})^{-1}\hat{\mathbf{G}}_{j}^{\top}\hat{\mathbf{y}}_{j}$

10

11 return $\mathbf{A}\mathbf{x}_{j}$

12

// Worker node $i$:

13 for  _$j=1:K$_ do

14 Determine $\ell_{i,j}$ by applying the policy trained using Algorithm 1.

15 Upon receiving $\mathbf{x}_{j}$:

16 Select $\ell_{i,j}$ rows from $\hat{\mathbf{A}}$ to construct
$\hat{\mathbf{A}}_{i,j}$.

17 Divide $\hat{\mathbf{A}}_{i,j}$ into batches
$\hat{\mathbf{A}}_{i,j,k},k\in[w_{i,j}]$

18 for  _$k=1:w_{i,j}$_ do

19 if  _Acknowledgement received_ then

20 break

21 Compute $\hat{\mathbf{A}}_{i,j,k}\mathbf{x}_{j}$.

22 Send $\hat{\mathbf{A}}_{i,j,k}\mathbf{x}_{j}$ back to the master node.

23

Algorithm 2 Batch Processing Based Coded Scheme

##  IV Simulation Studies

In this section, we conduct simulation studies to evaluate the performance of
the proposed MADDPG- and batch-processing based algorithm. As an illustration,
we consider a MAHC system formed by multiple UAVs. All experiments are run on
an Alienware Desktop with 32GB memory, 16-cores CPU with 3.6GHz. In the rest
of the section, we first describe the simulation settings and then show the
results.

###  IV-A Simulation Settings

####  IV-A1 Environment Description

We consider a MAHC system that consists of multiple UAVs flying at the same
altitude [22]. Each UAV is equipped with a micro-computer with different
computing powers. The communication between two UAVs is achieved through
directional antennas for extended communication range and higher bandwidth.
Assume the directional antennas are always aligned during the movement, which
can be achieved by the control algorithms developed in [22]. We can then
describe the time to transmit a matrix $\mathbf{X}\in\mathbb{R}^{m_{1}\times
n_{1}}$ between two UAVs as $T_{comm}(\mathbf{X})=\frac{m_{1}\times
n_{1}\times u}{C}$, where $u$ (bits) is the average size of the elements in
matrix $\mathbf{X}$. $C=Wlog_{2}(1+\frac{S}{Noise})$ (bits/sec) is the data
rate, according to the Shannon’s theory, where $W$ (Hz) is the communication
bandwidth between the two UAVs, $Noise$ (Watts) is the noise power, which is
assumed to be constant. $S$ (Watts) is the signal power that can be modeled by
$S=10^{\frac{(S_{d}-30)}{10}}$, with
$S_{d}=P_{t}+20\log_{10}(\lambda)-20\log_{10}(4\pi)-20\log_{10}(d)+G_{l\mid
dBi}+\omega$. Here, $P_{t}$ (dBm) is the transmitting power, $\lambda$ is the
wave length, $G_{l\mid dBi}$ is the sum of the transmitting and receiving
gains, $\omega$ is Gaussian noise with zero mean and variance $\sigma$, and
$d$ is the distance between the two UAVs [22].

To capture the computation time, we adopt the modeling technique used in many
recent studies [23, 13]. Particularly, we assume the time
$T_{comp}(\mathbf{A},\mathbf{x})$ to compute a matrix-vector multiplication
task $\mathbf{A}\mathbf{x}$ with $\mathbf{A}\in\mathbbm{R}^{\ell\times m}$ and
$\mathbf{x}\in\mathbbm{R}^{m\times 1}$ follows an exponential distribution
defined by:

$\operatorname{P}\left[T_{comp}(\mathbf{A},\mathbf{x})\leq
t\right]=1-e^{-\frac{\beta}{\ell}\left(t-\alpha\ell\right)}$ (11)

where $t\geq\alpha\ell$. $\beta$ and $\alpha$ are straggling and shift
parameters, respectively, and both $\beta$ and $\alpha$ are positive
constants.

To describe the mobility of the UAVs, we here adopt a simple model:
$\mathbf{p}(t^{\prime})=\mathbf{p}(t)+\mathbf{v}(t)(t^{\prime}-t)$, where
$\mathbf{p}(t)$ represents the position of a UAV at time $t$, and
$t^{\prime}>t$. In simulations, we assume the velocity $\mathbf{v}(t)$ of each
UAV is constant, i.e., $\mathbf{v}(t)=\mathbf{v}$.

####  IV-A2 Distributed Computing Schemes

For comparison, we consider the following three distributed computing schemes
as the benchmarks:

  * •

Uniform Uncoded: This is an uncoded computation scheme which divides the
computation loads equally, i.e., $\ell_{i}=\frac{p}{N}$, $\forall i\in[N]$.

  * •

Load-Balanced Uncoded [24]: This is an uncoded computation scheme which
divides the computation loads according to the computing capabilities of the
worker nodes. In particular, the computation load assigned to each worker node
$i$ is inversely proportional to the expected time for this node to compute an
inner product, i.e.,
$\ell_{i}\propto(\frac{\beta_{i}}{\alpha_{i}\beta_{i}+1})$ and
$\sum_{i=1}^{N}\ell_{i}=p$.

  * •

HCMM [24]: This is a state-of-the-art coded scheme that computes the load
number by $\ell_{i}=\frac{p}{h\lambda_{i}}$, where $\lambda_{i}$ is the
positive solution to
$e^{\beta_{i}\lambda_{i}}=e^{\alpha_{i}\beta_{i}}(\beta_{i}\lambda_{i}+1)$,
$h=\sum_{i=1}^{N}\frac{\beta_{i}}{1+\beta_{i}\lambda_{i}}$, and $\beta_{i}$,
$\alpha_{i}$ are the straggling and shift parameters for worker node $i$,
respectively.

Note that both load-balanced uncoded and HCMM schemes require explicit
knowledge of the nodes’ computing capabilities, which is not required by our
method.

####  IV-A3 Computation Scenarios

We consider three computation scenarios described as follows:

  * •

Scenario 1: $N=3$, $p=6000$.

  * •

Scenario 2: $N=4$, $p=8000$.

  * •

Scenario 3: $N=5$, $p=10000$.

In all scenarios, the total number of tasks is set to $K=30$, and the size of
each task $\mathbf{x}_{j}$ is $m=10000$. The initial position of each UAV is
sampled uniformly from the range $[-100,100]^{2}$, and its velocity is
randomly selected from the range $[-10m/s,10m/s]^{2}$ at each episode. The
computing parameter $\beta_{i}$ of each worker node $i$ is randomly sampled
from the range $[10^{4},10^{5}]$ and $\alpha_{i}$ is set to
$\alpha_{i}=\frac{1}{\beta_{i}}$. The communication model is configured by
setting $W=10^{4}$, $Noise=1.1\times 10^{-12}$, $S_{d}=6-20log_{10}(d)$, and
$\sigma=1$.

In our MADDPG- and batch processing based algorithm, the learning rates are
set to $\alpha=0.01$ and $\gamma=0.95$. To approximate the policies and
action-value functions, we use neural networks with four fully connected
layers. Each hidden layer has $64$ ReLU units. At the output layer, Sigmoid
units are used by the policy networks, and linear units are used by the
action-value networks. The weight for the penalty term in each reward function
is set to $c=200$.

###  IV-B Experimental Results

In this subsection, we first show the effectiveness of our algorithm and then
investigate the impact of an important parameter, batch size $b_{i,j}$, on the
computing performance. Comparison results with the benchmark schemes are
presented at the end.

####  IV-B1 Effect of Training

Fig. 1 shows the training reward of our algorithm in the three scenarios. Each
value is obtained by averaging the total rewards of all agents for every $250$
iterations over $1000$ episodes. We can see that in all scenarios, the average
total reward increases and finally converges with the increasing numbers of
training iterations.

Figure 1: Training reward of our algorithm in different scenarios.

####  IV-B2 Impact of Batch Size

We investigate the impact of batch size $b_{i,j}$ on the performance of our
algorithm, by varying $b_{i,j}$. For each value of $b_{i,j}$, we run Algorithm
2 to measure the task completion times, with load numbers $\ell_{i,j}$
determined by the policies trained using Algorithm 1. To also understand the
resilience of our algorithm to uncertain system disturbances, we further
consider the scenario with one uncertain straggler and the scenario without
any stragglers. The straggler is introduced by randomly selecting a worker
node at each iteration to sleep for 10 times of the computation time of that
node using the $time.sleep()$ function.

Fig. 2 shows the total completion time of all $K$ tasks averaged over $20$
episodes in different scenarios. As we can see, with the increase of batch
size, the performance degrades, which is consistent with our findings [14]. In
the following comparative studies, we set the batch size to $b_{i,j}=1$,
$\forall i\in[N]$ and $j\in[K]$.

Figure 2: The average total task completion time for different batch sizes
when there is (a) no straggler and (b) one uncertain straggler in different
scenarios.

####  IV-B3 Comparative Studies

We compare our MADDPG- and batch processing based algorithm with the three
benchmark schemes in different scenarios. For the load-balanced uncoded and
HCMM schemes, computing parameters $\beta_{i}$ and $\alpha_{i}$ are assumed to
be known.

The average total task completion time of each scheme in different scenarios
is shown in Fig. 3. As we can see, our algorithm achieves the best performance
in all scenarios regardless of whether there are any uncertain stragglers.
This verifies the effectiveness of our algorithm in addressing node mobility,
and also demonstrates its high computational efficiency and resilience to
uncertain system disturbances. Of interest, when no stragglers are present
(see Fig. 3), the two uncoded schemes outperform HCMM. This is because HCMM
introduces redundant computations, causing additional computation overhead.
However, when stragglers are present (see Fig. 3), HCMM achieves better
performance than the two uncoded schemes, due to the use of the coding
technique. Additionally, the uniform uncoded scheme shows the worst
performance in most scenarios as it ignores the heterogeneity nature of the
computing nodes.

Figure 3: The average task completion time of each scheme when there is (a) no
straggler and (b) one uncertain straggler in different scenarios.

##  V Conclusion

In this paper, we present a MARL-based coded computation framework for the
MAHC system that consists of multiple mobile computing devices sharing
resources among themselves. This framework addresses both node mobility and
the heterogeneity nature of mobile devices, and can be applied to any MAHC
systems without having to know their communication or computing
characteristics. Moreover, our framework, which applies the MADDPG, a state-
of-the-art MARL algorithm, enables the load allocation to be determined in a
decentralized manner, thus helping to relieve the computing burden for the
master node. It also adopts a batch processing procedure developed in our
previous studies to further improve the performance in the aspects of both
efficiency and resilience to uncertain system disturbances. The comprehensive
simulation studies show that our MADDPG- and batch processing based algorithm
achieves the best performance compared with existing uncoded and coded
schemes, including the uniform uncoded, load balanced uncoded, and HCMM. In
the future work, we will extend this study to consider more general scenarios,
such as varying node velocities and task sizes.

## Acknowledgement

We would like to thank the National Science Foundation (NSF) under Grants
CI-1953048/1730675/1730570/1730325, ECCS-1953049/1839804, CAREER-2048266 and
CAREER-1714519 for the support of this work.

## References

  * [1] S. Wang, X. Zhang, Y. Zhang, L. Wang, J. Yang, and W. Wang, “A survey on mobile edge networks: Convergence of computing, caching and communications,” _Ieee Access_ , vol. 5, pp. 6757–6779, 2017.
  * [2] X. Cao, L. Liu, Y. Cheng, and X. Shen, “Towards energy-efficient wireless networking in the big data era: A survey,” _IEEE Communications Surveys & Tutorials_, vol. 20, no. 1, pp. 303–332, 2017.
  * [3] I. Yaqoob, E. Ahmed, A. Gani, S. Mokhtar, M. Imran, and S. Guizani, “Mobile ad hoc cloud: A survey,” _Wireless Communications and Mobile Computing_ , vol. 16, no. 16, pp. 2572–2589, 2016.
  * [4] B. Li, Y. Pei, H. Wu, and B. Shen, “Heuristics to allocate high-performance cloudlets for computation offloading in mobile ad hoc clouds,” _The Journal of Supercomputing_ , vol. 71, no. 8, pp. 3009–3036, 2015.
  * [5] L. Liu, C. Chen, Q. Pei, S. Maharjan, and Y. Zhang, “Vehicular edge computing and networking: A survey,” _Mobile Networks and Applications_ , pp. 1–24, 2020.
  * [6] B. Wang, J. Xie, S. Li, Y. Wan, Y. Gu, S. Fu, and K. Lu, “Computing in the air: An open airborne computing platform,” _IET Communications_ , vol. 14, no. 15, pp. 2410–2419, 2020.
  * [7] B. Wang, J. Xie, S. Li, Y. Wan, S. Fu, and K. Lu, “Enabling high-performance onboard computing with virtualization for unmanned aerial systems,” in _2018 International Conference on Unmanned Aircraft Systems (ICUAS)_. IEEE, 2018, pp. 202–211.
  * [8] K. Lee, R. Pedarsani, D. Papailiopoulos, and K. Ramchandran, “Coded computation for multicore setups,” in _Proc. of IEEE ISIT 2017_ , 2017, pp. 2413–2417.
  * [9] K. Zhang, Z. Yang, and T. Başar, “Multi-agent reinforcement learning: A selective overview of theories and algorithms,” _arXiv preprint arXiv:1911.10635_ , 2019.
  * [10] K. Lee, M. Lam, R. Pedarsani, D. Papailiopoulos, and K. Ramchandran, “Speeding Up Distributed Machine Learning Using Codes,” _IEEE Transactions on Information Theory_ , vol. 64, no. 3, pp. 1514–1529, mar 2018\.
  * [11] R. Tandon, Q. Lei, A. G. Dimakis, and N. Karampatziakis, “Gradient Coding: Avoiding Stragglers in Distributed Learning,” in _Proc. of the 34th International Conference on Machine Learning_. PMLR, 2017, pp. 3368–3376.
  * [12] M. Rudow, K. Rashmi, and V. Guruswami, “A locality-based approach for coded computation,” _arXiv preprint arXiv:2002.02440_ , 2020.
  * [13] A. Reisizadeh, S. Prakash, R. Pedarsani, and S. Avestimehr, “Coded computation over heterogeneous clusters,” in _Proc. of IEEE ISIT 2017_ , 2017.
  * [14] B. Wang, J. Xie, K. Lu, Y. Wan, and S. Fu, “On batch-processing based coded computing for heterogeneous distributed computing systems,” _arXiv preprint arXiv:1912.12559_ , 2019.
  * [15] ——, “Coding for heterogeneous uav-based networked airborne computing,” in _2019 IEEE Globecom Workshops (GC Wkshps)_. IEEE, 2019, pp. 1–6.
  * [16] Y. Keshtkarjahromi, Y. Xing, and H. Seferoglu, “Dynamic heterogeneity-aware coded cooperative computation at the edge,” in _Proc. of ICNP 2018_. IEEE, sep 2018, pp. 23–33.
  * [17] H. Lu, C. Gu, F. Luo, W. Ding, and X. Liu, “Optimization of lightweight task offloading strategy for mobile edge computing based on deep reinforcement learning,” _Future Generation Computer Systems_ , vol. 102, pp. 847–861, 2020.
  * [18] J. Chen, Z. Wei, S. Li, and B. Cao, “Artificial intelligence aided joint bit rate selection and radio resource allocation for adaptive video streaming over f-rans,” _IEEE Wireless Communications_ , vol. 27, no. 2, pp. 36–43, 2020.
  * [19] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” in _Advances in neural information processing systems_ , 2017, pp. 6379–6390.
  * [20] Q. Y. Kenny _et al._ , “Indicator function and its application in two-level factorial designs,” _The Annals of Statistics_ , vol. 31, no. 3, pp. 984–994, 2003.
  * [21] R. S. Sutton and A. G. Barto, “An introduction to reinforcement learning, chapter 3,” 2018.
  * [22] M. Liu, Y. Wan, S. Li, F. L. Lewis, and S. Fu, “Learning and uncertainty-exploited directional antenna control for robust long-distance and broad-band aerial communication,” _IEEE Transactions on Vehicular Technology_ , vol. 69, no. 1, pp. 593–606, 2019.
  * [23] K. Lee, M. Lam, R. Pedarsani, D. Papailiopoulos, and K. Ramchandran, “Speeding up distributed machine learning using codes,” in _Proc. of IEEE ISIT 2016_. IEEE, aug 2016, pp. 1143–1147.
  * [24] A. Reisizadeh, S. Prakash, R. Pedarsani, and A. S. Avestimehr, “Coded computation over heterogeneous clusters,” _IEEE Transactions on Information Theory_ , vol. 65, no. 7, pp. 4227–4242, 2019.