# Confidence-Building Measures for Artificial Intelligence: Workshop
Proceedings00footnotetext: All authors provided substantive contributions to
the paper through sharing their ideas as participants in the workshop, writing
the paper, and/or editorial feedback and direction. The first two authors are
listed in order of contribution, and the remaining authors are listed
alphabetically. Some workshop participants have chosen to remain anonymous.
The claims in this paper do not represent the views of any author’s
organization. For questions about this paper, contact Sarah Shoker at
<EMAIL_ADDRESS>and Andrew Reddie at<EMAIL_ADDRESS>
Sarah Shoker1***Significant contribution, including writing, providing
detailed input for the paper, research, workshop organization, and setting the
direction of the paper., Andrew Reddie2******Significant contribution,
including providing detailed input for the paper, research, workshop
organization, and setting the direction of the paper., Sarah Barrington2, Ruby
Booth3, Miles Brundage1, Husanjot Chahal1,
Michael Depp4, Bill Drexel4, Ritwik Gupta2, Marina Favaro5, Jake Hecla2, Alan
Hickey1,
Margarita Konaev6, Kirthi Kumar2, Nathan Lambert7, Andrew Lohn6, Cullen
O’Keefe1, Nazneen Rajani7,
Michael Sellitto5, Robert Trager8, Leah Walker2, Alexa Wehsener9, Jessica
Young10
1OpenAI, 2University of California, Berkeley, 3Berkeley Risk and Security Lab,
4Center for a New American Security, 5Anthropic, 6Center for Security and
Emerging Technology,
7Hugging Face, 8Centre for the Governance of AI, 9Institute for Security and
Technology,
10Microsoft

(August 2023)

###### Abstract

Foundation models could eventually introduce several pathways for undermining
state security: accidents, inadvertent escalation, unintentional conflict, the
proliferation of weapons, and the interference with human diplomacy are just a
few on a long list. The Confidence-Building Measures for Artificial
Intelligence workshop hosted by the Geopolitics Team at OpenAI and the
Berkeley Risk and Security Lab at the University of California brought
together a multistakeholder group to think through the tools and strategies to
mitigate the potential risks introduced by foundation models to international
security. Originating in the Cold War, confidence-building measures (CBMs) are
actions that reduce hostility, prevent conflict escalation, and improve trust
between parties. The flexibility of CBMs make them a key instrument for
navigating the rapid changes in the foundation model landscape. Participants
identified the following CBMs that directly apply to foundation models and
which are further explained in this conference proceedings: 1. crisis hotlines
2. incident sharing 3. model, transparency, and system cards 4. content
provenance and watermarks 5. collaborative red teaming and table-top exercises
and 6. dataset and evaluation sharing. Because most foundation model
developers are non-government entities, many CBMs will need to involve a wider
stakeholder community. These measures can be implemented either by AI labs or
by relevant government actors.

## 1 Introduction

Foundation models could eventually introduce several opportunities for
undermining state security: accidents, inadvertent escalation, unintentional
conflict,111In this context, accidents occur when AI systems malfunction.
Inadvertent escalation happens due to inappropriate use of AI systems by
leaders or operators that intensify situations. Unintentional conflict occurs
when uncertainties in algorithm behavior hinder the ability of states to
signal effectively to adversaries, potentially increasing the likelihood of
conflict despite the ultimate intentions of involved states. [50] the
proliferation of weapons,[44] and the interference with human diplomacy are
just a few on a long list.[95] Meanwhile, new defense and security actors
continue to develop foundation model capabilities,[75, 85, 69, 29] increasing
the risk of an international crisis even further.

The Confidence-Building Measures for Artificial Intelligence workshop hosted
by the Geopolitics Team at OpenAI and the Berkeley Risk and Security Lab
(BRSL) at the University of California brought together participants from AI
labs, government, academia, and civil society to propose tools and strategies
to mitigate the potential risks introduced by foundation models to
international security. By foundation models, we mean models that use vast
amounts of data, self-supervision and deep learning methods which “can be
adapted…to a wide range of downstream tasks.”[10] The workshop included a mix
of presentations and breakout groups, where participants had the opportunity
to design possible confidence-building measures (CBMs). Together, participants
identified the following CBMs that directly apply to foundation models:

  * •

crisis hotlines

  * •

incident sharing

  * •

model, transparency, and system cards

  * •

content provenance and watermarks

  * •

collaborative red teaming exercises

  * •

table-top exercises

  * •

dataset and evaluation sharing

Popularized during the Cold War, CBMs represent “measures that address,
prevent, or resolve uncertainties among states. Designed to prevent the
escalation of hostilities and build mutual trust among erstwhile adversaries,
CBMs can be formal or informal, unilateral, bilateral, or multilateral, [such
as] military or political, and can be state-to-state or non-governmental.”[25]
Because states do not have perfect information about the capabilities or
intentions of their allies and adversaries, formal and informal rules can
establish predictability around state behavior, which in turn has the
potential to reduce misunderstandings and miscommunications between state
governments. This is in the interest of all parties.

Historical examples of CBMs include direct call lines between countries to
communicate during nuclear crises, reporting on weapon transfers between
states, inviting observers to witness military exercises that an outside
nation might otherwise construe as threatening, establishing clear “rules of
the road” for how adversarial navies should interact on the high seas in
peacetime, data exchanges on troop movements such as those mandated by the
Treaty on Conventional Forces in Europe, or on-site monitoring of technology
capabilities. In contrast to domestic or regional AI regulations that govern
the relationship between companies and consumers, CBMs target and address the
risks associated with state-to-state interactions by introducing
predictability into a typically opaque international environment. While CBMs
can target the prevention of a range of harms, workshop participants focused
on CBMs that mitigate human rights abuses, the proliferation of unconventional
weapons, and escalation due to misperceptions exacerbated by foundation
models.

Defense strategies now routinely address the risks and opportunities
associated with artificial intelligence, with some governments and think tanks
calling explicitly for confidence-building measures.[19] Yet with the notable
exception of the United Kingdom’s Integrated Review Refresh 2023, most
governments have not fully grappled with the implications of military AI, much
less foundation models.222Page 56 [88] Though many existing defense documents
do not directly target foundation models, governments can still fold the CBMs
identified in these proceedings into existing AI commitments, such as the U.S
Government’s Political Declaration on Responsible Military Use of Artificial
Intelligence and Autonomy.[14]

Building on the literature addressing the risks of AI to international
security, this workshop focused on generating practical CBMs that apply to
foundation models. The CBMs identified in these proceedings are not exhaustive
or equally feasible in today’s international climate. Where appropriate, we
outline political and technical limitations that could interfere with the
CBM’s success.

### 1.1 A Note on Terminology: Foundation Models, Generative AI, and Large
Language Models

For the sake of brevity, we use the term ‘foundation model’ to refer to both
base and fine-tuned models, generative AI, and large language models. Where
appropriate, we identify the specific type of AI model the CBM is meant to
address. The terms foundation model, large language model, and generative AI
are often used interchangeably, but there are significant, if imprecise,
differences between these terms. As Helen Toner notes, these terms do not have
“crisp boundaries…[but]…have emerged as attempts to point to a cluster of
research directions and AI systems that have become especially noteworthy in
recent years.”[91]

Foundation models are built using deep learning and self-supervision learning
methods and use vast amounts of data which, according to a 2022 paper by Rishi
Bommasani et al. at Stanford University, “can be adapted (e.g. fine-tuned) to
a wide range of downstream tasks.”[10] The large amount of data and
computational power used to train foundation models have led to impressive
improvements across a variety of domains.333ibid.

While foundation models are often associated with generative AI applications
like language and imagery (see below), these models can also be applied to
domains such as robotics, human-machine interaction, reasoning, and sentiment
analysis. On the other hand, generative AI is a narrower category of AI that
includes models and algorithms capable of generating media. These models
produce content like text, audio, imagery, and software code. Many public-
facing models that are available today have already been fine-tuned on a
foundation model. For example, ChatGPT models are fine-tuned on foundation
models called GPT-3.5 and GPT-4, while Stability AI uses foundation models
like StableLM to generate imagery.

### 1.2 Why do we need confidence-building measures for foundation models?

There is no shortage of historical crises where misperception or
miscommunication led to military escalation that neither side
wanted.444Misperception continues to be a popular research area for scholars
of military conflict, and some researchers suggest that the academic existence
of international relations is fundamentally linked to managing problems
related to information asymmetry and the anarchical conditions that make
misperception possible. Misperception plays a prominent causal role in the
bloodiest wars of the 20th century, whether that be in both World Wars, Cold
War ‘proxy’ conflicts like Vietnam, Korea, and Afghanistan, or more recent
21st century conflicts like the Second Gulf War and Syrian Civil War. There
are ample cases of militaries mistakenly targeting civilian planes and killing
most or all civilians onboard,[28] and there are numerous historical false
positives that only narrowly avoided nuclear exchange.555For a full list of
nuclear false alarms, please visit compendium of events.[23]

The flexibility of CBMs make them a key instrument for navigating the rapid
changes in the foundation model landscape. AI is a general purpose “enabling
technology” rather than a military technology in and of itself.666Page 6 [79]
For instance, current rule-making at the United Nations Convention on Certain
Conventional Weapons (UN CCW) focuses on weapons identified by the forum,[90]
which excludes many AI applications–such as generative AI–that are not
obviously categorized as a ‘weapon’ but that can nevertheless influence the
direction of international conflict. In particular, their non-binding, build-
as-you-go nature allows the CBMs to grow in specificity as the technology
necessarily evolves. This is essential, since it is not obvious what
capabilities foundation models possess after they are trained and new
capabilities are often revealed only after further red teaming and conducting
safety evaluations. Though several benchmarks exist for assessing foundation
models, they overwhelmingly point to rapid improvement in domain knowledge and
deduction.[30, 46, 76] These capabilities are already associated with
international security risks like providing information on the construction of
conventional and unconventional weapons.777Page 12 [44]

CBMs do not overrule or subvert important efforts at fora like the United
Nations and can act as an accompaniment to ongoing international regulatory
discussions. CBMs are, however, uniquely equipped to target risks associated
with foundation models due to the speed of their innovation and proliferation.
In comparison to formal rules or international treaties, CBMs can lower
coordination costs (such as time and money spent on bargaining) by reducing
the number of negotiating parties involved in discussions. CBMs are often
voluntary, which can incentivize participation from parties who are reluctant
to risk the full weight of their national credibility on formal treaties. CBMs
are more easily modified (and discarded).[51] CBMs can also ‘start small’ and
build into formal rules, an especially useful feature in a low-trust
international environment.[50]

Model performance and model safety are also separate research pursuits,
meaning that the performance of foundation models can improve with little
change to their safety profile. A large language model that can generate
information about nuclear physics is an example of a capability, while a large
language model that refuses a user request to output specific details about
bomb-building is an example of a safety mitigation. To date, AI labs have
tackled the gap between model performance and safety by investing in a range
of sociotechnical measures. Such measures include research into
interpretability and alignment,[102, 9, 6, 32, 86, 103, 104, 24] public
disclosure of risks through system cards[44] and transparency notes,[21]
delaying the release of models until sufficient safety mitigations have been
implemented,888Page 19 [44] and [67] and open-sourcing evaluations[16, 33, 35]
and provenance research.999For example, as part of its Content Authenticity
Initiative (CAI), Adobe open-sourced JavaScript SDK and Rust SDK, which is
“designed to let developers build functions displaying content credentials in
browsers, or make custom desktop and mobile apps that can create, verify and
display content credentials.”[43] Despite these efforts, the machine learning
community is in general consensus that harm mitigations need further
improvement to keep up with the rapidly increasing performance of LLMs.[5, 64,
40]

This landscape is further challenged by typical state behavior at the
international level. States are often reluctant to engage in cooperative
security agreements that require too much transparency into national
capabilities. They are even less likely to place limits on the development of
their own capabilities in the absence of any guarantees that their adversaries
will do the same.101010For example, France’s Defence Ethics Committee advised
the continuation of research into autonomy and weapons systems, citing, among
other reasons, the need to “counter enemy development of LAWS; and…to be able
to defend ourselves against this type of weapon in the likely event of their
use by an enemy State or terrorist group against our troops or
population.”[74] However, because performance and safety research are two
different research streams, it is possible to coordinate on security while
limiting availability of research into performance improvements. This
unintended silver-lining is known to AI labs, which is why commercial labs are
often willing to open-source safety research into evaluations and provenance
technologies.

### 1.3 An Overview of CBMs for Foundation Models

Drawing from the list published by the United Nations Office of Disarmament
Affairs, these proceedings organize CBMs under four categories: communication
and coordination, observation and verification, cooperation and integration,
and transparency.[82] These categories are not discrete; many CBMs can
comfortably fit into more than one category.

Because most foundation model developers are non-government entities, many
CBMs will need to involve a wider stakeholder community. These measures can be
implemented either by AI labs or by relevant government actors.111111We opted
to exclude a discussion on cyber risks from the scope of this paper since
legal advances published in the Tallinn Manuals, NATO announcements on what
counts as ‘cyber war,’ and norm setting at the UN Group of Governmental
Experts on state behavior in cyberspace means that the topic deserves its own
devoted forum. Throughout the paper, we provide examples of adjacent
technologies that have contributed to international crises, with the
understanding that these examples can help us better anticipate the risks
posed by foundation models, which are currently nascent or empirically
unconfirmed.

### 1.4 Communication and Coordination

Communication and coordination CBMs reduce misunderstandings and
misperceptions that, if left unaddressed, could escalate into conflict. The
workshop identified two communication and coordination challenges that could
be remedied using communication and coordination CBMs: misperceptions about
authenticity of the content, and misperceptions concerning who authorized a
decision.

First, on the topic of content authenticity, several workshop participants
reiterated that foundation models and, specifically, generative AI, can be
used to perpetuate ‘truth decay,’ or increased public distrust towards the
information reported by political leaders and other experts. That distrust, in
turn, complicates reporting on international events and crises.[41, 73] For
example, in March 2022, a widely circulated deepfake video on social media
showed Ukrainian President Volodymyr Zelenskyy instructing soldiers to
surrender to Russian forces.[4] Individuals may soon speak with interactive
deepfakes, where the deepfake is both able to pause appropriately for the
other speaker and use predictive modeling and synthetic audio to carry on a
conversation.[53] And we could see the use of compositional deepfakes–not just
one fake video or image, but many of them–released over time in between real
events, to create a synthetic history that seems believable.121212ibid.

Second, strong communication and coordination CBMs allow human actors to
account for the ambiguity an AI injects into a system or team.[50] AI systems
are often designed with the intention of supporting or augmenting human
decision-making, making it challenging to disentangle the contributions of
human operators. AI systems may also generate outputs or decisions that can be
misinterpreted or misunderstood by human operators or other AI systems; in
some cases the integration of AI in human-machine teams131313We define ‘human
machine’ team as “a relationship–one made up of at least three equally
important elements: the human, the machine, and the interactions and
interdependencies between them.”[12] can obfuscate whether AI was the
(inadvertent, accidental, or intentional) cause of military escalation. A case
in point is the 1988 tragedy of Iran Air Flight 655, which was targeted by an
Aegis cruiser–the most sophisticated anti-aircraft weapon system at the
time–on the order of the USS Vincennes, killing 290 civilians. The accident
was blamed on a number of factors: the Aegis incorrectly identified the
commercial airliner as a military aircraft; the commander of the Vincennes was
characterized as being unnecessarily aggressive in a high-pressure atmosphere
prone to misinterpretation; a nearby US navy ship, the USS Sides, had a human
commander who correctly deduced that Iran Air Flight 655 was a civilian
aircraft, but believed the Aegis’s identification system to be technologically
superior to his own human judgement and did not share his assessment with the
Vincennes.[45, 38] The Aegis radar system did eventually identify the Iran Air
Flight 655 as a civilian aircraft, but the human operators chose to accept the
first reading. The Iran Air Flight 655 accident features many of the
challenges that exist in today’s human-machine teams: overtrust and a
reluctance to contest the decisions made by the system, misunderstanding the
threat due to existing geopolitical hostilities, and cherry-picking evidence
to support one’s interpretation of events. The introduction of AI to this
atmosphere, which promises to increase the speed of targeting and analysis
using a black-boxed technology, makes it even more necessary to identify
communication pathways to prevent accidents.

Hotlines

The ability to interpret human intentions can become more challenging when
communication integrates with or is supplanted by a non-human entity. Hotlines
can assist with clarifying the ‘who’ or ‘what’ was responsible for military
escalation, and for clarifying red lines[99] to avoid crossing them in the
first place.141414For more information on how hotlines can clarify red lines,
see: [98] Workshop participants noted that competitor states could establish
communication links to reduce friction during political crises, building on
state-to-state hotlines that exist today for the management of military
crises.

Despite their prominent role in mitigating nuclear crises, recent political
events have underscored the reality that security norms will inform when
parties use–or refuse–a phone call. This point was made especially evident
during the February 2023 crisis involving a Chinese spy balloon traveling
across the United States, and the subsequent refusal by the People’s
Liberation Army (PLA) to answer a hotline call from U.S Defense Secretary
Lloyd Austin. Immediately following the crisis, researchers offered several
explanations for the PLA’s behavior that pointed to a discrepancy between how
both military powers interpreted the threat landscape. Some stated that the
PLA viewed CBMs and transparency as “disadvantageous” and a normalization “of
increasingly brazen behavior.” Another researcher stated that U.S military
norms prize the avoidance of military escalation, while “[i]n the Chinese
system, the impulse is to not be blamed for a mistake” or to be the person who
reports the message to their political or military leaders.[62] It is worth
noting that hotline usage becomes even more complicated in a world with three
major military powers, where incentives could exist for one actor to exploit
crisis communication between the two other states.

The successful use of hotlines may require that parties share common values
about the risks of foundation models and a mutual belief that CBMs reduce the
risk of unnecessary military escalations. States routinely disagree about the
severity of threats and will pursue technologies to keep their own borders
safe, even at the expense of global security. Other CBMs listed in this
document, such as collaborative red teaming, emergency-response tabletop
games, and incident sharing, can supply the necessary data for assessing the
risk landscape while reinforcing the idea that CBMs will not undermine any
single state’s security. As an area of future study, participants recommended
research on understanding policymaker perceptions about foundation model risks
to international security and ensuring that incentives for participating in
CBMs address country-specific social values.

Incident Sharing

Incident-sharing is a common practice across sectors where public safety is
paramount, such as electric vehicles, cybersecurity, aviation, and healthcare.
Information sharing about security incidents or ‘near misses’ is used to
improve safety and reduce the likelihood of new accidents. With regards to
autonomy in military systems, Michael Horowitz and Paul Scharre have
previously suggested that an “‘international autonomous incidents agreement’
that focuses on military applications of autonomous systems, especially in the
air and maritime environments…would reduce risks from accidental escalation by
autonomous systems, as well as reduce ambiguity about the extent of human
intention behind the behavior of autonomous systems”.[52] This problem is
documented in technology modernization for defense. For example, the
introduction of touchscreen controls to the USS John S McCain, combined with
crew confusion about the different settings associated with these controls,
contributed to the largest maritime accident involving the US Navy in the last
40 years and left 10 sailors dead.[94]

Open-source AI incident-sharing initiatives already exist, such as the AI,
Algorithmic, and Automation Incidents and Controversies (AIAAIC) and AI
Incident Databases.[3, 77] As of April 2023, these open-source databases
primarily featured journalistic investigations, which sometimes include
incidents on generative AI and international security, like the recent
deepfake of the US Pentagon explosion.[55] Participants suggested a comparable
database for international security incidents caused by foundation models,
with a possible focus on unusual vulnerabilities and emergent model behaviors.

Workshop participants raised several questions that remain unresolved. Namely,
it is unclear which model behaviors and misuses would qualify as an
“incident,” the incentives for parties to participate in an incident-sharing
agreement, and how those parties can assure accurate reporting while
respecting intellectual property rights and user privacy. A distinction might
exist between new and dangerous model capabilities versus the large-scale
misuse of the model. The former category could include behaviors linked to
improvements of the model such as the ability to manipulate users or design a
synthetic biochemical agent. The latter category could entail large-scale
misuse campaigns, such as using models to create spam or disinformation.

Other industries resolve these challenges through data anonymization and
encryption, trusted third-party agreements, access controls, NDAs, and
security audits. Some types of incident sharing can leverage existing
professional relationships between labs and could be as simple as hosting an
informal meeting amongst parties. However, incident-sharing may require a
multilateral entity that coordinates incident collection across multiple
parties. Workshop participants noted that an AI incident-response entity could
be analogous to existing Computer Emergency Response Teams (CERT) found in the
international cybersecurity domain.151515For examples on national CERTs,
please see: [93, 18]

## 2 Transparency

AI systems may produce unintended outcomes due to biases in training data,
algorithmic errors, or unforeseen interactions with other systems. To name a
few examples: foundation models used to summarize ISR data can introduce
artifacts into the data that impacts a military response. Plausible outputs
that are actually false, known as "hallucinations",[89] can be difficult to
detect in a fast-paced and high-pressure military environment. Moreover,
labeling practices can contribute to bias by privileging some worldviews over
others, a serious risk for intelligence analysts conducting even routine tasks
like report retrieval and summarization.161616Scholars working within science
and technology studies frequently note that the labeling of datasets reveal
the political preferences and biases of the labellers, which has direct
consequences for the model’s performance. As Kate Crawford and Trevor Paglen
note, “the automated interpretation of images is an inherently social and
political project, rather than a purely technical one”[26] Compounding this
problem is that models do not perform equally well across languages and it is
seldom clear whose values should be reflected in model generations. Finally,
prompt injection attacks, a type of data poisoning and security exploitation,
can alter model outputs.[72] Prompt injection attacks are made easier when the
adversary has access to the training data or model weights.

Some workshop participants also cited the problem of information overload.
Even if accurate, too much information creates its own set of risks. States
are often hesitant to escalate military activity because they are distrustful
of their own predictions and intelligence about adversaries.[87] For example,
machine learning could improve sensors to a point which renders the sea domain
overly transparent and erodes the deterrent capacity of second strike
forces.[59] In general, however, access to accurate information trends towards
international stability.[57, 36, 31] To address the challenges posited above,
workshop participants explored a variety of confidence-building measures.
These are outlined, in brief, below.

### 2.1 Transparency Reports, Model and System Cards

System cards are documents that detail intended use cases, limitations, and
the results of red teaming,171717Similarly, the Data Nutrition Project draws
inspiration from nutrition labels found on food packaging. For an example from
OpenAI, see [39] comparable to documentation practices found in industries
such as aerospace, medicine, and pharmaceuticals.[80, 70] In domestic
settings, proponents of system cards argue that they can help policymakers
better understand the capabilities and limitations of AI systems, informing
oversight and regulation.[70] System cards do not require a third party to
have access to the model itself, meaning that they can introduce transparency
about capabilities while not revealing research details that would enable
reverse-engineering of the model.

For foundation models used in defense domains, system cards should also
include risks associated with human-machine interaction and overreliance,
which can help outside observers interpret a system’s behavior in the event of
an accident or escalation. (It is not always possible to know who or what is
responsible for a system’s strange behavior.) For example, a 2021 UN Security
Council report described a Turkish-made Kargu-2 drone in Libya as an instance
where a lethal autonomous weapons system was deployed in violent conflict, a
description that generated significant controversy in the international
security community and highlighted the uncertainty involved with understanding
the behavior of human-machine teams.[60]

For best effect, system cards should be readable and easily accessible. Many
of today’s system cards are found on code repository websites like Github,
sites which tend not to be frequented by policymakers, and written in formats
that those outside the field of machine learning can sometimes find
inaccessible.

Like other measures found in these proceedings, there are limitations to model
and system cards. Specifically, outside parties can experience difficulty
verifying the results of model and system cards. Limitations can exist in non-
adversarial and non-military contexts, too. Foundation models are often
unavailable to third parties, with the exception of base models made available
by companies that open source their models. When foundation models are made
publicly available, they are often refined through reinforcement learning from
human feedback (RLHF), as seen in the InstructGPT models that power
ChatGPT.[66] In general, publicly available fine-tuned models tend to be safer
due to safety measures implemented after the base model has completed
training.181818Fine-tuning can also be used to make the model less safe, as
actors could fine-tune base models on harmful or otherwise undesirable
information. A recent example is the controversial release of ‘GPT-4Chan,’
where an open source model developed by Eleuther AI called GPT-J was finetuned
on 4Chan forum posts. The model was hosted on the HuggingFace hub and even
contained a model card before HuggingFace made the decision to restrict access
and eventually remove the model. See:
https://huggingface.co/ykilcher/gpt-4chan/discussions/1 Transparency reports,
model and system cards often document the capabilities and limitations of the
base model, making it difficult for third parties to replicate or otherwise
validate the findings in these documents. This limitation is especially
problematic in the context of international security, where adversaries may
have several reasons to exaggerate or under-report the capabilities of their
base models.

For this reason, model and system cards should be supported by other
coordination activities, such as collaborative red teaming (explained in the
section on ‘Cooperation, Collaboration, and Integration.’)

### 2.2 Observation and Verification

Parties can also agree to using empirical methods to observe and verify that
actors are complying with agreements. These techniques do not normally
guarantee full transparency, since states are reluctant to reveal the full
scope of their military capabilities, as explained above. The Biological
Weapons Convention (BWC), Chemical Weapons Convention (CWC), and the Treaty on
the Non-Proliferation of Nuclear Weapons (NPT) all include mechanisms for
third-party verification. Verification is also a key feature in agreements
outside the UN. For example, the Open Skies Treaty allows signatories to fly
observation aircraft to collect data on some military capabilities and
activities.[92]

The success of verification is often dependent on the availability of
detection and monitoring technologies. For this reason, detecting AI misuse in
the international context has emphasized restrictions on hardware, since
software can easily proliferate and evade monitoring. U.S. efforts have so far
focused on constraining the supply of semiconductors and semiconductor
manufacturing materials through export controls (though export controls are
not necessarily confidence-building measures.)191919Laws that restrict the
transfer of goods, technologies, and services to entities outside the country.
Some export controls restrict this transfer to specific actors in another
country (‘end-user control’) or restrict how the technology can be used (‘end-
use’). These controls have mostly targeted two countries: Russia due to its
invasion of Ukraine, and the PRC, which the 2022 U.S. National Security
Strategy identifies as the “only competitor with the intent, and,
increasingly, the capacity to reshape the international order.”202020Page 8
[8] However, algorithmic improvements, fine-tuning, the wide availability of
consumer-grade LLM APIs, and open source alternatives mean that hardware
controls are likely to be insufficient for preventing misuse. Moreover, while
technology denial can constrain the range of choices available to any
particular state, it does not address the problem of states’ behavior with
technologies already at their disposal.

### 2.3 Content Provenance and Watermarking

Content provenance and watermarking methods assist with the disclosure and
detection of AI content and can reduce misperceptions by establishing norms
around the use of generated content during international crises. Provenance
and watermarking methods can improve traceability, alleviate concerns about
the origin of the AI generated or edited content, and promote trust among
parties. If properly vetted against adversarial manipulation, they can also
help states use AI-generated products more confidently, knowing that the
outcomes can be traced back to their source. States can verify if the AI
systems deployed by other parties adhere to agreed-upon guidelines or
restrictions, making it easier to address any potential violations.

Content provenance is an ongoing and politically salient area of research,
development, and adoption. For example, the Coalition for Content Provenance
and Authenticity (C2PA), whose members include Adobe, Microsoft, Intel, BBC,
Sony, and Truepic, is an industry-led initiative that develops technical
standards for establishing the source and history of media content. Born to
address the “prevalence of misleading information online,” the coalition also
offers a set of technical specifications for developers and guidelines to help
users reason through the provenance of media content. As per the C2PA
specifications, provenance methods can be split between "hard" and "soft"
bindings, with the former including methods for applying unique identifiers to
data assets and other cryptographic methods.212121See section 2.4: [15] For
instance, a C2PA manifest using cryptographically-bound provenance can include
information about the origin of a piece of content (such as the AI model and
version used to create it) and edits made to the content over time. A full
survey of AI provenance methods is out of scope for this paper, but is well
worth further research to determine which methods can be applied to improve
state-to-state interactions and how these efforts complement each
other.222222For a list of disclosure methods, please see [78]

One of the most popular and readily-available AI disclosure methods in use
today is "watermarking" (which the C2PA describes as a "soft" binding because
they are more easily undermined in comparison to a "hard" binding.)
Watermarking involves embedding low probability sequences of tokens into the
outputs produced by AI systems, which can serve as a verification mechanism to
confirm the authenticity and integrity of AI generations. Watermarks are
traceable, meaning that they can enable parties to trace AI-generated outcomes
back to their source system, thereby allowing stakeholders to identify which
AI model was used and who was responsible for deploying it. However,
watermarks are also accompanied by a severe restriction: they are not tamper-
proof. For example, bad actors can use "paraphrasing attacks" to remove text
watermarks, spoofing to infer hidden watermark signatures, or even add
watermarks to authentic content.[84]

Because watermarking for large language models is a nascent area of research,
the technique is mostly useful for photorealistic imagery, though watermarking
for AI imagery also faces many limitations. Many AI-image generators that are
publicly available today are already accompanied by watermarking methods, but
these methods can be adversarially circumvented using post-processing methods
on the image. For example, some watermarks for AI images can be removed
through JPEG compression.[58] Due to such limitations, provenance tools should
be frequently red teamed to verify their resilience against adversarial
tampering, and C2PA provides security guidance to protect against attackers
trying to tamper with provenance methods.

Watermarking is publicly popular, as demonstrated by a Vox Media survey that
found that 78 percent of American adults believe that AI-generated media
should be clearly disclosed.[101] Provenance and watermarks for imagery and
audio feature prominently in the recent commitments made by AI companies to
White House.[34] Despite its popularity with the U.S. public, the adoption of
disclosure methods could also be contentious for both commercial and political
reasons. First, small developers may not have the resources to invest in and
apply provenance and watermarking technology. For this reason, open provenance
standards and open sourcing AI detection technologies should be encouraged to
help reduce the cost of security. Second, AI developers may also be reluctant
to use watermarks on imagery for fear of alienating their consumers. Third,
and as seen in the cybersecurity domain, states prefer certain technologies
precisely because the malicious behavior is difficult to trace back to the
belligerent party. States often exploit technical and political ambiguities
about "what counts" as an escalatory military behavior so that they can
continue to engage in conflict below the threshold of war and avoid
attribution.[83] Parties could exploit generative AI for the same reason,
since it is currently unclear how the use of such models are interpreted by
competitor states. While the proliferation of foundation models means that
provenance and watermarking is unlikely to be applied evenly by all
developers, states can commit–even unilaterally–to using such technologies in
diplomatic and security activities.

### 2.4 Policies and Procedures

Rather than providing information about the models and systems that might be
used, states could share information about the processes or procedures for
assuring that they are safe. This could involve sharing baseline testing and
best practices used to verify and validate AI-enabled systems. Since safety
assurance is typically a mutually-aligned goal, some even envision that
development of baseline testing techniques and procedures could become a
collaborative effort among allies and adversaries.[96, 71]

In addition to testing, publishing the policies and procedures for acquiring
and approving AI-enabled systems can also provide confidence that systems are
developed responsibly without divulging intellectual property. This could
involve disclosing the minimum standards for performance such as the safety-
integrity levels that exist for other safety-critical systems.[65] An option
that reveals even less potentially sensitive information is to publicly name
the responsible parties for approving the development, acquisition, and use of
potentially worrisome capabilities. Even simply defining the capabilities that
would require those extra approvals could help provide some clarity. DoD
Directive 3000.09 does not satisfy all advocates, but it does make progress at
providing clarity around some of these issues.

## 3 Cooperation, Collaboration, and Integration

Of course, many of the measures discussed above require AI labs and
governments to collaborate and address the most proximate risks. Parties can
coordinate security activities for the purpose of building trust and learning
from one another. In higher trust environments, these activities can encourage
transparency around military capabilities. In low trust environments, even
simulation exercises can be difficult to organize.

### 3.1 Collaborative Red Teaming Exercises

Workshop participants advocated for collaborative red teaming, in
(coincidental) alignment with the Biden Administration’s recent announcement
on responsible AI innovation, which featured a “public evaluation of
generative AI systems.”232323In May 2023, The Biden Administration announced
that major AI developers, including Anthropic, Google, Hugging
https://www.overleaf.com/project/64b6a3eadcc06995e5dc3666Face, Microsoft,
NVIDIA, OpenAI, and Stability AI, will participate in a public evaluation of
generative AI systems at DEFCON 31’s AI Village, with the aim of determining
whether these models align with the Biden Administration’s “Blueprint for an
AI Bill of Rights” and “AI Risk Management Framework.” See: [54] Collaborative
red-teaming in the United States is currently in development as a public
transparency and multistakeholder activity being held at DefCon 2023. The
event includes participation by several companies, including Google, Hugging
Face, Microsoft, NVIDIA, OpenAI, and Stability AI. Multilateral exercises
similarly exist for cybersecurity, such as the annual Locked Shields exercise
hosted by NATO’s Cyber Defence Centre of Excellence (CCDCOE); in 2022, the
red-team, blue-team exercise was attended by over two thousand cyber experts
from thirty-two countries.[17] Unlike vulnerability discovery for cyber
systems, red teaming foundation models often refers to capability discovery
and require little background in machine learning. In turn, red team
activities can improve emergency preparedness by exposing relevant
stakeholders to the risks associated with foundation models.

### 3.2 Table-Top Exercises

Table-top exercises bring together stakeholders to simulate and discuss their
responses to potential accidents or seemingly intractable solutions, improving
crisis readiness and joint planning, and reducing the likelihood of
misunderstandings during real-world conflicts. TTXs can also enhance
coordination; states can develop a better understanding of each other’s
emergency procedures, identify areas where their response mechanisms or
capabilities need improvement, and share best practices. TTXs between
competitors, such as in track 2 diplomatic contexts, can improve the mutual
understanding of intentions and surface risks or scenarios that might not have
been considered.

Like red-teaming exercises, international fora can act as hosts for these
events. The CCDCOE could integrate foundation models into Locked Shields to
explore the prospect of cyber vulnerabilities, while the United Nations
Institute for Disarmament Research (UNIDIR) could conduct a red-teaming
exercise linked to the aims of the CCW discussions on autonomy in weapons
systems. Because table-top exercises often serve as pedagogical tools, they
can also be thought of as a ‘training and education’ CBM.

### 3.3 Dataset and Evaluation Sharing

Dataset sharing allows for the integration of safety standards across labs.
Not to be confused with incident sharing, AI labs can collaborate on
"refusals" by sharing datasets that focus on identifying and addressing safety
or ethical concerns in AI-generated outputs. In the context of foundation
models, "refusals" refer to instances where the AI system intentionally does
not generate an output or refrains from providing a response to a user query
due to safety or ethical concerns. This may occur when the requested output
could potentially lead to harmful consequences, promote misinformation, or
violate the ethical guidelines and policies set by the AI developers. Sharing
such datasets can contribute to the development of more robust, aligned, and
responsible AI systems.

In the area of international security, these datasets can contain information
related to dual-use scientific information and that are legal to share across
parties, such as in domains like chemical, biological, radiological, and
nuclear science (CBRN). Red teaming has demonstrated that the interaction
between LLMs and CBRN can introduce new proliferation pathways, potentially
empowering non-state threat actors.242424See page 12, 16: [44] Sharing
datasets allows AI labs to establish and integrate common benchmarks for
evaluating the effectiveness of refusal mechanisms in LLMs. Datasets and
evaluation sharing can also help to improve red teaming in small labs that do
not have the resources to contract a large group of external experts.

There are limitations to dataset and evaluation sharing, especially as they
relate to issues like national security. These limitations involve the
regulation of dual-use items, the technical shortcomings of refusals, and
expected future improvements in foundation models. Some scientific information
is regulated to prevent the spread of information that can be used for weapons
proliferation. For example, the U.S. International Traffic in Arms Regulation
(ITAR) establishes controls on the import and export of technologies on the
United States Munitions List (USML). The USML includes some dual-use
technologies, and the regulations include rules surrounding the distribution
of information about those technologies to foreign actors,[37] adding a layer
of complexity to red teaming and the development of safety mitigations. As a
result, many labs avoid this problem by red teaming information that is not
controlled.

On a more general note, it is not yet clear if models will "discover" new
biochemical compounds in the future and, if so, whether these discoveries may
introduce new security vulnerabilities. Refusals target capabilities that have
already been discovered, meaning they are a powerful but limited solution in
the domain of biochemical discovery.

Despite these limitations, there are still benefits to sharing datasets that
contain public, but sometimes difficult to find, information. Because
information contained in these datasets are public, trialing dataset sharing
today versus later is a comparatively lower risk endeavor, the main obstacle
being the leveraging of scientific talent for capability discovery. In
comparison, future data-sharing may be a higher risk activity if foundational
models take on a greater role in biochemical discovery. And like all CBMs, the
sharing of datasets demonstrates a commitment to transparency and responsible
AI development, which can contribute to building trust among AI labs across
countries, policymakers, and the broader public.

## 4 Conclusion

States are often reluctant to limit their technical capabilities. This is
especially true in the face of heightened international competition and when
confronting uncertainties around a new technology.252525For example, France’s
Defence Ethics Committee advised the continuation of research into autonomy
and weapons systems, citing, among other reasons, the need to “counter enemy
development of LAWS; and…to be able to defend ourselves against this type of
weapon in the likely event of their use by an enemy State or terrorist group
against our troops or population.” [74] However, military interest in AI, and
increasingly in foundation models and generative AI capabilities, has
intensified the urgency of establishing an international code of conduct for
state behavior, as the Political Declaration on Military Uses of Artificial
Intelligence and Autonomy illustrates.[50] As Rebecca Hersman notes in her
work on the potentially new escalation dynamics caused by emerging
technologies, “[u]nlike traditional concepts of escalation, which suggest
linear and somewhat predictable patterns from low-level crisis to all-out
nuclear war, escalatory pathways in this new era of strategic competition will
be less predictable.”[47] While CBMs are not a complete solution for the
international system’s various insecurities, they do offer a set of options
for reducing the likelihood of violent conflict caused by misinterpretation
and miscalculation.

Though this workshop was primarily designed for thinking about solutions,
participants identified a number of risks that could undermine the feasibility
of CBM adoption. First, workshop attendees highlighted the information
disparity between technologists and policymakers. The speed of politics–and
diplomacy, in particular–often lags behind the speed of capability
development, compounding the challenges of establishing appropriate CBMs in
step with emerging tools. Policymakers may struggle to negotiate or provide
assurances to their counterparts in other countries if they are unaware of the
capabilities that exist within their borders.

Participants called for an increase in candid multistakeholder conversations
to alleviate this problem. While the CBMs for AI workshop served as a space to
discuss the impact of foundation models on international security, multi-
stakeholder opportunities have predominantly been sporadic and reliant on the
initiative of voluntary contributors to organize what are often time-consuming
meetings.

Second, there are different options with respect to who would coordinate
implementation and adoption of these CBMs, each with tradeoffs and drawbacks.
For example, incident sharing demands not just funding, but also a reliable
third party with sufficient staff to manage intake and ensure database
quality. Participants suggested a variety of mechanisms to address this
coordination issue, ranging from the addition of new offices within existing
government agencies like the U.S. Office of Science and Technology Policy and
parallel agencies in other countries, to forming international institutions
that would oversee compliance and distribute benefits for participating states
and labs. Two sandbox groups independently noted the oft-used and potentially
problematic analogy of an ‘IAEA for AI’ as a comparable entity. Workshop
participants suggested that states that abide by monitoring and verification
norms could gain access to data pools, with initial access granted to testing
and evaluation (T&E) infrastructure, followed by access to data and subsidies
to support the T&E infrastructure of fledgling companies, further driving
innovation and progress in the field and especially in safety research.
However, for countries that are already data-rich, such as the United States
and China, incentives focused on data sharing may be insufficient.

Third, it was unclear which incentives would encourage discussion and adoption
of CBMs for different states. Disparities in military and technological
capabilities among states may create resistance to CBMs, as some countries may
believe that CBMs will disproportionately disadvantage them and data sharing
incentives may be insufficient to overcome this perception. This belief may
make a commitment to more intrusive monitoring and verification politically
intractable in the near-term. There is an established literature in
international relations that addresses the rational, normative, and
psychological basis for participating in international agreements and arms
control;[56, 11, 13, 2, 100, 42, 7, 63, 20, 1, 68, 27, 61, 81, 48] this
literature provides a foundation for the development of incentives that could
target the adoption of CBMs for foundation models.

Despite the challenge of transnational coordination, confidence-building
measures that target foundation models are important for international
stability. Some of the suggestions in this document are already being
developed as consumer protections, meaning that much of the remaining work
will be on persuading parties—both private and public—that the adoption of
CBMs will be beneficial for international security, Equally promising are the
growing calls for international coordination on AI by governments, technology
companies, and civil society in the face of state-to-state competitive
tensions.[49, 22, 97] These calls carve an opening for increased transnational
dialogue between civil societies and scientific communities. As non-government
actors become increasingly responsible for steering technologies that have
global ramifications, many of the sociotechnical solutions that reduce
misperception will need to be implemented at the technical layer and in
collaboration with private actors.

## Acknowledgements

The authors would like to thank Wyatt Hoffman, Lauren Kahn, Philip Reiner, and
Joel Parish for their valuable feedback on earlier versions of this
manuscript.

## References

  * [1] Kenneth L. Adelman  “Arms Control With and Without Agreements”  In _Foreign Affairs_ , 1984
  * [2] Emanuel Adler  “The Emergence of Cooperation: National Epistemic Communities and the International Evolution of the Idea of Nuclear Arms Control”  In _International Organization_ 46.1 [MIT Press, University of Wisconsin Press, Cambridge University Press, International Organization Foundation], 1992, pp. 101–145  JSTOR: 2706953
  * [3] “AIAAIC”, https://www.aiaaic.org/home
  * [4] Bobby Allyn  “Deepfake Video of Zelenskyy Could Be ’tip of the Iceberg’ in Info War, Experts Warn”  In _NPR_ , 2022
  * [5] “Core Views on AI Safety: When, Why, What, and How”  In _Anthropic_ , https://www.anthropic.com/index/core-views-on-ai-safety
  * [6] Yuntao Bai et al.  “Constitutional AI: Harmlessness from AI Feedback”  arXiv, 2022  DOI: 10.48550/arXiv.2212.08073
  * [7] Richard K. Betts  “Systems for Peace or Causes of War? Collective Security, Arms Control, and the New Europe”  In _International Security_ 17.1 The MIT Press, 1992, pp. 5–43  DOI: 10.2307/2539157
  * [8] Joe Biden  “2022 US National Security Strategy”, 2022
  * [9] Steven Bills et al.  “Language Models Can Explain Neurons in Language Models”, 2023
  * [10] Rishi Bommasani et al.  “On the Opportunities and Risks of Foundation Models”  arXiv, 2022  DOI: 10.48550/arXiv.2108.07258
  * [11] Bernard Brodie  “On the Objectives of Arms Control”  In _International Security_ 1.1 The MIT Press, 1976, pp. 17–36  DOI: 10.2307/2538574
  * [12] “Building Trust in Human-Machine Teams”  In _Brookings_ , https://www.brookings.edu/articles/building-trust-in-human-machine-teams/
  * [13] Hedley Bull  “Arms Control and World Order”  In _International Security_ 1.1 The MIT Press, 1976, pp. 3–16  DOI: 10.2307/2538573
  * [14] Bureau of Arms Control  “Political Declaration on Responsible Military Use of Artificial Intelligence and Autonomy - United States Department of State”, 2023
  * [15] “C2PA Security Considerations :: C2PA Specifications”  In _C2PA_ , https://c2pa.org/specifications/specifications/1.0/security/Security_Considerations.html#_provenance_model
  * [16] “Overview - C2PA”  In _C2PA_ , https://c2pa.org/
  * [17] “Locked Shields”  In _CCDCOE_ , https://ccdcoe.org/exercises/locked-shields/
  * [18] “CERT-EU – Computer Emergency Response Team | European Union”, https://european-union.europa.eu/institutions-law-budget/institutions-and-bodies/search-all-eu-institutions-and-bodies/computer-emergency-response-team-eu-institutions-bodies-and-agencies-cert-eu_en
  * [19] “Chapter 4 - NSCAI Final Report”
  * [20] Abram Chayes  “An Inquiry into the Workings of Arms Control Agreements”  In _Harvard Law Review_ 85.5 The Harvard Law Review Association, 1972, pp. 905–969  DOI: 10.2307/1339933
  * [21] ChrisHMSFT  “Transparency Note for Azure OpenAI - Azure Cognitive Services”, https://learn.microsoft.com/en-us/legal/cognitive-services/openai/transparency-note, 2023
  * [22] Joseph Clark  “DOD Committed to Ethical Use of Artificial Intelligence”  In _U.S. Department of Defense_ , https://www.defense.gov/News/News-Stories/Article/Article/3429864/dod-committed-to-ethical-use-of-artificial-intelligence/https%3A%2F%2Fwww.defense.gov%2FNews%2FNews-Stories%2FArticle%2FArticle%2F3429864%2Fdod-committed-to-ethical-use-of-artificial-intelligence%2F, 2023
  * [23] “Close Calls with Nuclear Weapons”, 2015
  * [24] Michael K. Cohen, Marcus Hutter and Michael A. Osborne  “Advanced Artificial Agents Intervene in the Provision of Reward”  In _AI Magazine_ 43.3, 2022, pp. 282–293  DOI: 10.1002/aaai.12064
  * [25] “Confidence-Building Measures | Cross-Strait Security Initiative | CSIS”, https://www.csis.org/programs/international-security-program/isp-archives/asia-division/cross-strait-security-1
  * [26] Kate Crawford and Trevor Paglen  “Excavating AI”  In _Excavating AI_ , https://excavating.ai
  * [27] Ivo H. Daalder  “The Future of Arms Control”  In _Survival_ 34.1 Routledge, 1992, pp. 51–73  DOI: 10.1080/00396339208442630
  * [28] Ron DePasquale  “Civilian Planes Shot Down: A Grim History”  In _The New York Times_ , 2020
  * [29] Jeffrey Ding and Jenny Xiao  “Recent Trends in China’s Large Language Model Landscape”  In _Centre for the Governance of AI_ , 2023
  * [30] Dheeru Dua et al.  “DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs”  arXiv, 2019  DOI: 10.48550/arXiv.1903.00161
  * [31] Charles A. Duelfer and Stephen Benedict Dyson  “Chronic Misperception and International Conflict: The US-Iraq Experience.”  In _International Security_ 36.4, 2011, pp. 73–100
  * [32] Nelson Elhage et al.  “Toy Models of Superposition”  arXiv, 2022  DOI: 10.48550/arXiv.2209.10652
  * [33] Paul England et al.  “AMP: Authentication of Media via Provenance”  arXiv, 2020  DOI: 10.48550/arXiv.2001.07886
  * [34] “Ensuring Safe, Secure, and Trustworthy AI”
  * [35] “Evals”, OpenAI, 2023
  * [36] James D. Fearon  “Rationalist Explanations for War”  In _International Organization_ 49.3, 1995, pp. 379–414  JSTOR: 2706903
  * [37] “International Traffic in Arms Regulations: U.S. Munitions List Categories I, II, and III”  In _Federal Register_ , https://www.federalregister.gov/documents/2020/01/23/2020-00574/international-traffic-in-arms-regulations-us-munitions-list-categories-i-ii-and-iii, 2020
  * [38] “Formal Investigation into the Circumstances Surrounding the Downing of Iran Air Flight 655 on 3 July 1988”, 1988, pp. 153
  * [39] “DallE-2 System Card”  In _GitHub_ , https://github.com/openai/dalle-2-preview/blob/main/system-card.md
  * [40] Amelia Glaese et al.  “Improving Alignment of Dialogue Agents via Targeted Human Judgements”  arXiv, 2022  DOI: 10.48550/arXiv.2209.14375
  * [41] Josh A. Goldstein et al.  “Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigations”  arXiv, 2023  DOI: 10.48550/arXiv.2301.04246
  * [42] Rose Gottemoeller  “Rethinking Nuclear Arms Control”  In _The Washington Quarterly_ 43.3 Routledge, 2020, pp. 139–159  DOI: 10.1080/0163660X.2020.1813382
  * [43] Leigh Mc Gowran  “Adobe Launches Open-Source Tools to Tackle Visual Misinformation”  In _Silicon Republic_ , https://www.siliconrepublic.com/enterprise/adobe-digital-misinformation-cai-developer-tools, 2022
  * [44] “GPT-4 System Card”  OpenAI, 2023
  * [45] “H-020-1: USS Vincennes Tragedy”, http://public1.nhhcaws.local/content/history/nhhc/about-us/leadership/director/directors-corner/h-grams/h-gram-020/h-020-1-uss-vincennes-tragedy–.html
  * [46] Dan Hendrycks  “Measuring Massive Multitask Language Understanding”, 2023
  * [47] Rebecca Hersman  “Wormhole Escalation in the New Nuclear Age”  In _Texas National Security Review_ , https://tnsr.org/2020/07/wormhole-escalation-in-the-new-nuclear-age/, 2020
  * [48] Stephen Herzog  “After the Negotiations: Understanding Multilateral Nuclear Arms Control”  In _Yale Graduate School of Arts and Sciences Dissertations_ , 2021
  * [49] Kathleen Hicks  “Opinion | What the Pentagon Thinks About Artificial Intelligence”  In _POLITICO_ , https://www.politico.com/news/magazine/2023/06/15/pentagon-artificial-intelligence-china-00101751, 2023
  * [50] Michael C. Horowitz and Lauren Kahn  “Leading in Artificial Intelligence through Confidence Building Measures”  In _The Washington Quarterly_ 44.4 Routledge, 2021, pp. 91–106  DOI: 10.1080/0163660X.2021.2018794
  * [51] Michael C. Horowitz, Lauren Kahn and Casey Mahoney  “The Future of Military Applications of Artificial Intelligence: A Role for Confidence-Building Measures?”  In _Orbis_ 64.4, 2020, pp. 528–543  DOI: 10.1016/j.orbis.2020.08.003
  * [52] Michael C. Horowitz and Paul Scharre  “AI and International Stability: Risks and Confidence-Building Measures”, https://www.cnas.org/publications/reports/ai-and-international-stability-risks-and-confidence-building-measures, 2021
  * [53] Eric Horvitz  “On the Horizon: Interactive and Compositional Deepfakes”  In _INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION_ , 2022, pp. 653–661  DOI: 10.1145/3536221.3558175
  * [54] The White House  “FACT SHEET: Biden-Harris Administration Announces New Actions to Promote Responsible AI Innovation That Protects Americans’ Rights and Safety”  In _The White House_ , https://www.whitehouse.gov/briefing-room/statements-releases/2023/05/04/fact-sheet-biden-harris-administration-announces-new-actions-to-promote-responsible-ai-innovation-that-protects-americans-rights-and-safety/, 2023
  * [55] “Incident 543: Deepfake of Explosion Near US Military Administration Building Reportedly Causes Stock Dip”, https://incidentdatabase.ai/cite/543/, 2020
  * [56] Robert Jervis  “Arms Control, Stability, and Causes of War”  In _Political Science Quarterly_ 108.2 [Academy of Political Science, Wiley], 1993, pp. 239–253  DOI: 10.2307/2152010
  * [57] Robert Jervis  “Cooperation Under the Security Dilemma”  In _World Politics_ 30.2, 1978, pp. 167–214
  * [58] Zhengyuan Jiang, Jinghuai Zhang and Neil Zhenqiang Gong  “Evading Watermark Based Detection of AI-Generated Content”  arXiv, 2023  DOI: 10.48550/arXiv.2305.03807
  * [59] James Johnson  “Artificial Intelligence, Drone Swarming and Escalation Risks in Future Warfare”  In _The RUSI Journal_ 165.2 Routledge, 2020, pp. 26–36  DOI: 10.1080/03071847.2020.1752026
  * [60] Will Knight  “Autonomous Weapons Are Here, but the World Isn’t Ready for Them”  In _Wired_
  * [61] Sarah E. Kreps  “The Institutional Design of Arms Control Agreements”  In _Foreign Policy Analysis_ 14.1, 2018, pp. 127–147  DOI: 10.1093/fpa/orw045
  * [62] Howard LaFranchi  “US-China Conundrum: Can Hotline Diplomacy Work If Trust Isn’t a Goal?”  In _Christian Science Monitor_ , 2023
  * [63] “Arms Control: Cooperative Security in a Changing Environment”  Boulder, Colo: Lynne Rienner Publishers, 2002
  * [64] Jan Leike, John Schulman and Jeffrey Wu  “Our Approach to Alignment Research”, https://openai.com/blog/our-approach-to-alignment-research, 2022
  * [65] Andrew J. Lohn  “Estimating the Brittleness of AI: Safety Integrity Levels and the Need for Testing Out-Of-Distribution Performance”  arXiv, 2020  DOI: 10.48550/arXiv.2009.00802
  * [66] Ryan Lowe and Jan Leike  “Aligning Language Models to Follow Instructions”, https://openai.com/research/instruction-following, 2022
  * [67] “Microsoft Turing Academic Program (MS-TAP)”  In _Microsoft Research_
  * [68] Steven E. Miller  “Politics over Promise: Domestic Impediments to Arms Control”  In _International Security_ 8.4 The MIT Press, 1984, pp. 67–90  DOI: 10.2307/2538563
  * [69] Dan Milmo and Alex Hern  “UK to Invest £900m in Supercomputer in Bid to Build Own ‘BritGPT”’  In _The Guardian_ , 2023
  * [70] Margaret Mitchell et al.  “Model Cards for Model Reporting”  In _Proceedings of the Conference on Fairness, Accountability, and Transparency_ , 2019, pp. 220–229  DOI: 10.1145/3287560.3287596
  * [71] Forrest E. Morgan et al.  “Military Applications of Artificial Intelligence: Ethical Concerns in an Uncertain World”, 2020
  * [72] “Exploring Prompt Injection Attacks”  In _NCC Group Research Blog_ , https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/, 2022
  * [73] Philip Oltermann  “European Politicians Duped into Deepfake Video Calls with Mayor of Kyiv”  In _The Guardian_ , 2022
  * [74] “Opinion On The Integration Of Autonomy Into Lethal Weapon Systems”, 2021, pp. 5–6
  * [75] “Palantir Artificial Intelligence Platform”  In _Palantir_ , https://www.palantir.com/platforms/aip/
  * [76] “Papers with Code - MMLU Benchmark (Multi-task Language Understanding)”, https://paperswithcode.com/sota/multi-task-language-understanding-on-mmlu
  * [77] “AI Incidents Database”  In _Partnership on AI_ , https://partnershiponai.org/workstream/ai-incidents-database/
  * [78] “PAI’s Responsible Practices for Synthetic Media”  In _Partnership on AI - Synthetic Media_ , https://syntheticmedia.partnershiponai.org/
  * [79] Iona Puscas  “Confidence-Building Measures for Artificial Intelligence: A Framing Paper”  In _United Nations Institute for Disarmament Research_ , 2022
  * [80] Inioluwa Deborah Raji et al.  “Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing”  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , FAT* ’20  New York, NY, USA: Association for Computing Machinery, 2020, pp. 33–44  DOI: 10.1145/3351095.3372873
  * [81] Andrew William Reddie  “Governing Insecurity: Institutional Design, Compliance, and Arms Control”, 2019
  * [82] “Repository of Military Confidence-Building Measures – UNODA”
  * [83] Thomas Rid and Ben Buchanan  “Attributing Cyber Attacks”  In _The Journal of Strategic Studies_ 38.1-2, 2015, pp. 4–37  DOI: 10.1080/01402390.2014.977382
  * [84] Vinu Sankar Sadasivan et al.  “Can AI-Generated Text Be Reliably Detected?”  arXiv, 2023  DOI: 10.48550/arXiv.2303.11156
  * [85] “Donovan: AI-powered Decision-Making for Defense. | Scale AI”  In _ScaleAI_ , https://scale.com/donovan
  * [86] Rohin Shah et al.  “Goal Misgeneralization: Why Correct Specifications Aren’t Enough For Correct Goals”  arXiv, 2022  DOI: 10.48550/arXiv.2210.01790
  * [87] Glenn Herald Snyder  “Deterrence and Defense”  Princeton University Press, 1961
  * [88] Rishi Sunak  “Integrated Review Refresh 2023”, 2023, pp. 56
  * [89] Adrian Tam  “A Gentle Introduction to Hallucinations in Large Language Models”  In _MachineLearningMastery.com_ , 2023
  * [90] “The Convention on Certain Conventional Weapons – UNODA”
  * [91] Helen Toner  “What Are Generative AI, Large Language Models, and Foundation Models?”  In _Center for Security and Emerging Technology_ , 2023
  * [92] “Treaty on Open Skies”, 1992
  * [93] “US-CERT (United States Computer Emergency Readiness Team) - Glossary | CSRC”, https://csrc.nist.gov/glossary/term/us_cert
  * [94] “NTSB Accident Report on Fatal 2017 USS John McCain Collision off Singapore”  In _USNI News_ , 2019
  * [95] Alexander Ward, Matt Berg and Lawrence Ukenye  “Shaheen to Admin: Get Me the Black Sea Strategy”  In _POLITICO_ , https://www.politico.com/newsletters/national-security-daily/2023/03/21/shaheen-to-admin-get-me-the-black-sea-strategy-00088048, 2023
  * [96] Alexa Wehsener, Andrew W. Reddie, Leah Walker and Philip Reiner  “AI-NC3 Integration in an Adversarial Context: Strategic Stability Risks and Confidence Building Measures”  In _Institute for Security and Technology_ , https://securityandtechnology.org/virtual-library/reports/ai-nc3-integration-in-an-adversarial-context-strategic-stability-risks-and-confidence-building-measures/
  * [97] Shen Weiduo  “OpenAI CEO Calls for Global Cooperation on AI Regulation, Says ‘China Has Some of the Best AI Talent in the World’ - Global Times”  In _Global Times_ , https://www.globaltimes.cn/page/202306/1292326.shtml, 2023
  * [98] Bill Whitaker  “When Russian Hackers Targeted the U.S. Election Infrastructure”  In _CBS News_ , https://www.cbsnews.com/news/when-russian-hackers-targeted-the-u-s-election-infrastructure/, 2018
  * [99] Albert Wolf  “Backing Down: Why Red Lines Matter in Geopolitics”  In _Modern War Institute_ , https://mwi.westpoint.edu/geopolitical-costs-red-lines/, 2016
  * [100] Jon Brook Wolfsthal  “Why Arms Control?”  In _Daedalus_ 149.2 [American Academy of Arts & Sciences, The MIT Press], 2020, pp. 101–115  JSTOR: 48591315
  * [101] Edwin H. Wong  “What Americans Are Really Excited about — and Scared of — When It Comes to AI”  In _Vox Media_ , https://www.voxmedia.com/2023/6/26/23769834/what-americans-are-really-excited-about-and-scared-of-when-it-comes-to-ai, 2023
  * [102] Jeff Wu et al.  “Recursively Summarizing Books with Human Feedback”  arXiv, 2021  DOI: 10.48550/arXiv.2109.10862
  * [103] Denny Zhou et al.  “Least-to-Most Prompting Enables Complex Reasoning in Large Language Models”  arXiv, 2023  DOI: 10.48550/arXiv.2205.10625
  * [104] Daniel Ziegler et al.  “Adversarial Training for High-Stakes Reliability”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 9274–9286