# AGI labs need an internal audit function

Jonas Schuett
Centre for the Governance of AI
Contact<EMAIL_ADDRESS>
###### Abstract

The paper argues that organizations that have the stated goal of building
artificial general intelligence (AGI) need an internal audit function. First,
it explains what internal audit is: a specific team that performs an ongoing
assessment of an organization’s risk management practices and reports directly
to the board of directors, while being organizationally independent from
senior management. Next, the paper discusses the main benefits of internal
audit for AGI labs: it can make their risk management practices more
effective; ensure that the board of directors has a more accurate view of the
current level of risk and the effectiveness of the lab’s risk management
practices; signal that the lab follows best practices in corporate governance;
and serve as a contact point for whistleblowers. However, AGI labs should be
aware of a number of limitations: internal audit adds friction; there is not
much empirical evidence in support of the above-mentioned benefits; the
benefits depend on the people involved and their ability and willingness to
identify ineffective risk management practices; setting up and maintaining an
internal audit team is costly; and it should only be seen as an additional
“layer of defense”, not a silver bullet against emerging risks from AI.
Finally, the paper provides a blueprint for how AGI labs could set up an
internal audit team and suggests concrete things the team would do on a day-
to-day basis. These suggestions are based on the International Standards for
the Professional Practice of Internal Auditing Standards. In light of rapid
progress in AI research and development, AGI labs need to professionalize
their risk management practices. Instead of “reinventing the wheel”, they
should follow existing best practices in corporate governance. This will not
be sufficient as they approach AGI, but they should not skip this obvious
first step.

##  1 Introduction

A number of leading AI companies, including OpenAI, Google DeepMind, and
Anthropic, have the stated goal of building artificial general intelligence
(AGI)—AI systems that achieve or exceed human performance across a wide range
of cognitive tasks. In pursuing this goal, they may develop and deploy AI
systems that pose particularly significant risks. For example, as models get
scaled up, new capabilities can emerge unpredictably [41, 127],111Note that a
recent paper expressed doubts about this phenomenon [113]. some of which might
be dangerous [119]. Capabilities that would be particularly concerning include
situational awareness [29, 95, 99], power-seeking [124, 25, 123, 80, 99],
persuasion [86, 99], and long-horizon planning [99]. It has been argued that
some capabilities could lead to catastrophic outcomes [25, 95]. Against this
background, managing the risks of increasingly capable AI systems is an
important and urgent challenge.

To rise to this challenge, AGI labs need to strengthen their risk management
practices. In some cases, they need to develop new solutions themselves, such
as model type-specific risk taxonomies [128]. In other cases, it would be more
appropriate to learn from other industries. Instead of “reinventing the
wheel”, they should apply existing best practices to an AI context. One such
practice is internal audit. Note that I use “internal audit” as a technical
term here. It refers to a specific team that is organizationally independent
from senior management. This team performs an ongoing assessment of the
organization’s risk management practices and reports its findings directly to
the board of directors, especially if they discover ineffective practices. In
doing so, the board of directors gets more objective information about the
organization’s risk management practices. In a recent expert survey (_N_ =
51), 86% of respondents agreed or strongly agreed that AGI labs should have an
internal audit team [116]). Internal audit is also very common in many other
industries [87, 46, 3]. Yet, based on public information, AGI labs do not seem
to have internal audit teams.

Although there is some literature on the intersection of AI and internal
audit, most of it is about how internal auditors can use AI [30, 74]. There is
only limited work on how AI companies can use internal audit. The most
relevant piece is an article that applies the Three Lines of Defense model, a
risk management framework where internal audit serves as the third line, to an
AI context [114]. Here, I conduct a more in-depth analysis of the third line.
Besides that, the Institute of Internal Auditors (IIA) has published a three-
part series, in which they propose an AI auditing framework [55, 56, 58]. In
the first two parts, they specifically discuss the role of internal audit,
though the relevant passages are rather short. There is only one study that
mentions internal audit in the context of AGI labs [116], but only as part of
a broader expert survey on best practices in AGI safety and governance. There
does not seem to be an investigation of the benefits and limitations of
internal audit at AGI labs. There are also no practical recommendations for
how AGI labs could set up an internal audit team. Against this background, the
paper seeks to answer two research questions (RQs):

  * •

RQ1: Should AGI labs set up an internal audit team? In particular, what are
the key benefits and limitations of internal audit at AGI labs?

  * •

RQ2: How could AGI labs set up an internal audit team and what exactly would
that team do?

Figure 1: Organizational context of internal audit

The paper has two areas of focus. First, it focuses on AGI labs. I define
“AGI” as AI systems that reach or exceed human performance across a wide range
of cognitive tasks [116].111There is no generally accepted definition of the
term “AGI”. According to Goertzel [43], the term was first used by Gubrud [51]
in the article “Nanotechnology and international security”. It was popularized
through the book “Artificial general intelligence” edited by Goertzel and
Pennachin [45]. For more information on how to make this definition more
concrete, I refer to the relevant literature [44, 92, 12, 94]. Different
definitions emphasize different elements. For example, in their charter,
OpenAI uses a definition that focuses on economic value: “highly autonomous
systems that outperform humans at most economically valuable work” [98]. But
note that they have recently used a simplified definition: “AI systems that
are generally smarter than humans” [2]. The term “AGI” is related to the terms
“strong AI” [118], “superintelligence” [17, 19], and “transformative AI” [50].
I define “AGI labs” as organizations that have the stated goal of building AGI
[116]. This currently includes OpenAI [98, 2], Google DeepMind [49, 75, 81,
103], and Anthropic.222Anthropic does not use the term “AGI” in their public
communication. They mainly talk about “frontier models” or “transformative AI
systems” [6]. However, Anthropic is generally considered to pursue the goal of
building AGI. However, my analysis and recommendations might also apply to
other AI companies (e.g. Microsoft and Meta). If they continue doing similar
research (e.g. training very large models), then many of the same risks will
emerge regardless of whether or not they have the stated goal of building AGI.
They might also become increasingly focused on building AGI over time. Second,
the paper focuses on internal audit’s ability to reduce risks. By “risk”, I
mean the “combination of the probability of occurrence of harm and the
severity of that harm” [68].333Note that there are other definitions of risk
[67, 28]. In terms of severity, I focus on adverse effects on large groups of
people and society as a whole, especially threats to their lives and physical
integrity. I am less interested in financial losses and risks to organizations
themselves (e.g. litigation or reputation risks). In terms of likelihood, I
also consider low-probability, high-impact risks, sometimes referred to as
“black swans” [121, 11, 78]. The two main sources of harm (“hazards”) from AI
are accidents [4, 10] and cases of misuse [21, 125, 47, 5, 52].

The paper proceeds as follows. Section 2 explains what internal audit is.
Section 3 discusses the main benefits and limitations of internal audit at AGI
labs. Section 4 provides a blueprint for how AGI labs could set up an internal
audit team. Section 5 suggests concrete things that team would do on a day-to-
day basis. Section 6 concludes with questions for further research.

##  2 What is internal audit?

In this section, I define the term “internal audit” (Section 2.1), discuss
similarities and differences between internal audit and related governance
mechanisms (Section 2.2), and show that internal audit is a well-established
governance mechanism (Section 2.3).

###  2.1 Terminology

The term “internal audit” might cause confusion because it can be interpreted
in two different ways.

First interpretation. “Internal audit” can refer to a specific team that
performs a specific activity. The internal audit team consists of internal
audit professionals, typically Certified Internal Auditors [63], though a
certificate is not required in any way. The team is led by a chief audit
executive (CAE) who reports directly to the board of directors, more
specifically their audit committee. The team is organizationally independent
from senior management, i.e. senior management cannot interfere with their
work. Figure 1 illustrates this organizational context. The internal audit
activity involves an ongoing assessment of the organization’s risk management
practices. The findings of that assessment are reported to the board of
directors. In doing so, internal audit provides independent and objective
assurance about the organization’s risk management practices. Against this
background, you might also call it an “assurance team”, but “internal audit”
is the technical term.444The Institute of Internal Auditors (IIA) defines
internal audit as “an independent, objective assurance and consulting activity
designed to add value and improve an organization’s operations. It helps an
organization accomplish its objectives by bringing a systematic, disciplined
approach to evaluate and improve the effectiveness of risk management,
control, and governance processes” [64].

Second interpretation. “Internal audit” can also refer to an audit that is
done internally, as opposed to a third-party audit. The basic idea behind
auditing is that an auditor (e.g. an audit firm) evaluates whether the object
of the audit (e.g. an AI system or the way an organization is governed)
complies with predefined audit criteria (e.g. regulations or standards) [53,
22, 90]. For example, during a financial audit, a certified accounting firm
like KPMG or EY evaluates whether a company has prepared its financial
statement in accordance with a recognized accounting standard like IFRS or US
GAAP. Similarly, during a model audit, an audit firm like ORCAA might evaluate
a model against a specific fairness benchmark. An audit is external if the
auditor is not part of the organization that is audited [107, 109]. It is
internal if the auditor is part of that organization [108].

Difference between the two interpretations. The two interpretations overlap,
but there are also important differences. Internal audit as a specific team
and activity has a precisely defined scope (assessing the organization’s risk
management practices), purpose (providing independent and objective assurance
to the board), and position within the organization (it is organizationally
independent from management and reports directly to the board). In contrast,
internal audit as the opposite of external audit is less precisely defined. It
could assess models, applications, or governance structures [91], which might
also include the organization’s risk management practices. It can also provide
assurance to the board, but it does not have to. For example, it could also
assess whether the organization’s risk management practices comply with the
requirements set out in the EU AI Act [115] and report their findings to the
legal and compliance team, or it could try to identify ways in which an
unreleased model can be misused and report their findings to the research
lead. Finally, the term often refers to an activity, not a team. This activity
is usually not ongoing, and the people who conduct the audit can have varying
relationships with other stakeholders.

For the purpose of this paper, I follow the first interpretation: I mean the
specific team that performs a specific activity, not the opposite of a third-
party audit.

###  2.2 Related governance mechanisms

Below, I discuss the similarities and differences between internal audit and
three related governance mechanisms: third-party audits, ethics boards, and
institutional review boards (IRBs).

Third-party audits. In the previous section, I have discussed the similarities
and differences between different interpretations of internal audit (Section
2.1). Here, I contrast internal audit with external audit. In terms of
similarities, they both provide independent and objective information to the
board of directors—they are “assurance providers”. They could both assess the
effectiveness of the lab’s risk management practices. While this is a defining
characteristic of internal audit, not all third-party governance audits do
this, even if it is recommended [91, 89]. Finally, internal and external
auditors are both expected to comply with certain codes of conducts and
professional standards [62]. In terms of differences, internal audit is an
ongoing activity, while external audit is typically, though not always, a
snapshot in time. For that reason, and because AGI labs may be reluctant to
share certain information with external auditors (e.g. sensitive information
about ongoing research projects), internal auditors have access to more
information. As a consequence, they have a better understanding of the inner
workings of the company, including less tangible factors like norms and
culture. Taken together, these differences suggest that AGI labs should see
external audit as a complement to internal audit, not a substitute [114]. This
is also reflected in the Three Lines of Defense model, which highlights the
need for both [60].

Ethics boards. An ethics board is a collective body intended to promote an
organization’s ethical behavior. Some AI companies already have an ethics
board. One of the most prominent examples is Meta’s Oversight Board which
reviews the content on Facebook and Instagram [101, 77, 129]. But there have
also been a number of failures. For example, Google’s Advanced Technology
External Advisory Council (ATEAC) was shut down only one week after its
announcement [126]. Ethics boards can take various forms and shapes. They can
have different responsibilities (e.g. advising the board of directors,
overseeing model releases, or interpreting ethics principles), their decisions
may or may not be binding, and they can be part of the lab or a separate legal
entity [117]. In principle, a lab could design an ethics board that is similar
to internal audit. The board would have to be organizationally independent
from senior management and review the lab’s risk management practices [117].
However, there do not seem to be any ethics boards that are set up in that
way. And even if a lab decides to design an ethics board that resembles
internal audit, it should only complement internal audit, not replace it. It
should be seen as an additional layer of assurance [117].

IRBs. An IRB is a committee, typically within a university, that reviews the
methods of proposed research on human subjects to protect them from physical
or psychological harm (e.g. during clinical trials). They are common in
medical research and the social sciences, but rare in computer science [70,
14]. However, it has been suggested that AI companies should also have IRBs
[16]. Google DeepMind already has the equivalent of an IRB, an
interdisciplinary committee that oversees their human rights policy [48],
which played a key role in their release of AlphaFold [76, 73, 122]. IRBs and
internal audit are only similar insofar as they are both intended to ensure
ethical conduct and reduce risks to individuals. However, there are noticeable
differences. In particular, IRBs have a much narrower scope: they only focus
on human subject research. Even though it has been proposed to adapt IRBs to
an AI context [14, 105], other differences remain. In particular, based on
existing proposals, they would still not review the lab’s governance. Unlike
internal audit, they would not “supervise the supervisors”.555One could think
of the risk management team as “supervisors” of the research and product
development teams, and internal audit as the “supervisor” of risk management.
This is, of course, an oversimplification. I use that phrase to emphasize that
internal audit plays a meta function. They do not assess risks, but the
effectiveness of risk management practices. IRBs are therefore not a
substitute for internal audit.

###  2.3 How established is internal audit?

Internal audit is a well-established governance mechanism that is very common
in many other industries.

Profession. There is an entire profession of internal auditors. There are
professional associations (e.g. the Institute of Internal Auditors), standards
[57], and certification programs [63]. For more information about the current
state of the internal audit profession, see the benchmarking reports by the
Institute of Internal Auditors [61] and Protiviti [106].

Standards and best practices. There are generally accepted best practices.
Most notably, the Institute of Internal Auditors (IIA) publishes the
International Standards for the Professional Practice of Internal Auditing
[57]. Note that they plan to update the standards in 2023 [65]. The IIA also
publishes best practices for the Three Lines of Defense model, in which
internal audit acts as the third line [54, 60]. The Three Lines of Defense
model is a risk management framework that helps organizations to assign and
coordinate risk management roles and responsibilities. It essentially answers
the question: who is responsible for risk management? Schuett suggests
equivalent roles and responsibilities in an AI context [114].

Literature. There is an extensive body of literature on internal audit,
including literature reviews [79, 112], literature on its effectiveness [82,
37, 69], and literature on its history [120, 110, 13].

Who has an internal audit team? Most public companies have an internal audit
team. Based on their proxy statements, a document that public companies in the
US must file with the Securities and Exchange Commission (SEC), all big tech
companies (e.g. Alphabet [1], Microsoft [87], and Meta [85]), hardware
companies (e.g. NVIDIA [97] and Intel [66]), banks (e.g. Goldman Sachs [46]
and JP Morgan Chase [71]), and airlines (e.g. American Airlines [3] and Delta
Airlines [33]) seem to have internal audit teams, to name just a few
industries. In some jurisdictions, having an internal audit team is even a
legal requirement, such as for banks in the EU [38]. However, based on public
information, OpenAI, Google DeepMind, and Anthropic do not seem to have
internal audit teams.

##  3 Why AGI labs need an internal audit team

In this section, I discuss the main benefits (Section 3.1) and limitations
(Section 3.2) of internal audit at AGI labs. I conclude that AGI labs should
seriously consider setting up an internal audit team.

###  3.1 Benefits

Setting up an internal audit team would have a number of benefits. Below, I
discuss four benefits that seem particularly relevant for AGI labs.

Improving risk management practices. First, current risk management practices
might no longer be fit-for-purpose. The level of risk from state-of-the-art AI
systems is increasing. Safety concerns that once were theoretical might soon
become real problems. For example, as models get scaled up, new capabilities
can emerge unpredictably [41, 127],666Note that a recent paper expressed
doubts about this phenomenon [113]. some of which might be dangerous [119].
Capabilities that would be particularly concerning include situational
awareness [29, 95, 99], power-seeking [124, 25, 123, 80, 99], persuasion [86,
99], and long-horizon planning [99]. The risk management practices at AGI labs
need to keep up with these developments. This requires a more proactive
approach: labs need to identify potential risk management failures before
someone is harmed. They need to know whether or not their current practices
are fit-for-purpose and constantly re-evaluate and update them. Internal audit
can help with that. It would assess the effectiveness of risk management
practices, identify shortcomings, and report them to the board of directors,
which could then engage with senior management to fix them. Without such an
effort, it seems plausible that at least some shortcomings will remain
unnoticed [114].

Providing assurance to the board of directors. Second, internal audit can
ensure that the board of directors has a more accurate view of the current
level of risk and the effectiveness of the lab’s risk management practices.
This is important because the board plays a key role in the corporate
governance of AGI labs [26]. It sets the strategic priorities, is responsible
for risk oversight, and has significant influence over senior management (e.g.
it can replace chief executive officer [CEO]). However, many board members
only work part-time and rely on information provided to them by senior
management. This information might be biased. For example, deploying a new
model might be good for the career of the head of research, so they might play
down the associated risks. They might tell the board what they think it wants
to hear, not what it needs to hear. The board might therefore not get an
accurate view of the current level of risk and the effectiveness of the labs’s
risk management practices. This situation can be phrased as a principal-agent-
problem between the board, which is legally responsible for risk oversight
(principal), and the executives, who are responsible for risk-related
activities like research, product development, and risk management (agents).
Internal audit can tackle this problem. It acts as the board’s independent
ally in the company [32]. It is often described as the board’s “eyes and ears”
[60]. While the chief risk officer (CRO)—a senior executive who is responsible
for risk management—reports the current level of risk and outcomes of risk
management activities to the board, the chief audit executive (CAE) would tell
the board how much they can trust these reports (e.g. “their method for
evaluating risks is flawed”). Since internal audit is organizationally
independent from management, they are less biased and more objective. With
these two reporting lines, the board has a more complete picture. It seems
plausible that internal audit substantially improves the board’s information
base [114].

Signaling responsibility. Third, labs that have an internal audit team might
be perceived as being more responsible [114]. Labs face increasing pressure to
improve their governance practices. As they transition from startups to more
established corporations, many stakeholders (business partners, regulators,
etc.) expect them to follow best practices in corporate governance. In a
recent survey among experts from AGI labs, academia, and civil society (_N_ =
51), 86% of respondents agreed or strongly agreed that AGI labs should have an
internal audit team [117]. Following best practices might also be necessary to
reduce legal risks. For example, labs might increasingly become subject to
tort law claims. If they release a product and that product harms a user, that
user might claim compensation from the lab. The lab might then claim recourse
from the responsible executive. By following best practices in corporate
governance, executives might be able to demonstrate that they have not
violated their duty of care and thus exculpate themselves. In general, setting
up an internal audit team would be a credible signal that a lab aims to
professionalize their risk management practices. Since none of the AGI labs
currently has an internal audit team (based on public information), the first
lab would likely get most of the PR benefits. They have the opportunity to be
“ahead of the curve”.

Contact for whistleblowers. Fourth, internal audit could serve as a contact
for whistleblowers. Detecting misconduct is often difficult: it is hard to
observe from the outside, while insiders might not report it because they face
a conflict between personal values and loyalty [72, 35], or because they fear
retaliation [15]. For example, an engineer might become convinced that a model
shows early signs of power-seeking behavior [95, 124, 25, 123, 80, 99], but
the research lead wants to release the model anyway and threatens to fire the
engineer if they speak up. In such cases, whistleblower protection is vital.
This might become even more important as commercial pressure increases because
it might incentevize labs to cut corners on safety [9, 93]. Internal audit
could protect whistleblowers by providing a trusted contact point. It would be
more trustworthy than other organizational units because it is organizational
independent from management. But since it would still be part of the
organization, confidentiality would be less of a problem. This can be
particularly important if the information is highly sensitive and its
dissemination could be harmful in itself [125, 18, 20]. The internal audit
team could report the case to the board of directors who could engage with
management to address the issue. It could also advise the whistleblower on
other steps they could take to protect themselves or do something about the
misconduct. While this is not a typical role of internal audit, it has been
discussed in the literature [72].

###  3.2 Limitations

However, the benefits of internal audit might be limited and there might also
be a few downsides. I think they can be overcome, but labs should take them
seriously. Below, I discuss five main limitations.

Friction. First, internal audit can add friction. The internal audit team
interacts with many different people, including C-suite executives and senior
researchers. To them, internal audit might seem annoying, distracting, and
bureaucratic. They might even be actively opposed if they fear that internal
audit discovers flaws in their work. Internal audit might also (indirectly)
delay decisions. For example, OpenAI spent six months on safety research, risk
assessment, and iteration before releasing GPT-4 [99]. While I welcome this
level of scrutiny, it seems plausible that an internal audit team would have
found flaws in some of their risk assessment methods. Depending on the
severity of the flaws, internal audit would have escalated the issue to the
board, which might have started an investigation that would have ideally
resulted in an improvement of the methods and a repetition of the initial
assessment. This could have delayed the release for additional weeks or
months. To be clear, such a delay would not only be in the interest of
society, it might also be in the lab’s own interest. Rushed releases can
expose labs to significant financial and reputational risks. For example,
Alphabet’s share price dropped 9%—$100 billion in market value—after Google’s
chatbot Bard made some mistakes in a public demo [31]. Similarly, Microsoft
faced negative press coverage after users reported that Bing Chat sometimes
gives bizarre answers and even threatens users [111, 104].

Lack of empirical evidence. Second, there is not much empirical evidence in
support of the above-mentioned benefits. Since there are no studies on the
effectiveness of internal audit in an AI context, I have to rely on studies
from other industries [82, 37, 69]. Overall, the empirical literature suggests
that internal audit has in fact some merits. For example, it is associated
with increased financial performance [69], a decline in perceived risk [24],
strengthened internal controls [83, 100], and improved fraud prevention [27,
84, 34], among other things. However, these findings do not correspond to the
above-mentioned benefits. And even if they did, they might not generalize to
an AI context. AGI labs are structurally different from other companies and
the risks from AI are broader than risks from other products and services
[114]. Against this background, the benefits remain somewhat speculative and
based on abstract plausibility considerations. I also expect the benefits to
depend heavily on the concrete way in which the internal audit team is set up
(e.g. how large the team is, how competent its members are, and how much
management supports it). For more information on the drivers of internal audit
effectiveness, I refer to the relevant literature [8, 36].

Benefits depend on individuals. Third, the benefits depend on the individuals
who perform the internal audit activities. Simply having an internal audit
team is not sufficient to seize the above-mentioned benefits. The value of
internal audit mainly depends on the people involved and their ability and
willingness to identify ineffective risk management practices. But having able
and willing people is also not sufficient. Both are necessary and only
together can they be sufficient (though other factors, such as information
sharing between different organizational units, might also play a role).

Costs. Fourth, setting up and maintaining an internal audit team is costly.
The biggest cost positions will likely be the initial change process (e.g.
costs for external consultants and training of personnel) and salaries (e.g.
for hiring a chief audit executive [CAE] and internal audit professionals).
Since costs are highly context-specific, it is not possible to estimate them
in the abstract. For example, hiring a large team of Certified Internal
Auditors [63] and an experienced chief audit executive (CAE) will be orders of
magnitudes more expensive than asking one or two employees to assess the
effectiveness of the lab’s risk management practices on a part-time basis.
Labs need to weigh the costs with the above-mentioned benefits. Since there
are cheap ways to capture at least some of the value that internal audit
offers, I expect the benefits to outweigh the costs in most cases.

No silver bullet. Fifth, internal audit is not a silver bullet. Setting up an
internal audit team will not be sufficient to reduce risks to an acceptable
level. Internal audit should be seen as an additional “layer of defense”, as
yet another mechanism in a portfolio of mechanisms. In particular, it seems
unlikely that setting up an internal audit team and following current best
practices will be sufficient as labs approach AGI. Labs will have to adjust
and improve their internal audit activities over time. This will likely
require fundamental governance innovations. Despite this, internal audit might
feel like a substantial safety improvement, which could lead to a false sense
of security. However, I do not expect this to be a major issue. I doubt that
people at labs would overly rely on the team. Nobody would think of internal
audit as a silver bullet—“there is no silver bullet” [23].

##  4 How to set up an internal audit team

In this section, I suggest ways in which AGI labs could set up an internal
audit team. My suggestions are based on the International Standards for the
Professional Practice of Internal Auditing, especially the Attribute Standards
which focus on the organizational context [57]. To set up an internal audit
team, AGI labs need to write an internal audit charter (Section 4.1), appoint
a chief audit executive (Section 4.2), assemble a team of internal audit
professionals (Section 4.3), and create a quality assurance and improvement
program (Section 4.4).

###  4.1 Internal audit charter

First, AGI labs need to define the responsibilities, scope, authority, and
reporting lines of the internal audit team. These foundational decisions
should be documented in a so-called “internal audit charter” (Attribute
Standards 1000–1010 [57, 59]).

Responsibilities. I would suggest the following definition of internal audit’s
responsibilities: “The internal audit team is responsible for providing
independent and objective assurance about the organization’s risk management
practices to the board of directors. To this end, they assess the
effectiveness of the organization’s risk management practices, report their
findings to the board and inform senior management, and advise the board on
risk-related matters.”

Scope. While it makes sense to limit the scope of specific internal audit
engagements (Section 5.1), it is very uncommon to limit the scope of the
entire internal audit team. They should be responsible for assessing all risk
management practices (Performance Standards 2120). In particular, labs should
not limit the scope to specific types of risk, though it might make sense to
take a risk-based approach and focus on the most severe risks, including
existential risks. In addition to that, the internal audit team could assess
other governance structures (Performance Standards 2110 [57]) and internal
controls (Performance Standards 2130 [57]), but this is usually not a core
internal audit activity.

Authority. To do their job, the internal audit team needs to access documents,
attend meetings, talk to people, etc. As this might involve access to
sensitive information (e.g. unreleased models) and high-stakes conversations
(e.g. between senior researchers), their authority should be specified in more
detail. They should also be able to “enforce” access (e.g. by reporting denied
access to the board of directors who can then engage with senior management).

Reporting lines. The CAE should report directly to the board of directors,
ideally their audit committee. They should also inform senior management, but
senior management should not be able to interfere with internal audit’s work.
In other words, internal audit should be organizationally independent from
senior management (Attribute Standard 1110 [57]). These reporting lines are an
essential feature of internal audit, not a “nice-to-have”.

###  4.2 Chief audit executive

Next, labs need to appoint a chief audit executive (CAE) who leads the
internal audit team (Attribute Standards 1111, 1112 [57]). (Note that the
title does not matter. They are sometimes called “Head of Internal Audit” or
“Director of Internal Audit”, but CAE seems to be most common.)

Qualities. The CAE should have many of the same qualities as internal auditors
(Section 4.3). It might be even more important for this person to have the
bigger picture in mind (e.g. addressing the trade-off between more rigorous
risk management practices and curtailing the benefits from new models). It
might also be more important for this person to have practical internal audit
experience (e.g. maybe they used to be CAE at another tech company). They
should also be able and willing to speak up and have difficult conversations
with senior management and the board of directors. It might be less important
for this person to have deep AI expertise.

Recruiting. I doubt that there are any candidates who check all the boxes. But
I do not have any recommendations on what to do about it.

###  4.3 Internal auditors

The CAE then assembles a team of internal audit professionals (Attribute
Standards 1300–1322 [57]).

Size of the team. It is difficult to comment on team size without further
context. But it will often make sense to start small and scale as needed. For
example, starting with two full-time employees (FTEs), plus the CAE, and then
growing the team to three to five FTEs over the next two years would be
sensible for most labs. Though, depending on how much the lab grows, how risky
future models are, and how much pressure there is to improve their risk
management practices, a larger team might be necessary.

Qualities. Internal audits should have a number of qualities. They can be
broken down into three clusters. First, internal auditors should have AI
expertise. They should be familiar with the development and deployment process
of state-of-the-art AI systems, such as GPT-4 [99] or Claude [7]. They should
have a good understanding of the associated risks, including potential future
risks. Second, they should have internal audit expertise. They should know
what internal audit is and how it works. Being a Certified Internal Auditor
[63] would be an advantage, but this is not strictly necessary. Ideally, they
would also have practical experience (e.g. as an internal audit professional
at a big tech company). Third, they should have a number of other desirable
qualities (Attribute Standards 1120, 1130, 1200–1230 [57]). They should be
independent and objective. They should avoid conflicts of interest and not
have any other roles. Any impairment of independence or objectivity should be
disclosed. They should also be mission-aligned, i.e. care deeply about
“ensuring that AGI benefits all of humanity” [98] or “solving intelligence to
advance science and benefit humanity” [49]. It is also important that internal
auditors are able to deal with the difficult trade-off between reducing risks
by making risk management practices more rigorous and sacrificing some
benefits by delaying a model release, not releasing the model, or restricting
access. They should also be highly trustworthy. As they will have access to
confidential information, they need to be discreet. They should have a “scout
mindset” [40], i.e. cultivate an attitude of curiosity and openness to
evidence, be truth-seeking and epistemically honest. Finally, they should pay
attention to detail, always double-check their work, and strive for constant
improvement.

Recruiting. I doubt that there are any candidates who have all of the above-
mentioned qualities. To recruit candidates, labs could therefore take two
approaches. First, they could “bring AI to internal audit”, i.e. hire internal
audit professionals with experience in another industry and train them in AI.
One disadvantage of this approach would be that internal auditors from other
industries might be insensitive to the stakes at hand. In their previous
roles, they were concerned with litigation risks, PR risks, etc., but at AGI
labs they would have to deal with potentially existential risks. Second, labs
could “bring internal audit to AI”, i.e. train AI governance experts in
internal audit (e.g. by reading internal audit standards and educational
resources, and learning-by-doing). The main advantage of this approach would
be that it might be easier to find candidates. But there are also several
disadvantages. It is unclear if AI governance experts would be able to get up
to speed quickly enough. They might also not understand what is politically
feasible. Finally, hiring “weird” AI governance experts instead of seasoned
professionals might pose PR risks. Both approaches have advantages and
disadvantages. Labs should arguably do a mix of both.

###  4.4 Quality control

Finally, the CAE should create a quality assurance and improvement program
(Attribute Standards 1300–1322 [57]). Internal audit tells the board how much
they can trust senior management (“their method for evaluating risk is
flawed”), but the board also needs to know how much they can trust internal
audit (“are they able to assess whether the method is flawed?”). The purpose
of the quality assurance and improvement program is to assess the
effectiveness of internal audit activities. The review should be done both
internally (e.g. annually) and externally (e.g. every two to three years).
There are dedicated organizations who offer external reviews of internal audit
activities, but it could also be done by one or more trusted individuals with
relevant expertise. The CAE should report the results of the reviews to the
board of directors.

Once an internal audit team has been set up, they can start assessing the
lab’s risk management practices. In the next section, I will provide more
details of what that might entail.

##  5 What an internal audit team would do

In this section, I suggest concrete things internal audit teams at AGI labs
could do. Again, my suggestions heavily draw from the International Standards
for the Professional Practice of Internal Auditing, in particular the
Performance Standards which describe the internal audit activities [57].
Internal auditing is typically performed in sprints (so-called “engagements”).
Every engagement consists of four steps: planning (Section 5.1), gathering
information (Section 5.2), assessment (Section 5.3), and reporting (Section
5.4). However, in practice, it will be messier than this. The five steps will
overlap and there will be some back-and-forth between them. There will usually
be several engagements in parallel.

###  5.1 Planning

At the beginning of each engagement, the CAE should specify the objective,
scope, and resource requirements (Performance Standards 2200–2240 [57]).
Objectives can be specific/closed (e.g. “what is the false positive rate?” or
“does the risk management process comply with Article 9 of the EU AI Act?”
[115]). But they can also be more general/open-ended (e.g. “find as many flaws
as possible”). The scope of individual engagements can be limited to specific
policies or processes (e.g. the pre-deployment risk assessment), specific
risks (e.g. misuse risks), specific models (e.g. AlphaFold or DALL·E 2), or
specific flaws (e.g. robustness against circumventions). Resource requirements
include the number of FTEs (ideally two or more per engagement), the duration
of the engagement (e.g. two to four weeks), and the budget (e.g. for
commissioning an external red team).

###  5.2 Gathering information

The actual work begins with gathering information about the risk management
practices to be assessed (Performance Standard 2310 [57]). The internal audit
team should seek answers to the following questions: What techniques are risk
managers supposed to use? How are they supposed to document and report them?
How else could they do it? How do they actually do it? Who does it? When and
how often do they do it? The internal audit team could get these information
by: (1) reviewing the relevant literature (e.g. about best practices in other
industries), (2) reviewing documents (e.g. the deployment policy), (3)
attending meetings (e.g. regular meetings of development teams), (4)
conducting interviews (e.g. with junior and senior engineers), or (5)
commissioning external reviews (e.g. a governance audit [89] or a red team
[88, 42, 102]).

###  5.3 Assessment

Based on this information, the internal audit team conducts an assessment of
the effectiveness of risk management practices (Performance Standards 2120,
2320 [57]).

Object of assessment. Internal audit should try to identify flaws in every
step of the risk management process: (1) relevant risks might not be
identified, (2) the likelihood or severity of relevant risks might be
misjudged, (3) the level of risk might incorrectly be judged acceptable, and
(4) risks might not be reduced to an acceptable level. Note that there are
different conceptualizations of the risk management process [28, 67, 96]. The
above is based on ISO 31000:2019 [67]. Flaws can have different causes: (1)
There might be methodological reasons. Risk managers might not use appropriate
methods (e.g. they might not be aware of methods that are considered best
practice in other industries), appropriate methods might not exist (e.g.
dangerous capabilities evals [119]), they might rely on a single method (e.g.
risk taxonomies), or not seek external input. (2) Risk managers might also
lack resources. They might not have enough expertise (e.g. to anticipate
unprecedented risks), time (e.g. to use multiple methods or double check their
work), information (e.g. about the details of an ongoing research project), or
power (e.g. to enforce compliance with a safety policy). (3) Practices might
not sufficiently account for cognitive biases (e.g. scope neglect or
availability bias) and other human errors (e.g. risk managers might forget a
risk when going through a risk taxonomy). (4) Flaws can have many other
reasons. For example, there might be pressure to release a new model quickly,
there might be deliberate attempts to circumvent measures, or the model itself
might be deceptive (e.g. future systems might become situationally aware and
“pretend” to be safe during assessments). It goes without saying that the list
is not exhaustive. In addition to assessing the risk management process,
internal audit could assess other governance structures or internal controls
(Performance Standards 2110, 2130 [57]), but this is usually not a core
internal audit activity.

Assessment methods. The internal audit team might use the following assessment
methods: (1) understanding and replication (e.g. if a model is used to
quantify risk, calculate the risk yourself), (2) stress testing, (3)
experimentation (e.g. asking an engineer to violate a policy and see if they
get caught), (4) scenario planning, and (5) armchair reasoning (e.g. how could
you theoretically circumvent a measure?).

Frequency of assessments. Internal auditing is an ongoing activity. Although
many risk management activities are only performed during the development of a
new model, internal audit’s work should not be tied to the development and
deployment process. It might also make sense to assess certain practices
periodically (e.g. conducting an annual review of best practices as described
in AI risk management standards and in the literature). Sometimes, it will be
necessary to conduct ad hoc assessments (e.g. if someone informs the internal
audit team that a certain risk-related policy might be ineffective).

###  5.4 Reporting

The CAE reports the findings of the assessment to the board of directors and
informs senior management (Performance Standards 2400–2450, 2600 [57]).

What to report. Reports need to contain at least: (1) information about the
objectives and scope of the engagement, (2) key findings from the assessment,
and (3) recommendations or an overall opinion. Internal audit should also
report positive findings (“the assessed practices seem adequate”) and findings
about unnecessary practices. In addition to that, the CAE should report
updates (e.g. if they changed their views) and corrections (e.g. if they
become aware of a past mistake). If the CAE concludes that senior management
approves an unacceptable level of risk, they should discuss the matter with
senior management. If this is not sufficient, they must escalate the matter to
the board.

How to report. Reports must be accurate (i.e. free from errors and
distortions), objective (i.e. unbiased, balanced, and impartial), clear (i.e.
understandable, avoiding jargon and technical language), concise (i.e. to the
point, avoiding lengthy elaborations), constructive (i.e. making actionable
recommendations), and complete (i.e. they include everything the addressee
needs to know). The form of the report does not matter. A Google Doc or email
can be sufficient. It might make sense to create a template.

When to report. The reporting schedule depends on the particular context and
the preferences of the board. They might want to receive a report after every
engagement or periodic summary reports (e.g. before meetings of the risk
committee).

Addressees. The CAE reports to the board of directors, ideally its audit
committee, and informs senior management, ideally the CRO. The CAE needs
permission to share reports with external stakeholders.

##  6 Conclusion

Contributions. This paper has made two main contributions. First, it has
introduced internal audit to the AGI governance discourse. Although internal
audit is a well-established governance mechanism in many other industries, it
has been largely neglected in the debate around the governance of AGI labs.
Second, the paper contributes to efforts to professionalize risk management
practices at AGI labs. It has provided a blueprint for how AGI labs could set
up an internal audit team. It has also suggested concrete things the team
would do on a day-to-day basis. These suggestions can inform key decision-
makers at labs and serve as a basis for discussion.

Questions for further research. The paper left many questions unanswered and
more research is needed. In particular, I have not answered the question of
how AGI labs can be incentivized to actually set up an internal audit team.
For example, should an internal audit function be recommended in the NIST AI
Risk Management Framework [96] or in harmonized standards on risk management
that accompany the proposed EU AI Act [39, 114]? Since my arguments are mostly
theoretical, they also need to be verified empirically. An industry case study
similar to the one that Mökander and Floridi conducted could be a first step
[89].

AGI labs need to professionalize their risk management practices. Instead of
“reinventing the wheel”, they should follow existing best practices in
corporate governance. This will not be sufficient as labs approach AGI, but
they should not skip this obvious first step. I hope that this paper can
contribute to such efforts. In light of recent progress in AI research and
development, this is urgently needed.

## Acknowledgements

I am grateful for valuable comments and feedback from Leonie Koessler, Markus
Anderljung, James Ginns, Lennart Heim, Ben Garfinkel, and Jide Alaga. All
remaining errors are my own.

## References

  * [1] Alphabet.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/Q23E-WQWP, 2022.
  * [2] S. Altman.  Planning for AGI and beyond.  https://openai.com/blog/planning-for-agi-and-beyond, 2023.
  * [3] American Airlines.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/9WUA-7WPU, 2022.
  * [4] D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, and D. Mané.  Concrete problems in AI safety.  arXiv preprint arXiv:1606.06565, 2016.
  * [5] M. Anderljung and J. Hazell.  Protecting society from AI misuse: When are restrictions on capabilities warranted?  arXiv preprint arXiv:2303.09377, 2023.
  * [6] Anthropic.  Core views on AI safety: When, why, what, and how.  https://www.anthropic.com/index/core-views-on-ai-safety, 2023.
  * [7] Anthropic.  Introducing Claude.  https://www.anthropic.com/index/introducing-claude, 2023.
  * [8] M. Arena and G. Azzone.  Identifying organizational drivers of internal audit effectiveness.  International Journal of Auditing, 13:1, 2009.
  * [9] S. Armstrong, N. Bostrom, and C. Shulman.  Racing to the precipice: A model of artificial intelligence development.  AI & Society, 31:201–206, 2016.
  * [10] Z. Arnold and H. Toner.  AI accidents: An emerging threat.  Center for Security and Emerging Technology, Georgetown University, 2021.
  * [11] T. Aven.  On the meaning of a black swan in a risk context.  Safety Science, 57:44–51, 2013.
  * [12] M. Barnett.  When will the first general AI system be devised, tested, and publicly announced?  https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence, 2020\.
  * [13] J. Behrend and M. Eulerich.  The evolution of internal audit research: A bibliometric analysis of published documents (1926–2016).  Accounting History Review, 29(1):103–139, 2019.
  * [14] M. S. Bernstein, M. Levi, D. Magnus, B. A. Rajala, D. Satz, and Q. Waeiss.  Ethics and society review: Ethics reflection as a precondition to research funding.  Proceedings of the National Academy of Sciences, 118(52), 2021.
  * [15] B. Bjørkelo.  Workplace bullying after whistleblowing: Future research and implications.  Journal of Managerial Psychology, 28(3):306–323, 2013.
  * [16] R. Blackman.  If your company uses AI, it needs an institutional review board.  https://hbr.org/2021/04/if-your-company-uses-ai-it-needs-an-institutional-review-board, 2021\.
  * [17] N. Bostrom.  How long before superintelligence?  International Journal of Futures Studies, 2, 1998.
  * [18] N. Bostrom.  Information hazards: A typology of potential harms from knowledge.  Review of Contemporary Philosophy, 10:44–79, 2011.
  * [19] N. Bostrom.  Superintelligence: Paths, dangers, strategies.  Oxford University Press, 2014.
  * [20] N. Bostrom.  The vulnerable world hypothesis.  Global Policy, 10(4):455–476, 2019.
  * [21] M. Brundage, S. Avin, J. Clark, H. Toner, P. Eckersley, B. Garfinkel, A. Dafoe, P. Scharre, T. Zeitzoff, B. Filar, H. Anderson, H. Roff, G. C. Allen, J. Steinhardt, C. Flynn, S. Ó. hÉigeartaigh, S. Beard, H. Belfield, S. Farquhar, C. Lyle, R. Crootof, O. Evans, M. Page, J. Bryson, R. Yampolskiy, and D. Amodei.  The malicious use of artificial intelligence: Forecasting, prevention, and mitigation.  arXiv preprint arXiv:1802.07228, 2018.
  * [22] M. Brundage, S. Avin, J. Wang, H. Belfield, G. Krueger, G. Hadfield, H. Khlaaf, J. Yang, H. Toner, R. Fong, T. Maharaj, P. W. Koh, S. Hooker, J. Leung, A. Trask, E. Bluemke, J. Lebensold, C. O’Keefe, M. Koren, T. Ryffel, J. Rubinovitz, T. Besiroglu, F. Carugati, J. Clark, P. Eckersley, S. de Haas, M. Johnson, B. Laurie, A. Ingerman, I. Krawczuk, A. Askell, R. Cammarota, A. Lohn, D. Krueger, C. Stix, P. Henderson, L. Graham, C. Prunkl, B. Martin, E. Seger, N. Zilberman, S. Ó. hÉigeartaigh, F. Kroeger, G. Sastry, R. Kagan, A. Weller, B. Tse, E. Barnes, A. Dafoe, P. Scharre, A. Herbert-Voss, M. Rasser, S. Sodhani, C. Flynn, T. K. Gilbert, L. Dyer, S. Khan, Y. Bengio, and M. Anderljung.  Toward trustworthy AI development: Mechanisms for supporting verifiable claims.  arXiv preprint arXiv:2004.07213, 2020.
  * [23] M. Brundage, K. Mayer, T. Eloundou, S. Agarwal, S. Adler, G. Krueger, J. Leike, and P. Mishkin.  Lessons learned on language model safety and misuse.  https://openai.com/research/language-model-safety-and-misuse, 2022\.
  * [24] J. V. Carcello, M. Eulerich, A. Masli, and D. A. Wood.  Are internal audits associated with reductions in perceived risk?  Auditing: A Journal of Practice & Theory, 30(3):55–73, 2020.
  * [25] J. Carlsmith.  Is power-seeking AI an existential risk?  arXiv preprint arXiv:2206.13353, 2022.
  * [26] P. Cihon, J. Schuett, and S. D. Baum.  Corporate governance of artificial intelligence in the public interest.  Information, 12(7), 2021.
  * [27] P. Coram, C. Ferguson, and R. Moroney.  Internal audit, alternative internal audit structures and the level of misappropriation of assets fraud.  Accounting and Finance, 48(4):543–559, 2008.
  * [28] COSO.  Enterprise risk management: Integrating with strategy and performance.  https://perma.cc/5Z3G-KD6R, 2017.
  * [29] A. Cotra.  Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover.  https://www.alignmentforum.org/posts/pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to, 2022\.
  * [30] B. Couceiro, I. Pedrosa, and A. Marini.  State of the art of artificial intelligence in internal audit context.  In 2020 15th Iberian Conference on Information Systems and Technologies (CISTI), pages 1–7, 2020.
  * [31] M. Coulter and G. Bensinger.  Alphabet shares dive after Google AI chatbot Bard flubs answer in ad.  https://www.reuters.com/technology/google-ai-chatbot-bard-offers-inaccurate-information-company-ad-2023-02-08, 2023\.
  * [32] H. Davies and M. Zhivitskaya.  Three lines of defence: A robust organising framework, or just lines in the sand?  Global Policy, 9:34–42, 2018.
  * [33] Delta Airlines.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/5EJ8-D72B, 2023.
  * [34] G. Drogalas, M. Pazarskis, E. Anagnostopoulou, and A. Papachristou.  The effect of internal audit effectiveness, auditor responsibility and training in fraud detection.  Journal of Accounting and Management Information Systems, 16(4):434–454, 2017.
  * [35] J. Dungan, A. Waytz, and L. Young.  The psychology of whistleblowing.  Current Opinion in Psychology, 6:129–133, 2015.
  * [36] L. Erasmus and P. Coetzee.  Drivers of stakeholders’ view of internal audit effectiveness: Management versus audit committee.  Managerial Auditing Journal, 33(1):90–114, 2019.
  * [37] A. Eulerich and M. Eulerich.  What is the value of internal auditing? A literature review on qualitative and quantitative perspectives.  Maandblad Voor Accountancy En Bedrijfseconomie, 94(3/4):83–92, 2020\.
  * [38] European Banking Authority.  Final report on guidelines on internal governance under Directive 2013/36/EU (EBA/GL/2021/05).  https://perma.cc/RCD8-V99V, 2021.
  * [39] European Commission.  Proposal for a regulation laying down harmonized rules on artificial intelligence (Artificial Intelligence Act) (COM(2021) 206 final).  https://perma.cc/4YXM-38U9, 2021.
  * [40] J. Galef.  The scout mindset: Why some people see things clearly and others don’t.  Penguin Random House, 2021.
  * [41] D. Ganguli, D. Hernandez, L. Lovitt, A. Askell, Y. Bai, A. Chen, T. Conerly, N. Dassarma, D. Drain, N. Elhage, S. El Showk, S. Fort, Z. Hatfield-Dodds, T. Henighan, S. Johnston, A. Jones, N. Joseph, J. Kernian, S. Kravec, B. Mann, N. Nanda, K. Ndousse, C. Olsson, D. Amodei, T. Brown, J. Kaplan, S. McCandlish, C. Olah, D. Amodei, and J. Clark.  Predictability and surprise in large generative models.  In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764, 2022.
  * [42] D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Kadavath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, A. Jones, S. Bowman, A. Chen, T. Conerly, N. DasSarma, D. Drain, N. Elhage, S. El-Showk, S. Fort, Z. Hatfield-Dodds, T. Henighan, D. Hernandez, T. Hume, J. Jacobson, S. Johnston, S. Kravec, C. Olsson, S. Ringer, E. Tran-Johnson, D. Amodei, T. Brown, N. Joseph, S. McCandlish, C. Olah, J. Kaplan, and J. Clark.  Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned.  arXiv preprint arXiv:2209.07858, 2022.
  * [43] B. Goertzel.  Who coined the term “AGI”?  https://goertzel.org/who-coined-the-term-agi, 2011.
  * [44] B. Goertzel.  Artificial general intelligence: Concept, state of the art, and future prospects.  Journal of Artificial General Intelligence, 5(1):1–46, 2014.
  * [45] B. Goertzel and C. Pennachin.  Artificial General Intelligence.  Springer, 2007.
  * [46] Goldman Sachs.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/JFK5-Z29L, 2022.
  * [47] J. A. Goldstein, G. Sastry, M. Musser, R. DiResta, M. Gentzel, and K. Sedova.  Generative language models and automated influence operations: Emerging threats and potential mitigations.  arXiv preprint arXiv:2301.04246, 2023.
  * [48] Google DeepMind.  Human rights policy.  https://www.deepmind.com/human-rights-policy, 2022.
  * [49] Google DeepMind.  https://www.deepmind.com/about, 2023.
  * [50] R. Gruetzemacher, D. Paradice, and K. B. Lee.  Forecasting transformative AI: An expert survey.  arXiv preprint arXiv:1901.08579, 2019.
  * [51] M. A. Gubrud.  Nanotechnology and international security.  https://web.archive.org/web/20110427135521/http://www.foresight.org/Conferences/MNT05/Papers/Gubrud/index.html, 1997\.
  * [52] J. Hazell.  Large language models can be used to effectively scale spear phishing campaigns.  arXiv preprint arXiv:2305.06972, 2023.
  * [53] IEEE.  1028-2008 — IEEE standard for software reviews and audits.  https://doi.org/10.1109/IEEESTD.2008.4601584, 2022.
  * [54] Institute of Internal Auditors.  IIA position paper: The three lines of defense in effective risk management and control.  https://perma.cc/NQM2-DD7V, 2013.
  * [55] Institute of Internal Auditors.  Artificial intelligence: Considerations for the profession of internal auditing (Part I).  https://perma.cc/K8WQ-VNFZ, 2017.
  * [56] Institute of Internal Auditors.  The IIA’s artificial intelligence auditing framework: Practical applications (Part A).  https://perma.cc/U93U-LN75, 2017.
  * [57] Institute of Internal Auditors.  International standards for the professional practice of internal auditing.  https://perma.cc/AKU7-8YWZ, 2017.
  * [58] Institute of Internal Auditors.  The IIA’s artificial intelligence auditing framework: Practical applications (Part B).  https://perma.cc/826X-Y3L7, 2018.
  * [59] Institute of Internal Auditors.  The internal audit charter.  https://perma.cc/G2M7-F98U, 2019.
  * [60] Institute of Internal Auditors.  The IIA’s three lines model: An update of the three lines of defense.  https://perma.cc/GAB5-DMN3, 2020.
  * [61] Institute of Internal Auditors.  North American pulse of internal audit: Benchmarks for internal audit leaders.  https://perma.cc/XAQ4-LPG7, 2022.
  * [62] Institute of Internal Auditors.  About the profession frequently asked questions.  https://perma.cc/P9X5-T37M, 2023.
  * [63] Institute of Internal Auditors.  Certified internal auditor.  https://perma.cc/ZHY8-H2DP, 2023.
  * [64] Institute of Internal Auditors.  Definition of internal audit.  https://perma.cc/XD5G-VP22, 2023.
  * [65] Institute of Internal Auditors.  IPPF evolution: Public comment on proposed standards now open.  https://www.theiia.org/en/standards/ippf-evolution, 2023.
  * [66] Intel.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/STG7-EU4Q, 2022.
  * [67] ISO.  31000:2018 Risk management — Guidelines.  https://www.iso.org/standard/65694.html, 2018.
  * [68] ISO/IEC.  Guide 51:2014 Safety aspects — Guidelines for their inclusion in standards.  https://www.iso.org/standard/53940.html, 2014.
  * [69] L. Jiang, W. F. Messier, and D. A. Wood.  The association between internal audit operations-related services and firm operating performance.  Auditing: A Journal of Practice & Theory, 39(1):101–124, 2020\.
  * [70] S. R. Jordan.  Designing artificial intelligence review boards: creating risk metrics for review of AI.  In 2019 IEEE International Symposium on Technology and Society (ISTAS), pages 1–7, 2019.
  * [71] JPMorgan Chase.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/ZL4G-TRXK, 2022.
  * [72] P. B. Jubb.  Whistleblowing: A restrictive definition and interpretation.  Journal of Business Ethics, 21:77–94, 1999.
  * [73] J. M. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Zídek, A. Potapenko, A. Bridgland, C. Meyer, S. A. A. Kohl, A. Ballard, A. Cowie, B. Romera-Paredes, S. Nikolov, R. Jain, J. Adler, T. Back, S. Petersen, D. A. Reiman, E. Clancy, M. Zielinski, M. Steinegger, M. Pacholska, T. Berghammer, S. Bodenstein, D. Silver, O. Vinyals, A. W. Senior, K. Kavukcuoglu, P. Kohli, and D. Hassabis.  Highly accurate protein structure prediction with AlphaFold.  Nature, 596:583–589, 2021.
  * [74] S. B. Kahyaoglu and T. Aksoy.  Artificial intelligence in internal audit and risk assessment.  In U. Hacioglu and T. Aksoy, editors, Financial ecosystem and strategy in the digital era, pages 179–192. Springer, 2021.
  * [75] K. Kavukcuoglu.  Real-world challenges for AGI.  https://www.deepmind.com/blog/real-world-challenges-for-agi, 2022\.
  * [76] K. Kavukcuoglu, P. Kohli, L. Ibrahim, D. Bloxwich, and S. Brown.  How our principles helped define AlphaFold’s release.  https://www.deepmind.com/blog/how-our-principles-helped-define-alphafolds-release, 2022\.
  * [77] K. Klonick.  The Facebook Oversight Board: Creating an independent institution to adjudicate online free expression.  Yale Law Journal, 129(2418), 2020.
  * [78] N. Kolt.  Algorithmic black swans.  Washington University Law Review, 101, 2023.
  * [79] A. Kotb, H. Elbardan, and H. Halabi.  Mapping of internal audit research: A post-Enron structured literature review.  Accounting, Auditing & Accountability Journal, 33(8):1969–1996, 2020.
  * [80] V. Krakovna and R. Shah.  Some high-level thoughts on the DeepMind alignment team’s strategy.  https://www.alignmentforum.org/posts/a9SPcZ6GXAg9cNKdi/linkpost-some-high-level-thoughts-on-the-deepmind-alignment, 2023\.
  * [81] M. Kruppa.  Google DeepMind CEO says some form of AGI possible in a few years.  https://www.wsj.com/articles/google-deepmind-ceo-says-some-form-of-agi-possible-in-a-few-years-2705f452, 2023\.
  * [82] R. Lenz and U. Hahn.  A synthesis of empirical internal audit effectiveness literature pointing to new research opportunities.  Managerial Auditing Journal, 30(1):5–33, 2015.
  * [83] S. Lin, M. Pizzini, M. Vargus, and I. R. Bardhan.  The role of the internal audit function in the disclosure of material weaknesses.  The Accounting Review, 86(1):287–323, 2011.
  * [84] Y. Ma’ayan and A. Carmeli.  Internal audits as a source of ethical behavior, efficiency, and effectiveness in work units.  Journal of Business Ethics, 137(2):347–363, 2016.
  * [85] Meta.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/G7AG-R6DN, 2022.
  * [86] Meta Fundamental AI Research Diplomacy Team (FAIR), A. Bakhtin, N. Brown, E. Dinan, G. Farina, C. Flaherty, D. Fried, A. Goff, J. Gray, H. Hu, A. P. Jacob, M. Komeili, K. Konath, M. Kwon, A. Lerer, M. Lewis, A. H. Miller, S. Mitts, A. Renduchintala, S. Roller, D. Rowe, W. Shi, J. Spisak, A. Wei, D. Wu, H. Zhang, and M. Zijlstra.  Human-level play in the game of Diplomacy by combining language models with strategic reasoning.  Science, 378(6624):1067–1074, 2022.
  * [87] Microsoft.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/6NYQ-ZTMB, 2022.
  * [88] P. Mishkin, L. Ahmad, M. Brundage, G. Krueger, and G. Sastry.  DALL·E 2 preview: Risks and limitations.  https://github.com/openai/dalle-2-preview/blob/main/system-card.md, 2022\.
  * [89] J. Mökander and L. Floridi.  Operationalising AI governance through ethics-based auditing: An industry case study.  AI and Ethics, pages 1–18, 2022.
  * [90] J. Mökander, J. Morley, M. Taddeo, and L. Floridi.  Ethics-based auditing of automated decision-making systems: Nature, scope, and limitations.  Science and Engineering Ethics, 27(44), 2021.
  * [91] J. Mökander, J. Schuett, H. R. Kirk, and L. Floridi.  Auditing large language models: A three-layered approach.  AI and Ethics, 2023.
  * [92] L. Muehlhauser.  What is AGI?  https://intelligence.org/2013/08/11/what-is-agi, 2013.
  * [93] W. Naudé and N. Dimitri.  The race for an artificial general intelligence: Implications for public policy.  AI & Society, 35:367–379, 2020.
  * [94] R. Ngo.  https://twitter.com/RichardMCNgo/status/1643310525697105935, 2023\.
  * [95] R. Ngo, L. Chan, and S. Mindermann.  The alignment problem from a deep learning perspective.  arXiv preprint arXiv:2209.00626, 2022.
  * [96] NIST.  Artificial Intelligence Risk Management Framework (AI RMF 1.0).  https://doi.org/10.6028/NIST.AI.100-1, 2023.
  * [97] NVIDIA.  Notice of 2022 annual meeting of stockholders and proxy statement.  https://perma.cc/C24T-FYPR, 2022.
  * [98] OpenAI.  Charter.  https://openai.com/charter, 2018.
  * [99] OpenAI.  GPT-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [100] A. A. Oussii and N. Boulila Taktak.  The impact of internal audit function characteristics on internal control quality.  Managerial Auditing Journal, 33(5):450–469, 2018.
  * [101] Oversight Board.  https://www.oversightboard.com, 2023.
  * [102] E. Perez, S. Huang, F. Song, T. Cai, R. Ring, J. Aslanides, A. Glaese, N. McAleese, and G. Irving.  Red teaming language models with language models.  arXiv preprint arXiv:2202.03286, 2022.
  * [103] B. Perrigo.  DeepMind’s CEO helped take AI mainstream. Now he’s urging caution.  https://time.com/6246119/demis-hassabis-deepmind-interview, 2023\.
  * [104] B. Perrigo.  The new AI-powered Bing is threatening users. That’s no laughing matter.  https://time.com/6256529/bing-openai-chatgpt-danger-alignment, 2023\.
  * [105] M. Petermann, N. Tempini, I. K. Garcia, K. Whitaker, and A. Strait.  Looking before we leap.  Ada Lovelace Institute, 2022.
  * [106] Protiviti.  Internal auditing around the world: Building on experience to shape the future auditor.  https://perma.cc/24T4-ZTLJ, 2022.
  * [107] I. D. Raji and J. Buolamwini.  Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial AI products.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 429–435, 2019.
  * [108] I. D. Raji, A. Smart, R. N. White, M. Mitchell, T. Gebru, B. Hutchinson, J. Smith-Loud, D. Theron, and P. Barnes.  Closing the AI accountability gap: Defining an end-to-end framework for internal algorithmic auditing.  arXiv preprint arXiv:2001.00973, 2020.
  * [109] I. D. Raji, P. Xu, C. Honigsberg, and D. Ho.  Outsider oversight: Designing a third party audit ecosystem for AI governance.  In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 557–571, 2022.
  * [110] S. Ramamoorti.  Internal auditing: History, evolution, and prospects.  In S. R. A. D. Bailey, A. A. Gramling, editor, Research opportunities in internal auditing, pages 1–23, 2003.
  * [111] K. Roose.  A conversation with Bing’s chatbot left me deeply unsettled.  https://www.nytimes.com/2023/02/16/technology/bing-chatbot-microsoft-chatgpt.html, 2023\.
  * [112] M. Roussy and A. Perron.  New perspectives in internal audit research: A structured literature review.  Accounting Perspectives, 17(3):345–385, 2021.
  * [113] R. Schaeffer, B. Miranda, and S. Koyejo.  Are emergent abilities of large language models a mirage?  arXiv preprint arXiv:2304.15004, 2023.
  * [114] J. Schuett.  Three lines of defense against risks from AI.  arXiv preprint arXiv:2212.08364, 2022.
  * [115] J. Schuett.  Risk management in the Artificial Intelligence Act.  European Journal of Risk Regulation, pages 1–19, 2023.
  * [116] J. Schuett, N. Dreksler, M. Anderljung, D. McCaffary, L. Heim, E. Bluemke, and B. Garfinkel.  Towards best practices in AGI safety and governance: A survey of expert opinion.  arXiv preprint arXiv:2305.07153, 2023.
  * [117] J. Schuett, A. Reuel, and A. Carlier.  How to design an AI ethics board.  arXiv preprint arXiv:2304.07249, 2023.
  * [118] J. R. Searle.  Minds, brains, and programs.  Behavioral and Brain Sciences, 3(3):417–424, 1980.
  * [119] T. Shevlane, S. Farquhar, B. Garfinkel, M. Phuong, J. Whittlestone, J. Leung, D. Kokotajlo, N. Marchal, M. Anderljung, N. Kolt, L. Ho, D. Siddarth, S. Avin, W. Hawkins, B. Kim, I. Gabriel, V. Bolina, J. Clark, Y. Bengio, P. Christiano, and A. Dafoe.  Model evaluation for extreme risks.  arXiv preprint arXiv:2305.15324, 2023.
  * [120] L. F. Spira and M. Page.  Risk management: The reinvention of internal control and the changing role of internal audit.  Accounting, Auditing & Accountability Journal, 16(4):640–661, 2003\.
  * [121] N. N. Taleb.  The black swan: The impact of the highly improbable.  Random House, 2007.
  * [122] K. Tunyasuvunakool, J. Adler, Z. Wu, T. Green, M. Zielinski, A. Zídek, A. Bridgland, A. Cowie, C. Meyer, A. Laydon, S. Velankar, G. J. Kleywegt, A. Bateman, R. Evans, A. Pritzel, M. Figurnov, O. Ronneberger, R. Bates, S. A. A. Kohl, A. Potapenko, A. J. Ballard, B. Romera-Paredes, S. Nikolov, R. Jain, E. Clancy, D. A. Reiman, S. Petersen, A. W. Senior, K. Kavukcuoglu, E. Birney, P. Kohli, J. M. Jumper, and D. Hassabis.  Highly accurate protein structure prediction for the human proteome.  Nature, 596:590 – 596, 2021.
  * [123] A. M. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli.  Optimal policies tend to seek power.  arXiv preprint arXiv:1912.01683, 2023.
  * [124] A. M. Turner and P. Tadepalli.  Parametrically retargetable decision-makers tend to seek power.  arXiv preprint arXiv:2206.13477, 2022.
  * [125] F. Urbina, F. Lentzos, C. Invernizzi, and S. Ekins.  Dual use of artificial-intelligence-powered drug discovery.  Nature Machine Intelligence, 4(3):189–191, 2022.
  * [126] K. Walker.  An external advisory council to help advance the responsible development of AI.  https://blog.google/technology/ai/external-advisory-council-help-advance-responsible-development-ai, 2019\.
  * [127] J. Wei, Y. Tay, R. Bommasani, C. Raffel, B. Zoph, S. Borgeaud, D. Yogatama, M. Bosma, D. Zhou, D. Metzler, E. H. Chi, T. Hashimoto, O. Vinyals, P. Liang, J. Dean, and W. Fedus.  Emergent abilities of large language models.  arXiv preprint arXiv:2206.07682, 2022.
  * [128] L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, Z. Kenton, S. Brown, W. Hawkins, T. Stepleton, C. Biles, A. Birhane, J. Haas, L. Rimell, L. A. Hendricks, W. Isaac, S. Legassick, G. Irving, and I. Gabriel.  Ethical and social risks of harm from language models.  arXiv preprint arXiv:2112.04359, 2021.
  * [129] D. Wong and L. Floridi.  Meta’s Oversight Board: A review and critical assessment.  Minds and Machines, pages 1–24, 2022.