$^{1}$ School of Computing, Engineering and Digital Technologies,  Teesside University, Middlesbrough, UK TS1 3BA

$^2$ Machine Learning Group, Université Libre de Bruxelles, Boulevard du Triomphe CP212, 1050 Brussels, Belgium

$^3$ Artificial Intelligence Lab, Vrije Universiteit Brussel, Boulevard de la Plaine 2, 1050 Brussels, Belgium
$^{4}$	INESC-ID and Instituto Superior Tecnico, Universidade de Lisboa, Portugal

$^5$ NOVA Laboratory for Computer Science and Informatics (NOVA LINCS),
Universidade Nova de Lisboa,
2829-516 Caparica, Portugal

$^\star$ Corresponding author: The Anh Han<EMAIL_ADDRESS>
§ ABSTRACT

With the introduction of Artificial Intelligence (AI) and related technologies in our daily lives, fear and anxiety about their misuse as well as the hidden biases in their creation have led to a demand for regulation to address such issues.  Yet blindly regulating an innovation process that is not well understood, may stifle this process and reduce benefits that society may gain from the generated technology, even under the best intentions. In this paper, starting from a baseline model that captures the fundamental dynamics of a race for domain supremacy using AI technology, we demonstrate how socially unwanted outcomes may be produced when sanctioning is applied unconditionally to risk-taking, i.e. potentially unsafe, behaviours. As an alternative to resolve the detrimental effect of over-regulation, we propose a voluntary commitment approach wherein technologists have the freedom of choice between independently pursuing their course of actions or establishing binding agreements to act safely,  with sanctioning of those that do not abide to what they pledged. Overall, this work reveals for the first time how voluntary commitments, with sanctions either by peers or an institution, leads to socially beneficial outcomes in all scenarios envisageable in a short-term race towards domain supremacy through AI technology.  These results are directly relevant for the design of governance and regulatory policies that aim to ensure an ethical and responsible AI technology development process.

Keywords: Evolutionary Game Theory, AI development race, Commitments, Incentives, Safety.

§ INTRODUCTION

With the rapid advancement of  AI and related technologies, there has been significant fear and anxiety about their potential misuse as well as the social and ethical consequences that may result from biases within the design of such systems [45, 5, 42].
While expectations associated with these advanced technologies increase and monetary profits stimulate rapid deployment, there is a serious risk for taking unethical or risky short cuts to enter a market first with the next innovation, ignoring safety checks and ethical development procedures.
As different disagreeable examples have emerged [12], governments and regulating bodies have been catching up by debating new forms and frameworks for regulating this technology     [4, 8, 43]; notably, the recent EU White Paper on AI.
Such debates have produced proposals for mechanisms on how to avoid, mediate, or regulate the development and deployment of  AI  [4, 8, 19, 40, 24, 47, 35, 43, 2, 38]. Essentially, regulatory measures such as restrictions and incentives are proposed to limit harmful and risky practices in order to promote beneficial designs [4].  Examples of such approaches [4] include financially supporting the research into beneficial AI [34] and  making AI companies pay
fines when found liable for the consequences of harmful AI [20].

Although these regulatory measures may provide solutions for particular scenarios, one needs to ensure that they do not overshoot their targets since over-regulation could stifle innovation,  potentially hindering investments into the development of novel innovations as they become too risky endeavors [21, 32].
Worries  have been expressed by different organisations/societies that too strict policies may unnecessarily affect the benefits and societal advances that novel AI technologies may have to offer [18].
Regulations affect moreover big and small tech companies differently: A highly regulated domain makes it more difficult for small new start-ups, introducing an inequality and dominance of the market by a few big players [32]. It has been emphasised that neither over-regulation nor a laissez-faire approach suffices when aiming to regulate AI technologies [15]. In order to find a balanced answer, one clearly needs to have first an understanding of how a competitive development dynamic may work.

Starting  from a baseline game theoretical model, referred to as the DSAIR model [28] (a model of domain supremacy through an AI race), which defines the process through which multiple stake-holders aim for market supremacy,  we demonstrate first that  unconditional sanctioning will negatively influence social welfare in certain conditions of a short-term race towards domain supremacy through AI technology.  Afterwards, we examine the DSAIR model with an alternative mechanism for resolving the issue that can be shown to lead to less detrimental effects. Our approach is to allow technologists or race participants to voluntarily commit themselves to safe innovation procedures, signaling to others their intentions. Specifically, this  bottom-up, binding  agreement (or commitment) is established for those who want to take a safe choice,  with sanctioning applied to violators of such an agreement.

Previous Developments.
As was shown in [28]  participants either follow safety precautions (the SAFE option) or ignore them (the UNSAFE option) in each step of the development process of the DSAIR model. The main assumption in the model was that it requires more time and more effort to comply with the precautionary requirements, making the SAFE option not only costlier, but  also slower compared to the UNSAFE option.  Accordingly, it was assumed that in playing SAFE, participants must pay a cost $c > 0$, whereas playing UNSAFE costs nothing.  Furthermore, whenever playing UNSAFE, the development speed differs and is $s > 1$  whereas in playing SAFE the speed is simply normalised to 1.
Decisions to act SAFE or UNSAFE in AI development are repeated until one or more teams attain the designated objective, which can be translated into having completed $W$ development steps, on average [28]. As a result, they earn a large benefit or prize $B$ (e.g. windfall profits [38]), equally shared among those  reaching the target at the same time. A development disaster or a setback  may however come to occur with some probability, which is presumed to increase with the number of times that safety requirements were ignored by the winning team(s) at each step.
Whenever a disaster of this kind occurs, all the benefits of a risk-taking participant are lost.  This risk probability is denoted by $p_r$ (see the Models and Methods section for more details).

Behavioural regions (zones) as identified in [28] when the time to reach domain supremacy is short. Region II: Inside the plots, the two solid lines  delimit the boundaries wherein the collective prefers safety compliant behavior, yet unsafe development is individually preferred. Regions III and I exhibit where unsafe (respectively, safe) development is both the individually and collectively preferred outcome

It was observed in the DSAIR model that, in case the time-scale to reach the target is short, so that over the whole of the development process the average of the accumulated benefit is much smaller than the  final benefit $B$, only for a certain window of parameter settings societal interest conflicts with the individual ones: In that region, individual unsafe behaviour dominates, despite that safe development would lead to a larger collective outcome or social welfare (cf. region II in Figure <ref>).
From the regulatory perspective, it is only region II that thus requires governance in order to promote or enforce safe actions in order to avoid any disaster that may occur during the technology development race.

A peer punishment mechanism against unsafe behaviour was proposed in order to mediate the behavior in that region (without affecting the desirable safe outcome in region I) [25]. It however may lead to a reduction of the societal welfare that can be obtained in region III (see Figure <ref>) where the desired unsafe (risk-taking/innovative) behaviour becomes significantly reduced whenever punishment is not very costly to the punisher while strongly affecting the punished.  The problem with applying system-wide sanctioning for any risk-taking behavior is that it does not take into account the region wherein the AI innovation is taking place, as was visualised by Figure <ref>. That is, if it is the case that a development race falls in region III (low risk and innovation is collectively beneficial and preferred), unsafe behaviour should not be punished as it is beneficial for the overall social welfare. To enact such a region-dependent targeted punishment scheme, one would require the ability to estimate exactly the risk level associated with each AI development scenario, i.e. knowing beforehand the risk as well as speed of development. Clearly that may not  be easy due to the lack of data on how such an AI innovation dynamics works.  This is especially true  in the early stages of the development and adoption of many new technologies, which has become known as the so-called  double-bind problem [13]: the impact of technologies including  AI ones  cannot be predicted until it becomes a reality, and controlling or changing it is difficult or no longer possible at that point.

In this article an alternative solution is proposed that circumvents the need of being able to estimate correctly the speed of development and risk in order to appropriately regulate UNSAFE behavior: race participants can choose whether or not to establish a bilateral commitment to act safely, which also exposes them to a sanction in case they do not uphold their commitment.  While, on the one had, the development teams are allowed to work in an UNSAFE manner without repercussions if they do not commit; a prior agreement allows, on the other hand, the safety compliant participants to identify easily unsafe ones while having also the capacity to punish the dishonest ones (who might also be sanctioned by an external party such as a   regulating  institution).

Our results reveal that this freedom of choice, if enabled through a prior bilateral commitment to SAFE actions, can, on the one hand, avoid  over-regulating unsafe behaviour and, on the other, improve significantly the desired safety outcome in the dilemma zone when compared to punishment alone. Interestingly, this type of binding pledges has been argued to be of relevance within other types of global conflicts and dilemmas, such as  environmental governance [3, 11].

§ MODELS AND METHODS

We first recall the DSAIR model with pairwise interactions then extend it with the option to bilaterally commit to acting safely and associated punishment of violations of such commitments.
The Evolutionary Game Theory (EGT) methods being used to analyse the models will then be described.

§.§ Summary of the DSAIR model and prior results

The DSAIR model [28] was originally defined as a two-player game repeated with a certain probability, consisting thus on average of $W$ rounds[An N-player version of this game was discussed also in [28].  Yet in order to keep things easy to access, we focus here on the two-player scenario.].
At each round of development,  players gather benefits arising from their intermediate AI developments, subject to whether or not they chose to act UNSAFE or SAFE. Presuming some fixed benefit, $b$, resulting from the AI market, the teams will share this gain proportionately to their development speed.
Accordingly, at every round of the race one can write a payoff matrix denoted by $\Pi$ with respect to row players $i$, whose entries are denoted by $\Pi_{ij}$ ($j$ corresponding to some column),  as shown

\begin{equation}
\Pi =  \bordermatrix{~ & \textit{SAFE} & \textit{UNSAFE}\cr
\textit{SAFE} & -c + \frac{b}{2} &-c +  \frac{b}{s+1}      \cr
\textit{UNSAFE} &  \frac{s b}{s+1}   & \frac{b}{2}   \cr
\end{equation}

The payoff matrix can be explained as follows. Firstly, wherever there is an interaction between two players selecting the SAFE action, each shall pay a cost $c$ and the resulting benefit $b$ is shared. Differently, whenever interaction is between two players selecting the UNSAFE action, they shall share benefit $b$ without having had to pay cost $c$.
Whenever an UNSAFE choice is matched with a SAFE one, the SAFE choice necessitates a cost $c$ and receives a (smaller) part $b/(s+1)$ of  $b$, whereas the UNSAFE choice collects a larger  $sb/(s+1)$ whilst not ever having had to pay $c$.  Note that $\Pi$ is a simplification of the matrix defined in [28] for, in the current time-scale, it was  shown that the parameters as defined here sufficiently explain the obtained results.

We analyse the evolutionary outcomes of this game in a well-mixed and finite population consisting of $Z$ players. Given the choices each player can make and the fact that these choices need to be repeated for $W$ round, each adopts player one of the two following strategies [28]:

* AS: complies every time with safety precautions, acting thus SAFE in every round.

* AU: complies not once with safety precautions, acting thus UNSAFE in every round.

The averaged payoffs for AS vs AU are expressed by payoff matrix

\begin{equation}
\bordermatrix{~ & \textit{AS} & \textit{AU} \cr
\textit{AS} & \frac{B}{2W} +\Pi_{11} & \Pi_{12}    \cr
\textit{AU} &  p \left(\frac{sB}{W} + \Pi_{21}\right)   &  p \left(\frac{sB}{2W} +\Pi_{22}\right)  \cr
%    \textit{CS} &  \frac{B}{2W} +\Pi_{11}    & \frac{s}{W}  \left( \Pi_{12} + (\frac{W}{s} - 1) \Pi_{22}  \right) &  \frac{B}{2W} +\Pi_{11}    \cr
\end{equation}

wherein,  for presentation purposes alone, let us denote $p = 1 - p_r$ (note that $p_r$ was explained in the Introduction section).

As has been shown in [28], by contemplating where AU is risk-dominant against AS (cf. Methods below), then three distinct regions are identifiable within the parameter space $s$-$p_r$ (cf. Figure <ref>): (I)  if $ p_r > 1-\frac{1}{3s}$, AU is risk-dominated by  AS:   safety compliance affords both the collectively preferred  outcome and the one evolution selects; (II) if $1-\frac{1}{3s} > p_r > 1-\frac{1}{s}$:  even though  safety compliance is the more desirable strategy for ensuring the highest collective outcome,  the  social learning dynamics     leads  the population to the state within which safety precautions have been mostly ignored;  (III) if $p_r <  1-\frac{1}{s}$ (AU is risk-dominant against  AS),  then unsafe development is both collectively preferred and selected by the social learning dynamics.

So it is important to remember for the rest of the paper that UNSAFE actions are preferred and established in zone III, SAFE actions are preferred and established in zone I and a conflict exists between the individual and the collective in zone II as the former prefers UNSAFE actions and the latter SAFE actions.

Behavioural dominance in different zones for varying $p_r$, in presence of prior commitments  (top row: panels a, b) and comparison of its overall unsafe behaviour against when there are no commitments (bottom row: panels c, d). The black dotted lines in panels a and b indicate the total unsafe frequency (i.e. the sum of AU-in and AU-out frequencies). The desired collective behaviour is indicated for each zone (i.e. unsafe in zone III and safe in zones I and III). We show results for  two important scenarios: when efficient punishment can be made for a small cost (left column: $s_\alpha = 0.3$, $s_\beta = 1$) and when punishment is not highly efficient  (right column:, $s_\alpha = 1$, $s_\beta = 1$).

Parameters:  $b = 4$, $c = 1$, $s = 1.5$, $W = 100$, $B = 10^4$, $\beta = 1$,  $Z = 100$.

§.§ Introducing commitment strategies

We now extend the DSAIR model with strategies that can bilaterally commit to safety compliant behavior: Before an interaction, participants can commit to play SAFE in each round. The commitment stands when all parties agree.
The players can refuse to commit, preferring to proceed without being pushed into the safe direction and being able to take risks. Those that committed but later select the UNSAFE action are potentially subject to sanctioning. Two sanctioning scenarios are considered here: (a) peer punishment (PP), which is performed by the co-player who kept her side of the deal, and (b) institutional punishment (IP), which is performed by a third-party that is not actively participating in the race for supremacy in some domain through AI (e.g. the European Union or United Nations). Each player has the freedom of behavioural choice whereby those who do not commit will not be punished when playing UNSAFE in the DSAIR model. We call the latter behavior "honest" unsafe behavior whereas those that act unsafely after committing are referred to as "dishonest".

Sanctioning an opponent who  played UNSAFE in a previous round consists in  imposing  a  reduction $s_\beta$ on the opponent's speed  [25]. In case of PP, the punishing player also incurs a  reduction $s_\alpha$ on her own speed.
Committing may also be costly for all that do as they give up other choices. A commitment cost $\epsilon$ (per round) is thus introduced for those performing this pre-play action.

Transitions and stationary distributions when agreement is present (top row) against  when it is absent (bottom row), for three regions. For clarity, only stronger transitions (than the ones in the opposite directions) are shown; no transition either way means neutral.  The choice of $s_\alpha$ and $s_\beta$ values were chosen to illustrate the main difference  between with vs without commitment scenarios, in the three zones.
Parameters: $b = 4$, $c = 1$, $s = 1.5$, $W = 100$, $B = 10^4$, $\beta = 1$,  $Z = 100$.

With the possibility of joining or not a commitment to behave safely and sanctioning dishonest unsafe behaviour, one can now  define the possible strategies.  AS and AU (as defined above) can either commit to safe actions, and furthermore, when involved in a commitment, decide whether to punish a dishonest  co-player. If no commitment can be made, the player will select the UNSAFE action[We also consider the version where these players unconditionally play SAFE regardless of the commitment. The safety outcomes in all regions are similar, just the strategies' dominance is slightly different. Results are provided in Appendix (see Figures A3 and A4).]. The choices listed before lead to five strategies:

* AS-in: willing to commit,  plays SAFE when commitment is in place and UNSAFE otherwise, but does not use (costly) punishment. This strategy can be considered  a second-order free-rider on the punishment effort of others;

* AS-out: does not commit, but always selects the SAFE action;

* AU-in: claiming to commit to the SAFE action, but always plays UNSAFE in the interaction. This strategy makes a  commitment, trying to exploit safe players who only want to interact under an agreement;

* AU-out: does not commit and plays  UNSAFE in the interaction. This strategy wants to freely take risk or innovate without any repercussions otherwise imposed by a commitment to follow SAFE actions;

* PS: willing to commit and plays SAFE when the other player also commits; plays UNSAFE otherwise,  and also punishes an UNSAFE action of a co-player that committed with her. This strategy is only present  in the case of PP, and is not present in the IP sanctioning model.

§.§ Evolutionary Dynamics of Finite Populations

Herein are adopted the methods of EGT for finite populations [41, 44, 30], whether in the analytical or numerical results obtained here.   In such settings,  the payoffs of players stand for their social success or fitness, and  the evolutionary dynamics is shaped by social learning, in accordance to which  the players that are most successful  will tend more often to be copied  by other players. The so-called pairwise rule of comparison is utilised to model social learning      [44],  which ensures   a player $A$ with fitness $f_A$ resorts to adopt the strategy of player $B$ with fitness $f_B$ with a probability established by the Fermi function,
$P_{A,B}=\left(1 + e^{-\beta(f_B-f_A)}\right)^{-1}$,
where  the intensity of selection is conveniently described by $\beta$.
In a population wherein several strategies are in co-presence, their long-term frequency can  be  computed simply by calculating  the stationary distribution of a Markov chain the states of which represent each  strategy.
Absent behavioural exploration  or mutations, the end states of evolution are inevitably monomorphic.    Meaning whenever
such a state is reached, escape by imitation is impossible. Hence, we presume further that, given some mutation probability,  each agent may     freely explore its behavioural space (which consists of the two actions, UNSAFE  and SAFE, in our case), by randomly adopting an action as a result of mutation.  At the limit of a small  probability of mutating,    the population is comprised  of at most one of two strategies at any time. Accordingly,  the social dynamics is describable utilising a Markov Chain, in which each state  represents a monomorphic population whose transition probabilities to another state are expressed by the fixation probability of one single mutant [31, 36].    The  Markov Chain's stationary distribution depicts the average time   the whole population spends at each monomorphic end state (see the examples  in Figure <ref> for illustration).

Let $\pi_{X,Y}$ denote the payoff some strategist $X$ gathers from a pairwise interaction with some strategist $Y$ (as defined in the payoff matrix). Assume there exist two strategies at most in the population, for example $k$ agents using  strategy A ($0 \leq k \leq Z$)  and $(Z-k)$ agents using instead strategy B. Hence, the (average) payoff of agents  using  A and B can be formulated, respectively, as

\begin{equation}
\label{eq:PayoffA}
\begin{split}
\Pi_A(k) &=\frac{(k-1)\pi_{A,A} + (Z-k)\pi_{A,B}}{Z-1},\\
\Pi_B(k) &=\frac{k\pi_{B,A} + (Z-k-1)\pi_{B,B}}{Z-1}.
\end{split}
\end{equation}

As a result, at each step in time, the probability    of changing by $\pm$1 of the number of  $k$ agents using strategy A is specified as [44]

\begin{equation}
T^{\pm}(k) = \frac{Z-k}{Z} \frac{k}{Z} \left[1 + e^{\mp\beta[\Pi_A(k) - \Pi_B(k)]}\right]^{-1}.
\end{equation}

The fixation probability of a single mutant adopting strategy A,in a population $(Z-1)$ of  agents    adopting B, is defined by [44, 36]

\begin{equation}
\label{eq:fixprob}
\rho_{B,A} = \left(1 + \sum_{i = 1}^{Z-1} \prod_{j = 1}^i \frac{T^-(j)}{T^+(j)}\right)^{-1}.
\end{equation}

When considering a set  $\{1,...,s\}$ of different strategies, such probabilities of fixation  define the Markov Chain transition matrix $M = \{T_{ij}\}_{i,j = 1}^s$, with $T_{ij, j \neq i} = \rho_{ji}/(s-1)$ and  $T_{ii} = 1 - \sum^{s}_{j = 1, j \neq i} T_{ij}$. The normalized eigenvector    of the transposed matrix of $M$ associated with eigenvalue 1 produces the above defined  stationary distribution  [31], depicting the relative time the population stays adopting each of the strategies.

§.§ Risk-dominance

A major standpoint of comparison of  two strategies A and B is in which direction the transition is more probable or stronger, the one of some B mutant  fixating in a population of agents that employ A, $\rho_{A,B}$,  or that of a mutant A fixating in the population of agents that employ B, $\rho_{B,A}$.    At the limit, for a large enough population size (i.e. a large $Z$), the condition  simplifies to [41]

\begin{equation}
\label{eq:risk_dominance Equation}
\pi_{A,A} + \pi_{A,B} > \pi_{B,A} +  \pi_{B,B}.
\end{equation}

Frequency of unsafe  in all regions for varying $s_\alpha$ and $s_\beta$, when commitment is absent (top row) vs when it is  in use (bottom row).
Parameters:  $b = 4$, $c = 1$, $s = 1.5$, $W = 100$, $B = 10^4$, $\beta = 1$,  $Z = 100$.

§ RESULTS

We first focus on analysing the system of self-enforcing bilateral commitment (PP)  then showing the results  of bilateral commitment complemented with institutional enforcement (IP).

§.§ Self-enforcing bilateral commitments

In Figure <ref> (top row), we show the stationary distributions of the five strategies for varying $p_r$ across the three zones I, II and III.
We show results for  two important scenarios: when efficient punishment can be made for a small cost (left column: $s_\alpha = 0.3$, $s_\beta = 1$) and when punishment is not highly efficient  (right column:, $s_\alpha = 1$, $s_\beta = 1$). Punishment is considered efficient when its effect  ($s_\alpha$) on the development speed of the player  performing the punishment is significantly smaller  than the  effect ($s_\beta$) it has on the player undergoing the punishment.

In both cases, AU-out dominates when $p_r$ is small (zone III), PS dominates when it is intermediate (first part of zone II),  while AS-out dominates when it is large (part of zone II and zone I). It is important to notice that these results reflect the most desirable outcomes for all three zones: risky innovation in III and safety compliance in I and II (see the black line in the bottom row of Figure <ref>). Moreover,  a more efficient punishment leads to better safety outcome in zone II (see panels c and d). It is worth noting that inefficient punishment may result in the presence of AU-in strategists, in other words cheaters. It is thus important in this self-organised commitment solution that participants have an effective mechanism to punish commitment violators. One solution could be directly affecting the player's publicly available reputation.

To clarify the benefit of voluntary bilateral commitment, Figure <ref> (bottom row) compares the overall frequency of  unsafe/risk-taking when commitment is possible (black line) vs when it is absent (red line). In the absence of bilateral commitments,
over-regulation  occurs, i.e. safety compliance is abundant but not desired for a large part of zone III when efficient punishment can be carried out for a small cost (panel c), while a safety dilemma occurs (i.e. safety compliance is desired but infrequent) for a large part of zone II when sanctioning is not highly efficient.

These observations can be better understood by examining
the transitions and stationary distributions in Figure <ref>.
In region III (first column, low risk, where risky innovation is desired), low-cost highly-efficient punishment can lead to significant reduction of innovation since PS dominates AU in absence of bilateral commitments (panel d). Addition of AU-out provides an escape as this strategy is not punished since it never commits to safe course of actions, thereby dominating PS (panel a).  In region II (dilemma zone, safety is wanted but needs to be enforced), when punishment is not highly efficient, there is a large amount of unwanted AU in the population (panel e). As AU-out is dominated by PS, since PS plays also UNSAFE when not having a commitment partner, it leads to a lower frequency of  unsafe behaviour (comparing panels b and e). In region III, addition AU-out does not change the desired outcome of the dominance of safety compliance.

In Figure <ref>, we show that these remarkable observations regarding voluntary bilateral commitment are robust for different regimes of punishment effectiveness. In particular, when comparing the unsafe frequency with commitments against when it is absent for varying  $s_\alpha$ and $s_\beta$.
In region III, over-regulation occurs when there is only punishment whenever $s_\alpha$ is sufficiently small and $s_\beta$ is large (purple area, top row, first column), but when an agreement is in place that is not the case  (bottom row, first column). In  region II with the lower range of $p_r$ (second column, $p_r = 0.6$), unsafe frequency is lower in the latter in for most $s_\alpha$ and $s_\beta$. In the higher range  of region II and region III (third and fourth columns, with $p_r = 0.75$ and $p_r = 0.9$), desired safe behaviour is dominant  in both cases.

It's noteworthy that all results are robust also for other regimes of weaker or stronger selection intensities, i.e.  $\beta = 0.1 $  or $\beta = 10$ (see e.g. Figures <ref> and A2 in Appendix).

§.§ Institutionally governed bilateral commitments

When assuming that it is an institution that governs the dynamics of the race to supremacy through AI in some domain, punishing unsafe behaviour if a bilateral commitment is violated. Similarly to PP, IP reduces the development speed of a participant that selected the UNSAFE action by $s_\beta$.

On the one hand, when no bilateral commitments can be made, the population consists of two strategies, AS and AU, where AU's speed will be $s - s_\beta$ when being punished (by the institution). On the other hand, when  voluntary commitments are possible, there will only be the four strategies,  AS-in, AU-in, AS-out, and AU-out. There is no PS strategy as there is no peer sanctioning, which implies also no parameter $s_\alpha$  given that the institution is not part of the population[Future work might look at how to minimise the cost and efforts from the institution while  ensuring a desired level of safety compliance  [29, 22, 9, 48, 14]].
Concretely, the institution can only sanction AU-in when both players committed. In that case, its speed will be reduced, from $s$ to $s -s_\beta$. AU-out as before will be free of any sanction.

In Figure <ref>, we compare the effect of institutional sanctioning with and without the option of having prior voluntary bilateral commitments between the two players. The outcomes are similar to the case of PP discussed in the previous section. In particular, one can observe that the presence of commitments can significantly improve safety outcome in region II without being detrimental to the desired behaviour and thus social outcome in the other two regions. Without commitments, institutional punishment leads again to over-regulation of unsafe/innovation in region  III, which, as was mentioned before, is not a desired outcome. Moreover, with a smaller  $s_\beta$ (compare  top and bottom rows of the figure), the outcomes remain  similar, with a  lower level of  over-regulation but weaker results for region II.

Institutional punishment, with (left column) or without a safety agreement (right column). We report the frequency of unsafe behaviour (UNSAFE) in a population of AU and AS in the latter case and a population of AS-in, AU-in, AS-out, AU-out, in the former case. An institutional punishment reduces the speed of AU in the latter and AU-in (when an agreement is formed) in the former, by $s_\beta = 1$ (top row) and by $s_\beta = 0.5$ (bottom row). The lines within the figures are as in Figure 1.   Parameters:  $b = 4$, $c = 1$, $W = 100$, $B = 10^4$, $\beta = 1$,  $Z = 100$.

In short, we observe that freedom of choice between bilateral commitments to safety regulations (which can be self-enforcing or institutionally enforced) and unilateral risky/unsafe endeavours provides an efficient solution to ensure desirable outcomes in all scenarios of the DSAIR model. First, it alleviates the problem of over-regulation when the risk is low (which happens when punishment can be very efficient), while at the same time it  reduces the frequency of unsafe behaviour in the dilemma zone (region II). Furthermore,  the desirable safety compliance outcomes in region I remain unaffected.

§ DISCUSSION AND CONCLUSIONS

This paper proposes and analyses, using a multi-agent and population dynamics modelling approach,  a novel solution that promotes desirable safety compliance in a technology development race, while at the same time avoiding stifling beneficial innovation due to over-regulation. We base our study on  a previously proposed EGT model  [28] that describes  the dynamics of a competition between safety and risk-taking (unsafe) behaviours, within  an (alleged) race for  supremacy through AI in a marketable domain.
We show that, by allowing race participants the freedom of choosing to enter or not in bilateral commitments to act safely and avoid risks, accepting thus to be sanctioned in case of misbehavior,  high levels of the most beneficial behaviour as a whole, are achieved in all regions of the parameter space.

This system of voluntary bilateral commitments, either depending on sanctioning actions of peers or by an institution, provides a mechanism to overcome the problems associated with over-regulation, which  might occur whenever risk taking behaviour, which may be perceived as unsafe, is unconditionally penalised without taking into account the true risk level (in relation to the  cost and benefit of the safety compliance and of the opting for risky behaviour), as shown in  [25].
On the one hand, allowing for participants to explicitly commit to safety precautions in their development process ensures higher levels of safety compliance when that is desired, because then unsafe/risk-taking strategies become either clearly identified  (AU-out) and suitably coped with, or punished through the binding commitment (AU-in).  On the other hand, the commitment option enables the (honest) risk-taking behaviour (namely, AU-out) to prevail  whenever it is collectively preferred, since then it is not punished, as a result of not joining the commitment in the first place.

There have been several theoretical modelling studies based on EGT,  showing the benefit of establishing  prior commitments or agreements for promoting the evolution of certain positive behaviours, such as  cooperation and coordination in a population of self-regarding agents, see e.g.    [27, 39, 23, 37]. Behavioral  experiments with human subjects have also been performed showing the promoting role of commitments for cooperative behaviours, see e.g.   [10, 11]. In all such contexts, it is well-defined what is the correct (positive) behaviour most beneficial collectively and thus the one that should be promoted.
In our case that no longer holds, since whether or not a  behaviour is the most beneficial collectively depends on the behavioural region in which the race occurs at the time. This region is not well defined, especially when data is not abundantly available.  For example, we might need to wait for a technology to be audited and even largely adopted by users to know the level of risk associated with a particular technology (but then it might be too late already to provide pertinent regulations). Such level of uncertainty can also lead to other externalities, such as the emergence of strongly polarized positions [17], an effect not yet characterized by our model.

Interestingly, we show here that establishing the possibility of a voluntary bilateral commitments is as well highly effective in promoting collectively desired/preferred behaviour in all cases.
Commitment to cooperative actions in social dilemmas work because they allow cooperators to avoid free-riding strategies, for the case of an AI race, because they permit players the freedom to follow their preferred course of development without being punished (as long as they are being honest). One can also see it as a particular form of binding signal or pledge. The results of this work are in line with  Hadfield's arguments [21] in the general context of technology globalisation; that is, legal contracts and  institutions should allow freedom (via coordination) instead of exacting it from individuals in a population. For the sake of clarity, here we illustrate this idea with a pairwise race model; however, as shown in [28], this model can be easily extended to N-player interactions, with a concomitant increase in complexity, yet leading to the same qualitative messages. Future work may explicitly consider N-player commitments and more complex communication and bottom-up dynamics typical of this type of agreements, see e.g. [23, 26, 46].

Last but not least, in [38] the authors propose the so-called  Windfall Clause for mediating the tension in AI competition.  It is proposed there to arrange prior commitments or agreements from the race participants (e.g., companies, governments) for sharing a large part of the windfall benefit (i.e. the large prize $B$ in our model) with society. As discussed in that paper, it requires significant legal infrastructures to be put in place to enable  such a mechanism. Our approach does not so require because legal contracts can be used to ensure  compliance with prior safety agreements.

It is noteworthy that, although we focus  in this paper on an AI development race, the model proposed can be  more generally applicable to other kinds of long-term  situations of competition, such as the development of technological innovation  and its racing for patents in which there is a significant advantage (i.e. a large $B$) to be gotten in being one of the first ever to reach an important target [16, 7, 33].
Other important domains involve pharmaceutical and vaccines  development race, where companies might attempt to cut a few corners by not  following strictly the safe clinical trial protocols, with the view to be the first ones to develop and put on the market some pharmaceutical product, in order to reap the highest possible share of benefit in the market [1, 6]. However,  certain aspects or factors of the current model might need to be revised.
For example,  the large profit  $B$ for a vaccine development race would be more difficult to achieve than in the case of AI, since a developed vaccine needs to be approved by suitable authorities before users have trust in it and use it.
Vaccine developers only can generate the profit $B$ if they could show evidence of the safety of the vaccine and that they have closely followed  all the safety procedures.
In case of AI, the winning developers  can simply deploy the technologies and get the profit B (in most cases). One can expand the current model to capture also the vaccine race (and also generalised the current AI race model) by for example having a new parameter to capture when $B$ can be generated for the race winners, which depends on the nature of the race (e.g. vaccine vs AI) and also the frequency of safety compliance in the past. our future works will address  these issues.

§ ACKNOWLEDGEMENTS

T.A.H., L.M.P., T.L. and T.C., are supported by Future of Life Institute grant RFP2-154.  L.M.P. acknowledges support from FCT/MEC NOVA LINCS PEst UID/CEC/04516/2019. F.C.S. acknowledges support from FCT Portugal (grants PTDC/EEI-SII/5081/2014, PTDC/MAT/STA/3358/2014). T.L. acknowledges support by the FuturICT2.0 (www.futurict2.eu) project funded by the FLAG-ERA JCT 2016. T.A.H. is also supported by a Leverhulme Research Fellowship (RF-2020-603/9).

§ APPENDIX

§.§ Effect of larger intensity of selection
See Figures A1 and A2.

§.§ Other scenarios of modelling

See Figures A3 and A4.

Same as  Figure 2,  with other values of  $\beta$.

Frequency of unsafe  in all regions for varying $s_\alpha$ and $s_\beta$, when safety agreement is  in use, for different values of intensities of selection $\beta$.  Similar observations as in the main text.   Parameters:  $b = 4$, $c = 1$, $s = 1.5$, $W = 100$, $B = 10^4$,   $Z = 100$.

When AU-in  always plays SAFE in an interaction. Behavioural dominance of in different zones for varying $p_r$, in presence of prior commitment  (top row: panels a, b) and comparison of its overall unsafe behaviour against when commitment is absent  (bottom row: panels c, d). We show results for  two important scenarios: when efficient punishment can be made for a small cost (left column: $s_\alpha = 0.3$, $s_\beta = 1$) and when punishment is not highly efficient  (right column:, $s_\alpha = 1$, $s_\beta = 1$).
In both cases, AU-out dominates when $p_r$ is small (zone III), PS dominates when it is intermediate (first part of zone II),  while PS and AU-in  together dominate when it is large (part of zone II and zone I). That results in the fact that when a commitment is present desirable outcomes are achieved in all three zones: unsafe/innovation in III and safe in I and II, see bottom row. In absence of a commitment,
over-regulation  occurs (i.e. safe is abundant but not desired) for a large part of zone I when efficient punishment can be done for a small cost (panel c), while safety dilemma occurs (i.e. safe is desired but infrequent) for a large part of zone II when punishment is not highly efficient.   Parameters:  $b = 4$, $c = 1$, $s = 1.5$, $W = 100$, $B = 10^4$, $\beta = 1$,  $Z = 100$.

When AU-in  always play SAFE in an interaction. Transitions and stationary distributions when a commitment is present (top row) against  when it is absent (bottom row), for three regions (Only stronger transitions are shown; no transition either way means neutral). In region III (first column, low risk, where UNSAFE or innovation is wanted), low-cost highly efficient punishment can lead to significant reduction of innovation since PS dominates AU in absence of a commitment (panel d). Addition of AU-out provides an escape as this strategy is not punished, thereby dominating  PS (panel a).  In region II (dilemma zone, safety is wanted), when punishment is not highly efficient, there is still some amount of unwanted AU in the system. As AU-out is dominated by PS since PS plays UNSAFE in absence of an commitment, it leads to lower unsafe behaviour. In region III, addition AU-out does not change the desired outcome of safety.  Parameters: $b = 4$, $c = 1$, $s = 1.5$, $W = 100$, $B = 10^4$, $\beta = 1$,  $Z = 100$.

[1]
Abbott, F. M., Dukes, M. N. G., and Dukes, G. (2009).
Global pharmaceutical policy: ensuring medicines for tomorrow's
Edward Elgar Publishing.

[2]
Askell, A., Brundage, M., and Hadfield, G. (2019).
The Role of Cooperation in Responsible AI Development.
arXiv preprint arXiv:1907.04534.

[3]
Barrett, S. (2003).
Environment and statecraft: The strategy of environmental
treaty-making: The strategy of environmental treaty-making.
OUP Oxford.

[4]
Baum, S. D. (2017).
On the promotion of safe and socially beneficial artificial
AI & Society, 32(4):543–551.

[5]
Bostrom, N. (2014).
Superintelligence: paths, dangers, strategies.

[6]
Burrell, R. and Kelly, C. (2020).
The covid-19 pandemic and the challenge for innovation policy.
Available at SSRN 3576481.

[7]
Campart, S. and Pfister, E. (2014).
Technological races and stock market value: evidence from the
pharmaceutical industry.
Economics of Innovation and New Technology, 23(3):215–238.

[8]
Cave, S. and ÓhÉigeartaigh, S. (2018).
An AI Race for Strategic Advantage: Rhetoric and Risks.
In AAAI/ACM Conference on Artificial Intelligence, Ethics and
Society, pages 36–40.

[9]
Chen, X., Sasaki, T., Brännström, Å., and Dieckmann, U. (2015).
First carrot, then stick: how the adaptive hybridization of
incentives promotes cooperation.
Journal of The Royal Society Interface, 12(102):20140935.

[10]
Chen, X.-P. and Komorita, S. S. (1994).
The effects of communication and commitment in a public goods social
Organizational Behavior and Human Decision Processes,

[11]
Cherry, T. L. and McEvoy, D. M. (2013).
Enforcing compliance with environmental agreements in the absence of
strong institutions: An experimental analysis.
Environmental and Resource Economics, 54(1):63–77.

[12]
Coeckelbergh, M. (2020).
AI ethics.
MIT Press.

[13]
Collingridge, D. (1980).
The social control of technology.
New York : St. Martin's Press.

[14]
Couto, M. C., Pacheco, J. M., and Santos, F. C. (2020).
Governance of risky public goods under graduated punishment.
Journal of Theoretical Biology, 505:110423.

[15]
Dawson, D., Schleiger, E., Horton, J., McLaughlin, J., Robinson, C., Quezada,
G., Scowcroft, J., and S, H. (2019).
Artificial Intelligence: Australia’s Ethics Framework.
Technical report, Data61 CSIRO, Australia.

[16]
Denicolò, V. and Franzoni, L. A. (2010).
On the winner-take-all principle in innovation races.
Journal of the European Economic Association, 8(5):1133–1158.

[17]
Domingos, E. F., Grujić, J., Burguillo, J. C., Kirchsteiger, G., Santos,
F. C., and Lenaerts, T. (2020).
Timing uncertainty in collective risk dilemmas encourages group
reciprocation and polarization.
iScience, 23(12):101752.

[18]
EDRI (2021).
Civil society calls for AI red lines in the European Union’s
Artificial Intelligence proposal.
Technical report, European Commission.
Accessed January-29-2021.

[19]
Geist, E. M. (2016).
It's already too late to stop the ai arms race: We must manage it
Bulletin of the Atomic Scientists, 72(5):318–321.

[20]
Gurney, J. K. (2013).
Sue my car not me: Products liability and accidents involving
autonomous vehicles.
U. Ill. JL Tech. & Pol'y, page 247.

[21]
Hadfield, G. K. (2017).
Rules for a flat world: why humans invented law and how to
reinvent it for a complex global economy.
Oxford University Press.

[22]
Han, T. A., Lynch, S., Tran-Thanh, L., and Santos, F. C. (2018).
Fostering cooperation in structured populations through local and
global interference strategies.
In IJCAI-ECAI'2018, pages 289–295.

[23]
Han, T. A., Pereira, L. M., and Lenaerts, T. (2017a).
Evolution of commitment and level of participation in public goods
Autonomous Agents and Multi-Agent Systems, pages 1–23.

[24]
Han, T. A., Pereira, L. M., and Lenaerts, T. (2019).
Modelling and Influencing the AI Bidding War: A Research Agenda.
In Proceedings of the AAAI/ACM conference AI, Ethics and
Society, pages 5–11.

[25]
Han, T. A., Pereira, L. M., Lenaerts, T., and Santos, F. C. (2021).
Mediating Artificial Intelligence Developments through Negative and
Positive Incentives.
PLOS ONE, 16(1):e0244592.

[26]
Han, T. A., Pereira, L. M., Martinez-Vaquero, L. A., and Lenaerts, T. (2017b).
Centralized vs. personalized commitments and their influence on
cooperation in group interactions.
In AAAI, pages 2999–3005.

[27]
Han, T. A., Pereira, L. M., Santos, F. C., and Lenaerts, T. (2013).
Good agreements make good friends.
Scientific reports, 3(2695).

[28]
Han, T. A., Pereira, L. M., Santos, F. C., and Lenaerts, T. (2020).
To Regulate or Not: A Social Dynamics Analysis of an Idealised AI
Journal of Artificial Intelligence Research, 69:881–921.

[29]
Han, T. A. and Tran-Thanh, L. (2018).
Cost-effective external interference for promoting the evolution of
Scientific reports, 8(1):1–9.

[30]
Hindersin, L., Wu, B., Traulsen, A., and García, J. (2019).
Computation and simulation of evolutionary game dynamics in finite
Scientific reports, 9(1):1–21.

[31]
Imhof, L. A., Fudenberg, D., and Nowak, M. A. (2005).
Evolutionary cycles of cooperation and defection.
Proc. Natl. Acad. Sci. U.S.A., 102:10797–10800.

[32]
Lee, K.-F. (2018).
AI superpowers: China, Silicon Valley, and the new world order.
Houghton Mifflin Harcourt.

[33]
Lemley, M. A. (2012).
The myth of the sole inventor.
Michigan Law Review, pages 709–760.

[34]
McGinnis, J. O. (2010).
Accelerating ai.
Nw. UL Rev., 104:1253.

[35]
Nemitz, P. (2018).
Constitutional democracy and technology in the age of artificial
Philosophical Transactions of the Royal Society A: Mathematical,
Physical and Engineering Sciences, 376(2133):20180089.

[36]
Nowak, M. A., Sasaki, A., Taylor, C., and Fudenberg, D. (2004).
Emergence of cooperation and evolutionary stability in finite
Nature, 428:646–650.

[37]
Ogbo, N. B., Elgarig, A., and Han, T. A. (2021).
Evolution of coordination in pairwise and multi-player interactions
via prior commitments.
Adaptive Behavior (In Press).
Preprint arXiv:2009.11727.

[38]
O'Keefe, C., Cihon, P., Garfinkel, B., Flynn, C., Leung, J., and Dafoe, A.
The windfall clause: Distributing the benefits of ai for the common
In Proceedings of the AAAI/ACM Conference on AI, Ethics, and
Society, pages 327–331.

[39]
Sasaki, T., Okada, I., Uchida, S., and Chen, X. (2015).
Commitment to cooperation and peer punishment: Its evolution.
Games, 6(4):574–587.

[40]
Shulman, C. and Armstrong, S. (2009).
Arms control and intelligence explosions.
In 7th European Conference on Computing and Philosophy (ECAP),
Bellaterra, Spain, July, pages 2–4.

[41]
Sigmund, K. (2010).
The Calculus of Selfishness.
Princeton University Press.

[42]
Stix, C. and Maas, M. (2020).
Bridging the gap: the case for an ‘incompletely theorized
agreement’on ai policy.

[43]
Taddeo, M. and Floridi, L. (2018).
Regulate artificial intelligence to avert cyber arms race.
Nature, 556(7701):296–298.

[44]
Traulsen, A., Nowak, M. A., and Pacheco, J. M. (2006).
Stochastic dynamics of invasion and fixation.
Phys. Rev. E, 74:11909.

[45]
Tzachor, A., Whittlestone, J., Sundaram, L., et al. (2020).
Artificial intelligence in a crisis needs ethics with urgency.
Nature Machine Intelligence, 2(7):365–366.

[46]
Vasconcelos, V. V., Santos, F. C., and Pacheco, J. M. (2013).
A bottom-up institutional approach to cooperative governance of risky
Nature Climate Change, 3(9):797–801.

[47]
Vinuesa, R., Azizpour, H., Leite, I., Balaam, M., Dignum, V., Domisch, S.,
Felländer, A., Langhans, S., Tegmark, M., and Nerini, F. F. (2020).
The role of artificial intelligence in achieving the sustainable
development goals.
Nature Communications, 11(233).

[48]
Wang, S., Chen, X., and Szolnoki, A. (2019).
Exploring optimal institutional incentives for public cooperation.
Communications in Nonlinear Science and Numerical Simulation,