# The Formation of the Eccentric Millisecond Pulsar by the Accretion-induced
Collapse of the Massive White Dwarf

D. Wang, 1 B.P. Gong, 1
1Department of Physics, Huazhong University of Science and Technology, Wuhan
430074, China E-mail<EMAIL_ADDRESS>
(Accepted XXX. Received YYY; in original form ZZZ)

###### Abstract

The millisecond pulsar(MSP) is believed to be an old neutron star(NS) having
undergone spin-up by the accreting material from the donor. Whereas, the
discovery of eccentric millisecond pulsars (eMSPs) in the Galactic field
challenges such a scenario producing MSP-white dwarf (WD) only in the circular
orbit. As orbital periods and companion mass of these eMSPs are located in a
narrow range, a reasonable postulation is that they have the same origin.
Although many models have been proposed to interpret their origin, however,
the origin of the narrow range of the orbital period is still an open
question. The accretion-induced collapse(AIC) of the ONe WD is considered to
be an important pathway to form MSP, which was expected to result in the
formation of MSP in the circular orbit due to tidal circularization. Here we
revisited this scenario by the binary population synthesis including the
specific circularization calculation. Our results indicate that binaries with
insufficient circularization in this scenario can evolve into the eMSPs. The
narrow initial binary parameters required by insufficient circularization can
naturally account for the narrow range of the orbital period. Although the
evolution of WD’s AIC process has not been well understood, the characteristic
of a narrow range in the orbital period of eMSPs can still set constraints on
the physics of their evolution.

###### keywords:

(stars:) pulsars: general – (stars:) white dwarfs – (stars:) binaries: general
– stars: neutron – stars: evolution

##  1 Introduction

Millisecond pulsars(MSPs) are the old neutron stars(NSs) having gained angular
momentum by accretion of material from their donor stars, called the recycling
model which is supported by many observations(Lorimer, 2008). During the
accreting stages, the orbit is expected to be circularized by the tidal
interaction. The accretion is terminated while the donor stars evolve into the
white dwarfs(WDs) so that the remnant MSP-WD systems are expected to be
circular.

However, the discovery of eccentric millisecond pulsars(eMSPs) in the Galactic
field challenges such a recycling model. The first reported eMSP is PSR
J1903+0327, which has a main-sequence star(MS) companion and a very large
eccentricity $e\simeq 0.44$(Champion et al., 2008). It is the only eMSP that
has an MS companion, suggested to be formed from hierarchical triple
system(Freire et al., 2011). Subsequently, five eMSPs with WD companion are
discovered(Bailes, 2010; Barr et al., 2013; Deneva et al., 2013; Knispel et
al., 2015; Camilo et al., 2015). Those binaries have similar narrow orbital
period(22-32 d) and eccentricity(0.08-0.14). Their companions satisfy the
mass-orbital period relation similar to the conventional pulsars(Ginzburg &
Chiang, 2022), implying that they have undergone the same accretion process in
the recycling process despite their much larger eccentricity. Such
similarities suggest that they are not formed from the chaotic triple systems,
producing systems with a wider range of orbital periods.

As a result, several models have been proposed to explain eMSP systems.
Antoniadis (2014) suggested that the ejected material of the donor can form a
circumbinary disk during the hydrogen flash process, then the dynamical
interaction between such a disk and binary can excite a high eccentricity. In
this model, the orbital period and eccentricity of the binary depend on
properties of the disk such as lifetime, mass, position, and so on, which are
very poorly understood. Consistent with the observation requires strong
assumptions on the properties of the disk. Han & Li (2021) and Tang et al.
(2023) proposed that the mass ejection from proto-WD could be asymmetrical,
which thereby causes eccentricity via kick velocity regardless of whether the
circumbinary disk is formed or not. Recently, Ginzburg & Chiang (2022) found
that the orbital periods of eMSPs are close to the convective eddy turnover
time of the red giant. They thus proposed that the eccentricities of eMSPs are
induced by the resonant convection.

Other models drop the assumption that neutron stars arise from the iron-core
collapse of massive stars, as do conventional models. Freire & Tauris (2013)
suggested that the eMSP might be formed by the accretion-induced collapse(AIC)
of a massive white dwarf. To avoid orbital circularization after the AIC from
happening, the AIC event is required to take place while the companion has
exhausted its envelope. They assumed that the WD is super-Chandrasekhar with
high spin, which collapses while the critical mass reaches the current mass as
spin-down. Then the orbital eccentricity is induced by the mass loss and kick
via the AIC. Such a scenario is the so-called rotationally delayed accretion-
induced collapse(RD-AIC) model which is then adopted by (Jiang et al., 2015;
Jiang et al., 2021). They suggested that the eMSP is possibly produced by the
RD-AIC onto the NS, resulting in the strange quark star(SS). The mass of the
newborn neutron star is less than that of its progenitor because of the mass
loss during the AIC process. Consequently, the measured pulsar mass of two
eMSPs being larger than the Chandrasekhar limit is difficult to be explained
by the RD-AIC of the WD(Zhu et al., 2019; Barr et al., 2017). Though such RD-
AIC is difficult to account for the narrow orbital period range of eMSP, it
can easily interpret the presence of both high and low eccentric systems
within this range, i.e. via stemming from different evolutionary channels.

The AIC evolution of WD has been extensively studied(Wang, 2018; Liu et al.,
2018; Wang et al., 2022; Tauris et al., 2013). When the mass of an accreting
One WD reaches a critical mass, it collapses into NS. The eccentric orbit
originating in the AIC is usually expected to be circularized during the
subsequent accretion process. However, if the lifetime of the giant star
companion after the onset of AIC is less than the circularization timescale,
the NS-WD system left behind will be eccentric. Such insufficient circularized
systems may exist in a small parameter space, which can explain the narrow
orbital range observed.

Serylak et al. (2022) recently measured the companion mass of PSR J0955-6150
and found it to be significantly lower than predicted, a result they believe
rules out all current models. Such deviation can be explained by the kick
event that changes the orbital period. However, their calculation show that
the RD-AIC model as a pure kick model always predicts larger eccentricities
when the orbital period is met in accordance with observations. If the binary
undergoes an insufficient circularization after the kick, then it can have a
lower eccentricity than predicted by the RD-AIC model and thus satisfy the
observation.

To investigate such a possibility, the orbital eccentric evolution needs to be
considered in the AIC evolution of the WD. Hurley et al. (2010) made a binary
population synthesis(BPS) that contains orbital eccentric evolution. However,
the distribution of eccentricity was not addressed in the literature. Chen et
al. (2011) adopted the same BPS code to study the eccentric MSP with only the
non-degenerate companion.

In this paper, we make a similar BPS to study the eccentric MSP-WD systems and
then compare them with the observations. In Section 2, the numerical method is
exhibited. The simulation results are shown in Section 3. The discussion and
summary are presented in Section 4 and Section 5 respectively.

##  2 Numerical Methods

Our simulation of the binary evolution is performed based on the rapid binary
star evolution(BSE) code (Hurley et al., 2002). It uses the approximately
analytic formula to describe the evolution of single stellar (Hurley et al.,
2000). The orbital evolution includes the mass variation via Roche lobe
overflow(RLO) and stellar winds, kick velocity of collapse and common
envelope(CE) evolution, and the orbital angular momentum evolution via the
gravitational wave radiation, magnetic braking, and tide.

Once the mass of the WD reaches $1.44\mathrm{M}_{\sun}$ with mass growth by
accretion, the oxygen-neon(ONe) WD will collapse into a NS of
$1.3\mathrm{M}_{\sun}$ while the carbon-oxygen(CO) WD will undergo a Type Ia
supernova(SNe Ia). The eccentricity will be induced by the mass loss and kick
velocity during the AIC (Hurley et al., 2002).

###  2.1 The WD mass accumulation

In the BSE, a white dwarf ejects some accreted material at low mass transfer
rates and regains its envelope at high accretion rates to become a giant
star(Hurley et al., 2002; Hurley et al., 2010). It is different from the white
dwarf mass growth model widely used in studies of accreting white dwarfs
(Tauris et al., 2013; Wang et al., 2009, 2014; Wang, 2018). We use this more
commonly used model described below in this subsection and replace the
original model in the BSE with it.

Following Hachisu et al. (1999), the mass growth of the He-layer by the
H-shell burning is dependent on the mass loss of the companion $\dot{M}_{2}$,
given by

$\dot{M}_{\mathrm{He}}=\eta_{\mathrm{H}}\left|\dot{M}_{2}\right|,$ (1)

where the mass accumulation efficiency of H-shell burning $\eta_{\mathrm{H}}$
is

$\eta_{\mathrm{H}}=\begin{cases}\dot{M}_{\mathrm{cr}}/\left|\dot{M}_{2}\right|,&\left|\dot{M}_{2}\right|>\dot{M}_{\mathrm{cr,H}}\\\
1,&\dot{M}_{\mathrm{cr,H}}\geq\left|\dot{M}_{2}\right|\geq\dot{M}_{\text{low,H
}}\\\ 0,&\left|\dot{M}_{2}\right|<\dot{M}_{\text{low,H }}\end{cases}.$ (2)

Here the critical mass transfer rate for the steady hydrogen burning
$\dot{M}_{\mathrm{cr,H}}$ is

$\dot{M}_{\mathrm{cr,H}}=5.3\times
10^{-7}\frac{(1.7-X)}{\mathrm{X}}\left(\frac{M_{\mathrm{WD}}}{\mathrm{M}_{\sun}}-0.4\right)\mathrm{M}_{\sun}\mathrm{yr}^{-1},$
(3)

where the $M_{\mathrm{WD}}$ is the WD mass and $X$ is the hydrogen mass
fraction of the accreted material, and the minimum mass transfer rate at which
a WD can accumulate mass by H-shell burning is $\dot{M}_{\text{low,H
}}=\frac{1}{8}\dot{M}_{\mathrm{cr,H}}$.

The newborn He-layer will continue to burn, then the mass growth of the WD by
the He-layer burning is given by

$\dot{M}_{\mathrm{WD}}=\eta_{\mathrm{He}}\cdot\dot{M}_{\mathrm{He}},$ (4)

where the mass accumulation efficiency of the He-layer burning
$\eta_{\mathrm{He}}$ is

$\eta_{\mathrm{He}}=\begin{cases}\frac{\dot{M}_{\mathrm{cr,He}}}{\left|\dot{M}_{\mathrm{He}}\right|},&\left|\dot{M}_{\mathrm{He}}\right|>\dot{M}_{\mathrm{cr,He}}\\\
\eta_{\mathrm{He}}^{\prime},&\dot{M}_{\mathrm{cr,He}}>\left|\dot{M}_{\mathrm{He}}\right|\geq\dot{M}_{\mathrm{low,He}}\\\
0,&\left|\dot{M}_{\mathrm{He}}\right|<\dot{M}_{\mathrm{low,He}}\end{cases}.$
(5)

The critical mass transfer rate for the steady He-layer burning is(Nomoto,
1982)

$\dot{M}_{\mathrm{cr,He}}=7.2\times
10^{-6}\left(\frac{M_{\mathrm{WD}}}{\mathrm{M}_{\sun}}-0.6\right)\mathrm{M}_{\sun}\mathrm{yr}^{-1},$
(6)

and the minimum mass transfer rate of He-layer burning is(Woosley et al.,
1986)

$\dot{M}_{\text{low,He}}=4.0\times 10^{-8}\mathrm{M}_{\sun}\mathrm{yr}^{-1}.$
(7)

And the $\eta_{\mathrm{He}}^{\prime}$ is taken from the linear interpolation
of the numerical results by Kato & Hachisu (2004).

The materials that can not be accumulated onto the WD’s surface are assumed to
be blown away as the wind carrying the specific orbital angular momentum of
the WD (Hurley et al., 2002).

###  2.2 Circularization

If the circularization is insufficient after the occurrence of AIC, then the
remnant NS-WD system will be eccentric. Thus the only important issue is the
circularization mechanism of the NS and donor.

The circularization mechanism during stable mass transfer is tidal
dissipation. In the tidal model used by the BSE, the tidal dissipation occurs
in the envelope of the star, then the circularization efficiency is positively
related to the envelope mass. The detailed formula can be found in section 2.3
of the Hurley et al. (2002).

If the mass transfer between the NS and donor is unstable, the binary will
enter the CE stage, after which the binary is usually believed to be circular.
However, the simulation of the CE evolution of the eccentric binary shows that
eccentric binaries are only partially circularized(Glanz & Perets, 2021). In
the BSE, the CE evolution is described by the $\alpha\lambda$ model, in which
the initial binding energy of the envelope is calculated by

$E_{\mathrm{bind},\mathrm{i}}=-\frac{G}{\lambda}\left(\frac{M_{1}M_{\mathrm{env}1}}{R_{1}}+\frac{M_{2}M_{\mathrm{env}2}^{\prime}}{R_{2}}\right),$
(8)

and the initial orbital energy of the cores is

$E_{\mathrm{orb},\mathrm{i}}=-\frac{1}{2}\frac{GM_{\mathrm{c}1}M_{\mathrm{c}2}^{\prime}}{\mathrm{a}_{\mathrm{i}}}.$
(9)

Then orbital energy of the cores is transferred to the envelope during the CE
evolution with an efficiency $\alpha$.111In the BSE, The defined initial
orbital energy contains only the core component, which is not the same as the
calculation using the entire star. The $\alpha\approx 3$ in the BSE
corresponds to $\alpha=1$ in the latter (Hurley et al., 2010), representing
that all the orbital energy is transferred to the envelope.

The CE stage is terminated while the envelope has enough energy to be ejected,
so that the final orbital energy of the cores $E_{\mathrm{orb},\mathrm{f}}$ is
read,

$E_{\mathrm{bind},\mathrm{i}}=\alpha\left(E_{\mathrm{orb},\mathrm{f}}-E_{\mathrm{orb},\mathrm{i}}\right),$
(10)

where the final semi-axis $\mathrm{a}_{\mathrm{f}}$ can be derived by

$E_{\mathrm{orb},\mathrm{f}}=-\frac{1}{2}\frac{GM_{\mathrm{c}1}M_{\mathrm{c}2}^{\prime}}{\mathrm{a}_{\mathrm{f}}}.$
(11)

The initial circularized orbital energy of the cores is defined as
$E_{\mathrm{circ}}=E_{\mathrm{orb},\mathrm{i}}/(1-e^{2})$. If
$E_{\mathrm{orb},\mathrm{f}}<E_{\mathrm{circ}}$, the final eccentricity is
given by

$e_{f}=\sqrt{1-\frac{E_{\mathrm{orb},\mathrm{f}}}{E_{\mathrm{circ}}}},$ (12)

and $e_{f}=0$ if $E_{\mathrm{orb},\mathrm{f}}\geq E_{\mathrm{circ}}$.

###  2.3 Criteria of the CE

The criteria for entering the CE stage are also very critical. Whether or not
a binary enters the CE phase after the AIC determines which circularization
mechanism will work. Also, if the CE proceeds during the accretion phase of
the WD, the short lifetime of the CE will prevent the WD from accreting enough
mass for AIC.

Now the CE criteria is unclear. Tauris et al. (2013) adopted
$\dot{M}_{2}=3\dot{M}_{\mathrm{cr,H}}$ and Wang (2018) used an unstable
increases of $\dot{M}_{2}$ as the CE criteria. Thus they predict different
parameters space on the occurrence of AIC. The BSE adopts the mass ratio as
the CE criteria, which is not going to be changed by this work. But it is
worth noting that the simulation results are strongly dependent on this
assumption.

###  2.4 Binary population synthesis

On the choice of initial parameter of binaries, we adopt the setting of Hurley
et al. (2002), in which all binaries are assumed to be initially circular. The
initial parameter space of binaries is as follows: $0.8-80\mathrm{M}_{\sun}$
for the primary mass $M_{1}$, $0.1-80\mathrm{M}_{\sun}$ for the secondary mass
$M_{2}$, $3-10^{4}\mathrm{R}_{\sun}$ for the binary separation $a$. All
parameters are taken from $n_{\chi}$ grid points in the logarithmic space.
Thus, the gird point interval is

$\delta\ln\chi=\frac{\ln\chi_{\max}-\ln\chi_{\min}}{n_{\chi}-1}$ (13)

for each initial binary parameter $\chi$($M_{1}$, $M_{2}$, $a$). The grid
number $n_{\chi}$ we use is 400.

There are three evolutionary channels that depend on the type of companion
star at the time of the AIC(Wang & Liu, 2020): the MS channel, the red
gaint(RG) channel, and the helium star(He) channel. The He channel will form
more massive companions than observation (Tauris et al., 2013), which implies
that eMSPs are formed from the MS/RG channel. Thus we only concern the
detached eccentric($e>0.01$) NS-WD systems formed from the AIC of WD in these
channels.

As long as a binary evolves to such a system, it corresponds to a birthrate of

$\delta r=S\Phi(\ln M_{1})\varphi(\ln M_{2})\Psi(\ln a_{2})\delta\ln
M_{1}\delta\ln M_{2}\delta\ln a\ ,$ (14)

where $S$ is the star formation rate assumed to be a constant
$S=7.6085\mathrm{yr}^{-1}$. And the contribution to the expected number of the
concerned systems is

$\delta n=\delta r\delta t,$ (15)

where $\delta t$ is the lifetime of the concerned systems for each grid.

The primary mass distribution $\Phi(\ln M_{1})$ can be derived by

$\Phi(\ln M_{1})=M_{1}\xi(M_{1}),$ (16)

where the $\xi(M_{1})$ is the initial mass function(IMF) given by Kroupa et
al. (1993),

$\xi(m)=\begin{cases}0.29056m^{-1.3}&0.1<m\leq 0.5\\\
0.15571m^{-2.2}&0.5<m\leq 1.0\\\ 0.15571m^{-2.7}&1.0<m<\infty\end{cases}.$
(17)

The distribution of the secondary mass can be obtained from the mass ratio
$q_{2}$,

$\varphi\left(\ln M_{2}\right)=\frac{M_{2}}{M_{1}}=q_{2},$ (18)

which is assumed to satisfy a uniform distribution in the range of $[0,1]$.

The distribution of separation $a$ is assumed to be

$\Psi(\ln a)=0.12328,$ (19)

which is normalized within the range of 3 and $10^{4}\mathrm{R}_{\sun}$.

In the BSE code, the kick velocity is taken from Maxwellian distribution, and
the direction is uniformly distributed in the full solid angles. In previous
studies using BSE, each grid point was sampled only once, i.e., only a
particular kick velocity direction was simulated. This results in the problem
of under-sampling, because the binary parameters after kick strongly depend on
the direction of the kick velocity. Therefore, the outcome of the BPS using
BSE change significantly with the random seed for the black holes(BHs) and NS
systems which have undergone the kick event(Hurley et al., 2002; Chen et al.,
2011).

To improve this under-sampling problem, we also include the kick direction
parameter in the grid of the initial binary parameters. In such a case, the
new contribution to the expected number is read,

$\delta n_{kick}=\delta nP(\sin\phi)Q(\omega)\delta\sin\phi\delta\omega,$ (20)

where the kick direction is determined by the two angles $\phi$ and $\omega$,
defined in the Hurley et al. (2002). Assuming a uniform distribution of
direction in the full solid angles, such an angle distribution is read,

$P(\sin\phi)=0.5$ (21)

for $\sin\phi$ in the $[-1,1]$, and

$Q(\omega)=\frac{1}{2\pi}$ (22)

for $\omega$ in the $[0,2\pi]$. The number of the grid points of the two
angels of the kick velocity is taken as 10.

The introduction of the kick parameters increases the number of grid points,
so that longer computation time is required. Here we use a two-step approach
to perform simulations. Firstly, the simulations of the grid points without
kick parameters are made. Secondly, the points where AIC can occur are
selected. And finally, each selected grid point is simulated again with a
different kick direction.

Two values of the $\alpha$ are adopted in the CE evolution, $\alpha=1$ and
$\alpha=3$. Although the small kick velocity of the WD’s AIC is widely adopted
in the previous studies Hurley et al. (2010); Tauris et al. (2013); Freire &
Tauris (2013), there are studies adopt large kick velocity (Chen et al., 2011,
and the references therein). Here we adopt both small and large kick
velocities, which are chosen as 0, 10, 30, 50, 70, 90, 110, 130, 150, 170, and
190 km/s. The metallicity is taken as $Z=0.02$ and the other parameters in the
BSE are the same as those of Hurley et al. (2010).We evolve these binaries for
12Gyr.

##  3 Results

###  3.1  The expected number of eMSPs

The orbital periods of observed MSPs with helium WD companion are within 0.1
to 1000 days (Ginzburg & Chiang, 2022). Like Hurley et al. (2010), we define
the detectable systems as those with $0.1<P_{b}<1000$ days.

The expected number of eccentric MSP-WD systems with each kick velocity is
shown in Fig.1. It decreases as the kick velocity increases in both cases of
$\alpha=1$ and $\alpha=3$. For the detectable systems, it increases and
subsequently decreases as the kick velocity increases. When the kick velocity
is small, there are more systems with $\alpha=3$ than those with $\alpha=1$,
but for detectable eMSPs, more systems reside in $\alpha=1$ rather than within
$\alpha=3$. As the kick velocity increases, the number of systems with
$\alpha=1$ and $\alpha=3$ as well as the number of corresponding detectable
systems converges, i.e., the undetectable systems disappear. And the number of
systems with $\alpha=3$ is much smaller than the number of systems with
$\alpha=1$ when the kick velocity is large.

Figure 1: The relation between the kick velocity and expected number of the
eccentric MSP-WD systems.

Such a result is a bit counterintuitive. A larger kick speed will lead to a
larger eccentricity after the kick event, which in turn will generate more
eMSPs(Freire & Tauris, 2013; Han & Li, 2021). However, this is opposite to the
result shown in Fig.1. The reason is that the larger kick velocity will also
make the binary easier to be disrupted or become tighter so that it is easier
to merge or be circularized.

###  3.2 The distribution of eMSPs parameters

The distribution of $e/m_{2}-P_{b}$ of the detectable eMSPs is shown in Fig.2
and Fig.3, corresponding to the cases of $\alpha=1$ and $\alpha=3$
respectively. Such distributions of all eMSPs is shown in the appendix A We
only exhibit results for $v_{kick}=$ 10, 50 and 190 km/s corresponding to low,
medium, and high kick velocity, respectively. There is a significant
discrepancy between the results for $\alpha=1$ and $\alpha=3$. The case of
$\alpha=1$ corresponds to the more diffuse distribution of parameters and
$\alpha=3$ demonstrates some concentrated areas of parameters. The
distribution of parameters for medium and high kick velocities is very similar
but differs profoundly from that of low kick velocity. In the case of low kick
velocity, the distribution of $P_{b}$ shows a double-peaked structure for both
$\alpha=1$ and $\alpha=3$.

Figure 2: The distribution of the parameters of the detectable eMSPs with
$\alpha=1$. The top and middle panels show the distribution of $m_{2}-P_{b}$
and $e-P_{b}$ respectively, and the lower panel shows the distribution of
$P_{b}$. The colorbar represent the expected number defined by Eq.20. The
yellow region represent the orbital period range of 20 35 days, which the
observational eMSP locate in.

The plots from left to right represent the results with $v_{kick}=$ 10, 90,
and 190 km/s respectively.

Figure 3: As in Fig.2, but with $\alpha=3$.

The orbital period range of observational eMSP is 20 35 days. The orbital
period distribution of the simulation that fits the observation should have a
peak near this range. The Fig.2 and Fig.3 show that the parameters that best
fit observations are $\alpha=3$ and $v_{kick}=$ 10 km/s. In this case,
although the orbital periods of eMSPs are mostly distributed over 100 days,
there is a peak below 100 days that appears to match the observations. To
compare more intuitively the simulation results corresponding to this peak
with the observed eMSPs, the distribution of $e/m_{2}-P_{b}$ of the simulated
eMSPs with $0.1<P_{b}<100$ days are shown as a scatter plot in Fig.4. With
$\alpha=3$ and $v_{kick}=$ 10 km/s, the orbital period distributions of the
eMSPs obtained from the simulations well match with those of the observed
eMSPs.

Figure 4: As in Fig.3 with $v_{kick}=$ 10 km/s, but as a scatter plot to show
each simulated point. The solid red star points represent the parameters of
the five observed eMSPs, which are taken from Serylak et al. (2022) and the
references therein. For the companion mass of PSR J1618-3921, the minimum mass
estimated from the mass function is adopted and taken from ATNF catalog
(Manchester et al., 2005). In the top panel, the green, orange, and blue lines
represent the $m_{2}-P_{b}$ relationships by Rappaport et al. (1995) and
Tauris & Savonije (1999) for Pop.I and Pop.II respectively. In the bottom
panel, the dashed orange and solid blue lines represent the distribution of
the simulated and observed samples respectively.

However, the simulated companion mass of the eMSPs is smaller than the
observed one. This difference essentially represents the deviation of the
simulated $m_{2}-P_{b}$ relationship obtained from the simulations and the
observed values. This deviation may be caused by the kick increasing the
orbital period, which also leads to the dispersion of the orbital period in
Fig.4. But such an effect can also reduce the orbital period, the theoretical
curve should cross the distribution of the orbital period. Thus it can not
explain this deviation.

The $m_{2}-P_{b}$ relationship was first discovered by Rappaport et al. (1995)
and subsequently obtained by Tauris & Savonije (1999) with an updated stellar
evolution code. Their results are compared with observation and our
simulations as shown in Fig.4. The $m_{2}-P_{b}$ relationship from Tauris &
Savonije (1999) for Population I ($Z=0.02$) consist best with the observation.
The discrepancy between their results stems from the different stellar
evolutionary codes they use (Tauris & Savonije, 1999). Therefore, the
difference between the results of ours and theirs may also originate in the
same reason.

The $m_{2}-P_{b}$ relationship also depends on metallicity (Ergma et al.,
1998; Tauris & Savonije, 1999). For a given $P_{b}$, $m_{2}$ increases as
metallicity decreases. Therefore, simulation with $\alpha=3$ and $v_{kick}=$
10 km/s for the Population II ($Z=0.001$) are performed as shown in Fig.5. In
such a case, the $m_{2}-P_{b}$ relationship obtained from the simulation well
agrees with the observed values222Some points that do not follow the relation
satisfied by other points. They correspond to the situation that the companion
evolve into the He star and then into the He giant star and finally into the
CO WD after the AIC event.. However, the peaks in the $P_{b}$ distribution are
wider than that with $z=0.02$, indicating a more diffused $P_{b}$ than the
observed values. In addition, the number of eMSPs is also greater than those
with $z=0.02$.

Figure 5: As in Fig.4, but for Pop.II($Z=0.001$).

Although the simulated $m_{2}-P_{b}$ distribution for Pop.II is more
consistent with the observations, the $m_{2}-P_{b}$ relationship from Tauris &
Savonije (1999) suggests that the observations are more consistent with Pop.I.
Since Tauris & Savonije (1999) uses an updated numerical stellar evolution
code, whereas BSE uses a semi-analytic stellar evolution(Hurley et al., 2002),
the results by Tauris & Savonije (1999) should be somewhat more accurate. The
observed eMSPs are more likely to have evolved from Pop.II.

##  4 Discussion

The result that best fits the observation implies that there are eMSPs with
orbital periods between 100 and 1000 days, which have not been found in the
observation. The possible reason for such a contradiction is due to the
assumption that the NS born through WD’s AIC is the MSP. For a NS to be a
pulsar, in addition to its radiation cone pointing towards the Earth, its
magnetic field $B$ and rotation period $P$ are required to satisfy
$B/P^{2}>0.17\times 10^{12}\mathrm{G}/\mathrm{s}^{2}$, which is above the
death line (Bhattacharya et al., 1992). A NS of radius 10km produced by the
collapse of a typical WD with radius 3000 km and magnetic field
$10^{3}\mathrm{G}$ corresponds to a surface magnetic field of
$10^{8}\mathrm{G}$ (Freire & Tauris, 2013), which is the typical value for
MSP.

The evolution of the NS’s rotation period is ignored in our simulation, which
depends on two factors: one is the initial rotation period determined by the
rotation of the WD as the progenitor, and the other is the NS’s rotation
evolution during NS accretion(Chen et al., 2011). Since the eMSP discussed in
this paper is insufficiently circularized, which implies that the lifetime of
the NS during the accretion phase is short, the latter can be neglected. For
the former, the rotation evolution of the WD needs to be considered, which may
affect the mass of the NS due to the possible differential radiation of the WD
(Yoon & Langer, 2004, 2005).

Two of the observed eMSPs have pulsar mass of more than
$1.44\mathrm{M}_{\sun}$ (Zhu et al., 2019; Barr et al., 2017), which is the
maximum mass of WDs in our simulations. Since the mass of the eMSPs produced
by AIC is smaller than those of the progenitors, our simulations cannot
produce MSPs with such a large mass. Even in the RD-AIC model, the white dwarf
can only have a maximum mass of $1.48\mathrm{M}_{\sun}$, which is the maximum
mass of a rigidly rotating WD (Freire & Tauris, 2013). However, if the white
dwarf has a differential rotation, it can have a maximum mass of up to
$4\mathrm{M}_{\sun}$ (Yoon & Langer, 2004). The angular momentum carried by
the WD accreting material is first transferred to the outer layers of the WD,
and then to the inner sides, thus redistributing the angular momentum. If the
transfer of angular momentum is not efficient enough, the WD in accretion will
be in a differential rotation rather than a rigid one.

The evolution of the differential accreting WD has been studied by Yoon &
Langer (2005). The evolutionary path of the accreting WD depends on the
angular momentum accretion efficiency, the angular momentum transport
efficiency, and the angular momentum loss due to gravitational radiation
induced by the r-mode instability. Whether accretion terminates before the WD
reaches the critical mass will determine whether the collapse is delayed. The
RD-AIC model is the delayed collapse of the rigidly rotational WD, which is
one of the cases that Freire & Tauris (2013); Yoon & Langer (2005) considered.

The initial rotation period of the NS is also likely to be constrained by the
observation. The AIC of the WD is one of the possible origins of the fast blue
optical transients(FBOT) (Yu et al., 2019a; Yu et al., 2019b). And Liu et al.
(2022) found that the rotation period of the remnant NS after FBOT event is
$9.1_{-4.4}^{+9.3}\mathrm{~{}ms}$, which slightly matches the observational
value of the eMSPs. However, the magnetic field of the NS they found is very
large $11_{-7}^{+18}\times 10^{14}\mathrm{G}$, which contradicts our
assumptions.

The change of mass limit of the WD with differential rotation will
significantly the evolution of the AIC or SNe Ia, making the parameter space
smaller(Chen & Li, 2009). Although our results well explain the narrow orbital
period range of the observation, it will make a difference if the rotation
evolution is considered. Even so, due to the required short lifetime of the
companion after the kick event, the results of the simulations with the
rotation evolution are expected to be able to have a narrow orbital period
distribution for some specific parameters.

The AIC evolution of the WDs contains direct AIC of the Chandrasekhar WDs, the
direct and delay AIC of the super-Chandrasekhar WD (Yoon & Langer, 2005). And
the rotation of the WDs also affects the rotation of the newborn NS. Thus a
comprehensive simulation of the AIC evolution of the WD taking into account
the rotation evolution of the WD and the NS is necessary, which will be
included in our future simulations.

The direct AIC of the super-Chandrasekhar WD with insufficient circularization
is a possible way to form PSR J0955-6150. The pure kick simulation show that
the kick event can change the orbital period to the observational value of
24.58 days (Serylak et al., 2022). The larger eccentricity than observation
induced by such kick event can decrease to the observational value of 0.11 by
tidal circularization while the companion evolves to the white dwarf. However,
the circularization will also decrease the orbital period, so whether the
final orbital period can match observation is questionable.

Serylak et al. (2022) also found that the spin axis of the MSP does not align
with the direction of the orbital angular momentum vector, and the
misalignment angle $\delta>4.8$ deg is surprisingly huge. Such a misalignment
angle can also be caused by the kick event which can change the direction of
the orbital angular momentum, then constrain the kick velocity.

Here we use method described by Hurley et al. (2002) to calculate the orbital
period $P_{b,0}$, eccentricity $e_{0}$ and misalignment angle after kick
event. Following the calculation of Serylak et al. (2022), the mass of the
white dwarf and orbital period in the pre-AIC stage is set to 1.96
$\mathrm{M}_{\sun}$ and 14.2 days(Pop. I), respectively. After the kick event,
the circularization will decrease both the orbital period and eccentricity. In
this process, orbital angular momentum $L=\sqrt{GMa(1-e^{2})}$ is roughly
conserved, thus the final orbital period can be derived by
$P_{b,f}=\left(\frac{1-e_{f}}{1-e_{0}}\right)^{\frac{3}{2}}P_{b,0}$, where the
final eccentricity $e_{f}$ is the observational value 0.11.

Figure 6: The distribution of misalignment angle and orbital period of the
simulation for direct AIC with insufficient circularization process of PSR
J0955-6150. The red star represents the parameters of PSR J0955-6150. The
probabilities are calculated under the assumption of isotropic kick velocity
direction.

The simulation result is shown in Fig.6. Only post-AIC eccentricity
$e_{0}>e_{f}=0.11$ is considered. while kick velocity is 10 km/s, the final
orbital period can not match observation even if allow all misalignment
angles. For the higher kick velocity 20 km/s, the final orbital period range
is broadened, and both the final orbital period and misalignment can match
observations. The edges of the parameter space have the maximum probability,
thus the kick velocity is likely between 10 and 20 km/s.

##  5 Conclusion

We suggest that the eMSP is formed from the AIC of WD under insufficient
circularization. By performing a binary population synthesis with BSE, we find
that the distribution of $e$ and $m_{2}$ with $\alpha=3$ and
$v_{\mathrm{kick}}=10$ km/s well fit the observations. In particular, it
allows a natural interpretation of the narrow range of orbital periods of the
observational eMSPs. In addition, we find that smaller kick velocities are
more likely to produce eMSPs, which is somewhat counterintuitive.

Despite our simulation ignoring the rotation evolution of the WD and MSP, the
results still show that the small parameter space corresponding to the
insufficient circularization can explain the narrow orbital period range in
the observation. Such a result indicates that this scenario is a potential
mechanism to form observational MSPs. The more realistic scenario need to
include direct and delay AIC of the super-Chandrasekhar WD. The comprehensive
simulation in the future needs to take into account the rotation evolution of
the WD and the NS.

## Acknowledgements

This work was supported by the National Key Research and Development Program
of China(No. 2020YFC2201400) and National SKA Program of China
(2020SKA0120300). We acknowledge Beijng PARATERA Tech CO.,Ltd. for providing
HPC resources that have contributed to the research results reported within
this paper.

## Data Availability

The result and code of this work are available from the corresponding authors
at reasonable request.

## References

  * Antoniadis (2014) Antoniadis J., 2014, The Astrophysical Journal Letters, 797, L24
  * Bailes (2010) Bailes M., 2010, New Astronomy Reviews, 54, 80
  * Barr et al. (2013) Barr E. D., et al., 2013, Monthly Notices of the Royal Astronomical Society, 435, 2234
  * Barr et al. (2017) Barr E., Freire P., Kramer M., Champion D., Berezina M., Bassa C., Lyne A., Stappers B., 2017, Monthly Notices of the Royal Astronomical Society, 465, 1711
  * Bhattacharya et al. (1992) Bhattacharya D., Wijers R. A., Hartman J. W., Verbunt F., 1992, Astronomy and Astrophysics, 254, 198
  * Camilo et al. (2015) Camilo F., et al., 2015, The Astrophysical Journal, 810, 85
  * Champion et al. (2008) Champion D. J., et al., 2008, Science, 320, 1309
  * Chen & Li (2009) Chen W.-C., Li X.-D., 2009, The Astrophysical Journal, 702, 686
  * Chen et al. (2011) Chen W.-C., Liu X.-W., Xu R.-X., Li X.-D., 2011, Monthly Notices of the Royal Astronomical Society, 410, 1441
  * Deneva et al. (2013) Deneva J., Stovall K., McLaughlin M., Bates S., Freire P., Martinez J., Jenet F., Bagchi M., 2013, The Astrophysical Journal, 775, 51
  * Ergma et al. (1998) Ergma E., Sarna M. J., Antipova J., 1998, Monthly Notices of the Royal Astronomical Society, 300, 352
  * Freire & Tauris (2013) Freire P. C., Tauris T. M., 2013, Monthly Notices of the Royal Astronomical Society: Letters, 438, L86
  * Freire et al. (2011) Freire P., et al., 2011, Monthly Notices of the Royal Astronomical Society, 412, 2763
  * Ginzburg & Chiang (2022) Ginzburg S., Chiang E., 2022, Monthly Notices of the Royal Astronomical Society: Letters, 509, L1
  * Glanz & Perets (2021) Glanz H., Perets H. B., 2021, Monthly Notices of the Royal Astronomical Society, 507, 2659
  * Hachisu et al. (1999) Hachisu I., Kato M., Nomoto K., Umeda H., 1999, The Astrophysical Journal, 519, 314
  * Han & Li (2021) Han Q., Li X.-D., 2021, The Astrophysical Journal, 909, 161
  * Hurley et al. (2000) Hurley J. R., Pols O. R., Tout C. A., 2000, Monthly Notices of the Royal Astronomical Society, 315, 543
  * Hurley et al. (2002) Hurley J. R., Tout C. A., Pols O. R., 2002, Monthly Notices of the Royal Astronomical Society, 329, 897
  * Hurley et al. (2010) Hurley J. R., Tout C. A., Wickramasinghe D. T., Ferrario L., Kiel P. D., 2010, Monthly Notices of the Royal Astronomical Society, 402, 1437
  * Jiang et al. (2015) Jiang L., Li X.-D., Dey J., Dey M., 2015, The Astrophysical Journal, 807, 41
  * Jiang et al. (2021) Jiang L., Wang N., Chen W.-C., Liu W.-M., Leng C.-W., Yuan J.-P., Qian X.-L., 2021, Research in Astronomy and Astrophysics, 21, 231
  * Kato & Hachisu (2004) Kato M., Hachisu I., 2004, The Astrophysical Journal, 613, L129
  * Knispel et al. (2015) Knispel B., et al., 2015, The Astrophysical Journal, 806, 140
  * Kroupa et al. (1993) Kroupa P., Tout C. A., Gilmore G., 1993, Monthly Notices of the Royal Astronomical Society, 262, 545
  * Liu et al. (2018) Liu D., Wang B., Chen W., Zuo Z., Han Z., 2018, Monthly Notices of the Royal Astronomical Society, 477, 384
  * Liu et al. (2022) Liu J.-F., Zhu J.-P., Liu L.-D., Yu Y.-W., Zhang B., 2022, arXiv preprint arXiv:2206.03303
  * Lorimer (2008) Lorimer D. R., 2008, Living reviews in relativity, 11, 1
  * Manchester et al. (2005) Manchester R. N., Hobbs G. B., Teoh A., Hobbs M., 2005, The Astronomical Journal, 129, 1993
  * Nomoto (1982) Nomoto K., 1982, The Astrophysical Journal, 253, 798
  * Rappaport et al. (1995) Rappaport S., Podsiadlowski P., Joss P., Di Stefano R., Han Z., 1995, Monthly Notices of the Royal Astronomical Society, 273, 731
  * Serylak et al. (2022) Serylak M., et al., 2022, Astronomy & Astrophysics, 665, A53
  * Tang et al. (2023) Tang W.-S., Gao S.-J., Li X.-D., 2023, Monthly Notices of the Royal Astronomical Society, 519, 2951
  * Tauris & Savonije (1999) Tauris T. M., Savonije G. J., 1999, arXiv preprint astro-ph/9909147
  * Tauris et al. (2013) Tauris T. M., Sanyal D., Yoon S.-C., Langer N., 2013, Astronomy & Astrophysics, 558, A39
  * Wang (2018) Wang B., 2018, Monthly Notices of the Royal Astronomical Society, 481, 439
  * Wang & Liu (2020) Wang B., Liu D., 2020, Research in Astronomy and Astrophysics, 20, 135
  * Wang et al. (2009) Wang B., Meng X., Chen X., Han Z., 2009, Monthly Notices of the Royal Astronomical Society, 395, 847
  * Wang et al. (2014) Wang B., Justham S., Liu Z.-W., Zhang J.-J., Liu D.-D., Han Z., 2014, Monthly Notices of the Royal Astronomical Society, 445, 2340
  * Wang et al. (2022) Wang B., Liu D., Chen H., 2022, Monthly Notices of the Royal Astronomical Society, 510, 6011
  * Woosley et al. (1986) Woosley S., Taam R. E., Weaver T. A., 1986, The Astrophysical Journal, 301, 601
  * Yoon & Langer (2004) Yoon S.-C., Langer N., 2004, Astronomy & Astrophysics, 419, 623
  * Yoon & Langer (2005) Yoon S.-C., Langer N., 2005, Astronomy & Astrophysics, 435, 967
  * Yu et al. (2019a) Yu Y.-W., Chen A., Wang B., 2019a, The Astrophysical Journal Letters, 870, L23
  * Yu et al. (2019b) Yu Y.-W., Chen A., Li X.-D., 2019b, The Astrophysical Journal Letters, 877, L21
  * Zhu et al. (2019) Zhu W., et al., 2019, The Astrophysical Journal, 881, 165

##  Appendix A  Results for the full orbital period range

Figure 7: As in Fig.2, but for the full orbital period range. Figure 8: As in
Fig.7, but with $\alpha=3$.


# Low-impact agency: review and discussion.

Danilo Naiff1, Shashwat Goel2

<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
)

###### Abstract

Powerful artificial intelligence poses an existential threat if the AI decides
to drastically change the world in pursuit of its goals. The hope of low-
impact artificial intelligence is to incentivize AI to not do that just
because this causes a large impact in the world. In this work, we first review
the concept of low-impact agency and previous proposals to approach the
problem, and then propose future research directions in the topic, with the
goal to ensure low-impactedness is useful in making AI safe.

##  1 Introduction.

The problem of artificial intelligence safety can be seen as can be seen as
ensuring an agent with the power of causing harm chooses to not do so. In the
limit, the agent can be powerful enough that causing existential catastrophe
is within its limit, and it has incentives to doing so [6], so our task is to
guarantee that it chooses not to. A possible approach is penalize changes in
the world caused by agent, leading to the agent not causing catastrophe
because that leads to large changes in the world[24]. The hope is that this is
a relatively easy objective to align the agent with, as opposed to aligning it
with the full range of human values.

So, our desideratum is that the AI achieves something while doing as little in
the world as possible. This is related to Yudkowski’s strawberry problem[25].

The strawberry problem:

Make an AI place, onto this particular plate, two strawberries identical down
to the cellular level, and do nothing else.

An AI that solves this task is powerful, because it can do something that lies
way beyond our current technological capacity, steerable, because we made the
AI solve this task, and safe, because it didn’t any other unnecessary harmful
action. Therefore, low-impact agency concerns itself with the ”do nothing
else” part, just because doing more is unnecessarily impactful1.

Figure 1: Achieving safety through low-impact versus safety through value
alignment and non-safety.

However, as with most alignment problems, formalizing low-impact is elusive,
especially if we are to implement our formalization in AI. As we will see
below, we find that although the idea seems intuitively easy, when we try to
give form to this intuition we find many complications, as is commonplace in
the field of alignment.

##  2 What we mean by low-impact?

An intuitive definition of a low-impact action is one in that the action
didn’t make much of a difference in the world, compared to what would
otherwise happen. Therefore, for an AI to evaluate whether an action is low-
impact or not, it needs a default world for comparison, a measure of impact to
compare to, and a prediction model to check the effect of that action.
Therefore, we can decompose the low-impact problem in three parts:

  * •

The baseline problem: If we want to consider that some event had low-impact on
the world, we should compare that event to a world where it does not happen.
We must have a ”default world” to which to compare our agent’s actions to
measure their impact. However, the main intuitive definitions for such
”default worlds” have shortcomings.

  * •

The impact measure problem: Any action changes the world in an extremely large
number of ways at the atomic level, but most of these changes are
insignificant. Therefore, a meaningful definition of impact depends on what
are changes humans care about.

  * •

The world model problem Even if we coarse-grained our world, any action of the
agent may end up changing the world in unpredictable ways due to chaotic
effects (eg: weather). Therefore, the agent should use its world-model to
consider the predictable impact of their actions.

Next, we investigate and review each of these components.

###  2.1 The baseline problem

If we want to consider that some event had low-impact on the world, we should
compare that event to a world where the event does not happen. So, by its very
nature, low-impactness involves comparisons. The event is in the past, we are
doing counterfactual reasoning, and if the event is in the future, we are
engaged with causal reasoning. The wealth of centuries-old literature on these
topics[9, 8] should already give a hint that defining baselines is more
complicated than it seems.

Following [16, 15], we consider three baseline worlds:

  * •

The initial state baseline, where we assume that the base world is the one
where the agent started with.

  * •

The initial inaction baseline, where we assume the base world to be the one
where the agent has never acted at all.

  * •

The future inaction baseline, where we assume the base world is the one where
the agent does not act in the future.

We represent this schematically in Figure 2. Next, we expand on these three
baselines below, while also considering a formalization in a Markov decision
process (MDP) context, following [16]. However, these should be seen more as
illustrations and first steps, than as formalizations applicable to real-world
problems.

Figure 2: Comparison of initial state, initial inaction, and future inaction
baselines. The agent from the current state (orange) compares the result of
following its policy (red) to the three baseline states in blue.

####  2.1.1 The initial state baseline

In the initial state baseline, the default world is the initial state the
agent first was turned on. Therefore, the agent should compare his actions to
the world where he first appeared. The MDP formalization is simple in this
case. Let $(\mathcal{S},\mathcal{A},T)$ be a possibly stochastic MDP/R, with
state space $\mathcal{S}$, action space $\mathcal{A}$, and transition
$T:\mathcal{S}\times\mathcal{A}\to\Delta\mathcal{S}$. Assume our agent starts
at $s_{0}$. We have that, for any time step $t$, our initial state baseline is
then given by $s^{b}_{t}=s_{0}$.

This baseline suffers from the interference problem. Since the actions of
other agents and the environment already result in changes, the agent has
incentives to force the environment back to its initial state, and thus may
undesirably interfere with the world and its agents. Therefore, we don’t
investigate this baseline further, as this level of stagnation does not seem
desirable in the real world.

####  2.1.2 The initial inaction baseline

The initial inaction baseline proposes as the baseline world the one where the
agent has never acted. In [5], the default world proposed is where the agent
has never been turned on. Therefore the inaction baseline involves
counterfactual reasoning, since the AI is turned on. We also consider as the
baseline a world where the agent just outputs an action we would consider as
”doing nothing”, such as an empty string.

Again considering an MDP/R $(\mathcal{S},\mathcal{A},T)$, we assume the action
space contains a do-nothing action $\varnothing\in\mathcal{A}$. If our agent
starts at $s_{0}$, and is currently at state $s_{t}$, we choose our initial
inaction baseline state as $s^{b}_{t}=T^{t}_{\varnothing}(s_{0})$. Since here
the transition is stochastic, $s^{b}_{t}$ is in general a random variable.

However, one problem with this approach is the offsetting problem. Suppose
that, from the initial state, the agent has already taken some impactful
action. That may be either because the agent did not engage with low-impact
reasoning at the time, or because the low-impact requirement was outweighed by
another incentive to take the action. Now, the initial inaction baseline is
that the agent should now reverse its actions as much as possible, with no
consideration for the possible harm that reversal may cause.

####  2.1.3 The future inaction baseline

In the future inaction baseline the default world is where the agent does not
act anymore in the future. So we avoid counterfactual reasoning and are left
with causal reasoning. The hope is that we avoid both the offsetting and the
interference problem, while still following our intuition of what a default
world is. In the MDP/R context, if $s_{t}$ is the state just before taking
action $a_{t}$, then we have, for any $\tau>0$, the future inaction baseline
is given by $s^{b}_{t+\tau}=T^{\tau}_{\varnothing}(s^{b}_{t})$.

The future inaction baseline still runs into problems. First, as [16] notes,
if our agent is a self-driving car, about to pass through a curve, the do
nothing action would lead to the car crashing. More generally, for any task
that requires some sequence of actions for a period of time, and that sudden
interruption is harmful, this frame seems to lead to harm.

The second issue with the future inaction baseline is ensuring that a future
window is long enough to consider the delayed impacts of the agent’s actions.
In the MDP/R context, it means that we must make an inaction rollout[23] to
evaluate the delayed effect of action $a_{t}$, considering that after taking
this action, the machine does nothing else. So we compare $s^{b}_{t+\tau}$
with $s^{a_{t}}_{t+\tau}=T_{\varnothing}^{\tau-1}(T(s_{t},a_{t}))$.

We may also avoid using fixed rollouts if we, at time step $t$, instead of
just choosing a single action $a_{t}$, chooses an entire policy $\pi$ to
follow from steps $t$ to $t+\tau$. Now, we have that such a policy would be
equivalent to choosing a $\tau$-sized initial inaction baseline starting at
$t$. Thus being prey to offsetting in that window, and to delayed impact after
that window.

However, this middle-ground may be desirable between a complete initial
inaction baseline and a complete future inaction baseline. After all, not all
offsets are bad. If some actions require us to have some short-term temporary
impact, we want that impact to be undone. When the robot that has to open a
door to get grociers, we want it to close the door before leaving [15]. So it
may seem that some short-term offsetting incentive is desirable, and even
undesirable short-term offsetting may not be too bad. That is less true when
considering long-term offsetting, since trying to reverse changes done long
ago may result in severe harm as other agents or humans may have made
decisions based on the change.

###  2.2 The impact measure problem.

We need a way of measuring the difference between the actual world state from
the baseline. What [5] calls the fundamental problem is that every single
action has a large impact in terms of atoms being rearranged, as each change
at the atomic level results in further atoms being rearranged in the next time
step, and so on. However, rather than atomic changes, we only care about
relevant ways in which the action changes the world, with some more changes
being more important than others.

One attempt at specifying ”relevant changes” may be to focus on the state of
macroscopic objects in the world, such as animals, mountains, planets, and
galaxies. We can try to get those features by some natural coarse-graining
from micro to macro. However, this does not work, as seen in the following
argument[4]:

The orbit imbalance problem:

Suppose that the AI realizes that its actions have slightly imbalanced the
Earth in one direction, and that, within a billion years, this will cause
significant deviations in the orbits of the planets, deviations it can
estimate. Compared with that amount of mass displaced, the impact of killing
all humans everywhere is a trivial one indeed.

Now, this seems wrong because humans don’t care about orbit displacement, or
about the distance between galaxies as much as human lives. So, specifying the
”value domain” on what is relevant for humans is necessary if we want to
measure impact. This makes impact learning related to [22], although, as we
discuss in Section 3, there are some important differences between the two
approaches.

Moreover, almost any action will have vastly diverging long-term effects. For
instance, if I decide to lift my finger now, weather effects can indirectly
delay a conception event by seconds, making the fecundated individual have
another genetic makeup, which will heavily change the future trajectory of the
world111This is not only a problem for low-impact agency but for moral
philosophy more generally[12, 13].. Yet we do not want to consider our system
to consider these chaotic effects when making our decision. When one
informally talks about avoiding side effects, it usually means effects that
can be somewhat predicted. So, any measure of low-impactedness should consider
some measure of predictable impact. In Figure 3, we show schematically the
idea behind each different measure.

Figure 3: How each measure makes the agent avoid catastrophic actions.

####  2.2.1 The twenty-billion question measure.

In [5], is proposed a way of measuring whether the agent has low-impact or
not: divide the world in a way that is precisely as coarse that the change of
features leads to impact (unlike atom positions) but we still have a highly
detailed description of the world. We could have billions of such features
like the air pressure in Dhaka, the mean luminosity in the poles, the closing
price of the Shangai stock exchange etc. We need to choose these features in a
manner that they don’t change when the agent exists and has low-impact. We
would not have to measure these if our agent is powerful enough to roughly
estimate them, and we can ask the agent to not deviate from the baseline it
estimates the world is in222We are assuming the agent does not tamper with his
own world-model so that he ”cheats” on himself. This is similar to the general
reward tampering problem [11].

It is useful to give a formalism in a simple MDP/R setting. Consider the MDP/R
$(\mathcal{S},\mathcal{A},T,\gamma)$. We consider $s^{b}_{t}$ to be a baseline
state at time $t$ (which may be a random variable). Now, let
$\\{\phi_{i}:\mathcal{S}\to\mathbb{R}\\}_{i=1}^{N}$ be $N$ pre-determined
feature functions. Then, for state $s_{t}$, we can define a feature vector
$\Phi(s_{t})$ whose $i$-th element is $\phi_{i}(s_{t})$. Similarly, we have a
baseline random vector $\Phi^{b}_{t}=\Phi(s^{b}_{t})$. Then, we can choose
some divergence $D$ between random vectors so that
$D(\Phi(s_{t}),\Phi^{b}_{t})$ is our measure of the difference between states
$s_{t}$ and baseline state $s^{b}_{t}$. For instance, we can consider as the
deviation the expected max-norm

$D(\Phi(s_{t}),\Phi^{b}_{t})=\mathbb{E}\left[\lVert\Phi(s_{t})-\Phi^{b}_{t}\rVert_{\infty}\right]$
(1)

So, considering some baseline $s^{t}_{b+1}$, our measure of impact of action
$a_{t}$ at state $s_{t}$ is

$D(\Phi(T(s_{t},a_{t}),\Phi^{b}_{t+1})).$ (2)

Of course, if we want to consider the agent’s beliefs, we can let $T$ and
$\phi_{i}$ be estimated by the own agent to get his (belief of) measure of
impact of action $a_{t}$ at state $s_{t}$.

####  2.2.2 Attainable utility.

In [23], the insight is that high-impact actions make many goals either easier
or harder to achieve. This suggests preserving the attainability of a variety
of goals as an impact measure. As a bonus, an agent maintaining attainability
is incentivized to not amass power nor to self-improve.

We again formalize this idea by using a MDP/R
$(\mathcal{S},\mathcal{A},T,\gamma)$, but now we use the stepwise inaction
baseline. Let $\mathcal{R}=\\{R_{1},\ldots,R_{N}\\}$ be a set of $N$ reward
functions $R_{i}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$. We let
$Q^{*}_{i}:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ be the optimal Q-value
function for reward $R_{i}$. Then, we can follow the previous formalism and
define the state-action vectors $\Phi(s_{t},a_{t})$, whose $i$-th element is
$Q^{*}_{i}(s_{t},a_{t})$, and the baseline vector $\Phi^{b}_{t}$ whose $i$-th
element is $Q^{*}_{i}(s_{t},\varnothing)$. Here, $\Phi^{b}_{t}$ is not random,
since its elements are optimal Q-values for the rewards $R_{i}$. We then
measure impact by the divergence between these vectors. In [23], the
normalized L1 distance

$\frac{\lVert\Phi(s_{t},a_{t})-\Phi^{b}_{t}\rVert_{1}}{\lVert\Phi^{b}_{t}\rVert_{1}}=\frac{\sum_{i=1}^{N}\big{|}Q^{*}_{i}(s_{t},a_{t})-Q^{*}_{i}(s_{t},\varnothing)\big{|}}{\sum_{i=1}^{N}\big{|}Q^{*}_{i}(s_{t},\varnothing)\big{|}}$

is used as the impact measure of action $a_{t}$ at state $s_{t}$.

####  2.2.3 Reachability-based measures.

High-impact actions are intuitively more irreversible in many different
aspects. Therefore, if we can penalize the disruption of the ability to reach
an old state from a new one, we can penalize the impact of actions. This is
the idea built on in [16, 17] to define reachability-based measures for
measuring impact.

Consider a MDP/R $(\mathcal{S},\mathcal{A},T,\gamma)$. For states
$x,y\in\mathcal{S}$. Let $r_{y}$ be the reward function given by $r_{y}(x)=1$
if $x=y$ and $0$ otherwise. We define then the reachability from $x$ to $y$ as
the $r_{y}$-optimal value function at $x$, given by
$R(x;y)=V^{*}_{r_{y}}(x)=\max_{\pi}\mathbb{E}\gamma^{N_{\pi}(x;y)}$, where
$N_{\pi}(x;y)$ is the number of taken to reach $y$ from $x$ following policy
$\pi$ (which is a random variable since our transition can be is stochastic).
We define the unreachability deviation measure as between state $s_{t}$ and a
baseline state $s^{b}_{t}$ as

$d_{UR}(s_{t};s^{b}_{t})=1-R(s_{t};s^{b}_{t}).$ (3)

Then, a measure of expected impact of action $a_{t}$ at state $s_{t}$ (for
some baseline $s^{b}_{t+1}$ is given by

$d_{UR}(T(s_{t};a_{t});s^{b}_{t+1}).$ (4)

Notice that, if we consider the stepwise inaction baseline, this reward is
similar to the attainable utility for a single reward $R_{1}$, except that our
reward is not fixed, and is in general a random variable, since
$\mathcal{R}=\\{r_{s^{b}_{t+1}}\\}$.

Now, a problem with the above reward is that it considers all irreversible
actions to be equal, since if we cannot reach $s^{b}_{t+1}$ from
$s_{t+1}=T(s_{t};a_{t})$, we have $R(s_{t+1};s^{b}_{t+1})=0$. To mitigate
this, we can instead consider the reachability from $s_{t+1}$ from all states
$s\in\mathcal{S}$, and compare to the reachability of $s^{b}_{t+1}$, giving
the relative reachability measure, defined as

$d_{RR}(s_{t},s^{b}_{t})=\frac{1}{|\mathcal{S}|}\sum_{s\in\mathcal{S}}\max\left(R(s^{b}_{t};s)-R(s_{t};s),0\right).$
(5)

Both relative reachability and attainable utility can be considered as an
instance of a general class of value difference measure of impact333Notice
that we are using ”value” in a technical sense of optimal path for reward
functions., based on a divergence

$d_{VD}(s_{t};s^{b}_{t})=\sum_{r\in\mathcal{R}}w_{r}f(V_{r}(s^{b}_{t})-V_{r}(s_{t})).$
(6)

In relative reachability, we have $f=\operatorname{relu}$,
$w_{r}=1/|\mathcal{S}|$, and $\mathcal{R}=\\{r_{s}\\}_{s\in\mathcal{S}}$,
while in attainable utility we have $f=\operatorname{abs}$,
$w_{r}=1/\sum_{r\in\mathcal{R}}Q(r,\varnothing)$ and $\mathcal{R}$ pre-
determined.

Finally, in [18], it is proposed a variant of the relative reachability
measure (and equivalent to it in deterministic MDP/R), called the future tasks
approach. The idea is originally proposed as a reward based on the ability to
reach future states when compared to the baseline, but we can rewrite it in a
measure of impact given by

$\operatorname{d}_{FT}(s_{t},s_{t}^{b})=1-\frac{D(s_{t})}{|\mathcal{S}|}\sum_{s\in
S}\mathbb{E}\left[\gamma^{\max(N_{s}(s_{t})-N_{s}(s_{t}^{b}),0)}\gamma^{N_{s}(s^{b}_{t})}\right],$
(7)

with $N_{s}(s_{t})$ being the minimal number of steps necessary to reach state
$s$ from state $s_{t}$, and $D(s_{t})=1$ if the state $s_{t}$ is terminal and
$1-\gamma$ otherwise. Here, we can see that the $\gamma^{N}_{s}(s^{b}_{t})$
inside the expected value acts as a sort of weight for how much diminishing
reachability to state $s$ is important when compared to the reachability from
the baseline.

####  2.2.4 Other proposals for impact measure

In [5], a variety of low-impactedness definitions are proposed which we
summarize here in a state/action paradigm.

The key idea is that having evidence that the agent took such an action should
give more information about the future of the world than some other fact. In
the framework of [5], this means that knowing that the AI is turned on ($X$),
relative to it not being turned on ($\neg X$) should be important relative to
some other facts $f$ from a set of facts $\mathcal{F}$. If we consider the
state/action framework, knowing that the AI took action $a$ (or a sequence of
such actions) should give us more information than it taking the noop action
$\varnothing$.

But how do we turn this into a measure? One approach is to consider a set of
utility functions $\mathcal{U}$, and see how much, for each utility function
$u\in\mathcal{U}$ and fact $f\in\mathcal{F}$, the random variable $u|(f\cup
a)$ compares to $u|(f\cup\varnothing)$. For instance, we can compare expected
values, having the impact measure of action $a$ (at some state $s$) as

$\max_{u\in\mathcal{U},f\in\mathcal{F}}\big{|}\mathbb{E}[u|(f\cup
a)]-\mathbb{E}[u|(f\cup\varnothing)]\big{|}.$

A similar proposal is to consider an unimpactful event to be undetectable. The
original formalization works as follows444where we again use $a$ and
$\varnothing$ in place of $X$ and $\neg X$.: suppose that the (bounded) agent
uses a probability estimate $P^{\prime}$. He can use his model of a superior
agent to get estimates of an ideal unbounded agent probability estimate $P$.
Now, let $b$ be some background information before the agent took either
action $a$ or action $\varnothing$, and let $\mathcal{G}$ be some set of
possible events in the future of the agent. Then, we want that the probability
distribution of $g\in\mathcal{G}$ to be identical (to an extent) whether $a$
or $\varnothing$ happened. That is, for an event $g\in G$, we want
$P(g|a,b)\approx P(g|\varnothing,b)$. Since the agent has a personal
probability estimate on $P$, this results in a measure of impact

$\mathbb{E}^{\prime}[P(g|a,b)-P(g|\varnothing,b)].$

In [7], a feature-based impact penalty is proposed, very similar to the
twenty-billion impact penalty, except that the measure of impact is relative
to the previous time step, and we consider the impact measure itself to be
random, instead of being a divergence between random variables. Considering
$d$ a distance function between feature vectors $\Phi$ of $s$, we have that
the impact measure is given by

$\gamma d(\Phi(T(s_{t},a_{t}),\Phi^{b}_{t+1}))-d(\Phi(s_{t}),\Phi^{b}_{t}).$

Interestingly, it is shown that, by the end of a trajectory, this relative
impact measure ends up depending only on the terminal state $s_{T}$, compared
to the baseline state $s^{b}_{T}$, given by
$\gamma^{T}d(\Phi(s_{T}),\Phi^{b}_{T})$.

####  2.2.5 Subagents and indexical impact measures.

The attainable utility and reachability measures are indexical measures, in
the sense that the divergence between worlds is measured on the difference for
the agent to attain certain states in the world. In relative reachability, an
impact is diminishing the ability to reach different states in the world,
while in attainable utility, an impact is both diminishing and increasing the
ability to achieve a predetermined number of reward functions.

In [3], it is shown that impact measures that constrain the agent to gain
power can be bypassed, by letting the agent create subagents. The conclusions
are:

  * •

If the indexical impact measure incentivizes the agent to not decrease its
ability to do something, then creating subagents does not bypass this
incentive.

  * •

If the indexical impact measure incentivizes the agent to not increase its
ability to do something, then creating subagents can bypass this incentive.

The reason is that indexical impact measures are defined for the agent’s
states and actions, not the subagents actions. There is a caveat here, in that
if the subagent is to bypass the restrictions, then it must not be controlled
by the agent when it begins to act. Otherwise, the subagent’s actions are part
of the agent’s actions. Also, the subagent must not be somewhat restricted if
the act of creating the subagent is not to be penalized by the impact measure.
However, those restrictions end up being loose[3], thus not actually
constraining the agent to gain power. Also, the subagent is a symptom of a
more general issue. For instance, the agent can instead of creating a sub-
agent convince the human to do its bidding555This problem is analogous to the
AI Boxing problem[1]..

Now, the above argument applies to subagents that can do more than the agent,
thus bypassing the restriction to not increasing its ability, but not to
subagents that hinders the agent. So the subagent problem does not directly
break reachability-based measures. However, it allows power gain, and power
gain by itself breaks reachability constraints. This is so because a powerful
enough agent can take impactful actions and maintain its ability to achieve
goals, since the action space available to a powerful agent is very large.

Notice that for non-indexical impact measures such as the twenty-billion
question, whether the agent creates a subagent or not is irrelevant since we
are only caring about states of the world. Of course, power gain will increase
its capability to have high impact even under the twenty-billion constraint,
but it seems like it hinders less the twenty-billion measure than it hinders
reachability measures.

###  2.3 World models and impact.

When considering our definitions in MDP/R settings, it is easy to forget we
should extend those definitions to the real world, because that is where our
agent will act. Yet, even if the world is deterministic, no agent will ever
have access to the true transition of the universe. Therefore, low-
impactedness will be computed according to the agent’s own world model, which
will be necessarily imperfect. This world model will be used to compute both
the baseline and the deviation from that baseline for a given action.

Since world models are subject to change, we must consider how the evaluation
of low-impact responds to that change. In particular, if the agent learns more
about the world, we don’t want for its evaluation of low-impact to decorrelate
from human intuition of what is low-impact666This possibility is known as
ontological crisis[20]. It should be noted that even humans are subjected to
it. For instance, losing faith in a religion will make some actions that were
considered very high-impact, such as blasphemy in private, become of low-
impact.. In the parlance of model splintering[2], we don’t want the concept of
low-impact to splinter under model refinement, and, if splintering ends up
happening, we want the chosen refactoring of low-impact to still correlate
with what we care about. Figure 4 schematically represents this danger.

Figure 4: Effect of changing world model.

Another issue is that, since impact is evaluated through the agent world
model, the agent has the incentive to change its own model to ”believe” it is
achieving low-impact according to its metrics, while not necessarily doing so
in the real world. This is analogous to the problem of reward tampering[11]
and wireheading[21]. However, in our view, the incentive to tamper with one
own’s world model is even greater, because world-model tampering is of
instrumental value to achieve goals in the real world, not just in getting
good reward signals.

To our knowledge, the literature on both problems, when related to low-impact
approaches, is lacking, thus making this section short. This is unfortunate,
since we see tackling these problems as important as tackling the baseline
problem and the impact measure problem, if we are to have robust low-impact
approaches.

##  3 Impact learning and value learning

Note that fully solving the value learning probably solves our act with
minimal side effects problem. In particular, if our reward function is
complete, it can assign appropriate negative rewards to unintended side
effects. So complete value learning is sufficient for achieving our goal. We
will call then the subset of value learning necessary for solving low-impact
agency impact value learning. Yet, there seems to be a few ways in which
impact value learning seems easier than complete value learning

First, impact value learning may be more robust to model misspecification than
complete value learning. First, at least in test time, we want the agent to
take actions that leave us in the low-impact region. So, if $d$ is our impact
function, and $\epsilon$ is our impact tolerance, we only care about getting
the impact function right for actions at states in the domain
$\\{(s,s^{\prime});d(s,s^{\prime})<\epsilon\\}$ right. Of course, in training
time, or when the agent is optimizing for some action, the agent may find
itself in the domain $\\{(s,s^{\prime});d(s,s^{\prime})>\epsilon\\}$. However,
as long as $d$ is correct enough to direct us to the low-impact region we care
less about the exact value of $d$ in that domain. Moreover, we have a
direction where we can err more, that is assuming something is more impactful
than it actually is. This will constrain our agent actions more than
necessary, so sacrificing some capabilities and adding alignment tax to
constraining impactful actions, in particular performance that in the
classification of [19].

Moreover, in evaluating whether something has a high impact, we can be a bit
looser in defining our features, as long as changing these looser features
results in an impact on features care about. That is, assume $\bar{\phi}$ is a
feature function we don’t care about per se, but such that if
$\tilde{\phi}(s)$ is significantly different from $\bar{\phi}(s^{\prime})$,
then a feature we care about $\phi(s)$ is significantly different from
$\phi(s^{\prime})$. Then we may use $\bar{\phi}$ as a proxy for $\phi$ when
measuring impact. An analogy is that we consider the moon disappearing a high
impact, even if we don’t care about the moon by itself, but just because of
all of the knock-on effects on the things we care about.

Finally, if we are to deploy low-impact AI in the real world, the AI should be
able to aggregate multiple human preferences on what constitutes low-impact.
Yet, this aggregation seems easier than aggregating human values, for some
reasons. First, there may be more agreement between humans on what constitutes
impact, compared to what is valuable or not. That is, letting $A$ and $B$ be
two humans, with impact functions $d_{A}$, $d_{B}$ and utility functions
$u_{A}$, $u_{B}$, $d_{A}$ is closer to $d_{B}$ than $u_{A}$ is to $u_{B}$. In
other words, Both Gandhi and Hitler would agree on whether invading Poland is
impactful, but not about its utility. Moreover, disagreement between humans on
impact values for difference actions may be more easily solved than
disagreements on values. With impacts, we can take a conservative approach
such as ”an act is impactful if it is impactful to any human”. For instance,
for humans $A$ and $B$, if we take $d_{A,B}=\max(d_{A},d_{B})$, minimizing
$d_{A,B}$ would be acceptable in the conservative viewpoint.

##  4 Achieving goals with low Impact

What we want is for our agent to have low-impact while still completing its
tasks. More specifically, if $V$ is a value function for completing some task,
and $P$ is a penalty function associated with the disvalue of causing impact,
we want to maximize $V$ while minimizing $P$. The obvious way for doing that
is by using $P$ as a regularizer, with regularization constant $\mu>0$. So we
have the regularized objective $V_{\mu,P}=V-\mu P$ that we want to maximize.
This is the approach used in [16, 18, 7, 23], but as [5] notes, this approach
is not without its own issues.

The problem is that $\mu$ is a hyperparameter that we ourselves have to fit.
Let $\mu$ be too high and our agent does nothing, while if $\mu$ is too low
our agent may cause unintended high impact. In principle, a safe way to tune
$\mu$ is to start with a very high $\mu$ that essentially does nothing, and
lower $\mu$ until the agent completes the desired task, hopefully without
harmful consequences. This assumes that the transition from ”safe and
ineffective” to ”harmful and effective” is smooth, and we can find a ”safe and
effective” middle. However, if that is not the case, it will be hard to find
the ”safe and effective range”, and this range may be unstable. Moreover, it
is not clear how well we can evaluate whether we left the ”safe” range. After
all, some long-term impacts of some actions may be unforeseen by us, but known
to the agent, so we may get the ”safe” range wrongly777This seems related to
the eliciting latent knowledge problem[10], in that we want to know what the
agent knows about what is a safe range, without having to evaluate ourselves..
Therefore, it seems that even with a correct impact penalty function, using
this penalty as a satisfactory constraint is an open problem.

While high change in utility occurs from high impact actions, high impact
actions may still have low change in utility between states. A reframing that
perhaps slightly simplifies the conservative AI problem is to constrain the
agent to not take high impact low (or highly uncertain) value actions. For
example, choosing to restrain a human that the model believes might commit a
crime has debatable value, perhaps slightly positive, but is high impact and
therefore should be avoided. The value-impact tradeoff is similar to the
return-risk tradeoff, where we wish to maximize expected value within
palatable amounts of risk. The whole motivation to care about low impact does
stem from mitigating the risk of catastrophic actions afterall. It may be
productive to explore this analogy further by designing objectives inspired by
literature on the risk-reward tradeoff in economics. For example, its
plausible that the agent shouldn’t take actions that are not on the efficient
frontier of the impact-value tradeoff curve.

##  5 Further discussion and conclusion

###  5.1 Low-impact from MDP to POMDP

Any approach to low-impact that ignores that impact depends on imperfect world
models, that are subject to both refinement and tampering, may be limited in
practice. Also, the approach should also consider that low-impact should
depend on features of the world we care about, so the concept of low-
impactedness should derive from values.

A possible framework that is still simplified, but do not ignore these
problems is the POMDP/R setting $(\mathcal{S},\mathcal{A},T,\mathcal{O},O)$,
extending with an observation model $O:\mathcal{S}\to\Delta\mathcal{O}$. Now,
we assume that the agent does not have access to neither $T$ nor
$\mathcal{O}$. Instead, letting $\mathcal{O}^{<\mathbb{N}}$ be the set of
finite sequences of observations, the agent has access to an imperfect
probabilistic world model
$\tilde{T}:\mathcal{O}^{<\mathbb{N}}\times\mathcal{A}\to\Delta\mathcal{O}$.
The agent has then to use $\tilde{T}$ to both infer the baseline and the
impact measure. The questions in this frame become:

  * •

Can the agent infer an impact measure
$d:\mathcal{O}^{<\mathbb{N}}\times\mathcal{O}^{<\mathbb{N}}\to\mathbb{R}$ from
human data? Is this easier than inferring human reward? And what does a
misspecified impact measure $\tilde{d}$ behave in presence of a true impact
measure $d$?

  * •

What agents can do to bypass this impact measure888An obvious answer is to
tamper with its own impact measure model and world-model, so we already have
to assume some restrictions here.? And what a restricted agent such as an
oracle or an LLM can do to bypass these measures? And if not, can we prove it?

  * •

How do impact measures behave under refinements $\tilde{T}^{*}$? In
particular, how does splintering break our previous impact measure?

From this, some possible steps in a research agenda would be:

  * •

Reframing proposed approaches in the POMDP setting with an imperfect world
model. Analyze, through a formal model of refinement and refactoring, how
these approaches behave under refactoring and splintering.

  * •

Analyzing whether those approaches are learnable through human data. Don’t
ignore the fact that humans are biased and flawed, and have different
preferences.

  * •

Investigating, likely through some formal agency model, whether those
approaches break with a capable enough agent. Moreover, study what constraints
should we assume to the AI, if those approaches don’t break.

  * •

Creating simplified toy world models that still obey the POMDP/R setting, and
the imperfect world model assumption, and test approaches in these simplified
toy world models. I think that we can simulate a lot in those world models.

###  5.2 Low-impact agency and RLHF

One may argue that progress in the has been constrained by debates over
definitions which have each turned out to have some shortcomings.In fact, one
issue with existing impact measures such as attainable utility is that they
seem to conflate impact with utility. As discussed in Section 4, utility and
impact are correlated but distinct concepts. Thus, we first need a way to
measure impact independently from utilities. Reinforcement Learning using
Human Feedback [14] has been proposed as a way to align models with human
intuitions. Since we wish to define ’impact’ consistent with human
intutitions, RLHF may be a way to initially circumvent this problem altogether
by making the model learn a fuzzy conception of ’impact’ specific to the
environment. Particularly, it seems useful to defer impact learning to
finetuning once the model already has a world model which it can use to
connect it’s actions, their outcomes on the world, and how humans rated their
impact. This circumvents the problem of updates to the world-model splintering
the agent’s learnt concept of impact as the finetuning can be designed to not
change the world-model (one example is training a separate impact classifier,
similar to the injury classifier [26]).

ChatGPT can be seen as an example of using RLHF to constrain the agent from
giving ’high impact’ outputs. We can view the creators decision as
constraining the goals of the model to being informative in a subset of
conversations and defined the preferences learnt by the model, biases, ability
to act as a romantic partner etc. as side effects. While correctly doing these
may be valuable to some users, this is outweighed by the risk of negative
impacts and thus ChatGPT simply states it is not designed to handle such
conversations (noop action). This just shows that ChatGPT can be viewed as an
instance of the value-impact framework, which is more general. Note that
ChatGPT’s success in preventing harmful outputs has been limited with numerous
examples being constructed that circumvent safety checks. It is possible that
a more general finetuning to avoid high-impact behaviours rather than the
specific case-by-case (recommendations, bias, research production etc.) in
ChatGPT can make it harder to circumvent safety checks. However, the model may
simply learn to manipulate the low-impact check and exploit it’s mis-
specifications. Therefore, studies are needed to know to what extent this
occur, and how this can be fixed.

###  5.3 Conclusion

Although low-impact agency seems like a promissing path toward safe AI, at
least in the medium term, there are a lot theoretical and practical issues in
not only implementing low-impactedness, but even defining it. This gives us
the temptation to study simplified settings, where we can understand the
problem more easily. However, we should not oversimplify the problem. We
prefer to explicitly say that some solution depends on some part of the
problem being solved, and pointing toward that part, than simply assuming the
part away. After all, the final objective is to implement a robust low-impact
approach in an AI system that will act in the real world, with all its
complications, and not cause harm to it.

## References

  * [1] https://www.lesswrong.com/tag/ai-boxing-containment.
  * [2] Stuart Armstrong.  Model splintering: moving from one imperfect model to another.  https://www.lesswrong.com/s/u9uawicHx7Ng7vwxA/p/k54rgSg7GcjtXnMHX, 2020\.
  * [3] Stuart Armstrong.  Subagents and impact measures, full and fully illustrated.  https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/mdQEraEZQLg7jtozn, 2020\.
  * [4] Stuart Armstrong.  General alignment plus human values, or alignment via human values?  https://www.lesswrong.com/s/u9uawicHx7Ng7vwxA/p/3e6pmovj6EJ729M2i, 2021\.
  * [5] Stuart Armstrong and Benjamin Levinstein.  Low impact artificial intelligences.  CoRR, abs/1705.10720, 2017.
  * [6] Nick Bostrom.  Superintelligence: Paths, Dangers, Strategies.  Oxford University Press, Oxford, UK, 2014.
  * [7] Lewis Hammond Alessandro Abate Charlie Griffin, Joar Skalse.  All’s well that ends well: Avoiding side effects with distance-impact penalties.  In 36th Conference on Neural Information Processing, 2022.
  * [8] Stanford Encyclopedia.  Counterfactuals.  https://plato.stanford.edu/entries/counterfactuals/, 2019.
  * [9] Stanford Encyclopedia.  The metaphysics of causation.  https://plato.stanford.edu/entries/causation-metaphysics/, 2022\.
  * [10] Paul Christiano et al.  Arc’s first technical report: Eliciting latent knowledge.  https://www.lesswrong.com/posts/qHCDysDnvhteW7kRd/arc-s-first-technical-report-eliciting-latent-knowledge, 2021\.
  * [11] Tom Everitt et al.  Designing agents incentives to avoid reward tampering.  https://deepmindsafetyresearch.medium.com/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd, 2019\.
  * [12] Hilary Greaves.  Cluelessness.  Proceedings of the Aristotelian Society, 116(3):311–339, 2016.
  * [13] Hillary Greaves.  Evidence, cluelessness and the long-term.  https://forum.effectivealtruism.org/posts/LdZcit8zX89rofZf3/evidence-cluelessness-and-the-long-term-hilary-greaves, 2020\.
  * [14] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz.  Policy shaping: Integrating human feedback with reinforcement learning.  Advances in neural information processing systems, 26, 2013.
  * [15] Victoria Krakovna.  Tradeoff between desirable properties for baseline choices.  https://www.lesswrong.com/posts/nLhfRpDutEdgr6PKe/tradeoff-between-desirable-properties-for-baseline-choices, 2020\.
  * [16] Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic, and Shane Legg.  Penalizing side effects using stepwise relative reachability.  2018\.
  * [17] Victoria Krakovna, Laurent Orseau, Miljan Martic, and Shane Legg.  Measuring and avoiding side effects using relative reachability.  CoRR, abs/1806.01186, 2018.
  * [18] Victoria Krakovna, Laurent Orseau, Richard Ngo, Miljan Martic, and Shane Legg.  Avoiding side effects by considering future tasks.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19064–19074. Curran Associates, Inc., 2020.
  * [19] Jan Leike.  Distinguishing three alignment taxes.  https://aligned.substack.com/p/three-alignment-taxes, 2022.
  * [20] LessWrong.  Ontology crisis.  https://www.lesswrong.com/tag/ontological-crisis.
  * [21] LessWrong.  Wireheading.
  * [22] Rohin Shah.  Value learning.  https://www.alignmentforum.org/s/4dHMdK5TLN6xcqtyc, 2018.
  * [23] Alexander Matt Turner, Dylan Hadfield-Menell, and Prasad Tadepalli.  Conservative agency via attainable utility preservation.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, AIES ’20, page 385–391, New York, NY, USA, 2020. Association for Computing Machinery.
  * [24] Unknown.  Low-impact.  https://arbital.com/p/low_impact/, 2018.
  * [25] Eliezer Yudkowsky.  No title.  https://twitter.com/esyudkowsky/status/1070095112791715846?lang=en, 2018\.
  * [26] Daniel M. Ziegler, Seraphina Nix, Lawrence Chan, Tim Bauman, Peter Schmidt-Nielsen, Tao Lin, Adam Scherlis, Noa Nabeshima, Ben Weinstein-Raun, Daniel de Haas, Buck Shlegeris, and Nate Thomas.  Adversarial training for high-stakes reliability.  https://arxiv.org/abs/2205.01663, 2022.


# Preparation of iron STM tips suitable for room temperature spectroscopy

Jędrzej Tepper  Huygens-Kamerlingh Onnes Laboratory, Leiden University Niels
Bohrweg 2, 2333CA Leiden  Jan M. van Ruitenbeek  Huygens-Kamerlingh Onnes
Laboratory, Leiden University Niels Bohrweg 2, 2333CA Leiden

###### Abstract

A method for preparing Fe tips suitable for spectroscopy in a room temperature
ultra-high vacuum system is presented.

The successful operation of a scanning tunneling microscope (STM) critically
depends on the atomic-scale composition of the probe tip. At present it is not
possible to fully control the apex structure of the tip at a relevant scale,
which has led many to investigate effective procedures that improve the
probability of performing a successful STM experiment. Such optimal procedures
depend on the metal that one selects as the tip material. Conventional STM
probes are often made out of tungsten, due to its mechanical stability
stemming from strong chemical bonds inside the material. Another popular
choice is platinum/iridium alloy, which is well suited for microscopes
operating in non-vacuum conditions, due to its chemical stability. One of the
more recent extensions of the original method conceived by Binning and Rohrer
Binnig  _et al._ (1982) consists of using magnetic tips to study local spin
dependent properties. In this work, a method of preparing bulk iron tips
suitable for room temperature spectroscopy is presented.

Figure 1: SEM images of Fe tips produced by electrochemical etching followed
by dry etching, performed with 5keV Ar ions, while the tip is held at a
potential of 500V. Left before, right after, each at three levels of
magnification, as indicated by the scale bars. Figure 2: SEM images, as in
Fig. 1, for the tip held at 0V during dry etching. Figure 3: From left:
electrochemically etched tip, the same tip after 60min of 1.2µA 5keV Ar
bombardment, with the tip held at 500V, 250V and 0V. Surprisingly, no blunting
can be observed, even without deflection.

The first step consists of electrochemical etching of the wire. This step
provides the necessary structure that will be used in STM to establish a
tunneling junction. Apices with radius below 50 nm can be considered a
sufficiently small structure to exclude any detrimental tip-size effects on
the acquired data in the greater majority of STM experiments. However, the
requirements towards room-temperature spectroscopy measurements are more
demanding due to the increased mobility of the surface atoms constituting the
probe. It is the external surface of the probe that will conduct current.
Therefore, an appropriate geometry for the probe is only a first prerequisite.
This can be inspected at the relevant scale using scanning electron microscopy
(SEM). For spectroscopy, even more important are the outer atomic layers of
the material, which may only be visible in transmission electron microscopy
(TEM), which is often not trivial. However, one can verify using distance/bias
dependence (z/V) spectroscopy that tips etched electrochemically out of nickel
or iron wire, upon introduction to UHV system and water removal, do not
exhibit metal-vacuum-metal tunelling characteristics. Instead, significant
semi-conducting behavior is observed. Widely used tip-sample interaction
procedures inside the ultra-high vacuum (UHV) STM, such as indenting of the
sample surface with the tip,Tewari  _et al._ (2017) or field evaporation,Chen
(2021) can be used to expose a clean metal apex. However, the stability of the
junctions formed in such a way is unsuitable for long spectroscopy
measurements at room temperature. We suspect that residues left by the
electrochemical process migrate towards the apex during the operation of the
STM.

Various techniques have been developed to mitigate this issue, such as ion
milling of a rotating tip at an angle,Biegelsen  _et al._ (1987) similar to a
pencil sharpener. In the recent work by Schmucker et al.Schmucker  _et al._
(2012), the effects are studied of on-axis Ar ion bombardment on the
electrochemically etched tungsten and platinum/iridium tips, kept at different
potentials. This approach allows for spatially selective deflection of the
incoming ions, as the electric field gradient around the sharpest structure
will be higher than that around the shaft of the tip. The effectiveness of
this method applied to the tips made out of iron is studied here. Whereas
Schmucker et al. optimized the method for sharpest tip structures that can be
employed for nanosculpting, for our purposes a tip with radius below 50 nm is
sufficient, but we require low levels of contamination and high levels of
long-term stability.

Our first study was aimed at determining the scale at which material removal
takes place during the process. Tips are exposed to Ar ion currents in a
dedicated UHV system, with base pressures below $5\cdot 10^{-9}$ mbar, which
rises to about $2\cdot 10^{-7}$ mbar during the sputtering process. The
results are initially judged from images obtained in a SEM. The first set of
images compares the effects of 5kV Ar ions directed at the tip, while the
latter is kept at deflection potentials of 500V (Fig. 1) and 0V (Fig. 2). At
the lowest magnification, the results for the dry etching are similar, and any
influence of the applied deflection potential cannot be distinguished. We only
observe a general smoothing of the electrochemically etched surface. However,
effects of the deflection potential are clear at ten times higher
magnification, where the tip without deflection can no longer be considered a
good candidate for STM experiments, in view of its increased tip radius. It
can be concluded that for 5kV Ar ions, 500V is enough to protect the sharpest
structure formed in the electrochemical process against blunting by the ion
bombardment.

This observation was utilized in the next experiment (Fig. 3), where
subsequent 5kV Ar ion treatments have been applied with a gradual reduction of
deflection. Surprisingly, the initially cleaned apex structure survived all
subsequent etching steps with only minor changes, and no significant blunting
or sharpening was observed, indicating mechanical stability of the apex
cleaned with the initial deflection potential.

The amount of material displaced in the sputtering process depends on the
kinetic energy of the ions. Gradual reduction of this parameter, akin to grit
sizes of sanding paper, removes smaller and smaller volumes of material upon
impact. This property was exploited in a subsequent experiment, where an
electrochemically etched apex is subjected to ion bombardment with gradually
decreasing kinetic energy and no deflection (Fig. 4). Not only this treatment
preserved the majority of the apex but also created nano-asperities.
Subsequent etching with high deflecting potential and again gradually
decreasing energy preserved and smoothened the entire nano-tip created during
the first etching. Although this procedure produces vary sharp tip structures
which may be useful for applications such as tip-induced nanostructuring, the
stability of the tips is insufficient for our purposes.

Figure 4: Electrochemically etched tip before dry etching (a); subsequently
dry etched without deflection potential, using a gradually reducing kinetic
energy (b), subsequently dry etched with 500V deflection potential, and with
gradual reduction of the kinetic energy (c); close up of the created nano-
structure (d).

Another promising approach could be to first blunt the stressed apex created
during the drop-off phase of electrochemical etching, or remove it, and
afterward use dry etching to sharpen it up again. While an initial experiment,
using 2.2µA of 2kV Ar ions applied for 120min, showed that this was enough to
fully remove the thin and poorly supported part of the tip, in a different
experiment, with 0.6µA for 60min, it was not. Efforts to sharpen the apex
using 1µA of 5kV Ar ions with 100V deflection potential for an hour showed
little promise. Also, the subsequent treatment of 0.6µA at 2.5kV did not
change the outlook, and the tip did not perform well in STM experiments.

Figure 5: The effects of Ar ion etching of the tip for various settings of the
deflection potential, keeping the ion beam at 2keV and 0.6µA for 60 min. (a)
Deflection potential of 200V, (b) 100V, and (c) 50V. Each image shows the tip
before (left) and after (right), at two levels of magnification.

Finally, experiments were conducted to determine how large a deflection
potential is actually needed to preserve the apex while removing residues left
by the electrochemical etching. In all experiments, 2keV Ar ions were used for
60min at 0.6µA. For etching performed with the tip kept at 200V (Fig. 5a) and
100V (Fig. 5b) the actual apex contour in the SEM micrographs is significantly
better defined. For 50V (Fig. 5c) some smoothing and improvement of definition
is observed, however, multiple tip apices can be seen, indicating the onset of
insufficient protection of the sharpest feature.

Our investigations show that for dry etching applied after electrochemical
etching without deflecting potential, and with decreasing energies ultra-sharp
asperities can be created. Those can be smoothed with the same approach but
this time utilizing deflection. Dry etching with deflecting potential always
results in, both, preservation of the electrochemically etched apex as well as
cleaning of its outer surface. Therefore this can be recommended as a safe
step before introducing the tip to the STM.

Figure 6: (Top) STM topography on $210\times 210$ nm of Co islands on Cu(111),
acquired with a room temperature UHV STM, at a sample bias of -316mV, and
current setpoint of -100pA. Post-processing involved plane fitting and
subtraction in order to remove average slopes. (Bottom) Density of states map
acquired simultaneously with the topography from the top image, using 30mV
bias modulation at 17.7758 kHz. An X-offset was subtracted. Defect-induced
modifications of the local density of states are clearly imaged, which are
invisible in topography.

Interestingly, clean metal-vacuum-metal behavior is preserved for the tip that
is dry etched in one UHV chamber, and then transferred through ambient to the
UHV-STM chamber, as long as water is removed prior to usage. The high-quality
data shown in Fig. 6 is a testimony to the junction stability achieved in such
a way. The image shows roughly triangular shaped Co islands of 2 or 3 atomic
layers in height, on top of flat Cu atomic terraces. There are no apparent tip
condition changes during the scan, a fact that can be appreciated in the
topography that has not been corrected with any line-by-line offset
subtraction. However, what is significantly harder to achieve is a very good
room temperature measurement stability for AC conductance spectroscopy using a
lock-in signal and sample bias modulation. While a minor x-offset has been
subtracted from the density of states map in Fig. 6 (bottom), the signal is
stable and reproducible and can be used to uncover otherwise invisible
material features. Note in particular the small round features in many places,
especially in the lower left corner of the image, which are due to defects,
most likely sub-surface defects.Schmid, Crampin, and Varga (2000)

In summary, Ar ion etching of electrochemically etched iron tips has been
studied. We obtain improvement of the junction stability for tips that have
been subjected to a dry etching procedure before introduction to UHV-STM. Ion
bombardment at 5 kV and a few µA Ar ions, while maintaining the tip at 500V
for protection, results in reproducibly stable tips that are sharp enough
(below 50 nm tip radius). This removes the semiconducting tunneling behaviour
observed after electrochemical etching. This finding suggests that any residue
left by the electrochemical etching is more problematic than the one left
after dry etching and subsequent transfer through ambient conditions. The
procedure has been tested for ferromagnetic Fe, but similar results were
obtained for Ni. The success of the procedure is almost exclusively limited by
the first step (electrochemical etching), from which we need to select tips
that are sufficiently sharp and straight.

## References

  * Binnig  _et al._ (1982) G. Binnig, H. Rohrer, C. Gerber, and E. Weibel, “Tunneling through a controllable vacuum gap,” Appl. Phys. Lett. 40, 178–180 (1982).
  * Tewari  _et al._ (2017) S. Tewari, K. Bastiaans, M. Allan, and J. van Ruitenbeek, “Robust procedure for creating and characterizing the atomic structure of scanning tunneling microscope tips,” Beilstein J. Nanotechn. 8, 2389–2395 (2017).
  * Chen (2021) D. Chen,  _Introduction to Scanning Tunneling Microscopy_ (Oxford Science Publications, 2021).
  * Biegelsen  _et al._ (1987) D. Biegelsen, F. Ponce, J. Tramontana, and S. Koch, “Ion milled tips for scanning tunneling microscopy,” Appl. Phys. Lett. 50, 696–698 (1987).
  * Schmucker  _et al._ (2012) S. Schmucker, N. Kumar, J. Abelson, S. Daly, G. Girolami, M. R. Bischof, D. Jaeger, R. Reidy, B. Gorman, J. Alexander, J. Ballard, J. Randall, and J. Lyding, “Field-directed sputter sharpening for tailored probe materials and atomic-scale lithography,” Nat. Commun. 3, 935 (2012).
  * Schmid, Crampin, and Varga (2000) M. Schmid, S. Crampin, and P. Varga, “Stm and sts of bulk electron scattering by subsurface objects,” J. Electr. Spectr. Rel. Phen. 109, 71–84 (2000).