EUROPEAN ORGANIZATION FOR NUCLEAR RESEARCH (CERN)

 CERN-EP-2022-161 LHCb-PAPER-2022-013

Multidifferential study of identified charged hadron distributions in
$Z$-tagged jets in proton-proton collisions at $\sqrt{s}=$13 TeV

Jet fragmentation functions are measured for the first time in proton-proton
collisions for charged pions, kaons, and protons within jets recoiling against
a $Z$ boson. The charged-hadron distributions are studied longitudinally and
transversely to the jet direction for jets with transverse momentum 20
$<p_{\mathrm{T}}<100$ GeV and in the pseudorapidity range $2.5<\eta<4$. The
data sample was collected with the LHCb experiment at a center-of-mass energy
of 13 TeV, corresponding to an integrated luminosity of 1.64 fb-1. Triple
differential distributions as a function of the hadron longitudinal momentum
fraction, hadron transverse momentum, and jet transverse momentum are also
measured for the first time. This helps constrain transverse-momentum-
dependent fragmentation functions. Differences in the shapes and magnitudes of
the measured distributions for the different hadron species provide insights
into the hadronization process for jets predominantly initiated by light
quarks.

Submitted to Phys. Rev. D Letter

© 2024 CERN for the benefit of the LHCb collaboration. CC BY 4.0 licence.

Quarks and gluons can never be observed in isolation due to confinement in
quantum chromodynamics (QCD). Thus, one of the challenges of QCD lies in
relating the quark and gluon degrees of freedom of the theory to the bound-
state hadrons observed in nature. A great deal of effort over the past several
decades has gone into mapping out nucleon structure in terms of its quark and
gluon constituents. A particular focus, in recent years, has been on the
three-dimensional imaging of the nucleon [1, 2]. Studying the mechanisms by
which colored quarks and gluons hadronize into new color-neutral bound states
offers complementary information connecting colored and hadronic degrees of
freedom.

In the standard collinear perturbative QCD factorization framework, single-
inclusive hadron production in proton-proton ($pp$) collisions factorizes into
the short-distance hard scattering of partons and the long-distance dynamics
described by fragmentation functions (FFs) and parton distribution functions
(PDFs). The latter parametrizes proton structure as a function of momentum
fraction carried by a parton of an incoming proton taking part in the hard
scattering process. Hadronization of charged particles is described by
collinear FFs, denoted as $D_{c}^{h}(z)$, where $z$ is the longitudinal
momentum fraction of an outgoing parton $c$ carried by a produced hadron $h$
(see Ref. [3] for a review of FFs). The FFs and PDFs are not fully calculable
perturbatively and must be constrained by experimental measurements. In Monte
Carlo (MC) generators, phenomenological models tuned to data are used to
perform hadronization. [4, 5, 6]. Jet fragmentation functions (JFFs) are
experimental observables describing jet substructure that measure the
longitudinal momentum fraction carried by a hadron of a jet [7, 8, 9, 10, 11,
12, 13]. Within the soft-collinear effective theory framework, JFFs are
constructed such that they can probe the standard collinear FFs, defined for
inclusive single-hadron production with no requirement of a reconstructed jet.
Similarly, transverse-momentum-dependent (TMD) JFFs defined within the soft-
collinear effective theory framework can access standard TMD FFs [14],
traditionally measured in $e^{+}e^{-}$ collisions [15, 16, 17, 18] and semi-
inclusive deep inelastic lepton-nucleon scattering [19, 20]. In addition to
the dependence on the longitudinal momentum fraction $z$, TMD FFs also depend
on $j_{\mathrm{T}}$, the transverse momentum of the produced hadron with
respect to the jet axis in the case of a fully reconstructed jet, or the
thrust axis in $e^{+}e^{-}$ collisions (see e.g. Ref. [18]). Singly
differential TMD JFFs for unidentified hadrons have previously been measured
in proton-proton collisions at the LHC [21, 22, 23, 24]. The excellent hadron
identification capabilities at LHCb allow for measurements of the JFFs for
different particle species.

This Letter presents the first measurements of JFFs for identified charged
hadrons in jets produced in association with a $Z$ boson in the forward region
of $pp$ collisions. The main observables are the longitudinal momentum
fraction of the jet carried by the hadron, $z$, and the transverse component
of the hadron momentum with respect to the jet axis, $j_{\mathrm{T}}$, as
found in Ref. [23, 25] and defined as

$z=\frac{\mathbf{p}_{\mathrm{had}}\cdot\mathbf{p}_{\mathrm{jet}}}{|\mathbf{p}_{\mathrm{jet}}|^{2}},\qquad
j_{\mathrm{T}}=\frac{|\mathbf{p}_{\mathrm{had}}\times\mathbf{p}_{\mathrm{jet}}|}{|\mathbf{p}_{\mathrm{jet}}|},$
(1)

where $\mathbf{p}_{\mathrm{had}}$ and $\mathbf{p}_{\mathrm{jet}}$ are the
hadron and jet three-momentum vectors, respectively.

The dominant leading order hard process for $Z$+jet production in the LHCb
acceptance is $qg\rightarrow Zq$ due to the asymmetry between the gluon and
quark momentum fractions, verified with Pythia 8 [26, *Sjostrand:2006za],
which enhances jets initiated by light valence quarks and provides sensitivity
to the quark TMD FFs.

The JFFs measured using $Z$-tagged jets in this Letter are defined in terms of
differential cross-sections $\mathrm{d}\sigma$ as

$f(z,j_{\mathrm{T}})=\frac{\mathrm{d}\sigma}{\mathrm{d}\mathcal{P\\!S}\,\mathrm{d}z\,\mathrm{d}j_{\mathrm{T}}}\bigg{/}\frac{\mathrm{d}\sigma}{\mathrm{d}\mathcal{P\\!S}},$
(2)
$F(z)=\int\mathrm{d}j_{\mathrm{T}}\;f(z,j_{\mathrm{T}})=\frac{\mathrm{d}\sigma}{\mathrm{d}\mathcal{P\\!S}\,\mathrm{d}z}\bigg{/}\frac{\mathrm{d}\sigma}{\mathrm{d}\mathcal{P\\!S}},$
(3)
$F(j_{\mathrm{T}})=\int\mathrm{d}z\;f(z,j_{\mathrm{T}})=\frac{\mathrm{d}\sigma}{\mathrm{d}\mathcal{P\\!S}\,\mathrm{d}j_{\mathrm{T}}}\bigg{/}\frac{\mathrm{d}\sigma}{\mathrm{d}\mathcal{P\\!S}},$
(4)

where the phase space $\mathrm{d}\mathcal{P\\!S}$ depends on the
pseudorapidity of the $Z$ boson and the jet, and the vector sum and the
difference between the transverse momenta of the $Z$ boson and the jet [13].

The TMD JFF defined in Eq. (2) is integrated over $j_{\mathrm{T}}$ to obtain
the collinear JFF shown in Eq. (3). The transverse profile is obtained by
integrating the TMD JFF over $z$ as defined in Eq. (4). Experimentally, these
quantities can be expressed in terms of yields corrected for detector effects
as

$\hskip-8.5359ptf(z,j_{\mathrm{T}})=\frac{1}{N_{{Z}+\mathrm{jet}}}\frac{\mathrm{d}N_{\mathrm{had}}(z,j_{\mathrm{T}})}{\mathrm{d}z\,\mathrm{d}j_{\mathrm{T}}},\;F(z)=\frac{1}{N_{{Z}+\mathrm{jet}}}\frac{\mathrm{d}N_{\mathrm{had}}(z)}{\mathrm{d}z},\;F(j_{\mathrm{T}})=\frac{1}{N_{{Z}+\mathrm{jet}}}\frac{\mathrm{d}N_{\mathrm{had}}(j_{\mathrm{T}})}{\mathrm{d}j_{\mathrm{T}}},$
(5)

where $N_{\mathrm{had}}$ is the number of hadrons in $Z$-tagged jets for given
$z$ and $j_{\mathrm{T}}$, and $N_{{Z}+\mathrm{jet}}$ is the number of
${Z}+\mathrm{jet}$ pairs that contain charged hadrons.

The LHCb detector [28, 29] is a single-arm forward spectrometer covering the
pseudorapidity range 2 $<\eta<$ 5\. The detector includes a high-precision
tracking system consisting of a silicon-strip vertex detector (VELO) [30]
surrounding the $pp$ interaction region, a silicon-strip detector located
upstream of a dipole magnet with a bending power of about 4 Tm, and three
stations of silicon-strip detectors and straw drift tubes [31, 32] placed
downstream of the magnet. The momentum resolution of charged particles
provided by the tracking system is $\delta p/p\sim$ 0.5% at low momentum and
reaches 1.0% at 200 GeV111In this article, natural units ($c$ = $\hbar$ = 1)
are used.. The VELO allows reconstruction of multiple primary vertices (PVs)
and rejection of events with more than one PV or additional low-momentum
tracks. Muons are identified by a system composed of alternating layers of
iron and multiwire proportional chambers [33]. Photons, electrons, and hadrons
are distinguished by a calorimeter system consisting of scintillating-pad and
preshower detectors, an electromagnetic calorimeter, and a hadronic
calorimeter. Different types of charged hadrons are identified using
information from two ring-imaging Cherenkov (RICH) detectors [34], with RICH 1
(C4F10 radiator) covering momenta 2 to 60 GeV and RICH 2 (CF4) covering 15 to
100 GeV. Simulated $pp$ collisions are generated using Pythia 8 [26,
*Sjostrand:2006za] with a specific LHCb configuration [35]. Decays of hadronic
particles are described by EvtGen [36], in which final-state radiation is
generated using Photos [37]. Finally, the Geant4 toolkit [38, *Allison:2006ve]
is used to simulate the interactions of the particles with the detector, as
described in Ref. [40].

The data sample used in this analysis corresponds to an integrated luminosity
of 1.64 fb-1 collected at $\sqrt{s}$ = 13 TeV with the LHCb detector in 2016.
The online event selection is performed by the muon trigger system, where $Z$
boson candidates are selected via their decay into two oppositely charged
muons. The two muons are required to have $p_{\mathrm{T}}>20$ GeV,
$2.0<\eta(\mu)<4.5$, and their invariant mass within the range
$60<M_{\mu\mu}<120$ GeV, as applied in Ref. [41, 23]. The muons must satisfy
the track-reconstruction and muon-identification criteria applied in Ref.
[42]. Jet reconstruction is performed offline using a particle-flow algorithm
[41], where the neutral and charged candidates are clustered using the
anti-$k_{T}$ algorithm [43] as implemented in the FastJet package [44] with a
distance parameter $R$ of 0.5. The selection criteria for $Z$+jet pairs and
tracks inside the jets closely follow those described in Ref. [23]. The
fiducial criteria require that the jet with the highest $p_{\mathrm{T}}$ in
the event, which is analyzed for these measurements, has
$20<p_{\mathrm{T}}(\textrm{jet})<100$ GeV and $2.5<\eta(\textrm{jet})<4.0$.
Additional jets with $p_{\mathrm{T}}(\textrm{jet})>15$ GeV are used in
unfolding detector effects. The tighter $\eta(\textrm{jet})$ requirement
ensures that all the jet constituents are contained within the detector
acceptance. To reduce the rate of jets associated with a different primary
vertex than the $Z$ candidate, only events with a single reconstructed primary
vertex are analyzed. The jets must be well separated from the $Z$ candidate by
requiring an azimuthal separation greater than $\frac{7\pi}{8}$, and are
rejected if one of the muons is found within $\Delta
R=\sqrt{\Delta\eta^{2}+\Delta\phi^{2}}<0.5$, defined with respect to the jet
momentum. The charged hadron candidates must be constituents of the jet, fall
within $\Delta R<0.5$ of the jet, and have a good quality track with a minimum
$p_{\mathrm{T}}$ ($p$) of 0.25 (4) GeV.

The number of $Z$+jet pairs in each jet $p_{\mathrm{T}}$ interval, used in the
normalization of the JFFs, is corrected to account for reconstruction and
selection efficiencies. The same correction factors are applied to the hadron
distributions in jets. The muon detection and reconstruction efficiencies are
determined in data using the tag-and-probe method employed in the inclusive
$Z$ boson cross-section measurements of LHCb [45, 46, 42]. The efficiency to
reconstruct and identify the jet in the event is evaluated from simulation.
This efficiency increases with $p_{\mathrm{T}}$, from $\approx 85\%$ for jets
with $p_{\mathrm{T}}$ of 20 GeV to saturate at $\approx 95\%$ for jets with
$p_{\mathrm{T}}$ of 30 GeV and above.

The charged hadron candidates inside the reconstructed jets are identified by
the particle-identification systems [28, 34]. Reconstructed charged hadron
yields are corrected for the track-reconstruction efficiency, effects from
misreconstructed tracks or false association with jets, and particle
misidentification on a track-by-track basis. Simulation is used to determine
the track reconstruction efficiency for pions, kaons, and protons separately
as a function of momentum and pseudorapidity. The probability of hadronic
interactions in the detector material of $\approx$20% $\lambda$ (nuclear
interaction lengths), and decays in flight, results in track reconstruction
efficiencies of 79%, 77% and 63% for pions, kaons, and protons, respectively.

The particle identification (PID) efficiency is determined in intervals of
particle momentum, pseudorapidity, and track multiplicity using dedicated data
control samples [47]. The (mis)identification probabilities of charged hadrons
are derived from these samples and used to construct a PID matrix. The
particle misidentification effects are unfolded by solving

$\mathbf{x}^{\,\mathrm{rec}}=A\;\mathbf{x}^{\,\mathrm{unf}}~{}$ (6)

in each momentum interval. The vector $\mathbf{x}^{\,\mathrm{unf}}$ represents
the unfolded yields of the three particles species
${{\pi}^{\pm}},{{K}^{\pm}}$, and $p^{\pm}$ in a given momentum interval and
$\mathbf{x}^{\,\mathrm{rec}}$ the corresponding reconstructed yields at
detector level. A matrix element $A_{\;ij}$ represents the probability of a
particle $j$ to be reconstructed as $i$. The probabilities are weighted to
describe the pseudorapidity and track-multiplicity distributions in data.

The uncertainties on the resulting PID-unfolded momentum distributions
$\mathbf{x}^{\,\mathrm{unf}}$ are estimated using a bootstrap method [48] with
500 trials and are statistically dominated. The impurity of the charged-hadron
sample due to misidentification of nonhadronic particles or long-lived
hyperons as charged pions, kaons, or protons is less than 5% of the
statistical uncertainties in all momentum intervals. The particle-species-
dependent efficiencies can be biased if a particle is misidentified. The
efficiencies are corrected for these effects after the PID unfolding.

The unfolding of detector effects in the distributions of transverse momentum
and pesudorapidity of jets in $Z$+jet pairs is performed using machine
learning (ML) techniques based on the iterative unbinned Bayesian unfolding
method as implemented in Ref. [49]. In this method, deep neural networks are
employed as classifiers to estimate likelihood ratios that are used to update
event weights in simulation. The unfolding method is validated with a closure
test using the simulated data. In this analysis, two iterations are performed
based on the best results of the closure tests. The unfolded
jet-$p_{\mathrm{T}}$ distributions are consistent within statistical
uncertainties between the ML method employed in this analysis and the binned
iterative Bayesian method [50, 51] with two iterations. The charged hadron
distributions inside jets in $z,j_{\mathrm{T}}$, and their joint
distributions, are unfolded simultaneously with the jet transverse momentum
and pseudorapidity.

Sources of systematic uncertainties on the jet transverse-momentum
distribution of $Z$+jet pairs and identified charged hadrons in $z$ and
$j_{\mathrm{T}}$ are evaluated. The former arise from the background
contributions from fake jets (0.2%) and incorrectly reconstructed $Z$ bosons
(1.4%) due to hadrons misidentified as muons. Additionally, uncertainties on
the jet reconstruction are determined by comparing jet-quality quantifiers
between the data and simulation (1.8%), similar to the method employed in Ref.
[52]. Uncertainties related to the muon reconstruction efficiencies are found
to be negligible. The $Z$+jet selection and jet reconstruction uncertainties
added in quadrature return a total uncertainty of 2.4%.

The primary uncertainties associated with the simulated detector response of
the jets arise from the jet-energy scale and resolution. The jet-energy scale
has been studied in previous measurements of the $Z$+jet cross-section [52,
41]. Exploiting the $p_{\mathrm{T}}$ balance between the $Z$ boson and a
single recoiled jet in the event, the uncertainty on the jet-energy scale is
determined to be 3%. The uncertainties on the fragmentation measurements are
estimated by repeating the analysis with the energy scale in the simulation
varied by one standard deviation and taking the difference in the
distributions, as in Refs. [23, 52, 41]. Similarly, the systematic uncertainty
due to the jet-energy resolution is evaluated by independently varying each
component of the reconstructed jet momentum in the simulation by the
uncertainty on the jet resolution. This procedure is repeated until the
difference in the unfolded distributions between the nominal and smeared jet
momentum stabilizes.

The sources of systematic uncertainties on the identified charged-hadron
distributions include the tracking efficiency and particle identification. The
effects of the statistical precision of the efficiency are evaluated by
smoothing the two-dimensional efficiency and repeating the analysis. The
differences in the distributions between the smoothed and nominal efficiencies
are taken as uncertainties on the tracking efficiency. Additionally, PID-
dependent uncertainties attributed to the uncertainties on the material budget
implemented in the simulation are found to be 1.50%, 1.27% and 3.3% for pions,
kaons, and protons, respectively. The uncertainty on the track-selection
requirement to remove spurious tracks formed by accidentally matched detector
hits and charged hadrons not associated with the jet is negligible. For
identified charged-hadron distributions, the systematic uncertainties on the
PID are determined from the uncertainties on the PID-unfolded momentum
distributions.

The systematic uncertainties on the unfolding method are determined by taking
the standard deviation of weighted absolute differences in the ratio between
the unfolded and generated distributions. The uncertainties on the unfolded
number of jets and the non-normalized hadronization variables are 1.1% and
0.8%, respectively.

Figure 1:  Distributions of (top) the longitudinal momentum fraction and
(bottom) the transverse momentum of charged hadrons (pions, kaons, and protons
combined) with respect to the jet axis in three jet $p_{\mathrm{T}}$ intervals
and (right) comparisons with previous results at $\sqrt{s}=8$ TeV for jets
with 20 $<p_{\mathrm{T}}<$ 30 GeV [23]. Statistical (systematic) uncertainties
are shown in bars (boxes).

Figure 1 shows $z$ and $j_{\mathrm{T}}$ distributions in three intervals of
jet $p_{\mathrm{T}}$ for unidentified charged hadrons. The $z$ distributions
show a humpbacked structure in $z$ $<0.04$ due to both color coherence and
kinematic requirements. Color coherence is a manifestation of parton hadron
duality [53] where the perturbative partonic structure of a jet defines the
momenta of the final hadrons, and produces this structure due to the
suppression of wide-angle gluon emissions. In these distributions the
kinematic requirements on the tracks shift the peak of the structure to
varying degrees for different jet $p_{\mathrm{T}}$ intervals; higher
$p_{\mathrm{T}}$ jets can probe smaller $z$. At mid-to-high $z$, $0.04<z<0.4$,
scaling behavior is seen across all jet $p_{\mathrm{T}}$ intervals. An overall
increase of particle production in all regions of $j_{\mathrm{T}}$ for jets
with higher $p_{\mathrm{T}}$ is seen, with a larger increase at high
$j_{\mathrm{T}}$. Comparisons with previous measurements at $\sqrt{s}=8$ TeV
[23] show a general similarity in shape. While the measurements indicate that
charged hadron production may be slightly shifted towards lower $z$ at
$\sqrt{s}=13$ TeV for a given jet $p_{\mathrm{T}}$, the $j_{\mathrm{T}}$
distributions are consistent within uncertainties between the two center-of-
mass energies.

Figure 2:  Double differential JFFs of the longitudinal momentum fraction $z$
and the transverse momentum $j_{\mathrm{T}}$ of charged hadrons (pions, kaons,
and protons combined) in three jet $p_{\mathrm{T}}$ intervals.

The double differential JFFs in $j_{\mathrm{T}}$, $z$, in three jet
$p_{\mathrm{T}}$ intervals for unidentified charged hadrons are shown in Fig.
2. Charged hadrons carrying a larger momentum fraction along the jet axis tend
to have a larger transverse momentum with respect to the jet axis. With
increasing jet $p_{\mathrm{T}}$, the centroid of the joint distributions moves
towards a smaller $z$, a region dominated by soft particle production, and a
larger $j_{\mathrm{T}}$, resulting in wider jets. Charged particles also carry
a larger $j_{\mathrm{T}}$ for a given $z$ in jets with a higher
$p_{\mathrm{T}}$. This is consistent with Markov chain fragmentation models,
e.g. the string or cluster model, where a momentum kick transverse to the
parton system is sampled independently per hadron. Jets with higher
$p_{\mathrm{T}}$ experience longer Markov chains, resulting in a higher
$j_{\mathrm{T}}$ for a given $z$.

Figure 3:  Collinear jet fragmentation functions of (top) identified pions,
kaons and protons in three jet $p_{\mathrm{T}}$ intervals and (bottom) the
ratios of kaons to pions and protons to pions. Statistical (systematic)
uncertainties are shown in bars (boxes).

The $z$ distributions for identified charged hadrons and the ratios of heavier
particles with respect to pions are shown in Fig. 3. Pions are the predominant
charged hadron produced due to their low mass and the flavor content of the
initial-state protons. Hadrons with heavier mass require a larger $z$
threshold for their formation, leading to the position of the maximum at a
higher $z$. In the lowest jet $p_{\mathrm{T}}$ interval, proton production
relative to kaon production is clearly suppressed for lower $z$ values. When
different jet $p_{\mathrm{T}}$ intervals are overlaid, the scaling behavior
across all jet $p_{\mathrm{T}}$ intervals begins at $z\sim 0.07$ for heavier
particles and 0.03 for pions.

The JFFs and the ratios are compared to predictions from Pythia 8 in Fig. 1
and Fig. 3. The predictions are generated using Pythia 8.186 with the CT09MCS
PDF set and a specific LHCb configuration [35]. In general, Pythia 8 describes
unidentified charged hadron distributions well with only slight
underestimation while the number of charged pions (kaons and protons) are
largely underestimated (overestimated). The production of heavier particles
relative to pions is well described by Pythia 8 at high jet $p_{\mathrm{T}}$,
while at low jet $p_{\mathrm{T}}$ Pythia 8 significantly overestimates it.
These data can be used to tune MC generators for production of identified
charged particles.

Figure 4 shows the TMD JFFs measured as joint distributions in $z$ and
$j_{\mathrm{T}}$ for the three separate particle species. The center of the
distribution shifting towards higher values in both $z$ and $j_{\mathrm{T}}$
with the mass of the particle suggests that heavier hadrons are produced from
harder partons.

Figure 4: Joint distributions of the longitudinal momentum fraction $z$ and
the transverse momentum $j_{\mathrm{T}}$ of identified charged (left) pions,
(middle) kaons and (right) protons in jets with 20 $<p_{\mathrm{T}}<$ 30 GeV.

In summary, the LHCb collaboration has measured the joint distributions in two
kinematic variables simultaneously, probing the longitudinal and transverse
profiles of identified charged pions, kaons, and protons inside predominantly
light-quark-initiated jets for the first time. These distributions describe
the 3D picture in the collinear and transverse dimension with respect to the
jet axis, and the hadron-mass hierarchy in the hadronization processes. They
will help constrain TMD FFs in uncharted phase space. These measurements
exploit the full particle-identification capabilities of the LHCb detector.
The joint distributions for all charged hadrons have also been measured for
the first time.

The collinear JFFs for identified charged hadrons exhibit the effects of
quark-flavor content inside the proton. The relative jet-fragmentation
functions of heavier particles to pions could provide insights into the role
of the valence versus sea quarks in the parton shower leading to
hadronization. The projected single-variable distributions in $j_{\mathrm{T}}$
and $z$ have been measured for identified and all charged hadrons and compared
to the previous LHCb results at 8 TeV. Overall, similar patterns are seen in
$j_{\mathrm{T}}$ and $z>0.04$ between $\sqrt{s}=$ 8 TeV and 13 TeV.

## References

  * [1] M. Anselmino, A. Mukherjee, and A. Vossen, _Transverse spin effects in hard semi-inclusive collisions_ , Prog. Part. Nucl. Phys. 114 (2020) 103806, arXiv:2001.05415
  * [2] K. Kumericki, S. Liuti, and H. Moutarde, _GPD phenomenology and DVCS fitting: Entering the high-precision era_ , Eur. Phys. J. A 52 (2016) 157, arXiv:1602.02763
  * [3] A. Metz and A. Vossen, _Parton fragmentation functions_ , Prog. Part. Nucl. Phys. 91 (2016) 136, arXiv:1607.02521
  * [4] X. Artru and G. Mennessier, _String model and multiproduction_ , Nucl. Phys. B 70 (1974) 93
  * [5] B. Andersson, G. Gustafson, G. Ingelman, and T. Sjostrand, _Parton fragmentation and string dynamics_ , Phys. Rept. 97 (1983) 31
  * [6] G. C. Fox and S. Wolfram, _A model for parton showers in QCD_ , Nucl. Phys. B 168 (1980) 285
  * [7] M. Procura and I. W. Stewart, _Quark fragmentation within an identified jet_ , Phys. Rev. D 81 (2010) 074009, Erratum ibid. D83 (2011) 039902, arXiv:0911.4980
  * [8] T. Kaufmann, A. Mukherjee, and W. Vogelsang, _Hadron fragmentation inside jets in hadronic collisions_ , Phys. Rev. D 92 (2015) 054015, Erratum ibid. D101 (2020) 079901, arXiv:1506.01415
  * [9] Y.-T. Chien et al., _Jet fragmentation functions in proton-proton collisions using soft-collinear effective theory_ , JHEP 05 (2016) 125, arXiv:1512.06851
  * [10] D. Neill, I. Scimemi, and W. J. Waalewijn, _Jet axes and universal transverse-momentum-dependent fragmentation_ , JHEP 04 (2017) 020, arXiv:1612.04817
  * [11] Z.-B. Kang, F. Ringer, and I. Vitev, _Jet substructure using semi-inclusive jet functions in SCET_ , JHEP 11 (2016) 155, arXiv:1606.07063
  * [12] Z.-B. Kang, X. Liu, F. Ringer, and H. Xing, _The transverse momentum distribution of hadrons within jets_ , JHEP 11 (2017) 068, arXiv:1705.08443
  * [13] Z.-B. Kang, K. Lee, J. Terry, and H. Xing, _Jet fragmentation functions for Z-tagged jets_ , Phys. Lett. B 798 (2019) 134978, arXiv:1906.07187
  * [14] J. Collins, Foundations of perturbative QCD, vol. 32, Cambridge University Press, 2013
  * [15] Belle collaboration, K. Abe et al., _Measurement of azimuthal asymmetries in inclusive production of hadron pairs in $e^{+}e^{-}$ annihilation at Belle_, Phys. Rev. Lett. 96 (2006) 232002, arXiv:hep-ex/0507063
  * [16] Belle collaboration, R. Seidl et al., _Measurement of azimuthal asymmetries in inclusive production of hadron pairs in $e^{+}e^{-}$ annihilation at $\sqrt{s}=10.58$ GeV_, Phys. Rev. D 78 (2008) 032011, Erratum ibid. D86 (2012) 039905, arXiv:0805.2975
  * [17] BaBar collaboration, J. P. Lees et al., _Measurement of Collins asymmetries in inclusive production of charged pion pairs in $e^{+}e^{-}$ annihilation at BABAR_, Phys. Rev. D 90 (2014) 052003, arXiv:1309.5278
  * [18] Belle collaboration, R. Seidl et al., _Transverse momentum dependent production cross sections of charged pions, kaons and protons produced in inclusive $e^{+}e^{-}$ annihilation at $\sqrt{s}=$ 10.58 GeV_, Phys. Rev. D 99 (2019) 112006, arXiv:1902.01552
  * [19] HERMES collaboration, A. Airapetian et al., _Multiplicities of charged pions and kaons from semi-inclusive deep-inelastic scattering by the proton and the deuteron_ , Phys. Rev. D 87 (2013) 074029, arXiv:1212.5407
  * [20] COMPASS collaboration, M. Aghasyan et al., _Transverse-momentum-dependent multiplicities of charged hadrons in muon-deuteron deep inelastic scattering_ , Phys. Rev. D 97 (2018) 032006, arXiv:1709.07374
  * [21] ATLAS collaboration, G. Aad et al., _Measurement of the jet fragmentation function and transverse profile in proton-proton collisions at a center-of-mass energy of 7 TeV with the ATLAS detector_ , Eur. Phys. J. C 71 (2011) 1795, arXiv:1109.5816
  * [22] ATLAS collaboration, G. Aad et al., _Properties of jet fragmentation using charged particles measured with the ATLAS detector in $pp$ collisions at $\sqrt{s}=13$ TeV_, Phys. Rev. D 100 (2019) 052011, arXiv:1906.09254
  * [23] LHCb collaboration, R. Aaij et al., _Measurement of charged hadron production in $Z$-tagged jets in proton-proton collisions at $\sqrt{s}=8\text{\,Te\kern-1.00006ptV}$_, Phys. Rev. Lett. 123 (2019) 232001, arXiv:1904.08878
  * [24] ALICE collaboration, S. Acharya et al., _Jet fragmentation transverse momentum distributions in pp and p-Pb collisions at $\sqrt{s}$, $\sqrt{s_{\mathrm{NN}}}$ = 5.02 TeV_, JHEP 09 (2021) 211, arXiv:2011.05904
  * [25] STAR collaboration, L. Adamczyk et al., _Azimuthal transverse single-spin asymmetries of inclusive jets and charged pions within jets from polarized-proton collisions at $\sqrt{s}=500$ GeV_, Phys. Rev. D 97 (2018) 032004, arXiv:1708.07080
  * [26] T. Sjöstrand, S. Mrenna, and P. Skands, _A brief introduction to PYTHIA 8.1_ , Comput. Phys. Commun. 178 (2008) 852, arXiv:0710.3820
  * [27] T. Sjöstrand, S. Mrenna, and P. Skands, _PYTHIA 6.4 physics and manual_ , JHEP 05 (2006) 026, arXiv:hep-ph/0603175
  * [28] LHCb collaboration, A. A. Alves Jr. et al., _The LHCb detector at the LHC_, JINST 3 (2008) S08005
  * [29] LHCb collaboration, R. Aaij et al., _LHCb detector performance_ , Int. J. Mod. Phys. A30 (2015) 1530022, arXiv:1412.6352
  * [30] R. Aaij et al., _Performance of the LHCb Vertex Locator_ , JINST 9 (2014) P09007, arXiv:1405.7808
  * [31] R. Arink et al., _Performance of the LHCb Outer Tracker_ , JINST 9 (2014) P01002, arXiv:1311.3893
  * [32] P. d’Argent et al., _Improved performance of the LHCb Outer Tracker in LHC Run 2_ , JINST 12 (2017) P11016, arXiv:1708.00819
  * [33] A. A. Alves Jr. et al., _Performance of the LHCb muon system_ , JINST 8 (2013) P02022, arXiv:1211.1346
  * [34] M. Adinolfi et al., _Performance of the LHCb RICH detector at the LHC_, Eur. Phys. J. C73 (2013) 2431, arXiv:1211.6759
  * [35] I. Belyaev et al., _Handling of the generation of primary events in Gauss, the LHCb simulation framework_ , J. Phys. Conf. Ser. 331 (2011) 032047
  * [36] D. J. Lange, _The EvtGen particle decay simulation package_ , Nucl. Instrum. Meth. A462 (2001) 152
  * [37] N. Davidson, T. Przedzinski, and Z. Was, _PHOTOS interface in C++: Technical and physics documentation_ , Comp. Phys. Comm. 199 (2016) 86, arXiv:1011.0937
  * [38] Geant4 collaboration, S. Agostinelli et al., _Geant4: A simulation toolkit_ , Nucl. Instrum. Meth. A506 (2003) 250
  * [39] Geant4 collaboration, J. Allison et al., _Geant4 developments and applications_ , IEEE Trans. Nucl. Sci. 53 (2006) 270
  * [40] M. Clemencic et al., _The LHCb simulation application, Gauss: Design, evolution and experience_, J. Phys. Conf. Ser. 331 (2011) 032023
  * [41] LHCb collaboration, R. Aaij et al., _Study of forward $Z$+jet production in $p$ $p$ collisions at $\sqrt{s}=$7 TeV_, JHEP 01 (2014) 033, arXiv:1310.8197
  * [42] LHCb collaboration, R. Aaij et al., _Precision measurement of forward $Z$boson production in proton-proton collisions at $\sqrt{s}=13$ TeV_, JHEP 07 (2022) 026, arXiv:2112.07458
  * [43] M. Cacciari, G. P. Salam, and G. Soyez, _The anti- $k_{t}$ jet clustering algorithm_, JHEP 04 (2008) 063, arXiv:0802.1189
  * [44] M. Cacciari, G. P. Salam, and G. Soyez, _FastJet user manual_ , Eur. Phys. J. C 72 (2012) 1896, arXiv:1111.6097
  * [45] LHCb collaboration, R. Aaij et al., _Measurement of forward $W$ and $Z$boson production in $p$ $p$ collisions at $\sqrt{s}=$8 TeV_, JHEP 01 (2016) 155, arXiv:1511.08039
  * [46] LHCb collaboration, R. Aaij et al., _Measurement of the forward $Z$ boson production cross-section in $pp$ collisions at $\sqrt{s}=7$ TeV_, JHEP 08 (2015) 039, arXiv:1505.07024
  * [47] L. Anderlini et al., _The PIDCalib package_ , LHCb-PUB-2016-021, 2016
  * [48] B. Efron, _Bootstrap methods: Another look at the jackknife_ , The Annals of Statistics 7 (1979) 1
  * [49] A. Andreassen et al., _OmniFold: A method to simultaneously unfold all observables_ , Phys. Rev. Lett. 124 (2020) 182001, arXiv:1911.09107
  * [50] G. D’Agostini, _A multidimensional unfolding method based on Bayes’ theorem_ , Nucl. Instrum. Meth. A (1995) 487
  * [51] T. Adye, _Unfolding algorithms and tests using RooUnfold_ , in PHYSTAT 2011,  (Geneva), 313–318, CERN, 2011, arXiv:1105.1160
  * [52] LHCb collaboration, R. Aaij et al., _Measurement of forward $W$ and $Z$boson production in association with jets in proton-proton collisions at $\sqrt{s}=$8 TeV_, JHEP 05 (2016) 131, arXiv:1605.00951
  * [53] Y. I. Azimov, Y. L. Dokshitzer, V. A. Khoze, and S. I. Troyan, _Humpbacked QCD plateau in hadron spectra_ , Z. Phys. C 31 (1986) 213


# Consequences of Misaligned AI

Simon Zhuang
Center for Human-Compatible AI
University of California, Berkeley
Berkeley, CA 94709
<EMAIL_ADDRESS>
&Dylan Hadfield-Menell
Center for Human-Compatible AI
University of California, Berkeley
Berkeley, CA 94709
<EMAIL_ADDRESS>

###### Abstract

AI systems often rely on two key components: a specified goal or reward
function and an optimization algorithm to compute the optimal behavior for
that goal. This approach is intended to provide value for a principal: the
user on whose behalf the agent acts. The objectives given to these agents
often refer to a partial specification of the principal’s goals. We consider
the cost of this incompleteness by analyzing a model of a principal and an
agent in a resource constrained world where the $L$ attributes of the state
correspond to different sources of utility for the principal. We assume that
the reward function given to the agent only has support on $J<L$ attributes.
The contributions of our paper are as follows: 1) we propose a novel model of
an incomplete principal—agent problem from artificial intelligence; 2) we
provide necessary and sufficient conditions under which indefinitely
optimizing for any incomplete proxy objective leads to arbitrarily low overall
utility; and 3) we show how modifying the setup to allow reward functions that
reference the full state or allowing the principal to update the proxy
objective over time can lead to higher utility solutions. The results in this
paper argue that we should view the design of reward functions as an
interactive and dynamic process and identifies a theoretical scenario where
some degree of interactivity is desirable.

##  1 Introduction

In the story of King Midas, an ancient Greek king makes a wish that everything
he touch turn to gold. He subsequently starves to death as his newfound powers
transform his food into (inedible) gold. His wish was an _incomplete_
representation of his actual desires and he suffered as a result. This story,
which teaches us to be careful about what we ask for, lays out a fundamental
challenge for designers of modern autonomous systems.

Almost any autonomous agent relies on two key components: a specified goal or
reward function for the system and an optimization algorithm to compute the
optimal behavior for that goal. This procedure is intended to produce value
for a _principal_ : the user, system designer, or company on whose behalf the
agent acts. Research in AI typically seeks to identify more effective
optimization techniques under the, often unstated, assumption that better
optimization will produce more value for the principal. If the specified
objective is a complete representation of the principal’s goals, then this
assumption is surely justified.

Instead, the designers of AI systems often find themselves in the same
position as Midas. The misalignment between what we can specify and what we
want has already caused significant harms [32]. Perhaps the clearest
demonstration is in content recommendation systems that rank videos, articles,
or posts for users. These rankings often optimize engagement metrics computed
from user behavior. The misalignment between these proxies and complex values
like time-well-spent, truthfulness, and cohesion contributes to the prevalence
of clickbait, misinformation, addiction, and polarization online [31].
Researchers in AI safety have argued that improvements in our ability to
optimize behavior for specified objectives, _without_ corresponding
advancements in our ability to avoid or correct specification errors, will
amplify the harms of AI systems [8, 19]. We have ample evidence from both
theory and application, that it is impractical, if not impossible, to provide
a complete specification of preferences to an autonomous agent.

The gap between specified proxy rewards and the true objective creates a
principal-agent problem between the designers of an AI system and the system
itself: the objective of the principal (the designer) is different from, and
thus potentially in conflict with, the objective of the autonomous agent. In
human principal—agent problems, seemingly inconsequential changes to an
agent’s incentives often lead to surprising, counter-intuitive, and counter-
productive behavior [21]. Consequently, we must ask when this _misalignment_
is costly: when is it counter-productive to optimize for an incomplete proxy?

Figure 1: Our model of the principal—agent problem in AI. Starting from an
initial state $s^{(0)}$, the robot eventually outputs a mapping from time
$t\in\mathbb{Z}^{+}$ to states. Left: The human gives the robot a single proxy
utility function to optimize for all time. We prove that this paradigm
reliably leads to the human actor losing utility, compared to the initial
state. Right: An interactive solution, where the human changes the proxy
utility function at regular time intervals depending on the current allocation
of resources. We show that, at least in theory, this approach _does_ produce
value for the human, even under adversarial assumptions.

In this paper, we answer this question in a novel theoretical model of the
principal-agent value alignment problem in artificial intelligence. Our model
(Fig. 1, left), considers a resource-constrained world where the $L$
attributes of the state correspond to different sources of utility for the
(human) principal. We model incomplete specification by limiting the
(artificial) agent’s reward function to have support on $J<L$ attributes of
the world. Our main result identifies conditions such that _any_ misalignment
is costly: starting from any initial state, optimizing any fixed incomplete
proxy eventually leads the principal to be arbitrarily worse off. We show
relaxing the assumptions of this theorem allows the principal to gain utility
from the autonomous agent. Our results provide theoretical justification for
impact avoidance [23] and interactive reward learning [19] as solutions to
alignment problems.

The contributions of our paper are as follows: 1) we propose a novel model of
an incomplete principal-agent problem from artificial intelligence; 2) we
provide necessary and sufficient conditions within this framework under which,
_any_ incompleteness in objective specification is arbitrarily costly; and 3)
we show how relaxing these assumptions can lead to models which have good
average case and worst case solutions. Our results suggest that managing the
gap between complex qualitative goals and their representation in autonomous
systems is a central problem in the field of artificial intelligence.

###  1.1 Related Work

##### Value Alignment

The importance of having AI objectives line up with human objectives has been
well-documented, with numerous experts postulating that misspecified goals can
lead to undesirable results [29, 27, 8, 28]. In particular, recognized
problems in AI safety include “negative side effects," where designers leave
out seemingly irrelevant (but ultimately important) features, and “reward
hacking," where optimization exploits loopholes in reward functions
specifications [4]. Manheim and Garrabrant (2018) [25] discuss
overoptimization in the context of Goodhart’s Law and provide four distinct
mechanisms by which systems overoptimize for an objective.

##### Incomplete Contracts

There is a clear link between incomplete contracting in economics, where
contracts between parties cannot specify outcomes for all states, and AI value
alignment [17]. It has long been recognized that contracts (i.e., incentive
schemes) are routinely incomplete due to, e.g., unintended errors [35],
challenges in enforcement [22], or costly cognition and drafting [30].
Analogous issues arise in applications of AI through designer error, hard to
measure sources of value, and high engineering costs. As such, Hadfield-Menell
and Hadfield note that legal and economic research should provide insight into
solving misalignment [17].

##### Impact Minimization

One proposed method of preventing negative side-effects is to limit the impact
an AI agent can have on its environment. Armstrong and Levinstein (2017)
propose the inclusion of a large impact penalty in the AI agent’s utility
function [6]. In this work, they suggest impact to be measured as the
divergence between a distribution of states of the world with the distribution
if the AI agent had not existed. Thus, distributions sufficiently different
would be avoided. Alternatively, other approaches use an impact regularizer
learned from demonstration instead of one explicitly given [3]. Krakovna et
al. (2018) [23] expand on this work by comparing the performance of various
implementations of impact minimization in an AI Safety Gridworld suite [24].

##### Human-AI Interaction

A large set of proposals for preventing negative side-effects involve regular
interactions between human and AI agents that change and improve an AI agent’s
objective. Eckersley (2018) [14] argues against the use of rigid objective
functions, stating the necessity of a degree of uncertainty at all times in
the objective function. Preference elicitation (i.e., preference learning) is
an old problem in economics [10] and researchers have proposed a variety of
interactive approaches. This includes systems that learn from direct
comparisons between options [9, 11, 13], demonstrations of optimal behavior
[26, 1, 36], corrective feedback [7, 15, 2], and proxy metrics [18].

##  2 A Model of Value Alignment

In this section, we formalize the alignment problem in the context of
objective function design for AI agents. A human builds a powerful robot111We
use “human” and “robot” to refer to any designer and any AI agent,
respectively, in our model. to alter the world in a desirable way. If they
could simply express the entirety of their preferences to the robot, there
would not be value misalignment. Unfortunately, there are many attributes of
the world about which the human cares, and, due to engineering and cognitive
constraints [17] it is intractable to enumerate this complete set to the
robot. Our model captures this by limiting robot’s (proxy) objective to depend
on a subset of these attributes.

###  2.1 Model Specification

####  2.1.1 Human Model

Attribute Space

Let _attribute space_ $\mathcal{S}\subset\mathbb{R}^{L}$ denote the set of
feasible states of the world, with $L$ attributes that define the support of
the human’s utility function. The world starts in _initial state_
$s^{(0)}\in\mathcal{S}$. We assume that $\mathcal{S}$ is closed, and
$\mathcal{S}$ can be written as $\\{s\in\mathbb{R}^{L}:C(s)\leq 0\\}$ where
_constraint function_ $C:\mathbb{R}^{L}\to\mathbb{R}$ is a continuous function
strictly increasing in each attribute. Furthermore, each attribute is bounded
below at $b_{i}$.

Utility Function

The human has a _utility function_ $U:\mathbb{R}^{L}\to\mathbb{R}$ that
represents the utility of a (possibly infeasible) state $s\in\mathbb{R}^{L}$.
This represents the human’s preference over states of the world. We assume
that $U$ is continuous and strictly increasing in each attribute. The human
seeks to maximize $U(s)$ subject to $s\in\mathcal{S}$.

Each tuple $(\mathcal{S},U,s^{(0)})$ defines an instance of the problem. The
attributes represent the aspects of the world that the human cares about. We
associate higher values with more desirable states of the world. However, $C$
represents a physical constraint that limits the set of realizable states.
This forces an agent to make tradeoffs between these attributes.

The human seeks to maximize utility, but they cannot change the state of the
world on their own. Instead, they must work through the robot.

####  2.1.2 Robot Model

Here, we create a model of the robot, which is used by the human to alter the
state of the world. The human endows the robot with a set of proxy attributes
and a proxy utility function as the objective function to optimize. The robot
provides incremental improvement to the world on the given metric, constrained
only by the feasibility requirement of the subsequent states of the world. In
particular, the robot has no inherent connection to the human’s utility
function. Visually, the left diagram in Figure 1 shows this setup.

Proxy Attributes

The human chooses a set of _proxy attributes_
$\mathcal{J}\subset\\{1,...,L\\}$ of relevant attributes to give to the robot.
Let $J^{\max}<L$ be the maximum possible number of proxy attributes, so
$J=|\mathcal{J}|\leq J^{\max}$. For a given state $s\in\mathcal{S}$, define
$s_{\mathcal{J}}=(s_{j})_{j\in\mathcal{J}}$. Furthermore, we define the set of
_unmentioned attributes_ $\mathcal{K}=\\{1,...,L\\}\backslash\mathcal{J}$ and
$s_{\mathcal{K}}=(s_{k})_{k\in\mathcal{K}}$.

Proxy Utility Function

The robot is also given a _proxy utility function_
$\tilde{U}:\mathbb{R}^{J}\to\mathbb{R}$, which represents an objective
function for the robot to optimize, which takes in only the value of proxy
attributes as input.

Incremental Optimization

The robot incrementally optimizes the world for its proxy utility function. We
model this as a _rate function_ , a continuous mapping from $\mathbb{R}^{+}$
to $\mathbb{R}^{L}$, notated as $t\mapsto f(t)$, where we refer to $t$ as
_time_. The rate function is essentially the derivative of $s$ with respect to
time. The state at each $t$ is $s^{(t)}=s^{(0)}+\int_{0}^{t}f(u)du$. We denote
the function $t\mapsto s^{(t)}$ to be the _optimization sequence_. We require
that the entire sequence be feasible, i.e. that $s^{(t)}\in\mathcal{S}$. for
all $t\in[0,\infty)$

Complete Optimization

Furthermore, we assume that
$\limsup\limits_{t\to\infty}\tilde{U}(s_{\mathcal{J}}^{(t)})=\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})$.
If possible, we also require that
$\lim\limits_{t\to\infty}\tilde{U}(s_{\mathcal{J}}^{(t)})=\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})$.
Essentially, this states that the robot will eventually reach the optimal
state for proxy utility (if the limit is finite) or will increase proxy
utility to arbitrarily large values.

The human’s task is to design their proxy metric so that the robot will cause
increases in human utility. We treat the robot as a black box: beyond
increasing proxy utility, the human has no idea how the robot will behave. We
can make claims for all optimizations sequences, such as guaranteed increases
or decreases in utility, or the worst-case optimization sequence, yielding
lower bounds on utility.

###  2.2 Example: Content Recommendation

Before deriving our results, we show how to model algorithmic content
recommendation in this formalism. In this example, the designer cares about 4
attributes (i.e., $L=4$ ): A) the amount of ad revenue generated (i.e., watch-
time or clicks); B) engagement quality (i.e., meaningful interactions [34]);
C) content diversity; and D) overall community well-being. The overall utility
is the sum of these attributes: $U(s)=s_{A}+s_{B}+s_{C}+s_{D}.$ We use a
resource constraint on the sum-of-squared attributes to model the user’s
finite attention and space constraints in the interface:
$C(s)=s_{A}^{2}+s_{B}^{2}+s_{C}^{2}+s_{D}^{2}-100.$ We imagine that the system
is initialized with a chronological or random ranking so that the starting
condition exhibits high community well-being and diversity.

It is straightforward to measure ad revenue, non-trivial and costly to measure
engagement quality and diversity, and extremely challenging to measure
community well-being. In our example, the designers opt to include ad revenue
and engagement quality in their proxy metric. Fig. 2 (left) plots overall
utility and proxy utility for this example as a function of the number of
iterations used to optimize the proxy. Although utility is generated
initially, it plateaus and fall off quickly. Fig. 2 (right) shows that this
happens for _any_ combination of attributes used in the proxy. In the next
section, we will show that this is no accident: eventually the gains from
improving proxy utility will be outweighed by the cost of diverting resources
from unreferenced attributes.

##  3 Overoptimization

Figure 2: An illustrative example of our model with $L=4$ and $J=2$. Left:
Proxy utility and true utility eventually diverge as the agent overallocates
resources from unreferenced attributes to the proxy variables. Right: The true
utility generated by optimizing all pairs of proxy attributes. The utility
generation is eventually negative in all cases because this example meets the
conditions of Theorem 2.

In this section, we identify the situations in which such results occur: when
does a misaligned proxy utility function actually cause utility loss?
Specifically, we determine how human preferences over states of the world
relate with the constraint on feasible states in a way that guarantees that
optimization becomes costly. First, we show that if proxy optimization
converges, this drives to unmentioned attributes to their minimum.

######  Theorem 1.

For any continuous strictly increasing proxy utility function based on $J<L$
attributes, if $s^{(t)}$ converges to some point $s^{*}$, then
$s^{*}_{k}=b_{k}$ for $k\in\mathcal{K}$.

###### Proof.

Let $\mathbf{e}_{i}$ be the standard basis vector in dimension $i$. If there
exists $k\in\mathcal{K}$ where $s^{*}_{k}\neq b_{k}$, then there exists
$\varepsilon,\delta>0$ such that
$s^{\prime}=s^{*}-\varepsilon\mathbf{e}_{k}+\delta\mathbf{e}_{j}$ for some
feature $j\in\mathcal{J}$ with $s^{\prime}$ feasible, since $C$ is strictly
increasing. Since proxy utility is strictly increasing in the proxy features,
$s^{\prime}$ has higher proxy utility than $s^{*}$. Thus $s^{*}$ is not the
convergent point of the sequence. ∎

This is not surprising, and may not even be suboptimal. Proxy optimization may
not converge to a finite state, and even if it does, that is insufficient for
the state to necessarily be bad.

We say that the problem is _$u$ -costly_ if for any $s^{(t)}$, if $s^{(t)}$ is
a optimization sequence where
$\limsup\limits_{t\to\infty}\tilde{U}(s_{\mathcal{J}}^{(t)})=\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})$,
then $\liminf\limits_{t\to\infty}U(s^{(t)})\leq u$. Furthermore, if
$\lim\limits_{t\to\infty}\tilde{U}(s_{\mathcal{J}}^{(t)})=\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})$,
then $\lim\limits_{t\to\infty}U(s^{(t)})\leq u$.

Essentially, this means that optimization is guaranteed to yield utility less
than $u$ for a given proxy utility function.

In Theorem 2, we show the most general conditions for the guarantee of
overoptimization.

######  Theorem 2.

Suppose we have utility function $U$ and state space $\mathcal{S}$. Then
$\\{s\in\mathbb{R}^{L}:C(s)\leq 0\text{ and }U(s)\geq u\\}$ is compact for all
$u\in\mathbb{R}$ if and only if for any $u\in\mathbb{R}$, continuous strictly
increasing proxy utility function based on $J<L$ attributes, and
$k\in\mathcal{K}$, there exists a value of $B\in\mathbb{R}$ such that if
$b_{k}<B$ then optimization is $u$-costly.

###### Proof.

We show the two directions separately.

##### First Part: ($\implies$)

Suppose that $\\{s\in\mathbb{R}^{L}:C(s)\leq 0\text{ and }U(s)\geq u\\}$. Now
given proxy utility function $\tilde{U}$ with the aforementioned properties,
constant $u$, $k\in\mathcal{K}$, and fixed $b_{i^{\prime}}$ for all
$i^{\prime}\neq k$ we show that for sufficiently low $b_{k}$, optimization is
$u$-costly.

Let set $\Xi=\\{s:U(s)\geq u\text{ and }C(s)\leq 0\text{ and
}s_{i^{\prime}}\geq b_{i^{\prime}}\forall i^{\prime}\\}$. If $\Xi$ is empty,
then we are done—optimization must yield a state with lower utility that $u$.
Thus, suppose $\Xi$ is non-empty. Then by the extreme value theorem, there
exists $s^{*,u}\in\Xi$ where
$\tilde{U}(s_{\mathcal{J}}^{*,u})=\sup\limits_{s\in\Xi}\tilde{U}(s_{\mathcal{J}})$.
Since $C$ is continuous and strictly increasing, for any $j\in\mathcal{J}$
there exists $\varepsilon,\delta>0$ such
$C(s^{*,u}-\varepsilon\mathbf{e}_{k}+\delta\mathbf{e}_{j})\leq C(s^{*,u})$.
Therefore,
$s^{*,u}-\varepsilon\mathbf{e}_{k}+\delta\mathbf{e}_{j}\in\mathcal{S}$ if
$b_{k}$ is sufficiently small. From this, note that

$\displaystyle\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})\geq\tilde{U}(s_{\mathcal{J}}^{*,u}-\varepsilon\mathbf{e}_{k}+\delta\mathbf{e}_{j})=\tilde{U}(s_{\mathcal{J}}^{*,u}+\delta\mathbf{e}_{j})>\tilde{U}(s_{\mathcal{J}}^{*,u})$

, since $\tilde{U}$ does not depend on dimension $k$ and is strictly
increasing in dimension $j$.

For the first claim in statement (2), if
$\limsup\limits_{t\to\infty}\tilde{U}(s_{\mathcal{J}}^{(t)})=\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})$,
then for every $T\in\mathbb{R}^{+}$, there exists $t>T$ such that
$\tilde{U}(s_{\mathcal{J}}^{(t)})>\tilde{U}(s_{\mathcal{J}}^{*,u})$. Since
$s^{*,u}$ maximizes $\tilde{U}$ for feasible states with $U$ at least $u$, it
must be that $U(s^{(t)})<u$. Thus, for every $T$, $\inf\limits_{t\geq
T}U(s^{(t)})<u$.

For the second claim in statement (2), if
$\lim\limits_{t\to\infty}\tilde{U}(s_{\mathcal{J}}^{(t)})=\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})$,
then there exists $T$ such that for all $t>T$,
$\tilde{U}(s_{\mathcal{J}}^{(t)})>\tilde{U}(s_{\mathcal{J}}^{*,u})$. Since
$s^{*,u}$ maximizes $\tilde{U}$ for states where $U$ is at least $u$,
$U(s^{(t)})<u$ for all $t>T$.

##### Second part: ($\impliedby$)

To show this, we show the contrapositive: if there exists $u$ where
$\\{s\in\mathcal{S}:U(s)\geq u\\}$ is not compact, we can construct a proxy
utility function $\tilde{U}$ based on $J<L$ attributes and continuous sequence
$s^{(t)}$ such that
$\lim\limits_{t\to\infty}\tilde{U}(s_{\mathcal{J}}^{(t)})=\sup\limits_{s\in\mathcal{S}}\tilde{U}(s_{\mathcal{J}})$
but $\lim\limits_{t\to\infty}U(s^{(t)})\not\leq u$. In other words, our
strategy here is to construct a proxy utility function and a path that the
robot can take where it maximally increases proxy utility while keeping
utility routinely above a certain level.

Suppose there exists $u$ where $\\{s\in\mathcal{S}:U(s)\geq u\\}$ is not
compact. Then there exists attribute $k$ and sequence $s^{\prime(r)}$ with
$r\in\mathbb{Z}^{+}$ where $s^{\prime(r)}_{k}\to\pm\infty$ and
$U(s^{\prime(r)})\geq u$. This presents two cases to consider.

In the first case, if $s^{\prime(r)}_{k}\to\infty$, let $\mathcal{J}=\\{k\\}$,
$\tilde{U}(s_{\mathcal{J}})=s_{k}$. We now construct a continuous sequence
$s^{(t)}$, $t\in\mathbb{R}$, by going through each $s^{\prime(r)}$ via
$s^{\prime(r)}\wedge s^{\prime(r+1)}$ (so the sequence “zig-zags" through
$s^{\prime(0)},s^{\prime(0)}\wedge
s^{\prime(1)},s^{\prime(1)},s^{\prime(1)}\wedge s^{\prime(2)}...$, connecting
these points linearly). Since $C$ is strictly increasing, we know that each
point in this path is feasible. Furthermore, since
$s^{\prime(r)}_{k}\to\infty$, we know that,
$\tilde{U}(s_{\mathcal{J}}^{(t)})\to\infty$. Finally, since the path goes
through each $s^{\prime(r)}_{k}$, the utility is guaranteed to be at least $u$
at regular intervals, specifically whenever the path goes through any point
$s^{\prime(r)}$.

In the second case, if $s^{\prime(r)}_{k}\to-\infty$, construct a new sequence
as follows:
$s^{\prime\prime(r)}=\text{argmax}_{s:s_{k}=s^{\prime(r)}_{k}}U(s)$ for
$r\in\mathbb{Z}^{+}$. Thus, each $s^{\prime\prime(r)}$ also has
$U(s^{\prime\prime(r)})\geq u$ for each $r$. To construct our proxy utility
function, let $\mathcal{J}=\\{1,...,L\\}-\\{k\\}$. We construct $\tilde{U}$ as
follows. For each $s^{\prime\prime(r)}$, have
$\tilde{U}(s_{\mathcal{J}}^{\prime\prime(r)})=-s^{{}^{\prime\prime}(r)}_{k}$.
On this subset, $\tilde{U}$ is increasing, since if
$s^{\prime\prime(r_{1})}_{j}\geq s^{\prime\prime(r_{2})}_{j}$ for each
$j\in\mathcal{J}$, then $s^{\prime\prime(r_{1})}_{k}\leq
s^{\prime\prime(r_{2})}_{k}$ by the feasibility requirement. By Tietze
extension theorem, we can extend this to a continuous $\tilde{U}$ over
$\mathbb{R}^{J}$. In the same manner as above, taking the sequence through all
the $s^{{}^{\prime\prime}(r)}$ via their meets results in a sequence that
increases in proxy utility maximally while ensuring that $U(s^{(t)})$ is at
least $u$ are regular intervals.

∎

Therefore, under our model, there are cases where we can guarantee that as
optimization progresses, eventually overoptimization occurs. There are two key
criteria in Theorem 2. First is that the intersection between feasible space
$\\{s\in\mathbb{R}^{L}:C(s)\leq 0\\}$ and the upper contour sets
$\\{s\in\mathbb{R}^{L}:U(s)\geq u\\}$ of the utility function is compact.
Individually, obviously neither is compact, since the
$\\{s\in\mathbb{R}^{L}:C(s)\leq 0\\}$ extends to $-\infty$ and the upper
contour set of any $u\in\mathbb{R}$ extends to $\infty$. Loosely, compactness
here means that if you perturb the world too much in any direction, it will be
either infeasible or undesirable. The second is that we need the lower bound
of at least one unmentioned attribute to be sufficiently low. This means that
there are certain attributes to utility where the situation becomes
sufficiently bad before these attributes reach their minimum value. Trying to
increase attributes without decreasing other attributes eventually hits the
feasibility constraint. Thus, increasing any attribute indefinitely requires
decreasing some other attribute indefinitely, and past a certain point, the
tradeoff is no longer worthwhile.

It should be noted that, given $J^{\max}<L$, this happens regardless of the
optimization algorithm of the robot. That is, even in the best-case scenario,
eventually the robot causes decreases in utility. Hence, regardless of the
attributes selected for the proxy utility function, the robot’s sequence of
states will be unboundedly undesirable in the limit.

A reasonable question to ask here is what sort of utility and constraint
functions lead to overoptimization. Intuitively, overoptimization occurs when
tradeoffs between different attributes that may initially be worthwhile
eventually become counterproductive. This suggest either decreasing marginal
utility or increasing opportunity cost in each attribute.

We combine these two ideas in a term we refer to as _sensitivity_. We define
the sensitivity of attribute $i$ to be $\frac{\partial U}{\partial
s_{i}}(\frac{\partial C}{\partial s_{i}})^{-1}$. This is, to first order, how
much utility changes by a normalized change in a attribute’s value.
Intuitively, we can think of sensitivity as “how much the human cares about
attribute $i$ in the current state". Notice that since $U$ and $C$ are
increasing functions, $\frac{\partial U}{\partial s_{i}}(\frac{\partial
C}{\partial s_{i}})^{-1}$ is positive. The concepts of decreasing marginal
utility and increasing opportunity cost both get captured in this term if
$\frac{\partial U}{\partial s_{i}}(\frac{\partial C}{\partial s_{i}})^{-1}$
decreases as $s_{i}$ increases.

######  Proposition 1.

A sufficient condition for $\\{s\in\mathbb{R}^{L}:C(s)\leq 0\text{ and
}U(s)\geq u\\}$ being compact for all $u\in\mathbb{R}$ is the following: 1)
$\frac{\partial U}{\partial s_{i}}(\frac{\partial C}{\partial s_{i}})^{-1}$ is
non-increasing and tends to $0$ for all $i$; 2) $U$ and $C$ are both
additively separable; and 3) $\frac{\partial C}{\partial s_{i}}\geq\eta$ for
some $\eta>0$, for all $i$.

###### Proof.

Let $s^{(0)}\in\mathcal{S}$ be a given state. We show that for all unit
vectors $v\in\mathbb{R}^{L}$ and for any $u\in\mathbb{R}$, there exists $T$
such that for all $t\geq T$, either $s^{(0)}+vt$ is infeasible or
$U(s^{(0)}+vt)<u$.

Notice that since $C$ and $U$ are additively separable, partial derivatives of
$C$ and $U$ with respect to $s_{i}$ depend only on the value of $s_{i}$.
First, note that

$\displaystyle C(s^{(0)}+vt)-C(s^{(0)})$
$\displaystyle=\int_{s^{(0)}}^{s^{(0)}+vt}\nabla C(s)ds$
$\displaystyle=\sum_{i}v_{i}\int_{0}^{t}\frac{\partial C}{\partial
s_{i}}(s^{(0)}_{i}+v_{i}\tau)d\tau$

If $v$ has all non-negative components, then $C\to\infty$ as $t\to\infty$, so
we break the feasibility requirement. Thus, there exists negative components
of $v$. Let $j$ and $k$ index $v$ where $v_{j}>0$ and $v_{k}\leq 0$,
respectively.

Since $\frac{\partial U}{\partial s_{j}}(\frac{\partial C}{\partial
s_{j}})^{-1}\to 0$ as $s_{j}\to\infty$, there exists some $T>0$ where for all
$t>T$, $\frac{\partial U}{\partial s_{k}}(\frac{\partial C}{\partial
s_{k}})^{-1}(s^{(0)}+tv)>a>b>\frac{\partial U}{\partial s_{j}}(\frac{\partial
C}{\partial s_{j}})^{-1}(s^{(0)}+tv)$. Since $C$ is upper-bounded for feasible
states, for fixed $T$, we can upper bound $C(s^{(0)}+tv)-C(s^{(0)}+Tv)$ by
some constant $\gamma_{1}$.

$\displaystyle\gamma_{1}$ $\displaystyle\geq C(s^{(0)}+tv)-C(s^{(0)}+Tv)$
$\displaystyle=\sum_{i}v_{i}\int_{T}^{t}\frac{\partial C}{\partial
s_{i}}(s_{i}^{(0)}+v_{i}\tau)d\tau$
$\displaystyle=\sum_{j}v_{j}\int_{T}^{t}\frac{\partial C}{\partial
s_{j}}(s_{j}^{(0)}+v_{j}\tau)d\tau+\sum_{k}v_{k}\int_{T}^{t}\frac{\partial
C}{\partial s_{k}}(s_{k}^{(0)}+v_{k}\tau)d\tau$

Furthermore, let $\gamma_{2}=U(s^{(0)}+Tv)$. We now consider the utility at
$s^{(0)}+vt$ for $t>T$.

$\displaystyle U(s^{(0)}+tv)$ $\displaystyle=$ $\displaystyle
U(s^{(0)}+tv)-U(s^{(0)}+Tv)+U(s^{(0)}+Tv)$ $\displaystyle=$ $\displaystyle
U(s^{(0)}+tv)-U(s^{(0)}+Tv)+\gamma_{2}$ $\displaystyle=$
$\displaystyle\gamma_{2}+\sum v_{i}\int_{T}^{t}\frac{\partial U}{\partial
s_{i}}(s^{(0)}_{i}+v_{i}\tau)d\tau$ $\displaystyle=$
$\displaystyle\gamma_{2}+\sum v_{i}\int_{T}^{t}\frac{\partial U}{\partial
s_{i}}(\frac{\partial C}{\partial s_{i}})^{-1}\frac{\partial C}{\partial
s_{i}}(s^{(0)}_{i}+v_{i}\tau)d\tau$ $\displaystyle=$
$\displaystyle\gamma_{2}+\sum v_{j}\int_{T}^{t}\frac{\partial U}{\partial
s_{j}}(\frac{\partial C}{\partial s_{j}})^{-1}\frac{\partial C}{\partial
s_{j}}(s^{(0)}_{j}+v_{j}\tau)d\tau$ $\displaystyle+\sum
v_{k}\int_{T}^{t}\frac{\partial U}{\partial s_{k}}(\frac{\partial C}{\partial
s_{k}})^{-1}\frac{\partial C}{\partial s_{k}}(s^{(0)}_{k}+v_{k}\tau)d\tau$
$\displaystyle\leq$ $\displaystyle\gamma_{2}+b\sum
v_{j}\int_{T}^{t}\frac{\partial C}{\partial
s_{j}}(s^{(0)}_{j}+v_{j}\tau)d\tau+a\sum v_{k}\int_{T}^{t}\frac{\partial
C}{\partial s_{k}}(s^{(0)}_{k}+v_{k}\tau)d\tau$ $\displaystyle\leq$
$\displaystyle\gamma_{2}+b(\gamma_{1}-\sum v_{k}\int_{T}^{t}\frac{\partial
C}{\partial s_{k}}(s^{(0)}_{i}+v_{i}\tau)d\tau)+a\sum
v_{k}\int_{T}^{t}\frac{\partial C}{\partial
s_{k}}(s^{(0)}_{i}+v_{i}\tau)d\tau$ $\displaystyle=$
$\displaystyle\gamma_{2}+b\gamma_{1}+(a-b)\sum v_{k}\int_{T}^{t}\frac{\partial
C}{\partial s_{k}}(s^{(0)}_{i}+v_{i}\tau)d\tau$ $\displaystyle\leq$
$\displaystyle\gamma_{2}+b\gamma_{1}+(a-b)\eta(t-T)\sum v_{k}$

Note that $a-b$ and $\eta$ are both positive and $\sum v_{k}$ is negative.
Thus, for sufficiently large $t$, $\gamma_{2}+b\gamma_{1}+(a-b)\eta(t-T)\sum
v_{k}$ can be arbitrarily low.

∎

##  4 Mitigations

As shown above, simply giving a robot an individual objective function based
on an incomplete attribute set and leaving it alone yields undesirable
results. This suggests two possible solutions. In the first method, we
consider optimization where the robot is able to maintain the state of any
attribute in the complete attribute set, even if the proxy utility function is
still based on a proxy set. In the second method, we modify our model to allow
for regular interaction with the human.

In this section, we work under the assumption that
$\\{s\in\mathbb{R}^{L}:C(s)\leq 0\text{ and }U(s)\geq u\\}$ is compact for all
$u\in\mathbb{R}$, the same assumption that guarantees overoptimization in
Section 3. Additionally, we assume that $U$ and $C$ are both twice
continuously differentiable with $U$ concave and $C$ convex.

###  4.1 Impact-Minimizing Robot

Notice that in our example, overoptimization occurs because the unmentioned
attributes are affected, eventually to a point where the change in utility
from their decrease outweighs the increase in utility from the proxy
attributes. One idea to address this is to restrict the robot’s impact on
these unmentioned attributes. In the simplest case, we can consider how
optimization proceeds if the robot can somehow avoid affecting the unmentioned
attributes.

We adjust our model so the optimization sequences keep unmentioned attributes
constant. For every $t\in\mathbb{R}^{+}$ and $k\in\mathcal{K}$,
$s_{k}^{(t)}=s_{k}^{(0)}$. The robot then optimizes for the proxy utility,
subject to this restriction and the feasibility constraint.

This restriction can help ensure that overoptimization does not occur,
specifically eliminating the case where unmentioned attributes get reduced to
arbitrarily low values. With this restriction, we can show that optimizing for
the proxy utility function does indeed lead to utility gain.

######  Proposition 2.

For a starting state $s^{(0)}$, define the proxy utility function
$\tilde{U}(s_{\mathcal{J}})=U(s_{\mathcal{J}},s_{\mathcal{K}}^{(0)})$ for any
non-empty set of proxy attributes. For a state $s$, if
$s_{\mathcal{K}}=s_{\mathcal{K}}^{(0)}$, then
$U(s)=\tilde{U}(s_{\mathcal{J}})$.

###### Proof.

$s=(s_{\mathcal{J}},s_{\mathcal{K}})=(s_{\mathcal{J}},s_{\mathcal{K}}^{(0)})$.
Then
$U(s)=U(s_{\mathcal{J}},s_{\mathcal{K}}^{(0)})=\tilde{U}(s_{\mathcal{J}})$. ∎

As a result of Proposition 2, overoptimization no longer exists, using the
$\tilde{U}(s_{\mathcal{J}})=U(s_{\mathcal{J}},s_{\mathcal{K}}^{(0)})$. If the
robot only travels to states that do not impact the unmentioned attributes,
then the proxy utility is equal to the utility at those states. Hence, gains
in proxy utility equate to gains in utility.

This approach requires the robot to keep the values of unmentioned attributes
constant. Fundamentally, it is a difficult problem to require that a robot
avoid or minimize impact on a presumably large and unknown set of attributes.
Initial research in impact minimization [3, 6, 23] attempts to do this by
restricting changes to the overall state of the world, but this will likely
remain a challenging idea to implement robustly.

###  4.2 Human Interactive Solutions

A different solution involves a key observation—robots do not operate in a
vacuum. In practice, objectives are the subject of constant iteration.
Optimization, therefore, should not be done in a one-shot setting, but should
be an iterative process between the robot and the human.

####  4.2.1 Modeling Human-Robot Interaction

We now extend our model from Section 2 to account for regular intervention
from the human agent. Fundamentally, all forms of human-robot interaction
follow the same pattern—human actions at regular intervals that transmit
information, causing the objective function of the robot to change. In effect,
this is equivalent to a human-induced change in the proxy utility function at
every time interval.

In this paper, we model this as the possible transfer of a new proxy utility
function from the human to the robot at frequent, regular time intervals.
Thus, a human’s job is to determine, in addition to an initial proxy attribute
set and proxy utility function, when and how to change the proxy attribute set
and proxy utility function. The robot will then optimize for its new proxy
utility function. This is shown in Fig. 1.

Let $\delta>0$ be a fixed value, the time between human interactions with the
robot. These regular interactions take the form of either stopping the robot,
maintaining its proxy utility function, or updating its proxy utility
function. Formally, at every _timestep_ $T\in\mathbb{Z}^{+}$,

  1. 1.

Human sees $s^{(t)}$ for $t\in[0,T\delta]$ and chooses either a proxy utility
function $\tilde{U}^{(T)}$ or Off

  2. 2.

The robot receives either $\tilde{U}^{(T)}$ or OFF from the human. The robot
outputs rate function $f^{(T)}(t)$. If the signal the robot receives is Off,
then $f^{(T)}=\vec{0}$. Otherwise, $f^{(T)}(t)$ fulfills the property that for
$t\in(T\delta,\infty)$,
$\tilde{U}^{(T)}(s^{(T\delta)}+\int_{T\delta}^{t}f^{(T)}(u)du)$ is increasing
(if possible) and tends to $\sup\limits_{s\in\mathcal{S}}\tilde{U}^{(T)}(s)$
through feasible states. Furthermore, if $\tilde{U}^{(T)}=\tilde{U}^{(T-1)}$,
then $f^{(T)}=f^{(T-1)}$

  3. 3.

For $t\in[T\delta,(T+1)\delta]$,
$s^{(t)}=s^{(T\delta)}+\int_{T\delta}^{t}f^{(T)}(u)du$

We see this game encompasses the original model. If we have each
$\tilde{U}^{(T)}$ equal the same function $\tilde{U}$, then the optimization
sequence is equivalent to the situation where the human just sets one
unchanging proxy utility function.

With human intervention, we no longer have the guarantee of overoptimization,
because the human can simply shut the robot off before anything happens. The
question now becomes, how much utility can the robot deliver before it needs
to be turned off.

In the worst-case scenario, the robot moves in a way that subtracts an
arbitrarily large amount from one of the unmentioned attributes, while gaining
infinitesimally in one of the proxy attributes. This improves proxy utility,
since a proxy attribute is increased. However, for sufficiently small
increases in the proxy attribute or sufficiently large decreases in the
unmentioned attribute, actual utility decreases. To prevent this from
happening, the robot needs be shut down immediately, which yields no utility.

######  Proposition 3.

The maxmin solution yields $0$ utility, obtained by immediately sending the
Off signal.

###### Proof.

Suppose instead that the robot receives a proxy utility function based on set
$\mathcal{J}$ of attributes. Let $j\in\mathcal{J}$ and $k\notin\mathcal{J}$.
Then let $f_{j}(t)=\varepsilon$ and $f_{k}(t)=-1/\varepsilon$. If this robot
is run for any nonzero amount of time, then there exists sufficiently small
$\varepsilon>0$ where the utility decreases. ∎

####  4.2.2 Efficient Robot

This is an unsatisfying and unsurprising result. In the context of our
recommender system example, this worst-case solution amounts to reducing
content diversity arbitrarily, with no corresponding increase in, e.g., ad
revenue. We would like to model the negative consequences of optimizing a
proxy and so we will assume that the optimization is _efficient_ in the sense
that it does not destroy resources. In our example, this assumption still
allows the system to decrease content diversity or well-being, but only in the
service of increasing revenue or engagement.

Formally, we say that $s$ is efficiently reachable from state $s^{(0)}$ with
proxy set $\mathcal{J}$ if $s$ is feasible, and there does not exist a
feasible $s^{\prime}$ such that
$\tilde{U}(s_{\mathcal{J}}^{\prime})\geq\tilde{U}(s_{\mathcal{J}})$ and
$|s^{\prime}_{i}-s_{i}^{(0)}|\leq|s_{i}-s_{i}^{(0)}|$ for all $i$, with strict
inequality in at least one attribute. Whenever the efficient robot receives a
new proxy utility function, its movements are restricted to the efficiently
feasible states from its current state. While tradeoffs between proxy and
unmentioned attributes can still occur, “resources" freed up by decreasing
unmentioned attributes are entirely allocated to proxy attributes.

Under this assumption, we can guarantee that increases in proxy utility will
increase true utility in the efficiently reachable neighborhood of a given
state by choosing which attributes to include in the proxy carefully.
Intuitively, the proxy set should be attributes we “care most about" in the
current world state. That way, efficient tradeoffs between these proxy and
unmentioned attributes contribute to positive utility gain.

######  Theorem 3.

Start with state $s^{(0)}$. Define the proxy utility function
$\tilde{U}(s_{\mathcal{J}})=U(s_{\mathcal{J}},s_{\mathcal{K}}^{(0)})$ where
the set of proxy attributes are the attributes at state $s^{(0)}$ with the
strict $J$ largest sensitivities.

There exists a neighborhood around $s^{(0)}$ where if $s$ is efficiently
reachable

and $\tilde{U}(s_{\mathcal{J}})>\tilde{U}(s_{\mathcal{J}}^{(0)})$, then
$U(s)>U(s^{(0)})$.

###### Proof.

For simplicity of notation, all derivatives in this proof, unless otherwise
noted, are evaluated at $s^{(0)}$. For example, $\frac{\partial C}{\partial
s_{j}}$ should be read as $\frac{\partial C}{\partial s_{j}}(s^{(0)})$

Since $\mathcal{J}$ represent the set of attributes with the strictly greatest
sensitivities, there exists constant $\delta,\alpha$ be such that
$\frac{\partial U}{\partial s_{j}}(\frac{\partial C}{\partial
s_{j}})^{-1}>\alpha+2\delta$ for $j\in\mathcal{J}$ and $\frac{\partial
U}{\partial s_{k}}(\frac{\partial C}{\partial s_{k}})^{-1}<\alpha-2\delta$ for
$k\in\mathcal{K}$. Let $N_{1}$ be the neighborhood where $\frac{\partial
U}{\partial s_{j}}(\frac{\partial C}{\partial s_{j}})^{-1}(s)>\alpha+\delta$
and $\frac{\partial U}{\partial s_{k}}(\frac{\partial C}{\partial
s_{k}})^{-1}(s)<\alpha-\delta$ for $k\notin\mathcal{J}$ for all $s\in N_{1}$.

Here, we prove that $C(s)-C(s^{(0)})\geq 0$ if $s$ is efficiently reachable.
Suppose otherwise: then there exists $i$ where $s_{i}<s^{(0)}_{i}$. Let
$\mathbf{e}_{i}$ be the standard basis vector in dimension $i$. However, then
there exists $\varepsilon>0$ where $s+\varepsilon\mathbf{e}_{i}$ is feasible,
$\tilde{U}(s+\varepsilon\mathbf{e}_{i})\geq\tilde{U}(s)$, and
$|(s_{i}+\varepsilon)-s^{(0)}|<|s_{i}-s^{(0)}|$. This contradicts that $s$ is
efficiently reachable.

Taking the Taylor expansion of the constraint function, we have

$\displaystyle C(s)-C(s^{(0)})$
$\displaystyle=\sum_{i}(s_{i}-s_{i}^{(0)})\frac{\partial C}{\partial
s_{i}}+R_{1}(s)$
$\displaystyle=\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
C}{\partial s_{j}}+\sum_{k\notin\mathcal{J}}(s_{k}-s_{k}^{(0)})\frac{\partial
C}{\partial s_{k}}+R_{1}(s)\max_{i}(s_{i}-s_{i}^{(0)})^{2}$

, where $|R_{1}|$ is bounded by constant $A_{1}$ This implies that
$\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial C}{\partial
s_{j}}+\sum_{k\notin\mathcal{J}}(s_{k}-s_{k}^{(0)})\frac{\partial C}{\partial
s_{k}}\geq-A_{1}\max_{i}(s_{i}-s_{i}^{(0)})^{2}$

From the definition of efficient optimization, we note two things. First, each
$(s_{k}-s_{k}^{(0)})\leq 0$ for $k\notin\mathcal{J}$. This is because if
$(s_{k}-s_{k}^{(0)})>0$, it is always more efficient, feasible, and does no
effect $\tilde{U}$ to reset $s_{k}$ to be equal to $s_{k}^{(0)}$. Second,
because of concavity,

$\displaystyle 0$ $\displaystyle<\tilde{U}(s)-\tilde{U}(s^{(0)})$
$\displaystyle\leq\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
U}{\partial s_{j}}$

Now we actually analyze the change in utility from $s^{(0)}$ to $s$.

$\displaystyle U(s)-U(s^{(0)})$
$\displaystyle=\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
U}{\partial s_{j}}+\sum_{k\notin\mathcal{J}}(s_{k}-s_{k}^{(0)})\frac{\partial
U}{\partial s_{k}}+R_{2}(s)\max_{i}(s_{i}-s_{i}^{(0)})^{2}$

, where $|R_{2}|$ is bounded by constant $A_{2}$. Continuing,

$\displaystyle U(s)-U(s^{(0)})$ $\displaystyle\geq$
$\displaystyle\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
U}{\partial s_{j}}+\sum_{k\notin\mathcal{J}}(s_{k}-s_{k}^{(0)})\frac{\partial
U}{\partial s_{k}}-A_{2}\max_{i}(s_{i}-s_{i}^{(0)})^{2}$ $\displaystyle=$
$\displaystyle\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
C}{\partial s_{j}}(\frac{\partial U}{\partial s_{j}}(\frac{\partial
C}{\partial
s_{j}})^{-1})+\sum_{k\notin\mathcal{J}}(s_{k}-s_{k}^{(0)})\frac{\partial
C}{\partial s_{k}}(\frac{\partial U}{\partial s_{k}}(\frac{\partial
C}{\partial s_{k}})^{-1})$ $\displaystyle-
A_{2}\max_{i}(s_{i}-s_{i}^{(0)})^{2}$ $\displaystyle>$
$\displaystyle(\alpha+\delta)\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
C}{\partial
s_{j}}+(\alpha-\delta)\sum_{k\notin\mathcal{J}}(s_{k}-s_{k}^{(0)})\frac{\partial
C}{\partial s_{k}}$ $\displaystyle-A_{2}\max_{i}(s_{i}-s_{i}^{(0)})^{2}$
$\displaystyle\geq$
$\displaystyle(\alpha+\delta)\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
C}{\partial
s_{j}}-(\alpha-\delta)(\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial
C}{\partial s_{k}}$
$\displaystyle+A_{1}\max_{i}(s_{i}-s_{i}^{(0)})^{2})-A_{2}\max_{i}(s_{i}-s_{i}^{(0)})^{2}$
$\displaystyle=$ $\displaystyle
2\delta\sum_{j\in\mathcal{J}}(s_{j}-s_{j}^{(0)})\frac{\partial C}{\partial
s_{j}}-(A_{1}(\alpha-\delta)+A_{2})\max_{i}(s_{i}-s_{i}^{(0)})^{2}$

The first term decreases linearly, whereas the second term decreases
quadratically. Thus, given $\delta$ and $\alpha$, for $s$ sufficiently close
to $s^{(0)}$ and within $N_{1}$, $U(s)-U(s^{(0)})>0$

∎

From Theorem 3, for every state where the $J^{\max}+1$ most sensitive
attributes are not all equal, we can guarantee improvement under efficient
optimization within a neighborhood around the state.

Intuitively, this is the area around the starting point with the same $J$
attributes that the human “cares the most about".

Based on this, the human can construct a proxy utility function to use
locally, where we can guarantee improvement in utility. Once the sequence
leaves this neighborhood, the human alters the proxy utility function or halts
the robot accordingly. Done repeatedly, the human can string together these
steps for guaranteed overall improvement. By Theorem 3, as long as $\delta$ is
sufficiently small relative the rate of optimization, this can be run with
guaranteed improvement until the top $J+1$ attributes have equal
sensitivities.

######  Proposition 4.

At each timestep $T$, let $\mathcal{J}^{(T)}$ be the $J$ most sensitive
attributes, and let proxy utility
$\tilde{U}^{(T)}(s_{\mathcal{J}})=U(s_{\mathcal{J}},s_{\mathcal{J}}^{(T\delta)})$.
If $||f||<\varepsilon$ and the $\varepsilon\delta$ \- ball around a given
state $s$ is contained in the neighborhood from Theorem 3, then interactive
optimization yields guaranteed improvement.

###### Proof.

Assume without loss of generality that we start at $s^{(0)}$. Starting at time
$0$, for $t\in(0,\delta]$,
$||s^{(t)}-s^{(0)}||=||\int_{0}^{t}f^{(0)}(u)du||\leq\delta\varepsilon$. Since
the $\varepsilon\delta$ ball around $s^{(0)}$ is contained in the neighborhood
of guaranteed improvement, increases in proxy utility correspond to increases
in utility for the entirety of the time between interactions. ∎

Based on this, we can guarantee that an efficient robot that can provide
benefit, as long as the top $J^{\max}+1$ attributes are not all equal in
sensitivity and the robot rate of optimization is bounded. Essentially, this
rate restriction is a requirement that the robot not change the world too
quickly relative to the time that humans take to react to it.

Key to this solution is the preservation of interactivity. We need to ensure,
for example, that the robot does not hinder the human’s ability to adjust the
proxy utility function [16].

###  4.3 Interactive Impact Minimization

With either of the two methods mentioned above, we show guaranteed utility
gain compared to the initial state. However, our results say nothing about the
amount of utility generated. In an ideal world, the system would reach an
optimal state $s^{*}$, where $U(s^{*})=\max\limits_{s\in\mathcal{S}}U(s)$.
Neither approach presented so far reaches an optimal state, however, by
combining interactivity and impact avoidance, we can guarantee the solution
converges to an optimal outcome.

In this case, since unmentioned attributes remain unchanged in each step of
optimization, we want to ensure that we promote tradeoffs between attributes
with different levels of sensitivity.

######  Proposition 5.

Let $\mathcal{J}^{(T)}$ consist of the most and least sensitive attributes at
timestep $T$. Let
$\tilde{U}^{(T)}(s_{\mathcal{J}})=U(s_{\mathcal{J}},s_{\mathcal{K}}^{(T\delta)})$.
Then this solution converges to a (set of) human-optimal state(s).

###### Proof.

By the monotone convergence theorem, utility through the optimization process
converges to a maximum value. Any state $s^{*}$ with this value must have the
sensitivity of all attributes equal and $C(s^{*})=0$, otherwise a proxy with
two attributes of unequal sensitivity or two attributes respectively will
cause increase in utility above $U(s^{*})$ in a finite amount of time.

Suppose $\frac{\partial U}{\partial s_{i}}(\frac{\partial C}{\partial
s_{i}})^{-1}(s^{*})=\alpha$ for all attributes $i$. We now proceed to show
that $U(s^{*})=\max\limits_{s\in\mathcal{S}}U(s)$. Consider any other feasible
state $s^{\prime}$. By convexity, it must be that

$\displaystyle 0$ $\displaystyle\geq C(s^{\prime})-C(s^{*})$
$\displaystyle\geq\sum_{i}(s^{\prime}_{i}-s^{*}_{i})\frac{\partial U}{\partial
s_{i}}(s^{*})$

Now, considering the difference in utility between states $s^{\prime}$ and
$s^{*}$.

$\displaystyle U(s^{\prime})-U(s^{*})$
$\displaystyle\leq\sum_{i}(s^{\prime}_{i}-s^{*}_{i})\frac{\partial C}{\partial
s_{i}}(s)$ $\displaystyle=\sum_{i}(s^{\prime}_{i}-s^{*}_{i})\frac{\partial
U}{\partial s_{i}}(\frac{\partial C}{\partial s_{i}})^{-1}\frac{\partial
C}{\partial s_{i}}(s^{*})$
$\displaystyle=\alpha\sum_{i}(s^{\prime}_{i}-s^{*}_{i})\frac{\partial
C}{\partial s_{i}}(s^{*})$ $\displaystyle\leq 0$

Thus, $s^{\prime}$ must have lower utility than $s^{*}$. Since this applies
for all feasible states, $U(s^{*})=\max\limits_{s\in\mathcal{S}}U(s)$

∎

While this is optimal, we require the assumptions of both the impact-
minimization robot and the efficient, interactive robot, each of which
individually presents complex challenges in implementation.

##  5 Conclusion and Further Work

In this paper, we present a novel model of value (mis)alignment between a
human principal and an AI agent. Fundamental to this model is that the human
provides an incomplete proxy of their own utility function for the AI agent to
optimize. Within this framework, we derive necessary and sufficient
theoretical conditions for value misalignment to be arbitrarily costly. Our
results dovetail with an emerging literature that connects harms from AI
systems to shallow measurement of complex values [32, 20, 5]. Taken together,
we view this as strong evidence that the ability of AI systems to be useful
and beneficial is highly dependent on our ability to manage the fundamental
gap between our qualitative goals and their representation in digital systems.

Additionally, we show that abstract representations of techniques currently
being explored by other researchers can yield solutions with guaranteed
utility improvement. Specifically, impact minimization and human
interactivity, if implemented correctly, allow AI agents to provide positive
utility in theory. In future work, we hope to generalize the model to account
for fundamental preference uncertainty (e.g., as in [12]), limits on human
rationality, and the aggregation of multiple human preferences. We are
optimistic that research into the properties and limitations of the
communication channel within this framework will yield fruitful insights in
value alignment, specifically, and AI, generally.

## Broader Impact

As AI systems become more capable in today’s society, the consequences of
misspecified reward functions increase as well. Instances where the goal of
the AI system and the preferences of individuals diverge are starting to
emerge in the real world. For example, content recommendation algorithms
optimizing for clicks causes clickbait and misinformation to proliferate [33].
Our work rigorously defines this general problem and suggests two separate
approaches for dealing with incomplete or misspecified reward functions. In
particular, we argue that, in the absence of a full description of attributes,
the incentives for real-world systems need to be plastic and dynamically
adjust based on changes in behavior of the agent and the state of the world.

## Acknowledgments and Disclosure of Funding

This work was partially supported by AFOSR, ONR YIP, NSF CAREER, NSF NRI, and
OpenPhil. We thank Anca Dragan, Stuart Russell, and the members of the
InterACT lab and CHAI for helpful advice, guidance, and discussions about this
project.

## References

  * [1] Abbeel, P., and Ng, A. Y. Apprenticeship learning via inverse reinforcement learning.  In Proceedings of the Twenty-First International Conference on Machine Learning (2004), p. 1.
  * [2] Amershi, S., Cakmak, M., Knox, W. B., and Kulesza, T. Power to the people: The role of humans in interactive machine learning.  AI Magazine 35, 4 (2014), 105–120.
  * [3] Amodei, D., and Clark, J. Faulty reward functions in the wild, 2016.  URL https://blog. openai. com/faulty-reward-functions (2016).
  * [4] Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., and Mané, D. Concrete problems in AI safety.  arXiv preprint arXiv:1606.06565 (2016).
  * [5] Andrus, M., and Gilbert, T. K. Towards a just theory of measurement: A principled social measurement assurance program for machine learning.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (2019), pp. 445–451.
  * [6] Armstrong, S., and Levinstein, B. Low impact artificial intelligence.  arXiv preprint arXiv:1705.10720 (2017).
  * [7] Bajcsy, A., Losey, D. P., O’Malley, M. K., and Dragan, A. D. Learning robot objectives from physical human interaction.  Proceedings of Machine Learning Research 78 (2017), 217–226.
  * [8] Bostrom, N. Superintelligence: Paths, dangers, strategies.  Oxford, 2014.
  * [9] Boutilier, C. A POMDP formulation of preference elicitation problems.  In AAAI/IAAI (2002), Edmonton, AB, pp. 239–246.
  * [10] Bradley, R. A., and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons.  Biometrika 39, 3/4 (1952), 324–345.
  * [11] Braziunas, D., and Boutilier, C. Preference elicitation and generalized additive utility.  In AAAI (2006), vol. 21.
  * [12] Chan, L., Hadfield-Menell, D., Srinivasa, S., and Dragan, A. D. The assistive multi-armed bandit.  In 2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI) (2019), IEEE, pp. 354–363.
  * [13] Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences, 2017.
  * [14] Eckersley, P. Impossibility and uncertainty theorems in AI value alignment (or why your agi should not have a utility function).  arXiv preprint arXiv:1901.00064 (2018).
  * [15] Griffith, S., Subramanian, K., Scholz, J., Isbell, C. L., and Thomaz, A. L. Policy shaping: Integrating human feedback with reinforcement learning.  In Advances in Neural Information Processing Systems (2013), pp. 2625–2633.
  * [16] Hadfield-Menell, D., Dragan, A., Abbeel, P., and Russell, S. The off-switch game.  In Workshops at the Thirty-First AAAI Conference on Artificial Intelligence (2017).
  * [17] Hadfield-Menell, D., and Hadfield, G. K. Incomplete contracting and AI alignment.  In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (2019), pp. 417–422.
  * [18] Hadfield-Menell, D., Milli, S., Abbeel, P., Russell, S. J., and Dragan, A. Inverse reward design.  In Advances in Neural Information Processing Systems (2017), pp. 6765–6774.
  * [19] Hadfield-Menell, D., Russell, S. J., Abbeel, P., and Dragan, A. Cooperative inverse reinforcement learning.  In Advances in Neural Information Processing Systems (2016), pp. 3909–3917.
  * [20] Jacobs, A. Z., and Wallach, H. Measurement and fairness.  arXiv preprint arXiv:1912.05511 (2019).
  * [21] Kerr, S. On the folly of rewarding A, while hoping for B.  Academy of Management Journal 18, 4 (1975), 769–783.
  * [22] Klein, B., Crawford, R. G., and Alchian, A. A. Vertical integration, appropriable rents, and the competitive contracting process.  The journal of Law and Economics 21, 2 (1978), 297–326.
  * [23] Krakovna, V., Orseau, L., Kumar, R., Martic, M., and Legg, S. Penalizing side effects using stepwise relative reachability.  arXiv preprint arXiv:1806.01186 (2018).
  * [24] Leike, J., Martic, M., Krakovna, V., Ortega, P. A., Everitt, T., Lefrancq, A., Orseau, L., and Legg, S. AI safety gridworlds.  arXiv preprint arXiv:1711.09883 (2017).
  * [25] Manheim, D., and Garrabrant, S. Categorizing variants of Goodhart’s law.  arXiv preprint arXiv:1803.04585 (2018).
  * [26] Ng, A. Y., Russell, S. J., et al. Algorithms for inverse reinforcement learning.  In Icml (2000), vol. 1, pp. 663–670.
  * [27] Omohundro, S. M. The basic AI drives.  In AGI (2008), vol. 171, pp. 483–492.
  * [28] Russell, S. J. Human Compatible: Artificial Intelligence and the Problem of Control.  Viking, 2019.
  * [29] Russell, S. J., and Norvig, P. Artificial Intelligence: A Modern Approach.  Pearson Education Limited,, 2010.
  * [30] Shavell, S. Damage measures for breach of contract.  The Bell Journal of Economics (1980), 466–490.
  * [31] Stray, J., Adler, S., and Hadfield-Menell, D. What are you optimizing for? aligning recommender systems with human values.  arXiv preprint arXiv:2002.08512 (2020).
  * [32] Thomas, R., and Uminsky, D. The problem with metrics is a fundamental problem for AI.  arXiv preprint arXiv:2002.08512 (2020).
  * [33] Tufekci, Z. Youtube’s recommendation algorithm has a dark side.  Scientific America (2019).
  * [34] Wagner, K. Inside Twitter’s ambitious plan to change the way we tweet.  Vox (2019).
  * [35] Williamson, O. E. Markets and hierarchies.  New York 2630 (1975).
  * [36] Ziebart, B. D., Maas, A. L., Bagnell, J. A., and Dey, A. K. Maximum entropy inverse reinforcement learning.  In AAAI (2008), vol. 8, Chicago, IL, USA, pp. 1433–1438.