# Explore and Control with Adversarial
Surprise

Arnaud Fickinger 1, Natasha Jaques∗12, Samyak Parajuli1, Michael Chang1,
Nicholas Rhinehart1, Glen Berseth1, Stuart Russell1, Sergey Levine12
1UC Berkeley
2Google Research, Brain Team
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Equal contribution.

###### Abstract

Unsupervised reinforcement learning (RL) studies how to leverage environment
statistics to learn useful behaviors without the cost of reward engineering.
However, a central challenge in unsupervised RL is to extract behaviors that
meaningfully affect the world and cover the range of possible outcomes,
without getting distracted by inherently unpredictable, uncontrollable, and
stochastic elements in the environment. To this end, we propose an
unsupervised RL method designed for high-dimensional, stochastic environments
based on an adversarial game between two policies (which we call Explore and
Control) controlling a single body and competing over the amount of
observation entropy the agent experiences. The Explore agent seeks out states
that maximally surprise the Control agent, which in turn aims to minimize
surprise, and thereby manipulate the environment to return to familiar and
predictable states. The competition between these two policies drives them to
seek out increasingly surprising parts of the environment while learning to
gain mastery over them. We show formally that the resulting algorithm
maximizes coverage of the underlying state in block MDPs with stochastic
observations, providing theoretical backing to our hypothesis that this
procedure avoids uncontrollable and stochastic distractions. Our experiments
further demonstrate that Adversarial Surprise leads to the emergence of
complex and meaningful skills, and outperforms state-of-the-art unsupervised
reinforcement learning methods in terms of both exploration and zero-shot
transfer to downstream tasks.

##  1 Introduction

Reinforcement learning methods have attained impressive results across a
number of domains (e.g., Berner et al. (2019); Kober et al. (2013); Levine et
al. (2016); Vinyals et al. (2019)). However, current RL methods typically
require a large number of samples for each new task (Dann et al., 2018). In
other areas of machine learning, an effective way to mitigate high data
requirements has been the use of unsupervised or self-supervised learning
(Sutskever et al., 2014; Radford et al., 2019). Similarly, humans and animals
seem to be able to learn rich priors from their own experience without being
told what to do, and children engage in structured but unsupervised play in
part as a way to acquire a functional understanding of the world (Smith &
Gasser, 2005). Based on this intuition, unsupervised RL methods rely on
intrinsic motivation (IM): task-agnostic objectives that incentivize the agent
to autonomously explore the world and learn behaviors that can be used to
solve a range of downstream tasks with little supervision. A general strategy
is to exactly or approximately express this task-agnostic objective in the
form of a reward function that uses only environment statistics, and then
optimize it using standard RL algorithms.

A reasonable goal for a good unsupervised learning algorithm is to fully
explore the state space of the environment, since this ensures the agent will
have the experience which with to learn the optimal policy for a downstream
task. Therefore, past work on IM has frequently focused on novelty-seeking
agents that maximize surprise or prediction error (Achiam & Sastry, 2017;
Schmidhuber, 1991; Yamamoto & Ishikawa, 2010; Pathak et al., 2017; Burda et
al., 2018). However, these methods are vulnerable to becoming distracted by
inherently stochastic elements of the environment, such as a “noisy TV”
(Schmidhuber, 2010). In contrast, active inference researchers inspired by
biological agents have focused on developing agents that seek to control their
environment and minimize surprise (Friston, 2009; Friston et al., 2009; 2016;
Berseth et al., 2021). These methods suffer from the opposite issue, the “dark
room problem”, in which a surprise-minimizing agent in a low-entropy
environment does not need to learn any behaviors at all in order to satisfy
its objective (Friston et al., 2012). Yet humans seem to maintain a balance
between optimizing for both novelty and familiarity. For example, a child in a
play room does not just try to toss their toys on the floor in every possible
pattern, or immediately put them away in the toy box, but instead tries to
stack them together, find new uses for parts, or combine them in various
structured ways.

We argue that an effective unsupervised RL method should find the right
balance between exploration and control. With this goal in mind, we introduce
a new algorithm based on an adversarial game between two policies, which take
turns sequentially acting for the same RL agent. The goal of the Control
policy is to minimize surprise, by learning to manipulate its environment in
order to return to safe and predictable states. In turn, the Explore policy is
novelty-seeking, and attempts to maximize surprise for the Control policy,
putting it into a diverse range of novel states. When combined, the two
adversaries engage in an arms race, repeatedly putting the agent into
challenging new situations, then attempting to gain control of those
situations. Figure 8 shows an illustration of the method, including a sample
interaction. Rather than simply adding noise to the environment, the Explore
policy learns to adapt to the Control policy, and to search for increasingly
challenging situations from which the Control policy must recover. Thus our
method, Adversarial Surprise (AS), leverages the power of multi-agent training
to generate a curriculum of increasingly challenging exploration and control
problems, leading to the emergence of complex, meaningful behaviours.

The contributions of this paper are: i) The Adversarial Surprise algorithm;
ii) Theoretical results which prove that when the environment is formulated as
a stochastic Block MDP (Du et al., 2019), traditional surprise-maximizing
methods will fail to fully explore the underlying state space, but Adversarial
Surprise will succeed; iii) Empirical evidence which supports the theoretical
results, and shows that AS fully explores the state space of both Block MDPs
and traditional benchmarking environments like VizDoom; iv) Experiments that
compare AS to state-of-the-art unsupervised RL baselines Random Network
Distillation (RND) (Burda et al., 2018), Asymmetric Self-Play (ASP)
(Sukhbaatar et al., 2017), Adversarially Guided Actor-Critic (AGAC) (Flet-
Berliac et al., 2021), and Surprise Minimizing RL (SMiRL) (Berseth et al.,
2021), and show that AS is able to explore more effectively, learn more
meaningful behaviors, and achieve higher task reward when transferred zero-
shot to common benchmarking environments Atari and VizDoom. Videos of our
agents are available on the project website:
https://sites.google.com/corp/view/adversarialsurprise/home, and show that AS
is able to learn interesting, emergent behaviors in Atari, VizDoom, and
MiniGrid, even without ever having trained with the game reward.

Figure 1: Adversarial Surprise is a multi-agent competition in which two
policies take turns controlling a single agent. The Explore policy acts first,
and tries to put the agent into surprising, high entropy states. On its turn,
the Control policy tries to minimize surprise by finding familiar, low-
entropy, predictable states. As training continues, the competition drives the
agents to learn increasingly complex behaviors. In the above example, the
Control policy eventually learns to pick up a key and lock the door to prevent
the Explore agent from taking it into the room with randomly moving objects (a
noisy TV state).

##  2 Related Work

Novelty-seeking and exploration methods lead the agent to increase coverage of
the environment. A simple way to implement novelty-seeking is to maximize the
prediction error of a world model (Achiam & Sastry, 2017; Schmidhuber, 1991;
Yamamoto & Ishikawa, 2010; Pathak et al., 2017; Burda et al., 2018; Raileanu &
Rocktäschel, 2020; Zhang et al., 2020b). Random Network Distillation (RND)
(Burda et al., 2018) is a highly effective example of one such method.
However, a major problem of prediction-error-based methods is that the
intrinsic reward is not useful when the environment contains aleatoric
uncertainty, i.e. inherently stochastic elements. This problem is often
referred to as the noisy TV problem, after Schmidhuber (2010) used the example
of an agent becoming stuck staring at static on a TV screen. We show in this
work that RND is indeed vulnerable to this problem, performing poorly in
stochastic environments.

Several works have proposed solutions to deal with aleatoric uncertainty. For
example, some approximate information gain on the agent’s dynamics model of
the environment (Houthooft et al., 2016; Still & Precup, 2012; Bellemare et
al., 2016; Pathak et al., 2017; Schmidhuber, 1991), using variational Bayes or
ensemble disagreement (Shyam et al., 2019; Pathak et al., 2019; Houthooft et
al., 2016). However, implementing such Bayesian procedures is difficult,
because it requires scalable and effective modeling of epistemic uncertainty,
which itself is a major open problem with high-dimensional models such as
neural networks (Bhattacharya & Maiti, 2020). Another method based on
maximizing information gain between the agent’s actions and future state is
known as empowerment (Klyubin et al., 2005; Salge et al., 2014; Eysenbach et
al., 2018; Sharma et al., 2019), but can also be difficult to approximate for
high-dimensional states (Karl et al., 2015; Zhao et al., 2020; de Abril &
Kanai, 2018; Zhang et al., 2020a; Mohamed & Rezende, 2015; Gregor et al.,
2016; Hansen et al., 2019). Instead of attempting to directly approximate
information gain, our approach maximizes state coverage via an adversarial
competition over observation entropy. As we will show, this recovers an
effective coverage strategy even in the presence of rich, stochastic
observations, and performs well in practice.

The asymptotic policy learned by standard novelty-seeking methods is not
exploratory. Recent work tries to learn a policy that approximately maximizes
the state marginal entropy at convergence (Hazan et al., 2019; Lee et al.,
2019). The state marginal entropy is hard to compute in general, and recent
work has proposed various approximations (Seo et al., 2021; Liu & Abbeel,
2021; Mutti et al., 2021). We prove that even with only noisy observations of
the underlying state, our method asymptotically maximizes the state marginal
entropy at convergence under some assumptions.

Surprise minimization and active inference: The design of the Control agent in
our method draws on ideas from surprise minimization and active inference
(Friston, 2009; Friston et al., 2009; 2016). The free energy principle,
originating in the neuroscience community, argues that complex niche-seeking
behaviors of biological systems are the result of minimizing long-term average
surprise on system sensors, leading agents to stay in safe and stable states
(Friston, 2009). SMiRL is an unsupervised RL method that leverages surprise
minimization as a means to discover skills that stabilize an otherwise chaotic
environment (Berseth et al., 2021). However, such approaches require strong
assumptions on the stochasticity of the environment. In low-entropy
environments, surprise minimization will not lead to learning interesting
behavior due to the dark room problem, in which the agent is not incentivized
to explore the environment to find a better niche (Friston et al., 2012). Our
method does not require that the environment is stochastic, since the Explore
agent itself drives the Control agent into situations from which surprise
minimization is challenging.

Multi-Agent competition has been shown to drive the emergence of complex
behavior (Baker et al., 2019; Dennis et al., 2020; Xu et al., 2020; Leibo et
al., 2019; Schmidhuber, 1997; Campero et al., 2020). Asymmetric Self-Play
(ASP) aims to learn increasingly complex skills via a competition between two
policies, where one policy (Bob) tries to imitate or reverse the trajectory of
the other policy (Alice) (Sukhbaatar et al., 2017; OpenAI et al., 2021). Our
empirical results compare to ASP, and demonstrate that it can fail in
stochastic environments, because Alice can easily produce random trajectories
which are very difficult for Bob to imitate. Similar to ASP, Adversarially-
Guided Actor Critic (AGAC) (Flet-Berliac et al., 2021) introduces an adversary
agent which attempts to mimic the action distribution of the actor, while the
actor tries to differentiate itself from the adversary. Like these methods,
Adversarial Surprise uses a two-player game to induce exploration and emergent
complexity. However, our objective is general and information-theoretic,
focusing on minimization or maximization of surprise rather than reaching
specific states. Unlike these methods, we provide theoretical results showing
AS provides a principled approach to maximizing state coverage in stochastic
environments. Finally, our method is reminiscent of recent work that learns
separate exploration and exploitation policies but without competition between
them (Badia et al., 2020b; a; Campos et al., 2021)

##  3 Background

Markov Decision Process (MDP): An MDP is a tuple $(\mathcal{S,A,T},r,\gamma)$,
where $s\in\mathcal{S}$ are states, $a\in\mathcal{A}$ are actions, $r(a,s)$ is
the reward function, and $\gamma\in[0,1)$ is a discount factor. At each
timestep $t$, an RL agent selects an action $a_{t}$ according to its policy
$\pi(a_{t}|s_{t})$, receives reward $r(a_{t},s_{t})$, and the environment
transitions to the next state according to $\mathcal{T}(s_{t+1}|s_{t},a_{t})$.
The agent uses RL to maximize its cumulative discounted reward over the
episode: $R=\sum_{t=0}^{T}\gamma^{t}r^{i}(a_{t},s_{t})$.

Block MDP (BMDP): A BMDP (Du et al., 2019) extends the MDP formalism to the
case where the agent cannot observe the true state $s$, but rather observes
rich observations $o\sim p(O|s)$. Unlike a traditional partially-observed MDP
(POMDP), a BMDP has an additional disjointness assumption: for any
$s,s^{\prime}\in S$, $s\neq
s^{\prime}\Rightarrow\text{supp}(p(O|s))\cap\text{supp}(p(O|s))=\emptyset$.
The same assumption has been used in several prior works (Azizzadenesheli et
al., 2016; Dann et al., 2018; Krishnamurthy et al., 2016; Misra et al., 2020),
and enables the BMDP to naturally capture a wide range of environments which
have rich, high-dimensional observations. We are interested in stochastic
BMDPs, in which the observation distribution is inherently entropic for some
states, i.e. $\exists s:H(p(O|s))>0$, where $H$ is the entropy. This is a
conceptual model of stochastic environments with uncontrollable factors.

Maximizing state marginal entropy: The state marginal distribution
$d^{\pi}(s)$ of a policy $\pi$ is the probability that the policy visits state
$s$:
$d^{\pi}(s)=\mathbb{E}_{a\sim\pi,s\sim\mathcal{T}}[\frac{1}{T}\sum_{t=0}^{T}\gamma^{t}\mathbbm{1}(s_{t}=s)]$
(Lee et al., 2019). The observation marginal density is defined analogously:

$\displaystyle d^{\pi}(o)=\mathbb{E}_{a\sim\pi,s\sim\mathcal{T},o\sim
p(\mathcal{O}|s)}\left[\frac{1}{T}\sum_{t=0}^{T}\gamma^{t}\mathbbm{1}(o_{t}=o)\right]$
(1)

We are interested in agents that are able to fully explore the underlying
state space of a stochastic BMDP; that is, agents that can maximize the
entropy of the state marginal distribution: $H(d^{\pi}(s))$. However, in the
context of a BMDP, agents cannot directly observe the underlying state $s$ and
must attempt to maximize $H(d^{\pi}(s))$ purely through their ability to alter
$d^{\pi}(o)$.

##  4 Adversarial Surprise

The design of our method is guided by the following desiderata: i) It should
be able to learn meaningful behaviors even if the environment has high-
dimensional, stochastic observations, and avoid the noisy TV problem. Thus,
our method should favour low-entropy states when possible, while still seeking
novelty. ii) It should enable the emergence of skills. Since multi-agent
training can lead to the emergence of complex skills, we create a multi-agent
adversarial game between two policies. iii) The optimized objective should be
theoretically sound, in the sense that we can provably show that it optimizes
a well-defined exploration metric under some assumptions. In our case, we will
show that our two-player game maximizes coverage of the underlying state space
of a stochastic BMDP.

To achieve these goals, we design an adversarial game that pits two policies
against each other in a competition over the amount of surprise encountered
through exploration. Specifically, we learn an Explore policy, $\pi^{E}$, and
a Control policy, $\pi^{C}$. The goal of the Control policy is to minimize its
own surprise, or observation entropy, using a learned density model
$p_{\theta}(o)$. The Explore policy’s goal is to maximize the surprise that
the Control policy experiences. The policies take turns taking actions for the
agent, switching back and forth throughout the episode. To simplify notation,
assume the policy taking actions switches every $k$ steps, such that for the
first $k$ steps, $a_{t}\sim\pi^{E}(a_{t}|o_{t})$ (the Explore policy acts),
then for timesteps $k-2k$, $a_{t}\sim\pi^{C}(a_{t}|o_{t})$ (the Control policy
acts). The policies continue switching until the end of the episode. Each
policy is given $k$ steps to act to enable it to reach states that will be
challenging for the other policy to recover from, thus facilitating learning
more complex and long-term exploration and control behaviors (see Figure 8).

Surprise is computed using a learned density model that estimates the agent’s
likelihood of experiencing observation $o$, $p_{\theta}(o)$. Following Berseth
et al. (2021), $p_{\theta}(o)$ is re-initialized each episode and trained
using maximum likelihood estimation (MLE) to fit the observations of the agent
within a single episode, which are stored in a buffer $\beta$. The density
model is either represented using a multivariate Gaussian distribution fit to
the pixels within $o$ (as in Berseth et al. (2021)), or using independent
categorical distributions for each pixel. For more details see Appendix C.

Because the Control policy is surprise-minimizing, its reward is
$r^{C}_{t}=\log p_{\theta}(o_{t})$, which resembles SMiRL (Berseth et al.,
2021), except using the observation in place of the state. The goal of the
Explore policy is to maximize the observation surprise  during the Control
agent’s turn. This creates an adversarial game, in which the Explore policy
attempts to find surprising situations with which to expose the Control
policy, and the Control policy’s objective is to recover from them. Therefore,
the Explore policy’s reward is based on the surprise for the observations of
the Control policy. Assume that the Control policy’s turn begins at timestep
$t^{C}$, and it receives a total reward of
$R^{i}=\sum_{t=t^{C}}^{t^{C}+k}\gamma^{k}\log p_{\theta}(o_{t})$ for that
turn. Then, the Explore policy’s reward is $-R^{i}$, and is applied to the
last timestep of the Explore policy’s turn (i.e. timestep $t^{C}-1$). Thus,
Adversarial Surprise defines the following adversarial game between the two
policies:

$\displaystyle\max_{\pi^{E}}\min_{\pi^{C}}-\mathbb{E}\left[\sum_{t=t^{C}}^{t^{C}+k}\log
p_{\theta}(o_{t})\right],$ (2)

where the Explore policy can only effect $p_{\theta}(o_{t})$ through the final
state that it produces at the end of its turn, which is the initial state for
the Control policy. Algorithm 1 (Appendix D) gives the full training
procedure.

We can show that optimizing these rewards leads the two players to compete
over the amount of observation entropy, where the Explore policy’s objective
is to maximize observation entropy:

$\displaystyle\mathcal{J}^{E}=-\mathbb{E}\left[\sum_{t=t^{C}}^{t^{C}+k}\log
p_{\theta}(o_{t})\right]\approx H(d^{\pi_{C}}(o)),$ (3)

And the Control policy’s goal is to minimize its observation entropy:
$-H(d^{\pi_{C}}(o))$. We prove the relationship between $\log p_{\theta}(o)$
and the observation entropy in Appendix A.

Implementation details: We parameterize the policies for the Explore and
Control policy using deep neural networks (NN) with parameters $\phi^{E}$ and
$\phi^{C}$, respectively. The policy is based on a convolutional NN, which
takes in the last 4 observation frames. The networks are trained using
Proximal Policy Optimization (PPO) (Schulman et al., 2017); further details
are given in Appendix C. We have found it helpful to only compute the surprise
reward using observations from the second half of the Control policy’s turn.
This gives the agent greater ability to take actions that may lead to initial
surprise, but reduce entropy over the long term. Finally, we allow the Explore
and Control policys to act for a different number of timesteps (that is, we
have a separate $k^{C}$ and $k^{E}$), enabling us to tune the emphasis on
exploration or control depending on the environment.

###  4.1 Adversarial Surprise maximizes state coverage in Block MDPs

We will show that AS covers the underlying state space of a Block MDP, under
some assumptions about the density of states with low observation entropy.
Full proofs are in Appendix A.

We are interested in maximizing the entropy of the marginal state distribution
$H(d^{\pi}(s))$, using a density model of the observation, $p_{\theta}(o)$. To
that end, we prove the following lemma in Appendix A:

######  Lemma 1.

The cumulative surprise measured by the observation density model
$p_{\theta}(o)$ forms an upper bound of the observation marginal entropy
$H(d^{\pi}(o))$, which becomes tight when the observation density model fits
the observation marginal $d^{\pi}(o)$:
$-\mathbb{E}_{\pi}\sum_{t=0}^{\infty}\log p_{\theta}(o_{t})\geq
H(d^{\pi}(o))$.

######  Lemma 2.

Given the disjointness assumption of the BMDP stated in Section 3, we show a
useful relation between observation marginal entropy and state marginal
entropy:

$\displaystyle
H(d^{\pi}(o))=\mathbb{E}_{d^{\pi}(s)}H(p(O|S=s))+H(d^{\pi}(s)).$ (4)

See Appendix A for the proof. Equation 4 shows that maximizing entropy in the
marginal observation distribution $d^{\pi}(o)$ amounts to maximizing two
terms: the emission entropy $H(p(O|S))$, and the state marginal entropy
$H(d^{\pi}(s))$.

Denote by $\mu^{*}_{RND}$ the state stationary distribution of an RND agent
trying to maximize the observation stationary distribution. It follows by
lemma 2 that $\mu^{*}_{RND}$ is the solution to the entropy-regularized LP

$\mu^{*}_{RND}=\operatorname*{arg\,max}_{\mu}\langle\mu,h\rangle+H(\mu)$

where $h$ is the vector indexed by $S$ giving the emission entropy of all
states. This program has a closed form solution which is given by
${\mu^{*}_{RND}}_{i}=\frac{e^{h_{i}}}{\sum_{j}e^{h_{j}}}.$ Therefore, the
distribution exponentially favors states with high emission entropy. This
explains why algorithms which maximize observation entropy fail to explore
environments with noisy TV states.

In Adversarial Surprise, the Controller policy is actively trying to transform
the observation distribution given by the Explorer policy into a low entropic
distribution after $T$ steps. Therefore the objective can be written as

$\min_{M}\max_{\mu}\langle M\mu,h\rangle+H(M\mu)$

where $M$ must lie in $\\{P^{T}_{\pi}:\pi\text{ is a stationary policy}\\}.$
Therefore the Explorer policy is now constrained to find a maximum in the
image of $M$. We provide now a sufficient condition on the structure of the
BMDP such that the Control policy can choose a matrix $M$ that induces the
Explore policy to maximize the state entropy by maximizing the observation
entropy after $T$ steps.

We first define a (semiquasi)metric on the latent state space:
$\tilde{d}(s,s^{\prime})=\min\\{k:\exists\pi,P^{\pi}_{k}(s^{\prime}|s)=1\\}$,
where by convention $P^{\pi}_{0}(s|s)=1$ for all $s$. In other words,
$\tilde{d}(s,s^{\prime})=k$ if there is a policy that reaches $s^{\prime}$
from $s$ in $k$ steps with probability 1. We symmetrize this metric by
defining the following semimetric:
$d(s,s^{\prime})=\max\\{\tilde{d}(s,s^{\prime}),\tilde{d}(s^{\prime},s)\\}$

######  Definition 1.

We now give a formal definition of “dark rooms”. We say that a state $s$ is a
dark room if it has minimal emission entropy: $H(p(O|S=s))=\min_{s\in
S}H(p(O|S=s))$.

######  Assumption 1.

We assume that for every state $s$, there is a dark room $s^{\prime}$ such
that $d(s,s^{\prime})=T$. That is, the set of dark rooms is a $T$-cover of the
state space with respect to $d$, and from every state there is a dark room can
be reached in exactly $T$ steps.

We can now state our main result:

######  Theorem 1.

Under the BMDP assumption and Assumption 1, the Markov chain induced by the
following AS game:

$\displaystyle\max_{d^{\pi_{E}}(s_{0})}\;\min_{d_{1:T}^{\pi_{C}}(s|s_{0})}H\left(d_{T}^{\pi_{C}}(o)\right)$

$T$-covers the latent state space, i.e., for all states $s$, there is a state
$s^{\prime}$ such that $d^{\pi}(s^{\prime})>0$ and $d(s,s^{\prime})\leq T$,
where $d^{\pi}$ is the state marginal distribution induced by the game between
the Explore ($\pi_{E}$) and Control ($\pi_{C}$) policies.

###### Proof.

(sketch.) Assumption 1 guarantees that for any states that the Explore policy
reaches, the Control policy can find a low-emission-entropy state within its
turn ($T$ timesteps), so that $H(p(O|s))$ is minimized. Thus, by Lemma 2, when
the Control policy is factored out, the objective of the Explore policy
becomes:

$\displaystyle\mathcal{J}^{E}=H(d^{\pi}(o))\approx H(d^{\pi}(s)).$ (5)

Therefore the Explore policy’s objective is to maximize coverage of the BMDP
state space. ∎

##  5 Experimental results

We now present experimental results111AS performance is measured across the
full episode, including both the Explore and Control turns. designed to assess
the following questions:

Q1. State coverage: How well does AS explore the underlying state space in a
stochastic BMDP, as compared to baselines? Given the theoretical results in
Section 4.1, we hypothesize that AS will fully explore the environment, but
other methods will become distracted by stochastic elements.

Q2. Zero-shot transfer: Does AS learn behaviors that are useful for downstream
tasks? To assess this, we train AS and baseline methods using only intrinsic
reward, then transfer the agents zero-shot to standard benchmark environments
in Atari (Bellemare et al., 2013) and VizDoom (Kempka et al., 2016b), and
measure the amount of game reward obtained. While there is no reason to expect
AS to always correlate with the objectives in arbitrary MDPs, we expect that
the twin goals of maximizing coverage while achieving high control should
correlate well with objectives in many reasonable MDPs. This is particularly
true of games, which have a notion of progress that roughly corresponds to
coverage, but at the same time have many dangerous states that could result in
‘death’, which leads to an unexpected jump back to the starting state. We
hypothesize that AS should, without even being aware of the task reward,
perform well in these environments. Comparing to prior methods in these
domains is interesting, because prior work has variously argued that both
novelty-seeking exploration methods (Burda et al., 2018) and surprise-
minimization methods (Berseth et al., 2021) should be expected to achieve high
scores in these games.

Q3. Emergence of complexity: If AS is able to achieve high zero-shot transfer
performance, we hypothesize that it will be because the adversarial game
drives the acquisition of increasingly complex observable behaviors.
Therefore, we track the emergence of specific skills throughout training, and
qualitatively examine final performance in benchmark environments to identify
meaningful skills.

Baselines: We compare AS to four competitive unsupervised RL baselines: i)
Asymmetric Self-Play (ASP) (Sukhbaatar et al., 2017), a state-of-the-art
multi-agent curriculum method; ii) Random Network Distillation (RND) (Burda et
al., 2018), a state-of-the-art exploration method; iii) Adversarially Guided
Actor-Critic (AGAC) (Flet-Berliac et al., 2021), a recently proposed
adversarial exploration method; and iv) Surprise Minimizing RL (SMiRL)
(Berseth et al., 2021), a recently proposed intrinsic motivation based on
active inference. All methods use the same PPO implementation, with
hyperparameters given in Appendix C.

Environments: To evaluate the above hypotheses, we use three types of
environments. To obtain environments that match the precise specifications of
the stochastic BMDP formalism, and present an exploration challenge, we
constructed a custom family of procedurally generated navigation tasks based
on MiniGrid (Chevalier-Boisvert et al., 2018). These environments contain
rooms that are either empty (dark), or contain stochastic elements such as
flashing lights that randomly change color. They also contain elements such as
doors that can be opened with keys, and switches that, when flipped, stop or
start the stochastic elements moving. As in MiniGrid, the agent only sees a
5x5 window of the true state. Each room of size 12 represents a hard
exploration task for the agent (see Figure 2(a)).While gridworlds allow us to
easily interpret the results and behaviors of the agent, and were used by ASP,
SMiRL, and AGAC, we also compare to prior work on two standardized benchmarks
with high-dimensional observations. Specifically, we use Atari ALE (Bellemare
et al., 2013), which was used by both SMiRL and RND to establish their
effectiveness, and the ViZDoom environment (Kempka et al., 2016b), which was
used by AGAC and SMiRL. Due to limited computational resources, we do not
conduct experiments in all possible Atari games (which is consistent with
prior work (Berseth et al., 2021; Burda et al., 2018)), but we do not cherry-
pick results; we show results for each of the games that we test.

(a) Environment

(b) Cumulative exploration

(c) Exploration within an episode

Figure 2: State coverage in stochastic BMDPs: the number of rooms (out of 4)
explored both (b) cumulatively over training, and (c) within an episode. AS
explores better than the state-of-the-art exploration methods RND and ASP,
which become distracted by noisy elements. AS outperforms SMiRL, which
minimizes observation surprise by turning to face the wall, or remains in dark
rooms.

Q1: State coverage: We measure state coverage in the procedurally-generated
BMDPs using the number of rooms the agent visits (since many different
observations can be generated from a single room). The results for all
algorithms are shown in Figure 2. We measure coverage cumulatively, over the
course of training (Figure 2(b)), to assess whether each method will lead the
agent to collect experience from all possible states. This measure is relevant
to whether the technique can be used as an effective exploration bonus to aid
learning a downstream task. We also measure the number of rooms explored
within each episode (Figure 2(c)). This allows us to assess whether the
asymptotic policy learned by the algorithms continues to explore once it has
converged.

(a) Doom environment

(b) AS (ours)

(c) RND

(d) SMiRL

Figure 3: State coverage in Doom: the heatmaps show $\log p(s)$ over the first
1000 training steps, where $s$ is the agent’s $x$-axis position in Doom. Black
areas indicate $p(s)=0$, meaning the agent has never visited these
coordinates. These results show that AS fully explores the state space, while
RND fails to explore both edges and SMiRL fails to explore the right edge.

As predicted by our theoretical analysis, we see that AS learns to more fully
explore the environments than prior work, visiting all possible rooms over a
lifetime (Figure 2(b)) and significantly more rooms per episode (Figure 2(c)).
It learns more quickly and explores more thoroughly than RND, which becomes
distracted by stochastic elements that lead to high prediction error.
Stochasticity also hinders learning for ASP, since Alice can easily produce
random goals that are difficult for Bob to replicate. Finally, we see that
SMiRL, which is designed for fully observed environments, does not explore
effectively because it suffers from the dark room problem – it prefers to stay
within the empty rooms, and not venture into rooms with high-entropy,
stochastic elements.

Figure 3 shows coverage of the latent state space in the VizDoom Take Cover
environment as positions along the x-axis. In this environment, agents
navigate horizontally while seeing a partial view of the underlying state.
Figure 3 reveals that only AS covers all the positions in the map in the first
1000 train steps, whereas RND fails to explore the edges, and SMiRL fails to
explore the right edge.

(a) Assault

(b) Berzerk

(c) Freeway

(d) Space Invaders

Figure 4: Zero-shot transfer in Atari: Each method is trained in Atari using
only intrinsic reward, then transferred zero-shot to optimizing game reward.
Plots show game reward, where error bars are the 95% Confidence Interval (CI)
of 5 seeds. The games reward behavior such as staying alive and shooting
enemies, so obtaining higher reward indicates the agent has learned meaningful
behaviors. Across 3/4 environments, AS outperforms RND, ASP, AGAC, and SMiRL,
showing AS provides a general way to learn useful behaviors in the absence of
external reward.

Q2. Zero-shot transfer: Figures 4 and 5 show how each method performs when
transferred zero-shot to the task of optimizing game reward in several Atari
environments and VizDoom. Because the

Figure 5: Zero-shot transfer in Doom. Consistent with the Atari results, AS
learns more meaningful behaviors (i.e. moving while avoiding enemy bullets)
than other techniques. This leads to higher environment reward.

game rewards complex behaviors like shooting or avoiding enemies, a high game
reward indicates the agent has learned interesting skills, purely from
optimizing the intrinsic objective. Across the environments, AS performs
better than RND, SMiRL, AGAC, and ASP. While RND is sometimes effective, its
performance often decreases over time due the bonus from the prediction error
shrinking as more states become familiar. Further, maximizing novelty in
environments like Freeway, Space Invaders, and Doom can lead to the agent
dying, corresponding to low reward. SMiRL performs best in Freeway, where
minimizing entropy corresponds closely to staying alive and not being hit by
cars. However, SMiRL performs poorly in the other environments because it
avoids entropy by hiding from enemies (staying in dark rooms when they are
available). ASP also performs poorly because it is possible for Alice to
quickly reach states which Bob cannot easily replicate, preventing the
algorithm from learning meaningful behaviors. For AGAC, the adversary cannot
accurately predict the actions of the agent given an observation when the
observation has a high entropy emission probability. In contrast, AS
consistently obtains high returns across all environments, indicating that
optimizing for both exploration and control provides a broadly useful
inductive bias for learning interesting behaviors in the absence of external
reward.

Q3. Emergence of complexity: Due to space constraints, we show results for the
emergence of a series of interesting behaviors in the BMDP environments in
Appendix B. Figure 7 reveals that the multi-agent game induced by AS leads to
a curriculum with multiple training phases: the Control agent first learns to
go to a dark room, the Explore agent learns to go back to a noisy TV, and the
Control agent responds by learning to lock a door to make it more difficult
for the Explore agent to expose it to surprising states. This highlights the
potential of Adversarial Surprise to learn long-term surprise-minimizing
behaviors. Figure 6 demonstrates that RND and ASP do not learn to try taking
even simple actions to control the environment, such as flipping a switch to
stop stochastic elements from moving. In AGAC, the actor chooses actions that
maximize the discrepancy between its policy and the Adversary’s. Although this
proves to be effective for exploration between subsequent trajectories, as
shown in Figure 5, the interaction between the Control and Explore policies in
Adversarial Surprise is able to facilitate more complex behaviors by not only
finding novel states to explore but determining when to exhibit control over
safe states. The emergence of interesting behaviour in both Atari and Vizdoom
is evidenced in the videos available at:
https://sites.google.com/corp/view/adversarialsurprise/home. Purely through
optimizing the Adversarial Surprise objective, AS agents learn complex
behaviors such as hiding from enemies behind walls or barriers, shooting
enemies in Space Invaders, Berzerk, Assault, and Doom, and hopping across the
road in Freeway.

##  6 Discussion

We proposed Adversarial Surprise as a general approach for unsupervised RL.
Adversarial Surprise corresponds to a two-player adversarial game, in which
two policies compete over the amount of surprise, or observation entropy, that
an agent experiences. Reminiscent of Dr. Jekyll and Mr. Hyde, the Explore
policy acts to expose the Control policy to highly entropic states from which
it must recover by learning to manipulate the environment. We show that AS
produces increasingly complex control and exploration strategies, and is able
to address exploration in stochastic Block MDPs. In such environments, prior
methods can become distracted by noisy elements, or suffer from the dark room
problem.We show both theoretically and empirically that AS is robust against
these issues, and learns to explore the environment more thoroughly, and
control it more effectively, than state-of-the-art prior works like RND, ASP,
AGAC, and SMiRL.

Future work: Our evaluation of AS focuses on coverage and unsupervised
exploration, where we demonstrate that AS improves over novelty-seeking,
surprise minimization, adversarial, and goal-setting methods in stochastic
BMDPs and standard benchmarks like Atari and Doom. However, the potential
value of unsupervised RL methods extends more broadly: such methods could be
used to acquire skills for downstream task learning, controlling an
environment to reach states from which more behaviors could be performed
successfully. Future work could study how AS and its extensions could enhance
applications like robotics, for example by collecting data for downstream
reward-guided learning. Further, we see a potentially exciting method which
combines AS with hierarchical RL, by training a meta-policy to select when to
invoke the Explore and Control sub-policies. In this way, the meta-policy
could explicitly decide when to explore and when to exploit.

## References

  * Achiam & Sastry (2017) Joshua Achiam and Shankar Sastry.  Surprise-based intrinsic motivation for deep reinforcement learning.  _arXiv preprint arXiv:1703.01732_ , 2017.
  * Azizzadenesheli et al. (2016) Kamyar Azizzadenesheli, Alessandro Lazaric, and Animashree Anandkumar.  Reinforcement learning of pomdps using spectral methods.  In _Conference on Learning Theory_ , pp. 193–256. PMLR, 2016.
  * Badia et al. (2020a) Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell.  Agent57: Outperforming the atari human benchmark.  In _International Conference on Machine Learning_ , pp. 507–517. PMLR, 2020a.
  * Badia et al. (2020b) Adrià Puigdomènech Badia, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Bilal Piot, Steven Kapturowski, Olivier Tieleman, Martín Arjovsky, Alexander Pritzel, Andew Bolt, et al.  Never give up: Learning directed exploration strategies.  _arXiv preprint arXiv:2002.06038_ , 2020b.
  * Baker et al. (2019) Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch.  Emergent tool use from multi-agent autocurricula.  _arXiv preprint arXiv:1909.07528_ , 2019.
  * Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, 2013.
  * Bellemare et al. (2016) Marc G Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.  Unifying count-based exploration and intrinsic motivation.  _arXiv preprint arXiv:1606.01868_ , 2016.
  * Berner et al. (2019) Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Berseth et al. (2021) Glen Berseth, Daniel Geng, Coline Manon Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine.  Smirl: Surprise minimizing reinforcement learning in unstable environments.  2021\.  URL https://openreview.net/forum?id=cPZOyoDloxl.
  * Bhattacharya & Maiti (2020) Shrijita Bhattacharya and Tapabrata Maiti.  Statistical foundation of variational bayes neural networks.  _arXiv preprint arXiv:2006.15786_ , 2020.
  * Burda et al. (2018) Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_ , 2018.
  * Campero et al. (2020) Andres Campero, Roberta Raileanu, Heinrich Küttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette.  Learning with amigo: Adversarially motivated intrinsic goals.  _arXiv preprint arXiv:2006.12122_ , 2020.
  * Campos et al. (2021) Víctor Campos, Pablo Sprechmann, Steven Stenberg Hansen, Andre Barreto, Steven Kapturowski, Alex Vitvitskyi, Adria Puigdomenech Badia, and Charles Blundell.  Beyond fine-tuning: Transferring behavior in reinforcement learning.  In _ICML 2021 Workshop on Unsupervised Reinforcement Learning_ , 2021\.
  * Chevalier-Boisvert et al. (2018) Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal.  Minimalistic gridworld environment for openai gym.  https://github.com/maximecb/gym-minigrid, 2018.
  * Dann et al. (2018) Christoph Dann, Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert E Schapire.  On oracle-efficient pac rl with rich observations.  _arXiv preprint arXiv:1803.00606_ , 2018.
  * de Abril & Kanai (2018) Ildefons Magrans de Abril and Ryota Kanai.  A unified strategy for implementing curiosity and empowerment driven reinforcement learning.  _arXiv preprint arXiv:1806.06505_ , 2018.
  * Dennis et al. (2020) Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine.  Emergent complexity and zero-shot transfer via unsupervised environment design.  _arXiv preprint arXiv:2012.02096_ , 2020.
  * Dhariwal et al. (2017) Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, Yuhuai Wu, and Peter Zhokhov.  Openai baselines.  https://github.com/openai/baselines, 2017.
  * Du et al. (2019) Simon Du, Akshay Krishnamurthy, Nan Jiang, Alekh Agarwal, Miroslav Dudik, and John Langford.  Provably efficient rl with rich observations via latent state decoding.  In _International Conference on Machine Learning_ , pp. 1665–1674. PMLR, 2019.
  * Eysenbach et al. (2018) Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Flet-Berliac et al. (2021) Yannis Flet-Berliac, Johan Ferret, Olivier Pietquin, Philippe Preux, and Matthieu Geist.  Adversarially guided actor-critic.  _arXiv preprint arXiv:2102.04376_ , 2021.
  * Friston (2009) Karl Friston.  The free-energy principle: a rough guide to the brain?  _Trends in cognitive sciences_ , 13(7):293–301, 2009.
  * Friston et al. (2012) Karl Friston, Christopher Thornton, and Andy Clark.  Free-energy minimization and the dark-room problem.  _Frontiers in psychology_ , 3:130, 2012.
  * Friston et al. (2016) Karl Friston, Thomas FitzGerald, Francesco Rigoli, Philipp Schwartenbeck, Giovanni Pezzulo, et al.  Active inference and learning.  _Neuroscience & Biobehavioral Reviews_, 68:862–879, 2016\.
  * Friston et al. (2009) Karl J Friston, Jean Daunizeau, and Stefan J Kiebel.  Reinforcement learning or active inference?  _PloS one_ , 4(7):e6421, 2009.
  * Gregor et al. (2016) Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra.  Variational intrinsic control.  _arXiv preprint arXiv:1611.07507_ , 2016.
  * Hansen et al. (2019) Steven Hansen, Will Dabney, Andre Barreto, Tom Van de Wiele, David Warde-Farley, and Volodymyr Mnih.  Fast task inference with variational intrinsic successor features.  _arXiv preprint arXiv:1906.05030_ , 2019.
  * Hazan et al. (2019) Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest.  Provably efficient maximum entropy exploration.  In _International Conference on Machine Learning_ , pp. 2681–2691. PMLR, 2019.
  * Houthooft et al. (2016) Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel.  Vime: Variational information maximizing exploration.  _arXiv preprint arXiv:1605.09674_ , 2016.
  * Karl et al. (2015) Maximilian Karl, Justin Bayer, and Patrick van der Smagt.  Efficient empowerment.  _arXiv preprint arXiv:1509.08455_ , 2015.
  * Kempka et al. (2016a) Michal Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaskowski.  Vizdoom: A doom-based AI research platform for visual reinforcement learning.  _CoRR_ , abs/1605.02097, 2016a.  URL http://arxiv.org/abs/1605.02097.
  * Kempka et al. (2016b) Michał Kempka, Marek Wydmuch, Grzegorz Runc, Jakub Toczek, and Wojciech Jaśkowski.  ViZDoom: A Doom-based AI research platform for visual reinforcement learning.  In _IEEE Conference on Computational Intelligence and Games_ , pp. 341–348, Santorini, Greece, Sep 2016b. IEEE.  URL http://arxiv.org/abs/1605.02097.  The best paper award.
  * Klyubin et al. (2005) Alexander S Klyubin, Daniel Polani, and Chrystopher L Nehaniv.  All else being equal be empowered.  In _European Conference on Artificial Life_ , pp. 744–753. Springer, 2005.
  * Kober et al. (2013) Jens Kober, J Andrew Bagnell, and Jan Peters.  Reinforcement learning in robotics: A survey.  _The International Journal of Robotics Research_ , 32(11):1238–1274, 2013.
  * Krishnamurthy et al. (2016) Akshay Krishnamurthy, Alekh Agarwal, and John Langford.  Pac reinforcement learning with rich observations.  _arXiv preprint arXiv:1602.02722_ , 2016.
  * Lee et al. (2019) Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdinov.  Efficient exploration via state marginal matching.  _arXiv preprint arXiv:1906.05274_ , 2019.
  * Leibo et al. (2019) Joel Z Leibo, Edward Hughes, Marc Lanctot, and Thore Graepel.  Autocurricula and the emergence of innovation from social interaction: A manifesto for multi-agent intelligence research.  _arXiv preprint arXiv:1903.00742_ , 2019.
  * Levine et al. (2016) Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel.  End-to-end training of deep visuomotor policies.  _The Journal of Machine Learning Research_ , 17(1):1334–1373, 2016.
  * Liu & Abbeel (2021) Hao Liu and Pieter Abbeel.  Behavior from the void: Unsupervised active pre-training.  _arXiv preprint arXiv:2103.04551_ , 2021.
  * Misra et al. (2020) Dipendra Misra, Mikael Henaff, Akshay Krishnamurthy, and John Langford.  Kinematic state abstraction and provably efficient rich-observation reinforcement learning.  In _International conference on machine learning_ , pp. 6961–6971. PMLR, 2020.
  * Mohamed & Rezende (2015) Shakir Mohamed and Danilo Jimenez Rezende.  Variational information maximisation for intrinsically motivated reinforcement learning.  _arXiv preprint arXiv:1509.08731_ , 2015.
  * Mutti et al. (2021) Mirco Mutti, Lorenzo Pratissoli, and Marcello Restelli.  Task-agnostic exploration via policy gradient of a non-parametric state entropy estimate.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, pp. 9028–9036, 2021.
  * OpenAI et al. (2021) OpenAI OpenAI, Matthias Plappert, Raul Sampedro, Tao Xu, Ilge Akkaya, Vineet Kosaraju, Peter Welinder, Ruben D’Sa, Arthur Petron, Henrique Ponde de Oliveira Pinto, et al.  Asymmetric self-play for automatic goal discovery in robotic manipulation.  _arXiv preprint arXiv:2101.04882_ , 2021.
  * Pathak et al. (2017) Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _International Conference on Machine Learning_ , pp. 2778–2787. PMLR, 2017.
  * Pathak et al. (2019) Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta.  Self-supervised exploration via disagreement.  In _International Conference on Machine Learning_ , pp. 5062–5071. PMLR, 2019.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raileanu & Rocktäschel (2020) Roberta Raileanu and Tim Rocktäschel.  Ride: Rewarding impact-driven exploration for procedurally-generated environments.  _arXiv preprint arXiv:2002.12292_ , 2020.
  * Salge et al. (2014) Christoph Salge, Cornelius Glackin, and Daniel Polani.  Empowerment–an introduction.  In _Guided Self-Organization: Inception_ , pp. 67–114. Springer, 2014.
  * Schmidhuber (1991) Jürgen Schmidhuber.  A possibility for implementing curiosity and boredom in model-building neural controllers.  In _Proc. of the international conference on simulation of adaptive behavior: From animals to animats_ , pp. 222–227, 1991.
  * Schmidhuber (1997) Jürgen Schmidhuber.  What’s interesting?  _IDSIA Technical Report_ , 1997.
  * Schmidhuber (2010) Jürgen Schmidhuber.  Formal theory of creativity, fun, and intrinsic motivation (1990–2010).  _IEEE Transactions on Autonomous Mental Development_ , 2(3):230–247, 2010.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Seo et al. (2021) Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee.  State entropy maximization with random encoders for efficient exploration.  _arXiv preprint arXiv:2102.09430_ , 2021.
  * Sharma et al. (2019) Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman.  Dynamics-aware unsupervised discovery of skills.  _arXiv preprint arXiv:1907.01657_ , 2019.
  * Shyam et al. (2019) Pranav Shyam, Wojciech Jaśkowski, and Faustino Gomez.  Model-based active exploration.  In _International Conference on Machine Learning_ , pp. 5779–5788. PMLR, 2019.
  * Smith & Gasser (2005) Linda Smith and Michael Gasser.  The development of embodied cognition: Six lessons from babies.  _Artificial life_ , 11(1-2):13–29, 2005.
  * Still & Precup (2012) Susanne Still and Doina Precup.  An information-theoretic approach to curiosity-driven reinforcement learning.  _Theory in Biosciences_ , 131(3):139–148, 2012\.
  * Sukhbaatar et al. (2017) Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, and Rob Fergus.  Intrinsic motivation and automatic curricula via asymmetric self-play.  _arXiv preprint arXiv:1703.05407_ , 2017.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  In _Advances in neural information processing systems_ , pp. 3104–3112, 2014.
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Xu et al. (2020) Kelvin Xu, Siddharth Verma, Chelsea Finn, and Sergey Levine.  Continual learning of control primitives: Skill discovery via reset-games.  _arXiv preprint arXiv:2011.05286_ , 2020.
  * Yamamoto & Ishikawa (2010) Naoyuki Yamamoto and Masumi Ishikawa.  Curiosity and boredom based on prediction error as novel internal rewards.  In _Brain-Inspired Information Technology_ , pp. 51–55. Springer, 2010.
  * Zhang et al. (2020a) Jin Zhang, Jianhao Wang, Hao Hu, Tong Chen, Yingfeng Chen, Changjie Fan, and Chongjie Zhang.  Metacure: Meta reinforcement learning with empowerment-driven exploration.  _arXiv preprint arXiv:2006.08170_ , 2020a.
  * Zhang et al. (2020b) Tianjun Zhang, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E Gonzalez, and Yuandong Tian.  Bebold: Exploration beyond the boundary of explored regions.  _arXiv preprint arXiv:2012.08621_ , 2020b.
  * Zhao et al. (2020) Ruihan Zhao, Pieter Abbeel, and Stas Tiomkin.  Efficient online estimation of empowerment for reinforcement learning.  _arXiv preprint arXiv:2007.07356_ , 2020.

##  Appendix A Proof details

Firstly, we notice that we have a simple relation between marginal observation
entropy and marginal state entropy by the structure of the POMDP:

$\displaystyle d^{\pi}(o)$
$\displaystyle=(1-\gamma)\sum_{t}\gamma^{t}p(o_{t}=o)$ (6)
$\displaystyle=(1-\gamma)\sum_{t}\gamma^{t}\sum_{s}p(s_{t}=s)p(o|s)$ (7)
$\displaystyle=(1-\gamma)\sum_{s}p(o|s)\sum_{t}\gamma^{t}p(s_{t}=s)$ (8)
$\displaystyle=\sum_{s}p(o|s)d^{\pi}(s)$ (9)

We can use this relation to prove the following lemma:

######  Lemma 3.

The cumulative surprise measured by the observation density model
$p_{\theta}(o)$ forms an upper bound of the observation marginal entropy
$H(d^{\pi}(o))$, which becomes tight when the observation density model fits
the observation marginal $d^{\pi}(o)$.

###### Proof.

$\displaystyle-\mathbb{E}_{\pi}\log p_{\theta}(o)$
$\displaystyle=-\sum_{s}d^{\pi}(s)\sum_{o}p(o|s)\log p_{\theta}(o)$ (10)
$\displaystyle=-\sum_{o}d^{\pi}(o)\log p_{\theta}(o)$ (11) $\displaystyle\geq
H(d^{\pi}(o))$ (12)

where the last inequality is because $-\sum_{o}d^{\pi}(o)\log
p_{\theta}(o)-H(d^{\pi}(o))=KL(d^{\pi}(o)||p_{\theta}(o))\geq 0$ ∎

We suppose that we are in the Block MDP setting:

######  Assumption 2 (Block MDP).

We suppose that for any $s,s^{\prime}\in S$, $s\neq s^{\prime}\Rightarrow
supp(p(O|s))\cap supp(p(O|s))=\emptyset$

In this case the marginal observation entropy can also be simply related to
the marginal state entropy:

######  Lemma 4.

In a block MDP (BMDP) Misra et al. (2020), by noticing that $H(S|O)=0$, we can
decompose the observation marginal entropy as follows:

$\displaystyle H(d^{\pi}(o))=\mathbb{E}_{d^{\pi}(s)}H(O|S=s)+H(d^{\pi}(s))$
(13)

###### Proof.

$\displaystyle H(d^{\pi}(o))$
$\displaystyle=\sum_{o,s}p(o|s)d^{\pi}(s)\log(\sum_{s}p(o|s)d^{\pi}(s))$ (14)
$\displaystyle=\sum_{s}d^{\pi}(s)\sum_{o\in\mathcal{O}_{s}}p(o|s)\log(p(o|s)d^{\pi}(s))$
(15) $\displaystyle=\sum_{s}d^{\pi}(s)[\sum_{o\in\mathcal{O}_{s}}p(o|s)\log
p(o|s)+\log d^{\pi}(s)]$ (16)
$\displaystyle=\mathbb{E}_{d^{\pi}(s)}H(O|S=s)+H(d^{\pi}(s))$ (17)

∎

that we can also obtain by simply noticing that $H(S|O)=0$ in the Block MDP
setting and writing the mutual information between $S$ and $O$.

Suppose that we have a small number of latent states with rich observations,
that is, there is $s$ such that $H(O|S=s)\gg\log|\mathcal{S}|$. In this case,
if we are trying to maximize the marginal observation entropy, we have:

$\displaystyle\max_{d^{\pi}(s)}H(d^{\pi}(o))\approx\max_{d^{\pi}(s)}\mathbb{E}_{d^{\pi}(s)}H(O|S=s)$
(18)

The RHS is maximized by taking:

$\displaystyle d^{\pi}(s)=I(s=\operatorname*{arg\,max}H(O|S=s))$ (19)

That is, we are stuck in a noisy TV.

On the contrary, if we are trying to minimize the marginal observation
entropy, equation Eq.17 gives us the exact minimizer:

$\displaystyle d^{\pi}(s)=I(s=\operatorname*{arg\,min}H(O|S=s))$ (20)

That is, we are stuck in a dark room.

Now suppose that we are optimizing the following objective:

$\displaystyle\max_{d^{\pi_{A}}(s_{0})}\min_{d_{1:T}^{\pi_{B}}(s|s_{0})}H(d_{1:T}^{\pi_{B}}(o))$
(21)

######  Definition 2.

We define the following semiquasimetric in the state space:

$\displaystyle\tilde{d}(s,s^{\prime})$
$\displaystyle=\min\\{k:\exists\pi,P^{\pi}_{k}(s^{\prime}|s)=1\\}$ (22)

where by convention $P^{\pi}_{0}(s|s)=1$ for all $s$. In other words,
$d(s,s^{\prime})=k$ if state there is a policy that reaches $s^{\prime}$ from
$s$ in $k$ steps with probability 1.

######  Definition 3.

We define the following semimetric:

$\displaystyle d(s,s^{\prime})$
$\displaystyle=\max\\{\tilde{d}(s,s^{\prime}),\tilde{d}(s^{\prime},s)\\}$ (23)

######  Definition 4.

We say that a state $s$ is a dark room if it has minimal emission entropy:

$\displaystyle H(O|S=s)=\min_{s\in S}H(O|S=s)$ (24)

######  Assumption 3.

We make three assumptions concerning the density of dark rooms:

  1. (a)

We suppose that for every state $s$, there is a dark room such that
$d(s,s^{\prime})\leq T$. That is, the set of dark rooms is a $T$-cover of the
state space with respect to $d$.

  2. (b)

We suppose that for every state $s$, there is a dark room such that
$P^{\pi}_{T}(s^{\prime}|s)=1$, that is a dark room can be reached in exactly
$T$ steps.

  3. (c)

We suppose that for any state $s$ and any dark room $s^{\prime}$, if
$\tilde{d}(s,s^{\prime})\leq T$, then $d(s,s^{\prime})\leq T$, that is if we
can reach a dark room from a state $s$ in less than $T$ steps, then we can
also reach $s$ from this dark room is less than $T$ steps.

######  Theorem 2.

Under Assumptions 2 and 3, the Markov chain induced by the following AS game:

$\displaystyle\max_{d^{\pi_{A}}(s_{0})}\min_{d_{1:T}^{\pi_{B}}(s|s_{0})}H(d_{T}^{\pi_{B}}(o))$
(25)

$T$-covers the state space, that is for every state $s$, there is a state
$s^{\prime}$ such that $d^{\pi}(s^{\prime})>0$ and $d(s,s^{\prime})\leq T$,
where $d^{\pi}$ is the marginal induced by the game between $A$ and $B$.

###### Proof.

Given an initial state $s_{0}$, Eq.17 shows that the controller will always
reach the same state of lowest emission entropy at step $T$. By assumption
3(b) the Controller can always reach a dark room with probability 1 in exactly
T steps. Therefore the game is equivalent to the following constrained
objective:

$\displaystyle\max_{d^{\pi}(s)}\mathbb{E}_{d^{\pi}(s)}H(O|S=s)+H(d^{\pi}(s))$
(26) $\displaystyle\text{s.t. }d^{\pi}(\\{s:H(O|S=s)=\min_{s}H(O|S=s)\\})=1$
(27)

For any state marginal satisfying the constraint we have:

$\displaystyle\mathbb{E}_{d^{\pi}(s)}H(O|S=s)=C$ (28)

where $C$ is a constant. Therefore, with probability 1, maximizing this
objective is equivalent to maximizing the state marginal entropy in the set of
dark rooms which form a $T$-cover of the state space by assumption 3(a).
Therefore the Markov chain induced by the game $T$-covers the state space.
Indeed, suppose by contrapositon that this is not the case. That is, there is
$s$ such that for any $s^{\prime}$ we have:

$\displaystyle d^{\pi}(s^{\prime})=0\vee d(s,s^{\prime})>T$ (29)

Since the set of dark rooms is a T-cover by assumption 3(a), we know that
there is a dark room $s^{\prime\prime}$ such that $d(s,s^{\prime\prime})\leq
T$, which implies that $d^{\pi}(s^{\prime\prime})=0$. Therefore the state
marginal entropy in the set of dark rooms is not maximized and the objective
is not optimized.

∎

##  Appendix B Additional experimental results

###  B.1 Emergence of Complexity

Figure 6: Q2. Emergence of Complexity: the average number of times the agent
flip a switch to stop lights from flashing. ASP and RND do not learn to press
the switch, while SMiRL and AS both press the switch a similar number of
times. Resetting the AS buffer more frequently enables it to exceed even SMiRL
in taking actions to control the environment.

To show that Adversarial Surprise leads to emergence of complexity by phases,
we plot the temporal acquisition of two behaviors in the MiniGrid environment.
The results are shown in Figure 7. This environment includes a dark room and a
noisy room separated by a door. The position of the door changes at each
episode. Initially, the agent is inside the noisy room and the door is open.
One episode consists of 96 steps: the Control policy takes control of the
agent during 32 steps, then the Explore policy takes control of the agent
during 32 steps, finally the Control policy takes control of the agent during
32 steps. The first acquired behavior by the Control policy is identifying
where the door is and going to the dark room during the first round. It is a
short-term surprise minimizing behavior and an agent trained with a SMIRL
objective can converge to it. However, the Explore policy learns to go back to
the noisy room and to reach the farthest point from the door such that the
Control policy does not have the time to reach a state of minimum entropy
before the surprise of the agent is computed in the reward. This in turn
incentivizes the acquisition of a more complex behavior by the Control policy:
it learns to go in the dark room and to lock the agent inside by closing the
door during the first round, making it harder for the Explore policy to learn
to reach a state that will surprise the agent during the Control policy’s
second round. This behavior reminiscent of Dr. Jekyll and Mr. Hyde highlights
the potential of Adversarial Surprise to learn long-term surprise-minimizing
behaviors.

We also investigate the behaviors learned in a similar environment, which
contains a switch that enables the agent to stop stochastic elements from
changing color. Figure 6 measures how often agents trained with different
techniques learn to press the switch. Since RND has no incentive to learn to
control the environment, it never learns to press the switch. A similar result
is observed for ASP, since reducing the entropy would make it easier for the
Bob agent to replicate the Alice agent’s final state. Thus, ASP will not
always lead the agent to learn all possible behaviors relevant to controlling
the environment. Both SMiRL and AS learn to take actions to reduce entropy.
However, when we train AS by resetting the buffer $\beta$ used to fit the
density model $p_{\theta}(o)$ after each round (that is, after both the
Explore and Control policy have taken one turn), rather than after each
episode, we see that AS increases the number of actions it takes to reduce
entropy even over SMiRL. This is likely because resetting the buffer removes
any incentive to return to states that the agent has previously seen within
its lifetime, and instead gives a stronger incentive to reduce entropy
immediately.

(a) The Control policy learns to lock the agent in the dark room to minimize
long-term surprise.

(b) Acquisition of two behaviors by phases in order of complexity and long-
term impact on the surprise.

Figure 7: Q3. Emergence of Complexity: a) An example environment which
contains a key that can be used to lock a door separating the agent from
stochastic elements. b) We observe two relatively short phase transitions
separating three learning phases with three clearly distinguishable behaviors:
randomly exploring, going to the dark room, locking the agent in the dark room
(left). This is evidence of an emergent curriculum induced by the multi-agent
competition.

###  B.2 Ablation Study on the Explorer and Controller Horizon

We perform an ablation study on the length of each round for the Explore and
Control policy in the MiniGrid environment and observe that the optimal
horizon is between 8 and 16.

Figure 8: Ablation Study on the Explorer and Controller Horizon.

##  Appendix C Hyperparameter details

In every experiment, both the explorer’s and controller’s policies receive a
stack of the 4 last observations, a sufficient statistic of the current
observation density model and the index of the current time step. The 4 last
observations are stacked with the sufficient statistic before being fed into a
convolutional layer. Then the output of the convolutional layer is stacked
with the index of the current time step before being fed into a feed forward
layer. At every step of the second-half of the controller’s trajectories, we
compute the error of the current observation with respect to the current
observation density model and then feed the adversarial surprise buffer with
the current observation to update the density model. The negative error is the
reward of the controller for the current time step. By default, buffer is
reset at the beginning of every episode. We also experiment with when to reset
the buffer $\beta$; we find that resetting the buffer after each round (after
the Explore policy and Control policy each take one turn) can sometimes
improve performance. In our round buffer variant, used in 6, the buffer is
reset at every round.

###  C.1 Minigrid

We use 3 convolutional layers with 16, 32 and 64 output channels and a stride
of 2 for every layers. For the MiniGrid experiments using $7\times 7$
observations, for our observation density model we use $7\times 7$ independent
categorical distributions with 12 classes each instead of independent Gaussian
distributions, which significantly improve the results. The sufficient
statistic is the set of $7\times 7\times 12$ probabilities. In one episode,
each agent acts during 2 rounds of 32 steps per round (Explorer-Controller-
Explorer-Controller), for a total of 128 steps. We use 16 parallel
environments for AS, RND, ASP and SMIRL. For the baselines, we base our RND
implementation on the github repo from jcwleo, which is a reliable PyTorch
implementation (https://github.com/jcwleo/random-network-distillation-
pytorch), our ASP implementation on the github repo from the author
(https://github.com/tesatory/hsp), our SMIRL implementation on the github repo
from the author (https://github.com/Neo-X/SMiRL_Code), and our AGAC
implementation is based on the Pytorch version of the github repo from the
author (https://github.com/yfletberliac/adversarially-guided-actor-
critic/tree/main/agac_torch.

###  C.2 Atari

We use the Atari pre-processing wrappers from Dhariwal et al. (2017) before
feeding input into a five-layer architecture with three convolutional layers
with 4, 32, and 64 output channels, kernel sizes of 8,4,3, and strides of 4,
2, and 1 for the explorer. We downsize the input images to $4\times 20\times
20$ and for our observation density model we use $4\times 20\times 20$
independent Gaussian distributions. The sufficient statistic is the set of
means and standard deviations of the $4\times 20\times 20$ Gaussian
distributions. For the Atari environments, we run the explorer for 64 steps
and controller for 128 steps and alternate until the end of an episode and
reset the buffer after every life lost. The four Atari environments used for
testing are shown in Figure 9

###  C.3 Doom

We use the _Take Cover_ scenario on the ViZDoom Kempka et al. (2016a)
platform. We feed the input to a five-layer architecture with three
convolutional layers with 32, 32, and 64 output channels, kernel sizes of
4,4,3, and strides of 2, 2, and 1 for the explorer. We downsize the input
images to $4\times 20\times 20$ and for our observation density model we use
$4\times 20\times 20$ independent Gaussian distributions. The sufficient
statistic is the set of means and standard deviations of the $4\times 20\times
20$ Gaussian distributions. We similarly run the explorer for 64 steps and
controller for 128 steps and alternate until the end of an episode and reset
the buffer after a life has been lost. The scenario used for testing is shown
in Figure 5

(a) Freeway

(b) Berzerk

(c) Assault

(d) Space Invaders

Figure 9: The four Atari environments we used to test each method.

###  C.4 Plots

For each environment and each method, we run a total of six random seeds and
compare the top three, as a measure of the best-case performance. Plots are
generated by smoothing the obtained returns using a rolling window of steps
corresponding to 25, 15, 10, 30 for Freeway, 30, 40, 30, 20 for Berzerk,
10,20,5,1 for Assault, and 10, 15, 10, 2 for Space Invaders. Then, we plot
both the mean and error bars representing a 95% confidence interval over the
seeds.

A website with videos of the experiments is available here:
https://sites.google.com/view/adversarial-surprise/home.

##  Appendix D Algorithm

Randomly initialize $\phi^{E}$ and $\phi^{C}$;

for  _episode = $0,...,M$_ do

Initialize $\theta,R^{i}=0$, $\beta\leftarrow\\{\\}$, explore_turn = True,
$t^{C}=k$, $s_{0}\sim p(s_{0}),o_{0}\sim p(O_{0}|s_{0})$;

for  _$t\leftarrow 0$ to $T$_ do

if  _explore_turn_ then

$a_{t}\sim\pi^{E}(o_{t},h^{E}_{t})$ ;

// Explore

else

$a_{t}\sim\pi^{C}(o_{t},h^{C}_{t})$ ;

// Control

end if

$s_{t+1}\sim\mathcal{T}(s_{t+1}|s_{t},a_{t})$, $o_{t+1}\sim
p(O_{t+1}|s_{t+1})$ ;

// Environment step

$r^{i}_{t}=\log p_{\theta}(o_{t+1})$ ;

// Compute intrinsic reward

if  _not explore_turn and $t-t^{C}>k/2$_ then

$R^{i}=R^{i}+r^{i}$ ;

end if

$\beta=\beta\cup\\{o_{t},a_{t},o_{t+1}\\}$ ;

// Update buffer

if  _$t==t^{C}$_ then

explore_turn = not explore_turn ;

// Switch turns

$t^{C}=t^{C}+2k$;

end if

$\theta_{t+1}=$MLE_update$(\beta,\theta_{t})$ ;

// Fit density model

end for

$\phi^{E}=$RL_update$(\beta,-R^{i})$ ;

// Train Explore policy $\pi^{E}$ with reward $-R^{i}$

$\phi^{C}=$RL_update$(\beta,R^{i})$ ;

// Train Control policy $\pi^{C}$ with reward $R^{i}$

end for

Algorithm 1 Adversarial Surprise

##  Appendix E Broader impact:

AS is an intrinsic motivation method aimed at enhancing learning in
intelligent agents. There has been a rich history of previous work in this
area, where the goal has been to develop more general algorithms for learning
in the absence of task-specific rewards. This work is in line with previous
work and should not directly cause broader harms to society. We are not using
any dataset or tool that will perpetuate or enforce bias. Nor does our method
make judgements on our behalf that could be misaligned with our values. A
potential positive impact of this work may be providing interesting insight
into the nature of intelligence and learning.


# Programmers Prefer Individually Assigned Tasks vs. Shared Responsibility

Adela Krylova  0000-0001-9814-9759<EMAIL_ADDRESS>Innopolis
UniversityInnopolisRussia ,  Roman Makarov  0009-0009-3364-0497
<EMAIL_ADDRESS>Innopolis UniversityInnopolisRussia ,  Sergei
Pasynkov  0000-0001-0000-0000<EMAIL_ADDRESS>Innopolis
UniversityInnopolisRussia and  Yegor Bugayenko  0000-0001-6370-0678
<EMAIL_ADDRESS>HuaweiMoscowRussia

###### Abstract.

In traditional management, tasks are typically assigned to individuals, with
each worker taking full responsibility for the success or failure of a task.
In contrast, modern Agile, Lean, and eXtreme Programming practices advocate
for shared responsibility, where an entire group is accountable for the
outcome of a project or task. Despite numerous studies in other domains, the
preferences of programmers have not been thoroughly analyzed. To address this
gap, we conducted a survey featuring seven situational questions and collected
the opinions of 120 software development practitioners. Our findings reveal
that programmers prefer tasks to be assigned to them on an individual basis
and appreciate taking personal responsibility for failures, as well as
receiving individual rewards for successes. Understanding these preferences is
crucial for project managers aiming to optimize team dynamics and ensure the
successful completion of software projects.

agile, project management, shared responsibility, individual responsibility

##  1\. Introduction

Traditional management theory posits that workers’ commitment is higher if
tasks are assigned to them individually, with each individual taking full
responsibility for failure and success alike (Drucker, 2006b, a; Welch, 2011;
Iacocca, 1984). Contrary to this “archaic” philosophy, proponents of lean
management practices argue that assigning responsibility for failures and
successes to the entire team is a more effective way of managing programmers
(Beck, 2000; Siakas and Siakas, 2007; Yu and Petter, 2014). Despite the
discovery of “social loafing” in 1913 (Ingham et al., 1974; Kravitz and
Martin, 1986), the debate remains unresolved (McCarthy et al., 2019): Who
should bear responsibility for a missed deadline, an individual programmer or
the entire team? In other words, is individual or group responsibility,
defined as “an obligation to satisfactorily perform or complete a task”
(McGrath and Whitty, 2018), more conducive to better management of software
teams?

This debate is not purely metaphysical or philosophical (Beecham et al., 2006,
2008). Its outcome has practical implications on management practices in
software development teams. If we prove that individual responsibility is more
preferred by programmers, we may, for example, consider restructuring
compensation mechanisms of programmers in order to reward them for smaller
individual contributions (on per-task basis), rather than for larger results
of the entire team (annual bonuses) (Patterson and Bugayenko, 2020; Bugayenko,
2020). We may also reconsider the “only positive encouragement” paradigm and
enable certain forms of punishment for personal failures, like missed
deadlines.

The research questions that we ask in this study are the following:

RQ1:

To what extent do software developers perceive themselves as solely
responsible for the success or failure of the tasks assigned to them?

RQ2:

How does the level of professional development influence a software
developer’s understanding and acceptance of responsibility?

In order to answer these questions we created a survey of seven situational
questions. Each question imaginatively places a respondent into a situation
where he/she must either take a responsibility for his/her actions or shift it
to the group. We tried to structure the questions such a way that a respondent
does not understand the intent of our research. We distributed the
questionnaire among 120 programmers, spanning various experience levels.

The survey results identified three key factors influencing programmers’
responsibility within a team setting: individual task ownership, clarity of
task definitions, and the role of external factors. Individual ownership
fosters accountability, while clear task descriptions enhance engagement.
Moreover, we discovered that professional experience does not significantly
impact responsibility perception.

The rest of the paper is organized as following. Section 2 shows a few
previous studies related to the problem we research. Section 3 explains our
method of study. Section 4 presents the results we obtained. Section 5
discusses our findings and Section 6 highlights limitations of our method.

##  2\. Related Work

In software development, understanding how responsibility works among
developers is crucial for successful project management (DeMarco and Lister,
2013; Franca et al., 2008; Brooks, 1975). While numerous studies have explored
different aspects of collective vs. individual responsibility (Narveson, 2002;
Doorewaard et al., 2002), there exists a notable research gap in the
investigation of critical factors influencing responsibility within software
developers (Celepkolu and Boyer, 2018). This chapter reviews existing
literature to highlight the scarcity of research addressing responsibility
within the developer community, providing a foundation for our investigation.

Deak et al. (2016) conducted a study where they described motivating and
demotivating factors for testers. Factors that can have a detrimental impact
on motivation include time constraints, lack of recognition, ineffective
management, and strained relationships. On the other hand, both extrinsic and
intrinsic elements contribute to positive motivation.

Dorairaj et al. (2012) provided an analysis of the factors leading to a lack
of trust in distributed Agile teams, thus negatively impacting responsibility.
These factors include the absence of a sense of belonging, feelings of
vulnerability, inadequate team bonding, insufficient cultural understanding,
the absence of face-to-face interaction, and ineffective communication.

Study by Vishnubhotla and Vishnubhotla (2019) explores capability measurement
in Agile team formation. Survey results from 60 practitioners align with the
SLR, identifying 127 individual and 28 team capability measures. Notably,
seven individual and one team measure were newly identified.

Austin (2001) presented an agency model that investigates the effects of time
pressure on software quality in the context of software development. The
findings suggest that systematically adding slack to deadlines may not always
be an effective strategy for minimizing costs, and that setting deadlines
separately from planning estimates can lead to improved outcomes.

To our knowledge, there has not been any systematic study of an individual vs.
team responsibility dichotomy. It still remains unclear whether programmers
prefer to be personally responsible for their tasks or work with jobs assigned
to their teams.

##  3\. Method

We created a survey method to gather responses about the attitudes toward
responsibility among a community of software developers, spanning various
experience levels. We designed a survey utilizing Google Forms, an online
platform designed for executing surveys.

The first question in our survey served for confirming that the respondents
belongs to the target group. Following seven situational questions (Table 1)
were directly related to our research questions. Then, final three questions
collected demographic profile of respondents. We ended the survey with a non-
mandatory open-ended question. All questions were written in English.

The seven situational questions (Table 1) were intentionally designed to
conceal the purpose of the survey. We expected respondents to imagine work
situations and to respond to them honestly, thus making it possible for us to
understand what are their most probable behavior would be.

Question |  Type |  Possible answers |  Interpretation |  Results
---|---|---|---|---
Q1: “You have two assignments to complete during this working day: one that was assigned to you personally, another that you work on with your team. Which one would you start with?” |  [1]–[4] Likert Scale |  [1] for the first task and [4] for the second task |  Answers closer to [4] contribute toward the strong sense of individual responsibility (RQ1) |  [1] 40.3% [2] 19.3% [3] 25.2% [4] 15.1%
Q2: “A project that you worked on during your probationary period has finished successfully. How likely do you think that you will be offered to join another project?” |  [1]–[4] Likert Scale |  [4] means very likely and [1] means very unlikely |  Responses close to [1] show smaller responsibility for the success of the project, probably due to lower experience level (RQ2) |  [1] 1.7% [2] 12.6% [3] 38.7% [4] 47.1%
Q3: “A recent project, which you were a part of, was closed by the company due to unmet deadlines and incomplete functionality. What should be done?” |  Closed question |  [1] if project managers should be fired; [2] if development team; [3] if the whole team; [4] if no one should be penalized |  [4] means preference of group responsibility (RQ1 + RQ2) |  [1] 56.1% [2] 0.0% [3] 18.2% [4] 25.8%
Q4: “Suppose your laptop broke down and you were not able to finish your work on time. Do you think that it is right to penalize you in this case?” |  [1]–[4] Likert Scale |  [1] for “absolutely wrong” and [4] for “absolutely right” |  Answers close to [4] show strong individual responsibility (RQ1) |  [1] 36.4% [2] 29.5% [3] 19.7% [4] 14.4%
Q5: “Given two tasks, you can’t complete both due to a personal fault. The first task is crucial for the client, but you won’t suffer much personally if it’s delayed. The second task carries a penalty for missing the deadline, though it’s less important for the overall project. Which task would you prioritize?” |  [1]–[4] Likert Scale |  [1] for the first task and [4] for the second task |  Answers close to [1] show stronger responsibility towards client satisfaction (RQ1), while [4] shows low personal responsibility |  [1] 37.9% [2] 18.2% [3] 12.9% [4] 31.1%
Q6: “You got a task with an unclear description. Because of your mistake, you noticed it just before the deadline. Would you finish it the way you think is right without asking for help?” |  [1]–[4] Likert Scale |  [1] means doing the task on your own and [4] means asking for clarification |  Answers close to [1] show lower personal responsibility (RQ1) |  [1] 11.4% [2] 13.6% [3] 20.5% [4] 54.5%
Q7: “You have a project part to complete, but you can only start when your colleague finishes theirs. They say they’ll be late by two days and ask you not to inform the manager. What would you do?” |  [1]–[4] Likert Scale |  [1] means informing project manager and [4] means letting programmer do their job |  [1] shows that the individual responsibility is more important for the respondent, while [4] means waiting, hence loosing resources (RQ1) |  [1] 42.4% [2] 26.5% [3] 16.7% [4] 14.4%
Table 1. All questions from our online survey, together with possible answers
for each question, and the actual results collected (in the last column).

##  4\. Results

This section presents the results obtained from our survey, while the entire
collected dataset is available in open GitHub
repository111https://github.com/rmakarovv/Responsibility_Study. Upon gathering
responses, we analyzed participants’ work experience, scrutinizing the
distribution patterns after classifying participants into two groups: the
first group comprising junior and middle programmers, and the second
consisting of senior software developers. Our categorization was done that
based on the assumption that there could be a gap in working habits between
beginners and advanced specialists.

Consciously opting for a modest number of questions, we aimed to make the
survey interesting and easy to navigate for the respondents, ensuring a
straightforward completion process. This decision proved to be successful,
evidenced by a high completion rate—every respondent who initiated the survey
concluded the entire questionnaire.

We recruited respondents from posts on social networks, and in total, we
gathered responses from 120 individuals. All participants completed all
mandatory questions, with 20 of them providing answers to the final open-ended
question. Upon analyzing the work experience of our respondents, we discovered
that all our respondents have a solid experience as a programmers except one.
Only one respondent did not prove to belong to our target group
(“programmers”) Hence, we decided to exclude this particular response.

Our respondents disclosed the following information about themselves:

  * •

“How much code did you write during the last three months?”
More than 1000 lines (70.6%); Up to 1000 lines (23.5%); Up to 100 lines
(5.9%); Nothing (0.0%).

  * •

“What is your professional level?”
Senior (52.1%); Middle (31.9%); In my field there is not such a segmentation
(5.0%); Junior or internship (10.9%).

  * •

“Do you currently work?”
Yes, full-time (86.6%); No, but I have some working experience (3.4%);
Currently searching for a job (4.2%); Yes, part-time (5.9%).

Our research questions were framed as “To what extent do software developers
perceive themselves as solely responsible for the success or failure of the
tasks assigned to them?” and “How does the level of professional development
influence a software developer’s understanding and acceptance of
responsibility?.” Based on these questions, we designed a survey that is
presented in Table 1, along with the response distribution.

###  4.1. RQ1: To what extent do software developers perceive themselves as
solely responsible for the success or failure of the tasks assigned to them?

The results of 1 show that 69.6% of workers feel more responsible for
completing tasks assigned solely to them. Similarly, 1 reveals that a majority
of developers (85.8%) tend to take responsibility for the success of a
project. These findings suggest that individual ownership is crucial for
fostering a sense of accountability. However, in teamwork environments where
tasks are shared, developers may feel that their work could be done by someone
else, leading to a decreased sense of ownership and ultimately, a lower
perception of personal responsibility.

The results of 1 revealed that clearly defined tasks can significantly enhance
developers’ sense of responsibility.

###  4.2. RQ2: How does the level of professional development influence a
software developer’s understanding and acceptance of responsibility

After collecting responses, we divided them into two categories based on
participant experience: junior/middle developers and senior developers.
Analyzing their responses revealed similar distributions for both groups
across all questions. Thus, contrary to our initial expectations, the findings
did not reveal a significant difference in the perception of responsibility
between less experienced and more experienced developers. This suggests that
the underlying trait of responsibility is primarily personal and not
significantly impacted by age or professional experience. While professional
development can enhance knowledge and skills, it may not significantly impact
an individual’s innate sense of responsibility. Hence, answering the RQ2, the
level of professional development does not influence a software developer’s
understanding acceptance of responsibility.

##  5\. Discussion

The focus of this study was to explore the perception of responsibility among
software developers, with a particular emphasis on the distinctions among
developers of different experience levels. The findings reveal several key
factors influencing the sense of responsibility in these two groups.

Why the professional experience of a programmer does not impact the
responsibility for an assigned task? We believe that the correlation between
professional experience and responsibility for assigned tasks is not absolute
due to several factors. Firstly, experience alone does not guarantee the
acquisition of essential soft skills, such as effective communication,
adaptability, and a strong work ethic, which are crucial for assuming
responsibility. Individuals may accumulate years of technical experience but
lack the interpersonal and organizational skills necessary for responsible
task handling. Secondly, work environments and cultures vary, and some
organizations may foster a sense of responsibility at all experience levels,
emphasizing individual qualities over tenure. Additionally, the dynamic nature
of the technology field requires continuous learning and adaptation, and a
less experienced but proactive individual may demonstrate greater
responsibility by actively seeking knowledge and staying current.

How do software developers perceive their responsibility in the face of
external factors that hinder their work? Most respondents feel responsible for
the results of their work. However, if some outer factors hinder their work,
such as broken laptop (1), they do not feel responsible for missing the
deadline. While developers generally accept responsibility for their work (1),
they may attribute a portion of the responsibility for missed deadlines to
external factors like equipment malfunctions, as we found out that 47.9% of
respondents would not feel responsible for failing the task if their laptop
broke down (1). Hence, organizations should acknowledge this perspective and
provide support mechanisms to help developers mitigate the impact of these
challenges.

Why does Agile emphasize team responsibility, while the opinion of developers
show that people prefer individual responsibility? Contemporary software
development reached high level of complexity; tasks often require
collaboration of several specialists—that makes it impractical to assign them
to individuals. We believe that Agile underscores shared responsibility as a
response to the need for collaborative efforts in addressing multifaceted
challenges. In the context of software development, where a diverse skill set
is crucial, a collective, team-oriented approach becomes essential.
Nevertheless, it is crucial to recognize the individual nature of preferences,
as there are scenarios where assigning individual responsibility may indeed
prove more effective, depending on the task’s nature and an individual’s
specific strengths.

How Can Management Help Programmers Become More Responsible for Their Tasks?
First, as programmers prefer working on tasks assigned to them individually,
according to 1, management should strive to decompose the project scope into
smaller elements and then assign tasks to programmers on an individual basis.
Second, because programmers appreciate it when both success (1) and failure
(1) are attributed personally rather than at a group level, management should
establish motivation mechanisms where both positive and negative incentives
are distributed based on individual achievements (Delfgaauw et al., 2020).
Third, as programmers do not appreciate being punished for failures, according
to 1, management should lean more towards mechanisms of positive
encouragement. Fourth, given that programmers, according to 1, do not hesitate
to seek clarification on the scope of tasks assigned to them, management
should facilitate open communication between programmers and product owners,
thus enabling earlier refinement of requirements, especially in distributed
teams. There could be additional recommendations for managers along these
lines.

##  6\. Limitations

It is essential to recognize the limitations of this study. With a sample size
of 120 respondents, the study’s scope is relatively narrow. Additionally, the
respondents were primarily drawn from personal connections, with a significant
portion being Russian individuals. This sampling approach could limit the
generalizability of the findings to the wider population of software
developers. Expanding the sample size and diversifying the respondent pool
could provide a more thorough understanding of responsibility perceptions
across the broader software development community.

##  7\. Conclusion

This study looked closely at how software developers see responsibility,
focusing on differences between those with more and less experience. We found
that programmers feel more responsible for tasks assigned to them individually
rather than to the group to which programmers belong. Surprisingly, the level
of experience did not show a big difference in responsibility perception,
suggesting it is a personal trait unrelated to professional growth.
Recognizing study limits, such as a small sample of participants from
different backgrounds, cautions against making broad conclusions. However, we
depicted significant factors and answered set research questions. Moreover, we
provide a foundation for future research in the field of software development.

##  8\. Acknowledgement

We would like to thank Nikita Bogdankov, Dilyara Farkhutdinova and Alexey
Potyomkin for their contributive review of our draft.

## References

  * (1)
  * Austin (2001) Robert D. Austin. 2001\.  The Effects of Time Pressure on Quality in Software Development: An Agency Model.  _Information Systems Research_ (2001).  https://doi.org/10.1287/isre.12.2.195.9699
  * Beck (2000) Kent Beck. 2000\.  _Extreme Programming Explained: Embrace Change_.  Addison-Wesley Professional.
  * Beecham et al. (2006) Sarah Beecham, Nathan Baddoo, Tracy Hall, Hugh Robinson, and Helen Sharp. 2006.  Protocol for a Systematic Literature Review of Motivation in Software Engineering.  (2006).
  * Beecham et al. (2008) Sarah Beecham, Nathan Baddoo, Tracy Hall, Hugh Robinson, and Helen Sharp. 2008.  Motivation in Software Engineering: A Systematic Literature Review.  _Information and Software Technology_ 50, 9-10 (2008), 860–878.
  * Brooks (1975) Frederick P. Brooks. 1975\.  _The Mythical Man-Month_.  Addison-Wesley.
  * Bugayenko (2020) Yegor Bugayenko. 2020\.  To Measure or Not to Measure.  https://www.yegor256.com/200623.html.  [Online; accessed 05-03-2024].
  * Celepkolu and Boyer (2018) Mehmet Celepkolu and Kristy Elizabeth Boyer. 2018.  The Importance of Producing Shared Code Through Pair Programming. In _Proceedings of the 49th Technical Symposium on Computer Science Education_. 765–770.
  * Deak et al. (2016) Anca Deak, Tor Stålhane, Tor Stålhane, and Guttorm Sindre. 2016\.  Challenges and Strategies for Motivating Software Testing Personnel.  _Information & Software Technology_ (2016).  https://doi.org/10.1016/j.infsof.2016.01.002
  * Delfgaauw et al. (2020) Josse Delfgaauw, Robert Dur, and Michiel Souverijn. 2020\.  Team Incentives, Task Assignment, and Performance: A Field Experiment.  _The Leadership Quarterly_ 31, 3 (2020), 101241\.  https://doi.org/10.1016/j.leaqua.2018.03.003
  * DeMarco and Lister (2013) Tom DeMarco and Tim Lister. 2013.  _Peopleware: Productive Projects and Teams_.  Addison-Wesley.
  * Doorewaard et al. (2002) Hans Doorewaard, Geert Van Hootegem, and Rik Huys. 2002\.  Team Responsibility Structure and Team Performance.  _Personnel Review_ 31, 3 (2002), 356–370.
  * Dorairaj et al. (2012) Siva Dorairaj, James Noble, Petra Malik, and Petra Malik. 2012\.  Understanding Lack of Trust in Distributed Agile Teams: A Grounded Theory Study.  _Null_ (2012).  https://doi.org/10.1049/ic.2012.0011
  * Drucker (2006a) Peter Drucker. 2006a.  _The Practice of Management_.  Harper Business.
  * Drucker (2006b) Peter Drucker. 2006b.  _The Effective Executive: The Definitive Guide to Getting the Right Things Done_.  Harper Business.
  * Franca et al. (2008) A. Cesar C. Franca, Evisson F. Lucena, Fabio QB da SILVA, and Hermano P. Moura. 2008.  A Qualitative Research on Software Projects Team Building. In _Proceedings of the CONTECSI–5ª International Conference on Technology and Information Systems_.
  * Iacocca (1984) Lee Iacocca. 1984\.  _Iacocca: An Autobiography_.  Bantam Books.
  * Ingham et al. (1974) Alan G. Ingham, George Levinger, James Graves, and Vaughn Peckham. 1974.  The Ringelmann Effect: Studies of Group Size and Group Performance.  _Journal of Experimental Social Psychology_ 10, 4 (1974), 371–384.  https://doi.org/10.1016/0022-1031(74)90033-X
  * Kravitz and Martin (1986) David A. Kravitz and Barbara Martin. 1986.  Ringelmann Rediscovered: The Original Article.  (1986).  https://doi.org/10.1037/0022-3514.50.5.936
  * McCarthy et al. (2019) Stephen McCarthy, Paidi O’Raghallaigh, Ciara Fitzgerald, and Frédéric Adam. 2019.  Towards a Framework for Shared Understanding and Shared Commitment in Agile Distributed ISD Project Teams.  (2019).
  * McGrath and Whitty (2018) Stephen Keith McGrath and Stephen Jonathan Whitty. 2018.  Accountability and Responsibility Defined.  _International Journal of Managing Projects in Business_ 11, 3 (2018), 687–707.
  * Narveson (2002) Jan Narveson. 2002\.  Collective Responsibility.  _The Journal of Ethics_ 6 (2002), 179–198.
  * Patterson and Bugayenko (2020) David Patterson and Yegor Bugayenko. 2020.  Bringing Industry Back to Conferences, and Paying for Results.  _Communications of the ACM_ 63, 11 (2020), 12–13.
  * Siakas and Siakas (2007) Kerstin V. Siakas and Errikos Siakas. 2007.  The Agile Professional Culture: A Source of Agile Quality.  _Software Process: Improvement and Practice_ 12, 6 (2007), 597–610.
  * Vishnubhotla and Vishnubhotla (2019) Sai Datta Vishnubhotla and Sai Datta Vishnubhotla. 2019.  Supplementary Material Of: ”Understanding the Perceived Relevance of Capability Measures: A Survey of Agile Software Development Practitioners”.  _Null_ (2019).  https://doi.org/null
  * Welch (2011) Jack Welch. 2011\.  _Jack: Straight from the Gut_.  Business Plus.
  * Yu and Petter (2014) Xiaodan Yu and Stacie Petter. 2014.  Understanding Agile Software Development Practices Using Shared Mental Models Theory.  _Information and software technology_ 56, 8 (2014), 911–921.