# Continual Learning of Control Primitives:
Skill Discovery via Reset-Games

Kelvin Xu
UC Berkeley
<EMAIL_ADDRESS>
&Siddharth Verma∗
UC Berkeley
<EMAIL_ADDRESS>
Chelsea Finn
Stanford University
<EMAIL_ADDRESS>
&Sergey Levine
UC Berkeley
<EMAIL_ADDRESS>equal contribution

###### Abstract

Reinforcement learning has the potential to automate the acquisition of
behavior in complex settings, but in order for it to be successfully deployed,
a number of practical challenges must be addressed. First, in real world
settings, when an agent attempts a task and fails, the environment must
somehow “reset" so that the agent can attempt the task again. While easy in
simulation, this could require considerable human effort in the real world,
especially if the number of trials is very large. Second, real world learning
often involves complex, temporally extended behavior that is often difficult
to acquire with random exploration. While these two problems may at first
appear unrelated, in this work, we show how a single method can allow an agent
to acquire skills with minimal supervision while removing the need for resets.
We do this by exploiting the insight that the need to “reset" an agent to a
broad set of initial states for a learning task provides a natural setting to
learn a diverse set of “reset-skills". We propose a general-sum game
formulation that balances the objectives of resetting and learning skills, and
demonstrate that this approach improves performance on reset-free tasks, and
additionally show that the skills we obtain can be used to significantly
accelerate downstream learning. 111code is available at
https://github.com/siddharthverma314/clcp-neurips-2020

###  1 Introduction

For reinforcement learning (RL) methods to be successfully deployed in the
real world, they must solve a number of distinct challenges. First, agents
that learn in real world settings, such as robotics, must contend with non-
episodic learning processes: when the agent attempts the task and fails, it
must start from that resulting failure state for the next attempt. Otherwise,
it would require a manually-provided “reset,” which is easy in simulated
environments, but takes considerable human effort in the real world, thereby
reducing autonomy – the very thing that makes RL appealing. Second, complex
and temporally extended behaviors can be exceedingly hard to learn with naïve
exploration [25, 24]. Agents that are equipped with a repertoire of skills or
primitives (e.g., options [43]) could master such temporally extended tasks
much more efficiently. These two problems may on the surface seem unrelated.
However, the non-episodic learning problem may be addressed by learning
additional skills for resetting the system. Could these same skills also
provide a natural way to accelerate learning of downstream tasks? This is the
question we study in this paper. Our goal is to understand how diverse “reset
skills” can simultaneously enable non-episodic reset-free learning and, in the
process, acquire useful primitives for accelerating downstream reinforcement
learning.

Prior works have explored automated skill learning using intrinsic,
unsupervised reward objectives [20, 13, 40], acquiring behaviors that can then
be used to solve downstream tasks with user-specified objectives efficiently,
often by employing hierarchical methods [11, 43, 12]. These skill discovery
methods assume the ability to reset to initial states during this “pre-
training” phase, and even then face a challenging optimization problem:
without any additional manual guidance, the space of potential behaviors is so
large that practically useful skill repertoires can only be discovered in
relatively simple and low dimensional domains. Therefore, we need a compromise
– a more “focused” method that is likely to produce skills that are relevant
for downstream problems, without requiring these skills to be defined
manually.

Figure 1: An outline of our approach. A reset policy
$\pi^{\text{reset}}_{\phi}$ provides a state ${\mathbf{s}_{0}}$ to a forward
policy $\pi_{\theta}$ to learn a task using a learned skill. Upon completion,
the final states, $\mathbf{s}_{T-1}$ is reset with another selected skill. In
real world settings, where all learning is continuous and sample efficiency is
critical, we present a method that leverages the insight that the non-episodic
learning problem can be addressed by learning a set of skills of resetting the
system. We additionally show that these skills can be used to accelerate
downstream learning.

In this paper, we study how these issues can be addressed together with
removing manual resets. While the problems of skill discovery and reset-free
learning may at first appear unrelated, we observe that reset-free learning of
a given task can be enhanced by access to varied and diverse “reset skills,”
which force the task policy to succeed from a variety of starting states. At
the same time, the skills can be forced to acquire more complex behaviors,
through an objective that encourages them to discover states that are
challenging for the task policy. This means that the same mechanism that can
address the “reset” problem can also serve as a skill discovery method.
Performing a single given task can be viewed as a “funneling” process:
transitioning from a wide distribution over initial conditions to a narrower
distribution over states where a given task has been completed successfully.
Reversing this process therefore can provide coverage over a broader range of
states, while still keeping this exploration process grounded to situations
from which the initial task is solvable.

The main contribution of this paper is a “reset game” that implements this
idea as a general-sum game between two players: a task policy player that
attempts to perform a given task, and a reset policy player, corresponding to
a repertoire of distinct skills, that attempts to perturb the state to make
the task harder for the task policy while _simultaneously_ producing a diverse
set of skills (see Figure 1). The result of this process is a setting where
the challenge of learning without resets is instead viewed as an opportunity
to learn a diverse set of behavioral primitives, which can then be used to
accelerate downstream reinforcement learning, both removing the need for
human-provided resets and acquiring skills that can be used for downstream
tasks. We show that by grounding the skill learning problem in this way, we
can design a method that (1) enables an agent to learn a task without
requiring oracle resets, which is a non-trivial requirement in real-world
applications, and (2) learns a broader set of skills compared to prior
unsupervised approaches, which substantially improves effectiveness on down-
stream long-horizon tasks. We demonstrate the efficacy of our approach both on
previously studied robotics-themed reset-free RL problems, and tasks that
reflect domains studied in prior unsupervised skill learning work,
demonstrating both improved reset-free learning performance and improved
performance on downstream tasks with hierarchical controllers.

###  2 Related Work

Learning without access to resets has been studied in prior work with a focus
on automation, safety, and learning compound controllers [23, 13, 8].
Eysenbach et al. [13] propose to learn a reset controller for safe learning,
but assume access to ground truth rewards and an oracle function that can
detect resets. Closely related to our work, Zhu et al. [54] learn a
perturbation controller to handle the reset-free setting, but importantly does
not learn a set of skills. In contrast, we propose to learn a broad set of
skills for performing resets, which are “anchored” to a specific forward RL
task by requiring that the reset skills produce states that are diverse and
challenging. We show that this anchoring leads to skills more suited for
downstream learning, while improving reset-free performance.

Autonomous acquisition of skills is related to the learning with intrinsic
motivation, which considers the unsupervised setting where an agent must learn
without extrinsic reward [39, 9, 26, 4, 3]. In the direction of curiosity-
driven exploration, recent work has used state novelty to reward an agent [5,
44, 33]. Similarly, other work has instead used the empowerment maximization
principle [38] to define mutual information based objectives from which to
learn behavioral primitives [30, 20, 16, 14, 40]. Our method incorporates
unsupervised skill discovery into reset-free learning, building on mutual
information methods for skill discovery. In contrast to these prior methods,
our approach uses a forward task to implicitly “anchor” the skills. We show
experimentally that this allows us to learn varied skills in the reset-free
setting, and also produces better skills for downstream task learning.

The goal of acquiring primitives for the purpose of enabling efficient
learning on downstream tasks is also a central goal of hierarchical
reinforcement learning (HRL), which has been a long-standing area of research
[11, 51, 32, 43, 12]. Recent work has investigated deep hierarchical agents
[1, 48, 16, 31], building on methods that explicitly design a hierarchy over
actions [32, 12], or utilize options [43, 35]. Our work is also motivated by
acquiring temporal abstraction through experience, although we seek to do so
under a problem setting that is reset-free. Our work can be seen as
complementary to prior approaches in HRL, as the skills we acquire can be used
by an HRL agent to accelerate downstream learning. We demonstrate this
empirically in Section 5.

Central to our method is formulating the problem of acquiring diverse skills
as an adversarial two player game. Adversarial games have been proposed in RL
with the goal of improving exploration [42, 27], learning goal conditioned
policies [36] and generating an automated curriculum [45, 41, 2]. Our work
resembles the reset variant of asymmetric self-play [42], though our approach
differs in two ways. First, while the reset variant of asymmetric self-play
defines “Bob’s” learning objective using its ability to reset “Alice’s”
trajectories, the problem setting is not in fact reset-free as Alice is reset
at every episode. Second, our method does not involve setting new goals or use
time to completion as a proxy reward, but rather makes use of a task reward to
measure which reset states are challenging. Unlike prior methods that
formulate an adversarial game around goal reaching [42, 36, 17], our method
does not involve setting or reaching goals, but rather uses a single task to
focus the forward policy, providing opportunities for the reset skills to
learn varied adversarial perturbations. Unlike methods that adversarially vary
the environment [6, 50], our method does not require any additional privileged
capability to change the environment parameters. Instead, the reset skills use
the same action space as the forward policy to discover states from which the
original task is difficult. Finally, unlike all of the previously listed
methods, our approach enables learning in a reset-free setting – a setting
where, as we show experimentally in Section 5, prior methods struggle to learn
effectively.

###  3 Preliminaries

Our eventual goal is to devise a reset-free, non-episodic learning framework.
We first describe standard episodic RL. An RL problem is defined on a Markov
decision process (MDP), represented by the tuple: $\mathcal{M}=$
($\mathcal{S},\mathcal{A},\mathcal{P}_{\mathbf{s}},r,\gamma,\mathcal{P}_{0}$),
where $\mathcal{S}$ is a set of continuous states and $\mathcal{A}$ is a set
of continuous actions,
$\mathcal{P}_{\mathbf{s}}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$
is the transition probability density,
$r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function,
$\gamma$ is the discount factor and $\mathcal{P}_{0}$ is the initial state
distribution. The $\gamma$-discounted return $R(\tau)$ of a trajectory
$\tau=({\mathbf{s}_{0}},{\mathbf{a}_{0}},\dots\mathbf{s}_{T-1},\mathbf{a}_{T-1})$
is $\sum_{t=0}^{T-1}\gamma^{t}r(\mathbf{s}_{t},\mathbf{a}_{t})$. In episodic,
finite horizon tasks of length $T$, the goal is to learn a policy
$\pi_{\theta}:\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$ that
maximizes the objective
$J(\pi_{\theta})=\mathbb{E}_{\pi_{\theta},\mathcal{P}_{0},\mathcal{P}_{\mathbf{s}}}\left[R(\tau)\right]$.
Here, $\theta$ denotes the parameters of the policy $\pi_{\theta}$, which are
learned by iteratively sampling episodes, where at the end of each episode, a
new initial state is ${\mathbf{s}_{0}}$ is sampled from the initial state
distribution $\mathcal{P}_{0}$. While such resets can be easily obtained in
simulated tasks (e.g., Atari [29]), this is significantly more challenging in
real-world physical tasks, as it requires human intervention in the form of
manual resets or the availability of a hard coded reset [28, 47, 37, 52, 21].

Our method will simultaneously remove the need for manual resets and, at the
same time, learn a varied set of skills by resetting the environment in
different ways. To learn this set of skills, we build on previously proposed
episodic skill learning methods, which we review here. Following prior work
[14, 40], skills can be represented by conditioning the policy on a latent
skill index $\mathbf{z}$, which is sampled from $\mathcal{P}(\mathbf{z})$ and
kept fixed over a temporally extended period. Every distinct value of
$\mathbf{z}$ should correspond to a distinct behavior. One objective for
achieving this proposed in prior work [20, 14, 40, 34] is the mutual
information (MI) between skills and the states they visit:
$\mathcal{I}(\mathbf{s};\mathbf{z})=\mathcal{H}(\mathbf{s})-\mathcal{H}(\mathbf{s}|\mathbf{z})$.
Maximizing $\mathcal{I}(\mathbf{s};\mathbf{z})$ entails maximizing the state
entropy (high state coverage) while minimizing conditional state entropy (high
predictability for each $\mathbf{z}$). This is typically combined with
minimizing the MI between actions ($\mathbf{a}$) and the state/context
($\mathbf{s},\mathbf{z}$), so that skills are distinguished by the states they
visit rather than the actions they take. The resulting objective [14] is:

$\displaystyle\mathcal{I}(\mathbf{s};\mathbf{z})-\mathcal{I}(\mathbf{a};\mathbf{s},\mathbf{z})$
$\displaystyle=\mathbb{E}_{\pi}\left[\log\frac{p(\mathbf{z}|\mathbf{s})}{p(\mathbf{z})}-\log\frac{\pi(\mathbf{a}|\mathbf{s},\mathbf{z})}{\pi(\mathbf{a})}\right]$
$\displaystyle\geq\mathbb{E}_{\pi}\left[\log
q_{\omega}(\mathbf{z}|\mathbf{s})-\log
p(\mathbf{z})-\log\pi(\mathbf{a}|\mathbf{s},\mathbf{z})\right]=\mathcal{G}(\theta,\omega),$

where we replace $p(\mathbf{z}|\mathbf{s})$ with an approximate learned
discriminator $q_{\omega}(\mathbf{z}|\mathbf{s})$ with parameters $\omega$ to
obtain a variational lower bound. We can maximize $\mathcal{G}(\theta,\omega)$
with RL using the pseudo-reward:

$r_{\text{skill}}(\pi,q_{\omega})=\log q_{\omega}(\mathbf{z}|\mathbf{s})-\log
p(\mathbf{z})-\log\pi(\mathbf{a}|\mathbf{s},\mathbf{z}).$ (1)

The skill learning algorithm alternates between learning the skills with this
pseudo-reward and optimizing for a discriminator that is able to discriminate
between the skills.

###  4 Learning Skills via the Reset Game

To learn without resets, our approach aims to learn both a forward policy
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$ and a set of reset skills, which we
denote $\pi^{\text{reset}}_{\phi}(\mathbf{a}|\mathbf{s},\mathbf{z})$. We first
describe the optimization objectives for each of the policies. Then, we
describe how we can balance these two objectives using a game theoretic
formulation. Finally, we instantiate this method and present a practical
algorithm, where a single goal-oriented forward policy is reset by multiple
different reset skills. While learning multiple skills and learning without
resets may at first seem like largely unrelated problems, we make the
observation that a sufficiently diverse set of skills can serve to alleviate
the need for a manual reset, because different policies (i.e.,
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$ and
$\pi^{\text{reset}}_{\phi}(\mathbf{a}|\mathbf{s},\mathbf{z})$) can reset _each
other_ into different states. By forcing $\pi_{\theta}(\mathbf{a}|\mathbf{s})$
to succeed from the final states of a range of different reset skills,
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$ becomes proficient at performing the
task from many states. By forcing the reset skills to all perturb
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$ in different ways, the reset skills
themselves are forced to differentiate and learn various behaviors, making
them suitable for downstream learning of more complex tasks. In this way, the
problems of learning without resets and learning diverse skills are a natural
fit to be addressed jointly.

Our goal is to devise a method such that our policies have the following
properties: (1) executing the reset policy allows the forward policy to learn
to solve the task with standard episodic RL (i.e., without an oracle reset to
the initial state distribution), (2) through repeated resetting, the reset
policy is able to discover skills that may be useful for downstream tasks.

Figure 2: An outline of our approach. Our method first begins by (1) sampling
a skill from a prior distribution that is used to condition the reset policy.
Next, the reset agent acts in the environment in order to bring the agent to
the initial state for the forward policy (2). The states of the reset policy
are passed to additionally to a learned discriminator which tries to determine
which skill was used to generate the final state (3). Next, the forward policy
tries to solve the task (4) and all agents attempt to learn by maximizing
their respective objective (5).

####  4.1 The Reset Game

We first describe the reset game that represents the core of our method, and
then in Section 4.2 extend this to incorporate multiple skills by conditioning
$\pi^{\text{reset}}_{\phi}$ on $\mathbf{z}$. Our proposed reset game enables
reset-free learning via a competitive interaction between
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$ and
$\pi^{\text{reset}}_{\phi}(\mathbf{a}|\mathbf{s})$, specified as follows:

$\max_{\pi_{\theta}}\;\;\;\mathcal{J}^{\text{forward}}(\pi_{\theta},\pi^{\text{reset}}_{\phi}),\;\;\max_{\pi^{\text{reset}}_{\phi}}\;\;\;\mathcal{J}^{\text{reset}}(\pi_{\theta},\pi^{\text{reset}}_{\phi}),$
(2)

where $\mathcal{J}^{\text{forward}}$ and $\mathcal{J}^{\text{reset}}$ are the
expected returns, which we define below. Both $J^{\text{forward}}$ and
$J^{\text{reset}}$ depend on both $\pi_{\theta}$ and
$\pi^{\text{reset}}_{\phi}$, since the starting state for each policy is
obtained by running the other policy (i.e.,
$J^{\text{forward}}(\pi_{\theta},\pi^{\text{reset}}_{\phi})=\mathbb{E}_{\pi_{\theta},\pi^{\text{reset}}_{\phi},\mathcal{P}_{\mathbf{s}}}\left[R(\tau)\right]$,
and vice-versa).

$\mathcal{J}^{\text{forward}}$ is the task objective, which forces the task
policy $\pi_{\theta}(\mathbf{a}|\mathbf{s})$ to learn to perform the given
task. $\mathcal{J}^{\text{reset}}$ is an objective that causes the reset
policy $\pi^{\text{reset}}_{\phi}(\mathbf{a}|\mathbf{s})$ to discover states
from which it is difficult for the task policy to succeed at the task, while
at the same time promoting diverse and varied resets. We provide specific
definitions for these objectives in Section 4.2, though a number of different
options are possible. Crucially, this is not a zero-sum game. However, due to
the fact that the reset policy and task policy must alternate, the reset game
has a pre-specified turn order. This form of sequential game is known as a
Stackelberg game [49], and is closely connected to the field of bi-level
optimization [10]. Unlike general games, these games are known to be solvable
(using game theoretic measures) by gradient based learning algorithms [15],
even when the game is not zero-sum. Formally, a Stackelberg game is defined as
an optimization problem where the “leader” must optimize the following
optimization problem:

$\displaystyle\max_{x_{1}\in
X_{1}}\;\\{\,f_{1}(x_{1},x_{2})\;\;|\;\;x_{2}\in\operatorname*{arg\,max}_{y\in
x_{2}}f_{2}(x_{1},y)\,\\}.$

The follower optimizes the function $\operatorname*{arg\,max}_{x_{2}\in
X_{2}}f_{2}(x_{1},x_{2})$. Taking the reset policy as the leader in our
setting results in the following optimization problem:

$\max_{\pi^{\text{reset}}_{\phi}}\;\;\;\\{\,\mathcal{J}^{\text{reset}}(\pi_{\theta}^{*},\pi^{\text{reset}}_{\phi})\;|\;\;\pi_{\theta}^{*}\in\operatorname*{arg\,max}_{\pi_{\theta}}\;\mathcal{J}^{\text{forward}}(\pi_{\theta},\pi^{\text{reset}}_{\phi})\,\\}.$
(3)

Based on this connection, we can devise a gradient-based algorithm to solve
the reset game in Equation 2, so long as the reset policy and task policy
learn at different rates (e.g., with the task policy learning faster). We
present such an algorithm in Section 4.3, but first we describe how the reset
policy can discover diverse skills during the reset.

####  4.2 Mining Diverse Skills from Diverse Resets

Recall that our goal is to allow the forward policy to learn to optimize its
task reward $r$, as if a reset mechanism were present, and for our reset
mechanism to provide challenging and diverse resets. Towards our first goal,
we propose to optimize the forward RL policy from the initial state
distribution implied by executing $\pi^{\text{reset}}_{\phi}$ for $T_{reset}$
time steps. Towards our second goal, we propose to use the skill learning
reward $r_{\text{skill}}$ defined in Equation 1 to simultaneously acquire a
repertoire of reset skills that are forced to diversify. The underlying mutual
information objective encourages high coverage of initial states, but to
ensure that these states are challenging for $\pi_{\theta}$, we add the
negative return of $\pi_{\theta}$ to the objective for each reset skill.
Putting these terms in the game defined in Equation 3 results in the following
optimization problem:

$\displaystyle\max_{\phi}$
$\displaystyle\;\;\underbrace{\mathbb{E}_{{\mathbf{s}_{t}}^{\prime},{\mathbf{a}_{t}}^{\prime}\sim\pi^{\text{reset}}_{\phi}}\left[\sum_{t=0}^{T_{reset}-1}\gamma^{t}r_{skill}({\mathbf{a}_{t}}^{\prime},{\mathbf{s}_{t}}^{\prime})-\lambda\;\mathbb{E}_{\pi_{\theta^{*}}}\left[\sum_{t=0}^{T-1}\gamma^{t}r({\mathbf{a}_{t}},{\mathbf{s}_{t}})\right]\right]}_{\mathcal{J}^{\text{reset}}(\pi_{\theta},\pi^{\text{reset}}_{\phi})}$
(4) $\displaystyle\text{such that
}\theta^{*}=\operatorname*{arg\,max}_{\theta}\>\underbrace{\mathbb{E}_{\pi_{\theta}}\left[\sum_{t=0}^{T-1}\gamma^{t}r({\mathbf{a}_{t}},{\mathbf{s}_{t}})\right]}_{\mathcal{J}^{\text{forward}}(\pi_{\theta},\pi^{\text{reset}}_{\phi})}.$
(5)

The hyperparameter $\lambda$ controls the relative importance of the two terms
of $\mathcal{J}^{\text{reset}}(\pi_{\theta},\pi^{\text{reset}}_{\phi})$.
Intuitively, setting $\lambda=0$ reduces the objective of the reset controller
to prior skill learning methods [14], while $\lambda>0$ requires the reset
controller to reach more challenging terminal states. As the task policy and
reset skills improve, they provide a curriculum for one another, with the task
policy pushing the skills to reach more distant states, and the skills pushing
the policy to be effective from a larger range of initializations. As we
demonstrate in our experiments, this results in more complex and varied
skills, as well as faster reset-free learning for $\pi_{\theta}$. To reflect
the synthesis of resets and skill learning, we refer to our specific
instantiation of the reset game as Learning Skills from Resets (LSR).

Running this reset game produces an effective forward policy
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$, as well as a set of skills defined by
$\pi^{\text{reset}}_{\phi}(\mathbf{a}|\mathbf{s},\mathbf{z})$. The latter can
be used to solve more complex downstream tasks in a hierarchical RL framework,
using a higher-level policy $\pi^{\text{hi}}(\mathbf{z}|\mathbf{s})$ to
command the skill $\mathbf{z}$ that should be executed at each (higher-level)
time step. We demonstrate this capability in our experiments.

Algorithm 1 Learning Skillful Resets (LSR)

1: Input: Environment

2: Initialize: policy $\pi_{\theta}$, reset policy
$\pi^{\text{reset}}_{\phi}$, discriminator
$q_{\omega}({\mathbf{s}_{t}}|{\mathbf{a}_{t}})$, prior $p(z)$

3: for $N$ iterations do

4: Sample skill $\mathbf{z}\sim p(\mathbf{z})$

5: # Set environment

6: for $t\leftarrow 0\dots T_{\text{reset}-1}$ do

7: Sample
${\mathbf{a}_{t}}~{}\sim\pi^{\text{reset}}_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}},\mathbf{z})$

8: Step env
${\mathbf{s}_{t+1}}~{}\sim\mathcal{P}_{\mathbf{s}}({\mathbf{s}_{t+1}}|{\mathbf{s}_{t}},{\mathbf{a}_{t}})$,
compute reward for reset controller
$r_{t}^{reset}=q_{\omega}({\mathbf{s}_{t}}|{\mathbf{a}_{t}})$

9: end for

10: # Solve task

11: for $t\leftarrow 0\dots T-1$ do

12: Sample
${\mathbf{a}_{t}}~{}\sim\pi_{\theta}({\mathbf{a}_{t}}|{\mathbf{s}_{t}})$

13: Step env
${\mathbf{s}_{t+1}}~{}\sim\mathcal{P}_{\mathbf{s}}({\mathbf{s}_{t+1}}|{\mathbf{s}_{t}},{\mathbf{a}_{t}})$,
obtain environment reward $r_{t}$

14: end for

15: Update reset policy’s final reward
$r_{T}^{reset}=-\sum_{t=0}^{T}\gamma^{t}r_{t}({\mathbf{a}_{t}},{\mathbf{s}_{t}})$

16: Update $\pi^{\text{reset}}_{\phi}$, $\pi_{\theta}$ to maximize respective
return using SAC

17: Update discriminator $q_{\omega}$ using Adam.

18: end for

####  4.3 Algorithm Summary: Optimizing the Reset Game

We could optimize the task policy and reset skills as per Equation 4 via a bi-
level optimization, where the parameters of $\pi^{\text{reset}}_{\phi}$ are
held fixed while $\pi_{\theta}$ is optimized to convergence. However, in an RL
setting, this would require an excessive number of samples. A substantially
more efficient method alternates gradient steps on both objectives, with
different learning rates. Since the reset game corresponds to a Stackelberg
game, gradients-based learning algorithms with respect to these objectives
exist that have finite time high probability bounds for local convergence in
the case of non zero-sum objectives [15]. Optimizing such a problem can be
done approximately by using a two-timescale algorithm, where the leader is
optimized at a smaller learning rate [15]. This intuitively results in a reset
controller that changes “slower” relative to the forward policy. In our
implementation, we approximately optimize
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$ and
$\pi^{\text{reset}}_{\phi}(\mathbf{a}|\mathbf{s},\mathbf{z})$ using soft
actor-critic (SAC) [22].

We outline the pseudo-code of our approach in Algorithm 1. At each iteration,
we sample a reset skill $\mathbf{z}\sim p(\mathbf{z})$, execute that skill by
following
$\pi^{\text{reset}}_{\phi}({\mathbf{a}_{t}}|{\mathbf{s}_{t}},\mathbf{z})$ to
reset to a challenging initial state, then execute
$\pi_{\theta}(\mathbf{a}|\mathbf{s})$ to attempt the task. The experience
collected during these trials is then used to update the corresponding
policies.

###  5 Experiments

In this section we aim to experimentally answer the following questions: (1)
How does our approach compare to prior methods in the reset free domain? (2)
How do resets and state representation choice effect the skills we learn? (3)
How do the skills learned by our approach compare to prior work? Specifically,
can we learn better hierarchical controllers using the primitives learned by
our approach (LSR)? We first describe our experimental setup, evaluation
metrics, and the prior methods to which we will compare. We leave detailed
discussion of hyperparameters and environment parameters to the Appendix
Appendix.

Experimental setup. To study reset-free learning, we use the three-fingered
hand repositioning task proposed by Zhu et al. [54], where the hand must
position an object in the center of a bin, starting from any position. This is
the most challenging domain considered by Zhu et al. [54] for reset-free
learning. We refer to this environment as the DClaw-ManipulateFreeObject
environment. We follow the evaluation protocol of Zhu et al. [54], using an
evaluation metric that measures the final position and orientation of the
object compared to the goal position evaluated from a set of initial positions
that is consistent across all methods.

Figure 3: Diagram of the hierarchical locomotion tasks considered in this
work. The agent must first acquire locomotion skills in a reset game (left),
where the task policy must learn to walk to the origin of the workspace. The
skills are subsequently used as the action space for a hierarchical policy,
which must learn complex hierarchical navigation tasks (second and third
image). The far-right image shows the paths taken by a hierarchical policy
using our learned reset skills.

To evaluate our second experimental hypothesis, we evaluate skills acquired
with reset-free learning for an ant locomotion task on two hierarchical
navigation problems, shown in Figure 3. In these experiments, the agent must
first learn a set of skills (without resets) to control an Ant quadrupedal
robot to walk to the center of the workspace (far left), and subsequently
transfer those skills to a waypoint navigation task and a maze traversal
environment, which we refer to as Ant-Waypoint and Ant-MediumMaze (center,
right plot respectively). For all methods, we report a return, which we
normalize using the return of an agent that successfully solves the task and a
random agent.

Figure 4: Reset-free learning comparison (lower is better). The error bars
show 95% bootstrap confidence intervals for average performance. We average
each method over 4 seeds. Our method (blue) outperforms prior methods (orange,
green). Compared to R3L [54] (orange), our approach converges $\sim$28.6%
faster. Our ablations show that the most important aspect of our approach is
having multiple resetters. This indicates that reset state convergence is
crucial to good performance, in contrast to prior methods [13] that learn a
explicit reset policy.

Comparisons and baselines. We compare our approach to prior methods, which are
trained either with or without resets. In the DClaw environments, we compare
to two prior methods that learn reset controllers: leave no trace (LNT) [13],
and the R3L perturbation controller (R3L) [54]. In order to fairly compare all
prior methods, we use the same underlying RL algorithm, soft actor-critic
(SAC) [22], across all methods, and use the same exploration bonus used in R3L
[54], random network distillation [7], which is added to the reward for all
methods.

We also compare to prior skill learning methods on the Ant task: Diversity is
All You Need (DIAYN) [14] and Dynamics-Aware Discovery of Skills (DADS) [40].
DADS and DIAYN are unsupervised skill learning procedures that aim to learn a
diverse set of skills in an environment. For this domain, prior work has
differed in the choice of state representation, either using the full state
[14], which is a generally more challenging setting, or a reduced $(x,y)$
coordinate state representation. For our experiments, in order to understand
the effect of resets and state representation choice, we experiment with both
the full-state representation and reduced $(x,y)$ coordinate state space, in
addition to running all methods with resets or reset-free. In the case of DADS
[40], which uses continuous skills, we discretize the skill space to allow us
to use the same hierarchical learner.

Figure 5: A visualization of the learned skills (top) and performance on a
hierarchical downstream locomotion task (bottom) under different state
representation choices and reset availability. We visualize the $(x,y)$
trajectories of learned skills. In comparison to prior methods, LSR is able to
learn skills that are able to robustly move the ant much further from the
origin compared to prior approaches which either need the $(x,y)$-prior or
resets. This allows us to substantially improve performance in a downstream
hierarchical locomotion task (bottom, top-left), where a meta-controller using
only the skills is able to navigate a maze much more effectively. . Figure 6:
Performance on the Ant-Waypoints environment compared to DADS and DIAYN. DADS
and DIAYN makes small progress on this simpler domain (orange), but our method
(blue) is still able to successfully navigate between the waypoints more
quickly.

Reset-free learning results. The results in Figure 4 show that our approach
substantially improves reset-free performance. In contrast to prior approaches
(orange and green), our method converges approximately 200 iterations faster
on the most challenging task (DClaw-ManipulateFreeObject), which is a 28.6%
improvement. We hypothesize that is due to the improved state coverage of our
approach, as an explicit reset policy (green curve) [13] performs much worse.
This is supported by our ablations, which show that having “Multiple-
Resetters” (red), where we remove the diversity bonus, accounts for much of
the method’s improvements. Using a single adversary (purple-curve), where we
run our approach with a single skill, performs comparably to R3L, and
substantially worse than our method. Note that prior techniques that have
formulated adversarial games, such as asymmetric self play (ASP) [42],
generally use a single adversary.222While ASP shares many properties with our
method, such as the use of a game formulation, it was infeasible to compare on
these tasks due to differences in assumptions: ASP does not consider reset-
free learning, does not aim to optimize a given task, and requires goal-
reaching policy formulations for both players. However, we can view the
“Single Adversary” baseline as the closest to ASP in our experiments.

Choice of state representation and the effect of resets. In the Ant domain, we
find that our approach allows us to robustly learn skills across reset
settings and state representation choice. We visualize the $(x,y)$
trajectories of the learned reset skills qualitatively in Figure 8 (top),
along with skills learned by prior methods with and without resets [14, 40].
Both prior methods (DIAYN and DADS) learn short directional walking skills
with resets (top, right two columns), similar to those reported in prior work
[14, 40], but generally fail to learn meaningful skills in the reset-free
setting. Performance is generally improved by using the $(x,y)$ prior as the
state representation, but this is not robust in the reset-free setting for
prior methods. In contrast, the skills learned by our method are capable of
moving the ant much further than prior methods that learn in _either_ the
reset-free or reset-based setting, with both the full-state and $(x,y)$
representation, as illustrated in the figure.

Skill learning and hierarchical control results. Finally, we compare our
approach to prior methods by evaluating the quantitative performance on
downstream hierarchical tasks. we see that the skills produced by our approach
improves performance in the simpler Ant-WayPoints navigation domain Fig. 6,
and these improvements are much more pronounced in the more challenging Ant-
MediumMaze task, where our approach leads to substantially more effective
downstream learning, as shown in Fig. 8 (right), compared to prior methods. We
normalize each curve in Fig. 8 by the return of the best performing policy. A
visualization of the path taken through the maze by our hierarchical policy in
the reset-free, full state representation setting is shown in Fig. 3 (far
right). We provide a similar analysis on the Ant-Waypoints task in Appendix.
B.

###  6 Discussion, Limitations and Future Work

We proposed a method that simultaneously addresses two challenges faced by
real-world RL systems: learning continually without manually provided resets,
and acquiring diverse skills for solving long-horizon downstream tasks. While
these two problems may at first appear unrelated, we show that learning
diverse skills actually enables more effective reset-free learning, while
learning the skills in an adversarial manner with a task policy actually makes
the skills themselves more effective, and therefore better suited for
downstream long-horizon problems. Our work assumed environments that are
reversible, which is a strong assumption assumption in the real world. We also
assumed a defined forward task, whereas a fully autonomous real world agent
would need to have the ability to verify its own task success. A particularly
exciting direction for future work would be to extend the reset game to learn
a zero or few-shot success classifier, or design a system that explore
conservatively and request limited human intervention when encountering a non-
communicating state.

### Broader Impact Statement

The central goal of reinforcement learning is to allow agents to autonomously
acquire complex behavior from interacting with the world. The central focus on
this work is to address challenges in the deployment of reinforcement learning
in the real world, where all learning is continual and sample efficient is
important. We introduce our approach to remove the assumption of an oracle
reset function and to allow the agent to simultaneously acquire temporal
abstraction. In order to maximally benefit from deploying reinforcement
learning agents in the real world, these challenges must be addressed by one
method or another.

Nevertheless, as the underlying goal of our research in reinforcement learning
is to enable the automatic acquisition of complex behavior, we consider the
societal impact of our research to be closely related to automation. While the
broader impact of having increased automation is difficult to predict, it is
likely to have both negative and positive consequences. Having agents that can
be deployed in unstructured environments could have the potential to automate
dangerous tasks, but also could result in more complex consequences, including
changes in the employment landscape, and applications with complex ethical
implications, such as autonomous weapons.

### Acknowledgments and Disclosure of Funding

We thank the anonymous reviewers, whose comments greatly improved this work.
We also thank Parsa Mahmoudieh, Michael Chang, Justin Fu and Ashvin Nair for
feedback on an earlier draft of this paper. We additionally would like to give
a special thanks to Justin Yu for clarifying details and helping reproduce the
R3L experiments. This research was supported by the National Science
Foundation under IIS-1700696, the Office of Naval Research, ARL DCIST CRA
W911NF-17-2-0181, Berkeley DeepDrive, and compute support from Google.

### References

  * Bacon et al. [2017] P.-L. Bacon, J. Harb, and D. Precup.  The option-critic architecture.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Baker et al. [2019] B. Baker, I. Kanitscheider, T. Markov, Y. Wu, G. Powell, B. McGrew, and I. Mordatch.  Emergent tool use from multi-agent autocurricula.  _arXiv preprint arXiv:1909.07528_ , 2019.
  * Baranes and Oudeyer [2013] A. Baranes and P.-Y. Oudeyer.  Active learning of inverse models with intrinsically motivated goal exploration in robots.  _Robotics and Autonomous Systems_ , 61(1):49–73, 2013.
  * Barto [2013] A. G. Barto.  Intrinsic motivation and reinforcement learning.  In _Intrinsically motivated learning in natural and artificial systems_ , pages 17–47. Springer, 2013.
  * Bellemare et al. [2016] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos.  Unifying count-based exploration and intrinsic motivation.  In _Advances in Neural Information Processing Systems_ , pages 1471–1479, 2016.
  * Brant and Stanley [2017] J. C. Brant and K. O. Stanley.  Minimal criterion coevolution: a new approach to open-ended search.  In _Proceedings of the Genetic and Evolutionary Computation Conference_ , pages 67–74, 2017.
  * Burda et al. [2018] Y. Burda, H. Edwards, A. Storkey, and O. Klimov.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_ , 2018.
  * Chatzilygeroudis et al. [2018] K. Chatzilygeroudis, V. Vassiliades, and J.-B. Mouret.  Reset-free trial-and-error learning for robot damage recovery.  _Robotics and Autonomous Systems_ , 100:236–250, 2018.
  * Chentanez et al. [2005] N. Chentanez, A. G. Barto, and S. P. Singh.  Intrinsically motivated reinforcement learning.  In _Advances in neural information processing systems_ , pages 1281–1288, 2005.
  * Colson et al. [2007] B. Colson, P. Marcotte, and G. Savard.  An overview of bilevel optimization.  _Annals of operations research_ , 153(1):235–256, 2007.
  * Dayan and Hinton [1993] P. Dayan and G. E. Hinton.  Feudal reinforcement learning.  In _Advances in neural information processing systems_ , pages 271–278, 1993.
  * Dietterich [2000] T. G. Dietterich.  Hierarchical reinforcement learning with the maxq value function decomposition.  _Journal of artificial intelligence research_ , 13:227–303, 2000.
  * Eysenbach et al. [2017] B. Eysenbach, S. Gu, J. Ibarz, and S. Levine.  Leave no trace: Learning to reset for safe and autonomous reinforcement learning.  _arXiv preprint arXiv:1711.06782_ , 2017.
  * Eysenbach et al. [2018] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine.  Diversity is all you need: Learning skills without a reward function.  _arXiv preprint arXiv:1802.06070_ , 2018.
  * Fiez et al. [2019] T. Fiez, B. Chasnov, and L. J. Ratliff.  Convergence of learning dynamics in stackelberg games.  _arXiv preprint arXiv:1906.01217_ , 2019.
  * Florensa et al. [2017] C. Florensa, Y. Duan, and P. Abbeel.  Stochastic neural networks for hierarchical reinforcement learning.  _arXiv preprint arXiv:1704.03012_ , 2017.
  * Florensa et al. [2019] C. Florensa, J. Degrave, N. Heess, J. T. Springenberg, and M. Riedmiller.  Self-supervised learning of image embedding for continuous control.  _arXiv preprint arXiv:1901.00943_ , 2019.
  * Fu et al. [2018] J. Fu, A. Singh, D. Ghosh, L. Yang, and S. Levine.  Variational inverse control with events: A general framework for data-driven reward definition.  In _Advances in Neural Information Processing Systems_ , pages 8538–8547, 2018.
  * Fu et al. [2020] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine.  D4RL: Datasets for deep data-driven reinforcement learning.  _https://arxiv.org/abs/2004.07219_ , 2020.
  * Gregor et al. [2016] K. Gregor, D. J. Rezende, and D. Wierstra.  Variational intrinsic control.  _arXiv preprint arXiv:1611.07507_ , 2016.
  * Haarnoja et al. [2018a] T. Haarnoja, V. Pong, A. Zhou, M. Dalal, P. Abbeel, and S. Levine.  Composable deep reinforcement learning for robotic manipulation.  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 6244–6251. IEEE, 2018a.
  * Haarnoja et al. [2018b] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _arXiv preprint arXiv:1801.01290_ , 2018b.
  * [23] W. Han, S. Levine, and P. Abbeel.  Learning compound multi-step controllers under unknown dynamics.  In _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6435–6442. IEEE.
  * Kakade et al. [2003] S. M. Kakade et al.  _On the sample complexity of reinforcement learning_.  PhD thesis, 2003.
  * Kearns and Singh [2002] M. Kearns and S. Singh.  Near-optimal reinforcement learning in polynomial time.  _Machine learning_ , 49(2-3):209–232, 2002.
  * Klyubin et al. [2005] A. S. Klyubin, D. Polani, and C. L. Nehaniv.  Empowerment: A universal agent-centric measure of control.  In _2005 IEEE Congress on Evolutionary Computation_ , volume 1, pages 128–135. IEEE, 2005.
  * Lee et al. [2019] L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov.  Efficient exploration via state marginal matching.  _arXiv preprint arXiv:1906.05274_ , 2019.
  * Levine et al. [2016] S. Levine, C. Finn, T. Darrell, and P. Abbeel.  End-to-end training of deep visuomotor policies.  _The Journal of Machine Learning Research_ , 17(1):1334–1373, 2016.
  * Mnih et al. [2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mohamed and Rezende [2015] S. Mohamed and D. J. Rezende.  Variational information maximisation for intrinsically motivated reinforcement learning.  In _Advances in neural information processing systems_ , pages 2125–2133, 2015.
  * Nachum et al. [2018] O. Nachum, S. S. Gu, H. Lee, and S. Levine.  Data-efficient hierarchical reinforcement learning.  In _Advances in Neural Information Processing Systems_ , pages 3303–3313, 2018.
  * Parr and Russell [1998] R. Parr and S. J. Russell.  Reinforcement learning with hierarchies of machines.  In _Advances in neural information processing systems_ , pages 1043–1049, 1998.
  * Pathak et al. [2017] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 16–17, 2017.
  * Pong et al. [2019] V. H. Pong, M. Dalal, S. Lin, A. Nair, S. Bahl, and S. Levine.  Skew-fit: State-covering self-supervised reinforcement learning.  _arXiv preprint arXiv:1903.03698_ , 2019.
  * Precup [2001] D. Precup.  Temporal abstraction in reinforcement learning.  2001\.
  * Racaniere et al. [2019] S. Racaniere, A. K. Lampinen, A. Santoro, D. P. Reichert, V. Firoiu, and T. P. Lillicrap.  Automated curricula through setter-solver interactions.  _arXiv preprint arXiv:1909.12892_ , 2019.
  * Rajeswaran et al. [2017] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ , 2017.
  * Salge et al. [2014] C. Salge, C. Glackin, and D. Polani.  Empowerment–an introduction.  In _Guided Self-Organization: Inception_ , pages 67–114. Springer, 2014.
  * Schmidhuber [1991] J. Schmidhuber.  Curious model-building control systems.  In _Proc. international joint conference on neural networks_ , pages 1458–1463, 1991.
  * Sharma et al. [2019] A. Sharma, S. Gu, S. Levine, V. Kumar, and K. Hausman.  Dynamics-aware unsupervised discovery of skills.  _arXiv preprint arXiv:1907.01657_ , 2019.
  * Silver et al. [2018] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  _Science_ , 362(6419):1140–1144, 2018.
  * Sukhbaatar et al. [2017] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam, and R. Fergus.  Intrinsic motivation and automatic curricula via asymmetric self-play.  _arXiv preprint arXiv:1703.05407_ , 2017.
  * Sutton et al. [1999] R. S. Sutton, D. Precup, and S. Singh.  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  _Artificial intelligence_ , 112(1-2):181–211, 1999.
  * Tang et al. [2017] H. Tang, R. Houthooft, D. Foote, A. Stooke, O. X. Chen, Y. Duan, J. Schulman, F. DeTurck, and P. Abbeel.  # exploration: A study of count-based exploration for deep reinforcement learning.  In _Advances in neural information processing systems_ , pages 2753–2762, 2017.
  * Tesauro [1995] G. Tesauro.  Temporal difference learning and td-gammon.  1995\.
  * Van Hasselt et al. [2016] H. Van Hasselt, A. Guez, and D. Silver.  Deep reinforcement learning with double q-learning.  In _Thirtieth AAAI conference on artificial intelligence_ , 2016.
  * Vecerik et al. [2017] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. Riedmiller.  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _arXiv preprint arXiv:1707.08817_ , 2017.
  * Vezhnevets et al. [2017] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg, D. Silver, and K. Kavukcuoglu.  Feudal networks for hierarchical reinforcement learning.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 3540–3549. JMLR. org, 2017.
  * Von Stackelberg [1934] H. Von Stackelberg.  _Marktform und Gleichgewicht_.  1934\.
  * Wang et al. [2019] R. Wang, J. Lehman, J. Clune, and K. O. Stanley.  Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions.  _arXiv preprint arXiv:1901.01753_ , 2019.
  * Wiering and Schmidhuber [1997] M. Wiering and J. Schmidhuber.  Hq-learning.  _Adaptive Behavior_ , 6(2):219–246, 1997.
  * Yahya et al. [2017] A. Yahya, A. Li, M. Kalakrishnan, Y. Chebotar, and S. Levine.  Collective robot reinforcement learning with distributed asynchronous guided policy search.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 79–86. IEEE, 2017.
  * Zhu et al. [2019] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar.  Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pages 3651–3657. IEEE, 2019.
  * Zhu et al. [2020] H. Zhu, J. Yu, A. Gupta, D. Shah, K. Hartikainen, A. Singh, V. Kumar, and S. Levine.  The ingredients of real world robotic reinforcement learning.  In _International Conference on Learning Representations_ , 2020.

## Appendix

We first describe the details of environments used in the paper, and next list
the hyperparameters used to train all the agents. All environments use the
MuJoCo 2.0 simulator.

###  Appendix A Environment Details

####  A.1 Ant Task

The Ant environment is equivalent to the standard gym Ant-v3 environment
except that the gear ratio is reduced from $(-120,120)$ to $(-30,30)$. The use
of this lower gear ratio is consistent with prior work [14]. The observation
space is the environment $qpos$ and $qvel$. The rewards are a weighted
combination of three terms: (1) a negative reward of the distance from the
center, (2) a positive reward for reaching a threshold from the center and
(3). The equation used is for (1) and (2) is given below:

$r(d)=e^{-d^{2}/2}+{\rm num\ goals\ completed}$

where $d$ is the $\ell-2$ norm of the distance from the origin. A negative
reward ($-300$) is also given to the agent for flipping over. If the reset
policy terminates the episode in this manner then the forward policy is not
executed. If however, the forward policy terminates the episode, the forward
policy receives the flip cost and the reset policy includes the negation of
this cost in its reward as described in Sec. 4.2. Each agent is given a time
horizon of $T_{\text{reset}}=T=200$. In this ant task, the game was run for
$7000$ episodes.

####  A.2 Ant-Waypoints Task

The aim of this task is to navigate between a set of pre-defined waypoints.
The waypoints defined are $[(0,5),(5,5),(5,0)]$. The reward structure is the
same as the constructed ant environment described in section A.1, conditioned
on the current waypoint.

The lower level controller was constructed from the reset and forward skills
learned from the adversarial game. The $(x,y)$ position of the state were re-
normalized so that the ant appeared to be at the origin. The higher level
controller was trained using an implementation of Double DQN [46] using the
2-D coordinates of the Ant as the state.

####  A.3 Ant-MediumMaze Task

We use the base environment from D4RL using a single
goal333https://github.com/rail-berkeley/d4rl [19]. The reward is the $\ell-2$
distance to the target position. The target position is the top right corner
and the bottom position is the bottom left corner. Each episode is of a length
30 where each step represents a single skill.

####  A.4 ManipulateFreeObject Task

The object is a 6 DoF three pronged object that is 15cm in diameter placed in
a workspace that is 30 cm $\times$ 30 cm box. We follow the goal configuration
of Zhu et al. [53], using $(x,y,\theta)=(0,0,-\frac{\pi}{2})$, and use the
same 15 initial configurations used for evaluation. Experiments were performed
using a state representation that comprised of the object $(x,y)$ position,
and $\sin$ and $\cos$ encoding of its orientation, the claw’s pose, and the
last action taken.

###  Appendix B Additional Experiments

####  B.1 Understanding the effects of resets and state representation choice
on Ant-Waypoints

Here, we provide an analysis of the effects of resets and state representation
choice on downstream task performance on the Ant-Waypoints task. We vary the
condition under which the policy is learned (i.e., reset or reset-free) and
the state representation (full state versus $(x,y)$-prior). In general, we
find that only with the availability of the $(x,y)$-prior and resets are prior
methods able to make significant progress (Fig. 7). The different relative
performance of the same skills on different downstream tasks (Waypoints vs
Maze) also suggests that learning a single action representation may not
always be ideal. This suggests future work that combines downstream
adaptation, or potentially mixing in primitive actions to allow for more
granular control.

Figure 7: A evaluation of the skills learned when varying the state
representation ($(x,y)$-prior or full state) and initial state distribution
(reset-free or with resets) on the Ant-Waypoints task. We normalize the return
of plot in terms of the best performing algorithm. Similar to the Ant-
MediumMaze task, we find that prior methods can only make progress on the task
when resets or additional information in the form of the $(x,y)$-prior is
available.

####  B.2 Effect of varying the number of skills on downstream performance

Figure 8: A plot showing the effect of varying the number of skills on the
Ant-Waypoints tasks. We find that increasing the number of skills improves
performance (16, yellow curve), the value used in our experiments (10, purple
curve) is far from the best, and many other values attain similar performance.
The number of skills used in our experiments were chosen to be roughly around
the order of previous work [14].

###  Appendix C Algorithm Details

The adversarial game comprises of a reset policy and a forward policy as
described in the paper. The forward policy receives rewards as given by the
environment at every step. The reset policy receives rewards from DIAYN at
every step and a game reward at the last step. This game reward is the sum of
the rewards received by the forward policy when played from the final state of
the reset policy trajectory. Each agent is given a fixed number of steps at
each turn. Environment termination was handled for individually for both
agents as follows. If the reset policy terminates, the forward policy does not
take any steps in the environment and is not given any reward. If forward
policy terminates, the reset policy is given rewards based on the forward
policy as normal.

In the following sections, curly brackets indicate that a parameter was
searched over.

####  C.1 DClaw Environment Parameters

The experiments conducted on the DClaw-ManipulateFreeObject domain was
conducted using the following parameters. The grid search parameters for DIAYN
were identical to those used in our method except where $\lambda=0$.

Hyperparameters | Value
---|---
Actor-critic architecture | FC(256, 256)
RND architecture | FC(256, 256, 512)
Classifier architecture | FC(256, 256)
Optimizer | Adam
Learning rate | {3e-3, 3e-4}
Classifier steps per iteration | 5
$\gamma$ | 0.99
$\lambda$ | {0.1, 0.5}
$r_{\text{skill}}$ scale | 0.1
target update | 0.005
Batch size | 256
Classifier batch size | 128

We follow [54] and provide 200 goal images for all experiments which are used
to learn a VICE [18] based reward function. We similarly omit the
$-\log\pi(\mathbf{s}|\mathbf{a})$. We also follow Burda et al. [7] and
normalize the prediction errors. In our implementation of LSR, we found used a
dimensionality of $\mathbf{z}$ of 2, implemented as a separate network as
opposed to one network. The reset controller [13] and R3L baseline [54] follow
the hyperparameters above. The reset controller’s results were averaged over
the 3 positions suggested by Zhu et al. [54].

####  C.2 Ant Environment Parameters

The experiments conducted on the Ant domain were conducted using the following
parameters. The grid search parameters for DIAYN were identical to those used
in our method except where $\lambda=0$.

Hyperparameters | Value
---|---
Actor-critic architecture | FC(256, 256)
Classifier architecture | FC(256, 256)
Optimizer | Adam
Learning rate | {1e-3, 2e-4}
Training steps per agent |  {$200,200$}
$\gamma$ | 0.99
Episode Length | 1000
$\lambda$ | 0.5
Number of skills | 10
$r_{\text{skill}}$ scale |  [$0.01,10$], chose 2
target update | 0.005
Batch size | 256
Classifier batch size | 256

####  C.3 Hierarchical Controller Parameters

The hierarchical controller is a Double-DQN [46] network that takes as input
the ${\mathbf{s}_{t}}$ and outputs a Q-function over the skills
$Q({\mathbf{s}_{t}},\mathbf{z})$.

Hyperparameters | Value
---|---
Architecture | FC(128)
Exploration fraction | $0.3$
Final epsilon j | 0.0
Learning rate $\alpha$ | 0.0001
Reward Scale | {0.1, 0.3}
Batch size | 256
Maximum horizon (T) | 30
Total epochs | 500
Replay Buffer size | {1000, 10,000}
Critic update frequency | {5, 10}
Critic updates per epoch | {100, 200}

An episode step corresponds to a 150 length rollout by the lower level
controller in the environment.

####  C.4 DADS

To allow for fair comparison, we run DADS without an $(x,y)$ prior. We
additionally discretize the continuous skill space into a space of 10 discrete
skills to allow for us to use the same meta-controller architecture. We used
the open source implementation of DADS [40] 444https://github.com/google-
research/dads with the following parameters:

Skill parameters |
---|---
Number of skills | 10
Skill type | Discrete
Dynamics parameters |
train steps | 8
learning rate | 3e-4
batch size | 256
Path length | 200
Agent parameters |
Architecture | FC(512)
Learning rate | 3e-4
Entropy | 0.1
Train steps | 64
Batch size | 256
$\gamma$ | 0.99


# Secluded and Putative Flipped Dark Matter and Stueckelberg Extensions of the
Standard Model

E. C. F. S. Fortes  V. Pleitez  F. W. Stecker

###### Abstract

We consider here three dark matter models with the gauge symmetry of the
standard model plus an additional local $U(1)_{D}$ factor. One model is truly
secluded and the other two models begin flipped, but end up secluded. All of
these models include one dark fermion and one vector boson that gains mass via
the Stueckelberg mechanism. We show that the would be flipped models provide
an example dark matter composed of “almost least interacting particles”
(ALIPs). Such particles are therefore compatible with the constraints obtained
from both laboratory measurements and astrophysical observations.

##  1 Introduction

The existence of dark matter (DM) requires physics beyond standard model (SM).
The so-called dark matter comprises approximately four fifths of the mass of
the universe as manifested by its gravitational influence (e.g., [1]).
However, at the present time its other fundamental characteristics are still
illusive. This is because, aside from its gravitational effects, no other
definitive physical evidence of its nature has been found. In particular,
severe limits on its interaction with standard model particles have been
obtained in the laboratory [2]–[5]). This dark matter could well be in hidden
sectors comprised of additional particles and forces [6]. Such particles may
have either minuscule or nonexistent interactions with SM particles.

Interesting models for physics beyond the SM can be constructed by considering
the possibility of an extra local $U(1)_{D}$ symmetry in addition to the
standard SM gauge symmetries. If the electric charge has a component on the
new factor, the model is called “flipped” otherwise, the model is called
“secluded”. In both cases, the vector boson related with the $U(1)_{D}$
symmetry factor may gain a mass from a spontaneously symmetry breaking through
a singlet scalar, and or by the Stueckelberg mechanism [7] if the vector boson
couples with a conserved vector current.

These models present two possibilities. In the secluded model the dark vector
does not have a projection on the photon field. Cross sections with SM
particles in the laboratory are negligible, but indirect astrophysical signals
of dark matter annihilation are potentially observable [8]. On the other hand,
in flipped models, the dark vector boson has a component along the photon
field and the electric charge operator changes with respect to its definition
in the standard model. In principle, this can produce examples of milli-
charged dark matter. In both cases, the vector boson related with the
$U(1)_{D}$ symmetry factor may gain a mass from a spontaneously symmetry
breaking through a singlet scalar, or by the Stueckelberg mechanism [7] if the
vector boson couples with a conserved vector current.

Here we will compare and discuss the implications of three models with the
gauge symmetry of the SM plus an extra local $U(1)_{D}$ factor with the
particle content of the dark sector consisting of a Dirac fermion and a vector
field, where the vector field gets mass only by the Stueckelberg mechanism.
One model is the secluded model in Ref. [8]. The other models start out
as“flipped” models [9, 10] but end up as secluded models after the constraint
on the possible neutrino electric charge is taken into account. This is why we
will refer to these models as “putative flipped models”. In the secluded
models the dark vector mixes in the kinetics term with the vector boson $B$
related with the $U(1)_{Y}$ factor of the SM before the spontaneous
electroweak symmetry breaking (SESB). In the first flipped model there is no
kinetic mixing but the vector boson $B$ gets a Stueckelberg mass after the
SESB mass, besides the mass obtained by the usual Higgs mechanism. This model
has been called Stueckelberg Extension of the Standard Model (StESM). In the
second flipped model, a kinetics mixing is added to the StESM.

Unlike in the case of the secluded models, it has been claimed that in the
flipped models the dark fermion is millicharged. However we will show here
that this is not true because in those models, either the neutrinos gain an
electric charge or the electric charges of the known fermions are modified.
This result imposes strong constraints on the mixing angles for the neutral
vector particles. For all practical purposes, this leaves only a single angle,
viz., the electroweak angle $\theta_{W}$.

We will show that the experimental constraints on the electric charge of the
neutrinos and on the measured electric charge of the electron imply that the
dark fermion must be almost as neutral as the neutrino. Such bounds do not
appear in the secluded model because in that model neutrinos do not gain an
electric charge and the charged fermions get the correct charge,
$q_{i}gs_{W}$. Thus in case of the StESM, with or without kinetics mixing, the
dark sector decouples from the SM particles. Therefore, in the StESM model the
dark matter fermion will not produce any observable effects in either
laboratory or astrophysical settings. The StESMs are in fact examples dark
matter composed of “almost least interacting particles” (ALIPs). Such
particles are compatible with the constraints obtained from both laboratory
measurements and astrophysical observations.

The outline of the paper is as follows: In Section 2 we briefly review the
secluded model. We present this model in a different way from that in [8] in
order to clarify the difference between this model and the flipped models
discussed in Sections 3 and 4. We consider the StESM without [9], and with the
kinetics [10], in Sections. 3 and 4 respectively. In Section 5 we show that in
fact both StESM models are examples of almost least interacting particles
(“ALIPs”). A discussion of the behavior of such particles is given in Section
6. Our conclusions appear in Section 7.

##  2 Secluded Stueckelberg Model

A form of the Stueckelberg model for secluded DM was proposed in Ref. [8].
Here we briefly review rewritten the neutral interactions of model in order to
compare this with the flipped models discussed in Secs. III and IV. We assume
a kinetic mixing between a dark vector boson, $V^{\prime}$, and the $U(1)_{Y}$
factor of the standard model (SM) with kinetic mixing with the $V^{\prime}$
occurring before spontaneous symmetry breaking being denoted by $B^{\prime}$.
We will denote $B$ and $V$ the vector fields associated with the $U(1)_{D}$
factor and the dark vector in the basis with diagonal kinetics terms. If we
start with

$\displaystyle\mathcal{L^{\prime}}_{kin}$ $\displaystyle\\!=\\!$
$\displaystyle\frac{M_{V}^{2}}{2}V^{\prime}_{\mu}V^{\prime\mu}\\!-\\!\frac{1}{4}V^{\prime}_{\mu\,\nu}V^{\prime\mu\nu}+\frac{g_{VB}}{2}V^{\prime}_{\mu\,\nu}B^{\prime\mu\nu}-\frac{1}{4}B^{\prime}_{\mu\nu}B^{\prime\mu\nu},$
(2.1)

making the transformation

$\left(\begin{array}[]{c}V^{\prime}\\\
B^{\prime}\end{array}\right)=\frac{1}{\sqrt{1-g^{2}_{VB}}}\left(\begin{array}[]{cc}1&0\\\
g_{VB}&\sqrt{1-g^{2}_{VB}}\\\ \end{array}\right)\left(\begin{array}[]{c}V\\\
B\end{array}\right),$ (2.2)

we obtain the diagonal mixing terms:

$\displaystyle\mathcal{L}_{kin}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\bar{M}_{V}^{2}V_{\mu}V^{\mu}-\frac{1}{4}V_{\mu\nu}V^{\mu\nu}-\frac{1}{4}B_{\mu\nu}B^{\mu\nu},$
(2.3)

where $\bar{M}^{2}_{N}=M^{2}_{V}/\sqrt{1-g^{2}_{VB}}$.

In the covariant derivatives acting on $H=(0,\,v/\sqrt{2})^{T}$ (unitary
gauge) we use $B^{\prime}$,

$\mathcal{D}_{\mu}^{H}=\mathbf{1}\partial_{\mu}+i\frac{g}{2}\vec{\tau}\cdot\vec{W}+i\frac{g^{\prime}}{2}\mathbf{1}B^{\prime}_{\mu},$
(2.4)

where $B^{\prime}$ is given in Eq. (2.2). After including the bare term in Eq.
(2.3) for the field $V$ and the spontaneously symmetry breaking with
$v\not=0$, we obtain the mass matrix in the basis $(V\,B\,W_{3})$ of Ref. [8]
:

$M^{2}_{FPS}=\frac{g^{2}v^{2}}{4c^{2}_{W}}\left(\begin{array}[]{ccc}\xi^{2}s^{2}_{W}+r&\xi
s^{2}_{W}&-\xi c_{W}s_{W}\\\ \xi s^{2}_{W}&s^{2}_{W}&-c_{W}s_{W}\\\ -\xi
c_{W}s_{W}&-c_{W}s_{W}&c^{2}_{W}\end{array}\right),$ (2.5)

where $c_{W}\equiv\cos\theta_{W},...$, $\xi=g_{VB}/\sqrt{1-g^{2}_{VB}}$,
$r=M^{2}_{V}/M^{2}_{Z}$, $M_{Z}=gv/2c_{W}$. The matrix in (2.5) is
diagonalized using the matrix

$\left(\begin{array}[]{c}V\\\ B\\\
W_{3}\end{array}\right)=\left(\begin{array}[]{ccc}c_{\alpha}&s_{\alpha}&0\\\
s_{W}s_{\alpha}&-s_{W}c_{\alpha}&c_{W}\\\
-c_{W}s_{\alpha}&c_{W}c_{\alpha}&s_{W}\end{array}\right)\left(\begin{array}[]{c}Z^{\prime}\\\
Z\\\ A\end{array}\right),$ (2.6)

where $\tan(2\alpha)=2\xi s_{W}/(1-\xi^{2}s^{2}_{W}-r)$.

As in Eq. (2.4) we have also to use $B^{\prime}$ in the covariant derivative
acting in the fermion sector. We first consider the lepton sector for both
left-handed and right-handed leptons:

$\mathcal{D}_{\mu}^{L_{L}}=\mathbf{1}\partial_{\mu}+i\frac{g}{2}\vec{\tau}\cdot\vec{W}-i\frac{g^{\prime}}{2}\mathbf{1}B^{\prime}_{\mu},\quad\mathcal{D}_{\mu}^{l_{R}}=\mathbf{1}\partial_{\mu}-i2\frac{g^{\prime}}{2}\mathbf{1}B^{\prime}_{\mu}.$
(2.7)

As in PDG, we parametrize the neutral currents as follows:

$\mathcal{L}_{NC}=-\frac{g}{2c_{W}}\bar{\psi}[\gamma^{\mu}(g^{\psi}_{V}-\gamma_{5}g^{\psi}_{A})Z_{\mu}+\gamma^{\mu}(f^{\psi}_{V}-\gamma_{5}f^{\psi}_{A})Z_{\mu}]\psi$
(2.8)

From (2.7) we get

$\begin{array}[]{c}\mathcal{L}^{leptons}_{NC}=-\frac{g}{2c_{W}}[\bar{\nu}_{L}\gamma^{\mu}(-W_{3}c_{W}+B^{\prime}s_{W})_{\mu}\nu_{L}\\\
\\\
+\bar{l}_{L}\gamma^{\mu}(W_{3}c_{W}+B^{\prime}s_{W})_{\mu}l_{L}+2s_{W}\bar{l}_{R}\gamma^{\mu}l_{R}B^{\prime}_{\mu}]\end{array}$
(2.9)

Neutrinos:

Using the same parametrization as in Eq. (2.8), (2.2) and (2.6) in (2.9), we
obtain

$\begin{array}[]{c}Q^{\nu}=0,\quad
g^{\nu}_{V}=g^{\nu}_{A}=\frac{1}{2}(c_{\alpha}-\xi s_{W}s_{\alpha}),\\\ \\\
\quad f^{\nu}_{V}=f^{\nu}_{A}=\frac{1}{2}(s_{\alpha}+\xi
s_{W}c_{\alpha}),\end{array}$ (2.10)

where $Q^{\psi}$ denote the electric charge of the particle $\psi$. Only when
$s_{\alpha}=0$, neutrinos have the SM interactions with $Z$ i.e.
$g^{\nu}_{V}=g^{\nu}_{A}=1/2$, but even in this case, they interacts with
$Z^{\prime}$ with a strength proportional to $\xi$. Unlike in other models
with kinetic mixing, neutrinos do not couple with the photon.

Charged Leptons:

As in the previous case we obtain that their electromagnetic interactions are
as those of SM, $e=gs_{W}=g^{\prime}c_{W}$ and the neutral coupling to $Z$ and
$Z^{\prime}$ are

$\displaystyle f^{l}_{V}=-\frac{1}{2}\left[(1-4s^{2}_{W})s_{\alpha}-3\xi
s_{W}c_{\alpha}\right],f^{l}_{A}=-\\!\\!\frac{1}{2}(s_{\alpha}+\xi
s_{W}c_{\alpha})$ $\displaystyle
g^{l}_{V}=\frac{1}{2}\left[(-1\\!\\!+\\!\\!4s^{2}_{W})c_{\alpha}\\!\\!-\\!\\!3\xi
s_{W}s_{\alpha}\right],g^{l}_{A}\\!\\!=-\\!\\!\frac{1}{2}\left(c_{\alpha}-\xi
s_{W}s_{\alpha}\right).$ (2.11)

Here $g^{l}_{V,A}$ coincides with the SM expressions when $s_{\alpha}=0$, as
in [8] if we use Eqs. (A.23) and Eqs. (A.24) for $g^{l}_{L},g^{l}_{R}$,
respectively and define $g^{l}_{V}=(1/2)(g^{l}_{L}+g^{l}_{R})$ and
$g^{l}_{A}=(1/2)(g^{l}_{L}-g^{l}_{R})$.

Quarks:

In this sector we have

$\begin{array}[]{c}\mathcal{D}_{\mu}^{Q_{L}}=\mathbf{1}\partial_{\mu}+i\frac{g}{2}\vec{\tau}\cdot\vec{W}+i\frac{1}{3}\frac{g^{\prime}}{2}\mathbf{1}B^{\prime}_{\mu},\\\
\\\
\mathcal{D}_{\mu}^{u_{R}}=\mathbf{1}\partial_{\mu}+i\frac{4}{3}\frac{g^{\prime}}{2}\mathbf{1}B^{\prime}_{\mu},~{}~{}\mathcal{D}_{\mu}^{d_{R}}=\mathbf{1}\partial_{\mu}-i\frac{2}{3}\frac{g^{\prime}}{2}\mathbf{1}B^{\prime}_{\mu}.\end{array}$
(2.12)

From Eqs. (2.12), using Eq. (2.2) and Eq. (2.6), we obtain

$\displaystyle
g^{u}_{V}=\frac{1}{2}\left[\left(1-\frac{8}{3}s^{2}_{W}\right)c_{\alpha}+\frac{5}{3}\xi
s_{W}s_{\alpha}\right],\;g^{u}_{A}=\frac{1}{2}\left(c_{\alpha}-\xi
s_{W}s_{\alpha}\right),$ $\displaystyle
f^{u}_{V}=\frac{1}{2}\left[-\frac{5}{3}\xi
s_{W}c_{\alpha}+s_{\alpha}-\frac{8}{3}s_{W}^{2}c_{\alpha}\right],\;f^{u}_{A}=\frac{1}{2}\left[\xi
s_{W}c_{\alpha}+s_{\alpha}\right],$ $\displaystyle
g^{d}_{V}=\frac{1}{2}\left[\left(-1+\frac{4}{3}s^{2}_{W}\right)c_{\alpha}-\frac{1}{3}\xi
s_{W}s_{\alpha}\right],\;g^{d}_{A}=-\frac{1}{2}\left(c_{\alpha}-\xi
s_{W}s_{\alpha}\right),$ $\displaystyle
f^{d}_{V}=\frac{1}{2}\left[\frac{1}{3}\left(c_{\alpha}\xi
s_{W}+4s_{W}^{2}s_{\alpha}\right)-s_{\alpha}\right],\;f^{d}_{A}=-\frac{1}{2}\left[c_{\alpha}\xi
s_{W}+s_{\alpha}\right].$ (2.13)

The angle $\alpha$ is constrained by the neutral current data. However,these
constraints are weaker than the astronomical observations that imply
$s_{\alpha}\simeq-4.7\times 10^{-7}$ [8].

The Dark Fermion:

Finally the neutral interactions of the dark fermion $\eta$, which has
couplings as $g_{\eta}\bar{\eta}\gamma^{\mu}\eta V_{\mu}$ and we have:

$\begin{array}[]{c}Q^{\eta}=0,\;\;\;g^{\eta}_{V}=\frac{g_{\eta}}{\sqrt{1-g^{2}_{VB}}}s_{\alpha},\;\;\;g^{\eta}_{A}=0,\\\
\\\
f^{\eta}_{V}=\frac{g_{\eta}}{\sqrt{1-g^{2}_{VB}}}c_{\alpha},\;\;\;f^{\eta}_{A}=0.\end{array}$
(2.14)

In this model the dark fermion and neutrino do not carry electric charge and
that the couplings of the dark fermion with $Z$ and $Z^{\prime}$ are vectorial
only. The model has a free angle, $\alpha$, and is possible to constraint it
as it was done in Ref. [8].

##  3 Flipped Stueckelberg model I

In this model a dark fermion and dark Stueckelberg vector boson coupled with a
conserved vector current are introduced, but this time without kinetic mixing.
Moreover, a Stueckelberg mass for the vector boson of SM $U(1)_{Y}$ factor is
added. Since there is no kinetic mixing, the field $B$ in the covariant
derivative is already the field coupled with the SM fields. This field gains a
Stueckelberg bare mass term. This model has been dubbed Stueckelberg extension
of the standard model (StESM), and it was proposed in Ref. [9].

When $\xi=0$ (no kinetic mixing) in Eq. (2.4), and introducing a bare mass
$M_{B}$ for the field $B$, we have the model of Ref. [9] in which the mass
matrix of the neutral vector bosons is given by

$M^{2}_{KN}=\frac{g^{2}v^{2}}{4c^{2}_{W}}\left(\begin{array}[]{ccc}r^{2}_{1}&r_{1}r_{2}&0\\\
r_{1}r_{2}&r^{2}_{2}+s^{2}_{W}&-c_{W}s_{W}\\\
0&-c_{W}s_{W}&c^{2}_{W}\end{array}\right)$ (3.1)

where $r_{1}=M_{V}/M_{Z}$, $r_{2}=M_{B}/M_{Z}$ [$M_{B}(M_{V})$] was denoted by
$M_{2}(M_{1})$ in Ref. [9]).

We assume that the known fermions are not charged under the dark $U(1)_{D}$,
hence the covariant derivatives are the usual ones and the mixing with the
dark sector is only through the mass matrix involving $V,B,W_{3}$. The matrix
in (3.1) is diagonalized by the following orthogonal matrix, O:

${\bf
O}=\left(\begin{array}[]{ccc}c_{\psi}c_{\phi}-s_{\theta}s_{\phi}s_{\psi}&-s_{\psi}c_{\phi}-s_{\theta}s_{\phi}c_{\psi}&-c_{\theta}s_{\phi}\\\
c_{\psi}s_{\phi}+s_{\theta}c_{\phi}s_{\psi}&-s_{\psi}s_{\phi}+s_{\theta}c_{\phi}c_{\psi}&c_{\theta}c_{\phi}\\\
-c_{\theta}s_{\psi}&-c_{\theta}c_{\psi}&s_{\theta}\end{array}\right),$ (3.2)

so that

$\left(\begin{array}[]{c}V\\\ B\\\ W_{3}\end{array}\right)={\bf
O}\left(\begin{array}[]{c}Z^{\prime}\\\ Z\\\ A\end{array}\right),$ (3.3)

where

$\tan\phi=\frac{M_{B}}{M_{V}},\quad\tan\psi=\frac{\tan\theta_{W}M^{2}_{W}}{c_{W}[M^{2}_{Z^{\prime}}(1-\tan^{2}\theta_{W})M^{2}_{W}]}\tan\phi.$
(3.4)

One of the angles in the matrix in (3.2) has to be, within the experimental
error, equal to the weak mixing angle, $\theta_{W}$.

Unlike the matrix in equation (2.6), in this case, $V$ has a component along
the photon field. This is a flipped model. The dark fermion and neutrinos will
get an electric charge and the known charged fermion will also get a small
non-standard electric charge. Explicitly

$\displaystyle
Q^{\nu}=-\frac{1}{2}gs_{W}(-1+c_{\phi}),\;\;\;Q^{\eta}=-g_{\eta}c_{W}s_{\phi},$
$\displaystyle
Q^{l}_{L}=-\frac{1}{2}gs_{W}(1+c_{\phi}),\;\;Q^{l}_{R}=-gs_{W}c_{\phi},$
$\displaystyle
Q^{u}_{L}=\frac{1}{2}gs_{W}\left(1+\frac{1}{3}c_{\phi}\right),\;\;Q^{u}_{R}=\frac{2}{3}gs_{W}c_{\phi},$
$\displaystyle
Q^{d}_{L}=-\frac{1}{2}gs_{W}\left(1-\frac{1}{3}c_{\phi}\right),\;\;Q^{d}_{R}=-\frac{1}{3}gs_{W}c_{\phi},$
(3.5)

where $gs_{W}=|e|$. Notice that only if $\phi=0$,
$Q^{l}_{L}=Q^{l}_{R}=-gs_{W}$ and $Q^{\nu}=0$. Similarly for quarks, only when
$\phi=0$ we obtain the usual electric charge for the known charged fermions.
On the other hand, astrophysical constraints on neutrino electric charge imply
$Q^{\nu}<10^{-19}|e|$ [11]. Stronger constraints come from $\beta$-decay:
$Q^{\nu}<10^{-21}|e|$ [12, 13]. In fact, in order to $Q^{\nu}$ be compatible
with the experimental upper limit on a possible neutrino electric charge then
$(1-c_{\phi})<2\times 10^{-21}$. This implies that $\phi\approx 0$ and
$M_{B}\approx 0$. In accordance with the definition in Eq. (3.4), the angle
$\psi\approx 0$ is also practically zero. Hence, the model has only one angle:
$\theta_{W}$.

We can calculate the neutral current couplings in the fermion sector using
still all the angles in (3.2).

Neutrinos:

$\displaystyle
g^{\nu}_{V}=g^{\nu}_{A}=\frac{1}{2}[(c_{W}^{2}+s^{2}_{W}c_{\psi})c_{\phi}-s_{W}s_{\psi}s_{\phi}],$
$\displaystyle
f^{\nu}_{V}=f^{\nu}_{A}=\frac{1}{2}\left[(s_{W}^{2}c_{\phi}+c_{W}^{2})s_{\psi}+s_{W}s_{\psi}c_{\phi}\right].$
(3.6)

Charged Leptons:

$\displaystyle
g^{l}_{V}=\frac{1}{2}[(1-3s^{2}_{W}c_{\phi}-s^{2}_{W})c_{\psi}+3s_{W}s_{\phi}s_{\psi})],$
$\displaystyle
g^{l}_{A}=\frac{1}{2}[(c^{2}_{W}+s^{2}_{W}c_{\phi})c_{\psi}-s_{W}s_{\phi}s_{\psi}],$
$\displaystyle
f^{l}_{V}=\frac{1}{2}\left[\left(c^{2}_{W}-3s^{2}_{W}c_{\phi}\right)s_{\psi}-3s_{W}c_{\psi}s_{\phi}\right],$
$\displaystyle
f^{l}_{A}=\frac{1}{2}\left[\left(c^{2}_{W}+s^{2}_{W}c_{\phi}\right)s_{\psi}+s_{W}c_{\psi}s_{\phi}\right].$
(3.7)

$Up$-quarks:

$\displaystyle
g^{u}_{V}=\frac{1}{2}\left[\left(c^{2}_{W}-\frac{5}{3}s^{2}_{W}c_{\phi}\right)c_{\psi}+\frac{5}{3}s_{W}s_{\phi}s_{\psi}\right],$
$\displaystyle
g^{u}_{A}=\frac{1}{2}\left[(c^{2}_{W}+s^{2}_{W}c_{\phi})c_{\psi}-s_{W}s_{\psi}s_{\psi}\right],$
$\displaystyle
f^{u}_{V}=\frac{1}{2}\left[\left(c^{2}_{W}-\frac{5}{3}s^{2}_{W}c_{\phi}\right)s_{\psi}-\frac{5}{3}s_{W}s_{\phi}c_{\psi}\right],$
$\displaystyle
f^{u}_{A}=\frac{1}{2}\left[(c^{2}_{W}+s^{2}_{W}c_{\phi})s_{\psi}+s_{W}s_{\phi}c_{\psi}\right]$
(3.8)

$Down$-quarks:

$\displaystyle
g^{d}_{V}=\frac{1}{2}\left[\left(-c^{2}_{W}+\frac{1}{3}s^{2}_{W}c_{\phi}\right)c_{\psi}-\frac{1}{3}s_{W}s_{\phi}s_{\psi}\right],$
$\displaystyle
g^{d}_{A}=-\frac{1}{2}\left[\left(c^{2}_{W}+s^{2}_{W}c_{\phi}\right)c_{\psi}-s_{W}s_{\phi}s_{\psi}\right],$
$\displaystyle
f^{d}_{V}=-\frac{1}{2}\left[\left(c^{2}_{W}+s^{2}_{W}c_{\phi}\right)s_{\psi}+s_{W}s_{\phi}c_{\psi}\right],$
$\displaystyle
f^{d}_{A}=-\frac{1}{2}\left[\left(c^{2}_{W}+s^{2}_{W}c_{\phi}\right)s_{\psi}+s_{W}s_{\phi}c_{\psi}\right].$
(3.9)

The Dark Fermion:

$\displaystyle
g^{\eta}_{V}=-g_{\eta}(s_{\psi}c_{\phi}+s_{W}s_{\phi}c_{\psi}),\;\;g^{\eta}_{A}=0,$
$\displaystyle
f^{\eta}_{V}=g_{\eta}(c_{\psi}c_{\phi}-s_{W}s_{\phi}s_{\psi}),\;\;f^{\eta}_{A}=0.$
(3.10)

Notice that the dark fermion has only vector interactions. However, if we now
take into account the neutrino electric charge constraint discussed above,
$\phi=0$ in order for $Q^{\nu}$ in equation (3.5) to be compatible with the
upper experimental bound limits. Because of the relations in (3.4), $\psi=0$
as well. In this situation all of the coupling constants $g_{V},g_{A}$ thus
reduce to their SM values. Therefore the dark fermion couples only with
$Z^{\prime}$ and $Q^{\eta}=0$. Hence there is no millicharged dark matter in
the model of Ref. [9]. The $B$ vector of the standard model cannot receive (if
any) a considerable Stueckelberg bare mass.

Above we assumed that $\theta_{W}=\theta$, another possibility used in Ref.
[9] is $\tan\theta=\theta_{W}c_{\phi}$. Below we show that, in fact,
$\tan\theta=\tan\theta_{W}c_{\phi}$ implies $\tan\theta=\tan\theta_{W}$.

Let us again consider the electric charge of the known leptons in the flipped
models of Sec. 3 and 4. As in the SM, the interaction of leptons with the
vector $W_{3}$ and $B$ fields are given by

$\mathcal{L}_{l}=-\frac{g}{2}\left[\bar{\nu}_{L}(W_{3}-\tan\theta_{W}B)\bar{\nu}_{L}+\bar{l}_{L}(-W_{3}-\tan\theta_{W}B)\bar{l}_{L}\right]+g_{Y}\bar{l}_{R}(B)l_{R},$
(3.11)

where $g_{Y}$ is the gauge coupling of the $U(1)_{Y}$ factor of the SM. From
(3.2) we obtain

$\displaystyle
W_{3}=-c_{\theta}s_{\psi}Z^{\prime}-c_{\theta}c_{\psi}Z+s_{\theta}A,$
$\displaystyle
B=(c_{\psi}s_{\phi}+s_{\theta}c_{\phi}s_{\psi})Z^{\prime}+(-s_{\psi}s_{\phi}+s_{\theta}c_{\phi}c_{\psi})Z+c_{\theta}c_{\phi}A.$
(3.12)

Using equations (3.12) and (3.11) we find that the electric charge of the
leptons are given by

$\displaystyle Q^{\nu}=-\frac{g}{2c_{W}}\cdot
c_{W}c_{\theta}(\tan\theta-\tan\theta_{W}c_{\phi}),$ $\displaystyle
Q^{l}_{L}=\frac{g}{2c_{W}}\cdot
c_{W}c_{\theta}(\tan\theta+\tan\theta_{W}c_{\phi}),\quad
Q^{l}_{R}=g\tan\theta_{W}c_{\theta}c_{\phi},$ (3.13)

Here we have two possibilities.

  1. a)

$\theta=\theta_{W}$ and in this case we have in Eq. (3.13)

$\displaystyle Q^{\nu}=-\frac{gs_{W}}{2}(1-c_{\phi}),$ $\displaystyle
Q^{l}_{L}=\frac{g}{2}s_{W}(1+c_{\phi}),\quad Q^{l}_{R}=gs_{W}c_{\phi},$ (3.14)

as in Eq. (3.5). Hence, as we said $\phi$ is for all practical purposes zero.

Or we can choose

  2. b)

$\tan\theta=\tan\theta_{W}c_{\phi}$. However, in case b) we have two
possibilities: doing $\tan\theta_{W}c_{\phi}\to\tan\theta$ in Eq. (3.13), we
obtain

$Q^{\nu}=0,\quad Q^{l}_{L}\equiv Q^{l}_{R}=gs_{\theta},$ (3.15)

or, doing in (3.13) $\tan\theta\to\tan\theta_{W}c_{\phi}$ we obtain

$Q^{\nu}=0,\quad Q^{l}_{L}=Q^{l}_{R}=gc_{\theta}\tan\theta_{W}c_{\phi}.$
(3.16)

##  4 Flipped Stueckelberg model II

It is possible to add a kinetic mixing term as the $g_{VB}$ term in Eq. (2.1)
to the flipped model I that was considered in Sec. 3. This was done in Ref.
[10], where the coupling constant $g_{VB}$ was denoted by $\delta$. In
principle, such models have two parameters in the dark vector sector: a
kinetic mixing, as in the secluded model in Sec. 2, and a bare Stueckelberg
mass $M_{B}$ as in the flipped I model in Sec. 3.

In this case the orthogonal matrix in Eq. (3.2) is still the same and all the
results in Eqs. (3.5) – (3.10) are valid. Hence all the results in Sec. 3 are
still valid. The neutrino and the dark fermion $\eta$ would have an electric
charge as in Eqs. (3.5). All of the couplings in Eqs. (3.6) – (3.10) are the
same as well.

In this model, as in the previous model, the experimental constraints of a
possible neutrino electric charge imply that $\phi\approx 0$. However, the
definition in Eq. (3.4) is replaced by [10]:

$\tan\phi=\bar{\epsilon},\;\;\;\tan
2\psi=\frac{2s_{W}M^{2}_{Z}\tan\phi}{M^{2}_{V}-M^{2}_{Z}+(M^{2}_{V}-M^{2}_{Z}-M^{2}_{W})\bar{\epsilon}^{2}}$
(4.1)

where $\bar{\epsilon}=r_{1}-\delta$ (this is the parameter in the kinetics
mixing in the notation of Ref. [10]) and as in the case of the model in the
previous section, when $\phi=0$, then $\psi=0$ and the dark sector decouple
from the SM particles. Thus, this model actually becomes secluded and is
similar to that in Sec. 2. Thus, there is no millicharged dark matter in this
case. However, in this case $\tan\phi=0$ means, according to Eq. (4.1) that
$\epsilon=\delta$ and then it is possible to have a bare mass for the vector
boson $B$ but the dark sector decouple from the SM particles.

Notice that (3.15) and (3.16) are compatible with each other if, and only if,
$\theta=\theta_{W}$ and $c_{\phi}=0$ (or extremely small). Hence, we can from
the very beginning use $\theta=\theta_{W}$ as we have done in Secs. 3 and 4.

##  5 LIPs and ALIPs

In the dark sector we can distinguish two cases of DM models that are
consistent with the observational astrophysical and laboratory constraints:
(1) particles that interact with the particles of the standard model only
through gravitational interactions, labeled Least Interacting Particles (LIPs)
[15], and (2) particles which we call Almost LIPs (ALIPs). ALIPs interact with
the particles of the standard model only by gravitational interactions, but
also interact with each other through new interactions and which may have
their own symmetry (see Table 1).

A true LIP particle is one that interacts only gravitationally. Examples of
true LIPs are primordial black holes. Such particles, in order to make up the
dark matter, must be stable against Hawking radiation [16] up until the
present epoch. Thus their mass must be greater than $10^{15}$ g. Astrophysical
constraints on such primordial black holes are discussed in Ref. [17] and
[18].

In the models considered in Secs. 3 and 4, the dark sector consists of a
massive vector boson $Z^{\prime}$ and a Dirac fermion $\eta$ that couples to
the standard model particles mainly via gravitation. In such models the fields
in the dark sector can explain the matter relic density but no direct effects
can be observed in astrophysical processes or in laboratory. The Kors and Nath
model [9] is an ALIP model, since the angle $\phi$ is not exactly zero but
rather is a small number, taken to be compatible with the upper bound of a
possible neutrino electric charge. This model has no angles besides the weak
angle, $\theta_{W}$, in the visible sector.

Let us assume only that $m_{V}>m_{\eta}$. In the early universe
$Z^{\prime}+Z^{\prime}\leftrightarrow\eta+\bar{\eta}$, and $Z^{\prime}$ and
$\eta$ were in equilibrium. Then at a given later time, say $t_{0}$, only
$V+V\to\eta+\bar{\eta}$ could occur and the fermion $\eta$ decoupled. The rate
of this reaction depends on the coupling constant $g_{\eta}$ and the masses of
the fermion and vector bosons. When $\Gamma^{-1}>H^{-1}(t_{0})$, $\eta$
decoupled and its number remained constant thereafter, thus accounting for the
27% of cold dark matter (CDM) observed by Planck Collaboration at present [1].

Denoting $\rho_{{}_{crit}}$ the critical density of the universe

$\Omega_{{}_{CDM}}=\frac{\rho_{{}_{CDM}}}{\rho_{crit}}=0.258,$ (5.1)

being

$\rho_{{}_{crit}}=1.054\times
10^{-5}\,h^{2}\,\frac{\textrm{GeV}}{\textrm{cm}^{3}}=0.48\times
10^{-5}\,\frac{\textrm{GeV}}{\textrm{cm}^{3}},$ (5.2)

where we have used $h=0.678$ [19].

From (5.1) and (5.2) we obtain

$\displaystyle\rho_{{}_{CDM}}$ $\displaystyle=$ $\displaystyle 1.24\times
10^{-6}\,\frac{\textrm{GeV}}{\textrm{cm}^{3}}=1.24\,\frac{\textrm{GeV}}{\textrm{m}^{3}},$
(5.3) $\displaystyle=$ $\displaystyle 2.22\times
10^{-24}\frac{\textrm{g}}{\textrm{m}^{3}}.$

This shows that an ALIP with mass of 1.24 GeV per cubic meter can explain the
observed relic density. The ALIP mass is constrained by the unitary limit for
a Dirac fermion

$\Omega_{tot}h^{2}\sim 0.5\geq 8.7\times 10^{-6}[m_{\chi}({\rm TeV})]^{2},$
(5.4)

which gives an upper limit on an ALIP Dirac fermion mass of $\sim 240$ TeV
[20]. This is in contrast to LIP particles that can much larger masses.

It is important to notice the difference between the model in Sec. 2 and those
in Sec. 4. In both there is a mixing in the vector kinetic terms, but as in
the former, the kinetic terms are diagonalized before the spontaneous symmetry
breaking (SSB). On the other hand, in the model of Sec. 4 [9, 10], the
diagonalization of the kinetic mixing is done after the SSB,
$M^{2}_{FLN}=K^{T}M^{2}_{KN}K$ where $K$ is the $3\times 3$ extension of the
matrix defined in Eq. (2.2) and $M^{2}_{KN}$ is the matrix in Eq. (3.1). The
result is the matrix of Eq. (A2) of Ref. [10] that has determinant zero. In
fact, if in the model of Sec. 4 the kinetics mixing had occurred before the
SSM, the mass matrix of the neutral vector bosons would be

$M^{2}_{FLN}=\frac{g^{2}v^{2}}{4c^{2}_{W}}\left(\begin{array}[]{ccc}\xi^{2}s^{2}_{W}+r^{2}_{1}&\xi
s^{2}_{W}+r_{1}r_{2}&-\xi c_{W}s_{W}\\\ \xi
s^{2}_{W}+r_{1}r_{2}&s^{2}_{W}+r^{2}_{2}&-c_{W}s_{W}\\\ -\xi
c_{W}s_{W}&-c_{W}s_{W}&c^{2}_{W}\end{array}\right),$ (5.5)

that has a determinant equal to $-c^{2}_{W}(-4+r_{1})r_{1}r^{2}_{2}$ and the
photon acquires a nonzero mass.

##  6 Small Scales

In this section we will give a brief preview about the model behavior under
small scales. We have analysed the self-interacting dark matter case. For this
calculation we have considered the process $\eta+\eta\rightarrow\eta+\eta$. In
principle this calculation is valid for the Flipped Stueckelberg model I and
II since we leave $g_{\eta}$ as a generic coupling.

With this process we can have a good estimative of the other related processes
like $\eta+\bar{\eta}\rightarrow\eta+\bar{\eta}$ since in this case we have
the crossing symmetry.

Lets consider now the process $\eta+\eta\rightarrow\eta+\eta$. We had taken
into account the statistical factor for identical particles and made the
substitution for $s=4m_{\eta}^{2}+m_{\eta}^{2}v^{2}$ and we had taken the
limit of $v\rightarrow 0$. In the Eq. (6.1) we express $\sigma/m$. This number
must be in the 0.5 cm2/g $<\sigma/m<$ 1 cm2/g range in order to solve the
core-cusp and the too-big-to-fail problem on small scales. These limits are in
agreement with the other astrophysical constraints on larger scales.

$\displaystyle\frac{\sigma}{m}$ $\displaystyle=$
$\displaystyle\frac{g_{\eta}^{4}m_{\eta}}{16\pi M_{ZP}^{4}}.$ (6.1)

So, for these type of model we should use Eq. (6.1) to give a estimative of
the cross-section over the mass.

Now we consider the dwarf galaxy problem. For the central region of a typical
dwarf galaxy according to Ref. [23] we should have $\rho_{DM}\sim
0.1M_{\odot}$/pc3 and $v\sim 50$ km/s [24]. If we convert this constraint to
units of g/m3 and compare this number with the mass density obtained in
equation (5.3) for our ALIP model, we find the required $\rho_{DM}\sim
6.73\times 10^{-18}$ g/m3 is six orders of magnitude larger than the number
obtained in equation (5.3). Thus, this model cannot explain the missing
satellite problem.

##  7 Conclusions

Of the three models that we have considered here, only the secluded model of
Sec. 2, i.e., in Ref. [8], may induce effects that can be observed by
astronomers or in laboratory. However, recent experiments trying to observe
direct effect of WIMPS, viz., XENON1T [2], LUX [4], and PANDA-II [5], have not
obtained any evidence for dark matter particles, with the upper limits on the
spin-independent cross section reaching values as low as $\sim 10^{-46}$ cm2.
[21]

Astrophysical searches for dark matter (DM) annihilation into $\gamma$-rays
have also been somewhat disappointing. While the $\gamma$-ray excess in the
galactic center region (GCE) can have a DM annihilation interpretation (e.g.,
[8, 22]), such an interpretation appears to be somewhat in conflict with upper
limits from dwarf spheroidal satellite galaxies, at least if the main
annihilation models are through the $t\bar{t}$ and $\tau\bar{\tau}$ channels
[25]. A strong alternate interpretation of the GCE is that it is made up of
emission from point sources such as pulsars [26, 27].

Given these present negative empirical results, the possible existence of dark
matter particles that mainly, or possibly only interact gravitationally with
SM particles (ALIPs or LIPs) is an interesting alternative theoretical option
to explore, as we have considered here.

## Acknowledments

VP would like to thank CNPq for partial support and is also thankful for the
support of FAPESP funding Grant No. 2014/19164-6 and to G. E. A. Matsas for
useful discussions.

## References

  * [1] P. A. R. Ade et al. [Planck Collaboration],
  * [2] E. Aprile et al. [XENON Collaboration], [arXiv:1705.06655] [astro-ph.CO].
  * [3] E. Aprile et al. [XENON100 Collaboration], Phys. Rev. D 94, no. 12, 122001 (2016)
  * [4] D. S. Akerib et al. [LUX Collaboration], Phys. Rev. Lett. 118, no. 2, 021303 (2017).
  * [5] C. Fu et al. [PandaX-II Collaboration], Phys. Rev. Lett. 118, no. 7, 071301 (2017).
  * [6] M. Pospelov, A. Ritz, and M. Voloshin, Physics Letters B 662, 53 (2008).
  * [7] E. C. G. Stueckelberg, Helv. Phys. Acta 11, 225 (1938).
  * [8] E. C. F. S. Fortes, V. Pleitez and F. W. Stecker, Astropart. Phys. 74, 87 (2016).
  * [9] B. Kors and P. Nath, Phys. Lett. B 586, 366 (2004).
  * [10] D. Feldman, Z. Liu and P. Nath, Phys. Rev. D 75, 115001 (2007).
  * [11] A. I. Studenikin and I. Tokarev, Nucl. Phys. B 884, 396 (2014).
  * [12] M. Marinelli and G. Morpurgo, Phys. Lett. B 137, 439 (1984).
  * [13] J. Baumann, J. Kalus, R. Gahler and W. Mampe, Phys. Rev. D 37, 3107 (1988).
  * [14] S. Davidson, S. Hannestad and G. Raffelt, JHEP 0005, 003 (2000).
  * [15] G. E. A. Matsas, J. C. Montero, V. Pleitez and D. A. T. Vanzella, Contributions to Topics in Theoretical Physics: Festschrift for A. H. Zimerman, Instituto de Física Teórica/UNESP- São Paulo, 1998, Edited by H. Aratyn, L. A. Ferreira and J. F. Gomes, p. 219; hep-ph/9810456 (1998).
  * [16] S. Hawking, Mon. Not. R. astr. Soc. 1̱52 75 (1971).
  * [17] B. J. Carr, Phys. Rev. D 81, 104019 (2010).
  * [18] S. J. Clark et al. Phys. Rev. D 95, 083006 (2017).
  * [19] C. Patrignani et al. [Particle Data Group], Chin. Phys. C 40, no. 10, 100001 (2016).
  * [20] K. Griest and M. Kamionkowski, Phys. Rev. Lett. 64, 615 (1990).
  * [21] M. Ablikim et al. [BESIII Collaboration], Phys. Lett. B 774, 252 (2017).
  * [22] Abazajian, K. N. and Keeley, R. E., Phys. Rev. D 93, 083514, (2016),
  * [23] S. H. Oh, W. J. G. de Blok, E. Brinks, F. Walter and R. C. Kennicutt, Jr, Astron. J. 141, 193 (2011) doi:10.1088/0004-6256/141/6/193 [arXiv:1011.0899 [astro-ph.CO]].
  * [24] S. Tulin and H. B. Yu, arXiv:1705.02358 [hep-ph].
  * [25] Albert, A. et al. Fermi-LAT and Collaborations, Astrophys. J., 834, 110 (2017), [arXiv:1611.03184][astro-ph.HE].
  * [26] Bartels, R. and Krishnamurthy, S. and Weniger, C., Physical Review Letters 116, 051102 (2016).
  * [27] M. Ackermann et al. [Fermi-LAT Collaboration], The Fermi Galactic Center GeV Excess and Implications for Dark Matter, Astrophys. J. 840, no. 1, 43 (2017)

Particle/Interactions | Gravitational | weak | electromagnetic | strong | dark interactions
---|---|---|---|---|---
Quarks | + | + | + | + | -
Charged leptons | + | + | + | - | -
Neutrinos | + | + | - | - | -
ALIPS | + | - | - | - | +
LIPS | + | - | - | - | -
Table 1: Particle types and their interactions.