# Robust Multi-Modal Policies for Industrial Assembly via Reinforcement
Learning and Demonstrations: A Large-Scale Study

Jianlan Luo* Oleg Sushkov* Rugile Pevceviciute* Wenzhao Lian Chang Su

Mel Vecerik Ning Ye Stefan Schaal Jon Scholz

Google X
Google Deepmind
*Equal contribuition  Jianlan Luo*
Wenzhao Lian
Ning Ye  Google X
Google X
Google X
Oleg Sushkov*
Chang Su
Stefan Schaal  Google Deepmind
Google X
Google X
Rugile Pevceviciute*
Mel Vecerik
Jon Scholz  Google Deepmind
Google Deepmind  Google Deepmind

###### Abstract

Over the past several years there has been a considerable research investment
into learning-based approaches to industrial assembly, but despite significant
progress these techniques have yet to be adopted by industry. We argue that it
is the prohibitively large design space for Deep Reinforcement Learning (DRL),
rather than algorithmic limitations per se, that are truly responsible for
this lack of adoption. Pushing these techniques into the industrial mainstream
requires an industry-oriented paradigm which differs significantly from the
academic mindset. In this paper we define criteria for industry-oriented DRL,
and perform a thorough comparison according to these criteria of one family of
learning approaches, DRL from demonstration, against a professional industrial
integrator on the recently established NIST assembly benchmark. We explain the
design choices, representing several years of investigation, which enabled our
DRL system to consistently outperform the integrator baseline in terms of both
speed and reliability. Finally, we conclude with a competition between our DRL
system and a human on a challenge task of insertion into a randomly moving
target. This study suggests that DRL is capable of outperforming not only
established engineered approaches, but the human motor system as well, and
that there remains significant room for improvement. Videos can be found on
our project website:https://sites.google.com/view/shield-nist.

**footnotetext: These authors contributed equally ††Email: {jianlanluo,
sushkov, rugile, wenzhaol, suchang, vec, nye, sschaal<EMAIL_ADDRESS>
##  I INTRODUCTION

Since the rise of deep-learning there has been an increasing interest in using
learning-based methods to solve industrial manipulation tasks. The two
dominant families of industry-inspired tasks using deep-learning today are
mixed bin-picking, which poses a significant perception challenge, and
insertion, which emphasizes contact-rich control and multi-modal feedback.
While mixed-picking has seen some adoption in industry, the stigma of
complexity, brittleness, and long training times for reinforcement-learning
(RL) systems has hampered their adoption for industrial insertion.

Figure 1: An overview of our method: (1) gather human demonstration data by
teleportation, (2) robot executes its current policy from the latest snapshot,
(3) human perform occasional on-policy correction if necessary, (4)
asynchronous training of an improved DDPGfD agent:SHIELD.

Instead, most industrial insertion robots today operate in constrained
environments and execute sequences of manually pre-programmed force-control or
vision-control primitives. This approach imposes a barrier to providing higher
levels of autonomy at a reasonable cost, since each manipulation skill
requires significant time and effort to engineer. As a result, entire sectors
of low-volume manufacturing are precluded from accessing the benefits of
robotic automation.

Deep reinforcement learning (DRL) offers the potential to autonomously train
motor skills with little or no additional engineering effort. However, these
methods have not been pushed to industry-acceptable standards, nor
systematically evaluated against industry acceptable benchmarks. Previous
attempts to demonstrate learned insertion tasks [1, 2, 3, 4, 5, 6, 7] have
varied wildly in the task definition, action space, connector type, and sensor
observations. Fortunately, the recently introduced NIST assembly benchmark [8]
provides an externally defined set of representative industrial assembly tasks
with clearly defined metrics.

Inspired by the NIST benchmark, we argue that there are three key attributes
for DRL to succeed in industrial robotic settings:††Safety is often cited as a
fourth factor but is omitted from this list because, in the “last-inch”
regimes we consider here, safety can be attained by an appropriate
impedance/admittance controller. The lack of compliant industrial robots is
thus another significant obstacle to deploying DRL in the real world, but is
not addressed by this work.

  1. 1.

Efficient. It must be off-policy, and have a proper mechanism for injecting
suitable priors into the learning process, such that one general RL algorithm
can quickly adapt to new tasks by adapting task-specific priors. In this paper
we argue that human demonstration is a simple and natural way to provide such
prior information.

  2. 2.

Economical. The algorithm must bring value to the automation process. This can
be done by enabling solutions to problems that are infeasible with existing
techniques while being simple enough to deploy in practice.

  3. 3.

Evaluated. The resulting algorithm has to be evaluated extensively on
industrial benchmarks using industry-relevant metrics such as reliability and
cycle time. Generalization must be evaluated in a fair and convincing manner
against state-of-the-art industry solutions, and not just straw-man
“classical” approaches.

These “three E’s of industrial reinforcement learning” are critical for any
method to be viable in real-world settings. With these in mind, we present
Super-Human InsErtion using Learning from Demonstration (SHIELD), a collection
of design choices for the well-known DDPGfD algorithm which reflect several
years of experimentation on industrial-insertion settings. These choices are
individually minor, but collectively produce a system that, to our knowledge,
significantly outperforms any published learning approach to industrial
insertion in terms of speed and robustness to variability in scene
configuration.

The primary contributions of this paper can be summarized as follows:

  1. 1.

We conduct the first large scale systematic evaluation of an RL algorithm on
an industrial robotic manipulation benchmark (NIST assembly board challenge),
showing high reliability and robustness: 99.8% out of 13K trials.

  2. 2.

We describe in detail the SHIELD design choices which allow DDPGfD to attain
these results.

  3. 3.

We open-source a dataset containing all interaction data collected during our
NIST experiments to support future work on machine learning for industrial
manipulation.

  4. 4.

We present two challenge tasks and a comparison against human performance
(actual, not robot tele-operation) to demonstrate potential capabilities of
this paradigm: a) HDMI cable insertion into a moving socket, and b) blind key-
insertion into a randomly-positioned lock, requiring searching “in the dark”.

##  II PROBLEM STATEMENT

We aim to systematically evaluate our method in an industrial robotic
benchmark and beyond. To this end we consider three representative assembly
tasks in the NIST assembly board as shown in Fig.2. In each of these tasks,
the goal is to insert an industrial connector (e.g. USB), into a corresponding
socket.

To test the generalization capability of the learned polices, we ask the agent
to perform connector insertions at new locations and orientations using the
previously trained policies without any fine-tuning. For each of these tasks,
the agent was trained with three different input modalities: 1) proprioceptive
information from the robot (Tool Center Point (TCP) pose and velocity) 2)
proprioceptive information and wrist force/torque measurement 3)
proprioceptive information, wrist force/torque measurement and wrist camera
images. The agent is able to solve these tasks under all these modalities, and
perform zero-shot transfer to new connector locations and orientations. In
particular for policies trained with vision, we further test their
generalization capabilities by randomly moving the NIST board by hand during
the insertion process.

Figure 2: NIST robotic challenge board setup: (a) Robot inserting a connector,
(b) Three representative connectors.

We also consider two more challenging tasks as shown in Fig.7 and 9. In the
first task, the robot is asked to learn a policy to insert an HDMI connector
into a socket that is held by another robot that moves along a randomized
trajectory during the insertion process. In this task, the agent was not only
expected to solve it reliably despite the moving target, but also to solve it
as quickly as a human could, which is a critical threshold in the
manufacturing industry.

For the second task, we consider a key-lock insertion with situations where no
visual information is available; these are typical scenarios in today’s
manufacturing sites. The agent has to learn an appropriate searching strategy
solely based on proprioceptive information and wrist force/torque measurement.
This task is different and more difficult than the ones on the NIST board,
which also use the same input modality: 1) the lock and key are more delicate
pieces of hardware, and as they are made from metal they don’t provide any
material compliance; thus requiring even finer force-based motion to complete
the task. 2) the starting position of the key is randomized and largely off
the lock, which is not fully observable to the policy. In this case, the agent
must learn to search for the keyhole position on the surface by maximum use of
sensor readings.

##  III RELATED WORK

[3, 2] showed that it was possible to train a visual+force feedback controller
from scratch to solve deformable connector insertion tasks in under 1 hr. This
method was built upon off-policy RL from demonstration (specifically DDPGfD)
and used unsupervised pretrained visual features (VAE) to increase sample
efficiency. The main contribution of this approach was to demonstrate that a
purely learning-based solution to connector insertion was practical without
relying on simulation. However, results were only shown on synthetic
3D-printed parts, rather than real industrial connectors.

[6] introduces a sim2real approach that uses meta-learning to train an agent
in simulation to quickly adapt to errors at (meta) test time, and demonstrated
that this fast-adaptation ability transfers to the real-world. However, the
setting was limited to 3-dim position-only actions, a 3mm workspace, and used
a height-based reward function that only works in the immediate neighborhood
of the socket.

[5] describes a method which combines a model-based RL algorithm (iLQG) with
an operational space force controller to solve high-precision insertion tasks.
Using a guided policy search method they also train a neural network
controller which is designed to explicitly consider force/torque measurements.
The results show that this controller is able to generalize and solves peg-in-
hole insertion tasks under unseen initial configurations of the prop
positions. However, this method does not demonstrate a very high reliability
when generalizing to unseen peg-hole displacements.

[9] introduces a method where a control policy is defined as the superposition
of a user defined controller and a learned residual RL policy. The algorithm
delegates the difficult parts of a task (such as contacts and external object
dynamics) to a learned RL policy. This allows to approach problems that are
common in manufacturing and are difficult to solve using conventional methods.
The algorithm is demonstrated on an insertion task of a block between two
other freestanding blocks. The results show that the learned policy is able to
complete insertions when the blocks are displaced and the scripted policy
fails. However, they do not demonstrate high robustness and reliability and
the selected task does not require the policy to be as precise as industrial
connector insertion.

In principle any of these approaches is capable of competitive performance vs.
scripted baselines. However, we pursue DDPGfD due to its comparative
simplicity, which is an important factor when attempting to search the design
space for our setting.

##  IV PRELIMINARIES

We consider all tasks here that can be described as moving already-grasped
objects to their goal poses. We use a sparse reward system: 1) in the case
without vision, the agent will get a binary reward at the termination of one
trial episode: it will get a reward of one if its final position is within
some tolerance of the target location, otherwise it will get a reward of zero;
2) in the case with vision inputs, we train an additional visual reward
classifier to assign this binary reward. The goal of these tasks can be
measured as maximizing such collected rewards. We make no particular
assumptions about the encountered dynamics during tasks, especially during
contacts. All dynamics need to be learned by the robot from interaction with
the environment. Let ${\mathbf{s}}_{t}$ and ${\mathbf{a}}_{t}$ denote robot
states and actions respectively;
${\mathbf{r}}({\mathbf{s}}_{t},{\mathbf{a}}_{t})$ be the reward function
related to the task, $T$ be the time horizon of a task. Our problem can be
formulated as finding

$\displaystyle\max_{{\mathbf{a}}_{1},{\mathbf{a}}_{2}...{\mathbf{a}}_{T}}\sum_{t=1}^{T}{\mathbf{r}}({\mathbf{s}}_{t},{\mathbf{a}}_{t})$
$\displaystyle
s.t.\quad{\mathbf{s}}_{t+1}=f({\mathbf{s}}_{t},{\mathbf{a}}_{t})\,\,t=1,2...T-1,$

where $f$ governs the (unknown) system dynamics, and ${\mathbf{s}}$,
${\mathbf{a}}$ can also be subject to other algebraic constraints.

##  V METHODS

The workflow of our framework can be seen in Fig.1, and can be described as
follows:

  * •

Collect initial human demonstrations, saved to a replay buffer.

  * •

(Optional) Train visual features on demonstration dataset.

  * •

Roll out the current policy to collect more data.

  * •

A human operator performs on-policy correction if necessary.

  * •

(Optional) Add a learning curriculum.

  * •

Actor and critic networks are trained asynchronously w.r.t. data collection.

###  V-A Reinforcement learning from demonstration

RL methods are guided by reward signals. For contact-rich high-precision
manipulation tasks as presented in this paper, the agent needs to perform
sequenced fine-grained motion to achieve success. Using a shaped reward
function to guide the RL agent requires a large amount of engineering effort
to extract the necessary state information. On the other hand a pure sparse
reward poses an extremely difficult exploration problem. Incorporating human
demonstrations is a natural way to mitigate this issue [10, 2, 3, 11, 12, 1,
13, 14, 15, 16]. A demonstrator can express intent by using a remote guiding
device as in Fig.1. This is an intuitive way of showing the agent possible
solutions to the task, and can bootstrap a reasonable initial policy. We adopt
and extend the Deep Deterministic Policy Gradient from Demonstration (DDPGfD)
framework [2], in that it has several convenient mechanism dealing with human
demonstrations. The original DDPGfD algorithm and our modifications (in red)
can be seen in Alg.1. There are three modifications that significantly
increase performance of the original DDPGfD algorithm:

  * •

We remove the independent Gaussian noise added to the actor network – thus we
have a deterministic policy.

  * •

The replay buffer retains all experience data and all transitions are weighted
equally (no prioritization), actions from the human operator are regarded as
the agent’s own.

  * •

We also add task and action space curriculum to gradually increase the
difficulty level of learning for the agent.

  * •

We occasionally use the remote teaching device to override the agent’s actions
if it is not performing well, thus performing on-policy correction.

Our rationale for these changes is as follows. DDPG is a continuous variant of
Deep Q learning, which is an off-policy RL algorithm, and it minimizes Bellman
backup error as in Eq.1:

$L(\phi,\mathcal{D})=\underset{(s,a,r,s^{\prime})\sim\mathcal{D}}{E}\bigg{[}\bigg{(}Q_{\phi}(s,a)-\Big{(}r+\gamma
Q_{\phi}(s^{\prime},\pi_{\theta}(s^{\prime}))\Big{)}\bigg{)}^{2}\bigg{]}.$ (1)

Here, $\phi$ and $\theta$ are parameters for the Q network and policy network;
replay buffer $\mathcal{D}$ contains data collected by an exploration policy;
which is usually formed by adding exploration Gaussian noise (could be state
dependent or not) to the acting policy. We remove this Gaussian noise for two
reasons: 1) exploration is addressed via demonstrations, and 2) i.i.d. random
noise decreases the probability of successfully navigating narrow passages,
which is precisely the problem we face in insertion tasks. This way we can
easily reproduce behaviors from training in testing scenarios. In our
experiments we found that the noise due to the stochastic learning process,
coupled with the demonstration trajectories, provides sufficient exploration,
despite the actor being deterministic.

It is not uncommon to setup a prioritization mechanism in the replay buffer as
seen in previous work [17]. However, in our regime we’ve found that
prioritization schemes, e.g. using TD-error, consistently hurt performance. We
speculate that this is due to the highly off-policy regime in which we operate
in RLfD – not only does the replay buffer contain human demonstrations, but we
also never evict samples, so early data becomes increasingly off-policy as
training progresses.††See [18] for a discussion on this effect. Instead, we
always load all data seen so far into the replay buffer, and use uniform
sampling to make sure that every piece of data gets utilized equally.

The third modification we introduce is for a human operator to intervene and
correct the RL agent if necessary. This “on-policy” correction has been used
in previous works in imitation learning methods to correct distributional
shift between the offline expert dataset and the online policy, and to
incrementally train an agent to solve problems it encounters during
learning[19]. This DAGGER approach is particularly intuitive for insertion
tasks. Consider the situation depicted in Fig.3(a) in which the plug has
fallen to the outside of the socket. The correct strategy would be to lift up
to avoid that local optimum and try moving to the top of the socket again to
perform insertion. However, the agent has no mechanism of its own to develop
this recovery behavior since the reward is sparse, and there is no local
information guiding the agent to escape that local optimum. The human can
simply intervene to guide the agent back into the region in which it has
previously succeeded, and thanks to the off-policy structure of the algorithm
the agent can incorporate this correction to future policies. In practice, we
found around ten such corrections would be enough for the agent to succeed in
the task.

Algorithm 1 Robust DDPGfD

1:policy parameter $\theta$, target policy parameter $\theta^{\prime}$, Q
network parameter $\phi$, target Q network parameter $\phi^{\prime}$

2:for $t$ $\in$ [0, _episode_length_] do

3: Collect demonstration, save to replay buffer $\mathcal{B}$

4: Select an action $a_{t}=\pi_{\theta}(s_{t})$ $+\sigma$, where
$\sigma\sim\mathcal{N}(0,\Sigma)$

5: Take action $a_{t}$, observe$(s_{t},a_{t},s_{t+1},r_{t})$, add it to
$\mathcal{B}$

6: On-policy correction $a_{t}^{\prime}$ if necessary, overriding $a_{t}$

7: Compute TD target with minibatch data from $\mathcal{B}$ and target
networks $Q_{\phi^{\prime}}$ and $\pi_{\theta^{\prime}}$

8: Update $\theta$ and $pi$ with the TD target

9: Update target networks $\theta^{\prime}$ and $\phi^{\prime}$ (Polyak
averaging)

10:end for

Figure 3: An illustration of on-policy correction: (a)the robot fails behind
the sidewall of the male connector, (b)human engages by using remote teaching
device, (c)human corrects robot’s behavior

###  V-B Relative Coordinates and Goal Randomization

In order to not over-fit to any particular absolute coordinate position, we
introduce a mechanism allowing the agent to generalize to new locations. We
drop all usage of pose information in the robot’s base frame; instead we treat
the robot’s reset pose as the “origin” for the task, and express all relevant
proprioceptive information w.r.t. that origin.

From the agent’s perspective this is equivalent to physically moving the goal,
but requires no additional mechanical apparatus beyond the robot itself. A
convenient consequence of this parameterization is that it is amenable to
fine-grained control of the distribution of poses, which enabled the
curriculum approach we describe in Section VI-B1. We detail this procedure in
the Alg.2.

Algorithm 2 Relative coordinates and goal randomization

1:perturbation magnitude $\epsilon$, nominal resetting pose $T_{0}$ w.r.t.
robot base

2:for $i$ $\in$ [0, _total_episodes_] do

3: Sample a noise $\sigma$ from uniform distribution
$\mathcal{U}[-\epsilon,\epsilon]$

4: Get randomized resetting pose $T_{i}=T_{0}+\sigma$ w.r.t. base

5: for $t$ $\in$ [0, _episode_length_] do

6: Get current pose $T_{i}^{t}$ w.r.t. base

7: Calculate relative pose $T_{rel}^{t}=T_{i}^{t}\times T_{i}^{-1}$

8: Record $T_{rel}^{t}$ to current state $s_{i}^{t}$

9: end for

10:end for

###  V-C Pre-trained Visual Features

As discussed previously, our goal is to train control policies in the real
world in a timescale that is feasible for industrial use cases. [20]
demonstrated that it is possible to train a USB-insertion policy directly from
pixels, but it took 8 hours to reach 80% performance, which does not meet our
requirements. Instead of training image features from reward, we advocate pre-
training visual features using unsupervised learning objectives. In this work
used a variational auto-encoder (VAE), but there are numerous visual
representation-learning approaches and this is an area of active research.

To gather this dataset we use the remote teaching device to jog the robot
around the workspace to collect camera images to supplement the demonstration
dataset. These images are necessary as the demonstration data typically only
covers the part of the state-space seen by a near-optimal policy, which would
lead to poor features when run outside this distribution. This would be
particularly problematic since we regress a human-labeled reward signal on the
VAE features.

The VAE training objective can be seen in Eq.2:

$\mathcal{L}(\theta,\phi;x)=\mathbb{E}_{q_{\phi}}(z|x)\Big{[}\text{log}p_{\theta}(x|z)\Big{]}-{\beta}D_{KL}\Big{(}q_{\phi}(z|x)||p(z)\Big{)}.$
(2)

Here, $x$ are inputs to the VAE, which in our case are camera images; $z$ is
the latent code, usually represented by a Gaussian distribution parameterized
by mean and variance. The fist part in this loss function aims to reconstruct
the original inputs; $p(z)$ in the second term of loss function is called the
Prior that we often pick as a unit Gaussian distribution, the posterior
$q_{\phi}(z|x)$ is asked to stay close with this prior distribution in the
sense of some divergence measure.

After the VAE is trained, we fix the weights and discard the decoder. The
encoder is used to extract latent codes from images, and these are passed to
the agent. We’ll detail the network architecture in the following section.

###  V-D Multi-modal policies

Figure 4: Different modality inputs to the policy.

We train and test the RL agent under four different sets of input modalities
as in Fig.4. In Fig.4(a), the agent uses only the proprioceptive sensors of
the robot arm. In Fig.4(b), we add wrist force/torque sensor on top of the
proprioceptive information. In the Fig.4(c),(d), pre-trained visual features
are made available to the agent; with the difference that, in (d), we rely
purely on vision and force/torque, and the VAE is also used as reward detector

##  VI EXPERIMENTS

We report our experimental results in three sections: first the NIST board
insertion tasks, which we evaluate with the first three input modalities as in
Fig.4; second the moving target HDMI insertion, which we evaluate with the
last input modality; last the key-lock insertion results, which doesn’t use
vision and performs long range “in-the-dark search”.

###  VI-A NIST board insertion results

####  VI-A1 Experiment setup

The experiment setup is shown in Fig.2. For each connector, we train policies
with three different observation modalities: 1) proprioceptive information
only, 2) proprioceptive information + wrist force/torque, 3) proprioceptive
information + wrist force/torque + vision. For each trained policy, we
evaluate its success rate on the trained location, with some perturbation
noise added to the starting pose. We then move the female connector to a new
location and 45 degree apart from its original orientation along the Z axis;
and evaluate the success rate. To evaluate the generalization potential for
vision-based polices, the agent performs insertion while a person is manually
moving the NIST board as in Fig.5. This is a setting the agent was not trained
on, but it verifies our hypothesis in Sec.V-B: the trained policy should
generalize to any new condition as long as the relative transformation between
start state and goal is within the range of training randomization. For all
experiments, we use a Kuka iiwa arm, and control the end-effector Cartesian
velocity using a customized impedance controller.

We use a single Nvidia Titan RTX GPU for training. We use the ground truth
pose to provide sparse reward label, i.e., if the current pose of the TCP is
within some tolerance of target pose. Each episode lasts a maximum 10 seconds;
early termination occurs on successful insertion or if the robot end-effector
leaves the specified workspace. For vision-based experiments, we use a off-
the-shelf USB camera, crop its image size to 320$\times$240.

Figure 5: An illustration of dynamic the insertion task.

####  VI-A2 Baselines

For a fair and convincing comparison, we contracted two independent robotic
solution vendors to solve the same NIST board challenge. They used off-the-
shelf tools and additional engineering on their side. We use their results as
a baselines. We considered pure RL methods without demonstrations as an
additional baseline, but none were able to solve these tasks and are not
included in the discussion. Below we describe the vendors’ approaches:

##### Vendor’s approach

The robot moves to a predefined pose in the given female connector’s frame
where it’s close to establishing contact. The robot then moves along the
board’s normal direction until an above-threshold force is detected. The robot
regulates the force along the board normal direction at a predefined value and
0 torque along all 3 axes, meanwhile, searching in a spiral pattern in the
board plane[21]. The procedure stops when the movement along the board normal
direction exceeds a threshold, i.e., the plug is considered aligned and is
inserted into the hole.

##### Vendor’s approach

The nominal target female connector pose is defined such that the male
connector and the female connector are perfectly aligned for insertion. Then
we inject a 3D perturbation error: along the two axes in translation in
parallel with the assembly board plane ($x,y$), and in rotation around the
board’s normal direction ($rz$). We start the insertion attempts using the
nominal pose, repeating for 10 times with $rz$ rotation error smaller than 2
degrees. Then we increase the translation perturbation by 0.5mm along $x$
and/or $y$ axis following an outwards spiral pattern on a 2D grid, and repeat
10 attempts at each grid point. This process continues until consecutive
failures are observed.

####  VI-A3 Results

It is expected that vendors can solve these NIST assembly tasks with off-the-
shelf tools. We are more interested in understanding how they solve these
tasks and how well their solutions generalize. From task to task, the only
change is the target pose; there is no additional engineering for each
individual connector. On the vendors’ side, a fair amount of time and effort
is spent to engineer a solution and then fine-tune parameters for each
connector type. More importantly, our methods are far more generalizable. They
outperform vendor solutions by large margins in terms of perturbation ranges
while keeping nearly 100% success rates. This is a key performance indicator
in semi-constrained environments with some level of uncertainties. Fig.I shows
the comparison of maximum perturbation of each connector. Table I presents
complete data for each connector under different testing conditions.

Figure 6: Comparison of maximum perturbation between our method and vendor’s.

Waterproof USB DSUB pose + perturbation 100% (931/931) 100% (1357/1357) 100%
(839/839) pose + 45 deg 100% (1035/1035) 99.4% (623/627) 100% (960/960) pose +
F/T + perturbation 100% (801/801) 98.9% (791/800) 100% (1547/1547) pose + F/T
+ 45 deg 100% (662/662) 98.5% (639/649) 100% (645/645) pose + F/T + vision +
perturbation 100% (653/653) 100% (802/802) 100% (788/788) Total: 99.8%
(13069/13096)

TABLE I: Results of our method on three different connectors under different
input modalities. “perturbation” means that perturbations noise is added to
its nominal resetting poses during training. “45 deg” means that it was tested
on a different location and rotated 45 degree apart around the Z axis.

In total, we have performed 13096 trials, with a 99.8% success rate. This
strongly implies our method is robust and reliable. Most failures occurred
with the USB connector, mainly due to the floating metal part inside the
socket; active hard contact can cause changing dynamics. In the future we plan
to look at ways to encourage a more gentle insertion policy.

####  VI-A4 Dynamic insertion

As mentioned in Sec.VI-A1, we test the vision-based policies in more
challenging conditions. The agent needs to perform an insertion while a person
is manually moving the NIST board as in Fig.5. This is a much harder task than
static insertions, the agent’s policy must be reactive so that it quickly
adapts to changes in the environment. In essence this a closed-loop visual
servoing policy, emerging from training using relative coordinates and goal
randomization. Results show that our methods are able to solve these very
challenging tasks with over 95% success rate out of 50 trials. Most failures
occur when the person generates high-frequency movement that is beyond the
impedance controller’s operating frequency, or far exceeding the training
perturbation range. Complete videos can be seen on our website.

###  VI-B Moving HDMI Insertion

We have so far evaluated the robustness of our methods; another important
metric in industrial robotics is cycle time. In this experiment, we evaluate
the cycle time of our methods by asking the agent to do dynamic socket
insertions with maximum possible speed. The goal is to demonstrate that the
reliability and speed of our approach matches that of human hand-eye
coordination.

####  VI-B1 Experiment Setup

The experiment involves the insertion of an HDMI plug into a dynamically
moving socket. Socket motion is achieved by mounting it to a secondary robot
arm. Neither the pose or velocity of the socket is given to the RL agent.

The HDMI socket is rigidly mounted to the flange of a 7-DOF Sawyer arm, the
plug is rigidly attached to the flange of a 7-DOF Kuka IIWA robot. The IIWA
has a wrist sensor providing Cartesian force-torque sensing, and a wrist
mounted USB camera. At the start of each episode, the start position of the
socket is randomized, the start pose of the plug is 15cm from the mean socket
pose. During the episode the socket moves in a random circular motion in the
plane perpendicular to the direction of insertion. In this case, we hide the
pose information for the RL agent, it only has access to local TCP-frame
velocity, wrist force/torque, and vision as in the fourth input modality in
Fig.4. The RL agent is controlling the robot at 20 HZ.

Figure 7: The cell setup for moving target HDMI insertion.

##### Task Curriculum

To successfully train an RL agent for this challenging task, we employed a
task curriculum. The task starts off with no randomization of the socket pose
and no socket movement. As the agent improves, the task difficulty is
increased, first by increasing the socket pose randomization, and then by
increasing the movement speed of the socket. At the conclusion of the
curriculum the socket initial position is randomized by up to +-6cm, and the
movement speed is up to 2.5cm/s.

##### Action Space Curriculum

As we are aiming to minimize the insertion time, the agent needs to be able to
request large velocities. However, an untrained agent may cause damage to the
robot or environment. To improve training and safety we used a curriculum
approach to the action space. Initially the available action space is quite
small and conservative, but is grown exponentially (up to a maximum) as the
agent policy becomes more reliable.

####  VI-B2 Agent Results

We trained the RL agent on the task for 12 hours, with the replay buffer
initialized with 25 successful demonstration trajectories. At the end of
training we performed an evaluation run with the task randomization and agent
action space set to maximum. We recorded the success/failure of each episode
and the amount of time taken per episode. The agent had a 100% insertion
success rate over 1637 episodes, and a mean insertion time of 1093ms.

Figure 8: Insertion completion time comparison between our method and human

####  VI-B3 Human Results

Our aim is to compare the RL agent performance to that of human hand-eye
coordination. The setup of the human trials is similar to the RL agent, except
the IIWA robot is removed and the human holds the HDMI plug in their hand. At
the start of each episode the human participant is told to keep their their
eyes closed and their hand positioned roughly 15cm from the mean position of
the socket. The robot holding the socket resets to a randomized position, and
a beep is played over speakers to indicate the start of the episode. The human
participant then opens their eyes and inserts the plug as fast as they can
into the moving socket. We decided upon this setup to maximally match how the
RL agent and the human participants experience the task. A total of 33
attempts were made by 2 human participants, with 100% insertion success rate
and a mean insertion time of 1041ms.

These experimental results show that the reliability and insertion time of the
RL agent is comparable to that of human hand-eye coordination.

###  VI-C Key-lock insertion

As mentioned before, the plugs and sockets involved in previous insertion
experiments were mainly industrial connectors, which have a particular range
of sizes and material compliance characteristics. In this experiment we use a
standard household metal lock and key as the plug and socket. The purpose of
this experiment is to demonstrate that an RL agent can learn to insert objects
with small surface area and clearance, as well as long range (when compared to
the plug clearance) searching policies. The latter increases the range of
perturbations than can be handled without a vision system when compared to
industry standard search algorithms, which massively reduces system
complexity.

Figure 9: The cell setup for key insertion experiment.

####  VI-C1 Experiment Setup

This experiment consists of a key inserting into a lock with a RL-controlled
robot. The key is attached to the end effector of 7-DOF Kuka IIWA robot while
the lock is rigidly mounted in the cell.

The robot is controlled by the RL agent at a rate of 20 Hz with a Cartesian
velocity action space. The agent has access to full proprioceptive information
and wrist force/torque as in the second modality in Fig.4; we also adopt the
same relative coordinate and randomization mechanism as before.

At the start of each episode the initial position of the robot is randomized.
The key is always 5cm away from the lock but the position in x, y axis is
sampled randomly (where z axis is aligned with the key). This offset is not
observable since the TCP pose given to the agent is relative to the start of
the episode. This makes the task more challenging: it forces the agent to
learn a searching policy in order to find the lock opening and complete the
insertion.

The episode terminates when the insertion is completed (which can be
determined by the true TCP pose), the robot leaves the workspace, time limit
of 15s is exceeded or the force/torque readings exceed a limit (to avoid
damaging the key).

##### Starting pose randomization curriculum

We employed a curriculum of the starting pose randomization. The offset
magnitude in X/Y axis is sampled from a continuous uniform distribution
between -$b$ and $b$. We start with $b=0$ and slowly increase it as the agent
is learning. This allowed the agent to learn insertion policies with up to
0.5cm offset in x and y axes, which the agent failed to do without the
curriculum.

####  VI-C2 Results

The results are summarized in table II, measuring the success rate of the RL
agent given different maximum starting pose offset. The ‘randomization’ column
indicates maximum offset in X/Y axes that can be sampled at the start of the
episode. These results show that our SHIELD agent is capable of learning
insertion policies for objects of small clearance and surface area, as well as
a long range search policy. The RL agent increases the range of a search
policy to 5mm compared to vendors’ results of 2mm in Section VI-A Fig.6.

Maximum Pose Randomization | Insertion Success Rate
---|---
0.1cm | 92% (92 / 100)
0.3cm | 88% (88 / 100)
0.5cm | 87% (87 / 100)
TABLE II: Key insertion results. The pose randomization value represents the
maximum distance from the central position (right in front of the lock) to the
sampled random positions in x,y axes (where z axis is aligned with the key)

##  VII OPEN-SOURCE DATASET

We colected a large amount of data during the NIST board evaluations, and we
open sourced this dataset in the hopes of building interest in this setting
within the machine-learning community.

The dataset consist of around 50 hours of robot interaction time across
different tasks and input modalities; reward labels are also included. Further
details can be found on our website:https://sites.google.com/view/shield-nist

##  VIII DISCUSSION AND FUTURE WORK

We present a framework for learning robust robotic manipulation policies in
industrial settings. We systematically evaluated the performance of our
approach in the industrial robotic manipulation benchmark, NIST board. We show
99.8% overall success rate out of 13K trials under a variety of different
testing scenarios. The effectiveness of our approach is further demonstrated
on two very challenging robotic tasks. The first is a moving target HDMI
insertion task, the agent achieved a 100% success rate out of 1637 trials and
matched human performance in terms of cycle time. The second task is key-lock
insertion, the agent learned a policy to successfully insert a metal key into
a narrow clearance key-hole in the presence of initial condition
randomization.

There are two key takeaways from our results:

  * •

The NIST board assembly challenge represents a class of tasks solvable by
robotic system integrators available today. However, these solutions require a
semi-constrained environment and additional engineering effort for each
individual assembly task, even if these are very similar to one another. Our
approach mitigates these issues by leveraging demonstrations and training time
randomization. Our approach can be seen as a “general template” to solve a
class of similar tasks with almost no parameter tuning except requiring a few
demonstrations. Our learned policies outperform the traditional solutions by a
large margin in terms of handling perturbations and generalization, verified
by a large amount of real-world trials. This is essential to deploying any
industrial robotic solution to the field.

  * •

We also show that our method is able to unlock new applications for industrial
robots. This can enable robots to enter unconstrained manufacturing
environments and industries where the automation level remains low.

We believe that our approach, together with the evaluation presented, would be
the first step in applying learning-based robotic manipulation to the
industrial robot field.

Future work can be regarded as following directions:

  * •

Much larger-scale evaluation to verify the “99.999%” success rate, which can
further convince the community of the tech-readiness of RL for industrial
robotics.

  * •

Better visual representations for cluttered scenes. We found our vision model
was not particularly robust to environmental changes such as lighting,
background, distractions etc. VAE’s loss function has a reconstruction term
which attempts to model the entire scene, rather than focusing on the objects
relevant to the task. As the objects become smaller in the visual field we
find the performance of these agent decreases.

  * •

We can further shorten the training time by leveraging recent advances in
offline reinforcement learning plus online fine-tuning [22, 23, 24, 25].

## References

  * [1] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov and Sergey Levine  “Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations”  In _Proceedings of Robotics: Science and Systems_ , 2018  DOI: 10.15607/RSS.2018.XIV.049
  * [2] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe and Martin Riedmiller  “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards”  In _arXiv preprint arXiv:1707.08817_ , 2017
  * [3] Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester and Jon Scholz  “A practical approach to insertion with variable socket position using deep reinforcement learning”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 754–760  IEEE
  * [4] Jianlan Luo, Eugen Solowjow, Chengtao Wen, Juan Aparicio Ojea and Alice M Agogino  “Deep reinforcement learning for robotic assembly of mixed deformable and rigid objects”  In _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2018, pp. 2062–2069  IEEE
  * [5] Jianlan Luo, Eugen Solowjow, Chengtao Wen, Juan Aparicio Ojea, Alice M Agogino, Aviv Tamar and Pieter Abbeel  “Reinforcement learning on variable impedance controller for high-precision robotic assembly”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 3080–3087  IEEE
  * [6] Gerrit Schoettler, Ashvin Nair, Juan Aparicio Ojea, Sergey Levine and Eugen Solowjow  “Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks”  In _arXiv preprint arXiv:2004.14404_ , 2020
  * [7] Tadanobu Inoue, Giovanni De Magistris, Asim Munawar, Tsuyoshi Yokoya and Ryuki Tachibana  “Deep reinforcement learning for high precision assembly tasks”  In _Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on_ , 2017, pp. 819–825  IEEE
  * [8] Kenneth Kimble, Karl Van Wyk, Joe Falco, Elena Messina, Yu Sun, Mizuho Shibata, Wataru Uemura and Yasuyoshi Yokokohji  “Benchmarking protocols for evaluating small parts robotic assembly systems”  In _IEEE robotics and automation letters_ 5.2 IEEE, 2020, pp. 883–889
  * [9] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow and Sergey Levine  “Residual reinforcement learning for robot control”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 6023–6029  IEEE
  * [10] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba and P. Abbeel  “Overcoming Exploration in Reinforcement Learning with Demonstrations”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 6292–6299  DOI: 10.1109/ICRA.2018.8463162
  * [11] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, Gabriel Dulac-Arnold, John Agapiou, Joel Leibo and Audrunas Gruslys  “Deep Q-learning From Demonstrations”  In _Proceedings of the AAAI Conference on Artificial Intelligence_ 32, 2018  URL: https://ojs.aaai.org/index.php/AAAI/article/view/11757
  * [12] Dan Horgan, John Quan, David Budden, Gabriel Barth-Maron, Matteo Hessel, Hado Hasselt and David Silver  “Distributed Prioritized Experience Replay”, 2018  arXiv:1803.00933 [cs.LG]
  * [13] Scott Fujimoto, David Meger and Doina Precup  “Off-Policy Deep Reinforcement Learning without Exploration”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, Proceedings of Machine Learning Research  PMLR, 2019, pp. 2052–2062  URL: http://proceedings.mlr.press/v97/fujimoto19a.html
  * [14] Gabriel Dulac-Arnold, Daniel Mankowitz and Todd Hester  “Challenges of Real-World Reinforcement Learning”, 2019  arXiv:1904.12901 [cs.LG]
  * [15] Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando Freitas and Nicolas Heess  “Reinforcement and Imitation Learning for Diverse Visuomotor Skills”, 2018  arXiv:1802.09564 [cs.RO]
  * [16] Junhyuk Oh, Yijie Guo, Satinder Singh and Honglak Lee  “Self-Imitation Learning”  In _Proceedings of the 35th International Conference on Machine Learning_ 80, Proceedings of Machine Learning Research  Stockholmsmässan, Stockholm Sweden: PMLR, 2018, pp. 3878–3887  URL: http://proceedings.mlr.press/v80/oh18b.html
  * [17] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel and Wojciech Zaremba  “Hindsight Experience Replay”  In _Advances in Neural Information Processing Systems_ 30 Curran Associates, Inc., 2017  URL: https://proceedings.neurips.cc/paper/2017/file/453fadbd8a1a3af50a9df4df899537b5-Paper.pdf
  * [18] Aviral Kumar, Abhishek Gupta and Sergey Levine  “Discor: Corrective feedback in reinforcement learning via distribution correction”  In _arXiv preprint arXiv:2003.07305_ , 2020
  * [19] Stéphane Ross, Geoffrey Gordon and Drew Bagnell  “A reduction of imitation learning and structured prediction to no-regret online learning”  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , 2011, pp. 627–635
  * [20] Serkan Cabi, Sergio Gómez Colmenarejo, Alexander Novikov, Ksenia Konyushkova, Scott Reed, Rae Jeong, Konrad Zolna, Yusuf Aytar, David Budden and Mel Vecerik  “Scaling data-driven robotics with reward sketching and batch reinforcement learning”  In _arXiv preprint arXiv:1909.12200_ , 2019
  * [21] ABB  “Application manual Force Control”, 2018  ABB
  * [22] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar and Sergey Levine  “COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning”, 2020  arXiv:2010.14500 [cs.LG]
  * [23] Ashvin Nair, Murtaza Dalal, Abhishek Gupta and Sergey Levine  “Accelerating Online Reinforcement Learning with Offline Datasets”, 2020  arXiv:2006.09359 [cs.LG]
  * [24] Aviral Kumar, Aurick Zhou, George Tucker and Sergey Levine  “Conservative Q-Learning for Offline Reinforcement Learning”, 2020  arXiv:2006.04779 [cs.LG]
  * [25] Sergey Levine, Aviral Kumar, George Tucker and Justin Fu  “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”, 2020  arXiv:2005.01643 [cs.LG]


# The Future of Primordial Black Holes:
Open Questions and Roadmap

Antonio Riotto Département de Physique Théorique, Université de Genève,
24 quai Ansermet, CH-1211 Genève 4, Switzerland  Gravitational Wave Science
Center (GWSC), Université de Genève,
CH-1211 Geneva, Switzerland  Joe Silk Institut d’Astrophysique, UMR 7095 CNRS,
Sorbonne Université,
98bis Bd Arago, 75014 Paris, France  Department of Physics and Astronomy, The
Johns Hopkins University,
Baltimore MD 21218, USA  Beecroft Institute of Particle Astrophysics and
Cosmology, Department of Physics, University of Oxford, Oxford OX1 3RH, UK

###### Abstract

We discuss some of the the open questions and the roadmap in the physics of
primordial black holes. Black holes are the only dark matter candidate that is
known to actually exit. Their conjectured primordial role is admittedly based
on hypothesis rather than fact, most straightforwardly as a simple extension
to the standard models of inflation, or even, in homage to quantum physics,
more controversially via a slowing-down of Hawking evaporation. Regardless of
one’s stance on the theoretical basis for their existence, the possibility of
primordial black holes playing a novel role in dark matter physics and
gravitational wave astronomy opens up a rich astrophysical phenomenology that
we lay out in this brief overview.

##  1 Introduction

Primordial black holes (PBHs) provide an attractive way to resolve the dark
matter problem. Unlike essentially all other dark matter particle candidates,
BHs actually exist, and there are several robust prospects for determining
whether there is a primordial component over a broad mass range. Theoretically
PBHs may populate the universe in a humongous range of masses from
$10^{-20}\,M_{\odot}$ to $10^{20}\,M_{\odot}$. They could be the dark matter
for masses around $10^{-12}\,M_{\odot}$. Moreover even if PBHs are dark
matter-subdominant, , they could be responsible for some of the currently
observed mergers of BHs in the solar mass range or they may play key roles as
rare IMBH (Intermediate Mass BHs in the range $(10^{3}-10^{6})M_{\odot}$ that
may seed both SMBH (Supermassive Black Holes observed in the mass range
$(10^{6}-10^{10})M_{\odot}$ and massive galaxies in the early universe.

PBHs formed very early and hence may be ubiquitous at the highest redshifts
accessible by direct imaging, that is $z\lower
2.15277pt\hbox{$\;\buildrel<\over{\sim}\;$}100,$ when there was little
astrophysical competition. This provides us with the possibility of finding
truly unique high redshift signatures of PBHs, for example via gravity wave
interferometers such as Einstein Telescope and Cosmic Explorer that will probe
$z\lower 2.15277pt\hbox{$\;\buildrel<\over{\sim}\;$}20$, for potential sources
of EMRI signals.

Well before these telescopes are in action, there will be 21cm dark ages
telescopes destined for the lunar far side, such as LUSEE-Night and DSL to
probe PBH accretion imprints on diffuse hydrogen absorption against the CMB to
$z\lower 2.15277pt\hbox{$\;\buildrel<\over{\sim}\;$}100$, both scheduled for
deployment in 2026. And on a similar time-scale we anticipate LIGO-class
gravity wave telescopes to make significant inroads on the possible existence
of solar mass black holes as well as black holes populating the ”forbidden”
region in the pair instability gap, at masses $\lower
2.15277pt\hbox{$\;\buildrel>\over{\sim}\;$}50\,M_{\odot}.$ Such observations
would provide further evidence for the existence of PBHs.

We review some of these arguments in this chapter, with particular attention
to the open questions, challenges and future opportunities.

##  2 Some open questions

How well do we know the PBH abundance? And over what mass range? How well can
we ascertain the PBH clustering properties and therefore the GW signal from
binary mergers via LIGO and future terrestrial GW interferometers? And can we
use the vicinities of SMBH amd IMBH, most likely rich in dark matter and where
PBH abundances are in consequence likely to be enhanced, as potential sources
of EMRI signals for experiments such as LISA? We briefly discuss such open
theoretical and observational questions below.

###  2.1 What is the abundance of PBHs?

The formation of a PBH in the early universe is a rare event and knowing the
precise formation probability represents, even nowadays, a challenge. This
issue is of fundamental importance as the formation probability is one of the
key ingredients to calculate the current PBH abundance and mass function which
not only enter in many observables, such as the merger rate of BH binaries,
but which are also routinely used to express current constraints from various
observations.

To illustrate the difficulty in assessing the precise abundance of PBHs, let
us consider the most standard scenario in the literature where the PBHs are
formed by the collapse of large inhomogeneities generated during the
inflationary stage. The latter delivers a stochastic quantity, the curvature
perturbation $\zeta$ on superhorizon scales, whose properties are known
statistically. As we mentioned, PBHs are rare events which require a given
observable, e.g. the density contrast $\delta$ or the compaction function $C$,
to be above a typically large critical value. This implies that the
calculation of the PBH abundance requires going beyond linear perturbation
theory and the knowledge of the tail of the probability where the latter is
typically far from being a simple Gaussian distribution. Nonlinearities enter
into the game even when the initial curvature perturbation is Gaussian (which
is typically not) because of the nonlinear relation between the
overdensity/compaction function an the curvature perturbation itself. The
logic chain is (in radiation)

$\textrm{non-Gaussian}\,\zeta(r)\,\rightarrow\textrm{non-Gaussian compaction
function}\,C(r)=C_{1}(r)-\frac{3}{8}C_{1}^{2}(r),\,\,\,C_{1}=-\frac{4}{3}r\zeta^{\prime}(r).$
(1)

The presence of nonlinearities at the various stages of the calculation makes
the calculation cumbersome. One more complication comes from the fact that the
critical threshold $C_{c}$ to form a PBH depends on the shape of the initial
perturbation, specifically on the curvature around the maximum $r_{m}$ of the
compaction function

$C_{c}=C_{c}(q),\,\,\,\,q=-\frac{r_{m}^{2}C^{\prime\prime}(r_{m})}{4C(r_{m})},$
(2)

where primes indicate derivatives with respect to the radius and we are
assuming spherical symmetry. The formation probability therefore needs the
knowledge of the exact probability $P[C(r_{m}),C^{\prime\prime}(r_{m})]$.
Given an initial profile of the curvature perturbation, one neeeds therefore
to compute the corresponding critical profile for the compaction function. The
latter determines the position of the maximum $r_{m}$ in units of the typical
momentum scale $k_{\star}$ of the power spectrum of the curvature perturbation
and the correlators derived from the compaction function, which in turn
determine its non-Gaussian statistics. Even though the dependence on the
profile may be accounted for integrating over all possible values of the
curvature of the compaction function at its maximum in real space or realizing
that in some cases the broadest profiles count [1], the position of the
maximum $r_{m}$ is typically calculated only for the average profile of the
curvature perturbation and this automatically introduces some intrinsic and
unavoidable uncertainties due to the statistical nature of the observables. A
possible, albeit incomplete, way to proceed might be to find an observable
whose critical value does not depend on its profile, e.g. the volume average
of the compaction function, as suggested in Ref. [2]

$\overline{C}(r_{m})=\frac{3}{R_{m}^{3}}\int_{0}^{R_{m}}{\rm
d}x\,x^{2}\,C(x),\,\,\,R_{m}=r_{m}\,e^{\zeta(r_{m})},$ (3)

but no calculation exists in the literature so far and no understanding of why
the volume average of the compaction function has a critical threshold which
does not depend on the profile. Even without the need of mentioning other
possible problems, e.g. dealing with the nonlinearities entering in the
radiation transfer function when perturbations re-enter the horizon [3] or the
need to deal with the phenomenon of operator mixing [4], we believe the
calculation of the PBH abundance in the standard scenario is still an open
issue which deserves further study. The same is true in alternative mechanisms
to generate the PBHs, e.g. in supercooled first-order phase transitions [5]
where the threshold criterion does not suffice.

###  2.2 What is the effect of PBH clustering?

Another open issue worth mentioning is the clustering of PBHs. It may
significantly affect the merger rates of coalescing binaries and,
consequently, the gravitational wave signal measured by the LIGO/Virgo
collaboration and by future experiments. Indeed, even if isolated PBH binaries
are not significantly affected by encounters with a third PBH, PBH binaries
residing in PBH clusters might suffer such interactions, thus modifying the
expected merger rate. If PBHs cluster or not is therefore an important
question to address. A significant PBH clustering might also modify
constraints arising from microlensing observations and from the cosmic
microwave background. Clustering might also be relevant in determining PBH
spins. While the gravitational collapse of a spherical overdensity during
radiation domination generates nearly spinless PBHs [6], they may acquire a
large spin in the presence of accretion [7]. Indeed, if PBHs cluster they can
more easily merge to form binaries at late times, so that the resulting PBH
coming from such merger will have a non-zero value even when no accretion is
present.

Even though PBHs in the standard scenario formed through the collapse of large
overdensities are initially not clustered [8, 9], PBH clusters form not long
after matter-radiation equality [10] if $f_{\rm PBH}$ is large. Current N-body
simulations are limited to large redshifts $z\sim 10^{2}$ and a chief
challenge is how to extrapolate to lower redshifts relevant for some
observational constraints. Analytical insight is certainly called for in
understanding the complex evolution of the PBH population.

PBHs, even if rare, are potentially important for seeding galaxy and even
quasar formation at very high redshift [11]. These would be intermediate mass
PBHs that enhanced local growth by gravitational instability of cold dark
matter. in such a scenario, both massive galaxies and SMBH would form at
epochs much earlier than in the standard model of structure formation.
Moreover SMBH formation would precede or be contemporaneous with massive
galaxy formation, as actually favored by recent JWST observations [12].

###  2.3 What fraction of the currently observed GW events can be ascribed to
PBHs?

Thanks to the many binary BH events detected so far with many more expected in
the next few years, gravitational-wave astronomy is about to become a
population study. Currently hierarchical Bayesian analysis combining
astrophysical formation models and PBHs may only constrain the fraction of a
putative subpopulation of PBHs in the data (see for instance Ref. [13]). The
exciting possibility that PBHs are contributing to current observations will
be verified only by further reducing uncertainties in astrophysical and
primordial formation models. This is another fundamental open issue.

###  2.4 Are PBHs the Dark Matter?

The idea that PBH can comprise most of the dark matter is one of the main
motivations for studying PBH. Unfortunately, observational constraints
eliminate this possibility for most of the range of possible PBH masses, with
a notable exception around asteroid mass PBH, spanning several decades in mass
around $\sim 10^{-12}\,\rm M_{\odot}$ down to the limit $\sim 10^{-16}\,\rm
M_{\odot}$, obtained from the isotropic x-ray and soft gamma ray background
observational limits on fluxes produced by PBHs currently undergoing Hawking
evaporation [14].

In the standard formation scenario of PBH, it is unavoidable that
gravitational waves are generated with a frequency which today is in the mHz
range, exactly where the LISA mission has maximum sensitivity [15]. The
scenario of PBH as dark matter can be therefore tested in the future by LISA
by measuring the GW two-point correlator. The fact that the asteroid mass
range is still unconstrained is due to the fact that the microlensing
constraints are ineffective around the value $10^{-11}\,M_{\odot}$ under which
the geometric optics approximation is no longer valid and the constraints from
the presence of neutron stars in globular clusters is based on extreme
assumptions about the dark matter density. It is of tantamount importance to
come out with possible ideas for constraining or identifying PBH in the
asteroid mass range.

One promising approach is that PBH capture leads to neutron star conversion to
BH. This could occur in dense star clusters that contain DM, as may be the
case for nuclear star clusters, and NS conversion would occur in the case of
capture of such ”endoparasitic” PBHs for PBH masses larger than $\sim
10^{-11}\,M_{\odot}$ [16]. Such a phenomenon may produce a deficit of pulsars
in our GC as possibly observed [17]. Another interesting avenue is to observe
the number of massive main-sequence stars in ultra-faint dwarfs which should
be suppressed if the dark matter is made of asteroid-mass PBHs [18], making
measured the mass distribution of stars depleted in the high-mass range.

Furthermore, near-extremal PBHs provide an intriguing means of making
evaporating PBHs stable on cosmological scales. Formation is simplest for low
mass five-dimensional PBHs that initially act like four-dimensional PBHs,
Hawking radiating down to the radius of the extra dimension where their
effective temperature is effectively zero, to attain a stable mass [19]. These
are generated in higher dimensional scenarios and Hawking radiation is
generically found to be slowed down [20]. Other scenarios for producing near-
extremal PBH include formation of maximally rotating or charged PBH at very
early epochs [21], as well as via the quantum gravity phenomenon of so-called
memory burden suppression [22]. Detectability is feasible down to scales as
small as a few Planck masses for charged PBH relics via terrestrial detectors
[23] or equally for high energy particle emission from occasional binary
merger events [24].

##  3 The PBH Roadmap

Future experiments abound, ranging from approved projects to imminently enter
the detailed design phase, to provide significantly enhanced constraints on
PBH scenarios, including terrestrial laser interferometers (Einstein
Telescope, Cosmic Explorer)[25], and space laser interferometers (LISA, DECIGO
[26], TianQin and Taiji [27]), to atomic interferometers on the ground with
demonstrators under construction and extensions planned to km scales (AION
MAGIS, MAGIA, MIGA, ELGA, ZAIGA), as well as in space (AEDGE) [28], and
designs for lunar interferometer/seismometer arrays (LGWA, GLOC)[29].

The science goals focus on bridging frequency gaps in the LIGO/VIRGO/KAGRA
frequency range that has so far led to hundreds of detections of BH mergers in
the $(10-100)M_{\odot}$ range by going both to higher and lower frequencies.
The new science that can be explored ranges from the masses and equation of
state of neutron stars as well as multimessenger signals from core collapse
supernovae to kpc distances (kHz to Hz range) to SMBH and IMBH detections and
BH inspiralling (Hz to mHz, and eventually $\mu$Hz). One would ultimately
approach the pulsar timing array sensitivities (NANOGrav, SKA) at nanohertz
frequencies to enable exploration of first order very early universe phase
transitions, and presence of loop decays in a network of cosmic strings [30].

We next address various “smoking-gun” signatures that may identify PBHs,
notably including high redshift mergers and sub-solar mass PBHs.

###  3.1 High redshift mergers

The PBH model predicts a binary merger rate density which grows monotonically
with redshift [31, 32, 33]. Focusing those binaries generated at early epochs,
one has a well-defined time evolution with redshift

$R^{\rm EB}(z)\propto\left(\frac{t(z)}{t(z=0)}\right)^{-34/37},$ (4)

extending up to redshifts $z\lower
2.15277pt\hbox{$\;\buildrel>\over{\sim}\;$}{\cal O}(10^{3})$. Since mergers
from BHs of astrophysical origin switch off at redshifts $z\lower
2.15277pt\hbox{$\;\buildrel>\over{\sim}\;$}(20-30)$ [34, 35, 36, 37, 38, 39,
40, 41, 42, 43], detecting merger redshifts at $z\lower
2.15277pt\hbox{$\;\buildrel>\over{\sim}\;$}30$ will represent smoking guns for
primordial binaries [44, 45, 46]. Of course, large measurement uncertainties
on the inferred luminosity distance due to the low SNR at 3G detectors [46,
47, 48, 49] may bring relatively large errors, but recent studies show that
identifying merger rate evolution at redshift larger than ${\cal O}(10)$ will
allow to constrain PBH populations up to abundances as low as $f_{\rm
PBH}\approx 10^{-5}$ [50] in the solar mass range, even accounting for the
contamination of Pop III binaries, see Fig. 1.

Figure 1: Posteriors of $f_{\rm PBH}$ given a lognormal PBH mass function
centered at $M_{c}=30\,M_{\odot}$ and variance equal to 0.3 at $\beta_{\rm
PBH}=0,1/9,1/5,1/3$, and $1/2$, being $\beta_{\rm PBH}$ the fraction of the
PBH merger rate to the total merger rate. The solid black lines indicate the
95% CIs. From Ref. [50].

###  3.2 Sub-solar PBHs

The detection of sub-solar BHs in a compact binary merger is regarded as one
of the smoking gun signatures of PBHs. Indeed, BHs of astrophysical origin are
not expected. The sub-solar mass range may be however populated by other
compact objects like neutron stars, white dwarfs, or exotic compact objects
[51] (e.g. boson stars [52]).

GW signatures of any population of PBH below the astrophysical production
provide a powerful statistical constraint on any solar or subsolar component,
as evidenced by the sharp decline observed in 03 events. There could of course
be rare events generated by solar or subsolar PBHs that might contribute to DM
at the ${\cal O}(10\%)$ level. One key ingredient is therefore to distinguish
PBHs from other compact objects, e.g. through the measurements of the tidal
disruption and tidal deformability $\Lambda$. A recent analysis has shown that
for subsolar neutron-star binaries, the O4 and O5 projected sensitivities
would allow measuring the effect of tidal disruption on the waveform in a
large portion of the parameter space, also constraining the tidal
deformability at ${\cal O}(10\%)$ level, thus excluding a primordial origin of
the binary. Viceversa, for subsolar PBH binaries, model-agnostic upper bounds
on the tidal deformability can rule out neutron stars or more exotic
competitors. Assuming events similar to the sub-threshold candidate SSM200308
reported in LVK O3b data are PBH binaries, O4 projected sensitivity would
allow ruling out the presence of neutron-star tidal effects at $\approx
3\sigma$ C.L., thus strengthening the PBH hypothesis. Future experiments would
lead to even stronger ($>5\sigma$) conclusions on potential discoveries of
this kind, see Fig. 2.

PBHs of lunar mass, a regime where hints of unaccounted-for microlensing
events have been reported by the OGLE survey of the galactic bulge [53], could
be sufficiently numerous that nearby binaries might be detectable with future
generation detectors [54].

Figure 2: Precision of the deformability parameter $\Lambda$ at both Ad. LIGO
and ET. The binary is assumed to have spinless components and negligible
eccentricity and deformability, as predicted by the PBH scenario. The solid
(dashed) line indicates the result for mass ratios 1 $(1/4)$. From Ref. [47].

###  3.3 Plugging the pair instability gap with PBH?

An equally powerful mass challenge for PBH arises from the pair instability
gap. This is generated by the catastrophic core collapse of massive stars that
results from pair-instability SNe, and predicts a deficit of astrophysical BH
between $\sim 50\,M_{\odot}$ and $\sim 120\,M_{\odot}.$ The full LIGO-Virgo O3
data set reveals a definitive decline in merger rates above $\sim
40\,M_{\odot}$ [55] but with several candidates filling this gap at high
credibility [56].

Could PBH populate the pair instability gap? A smoking gun signature would be
a feature in the BH mass spectrum, most likely indicative of a new formation
mechanism.

###  3.4 PBH eccentricity

Even though PBH binaries are formed with large eccentricity at high redshift,
they have enough time to circularize before the GW signal can enter the
observation band of current and future detectors. This implies that an
observation of a non-zero eccentricity $e$ would rule out the interpretation
as a primordial binary formed in the early Universe, while it may still be
compatible with a PBH binary formed in the late-time universe [57, 58].

###  3.5 PBH spin

The PBHs form in the early universe through the standard scenario of the
collapse of large overdensities inherit a characteristic mass-spin
correlations induced by accretion effects [7]. Using this criterion for
determining the possible primordial nature of individual GW events would
require reducing uncertainties on the accretion model and studied at the
population level [59].

Fig. 3 presents a flowchart from Ref. [47] one might follow to assess the
primordial nature of a binary merger based on measurements of the redshift
$z$, eccentricity $e$, tidal deformability $\Lambda$, component masses $m$,
and dimensionless spins $\chi$.

Figure 3: Flowchart from Ref. [47] to assess the primordial nature of a binary
merger based on measurements of the redshift $z$, eccentricity $e$, tidal
deformability $\Lambda$, component masses $m$, and dimensionless spins $\chi$.
Each arrow indicates if the condition in the box is met (green) or violated
(red).

###  3.6 Future gamma-ray telescopes

PBHs have a range of potentially interesting direct gamma ray signals
associated with Hawking evaporation. These include the 511 keV gamma-ray line,
produced by electron-positron pair-annihilation, where positrons originate
from black hole evaporation. The INTEGRAL detection of the Large Magellanic
Cloud provides one of the strongest bounds attainable with present
observations, and should be greatly improved upon by future MeV gamma-ray
telescopes such as GECCO [60], as well as AMEGO [61] and ASTROGRAM [62], among
others [63].

Exploding PBHs, the late-time limit of Hawking evaporation for small PBHs,
provide intriguing candidates for future very high energy gamma ray telescopes
that may address transient phenomena, that pertain especially to the new
frontier of PBHs of ultralralow masses.

The Hawking temperature characterizes the effective temperature as inversely
proportional to PBH mass and is constrained as discussed previously by soft
gamma ray constraints at the $\sim 10^{17}\rm gm$ PBH mass scale,
incorporating appropriate spectral corrections due to nonthermal aspects of
the complex emission processes [64]. Once the allowable PBH mass range is
extended to much lower masses due to slowing or suppression of Hawking
radiation, as proposed in recent discussions of near-extremal and higher
dimensional PBHs, intriguing prospects arise for high energy gamma ray
experiments [21, 20, 65, 22]. As previously noted, new observational terrain
opens up for exploring near-extremal PBH detection both directly via
terrestrial experiments [23] and indirectly via binary merger events [24],
where constraints at much higher gamma ray or even neutrino energies could
become important.

Such gamma ray experiments include HAWC (100 GeV -50 TeV) and CTA (20 GeV-300
TeV) [66] at TeV scales, and at PeV-scale photon energies LHAASO [67] and SWGO
[68]. Such searches will be closely coordinated with multimessenger projects
spanning next generation underground gravitational wave telescopes (EINSTEIN
(ET), COSMIC EXPLORER (CE)) and FRB observatories (CHORD) [69].

##  4 Conclusions

Support for the existence of PBHs hypothesis comes from a range of arguments
spanning dark matter physics, inflationary models, astrophysics, gravitational
microlensing and quantum cosmology. None are yet conclusive but the cumulative
support for the existence of PBHs seems reasonably compelling. More to the
point however is that there is a rich astrophysical agenda of open questions
that contribute to a future road map for a plethora of research projects.

Two of the most fundamental PBH probes that distinguish them from
astrophysical counterparts are their redshift and mass distributions.This
combination provides an armoury of smoking guns that will inevitably enrich
the next generation of gravitational wave and gravitational microlensing
experiments. Confirmation will require improved statistics as single events
may not be conclusive. it is only too easy to come up with rare events that
can masquerade as PBH candidates, for example by appealing to higher order
generations of merging events or to non-Gaussianity. We have laid out a
possible future road map for definitively assessing whether PBHs can present a
viable candidate for dark matter, or possibly contribute to other
astrophysical or even quantum cosmology anomalies.

## References

  * [1] A. Ianniccari, A. J. Iovino, A. Kehagias, D. Perrone, and A. Riotto.  The Primordial Black Hole Abundance: The Broader, the Better.  2 2024.
  * [2] Albert Escrivà, Cristiano Germani, and Ravi K. Sheth.  Universal threshold for primordial black hole formation.  Phys. Rev. D, 101(4):044022, 2020.
  * [3] Valerio De Luca, Alex Kehagias, and Antonio Riotto.  How well do we know the primordial black hole abundance: The crucial role of nonlinearities when approaching the horizon.  Phys. Rev. D, 108(6):063531, 2023.
  * [4] Gabriele Franciolini, Andrea Ianniccari, Alex Kehagias, Davide Perrone, and Antonio Riotto.  Renormalized Primordial Black Holes.  11 2023.
  * [5] Marcos M. Flores, Alexander Kusenko, and Misao Sasaki.  Revisiting formation of primordial black holes in a supercooled first-order phase transition.  2 2024.
  * [6] V. De Luca, V. Desjacques, G. Franciolini, A. Malhotra, and A. Riotto.  The initial spin probability distribution of primordial black holes.  JCAP, 05:018, 2019.
  * [7] V. De Luca, G. Franciolini, P. Pani, and A. Riotto.  Constraints on Primordial Black Holes: the Importance of Accretion.  Phys. Rev. D, 102(4):043505, 2020.
  * [8] Yacine Ali-Haïmoud.  Correlation function of high-threshold regions and application to the initial small-scale clustering of primordial black holes.  Physical Review Letters, 121(8), aug 2018.
  * [9] Vincent Desjacques and Antonio Riotto.  Spatial clustering of primordial black holes.  Phys. Rev. D, 98(12):123533, 2018.
  * [10] Derek Inman and Yacine Ali-Haïmoud.  Early structure formation in primordial black hole cosmologies.  Phys. Rev. D, 100(8):083528, 2019.
  * [11] Bernard Carr and Joseph Silk.  Primordial black holes as generators of cosmic structures.  Monthly Notices of the Royal Astronomical Society, 478(3):3756–3775, May 2018.
  * [12] Joseph Silk, Mitchell Begelman, Colin Norman, Adi Nusser, and Rosemary Wyse.  Which came first: supermassive black holes or galaxies? insights from jwst, 2024.
  * [13] Gabriele Franciolini, Vishal Baibhav, Valerio De Luca, Ken K. Y. Ng, Kaze W. K. Wong, Emanuele Berti, Paolo Pani, Antonio Riotto, and Salvatore Vitale.  Searching for a subpopulation of primordial black holes in LIGO-Virgo gravitational-wave data.  Phys. Rev. D, 105(8):083526, 2022.
  * [14] J. Iguaz, P.D. Serpico, and T. Siegert.  Isotropic x-ray bound on primordial black hole dark matter.  Physical Review D, 103(10), May 2021.
  * [15] N. Bartolo, V. De Luca, G. Franciolini, M. Peloso, D. Racco, and A. Riotto.  Testing primordial black holes as dark matter with LISA.  Phys. Rev. D, 99(10):103521, 2019.
  * [16] Chloe B. Richards, Thomas W. Baumgarte, and Stuart L. Shapiro.  Accretion onto a small black hole at the center of a neutron star.  Physical Review D, 103(10), May 2021.
  * [17] Jason Dexter and Ryan M. O’Leary.  The peculiar pulsar population of the central parsec.  The Astrophysical Journal, 783(1):L7, February 2014.
  * [18] Nicolas Esser, Sven De Rijcke, and Peter Tinyakov.  The impact of primordial black holes on the stellar mass function of ultra-faint dwarf galaxies.  11 2023.
  * [19] Jose A. de Freitas Pacheco, Elias Kiritsis, Matteo Lucca, and Joseph Silk.  Quasiextremal primordial black holes are a viable dark matter candidate.  Physical Review D, 107(12), June 2023.
  * [20] Luis A. Anchordoqui, Ignatios Antoniadis, and Dieter Lüst.  Dark dimension, the swampland, and the dark matter fraction composed of primordial black holes.  Phys. Rev. D, 106(8):086001, October 2022.
  * [21] J. A. de Freitas Pacheco and Joseph Silk.  Primordial Rotating Black Holes.  Phys. Rev. D, 101(8):083022, 2020.
  * [22] Ana Alexandre, Gia Dvali, and Emmanouil Koutsangelas.  New Mass Window for Primordial Black Holes as Dark Matter from Memory Burden Effect.  2 2024.
  * [23] Benjamin V. Lehmann, Christian Johnson, Stefano Profumo, and Thomas Schwemberger.  Direct detection of primordial black hole relics as dark matter.  Journal of Cosmology and Astroparticle Physics, 2019(10):046–046, October 2019.
  * [24] Yang Bai and Nicholas Orlofsky.  Primordial extremal black holes as dark matter.  Physical Review D, 101(5), March 2020.
  * [25] Guillem Domènech and Misao Sasaki.  Probing primordial black hole scenarios with terrestrial gravitational wave detectors, 2024.
  * [26] Seiji Kawamura et al.  Current status of space gravitational wave antenna DECIGO and B-DECIGO.  PTEP, 2021(5):05A105, 2021.
  * [27] Yungui Gong, Jun Luo, and Bin Wang.  Concepts and status of chinese space gravitational wave detection projects.  Nature Astronomy, 5(9):881–889, September 2021.
  * [28] John Ellis, Malcolm Fairbairn, Juan Urrutia, and Ville Vaskonen.  Probing supermassive black hole seed scenarios with gravitational wave measurements, 2023.
  * [29] Joris van Heijningen et al.  The payload of the Lunar Gravitational-wave Antenna.  J. Appl. Phys., 133(24):244501, 2023.
  * [30] John Ellis, Malcolm Fairbairn, Gabriele Franciolini, Gert Hütsi, Antonio Iovino Jr. au2, Marek Lewicki, Martti Raidal, Juan Urrutia, Ville Vaskonen, and Hardi Veermäe.  What is the source of the pta gw signal?, 2023.
  * [31] Yacine Ali-Haïmoud, Ely D. Kovetz, and Marc Kamionkowski.  Merger rate of primordial black-hole binaries.  Phys. Rev. D, 96(12):123523, 2017.
  * [32] Martti Raidal, Christian Spethmann, Ville Vaskonen, and Hardi Veermäe.  Formation and Evolution of Primordial Black Hole Binaries in the Early Universe.  JCAP, 02:018, 2019.
  * [33] V. De Luca, G. Franciolini, P. Pani, and A. Riotto.  Primordial Black Holes Confront LIGO/Virgo data: Current situation.  JCAP, 06:044, 2020.
  * [34] Tomoya Kinugawa, Kohei Inayoshi, Kenta Hotokezaka, Daisuke Nakauchi, and Takashi Nakamura.  Possible Indirect Confirmation of the Existence of Pop III Massive Stars by Gravitational Wave.  Mon. Not. Roy. Astron. Soc., 442(4):2963–2992, 2014.
  * [35] Tomoya Kinugawa, Akinobu Miyamoto, Nobuyuki Kanda, and Takashi Nakamura.  The detection rate of inspiral and quasi-normal modes of Population III binary black holes which can confirm or refute the general relativity in the strong gravity region.  Mon. Not. Roy. Astron. Soc., 456(1):1093–1114, 2016.
  * [36] Tilman Hartwig, Marta Volonteri, Volker Bromm, Ralf S. Klessen, Enrico Barausse, Mattis Magg, and Athena Stacy.  Gravitational Waves from the Remnants of the First Stars.  Mon. Not. Roy. Astron. Soc., 460(1):L74–L78, 2016.
  * [37] K. Belczynski, T. Ryu, R. Perna, E. Berti, T. L. Tanaka, and T. Bulik.  On the likelihood of detecting gravitational waves from Population III compact object binaries.  Mon. Not. Roy. Astron. Soc., 471(4):4702–4721, 2017.
  * [38] Kohei Inayoshi, Ryosuke Hirai, Tomoya Kinugawa, and Kenta Hotokezaka.  Formation pathway of Population III coalescing binary black holes through stable mass transfer.  Mon. Not. Roy. Astron. Soc., 468(4):5020–5032, 2017.
  * [39] Boyuan Liu and Volker Bromm.  The Population III origin of GW190521.  Astrophys. J. Lett., 903(2):L40, 2020.
  * [40] Boyuan Liu and Volker Bromm.  Gravitational waves from Population III binary black holes formed by dynamical capture.  Mon. Not. Roy. Astron. Soc., 495(2):2475–2495, 2020.
  * [41] Tomoya Kinugawa, Takashi Nakamura, and Hiroyuki Nakano.  Chirp Mass and Spin of Binary Black Holes from First Star Remnants.  Mon. Not. Roy. Astron. Soc., 498(3):3946–3963, 2020.
  * [42] Ataru Tanikawa, Hajime Susa, Takashi Yoshida, Alessandro A. Trani, and Tomoya Kinugawa.  Merger rate density of Population III binary black holes below, above, and in the pair-instability mass gap.  Astrophys. J., 910(1):30, 2021.
  * [43] Neha Singh, Tomasz Bulik, Krzysztof Belczynski, and Abbas Askar.  Exploring compact binary populations with the Einstein Telescope.  Astron. Astrophys., 667:A2, 2022.
  * [44] Savvas M. Koushiappas and Abraham Loeb.  Maximum redshift of gravitational wave merger events.  Phys. Rev. Lett., 119(22):221104, 2017.
  * [45] V. De Luca, G. Franciolini, P. Pani, and A. Riotto.  Bayesian Evidence for Both Astrophysical and Primordial Black Holes: Mapping the GWTC-2 Catalog to Third-Generation Detectors.  JCAP, 05:003, 2021.
  * [46] Ken K. Y. Ng, Shiqi Chen, Boris Goncharov, Ulyana Dupletsa, Ssohrab Borhanian, Marica Branchesi, Jan Harms, Michele Maggiore, B. S. Sathyaprakash, and Salvatore Vitale.  On the Single-event-based Identification of Primordial Black Hole Mergers at Cosmological Distances.  Astrophys. J. Lett., 931(1):L12, 2022.
  * [47] Gabriele Franciolini, Roberto Cotesta, Nicholas Loutrel, Emanuele Berti, Paolo Pani, and Antonio Riotto.  How to assess the primordial origin of single gravitational-wave events with mass, spin, eccentricity, and deformability measurements.  Phys. Rev. D, 105(6):063510, 2022.
  * [48] Matteo Martinelli, Francesca Scarcella, Natalie B. Hogg, Bradley J. Kavanagh, Daniele Gaggero, and Pierre Fleury.  Dancing in the dark: detecting a population of distant primordial black holes.  JCAP, 08(08):006, 2022.
  * [49] Ken K. Y. Ng et al.  Measuring properties of primordial black hole mergers at cosmological distances: Effect of higher order modes in gravitational waves.  Phys. Rev. D, 107(2):024041, 2023.
  * [50] Ken K. Y. Ng, Gabriele Franciolini, Emanuele Berti, Paolo Pani, Antonio Riotto, and Salvatore Vitale.  Constraining High-redshift Stellar-mass Primordial Black Holes with Next-generation Ground-based Gravitational-wave Detectors.  Astrophys. J. Lett., 933(2):L41, 2022.
  * [51] Vitor Cardoso and Paolo Pani.  Testing the nature of dark compact objects: a status report.  Living Rev. Rel., 22(1):4, 2019.
  * [52] Huai-Ke Guo, Kuver Sinha, and Chen Sun.  Probing Boson Stars with Extreme Mass Ratio Inspirals.  JCAP, 09:032, 2019.
  * [53] Hiroko Niikura, Masahiro Takada, Shuichiro Yokoyama, Takahiro Sumi, and Shogo Masaki.  Constraints on earth-mass primordial black holes from ogle 5-year microlensing events.  Physical Review D, 99(8), April 2019.
  * [54] Kaiki Taro Inoue and Takahiro Tanaka.  Gravitational waves from sub-lunar-mass primordial black-hole binaries: A new probe of extradimensions.  Physical Review Letters, 91(2), July 2003.
  * [55] The LIGO Scientific Collaboration, the Virgo Collaboration, the KAGRA Collaboration, and R. Abbott et al.  The population of merging compact binaries inferred using gravitational waves through gwtc-3, 2022.
  * [56] Digvijay Wadekar, Javier Roulet, Tejaswi Venumadhav, Ajit Kumar Mehta, Barak Zackay, Jonathan Mushkin, Seth Olsen, and Matias Zaldarriaga.  New black hole mergers in the ligo-virgo o3 data from a gravitational wave search including higher-order harmonics, 2023.
  * [57] Ilias Cholis, Ely D. Kovetz, Yacine Ali-Haïmoud, Simeon Bird, Marc Kamionkowski, Julian B. Muñoz, and Alvise Raccanelli.  Orbital eccentricities in primordial black hole binaries.  Phys. Rev. D, 94(8):084013, 2016.
  * [58] Yi-Fan Wang and Alexander H. Nitz.  Prospects for detecting gravitational waves from eccentric subsolar mass compact binaries.  Astrophys. J., 912(1):53, 2021.
  * [59] Gabriele Franciolini and Paolo Pani.  Searching for mass-spin correlations in the population of gravitational-wave events: The GWTC-3 case study.  Phys. Rev. D, 105(12):123024, 2022.
  * [60] Mrunal Korwar and Stefano Profumo.  Updated constraints on primordial black hole evaporation.  Journal of Cosmology and Astroparticle Physics, 2023(05):054, May 2023.
  * [61] Henrike Fleischhack.  Amego-x: Mev gamma-ray astronomy in the multimessenger era, 2021.
  * [62] Manuela Mallamaci, A. De Angelis, V. Tatischeff, R. Rando, M. Tavani, U. Oberlack, R. Walter, G. Ambrosi, A. Argan, P. von Balmoos, D. Bastieri, E. Bernardini, S. Brandt, A. Bulgarelli, A. Bykov, V. Fioretti, I.A. Grenier, L. Hanlon, D. Hartmann, M. Hernanz, G. Kanbach, I. Kuvvetli, P. Laurent, M. Mariotti, M.N. Mazziotta, J. Mc Enery, S. Mereghetti, A. Morselli, K. Nakazawa, M. Pearce, E. Prandini, J. Rico, R. Curado da Silva, X. Wu, Andrzej Zdziarski, and A. Zoglauer.  All-Sky-ASTROGAM: a MeV Companion for Multimessenger Astrophysics.  PoS, ICRC2019:579, 2019.
  * [63] Simeon Bird, Andrea Albert, Will Dawson, Yacine Ali-Haïmoud, Adam Coogan, Alex Drlica-Wagner, Qi Feng, Derek Inman, Keisuke Inomata, Ely Kovetz, Alexander Kusenko, Benjamin V. Lehmann, Julian B. Muñoz, Rajeev Singh, Volodymyr Takhistov, and Yu-Dai Tsai.  Snowmass2021 cosmic frontier white paper: Primordial black hole dark matter.  Physics of the Dark Universe, 41:101231, August 2023.
  * [64] Jérémy Auffinger.  Primordial black hole constraints with hawking radiation—a review.  Progress in Particle and Nuclear Physics, 131:104040, July 2023.
  * [65] F. Hoyle and R. A. Lyttleton.  On the accretion of interstellar matter by stars.  Proceedings of the Cambridge Philosophical Society, 36(3):325, January 1940.
  * [66] Xavier Boluna, Stefano Profumo, Juliette Blé, and Dana Hennings.  Searching for exploding black holes, 2024.
  * [67] Zhen Cao et al.  Ultrahigh-energy photons up to 1.4 petaelectronvolts from 12 $\gamma$-ray Galactic sources.  Nature, 594(7861):33–36, 2021.
  * [68] J. Hinton and SWGO Collaboration.  The Southern Wide-field Gamma-ray Observatory: Status and Prospects.  In 37th International Cosmic Ray Conference, page 23, March 2022.
  * [69] Keith Vanderlinde, Adrian Liu, Bryan Gaensler, Dick Bond, Gary Hinshaw, Cherry Ng, Cynthia Chiang, Ingrid Stairs, Jo-Anne Brown, Jonathan Sievers, Juan Mena, Kendrick Smith, Kevin Bandura, Kiyoshi Masui, Kristine Spekkens, Leo Belostotski, Matt Dobbs, Neil Turok, Patrick Boyle, Michael Rupen, Tom Landecker, Ue-Li Pen, and Victoria Kaspi.  The canadian hydrogen observatory and radio-transient detector (chord).  2019\.