# An Autonomous Non-monolithic Agent with Multi-mode Exploration based on
Options Framework ††thanks: This work was supported by the Australian Research
Council through Discovery Early Career Researcher Award DE200100245 and
Linkage Project LP210301046.

JaeYoon Kim  Junyu Xuan  Christy Liang  Farookh Hussain

###### Abstract

Most exploration research on reinforcement learning (RL) has paid attention to
‘the way of exploration’, which is ‘how to explore’. The other exploration
research, ‘when to explore’, has not been the main focus of RL exploration
research. The issue of ‘when’ of a monolithic exploration in the usual RL
exploration behaviour binds an exploratory action to an exploitational action
of an agent. Recently, a non-monolithic exploration research has emerged to
examine the mode-switching exploration behaviour of humans and animals. The
ultimate purpose of our research is to enable an agent to decide when to
explore or exploit autonomously. We describe the initial research of an
autonomous multi-mode exploration of non-monolithic behaviour in an options
framework. The higher performance of our method is shown against the existing
non-monolithic exploration method through comparative experimental results.

###### Index Terms:

non-monolithic exploration, autonomous multi-mode exploration, options
framework

##  I Introduction

Exploration is the crucial part of RL algorithms because it gives an agent the
choice to uncover unknown states. There have been many RL exploration research
studies with various viewpoints, such as intrinsic reward [1], [2], [3], [4],
skill discovery [5], [6], [7], Memory base [8], [9], [10], [11], and Q-value
base [12]. Although exploration research has evolved, it has concentrated on
‘how to explore’, which is how an agent selects an exploratory action.
However, the exploration research regarding ‘when to explore’ has not been
researched in earnest.

There are two types of methodology regarding ’when to explore’, which are
monolithic exploration and non-monolithic exploration. The noise-based
monolithic exploration, a representative monolithic exploration, is that a
noise, which is usually sampled from a random distribution, is added to the
original action of a behaviour policy before putting the final action to an
environment. The original action of policy and the noise to be added act as an
exploitation and exploration respectively. Hence, the behaviour policy using
monolithic exploration is affiliated to a time-homogeneous behaviour policy.
However, in a non-monolithic exploration, the original action of a behaviour
policy is not added to a noise. They act for their own purpose at a separate
step. Therefore, the behaviour policy using a non-monolithic exploration
belongs to a heterogeneous mode-switching one (Fig. 1).

We have investigated the initial research [13] of non-monolithic exploration.
As the tentative work, there are still several limitations. Firstly, there is
only one exploration policy (we call it one-mode exploration). An agent can
require more choice of entropy of exploration mode which denotes more
exploration modes greater than one-mode exploration. Secondly, the period of
exploration to be controlled should be not fixed but variable. Thirdly, the
research takes advantage of a simple threshold hyper-parameter function, which
is named ‘homeostasis’, for the variable scale of trigger signals for
switching exploration or exploitation. However, there should be a natural
switching mechanism by using the policy itself. It also claims other informed
triggers, which are action-mismatch-based triggers and variance-based
triggers.

Figure 1: An example of noise-based monolithic exploration (left) and non-
monolithic exploration (right). The final action, which is a scalar in this
example for the well understanding explanation, denotes the action of an agent
represented with a solid circle at each step. The solid line denotes the
exploitation, an original action of a behaviour policy. The solid circle in
the noise-based monolithic exploration is a final action which combines the
original action of a behaviour policy and a sampled bounded noise at each
step. However, the solid circle in the non-monolithic exploration is defined
according to the mode of each step, i.e. exploitation which is an an original
action of a behaviour policy or exploration which is a random noise or a
policy.

In this paper, we propose an autonomous non-monolithic agent with multi-mode
exploration based on an options framework to resolve the above-mentioned
considerations. Specifically, we adopt a Hierarchical Reinforcement Learning
(HRL) as an options framework chaining together a sequence of exploration
modes and exploitation in order to achieve switching behaviours at intra-
episodic time scales. Thus, we can achieve a multi-mode exploration with the
different entropy. In order to enable autonomous switching between exploration
policies and an exploitation policy where the switching is based on intrinsic
signals, we adapt a guided exploration using a reward modification of each
switching mode. A robust optimal policy is also researched to maintain the
potential performance.

Meanwhile, for this research the following 5 questions should be answered. How
can an options framework be adopted in order to take advantage of the context
of a HRL for exploration modes and exploitation? How does an agent have the
flexibility of the exploration period? How does an agent get more entropy
choice of exploration mode? How can an agent determine the switching of non-
monolithic multi-mode exploration by itself without any subsidiary function
such as ‘homeostasis’? How does an agent avoid the inherent disturbance of a
policy but have a robust optimal policy?

It is worth mentioning that there are no similar works in the literature, so
the reference methods are partially based on the method proposed in [13] even
though their work is not based on an options framework. In the end, our
exploration method shows a better performance.

The contributions are summarized as follows.

  * •

Development of an options framework model supporting an autonomous non-
monolithic multi-mode exploration: We introduce a novel HRL model architecture
to support an autonomous non-monolithic multi-mode exploration for the first 3
research questions.

  * •

Development of a switching method for a non-monolithic exploration by using an
inherent characteristic of a policy: Our model use a guided exploration with a
reward modification for the fourth research question.

  * •

Improved robustness of the policy: A robust optimal policy can be ensured by
taking advantage of an evaluation process for the last research question.

The rest of this research is explained as follows. Section II surveys the
research of exploration and HRL related to our research. Section III explains
our proposed model. Section IV describes the experiments for the performance
measurement of our model compared with a non-monolithic model, [13], as a
reference model and a monolithic exploration, HIRO. We discuss several
acknowledged issues from the experiment in Section V. Finally, we present the
conclusion of the current research and suggestions for future works in Section
VI.

Figure 2: The architecture of our suggested model (right) compared with that
of the reference paper using a homeostasis [13] (left)

##  II Related work

###  II-A Options framework

The set of option, which is a generalized concept of action, over an MDP is
comprised of a semi-Markov decision process (SMDP). Semi-MDPs are defined to
deal with the different levels of an abstract action based on the variable
period. HRL is a representative generalization of reinforcement learning where
the environment is modelled as a semi-MDP [14].
Each action in non-monolithic exploration mode which adopts a multi-mode
exploration has different effect during the different period. Thus the
sequence of action is defined by taking advantage of an option framework for a
multi-mode exploration.

###  II-B Exploration

The various events for triggers have been considered whether they are acquired
from uncertainty or not [15], [16], [17].

The experiment of [18] shows the efficacy of robot behaviour learning from
self-exploration and a socially guided exploration supported by a human
partner. [19] claims about the Bayesian framework which supports changing
dynamics online and prevents conservativeness by using a variance bonus
uncovering the level of transition of adversity. [20] claims Tactical
Optimistic and Pessimistic (TOP) estimation for the value estimation strategy
of optimistic and pessimistic online by using a quantile approximation [21].
Hence, the belief distribution is constructed by following quantile
estimation:

$q_{\tilde{Z}^{\pi}(s,a)}=q_{\bar{Z}(s,a)}+\beta q_{\sigma(s,a)}$ (1)

where $q_{\bar{Z}(s,a)}$ and $q_{\sigma(s,a)}$ are the mean and standard
deviation of quantile estimation respectively and Z(s,a) is a return random
variable. The belief distribution is optimistic and pessimistic when
$\beta\geq 0$ and $\beta<0$ respectively.

[13] claims the importance of a non-monolithic exploration against a
monolithic exploration. Its representative non-monolithic exploration method
utilizes ‘homeostasis’ based on the difference of value function between k
steps which is referred to as the following ’value promise discrepancy’,

$D_{promise}(t-k,t):=\;\big{|}V(s_{t}-k)-\sum_{i=0}^{k-1}\gamma^{i}R_{t-i}-\gamma^{k}V(s_{t})\big{|}$
(2)

where $V(s)$ is the agent’s value estimate at state s, $\gamma$ is a discount
factor and $R$ is the reward.

The above-mentioned researches regarding the guidance-exploration framework,
robust-MDP research and the adaptive optimism inspired our research on the
basis of [13].

###  II-C Hierarchical RL

The Semi-Markov Decision Process(SMDP) takes advantage of options defining a
domain knowledge and can reuse the solutions to sub-goals [14]. [22] claims
that an Adaptive Skills, Adaptive Partitions framework supports learning near-
optimal skills which are composed automatically and concurrently with skill
partitions, in which an initial misspecified model is corrected. [23] proposes
an algorithm to solve tasks with sparse reward in which the research suggests
an algorithm to accelerate exploration with the construction of options
minimizing the cover time. [24] also deals with a sparse reward environment.
Thus, it formalizes the concept of fast and slow curiosity for the purpose of
stimulating a long-time horizon exploration. The option-critic architecture
has the intra policy, which follows the option chosen by the policy over
options until the end of the condition of option termination [25]. [26] claims
that each policy in HRL, which utilizes a flow-based deep generative model for
retaining a full expressivity, is trained through a latent variable with a
bottom-up layer-wise method. HIRO claims the method to synchronize adjacent
levels of hierarchical reinforcement learning to efficiently train the higher
level policy.

Our model makes use of HIRO for our exploration research because it is a
traditional goal-conditioned HRL.

##  III Our model

From the rising issue of value promise discrepancy used in [13], our research
pays close attention to an autonomous multi-mode non-monolithic exploration
model where an agent makes an action as to when an exploration mode starts and
exits by itself. In addition, the expected model takes advantage of its
inherent characteristic for the action. For the purpose, our research adopts
an options framework.

Table I: Key Notations.

Symbol |  Meaning
---|---
$t$ |  action step
$state$ |  current state
$next\\_state$ |  next state
$Top$ |  The highest level
$Middle$ |  The higher level
$Low$ |  The lower level
$Action$ |  The action of Low level (The action of $\pi^{TD3}_{L}$)
$target\\_pos$ |  The context of Top and Middle
$goal$ |  The current lower-level context of three sub-policies of Middle level (The current goal for $\pi^{TD3}_{L}$)
$next\\_goal$ |  The next lower-level context of three sub-policies of Middle level (The next goal for $\pi^{TD3}_{L}$)
$R$ |  The reward received from an environment (The sign of R is negative in Ant domain of OpenAI Gym)
$\pi^{PPO}_{T}$ |  The policy(on-policy) of Top level
$\pi^{TD3}_{M}$ |  The policy(off-policy) of Middle level
$\pi^{PPO}_{M}$ |  The policy(on-policy) of Middle level
$\pi^{RND}_{M}$ |  The policy (The uniform random for random policy) of Middle level
$\pi^{TD3}_{L}$ |  The policy(off-policy) of Low level
$g^{\text{expl-mode}}$ |  The action of $\pi^{PPO}_{T}$ of Top level
$\alpha_{g^{\text{expl-mode}}}$ |  The preset value of $\alpha$ according to $g^{\text{expl-mode}}$
$S\\_O_{g^{\text{expl-mode}}}$ |  The reference value of $Success\\_ratio$ according to $g^{\text{expl-mode}}$
$loss$ |  The loss of $\pi^{PPO}_{T}$ of Top level
$S\\_E$ |  The $Success\\_rate$ of evaluation function of $\pi^{TD3}$ of Middle level
$Done\\_m$ |  The count of Done during the horizon of Top level
$R\\_m$ |  The sum of R during the horizon of Top level
$Count\\_m$ |  The count during the horizon of Top level
$S\\_O\\_m$ |  The ratio of success count regarding $g^{\text{expl-mode}}$ of $\pi^{PPO}_{T}$ during the horizon of Top level
$\rho$ |  The preset value of target rate, i.e. the average number of switches of the reference model

An options framework especially in a goal-conditioned HRL is the appropriate
consideration to control the multi-mode exploration through a fully state-
dependent hierarchical policy. For the first research question proposed in
Section I, our model has three levels of HRL as shown in Fig. 2 together with
the implemented model of [13]. Our model names each of the levels according to
the height of the level: $\;Top,\;Middle\;\text{and}\;Low$. The policies are
in each level:$\;\pi^{PPO}_{T}$ for Top,$\;\pi^{TD3}_{M}$,$\;\pi^{PPO}_{M}$
and $\pi^{RND}_{M}$ for Middle and $\pi^{TD3}_{L}$ for Low.

The hierarchical control process is easy to systematically construct a multi-
mode exploration, $g^{\text{expl-mode}}$, as the option against a function
control such as homeostasis. The exploration mode policy, $\pi^{PPO}_{T}$, can
choose one of three policies of Middle level as follows

$g^{\text{expl-mode}}\sim\pi^{PPO}_{T}.$ (3)

Therefore, the value of $g^{\text{expl-mode}}$ denotes one of two exploration
modes, which are uniform random and PPO, or one exploitation which is TD3. It
also provides several control benefits for exploration as there are the
inherent characteristics of the options framework.

In order to accomplish the purpose of our research, our options framework
model comprises four elements: the inherent switching mode decision of the
policy itself, empowering more entropy degrees for exploration, a guided
exploration mode, and the use of an evaluation process for robustness.

###  III-A The inherent switching mode decision of a policy itself

Since the inherent training method of $\pi^{PPO}_{T}$ is used in our model,
one policy of Middle level can be chosen according to an option,
$g^{\text{expl-mode}}$, of $\pi^{PPO}_{T}$. $\pi^{PPO}_{T}$ is synthesizing
the reward-maximization of policy on all modes into its own policy without a
subsidiary aid. This leads to the fact that the period of both exploration and
exploitation is controlled by the inherent characteristic of an agent. In the
end, all characteristic of the non-monolithic exploration mode policy can be
integrated to the reward-maximization of policy for the second research
question in Section I. We can verify the choice of a switching mode on the
count of each exploration mode as shown in Section IV.

###  III-B Empowering more entropy choice for exploration

Our model pursues multi-mode exploration for the exploration mode policy
according to the degree of entropy of exploration mode as the degree of
optimism. Our model has two exploration modes, which are a $\pi^{RND}_{M}$ and
a $\pi^{PPO}_{M}$, and one exploitation policy, $\pi^{TD3}_{M}$, in Middle
level for the third research question in Section I. Thus, while an agent is
being trained, we hypothesize that the degree of each entropy of three
policies is as follows,

$\textsc{$\textit{H}\Bigl{(}\pi^{RND}_{M}$}\Bigr{)}>\textsc{$\textit{H}\Bigl{(}\pi^{PPO}_{M}$}\Bigr{)}>\textsc{$\textit{H}\Bigl{(}\pi^{TD3}_{M}$}\Bigr{)}$
(4)

where $\textsc{$\textit{H}\Bigl{(}\pi^{\;\text{\textbullet}}_{M}$}\Bigr{)}$
denotes the overall entropy of a policy $\pi^{\;\text{\textbullet}}_{M}$.

Our model just consumes PPO for an exploration mode so that it will be
discarded at the end of training. Our model takes care of only off-policy,
TD3, as a final target policy. Meanwhile, PPO and TD3 are trained together
whenever a data occurs due to one of three sub-policies of Middle level. If
PPO is trained to some degree, our model expects that the performance of PPO
is higher than the performance of uniform random regarding the result of
exploration.

Algorithm 1 Multi-exploration mode based on options framework

1:Initialize:

2: Set the value of $\alpha_{g^{\text{expl-mode}}}$ according to
$g^{\text{expl-mode}}$ of $\pi^{PPO}_{T}$
Set the value of $S\\_O_{g^{\text{expl-mode}}}$ according to $g^{\text{expl-
mode}}$ of $\pi^{PPO}_{T}$

3:procedure Evaluate_$\pi^{TD3}_{M}$($...,\pi^{TD3}_{M},\pi^{TD3}_{L},...$)

4: According to $\pi^{TD3}_{M}\;and\;\pi^{TD3}_{L}$, compute S_E

5:end procedure

6:procedure $\textit{Train}_{T}$($...,\;S\\_E,\;g^{\text{expl-mode}},...$)

7: if $g^{\text{expl-mode}}$ is Random uniform or PPO then

8: $loss_{final}\;=\;loss\;+\;S\\_E\;*\;loss$

9: else

10: $loss_{final}\;=\;loss\;-\;S\\_E\;*\;loss$

11: end if

12:end procedure

13:for $t=0,\ldots,T-1$ do

14: $action\leftarrow Clamp\\_Max(\pi^{TD3}_{L}(state,\;goal)+Noise)$

15: Execute action, observe $R$ and $next\\_state$

16: $Done\leftarrow Judge\\_success(state,\;target\\_pos)$

17: Increase $Done\\_m$ by 1 if Done is True

18: Increase $R\\_m$ by $R$

19: Increase $Count\\_m$ by 1

20: if the horizon of Top level then

21: $S\\_O\\_m\leftarrow Done\\_m\;/\;Count\\_m$;

22: $R_{final}\leftarrow R\\_m+\alpha_{g^{\text{expl-mode}}}*R\\_m$

23: if $S\\_O\\_m\;>=\;S\\_O_{g^{\text{expl-mode}}}$ then

24: $Done\\_m\leftarrow True$

25: end if

26: if the training time of Top level then

27: $\textit{Train}_{T}$$(...,\;S\\_E,\;g^{\text{expl-mode}},...)$

28: end if

29: if the starting mode then

30: $g^{\text{expl-mode}}\;\leftarrow\;Random\;policy$

31: else

32: $g^{\text{expl-mode}}\;\sim\;\pi^{PPO}_{T}$

33: end if

34: $R\\_m,Count\\_m,Done\\_m\;\leftarrow\;0$

35: end if

36: if the horizon of Middle level then

37: By $g^{\text{expl-mode}}$, $next\\_goal\sim\;\pi^{RND}_{M}$,
$\pi^{PPO}_{M}$ or $\pi^{TD3}_{M}$

38: if the training step of $\pi^{PPO}_{M}$ then

39: Train_$\pi^{PPO}_{M}(...)$

40: end if

41: end if

42: $state\leftarrow next\\_state$; $goal\leftarrow next\\_goal$

43: if the training step of $\pi^{TD3}_{M}$ then

44: Train_$\pi^{TD3}_{M}(...)$

45: end if

46: if the evaluation step of Middle level then

47: Evaluate_$\pi^{TD3}_{M}(...,\;\pi^{TD3}_{M},\;\pi^{TD3}_{M},...)$

48: end if

49:end for

###  III-C Guided exploration

There are two phases of a potential reward progress during the training of our
agent. Thus, our model takes a guided exploration into consideration for the
agent in order to keep the first phase. Since our model pursues an options
framework in a goal-conditioned HRL, the exploration mode policy can follow a
reward-maximizing policy so that the modification of reward $R$ from an
environment is conducted with a preset parameter $\alpha_{g^{\text{expl-
mode}}}$ as

$\displaystyle R_{final}=R+\alpha_{g^{\text{expl-mode}}}*R$ (5)

where $R_{final}$ denotes a modified reward according to a preset parameter
$\alpha_{g^{\text{expl-mode}}}$ and an environment reward $R$.

The value of $\alpha_{g^{\text{expl-mode}}}$ is differently or sometimes
equally preset according to the type of $g^{\text{expl-mode}}$ as

$\alpha_{\text{uniform\;random}}>\alpha_{\text{ppo}}>\textit{or equal
to\;}\alpha_{\text{td3}}$ (6)

where
$\alpha_{\text{uniform\;random}},\;\alpha_{\text{ppo}},\;and\;\alpha_{\text{td3}}$
denotes a preset parameter $\alpha_{g^{\text{expl-mode}}}$ of
$\;\pi^{RND}_{M}$,$\;\pi^{PPO}_{M}$ and $\pi^{TD3}_{M}$ for Middle
respectively.

Finally, since the value of $R_{final}$ is utilized in the training of the
exploration mode policy, a reward-maximized option for the fourth research
question in Section I is preferred by the exploration mode policy depending on
the value of $\alpha_{g^{\text{expl-mode}}}$. As the value of
$\alpha_{g^{\text{expl-mode}}}$ gets bigger, the occurrence probability of its
exploration mode gets smaller.

(a)

(b)

(c)

(d)

(e)

Figure 3: The count of exploration modes and exploitation and the reward and
success rate of higher level policy for our model, Ref:Uniform random, Ref:PPO
and HIRO in Ant Push

(a)

(b)

(c)

(d)

(e)

Figure 4: The count of exploration modes and exploitation and the reward and
success rate of higher level policy for our model, Ref:Uniform random, Ref:PPO
and HIRO in Ant Fall

###  III-D Evaluation for robustness

For the second phase of the potential reward progress of our agent, our model
adopts the online evaluation process to keep a robust optimal policy. The
occurrence of success rate in the online evaluation process shows that the
performance of an agent enters the second stage of reward progress in this
research. From the second stage, our agent is required to have robust optimal
policy by using online evaluation process. The online process evaluating the
off-policy, $\pi^{TD3}_{M}$, operates every preset step. Then, it outputs the
success rate, $S\\_E$, according to the type of $g^{\text{expl-mode}}$. Thus,
$loss_{final}$ of the exploration mode policy, $\;\pi^{PPO}_{T}$, for the
fifth research question in Section I is calculated in this research as

$loss_{final}=loss+S\\_E*loss$ (7)

where $loss_{final}$ is a modified loss according to $S\\_E$.

In our model, as the online value of $S\\_E$ increases, the loss of the
exploration mode policy in the mode of uniform random and online policy
becomes bigger than its online original loss.

##  IV Experiments

The control of multi-mode exploration of our model as an autonomous non-
monolithic agent is shown by the count of exploration modes and exploitation,
since their counts are critical for the analysis of our model. Each count
describes the current situation of reward-maximization of policy on all modes.
Through the analysis, we aim to answer the following crucial question. Can our
model show better performance than that of the representative model of the
reference paper, [13], and a noise-based monolithic exploration policy? We
evaluate our model and them in two tasks, Ant Push and Ant Fall, of Ant domain
of OpenAI Gym. The reference models for the comparison are two models of [13],
XU-intra(100, informed, $p^{*}$, X) and XI-intra(100, informed, $p^{*}$, X),
which are called ’Ref:Uniform random’ and ’Ref:PPO’ respectively in our
reference model111Please read the section 3.1 of [13] for the experimental
details. PPO is utilized for the intrinsic explore mode of our reference
model. A noise-based monolithic exploration policy is HIRO, which is composed
of TD3 at each level. Our model and two reference models are also implemented
based on HIRO. In order to evaluate the best performance among three models,
we have the four analysis items as follows:

  1. 1.

How many counts are assigned to each policy through whole training steps?

  2. 2.

How does the transition of our model between exploration mode and exploitation
occur compared with the forced exploration transition of reference model?

  3. 3.

How much is the difference between uniform random and on-policy as the
exploration policy of our model based on a guided exploration strategy?

  4. 4.

How much does the evaluation process influence the performance of the second
reward phase?

The results of Ant Push and Ant Fall are represented in Fig. 3 and Fig. 4
respectively. Moreover, our source and implementation details are available
online222https://github.com/jangikim2/An-Autonomous-Non-monolithic-Agent-with-
Multi-mode-Exploration-based-on-Options-Framework. Algorithm 1 shows the main
part of algorithm which is implemented on the reference code.

(a)

(b)

(c)

(d)

(e)

Figure 5: Three types of ablation study against our normal model in Ant Push

###  IV-A Comparison with the reference paper and pure off-policy

####  IV-A1 Ant Push

Our model outperforms all other models through almost all training steps. The
exploitation of our model and two reference models occurs during the most of
the training steps. The exploration mode of our model and two reference models
takes place less than the exploitation of them. HIRO shows the best
performance in the early period but quickly loses the potential through whole
training steps as the other models takes advantage of the diverse exploration
modes. The performance of ‘Ref:Uniform random’ is better than that of
‘Ref:PPO’.

The exploration mode of Uniform random of our model and ‘Ref:Uniform random’
does not take place for a long time, but for a short time and gradually.
Meanwhile, more exploration of ‘Ref:PPO’ occurs than that of ‘Ref:Uniform
random’ according to a preset target rate $\rho$ where the incessant
exploration occurs after the starting mode.

After the starting mode in Algorithm 1, the PPO exploration mode of our model
has about 3600 steps, which is more than the Uniform random exploration mode
of our model, which is about 2100 steps. Most of the PPO exploration mode of
our model occurs before 1M steps. The comparison of the total steps of the two
exploration modes and exploitation of our model is

$\textit{Total\\_Step}\Bigl{(}\pi^{TD3}_{M}\Bigr{)}>>\textit{Total\\_Step}\Bigl{(}\pi^{PPO}_{M}\Bigr{)}>\textit{Total\\_Step}\Bigl{(}\pi^{RND}_{M}\Bigr{)}$
(8)

where $\textit{Total\\_Step}\Bigl{(}\pi^{\text{\;\textbullet}}_{M}\Bigr{)}$
denotes total conducted steps of a policy of $\pi^{\text{\;\textbullet}}_{M}$.

The guided exploration strategy produces the exploration mode of our model
based on the modification of reward, Equation (5), and the ratio of success
count $S\\_O\\_m$.

The second phase in Ant PUSH task starts from the steps when the reward occurs
above -100 since the success rate, $S\\_E$, of the evaluation process occurs
from about 500K and passes 0.6 at 1M steps. The situation of collapsed reward
does not take place for a long time because of the loss modification, Equation
(7), relying on the evaluation process.

####  IV-A2 Ant Fall

Our model shows a competitive performance against all other models after 3M
steps. The preset of reward modification of Ant Fall is different from that of
Ant Push, which means that $\alpha_{\text{on-policy}}$ is equal to
$\alpha_{\text{off-policy}}$. The exploitation of our model occurs over 1M
steps less than that of the two reference models, which is different from the
situation of Ant Push. The performance of HIRO is stationary and decrease in
the latter part. Unlike Ant Push, the performance of ‘Ref:PPO’ is better than
that of ‘Ref:Uniform random’.

The exploration mode of ‘Ref:Uniform random’ and our model takes place for
longer than that of ‘Ref:Uniform random’ and our model in Ant Push. Meanwhile,
the exploration mode of ‘Ref:PPO’ and that of ‘Ref:Uniform random’ are almost
the same since a preset target rate $\rho$ for ‘Ref:Uniform random’ and
‘Ref:Uniform random’ is the same.

Although $\alpha_{\text{on-policy}}$ is equal to $\alpha_{\text{off-policy}}$,
since the ratio of success count regarding each action of the second level is
also modified, after the starting mode, the total step comparison of two
exploration mode and exploitation of our model is through whole training steps
as

$\textit{Total\\_Step}\Bigl{(}\pi^{TD3}_{M}\Bigr{)}>\textit{Total\\_Step}\Bigl{(}\pi^{PPO}_{M}\Bigr{)}>>\textit{Total\\_Step}\Bigl{(}\pi^{RND}_{M}\Bigr{)}.$
(9)

Unlike the Ant PUSH task, the second phase in the Ant Fall task suffers a drop
of reward between 4M and 4.5 M steps. The success rate of the evaluation
process stays between 0.5 and 0.6 during the period. The recovery of reward
quickly takes place due to the success rate compared with IV-B2.

###  IV-B Ablation study

We investigate our model without the reward modification, the loss
modification and both modifications. Fig. 5 shows the results of the
experiment compared with our normal model in the Ant Push task. Therefore, the
part related to our normal model in Ant Push is removed for the purpose of
experimenting each case.

####  IV-B1 Without the reward modification

While the exploitation has less steps than our normal model, the exploration
of Uniform random and PPO has more steps than our normal model. The
performance of reward and success rate slowly increase.

####  IV-B2 Without the loss modification

Again, the two exploration modes have more steps than our normal model and the
exploitation has less steps than our normal model. It shows a drop of reward
between 2.2M steps and 3M steps due to the increase in PPO exploration.
Although the success rate is better than that of our normal model during the
period, its performance is worse than that of our normal model.

####  IV-B3 Without both the reward modification and the loss modification

Too many explorations and less exploitation cause the worst performance.

##  V Discussion

###  V-A The effect of on-policy for exploration

When the on-policy operates in the beginning of exploration, the performance
of on-policy, $\;\pi^{PPO}_{M}$, is not competitive. However, after it is
trained by itself or other policies to some extent, the performance of on-
policy shows better performance than the random policy. Meanwhile, In practice
$\;\pi^{PPO}_{T}$ is likely not to suffer a local minima due to the three
policies of Middle level.

###  V-B The effect of reward modification

In Ant Fall task, the performance of our model in the early steps up to about
2M steps lags behind all other models. The reason is that the on-policy
operates for long time up to then since $\alpha_{\text{on-policy}}$ is equal
to $\alpha_{\text{off-policy}}$. The reward modification for the guided
exploration takes advantage of the fixed value of $\alpha_{g^{\text{expl-
mode}}}$, which is not an adaptive strategy.

###  V-C The effect of loss modification

The occurrence of on-policy and random policy in the Ant Fall task between 4M
and 4.5M steps gives rise to a drop in performance of the agent. In
particular, the modeling of uncertainty reflecting the success rate, $S\\_E$,
can be considered. The higher $S\\_E$ is, the lower the uncertainty is. Thus,
$S\\_E$ is related to the uncertainty.

##  VI Conclusion

In order to overcome the issues of a non-monolithic exploration, this paper
introduces an autonomous non-monolithic agent with multi-mode exploration
based on options framework. We reveal the potential of our model to follow a
behaviour thought of humans and animals. Our model takes advantage of the
difference in the degree of entropy of each exploration policy with a
guidance-exploration framework. A robust optimal policy can be expected due to
the evaluation process. The research on a guided exploration of the adaptive
strategy for the multi-mode exploration of an autonomous non-monolithic agent
is required. The further research on the modeling of $S\\_E$ in the agent is
also required for the robust optimal policy.

## References

  * [1] J. Fu, J. Co-Reyes, and S. Levine, “Ex2: Exploration with exemplar models for deep reinforcement learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [2] Y. Burda, H. Edwards, A. Storkey, and O. Klimov, “Exploration by random network distillation,” _arXiv preprint arXiv:1810.12894_ , 2018.
  * [3] H. Tang, R. Houthooft, D. Foote, A. Stooke, O. Xi Chen, Y. Duan, J. Schulman, F. DeTurck, and P. Abbeel, “#Exploration: A study of count-based exploration for deep reinforcement learning,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [4] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, and A. A. Efros, “Large-scale study of curiosity-driven learning,” _arXiv preprint arXiv:1808.04355_ , 2018.
  * [5] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine, “Diversity is all you need: Learning skills without a reward function,” _arXiv preprint arXiv:1802.06070_ , 2018.
  * [6] K. Gregor, D. J. Rezende, and D. Wierstra, “Variational intrinsic control,” _arXiv preprint arXiv:1611.07507_ , 2016.
  * [7] J. Achiam, H. Edwards, D. Amodei, and P. Abbeel, “Variational option discovery algorithms,” _arXiv preprint arXiv:1807.10299_ , 2018.
  * [8] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, and A. Bolt, “Never give up: Learning directed exploration strategies,” _arXiv preprint arXiv:2002.06038_ , 2020\.
  * [9] N. Savinov, A. Raichuk, R. Marinier, D. Vincent, M. Pollefeys, T. Lillicrap, and S. Gelly, “Episodic curiosity through reachability,” _arXiv preprint arXiv:1810.02274_ , 2018.
  * [10] A. Ecoffet, J. Huizinga, J. Lehman, K. O. Stanley, and J. Clune, “Go-explore: a new approach for hard-exploration problems,” _arXiv preprint arXiv:1901.10995_ , 2019.
  * [11] Y. Guo, J. Choi, M. Moczulski, S. Feng, S. Bengio, M. Norouzi, and H. Lee, “Memory based trajectory-conditioned policies for learning from sparse rewards,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 4333–4345, 2020.
  * [12] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy, “Deep exploration via bootstrapped dqn,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [13] M. Pislar, D. Szepesvari, G. Ostrovski, D. Borsa, and T. Schaul, “When should agents explore?” _arXiv preprint arXiv:2108.11811_ , 2021.
  * [14] R. S. Sutton, D. Precup, and S. Singh, “Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning,” _Artificial intelligence_ , vol. 112, no. 1-2, pp. 181–211, 1999.
  * [15] M. A. Wiering and H. Van Hasselt, “Ensemble algorithms in reinforcement learning,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 38, no. 4, pp. 930–936, 2008.
  * [16] J. Buckman, D. Hafner, G. Tucker, E. Brevdo, and H. Lee, “Sample-efficient reinforcement learning with stochastic ensemble value expansion,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [17] S. Flennerhag, J. X. Wang, P. Sprechmann, F. Visin, A. Galashov, S. Kapturowski, D. L. Borsa, N. Heess, A. Barreto, and R. Pascanu, “Temporal difference uncertainties as a signal for exploration,” _arXiv preprint arXiv:2010.02255_ , 2020.
  * [18] A. L. Thomaz and C. Breazeal, “Experiments in socially guided exploration: Lessons learned in building robots that learn with and without human teachers,” _Connection Science_ , vol. 20, no. 2-3, pp. 91–110, 2008.
  * [19] E. Derman, D. Mankowitz, T. Mann, and S. Mannor, “A bayesian approach to robust reinforcement learning,” in _Uncertainty in Artificial Intelligence_. PMLR, 2020, Conference Proceedings, pp. 648–658.
  * [20] T. Moskovitz, J. Parker-Holder, A. Pacchiano, M. Arbel, and M. Jordan, “Tactical optimism and pessimism for deep reinforcement learning,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 12 849–12 863, 2021.
  * [21] W. Dabney, M. Rowland, M. Bellemare, and R. Munos, “Distributional reinforcement learning with quantile regression,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 32, 2018, Conference Proceedings.
  * [22] D. J. Mankowitz, T. A. Mann, and S. Mannor, “Adaptive skills adaptive partitions (asap),” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [23] Y. Jinnai, J. W. Park, D. Abel, and G. Konidaris, “Discovering options for exploration by minimizing cover time,” in _International Conference on Machine Learning_. PMLR, 2019, Conference Proceedings, pp. 3130–3139.
  * [24] N. Bougie and R. Ichise, “Fast and slow curiosity for high-level exploration in reinforcement learning,” _Applied Intelligence_ , vol. 51, no. 2, pp. 1086–1107, 2021.
  * [25] P.-L. Bacon, J. Harb, and D. Precup, “The option-critic architecture,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 31, no. 1, 2017.
  * [26] T. Haarnoja, K. Hartikainen, P. Abbeel, and S. Levine, “Latent space policies for hierarchical reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 1851–1860.


11institutetext: 1 Scuola Superiore Meridionale, Largo San Marcellino 10,
80138 Napoli, Italy,
2 Istituto Nazionale di Fisica Nucleare, Sezione di Napoli, Complesso
Universitario di Monte S. Angelo, Via Cintia Edificio 6, 80126 Napoli, Italy
3 Department of Physics, University of Vienna, Boltzmanngasse 5, A-1090
Vienna, Austria
4 Institute of Astrophysics, FORTH, Dept. of Physics, University Campus,
GR-71003 Heraklion, Greece
5 Max-Planck-Instutut für Radioastronomie, Auf dem Hügel 69, 53121, Bonn, DE

Post-Newtonian approximation; perturbation theory; related approximations
Pulsars Gravitational waves

# Analytical coordinate time at first post-Newtonian order

Vittorio De Falco1,2<EMAIL_ADDRESS>Emmanuele Battista3
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>John Antoniadis4,5<EMAIL_ADDRESS>
###### Abstract

In this letter, we exploit the Damour-Deruelle solution to derive the
analytical expression of the coordinate time in terms of the polar angle. This
formula has advantageous applications in both pulsar timing and gravitational-
wave theory.

###### pacs:

04.25.Nx

###### pacs:

97.60.Gb

###### pacs:

04.30.-w

##  1 Introduction

During the past few decades, General Relativity (GR) has been subjected to a
number of experiments beyond the weak-field regime of the Solar System. A
crucial astrophysical testbed of GR is represented by compact binary systems.
However, due to the non-linear geometric structure of GR, the dynamics of a
binary system is contained in the gravitational field equations and it is
governed by retarded-partial-integro differential equations [1, 2, 3]. These
issues complicate subsequent calculations, making it difficult to test GR
against observations.

The aforementioned criticality has been solved via approximation strategies
and by adopting special coordinate systems (e.g., harmonic coordinates), at
the price of losing the general covariance of the GR theory. Furthermore, the
gravitational source is assumed to be post-Newtonian (PN), i.e., slowly
moving, weakly self-gravitating, and weakly stressed [1, 2]. This allows us to
apply in the near zone the PN approximation method [4], which produces
instantaneous potentials with no retardation effects [1, 2]. In addition,
assuming that the bodies are well separated, we can deal with test particles
and ordinary differential equations [2, 3]. The bodies’ motion occurs in the
Newtonian absolute Euclidean space, in which we add the PN corrections.
Moreover, these dynamical equations preserve their relativistic nature, since
they remain invariant under a global PN-expanded Lorentz transformation [2].

The PN method has been extensively employed to investigate several aspects of
the relativistic two-body dynamics, including the back-reaction of
gravitational radiation in binary pulsars [5, 6, 7, 8] and the direct
detection of gravitational waves (GWs) from coalescing compact binaries [2].
The PN formalism has also played a central role in precision tests of gravity
theories [9] and neutron star mass measurements in binary pulsars [10]. These
assessments extensively use the quasi-Keplerian analytical solution of Damour
and Deruelle, describing the quasi-elliptic 1PN-accurate GR motion of a two-
body system [11].

In this letter, we start from the latter reference to derive, for the very
first time, an analytical expression of the coordinate time, $t$, in terms of
the polar angle, $\varphi$, at the 1PN level, namely the function
$t=t(\varphi)$. The motivation for this work takes its origin from the need to
speed up our numerical simulations for computing the gravitational waveforms
and the fluxes from inspiralling binaries framed in GR (and, as recently
proposed in the literature, also in Einstein-Cartan theory [12]). As the two-
body dynamics can be readily analysed via the relative distance $R=R(\varphi)$
[11], then all dynamical quantities can depend on $\varphi$ and hence it is
natural to invest efforts for determining $t(\varphi)$.

This analytical formula can be used to replace numerical derivation schemes
currently exploited in pulsar timing software such as TEMPO [5], TEMPO2 [13],
and PINT [14]. Similarly, it may also be exploited in coherent pulsar search
algorithms [15, 16], in which a very large number of trial timing solutions
must be generated and compared with the data, in a manner similar to template
matching in ground-based GW astronomy.

##  2 Preliminaries

We consider a binary system composed of two PN weakly self-gravitating, slowly
moving, and widely separated bodies with masses $m_{1}\geq m_{2}$, total mass
$M:=m_{1}+m_{2}$, reduced mass $\mu:=\tfrac{m_{1}m_{2}}{M}$, symmetric mass
ratio $\nu:=\tfrac{\mu}{M}$, position vectors $\boldsymbol{r}_{1}$,
$\boldsymbol{r}_{2}$, and velocities $\boldsymbol{v}_{1}:=\frac{{\rm
d}\boldsymbol{r}_{1}}{{\rm d}t}$, $\boldsymbol{v}_{2}:=\frac{{\rm
d}\boldsymbol{r}_{2}}{{\rm d}t}$.

Let us define the relative position,
$\boldsymbol{R}:=\boldsymbol{r}_{1}(t)-\boldsymbol{r}_{2}(t)$, and the
relative velocity,
$\boldsymbol{V}:=\boldsymbol{v}_{1}(t)-\boldsymbol{v}_{2}(t)$, vectors. Then,
likewise the Newtonian case, the description of the motion can be simplified
by choosing a fixed orthogonal reference frame
$(\boldsymbol{x},\boldsymbol{y})$ centered in the barycenter of the binary
system, which, without loss of generality, is supposed to be static. We also
introduce $\varphi$ as the angle between $\boldsymbol{R}$ and the
$\boldsymbol{x}$-axis, which is measured counterclockwise (see Fig. 1). In
this frame, one can write

$\displaystyle\boldsymbol{r}_{1}(t)$
$\displaystyle=\frac{\mu}{m_{1}}\boldsymbol{R}(t)+\frac{\mu(m_{1}-m_{2})}{2M^{2}c^{2}}\left(V^{2}-\frac{GM}{R}\right)\boldsymbol{R}(t)$
$\displaystyle+{\rm O}\left(c^{-4}\right),$ (1a)
$\displaystyle\boldsymbol{r}_{2}(t)$
$\displaystyle=-\frac{\mu}{m_{2}}\boldsymbol{R}(t)+\frac{\mu(m_{1}-m_{2})}{2M^{2}c^{2}}\left(V^{2}-\frac{GM}{R}\right)\boldsymbol{R}(t)$
$\displaystyle+{\rm O}\left(c^{-4}\right),$ (1b)

Figure 1: The barycentric coordinate system.

where $R:=|\boldsymbol{R}|$ and $V:=|\boldsymbol{V}|$.

In the GR framework, we consider the 1PN-accurate _Damour-Deruelle treatment_
, where the total conserved energy $E=E_{0}+\frac{1}{c^{2}}E_{1}+{\rm
O}\left(c^{-4}\right)$ and the angular momentum
$\boldsymbol{J}=\boldsymbol{J}_{0}+\frac{1}{c^{2}}\boldsymbol{J}_{1}+{\rm
O}\left(c^{-4}\right)$ of the system are [11]

$\displaystyle E_{0}$ $\displaystyle:=\frac{1}{2}V^{2}-\frac{GM}{R},$ (2a)
$\displaystyle E_{1}$
$\displaystyle:=\frac{GM}{2R}\left[(3+\nu)V^{2}+\nu\left(\frac{\boldsymbol{R}\cdot\boldsymbol{V}}{R}\right)^{2}+\frac{GM}{R}\right]$
$\displaystyle+\frac{3}{8}V^{4}(1-3\nu),$ (2b) $\displaystyle J_{0}$
$\displaystyle:=|\boldsymbol{J}_{0}|=|\boldsymbol{R}\times\boldsymbol{V}|,$
(2c) $\displaystyle J_{1}$
$\displaystyle:=|\boldsymbol{J}_{1}|=|\boldsymbol{R}\times\boldsymbol{V}|\left[\frac{(1-3\nu)}{2}V^{2}+(3+\nu)\frac{GM}{R}\right],$
(2d) $\displaystyle J$
$\displaystyle:=|\boldsymbol{J}|=J_{0}+\frac{1}{c^{2}}J_{1},$ (2e)

where $|\boldsymbol{R}\times\boldsymbol{V}|=R^{2}\frac{{\rm d}\varphi}{{\rm
d}t}$ and $(\boldsymbol{R}\cdot\boldsymbol{V})=\frac{{\rm d}R}{{\rm d}t}$. It
is important to note that the first integrals (2) can be easily calculated
once the initial conditions are assigned. Therefore, at the initial time
$t_{\rm in}$ we have

$\displaystyle R(t_{\rm in})$ $\displaystyle=R_{\rm in},\quad\frac{{\rm
d}R}{{\rm d}t}(t_{\rm in})=\dot{R}_{\rm in},$ $\displaystyle\varphi(t_{\rm
in})$ $\displaystyle=\varphi_{\rm in},\quad\frac{{\rm d}\varphi}{{\rm
d}t}(t_{\rm in})=\beta\sqrt{\frac{GM}{R_{\rm in}^{3}}},$ (3)

where $0<\beta\leq 1$ with $\beta=1$ corresponding to circular orbits. Having
defined the parameters

$\displaystyle Q_{0}$ $\displaystyle:=\frac{2GM}{R_{0}},$ (4a) $\displaystyle
h_{0}$ $\displaystyle:=\frac{J_{0}}{GM},$ (4b) $\displaystyle e_{0}$
$\displaystyle:=\sqrt{1+2E_{0}h_{0}^{2}},\qquad(0\leq e_{0}<1),$ (4c)
$\displaystyle W_{0}$
$\displaystyle:=\frac{J_{1}}{J_{0}}-\frac{3}{h_{0}^{2}},$ (4d)
$\displaystyle\tilde{K}$ $\displaystyle=1-\frac{1}{c^{2}}\frac{3}{h_{0}^{2}},$
(4e)

the 1PN expansion of the radius $R=R_{0}+\frac{1}{c^{2}}R_{1}+{\rm
O}\left(c^{-4}\right)$ reads as

$\displaystyle R_{0}$
$\displaystyle:=\frac{h_{0}^{2}GM}{1+e_{0}\cos\left[\left(\varphi-\varphi_{\rm
in}\right)\tilde{K}\right]}\,,$ (5a) $\displaystyle R_{1}$
$\displaystyle:=\frac{1}{2}\left\\{G\mu-\frac{E_{0}^{2}R_{0}^{2}}{GMe_{0}}\cos\left[\left(\varphi-\varphi_{\rm
in}\right)\tilde{K}\right]\biggr{[}\nu-15\right.$
$\displaystyle\left.\left.+\frac{4W_{0}}{E_{0}}+\frac{2E_{1}}{E_{0}^{2}}\right]+4R_{0}\left[\frac{E_{0}}{2}(\nu-4)+W_{0}\right]\right\\}.$
(5b)

From the above equations, it is clear that the Damour and Deruelle solution
links the relative radius $R$ with the polar angle $\varphi$.

##  3 Analytical expression of time

We start from Eq. (2.16) in Ref. [11], which naturally expresses the time $t$
in terms of the polar angle $\varphi$ up to ${\rm O}(c^{-4})$ corrections,

${\rm d}t=\frac{{\rm d}\varphi}{\frac{H}{R^{2}}+\frac{I}{R^{3}}},$ (6)

where

$\displaystyle H$ $\displaystyle=J\left[1+(3\nu-1)\frac{E}{c^{2}}\right],$
(7a) $\displaystyle I$ $\displaystyle=(2\nu-4)\frac{GMJ}{c^{2}}.$ (7b)

Substituting the PN expansions (2), (4), and (5) into Eq. (6), we expand the
ensuing expression in power series of $1/c\to 0$ and up to the first order. In
this way, after a lengthy calculation, we obtain the PN-expanded differential
equation for the coordinate time, which, up to ${\rm O}(c^{-4})$ terms, reads
as

$\displaystyle{\rm d}t$
$\displaystyle=\frac{R_{0}^{2}}{J_{0}}\biggr{\\{}1+\frac{1}{c^{2}}\biggr{[}E_{0}(1-3\nu)+\frac{2R_{1}}{R_{0}}$
$\displaystyle\qquad\qquad-2Q_{0}(\nu-2)-\frac{J_{1}}{J_{0}}\biggr{]}\biggr{\\}}{\rm
d}\varphi.$ (8)

This amounts to solve the integral111Since the two-body equations of motion in
GR constitutes an autonomous dynamical system (i.e., independent of the time
$t$) [11], we can assume $t_{\rm in}=0$ without loss of generality.

$\displaystyle t=\int g(\tilde{K}\bar{\varphi}){\rm d}\bar{\varphi},$ (9)

where $\bar{\varphi}:=\varphi-\varphi_{\rm in}$ and

$\displaystyle g(\tilde{K}\bar{\varphi})$
$\displaystyle:=g_{1}(\tilde{K}\bar{\varphi})+g_{2}(\tilde{K}\bar{\varphi})+g_{3}(\tilde{K}\bar{\varphi}),$
(10)

with

$\displaystyle g_{1}(\tilde{K}\bar{\varphi})$ $\displaystyle:=A_{1}R_{0},$
(11a) $\displaystyle g_{2}(\tilde{K}\bar{\varphi})$
$\displaystyle:=A_{2}R_{0}^{2},$ (11b) $\displaystyle
g_{3}(\tilde{K}\bar{\varphi})$
$\displaystyle:=A_{3}\cos(\tilde{K}\bar{\varphi})R_{0}^{3}.$ (11c)

The coefficients $A_{1},A_{2},A_{3}$ occurring in Eq. (11) can be obtained by
substituting the expression (5b) into Eq. (9). In this way, we end up with

$\displaystyle A_{1}$ $\displaystyle:=\frac{4-\nu}{c^{2}h_{0}},$ (12a)
$\displaystyle A_{2}$
$\displaystyle:=\frac{J_{0}\left[c^{2}-E_{0}(\nu+7)+4W_{0}\right]-J_{1}}{c^{2}J_{0}^{2}},$
(12b) $\displaystyle A_{3}$
$\displaystyle:=-\frac{E_{0}^{2}(\nu-15)+4E_{0}W_{0}+2E_{1}}{c^{2}GME_{0}J_{0}}.$
(12c)

If we integrate Eq. (9) numerically, we get a _monotonically increasing_
function $t(\varphi)$ (see Fig. 2).

Figure 2: Numerical integration of $t(\varphi)$ for $\varphi\in[0,20\pi]$. The
following values have been used: $m_{1}=1.60\ M_{\odot}$, $m_{2}=1.17\
M_{\odot}$, $R_{\rm in}=100\frac{GM}{c^{2}}=410.83\ {\rm km}$, $\dot{R}_{\rm
in}=0$, $\beta=0.7$, $E_{0}=-6.80\times 10^{14}\ {\rm J}$, $E_{1}=1.07\times
10^{30}\ {\rm kg}$, $J_{0}=8.62\times 10^{12}\ {\rm kg\ m^{2}\ s^{-1}}$,
$J_{1}=2.57\times 10^{28}\ {\rm kg\ s}$.

It is useful to write Eq. (5a) equivalently as

$\displaystyle R_{0}=\frac{1}{B_{1}+B_{2}\cos(\tilde{K}\bar{\varphi})},$ (13)

where

$\displaystyle B_{1}$ $\displaystyle:=\frac{1}{h_{0}^{2}GM},\qquad
B_{2}:=\frac{e_{0}}{h_{0}^{2}GM},$ (14)

with $B_{1}\geq B_{2}\geq 0$ (cf. Eq. (4c)). Let us also define

$\displaystyle f_{i}(\varphi):=\int g_{i}(\tilde{K}\bar{\varphi}){\rm
d}\bar{\varphi},\quad i=1,2,3.$ (15)

The solution $t(\varphi)$ is obtained by adding the above three integrals (cf.
Eqs. (9)–(11)), which we now solve separately. For each integral, we use the
same integration strategy: we first make the substitution
$x=\tilde{K}\bar{\varphi}$ and then convert the trigonometric functions into
polynomials through _Weierstrass substitution_ $\tau=\tan(x/2)$, leading to
the following transformations:

$\displaystyle{\rm d}x$ $\displaystyle=\frac{2}{1+\tau^{2}}{\rm
d}\tau,\qquad\cos x=\frac{1-\tau^{2}}{1+\tau^{2}}.$ (16)

For the function $f_{1}$, we get

$\displaystyle f_{1}(\varphi)$
$\displaystyle=\frac{2A_{1}}{\tilde{K}}\int\frac{1}{C_{1}+C_{2}\tau^{2}}{\rm
d}\tau$
$\displaystyle=\frac{2A_{1}}{\tilde{K}}\frac{\arctan\left(\sqrt{\frac{C_{2}}{C_{1}}}\tau\right)}{\sqrt{C_{1}C_{2}}},$
(17)

where we have set (cf. Eq. (14))

$\displaystyle C_{1}$ $\displaystyle=B_{1}+B_{2}\geq 0,\qquad
C_{2}=B_{1}-B_{2}\geq 0.$ (18)

For $f_{2}$, we have the following result:

$\displaystyle f_{2}(\varphi)$
$\displaystyle=\frac{2A_{2}}{\tilde{K}}\int\frac{1+\tau^{2}}{(C_{1}+C_{2}\tau^{2})^{2}}{\rm
d}\tau$
$\displaystyle=\frac{2A_{2}}{\tilde{K}}\biggr{[}\frac{C_{1}+C_{2}}{2(C_{1}C_{2})^{3/2}}\arctan\left(\sqrt{\frac{C_{2}}{C_{1}}}\tau\right)$
$\displaystyle-\frac{\tau(C_{1}-C_{2})}{2C_{1}C_{2}\left(C_{1}+C_{2}\tau^{2}\right)}\biggr{]}.$
(19)

Finally, the expression of $f_{3}$ is given by

$\displaystyle f_{3}(\varphi)$
$\displaystyle=\frac{2A_{3}}{\tilde{K}}\int\frac{1-\tau^{4}}{(C_{1}+C_{2}\tau^{2})^{3}}{\rm
d}\tau$
$\displaystyle=\frac{2A_{3}}{\tilde{K}}\biggr{[}-\frac{3\left(C_{1}^{2}-C_{2}^{2}\right)\arctan\left(\sqrt{\frac{C_{2}}{C_{1}}}\tau\right)}{8(C_{1}C_{2})^{5/2}}$
$\displaystyle+\frac{\tau
C_{1}(3C_{1}^{2}+5C_{2}^{2})+(5C_{1}^{2}+3C_{2}^{2})C_{2}\tau^{3}}{8C_{1}^{2}C_{2}^{2}\left(C_{1}+C_{2}\tau^{2}\right)^{2}}\biggr{]}.$
(20)

Figure 3: Function $f(\varphi)$ for $\varphi\in[0,20\pi]$, where the same
numerical values as in Fig. 2 have been used.

The plot of the function
$f(\varphi):=f_{1}(\varphi)+f_{2}(\varphi)+f_{3}(\varphi)$ is shown in Fig. 3.
It is clear that $f(\varphi)$ is discontinuous, since its different periodic
branches are not smoothly connected to each other; furthermore, it does not
reproduce the behavior of Fig. 2. Therefore, we have to add to it an “
_accumulation function_ ”, which takes into account that time is a
monotonically increasing function, while trigonometric functions and their
related information are reset after each period. To this end, we define the
following _characteristic period_

$\displaystyle P_{\varphi}:=\frac{\pi}{\tilde{K}},$ (21)

and then the accumulation function reads as

$\displaystyle F_{n}(\varphi):=\begin{cases}0,&{\rm if}\
\bar{\varphi}\in[0,P_{\varphi}],\\\ 2nf(P_{\varphi}),&{\rm if}\
\bar{\varphi}\in[P_{\varphi}(2n+1),P_{\varphi}(2n+2)],\end{cases}$ (22)

where $n\in\mathbb{N}$. For a generic $\varphi$, the related value of $n$ can
be calculated considering $q:=[(\bar{\varphi}-P_{t})/P_{t}]$, where $[\cdot]$
stands for the integer part of a number. Thus, if $q$ is an even number, then
$n=(q+2)/2$; on the other hand, if $q$ is an odd number, then $n=(q+1)/2$.
Therefore, we can conclude that the correct _analytical_ form of $t(\varphi)$
is

$\displaystyle t(\varphi)=f(\varphi)+F_{n}(\varphi)+{\rm
O}\left(c^{-4}\right).$ (23)

Figure 4 shows the agreement between the numerical solution and the analytical
expression (23).

Figure 4: Function $t(\varphi)$ for $\varphi\in[0,20\pi]$, where the same
numerical values as in Fig. 2 are employed. The black continuous line
represents the numerical solution, whereas the red dashed line the analytical
expression (23).

##  4 Discussion and conclusions

The formula for the coordinate time determined here, see Eq. (23), represents
a new analytical result in the GR PN literature. The strength of our method
relies upon two fundamental steps: (1) computing integrals (cf. Eqs. (3), (3),
and (3)), which can be easily performed via a symbolic program; (2) tune the
determined solution via an accumulation function (cf. Eq. (22)), which makes
the function of point (1) continuous. The latter procedure constitutes the
original aspect of the proposed analytical strategy.

At the 1PN level, this formula can be used to replace numerical schemes due to
its simpler and faster implementation with no approximation costs.
Importantly, our method can be generalized and extended to higher PN orders,
where an analytical formula (in whatever form is presented) may improve speed
and accuracy.

We stress that the variable $\varphi$ naturally stems from the PN equations of
motion for conservative binary systems. In fact, these are usually given as
second-order differential equations of position with respect to the time $t$.
Due to the conservation of energy and angular momentum, they can be divided
into two coupled first-order ordinary differential equations of the radius $R$
and the angle $\varphi$ with respect to the time $t$. Then, it is possible to
obtain an orbital equation, where we derive $R$ with respect to $\varphi$.
This normally allows us to determine $R(\varphi)$, rather than $R(t)$. This
constitutes a very fundamental step that then permits to find out
$t(\varphi)$. We would like to underline again that the angle $\varphi$
assumes a fundamental role in dealing with the two-bodies’ equations of motion
both in Newtonian physics and in GR. Indeed, it replaces the time $t$ and in
this way all dynamical quantities can be expressed in terms of $\varphi$,
offering thus the possibility to achieve analytical results.

It is worth noting that pulsar timing recipes often make use of the inverse of
Eq. (23), i.e., the function $\varphi(t)$, whose calculation still requires a
numerical inversion. Here, our simplified solution can be exploited as a mean
of speeding up such computations. Our finding can be generally applied for
timing the orbital period of compact binary systems during the inspiral phase
[17]. In addition, this formula could be extremely useful for extracting
fundamental information from the _holy grail binary system_ , constituted by a
pulsar and a black hole, as well as for providing significant tests of gravity
[18].

###### Acknowledgements.

V.D.F. and E.B. are grateful to Gruppo Nazionale di Fisica Matematica of
Istituto Nazionale di Alta Matematica for support, to Caterina Tiburzi for
continuous help, and to Luigi Stella for illuminating discussions. The authors
thank Alessandro Ridolfi for useful suggestions. VDF acknowledges the support
of INFN sez. di Napoli, iniziative specifiche TEONGRAV. E.B. acknowledges the
support of the Austrian Science Fund (FWF) grant P32086. J.A. acknowledges the
support of the Stavros Niarchos Foundation (SNF) and the Hellenic Foundation
for Research and Innovation (HFRI) under the 2nd Call of “Science and Society
– Action Always strive for excellence – “Theodoros Papazoglou” (Project
Number: 01431).

## References

  * [1] Maggiore M. Gravitational Waves. Vol. 1: Theory and Experiments Oxford Master Series in Physics (Oxford University Press) 2007.
  * [2] Blanchet L. Living Reviews in Relativity1720142.
https://doi.org/10.12942/lrr-2014-2

  * [3] Poisson E. Will C. M. Gravity: Newtonian, Post-Newtonian, Relativistic (Cambridge University Press) 2014.
  * [4] Lorentz H. A. Droste J. The Motion of a System of Bodies under the Influence of their Mutual Attraction, According to Einstein’s Theory (Springer Netherlands, Dordrecht) 1937 pp. 330–355.
https://doi.org/10.1007/978-94-015-3445-1_11

  * [5] Taylor J. H. Weisberg J. M. The Astrophysical Journal3451989434.
  * [6] Damour T. Taylor J. H. The Astrophysical Journal3661991501.
  * [7] Kramer M., Stairs I. H. et al. Science314200697.
  * [8] Kramer M., Stairs I. H. et al. Physical Review X112021041050.
  * [9] Wex N. Testing Relativistic Gravity with Radio Pulsars (Feb. 2014).
  * [10] Özel F. Freire P. Annual Review of Astronomy and Astrophysics542016401.
  * [11] Damour T. Deruelle N. Ann. Inst. Henri Poincaré Phys. Théor431985107.
  * [12] Battista E. De Falco V. Eur. Phys. J. C822022628.
  * [13] Edwards R. T., Hobbs G. B. Manchester R. N. Monthly Notices of the Royal Astronomical Society37220061549.
  * [14] Luo J., Ransom S., Demorest P., Ray P. S., Archibald A., Kerr M., Jennings R. J., Bachetti M., van Haasteren R., Champagne C. A., Colen J., Phillips C., Zimmerman J., Stovall K., Lam M. T. Jenet F. A. The Astrophysical Journal911202145.
  * [15] Lentati L., Champion D. J., Kramer M., Barr E. Torne P. MNRAS47320185026.
  * [16] Freire P. C. C. Ridolfi A. MNRAS47620184794.
  * [17] Postnov K. A. Yungelson L. R. Living Reviews in Relativity1720143.
  * [18] Chattopadhyay D., Stevenson S., Hurley J. R., Bailes M. Broekgaarden F. MNRAS50420213682.


titleCapturing Non-Markovian Dynamics on Near-Term Quantum

titleLearning Non-Markovian Quantum Noise from
Moiré-Enhanced Swap Spectroscopy with Deep Evolutionary Algorithm

Cambridge University Press

editionSecond edition
titleMatrix Analysis

Cambridge University Press

titleConvex Optimization

titleCVXPY: A Python-embedded modeling language for convex
journaltitleJournal of Machine Learning Research

titleA rewriting system for convex optimization problems
journaltitleJournal of Control and Decision

Springer-Verlag Berlin Heidelberg

titlePerturbation theory for linear operators

The Institute of Mathematical Statistics
the Bernoulli Society

titleEmbeddable Markov Matrices
journaltitleElectron. J. Probab.

titleQuantum advantage in simulating stochastic processes

titleConditionally completely positive maps on operator algebras
journaltitleThe Quarterly Journal of Mathematics

Springer-Verlag Berlin Heidelberg

subtitleAlgebraic Curves, Algebraic Manifolds and Schemes
titleAlgebraic Geometry I

Claypool Publishers

seriesSynthesis Lectures on Mathematics and Statistics
titleJordan Canonical Form: Theory and Practice

American Mathematical Society

issn00029939, 10886826
titleDense Sets of Diagonalizable Matrices
journaltitleProceedings of the American Mathematical Society

booktitleProceedings 38th Annual Symposium on Foundations of
Computer Science
titleComputing integral points in convex semi-algebraic sets

titleInteger programming with a fixed number of variables
journaltitleMathematics of operations research

Springer US

booktitleEncyclopedia of Optimization
titleInteger programming: branch and cut algorithmsInteger
Programming: Branch and Cut Algorithms
Boston, MA

given=Michael J.W.,

Informa UK Limited

titleFinding the Kraus decomposition from a master equation and
vice versa
journaltitleJournal of Modern Optics