# Leveraging the Variance of Return Sequences for Exploration Policy

Zerong Xi
Department of Computer Science
University of Central Florida
Orlando, FL 32765, USA
<EMAIL_ADDRESS>
&Gita Sukthankar
Department of Computer Science
University of Central Florida
Orlando, FL 32765, USA
<EMAIL_ADDRESS>

###### Abstract

This paper introduces a method for constructing an upper bound for exploration
policy using either the weighted variance of return sequences or the weighted
temporal difference (TD) error. We demonstrate that the variance of the return
sequence for a specific state-action pair is an important information source
that can be leveraged to guide exploration in reinforcement learning. The
intuition is that fluctuation in the return sequence indicates greater
uncertainty in the near future returns. This divergence occurs because of the
cyclic nature of value-based reinforcement learning; the evolving value
function begets policy improvements which in turn modify the value function.
Although both variance and TD errors capture different aspects of this
uncertainty, our analysis shows that both can be valuable to guide
exploration. We propose a two-stream network architecture to estimate weighted
variance/TD errors within DQN agents for our exploration method and show that
it outperforms the baseline on a wide range of Atari games.

##  1 Introduction

Having a good exploration policy is an essential component of achieving sample
efficient reinforcement learning. Most RL applications use two heuristics,
visitation counts and time, to guide exploration. Count-based exploration
(Kocsis & Szepesvári, 2006) assumes that it is worth allocating the
exploration budget towards previously unexplored actions by awarding
exploration bonuses based on action counts. Time-based exploration (Kaelbling
et al., 1996) is usually implemented using a Boltzmann distribution that
reduces exploration during later stages of the learning process. This paper
presents an analysis of the benefits and drawbacks of weighted sequence
variance for guiding exploration; we contrast the performance of weighted
variance with the more familiar weighted temporal difference (TD) error.

Our intuition about the merits of weighted variance as a heuristic to guide
exploration is as follows. Imagine that the returns are being summed in a
potentially infinite series. Weighted variance can be computed online in order
to estimate the convergence speed of the series for a specific state-action
pair. We construct an upper bound using uncertainty, modeled as the weighted
standard deviation, as an exploration bonus to guide action selection.
Fluctuation in the return sequence may foretell greater uncertainty in the
near future returns that should be rectified through allocation of the
exploration budget. Value-based RL algorithms are particularly susceptible to
divergence, since improvements in the value function result in rapid policy
changes which in turn affect the value estimation. Unlike event counts,
weighted variance is more sensitive to the dynamics of the return sequence; if
multiple visitations yield consistent reward, weighted variance will quickly
prioritize a different state-action sequence even if the total event counts
are smaller.

However, computing weighted variance within a deep reinforcement learning
framework is a challenging problem, due to the instability of deep neural
networks. Simply computing the variance directly from output of DQN risks
overestimating the error. The second contribution of the paper is introducing
two-stream network architecture to estimate either weighted variance or TD
errors within DQN agents. Our new architecture (Variance Deep Q Networks) uses
a separate $\sigma$ stream to estimate a weighted standard deviation of the
outputs from the original stream.

##  2 Related Work

Several groups have proposed strategies for balancing exploration/exploitation
in deep reinforcement learning including 1) extensions on count-based methods
(Bellemare et al., 2016; Ostrovski et al., 2017; Tang et al., 2017); 2) noise
injection techniques (Fortunato et al., 2018; Plappert et al., 2018); 3)
improving uncertainty estimation (Osband et al., 2016; Nikolov et al., 2019);
4) driving exploration with intrinsic motivation (Chentanez et al., 2005;
Pathak et al., 2017) and 5) entropy-guided architectures (Haarnoja et al.,
2017; Hazan et al., 2019). Our proposed weighted-variance guided exploration
technique is a compatible addition to some of these other techniques (see
Section 6 for further details).

Moving from tabular to deep reinforcement learning makes the problem of
estimating quantities such as counts and variance more challenging. Ostrovski
et al. (2017) showed how count-based techniques could be generalized to neural
networks by learning a density model on state sequences; pseudocounts for
states are then derived from the density model’s recoding probability. In
contrast our model learns the weighted variance over the return sequence
rather than the state sequence.

Bootstrapped DQN (Osband et al., 2016) uses the classic bootstrap principle to
estimate uncertainty over Q-values. Agent actions are selected using a single
Q function sampled from the posterior distribution. Rather than dithering like
noise based models, bootstrapped DQN promotes deep exploration by maintaining
policy continuity with regards to the single sampled Q-function. Uncertainty
in our proposed architecture is quantified by the weighted standard deviation
of the return sequences instead of through data partitions, but like
Bootstrapped DQN, our architecture uses multiple (two) heads.

The use of randomness or noise to drive exploration is a common theme across
many approaches. NoisyNets (Fortunato et al., 2018) directly injects noise
into the weights of the neural networks; the parameters of the noise are
learned in combination with the network weights. Like NoisyNet, our
uncertainty is learned directly by the network, reducing the need for extra
hyperparameters.

Prior work has also analyzed the dynamics of value functions in reinforcement
learning. Sutton & Barto (2018) show how policy and value functions change
throughout the learning process. Like them, we believe that the variation of
the sequence is predictive of the uncertainty of the near future returns.
Variation in the return sequence can be modeled as a slower convergence trend
layered with transient fluctuations. Our method builds on this finding; see
Appendix B for an empirical analysis showing how weighted variance reacts to
the dynamics of raw, smoothed, and residual return sequences.

##  3 Background

Our aim is to learn an action policy for a stochastic world modeled by a
Markov Decision Process by balancing the exploration of new actions and the
exploitation of actions known to have a high reward. This is done by learning
a value function ($Q(s,a)$) using the discounted return information ($G(s,a)$)
and learning rate ($\alpha$):

$Q(s,a)=Q(s,a)+\alpha\cdot(G(s,a)-Q(s,a))$ (1)

Actions are selected using the learned value function. This paper illustrates
how our weighted variance exploration approach can be integrated into agents
using deep Q-learning.

Deep Q Networks (Mnih et al., 2015) utilize deep neural networks as
approximators for action-value functions in Q learning algorithms. The
updating procedure of the function is formulated as an optimization problem on
a loss function:

$L_{\text{DQN}}(\zeta)=\mathbb{E}_{(s,a,r,s^{\prime})\sim
D}\left[\left(r+\gamma\cdot\text{max}_{b\in\mathcal{A}}Q(s^{\prime},b;\zeta^{-})-Q(s,a;\zeta)\right)^{2}\right]$
(2)

where $\zeta$ are the parameters of the network, $\mathcal{A}$ is a set of
valid actions and $D$ is a distribution over a replay buffer of previously
observed transitions. A target network with parameters $\zeta^{-}$ is
regularly synchronized with $\zeta$ and used to estimate the action values of
the successor states; the use of a target network promotes estimation
stability. Since the original introduction of DQN, several improvements to the
updating procedure and network architecture have been proposed.

Double DQN (Hasselt et al., 2016) updates the network according to a different
rule in which the action for the successor state is selected based on the
target network rather than the updating network. This change alleviates the
overestimation problem by disentangling the estimation and selection of action
during optimization steps. The loss function for Double DQN is:

$L_{\text{DDQN}}=\mathbb{E}_{(s,a,r,s^{\prime})\sim
D}\left[\left(r+\gamma\cdot
Q(s^{\prime},\text{argmax}_{b\in\mathcal{A}}Q(s^{\prime},b;\zeta);\zeta^{-})-Q(s,a;\zeta)\right)^{2}\right].$
(3)

Dueling DQN (Wang et al., 2016) uses a two-stream network architecture to
separate the estimation of state value and action advantage in order to
accelerate learning in environments with a large number of valid actions. The
estimation of Q value is a function of value stream $V(\cdot,\cdot;\cdot)$ and
advantage stream $A(\cdot,\cdot;\cdot)$ such that

$Q(s,a;\zeta)=V(s,a;\zeta)+A(s,a;\zeta)-\frac{\sum_{b\in\mathcal{A}}A(s,b;\zeta)}{N_{\text{actions}}}.$
(4)

Prioritized Replay (Schaul et al., 2015) selects the experience used for the
optimation procedure from a replay buffer with a priority-based sampling
method instead of a uniform one. Sampling probabilities are proportional to
the TD errors.

Our proposed technique guides exploration and can be coupled with other
improvements to the reinforcement learning process. Although we construct our
models using the above extensions, our technique is solely for guiding
exploration and can benefit from other improvements, such as Bootstrapped DQN
(Osband et al., 2016), Distributional DQN (Bellemare et al., 2017; Dabney et
al., 2018), and Multi-Step Learning (Sutton & Barto, 2018). Moreover, it can
be combined with the other exploration methods, such as count-based methods
(Ostrovski et al., 2017) or Noisy DQN (Fortunato et al., 2018), to guide
exploration during different training stages. We discuss this further in
Section 6.

##  4 Measuring Variance for Exploration

During Monte Carlo policy evaluation, the value function $Q(s,a)$ for a
particular state-action pair is updated using a sequence of returns
$\mathcal{G}_{n}(s,a)=G_{1}(s,a),G_{2}(s,a),...,G_{n}(s,a)$. This series can
start diverging due to the cyclic nature of value-based approaches; the
evolving value function begets policy improvements which in turn modify the
value function. We believe that the agent should leverage information from
these variations to quantify uncertainty in order to explore non-optimal, but
still promising, actions. Specifically, agents can follow a greedy exploration
policy based on an upper bound:

$\pi(s)=\text{argmax}_{a\in\mathcal{A}}Q(s,a)+\sigma(s,a)\cdot c,$ (5)

where $\sigma$ is a measurement of uncertainty and $c$ is a fixed hyper-
parameter which adjusts the extent of exploration.

To measure the uncertainty of returns for a specific state-action pair, we
propose 1) a weighted variance estimation method for general RL, 2) a neural
network architecture, and 3) novel optimizing strategy which explicitly
estimates either weighted variance or weighted TD error in the DRL
configuration.

###  4.1 Reinforcement Learning with Variance Estimation

Although the vanilla form of sequence variance doesn’t reflect the higher
importance of the recent returns, we define the uncertainty as an
exponentially decaying weighted standard deviation

$\sigma_{n}(s,a)=\sqrt{\frac{\sum_{i=1}^{n}(1-\alpha)^{n-i}(G_{i}(s,a)-Q_{n}(s,a))^{2}}{\sum_{i=1}^{n}(1-\alpha)^{n-i}}},$
(6)

where $Q_{n}(s,a)$ is the value function which is updated using
$\mathcal{G}_{n}(s,a)$ and $\alpha$ (the update step size) in equation 1.

The update formula for $\sigma$ is as follows

$\sigma_{n+1}(s,a)=\sqrt{(1-\alpha)\left[\sigma_{n}^{2}(s,a)+(Q_{n+1}(s,a)-Q_{n}(s,a))^{2}\right]+\alpha(G_{n+1}(s,a)-Q_{n+1}(s,a))^{2}}.$
(7)

The first term inside the square root represents the adjusted estimation of
variance on $\mathcal{G}_{n}(s,a)$ with the updated $Q_{n+1}(s,a)$, and the
second term is the estimation from the incoming $G_{n+1}(s,a)$. The details of
the updating formula are presented in Appendix A.

When updates are performed using the above formula, $\sigma(s,a)$ is biased
during the early stage, due to the undecidable prior $\sigma_{0}(s,a)$ as well
as the bias incipient to the usage of a small $n$. We propose two strategies
for initializing the $\sigma$ function: 1) warming up with an
$\epsilon$-greedy method with $\epsilon$ decayed to 0 to ensure a gradual
transition to our exploration policy, which effectively starts with a larger
$n$; 2) initializing $\sigma_{0}(s,a)$ as a large positive value to encourage
uniform exploration during early stages, which is theoretically sound since
the variance of the value of a state-action pair is infinitely large if it has
never been visited.

###  4.2 Variance Deep Q Networks (V-DQN)

Our new algorithm, V-DQN, incorporates weighted variance into the exploration
process of training DQNs.

Algorithm 1 Variance DQN (V-DQN)

Input: exploration parameter $c$; minibatch $k$; target network update step
$\tau$;
Input: initial network parameters $\zeta$; initial target network parameter
$\zeta^{-}$;
Input: Boolean DOUBLE

1:Initialize replay memory $\mathcal{H}=\emptyset$

2:Observe $s_{0}$

3:for $t\in\\{1,...,T\\}$ do

4: Select an action
$a\leftarrow$argmax${}_{b\in\mathcal{A}}Q(s,b;\zeta)+|\sigma(s,b;\zeta)|\cdot
c$

5: Sample next state $s\sim P(\cdot|s,a)$ and receive reward $r\leftarrow
R(s,a)$

6: Store transition $(s_{t-1},a,r,s_{t})$ in $\mathcal{H}$

7: for $j\in\\{1,...,k\\}$ do

8: Sample a transition $(s_{j},a_{j},r_{j},s^{\prime}_{j})\sim D(\mathcal{H})$
$\triangleright$ $D$ can be uniform or prioritised replay

9: if $s^{\prime}_{j}$ is a terminal state then

10: $G\leftarrow r_{j}$

11: else if DOUBLE then

12:
$b^{\ast}(s^{\prime}_{j})=\text{argmax}_{b\in\mathcal{A}}Q(s^{\prime}_{j},b;\zeta)$

13: $G\leftarrow r_{j}+\gamma
Q(s^{\prime}_{j},b^{\ast}(s^{\prime}_{j});\zeta^{-})$

14: else

15: $G\leftarrow
r_{j}+\gamma\text{max}_{b\in\mathcal{A}}Q(s^{\prime}_{j},b;\zeta^{-})$

16: end if

17: $\hat{\sigma}\leftarrow G-Q(s_{j},a;\zeta)$

18: Do a gradient step with loss
$(G-Q(s_{j},a;\zeta))^{2}+({\hat{\sigma}}^{2}-\sigma^{2}(s_{j},a;\zeta))^{2}$

19: end for

20: if $t\equiv 0\;(\text{mod}\;\tau)$ then

21: Update the target network $\zeta^{-}\leftarrow\zeta$

22: end if

23:end for

Due to the known instability of deep neural networks during the training
process, it is risky to calculate the weighted variance from composing
multiple estimations (e.g., the state-action values before and after the
optimization step). Instead of computing the target variance as a byproduct,
we propose a two-stream neural network architecture along with an novel loss
function to allow end-to-end training while estimating the weighted standard
deviation. It simplifies the optimization for variance by sacrificing the
accuracy, which we believe is an acceptable compromise since the deep neural
networks with gradient descent cannot strictly follow the above updating
formula. Our empirical results demonstrate the effectiveness of the
simplification.

Variance DQN uses neural networks with a separate $\sigma$-stream to estimate
a weighted standard deviation of the outputs from the original stream on
moving targets, which is common in the context of deep reinforcement learning
where the value function improves as the policy evolves. In practice, the
$\sigma$-stream shares lower layers, e.g. convolutional layers, with the
original stream to reduce computational demands.

The loss function for Variance DQN is a sum of mean square error losses on the
original stream, which is identical to formula 2 (for DQN) or formula 3 (for
Double DQN), and the square of the $\sigma$-stream:

$\displaystyle L_{\text{V-DQN}}=\mathbb{E}_{(s,a,r,s^{\prime})\sim
D}\left[\left(G-Q(s,a;\zeta)\right)^{2}+\left((G-Q(s,a;\zeta))^{2}-\sigma^{2}(s,a;\zeta)\right)^{2}\right]$
(8) $\displaystyle s.t.\;\;\;$ $\displaystyle
G=\begin{cases}r+\gamma\cdot\text{max}_{b\in\mathcal{A}}Q(s^{\prime},b;\zeta^{-})&\text{for
DQN}\\\ r+\gamma\cdot
Q(s^{\prime},\text{argmax}_{b\in\mathcal{A}}Q(s^{\prime},b;\zeta);\zeta^{-})&\text{for
Double DQN}\end{cases}$ (9)

It is worth noting that the Q function used in the second part of the loss
function on the $\sigma$-stream doesn’t contribute to gradients directly.
Therefore, the optimization steps are in effect unchanged for the original
stream on the Q value, except for the shared lower layers. While the sign of
the $\sigma$-stream’s output is eliminated in the loss function, we need to do
the same during the exploration process. The modified exploration policy is

$\pi(s)=\text{argmax}_{a\in\mathcal{A}}Q(s,a)+|\sigma(s,a)|\cdot c$ (10)

The full procedure is shown in Algorithm 1.

We also propose a variant of our method (TD-DQN) which updates $\sigma$-stream
with absolute temporal difference error. The loss function for TD-DQN is

$L_{\text{TD-DQN}}=\mathbb{E}_{(s,a,r,s^{\prime})\sim
D}\left[\left(G-Q(s,a;\zeta)\right)^{2}+\left(|G-Q(s,a;\zeta)|-\sigma(s,a;\zeta)\right)^{2}\right]$
(11)

where $G$ is the same as Equation 9.

Both networks measure the uncertainty of the Q value based on the return
history in order to construct an upper bound for exploration policy. The
difference between the approaches can be interpreted based on their implicit
usage of different distance metrics: Euclidean (Variance DQN) vs. Manhattan
(TD-DQN). Generally, V-DQN is more sensitive to fluctuations in the return
sequence, investing a greater amount of the exploration budget to damp
variations.

There may be applications in which it is is valuable to estimate still higher
order statistics, such as the skew or kurtosis of the return sequence.
However, this sensitivity can also sabotage exploration by directing resources
away from promising areas of the state space that are slowly trending towards
convergence; overemphasizing the elimination of small variations could
ultimately result in longer training times. While the $c$ hyper-parameter in
our exploration policy adjusts the trade-off between exploration and
exploitation, the choice of the distance metrics used to measure sequence
variation determines the distribution of exploration time.

##  5 Results

To illustrate how weighted variance improves exploration, this paper first
presents results on its usage in tabular Q-learning for the Cartpole inverted
pendulum problem. Then we report the performance of our proposed techniques
(V-DQN and TD-DQN) on the Atari game benchmark. The results show that our two
stream architecture for guiding exploration with weighted variance or weighted
temporal difference outperforms the standard DDQN benchmark.

###  5.1 Cartpole

To demonstrate the effectiveness of our Variance Estimation (VE) method, we
compare it with $\epsilon$-greedy on Cartpole balancing problem. For this
experiment, we use the classic Q-learning algorithm (Watkins & Dayan, 1992)
with a tabular look-up Q-table.

The Cartpole environment has a 4-dimensional continuous state space
$\mathcal{S}=\mathbb{R}^{4}$ and a discrete action space $\mathcal{A}=${Push
Left, Push Right, Do Nothing}. It provides a reward of 1 in every time step.
The episode terminates if any of the following conditions is met: 1) the
episode length is greater than 500, 2) the pole falls below a threshold angle,
3) the cart drifts from the center beyond a threshold distance. With this
setting, the maximum accumulated reward any policy can achieve is 500. To
apply the tabular Q-learning configuration, the continuous state space is
discretized into 18,432 discrete states by dividing {Cart Position, Cart
Velocity, Pole Angle, Pole Velocity} into {$12,8,16,12$} intervals.

(a) Episode reward ($\epsilon_{\text{eval}}=0$)

(b) Episode reward ($\epsilon_{\text{eval}}=0.05$)

Figure 1: Average episode rewards in the Cartpole balancing problem. The
curves and the shadowed areas represent the means and the quartiles over 9
independent runs. The models are evaluated for 10 evaluation episodes every
200 training episodes.

With grid search, $\epsilon$-greedy achieves the best performance when the
discounting factor $\gamma=1.0$ and the exploration rate $\epsilon$ decays
from 1.0 to 0.01 in 5000 episodes.

For Variance Estimation, we experimented on both initialization methods as
well as a combination of them. A similar configuration is applied on warming
up with the $\epsilon$-greedy method in which $\epsilon$ decays from 1.0 to
0.0 during 5000 episodes. The initial standard deviation is set to 5000 for
initializing the $\sigma_{0}$ method to ensure sufficient early visits on
states. The combination method warms up with $\epsilon$-greedy while retaining
the large initial standard deviation; it uses the same hyper-parameters. The
value of $c$ is set to 1.5 for initializing the $\sigma_{0}$ method and 0.5
for the other two.

To reduce the possibility of over-fitting, we evaluate the models with an
additional environment in which the agent has a probability
$\epsilon_{\text{eval}}=0.05$ of acting randomly. All of our methods
outperform $\epsilon$-greedy consistently for both evaluation settings. When
the training time is prolonged, the baseline method generally achieves similar
scores to our methods, but requires approximately 10 times the training
episodes.

###  5.2 Atari Games

We evaluate our DQN-based algorithms on 55 Atari games from the Arcade
Learning Environment (ALE) (Bellemare et al., 2013), simulated via the OpenAI
Gym platform (Brockman et al., 2016). Defender and Surround are excluded
because they are unavailable in this platform. The baseline method (denoted as
DDQN) is DQN (Mnih et al., 2015) with all the standard improvements including
Double DQN (Hasselt et al., 2016), Dueling DQN (Wang et al., 2016) and
Prioritized Replay (Schaul et al., 2015).

Our network architecture has a similar structure to Dueling DQN (Wang et al.,
2016), but with an additional $\sigma$-stream among fully connected layers.
The 3 convolutional layers have 32 8$\times$8 filters with stride 4, 64
4$\times$4 filters with stride 2, 64 3$\times$3 filters with stride 1
respectively. Then the network splits into three streams of fully connected
layers, which are value, advantage and $\sigma$ streams. Each of the streams
has a hidden fully connected layer with 512 units. For the output layers, the
value stream has a single output while both advantage and $\sigma$ streams
have the same number of outputs as the valid actions.

The random start no-op scheme in Mnih et al. (2015) is used here in both
training and evaluation episodes. The agent repeats no-op actions for a
randomly selected number of times between 1 to 30 in the beginning to provide
diverse starting conditions to alleviate over-fitting. Evaluation takes place
after freezing the network every 250K training steps (1M frames). The scores
are the averages of episode rewards over 125K steps (500K frames) where
episodes are truncated at 27K steps (108K frames or 30 minutes of simulated
play).

We use the Adam optimizer (Kingma & Ba, 2015) with a learning rate of
$6.25\times 10^{-5}$ and a value of $1.5\times 10^{-4}$ for Adam’s $\epsilon$
hyper-parameter over all experiments. The network is optimized on a mini-batch
of 32 samples over prioritized replay buffer every 4 training steps. The
target network is updated every 30K steps.

The exploration rate of DDQN decays from 1.0 to 0.01 in 250K steps (1M frames)
and retains that value until the training ends. Our methods do not rely on
$\epsilon$-greedy so that is simply set to 0 for all the steps. Instead, the
value of $c$ impacts the actual exploration rates of our methods, which are
defined here to be the proportion of actions different from the optimal ones
based on the current Q value function. Empirically, the performance on Atari
games does not vary significantly over a wide range of $c$ values, which is an
unusual finding. A possible explanation is that most of the actions in those
games are not critical. To keep the exploration policy from drifting too far
from the exploitation policy, we set $c$ to be 0.1 for both V-DQN and TD-DQN
over all experiments to keep the average exploration rates of the majority of
the Atari games to reside roughly between 0.01 and 0.1.

A summary of the results over all 55 Atari games is reported in Table 1. The
raw and the normalized scores for individual games are compiled in Table 4 and
Table 3 in Appendix D. To compare the performance of agents over different
games, the scores are normalized with human scores

$\text{Score}_{\text{Normalized}}=100\times\frac{\text{Score}_{\text{Agent}}-\text{Score}_{\text{Random}}}{\text{Score}_{\text{Human}}-\text{Score}_{\text{Random}}}$
(12)

where both the random and the human scores were taken from Schaul et al.
(2015).

Figure 2: Improvement in normalized scores of V-DQN over DDQN in 200M frames

Figure 3: Improvement in normalized scores of TD-DQN over DDQN in 200M frames

The results clearly show that our proposed methods for guiding exploration,
V-DQN and TD-DQN, both improve on the standard DDQN benchmark. Although there
are small differences in the ranking, both versions perform well in the same
games, and underperform the benchmark in a small set of games. The mean and
median statistics do not reveal significant differences between V-DQN and TD-
DQN. Our intuition remains that V-DQN is likely to more sensitive to
fluctuations and will allocate more exploration budget to damp them out.

| DDQN | V-DQN | TD-DQN
---|---|---|---
Median | 151% | 164% | 164%
Mean | 468% | 547% | 533%
Table 1: Summary of normalized scores. See Table 3 in Appendix D for full
results.

(a) Mean normalized scores of all Atari games over 200M frames.

(b) Median normalized scores of all Atari games over 200M frames.

Figure 4: The mean and median of the normalized training curve over all 55
Atari games

##  6 Conclusion

This paper presents an analysis of the benefits and limitations of weighted
variance for guiding exploration. Both weighted convergence and its close
cousin, weighted temporal difference, can be used to quantify the rate of
convergence of the return series for specific state-action pairs. The return
dynamics of value-based reinforcement learning is particularly susceptible to
diverging as value improvements beget policy ones. This paper introduces a new
two-stream network architecture to estimate both weighted variance/TD errors;
both our techniques (V-DQN and TD-DQN) outperform DDQN on the Atari game
benchmark. We have identified two promising directions for future work in this
area: 1) unifying exploration architectures to ameliorate the cold start
issues and 2) adapting our exploration strategies to other deep reinforcement
learning models.

Addressing Cold Start with Unified Exploration While our methods capture the
divergence of return sequences, they suffer from the “cold start” problem. It
is unlikely that they will perform well for either empty or short sequences.
To address this, we propose two simple initialization methods for tabular
configurations in this paper. This issue is somewhat alleviated by the
generalization capacity inherent to function approximators like deep neural
networks. However, larger state space where most of the states will never be
visited still pose a problem. Our method can further benefit from unification
with the other exploration methods. Count-based upper confidence bound (UCB)
methods (Ostrovski et al., 2017) have a stronger effect on balancing the
visits among states in the early stage. This effect decays gradually as visits
increase. This characteristic makes it a natural complement for our sequence-
based methods. Noisy DQN (Fortunato et al., 2018) is another option that
assigns greater randomness to less visited states. Our method focuses on
promising actions whereas Noisy DQN chooses actions more randomly in those
areas of the state space. We ran some experiments on a rudimentary design in
which the linear layers of Q-value stream were replaced with noisy ones; our
preliminary results (not reported) show an improvement by hybridizing the two
architectures.

Beyond Q Learning The key intuition behind our methods is that exploration
should be guided with with a measure of historical variation. Generally, the
returns based on the estimated Q values of successive states are more
consistent than those built on purely episodic experience. Therefore, it is
natural to extend the idea of using return sequences to algorithms where state
or action values are available, such as actor-critic methods. In policy
gradient algorithms that strictly rely on observed reward, a history of
individual action preferences is a good candidate for measuring variation.

## References

  * Bellemare et al. (2016) Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.  Unifying count-based exploration and intrinsic motivation.  In _Advances in Neural Information Processing Systems_. 2016.
  * Bellemare et al. (2013) Marc G. Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , May 2013.
  * Bellemare et al. (2017) Marc G. Bellemare, Will Dabney, and Rémi Munos.  A distributional perspective on reinforcement learning.  In _Proceedings of the International Conference on Machine Learning_ , 2017.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym, 2016.
  * Chentanez et al. (2005) Nuttapong Chentanez, Andrew G. Barto, and Satinder P. Singh.  Intrinsically motivated reinforcement learning.  In _Advances in Neural Information Processing Systems_. 2005.
  * Dabney et al. (2018) Will Dabney, Mark Rowland, Marc G. Bellemare, and Rémi Munos.  Distributional reinforcement learning with quantile regression.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Fortunato et al. (2018) Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg.  Noisy networks for exploration.  In _International Conference on Learning Representations_ , 2018.
  * Haarnoja et al. (2017) Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In _Proceedings of the International Conference on Machine Learning_ , 2017.
  * Hasselt et al. (2016) Hado van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2016.
  * Hazan et al. (2019) Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest.  Provably efficient maximum entropy exploration.  In _International Conference on Learning Representations_ , 2019.
  * Kaelbling et al. (1996) Leslie Pack Kaelbling, Michael L. Littman, and Andrew W. Moore.  Reinforcement learning: A survey.  _J. Artif. Intell. Res._ , 4:237–285, 1996.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Lei Ba.  Adam: A method for stochastic optimization.  In _International Conference on Learning Representations_ , 2015.
  * Kocsis & Szepesvári (2006) Levente Kocsis and Csaba Szepesvári.  Bandit based monte-carlo planning.  _Machine Learning: ECML_ , 2006:282–293, 09 2006.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis.  Human-level control through deep reinforcement learning.  _Nature_ , Feb 2015.
  * Nikolov et al. (2019) Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause.  Information-directed exploration for deep reinforcement learning.  In _International Conference on Learning Representations_ , 2019.
  * Osband et al. (2016) Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy.  Deep exploration via bootstrapped dqn.  In _Advances in Neural Information Processing Systems_. 2016.
  * Ostrovski et al. (2017) Georg Ostrovski, Marc G. Bellemare, Aäron van den Oord, and Rémi Munos.  Count-based exploration with neural density models.  In _Proceedings of the International Conference on Machine Learning_ , 2017.
  * Pathak et al. (2017) Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell.  Curiosity-driven exploration by self-supervised prediction.  In _Proceedings of the International Conference on Machine Learning_ , 2017.
  * Plappert et al. (2018) Matthias Plappert, Rein Houthooft, Prafulla Dhariwal, Szymon Sidor, Richard Y. Chen, Xi Chen, Tamim Asfour, Pieter Abbeel, and Marcin Andrychowicz.  Parameter space noise for exploration.  In _International Conference on Learning Representations_ , 2018.
  * Schaul et al. (2015) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized Experience Replay.  _Proceedings of the International Conference on Learning Representations_ , 2015.
  * Sutton & Barto (2018) Richard S. Sutton and Andrew G. Barto.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA, 2018.  ISBN 0262039249.
  * Tang et al. (2017) Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAI Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel.  #exploration: A study of count-based exploration for deep reinforcement learning.  In _Advances in Neural Information Processing Systems_. 2017.
  * Wang et al. (2016) Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Van Hasselt, Marc Lanctot, and Nando De Frcitas.  Dueling Network Architectures for Deep Reinforcement Learning.  _the International Conference on Machine Learning_ , 2016.
  * Watkins & Dayan (1992) Christopher J. C. H. Watkins and Peter Dayan.  Q-learning.  In _Machine Learning_ , 1992.

##  Appendix A Updating Formula in Variance Estimation

Here we show that the weighted standard deviation in equation 6 can be
obtained by updating $\sigma$ with formula 7 in Variance Estimation. For
simplicity, the inputs to all the following functions are hidden as they are
the same state-action pair $(s,a)$.

While being updated with equation 1, we have

$\displaystyle Q_{n}$
$\displaystyle=(1-\alpha)^{n}Q_{0}+\sum_{i=1}^{n}\alpha(1-\alpha)^{n-i}G_{i}$
(13) $\displaystyle=\alpha\sum_{i=1}^{n}(1-\alpha)^{n-i}G_{i}$ (14)

The equal sign in formula 14 is established by setting $Q_{0}=0$.

If we expand $1/\alpha$ as a power series

$\displaystyle\frac{1}{\alpha}$
$\displaystyle=\frac{1-(1-\alpha)^{n}}{1-(1-\alpha)}\cdot\frac{1}{1-(1-\alpha)^{n}}$
(15)
$\displaystyle=\sum_{i=0}^{n-1}(1-\alpha)^{i}\cdot\frac{1}{1-(1-\alpha)^{n}}$
(16) $\displaystyle\approx\sum_{i=0}^{n-1}(1-\alpha)^{i}$ (17)
$\displaystyle=\sum_{i=1}^{n}(1-\alpha)^{n-i}$ (18)

The approximate sign is used here to reflect that the second term of the
multiplication approximates 1 as $n\xrightarrow{}\infty$ because
$\alpha\in(0,1]$. The value function can be re-written as an exponentially
weighted sum of the return sequence by applying the above expansion of power
series:

$Q_{n}\approx\frac{\sum_{i=1}^{n}(1-\alpha)^{n-i}G_{i}}{\sum_{i=1}^{n}(1-\alpha)^{n-i}}$
(19)

We denote $A_{n}=\sum_{i=1}^{n}(1-\alpha)^{n-i}$, then

$\displaystyle\sigma_{n+1}^{2}$
$\displaystyle=\frac{\sum_{i=1}^{n+1}(1-\alpha)^{n+1-i}(G_{i}-Q_{n+1})^{2}}{A_{n+1}}$
(20)
$\displaystyle=\frac{\sum_{i=1}^{n}(1-\alpha)^{n+1-i}(G_{i}-Q_{n+1})^{2}}{A_{n+1}}+\frac{(G_{n+1}-Q_{n+1})^{2}}{A_{n+1}}$
(21)
$\displaystyle=(1-\alpha)\cdot\frac{\sum_{i=1}^{n}(1-\alpha)^{n-i}(G_{i}-Q_{n+1})^{2}}{A_{n+1}}+\frac{(G_{n+1}-Q_{n+1})^{2}}{A_{n+1}}$
(22)
$\displaystyle=(1-\alpha)\cdot\frac{\sum_{i=1}^{n}(1-\alpha)^{n-i}(G_{i}-Q_{n}+Q_{n}-Q_{n+1})^{2}}{A_{n+1}}+\frac{(G_{n+1}-Q_{n+1})^{2}}{A_{n+1}}$
(23) $\displaystyle\begin{aligned}
=&(1-\alpha)\cdot\left[\frac{\sum_{i=1}^{n}(1-\alpha)^{n-i}(G_{i}-Q_{n})^{2}}{A_{n+1}}+\frac{\sum_{i=1}^{n}(1-\alpha)^{n-i}(Q_{n}-Q_{n+1})^{2}}{A_{n+1}}\right.\\\
&\left.+\frac{2\sum_{i=1}^{n}(1-\alpha)^{n-i}(G_{i}-Q_{n})(Q_{n}-Q_{n+1})}{A_{n+1}}\right]+\frac{(G_{n+1}-Q_{n+1})^{2}}{A_{n+1}}\end{aligned}$
(24)
$\displaystyle\approx(1-\alpha)\cdot\frac{A_{n}}{A_{n+1}}\cdot\left[\sigma_{n}^{2}+(Q_{n+1}-Q_{n})^{2}+0\right]+\frac{(G_{n+1}-Q_{n+1})^{2}}{A_{n+1}}$
(25)
$\displaystyle\approx(1-\alpha)\left[\sigma_{n}^{2}+(Q_{n+1}-Q_{n})^{2}\right]+\alpha(G_{n+1}-Q_{n+1})^{2}$
(26)

where the first approximate sign comes from formula 19 and the second one
comes from formula 17.

Though the updating formula is biased as $n$ doesn’t approach infinity in
practice, the bias is negligible. Because all the above approximations are
rooted in formula 17 and $(1-\alpha)^{n}$ converges to 0 rapidly as $n$ grows.

##  Appendix B An Empirical Analysis

(a) Return sequence

(b) $\sigma$ value

Figure 5: Raw return sequences (a) and corresponding $\sigma$ values (b) over
visitations.

(a) Return sequence

(b) $\sigma$ value

Figure 6: Smoothed return sequences (a) and corresponding $\sigma$ values (b)
over visitations.

(a) Return sequence

(b) $\sigma$ value

Figure 7: Residual return sequences (a) and corresponding $\sigma$ values (b)
over visitations.

Sutton & Barto (2018) prove that the policy and value functions monotonically
improve over sweeps in policy iteration. In this simple form, return sequences
for individual state-action pairs are monotonically increasing and usually
converge fast. However, the same statement is not true when the policy is
updated with Monte Carlo settings where the returns are random samples. In
this situation, fluctuations in returns are inevitable due to the stochastic
property; an incremental updating scheme is adopted to stabilize learning as
well as avoid the excessive impact of malicious samples.

Here we illustrate the characteristics of the return sequences and analyze how
variance guides exploration. The raw return sequences shown in Figure 5(a) are
randomly sampled from frequently visited state-action pairs in the Cartpole
balancing problem and truncated to be of the same length. To disentangle the
impact of the convergence trend from transient fluctuations, smoothed versions
of those sequences are extracted from the raw sequences (shown in Figure
6(a)), while the residuals are shown in Figure 7(a). Then we apply our
Variance Estimation method on each of the sequences independently and show the
$\sigma$ values over visits.

Since the $\sigma$ values are directly integrated into the Q values used by
our exploration policy (shown in Equation 5), a greater $\sigma$ value usually
results in an increase in exploration budget. Meanwhile, the nature of
weighted sum balances the importance of learned Q value and the history of its
change which is captured by $\sigma$.

In Figure 6(b), we observe that the $\sigma$ value is greater when the return
sequence changes at a faster rate. As the sequence converges, the $\sigma$
value approaches zero. An interesting but unobvious observation is that the
$\sigma$ value spikes when there is change in the convergence rate of the
return sequence. Since variance is essentially a Euclidean distance metric, it
is capable of capturing second-order information.

Sequences in Figure 7(a) isolate the impacts of transient fluctuations from
the overall trend. We observe that whenever there is an excessive fluctuation,
the $\sigma$ value spikes to a high magnitude to demand immediate exploration.
Once the return goes back to its normal range, the $\sigma$ value decreases
simultaneously. Those quick responses are useful since excessive fluctuations
are harmful to the estimation of Q values. A timely investigation of
exploration budget eliminates this negative impact before it propagates to
more states. Meanwhile, frequent fluctuations beget an increase in $\sigma$
value and result in more exploration to determine its value.

In conclusion, the exploration policy on our constructed upper bound
effectively allocates exploration budget to accelerate convergence in
important states as well as alleviate the impact of fluctuations.

##  Appendix C Hyperparameters

Table 2: Atari DQN Hyperparameters Hyperparameter | Value |  Description
---|---|---
$c_{\text{V-DQN}}$ | 0.1 |  Weighting factor of $\sigma$-stream in exploration policy in V-DQN
$c_{\text{TD-DQN}}$ | 0.1 |  Weighting factor of $\sigma$-stream in exploration policy in TD-DQN
mini-batch size | 32 |  Size of mini-batch sample for gradient step
replay buffer size | 1M |  Maximum number of transitions stored in the replay buffer
initial replay buffer size | 50K |  Number of transitions stored in the replay buffer before optimization starts
optimization frequency | 4 |  Number of actions the agent takes between successive network optimization steps
update frequency | 30000 |  Number of steps between consecutive target updates
$\epsilon_{\text{init}}$ | 1.00 |  Initial exploration rate of $\epsilon$-greedy method
$\epsilon_{\text{final}}$ | 0.01 |  Final exploration rate of $\epsilon$-greedy method
$N_{\epsilon}$ | 1M |  Number of actions that the exploration rate of $\epsilon$-greedy method decays from initial value to final value
$\alpha$ | 0.0000625 |  Adam optimizer learning rate
$\epsilon_{\text{ADAM}}$ | 0.00015 |  Adam optimizer parameter
evaluation frequency | 250K |  Number of actions between successive evaluation runs
evaluation length | 125K |  Number of actions per evaluation run
evaluation episode length | 27K |  Maximum number of action in an episode in evaluation runs
max no-op | 30 |  Maximum number of no-op actions before the episode starts

##  Appendix D Experimental Results on Atari Games

Figure 8: Training curve on Atari games from a single training run each. Episodes start with up to 30 no-op actions. Each data point is an average of episode rewards from 500K frames of evaluation runs, and smoothed over 10 data points. Game | DDQN | V-DQN | TD-DQN
---|---|---|---
Alien | 123% | 130% | 76%
Amidar | 99% | 97% | 97%
Assault | 764% | 1659% | 1324%
Asterix | 537% | 1280% | 1490%
Asteroids | 2% | 1% | 1%
Atlantis | 7445% | 7334% | 7714%
BankHeist | 165% | 158% | 170%
BattleZone | 168% | 199% | 149%
BeamRider | 143% | 153% | 156%
Berzerk | 51% | 49% | 49%
Bowling | 7% | 52% | 18%
Boxing | 902% | 903% | 904%
Breakout | 1764% | 1851% | 1644%
Centipede | 57% | 64% | 64%
ChopperCommand | 39% | 237% | 111%
CrazyClimber | 639% | 658% | 646%
DemonAttack | 563% | 1415% | 1700%
DoubleDunk | 1568% | -145% | -90%
Enduro | 267% | 284% | 281%
FishingDerby | 151% | 164% | 164%
Freeway | 130% | 131% | 131%
Frostbite | 108% | 116% | 112%
Gopher | 1071% | 1689% | 1665%
Gravitar | 14% | 22% | 25%
Hero | 83% | 81% | 82%
IceHockey | 139% | 227% | 253%
Jamesbond | 890% | 1315% | 1071%
Kangaroo | 354% | 529% | 552%
Krull | 909% | 917% | 931%
KungFuMaster | 150% | 199% | 214%
MontezumaRevenge | -1% | -0% | -1%
MsPacman | 32% | 33% | 29%
NameThisGame | 207% | 210% | 218%
Phoenix | 134% | 852% | 531%
Pitfall | 5% | 5% | 5%
Pong | 116% | 115% | 115%
PrivateEye | -1% | -1% | -1%
Qbert | 146% | 150% | 160%
Riverraid | 136% | 152% | 156%
RoadRunner | 826% | 827% | 724%
Robotank | 1006% | 1071% | 1051%
Seaquest | 6% | 122% | 226%
Skiing | 43% | 37% | 35%
Solaris | -5% | 50% | 40%
SpaceInvaders | 889% | 1527% | 1318%
StarGunner | 922% | 984% | 760%
Tennis | 197% | 147% | 146%
TimePilot | 387% | 365% | 366%
Tutankham | 215% | 108% | 109%
UpNDown | 366% | 516% | 707%
Venture | -1% | 36% | 18%
VideoPinball | 425% | 425% | 425%
WizardOfWor | 168% | 309% | 206%
YarsRevenge | 61% | 98% | 95%
Zaxxon | 151% | 175% | 183%
Table 3: Normalized scores. Game | Random | Human | DDQN | V-DQN | TD-DQN
---|---|---|---|---|---
Alien | 128.3 | 6371.3 | 7807.3 | 8236.9 | 4895.4
Amidar | 11.8 | 1540.4 | 1521.6 | 1495.0 | 1494.0
Assault | 166.9 | 628.9 | 3697.5 | 7829.8 | 6284.9
Asterix | 164.5 | 7536.0 | 39782.0 | 94509.1 | 110013.6
Asteroids | 871.3 | 36517.3 | 1464.3 | 1317.6 | 1278.0
Atlantis | 13463.0 | 26575.0 | 989675.0 | 975100.0 | 1024975.0
BankHeist | 21.7 | 644.5 | 1050.1 | 1007.7 | 1082.3
BattleZone | 3560.0 | 33030.0 | 53153.8 | 62133.3 | 47460.0
BeamRider | 254.6 | 14961.0 | 21296.0 | 22765.7 | 23234.0
Berzerk | 196.1 | 2237.5 | 1228.1 | 1205.0 | 1190.7
Bowling | 35.2 | 146.5 | 42.7 | 93.6 | 54.7
Boxing | -1.5 | 9.6 | 98.6 | 98.7 | 98.8
Breakout | 1.6 | 27.9 | 465.5 | 488.3 | 434.1
Centipede | 1925.5 | 10321.9 | 6695.1 | 7271.9 | 7282.8
ChopperCommand | 644.0 | 8930.0 | 3900.0 | 20289.7 | 9835.0
CrazyClimber | 9337.0 | 32667.0 | 158346.2 | 162828.0 | 159952.0
DemonAttack | 208.3 | 3442.8 | 18418.2 | 45977.5 | 55200.6
DoubleDunk | -16.0 | -14.4 | 9.1 | -18.3 | -17.4
Enduro | -81.8 | 740.2 | 2113.5 | 2250.2 | 2225.0
FishingDerby | -77.1 | 5.1 | 46.6 | 57.7 | 57.7
Freeway | 0.1 | 25.6 | 33.2 | 33.6 | 33.6
Frostbite | 66.4 | 4202.8 | 4516.2 | 4883.8 | 4702.4
Gopher | 250.0 | 2311.0 | 22331.4 | 35061.4 | 34555.7
Gravitar | 245.5 | 3116.0 | 637.5 | 869.5 | 976.1
Hero | 1580.3 | 25839.4 | 21606.6 | 21244.0 | 21476.5
IceHockey | -9.7 | 0.5 | 4.5 | 13.5 | 16.1
Jamesbond | 33.5 | 368.5 | 3014.2 | 4438.8 | 3622.5
Kangaroo | 100.0 | 2739.0 | 9450.0 | 14057.6 | 14679.4
Krull | 1151.9 | 2109.1 | 9855.5 | 9930.0 | 10065.0
KungFuMaster | 304.0 | 20786.8 | 31070.7 | 40984.4 | 44177.4
MontezumaRevenge | 25.0 | 4182.0 | 0.0 | 9.1 | 3.3
MsPacman | 197.8 | 15375.0 | 5038.5 | 5246.7 | 4585.7
NameThisGame | 1747.8 | 6796.0 | 12181.1 | 12359.4 | 12774.7
Phoenix | 1134.4 | 6686.2 | 8568.6 | 48424.7 | 30623.9
Pitfall | -348.8 | 5998.9 | 0.0 | 0.0 | 0.0
Pong | -18.0 | 15.5 | 20.8 | 20.6 | 20.6
PrivateEye | 662.8 | 64169.1 | 100.0 | 200.0 | 100.0
Qbert | 183.0 | 12085.0 | 17551.4 | 18051.7 | 19250.0
Riverraid | 588.3 | 14382.2 | 19322.9 | 21545.0 | 22073.8
RoadRunner | 200.0 | 6878.0 | 55381.7 | 55437.1 | 48526.7
Robotank | 2.4 | 8.9 | 67.8 | 72.0 | 70.7
Seaquest | 215.5 | 40425.8 | 2789.8 | 49230.8 | 91277.1
Skiing | -15287.4 | -3686.6 | -10314.1 | -10990.9 | -11215.2
Solaris | 2047.2 | 11032.6 | 1572.0 | 6497.6 | 5615.0
SpaceInvaders | 182.6 | 1464.9 | 11580.8 | 19757.7 | 17086.2
StarGunner | 697.0 | 9528.0 | 82076.7 | 87577.8 | 67768.6
Tennis | -21.4 | -6.7 | 7.5 | 0.2 | 0.0
TimePilot | 3273.0 | 5650.0 | 12460.7 | 11941.4 | 11975.0
Tutankham | 12.7 | 138.3 | 283.0 | 147.8 | 150.1
UpNDown | 707.2 | 9896.1 | 34346.4 | 48118.3 | 65673.9
Venture | 18.0 | 1039.0 | 9.6 | 382.9 | 197.1
VideoPinball | 20452.0 | 15641.1 | 584388.2 | 632013.8 | 631348.0
WizardOfWor | 804.0 | 4556.0 | 7115.1 | 12388.1 | 8551.7
YarsRevenge | 1476.9 | 47135.2 | 29332.9 | 46319.6 | 44894.4
Zaxxon | 475.0 | 8443.0 | 12488.4 | 14409.8 | 15028.3
Table 4: Raw Scores.


# Oscillator Strengths for Ultraviolet Transitions in P II – The Multiplet at
1308 Å

M.S. Brown  Department of Physics and Astronomy, University of Toledo, Toledo,
OH 43606  R.B. Alkhayat  Department of Physics and Astronomy, University of
Toledo, Toledo, OH 43606  R.E. Irving  Department of Physics and Astronomy,
University of Toledo, Toledo, OH 43606  N. Heidarian  Department of Physics
and Astronomy, University of Toledo, Toledo, OH 43606  J. Bancroft Brown
Department of Physics and Astronomy, University of Toledo, Toledo, OH 43606
S.R. Federman  Department of Physics and Astronomy, University of Toledo,
Toledo, OH 43606  S. Cheng  Department of Physics and Astronomy, University of
Toledo, Toledo, OH 43606  L.J. Curtis  Department of Physics and Astronomy,
University of Toledo, Toledo, OH 43606

###### Abstract

We report lifetimes, branching fractions, and the resulting oscillator
strengths for transitions within the P II multiplet (3$s^{2}$3$p^{2}$
${}^{3}P$ $-$ 3$s$3$p^{3}$ ${}^{3}P^{\rm o}$) at 1308 Å. These comprehensive
beam-foil measurements, which are the most precise set currently available
experimentally, resolve discrepancies involving earlier experimental and
theoretical results. Interstellar phosphorus abundances derived from
$\lambda$1308 can now be interpreted with greater confidence. In the course of
our measurements, we also obtained an experimental lifetime for the 3$p$4$s$
${}^{3}P_{0}^{\rm o}$ level of P IV. This lifetime agrees well with the
available theoretical calculation.

atomic data — ISM: abundances — ISM: atoms — methods:laboratory — ultraviolet:
ISM

##  1 Introduction

Singly$-$ionized phosphorus, which is the element’s dominant charge state in
neutral diffuse interstellar gas, is used to probe depletion onto interstellar
grains in the Galaxy (e.g., Jura & York, 1978; Dufton et al., 1986; Jenkins et
al., 1986; Lebouteiller et al., 2005; Cartledge et al., 2006; Jenkins, 2009)
and in the Magellanic Clouds (e.g., Mallouris et al., 2001; Tchernyshyov et
al., 2015). Knowledge of the amount of depletion in terms of gas density then
allows studies of metallicity and the nucleosynthetic history in more distant
galaxies and damped Lyman-$\alpha$ systems (e.g., Molaro et al., 2001;
Levshakov et al., 2002; Pettini et al., 2002; Dessauges-Zavadsky et al., 2007;
Som et al., 2015; De Cia et al., 2016) via P II absorption. Two transitions
originating from the ground state ($\lambda\lambda$1153, 1302) in this ion are
the most commonly used ones for Galactic studies (e.g., Jenkins et al., 1986;
Cartledge et al., 2006; Jenkins, 2009). These transitions have moderate to
moderately large oscillator strengths ($f$-values), as noted in the most
recent compilation of (Morton, 2003); the line at 1302 Å is the weaker of the
two. Morton recommended the theoretical $f$-values of (Hibbert, 1988) for the
multiplet ($\lambda$1308) containing the line at 1302 Å. Hibbert’s computed
lifetimes for the upper levels 3$s$3$p^{3}$ ${}^{3}P_{j}^{\rm o}$ are in close
agreement with the experimental results of Livingston et al. (1975), who
quoted about 20% accuracy. Other experimental results (Savage & Lawrence,
1966; Curtis et al., 1971; Smith, 1978) for this multiplet, however, differ
considerably (factors of 2) from those of Livingston et al. (1975). The early
theoretical work of Beck & Sinanoğlu (1972) agreed reasonably well with the
experimental results available at the time (Savage & Lawrence, 1966; Curtis et
al., 1971). The theoretical results of Brage et al. (1993) lie between those
of Curtis et al. (1971) and Livingston et al. (1975). More recent large-scale
calculations do not provide a resolution. While the results of Tayal (2003)
and Froese Fischer et al. (2006) agree very well, the $f$-values are nearly
twice those given by Hibbert (1988). The importance of P II oscillator
strengths in astronomical studies is discussed in the recent compilation by
Cashman et al. (2017), who prefer using the results from Froese Fischer et al.
(2006).

These conflicting results have forced astronomers to derive empirical
$f$-values from interstellar spectra, but the situation remains
unsatisfactory. Based on their comparison of $b$-values for interstellar S II
and P II lines, Harris & Mas Hesse (1986) suggested a value of 9.2 for the
ratio of $f$-values for the lines at 1153 and 1302 Å. Dufton et al. (1986), on
the other hand, adopted earlier calculations by Hibbert (1986) and in the
analysis of their data that resulted in a ratio of 22. This is similar to the
ratio obtained by Hibbert (1988). For comparison, Brage et al. (1993) obtained
a ratio of 11.2, Tayal (2003) a ratio of 11.5, and Froese Fischer et al.
(2006) a ratio of 12.1.

We seek a resolution to the question of the most appropriate $f$-values to use
for interstellar studies by extending our earlier results on the multiplet
containing the line at 1153 Å (Federman et al., 2007) through measurements of
the pertinent data for the multiplet $\lambda$1308\. Through beam-foil
spectroscopic techniques, we present experimental lifetimes and branching
fractions to derive $f$-values. Our empirical results are the most
comprehensive to date and are of sufficient accuracy to discern the most
consistent theoretical efforts.

##  2 Experimental Details

The measurements were performed at the Toledo Heavy Ion Accelerator. Details
describing this facility can be found in our earlier papers (e.g., Federman et
al., 1992; Haar et al., 1993; Schectman et al., 2000). Here we focus on the
specfics relevant to the P II data for the multiplet at $\lambda$1308\. As in
our earlier work on P II (Federman et al., 2007), phosphorus ions were
produced utilizing a low-temperature oven and subsequent charging through
interactions in an argon plasma. Lifetime measurements and the spectra for
branching fractions were obtained using energies of 170 and 220 keV. Because
the lines in this multiplet were relatively weak, only forward decay curves
were acquired (i.e., moving away from the slit of the monochromator). Beam
divergence, foil thickening, and nuclear scattering were investigated while
interpreting our data (see Federman et al., 2007). Our ion source was
typically set to produce P+ prefoil beam currents of 100 nA. These ions passed
through carbon foils whose thicknesses were 2.4 $\mu$g cm-2 and emerged both
in a variety of charge states and excited states. An Acton 1 m normal-
incidence vacuum ultraviolet monochromator with 2400 line mm-1 gratings blazed
at 800 Å was used for the measurements. Lifetime measurements and the spectrum
of P IV were obtained with one grating; the P II spectra were obtained with a
newer grating of the same specifications. The only noticeable effect involved
the instrumental linewidth, which changed from 0.22 to 0.27 Å. A Galileo
channeltron was utilized to detect the emitted radiation at the exit slit of
the monochromator.

Blending among spectral features and the relative weakness of the lines of
interest prevented us from performing a complete set of measurements. We
instead focused on those which would yield accurate results. Lifetimes for the
$J$ $=$ 1 and 2 levels in the upper state 3$s$3$p^{3}$ ${}^{3}P^{\rm o}$ were
obtained from decay curves involving transitions at 1301.87 and 1310.70 Å,
respectively, to the ground state 3$s^{2}$3$p^{2}$ ${}^{3}P$. Systematic
effects were studied by the acquisition of decay curves for $J$ $=$ 2 at two
energies. Multiexponential fits were used to extract lifetimes. However, in
most instances a single exponential sufficed. Furthermore, the error bars
associated with the lifetimes take into account both statistical and
experimental uncertainties. The decay curve for the $J$ $=$ 2 line obtained in
the forward direction was best fit by two exponentials; we ascribe the short-
lived decay component of the fit to a transition in P IV at 655 Å seen in
second order (3$p^{2}$ ${}^{3}P_{1}$ – 3$p$4$s$ ${}^{3}P_{0}^{\rm o}$). We
only saw the effects of P IV in one decay curve because the feature is present
on the red side of the P II line and therefore depended on the placement of
the grating when peaking up on $\lambda$1310\. Additional measurements at 205
keV were made on the P IV line at 655 Å for further confirmation of its short
lifetime. Figure 1 shows the decay curve for the ${}^{3}P_{2}$ $-$
${}^{3}P_{2}^{\rm o}$ line of P II at 1310 Å at a beam energy of 170 keV.

Cascades from higher lying levels, which affect the populations of the levels
of interest, could impact our lifetime measurements. Large-scale computations
(Hibbert, 1988; Tayal, 2003; Froese Fischer et al., 2006), however, suggest
that repopulation is not a concern in our experiment. These calculations
indicate that decays from the 3$s^{2}$3$p$4$p$ ${}^{3}D$, ${}^{3}P$, and
${}^{3}S$ states may repopulate the 3$s$3$p^{3}$ ${}^{3}P^{\rm o}$ state, with
typical lifetimes of 7 to 12 ns. We do not find clear evidence for lifetimes
in this range from our multiexponential fits at the precision of our
measurements. More importantly, Hibbert (1988) and Froese Fischer et al.
(2006) determined multiplet Einstein $A$ coefficents for transitions between
these states that were less than a few times $10^{7}$ s-1; this suggests the
transitions may be too weak to play a significant role in the repopulation of
the levels in question.

When more than one decay channel is present, a combination of lifetimes and
branching fractions are needed to derive oscillator strengths. The theoretical
results of Hibbert (1988), Tayal (2003), and Froese Fischer et al. (2006)
indicate that intercombination lines arising from the 3$s$3$p^{3}$
${}^{3}P^{\rm o}$ state are weak and are below the sensitivity of our
experiment. Therefore, we focused on branching among the dipole-allowed
transitions between 1300 and 1312 Å. Two scans were required to cover the
multiplet in which there was an overlap in the region between 1304 and 1306 Å.
The overlap region was utilized to normalize the spectra covering the
multiplet. The added signal in this portion of the spectrum aided in fitting
the blend of lines. A further complication was the numerous weak lines from
the P IV multiplet $\lambda$655 seen in second order at these wavelengths, as
well the P I lines appearing in first order at 1305.0, 1305.7 and 1306.0 Å
(Kelly, 1987). In order to minimize effects of the rapidly decaying P IV
features, spectral scans were acquired 4 mm upstream from the monochromator
entrance slit at a beam energy of 170 keV.

As a guide to help determine the amount of contamination from the P IV lines
in P II spectra, a spectrum from 650 to 658 Å with a beam energy of 205 keV
was acquired with the foil position zero mm upstream of the monochromator
entrance slit (see Figure 2). The P IV spectrum was fit by allowing the
dispersion, line intensities, and the multiplet wavelength to vary while
keeping the line separations fixed to the wavelengths in Kelly (1987). The
intensities from this fit were scaled to those seen in the P II spectrum at 4
mm upstream with a dispersion of 0.27 Å, and the scaled P IV lines were
subtracted from the P II spectrum. The impact of the contamination from P I
lines on the P II multiplet under study was deduced from a spectrum obtained
at 120 keV at zero mm offset. The least blended P I line at 1306.0 Å was fit
and the intensities for the other lines scaled to the intensities in Kelly
(1987) before subtracting the P I lines from the P II spectrum at 4 mm. Figure
3a shows the uncorrected P II spectrum with contaminating features and the
corrected P II spectrum is displayed in Figure 3b.

Branching fractions are usually determined by measuring the relative
integrated intensities from lines with a common upper level through Gaussian
fits. The fitting procedure indicated that the line widths were
indistinguishable from one another; we, therefore, relied on the intensities.
The spectral interval is small enough that systematic differences in
instrumental response are not a concern. For determining the uncertainty in
branching fractions, the statistical error estimated from the uncertainty in P
II line intensities was added in quadrature to the uncertainty in lifetime for
the corresponding upper level. The systematic error from the P I and P IV
blends was also considered; it was inferred from relative intensities of the
contaminating lines to the P II lines, which did not exceed 5 to 10%. These
systematic errors were added in quadrature to the statistical errors to
determine the total errors in the branching fractions.

##  3 Results and Discussion

Tables 1 through 3 show the results of our lifetime measurements, branching
fractions, and derived oscillator strengths, respectively. In Table 1, we
present our lifetime measurements as well as comparisons with earlier
experiments and theoretical calculations. The lifetime of the 3$s$3$p^{3}$
${}^{3}P^{\rm o}$ $J_{u}$=2 level was obtained from the weighted average of
measurements of the $\lambda$1310 line at two beam energies, 170 and 220 keV.
Figure 1 shows the decay curve for the line at 1310 Å at a beam energy of 170
keV. Our lifetime measurements are most consistent with the beam-foil results
of Livingston et al. (1975) and with the theoretical calculations of Brage et
al. (1993), Tayal (2003), and Froese Fischer et al. (2006). These theoretical
efforts were based on versions of the multiconfiguration-Hartree-Fock (MCHF)
technique, with corrections for relativistic effects. However, the two
experiments using the phase shift method (Savage & Lawrence, 1966; Smith,
1978) yielded much shorter lifetimes. Our experimental values differ slightly
from the lifetimes given by Hibbert (1988). We note that $\lambda$1304.49
associated with $J_{u}=0$ is too blended with $\lambda$1304.68, making it
difficult to measure the lifetime for this level. Therefore, we do not list a
multiplet value in our results for oscillator strength. However, the
theoretical calculations (Hibbert, 1988; Tayal, 2003) predicted that the
levels with higher angular momenta have longer lifetimes.

Our lifetime for the 3$p$4$s$ ${}^{3}P_{0}^{\rm o}$ level of P IV is $0.27\pm
0.04$ ns, a value that agrees with the theoretical prediction of Gupta &
Msezane (2002) (0.2205 ns), who used the CIV3 code of Hibbert (1975). The
isolated measurement confirms our supposition that the short-lived decay found
for the $J$ $=$ 2 level of P II 3$s$3$p^{3}$ ${}^{3}P^{\rm o}$ (see Figure 1)
arises from P IV. The result for P IV leads us to the following suggestions
regarding earlier experimental determinations. First, the shorter lifetime
obtained by Curtis et al. (1971), also based on beam-foil spectroscopy, could
be the result of contamination from the P IV decay. Second, Livingston et al.
(1975) noted the presence of a second decay besides the one associated with P
II with a lifetime of $1.2\pm 0.4$ ns that they ascribed to a blend with O I.
Within the precision of their measurements, the shorter lifetime could instead
arise from P IV.

Hibbert (1988), Tayal (2003), and Froese Fischer et al. (2006) provided a
complete set of results for the multiplet at 1308 Å, from which branching
fractions can be inferred. The branching fractions from the three calculations
agree well and show slight differences from those obtained from LS coupling
rules (see Table 2). LS coupling gives relative strengths of 0.750 and 0.250
for the transitions to $J_{l}$ $=$ 2 and $J_{l}$ $=$ 1\. Hibbert (1988) found
values of 0.758 and 0.242, while Tayal (2003) gives branching fractions of
0.756 and 0.241 and Froese Fischer et al. (2006) 0.756 and 0.244. Some minor
variation is seen among the theoretical calculations for the $J_{u}$ $=$ 1
branching fractions, where LS coupling gives 0.417, 0.250, and 0.333 for
decays to $J_{l}$ $=$ 2, 1, and 0. Hibbert (1988) obtained respective values
of 0.378, 0.276, and 0.346, while those of Tayal (2003) are 0.391, 0.270, and
0.339 and those of Froese Fischer et al. (2006) are 0.414, 0.254, and 0.332.
Also, shown are the results of a semi-empirical calculation (Bancroft Brown et
al., 2011) in which the mixing between LS-coupled basis states is determined
from the observed energy levels. This simple approach has worked well for
forbidden lines in a number of sequences but clearly fails to account
precisely for the configuration interactions in the present case.

Although the three computations yielded very similar branching fractions, we
obtained experimental values to check for the possibility of unexpected
anomalies and they are also shown in Table 2. The results from fitting our
spectrum of the multiplet are consistent with these predictions, but they do
not favor one over another because the experimental uncertainties are too
large. The experimentally determined branching fractions for the $J_{u}$ $=$ 2
level are $0.22\pm 0.02$ ($J_{l}$ $=$ 1) and $0.78\pm 0.05$ ($J_{l}$ $=$ 2),
while for the $J_{u}$ $=$ 1 level they are $0.36\pm 0.03$ ($J_{l}$ $=$ 0),
$0.27\pm 0.03$ ($J_{l}$ $=$ 1), and $0.37\pm 0.04$ ($J_{l}$ $=$ 2). We did not
apply a correction to the intensities between spectra at 4 mm upstream and at
zero offset relative to the entrance slit because our precision for lifetimes
could not discern the very small variation with $J_{u}$ seen in the
calculations.

Finally, we compare our results on oscillator strength derived from our
lifetimes and branching fractions in Table 3. Our experimental results for
oscillator strengths are in very good agreement with those of Tayal (2003) and
Froese Fischer et al. (2006). The calculations from Hibbert (1988), which
formed the basis of Morton’s (2003) compilation, are lower by about 30%. These
oscillator strengths are smaller because of the cancellations in the dipole
matrix elements due to the strong configuration interactions between
3$s$3$p^{2}$ and 3$s^{2}$3$pnd$. Moreover, the ratio of our experimental
oscillator strengths for P II $\lambda$1153 obtained from Federman et al.
(2007) and P II $\lambda$1302 acquired from this work is $13.9\pm 2.0$, which
is in very good agreement with the ratio given by MCHF calculations (Brage et
al., 1993; Tayal, 2003; Froese Fischer et al., 2006). Thus, our measurements
provide support for the assessment on P II $f$-values given by Cashman et al.
(2017).

##  4 Conclusions

Results of beam-foil measurements yielding lifetimes and branching fractions
for the transitions associated with the P II multiplet at 1308 Å were
presented. These were combined to yield oscillator strengths that are needed
for studies of the phosphorus abundance in the diffuse interstellar medium of
our Galaxy and more distant ones. These experimental results represent the
most precise values for this multiplet to date. Our branching fractions agree
quite well with LS coupling rules but cannot differentiate between them and
theoretical calculations given by Hibbert (1988), Tayal (2003), and Froese
Fischer et al. (2006). Our results for lifetimes and oscillator strengths
agree with the MCHF results of Tayal (2003) and Froese Fischer et al. (2006).
Furthermore, the $f$-value ratio from our beam foil techniques for lines at
1153 and 1302 Å is $13.9\pm 2.0$, very similar to the ratio obtained from MCHF
calculations. The consistency now revealed between experiment and theory for
these transitions in P II will provide more secure phosphorus abundances for
interstellar matter.

This work was supported by NASA grant NNG06GC70G. J. Bancroft Brown and R.E.
Irving acknowledge support by the National Science Foundation under Grant No.
PHY- 0648963. We thank Dave Ellis for useful conversations.

## References

  * Bancroft Brown et al. (2011) Bancroft Brown, J., Brown, M. S., Cheng, S., et al. 2011, CaJPh, 89, 413
  * Beck & Sinanoğlu (1972) Beck, D., & Sinanoğlu, O. 1972, PhRvL, 28, 945
  * Brage et al. (1993) Brage, T., Merkelis, G., & Froese Fischer, C. 1993, PhLA, 174, 111
  * Cartledge et al. (2006) Cartledge, S. I. B., Lauroesch, J. T., Meyer, D. M., & Sofia, U. J. 2006, ApJ, 641, 327
  * Cashman et al. (2017) Cashman, F. H., Kulkarni V. P., Kisielius R., Ferland G. J., & Bogdanovich P. 2017, ApJS, 230,8
  * Curtis et al. (1971) Curtis, L. J., Berry, H. G., & Bromander, J. 1971, PhLA, 34, 169
  * De Cia et al. (2016) De Cia, A., Ledoux, C., Mattsson, L., et al. 2016, A&A, 596, A97
  * Dessauges-Zavadsky et al. (2007) Dessauges-Zavadsky, M., Calura, F., Prochaska, J. X., et al. 2007, A&A, 470, 431
  * Dufton et al. (1986) Dufton, P. L., Keenan, F. P., & Hibbert, A. 1986, A&A, 164, 179
  * Federman et al. (1992) Federman, S. R., Beideck, D. J., Schectman, R. M., & York, D. G. 1992, ApJ, 401, 367
  * Federman et al. (2007) Federman, S. R., Brown, M., Torok, S., et al. 2007, ApJ, 660, 919
  * Froese Fischer et al. (2006) Froese Fischer, C., Tachiev, G., & Irimia, A. 2006, ADNDT, 92, 607
  * Gupta & Msezane (2002) Gupta, G. P., & Msezane, A. Z. 2002, PhyS, 66, 43
  * Haar et al. (1993) Haar, R. R., Beideck, D. J., Curtis, L. J., et al. 1993, NIMPB, 79, 746
  * Harris & Mas Hesse (1986) Harris, A. W., & Mas Hesse, J. M. 1986, ApJ, 308, 240
  * Hibbert (1975) Hibbert, A. 1975, CoPhC, 9, 141
  * Hibbert (1986) 1986, JPhB, 19, L455
  * Hibbert (1988) 1988, PhyS, 38, 37
  * Jenkins (2009) Jenkins, E. B. 2009, ApJ, 700, 1299
  * Jenkins et al. (1986) Jenkins, E. B., Savage, B. D., & Spitzer, L., Jr. 1986, ApJ, 301, 355
  * Jura & York (1978) Jura, M., & York, D. G. 1978, ApJ, 219, 861
  * Kelly (1987) Kelly, R. L. 1987, JPCRD, 16, Suppl. 1
  * Lebouteiller et al. (2005) Lebouteiller, V., Kuassivi, & Ferlet, R. 2005, A&A, 443, 509
  * Levshakov et al. (2002) Levshakov, S. A., Dessauges-Zavadsky, M., D’Odorico, S., & Molaro, P. 2002, ApJ, 565, 696
  * Livingston et al. (1975) Livingston, A. E., Kernahan, J. A., Irwin, D. J. G., & Pinnington, E. H. 1975, PhyS, 12, 223
  * Mallouris et al. (2001) Mallouris, C., Welty, D. E., York, D. G., et al. 2001, ApJ, 558, 133
  * Molaro et al. (2001) Molaro, P., Levshakov, S. A., D’Odorico, S., et al. 2001, ApJ, 549, 90
  * Morton (2003) Morton, D. C. 2003, ApJS, 149, 205
  * Pettini et al. (2002) Pettini, M., Rix, S. A., Steidel, C. C., et al. 2002, ApJ, 569, 742
  * Savage & Lawrence (1966) Savage, B. D., & Lawrence, G. M. 1966, ApJ, 146, 940
  * Schectman et al. (2000) Schectman, R. M., Cheng, S., Curtis, L. J., et al. 2000, ApJ, 542, 400
  * Smith (1978) Smith, W. H. 1978, PhyS, 17, 513
  * Som et al. (2015) Som, D., Kulkarni, V. P., Meiring, J., et al. 2015, ApJ, 806, 25
  * Tayal (2003) Tayal, S. S. 2003, ApJS, 146, 459
  * Tchernyshyov et al. (2015) Tchernyshyov, K., Meixner, M., Seale, J., et al. 2015, ApJ, 811, 78

Table 1: P II Lifetimes for 3$s$3$p^{3}$ ${}^{3}P^{\rm o}$ Levels $J_{u}$ |  $\tau$ (ns)
---|---
| Present |  SLaaSavage & Lawrence (1966) $-$ phase shift experiment. |  CMBbbCurtis et al. (1971) $-$ beam-foil experiment. |  LKIPccLivingston et al. (1975) $-$ beam-foil experiment. |  SddSmith (1978) $-$ phase shift experiment. |  HeeHibbert (1988) $-$ configuration interaction calculation. |  BMFffBrage et al. (1993) $-$ multi-configuration Hartree-Fock calculation. |  TggTayal (2003) $-$ multi-configuration Hartree-Fock calculation. |  FTIhhFroese Fischer et al. (2006) $-$ multi-configuration Hartree-Fock calculation.
1 | $14.0\pm 0.8$ | $6.4\pm 0.8$ | $9.0\pm 0.5$ | $15\pm 4$ | $5.4\pm 0.6$ | 20.7 | $\ldots$ | 12.5 | 12.2
2 | $14.6\pm 0.5$ | $6.4\pm 0.8$ | $9.0\pm 0.5$ | $15\pm 4$ | $5.4\pm 0.6$ | 22.6 | 12.3 | 13.4 | 13.0
Table 2: P II Branching Fractions for 3$s$3$p^{3}$ ${}^{3}P^{\rm o}$ Levels Transition |  $J_{l}$ |  PresentaaThis work – beam-foil experiment. |  HbbHibbert (1988). |  TccTayal (2003). |  FTIddFroese Fischer et al. (2006). |  BBeeBancroft Brown et al. (2011) $-$ semi-empirical method. | LS coupling
---|---|---|---|---|---|---|---
${}^{3}P_{J}$ $-$ ${}^{3}P_{1}^{\rm o}$ | 0 | $0.36\pm 0.03$ | 0.346 | 0.339 | 0.332 | $\ldots$ | 0.333
| 1 | $0.27\pm 0.03$ | 0.276 | 0.270 | 0.254 | $\ldots$ | 0.250
| 2 | $0.37\pm 0.04$ | 0.378 | 0.391 | 0.414 | $\ldots$ | $0.417$
${}^{3}P_{J}$ $-$ ${}^{3}P_{2}^{\rm o}$ | 1 | $0.22\pm 0.02$ | 0.242 | 0.241 | 0.244 | 0.318 | 0.250
| 2 | $0.78\pm 0.05$ | 0.758 | 0.759 | 0.756 | 0.682 | 0.750
Table 3: P II Oscillator Strengths for the Multiplet 3$s^{2}$3$p^{2}$ ${}^{3}P$ $-$ 3$s$3$p^{3}$ ${}^{3}P^{\rm o}$ $\lambda_{ul}$ (Å) | $J_{l}$ | $J_{u}$ |  $f$-value ($\times 10^{-3}$) |
---|---|---|---|---
|  |  | Present |  SLaaSavage & Lawrence (1966). |  CMBbbCurtis et al. (1971). |  BSccBeck & Sinanoğlu (1972) $-$ nonclosed many-electron theory. |  LKIddLivingston et al. (1975). |  SeeSmith (1978). |  HffHibbert (1988). |  BMFggBrage et al. (1993). |  ThhTayal (2003). |  FTIiiFroese Fischer et al. (2006). |  MjjMorton (2003) $-$ compilation.
1310.70 | 2 | 2 | $13.8\pm 1.0$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  8.4kkBased on length formalism. | $\ldots$ |  14.6kkBased on length formalism. | 15.1 | 8.4
| $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  8.0llBased on velocity formalism. | $\ldots$ |  14.6llBased on velocity formalism. | $\ldots$ | $\ldots$
1309.87 | 2 | 1 | $4.1\pm 0.5$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  2.8kkBased on length formalism. | $\ldots$ |  4.8kkBased on length formalism. | 5.3 | 2.8
| $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  2.6llBased on velocity formalism. | $\ldots$ |  4.8llBased on velocity formalism. | $\ldots$ | $\ldots$
1305.50 | 1 | 2 | $6.4\pm 0.6$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  4.5kkBased on length formalism. | $\ldots$ |  7.6kkBased on length formalism. | 8.1 | 4.5
| $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  4.5llBased on velocity formalism. | $\ldots$ |  7.7llBased on velocity formalism. | $\ldots$ | $\ldots$
1304.68 | 1 | 1 | $4.9\pm 0.6$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  3.4kkBased on length formalism. | $\ldots$ |  5.5kkBased on length formalism. | 5.4 | 3.4
| $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  3.3llBased on velocity formalism. | $\ldots$ |  5.6llBased on velocity formalism. | $\ldots$ | $\ldots$
1304.49 | 1 | 0 | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  4.5kkBased on length formalism. | $\ldots$ |  7.0kkBased on length formalism. | 7.2 | 4.5
| $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  4.4llBased on velocity formalism. | $\ldots$ |  7.0llBased on velocity formalism. | $\ldots$ | $\ldots$
1301.87 | 0 | 1 | $19.6\pm 2.0$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  12.7kkBased on length formalism. | $\ldots$ |  20.7kkBased on length formalism. | 21.0 | 12.7
| $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  12.6llBased on velocity formalism. | $\ldots$ |  21.0llBased on velocity formalism. | $\ldots$ | $\ldots$
Multiplet | $\ldots$ | $\ldots$ | $\ldots$ | $40\pm 5$ | $29\pm 2$ |  38.2kkBased on length formalism. | $17\pm 5$ | 48 |  11.8kkBased on length formalism. |  22.6kkBased on length formalism. | $\ldots$ | $\ldots$ | 11.8
| $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ | $\ldots$ |  43.6llBased on velocity formalism. | $\ldots$ | $\ldots$ |  11.4llBased on velocity formalism. |  23.0llBased on velocity formalism. | $\ldots$ | $\ldots$ | $\ldots$

Figure 1: The measured P II decay curve for the line at 1310 Å for a beam
energy of 170 keV. The post-foil beam velocity at this energy was 1.0068 mm
ns-1, thus establishing the time since excitation for a given foil position.
The foil was moved relative to the monochromator entrance slit in increments
of 0.1 mm until it was displaced 1 mm; then the increments were increased to
0.5 mm. A two-exponential fit to the data is shown by the solid curve. Figure
2: A spectrum of the P IV multiplet at 655 Å for a beam energy of 205 keV. The
total angular momentum quantum number for each upper fine structure level is
indicated. Numbers above the spectrum refer to $J_{u}$ for different
transitions. The best fit to the lines in the multiplet is shown by the solid
curve.

Figure 3: Panel (a) shows the uncorrected spectrum of the P II multiplet at
1308 Å with contaminating features. Dash-dotted tick marks indicate the
position of P I lines. Solid tick marks show the position of P IV lines.
Gaussian profiles reveal the strength of the contaminating features. Panel (b)
represents to the corrected spectrum of the P II multiplet at 1308 Å. See Fig.
2 for further details.