# Sparse Attention Guided Dynamic Value Estimation for Single-Task Multi-Scene
Reinforcement Learning

Jaskirat Singh  Liang Zheng

###### Abstract

Training deep reinforcement learning agents on environments with multiple
levels / scenes from the _same task_ , has become essential for many
applications aiming to achieve generalization and domain transfer from
simulation to the real world (Wortsman et al., 2019; Cobbe et al., 2019b).
While such a strategy is helpful with generalization, the use of multiple
scenes significantly increases the variance of samples collected for policy
gradient computations. Current methods, effectively continue to view this
collection of scenes as a single Markov decision process (MDP) and thus, learn
a scene-generic value function $V(s)$. However, we show that the sample
variance for a multi-scene environment is best minimized by treating each
scene as a distinct MDP, and then learning a joint value function
$V(s,\mathcal{M})$ dependent on both state $s$ and MDP $\mathcal{M}$. We
further demonstrate that the true joint value function for a multi-scene
environment, follows a multi-modal distribution which is not captured by
traditional CNN / LSTM based critic networks. To this end, we propose a
dynamic value estimation (DVE) technique, which approximates the true joint
value function through a _sparse_ attention mechanism over multiple value
function hypothesis / modes. The resulting agent not only shows significant
improvements in the final reward score across a range of OpenAI ProcGen
environments, but also exhibits enhanced navigation efficiency and provides an
implicit mechanism for unsupervised state-space skill decomposition.

Multi-Scene Reinforcement Learning, Variance Reduction, Dynamic Value
Estimation

##  1 Introduction

While the field of reinforcement learning has shown tremendous progress in the
recent years, generalization across variations in the environment dynamics
remains out of reach for most state-of-the-art deep RL algorithms (Rajeswaran
et al., 2017; Zhang et al., 2018a; Whiteson et al., 2011). In order to achieve
the generalization objective, many deep RL approaches attempt to train agents
on environments comprising of multiple levels or scenes from the same task
(Wortsman et al., 2019; Cobbe et al., 2019b; Zhu et al., 2017; Justesen et
al., 2018; Cobbe et al., 2019a; Kanagawa & Kaneko, 2019; Juliani et al.,
2019). Although incorporating a wider source of data distribution in the
training itself has shown promise in bridging the train and test performance,
the inclusion of multiple scenes, each defined by a distinct underlying MDP,
significantly increases the variance of samples collected for policy gradient
computations (Cobbe et al., 2019a; Song et al., 2019).

The current approaches using multi-scene environments for training usually
deal with the high variance problem by deploying multiple actors for
collecting a larger and varied range of samples. For instance, (Zhu et al.,
2017; Wortsman et al., 2019) use multiple asynchronous actor critic (A3C)
models when training on the AI2-THOR framework based visual navigation task.
Similarly, (Cobbe et al., 2019a, b) deploy parallel workers to stabilize
policy gradients for multi-level training on procedurally-generated game
environments (Justesen et al., 2018). While parallel sample collection helps
in stabilizing the learning process, the obvious disadvantages of lower sample
efficiency and higher hardware constraints, suggest the need for specialized
variance reduction techniques in multi-scene RL.

Most RL generalization benchmarks (Nichol et al., 2018; Zhang et al., 2018b;
Cobbe et al., 2019b; Igl et al., 2019) _effectively_ treat the collection of
scenes as a _single-MDP environment_. That is, a common and scene generic
value function $V(s)$ is learned across all levels. By comparison, we propose
an improved variance reduction formulation, which instead shows that the
sample variance for a multi-scene environment is best minimized by
acknowledging each scene as a separate MDP and then learning a joint value
function $V(s,\mathcal{M})$ dependent on both state $s$ and MDP $\mathcal{M}$.

However, given the lack of information about the operational level at train /
test times, estimating the joint value function $V(s,\mathcal{M})$ presents a
challenging problem. To address this, we first show that the underlying true
joint value function samples follow a multi-modal distribution. We then use
this insight to propose a dynamic value estimation strategy, which
approximates the overall value distribution through a progressively learned
_sparse_ attention mechanism over the corresponding distribution modes. The
sparse attention guided dynamic networks not only result in huge improvements
in total reward on the OpenAI ProcGen benchmark, but also exhibit semantically
desirable properties like enhanced navigation efficiency and provide an
implicit framework for unsupervised state space decomposition.

The main contributions of this paper are summarized below.

  * •

Enhanced Variance Reduction: We propose an improved variance reduction
formulation which shows that the sample variance for a multi-scene environment
is best minimized by treating the each scene as a distinct MDP, and then
learning a joint value function $V(s,\mathcal{M})$ dependent on both state $s$
and MDP $\mathcal{M}$.

  * •

Clustering Hypothesis: We show that the true scene-specific value function
distribution is best described using a mixture model with multiple dominant
modes, which are not fully captured by the current CNN or LSTM based critic
networks.

  * •

Novel Critic Module: We propose a novel critic model which approximates the
true multi-modal value function distribution through a _sparse_ attention
mechanism over multiple value function hypothesis / modes.

  * •

Implicit State-Space Decomposition: We demonstrate that the learned sparse
attention divides the overall state space into distinct sets of game skills.
The collection of these skills represents a curriculum that the agent must
master for effective game play.

  * •

Navigation Efficiency: Through both quantitative and qualitative results, we
show that the sparse dynamic model leads to huge improvements in the
navigation efficiency of the resulting agent. Furthermore, the high navigation
efficiency of our method and its tendency to limit unnecessary exploration,
presents an effective alternative to explicit reward-shaping (Zhu et al.,
2017; Wortsman et al., 2019; Laud, 2004; Laud & DeJong, 2003), for penalizing
longer episode-lengths / reward-horizons in multi-scene reinforcement
learning.

##  2 Problem Setup

A typical multi-scene environment is characterized by a set of possible MDPs
${\mathcalbf{M}}:\\{\mathcal{M}_{1},\mathcal{M}_{2},...\mathcal{M}_{N}\\}$,
each defined by its own state space $\mathcal{S}_{\mathcal{M}}$, transition
probabilities $P_{\mathcal{M}}(s_{t+1}|s_{t},a_{t})$, reward function
$r_{\mathcal{M}}(s_{t},a_{t},s_{t+1})$ and discount factor $\gamma$. An agent
with action space $\mathcal{A}$ interacts with a randomly chosen and unknown
MDP $\mathcal{M}\in\mathcalbf{M}$, to generate a trajectory
$\tau:(s_{0},a_{0},s_{1}...s_{T})$ with total reward
$\mathcal{R}_{\tau}=\sum_{t=0}^{T-1}\gamma^{t}r_{\mathcal{M}}(s_{t},a_{t},s_{t+1})$.
The goal of the agent is to maximize the expected trajectory rewards over the
entire MDP set $\mathcalbf{M}$, _i.e._
$\mathbf{E}_{\tau,\mathcal{M}}\left[\mathcal{R}_{\tau,\mathcal{M}}\right]$.

##  3 Motivation

In this section, we present a step-by-step analysis which motivates the final
method presented in Sec. 4. We first begin by proposing an improved variance
reduction formulation for multi-scene environments in Sec. 3.1. We then
demonstrate the multi-modal nature of the joint value distribution in Sec.
3.2. Finally we outline the main idea behind dynamic value estimation in Sec.
3.3.

###  3.1 Variance Reduction for Multi-Scene Environments

For a single-MDP environment, with policy network $\pi$ (parameterized by
$\theta$) and an action-value function $Q(s,a)$, the general expression for
computing policy gradients with minimal possible sample variance can be
written as, (Greensmith et al., 2004; Schulman et al., 2015),

$\nabla_{\theta}J=\mathbf{E}_{s,a}\left[(\nabla_{\theta}\log\pi(a|s))\
\psi(s,a)\right],$ (1)

where $\psi(s,a)=Q(s,a)-V(s)$ is the advantage function. Similarly for a
multiple-scene environment, it can be shown that the optimal formulation for
minimizing total sample variance is given by (please refer supp. material for
proof),

$\nabla_{\theta}J=\mathbf{E}_{s,a,\mathcal{M}}\left[(\nabla_{\theta}\log\pi(a|s))\
\psi(s,a,\mathcal{M})\right],$ (2)

where $\psi(s,a,\mathcal{M})=Q(s,a,\mathcal{M})-V(s,\mathcal{M})$. Here
$Q(s,a,\mathcal{M})$ and $V(s,\mathcal{M})$ represent the action-value and
value function respectively for the particular MDP $\mathcal{M}$. However,
since most of the times knowledge about the operational MDP $\mathcal{M}$ is
unknown to the agent, the current policy gradient methods continue to use a
single scene-generic value function estimate $\hat{V}(s)$ for variance
reduction. However, $\hat{V}(s)$ then is essentially an estimate of the global
average over the underlying scene-specific value functions
$\\{V_{\mathcal{M}_{1}}(s),V_{\mathcal{M}_{2}}(s),...,V_{\mathcal{M}_{N}}(s)\\}$,
and thus gives a poor approximation of the joint value function for a given
MDP.

We next show that such a simplification is not necessary and present an
approach for obtaining a better approximation for the joint value function
$V(s,\mathcal{M})$.

[capbesideposition=right,center,capbesidewidth=5cm]figure[]

Figure 1: Clustering Hypothesis. Column 1-2 demonstrate multi-modal nature of
the true value function distribution, for an intermediate policy $\pi$, on a
set of randomly selected 50 levels from the CoinRun environment. The true
value estimate for a state image shown in Column 1 can be characterized by one
of the many clusters. Column 3: In contrast, the LSTM based value predictions,
though showing some variance with MDP $\mathcal{M}$, fail to capture the
multiple dominant modes exhibited by the true value function distribution.

###  3.2 Clustering Hypothesis

Training on multi-scene environments over the same domain task can lead to
ambiguity in value function estimation. That is, two visually similar states
could have very different value function estimates corresponding to distinct
scenes / levels. In this section, we empirically demonstrate that unlike a
single-scene environment, the true value function for a multi-scene
environment (having scenes with similar state spaces), is better described by
a multi-modal distribution.

Empirical Demonstration on OpenAI CoinRun111Please refer supplementary
materials for further details.. To test the above hypothesis, we finetune
separate critic networks over a fixed policy $\pi$, to obtain the true MDP-
specific value function estimates
$\\{V(s,\mathcal{M}_{1}),V(s,\mathcal{M}_{2}),...,V(s,\mathcal{M}_{50})\\}$
corresponding to a random selection of 50 levels from the CoinRun ProcGen
environment (Cobbe et al., 2019b). We then use a Gaussian Mixture Model (GMM)
for fitting these $V(s,\mathcal{M}_{i})\\{i\in[1,50]\\}$ samples. Results are
shown in Fig. 1. We clearly observe that the true value function estimates
form a multi-modal distribution that is not captured by traditional CNN or
LSTM based critic networks.

###  3.3 Minimizing Value Prediction Error

Theorem 1. The sample variance ($\nu$) for policy gradients defined by Eq. 2,
can be minimized by reducing the prediction error $\epsilon$ between the true
joint value function $V(s,\mathcal{M})$ and the predicted estimate
$\hat{V}(s,\mathcal{M})$, where

$\epsilon=\mathbf{E}_{s,\mathcal{M}}\left[\left(V(s,\mathcal{M})-\hat{V}(s,\mathcal{M})\right)^{2}\right].$
(3)

The proof is provided in supp. materials. We now use the insight provided by
Theorem 1 to propose the following solution for reducing the policy gradient
sample variance.

Proposed Solution: While the exact estimation of the true value function is
infeasible without knowledge of MDP $\mathcal{M}$, we use the results of
Section 3.2, to assert that the prediction error can be reduced by
approximating the value function as the mean value of the cluster to which the
current MDP belongs. Fig. 2 provides an overview of this idea.

Figure 2: Proposed Approach. Traditional methods learn a single scene-generic
value function estimate $V(s)$ which leads to high prediction error
($\epsilon_{1}>\epsilon_{2}$). In contrast we propose that the overall
prediction error can be significantly reduced by approximating the joint value
function $V(s,\mathcal{M})$ as the mean of the nearest cluster.

##  4 Our method

The solution proposed in Sec. 3.3 can be implemented through a sparse
attention mechanism over the value function modes $\mu_{i}$, wherein the
attention parameters are $1$ for the closest cluster and $0$ otherwise.
However the sparse attention parameters are not fixed, as the cluster to which
an MDP belongs is expected to change while training the RL agent. To address
this, we first model the predicted value function through a generalized
attention mechanism over the value function modes (Sec. 4.1) and then propose
a novel loss function which progressively enforces sparse attention parameters
based on the training dynamics (Sec. 4.2).

###  4.1 Continuous Dynamic Estimation Formalism

Mathematically, given that the _true_ value function follows a Gaussian
Mixture Model (GMM) like,

$P\left(V(s_{t},\mathcal{M})|s_{t}\right)=\sum_{i=1}^{N_{b}}p_{i}\
\mathcal{N}(V(s_{t},\mathcal{M})|\mu_{i},\sigma_{i}^{2}),$ (4)

we propose to model the _predicted_ value function as,

$\hat{V}(s_{t},\mathcal{M})=\sum_{i=1}^{N_{b}}\alpha_{i}(s_{t},\mathcal{M})\
\mu_{i}(s_{t}),\\\ s.t.\quad\alpha_{i}>0,\quad\sum_{i}^{N_{b}}\alpha_{i}=1.$
(5)

That is, given a state $s_{t}$, we predict $N_{b}$ distinct value function
hypotheses $\\{\mu_{1}(s),\mu_{2}(s),...,\mu_{N_{b}}(s)\\}$ (one corresponding
to each cluster). The final value prediction is then modelled as the weighted
combination of these value hypotheses using attention parameters $\alpha_{i}$.
Wherein, the attention parameters $\alpha_{i}(s_{t},\mathcal{M})$ are used to
capture the similarity between the $i^{th}$ value hypothesis and the true
value for MDP $\mathcal{M}$. In practice, since the current MDP $\mathcal{M}$
is not known, the parameters $\alpha_{i}$ are learned from (state, episode
trajectory) pairs $\\{s_{t},\tau^{t-}\\}$
($\tau^{t-}:\\{s_{0},a_{0},....s_{t-1}\\}$ is the trajectory till time $t-1$),
using a Long Short Term Memory (LSTM) network.

###  4.2 Enforcing Sparse Attention

We first define two metrics to describe the attention parameter distribution,
_confusion_ and _contribution_. Confusion ($\delta$) is a measure of
uncertainty as to which cluster, the current state-trajectory pair
$\\{s_{t},\tau^{t-}\\}$ belongs to. On the other hand, contribution ($\rho$),
as the name suggests, determines the ‘contribution’ of a cluster in the
overall value function estimation across a general trajectory sequence
$\tau:\\{s_{0},a_{0},s_{1},a_{1},...s_{T}\\}$. Formally, confusion and
contribution are defined as,

$\displaystyle\delta(s_{t},\tau^{t-})=\frac{1}{N_{b}.\sum_{i}\alpha^{2}_{i}(s_{t},\tau^{t-})},$
(6)
$\displaystyle\rho_{i}(\tau)=\frac{1}{T}\sum_{t=1}^{T}\delta(s_{t},\tau^{t-})\
\alpha_{i}(s_{t},\tau^{t-}).$ (7)

We now note that an increase in sparsity of cluster assignments
$\\{\alpha_{1},\alpha_{2},\dots\alpha_{N_{b}}\\}$ is equivalent to
maximization of their $l_{2}$ norm. Thus using Eq. 6, it corresponds to
minimization of the confusion ($\delta$) metric. However, a mere enforcement
of sparsity may encourage convergence to solutions where only one of the
clusters is active. We also want to ensure that each cluster contributes
equally in the $(s,\mathcal{M})$ space. To achieve this, we propose the
following _confusion-contribution loss_ ,

$L^{CC}=k_{1}\
\mathbf{E}_{s_{t},\tau^{t-}}\left[\log\delta(s_{t},\tau^{t-})\right]+\\\
k_{2}\
\mathbf{E}_{\tau}\left[\log\left(\sum_{i}^{N_{b}}\rho^{2}_{i}(\tau)\right)\right].$
(8)

We emphasize that the state space must have already been well explored222For
OpenAI ProcGen, we consider state space to be sufficiently explored when the
avg. episode length plateaus / starts decreasing. by the agent, prior to the
application of confusion-contribution loss. If applied prematurely, due to the
continuous nature of neural networks, the sparse cluster assignment is
incorrectly generalized across the entire state space. This would lead to a
detrimental impact on value function estimation for the currently unexplored
states. Also, such a mistake is hard to recover from, because for any state
$s\in\mathcal{S}$, the sparse assignment ensures that the gradients for all
but one cluster are approximately zero.

##  5 Related Work

Meta Reinforcement Learning. (Duan et al., 2016; Wang et al., 2016) previously
proposed the use of recurrent neural networks and episode trajectories as a
meta-RL approach for adapting to environment dynamics. While in theory, an
LSTM is capable of learning multi-modal distributions, we find that in
practice the vanilla-LSTM based conditional value function distribution (for a
given state) is usually characterized by a single dominant mode (refer Fig.
1), and thus fails to capture the multi-modal nature of true value function
distribution. In contrast, our method explicitly forces multiple dominant
modes while estimating the cluster means $\mu_{i}$ and uses episode
trajectories to compute the assignment $(\alpha_{i})$ of the current state
sample to each cluster333In this paper, we occasionally use the term cluster
to refer to the value function hypothesis, since each hypothesis represents
the mean of a distinct cluster in the true value distribution..

Recurrent Independent Mechanisms. (Goyal et al., 2019) propose modular
structures called Recurrent Independent Mechanisms (RIMs) which specialize to
different dynamic processes within an environment, and communicate sparingly
through a sparse attention mechanism. While RIMs and our method share the idea
of having sparse attention, our work differs significantly as RIMs use
separate recurrent models with independent dynamics, whereas we deploy a
single actor-critic network with shared dynamics. Another important
distinction is that (Goyal et al., 2019) assume a sparse structure from the
beginning and hence require environments with clearly independent dynamic
processes. In contrast, the sparse assignment in our method is learned
progressively and only after sufficient exploration, which allows for a more
informed division of the state space (Sec. 8.1).

Generalization in Reinforcement Learning. Recently, multi-scene environments
have been extensively used to study and address the problem of overfitting in
RL. (Cobbe et al., 2019b) deploy standard regularization techniques from
supervised learning like dropout, batch-normalization, L2-regularization to
counter overfitting when training on the multi-scene CoinRun environment.
(Rajeswaran et al., 2016) learn robust policies over an ensemble of scenes by
formulating the overall objective as the expected reward over scenes with the
worst performance. Noise injection techniques like (Igl et al., 2019; Tobin et
al., 2017) add noise to the model parameters in order to improve the
generalization capability. While effective, these works are seen to reduce
overfitting at some expense to the training performance. Our work is different
as it doesn’t focus on generalization specifically, but improves both training
and test time performance by learning a better value function estimate.
Furthermore, we note that our approach is orthogonal to these works and would
likely also benefit from use of regularization methods.

Distributional RL. Recent works like (Bellemare et al., 2017; Dabney et al.,
2018; Choi et al., 2019) aim to directly learn the value function distribution
instead of modelling the expected return. Our work differs in the following
aspects. First, distributional RL methods need to discretize the return space
using a high number of support locations/nodes (_e.g._ $N=200$ for (Bellemare
et al., 2017)) to approximate the overall value distribution. In contrast, we
approximate the joint value function through only the modes of the underlying
distribution and thus require very few output nodes ($N\in[2,5]$). Second, the
current work on distributional RL is limited to off-policy methods with a
shared replay buffer. The use of a large replay buffer implies that the
overall sample distribution changes minimally from one update to another. This
is in sharp contrast to the high variance seen in on-policy training which is
the focus of current work. To the best of our knowledge, our work is the first
stable method for approximating value distribution in on-policy RL.

Bootstrapped DQN. (Osband et al., 2016) propose the use of multi-head
Q-networks for bootstrapped DQNs. However, they aim to facilitate deep
exploration and hence select the Q-network head for a given episode randomly.
In contrast, we aim to reduce the prediction error with the true value
function distribution and also present a novel approach which progressively
learns the most representative cluster for each state $s\in\mathcal{S}$
through the confusion-contribution loss.

In addition to the above, we note that (Liang et al., 2018) propose an
attention-based value function network for model-based reinforcement learning.
However, they consider an attention over past trajectory states, whereas we
consider an attention over multiple value function hypotheses.

##  6 Evaluation on OpenAI Procgen

[capbesideposition=right,center,capbesidewidth=5cm]figure[]

Figure 3: Learning curves for LSTM-PPO, sparse and non-sparse dynamic models,
illustrating differences in sample complexity, total reward and episode
lengths. We clearly observe that our method (with sparse attention) achieves
higher total rewards (top-row) while needing much shorter episode lengths
(bottom-row) per reward unit.

| Total Reward |  Navigation Efficiency [$\times 10^{-2}$]
---|---|---
|  |  |  |  | (Total reward / Episode length) |  |
Class | Environment | PPO | DVE (Ours) | Sparse DVE (Ours) | PPO | DVE (Ours) | Sparse DVE (Ours)
Class 1 | CoinRun | 7.75 | 9.16 | 9.62 | 6.15 | 11.59 | 15.48
CaveFlyer | 6.82 | 9.02 | 11.57 | 3.02 | 3.27 | 15.39
Climber | 7.50 | 8.14 | 10.17 | 4.21 | 3.59 | 5.92
Jumper | 6.61 | 6.52 | 6.65 | 2.80 | 3.61 | 8.43
Class 2 | Plunder | 7.13 | 17.16 | 18.42 | 1.44 | 2.19 | 2.48
DodgeBall | 10.98 | 11.25 | 12.76 | 2.73 | 2.44 | 4.45
FruitBot | 7.33 | 18.32 | 23.08 | 4.25 | 5.02 | 6.17
StarPilot | 17.94 | 18.08 | 19.81 | 5.48 | 5.28 | 6.19

Table 1: Final performance comparison. We clearly see that the sparse dynamic
model results in huge gains in both the total reward as well as the overall
navigation efficiency _i.e._ it achieves higher rewards while needing much
shorter episode lengths (per reward unit).

###  6.1 Training Configurations

CNN-LSTM Baseline. The baseline model closely follows4 the one described in
(Cobbe et al., 2019a). Both actor and critic share an IMPALA-CNN network
(Espeholt et al., 2018) modeled in the form of an LSTM (Hochreiter &
Schmidhuber, 1997). The output from the LSTM is then fed to both actor and
critic separately, and is followed by a single fully-connected (FC) layer to
compute the action scores and value function, respectively. The choice of
hyper parameters is kept similar to that described in (Cobbe et al., 2019a) in
order to allow for comparison on the baseline values 444 (Cobbe et al., 2019a)
use an IMPALA CNN network as the baseline, while we use an IMPALA CNN-LSTM
network., though, local hyper-parameter search was performed for each game to
achieve the best average reward score.

Dynamic. The dynamic model is quite similar to the baseline and only requires
few changes in the critic network to model the dynamic estimate described by
Eq. 5. The attention parameters $\alpha_{i}(s_{t},\tau^{t-})$ are modeled
using a fully connected layer followed by a softmax function, while the mean
estimates $\mu_{i}(s_{t})$ are learned using a single FC layer. The number of
dynamic clusters/nodes $N_{b}$ is treated as a hyperparameter555The Bayesian
optimal choice for the number of clusters $N_{b}$ can be determined by
minimizing the Akaike Information Criterion (AIC) (Akaike, 1987; Forster &
Sober, 2011) for the learned value function distribution. We demonstrate this
in supp. material.. For most ProcGen environments, the optimal choice for
$N_{b}$ is found to lie within the range $[2,5]$.

Sparse Dynamic. The only difference with the original dynamic model is the
additional application of the confusion-contribution loss $(L_{CC})$ during
the training process. A hyperparameter selection of $\\{k_{1}=0.1,k_{2}=1\\}$
was seen to give competitive results for most ProcGen environments.

All the three configurations are trained using the Proximal policy
optimization (PPO) (Schulman et al., 2017) algorithm, which is ran with 4
parallel workers for gradient computations as this is seen to enhance
performance. Each worker is trained for 50M steps, thus equating to a total of
200M steps across all the 4 workers. All results are reported as the average
across 4 runs using 500 levels for training.

###  6.2 Test Environments

We test our method on 8 OpenAI ProcGen (Cobbe et al., 2019a) environments:
Dodgeball, FruitBot, StarPilot, CoinRun, CaveFlyer, Climber, Jumper and
Plunder. Note that each game is characterized by a different rate of state
exploration. Thus, depending upon the type of environment, we adopt the
following strategies for applying the sparse loss.

Class-1. For games allowing rapid state space exploration at the beginning,
the confusion-contribution loss can be applied quite early to promote
sparsity. In fact, because the policy gradient and value function loss
dominate the initial training updates, we apply the confusion-contribution
loss from the start. CoinRun, CaveFlyer, Climber and Jumper belong to this set
and are labelled as _class-1_ environments.

Class-2. In contrast, other games display a much more gradual expansion of
explored state space, exhibiting a positive correlation between episode
lengths and the total reward. The sparse loss for such environments can only
be applied after the rate of increase of average episode length has declined.
Thus, the application of confusion-contribution loss is usually preceded by
pre-training666For maintaining consistency, a pre-training procedure same as
the one described above, is followed for all training configurations on
class-2 environments. with the original dynamic model for 50M timesteps (per
worker). Games like Plunder, Dodgeball, FruitBot and StarPilot are part of
this set and are labelled as _class-2_ environments.

Figure 4: Visualizing Cluster Features. Examples of key obstacles types
learned by each cluster in the CoinRun Environment. We note that the sparse
training divides the overall state space into a distinct sets of game skills.

##  7 Results

Total Reward and Sample Complexity. Table 6 reports our results on various
OpenAI-ProcGen environments. We clearly see that the use of dynamic value
estimation (with sparse attention) leads to huge improvements in both the
final reward and sample complexity (refer Fig. 3) over the baseline model. For
instance, our method (with sparse attention) results in an increase of 69.7%,
158.3% & 215.1% in the average episode reward, for the CaveFlyer, Plunder and
FruitBot environments respectively.

Navigation Efficiency. In addition to reporting results for average episode
reward, we also compare the model performance based on the agent’s efficiency
in completing a game level. The navigation efficiency is thus measured by the
ratio of the final reward and average episode length. Results are reported in
Table 6. We clearly see that the sparse model leads to better reward scores
while on average, using much fewer timesteps per episode777Note that the
ProcGen environments have no explicit penalty for discouraging longer episode
lengths.. For instance, we see that the sparse dynamic model leads to an
increase of 151.7%, 409.1% & 200% in the reported navigation efficiency
scores, for the CoinRun, CaveFlyer, and Jumper environments respectively. We
discuss more on this phenomenon in Sec. 8.2.

Requirement for sufficient state space exploration. Interestingly, we also see
that a saturation in the rate of state space exploration is necessary for
getting gains with the sparse model. This is illustrated through the training
curves for the game of FruitBot (refer Fig. 3), where relative gains over the
non-sparse dynamic model occur only after a decline in the rate of increase of
average episode length.

##  8 Analysis

###  8.1 Learning Implicit State Space Decomposition

A key advantage of our method can be seen in its ability to achieve an
unsupervised division of the state space into distinct sets of game skills.
The state space decomposition is achieved through the sparse cluster
assignments, wherein the network learns to assign each state, trajectory pair
$\\{s_{t},\tau^{t-}\\}$ to a distinct value cluster. In this section, we use
this sparse property of our method to visualize different obstacle types
characteristic of each cluster in the CoinRun Environment.

To visualize the distinguishing features for each cluster, we first extract
the set of states $\mathcal{S}_{i}$ for which each cluster is active. The
latent representations (output of the LSTM network) for these states are used
to map each $s\in\mathcal{S}_{i}$ to a two dimensional embedding space using
TSNE (Maaten & Hinton, 2008). This embedding is then manually analysed for
clusters to then identify the salient obstacle classes.

Fig. 4 shows some key obstacle types for each cluster. We observe that each
cluster is responsible for predicting the value function on a distinct set of
obstacles/skills. For instance, _Cluster-1_ is responsible for value
estimation in cases like double-jump from one side to another (Skill-1) and
crossing over moving enemies (Skill-2). On the other hand, _Cluster-2_ handles
landing after jumps from higher ground (Skill-1) and high jumps with very
limited visibility of coming obstacles (Skill-2). Finally, _Cluster-3_ takes
care of precision climbs (Skill-1) and jumps over wide valleys (Skill-2).

Thus, we see that each disjoint state space set $\mathcal{S}_{i},\
i\in[1,N_{b}]$ represents a distinct curriculum of game skills that must be
learned for mastering the overall multi-scene game environment. This division
is semantically desirable and is analogous to the human learning paradigm
wherein it is quite common to break down a complex task into a set of
manageable skills before attempting the complete task.

Figure 5: Comparing Navigation Efficiency. Demonstrating qualitative
difference between trajectories for sparse and non-sparse dynamic agents. Our
method shows higher efficiency in navigating to the final goals (red & green
spheres).

###  8.2 Navigation Efficiency

A peculiar feature arising as a result of learning sparse attention parameters
(over non-sparse dynamic model) can be seen in terms of the highly improved
_navigation efficiency_. That is, the agent on average uses far fewer time-
steps per episode while achieving similar or better reward score. This massive
difference in time-steps results from two reasons.

  * •

The tendency to use fewer time-steps is a direct consequence of optimizing the
discounted reward function with $\gamma<1$ (Schulman et al., 2015). As a
result, the agent is incentivized to minimize the number of steps between the
current state and the next reward. Hence, more accurate policy updates (with
lower sample variance) should lead to fewer timesteps.

  * •

As explained in Sec. 4.2, the expansion in state space after application of
the confusion-contribution loss can lead to potential errors in value function
estimation. Thus, the sparse dynamic agent learns to maximize the utilization
of the already explored state space.

In this section, we will qualitatively analyse how learning a more accurate
value function leads to shorter (and efficient) episode trajectories for the
sparse agent. Note that the computation of a suboptimal value function at a
_critical_ environment state (_e.g._ a tricky obstacle) can cause the agent to
underestimate the _advantage_ of choosing an action which leads to a faster
route to the final destination / goal. We next try to identify these
_critical_ states by comparing episode trajectories for the sparse and non-
sparse dynamic agents on the CaveFlyer environment.

Game Description. The goal of the Caveflyer environment (refer Fig. 5) is to
destroy the red spheres and finally reach the green sphere while avoiding
intermediate obstacles. The agent receives a small reward of +3 on destroying
a red sphere and an end of episode reward of +10 on successfully reaching the
green one. Direct collisions with an obstacle or the red sphere cause
immediate episode termination.

Trajectories for both sparse and non-sparse dynamic agents are shown in Fig.
5. We see that the non-sparse agent after destruction of the red sphere
(critical state), effectively restarts its search for the next target, while
often revisiting already encountered states. In contrast, the sparse agent
with its more accurate value estimates, realizes that the expected value for
exploring unseen parts of the cave is much higher than revisiting previous
states. Doing so not only helps the sparse agent reach the end goals much
faster, but also eliminates the need for evading obstacles that it has already
crossed (thereby increasing the episode success rate).

We also note that, the balance between the sparse model’s reluctance towards
state space expansion and maximization of total reward can be modulated
through the scale of coefficients of the confusion-contribution loss. In this
regard, the high navigation efficiency of our method provides an effective
alternative to designing explicit reward shaping penalties (Zhu et al., 2017;
Wortsman et al., 2019; Laud, 2004) for promoting reduced episode lengths.

##  9 Conclusion

This paper introduces a novel dynamic value estimation strategy for enhanced
variance reduction in multi-scene reinforcement learning environments. The
proposed approach significantly outperforms the baseline model on a range of
OpenAI ProcGen environments, while exhibiting much higher navigation
efficiency to complete a game level. Additionally, we observe that the learned
sparse attention parameters divide the overall state space into disjoint
subsets. We show that each subset focuses on a distinct set of game-skills,
which is semantically desirable and draws a strong parallel with the human
learning paradigm.

## References

  * Akaike (1987) Akaike, H.  Factor analysis and aic.  In _Selected papers of hirotugu akaike_ , pp. 371–386. Springer, 1987.
  * Anderson et al. (2018) Anderson, P., Chang, A., Chaplot, D. S., Dosovitskiy, A., Gupta, S., Koltun, V., Kosecka, J., Malik, J., Mottaghi, R., Savva, M., et al.  On evaluation of embodied navigation agents.  _arXiv preprint arXiv:1807.06757_ , 2018.
  * Bellemare et al. (2017) Bellemare, M. G., Dabney, W., and Munos, R.  A distributional perspective on reinforcement learning.  In _International Conference on Machine Learning_ , pp. 449–458. PMLR, 2017.
  * Chancán & Milford (2019) Chancán, M. and Milford, M.  From visual place recognition to navigation: Learning sample-efficient control policies across diverse real world environments.  _arXiv preprint arXiv:1910.04335_ , 2019.
  * Choi et al. (2019) Choi, Y., Lee, K., and Oh, S.  Distributional deep reinforcement learning with a mixture of gaussians.  In _2019 International Conference on Robotics and Automation (ICRA)_ , pp. 9791–9797. IEEE, 2019.
  * Cobbe et al. (2019a) Cobbe, K., Hesse, C., Hilton, J., and Schulman, J.  Leveraging procedural generation to benchmark reinforcement learning.  _arXiv preprint arXiv:1912.01588_ , 2019a.
  * Cobbe et al. (2019b) Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J.  Quantifying generalization in reinforcement learning.  In _International Conference on Machine Learning_ , pp. 1282–1289, 2019b.
  * Dabney et al. (2018) Dabney, W., Rowland, M., Bellemare, M., and Munos, R.  Distributional reinforcement learning with quantile regression.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32, 2018.
  * Duan et al. (2016) Duan, Y., Schulman, J., Chen, X., Bartlett, P. L., Sutskever, I., and Abbeel, P.  Rl2: Fast reinforcement learning via slow reinforcement learning.  _arXiv preprint arXiv:1611.02779_ , 2016.
  * Espeholt et al. (2018) Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In _International Conference on Machine Learning_ , pp. 1407–1416, 2018.
  * Forster & Sober (2011) Forster, M. and Sober, E.  Aic scores as evidence: a bayesian interpretation.  In _Philosophy of Statistics_ , pp. 535–549. Elsevier, 2011.
  * Goyal et al. (2019) Goyal, A., Lamb, A., Hoffmann, J., Sodhani, S., Levine, S., Bengio, Y., and Schölkopf, B.  Recurrent independent mechanisms.  _arXiv preprint arXiv:1909.10893_ , 2019.
  * Greensmith et al. (2004) Greensmith, E., Bartlett, P. L., and Baxter, J.  Variance reduction techniques for gradient estimates in reinforcement learning.  _Journal of Machine Learning Research_ , 5(Nov):1471–1530, 2004.
  * Hochreiter & Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Igl et al. (2019) Igl, M., Ciosek, K., Li, Y., Tschiatschek, S., Zhang, C., Devlin, S., and Hofmann, K.  Generalization in reinforcement learning with selective noise injection and information bottleneck.  In _Advances in Neural Information Processing Systems_ , pp. 13956–13968, 2019.
  * Juliani et al. (2019) Juliani, A., Khalifa, A., Berges, V.-P., Harper, J., Teng, E., Henry, H., Crespi, A., Togelius, J., and Lange, D.  Obstacle tower: A generalization challenge in vision, control, and planning.  _arXiv preprint arXiv:1902.01378_ , 2019.
  * Justesen et al. (2018) Justesen, N., Torrado, R. R., Bontrager, P., Khalifa, A., Togelius, J., and Risi, S.  Illuminating generalization in deep reinforcement learning through procedural level generation.  _arXiv preprint arXiv:1806.10729_ , 2018.
  * Kanagawa & Kaneko (2019) Kanagawa, Y. and Kaneko, T.  Rogue-gym: A new challenge for generalization in reinforcement learning.  In _2019 IEEE Conference on Games (CoG)_ , pp. 1–8. IEEE, 2019\.
  * Kolve et al. (2017) Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L., Herrasti, A., Gordon, D., Zhu, Y., Gupta, A., and Farhadi, A.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_ , 2017.
  * Laud & DeJong (2003) Laud, A. and DeJong, G.  The influence of reward on the speed of reinforcement learning: An analysis of shaping.  In _Proceedings of the 20th International Conference on Machine Learning (ICML-03)_ , pp. 440–447, 2003.
  * Laud (2004) Laud, A. D.  Theory and application of reward shaping in reinforcement learning.  Technical report, 2004.
  * Liang et al. (2018) Liang, X., Wang, Q., Feng, Y., Liu, Z., and Huang, J.  Vmav-c: A deep attention-based reinforcement learning algorithm for model-based control.  _arXiv preprint arXiv:1812.09968_ , 2018.
  * Maaten & Hinton (2008) Maaten, L. v. d. and Hinton, G.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(Nov):2579–2605, 2008.
  * Nichol et al. (2018) Nichol, A., Pfau, V., Hesse, C., Klimov, O., and Schulman, J.  Gotta learn fast: A new benchmark for generalization in rl.  _arXiv preprint arXiv:1804.03720_ , 2018.
  * Osband et al. (2016) Osband, I., Blundell, C., Pritzel, A., and Van Roy, B.  Deep exploration via bootstrapped dqn.  _arXiv preprint arXiv:1602.04621_ , 2016.
  * Parisotto et al. (2015) Parisotto, E., Ba, J. L., and Salakhutdinov, R.  Actor-mimic: Deep multitask and transfer reinforcement learning.  _arXiv preprint arXiv:1511.06342_ , 2015.
  * Pennington et al. (2014) Pennington, J., Socher, R., and Manning, C. D.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pp. 1532–1543, 2014.
  * Rajeswaran et al. (2016) Rajeswaran, A., Ghotra, S., Ravindran, B., and Levine, S.  Epopt: Learning robust neural network policies using model ensembles.  _arXiv preprint arXiv:1610.01283_ , 2016.
  * Rajeswaran et al. (2017) Rajeswaran, A., Lowrey, K., Todorov, E. V., and Kakade, S. M.  Towards generalization and simplicity in continuous control.  In _Advances in Neural Information Processing Systems_ , pp. 6550–6561, 2017.
  * Schulman et al. (2015) Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P.  High-dimensional continuous control using generalized advantage estimation.  _arXiv preprint arXiv:1506.02438_ , 2015.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Song et al. (2019) Song, X., Du, Y., and Jackson, J.  An empirical study on hyperparameters and their interdependence for rl generalization.  _arXiv preprint arXiv:1906.00431_ , 2019.
  * Tobin et al. (2017) Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P.  Domain randomization for transferring deep neural networks from simulation to the real world.  In _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pp. 23–30. IEEE, 2017.
  * Wang et al. (2016) Wang, J. X., Kurth-Nelson, Z., Tirumala, D., Soyer, H., Leibo, J. Z., Munos, R., Blundell, C., Kumaran, D., and Botvinick, M.  Learning to reinforcement learn.  _arXiv preprint arXiv:1611.05763_ , 2016.
  * Whiteson et al. (2011) Whiteson, S., Tanner, B., Taylor, M. E., and Stone, P.  Protecting against evaluation overfitting in empirical reinforcement learning.  In _2011 IEEE symposium on adaptive dynamic programming and reinforcement learning (ADPRL)_ , pp. 120–127. IEEE, 2011.
  * Wortsman et al. (2019) Wortsman, M., Ehsani, K., Rastegari, M., Farhadi, A., and Mottaghi, R.  Learning to learn how to learn: Self-adaptive visual navigation using meta-learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 6750–6759, 2019.
  * Zhang et al. (2018a) Zhang, A., Ballas, N., and Pineau, J.  A dissection of overfitting and generalization in continuous reinforcement learning.  _arXiv preprint arXiv:1806.07937_ , 2018a.
  * Zhang et al. (2018b) Zhang, C., Vinyals, O., Munos, R., and Bengio, S.  A study on overfitting in deep reinforcement learning.  _arXiv preprint arXiv:1804.06893_ , 2018b.
  * Zhu et al. (2017) Zhu, Y., Mottaghi, R., Kolve, E., Lim, J. J., Gupta, A., Fei-Fei, L., and Farhadi, A.  Target-driven visual navigation in indoor scenes using deep reinforcement learning.  In _2017 IEEE international conference on robotics and automation (ICRA)_ , pp. 3357–3364. IEEE, 2017.

##  Appendix A Additional Evaluation on the Visual Navigation Task

The proposed dynamic value estimation strategy (with sparse attention) is
highly effective and widely generalizable to any task domain utilizing multi-
scene environments. In this section, we demonstrate its further efficacy on
the domain of visual navigation, which heavily relies on using multiple-scene
environments for achieving better generalization performance.

###  A.1 Task Definition

The task of visual navigation consists of a set of scenes
$\mathcalbf{S}=\\{\mathcal{S}_{1},\mathcal{S}_{2},...\mathcal{S}_{n}\\}$, and
a set of possible object classes
$\mathcalbf{O}=\\{\mathcal{O}_{1},\mathcal{O}_{2}..\mathcal{O}_{m}\\}$. Note
that the set $\mathcalbf{S}$ is just another annotation for the MDP set
$\mathcalbf{M}=\\{\mathcal{M}_{1},\mathcal{M}_{2},...\mathcal{M}_{n}\\}$, as
each scene $\mathcal{S}_{i}$ represents a distinct underlying MDP and is
characterized by a different room setup, distribution of objects and lighting
conditions.

Navigation Task Description. A single navigation task $\mathcal{T}$ consists
of an agent with action space $\mathcal{A}$ situated at a random starting
position $p$ in the one of the scenes $\mathcal{S}_{i}$. The goal of the agent
is to reach an instance of the target class $\mathcal{O}_{k}$ (given as a
Glove embedding (Pennington et al., 2014)) within a certain number of steps.
The agent then continues interacting with the environment using a policy
$\pi_{\theta}$ until it chooses a termination action. The episode is
considered a success, if and only if, at the time of termination, the target
object is sufficiently close and in the field of view of the agent.

###  A.2 Experimental design

The performance on the visual navigation task is measured using the following
evaluation metrics,

  * •

SPL. Success weighted by Path Length (SPL) (Anderson et al., 2018) provides a
measure for the navigation efficiency of the agent and is given by
$\frac{1}{N}\sum_{i=1}^{N}S_{i}\frac{L_{i}}{\max(P_{i},L_{i})}$.

  * •

Success rate. The average rate of success: $\frac{1}{N}\sum_{i=1}^{N}S_{i}$.

  * •

Total reward. Average total episode reward:
$\frac{1}{N}\sum_{i=1}^{N}\mathcal{R}_{i}$.

In above, $N$ is the number of episodes, $S_{i}\in\\{0,1\\}$ indicates the
success of an episode, $P_{i}$ is the path length, $L_{i}$ is the optimal path
length to any instance of the target object class in that scene, and
$\mathcal{R}_{i}$ is the episode reward. The baseline results are reported
using the non-adaptive and self-adapting (SAVN) A3C models from (Wortsman et
al., 2019) with 12 asynchronous workers. We then modify the critic network as
per Sec. 4 of the main paper to get the dynamic training configuration for
both baselines.

The agent is trained using the AI2-THOR environment (Kolve et al., 2017) which
consists of 120 distinct scenes. A train/val/test split of 80:20:20 is used
for selecting the best model based on highest success rate. Note that the
final results are reported after 5M episodes of training which equates to
$\approx$ 96 hours of training time on 2 Nvidia RTX 2080 Ti GPUs.

###  A.3 Results

As described in Table 2, the dynamic A3C model results in significant
improvements across all 3 performance metrics for both self-adaptive and non-
adaptive baselines. Furthermore, we note that the dynamic model provides a
huge boost in sample efficiency of the navigation agent (refer Fig. 6). For
instance, the dynamic nonadaptive A3C model reaches a test success rate of
$\approx 30$% after only 2M episodes, which in contrast with the non-adaptive
baseline takes around 4M episodes. Similarly, the dynamic model for self-
adaptive A3C achieves a test success rate of $\approx 40$% after only 1M
episodes while the corresponding baseline has a success rate of only $30.8$%.
Given the huge amounts of training time required for this task, the dynamic
model promises to be a real asset for applications with much stricter time
constraints. For instance, the high sample efficiency of our method would be
highly useful for quick fine-tuning in mobile navigation robots operating in
the real world (Chancán & Milford, 2019).

Method | SPL | Success | Total Reward
---|---|---|---
A3C | 14.3 | 31.8 | 1.413
Dynamic A3C | 15.4 | 36.5 | 1.638
SAVN | 15.19 | 37.1 | 1.652
Dynamic SAVN | 14.81 | 38.7 | 1.824

Table 2: Comparison on key evaluation metrics for visual navigation. We
observe improvements when using the dynamic critic network on both the non-
adaptive and adaptive baselines.

Figure 6: Comparing sample efficiency at training time for A3C and Dynamic A3C
model on the visual navigation task.

##  Appendix B Policy Gradient Formulation for Multi-Scene Environments

In this section, we provide a mathematical derivation for extending the
variance reduction formulation for policy gradient algorithms to multi-scene
environments. As stated in the main paper, the derived / improved formulation
shows that the sample variance for a multi-scene environment is best minimized
by learning a joint value function $V(s,\mathcal{M})$ dependent on both state
$s$ and MDP $\mathcal{M}$.

###  B.1 Policy Gradient Equation

Policy gradient algorithms aim to optimize the policy $\pi(a|s)$ by maximizing
the expected trajectory reward. For a multiple-scene environment operating
over an MDP set
${\mathcalbf{M}}:\\{\mathcal{M}_{1},\mathcal{M}_{2},...\mathcal{M}_{N}\\}$,
the expected reward is over different trajectories $\tau$ and the set of
possible MDPs $\mathcalbf{M}$. Thus,

$J=\mathbf{E}_{\tau,\mathcal{M}}[\mathcal{R}_{\tau,\mathcal{M}}],$ (9)

where $\mathcal{R}_{\tau,\mathcal{M}}$ is the total reward for a trajectory
$\tau$ sampled using MDP $\mathcal{M}$. The above equation can be broken down
as,

$\displaystyle J=\sum_{\mathcal{M}}\sum_{\tau}P(\tau,\mathcal{M})\
\mathcal{R}_{\tau,\mathcal{M}}.$ (10) $\displaystyle
J=\sum_{\mathcal{M}}P(\mathcal{M})\left(\sum_{\tau}P(\tau|\mathcal{M})\
\mathcal{R}_{\tau,\mathcal{M}}\right).$ (11)

The term $\sum_{\tau}P(\tau|\mathcal{M})\ \mathcal{R}_{\tau,\mathcal{M}}$ can
be understood as the expected reward over trajectories given a fixed
underlying MDP $\mathcal{M}$. We denote this conditional expectation using
$J_{\mathcal{M}}$. Thus,

$\displaystyle J_{\mathcal{M}}=\sum_{\tau}P(\tau|\mathcal{M})\
\mathcal{R}_{\tau,\mathcal{M}}$ (12) $\displaystyle\implies
J=\sum_{\mathcal{M}}P(\mathcal{M})\ J_{\mathcal{M}}.$ (13)

Computing gradients for both sides with respect to policy parameter $\theta$:

$\displaystyle\nabla_{\theta}J=\sum_{\mathcal{M}}P(\mathcal{M})\
\nabla_{\theta}(J_{\mathcal{M}}).$ (14)

Adopting single-MDP policy gradient derivation procedure from
(sutton2018reinforcement) for Eq. 12, the gradient with respect to
$J_{\mathcal{M}}$ can be written as:

$\displaystyle\nabla_{\theta}(J_{\mathcal{M}})=\sum_{s}\sum_{a}P(s,a|\mathcal{M})\
[\nabla_{\theta}\log\pi(a|s)]\ Q(s,a,\mathcal{M}),$ (15)

where $\pi(a|s)$ represents the scene-generic policy function and
$Q(s,a,\mathcal{M})$ is the action value function for the specific MDP
$\mathcal{M}$. Combining Eq. 14 and 15,

$\displaystyle\nabla_{\theta}J=\sum_{\mathcal{M}}P(\mathcal{M})\
\sum_{s}\sum_{a}P(s,a|\mathcal{M})\ [\nabla_{\theta}\log\pi(a|s)]\
Q(s,a,\mathcal{M}).$ (16)
$\displaystyle\nabla_{\theta}J=\sum_{\mathcal{M}}\sum_{s}\sum_{a}P(s,a,\mathcal{M})\
[\nabla_{\theta}\log\pi(a|s)]\ Q(s,a,\mathcal{M}).$ (17)

Writing Eq. 17 in the form of an expectation mean, the expression for policy
gradients in multi-scene environments is given by,

$\boxed{\nabla_{\theta}J=\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)\
Q(s,a,\mathcal{M})\right]}.$ (18)

###  B.2 Variance Reduction

The goal of variance reduction is to find a distribution with same mean as Eq.
18, but reduced variance so as to be able to compute the sample mean with
minimum error using a limited number of samples. In this section, we derive
the optimal formulation for variance reduction in _multi-scene_ environments.

Lemma 1. $Q(s,a,\mathcal{M})$ can be replaced by
$\psi(s,a,\mathcal{M})=Q(s,a,\mathcal{M})-f(s,\mathcal{M})$, where $f$ is any
general function, without affecting the overall policy gradient from Eq. 18.

Proof: Let the new policy gradient be $\nabla_{\theta}J^{\prime}$,

$\displaystyle\nabla_{\theta}J^{\prime}=\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)\
\psi(s,a,\mathcal{M})\right].$ (19)
$\displaystyle\nabla_{\theta}J^{\prime}=\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)\
(Q(s,a,\mathcal{M})-f(s,\mathcal{M}))\right].$ (20)
$\displaystyle\nabla_{\theta}J^{\prime}=\nabla_{\theta}J-\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)\
f(s,\mathcal{M})\right].$ (21)
$\displaystyle\nabla_{\theta}J^{\prime}=\nabla_{\theta}J-\sum_{\mathcal{M}}\sum_{s}\sum_{a}P(s,a,\mathcal{M})\left(\nabla_{\theta}\log\pi(a|s)\right)\
f(s,\mathcal{M}).$ (22)
$\displaystyle\nabla_{\theta}J^{\prime}=\nabla_{\theta}J-\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\
f(s,\mathcal{M})\ \sum_{a}P(a|s,\mathcal{M})\ [\nabla_{\theta}\log\pi(a|s)].$
(23)

Since we learn a common policy $\pi(a|s)$ for the entire MDP-set
$\mathcalbf{M}$, $P(a|s,\mathcal{M})=\pi(a|s)$,

$\displaystyle\nabla_{\theta}J^{\prime}=\nabla_{\theta}J-\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\
f(s,\mathcal{M})\ \sum_{a}\pi(a|s)\ \nabla_{\theta}\log\pi(a|s).$ (24)
$\displaystyle\nabla_{\theta}J^{\prime}=\nabla_{\theta}J-\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\
f(s,\mathcal{M})\ \left(\sum_{a}\ \nabla_{\theta}\ \pi(a|s)\right).$ (25)
$\displaystyle\nabla_{\theta}J^{\prime}=\nabla_{\theta}J-\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\
f(s,\mathcal{M})\ \left(\nabla_{\theta}\sum_{a}\ \pi(a|s)\right).$ (26)
$\displaystyle\nabla_{\theta}J^{\prime}=\nabla_{\theta}J-\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\
f(s,\mathcal{M})\ \cancelto{0}{\left(\nabla_{\theta}.1\right)}.$ (27)
$\displaystyle\implies\boxed{\nabla_{\theta}J^{\prime}=\nabla_{\theta}J}.$
(28)

Hence, without loss of generality we can write the multi-scene policy gradient
equation as,

$\boxed{\nabla_{\theta}J=\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)\
(Q(s,a,\mathcal{M})-f(s,\mathcal{M}))\right]}.$ (29)

Lemma 2. The optimal function $f(s,\mathcal{M})$ for minimizing the policy
gradient sample variance is equal to the joint value function
$V(s,\mathcal{M})$.

Proof: For a given tuple $\\{s,a,\mathcal{M}\\}$, the sample value
$\mathcal{X}$ is given by $\left[\nabla_{\theta}\log\pi(a|s)\right]\
\psi(s,a,\mathcal{M})$. Thus, the sample variance can be written down as,

$\displaystyle Var(\mathcal{X})=E[\mathcal{X}^{2}]-(E[\mathcal{X}])^{2}.$ (30)
Given that the function $f(s,\mathcal{M})$ is parameterized by $\phi$:
$\displaystyle\operatorname*{arg\,min}_{\phi}Var(\mathcal{X})=\operatorname*{arg\,min}_{\phi}\left(E[\mathcal{X}^{2}]-(E[\mathcal{X}])^{2}\right).$
(31) From Lemma 1, we know that the expected value of the samples remains
unchanged for any function $f$.
$\displaystyle\implies\operatorname*{arg\,min}_{\phi}Var(\mathcal{X})=\operatorname*{arg\,min}_{\phi}E[\mathcal{X}^{2}],$
(32)
$\displaystyle\operatorname*{arg\,min}_{\phi}Var(\mathcal{X})=\operatorname*{arg\,min}_{\phi}\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\left(\nabla_{\theta}\log\pi(a|s)\right)\
\psi(s,a,\mathcal{M})\right)^{2}\right].$ (33) Assuming independence between
policy network gradients and the action value function,
$\displaystyle\operatorname*{arg\,min}_{\phi}Var(\mathcal{X})=\operatorname*{arg\,min}_{\phi}\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)^{2}\right]\
\mathbf{E}_{s,a,\mathcal{M}}\left[\psi^{2}(s,a,\mathcal{M})\right],$ (34)
$\displaystyle\operatorname*{arg\,min}_{\phi}Var(\mathcal{X})=\operatorname*{arg\,min}_{\phi}\mathbf{E}_{s,a,\mathcal{M}}\left[\psi^{2}(s,a,\mathcal{M})\right],$
(35)
$\displaystyle\operatorname*{arg\,min}_{\phi}Var(\mathcal{X})=\operatorname*{arg\,min}_{\phi}\mathbf{E}_{s,a,\mathcal{M}}\left[\left(Q(s,a,\mathcal{M})-f_{\phi}(s,\mathcal{M})\right)^{2}\right].$
(36) For optimal $\phi,\quad\nabla_{\phi}Var(\mathcal{X})=0$ (37)
$\displaystyle\implies\nabla_{\phi}Var(\mathcal{X})\equiv\nabla_{\phi}\
\mathbf{E}_{s,a,\mathcal{M}}\left[\left(Q(s,a,\mathcal{M})-f_{\phi}(s,\mathcal{M})\right)^{2}\right]=0.$
(38) $\displaystyle 2\ \nabla_{\phi}f\ .\
\mathbf{E}_{s,a,\mathcal{M}}\left[f_{\phi}(s,\mathcal{M})-Q(s,a,\mathcal{M})\right]=0.$
(39)
$\displaystyle\sum_{\mathcal{M}}\sum_{s}\sum_{a}P(s,a,\mathcal{M})\left[f_{\phi}(s,\mathcal{M})-Q(s,a,\mathcal{M})\right]=0.$
(40)
$\displaystyle\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\sum_{a}P(a|s,\mathcal{M})\left[f_{\phi}(s,\mathcal{M})-Q(s,a,\mathcal{M})\right]=0.$
(41)
$\displaystyle\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\left[f_{\phi}(s,\mathcal{M})-\sum_{a}\pi(a|s)\
Q(s,a,\mathcal{M})\right]=0.$ (42)
$\displaystyle\sum_{\mathcal{M}}\sum_{s}P(s,\mathcal{M})\left[f_{\phi}(s,\mathcal{M})-V(s,\mathcal{M})\right]=0.$
(43)
$\displaystyle\mathbf{E}_{s,\mathcal{M}}\left[f_{\phi}(s,\mathcal{M})-V(s,\mathcal{M})\right]=0.$
(44) The above relation has to hold for any batch sample across
$(s,\mathcal{M})$, which implies that,
$\displaystyle\boxed{f_{\phi}(s,\mathcal{M})=V(s,\mathcal{M})}.$ (45)

Thus using Lemma 1 & 2, we can write the optimal policy gradient formulation
leading to minimum sample variance for a _multi-scene_ environment as,

$\boxed{\nabla_{\theta}J=\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)\
(Q(s,a,\mathcal{M})-V(s,\mathcal{M}))\right]}.$ (46)

##  Appendix C Relationship between sample variance and value prediction error

Theorem 1. The sample variance ($\nu$) for policy gradients defined by Eq. 46,
can be minimized by reducing the prediction error $\epsilon$ between the true
joint value function $V(s,\mathcal{M})$ and the predicted estimate
$\hat{V}(s,\mathcal{M})$, where

$\epsilon=\mathbf{E}_{s,\mathcal{M}}\left[\left(V(s,\mathcal{M})-\hat{V}(s,\mathcal{M})\right)^{2}\right].$
(47)

Proof: The equivalent sample variance ($\nu$) for policy gradients defined by
Eq. 46, can be approximated as,

$\displaystyle\nu$ $\displaystyle\approx\kappa\ .\
\mathbf{E}_{s,a,\mathcal{M}}\left[\psi^{2}(s,a,\mathcal{M})\right]$ (48)
$\displaystyle=\kappa\ .\
\mathbf{E}_{s,a,\mathcal{M}}\left[\left(Q(s,a,\mathcal{M})-\hat{V}(s,\mathcal{M})\right)^{2}\right],$
(49)

where
$\kappa=\mathbf{E}_{s,a,\mathcal{M}}\left[\left(\nabla_{\theta}\log\pi(a|s)\right)^{2}\right]$
and $\hat{V}(s,\mathcal{M})$ represents the predicted value function. Now,
using the true value function
$V(s,\mathcal{M})=\mathbf{E}_{a}\left[Q(s,a,\mathcal{M})\right]$, Eq. 49 can
be decomposed as,

$\nu\approx\kappa\ .\
\mathbf{E}_{s,a,\mathcal{M}}\left[\left(Q(s,a,\mathcal{M})-V(s,\mathcal{M})\
+\ V(s,\mathcal{M})-\hat{V}(s,\mathcal{M})\right)^{2}\right]$ (50)
$\nu\approx\underbrace{\kappa\ .\
\mathbf{E}_{s,a,\mathcal{M}}\left[\left(Q(s,a,\mathcal{M})-V(s,\mathcal{M})\right)^{2}\right]}_{\text{minimal
possible variance}}+\underbrace{\kappa\ .\
\mathbf{E}_{s,\mathcal{M}}\left[\left(V(s,\mathcal{M})-\hat{V}(s,\mathcal{M})\right)^{2}\right]}_{\text{prediction
error}}+\cancelto{0}{(\dots)}.$ (51)

Thus, we see that the policy gradient sample variance $(\nu)$ can be minimized
by reducing the error between the true value function $V(s,\mathcal{M})$ and
the predicted estimate $\hat{V}(s,\mathcal{M})$.

##  Appendix D Clustering Hypothesis: Estimating MDP-Specific Value Function

###  D.1 True Value Function

In order to make any comments about the distribution of the value function
$V(s,\mathcal{M})$ over $s\in\mathcal{S}\ \&\ \mathcal{M}\in\mathcalbf{M}$, we
first need to be able to compute the true scene-specific value estimates for
any state $s\in\mathcal{S}$ and a particular MDP $\mathcal{M}$. While training
an expert model for each level (as in (Parisotto et al., 2015)) is possible,
it suffers with the following flaws,

  1. 1.

The resulting value estimates $\\{V_{1}(s),V_{2}(s),...V_{n}(s)\\}$ would not
be over the same policy $\pi$.

  2. 2.

The learned CNN backbone for each expert model will only be able to correctly
process state images belonging to the corresponding level. This would lead to
erroneous true value predictions for state images belonging to other levels.

We instead propose the following strategy for estimation of the true MDP-
specific value function,

  * •

Train a general actor critic model over the entire MDP-set $\mathcalbf{M}$ to
learn an intermediate but sufficiently good policy $\pi$.

  * •

Freeze all the weights except the non-shared critic branch (usually a single
fc layer)

  * •

Fine-tune this semi-frozen network, individually on each MDP $\mathcal{M}_{i}$
using only the critic value loss to get the true MDP-specific value function
estimates.

###  D.2 LSTM based Value Function

While demonstrating the insufficiency of traditional CNN/LSTM based networks,
we argue that the value function learned by a CNN/LSTM network does not
capture multiple dominant modes exhibited by the true value distribution. In
this section, we outline the process used for computing the scene-specific
value estimates learned by a CNN-LSTM critic network which was jointly trained
on the entire MDP-set $\mathcalbf{M}$.

  * •

A CNN-LSTM network trained upto an intermediate policy $\pi$, is used to
generate a dataset consisting of tuples of $\\{$state images $s$, value
estimates $\hat{V}(s)$ and CNN features $\phi(s)\\}$. The associated samples
are collected over 5k episode runs with the multi-level environment.

  * •

In order to compute the value distribution samples for a target state $s_{t}$,
we first extract a set of 200 state images
$(\mathcal{S}_{c}:\\{s_{1},s_{2}\dots s_{200}\\})$, most similar to the target
state in terms of the learned CNN features $\phi$. These states are then
filtered to remove possible duplicates by setting a minimum threshold distance
$||\phi(s_{t})-\phi(s)||$ in the feature space $\phi$.

  * •

As a final check, each state $s\in\mathcal{S}_{c}$ is manually examined to
ensure semantic similarity with the target state $s_{t}$.

  * •

Finally, the learned scene-specific value distribution is analysed by
computing the histogram for the value estimates $\hat{V}(s)$ for
$s\in\mathcal{S}_{c}$.

##  Appendix E Finding the Optimal Number of Dynamic Nodes

A critical component in training the dynamic model is the selection of number
of dynamic nodes / clusters $N_{b}$. While it is possible to treat it as
another hyper-parameter, we present an alternate approach for the same. This
approach not only gives us the number of optimal clusters, but also
strengthens our belief in the function of the proposed critic network.

Recall that, given our motivation from the clustering hypothesis, we believe
that the true scene-specific value function distribution resembles a Gaussian
Model Mixture (GMM). The final dynamic model needs to learn this distribution
not just across different MDPs belonging to the set $\mathcalbf{M}$ but also
for all states $s\in\mathcal{S}$. Thus, to find out the optimal number of
gaussian clusters, we begin by approximating the multi-variate distribution
$V(s,\mathcal{M})$ using discrete samples
$\\{s_{j},\mathcal{M}_{i},V(s_{j},\mathcal{M}_{i})\\}$ for $s_{j}\in S\ \&\
\mathcal{M}_{i}\in\mathcalbf{M}$.

For the sample collection, we first obtain an intermediate policy $\pi$ and
MDP specific value estimate networks $\hat{V}_{i}(s)$ for a random selection
of 500 levels from the CoinRun ProcGen environment, using the strategy from
Section D.1. While incorporating the entire state space is infeasible, we try
to minimize the error by sampling a large collection of 1000 states from the
different levels using the common policy $\pi$. Next for each of these states
$s_{j},\ j\in[1,1000]$, we obtain the corresponding level specific value
function estimates using $\hat{V}_{i}(s_{j}),i\in[1,500]$.

The generated dataset representing samples from the multi-variate distribution
$V(s,\mathcal{M})$ has shape [N $\times$ K], where N is number of levels and K
is the number of states. The dataset is then fitted using a GMM with variable
number of components $C\in[1,10]$. We finally use the Akaike Information
Criterion (AIC) from (Akaike, 1987), to determine the optimal choice for the
number of clusters (refer Fig. 7).

[capbesideposition=right,center,capbesidewidth=5cm]figure[]

Figure 7: Finding optimal number of dynamic clusters. AIC score curves for
different number of clusters and training levels N. The local minima
represents the Bayesian optimal choice (Forster & Sober, 2011) for the number
of required dynamic / cluster nodes $N_{b}$.

Result. As shown in Fig. 7, the optimal number of clusters (point of minima in
the AIC/N curve) increases with the number of training levels. We also note
that the AIC/N curve becomes less steep, as the number of training levels
increases. This implies that given a sufficient number of training levels, the
dynamic model’s performance shows low sensitivity (higher robustness) to the
selection of the hyper-parameter $N_{b}$.


# Effective water/water contact angle at the base of an impinging jet

Théophile Gaichies  Anniina Salonen  Emmanuelle Rio
<EMAIL_ADDRESS>Université Paris-Saclay, CNRS,
Laboratoire de Physique des Solides, 91405, Orsay, France.  Arnaud Antkowiak
Sorbonne Université, CNRS, Institut Jean le Rond $\partial$’Alembert, F-75005
Paris, France

###### Abstract

The base of a jet impinging on an ultrapure water bath is studied
experimentally. At the impact point, a train of capillary waves develops along
the jet. By performing Particle Tracking Velocity measurements, we show that
there is a boundary layer separation between the jet and the meniscus. We thus
describe the shape of this meniscus with a hydrostatic model. A striking
observation is the existence of an effective non-zero water/water contact
angle between the jet and the meniscus. The rationalization of this finite
contact angle requires a full description of the shape of the interface. By
doing an analytical matching between the meniscus and the jet, we show that
the capillary waves can be considered as reflected waves present to ensure
pressure continuity. It is finally shown that the value of the apparent
contact angle is fixed by energy minimization, with an excellent agreement
between prediction and experiment for small jets.

††preprint: APS/123-QED

Liquid-air interfaces are geometrical objects with an intimate link between
their shape and their properties. They give rise to purely geometrical
questions such as surface minimization problems [1, 2]. A physical
illustration is foams, whose mechanical stability is linked to the local
bubble packing [3] and whose coarsening is fixed by topology [4, 5].
Interfacial geometry can also play a critical role in dynamical situations.
For example, the shape of the cusp at the rear of a sliding drop or an
entrained film fixes the wetting transition [6, 7]. More complex interfacial
topologies exist in multiphase flows, such as breaking waves [8] or aeration
through weirs or cascades [9]. A paradigm for these flows is the plunging jet,
which can entrain air when it impacts a bath [8]. Its geometrical features,
such as the shape of the cusp in the viscous case [10, 11], or surface
disturbances when inviscid [12], control the whole behavior of the system.

In this Letter, we focus on the base of water jets impacting a quiescent water
bath. We report on an intriguing observation: the apparition of an apparent
non-zero contact angle at the water/water junction, see Fig. 1(c). We
rationalize the presence of this contact angle and provide a description of
the whole shape of this interface.

Figure 1: (a) Experimental image of a water jet of radius $R=100\ \mu$m and
speed $V=2.1$ $\mathrm{m.s^{-1}}$ impacting a bath of water. (b) Zoom on the
capillary waves developing at the jet surface fitted by a function of the form
$r(z)=R+a\ \cos(kz+\varphi)\exp(-\frac{z}{z_{\mathrm{att}}})$. (c) Zoom on the
connection between the meniscus and the jet, where the non-zero contact angle
appears. (d) Comparison between the values of $z_{\mathrm{att}}$ and $\lambda$
extracted from the experimental fit and the ones calculated with Eq. 6 in
ref.[13] for a jet of radius $R=160\ \mu$m.

When a water jet plunges in a water bath, a train of stationary capillary
waves will be created at its base, and decay upstream (Figure 1(a) and (b))
[14]. Their wavelength can be obtained by equating the phase velocity of the
waves with the speed of the jet [15]. The damping coefficient has been modeled
and linked to the properties of the fluid [13, 16]. However, less attention
has been given to the meniscus at the connection between the jet and the bath,
where it is clear that there exists an apparent contact angle. This is
surprising as the only phases at play are water and air, hence no contact
angle is expected.

To study this contact angle, we produce jets of ultrapure water (resistivity
larger than 18.2 M$\Omega$.cm) with radii $R$ ranging from 50 to 400 $\mu$m,
using a syringe pump (Harvard Apparatus PHD ULTRA) connected to a nozzle when
$R<100\ \mu$m and a pressurized reservoir otherwise. As shown in [17], a small
amount of surfactants in the recipient bath modifies significantly the wave
pattern, so great care was taken to avoid contamination of the ultrapure water
in our experiments. The pressure in the reservoir is imposed by a pressure
controller (Elveflow OB1 Mk4) connected to a network of compressed air. These
setups can create laminar jets with speeds between $1.9$ and $8\
\mathrm{m.s^{-1}}$ for $R<100\ \mu m$ and between $1.4$ and $5\
\mathrm{m.s^{-1}}$ for bigger ones. The base of the jet is imaged using
telecentric lenses (SilverTl 2x or 4x Edmund Optics) to obtain sharp
interfaces and a digital camera (Basler a2A1920-160umPRO). The images were
typically taken with an exposure time of 700 $\mu$s.

Measuring the observed contact angle is not straightforward, as it is not
possible to place the contact line between the jet and the meniscus in a non-
arbitrary manner. To overcome this problem, we propose to measure the contact
angle by fitting the shape of the interface. Despite the jet having a speed of
a few meters per second, the meniscus at the base of the jet looks very
similar to the one climbing a static fiber plunged in a liquid bath (visual
comparison in Fig.2(a) and (b)), as was previously noticed by Patrascu $et\
al.$ [13]. We performed PTV measurements below the interface, using small
silica particles (of radius between 30 and 80 $\mu$m), and a fast camera
(Photron FASTCAM NOVA S9) recording 6000 images per second. These images were
then analysed with Python. The results can be seen in Fig.2(c) and (d) for two
different jet radii and velocities. The measurements confirm that there is a
boundary layer separation between the dynamic jet and the static meniscus,
with a typical velocity ratio of 20 between the velocity in the dynamic jet
and in the still zone of the meniscus. This justifies the use of hydrostatic
equations to describe the shape of the meniscus.

The shape of the static meniscus is fixed by a balance between the hydrostatic
pressure $\rho gz$, with $\rho$ the liquid density, $g$ the acceleration of
gravity and $z$ the surface elevation and the Laplace pressure
$\gamma\mathcal{C}$, with $\gamma$ the liquid/air surface tension and
$\mathcal{C}$ the curvature of the interface. The typical length scale of the
meniscus is thus the capillary length $\ell_{c}=\sqrt{\frac{\gamma}{\rho g}}$.
An equation for $z(r)$, with $r$ the coordinate in the radial direction, in
the non-linear part of the meniscus attached to a fiber of radius $r_{0}$ with
a non-zero contact angle has been proposed by James [18]. The linear small
slope solution is $z(r)=AK_{0}(\frac{r}{l_{c}})$, where $A$ is a constant and
$K_{0}$ a modified Bessel function of the second kind of order zero. By
matching it with the equation of a catenoid satisfying the boundary condition
$z^{\prime}(r_{0}/r_{0})=-\mathrm{cotan}(\theta)$ where $\theta$ is the
contact angle, he obtains, in the vicinity of the fiber

$\begin{split}z(r)=&r_{0}\cos(\theta)\left[-\ln(\epsilon)+\ln(4)-\Gamma\right.\\\
&\left.-\ln\left(\frac{r}{r_{0}}+\left(\left(\frac{r}{r_{0}}\right)^{2}-\cos^{2}\left(\theta\right)\right)^{\frac{1}{2}}\right)\right],\end{split}$
(1)

where $\epsilon=\sqrt{\frac{\rho gR^{2}}{\gamma}}$ is the square root of the
Bond number and $\Gamma$ is Euler’s constant. We use James’ equation to fit
the meniscii in Fig. 2(a) and (b). The contact angle is used as a free
parameter and can be different on the right and on the left. This leads to
slightly different values of the contact angle on the jet, which will be taken
into account while evaluating the error bars. On the fiber, the difference is
larger, probably due to impurities at its surface. The agreement between the
fit and the experimental shape confirms that this region can be considered as
static, and that there exists a non-zero contact angle between the meniscus
and the jet. Replacing directly the fiber radius $r_{0}$ in James’ equation
with the jet radius $R$ proves unsatisfactory, for the actual radius is
altered with the capillary wavefield. We thus need to write proper matching
conditions between the capillary wavefield and the static meniscus.

Figure 2: (a) Fit of the meniscus, on the right and on the left, by equation
(1) for a jet of radius $R=280\ \mu$m and speed $V=2.65\ \mathrm{m.s^{-1}}$
and for (b) a glass fiber of $R=250\ \mu$m. (c) Results of the PTV experiments
under the water surface for a jet of radius $R=280\ \mu$m and speed $V=2\
\mathrm{m.s^{-1}}$ and a jet of radius $R=80\ \mu$m and speed $V=2.8\
\mathrm{m.s^{-1}}$

In the following, we propose a matching between James’ description of the
meniscus and the jet decorated with capillary waves. The parameters needed to
describe the meniscus using James’ equation are $r_{0}$ and $\theta$. The
capillary wavefield is approximated by an attenuated sinusoidal wave
$r(z)=R+a\ \cos(kz+\varphi)\exp(-\frac{z}{z_{\mathrm{att}}})$, with
$k=\frac{2\pi}{\lambda}$ the wavenumber, $\lambda$ the wavelength, $a$ the
amplitude, $\varphi$ a phase shift, and $z_{\mathrm{att}}$, the attenuation
length. $k$ and $z_{\mathrm{att}}$ are predicted by the model of Patrascu
(equation (6) in reference [13]). It agrees very well with the fitted values
of $z_{\textrm{att}}$ and $\lambda$, as can be seen in Fig. 1d. We then have
five unknowns $\theta,\ r_{0},\ z_{0},\ \varphi,\mathrm{and}\ a$. The first
matching condition is the radial matching position, which gives
$r_{0}=r(z_{0})$. The height matching position $z_{0}$ is given by James’
equation $z_{0}=z(r_{0})$. The slope matching yields $\frac{\partial
r}{\partial z}=-\tan(\theta)$. The last condition is the matching between the
jet curvature

$\kappa_{\textrm{jet}}=\frac{1+r^{\prime}(z)^{2}-r(z)r^{\prime\prime}(z)}{r(z)[1+r^{\prime}(z)^{2}]^{3/2}}$

and the meniscus curvature $\kappa_{\textrm{m}}=0$, as the meniscus is a
catenoid. Practically we used a linearized version of $\kappa_{\text{jet}}$ as
full nonlinear computations did not show significant differences.

This curvature matching is in fact a pressure matching between the jet, which
has an overpressure $\Delta P=\frac{\gamma}{R}$, and the meniscus, which has
an underpressure $\Delta P=-\rho gz$. Capillary wave emission stems from this
pressure mismatch. The reflected waves lower the curvature of the impinged jet
base to accommodate with the meniscus’ pressure, as sketched in Fig.3a. 3a.
This is similar to the stationary wave rippling the air cavity entrained
behind an object dropped in a liquid[19]. In this case, a reflected stationary
wave is needed so that the cavity is attached to the side of the moving
object. These situations are reminiscent of classical problems in wave
physics, where boundary conditions are impossible to meet without a reflected
wave [20].

With five unknowns but only four matching conditions, the problem is under-
constrained as stated. However, if we set a $\theta$, we can get all the other
values thanks to the following system:

$\begin{split}&r_{0}=\frac{R\left(R^{2}+z_{\mathrm{att}}^{2}(k^{2}R^{2}-2)+2z_{\mathrm{att}}R\tan(\theta)\right)}{R^{2}+z_{\mathrm{att}}^{2}(k^{2}R^{2}-1)},\\\
&z_{0}=r_{0}\cos(\theta)\left(\log(\frac{4R}{r_{0}\epsilon(1+\sin(\theta))})-\Gamma\right),\\\
&\varphi=-kz_{0}+\arctan\left(\frac{r_{0}-R-z_{\mathrm{att}}\tan(\theta)}{kz_{\mathrm{att}}(R-r_{0})}\right),\\\
&a=\frac{(r_{0}-R)\exp(z_{0}/z_{\mathrm{att}})}{\cos(kz_{0}+\varphi)},\end{split}$
(2)

where $\varphi$ can be shifted by $\pi$ to get a positive amplitude. To test
that our equations of matching indeed set the amplitude of the capillary
waves, we follow this fitting procedure: for a jet of given $R$ and $V$, we
first get $k$ and $z_{\mathrm{att}}$ by fitting the wave pattern at the base
of the jet, or from Patrascu equation (that only depends on $R$ and $V$) when
the waves become barely visible on our images. This fixes $r_{0}(\theta)$
through Eq. 2. We then obtain $\theta$ by fitting the meniscus with James’
equation.

Figure 3: (a) Sketch of the jet impacting a bath. In the first virtual case, a
straight jet connects to the meniscus, and exposes the impossibility of a
pressure continuity between the two regions. In the second case, the capillary
waves allow for pressure continuity. (b) Fitted interfaces of jets of various
radii and speeds. The red line is the fitted meniscus, the blue line the
wavefield with the calculated amplitude, and the violet cross the calculated
matching point $(r_{0},z_{0})$.

When $\theta$ is fitted, we inject it in system 2 and deduce the values of
$z_{0},\ \varphi$ and $a$. In Fig.3b, the resulting wavefield is plotted on
the image, together with James’ equation for the fitted $\theta$. We can see
that, for various radii and jet velocities, the obtained amplitude agrees very
well with the experimental one. This confirms that the pressure matching
between the meniscus and the jet sets the amplitude of the capillary
wavefield.

To get a complete model of the interface, we miss a prediction for $\theta$,
as all $\theta$ are compatible with our equations of matching. However, the
total energy of the system depends on $\theta$. As can be seen in Fig.4a, when
$\theta$ is small, the meniscus joins the jet at a higher altitude, resulting
in an excess surface compared to a flat meniscus. When $\theta$ is large, the
amplitude of the capillary waves is larger, so the excess of surface compared
to an unperturbed jet is higher. The kinetic energy excess caused by the
capillary waves is equal to the surface energy excess according to
Chandrasekhar (equation (154) and (57) of chapter XII in [21]). It is then
possible to calculate the difference in energy $\Delta E$ with a virtual
cylindrical waveless jet and flat meniscus as a function of $\theta$:

$\begin{split}\Delta E_{\mathrm{tot}}(\theta)=\
2\pi\gamma&\left(2\int_{z_{0}(\theta)}^{\infty}[R(z,\theta)\sqrt{1+\frac{\partial
R(z,\theta)}{\partial z}^{2}}-R]dz\right.\\\
&\left.+\int_{r_{0}(\theta)}^{\infty}r(\sqrt{1+\frac{\partial
z(r,\theta)}{\partial r}^{2}}-1)dr\right)\\\ &+\rho
g\int_{r_{0}(\theta)}^{\infty}\pi rz(r,\theta)^{2}dr,\end{split}$ (3)

where the last term accounts for the gravitational potential energy of the
meniscus 111The meniscus energy term has here been computed using the
uniformly valid composite solution of James, see §6 of james [18]. The sum of
the jet and meniscus energies is plotted in Fig. 4a, revealing a value
$\theta_{\mathrm{min}}$ which minimizes $\Delta E_{\mathrm{tot}}$. This
$\theta_{\mathrm{min}}$ depends on $V$ and $R$ as these variables set $k$ and
$z_{\mathrm{att}}$, which affect $\Delta E_{\mathrm{jet}}$. In Fig.4b, the
prediction of $\theta_{\mathrm{min}}$ is plotted together with the
experimental data as a function of $V$ for different radius. The uncertainty
for the experimental value of $\theta_{\mathrm{w}}$ comes from the numerical
uncertainty on the fit of the meniscus. For small jets ($R=56\ \mathrm{\mu m}$
and $R=76\ \mathrm{\mu m}$), the model agrees well with the experiments
without any fitting parameters, especially when $V>3\ \mathrm{m.s^{-1}}$. For
lower speeds, $a/R$ is not negligible and capillary waves are not well
described by an attenuated sinusoid [23] (see the first image in Fig.3b), so
we expect a bigger difference with our model. For wider jets, non
stationarities, and a certain degree of swirl [13] can appear (see Fig.4b),
which result in deviations from our model, as our assumptions of stationarity
and axisymmetry are no longer fulfilled.

Figure 4: (a) Energy excess of the meniscus ($\Delta E_{\mathrm{jet}}$), of
the jet ($\Delta E_{\mathrm{meniscus}}$) and their sum ($\Delta
E_{\mathrm{tot}}$) plotted against the speed for a jet of radius $R=190\
\mu$m. The inserted sketches show the difference between the shape of the
interface for a low and high contact angle. (b) Apparent contact angle fitted
and predicted by our model for jets of various radii, plotted against the
speed of the jet. The two experimental images show two reasons for the
deviation from our model, the first one being instationarity (at the top) and
the second one (at the bottom) the loss of axisymmetry, even when the
stationarity is restored.

In conclusion, the observation of an apparent contact angle when a jet impacts
a bath has been rationalized. Capillary waves are due to a pressure matching
between the jet and the meniscus, and energy minimization fixes the effective
contact angle at least for small jets. Our data are in excellent agreement
with this prediction for tiny jets. Discrepancies appearing for wider jets
underline the necessity to better understand the non-stationary and asymmetric
regimes.

## References

  * [1] Richard Courant.  The existence of a minimal surface of least area bounded by prescribed jordan arcs and prescribed surfaces.  Proceedings of the National Academy of Sciences, 24(2):97–101, 1938\.
  * [2] Joseph Antoine Ferdinand Plateau.  Statique expérimentale et théorique des liquides soumis aux seules forces moléculaires, volume 2.  Gauthier-Villars, 1873.
  * [3] S Heitkam, W Drenckhan, and J Fröhlich.  Packing spheres tightly: influence of mechanical stability on close-packed sphere structures.  Physical review letters, 108(14):148302, 2012.
  * [4] John Von Neumann.  Metal interfaces.  American Society for Metals, Cleveland, 108:108–110, 1952.
  * [5] Sascha Hilgenfeldt, Andrew M Kraynik, Stephan A Koehler, and Howard A Stone.  An accurate von neumann’s law for three-dimensional foams.  Physical review letters, 86(12):2685, 2001.
  * [6] Nolwenn Le Grand, Adrian Daerr, and Laurent Limat.  Shape and motion of drops sliding down an inclined plane.  Journal of Fluid Mechanics, 541:293–315, 2005.
  * [7] Jacco H Snoeijer, Giles Delon, Marc Fermigier, and Bruno Andreotti.  Avoided critical behavior in dynamically forced wetting.  Physical review letters, 96(17):174504, 2006.
  * [8] Kenneth T Kiger and James H Duncan.  Air-entrainment mechanisms in plunging jets and breaking waves.  Annual Review of Fluid Mechanics, 44:563–596, 2012.
  * [9] DA Ervine.  Air entrainment in hydraulic structures: A review.  Proceedings of the Institution of Civil Engineers-Water Maritime and Energy, 130(3):142–153, 1998.
  * [10] Jens Eggers.  Air entrainment through free-surface cusps.  Physical review letters, 86(19):4290, 2001.
  * [11] Élise Lorenceau, David Quéré, and Jens Eggers.  Air entrainment by a viscous jet plunging into a bath.  Physical review letters, 93(25):254501, 2004.
  * [12] Yonggang Zhu, Hasan N Oğuz, and Andrea Prosperetti.  On the mechanism of air entrainment by liquid jets at a free surface.  Journal of Fluid Mechanics, 404:151–177, 2000.
  * [13] Claudiu Patrascu and Corneliu Balan.  Decay of stationary capillary waves on impinging liquid jets.  Physics of Fluids, 33(2), 2021.
  * [14] D Sklavenites.  Wave patterns on a water column.  American Journal of Physics, 65(3):225–230, 1997.
  * [15] J. Lighthill.  Waves in Fluids.  Cambridge University Press, 1978.
  * [16] KM Awati and T Howes.  Stationary waves on cylindrical fluid jets.  American Journal of Physics, 64(6):808–811, 1996.
  * [17] Matthew J Hancock and John WM Bush.  Fluid pipes.  Journal of Fluid Mechanics, 466:285–304, 2002.
  * [18] David F James.  The meniscus on the outside of a small circular cylinder.  Journal of Fluid Mechanics, 63(4):657–664, 1974.
  * [19] Torben Grumstrup, Joseph B Keller, and Andrew Belmonte.  Cavity ripples observed during the impact of solid objects into liquids.  Physical review letters, 99(11):114502, 2007.
  * [20] John Billingham and Andy C King.  Wave motion.  Number 24. Cambridge university press, 2000.
  * [21] Subrahmanyan Chandrasekhar.  Hydrodynamic and hydromagnetic stability.  Courier Corporation, 2013.
  * [22] The meniscus energy term has here been computed using the uniformly valid composite solution of James, see §6 of james [18].
  * [23] J.-M. Vanden-Broeck, T. Miloh, and B. Spivack.  Axisymmetric capillary waves.  Wave Motion, 27(3):245–256, apr 1998.