# Understanding the Effects of Second-Order Approximations in Natural Policy
Gradient Reinforcement Learning

Brennan Gebotys  University of Waterloo
<EMAIL_ADDRESS>Alexander Wong  University of Waterloo
<EMAIL_ADDRESS>David A Clausi  University of Waterloo
<EMAIL_ADDRESS>
###### Abstract

Natural policy gradient methods are popular reinforcement learning methods
that improve the stability of policy gradient methods by utilizing second-
order approximations to precondition the gradient with the inverse of the
Fisher-information matrix. However, to the best of the authors’ knowledge,
there has not been a study that has investigated the effects of different
second-order approximations in a comprehensive and systematic manner. To
address this, five different second-order approximations were studied and
compared across multiple key metrics including performance, stability, sample
efficiency, and computation time. Furthermore, hyperparameters which aren’t
typically acknowledged in the literature are studied including the effect of
different batch sizes and optimizing the critic network with the natural
gradient. Experimental results show that on average, improved second-order
approximations achieve the best performance and that using properly tuned
hyperparameters can lead to large improvements in performance and sample
efficiency ranging up to +181%. We also make the code in this study available
at https://github.com/gebob19/natural-policy-gradient-reinforcement-learning.

##  I Introduction

The policy gradient method [18] is a popular optimization method for
reinforcement learning problems; however, it suffers from unstable training
due to high-variance gradient estimates [14]. A promising solution is to
leverage natural policy gradient methods, which preconditions the gradient
with the inverse of the Fisher-information matrix [2] to restrict how much the
model can change between training iterations.

For neural networks, which can have tens of millions of parameters, directly
computing the inverse of the Fisher-information matrix is intractable since
the Fisher-information matrix is an $n_{\theta}\times n_{\theta}$ matrix,
where $n_{\theta}$ is the number of parameters in the model. This led to the
research community developing new approximations for the inverse of the
Fisher-information matrix which led to having to choose the most appropriate
approximation method. Furthermore, to generate effective performance, setting
the batch size and how to optimize the critic network are also important
details. To the best of the authors’ knowledge, there has not been a study
that has holistically investigated strategies for choosing these details to
achieve high performance with natural policy gradient methods in a
comprehensive and systematic manner.

In this paper, we study the effects of five different second-order
approximations: Hessian-free optimization (HF), diagonal approximations
(Diagonal), Kronecker-factored approximate curvature (KFAC), Eigenvalue-
corrected Kronecker factorization approximate curvature (EKFAC), and Time-
efficient natural gradient descent (TENGraD). Across these approximations, we
investigate hyperparameters which aren’t typically given importance in policy
gradient research including, the effect of using different batch sizes and the
effect of using different optimization methods on the critic network. We also
study these approximations and hyperparameters across multiple reinforcement
learning environments and record statistics on performance, training
stability, sample efficiency, and computation time.

##  II Background

In this section, we first define the notation used throughout the paper,
followed by a brief overview of the policy gradient method, the natural policy
gradient method, and the five different second-order approximations which can
be used to implement the natural policy gradient method.

We consider an infinite-horizon discounted Markov decision process, defined by
the tuple ($\mathcal{S}$, $\mathcal{A}$, $p$, $r$, $\gamma$) which defines the
state distribution $\mathcal{S}$, action distribution $\mathcal{A}$, state
transition function $p$, reward function $r$, and discount factor $\gamma$.
Specifically, at time $t$, an agent is in a state $s_{t}\in\mathcal{S}$ and
samples an action $a_{t}\in\mathcal{A}$ from its policy, $\pi(a_{t}|s_{t})$.
The environment then produces a reward $r(a_{t},s_{t})$ and a new state
$s_{t+1}\in\mathcal{S}$ according to the transition probability function
$p(s_{t+1}|a_{t},s_{t})$. Repeating this process $n$ times produces a
trajectory, $\tau$, of length $n$.

###  II-A Policy Gradient Methods

The policy gradient method [18] parameterizes its policy with parameters
$\theta$ and aims to minimize the negative expected reward of its
trajectories:

$\underset{\theta}{\mathop{\mathrm{argmin}}}\,L(\theta)=\underset{\theta}{\mathop{\mathrm{argmin}}}\,-\mathbb{E}_{\tau\sim\pi(\tau|\theta)}[r(\tau)\,|\,\pi(\cdot|\theta)]$
(II.1)

where for each time step $t$ in the trajectory $\tau$:
$r(\tau)_{t}=\sum_{i=0}^{\infty}\gamma^{i}r(a_{t+i},s_{t+i})$ is the
discounted cumulative reward after taking action $a_{t}$ in state $s_{t}$ and
$\pi(\cdot|\theta)_{t}=\pi(a_{t}|s_{t},\theta)$ is the policy’s probability of
taking action $a_{t}$. In general, to optimize (II.1), the policy gradient
method uses the following gradient:

$\displaystyle\nabla_{\theta}L(\theta)$
$\displaystyle=-\mathbb{E}_{\tau\sim\pi(\tau|\theta)}[\Psi\nabla_{\theta}\text{log}\pi(\cdot|\theta)]$
(II.2)

where $\Psi$ is typically chosen as an advantage function, $A$, which has the
general form of: $A(a_{t},s_{t})=r(\tau)_{t}-V(s_{t})$, where $V(s_{t})$ is
the expected cumulative reward of the policy in state $s_{t}$. In our
experiments, we set $\Psi$ to use the generalized advantage estimation method
(GAE) [16] which involves training a critic network to estimate $V(s_{t})$.
While the policy gradient method directly optimizes (II.2), doing so often
results in high-variance gradient estimates which leads to unstable training
and poor performance [14].

###  II-B Natural Policy Gradient Methods

To stabilize training, the natural policy gradient method [2] restricts how
much the policy can change across training iterations by adding a
Kullback–Leibler divergence constraint ($D_{KL}$) between policy iterations:

$\theta^{(k+1)}=\underset{\theta^{(k+1)}}{\mathop{\mathrm{argmin}}}\,L(\theta^{(k+1)})+\lambda
D_{KL}(\pi(\cdot|\theta^{(k)}),\pi(\cdot|\theta^{(k+1)}))$ (II.3)

where $\theta^{(k)}$ defines the parameters at optimization iteration $k$. To
approximately solve (II.3), we use a first-order taylor-series approximation
of $L$ around $\theta^{(k)}$, a second-order taylor-series approximation of
$D_{KL}$, and solve for the optimum (see [6] and [4] for a full derivation).
In doing so, we derive the natural policy gradient method,

$\theta^{(k+1)}=\theta^{(k)}-{\lambda}^{-1}{F}^{-1}\nabla L(\theta^{(k)})$
(II.4)

where $F$ is the Fisher-information matrix:

$F=\mathbb{E}_{\pi(\cdot|\theta)}[\nabla_{\theta}\log\pi(\cdot|\theta)\nabla_{\theta}\log\pi(\cdot|\theta)^{T}]$
(II.5)

Figure 1: Diagram of the two hyperparameters introduced in this paper for
natural policy gradient methods: tuning the batch size for performance and
optimizing the critic method using the natural gradient. These strategies can
be applied across all second-order approximations for any reinforcement
learning environment.

##  III Second-Order Approximations

A key problem is that the Fisher-information matrix ($F$) is an
$n_{\theta}\times n_{\theta}$ matrix, where $n_{\theta}$ is the number of
parameters in the model. For neural networks which can have tens of millions
of parameters, directly computing the inverse of the Fisher-information matrix
is intractable. This has led to rising interest in the research community to
develop new ways to approximate the inverse of the Fisher-information matrix
for neural networks. However, most of these approximations have never been
fully investigated in reinforcement learning (RL) tasks. Below, we review five
of the most popular approximations and their related work in RL.

Diagonal Approximations (Diagonal) [11]: Diagonal approximations use a
diagonal approximation of $F$. This makes storage and inversion easy, however,
it ignores a large amount of curvature information. While some RL research [7]
indirectly use the diagonal approximation by using the Adam optimizer [9],
there has not been any specific focus on understanding the fundamental trade-
offs when using Diagonal approximations compared to other approximations when
realizing the natural policy gradient.

Hessian-free Approximations (HF) [12]: Hessian-free approximations use matrix-
vector products and the conjugate gradient method to approximately solve for
${F}^{-1}\nabla L(\theta)$. However, computing the conjugate gradient at each
iteration can be computationally expensive. One of the most popular
implementations of the natural policy gradient which used the HF approximation
is Trust region policy optimization (TRPO) [15] which used a backtracking line
search method to achieve state-of-the-art results in continuous control tasks
at the time.

Kronecker-factor Approximations (KFAC/EKFAC) [13]: Kronecker-factor
approximations use a block-diagonal approximation of $F$ which assumes each
layer is independent of the others. KFAC [13] decomposes the Fisher-
information matrix using the Kronecker product. EKFAC [5] uses a different
approximation based on KFAC which showed improved results across computer
vision experiments. In RL, Actor-critic using kronecker-factored trust region
(ACKTR) [19] used KFAC optimization and step-size clipping to realize the
natural policy gradient and showed improved results compared to TRPO. EKFAC
has never been investigated in the RL literature.

Woodbury Approximations (TENGraD) [17]: While EKFAC proposes an improved
approximation to KFAC, TENGraD provides an exact block-diagonal approximation
using the Woodbury identity. However, TENGraD requires constructing a $m\times
m$ matrix where $m$ is the batch size which can be computationally expensive
when using large batch sizes. TENGraD has never been investigated in the RL
literature.

##  IV Key Hyperparameters

In this section, we give a brief introduction to the hyperparameters we
investigate. This includes batch size tuning and using the natural gradient to
optimize the critic network. Figure 1 shows a diagram of the two
hyperparameters.

###  IV-A Batch Size Tuning

In the policy gradient setting, the batch size defines how many training
examples to collect in the environment before using the collected data to
update the model. Using a larger batch size will result in more examples, for
a more stable update, but fewer model updates per environment step.
Comparatively, using a smaller batch size will result in fewer examples, for a
less stable update, but more updates per environment step. Although the
importance of the batch size is not emphasized much in the literature, we find
that tuning the batch size is a critical hyperparameter that can significantly
affect the final performance and other metrics of the model (a similar result
was found for Q-learning methods [8]).

To understand how a tuned batch size can improve the baseline performance, we
tune the batch size of the baseline models based on the performance metric.

Figure 2: Average return vs the number of environment steps of five different second-order approximations across seven different MuJoCo environments using the baseline hyperparameters. All experiments are across 10 random seeds. The solid-colored line represents the mean value across the different seeds and the shaded region represents one standard deviation above and below the mean. TENGraD (red) achieves the best performance across most environments. HF is the most stable across seeds. Approximation | Performance | Stability | Sample Efficiency | Computation Time
---|---|---|---|---
Diagonal | 12.52 | -12338.0 | -2170.0 | -37.051
HF | 68.73 | -107.0 | -1708.0 | -37.282
KFAC | 29.03 | -178.0 | -1263.0 | -37.430
EKFAC | 61.85 | -235.0 | -1316.0 | -37.624
TENGraD | 85.69 | -182.0 | -1425.0 | -52.861
TABLE I: Mean metrics of five different approximations across seven different
MuJoCo environments. The best result for each metric is in bold. Larger values
are better. TENGraD achieves the best performance score; HF achieves the best
stability score; KFAC achieves the best sample efficiency score, and Diagonal
achieves the best computation time score.

###  IV-B Natural Critic Optimization

Another hyperparameter is to improve the optimization of the critic network
using the natural gradient. While some work investigated using clipped targets
to stabilize training [3], ACKTR [19] used the natural gradient to further
improve the stability and showed improved performance using KFAC optimization
for both the policy and the critic network.

While ACKTR investigated applying only KFAC optimization to both the policy
network and the critic network, we investigate five different second-order
approximations for the policy network and for each of these we investigate
five second-order approximations for tuning the critic network. To achieve
this, we leverage the models with a tuned batch size and use grid search to
find the best approximation to optimize the critic network with. The results
are shown in Section V-B.

##  V Experiments

In this section, we investigate the performance of the five second-order
approximations and the effect of our hyperparameters. Furthermore, in doing
so, we also investigate the performance of EKFAC and TENGraD approximations
for natural policy gradient reinforcement learning for the first time in the
literature. We compare each approximation across seven different MuJoCo
environments: HalfCheetah-v2, Ant-v2, Hopper-v2, Humanoid-v2, Walker2d-v2,
Reacher-v2, and Swimmer-v2. All experiments were run across 10 different
random seeds.

Throughout the experiments, we study four metrics: Performance which is the
maximum average return achieved by the policy across training; Stability which
is the average standard deviation (across seeds) across training; Sample
Efficiency which is the number of environment steps to achieve a specific
performance threshold; and Computation Time which is the amount of time to
take 100K environment steps while training. More information on these metrics
is available in Appendix B.

For clarity purposes, to make all metrics correspond with ‘larger is better’,
we flip the sign of the metrics where smaller values are better (i.e.,
Stability, Sample Efficiency, and Computation Time). For example, if two
agents achieve an average standard deviation (across seeds) of 500 and 600
(where 500 is a smaller standard deviation and is thus more stable), we flip
the signs to get -500 and -600, in which case the policy with the larger value
of -500 is more stable and is therefore considered better. All hyperparameter
information can be found in Appendix C.

###  V-A Baseline

Figure 3: Normalized performance scores of all second-order approximations across four aggregate metrics with 95% stratified bootstrap confidence intervals. A larger score in the median, IQM, and mean aggregations and a smaller score in the optimality gap aggregation corresponds to better performance. We see TENGraD statistically outperforms all other approximations across IQM, mean, and the optimality gap aggregation metrics. The median aggregate metric shows a trend that newer second-order approximations lead to better median performance. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 1687 (+67%) | -439 (+16%) | -2775 (-50%) | -22 (+18%)
Ant-v2 | 825 (+94%) | -178 (+100%) | -1025 (+10%) | -43 (+18%)
Hopper-v2 | 527 (-55%) | -48 (+87%) | NaN | -27 (+16%)
Humanoid-v2 | 486 (-33%) | -20 (+72%) | NaN | -50 (+19%)
Walker2d-v2 | 526 (-71%) | -59 (+83%) | -2175 (-222%) | -28 (+18%)
Reacher-v2 | -12 (+14%) | -304 (+92%) | -625 (-34%) | -21 (+19%)
Swimmer-v2 | 33 (-10%) | -4 (+7%) | NaN | -22 (+15%)
TABLE II: Performance of the Diagonal approximation across seven different environments with a tuned batch size and where the critic network is optimized with the natural gradient. The percentage improvement compared to the baseline is in brackets. The largest improvement for each metric is in bold. We see performance, stability and sample efficiency improvements up to +94%, +100%, and +10% with up to a +19% improvement in the speed score. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 1506 (+32%) | -216 (-423%) | -1120 (+62%) | -29 (-6%)
Ant-v2 | 1573 (+65%) | -143 (-189%) | -1370 (+57%) | -54 (-3%)
Hopper-v2 | 504 (-78%) | -45 (+85%) | NaN | -34 (-3%)
Humanoid-v2 | 499 (-39%) | -13 (+60%) | NaN | -62 (-3%)
Walker2d-v2 | 398 (-86%) | -35 (+89%) | NaN | -36 (-4%)
Reacher-v2 | -4 (+11%) | -0 (+63%) | -270 (+44%) | -29 (-9%)
Swimmer-v2 | 47 (+15%) | -5 (-57%) | -820 (+35%) | -30 (-12%)
TABLE III: Performance of the HF approximation across seven different environments with a tuned batch size and where the critic network is optimized with the natural gradient. The percentage improvement compared to the baseline is in brackets. The largest improvement for each metric is in bold. We see performance, stability and sample efficiency improvements up to +65%, +89%, and +62% with at most a 12% decrease in the speed score. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 2319 (+38%) | -396 (-52%) | -480 (+29%) | -27 (-1%)
Ant-v2 | 2593 (+65%) | -386 (-106%) | -240 (+38%) | -52 (+0%)
Hopper-v2 | 494 (-60%) | -86 (+85%) | NaN | -32 (+2%)
Humanoid-v2 | 451 (-36%) | -34 (+48%) | NaN | -62 (-0%)
Walker2d-v2 | 337 (-24%) | -95 (+36%) | NaN | -35 (+1%)
Reacher-v2 | -7 (+19%) | -1 (-40%) | -360 (-14%) | -26 (+1%)
Swimmer-v2 | 52 (+53%) | -15 (-87%) | -1770 (+53%) | -27 (+1%)
TABLE IV: Performance of the KFAC approximation across seven different environments with a tuned batch size and where the critic network is optimized with the natural gradient. The percentage improvement compared to the baseline is in brackets. The largest improvement for each metric is in bold. We see performance, stability and sample efficiency improvements up to +65%, +85%, and +53% with at most a 1% decrease in the speed score. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 2513 (+11%) | -603 (-17%) | -570 (+0%) | -27 (-1%)
Ant-v2 | 2345 (+50%) | -325 (-10%) | -375 (+38%) | -52 (-0%)
Hopper-v2 | 495 (-74%) | -66 (+86%) | NaN | -33 (+4%)
Humanoid-v2 | 456 (-37%) | -21 (+35%) | NaN | -63 (-0%)
Walker2d-v2 | 403 (-72%) | -72 (+77%) | NaN | -36 (-2%)
Reacher-v2 | -6 (+10%) | -1 (+36%) | -615 (-5%) | -27 (-2%)
Swimmer-v2 | 37 (-11%) | -7 (+12%) | -2910 (-169%) | -27 (-1%)
TABLE V: Performance of the EKFAC approximation across seven different environments with a tuned batch size and where the critic network is optimized with the natural gradient. The percentage improvement compared to the baseline is in brackets. The largest improvement for each metric is in bold. We see EKFAC achieves improvements in performance, stability, and sample efficiency of up to +50%, +86%, and +38% respectively with at most a 1% decrease in the speed score. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 2058 (+15%) | -592 (-69%) | -788 (+52%) | -44 (-5%)
Ant-v2 | 3855 (+113%) | -464 (-153%) | -652 (+59%) | -71 (-4%)
Hopper-v2 | 1510 (-14%) | -148 (+35%) | -4256 (-1008%) | -50 (-4%)
Humanoid-v2 | 640 (-26%) | -21 (+60%) | NaN | -79 (-2%)
Walker2d-v2 | 1614 (-41%) | -267 (+41%) | -1296 (-89%) | -53 (-4%)
Reacher-v2 | -5 (+13%) | -1 (-16%) | -208 (+43%) | -43 (-4%)
Swimmer-v2 | 130 (+181%) | -49 (-874%) | -172 (+86%) | -44 (-4%)
TABLE VI: Performance of the TENGraD approximation across seven different
environments with a tuned batch size and where the critic network is optimized
with the natural gradient. The percentage improvement compared to the baseline
is in brackets. The largest improvement for each metric is in bold. We see
TENGraD achieves improvements in performance, stability, and sample efficiency
of up to +181%, +60%, and +86% respectively with at most a 5% decrease in the
speed score.

We first investigate each approximation’s baseline performance using the same
batch size and using SGD to optimize the critic network. The training curves
of the different approximations are shown in Figure 2. The solid-colored line
represents the mean value across the random seeds and the shaded region
represents one standard deviation above and below the mean. We also show the
mean metrics across the seven different environments in Table I.

Performance Since each environment has different reward scales, to weigh each
environment’s score equally, we normalize the performance scores in Table I
and Figure 3.

Figure 2 shows that no approximation outperforms all other approximations
across all environments and shows that the best approximation depends on the
environment. Table I shows TENGraD achieves the best performance score across
all approximations, followed by HF and EKFAC, followed by KFAC and lastly
Diagonal. Table I also shows that improved approximations can lead to improved
performance since TENGraD achieves the best mean performance score across all
environments and is the most accurate approximation. We also see that the
Diagonal achieves the lowest performance score and that in some environments
(Ant-v2 and Reacher-v2 shown in Figure 2) it diverges near the end of the
training; likely because it ignores a large amount of curvature leading to
unstable updates.

Figure 3 shows normalized performance scores across four aggregate metrics
with 95% stratified bootstrap confidence intervals from [1] including median,
interquartile mean (IQM), mean, and the optimality gap across all
approximations. A larger score in the median, IQM, and mean aggregations and a
smaller score in the optimality gap aggregation corresponds to better
performance.

We see TENGraD significantly outperforms the other approximations in terms of
the IQM, mean, and optimality gap aggregation metrics because it achieves the
best score and there is no overlap between its confidence intervals and the
others. EKFAC and HF achieve the second-best scores however, their confidence
intervals overlap which shows their performance is not significantly different
from each other. Lastly, we see KFAC achieves the third-best score, followed
by Diagonal with the worst score. Analyzing the median metrics in Figure 3 is
not as straightforward due to a large amount of overlap in confidence
intervals; this is not surprising since the median is a poor measure of
overall performance [1]. We see TENGraD is significantly better than all other
approximations other than EKFAC; and that EKFAC is significantly better than
Diagonal. Interestingly, the median aggregate metric shows a trend that newer
second-order approximations lead to better median performance.

Stability HF achieves the best stability score, closely followed by KFAC,
followed by TENGraD and EKFAC, and lastly, with a significantly lower score,
Diagonal. HF achieves the best stability score, likely because it utilizes the
conjugate gradient algorithm for its update. In contrast, Diagonal achieves
the lowest score, likely because its approximation ignores a large amount of
curvature leading to inaccurate gradient scaling. We see in the HalfCheetah-v2
and Reacher-v2 environments, Diagonal is unable to remain stable throughout
training and diverges at the end of training.

Sample Efficiency KFAC achieves the best sample efficiency score, closely
followed by EKFAC and TENGraD, followed by HF and Diagonal.

Computation Time Diagonal achieves the best computation time score, followed
by HF, followed by KFAC and EKFAC, and lastly TENGraD. Diagonal achieves the
best scores due to its simple and efficient implementation, only requiring an
element-wise multiplication and division. Whereas, TENGraD achieves the lowest
score due to it having to periodically compute the inverse of a very large
$m\times m$ matrix where $m$ is the batch size.

###  V-B Hyperparameters

We now investigate how we can leverage a performance-tuned batch size and use
the natural gradient to optimize the critic network to achieve performance
improvements across all approximations in Tables II \- VI. For succinctness,
we move the results of batch size tuning in Appendix A. In all the tables we
show the percentage change compared to the baseline model in brackets and bold
the largest improvement for each metric. If the policy is unable to achieve
the sample efficiency threshold compared to the baseline we show ‘NaN’.

Key Trends: Overall, we see tuning the batch size and using the natural
gradient to optimize the critic network can lead to significant improvements
in performance, training stability, sample efficiency, and speed in many
reinforcement learning environments across all five natural gradient
approximations. Specifically, we see TENGraD achieves improvements in
performance, stability, and sample efficiency of up to +181%, +60%, and +86%
respectively with at most a 5% decrease in the speed score. The results of
applying the strategies across all the other approximations are summarized in
their respective table captions.

Interestingly, we see that using the hyperparameters the metrics are usually
improved in the HalfCheetah-v2, Ant-v2, Reacher-v2, and Swimmer-v2
environments, however, the other environments such as Hopper-v2, Humanoid-v2,
and Walker2d-v2 see results decrease compared to the baseline. This may be
because the strategies are tuned on the HalfCheetah-v2 environment which is
much similar to the environments in which the policy performed well in and is
not similar to the environments in which the policy performed poorly. We
hypothesise that tuning the strategies on the poor performance environments
can result in similar performance improvements seen in the other environments.
We leave this for future research.

Another trend we see is that when the performance score is improved usually
the stability score decreases; and when the performance score is improved, we
see a similar improvement in the sample efficiency score.

##  VI Conclusion

In this paper, we studied five different second-order approximations across
multiple key metrics to better understand how each approximation affects the
performance of the natural policy gradient. Furthermore, we investigated the
effect of two different hyperparameters: the batch size and the optimization
method for the critic network, and showed they have a large effect on the
final performance, even though they aren’t typically acknowledged in the
literature.

We found that properly tuning the hyperparameters can lead to large
improvements in performance and sample efficiency ranging up to +181% and +86%
respectively across the MuJoCo control benchmarks and that TENGraD achieved
the best performance out of all the approximations.

We also find that there is a fundamental trade-off between achieving high
performance and maintaining stability throughout training, and that
performance is positively correlated with sample efficiency. We hope this
research helps expand our field’s understanding of how different
approximations and hyperparameters affects the natural policy gradient and
helps practitioners efficiently leverage natural policy gradient methods for
real-world reinforcement learning tasks.

## References

  * [1] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare.  Deep reinforcement learning at the edge of the statistical precipice.  Advances in Neural Information Processing Systems, 34, 2021.
  * [2] S. I. Amari.  Natural Gradient Works Efficiently in Learning.  Neural Computation, 10(2):251–276, 1998.
  * [3] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, L. Rudolph, and A. Madry.  Implementation matters in deep policy gradients: A case study on ppo and trpo.  arXiv preprint arXiv:2005.12729, 2020.
  * [4] B. Gebotys.  Natural gradient descent without the tears.  https://gebob19.github.io/natural-gradient/, 2021.
  * [5] T. George, C. Laurent, X. Bouthillier, N. Ballas, and P. Vincent.  Fast approximate natural gradient descent in a kronecker-factored eigenbasis.  Advances in Neural Information Processing Systems, 2018-Decem(NeurIPS):9550–9560, 2018.
  * [6] R. Grosse.  Neural net training dynamics, winter 2021: Chapter 3: Metrics.  https://www.cs.toronto.edu/~rgrosse/courses/csc2541_2021/readings/L03_metrics.pdf, 2021\.
  * [7] S. Gu, T. Lillicrap, Z. Ghahramani, R. E. Turner, and S. Levine.  Q-prop: Sample-efficient policy gradient with an off-policy critic.  arXiv preprint arXiv:1611.02247, 2016.
  * [8] K. Kielak.  Importance of using appropriate baselines for evaluation of data-efficiency in deep reinforcement learning for atari.  arXiv preprint arXiv:2003.10181, 2020.
  * [9] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [10] I. Kostrikov.  pytorch-trpo.  https://github.com/ikostrikov/pytorch-trpo, 2018.
  * [11] Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller.  Efficient backprop.  In Neural networks: Tricks of the trade, pages 9–48. Springer, 2012\.
  * [12] J. Martens.  Deep learning via Hessian-free optimization.  ICML 2010 - Proceedings, 27th International Conference on Machine Learning, pages 735–742, 2010.
  * [13] J. Martens and R. Grosse.  Optimizing neural networks with Kronecker-factored approximate curvature.  32nd International Conference on Machine Learning, ICML 2015, 3:2398–2407, 2015.
  * [14] S. Mohamed, M. Rosca, M. Figurnov, and A. Mnih.  Monte carlo gradient estimation in machine learning.  J. Mach. Learn. Res., 21(132):1–62, 2020.
  * [15] J. Schulman, S. Levine, P. Moritz, M. Jordan, and P. Abbeel.  Trust region policy optimization.  32nd International Conference on Machine Learning, ICML 2015, 3:1889–1897, 2015.
  * [16] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel.  High-dimensional continuous control using generalized advantage estimation.  arXiv preprint arXiv:1506.02438, 2015.
  * [17] S. Soori, B. Can, B. Mu, M. Gürbüzbalaban, and M. M. Dehnavi.  Tengrad: Time-efficient natural gradient descent with exact fisher-block inversion.  arXiv preprint arXiv:2106.03947, 2021.
  * [18] R. S. Sutton, D. A. McAllester, S. P. Singh, and Y. Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In Advances in neural information processing systems, pages 1057–1063, 2000.
  * [19] Y. Wu, E. Mansimov, S. Liao, R. Grosse, and J. Ba.  Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation.  Advances in Neural Information Processing Systems, 2017-Decem:5280–5289, 2017.

##  Appendix A Tuned Batch Size Results

We show the results of tuning the batch size across all approximations in
Tables VII \- X. We found KFAC’s tuned batch size was the same as its baseline
batch size so we don’t include its results. We see large improvements in
performance, stability, sample efficiency, and speed across all approximations
showing that tuning the batch size is a key strategy for achieving the best
performance.

##  Appendix B More Details on the Metrics

For the sample efficiency metric, the performance thresholds are computed for
each environment as the lowest performance score across all approximations in
the baseline experiments. The thresholds are then held constant throughout the
rest of the experiments. For clarity purposes, the threshold scores are
divided by 1000. The exact thresholds used are made available in our code
repository.

In Table I, since each environment has different reward scales, to weigh each
environment’s score equally, we normalize the performance scores of each
environment by dividing by the highest score achieved in that environment
across all approximations and multiplying it by 100, making the largest
achievable reward 100. We don’t scale the values by 100 for Figure 3.

##  Appendix C Hyperparameters

###  C-A Baseline

For the baseline model, we tuned the following hyperparameters for one million
environment steps: damping parameter with values (1e-1 1e-2), critic learning
rate with values (1e-2 1e-3), whether to use a backtracking line search or not
based on [15], when not using line search we use step-size clipping used in
[19] and tune the maximum learning rate with values (1e-1 1e-2). All
approximations used a batch size of 15K except for TENGraD which used a batch
size of 6K because our GPUs didn’t have enough memory for a batch size of 15K.
We chose 15K because it is commonly used in the literature [15, 19].

Although TENGraD uses a smaller batch size which may be seen as unfair due to
the increased number of optimization steps, comparing all approximations with
their tuned batch sizes (with available batch sizes smaller than TENGraD’s
batch sizes), TENGraD still achieves the highest performance score.

All hyperparameters used in all the experiments can be found in our code
repository.

###  C-B Batch Size Tuning

For the batch size model, we tuned the batch size for five million environment
steps with values (25 20 15 10 5) except for TENGraD, because of limited GPU
memory, which we searched over values (10 8 6 4). EKFAC’s performance on
HalfCheetah-v2 (which it was tuned on) decreased with the tuned batch size so
we used the baseline batch size when applying the natural critic optimization.

###  C-C Natural Critic Optimization

For the natural critic optimization model, we applied a grid search over the
following hyperparameters for five million environment steps: natural gradient
approximation with values (Diagonal HF KFAC EKFAC TENGraD), critic damping
parameter with values (1e-1 1e-2), whether to use a backtracking line search
or not based on [15], when not using line search we use step-size clipping
used in [19] and tune the maximum learning rate with values (1e-1 1e-2).

##  Appendix D Model Architectures

The policy and critic models consisted of three linear layers with 64 hidden
units each using the tanh activation function, except for the final output.
Following standard practice, all layers were initialized with orthogonal
initialization.

##  Appendix E Code

For fisher approximation implementations we implemented the diagonal
ourselves, we used [10] as a reference when implementing HF. we used the code
provided by EKFAC [5] for EKFAC and KFAC, and we used the author’s provided
code for the implementation of TENGraD.

Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 1371 (+36%) | -257 (+51%) | -2250 (-22%) | -22 (+18%)
Ant-v2 | 668 (+57%) | -100 (+100%) | -1575 (-38%) | -43 (+17%)
Hopper-v2 | 1500 (+29%) | -330 (+10%) | -1125 (+5%) | -27 (+17%)
Humanoid-v2 | 749 (+3%) | -42 (+42%) | -4775 (-9%) | -50 (+19%)
Walker2d-v2 | 1802 (+0%) | -210 (+40%) | -1275 (-89%) | -28 (+18%)
Reacher-v2 | -12 (+8%) | -1912 (+48%) | -775 (-67%) | -21 (+18%)
Swimmer-v2 | 36 (-2%) | -4 (+18%) | -4025 (-19%) | -22 (+18%)
TABLE VII: Performance of the Diagonal approximation across seven different environments with a tuned batch size. The percentage improvement compared to the baseline is in brackets. The largest improvement for each metric is in bold. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 1338 (+17%) | -171 (-314%) | -1900 (+35%) | -28 (-5%)
Ant-v2 | 1123 (+18%) | -92 (-85%) | -2630 (+17%) | -54 (-3%)
Hopper-v2 | 2358 (+3%) | -340 (-14%) | -510 (+17%) | -35 (-4%)
Humanoid-v2 | 991 (+21%) | -47 (-42%) | -3680 (+16%) | -63 (-3%)
Walker2d-v2 | 3065 (+8%) | -298 (+7%) | -690 (+6%) | -37 (-5%)
Reacher-v2 | -5 (+1%) | -0 (+24%) | -360 (+25%) | -28 (-7%)
Swimmer-v2 | 41 (-1%) | -5 (-49%) | -1200 (+5%) | -29 (-10%)
TABLE VIII: Performance of the HF approximation across seven different environments with a tuned batch size. The percentage improvement compared to the baseline is in brackets. The largest improvement for each metric is in bold. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 2200 (-3%) | -460 (+11%) | -430 (+25%) | -28 (-5%)
Ant-v2 | 1725 (+10%) | -346 (-18%) | -400 (+33%) | -54 (-4%)
Hopper-v2 | 1097 (-43%) | -457 (+5%) | NaN | -34 (+4%)
Humanoid-v2 | 788 (+8%) | -80 (-148%) | -3940 (+17%) | -65 (-4%)
Walker2d-v2 | 1337 (-7%) | -519 (-66%) | -400 (+33%) | -36 (-3%)
Reacher-v2 | -8 (-14%) | -1 (+17%) | -370 (+37%) | -27 (-4%)
Swimmer-v2 | 41 (-1%) | -7 (+1%) | -2430 (-125%) | -29 (-8%)
TABLE IX: Performance of the EKFAC approximation across seven different environments with a tuned batch size. The percentage improvement compared to the baseline is in brackets. The largest improvement for each metric is in bold. Environment | Performance | Stability | Sample Efficiency | Speed
---|---|---|---|---
HalfCheetah-v2 | 1943 (+8%) | -371 (-6%) | -1220 (+25%) | -42 (-1%)
Ant-v2 | 1998 (+10%) | -236 (-28%) | -1356 (+14%) | -69 (-1%)
Hopper-v2 | 1870 (+7%) | -318 (-40%) | -340 (+11%) | -49 (-1%)
Humanoid-v2 | 972 (+13%) | -49 (+6%) | -3728 (+11%) | -79 (-2%)
Walker2d-v2 | 2788 (+2%) | -576 (-27%) | -496 (+27%) | -51 (-0%)
Reacher-v2 | -6 (-14%) | -1 (-63%) | -268 (+27%) | -42 (-0%)
Swimmer-v2 | 45 (-3%) | -6 (-24%) | -452 (+64%) | -43 (-1%)
TABLE X: Performance of the TENGraD approximation across seven different
environments with a tuned batch size. The percentage improvement compared to
the baseline is in brackets. The largest improvement for each metric is in
bold.


# The evaporation of concentrated polymer solutions is insensitive to relative
humidity

Max Huisman  SUPA and School of Physics and Astronomy, The University of
Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United Kingdom  Paul
Digard  Department of Infection and Immunity, The Roslin Institute, The
University of Edinburgh, Easter Bush Campus, Edinburgh EH25 9RG, United
Kingdom  Wilson C. K. Poon  SUPA and School of Physics and Astronomy, The
University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United
Kingdom  Simon Titmuss  SUPA and School of Physics and Astronomy, The
University of Edinburgh, Peter Guthrie Tait Road, Edinburgh EH9 3FD, United
Kingdom

###### Abstract

A recent theory suggests that the evaporation kinetics of macromolecular
solutions is insensitive to the ambient relative humidity (RH) due to the
formation of a ‘polarisation layer’ of solutes at the air-solution interface.
We confirm this insensitivity up to RH $\approx 80\%$ in the evaporation of
polyvinyl alcohol solutions from open-ended capillaries. To explain the
observed drop in evaporation rate at higher RH, we need to invoke compressive
stresses due to interfacial polymer gelation. Moreover, RH-insensitive
evaporation sets in earlier than theory predicts, suggesting a further role
for a gelled ‘skin’. We discuss the relevance of these observations for
respiratory virus transmission via aerosols.

††preprint: APS/123-QED

‘Hindered evaporation’ from a water-air interface through some barrier is
ubiquitous in applications. For example, as paint or ink dries, a ‘skin’ may
form rapidly at the interface [1]. This affects the ‘open time’ during which a
second coat may be applied without imperfections of the first coat showing
through [2]. The skin may buckle [3] or give rise to bubbles [4] and diminish
coating quality. Such applications have motivated a number of fundamental
studies [5, 6, 7, 4, 8].

Hindered evaporation is also important for understanding biological response
to environmental relative humidity, RH = $a_{e}\times 100\%$, where $a_{e}$ is
the water activity in the air. The evaporation rate of water through the inert
wax cuticle of a leaf is proportional to $(1-a_{e})$, but is humidity
independent in human skin for RH $\lesssim 85\%$ [9]. It is suggested that the
phase behavior of the lipid mixture in the stratum corneum, the outer layer of
skin, turns it into an active barrier [10] that responds to changes in $a_{e}$
to maintain ‘evaporative homeostasis’. The observation of liquid crystallinity
at the water-air interface of lipid solutions displaying RH-independent
evaporation rate up to RH $\approx 80\%$ [9] supports this picture.

A recent theory [11] suggests that RH-independent evaporation does not require
such ‘active’ response, but occurs whenever a concentrated ‘polarisation
layer’ of solutes builds up at the water-air interface due to the advective
flux towards the interface driven by evaporation balanced by an opposite
diffusive flux. The evaporation rate is primarily controlled by
$a(\varphi_{i})$, the water activity at the interface with solute volume
fraction $\varphi_{i}$. If $a(\varphi)$ drops sharply enough at high
$\varphi$, the evaporation rate becomes RH insensitive. This effect is
enhanced if the mutual diffusivity of the solution $D(\varphi)$ has a similar
dependence on $\varphi$. Since such $a(\varphi)$ and $D(\varphi)$ occur widely
in macromolecular solutions, RH-insensitive evaporation should be generic even
without ‘active’ polarisation layers. Using measured $a(\varphi)$ and
$D(\varphi)$, Salmon et al. predict RH-independent evaporation of polyvinyl
alcohol (PVA) solutions up to near saturation (RH $\to 100\%$).

There are multiple motivations for verifying this theory. Fundamentally, it is
important to know whether there is indeed a generic mechanism for RH-
insensitive evaporation, and, if so, what active response [10] may add.
Applications to coatings will also benefit from a verified predictive theory
for the effect of RH.

More topically, the respiratory droplets that transmit SARS-CoV2 and other
pathogens contain a mixture of salt, lipids and glycoproteins (mucins), the
latter at high concentrations deep inside the lungs [12]. Their evaporative
kinetics controls air-borne transmission [13, 14, 15, 16, 17]. Previous
empirical studies suggest an intriguing non-monotonic dependence of viral
viability on RH [18]. A meta-analysis correlates SARS-CoV2 infectivity with RH
but provides no clear picture [19]. As evaporative kinetics probably plays a
role in explaining seasonal variability of respiratory virus transmission [20,
21], recent studies have considered macromolecular effects on the RH
dependence of droplet drying and virus viability [22, 23, 24].

Figure 1:  (a) Experimental schematic showing one of the five
$50\text{\,}\mathrm{mm}$ capillaries emerging from a $30\text{\,}\mathrm{mm}$
section of $20\text{\,}\mathrm{mm}$ diameter perspex tubing glued to a
microscope slide. The 5 capillaries increase evaporation rate, thus allowing
faster experiments. (b) Mass loss, $m(t)$, of pure water (H2O, open diamonds)
and H2O + PVA at $\varphi=0.008$ (filled circles) solutions at RH $=50$% and
$T=$291\text{\,}\mathrm{K}$$. Inset: time exponents $\beta$ of a power-law fit
to the late-time $m(t)$ for PVA solutions at different RH. (c) PVA $m(t)$ at
different RH plotted against time $t$. Inset PVA $m(t)$ versus $t^{1/2}$.
Dashed lines are linear fits at long times, after a transient period indicated
by the dotted line where $t>300~{}$min or $t^{1/2}>15~{}$min1/2. (d)
$\blacktriangle$: the mass loss rate of pure water, $\alpha_{0}=\dot{m}(t)$,
normalised to the value at RH = 25%; $\triangle$: the same quantity for 0.9%
(w/w) NaCl solutions; $\bullet$: the slope of the linear portions in the inset
of part (c) for $\varphi=0.008$ PVA solutions, ${\rm d}m(t)/{\rm
d}t^{1/2}=(2\sqrt{D_{0}}\rho A)\alpha$, normalised to the value at RH = 25%.

Motivated by these reasons, we have tested experimentally the theory of Salmon
et al. for RH-insensitive hindered evaporation [11]. One of the geometries
they treated, unidirectional drying from one end of a capillary whose other
end is connected to a constant solute concentration reservoir, was previously
used to study the drying of lipidic [9, 25, 26] and saliva-containing
solutions [23]. They made testable predictions for PVA solutions up to a
single numerical prefactor.

Based on a previous set-up [9, 25, 27], Fig. 1(a), we connected 5 rectangular
borosilicate capillaries
($$0.20\text{\,}\mathrm{mm}$\times$4\text{\,}\mathrm{mm}$$, VitroTubes) to a
liquid reservoir. To ensure evaporation only at the open end of each
capillary, we covered the liquid in the reservoir with a thin layer of
1-Octadecene (Sigma Aldrich). The set-up rested on a Sartorious Secura 224-1S
high-precision scale to quantify evaporative mass loss. A sealed enclosure
over the sample connected to a Cellkraft P-10A humidifier controlled the RH
and temperature $T$, with set-points confirmed using external probes. This
obviates the need for blowing constant-RH air at the interface [9], which may
perturb drying. The humidifier airflow was tuned to minimize disturbance on
mass measurements.

A drop shape analyser (Kruss EasyDrop DSA20E) observes the side view at the
open end of the capillary. We adjusted the reservoir level to obtain an
initial flat water-air interface. The degree of flatness did not visibly
change over an experiment, thus minimising curvature effects [28]. The mass
loss rate of water should then follow

$\dot{m}(t)=kAc_{\rm{sat}}(1-a_{\rm{e}})\stackrel{{\scriptstyle\rm
def}}{{=}}\alpha_{0},$ (1)

with $A$ the capillary’s cross section, and $k$ and $c_{\rm{sat}}$ the mass
transfer coefficient and the saturation concentration of water in air [29].
Indeed, $m(t)$ is linear at RH = 50%, Fig. 1(b), and $\alpha_{0}$ is strictly
proportional to $(1-a_{\rm e})$ [30].

At early times, the evaporation of a polymer solution with volume fraction
$\varphi\ll 1$ should follow Eq. 1. Once a significant polarisation layer
forms, the reduced interfacial water activity, $a(\varphi_{i})$, controls the
evaporation rate, which now reads
$\dot{m}(t)=kAc_{\rm{sat}}(a(\varphi_{i})-a_{\rm{e}})$. As the polarisation
layer grows, $\varphi_{i}$ increases asymptotically towards $\varphi^{*}$
given by $a(\varphi^{*})=a_{\rm e}$, and the evaporation rate becomes sub-
linear as $a(\varphi_{i})\to a(\varphi^{*})$. Solving the case for constant
$D(\varphi)=D_{0}$, the single-coil diffusivity, to highlight the role of
water activity, Salmon et al. predict

$\dot{m}(t)=f\rho
A\sqrt{\frac{\varphi^{*}D_{0}}{2\varphi_{0}t}}\stackrel{{\scriptstyle\rm
def}}{{=}}\alpha\rho A\sqrt{\frac{D_{0}}{t}},$ (2)

where $f$ is a numerical constant of order unity,
$\rho\approx$998\text{\,}\mathrm{k}\mathrm{g}\mathrm{m}^{-3}$$ is the mass
density of the water and $\varphi_{0}$ is the reservoir concentration. At
$298\text{\,}\mathrm{K}$, 99% hydrolyzed PVA with molecular weight
$M_{\rm{W}}=$ 85k-124k has a hydrodynamic radius
$R_{\rm{H}}=10.5\pm$0.5\text{\,}\mathrm{nm}$$ [31]. For our PVA with about 50%
higher $M_{\rm w}$, $R_{\rm{H}}\approx\sqrt{1.5}\times 10.5~{}\rm{nm}\approx
13~{}\rm{nm}$, from which we estimate
$D_{0}\approx$16\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}^{2}\mathrm{s}^{-1}$$.
Equation 2 also defines a dimensionless $\alpha\propto{\rm d}m/{\rm d}t^{1/2}$
(cf. Fig. 1(c) inset) to facilitate comparison with data,

$\alpha=(2\rho A\sqrt{D_{0}})^{-1}\frac{{\rm d}m}{{\rm
d}t^{1/2}}=f\sqrt{\frac{\varphi^{*}}{2\varphi_{0}}}.$ (3)

We dissolved PVA ($M_{\rm w}=$ 146k-186k, 99% hydrolized, Sigma Aldrich) in
milli-Q water. Figure 1(b) shows $m(t)$ for $\varphi=0.008$ ($=1\%$ (w/w)111We
use a mass density of PVA, $\rho_{\mathrm{PVA}}=1250\,\mathrm{kgm^{-3}}$
correpsonding to the middle of the 1190-1310 $\mathrm{kgm^{-3}}$ range given
at https://pubchem.ncbi.nlm.nih.gov/compound/11199 ) at
$T=$291\text{\,}\mathrm{K}$$ and RH = 50%. An initially linear regime of
constant evaporation rate becomes sub-linear as time progresses. For each RH
studied, Fig. 1(c), the sub-linear ‘falling rate’ regime is consistent with a
$t^{1/2}$ scaling (inset) [11].

Figure 2:  Comparison of the measured evaporation prefactor $\alpha$ as a
function of RH ( $\bullet$, with error bars giving the standard deviation)
with theoretical predictions [11] for constant polymer diffusivity and a water
activity $a(\varphi)$ without elasticity, Eq. 4 (dashed line), and with
elasticity, Eq. 6 (full line).

To highlight the near-RH independence up to 80% and contrast this with the
behavior of pure water, Fig. 1(d) plots $\alpha_{0}/\alpha_{0}^{(25)}$ and
$\alpha/\alpha^{(25)}$ against RH, where the denominators are the respective
values at RH = 25% 222While $\alpha^{\prime}=\dot{m}(t)$ is the evaporation
rate for H2O, it is incorrect to call $\alpha\propto{\rm dm}/{\rm d}t^{1/2}$
an ‘evaporation rate’ for PVA (or lipid [9]) solutions: see Eq. 2.. This
difference is not seen for 0.9% (w/w) NaCl solutions, Fig. 1(d) [$\triangle$]:
the macromolecular nature of PVA is key. Moreover, no ordering is needed:
crossed-polar observation of the open end of capillaries with evaporating PVA
solution showed no liquid crystallinity.

To test the theory further, consider the absolute value of the evaporative
prefactor, $\alpha$, in the falling rate regime. Experimental values
calculated using measured ${\rm d}m/{\rm d}t^{1/2}$ and known $(\rho,A,D_{0})$
as input, Eq. 3, are shown in Fig. 2 ( $\bullet$). To compare with theory, we
solved $a(\varphi^{*})=a_{e}$ to give $\varphi^{*}=a^{-1}(a_{e})$ using a
parameterisation of the experimentally measured $a(\varphi)$ [34, 11]

$a(\varphi)=(1-\varphi)\exp{[\varphi+\chi\varphi^{2}]},$ (4)

with the Flory-Huggins parameter

$\chi(\varphi)=3.94-3.42(1-\varphi)^{0.09}.$ (5)

This ‘cliff-like’ $a(\varphi)$, Eq. 4, means $\varphi^{*}=a^{-1}(a_{e})$
varies little for $0<\rm RH\lesssim 95\%$, engendering even lower RH-
dependence for $\alpha\propto\sqrt{\varphi^{*}}$, Eq. 3. Our predicted
$\varphi^{*}$ and a prefactor of $f=1.12$, Eq. 3, account quantitatively for
the observed $\alpha(\rm RH)$ up to RH $\approx 80\%$, Fig. 2 (dashed line);
discrepancies at higher RH require explanation.

Figure 3:  Bright-field image of the polarization layer at the water-air
interface at the end ($t\sim 10^{3}~{}$min) of a typical experiment at
RH$=50\%$.

Bright-field imaging of the polarisation layer, Fig. 3, shows late-stage
delamination consistent with quasi-periodic buckling, allowing air ingress to
give bright-field contrast. Such buckling is typical of a stiff film atop a
compliant substrate when the film is under considerable compressive stress
[35]. In our case, we have a stiff gel skin [36, 37, 1] covering a more
compliant, viscoelastic polarisation layer. The resulting stress contributes
to the osmotic pressure [38] and modifies Eq. 4 to

$a(\varphi)=(1-\varphi)\exp{\left[\varphi+\chi\varphi^{2}-\frac{K_{\rm{g}}\nu_{1}}{k_{\rm{B}}T}\ln{\left(\frac{\varphi}{\varphi_{g}}\right)}\right]},$
(6)

with $\nu_{1}$ the molecular volume of water and $K_{\rm g}$ the osmotic
modulus of the gelled skin [30]. Equation 6 well fits our data up to RH
$\approx 90\%$, Fig. 2 (solid line), with
$K_{\rm{g}}=10\pm$1\text{\,}\mathrm{MPa}$$ and $\varphi_{g}=0.24\pm 0.02$ as
fit parameters.

As already noted, the apparently linear late time
($t\gtrsim$300\text{\,}\mathrm{min}$$) data in Fig. 1(c) [inset] seems
consistent with $m(t)\sim t^{0.5}$. Fitting this data to $m(t)\sim t^{\beta}$
gives $0.5<\beta<0.6$, with no systematic dependence on RH, Fig. 1(b) [inset]:
evaporation is always somewhat faster than theory [11] predicts, probably due
to the parallel pathways provided by air ingress following the buckling
delamination of the polymer skin from the glass wall, giving higher
permeability than an intact polarisation layer.

Figure 4:  The dependence of evaporation rate on time plotted on log-log
scales. Dotted lines represent for early times the the dilute evaporation
regime, Eq. 1, and for long times the diffusion limited evaporation regime,
Eq. 2 (with $f=1$), the transition point is taken as the intercept.

For a final test of theory, we show $\dot{m}(t)$ in Fig. 4. Salmon et al. [11]
predict that, before a polarisation layer is established, $\dot{m}(t)$ should
be constant, $\sim t^{0}$, and essentially that of pure water and decreases
linearly with RH, Eq. 1. Including the nearly-universal $t^{-1/2}$ scaling,
Eq. 2 with $f=1$, on the same plot defines a critical time, $t_{c}$, for the
transition between these two regimes. Measured $\dot{m}(t)$ (full lines) and
theory (dashed lines) disagree on two accounts. First, rather than a
systematic decrease with RH, the constant $\dot{m}$ at short times appears to
vary randomly with RH. This scatter is comparable to the reproducibility
between runs at a single RH, suggesting that $\dot{m}$ may be RH independent
at early times. Secondly, the observed cross-over from $t^{0}$ to $t^{-1/2}$
scaling occurs at a time that is randomly scattered about
$t\lesssim${10}^{4}\text{\,}\mathrm{s}$$, while the theoretical ‘knee’
systematically shifts to longer times as RH increases.

These discrepancies motivate the search for extra physics beyond Salmon et al.
[11], which may again relate to a gelled polymer skin at the air-solution
interface. Its formation, in $\lesssim$1\text{\,}\mathrm{s}$$ in our case
[30], gives rise to an interfacial concentration that remains close to
$\varphi_{g}$ for some time [37, 36, 39]. In this ‘skin-limited’ period, the
water flux into the skin from the solution is set by the pressure gradient at
the skin [40], and therefore by $\varphi=\varphi_{g}$, rather than
$\varphi^{*}=a^{-1}(a_{e})$ [11]. This predicts RH-independent, constant
evaporation rate at early times, consistent with the observed random scatter
of $\dot{m}$ values.

The rate at which the interface concentration increases from $\varphi_{g}$ to
$\varphi^{*}$ is set by an RH-independent skin-limited evaporation rate;
moreover, we have seen that $\varphi^{*}$ is RH insensitive. So, the duration
of the skin-limited period, $t_{c}$, is also insensitive to RH, as observed.

In summary, a PVA-water mixture is found to evaporate almost independently of
the ambient RH during early constant evaporation rate and late falling-rate
regimes. The latter agrees with a theoretical model [11] predicated on the
formation of a polymer polarization layer at the air-water interface driven by
advection. We find evidence that the interfacial gelation of the PVA
contributes to the early-stage RH-insensitivity. These findings may prove
significant for the evaporation of pathogen-containing respiratory droplets,
which include surfactants and gel-forming macromolecules that have been shown
to encapsulate dried droplets [41, 18]. Our tractable experimental system
provides a baseline for investigating the role of these more complex solutes
in viral transmission via respiratory droplets [42].

We thank Paul Harris and Andrew Schofield for technical assistance and Jean-
Baptiste Salmon, Jay Marsden, Veronica McKinny, Davide Marenduzzo and Patrick
Warren for insightful discussions. Aidan Brown lent us a humidifier purchased
under EPSRC EP/S001255/1. MH was funded by the University of Edinburgh.

For the purpose of open access, the author has applied a Creative Commons
Attribution (CC BY) licence to any Author Accepted Manuscript version arising
from this submission.

## References

  * Tirumkudulu and Punati [2022] M. S. Tirumkudulu and V. S. Punati, Langmuir 38, 2409 (2022).
  * Holl  _et al._ [2001] Y. Holl, J. L. Keddie, P. J. McDonald, and W. A. Winnik, Drying modes of polymer colloids, in  _Film Formation in Coatings_ (American Chemical Society, 2001) Chap. 1, pp. 2–26.
  * Pauchard and Allain [2003] L. Pauchard and C. Allain, Europhys. Lett. (EPL) 62, 897 (2003).
  * Arai and Doi [2012] S. Arai and M. Doi, Eur. Phys. J. E 35, 57 (2012).
  * Bornside  _et al._ [1989] D. E. Bornside, C. W. Macosko, and L. E. Scriven, J. Appl. Phys. 66, 5185 (1989).
  * de Gennes [2002] P. G. de Gennes, Eur. Phys. J. E 7, 31 (2002).
  * Okuzono  _et al._ [2006a] T. Okuzono, K. Ozawa, and M. Doi, Phys. Rev. Lett. 97, 136103 (2006a).
  * Raju  _et al._ [2022] L. T. Raju, C. Diddens, J. Rodríguez-Rodríguez, M. N. van der Linden, X. Zhang, D. Lohse, and U. Sen,  (2022), arXiv:2211.06528 .
  * Roger  _et al._ [2016] K. Roger, M. Liebi, J. Heimdal, Q. D. Pham, and E. Sparr, Proc. Natl. Acad. Sci. (USA) 113, 10275 (2016).
  * Sparr and Wennerström [2001] E. Sparr and H. Wennerström, Biophys. J. 81, 1014 (2001).
  * Salmon  _et al._ [2017] J. B. Salmon, F. Doumenc, and B. Guerrier, Phys. Rev. E 96, 032612 (2017).
  * Nicas  _et al._ [2005] M. Nicas, W. W. Nazaroff, and A. Hubbard, J. Occup. Environ. Hyg. 2, 143 (2005).
  * Wells [1934] W. F. Wells, Am. J. Epidemiol. 20, 611 (1934).
  * Xie  _et al._ [2007] X. Xie, Y. Li, A. T. Chwang, P. L. Ho, and W. H. Seto, Indoor Air 17, 211 (2007).
  * Netz [2020] R. R. Netz, J. Phys. Chem. B 124, 7093 (2020).
  * Rezaei and Netz [2021] M. Rezaei and R. R. Netz, Phy. Fluids 33, 091901 (2021).
  * Dbouk and Drikakis [2020] T. Dbouk and D. Drikakis, Physics of Fluids 32, 093312 (2020).
  * Huynh  _et al._ [2022] E. Huynh, A. Olinger, D. Woolley, R. K. Kohli, J. M. Choczynski, J. F. Davies, K. Lin, L. C. Marr, and R. D. Davis, Proc. Natl. Acad. Sci. (USA) 119, e2109750119 (2022).
  * Mecenas  _et al._ [2020] P. Mecenas, R. T. d. R. M. Bastos, A. C. R. Vallinoto, and D. Normando, PLOS ONE 15, 1 (2020).
  * Ali  _et al._ [2022] S. T. Ali, B. J. Cowling, J. Y. Wong, D. Chen, S. Shan, E. H. Lau, D. He, L. Tian, Z. Li, and P. Wu, Sci. Total Environ. 818, 151724 (2022).
  * Azziz Baumgartner  _et al._ [2012] E. Azziz Baumgartner, C. N. Dao, S. Nasreen, M. U. Bhuiyan, S. Mah-E-Muneer, A. A. Mamun, M. A. Y. Sharker, R. U. Zaman, P.-Y. Cheng, A. I. Klimov, M.-A. Widdowson, T. M. Uyeki, S. P. Luby, A. Mounts, and J. Bresee, J. Infect. Dis. 206, 838 (2012).
  * Poon  _et al._ [2020] W. C. K. Poon, A. T. Brown, S. O. L. Direito, D. J. M. Hodgson, L. Le Nagard, A. Lips, C. E. MacPhee, D. Marenduzzo, J. R. Royer, A. F. Silva, J. H. J. Thijssen, and S. Titmuss, Soft Matter 16, 8310 (2020).
  * Merhi  _et al._ [2022] T. Merhi, O. Atasi, C. Coetsier, B. Lalanne, and K. Roger, Proc. Natl. Acad. Sci. (USA) 119, e2204593119 (2022).
  * Pease  _et al._ [2022] L. F. Pease, N. Wang, G. R. Kulkarni, J. E. Flaherty, and C. A. Burns, Int. Commun. Heat Mass Transf. 131, 105746 (2022).
  * Roger  _et al._ [2018] K. Roger, E. Sparr, and H. Wennerström, Phys. Chem. Chem. Phys. 20, 10430 (2018).
  * Roger and Crassous [2021] K. Roger and J. J. Crassous, Proc. Natl. Acad. Sci. (USA) 118, e2105530118 (2021).
  * Andersson  _et al._ [2018] J. M. Andersson, K. Roger, M. Larsson, and E. Sparr, ACS Cent. Sci. 4, 1315 (2018).
  * Thomson [1872] W. Thomson, Proc. R. Soc. Edin. 7, 63–68 (1872).
  * Cussler [1997] E. Cussler,  _Diffusion: Mass Transfer in Fluid Systems_ , Cambridge Series in Chemical Engineering (Cambridge University Press, 1997).
  * [30] See electronic supplement at http://XXX.
  * Perfetti  _et al._ [2020] M. Perfetti, N. Gallucci, I. R. Krauss, A. Radulescu, S. Pasini, O. Holderer, G. D’Errico, G. Vitiello, G. O. Bianchetti, and L. Paduano, Macromolecules 53, 852 (2020).
  * Note [1] We use a mass density of PVA, $\rho_{\mathrm{PVA}}=1250\,\mathrm{kgm^{-3}}$ corresponding to the middle of the 1190-1310 $\mathrm{kgm^{-3}}$ range given at https://pubchem.ncbi.nlm.nih.gov/compound/11199.
  * Note [2] While $\alpha^{\prime}=\dot{m}(t)$ is the evaporation rate for H2O, it is incorrect to call $\alpha\propto{\rm dm}/{\rm d}t^{1/2}$ an ‘evaporation rate’ for PVA (or lipid [9]) solutions: see Eq. 2.
  * Jeck  _et al._ [2011] S. Jeck, P. Scharfer, W. Schabel, and M. Kind, Chem. Eng. Process. 50, 543 (2011).
  * Li  _et al._ [2012] B. Li, Y.-P. Cao, X.-Q. Feng, and H. Gao, Soft Matter 8, 5728 (2012).
  * Okuzono  _et al._ [2006b] T. Okuzono, K. Ozawa, and M. Doi, Phys. Rev. Lett. 97, 136103 (2006b).
  * Ozawa  _et al._ [2006] K. Ozawa, T. Okuzono, and M. Doi, Jpn. J. Appl. Phys. 45, 8817 (2006).
  * Leibler and Sekimotot [1993] L. Leibler and K. Sekimotot, Macromolecules 26, 6937 (1993).
  * Okuzono and Doi [2008] T. Okuzono and M. Doi, Phys. Rev. E 77, 030501(R) (2008).
  * Punati and Tirumkudulu [2022] V. S. Punati and M. S. Tirumkudulu, Soft Matter 18, 214 (2022).
  * Vejerano and Marr [2018] E. P. Vejerano and L. C. Marr, J. Royal Soc. Interface 15, 1 (2018).
  * Zanin  _et al._ [2016] M. Zanin, P. Baviskar, R. Webster, and R. Webby, Cell Host & Microbe 19, 159 (2016).