# Off-Policy Policy Gradient Algorithms
by Constraining the State Distribution Shift

Riashat Islam1
&Komal K. Teru1
&Deepak Sharma1††footnotemark:
&Joelle Pineau1,2
Equal contribution

###### Abstract

Off-policy deep reinforcement learning (RL) algorithms are incapable of
learning solely from batch offline data without online interactions with the
environment, due to the phenomenon known as extrapolation error. This is often
due to past data available in the replay buffer that may be quite different
from the data distribution under the current policy. We argue that most off-
policy learning methods fundamentally suffer from a state distribution shift
due to the mismatch between the state visitation distribution of the data
collected by the behaviour and target policies. This data distribution shift
between current and past samples can significantly impact performance of most
modern off-policy based policy optimization algorithms. In this work, we first
do a systematic analysis of state distribution mismatch in off-policy
learning, and then develop a novel off-policy policy optimization method to
constraint the state distribution shift. To do this, we first estimate the
state distribution based on features of the state, using a density estimator
and then develop a novel constrained off-policy gradient objective that
minimizes the state distribution shift. Our experimental results on continuous
control tasks show that minimizing this distribution mismatch can
significantly improve performance in most popular practical off-policy policy
gradient algorithms.

##  1 Introduction

Most off-policy deep RL algorithms that seek to train agents from a fixed
batch setting are forced to rely on online sample collection (using a
behaviour policy as an instance of the current policy, e.g with noise added to
action space) since the data under the behaviour policy can often be
uncorrelated with the data that would have been collected under the current
policy. Due to this, most deep RL algorithms cannot rely on purely offline
batch data. This is fundamentally due to the data distribution shift leading
to an extrapolation error as coined by Fujimoto et al. (2018a).

In this work, we propose a constrained policy optimization method based on
minimizing the state distribution shift between the target and behaviour
policy state distribution. Policy gradient methods such as the Deep
Deterministic Policy Gradients (DDPG) (Lillicrap et al., 2015), Twin Delayed
Deterministic Policy Gradients (TD3) (Fujimoto et al., 2018b) and Soft Actor-
Critic (SAC) (Haarnoja et al., 2018) are very popular for solving continuous
control tasks, and the off-policy nature of these algorithms due to the
ability to learn from replay buffer data often makes them more sample
efficient compared to the on-policy counterparts such as TRPO (Schulman et
al., 2015). However, these methods are incapable to learn solely from off-
policy data, and often requires online samples in a growing batch off-policy
setting for good performance. We first analyse the sample efficiency of
popular off-policy based policy gradient methods such as DDPG, TD3, and SAC
(Lillicrap et al., 2015; Fujimoto et al., 2018b; Haarnoja et al., 2018). We
interpolate between pure batch setting and online sample collection to
investigate how much these methods can rely on past data. We empirically show
that they have to collect data online to perform well, in a growing batch off-
policy setting. We hypothesize that this is mostly because of a lack of state
distribution correction in off-policy actor-critic methods, resulting in a
state distribution shift which further amplifies the effects of extrapolation
error. Although recent works have aimed towards state distribution correction
(Liu et al., 2019), such methods are often not scalable to the deep RL
continuous control tasks based on popular value-gradient based algorithms
(Silver et al., 2014).

We propose an algorithm that constrains the state distribution to account for
the mismatch in the data distribution under the current policy and past
behaviour policies. To do this, we first present a novel method to estimate
the state distribution with state features based on the state occupancy
measure using a density estimator (e.g a variational auto-encoder (Kingma and
Welling, 2014)). We then constrain the policy gradient update to minimize the
KL divergence between the behaviour and target state distributions. Based on
our proposed state distribution constrained policy gradient algorithm, we find
that correcting for the mismatch in the data distribution can significantly
improve performance in off-policy based methods. The key contributions of our
work are as follows :

  * •

We analyse the sample efficiency of popular off-policy actor-critic
algorithms. We train agents using sample buffers that interpolate between
current and past samples. We also train them in a pure batch setting, in which
the replay buffer is fixed. We find that even off-policy based methods rely on
a growing batch of online samples. We hypothesize that this is due to the
mismatch in state distributions under the behaviour and learned policies.

  * •

We present a novel method to estimate the state distribution using state
features based on the state occupancy measure. We use a parameterized density
estimator that directly uses the state features to estimate the state
distribution $d_{\pi}(s)$, where we use a policy architecture to output the
state features based on state visitation. This coupling ensures that we can
find a gradient estimate $\nabla_{\theta}d_{\pi_{\theta}}(s)$ w.r.t to the
policy parameters $\theta$ for the state distribution, even though we use a
separately parameterized density estimator with parameters $\phi$.

  * •

We then present an off-policy policy gradient algorithm that relies on
estimating the state distributions under the behaviour and target policies, to
constrain and minimize the state distribution mismatch. Our proposed algorithm
uses a KL-regularized constraint for the state distributions
$KL(d_{\mu}||d_{\pi})$, to minimize the state distribution shift and improve
the policy by moving away from the state distribution under the behaviour
policy. Our algorithm provides a similar trust region constraint as in
Schulman et al. (2015) based on KL between the state instead of policy
distributions.

##  2 Preliminaries

We consider a finite horizon MDP
$M=<\mathcal{S},\mathcal{A},\mathcal{P},r,\gamma>$ with a continuous state
space $\mathcal{S}$ and continuous action space $\mathcal{A}$, with transition
dynamics $\mathcal{P}$, rewards $r$ and discount factor $\gamma$. The goal is
to find a parameterized policy $\pi_{\theta}(a,s)$ which can maximize the
expected discounted return $J(\pi)=\operatorname{\mathbb{E}}_{s\sim
d_{\pi}(s),a\sim\pi(a|s)}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}]$, where
$d_{\pi}(s)$ is the stationary state distribution, given by
$d_{\pi}(s)=\frac{1}{\sum_{t=0}^{T}\gamma^{t}}\sum_{t=0}^{T}\gamma^{t}d^{t}_{\pi}(s)$.
In contrast, the discounted state distribution with discounting applied to
future states is given by
$d_{\pi}(s)=(1-\gamma)\sum_{t=0}^{\infty}\gamma^{t}\mathop{P}(s_{t}=s\mid
s_{0})$. We mostly consider policy gradient based methods, based on the policy
gradient theorem (Sutton et al., 2000). The policy gradient theorem is used to
maximize the cumulative return objective function given by
$\operatorname{\mathbb{E}}_{s\sim
d_{\pi}(s),a\sim\pi(a,s)}[\nabla_{\theta}\log\pi_{\theta}(a|s)Q^{\pi}(s,a)]$.
In off-policy based policy gradient methods, a different behaviour policy,
denoted by $\mu(a,s)$, is used to collect the data to obtain the policy
gradient with an importance sampling based correction for the mismatch in
action distributions (Precup, 2000). Most off-policy based actor-critic
algorithms are derived from the Off-PAC algorithm (Degris et al., 2012), where
the policy gradient is given by
$\nabla_{\theta}J(\theta)=\operatorname{\mathbb{E}}_{s\sim
d_{\mu}(s),a\sim\mu(a,s)}[\frac{\pi(a,s)}{\mu(a,s)}\nabla_{\theta}\log\pi_{\theta}(a|s)Q^{\pi}(s,a)]$.
In practice, an exploratory behaviour policy $\mu(a,s)$ is used which is often
coupled with the target policy, given by $\mu(a,s)=\pi(a,s)+\epsilon$ for
continuous action space MDPs.

##  3 Distribution Shift in Off-Policy Methods

Figure 1: Reward curves for DDPG with delayed buffer sampling average over 10
seeds. The baseline corresponds to $d=0$ delay. Delay is in episodes. We find
that the more delayed samples we use from the replay buffer, the worse the
performance gets, and it is always best to use random batch selection
containing past and present policies.

Figure 2: Reward curves for DPPG with windowed buffer sampling averaged over
10 seeds. Windowing is in episodes. The baseline corresponds to window size of
$W=10000$ episodes. We find that using the most recent samples in the replay
buffer is often most useful, as these samples have less distribution shift
compared to the data distribution under the target policy. This is as
expected, since in most of these algorithms we often use a behaviour policy
$\mu(a,s)=\pi(a,s)+\epsilon$ that is only separated by a noise $\epsilon$ in
the action selection

We first investigate to what extent state distribution shift matters in off-
policy based methods, by interpolating between very old data from the buffer
to the most recent data collected under the behaviour policy. We find that the
data distribution shift can significantly impact performance in policy
updates, where very old samples from the batch data can significantly worsen
performance. Our analysis suggests that off-policy based methods are highly
dependent on the online most recent samples collected under the behaviour
policy, especially since the behaviour policy is often coupled with the target
policy and separated by noise in the action space. We systematically analyse
the impact of state distribution shift in off-policy gradient methods, by
changing the sampling scheme used to collect batches of data from the replay
buffer. We motivate the impact of state distribution shift, by introducing two
new sampling schemes from the replay buffer. We use popular algorithms namely
Deep Deterministic Policy Gradient Algorithm (DDPG) (Todorov et al., 2012) and
its variants TD3 (Fujimoto et al., 2018a) and Soft Actor-Critic (Haarnoja et
al., 2018) and analyse the impact of state distribution shift in continuous
control Mujoco benchmark tasks. Additional experiment results are included in
the appendix.

Sampling from Delayed Replay Buffer : We sample transitions that were made at
least $d$ episodes ago. We use samples of data that were collected with older
policies, which may be significantly different compared to the current policy.
This lets us study the impact of distribution shift in these methods. Sampling
from Windowed Replay Buffer :  We can also sample transitions only from a
window of the latest $W$ episodes or roll-outs. This lets us study the extent
to which off-policy methods rely on recent samples.

Experiment Results with Delayed Replay Buffer: From Figure 1, we observe that
the baseline (corresponding to delay = 0) consistently performs the best. As
we increase the delay, we are effectively increasing the state distribution
shift. When the distribution of data in the buffer does not correspond with
the distribution induced by the current policy, this leads to a poor value
function approximation. Therefore, our experimental results in Figure 1 verify
the hypothesis that increase in distribution shift degrades the performance of
the policy since policy improvement with the policy gradient update relies on
out of state-distribution samples, which may not be visited by the target
policy at all.

Experiment Results with Windowed Replay Buffer: are as shown in Figure 2. We
hypothesize that when the data used for state-action value updates is
inadequate, the values of missing state-action pairs are extrapolated by the
critic, causing errors in the action values. Bootstrapping with erroneous
values propagates the error to all state-action values.

Summary : From our experimental studies with the different sampling schemes
from the replay buffer, we can conclude that the state distribution shift
indeed can degrade performance of off-policy policy gradient algorithms. When
using delayed buffer for sampling batches of data, we find that the baseline
with random sampling performs best since it can explicitly use both the online
and offline samples under the target policy, whereas the results with a
delayed window would only use past data that has a significant state
distribution shift. This also explains why methods such as prioritized
experience replay (Schaul et al., 2016) can be useful in off-policy settings.
Windowed buffer experiments suggest that a wide range of samples are necessary
to cover as much state space as possible, and in cases with sufficient state
space coverage even with smaller windows, using the whole buffer doesn’t give
any particular advantage.

##  4 Constraining the State Distribution Shift

In this work, we hypothesize that the inability of off-policy deep RL
algorithms to learn solely from batch offline data is due to the mismatches in
the data distribution under the behaviour and target policies. Since the
behaviour policy and updating the target policy based on past experience data
can lead to state visitation distribution which is quite different than what
would have been the state distribution under the target policy, this mismatch
in the state distributions can lead to divergence in off-policy learning
algorithms. To remedy this, we propose a trust region KL constraint based on
the state distributions under the behaviour policy $d_{\mu}(s)$ and the target
policy $d_{\pi}(s)$. Our proposed algorithm is based on minimizing the state
distribution shift to constrain the target policy updates towards states that
are more likely to be visited by the target policy, even though in off-policy
RL the samples are drawn from the behaviour policy. In other words, during
policy improvement and gradient update, we constrain the target policy
parameters $\theta$ towards the target policy state distribution
$d_{\pi_{\theta}}(s)$. However, this is often difficult in practice and can
lead to sample inefficiency since the target policy is not often accessible in
off-policy learning making it difficult to estimate $d_{\pi_{\theta}}(s)$. To
this end, we use an approximation where we use the near on-policy samples
where the samples under the most recent behaviour policy
$\mu=\pi_{\theta}(s)+\epsilon$ which is coupled with the target policy and
separated by a noise $\epsilon$ (as in algorithms such as DDPG), instead of
re-using any samples from the experience replay buffer data. We emphasize that
most off-policy actor-critic methods do not correct for the state distribution
shift under behaviour and target policies (Imani et al., 2018; Degris et al.,
2012; Silver et al., 2014; Haarnoja et al., 2018) resulting in the state
distribution shift.

###  4.1 Estimating the State Distribution

To constrain the state distribution shift, we present a novel method towards
estimating the state distribution based on a density estimator, such that we
can still regularize the policy gradient update based on a gradient estimate
w.r.t to the state distribution, ie, we can take the gradient
$\nabla_{\theta}d_{\pi_{\theta}(s)}$ in the policy gradient update. This is
often difficult in practice since we require finding the gradient w.r.t to the
state distribution $d_{\pi_{\theta}(s)}$. We highlight that in our approach,
we are not estimating the discounted stationary state distribution, as we are
considering finite horizon episodic tasks, but rather estimating the state
density based on the state visitation frequency. We estimate the state
distribution based on a density estimator as follows : We use a separately
parameterized density estimator with parameters $\phi$, such that we can
estimate the log probability density based on the state visitation $\log
p(s)$, which we denote by $\log d_{\pi}(s)$. In our approach, we use a
generative model such as a variational auto-encoder (VAE) to obtain lower
bound approximations to $\log p(s)$. Since we require the log probability
density to be dependent on the policy parameters $\theta$, we therefore
estimate the state distribution based on state features $\psi_{\pi}(s)$ which
is directly influenced by the policy $\pi_{\theta}(s)$. Our model is given as
follows : we use a policy network which outputs not just the action
probabilities $\pi_{\theta}(a|s)$ but also the state features $\psi(s)$. We
can therefore represent the policy network jointly as $\pi(a,\psi(s)|s)$. The
state features $\psi_{\pi}(s)$ are given as input to the VAE density
estimator, parameterized by $\phi$, which maps the features into a latent
space representation $q_{\phi}(z|\psi(s)$ that captures a lower dimensional
learnt representation of the state features, and the VAE trained to optimize
the following lower bound based on the
$KL(q_{\phi}(Z\mid\psi_{\theta}(S)||p(\psi_{\theta}(S)))$.

$\begin{split}\log\psi(s)&=\log d_{\pi}(S)=\log p(S)\\\
&>=\operatorname{\mathbb{E}}_{q_{\phi}(Z\mid\psi_{\theta}(s)}[\log
p_{\phi}(S\mid\psi_{\theta}(s)]\\\
&-KL(q_{\phi}(Z\mid\psi_{\theta}(S)||p(\psi_{\theta}(S)))\\\
&=\mathcal{L}(\phi,\theta)\\\ \end{split}$ (1)

In equation 1 above, the VAE takes as input the state features, instead of the
raw states, that are dependent on the policy $\psi_{\pi}(s)$ since it is the
output from the policy network $\pi(a|s)$ and directly depends on the states
visited by the policy. The encoder in this case learns a latent representation
of the state features and the VAE is therefore trained to maximize the log
probability of the state features $\log\psi(s)$ instead of the raw states
$\log p(s)$. Our estimated state distribution is therefore based on the state
features instead of the raw states, which we find to be a more practically
feasible approach given the type of benchmark tasks considered for most deep
RL algorithms. To maintain clarity in notation, in the text as follows we will
denote this as $d_{\pi}(s)$ and not $d_{\pi}(\psi(s))$, even though our model
is based on the state features conditioned on the policy. Since the VAE takes
as input the state features directly $\psi_{\pi_{\theta}}(s)$ or
$\psi_{\mu}(s)$ (where $\mu(a,s)$ is closely related to $\pi_{\theta}(a,s)$ in
practice), this ensures that there exists a gradient w.r.t to the ELBO term is
dependent on both the parameters $\theta$ and $\phi$. In other words, while
the VAE is separately parameterized by $\phi$, there also exists a gradient
term w.r.t policy parameters $\theta$ for the ELBO term, since $\psi_{\pi}(s)$
is also directly dependent on policy parameters $\theta$. Based on the states
under behaviour policy $\mu(a,s)$ which we denote by $s_{\mu}$ and the target
policy $\pi(a,s)$ denoted by $s_{\pi}$ we can therefore obtain lower bound
approximations to the state distributions under the behaviour policy
$d_{\mu}(s)$ and the target policy $d_{\pi}(s)$. This allows us to constrain
the policy gradient update with a trust region constraint based on the state
distribution mismatch $KL(d_{\mu}(s)||d_{\pi}(s))$. Below we include details
about estimating $d_{\mu}(s)$ and $d_{\pi}(s)$ which is key to our proposed
algorithm.

Estimating $d_{\pi}(s)$ :  To estimate $d_{\pi}(s)$, we require samples that
are collected by the target policy $\pi(a,s)$ with which to train the density
estimator to obtain $d_{\pi}(s)$. However, in off-policy methods, since we do
not explicitly use $\pi(a,s)$ to collect samples, we instead use the online
samples that are collected based on the current roll-out. Therefore, in
practice, instead of truly estimating $d_{\pi}(s)$, we instead compute an
approximation $d_{\pi}(s)+\epsilon(s)$, which we can denote by
$d_{\pi+\epsilon}(s)$. We then separately store the current online samples
with $\mu(a,s)=\pi_{\theta}(a,s)+\epsilon$ in a separate online reply buffer
based on the current roll-out. These samples are then used to estimate
$d_{\pi}(s)$ by training the VAE density estimator with the online samples to
obtain a crude approximation for $d_{\pi}(s)\approx d_{\pi+\epsilon}(s)$.

Estimating $d_{\mu}(s)$ :  To estimate the off-policy state distribution is
more convenient, since we can now directly use the samples that are stored in
the replay buffer memory. The replay buffer stores all the off-policy samples
collected under $\mu(a,s)$ with which we can train the density estimator to
obtain $d_{\mu}(s)$.

###  4.2 Off-Policy Policy Gradient with Constrained Distribution Shift

We propose an algorithm that constraints the policy gradient update to account
for the mismatch in the state distribution induced by the mismatch between the
target and behaviour policies. Our algorithm is based on a constrained KL
regularizer on the policy gradient, which instead of constraining the policies
between current and old policies (e.g as in TRPO (Schulman et al., 2015),
propose a similar trust region update based on the state distributions. We
estimate an approximation to the state distribution between the target and
behaviour state samples to obtain $d_{\pi}(s)$ and $d_{\mu}(s)$. However,
since we cannot exactly compute $d_{\pi}(s)$ as the target policy is not
accessible in an off-policy learning setting, we compute a near on-policy
state distribution $d_{\pi+\epsilon}(s)$ based on using the most recent
samples collected under the behaviour policy. In other words, we estimate
$d_{\mu}(s)$ using state samples from the replay buffer, whereas
$d_{\pi+\epsilon}(s)$ is estimated based on the current set of trajectories
under the behaviour policy. This therefore gives us the following state
distribution based trust region constrained off-policy policy gradient
objective where the $\mathbb{E}$ is with respect to the behaviour policy
samples and we use a regularizer $KL(d_{\mu}(s)||d_{\pi_{\theta}}(s)))$, such
that the overall gradient objective is :

$J(\pi_{\theta})=\operatorname{\mathbb{E}}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}(s,a)-KL(d_{\mu}(s_{\mu})||d_{\pi_{\theta}+\epsilon}(s_{\pi})))]$
(2)

where for clarity and explanation, the $\mathbb{E}$ is w.r.t off-policy states
sampled from the replay buffer denoted by $s_{\mu}$ while we also sample the
recent most samples generated by $\mu(a,s)=\pi_{\theta}(a,s)+\epsilon$ from
the replay buffer, denoted by $s_{\pi}$. The $\mathbb{E}$ samples both
$s_{\pi+\epsilon}$ and $s_{\mu}$ based on which we estimate $d_{\mu}(s)$ and
$d_{\pi_{\theta}+\epsilon}(s)$. Note that, our approach does not require
additional samples compared to the benchmark algorithms and both algorithms
would require same amount of samples. Indeed, we simply write the above
objective to clarify that the KL regularizer is based on $d_{\mu}(s)$ and
$d_{\pi+\epsilon}(s)$ for constraining the state distribution shift. Based on
the above objective, we therefore get the following trust region regularized
off-policy gradient update that minimizes for the state distribution shift
with the $KL(d_{\mu}(s)||d_{\pi}(s))$.

$\nabla_{\theta}J(\theta)=\operatorname{\mathbb{E}}_{s\sim d_{\mu}(s),s\sim
d_{\pi_{\theta}(s)}}[\nabla_{\theta}Q^{\pi}(s,\pi_{\theta}(s))-\nabla_{\theta}KL(d_{\mu}(s)||d_{{\pi_{\theta}}+\epsilon}(s))]$
(3)

In the above expression, the gradient w.r.t $\theta$ exists for the KL term
since we estimate $d_{\pi_{\theta}(s)}$ with a density estimator based on
state features $\psi_{\pi_{\theta}(s)}$ that are directly dependent on the
policy parameters $\theta$. Therefore, backpropagating the gradients w.r.t to
the KL term in the above equation will pass the gradients w.r.t to the policy
parameters $\theta$. For clarity and notation, we can further expand the
gradient term above by re-writing with separate $\mathbb{E}$ terms since only
the second KL term depends on the states sampled from
$d_{\pi_{\theta}+\epsilon(s)}$

$\nabla_{\theta}J(\theta)=\operatorname{\mathbb{E}}_{s\sim
d_{\mu}(s)}[\nabla_{\theta}Q^{\pi}(s,\pi_{\theta}(s))]-\operatorname{\mathbb{E}}_{s\sim
d_{\mu}(s),s\sim d_{\pi}(s)}[\nabla_{\theta}[\log d_{\mu}(s)-\log
d_{\pi_{\theta}}(s)]]$ (4)

In equation 4, the second term is the KL term, the gradient is w.r.t to
$\theta$ for the state distribution $d_{\pi_{\theta}}(s)$. This can be
interpreted as a trust region regularizer based on the state distributions.

Algorithm :  Our proposed algorithm is outlined in the algorithm box given in
Appendix. We use a policy network that outputs both the action distributions
and the state features $\pi_{\theta}(a,\psi(s)|s)$. As in any off-policy
gradient method, we use a behaviour policy $\mu(a,s)$ which is related or
coupled with the target policy, e.g $\mu(a,s)=\pi(a,s)+\epsilon$. We store all
the samples in a replay buffer memory, using which the off-policy state
distribution $d_{\mu}(s)$ is estimated with a VAE. We use the current online
samples with $\mu(a,s)$ to estimate $d_{\pi+\epsilon}$ which we assume to
closely approximate $d_{\pi}$ since we assume that the additional $\epsilon$
noise would only change the state visitation distribution by an $\epsilon$
amount. The proposed KL regularizer $KL(d_{\mu}(s)||d_{\pi}(s))$ can then be
used with any off-policy gradient update such as in DDPG, SAC or TD3 with a
$\lambda$ weighted regularizer. By minimizing this KL term, our algorithm can
therefore minimize the state distribution shift in off-policy actor-critic
algorithms.

##  5 Experimental Results

We first demonstrate our approach in the growing batch where the agent keeps
storing all instances of past experiences in the replay buffer while it
learns. Our experimental analysis is based on a range of continuous control
Mujoco tasks (Todorov et al., 2012) and we compare our proposed algorithm with
standard state-of-the-art off-policy policy based methods including DDPG
(Lillicrap et al., 2015), TD3 (Fujimoto et al., 2018b) and Soft Actor-Critic
(SAC) (Haarnoja et al., 2018). In all our experimental results we compare with
the above baselines and add our state distribution KL regularizer with a
$\lambda$ weighting to show the significance of minimizing the state
distribution shift. Detailed experimental results and ablation studies are
also included in the appendix, and we provide code for our algorithm as
supplementary. We show that minimizing the state distribution shift can
significantly improve performance over baselines. All our experimental results
are averaged over $5$ random seeds.

Figure 3: Performance improvements in DDPG algorithm with the proposed
"StateKL" regularizer

Comparisons with DDPG : We first compare our proposed algorithm with DDPG,
where we examine the significance of our proposed approach on several
continuous control benchmark tasks. As shown in figure 3, we find that with
the StateKL regularizer, we can achieve significant performance improvements
across all the environments.

Figure 4: Performance improvements in TD3 algorithm with the proposed
"StateKL" regularizer

Comparisons with TD3 :  We then analyse the usefulness of minimizing the state
distribution shift on TD3 framework as shown in figure 4 and find similar
improvements in performance across the environments.

Figure 5: Performance improvements in SAC algorithm with the proposed
"StateKL" regularizer

Soft Actor-Critic (SAC) :  Finally, we test the significance of our proposed
approach on the SAC framework. In two out of the three environments, we find
that StateKL regularizer improves the performance over baseline SAC algorithm.

Summary :  In all our experimental results comparing with standard off-policy
state-of-the-art policy optimization baselines, we find that adding the
constraint with the state distribution shift regularizer can significantly
improve performance over the baseline algorithms. We note that we do not
compare with other baselines here, since our work is the first, to our
knowledge, to propose such constraints minimizing the state distribution
shift. This is also partly because of our novel framework to estimate the
state distribution as well. While other works correct for the mismatch in the
state distribution by adding an importance correction
$\frac{d_{\pi}}{d_{\mu}}$ (Liu et al., 2019), they do not compare in the deep
RL continuous control setting, but rather shows simple benchmark tasks making
the algorithms incomparable to ours.

##  6 Constrained State Distribution Shift in Batch Off-Policy Learning

We further use our proposed constraint on the state distribution shift in off-
policy, in the pure fixed batch learning setting (Fujimoto et al., 2018a) with
off-policy policy gradient algorithms. As such, the agent is given only a
fixed batch of data collected using a pre-trained expert policy, and the goal
is to perform policy improvements given this fixed batch of data without
further online interactions with the environment. We use the same experimental
setup as proposed in the Batch Constrained deep Q-learning (BCQ) algorithm
(Fujimoto et al., 2018a). The BCQ algorithm uses a separate generative model
to sample new actions based on data from the batch and use a constrain between
the new sampled actions and the actions available in the batch to constrain
the policy updates. In our work, we build on the framework of BCQ and use the
same setup with additional constraint with the state distribution shift
regularizer. We analyze to what extend constraining the state distribution
shift $KL(d_{\mu}||d_{\pi_{\theta}+\epsilon})$, in addition to constraining
the action space as in BCQ, can help in improving performance of standard off-
policy policy based algorithms in the fixed batch setting. As in before, we
use all the old instances of the expert data to estimate state distribution
$d_{\mu}(s)$, and use the recent most near on-policy samples to estimate
$d_{\pi}(s)$ to compute KL. Experimental results below shows the significance
of our proposed state KL regularizer for constraining the state distribution
shift in batch off-policy learning.

Figure 6: Comparisons with the BCQ Algorithm where we use our proposed KL
regularizer for constraining the state distribution shift, in addition to the
constrained action space as proposed in the BCQ algorithm for expert batch
data

Figure 7: Comparisons with the BCQ Algorithm where we use our proposed KL
regularizer for constraining the state distribution shift, in addition to the
constrained action space as proposed in the BCQ algorithm for transient batch
data

In our experiments, we analyse two different batch settings $(i)$ Expert batch
: A trained DDPG policy is used to collect the fixed batch of data used to
train our agent, and $(ii)$ Transient batch: We train a DDPG policy and
collect all the transitions experienced during the training process. Figure 6
shows comparisons with BCQ algorithm in the pure batch setting where a given
pre-trained expert policy was used to generate the buffer. We show
significance of constraining the distribution shift between the near on-policy
samples and samples from a very old policy. In figure 7 we further compare
with BCQ where a growingly trained expert policy was used to collect the
buffer. We do this since in this setting it is more meaningful to compute the
near on-policy state distribution as we use the recent most expert samples to
compute $d_{\pi}(s)$ and old samples from batch to compute $d_{\mu}(s)$.

##  7 Discussion and Conclusion

In this work, we first analyse the ability of off-policy gradient based
methods to re-use past samples for data efficiency, and find that off-policy
methods are not truly data efficient due to their inability to use very old
off-policy data. This is because the old samples can have a state distribution
which is quite different than the target policy’s state distribution, leading
to a state distribution shift in off-policy learning. Based on our empirical
findings, we then propose a trust region constrained off-policy gradient
update which can minimize the state distribution shift between target and
behaviour policy state distributions $d_{\pi}$ and $d_{\mu}$. We showed that
by estimating the state distribution, based on the state visitation, using a
separate density estimator, we can get estimates of the state distribution for
both the target $d_{\pi}(s)$ and the behaviour $d_{\mu}(s)$ policies. We
present a novel approach towards estimating the state distribution, which uses
an explicit density estimator that takes as input the visited state features
under a given policy. Existing off-policy gradient based methods do not
correct for the state distribution mismatch, and in this work we show that
instead of computing the ratio over state distributions, we can instead
minimize the $KL(d_{\mu}(s)||d_{\pi}(s))$ between the target and behaviour
state distributions to account for the state distribution shift in off-policy
learning. In our experimental results, we demonstrated the usefulness of
minimizing this state distribution shift and showed significant improvements
in performance across several continuous control tasks.

## References

  * Fujimoto et al. (2018a) Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  _CoRR_ , abs/1812.02900, 2018a.  URL http://arxiv.org/abs/1812.02900.
  * Lillicrap et al. (2015) Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ , 2015.
  * Fujimoto et al. (2018b) Scott Fujimoto, Herke van Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , pages 1582–1591, 2018b.  URL http://proceedings.mlr.press/v80/fujimoto18a.html.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , pages 1856–1865, 2018.  URL http://proceedings.mlr.press/v80/haarnoja18b.html.
  * Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz.  Trust region policy optimization.  In _Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, Lille, France, 6-11 July 2015_ , pages 1889–1897, 2015\.  URL http://jmlr.org/proceedings/papers/v37/schulman15.html.
  * Liu et al. (2019) Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill.  Off-policy policy gradient with state distribution correction.  _CoRR_ , abs/1904.08473, 2019.  URL http://arxiv.org/abs/1904.08473.
  * Silver et al. (2014) David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In _International Conference on Machine Learning_ , 2014.
  * Kingma and Welling (2014) Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  In _2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings_ , 2014.  URL http://arxiv.org/abs/1312.6114.
  * Sutton et al. (2000) Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour.  Policy gradient methods for reinforcement learning with function approximation.  In _Advances in neural information processing systems_ , 2000.
  * Precup (2000) Doina Precup.  Eligibility traces for off-policy policy evaluation.  _Computer Science Department Faculty Publication Series_ , page 80, 2000.
  * Degris et al. (2012) Thomas Degris, Martha White, and Richard S Sutton.  Off-policy actor-critic.  _arXiv preprint arXiv:1205.4839_ , 2012.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033. IEEE, 2012.
  * Schaul et al. (2016) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_ , 2016.  URL http://arxiv.org/abs/1511.05952.
  * Imani et al. (2018) Ehsan Imani, Eric Graves, and Martha White.  An off-policy policy gradient theorem using emphatic weightings.  In _Advances in Neural Information Processing Systems_ , 2018.
  * Liu et al. (2018) Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou.  Breaking the curse of horizon: Infinite-horizon off-policy estimation.  In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, 3-8 December 2018, Montréal, Canada._ , pages 5361–5371, 2018.  URL http://papers.nips.cc/paper/7781-breaking-the-curse-of-horizon-infinite-horizon-off-policy-estimation.
  * Gelada and Bellemare (2019) Carles Gelada and Marc G. Bellemare.  Off-policy deep reinforcement learning by bootstrapping the covariate shift.  _CoRR_ , abs/1901.09455, 2019.  URL http://arxiv.org/abs/1901.09455.
  * Hallak and Mannor (2017) Assaf Hallak and Shie Mannor.  Consistent on-line off-policy evaluation.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , pages 1372–1383, 2017.  URL http://proceedings.mlr.press/v70/hallak17a.html.

##  8 Appendix

###  8.1 Related Work

Off-policy policy gradient methods are popular for continuous control tasks
[Lillicrap et al., 2015, Fujimoto et al., 2018b, Haarnoja et al., 2018] and
are mostly derived from the Off-PAC algorithm [Degris et al., 2012, Imani et
al., 2018]. However, none of the previous works accounted for the mismatch in
state distributions in off-policy gradient methods. Recently, Liu et al.
[2019] have introduced an off-policy method that can be used with batch data,
while correcting for the difference in the state distributions between current
target and behaviour policies. Their approach builds on recent works on off-
policy policy evaluation [Liu et al., 2018] by computing a direct ratio over
the stationary state distributions under the target and behaviour policies.
However, one drawback of their approach is the need to explicitly know the
model dynamics in order to compute the stationary distributions. In other
words, their approach is not completely model-free and they do not use an
explicit estimator of the stationary distributions unlike ours, which makes
their algorithm difficult to scale up in practice. Recently, Gelada and
Bellemare [2019] also introduced an off-policy correction for the state
distributions, where the ratio is directly estimated by bootstrapping. They
proposed an algorithm to estimate this ratio directly, based on the covariate
shift approach from Hallak and Mannor [2017] and extended their results in the
deep reinforcement learning framework. In contrast to Gelada and Bellemare
[2019] which requires estimating this ratio, where the policy gradient
estimator if used with their approach would have high variance in gradient
estimators, we instead correct for the mismatch in state distributions by
minimizing the state distribution shift based on the KL divergence between the
state distributions.

###  8.2 Algorithm

Algorithm 1 Off-Policy Policy Gradient with State Distribution Shift
Minimization

0: A policy $\pi_{\theta}(a,\psi(s)\mid s)=\psi(s)p_{\theta}(a\mid s)$ and
$\lambda$ regularization coefficient

0: The number of episodes, $E$ and update interval, $N$.

1: for $e=1$ to $E$ do

2: Take action $a_{t}$,get reward $r_{t}$ and observe next state $s_{t+1}$

3: Store tuple ($s_{t},a_{t},r_{t+1},s_{t+1}$) as trajectory rollouts or in
replay buffer $\mathcal{B}$

4: Estimate target policy state distribution $d_{\pi+\epsilon}(s)$ by
maximizing variational lower bound (ELBO) using samples collected by
$\pi(a,s)+\epsilon$

5: Estimate behaviour policy state distribution $d_{\mu}$ by maximizing
variational lower bound (ELBO) using samples collected by $\mu(a,s)$

6: Update policy parameters $\theta$ following any policy gradient method,
according to
$\begin{split}\nabla_{\theta}\tilde{J}(\theta)&=\operatorname{\mathbb{E}}_{s_{\mu}\sim
d_{\mu}(s),s_{\pi}\sim
d_{\pi}(s)}\Bigl{[}\nabla_{\theta}Q^{\pi_{\theta}}(s,\pi_{\theta}(s))\\\
&-\lambda\nabla_{\theta}KL(d_{\mu}(s_{\mu})||d_{\pi_{\theta}}(s_{\pi}))\Bigr{]}\\\
\end{split}$ (5)

7: end for

###  8.3 Additional Results : Distribution Shift in Off-Policy Methods

Delayed Buffer :  We include additional results here for the delayed buffer
experiments with both SAC and TD3 in Figure 8 and 9. In the main draft, we
demonstrated the impact of using old samples from the replay buffer in DDPG.
In the results shown here, we find similar trends in results with both TD3 and
SAC as well, where performance drops as we use out of distribution batch
samples for training.

Figure 8: Reward curves for SAC with windowed buffer sampling averaged over 10
seeds. Windowing is in episodes. The baseline corresponds to window size of
$W=10000$ episodes.

Figure 9: Reward curves for TD3 with windowed buffer sampling averaged over 10
seeds. Windowing is in episodes. The baseline corresponds to window size of
$W=10000$ episodes.

Summary - Delayed Buffer Experiments :  Our experimental results across all
off-policy policy gradient based algorithms therefore conclude that as we
increase the distribution shift, by using older samples from the replay
buffer, the performance of off-policy policy gradient algorithms degrades.
This validates our hypothesis that distribution shift impacts the performance
in off-policy learning.

Windowed Buffer :  Additional results for the windowed buffer experiments with
TD3 and SAC are shown in Figure 10 and 11. Similar patterns as demonstrated
for DDPG in the main draft hold for both these algorithms across all tasks. In
particular, we see that environments with shorter buffer lengths (Hopper and
Walker2d) perform worse with shorter windows. This is likely because they have
sparsely distributed samples in the buffer, causing erroneous action value
estimates.

Figure 10: Reward curves for SAC with windowed buffer sampling averaged over
10 seeds. Windowing is in episodes. The baseline corresponds to window size of
$W=10000$ episodes.

Figure 11: Reward curves for TD3 with windowed buffer sampling averaged over
10 seeds. Windowing is in episodes. The baseline corresponds to window size of
$W=10000$ episodes.

Summary - Windowed Buffer Experiments :  All our experiments show that with
the buffer restricted to only the most recent samples, performance of the
algorithms under study deteriorates. This suggests that they need data
sufficiently spread across the state space for better generalization of action
values.

###  8.4 Ablation Studies with State Distribution Shift

In this section, we include additional experimental results with different
$\lambda$ weightings for the StateKL regularizer. We include results for a
range of $\lambda$ and compare our results with baseline off-policy algorithms
including DDPG, TD3 and SAC in Figures 12, 13, and 14.

Figure 12: Ablation studies with different $\lambda$ StateKL regularizers in
DDPG

Figure 13: Ablation studies with different $\lambda$ StateKL regularizers in
TD3

Figure 14: Ablation studies with different $\lambda$ StateKL regularizers in
SAC


# On Convolutional Vision Transformers for Yield Prediction

Alvin Inderka Mail<EMAIL_ADDRESS>Department of Computer Science IV,
University of Bonn
Friedrich-Hirzebruch-Allee 8, 53121, Bonn, Germany  Florian Huber Mail:
<EMAIL_ADDRESS>Department of Computer Science IV, University of Bonn
Friedrich-Hirzebruch-Allee 8, 53121, Bonn, Germany  Volker Steinhage Mail:
<EMAIL_ADDRESS>Department of Computer Science IV, University of Bonn
Friedrich-Hirzebruch-Allee 8, 53121, Bonn, Germany

###### Abstract

While a variety of methods offer good yield prediction on histogrammed remote
sensing data, vision Transformers are only sparsely represented in the
literature. The Convolution vision Transformer (CvT) is being tested to
evaluate vision Transformers that are currently achieving state-of-the-art
results in many other vision tasks. CvT combines some of the advantages of
convolution with the advantages of dynamic attention and global context fusion
of Transformers. It performs worse than widely tested methods such as XGBoost
and CNNs, but shows that Transformers have potential to improve yield
prediction.

##  1 Introduction

Precise yield predictions are used for economic and logistical planning by
farmers, traders, producers, and political actors. They can avoid
misallocations of food and financial risks for farmers in times of climate
change and its impact on yields. While ground-level measurements are cost-
intensive and cannot be applied on a large scale, remote sensing data provide
a cost-effective and well-studied database of plant condition at regional
level. In addition, the USDA (2021) maintains freely accessible county-level
soybean yield data statistics for the United States. Using this data,
traditional machine learning methods and deep learning methods could be
developed and refined. The often more successful deep learning methods are
mainly Artificial Neural Networks (ANN), Deep Neural Networks (DNN),
Convolutional Neural Networks (CNN), Long-Short Term Memory Networks (LSTM),
CNN-LSTMs and some similar variants of these (Muruganantham et al., 2022). The
use of more recent vision Transformers for yield prediction, on the other
hand, is sparsely represented and will be examined in this brief elaboration
using the Convolutional vision Transformer (CvT) by Wu et al. (2021).

##  2 Related Work

A variety of studies addressing yield prediction with remote sensing data and
deep learning applications. Muruganantham et al. (2022) provides a compact
overview of the prevailing deep learning approaches in a systematic literature
review. Van Klompenburg et al. (2020) offers a similar systematic literature
review with the additional consideration of traditional machine learning
methods. A general overview of Transformers in remote sensing is given by
Aleissaee et al. (2023). However, despite the fairly wide scope of the work,
regression is not discussed by Aleissaee et al. (2023) and, in general,
regression by vision Transformers is not well represented in the literature as
regression is a rather unusual vision task.

###  2.1 Yield Prediction

You et al. (2017) proposed a CNN and a LSTM pipeline for soybean yield
prediction in the USA. For the purpose of dimensionality reduction, the
satellite data of each county are preprocessed to a normalized histogram for
each band and is fed afterwards to the CNN or LSTM. Additionaly You et al.
(2017) proposed an Gaussian Process appended to the CNN and LSTM, which adds
tempo-spatial information to the model and improves the error significantly.
Sun et al. (2019) combines the advantages of spatial feature extraction of
CNNs and the capability of revealing phenological characteristic of LSTMs
while using similar histograms. The resulting CNN-LSTM Model extracts the
features by a CNN and uses an LSTM on the resulting feature map to learn
temporal features. This method proposed by Sun et al. (2019) achieves a higher
performance than CNNs and LSTMs individually and enables a satisfactory result
at an earlier stage before the harvest. Shooka et al. (2020) is using a
stacked LSTM and a Temporal Attention Model, which basically uses a Temporal
Attention Mechanism applied to the stacked LSTM for soybean yield prediction
on weather data given by weather.com and yield, genotype and location data
given by uniform soybean tests (UST) from the USDA. With the help of a greedy
search, the features with the greatest influence on the stacked LSTM are
picked out and the weights of the Temporal Attention Mechanism provide
information about the temporal dependencies of the Temporal Attention Model,
which gives a certain explainability of this deep learning models. The LSTM
and the Temporal Attention Model of Shooka et al. (2020) have a similar
performance, but both are more performant than a Random Forest (RF) and LASSO
regression. Another approach to promote explainability of yield prediction is
an XGBoost approach by Huber et al. (2022), which uses MODIS satellite data
and Daymet weather data to propose an explainable model with state-of-the-art
performance. Liu et al. (2022) use a Transformer model named Informer to
predict rice yield in the Indo-Gangetic Plains of India. In addition to the
official yield data, 8 different sequential variables such as the Normalized
Difference Vegetation Index (NDVI) were calculated using MODIS datasets and
two other environmental datasets. Liu et al. (2022) show that the Informer
performs better on this task than Attention-based LSTM (AtLSTM), XGBoost,
LASSO Regression and a Random Forest (RF), with AtLSTM being the second best
method. Bi et al. (2023) uses a pipeline of two standard vision Transformers
named ViT for feature extraction of near-field images. The images are
preprocessed so that one vision Transformer is responsible for soil and one
for soybean plants. The respective outputs are multiplied and fed into another
Transformer, where they are processed with seed information and fully
connected neural networks (FCNN). The resulting performance exceeds the
performance of a CNN-LSTM.

###  2.2 Vision Transformer

As discussed by Aleissaee et al. (2023), there is a variety of vision
Transformer models for different remote sensing tasks. Due to this diversity
of models and the simultaneous uncertainty regarding the performance of the
models in regression and especially yield prediction, only a few widely used
models were examined for analysis. Following the success of Transformers in
NLP, Dosovitskiy et al. (2020) achieves with Vision Transformer (ViT) an
adaptation of the Transformer and its pipeline to images using large training
datasets that outperform leading CNNs in some image classification benchmarks.
The vision Transformers based on this offer various capabilities and are often
presented with state-of-the-art results, especially in the tasks of image
classification, object detection and semantic segmentation. Based on our
previous experience with other methods, two popular vision Transformers in
particular were considered for testing. It should be noted that due to the
large number of possible vision Transformers and the lack of information on
their performance with regard to the yield prediction problem, the selection
of the following two Transformers and the choice of the evaluated CvT does not
claim to be optimal. Liu et al. (2021) present the Swin Transformer, a general
purpose Transformer that achieves good performance in image classification,
object detection, and semantic segmentation. The Swin Transformer reduces the
complexity of ViT, which is quadratic to the image size, to a linear one using
a shifted window approach. In this approach, self-attention is only used
within non-overlapping windows and the relationships between windows are
solved by alternating window partitioning between layers. The Convolutional
vision Transformer (CvT) evaluated below originates from Wu et al. (2021) and
combines the advantages of CNNs and Transformers. CvT is explained in chapter
4. Raghu et al. (2021) analyze the internal representation structure of vision
Transformers (using ViT) in comparison to CNNs (using ResNet), showing that
the strong inductive bias of spatial equivariance in CNNs is particularly
advantageous in the use of local information. ViT can only use this local
information effectively with the help of large amounts of data or large-scale
pretraining, but has advantages in the global use of information due to self-
attention.

##  3 Data

In order to ensure the best possible comparability with other models, the
processed dataset from Huber et al. (2022) is used and reduced to the test
years 2018 to 2021 for an accelerated evaluation. As the data set originates
directly from Huber et al. (2022), reference is made to it for a more detailed
description. A brief summary of the data and its processing according to Huber
et al. (2022) is provided below.

The dataset covers the 13 US states with the majority of US soybean acreage.
Yield data from the USDA (2021) is used at district level for the years 2003
to 2021 inclusive. The satellite data comes from NASA’s Moderate Resolution
Imaging Spectroradiometers (MODIS). In particular, the products MOD09A1
(Vermote, E., 2021) with 7 bands for spectral surface reflectance and MYD11A2
(Wan, Z., Hook, S., Hulley, G., 2021) with 2 bands for land surface
temperature are used. These products contain a resolution of 500 m per pixel
and consist of an 8-day composition of the images. The daily average water
vapor pressure and the daily absolute precipitation of the Daymet V41 dataset
(Thornton et al., 2014) are used for the weather. The resolution of 1 km is
reduced to 500 m using Google Earth Engines image pyramid and the daily data
is aggregated to 8-day compositions. The United States Census Bureau TIGER
dataset (U.C:, 2018) from 2018 is used to limit the respective satellite data.
In addition, an upscaled version of the USDA NASS Cropland Data Layer is used
to mask the soybean fields. All satellite data can be found in the Google
Earth Engine catalog.

The given data is cropped to an end-of-year harvest period between the 49th
and 321st day of the year and assigned to the respective district. The pixels
containing soybean fields are then extracted from the satellite data for each
district and converted into a histogram per band with 32 equally distributed
bins of frequencies. Thus, a resulting data point of a district for one year
consists of 11 histograms with a width of 34 8-day intervals and a height of
32 bins. In addition to identifying information on the respective district,
the data point is also annotated with the corresponding USDA yield data as
ground truth. This results in 14,543 data points with an average yield of
45.26 bushels per acre and a standard deviation of 10.80 bushels per acre.

##  4 Methods

Inspired by the success of CNN models in yield prediction and Vision
Transformer in different Tasks, we evaluate the Convolutional vision
Transformer (CvT) (Wu et al., 2021), which is proposed for image
classification. CvT combines the advantages of CNNs (local receptive fields,
shared weights, spatial subsampling) and Transformer (dynamic attention,
global context fusion, better generalization). As we do not make any
substantial adjustments to CvT, the following is a summary of the main
components of CvT according to Wu et al. (2021). Details can be found in the
reference. The adjustments to use CvT as a model for predicting soybean yield
using satellite imagery are described afterwards.

CvT as proposed by Wu et al. (2021) contains 3 stages with a similar
structure. Starting with the input image, in our case our multi dimensional
histogram, the Convolutional Token Embedding Layer generates tokens by the use
of convolution followed by flattening and layer normalization. These tokens
correspond to the tokens, often vectorized words, of the Transformers in NLP.
The number and feature dimension of these can be adjusted using the stride of
the convolution. The following Convolutional Transformer Block essentially
corresponds to the usual procedure of Multi-Head Self-Attention with the
modification of the initial linear projections of the input tokens to
convolutional projections with depth-wise separable convolutions. To use an
convolutional projection, it is necessary to reshape the tokens into a 2
dimensional token map and flatten the result of the convolution again. Three
different convolutional projections are used to determine query, key and
value. In particular, it is possible to undersample using a stride of 2 for
key and value and thus reduce the number of tokens by a factor of 4. Query,
key and value are then used for the Multi-Head Attention (MHA) and generate a
token map with the help of residual connections, normalization and an MLP, as
known from the Multi-Head Self-Attention. The next stage of the same form can
be applied to the token and a spatial downsampling can therefore be achieved.
A classification token is added to the last stage. The classification token is
passed to a final MLP, which finally determines the class or, in our case, the
yield.

The adaptation of CvT for use in soybean prediction is basically the
adaptation of the input and output of the model and does not require
structural changes in this adaption. In addition to the changed input
dimension, all augmentation methods must be disabled. The output is changed to
one class to represent a regression and its output value is learned using the
mean squared error as loss. The evaluation is then carried out using the mean
squared error (MSE), its root (RMSE) and the R² metric. The MSE and RMSE
should therefore be minimized in their role as error values, while the R²
metric should be maximized in its role as a coefficient of determination. All
data points from one year (the test year) are used for evaluation and all data
points from all previous years are used for training. In addition, to
calculate the validation loss and to find the resulting model with the best
performance (similar to an early stopping), 10% of the training data points
are randomly separated from the training dataset to form a validation dataset.

##  5 Results

###  5.1 Configurations of CvT

To evaluate the use of CvT for soybean yield prediction, the standardized
models according to Wu et al. (2021) are tested with the corresponding mostly
unchanged hyperparameters. The number of epochs is set to 150 for CvT-13, 200
for CvT-21 and 250 for CvT-W24. After this number of epochs, the loss hardly
shows any further changes. The learning rate is manually set to 0.00025, as
slightly different learning rates and lower learning rates of the larger
models do not result in any advantages. The evaluation is carried out using
the root mean squared error (RMSE) in bushels per acre averaged over the years
2018, 2019, 2020 and 2021. 90% of the remaining data points in years before
the respective test year are used for training and 10% for validation. To
counteract randomness, all results are averaged over 4 runs. The results can
be seen in Table 1. Performance improves slightly as the size of the model
increases. At the same time, the runtime increases considerably, meaning that
the trade-off between performance and runtime is expensive in terms of
runtime.

| CvT-13 | CvT-21 | CvT-W24
---|---|---|---
Year | RMSE | R² | RMSE | R² | RMSE | R²
2018 | 6.64 | 0.580 | 6.34 | 0.618 | 6.38 | 0.613
2019 | 5.88 | 0.517 | 5.96 | 0.503 | 5.92 | 0.510
2020 | 7.06 | 0.504 | 7.11 | 0.495 | 6.74 | 0.547
2021 | 6.61 | 0.747 | 6.14 | 0.783 | 6.35 | 0.767
AVG | 6.55 | 0.587 | 6.39 | 0.600 | 6.35 | 0.609
Table 1: Results of the different standard CvT models as RMSE in bushels per acre. | CvT-13 | CvT-13 KV-Stride 1 | CvT-21 | CvT-21 KV-Stride 1
---|---|---|---|---
Year | RMSE | R² | RMSE | R² | RMSE | R² | RMSE | R²
2018 | 6.64 | 0.580 | 6.30 | 0.622 | 6.34 | 0.618 | 6.67 | 0.577
2019 | 5.88 | 0.517 | 5.58 | 0.563 | 5.96 | 0.503 | 5.73 | 0.540
2020 | 7.06 | 0.504 | 7.07 | 0.502 | 7.11 | 0.495 | 7.00 | 0.512
2021 | 6.61 | 0.747 | 6.35 | 0.767 | 6.14 | 0.783 | 6.22 | 0.776
AVG | 6.55 | 0.587 | 6.33 | 0.613 | 6.39 | 0.600 | 6.41 | 0.601
Table 2: CvT-13 and CvT-21 with a smaller stride of the Convolutional
Projection of key and value (KV-Stride; originally it is 2). In this way,
advantages in efficiency are cancelled by a larger number of key and value
tokens.

Due to the use of CvT on histograms with a low resolution of $34\times 32$
pixels, the assumption regarding the smoothness of neighboring pixels must be
critically examined. This is used to reduce the computational costs in the
Convolutional Transformer Block. Key and value are convolutional projected
with a stride of 2, so that the following Multi-Head Attention receives a
quarter of the originally required tokens for key and value. Table 2 shows the
results of CvT with a stride of 1 of the convolutional projection of key and
value and therefore a 4 times more expensive multi-head self-attention. It can
be seen that this increases the performance of the small CvT-13 model, which
then achieves the best result with a lower runtime than its larger
competitors. This pattern does not carry over to CvT-21.

###  5.2 Comparison between XGBoost, CNN and CvT

The CvT-W24 model achieved the best performance of the standard configurations
tested. Table 3 compares the RMSE and the R² metric of the CNN according to
You et al. (2017) and the XGBoost method according to Huber et al. (2022). The
trivial application of CvT-W24 is subject to both XGBoost and CNN. CvT-W24 has
the longest runtime of the compared methods.

| XGBoost | CNN | CvT-W24 Plain
---|---|---|---
Year | RMSE | R² | RMSE | R² | RMSE | R²
2018 | 4.51 | 0.76 | 6.15 | 0.63 | 6.38 | 0.613
2019 | 4.21 | 0.76 | 5.52 | 0.57 | 5.92 | 0.510
2020 | 4.22 | 0.80 | 6.66 | 0.55 | 6.74 | 0.547
2021 | 4.55 | 0.82 | 5.12 | 0.85 | 6.35 | 0.767
AVG | 4.37 | 0.79 | 5.86 | 0.65 | 6.35 | 0.609
Table 3: Annual results of CvT-W24 in bushels per acre compared to CNN according to You et al. (2017) (without a gaussian process) and XGBoost according to Huber et al. (2022). The results are taken directly from Huber et al. (2022) and take the Daymet data into account. | XGBoost | CNN | CvT-13
---|---|---|---
Year | RMSE | R² | RMSE | R² | RMSE | R²
2018 | 5.27 | 0.68 | 7.30 | 0.50 | 7.92 | 0.40
2019 | 4.65 | 0.70 | 9.73 | -0.49 | 9.75 | -0.33
2020 | 5.35 | 0.65 | 8.24 | 0.32 | 8.82 | 0.22
2021 | 6.04 | 0.61 | 8.11 | 0.62 | 8.85 | 0.55
AVG | 5.33 | 0.66 | 8.35 | 0.24 | 8.83 | 0.21
Table 4: Annual results of CvT-13 in bushels per acre compared to CNN
according to You et al. (2017) (without a gaussian process) and XGBoost
according to Huber et al. (2022) for in-year prediction. The results are taken
directly from Huber et al. (2022) and take the Daymet data into account.

In order to test possible advantages of CvT in an earlier prediction, an in-
year prediction is also carried out. This contains a time span from the 49th
to the 201st day of each year and therefore ends before the harvest. The
horizontal histogram contains 19 instead of 34 8-day intervals. The results of
CvT-13 on this in-year prediction, as well as comparative results from XGBoost
and the CNN, can be seen in Table 4. The results show a similar situation to
the end-of-year results.

##  6 Conclusion

The Convolutional vision Transformer shows a lower performance than the
competing methods CNN and XGBoost. Even if it were possible to close the gap
to the CNN method by a very fine adjustment of the hyperparameters, this would
not lead to any advantage over the CNN and the performance would continue to
lag behind XGBoost. The higher performance of the CNN compared to the vision
Transformer CvT suggests that the local correlations are more important than
the global correlations in the histograms. As mentioned in chapter 2 the use
of local features of the CvT could be improved with larger training datasets
or large-scale pretraining, which is difficult due to the special input type
of low resolution histograms. Other methods such as the SWIN Transformer
mentioned in chapter 2 or transformers that additionally take into account
temporal modeling could also improve the performance of the Transformers for
yield prediction in further work. Furthermore, it is conceivable that the
processing of the comparatively small histograms is unsuitable for exploiting
the strengths of the Transformers, which lie particularly in long range
dependencies. Therefore, we see a potential for successful application of
Transformer in an improved pre-processing of the soybean field pixels for
tokenization.

## Acknowledgements

This work was partially done within the project “Artificial Intelligence for
innovative Yield Prediction of Grapevine” (KI-iRepro). The project is
supported by funds of the Federal Ministry of Food and Agriculture (BMEL)
based on a decision of the Parliament of the Federal Republic of Germany. The
Federal Office for Agriculture and Food (BLE) provides coordinating support
for artificial intelligence (AI) in agriculture as funding organisation, grant
number FKZ 28DK128B20.

## References

  * Aleissaee et al. [2023] Abdulaziz Amer Aleissaee, Amandeep Kumar, Rao Muhammad Anwer, Salman Khan, Hisham Cholakkal, Gui-Song Xia, and Fahad Shahbaz Khan.  Transformers in remote sensing: A survey.  _Remote Sensing_ , 15(7):1860, 2023.
  * Bi et al. [2023] Luning Bi, Owen Wally, Guiping Hu, Albert U Tenuta, Yuba R Kandel, and Daren S Mueller.  A transformer-based approach for early prediction of soybean yield using time-series images.  _Frontiers in Plant Science_ , 14:1173036, 2023.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Huber et al. [2022] Florian Huber, Artem Yushchenko, Benedikt Stratmann, and Volker Steinhage.  Extreme gradient boosting for yield estimation compared with deep learning approaches.  _Computers and Electronics in Agriculture_ , 202:107346, 2022.
  * Liu et al. [2022] Yuanyuan Liu, Shaoqiang Wang, Jinghua Chen, Bin Chen, Xiaobo Wang, Dongze Hao, and Leigang Sun.  Rice yield prediction and model interpretation based on satellite and climatic indicators using a transformer method.  _Remote Sensing_ , 14(19):5045, 2022.
  * Liu et al. [2021] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 10012–10022, 2021.
  * Muruganantham et al. [2022] Priyanga Muruganantham, Santoso Wibowo, Srimannarayana Grandhi, Nahidul Hoque Samrat, and Nahina Islam.  A systematic literature review on crop yield prediction with deep learning and remote sensing.  _Remote Sensing_ , 14(9):1990, 2022.
  * Raghu et al. [2021] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy.  Do vision transformers see like convolutional neural networks?  _Advances in Neural Information Processing Systems_ , 34:12116–12128, 2021.
  * Shooka et al. [2020] Johnathon Shooka, Tryambak Gangopadhyayb, Linjiang Wub, Baskar Ganapathysubramanianb, Soumik Sarkarb, and Asheesh K Singha.  Crop yield prediction integrating genotype and weather variables using deep learning.  _arXiv preprint arXiv:2006.13847_ , 2020.
  * Sun et al. [2019] Jie Sun, Liping Di, Ziheng Sun, Yonglin Shen, and Zulong Lai.  County-level soybean yield prediction using deep cnn-lstm model.  _Sensors_ , 19(20):4363, 2019.
  * Thornton et al. [2014] Peter E Thornton, Michele M Thornton, Benjamin W Mayer, Nate Wilhelmi, Yaxing Wei, Ranjeet Devarakonda, and Robert B Cook.  Daymet: Daily surface weather data on a 1-km grid for north america, version 2., 1 2014.  URL https://www.osti.gov/biblio/1148868.
  * U.C: [2018] Bureau U.C:.  Tiger: Us census counties 2018.  https://developers.google.com/earth-engine/data
sets/catalog/TIGER_2018_Counties, 2018.  Accessed 01 July 2021.

  * USDA [2021] USDA.  USDA nass quick stats database.  https://quickstats.nass.usda.gov/, 2021.  (Accessed 01 July 2021).
  * Van Klompenburg et al. [2020] Thomas Van Klompenburg, Ayalew Kassahun, and Cagatay Catal.  Crop yield prediction using machine learning: A systematic literature review.  _Computers and Electronics in Agriculture_ , 177:105709, 2020.
  * Vermote, E. [2021] Vermote, E.  MODIS/Terra Surface Reflectance 8-Day L3 Global 500m SIN Grid V061 [Data set]. NASA EOSDIS Land Processes DAAC.  https://doi.org/10.5067/MODIS/MOD09A1.061, 2021.  Accessed 01 July 2021.
  * Wan, Z., Hook, S., Hulley, G. [2021] Wan, Z., Hook, S., Hulley, G.  MODIS/Aqua Land Surface Temperature/Emissivity 8-Day L3 Global 1km SIN Grid V061 [Data set]. NASA EOSDIS Land Processes DAAC.  https://doi.org/10.5067/MODIS/MYD11A2.061, 2021.  Accessed 01 July 2021.
  * Wu et al. [2021] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 22–31, 2021.
  * You et al. [2017] Jiaxuan You, Xiaocheng Li, Melvin Low, David Lobell, and Stefano Ermon.  Deep gaussian process for crop yield prediction based on remote sensing data.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2017.