# Learning-based agricultural management in partially observable environments
subject to climate variability

Zhaoan Wang  Shaoping Xiao  Junchao Li  Jun Wang

(June 2023)

###### Abstract

Agricultural management, with a particular focus on fertilization strategies,
holds a central role in shaping crop yield, economic profitability, and
environmental sustainability. While conventional guidelines offer valuable
insights, their efficacy diminishes when confronted with extreme weather
conditions, such as heatwaves and droughts. In this study, we introduce an
innovative framework that integrates Deep Reinforcement Learning (DRL) with
Recurrent Neural Networks (RNNs). Leveraging the Gym-DSSAT simulator, we train
an intelligent agent to master optimal nitrogen fertilization management.
Through a series of simulation experiments conducted on corn crops in Iowa, we
compare Partially Observable Markov Decision Process (POMDP) models with
Markov Decision Process (MDP) models. Our research underscores the advantages
of utilizing sequential observations in developing more efficient nitrogen
input policies. Additionally, we explore the impact of climate variability,
particularly during extreme weather events, on agricultural outcomes and
management. Our findings demonstrate the adaptability of fertilization
policies to varying climate conditions. Notably, a fixed policy exhibits
resilience in the face of minor climate fluctuations, leading to commendable
corn yields, cost-effectiveness, and environmental conservation. However, our
study illuminates the need for agent retraining to acquire new optimal
policies under extreme weather events. This research charts a promising course
toward adaptable fertilization strategies that can seamlessly align with
dynamic climate scenarios, ultimately contributing to the optimization of crop
management practices.

###### keywords:

Fertilization Management , Reinforcement Learning , Recurrent Neural Networks
, Partially Observable Environments , Decision-making , Climate Variability

[inst1]organization=Department of Mechanical Engineering, Iowa Technology
Institute, University of Iowa,addressline=3131 Seamans Center, city=Iowa City,
postcode=52242, state=Iowa, country=USA [inst2]organization=Department of
Chemical and Biochemical Engineering, Iowa Technology Institute, University of
Iowa,addressline=4133 Seamans Center, city=Iowa City, postcode=52242,
state=Iowa, country=USA

##  1 Introduction

According to a 2022 report from the United States Department of Agriculture
(USDA) [1], total farm production nearly tripled from 1948 to 2017. However,
despite the growth, there remains a global food shortage. The Food and
Agriculture Organization (FAO) estimated that approximately 828 million people
were experiencing hunger in 2022. Given this pressing issue, it becomes
imperative to leverage new technologies to boost farm production, and one such
solution is Precision Agriculture (PA) [2]. Precision agriculture, also known
as “precision farming” or “prescription farming,” utilizes information and
technology-based agricultural management systems. These systems enable farmers
to precisely tailor their soil and crop management practices to various
weather/soil conditions on individual farmlands.

In a modern community, PA is an emerging field aimed at enhancing the
efficiency and sustainability of agricultural practices [3]. Precision
agriculture often employs advanced technologies such as remote sensing,
robotics, Machine Learning (ML), and Artificial Intelligence (AI) techniques.
Monitoring plant health and detecting diseases are vital aspects of
sustainable agriculture. Yet, manual disease detection is labor-intensive,
necessitating significant expertise, effort, and extended processing time.
Researchers have turned to image recognition algorithms as a solution,
achieving promising results in plant disease identification [4]. Additionally,
AI’s potential in forecasting crop yields has gained significant attention.
Some researchers have used satellite imagery to develop models that predict
yields, often incorporating crop identification maps and meteorological data.
These models have been applied to forecast yields for crops such as wheat,
rice, cotton, and sugarcane, especially in regions like the Indus Basin in
Pakistan, demonstrating satisfactory performance [5].

As one of the important components in PA, learning-based agricultural
management represents a substantial departure from traditional farming
methods, which often rely on human intuition and experience. Learning-based
agricultural management adopts a more data-driven approach [6] with the
overarching goals of increasing efficiency, reducing waste, protecting the
environment, and improving the sustainability of farming practices. A notable
example is seen in the work of Vij and co-authors [7], who predicted the
irrigation needs of farmland. They achieved this by using intelligent systems
to monitor ground parameters, including soil moisture, soil temperature, and
environmental conditions such as air temperature, ultraviolet rays, light
radiation, and the relative humidity of the fields. Additionally, they
incorporated weather forecast data sourced from the internet.

In previous studies of agricultural management, researchers traditionally
collected and analyzed historical data to identify empirical regularities,
which could then inform future agricultural policies and practices [8]. With
the continuous advancement of computer simulation technology, specialized
software tools such as Decision Support System for Agrotechnology Transfer
(DSSAT) [9], Agricultural Production Systems Simulator (APSIM) [10], and
AquaCrop [11] have been developed. These simulation tools are designed to
model various aspects of crop growth, yield, water, and nutrient requirements
in response to environmental conditions. Particularly, DSSAT has gained
widespread recognition and has been employed for over 30 years to simulate
crop behavior and responses to environmental variables, making it a valuable
resource for crop simulation studies.

The above-mentioned software tools can effectively approximate the growth
process of crops and predict the final yields by considering management
parameters and environmental conditions such as temperature, humidity, soil
property, and other influential factors. Among those factors, nitrogen
fertilizer stands out as a crucial element that can be managed and controlled.
Nitrogen is the primary nutrient that profoundly affects crop growth and yield
production. However, an excessive application of nitrogen fertilizer can lead
to substantial detrimental effects on the environment [12], including nitrate
leaching. Therefore, it becomes imperative to implement effective nitrogen
management strategies to balance optimizing crop yields, minimizing
environmental damage, and sustaining farmers’ income.

As one subset of ML, Reinforcement Learning (RL) empowers computer programs,
acting as agents, to control unknown and uncertain dynamical systems while
pursuing specific tasks [13, 14]. This approach has garnered increasing
attention from researchers interested in determining optimal strategies for
agricultural management [15]. Notably, the DSSAT, a widely recognized
agricultural simulation tool [9], has been extended to a realistic simulation
environment known as Gym-DSSAT [16]. In this environment, RL agents can
effectively learn fertilization and irrigation management strategies when
provided with soil property data and weather history/forecast information.
Specifically, Gautron et al. [16] proved that RL agents could discover
interesting crop management policies in simulated conditions and gym-DSSAT and
simulated worldwide growing conditions. In addition, Wu et al. [17] recently
demonstrated that the RL-trained policies outperformed empirical methods,
resulting in higher or similar crop yields while using fewer fertilizers. Sun
et al. [18] conducted research wherein they formulated a reinforcement
learning-driven irrigation control method. This technique has the potential to
substantially enhance net gains by accounting for both crop yield and water
expenditure.

The aforementioned works [16, 17, 18] predominantly assumed the agricultural
environment was fully observable, leading to the mathematical formulation of
corresponding RL problems as Markov Decision Process (MDP) problems. In MDP,
each state of the environment is expected to encompass all the necessary
information for the agent to determine the best action for optimizing the
objective function. However, questions arise regarding whether the state
variable listed in Gym-DSSAT can comprehensively represent the state of the
agricultural environment [19]. Moreover, certain state variables, such as the
index of plant water stress, daily nitrogen denitrification, and daily
nitrogen plant population uptake, may be challenging to measure and access.

This issue mirrors many real-world applications where agents lack complete
knowledge to determine the environment’s state precisely. In such cases,
agents often only have access to uncertain or incomplete observations of the
states. This challenge may be addressed by the Partially Observable Markov
Decision Process (POMDP) framework [20]. While POMDP was mentioned in the
context of Gym-DSSAT [16] during its introduction, the specific solution was
not detailed. Recently, Tao et al. [25] employed Imitation Learning (IL) to
develop management policies that require only a minimal number of state
variables by mirroring the actions of the RL policies learned with full
observation. They discovered that the policies, after being learned under
partial observation, demonstrated decisions almost identical to those trained
with RL under full observation.

On the other hand, climate variability is another critical factor in
agriculture and its management, encompassing changes in temperature,
precipitation, wind patterns, and other meteorological elements occurring over
various temporal and spatial scales [27]. Weather conditions, especially
extreme weather events, can significantly impact final crop yields. For
example, Motha and Baier [28] conducted a study analyzing the time series of
corn yields from 1895 to 2002 in the state of Iowa. They identified
substantial agricultural losses in 1988 due to one of the worst droughts
during the growing season in modern history. Additionally, flooding caused
almost a 50% drop in Iowa’s corn production in 1993 compared to the previous
year. Therefore, it is crucial for the learning agent of agricultural
management to be adaptive to climate variability.

This paper presents a framework for optimizing nitrogen fertilization while
considering the agricultural environment as partially observable.
Additionally, we investigate the impact of climate variability on nitrogen
fertilization and crop production. Our contributions are twofold.

First, our study demonstrates the effectiveness of formulating the
agricultural environment as a POMDP in generating superior policies (i.e.,
management strategies) compared to using an MDP, which has been the assumption
in most prior works [16, 17, 18]. This conclusion contrasts with the findings
of Tao et al. [25], where the agent was initially pre-trained in MDP and
subsequently in POMDP through imitation learning, resulting in similar policy
performance. Furthermore, our approach enables the agent to learn optimal
policies within the POMDP framework directly. Specifically, we employ a model-
free RL method that incorporates RNNs to solve POMDP problems.

Secondly, we investigate and quantify the influence of climate variability on
agricultural practices and crop production. We particularly emphasize two
extreme events: a heatwave in 1983 and a drought in 1988. These case studies
illustrate the adaptability of RL agents to learn optimal nitrogen
fertilization policies under extreme conditions. To the best of our knowledge,
no similar systematic investigations have been reported in the literature on
learning-based agricultural management.

The structure of this paper is organized as follows. In Section 2, we present
the formulations of MDP and POMDP and introduce Q learning, a model-free RL
method. Section 3 sets up and compares various MDP and POMDP models. In
Section 4, we delve into the impact of climate variability on crop yield and
nitrogen fertilizer usage, including the study of two extreme weather events.
Finally, we conclude the paper in Section 5 and outline avenues for future
research.

##  2 Methodology

In this study, we utilize an RL approach. During a learning process, as
illustrated in Figure 1, the agent interacts with the agricultural environment
by taking actions in agricultural management and receiving rewards as
feedback. The MDP, a mathematical framework that describes the environment and
its interaction with the agent, assumes the environment is fully observable.
Under this assumption, the agent can completely identify the environment’s
current state (i.e., configuration) and learn how to make optimal decisions
accordingly. However, in most real-world applications, the agent only receives
incomplete information, which cannot be used to identify the current state of
the environment. Therefore, the MDP is unsuitable, and the POMDP must be
adopted. This section will provide mathematical definitions for both MDP and
POMDP. Subsequently, we will introduce a model-free RL method called Q
learning and then extend it by incorporating RNNs to solve POMDP problems.

Figure 1: The reinforcement learning approach.

###  2.1 MDP and POMDP

######  Definition 2.1 (MDP).

An MDP can be generally denoted by a tuple
$\mathcal{P}=\left(S,A,T,s_{0},R\right)$, where:

  * 1.

$S=\\{s_{1},...,s_{n}\\}$ is a finite set of states.

  * 2.

$A=\\{a_{1},...,a_{m}\\}$ is a finite set of actions. In particular, $A(s)$
represents a set of actions that the learning agent can take at state $s$.

  * 3.

$T:S\times A\times S\rightarrow\left[0,1\right]$ is a function representing
the transition probability from state $s\in S$ to state $s^{\prime}\in S$
after the agent takes action $a\in A(s)$. It satisfies $\sum_{s^{\prime}\in
S}T(s,a,s^{\prime})=1$.

  * 4.

$s_{0}\in S$ is the initial state.

  * 5.

$R:S\times A\times S\rightarrow R$ is a reward function as
$R(s,a,s^{\prime})$. The reward function may have other formulations, such as
$R(s^{\prime})$ or $R(s,a)$.

When we use RL approaches to solving MDP problems, it is crucial to grasp how
an agent interacts with its environment. When the agent engages with the
environment, it makes decisions based on its knowledge of the current state,
denoted as $s$. Once the agent selects an action, represented as $a$, the
environment transitions to a new state, which we denote as $s^{\prime}$. This
transition occurs with a probability determined by the function
$T(s,a,s^{\prime})$. Simultaneously, the agent receives immediate feedback in
the form of a reward, denoted as $R(s,a,s^{\prime})$.

######  Definition 2.2 (POMDP).

A POMDP can be generally denoted by a tuple
$\mathcal{P}=\left(S,A,T,s_{0},R,O,\Omega\right)$, where $S,A,T,S_{0},$ and
$R$ are defined as the same in MDP (Definition 2.1), and

  * 1.

$O=\\{o_{1},...,o_{z}\\}$ is a finite set of observations. $O(s)$ is a set of
possible observations the agent can perceive at state $s$.

  * 2.

$\Omega:S\times A\times O\rightarrow\left[0,1\right]$ is a function
representing the observation probability that the agent can perceive at state
$s^{\prime}\in S$ after taking action $a\in A(s)$. This function satisfies
$\sum_{o\in O}\Omega(s^{\prime},a,o)=1$.

When the agent receives only partial information about the environment’s
current state, its decision-making relies on both past and current
observations. Once the agent takes a selected action and transitions to the
next state $s^{\prime}$, it perceives a new observation $o\in O(s^{\prime})$
with a probability described by $\Omega(s^{\prime},a,o)$. The primary
objective of an intelligent agent is to learn an optimal policy that maximizes
the expected return, as formulated below. This expected return represents the
cumulative rewards starting from the current state.

$U(s)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R(s_{t},a_{t},s_{t+1})\Big{|}s_{t=0}=s\right]$
(1)

where $s_{t}$ denotes the agent’s state at time $t$. $\gamma\in[0,1]$ is the
discount factor to balance the importance between immediate and future
rewards.

There have been several model-based approaches [13] to solving POMDP problems.
These approaches commonly seek optimal policies in the belief state space
rather than the state space defined in the POMDP (Definition 2.2). A belief
state is a probability distribution encompassing all the possible states where
the agent could be. It can be dynamically updated based on transition and
observation probabilities during the learning process. When using the model-
based approach, a POMDP problem transforms into a search for optimal policies
within a corresponding MDP defined in the belief state space. However, in this
study, the agent lacks knowledge of the transition and observation
probabilities, rendering model-free RL methods is an appropriate choice.

###  2.2 Q-learning and Deep Q-Network

The expected return in Equation (1) also defines the state value function,
denoted as $V(s)$, at state $s$. Similarly, there is another value function,
$Q(s,a)$, referred to as state-action value, action value, or $Q$ value. It
represents the total reward an agent can accumulate over the long run after
taking action $a$ at state $s$. In the realm of RL, there are two main
categories of methods: value-based and policy-based. Policy-based methods seek
optimal policies directly, while value-based RL methods focus on determining
optimal value functions. Subsequently, optimal policies can be derived through
greedy action selection.

Q-learning [26] is a model-free, value-based RL method in which the agent
tends to achieve optimal state-action values. As a tabular method, the naïve
Q-learning employs a Q-table to store $Q$ values and quantify the best action
with the highest $Q$ value for the agent to choose. On the other hand, $Q$
values in the Q-table are updated via bootstrapping when the agent interacts
with its environment. In each episode, $Q(s,a)$ is updated at each step as
below after taking action $a$ at state $s$, following the Bellman equation
[21].

$Q_{new}(s,a)=Q(s,a)+\alpha[R(s,a,s^{\prime})+\gamma\max_{a^{\prime}\in
A}Q(s^{\prime},a^{\prime})-Q(s,a)]$ (2)

where $\alpha$ is the learning rate, enhancing the efficiency and stability of
Q-value convergence.

Usually, the $\epsilon$-greedy technique is employed. This means that there is
a probability of $\epsilon$ for the agent to choose a non-optimal action,
allowing it to explore the state space, in addition to exploiting the current
policy. Once the optimal value function, $Q^{*}(s,a)$, is reached, the optimal
policy can be extracted as $\xi^{*}(s)=\arg\max_{a\in A}Q^{*}(s,a)$.

However, tabular Q-learning becomes unsuitable when dealing with a large or
infinite state space, such as agricultural environments. This challenge can be
addressed by replacing the Q-table with deep neural networks (DNNs), known as
Q-networks, to estimate $Q$ values. This approach falls under the umbrella of
Deep Reinforcement Learning (DRL) [22], and in this study, we employed the
Deep Q-Network (DQN) [23], which is an extension of Q-learning.

Deep Q-Network consists of two Q-networks: an evaluation Q-network, denoted as
$Q_{e}(s,a;\theta_{e})$, and a target Q-network, denoted as
$Q_{t}(s,a;\theta_{t})$. Here, $\theta_{e}$ and $\theta_{t}$ represent the
network weights, which are to be trained and updated through the experience
replay memory [24]. During the learning process, an experience is generated at
each step in the form of $(s,a,s^{\prime},R)$ and stored in a memory pool.
Simultaneously, a set of these experiences, referred to as a mini-batch, is
selected from the memory pool to train and update the evaluation Q-network.
The Bellman equation presented in Equation (2) is modified as follows.

$Q_{new}(s,a)=Q_{e}(s,a;\theta_{e})+\alpha\left[R(s,a,s^{\prime})+\gamma\max_{a^{\prime}\in
A}Q_{t}(s^{\prime},a^{\prime};\theta_{t})-Q_{e}(s,a;\theta_{e})\right]$ (3)

It is worth mentioning that the target Q-network is not trained by holding
fixed weights until copying from the evaluation Q-network, i.e.,
$\theta_{t}=\theta_{e}$, once in a while. Instead, this approach allows the
target network to update incrementally, known as a soft update. In a soft
update, the target network slowly tracks changes in the evaluation network,
which helps improve stability and convergence during training.

In MDP, the agent has a complete knowledge of the environment, and the value
function assesses available actions at the current state for decision-making.
However, when the environment becomes partially observable, the agent must
rely on a history of perceived information, typically a sequence of
observations, to make informed decisions. As a result, the Q-networks in DQN
take this sequence of observations as the input and produce corresponding $Q$
values. Furthermore, the policy or agent function now maps a sequence of
observations to an action. In such cases, RNNs emerge as strong candidates for
Q-networks because of their proficiency in handling sequential or time-series
data.

In a recent study by Li et al. [13], an RNN-based DQN was proposed for
robotics motion planning in partially observable environments. They utilized
Long Short-Term Memory (LSTM) [29] in their Q-networks to process sequences of
observations. In this study, we opt for Gated Recurrent Units (GRU) [30],
another advanced RNN architecture, within our Q-networks to model temporal
dependencies among observations, incorporating feedback loops within the
network structure. We conduct a comparison between LSTM-based DQN and GRU-
based DQN, and both approaches yield similar results. However, due to its
forget gate and the absence of an output gate, a GRU cell has fewer
parameters, which results in increased efficiency and reduced training time
compared to LSTM.

Figure 2: GRU-based Q-network architecture

The architecture of the GRU-based Q-network is depicted in Figure 2. The
sequence of observations, denoted as
$\mathbf{o}_{t}=(o_{t-j},o_{t-j+1},...,o_{t})$, has a length of $j+1$. Since Q
networks take the observation sequences as input, we redefine the evaluation
Q-network as $Q_{E}(\mathbf{o}_{t},a_{t};\theta_{E})$ and the target Q-network
as $Q_{T}(\mathbf{o}_{t},a_{t};\theta_{T})$. During the learning process,
after the agent reaches the next state, it receives a reward $R_{t}$ and
perceives a new observation $o_{t+1}$. A new observation sequence is then
generated as $\mathbf{o}_{t+1}=(o_{t-j+1},o_{t-j+2},...,o_{t+1})$.
Consequently, a new data sample (or experience), e.g.,
$(\mathbf{o}_{t},a_{t},R_{t},\mathbf{o}_{t+1})$, is formed and recorded in the
replay memory to update the evaluation Q-network. Furthermore, Equation (3)
can be expressed as

$Q_{new}(\mathbf{o}_{t},a_{t})=Q_{E}(\mathbf{o}_{t},a_{t};\theta_{E})+\alpha\left[R_{t}+\gamma\max_{a_{t+1}}Q_{T}(\mathbf{o}_{t+1},a_{t+1};\theta_{T})-Q_{E}(\mathbf{o}_{t},a_{t};\theta_{E})\right]$
(4)

After the learning finishes, the evaluation Q-network is converged, and the
optimal Q values can be estimated. Furthermore, the optimal policy can be
derived by

$\xi^{*}(\mathbf{o})=\arg\max_{a\in A}Q^{*}(\mathbf{o},a).$ (5)

##  3 Agriculture management as a POMDP problem

In this study, we utilize maize crop growth in Iowa as a case study to
demonstrate that the agricultural environment, represented by DSSAT, is
partially observable. Our study encompasses the years 1965, 1980, 1999, and
2020. We obtain the corresponding weather data from the Iowa State University
Soil Moisture Network [31], which includes daily maximum temperature, minimum
temperature, solar radiation, and precipitation. However, due to a lack of
comprehensive data, we rely on the soil property data from 1999 and apply it
to the other years in our study. The basic Gym-DSSAT input file from 1999
weather and soil data and the DRL code for this research can be found in our
GitHub repository. (https://github.com/ZhaoanWang/Learning-based-agricultural-
Management).

###  3.1 Model setup

The Gym-DSSAT employs factored representations, utilizing a total of 28
internal variables, as detailed in Table 1. Many studies [16, 17] have used
these variables to represent the agricultural environment’s state. They have
framed learning-based agricultural management problems as MDP problems,
assuming full observability of the environment. This assumption implies that
the environment possesses the Markov property, allowing the agent to make
decisions based on the immediately-received state variables. However, it is
important to note that there is no conclusive evidence to demonstrate that
these 28 internal variables can entirely determine the state of the
agricultural environment. Furthermore, not all of these variables are easily
observable or accessible. This study uses the first 10 variables from Table 1
as observation variables. This approach enables the agent to make decisions
based on both current and previous observations. Consequently, the original
problem is transformed into a POMDP problem.

Variable |  Description
---|---
cumsumfert |  cumulative nitrogen fertilizer applications (kg/ha)
dap |  days after planting
istage |  DSSAT maize growing stage
pltpop |  plant population density (plant/m2)
rain |  rainfall for the current day (mm/d)
sw |  volumetric soil water content in soil layers (cm3 [water] / cm3 [soil])
tmax |  maximum temperature for the current day (∘C)
tmin |  minimum temperature for the current day (∘C)
vstage |  vegetative growth stage (number of leaves)
xlai |  plant population leaf area index
cleach |  cumulative nitrate leaching (kg/ha)
cnox |  cumulative nitrogen denitrification (kg/ha)
dtt |  growing degree days for the current day (C/d)
es |  actual soil evaporation rate (mm/d)
grnwt |  grain weight dry matter (kg/ha)
nstres |  index of plant nitrogen stress
pcngrn |  massic fraction of nitrogen in grains
rtdep |  root depth (cm)
runoff |  calculated runoff (mm/d)
srad |  solar radiation during the current day (MJ/m2/d)
swfac |  index of plant water stress
tleachd |  daily nitrate leaching (kg/ha)
tnoxd |  daily nitrogen denitrification (kg/ha)
topwt |  above the ground population biomass (kg/ha)
totir |  total irrigated water (mm)
trun |  daily nitrogen plant population uptake (kg/ha)
wtdep |  depth to water table (cm)
wtnup |  cumulative plant population nitrogen uptake (kg/ha)
Table 1: Internal state variables of the agricultural environment.

This section explores four problems using different MDP and POMDP models. We
then compare corn yields and N fertilizer usages resulting from the optimal
policies ${-}$ i.e., management strategies learned by the agent ${-}$ in the
previously mentioned years. These problem types are as follows:

  * 1.

MDP-28: Markov decision process problems with all 28 internal variables as
state variables.

  * 2.

POMDP-28: Partially observable Markov decision process problems with all 28
internal variables as observation variables.

  * 3.

MDP-10: Markov decision process problems with the first 10 internal variables
as state variables.

  * 4.

POMDP-10: Partially observable Markov decision process problems with the first
10 internal variables as observation variables.

Given that maize crops in Iowa are typically rain-fed [17], this study
excludes daily irrigation considerations and concentrates on nitrogen
fertilization. Consequently, the action space encompasses various quantities
of nitrogen that can be applied in a single day. Mathematically, the action
space is discretized as ${10k(kg/ha)}$ nitrogen input, where $k$ ranges from 1
to 20.

In a given day ‘$d_{t}$,’ after taking action, which involves applying an
amount of nitrogen $N_{t}$, the agent receives a reward defined as:

$R(d_{t},N_{t})=\left\\{\begin{array}[]{cc}w_{1}Y-w_{2}N_{t}-w_{3}L_{t}&\mbox{at
harvest}\\\ -w_{2}N_{t}-w_{3}L_{t}&\mbox{otherwise}\end{array}\right.$ (6)

where $Y$ represents the corn yield at harvest, and $L_{t}$ denotes nitrate
leaching on a particular day $t$. The weight coefficients, $w_{1}$ and
$w_{2}$, are determined by the prevailing prices of corn and nitrogen input in
each simulated year, as listed in Table 2. Particularly, $w_{1}$ corresponds
to the price of corn per kilogram [32], and $w_{2}$ is based on the price of
nitrogen per kilogram, calculated using 45% urea nitrogen to determine the
price of 100% nitrogen [33]. In addition, $w_{3}$ is the weight assigned to
nitrate leaching, calculated as a multiple of $w_{2}$. The specific multiple
is 5, as indicated in [17].

Year |  w1 |  w2 |  w3
---|---|---|---
1965 |  0.03819 |  0.26 |  1.04
1980 |  0.07953 |  0.49 |  1.96
1999 |  0.07087 |  0.39 |  1.95
2020 |  0.1827 |  0.87 |  3.48
Table 2: Weight coefficients used in reward functions

When using DQN to solve MDP problems, as defined above, Q-networks are fully
connected networks that take state variables as the input and output $Q$
values for each action. The network architecture consists of 3 hidden layers
with 256 units in each layer, and the rectified linear activation function
(ReLU) is used. On the other hand, when solving POMDP problems, Q-networks
take a sequence of observations as input, and each observation is a vector of
observation variables. We test various sequence lengths and find that 5 time
steps (i.e., 5 days) are the proper length. The GRU layer in the Q-networks
has one hidden layer with 64 units, and its output is passed to a fully
connected network, which is the same as the one used in solving MDP problems
to calculate $Q$ values.

The training process includes 6000 episodes, each lasting 180 steps (i.e.,
days). We set the discount factor to 0.99. To update the neural networks, we
utilize Pytorch and Adam optimizer [34] with an initial learning rate of 1e-5
and a batch size of 640. The simulations are conducted on two machines: one
equipped with an Intel Core i7-12700K processor, NVIDIA GeForce RTX 3070 Ti
graphics card, and 64GB RAM, while the other featured an AMD 5800h processor,
NVIDIA GeForce RTX 3070 graphics card, and 32GB RAM.

###  3.2 Results and discussions

Table 3 displays the accumulated rewards each year, reflecting the performance
of optimal policies learned by the agent within various models. Notably, the
MDP-28 model surpasses the MDP-10 model, indicating that the inclusion of 28
state variables enhances decision-making by providing more information if the
agent can only access the current agricultural state (i.e., configuration). On
the other hand, all POMDP models outperform the MDP-28 model as hypothesized,
shedding light on the fact that the agricultural environment is only partially
observable through the internal state variables listed in Table 1. This
finding suggests that the agent benefits from leveraging a history of
observations to formulate better policies. Intriguingly, the optimal policies
derived from POMDP-28 and POMDP-10 exhibit striking similarity, implying that
employing 10 observation variables is well-informed for the agent’s decision-
making in the Gym-DSSAT environment.

Year |  MDP-28 |  POMDP-28 |  MDP-10 |  POMDP-10
---|---|---|---|---
1965 |  235 |  350 |  187 |  350
1980 |  594 |  612 |  460 |  612
1999 |  515 |  584 |  435 |  584
2020 |  1435 |  1471 |  1200 |  1466
Table 3: Accumulative rewards from optimal policies trained by different
models.

In Table 4, we compare the outcomes of 1999, which include corn yield,
nitrogen input, and nitrate leaching, obtained from the optimal policies
learned by the agent within different models. Surprisingly, all four optimal
policies achieve similar corn yields. However, notable differences emerge in
nitrogen usage and nitrate leaching. Policies learned within POMDP models
recommend significantly less nitrogen usage than those from MDP models. This
reduction in nitrogen input also results in less nitrate leaching,
contributing to higher accumulated rewards for the POMDP-induced policies than
their MDP counterparts.

Policy from |  Yield (kg/ha) |  Nitrogen input (kg/ha) |  Nitrate leaching (kg/ha)
---|---|---|---
MDP-28 |  9247 |  360 |  0.14
POMDP-28 |  9243 |  180 |  0.12
MDP-10 |  9226 |  560 |  0.20
POMDP-10 |  9243 |  180 |  0.12
Expert policy 1 |  6236 |  56 |  0.12
Expert policy 2 |  9247 |  224 |  0.26
Table 4: Outcomes of 1999 from various optimal policies, compared to the
expert policies, which respectively result in total rewards of 425 and 567.

Additionally, we provide the results from two expert policies offered by Gym-
DSSAT in Table 4. The first expert policy employs minimal nitrogen input and
consequently yields much less corn than other policies. In contrast, the
second expert policy utilizes a more substantial amount of nitrogen and yields
more corn than the first expert policy, aligning closely with the outcomes of
the learned optimal policies. However, this second expert policy also leads to
the highest level of nitrate leaching, although its nitrogen usage is less
than the optimal policies from MDP models.

Figure 3: Comparison of fertilizer managements based on different optimal
policies from MDP-10 and POMDP-10 models.

As seen in Figure 3, we can observe the nitrogen fertilization schedules and
quantities based on the optimal policies derived from MDP and POMDP models.
Notably, the MDP-10 policy applies nitrogen more frequently and in larger
amounts than the policy generated by the POMDP-10 model. This discrepancy
likely arises from the benefits of reduced nitrogen input in minimizing
nitrate leaching, while frequent applications effectively support corn growth.
Furthermore, a common trend emerges when we align nitrogen applications with
weather data, particularly focusing on precipitation: Both policies avoid
fertilizing on rainy days to prevent nitrate leaching effectively.

In accordance with a 1999 report titled ’Iowa Crop Performance Test- Corn
District 2,’ published by Iowa State University, corn yielded for various
brands ranged between 146 bu/acre and 192 bu/acre, with an average yield of
169.4 bu/acre. It is worth noting that these numbers represented yields of wet
corn. To make accurate comparisons with our simulation results, these wet
yields must be converted to dry corn yields. The report specified an average
moisture content of 17.5% for the wet corn. In our conversion process, we use
a moisture content of 14% for dry corn, as referenced in a study by Yakoub et
al. [35]. After converting to dry yield, the range of corn yield in 1999 spans
from 8507.53 kg/ha to 11197.74 kg/ha, with an average yield of 9868.59 kg/ha.

Our simulation results, averaging approximately 9243 kg/ha from optimal
policies in Table 4, exhibit a deviation of about 6.4% from the actual yield.
This aligns with variations observed in some prior studies (e.g., 13% and
9.5%) that compared the DSSAT simulation results with the actual crop
productions [35, 36]. It is worth noting that the DSSAT maize models,
specifically CERES, were initially published 37 years ago [37]. As such, they
may not perfectly capture modern maize production methods, which have evolved
substantially over the years. Nonetheless, the optimal policies derived from
POMDP models recommend reduced nitrogen usage (as shown in Table 4) compared
to the second expert policy, which we will use as a baseline for our
subsequent studies in this paper.

In order to assess the influence of nitrate leaching on both nitrogen
fertilization practices and corn yield, we conduct a series of simulations to
obtain optimal policies from the POMDP-10 model involving different values of
$w_{3}$, ranging from 0 to 50 times the value of $w_{2}$. Throughout the
comparisons, the overall quantity of nitrogen usage remains constant, while
the timing of fertilization applications is adjusted according to the
specified $w_{3}$ values. Upon closer examination, we find that assigning a
higher weight to nitrate leaching in the reward function (as defined in
Equation (6)) leads to a reduction in nitrate leaching. This reduction is
achieved by minimizing fertilization on rainy days whenever possible. For the
subsequent studies detailed in this paper, we choose to maintain $w_{3}$ five
times of $w_{2}$ in the reward function. This decision aligns with the
approach used in a previously referenced study [17].

##  4 Impact of climate variability on agriculture management

In this section, we utilize weather data from 1999 as a baseline.
Subsequently, we introduce variations in temperature and precipitation to
analyze the influence of climate variability. In addition, we delve into the
agricultural ramifications of specific extreme events, notably the 1983
heatwave and the 1988 drought. We also examine how optimal policies of
fertilization management adapt in response to these events. It is important to
note that the soil data used in all simulations remains consistent with the
conditions observed in 1999.

###  4.1 Impact of higher temperature

Over the past 70 years, the global climate patterns have undergone significant
changes primarily driven by anthropogenic activities, with temperature
increases being one of the most prominent aspects [38]. Historical data
reveals a steady rise in average temperatures, within a 0.98-degree Celsius
increase since 1880. Notably, this warming trend has accelerated in recent
decades, with a 0.94-degree Celsius rise recorded in the last 60 years alone.
In this study, we systematically elevated the daily average temperature from
the 1999 baseline by increments of 0.5, 1, 2, and 5 degrees Celsius [39]
throughout the year while following a consistent pattern. The precipitation
remains the same as in 1999, so we can investigate the impact of temperature
variation on fertilization management and corn yield.

In this section, we examine two categories of optimal policies. The first
category is the ‘1999 policy,’ which replicates the optimal policy learned by
the agent in the POMDP-10 model under the actual weather data from 1999, as
outlined in Table 4. The ‘1999 policy’ remains unchanged even when subjected
to ‘hotter’ weather conditions, allowing us to assess its adaptability to
elevated temperatures. It’s important to note that this policy will generate
different fertilizer management plans under varying weather conditions. The
second category of policy consists of ‘optimal policies’ that the agent re-
learns in response to elevated temperatures. In addition, we will investigate
Expert policy 2 as detailed in Table 4.

Table 5 presents the agricultural outcomes, including corn yield, nitrogen
input, and nitrate leaching, based on different fertilization management
policies. The table illustrates that a 0.5-degree Celsius increase in
temperature corresponds to a boost in corn yield and a slight uptick in
fertilizer consumption (by the ‘optimal policy’). As temperatures continue to
rise, corn yields also increase, while fertilizer usage begins to drop
slightly. This trend is likely due to the temperature approaching the optimal
growth range for corn as it escalates.

temperature increment | 1999 policy | Optimal policies | Expert policy
---|---|---|---
$+0^{\circ}$C |  |  |
Yield (kg/ha) | 9243 | 9243 | 9247
Nitrogen input (kg/ha) | 180 | 180 | 224
Nitrate leaching (kg/ha) | 0.12 | 0.12 | 0.26
$+0.5^{\circ}$C |  |  |
Yield (kg/ha) | 9784 | 10295 | 10303
Nitrogen input (kg/ha) | 180 | 190 | 224
Nitrate leaching (kg/ha) | 0.11 | 0.10 | 0.28
$+1^{\circ}$C |  |  |
Yield (kg/ha) | 10416 | 10425 | 10426
Nitrogen input (kg/ha) | 170 | 160 | 224
Nitrate leaching (kg/ha) | 0.10 | 0.10 | 0.29
$+2^{\circ}$C |  |  |
Yield (kg/ha) | 9352 | 9357 | 9337
Nitrogen input (kg/ha) | 140 | 120 | 224
Nitrate leaching (kg/ha) | 0.09 | 0.09 | 0.29
$+5^{\circ}$C |  |  |
Yield (kg/ha) | 4901 | 4873 | 4901
Nitrogen input (kg/ha) | 250 | 60 | 224
Nitrate leaching (kg/ha) | 0.08 | 0.07 | 0.27
Table 5: Comparison of different policies when temperature increases.

Our simulations run from April 10th to October 30th, with an average air
temperature of 10.9 degrees Celsius in 1999 on the planting day (May 27th),
aligning with the recommended corn planting temperature of above 10 degrees
Celsius by experts [40]. However, when temperatures surge by more than 2.5
degrees, corn yield starts to decline, possibly indicating that the heat
becomes too intense for healthy corn growth. This hypothesis gains further
support when the temperature is raised by 5 degrees Celsius, resulting in a
significant drop in corn yield to just half its original volume under the
actual weather conditions of 1999.

While examining the data presented in Table 5, it becomes evident that various
policies exhibit similar trends in corn yield as daily temperature
systematically increases. However, notable differences emerge in nitrogen
input when corn yield starts to decline significantly due to substantial
temperature escalation. The ‘optimal policies,’ which consistently employ
significantly less nitrogen than the ‘1999 policy,’ lead to higher rewards.
Specifically, when the temperature increases by 5 degrees, the ’optimal
policy’ yields a reward (representing the net income) 30% higher than the
‘1999 policy.’ In contrast, the expert policy maintains the same fertilizer
usage but results in substantially higher nitrate leaching compared to other
policies.

These results underscore the adaptability of the fixed policy to small
temperature escalation, while also highlighting the need of the agent to
update the optimal policy under extreme temperature conditions. This finding
has important implications for PA in response to climate change in the future.
It suggests that as temperatures continue to rise, there will be a growing
need to develop and implement more dynamic and responsive agricultural
management policies to maximize yields and minimize environmental impacts.

###  4.2 Impact of insufficient precipitation

We also examine the influence of rainfall on corn yield and fertilization
management. After analyzing the historical rainfall data since 1950, we find
no consistent linear pattern in yearly rainfall. For this study, we still use
the actual weather conditions of 1999 as a reference and reduce daily rainfall
by 20%, 35%, 50%, 65%, and 80% throughout the year while maintaining a
consistent pattern. Temperatures remain the same as in 1999. Notably, we
choose not to increase precipitation, as doing so might introduce the risk of
flood damage to the crops, which falls beyond the predictive capabilities of
DSSAT.

Precipitation reduction | 1999 policy | Optimal policies | Expert policy
---|---|---|---
0% |  |  |
Yield (kg/ha) | 9243 | 9243 | 9247
Nitrogen input (kg/ha) | 180 | 180 | 224
Nitrate leaching (kg/ha) | 0.12 | 0.12 | 0.26
20% |  |  |
Yield (kg/ha) | 8652 | 8930 | 9108
Nitrogen input (kg/ha) | 180 | 160 | 224
Nitrate leaching (kg/ha) | 0.006 | 0.008 | 0.115
35% |  |  |
Yield (kg/ha) | 8192 | 8408 | 8808
Nitrogen input (kg/ha) | 180 | 160 | 224
Nitrate leaching (kg/ha) | 0.006 | 0.008 | 0.019
50% |  |  |
Yield (kg/ha) | 7164 | 7350 | 7604
Nitrogen input (kg/ha) | 180 | 130 | 224
Nitrate leaching (kg/ha) | 0.0006 | 0.001 | 0.009
65% |  |  |
Yield (kg/ha) | 4756 | 5658 | 5587
Nitrogen input (kg/ha) | 180 | 120 | 224
Nitrate leaching (kg/ha) | 0.0005 | 0.0006 | 0.0008
80% |  |  |
Yield (kg/ha) | 2406 | 4360 | 4025
Nitrogen input (kg/ha) | 180 | 100 | 224
Nitrate leaching (kg/ha) | 0.0005 | 0.0005 | 0.0007
Table 6: Comparison of different policies when precipitation decreases.

Table 6 presents the simulated outcomes under varying precipitation scenarios.
Similar to Table 5, this table compares corn yield, nitrogen input, and
nitrate leaching between various policies, including the ‘1999 policy,’
optimal policies, and an expert policy. It can be seen that when precipitation
decreases, the corn yield is significantly impacted. The corn yield can be
less than half the standard value when the weather is severely dry. Given that
corn is a moisture-intensive crop [41], the results are convincing.

In addition, while adhering to the ‘1999 policy,’ fertilization practices
remain unchanged. However, the ‘optimal policies’ utilize less fertilizer due
to reduced precipitation, which doesn’t significantly impact nitrate leaching.
We also include the results from the expert policy. In line with the earlier
simulation results regarding temperature variability, both the optimal
policies and the expert policy yield similar corn productions. The primary
distinction between the two policies is the quantity of nitrogen applied.

Overall, Table 6 also illustrates that optimal policies consistently
outperform the ‘1999 policy’ in corn yield, especially in conditions with
significantly low rainfall. This finding aligns with our conclusion in the
study of the impact of higher temperatures on agriculture and agricultural
management. The optimal policy learned under typical weather conditions
demonstrates adaptability when faced with minor precipitation fluctuations.
However, in the case of a significant reduction in precipitation, the agent
must acquire a new optimal policy. When comparing Table 6 with Table 5, it
becomes evident that reductions in precipitation have a more pronounced impact
than temperature increases. This highlights the crucial role that humidity
levels play in corn cultivation. It is important to note that this study
doesn’t consider irrigation as a factor.

###  4.3 Heat wave and drought

To further study the impact of extreme weather events on agriculture and
fertilization management, we consider two real scenarios that occurred in
Iowa: the heat wave in 1983 and the drought in 1988. Previous research [28]
indicated that these extreme weather events led to 32% and 38.5% reduction in
Iowa’s corn yields compared to the previous years.

To accurately simulate these events, we source data from the Iowa
Environmental Mesonet (IEM) for daily maximum and minimum temperatures, as
well as precipitations, for the years 1982, 1983, 1987, and 1988.
Additionally, we gather information on planting and harvesting dates from the
’THE 1983 IOWA CORN YIELD TEST REPORT District 2’ and ’Iowa Corn Yield Test
Report (Iowa State University) District 1 December 1988.’ According to the
reports, corn was planted on May 7th & 8th and harvested on October 20th &
21st in 1983, and on May 3rd and harvested on October 4th & 5th in 1988.

Figure 4: Monthly average temperatures from May to October in 1982 and 1983.

Figure 5: Monthly total precipitations from May to October in 1982 and 1983.

Figures 4 and 5 depict the comparisons of monthly average temperatures and
precipitations for the months of April through October in 1982 and 1983. It
can be seen that the average temperatures in June, July, August, and September
of 1983 were higher than in 1982 by 3.1 degrees, 1.7 degrees, 4.3 degrees, and
1.5 degrees, respectively. However, the overall precipitation in 1983 was
higher than in 1982, especially in June 1983. Therefore, there was a heatwave
in 1983 but not a drought.

Figure 6: Monthly average temperatures from May to October in 1987 and 1988.

Figure 7: Monthly total precipitation from May to October in 1987 and 1988.

Figure 6 and Figure 7 compare monthly average temperatures and precipitation
between 1987 and 1988. The average temperature in August of 1988 was
significantly higher than in 1987, with a difference of 6.3 degrees, larger
than those in August of 1982 and 1983. However, the precipitation in August
was considerably lower in 1988 compared to 1987, representing approximately a
400% reduction in rainfall. Although there were more rains in July and
September of 1988 compared to 1987, the severe reduction in precipitation
during August, a crucial corn growth stage, indicates that Iowa experienced a
drought in 1988, emphasizing the significance of this dry period more than the
heatwave.

Our simulations utilize the actual weather data of 1982, 1983, 1987, and 1988
while maintaining the soil data consistent with 1999. To compare corn yield
and nitrogen input between 1982 and 1983, the agent first learns an optimal
policy for 1982, which is then applied to 1983. This is compared to the
optimal policy that the agent learns specifically for 1983. We follow the same
procedure for the comparison between 1987 and 1988. The results of these
comparisons are presented in Table 7.

Year | Policy | Corn yield (kg/ha) | Nitrogen input (kg/ha)
---|---|---|---
1982 | Optimal policy | 10923 | 270
1983 | 1982 policy | 7318 | 320
1983 | Optimal policy | 8098 | 110
1987 | Optimal policy | 9963 | 180
1988 | 1987 policy | 3820 | 200
1988 | Optimal policy | 4344 | 130
Table 7: Comparison of simulation results for 1982 and 1983 & 1987 and 1988

An optimal policy of fertilization management is learned for 1982, and its
effects on corn yield and nitrogen input are documented in Table 7. However,
when the same policy is applied in 1983, we observed a drop of 33% in
production due to the heatwave. This decline closely mirrors historical data
from the USDA for Iowa, which indicates a 32% decrease in corn production in
1983 compared to the previous year (1982). This suggests that the
fertilization strategy developed in 1982 may not have been well-suited to
handle the extreme weather event, i.e., heatwave, in 1983.

Interestingly, despite a significant decrease in corn yield, nitrogen input
increases by 18.5% in 1983 compared to 1982 when using the ‘1982 policy.’ This
suggests that the fertilization application is not adjusted to match the
unique conditions of 1983, potentially leading to an excessive use of
nitrogen. The agent learns a separate optimal policy specifically for 1983 to
address these challenges. The induced fertilization strategy results in a
higher corn yield by 10% while significantly reducing nitrogen input.

A similar pattern is observed when assessing the impact of the 1988 drought on
agricultural outcomes. When the optimal policy learned for 1987 is applied in
1988, it results in a significant reduction in corn production, amounting to a
61.5% decrease, as depicted in Table 7. However, when the optimal policy
specifically tailored by the agent for 1988 is employed, the production
reduction was slightly less severe at 56%, but it came with a substantial
reduction in nitrogen input.

It is important to note that the simulated reduction in corn yield, as seen in
the study, greatly exceeded the actual drop, which is 38.5%. This discrepancy
can be attributed to the fact that this study doesn’t account for irrigation
in the agricultural management strategy. Corn has a high water requirement and
is particularly sensitive to drought, so the omission of irrigation likely
contributed to the larger simulated yield reduction.

##  5 Conclusion and future works

Optimizing crop management strategies is essential for maximizing yield,
reducing costs, and mitigating environmental impacts. In this study, we
introduced a framework that combines DRL with RNNs, utilizing Gym-DSSAT to
determine optimal nitrogen fertilization strategies. Our findings reveal that
the agricultural environment, as represented by Gym-DSSAT, is partially
observable. This differs from the assumptions made in previous studies, where
the state of the agricultural environment was assumed to be entirely
determined from the currently observed internal variables provided by Gym-
DSSAT. To address this challenge, we compared POMDP models to MDP models. Our
results indicated that leveraging a sequence of observations allows the agent
to learn and implement more effective policies for nitrogen fertilization
management.

We also applied our developed framework to assess the impacts of climate
variability on agricultural outcomes and management, with a particular focus
on scenarios involving higher temperatures and inadequate precipitation. We
found that the pre-learned optimal policy proves adaptable under minor climate
variability but falls short in extreme weather conditions. This study
underscores the critical importance of tailoring fertilization management
practices to the specific weather conditions of each year, especially in the
face of extreme weather events. Such adaptability is essential for optimizing
crop yield while simultaneously minimizing nitrogen input. It recognizes the
need for agriculture to maintain flexibility in response to the variable and,
at times, extreme influences of weather and other factors on crop performance.

Due to data limitations, the simulations presented in this paper rely solely
on the 1999 dataset, particularly for soil properties. Going forward, we aim
to compile a comprehensive historical soil dataset for the relevant
agricultural lands. With this enriched dataset, we aspire to conduct more
representative and accurate simulations, thereby enhancing our findings’
precision and reliability.

While this paper primarily focused on nitrogen fertilization, our future
research will incorporate irrigation management, particularly in addressing
severe drought events. Furthermore, we intend to gather comprehensive cost
data for the relevant year, encompassing machine, labor, and other expenses.
Integrating these variables into our reward function will enable our model to
replicate farmers’ net incomes more accurately.

When studying the impact of temperature and precipitation variability on
agriculture, we maintain the same patterns consistent with those observed in
1999. We acknowledge the limitation of solely relying on historical data for
simulations. In subsequent research, we plan to generate random weather
scenarios based on real data to introduce weather uncertainty and perform
uncertainty quantification of agricultural management and outcomes.

## CRediT authorship contribution statement

Zhaoan Wang: Conceptualization, Data curation, Investigation, Methodology,
Writing – original draft. Shaoping Xiao: Conceptualization, Methodology,
Supervision, Funding acquisition, Writing - review & editing. Junchao Li:
Methodology, Writing - review & editing. Jun Wang: Conceptualization, Funding
acquisition, Writing - review & editing.

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that could have appeared to influence the work reported
in this paper.

## Data availability

The data used in this study are freely available. Some data files and codes
are available on the GitHub site (https://github.com /ZhaoanWang /Learning-
based-agricultural-Management). You can contact the corresponding author, Mr.
Zhaoan Wang<EMAIL_ADDRESS>if you want to use the data.

## Acknowledgments

The authors received support from the University of Iowa OVPR
Interdisciplinary Scholars Program for this study. Z.W., J.L., and S.X.
received supports from the U.S. Department of Education (E.D. #P116S210005)
and the National Science Foundation (#2226936).

## Declaration of Generative AI and AI-assisted technologies in the writing
process

During the preparation of this work, the authors used GPT-3.5 in order to
improve the readability and language during the writing process. After using
this tool/service, the authors reviewed and edited the content as needed and
took full responsibility for the content of the publication.

## References

  * [1] Wang, S.L., Robert, A., Hertz, T., Xu, S., 2022. Farm Labor, Human Capital, and Agricultural Productivity in the United States. U.S. Department of Agriculture.
  * [2] Zhang, N., Wang, M., Wang, N., 2002. Precision agriculture—a worldwide overview. Comput. Electron. Agric. 36(2-3), 113-132. https://doi.org/10.1016/S0168-1699(02)00096-0
  * [3] Duru, M., Therond, O., Martin, G. et al., 2015. How to implement biodiversity-based agriculture to enhance ecosystem services: a review. Agron. Sustain. Dev. 35, 1259–1281. https://doi.org/10.1007/s13593-015-0306-1
  * [4] Khirade S D, Patil A B., 2015. Plant disease detection using image processing. 2015 International Conference on Computing Communication Control and Automation, Pune, India, pp. 768-771. doi: 10.1109/ICCUBEA.2015.153.
  * [5] Bastiaanssen, W. G., Ali, S., 2003. A new crop yield forecasting model based on satellite measurements applied across the Indus Basin, Pakistan. Agriculture, ecosystems & environment,94(3): 321-340. https://doi.org/10.1016/S0167-8809(02)00034-8
  * [6] Jha, K., Doshi, A., Patel, P., Shah, M., 2019, A comprehensive review on automation in agriculture using artificial intelligence. Artificial Intelligence in Agriculture, 2: 1-12. https://doi.org/10.1016/j.aiia.2019.05.004
  * [7] Vij A, Vijendra S, Jain A, et al., 2020. IoT and machine learning approaches for automation of farm irrigation system. Procedia Computer Science, 167: 1250 1257. https://doi.org/10.1016/j.procs.2020.03.440
  * [8] Cassman, K. G., Dobermann, A., Walters, D. T.,2002. Walters. Agroecosystems, Nitrogen-use Efficiency, and Nitrogen Management. AMBIO: A Journal of the Human Environment, 31(2): 132-140. https://doi.org/10.1579/0044-7447-31.2.132
  * [9] Jones, J. W., Hoogenboom, G., Porter, C. H., et al., 2003. The DSSAT cropping system model. European journal of agronomy, 18(3-4), 235-265. https://doi.org/10.1016/S1161-0301(02)00107-7
  * [10] Keating B A, Carberry P S, Hammer G L, et al., 2003. An overview of APSIM, a model designed for farming systems simulation. European journal of agronomy, 18(3-4): 267-288. https://doi.org/10.1016/S1161-0301(02)00108-9
  * [11] Steduto P, Hsiao T C, Raes D, et al., 2009. AquaCrop—The FAO crop model to simulate yield response to water: I. Concepts and underlying principles. Agronomy Journal, 101(3): 426-437. https://doi.org/10.2134/agronj2008.0139s
  * [12] Sutton, M. A., Oenema, O., Erisman, J. W., Leip, A., van Grinsven, H., Winiwarter, W. 2011. Too much of a good thing. Nature, 472(7342): 159-161. https://doi.org/10.1038/472159a
  * [13] Li, J. C., Cai, M., Wang, Z. A., and Xiao, S. P., 2023. Model-based motion planning in POMDPs with temporal logic specifications. Advanced Robotics, 37(14), 871-886. https://doi.org/10.1080/01691864.2023.2226191
  * [14] Cai, M., Xiao, S. P., Li, J.C., and Kan, Z., 2023. Safe reinforcement learning under temporal logic with reward design and quantum action selection. Scientific Reports, 13, 1925. https://doi.org/10.1038/s41598-023-28582-4
  * [15] Overweg, H., Berghuijs, H. N., Athanasiadis, I. N., 2021. CropGym: a reinforcement learning environment for crop management. arXiv preprint arXiv:2104.04326. https://doi.org/10.48550/arXiv.2104.04326
  * [16] Gautron, R., Padrón, E. J., Preux, P., Bigot, J., Maillard, O. A., Emukpere, D., 2022. gym-DSSAT: a crop model turned into a Reinforcement Learning environment. Research Report RR-9460, Inria Lille. pp.31. ffhal-03711132v4f. https://doi.org/10.48550/arXiv.2207.03270
  * [17] Wu Y., Zhang Y., Zhang C., Castro da Silva, B., 2022. Optimizing Nitrogen Management With Deep Reinforcement Learning and Crop Simulations. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1712-1720. https://doi.org/10.48550/arXiv.2204.10394
  * [18] Sun L, Yang Y, Hu J, et al., 2017. Reinforcement learning control for water-efficient agricultural irrigation. 2017 IEEE International Symposium on Parallel and Distributed Processing with Applications and 2017 IEEE International Conference on Ubiquitous Computing and Communications (ISPA/IUCC), Guangzhou, China 1334-1341. doi: 10.1109/ISPA/IUCC.2017.00203.
  * [19] Puterman M L., 2014. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons
  * [20] Astrom K J., 1965. Optimal control of Markov processes with incomplete state information. Journal of mathematical analysis and applications, 10(1): 174-205.DOI:10.1016/0022-247X(65)90154-X
  * [21] Sutton, R. S., & Barto, A. G., 2018. Reinforcement learning: An introduction. MIT press.
  * [22] Arulkumaran K, Deisenroth M P, Brundage M, et al., 2017. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6): 26-38. doi: 10.1109/MSP.2017.2743240.
  * [23] Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M., 2013. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. https://doi.org/10.48550/arXiv.1312.5602
  * [24] Lin L J., 2002. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8: 293-321. DOI: 10.1007/BF00992699
  * [25] Tao R, Zhao P, Wu J, et al., 2022. Optimizing crop management with reinforcement learning and imitation learning. arXiv preprint arXiv:2209.09991. https://doi.org/10.48550/arXiv.2209.09991
  * [26] Watkins, C. J., Dayan, P., 1992. Q-learning. Machine learning, 8, 279-292. https://doi.org/10.1007/BF00992698
  * [27] Pachauri, R. K., Allen, M. R., Barros, V. R., Broome, J., Cramer, W., Christ, R., et al. (2015). Climate change 2014: synthesis report. Contribution of Working Groups I, II and III to the fifth assessment report of the Intergovernmental Panel on Climate Change. Ipcc.
  * [28] Motha R P, Baier W., 2005. Impacts of present and future climate change and climate variability on agriculture in the temperate regions: North America. Climatic Change, 70(1-2): 137-164. https://doi.org/10.1007/s10584-005-5940-1
  * [29] Hochreiter S, Schmidhuber J., 1997. Long short-term memory. Neural computation, 9(8): 1735-1780. doi: 10.1162/neco.1997.9.8.1735.
  * [30] K. Cho et al., 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 1724–1734. doi: 10.3115/v1/d14-1179.
  * [31] The Iowa Environmental Mesonet-ISU Soil Moisture Network. https://mesonet.agron.iastate.edu/agclimate/#tmpf
  * [32] [dataset]USDA. All fertilizer use and price tables in one workbook. https://www.ers.usda.gov/webdocs/DataFiles/50341/fertilizeruse.xls?v=0
  * [33] [dataset]ISU. Iowa Cash Corn and Soybean Prices. https://www.extension.iastate.edu/agdm/crops/pdf/a2-11.pdf
  * [34] Kingma D P, Ba J., 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. https://doi.org/10.48550/arXiv.1412.6980
  * [35] Yakoub, A., Lloveras, J., Biau, A., Lindquist, J. L., Lizaso, J. I., 2017. Testing and improving the maize models in DSSAT: Development, growth, yield, and N uptake. Field Crops Research, 212: 95-106. https://doi.org/10.1016/j.fcr.2017.07.002
  * [36] Nouna B B, Katerji N, Mastrorilli M., 2000. Using the CERES-Maize model in a semi-arid Mediterranean environment. Evaluation of model performance. European Journal of Agronomy, 13(4): 309-322. https://doi.org/10.1016/S1161-0301(00)00063-0
  * [37] Jones, C.A., and J.R. Kiniry (Eds.)., 1986. CERES-Maize: A simulation model of maize growth and development. Texas A&M Univ. Press, College Station.
  * [38] Rabatel A, Francou B, Soruco A, et al., 2013. Current state of glaciers in the tropical Andes: a multi-century perspective on glacier evolution and climate change. The Cryosphere, 2013, 7(1): 81-102. https://doi.org/10.5194/tc-7-81-2013
  * [39] NASA. Global climate change. https://climate.nasa.gov/
  * [40] Abendroth L J, Woli K P, Myers A J W, et al., 2017. Yield‐based corn planting date recommendation windows for Iowa. Crop, Forage & Turfgrass Management, 3(1): 1-7. https://doi.org/10.2134/cftm2017.02.0015
  * [41] Di Paolo, E., Rinaldi, M., 2008. Yield response of corn to irrigation and nitrogen fertilization in a Mediterranean environment. Field Crops Research, 105(3), 202-210. https://doi.org/10.1016/j.fcr.2007.10.004