# Why are Mountaintops Cold? The Transition of Surface Lapse Rate on Dry
Planets

###### Abstract

Understanding surface temperature is important for habitability. Recent work
on Mars has found that the dependence of surface temperature on elevation
(surface lapse rate) converges to zero in the limit of a thin
$\mathrm{CO_{2}}$ atmosphere. However, the mechanisms that control the surface
lapse rate are still not fully understood. It remains unclear how the surface
lapse rate depends on both greenhouse effect and surface pressure. Here, we
use climate models to study when and why “mountaintops are cold”. We find the
tropical surface lapse rate increases with the greenhouse effect and with
surface pressure. The greenhouse effect dominates the surface lapse rate
transition and is robust across latitudes. The pressure effect is important at
low latitudes in moderately opaque ($\tau\sim 0.1$) atmospheres. A simple
model provides insights into the mechanisms of the transition. Our results
suggest that topographic cold-trapping may be important for the climate of
arid planets.

Geophysical Research Letters

Department of the Geophysical Sciences, University of Chicago, Chicago, IL,
60637, USA Jet Propulsion Laboratory, California Institute of Technology,
Pasadena, CA, 91109, USA

Bowen<EMAIL_ADDRESS>
Surface lapse rate robustly increases with atmospheric longwave optical
thickness (greenhouse effect) in a general circulation model.

Increased pressure further contributes to a tropical surface lapse rate
increase in moderately opaque atmospheres.

A simple model, assuming weak temperature gradient and highland convective
adjustment, provides insight into the mechanisms.

## Plain Language Summary

Understanding surface temperature on a planet is important for life on Earth
and beyond. On Earth, we know “mountaintops are cold”, which means that
surface temperature decreases with elevation. However, this idea does not
apply on present-day Mars. Here, we investigate when and why the Earth-based
understanding holds for planets with different types of atmospheres. Using a
global climate model, we show that both the greenhouse effect (atmospheric
infrared opacity) and the pressure effect (atmospheric turbulence) are
important. The weaker the greenhouse effect, or the thinner the atmosphere,
the slower the surface cools with elevation. The greenhouse effect plays the
dominant role, but in moderately opaque atmospheres, the pressure effect
becomes important as well. Our work reveals a novel connection between climate
and geomorphology. For example, on a planet with a relative thin pure
$\mathrm{O_{2}}$ or $\mathrm{N_{2}}$ atmosphere, we do not expect that
“mountaintops are cold”.

##  1 Introduction

Surface temperature, $T_{s}$, is fundamental for understanding habitability
[Seager (2013)]. In addition to the direct implications of surface temperature
for life, the distribution of surface temperature defines the cold trap (where
moisture tends to condense and accumulate), which regulates the hydrological
cycle on an arid planet [Mitchell Lora (2016), Ding Wordsworth (2020)].

There are two paradigms for estimating the distribution of surface
temperature. The first paradigm, which we refer to as “radiation deficits are
cold”, states that the coldest region is close to the time-mean minimum of
solar radiation. These regions are the deficits of net radiation flux at the
top of the atmosphere. For example, the poles are regions of radiation deficit
for Earth and modern Mars, the night hemisphere is the region of radiation
deficit for synchronously rotating exoplanets [Wordsworth (2015)], and the
tropics were the region of radiation deficit for pre-modern Mars at times when
the obliquity was very high [Forget . (2006)]. This paradigm focuses on the
large-scale pattern of surface temperature, and has been extensively studied
[<]e.g.,¿held1993,forget2006,kaspi2015,wordsworth2015atmospheric.

The second paradigm, which we refer to as “mountaintops are cold,” emphasizes
local processes and emerged long before the modern era. This paradigm states
that the change of surface temperature with elevation should follow that of
the atmosphere, which can be quantified as the lapse rate. For a well-mixed,
isolated atmospheric column without condensible species, the lapse rate is the
dry adiabat, $\Gamma_{ad}$:

$\Gamma_{ad}=-\frac{dT_{a}}{dZ}=\frac{g}{c_{p}}$ (1)

where $T_{a}$ is atmospheric temperature, $Z$ is height, $g$ is gravity, and
$c_{p}$ is the specific heat of air. If surface temperature also follows this
adiabatic lapse rate, we therefore expect colder temperature at higher
elevations. Taken together, these two paradigms predict how surface
temperature changes horizontally (by solar radiation) and following surface
elevation (by gravity and atmospheric composition).

However, the idea that “mountaintops are cold” does not apply to Mars today.
Surface lapse rate, $\Gamma_{s}$, (the change of surface temperature with
elevation) is weak on modern Mars [Sagan Pollack (1968)]. Recent work on early
Mars has linked the Martian surface lapse rate to the atmosphere’s evolution
[Forget . (2013), Wordsworth . (2013), Wordsworth (2016), Kite (2019)].
Specifically, $\Gamma_{s}$ is close to $\Gamma_{ad}$ only for scenarios with
thick $\mathrm{CO_{2}}$ atmospheres. For thin $\mathrm{CO_{2}}$ atmospheres,
$\Gamma_{s}$ is close to zero. From the perspective of surface energy budgets,
forget2013 and Wordsworth2016 suggested that both the sensible heat flux,
$SH$, and atmospheric longwave heating, $LW_{a}$, are important in modulating
$T_{s}$ when the $\mathrm{CO_{2}}$ atmosphere goes from thick to thin.
However, the mechanisms that control the transition in $\Gamma_{s}$ are not
fully understood. It remains unclear whether $SH$ or $LW_{a}$ is more
important for the change of $\Gamma_{s}$ seen in earlier work, and what
controls the changes in $SH$ and $LW_{a}$. Relatedly, it is not clear in how
far the surface lapse rate is sensitive primarily to the change in the surface
pressure, versus the change in the greenhouse effect. While the surface
pressure and greenhouse effect are directly linked in a pure $\mathrm{CO_{2}}$
atmosphere, understanding the role of these two distinct effects is important
to predict the surface lapse rate on planets with different atmospheres. Are
mountaintops still cold for planets with, for example, a thick, pure
$\mathrm{O_{2}}$ atmosphere (high pressure, weak greenhouse effect)? What
about a thin, fluoride atmosphere (e.g., $\mathrm{CF_{4}}$ or
$\mathrm{SF_{6}}$, low pressure, strong greenhouse effect), as suggested by
marinova2005?

In this paper, we seek a better understanding of when and why “mountaintops
are cold”, for planets with different greenhouse gas forcings and atmospheric
pressures. Following koll2016, we focus on “dry planets” (idealized planets
forced by gray radiation) to gain a basic understanding of the phenomenon. We
introduce our methodology in Section 2. We present and analyse the results in
Section 3. Section 4 includes our conclusion, limitations of this research,
and implications for future work on different planets.

##  2 Methods

###  2.1 General Circulation Model (GCM)

We use the MarsWRF GCM [Richardson . (2007), Toigo . (2012)] to investigate
temperature distribution across different atmospheres. The model resolution is
72$\times$36$\times$40 gridpoints in longitude/latitude/height. All
simulations are run for 20 years with 5 years of spin up and averages taken
over the last 15 years.

To aid understanding, we use idealized simulations with the following
settings. The radiative transfer is computed using a gray gas scheme. Under
the scheme, the longwave absorption coefficient, $\kappa$, is varied, allowing
us to decouple the greenhouse effect from surface pressure. The shortwave
Rayleigh scattering and absorption are set to zero. Surface albedo is
uniformly zero. We also carry out simulations with a pure $\mathrm{CO_{2}}$
atmosphere, using a correlated-k scheme for radiative transfer [Mischna .
(2012)] to validate our simulations against earlier studies (Supplementary
Information A).

For the default simulations, the planetary obliquity and orbital eccentricity
are set to zero, with solar constant 75$\%$ of the modern Martian value,
representing the faint young Sun. Diurnal cycles are disabled. Planetary size
and rotation rate are set to Mars values. We explore different planetary
climates over a 2D parameter space: varying mean surface pressure, $p_{s}$,
and mean surface longwave optical depth, $\tau$, between 0.01 bar and 5
bar111MarsWRF requires $p_{s}$ to be multiples of modern Mars pressure (610
Pa). Here we choose the multiplier to be 2, 17, 167, 833, which correspond to
$p_{s}$ being 0.012, 0.103, 1.02, 5.08 bar. and between 0.003 and 5,
respectively. The range of values is chosen so as to compare with earlier work
[Forget . (2013), Kamada . (2021)]. We note that higher values of $p_{s}$ or
$\tau$ lead to an energy flux imbalance at the top of the atmosphere ($>5\%$
imbalance compared to the net shortwave flux) in MarsWRF. Our simulations are
performed with an idealized topography, which is a Gaussian-shaped mountain
placed at the equator (blue dashed contours in Fig. 1a $\&$ Fig. 1b):

$Z_{s}=6000\times e^{-\frac{1}{2}\frac{X^{2}}{9^{2}}}\times
e^{-\frac{1}{2}\frac{Y^{2}}{7^{2}}}$ (2)

where $Z_{s}$ is surface elevation (in meters), and $X$ and $Y$ are longitude
and latitude grid points ($-35.5\leq X\leq 35.5$, $-17.5\leq Y\leq 17.5$),
respectively.

The surface sensible heat flux is given by:

$SH=\rho c_{p}C_{h}U^{*}(\theta_{a}-\theta_{s})$ (3)

where $\rho$ is near-surface air density, $c_{p}$ is the specific heat
capacity of air, $C_{h}$ is a heat exchange coefficient, $U^{*}$ is friction
velocity, $\theta_{a}$ is near-surface potential temperature, and $\theta_{s}$
is surface potential temperature, respectively. Both $C_{h}$ and $U^{*}$ are
calculated inside MarsWRF’s surface layer scheme [Zhang Anthes (1982)], which
uses Monin-Obukhov similarity and accounts for four stability categories:
stable, mechanically induced turbulence, unstable forced convection, and
unstable free convection.

To connect our idealized simulations to more Mars-relevant scenarios, we
perform the following sensitivity tests: (1) obliquity set to $20^{\circ}$;
(2) obliquity set to $20^{\circ}$ and with condensation and sublimation of ice
caps [Chow . (2019)] (no atmospheric collapse is found in this case); and (3)
modern Mars topography. For each set of sensitivity tests, we vary $\tau$
while fixing $p_{s}$ to 1 bar, and we vary $p_{s}$ and while fixing $\tau$ to
0.1.

We also perform two sets of mechanism-denial experiments to verify the role of
sensible heat flux, $SH$, as well as the role of atmospheric mass in
modulating $SH$. In the first set of mechanism-denial experiments, $SH$ is
forced to be 0. In the second set of mechanism-denial experiments, the value
of $\rho$ in Eq. (3) is held fixed at the reference value for a 1-bar
atmosphere, thereby eliminating the direct effect of surface pressure on the
surface turbulent heat flux. The effect of varying atmospheric mass is still
considered in all other components of the model, and $SH$ is allowed to change
as a result of indirect effects.

###  2.2 Definition of the orographic temperature control: relative surface
lapse rate, $\gamma$

The relationship between surface temperature, $T_{s}$, and elevation, $Z_{s}$,
is quantified via the surface lapse rate, $\Gamma_{s}$:

$\Gamma_{s}=-\frac{dT_{s}}{dZ_{s}}$ (4)

where $dT_{s}/dZ_{s}$ is quantified by calculating a linear regression of the
time-mean model output in the tropical belt $20^{\circ}$ N - $20^{\circ}$ S
(see the red dashed lines in Fig. 1a and white dashed lines in Fig. 1b). We
also analyzed the effect of topography in the mid-latitudes (Supplementary
Information B) to test the sensitivity to the choice of latitude.

Furthermore, we define the relative surface lapse rate, $\gamma$, as the
surface lapse rate scaled by the atmospheric dry adiabat:

$\gamma=\frac{\Gamma_{s}}{\Gamma_{a}}\times 100\%$ (5)

Thus we expect, for the Earth-like regime (“mountaintops are cold”), $\gamma$
to be close to $100\%$.

##  3 Results

###  3.1 The transition of surface lapse rate

We first examine the horizontal distribution of temperature in our GCM
simulations. Fig. 1a$\&$1b show the typical annual mean surface temperature,
$T_{s}$, in different climates. In all simulations, $T_{s}$ decreases with
increasing latitude. This is consistent with our default setting of obliquity
to $0^{\circ}$ (polar cold traps are created by radiation deficits). The
pattern of near-surface atmospheric temperature, $T_{a}$, follows $T_{s}$
closely (Supplementary Information C), with minor modulation by the winds
across the elevated topography [Wordsworth . (2015)].

For the topographic control on surface temperature, we find two opposing
limits for thick and thin atmospheres. In the thick atmosphere limit
($p_{s}=5$ bar, $\tau=5$), we find “mountaintops are cold” (Fig. 1a): $T_{s}$
decreases with $Z_{s}$ ($\gamma\rightarrow 100\%$). In the thin atmosphere
limit ($p_{s}=0.01$ bar, $\tau=0.01$), the $T_{s}$ distribution becomes
zonally banded (Fig. 1b), with almost no dependence on topography
($\gamma\rightarrow 0$).

The transition of the tropical surface temperature distribution across
different climates can be quantified as the change in the relative surface
lapse rate, $\gamma$, with varying surface longwave optical depth, $\tau$, and
surface pressure, $p_{s}$ (Fig. 1c). We find $\gamma$ increases with $\tau$
and $p_{s}$. However, the role of the greenhouse effect (i.e., variations in
$\tau$) and the pressure effect (variations in $p_{s}$) are not symmetric.
Within our parameter space, we find $\gamma$ always increases significantly
with $\tau$ for any given $p_{s}$, but $\gamma$ increases significantly with
$p_{s}$ only for intermediate values of $\tau$ ($\tau\sim 0.1$). When
$\tau\leq 0.01$, we find $\gamma\approx 0$ for all values of $p_{s}$. When
$\tau>1$, $\gamma$ is close to saturation and increases only slowly with
$p_{s}$. Sensitivity of $\gamma$ on $p_{s}$ is smaller than that on $\tau$
even at intermediate $\tau$. For example, starting from $\tau=0.1$, $p_{s}=1$
bar (i.e., a cold early Mars), decreasing $\tau$ by one order of magnitude
leads to $\gamma$ decreasing from 37$\%$ to 7$\%$ (red solid line in Fig. 1d),
while decreasing $p_{s}$ by one order of magnitude leads to $\gamma$
decreasing from 37$\%$ to 15$\%$ (red solid line in Fig. 1e).

We also explore the change of $\gamma$ with $\tau$ and $p_{s}$ in sensitivity
tests. Here we focus on Mars-relevant scenarios when the obliquity is non-
zero, atmospheric condensation occurs, or the topography is different. We find
$\gamma$ is slightly smaller, but still increases with $\tau$ and $p_{s}$,
when the obliquity is non-zero (compare the blue solid line to the red solid
line in Fig. 1d$\&$1e). The relationship between $\gamma$ and $\tau$ and
$p_{s}$ also holds when the atmosphere partially condenses and sublimates
seasonally (blue dotted lines in Fig. 1d$\&$1e). Changing to Mars topography
decreases the sensitivity of $\gamma$ on $\tau$ and, in particular, $p_{s}$,
although the qualitative results remain robust (magenta lines in Fig.
1d$\&$1e, also see Supplementary Information A). Especially, the $p_{s}$
sensitivity becomes very small. The surface lapse rate in the mid-latitudes
(discussed in Supplementary Information B) ) also shows similar sensitivity to
$\tau$, but is virtually insensitive to $p_{s}$. In conclusion, we confirm
that both the greenhouse effect and the pressure effect can contribute to the
lapse rate transition, as suggested by earlier studies [Forget . (2013),
Wordsworth (2016)], but the greenhouse effect dominates and is more robust.

###  3.2 Surface energy budgets

How do the longwave optical depth and surface pressure control the surface
lapse rate in different climates? Wordsworth2016 proposed using the surface
energy budget to understand the mechanisms controlling the surface lapse rate.
The surface energy budget is:

$SW+LW_{a}=LW_{s}+SH$ (6)

where $SW$ is the net shortwave heating from the star, $LW_{a}$ is the
longwave heating from the atmosphere (greenhouse effect), and $LW_{s}$ is the
longwave cooling by surface emission, which is directly related to $T_{s}$:

$LW_{s}=\sigma T_{s}^{4}$ (7)

where $\sigma=5.67\times 10^{-8}\;\mathrm{W/m^{2}/K^{4}}$ is the Stefan-
Boltzmann constant. $SH$ is the sensible heat flux. There is no latent heat
term in the surface energy budget because water vapor and $\mathrm{CO_{2}}$
condensation are disabled in our default simulations.

To better understand the role of atmospheric optical thickness and pressure on
the surface temperature structure, we analyze the surface energy budget in our
simulations with varying $\tau$ and $p_{s}$. To visualize the model output, we
apply a tropical meridional average ($20^{\circ}$ N - $20^{\circ}$ S) and
time-average to the model output. With this approach, the temperature gradient
due to solar insolation is minimized. The annual mean longitudinal variation
in $T_{s}$ corresponds to the $LW_{s}$ term (see Eq. 7). For example, the red
line in Fig. 2c indicates a temperature minimum at longitude = $0^{\circ}$ in
the tropics, which corresponds to the highland in our idealized topography
(Fig. 1a).

Figure 2: Time-averaged surface energy budgets for typical scenarios: (a)
$p_{s}$ = 0.01 bar, $\tau$ = 0.01, (b) $p_{s}$ = 0.01 bar, $\tau$ = 0.1, (c)
$p_{s}$ = 0.01 bar, $\tau$ = 5, (d) $p_{s}$ = 5 bar, $\tau$ = 0.01, (e)
$p_{s}$ = 5 bar, $\tau$ = 0.1, (f) $p_{s}$ = 5 bar, $\tau$ = 5. Relative
surface lapse rate (surface lapse rate scaled by adiabat), $\gamma$, for each
case is indicated in the upper-left corner. $SW$ is the net shortwave heating
from the star, $LW_{a}$ is the longwave heating from the greenhouse effect,
$LW_{s}$ is the longwave cooling by surface emission, and $SH$ is the cooling
by sensible heat flux, respectively. Each term is meridionally averaged within
the tropics ($20^{\circ}$N - $20^{\circ}$S). A dip in the red curve indicates
a correlation between $T_{s}$ and topography (lower $T_{s}$, thus lower
emission over the highlands - see Eq. 7).

We find three typical scenarios for the zonal structure of the tropical
surface energy budget (Fig. 2, see Supplementary Information D for all cases).
For an atmosphere that is optically transparent ($\tau<0.1$, Fig. 2a$\&$2d),
or thin and optically intermediate ($p_{s}\leq 0.1$ bar, $\tau\sim 0.1$, Fig.
2b), the major balance is between surface emission ($LW_{s}$, red lines) and
shortwave absorption ($SW$, blue lines). The other terms are small. Since $SW$
does not vary with topographic elevation in our model setup, $LW_{s}$ (and
thus $T_{s}$) can’t vary much either. Hence, $\gamma$ is close to zero under
this scenario. For optically thick atmospheres ($\tau\geq 1$, Fig. 2c$\&$2f),
the dominant balance is between surface emission, $LW_{s}$, and longwave
heating ($LW_{a}$, cyan lines). The longwave heating is weaker over the
highlands compared to the lowlands, thus the highland surface is colder.
Although the magnitude of $SH$ is non-negligible for massive atmospheres
($p_{s}\leq 1$ bar), the spatial variations of $SH$ are small under this
scenario. This is consistent with our earlier results that $\gamma$ is
dominated by $\tau$ when $\tau\geq 1$ (Fig. 1c). For massive, moderately
opaque atmospheres ($p_{s}\geq 1$ bar, $\tau\sim 0.1$, Fig. 1e), $SH$
variations are large enough to generate a significant pattern in $LW_{s}$ (and
thence $T_{s}$), while $LW_{a}$ is still small. Therefore, in this regime,
variations in $SH$ are important for the surface lapse rate.

Figure 3: Time-averaged surface energy budgets (unit in $\mathrm{W/m^{2}}$)
for all cases in Fig. 1c. Same as Fig. 2, each variable is meridionally
averaged within the tropics ($20^{\circ}$N - $20^{\circ}$S). (a) Contrast of
surface emission, $LW_{s}$, (b) contrast of greenhouse effect, $LW_{a}$, (c)
contrast of sensible heat flux, $SH$.

To illustrate the role of spatial variations in the longwave radiation and
surface heat flux across the full parameter regime, Fig. 3 shows the highland-
lowland contrast as a function of $\tau$ and $p_{s}$ in our default
simulations. Here, we focus on the difference between the maximum and minimum
values (positive-definite) to visualize the highland-lowland contrast. By
definition, the change in surface emission contrast (Fig. 3a) reflects the
surface lapse rate transition (Fig. 1c). The surface emission contrast
$LW_{s,max}-LW_{s,min}$ is balanced by contrasts in the greenhouse forcing,
$LW_{a}$, and sensible heat flux, $SH$. The change in the $LW_{a}$ contrast
follows $\tau$ and dominates the transition (Fig. 3b). The $SH$ contrast
additionally contributes to the surface energy budget pattern when the
atmosphere is moderately opaque (Fig. 3c).

Wordsworth2016 proposed that $p_{s}$ affects $\gamma$ by modulating $\rho$ in
the equation of $SH$ (Eq. 3). To test the importance of $SH$ in general, and
the role of $\rho$ in the surface heat exchange equation in particular, we
perform two sets of mechanism-denial experiments with modified $SH$. In the
first set of simulations, we manually disable $SH$, which significantly
reduces $\gamma$ when the atmosphere is intermediately opaque ($\tau\sim
0.1$), such that $\gamma$ remains small at all surface pressures (solid cyan
lines in Fig. 1d$\&$1e). However, $\gamma$ still becomes large for very large
$\tau$. In the second set of simulations, we fix $\rho$ at the reference value
for a 1-bar atmosphere in the equation for $SH$ (Eq. 3), which does not
significantly change the sensitivity of $\gamma$ to $p_{s}$ (dotted cyan line
in Fig. 1e). Thus, $SH$ is necessary for the increase of $\gamma$ with $p_{s}$
at intermediate $\tau$, but the increase of $\rho$ in Eq. (3) is not the
primary mechanism.

###  3.3 Surface lapse rate transition in a two-column model

To improve our understanding of the mechanisms that cause the surface lapse-
rate transition, we construct a simpler two-column model, building on previous
work by yang2014, wordsworth2015atmospheric, and koll2016. Similar to
yang2014, our model is formulated by requiring energy balance in the free
troposphere and at the surface (Eq. 8 - Eq. 11), making the Weak Temperature
Gradient (WTG) approximation (Eq. 12), and enforcing convective stability or
neutrality (Eq. 13 - Eq. 14). The model is therefore built on the hypothesis
that the key ingredients to explain the surface lapse rate include 1) a WTG in
the free troposphere, 2) convection, which maintains an adiabatic lapse-rate
if and only if the radiative-advective equilibrium solution is unstable, and
3) radiative-advective equilibrium (i.e. negligible turbulent heat flux) when
the solution is stable. The model equations are:

$SW-F_{c,HL}+\epsilon_{HL}\sigma T_{a,HL}^{4}-\sigma T_{s,HL}^{4}=0$ (8)
$F_{c,HL}-F_{a}-F_{H}+\epsilon_{HL}\sigma T_{s,HL}^{4}-2\epsilon_{HL}\sigma
T_{a,HL}^{4}=0$ (9)
$F_{c,LL}+\frac{\alpha}{1-\alpha}F_{a}-F_{H}+\epsilon_{LL}\sigma
T_{s,LL}^{4}-2\epsilon_{LL}\sigma T_{a,LL}^{4}=0$ (10) $SW-
F_{c,LL}+\epsilon_{LL}\sigma T_{a,LL}^{4}-\sigma T_{s,LL}^{4}=0$ (11)
$T_{a,HL}-T_{a,LL}=0$ (12)
$DSE_{s,HL}\begin{cases}=DSE_{a,HL}&\text{convective highland, solve for
$F_{c,HL}$}\\\ <DSE_{a,HL}&\text{stratified highland,
$F_{c,HL}=0$}\end{cases}$ (13)
$DSE_{s,LL}\begin{cases}=DSE_{a,LL}&\text{convective lowland, solve for
$F_{c,LL}$}\\\ <DSE_{a,LL}&\text{stratified lowland, $F_{c,LL}=0$}\end{cases}$
(14)

Here, $T_{s,HL}$ and $T_{s,LL}$ are the surface temperatures of the highland
and lowland, respectively; $T_{a,HL}$ and $T_{a,LL}$ are the free-tropospheric
temperature of the highland and lowland, respectively; $F_{c,HL}$ and
$F_{c,LL}$ are the convective heat flux from the surface to the free
troposphere in the highland and lowland column, respectively; $F_{a}$
represents atmospheric heat transport between the columns; $F_{H}$ represents
the atmospheric heat outflow from the tropical band caused by the Hadley
circulation; $\alpha$ is the surface area fraction of the highland within the
latitudinal belt; $\epsilon_{HL}$ and $\epsilon_{LL}$ are the atmospheric
emissivities of the two columns; and $DSE$ is the dry static energy at the
respective location. The values of $\epsilon$ and $DSE$ are calculated as:

$\epsilon_{HL}=\frac{p_{s,HL}}{p_{s}}\tau$ (15)
$\epsilon_{LL}=\frac{p_{s,LL}}{p_{s}}\tau$ (16) $DSE_{i}=c_{p}T_{i}+gZ_{i}$
(17)

The choices of parameters are explained in Supplementary Information E. Using
the seven equations (8)-(14), we numerically determine solutions for the seven
dependent variables of the model: surface temperature ($T_{s,HL}$ and
$T_{s,LL}$), atmospheric temperature ($T_{a,HL}$ and $T_{a,LL}$), atmospheric
heat transport between the columns $F_{a}$, and convective heat flux
($F_{c,HL}$ and $F_{c,LL}$). We note that the model can be further simplified
by combining Eq. (9), (10) and (12) into a single equation to eliminate
$F_{a}$ and merge $T_{a,HL}$ and $T_{a,LL}$ into a single unknown
($T_{a,HL}=T_{a,LL}\equiv T_{a}$). Moreover, since convection is never active
in the lowland for any of the presented solutions, we can obtain the same
results by setting $F_{c,LL}=0$ and eliminating Eq. (14). However, we found
that these simplifications provide no additional insight, and numerical
solution is trivial with either formulation. Similar to
wordsworth2015atmospheric, our model is constructed for cases when $\tau\ll
1$. For optically thick atmospheres ($\tau>1$), the surface is no longer
radiatively heated by the same atmospheric layer that emits to space, thus the
single-layer-atmosphere approximation breaks down.

Figure 4: The transition of relative lapse rate, $\gamma$, in the two-column
model. (a) The dependence of $\gamma$ on the greenhouse effect, $\tau$. Solid
line (def): default case - same as the red dotted line in Fig. 1d. Dashed line
(fixed $\epsilon$): the case with uniform greenhouse forcing above highlands
and lowlands. Dotted line ($F_{a}=0$): the case with no heat advection between
the highland and lowland atmosphere. Dash-dotted line ($F_{c}=0$): the case
with no convection between the atmosphere and surface. (b) The dependence of
$\gamma$ on the pressure effect. Solid line (def): default case - same as the
red dotted line in Fig. 1e. Dashed line (fixed $F_{H}$): the case with
meridional heat advection fixed to the 1 bar value ($F_{H}=6.5528$ W/m2).

The two-column model is capable of reproducing the increase of $\gamma$ with
$\tau$ and $p_{s}$ (red dotted lines, Fig. 1d$\&$1e). From the surface energy
budget perspective, our two-column model is qualitatively consistent with the
transitions in the highland-lowland contrast from the GCM (Supplementary
Information E).

In addition to confirming that the assumptions entering the two-column model
formulation appear to be sufficient to understand the lapse rate transition,
the model provides some insight into the specific mechanisms. The surface
energy budget analysis discussed above indicated that differential longwave
radiation between the lowland and highland is important to understand the
surface lapse rate at high optical thickness. A naive interpretation is that
$LW_{a}$ is smaller over the highland simply because the overlying atmosphere
is less massive and hence has a weaker greenhouse effect. We can test this
hypothesis in the two column model by eliminating the difference in $\epsilon$
between the two columns (setting
$\frac{p_{s,HL}}{p_{s}}=\frac{p_{s,LL}}{p_{s}}=1$ in Eq. 15$\&$16). We find
the change of $\gamma$ with $\tau$ persists in this sensitivity experiment,
and indeed is only weakly affected (compare solid and dashed lines in Fig.
4a).

So what instead explains the transition with $\tau$? In the two-column model,
the increase of the surface lapse rate, $\gamma$, is directly related to the
lapse rate in the lowland column, due to the assumption of WTG in the
atmosphere and convective adjustment over the highlands. The importance of
these two assumptions can be illustrated by setting either $F_{a}$ or $F_{c}$
to zero (by modifying Eq. 12 or Eq. 13), which changes the relationship
between $\gamma$ and $\tau$ significantly (dotted line and dash-dotted line in
Fig. 4a). Notably, without convective heat transport between the surface and
atmosphere, the sensitivity of $\gamma$ effectively reproduces the GCM
simulations without $SH$ (solid cyan line in Fig. 1d). WTG and convective
adjustment link the surface lapse rate to the lowland atmospheric lapse rate,
which, in turn, is governed by radiative-advective equilibrium. As discussed
in payne2015 and cronin2016, the lapse rate of a column in radiative-advective
equilibrium increases under increasing greenhouse forcing, as the increased
radiative flux cools the atmosphere and heats the surface.

In the next step we examine the sensitivity to the global mean surface
pressure, $p_{s}$. In the two-column model, $p_{s}$, by construction, affects
the solution only via its indirect effect on the meridional heat transport
$F_{H}$, which is here diagnosed from the GCM simulations. The sensible heat
flux, $SH$, which in the real world is associated with complex boundary layer
physics [Joshi . (2020)], is implied by Eq. 13$\&$14\. Consistent with
kaspi2015, we find that higher $p_{s}$ (and thus greater atmospheric mass)
drives larger $F_{H}$. Following our argument above, atmospheric heat flux
divergence from the tropics, $F_{H}$, leads to a reduction of the net
atmospheric heat flux convergence over the lowlands, which leads to a
reduction in the atmospheric lapse rate (and, hence, a reduction in the
surface lapse rate, $\gamma$). The mechanism can be illustrated by fixing
$F_{H}$. As expected, we find that when $F_{H}$ is fixed, the sensitivity of
$\gamma$ to $p_{s}$ disappears (Fig. 4b).

Taken together with the GCM simulations, the results suggest that both a weak
temperature gradient and highland convection are important for explaining the
sensitivity of $\gamma$ to the greenhouse effect and pressure effect.
Meanwhile, the spatial variations in the column-integrated greenhouse gas and
the near-surface air density do not play a major role in the sensitivity of
the surface lapse rate.

##  4 Discussion and Summary

“Radiation deficits are cold” and “mountaintops are cold” comprise the usual
expectation for the distribution of surface temperature, $T_{s}$. Here, using
a GCM simulating a fast-rotating, dry planet, we argue that whether
“mountaintops are cold” depends on both the greenhouse effect (longwave
optical depth, $\tau$) and surface pressure, $p_{s}$. Specifically, the
dependence of $T_{s}$ on surface elevation is quantified as the tropical
surface lapse rate relative to the adiabat, $\gamma$. We find that $\gamma$
increases with $\tau$ and $p_{s}$. However, the roles of the greenhouse effect
and pressure effect are not symmetric. At all surface pressures, we find that
$\gamma$ is close to zero for very small $\tau$, and approaches 100$\%$ as
$\tau\gg 1$. Surface pressure plays a significant role at intermediate $\tau$,
where more massive atmospheres tend to have larger $\gamma$ in the tropics,
but the effect of $p_{s}$ is less robust. From a surface energy budget
perspective, spatial variations in the downwelling atmospheric longwave
radiation $LW_{a}$ balance the variations in the upward longwave radiation
associated with the topographic surface temperature variations for optically
thick atmospheres ($\tau>1$). For optically moderate atmospheres ($\tau\sim
0.1$) the variations in surface longwave emission $LW_{s}$, associated with
the topographic temperature variations, are maintained by variations in the
sensible heat flux, $SH$. Large $\gamma$ requires a weak temperature gradient
in the atmosphere and effective coupling between the surface and the
atmospheric temperature, where the coupling can occur either radiatively or
via $SH$. The surface lapse rate transition can be reproduced in a two-column,
two-layer model, consisting of a convective highland column together with a
stable lowland column, coupled via the weak temperature gradient assumption in
the atmosphere. The two-column model suggests that weak temperature gradient
and highland convection are important to explain the lapse rate transition.
Increases in optical thickness or surface pressure then affect the tropical
surface lapse rate by destabilizing the atmospheric lapse rate over the
lowlands.

This paper focuses on the surface temperature distribution on fast-rotating,
dry planets. We can speculate how the conclusions might differ on other
planets, although future work should use GCMs to verify these predictions. (1)
For tidally locked planets, solar insolation never reaches the permanent night
hemisphere, such that the nightside, heated by advection, is stably stratified
[Joshi . (2020), Ding Wordsworth (2021)]. Thus, the mountaintops in the night
hemisphere might be warm depending on the strength of the thermal inversion.
(2) For warm, wet planets, water vapor modulates the atmospheric lapse rate
from a dry adiabat towards the moist adiabat, which is likely to similarly
affect the surface lapse rate for thick atmospheres. (3) For cold, wet
planets, the ice-albedo feedback is important. The existence of ice would
decrease absorbed solar insolation. In the optically thin limit, one would
then expect to find a surface temperature discontinuity near the snowline. In
the optically thick limit, our mechanism suggests that the surface lapse rate
still approaches the adiabat, independent of the presence of snow or ice
(which is broadly consistent with present-day Earth). (4) The influence of
spectral properties of real gas species are ignored in this work. For real gas
species, surface pressure and greenhouse effect are coupled due to the effect
of pressure broadening and collision-induced absorption (CIA) [Ding Wordsworth
(2019)]. Meanwhile, for optically thick atmospheres with typical greenhouse
species (e.g., $\mathrm{CO_{2}}$, $\mathrm{H_{2}O}$), the surface can still
emit to space through the spectral windows. Different greenhouse gas emission
spectra are therefore likely to affect the quantitative results, although we
have found that changing optical thickness in our gray atmosphere
qualitatively reproduces the effect of increasing $\mathrm{CO_{2}}$. (5)
Rayleigh scattering by atmospheric molecules (which influences shortwave
heating, $SW$, and is related to pressure) is ignored in this paper. It was
found to be unimportant within the parameter space used in our study. But the
role of reflection by clouds remains unknown.

Inspired by earlier research on Mars [Forget . (2013), Wordsworth . (2013),
Wordsworth (2016), Kite (2019)], our work suggests that early Martian
sedimentary geology might be explained by a new end-member option for the
climate: “high $T_{s}$ \+ non-$\mathrm{CO_{2}}$ greenhouse gases + low
$p_{\mathrm{CO_{2}}}$”. Under this scenario, changes in fluvial patterns may
arise from non-$\mathrm{CO_{2}}$ greenhouse forcings rather than the loss of a
$\mathrm{CO_{2}}$-dominated atmosphere [Kite . (2022)]. The
non-$\mathrm{CO_{2}}$ greenhouse forcing could be water ice cloud radiative
forcings [Urata Toon (2013), Kite . (2021)], or
$\mathrm{H_{2}}+\mathrm{CO_{2}}$ CIA [Wordsworth . (2017), Turbet Forget
(2021)]. Future work should further test the correlation between river
locations and elevation with non-$\mathrm{CO_{2}}$ greenhouse forcings and low
atmospheric pressure ($p_{s}<1$ bar).

Our work can be connected to other planets in the solar system. Based on
Akatsuki infrared measurements, recent analysis show that surface temperature
on Venus depends little on insolation compared to elevation [Singh (2019)].
However, the elevation dependence is small compared to the adiabat. It remains
unclear whether the unexpectedly small dependence is a result of physical
processes [Esposito (1984)] or instrument error [Iwagami . (2018)].
Fortunately, forthcoming missions (DAVINCI, VERITAS, and EnVision) will extend
the understanding with new data on atmospheric thermal structure and surface
properties. Meanwhile, modeling on Titan has found robust correlation between
temperature and elevation [Lora . (2019)]. The greenhouse effect on Titan is
provided by methane and $\mathrm{N_{2}}$-$\mathrm{N_{2}}$ CIA. Since the
lifetime of methane is short on geological timescales, it is possible that
Titan has undergone a lapse rate transition after a methane outburst event
[Tokano Lorenz (2021)].

With respect to the habitability of exoplanets, our work also highlights the
potential role of topography in creating cold traps in optically thick
atmospheres, in addition to the cold traps created by radiation deficts and
atmospheric circulation [Ding Wordsworth (2020)]. Most exoplanet GCMs assume
no topography. However, topography allows a greater chance for water to
condense. Future work could focus on different potential climate regimes under
the competition of these cold traps, as well as their influences on the
hydrological cycle and long-term planetary evolution (e.g., the transition
between snowball and habitable climates).

###### Acknowledgements.

This research was funded by NASA (80NSSC20K0144+80NSSC18K1476) and NSF award
AGS-2033467. We thank Robin Wordsworth and an anonymous reviewer for comments
that led to an improved manuscript. We also thank Tiffany Shaw for helpful
discussions. Our simulations were completed with resources provided by the
University of Chicago Research Computing Center (RCC). A portion of this work
was carried out at the Jet Propulsion Laboratory, California Institute of
Technology, under a contract with the National Aeronautics and Space
Administration (80NM0018D0004).

## Open Research

Data necessary to reproduce the figures in this paper is publicly available on
Zenodo [Fan (2023)].

## References

  * Chow . (2019) chow2019flowChow, KC., Xiao, J., Chan, KL. Wong, CF.  2019\.  Flow associated with the condensation and sublimation of polar ice caps on Mars Flow associated with the condensation and sublimation of polar ice caps on Mars.  Journal of Geophysical Research: Planets12461570–1580.
  * Cronin Jansen (2016) cronin2016Cronin, TW. Jansen, MF.  2016\.  Analytic radiative-advective equilibrium as a model for high-latitude climate Analytic radiative-advective equilibrium as a model for high-latitude climate.  Geophysical Research Letters431449–457.
  * Ding Wordsworth (2019) ding2019newDing, F. Wordsworth, RD.  2019\.  A new line-by-line general circulation model for simulations of diverse planetary atmospheres: initial validation and application to the exoplanet GJ 1132b A new line-by-line general circulation model for simulations of diverse planetary atmospheres: initial validation and application to the exoplanet GJ 1132b.  The Astrophysical Journal8782117.
  * Ding Wordsworth (2020) Ding2020Ding, F. Wordsworth, RD.  2020\.  Stabilization of dayside surface liquid water via tropopause cold trapping on arid slowly rotating tidally locked planets Stabilization of dayside surface liquid water via tropopause cold trapping on arid slowly rotating tidally locked planets.  The Astrophysical Journal Letters8911L18.
  * Ding Wordsworth (2021) Ding2021Ding, F. Wordsworth, RD.  2021\.  Multiple Moist Climate Equilibrium States on Arid Rocky M-dwarf Planets: A Last-saturation Tracer Analysis Multiple moist climate equilibrium states on arid rocky M-dwarf planets: A last-saturation tracer analysis.  The Planetary Science Journal25201.
  * Esposito (1984) esposito1984sulfurEsposito, LW.  1984\.  Sulfur dioxide: Episodic injection shows evidence for active Venus volcanism Sulfur dioxide: Episodic injection shows evidence for active Venus volcanism.  Science22346401072–1074.
  * Fan (2023) bowen_fan_2023_8404011Fan, B.  2023\.  Why are Mountaintops Cold? The Transition of Surface Lapse Rate on Dry Planets [Dataset]. Why are mountaintops cold? the transition of surface lapse rate on dry planets [dataset].  Zenodo.  10.5281/zenodo.8404011
  * Forget . (2006) forget2006Forget, F., Haberle, RM., Montmessin, F., Levrard, B. Head, JW.  2006\.  Formation of glaciers on Mars by atmospheric precipitation at high obliquity Formation of glaciers on Mars by atmospheric precipitation at high obliquity.  Science3115759368–371.
  * Forget . (2013) forget2013Forget, F. .  2013\.  3D modelling of the early Martian climate under a denser $\mathrm{CO_{2}}$ atmosphere: Temperatures and $\mathrm{CO_{2}}$ ice clouds 3D modelling of the early Martian climate under a denser $\mathrm{CO_{2}}$ atmosphere: Temperatures and $\mathrm{CO_{2}}$ ice clouds.  Icarus222181–99.
  * Held (1993) held1993Held, IM.  1993\.  Large-scale dynamics and global warming Large-scale dynamics and global warming.  Bulletin of the American Meteorological Society742228–242.
  * Iwagami . (2018) iwagami2018initialIwagami, N. .  2018\.  Initial products of Akatsuki 1-$\mu$m camera Initial products of Akatsuki 1-$\mu$m camera.  Earth, planets and space7011–19.
  * Joshi . (2020) joshi2020Joshi, M., Elvidge, A., Wordsworth, R. Sergeev, D.  2020\.  Earth’s Polar Night Boundary Layer as an Analog for Dark Side Inversions on Synchronously Rotating Terrestrial Exoplanets Earth’s polar night boundary layer as an analog for dark side inversions on synchronously rotating terrestrial exoplanets.  The Astrophysical Journal Letters8922L33.
  * Kamada . (2021) kamada2021Kamada, A., Kuroda, T., Kasaba, Y., Terada, N. Nakagawa, H.  2021\.  Global climate and river transport simulations of early Mars around the Noachian and Hesperian boundary Global climate and river transport simulations of early Mars around the noachian and hesperian boundary.  Icarus368114618.
  * Kaspi Showman (2015) kaspi2015Kaspi, Y. Showman, AP.  2015\.  Atmospheric dynamics of terrestrial exoplanets over a wide range of orbital and atmospheric parameters Atmospheric dynamics of terrestrial exoplanets over a wide range of orbital and atmospheric parameters.  The Astrophysical Journal804160.
  * Kite (2019) Kite2019Kite, ES.  2019\.  Geologic constraints on early Mars climate Geologic constraints on early Mars climate.  Space Science Reviews215110.
  * Kite . (2022) Kite2022Kite, ES., Mischna, MA., Fan, B., Morgan, AM., Wilson, SA. Richardson, MI.  2022\.  Changing spatial distribution of water flow charts major change in Mars’ greenhouse effect Changing spatial distribution of water flow charts major change in Mars’ greenhouse effect.  Science Advances821eabo5894.
  * Kite . (2021) kite2021Kite, ES., Steele, LJ., Mischna, MA. Richardson, MI.  2021\.  Warm early Mars surface enabled by high-altitude water ice clouds Warm early Mars surface enabled by high-altitude water ice clouds.  Proceedings of the National Academy of Sciences11818.
  * Koll Abbot (2016) koll2016Koll, DD. Abbot, DS.  2016\.  Temperature structure and atmospheric circulation of dry tidally locked rocky exoplanets Temperature structure and atmospheric circulation of dry tidally locked rocky exoplanets.  The Astrophysical Journal825299.
  * Lora . (2019) lora2019modelLora, JM., Tokano, T., d’Ollone, JV., Lebonnois, S. Lorenz, RD.  2019\.  A model intercomparison of Titan’s climate and low-latitude environment A model intercomparison of Titan’s climate and low-latitude environment.  Icarus333113–126.
  * Marinova . (2005) marinova2005Marinova, MM., McKay, CP. Hashimoto, H.  2005\.  Radiative-convective model of warming Mars with artificial greenhouse gases Radiative-convective model of warming Mars with artificial greenhouse gases.  Journal of Geophysical Research: Planets110E3.
  * Mischna . (2012) Mischna2012Mischna, MA., Lee, C. Richardson, M.  2012\.  Development of a fast, accurate radiative transfer model for the Martian atmosphere, past and present Development of a fast, accurate radiative transfer model for the Martian atmosphere, past and present.  Journal of Geophysical Research (Planets)117E10E10009.
  * Mitchell Lora (2016) mitchell2016Mitchell, JL. Lora, JM.  2016\.  The climate of Titan The climate of Titan.  Annual Review of Earth and Planetary Sciences44353–380.
  * Payne . (2015) payne2015Payne, AE., Jansen, MF. Cronin, TW.  2015\.  Conceptual model analysis of the influence of temperature feedbacks on polar amplification Conceptual model analysis of the influence of temperature feedbacks on polar amplification.  Geophysical Research Letters42219561–9570.
  * Richardson . (2007) Richardson2007Richardson, MI., Toigo, AD. Newman, CE.  2007\.  PlanetWRF: A general purpose, local to global numerical model for planetary atmospheric and climate dynamics PlanetWRF: A general purpose, local to global numerical model for planetary atmospheric and climate dynamics.  Journal of Geophysical Research (Planets)112E9E09001.
  * Sagan Pollack (1968) Sagan1968Sagan, C. Pollack, JB.  1968\.  Elevation Differences on Mars Elevation Differences on Mars.  Journal of Geophysical Research731373.
  * Seager (2013) Seager2013Seager, S.  2013\.  Exoplanet habitability Exoplanet habitability.  Science3406132577–581.
  * Singh (2019) singh2019venusSingh, D.  2019\.  Venus nightside surface temperature Venus nightside surface temperature.  Scientific Reports911137.
  * Toigo . (2012) Toigo2012Toigo, AD., Lee, C., Newman, CE. Richardson, MI.  2012\.  The impact of resolution on the dynamics of the martian global atmosphere: Varying resolution studies with the MarsWRF GCM The impact of resolution on the dynamics of the martian global atmosphere: Varying resolution studies with the MarsWRF GCM.  Icarus2211276-288.
  * Tokano Lorenz (2021) tokano2021paleoclimateTokano, T. Lorenz, RD.  2021\.  Paleoclimate evolution on Titan after episodic massive methane outgassing simulated by a global climate model Paleoclimate evolution on Titan after episodic massive methane outgassing simulated by a global climate model.  Journal of Geophysical Research: Planets12612e2021JE007081.
  * Turbet Forget (2021) turbet2021Turbet, M. Forget, F.  2021\.  3-D Global modelling of the early Martian climate under a dense $\mathrm{CO_{2}}$+$\mathrm{H_{2}}$ atmosphere and for a wide range of surface water inventories 3-D Global modelling of the early Martian climate under a dense $\mathrm{CO_{2}}$+$\mathrm{H_{2}}$ atmosphere and for a wide range of surface water inventories.  arXiv e-printsarXiv:2103.10301.
  * Urata Toon (2013) urata2013Urata, RA. Toon, OB.  2013\.  Simulations of the martian hydrologic cycle with a general circulation model: Implications for the ancient martian climate Simulations of the martian hydrologic cycle with a general circulation model: Implications for the ancient martian climate.  Icarus2261229–250.
  * Wordsworth (2015) wordsworth2015atmosphericWordsworth, R.  2015\.  Atmospheric heat redistribution and collapse on tidally locked rocky planets Atmospheric heat redistribution and collapse on tidally locked rocky planets.  The Astrophysical Journal8062180.
  * Wordsworth . (2013) wordsworth2013Wordsworth, R., Forget, F., Millour, E., Head, J., Madeleine, JB. Charnay, B.  2013\.  Global modelling of the early Martian climate under a denser $\mathrm{CO_{2}}$ atmosphere: Water cycle and ice evolution Global modelling of the early Martian climate under a denser $\mathrm{CO_{2}}$ atmosphere: Water cycle and ice evolution.  Icarus22211–19.
  * Wordsworth . (2017) wordsworth2017Wordsworth, R. .  2017\.  Transient reducing greenhouse warming on early Mars Transient reducing greenhouse warming on early Mars.  Geophysical Research Letters442665–671.
  * Wordsworth (2016) Wordsworth2016Wordsworth, RD.  2016\.  The Climate of Early Mars The climate of early Mars.  Annual Review of Earth and Planetary Sciences441381-408.
  * Wordsworth . (2015) Wordsworth2015Wordsworth, RD., Kerber, L., Pierrehumbert, RT., Forget, F. Head, JW.  2015\.  Comparison of “warm and wet” and “cold and icy” scenarios for early Mars in a 3-D climate model Comparison of “warm and wet” and “cold and icy” scenarios for early Mars in a 3-D climate model.  Journal of Geophysical Research: Planets12061201–1219.
  * Yang Abbot (2014) yang2014Yang, J. Abbot, DS.  2014\.  A low-order model of water vapor, clouds, and thermal emission for tidally locked terrestrial planets A low-order model of water vapor, clouds, and thermal emission for tidally locked terrestrial planets.  The Astrophysical Journal7842155.
  * Zhang Anthes (1982) zhang1982Zhang, D. Anthes, RA.  1982\.  A high-resolution model of the planetary boundary layer—Sensitivity tests and comparisons with SESAME-79 data A high-resolution model of the planetary boundary layer—sensitivity tests and comparisons with SESAME-79 data.  Journal of Applied Meteorology (1962-1982)1594–1609.


# Reinforcement Learning Problem Solving with Large Language Models

Sina Gholamian
Thomson Reuters AI Labs
Toronto, Canada
<EMAIL_ADDRESS>
&Domingo Huh
Thomson Reuters AI Labs
Toronto, Canada
<EMAIL_ADDRESS>

###### Abstract

Large Language Models (LLMs) encapsulate an extensive amount of world
knowledge, and this has enabled their application in various domains to
improve the performance of a variety of Natural Language Processing (NLP)
tasks. This has also facilitated a more accessible paradigm of conversation-
based interactions between humans and AI systems to solve intended problems.
However, one interesting avenue that shows untapped potential is the use of
LLMs as Reinforcement Learning (RL) agents to enable conversational RL problem
solving. Therefore, in this study, we explore the concept of formulating
Markov Decision Process-based RL problems as LLM prompting tasks. We
demonstrate how LLMs can be iteratively prompted to learn and optimize
policies for specific RL tasks. In addition, we leverage the introduced
prompting technique for episode simulation and Q-Learning, facilitated by
LLMs. We then show the practicality of our approach through two detailed case
studies for “Research Scientist” and “Legal Matter Intake” workflows.

Reinforcement Learning Problem Solving with Large Language Models

Sina Gholamian Thomson Reuters AI Labs Toronto, Canada
<EMAIL_ADDRESS>Domingo Huh Thomson Reuters AI Labs Toronto,
Canada<EMAIL_ADDRESS>

##  1 Introduction

The recent inception of Generative Large Language Models into the limelight
has significantly influenced and reshaped the applications of Natural Language
Processing (NLP) and the ways that LLMs contribute to it Zhao et al. (2023).
Because LLMs are trained on a massive set of internet data, this comprehensive
knowledge gives them the capabilities to extract semantics and respond
coherently in a human-style conversation. Furthermore, fine-tuning through
Reinforcement Learning with Human Feedback (RLHF) Ouyang et al. (2022) has
enabled LLMs to accurately follow input prompt instructions. These
capabilities have allowed LLMs to be used for a variety of out-of-the-box
applications, including chatbot agents Dan et al. (2023); Limna et al. (2023),
coding Vaithilingam et al. (2022); OpenAI (2023), and general common-sense
reasoning Wei et al. (2022). In addition, some studies have shown that LLMs
can be further leveraged for knowledge reasoning in professional domains such
as in medical Liévin et al. (2022) and legal Hakimi Parizi et al. (2023)
fields.

In RL problems, one goal can be to optimize an objective function (e.g., the
sum of the expected long-term rewards) as an agent interacts with the
environment and receives feedback. Deep RL Mnih et al. (2015), a paradigm that
implements the agent or policy with a deep learning model, has gained
attention as a function approximator to implement RL agents and policies. Once
the Deep RL model is trained, it can receive a set of observations as inputs,
and output the corresponding action to navigate the agent to the next state.
RL agents, and their policy implementations as RL models, do not necessarily
have well-defined natural language interfaces, which limits their
implementation accessibility to non-technical users. To this aim, prior work
Woodward et al. (2020) has explored the advantages of expanding the RL agent
interfaces to unstructured formats and has achieved improved task performance.

In parallel, LLM-based prompting has introduced a new interaction paradigm
between AI systems and end-users that is more intuitive and accessible to
every-day and non-technical users. In addition, recent developments in
prompting strategies, such as Chain-of-Thought (CoT) prompting Wei et al.
(2022), have improved LLMs’ abilities to decompose complex tasks into
manageable substeps and further enhance their reasoning and planning
capabilities. This motivates the exploration of whether RL problems can be
interactively solved through iterative prompting of the LLMs. Based on a set
of input and output requirements for the RL task at hand, the LLM can arguably
execute the task, evaluate whether the requirements are fully met, and
determine if additional iterations are required to satisfy all the
requirements. Subsequently, the interesting question of whether the embedded
knowledge and reasoning capabilities of LLMs can be leveraged through
iterative prompting to allow LLMs to function as RL agents remains unanswered.

Consequently, in this research, we propose a new framework that leverages the
reasoning and problem-solving capabilities of LLMs to align them for RL
problem-solving. For this process, we leverage the latest paradigm of human
and AI systems interactions, i.e., natural language prompting of LLMs. More
specifically, we make the following contributions:

  * •

We introduce an iterative prompting strategy for communicating with LLMs and
formulate RL problems based on Markov Decision Process and Q-Learning into LLM
prompting tasks. We then iteratively leverage the LLM’s knowledge and self-
reflection to optimize the RL problem and extract the desired output from the
LLM.

  * •

We propose the integration of episode generation and simulation into the
prompting chain and thus enable LLM-based policy learning, and, subsequently,
we solicit the optimal policy outcomes (i.e., episodes) from the LLM.

  * •

We provide two detailed case studies of how our approach can be effectively
implemented to extract optimal policies for the Research Scientist and Legal
Matter Intake workflows.

##  2 Prior Art

While the primary applications of LLMs are in NLP tasks, and are receiving
significant attention from the research community, the applications of LLMs in
adjacent areas such as Reinforcement Learning (RL) have the potential to grow
Wang et al. (2023); Xi et al. (2023). Previously, Janner et al. Janner et al.
(2021) proposed an RL formulation via a sequence-to-sequence learning model
based on the transformers architecture. However, this modeling lacks the more
accessible natural language and prompt-based interactions with LLMs and human
users. Consequently, recent studies have explored building RL agents with LLMs
or enabling interactions between LLMs and RL agents to improve the
accomplishment of specific tasks, to name a few, robot control Hu et al.
(2023) and game playing Xu et al. (2023).

Prior work has shown prompt engineering scenarios can help in decomposing
complex tasks and thus reach better outcomes with LLMs. Wei et al. Wei et al.
(2022) introduced Chain-of-Thought (CoT) prompting, which prompts the LLM to
solve a task in a step-by-step manner. Similarly, Tree-of-Thoughts (ToT) Yao
et al. (2023) and Graph-of-Thoughts (GoT) Besta et al. (2023) expand on CoT
and create tree and graph structures of prompts, such that different paths of
the tree can break down the task into varied sets of smaller substeps. On an
adjacent track, self-reflection prompting Shinn et al. (2023) aims to
iteratively refine the output of the LLM and get closer to the desired output.
Our approach is different from the aforementioned prompting methods as we
introduce a framework for solving RL tasks with iterative prompting techniques
that take into consideration the requirements of reaching an optimal RL
policy.

In essence, compared to prior approaches, we argue that our approach to
successfully implementing natural language and prompt-based interactions with
LLMs would be a more intuitive and potentially transformative method for
interactions between human users and AI systems. Thus, it naturally represents
the forthcoming paradigm for optimizing RL problems.

##  3 Preliminaries

An RL problem is often formally defined as a Markov Decision Process (MDP)
Puterman (2014), where MDP provides a mathematical definition of the
environment that the RL agent is interacting with. An MDP is defined as a
tuple of $(S,A,P,R,\gamma)$, where $S=\\{s_{1},s_{2},\ldots,s_{n}\\}$ denotes
a set of states that an agent can be within the environment;
$A=\\{a_{1},a_{2},\ldots,a_{m}\\}$ represents a set of actions that agent can
take at each state; $P$ is a state transition probability matrix that given
the current state $s$ and action $a$ at time $t$, represents the probability
of ending up in state $s^{\prime}$ at the next time step, i.e.,
$P(s^{\prime}|s,a)=Pr(S_{t+1}=s^{\prime}\mid S_{t}=s,A_{t}=a)$; $R$ is a
reward function that represents the expected reward for taking action $a$ in
state $s$ and moving to state $s^{\prime}$,
$R(s,a,s^{\prime})=\mathbb{E}[R_{t+1}\mid
S_{t}=s,A_{t}=a,S_{t+1}=s^{\prime}]$; $\gamma\in[0,1]$ is a discount factor
that encourages more immediate rewards for $\gamma<1$. Based on this framework
we can define a policy ($\pi$) that is a strategy (i.e., function) that the
agent can leverage to decide the next action to take based on the current
state: $\pi(a|s)=Pr(A_{t}=a\mid S_{t}=s)$.

###  3.1 Q-Learning

One popular RL algorithm is Q-Learning which uses a Q-table to store the
expected rewards for state-action pairs. The agent then uses this table to
select the best action based on its current state. The Q-value update formula
that chooses the action that maximizes the expected sum of rewards at time
step $t$ is defined as:

$\displaystyle Q(s_{t},a_{t})\leftarrow$ $\displaystyle
Q(s_{t},a_{t})+\alpha\times$
$\displaystyle[r_{t+1}+\gamma\max_{a}Q(s_{t+1},a)-Q(s_{t},a_{t})]$

where $\alpha$ is the learning rate, $\gamma$ is the discount factor,
$r_{t+1}$ is the reward received after taking action $a_{t}$ at state $s_{t}$,
and $\max_{a}Q(s_{t+1},a)$ represents the expected reward for the action that
results in maximum reward at the next state. This algorithm can be augmented
to balance both exploitation and exploration through $\epsilon$-greedy
approach. With probability $p=\epsilon$, the agent takes a randomly chosen
action from the set of all possible actions $A$ to encourage exploration, and
otherwise selects the action that maximizes the reward, i.e., exploitation:

$a_{t}=\begin{cases}a\sim\text{Uniform}(A)&:p=\epsilon\\\
\underset{a}{\arg\max}\ Q(s_{t},a)&:p=1-\epsilon\end{cases}$

##  4 Methodology

We propose utilizing LLMs to formulate the RL problem as a prompting task and
then leverage the internal knowledge of LLMs to train a Q-Learning RL agent,
and thereby optimize the desired policy. Initially, it is necessary to clearly
translate and define the communication of the requirements for an RL problem
to an LLM in text-based prompts. Therefore, we define accurate steps to
transform an RL problem into a set of prompts to elicit the desired outputs,
in this case, the optimized policy, from the LLM.

###  4.1 Prompt Framework

Based on the preliminaries discussed in the previous section for defining an
RL problem within the MDP framework, we first outline a framework to provide
the RL problem requirements through different sections of the prompt.
Specifically, based on the tuple $(S,A,P,R,\gamma)$, we initially establish
the context of the RL problem, followed by detailing states, actions, and
rewards. We then compile these necessary steps in two elaborative case
studies.

#### Context Setup.

We define the problem context for the LLM, which sets the expected behavior of
the LLM and outlines the inputs for the MDP-based RL problem. This context
setup is crucial as it aligns the LLM to interpret and respond to the RL
problem requirements.

Problem Context: You are an RL agent tasked with maximizing cumulative reward
for a given task. You will be provided with the task, states, possible actions
at each state, and rewards.

#### Task.

We then specify the RL task that LLM is intended to optimize. The placeholder
provided below indicated that it can be filled in with any arbitrary task. For
example, the task targeted for optimization could be the “Workflow of a
research scientist”.

Task: {place_holder}

#### Inputs.

Once the task is specified, we include the necessary inputs to formulate the
task as a MDP-based RL problem. We specifically provide details on states,
actions, and rewards. It is important to note that additional inputs such as
$\gamma$ and $\epsilon$ can also be provided if required. If these parameters
are not specified, the LLM will decide them.

States: $\\{s_{1},s_{2},\ldots,s_{n}\\}$ Possible actions:
$\\{a_{1},a_{2},\ldots,a_{m}\\}$ Rewards: $\\{r_{1},r_{2},\ldots,r_{k}\\}$

#### Requirements.

This section can be added if specific task-related requirements are necessary.
As a part of the requirements, we include episode simulation with the LLM. RL
agents require episodes of interactions with the environment to receive
rewards and thereby evaluate different decisions. However, real-world
interactions can often be time-consuming and costly. Hence, in line with our
concept of utilizing LLMs as RL optimizers and agents, we also leverage LLMs
as environment simulators. More specifically, in this setup, we request the
LLM to generate episodes that begin from the start state and conclude at a
terminal state. These episodes are then utilized for Q-Learning with the LLM.
Below, we have listed some example requirements for Q-Learning and we provide
more detailed case studies in the later sections.

Requirements: 1. Solve it with Q-Learning. 2. Simulate the environment for
$\\{n\\}$ episodes. 3. Episodes MUST begin at "Start" and finish at "End". 4.
…

#### Outputs.

We then instruct the LLM regarding the expected output. We ask the LLM to
return the Q table and values for the optimal episode. This way we can
evaluate the effectiveness of our framework in achieving the optimal policy.

Output: Print the Q-table, and list the state/action pairs and their
"Q-values" for the optimal episode from "Start" to "End".

#### Iterative Check.

This step revisits the output of the LLM from the last iteration to confirm if
it meets the requirements. Iterations can continue until the desired output or
the maximum number of trials is reached. Similar approaches, like self-
reflection Shinn et al. (2023), have been shown to improve the alignment with
the intended task and elicitation of desired outputs from LLMs.

Did your output satisfy all of the following requirements? If not, you MUST
take a fresh approach and execute it if necessary.

{repeat “Task/States/Actions/Rewards” and “Requirements” from above}

###  4.2 Algorithm

Algorithm 1 outlines the pseudocode for algorithmically implementing RL
problem formulation to solve an arbitrary RL optimization problem. First,
task-specific inputs are defined, followed by the crafting of a generic prompt
incorporating these inputs,GeneratePrompt(.) at Line 4. We then iterate in the
while loop on Lines 6-11 to check whether the requirements are met or the
maximum number of iterations is reached. If the LLM-generated output does not
meet the requirements, the LLM is re-prompted with the Requirements U on Line
7. Finally, the output from the LLM, which should be the Q-table and the
optimal episode for the learned policy according to the RL task, is returned.
This algorithm can be generally applied to any prompt-based LLM to iteratively
find a solution to RL problems.

Input: Problem Context $C$, Task $T$, States $S$, Actions $A$, Rewards $R$,
Requirements $U$, and iterations $M$

Output: Q-Table for Policy $\pi$

1 $Satisfied\leftarrow False$;

2 $Max\\_Iter\leftarrow M$;

3 $iter\leftarrow 0$;

$Prompt\leftarrow GeneratePrompt(C,T,S,A,R,U)$;

/* e.g., Figure 4 */

4 $Output\leftarrow LLM(Prompt)$;

$Prompt\leftarrow GeneratePrompt(T,S,A,R,U)$;

/* e.g., Figure 5 */

5 while  _$iter <Max\\_Iter$_ do

6 $Satisfied\leftarrow LLM(Prompt,Output)$;

7 if  _$Satisfied$_ then

8 return $Output$;

9

10 else

11 $Output\leftarrow LLM(Prompt)$;

12 $iter\leftarrow iter+1$

13

14return $Output$;

Algo. 1 RL problem solving with LLMs

##  5 Case Studies

In this section, we provide two illustrative case studies on how our approach
can be applied to optimize different workflows.

###  5.1 Research Scientist Workflow

In this case study, we consider our task to be optimizing the workflow of a
Research Scientist. The professional is involved in daily research activities
like literature reviews and research publications. The goal is to learn and
optimize the sequence of steps that a researcher should take to optimize their
workflow.

To address this problem with Q-Learning through LLM agents, we first create a
simulation of the environment based on the provided input states, actions, and
rewards. Then, the Q-Learning algorithm is implemented and iterated for a
predefined number of episodes (e.g., 1000) to optimize the policy. Each
episode begins from the initial state, i.e., ‘Start’, and ends at the terminal
state, i.e., ’End’. To ensure convergence, other parameters, such as the
maximum number of steps per episode can also be defined. After training, the
leaned policy is stored in the Q-table and we can identify and extract
episodes with the highest cumulative rewards. As an example, we provide the
following states, actions, and rewards to be included in the LLM’s prompt. For
simplicity, we assume equal probabilities for existing transitions from one
state to another. Later, we will discuss how this condition can be relaxed by
deriving the probabilities from actual workflows.

States = {‘Start (ST)’, ‘Initiate Research (IR)’, ‘Literature Review (LR)’,
‘Experiment Plan (EP)’, ‘Experiment Execution (EE)’, ‘Data Analysis (DA)’,
‘Manuscript Drafting (MD)’, ‘Submission to Venue (SV)’, ‘Revision of
Manuscript (RM)’, ‘Peer Review (PR)’, ‘Result Publication (RP)’, ‘End (ED)’}

Figure 1: Research Scientist workflow.

Similarly, we define the possible actions for each state as follows. For every
state $s$ we list all possible actions $a$ that can be taken from that state,
$\forall s\in S|\text{ $s$: }[a_{p1},\ldots,a_{pn}]$, and for simplicity, we
assume that actions from each state initially have the same probability.
States and action combined define the state machine for the MDP. Figure 1
depicts the state machine for the research scientist workflow. While we have
aimed to represent a meaning flow in this case study, it is for demonstration
purposes and may not necessarily fully generalize to real-life scenarios.

Actions = {‘ST’: [‘IR’], ‘IR’: [‘LR’, ‘EP’], ‘LR’: [‘EP’, ‘MD’], ‘EP’: [‘EE’],
‘EE’: [‘DA’], ‘DA’: [‘MD’, ‘EP’], ‘MD’: [‘SV’, ‘RM’], ‘SV’: [‘RM’, ‘PR’],
‘RM’: [‘EE’, ‘SV’], ‘PR’: [‘RM’, ‘RP’], ‘RP’: [‘ED’], ‘ED’: [‘ED’], ‘ELSE’:
-inf}

Additionally, we assign rewards accordingly to optimize the workflow. In this
scenario, we aim to reach the ‘End’, i.e., the terminal state, as soon as
possible. Hence, for simplicity, we assign a reward of $-1$ to every state,
except the terminal state, which receives a reward of $0$.

$r(s)=\begin{cases}-1&\text{if }s\neq\text{`End (ED)'}\\\ 0&\text{if
}s=\text{`End (ED)'}\end{cases}$

Rewards = {‘ED’: 0, else: -1}

Lastly, we define the requirements for optimizing the task of “Research
Scientist Workflow” as follows.

Requirements: 1. Solve it with Q-Learning. 2. First, simulate the environment
for 1000 episodes and fill in the Q-table. 3. Episodes MUST begin at "Start"
and finish at "End". 4. …

The requirements ensure that the LLM initially simulates the MDP for a given
number of episodes and uses Q-Learning. This method could easily generalize to
a new set of requirements and algorithms, e.g., SARSA instead of Q-Learning.
Finally, we prompt the LLM to provide the output and iteratively check whether
it satisfies the requirements.

###  5.2 Legal Matter Intake Workflow

As the second case study, we illustrate the workflow for tracking legal
matters. This flow outlines the steps that legal associates within law firms
typically follow when receiving a new legal request from a client and managing
it through to the completion of the matter. The flow generally consists of the
following steps:

1. Matter Intake (MI):The first step involves taking in the client’s request and assessing whether the firm has the required expertise and resources to undertake this matter.  2. Conflict Assessment (CA):Upon intake, the firm or attorney involved is required to assess conflicts of interest to ensure they are not precluded from representing the potential client. In some cases, CA can be bypassed, for example, for existing clients.  3. Initial Assessment (IA):An associate within the legal firm then conducts an initial assessment to identify client’s case strengths and weaknesses. The associate will also come up with a plan on how to proceed with the case. This step may circle back to CA if further information are required after the initial assessment.  4. Client Communication (CC):In this step, the legal firm communicates with the potential client, briefing them on the outcomes of the initial assessment and providing an overview of the next legal steps. This also give an opportunity to the client to discuss any questions they may have.  5. Fee and Payment (FP):This step involves communicating the legal fees or estimations associated with the case to the client, such as hourly rates or any contingency fees. This step can be potentially bypassed assuming the client is already aware of the law firm’s fees.  6. Proposal Preparation (PP):Based on the initial assessment and the client’s needs, the legal professional prepares a proposal that outlines the scope of work, the estimated cost, and the expected outcome.  7. Proposal review (PR):Once the proposal is prepared, the legal professional reviews it with the client and address any questions or concerns. After the client agrees with the plan, the matter progresses to the Case Management step.  8. Case Management (CM):This step contains the bulk of the legal research work. CM entails that the legal professional will conduct legal research, draft legal documents, and potentially represent their client in court or other legal proceedings.  9. Billing (BI):At last, the law firm manages the billing and invoicing process, which includes tracking the time spent on the case, i.e., billable hours, preparing invoices, and sending them to the client for payment.

Based on the above-mentioned possible stages of the legal matter intake
process, we produce a feasible arrangement of states and potential transitions
among them in Figure 2. It is important to reiterate that this workflow is
intended for demonstration purposes only and might not accurately capture the
potential variability of this process.

Figure 2: Legal Matter Intake workflow.

Similar to the research scientist workflow, and based on Figure 2, States and
Actions for legal matter intake workflow are outlined below.

States = {‘Start (ST)’, ‘Matter Intake (MI)’, ‘Conflict Assessment (CA)’,
‘Initial Assessment (IA)’, ‘Client Communication (CC)’, ‘Fee and Payment
(FP)’, ‘Proposal Preparation (PP)’, ‘Proposal Review (PR)’, ‘Case Management
(CM)’, ‘Billing (BI)’, ‘End (ED)’}

Actions = {‘ST’: [‘MI’], ‘MI’: [‘CA’, ‘IA’], ‘CA’: [‘IA’, ‘MI’], ‘IA’: [‘CC’,
‘CA’], ‘CC’: [‘FP’, ‘IA’, ‘PP’], ‘FP’: [‘PP’, ‘CC’], ‘PP’: [‘PR’, ‘CC’, ‘IA’],
‘PR’: [‘CM’, ‘PP’, ‘FP’], ‘CM’: [‘BI’, ‘IA’], ‘BI’: [‘CM’, ‘ED’, ‘FP’], ‘ED’:
[‘ED’], ‘ELSE’: -inf}

For each state, possible transitions to other states are listed as valid
actions. At last, we include a provision in our framework to inform the LLM
that the remaining transitions are invalid, using a special notation at the
end of the list of actions: ‘ELSE’: -inf. This approach allows the Q-Learning
algorithm to model invalid transitions with a infinitely large negative
reward. As a result, in order to maximize cumulative reward, our framework, in
principle, avoids these actions. For legal matter intake workflow, Rewards,
Requirements, and the details of episode simulation closely follow the
discussion mentioned for research scientist workflow, as such for brevity, we
avoid repeating them here.

##  6 Evaluation

We experimented with our prompting framework with the GPT-4 model (knowledge
cutoff: April 2023) for both case studies which includes four configurations,
and for 10 iterations per configuration. In cases that the GPT model initially
fails to deliver the correct answer, we re-prompt it with “Did your output
satisfy all of the following requirements? If not, you MUST take a fresh
approach and execute it if necessary.”, and we repeat the states, actions,
rewards, and requirements (refer to Figure 5 in the appendices), with a
maximum of 5 iterations, $Max\\_Iter=5$. Table 1 summarizes the distribution
for the number of iterations and the expected reward for the optimal workflow
for both research scientist and legal matter intake. We also test two $\gamma$
(discount factor) configurations. We first do not specify (UNS) the $\gamma$
value and allow the LLM to decide its own value which can be variable on each
iteration. In the second configuration, we provide a fix value for $\gamma$ as
a part of the task requirements and then measure the variability of the
calculated optimal reward for both cases. Overall, we run the experiments for
40 iterations, 20 iterations for each use case.

###  6.1 Research Scientist

In 19 out of 20 iterations for this use case, our approach resulted in the
following optimal workflow for the research scientist: Start (ST) →Initiate
Research (IR) →Literature Review (LR) →Manuscript Drafting (MD) →Submission to
Venue (SV) →Peer Review (PV) →Result Publication (RP) →End (ED). This sequence
is optimal for this use case and bypasses common workflow steps such as
experiment planning and execution. This outcome confirms that the LLM is not
outputting a commonly followed flow based on its embedded knowledge if not
optimal. The optimal reward considering $\gamma=0.9$ is $-4.7$. Variations in
the LLM’s calculation of the expected reward for UNS configuration are
generally attributed to the use of different $\gamma$ values, and we observe
that setting $\gamma$ value as part of the input requirements results in zero
reward variations.

###  6.2 Legal Matter Intake

Similarly, for legal matter intake workflow, in all 20 iterations, the LLM was
able to reach the optimal flow: Start (ST) →Matter Intake (MI) →Initial
Assessment (IA) →Client Communication (CC) →Proposal Preparation (PP)
→Proposal Review (PR) →Case Management (CM) →Billing (BI) →End (ED). The
optimal reward considering $\gamma=0.9$ is $-5.2$, and similar to the first
use case, setting the $\gamma$ value as part of workflow requirements results
in zero variance in optimal values.

Task |  ($\bm{\gamma}$) |  Iterations
$\mathcal{N}(\mu,\sigma)$ |  Optimal Reward $\mathcal{N}(\mu,\sigma)$
---|---|---|---
Research Scientist |  UNS |  $(1.6,0.5)$ |  $(-4.3,1.8)$
|  $0.9$ |  $(2.0,0.5)$ |  $(-4.7,0.0)$
Legal Matter Intake |  UNS |  $(1.8,0.4)$ |  $(-4.7,2.0)$
|  $0.9$ |  $(1.5,0.5)$ |  $(-5.2,0.0)$
Table 1: Results for Research Scientist and Legal Matter Intake workflows.
UNS: Unspecified $\gamma$ value.

For both use cases, on average, our framework was able to reach the optimal
workflows with $\leq 2$ iterations, as illustrated on "Iterations"
distribution in Table 1. Our observation was that in most cases, LLM was able
to satisfy most of the requirements in solving the task optimization in the
first prompt, i.e., Figure 4. The second prompt, i.e., Figure 5, then allowed
the LLM to revisit the requirements and fix if some of the conditions are
missed in the first trial. Overall, our approach demonstrated the
effectiveness of iterative prompting in order to ensure that LLM is navigated
towards satisfying all task optimization requirements.

##  7 Discussion

###  7.1 RL Task Complexity

Although our provided use cases might seem limited, we believe as the
available prompt context of LLMs is increasing over 100K tokens OpenAI
(2023b); Anthropic (2023), the potential to handle more complex RL problems
using our proposed framework is growing. Additionally, with the recent
announcement of Custom GPTs OpenAI (2023a), we anticipate that LLMs that are
tailored to specific user preferences and requirements will further allow us
to optimize according to our RL framework. Therefore, our approach can be
integrated as a custom NLP module that customizes an LLM for optimizing RL
tasks, while allowing the end user to interact with LLM-based agent through
natural language.

###  7.2 User Logs

In our research scientist and leggal matter intake examples, we demonstrated
how a workflow can be optimized as an RL problem. This approach can be applied
to workflow logs from various personnel in an enterprise setting instead of
the simulated workflows that we generated with the LLM. By analyzing these
workflow logs and targeting metrics to optimize, such as task efficiency, we
can optimize the workflows of employees, and thus improve overall enterprise
efficiency. Additionally, we initially relied on the LLM to generate
simulations and episodes of interactions with provided states, actions, and
rewards. Compared to the LLM simulations, the available user log data can
better represent the actions and rewards for different states and result in a
more accurate implementation of our MDP framework. This aspect serves as an
interesting avenue for our future research.

###  7.3 LLM-based Planning

In our work, we primarily aimed to describe the plan of solving the use case
for LLM to follow. An extension of our work to consider is to perform this
step with the LLM as well Valmeekam et al. (2023). Such that we first ask the
LLM to generate several plans to solve the use cases and list down the steps
involved in each. We then navigate the LLM through the steps for each plan via
our prompting platform and collect the outcomes of different plans. Lastly,
the LLM can be also serve as a judge to select the best plan based on the
outcomes of different plans.

Furthermore, every step of the planning can also be delegated to specialized
multimodal LLM agents (MLLMs) Yin et al. (2023). To provide a concrete
example, we dive into the "Conflict Assessment" in the legal matter intake
flow as illustrate in Figure 3. Assume the agent needs to check for conflicts
through an interactive application called National Conflict Lookup Service
(NCLS). The information captured during the "Matter Intake" state includes the
client’s full name along with case-related details and is available for use by
the "Conflict Assessment" MLLM. The specialized MLLM has learned how to
successfully perceive the NCLS application and act on it by entering the
client’s full name into the appropriate text field, pressing the "Check for
conflicts" button, and resulting in either a "No conflict found" or "Conflict
found" status update, so it can transition to the next ideal workflow state.

The left box on Figure 3 provides a sketch on how the MLLM agent can learn to
complete the task through interactions with the NCLS environment, similar to
ScreenAgent Niu et al. (2024). Upon entering the "Conflict Assessment" state,
and with an interactive RL-based problem-solving approach in mind, the MLLM
agent observes the state’s default prompt for the task, e.g., "Check client’s
name for conflict". Assuming a first pass at performing the task, the Learning
Element Acts on the NCLS and Senses the observations. The Critic provides
feedback on the agent’s actions and allow it to refine its approach to
performing the task. In essence, the MLLM agent will learn and iteratively
rephrase the current state’s prompt in "Check for conflicts". We pursue MLLM-
based planning and task optimization as directions for our future work.

Figure 3: MLLM agent for Conflict Assessment state.

##  8 Conclusion

With the advent of Large Language Models (LLMs), recent research has aimed to
build on their reasoning and problem-solving capabilities and adapt them to
different complex tasks. In this study, we introduced a novel approach to
frame reinforcement learning problems as LLM prompting tasks as a novel way of
solving RL problems through the user and LLM interactions. We developed an LLM
prompting methodology that represents different elements of an RL problem
based on the Markov Decision Process. Additionally, we leveraged LLMs for
episode simulation and Q-Learning optimization and derived the optimal policy
and reward based on our iterative prompting framework. In future work, we plan
to expand our work to additional tasks and optimize workflow processes in the
enterprise environment with LLM-based planning.

##  9 Limitations

#### Scope.

Although we aimed to introduce a generic approach for RL problem formulation,
a few limited cases are experimented with. Thus, while we think our prompting
technique should be generalizable to other RL tasks, further experimentation
to verify broader evidence is required.

#### LLMs Variability.

Different LLMs may have noticeable differences in their performance on our
illustrated use cases, and thus, can impact our findings. In addition, in our
use cases, we also observe that LLM outputs for different iterations can be
variable and it can often be difficult to reproduce the same exact results
through prompting. For this reason, we conducted our study on a state-of-the-
art (SOTA) LLM to establish the initial viability of our framework. We believe
that as the LLMs continue to improve and as SOTA advances, this only further
validates the feasibility of expanding our RL framework to tackle more complex
RL or planning problems. Furthermore, to achieve more consistent LLM outputs
and structured data formats, approaches such as LLM Functions OpenAI (2023b)
can be leveraged.

#### Ethical Considerations.

As we use LLMs as the basis of our work, ethical considerations evolve on the
appropriateness of the generated outputs Head et al. (2023). We reason that
the chance of harmful outputs for our application should be minimal as our
framework aligns the prompts to specific RL tasks. Furthurmore, when analyzing
real user workflow logs, it is imperative to implement steps to ensure
privacy.

## References

  * Anthropic (2023) Anthropic. 2023.  Claude 2.1.  https://www.anthropic.com/index/claude-2-1.  Accessed: December 08, 2023.
  * Besta et al. (2023) Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, et al. 2023.  Graph of thoughts: Solving elaborate problems with large language models.  _arXiv preprint arXiv:2308.09687_.
  * Dan et al. (2023) Yuhao Dan, Zhikai Lei, Yiyang Gu, Yong Li, Jianghao Yin, Jiaju Lin, Linhao Ye, Zhiyan Tie, Yougen Zhou, Yilei Wang, et al. 2023.  Educhat: A large-scale language model-based chatbot system for intelligent education.  _arXiv preprint arXiv:2308.02773_.
  * Hakimi Parizi et al. (2023) Ali Hakimi Parizi, Yuyang Liu, Prudhvi Nokku, Sina Gholamian, and David Emerson. 2023.  A comparative study of prompting strategies for legal text classification.  In _Proceedings of the Natural Legal Language Processing Workshop 2023_ , pages 258–265, Singapore. Association for Computational Linguistics.
  * Head et al. (2023) Cari Beth Head, Paul Jasper, Matthew McConnachie, Linda Raftree, and Grace Higdon. 2023.  Large language model applications for evaluation: Opportunities and ethical implications.  _New Directions for Evaluation_ , 2023(178-179):33–46.
  * Hu et al. (2023) Bin Hu, Chenyang Zhao, Pu Zhang, Zihao Zhou, Yuanhang Yang, Zenglin Xu, and Bin Liu. 2023.  Enabling intelligent interactions between an agent and an llm: A reinforcement learning approach.
  * Janner et al. (2021) Michael Janner, Qiyang Li, and Sergey Levine. 2021.  Offline reinforcement learning as one big sequence modeling problem.  _Advances in neural information processing systems_ , 34:1273–1286.
  * Kang et al. (2023) Inwon Kang, Sikai Ruan, Tyler Ho, Jui-Chien Lin, Farhad Mohsin, Oshani Seneviratne, and Lirong Xia. 2023.  Llm-augmented preference learning from natural language.  _arXiv preprint arXiv:2310.08523_.
  * Liévin et al. (2022) Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther. 2022.  Can large language models reason about medical questions?  _arXiv preprint arXiv:2207.08143_.
  * Limna et al. (2023) Pongsakorn Limna, Tanpat Kraiwanit, Kris Jangjarat, Prapasiri Klayklung, and Piyawatjana Chocksathaporn. 2023.  The use of chatgpt in the digital era: Perspectives on chatbot implementation.  _Journal of Applied Learning and Teaching_ , 6(1).
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533.
  * Niu et al. (2024) Runliang Niu, Jindong Li, Shiqi Wang, Yali Fu, Xiyu Hu, Xueyuan Leng, He Kong, Yi Chang, and Qi Wang. 2024.  Screenagent: A vision language model-driven computer control agent.  _arXiv preprint arXiv:2402.07945_.
  * OpenAI (2023a) OpenAI. 2023a.  Chatgpt plugins.  https://openai.com/blog/chatgpt-plugins.  Accessed: December 10, 2023.
  * OpenAI (2023b) OpenAI. 2023b.  Function calling - openai api.  https://platform.openai.com/docs/guides/function-calling.  Accessed: 2024-02-29.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * OpenAI (2023a) OpenAI. 2023a.  Introducing GPTs.  https://openai.com/blog/introducing-gpts/.  Accessed: December 08, 2023.
  * OpenAI (2023b) OpenAI. 2023b.  New models and developer products announced at devday.  https://openai.com/blog/new-models-and-developer-products-announced-at-devday/.  Accessed: December 08, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Puterman (2014) Martin L Puterman. 2014.  _Markov decision processes: discrete stochastic dynamic programming_.  John Wiley & Sons.
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_.
  * Vaithilingam et al. (2022) Priyan Vaithilingam, Tianyi Zhang, and Elena L Glassman. 2022.  Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models.  In _Chi conference on human factors in computing systems extended abstracts_ , pages 1–7.
  * Valmeekam et al. (2023) Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. 2023.  On the planning abilities of large language models - a critical investigation.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Wang et al. (2023) Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837.
  * Woodward et al. (2020) Mark Woodward, Chelsea Finn, and Karol Hausman. 2020.  Learning to interactively learn and assist.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 2535–2543.
  * Xi et al. (2023) Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023.  The rise and potential of large language model based agents: A survey.  _arXiv preprint arXiv:2309.07864_.
  * Xu et al. (2023) Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2023.  Language agents with reinforcement learning for strategic play in the werewolf game.  _arXiv preprint arXiv:2310.18940_.
  * Yao et al. (2023) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_.
  * Yin et al. (2023) Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023\.  A survey on multimodal large language models.  _arXiv preprint arXiv:2306.13549_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_.

Problem Context: You are an RL agent tasked with maximizing cumulative reward
for a given task. You will be provided with the task, states, possible actions
at each state, and rewards. Task: Workflow of a research scientist. States =
{‘Start (ST)’, ‘Initiate Research (IR)’, ‘Literature Review (LR)’, ‘Experiment
Plan (EP)’, ‘Experiment Execution (EE)’, ‘Data Analysis (DA)’, ‘Manuscript
Drafting (MD)’, ‘Submission to Venue (SV)’, ‘Revision of Manuscript (RM)’,
‘Peer Review (PR)’, ‘Result Publication (RP)’, ‘End (ED)’} Actions = {‘ST’:
[‘IR’], ‘IR’: [‘LR’, ‘EP’], ‘LR’: [‘EP’, ‘MD’], ‘EP’: [‘EE’], ‘EE’: [‘DA’],
‘DA’: [‘MD’, ‘EP’], ‘MD’: [‘SV’, ‘RM’], ‘SV’: [‘RM’, ‘PR’], ‘RM’: [‘EE’,
‘SV’], ‘PR’: [‘RM’, ‘RP’], ‘RP’: [‘ED’], ‘ED’: [‘ED’], ‘ELSE’: -inf} Rewards =
{‘ED’: 0, else: -1} Requirements: 1. Solve it with Q-learning. gamma=0.9 2.
Transitions from one state to another are only allowed based on provided
’Actions’. Other actions are not possible. 3. First, simulate the environment
for 1000 episodes and fill in the Q-table for states. If an episode goes more
than 100 steps terminate that episode. 4. Episodes MUST begin at "Start" and
finish at "End". Output:
Print the Q-table, and list the state/action pairs and their "Q-values" for
the optimal episode from "Start" to "End". Figure 4: This prompt is used to
model the research scientist workflow based on the Markov Decision Process.
Did your output satisfy all of the following requirements? If not, you MUST
take a fresh approach and execute it if necessary.
Task: Workflow of a research scientist. States:
$\\{s_{1},s_{2},\ldots,s_{n}\\}$ Actions: $\\{a_{1},a_{2},\ldots,a_{m}\\}$
Rewards: $\\{r_{1},r_{2},\ldots,r_{k}\\}$ 1. Solve it with Q-Learning.
gamma=0.9 2. Transitions from one state to another are only allowed based on
provided ’Actions’. Transitions from one state to another are only allowed
based on possible actions. 3. First, simulate the environment for 1000
episodes and fill in the Q-table for states. If an episode goes more than 100
steps terminate that episode. 4. Episodes MUST begin at "Start" and finish at
"End". 5. Print the state/action pairs and their "Q-values" for the optimal
episode from "Start" to "End". Figure 5: This prompt is used for iterative
verification of the LLM outputs until reaching the desired outputs. Task,
States, Actions, and Rewards are place-holders for use case specific values,
e.g., values from Figure 4 can be used here for research scientist workflow.

##  Appendix A Prompts

Figures 4 and 5 compile all the prompts used for our demonstrated use cases.
Because of the generalizability in their designs, these prompts can readily
serve as a template for formulating additional RL-based use cases. In addition
to leveraging self-reflection Shinn et al. (2023), we also implemented
capitalization of critical instructions to emphasize the output requirements
and ensure more consistent generations from the LLM Kang et al. (2023). The
aforementioned prompting techniques, in addition to the code generation
capabilities of GPT OpenAI (2023a), allow the LLM to accurately implement and
simulate the MDP based on the provided input and thereby produce the optimal
policy.