# Model based reinforcement learning for stock trading optimization

Huifang Huang, Ting Gao, Luxuan Yang, Yi Gui, Jin Guo, Peng Zhang

(September 2021)

###### Abstract

Reinforcement learning (RL) is gaining attention by more and more researchers
in quantitative finance as the agent-environment interaction framework is
aligned with decision making process in many business problems. Most of the
current financial applications using RL algorithms are based on model-free
method, which still faces stability and adaptivity challenges. As lots of
cutting-edge model-based reinforcement learning (MBRL) algorithms mature in
applications such as video games or robotics, we design a new approach that
leverages resistance and support (RS) level as regularization terms for action
in MBRL, to improve the algorithm’s efficiency and stability. From the
experiment results, we can see RS level, as a market timing technique,
enhances the performance of pure MBRL models in terms of various measurements
and obtains better profit gain with less riskiness. Besides, our proposed
method even resists big drop (less maximum drawdown) during COVID-19 pandemic
period when the financial market got unpredictable crisis. Explanations on why
control of resistance and support level can boost MBRL is also investigated
through numerical experiments, such as loss of actor-critic network and
prediction error of the transition dynamical model. It shows that RS
indicators indeed help the MBRL algorithms to converge faster at early stage
and obtain smaller critic loss as training episodes increase.

##  1 Introduction

As the development of modern deep learning techniques, more and more cutting-
edge neural network structures have been widely used in various research and
application fields. These machine learning frameworks in return inspire
researchers from different background to design better neural network
architectures to improve the model performance and solve more complex
scenarios. One of the promising applications, quantitative finance, has
recently been taking advantage of AI techniques to make lots of innovative
algorithmic trading strategies for financial investment and portfolio
optimization.

Among all the machine learning frameworks, reinforcement learning (RL) has an
unique advantage that the interactive training process is aligned with human
decision making. During the past years, two basic optimization ways,
Q-learning and Actor-Critic have been developed to solve many RL problems, as
well as the combination of the two. Take financial applications for example,
Pastore et al. [Pastore et al.(2016)] analyze data on 46 players from online
games in financial markets and test whether the Q-Learning could capture these
players’ behaviours based on a riskiness measure. Their results indicate that
not all players are short-sighted, which contradicts with the naive-investor
hypothesis. Besides, Li et al. [Li et al.(2019b)] study the benefits of the
three different classical deep RL models DQN [Mnih et al.(2013)], Double DQN
[Van Hasselt et al.(2016)] and Dueling DQN [Wang et al.(2016)] by predicting
the stock price and conclude that DQN has the best performance. In addition,
some researchers simulate and compare the improved deep RL method with the
Adaboost algorithm and propose a hybrid solution. Interestingly, Lee et al.
(2019) [Lee et al.(2019)] apply CNN to a deep Q-network which takes stock
prices and volumes chart images as input to predict global stock markets. On
the policy gradient side, Kang et al. (2018) [Kang et al.(2018)] utilize the
state-of-art Asynchronous Advantage Actor-Critic algorithm (A3C [Mnih et
al.(2016)]) to solve the portfolio management problem and design a standalone
deep RL model. Moreover, Li et al. [Li et al.(2019a)] propose an adaptive deep
deterministic RL method (Adaptive DDPG) for some portfolio allocation task,
which incorporates optimistic or pessimistic deep RL algorithm based on the
influence of prediction error. Through analyzing data from Dow Jones 30
component stocks, the trading strategy outperforms traditional DDPG method
[Lillicrap et al.(2015)]. Sutta [Sornmayura(2019)] compares the performance of
an AI agent to the results of the buy-and-hold strategy and the expert trader
by testing 15 years’ forex data market with a paired t-Test. The findings show
that AI can beat the buy $\&$ hold strategy and commodity trading advisor in
FOREX for both EURUSD and USDJPY currency pairs.

However, there are still complex and challenging issues with reinforcement
learning for financial environment. First of all, financial marketing data, in
most scenarios appear to have complex non-linearity, even complex chaotic
behaviors and uncertainties including non-Gaussian noise, which causes finance
time series data has distribution shift [Cai and Wei(2020)] over time.
Besides, hidden interactions among different agencies can be unpredictably
difficult to understand. As it is shown from lots of stock marketing analysis,
agencies with large capital investment can sometimes cause dramatic price
fluctuation, leading to marketing panic and instability among the public.
Therefore, scholars are actively seeking intrinsic mechanics to solve the
above problems. On one hand, various data denoise methods are utilized to
preprocess the financial time series data. For example, Bao W, et al. (2017)
[Bao et al.(2017)] propose a new model for stock prediction, wavelet
transforms for decomposing and eliminating noise of the stock price time
series, which helps stacked auto-encoders (SAEs) for extracted high-level deep
features and long-short term memory (LSTM) to better forecast the stock price
and improve predictive accuracy and profitability performance. On the other
hand, researchers also create many kinds of indexes to optimize portfolio
gain. In short, all these make it hard for us to build an reinforcement
learning system which directly interacts with complex real marketing
environment, and this inspires us to investigate model-based RL for
algorithmic trading strategy optimization.

Most of the existing literature is based on model-free reinforcement learning
(MFRL) method. However, these algorithms are quite expensive to train,
considering high risk of investment loss at early stage in real financial
market environment, even using simulated domains (Mnih et al., 2015 [Mnih et
al.(2015)], Lillicrap et al., 2015 [Lillicrap et al.(2015)], Schulman et al.,
2017 [Schulman et al.(2017)]). A promising direction for improving sample
efficiency is to explore model-based reinforcement learning (MBRL) methods [Yu
et al.(2019)]. DynaQ [Peng et al.(2018)], PETS [Chua et al.(2018)], MBPO
[Janner et al.(2019)] are powerful model-based reinforcement algorithms in Gym
environment. Model-based RL algorithms can attain excellent sample efficiency,
but often lag behind the best model-free algorithms in terms of asymptotic
performance. Chua, Calandra et al. (2019) [Chua et al.(2018)] study how to
bridge this gap by employing uncertainty-aware dynamical models. They propose
a novel algorithm called PETS, which approaches the asymptotic performance of
several benchmark model-free algorithms, while requiring significantly fewer
samples. Janner et al. (2019) [Janner et al.(2019)] propose model-based Policy
Optimization (MBPO) algorithm to boost PETS through monotonic improvement at
each step and get state-of-the-art performance. All these excellent work
inspires us to apply model-based RL algorithms into the complex financial
marketing environment, and to the best of our knowledge, rare literature has
been found on utilizing MBRL into quantitative financial modeling.

Quantitative finance is an interdisciplinary subject that involves many
researchers from various fields with different background knowledge, and lots
of promising algorithmic trading strategies in real-world problems utilize
some financial technique analysis. For example, to find the best market timing
for trading, resistance level (the price at which the trader believes that the
seller’s power begins to overtake the buyer’s) and support level (the price at
which traders believe that the buyer’s power begins to overtake the seller’s)
are often considered as selling and buying market timing indicators
respectively. Therefore, we leverage the timing selection power of Resistance
Support Relative Strength (RSRS) and apply them into the classical MBRL
algorithms, aiming to find better optimized policies that can improve the MBRL
algorithms to be more stable, adaptive and efficient.

Overall, this paper makes the following major contributions:

  * •

Marriage of RSRS and MBRL. We design a financial environment for classic
model-based reinforcement learning (MBRL) algorithms and equip Resistance
Support Relative Strength (RSRS) into the RL framework to improve model
stability and adaptivity.

  * •

Business Performance. The Resistance Support Relative Strength (RSRS), as a
powerful timing selection technique, when embedded into a couple of MBRL
algorithms, help pure MBRL strategy get high improvement in seven different
measurements, from both risk management and portfolio optimization point of
view.

  * •

Model Performance. To better explain the performance discrepancies between
MBRL with and without RSRS as regularization for the actions, we also compare
the loss of actor-critic network among all the algorithms. From the plots of
critic loss, we can see that RSRS indeed help the selected MBRL algorithms to
converge faster at early stage and obtain lower error when the models are
convergent, therefore improves the model efficiency.

  * •

Performance of Transition Dynamics. There is some close connection between SAC
for policy optimization under transition dynamic modeling and stochastic
optimal control with stochastic differential equations as state constraints.
We also check the prediction error of the transition dynamical model in terms
of different coordinates projections, considering our state space is complex
and high dimensional.

We structure our paper as the following sessions: First, in Section 2.1, we
introduce some background knowledge of model-based RL, and then explain our
financial environment setup under model-based RL framework in Section 2.2 and
2.3. We also interpret the transition dynamic from a dynamical system point of
view and describe the Gaussian Process as some stochastic differential
equation (SDE), and correspondingly, the policy optimization of SAC is
connected with stochastic optimal control under the constraint of a SDE in
Section 2.4. Next, we explain the RSRS algorithm in detail in Section 3, and
present the pseudo code of our proposed model. Moreover, all the experiment
results are shown in Section 4, as well as detailed explanations on how and
why our proposed algorithm is better than pure MBRL without RSRS. Finally, we
summarize our conclusions and future research directions in Section 5.

##  2 Framework

###  2.1 Background: Model-based RL

Reinforcement learning aims to learn an optimal policy so as to maximize the
expectation of cumulative rewards in the process of interacting with the
environment. It mainly includes model-free and model-based methods in terms of
whether the agent interacts with purely real environment.

Model-free methods can be used to handle many complex tasks with the best
progressive performance. But usually this requires a lot of interaction with
the environment and has relatively high requirements of computing power.
However, model-based methods focus on building the environment model which we
called transition dynamic model, with high sampling efficiency. One of the
goals usually lies in improving the accuracy of estimations on state
transition probability. Moreover, when the learned transition model is close
to the real environment, the optimal strategy can be found directly through
some planning algorithms.The traditional planning pipeline includes two parts:
path search and trajectory optimization.

To be more specific, for path search, when the action space is continuous, CEM
(Cross-Entropy Method) is usually used. It is a Monte Carlo method mainly used
for optimization and importance sampling. When the action space is discrete,
the MCTS (Monte Carlo Tree Search) is considered when the search space is
huge. Trajectory optimization refers to the problem of solving the objective
function with constraints, that is, the optimal control problem. Trajectory
optimization generally includes shooting and collocation methods. Shooting
methods are more suitable for simple control and problems with no path
restrictions while collocation methods are more suitable for complex control
and problems with path restrictions.

In PETS, parametric models fitted by a neural network, and propose the
algorithm by combining the PE (Probabilistic Ensembles) model and the TS
(Trajectory Sampling) planning method. With cross entropy method (CEM), they
point out a better direction for the action, then sample based on it. Due to a
certain deviation between the established environmental model and the real
model, Janner and Fu et al. (2019) [Janner et al.(2019)] proposed a novel
model-based algorithm MBPO to improve PETs. After formulating and analyzing a
model-based RL with monotonic improvement at each step, they study the model
usage in policy optimization both theoretically and empirically. Then they
demonstrate that a simple procedure of using short rollout planning from real
data and SAC (soft actor-critic) [Haarnoja et al.(2018)] for optimization.

In complex and noisy environments, transition model error can be large.
Learning a good policy requires an accurate model, while getting an accurate
model requires a lot of interaction with the real environment. Pan and He
(2020) proposed M2AC (Masked Model based Actor-Critic) [Pan et al.(2020)],
which reduces the influence of model error through masking mechanism, and
alleviates the overfitting problems of MBRL in the case of small data size and
large noise in real environment. It is theoretically proved that the gap
between the real return and the masked ensemble model rollout value can be
bounded when the individual model being used is with a small error.

###  2.2 Space and Reward Definition

By considering our agency (investor) as the intelligent agent and financial
market as the corresponding environment, we model the stock trading problem as
a Markov Decision Process (MDP).

$\bullet$ State space $\mathcal{S}=[B,P,W,I]$: State space $\mathcal{S}$ is
the market information collected. $\forall s_{t}\in\mathcal{S}$ is the state
of the agent at time $t$, which includes four parts information: account
balance for agency $B_{t}$, current stock price $P_{t}$, cumulative holding
$W_{t}$, technical indicators $I_{t}$. Here the technical indicators $I_{t}$
is consisted of seven common technology factors: MACD [Appel(2003)], SMA30,
SMA60, BOLL [Bollinger(1992)], RSI [Ţăran-Moroşan(2011)], CCI [Lai et
al.(2020)], ADX. Table 1 describes the notations of state $s_{t}$ in detail.

We assume the experiment consist of D stocks.

Table 1: Notations of State  Notation | Defination
---|---
$B_{t}$ |  | Account balance at time t; $B_{t}\in\mathbb{R}_{+}$
---
$P_{t}$ |  | daily closing price of each stock; $P_{t}\in\mathbb{R}_{+}^{D}$
---
$W_{t}$ |  | cumulative holding shares of each stock; $W_{t}\in\mathbb{Z}_{+}^{D}$
---
MACD |  | Moving Average Convergence Divergence:
---
a momentum indicator displays trend
SMA30 |  | 30 day Simple Moving Average:
---
30 day closing price equal weighted average
SMA60 |  | 60 day Simple Moving Average:
---
60 day closing price equal weighted average
BOLL |  | Bollinger Bands:
---
judges the medium and long-term movement trend
RSI |  | Relative Strength Index:
---
identifies inflection points of a trend
CCI |  | Commodity channel index:
---
helps to find the degree of deviation of the price
ADX |  | Average Directional Index:
---
determines the strength of a trend

$\bullet$ Action Space $\mathcal{A}$: Action space $\mathcal{A}$ is a set of
available operations during the transaction on all D stocks. $\forall
a_{t}\in\mathcal{A}$ is the action taken by the agent at time $t$, assumed to
be finite dimensional and continuous. Here $a_{t}$ is a D-dimensional vector,
where the $i_{th}$ dimension represents the action performed on the $i_{th}$
stock.

Noting $W^{i}_{t}$ as the cumulative shares of $i_{th}$ stock in time $t$, the
available actions of the stock include buying, holding and selling. We assume
the maximum trading volume for a single stock at one step is 100 shares. The
details as follows:
• Buying: $a^{i}_{t}=+h$, $h$ shares can be bought and it leads to
$W^{i}_{t+1}=W^{i}_{t}+h$.
• Selling: $a^{i}_{t}=-h$, $h$ shares can be sold from the current holdings.
In this case, $W^{i}_{t+1}=W^{i}_{t}-h$.
• Holding: $a_{t}=0$, that means no change in $W^{i}_{t}$.
where $h\in[0,100]$ is a positive integer.

$\bullet$ Reward $r_{t}$: The reward function is a mapping
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. As is shown
in later equation (3), the direct reward $r_{t}$ of taking the action $a_{t}$
in state $s_{t}$ is defined as the percentage of asset amount change:

1)The asset amount of the agent is the sum of the remaining investment funds
and the current value of stocks holding. We note the asset amount of the agent
in time $t$ is $Asset_{t}$, and the calculation formula as equation:

$Asset_{t}=(B_{t}+P_{t}^{T}\cdot
W_{t})=B_{0}-\sum_{\tau=0}^{t}P_{\tau}^{T}\cdot
a_{\tau}+P_{t+1}^{T}\sum_{\tau=0}^{t}a_{\tau}$ (1)

2) Consider the transition cost, we notes the cost as $C_{t}$:

$C_{t}=P_{t}^{T}\cdot|a_{t}|\cdot cost_{percentage}$ (2)

the absolute value of $a_{t}$ here means taking the absolute value of each
component of a without changing the dimension of the vector $a_{t}$.

3) The calculation formula of direct reward $r_{t}$ is:

$r_{t}=\frac{Asset_{t+1}-Asset_{t}-C_{t}}{Asset_{t}}\times 100$ (3)

where $B_{t},P_{t}$ is shown as Table 1; $a_{t}$ is the action given by agent
at time $t$;$P_{t}^{T}\cdot W_{t}$ is the inner product of vectors.

For RL problem, the agent’s objective is to maximize its expected reward. The
reward can assess the quality of each action. What’s more, the agent optimizes
the strategy under the guidance of the reward. Noting the cumulative reward as
$R_{t}$ in time t:

$R_{t}=\Sigma_{\tau=1}^{t}r_{\tau}$ (4)

###  2.3 Transition Dynamic

One of the classification criteria of RL algorithms is whether dynamic model
of the environment exists. Model-based RL is appealing because the dynamic
model is reward-independent and it can easily benefit from all of the advances
in deep supervised learning to utilize high-capacity models [Chua et
al.(2018)]. However, the disadvantage of this method is that in many learning
tasks, the agent is often hard to obtain an accurate model of the real
environment, leading to big accumulated error when interacting with virtual
environment. PETS, MBPO and M2AC are robust algorithm to take a step toward
narrowing the gap between model-based and model-free RL methods. Therefore, we
apply these models to the financial market for stock trading optimization. We
start with the definition of transition probability in the following.

$\bullet$ Transition Probability $\mathcal{P}$: The change of market
conditions is abstracted as the state transition function.
$\mathcal{P}:S\times A\times S\rightarrow[0,1]$ is a function of probabilities
of state transitions.

$P_{s,s^{\prime}}^{a}=\mathcal{P}(s_{t+1}=s^{\prime}|s_{t}=s,a_{t}=a)$

.

In PETS, MBPO and M2AC, ensemble Gaussian Process is used to predict state
transition dynamics. From stochastic analysis, we know that the state $s_{t}$
satisfies the following stochastic differential equation (SDE):

$dX_{t}=b(X_{t},u_{t})d_{t}+\sigma(X_{t},u_{t})dw_{t}$ (5)

where the initial condition $X_{0}=x\in\Omega$, and
$a_{t}\subset\mathbb{R}^{da}$ is an $\mathbb{F}_{t}$-adapted control field and
$w_{t}$ is a $d_{w}$-dimensional $\mathbb{F}_{t}$-standard Brownian Motion.

$\bullet$ Policy $\pi$: Policy is a mapping characterized by the policy
$\pi:\mathcal{S}\rightarrow\mathcal{A}$. $\forall t\in{1,...,T}$. It is an
action sequence given by agent, the objective of policy $\pi$ given by agent
is to maximize its finally expected portfolio value.

The agent in state $s_{t}\in\mathcal{S}$ takes an action $a_{t}\in\mathcal{A}$
follows policy $\pi$, receives the reward $r_{t}=\mathcal{R}(s_{t},a_{t})$ and
transits to the next state $s_{t+1}$ according to the transition probability
$\mathcal{P}$. The agent interacts with the environment to collect the
trajectory, and then updates the strategy. The core optimization problem in RL
is to find the optimal strategy $\pi^{*}$ that optimize total accumulated
reward.

###  2.4 Policy Optimization

The actor-critic bridges the gap between value function approximation methods
and policy gradient methods for RL. This approach has proven to be able to
learn and adapt to large and complex environments, and has been used to play
popular video games, such as Doom [Wu and Tian(2017)]. Thus, the actor-critic
approach is applying for trading with a large stock portfolio [Xiong et
al.(2018)]. The Proximal Policy Optimization (PPO) Algorithm(Schulman et al.,
2017 [Schulman et al.(2017)]) is an actor-critic off-policy model-free RL
algorithm for discrete control and continuous control. However, it faces
serious sample inefficiency and requires a huge amount of sampling to learn,
which is unacceptable for real stock trading market training. Another typical
example of actor-critic algorithms is Deep deterministic policy gradient
(DDPG) algorithm (Lillicrap et al., 2015 [Lillicrap et al.(2015)]).
Researchers explore the potential of RL for stock trading with DDPG algorithm
and achieve some good results [Emami(2016), Azhikodan et al.(2019)]. While it
is very sensitive to various hyper-parameters during actual training, so the
excellent performance of DDPG on various benchmarks is actually crafted and it
is difficult to solve a lot of specific problem.

Tuomas Haarnoja proposed SAC Algorithm [Haarnoja et al.(2018)], an off-policy
method, to solve some issues of sampling inefficiency in PPO, and sensitivity
of hyper-parameters in DDPG. The biggest difference in SAC is its objective of
entropy maximization, which is a great benefit for maintaining stochasticity
in policy selection, and hence can help agent to search state space more
completely so as to avoid local optimal and sensitivity.

For random variable $x$ with probability density $p$, the entropy $H$ of $x$
can be defined as:

$H(p)=\underset{x\thicksim p}{\mathbb{E}}[-logp(x)]$ (6)

Consider entropy regularized RL, the reward will be added with an extra term
which is the entropy. At the same time, the influence of current action on
future rewards will be weakened with time going by, so we use discounted
reward instead of the directed reward. The soft value function (soft Q
function) becomes:

$\displaystyle Q_{soft}^{\pi}(s,a)=$
$\displaystyle\underset{(s_{t},a_{t})\thicksim\rho_{\pi}}{\mathbb{E}}[\Sigma_{t=0}^{\infty}\gamma^{t}r(s_{t},a_{t})+$
(7)
$\displaystyle\alpha\cdot\Sigma_{t=1}^{\infty}\gamma^{t}H(\pi(\cdot|s_{t}))|s_{0}=s,a_{0}=a]$

where $\rho_{\pi}$ represents the distribution of state-action pair of the
agent under policy $\pi$. Correspondingly, soft V function becomes:

$V_{soft}^{\pi}(s)=\underset{(s_{t},a_{t})\thicksim\rho_{\pi}}{\mathbb{E}}[\Sigma_{t=0}^{\infty}\gamma^{t}(r(s_{t},a_{t})+\alpha
H(\pi(\cdot|s_{t})))|s_{0}=s]$ (8)

With these definitions, $V^{\pi}$ and $Q^{\pi}$ and connected by

$V^{\pi}(s)=\underset{a\thicksim\pi}{\mathbb{E}}[Q^{\pi}(s,a)+\alpha
H(\pi(\cdot|s))]$ (9)

Thus, the Bellman equation can be written as:

$Q_{soft}^{\pi}(s,a)=\underset{s^{\prime}\thicksim\mathcal{P}(s^{\prime}|s,a)}{\mathbb{E}}[r(s,a)+\gamma(Q_{soft}^{\pi}(s^{\prime},a^{\prime})+\alpha
H(\pi(\cdot|s^{\prime})))]$ (10)

There are several versions of SAC for the past years. Here we use the older
version that learns a value function $V$ in addition to $Q$ function.

$Q_{soft}^{\pi}(s,a)=\underset{s^{\prime}\thicksim\mathcal{P}(s^{\prime}|s,a)}{\mathbb{E}}[r(s,a)+\gamma(V_{soft}^{\pi}(s^{\prime})]$
(11)

Thus, iteration of value function is

$V_{soft}^{\pi}(s)=\underset{a\thicksim\pi}{\mathbb{E}}[Q_{soft}^{\pi}(s,a)-\alpha
log\pi(a|s)]$ (12)

By clipped donate Q trick, the loss function of Q-network in SAC is:

$\mathcal{L}(\phi,D)=\underset{(s,a,r,s^{\prime})\in
D)}{\mathbb{E}}[Q_{\phi}(s,a)-y(r,s^{\prime},d)]^{2}$ (13)

where $Q_{\phi}$ and $Q_{target}$ are neural networks and

$y(r,s^{\prime},d)=r+\gamma(1-d)(\underset{j=1,2}{min}Q_{target,j}(s^{\prime},\tilde{a}^{\prime})-\alpha
log\pi_{\theta}(\tilde{a}^{\prime}|s^{\prime}))$ (14)

where $\tilde{a}^{\prime}\thicksim\pi_{\theta}(\cdot|s^{\prime})$ in which a
sample from $\pi_{\theta}(\cdot|s)$ is drawn by Gaussian distribution.

Now we formulate this problem as a stochastic optimal control problem and the
stochastic optimal control problem can be written as:

$\mathcal{J}^{u}(x)=\mathbb{E}[\int_{0}^{\infty}f(X_{s},u_{s})e^{-\gamma
s}ds|X_{0}^{u}=x]$ (15)

constrained on equation (5).
Define

$V(x)=\underset{u}{inf}\mathcal{J}^{u}(x)-\alpha log\pi(a|s)$ (16)

Then V satisfies the time-dependent HJB equation:

$\underset{u}{inf}{\mathcal{L}^{u}V(x,u)+f(x,u)-\gamma V(x)}=0$ (17)

where
$\mathcal{L}^{u}V(x)=\frac{1}{2}Tr(\sigma\sigma^{T}Hess(V))(x,u)+b(x,u)^{T}\nabla
V(x)$ is the generator of SDE (5).

##  3 Marriage of MBRL and RSRS

In this article, we use the Resistance Support Relative Strength (RSRS) market
timing indicator 111Junwei Liu, Xiaoxiao Zhou. Market Timing Based on
Resistance Support Relative Strength (RSRS). Everbright Securities Technical
Timing Series Report Series I, 2017.5.1. It is a technical indicator to choose
the best time for buying and selling by measuring the relative strength of
support and resistance [Lloyd Sr(2013)]. With the definitions of support level
and resistance levels in the financial field, we have:

$\bullet$ Resistance level: The price at which the trader believes that the
seller’s power begins to overtake the buyer’s, making it difficult for the
price to continue to rise or to pull back from the falling price.

$\bullet$ Support level: The price at which traders believe that the buyer’s
power begins to overtake the seller’s, thereby stopping the decline or
rebounding the rising price.

The RSRS indicator no longer regards resistance and support as a fixed value,
but as a variable in that report. This indicator is an expected judgment of
traders on the top and bottom of the current market state.

Researchers replaced the specific price threshold of support and resistance
with the relative strength of the daily high and low price changes. Let $\mu$
be the mean value of the historical RSRS slope, and $\sigma$ be the standard
deviation of the historical slope. Denote the resistance level as $S_{buy}$,
where $S_{buy}=\mu+\sigma$. Then denote the support level as $S_{sell}$, where
$S_{sell}=\mu-\sigma$. When the RSRS slope is greater than $S_{buy}$, buy the
whole position, and when it is less than $S_{sell}$, sell and close the
position. In this way, the drawdown can be controlled and the profit
performance of the strategy will be guaranteed.

###  3.1 RSRS Indicator

The RSRS indicator uses the magnitude of the high price change when the low
price of a certain stock changes by 1 unit to measure the strength of support
and resistance. That is, to regress the high price series and the low price
series in a certain period. The slope $\beta$ obtained by the model is a
specific indicator for measuring resistance support relative strength.

Taking the high price time series and low price time series of the previous N
days of a certain stock. We calculate the RSRS slope of the day:

$high=\alpha+\beta\times low+\epsilon$ (18)

where $\epsilon\backsim\mathcal{N}(0,\sigma_{\epsilon}^{2})$

###  3.2 Right Standard Score

For stocks in different periods of the market, the mean value of the slope
will fluctuate significantly. Therefore, it is not appropriate to directly use
the mean value of the slope as the timing index. It is a good choice to
standardize the slope value.

1) Taking the RSRS slope time series of the previous M days of a certain
stock. We calculate the standard score $RSRS_{std}$ of the RSRS slope of the
day:

$RSRS_{std}=\frac{RSRS-\mu_{M}}{\sigma_{M}}$ (19)

where $\mu_{M}$ is the average slope of the previous M days and $\sigma_{M}$
is the standard deviation of the previous M days.

2) In fact, when the slope is used to quantify the relative strength of
resistance support, its quantitative effect is largely affected by the effect
of the fitting itself. So further, we consider the fitting effect and weight
the standard scores with the coefficient of determination (the R-square value
in the regression model) to reduce the impact of the RSRS standard score,
which has a large absolute value but poor fitting effect, on the strategy.

$RSRS_{cor}=RSRS_{std}\times R^{2}$ (20)

where $R^{2}$ is the coefficient of determination in the regression model of
the day.

3) The RSRS indicator has excellent left-side forecasting ability by itself.
So we use the significant right-biased standard score correction as the timing
index in this paper.

$RSRS_{rightdev}=RSRS_{cor}\times RSRS$ (21)

4) According to the analysis before, the RSRS timing index is defined as:

$RSRS_{rightdev}=\frac{RSRS-\mu_{M}}{\sigma_{M}}\times R^{2}\times RSRS$ (22)

###  3.3 Proposed Model

Based on RSRS indicator explained in the previous session, we know it’s a
relatively strong policy to resist deep drop when a stock has big plunge
during some urgent time period. Therefore, we believe that combining this
strategy to classical RL algorithms can help to get the optimal strategy with
less volatility and more stability, meaning that we could get higher Sharpe
ratio and lower maximum drawdown. Among all the recent state-of-the-art model
based RL algorithms, we choose MBPO and M2AC as our illustrated backbone
models, while we think other models such as BMPO [Lai et al.(2020)] and MVE
[Feinberg et al.(2018)] may also have similar results. The pseudo code is put
in table 2 which uses MBPO as an example, and the case with M2AC is similar
and easy to obtain. For the sake of illustration, we note MBPO algorithm with
RSRS indicator as RSPO strategy and M2AC algorithm with RSRS indicator as RSAC
strategy later.

Table 2: RSPO Framework Algorithm of RSPO : MBPO algorithm with RSRS indicator
---
Initialize policy $\pi_{\theta}$, dynamic model $\mathcal{P}_{\phi}$,
environment dataset $\mathcal{D}_{env}$, model dataset $\mathcal{D}_{model}$
Initialize time window $l$, considering window M, buy threshold $rs_{buy}$,
sell threshold $rs_{sell}$ and maximum action $hmax$
for E epochs do
Train dynamic model $\mathcal{P}_{\phi}$ on $\mathcal{D}_{env}$
for N steps do
$\vec{y}\>=\>\alpha\>+\>\beta_{i}\cdot\vec{x}\>+\>\epsilon,\>(i\>=\>1,\>...,\>N)$,
where $\vec{y}$ and $\vec{x}$ corresponds to the high price
and low price sequence of length ${l}$ respectively
for M steps do
$\vec{y}\>=\>\alpha+\beta_{m}\cdot\vec{x}+\epsilon,\>(m\>=\>i-M+1,\>...,\>i)$
${\beta_{std}}_{i}\>=\>\frac{\beta_{i}\>-\>\mathbb{E}(\beta_{j})}{\sigma(\beta_{j})},\>j\>=\>i\>-\>M\>+\>1,\>...,\>i$
${\beta_{mod}}_{i}\>=\>{\beta_{std}}_{i}\>\times\>R^{2}$, where $R^{2}$ is the
coefficient of determination corresponds to $\beta_{i}$
${\beta_{rightdev}}_{i}\>=\>\beta_{mod}\>\times\>\beta_{i}$
Get action $a_{t}$ using policy $\pi_{\theta}$
if ${\beta_{rightdev}}_{i}\>>\>rs_{buy}$, then $a_{t}\>=\>hmax$
if ${\beta_{rightdev}}_{i}\><\>rs_{sell}$, then $a_{t}\>=\>-\>hmax$
Take action $a_{t}$ in environment ; add to $\mathcal{D}_{env}$
for L model rollouts do
Sample $s_{t}$ randomly from $\mathcal{D}_{env}$
Perform k-step model rollout starting from $s_{t}$ using policy $\pi_{\theta}$
; add to $\mathcal{D}_{model}$
for W gradient updates do
Update policy parameters on $\mathcal{D}_{model}$:
$\theta\>\leftarrow\>\theta\>-\>\lambda_{\pi}\>\triangledown_{\theta}\>(\mathcal{J}_{\pi}(\theta,\mathcal{D}_{model}))$

##  4 Experiment

In this session, we present some experiment results and corresponding
explanations, from which we can see how and why RSRS helps improving the
performance of model-based classical reinforcement learning models in stock
portfolio.

###  4.1 Dataset

All of our sample data are from Yahoo finance database
222https://www.yahoo.com. We conduct our experiments on 30 stocks selected by
their turnover rate within 180 days before Jan. 01, 2009 in The Standard and
Poor’s 500 (S&P 500) market. The selected stocks have relatively low turnover
rate so as to avoid big noise or fluctuations in unstable financial markets
with small market capitalization. Here, as a liquidity indicator, the
"turnover rate" refers to the frequency of stock changing hands in the market
within a certain period of time. In this way, we can better explore the
applicability and generalizability of the proposed agent in this paper.

For each experiment, the time span of the dataset is historical daily price
data from Jan. 1, 2009 to Jul. 3, 2021. Data from Jan. 1, 2009 to Jul. 3, 2016
(1888 days) is used as training set, and data from Jul. 4, 2016 to Jul. 3,
2018 (504 days) is used as validation set. The remaining data from Jul. 4,
2018 to Jul. 3, 2021 (755 days) is used as testing set. We train our agent on
training data, then select model hyper-parameters by evaluating metrics such
as annualized return, Sharpe ratio and maximum drawdown on validation data,
and finally apply the selected model onto the testing data, which is shown in
the following figures and tables. Notice that all the testing results are
averaged from 10 random experiments.

Here we setup the following hyperparameters for our experiments: initial
balance $B_{0}=1e6$ dollars, time window $l=10$, time period of mean and
standard deviation for RSRS $M=300$, two thresholds $rs_{buy}=1.0$,
$rs_{sell}=-0.4$, and maximum shares per transaction $hmax=100$.

###  4.2 Portfolio Performance

Five algorithms, PETS, MBPO, M2AC, RSPO, RSAC are used in our experiments. All
these strategies on testing data are compared with the Market Index baselines:
GSPC.

The evaluation metrics, we use the following measurements to evaluate proposed
model performance.

  * •

Annualized Return: the geometric average amount of money earned by an
investment strategy each year over a given time period.

  * •

Cumulative Return: reflects the overall effect of trading strategy in a
certain time range.

  * •

Annualized Volatility: annualized standard deviation of portfolio return shows
the robustness of the agent.

  * •

Sharpe Ratio [Sharpe(1998)]: the return earned per unit volatility which is a
widely-used measure of the performance of an investment.

  * •

Calmar Ratio: the annualized return earned per unit maximum drawdown in the
period.

  * •

Stability: Determines R-squared of a linear fit to the cumulative log returns.

  * •

Maximum Drawdown: the maximum loss from a peak-to-trough decline of an
investment, before a new peak is attained.

Overall performance with different measurements: In Table 3, we use the above
seven indicators to evaluate the performance of different RL algorithms. It
can be seen from the table that between July 2018 and July 2021, all the
model-based reinforcement learning algorithms are better than the baseline
method (GSPC), especially RSAC and RSPO, which are ranked as top 2 methods in
terms of all the measurements. To be more specific, first, from profit point
of view, RSAC has around 30% annualized return and 120% cumulative return,
which are nearly twice as high as those of GSPC. Second, from the risk
management point of view, RSAC and RSPO have relatively low annualized
volatility, which is aligned with high stability ratio and low maximum
drawdown, meaning that RSRS can help the agent to make more stable investment
decision with less riskiness. And finally, considering both risk management
and profit gain, Sharpe ratio and Calmar ratio show that RSAC and RSPO are
significantly better than others, indicating combining RSRS with some
classical reinforcement learning algorithms is a worthy direction to try.

Table 3: Strategy Performance in S&P 500. The specific performance of the six strategies on seven backtest indicators during the three-years backtest period from July 2018 to July 2021. | RSAC | RSPO | M2AC | MBPO | PETS | GSPC
---|---|---|---|---|---|---
| Annualized
---
Return
30.02% | 28.23% | 23.43% | 22.78% | 20.21% | 16.75%
| Cumulative
---
Return
119.56% | 110.62% | 87.87% | 84.96% | 73.60% | 59.04%
| Annualized
---
Volatility
22.78% | 23.05% | 23.58% | 24.91% | 24.15% | 23.05%
| Sharpe
---
Ratio
1.27 | 1.2 | 1.01 | 0.95 | 0.88 | 0.79
| Calmar
---
Ratio
105.14% | 106.86% | 72.57% | 64.58% | 61.11% | 49.37%
Stability | 94.63% | 95.54% | 89.08% | 88.96% | 90.03% | 72.82%
| Maximum
---
Drawdown
-28.55% | -26.41% | -32.28% | -35.28% | -33.07% | -33.92%

Comparison of Annualized Return: Table 4 shows the year-end returns of the six
strategies for each year. Analysis of the tabular data shows that between July
2019 and July 2020, due to the impact of COVID-19, the stock market sentiment
was very sluggish. The benchmark GSPC has only a yield of 4.66%, and M2AC,
MBPO, PETS, as our control groups, all have the highest yield less than 10%.
However, our proposed method with RSRS equipped into M2AC and MBPO, still has
maintained the annualized return close to be around 20% and 28%, showing that
the RSRS indicator can effectively resist plunge when the finance market gets
collapse during some unpredictable crisis.

However, from Jul. 2020 to Jul. 2021, the year following the rapid recovery of
the stock market, the yields of M2AC and MBPO increased sharply, outperforming
RSAC and RSPO. We speculate the reason for this might come from the low asset
amount of M2AC and MBPO in Jul. 2020.

Based on the above analysis, we believe that M2AC and MBPO algorithms are
comparatively sensitive to market sentiment, which shows that the RSRS
indicator could enhance the algorithms robustness against the violent
fluctuation of the finance market in a certain extent.

Table 4: Annualized return of the six strategies in each year from July 2018 to July 2021. | 2018.7.1-2019.7.1 | 2019.7.1-2020.7.1 | 2020.7.1-2021.7.1
---|---|---|---
RSAC | 29.90% | 19.53% | 38.33%
RSPO | 31.72% | 27.88% | 22.66%
M2AC | 23.22% | 5.94% | 42.25%
MBPO | 22.46% | 8.01% | 36.94%
PETS | 15.42% | 9.31% | 34.13%
GSPC | 9.51% | 4.66% | 36.87%

Comparison of Daily Return: For all 30 selected stocks, we show the portfolio
of the five algorithms and the corresponding baseline in Figure 1. It
illustrates the account value of six strategies in the experiment. Compared
the performance of RSAC, RSPO, M2AC and MBPO, it can be seen that after adding
the RSRS indicator, the income of RSAC (orange) and RSPO (dark blue) has
increased significantly and is consistently higher than the cumulative income
of PETS (light blue) and baseline (grey).

Figure 1: Returns. Time-yield curve of RSAC, RSPO and M2AC, MBPO, PETS,
Baseline.

###  4.3 Model Performance

Convergence of Critic Network: To guarantee reliability of our algorithms, we
also analyze the convergence of critic network. Figure 2 illustrates the
relationship between critic loss over training episodes, which appears that
all the algorithms converge in less than 50 episodes. In addition, the decline
pattern of the critic loss over first 30 training episodes also shows that
RSAC (orange) and RSPO (dark blue) converge faster than pure MBPO (purple) and
M2AC (pink) at early stage, indicating that RSRS has helped our proposed
method to be relatively efficient in terms of computational cost. The smaller
box inside Figure 2 shows the convergence in detail. In addition, Figure 3
depicts the logarithm of critic loss value over training episodes and we can
see that critic loss of RSAC (orange) and RSPO (dark blue) have consistently
lower error compared with MBPO and RSPO without RSRS.

Figure 2: Critic loss over training steps. Figure 3: Critic loss over training
steps.

Analysis of Transition Dynamic Model: The RL algorithms involved in this
article are all model-based, so we also analyze the performance of our dynamic
models. When training the transitional dynamics with quadruple data
$(s,a,s^{\prime},r)$ obtained from real environment, for any given $(s,a)$, we
can predict $s^{\prime}$ in virtual environment and therefore we also check
the convergence of variable $s^{\prime}$. Since our state is high dimensional,
here we take some components of state ’s’ as an illustration. For example,
Figure 4 depicts the absolute error between predicted ’next state’ and the
ground truth of ’next state’ projected onto the coordinate ’cumulative shares’
(denoted as $W_{t}$ in Table 1). It is shown that with the training iterations
increasing by, the predict shares from transition dynamics getting closer to
the true shares given by real environment. Moreover, Figure 5 shows the error
projected onto the component ’balance’ which is the cash in the account over
time (denoted as $B_{t}$ in Table 1).

Figure 4: Absolute error of cumulative shares Figure 5: Absolute error of
balance

##  5 Conclusion

In this paper, we propose a novel strategy which brings resistance support and
relative strength (RSRS) indicator into some powerful model-based RL methods
for stock trading optimization.

The RSAC and RSPO strategies take advantage of both the timing selection
nature from technical analysis in finance market and the intelligent agent-
environment interaction properties of model-based RL algorithms, leading to
improvements in terms of stability, risk management and profit gain,
especially the resistance to big marketing crisis. Experiment results based on
30 stocks selected form S$\&$P 500 market show that the proposed RSAC and RSPO
strategies can effectively strengthen the performance of the model-based
algorithms. For example, during COVID-19 pandemic period, our strategies
highly improve the performance of maximum drawdown and annualized return
measurements. To guarantee our observed results to be reliable, we also check
the convergence of agent’s critic loss and prediction error of the dynamic
model. Therefore, it inspires us that combining techniques from finance and
statistics with RL algorithms is an interesting research direction to study.

However, there are still some challenges that are open to us. For example, as
we known that there is close relationship between RL and stochastic optimal
control. After setting up this question as a stochastic optimal control
problem, how to prove the solution’s uniqueness and properties in appropriate
functional space, is still unknown to us. Moreover, the analysis of
generalization error bound of model-based RL is also important for us to
design more adaptive and stable RL algorithms.

## References

  * [1]
  * [Appel(2003)] Gerald Appel. 2003\.  Become your own technical analyst: How to identify significant market turning points using the moving average convergence-divergence indicator or macd.  _The Journal of Wealth Management_ 6, 1 (2003), 27–36.
  * [Azhikodan et al.(2019)] Akhil Raj Azhikodan, Anvitha GK Bhat, and Mamatha V Jadhav. 2019\.  Stock trading bot using deep reinforcement learning.  In _Innovations in Computer Science and Engineering_. Springer, 41–49.
  * [Bao et al.(2017)] W. Bao, J. Yue, Y. Rao, and P. Boris. 2017\.  A deep learning framework for financial time series using stacked autoencoders and long-short term memory.  _PLoS ONE_ 12, 7 (2017), e0180944.
  * [Bollinger(1992)] John Bollinger. 1992\.  Using bollinger bands.  _Stocks & Commodities_ 10, 2 (1992), 47–51.
  * [Cai and Wei(2020)] T Tony Cai and Hongji Wei. 2020.  Distributed gaussian mean estimation under communication constraints: Optimal rates and communication-efficient algorithms.  _arXiv preprint arXiv:2001.08877_ (2020).
  * [Chua et al.(2018)] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. 2018.  Deep reinforcement learning in a handful of trials using probabilistic dynamics models.  _arXiv preprint arXiv:1805.12114_ (2018).
  * [Emami(2016)] Patrick Emami. 2016\.  Deep deterministic policy gradients in tensorflow.  _My summaries of Machine Learning papers and investigations into various topics concerning artificial intelligence_ (2016).
  * [Feinberg et al.(2018)] Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael I Jordan, Joseph E Gonzalez, and Sergey Levine. 2018\.  Model-based value expansion for efficient model-free reinforcement learning. In _Proceedings of the 35th International Conference on Machine Learning (ICML 2018)_.
  * [Haarnoja et al.(2018)] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcem learning with a stochastic actor. In _International conference on machine learning_. PMLR, 1861–1870.
  * [Janner et al.(2019)] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. 2019\.  When to trust your model: Model-based policy optimization.  _arXiv preprint arXiv:1906.08253_ (2019).
  * [Kang et al.(2018)] Qinma Kang, Huizhuo Zhou, and Yunfan Kang. 2018.  An Asynchronous Advantage Actor-Critic Reinforcement Learning Method for Stock Selection and Portfolio Management. In _the 2nd International Conference_.
  * [Lai et al.(2020)] Hang Lai, Jian Shen, Weinan Zhang, and Yong Yu. 2020\.  Bidirectional Model-based Policy Optimization. In _International Conference on Machine Learning_. PMLR, 5618–5627.
  * [Lee et al.(2019)] J. Lee, R. Kim, Y. Koh, and J. Kang. 2019\.  Global Stock Market Prediction Based on Stock Chart Images Using Deep Q-Network.  _IEEE Access_ PP, 99 (2019), 1–1.
  * [Li et al.(2019a)] X Li, Y. Li, Y. Zhan, and X. Y. Liu. 2019a.  Optimistic Bull or Pessimistic Bear: Adaptive Deep Reinforcement Learning for Stock Portfolio Allocation.  _Papers_ (2019).
  * [Li et al.(2019b)] Y. Li, M. Nee, and V. Chang. 2019b.  An Empirical Research on the Investment Strategy of Stock Market based on Deep Reinforcement Learning model. In _4th International Conference on Complexity, Future Information Systems and Risk (COMPLEXIS 2019)_.
  * [Lillicrap et al.(2015)] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. 2015.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971_ (2015).
  * [Lloyd Sr(2013)] Tom K Lloyd Sr. 2013\.  _Successful Stock Signals for Traders and Portfolio Managers: Integrating Technical Analysis with Fundamentals to Improve Performance_.  John Wiley & Sons.
  * [Mnih et al.(2016)] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. 2016.  Asynchronous methods for deep reinforcement learning. In _International conference on machine learning_. PMLR, 1928–1937.
  * [Mnih et al.(2013)] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ (2013).
  * [Mnih et al.(2015)] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _nature_ 518, 7540 (2015), 529–533.
  * [Pan et al.(2020)] Feiyang Pan, Jia He, Dandan Tu, and Qing He. 2020\.  Trust the model when it is confident: Masked model-based actor-critic.  _Advances in neural information processing systems_ 33 (2020), 10537–10546.
  * [Pastore et al.(2016)] A. Pastore, U. Esposito, and E. Vasilaki. 2016.  Modelling stock-market investors as Reinforcement Learning agents. In _IEEE International Conference on Evolving $\&$ Adaptive Intelligent Systems_.
  * [Peng et al.(2018)] Baolin Peng, Xiujun Li, Jianfeng Gao, Jingjing Liu, Kam-Fai Wong, and Shang-Yu Su. 2018\.  Deep dyna-q: Integrating planning for task-completion dialogue policy learning.  _arXiv preprint arXiv:1801.06176_ (2018).
  * [Schulman et al.(2017)] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * [Sharpe(1998)] William F Sharpe. 1998\.  The sharpe ratio.  _Streetwise–the Best of the Journal of Portfolio Management_ (1998), 169–185.
  * [Sornmayura(2019)] Sutta Sornmayura. 2019\.  Robust forex trading with deep q network (dqn).  _ABAC Journal_ 39, 1 (2019).
  * [Ţăran-Moroşan(2011)] Adrian Ţăran-Moroşan. 2011\.  The relative strength index revisited.  _African Journal of Business Management_ 5, 14 (2011), 5855–5862.
  * [Van Hasselt et al.(2016)] Hado Van Hasselt, Arthur Guez, and David Silver. 2016\.  Deep reinforcement learning with double q-learning. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 30.
  * [Wang et al.(2016)] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. 2016\.  Dueling network architectures for deep reinforcement learning. In _International conference on machine learning_. PMLR, 1995–2003.
  * [Wu and Tian(2017)] Yuxin Wu and Yuandong Tian. 2017.  Training Agent for First-Person Shooter Game with Actor-Critic Curriculum Learning. In _ICLR_.
  * [Xiong et al.(2018)] Zhuoran Xiong, Xiao-Yang Liu, Shan Zhong, Hongyang Yang, and Anwar Walid. 2018.  Practical deep reinforcement learning approach for stock trading.  _arXiv preprint arXiv:1811.07522_ (2018).
  * [Yu et al.(2019)] Pengqian Yu, Joon Sern Lee, Ilya Kulyatin, Zekun Shi, and Sakyasingha Dasgupta. 2019.  Model-based deep reinforcement learning for dynamic portfolio optimization.  _arXiv preprint arXiv:1901.08740_ (2019).


HISTORICAL CURIOSITY:
HOW ASYMPTOTIC FREEDOM OF THE YANG-MILLS THEORY COULD HAVE BEEN DISCOVERED
THREE TIMES BEFORE GROSS, WILCZEK, AND POLITZER, BUT WAS NOT 111This article
was published in 2001, see Festschrift [1]. I added a couple of sentences in
2022.

M. Shifman

William I. Fine Theoretical Physics Institute, University of Minnesota,
Minneapolis, MN 55455, USA

Abstract

Asymptotic freedom as the basic property of QCD was discovered by Gross,
Wilczek, and Politzer in 1973. Personal recollections of David Gross which are
being published in this Volume vividly describe the historical background and
the chain of events which led to this fundamental breakthrough. Unfortunately,
I failed to obtain Politzer’s side of the story. Some details can be found in
an interview which Prof. Politzer gave to R. Crease and C. Mann on February
21, 1985 [2]. Below I acquaint the reader with the pre-1973 appearances of
asymptotic freedom which, unfortunately, went unnoticed.

The antiscreening of the gauge coupling constant in the non-Abelian gauge
theories had been detected three times before the actual discovery of
asymptotic freedom was made by David Gross, Frank Wilczek and David Politzer.
In two instances, the results of calculations exhibiting the antiscreening
were reported in the journal publications [3, 4] in the 1960’s (V. Vanyashin
and M. Terentev, 1965; I. Khriplovich, 1969). The third time this calculation
was apparently carried out in 1971 by G. ’t Hooft; the result was not
published, however. Since this story is rather curious (and, perhaps, even
instructive, in retrospect), I think, the reader deserves to be acquainted
with it. I hasten to add, though, that the above authors did not recognize the
importance of the antiscreening for the theory of strong interactions; nor did
they grasp possible implications. Therefore, by no means I want to imply that
the discovery of asymptotic freedom was made before 1973. There can be no
questions – the true discoverers of asymptotic freedom are Gross, Wilczek and
Politzer.

The paper of Vanyashin and Terentev, titled The Vacuum Polarization of a
Charged Vector Field, was published [3] in 1965. The authors considered
propagation of the charged massive vector field in the background
electromagnetic field. They obtained the Heisenberg–Euler effective Lagrangian
at one loop. The calculation was carried out in the unitary gauge. In modern
terms one can say that the authors found the one-loop effective Lagrangian in
the SU(2) massive Yang-Mills theory.

Although the mass of the vector field was treated as hard, rather than in the
framework of the Higgs mechanism, in the one-loop approximation the theory is
self-consistent. The power divergences cancel. The only remaining divergence
is logarithmic; it is exactly the same as in the SU(2) Yang-Mills theory with
the Higgs mechanism. The coefficient in front of

$\frac{e^{2}}{32\pi^{2}}\,E^{2}\,\ln\frac{M_{\rm UV}^{2}}{eE}$ (1)

in the large $E$ limit is nothing but the first coefficient in the $\beta$
function. (Here $M_{\rm UV}$ is the ultraviolet cutoff used in calculating the
Heisenberg–Euler Lagrangian). Vanyashin and Terentev found this coefficient to
be equal to

$-7=-\left(\frac{22}{3}-\frac{1}{3}\right)\,,$ (2)

(see Eq. (30) in their work). The first term, $-{22/}{3}$, is what one would
get for the massless Yang-Mills field, while $1/3$ is the contribution of the
longitudinal $W$ bosons. In the theory with the Higgs mechanism of the mass
generation this $1/3$ emerges as the contribution of the Higgs field.

Vanyashin and Terentev perfectly realized that the coefficient they found
represented the charge renormalization. Let me quote a passage from the
concluding part of the paper: “Unlike conventional electrodynamics, where the
renormalized charge is smaller than the bare one, our case gives
$e^{2}>e_{0}^{2}$ […] As is well-known, in electrodynamics the restriction
$0\leq Z\leq 1$ on the $Z$ factor (the charge renormalization) follows from
the most general principles […]. In our case it is impossible, generally
speaking, to get the restriction $0\leq Z\leq 1$ starting from these
principles […]. Nevertheless, the inequality $Z>1$ seems extremely
undesirable.”222In 2008 Arkady Vainshtein wrote (private communication to G.
t’Hooft): “Valentin Zakharov who was at Terentev’s seminar recollects […] that
the main problem Isaac Pomeranchuk had with Terentev’s work was positivity of
the spectral function in the Källen-Lehman representation of the polarization
operator. Pomeranchuk, of course, knew very well that $Z>1$ implies asymptotic
freedom in the UV but he viewed the theory of massive charged vector field as
nonphysical, particularly, because of the positivity problem. Terentev was not
able to give him a satisfactory answer. […] A simple relation between the
polarization operator and the coupling constant will be subsequently lost. A
derivation based on scattering amplitudes was carried out in [5] and later by
L. Lipatov when he considered reggezation.”

For a short time, around 1970, Terentev was my thesis adviser, and I had the
opportunity of discussing various physics issues with him. Certainly, he was
expert in the renormalization group equations, and was well aware of the fact
that $Z\leq 1$ in QED implied the growth of the effective charge in QED at
short distances, and vice versa. In the ITEP theory department this was a
“$2\times 2=4$” statement. Landau, Abrikosov and Khalatnikov at first made a
mistake in their 1954 work [6] on the effective charge in QED, and got the
$\beta$ function negative. For some time Landau was entertaining himself
working out the implications of the negative $\beta$ function – the fall off
of the effective charge at short distances, and so on. He recognized the
importance of the phenomenon right away and widely discussed it with his ITEP
colleagues and elsewhere. Only somewhat later a sign error was found by Boris
Ioffe and Alexei Galanin (See Geshkenbein’s note). In the published version
[6] the zero charge result (i.e. infrared freedom) was reported.

Why Terentev never returned to his 1965 work after the results of the SLAC-MIT
experiments on deep inelastic scattering became known, remains a mystery. The
experiments on deep inelastic scattering and the subsequent works of Bjorken
and Feynman were subject to intense scrutiny in ITEP theory division, prior to
the discovery of asymptotic freedom in 1973.

Khriplovich’s work [4] was specifically devoted to the one-loop corrections in
(massless) non-Abelian Yang-Mills theories. A non-covariant (Coulomb) gauge
was used, which is ghost-free. In this gauge the charge renormalization is
given by that of the $D_{00}$ component of the gauge field Green function,
much in the same way as in QED. The logarithmically divergent part of $D_{00}$
is presented in Eq. (61) of Ref. [4], from which it is straightforward to read
off the corresponding $Z$ factor (in the first line see the first two terms in
the square brackets). After taking into account an unconventional
normalization of the coupling constant there, one immediately recovers the
first coefficient in the $\beta$ function, the famous $-22/3$ (the theory
considered in Ref. [4] was the SU(2) gauge theory).

Moreover, one additionally learns from the discussion around Eq. (61) in [4]
that $-22/3$ is actually $-8+\frac{2}{3}$. The logarithms are of two different
types. One is non-dispersive, it corresponds to the instantaneous Coulomb
interaction and is responsible for antiscreening (“minus eight”).333For more
details see Sec. 1.3 of Ref. [7]. The second logarithm is normal dispersive;
in full accordance with the general principles it gives a positive
contribution (“two thirds”) characteristic of the normal screening. The
formula $-\frac{22}{3}=-8+\frac{2}{3}$ and the corresponding interpretation of
antiscreening vs. screening was rediscovered in the West only eight years
later [8].444Khriplovich was far away, at the Novosibirsk Institute for
Nuclear Physics, and, according to Vainshtein, was less knowledgeable about
implications of his $Z>1$ result for the $\beta$ function. Alas… he missed the
connection.

Now I come to ’t Hooft’s story. Here is an excerpt from 1998 ’t Hooft’s talk
[9] where ’t Hooft recollects on his work in 1971.

“To many physicists, the Bjorken scaling appeared to be a signal that no
quantum field theory would be fit to account for the observed strong
interactions. Why then did nobody compute the $\beta$ function for the Yang-
Mills theories? I had computed this function, and knew about its sign. In
1971, as soon as we knew how to renormalize the theory, I started a
preliminary calculation, showing the negative sign. Not realizing that this
was something totally novel, I could not imagine that this was all that was
needed to resolve the outstanding problems of strong interaction theory. I
actually thought that this scaling behavior was also known, but not considered
relevant for partons. Since I could not understand their problems, I turned
away from the strong interactions that had to be infinitely complicated.

By 1972, when Veltman and I had worked out many examples of gauge models, we
knew how to compute the renormalization counter terms, and I had found a way
to relate these to the equations governing the running coupling constants.
Veltman was not interested in these findings. According to him, they were only
relevant for Euclidean space, which has nothing to do with experimental
observations, all done in Minkowski space. On several occasions I mentioned to
him the possibility of a pure gauge theory for the strong interactions, but
his mind was made up. As long as they produced no answer to the quark
confinement problem, such ideas are useless, and no referee would accept a
publication suggesting such an idea.

In June 1972, a small meeting was organized in Marseilles by C.P. Korthals
Altes. Symanzik was there. Upon arrival, right at the airport, Symanzik and I
began to discuss our work. Symanzik explained that he worked on
$\lambda\phi^{4}$ theory with negative $\lambda$, because of its interesting
scaling properties. He had been attracted by the idea that theories with a
negative $\beta$ function would reproduce parton-like behavior, but, the only
models with such properties that he could think of were theories with a
negative squared coupling constant, $g^{2}<0$ or $\lambda<0$.

Naturally, I asked him how he would solve the problem that the energy appears
to be unbounded from below. He explained that perhaps this problem could be
cured by non-perturbative effects, which at that time were badly understood.
But I informed him of my finding that the coefficient determining the running
of the coupling strength, that he called $\beta(g^{2})$, for gauge theories is
negative. Symanzik was surprised and skeptical. “If this is true, it will be
very important, and you should publish this result quickly, and if you won’t,
somebody else will,” he said.

In the meeting, Symanzik ended his talk on negative $\lambda\phi^{4}$ theories
by the remark that further study on the $\beta$ functions for non-Abelian
gauge theories were called for. When it was time for questions I came forward
to write down on the blackboard the expression I had derived (with a negative
coefficient) […].

Why did I not follow Symanzik’s sensible advice to publish this result?
Weinberg had written a long paper in which he reported about the complicated
infrared structure of gauge theories, which made me believe he knew everything
about what was to be called infrared slavery, a direct consequence of
asymptotic freedom. Surely, I thought, he understands the scaling behavior of
gauge theories. David Gross had explained to me that the Bjorken scaling
cannot be accommodated for in any field theory, Abelian or not. Veltman was
telling me that scaling all components of the momentum vector would lead you
far away from the mass shell, whereas all experiments were looking at on-mass-
shell hadronic final states, so the deep-Euclidean region was unphysical and
uninteresting.”

I am grateful to L. Frankfurt and A. Vainshtein for reminding me certain
details in the Vanyashin-Terentev work.

## References

  * [1] At the Frontier of Particle Physics; Handbook of QCD, Ed. M. Shifman, (World Scientific, Singapore, 2001), Vol. 1, page 126.
  * [2] Excerpts from this interview are published in R. Crease and C. Mann, The Second Creation (Rutgers University Press, 1986, The Second Edition 1996).
  * [3] V.S. Vanyashin and M.V. Terentev, The Vacuum Polarization of a Charged Vector Field, ZhETF 48 (1965) 565 [Sov. Phys. – JETP 21 (1965) 375].
  * [4] I.B. Khriplovich, Green’s functions in theories with non-abelian gauge group, Yad. Fiz. 10, 409 (1969) [Sov. J. Nucl. Phys. 10 (1970) 235].
  * [5] E. B. Bogomolnyi, V. A. Novikov and M. A. Shifman, The small-distance behavior of the physical charge in non-Abelian gauge theories, Yad. Fiz. 20, 210 (1974) [Sov. J. Nucl. Phys. 20, 110 (1975); online on
https://www.academia.edu/74247005/].

  * [6] L. D. Landau, A. A. Abrikosov and I. M. Khalatnikov, An asymptotic expression for the photon Green function in Quantum Electrodynamics, Doklady AN SSSR, 95, 1177 (1954) [English version in: Collected Papers of L. D. Landau , (Oxford: Pergamon, 1965), p. 616-620].
  * [7] V. A. Novikov, L. B. Okun, M. A. Shifman, A. I. Vainshtein, M. B. Voloshin and V. I. Zakharov, Charmonium and Gluons Phys. Reports 41, 1 (1978),
  * [8] T. Appelquist, M. Dine and I. J. Muzinich, The Static Potential In Quantum Chromodynamics, Phys. Lett. B69 (1977) 231.
  * [9] G. ’ t Hooft, The Creation of Quantum Chromodynamica, in The Creation of Quantum Chromodynamics and the Effective Energy, Ed. L. N. Lipatov, (Bologna, 1998), p. 27.