# Generating Synergistic Formulaic Alpha Collections via Reinforcement
Learning

Shuo Yu  ,  Hongyan Xue  ,  Xiang Ao  Institute of Computing Technology,
Chinese Academy of SciencesUniversity of Chinese Academy of
SciencesBeijingChina<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>,  Feiyang Pan  ,  Jia He  ,  Dandan Tu  Huawei EI
Innovation LabChina<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and  Qing He  Institute of Computing Technology, Chinese Academy of
SciencesUniversity of Chinese Academy of SciencesBeijingChina<EMAIL_ADDRESS>
(2023)

###### Abstract.

In the field of quantitative trading, it is common practice to transform raw
historical stock data into indicative signals for the market trend. Such
signals are called alpha factors. Alphas in formula forms are more
interpretable and thus favored by practitioners concerned with risk. In
practice, a set of formulaic alphas is often used together for better modeling
precision, so we need to find synergistic formulaic alpha sets that work well
together. However, most traditional alpha generators mine alphas one by one
separately, overlooking the fact that the alphas would be combined later. In
this paper, we propose a new alpha-mining framework that prioritizes mining a
synergistic set of alphas, i.e., it directly uses the performance of the
downstream combination model to optimize the alpha generator. Our framework
also leverages the strong exploratory capabilities of reinforcement learning
(RL) to better explore the vast search space of formulaic alphas. The
contribution to the combination models’ performance is assigned to be the
return used in the RL process, driving the alpha generator to find better
alphas that improve upon the current set. Experimental evaluations on real-
world stock market data demonstrate both the effectiveness and the efficiency
of our framework for stock trend forecasting. The investment simulation
results show that our framework is able to achieve higher returns compared to
previous approaches.

Computational Finance, Stock Trend Forecasting, Reinforcement Learning

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
6–10, 2023; Long Beach, CA, USA††booktitle: Proceedings of the 29th ACM SIGKDD
Conference on Knowledge Discovery and Data Mining (KDD ’23), August 6–10,
2023, Long Beach, CA, USA††doi: 10.1145/3580305.3599831††isbn:
979-8-4007-0103-0/23/08††ccs: Computing methodologies Reinforcement
learning††ccs: Computing methodologies Search methodologies††ccs: Applied
computing Economics

##  1\. Introduction

Currently, it is almost a standard paradigm to transform raw historical stock
data into indicative signals for the market trend in the field of quantitative
trading (Qian, 2007). These signal patterns are called _alpha factors_ , or
alphas in short (Tulchinsky, 2015). Discovering alphas with high returns has
been a trendy topic among investors and researchers due to the close
relatedness between alphas and investment revenues.

The prevailing methods of discovering alphas can be in general divided into
two groups, namely machine learning-based and formulaic alphas. Most recent
research has focused on the former ones. These more sophisticated alphas are
often obtained via deep learning models, e.g., using sequential models like
LSTM (Hochreiter and Schmidhuber, 1997), or more complex ones integrating non-
standard data like HIST (Xu et al., 2021a) and REST (Xu et al., 2021b), etc.
On the other end of the spectrum, we have the alphas that can be represented
in simple formula forms. Such formulaic alphas are traditionally constructed
by human experts using their domain knowledge and experience, often expressing
clear economic principles. To name some, (Kakushadze, 2016) demonstrates 101
alpha factors tested on the US stock market. Recently, research has also been
conducted on frameworks that generate such formulaic alphas automatically (Cui
et al., 2021; Zhang et al., 2020; Lin et al., 2019b, a). These approaches are
able to find loads of new alphas rapidly without human supervision, while
still maintaining relatively high interpretability compared to the more
sophisticated machine learning-alphas.

Despite the existing approaches achieving remarkable success, however, they
still have disadvantages in different aspects. Machine learning-based alpha
factors are inherently complex and sometimes require more complex data other
than the price/volume features. In addition, although they are often more
expressive, they often suffer from relatively low explainability and
interpretability. As a result, when the performance of these “black box”
models unexpectedly deteriorates, it is hard for human experts to tune the
models accordingly. These algorithms are thus not favored under some
circumstances due to concerns about risks. On the other hand, while formulaic
alphas are more interpretable, previous research on this matter often focused
on finding a single alpha factor that predicts well on its own. Nonetheless,
it is often impossible to describe a complex and chaotic system such as the
stock market with simple rules that human researchers can comprehend. As a
compromise, a set of these alphas are oftentimes used together in practice,
instead of using them individually. However, when multiple of these
independently mined formula alphas are combined, the final prediction
performance may not improve much because not much consideration is put into
the synergistic effect between factors (see Section 4.2.2 for detail). In
addition, these alphas are often simple in their forms, and their underlying
mechanisms are often quite understandable. Once they are released to the
public and become well-known among practitioners, their performance may
deteriorate rapidly (Kakushadze, 2016).

Therefore, the question we are facing is: _Are we able to find a way to
automatically discover interpretable alpha factors, which work well with
downstream predictive models, without suffering possible performance
deterioration due to the alpha factors being widely known to the general
public?_

To solve the above challenge, we formulate a new research problem in this
paper, which is to find _synergistic formulaic alpha factor sets_. Using raw
stock price/volume data as the input, we aim to search for a set of formulaic
alpha factors instead of individual ones. Recall that finding a single well-
performing alpha on given data is already a hard problem to resolve since the
search space of valid formulas is vast and hard to navigate. The search space
for alpha mining is often even larger than that of a typical symbolic
regression problem (Petersen et al., 2021).

The most intuitive approach to this problem would be using genetic programming
(GP), performing mutations on expression trees to generate new alphas. In
fact, most previous work on this matter is based on genetic programming (GP)
(Lin et al., 2019b, a; Cui et al., 2021; Zhang et al., 2020), which is of
course not a serendipitous choice since GP methods generally excel at such
problems with large search spaces. However, GP algorithms often scale poorly
due to the complexity of maintaining and mutating a huge population (Petersen
et al., 2021). In addition, the main challenge remains that mining a set of
synergistic alphas all at once is an even harder problem with a much larger
search space, the scale of which makes most existing frameworks infeasible to
solve.

Hence, previous works mostly tried to find ways to simplify the problem of
alpha set mining, by mining alphas one by one and filtering out a subset of
them with respect to some similarity metric. The mutual information
coefficient (IC) between the pairs of alpha in the set is often employed as
the similarity “metric” (Cui et al., 2021; Zhang et al., 2020; Lin et al.,
2019b). However, as we will demonstrate below, adding a new alpha that is of
high IC to the ones in an existing pool of alpha may still bring a non-
negligible boost of performance to the combined result, and vice versa. This
phenomenon still exists even when the combination model is set to be a simple
linear regressor. Therefore, the traditional approach to determining whether a
set of alpha could be synergistic does not line up with the expected outcome.

To tackle the challenge that GP methods could be inefficient at exploring the
vast search space of formulaic alphas, our framework utilizes reinforcement
learning (RL) for achieving better results in exploration. Combined with the
strong expressiveness of deep neural networks, RL with its excellent
exploratory ability plays a predominant role in numerous areas. To list a few
examples, game playing (Schrittwieser et al., 2020), natural language
processing (Liu et al., 2020), symbolic optimization (Petersen et al., 2021),
and portfolio management (Wang et al., 2021). We implement a sequence
generator with constraints to ensure valid formulaic alpha generation and
employ a policy gradient-based algorithm to train the generator in the absence
of a direct gradient. Since traditional mutual-IC filtering methods do not
align well with the target of optimizing the combination model’s performance,
we propose to use directly the performance as the optimization objective of
our alpha generator. Under this new optimization scheme, our generator is able
to produce a synergistic set of alpha which fits the mine-and-combine
procedure in a more suitable way. To evaluate our alpha-mining framework, we
conduct extensive experiments over real-world stock data. Our experiment
results demonstrate that the formulaic alpha sets generated by our framework
perform better than those generated with previous approaches, shown both on
the prediction metrics and investment simulations.

Our contributions can be summarized as follows.

  * •

We propose a new optimization scheme that produces a set of alpha that suits
downstream tasks better, regardless of what actual form the combination model
takes.

  * •

We introduce a new framework for searching formulaic alpha factors based on
policy gradient algorithms, to utilize the strong exploratory power of
reinforcement learning.

  * •

We present a series of experimental results demonstrating the effectiveness of
our proposed framework. Additional experiments and case studies are also
conducted to demonstrate why mutual IC-based filtering techniques that are
previously commonly used may not work as expected when considering the
combined performance of an alpha set.

##  2\. Problem Formulation

###  2.1. Alpha Factor

We consider a stock market with $n$ stocks in a period of $T$ trading days in
total. On each trading day $t\in\left\\{1,2,\cdots,T\right\\}$, each stock $i$
corresponds to a feature vector $x_{ti}\in\mathbb{R}^{m\tau}$, comprised of
$m$ raw features such as opening/closing price in the recent $\tau$
days111This “unrolling” of historical data introduces redundancy. Namely,
feature vectors of a stock on consecutive trading days have overlapping
sections. This notation is chosen for the convenience of demonstration..
Finally, we define an _alpha factor_ $f$ as a function mapping feature vectors
of all stocks on a trading day $X\in\mathbb{R}^{n\times m\tau}$ into _alpha
values_ $z=f(X)\in\mathbb{R}^{n}$. We will use the word “alpha” for both an
alpha factor and its corresponding values in the following sections.

###  2.2. Alpha Factor Mining

To measure the effectiveness of an alpha, we calculate the information
coefficient (IC) between the true stock trend it aims to predict
$y_{t}\in\mathbb{R}^{n}$ and the factor values $f(X_{t})$. We denote the daily
IC function as $\sigma:\mathbb{R}^{n}\times\mathbb{R}^{n}\to[-1,1]$, which is
defined as the Pearson’s correlation coefficient:

(1)
$\sigma(u_{t},v_{t})=\frac{\sum_{i=1}^{n}(u_{ti}-\bar{u}_{t})(v_{ti}-\bar{v}_{t})}{\sqrt{\sum_{i=1}^{n}(u_{ti}-\bar{u_{t}})^{2}\sum_{i=1}^{n}(v_{ti}-\bar{v_{t}})^{2}}}.$

Such value can be calculated on every trading day between an alpha and the
prediction target. For convenience, we denote the IC values between two sets
of vectors averaged over all trading days as
$\bar{\sigma}(u,v)=\mathbb{E}_{t}\left[\sigma(u_{t},v_{t})\right]$.

We use the average IC between an alpha and the return to measure the
effectiveness of an alpha factor on a stock trend series
$y=\left\\{y_{1},y_{2},\cdots,y_{T}\right\\}$:

(2) $\bar{\sigma}_{y}(f)=\bar{\sigma}(f(X),y).$

As mentioned above, the output of a combination model can be seen as a “mega-
alpha”, mapping raw inputs into alpha values. Therefore, we denote the
combination model as $c(X;\mathcal{F},\theta)$, where
$\mathcal{F}=\left\\{f_{1},f_{2},\cdots,f_{k}\right\\}$ is a set of alphas to
combine, and $\theta$ denotes the parameters of the combination model. We
would like the combination model to be optimal w.r.t. a given alpha set
$\mathcal{F}$ on the training dataset $y$, that is:

(3) $\displaystyle c^{*}(X;\mathcal{F})$
$\displaystyle=c(X;\mathcal{F},\theta^{*}),\quad\textrm{where}$
$\displaystyle\theta^{*}$
$\displaystyle=\operatorname*{argmax}_{\theta}\bar{\sigma}_{y}(c(\cdot;\mathcal{F},\theta)).$

Conclusively, the task of mining a set of alphas can be defined as the
optimization problem
$\operatorname*{argmax}_{\mathcal{F}}c^{*}(\cdot;\mathcal{F})$.

Figure 1. (A) An example of a formulaic alpha. (B) Its equivalent expression
tree. (C) Its reverse Polish notation (RPN). Note that _BEG_ and _SEP_ are
sequence indicators later mentioned in our framework. (D). Step-by-step
computation of this alpha on an example time series.

###  2.3. Formulaic Alpha

Formulaic alphas are expressed as mathematical expressions, consisting of
various operators and the raw input features mentioned before. Some examples
of the operators are the elementary functions (like “$+$” and “$\log$”)
operated on one-day data, called cross-section operators, and operators that
require data from a series of days, called time-series operators (e.g.
“Min(close, 5)” gives the lowest closing price of a stock in the recent 5
days). A list of all the operators used in our framework is given in Appendix
A.

Such formulas can be naturally represented by an expression tree, with each
non-leaf node representing an operator, and children of a node representing
the operands. To generate such an expression, our model represents the
expression tree by its postorder traverse, with the children’s order also
defined by the traversing order. In other words, the model represents a
formula as its reverse Polish notation (RPN). It is easy to see that such
notation is unambiguous since the arities of the operators are all known
constants. See Figure 1 for an example of a formulaic alpha expression
together with its corresponding tree and RPN representations.

##  3\. Methodology

Figure 2. An overview of our alpha-mining framework. (A) An alpha generator
that generates expressions, optimized via a policy gradient algorithm. (B) A
combination model that maintains a weighted combination of principal factors
and, in the meantime, provides evaluative signals to guide the generator.

As illustrated in Figure 2, our alpha-mining framework consists of two main
components: 1) the _Alpha Combination Model_ , which combines multiple
formulaic alphas to achieve optimal performance in prediction, and 2) the _RL-
based Alpha Generator_ , which generates formulaic alphas in the form of a
token sequence. The performance of the Alpha Combination Model is used as the
reward signal to train the RL policy in the Alpha Generator using policy
gradient-based algorithms, such as PPO (Schulman et al., 2017). Repeating this
process, the generator is continuously trained to generate alphas that boost
the combination model, thereby enhancing the overall predictive power.

###  3.1. Alpha Combination Model

Considering the interpretability of the combined “mega-alpha”, the combination
model itself should also be interpretable. In this paper, we use a linear
model to combine the alphas.

The values evaluated from different alphas have drastically different scales,
which might cause problems in the following optimization steps. To counter
this effect, we centralize and normalize the alpha values with their average
and standard deviation. Since Pearson’s correlation coefficient is invariant
up to linear transformation, this transformation does not affect the
performance of the alphas when they are considered separately. Formally, we
introduce a normalization operator $\mathcal{N}$, that transforms a vector
such that its elements have a mean of 0, and the vector has a length of 1:

(4)
$\left[\mathcal{N}(u)\right]_{i}=\frac{u_{i}-\bar{u}}{\sqrt{\sum_{j=1}^{n}\left(u_{j}-\bar{u}\right)^{2}}}.$

We will omit explicitly writing the $\mathcal{N}$ operator for simplicity. For
the rest of this paper, we will assume that all the $f(X)$ evaluations and the
targets $y$ are normalized to have a mean of 0 and a length of 1 before
subsequent computations. In other words, treat $f$ as $\mathcal{N}\circ f$ and
$y$ as $\mathcal{N}(y)$.

Given a set of $k$ alpha factors
$\mathcal{F}=\left\\{f_{1},f_{2},\cdots,f_{k}\right\\}$ and their weights
$w=\left(w_{1},w_{2},\cdots,w_{k}\right)\in\mathbb{R}^{k}$, the combination
model $c$ is defined as follows:

(5) $c(X;\mathcal{F},w)=\sum_{j=1}^{k}w_{j}f_{j}(X)=z.$

We define the loss of the combination model as the mean squared error (MSE)
between model outputs and true stock trend values:

(6) $\mathcal{L}(w)=\frac{1}{nT}\sum_{t=1}^{T}\|z_{t}-y_{t}\|^{2}.$

To simplify the calculation of alpha combination, we have:

###### Theorem 3.1.

Let $\mathcal{F}$ be a set of $k$ alphas and $w$ be their respective weights,
the MSE loss $\mathcal{L}(w)$ can be represented as:

(7)
$\displaystyle\mathcal{L}(w)=\frac{1}{n}\left(1-2\sum_{i=1}^{k}w_{i}\bar{\sigma}_{y}(f_{i})+\sum_{i=1}^{k}\sum_{j=1}^{k}w_{i}w_{j}\bar{\sigma}(f_{i}(X),f_{j}(X))\right).$

The proof of this theorem is provided in Appendix B. Notice that there is no
$z_{t}$ term on the RHS of Equation 7. Once we have obtained
$\bar{\sigma}_{y}(f)$ for each alpha $f$ and their pairwise mutual
correlations $\bar{\sigma}(f_{i}(X),f_{j}(X))$, we can then calculate the loss
$\mathcal{L}(w)$ solely using these terms, saving time on calculating the
relatively large $z_{t}$ in each gradient descent step.

Considering time and space complexity, it is impractical to combine all
generated alphas together, because to calculate mutual correlation for each
pair of factors we need $\mathcal{O}(k^{2})$ evaluations of mutual IC. The
quadratic growth of this makes it expensive to apply the current procedure to
a large number of alphas. However, a few dozen of alphas will suffice for
practical uses. To a certain point, more alphas would not bring much more
increment in performance, following the law of diminishing returns. We will
demonstrate this effect in Section 4.2.2.

Input: Alpha set $\mathcal{F}=\left\\{f_{1},\cdots,f_{k}\right\\}$, weights
$w=\left\\{w_{1},\cdots,w_{k}\right\\}$ and a new alpha $f_{\textrm{new}}$

Output: Optimal alpha subset
$\mathcal{F}^{*}=\left\\{f_{1}^{\prime},\cdots,f_{k}^{\prime}\right\\}$,
optimal weights $w^{*}=\left(w_{1}^{\prime},\cdots,w_{k}^{\prime}\right)$

1
$\mathcal{F}\leftarrow\mathcal{F}\cup\left\\{f_{\textrm{new}}\right\\},w\leftarrow
w\|\mathrm{rand}()$;

2 foreach  _$f\in\mathcal{F}$_ do

3 Obtain $\bar{\sigma}_{y}(f)$ from calculation or cache;

4 foreach  _$f^{\prime}\in\mathcal{F}$_ do

5 Obtain $\bar{\sigma}(f(X),f^{\prime}(X))$ from calculation or cache;

6

7for  _$i\leftarrow 1$ to $num\\_gradient\\_steps$_ do

8 Calculate $\mathcal{L}(w)$ according to Equation 7;
$w\leftarrow\mathrm{GradientDescent}(\mathcal{L}(w))$;

9$p\leftarrow\operatorname*{argmin}_{i}|w_{i}|$;

10 $\mathcal{F}\leftarrow\mathcal{F}\backslash\left\\{f_{p}\right\\}$,
$w\leftarrow\left(w_{1},\cdots,w_{p-1},w_{p+1},\cdots,w_{k}\right)$;

return $\mathcal{F},w$;

Algorithm 1 Incremental Combination Model Optimization

After the alpha generator outputs a new alpha, the alpha is first added to the
candidate alpha set and assigned a random initial weight. Gradient descent is
then performed to optimize the weights with respect to the extended alpha set.
We also set a threshold to limit the size of the alpha set, leaving only the
principal alphas with the largest absolute weight. If the amount of alphas in
the extended set exceeds a certain threshold, the least principal alpha is
removed from the set together with its corresponding weight. The pseudocode of
the training procedure is shown in Algorithm 1.

###  3.2. Alpha Generator

The alpha generator models a distribution of mathematical expressions. As each
expression can be represented as a symbolic expression tree, we use the
reverse Polish notation (RPN) to represent it as a linear sequence, since
traditional auto-regressive generators can only deal with sequences. To
control and evaluate the generation process of valid expressions, we model the
generation process as a non-stationary Markov Decision Process (MDP). We will
describe the various components of the MDP below in the following paragraphs.
An overview of the MDP-based Alpha generator is shown in Figure 3.

####  3.2.1. Tokens

The token is an important abstraction in our framework. A token can be any of
the operators, the features, or constant values. Table 1 shows some examples
of such tokens. For the full list of operators, please refer to Section A; for
the full list of features we have chosen, please refer to Section 4.1.1.

Table 1. Tokens used in our framework.

Category | Examples
---|---
Operators |  _CS-Log_ , _CS-Add_ , _TS-Mean_ , _,_ $\dotsc$
Features |  _$open_ , _$volume_ , $\dotsc$
Constants | $-30,-10,-5,-2,-1,-0.5,-0.01,0.01,0.5,1,2,5,10,30$
Time Deltas | $10d,20d,30d,40d,50d$
Sequence Indicator |  _BEG_(begin), _SEP_(end of expression)

Figure 3. An illustration of our alpha generation framework.

####  3.2.2. State Space

Each state in the MDP corresponds to a sequence of tokens denoting the
currently generated part of the expression. The initial state is always _BEG_
, so a valid state always starts with _BEG_ and is followed by previously
chosen tokens. Since we aim for interpretability of the alphas, and too long
of a formula will instead be less interpretable, we cap the length threshold
of the formulas at 20 tokens.

####  3.2.3. Action Space

An action is a token that follows the current state (generated partial
sequence). It is obvious that an arbitrarily generated sequence is not
guaranteed to be the RPN of an expression, so we only allow a subset of
actions to be taken at a specific state to guarantee the well-formedness of
the RPN sequence. Please refer to Appendix C for more details.

####  3.2.4. Dynamics

Given a state and an action, we can obtain the next state deterministically.
The next state is generated by taking the current state’s corresponding
sequence and appending the action token at the end.

####  3.2.5. Rewards and Returns

The MDP does not give immediate rewards for partially formed sequences. At the
end of each episode, if the final state is valid, the state will be parsed to
a formulaic function and evaluated in the combination model shown in Algorithm
1. To encourage our generator to generate novel alphas, we will then evaluate
the new combination model with the new alpha added, and use the model’s
performance as the return of this episode. Since the reward varies together
with the components of the alpha pool, the MDP is non-stationary.

Contrary to common RL task settings, for alpha expression generation we do not
necessarily want to penalize longer episodes (longer expressions). In fact,
longer alphas that perform well are harder to find than shorter ones, due to
exponential explosion of the search space. Consequently, we set the discount
factor as $\gamma=1$ (no discount).

Input: Stock trend dataset $Y=\left\\{y_{t}\right\\}$

Output: Optimal alpha subset
$F^{*}=\left\\{f_{1}^{\prime},\cdots,f_{k}^{\prime}\right\\}$, optimal weights
$w^{*}=\left\\{w_{1}^{\prime},\cdots,w_{k}^{\prime}\right\\}$

1 Initialize $\mathcal{F}$ and $w$;

2 Initialize RL policy $\pi_{\theta}$ with parameters $\theta$ and replay
buffer $\mathcal{D}$;

3 for  _each iteration_ do

4 for  _each environment step_ do

5 $a_{t}\sim\pi_{\theta}(a_{t}|s_{t})$;

6 $s_{t+1}\leftarrow[s_{t},a_{t}]$;

7 if  _$a_{t}=\emph{SEP}\ or\ len(s_{t+1})\geq threshold$_ then

8 $f\leftarrow parse(s_{t+1})$;

9 Update $\mathcal{F}$, $w$ using $f$ and Algorithm 1;

10 $IC_{new}\leftarrow\bar{\sigma}_{y}(\sum_{i=1}^{k}w_{i}f_{i})$;

11 $r_{t}\leftarrow IC_{new}$;

12 else

13 $r_{t}\leftarrow 0$;

14
$\mathcal{D}\leftarrow\mathcal{D}\cup\left\\{(s_{t},a_{t},r_{t},s_{t+1})\right\\}$;

15 for  _each gradient step_ do

16 Use batch $\mathcal{B}\subset\mathcal{D}$ to do gradient descent on PPO
objective $\mathcal{L}^{CLIP}(\theta)$ to update $\theta$;

17

return $\mathcal{F},w$;

Algorithm 2 Alpha Mining Pipeline

####  3.2.6. Reinforcement Algorithm

Based on the MDP defined above, we use Proximal Policy Optimization (PPO)
(Schulman et al., 2017) to optimize a policy $\pi_{\theta}(a_{t}|s_{t})$ that
takes a state as input and outputs a distribution of action. An actual action
will be sampled from the output distribution.

PPO is an on-policy RL algorithm based on the trust region method. It proposed
a clipped objective $\mathcal{L}^{CLIP}$ as follows:

(8)
$\mathcal{L}^{CLIP}(\theta)=\hat{\mathbb{E}}_{t}\left[\min\left\\{r_{t}(\theta)\hat{A}_{t},\mathrm{clip}\left(r_{t}(\theta),1-\epsilon,1+\epsilon\right)\hat{A}_{t}\right\\}\right],$

where $r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{old}(a_{t}|s_{t})}$
and $\hat{A}_{t}$ is an estimator of the advantage function at timestep $t$.
Using the importance sampling mechanism, PPO can effectively take the biggest
possible improvement while keeping the policy in a trust region that avoids
accidental performance collapse.

Since our MDP has complicated rules for the legality of actions, an action
sampled from the full discrete action distribution predicted by the learned
policy is likely to be invalid as mentioned in Section 3.2.3. We adopt the
Invalid Action Masking mechanism (Huang and Ontañón, 2022) to mask out invalid
actions and just sample from the set of valid actions.

###  3.3. Network Architecture

The PPO algorithm requires the agent to have a value network and a policy
network. Under our experiment settings, the two networks share a base LSTM
feature extractor that converts token sequences into dense vector
representations. Separate value and policy “heads” are attached after the
LSTM. The values of hyperparameters are given in Appendix D.

###  3.4. Training with policy gradient-based methods

For the task of alpha mining, we do not require the agent to achieve
relatively high average returns in each episode, but place more importance on
the trajectories the agent takes in the whole training process. For this
reason, we maintain a pool of alphas without resetting between episodes. We
run the alpha generation procedure mentioned in Section 3.2 and optimize the
alpha combination model according to Section 3.1 repeatedly. In this way, we
train the policy to continuously generate novel alpha factors that bring
improvement to the overall prediction performance.

The proposed alpha mining process is shown in Algorithm 2. Our implementation
is publicly available222 https://github.com/RL-MLDM/alphagen/.

##  4\. Experiments

Our experiments are designed to investigate the following questions:

  * •

Q1: How does our proposed framework compare to prior alpha mining methods?

  * •

Q2: How well does our model scale as the alpha set size increases?

  * •

Q3: Compared to the more commonly used mutual correlation, why is combination
model IC a better metric?

  * •

Q4: How does our framework perform under more realistic trading settings?

###  4.1. Experiment Settings

####  4.1.1. Data

Our experiments are conducted on raw data from the Chinese A-shares
market333The stock price/volume data is retrieved from
https://www.baostock.com. Regarding dividend adjustment, the price/volume data
are all forward-dividend-adjusted respected to the adjustment factors on
2023/01/15.. We select 6 raw features as the inputs to our alphas: {open,
close, high, low, volume, vwap (volume-weighted average price)}. The target is
set to be the 20-day return of the stocks, selling/buying at the closing price
($\textrm{Ref}(\textrm{close},-20)/\textrm{close}-1$). The dataset is split by
date into a training set (2009/01/01 to 2018/12/31), a validation set
(2019/01/01 to 2019/12/31), and a test set (2020/01/01 to 2021/12/31). In the
following experiments, we will use the constituent stocks of the CSI300 and
the CSI500 indices of China A-shares as the stock set.

####  4.1.2. Compared Methods

To evaluate how well our framework performs against traditional formulaic
alpha generation approaches, we implemented two methods that are designed to
generate one alpha at a time. GP is a genetic programming model using the
alpha’s IC as the fitness measure to generate expression trees. This model is
implemented upon the gplearn444https://github.com/trevorstephens/gplearn
framework. PPO is a reinforcement learning method, based on the same PPO
(Schulman et al., 2017) algorithm and expression generator, and uses the
single alpha’s IC as the episode return instead of the combined performance
used in our full framework.

Since only using the top-most alpha to evaluate the frameworks are extremely
prone to overfitting on the training data, we also constructed alpha sets with
the ones generated by the two single alpha generators. The same combination
model is then applied to these alpha sets. Note that the generators still emit
alphas in a one-by-one manner, and are agnostic to the combination model’s
performance. The first method to construct the set (top) is to simply select
the top-$k$ alphas emitted by the generator with the highest IC on the
training set. The second method (filter) is to select the top-$k$ performing
alphas with a constraint that any pair of alpha from the set must not have a
mutual IC higher than 0.7.

To better evaluate the model performance, we also compared our approach to
several end-to-end machine learning models implemented in the open-source
library Qlib (Yang et al., 2020). The models receive 60 days’ worth of raw
features as the input, and are trained to predict the 20-day returns directly.
Note that these models do not generate formulaic alphas. The hyperparameters
of these models are set according to the benchmarks given by Qlib.

  * •

XGBoost (Chen and Guestrin, 2016) is an efficient implementation of gradient
boosting algorithms, which ensembles decision trees to predict stock trends
directly.

  * •

LightGBM (Ke et al., 2017) is another popular implementation of gradient
boosting.

  * •

MLP: A multilayer perceptron (MLP) is a type of fully-connected feedforward
artificial neural network.

To demonstrate the effect caused by stochasticity in the training process,
each experimental combination with an indeterministic training process is
evaluated with 10 different random seeds.

####  4.1.3. Evaluation Metrics

We choose two metrics to measure the performance of our models as follows.

  * •

IC, the Pearson’s correlation coefficient shown in Eq. 1.

  * •

Rank IC, the rank information coefficient. The rank IC tells how much the
ranks of our alpha values are correlated with the ranks of future returns.
Rank IC is defined by replacing Pearson’s correlation coefficient with
Spearman’s correlation coefficient. The rank IC is just the IC of ranked data,
defined as follows:

(9) $\sigma^{\mathrm{rank}}(u,v)=\sigma(r(u),r(v)),$

where $r(\cdot)$ is the ranking operator. The ranks of repeated values are
assigned as the average ranks that they would have been assigned to555For
example, $r(\left(3,-2,6,4\right))=\left(2,1,4,3\right)$, while
$r((3,-2,4,4))=\left(2,1,3.5,3.5\right)$..

Both of the metrics are the higher the better.

Table 2. Main results on CSI 300 and CSI 500. Values outside parentheses are
the means, and values inside parentheses are the standard deviations across 10
runs.

Method | CSI 300 | CSI 500
---|---|---
IC($\uparrow$) |  Rank IC($\uparrow$) |  IC($\uparrow$) |  Rank IC($\uparrow$)
MLP |  $\hphantom{-}0.0250$ |  $\hphantom{-}0.0401$ |  $0.0188$ |  $0.0458$
$\hphantom{-}(0.0068)$ |  $\hphantom{-}(0.0081)$ |  $(0.0018)$ |  $(0.0045)$
XGBoost |  $\hphantom{-}0.0404$ |  $\hphantom{-}0.0576$ |  $0.0353$ |  $0.0639$
LightGBM |  $\hphantom{-}0.0259$ |  $\hphantom{-}0.0324$ |  $0.0332$ |  $0.0609$
PPO_top* |  $-0.0166$ |  $-0.0144$ |  $0.0025$ |  $0.0295$
$\hphantom{-}(0.0028)$ |  $\hphantom{-}(0.0075)$ |  $(0.0076)$ |  $(0.0135)$
GP_top* |  $\hphantom{-}0.0078$ |  $\hphantom{-}0.0157$ |  $0.0200$ |  $0.0504$
$\hphantom{-}(0.0218)$ |  $\hphantom{-}(0.0271)$ |  $(0.0112)$ |  $(0.0160)$
PPO_filter* |  $-0.0044$ |  $\hphantom{-}0.0101$ |  $0.0042$ |  $0.0506$
$\hphantom{-}(0.0107)$ |  $\hphantom{-}(0.0107)$ |  $(0.0042)$ |  $(0.0052)$
GP_filter* |  $\hphantom{-}0.0183$ |  $\hphantom{-}0.0298$ |  $0.0117$ |  $0.0562$
$\hphantom{-}(0.0190)$ |  $\hphantom{-}(0.0227)$ |  $(0.0083)$ |  $(0.0105)$
Ours* |  0.0725 |  0.0806 |  0.0438 |  0.0727
$\hphantom{-}(0.0105)$ |  $\hphantom{-}(0.0106)$ |  $(0.0064)$ |  $(0.0112)$
*Optimal combination size in $\left\\{10,20,50,100\right\\}$

Figure 4. The results of ablation study. A pool size of 1 refers to settings that only evaluate the top-most alpha without using a combination model. Table 3. An example combination of 10 alphas. # |  Alpha |  Weight |  IC (CSI300)
---|---|---|---
1 |  $\mathrm{Var}(\mathrm{Greater}(\mathrm{Greater}(\mathrm{Var}(\mathrm{low},50),\mathrm{high}),\mathrm{open}),30)$ |  $-0.0295$ |  $\hphantom{-}0.0011$
2 |  $\mathrm{Max}((\mathrm{Min}(\mathrm{Max}(\mathrm{close},20)-30,20)+100)/30,20)$ |  $\hphantom{-}0.0515$ |  $\hphantom{-}0.0262$
3 |  $(\mathrm{Mad}(\mathrm{high},50)+0.5)\mathrm{vwap}/\mathrm{close}$ |  $\hphantom{-}0.0343$ |  $\hphantom{-}0.0447$
4 |  $\mathrm{Ref}(\mathrm{low},50)$ |  $\hphantom{-}0.0260$ |  $\hphantom{-}0.0241$
5 |  $\mathrm{Min}(\mathrm{high}/\mathrm{close},50)-\mathrm{Greater}(-0.05/\mathrm{close},-10)$ |  $\hphantom{-}0.0437$ |  $-0.0211$
6 |  $\mathrm{Delta}(\mathrm{high},20)+\mathrm{high}-12$ |  $-0.0997$ |  $\hphantom{-}0.0165$
7 |  $\mathrm{Less}(\mathrm{Min}(2(\mathrm{vwap}-\mathrm{volume}),30)+30,30\mathrm{vwap}/\mathrm{low})-5$ |  $\hphantom{-}0.0276$ |  $\hphantom{-}0.0025$
8 |  $\mathrm{Corr}(\mathrm{Greater}(\mathrm{Greater}(\mathrm{vwap},\mathrm{volume}),\mathrm{Greater}(\mathrm{close},$
$\mathrm{Greater}(\mathrm{Log}(\mathrm{Var}(\mathrm{volume},10)),10))/\mathrm{close}),\mathrm{close},10)$ |  $-0.0279$ |  $-0.0338$
9 |  $|\mathrm{low}-30|$ |  $\hphantom{-}0.0319$ |  $\hphantom{-}0.0073$
10 |  $\mathrm{Max}(1-\mathrm{Max}(\mathrm{Corr}(\mathrm{low},\mathrm{volume}\cdot\mathrm{Log}(10^{-4}\mathrm{Max}(\mathrm{volume},10)),10),10),30)$ |  $\hphantom{-}0.0312$ |  $\hphantom{-}0.0488$
Weighted Combination |  $\hphantom{-}0.0511$

###  4.2. Main Results

####  4.2.1. Comparison across all alpha generators

To answer Q1, we first compare our framework against several other alpha-
mining methods and direct stock trend forecasting baselines, including PPO,
GP, MLP, LightGBM, and XGBoost. Experiments are conducted on CSI300 and CSI500
stocks respectively.

The results are shown in Table 2. Our framework is able to achieve the highest
IC and rank IC across all the methods we compare to. Note that the framework
is only explicitly optimized against the IC metric. The non-formulaic alpha
models come in the second tier. The baseline formulaic alpha generators
perform poorly on the test set, especially the RL-based ones. The
reinforcement learning agent, when optimized only against single-alpha IC, is
prone to falling into local optima and thus overfitting on the training set,
and basically stops searching for new alphas after a certain amount of steps.
On the other hand, the GP-based methods maintaining a large population can
avoid the same problem, but still cannot produce alphas that are synergistic
when used together. The results also show that the filtering techniques cannot
solve the synergy problem consistently either.

####  4.2.2. Comparison of formulaic generators with varying pool capacity

To answer Q2, we study the four baseline formulaic alpha generators more
extensively, and compare them to our proposed framework. The models are
evaluated under pool sizes of $k\in\left\\{1,10,20,50,100\right\\}$. The
results are shown in Figure 4.

Compared to the baseline method PPO_filter, our method directly uses the
combination model’s performance as the reward to newly generated alphas. This
leads to a substantial improvement when the pool size increases, meaning that
our method can produce alpha sets with great synergy. Our method shows
scalability for pool size: even when the pool size is large enough, it can
still continuously find synergistic alphas that boost the performance over the
existing pool. Conversely, the combined performance of the alphas generated by
other approaches barely improves upon the case with just the top alpha,
meaning that these alpha factors have poor synergy. Furthermore, the ability
to control the reward of individual expressions under a certain alpha pool
configuration is granted by the flexibility of the RL scheme. The GP scheme of
maintaining a large population at the same time does not work well with fine-
grained fitness value control.

Also, we can see that for the CSI500 dataset, GP_filter performs worse than
GP_top on the IC metric when the pool size increases. This phenomenon
demonstrates that the traditionally used mutual-IC filtering is not always
effective, answering the question Q3.

###  4.3. Case Study

Table 3 shows an example combination of 10 alphas generated by our framework,
evaluated on the CSI300 constituent stock set. Most of the alpha pairs in this
specific set have mutual IC values over 0.7. Previous work (Lin et al.,
2019b)(Zhang et al., 2020) considered this to be too high for the individual
alphas to be regarded as “diverse”, yet these alphas are able to work well in
a synergistic manner. For example, the alphas #2 and #6 have a mutual IC of
$0.9746$, thus traditionally considered too similar to be useful
cooperatively. However, the combination $0.09317f_{2}-0.07163f_{6}$ achieves
an IC of $0.0458$ on the test set, even higher than the sum of the respective
ICs, showing the synergy effect.

Also, although alpha #1 only has an IC of 0.0011, it still plays a vital role
in the final combination. Once we remove alpha #1 from the combination and re-
train the combination weights on the remaining set, the combination’s IC drops
to merely $0.0447$. The two observations above show that neither the single
alpha IC nor the mutual IC between alpha pairs is a good indicator of how well
the combined alpha would perform, answering Q3.

One possible explanation for these phenomena is that: Although traditionally
these alphas are similar due to the high mutual IC, some linear combinations
of the alphas could point to a completely different direction from the
original ones. Consider two unit vectors in a linear space. The more similar
these two vectors are, the less similar either of these vectors is to the
difference between the two vectors, since the difference vector approaches to
be perpendicular to either of the original vectors as the vectors get closer.

Figure 5. Backtest results on CSI 300. The lines track the net worth of
simulated trading agents utilizing the various alpha-mining approaches.

###  4.4. Investment Simulation

To demonstrate the effectiveness of our factors in more realistic investing
settings, we use a simple investment strategy and conducted backtests in the
testing period (2020/01/01 to 2021/12/31) on the CSI300 dataset. We use a
simple _top- $k$/drop-$n$_ strategy to simulate the investment: On each
trading day, we first sort the alpha values of the stocks, and then select the
top $k$ stocks in that sorted list. We evenly invest across the $k$ stocks if
possible, but restrict the strategy to only buy/sell at most $n$ stocks on
each day to reduce excessive trading costs. In our experiment, $k$ is set to
50 and $n$ to 5.

We recorded the net worth of the respective strategies in the testing period,
of which a line chart is shown in Figure 5. Although our framework does not
explicitly optimize towards the absolute returns, the framework still performs
well in the backtest. Our framework is able to gain the most profit compared
to the other methods.

##  5\. Related Work

Formulaic alphas. The search space of formulaic alphas is enormous, due to the
large amount of possible operators and features to choose from. To our best
knowledge, all notable former work uses genetic programming to explore this
huge search space. (Lin et al., 2019b) augmented the gplearn library with
formulaic-alpha-specific time-series operators, upon which an alpha-mining
framework is built. (Lin et al., 2019a) further improved the framework to also
mine alphas with non-linear relations with the returns by using mutual
information as the fitness measure. (Zhang et al., 2020) used mutual IC to
filter out alphas that are too similar to existing ones, improving the
diversity of resulting alpha sets. PCA is carried out on the alpha values for
reducing the algorithmic complexity of computing the mutual ICs, and various
other tricks are also applied to aid the evolution process. AlphaEvolve (Cui
et al., 2021) evolves new alphas upon existing ones. It allows combinations of
much more complex operations (for example matrix-wise computations), and uses
computation graphs instead of trees to represent the alphas. This leads to
more sophisticated alphas and better prediction accuracy, although at the risk
of lowering the alphas’ interpretability. Mutual IC is also used as a measure
of alpha synergy in this work.

Machine learning-based alphas. The development of deep learning in recent
years has brought about various new ideas on how to accurately model stock
trends. Early work on stock trend forecasting treats the movement of each
stock as a separate time series, and applies time series models like LSTM
(Hochreiter and Schmidhuber, 1997) or Transformer (Vaswani et al., 2017) to
the data. Specific network structures catered to stock forecasting like the
SFM (Zhang et al., 2017) which uses a DFT-like mechanism have also been
developed. Recently, research has also been conducted on methods to integrate
non-standard data with the time series. REST (Xu et al., 2021b) fuses multi-
granular time series data together with historical event data to model the
market as a whole. HIST (Xu et al., 2021a) utilizes concept graphs on top of
the regular time series data to model shared commonness between future trends
of various stock groups. One specific type of machine learning-based model is
also worth mentioning. Decision tree models, notably XGBoost (Chen and
Guestrin, 2016), LightGBM (Ke et al., 2017), etc., are often considered
interpretable, and they could also achieve relatively good performance on
stock trend forecasting tasks. However, whether a decision tree with extremely
complex structure is considered “interpretable” is at least questionable. When
these tree models are applied to raw stock data, the high dimensionality of
input only exacerbates the aforementioned problem. Our formulaic alphas use
operators that apply to the input data in a more structured manner, making
them more easily interpretable by curious investors.

Symbolic regression. Symbolic regression (SR) concerns the problem of
discovering relations between variables represented in closed-form
mathematical formulas. SR problems are different from our problem settings
that there always exists a “groundtruth“ formula that precisely describes the
data points in an SR problem, while stock market trends are far too complex to
be expressed in the space of formulaic alphas. Nevertheless, there remain
similarities between the two fields since similar techniques can be used for
the expression generator and the optimization procedure. (Sahoo et al., 2018)
suggested using a custom neural network whose activation functions are
symbolic operators to solve the SR problem. (Petersen et al., 2021) proposed a
novel symbolic regression framework based on an autoregressive expression
generator. The generator is optimized using an augmented version of the policy
gradient algorithm that values the top performance of the agent more than the
average. (Mundhenk et al., 2021) developed a method similar to (Petersen et
al., 2021), but also introduced GP into the optimization loop, seeding the GP
population with RL outputs. (Valipour et al., 2021) applied the language model
pretraining scheme to symbolic regression, training a generative
autoregressive “language model” of expressions on a large dataset of synthetic
expressions.

Discussions. Although the term “formulaic alpha” is often tied down to
investing, the concept of simple and interpretable formulaic predictors that
could be combined into more expressive models is not limited to quantitative
trading scenarios. Our framework can be adapted to solve other time-series
forecasting problems, for example, energy consumption prediction (Deb et al.,
2017), anomaly detection (Ashok et al., 2018), biomedical settings (Tuarob et
al., 2017), etc. In addition, we chose the linear combination model in this
paper for its simplicity. Meanwhile, in theory, other types of interpretable
combination models, for example, decision trees can also be integrated into
our framework. In that sense, providing these combination models with these
features expressed in relatively straightforward formulas might help provide
investigators with more insights into how the models come to the final
results.

##  6\. Conclusion

In this paper, we proposed a new framework for generating interpretable
formulaic alphas to aid investors in quantitative trading. We proposed to
directly use the performance boost brought about by the newly added alpha to
the existing alpha combination as the metric for alpha synergy. As a result,
our framework can produce sets of alphas that could cooperate satisfactorily
with a combination model, notwithstanding the actual form of the combination
model. For the model to explore the vast search space of formulaic alphas more
effectively, we also formulated the alpha-searching procedure as an MDP and
applied reinforcement learning techniques to optimize the alpha generator.
Extensive experiments are conducted to demonstrate that the performance of our
framework surpasses those of all previous formulaic alpha-mining approaches,
and that our method can also perform well under more realistic trading
settings.

###### Acknowledgements.

The research work was supported by the National Key Research and Development
Program of China under Grant No. 2022YFC3303302, the National Natural Science
Foundation of China under Grant No.61976204. Xiang Ao is also supported by the
Project of Youth Innovation Promotion Association CAS, Beijing Nova Program
Z201100006820062.

## References

  * (1)
  * Ashok et al. (2018) Aditya Ashok, Manimaran Govindarasu, and Venkataramana Ajjarapu. 2018\.  Online Detection of Stealthy False Data Injection Attacks in Power System State Estimation.  _IEEE Trans. Smart Grid_ 9, 3 (2018), 1636–1646.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A Scalable Tree Boosting System. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , Balaji Krishnapuram, Mohak Shah, Alexander J. Smola, Charu C. Aggarwal, Dou Shen, and Rajeev Rastogi (Eds.). ACM, 785–794.
  * Cui et al. (2021) Can Cui, Wei Wang, Meihui Zhang, Gang Chen, Zhaojing Luo, and Beng Chin Ooi. 2021\.  AlphaEvolve: A Learning Framework to Discover Novel Alphas in Quantitative Investment. In _SIGMOD ’21: International Conference on Management of Data, Virtual Event, China, June 20-25, 2021_ , Guoliang Li, Zhanhuai Li, Stratos Idreos, and Divesh Srivastava (Eds.). ACM, 2208–2216.
  * Deb et al. (2017) Chirag Deb, Fan Zhang, Junjing Yang, Siew Eang Lee, and Kwok Wei Shah. 2017.  A review on time series forecasting techniques for building energy consumption.  _Renewable and Sustainable Energy Reviews_ 74 (2017), 902–924.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long Short-Term Memory.  _Neural Comput._ 9, 8 (1997), 1735–1780.
  * Huang and Ontañón (2022) Shengyi Huang and Santiago Ontañón. 2022\.  A Closer Look at Invalid Action Masking in Policy Gradient Algorithms. In _Proceedings of the Thirty-Fifth International Florida Artificial Intelligence Research Society Conference, FLAIRS 2022, Hutchinson Island, Jensen Beach, Florida, USA, May 15-18, 2022_ , Roman Barták, Fazel Keshtkar, and Michael Franklin (Eds.).
  * Kakushadze (2016) Zura Kakushadze. 2016\.  101 Formulaic Alphas.  arXiv:1601.00991
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017\.  LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 3146–3154.
  * Lin et al. (2019a) Xiaoming Lin, Ye Chen, Ziyu Li, and Kang He. 2019a.  _Revisiting Stock Alpha Mining Based On Genetic Algorithm_.  Technical Report. Huatai Securities Research Center.  https://crm.htsc.com.cn/doc/2019/10750101/3f178e66-597a-4639-a34d-45f0558e2bce.pdf
  * Lin et al. (2019b) Xiaoming Lin, Ye Chen, Ziyu Li, and Kang He. 2019b.  _Stock Alpha Mining Based On Genetic Algorithm_.  Technical Report. Huatai Securities Research Center.  https://crm.htsc.com.cn/doc/2019/10750101/f75b4b6a-2bdd-4694-b696-4c62528791ea.pdf
  * Liu et al. (2020) Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. 2020.  You Impress Me: Dialogue Generation via Mutual Persona Perception. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 1417–1427.
  * Mundhenk et al. (2021) T. Nathan Mundhenk, Mikel Landajuela, Ruben Glatt, Daniel M. Faissol, and Brenden K. Petersen. 2021\.  Symbolic Regression via Neural-Guided Genetic Programming Population Seeding.  (2021).  arXiv:2111.00053
  * Petersen et al. (2021) Brenden K. Petersen, Mikel Landajuela, T. Nathan Mundhenk, Cláudio Prata Santiago, Sookyung Kim, and Joanne Taery Kim. 2021\.  Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradients. In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_.
  * Qian (2007) Edward E Qian. 2007\.  _Quantitative equity portfolio management: modern techniques and applications_.  Chapman and Hall/CRC.
  * Sahoo et al. (2018) Subham S. Sahoo, Christoph H. Lampert, and Georg Martius. 2018.  Learning Equations for Extrapolation and Control. In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ _(Proceedings of Machine Learning Research, Vol. 80)_ , Jennifer G. Dy and Andreas Krause (Eds.). PMLR, 4439–4447.
  * Schrittwieser et al. (2020) Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. 2020\.  Mastering atari, go, chess and shogi by planning with a learned model.  _Nature_ 588, 7839 (2020), 604–609.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal Policy Optimization Algorithms.  (2017).  arXiv:1707.06347
  * Tuarob et al. (2017) Suppawong Tuarob, Conrad S. Tucker, Soundar R. T. Kumara, C. Lee Giles, Aaron L. Pincus, David E. Conroy, and Nilam Ram. 2017\.  How are you feeling?: A personalized methodology for predicting mental states from temporally observable physical and behavioral information.  _J. Biomed. Informatics_ 68 (2017), 1–19.
  * Tulchinsky (2015) I. Tulchinsky. 2015\.  _Finding Alphas: A Quantitative Approach to Building Trading Strategies_.  1–253 pages.
  * Valipour et al. (2021) Mojtaba Valipour, Bowen You, Maysum Panju, and Ali Ghodsi. 2021\.  SymbolicGPT: A Generative Transformer Model for Symbolic Regression.  (2021).  arXiv:2106.14131
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 5998–6008.
  * Wang et al. (2021) Zhicheng Wang, Biwei Huang, Shikui Tu, Kun Zhang, and Lei Xu. 2021.  DeepTrader: A Deep Reinforcement Learning Approach for Risk-Return Balanced Portfolio Management with Market Conditions Embedding. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_. AAAI Press, 643–650.
  * Xu et al. (2021a) Wentao Xu, Weiqing Liu, Lewen Wang, Yingce Xia, Jiang Bian, Jian Yin, and Tie-Yan Liu. 2021a.  HIST: A Graph-based Framework for Stock Trend Forecasting via Mining Concept-Oriented Shared Information.  (2021).  arXiv:2110.13716
  * Xu et al. (2021b) Wentao Xu, Weiqing Liu, Chang Xu, Jiang Bian, Jian Yin, and Tie-Yan Liu. 2021b.  REST: Relational Event-driven Stock Trend Forecasting. In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia (Eds.). ACM / IW3C2, 1–10.
  * Yang et al. (2020) Xiao Yang, Weiqing Liu, Dong Zhou, Jiang Bian, and Tie-Yan Liu. 2020.  Qlib: An AI-oriented Quantitative Investment Platform.  (2020).  arXiv:2009.11189
  * Zhang et al. (2017) Liheng Zhang, Charu C. Aggarwal, and Guo-Jun Qi. 2017\.  Stock Price Prediction via Discovering Multi-Frequency Trading Patterns. In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifax, NS, Canada, August 13 - 17, 2017_. ACM, 2141–2149.
  * Zhang et al. (2020) Tianping Zhang, Yuanqi Li, Yifei Jin, and Jian Li. 2020\.  AutoAlpha: an Efficient Hierarchical Evolutionary Algorithm for Mining Alpha Factors in Quantitative Investment.  _arXiv preprint arXiv:2002.08245_ (2020).

Table 4. All the operators used in our framework. CS: cross-section, TS: time-series, U: unary, B: binary. Operator | Category |  Descriptions
---|---|---
Abs($x$) | CS–U |  The absolute value $|x|$.
Log($x$) | CS–U |  Natural logarithmic function $\log(x)$.
$x+y$, $x-y$, $x\cdot y$, $x/y$ | CS–B |  Arithmetic operators.
$\mathrm{Greater}(x,y)$, $\mathrm{Less}(x,y)$ | CS–B |  The larger/smaller one of the two values.
$\mathrm{Ref}(x,t)$ | TS–U |  The expression $x$ evaluated at $t$ days before the current day.
$\mathrm{Mean}(x,t)$, $\mathrm{Med}(x,t)$, $\mathrm{Sum}(x,t)$ | TS–U |  The mean/median/sum value of the expression $x$ evaluated on the recent $t$ days.
$\mathrm{Std}(x,t)$, $\mathrm{Var}(x,t)$ | TS–U |  The standard deviation/variance of the expression $x$ evaluated on recent $t$ days.
$\mathrm{Max}(x,t)$, $\mathrm{Min}(x,t)$ | TS–U |  The maximum/minimum value of the expression $x$ evaluated on the recent $t$ days.
$\mathrm{Mad}(x,t)$ | TS–U |  The mean absolute deviation $\mathbb{E}\left[|x-\mathbb{E}\left[x\right]|\right]$ of the expression $x$ evaluated on the recent $t$ days.
$\mathrm{Delta}(x,t)$ | TS–U |  The relative difference of $x$ compared to $t$ days ago, $x-\mathrm{Ref}(x,t)$.
$\mathrm{WMA}(x,t)$, $\mathrm{EMA}(x,t)$ | TS–U |  Weighted moving average and exponential moving average of the expression $x$ evaluated on the recent $t$ days.
$\mathrm{Cov}(x,y,t)$ | TS–B |  The covariance between two time series $x$ and $y$ in the recent $t$ days.
$\mathrm{Corr}(x,y,t)$ | TS–B |  The Pearson’s correlation coefficient between two time series $x$ and $y$ in recent $t$ days.

##  Appendix A List of Operators

There are four types of operators used in our framework. The four types break
down into two groups: cross-section operators, and time-series operators.
Cross-section operators (denoted with “CS” in the table) only deal with data
on the current trading day, while time-series operators (denoted with “TS”)
take into consideration data from a consecutive period of time. Each of the
two groups further separates into unary (denoted with “U”) and binary (denoted
with “B”) operators that apply to one or two series respectively.

##  Appendix B Proof of Theorem 3.1

###### Proof.

We know that the elements of a vector $u$ that is centralized and normalized
(using the $\mathcal{N}$ operator mentioned above) have a variance of $1/n$,
since:

(10) $\displaystyle\mathrm{Var}\left[u\right]$
$\displaystyle=\mathbb{E}_{i}\left[u_{i}^{2}\right]-\mathbb{E}_{i}\left[u_{i}\right]^{2}$
$\displaystyle=\frac{1}{n}\|u\|^{2}-0$ $\displaystyle=\frac{1}{n}.$

Using the original definition of Pearson’s correlation coefficient, we have:

(11) $\displaystyle\sigma(u,v)$
$\displaystyle=\frac{\mathrm{Cov}\left[u,v\right]}{\sqrt{\mathrm{Var}\left[u\right]\cdot\mathrm{Var}\left[v\right]}}$
$\displaystyle=\mathbb{E}_{i}\left[\frac{(u_{i}-\bar{u})}{\sqrt{\mathrm{Var}\left[u\right]}}\cdot\frac{(v_{i}-\bar{v})}{\sqrt{\mathrm{Var}\left[v\right]}}\right]$
$\displaystyle=\mathbb{E}_{i}\left[\frac{\left[\mathcal{N}(u)\right]_{i}}{\sqrt{1/n}}\cdot\frac{\left[\mathcal{N}(v)\right]_{i}}{\sqrt{1/n}}\right]$
$\displaystyle=n\mathbb{E}_{i}\left[\left[\mathcal{N}(u)\right]_{i}\left[\mathcal{N}(v)\right]_{i}\right]$
$\displaystyle=\left\langle\mathcal{N}(u),\mathcal{N}(v)\right\rangle.$

That is to say, the Pearson’s correlation coefficient between two vectors
equals the inner product of the two vectors centralized and normalized.

Therefore the theorem can be proved as follows. Recall that $f_{i}(x_{t})$ and
$y_{t}$ are normalized.

(12) $\displaystyle n\mathcal{L}(w)$
$\displaystyle=\frac{1}{T}\sum_{t=1}^{T}\|z_{t}-y_{t}\|_{2}^{2}$
$\displaystyle=\mathbb{E}_{t}\left[\|z_{t}\|^{2}-2\left\langle
z_{t},y_{t}\right\rangle+\|y_{t}\|^{2}\right]$
$\displaystyle=\mathbb{E}_{t}\left[\left\|\sum_{i=1}^{k}w_{i}f_{i}(X_{t})\right\|^{2}-2\left\langle\sum_{i=1}^{k}w_{i}f_{i}(X_{t}),y_{t}\right\rangle+1\right]$
$\displaystyle=\mathbb{E}_{t}\left[\sum_{i=1}^{k}\sum_{j=1}^{k}w_{i}w_{j}\sigma(f_{i}(X_{t}),f_{j}(X_{t}))\right.$
$\displaystyle\qquad\left.-\ 2\sum_{i=1}^{k}w_{i}\left\langle
f_{i}(X_{t}),y_{t}\right\rangle+1\right]$
$\displaystyle=\sum_{i=1}^{k}\sum_{j=1}^{k}w_{i}w_{j}\bar{\sigma}(f_{i}(X),f_{j}(X))-2\sum_{i=1}^{k}w_{i}\bar{\sigma}_{y}(f_{i})+1.$

∎

##  Appendix C Expression Legality Guarantee

The legality of expressions divides into two parts: _Formal_ legality and
_semantic_ legality.

###  C.1. Formal Legality

An RPN can be built with a stack of expressions, constants, or raw features.
The RPN building procedure follows the rules below, and actions that may
violate these rules will be masked.

  * •

TS (time-series) operators must take a time-delta (e.g. 10d for a time-
difference of 10 days) as its last parameter;

  * •

Excluding the aforementioned time-delta, each operator must take enough
expressions as operands, according to the arity of the operator (one for
*-Unary, two for *-Binary);

  * •

A multi-token expression should not be equivalent to a constant;

  * •

The special SEP token (end of expression) is only allowed when the generated
sequence is already a valid RPN.

For example, when the stack (state) is currently [$open, 0.5], we can choose
the “Add” token (a binary operator), building an expression “Add($open, 0.5)”.
Meanwhile, the operator “Log” is not allowed here because “Log” will take
“0.5” and “Log(0.5)” is a constant; similarly, the operator “TS-Mean” is also
invalid because “Mean($open, 0.5)” is illegal.

###  C.2. Semantic Legality

Some expressions with correct forms might still fail to evaluate due to more
constraints imposed by the operators. For example, the logarithm operator
cannot be applied to a non-positive value. This kind of semantic invalidity is
not directly detected by the procedure mentioned in the last section. In our
experiments, these expressions are given the reward of -1 (the minimum value
of Pearson’s correlation coefficient) to discourage the agent from generating
these expressions.

##  Appendix D Hyperparameters

The LSTM feature extractor used in the RL agent has a 2-layer structure with a
hidden layer dimension of 128. A dropout rate of 0.1 is used in the LSTM
network. The separate value and policy heads are MLPs with two hidden layers
of 64 dimensions. PPO clipping range $\epsilon$ is set to 0.2.