# ADD: Augmented Disentanglement Distillation Framework for Improving Stock
Trend Forecasting

Hongshun Tang1, Lijun Wu2, Weiqing Liu2, Jiang Bian2
1Peking University
2Microsoft Research Asia
<EMAIL_ADDRESS>{Lijun.Wu, Weiqing.Liu<EMAIL_ADDRESS>This
work was done when the author was an intern at Microsoft Research Asia.

###### Abstract

Stock trend forecasting has become a popular research direction that attracts
widespread attention in the financial field. Though deep learning methods have
achieved promising results, there are still many limitations, for example, how
to extract clean features from the raw stock data. In this paper, we introduce
an _Augmented Disentanglement Distillation (ADD)_ approach to remove
interferential features from the noised raw data. Specifically, we present 1)
a disentanglement structure to separate excess and market information from the
stock data to avoid the two factors disturbing each other’s own prediction.
Besides, by applying 2) a dynamic self-distillation method over the
disentanglement framework, other implicit interference factors can also be
removed. Further, thanks to the decoder module in our framework, 3) a novel
strategy is proposed to augment the training samples based on the different
excess and market features to improve performance. We conduct experiments on
the Chinese stock market data. Results show that our method significantly
improves the stock trend forecasting performances, as well as the actual
investment income through backtesting, which strongly demonstrates the
effectiveness of our approach.

##  1 Introduction

Recently, stock trend forecasting has become an important research topic in
the financial field. In practice, the stock return can be divided into two
parts, one is the market return, which represents the overall trend of the
stock market, and the other one is the excess return, which displays the
changes of each stock affected by its own factors outside the market changes
[22]. Both returns provide valuable information to benefit real investors to
analyse the market trend and each specific stock so to help make investment
decision. Therefore, predicting market return and excess return are both
worthy in stock trend forecasting.

To model the prediction of stock return, traditional methods include
AutoRegressive [20], Moving Average [23] and Kalman Filters [3]. Nowadays, the
deep neural networks (DNNs) have shown great potential, which significantly
outperforms the conventional approaches [15, 19, 5]. Though promising results
are obtained, the _interferential features_ existed in the noised raw stock
data have seriously restricted the model performance [9, 32]. In this work, we
focus on the market and excess return predictions, and we find the factors
related to these two returns are highly entangled/correlated in the noised raw
data, which makes the market factors be interferential features for the excess
return prediction [12]. Similarly, the excess factors also hurt the
performance of market return prediction. Besides, other noisy interferential
factors also disrupt the two return predictions. Therefore, it is critical to
remove the interferential features so that both return predictions can be
benefited.

In order to decouple the entanglement between the market and excess factors,
we present an Augmented Disentanglement Distillation (ADD) approach, where 1)
a _disentanglement framework_ to separate the market and excess information,
and 2) a _dynamic self-distillation_ method to enhance the related return
features. Also, 3) a novel _data augmentation_ strategy is introduced, which
can greatly improve the performance for rare data samples. Specifically, our
disentanglement model consists of two separated encoders, two return
predictors and a reconstructive decoder. The market and excess features are
extracted by their respective encoders. By utilizing the corresponding
predictors (which predict the market/excess returns using market/excess
information), and two extra predictors with adversarial training (which
prevent the market/excess features predicting the excess/market returns), the
market and excess factors can be disentangled.

The self-distillation training method is leveraged to further enhance the
clean knowledge for market/excess features. Different from conventional
knowledge distillation, we dynamically distill the knowledge from the teacher
model with weights calculated according to the teacher performance, so to
better control the contribution from the teacher knowledge and raw data during
student model training.

Thanks to the reconstructive decoder and separated encoders, we augment the
stock training data in a novel way. Concretely, we feed the disentangled
features outputted by market/excess encoders from different original samples
into decoder, so to get the augmented new samples. In practice, we focus more
on augmenting those rare/hard samples.

We conduct experiments on the Chinese stock market of A-shares from $2007$ to
$2019$. In addition, we also add a trading strategy backtesting experiment to
evaluate our method in real investment market. The strong results of the
excess and market return predictions demonstrate the effectiveness of our
method, and the backtesting results further prove that our approach is
beneficial for real investment.

In summary, our main contributions are as follows:

  * •

We propose a disentanglement structure to separate the excess and market
information and remove the entangled relationship between them.

  * •

We develop a dynamic self-distillation method through iterative training to
further enhance the return features.

  * •

We introduce a novel augmentation method based on the disentanglement features
to improve model performance, especially for the rare samples.

  * •

Experiments show that our Augmented Disentanglement Distillation (ADD) method
can improve the stock trend forecasting and increase the investment income.

##  2 Method

###  2.1 Background

Before discussing our method, we first introduce the background and necessary
notations.

Stock return is often defined as the future change rate of stock price, which
is formulated as:

$\displaystyle r_{i}^{j}=\frac{price_{i}^{j+1}-price_{i}^{j}}{price_{i}^{j}},$
(1)

where $r_{i}^{j}$ and $price_{i}^{j}$ are the stock return and price of stock
$s_{i}$ on day $d_{j}$. The price can be opening price, closing price or
Volume Weighted Average Price, we choose closing price in the work. The daily
market return is the average of the stock returns of all stocks on a certain
day, which represents the overall trend of stock market. The excess return is
the difference between the stock return and market return, which represents
the changes of each stock outside the market changes.

We formalize the task as follows: $X$ denotes the stock features (see Section
4.1), $Y_{E}$ and $Y_{M}$ are the labels of excess and market returns. Given a
dataset $D=\\{(x,y_{e},y_{m})\\},x\in X,y_{e}\in Y_{E},y_{m}\in Y_{M}$, the
goal is to predict the excess returns $Y_{E}$ and the market returns $Y_{M}$
based on $X$.

The market return for all stocks is same in one day, which makes the number of
the market return values be far smaller than stock samples. Therefore, we
perform a classification task for the market return. We divide all market
returns into three categories by the thresholds calculated on the market
distribution from training set. The thresholds fulfill that each category
holding the same number of training samples, and they are also reused in
validation and test sets. The three categories are the market return ‘going
up’, ‘moving steady’ and ‘going down’. As for the excess prediction, we carry
on the regression task.

###  2.2 Disentanglement Framework

As discussed before, the market factors are interferential features for excess
return prediction, and vice versa. To separate the market and excess factors
for accurate predictions, we propose a disentanglement framework (shown in
Figure 1). The framework consists of two encoders, one decoder and four
predictors. Specifically, we use excess encoder EncE and market encoder EncM
to encode stock features $X$ into excess feature $f_{E}=\texttt{Enc}_{E}(X)$
and market feature $f_{M}=\texttt{Enc}_{M}(X)$. The decoder Dec takes the
disentangled $f_{E}$ and $f_{M}$ as inputs and outputs $\widehat{X}$, which is
to recover the original inputs $X$. The decoder is designed to reduce the loss
of information during encoding process, which plays an important role for
later augmentation (Section 2.4).

Figure 1: The overall disentanglement framework, which contains the excess and
market return predictors, the two corresponding adversarial predictors and the
reconstruction decoder.

To keep the features $f_{E}$ and $f_{M}$ only related to its own return, we
build two predictors for each disentangled feature, one for preserved
information (e.g., excess return predictor PreE) and the other one for
unrelated information (e.g., adversarial predictor AdvM). The objective is to
minimize the sum of three weighted terms: 1) the loss of preserved
information, 2) the negative adversarial loss of unrelated information, and 3)
the reconstruction loss of decoder.

Formally, let $\theta_{Enc}$ be the parameters of EncE and EncM,
$\theta_{Dec}$ be the parameters of Dec, $\theta_{Pre}$ be the parameters of
PreE and PreM, $\theta_{Adv}$ be the parameters of AdvE and AdvM. We define
$\mathcal{L}_{Pre}$ as the prediction losses for PreE and PreM,
$\mathcal{L}_{Adv}$ for AdvE and AdvM, $\mathcal{L}_{Rec}$ for Dec. As for
optimization, the objective is to simultaneously minimize $\mathcal{L}_{Pre}$
and $\mathcal{L}_{Rec}$ while maximize $\mathcal{L}_{Adv}$. $\lambda$ and
$\mu$ are used to weight the losses. The two training objectives are executed
alternately, just like the general process of adversarial training. Assume
that MSE represents mean square error, and CE represents cross entropy, the
whole training loss is:

$\displaystyle\mathcal{L}_{Pre}$
$\displaystyle=\texttt{MSE}(\texttt{Pre}_{E}(f_{E}),Y_{E})+\texttt{CE}(\texttt{Pre}_{M}(f_{M}),Y_{M}),$
$\displaystyle\mathcal{L}_{Adv}$
$\displaystyle=\texttt{MSE}(\texttt{Adv}_{E}(f_{M}),Y_{E})+\texttt{CE}(\texttt{Adv}_{M}(f_{E}),Y_{M}),$
$\displaystyle\mathcal{L}_{Rec}$ $\displaystyle=\texttt{MSE}(X,\widehat{X}).$
$\displaystyle\mathop{min}\limits_{\theta_{Enc},\theta_{Pre},\theta_{Dec}}\mathcal{L}_{1}$
$\displaystyle=\mathcal{L}_{Pre}-\lambda*\mathcal{L}_{Adv}+\mu*\mathcal{L}_{Rec},$
$\displaystyle\mathop{min}\limits_{\theta_{Adv}}\mathcal{L}_{2}$
$\displaystyle=\mathcal{L}_{Adv}.$ (2)

We choose GRU [7] network for our encoder and decoder. The last output of GRU
encoder will feed into a MLP layer for the disentangled features. Similarly,
the decoder takes the excess and market features as input and outputs the
recovered $\widehat{X}$ through another MLP layer.

###  2.3 Dynamic Self-Distillation

Though the market and excess factors can be separated by return predictors and
adversarial predictors, there may still exist unrelated factors which
influence the prediction performance. To further remove the interferential
features and enhance the representation, we incorporate a self-distillation
[36, 17] method to distill the disentangled feature information from the
teacher model to student model, where the teacher and student models are the
same disentanglement networks in our work.

Different from the conventional distillation, we introduce a dynamic self-
distillation method, where the knowledge weight for a sample is determined by
its own performance and the corresponding day performance from the teacher
model. The inspiration is to dynamically control the knowledge importance for
different samples. For example, if the teacher performance is not good on a
day, we should increase the weight for all samples on that day. Besides, if
one sample’s performance is not good, we increase the weight for that sample.
Otherwise, we reduce the weight of these samples.

To measure excess return performance, Information Coefficient (IC) is used for
day performance and Mean Squared Error (MSE) loss for each sample. Let
$ic_{max}$ and $ic_{min}$ be the maximum and minimum IC values among all days,
$mse_{max}$ and $mse_{min}$ be the maximum and minimum MSE values among all
samples. $ic^{j}$ is the IC for day $d_{j}$ and $mse_{i}^{j}$ is the MSE for
stock sample $s_{i}$ on day $d_{j}$. We define two biases $\beta_{day}$ and
$\beta_{sample}$, which are used to ensure the importance of model knowledge.
$\alpha$ is used for leveraging the influence of day and sample information.
The final knowledge weight $w_{i}^{j}$ for stock sample $s_{i}$ on day $d_{j}$
is formulated by the daily weight $wd^{j}$ and the sample weight $ws_{i}^{j}$
as follows:

$\displaystyle wd^{j}$
$\displaystyle=\beta_{day}+(1-\beta_{day})*\frac{ic_{max}-ic^{j}}{ic_{max}-ic_{min}},$
$\displaystyle ws_{i}^{j}$
$\displaystyle=\beta_{sample}+(1-\beta_{sample})*\frac{mse_{max}-mse_{i}^{j}}{mse_{max}-mse_{min}},$
$\displaystyle w_{i}^{j}$ $\displaystyle=\alpha*wd^{j}+(1-\alpha)*ws_{i}^{j}.$
(3)

In this way, we can build the self-distillation loss for encoders between
teacher and student models. Let $ht_{i}^{j}$ and $hs_{i}^{j}$ represent the
last step output of teacher model and student model respectively. The
distillation loss is:

$\displaystyle\mathcal{L}_{Dis}=\sum_{i}\sum_{j}w_{i}^{j}*\texttt{MSE}(ht_{i}^{j},hs_{i}^{j}).$
(4)

The distillation for market encoder is same as excess encoder after replacing
with the classification measure in Eqn. (2.3).

Define the parameter $\xi$ to control the weight of the distillation loss.
Combined with previous disentanglement training, the overall training
objectives are:

$\displaystyle\mathop{min}\limits_{\theta_{Enc},\theta_{Pre},\theta_{Dec}}\mathcal{L}_{1}$
$\displaystyle=\mathcal{L}_{Pre}-\lambda*\mathcal{L}_{Adv}+\mu*\mathcal{L}_{Rec}+\xi*\mathcal{L}_{Dis},$
$\displaystyle\mathop{min}\limits_{\theta_{Adv}}\mathcal{L}_{2}$
$\displaystyle=\mathcal{L}_{Adv}.$ (5)

###  2.4 Data Augmentation

Figure 2: The distribution of stock returns. As an example, to augment the
training data samples, we choose the samples in day 1 and day 2 which have
close market returns.

To further improve the model generalization, we propose a novel data
augmentation method to extend the stock training data, where the new data
samples are generated by the decoder with disentangled features from different
samples.

For a sample $p=\\{x^{p},y_{e}^{p},y_{m}^{p}$} and another one
$q=\\{x^{q},y_{e}^{q},y_{m}^{q}$}, we obtain a new sample based on the excess
feature $f_{E}^{p}$ from $p$ and the market feature $f_{M}^{q}$ from $q$.
$f_{E}^{p}$ and $f_{M}^{q}$ are then concatenated and fed into the decoder to
generate $\widehat{x}$. Therefore, the new sample is augmented as
{$\widehat{x},y_{e}^{p},y_{m}^{q}$}. Similarly, we can use the excess feature
$f_{E}^{q}$ from $q$ and the market feature $f_{M}^{p}$ from $p$ to generate
another new sample.

Now the question is how to choose samples $p$ and $q$. In principle, the
disentangled features should be taken from the two days that the market
returns are close. As shown in Figure 2, the distributions of excess and
market returns are close to each other. Therefore, if we choose from the days
that market returns are close, the augmented samples should also be close to
the original data. In addition, we pay more attention to the days that perform
not good enough (the days that IC measurement is low), so to improve the
performance of these rare and hard samples.

##  3 Training Algorithm

Algorithm 1 ADD Algorithm

1:Disentanglement model $M$, training dataset $D$

2:Train model $M$ on data $D$ by losses Eqn. (2.2)

3:while $M$ not converge do

4: Set teacher model $M_{t}=M$

5: Generate new samples dataset $\widehat{D}$ (_Section 2.4_)

6: Augment the training data $D=D\cup\widehat{D}$

7: Train new model $M$ on data $D$ by losses Eqn. (2.3) with knowledge
distilled from $M_{t}$ (_Section 2.3_)

8:end while

9:Return model $M$

The proposed Augmented Disentanglement Distillation (ADD) method can be seen
in Algorithm 1.

At first, we train the disentanglement model with excess and market return
predictors, and the adversarial predictors. The trained disentanglement model
will serve as the teacher model later. After that, we dynamically augment new
samples (Section 2.4) and take augmented ones together with raw samples as the
new training dataset, then we distill the knowledge from the teacher model
(Section 2.3) and train a new student model for return predictions. This
student model will server as teacher model in the next round, and this
training procedure can be iteratively performed until convergence.

##  4 Experiment

We conduct experiments on the Chinese stock market for both stock return
predictions and the trading strategy backtesting. The return prediction task
can reflect the predicting ability of our method while the backtesting result
can demonstrate the actual effect in the real investment application.

###  4.1 Data

The data source is China A-shares stock market, which is one of the fastest
growing stock markets in the world. We obtained $13$ years of stock data from
$2007$ to $2019$, with a total of $7,836,559$ samples. The data is split as
follows: the training set consists of $3,078,000$ data samples from $2007$ to
$2013$, the validation set contains $1,114,234$ samples from $2014$ to $2015$
and $3,062,042$ data samples for the test set from $2016$ to $2019$. We take
three years of test data, which is enough to verify the stability of the
method in time.

Each sample contains $360$ ($6\times 60$) dimensional volume and price
features, and the labels of the excess and the market returns. The format of
the data sample is introduced as in Section 2.1. The features are composed of
$6$ factors that reflect the important direction of the stocks, namely opening
price, closing price, highest price, lowest price, volume and Volume Weighted
Average Price (VWAP). Each factor takes the corresponding values of the past
$60$ days. Therefore, the features contain valuable time series information.

###  4.2 Setting

For the disentanglement model, we adopt the one-layer GRU [7] for the encoder
and decoder networks with hidden size $64$. For the four predictors, we
leverage the $2$-layer stacked MLP network with batch normalization [16], the
activation function is tanh. As introduced, the market return prediction is a
classification task while the excess return prediction is a regression task.
Hence, one three class softmax layer is used for market return predictors,
while one unit regression for excess return predictors. The regression loss is
MSE for excess returns and the classification loss is cross entropy for market
returns. We use Adam [18] optimizer with the learning rate $0.001$ for
training. Dropout [29] is used for the decoder output with value $0.5$, the
batch size is $5,000$.

Hyper | Represent | Value
---|---|---
$\lambda$ | weight of adversarial loss | $0.4$
$\mu$ | weight of reconstruction loss | $0.05$
$\xi$ | weight of distillation loss | $0.8$
$\beta_{day}$ | bias of daily weight | $0.5$
$\beta_{sample}$ | bias of sample weight | $0.2$
$\alpha$ | influence of daily and sample weight | $0.4$

Table 1: Hyper-parameter settings used in our ADD method.

For the hyper-parameters, we report their settings and the descriptions in
Table 1. Note that we also show the impact of these hyper-parameters in
Section 4.8.

###  4.3 Evaluation Metrics

In quantitative investment field, there are some unique indicators for
evaluating the performance of stock prediction. Concretely, for excess
returns, people often take Information Coefficient (IC) and Rank IC [22] as
evaluation metrics. IC stands for the Pearson Correlation Coefficient of the
predicted sequence and the labeled sequence, while Rank IC is the Spearman’s
Rank Correlation Coefficient of two sequences. Both IC and Rank IC can show
how closely the stock predictions match the stock results.

In addition, the classification performance of market return within each
category is also considered. We take macro F1 score, which is the harmonic
mean of the precision and recall, as another metric to evaluate the
performance in each class.

###  4.4 Compared Method

To demonstrate the effectiveness of our method, we make comparison with
previous baseline methods, which include:

Long Short-Term Memory (LSTM): As an effective model used for time series
data, LSTM [14] has been widely utilized in stock prediction [6, 25] that
achieves great performance.

Gated Recurrent Unit (GRU): GRU [7] is another version of recurrent neural
network, but has fewer parameters than LSTM. It has also been widely used for
stock prediction [24, 33]. This GRU baseline is same as the encoder network in
our framework, but our method incorporates other components and training
strategies instead of directly predicting the returns.

The two-step disentanglement method: [12] present a two-step method for return
predictions. It was the first time that the disentanglement model was
introduced to stock prediction, which also separates the market and the excess
information. However, their model structure and training method are quite
different from ours.

Method | Rank IC | IC
---|---|---
Pre_E$\uparrow$ | Adv_E$\downarrow$ | Pre_E$\uparrow$ | Adv_E$\downarrow$
LSTM | $0.1348$ | $-$ | $0.1350$ | $-$
GRU | $0.1353$ | $-$ | $0.1343$ | $-$
Two-step | $0.1375$ | $0.0627$ | $0.1362$ | $0.0619$
ADD (Ours) | $\bf{0.1480}$ | $\bf{0.0306}$ | $\bf{0.1435}$ | $\bf{0.0275}$
$-$Distill and Augment | $0.1396$ | $0.0354$ | $0.1380$ | $0.0353$
$-$Distill | $0.1409$ | $0.0352$ | $0.1391$ | $0.0348$
$+$Static Distill | $0.1437$ | $0.0339$ | $0.1408$ | $0.0323$
$-$Augment | $0.1460$ | $0.0307$ | $0.1419$ | $0.0279$
$+$Noise Augment | $0.1468$ | $0.0310$ | $0.1424$ | $0.0277$
Table 2: Performances of excess return predictor $\texttt{Pre}_{E}$ by excess
encoder $\texttt{Enc}_{E}$ and adversarial excess return predictor
$\texttt{Adv}_{E}$ by market encoder $\texttt{Enc}_{M}$.

###  4.5 Prediction Performance

We report the return prediction performances on test set for our approach and
baseline methods in this subsection.

First, the performances of excess return predictor PreE and adversarial
predictor AdvE are shown in Table 2. For LSTM and GRU baselines, they are only
used to predict the excess returns. As shown, our ADD method effectively
facilitates the ability of predicting excess return with remarkable promotion
of Rank IC and IC (e.g., $0.1480$ v.s. $0.1375$ Rank IC). For adversarial
prediction, our method successfully removes most of the excess information
from the market encoder EncM, which results a significant decline in the
excess return prediction. For example, the Rank IC for ours is $0.0306$ while
$0.0627$ of two-step work [12].

Second, the market return prediction of PreM and adversarial prediction AdvM
is shown in Table 3. Again, we show strong improvements of the accuracy and F1
score over the baseline methods. As expected, the adversarial predictor
achieves lower accuracy and F1 score than the two-step method [12], which
means the market feature has been removed more from the excess encoder EncE.

In a short summary, our ADD approach achieves satisfactory results on the
excess and market return predictions, and surpasses the previous works in a
large margin.

Method | Accuracy | F1 score
---|---|---
Pre${}_{M}\uparrow$ | Adv${}_{M}\downarrow$ | Pre${}_{M}\uparrow$ | Adv${}_{M}\downarrow$
LSTM | $42.05\%$ | $-$ | $0.4172$ | $-$
GRU | $42.56\%$ | $-$ | $0.4231$ | $-$
Two-step | $42.67\%$ | $35.49\%$ | $0.4235$ | $0.3543$
ADD (Ours) | $\bf{46.35\%}$ | $\bf{34.36\%}$ | $\bf{0.4628}$ | $\bf{0.3435}$
$-$Distill and Augment | $44.51\%$ | $35.18\%$ | $0.4491$ | $0.3520$
$-$Distill | $44.62\%$ | $35.07\%$ | $0.4495$ | $0.3518$
$+$Static Distill | $45.13\%$ | $34.87\%$ | $0.4510$ | $0.3501$
$-$Augment | $46.05\%$ | $34.46\%$ | $0.4602$ | $0.3439$
$+$Noise Augment | $46.14\%$ | $34.50\%$ | $0.4604$ | $0.3441$
Table 3: Performances of market return predictor PreM by market encoder EncM
and adversarial market return predictor AdvM by excess encoder EncE.

###  4.6 Ablation Studies

As introduced, our approach consists of a disentanglement framework, a dynamic
self-distillation and a novel data augmentation strategy. In order to reflect
the importance of each component, we conduct ablation studies in this section.
The detailed settings are as follows:

1\. ADD without distillation and augmentation: We remove the distillation and
augmentation components in ADD, only remain the disentanglement model.
Compared with [12], this setting can illustrate the advantage of our structure
for disentanglement modeling.

2\. ADD without distillation: We perform ADD method without distillation, so
as to identify the value of training with self-distillation.

3\. ADD with static distillation: We replace the dynamic self-distillation
part in our ADD method with a static distillation, which directly transfers
the knowledge from teacher to student model for each sample, without
considering any day performance or controlled weights.

4\. ADD without augmentation: We remove the augmentation part in ADD method,
which can verify the contribution of our novel augmentation strategy.

5\. ADD with noise augmentation: We replace the augmentation part with a
simple Gaussian noise $N(0,0.25)$ to the original data samples as new ones, to
prove our disentanglement based augmentation approach is effective.

The results for above ablation studies are presented in the last several lines
of Table 2 and Table 3. From these numbers, we can observe several findings:

  * •

Our _disentanglement_ network design is more suitable for stock trend
prediction (two-step method v.s. setting 1).

  * •

Both distillation and augmentation benefit the return predictions (ADD v.s.
setting 1).

  * •

Our novel augmentation method based on disentangled features is more effective
than the simple noise augmentation (setting 4 and 5).

  * •

Our distillation part contributes more than the augmentation in the ADD
framework (setting 2 and 4).

  * •

The _dynamic_ self-distillation is sorely important than the static
distillation (setting 2 and 3).

###  4.7 Impact of Augmentation Data

To give a more detailed analysis about data augmentation strategy, we
investigate the impact from the different number of the augmented samples to
the performance. The results on valid set are clarified in Table 4.

As expected, as we increase the number of augmented samples, the model
performance is enhanced until reaching a threshold, then the performance
starts to drop. Therefore, we use $1.0$ million augmented samples in our
experiment.

#Augment | Rank IC | IC | Accuracy | F1 score
---|---|---|---|---
$0$ | $0.1701$ | $0.1670$ | $47.40\%$ | $0.4735$
$250,000$ | $0.1703$ | $0.1674$ | $47.43\%$ | $0.4736$
$500,000$ | $0.1711$ | $0.1680$ | $47.55\%$ | $0.4749$
$\bf{1,000,000}$ | $\bf{0.1720}$ | $\bf{0.1691}$ | $\bf{47.69\%}$ | $\bf{0.4758}$
$1,500,000$ | $0.1717$ | $0.1690$ | $47.60\%$ | $0.4753$
Table 4: Impact of the different number of augmented data samples on valid
set.

###  4.8 Impact of the Hyper-parameters

For the sake of studying the hyper-parameters, we perform experiments with
multiple values of the hyper-parameters. We investigate the weight $\lambda$
of adversarial loss, $\xi$ of distillation loss and $\alpha$ of knowledge
weight, others differ little in our preliminary attempts. Specifically, we
vary $\xi$ in $[0.6,0.8,1.0]$, $\alpha$ in $[0.0,0.2,0.4,0.6,0.8,1.0]$ and
$\lambda$ in $[0.2,0.4,0.6]$ and show the varied Rank IC performances.

The influenced results on valid set are shown in Figure 3. It is obvious that
these hyper-parameters have a huge impact to the prediction performances. From
these searched results, $\xi=0.8$, $\alpha=0.4$ and $\lambda=0.4$ make a best
balance and contribution towards the final performance. Note that $\alpha=0.0$
and $\alpha=1.0$ mean that we use a single property instead of the combination
of daily performance and sample performance, this result also demonstrates the
necessary of the balanced combination process.

Figure 3: The performance for different hyper-parameter settings on validation
set.

###  4.9 Improvements for Rare Sample Performance

Among all the samples, we pay more attention to those hard and rare examples.
To evaluate the performance on rare samples, we first separate the test set
into three subsets according to the performance of baseline model.
Specifically, take the Rank IC for excess return and the probability value of
the correct class for market return from baseline model as the division
indicators, the test set is equally divided into three subsets: low set,
middle set, and high set, with the same number of samples in each subset.
Samples in the low set refer to the data that performs worst among the whole
data set, while samples in the high set achieve the best performance. After
splitting the test set, we then compare the performance of the baseline model
and our method on each subset, and calculate the ratio of samples that ADD
achieves better in each subset. The result is given in Table 5.

Test Subset | Ratio of Samples
---|---
Excess | Market
Low Set | $78.77\%$ | $62.46\%$
Middle Set | $70.46\%$ | $56.62\%$
High Set | $61.23\%$ | $52.91\%$
Table 5: The ratio of data samples that our ADD method achieves better on the
separated three subsets.

We can see our ADD achieves better than the baseline model among all subsets
with a large number of samples, especially on the low set (e.g., $78.77\%$
samples for excess return). Compared with middle set and high set, there are
more samples achieving better performance in the low set than other two
subsets (e.g., $78.77\%$ on low set v.s. $61.23\%$ on high set for excess
return). These results demonstrate our ADD indeed cares more about the rare
samples and improves their performances significantly.

###  4.10 Backtesting

Figure 4: The accumulated returns over time in backtesting experiments.

For verifying the role of our method in actual investment, we need to assess
the viability of a trading strategy based on our prediction by discovering how
it play out using real-world data, which is the idea of backtesting.

A widely-acknowledged method for backtesting is to pick the top-$k$ stocks
ranked by the excess return prediction to form up the portfolio for the next
trading day. In order to reduce the investment risk, traders usually
distribute the investing equally over these top-$k$ stocks. Thus, $k$ is a
hyper-parameter to balance between the robustness and profits of the
portfolio. In our experiments, we set up $4$ sets of comparative experiments
with $k$ to be $30$, $50$, $100$ and $200$ respectively. To further
approximate the real-world trading, we consider a transaction cost of $0.4\%$
for both buy and sell transactions. Actual accumulated returns are used to
evaluate the trading policies under these different strategies.

The backtesting results are given in Figure 4. Under different values of $k$,
our ADD method achieves higher accumulated income than the baseline models
(GRU baseline models). Over time, the income gap between the baseline models
and our ADD is increasing gradually, indicating that our method is stable and
appropriate along time. Our method has obvious advantages, no matter when the
prediction effect is better or when the performance is not that good. Through
backtesting, we verify that ADD leads to a promotion in the actual investment
that can have real practical value.

##  5 Related Work

###  5.1 Disentanglement Representation

Disentanglement representation extraction aims to figure out explanatory
factors of the input data to generate more meaningful features. Early attempts
including bilinear model [30], autoencoder [11], manifold learning [8],
Restricted Boltzmann Machine [26] and so on. Recently, Variational Autoencoder
(VAE) and Generative Adversarial Networks (GANs) are found to be more
powerful. Besides, researchers are also interested in explorations about the
ability of disentanglement representation to solve challenge tasks [2, 31,
21].

The disentanglement method is first introduced to stock prediction in [12],
which proposes a two-step disentanglement method by training the two encoders
separately. Unlike its asynchronous disentanglement, our disentanglement
process is fully end-to-end and trained synchronously. Besides, our novel
framework and training phases differ from their method a lot.

###  5.2 Knowledge Distillation

Knowledge distillation usually utilizes a teacher-student strategy, which
targets at transferring the knowledge from a strong teacher model to a student
model [4, 1, 13]. Previous purpose of knowledge distillation lies on model
compression, which intends to reduce the large model space of a teacher model
to a small student model while trying best to keep the performance. Techniques
include soft label matching [13], adding random perturbations into soft label
[28], hidden layer approximating [27] and attention weight mapping [35].

The performance of student model is always worse than the teacher model, self-
distillation is proposed to achieve comparable or even better performance than
teacher model, which distills the knowledge between teacher and student models
in an identical architecture [34, 10, 36, 17]. We adopt self-distillation but
differ from them in two key points. First, we propose a dynamic self-
distillation process by introducing some weights to leverage the knowledge
from teacher model and the information in raw data. Second, our distillation
method is performed with disentanglement network, which not only improves the
model performance, but also assists in eliminating interference factors.

##  6 Conclusion

Stock trend forecasting plays a more attractive role nowadays with the help of
deep learning approaches. In this work, we propose an Augmented
Disentanglement Distillation (ADD) approach to remove the inferential features
from the noised raw stock data, so as to extract clean information and improve
the market/excess return prediction performances. Our ADD contains a
disentanglement structure, a dynamic self-distillation method, as well as a
novel data augmentation strategy. Experiments on the Chinese stock market data
demonstrate the effectiveness of our approach for stock trend forecasting.
Also, the backtesting experiments also verify that ADD has real impact in the
actual investment to help traders increase income. For future works, we are
interested in simplifying the training strategy as well as designing a more
effective disentanglement structure to improve stock trend forecasting.

## References

  * Ba & Caruana [2014] Jimmy Ba and Rich Caruana.  Do deep nets really need to be deep?  In _Advances in neural information processing systems_ , pp. 2654–2662, 2014.
  * Bengio et al. [2013] Yoshua Bengio, Aaron Courville, and Pascal Vincent.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ , 35(8):1798–1828, 2013.
  * Bisoi & Dash [2014] Ranjeeta Bisoi and Pradipta K Dash.  A hybrid evolutionary dynamic neural network for stock market trend analysis and prediction using unscented kalman filter.  _Applied Soft Computing_ , 19:41–56, 2014.
  * Bucilua et al. [2006] C Bucilua, R Caruana, and A Niculescu-Mizil.  Model compression, in proceedings of the 12 th acm sigkdd international conference on knowledge discovery and data mining, 2006.
  * Chen et al. [2019] Chi Chen, Li Zhao, Jiang Bian, Chunxiao Xing, and Tie-Yan Liu.  Investment behaviors can tell what inside: Exploring stock intrinsic properties for stock trend prediction.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 2376–2384, 2019.
  * Chen et al. [2015] Kai Chen, Yi Zhou, and Fangyan Dai.  A lstm-based method for stock returns prediction: A case study of china stock market.  In _2015 IEEE international conference on big data (big data)_ , pp. 2823–2824. IEEE, 2015.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ , 2014.
  * Elgammal & Lee [2004] Ahmed Elgammal and Chan-Su Lee.  Separating style and content on a nonlinear manifold.  In _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , volume 1, pp. I–I. IEEE, 2004.
  * Feng et al. [2018] Fuli Feng, Huimin Chen, Xiangnan He, Ji Ding, Maosong Sun, and Tat-Seng Chua.  Enhancing stock movement prediction with adversarial training.  _arXiv preprint arXiv:1810.09936_ , 2018.
  * Furlanello et al. [2018] Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.  Born again neural networks.  _arXiv preprint arXiv:1805.04770_ , 2018.
  * Georghiades et al. [2001] Athinodoros S. Georghiades, Peter N. Belhumeur, and David J. Kriegman.  From few to many: Illumination cone models for face recognition under variable lighting and pose.  _IEEE transactions on pattern analysis and machine intelligence_ , 23(6):643–660, 2001.
  * Hadad et al. [2018] Naama Hadad, Lior Wolf, and Moni Shahar.  A two-step disentanglement method.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 772–780, 2018.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_ , 2015.
  * Hochreiter & Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Hu et al. [2018] Ziniu Hu, Weiqing Liu, Jiang Bian, Xuanzhe Liu, and Tie-Yan Liu.  Listening to chaotic whispers: A deep learning framework for news-oriented stock trend prediction.  In _Proceedings of the eleventh ACM international conference on web search and data mining_ , pp. 261–269, 2018.
  * Ioffe & Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  _arXiv preprint arXiv:1502.03167_ , 2015.
  * Kim et al. [2020] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang.  Self-knowledge distillation: A simple way for better generalization.  _arXiv preprint arXiv:2006.12000_ , 2020.
  * Kingma & Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Li et al. [2019] Chang Li, Dongjin Song, and Dacheng Tao.  Multi-task recurrent neural networks and higher-order markov random fields for stock price movement prediction: Multi-task rnn and higer-order mrfs for stock price classification.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1141–1151, 2019.
  * Li et al. [2016] Lili Li, Shan Leng, Jun Yang, and Mei Yu.  Stock market autoregressive dynamics: A multinational comparative study with quantile regression.  _Mathematical Problems in Engineering_ , 2016, 2016.
  * Locatello et al. [2019] Francesco Locatello, Gabriele Abbati, Thomas Rainforth, Stefan Bauer, Bernhard Schölkopf, and Olivier Bachem.  On the fairness of disentangled representations.  In _Advances in Neural Information Processing Systems_ , pp. 14611–14624, 2019.
  * McWilliams & Siegel [1997] Abagail McWilliams and Donald Siegel.  Event studies in management research: Theoretical and empirical issues.  _Academy of management journal_ , 40(3):626–657, 1997.
  * Metghalchi et al. [2012] Massoud Metghalchi, Juri Marcucci, and Yung-Ho Chang.  Are moving average trading rules profitable? evidence from the european stock markets.  _Applied Economics_ , 44(12):1539–1559, 2012\.
  * Minh et al. [2018] Dang Lien Minh, Abolghasem Sadeghi-Niaraki, Huynh Duc Huy, Kyungbok Min, and Hyeonjoon Moon.  Deep learning approach for short-term stock trends prediction based on two-stream gated recurrent unit network.  _Ieee Access_ , 6:55392–55404, 2018.
  * Nelson et al. [2017] David MQ Nelson, Adriano CM Pereira, and Renato A de Oliveira.  Stock market’s price movement prediction with lstm neural networks.  In _2017 International joint conference on neural networks (IJCNN)_ , pp. 1419–1426. IEEE, 2017.
  * Reed et al. [2014] Scott Reed, Kihyuk Sohn, Yuting Zhang, and Honglak Lee.  Learning to disentangle factors of variation with manifold interaction.  In _International Conference on Machine Learning_ , pp. 1431–1439, 2014.
  * Romero et al. [2014] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  _arXiv preprint arXiv:1412.6550_ , 2014.
  * Sau & Balasubramanian [2016] Bharat Bhusan Sau and Vineeth N Balasubramanian.  Deep model compression: Distilling knowledge from noisy teachers.  _arXiv preprint arXiv:1610.09650_ , 2016.
  * Srivastava et al. [2014] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958, 2014.
  * Tenenbaum & Freeman [2000] Joshua B Tenenbaum and William T Freeman.  Separating style and content with bilinear models.  _Neural computation_ , 12(6):1247–1283, 2000\.
  * van Steenkiste et al. [2019] Sjoerd van Steenkiste, Francesco Locatello, Jürgen Schmidhuber, and Olivier Bachem.  Are disentangled representations helpful for abstract visual reasoning?  In _Advances in Neural Information Processing Systems_ , pp. 14245–14258, 2019.
  * Wei [2016] Liang-Ying Wei.  A hybrid anfis model based on empirical mode decomposition for stock time series forecasting.  _Applied Soft Computing_ , 42:368–376, 2016.
  * Xu & Cohen [2018] Yumo Xu and Shay B Cohen.  Stock movement prediction from tweets and historical prices.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 1970–1979, 2018\.
  * Yim et al. [2017] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 4133–4141, 2017.
  * Zagoruyko & Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer.  _arXiv preprint arXiv:1612.03928_ , 2016.
  * Zhang et al. [2019] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pp. 3713–3722, 2019.