monthyeardate[]

# Deep Reinforcement Learning in Quantitative Algorithmic Trading: A Review

Tidor-Vlad Pricope
The University of Edinburgh
Informatics Forum, Edinburgh, UK, EH8 9AB
<EMAIL_ADDRESS>

###### Abstract

Algorithmic stock trading has become a staple in today’s financial market, the
majority of trades being now fully automated. Deep Reinforcement Learning
(DRL) agents proved to be to a force to be reckon with in many complex games
like Chess and Go. We can look at the stock market historical price series and
movements as a complex imperfect information environment in which we try to
maximize return - profit and minimize risk. This paper reviews the progress
made so far with deep reinforcement learning in the subdomain of AI in
finance, more precisely, automated low-frequency quantitative stock trading.
Many of the reviewed studies had only proof-of-concept ideals with experiments
conducted in unrealistic settings and no real-time trading applications. For
the majority of the works, despite all showing statistically significant
improvements in performance compared to established baseline strategies, no
decent profitability level was obtained. Furthermore, there is a lack of
experimental testing in real-time, online trading platforms and a lack of
meaningful comparisons between agents built on different types of DRL or human
traders. We conclude that DRL in stock trading has showed huge applicability
potential rivalling professional traders under strong assumptions, but the
research is still in the very early stages of development.

##  1 Introduction

Stock trading is the buying and selling of a company’s shares within the
financial market. The goal is to optimize the return on the investment of the
capital exploiting the volatility of the market through repetitive buy and
sell orders. Profit is generated when one buys at a lower price than it sells
on afterwards. Among the key challenges in the finance world, the increasing
complexity and the dynamical property of the stock markets are the most
notorious. Stiff trading strategies designed by experts in the field often
fail to achieve profitable returns in all market conditions [1]. To address
this challenge, algortihmic trading strategies with DRL are proposed.

Although automated low-frequency quantitative stock trading may seem like a
fancy choice of words, it means something really simple. Automated refers to
the fact that we are targeting algorithmic solutions, with very little to no
human intervention. Quantitative stock trading refers to the fact that we want
the trading agent developed to be able to take its decisions based on
historical data, on quantitative analysis, that is based on statistical
indicators and heuristics to dechiper patterns in the market and take
appropiate actions [1]. Finally, low-frequency trading assures that the
solutions that we seek resemble agents which can trade like a human being from
1 minute to a few days’ timeframe. This choice is natural and does not narrow
the literature down that much, as most high-frequency trading systems act at
the level of pico-second or milli-seconds and the studies in this area are
usually owned by high-profile companies which try to keep their research
hidden. The hardware for such studies is also really limited and represents a
reason for such a shortage of studies in high-frequency trading because it
does not require only a few very good GPUs but also really small latency
devices to connect to live datacenters that contain active stock prices.

For the scope of this paper, there are further limitations that should be
implied by the title and the abstract but nevertheless, they will be mentioned
here. Our attention is focused on studies building an end-to-end trading agent
using DRL as a core learning paradigm. We won’t discuss papers debating on
forecasting stock prices or the direction of the market with deep learning.
Although it is clear that one can develop a good trading agent if such a
system would be successful, usually, stock prices are very volatile and can’t
be predicted in such ways [2] leaving the job to a risk management system to
do the work. From the same reason, we won’t mention papers that try to combine
meta-heuristic algorithms (like Genetic Algorithms - GA [3] or Particle Swarm
Optimization - PSO [4]) with Reinforcement learning (RL) or multi-agent
solutions to this problem. We also rule out the portfolio management part [5]
because it is entirely a different system that should be taken into
consideration separately when constructing a trading engine [6, 7]. We focus
on the actual agent - learner that makes the trades, the allocation of funds
is not a priority to review in this paper.

It is no secret that the main theme of this paper is the applicability of AI
in finance, further narrowed down by multiple research questions. As we will
see, most of the work in this area is fairly new. The literature can answer
the question quite well, as several benchmarks have been a staple for quite
some time in algorithmic trading, like yearly return rate on a certain
timeframe or cumulative wealth [7] or Sharpe ratio [8]. An issue that is
usually problematic here relates to performance comparisons that can be
inconclusive. Obviously, some papers test their proposed approaches on
different environments and different timelines and may use a different metric
for assessing performance, but a common ground can be found if similar
settings are used. Moreover, it is not futile to assume that many studies from
the literature are kept private in this area, as it is a way to generate a lot
of profit, therefore, state-of-the-art methods can hardly be called out;
that’s why we are going to divide this review paper according to the main idea
of the methods used to solve the problem: critic-only RL, actor-only RL and
actor-critic RL approaches, which is actually a well-defined categorization of
RL in trading [9].

There is an obvious question that this paper tries to answer, that being: can
we make a successful trading agent which plays on the same timeframes as a
regular human trader using the RL paradigm making use of deep neural networks?
We explore solutions presenting multiple papers in the current literature that
propose approaches to the problem and actual concrete agents that we will be
able to assess in order to define the progression on this field of study. We
conclude that DRL has huge potential in being applying in algorithmic trading
from minute to daily timeframes, several prototype systems being developed
showing great success in particular markets. A secondary question that arises
naturally from this topic is: can DRL lead to a super-human agent that can
beat the market and outperform professional human traders? The motivation for
considering such a question is that, recently, it was showed that RL systems
can outperform experts in conducting optimal control policies [10], [11] or
can outperform even the best human individuals in complex games [12]. We shall
also try to respond to this question to some extent; we provide a partial
answer supported by particular studies [13] where there have been DRL agents
developed for real-time trading or for realistic back-testing [14] that can,
in theory, allow for such a comparison. We infer that DRL systems can compete
with professional traders with respect to the risk-adjusted return rates on
short (15-minute) or long (daily) timeframes in particular hand-picked
markets, although further research on this needs to be done in order to
decipher the true extent of DRL power in this environment.

##  2 Background

In this section, we provide the necessary background for the reader to get a
grasp of some basic concepts we shall recall throughout this review paper.

###  2.1 Reinforcement Learning

Reinforcement learning [15] is considered to be the third paradigm of
learning, alongside supervised and unsupervised learning. In contrast with the
other two, reinforcement learning is used to maximize the expected reward when
in an environment usually modelled as a Markov Decision Process (MDP). To be
more precise, having as input a state representation of the environment and a
pool of actions that can be made, we reach the next state with a certain
numerical reward of applying a picked action. In order to approximate the
expected reward from a state (or the value of a state), an action-value
function $Q$ can be used. This takes as input a state and a possible action
and outputs the expected $Q$ value [16] \- the reward that we can expect from
that state forward - the cumulative future rewards. In the literature, the
action-value function is referred to as the critic, one might say it’s because
it ”critiques” the action made in a state with a numerical value. However, RL
can directly optimize the policy that the agent needs to take in order to
maximize reward. If we view the policy as a probability distribution over the
whole action space, then policy gradient (PG) methods [17] can be applied to
optimize that distribution so that the agent chooses the action with the
highest probability, amongst all possible actions, that gives the highest
reward. Judging by the fact that this method directly controls to policy, in
the literature, it is referred as the actor. RL employs these two approaches
in the literature to form 3 main techniques: critic-only, actor-only, actor-
critic learning. In the critic-only, we use only the action-value function to
make decisions. We can take a greedy approach - always choosing the action
that gives the max possible future reward, or we can choose to explore more.
Note that the Q function can be approximated in many ways, but the most
popular and successful approach is with a neural network (Deep Q Network - DQN
[10] and its extensions [18]). In the actor only approach, we model only the
probability distribution over the state - the direct behaviour of the agent.
In actor-critic RL, the actor outputs an action (according to a probability
distribution) given a state and the critic (we can look at this as a feedback
that ”critiques” that action) outputs a $Q$ value (the future reward) of
applying the chosen action in that given state. Popular, state-of-the-art
algorithms like Advantage Actor Critic (A2C) [19, 20], Deep Deterministic
Policy Gradient (DDPG) [21, 22], Proximal Policy Optimization (PPO) [23, 24]
all follow this scheme.

###  2.2 Technical analysis in stock trading: stock market terminologies

The generally adopted stock market data is the sequence at regular intervals
of time such as the price open, close, high, low, and volume [1]. It’s a
difficult task, even for Deep Neural Networks (DNN), to get a solid grasp on
this data in this form as it holds a great level of noise, plus the non-
stationary trait. Technical indicators were introduced to summarize markets’
behaviour and make it easier to understand the patterns in it. They are
heuristics or mathematical calculations based on historical price, volume that
can provide (fig. 1) bullish signals (buy) or bearish signals (sell) all by
themselves (although not accurate all the time). The most popular technical
indicators that will be mentioned in the review as well are: Moving Average
Convergence Divergence (MACD) [25], Relative Strength Index (RSI) [26],
Average Directional Index (ADX) [27], Commodity Channel Index (CCI) [28], On-
Balance Volume (OBV) [29] and moving average and exponential moving average
[30]. There are many more indicators and the encyclopedia [31] explains nicely
each one of them.

Figure 1: Indicator showing buy and sell signals. Source: iqoption [32]

The challenge that comes with applying technical indicators in algorithmic
trading is the fact that there are so many of them and the majority do not
work in specific situations - providing false signals. Moreover, many are
correlated with each other - such dependencies need to be taken down as it can
affect the behaviour of a learning algorithm because of the overwhelming
obscure feature space. Solutions have been employed through deep learning -
performing dimensionality reduction (through an autoencoder [33]), feature
selection and extraction (through Convolutional Neural Networks - CNN [34] or
Long Short Term Memory Networks - LSTM [35]), hence eliminating some noise in
the raw market data.

These indicators alone can be a repertoire for a human trader and combinations
between them can represent by themselves winning strategies for some stocks.
That’s why there is actually a great amount of trading agents based on expert
systems rule-based with great success using only technical analysis. There are
also papers that present simple algorithms that are based only on data mining
and the use of one indicator that can achieve over 32% annual return [36].

##  3 Literature Review

In this section we review literature works regarding deep reinforcement
learning in trading. There are key points we follow in this review, tied to
the principal challenges that reinforcement learning faces when dealing with
financial trading: data quality and availability (high quality data might not
be free ot it might be limited for certain timeframes), the partial
observability of the environment (there will always be a degree of un-
observability in the financial market [14]) and the exploration/exploitation
dilemma [15] (a great deal of exploration is usually required for a successful
RL agent but this is not feasible in financial trading since random
exploration would inevitably generate huge amount of transaction costs and
loss of capital).

###  3.1 Critic-only Deep Reinforcement Learning

Overview

Before getting to review the studies in this category, it is important to
highlight the main limitations that such an approach suffers from. The critic
method (represented mainly by the DQN and its improved variations) is the most
published method in this area of research [19] (compared to the others: actor-
only and actor-critic). The main limitation is that a DQN aims to solve a
discrete action space problem; so, in order to transform it to a continuous
space (this can be needed as we will see in some reviewed papers), certain
tricks are to be applied. Moreover, a DQN works well for discrete state
representations, however, the prices of all the stocks are continuous-valued
entities. Applying this to multiple stocks and assets is also a huge
limitation as the state and action space grows exponentially [22]. Moreover,
the way that a reward function is defined in this context is crucial; critic-
only approaches being very sensible to the reward signals it receives from the
environment [1].

Related Works

Application of Deep Reinforcement Learning on Automated Stock Trading [37].
Inspired by the success of state-of-the-art Deep Q Networks in tackling games
like Atari [38] and imperfect information games like Poker [39], this paper
tries to apply the same idea to develop a trading agent considering the
trading environment as a game in which we try to maximize reward signaled by
profit. A variant of the DQN learning system is tested by the authors: Deep
Recurrent Q- Network (DRQN). The architecture is simple - they use a recurrent
neural network at the basis of DQN which can process temporal sequences better
and consider a second target network to further stabilize the process. The
authors use S&P500 ETF price history dataset to train and test the agent
against baseline benchmarks: buy and hold strategy – which is used by human
long-term investors and a random action-selected DQN trader. The data for
training and testing is obtained through Yahoo Finance and it contains 19
years of daily closing prices, the first 5 being used for training and the
rest for testing the agent. Therefore, a state representation is defined with
a sequence of adjusted close price data over a sliding window of 20 days.
Being a pure crittic approach, at any time, an agent has a discrete, finite
set of actions - it has to decide between 3 options: buy a share, sell a share
or do nothing. The reward is computed as the difference between the next day’s
adjusted close and the current day’s adjusted close price, depending on which
action was made.

As we can see in figure 2, the best agent, based on the DRQN system, is
successful and outperforms the baseline models which gives hope for
algorithmic trading. The annual expected return from these systems is around
22-23%, which is not bad, but it is far off from some of the other reviewed
agents in this paper that can get over 60% annual return [14].

Figure 2: Profit Curves of BH, DQN and DRQN on the SPY Test Dataset [37]

However, there are clear limitations on the study of this paper. First, only
one stock is considered as a benchmark. That is not necessarily wrong but it
does question the generality of the proposed approach to a larger portfolio,
even from our own experiments doing algorithmic trading, some stocks can be
very successful and some can lose all the allocated budget. Secondly, the
reward function is really weak, it does not address risk as the other reviewed
paper do, therefore it might be scary to let this model run on live data. A
quick fix would be to use the Sharpe ratio as a reward measurement. Third, the
agent is very constrained in what it can do - buying only one stock at a time
is very limiting, more actions could be added to allow for more shares to be
bought or sold. The feature representation is far from optimal, raw price data
contains a lot of noise - technical analysis features should have been used to
mitigate that. In terms of training/testing, it is concerning to us that the
training data is much smaller in size and it might not be representative
compared to the testing one - the 2006-2018 interval of time contains critical
market states like the 2008 financial crisis in which the agent probably won’t
know how to handle. Furthermore, very basic opponents have been chosen for
comparison purposes, even though the buy and hold strategy is objectively good
and can be really profitable in real life, being actually the main strategy
for investors, the paper lacks a direct comparison with other trading agents
from the literature on the same stock. Nevertheless, we would categorize this
paper as a solid pilot experiment into this wild field that can provide a
proof-of-concept for the utility of deep reinforcement learning in trading.

A strongly related work with this is the study Financial Trading as a Game: A
Deep Reinforcement Learning Approach [14] that also employs a DRQN-based
agent. However, there are key differences: it uses 16 features in total - raw
price data (open, high, low, close, volume (OHLCV)) plus indicators; a
substantially small replay memory is used during training; an action
augmentation technique is also introduced to mitigate the need for random
exploration by providing extra feedback signals for all actions to the agent.
This enables the use of greedy policies over the course of learning and shows
strong empirical performance compared to more commonly used $\epsilon$-greedy
exploration. Moreover, they sample a longer sequence for recurrent neural
network training. Limitations still exist: the reward function (defined here
as the portfolio log returns) is better as it normalizes the returns but still
does not specifically address risk and the action space size is still very
restricted (3 options).

The algorithm is applied on forex (FX) market from 2012 to 2017, 12 currency
pairs, 15-minute timeframe with realistic transaction costs. The results are
decent, the algorithm obtaining even 60% annual return on some currencies,
averaging about 10% with all the results being positive, which is encouraging
further research in this area. It is true that the authors haven’t tested this
algorithm in stock markets, only on FX, which would rule this paper out of
scope for our review. However, considering the fact that this paper presents a
better version of the previously-discussed algorithm, it is not wrong to
assume it may be applied with a decent amount of success to the stock market
as well. It’s interesting here to also address the research question regarding
the comparison with professional human traders. The environment for
backtesting happens to have realistic setting in this paper and yet, the best
result is a 60% annual return, which is high. Nevertheless, it is impossible
to just go with the numbers because we can’t really know what an average
professional trader or day-trader scores as profit rate in a year [40], but we
do know that 60% a year in profits is much more than 7% (the usual buy and
hold annual return rate on investment for a stable growing stock index like
S&P 500) and would quickly generate more and more money if continuous
reinvesting of funds is done and the algorithm is stable which is the main
goal of traders and investors. Therefore, from this perspective, we can assume
that, if consistent in trading, the algorithmic solution might be on par with
human traders that make a living out of this.

There are a few studies [1, 19] that employ critic-only deep reinforcement
learning methods to compare with the other ones. In Adaptive stock trading
strategies with deep reinforcement learning methods [1], Gated Recurrent Units
(GRUs) are introduced as a deep learning method to extract features from the
stock market automatically for its Deep Reinforcement Q-Learning system
(GDQN). The motivation for introducing this is the fact that stock market
movements cannot reveal the patterns or features behind the dynamic states of
the market. In order to achieve that, the authors present an architecture with
an update gate and reset gate function for the neural network. The reset gate
acts like a filter for the previous stock informationi, keeping only a portion
of it, and the update gate decides whether the hidden state will be updated
with the current information or not. This is a learnt way to tackle feature
engineering for this problem, similar approaches with autoencoder
architectures have been tried to achieve the same thing. To note, Dropout [41]
is used with GRU in the Q-network.

Figure 3: Gated Recurrent Unit (GRU) architecture [1]

Another contribution of this paper is the use of a new reward function that
better handles risk (something we repeatedly stated that it’s missing before):
the Sortino ratio (SR) [42]. The authors argue that this reward function would
better fair in a high-volatility market because it measures negative
volatility - it only factors in the negative deviation of a trading strategy’s
returns from the mean. As state representation, the usual OHLCV is used plus
some popular technical indicators like MACD, MA, EMA, OBV. The action space is
again restricted to 1 share per transaction, 3 options being available: buy,
sell or hold. Among the reviewed papers so far, this has the most promising
setup; the daily stock data used stretched from early 2008 to the end of 2018,
the first 8 years being used for training; it consisted of 15 US, UK and
Chinese stocks, each one of them tested individually. The best result of the
GDQN system was recorded on a Chinese stock: 171% return and 1.79 SR. There
was also recorded a SR of 3.49 and a return of 96.6% on another Chinese stock,
which is impressive. Just 2 out of 15 stocks were negative. Moreover, the
system wins the comparison with the baseline (Turtle strategy [43]) and it is
showed to achieve more stable returns than a state-of-the-art direct
reinforcement learning trading strategy [13]. GDQN is directly compared with
the actor-critic Gated Deterministic Policy Gradient (GDPG) also introduced in
this paper that we will review in the following sections. It is showed that
GDPG is more stable than GDQN and provides slightly better results. Our
criticism towards this study remains, like in the other ones, the limitation
in what the agent can do - with only 1 share per transaction at max; in
addition, transaction costs were not added in the equation, it would be
interesting to see how the return rates would change if they would be factored
in.

We will briefly mention Deep Reinforcement Learning for Trading [19] as well,
though their experiments focus on futures contracts and not stocks. It is very
similar with [37], implementing a LSTM based Q-network, but making use of some
technical indicators (RSI and MACD) to add to the feature space. As reward, it
is important to mention that volatility of the market is incorporated in order
to also consider the risk. This mechanism is used to scale up trade positions
while volatility is low and do the opposite, otherwise. The authors show that
show that the DQN system algorithms outperform baseline models (buy and hold,
Sign(R) [44, 45], MACD signal [25]) and deliver profits even under heavy
transaction costs. This paper also tests an actor-only algorithm (PG) and the
A2C algorithm to contrast with the baseline and with the DQN. We will get back
to this paper in the next sections.

As we mentioned in the first paragraph of this section, the critic-only
approach is the most published one in this field of financial trading; we had
to remove some (older) studies [46, 47, 48] because they were out of the scope
of this review paper - they weren’t using deep Q-learning, only Q-learning
with simple artificial neural networks (ANNs) or doing the feature extraction
beforehand through other methods and not deep learning.

###  3.2 Actor-only Deep Reinforcement Learning

Overview

With actor-only methods, because a policy is directly learnt, the action space
can be generalized to be continuous \- this is the main advantage of this
approach: it learns a direct mapping (not necessarily discrete) of what to do
in a particular state. A general concern we have discovered with this actor-
only approach is the longer time to train that it takes to learn optimal
policies, as also highlighted in [19]. This happens because, in trading,
learning needs a lot of samples to successfully train as otherwise, individual
bad actions will be considered good as long as the total rewards are great -
we don’t have a numerical value assigned to a particular state and action
(like in the critic-only approach) that can tell us different through an
expected outcome. However, this statement might be disputed [9], as in some
cases faster convergence of the learning process is possible.

Related Works

Deep Direct Reinforcement Learning for Financial Signal Representation and
Trading [13]. We briefly mentioned this study in the previous section (when
comparing to the GDQN) as a state-of-the-art direct reinforcement learning
trading strategy. This paper from 2017 specifically tries to answer the
question of beating experienced human traders with a deep reinforcement
learning based algorithmic-trader. This study is special because the authors
claim that (at the time) it represented the first use of Deep Learning for
real-time financial trading. Their proposed approach is a novel one. They
remodel the structure of a typical Recurrent Deep Neural Network for
simultaneous environment sensing and recurrent decision making that can work
in an online environment. Like in the first paper reviewed, they use a
Recurrent Neural Network for the RL part; however, they use Deep Learning for
feature extraction combined with fuzzy learning concepts [49, 50] to reduce
the uncertainty of the input data. They use fuzzy representations [51] (assign
fuzzy linguist values to the input data) as part of the data processing step;
after that, they use the deep learning for feature learning on the processed
data and this is fed to the RL agent to trade. Another novelty that the
authors introduce in this paper is a variant of Backpropagation through time
(BTT) called task-aware BPTT to better handle the vanishing gradient problem
in very deep architectures. To briefly explain this one, they use extra links
from the output to each hidden layer during the backpropagation making more
room for the gradient to flow.

What is important here to note is the fact that the proposed solution that not
use technical indicators, it utilizes the aforementioned system for raw data
processing and feature learning to make educated guesses through the RL system
to maximize profit and that is really impressive. we personally thought that
technical indicators would greatly improve a system’s performance if we aim
for the correct amount of un-correlated ones because of a simple reason: it
adds more insight to the dataset and more features that can unravel the hidden
patterns leading to a successful trading strategy. This does not mean that the
performance using only raw data and clever data processing would beat current
state-of-the-art models that make use of technical analysis but perhaps a
comparison between their proposed system using and not using technical
indicators would have been great to add. The experiments part is solid from a
comparison point of view between established strategies in the literature and
the proposed approaches; however, it lacks the testing on a diverse portfolio:
only 3 contracts are chosen (2 commodities – sugar and silver and one stock-
index future IF). It is actually addressed in the future work section that
this framework should be adapted in the future to account for portfolio
management. The comparisons (figure 4) on both the stock-index and commodity
future contracts show that the proposed Direct Deep Reinforcement (DDR) system
and its fuzzy extension (FDDR) generate significantly more total profit are
much robust to different market conditions than Buy and Hold, sparse coding-
inspired optimal training (SCOT) [52] and DDR system without fuzzy
representation for data processing. Moreover, on Sharpe ratio, the proposes
systems recorded values over 9, which is incredibly high.

Figure 4: Market behaviour (top), profit and loss (P&L) curves of different
trading systems with total profits (TP -middle) and moving Sharpe Ratio [53]
(SR - bottom) [13] Figure 5: S&P 500 stock index P%L curves on different
trading systems [13]

The authors also tested the algorithms on stock indexes - like S&P 500 (figure
5), where they can combine multi-market features to form a multi-FDDR agent.
The stock data stretched from 1990 to 2015, the first 8 years being used for
training and the transaction cost is set to 0.1% of the index value. We can
see (figure 5) that indeed the agents generate profits - although not that
high considering the length of the testing period (about 18 years).

One serious drawback of this study is the consistency of answering the main
question that is opened and addressed in it. There is no comparison to human
trading experts’ performance in the experiments section, nor any discussion
about this in the conclusion. Of course, the total profits displayed in the
experiments do not come close to what a successful professional day trader or
trading expert would achieve (because it does not come close to an actual
annual wage), but this does not negate the premise. This is because the
authors fail to mention the actual starting capital used in the experiments;
therefore, we can only rely on the Sharpe ratio results for a conclusion. As
we have mentioned before, the proposed approaches highlight remarkable high
values with this metric: in the range 9-20. We need to be careful in calling
this super-human performance, as investors usually consider a Sharpe ratio
over 1 as good and over 3 as excellent [54], but the paper does not mention if
it uses the annualized Sharpe ratio or not. For more intuition on this, a
ratio of 0.2-0.3 is in line with the broader market [55]. There is still a lot
of skepticism behind these claims though, as the agents were tested on
particular futures and commodities; one needs a broader portfolio to claim
super-human success even though, at first, the results seem to be pointing in
that direction. Nevertheless, the proposed agents are showed to generate a
positive result in the long run which is an accomplishment by itself as there
are some research studies [56] arguing that many individual investors hold
undiversified portfolios and trade actively, speculatively and to their own
detriment. So, if we switch our research question to address and average
trader, Deep Reinforcement learning approaches would probably be superior.

Quantitative Trading on Stock Market Based on Deep Reinforcement Learning [57]
is another study that explores a deep actor-only approach to quantitative
trading. Its main contributions are a throughout analysis of the advantage of
choosing a deep neural network (LSTM) compared to a fully connected one and
the impact of some combinations of the technical indicators on the performance
on the daily-data Chinese markets. This paper does prove that a deep approach
is better and picking the right technical indicators does make a difference if
we care about the return. The pure-performance results are mixed, the authors
show that the proposed method can make decent profit in some stocks but it can
also have oscillatory behaviour in others.

We have omitted impressive works like Enhancing Time Series Momentum
Strategies Using Deep Neural Networks [45] in this section as it focuses more
on the portfolio management part, plus it uses offline batch gradient ascent
methods to directly optimize the objective function (maximizing the reward and
minimizing the risk, Sharpe Ratio) which is fundamentally different from the
standard actor-only RL where a distribution needs to be learnt to arrive at
the final policy [19].

###  3.3 Actor-critic Deep Reinforcement Learning

Overview

The third type of RL, actor-critic framework aims to simultaneously train two
models at a time: the actor that learns how to make the agent respond in a
given state and the critic - measuring how good the chosen action really was.
This approach proved to be one of the most successful ones in the literature,
state-of-the-art actor-critic DRL algorithms like PPO (in an actor-critic
setting like it was presented in the original paper) or A2C solving complex
environments [23]. However, in spite of this success, this method seems still
unexplored; at least as of 2019 [19], it stayed among the least studied
methods in DRL for trading and as of current day, only a few new studies in
this area have been published. Something that actor-critic algorithms like PPO
do better than previous mentioned RL types is that it addresses well-known
problems of applying RL to complex environments. One of it is that the
training data that is generated during learning is itself dependent on the
current policy so that means the data distributions over observations and
rewards are constantly changing as the agent learns which is a major cause of
instability. RL also suffers from a very high sensitivity to initialization or
hyper-parameters: if the policy suffers a massive change (due to a high
learning rate, for example) the agent can be pushed to a region of the search
space where it collects the next batch of data over a very poor policy causing
it to maybe never recover again.

Related Works

Deep Reinforcement Learning for Automated Stock Trading: An Ensemble Strategy
[58]. This paper proposed an ensemble system (figure 6) using deep
reinforcement learning to further better the results in the literature with
multiple stock trading experiments. They introduce an ensemble deep model that
has 3 state-of-the-art DRL algorithms at core: PPO, A2C, DDPG. The authors
state that the ensemble strategy makes the trading more robust and reliable to
different market situations and can maximize return subject to risk
constraints. The experiments are solid, being based on 30 stocks from Dow
Jones portfolio. The features used to train the ensemble model are the
available balance, adjusted close price, shares already owned plus some
technical indicators: MACD, RSI, CCI and ADX. The choice for the action space
is solid as well: $\\{-k,...,-1,0,1,...,k\\}$, where $k$ and $-k$ presents the
number of shares we can buy and sell, so in total there are 2k+1 possibility
for one stock. This is later easily normalized to the continuous $[-1,1]$
action space for the main policy algorithms to work. It seems the authors do a
better job at portfolio management problem with this approach - not being
limited to buy only 1 share when in a great spot (like we saw in some of the
previously reviewed papers). However, all the features from all the 30 stocks
are merged, the aim being to also choose what stock to buy or sell – this is
another part of portfolio management which is different and maybe should have
been treated separately to the main trading algorithm.

Figure 6: Overview of the ensemble DRL strategy [58] Figure 7: Cumulative
return curvess on different strategies on US 30 stocks index. Initital
portfolio value: $1m. [58]

The way the ensemble strategy works is that a growing window training and a
rolling window validation time period are chosen continuously so that we can
pick the best performing algorithm over the validation set. This one is then
used for the next fixed period of time as our sole agent. The motivation
behind this is each trading agent is sensitive to different type of trends.
One agent performs well in a bullish trend but acts bad in a bearish trend.
Another agent is more adjusted to a volatile market. This paper also discusses
the impact of market crash or extreme instability to a trading agent. They
employ the financial turbulence index that measures extreme asset price
movements, using it together with a fixed threshold so that the agent sells
everything and stops for making any trades if the markets seem to crash or be
notoriously unstable. This is indeed a good idea; however, one can argue that
it’s these periods of extreme volatility that a very good trading agent can
exploit to make huge profits. The results of the experiments can be seen in
figure 7. It seems the PPO algorithm still managed to provide slightly higher
cumulative reward (at about 15% annually); however, the Sharpe ratio which is
the risk-adjusted reward is higher in case of the ensemble (which has 13%
return annually): $1.30$ compared to $1.10$. The proposed approach also beats
the Dow Jones Industrial average index which is a staple benchmark for this
portfolio and the min-variance portfolio allocation.

We personally think that a single model for a stock approach in this case will
lead to much higher returns, instead of trying to build a master model that
can handle all 30+ stocks at once. However, this still provides incredible
results for the defined scope – handling portfolio management and the stock
trading strategy simultaneously. The return is much greater than what you can
expect through a buy and hold strategy (around 7%) and the algorithm seems
stable enough during extreme market conditions (the March 2020 stock market
crash).

A strongly related paper with [58] is Practical Deep Reinforcement Learning
Approach for Stock Trading [22]. This paper only investigates the DDPG
algorithm (from the previously reviewed ensemble) under the same conditions
and with the same evaluation metrics but without any technical indicators as
input. There are other minor differences: historical daily prices from 2009 to
2018 to train the agent and test the performance; Data from early 2009 to late
2014 are used for training, and the data from 2015 is used for validation; we
test our agent’s performance on trading data, which is from early 2016 to late
2018. The results are pretty good: 25.87% annualized return on this US 30 (Dow
Jones) index, more than 15.93% obtained through the Min-Variance method and
16.40% the industrial average. The comparisons on Sharpe ratios only are also
favorable (1.79 against 1.45 and 1.27) which makes it more robust than the
others in balancing risk and return.

Stock Trading Bot Using Deep Reinforcement Learning [59] is an especially
interesting work because of the fact that it combines a deep reinforcement
learning approach with sentiment analysis [60] (external information from news
outlets) and proves that the proposed approach can learn the tricks of stock
trading. Such a study is definitely important because many people are
skeptical of the effectiveness of a pure quantitative learning algorithm in
trading. Although the most important part of their system is the RL agent,
this extra component that tries to predict future movements in the markets
through sentiment analysis on financial news is an extra feature to the Deep
Reinforcement Learning model. We can also view it as an extra precaution, an
important signal tool in case of critical news or as a fail-safe system in
case something major happens in the world that the pure quantitative trader is
not aware of, though this is not what the authors directly had in mind.

The proposed approach in this paper is fairly simple at core: they use DDPG
for the reinforcement learning agent and a recurrent convolutional neural
network (RCNN) for classification of news sentiment. For the supervised
learning part, the authors argue that a pure recurrent neural network (RNN)
approach here would fail because it won’t determine discriminative phrases in
a text, the advantage with convolutions being the max-pooling layers that come
afterwards that can extract a good representation for the input. However, the
recurring part is still needed as it provides contextual information to a
greater extent. Hence, RCNN is used for this task - together with word
embeddings, of course, for abstract text representation. Apart from the text
embeddings, the change in stock price is also added as input and the network
predicts a binary value $(0/1)$ \- whether the price will go up (or down) in
the following few days, this bit being merged with the information state for
the RL agent.

Figure 8: Training over 5 months with NASDAQ-GOOGL stock. Initial budget: $1m.
The red line indicates the agent’s assets, and the blue line indicates the
value of the stagnant stock [59]

The experiments involved market data on a daily timeframe for one stock, so
really not on a large scale, but let’s not forget that the aim for this paper
is more on the side of a proof of concept. The supervised learning system for
sentiment analysis (RCNN) needed to be trained beforehand in order to
accurately provide the extra feature for a possible future price movement
according to the news outlets. The authors used almost 96k news headlines for
training and around 31.5k for testing, in total, from 3.3k companies. The
network is composed of embedding + Convolution + LSTM + Output with around 26M
total model parameters. The results acquired are satisfactory: 96.88% accuracy
on test. However, information about the prior probability of the dataset used
in missing, there should be a mention about the distribution to fully claim
that such an accuracy is indeed good. Even though the experiments don’t show
an overwhelming increase in profits (figure 8), it does prove that the agent
does learn basic strategies: it does buy and sell continuously, it prefers to
hold when there is a continuous decrease in price and it always maintains a
higher value than the stagnant stock value. The authors also experimented with
the reward function, trying 3 variants: difference between RL-agent asset
value and stagnant asset value, difference between the cost at which the
stocks are sold and the cost at which the stocks were bought, a binary reward
representing if the action was profitable or not. The first 2 reward functions
failed, the agent did not manage to learn this way, however, the last one
managed to be the most successful and make the RL system work. Obviously, this
is not the best reward system, it does not account for risk at all but it will
sometimes work for the sole purpose of maximizing the gain.

There are two more work we have to include in this section.

First, the GDPG system [1] we briefly mentioned in the critic-only RL section
as a reference for comparison. The authors introduce GDPG as an actor-critic
strategy that combines the Q-Network from the GDQN system with a policy
network. It is showed (table 1) that this achieves more stable risk-adjusted
returns and outperforms the baseline Turtle trading strategy. To put things
into perspective, the turtle trading system is a well-known trend following
strategy that was originally taught by Richard Dennis back in 1979. It was
really successful back then and researchers use this system nowadays as a
valid benchmark when comparing trading performances. The basic strategy is to
buy futures on a 20-day high (breakout) and sell on a 20-day low, although the
full set of rules is more intricate.

Figure 9: Cumulative trade returns. In order: commodity, equity index and fixed income, FX and the portfolio of using all contracts [59] | GDQN | GDPG | Turtle
---|---|---|---
Symbol | SR | R(%) | SR | R(%) | SR | R(%)
AAPL | 1.02 | 77.7 | 1.30 | 82.0 | 1.49 | 69.5
GE | -0.13 | -10.8 | -0.22 | -6.39 | -0.64 | -17.0
AXP | 0.39 | 20.0 | 0.51 | 24.3 | 0.67 | 25.6
CSCO | 0.31 | 20.6 | 0.57 | 13.6 | 0.12 | -1.41
IBM | 0.07 | 4.63 | 0.05 | 2.55 | -0.29 | -11.7
Table 1: Comparison between GDQN, GDPG, Turtle strategy in some U.S. stocks
over a 3-year period (2016-2019) [1]

Secondly, there is [19] that we said we will get back to in section 3.1. This
study directly compares the A2C algorithm with an actor-only approach (PG) and
the critic-only approach we discussed in section 3.1. A graph with their
experimental results can be seen in figure 9; it seems the A2C comes in second
place on the whole portfolio return, this is explained by the authors saying
that A2C generates larger turnovers, leading to smaller average returns per
turnover. The experiments also showed that the A2C algorithm can monetize on
large moves without changing positions and also deal with markets that are
more mean-reverting [19].

##  4 Summary & Conclusion

Deep Reinforcement Learning is a growing field of interest in the financial
world. All the papers that were discussed here had their own experimental
settings, assumptions and constrains. This makes it hard to compare one
against another especially when the scope of the study is different between
them, a direct comparison is not feasible in this scenario. Nevertheless,
comparisons between different types of DRL methods were possible [37, 14, 19,
13] in the context of same study but also, limited, in the context of
different studies as we saw with [1] and [13]. We have observed that DRL can
be a powerful tool in quantitative low-frequency trading, a few studies with
more realistic setups [14, 1, 13] obtaining over 20% annual return with risk-
aware metrics also being used. We have come to the conclusion that such agents
can rival human traders in specific markets, but further research should be
conducted in this area.

There is academic research [61, 62] (granted not that recent compared to the
papers reviwed here) that claims the market cannot be predicted and taken
advantage of, supporting the hypothesis with data and experiments. With this
paper, we have also disputed such claims. It is true that some markets cannot
be predicted under some experimental setups, such as equities traded on a
daily basis, as the authors of the mentioned studies showed, but this does not
mean no market can be predicted in any setting - the above reviewed literature
backs that up.

A lot of reviewed papers focused on trading on a daily timeframe. There is a
fundamental problem with this that is barely brought up in the discussion
section of the reviewed studies. Over longer time scales, such as days and
weeks, market activity can be a direct result of complex interactions between
political news, legal rulings, public sentiment, social hype, business
decisions, etc. Large trades by institutional investors can also have a big
impact. These are not aspects that can be modeled or exploited by quantitative
algorithms (e.g. the well known March 2020 market crash caused by COVID-19).
However, if we zoom into the market activity for a single hour, minute, or
second, we can often see patterns. It is believed that these patterns are a
result of algorithmic rule-based trading that we can exploit with an
intelligent system.

Another concern that we have, especially for the testing stage of the proposed
solutions to the problem discussed here, is that many of the papers consider
years on end as testing intervals of time to assess the performance of a
learning algorithm. For proof-of-concept purposes, this is not a problem;
however, in a realistic setting, one would not deploy such an agent in the
wild, the stock market is dynamic and needs constant retraining as covariance
shifts. Trading/testing sessions of a fixed, relatively small length (2-3
months) would be much more relevant because we make use of more data (the
latest data units to be precise) and the learning algorithm would be much more
caught-up with the market in this way.

Future possible directions of research include:

  * •

More extensive experiments on live-trading platforms rather than backtesting
or very limited real-time trading.

  * •

Direct comparisons between DRL trading agents with human traders. For example,
having a professional day trader run against an algorithmic DRL agent over the
same market and interval of time in a controller experiment to observe which
one would get higher returns.

  * •

More comparisons among state-of-the-art DRL approaches under similar
conditions and data sources.

  * •

More research into the behaviour of DRL agents under critical market
conditions (stock market crashes).

Although researching these points would greatly advance the field of AI in
finance, there are non-trivial problems to be solved along the way, that even
studies we have reviewed in this paper, which claim that they have a realistic
setting, fall short of addressing some of them.

First, if we are relying on machine learning (pattern matching/DRL) for our
trading agent, we can’t hope to make good predictions on higher time scales,
as we have explained above. However, if we go too low (timeframewise), the
transaction fees might overcome the actual small gains of relative high
frequency trading. Therefore, picking the perect timescale is not a trivial
problem and should be addressed more.

Secondly, a really good model (according to backtesting) is not enough to
secure profits, as most of the papers we have reviewed seemed to suggest. As
we saw, the final step in finance research is often backtesting on historical
data. If the model does well, the researchers declare success, usually not
addressing the fact that the model would probably be questionable in a
production environment. This does not mean that there is no research there on
real-trading but if their new algorithm performed as well in the real-world,
they certainly would not publish a paper about it and give away their edge. To
expand on this matter, on many other fields (computer vision, voice
recognition), machine learning, train-test performance directly correlates
with live performance as the data distribution does not change significantly
over time, we can be pretty sure that a model performing well on the test set
also does so in production. In trading, our training, backtesting, and live
environments can become so different that we can’t really make any guarantees.

In third, the backtesting should be generally improved. It should simulate a
live trading environment really well, meaning latencies, non-standard order
types, exchange commission structures and slippages. And even so, latencies in
the real world may be stable during low-activity periods and spike during
high-activity periods, something that would be incredibly difficult to
simulate. For example, our model can be really good but the results are poor
in a live enviornment becaues the connection (latency) is high and we won’t be
able to observe new data fast enough before the competition processes it
already. It should also be bug-free and handle edge cases really well: in
simulation everything works perfectly, but in the real world we run into API
issues, request throttling, and random order rejections during busy periods.
Therefore, no matter how good the backtesting seems to be, it is still
fundamentally different from a live environment, but we can still try to amend
some of these issues.

## References

  * [1] Xing Wu, Haolei Chen, Jianjia Wang, Luigi Troiano, Vincenzo Loia, and Hamido Fujita.  Adaptive stock trading strategies with deep reinforcement learning methods.  Information Sciences, 2020.
  * [2] Mojtaba Nabipour, Pooyan Nayyeri, Hamed Jabani, Amir Mosavi, E Salwana, et al.  Deep learning for stock market prediction.  Entropy, 22(8):840, 2020.
  * [3] David E Goldberg.  Genetic algorithms.  Pearson Education India, 2006.
  * [4] James Kennedy and Russell Eberhart.  Particle swarm optimization.  In Proceedings of ICNN’95-International Conference on Neural Networks, volume 4, pages 1942–1948. IEEE, 1995.
  * [5] Robert G Cooper, Scott J Edgett, and Elko J Kleinschmidt.  Portfolio management.  Pegasus, New York, 2001.
  * [6] Farzan Soleymani and Eric Paquet.  Financial portfolio optimization with online deep reinforcement learning and restricted stacked autoencoder-deepbreath.  Expert Systems with Applications, page 113456, 2020.
  * [7] Jingyuan Wang, Yang Zhang, Ke Tang, Junjie Wu, and Zhang Xiong.  Alphastock: A buying-winners-and-selling-losers investment strategy using interpretable deep reinforcement attention networks.  In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1900–1908, 2019.
  * [8] William F Sharpe.  The sharpe ratio.  Journal of portfolio management, 21(1):49–58, 1994.
  * [9] Thomas G Fischer.  Reinforcement learning in financial markets-a survey.  Technical report, FAU Discussion Papers in Economics, 2018.
  * [10] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * [11] Yong-Duan Song, Qi Song, and Wen-Chuan Cai.  Fault-tolerant adaptive control of high-speed trains under traction/braking failures: A virtual parameter-based approach.  IEEE Transactions on Intelligent Transportation Systems, 15(2):737–748, 2013.
  * [12] David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al.  A general reinforcement learning algorithm that masters chess, shogi, and go through self-play.  Science, 362(6419):1140–1144, 2018.
  * [13] Yue Deng, Feng Bao, Youyong Kong, Zhiquan Ren, and Qionghai Dai.  Deep direct reinforcement learning for financial signal representation and trading.  IEEE transactions on neural networks and learning systems, 28(3):653–664, 2016.
  * [14] Chien Yi Huang.  Financial trading as a game: A deep reinforcement learning approach.  arXiv preprint arXiv:1807.02787, 2018.
  * [15] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.
  * [16] C. J. Watkins and Dayan P.  Q-learning.  Machine Learning, 1992.
  * [17] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  2014\.
  * [18] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
  * [19] Zihao Zhang, Stefan Zohren, and Stephen Roberts.  Deep reinforcement learning for trading.  The Journal of Financial Data Science, 2(2):25–40, 2020.
  * [20] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In International conference on machine learning, pages 1928–1937, 2016.
  * [21] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [22] Zhuoran Xiong, Xiao-Yang Liu, Shan Zhong, Hongyang Yang, and Anwar Walid.  Practical deep reinforcement learning approach for stock trading.  arXiv preprint arXiv:1811.07522, 2018.
  * [23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347, 2017.
  * [24] Zhipeng Liang, Hao Chen, Junhao Zhu, Kangkang Jiang, and Yanran Li.  Adversarial deep reinforcement learning in portfolio management.  arXiv preprint arXiv:1808.09940, 2018.
  * [25] Jamil Baz, Nicolas Granger, Campbell R Harvey, Nicolas Le Roux, and Sandy Rattray.  Dissecting investment strategies in the cross section and time series.  Available at SSRN 2695101, 2015.
  * [26] J Welles Wilder.  New concepts in technical trading systems.  Trend Research, 1978.
  * [27] Ikhlaas Gurrib et al.  Performance of the average directional index as a market timing tool for the most actively traded usd based currency pairs.  Banks and Bank Systems, 13(3):58–70, 2018.
  * [28] Mansoor Maitah, Petr Prochazka, Michal Cermak, and Karel Šrédl.  Commodity channel index: Evaluation of trading rule of agricultural commodities.  International Journal of Economics and Financial Issues, 6(1):176–178, 2016.
  * [29] William Wai Him Tsang, Terence Tai Leung Chong, et al.  Profitability of the on-balance volume indicator.  Economics Bulletin, 29(3):2424–2431, 2009.
  * [30] J Stuart Hunter.  The exponentially weighted moving average.  Journal of quality technology, 18(4):203–210, 1986.
  * [31] Robert W Colby and Thomas A Meyers.  The encyclopedia of technical market indicators.  Dow Jones-Irwin Homewood, IL, 1988.
  * [32] Iqoption trading website.  https://eu.iqoption.com/en, 2018.
  * [33] Wei Wang, Yan Huang, Yizhou Wang, and Liang Wang.  Generalized autoencoder: A neural network framework for dimensionality reduction.  In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 490–497, 2014.
  * [34] Yann LeCun, Yoshua Bengio, et al.  Convolutional networks for images, speech, and time series.  The handbook of brain theory and neural networks, 3361(10):1995, 1995.
  * [35] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [36] K Senthamarai Kannan, P Sailapathi Sekar, M Mohamed Sathik, and P Arumugam.  Financial stock market forecast using data mining techniques.  In Proceedings of the International Multiconference of Engineers and computer scientists, volume 1, 2010.
  * [37] Lin Chen and Qiang Gao.  Application of deep reinforcement learning on automated stock trading.  In 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS), pages 29–33. IEEE, 2019.
  * [38] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [39] Johannes Heinrich and David Silver.  Deep reinforcement learning from self-play in imperfect-information games.  arXiv preprint arXiv:1603.01121, 2016.
  * [40] Investopedia webstie average return over one year for professsional traders.  https://www.investopedia.com/articles/active-trading/053115/average-rate-return-day-traders.asp, 2020\.
  * [41] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research, 15(1):1929–1958, 2014\.
  * [42] Vivek Mohan, Jai Govind Singh, and Weerakorn Ongsakul.  Sortino ratio based portfolio optimization considering evs and renewable energy in microgrid power market.  IEEE Transactions on Sustainable Energy, 8(1):219–229, 2016.
  * [43] Curtis Faith.  The original turtle trading rules, 2003.
  * [44] Tobias J Moskowitz, Yao Hua Ooi, and Lasse Heje Pedersen.  Time series momentum.  Journal of financial economics, 104(2):228–250, 2012.
  * [45] Bryan Lim, Stefan Zohren, and Stephen Roberts.  Enhancing time-series momentum strategies using deep neural networks.  The Journal of Financial Data Science, 1(4):19–38, 2019.
  * [46] Zhiyong Tan, Chai Quek, and Philip YK Cheng.  Stock trading with cycles: A financial application of anfis and reinforcement learning.  Expert Systems with Applications, 38(5):4741–4755, 2011.
  * [47] Francesco Bertoluzzo and Marco Corazza.  Testing different reinforcement learning configurations for financial trading: Introduction and applications.  Procedia Economics and Finance, 3:68–77, 2012.
  * [48] Gordon Ritter.  Machine learning for trading.  Available at SSRN 3015609, 2017.
  * [49] George J Klir.  Fuzzy sets.  Uncentainty and Information, 1988.
  * [50] Nikhil R Pal and James C Bezdek.  Measuring fuzzy uncertainty.  IEEE Transactions on Fuzzy Systems, 2(2):107–118, 1994.
  * [51] Chin-Teng Lin, C. S. George Lee, et al.  Neural-network-based fuzzy logic control and decision system.  IEEE Transactions on computers, 40(12):1320–1336, 1991.
  * [52] Yue Deng, Youyong Kong, Feng Bao, and Qionghai Dai.  Sparse coding-inspired optimal trading system for hft industry.  IEEE Transactions on Industrial Informatics, 11(2):467–475, 2015\.
  * [53] John Moody and Matthew Saffell.  Learning to trade via direct reinforcement.  IEEE transactions on neural Networks, 12(4):875–889, 2001.
  * [54] Investopedia webstie sharpe ration.  https://www.investopedia.com/ask/answers/010815/what-good-sharpe-ratio.asp, 2020\.
  * [55] Daytrading webstie sharpe ration.  https://www.daytrading.com/sharpe-ratio, 2020.
  * [56] Brad M Barber and Terrance Odean.  The behavior of individual investors.  In Handbook of the Economics of Finance, volume 2, pages 1533–1570. Elsevier, 2013.
  * [57] WU Jia, WANG Chen, Lidong Xiong, and SUN Hongyong.  Quantitative trading on stock market based on deep reinforcement learning.  In 2019 International Joint Conference on Neural Networks (IJCNN), pages 1–8. IEEE, 2019.
  * [58] Hongyang Yang, Xiao-Yang Liu, Shan Zhong, and Anwar Walid.  Deep reinforcement learning for automated stock trading: An ensemble strategy.  Available at SSRN, 2020.
  * [59] Akhil Raj Azhikodan, Anvitha GK Bhat, and Mamatha V Jadhav.  Stock trading bot using deep reinforcement learning.  In Innovations in Computer Science and Engineering, pages 41–49. Springer, 2019.
  * [60] Rudy Prabowo and Mike Thelwall.  Sentiment analysis: A combined approach.  Journal of Informetrics, 3(2):143–157, 2009.
  * [61] Steve Thomas, Andrew Clare, and Nick Motson.  An evaluation of alternative equity indices-part 2: Fundamental weighting schemes.  Available at SSRN 2242034, 2013.
  * [62] Andrew Clare, Nick Motson, and Steve Thomas.  An evaluation of alternative equity indices-part 1: Heuristic and optimised weighting schemes.  Available at SSRN 2242028, 2013.


,

# Qubit gates with simultaneous transport in double quantum dots

Yi-Chao Li1,2, Xi Chen1, J. G. Muga2, E. Ya. Sherman2,3 1 Department of
Physics, Shanghai University, Shanghai 200444, Peoples Republic of China  2
Department of Physical Chemistry, UPV/EHU, Apdo. 644, 48080 Bilbao, Spain  3
IKERBASQUE Basque Foundation for Science, Bilbao, Spain<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

A single electron spin in a double quantum dot in a magnetic field is
considered in terms of a four-level system. By describing the electron motion
between the potential minima by spin-conserving tunneling and spin flip caused
by a spin-orbit coupling, we inversely engineer faster-than-adiabatic state
manipulation operations based on the geometry of four-dimensional (4D)
rotations. In particular, we show how to transport a qubit among the quantum
dots performing simultaneously a required spin rotation.

Keywords: coupled quantum dots, spin-orbit coupling effect, inverse
engineering

##  1 INTRODUCTION

Device architecture based on electrons confined in coupled quantum dots [1, 2,
3, 4] is considered as a potential and significant candidate for quantum
computation and quantum information processing. The advantages of this
architecture are based on the facts that electron spin is a natural qubit with
spin-up and spin-down states, mature semiconductor technology may be used, and
long coherence times on the scale of microseconds have been achieved in these
systems [5, 6]. Laboratories use electric, microwave or magnetic fields to
manipulate spin states, performing $10^{3}\thicksim 10^{5}$ operations in the
spin dephasing time [5, 6, 7, 8, 9, 10].

Scalability of quantum information devices is associated with several
architectures having the capability to transport qubits. In this paper we
theoretically explore a four-level model for a spin in a double quantum dot
(DQD) aiming at the possibilities to implement fast qubit transport with
simultaneous rotations. We achieve this goal for arbitrary rotations by
controlling the synchronized time dependence of interdot tunneling and spin-
orbit coupling (SOC). We inverse-engineer these time dependencies based on our
recent work [11] on the control of four-level systems. The method separates
population control from control of the phases of the bare state basis [12].
Populations can be mapped onto a 4D sphere so their evolution amounts to 4D
transformations controlled by the rotation Hamiltonian that may be engineered
from the target state (in our case via isoclinic rotations and quaternions). A
full Hamiltonian can then be constructed from the rotation Hamiltonian to
realize the desired phase changes. Arbitrary state manipulations require full
flexibility in the Hamiltonian, i.e., the possibility to implement the
different Hamiltonian matrix elements with specific time-dependences. In the
systems of interest, however, there are constraints that hinder certain
manipulations and transitions. In particular, in this paper we examine the
Hamiltonian structure that corresponds to combined tunneling and SOC
controllable couplings, and deduce the possible transformations.

Spin-orbit coupling in semiconductors consists of two main contributions due
to the Dresselhaus- and the Bychkov-Rashba-effect. The former is due to the
bulk inversion asymmetry of material and the latter results from the structure
inversion asymmetry, produced, e.g., by the confining potential or an external
electric field [13]. The practical advantage of the Rashba coupling is the
ability to manipulate it by an external electric field applied across the
semiconductor structure [14, 15]. The Rashba coupling controlled by a high-
frequency ac gate voltage [16] provides an effective method to control the
spin states in short times [17, 18].

This paper is organized as follows. In Section II, we introduce first the
method that parameterizes the time-dependent Hamiltonian and time evolution
operator of a four-level system by using isoclinic rotations and quaternions
[11]. Then we map the Hamiltonian of the spin in a DQD coupled by SOC and
tunneling onto this scheme. In Section III, we apply the method developed in
Section II to design the synchronized time dependences of the control
parameters to perform different qubit operations, such as the interdot
transport combined with spin rotations. Section IV provides discussion of the
results and their relation to other systems. Some details on the structure of
the Hamiltonian are presented in the Appendix.

##  2 ELECTRON IN A DOUBLE QUANTUM DOT: A 4D APPROACH

###  2.1 4D Hamiltonians and evolution operators

The wave function of a four-level system

$\psi(t)=\sum_{n=1}^{n=4}c_{n}(t)e^{i\varphi_{n}}|n{\rangle},$ (1)

where $c_{n},\varphi_{n}$ are real amplitudes and phases (we set
$\varphi_{1}=0$), and $\sum_{n}^{4}c^{2}_{n}=1$, can be decomposed as
$|\psi(t){\rangle}=K(t)|\psi_{r}(t){\rangle}$, where

$\psi_{r}(t)=\sum_{n=1}^{n=4}c_{n}(t)|n{\rangle}$ (2)

is a vector on the surface of a 4D sphere, and the phase information is
contained in

$K(t)=\sum_{n=1}^{n=4}e^{i\varphi_{n}}|n{\rangle}{\langle}n|.$ (3)

The states $|\psi(t){\rangle}$ and $|\psi_{r}(t){\rangle}$ evolve via
evolution operators $U(t)$ and $U_{r}(t)$ related by
$U_{r}(t)=K^{\dagger}(t)U(t)K(0)$,

$\displaystyle|\psi(t){\rangle}=U(t)|\psi(0){\rangle},$
$\displaystyle|\psi_{r}(t){\rangle}=U_{r}(t)|\psi_{r}(0){\rangle},$ (4)

where we set the initial time as $0$. Accordingly, the rotation-related
Hamiltonian in the Hilbert space is defined as

$H_{r}(t)=i\hbar\dot{U}_{r}(t)U_{r}^{\dagger}(t),$ (5)

and the total Hamiltonian is

$\displaystyle H(t)$ $\displaystyle=$ $\displaystyle
i\hbar\dot{U}(t)U^{\dagger}(t)$ (6) $\displaystyle=$ $\displaystyle
K(t)H_{r}(t)K^{\dagger}(t)+i\hbar\dot{K}(t)K^{\dagger}(t).$

To engineer $H_{r}$ for a specific rotation, it is convenient to express first
a general 4D rotation matrix as a product of two isoclinic rotation matrices
[19, 20]:

$U_{r}(t)=\left[\begin{array}[]{cccc}q_{w}&-q_{x}&-q_{y}&-q_{z}\\\
q_{x}&q_{w}&-q_{z}&q_{y}\\\ q_{y}&q_{z}&q_{w}&-q_{x}\\\
q_{z}&-q_{y}&q_{x}&q_{w}\\\
\end{array}\right]\left[\begin{array}[]{cccc}p_{w}&-p_{x}&-p_{y}&-p_{z}\\\
p_{x}&p_{w}&p_{z}&-p_{y}\\\ p_{y}&-p_{z}&p_{w}&p_{x}\\\
p_{z}&p_{y}&-p_{x}&p_{w}\\\ \end{array}\right],$ (7)

where $q_{i}$ and $p_{j}$ are components of two unit quaternions
$q=q_{w}+q_{x}\textbf{{i}}+q_{y}\textbf{{j}}+q_{z}\textbf{{k}}$ and
$p=p_{w}+p_{x}\textbf{{i}}+p_{y}\textbf{{j}}+p_{z}\textbf{{k}}$. We shall
parameterize them in terms of generalized 4D spherical angles [21, 22],

$\displaystyle q_{w}(t)$ $\displaystyle=$ $\displaystyle\cos\gamma_{1}(t),$
$\displaystyle q_{x}(t)$ $\displaystyle=$
$\displaystyle\sin\gamma_{1}(t)\cos\theta_{1}(t),$ $\displaystyle q_{y}(t)$
$\displaystyle=$
$\displaystyle\sin\gamma_{1}(t)\sin\theta_{1}(t)\cos\phi_{1}(t),$
$\displaystyle q_{z}(t)$ $\displaystyle=$
$\displaystyle\sin\gamma_{1}(t)\sin\theta_{1}(t)\sin\phi_{1}(t),$
$\displaystyle p_{w}(t)$ $\displaystyle=$ $\displaystyle\cos\gamma_{2}(t),$
$\displaystyle p_{x}(t)$ $\displaystyle=$
$\displaystyle\sin\gamma_{2}(t)\cos\theta_{2}(t),$ $\displaystyle p_{y}(t)$
$\displaystyle=$
$\displaystyle\sin\gamma_{2}(t)\sin\theta_{2}(t)\cos\phi_{2}(t),$
$\displaystyle p_{z}(t)$ $\displaystyle=$
$\displaystyle\sin\gamma_{2}(t)\sin\theta_{2}(t)\sin\phi_{2}(t),$ (8)

where $0\leq\phi_{1,2}\leq 2\pi$, $0\leq\theta_{1,2},\gamma_{1,2}\leq\pi$.
Thus by using $U(t)=K(t)U_{r}(t)K^{\dagger}(0)$ and
$H(t)=K(t)H_{r}(t)K^{{\dagger}}(t)+i\dot{K}(t)K^{{\dagger}}(t)$, we find the
parameterized forms for the evolution operator $U(t)$ and the Hamiltonian
$H(t)$. The explicit expressions are lengthy, and will not be reported here.

###  2.2 Single electron in a double quantum dot

Figure 1:  Schematic diagram of single electron states in a double quantum
dot.

Consider a single electron spin in a semiconductor DQD, for example made of
silicon or GaAs, with tunneling and Rashba spin-orbit coupling, as shown in
Fig. 1. We use a bare basis of spin up and down states localized in each well,
numbered as $|\psi_{L\downarrow}{\rangle}=|1{\rangle}$,
$|\psi_{R\downarrow}{\rangle}=|2{\rangle}$,
$|\psi_{R\uparrow}{\rangle}=|3{\rangle}$,
$|\psi_{L\uparrow}{\rangle}=|4{\rangle}$. Following the derivation in A, and
after an diagonal energy shift of $-{\Delta}/2,$ the Hamiltonian of this
system (see (32)) can be written as

$H_{0}(t)=\hbar\left[\begin{array}[]{cccc}0&\tau(t)&\alpha(t)&0\\\
\tau(t)&0&0&-\alpha(t)\\\ \alpha^{*}(t)&0&\Delta&\tau(t)\\\
0&-\alpha^{*}(t)&\tau(t)&\Delta\\\ \end{array}\right].$ (9)

Here $\tau(t)$ represents the tunneling coupling between the two quantum dots,
$\alpha(t)$ is the Rashba coupling, and $\Delta$ is a Zeeman splitting. All
these quantities have dimensions of frequency. Following the approach of
Mal’shukov et al. [16], we consider the time-dependent Rashba coupling in the
complex form $\alpha(t)=\alpha_{0}+\alpha_{1}(t)e^{i\omega t}$.

The Hamiltonian structure corresponds topologically to a diamond-configuration
[11], which, in the parametric expression of $H(t)$ we may impose with the
conditions

$\dot{\theta}_{1}=\dot{\theta}_{2}=\dot{\phi}_{1}=\dot{\phi}_{2}=0,\
\phi_{1}=\phi_{2}=0,$ (10)

see Fig. 2. Specifically, after substituting (10) in the parameterized form of
(6), the Hamiltonian $H(t)$ acquires the corresponding form

Figure 2: Schematic links between the bare states in Fig. 1 showing the
“diamond” structure of the transition. Here the positions of the levels are
not related to their energies.

$\displaystyle H(t)$ $\displaystyle=$
$\displaystyle\hbar\\{-\dot{\varphi}_{2}(t)|2{\rangle}{\langle}2|-\dot{\varphi}_{3}(t)|3{\rangle}{\langle}3|-\dot{\varphi}_{4}(t)|4{\rangle}{\langle}4|$
(11) $\displaystyle-$ $\displaystyle
i[e^{-i\varphi_{2}(t)}(\dot{\gamma}_{1}(t)\cos{\theta_{1}}+\dot{\gamma}_{2}(t)\cos{\theta_{2}})]|1{\rangle}{\langle}2|$
$\displaystyle+$ $\displaystyle
e^{-i\varphi_{3}(t)}(\dot{\gamma}_{1}(t)\sin{\theta_{1}}+\dot{\gamma}_{2}(t)\sin{\theta_{2}})|1{\rangle}{\langle}3|$
$\displaystyle+$ $\displaystyle
e^{i[\varphi_{2}(t)-\varphi_{4}(t)]}(-\dot{\gamma}_{1}(t)\sin{\theta_{1}}+\dot{\gamma}_{2}(t)\sin{\theta_{2}})|2{\rangle}{\langle}4|$
$\displaystyle+$ $\displaystyle
e^{i[\varphi_{3}(t)-\varphi_{4}(t)]}(\dot{\gamma}_{1}(t)\cos{\theta_{1}}-\dot{\gamma}_{2}(t)\cos{\theta_{2}})|3{\rangle}{\langle}4|]\\}$
$\displaystyle+$ $\displaystyle{\rm H.c.}$

To make $H_{0}(t)$ and $H(t)$ fully consistent, we further fix the angles as

$\displaystyle\dot{\theta}_{2}=\theta_{2}=\dot{\gamma}_{2}=\gamma_{2}=0,$
$\displaystyle\varphi_{2}(t)=-\frac{\pi}{2},\varphi_{3}(t)=-\Delta
t+\frac{\pi}{2},\varphi_{4}(t)=-\Delta t.$ (12)

Then (11) gives

$\eqalign{H(t)&=\hbar\\{\dot{\gamma}(t)[\cos\theta(|1{\rangle}{\langle}2|+|3{\rangle}{\langle}4|)-e^{i\Delta
t}\sin\theta(|1{\rangle}{\langle}3|-|2{\rangle}{\langle}4|)]\\\
&+\Delta(|3{\rangle}{\langle}3|+|4{\rangle}{\langle}4|)\\}+{\rm H.c.},}$ (13)

where we have simplified the notation as $\gamma(t)=\gamma_{1}(t)$,
$\theta=\theta_{1}$. Now we may impose $H_{0}(t)=H(t)$, as they have the same
structure, to find the following relations between control functions and
auxiliary angles,

$\displaystyle\tau(t)=\dot{\gamma}(t)\cos\theta,$
$\displaystyle\alpha(t)=-e^{i\Delta t}\dot{\gamma}(t)\sin\theta,$ (14)

which implies $\alpha_{0}=0$, $\alpha_{1}(t)=-\dot{\gamma}_{1}(t)$,
$\omega=\Delta$ (i.e., the external bias is in resonance with the Zeeman
frequency), and $\theta$ can be considered as a coupling mixing angle. Under
the conditions stated in Eqs. (2.2), the parameterized time-evolution operator
becomes

$\eqalign{U(t)=\\\
\left[\begin{array}[]{cccc}\cos\gamma(t)&-i\cos\theta\sin\gamma(t)&i\sin\gamma(t)\sin\theta&0\\\
-i\cos\theta\sin\gamma(t)&\cos\gamma(t)&0&-i\sin\gamma(t)\sin\theta\\\
ie^{-i\Delta t}\sin\gamma(t)\sin\theta&0&e^{-i\Delta
t}\cos\gamma(t)&-ie^{-i\Delta t}\cos\theta\sin\gamma(t)\\\ 0&-ie^{-i\Delta
t}\sin\gamma(t)\sin\theta&-ie^{-i\Delta t}\cos\theta\sin\gamma(t)&e^{-i\Delta
t}\cos\gamma(t)\end{array}\right]}.$ (15)

We impose the boundary condition
$\gamma(0)=2n\pi,\,n=\ldots,-2,-1,0,1,2,\ldots$, to guarantee $U(0)=1$ at the
initial time.

##  3 Applications

###  3.1 Qubit preparation

Assume that the four-level system is initialized in state $|1{\rangle}$ on the
left well and the objective is to prepare from it an arbitrary qubit in the
right well encoded in levels $|2{\rangle}$ and $|3{\rangle}$. Besides the
conditions in (2.2), we set $\psi(0)=(1,0,0,0)$ and
$\psi(T)=(b_{1},b_{2},b_{3},b_{4})$, where $T$ is the duration time and
$b_{n}$ are final complex amplitudes which satisfy $\sum_{n}|b_{n}|^{2}=1$. By
using $\psi(T)=U(T)\psi(0)$, we have

$\displaystyle b_{1}$ $\displaystyle=$ $\displaystyle\cos\gamma(T),$
$\displaystyle b_{2}$ $\displaystyle=$
$\displaystyle-i\cos\theta\sin\gamma(T),$ $\displaystyle b_{3}$
$\displaystyle=$ $\displaystyle ie^{-i\Delta T}\sin\gamma(T)\sin\theta,$
$\displaystyle b_{4}$ $\displaystyle=$ $\displaystyle 0.$ (16)

We can transfer $|1{\rangle}$ to any bare state except $|4{\rangle}$, or to
arbitrary superpositions of $|2{\rangle}$ and $|3{\rangle}$ (i.e., any qubit
on the right well) by imposing $\gamma(T)=(2n+1)\pi/2$, $n=0,\pm 1,...$.

As an example we shall perform a state transfer to
$b_{1}=0,b_{2}=1/2,b_{3}=e^{i{\pi}/{2}}\sqrt{3}/2,b_{4}=0$. Equation (3.1)
with

$\displaystyle\theta=-\frac{\pi}{3},\;\;\,\,\,\,T=\frac{3\pi}{2\Delta},$
$\displaystyle\gamma(0)=0,\;\;\,\,\gamma(T)=\frac{\pi}{2},$
$\displaystyle\dot{\gamma}(0)=0,\;\;\,\,\dot{\gamma}(T)=0,$ (17)

corresponds to the desired final state
$|\psi(T){\rangle}=-i(|2{\rangle}+e^{i\pi/2}\sqrt{3}|3{\rangle})/2$ within an
irrelevant global phase factor. An Ansatz for $\gamma(t)$ consistent with the
above boundary conditions is

$\gamma(t)=\frac{\pi}{4}\left[1-\cos\left(\frac{\pi t}{T}\right)\right].$ (18)

The resulting tunneling and Rashba SOC are calculated from (2.2) as

$\displaystyle\tau(t)=\frac{\pi^{2}}{4T}\sin\left(\frac{\pi
t}{T}\right)\cos\theta,$ $\displaystyle\alpha(t)=-e^{i\Delta
t}\frac{\pi^{2}}{4T}\sin\left(\frac{\pi t}{T}\right)\sin\theta,$ (19)

with the characteristic values $\tau,\alpha\propto{1}/{T}$. We can prepare a
qubit with an arbitrary relative phase by adjusting $\Delta$ and the operation
time $T$ as long as the tunneling and SOC are experimentally feasible. We plot
the time-dependence of the tunneling matrix elements, Rashba SOC, and
populations evolution of all bare states in Fig. 3 with parameters
corresponding to the $g^{*}-$factor of electron in GaAs ($g^{*}=-0.44$) with
$B=100$ mT, $\Delta\approx 2\pi\times 0.5$ GHz, and $T=3\pi/(2\Delta)=1.5$ ns
[23].

Figure 3:  (a) $T\tau(t)$ (dot-dashed blue line) and $T|\alpha(t)|$ (solid red
line). (b) Populations of $|1{\rangle}$ (dot-dashed blue line), $|2{\rangle}$
(dashed purple line), $|3{\rangle}$ (green triangles) and $|4{\rangle}$ (solid
red line) for the transfer from $|1{\rangle}$ to
$-i(|2{\rangle}+e^{i\pi/2}\sqrt{3}\ |3{\rangle})/2$. Parameters: $B=100$ mT,
$\Delta\approx 2\pi\times 0.5$ GHz, and $T=1.5$ ns.

###  3.2 Qubit transport and rotation

Our method may be applied to transport the qubit from one dot to the other
applying simultaneously some qubit rotation, i.e., to produce an arbitrary
gate. Suppose we have already prepared a qubit in the left dot in an arbitrary
superposition of $|1{\rangle}$ and $|4{\rangle}$ as
$\psi(0)=\cos\chi|1{\rangle}+e^{i\mu}\sin\chi|4{\rangle}$, where $\chi$ is the
initial amplitude mixing angle and $\mu$ is the initial relative phase. The
corresponding general final state with the unitary evolution operator (15) is
given by the amplitudes

$\displaystyle b_{1}$ $\displaystyle=$ $\displaystyle\cos\chi\cos\gamma(T),$
$\displaystyle b_{2}$ $\displaystyle=$ $\displaystyle Ae^{i\zeta_{A}},$
$\displaystyle b_{3}$ $\displaystyle=$ $\displaystyle Be^{i(\zeta_{B}-\Delta
T)},$ $\displaystyle b_{4}$ $\displaystyle=$
$\displaystyle\sin\chi\cos\gamma(T)e^{i(\mu-\Delta T)},$ (20)

where

$\displaystyle A$ $\displaystyle=$
$\displaystyle\frac{\sin\gamma(T)}{\sqrt{2}}\sqrt{1+\cos 2\chi\cos
2\theta+\cos\mu\sin 2\chi\sin 2\theta},$ $\displaystyle B$ $\displaystyle=$
$\displaystyle\frac{\sin\gamma(T)}{\sqrt{2}}\sqrt{1-\cos 2\chi\cos
2\theta-\cos\mu\sin 2\chi\sin 2\theta},$ $\displaystyle\zeta_{A}$
$\displaystyle=$
$\displaystyle-\arctan\left(\cot\mu+\cot\theta\frac{\cot\chi}{\sin\mu}\right),$
$\displaystyle\zeta_{B}$ $\displaystyle=$
$\displaystyle-\arctan\left(\cot\mu-\tan\theta\frac{\cot\chi}{\sin\mu}\right).$
(21)

We can inversely calculate the coupling mixing angle $\theta$ under the
condition that $\gamma(T)=n\pi/2,n=1,3,5,\cdots$, so that the amplitudes
$b_{1}$ and $b_{4}$ vanish, and for given desired final real amplitudes $A$
and $B$ we obtain:

$\theta=\pm\arccos\sqrt{\frac{A^{2}\cos
2\chi+\sin^{2}\chi(1-2\cos^{2}\chi\sin^{2}\mu)+2\,S\,\cos\mu\sin
2\chi}{1-\sin^{2}2\chi\sin^{2}\mu}},$ (22)

where

$S=\sqrt{4A^{2}B^{2}-\sin^{2}\mu\sin^{2}2\chi}.$ (23)

Notice that there is still a degree of freedom to control the final relative
phase of the qubit on the right. Suppose our target relative phase is
$\lambda=\zeta_{B}-\zeta_{A}-\Delta T$. By adjusting the operation time as
$T=(\zeta-\lambda)/\Delta$, where $\zeta=\zeta_{B}-\zeta_{A}$, the process
produces the desired relative phase $\lambda$. Now we can consider two
examples of application of (22).

Figure 4: (a) Transient bare populations for the transition from
$(|1{\rangle}+e^{i\pi/4}\sqrt{3}\ |4{\rangle})/2$ to $(e^{i\pi/4}\sqrt{3}\
|2{\rangle}+|3{\rangle})/2$. (b) Fidelity versus time $t$. The parameters are
$\gamma(T)=\pi/2$, $\mu=\pi/4$, $\chi=\pi/3$, $T=0.5$ ns, $\Delta=2\pi\times
0.5$ GHz and $\theta=0.685$ (as calculated by (22)), lines are $|1{\rangle}$
(dot-dashed blue line), $|2{\rangle}$ (dashed purple line), $|3{\rangle}$
(green triangles) and $|4{\rangle}$ (solid red line).

Figure 5: Scheme for qubit operations in a chain of quantum dots. The system
is initialized in state $|\downarrow{\rangle}$ in Dot 1 at time $t=0$; then,
from this initial state, a qubit is prepared in Dot 2 at a time $t_{1}$ (the
duration of the process is $T_{1}=t_{1}$). This qubit is transported to Dot 3
with an additional relative phase $\pi/4$ (process duration
$T_{2}=t_{2}-t_{1}$). Finally, a “NOT gate with transport” operation is
applied to flip and transport the qubit to Dot 4 in a process with duration
$T_{3}=t_{3}-t_{2}$. The upper figures represent each qubit on the
corresponding Bloch spheres.

Example 1: transport and phase gate. We assume that $A=\cos\chi$ and
$B=\sin\chi$ and substitute them into (22) to get $\theta=0$, which means that
$\tau=\dot{\gamma}(t)$ and $\alpha=0$. The final state is calculated as

$\displaystyle b_{1}$ $\displaystyle=$ $\displaystyle\cos\chi\cos\gamma(T),$
$\displaystyle b_{2}$ $\displaystyle=$ $\displaystyle-i\sin\gamma(T)\cos\chi,$
$\displaystyle b_{3}$ $\displaystyle=$ $\displaystyle-ie^{i(\mu-\Delta
T)}\sin\gamma(T)\sin\chi,$ $\displaystyle b_{4}$ $\displaystyle=$
$\displaystyle\sin\chi\cos\gamma(T)e^{-i\Delta T}.$ (24)

By letting $\gamma(t)$ evolve from 0 to $\pi/2$, the qubit is transported from
left to right and rotated by a relative phase factor $e^{-i\Delta T}$.

Example 2: transport and NOT gate. The NOT gate with transport swaps the
amplitudes between up and down states, so we set $A=\sin\chi$ and $B=\cos\chi$
in (22) to get

$\theta=\frac{\cos\mu\sin 2\chi}{\sqrt{1-\sin^{2}2\chi\sin^{2}\mu}}.$ (25)

We also impose $\lambda=-\mu$, so we fix the operation time as
$T=(\zeta+\mu)/\Delta$.

The population transfer of the NOT gate+transport operation is depicted in
Fig. 4. We use the same Ansatz for $\gamma(t)$ in (18) to keep
$\gamma(T)=\pi/2$ and choose for the example $\mu=\pi/4$, $\chi=\pi/3$. The
population is successfully swapped in Fig. 4 (a), and Fig. 4 (b) shows the
fidelity $F=|{\langle}\psi_{\rm trg}|\psi(t){\rangle}|^{2}$, where the target
state is $|\psi_{\rm trg}{\rangle}=\sin\chi
e^{i\mu}|2{\rangle}+\cos\chi|3{\rangle}$.

##  4 Discussion

By applying an approach based on four-dimensional rotations, we studied
electron charge and spin motion in tunneling- and spin-orbit coupled quantum
dots. By a proper synchronization of their time-dependences, we inversely
engineered the tunneling and spin-orbit coupling matrix elements to achieve
spin transport with simultaneous single qubit rotations in quantum information
transformations such as the qubit preparation and $U_{\rm phase}$ and $U_{\rm
NOT}$ gates. In a chain of quantum dots, these transport+rotation operations
may be applied sequentially for a long-distance qubit transfer in a multi-dot
architecture, where the ability of a coherent spin transfer has been recently
demonstrated [24, 25]. Figure 3 demonstrates these processes for a particular
sequence starting in Dot 1 and ending in Dot 4. We point out that this
technique can also be applied to heavy-hole systems, where the control of the
hole spin via the tunneling and strong SOC has been demonstrated for silicon-
based double quantum dots [26]. In addition, a similar approach can be used to
design the spin and mass transport of cold atoms in optically produced
potentials [27].

We acknowledge this paper is supported by the China Scholarship Council(CSC),
by the National Natural Science Foundation of China (Grant No. 11474193), the
Shuguang program (Grant No. 14SG35), the program of Shanghai Municipal Science
and Technology Commission (Grants No. 18010500400 and No. 18ZR1415500), by the
Spanish Ministry of Economy, Industry and Competitiveness (MINECO) and the
European Regional Development Fund FEDER through Grant No. FIS2015-67161-P
(MINECO/FEDER, UE), and the Basque Government through Grant No. IT986-16.

##  Appendix A Four-level Hamiltonian of a double quantum dot

We consider a single electron in a double quantum dot modeled by a one-
dimensional Hamiltonian, as can be realized in nanowire-based systems [28],
where the electron is tightly confined in the perpendicular directions, as

$H_{\rm DQD}(t)=\frac{\widehat{p}^{2}}{2m}+V(x).$ (26)

Here the first term is the kinetic energy with
$\widehat{p}=-i\hbar\partial/\partial x$ and $m$ is the electron effective
mass (e.g., in GaAs $m\approx 0.067$ of the free electron mass). We assume
that $V(x)$ is a spatially symmetric potential ($V(x)=V(-x)$) with two
equivalent minima at points $x_{0}>0$ and $-x_{0}$ and choose the basis for
the tunneling-related Hamiltonian as two approximate states of this potential,
localized in the vicinity of the points $-x_{0}$ and $x_{0},$ which we will
denote as $\widetilde{\psi}_{L(R)}(x)$ [29, 30], respectively. The Hamiltonian
in the basis becomes

$H_{\rm tun}=\hbar\left[\begin{array}[]{cccc}0&\tau\\\
\tau&0\end{array}\right],$ (27)

where $\tau$ is the tunneling rate between the two quantum dots determined by
deviation of the total potential $V(x)$ from its shape in the vicinity of the
minima [31]. Thus, by modifying $V(x)$ by a time-dependent external field, one
can produce time-dependent $\tau(t).$

A magnetic field $\mathbf{B}=(0,0,-B)$ along the $z-$axis causes the Zeeman
spin splitting corresponding to the Hamiltonian
$H_{Z}=-g^{*}\mu_{B}\sigma_{z}B/2$, where $g^{*}$ is the conduction band Landé
factor, $\mu_{B}$ is the Bohr magneton, and the level splitting is
$\Delta=\mu_{B}|g^{*}B|$ [32]. In the basis of $\sigma_{z}-$representation,
the eigenstates of $H_{Z}$ are given by: $|\psi_{\uparrow}{\rangle}=(1,0)^{\rm
T}$ and $|\psi_{\downarrow}{\rangle}=(0,1)^{\rm T}$.

The one-dimensional Rashba spin-orbit coupling is represented as

$H_{\rm SOC}=\frac{\widetilde{\alpha}}{\hbar}\widehat{p}\sigma_{y},$ (28)

where the $\widetilde{\alpha}$ is the corresponding coupling parameter.

We define the full four-state basis of a single electron in the DQD as

$|L(R)_{\uparrow(\downarrow)}{\rangle}=|\psi_{L(R)}{\rangle}\otimes|\psi_{\uparrow(\downarrow)}{\rangle}$
(29)

and obtain nonzero coupling Rashba terms calculated as (with
$\widehat{k}\equiv\widehat{p}/\hbar$)

$\displaystyle{\langle}L_{\downarrow}|H_{R}|R_{\uparrow}{\rangle}$
$\displaystyle=$
$\displaystyle{\langle}\downarrow|\sigma_{y}|\uparrow{\rangle}{\langle}L|\widehat{k}|R{\rangle}=-i\widetilde{\alpha}{\langle}L|\widehat{k}|R{\rangle},$
$\displaystyle{\langle}R_{\downarrow}|H_{R}|L_{\uparrow}{\rangle}$
$\displaystyle=$
$\displaystyle{\langle}\downarrow|\sigma_{y}|\uparrow{\rangle}{\langle}R|\widehat{k}|L{\rangle}=-i\widetilde{\alpha}{\langle}R|\widehat{k}|L{\rangle},$
(30)

and the Hermitian conjugate terms. The diagonal elements are all zero because
${\langle}L|\widehat{k}|L{\rangle}={\langle}R|\widehat{k}|R{\rangle}=0$ and
${\langle}L|\widehat{k}|R{\rangle}=-{\langle}R|\widehat{k}|L{\rangle}$. We
finally find in the basis
$\\{|L_{\downarrow}{\rangle},|R_{\downarrow}{\rangle},|R_{\uparrow}{\rangle},|L_{\uparrow}{\rangle}\\},$
corresponding to the $\\{|1{\rangle},|2{\rangle},|3{\rangle},|4{\rangle}\\}$
basis of the main text

$H_{R}=\hbar\left[\begin{array}[]{cccc}0&0&\alpha&0\\\ 0&0&0&-\alpha\\\
\alpha^{*}&0&0&0\\\ 0&-\alpha^{*}&0&0\\\ \end{array}\right],$ (31)

where $\alpha\equiv-i\widetilde{\alpha}{\langle}L|\widehat{k}|R{\rangle}$.

For symmetric $V(x)$, the full Hamiltonian in the
$\\{|1{\rangle},|2{\rangle},|3{\rangle},|4{\rangle}\\}$ basis of the main
text, taking into account spin-conserving and spin-flip tunnelings acquires
the form

$H_{0}=\hbar\left[\begin{array}[]{cccc}-{\Delta}/{2}&\tau(t)&\alpha(t)&0\\\
\tau(t)&-{\Delta}/{2}&0&-\alpha(t)\\\ \alpha^{*}(t)&0&{\Delta}/{2}&\tau(t)\\\
0&-\alpha^{*}(t)&\tau(t)&{\Delta}/{2}\\\ \end{array}\right].$ (32)

The time dependence $\alpha(t)$ in (32) comes from two main sources: time-
dependent $\widetilde{\alpha}$ due to ac external bias and time-dependent
overlap of the wave functions localized near the left ($-x_{0}$) and right
($x_{0}$) minimum of the potential $V(x).$

## References

## References

  * [1] Loss D and DiVincenzo D P 1998 Phys. Rev. A 57 120
  * [2] Oosterkamp T H, Fujisawa T, van der Wiel W G, Ishibashi K, Hijman R V, Tarucha S, and Kouwenhoven L P 1998 Nature 395 873
  * [3] Hu X and Sarma S D 2000 Phys. Rev. A 61 062301
  * [4] Hu X and Sarma S D 2001 Phys. Rev. A 64 042312
  * [5] Bluhm H, Foletti S, Neder I, Rudner M, Mahalu D, Umansky V, and Yacoby A 2011 Nat. Phys. 7 109
  * [6] Veldhorst M, Hwang J C C, Yang C H, Leenstra A W, de Ronde B, Dehollain J P, Muhonen J T, Hudson F E, Itoh K M, Morello A, and Dzurak A S 2014 Nat. Nanotechnol. 9 981
  * [7] Nowack K C, Koppens F H L, Nazarov Y V, and Vandersypen L M K 2007 Science 318 1430
  * [8] McNeil R, Kataoka M, Ford C, Barnes C, Anderson D, Jones G, Farrer I, and Ritchie D 2011 Nature 477 439
  * [9] Baart T A, Shafiei M, Fujita T, Reichl C, Wegscheider W, and Vandersypen L M K 2016 Nat. Nanotechnol. 11 330
  * [10] Baart T A, Jovanovic N, Reichl C, Wegscheider W, and Vandersypen L M K 2016 Appl. Phys. Lett. 109 043101
  * [11] Li Y C, Martínez-Cercós D, Martínez-Garaot S, Chen X, and Muga J G 2018 Phys. Rev. A 97 013830
  * [12] Kang Y H, Huang B H, Lu P M, and Xia Y 2017 Laser. Phys. Lett. 14 025201
  * [13] Fabian J, Matos-Abiague A, Ertler C, Stano P, and Žutić I 2007 Acta Physica Slovaca, 57 565
  * [14] Nitta J, Akazaki T, Takayanagi H, and Enoki T 1997 Phys. Rev. Lett. 78 1335
  * [15] Sawada A, Faniel S, Mineshige S, Kawabata S, Saito K, Kobayashi K, Sekine Y, Sugiyama H, and Koga T 2018 Phys. Rev. B 97 195303
  * [16] Mal’shukov A G, Tang C S, Chu C S, and Chao K A 2003 Phys. Rev. B 68 233307
  * [17] Sadreev A F and Sherman E Ya 2013 Phys. Rev. B 88 115302
  * [18] Echeverría-Arrondo C and Sherman E Ya 2013 Phys. Rev. B 88 115302
  * [19] Thomas F 2014 IEEE. Trans. Robot. 30 1037
  * [20] Pérez-Gracia A and Thomas F 2017 Adv. Appl. Clifford Alg. 27 523
  * [21] Sommerfeld A 1949 Partial Differential Equations in Physics (Academic Press, New York) p 227
  * [22] Muga J G and Wardlaw D M 1995 Phys. Rev. E 51 5377
  * [23] Petta J R, Johnson A C, Taylor J M, Laird E A, Yacoby A, Lukin M D, Marcus C M, Hanson M P, and Gossard A C 2005 Science 309 2180
  * [24] Flentje H, Mortemousque P A, Thalineau R, Ludwig A, Wieck A D, Bäuerle C, and Meunier T 2017 Nat. Commun. 8 501
  * [25] Fujita T, Baart T A, Reichl C, Wegscheider W, and Vandersypen L M K 2017 NPJ Quantum Information 3 22
  * [26] Bogan A, Studenikin S, Korkusinski M, Gaudreau L, Zawadzki P, and Sachrajda A S 2017 Landau-Zener-Stückelberg-Majorana interferometry of a single hole arXiv: 1711.03492.
  * [27] Kartashov Y V, Konotop V V, and Vysloukh V A 2018 Phys. Rev. A 97 063609
  * [28] Nadj-Perge S, Frolov S M, Bakkers E P A M, and Kouwenhoven L P 2010 Nature 468 1084
  * [29] Li X, Barnes E, Kestner J P, and Das Sarma S 2017 Phys. Rev. A 96 012309
  * [30] Burkard G, Loss D, and DiVincenzo D P 1999 Phys. Rev. B 59 2070
  * [31] Ashcroft N W and Mermin N D 1976 Solid State Physics (Saunders College, Philadelphia)
  * [32] Taking into account that we consider system with $g^{*}<0$, we take $\mathbf{B}$ antiparallel to the $z-$ axis for consistency with the main text.