# A Modular Framework
for Reinforcement Learning Optimal Execution

Fernando de Meer Pardo
School of Engineering
Zurich University of Applied Sciences
Switzerland
<EMAIL_ADDRESS>
& Cristoph Auth
School of Engineering
Zurich University of Applied Sciences
Switzerland
<EMAIL_ADDRESS>
& Florin Dascalu
School of Management and Law
Zurich University of Applied Sciences
Switzerland
<EMAIL_ADDRESS>
corresponding author

###### Abstract

In this article, we develop a modular framework for the application of
Reinforcement Learning to the problem of Optimal Trade Execution. The
framework is designed with flexibility in mind, in order to ease the
implementation of different simulation setups. Rather than focusing on agents
and optimization methods, we focus on the environment and break down the
necessary requirements to simulate an Optimal Trade Execution under a
Reinforcement Learning framework such as data pre-processing, construction of
observations, action processing, child order execution, simulation of
benchmarks, reward calculations etc. We give examples of each component,
explore the difficulties their individual implementations & the interactions
between them entail, and discuss the different phenomena that each component
induces in the simulation, highlighting the divergences between the simulation
and the behavior of a real market. We showcase our modular implementation
through a setup that, following a Time-Weighted Average Price (TWAP) order
submission schedule, allows the agent to exclusively place limit orders,
simulates their execution via iterating over snapshots of the Limit Order Book
(LOB), and calculates rewards as the $ improvement over the price achieved by
a TWAP benchmark algorithm following the same schedule. We also develop
evaluation procedures that incorporate iterative re-training and evaluation of
a given agent over intervals of a training horizon, mimicking how an agent may
behave when being continuously retrained as new market data becomes available
and emulating the monitoring practices that algorithm providers are bound to
perform under current regulatory frameworks.

_Keywords_ Optimal Trade Execution $\cdot$ Reinforcement Learning in Finance

##  1 Introduction

An execution in an electronic financial market context is the process in which
a number of units of a publicly traded financial asset are bought or sold
through said market. Optimal Execution algorithms aim to carry out this
process by dividing the transaction into child orders in such a way that a
series of metrics, such as the transaction costs incurred, the order slippage
(the effect of order submissions on market prices) and/or the Implementation
Shortfall (IS) (the difference between market prices at the start and end of
the transaction, referred to this way because it represents the implementation
cost difference between ideal and realized portfolios) are optimized.

Modern electronic financial markets are based on trading pools that implement
a continuous double auction mechanism. Market participants are able to send
two types of orders. Limit orders represent an open interest to buy/sell at a
maximum/minimum price. If they fail to instantly lead to a transaction, they
are entered into a market structure called the Limit Order Book (LOB) that
stores all limit orders and that matches compatible incoming orders of the
opposite side based on a “first in, first out” queuing system specific to each
price-level [38, 1, 20, 33]. Market orders, on the contrary, represent an
interest to buy or sell at the best available price determined by the limit
orders currently in the LOB. Limit orders guarantee a price but not the
execution and provide the market with liquidity, allowing other market
participants the chance to trade at known prices. In contrast, market orders
guarantee the execution but not the price at which it is resolved therefore
removing liquidity from the market.

Since the profitability of any investment heavily depends on the effectiveness
of its trade execution, in some jurisdictions explicit rulings regulate the
obligation of financial service providers to perform optimal/best execution
for their clients. The main aim of these rulings is to prevent financial
service providers from collecting rebates on transactions by executing client
orders at more unfavorable prices that those present in the public markets,
but they also force providers to choose a method of execution if the client
has not explicitly done so. Examples of this are the US Financial Industry
Regulatory Authority (FINRA) Rule $5310$ [24] which states that abiding
members shall "buy or sell […] so that the resultant price to the customer is
as favorable as possible under prevailing market conditions" and the EU’s
Markets in Financial Instruments Directive (MiFID) Article 21 of the L1
Directive [50] which states that "firms should monitor on a regular basis the
effectiveness of their [best execution] policy and correct any deficiencies
identified".

In other jurisdictions, regulatory bodies issue guidance laying out
requirements and expected good practices regarding best execution, as well as
showcase common practices that fail to meet standards. Examples of this are
the Swiss Financial Markets Authority (FINMA) Circular 2018/1 [23], Hong
Kong’s Securities and Futures Commission (SFC) 2018 Best Execution Report
[51], and the Monetary Authority of Singapore’s (MAS) 2020 Notice on Execution
of Client Orders [45]. All of these directives encourage financial service
providers to put in place operational mechanisms to continuously monitor
execution quality, document phenomena such as execution delays, disclose
specific execution arrangements to clients, and designate specific committees
in which both management, regulatory compliance units, and the teams
responsible of carrying out client executions collaborate on policy oversight.

This set of obligations, explainability concerns, early results in the study
of the Optimal Execution problem (see Bertsimas & Lo [11] and Almgren & Chriss
[6]), and specially the fact that the monitoring of performed executions has
to be carried out via comparison against market benchmarks such as the Volume
Weighted Average Price (VWAP) are among the factors that have contributed to
the domination of a small number of well-known algorithms over the commercial
offering of execution services [2]. Examples include algorithms that attempt
to directly capture the VWAP benchmark such as Percentage-Of-Volume algorithms
that aim to participate in a given $\%$ of market volume [31], Time Weighted
Average Price (TWAP) algorithms that submit child orders of the same volume at
a constant rate, and algorithms that interact with the bid/ask spread [14].
All of these algorithms incorporate a relatively limited amount of market
information in their decision-making process and decide order submissions
based on relatively straightforward easily explainable rules.

###  1.1 Related Work

Bertsimas & Lo [11] and Almgren & Chriss [6] are perhaps the most influential
seminal works on the mathematical treatment of Optimal Execution. The authors
conceive the market price process as a realization of an arithmetic random
walk and model market impacts (both temporary and permanent) via a family of
functions whose parameters can be estimated with historical data. They obtain
the optimal strategy, defined as the strategy that minimizes the expected cost
of trading under their assumptions (and considering only market orders), via
Dynamic Programming and Stochastic Control respectively. Extensions to the
original approach that model additional market phenomena via stochastic
processes include [4, 5, 18, 25, 26, 47, 13]. Another branch of the literature
focuses instead on the modelling of the order book directly [27, 10, 44, 3,
46, 30], again via stochastic modelling and solved via stochastic control
techniques. The main drawback of the stochastic modelling paradigm is model
risk, i.e., the choice of model depends on stringent assumptions, which may
not hold in reality, on the nature of the market phenomena being modelled. For
example, assumptions such as the independence of returns and linear market
impact functions lead to the naïve TWAP (placing equally sized orders over the
execution) being the only optimal strategy, which is difficult to justify
across all markets/assets in a real setting. Furthermore, the chosen
functional forms of the stochastic processes used in models often lead to
unavailable analytic solutions that only allow for approximations, costly
parameter calibrations, and/or complex optimization problems which can become
computationally intractable and/or run into convergence challenges.

[42] was the first work to explore the application of data-driven
Reinforcement Learning by recursively employing Q-learning and dynamic
programming on High Frequency market data. Later, general advances in Deep
Learning led to a new era of Reinforcement Learning research in which Neural
Networks started being used as agents [39, 12], and hence emerged a new
Optimal Execution paradigm based on Deep Reinforcement Learning (DRL).
Contrary to stochastic modelling, Deep Reinforcement Learning allows for a
data-driven, model-free approach to Optimal Execution in which we obtain
strategies through learning optimal policies for sequential decision making by
optimizing a cumulative reward function. The only necessary assumption is the
Markov property of the system at hand, i.e., that the optimal action at any
given state of the environment is independent of previous actions, which is
appropriate for our setting since we can include the results of previous
actions as internal variables of the state. A multitude of different setups
have been explored in the DRL literature, including Q-learning [43, 36, 48,
16, 32, 28], Policy Gradient Methods [8, 37, 22, 48, 7, 16], and
$\text{Sarsa}(\lambda)$ [7], among others.

The advantages of the Deep RL paradigm include being able to obtain, by
learning from experience on historical market data, adaptable execution
policies that are less susceptible to front-running and avoid some aspects of
model risk. Model choices in DRL are also inevitable, but they are usually not
made based on assumptions about the nature of the underlying system to
optimize, but rather on the empirical performance of the combination of the
Neural Network Architecture used as agent and the RL optimization algorithm.
The main disadvantages include the black-box nature of DRL execution
strategies and the difficulty that financial simulations, that constitute the
training environment for policies, involve.

Most of the works in the literature focus on the Neural Network Architectures
used as agents and the RL optimization methods employed, as it is common in
other areas of RL research. However, one crucial characteristic of RL applied
in market applications is that the true environment is inaccessible and thus
it is necessary to simulate its behaviour, be it via historical data or data
generation approaches. This challenge constitutes an additional level of
choice compared to RL settings with access to their true environment.
Different setups differ not only by the NN agents and RL optimization methods
they employ, but also by the type of market simulation they implement (that
is, by the environment of their RL setup). The performance of a given agent +
optimization method does not necessarily generalize to different simulation
setups and the ability of a setup to replicate the behavior of a real market
is in itself a major challenge. The works mentioned above differ not only by
the RL optimization methods mentioned to categorize them, but also by the
multitude of choices they make in order to simulate executions.

###  1.2 Our Contributions

We focus on the environment/simulation setup aspect of RL Optimal Execution.
Few works openly share the implementation of their proposed setup along with
the training data used, and often the hyper-parameters employed during the
experiments are not fully specified, leading to the impossibility of
implementing the simulation setups and reproducing results. Reproducible
setups are routinely benchmarked against proposed novel setups but often
implementation choices are omitted and/or differ across implementations,
making comparisons between models biased. It is also common practice to
compare setups by only training and evaluating on fixed intervals of the
historical data whereas, in a real setting, models will likely be retrained
continuously as new market data becomes available over time. Since the
performance of RL agents fluctuates between different market periods and the
effect that each of the different components of a setup has on the results can
be difficult to ascertain, being able to easily implement different setups,
train agents on them, and deploy monitoring protocols of agent behavior on
novel data are necessary steps to successfully exploit RL-based Optimal
Execution algorithms in real settings. However, the limited availability of
documentation and code implementations of setups presents a barrier for entry.

This work aims to lower this barrier of entry by showcasing a modular RL
framework that eases the implementation of different environments/simulation
setups by separately implementing the specific simulation choices of each
setup through different modules. Our work can be of interest to both
practitioners and academics since it provides a basis that can vertebrate and
serialize the implementation of all the aspects of RL Optimal Execution,
allowing for a more uniform way to implement, benchmark and monitor different
simulation setups.

In Section 2, we give a short introduction to RL and introduce the framework
by describing each module and specifying the functionalities it should
incorporate. We implement each module as a separate Python class and structure
them as depicted in Figure 2. We discuss the difficulties that the
implementation of each individual module entails, the conflicts arising from
their dependencies and their interactions with other modules.

In Section 3 we showcase the usage of the modular implementation by
implementing a setup intended to enhance a TWAP algorithm via RL, in the
spirit of Bertsimas & Lo [11] and Almgren & Chriss [6]. The setup lets the RL
agent decide the volume of limit orders submitted following the order
submission schedule of a TWAP algorithm and imposes a bucket scheduling logic:
At the beginning of execution, it divides the execution horizon in a number of
periods/buckets and assigns volume to be executed proportionally to each
bucket. At the end of each bucket, the remaining unexecuted volume is
submitted as a market order. The reasoning behind following this kind of order
scheduling logic, which is common practice in many algorithms, is that we
minimize the market risk, and thus the probability of a very unfavorable
execution. The execution of each limit order is simulated via iterating over
snapshots of the limit order book (LOB) and rewards are calculated as the $
improvement over the price achieved by the TWAP benchmark algorithm subject to
the same bucket scheduling logic over the same time horizon. As already
mentioned, these simulation choices make for a training environment that is
different from a real market, so we discuss the explicit and implicit
differences that our particular implementation induces in the simulation, and
emphasize the importance of considering this phenomenon, the training-
deployment distribution shift, as an essential aspect of Optimal Execution RL-
based algorithm development.

Finally, in Section 4, we showcase two different experiments. In the first
experiment, we iteratively re-train and evaluate different models over a set
of contiguous 5-day intervals of market data, mimicking how models may behave
out of sample when being continuously retrained and then immediately deployed
as new market data becomes available, thus providing a closer estimation of
live model performance. Alternatively, in the second experiment, we
investigate whether a model’s out of sample performance decays over time, to
estimate the consequences of not re-training a model.

##  2 A Modular Implementation for RL Optimal Execution

In this Section, after a brief introduction to RL, we introduce a modular
framework that allows for the separate implementation of all the different
functionalities and internal processes that an Optimal Execution RL
environment needs to incorporate. The core modules we will introduce are the
Data Feed, the Broker and the Execution Algo, see Table 1 for a summarized
description of their functionalities and Figure 2 for an illustration of their
dependency structure.

Class Name | Functionalities | Required Methods | Dependency
---|---|---|---
Data Feed | Data retrieval and pre-processing | reset() |  Broker attribute
Execution Algo | Execution Logic | reset() |  Broker attribute
Broker | Order Execution Simulation |  reset() & simulate_order() | environment attribute
gym environment | RL logic |  reset() & step() | None
Table 1: Proposed modules of the Optimal Execution RL framework.

###  2.1 Introduction

Reinforcement Learning is a Machine Learning framework in which the objective
is to learn rules for iterative decision making so that a specific numerical
quantity, called a reward, is optimized. The algorithm tasked with obtaining
rules is called the agent; it receives as input the state of the world $S_{t}$
via another algorithm, called the environment. The agent maps states to
actions via a special function called the policy. Given an action $A_{t}$
taken by the agent, the environment produces an updated value of the reward
$R_{t+1}$ as well as a new state of the world $S_{t+1}$, see Figure 1 for an
illustration. The main difference between RL and supervised learning problems
is that the new states of the world may be affected by the agent’s actions
(i.e. there is interaction between the components of the problem).

AgentEnvironment$\begin{array}[]{l}Action\\\
A_{t}\end{array}$$\begin{array}[]{l}Reward\\\
R_{t}\end{array}$$\begin{array}[]{l}State\\\
S_{t}\end{array}$$R_{t+1}$$S_{t+1}$ Figure 1: Components and flows of the RL
framework.

More formally, the RL framework is characterized as a Markov Decision Process
(MDP). The MDP framework describes the agent–environment interaction through
the three signals mentioned above, the actions of the agent $\\{A_{t}\\}_{t}$,
the states of the environment $\\{S_{t}\\}_{t}$ and the rewards
$\\{R_{t}\\}_{t}$. An MDP is described by a sequence of discrete time steps
$t_{0},...,T$ and a tuple
$\left\\{\mathcal{S},\mathcal{A}(S),\mathbb{P}\left(S^{\prime}\mid
S,A\right),R,\gamma\right\\}$ in which:

  1. 1.

$\mathcal{S}$ is the set of states so that each $S_{t}\in\mathcal{S}$.

  2. 2.

$\mathcal{A}(S)$ defines possible actions $A_{t}\in\mathcal{A}(S)$ that can be
taken in a state $S_{t}=S$.

  3. 3.

$\mathbb{P}\left(S^{\prime}\mid S_{t-1},A_{t-1}\right)$ represent the
transition probabilities of states $S^{\prime}\in\mathcal{S}$ given action
$A_{t-1}$ is chosen at a previous state $S_{t-1}$.

  4. 4.

$R$ and $\gamma$ represent the set of all possible rewards and the discount
factor, which determines how much more desirable are immediate rather than
latter rewards.

The transition probabilities have to fulfill the Markov Property: Given
$\mathbb{P}\left(S_{t}\mid
S_{0:t-1}\right)=\sum_{A_{t}}\mathbb{P}(A_{t})\mathbb{P}\left(S_{t}\mid
S_{0:t-1},A_{t}\right)$ (in integral form if the action space is continuous),
we have that:

$\mathbb{P}\left(S_{t}\mid S_{0:t-1}\right)=\mathbb{P}\left(S_{t}\mid
S_{t-1}\right)$ (1)

That is, the probabilities of arriving at any state $S_{t}$ depend exclusively
on the previous state $S_{t-1}$. The goal of an agent is to find a policy
$\pi$ that maximizes the expected discounted cumulative reward:

$\operatorname*{arg\,max}_{\pi}\mathbb{E}^{\pi}\left[\sum_{t=0}^{T-1}\gamma^{t}R_{t}\left(S_{t},A_{t}\right)\right]$
(2)

for which there exist multiple optimization methods [52, 19].

Figure 2: Modules Dependency Structure.

###  2.2 The Data Feed Class

Datasets of High-Frequency (HF) market data can come in many different
formats, with different levels of granularity, that allow for different
simulation setups. They can be broadly categorized (although there are
exceptions in specific markets) into what is commonly referred to as "Levels":

  1. 1.

Level 1 data: Consists of records of traded prices at different timestamps. It
may include the very first levels of the LOB, the best bid/ask and their
respective volumes as well.

  2. 2.

Level 2 data: Extends Level 1 data by including deeper levels of the LOB, that
is, the price levels greater/lower than the lowest ask/highest bid
respectively and their volumes. This is usually referred to as "snapshots" of
the LOB. They may have a fixed size or change at each timestamp. Often price
levels very far away from the best bid/ask are not recorded whereas in a real
market this can be a possibility.

  3. 3.

Level 3 data: Consists of the sequence of limit order placements, deletions
and executions. It is the most granular out of the three types, since the
other two can be reconstructed from it, and can lead to large datasets because
of this.

Besides the data format, the regularity at which each datapoint is recorded
also plays a very big role later on in the simulation. It can range from
having some form of HF data sampled at a constant rate to having a datapoint
for each order placement, deletion, execution, and other market events
(volatility halts, circuit breakers etc.). Data accessibility can vary between
different markets, and more granular data is often either pay-walled or
unavailable. Thus, being able to process different data formats is a
necessity. Because of this, we code Data Feeds as Python Subclasses, each with
different methods adapted to their specific data format.

Each Data Feed Subclass needs to be able to retrieve data iteratively in order
to allow the Broker to produce observations for the agent in the environment’s
step function. If we are dealing with a dataset with entries ordered by
timestamps, one of the easiest ways to do this is to have an internal state
that represents the current time of the simulation and signals to the next
datapoint to be retrieved from the data. One thing to keep in mind is that the
timestamps of order placements may not necessarily align with the timestamps
of the data entries, and thus it becomes necessary to incorporate logic in the
Data Feed that finds the closest datapoint after a given timestamp. In the
same way the environment incorporates a reset function called at the beginning
of each Episode, each Data Feed Subclass needs a reset function that will be
called simultaneously and reconfigures its internal state for the next
execution.

In addition, some setups simulate synthetic markets through artificial data
[15, 9, 17]. All of this logic can be accommodated as methods of a Data Feed
subclass in the same fashion as we do with historical data and the internal
state modified to suit the needs of the specific algorithm.

###  2.3 The Broker Class

An RL agent needs to be able to interact with its environment, the market
under an Optimal Execution setting, during the training process. In order to
implement this feedback mechanism between the market and the agent’s actions,
translated as order submissions, we need to be able to simulate order
executions. We call this module the Broker Class since it is responsible of
placing orders into the market, simulating their execution and keeping logs of
the trades carried out over the entire trading horizon.

When presented with an order, the Broker has to be able to simulate its
execution through an algorithm of choice, that may be different for each
Execution Algo, by interacting with its Data Feed. Each algorithm may react
differently to partial executions, thus the Broker needs to incorporate as
methods each of the different ways Execution Algos may act mid-execution.
Examples of these choices are submitting all orders as market orders, placing
limit orders but deleting them and replacing them with a price closer to the
best bid/ask after they have remained in the LOB for a given time, combining
market and limit orders simultaneously at each order submission etc.

The Broker needs to be able to process orders originating from different
Execution Algos, records like the trade logs, the LOB histories and others
need to be implemented as dictionaries whose keys represent the Execution
Algos being simulated. Since the Broker is responsible of keeping track of all
the execution related variables at any point of the simulation, it needs to be
able to modify the internal state of the corresponding Execution Algos, so
setting them as Broker attributes is the most straightforward way of
implementing this dependency. See Figure 3 for an illustration of a typical
Broker simulation workflow.

Just as the environment and the Data Feed, the Broker needs to include a reset
function that, when called at the beginning of each episode, will clear all
its internal logs, reset its Data Feed according to the new execution starting
time and reset its Execution Algos according to the new execution parameters.

Figure 3: Typical Broker simulation workflow. Each order placement and its
corresponding simulation represent a step of the RL agent.

###  2.4 The Execution Algo Class

Each Execution Algorithm divides a transaction into child orders across the
execution interval according to some internal logic. The order submission
schedule and each order’s volume may be decided a priori (like a TWAP
algorithm), may be determined via the algorithm’s internal state and/or market
conditions during the execution (like a RL algorithm) or may incorporate a
combination of both (like an algorithm that incorporates a buckets schedule).

The Execution Algo class encodes a representation of each execution algorithm
consisting of its actions indexed by their timestamps (algo_events), the
variables involved in each action (volume, type of order etc.), as well as
variables involved in the particular logic of the algorithm and an internal
state containing the execution parameters such as the trading direction, the
total volume to execute or the volume left to execute.

Just as the two classes before, each Execution Algo needs to also include a
reset function that, when called at the beginning of each episode, will reset
all the internal variables involved in the algorithm’s logic and reset its
execution parameters.

###  2.5 The gym environment Class

The environment is the core of the RL methodology, it is tasked with
processing the agent’s actions, calculating the corresponding rewards and
producing subsequent states. An environment can be implemented through the
specification of two required methods, the reset & step functions that carry
out the tasks previously described and specify the dimensions & types
(continuous or discrete) of the Observation and Action Spaces that the agent
will receive Observations and be allowed to draw actions from respectively.

####  2.5.1 Executions as Episodes

In a RL Optimal Execution setting we consider each Execution to be a different
training Episode. At the start of each Episode, the reset function is called
and the execution parameters are randomized. All setups incorporate the
following execution parameters:

  1. 1.

start_time: Timestamp of the beginning of the execution.

  2. 2.

exec_time: Duration of the current execution.

  3. 3.

trade_direction: Whether the transaction is a Buy/Sell.

  4. 4.

volume: The number of units of the asset to Buy/Sell in the Execution.

In addition to setup-specific parameters. Additionally, the reset function has
to reset the reward and info attributes, as well as the execution-specific
attributes of the environment, that is, the Broker and through its reset
function its Execution Algos and its Data Feed. It also needs to produce the
first state for the agent to process at the beginning of each Episode.

####  2.5.2 Order placements as steps

The step function is called when the agent produces an action after being
presented with a state of the environment. In a RL Optimal Execution setting,
it needs to incorporate the necessary logic to accomplish the following:

  1. 1.

Translate the agent’s actions into order submissions, determining its
submission type, order type and volume through interacting with the RL
Execution Algo.

  2. 2.

Simulate via the Broker the Execution of the orders submitted by all the
Execution Algos involved in the reward calculation up until the timestamp of
the next agent’s action.

  3. 3.

Calculate rewards at specific timestamps of the execution that depend on the
setup. Some setups reward the agent at every order submission, some at
specific timestamps, some only at the end of the execution etc.

Each environment needs to calculate rewards via interacting with the
trade_logs of the Broker. Different reward calculations can incentivize
different aspects of the RL Execution, not only the average price achieved,
but also the speed, the slippage incurred, the market vs limit order volume
ratio etc. As in all RL setups, the step function needs to also produce the
subsequent state the agent will be presented with via the Data Feed and
ascertain whether the current episode is over. See Figure 4 for an
illustration of the workflow of the step function.

Figure 4: Typical workflow for the environment step function.

###  2.6 UnitTests

For the most part the implementation of each of the modules described above
can be rather intuitive, but given that our aim is to implement numerous
different setups dealing with different data sources and performing different
calculations, it is advisable to implement a series of Unit Tests on each
implementation. Common aspects to check for include:

  1. 1.

Reproducibility: Given two Execution Algos with equal representations (meaning
they have equal algo_events and volumes_per_trade), they should produce
identical trade_logs when simulated by identical Brokers (meaning they share
all the attributes and thier Data Feeds have been reset to the same
timestamp). Failing to pass this test can mean that the Data Feed is
permanently modifying historical data and/or that the reset functions of some
modules are not properly implemented.

  2. 2.

Duplicity: Each datapoint from the historical data should be interacted with
strictly once. Failing to pass this test can mean that the Data Feed, or
modules that make use of it, are incorrectly iterating through the historical
data.

  3. 3.

Rounding Issues: At the end of each Execution, it is important to verify that
the executed volume accounted for in the trade_logs is equal to the initial
volume of the Execution. All calculations that involve market variables should
be done with Python Decimals rather than floats, in order to avoid rounding
errors and keep the rounding to the minimum tick size of the given market
consistent across the whole simulation. Failing to pass this test can mean
that the execution of some orders is not correctly implemented (for example,
unlike in a real market, submitting a market order vs a LOB snapshot does not
guarantee a full execution, since the LOB may not have enough volume to fully
consume the order).

For efficiency reasons, all the UnitTests should be implemented outside the
implementation of the environment, since they will introduce overhead to the
RL training. As a final note, it is important to implement each of the modules
described in this section with efficiency in mind, trying to avoid unnecessary
computations as much as possible and taking advantage of any achievable
efficiency improvements. An inefficient implementation of any of the
components can effectively make the training of a setup unmanageable in
practice.

##  3 Setup Example

In this section, we present an example implentation of an Optimal Execution
setup through our modular framework. We share the Python code implementation
in 111https://github.com/FernandoDeMeer/RL_Optimal_Execution. The setup we
choose to showcase aims to improve a TWAP benchmark algorithm via RL: we only
allow the agent to submit orders at the same timestamps as the TWAP benchmark
and to choose only the order volumes as 80%,100% or 120% of the TWAP volumes
through its actions. The prices of limit orders are set to be 1 tick away from
the best bid/ask and simulated according to a procedure we will shortly
present. Because of this, the RL agent is subject to a series of restrictions
that we implement via specific subclasses of the modules described in Section
2.

###  3.1 Data Feed Example

We use a dataset of Level 2 HF data consisting of snapshots of the LOB of
BTC/USDT futures of Binance, a crypto-currency exchange, up to 10 price levels
away from the best bid/ask for each side (adding up to 20 levels every
snapshot)111See: https://github.com/binance/binance-public-
data/tree/master/Futures_Order_Book_Download and recorded every 100 ms from
01/06/2021 to 20/06/2021. We have different files for each day, therefore we
partition the training and evaluation periods by days.

The snapshots are indexed by timestamps and thus make for a straightforward
implementation of the Historical Data Feed SubClass. The SubClass only needs
to:

  1. 1.

Load the dataset from source files and store it as an attribute.

  2. 2.

Implement a reset method that locates the closest LOB to a given timestamp and
records its position in the dataset.

  3. 3.

Fetch past LOBs at a given timestamp and iterate through the data in order to
allow the Broker to simulate order executions and build states for
environment.

Algorithm 1 Simulation of a Limit Buy Order Execution

Input: Limit Order $\mathcal{O}_{l}$, Data Feed $\mathbb{D}$, Execution Algo
$\mathbb{A}$, current timestamp $t$, market tick_size
Output: Execution of order $\mathcal{O}_{l}$ recorded in Broker.trade_logs

1:while $\mathcal{O}_{l}\text{{.volume}}>0\And
t<\mathbb{A}.\texttt{{next\\_event.time}}$ do:

2: $\mathcal{O}_{l}$.price =
$\mathbb{D}.\texttt{{current\\_LOB.get\\_best\\_bid()}}-\texttt{{tick\\_size}}$

3: Update $\mathbb{D}$ $\Rightarrow$ $\mathbb{D}.\texttt{{next\\_LOB()}}$

4: Update $t$ $\Rightarrow$ $t=\mathbb{D}.\texttt{{current\\_LOB.time}}$

5:
$\texttt{{trade\\_log}}=\texttt{{place\\_order}}(\mathcal{O}_{l},\mathbb{D}.\texttt{{current\\_LOB}})$,
append to Broker.trade_logs

6: if $\texttt{{trade\\_log.message}}==trade$ then

7: Update $\mathcal{O}_{l}\text{{.volume}}$ $\Rightarrow$
$\mathcal{O}_{l}\text{{.volume}}\mathrel{-}=\texttt{{trade\\_log.volume}}$

8: Update $\mathbb{A}\text{{.{remaining\\_volume}}}$ $\Rightarrow$
$\mathbb{A}\text{{.{remaining\\_volume}}}\mathrel{-}=\texttt{{trade\\_log.volume}}$

9: if $\mathcal{O}_{l}\text{{.volume}}=0$ then

10: break

11: if
$t\geq\mathbb{A}.\texttt{{next\\_event.time}}\And\mathbb{A}.\texttt{{next\\_event}}==\texttt{{limit\\_order}}$
then

12:
$\mathbb{A}.\texttt{{next\\_event.volume}}\mathrel{+}=\mathcal{O}_{l}\text{{.volume}}$

13: else

14: if $t\geq\mathbb{A}.\texttt{{next\\_event.time}}$ then

15: Submit a market order of $\mathbb{A}\text{{.{remaining\\_volume}}}$

Algorithm 2 Simulation of a Market Buy Order Execution

Input: Market Order $\mathcal{O}_{m}$, Data Feed $\mathbb{D}$, Execution Algo
$\mathbb{A}$, current timestamp $t$
Output: Execution of order $\mathcal{O}_{m}$ recorded in Broker.trade_logs

1:while $\mathcal{O}_{m}\text{{.volume}}>0$ do

2: Update $\mathbb{D}$ $\Rightarrow$ $\mathbb{D}.\texttt{{next\\_LOB()}}$

3: Update $t$ $\Rightarrow$ $t=\mathbb{D}.\texttt{{current\\_LOB.time}}$

4:
$\texttt{{trade\\_log}}=\texttt{{place\\_order}}(\mathcal{O}_{m},\mathbb{D}.\texttt{{current\\_LOB}})$,
append to Broker.trade_logs

5: Update $\mathcal{O}_{m}\text{{.volume}}$ $\Rightarrow$
$\mathcal{O}_{m}\text{{.volume}}\mathrel{-}=\texttt{{trade\\_log.volume}}$

6: Update $\mathbb{A}\text{{.{remaining\\_volume}}}$ $\Rightarrow$
$\mathbb{A}\text{{.{remaining\\_volume}}}\mathrel{-}=\texttt{{trade\\_log.volume}}$

###  3.2 Broker Example

As discussed in Section 2.2, there exists a spectrum of HF datasets with
different granularities. More granular datasets allow for simulations of order
executions that incorporate more market information. In our setup, we have
Level 2 Data in the form of LOB snapshots and thus can only use this
information to simulate the market response to order placements. When using
order book level data to simulate executions, it is advisable to employ a LOB
Class implementation that efficiently carries out the basic interactions
between order books and orders (placements, deletions and executions). In our
case, we use [21] which implements price levels as linked lists (by time of
arrival) and the Bids and Asks of each LOB as 2 red-black trees.

Algorithm 1 contains the pseudo-code of our chosen simulation of the execution
of limit buy orders. The volume of the limit order is decided by the RL agent
and the price is chosen as the best bid of the current LOB -1 tick (+1 in case
of a sell order). Then, the order is matched against the next LOB snapshot. We
record the placement and, if it leads to a trade (when the market moves
downwards w.r.t to the last LOB), we update the remaining volume of the
Execution Algo submitting the order accordingly and repeat the process with
the subsequent LOBs until either the volume of the order is fully consumed or
we reach the next order placement, in which case we add the remaining volume
to that of the new limit order, or reach the end of the Bucket, which submits
the remaining volume as a market order. In a real setting, this update of the
price of the standing order every snapshot would require cancelling it and
resubmitting a new order, but unlike in our simulation, the order could get
(partially) consumed between snapshots. This simulation approach executes only
after favorable market movements and thus incorporates the delay in execution
that limit orders entail.

Algorithm 2 contains the pseudo-code of our chosen simulation of the execution
of market buy orders. Unlike in a real market, when placing a market order in
a LOB snapshot, the market order might not be fully executed because the LOB
might lack enough volume on the corresponding side to do so, since the
snapshots carry only the 10 closest price levels to the best bid/ask. Because
of this, it also becomes necessary to iterate through LOBs in the execution of
market buy orders.

The environment of our setup calculates rewards based on the outperformance of
the RL agent w.r.t a TWAP algorithm following the same order submission
schedule. Because of this, we carry out the two previous Algorithms in
parallel in our simulation, that is, both the RL and the benchmark TWAP
algorithm have access to the same LOBs when placing orders; their actions do
not affect the other in any way. This is justified because we are not
interested in having both Algorithms compete, but rather just obtaining a
meaningful signal to train the RL Algo with.

By simulating the execution of limit and market orders via Algorithms 1 and 2
we are inducing the following phenomena, which distance the behaviour of our
simulation from that of a real market:

  1. 1.

Underestimating volume: Since we have only LOB snapshots, we are missing all
the order placements and executions that took place between snapshots and thus
may be underestimating the volume available in the market.

  2. 2.

Overestimating volume: We have no way of knowing whether the volumes appearing
at two LOBs with different timestamps originate from the same order placements
or not, thus we might be executing the same volume more than once and thus may
be overestimating the volume available in the market.

  3. 3.

Lack of LOB Depth: since the LOB snapshots we have available have always the
same size, we have no knowledge of the orders placed 10 levels away from the
best bid/ask. This makes it so that we ignore volumes placed at very
unfavourable price levels for our agent, specially for the execution of market
orders.

  4. 4.

No market impact: Since we are simulating via a static historical dataset, we
are not currently incorporating into the simulation the reaction that other
market participants would have to the actions of our agent, which would shift
the behaviour of a real market.

  5. 5.

Transaction fees: We are not taking into account the exchange fees incurred in
by our transactions. In Optimal Execution, fees may have a big impact on the
performance of an algorithm, specially if they’re charged on a per-order basis
such as market access fees.

  6. 6.

Latency: Contrary to most setups, we do introduce a delay for our agent’s
actions, since order submissions are executed against future LOBs rather than
the one corresponding to the observation that prompted the order submission.
However, we still assume that our submission will take less than 100ms to
reach the market, which may or mat not be realistic depending on the market at
hand.

Issues 1-3 can be improved via having access to more granular datasets (the
sequence of order placements, deletions and executions specifically) in
settings where they are available. Issues 5 & 6 require setup and market-
specific knowledge to improve but they are relatively easy to work around.
Incorporating Issue 4 requires a model-based approach and is the focus of an
active branch of the financial simulations literature, see [41, 34, 15, 29].

###  3.3 Execution Algo Example

We implement both the RL and the benchmark TWAP Algorithm as Execution Algos.
The TWAP is implemented via an order submission schedule that divides the
execution interval into segments referred to as Buckets, which are assigned
volume proportionally according to their duration. Each bucket populates the
algo_events of the Execution Algo with the timestamps of a given number of
equally distributed limit_orders along its duration and adds a last timestamp
to represent the bucket_bound (which signals the submission of all the
remaining bucket volume as a market order). See Figure 5 for an illustration
of this order submission schedule.

Since both types of events represent order submissions and the volumes of the
market orders depend on the remaining volume (which depends on market
movements), we only need to keep track of the limit order volumes through a
list we call volumes_per_trade and the list of either the limit_orders’ or the
bucket_bounds’ timestamps which allows us to check whether each event leads to
a limit order or a market order submission. The RL Algorithm is implemented
through inheriting all the attributes of the TWAP (since they share the order
submission schedule) and modifies during training its volumes_per_trade list,
depending on the agents actions.

Figure 5: TWAP order submission schedule vs executed trades following
Algorithms 1 and 2. Figure 6: RL order submission schedule vs executed trades
following Algorithms 1 and 2. Keep in mind that the RL agent decides its
schedule over the course of training by adding volume to running orders,
unlike the TWAP schedule which is decided a-priori without taking market
conditions into account.

###  3.4  gym environment Example

We implement the environment of our setup via OpenAI’s gym library [12], by
specifying its reset and step functions and a number of other submethods that
implement specific functionalities. We choose a Discrete action space of size
$n=3$ for the agent, representing the choice of submitting an additional
volume to the limit order being currently executed (see Algorithm 1)
$\\{0.8,1,1.2\\}*\text{TWAP}_{Volume}$ for actions $\\{0,1,2\\}$ respectively.
We also choose an Observation space of size $n=102$ consisting of a
concatenation of the volumes and normalized prices of the 10 closest price
levels in both the bids & asks belonging to the past 5 LOB snapshots at the
current timestamp (prices are normalized w.r.t to the mid-price of the latest
observation) as well as two internal variables, the $\%$ of the bucket volume
left to execute and the number of orders remaining inside the current bucket.
In addition, the environment is initiated with a Broker as an attribute which
in turn incorporates a Data Feed as described in previous sections.

####  3.4.1 reset function Example

In addition to resetting the attributes already mentioned in Section 2.5.1,
our setup reset function has to sample TWAP-specific attributes such as the
no_of_slices, rand_bucket_bounds_width, bucket_func and delete_vol, which
specify different aspects of the order submission schedule, reset the,
state_idx, order_idx and bucket_idx attributes that represent internal indexes
of the environment used to contextualize specific order submissions (such as
market orders at bucket bounds), declare the new RL and TWAP benchmark
Execution Algos with the attributes resampled above and build the first
observation of the Episode for the agent.

####  3.4.2 step function Example

The step function of our setup implements the logic that translates the agent
actions into order submissions via interacting with the Execution Algo,
simulates order submissions until the next algo_event via the Broker and
Algorithms 1 & 2, creates new states by interacting with the Data Feed and
calculates rewards at the end of each bucket as the total $ difference,
weighted by the volume executed, between the price achieved by the RL
Algorithm vs that of the TWAP Benchmark:

$R_{t}(S_{t},A_{t})=\mathbbm{1}_{t\in
T^{*}}(\text{RL}_{VWAP}-\text{TWAP}_{VWAP})$ (3)

In the case of selling and with opposite sign in the case of buying, where
$T^{*}=\\{t_{1},t_{2},...,t_{N}|t_{i}\text{ is a bucket bound}\\}$ is the set
of all the bucket bounds timestamps. The function follows the same workflow as
Figure 4. See Figure 6 for an illustration of an order submission schedule
decided by the RL agent.

##  4 Experiments

In this section, we showcase two different experiments aimed at training
agents and estimating their out of sample performance, carried out under the
setup described in Section 3. In the first, we compare the out of sample
performance of two agents continuously retrained over a set of 5-day periods
in order to emulate an iterative training procedure that incorporates new
market data as it becomes available. In the second, we investigate whether a
model’s out of sample performance decays over time if it is not continuously
retrained. We use Ray’s RLlib [35, 40], an open-source Python library, to
implement our experiments.

Figure 7: LSTM agent Architecture.

Figure 8: FCN agent Architecture.

###  4.1 Iterative Training Experiment

We compare two different agents: the first architecture incorporates two Dense
layers of 128 units each with ReLU activations followed by a LSTM layer with
256 units and two output dense layers of dimensions 3 and 1 for the action
logits $a_{t}$ and the value function $V_{t}$ respectively. We refer to this
architecture as the LSTM agent (see Figure 8 for reference). The second
architecture incorporates two Dense layers of 256 units each with tanh
activations and the same output layers as before. We refer to this
architecture as the FCN agent (see Figure 8 for reference). We train both
agents via the PPO Algorithm [49], one of the most popular policy-gradient
algorithms, due to its state-of-the-art performance in numerous benchmarks and
sample efficiency. We task both agents with buying 100 BTC over a 5-minute
window and benchmark them against a TWAP that divides the execution interval
into 10 buckets of 30s each and submits 9 limit orders in each bucket. Each
agent is able to choose the volumes of its orders (but not the prices and
type) as described in Section 3.

We train both agents over 3 different training periods, from 01/06/2021 to
05/06/2021, from 06/06/2021 to 10/06/2021 and from 11/06/2021 to 16/06/2021
and evaluate them over the 5 days following each period. For the second and
third training periods, we start training with the best agent from the
previous period. We train each agent over each training period for 125 PPO
iterations, which account to approx. 1M steps of the environment, see Figure 9
for a plot of the Average Reward per Iteration over the course of training.
The FCN agent manages to gradually improve its Average Reward over the first
training instance (first 125 Iterations) and then plateaus from then onward,
while the LSTM does not achieve the same improvement and stays with an average
reward centered around 0 during the whole training process.

Figure 9: Smoothed Average Reward per PPO Iteration of both agents over the
course of training, the FCN agent is plotted in blue and the LSTM agent in
orange.

After training, we evaluate each model over its corresponding evaluation
period for $10^{4}$ Episodes with the same execution parameters. The results
are summarized in Table 2. We can observe that the LSTM agent barely
outperforms the FCN over the first training period, but underperforms it more
significantly over the other two. In addition, the FCN agent progressively
improves its out of sample performance over the three periods after being
trained with more data, while the LSTM agent does not. Both agents barely
improve the TWAP benchmark out of sample, given that the $ outperformance
difference is reduced compared to the price of the asset (BTC had an average
price close to 35k USD during the train/eval periods). This finding
experimentally supports the results of [6, 11] which showed the TWAP to be the
optimal execution algorithm under random walk dynamics. Both agents have
trouble significantly generalizing the experience they have learned during
training to a new market period because the environment presents a dynamic
close to a random-walk and thus a low signal-to-noise ratio. Nonetheless, the
FCN agent consistently achieves to outperform out of sample. It is however
important to remember that our setup severely restricts the freedom of both
agents. Allowing them to trade more freely, across different timestamps, and
with more options for choosing prices, order types, and distributing volumes,
could perhaps allow for a greater outperformance (likely with a bigger
variance). Additionally, our limit order execution Algorithm 1 induces a delay
in order execution, only executing volume if the market moves in our favor.
Choosing a less restrictive simulation of order executions could also lead to
easier dynamics for the agents to learn, but may not be as representative of a
real market environment.

| Mean $\pm$ std
---|---
Training Period | 01/06/2021-05/06/2021 | 06/06/2021-10/06/2021 | 11/06/2021-16/06/2021
Evaluation Period | 06/06/2021-10/06/2021 | 11/06/2021-16/06/2021 | 16/06/2021-20/06/2021
Agent Name |  |  |
LSTM agent | 0.0616 $\pm$ 0.48 | -0.0080 $\pm$ 0.33 | 0.0418 $\pm$ 0.34
FCN agent | 0.0506 $\pm$ 0.46 | 0.0785 $\pm$ 0.39 | 0.1087 $\pm$ 0.40
Table 2: Out-Of-Sample performance of the iteratively trained models measured
as the mean of the $ difference between the RL achieved price (VWAP) vs that
of the TWAP benchmark. Dates are in dd/mm/yyyy format.

###  4.2 Performance Decay Experiment

In order to estimate the out of sample performance time decay of a model, we
take the FCN agent of the previous experiment trained from 01/06/2021 to
05/06/2021 and evaluate it on the two other evaluation periods, from
11/06/2021 to 16/06/2021 and from 16/06/2021 to 21/06/2021. As in the previous
experiment, we evaluate the agent for $10^{4}$ Episodes on each period, with
the same execution parameters as during training. The results are summarized
in Table 3. We can observe how, unlike its iteratively trained counterparts,
the agent trained only once does not improve its performance on the later
periods but rather performs similarly, not significantly decaying.

| Mean $\pm$ std
---|---
Training Period |  | 01/06/2021-05/06/2021 |
Evaluation Period | 06/06/2021-10/06/2021 | 11/06/2021-16/06/2021 | 16/06/2021-20/06/2021
Agent Name |  |  |
FCN agent | 0.0506 $\pm$ 0.46 | 0.0586 $\pm$ 0.42 | 0.0566 $\pm$ 0.34
Table 3: Out-Of-Sample performance of a model trained on a single period
measured via the mean of the $ difference between the RL achieved price (VWAP)
vs that of the TWAP benchmark. Dates are in dd/mm/yyyy format.

###  4.3 Hardware Considerations

For training and testing, we used a Nvidia DGX station equipped with a
40-thread CPU, 4x Tesla V100 32GB GPUs, and 256 GB of RAM. We observed that
during the training of the agent, there is no significant speed-up when the
agent’s policy network is using the GPU. Thus, we used only the CPU for
sampling the RL environment and also for training the agent. A couple of
reasons could explain the lack of speed-up increase when trying to use the
GPU. First, the policy network used by our RL agent is not big enough to take
full advantage of the Nvidia Tesla V100. Second, there is an overhead data
transfer between the sampling of the environment and the training of the
policy. Because our environment was developed only to be used on the CPU (as
it is usually the case in RL), there is a penalty when the information is
transferred from the RAM (CPU) memory to the GPU memory and vice-versa. In
addition, modern processors like ours are capable of doing efficient
computation by leveraging the Advanced Vector Extensions 512 (Intel AVX-512)
instructions when working with tensors.

##  5 Conclusions and Outlook

In this article, we have presented a modular framework for the implementation
of different RL Optimal Execution setups. The framework can accommodate setups
with different functionalities and simulation choices such as data sources,
order execution simulation procedures, execution algorithms with multiple
types of actions, arbitrary order submission schedules, and reward calculation
functions among others. We have presented each module together with their
intended functionalities and the methods they are required to incorporate. We
have stated the dependencies between modules and presented a concrete
implementation of an example setup that aims to improve a TWAP benchmark
through RL. The example setup trains a narrow class of execution algorithms
that follow a TWAP order submission schedule. We have shown how to implement
this through specific modules and discussed the explicit and implicit effects
that each module induces on the simulation. In our experiments, we have
trained and evaluated two different agents, over multiple market periods,
shown the out of sample outperformance of both models to be small due to the
restrictive nature of the environment and finally shown the performance of the
best performing model not to significantly decay over time.

We have made our implementation as well as the code necessary to replicate the
experiments publicly available. In future work, we plan on leveraging the
framework to implement more complex simulation setups that tackle the
deficiencies discussed in Section 3. Since ranking setups by their realism is
not free of challenges, a good approach to selecting agents to carry out live
executions will be to choose those that can consistently outperform benchmarks
over the biggest number of setups.

## References

  * [1] Market making and the changing structure of the securities industry.  Lexington Books, Lexington, Mass, 1985.
  * [2] Nasdaq nordic algorithm offering.  ttps://www.nasdaq.com/solutions/execution-algorithms.  Accessed: 2022-05-18.
  * [3] Aurélien Alfonsi, Antje Fruth, and Alexander Schied.  Optimal execution strategies in limit order books with general shape functions.  arXiv e-prints, page arXiv:0708.1756, August 2007.
  * [4] Robert Almgren.  Optimal execution with nonlinear impact functions and trading-enhanced risk.  Applied Mathematical Finance, 10:1 – 18, 2003.
  * [5] Robert Almgren.  Optimal trading in a dynamic market.  2008\.
  * [6] Robert Almgren and Neil Chriss.  Optimal execution of portfolio transactions.  Journal of Risk, pages 5–39, 2000.
  * [7] Olle Dahlén Axel Rantil.  Optimized trade execution with reinforcement learning.  Master’s thesis, Linköping University | Department of Computer and Information Science, 2018.
  * [8] Wenhang Bao and Xiao-yang Liu.  Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysis.  arXiv e-prints, page arXiv:1906.11046, June 2019.
  * [9] Wenhang Bao and Xiao-yang Liu.  Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysis.  arXiv e-prints, page arXiv:1906.11046, June 2019.
  * [10] Erhan Bayraktar and Michael Ludkovski.  Liquidation in Limit Order Books with Controlled Intensity.  arXiv e-prints, page arXiv:1105.0247, May 2011.
  * [11] Dimitris Bertsimas and Andrew W. Lo.  Optimal control of execution costs.  Journal of Financial Markets, 1:1–50, 1998.
  * [12] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  OpenAI Gym.  arXiv e-prints, page arXiv:1606.01540, June 2016.
  * [13] A Cartea and S Jaimungal.  Incorporating order-flow into optimal execution.  Mathematics and Financial Economics, 10(3):339–364, 2017.
  * [14] Riccardo Cesari, Massimiliano Marzo, and Paolo Zagaglia.  Effective Trade Execution.  arXiv e-prints, page arXiv:1206.5324, June 2012.
  * [15] Andrea Coletta, Matteo Prata, Michele Conti, Emanuele Mercanti, Novella Bartolini, Aymeric Moulin, Svitlana Vyetrenko, and Tucker Balch.  Towards Realistic Market Simulations: a Generative Adversarial Networks Approach.  arXiv e-prints, page arXiv:2110.13287, October 2021.
  * [16] Kevin Dabeacute;rius, Elvin Granat, and Patrik Karlsson.  Deep execution - value and policy based reinforcement learning for trading and beating market benchmarks.  SSRN Electronic Journal, 01 2019.
  * [17] K. Dabérius, E. Granat, and P. Karlsson.  Deep Execution - Value and Policy Based Reinforcement Learning for Trading and Beating Market Benchmarks.  SSRN, 2022.
  * [18] Ngoc-Minh Dang.  Optimal execution with transient impact.  Econometric Modeling: Capital Markets - Asset Pricing eJournal, 2014\.
  * [19] M.F. Dixon, I. Halperin, and P. Bilokon.  Machine Learning in Finance: From Theory to Practice.  Springer International Publishing, 2020.
  * [20] Ian Domowitz and Jianxin Wang.  Auctions as algorithms: Computerized trade execution and price discovery.  Journal of Economic Dynamics and Control, 18(1):29–60, 1994.  Special Issue on Computer Science and Economics.
  * [21] Michael Nguyen @dyn4mik3.  Orderbook matching engine implementation.  https://github.com/dyn4mik3/OrderBook, 2013.
  * [22] Yuchen Fang, Kan Ren, Weiqing Liu, Dong Zhou, Weinan Zhang, Jiang Bian, Yong Yu, and Tie-Yan Liu.  Universal Trading for Order Execution with Oracle Policy Distillation.  arXiv e-prints, page arXiv:2103.10860, January 2021.
  * [23] Swiss Financial Market Supervisory Authority (FINMA).  Circular 2018/1, 2018.  Available at:
https://www.finma.ch/~/media/finma/dokumente/dokumentencenter/myfinma/rundschreiben/finma-
rs-2018-01.pdf?sc_lang=en.

  * [24] US Financial Industry Regulatory Authority (FINRA).  Rule 5310, 2014.  Available at:
https://www.finra.org/rules-guidance/rulebooks/finra-rules/5310.

  * [25] P.A. Forsyth, J.S. Kennedy, S.T. Tse, and H. Windcliff.  Optimal trade execution: A mean quadratic variation approach.  Journal of Economic Dynamics and Control, 36(12):1971–1991, 2012\.
  * [26] Peter A. Forsyth.  A hamilton–jacobi–bellman approach to optimal trade execution.  Applied Numerical Mathematics, 61(2):241–265, 2011.
  * [27] Olivier Gueant, Charles-Albert Lehalle, and Joaquin Fernandez-Tapia.  Optimal portfolio liquidation with limit orders.  SIAM J. Finan. Math., 3, 06 2011.
  * [28] Dieter Hendricks and Diane Wilcox.  A reinforcement learning extension to the almgren-chriss framework for optimal trade execution.  In 2014 IEEE Conference on Computational Intelligence for Financial Engineering $\&$ Economics (CIFEr), pages 457–464, 2014.
  * [29] Masanori Hirano, Hiroki Sakaji, and Kiyoshi Izumi.  Policy gradient stock gan for realistic discrete order data generation in financial markets.  ArXiv, abs/2204.13338, 2022.
  * [30] Gur Huberman and Werner Stanzl.  Optimal Liquidity Trading*.  Review of Finance, 9(2):165–200, 01 2005.
  * [31] Sham M. Kakade, Michael Kearns, Y. Mansour, and Luis E. Ortiz.  Competitive algorithms for vwap and limit order trading.  In EC ’04, 2004.
  * [32] Michaël Karpe, Jin Fang, Zhongyao Ma, and Chen Wang.  Multi-Agent Reinforcement Learning in a Realistic Limit Order Book Market Simulation.  arXiv:2006.05574 [q-fin], September 2020.  arXiv: 2006.05574.
  * [33] Charles-Albert Lehalle, Olivier Gueant, and Julien Razafinimanana.  High-frequency simulations of an order book: a two-scale approach.  New Economic Windows, 9:73–92, 01 2011.
  * [34] Junyi Li, Xitong Wang, Yaoyang Lin, Arunesh Sinha, and Micheal P. Wellman.  Generating Realistic Stock Market Order Streams.  arXiv e-prints, page arXiv:2006.04212, June 2020.
  * [35] Eric Liang, Richard Liaw, Philipp Moritz, Robert Nishihara, Roy Fox, Ken Goldberg, Joseph E. Gonzalez, Michael I. Jordan, and Ion Stoica.  RLlib: Abstractions for Distributed Reinforcement Learning.  arXiv e-prints, page arXiv:1712.09381, December 2017.
  * [36] Siyu Lin and Peter A Beling.  Optimal liquidation with deep reinforcement learning.  In Proceedings of the 33rd Conference on Neural Information Processing Systems, Deep Reinforcement Learning Workshop, Vancouver, Canada, 2019\.
  * [37] Siyu Lin and Peter A. Beling.  An end-to-end optimal trade execution framework based on proximal policy optimization.  In Christian Bessiere, editor, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 4548–4554. International Joint Conferences on Artificial Intelligence Organization, 7 2020.  Special Track on AI in FinTech.
  * [38] Haim Mendelson.  Market behavior in a clearing house.  Econometrica, 50(6):1505–1524, 1982.
  * [39] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing Atari with Deep Reinforcement Learning.  arXiv e-prints, page arXiv:1312.5602, December 2013.
  * [40] Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang, William Paul, Michael I. Jordan, and Ion Stoica.  Ray: A Distributed Framework for Emerging AI Applications.  arXiv e-prints, page arXiv:1712.05889, December 2017.
  * [41] Yusuke Naritomi and Takanori Adachi.  Data augmentation of high frequency financial data using generative adversarial network.  2020 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology (WI-IAT), pages 641–648, 2020\.
  * [42] Yuriy Nevmyvaka, Yi Feng, and Michael Kearns.  Reinforcement learning for optimized trade execution.  volume 2006, pages 673–680, 01 2006.
  * [43] Brian Ning, Franco Ho Ting Lin, and Sebastian Jaimungal.  Double Deep Q-Learning for Optimal Execution.  arXiv e-prints, page arXiv:1812.06600, December 2018.
  * [44] Anna A. Obizhaeva and Jiang Wang.  Optimal trading strategy and supply/demand dynamics.  Journal of Financial Markets, 16(1):1–32, 2013.
  * [45] Market Authority of Singapore (MAS).  Notice on execution of client orders, 2020.  Available at:
https://www.mas.gov.sg/-/media/MAS/Regulations-and-Financial-
Stability/Regulations-Guidance-and-Licensing/Notice-SFA-04N16-on-Execution-of-
Customers-Orders.pdf.

  * [46] Silviu Predoiu, Gennady Shaikhet, and Steven Shreve.  Optimal execution in a general one-sided limit-order book.  SIAM Journal on Financial Mathematics, 2(1):183–212, 2011.
  * [47] Alexander Schied and Torsten Schoneborny.  Risk aversion and the dynamics of optimal liquidation strategies in illiquid.  Finance and Stochastics, 13, 04 2009.
  * [48] Matthias Schnaubelt.  Deep reinforcement learning for the optimal placement of cryptocurrency limit orders.  European Journal of Operational Research, 296(3):993–1006, 2022\.
  * [49] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal Policy Optimization Algorithms.  arXiv e-prints, page arXiv:1707.06347, July 2017.
  * [50] European Securities and Markets Authority (ESMA).  Markets in financial instruments directive, 2018.  Available at:
https://www.esma.europa.eu/policy-rules/mifid-ii-and-mifir.

  * [51] Hong Kong’s Securities and Futures Comission (SFC).  Report on the thematic review of best execution, 2018.  Available at:
https://www.hkma.gov.hk/media/eng/doc/key-information/guidelines-and-
circular/2018/20180209e1a2.pdf.

  * [52] Richard S. Sutton and Andrew G. Barto.  Reinforcement Learning: An Introduction.  The MIT Press, second edition, 2018.