# Nowcasting Stock Implied Volatility with Twitter

Thomas Dierckx  0000-0001-9585-0566<EMAIL_ADDRESS>Jesse Davis
0000-0002-3748-9263<EMAIL_ADDRESS>Wim Schoutens  0000-0001-8510-1510
<EMAIL_ADDRESS>
###### Abstract

In this study, we predict next-day movements of stock end-of-day implied
volatility using random forests. Through an ablation study, we examine the
usefulness of different sources of predictors and expose the value of
attention and sentiment features extracted from Twitter. We study the approach
on a stock universe comprised of the 165 most liquid US stocks diversified
across the 11 traditional market sectors using a sizeable out-of-sample period
spanning over six years. In doing so, we uncover that stocks in certain
sectors, such as Consumer Discretionary, Technology, Real Estate, and
Utilities are easier to predict than others. Further analysis shows that
possible reasons for these discrepancies might be caused by either excess
social media attention or low option liquidity. Lastly, we explore how our
proposed approach fares throughout time by identifying four underlying market
regimes in implied volatility using hidden Markov models. We find that most
added value is achieved in regimes associated with lower implied volatility,
but optimal regimes vary per market sector.

††journal: Journal of Empirical Financelabel1label1footnotetext: Declarations
of interest: none

[inst1]organization=Department of Statistics and Risk, KU
Leuven,addressline=Celestijnenlaan 200B, city=Leuven, postcode=3000,
country=Belgium

[inst2] organization=Department of Computer Science, KU Leuven,
addressline=Celestijnenlaan 200A, city=Leuven, postcode=3000, country=Belgium

Next-day movements in stock implied volatility can be predicted using random
forests.

Attention and sentiment features extracted from Twitter improve predictive
performance.

Predictive performance varies significantly across the 11 traditional stock
market sectors.

Implied volatility regimes identified by hidden Markov models provide
actionable insights on when the proposed approach works best per stock market
sector.

##  1 Introduction

Today’s age is characterized by an ever-increasing connected and opinionated
world. The widespread adoption of social media has caused significant changes
in the world across many domains, and more are probably to follow. In the case
of financial markets, participants now have access to countless online
platforms to share their thoughts and feelings on certain events. Proponents
of the Efficient Market Hypothesis (Fama, 1970) ought to be pleased. The
advent of mass media facilitates rapid information diffusion, possibly
propelling markets into a higher tier of price efficiency. However, behavioral
economists would argue that this type of media might very well influence
investors and incite herd behavior which in turn induces inefficiency (e.g.
Baker and Wurgler, 2006; Chiang and Zheng, 2010). Theory aside, this new
wealth of information has not escaped the notice of the financial
establishment. Indeed, data providers such as Bloomberg and Refinitiv now
offer extensive social media indicators to help financial institutions
navigate this new world. Although the competitive edge that resides in these
alternative data sources remains veiled in secrecy, an abundance of academic
studies have already tried quantifying the interplay between social media and
certain financial variables, providing insight into the predictive power of
the masses.

Existing research has mainly focused on the Twitter platform and its influence
on three prominent financial variables: stock price (e.g. Groß-Klußmann et
al., 2019; Schnaubelt et al., 2020), realized volatility (e.g. Karagozoglu and
Fabozzi, 2017), trading volume (e.g. Guijarro et al., 2019) or a combination
thereof (e.g. Oliveira et al., 2017; Li et al., 2018). Remarkably, current
literature completely overlooks the interaction between social media and the
market implied volatility of stocks. Derived from option prices, this variable
is deemed to be one of the more important parameters in the world of
derivatives. In contrast to historical volatility, this is a forward-looking
metric that indicates how much risk the market expects a certain asset to
exhibit in the coming period. As this variable serves as a proxy for both
market sentiment and option contract prices, the ability to predict its
movements would be advantageous for the practice of asset management and
market making alike.

Most prevailing studies in this domain have two important methodological
shortcomings. First, analysis is typically performed on either a handful of
arbitrarily chosen stocks or indices tracking the entire market (e.g. Groß-
Klußmann et al., 2019), omitting sector idiosyncrasies in the process. Second,
hypotheses are commonly tested on a relatively small time window ranging from
a month (e.g. Bollen et al., 2011) to a few years (e.g. Schnaubelt et al.,
2020). However, the ever-changing nature of financial markets warrants a
closer look into how the interplay between social media patterns and financial
variables evolves over longer periods of time. As patterns may emerge and
dissipate over time, a crucial aspect of analysis is often left out.

The contribution of this study is threefold. First, to the best of our
knowledge, we are the first to investigate to what extent a stock its one-day
ahead movement in implied volatility can be predicted using machine learning
on different combinations of feature sources, including Twitter. Second,
instead of arbitrarily choosing a handful of stocks for our study, we
diversified our stock universe across the 11 traditional US stock market
sectors yielding 165 stocks in total. This allowed us to measure and explore
the variability in predictive performance present among sectors. Lastly, we
examined predictive performance on an out-of-sample period spanning January
1st, 2013 till March 1st, 2019. This period is significantly larger than many
other studies and gave us the opportunity to not only be more robust, but also
examine predictive performance throughout time. Instead of performing a year
by year analysis of predictive performance, we used hidden Markov models to
identify four regimes in the implied volatility of a stock and gauged whether
performance varies across them. We argue that this alternative quantification
of time yields more actionable insight as it allows practitioners to better
anticipate future performance.

##  2 Preliminaries

This section presents background information on the key components used in
this study. First, Section 2.1 explains market implied volatility and its
relation to the world of derivatives. Second, Section 2.2 describes the random
forest machine learning model which is used to perform predictions. Lastly,
Section 2.3 describes the hidden Markov model which is used to quantify
regimes in market implied volatility.

###  2.1 Market Implied Volatility

In the world of derivatives, options are one of the most prominent types of
financial instruments. As sellers of options are exposed to risk for the
duration of the contract, they want to be properly compensated. Measuring this
risk requires considering the expected price fluctuations of the underlying
asset over the duration of the contract. This expectation is better known as
implied volatility and it varies with the strike price and duration of an
option contract. To obtain a more general measure, the implied volatility of
option contracts that expire on the same date can be combined into a single
implied volatility measure. A famous example of this is the CBOE Volatility
Index, which combines the implied volatility of different option contracts on
the SPX into an index that is better known as the VIX.

More concretely, the VIX is a measure of expected price fluctuations in the
S&P 500 Index over the next 30 days. It is famously known as the fear index
and is considered a reflection of investor sentiment on the condition of the
market. Equation 1, taken from the VIX white paper (CBOE, 2015), shows how to
compute the VIX for a given term $T$:

$\mathit{VIX}=100\times\sqrt{\frac{2}{T}\sum\limits_{i}\frac{\Delta
K_{i}}{K_{i}^{2}}e^{RT}Q(K_{i})-\frac{1}{T}\bigg{[}\frac{F}{K_{0}}-1\bigg{]}^{2}}$
(1)

where:

$T$ ${}={}$ |  is time to expiration
---|---
$F$ ${}={}$ |  is the forward index level derived from the index option prices
$K_{0}$ ${}={}$ |  is the first strike below the forward index level F
$K_{i}$ ${}={}$ |  is the first strike price of the $i^{th}$ out-of-the-money option: a call if $K_{i}>K_{0}$, a put if $K_{i}<K_{0}$, or both put and call if $K_{i}=K_{0}$
$R$ ${}={}$ |  is the risk-free rate to expiration
$\Delta K_{i}$ ${}={}$ |  is the interval between strike prices
$Q(K_{i})$ ${}={}$ |  is the midpoint of the bid-ask spread for each option with strike $K_{i}$

The equation for computing the VIX is applicable to any asset where option
contracts are available. Although this measure can be calculated for any
arbitrary term, the duration of the option contracts will seldom match the
chosen term $T$. Indeed, option contracts typically have fixed expiration
dates and there is no guarantee that there are option contracts available with
a duration equal to the given term. To overcome this obstacle, the VIX is
first calculated for the option contracts expiring right before and after the
desired target date. The VIX for the given term can then be calculated by
linearly interpolating between the two computed measures, as outlined in
(CBOE, 2015).

###  2.2 Random Forests

Random forests (Breiman, 2001) are a popular machine learning approach for
learning a predictive model. They consist of multiple different decision (or
regression) trees whose predictions are combined into one final prediction.
The combination is typically done by taking the mode (or average) of all
outputs. While several variations exist for learning a random forest, all of
them are relatively straightforward. We summarize one of many popular
procedures. Given data $D=\\{(x_{i},y_{i})\\}_{i=1}^{n}$, where each $x_{i}$
has $F$ features and for $k=1\dots K$ trees:

  1. 1.

Obtain subset $d$ by sampling $m<n$ examples with replacement from $D$.

  2. 2.

Train a decision tree on $d$ using a random subset features $f\subseteq F$,
i.e. using CART (Breiman et al., 1984).

The prediction for a regression problem can then be obtained by:

$\hat{y}_{i}=\frac{1}{K}\sum_{k=1}^{K}f_{k}(x_{i})$ (2)

where $f$ is a function in the set of all possible decision trees and $K$ is
the total number of trees in the ensemble.

The advantages of random forests include that they are fast to build, are not
affected by feature scaling, are robust to irrelevant predictors and noisy
data (Khoshgoftaar et al., 2011). Moreover, their method of constructing an
ensemble model by randomly subsampling both data points and features during
the learning process helps decorrelate the predictions made by the individual
trees, which in turn reduces overfitting on the training data.

###  2.3 Hidden Markov Models

Hidden Markov models (HMM) are a generative approach for modeling systems that
follow a Markov process (Rabiner and Juang, 1986). The main assumption is that
while this process $Z$ is hidden, it can be learned from an observable
sequence $X$ whose behaviour depends on $Z$. More formally, the HMM models the
joint distribution of a sequence of hidden states $Z$ and observations $X$
described by:

$P(Z_{1:K},X_{1:K})=P(Z_{1})P(X_{1}|Z_{1})\prod_{t=2}^{K}P(Z_{t}|Z_{t-1})P(X_{t}|Z_{t})$
(3)

Given the number of hidden states $K$ and observed sequence $X$, the model is
fully determined by its parameters $\pi$, $A$, and $B$ which represent the
initial state distribution, state transition model, and emission probabilities
model, respectively. The initial state distribution is a $K\times 1$ vector
denoting the probabilities that the process is each of the $K$ states in the
first timestep. The transition model is a $K\times K$ stochastic matrix where
each element $A_{i,j}$ denotes the probability of transitioning from state
$Z_{t-1,i}$ to $Z_{t,j}$ where $i,j\in\\{1,\dots,K\\}$. Lastly, the emissions
probability model is a $M\times K$ matrix, with $M$ representing the number of
distinct observations, whose elements $B_{k,j}$ denote the probability of
observing $X_{t,k}$ given state $Z_{t,j}$.

The three key tasks associated with hidden Markov models are:

  1. 1.

What is the probability that a sequence of observations $X$ was generated by a
given HMM?

  2. 2.

Given an HMM, what sequence of hidden states $Z$ best explains a given
sequence of observations $X$?

  3. 3.

Given a sequence of observations $X$, learn an HMM with parameters $\pi$, $A$,
and $B$ that would generate them.

The first two tasks can be solved using dynamic programming using the forward-
backward algorithm (Chang and Hancock, 1966) and Viterbi (Forney, 1973)
algorithm, respectively. The third problem is solved by the Baum-Welch
algorithm (Baum, 1972) which uses an iterative expectation-maximization
approach.

##  3 Methodology

The main goal of this study is to explore the following questions:

  1. 1.

To what extent are one-day ahead movements in end-of-day implied volatility
predictable, and do features extracted from Twitter improve performance?

  2. 2.

Does performance vary across the 11 different stock market sectors? If so, are
there any obvious factors that might explain this variability?

  3. 3.

Can we identify underlying market regimes in implied volatility that influence
the performance of our proposed approach?

We tackle the first question by performing an ablation study using random
forests on feature configurations including stock price, stock implied
volatility, and Twitter features. The study encompasses a universe of 165
stocks over an out-of-sample period spanning January 1st, 2013 till March 1st,
2019. To examine the second question, we diversified our stock universe over
the 11 traditional stock sectors and grouped predictive performance by stocks
belonging to the same sector. The third and last question was studied by using
a hidden Markov model to identify four distinct implied volatility regimes per
stock, after which predictive performance was grouped by regime.

The next few sections explain our methodology in more detail. First, Section
3.1 outlines the stock universe we used for our study. Second, Section 3.2
explains how we obtained the relevant data for each stock and how we
constructed features for prediction. Section 3.3 and Section 3.4 then
respectively show how we used machine learning to predict our target variable
and how we evaluated the performance of the approach. Lastly, Section 3.5
explains how we used hidden Markov models to identify regimes in implied
volatility which we later use to evaluate our prediction performance through
time.

###  3.1 Stock Universe Selection

In order to obtain a diversified universe of stocks, we looked at the popular
SPDR and Vanguard Electronic Traded Funds (ETF) that track the 11 traditional
US stock market sectors. For each sector, we selected the 15 most liquid
stocks based on their average daily dollar-weighted option volume for a total
of 165 stocks. Some stocks were excluded due to stock splits (i.e. we kept
GOOG and dropped GOOGL), a late introduction to the stock market (i.e. PYPL,
ROKU, and SNAP only got introduced after 2015), and ambiguous names making it
hard to obtain relevant tweets (i.e. DOW is a chemical company but also a
common alias for the Dow Jones Index). Note that we replaced the excluded
stocks to maintain 15 stocks per sector for our study. Table 1 provides a
concise overview of our stock universe. Refer to A for a full overview of
which stocks were selected per market sector.

Table 1: This table presents the 11 different stock market sectors together with their corresponding SPDR ETF symbol and number of stocks considered in this study. The symbols are used to denote sectors throughout this paper, but are not indicative of stocks only belonging to the SPDR ETF portfolio.  Symbol | Sector | Selected Stocks
---|---|---
XLC | Communication Services | 15
XLY | Consumer Discretionary | 15
XLP | Consumer Staples | 15
XLE | Energy | 15
XLF | Financials | 15
XLV | Healthcare | 15
XLB | Materials | 15
XLI | Industrials | 15
XLK | Technology | 15
XLRE | Real Estate | 15
XLU | Utilities | 15

###  3.2 Data Acquisition and Feature Generation

We consider data ranging from January 1st, 2011 through March 1st, 2019 for
three data sources:

  1. 1.

Stock price data which consists of historical end-of-day adjusted closing
prices for each stock in our universe downloaded from Yahoo Finance.

  2. 2.

Option contract price data which consists of historical end-of-day option
chains for each stock in our universe obtained from IVolatility.

  3. 3.

Twitter data which consists of all relevant tweets published for each stock.
These were collected by filtering on cashtags, which are popular string
identifiers authors use to indicate their message is about a certain stock
(i.e. a tweet about the Apple stock typically contains $AAPL). In contrast to
other research, we did not employ additional filtering techniques to discard
potential spam. Most additional filtering rules appear arbitrary and there
seems to be no clear evidence of their validity.

In total, four features were extracted per stock for each trading day. First,
we simply used the end-of-day adjusted closing price from the stock price
data. Second, we calculated the end-of-day 30-day implied volatility using the
VIX method on the option contract data. Third and last, we derived two
numerical features from our textual Twitter corpus: end-of-day total tweet
publication count and end-of-day average sentiment polarity. The former
represents the total number of published tweets on a given day. The latter was
obtained by performing sentiment analysis using VADER (Hutto and Gilbert,
2014), a lexicon- and rule-based sentiment model that is specifically well-
tailored to social media text, on individual tweets. This yields a sentiment
polarity score $s\in[-1,1]$ for each tweet, which was then used to compute the
daily average.

In an effort to capture temporal information residing in the original feature
timeseries, we generated two additional predictors per feature. To this end,
the daily difference (or first-order difference) and the difference between
the daily value and its exponential moving average of the last 10 trading days
was taken. Table 2 outlines the different data sources and their features used
in this study. Note that the original adjusted closing price was omitted, as
this is typically a non-stationary variable offering little value to a
prediction model.

Table 2: This table provides a summary of the features considered per data source. The first row indicates what original features were extracted, whereas the last three rows indicate (*) which features were considered for the actual study. Note that the last two rows denote a specific feature engineering technique applied to the original feature.  | Stocks | Options | Twitter
---|---|---|---
Extracted | Adj. Closing Price | Implied Volatility | Count, Sentiment
Original |  | * | *
1st Order Diff. | * | * | *
EMA(10) Diff. | * | * | *

###  3.3 Predicting Movements in Implied Volatility

This study aims to predict one-day ahead movements in a stock’s 30-day implied
volatility. Concretely, given information at the end of trading day $t$, we
predict whether implied volatility will have moved up or down by the end of
next trading day $t+1$. To do so, we construct a binary target variable for
day $t$ as:

$y_{t}=\begin{cases}1,&\text{if
$(\mathit{ivolatility}_{t+1}-\mathit{ivolatility}_{t})>0$}.\\\
0,&\text{otherwise}.\end{cases}$ (4)

where $\mathit{ivolatility_{t}}$ denotes the end-of-day implied volatility on
day $t$.

In order to predict our target variable, we used random forest classifiers
even though more powerful models may exist. For example, the highly popular
gradient boosted trees (Friedman, 2001) have been shown to generally perform
slightly better than random forests (Caruana et al., 2008; Caruana and
Niculescu-Mizil, 2006). However, they are very sensitive to hyper-parameter
configurations and require longer runtimes for training. The main goal of this
study is not to maximize predictive performance, but rather probe the
feasibility of our proposed approach. In addition, it has been suggested that
random forests might generally work better on noisy data (Khoshgoftaar et al.,
2011), which is especially convenient when working on financial data. Lastly,
we did not consider techniques from the domain of deep learning due to the
complexity of the models and relatively small number of data points in our
study.

Ultimately, we used 64 distinctive random forest configurations built using
Sklearn. Each random forest was built with 1000 trees and a unique combination
of different hyper-parameters that control maximum tree depth, the minimum
number of samples required to split an internal node, and the minimum number
of samples required to be in a leaf node. Each individual tree was built by
sampling the training dataset (with replacement) and only considering a random
number of $\sqrt{f}$ features where $f$ denotes the total amount of features.
The models were trained on a temporally ordered feature matrix $X$ of
dimension $T\times K$, obtained by using any subset of features $K$ from
Section 3.2 and period $T$. Table 3 specifies the possible random forest
configurations considered in this study.

Table 3: This table presents the different possible values considered for different hyper-parameters available in the random forest implementation of Sklearn. The default value is used for hyper-parameters not listed.  Hyper-parameter | Values
---|---
n_estimators | {1000}
max_depth | {4, 6, 8, 10}
min_samples_split | {5, 10, 15, 20}
min_samples_leaf | {1, 3, 5, 8}
random_state | {42}
bootstrap | yes
max_features | sqrt
Table 4: Example of expanding walk forward validation without where $t_{i}$ represents the feature vector of trading day $i$. In this example, a training window with an initial size $s=3$ is taken together with a testing window of size $k=1$. We therefore consistently use the feature vectors of past trading days to train a model (underlined) and subsequently test said model on trading day $t+k$ (bold).  Iteration | Variable roles
---|---
1 | $\underline{t_{1}\hskip 4.2679ptt_{2}\hskip 4.2679ptt_{3}}\hskip 4.2679pt\mathbf{t_{4}}\hskip 4.2679ptt_{5}\hskip 4.2679pt\dotsi\hskip 4.2679ptt_{n}$
2 | $\underline{t_{1}\hskip 4.2679ptt_{2}\hskip 4.2679ptt_{3}\hskip 4.2679ptt_{4}}\hskip 4.2679pt\mathbf{t_{5}}\hskip 4.2679pt\dotsi\hskip 4.2679ptt_{n}$
⋮ | ⋮
m | $\underline{t_{1}\hskip 4.2679pt\dotsi\hskip 4.2679ptt_{n-3}\hskip 4.2679ptt_{n-2}\hskip 4.2679ptt_{n-1}}\hskip 4.2679pt\mathbf{t_{n}}$

###  3.4 Experimental Evaluation

We evaluated the different random forest configurations using walk-forward
validation, a cross-validation technique designed specifically for temporally
ordered data. Classical cross-validation methods assume observations to be
independent. This assumption does not necessarily hold for timeseries data,
which inherently contains temporal dependencies among observations. To this
end, the dataset is repeatedly split up in training and test sets where
temporal order is accounted for. In our case, we used an expanding window of
initially 504 trading days to train the models, after which performance was
measured on the next out-of-sample 40 trading days. Table 5 shows an example
of this method where $t_{i}$ denotes the feature vector corresponding to
trading day $i$. Note that in this scenario, when given a total of $n$
observations, an expanding training window of length $t$ and an out-of-sample
test window of length $k$, you can construct a maximum of $n-t-k$ different
train-test splits. Ultimately, each configuration its performance is averaged
across all folds. We measured performance with the area under the receiver
operating characteristic curve metric (AUC hereafter).

Table 5: Example of walk-forward validation where $t_{i}$ represents the feature vector of trading day $i$. In this example, a training window with an initial size $s=3$ is taken together with a testing window of size $k=1$. We therefore consistently use the feature vectors of past trading days to train a model (underlined) and subsequently test said model on trading day $t+k$ (bold).  Iteration | Variable roles
---|---
1 | $\underline{t_{1}\hskip 4.2679ptt_{2}\hskip 4.2679ptt_{3}}\hskip 4.2679pt\mathbf{t_{4}}\hskip 4.2679ptt_{5}\hskip 4.2679pt\dotsi\hskip 4.2679ptt_{n}$
2 | $\underline{t_{1}\hskip 4.2679ptt_{2}\hskip 4.2679ptt_{3}\hskip 4.2679ptt_{4}}\hskip 4.2679pt\mathbf{t_{5}}\hskip 4.2679pt\dotsi\hskip 4.2679ptt_{n}$
⋮ | ⋮
m | $\underline{t_{1}\hskip 4.2679pt\dotsi\hskip 4.2679ptt_{n-3}\hskip 4.2679ptt_{n-2}\hskip 4.2679ptt_{n-1}}\hskip 4.2679pt\mathbf{t_{n}}$

###  3.5 Analyzing Performance through Time with Hidden Markov Models

The ever-changing nature of financial markets makes it hard to find approaches
that consistently work well. The ability to time approaches therefore becomes
an interesting perk. In an effort to evaluate how our proposed approach
weathers the evolution of the financial market through time, we look at its
performance under different market regimes.

We quantified market regimes as different states in mean implied volatility
using a hidden Markov model. For each stock, a different HMM model was trained
on its end-of-day implied volatility timeseries dating from January 1st, 2007
till December 31st, 2012 and was used out-of-sample thereafter. Analogous to a
study performed by Société Générale (Daviaud et al., 2020), four different
regimes were identified corresponding to low, medium, high, and very high mean
implied volatility. Table 6 specifies the hyper-parameter configuration used
to build hidden Markov models using the hmmlearn Python package.

Table 6: This table presents the hyper-parameter configuration used for building hidden Markov models using the hmmlearn package. The default value is used for hyper-parameters not listed.  Hyper-parameter | Values
---|---
n_components | 4
n_iter | 100
random_state | 42
emissions | Gaussian
algorithm | Viterbi

##  4 Experimental Results and Discussion

In this section, we present and discuss our experimental results. First,
Section 4.1 shows the results of our ablation study where in total seven
different feature configurations were considered. Section 4.2 then builds on
these results by looking at the performance of the best feature configuration
per market sector. Lastly, Section 4.3 looks at predictive performance across
different implied volatility regimes. Recall that throughout the remainder of
this section we may denote the 11 different stock market sectors by the symbol
of their equivalent SPDR ETF tracker. This is solely done out of convenience
and is not indicative of stocks only belonging to said ETF portfolio. Refer to
Table 1 for an overview of the sector symbols.

###  4.1 Ablation Study

The first objective of this study was to investigate to what extent daily
movements in end-of-day implied volatility can be predicted. To this end, we
obtained 11 different features from three different data sources (Section 3.2)
on which we built random forest classifiers to predict said target variable.
We assessed the effectiveness of the different data sources by performing an
ablation study where 7 different scenarios were considered, shown in Table 7.
In total, one-day ahead movements in implied volatility were predicted for 165
stocks (Section 3.1) spanning an out-of-sample period from January 1st, 2013
till March 1st, 2019.

Table 7: This tables shows the different feature scenarios considered in our ablation study together with their total number of features. Note that the third column indicates the usage of both original and derived features from the given feature source.  Scenario | Feature Source | Features
---|---|---
1 | Stock Price | 2
2 | Stock Price, Tweets | 8
3 | Implied Volatility | 3
4 | Implied Volatility, Tweets | 9
5 | Tweets | 6
6 | Stock Price, Implied Volatility | 5
7 | Stock Price, Implied Volatility, Tweets | 11

We compared the predictive performance of different scenarios to that of a
stratified dummy classifier, which makes the comparison more rigid than using
a simple random classifier. Indeed, implied volatility tends to go down more
often than it goes up. This causes a stratified dummy classifier to achieve a
median AUC of 51.8% across all 165 stocks versus a 50.0% achieved by a fully
random one.

Table 8 displays the median AUC achieved for each scenario averaged over the
entire selected stock universe and the difference in AUC between our approach
and the stratified dummy classifier. These results provide empirical evidence
that end-of-day movements in implied volatility can indeed be predicted. All
possible feature scenarios perform better than a purely random classifier that
achieves a median of 50.0% AUC. Moreover, 4 out of 7 scenarios outperform the
stratified dummy classifier that achieves a median of 51.8% AUC. The
commonality among these improved scenarios is the use of implied volatility
features, indicating that this is an important source of information.
Moreover, including features derived from tweets always yielded a better
median performance (S2 versus S1, S4 versus S3, and S7 versus S6). This
implies there is indeed a predictive interplay between information on Twitter
and future implied volatility. Lastly, using all possible features (S7)
yielded the best result overall, suggesting there are predictive patterns
among all three feature sources.

Table 8: This table displays the median predictive performance across all 165 stocks per feature configuration obtained by predicting daily end-of-day movements in implied volatility over the period of January 1st, 2013 to March 1st, 2019. Moreover, the second row shows how much the proposed approach does better than the stratified dummy classifier.  | S1 | S2 | S3 | S4 | S5 | S6 | S7
---|---|---|---|---|---|---|---
Median AUC | 51.1 | 51.6 | 53.6 | 54.2 | 50.9 | 54.3 | 55.1
Improvement | -0.6 | -0.1 | +1.9 | +2.5 | -0.8 | +2.7 | +3.4

###  4.2 Predictive Performance across Sectors

In this section we look at the best performing feature configuration (S7) and
its performance variability across 11 different stock market sectors. Figure 1
shows a box plot where the performance improvement of our proposed approach
versus the stratified dummy classifier for each individual stock is grouped by
sector. The results were obtained on an out-of-sample period spanning January
1st, 2013 till March 1st, 2019.

Figure 1: This box plot shows the performance improvement of our proposed
methodology using feature configuration S7 versus a stratified dummy
classifier on individual stocks grouped by sector. The red dotted line
represents the minimum threshold necessary to beat the stratified dummy
classifier. The blue dotted line represents the overall median improvement of
our proposed approach versus the stratified dummy classifier.

It is clear that our proposed methodology is generally able to beat the
stratified dummy classifier across all different sectors. The approach beats
the dummy classifier on 148 out of 165 stocks. However, there is a
considerable amount of variability present across different sectors. Indeed,
the approach does significantly better on XLRE and XLU, but predictions on
XLC, XLY, and XLK also do better comparatively. In contrast, XLI and XLB seem
to lack in performance. The next two subsections will showcase a preliminary
attempt to partially explain this variability in performance.

####  4.2.1 The Effect of Option Liquidity

The results from the previous section indicate that predictions on stocks from
both XLRE and XLU do significantly better. Remarkably, it turns out that
stocks in these two sectors are also significantly less liquid compared to
other sectors. Figure 2 respectively shows a box plot of median option
liquidity per sector, measured by the average daily dollar amount traded in
options, and a regression plot where the relationship between liquidity and
performance improvement versus the dummy classifier is outlined.

Figure 2: This figure presents a box plot (left) where the median of daily
stock option liquidity is grouped per sector and a regression plot (right)
where the relationship between stock option liquidity and performance
improvement versus the dummy classifier is outlined.

These results seem to suggest that there is indeed a weak negative correlation
between predictive performance and option liquidity, implying that less liquid
stocks are easier to predict. We hypothesize that the relatively undersized
liquidity of both XLRE and XLU in the options market is possibly accompanied
by a less efficient price discovery process. This in turn might cause these
markets to reflect new information more slowly, making them easier to predict
with the information at hand. However, we note that is only one possible
explanation and many other factors might lie at the basis of this phenomenon.

####  4.2.2 The Effect of Twitter Attention

Lower liquidity might partially explain why sectors such as XLRE and XLU seem
easier to predict, but it certainly does not tell the whole story. Indeed,
predictions on stocks from XLC, XLY, and XLK, examples of very liquid sectors,
also do better comparatively. Here, we hypothesize that this might be due to
the attention they receive on Twitter. Figure 3 respectively shows a box plot
of the median daily tweets published on stocks grouped per sector and a
regression plot where the relationship between liquidity and daily tweets is
outlined.

Figure 3: This figure presents a box plot (left) where the median of daily
tweet publication of stocks is grouped per sector and a regression plot
(right) where the relationship between a stock its daily tweets and option
liquidity is outlined.

The box plot on the left of Figure 3 shows that stocks in XLC, XLY, and XLK
receive significantly more attention on Twitter than others. Note that the
plot demonstrates a striking resemblance with the box plot showing liquidity
per sector in Figure 2. Indeed, the regression plot on the right shows a very
strong correlation between attention on Twitter and liquidity. These findings
seem to suggest that prediction is easier on stocks that are more popular on
social media. We investigated this phenomenon further by looking at the
improvement in predictive performance caused by features extracted from
Twitter per sector. More concrete, we looked at the difference in performance
between feature configuration S6, which uses stock and options features, and
S7, which combines features from S6 and Twitter features (Section 4.1). Figure
4 shows the median improvement of using Twitter features per stock grouped by
sector.

Figure 4: This figure shows the performance improvement of using Twitter
features together with features from S6, versus only using features from S6.
The red dotted line represents the minimum threshold necessary to beat the
feature configuration of S6.

With exception of XLE, most sectors seem to have a sizeable number of stocks
that benefit from using social media features. This is no surprise considering
the results presented in Section 4.1. However, it seems that sectors that
receive more social media attention seem to benefit the most. For example, the
three sectors XLC, XLY, and XLK that are most popular also benefit more
consistently followed closely by XLV. Two reasons might explain these results.
First, previous research has hinted at social media inciting herd behavior and
emotional reactions among investors, possibly driving inefficiency (e.g. Wang
and Wang, 2018; Oliveira et al., 2017; Groß-Klußmann et al., 2019). If this is
true, it makes sense that Twitter features provide more added value for
popular stocks. Second, our sentiment extraction technique on tweets is
imperfect and might impact these results as well. Without carefully filtering
out tweets that are advertisements or spam, we rely on the law of large
numbers to correctly estimate a stock its sentiment for a given day.
Naturally, daily sentiment estimation will therefore be better on stocks that
are more heavily tweeted about.

###  4.3 Performance across Implied Volatility Regimes

In this section, we look at the best-performing feature configuration (S7) and
its performance variability across four different market regimes in implied
volatility, identified by using a hidden Markov model. Table 9 shows the
median of all results across 165 stocks where, for each of the four regimes,
the columns respectively show how many days were spent in a regime, what the
average implied volatility was, the AUC of a stratified dummy classifier, and
how much better our approach did compared to the latter.

Table 9: This table shows the median number of days a stock resided in one out of four implied volatility regimes, together with each regime’s mean implied volatility, stratified dummy performance and the improvement of our approach. In total, 165 stocks were considered over a period spanning January 1st, 2013 till March 1st, 2019.  | Frequency | Implied Volatility | Dummy AUC | Improvement |
---|---|---|---|---|---
Low | 392 | 18.6 | 50.75 | +3.15 |
Medium | 452 | 22.3 | 50.65 | +3.83 |
High | 411 | 26.7 | 52.12 | +3.84 |
Very High | 231 | 35.3 | 54.92 | +1.47 |

In general, stocks seem to reside more in the lower implied volatility
environments. The stratified dummy classifier also appears to perform worse
here, implying that up and down movements are about equal in occurrence.
However, the dummy classifier performance picks up as implied volatility
increases. This makes sense from a theoretical perspective, as implied
volatility is deemed to be a mean reverting process characterized by big up
movements after which the variable slowly trails back down, resulting in more
down movements. Although our approach seems to outperform the dummy classifier
in all regimes, the added value seems to be most significant in the low to
high regimes. This may suggest that distressed markets are harder to predict
than their calmer counterparts. However, note that the improvement of our
approach seems to be slightly correlated with regime frequency. From a data
science perspective, this might imply that the model performs worse in less
frequent regimes because it simply had fewer examples to learn from. Lastly,
Table 10 offers a more finer-grained analysis where the median improvement for
each regime per sector is shown.

Table 10: This table shows the median improvement of our approach compared to a stratified dummy classifier for stocks grouped per market sector and implied volatility regime. The best and worst regime for each sector are respectively indicated by bold and underlined text styles. | Low | Medium | High | Very High
---|---|---|---|---
XLB | 1.87 | 3.06 | 1.02 | 0.77
XLC | 3.57 | 4.76 | 5.27 | 0.26
XLE | 2.55 | 2.46 | 4.0 | -2.55
XLF | 3.06 | 4.08 | 2.64 | -1.7
XLI | 0.05 | 1.53 | 0.17 | -1.5
XLK | 0.60 | 2.38 | 3.91 | 5.70
XLP | 2.72 | 3.83 | 3.23 | 1.53
XLRE | 8.5 | 6.38 | 4.93 | 4.25
XLU | 6.63 | 5.95 | 5.36 | 1.79
XLV | 4.25 | 3.23 | 1.62 | 2.13
XLY | 2.04 | 3.66 | 5.95 | 0.85

Again, we remark a significant amount of variability in performance across
both regimes and sectors. Optimal implied volatility regimes seem to differ
significantly for different sectors. In contrast, with exception of XLK,
sectors seem to comparatively do worse when implied volatility is very high.
As all sectors share roughly the same regime frequency distribution, no clear
reason emerges to explain the performance variability. One potential reason
might be due to sector idiosyncrasies. For example, defensive sectors such as
XLRE, XLU, and XLV seem to have lower optimal regimes than cyclical sectors
such as XLC, XLK, and XLY. However, this is not always the case. Lastly, we
have no explanation for the significant difference between XLK and the other
sectors. Here, our approach performs best in the highest implied volatility
regime and worst in the lowest. This is in stark contrast with the other
sectors where the opposite is true. Perhaps the speculative nature of
technology stocks is more sensitive to herd behaviour and therefore investor
irrationality.

##  5 Conclusion

In this study, we presented the first empirical evidence that one-day ahead
movements of end-of-day stock implied volatility can be predicted to a certain
extent, and that attention and sentiment features extracted from Twitter
improve the performance of the approach. These alternative features were not
able to predict implied volatility in isolation, but improved the approach
significantly when combined with predictors extracted from stock and options
data. This suggests that the interplay between these sources gives rise to
predictive patterns. By conducting our experiments on a diversified universe
of 165 US stocks, we were able to assess the predictive performance across 11
traditional stock market sectors and found that stocks in real estate,
utilities, consumer discretionary, communications, and technology were easier
to predict than others. Further analysis indicated that these differences
could potentially be explained by market inefficiencies caused by low option
liquidity in the real estate and utilities sector, and excessive Twitter
attention in the consumer discretionary, communications, and technology
sector. Lastly, using hidden Markov models, we evaluated the predictive
performance of our approach across four different implied volatility regimes.
Although it outperforms the dummy classifier in all four regimes, we found
that it yields the least improvement in the regime associated with the highest
average implied volatility. Moreover, we discovered that different stock
market sectors have different optimal regimes for the application of our
approach. By analyzing performance through the usage of regimes, we showed
that this alternative quantification of time provides additional insight into
the performance of models which in turn could help better anticipate their
future performance.

##  Appendix A Selected Stock Universe

This appendix contains more detailed information on the US stock universe used
in this study. Table 11 outlines which stocks were chosen for each traditional
market sector.

Table 11: This table presents the US stock universe that was used in this study. Note that for each traditional market sector we chose 15 stocks based on option liquidity. Materials | FCX | X | NEM | CLF | MOS
---|---|---|---|---|---
DD | IP | CF | AA | NUE
LYB | VMC | SHW | BLL | WRK
Communications | FB | NFLX | GOOG | T | TWTR
VZ | DIS | CMCSA | EA | YELP
ATVI | DISH | CHTR | Z | TTWO
Energy | XOM | OXY | CVX | COP | SLB
HAL | VLO | EOG | APA | KMI
HES | MPC | MRO | RIG | WMB
Financials | BAC | JPM | C | GS | WFC
AIG | BX | MS | AXP | CME
MET | USB | SCHW | COF | BLK
Industrials | GE | BA | CAT | LMT | UPS
UNP | FDX | DE | MMM | HON
CSX | NSC | EMR | NOC | ETN
Technology | AAPL | NVDA | MSFT | INTC | MU
IBM | QCOM | CSCO | CRM | AMD
MA | V | ORCL | ADBE | TXN
C. Staples | PG | WMT | PM | KO | MO
CL | COST | PEP | HLF | WBA
GIS | MNST | TSN | CAG | KR
Real Estate | SPG | WY | AMT | EQIX | IRM
CCI | PSA | AVB | VTR | O
HST | DLR | PLD | MAC | EQR
Utilities | SO | EXC | AEP | DUK | NRG
NEE | FE | D | PPL | ED
ETR | EIX | CNP | NI | SRE
Healthcare | PFE | JNJ | GILD | LLY | MRK
ABT | BMY | AMGN | UNH | CVS
ABBV | ISRG | MDT | CI | DHR
C. Discretionary | AMZN | TSLA | MCD | HD | F
CMG | GM | SBUX | EBAY | NKE
TGT | LOW | BBY | LULU | MGM

## References

  * Baker and Wurgler (2006) Baker, M., Wurgler, J., 2006\.  Investor sentiment and the cross-section of stock returns.  The Journal of Finance 61, 1645–1680.  doi:https://doi.org/10.1111/j.1540-6261.2006.00885.x.
  * Baum (1972) Baum, L.E., 1972.  An inequality and associated maximization technique in statistical estimation for probabilistic functions of Markov processes, in: Inequalities III: Proceedings of the Third Symposium on Inequalities, Academic Press. pp. 1–8.
  * Bollen et al. (2011) Bollen, J., Mao, H., Zeng, X., 2011.  The impact of social and conventional media on firm equity value: A sentiment analysis approach.  Journal of Computational Science 2, 1–8.  doi:https://doi.org/10.1016/j.jocs.2010.12.007.
  * Breiman (2001) Breiman, L., 2001.  Random forests.  Mach. Learn. 45, 5–32.  doi:https://doi.org/10.1023/A:1010933404324.
  * Breiman et al. (1984) Breiman, L., Friedman, J.H., Olshen, R.A., Stone, C.J., 1984\.  Classification and Regression Trees.  Wadsworth and Brooks, Monterey, CA.  doi:https://doi.org/10.1201/9781315139470.
  * Caruana et al. (2008) Caruana, R., Karampatziakis, N., Yessenalina, A., 2008.  An empirical evaluation of supervised learning in high dimensions, in: International Conference on Machine Learning (ICML), pp. 96–103.  doi:https://doi.org/10.1145/1390156.1390169.
  * Caruana and Niculescu-Mizil (2006) Caruana, R., Niculescu-Mizil, A., 2006\.  An empirical comparison of supervised learning algorithms, in: Proceedings of the 23rd International Conference on Machine Learning, Association for Computing Machinery, New York, NY, USA. p. 161–168.  doi:https://doi.org/10.1145/1143844.1143865.
  * CBOE (2015) CBOE, 2015.  Cboe volatility index white paper URL: https://www.cboe.com/micro/vix/vixwhite.pdf.
  * Chang and Hancock (1966) Chang, R., Hancock, J., 1966\.  On receiver structures for channels having memory.  IEEE Transactions on Information Theory 12, 463–468.  doi:https://doi.org/10.1109/TIT.1966.1053923.
  * Chiang and Zheng (2010) Chiang, T.C., Zheng, D., 2010\.  An empirical analysis of herd behavior in global stock markets.  Journal of Banking & Finance 34, 1911–1921.  doi:https://doi.org/10.1016/j.jbankfin.2009.12.014.
  * Daviaud et al. (2020) Daviaud, O., Korber, O., Mukhopadhyay, A., Ungari, S., 2020\.  Systematic trading in options.  Société Générale - Cross Asset Research .
  * Fama (1970) Fama, E.F., 1970.  Efficient capital markets: A review of theory and empirical work.  The Journal of Finance 25, 383–417.  doi:https://doi.org/10.2307/2325486.
  * Forney (1973) Forney, G.D., 1973.  The viterbi algorithm.  Proceedings of the IEEE 61, 268–278.  doi:https://doi.org/10.1109/PROC.1973.9030.
  * Friedman (2001) Friedman, J.H., 2001.  Greedy function approximation: A gradient boosting machine.  The Annals of Statistics 29, 1189 – 1232.  doi:https://doi.org/10.1214/aos/1013203451.
  * Groß-Klußmann et al. (2019) Groß-Klußmann, A., König, S., Ebner, M., 2019.  Buzzwords build momentum: Global financial twitter sentiment and the aggregate stock market.  Expert Systems with Applications 136, 171–186.  doi:https://doi.org/10.1016/j.eswa.2019.06.027.
  * Guijarro et al. (2019) Guijarro, F., Moya-Clemente, I., Saleemi, J., 2019.  Liquidity Risk and Investors’ Mood: Linking the Financial Market Liquidity to Sentiment Analysis through Twitter in the S&P500 Index.  Sustainability 11, 1–13.  doi:https://doi.org/10.3390/su11247048.
  * Hutto and Gilbert (2014) Hutto, C.J., Gilbert, E., 2014\.  Vader: A parsimonious rule-based model for sentiment analysis of social media text., in: Adar, E., Resnick, P., Choudhury, M.D., Hogan, B., Oh, A.H. (Eds.), ICWSM, The AAAI Press.
  * Karagozoglu and Fabozzi (2017) Karagozoglu, A.K., Fabozzi, F.J., 2017\.  Volatility wisdom of social media crowds.  The Journal of Portfolio Management 43, 136–151.  doi:https://doi.org/10.3905/jpm.2017.43.2.136.
  * Khoshgoftaar et al. (2011) Khoshgoftaar, T.M., Van Hulse, J., Napolitano, A., 2011.  Comparing boosting and bagging techniques with noisy and imbalanced data.  IEEE Transactions on Systems, Man, and Cybernetics \- Part A: Systems and Humans 41, 552–568.  doi:https://doi.org/10.1109/TSMCA.2010.2084081.
  * Li et al. (2018) Li, T., van Dalen, J., van Rees, P.J., 2018.  More than just noise? examining the information content of stock microblogs on financial markets.  Journal of Information Technology 33, 50–69.  doi:https://doi.org/10.1057/s41265-016-0034-2.
  * Oliveira et al. (2017) Oliveira, N., Cortez, P., Areal, N., 2017.  The impact of microblogging data for stock market prediction: using twitter to predict returns, volatility, trading volume and survey sentiment indices.  Expert systems with applications 73, 125–144.  doi:https://doi.org/10.1016/j.eswa.2016.12.036.
  * Rabiner and Juang (1986) Rabiner, L., Juang, B., 1986\.  An introduction to hidden markov models.  IEEE ASSP Magazine 3, 4–16.  doi:https://doi.org/10.1109/MASSP.1986.1165342.
  * Schnaubelt et al. (2020) Schnaubelt, M., Fischer, T.G., Krauss, C., 2020.  Separating the signal from the noise – financial machine learning for twitter.  Journal of Economic Dynamics and Control 114, 103895.  doi:https://doi.org/10.1016/j.jedc.2020.103895.
  * Wang and Wang (2018) Wang, G., Wang, Y., 2018.  Herding, social network and volatility.  Economic Modelling 68, 74–81.  doi:https://doi.org/10.1016/j.econmod.2017.04.018.


# Depth-CUPRL: Depth-Imaged Contrastive Unsupervised Prioritized
Representations in Reinforcement Learning for Mapless Navigation of Unmanned
Aerial Vehicles

Junior C. de Jesus1, Victor A. Kich2, Alisson H. Kolling2,
Ricardo B. Grando3, Rodrigo S. Guerra2, Paulo L. J. Drews-Jr1 1Junior C. de
Jesus and P. L. J. Drews-Jr are with the NAUTEC, Centro de Ciencias
Computacionais, Universidade Federal do Rio Grande - FURG, RS, Brazil.
<EMAIL_ADDRESS>A. Kich, Alisson H. Kolling and Rodrigo S. Guerra are
with the Universidade Federal de Santa Maria - UFSM, RS, Brazil.
<EMAIL_ADDRESS>B. Grando is with Technological University of
Uruguay, Rivera, Uruguay<EMAIL_ADDRESS>
###### Abstract

Reinforcement Learning (RL) has presented an impressive performance in video
games through raw pixel imaging and continuous control tasks. However, RL
performs poorly with high-dimensional observations such as raw pixel images.
It is generally accepted that physical state-based RL policies such as laser
sensor measurements give a more sample-efficient result than learning by
pixels. This work presents a new approach that extracts information from a
depth map estimation to teach an RL agent to perform the mapless navigation of
Unmanned Aerial Vehicle (UAV). We propose the Depth-Imaged Contrastive
Unsupervised Prioritized Representations in Reinforcement Learning(Depth-
CUPRL) that estimates the depth of images with a prioritized replay memory. We
used a combination of RL and Contrastive Learning to lead with the problem of
RL based on images. From the analysis of the results with Unmanned Aerial
Vehicles (UAVs), it is possible to conclude that our Depth-CUPRL approach is
effective for the decision-making and outperforms state-of-the-art pixel-based
approaches in the mapless navigation capability.

## SUPPLEMENTARY MATERIAL

Video of the experiments is available at: https://youtu.be/-iNnP3HLXmI.
Released code at: https://github.com/dranaju/curl_navigation.

##  I INTRODUCTION

The problem of autonomous robotics navigation can be defined in the idea of
perceiving the environment and planning a path that leads a robot to avoid
places that may represent a specific danger[1]. This first principle can be
solved by using senses to tell the robot’s environment; examples are laser
sensors, cameras, or depth maps. In the second principle for an autonomous
vehicle, it is necessary to use systems to plan the robot’s path to avoid
dangerous situations, such as collisions and unsafe conditions. Linking these
principles of navigation problems to UAVs, it is possible to see the great
difficulty found in contrasting means of performance and perception [2].

Deep Reinforcement Learning (Deep-RL) is a promising approach to tackle the
problem of autonomous robotics navigation. In this so-called mapless
navigation [3], the methods are capable of providing an action response for
continuous systems [4, 5], as well as the Deep-RL control of UAVs in simulated
environments [6]. It is possible to see the application of these techniques
for the mapless navigation of UAVs in models of unknown dynamics in many
environments[7, 8, 9, 10, 6]. However, one of the significant challenges of
Deep-RL techniques is the limitation given by the immense need for labeled
data which is difficult to collect in the real world [11]. It has been
empirically observed that high-dimensional Deep-RL observations with images of
raw pixels are inefficient in terms of sample [12]. Knowing this inefficiency
case in RL, better techniques to extract information from images are required.

Figure 1: System of Depth-CUPRL: The observations are passed through a network
that generates the depth images. These depth images are stored in a replay
memory, and it is sampled memories in the buffer to make the Contrastive and
Reinforcement Learning.

Since depth maps are a well-known source of perception data for navigation, we
propose a Deep-RL system to perform the mapless navigation of an UAV through
features extracted from a depth map image. We propose the Depth-Imaged
Contrastive Unsupervised Prioritized Representations in Reinforcement Learning
(Depth-CUPRL) that considers information from the depth of images with a
prioritized replay memory to perform the navigation of an UAV. We show that a
combination of Deep-RL and Contrastive Learning leads to better performance
when compared with other approaches of Deep-RL based on images. Fig. 1 shows
the system of states $D(I_{t},I_{t-1},I_{t-2})$ that will be used in networks
for training and its architecture.

The main contributions of this work are listed as follows:

  * •

We present a novel methodology based on depth images and contrastive learning
that can successfully perform mapless navigation and obstacle avoidance for an
UAV that manages to outperform image-based Deep-RL algorithms;

  * •

We also show that taking into account information from depth images we are
able to increase the performance by 43.9% in success rate in the best case
(ours) and by 26.9% in the worst depth-based method used by comparison.

  * •

We also show that prioritized experience replay memory further enhances
contrastive learning and generalization ability in deep-RL networks,
increasing the performance by 5.6% in success rate.

The paper is divided as follows: Section II presents a review of the relevant
works in the field. After that, Sections III and IV present details related to
the proposed methodology as well as a discussion of the developed tools.
Section V, the description of the vehicle and the environment are made and the
results obtained are presented. Finally, our contribution and future works are
summarized in Section VII.

##  II RELATED WORKS

The study around mapless navigation is extensively explored using terrestrial
mobile robots [3]. Autonomous navigation of aerial mobile robots using Deep-RL
approaches is less frequent and mainly focused on approaches that avoid the
use of visual information [10, 6] or using simplified information, without
contrastive learning [7, 8, 9, 13].

Rodriguez _et al._ [7] implemented a Deep-RL approach to solve the problem of
landing a UAV on a base that could be in motion or static. Sampedro _et al._
[8] suggested that an approach based on the Deep-RL technique could perform a
task of Search and Rescue in closed scenarios. They are modified through a
Deep Deterministic Policy Gradient (DDPG) [4]. In the work of [14], it was
shown that approaches based on the Soft Actor-Critic (SAC) algorithm [5] is
more effective for robot navigation than approaches based on the DDPG
algorithm.

Knowing that Deep-RL from high-dimensional observations (state) with raw pixel
images is inefficient in terms of sample [15], Laskin _et al._ [16] presented
a technique capable of extracting high-level characteristics from images with
raw pixels for the control of Deep-RL networks through these extracted
features. In addition, the CURL method showed a better performance than the
state-of-the-art algorithms such as SAC when dealing with images. However,
CURL is a method that was only applied in game-like environments with a third-
person view in the environment.

For navigation for mobile aerial vehicles, it is possible to find works like
Thomas _et al._ [17] which presents an algorithm for an autonomous UAV using
RL with self-attentive models. It showed that it can effectively complete the
vehicle navigation even when subjected to varying inputs in the algorithm.
This work highlights how the navigation works with the input state data with
noise or modifications. He _et al._ [9] used a Lobula Giant Moment Detector
(LGMD) to simplify vision information for Deep-RL in navigation and obstacle
avoidance of UAV. It performed missions in a complex environment with $80\%$
of success rate.

The present work differs from the related works by using information from a
depth image in a contrastive learning-based approach. Our distance sensing
information is yielded from a neural network capable of depth estimation in
monocular images instead of a distance sensor such as a Lidar. We also differ
by avoiding the high-dimensional observations problem by presenting a
contrastive learning structure. Also, we outperform the state-of-the-art by
proposing a prioritized experience replay memory system that further enhances
our approach. To the best of our knowledge, we are the first to propose such a
contrastive learning system with prioritized experience replay.

##  III THEORETICAL BACKGROUND

This section details the information of tools and techniques used to develop
our work.

###  III-A SAC-Based Approach

The main idea of the SAC technique is to use approximation functions that can
learn policies of continuous action space, characterizing this method as a
stochastic actor-critic [5]. The SAC algorithm uses a neural network to
approximate a policy to the actor-network. The state value is used to estimate
a value network $V$ and a value $Q$ with a critic network. These three
networks calculate the action prediction for the current state and generate a
time difference error signal for each time interval step. In addition,
intending to seek the maximization of the system’s rewards, the SAC also seeks
to maximize the entropy of the policy. Entropy refers to how unpredictable a
variable can be. If a random variable still assumes a single value, then the
strategy has zero entropy, encouraging exploration by the agent.

In general, it is recommended to make use of a replay memory that can save the
experiences of the agent during the [18] training session. The states,
actions, rewards, and new states that an agent has experienced in previous
episodes are saved in a memory buffer, being marked as an experience. This
replay memory technique can disrupt temporal correlations between different
episodes in the training phase. A significant performance increase in off-
policy Deep-RL algorithms is noticeable even with short memories. Regardless
of the increase in application capabilities with Deep-RL, replay memories can
affect an agent’s learning time.

Despite the use of replay memories, another network for the target is used
with a time delay, called the target network. The update of target network
weights can be defined as:

$\theta^{\prime}\leftarrow\tau\theta+(1-\tau)\theta^{\prime}$ (1)

with $\tau\ll 1$. This equation tells us that the target network is just a
copy of the main network using a Poliak-averaging [19] to update its
parameters that affect the learning stability of Deep-RL algorithms.

The SAC technique was used in the CURL as the learning method, and we used it
in our Depth-CUPRL approach as well. We also used this technique in a CNN-
based approach with the SAC algorithm (SAC CNN Prio.) to further enhance the
comparison with our proposed system.

###  III-B Contrastive Unsupervised Representations for Reinforcement Learning

Contrastive Unsupervised Representations for Reinforcement Learning (CURL) is
a general framework for combining contrastive learning with Deep-RL [16]. In
principle, any RL algorithm in the CURL pipeline can be used, either on-policy
or off-policy. The SAC technique is primarily used in this case [16]. CURL has
been shown to significantly improve sample efficiency over previous pixel-
based methods by performing contrastive learning simultaneously with an off-
policy RL algorithm.

A key component of CURL is the ability to learn rich representations of
dimensional data using contrastive unsupervised learning. Given a query $q$
and key $k$, the objective of contrastive learning is to secure that $q$
matches with $k$. A query and key are positive pairs if they are modified data
from the same observation $o(t)$, like an image observation. Fig. 2 shows that
CURL trains a visual representation $encoder$, ensuring that embeddings of
augmented versions of data $o_{q}$ and $o_{k}$ from observation $o(t)$ match
using a contrastive loss of the work from Van der Oord _et al._ [20].

Figure 2: CURL structure combines instance Contrastive Learning and
Reinforcement Learning.

The observations from the query $o_{q}$ are handled by encoder
$f_{\theta_{q}}$, returning $q=f_{\theta_{q}}(o_{q})$. All observations are
built from the minibatch ($i$) sampled from replay memory for the Deep-RL
update. The key is encoded and return $k=f_{\theta_{k}}(o_{k})$, where
$\theta_{k}$ is a Poliak-averaging version of the encoder query or
$\theta_{k}\leftarrow\tau\theta_{k}+(1-\tau)\theta_{q}$. The Deep-RL policy
and value function are built on top of the query encoder that is trained
together with the objectives of reinforcement learning and contrastive
learning:

$\mathcal{L}oss_{\theta_{q}}=\log\frac{\exp(q^{T}k_{0})}{\sum\nolimits_{0}^{i}\exp(q^{T}k_{i})}$
(2)

We further enhanced the CURL approach and added a prioritized replay memory
system to create ours. We implemented a version of the CURL classic and a
version with Depth Maps for a more extensively comparison. Both prioritized
replay memory system and unsupervised depth estimation techniques are
described below.

###  III-C Prioritized Replay Memory

In the replay of experience, the agent can use a replay memory that allows
data efficiency by storing past experiences of the agent to have the
opportunity to reprocess the data later. The data saved in memory refers to
the current state $s_{t}$, action $a_{t}$, reward $r_{t}$ and next state
$s_{t+1}$ of the agent. Furthermore, replay memories also ensure that updates
are made from reasonably stable data stored in memory which helps with network
convergence.

Figure 3: Unsupervised Depth estimation architecture. The depth network takes
the current as input and outputs a depth map $\widehat{D_{t}}$. The pose
network assists in the loss for the depth map estimation.

While replay memory allows processing data transitions in different orders
than if they were experienced, there is also the possibility to use [18]
prioritized memories. These prioritized experiences allow data transitions
with frequencies different from those experimentally tested depending on their
importance. That is to say, which experience is stored and which one is
repeated.

We added a prioritized replay memory system to our approach and to some of the
approaches that we used compared to. The idea is that such a prioritized
system increases the ability of the Deep-RL algorithm to learn the depth
representation.

###  III-D Unsupervised Depth Estimation

One hypothesis is that the human being developed a rich, structural
understanding of the world through our previous visual experience, which
consisted mainly of moving around and observing large numbers of scenes and
developing consistent models of our observations. From millions of such
observations, we learn about the regularities of the world - roads are flat,
buildings are straight, roads support cars, etc. We can apply this knowledge
to perceiving a new scene, even from a single monocular image [21].

Given this, works such as Bian _et al._ [22] explore learning to estimate
depth maps through image sequences. His approach is based on the insight that
a geometric view synthesis system performs consistently well when its
intermediate predictions of scene geometry and camera pose match the
fundamental physical truth. They proposed a framework for jointly training a
single-view depth CNN and a camera pose estimation CNN from unlabeled video
sequences, shown in Fig. 3. We adopted the network proposed by Bian _et al._
[22] to learn depth maps from monocular color images in our framework.

##  IV METHODOLOGY

In this work, we propose a contrastive prioritized Deep-RL system that
considers information from depth maps to perform the mapless navigation and
obstacle avoidance of an UAV. We named our system of Depth-CUPRL, as this
method is based on a CURL structure [16] using depth maps and prioritized
memories for navigation of an aerial vehicle. In the sequence, we explain our
proposed approach and the other approaches used in this work for comparison.

The Depth-CUPRL equation of motion is defined as:

$v_{t}=f^{Depth-CUPRL}(I_{t})$ (3)

where $I_{t}$ is the raw pixel-colored monocular input image and $v_{t}$ is
the velocity applied to the UAV. For Depth-CUPRL, the Equation 3 passes
through $Depth_{CNN}$ which generates a depth image of $I_{t}$, the CURL-based
network extracts information from this depth map and then passes through a
SAC-based network which gives the robot velocity. A schematic diagram of the
Depth-CUPRL architecture is illustrated in Fig. 1 and pseudo-code can be seen
in Algorithm 1. It uses a [22] dense block-based encoding network to generate
depth maps. The KITTI dataset [23] is used to train this network.

After training the Depth-CUPRL with the KITTI dataset, the network is
finetuned to improve depth estimation for the navigation environment. For
this, images are extracted in sequences of random actions to improve the
network’s performance that generates the scene’s depth. After that, the depth
estimated is stacked in 3 consecutive temporal frames as the input from the
encoder query and key, as shown in Figure 1. The images received in the
encoders that extract information return the latent spaces $q$ and $k$, where
$q$ is used in the Deep-RL network. The outputs of the Depth-CUPRL network are
the linear and angular velocities are used to high-level control the UAV.

Figure 4: Depth-CUPRL network architecture.

The network architecture used by the Deep-RL resembles the SAC network
presented in the works of de Jesus _et al._ [24], and Grando _et al._ [25].
Fig. 4 shows the architecture including the query encoders as convolutional
layers. The depth maps are processed by four convolutional layers and three
fully connected layers. The number of layers and nodes are based on the
proposal of Laskin _et al._ [16]. The output layer estimates angular and
linear velocity to be used in the robot. However, the action value is
normalized between $(-1;1)$ because of the hyperbolic tangent function
$(tanh)$, which is the activation function. Therefore, the angular velocity
and the linear velocity are also constrained between the ranges $-0.25$ to
$0.25$ $rad/s$ and $0$ to $0.22$ $m/s$ , respectively.

Algorithm 1 Depth-Imaged Contrastive Unsupervised Prioritized Representations
in Reinforcement Learning

1: Input: initial policy parameters $\phi$, Q-function parameters
$\theta_{1}$, $\theta_{2}$, empty replay buffer $M$, maximum number of
episodes $E$, maximum number of steps $T$

2: Image network: depth estimation network trained $D$, contrastive encoder
$\psi_{q}$ and $\psi_{k}$;

3: Set target parameters equal to main parameters
$\theta^{\prime}_{1}\leftarrow\theta_{1}$,
$\theta^{\prime}_{2}\leftarrow\theta_{2}$

4: for $episode=1$ until $E$ do

5: reset environment

6: for $t=1$ until $T$ do

7: Observe image state $I_{t}$

8: Image observation passed to depth map $d=D(I_{t})$

9: Depth map passed to $s=\psi_{q}(d)$

10: Observe state $s$ and select action $a\sim\pi_{\phi}(\cdot|s)$

11: Execute $a$ in the environment

12: Observe next state $s^{\prime}$, and reward $r$

13: Store $(s,a,r,s^{\prime},d)$ in replay buffer $M$

14: if update Reinforcement Learning then

15: Randomly sample a batch of transitions, $B=\\{(s,a,r,s^{\prime})\\}$ from
$M$

16: Compute targets for the Q functions:
$y(r,s^{\prime})=r+\gamma\left(\min_{i=1,2}Q_{\theta^{\prime}_{i}}(s^{\prime},a^{\prime})-\alpha\log\pi_{\phi}(a^{\prime}|s^{\prime})\right)$

17: Update the critic:
$\nabla_{\theta_{i=1,2}}\frac{1}{|B|}\sum_{(s,a,r,s^{\prime})\in
B}\left(Q_{\theta_{i}}(s,a)-y(r,s^{\prime})\right)^{2}$

18: Update policy:  $\nabla_{\phi}\frac{1}{|B|}\sum_{s\in
B}\left(\min_{i=1,2}Q_{\theta_{i}}(s,a_{\phi})-\alpha\log\pi_{\phi}\left(\left.a_{\phi}\right|s\right)\right)$

19: Updates target:
$\quad\theta^{\prime}_{i=1,2}\leftarrow\tau\theta_{i}+(1-\tau)\theta_{i}$

20: end if

21: if updade Constrative Learning then

22: Randomly sample batch of depth map $d$, $B(d)=\\{(s,a,r,s^{\prime},d)\\}$
from $M$

23: Random augmentation $aug()$ sample for $query$ and $key$:$\quad
d_{q}=aug(d)$$\quad d_{k}=aug(d)$

24: Computes the latent space of $query$ and $key$:$\quad
q=\psi_{q}(d_{q})$$\quad k=\psi_{k}(d_{k})$

25: Updates $query$:
$\mathcal{L}oss_{\psi_{q}}=\log\frac{\exp(q^{T}k_{0})}{\sum\nolimits_{i\in
B(d)}\exp(q^{T}k_{i})}$

26: Updates $key$:
$\quad{\psi_{k}}\leftarrow\tau{\psi_{k}}+(1-\tau){\psi_{k}}$

27: end if

28: end for

29: end for

###  IV-A Reward Function

The rewards and penalties system given to the agent is a simple one, based on
empirical knowledge created during the problem-solving process. Thus, the
network does a feed-forward and back-propagation step to learn the
hyperparameters. The reward function is formulated for the obstacle avoidance
task, and it is defined as:

$r(s_{t},a_{t})=\begin{cases}r_{navigating}\ \textrm{if}\ min_{x}\geq
c_{o},\\\ r_{collision}\ \textrm{if}\ min_{x}<c_{o},\end{cases}$ (4)

where a negative reward $(r_{collision}=-1)$ is given if the robot collides
with an obstacle. $c_{o}$ is equivalent to the robot’s distance of $62cm$ from
an obstacle or wall. These collision conditions cause the agent to be reset to
the initial training position. Furthermore, $r_{navigating}$ is given when the
robot is navigating the map, whose value was decided to be $0.01$. The
observation that the only thing we want the robot to avoid is colliding with
objects or walls defined the rewards without creating a misleading reward for
the agent to interpret the map [26].

###  IV-B Experimental Setup

(a) Real vehicle.

(b) Simulated vehicle.

Figure 5: Simulated vehicle based on the real Hydrone Vehicle.

The simulations are run with the Robot Operating System (ROS) [27] as backbone
managing the connections between the proposed methods and the simulations. The
chosen simulator is Gazebo for an easy connection with ROS and its realistic
physics simulation. The algorithms are developed using the PyTorch library and
OpenCV [28]. The robotic UAV platform used in the simulation is designed by
Grando et al. [29]. The vehicle is based on a real vehicle called Hydrone [30,
31]. The Hydrone vehicle is a quadrotor-like hybrid unmanned aerial underwater
vehicle, but only the aerial capabilities are evaluated here. Fig. 5 shows the
real vehicle and the final vehicle described in simulation. The model in ROS
can be controlled by linear and angular velocity in 6 degrees of freedom
$(x,y,z,roll,pitch,yaw)$.

For the vehicle’s base frame, the value of $1.93$ kg for mass is used. For the
collision of the frame, a box with dimensions of $0.47$ meters on a side and
$0.09$ meters in height was estimated. The four rotors are placed
symmetrically with an angle of $90^{\circ}$ between each and an exact distance
of $0.27$ meters from the center of the vehicle.

###  IV-C Simulated Environments

Both environments created in Gazebo can be seen in Figure 6. The proposed
environments are generated to show that the approach used in this work can
navigate without colliding with walls and obstacles. In the first environment,
four obstacles are allocated as illustrated in Figure 6(a), the addition of
these fixed obstacles produce a more challenging scenario for the robot.
Therefore, if the vehicle collides with the wall or any obstacle, a negative
reward is given for this action and the current episode stops. Fig. 6(b) show
a more complex navigation scenario. This environment presents a higher degree
of complexity than the first one since we add several obstacles.

(a) First environment.

(b) Second environment.

Figure 6: Simulated Environments used on Gazebo simulation.

(a)

(b)

Figure 7: Moving average of the reward over $100000$ and $300000$ training
steps for the first and second environment, respectively.

##  V EXPERIMENTAL RESULTS

This section presents the discussion of the results. Firstly, we present the
reward gathered during the training. This information allows the estimation of
the agent’s degree of learning in the environment, as the improved in terms of
reward is intrinsically linked to the agent’s performance in the environment
it must navigate. All environments used for training the network are provided
by the Grando _et al._ [29]. However, some changes are made in the source code
of the Gazebo simulation to use the camera of the simulated UAV.

It is important to note that a Deep-RL agent is trained for each proposed
environment. We compared our method Depth-CUPRL with the _CURL_ [16] having as
input the raw pixel image, and with an own version of the CURL called here
_CURL (Depth)_ with depth maps as input but without prioritized memory. We
also compared with a _SAC (CNN prio.)_ as adopted in Grando et al [25] but
with convolutional layers. The SAC (CNN prio.) uses depth maps as inputs of
the network and has prioritized memory. All networks tested in the work
followed the architecture proposed in Fig. 4. The initial position of the
vehicle for both training and testing in each of the scenarios is defined as
the Cartesian position (3.6, -2.6, 2.0) for the aerial environment. Only
linear and angular velocities are applied to the $yaw$ of the vehicle.

Our method is configured with a prioritized replay memory of $35000$ samples
for training in the first environment. Each episode has $1000$ steps $t$. The
network is evaluated at each $1000$ step. The evaluation of the networks is
carried out in such a way that it uses only the deterministic response of the
network without any noise for $10$ episodes. With this, it is possible to
evaluate the network’s response by an average of tests and having as the
maximum reward for evaluation of $10.00$.

The results of training the reward function of the first environment for the
proposed and tested networks are shown in Figure LABEL:fig:reward_1. In the
first episodes of training networks, negative rewards are noted. This
condition is because the algorithm has started and is still learning. This
episode reward means the robot is trying to maximize the task reward. All
networks are trained for $100000$ steps. It is possible to conclude that the
Depth-CUPRL, CURL (Depth), SAC (CNN prio.) networks successfully completed the
proposed task in this environment. Depth-CUPRL presents higher efficiency to
converge to the optimal reward. CURL based on RGB image [16] remains unstable,
failing to converge to an optimal result.

The results gathered in the second environment are shown in Fig.
LABEL:fig:reward_2. Given a higher complexity, a more significant number of
training interactions is necessary for this environment. We adopted a replay
memory of $140000$ and all methods are trained for almost $300000$ steps. If
compared with the previous reward functions, from Fig. LABEL:fig:reward_1, it
is possible to notice a more unstable reward for the trained networks. Despite
this, the Depth-CUPRL obtains the highest average reward. Thus, we can see the
advantage of using our technique in navigation and obstacle avoidance.

##  VI DISCUSSION

Table I: Comparison between our approach and other approach in the first environment. Bold letters highlights the best results. Algorithm | Image | Success | Crash | Success Rate (%)
---|---|---|---|---
CURL (Depth) | Depth | 1000 | 0 | 100%
CURL (Classic) | Pixel | 856 | 144 | 85.6%
SAC (CNN prio.) | Depth | 1000 | 0 | 100%
Depth-CUPRL | Depth | 1000 | 0 | 100%

After training in the first environment, a final model evaluation is performed
after the 1000 episodes. Table I shows the results obtained for all methods.

In this evaluation, it is possible to see that the algorithms that used
information from depth images were able to complete the 1000 tests with 100%
success. Specifically, CURL (Classic), which uses pixel-based images, crashed
many times in the evaluation. We can see in these results that the depth map
images approach had a better efficiency in navigating through the first
environment.

After training the approaches in the second environment, a final evaluation is
performed at the end of the 1000 episodes. Table II shows the obtained results
for all approaches.

Table II: Comparison between proposal and other networks in the second environment. Bold letters highlights the best results. Algorithms | Image | Success | Crash | Success Rate (%)
---|---|---|---|---
CURL (Depth) | Depth | 936 | 64 | 93.6%
CURL (Classic) | Pixel | 553 | 447 | 55.3%
SAC (CNN prio.) | Depth | 822 | 178 | 82.2%
Depth-CUPRL | Depth | 992 | 8 | 99.2%

In this evaluation, the depth image approaches, collaborating with the results
of Table I, had a better performance. Yet, none of the approaches were
completed the scenario without a crash. Our approach Depth-CUPRL got the best
performance crashing only 8 times of 1000, showing an overall better
performance of 5.6% in success rate. Only the approach that could not get a
success rate above 80% was the pixel-based method.

Taking into account information from depth images, we are able to increase the
performance by 43.9% in success rate in the best case (ours) and by 26.9% in
the worst depth-based method used by comparison (SAC (CNN prio)).

##  VII CONCLUSION AND FUTURE WORK

In this work, we presented a novel Deep-RL based on the information of the
estimated image depth of a simple monocular camera system to perform the
mapless navigation and obstacle avoidance of UAVs. We evaluated our approach
in two realistic scenario showing the Depth-CUPRL approach outperforms the
state-of-the-art in terms of image-based RL approach. The use of depth maps is
a commonplace in robotics and we show the advantage to use it in Deep-RL
methods instead of raw pixel images.

With our approach, the generated depth maps are used as inputs in a network
that learns these representations by image sequences. After that, this
information is extracted and the Deep-RL agent trained to perform the
vehicle’s navigation and make it wise enough to avoid collisions with objects.
The results show that the reward increases with the number of training
episodes in the environment.

The next steps will be focused in apply our approach also to the underwater
navigation of the Hydrone [30, 25] using underwater depth maps [32]. For these
underwater scenarios, we intended to improve the depth estimation method to
handle a high turbidity environment such as water and still be effective when
in an aerial environment. Furthermore, real-world results are being conduced.

## ACKNOWLEDGEMENTS

The authors would like to thank the VersusAI team. This work was partly
supported by the Technological University of Uruguay (UTEC), Federal
University of Santa Maria (UFSM), Federal University of Rio Grande (FURG), and
PRH-ANP.

## References

  * [1] S. F. Alves, J. M. Rosario, H. Ferasoli Filho, L. Rincon, R. Yamasaki, and A. Barrera, “Conceptual bases of robot navigation modeling, control and applications,” _Advances in Robot Navigation_ , p. 26, 2011.
  * [2] R. B. Grando, P. M. Pinheiro, N. P. Bortoluzzi, C. B. da Silva, O. F. Zauk, M. O. Piñeiro, V. M. Aoki, A. L. Kelbouscas, Y. B. Lima, P. L. Drews-Jr, and A. A. Neto, “Visual-based autonomous unmanned aerial vehicle for inspection in indoor environments,” in _IEEE LARS/SBR_ , 2020, pp. 1–6.
  * [3] L. Tai and M. Liu, “Towards cognitive exploration through deep reinforcement learning for mobile robots,” _arXiv preprint arXiv:1610.01733_ , 2016.
  * [4] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” in _ICLR_ , 2016.
  * [5] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _ICML_ , vol. 80, 2018, pp. 1861–1870.
  * [6] R. B. Grando, J. C. de Jesus, and P. L. Drews-Jr, “Deep reinforcement learning for mapless navigation of unmanned aerial vehicles,” in _IEEE LARS/SBR_ , 2020, pp. 1–6.
  * [7] A. Rodriguez-Ramos, C. Sampedro, H. Bavle, I. G. Moreno, and P. Campoy, “A deep reinforcement learning technique for vision-based autonomous multirotor landing on a moving platform,” in _IEEE/RSJ IROS_ , 2018, pp. 1010–1017.
  * [8] C. Sampedro, A. Rodriguez-Ramos, H. Bavle, A. Carrio, P. de la Puente, and P. Campoy, “A fully-autonomous aerial robot for search and rescue applications in indoor environments using learning-based techniques,” _Journal of Intelligent & Robotic Systems_, pp. 601–627, 2019.
  * [9] L. He, N. Aouf, J. F. Whidborne, and B. Song, “Integrated moment-based LGMD and deep reinforcement learning for UAV obstacle avoidance,” in _IEEE ICRA_ , 2020, pp. 7491–7497.
  * [10] R. B. Grando, J. C. de Jesus, V. A. Kich, A. H. Kolling, and P. L. J. Drews-Jr, “Double critic deep reinforcement learning for mapless 3d navigation of unmanned aerial vehicles,” _Journal of Intelligent & Robotic Systems_, vol. 104, no. 2, pp. 1–14, 2022.
  * [11] R. Bonatti, R. Madaan, V. Vineet, S. Scherer, and A. Kapoor, “Learning visuomotor policies for aerial navigation using cross-modal representations,” in _IEEE/RSJ IROS_ , 2020, pp. 1637–1644.
  * [12] B. M. Lake, T. D. Ullman, J. B. Tenenbaum, and S. J. Gershman, “Building machines that learn and think like people,” _Behavioral and brain sciences_ , vol. 40, 2017.
  * [13] B. Li, Z. Gan, D. Chen, and D. Sergey Aleksandrovich, “Uav maneuvering target tracking in uncertain environments based on deep reinforcement learning and meta-learning,” _Remote Sensing_ , vol. 12, no. 22, p. 3789, 2020.
  * [14] J. C. de Jesus, V. A. Kich, A. H. Kolling, R. B. Grando, M. A. d. S. L. Cuadros, and D. F. T. Gamarra, “Soft actor-critic for navigation of mobile robots,” _Journal of Intelligent & Robotic Systems_, vol. 102, no. 2, pp. 1–11, 2021.
  * [15] L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine _et al._ , “Model-based reinforcement learning for atari,” _arXiv preprint arXiv:1903.00374_ , 2019\.
  * [16] M. Laskin, A. Srinivas, and P. Abbeel, “Curl: Contrastive unsupervised representations for reinforcement learning,” in _ICML_ , 2020, pp. 5639–5650.
  * [17] D.-G. Thomas, D. Olshanskyi, K. Krueger, T. Wongpiromsarn, and A. Jannesari, “Interpretable uav collision avoidance using deep reinforcement learning,” _arXiv preprint arXiv:2105.12254_ , 2021.
  * [18] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized experience replay,” in _4th Int. Conference on Learning Representations, ICLR_ , Y. Bengio and Y. LeCun, Eds., 2016. [Online]. Available: http://arxiv.org/abs/1511.05952
  * [19] B. T. Polyak and A. B. Juditsky, “Acceleration of stochastic approximation by averaging,” _SICON_ , vol. 30, no. 4, pp. 838–855, 1992.
  * [20] A. Van den Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” _arXiv preprint arXiv:1807.03748_ , 2018\.
  * [21] Z. Yin and J. Shi, “Geonet: Unsupervised learning of dense depth, optical flow and camera pose,” in _IEEE CVPR_ , 2018, pp. 1983–1992.
  * [22] J.-W. Bian, H. Zhan, N. Wang, Z. Li, L. Zhang, C. Shen, M.-M. Cheng, and I. Reid, “Unsupervised scale-consistent depth learning from video,” _IJCV_ , pp. 1–17, 2021.
  * [23] A. Geiger, P. Lenz, C. Stiller, and R. Urtasun, “Vision meets robotics: The KITTI dataset,” _IJRR_ , vol. 32, no. 11, pp. 1231–1237, 2013.
  * [24] J. C. de Jesus, J. A. Bottega, M. A. Cuadros, and D. F. Gamarra, “Deep deterministic policy gradient for navigation of mobile robots in simulated environments,” in _19th ICAR_ , 2019, pp. 362–367.
  * [25] R. B. Grando, J. C. de Jesus, V. A. Kich, A. H. Kolling, N. P. Bortoluzzi, P. M. Pinheiro, A. Alves Neto, and P. L. J. Drews-Jr, “Deep reinforcement learning for mapless navigation of a hybrid aerial underwater vehicle with medium transition,” in _IEEE ICRA_ , 2021, pp. 1088–1094.
  * [26] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press, 2018.
  * [27] M. Quigley, K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, A. Y. Ng _et al._ , “ROS: an open-source robot operating system,” in _IEEE ICRA workshop on open source software_ , vol. 3, no. 3.2, 2009, p. 5.
  * [28] G. Bradski, “The OpenCV library,” _Dr. Dobb’s Journal: Software Tools for the Professional Programmer_ , vol. 25, no. 11, pp. 120–123, 2000.
  * [29] R. B. Grando, J. C. de Jesus, V. A. Kich, A. H. Kolling, and P. L. J. Drews-Jr, “Double critic deep reinforcement learning for mapless 3d navigation of unmanned aerial vehicles,” _Journal of Intelligent & Robotic Systems_, vol. 104, no. 2, pp. 1–14, 2022.
  * [30] P. L. Drews-Jr, A. A. Neto, and M. F. Campos, “Hybrid unmanned aerial underwater vehicle: Modeling and simulation,” in _IEEE/RSJ IROS_ , 2014, pp. 4637–4642.
  * [31] A. C. Horn, P. M. Pinheiro, R. B. Grando, C. B. da Silva, A. A. Neto, and P. L. Drews-Jr, “A novel concept for hybrid unmanned aerial underwater vehicles focused on aquatic performance,” in _IEEE LARS/SBR_ , 2020, pp. 1–6.
  * [32] P. L. Drews-Jr, E. R. Nascimento, S. S. Botelho, and M. F. M. Campos, “Underwater depth estimation and image restoration based on single images,” _IEEE computer graphics and applications_ , vol. 36, no. 2, pp. 24–35, 2016\.