# Can Interpretable Reinforcement Learning Manage Assets Your Way?

Charl Maree and Christian Omlin
Center for Artificial Intelligence Research
University of Agder
Grimstad, Norway
<EMAIL_ADDRESS>
Second affiliation: Chief Technology Office, Sparebank 1 SR-Bank, Stavanger,
Norway.

###### Abstract

Personalisation of products and services is fast becoming the driver of
success in banking and commerce. Machine learning holds the promise of gaining
a deeper understanding of and tailoring to customers’ needs and preferences.
Whereas traditional solutions to financial decision problems frequently rely
on model assumptions, reinforcement learning is able to exploit large amounts
of data to improve customer modelling and decision-making in complex financial
environments with fewer assumptions. Model explainability and interpretability
present challenges from a regulatory perspective which demands transparency
for acceptance; they also offer the opportunity for improved insight into and
understanding of customers. Post-hoc approaches are typically used for
explaining pretrained reinforcement learning models. Based on our previous
modeling of customer spending behaviour, we adapt our recent reinforcement
learning algorithm that intrinsically characterizes desirable behaviours and
we transition to the problem of asset management. We train inherently
interpretable reinforcement learning agents to give investment advice that is
aligned with prototype financial personality traits which are combined to make
a final recommendation. We observe that the trained agents’ advice adheres to
their intended characteristics, they learn the value of compound growth, and,
without any explicit reference, the notion of risk as well as improved policy
convergence.

_K_ eywords AI in banking; personalized services; asset management;
explainable AI; reinforcement learning; policy regularisation

##  1 Introduction

Financial service providers are employing ever-advancing methods to improve
the level of personalisation of their services [1, 2]. Artificial intelligence
(AI) is a promising tool in this pursuit in areas such as anti-money
laundering, trading and investment, and customer relationship management [3].
Examples of _personalised_ services are recommender systems for product sales
[4], risk evaluation for credit scoring [5], and segmentation for customer-
centric marketing [6]. More commonly, AI has been applied to stock trading via
ensemble learning [7], currency recognition using deep learning [8], stock
index performance through time-series modelling with feature engineering [9],
and investment portfolio management using reinforcement learning (RL) [10,
11]. These applications generally lack the personalisation needed to enhance
customer relations and support service delivery for growing customer bases.
The lack of explainability and interpretability has thus far hindered the
wider adoption of machine learning, mainly due to model opacity; model
understanding is essential in financial services [12, 13, 14]. We distinguish
between explainability and interpretability: explainability refers to a
symbolic representation of the knowledge a model has learned, while
interpretability is necessary for reasoning about a model’s predictions.

We have previously investigated the _interpretability_ of systems of multiple
RL agents [15]. A regularisation term in the objective function imposed a
desired agent behaviour _during_ training. For our current purpose of asset
management, our agents learn distinct optimal policies for continuously
distributing a fixed monthly amount across five assets: a savings account,
property, stocks, luxury items, and mortgage repayments. It is our intention
to align the agents’ characteristics and behaviour with personality traits -
openness, conscientiousness, extraversion, agreeableness, and neuroticism - as
proposed for modelling spending behaviour in [16]. A linear combination of the
resulting policies can provide investment advice that matches each customer’s
unique personality profile. This intrinsic interpretability may fulfill the
promise of a digital private assistant for personal wealth management.

##  2 Related Work

Recent evidence has revealed a causal relationship between spending patterns
and individual happiness [17]: we are happiest when our spending matches our
personality. For instance, extraverted individuals typically prefer spending
at a bar rather than at a bookshop, while the opposite may apply to
introverts. Our premise is that spending personality traits can be carried
over to asset management: we are happiest when our investment matches our
personality. For instance, conscientious investors may prefer the
predictability of property over the volatility of stocks. This is consistent
with the high affinity of conscientious spenders towards residential mortgages
[17]. It is compelling to expand the notion of personality traits from
spending to wealth creation, i.e., to base personal investment advice on
historical spending behaviour [18, 19].

RL has been extensively applied to stock portfolio management [20, 21, 22, 23,
24, 25], but not yet to _holistic_ asset management; the lack of model
transparency may be a contributing factor. Interpretation of RL agents
typically follows model training [26, 27, 28]; our ambition is to _impose_ a
desired characteristic behaviour _during training_ , thus making it an
intrinsic property of the agent. Based on a prior that defines a desired
behaviour, we extend the deep deterministic policy gradient (DDPG [29])
objective function with a regularisation term [15]. Formally, for each agent
$i$, this objective function is given by:

$\displaystyle
J(\theta_{i})=\mathbb{E}_{o_{i},a_{i}\sim\mathcal{D}}\left[R_{i}(o_{i},a_{i})\right]-\lambda
L_{i}$ (1) $\displaystyle
L_{i}=\frac{1}{M_{i}}\sum_{j=0}^{M_{i}}\left[\mathbb{E}_{a\sim\pi_{\theta_{i}}}(a_{j})-(a_{j}|\pi_{0_{i}}(a))\right]^{2}$

where $\theta_{i}$ is a set of parameters governing the policy, $\mathcal{D}$
is the replay buffer, $R_{i}(o_{i},a_{i})$ is the reward for action $a_{i}$
with the partial sate observation $o_{i}$, $\lambda\in\mathbb{R}_{\geq 0}$ is
a scaling parameter, $M_{i}$ is the number of actions, and $\pi_{0_{i}}$ is
the prior that defines the desired behaviour of the agent. Note that the prior
is independent of the state, which simplifies it and thus makes it
interpretable; this is a departure from traditional policy regularisation
methods such as KL-regularisation and entropy regularisation which aim to
improve learning convergence instead [30, 31]. Traditional regularisation
encourages state space exploration by increasing the entropy of the policy,
whereas our method guides agents’ learning towards the prior and thus imposes
a desired characteristic behaviour.

##  3 Empirical Methodology

The aim of this work was to create an interpretable AI for personal investment
management. We selected five assets in which a customer could invest a monthly
amount over a duration of 30 years: a savings account, property, a portfolio
of stocks, luxury expenditures, and additional mortgage payments. We include
luxury expenditure to the portfolio under the premise that it may increase
customer satisfaction in their portfolios [17]. We define luxury items as any
expenditure that may appeal to a person’s personality profile; people scoring
high on openness might derive joy from spending money on travelling, people
scoring high on extraversion may prefer to spend money on festivities with
other people [17], while other luxury items such as cars or artwork are also
possible. We modelled the growth rates of assets according to historical index
data, which we describe below.

###  3.1 Modelling Assumptions

We continuously distribute funds into assets based on the indices of the S&P
500 [32], Norwegian property [33], and the Norwegian interest rate [34]. In
addition, we invest in mortgages and luxury items. We show this data for a
30-year period in Figure 1.

Figure 1: Three asset value indices for a period of 30 years: The S&P 500
stock index, the Norwegian property index, and the Norwegian interest rate
index. All indices are relative to their respective values on 1 January 1992.
While the stock index performs the best overall, it has the highest volatility
and therefore the highest risk. Conversely, the interest rate index has the
lowest risk but also the lowest growth.

We make a number of assumptions which limit the scope of the portfolio and
simplify investment choices to make the characterization of agent behaviour
and interpretation of investment strategies tractable.

######  Assumption 1.

Asset growth rates can be modelled by their respective asset indices, i.e., a
stock portfolio may be modeled by a major stock index - e.g., the S&P 500 -,
and an investment in property by its corresponding index.

The outright investment in indices such as S&P 500 is very common; it will
return the growth rates according to these indices. This is a conservative
assumption as stock portfolio optimization frequently outperforms indices,
which may serve as a performance measure of the investment strategy [25].

To give personalised advice, we depart from the premise that there is a mere
correlation between spending behaviour and happiness. We are expanding the
notion of the causal relationship of spending patterns and customer
satisfaction to chart an investment strategy and provide advice that is
aligned with customer personality [17]. We rated assets in terms of their risk
and expected return from the historical values of the indices, and their
liquidity, capital requirements, and novelty from domain experts. They also
scored the different personality types’ affinity for different assets from the
interval $[-1,+1]$. Individuals with a high degree of openness may prefer the
liquidity of stocks over mortgage payments, whereas neurotic individuals may
prefer savings accounts over stocks in their portfolio. These coefficients,
shown in Table 1, are the weighted sums of the asset ratings and affinities.

Table 1: Coefficients relating asset risk, expected return, liquidity, capital requirement, and novelty to prototypical personality traits: openness (O), conscientiousness (C), extraversion (E), agreeableness (A), and neuroticism (N). The values are in the range $[-1,1]$. Investment | O | C | E | A | N
---|---|---|---|---|---
Savings | -0.11 | 0.08 | -0.15 | 0.51 | 0.68
Property | -0.15 | 0.32 | -0.22 | -0.36 | -0.24
Stocks | 0.82 | -0.61 | 0.95 | 0.42 | 0.12
Luxury | 0.16 | -0.51 | -0.07 | -0.80 | -0.81
Mortgage | -0.72 | 0.72 | -0.52 | 0.23 | 0.25

We define a Markov decision process (MDP) for a multi-agent RL setting. The
_states_ consist of customer age111Customer age is normalized to a range of
$[0,1]$, the values of the assets222Asset values are scaled by $1:10^{6}$, and
two market indicators for each of the three indices, i.e., their mean asset
convergence divergence (MACD)333MACD here is the difference between the
26-month and the 12-month exponential moving average of a trend. which
predicts trend reversals and relative strength index
(RSI)444$RSI=100-100/(1+\frac{P_{x}}{N_{x}})$ where $P_{x}$ and $N_{x}$ are
the average positive and negative changes to the index values respectively,
for $x$ periods. which corrects for potential false predictions by MACD.
_Rewards_ are the changes in portfolio values between time steps and _actions_
are the continuous distribution of funds across the portfolio of assets. We
make an initial loan of 2 million NOK in a mortgage and monthly investments
over 30 years totalling 3.34 million NOK:

######  Assumption 2.

The initial values for a portfolio consist of a mortgage of NOK 2 million and
a property valued at NOK 2 million. All other assets have zero initial value.

It is easy to adjust these initial portfolio assignments for different
individuals.

######  Assumption 3.

We make consistent monthly investments of 10 000 Norwegian kroner (NOK).

This can be easily modified for individual customers’ contributions.

There is a priori no lower limit on the investment amounts:

######  Assumption 4.

Property investment does not require bulk payments, i.e., smaller investments
can be made through property funds, trusts, or crowdfunding.

While investment in physical real estate normally requires larger deposits, we
allow our agents to invest smaller amounts into the property market, i.e., a
fraction of the monthly investment contribution specified in Assumption 3.
This is not a strong assumption as it is possible to invest smaller amounts in
property indices, trusts, funds, etc.

We assign interest rates for savings accounts at 5-10% below, and those of
mortgage accounts at 5-10% over the interest index. Individuals younger than
35 years receive the more beneficial interest rate, as is common in Norwegian
banks. Luxury items experience a depreciation of 20% per year; the
depreciation of luxury items is highly variable and depends on the item, e.g.,
while artwork may appreciate, cars typically depreciate rapidly:

######  Assumption 5.

Luxury items depreciate at 20% per year.

Dividends are normally included in the calculation of indices and monthly
transactions are relatively infrequent compared to high frequency trading:

######  Assumption 6.

Any additional income from investments - such as dividend payouts or rental
income - as well as costs such as transaction costs and fund management costs
are ignored.

###  3.2 Agents

We train five DDPG agents, one for each of the five personality traits. Using
Equation (1) we regularise their objective functions with a prior derived from
their respective personality traits in Table 1, e.g., the openness prior
$\pi_{0}^{O}$ places the most weight on stocks and avoids mortgage repayments,
property investment, and savings, while the conscientiousness prior
$\pi_{0}^{C}$ places the most weight on mortgage repayments and avoids stocks
and luxury expenditure. These priors, shown in Table 2, are probability
distributions across the investment channels and therefore add up to one.

Table 2: Regularisation priors $\pi_{0}^{a}$ for each agent $a\in$ {openness (O), conscientiousness (C), extraversion (E), agreeableness (A), and neuroticism (N)}. Investment | $\pi_{0}^{O}$ | $\pi_{0}^{C}$ | $\pi_{0}^{E}$ | $\pi_{0}^{A}$ | $\pi_{0}^{N}$
---|---|---|---|---|---
Savings | 0.00 | 0.07 | 0.00 | 0.44 | 0.64
Property | 0.00 | 0.28 | 0.00 | 0.00 | 0.00
Stocks | 0.84 | 0.00 | 1.00 | 0.36 | 0.12
Luxury | 0.16 | 0.00 | 0.00 | 0.00 | 0.00
Mortgage | 0.00 | 0.65 | 0.00 | 0.02 | 0.24

Agents’ actor and critic neural networks each consist of two fully connected
feed-forward layers with 2000 nodes in each layer. The actor networks each
have a final soft-max activation layer while the critic networks have no final
activations. We tuned the hyperparameters using a one-at-a-time parameter
sweep resulting in learning rates of $0.004$ and $0.001$ for the actors and
critics respectively, target network update parameters of $\tau=0.05$, and
regularisation coefficients of $\lambda=2$. Training batch sizes were 256 time
steps and we sized the replay buffer to hold 2048 transitions. Each iteration
collected 256 time steps and completed two training batches.

##  4 Results

Each of our investment agents learns an optimal investment strategy for their
respective prototypical personality traits, for instance, openness. The final
portfolio values after 334 months of investing according to these policies are
shown in Table 3. Given the common total investment of 3.34 million NOK, the
compound annual growth rate varies between 5.8% and 7.8% which is the maximum
return possible if investing in stocks only.

Table 3: Portfolio values of the five optimal policies for each of the
prototypical personality traits.

Policy | Final portfolio value (NOK 1M)
---|---
Openness | 22.4
Conscientiousness | 18.8
Extraversion* | 27.7
Agreeableness | 20.5
Neuroticism | 16.4
Personal agent | 20.3

*This agent’s regularisation prior was coincidentally the same as the optimal monetary policy $\pi^{M}$ and it achieved the maximum possible final portfolio value.

Note that these personalised policies did not achieve the same final portfolio
value. In fact, the optimum policy in monetary terms $\pi^{M}$ in this case
would have been to always buy stocks as shown in Figure 2; this is the default
policy an agent will converge towards when personality traits are ignored.
However, we postulate that this is not the ideal personal financial advice to
give to all individuals; some customers may be more averse to risk and will
thus prefer to avoid volatility in their portfolio. Our personalized agent
takes into account such preferences and, e.g., it recommends property
investments rather than stock investments.

Figure 2: The return on investment at every time step, calculated as the index
value at the final time step divided by the index value at the current time
step. It is clear that S&P 500 has the greatest return on investment at every
time step, except for a brief period in ca. 2000 where it was marginally below
the property index. Therefore, the optimum monetary policy $\pi^{M}$ is to
always invest the maximum amount into stocks.

Thus far, our agents have each separately learned an optimal investment
strategy for each prototypical personality trait. The aggregate policy is the
weighted sum of these individually learned policies: a customer has a blend of
personality traits which can be represented as a vector with five entries with
values within the range $[-1,+1]$. We calculate the inner product of the
normalized personality vector and the prototypical policies to arrive at the
aggregate investment policy. We show a representative aggregate investment
policy for a customer with a random personality profile in Figure 3. We
observe that the openness agent is the only agent to recommend spending on
luxury items; this is to be expected because its regularisation prior
$\pi_{0}^{O}$ is the only one with a non-zero coefficient for luxury
purchases. We also observe that the conscientiousness agent recommends
investing in property in early stages, followed by rigorous loan repayments in
the second half of the investment period. This suggests that our agent has
learned the concept of compound growth and its utility for portfolio
optimization. By contrast, the extraversion agent was steadfast in purchasing
stocks only, which is consistent with its regularisation prior $\pi_{0}^{E}$.
Unlike the conscientiousness agent, the agreeableness and neuroticism agents
consistently recommend investing in savings towards the end of the investment
period. In the early stages of the investment period, the agreeableness and
neuroticism agents utilize compound growth to increase the portfolio value; in
the latter phases, their regimen changes and they prefer the safety of savings
accounts. This is noteworthy because although risk is not explicitly part of
either the reward or regularisation functions, it is consistent with
traditional financial advice, which decreases the risk level with age.
Repeated training produces consistent results. We intend to elucidate this
observation in future work.

Figure 3: Investment strategies for different prototypical personality traits:
Figures (a) through (e) show the fractions of monthly investments for
different assets. They reveal the distinct investment strategies with changing
asset preferences for the five prototypical personality traits. In Figure (f)
we illustrate the investment strategy for a fictitious customer with a random
personality profile [openness, conscientiousness, agreeableness, extraversion,
and neuroticism] = [0.22, 0.87, 0.21, 0.92, 0.49]. The customer invests in a
mixture of assets throughout the investment period.

We observe that training converges quickly to the desired behaviour (see
Figure 4); the contribution of the regularisation term decreases rapidly,
which implies that the agent is learning the intended behaviour. We show the
regularisation term for the extraversion agent where the regularisation prior
$\pi_{0}^{E}$ matches the optimum monetary policy $\pi^{M}$ in Figure 4(a).
Further training causes no instability as is often observed in the DDPG
algorithm [31]. We hypothesize that this may be due to the agent
characteristics imposed by our regularisation whose effect may be similar to
entropy regularisation [31].

The actions of any linear combination of these agents, i.e., any personal
agent, are _interpretable_ through the intrinsic characterizations, i.e.,
priors, of each of the regularized agents.

(a) Learning curve for the regularisation term of the extraversion agent. (b)
A typical learning curve for the regularisation term of other agents.

(c) Learning curve where the regularisation term temporarily falls in a local
minimum but subsequently converges to zero.

Figure 4: The regularisation term $L$ for three different runs. In (a) the
regularisation prior $\pi_{0}^{E}$ of the extraverted agent coincides with the
optimum monetary policy $\pi^{M}$ and the policy converges within 5 time
steps. (b) shows a typical training run for the other agents which converges
within 100-200 training steps. (c) shows a training run where the
regularisation term appears to fall in local minimum for a time, but
eventually finds the optimum after about 200 training steps.

##  5 Conclusions and Directions for Future Work

We have presented a novel application of training RL agents to exhibit desired
characteristics and behaviours in asset management. The method is based on the
regularisation of the policy _during_ training. Here, we use prototypical
personality traits - openness, conscientiousness, agreeableness, extraversion,
and neuroticism - to define a set of priors which express their affinity
towards different assets and thus impose different investment strategies. This
makes the agents’ behaviour explicit and thus offers an explanation for their
recommendations. Our agents learn distinct optimal strategies for the
continuous distribution of monthly investments across a portfolio of
investment assets. We have shown that the agents learned to optimize total
rewards while adhering to their distinct priors. This makes it possible to
interpret the agents’ investment strategies.

Unlike traditional DDPG algorithms which may diverge with continuous training,
our regularisation results in quick and robust convergence. This could become
relevant if RL agents undergo continuous training to give personalized
investment advice to customers. The justification of this observation will be
subject to future research.

Our agents have learned the concept and utility of compound growth rates and
risk avoidance, which form part of the interpretation of their investment
strategies. These are solely based on the regularisation priors which express
their personality traits; the reward function makes no reference to the
personality traits. While the notion of compound growth may emerge from the
reward function, we do not yet know whether the notion of risk avoidance is
connected to the reward function or regularisation.

Here, we have chosen a linear combination of different, separately trained
agents aligned with the prototypical personality traits to arrive at an
aggregate investment advice. In the future, we will investigate whether the
orchestration of these agents can be learned to approach the optimum monetary
policy. This aggregation will need an explanation as well as interpretation to
understand its impact on the investment strategy. The hierarchical
orchestration of prototypical agents will be learned from real customers’
personality profiles. This will result in an explainable and interpretable
personalized financial investment advisor.

#### Funding

This study was partially funded by The Norwegian Research Council; project
number 311465.

## References

  * [1] Matteo Stefanel and Udayan Goyal.  Artificial intelligence & financial services: Cutting through the noise.  Technical report, APIS partners, London, England, 2019.
  * [2] Sunanda Vincent Jaiwant.  Artificial intelligence and personalized banking.  In Garg Vikas and Richa Goel, editors, Handbook of Research on Innovative Management Using AI in Industry 5.0, pages 74–87. IGI Global, Bengaluru, India, 2022.
  * [3] Joost van der Burgt.  General principles for the use of artificial intelligence in the financial sector.  Technical report, De Nederlandsche Bank, Amsterdam, The Netherlands, 2019\.
  * [4] Oladapo Oyebode and Rita Orji.  A hybrid recommender system for product sales in a banking environment.  Journal of Banking and Financial Technology, 4:15–25, 2020.
  * [5] Siddharth Bhatore, Lalit Mohan, and Raghu Reddy.  Machine learning techniques for credit risk evaluation: a systematic literature review.  Journal of Banking and Financial Technology, 4:111–138, 06 2020\.
  * [6] Darshana Desai.  Hyper-personalization: An AI-enabled personalization for customer-centric marketing.  In Surabhi Singh, editor, Adoption and Implementation of AI in Customer Relationship Management, pages 40–53. IGI Global, Maharashtra, India, 2022.
  * [7] Dhanya Jothimani and Surendra Yadav.  Stock trading decisions using ensemble-based forecasting models: a study of the indian stock market.  Journal of Banking and Financial Technology, 3:113–129, 2019.
  * [8] Qian Zhang, Weiqi Yan, and Mohan Kankanhalli.  Overview of currency recognition using deep learning.  Journal of Banking and Financial Technology, 3:59–69, 2019.
  * [9] Tien-Yu Hsu.  Machine learning applied to stock index performance enhancement.  Journal of Banking and Financial Technology, 5:1–13, 01 2021.
  * [10] Petter Kolm and Gordon Ritter.  Modern perspectives on reinforcement learning in finance.  SSRN Electronic Journal, pages 1–28, 2019.
  * [11] Thomas G. Fischer.  Reinforcement learning in financial markets - a survey.  Technical report, Friedrich-Alexander University Erlangen-Nuremberg, Institute for Economics, 2018.
  * [12] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  Information Fusion, 58:82–115, 2020.
  * [13] Longbing Cao.  Ai in finance: Challenges, techniques and opportunities.  Banking & Insurance eJournal, 2021.
  * [14] Charl Maree, Jan Erik Modal, and Christian W. Omlin.  Towards responsible ai for financial transactions.  In 2020 IEEE Symposium Series on Computational Intelligence (SSCI), pages 16–21, 2020.
  * [15] Charl Maree and Christian Omlin.  Reinforcement learning your way: Agent characterization through policy regularization.  arXiv, 2201.10003, 2022.
  * [16] Joe Gladstone, Sandra Matz, and Alain Lemaire.  Can psychological traits be inferred from spending? Evidence from transaction data.  Psychological Science, 30:1–10, 2019.
  * [17] Sandra C. Matz, Joe J. Gladstone, and David Stillwell.  Money buys happiness when spending fits our personality.  Psychological Science, 27:715–725, 2016.
  * [18] Charl Maree and Christian W. Omlin.  Clustering in recurrent neural networks for micro-segmentation using spending personality.  In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1–5, 2021.
  * [19] Charl Maree and Christian W. Omlin.  Understanding spending behavior: Recurrent neural network explanation and interpretation (in print).  In IEEE Computational Intelligence for Financial Engineering and Economics, pages 1–7, 2022.
  * [20] S. M. Bartram, J. Branke, G. D. Rossi, and M. Motahari.  Machine learning for active portfolio management.  The Journal of Financial Data Science, 3(3):9–30, 2021.
  * [21] Emmanuel Jurczenko.  Machine Learning for Asset Management: New Developments and Financial Applications.  Wiley - ISTE, London, United Kingdom, 2020.
  * [22] Q.Y.E. Lim, Q. Cao, and C. Quek.  Dynamic portfolio rebalancing through reinforcement learning.  Neural Computing and Applications, 33(24):1–15, 2021.
  * [23] Michael Pinelis and David Ruppert.  Machine learning portfolio allocation.  The Journal of Finance and Data Science, 8:35–54, 2022.
  * [24] Adrian Millea.  Deep reinforcement learning for trading—a critical survey.  Data, 6(11):119, 2021.
  * [25] Charl Maree and Christian W. Omlin.  Balancing profit, risk, and sustainability for portfolio management (in print).  In IEEE Computational Intelligence for Financial Engineering and Economics, pages 1–8, 2022.
  * [26] A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez.  Explainability in deep reinforcement learning.  Knowledge-Based Systems, 214(106685):1–24, 2021.
  * [27] Lindsay Wells and Tomasz Bednarz.  Explainable AI and reinforcement learning: A systematic review of current approaches and trends.  Frontiers in Artificial Intelligence, 4:1–48, 2021.
  * [28] S. Gupta, G Singal, and D. Garg.  Deep reinforcement learning techniques in diversified domains: A survey.  Archives of Computational Methods in Engineering, 28:4715–4754, 2021.
  * [29] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv, 1509.02971, 2019.
  * [30] Brian D. Ziebart.  Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy.  PhD thesis, Machine Learning Department, Carnegie Mellon University, December 2010.
  * [31] Tuomas Haarnoja, Haoran Tang, P. Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In International Conference on Machine Learning (ICML), 2017.
  * [32] Yahoo Finance.  Historical data for S&P500 stock index, 2022.  https://finance.yahoo.com/quote/%5EGSPC/history?p=%5EGSPC, Accessed on 30/01/2022.
  * [33] Statistics Norway.  Table 07221 - Price index for existing dwellings, 2022.  https://www.ssb.no/en/statbank/table/07221/, Accessed on 30/01/2022.
  * [34] Norges Bank.  Interest rates, 2022.  https://app.norges-bank.no/query/#/en/interest, Accessed on 30/01/2022.


Riemann surface of $f$ and then sew up the cut edges along $[0,+\infty].$ Then
we obtain two Riemann surfaces that are only joint at $1\in S,$ and the
boundary curve of these two surfaces compose the boundary curve $\Gamma_{f}.$
This completes the proof. ∎

##  14\. Proof of the main theorem

We first prove the main theorem under certain conditions.

######  Lemma 14.1.

Let $f:\overline{\Delta}\rightarrow S$ be a normal mapping that is not fat and
satisfies (a)–(d) of Theorem 7.1. Then

$A(f,\Delta)<2L(f,\partial\Delta),$

and

$A(f,\Delta)<h_{0}L(f,\partial\Delta)-A(f,\Delta),$

where $h_{0}$ is given by (1.3).

###### Proof.

The second inequality follows from the first one directly, for $h_{0}>4.$ So
we only prove the first inequality.

Since $f$ is not fat, for each component $D$ of $\Delta\backslash
f^{-1}([0,+\infty]),$ $f(\partial D)\backslash[0,+\infty]\neq\emptyset,$ and
so by (d) the interior $\alpha_{0}$ of $\left(\partial
D\right)\cap(\partial\Delta)$ is not empty, and then by Theorem 7.1,

$L(f,\left(\partial D\right)\cap(\partial\Delta))>L(f,(\partial
D)\backslash\left(\partial\Delta\right)),$

$f(\overline{D})$ is contained in some hemisphere of $S,$ which implies

$A(f(D))<2\pi,$

and $f$ restricted to $D$ is a homeomorphism.

Then we have

(14.1) $2L(f,\left(\partial D\right)\cap(\partial\Delta))>L(f,\left(\partial
D\right)\cap(\partial\Delta))+L(f,(\partial
D)\backslash\left(\partial\Delta\right))=L(f,\partial D).$

If $L(f,\partial D)\geq 2\pi,$ then we have

$A(f,D)=A(f(D))<2\pi\leq L(f,\partial D),$

and if $L(f,\partial D)<2\pi,$ then by Theorem 4.3 we have

$A(f,D)<L(f,\partial D),$

and then by (14.1), in both cases we have

(14.2) $A(f,D)<2L(f,\left(\partial D\right)\cap(\partial\Delta)).$

By Theorem 7.1 it is clear that for any pair $D_{1}$ and $D_{2}$ of distinct
components of $\Delta\backslash f^{-1}([0,+\infty]),$ $\left(\partial
D_{1}\right)\cap(\partial\Delta)$ and $\left(\partial
D_{2}\right)\cap(\partial\Delta)$ contains at most two common points, and on
the other hand, by the same theorem, $\Delta\backslash f^{-1}([0,+\infty])$
contains only finitely many components. Thus, we have

$L(f,\partial\Delta)=\sum_{D}L(f,\left(\partial
D\right)\cap(\partial\Delta)),$

where the sum runs over all components $D$ of $\Delta\backslash
f^{-1}([0,+\infty])$. Then, summing up (14.2), we have

$A(f,\Delta)=\sum_{D}A(f,D)<2\sum_{D}L(f,(\partial
D)\cap(\partial\Delta))=2L(f,\partial\Delta).$

This completes the proof. ∎

######  Lemma 14.2.

Let $f:\overline{\Delta}\rightarrow S$ be a normal mapping that is fat and
satisfies (a)–(d) of Theorem 7.1. Then

$A(f,\Delta)\leq h_{0}L(f,\partial\Delta)-4\pi.$

where $h_{0}$ is given by (1.3).

###### Proof.

By Theorem 13.1, there exist normal mappings
$g_{j}:\overline{\Delta}\rightarrow S,j=1,2,\dots,n+1,$ such that for each
$j\leq n+1$ the followings hold.

(e) Each $g_{j}$ satisfies all assumptions (a)–(d) of theorem 7.1,
$j=1,2,\dots,n+1$.

(f) Each $g_{j}$ is not fat, $j=1,2,\dots,n+1$.

(g) $g_{j}$ maps the diameter $[-1,1]$ of $\overline{\Delta}$ homeomorphically
onto the real interval $\overline{0,1}$ in $S$ or onto $\overline{1,\infty}$
in $S.$

(h)
$A(f,\Delta)=4n\pi+\sum_{j=1}^{n+1}A(g_{j},\Delta),L(f,\partial\Delta)=\sum_{j=1}^{n+1}L(g_{j},\partial\Delta).$

Let $j$ be any positive integer with $j\leq n+1$ and consider the mapping
$g_{j}.$ By (e) and (f), Lemma 14.1 applies. Then we have

(14.3) $A(g_{j},\Delta)<2L(g_{j},\partial\Delta),$

and then

$\displaystyle 4\pi+A(g_{j},\Delta)$ $\displaystyle<$ $\displaystyle
2L(g_{j},\partial\Delta)+4\pi$ $\displaystyle=$
$\displaystyle(2+\frac{4\pi}{L(g_{j},\partial\Delta)})L(g_{j},\partial\Delta),$

and, considering that $2+\frac{4\pi}{L(g_{j},\partial\Delta)}\leq 4$ in the
case $L(g_{j},\partial\Delta)\geq 2\pi,$ we have

(k) If $L(g_{j},\partial\Delta)\geq 2\pi,$ then

$4\pi+A(g_{j},\Delta)<4L(g_{j},\partial\Delta).$

By Theorem 4.3, we have

(l) If $\sqrt{2}\pi\leq L(g_{j},\partial\Delta)<2\pi,$ then

$4\pi+A(g_{j},\Delta)<4L(g_{j},\partial\Delta).$

By (g) and Theorem 4.4, we have

(m) If $L(g_{j},\partial\Delta)<\sqrt{2}\pi,$ then

$4\pi+A(g_{j},\Delta)\leq h_{0}L(g_{j},\partial\Delta),$

where $h_{0}$ is given by (1.3).

Summarizing (k)–(m) and the fact that $h_{0}>4$, we have in any case

$4\pi+A(g_{j},\Delta)\leq h_{0}L(g_{j},\partial\Delta),j=1,\dots,n+1,$

and then by (h), we have

$\displaystyle A(f,\Delta)$ $\displaystyle=$ $\displaystyle
4n\pi+\sum_{j=1}^{n+1}A(g_{j},\Delta)$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{n+1}\left(4\pi+A(g_{j},\Delta)\right)-4\pi$
$\displaystyle\leq$ $\displaystyle
h_{0}\sum_{j=1}^{n+1}L(g_{j},\partial\Delta)-4\pi$ $\displaystyle=$
$\displaystyle h_{0}L(f,\partial\Delta)-4\pi.$

This completes the proof. ∎

###### Proof of the Main Theorem.

Let $f:\overline{\Delta}\rightarrow S$ be any nonconstant holomorphic mapping
such that $f(\Delta)\cap E=\emptyset.$ Then for any positive number

(14.4) $\varepsilon<\frac{1}{4h_{0}}\min\\{4\pi,A(f,\Delta)\\},$

there exists a Jordan domain $D\subset\Delta$ such that $f$ restricted to $D$
is a normal mapping,

(14.5) $A(f,D)\geq A(f,\Delta)-\varepsilon\mathrm{\ and\ }L(f,\partial
D)<L(f,\Delta)+\varepsilon,$

and the following condition holds.

(1) Each natural edge of the restricted mapping $f|_{\overline{D}}$ has
spherical length strictly less than $\pi$.

Let $h$ be a homeomorphism from $\overline{\Delta}$ onto $\overline{D}$ and
let $F=f\circ h.$ Then by (1) $F:\overline{\Delta}\rightarrow S$ is a normal
mapping satisfying the assumption of Theorem 12.1 with $A(F,\Delta)=A(f,D)\
$and $L(F,\partial\Delta)=L(f,\partial D).$ Then by (14.5) we have that

(14.6) $A(F,\Delta)\geq A(f,\Delta)-\varepsilon\ \mathrm{and}\
L(F,\partial\Delta)<L(f,\Delta)+\varepsilon;$

and by Theorem 12.1 there exist a number of $m$ normal mappings
$f_{j}:\overline{\Delta}\rightarrow S,j=1,\dots m,$ such that

(14.7) $\sum_{j=1}^{m}A(f_{j},\Delta)\geq A(F,\Delta)\ \mathrm{and\
}\sum_{j=1}^{m}L(f_{j},\partial\Delta)\leq L(F,\partial\Delta),$

and for each $j$ the followings hold.

(A) Each natural edge of the boundary curve
$\Gamma_{f_{j}}=f_{j}(z),z\in\partial\Delta$, has spherical length strictly
less than $\pi$.

(B) $\Gamma_{f_{j}}=f_{j}(z),z\in\partial\Delta$, is locally convex in
$S\backslash E.$

(C) $f_{j}$ has no branched point in $S\backslash E.$

Then, by Lemma 7.2, for the above $\varepsilon$ and each $j\leq m,$ there
exists a normal mapping $g_{j}:\overline{\Delta}\rightarrow S$ such that

(14.8) $A(g_{j},\Delta)\geq
A(f_{j},\Delta),L(g_{j},\partial\Delta)<L(f_{j},\partial\Delta)+\frac{\varepsilon}{m},$

and $g_{j}$ satisfies (d) in Theorem 7.1 and (A)–(C), say, $g_{j}$ satisfies
all hypotheses of Theorem 7.1. Then by Lemmas 14.1 and 14.2 we have

(14.9) $A(g_{j},\Delta)\leq
h_{0}L(g_{j},\partial\Delta)-\min\\{A(g_{j},\Delta),4\pi\\},j=1,\dots,m.$

On the other hand, if for some $j_{0}\leq m,$ $A(g_{j_{0}},\Delta)\geq 4\pi,$
then we have

$\sum_{j=1}^{m}\min\\{A(g_{j},\Delta),4\pi\\}\geq 4\pi,$

and if $A(g_{j},\Delta)<4\pi$ for all $j\leq m,$ then we have, by (14.8),
(14.7), (14.6) and (14.4), that

$\displaystyle\sum_{j=1}^{m}\min\\{A(g_{j},\Delta),4\pi\\}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{m}A(g_{j},\Delta)\geq\sum_{j=1}^{m}A(f_{j},\Delta)\geq
A(F,\Delta)$ $\displaystyle>$ $\displaystyle
A(f,\Delta)-\varepsilon>\frac{4h_{0}-1}{4h_{0}}A(f,\Delta);$

and thus, in both cases, we have

(14.10)
$\sum_{j=1}^{m}\min\\{A(g_{j},\Delta),4\pi\\}\geq\min\\{\frac{4h_{0}-1}{4h_{0}}A(f,\Delta),4\pi\\}.$

Summing up the inequalities of (14.9), by (14.10) we have

$\displaystyle\sum_{j=1}^{m}A(g_{j},\Delta)$ $\displaystyle\leq$
$\displaystyle\sum_{j=1}^{m}h_{0}L(g_{j},\partial\Delta)-\sum_{j=1}^{m}\min\\{A(g_{j},\Delta),4\pi\\}$
$\displaystyle\leq$
$\displaystyle\sum_{j=1}^{m}h_{0}L(g_{j},\partial\Delta)-\min\\{\frac{4h_{0}-1}{4h_{0}}A(f,\Delta),4\pi\\}.$

By (14.8), (14.7) and (14.6) we have

$\displaystyle\sum_{j=1}^{m}h_{0}L(g_{j},\partial\Delta)$ $\displaystyle<$
$\displaystyle\sum_{j=1}^{m}h_{0}L(f_{j},\partial\Delta)+\varepsilon h_{0}\leq
h_{0}L(F,\partial\Delta)+\varepsilon h_{0}$ $\displaystyle<$ $\displaystyle
h_{0}L(f,\Delta)+2\varepsilon h_{0},$

i.e.

(14.12)
$\sum_{j=1}^{m}h_{0}L(g_{j},\partial\Delta)<h_{0}L(f,\Delta)+2\varepsilon
h_{0}.$

By (14.6)–(14.8) we have

(14.13) $A(f,\Delta)\leq\sum_{j=1}^{m}A(g_{j},\Delta)+\varepsilon.$

Therefore, we have by (14)–(14.13)

$\displaystyle A(f,\Delta)$ $\displaystyle\leq$
$\displaystyle\sum_{j=1}^{m}A(g_{j},\Delta)+\varepsilon$ $\displaystyle\leq$
$\displaystyle\sum_{j=1}^{m}h_{0}L(g_{j},\partial\Delta)-\min\\{\frac{4h_{0}-1}{4h_{0}}A(f,\Delta),4\pi\\}$
$\displaystyle<$ $\displaystyle h_{0}L(f,\Delta)+2\varepsilon
h_{0}+\varepsilon-\min\\{\frac{4h_{0}-1}{4h_{0}}A(f,\Delta),4\pi\\}$
$\displaystyle\leq$ $\displaystyle
h_{0}L(f,\Delta)+\frac{2h_{0}+1}{4h_{0}}\min\\{A(f,\Delta),4\pi\\}-\min\\{\frac{4h_{0}-1}{4h_{0}}A(f,\Delta),4\pi\\},$

and considering that $h_{0}>4,$ we have

$A(f,\Delta)<h_{0}L(f,\Delta).$

It remains to show that the lower is sharp.

We give an example to show that $h_{0}$ given by (1.3) is a sharp lower bound
of the constant $h$ in (1.1).

As in Section 1, we denote by $D$ the spherical disk in $S$ with diameter
$\overline{1,\infty},$ the shortest path in $S$ from $1$ to $\infty,$ and for
$l\in[\pi,\sqrt{2}\pi]$ denote by $D_{l}$ the domain contained in the disk $D$
such that the boundary $\partial D_{l}$ is composed of two congruent circular
arcs, each of which has endpoints $\\{1,\infty\\}$ and spherical length
$\frac{l}{2}$. Then, $l=L(\partial D_{l})$ and by (1.4) and (1.5), the number
$h_{0}$ given by (1.3) is the maximum value of the function
$\frac{4\pi+A(D_{l})}{l}$ and

$h_{0}=\frac{4\pi+A(D_{l_{0}})}{l_{0}}$

for some $l_{0}\in(\pi,\sqrt{2}\pi)$.

It is clear that $D_{l_{0}},$ regarded as a domain in $\mathbb{C},$ is an
angular domain whose vertex is $1$ and bisector is the ray $[1,+\infty)$ in
$\mathbb{C}$. We denote by $2\theta_{0}$ the value of the angle of this
angular domain. Then it is clear that $\theta_{0}<\frac{\pi}{2}.$

Let $M_{1}$ be the angular domain in $\mathbb{C}$ defined by

$M_{1}=\\{re^{i\theta};\ r>0,0<\theta<\frac{\theta_{0}}{m}\\}$

and let $\Sigma_{1}$ be the angular domain in $\mathbb{C}$ defined by

$\Sigma_{1}=\\{1+re^{i\theta};\ 0<r<+\infty,-\theta_{0}<\theta<\pi\\}.$

Then it is easy to construct a homeomorphism $f_{0}$ from the closure
$\overline{M_{1}}$ of $M_{1}$ in $\overline{\mathbb{C}}$ onto the closure
$\overline{\Sigma_{1}}$ of $\Sigma_{1}$ in $\overline{\mathbb{C}},$ such that
$f_{0}$ maps the ray $\arg z=\frac{\theta_{0}}{m}$ onto the ray $\arg z=\pi,$
maps the interval $[0,1]$ onto itself increasingly, maps the ray $[1,+\infty]$
onto the ray

$z=1+re^{-i\theta_{0}},r\in[0,+\infty]$

and $f_{0}$ is holomorphic on $M_{1}.$

Let

$M_{2}=e^{\frac{\theta_{0}}{m}}M_{1}=\\{e^{\frac{\theta_{0}}{m}}z;\ z\in
M_{1}\\}.$

Then, by the Schwarz symmetry principle, we can extend $f_{0}$ to be an open
and continuous mapping $f_{1}$ from the closed angular domain

$A_{1}=\overline{M_{1}\cup M_{2}}=\\{z\in\overline{\mathbb{C}};0\leq\arg
z\leq\frac{2\theta_{0}}{m}\\}$

onto $\overline{\mathbb{C}}$ such that $f_{1}\ $maps the segments

$l_{k}=\left\\{re^{i\frac{k\theta_{0}}{m}},r\in[0,1]\right\\},k=0,1,$

homeomorphically onto the interval $[0,1],$ respectively; $f_{1}$ maps the
segments

$L_{k}=\left\\{re^{i\frac{k\theta_{0}}{m}},r\in[1,+\infty]\right\\},k=0,1,$

homeomorphically onto the segments

$l^{-}=\left\\{1+re^{-i\theta_{0}},r\in[0,+\infty]\right\\}$

and

$l^{+}=\left\\{1+re^{i\theta_{0}},r\in[0,+\infty]\right\\},$

respectively; and $f_{1}$ restricted to the domain $A_{1}$ is a holomorphic
mapping that covers the domain $D_{l_{0}}$ two times and covers the domain
$\mathbb{C}\backslash\overline{D_{l_{0}}}$ one times.

Let

$A_{1}^{\ast}=A_{1}^{\circ}\cup l_{0}^{\circ}\cup l_{1}^{\circ},$

where $A_{1}^{\circ}$ is the interior of $A_{1},$ and let

$A_{k}^{\ast}=e^{i\frac{2\left(k-1\right)\theta_{0}}{m}}A_{1}^{\ast}=\\{e^{i\frac{2\left(k-1\right)\theta_{0}}{m}}z;z\in
A_{1}^{\ast}\\},k=2,\dots,m.$

Then for each $k=1,\dots,m$ we can defined a continuous function $f_{k}$ on
$A_{k}^{\ast}$ inductively: $f_{k+1}$ is obtained from $f_{k}$ by Schwarz
symmetry principle cross the symmetry axis

$l_{k}=\\{re^{i\frac{2k\theta_{0}}{m}},r\in[0,1]\\}.$

Let $H^{+}$ be the upper half plane $\mathrm{Im}z>0$ in $\mathbb{C}$ and let

$K=H^{+}\backslash\cup_{k=1}^{m-1}\\{re^{i\frac{2k\theta_{0}}{m}},r\in[1,+\infty)\\}.$

Then $K=\left(\cup_{k=1}^{m}A_{k}^{\ast}\right)\backslash(l_{0}\cup l_{m})$
and $f_{1},\dots,f_{m}$ can be patched to be a holomorphic function $f$
defined on $K\cup(-\infty,+\infty)$, by the Schwarz symmetric principle. It is
clear that $K$ is a simply connected domain and there is a conformal mapping
$h$ from $\Delta$ onto $K$ such that $h$ can be extended to be a continuous
function $\widetilde{h}$ such that when $z$ goes around $\partial\Delta$ once,
$\widetilde{h}(z)$ describes the boundary section $(-\infty,+\infty)$ of $K$
once and the boundary sections
$\\{re^{i\frac{k\theta_{0}}{m}},r\in[1,+\infty)\\}$ twice for $k=1,\dots,m-1.$

Let $g=f\circ\widetilde{h}.$ Then
$g:\overline{\Delta}\rightarrow\overline{\mathbb{C}}$ is a continuous mapping
that is holomorphic in $\Delta$ and when we regard $g$ as a mapping from
$\overline{\Delta}$ to $S,$ $g$ restricted to $\Delta$ covers $S\backslash
D_{l_{0}}$ by $m$ times and covers $D_{l_{0}}$ by $2m$ times and the boundary
curve $\Gamma_{g}=g(z),z\in\partial\Delta,$ covers $\partial D_{l_{0}}$ by $m$
times and covers the shortest path $\overline{0,1}$ in $S$ from $0$ to $1$ by
$2$ times.

Then we have

$A(g,\Delta)=4m\pi+mA(D_{l_{0}})$

and

$L(g,\partial\Delta)=2L(\overline{0,1})+mL(\partial D_{l_{0}})=\pi+ml_{0},$

and then

$\frac{A(g,\Delta)}{L(g,\partial\Delta)}=\frac{4m\pi+mA(D_{l_{0}})}{\pi+mL(\partial
D_{l_{0}})}<\frac{4\pi+A(D_{l_{0}})}{l_{0}}=h_{0}.$

It is clear that as $m\rightarrow+\infty,$
$\frac{A(g,\Delta)}{L(g,\partial\Delta)}=\frac{4m\pi+mA(D_{l_{0}})}{\pi+mL(\partial
D_{l_{0}})}$ converges to $h_{0}.$ This completes the proof. ∎

## References

  * [1] L. Ahlfors, Zur Theorie der Üherlagerung-Sflächen, Acta Math., 65 (1935), 157-194.
  * [2] F. Bernstein, Über die isoperimetrische Eigenschaft des Kreises auf der Kugeloberfläche und in der Ebene, Math. Ann., vol. 60 (1905), pp. 117-136.
  * [3] J. Dufresnoy, Sur les. domaines couverts. parles valeurs d’une. fonction. méromorphe. ou. algébroide, Ann. Sci. École. Norm. Sup. 58. (1941),. 179-259.
  * [4] W.K. Hayman, Meromorphic functions, Oxford, 1964.
  * [5] T. Radó, The Isoperimetric Inequality on the Sphere, Amer. Jour. Math., Vol.57, No.4. (Oct.,1935), pp.765-770.