# Singular plane curves and their interplay with algebra and combinatorics

Piotr Pokora

###### Abstract

In this survey we focus on various aspects of singular complex plane curves,
mostly in the context of their homological properties and the associated
combinatorial structures. We formulate some demanding open problems that can
indicate new directions in research, for instance by introducing weak Ziegler
pairs of curve arrangements.

Keywords 14N20, 51B05, 51A45, 14N25, 32S25

Mathematics Subject Classification (2020) singular plane curves; minimal free
resolutions; combinatorics

###### Contents

  1. 1 Introduction
  2. 2 Freeness of plane curves
  3. 3 Weak-combinatorics versus strong combinatorics and NTC
    1. 3.1 NTC versus triangular line arrangements
    2. 3.2 NTC holds for conic-line arrangements with nodes, tacnodes, and ordinary triple points
  4. 4 Weak Ziegler pairs

##  1 Introduction

The main goal of the present survey is to provide a coherent and short
introduction to the world of singular plane curves, which is a subject lying
on the boundary of algebra, geometry, and combinatorics. Our idea here is to
explore very interesting deep properties of algebraic curves and their
importance in the context of very recent big open problems in the so-called
combinatorial algebraic geometry. By combinatorial algebraic geometry we mean
the field that is based on close bridges between combinatorics and algebraic
geometry, and where these fields of research cooperate in full symbiosis. The
Leitmotif for our discussion here is the Numerical Terao’s Conjecture (we will
write NTC for short) and this problem builds the core of the survey. Roughly
speaking, NTC predicts that the freeness of reduced plane curves is determined
by the weak-combinatorics. This conjecture is somehow surprising and
absolutely not obvious. We are aware of the fact that NTC does not hold in the
class of line arrangements in the complex projective plane, and we deliver a
very detailed discussion regarding this subject, but up to right now there is
no single counterexample to this conjecture when we extend our consideration
to curves such that irreducible components are no longer, and not only, of
degree one. This is a very mysterious thing and we do not know how to explain
this phenomenon. However, in order to get some feeling about this problem, it
seems quite natural to explore new classes of plane curves in order to
understand which geometric/combinatorial properties can have an impact on the
failure of NTC. In the last few years researchers focused exactly on that area
of studies, namely to study specific classes of curves that are free and to
explore their geometric properties. The first very natural class for such
consideration is the class of rational plane curve arrangements with quasi-
homogeneous singularities. The main motivation standing behind this choice is
very natural. First, it generalizes the framework of line arrangements, and
secondly in this class the total Tjurina number of a given curve is determined
by its weak-combinatorics.

Let us outline the structure of the present survey. In the first part we focus
on the notion of free plane curves. Then we focus on the weak-combinatorics
that can be attached to a given reduced plane curve. We will show that this is
a very delicate problem and why we cannot use the notion of (strong)
combinatorics that is defied for line arrangements verbatim to the world of
plane curves. Next, we focus on NTC and we present a detailed outline on that
subject emphasizing the recent developments. We conclude our note by
introducing the notion of weak Ziegler pairs and by presenting an example of
such a pair in the class of line arrangements.

I should emphasize that there is an excellent recent survey devoted to free
plane curves and their geometric properties by Alex Dimca [8], but our survey
is strictly oriented on NTC and weak-combinatorics of plane curves, so we will
stay parallel with respect to Alex’s survey and we hope that our presentation
will play a complementary role.

We work exclusively in the projective setting over the complex numbers. Many
computations presented in that survey are performed using `SINGULAR` [4].

##  2 Freeness of plane curves

Let $S:=\mathbb{C}[x,y,z]$ be the graded ring of polynomials with complex
coefficients, and for a homogeneous polynomial $f\in S$ let us denote by
$J_{f}$ the Jacobian ideal associated with $f$, that is, the ideal of the form
$J_{f}=\langle\partial_{x}\,f,\partial_{y}\,f,\partial_{z}\,f\rangle$. We
assume in the whole survey that our plane curves $C\,:f=0$ are always reduced.
We will need the following definition.

######  Definition 2.1.

Let $p$ be an isolated singularity of a polynomial $f\in\mathbb{C}[x,y]$.
Since we can change the local coordinates, assume that $p=(0,0)$. Furthermore,
the number

$\mu_{p}=\dim_{\mathbb{C}}\left(\mathbb{C}[x,y]/\bigg{\langle}\frac{\partial
f}{\partial x},\frac{\partial f}{\partial y}\bigg{\rangle}\right)$

is called the (local) Milnor number of $f$ at $p$.

The number

$\tau_{p}=\dim_{\mathbb{C}}\left(\mathbb{C}[x,y]/\bigg{\langle}f,\frac{\partial
f}{\partial x},\frac{\partial f}{\partial y}\bigg{\rangle}\right)$

is called the (local) Tjurina number of $f$ at $p$.

######  Remark 2.2.

For a projective situation, with a point $p\in\mathbb{P}^{2}_{\mathbb{C}}$ and
a homogeneous polynomial $F\in\mathbb{C}[x,y,z]$, we can take local affine
coordinates such that $p=(0,0,1)$, and then the dehomogenization of $F$.
Finally, the total Tjurina number of a given reduced curve
$C\subset\mathbb{P}^{2}_{\mathbb{C}}$ is defined as

$\tau(C)=\sum_{p\in{\rm Sing}(C)}\tau_{p}.$

We will work mostly with quasi-homogeneous singularities.

######  Definition 2.3.

A singularity is called quasi-homogeneous if and only if there exists a
holomorphic change of variables so that the defining equation becomes weighted
homogeneous.

Recall that $f(x,y)=\sum_{i,j}c_{i,j}x^{i}y^{j}$ is weighted homogeneous if
there exist rational numbers $\alpha,\beta$ such that
$\sum_{i,j}c_{i,j}x^{i\cdot\alpha}y^{j\cdot\beta}$ is homogeneous. One can
show that if $f(x,y)$ is a convergent power series with an isolated
singularity at the origin, then $f(x,y)$ is in the ideal generated by the
partial derivatives if and only if $f$ is quasi-homogeneous. It means that in
the quasi-homogeneous case one has $\tau_{p}=\mu_{p}$, i.e., the local Tjurina
number of $p$ is equal to the local Milnor number of $p$. Moreover, if
$C\,:f=0$ is a reduced plane curve with only quasi-homogeneous singularities,
then

$\tau(C)=\sum_{p\in{\rm Sing}(C)}\tau_{p}=\sum_{p\in{\rm
Sing}(C)}\mu_{p}=\mu(C),$

which means that the total Tjurina number of $C$ is equal to the total Milnor
number of $C$.

Before we define the ultimate class of curves in our investigations, i.e.,
free plane curves, we introduce the following general definition.

######  Definition 2.4.

Let $C:f=0$ be a reduced curve in $\mathbb{P}^{2}_{\mathbb{C}}$ of degree $d$
given by $f\in S$. Denote by $M(f):=S/J_{f}$ the Milnor algebra. We say that
$C$ is $m$-syzygy when $M(f)$ has the following minimal graded free
resolution:

$0\rightarrow\bigoplus_{i=1}^{m-2}S(-e_{i})\rightarrow\bigoplus_{i=1}^{m}S(1-d-d_{i})\rightarrow
S^{3}(1-d)\rightarrow S$

with $e_{1}\leqslant e_{2}\leqslant...\leqslant e_{m-2}$ and $1\leqslant
d_{1}\leqslant...\leqslant d_{m}$.

######  Definition 2.5.

The $m$-tuple $(d_{1},...,d_{m})$ in Definition 2.4 is called the exponents of
$C$.

In the light of Definition 2.4, being $m$-syzygy curve is a homological
condition that is decoded by the shape of the minimal free resolution of the
Milnor algebra. Among $m$-syzygy curves, the most crucial one, from our very
subjective viewpoint, is the following.

######  Definition 2.6.

We say that a reduced curve $C\subset\mathbb{P}^{2}_{\mathbb{C}}$ of degree
$d$ is free if and only if $C$ is $2$-syzygy, and in that case
$d_{1}+d_{2}=d-1$.

In other words, a reduced plane curve $C\subset\mathbb{P}^{2}_{\mathbb{C}}$ is
free if the corresponding minimal free resolution of the Milnor algebra is
Hilbert-Burch. Now we want to present a geometric approach showing that the
freeness of plane curves is decoded by their total Tjurina numbers and one
special exponent.

######  Definition 2.7.

Let $C\subset\mathbb{P}^{2}_{\mathbb{C}}$ be a reduced curve given by $f\in
S_{d}$. We define the graded $S$-module of algebraic relations associated with
$f$ as

${\rm AR}(f)=\\{(a,b,c)\in S^{\oplus 3}:af_{x}+bf_{y}+cf_{z}=0\\}.$

Then the minimal degree of Jacobian relations among the partial derivatives of
$f$ is defined as

${\rm mdr}(f):=\min\\{r:\,{\rm AR}(f)_{r}\neq 0\\}.$

######  Remark 2.8.

In the light of Definition 2.4, we have

${\rm mdr}(f):=d_{1}.$

######  Remark 2.9.

Sometimes we also write ${\rm mdr}(C)={\rm mdr}(f)$, where $C:f=0$.

######  Example 2.10.

Let us consider the arrangement
$\mathcal{L}\subset\mathbb{P}^{2}_{\mathbb{C}}$ given by

$Q(x,y,z)=xyz.$

Using the language of the elementary projective geometry, $\mathcal{L}$ is the
fundamental triangle consising of $3$ nodes as the intersection points located
at the three fundamental points of $\mathbb{P}^{2}_{\mathbb{C}}$. Obviously
${\rm mdr}(Q)>0$ since there are no
$\alpha,\beta,\gamma\in\mathbb{C}\setminus\\{0\\}$ such that

$\alpha\partial_{x}f+\beta\partial_{y}f+\gamma\partial_{z}f=\alpha yz+\beta
xz+\gamma xy=0.$

On the other hand, it is easy to see that ${\rm mdr}(Q)=1$ since we have, for
instance, the following relation

$x\partial_{x}f-y\partial_{y}f=0.$

As we have seen so far, to show that a given reduced plane curve $C\,:f=0$ is
free, we need to compute the minimal free resolution of $M(f)$, which can be
difficult without a reasonable computer assistance. However, there is an
interesting result, which can be considered as a folkloric result, that might
be use successfully in some case. This result is known in the literature as
Saito’s criterion [6].

######  Theorem 2.11.

Let $C:f=0$ be a reduced curve in $\mathbb{P}^{2}_{\mathbb{C}}$ of degree $d$.
Let $r_{1},r_{2}\in{\rm AR}(f)$ be two non-trivial syzygies of the form
$r_{i}=(f_{1}^{i},f_{2}^{i},f_{3}^{i})$ with $i\in\\{1,2\\}$. Then $C$ is free
if and only if

${\rm Det}\begin{pmatrix}x&f_{1}^{1}&f_{1}^{2}\\\ y&f_{2}^{1}&f_{2}^{2}\\\
z&f_{3}^{1}&f_{3}^{2}\end{pmatrix}=c\cdot f,$

where $c$ is a non-zero constant.

It is worth recalling that the above Saito’s criterion holds in a very general
setting, i.e., we can formulate it even for reduced hypersurfaces
$V\subset\mathbb{P}^{N}_{\mathbb{C}}$, and it is a quite effective tool
provided that we see appropriate candidates for syzygies $r_{1},r_{2}$.

######  Example 2.12.

Let us come back to our canonical example of the fundamental triangle
$\mathcal{L}:xyz=0$. Observe that we have the following two natural syzygies:

$r_{1}=(x,-y,0),\quad r_{2}=(x,0,-z).$

Observe that

${\rm Det}\begin{pmatrix}x&x&x\\\ y&-y&0\\\ z&0&-z\end{pmatrix}=3xyz,$

hence $\mathcal{L}$ is free with exponents $(d_{1},d_{2})=({\rm
mdr}(Q),d-1-{\rm mdr}(Q))=(1,1)$.

As we have mentioned, Saito’s criterion is an effective tool provided that we
can find two suitable syzygies. On the other hand, the freeness of reduced
algebraic curves can be checked via the following deep result by du Plessis
and Wall [12].

######  Definition 2.13 (Freeness criterion).

Let $C\,:f=0$ be a reduced curve in $\mathbb{P}^{2}_{\mathbb{C}}$ of degree
$d$. Then the curve $C$ with $d_{1}={\rm mdr}(f)\leqslant(d-1)/2$ is free if
and only if

$(d-1)^{2}-r(d-r-1)=\tau(C).$ (1)

The reader can easily use the above criterion to verify, once again, that the
fundamental triangle $\mathcal{L}:xyz=0$ is free.

Finally, let us define the second most important class of curves in our
considerations, namely nearly-free curves.

######  Definition 2.14.

A reduced plane curve $C\subset\mathbb{P}^{2}_{\mathbb{C}}$ of degree $d$ is
nearly-free if $C$ is $3$-syzygy with $d_{1}+d_{2}=d$ and $d_{2}=d_{3}$.

This definition is somehow complicated to handle, but we have a very useful
result due to Dimca that allows us to check the nearly-freeness using the
information about the total Tjurina number and the minimal degree of the
Jacobian relations [7].

######  Theorem 2.15.

A reduced curve $C:f=0$ in $\mathbb{P}^{2}_{\mathbb{C}}$ of degree $d$ with
$d_{1}={\rm mdr}(f)$ is nearly free if and only if

$(d-1)^{2}-d_{1}(d-d_{1}-1)=\tau(C)+1.$ (2)

##  3 Weak-combinatorics versus strong combinatorics and NTC

In this section we elaborate about the notion of the weak-combinatorics and
strong combinatorics that can be attached to a reduced plane curve. We start
with the strong combinatorics that is defined for line arrangements.

######  Definition 3.1.

Let $\mathcal{L}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be an arrangement of $d$
lines. Then by the intersection lattice $L(\mathcal{L})$ we mean the set of
all flats, i.e., non-empty intersections of (sub)families of lines in
$\mathcal{L}$ with the order defined by the revers inclusion, i.e.,
$X\leqslant Y$ if and only if $Y\subset X$.

It turns out that the information delivered by the intersection lattice of a
given line arrangement $\mathcal{L}$ can be equivalently decoded by the so-
called Levi graph. Here we take a step forward and define the notion of the
Levi graph for arrangements consisting of smooth curves and admitting
arbitrary singularities.

######  Definition 3.2.

Let $\mathcal{C}=\\{C_{1},...,C_{k}\\}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be a
reduced curve consisting of $k$ smooth components, each of degree ${\rm
deg}\,C_{i}=d_{i}\geqslant 1$. Then the _Levi graph_ $G=(V,E)$ is a bipartite
graph with $V:=V_{1}\cup V_{2}=\\{x_{1},...,x_{s},y_{1},...,y_{k}\\}$, where
each vertex $y_{i}$ corresponds to a curve $C_{i}$, each vertex $x_{j}$
corresponds to an intersection point $p_{j}\in{\rm Sing}(\mathcal{C})$ and
$x_{j}$ is joined with $y_{i}$ by an edge in $E$ if and only if $p_{j}$ is
incident with $C_{i}$.

The above definition is a rather weak notion if we study arbitrary
arrangements of smooth plane curves. Consider the following example.

######  Example 3.3.

Let $\mathcal{L}$ be an arrangement of two lines intersecting along a single
point, and let $\mathcal{C}$ be an arrangement of two smooth conics
intersecting along one point being $A_{7}$ singularity. Then we can easily
construct the associated Levi graphs, and in both cases these graphs have the
same shape, namely

$G=(\\{x_{1},y_{1},y_{2}\\},\\{\\{x_{1},y_{1}\\},\\{x_{1},y_{2}\\}\\}).$

Obviously the Levi graph does not deliver any information about the types of
singularities of a given curve $\mathcal{C}$.

######  Example 3.4.

Let $\mathcal{C}=\\{C_{1},C_{2}\\}$ be an arrangement of two smooth conics
such that the singular locus of $\mathcal{C}$ consists of three points, two
simple nodes $p_{1},p_{2}$, and one tacnode $p_{3}$. Let $x_{i}$ be the vertex
corresponding to point $p_{i}$ with $i\in\\{1,2,3\\}$. Then the associated
Levi graphs has the following form:

$G=(\\{x_{1},x_{2},x_{3},y_{1},y_{2}\\},\\{\\{x_{1},y_{1}\\},\\{x_{1},y_{2}\\},\\{x_{2},y_{1}\\},\\{x_{2},y_{2}\\},\\{x_{3},y_{1}\\},\\{x_{3},y_{2}\\}\\}).$

Obviously, just by looking at $G$ and without any additional data, we cannot
distinguish nodes from tacnodes.

Now let us pass to the following fundamental definition for this survey.

######  Definition 3.5.

Let $C=\\{C_{1},...,C_{k}\\}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be a reduced
curve such that each irreducible component $C_{i}$ is smooth. The weak-
combinatorics of $C$ is a vector of the form
$(d_{1},...,d_{s};t_{1},...,t_{p})$, where $d_{i}$ denotes the number of
irreducible components of $C$ of degree $i$, and $t_{j}$ denotes the number of
singular points of a curve $C$ of a given analytic type $T_{j}$.

######  Example 3.6.

Let $\mathcal{C}=\\{C_{1},C_{2}\\}$ be an arrangement of two smooth conics
such that the singular locus of $\mathcal{C}$ consists of three points, namely
two simple nodes and one tacnode. Denote by $n_{2}$ the number of nodes and by
$t_{3}$ the number of tacnodes. Then the weak-combinatorics of $\mathcal{C}$
has the following form

$(d_{2};n_{2},t_{3})=(2;2,1).$

Why did we introduce these various notions of combinatorics that can be
attached to a given reduced curve $C$? It is all about Terao’s freeness
conjecture and NTC. We start with the classical Terao’s conjecture devoted to
line arrangements [18].

######  Conjecture 3.7 (Terao).

Let $\mathcal{A},\mathcal{B}\subset\mathbb{P}^{2}_{\mathbb{C}}$ be two line
arrangements such that their intersection lattices
$L(\mathcal{A}),L(\mathcal{B})$ are isomorphic. Assume that $\mathcal{A}$ is
free, then $\mathcal{B}$ must be free.

Notice that Terao’s freeness conjecture is widely open – we know that it holds
with up to $14$ lines [2], which might be slightly disappointing. However,
classical Terao’s conjecture is very demanding and in order to understand this
problem well one needs to understand the geometry of moduli spaces of line
arrangements that are free. It is very unclear whether the mentioned
conjecture holds in general, and based on that problem Dimca and Sticlaru in
[10] defined the class of nearly free curves and shortly afterwards it turned
out that nearly free curves might be crucial for Terao’s conjecture. It is
believed that if there exists a counterexample to Terao’s freeness conjecture,
then we should be able to find two line arrangements with isomorphic
intersection lattices, where one arrangement is free and the second
arrangement is nearly free. Seeing such fundamental problems, we can wonder
whether there is a natural way to extend the setting designed for Terao’s
freeness conjecture to higher degree curves, for instance to the case of
conic-line arrangements. We have the following example that comes from [17].

######  Example 3.8 (Counterexample to a naive Terao’s conjecture).

Consider the following conic-line arrangement

$\mathcal{CL}_{1}\,:\,xy\cdot(y^{2}+xz)\cdot(y^{2}+x^{2}+2xz)=0.$

Observe that the intersection point $P=(0:0:1)$ has multiplicity $4$.
Moreover, it is quasi-homogeneous, but not ordinary. Using `SINGULAR`, we can
check that $P$ is a singular point of type $Z_{1,0}$ with
$\tau_{P}=\mu_{P}=15$. One can show that $\mathcal{CL}_{1}$ is free with
exponents $(2,3)$. If we perturb a bit line $y=0$, taking for instance
$x-13y=0$, we obtain a new conic-line arrangement

$\mathcal{CL}_{2}\,:\,x\cdot(x-13y)\cdot(y^{2}+xz)\cdot(y^{2}+x^{2}+2xz)=0.$

In this new arrangement, the intersection point $P=(0:0:1)$ has multiplicity
$4$, but it is not longer quasi-homogeneous, and $\mathcal{CL}_{2}$ is not
free. In fact, the arrangement $\mathcal{CL}_{2}$ is nearly free exponents
$(d_{1},d_{2},d_{3})=(3,3,3)$. Obviously the Levi graphs of
$\mathcal{CL}_{1}$, $\mathcal{CL}_{2}$ are isomorphic.

The example presented above is crucial. Firstly, we see that (strong)
combinatorics for curve arrangements, i.e., where the irreducible components
are not just lines, does not determine the freeness, so that the classical
Terao’s conjecture cannot be naively extended. On the other hand, the above
example shows that the notion of weak-combinatorics might allow us to
distinguish arrangements in the context of the freeness property. From this
perspective, it is natural to ask whether the following conjecture can hold.

######  Conjecture 3.9 (Numerical Terao’s Conjecture).

Let $C_{1},C_{2}$ be two reduced curves in $\mathbb{P}^{2}_{\mathbb{C}}$ with
the property that they possess only smooth irreducible components and they
admit only quasi-homogeneous singularities. Suppose that $C_{1}$ and $C_{2}$
have the same weak-combinatorics and let $C_{1}$ be free, then $C_{2}$ is also
free.

The assumption that we consider only quasi-homogeneous singularities is
technical and not obvious at the first glance. If we assume that we our curves
admit only smooth irreducible components and quasi-homogeneous singularities,
then the total Tjurina number is determined by the weak-combinatorics. For
instance, it is well-known that ordinary intersection points for curve
arrangements of multiplicity $m>4$ are, in general, not quasi-homogeneous, see
[17, Example 4.2] or [5, Exercise 7.31]. The assumption that all singularities
are quasi-homogeneous allows us, for instance, to compute the local Tjurina
number of such ordinary singular points using only Milnor’s formula involving
the number of branches and the intersection indices.

In the forthcoming two sections we collect all recent developments regarding
NTC. First of all, we start our discussion in the setting of line
arrangements, and then we jump it the scenario of conic-line arrangements in
the complex plane.

###  3.1 NTC versus triangular line arrangements

This section is based on [14]. Here we want to show that NTC is false in the
class of line arrangements in the complex plane. In order to construct the
mentioned counterexample, we need to introduce a special class of line
arrangements, the so-called triangular arrangements in the complex projective
plane.

######  Definition 3.10.

An arrangement $\mathcal{L}$ of $abc$ lines is called triangular, if
$\mathcal{L}$ is given by an equation of type

$Q(x,y,z)=xyz\prod_{i=1}^{a-1}(x-\alpha_{i}y)\cdot\prod_{j=1}^{b-1}(y-\beta_{j}z)\cdot\prod_{i=k}^{c-1}(x-\gamma_{k}z)=0,$

where $\alpha_{i},\beta_{j},\gamma_{k}\in\mathbb{C}$.

The above definition is motivated by the well-known example of a supersolvable
line arrangement, called sometimes the full $n$th CEVA arrangement, or the
full monomial arrangement, and we denote it as $\mathcal{FMA}_{n}$. This
arrangement is given by the following equation depending on $n\geqslant 2$,
namely

$F_{n}(x,y,z)=xyz(x^{n}-y^{n})(y^{n}-z^{n})(x^{n}-z^{n})=0.$

Here we want to present the following result that is proved in [14, Theorem
6.2].

######  Theorem 3.11.

There exists a pair of line arrangements in the complex projective plane, each
of which has weak-combinatorics of the form

$(d_{1};t_{2},t_{3},t_{4},t_{5},t_{6})=(15;24,12,0,0,3),$

such that one is free and the second is nearly-free.

###### Proof.

The first line arrangement $\mathcal{L}_{1}$ is constructed by removing from
the full monomial arrangement $\mathcal{FMA}_{6}$ defined by

$F_{6}(x,y,z)=xyz(x^{6}-y^{6})(y^{6}-z^{6})(x^{6}-z^{6})=0$

the following lines

$\ell_{1}:x-z=0,\quad\ell_{2}:x-ez=0,\quad\ell_{3}:y-z=0,$
$\ell_{4}:y-ez=0,\quad\ell_{1}:x-e^{2}y=0,\quad\ell_{6}:x-e^{4}y=0,$

where $e^{2}-e+1=0$. We can check, using `SINGULAR`, that $\mathcal{L}_{1}$ is
free with exponents $(d_{1},d_{2})=(7,7)$.

The second line arrangement $\mathcal{L}_{2}$ is constructed as follows. Take
the full monomial arrangement $\mathcal{FMA}_{5}$ given by

$F_{5}(x,y,z)=xyz(x^{5}-y^{5})(y^{5}-z^{5})(x^{5}-z^{5})=0$

and remove the following three lines

$\ell_{1}:x-z=0,\quad\ell_{2}:x-y=0,\quad\ell_{2}:y-z=0.$

We can check that $\mathcal{L}_{2}$ is not free, but only nearly free with
exponents $(d_{1},d_{2},d_{3})=(6,9,9)$. ∎

From the above result we can extract the following crucial observations.

######  Corollary 3.12.

The Numerical Terao’s conjecture does not hold in the class of triangular line
arrangements in the complex projective plane.

######  Corollary 3.13.

In the class of line arrangements in the complex projective plane, the minimal
degree of non-trivial Jacobian relations is not determined by the weak-
combinatorics!

###  3.2 NTC holds for conic-line arrangements with nodes, tacnodes, and
ordinary triple points

In this section we report results devoted to the freeness of conic-line
arrangements with simple singularities. More precisely,
$\mathcal{CL}=\\{\ell_{1},...,\ell_{d},C_{1},...,C_{k}\\}\subset\mathbb{P}^{2}_{\mathbb{C}}$
be an arrangement of $d\geqslant 0$ lines and $k\geqslant 0$ smooth conics
admitting only the following quasi-homogeneous singularities (here we use
normal local forms presented in [1]):

$A_{k}$ with $k\geqslant 1$ |  $:\,x^{2}+y^{k+1}=0$,
---|---
$D_{k}$ with $k\geqslant 4$ |  $:\,y^{2}x+x^{k-1}=0$,
$X_{9}$ with $a^{2}\neq 4$ |  $:\,x^{4}+y^{4}+ax^{2}y^{2}=0$.

In particular, $A_{1}$ singularities are just nodes, $D_{4}$ singularities are
ordinary triple points, and $X_{9}$ singularities are ordinary quadruple
points. Obviously all singularities presented above are quasi-homogeneous. Our
first result is devoted to the case when we have only smooth conics, see [15,
Theorem A].

######  Theorem 3.14.

There does not exist any arrangement of $k\geqslant 2$ smooth conics with
$A_{1},A_{3},D_{4},X_{9}$ singularities that is free.

This result is optimal in the sense that if we add to the list of admissible
intersections singularities of type $A_{5}$, then we can find an arrangement
that is free, see [16, Remark 2.5]. On the other hand, this result stands in
odds with the picture of line arrangements where we can find many free
arrangements admitting nodes, triple and quadruple points.

Now we can focus on the case of conic-line arrangements
$\mathcal{CL}\subset\mathbb{P}^{2}_{\mathbb{C}}$ with $d\geqslant 1$ and
$k\geqslant 1$. We assume that our arrangements admit only singularities of
type $A_{1},A_{3},D_{4}$ – this is the first non-trivial situation that
distinguishes the geometry of conic arrangements and line arrangements. Our
main result, obtained in a joint paper with A. Dimca [9], delivers a complete
classification of such arrangements.

######  Theorem 3.15.

Let $\mathcal{CL}$ be an arrangement of $d\geqslant 1$ lines and $k\geqslant
1$ smooth conics having only nodes, tacnodes, and ordinary triple points as
singularities. Then $\mathcal{CL}$ is free if and only if one of the following
cases occur:

  1. (1)

$d=k=1$ and $\mathcal{CL}$ consists of a smooth conic and a tangent line,
which means we can only one tacnode.

  2. (2)

$d=2$, $k=1$ and $\mathcal{CL}$ consists of a smooth conic and two tangent
lines. In this case one node and two tacnodes.

  3. (3)

$d=3$, $k=1$ and either $\mathcal{CL}$ is a smooth conic inscribed in a
triangle, or $\mathcal{CL}$ is a smooth conic circumscribed in a triangle. In
the first case we have three nodes and three tacnodes, and in the second case
we have only three ordinary triple points as intersections.

  4. (4)

$d=3$, $k=2$ and $\mathcal{CL}$ consists of a triangle $\Delta$, a smooth
conic inscribed in $\Delta$, and another smooth conic circumscribed in
$\Delta$. In this case we have $5$ tacnodes and and three ordinary triple
points as singularities.

In particular, a free conic-line arrangement having only nodes, tacnodes, and
ordinary triple points is determined up to a projective equivalence by the
weak-combinatorics.

This theorem implies the following classification result.

######  Corollary 3.16.

Numerical Terao’s Conjecture holds for conic-line arrangements with nodes,
tacnodes, and ordinary triple points.

The following remark explains the situation when $k=0$, so we arrive at the
scenario of line arrangements with double and triple points as intersections.

######  Remark 3.17.

Numerical Terao’s Conjecture holds for line arrangements having only double
and triple points as singularities and it follows from the following
observations. First of all, a free line arrangement $\mathcal{A}:f=0$ has to
satisfy the condition that $d=\deg f\leqslant 9$, see [13] for an elementary
argument. Next, observe that the freeness condition implies that

$d_{1}+d_{2}=d-1\quad\text{ with }d_{1}\leqslant d_{2},$

and we have $(d-1)/2\geqslant d_{1}={\rm mdr}(f)$. This gives us
$d_{1}\leqslant 3$ or $d_{1}={\rm mdr}(f)=4$ and $d=9$. Note that if
$\mathcal{A}^{\prime}:f^{\prime}=0$ has the same weak-combinatorics as
$\mathcal{A}:f=0$, then $\tau(\mathcal{A})=\tau(\mathcal{A}^{\prime})$ since
the total Tjurina number of a reduced curve with quasi-homogeneous
singularities is determined by the weak-combinatorics. It implies, in
particular, that $r^{\prime}={\rm mdr}(f^{\prime})\leqslant{\rm mdr}(f)$, and
this follows from the maximality of the Tjurina number for free reduced
curves, see [12]. In the first case, using the complete classification of line
arrangements with ${\rm mdr}(f)\leqslant 3$ provided in [3], we can conclude
the statement. In the second case, we use again the maximality of the Tjurina
number of free curves [12] and we conclude that
$\tau(\mathcal{A})=\tau(\mathcal{A}^{\prime})=48$, and this maximal value is
obtained when the number of nodes is $0$ and the number of triple points is
equal to $12$. The only line arrangement with these invariants is the dual
Hesse arrangement of lines which is free with exponents $(d_{1},d_{2})=(4,4)$
and unique up to the projective equivalence.

##  4 Weak Ziegler pairs

This short section is devoted to an intriguing notion of Ziegler pairs. Let us
recall that topic is strictly motivated by the following question.

######  Question 4.1.

Is it true that for a given complex line arrangements the minimal free
resolution of the associated Milnor algebra is determined by its intersection
lattice?

At the very first glance we have some doubts since there is no reasonable
argument explaining that the shape of the resolution of the Milnor algebras
associated with line arrangements can be determined the combinatorial
structure. From our perspective, and from the perspective of NTC, we focus our
attention on the minimal degree of non-trivial Jacobian relations as a crucial
invariant for the whole resolution. This idea leads us to the following
object.

######  Definition 4.2 (Ziegler pair).

We say that two line arrangements
$\mathcal{L}_{1},\mathcal{L}_{2}\subset\mathbb{P}^{2}_{\mathbb{C}}$ form a
Ziegler pair if the intersection lattices of $\mathcal{L}_{1}$ and
$\mathcal{L}_{2}$ are isomorphic, but ${\rm mdr}(\mathcal{L}_{1})\neq{\rm
mdr}(\mathcal{L}_{2})$.

As we can see, Ziegler pairs have a strong connection with the possible
counterexamples to Terao’s freeness conjecture. Recall that for line
arrangements the total Tjurina number is determined by the intersection
lattice (since all singularities are quasi-homogeneous), so if we could find a
counterexample to Terao’s conjecture, it would be all about the minimal
degrees of non-trivial Jacobian relations being different. The first pair of
line arrangements with the same intersection posets but different minimal
degrees of non-trivial Jacobian relations was found by Ziegler [19], and we
will say a few words about this construction. This pair consists of two
arrangements having exactly $9$ lines with $6$ triple and $18$ double points
as intersections, but their geometries are different. In the first case, all
six triple points are on a conic, but in the second case, only $5$ triple
points are on a conic, and one point is off the conic. Geometrically speaking,
the condition that $6$ points are on a conic is unexpected, and this is a
crucial feature for this example. This construction has recently been
revisited and studied in detail by Dimca and Sticlaru [11].

However, the notion of Ziegler pairs is not suitable for curve arrangements,
in its whole generality, since we can detect some pathological situations.

######  Example 4.3.

Consider an arrangement of lines $\mathcal{L}$ given by

$Q(x,y,z)=xy.$

Clearly ${\rm mdr}(Q)=0$. Now we consider the arrangement of conics
$\mathcal{C}$ given by

$G(x,y,z)=(x^{2}-yz)(x^{2}+z^{2}-yz).$

Obviously these two different curve arrangements have the same Levi graphs and
both are free, but

$0={\rm mdr}(Q)\neq{\rm mdr}(G)=1.$

In order to avoid such situations, we define the following.

######  Definition 4.4 (Weak Ziegler pair).

Consider two reduced plane curves
$C_{1},C_{2}\subset\mathbb{P}^{2}_{\mathbb{C}}$ such that all irreducible
components of $C_{1}$ and $C_{2}$ are smooth. We say that a pair
$(C_{1},C_{2})$ forms a weak Ziegler pair if $C_{1}$ and $C_{2}$ have the same
weak-combinatorics, but they have different minimal degrees of non-trivial
Jacobian relations, i.e., ${\rm mdr}(C_{1})\neq{\rm mdr}(C_{2})$.

######  Proposition 4.5.

The line arrangements constructed in Theorem 3.11 form a weak Ziegler pair.

###### Proof.

The arrangements $\mathcal{L}_{1}$ and $\mathcal{L}_{2}$ have the same weak-
combinatorics, but

$7={\rm mdr}(\mathcal{L}_{1})\neq{\rm mdr}(\mathcal{L}_{2})=6,$

so they form a weak Ziegler pair. ∎

It is natural to wonder whether we can construct new weak Ziegler pairs in
different classes of curve arrangements, for instance in the class of conic-
line arrangements. Let us recall the following example by Schenck and
Tohăneanu from [17] providing the first weak Ziegler pair for conic-line
arrangements with ordinary but not quasi-homogeneous singularities.

######  Example 4.6.

Let us consider the arrangement $\mathcal{C}_{1}$ being the union of the
following five smooth conics:

$\begin{array}[]{*{3}c}C_{1}&:&(x-3z)^{2}+(y-4z)^{2}-25z^{2}=0\\\
C_{2}&:&(x-4z)^{2}+(y-3z)^{2}-25z^{2}=0\\\
C_{3}&:&(x+3z)^{2}+(y-4z)^{2}-25z^{2}=0\\\
C_{4}&:&(x+4z)^{2}+(y-3z)^{2}-25z^{2}=0\\\
C_{5}&:&(x-5z)^{2}+y^{2}-25z^{2}=0.\end{array}$

The arrangement $\mathcal{C}_{1}$ has $13$ ordinary singular points, $10$ of
these points are nodes, while at the points $(0:0:1),(1:i:0),(1:-i:0)$ all the
five conics meet. We can check that the quintuple point $q=(0:0:1)$ is not
quasi-homogeneous since $15=\tau_{q}\neq\mu_{q}=16$. If we add the following
lines

$\ell_{1}:z=0,\quad\ell_{2}:x-iy=0,\quad\ell_{3}:x+iy=0,$

then we obtain arrangement
$\mathcal{CL}_{1}=\\{C_{1},...,C_{5},\ell_{1},\ell_{2},\ell_{3}\\}$ which is
free with exponents $(d_{1},d_{2})=(6,6)$.

Next, let us consider the arrangement $\mathcal{C}_{2}$ being the union of the
following five smooth conics:

$\begin{array}[]{ccl}C_{1}^{\prime}&:&x^{2}+8y^{2}+21xy-xz-8yz=0\\\
C_{2}^{\prime}&:&x^{2}+5y^{2}+13xy-xz-5yz=0\\\
C_{3}^{\prime}&:&x^{2}+9y^{2}-4xy-xz-9yz=0\\\
C_{4}^{\prime}&:&x^{2}+11y^{2}+xy-xz-11yz=0\\\
C_{5}^{\prime}&:&x^{2}+17y^{2}-5xy-xz-17yz=0.\end{array}$

Observe that $\mathcal{C}_{2}$ is combinatorially and weak-combinatorially
identical to $\mathcal{C}_{1}$, but the quintuple points
$(0:0:1),(1:0:1),(0:1:1)$, where all the branches meet, are not quasi-
homogeneous since $15=\tau\neq\mu=16$. If we add the lines

$\ell_{1}^{\prime}:x=0,\quad\ell_{2}^{\prime}:y=0,\quad\ell_{3}^{\prime}:x+y-z=0,$

then we obtain arrangement
$\mathcal{CL}_{2}=\\{C_{1}^{\prime},...,C_{5}^{\prime},\ell_{1}^{\prime},\ell_{2}^{\prime},\ell_{3}^{\prime}\\}$
having the same strong combinatorics as $\mathcal{CL}_{1}$, and since all
singularities are ordinary, then the arrangements have the same weak-
combinatorics, but $\mathcal{CL}_{2}$ is not free. In fact, $\mathcal{CL}_{2}$
is just $4$-syzygy with exponents $(d_{1},d_{2},d_{3},d_{4})=(7,7,7,7)$.

######  Remark 4.7.

In the above example we can literally see the phenomenon that if singularities
of our curve are not quasi-homogeneous, then the total Tjurina number is not
determined by the weak-combinatorics.

We finish our survey with the following problem.

######  Problem 4.8.

Construct examples of weak Ziegler pairs in the class of curve arrangements
(so arrangements admitting not only lines as irreducible components) admitting
only quasi-homogeneous singularities.

## Acknowledgement

I want to thank Alex Dimca for all discussions regarding NTC and Ziegler
pairs.

Piotr Pokora is supported by the National Science Centre (Poland) Sonata Bis
Grant 2023/50/E/ST1/00025. For the purpose of Open Access, the author has
applied a CC-BY public copyright licence to any Author Accepted Manuscript
(AAM) version arising from this submission.

## References

  * [1] V. I. Arnold, Local normal forms of functions. Invent. Math. 35: 87 – 109 (1976).
  * [2] M. Barakat, L. Kühne, Computing the nonfree locus of the moduli space of arrangements and Terao’s freeness conjecture. Math. Comp. 92: 1431 – 1452 (2023).
  * [3] R. Burity, S. Tohăneanu, Logarithmic derivations associated to line arrangements. J. Algebra 581: 327 – 352 (2021).
  * [4] W. Decker, G.-M. Greuel, G. Pfister, and H. Schönemann,  Singular 4-1-1 — A computer algebra system for polynomial computations.  http://www.singular.uni-kl.de, 2018.
  * [5] A. Dimca, Topics on real and complex singularities. An introduction. Advanced Lectures in Mathematics. Braunschweig/Wiesbaden: Friedr. Vieweg & Sohn. 1987.
  * [6] A. Dimca, Hyperplane arrangements. An introduction. Universitext. Cham: Springer (ISBN 978-3-319-56220-9/pbk; 978-3-319-56221-6/ebook). xii, 200 p. (2017).
  * [7] A. Dimca, Freeness versus Maximal Global Tjurina Number for Plane Curves. Math. Proc. Camb. Philos. Soc. 163(1): 161 – 172 (2017).
  * [8] A. Dimca, On free curves and related open problems. arXiv:2312.07591.
  * [9] A. Dimca and P. Pokora, On conic-line arrangements with nodes, tacnodes, and ordinary triple points. J. Algebraic Combin. 56(2): 403 – 424 (2022).
  * [10] A. Dimca, G. Sticlaru, Free and Nearly Free Curves vs. Rational Cuspidal Plane Curves. Publ. Res. Inst. Math. Sci. 54(1): 163 – 179 (2018).
  * [11] A. Dimca, G. Sticlaru, From Pascal’s Theorem to the geometry of Ziegler’s line arrangements. arXiv:2312.11928.
  * [12] A. Du Plessis and C. T. C. Wall, Application of the theory of the discriminant to highly singular plane curves. Math. Proc. Camb. Philos. Soc. 126(2): 259 – 266 (1999).
  * [13] J. Kabat, On the freeness of Böröczky line arrangements. Period. Math. Hung. 78(1): 31 – 37 (2019).
  * [14] S. Marchesi and J. Vallés, Triangular arrangements on the projective plane. Épijournal de Géom. Algébr., EPIGA 7: Article 14, 20 p. (2023).
  * [15] P. Pokora, $\mathcal{Q}$-conic arrangements in the complex projective plane. Proc. Amer. Math. Soc. 151(7): 2873 – 2880 (2023).
  * [16] P. Pokora, On free and nearly free arrangements of conics admitting certain ${\rm ADE}$ singularities. Ann Univ Ferrara (2023). https://doi.org/10.1007/s11565-023-00481-6.
  * [17] H. Schenck, S. Tohăneanu, Freeness of conic-line arrangements in $\mathbb{P}^{2}$. Comment. Math. Helv. 84: 235 – 258 (2009).
  * [18] H. Terao, Generalized Exponents of a Free Arrangement of Hyperplanes and Shepherd-Todd-Brieskorn Formula. Invent. Math. 63: 159 – 179 (1981).
  * [19] G. Ziegler, Combinatorial construction of logarithmic differential forms. Adv. Math. 76: 116 – 154 (1989).

Piotr Pokora, Department of Mathematics, University of the National Education
Commission Krakow, Podchora̧żych 2, PL-30-084 Kraków, Poland.
E-mail address<EMAIL_ADDRESS>


# Evaluation of Reinforcement Learning Techniques for Trading on a Diverse
Portfolio

Ishan Khare
Stanford University
<EMAIL_ADDRESS>
Tarun Martheswaran
Stanford University
<EMAIL_ADDRESS>
Jonah Ezekiel
Stanford University
<EMAIL_ADDRESS>
Akshana Dassanaike-Perera
Stanford University
<EMAIL_ADDRESS>

###### Abstract

This work seeks to answer key research questions regarding the viability of
reinforcement learning over the S&P 500 index. The on-policy techniques of
Value Iteration (VI) and State–action–reward–state–action (SARSA) are
implemented along with the off-policy technique of Q-Learning. The models are
trained and tested on a dataset comprising multiple years of stock market data
from 2000-2023. The analysis presents the results and findings from training
and testing the models using two different time periods: one including the
COVID-19 pandemic years and one excluding them. The results indicate that
including market data from the COVID-19 period in the training dataset leads
to superior performance compared to the baseline strategies. During testing,
the on-policy approaches (VI and SARSA) outperform Q-learning, highlighting
the influence of bias-variance tradeoff and the generalization capabilities of
simpler policies. However, it is noted that the performance of Q-learning may
vary depending on the stability of future market conditions. Future work is
suggested, including experiments with updated Q-learning policies during
testing and trading diverse individual stocks. Additionally, the exploration
of alternative economic indicators for training the models is proposed.

†All authors contributed equally to this work.

## Introduction

This project aims to develop an optimal policy of trading signals for
algorithmic trading by comparing on- and off-policy reinforcement learning
techniques. Specifically, we are implementing Value Iteration (VI),
State–action–reward–state–action (SARSA), and Q-learning algorithms to
determine when to buy/sell our shares of a set of stocks. Using VI and SARSA,
we aim to learn the most optimal on-policy trading strategy given a policy
designed by hand. Using Q-Learning, we aim to learn the most optimal policy
for algorithmic trading. We also have trained our models on our set portfolio
over multiple years of data, and subsequently tested it in an attempt to
determine a policy ROI estimate. After obtaining this evaluation metric, we
can compare with the S&P 500, our selected baseline oracle. Overall, we aim to
answer the following questions this updated report:

  * •

Which reinforcement learning technique, on or off policy, is able to generate
a higher ROI? Why may this be?

  * •

Should we be using reinforcement learning rather than holding S&P 500? In
which situations is this true/false?

  * •

What further hyperparameters changes should we make in order to further
increase this ROI?

## Literature Review

In the realm of on-policy approaches, such as Value Iteration, Sumit Rawat
applied a on-policy based technique using the Bellman optimality equation. As
he knew that evaluation at every step may not be necessary, he elected to stop
after just one sweep (Rawat, 2005). Additionally, Lee utilized Generalized
Policy Iteration (GPI) to find an evaluation-based optimal policy (Lee, 2001).

Within off-policy approaches, which have been more popular recently, we see
several studies starting to implement Q-learning. Sun and colleagues found
that a combination of Mean Absolute Deviation (MAD) and Q-learning performs
better than other common reinforcement learning techniques. Specifically, he
used epsilon-greedy after pre-processing with MAD. (Sun et al., 2023). Also,
Chakole and colleagues used Q-learning to demonstrate its outperformance of
the Buy/Hold strategy and Decision-Tree strategies (Chakole et al., 2021).

However, there is research of techniques that combine both on and off-policy
approaches. Alimoradi adapts a league championship algorithm (LCA), hybridized
with both Sarsa and Q-learning (Alimoradi and Kashan, 2018). Using a
combination of these techniques, the researchers found that they were able to
outperform evolutionary programming and Buy & Hold strategies when trading
almost all companies.

Since our last literature review, we have officially decided that our project
will be focused on trading ETFs, specifically SPY. So, we start to search for
articles that traded SPY with reinforcement learning. Yang and his coauthors
explored a reward-based inverse reinforcement learning technique to extract
trading signals. Similarly to us, they used the buy and hold SPY ETF as a
benchmark. They were able to outperform this based on signals from new
sentiment (Yang et al., 2018). Based on research like Yang’s, we would like to
see how our tests of both off and on policy reinforcement learning techniques
compare.

## Dataset

For the purpose of our project, we decided to connect to the Yahoo Finance API
through Python, in order to get a time series of data for a certain ticker.
Luckily, through the "yfinance" library the data is collected in an easy-to-
use fashion, so pre-processing was not a significant consideration. The
information that we chose to use for each day was simply Open & Closing Price,
as well as High and Lows.

After obtaining dataframes of information on the price of SPY from 1980 to
2023, we moved forward to apply our techniques of Value Iteration, SARSA, and
Q-learning to develop actionable policies of trading signals.

## Baseline

Many are familiar with the S&P 500 index. This is a popular index that
represents the general trend of the stock market at any given point. In past
research works, people have tried to develop algorithmic techniques that
output signals to game the S&P 500. While the common economic theory is that a
stochastic set of choice is likely to do better, we hope to see if a carefully
designed artificial intelligence algorithm can refute this claim. Can we in
fact define and evaluate a policy that creates actionable insights on the
market? Is it better than the most simplified method to track the market?

As an update to our previous project proposal, we have done extensive research
to conclude that we will also use the "Buy & Hold Strategy" as a baseline.
This strategy is based off the saying that "time in the market beats timing
the market", which comes from the belief that the market will generally
increase over longer periods of time, so holding a stock is advantageous.

Additionally, the models described in the Literature Review provide another
set of baselines. These, alongside several other models that researchers have
developed have similar evaluation metrics to the ones we have elected to use.
Our project aims to learn an optimal policy for only trading SPY, so
comparison against the results of multi-stock models should be interpreted in
consideration of that context.

## Main Approach

As mentioned in previous sections, we plan to implement and test these three
common reinforcement learning approaches on trading company stock. We aim to
use VI, SARSA, and Q-Learning to develop an optimal trading policy for trading
SPY, an ETF which tracks the S&P 500. We are only trading one ETF to provide
clear analysis across the three methods that we are testing by eliminating
confounding variables that arise when trading multiple stocks.

As VI and SARSA is on-policy, we need to have a policy before entering the
market. Following the motto "buy low, sell high", we follow the
straightforward policy:

  * •

If the price we purchased is lower than the current price, choose a random
number of shares to sell

  * •

If the price we purchased is higher than the current price, choose a random
number of shares to buy

Now, for our off-policy approach we chose to implement Q-learning to our
financial data. Here, we do not need a specific policy to test, instead the
algorithm learns as it goes through the data, finding the $Q_{opt}$ value we
discussed in lecture.

In terms of data, we will do two sets of training/testing as follows

  * •

2000-2021 for the "SPY" ticker as our training set, and subsequently test on
2021-2023.

  * •

2000-2016 for the "SPY" ticker as our training set, and subsequently test on
2016-2023.

This is to see if including the COVID period in our training data has an
impact on our model. We hypothesize that it will, as the events that occurred
were unpredictable and the on/off policy approaches may not work as expected.

## Evaluation Metric

To evaluate the performance of our trading strategies in both the on/off
policy approaches, we elected to use traditional profit and loss (PNL) as our
evaluation metric. We focus on the total profit and loss over the timespan
that our model runs, with the amount of starting capital constant. We also
plan to track PnL over each day and subperiods, to see if we should focus on
specific time spans.

## Results and Analysis

Figure 1: This figure displays our results when training our models on the
years 2000 - 2015 and testing from 2016-present day. Note: we include COVID
years in our testing data and exclude COVID years in our training data. Figure
2: This figure displays our results when training our models on the years 2000
- 2022 and testing from 2022-present day. Note: we exclude COVID years in our
testing data.

As demonstrated across the two graphs, including market data from COVID in the
training dataset leads to superior results compared to our baseline. This is
likely due to the fact that the unprecedented conditions of COVID led to
unpredictable market conditions and behavior. During the beginning COVID,
there was a rapid market crash, followed by an even greater tech-stock-led
boom that was not seen in earlier training data. As a result, as demonstrated
by Figure 1, holding the S&P 500 beat out all of our reinforcement learning
strategies when COVID was excluded from our training data (and included in our
testing data). However, when we included COVID in our training data (and
excluded it from testing), all models beat the baseline. This may be
attributed to the increased robustness that the data from COVID provided and
also the stagnation of the S&P 500 during that time.

When comparing all three reinforcement learning strategies, we can see
multiple takeaways. First, although based on simple principles, the “buy low,
sell high” policy behind VI and SARSA proves to be fairly robust. In both
experimental runs, regardless of whether COVID was included or not, Q-Learning
leads to the greatest portfolio value during testing. Because Q-Learning is an
off-policy algorithm, it is able to learn a more optimal policy than the
provided policy for VI and SARSA that was hand-designed. However, during our
testing phase, Q-Learning performs the worst out of the three models.

We attribute this to the nature of the task at hand. During testing, we do not
update $Q_{\pi}$, which means that Q-Learning does not "learn" as time
progresses. As such, Q-Learning is unable to adapt the specific policy it
learned under previous market conditions to the new conditions it experiences
during testing. On the other hand, because VI and SARSA are on-policy and
utilize a simple, fixed model, they are able to perform better than Q-Learning
as they generalize better to unseen training data. This dynamic between
learning algorithms is likely due to the bias-variance tradeoff, where the
simplistic policies of VI and SARSA generalize far better when the models are
unable to "learn".

That being said, its important to recognize that both experimental runs test
on modern market conditions, which are quite unique to that of past
conditions. Although most COVID restrictions have been eliminated, the economy
has not fully recovered since COVID. Many communities are still feeling the
impact. Even for large companies, COVID-era trends still have not settled to a
post-pandemic world, such as the dynamic between online streaming and in-
person movies, online shopping and in-person shopping, etc.

As such, our results may not be fully generalize to future market conditions
that are more stable, but, for conditions that are similar to that of our
training data, we hypothesize that our Q-Learning will perform quite
competitively.

Revisiting the three questions we originally sought out to answer:

  * •

Which reinforcement learning technique, on or off policy, is able to generate
a higher ROI? Why may this be?

During training time, Q-Learning performs far better than VI and SARSA.
However, during testing, SARSA performs the best and VI outperforms
Q-Learning. We attribute the performance drop of Q-Learning to the bias-
variance trade off. We hypothesize that SARSA performs better than VI because
it also optimizes on next-action, which essentially adds another layer of
depth in its optimization.

  * •

Should we be using reinforcement learning rather than holding S&P 500? In
which situations is this true/false?

Based on our results, we conclude that holding the S&P500 is preferable during
times of large uncertainty or new market conditions, such as during COVID,
whereas reinforcement learning is preferable during market stagnation.

  * •

What further hyperparameters changes should we make in order to further
increase this ROI?

To further increase this ROI, we should focus on weighing near-term rewards,
as market predictability drastically decreases over larger time horizons.

## Future Work

Moving forward, there’s multiple experiments we would like to perform. First,
for Q-Learning, we would like to experiment with updating our Q-Learning
policy during testing as well and compare it to the S&P 500 baseline. Although
this wouldn’t following testing as defined in a supervised learning context,
it may make more sense in a reinforcement-learning context to have our model
constantly updating itself as it encounters new data.

Additionally, an interesting experiment would be to trade a diverse set of
individual stocks. As of now, we have only tested on the SPY, and have
received promising results. How may our policies be able to exploit the market
conditions of individual stocks to outperform a large index?

Finally, another possible avenue of research would be to train our models on
an alternative measure of price. Currently, our models are being trained on
the percentage change of the closing price. Instead, we could train our models
on other economic indicators such as the 40-day Simple Moving Average or the
Relative Strength Index.

## Financial Disclaimer

The results, analysis, and findings of this work are for general knowledge and
informational purposes only. Any investment or financial decisions should be
made based on one’s own research and judgment, or by consulting with a
qualified financial advisor. Investing in financial markets involves risks,
and one should carefully consider their own financial situation and tolerance
for risk before making any investment decisions. Please note that we the
authors cannot be held responsible for any actions one takes based on our
work.

## Code

Our GitHub is linked here: github.com/jonah-b-ezekiel/CS221.

## Non-Private Data

We downloaded market data from Yahoo! Finance’s API linked here which is open-
source and free to use: pypi.org/project/yfinance.

## References

  * Alimoradi and Kashan (2018) M. R. Alimoradi and A. H. Kashan.  A league championship algorithm equipped with network structure and backward q-learning for extracting stock trading rules.  _Applied soft computing_ , 68:478–493, 2018.
  * Chakole et al. (2021) J. B. Chakole, M. S. Kolhe, G. D. Mahapurush, A. Yadav, and M. P. Kurhekar.  A q-learning agent for automated trading in equity stock markets.  _Expert Systems with Applications_ , 163:113761, 2021.
  * Lee (2001) J. W. Lee.  Stock price prediction using reinforcement learning.  In _ISIE 2001. 2001 IEEE International Symposium on Industrial Electronics Proceedings (Cat. No. 01TH8570)_ , volume 1, pages 690–695. IEEE, 2001\.
  * Rawat (2005) S. Rawat.  _Stock market prediction using reinforcement learning_.  Utah State University, 2005.
  * Sun et al. (2023) Y. Sun, M. Gao, C. Yang, D. Yuan, P. Zhu, H. Dong, and N. Zhou.  Research on stock trading prediction based on MAD and Q-learning.  In Y. Zhong, editor, _Fifth International Conference on Computer Information Science and Artificial Intelligence (CISAI 2022)_ , volume 12566, page 1256649. International Society for Optics and Photonics, SPIE, 2023.  doi: 10.1117/12.2668175.  URL https://doi.org/10.1117/12.2668175.
  * Yang et al. (2018) S. Y. Yang, Y. Yu, and S. Almahdi.  An investor sentiment reward-based trading system using gaussian inverse reinforcement learning algorithm.  _Expert Systems with Applications_ , 114:388–401, 2018\.  ISSN 0957-4174.  doi: https://doi.org/10.1016/j.eswa.2018.07.056.  URL https://www.sciencedirect.com/science/article/pii/S0957417418304810.