# Ranks of elliptic curves and deep neural networks

Matija Kazalicki  Department of Mathematics
University of Zagreb
Bijenička cesta 30
10000 Zagreb
Croatia<EMAIL_ADDRESS>and  Domagoj Vlah  Department of Applied
Mathematics
Faculty of Electrical Engineering and Computing
University of Zagreb
Unska 3
10000 Zagreb
Croatia<EMAIL_ADDRESS>
##  1\. Abstract

Determining the rank of an elliptic curve $E/\mathbb{Q}$ is a hard problem,
and in some applications (e.g. when searching for curves of high rank) one has
to rely on heuristics aimed at estimating the analytic rank (which is equal to
the rank under the Birch and Swinnerton-Dyer conjecture).

In this paper, we develop rank classification heuristics modeled by deep
convolutional neural networks (CNN). Similarly to widely used Mestre-Nagao
sums, it takes as an input the conductor of $E$ and a sequence of normalized
$a_{p}$-s (where $a_{p}=p+1-\\#E(\mathbb{F}_{p})$ if $p$ is a prime of good
reduction) in some range ($p<10^{k}$ for $k=3,4,5$), and tries to predict rank
(or detect curves of “high” rank). The model has been trained and tested on
two datasets: the LMFDB and a custom dataset consisting of elliptic curves
with trivial torsion, conductor up to $10^{30}$, and rank up to $10$. For
comparison, eight simple neural network models of Mestre-Nagao sums have also
been developed.

Experiments showed that CNN performed better than Mestre-Nagao sums on the
LMFDB dataset (interestingly neural network that took as an input all Mestre-
Nagao sums performed much better than each sum individually), while they were
roughly equal on custom made dataset.

##  2\. Introduction

Let $E$ be an elliptic curve over $\mathbb{Q}$ with discriminant $\Delta$ and
conductor $N$. A celebrated Mordell’s Theorem states that the group of
rational points $E(\mathbb{Q})$ is a finitely generated abelian group
isomorphic to $E(\mathbb{Q})_{tors}\times\mathbb{Z}^{r}$, where
$E(\mathbb{Q})_{tors}$ is the torsion subgroup of $E(\mathbb{Q})$ and $r$ is
the rank of $E(\mathbb{Q})$. While the possible torsion subgroups were
completely classified by Mazur [Maz77] and easy to compute, the rank of the
elliptic curve is a much more mysterious quantity. Not only it is not known
what values can rank attain (although this question was already asked by
Poincaré [Poi01]), but also there is no agreement on whether the rank is
unbounded or not. Up until recently, the folklore conjecture was that the rank
is unbounded which was challenged by the series of papers [WDE+14, Wat15,
PPVW19] that predict (based on different heuristic models) that there are only
finitely many elliptic curves with rank greater than $21$. The current rank
record is $28$ by Elkies [Elk06]. For more information about the rank records
for curves with fixed torsion subgroup as well for rank in families see [Duj].

One of the reasons why it is hard to find curves of high rank is that
computationally it is costly to compute the rank of an elliptic curve, mainly
due to the fact that we don’t know how to efficiently find rational points on
elliptic curves. Descent algorithms commonly in use, eventually boil down to
the naive point search on some auxiliary curves. One way around this is to use
rank heuristics inspired by the Birch and Swinnerton-Dyer conjecture to filter
out probable candidates for high rank elliptic curves.

For each prime of good reduction $p\nmid\Delta$, we define
$a_{p}=p+1-\\#E(\mathbb{F}_{p})$. For $p|N$, we set $a_{p}=0,-1,$ or $1$ if,
respectively, $E$ has additive, split multiplicative or non-split
multiplicative reduction at $p$. The $L$-function attached to $E/\mathbb{Q}$
is then defined as an Euler product

$L_{E}(s)=\prod_{p|\Delta}\left(1-\frac{a_{p}}{p^{s}}\right)^{-1}\prod_{p\nmid\Delta}\left(1-\frac{a_{p}}{p^{s}}+\frac{p}{p^{2s}}\right)^{-1},$

which converges absolutely for $\Re(s)>3/2$ and extends to an entire function
by the Modularity theorem [Wil95, BCDT01]. Birch and Swinnerton-Dyer (BSD)
conjecture states that the order of vanishing of $L_{E}(s)$ at $s=1$ (the
quantity known as analytic rank) is equal to the rank of $E(\mathbb{Q})$.

Mestre [Mes82] and Nagao [Nag92], and later others [EK20, Bob13], motivated by
BSD conjecture, considered certain sums (see Section 3 for the list of sums
examined in this paper) which heuristically should be able to detect curves of
high analytic rank. We call all such sums Mestre-Nagao sums. For example, one
of these sums (see Section 2 in [EK20])

$\tilde{S_{5}}(B)=\sum_{\begin{subarray}{c}p<B,\\\ \textrm{ good
reduction}\end{subarray}}\log\left(\frac{p+1-a_{p}}{p}\right)$

has a property that $exp(-\tilde{S_{5}}(B))$ is the partial product of
$L_{E}(s)$

(2.1) $\prod_{\begin{subarray}{c}p<B,\\\ \textrm{ good
reduction}\end{subarray}}\left(1-a_{p}p^{-s}+p^{1-2s}\right)^{-1},$

evaluated at $s=1$ (ignoring the primes of bad reduction). One expects that
$\tilde{S_{5}}(B)$ should be large if $E$ has a large rank since then the
partial product should rapidly approach zero. This sum was used in [EK20] as a
first step in finding rank-record breaking curves with fixed cyclic torsion
$\mathbb{Z}/n\mathbb{Z}$ for $n=2,3,\ldots 7$.

Recently, some new fundamental results have been discovered with the
assistance of deep neural networks in topology and representation theory
[DVB+21], combinatorics [Wag21], as well as some applications to problems in
statistics [IV22]. In number theory, the utility of machine learning methods
was shown in [HLO20] where the authors, among other things, successfully used
logistic regression for classifying elliptic curves of rank zero and one.

In this paper, we investigate a deep learning algorithm for rank
classification based on convolutional neural networks (CNN). These networks
take as an input the conductor of the elliptic curve together with the
sequence of normalized $a_{p}$-s (i.e. $a_{p}/\sqrt{p}$) for $p$ in a fixed
range and output the rank of the elliptic curve. We compare its performance to
that of Mestre-Nagao sums (defined in Section 3). A priori, it is not clear
how to decide on the rank of the elliptic curve based on the value of its
Mestre-Nagao sum (see related question (1) in Section 7 of [EK20]), so we
train simple fully connected neural network to do that task for us. Since the
answer critically depends on the conductor of the elliptic curve, these
networks, besides Mestre-Nagao sum, take the conductor of the elliptic curve
as an input. Training these networks revealed the optimal cutoff of specific
Mestre-Nagao sum for rank classification (for cutoffs of $S_{5}$ see Figure
3).

Architecture and the training process of our neural networks are described in
Section 5. For training, we used two datasets. One is LMFDB database [LMF22]
which contains $3,824,372$ elliptic curves defined over $\mathbb{Q}$, divided
in $2,917,287$ isogeny classes, with rank between $0$ and $5$ and conductor
less then $300,000,000$. The other one is a custom made dataset which consists
of $2,033,965$ elliptic curves with trivial torsion, the conductor less than
$10^{30}$ and the rank in the range between $0$ and $10$. For more information
about datasets see Section 4.

For each neural network (CNN or one of the Mestre-Nagao sums, in total $9$) we
have performed $24$ tests by varying

  * a)

dataset - LMFDB or custom,

  * b)

range of $a_{p}$-s - we considered $a_{p}$-s for $p<10^{3},10^{4},$ and
$10^{5}$,

  * c)

test curves - uniformly selected ($20\%$ from the dataset) or all curves in
top conductor range (which is $[10^{8},10^{9}]$ for LMFDB and
$[10^{29},10^{30}]$ for custom dataset),

  * d)

type of classification - binary or all ranks (for LMFDB the rank range is from
$0$ to $5$, and for custom dataset from $0$ to $10$).

In binary classification, curves are labeled as either of low or high rank.
For LMFDB high rank means rank $4$ (we did not consider $19$ rank $5$ curves),
while for custom dataset high rank is $8,9$ or $10$. Here, the idea is to test
how good neural networks are in detecting elliptic curves of “high” rank.

The neural networks have been selected to maximize Matthews correlation
coefficient (MCC) or phi coefficient, which is generally regarded as a
balanced measure of the quality of the classification even if the classes are
of very different sizes [BJEA17]. For binary classification, MCC is computed
using the following formula

$MCC=\frac{TP\cdot TN-FP\cdot
FN}{\sqrt{(TP+FP)\cdot(TP+FN)\cdot(TN+EP)\cdot(TN+FN)}},$

where $TP,FN,TN,FP$ denote, respectively, the number of true positives, false
negatives, true negatives, and false positives. Note that $MCC$ lies in the
segment $[-1,1]$ and $MCC=1$ only in the case of perfect classification.

For comparison of Mathews correlation coefficients of CNN vs. $S_{5}$
classifiers for uniform test set see Table 1.

| Number of $a_{p}$-s used
---|---
Type of classifier | LMFDB | custom dataset
$p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$ | $p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$
CNN | 0.9507 | 0.9958 | 0.9992 | 0.6129 | 0.7218 | 0.7958
$S_{5}$ | 0.6132 | 0.7774 | 0.8463 | 0.4987 | 0.5990 | 0.6696
Table 1. Comparison of Mathews correlation coefficients of CNN vs. $S_{5}$
classifiers for uniform test set.

In all the tests performed on LMFDB dataset CNN outperformed Mestre-Nagao sums
(this is especially true in the classification of all ranks). For example, in
the all ranks classification with $p<10,000$ in uniform range MCC of CNN is
$0.9958$ (in particular, CNN misclassified only $0.25\%$ of the curves) while
the best Mestre-Nagao sum $S_{2}$ has $CNN=0.8697$ (it misclassified $8.1\%$
of the curves). For more details see Table 5. Similarly, in the all ranks
classification with $p<10,000$ and top conductor range, MCC of CNN is $0.9289$
(see confusion matrix in Figure 2) while the best Mestre-Nagao sum $S_{0}$ has
$MCC=0.5057$ (see Figure 4), which is remarkable!

We also trained a fully connected neural network $\Omega$ whose inputs were
all Mestre-Nagao sums {$S_{0},\ldots,S_{6}$} (together with the conductor),
and interestingly on the LMFDB dataset this network performed much better than
any sum individually. For example, in the uniform range case above it attained
$MCC=0.9602$ and in the top range case $MCC=0.7013$.

A binary classification on the LMFDB dataset (which tries to identify curves
of rank $4$) ended up being an easy task for both CNN and the majority of
Mestre-Nagao sums - in a uniform mode they demonstrated the prefect
classification even in $p<1,000$ range (see Table 5 and 6).

The classification on the custom dataset was much more challenging for both
CNN and Mestre-Nagao sums. For example, in all ranks classification with
$p<10,000$ and uniform range, while CNN with $MCC=0.7218$ (it misclassified
$23\%$ of curves while for $3\%$ of the curves prediction missed true rank for
more than $1$) still outperformed the best Mestre-Nagao sum $S_{1}$ with
$MCC=0.6890$ (it misclassified $26\%$ of curves), it performed just a little
bit better than the network $\Omega$ for which $MCC=0.7069$ (see Table 7).

In the top conductor range (for all ranks classification on the custom
dataset) all heuristics were much less efficient. This is expected, at least
for Mestre-Nagao sums, since the curves in this test set have large conductors
(at least $10^{29}$) and thus we need more $a_{p}$’s to determine their
$L$-function. Mestre-Nagao sums slightly outperformed CNN in this setting. For
example, in $p<10,000$ mode MCC of CNN was $0.3019$ (it misclassified $61\%$
of curves while for $12\%$ of the curves prediction missed true rank for more
than $1$), while the best Mestre-Nagao sum $S_{2}$ had $MCC=0.3291$. For
complete information see Table 8.

In binary classification on the custom dataset (which tries to identify curves
of rank $8,9$ or $10$), CNN outperformed each Mestre-Nagao sum individually,
but it was a little bit worse than the neural network trained on all Mestre-
Nagao sums. For example, in the top conductor range for $p<10,000$ MCC of CNN
was $0.6774$ while MCC of all sums was $0.7091$.

##  3\. Mestre-Nagao sums

In this section, we define and motivate Mestre-Nagao sums that we are going to
use for rank classification.

Let $\Lambda(n)=\begin{cases}\log{p}&\textrm{if $n=p^{m}$}\\\
0&\textrm{otherwise}\end{cases}$ be the von Mangoldt function, and for
elliptic curve $E$ defined over $\mathbb{Q}$ with conductor $N_{E}$ let

$c_{n}=\begin{cases}\alpha_{p}^{m}+\beta_{p}^{m},&\textrm{if $n=p^{m}$ and
$p\nmid N_{E}$},\\\ a_{p}^{m},&\textrm{ if $n=p^{m}$ and $p|N_{E}$},\\\
0,&\textrm{otherwise},\end{cases}$

where $\alpha_{p}$ and $\beta_{p}$, for $p\nmid N_{E}$, are eigenvalues of the
Frobenius morphism at $p$ (thus $a_{p}=\alpha_{p}+\beta_{p}$ and
$\alpha_{p}\beta_{p}=p$). Note that for $m>1$ we have

$\alpha_{p}^{m}+\beta_{p}^{m}=(\alpha_{p}^{m-1}+\beta_{p}^{m-1})a_{p}-p(\alpha_{p}^{m-2}+\beta_{p}^{m-2}),$

hence for $p\nmid N_{E}$ it follows $c_{p^{m}}=c_{p^{m-1}}a_{p}-pc_{p^{m-1}}$.
These number are related to analytic rank $r$ of elliptic curve $E/\mathbb{Q}$
by the following identity

$-\frac{L^{\prime}_{E}(s)}{L_{E}(s)}=\sum_{n=1}^{\infty}\frac{c_{n}\Lambda(n)}{n^{s}}=\frac{r}{s-1}+\cdots.$

We consider the following Mestre - Nagao sums.

$\displaystyle S_{0}(B)$
$\displaystyle=\frac{1}{\log{B}}\sum_{\begin{subarray}{c}p<B,\\\ \textrm{ good
reduction}\end{subarray}}\frac{a_{p}(E)\log{p}}{p},$ $\displaystyle S_{1}(B)$
$\displaystyle=S_{0}(B)-\frac{1}{B\log{B}}\sum_{n\leq B}c_{n}\Lambda(n),$
$\displaystyle S_{2}(B)$ $\displaystyle=\frac{1}{\log{B}}\sum_{n\leq
B}\frac{c_{n}\Lambda(n)}{n}-\frac{1}{B\log{B}}\sum_{n\leq B}c_{n}\Lambda(n),$
$\displaystyle S_{3}(B)$ $\displaystyle=\sum_{\begin{subarray}{c}p<B,\\\
\textrm{ good
reduction}\end{subarray}}\frac{-a_{p}(E)+2}{p+1-a_{p}(E)}\log{p},$
$\displaystyle S_{4}(B)$
$\displaystyle=\frac{1}{B}\sum_{\begin{subarray}{c}p<B,\\\ \textrm{ good
reduction}\end{subarray}}-a_{p}(E)\log{p},$ $\displaystyle S_{5}(B)$
$\displaystyle=\sum_{\begin{subarray}{c}p<B,\\\ \textrm{ good
reduction}\end{subarray}}\log\left(\frac{p+1-a_{p}(E)}{p}\right)+\sum_{\begin{subarray}{c}p<B,\\\
\textrm{ split mult.
reduction}\end{subarray}}\log\left(3/2\cdot\frac{p-1}{p}\right),$
$\displaystyle S_{6}(\Delta)$
$\displaystyle=\frac{\log{N_{E}}}{2\Delta\pi}-\frac{\log{2\pi}}{\Delta\pi}+\frac{-1}{\Delta\pi}\sum_{p\leq\exp(2\pi\Delta)}\log{p}\sum_{k=1}^{\lfloor
2\pi\Delta/\log{p}\rfloor}\frac{c_{p^{k}}}{p^{k/2}}\left(1-\frac{k\log{p}}{2\pi\Delta}\right)$
$\displaystyle+\frac{1}{\pi}\Re\left\\{\int_{-\infty}^{\infty}\frac{\Gamma^{\prime}}{\Gamma}(1+it)\left(\frac{\sin(\Delta\pi
t)}{\Delta\pi t}\right)^{2}dt\right\\}.$

The sum $S_{0}$ was analyzed in detail in [KM22], where it was shown that if
the Riemann hypothesis for $L_{E}(s)$ is true, and if there exist the limit
$\lim_{B\rightarrow\infty}S_{0}(B)$, then the limit is $-r+1/2$ where $r$ is
analytic rank of $E/\mathbb{Q}$. The relation between $S_{0}$ and analytic
rank comes from the Perron’s formula applied to

$\frac{1}{2\pi
i}\int-\frac{L_{E}^{\prime}(s)}{L_{E}(s)}\frac{x^{s}}{s(s-1)}ds,\quad
x>1,\,d>3/2,\,a\in\mathbb{R}.$

From the proof of Theorem 6. in [KM22], by modifying the error term of
$S_{0}(B)$ one obtains sums $S_{1}$ and $S_{2}$ (equations (3.11) and (3.12))
that have essentially the same limit as $S_{0}$, but faster convergence.

Sums $S_{3}$ and $S_{4}$ were considered by Nagao [Nag92]. Note that
$S_{3}(B)$ is logarithmic derivative of the partial Euler product (2.1)
evaluated at $s=1$ (ignoring the primes of bad reduction). Since
$\frac{L_{E}^{\prime}(s)}{L_{E}(s)}=\frac{r}{s-1}+\cdots$, it is reasonable to
assume that for elliptic curve with large analytic rank $r$ the corresponding
$S_{3}$ sum will be large (see Section 1.3.3 in [Cam99]). Sum $S_{5}$ is the
modification of $\tilde{S_{5}}$ (from Introduction) at primes of split
multiplicative reduction (see Section 7. in [EK20]).

Sum $S_{6}$ is described in [Bob13]. It is based on the explicit formula for
$L_{E}(s)$ relating $S_{6}(\Delta)$ to the sum over nontrivial zeros
$\frac{1}{2}+i\gamma$ of $L_{E}(s)$, $\sum_{\gamma}f(\gamma)$, where
$f(z)=f(z;\Delta)=\left(\frac{\sin(\Delta\pi z)}{\Delta\pi z}\right)^{2}$.
Note that $f(0)=1$ and $f(x)\geq 0$ for all real $x$, thus,
$\sum_{\gamma}f(\gamma)$ gives an upper bound for $r$. Assuming Riemann
Hypothesis for $L_{E}(s)$, one has
$\lim_{\Delta\rightarrow\infty}S_{6}(\Delta)=r$, but computing $S_{6}(\Delta)$
becomes infeasible once $\Delta$ gets a little larger than $4.4$. For example,
already to compute $S_{6}(2)$ one needs $a_{p}$’s for all $p<286,751$.

For computing $S_{6}(\Delta)$ we used C version [Sut12] of PSAGE [S+11]
rankbound algorithm by Bober.

##  4\. Datasets

We have been training our models on two datasets.

###  4.1. LMFDB

LMFDB database [LMF22] contains $3,824,372$ elliptic curves defined over
$\mathbb{Q}$, distributed in $2,917,287$ isogeny classes. We selected one
representative per isogeny class (since isogeny curves have equal
$L$-functions and consequently same analytic rank). Since there are only $19$
curves with rank $5$, we didn’t take any on them in consideration. See Table 2
for rank distribution. Moreover, $63.71\%$ of curves have trivial torsion,
while $31.18\%$ have $\mathbb{Z}/2\mathbb{Z}$ torsion.

rank | 0 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---
count | 1,404,510 | 1,887,132 | 493,291 | 37,334 | 2,086 | 19
proportion | 36.73% | 49.34% | 12.90% | 0.98% | 0.05% | 0.00%
Table 2. Distribution of ranks in LMFDB.

###  4.2. Custom dataset

Our custom dataset contains $2,074,863$ elliptic curves with trivial torsion
and conductor less than $10^{30}$. See Table 3 for the number of curves in a
specific conductor range. Curves have a rank between $0$ and $10$. See Table 4
for distribution of curves by rank.

range | $1,10^{5}$ | $10^{5},10^{10}$ | $10^{10},10^{15}$ | $10^{15},10^{20}$ | $10^{20},10^{25}$ | $10^{25},10^{29}$ | $10^{29},10^{30}$
---|---|---|---|---|---|---|---
proportion | 0.94% | 21.60% | 26.97% | 21.27% | 17.02% | 10.19% | 1.97%
Table 3. Distribution of conductors in custom dataset. rank | 0 | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---|---
count | $78,755$ | $121,966$ | $179,593$ | $435,195$ | $543,713$ | $405,129$
proportion | 3.79% | 5.87% | 8.65% | 20.97% | 26.20% | 19.52%
rank | 6 | 7 | 8 | 9 | 10 |
count | $210,439$ | $77,923$ | $19,272$ | $2,670$ | $201$ |
proportion | 10.14% | 3.75% | 0.92% | 0.12% | 0.01% |
Table 4. Distribution of ranks in custom dataset.

We used two methods for generating the dataset. First, we produced curves with
random Weierstrass coefficients. These were mostly curves of rank $0$ and $1$.

The curves of higher rank were obtained as random specializations of pencils
of cubics through randomly selected $k$ rational points in the plane, for
$k=2,3,\ldots,8$. Every such cubic (if nonsingular) was transformed to
Weierstrass form by the change of variables described in Chapter 8 of [Cas91].
We removed curves with nontrivial torsion subgroup.

Next, using PARI/GP [Par19] function ellrank we tried to compute ranks of all
previously generated curves (assuming the Parity conjecture). In the end, we
discarded those for which PARI/GP couldn’t find the rank. Note that ellrank in
computing the upper bound for the rank of the elliptic curve $E$ approximates
the rank of $\Sh(E/\mathbb{Q})[2]$ from below with the rank of
$\Sh(E/\mathbb{Q})[2]/2\Sh(E/\mathbb{Q})[4]$ (which it computes via Cassels
pairing), thus it can not determine the rank of $E$ if $\Sh(E)[4]$ is
nontrivial. Consequently, such curves are missing from our dataset.

##  5\. Neural network architectures and training procedure

The models we employ are convolutional neural networks (CNN). CNN could be
regarded as the composition of functions, so-called layers. Each layer is in
principle composition of a fittable affine mapping and an element-wise
nonlinear function. To detect and utilize any correlations in $a_{p}$ sequence
inherent to our data, for the affine mapping we use 1D matrix convolutional
operations, inspired by the success of 2D matrix convolutions in image
classification task [KSH12]. The point is that all Mestre-Nagao sums
inherently employ the same computation “rule” for each $a_{p}$, which we
expect could be efficiently modeled by a matrix convolution operation. Thus,
we can regard the training of a CNN as an optimization in the highly
dimensional parameterized space of Mestre-Nagao like heuristics.

Additionally, we use smaller fully connected neural networks (FCNN) for the
classification of curves using only precomputed values of selected Nagao-
Mestre sums and conductors.

All described models are implemented in Python using the PyTorch library
[PGM+19]. Models were trained using a server with dual Xeon 5218R CPUs (40
cores) equipped with 6 Nvidia Quadro RTX 5000 GPUs, each with 16 GB of RAM.

###  5.1. Architecture of CNNs

For each elliptic curve $E$, CNN takes as an input a matrix $M(E)$ of shape
$3\times\pi(N)$, where $N=10^{3},10^{4},10^{5}$ and the columns are indexed
with primes less than $N$. First row consist of $a_{p}$-s divided by
$\sqrt{p}$ (by Hasse theorem $a_{p}/\sqrt{p}\in[-2,2]$). Second row contains
constant sequence of normalized conductor, $\log{N_{E}}/\log{N_{max}}$, where
$N_{max}$ is the maximal conductor present in the dataset. Third row is a
sequence of $r_{n}$ (linear sweep from $-1$ do $1$) that for $n$-th prime
$p_{n}$ is equal to $-1+2n/\pi(N)$. In this way we input to the network the
relative position of prime $p$ in sequence of primes less than $N$.

Each CNN consist of the following sequence of convolutional layers:

  1. (1)

preparatory layer (increasing number of channels (rows in input matrix) from
$3$ to $64$)

  2. (2)

several input layers $I_{1},\ldots,I_{L_{1}}$, preserving the number of
channels and the length of the input sequence

  3. (3)

several reducing layers $R_{1},\ldots,R_{L_{2}}$, preserving the number of
channels and reducing the length of input sequence by a factor $2$
(convolution kernel stride is equal to $2$)

  4. (4)

several output layers $O_{1},\ldots,O_{L_{3}}$, preserving the number of
channels and the length of the input sequence

Each convolution layer is followed by an activation function and a batch
normalization layer, improving the convergence of the network training and
network generalization properties [IS15]. For the activation function, we
decided to use ReLU. Kernel size $KS$ is the same in each layer. The number of
reducing layers $L_{2}$ is chosen such that the length of the output of the
final reducing layer $R_{L_{2}}$ (and consequently the length of the output of
every output layer) is equal to $1$.

Following convolutional layers, we have one final fully connected layer
reducing dimensionality from $64$ to the number of classification labels
(ranks). Thus, our CNN architecture is fully determined with hyperparameters
$L_{1}$, $L_{2}$, $L_{3}$, and $KS$.

For instance, in the case of uniform range for LMFDB model with $p<10,000$ we
used the following values of hyperparameters: $L_{1}=0$, $L_{2}=11$,
$L_{3}=3$, $KS=17$. To find optimal values of hyperparameters we conducted
Bayesian optimization [Moč75] in the space of hyperparameters by training
altogether $500$ different variants of networks. Similar hyperparameters were
acquired for LMFDB in the case of $p<1,000$ and for $p<100,000$, due to
lengthy computations we just reused the hyperparameters for $p<10,000$, by
setting $L_{2}=14$. For each custom dataset CNN model, we took the same values
of hyperparameters as for the corresponding LMFDB model.

###  5.2. Architecture of FCNNs

Here the input to the network for each curve $E$ consists of one or more
precomputed values of Nagao-Mestre sums for $a_{p}$ in a certain range and
normalized conductor $N_{E}$, as in the case of CNNs. In every case, our FCNN
consists of an input linear layer having the size of the number of input
features (Nagao-Mestre sums and normalized conductor), four hidden linear
layers of size $128$, and an output linear layer of the size of the number of
classification labels (ranks). In between each of the linear layers, we use
dropout [NHK+14], which reduces overfitting during training, and ReLU
activation function. In our experience, the capacity of this network is more
than enough to learn the dependency of rank to the input.

###  5.3. Training procedure

To train a neural network means to optimize network parameters (weights in
convolutional and linear layers) in order to minimize a chosen loss function
over a given dataset.

#### Dataset

For neural network training, as customary the whole dataset is divided into
three disjoint subsets called training, validation, and test dataset. Only the
training and validation datasets are used in network training, thus the test
dataset is not ”seen“ by the network during the training and is only used at
the end to test the performance of already trained models. We used the $4:1$
ratio between the training and validation datasets.

#### Loss function

As we are training classifier networks, we decided to use the cross entropy
loss function as implemented in PyTorch library. Additionally, we supply
weights for the loss function to reflect on disproportional sizes of classes,
which is considered standard practice.

#### Optimizer hyperparameters

Our models are trained using the fast.ai library [H+18]. For the optimizer, we
use the usual choice of Adam algorithm [KB17]. For governing learning rate and
momentum scheduling we used a one-cycle policy with cosine annealing [ST18],
which ensures better convergence of model parameters to a broad optimum and
allows for better generalization of the trained model [Smi18]. For the
optimizer hyperparameters, we used different ranges of the maximum learning
rate for CNNs and FCNNs.

For CNNs the range was between $3.5\cdot 10^{-4}$ and $1.5\cdot 10^{-3}$.
Number of training epochs was equal to $40$, and batch size equal to $1024$
($p<10^{5}$) or $2048$ ($p<10^{3}$ and $p<10^{4}$). For Adam-specific
hyperparameters, we have used: $\beta_{2}=0.99$, $\epsilon=10^{-5}$, and
weight decay of $10^{-3}$.

For FCNNs, we employed Bayesian optimization for three hyperparameters:
dropout, maximum learning rate, and weight decay, by training $200$ different
models (networks) for each of $7$ Mestre-Nagao sums $\\{S_{0},\ldots,S_{6}\\}$
and also for $\Omega$, in each of the $24$ test variants. Finally, in each of
these $192$ tests performed we took the model having the best MCC on the test
set. The number of training epochs for every model was equal to $10$ and the
rest of the Adam-specific hyperparameters were the same as in CNN training.

##  6\. Results

We present rank classification results using the Mathews correlation
coefficient in each of the $216$ tests. We use two different datasets (LMFDB
and custom, see Section 4), three $a_{p}$-s ranges ($p<10^{3},10^{4},10^{5}$),
two modes for selecting test curves ($20\%$ uniformly selected or top
conductor range), and two types of classification (binary or all ranks).

For binary classification we always trained our networks for all ranks
classification and to obtain binary classification we merged all rank labels
into two categories, ”low“ and ”high“ ranks - effectively only reinterpreting
all ranks classification, but rerunning Bayesian optimization for
hyperparameters. For LMFDB low ranks are $0,1,2,3$ and high ranks are $4$, and
for custom dataset low ranks are $0,\ldots,7$ and high ranks are $8,9,10$.
Interestingly, this procedure produced much better binary classification
results than training directly a binary classifier model.

Regarding CNN models, for each out of $12$ test variants (varying two
datasets, three ranges, and two modes for selecting test curves) we trained
only $10$ different networks due to high computational expense. For instance
training of models in the worst case of $p<10^{5}$ took over $14$ hours per
model on a single GPU. On the other hand, training of FCNNs for Mestre-Nagao
sums was much less computational demanding, so for each of the $192$ test
cases, we trained at least $200$ models (which took several minutes per model
on a single GPU), using Bayesian optimization in the space of hyperparameters,
as described in Section 5.3. For each of the test cases, we selected the best-
performing model out of all trained models.

###  6.1. LMFDB

Test results are presented in Tables 5 and 6.

In Table 5 are maximal values of MCC obtained for different classifiers.
Classifiers are trained using $80\%$ of randomly selected curves from the
LMFDB dataset, having a conductor less than $10^{8}$. MCC is computed on the
other $20\%$ of the dataset, not seen during the training. Classifiers (CNN,
$\Omega$, and Mestre-Nagao sums from Section 3) are trained using values of
$a_{p}$-s for primes $p$ less than $10^{3}$, $10^{4}$ or $10^{5}$, and the
value of the elliptic curve conductor.

| Number of $a_{p}$-s used
---|---
Type of classifier | classify all ranks | binary classification
$p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$ | $p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$
CNN | 0.9507 | 0.9958 | 0.9992 | 1 | 1 | 1
$S_{0}$ | 0.6823 | 0.8435 | 0.9068 | 0.9877 | 0.9758 | 0.9729
$S_{1}$ | 0.6848 | 0.8507 | 0.9301 | 0.9968 | 1 | 1
$S_{2}$ | 0.7277 | 0.8697 | 0.9359 | 0.9938 | 1 | 1
$S_{3}$ | 0.6933 | 0.8499 | 0.9142 | 0.9420 | 0.9842 | 0.9726
$S_{4}$ | 0.2678 | 0.3015 | 0.1525 | 0.2188 | 0.1103 | 0.0744
$S_{5}$ | 0.6132 | 0.7774 | 0.8463 | 0.9968 | 0.9968 | 1
$S_{6}$ | 0.6969 | 0.8647 | 0.9381 | 1 | 1 | 1
$\Omega$ | 0.8685 | 0.9602 | 0.9826 | 1 | 1 | 1
Table 5. LMFDB with uniform test dataset.

Figure 1. Confusion matrix of CNN for $p<10^{5}$ and uniform test dataset.

In Table 6 classifiers are trained using all curves from the LMFDB dataset
having a conductor less than $10^{8}$. MCC is computed on all curves from the
LMFDB dataset having a conductor between $10^{8}$ and $10^{9}$, not seen
during the training.

| Number of $a_{p}$-s used
---|---
Type of classifier | classify all ranks | binary classification
$p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$ | $p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$
CNN | 0.5669 | 0.9289 | 0.9846 | 0.9811 | 0.9996 | 1
$S_{0}$ | 0.2880 | 0.5057 | 0.6545 | 0.9470 | 0.9750 | 0.9756
$S_{1}$ | 0.2791 | 0.4883 | 0.6658 | 0.9669 | 0.9996 | 1
$S_{2}$ | 0.2790 | 0.4968 | 0.6730 | 0.9609 | 0.9996 | 1
$S_{3}$ | 0.2897 | 0.5030 | 0.6574 | 0.9151 | 0.9716 | 0.9676
$S_{4}$ | 0.1352 | 0.1424 | 0.1850 | 0.2438 | 0.1411 | 0.0647
$S_{5}$ | 0.2960 | 0.3913 | 0.5261 | 0.9538 | 0.9917 | 0.9968
$S_{6}$ | 0.2632 | 0.4542 | 0.6416 | 0.9644 | 0.9996 | 1
$\Omega$ | 0.4433 | 0.7013 | 0.8530 | 0.9796 | 1 | 1
Table 6. LMFDB with top conductor range.

Figure 2. Confusion matrix of CNN for LMFDB and $p<10^{4}$ with top conductor
range.

Figure 3. Rank cutoffs of classifier $S_{5}$ as a function of a conductor,
trained on LMFDB with uniform test set and $p<10^{4}$. On x-axis are
$\log_{10}$ values of conductors and on y-axis are values of sum $S_{5}$. The
unexpected shape of the cutoff between ranks $3$ and $4$ is the consequence of
a small number of rank $4$ curves with a small conductor, which are present in
the dataset.

Figure 4. Confusion matrix of $S_{0}$ for LMFDB and $p<10^{4}$ with top
conductor range.

###  6.2. Custom dataset

Test results are presented in Tables 7 and 8.

In Table 7 are maximal values of MCC obtained for different classifiers.
Classifiers are trained using $80\%$ of randomly selected curves from the
custom dataset with a conductor less than $10^{29}$. As in the previous
section, MCC is computed on the other $20\%$ of the dataset, not seen during
the training.

| Number of $a_{p}$-s used
---|---
Type of classifier | classify all ranks | binary classification
$p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$ | $p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$
CNN | 0.6129 | 0.7218 | 0.7958 | 0.6695 | 0.8335 | 0.9425
$S_{0}$ | 0.5738 | 0.6782 | 0.7462 | 0.6761 | 0.8275 | 0.9175
$S_{1}$ | 0.5780 | 0.6890 | 0.7592 | 0.6484 | 0.8317 | 0.9309
$S_{2}$ | 0.5649 | 0.6761 | 0.7521 | 0.6407 | 0.8252 | 0.9206
$S_{3}$ | 0.5551 | 0.6616 | 0.7361 | 0.6515 | 0.8118 | 0.9074
$S_{4}$ | 0.2893 | 0.2472 | 0.2251 | 0.4271 | 0.4179 | 0.3874
$S_{5}$ | 0.4987 | 0.5990 | 0.6696 | 0.5230 | 0.6919 | 0.7956
$S_{6}$ | 0.5230 | 0.6509 | 0.7361 | 0.5179 | 0.7554 | 0.8961
$\Omega$ | 0.5999 | 0.7069 | 0.7807 | 0.6821 | 0.8527 | 0.9412
Table 7. Custom dataset with uniform test set

Figure 5. Confusion matrix of CNN for custom dataset and $p<10^{4}$ with
uniform test set.

Figure 6. Confusion matrix of $\Omega$ for custom dataset and $p<10^{4}$ with
uniform test set.

In Table 8 classifiers are trained using all curves from the custom dataset
with conductor less than $10^{29}$. MCC is computed on all curves from the
custom dataset with conductor between $10^{29}$ and $10^{30}$, not seen during
the training.

| Number of $a_{p}$-s used
---|---
Type of classifier | classify all ranks | binary classification
$p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$ | $p<10^{3}$ | $p<10^{4}$ | $p<10^{5}$
CNN | 0.2147 | 0.3019 | 0.3655 | 0.5449 | 0.6774 | 0.8082
$S_{0}$ | 0.2533 | 0.3233 | 0.3719 | 0.5866 | 0.6978 | 0.7900
$S_{1}$ | 0.2573 | 0.3291 | 0.3834 | 0.5649 | 0.6925 | 0.7946
$S_{2}$ | 0.2340 | 0.3118 | 0.3688 | 0.5472 | 0.6814 | 0.7856
$S_{3}$ | 0.2556 | 0.3189 | 0.3645 | 0.5688 | 0.6830 | 0.7807
$S_{4}$ | 0.1234 | 0.1228 | 0.1024 | 0.3966 | 0.3935 | 0.3842
$S_{5}$ | 0.2081 | 0.2858 | 0.3380 | 0.4860 | 0.6071 | 0.6880
$S_{6}$ | 0.1803 | 0.2757 | 0.3527 | 0.4321 | 0.6217 | 0.7326
$\Omega$ | 0.2622 | 0.3246 | 0.3905 | 0.5931 | 0.7091 | 0.8118
Table 8. Custom dataset with top conductor range.

###  6.3. Discussion

As expected, classifiers based on $S_{0}$, $S_{1}$, and $S_{2}$ have very
similar performance across all tested regimes (improved convergence of sums
$S_{1}$ and $S_{2}$ does not seem to improve the quality of the
classification). They have the best performance of all considered Mestre-Nagao
sums. Classifiers based on $S_{3}$ have overall slightly worse performance
than the first three sums. Peculiarly, $S_{4}$ underperforms in all tests, and
its performance, in contrast to every other sum, decreases as the number of
used $a_{p}$-s increases. Classifier $S_{5}$ performs well but compared to the
first four sums is in most cases significantly lesser. While classifier
$S_{6}$ is among the best on the curves having small conductors, its
performance fades on larger conductors. Classifier $\Omega$, which is based on
all Mestre-Nagao sums simultaneously, is practically always better than any
other sum-based classifier, especially on curves of a small conductor.

Finally, CNN based classifiers on the LMFDB dataset were significantly better
than $\Omega$ (and consequently any other sum-based classifier). On custom
datasets with a uniform test set, they were slightly better than $\Omega$ and
other classifiers, but on the custom dataset with top range conductor, they
performed worse. Notice that every CNN-based classifier was selected as the
best one out of only a $10$ trained model, while each Mestre-Nagao sum-based
classifier was selected out of at least $200$ trained models, whose
hyperparameters were additionally optimized using the Bayesian optimization
technique. We expect that the quality of CNN-based classification would
increase with more trained models and possibly surpass other models. As CNN
model training is much more computationally expensive, more training was
unfeasible for us. It is fascinating how CNN almost perfectly classifies LMDFB
with uniform test database (see Figure 1) - it misclassified only $0.045\%$ of
all curves! On the other hand for custom dataset both CNN and $\Omega$
classifiers perform similarly - their confusion matrices look almost identical
(see Figures 5 and 6).

Additionally, we noticed that all classifiers made the most mistakes in
classifying curves of rank $0$ and $1$. As a byproduct of training Mestre-
Nagao sum models, we learned the optimal cutoffs for rank classification (see
Figure 3), which is of interest when searching for curves of high rank (see
Section 7 of [EK20]).

##  7\. Future research

This paper left us with some open questions which may be addressed in the
future projects:

  1. (1)

CNN and $\Omega$ models could be used as a substitution for the current use of
the Mestre-Nagao sums in the search for elliptic curves of high rank. Also,
these models could be trained on other elliptic curves datasets, such as
curves with non-trivial torsion, and with additional input of the root number
(to employ the Parity conjecture).

  2. (2)

It is unclear why CNN works much better than all of the other Mestre-Nagao sum
based models on the curves from the LMFDB (or in general on curves of a small
conductor). Did CNN discover some new mathematics?

  3. (3)

Suboptimal choices in CNN model architecture and optimizer hyperparameters
(see Section 5) may lead to underperforming classifiers. One could perform
more extensive optimization of hyperparameters, especially in the case of the
custom dataset.

  4. (4)

Using some of the more advanced neural network architectures, originally
developed for natural language processing tasks (see [VSP+17]) may lead to
better performing models. The idea is that such architectures can more easily
extract distant correlations in the $a_{p}$-s sequence.

## Acknowledgments

The first author was supported by the Croatian Science Foundation under the
project no. IP-2018-01-1313, and by the QuantiXLie Center of Excellence, a
project co-financed by the Croatian Government and European Union through the
European Regional Development Fund - the Competitiveness and Cohesion
Operational Programme (Grant KK.01.1.1.01.0004). The second author was
supported by Croatian Science Foundation (HRZZ) grant PZS-2019-02-3055 from
“Research Cooperability” program funded by the European Social Fund.

## References

  * [BCDT01] Christophe Breuil, Brian Conrad, Fred Diamond, and Richard Taylor.  On the modularity of elliptic curves over $\mathbb{Q}$: wild 3-adic exercises.  J. Am. Math. Soc., 14(4):843–939, 2001.
  * [BJEA17] Sabri Boughorbel, Fethi Jarray, and Mohammed El-Anbari.  Optimal classifier for imbalanced data using matthews correlation coefficient metric.  PLOS ONE, 12(6):1–17, 06 2017.
  * [Bob13] Jonathan W. Bober.  Conditionally bounding analytic ranks of elliptic curves.  In ANTS X—Proceedings of the Tenth Algorithmic Number Theory Symposium, volume 1 of Open Book Ser., pages 135–144. Math. Sci. Publ., Berkeley, CA, 2013.
  * [Cam99] Garikai Campbell.  Finding Elliptic Curves and Families of Elliptic Curves over $\mathbb{Q}$ of Large Rank.  PhD thesis, Rutgers University, 1999.
  * [Cas91] J. W. S. Cassels.  Lectures on elliptic curves, volume 24 of London Mathematical Society Student Texts.  Cambridge University Press, Cambridge, 1991.
  * [Duj] Andrej Dujella.  Infinite families of elliptic curves with high rank and prescribed torsion.  https://web.math.pmf.unizg.hr/~duje/tors/generic.html.
  * [DVB+21] A. Davis, P. Veličković, L. Buesing, et al.  Advancing mathematics by guiding human intuition with ai.  Nature, 600:70–74, 2021.
  * [EK20] Noam D. Elkies and Zev Klagsbrun.  New rank records for elliptic curves having rational torsion.  In ANTS XIV. Proceedings of the fourteenth algorithmic number theory symposium, Auckland, New Zealand, virtual event, June 29 – July 4, 2020, pages 233–250. Berkeley, CA: Mathematical Sciences Publishers (MSP), 2020\.
  * [Elk06] Noam D. Elkies.  $Z^{28}$ in $E(\mathbb{Q})$.  Number Theory Listserver, 2006.
  * [H+18] Jeremy Howard et al.  fastai.  https://github.com/fastai/fastai, 2018.
  * [HLO20] Yang-Hui He, Kyu-Hwan Lee, and Thomas Oliver.  Machine-learning arithmetic curves.  arXiv:2012.04084, 2020.
  * [IS15] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.  cite arxiv:1502.03167.
  * [IV22] Tomislav Ivek and Domagoj Vlah.  Reconstruction of incomplete wildfire data using deep generative models.  arXiv:2201.06153, 2022.
  * [KB17] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2017.
  * [KM22] Seoyoung Kim and M. Ram Murty.  From the Birch and Swinnerton-Dyer conjecture to Nagao’s conjecture.  to appear in Math. Comp., 2022.  With an appendix by Andrew V. Sutherland.
  * [KSH12] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  Advances in Neural Information Processing Systems, 25:1106–1114, 2012.
  * [LMF22] The LMFDB Collaboration.  The L-functions and modular forms database.  http://www.lmfdb.org, 2022.  [Online; accessed 4 July 2022].
  * [Maz77] B. Mazur.  Modular curves and the Eisenstein ideal.  Publ. Math., Inst. Hautes Étud. Sci., 47:33–186, 1977.
  * [Mes82] Jean-François Mestre.  Construction d’une courbe elliptique de rang $\geq 12$.  C. R. Acad. Sci., Paris, Sér. I, 295:643–644, 1982.
  * [Moč75] J. Močkus.  On Bayesian Methods for Seeking the Extremum, pages 400–404.  Springer Berlin Heidelberg, Berlin, Heidelberg, 1975.
  * [Nag92] Koh-ichi Nagao.  Examples of elliptic curves over $\mathbb{Q}$ with rank $\geq 17$.  Proc. Japan Acad., Ser. A, 68(9):287–289, 1992.
  * [NHK+14] Srivastava Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research, 15(56):1929–1958, 2014.
  * [Par19] PARI/GP, version 2.11.1.  Bordeaux, 2019.  http://pari.math.u-bordeaux.fr/.
  * [PGM+19] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada, pages 8024–8035, 2019.
  * [Poi01] H. Poincaré.  Sur les propriétés arithmétiques des courbes algébriques.  Journ. de Math. (5), 7:161–233, 1901.
  * [PPVW19] Jennifer Park, Bjorn Poonen, John Voight, and Melanie Matchett Wood.  A heuristic for boundedness of ranks of elliptic curves.  J. Eur. Math. Soc. (JEMS), 21(9):2859–2903, 2019.
  * [S+11] William A. Stein et al.  Purple SAGE, 2011.  http://purple.sagemath.org/.
  * [Smi18] Leslie N. Smith.  A disciplined approach to neural network hyper-parameters: Part 1 – learning rate, batch size, momentum, and weight decay, 2018.
  * [ST18] Leslie N. Smith and Nicholay Topin.  Super-convergence: Very fast training of neural networks using large learning rates, 2018.
  * [Sut12] Andrew Sutherland.  smalljac, 2012.  https://math.mit.edu/~drew/.
  * [VSP+17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [Wag21] Adam Z. Wagner.  Constructions in combinatorics via neural networks.  arXiv:2104.14516, 2021.
  * [Wat15] Mark Watkins.  A discursus on 21 as a bound for ranks of elliptic curves over q, and sundry related topics.  http://magma.maths.usyd.edu.au/~watkins/papers/DISCURSUS.pdf, 2015\.
  * [WDE+14] Mark Watkins, Stephen Donnelly, Noam D. Elkies, Tom Fisher, Andrew Granville, and Nicholas F. Rogers.  Ranks of quadratic twists of elliptic curves.  In Numéro consacré au trimestre “Méthodes arithmétiques et applications”, automne 2013, pages 63–98. Besançon: Presses Universitaires de Franche-Comté, 2014.
  * [Wil95] Andrew Wiles.  Modular forms, elliptic curves, and Fermat’s Last Theorem.  In Proceedings of the international congress of mathematicians, ICM ’94, August 3-11, 1994, Zürich, Switzerland. Vol. I, pages 243–245. Basel: Birkhäuser, 1995.