# Parametrizing the moduli of non-special curves of genus five and its
application to finding curves with many rational points

Momonari Kudo and Shushi Harashita Department of Mathematical Informatics,
Graduate School of Information Science and Technology, The University of
TokyoGraduate School of Environment and Information Sciences, Yokohama
National University.

###### Abstract

In algebraic geometry, it is important to give good parametrizations of spaces
of curves, theoretically and also practically. In particular, the case of non-
hyperelliptic curves is the central issue. In this paper, we give a very
effective parametrizations of the moduli of curves of genus $5$ which are
neither hyperelliptic nor trigonal. After that, we construct an algorithm for
a complete enumeration of non-special curves of genus $5$ having more rational
points than a prescribed number, where “non-special” here means non-
hyperelliptic and non-trigonal with mild singularities of the associated
sextic model which we propose. As an application, we execute an implementation
on computer algebra system MAGMA of the algorithm for curves over the prime
field of characteristic $3$.

000Keywords:  Algebraic curves, Rational points, Curves of genus five,
Canonical curves
2010 Mathematical Subject Classification:  14G05, 14G15, 14H10, 14H45, 14H50,
14Q05, 68W30

##  1 Introduction

Let $K$ be a field and let $\mathbb{P}^{n}$ denote the projective $n$-space
over $K$. Parameterizing the space of curves over $K$ of given genus is an
important task in algebraic geometry, number theory and arithmetic geometry.
In particular, it is meaningful to construct families of curves, with explicit
defining equations, that contain all curves of genus $g$. For the
hyperelliptic case, it is well-known that any hyperelliptic curve over $K$ of
genus $g$ is the normalization of $y^{2}=f(x)$ for a square-free polynomial
$f(x)\in K[x]$ of degree $2g+1$ or $2g+2$ if the characteristic of $K$ is odd.
In this case, the defining equation $y^{2}=f(x)$ is assumed to have at most
$2g-1$ (resp. $2g$) unknown coefficients if $K$ is algebraically closed (resp.
finite). However, for a family of non-hyperelliptic curves, it is not so easy
in general to find their defining equations, with parameters (unknown
coefficients) as few as possible, by choosing a suitable model of the family.
Note that for $g\geq 2$, a curve is non-hyperelliptic if and only if the
canonical sheaf is very ample (cf. [6, Chap. IV, Prop. 5.2]). For a non-
hyperelliptic curve, its image of the embedding by the canonical linear system
is called a canonical curve.

For genus $3$, we find a nice parametrization in the paper by Lercier,
Ritzenthaler, Rovetta and Sijsling [13]. For genus $4$, a canonically embedded
curve is the complete intersection of a quadric and a cubic in
${\mathbb{P}}^{3}$. In [7] and [10], we discussed reductions of the space of
pairs of quadratic forms and cubic forms by the projective general linear
group $\operatorname{PGL}_{4}(K)$ of degree $4$.

The next target is the case of $g=5$. In this case, it is known that there are
two types of non-hyperelliptic curves; trigonal or not. For the trigonal case,
we already studied a quintic model in $\mathbb{P}^{2}$ of a trigonal curve of
genus $5$ in order to enumerate superspecial ones over small finite fields
[9]. Under some assumptions, we presented reductions of quintic forms defining
trigonal curves of genus $5$, see [9, Section 3] for details. The remaining
case is that the curve is non-hyperelliptic and non-trigonal, and it is known
that the curve is realized as the complete intersection of three quadratic
hypersurfaces in $\mathbb{P}^{4}$ (cf. [6, Chap. IV, Example 5.5.3 and
Exercises 5.5]). This case is quite more difficult than the hyperelliptic and
trigonal cases, since we need so many parameters to give three quadratic forms
in 5 variables. Although it is natural to reduce the parameters by using the
natural action by $\operatorname{PGL}_{5}(K)$, it would be hopeless to give an
efficient reduction in this way, since the group $\operatorname{PGL}_{5}(K)$
is huge and complicated.

In this paper, we give a new effective parameterization of the space of
canonical and non-trigonal curves of genus $5$. Specifically, it will be
proved that any canonical and non-trigonal curve $C$ of genus $5$ is bi-
rational to a sextic $C^{\prime}=V(F)$ in $\mathbb{P}^{2}$. Note that, in most
cases, $C^{\prime}$ has five double points, or one triple point and two double
points (in such a case we say that $C$ is non-special). We also show that the
dimension of the space of non-special curves with fixed singularities is at
most $12$, which is just(!) the dimension of the moduli space of curves of
genus $5$. More precisely, the coefficients of $F$ have linear expressions in
$12$ parameters, and the expressions can be all computed.

As an applications of this parametrization, we present an algorithm to
enumerate non-special curves $C$ over a finite field $K$ of genus $5$ with
prescribed number of rational points. Executing the algorithm for
$K=\mathbb{F}_{3}$ on Magma, we obtain the below theorem. Note that we
determined all the possible positions of singular points before the execution.

######  Theorem 1.

The maximal number of $\mathbb{F}_{9}$-rational points on a non-special curve
$C$ of genus $5$ defined over $\mathbb{F}_{3}$ (and not over $\mathbb{F}_{9}$)
is $32$. Moreover, there are exactly four $\mathbb{F}_{9}$-isogeny classes of
Jacobian varieties of non-special curves $C$ of genus $5$ over
$\mathbb{F}_{3}$ with 32 $\mathbb{F}_{9}$-rational points, whose Weil
polynomials are

  1. (1)

$(t^{2}+2t+9)(t^{2}+5t+9)^{4}$,

  2. (2)

$(t+3)^{2}(t^{4}+8t^{3}+32t^{2}+72t+81)^{2}$,

  3. (3)

$(t+3)^{4}(t^{2}+2t+9)(t^{2}+4t+9)^{2}$,

  4. (4)

$(t+3)^{6}(t^{2}+2t+9)^{2}$.

In Section 5, examples of non-special curves $C$ over $\mathbb{F}_{3}$ with
$\\#C(\mathbb{F}_{9})=32$ will be given.

As in the web-site manypoints.org [5], the maximal number of
$\\#C(\mathbb{F}_{9})$ of curves $C$ of genus $5$ over $\mathbb{F}_{9}$ is
unknown but is known to belong between $32$ and $35$ (this upper bound is due
to Lauter [12]). On the web-site, three examples of $C$ with 32
$\mathbb{F}_{9}$-rational points are listed. The above theorem gives at least
one new example. More concretely, the Weil polynomial of Fischer’s example
$(x^{4}+1)y^{4}+2x^{3}y^{3}+y^{2}+2xy+x^{4}+x^{2}=0$ is
$(t^{2}+2t+9)(t^{2}+5t+9)^{4}$. In fact, this curve appears in our
computation, since we obtain a sextic form in $1/x$ and $1/y$ (having distinct
5 singular points) by dividing the example by $x^{4}y^{4}$. The example of
Ramos-Ramos [14] (submitted by Ritzenthaler to the site)
$y^{8}=a^{2}x^{2}(x^{2}+a^{7})$ with $a^{2}+a+2=0$ has Weil polynomial
$(t+3)^{6}(t^{2}+2t+9)^{2}$. This curve is defined over $\mathbb{F}_{9}$, but
the above theorem found a curve over the prime field $\mathbb{F}_{3}$ with the
same Weil polynomial. Anyway, from this theorem, we see that if one wants to
find curves of genus $5$ over $\mathbb{F}_{9}$ with $\\#C(\mathbb{F}_{9})>32$,
one needs to search those not defined over $\mathbb{F}_{3}$ or curves whose
sextic models have more complex singularities.

Our explicit parametrization and the algorithm presented in this paper may
lead to fruitful applications both for theory and computation, such as
classifying canonical and non-trigonal curves of genus $5$ with invariants
(Hasse-Witt rank and Ekedahl-Oort type and so on). Some open problems will be
summarized in Section 6. One of our future works is to enumerate superspecial
non-special curves of genus $5$ over finite fields.

### Acknowledgments

The contents of this paper was presented at Effective Methods in Algebraic
Geometry (MEGA 2021). The authors thank the anonymous referees assigned by the
organization of this conference for their comments and suggestions. All of
them are taken into account for improving the presentation of the paper. The
author is also very grateful to Christophe Ritzenthaler for helpful comments.
This work was supported by JSPS Grant-in-Aid for Scientific Research (C)
17K05196 and 21K03159, and JSPS Grant-in-Aid for Young Scientists 20K14301.

##  2 Non-hyperelliptic and non-trigonal curves of genus $5$

In this section, we study curves of genus $5$ which are neither hyperelliptic
nor trigonal. In order to parameterize these curves, we propose using the
realization of these curves by their sextic models in $\mathbb{P}^{2}$. This
method is much more effective than using the realization by the complete
intersection of three quadratic forms in ${\mathbb{P}}^{4}$. Here we give an
explicit construction of the sextic model for our purpose. Although one may
find a more conceptional way to construct such a sextic model in a general
curve in [1, Chap. VI, Exercises, F-24 on p. 275], it requires some
assumptions and does not explain all of this section.

###  2.1 Sextic models

The canonical model of a non-hyperelliptic and non-trigonal curve of genus $5$
is the intersection of three quadrics in ${\mathbb{P}}^{4}$. Let $C$ be such a
curve, say $V(\varphi_{1},\varphi_{2},\varphi_{3})$ in ${\mathbb{P}}^{4}$ for
three quadratic forms $\varphi_{1}$, $\varphi_{2}$ and $\varphi_{3}$ in
$x_{0},x_{1},x_{2},x_{3},x_{4}$. A sextic model associated to $C$ is obtained
by additional data: two points $P$ and $Q$ on $C$. We find a short explanation
of this construction in [6, Chap. IV, Example 5.5.3], but that appears away
from the context of looking at the space of curves. Here we give an explicit
construction toward parameterizing the space of these curves. By a linear
transformation, we may assume that

$P=(1:0:0:0:0)\quad\text{and}\quad Q=(0:0:0:0:1).$

Since $\varphi_{i}$ vanishes at $P$ and $Q$, the quadratic forms $\varphi_{i}$
($i=1,2,3$) must be of the form

$\varphi_{i}=a_{i}\cdot x_{0}x_{4}+f_{i}\cdot x_{0}+g_{i}\cdot x_{4}+h_{i},$
(2.1.1)

where $a_{i}\in K$ and $f_{i}$ and $g_{i}$ are linear forms in
$x_{1},x_{2},x_{3}$ and $h_{i}$ is a quadratic form in $x_{1},x_{2},x_{3}$.
Put

$(v_{1},v_{2},v_{3}):=-(h_{1},h_{2},h_{3})\cdot\Delta_{A}$

where $\Delta_{A}$ is the adjugate matrix of

$A:=\begin{pmatrix}a_{1}&a_{2}&a_{3}\\\ f_{1}&f_{2}&f_{3}\\\
g_{1}&g_{2}&g_{3}\end{pmatrix}.$

Since $(v_{1},v_{2},v_{3})=\det(A)(x_{0}x_{4},x_{0},x_{4})$ on $C$, the sextic

$C^{\prime}:\quad\det(A)\cdot v_{1}-v_{2}v_{3}=0$

in ${\mathbb{P}}^{2}=\operatorname{Proj}K[x_{1},x_{2},x_{3}]$ is birational to
$C$. Here we need to see that $\det A\neq 0$ holds generically. Indeed, if
$\det A$ were identically zero, then from (2.1.1) we have $(3-{\rm rank}A)$
quadratic forms only in $x_{1},x_{2},x_{3}$. If ${\rm rank}A<2$, then this
contradicts with $C$ is irreducible. If ${\rm rank}A=2$, then let $\psi$ be
the quadratic form in $x_{1},x_{2},x_{3}$; then there is a dominant morphism
$C\to V(\psi)\subset{\mathbb{P}}^{2}$, which turns out to be of degree $2$.
This contradicts with the assumption that $C$ is not hyperelliptic.

This construction of the sextic $C^{\prime}$ from $C$ with $P$ and $Q$ is
canonical in the following sense:

######  Proposition 2.1.1.

  1. (1)

Suppose
$\langle\varphi_{1},\varphi_{2},\varphi_{3}\rangle=\langle\phi_{1},\phi_{2},\phi_{3}\rangle$
as a linear space over a field defining $\varphi_{i}$ and $\phi_{i}$ for all
$i=1,2$ and $3$. Then $C^{\prime}$ obtained from
$\varphi_{1},\varphi_{2},\varphi_{3}$ is the same as that obtained from
$\phi_{1},\phi_{2},\phi_{3}$.

  2. (2)

The coordinate change

$(x_{0},x_{1},x_{2},x_{3},x_{4})\mapsto(x_{4},x_{1},x_{2},x_{3},x_{0})$

does not change the sextic.

  3. (3)

The coordinate change

$(x_{0},x_{1},x_{2},x_{3},x_{4})\mapsto(x_{0}+\phi,x_{1},x_{2},x_{3},x_{4}+\psi)$

with any linear $\phi,\psi$ in $x_{1},x_{2},x_{3}$ does not change the sextic.

###### Proof.

(1) Let $B$ be the square matrix of size $3$ such that
$(\varphi_{1},\varphi_{2},\varphi_{3})B=(\phi_{1},\phi_{2},\phi_{3})$. Let
$v_{i}^{\prime}$ be $v_{i}$ associated to $\phi_{1},\phi_{2},\phi_{3}$. Then

$(v^{\prime}_{1},v^{\prime}_{2},v^{\prime}_{3})=-(h_{1},h_{2},h_{3})B\cdot\det(AB)(AB)^{-1}=\det(B)(v_{1},v_{2},v_{3}).$

Hence we have
$\det(AB)v^{\prime}_{1}-v^{\prime}_{2}v^{\prime}_{3}=\det(B)(\det(A)v_{1}-v_{2}v_{3})$.

(2) The matrix $A$ for the new coordinate is

$A^{\prime}:=\begin{pmatrix}a_{1}&a_{2}&a_{3}\\\ g_{1}&g_{2}&g_{3}\\\
f_{1}&f_{2}&f_{3}\end{pmatrix}.$

Let $v_{i}^{\prime}$ be $v_{i}$ associated to the new coordinate. It is
straightforward to see
$(v^{\prime}_{1},v^{\prime}_{2},v^{\prime}_{3})=(-v_{1},v_{3},v_{2})$. Thus
the sextic does not change.

(3) Let $A^{\prime}$, $h^{\prime}_{i}$ and $v^{\prime}_{i}$ be the matrix $A$,
the quadratic form $h_{i}$ and $v_{i}$ for the new coordinate. Then we have

$A^{\prime}=\begin{pmatrix}1&0&0\\\ \psi&1&0\\\
\phi&0&1\end{pmatrix}\begin{pmatrix}a_{1}&a_{2}&a_{3}\\\ f_{1}&f_{2}&f_{3}\\\
g_{1}&g_{2}&g_{3}\end{pmatrix}$

and $h^{\prime}_{i}=h_{i}+a_{i}\phi\psi+f_{i}\phi+g_{j}\psi$. The equation
$\det(A^{\prime})v^{\prime}_{1}-v^{\prime}_{2}v^{\prime}_{3}=\det(A)v_{1}-v_{2}v_{3}$
is derived by a straightforward computation, which is tedious but has some
beautiful cancellations. $\Box$

From Proposition 2.1.1 (1) and (2), we have

######  Corollary 2.1.2.

Let $K$ be a field. If $C$ and the divisor $P+Q$ on $C$ for distinct two
points $P$ and $Q$ is defined over $K$, then the associated sextic
$C^{\prime}$ is defined over $K$.

This corollary suggests us to use the sextic realization to find/enumerate
curves over $\mathbb{F}_{q}$ of genus $5$ with many
$\mathbb{F}_{q^{2}}$-rational points, since the assumption that $P+Q$ is
defined over $\mathbb{F}_{q}$ is satisfied if $C$ has many (a few)
$\mathbb{F}_{q^{2}}$-rational points. The case we mainly treat in this paper
is the case of $q=3$. Actually, the maximal number $C(\mathbb{F}_{9})$ for
curves of genus $5$ over $\mathbb{F}_{9}$ is unknown, as mentioned in Section
1. We give an implementation of our algorithm, restricting ourselves the case
that $C$ is defined over $\mathbb{F}_{3}$.

In general, the curves defined by these sextic forms may have nasty
singularities. But in most cases, $C^{\prime}$ has five singular points with
multiplicity $2$. The number of monomials of degree $6$ in three variables is
$28$. For each singular point, we have three linear equations which assure
that the point is singular . Considering a scalar multiplication to the whole
sextic, the number of free parameters is $28-5\times 3-1=12$, see Propositions
2.1.3 and 2.2.2 below for the linear independence of $5\times 3$ linear
equations. This is just the dimension of the moduli space of curves of genus
$5$. Note that both of the dimension of the space of two points on $C$ and
that of five points on ${\mathbb{P}^{2}}$ modulo the action of
$\operatorname{PGL}_{3}$ are equal to two. This says that the parametrization
by the sextic models is very effective.

######  Proposition 2.1.3.

Let $\\{P_{1},\ldots,P_{5}\\}$ be distinct five points of
${\mathbb{P}}^{2}(K)$. Assume that any distinct four points in
$\\{P_{1},\ldots,P_{5}\\}$ are not contained in a line. Then the space of
sextics with double points at $P_{1},\ldots,P_{5}$ up to scalar
multiplications is of dimension $12$.

###### Proof.

By a linear transformation by an element of $\operatorname{PGL}_{3}(K)$, we
may assume that $P_{1}=(1:0:0)$, $P_{2}=(0:1:0)$ and $P_{3}=(0:0:1)$. Let
$K[x,y,z]$ be the coordinate ring of ${\mathbb{P}}^{2}$. Considering the
permutation of $\\{x,y,z\\}$ by the symmetric group of degree $3$, we may
assume that $P_{4}=(b:c:1)$ and $P_{5}=(d:e:1)$, by the assumption of four
points. Then, any sextic with $z=1$ having singularity at $P_{1}$, $P_{2}$ and
$P_{3}$ is of the form

$\displaystyle F$ $\displaystyle=$ $\displaystyle
a_{1}x^{4}y^{2}+a_{2}x^{4}y+a_{3}x^{4}+a_{4}x^{3}y^{3}+a_{5}x^{3}y^{2}+a_{6}x^{3}y+a_{7}x^{3}+a_{8}x^{2}y^{4}+a_{9}x^{2}y^{3}+a_{10}x^{2}y^{2}$
$\displaystyle+a_{11}x^{2}y+a_{12}x^{2}+a_{13}xy^{4}+a_{14}xy^{3}+a_{15}xy^{2}++a_{16}xy+a_{17}y^{4}+a_{18}y^{3}+a_{19}y^{2}.$

The condition that this sextic has singularity at $P_{4}$ (resp. $P_{5}$) is
described by three linear equations in $a_{1},\ldots,a_{19}$ obtained from the
condition that $F$, $\frac{\partial F}{\partial x}$ and $\frac{\partial
F}{\partial x}$ are zero at $P_{4}$ (resp. $P_{5}$). The coefficients of these
six linear equation makes a $6\times 19$ matrix $M$. It suffices to show that
$M$ is of rank $6$. By a direct computation, the determinant of the minor
(square matrix of size $6$) corresponding to the coefficients of
$a_{7},a_{11},a_{12},a_{15},a_{18},a_{19}$ is $(be-cd)^{5}(be+cd)$ and that of
$a_{6},a_{7},a_{9},a_{10},a_{11},a_{12}$ is $b^{6}d^{6}(c-e)^{5}$ and that of
$a_{5},a_{10},a_{14},a_{15},a_{18},a_{19}$ is $c^{6}e^{6}(b-d)^{5}$. If the
rank of $M$ were less than $6$, then $(be-cd)^{5}(be+cd)=0$,
$b^{6}d^{6}(c-e)^{5}=0$ and $c^{6}e^{6}(b-d)^{5}=0$ has to hold. But this
contradicts with the assumption of four points. Indeed if $b=0$ then we have
$c=0$ or $d=0$ by the first equation, and then $P_{1}=P_{4}$ holds or
${P_{1},P_{3},P_{4},P_{5}}$ is contained in the line $y=0$. Thus $b\neq 0$.
Similarly $c,d$ and $e$ are not zero. Then we have $b=d$ and $c=e$, which says
$P_{3}=P_{4}$. This is absurd. Hence $M$ has to be of rank $6$. $\Box$

###  2.2 Non-special curves of genus 5

As mentioned in the last paragraph of the previous subsection, sextics
$C^{\prime}$ may have various kinds of singularities. It would be natural that
we start with the case that the singularity is milder. This paper studies the
case that

$g(C)=g(C^{\prime})-\sum_{P^{\prime}\in
C^{\prime}}\frac{m_{P^{\prime}}(m_{P^{\prime}}-1)}{2}$ (2.2.1)

holds with $g(C^{\prime})=(6-1)(6-2)/2=10$, where $g(D)$ is the arithmetic
genus of a curve $D$ and $m_{P^{\prime}}$ is the multiplicity of $C^{\prime}$
at $P^{\prime}$. Note that in general the left hand side of (2.2.1) is less
than or equal to the right hand side. See Remark 2.2.4 below for a geometric
meaning of (2.2.1). In order to have (2.2.1) in our case (i.e., $g(C)=5$), the
multiple set of $m_{P^{\prime}}$ with $m_{P^{\prime}}\geq 2$ is either of
$\\{2,2,2,2,2\\}$ and $\\{3,2,2\\}$.

######  Definition 2.2.1.

We say that $C$ is non-special if a sextic model $C^{\prime}$ associated to
$(C,P+Q)$ satisfies (2.2.1). More concretely, there are two cases for non-
special curves:

  1. (I)

$C^{\prime}$ has five singular points ${P_{1},\ldots,P_{5}}$ with multiplicity
two.

  2. (II)

$C^{\prime}$ has one triple point $P_{1}$ and two double points $P_{2}$ and
$P_{3}$.

If $(C,P+Q)$ is defined over $K$, then in case (I) the divisor
$P_{1}+\cdots+P_{5}$ is defined over $K$, and in case (II) the triple point
$P_{1}$ is defined over $K$ and the divisor $P_{2}+P_{3}$ is defined over $K$.
For example in case (I) the absolute Galois group $G_{K}$ of $K$ makes
permutations of $\\{P_{1},\ldots,P_{5}\\}$. It is straightforward to see that
the patterns of the $G_{K}$-orbits in $\\{P_{1},\ldots,P_{5}\\}$ is either of
$(1,1,1,1,1)$, $(1,1,1,2)$, $(1,2,2)$, $(1,1,3)$, $(2,3)$, $(1,4)$ and $(5)$,
where for example $(1,2,2)$ means that $\\{P_{1},\ldots,P_{5}\\}$ consists of
three $G_{K}$-orbits each of which has cardinarity $1$, $2$ and $2$
respectively. In Section 4, in the case of $K=\mathbb{F}_{3}$ we shall give an
explicit classification of positions of $\\{P_{1},\ldots,P_{5}\\}$ in
$\mathbb{P}^{2}$ up to $\operatorname{PGL}_{3}(\mathbb{F}_{3})$.

Here is another constraint for the position of the singular points.

######  Proposition 2.2.2.

In case (I), if distinct four elements of
$\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$ are contained in a line, then
$C^{\prime}$ is geometrically reducible (i.e.,
$C_{\overline{K}}:=C\times_{\mathrm{Spec}(K)}\mathrm{Spec}(\overline{K})$ is
reducible, where $\overline{K}$ is the algebraic closure of $K$).

###### Proof.

Suppose that $P_{1},\ldots,P_{4}$ are contained in a line. By a linear
transformation over $\overline{K}$, we may assume

$P_{1}=(0:0:1),\quad P_{2}=(1:0:1),\quad P_{3}=(c:0:1),\quad P_{4}=(d:0:1),$

where $0,1,c$ and $d$ are muturally distinct. Let $f$ be the sextic (in $x$
and $y$) of $C^{\prime}$ obtained by substituting $1$ for $z$. Since $P_{1}$
is a singular points, the smallest degree of the non-zero terms of $f$ is two.
Let $a_{i}$ be the $x^{i}y^{0}$-coefficient of $F$ for $i=2,3,4,5,6$. Then the
constant term and the degree-one term of the Taylor expansion of $f$ at
$(x,y)=(b,0)$ are $\sum_{i=2}^{6}b^{i}a_{i}$ and $\sum_{i=2}^{6}ib^{i-1}a_{i}$
for $b\in\\{1,c,d\\}$. These are zero, as $C$ is singular at $P_{i}$ for
$i=1,\ldots,5$. Since

$\det\begin{pmatrix}1&1&1&1&1\\\ c^{2}&c^{3}&c^{4}&c^{5}&c^{6}\\\
2c&3c^{2}&4c^{3}&5c^{4}&6c^{5}\\\ d^{2}&d^{3}&d^{4}&d^{5}&d^{6}\\\
2d&3d^{2}&4d^{3}&5d^{4}&6d^{5}\end{pmatrix}=c^{4}d^{4}(c-1)^{2}(d-1)^{2}(c-d)^{4},$

we have that $a_{i}$ are zero for all $i=2,\ldots,6$. This says that $f$ is
divided by $y$, in particular $C^{\prime}$ is geometrically reducible. $\Box$

We have a similar result also in case (II):

######  Proposition 2.2.3.

In case (II), if $P_{1}$, $P_{2}$ and $P_{3}$ are contained in a line, then
$C^{\prime}$ is geometrically reducible.

###### Proof.

Suppose that $P_{1}$, $P_{1}$ and $P_{3}$ are contained in a line. By a linear
transformation over $\overline{K}$, we may assume

$P_{1}=(0:0:1),\quad P_{2}=(1:0:1),\quad P_{3}=(c:0:1)$

where $0$, $1$ and $c$ are muturally distinct. Let $f$ be the sextic (in $x$
and $y$) of $C^{\prime}$ obtained by substituting $1$ for $z$. Since $P_{1}$
is a triple point, the smallest degree of the non-zero terms of $f$ is three.
Let $a_{i}$ be the $x^{i}y^{0}$-coefficient of $F$ for $i=3,4,5,6$. Then the
constant term and the degree-one term of the Taylor expansion of $f$ at
$(x,y)=(b,0)$ are $\sum_{i=3}^{6}b^{i}a_{i}$ and $\sum_{i=3}^{6}ib^{i-1}a_{i}$
for $b\in\\{1,c\\}$. These are zero, as $C$ is singular at $P_{i}$ for
$i=1,\ldots,5$. Since

$\det\begin{pmatrix}1&1&1&1\\\ 3&4&5&6\\\ c^{2}&c^{3}&c^{4}&c^{5}\\\
2c&3c^{2}&4c^{3}&5c^{4}\end{pmatrix}=c^{4}(c-1)^{4},$

we have that $a_{i}$ is zero for all $i=3,\ldots,6$. This says that $f$ is
divided by $y$, in particular $C^{\prime}$ is geometrically reducible. $\Box$

The next remark explains what (2.2.1) means.

######  Remark 2.2.4.

We claim that if the desingularization $C\to C^{\prime}$ satisfies (2.2.1),
then, at each singular point $P\in C^{\prime}$, the desingularization of
$C^{\prime}$ at $P$ is obtained by the one-time blow-up centered at $P$.
Indeed, by a linear coordinate change, the structure ring around $P$ is
locally described as $\mathbb{F}_{q}[X,Y]/(G)$ with $G\in\mathbb{F}_{q}[X,Y]$,
where $P$ is the origin. Write $G=G_{m}+G_{m+1}+\cdots$, where $m=m_{P}$ is
the multiplicity at $P$ and $G_{i}$ is the part of degree $i$. By a linear
coordinate change again, we may assume that the $Y^{m}$-coefficient of $G_{m}$
is not zero if we assume $q-1\geq m$ (which is satisfied in our case: $q=9$
and $m\leq 3$). Then, the morphism from the strict transform of the blow-up to
$C^{\prime}$ is locally described by the ring homomorphism

$\mathbb{F}_{q}[X,Y]/(G)\to\mathbb{F}_{q}[X,Z]/(\tilde{G})$

with ${\tilde{G}}(X,Z)=G(X,ZX)X^{-m}$ sending $(X,Y)$ to $(X,XZ)$. Its
cokernel is the $\frac{m(m-1)}{2}$-dimensional $\mathbb{F}_{q}$-vector space
generated by $Z^{i}X^{j}$ for $1\leq i\leq m-1$ and $0\leq j\leq i-1$. Thus,
we have the claim.

Finally, we give a remark about the number of rational points on a non-special
curve $C$ over finite field $\mathbb{F}_{q}$. Each point $\in C$ over $P$ by
this blow-up corresponds to a root of ${\tilde{G}}_{m}(1,Z)=0$, where
${\tilde{G}}_{m}(1,Z)={\tilde{G}}_{m}(X,ZX)X^{-m}$ with the notation of Remark
2.2.4. Hence, we have the following formula

$\\#C(\mathbb{F}_{q})=\\#C^{\prime}(\mathbb{F}_{q})+\sum_{P\in
C^{\prime}(\mathbb{F}_{q})}\left(\\#V(h_{P})(\mathbb{F}_{q})-1\right),$
(2.2.2)

where $h_{P}$ is the $G_{m}$ above for $P$, i.e., the homogeneous part of the
least degree (i.e., $m_{P}$) of the Taylor expansion at $P$ of the defining
polynomial of an affine model containing $P$ of $C^{\prime}$ and $V(h_{P})$ is
the closed subscheme of ${\mathbb{P}}^{1}$ defined by the ideal $(h_{P})$. If
$h_{P}$ is quadratic, then $\\#V(h_{P})(\mathbb{F}_{q})-1$ is equal to $1$ if
the discriminant $\Delta(h_{P})$ of $h_{P}$ is nonzero square and to $-1$ if
$\Delta(h_{P})$ is nonzero non-square and to $0$ if $\Delta(h_{P})=0$. From a
computational point of view, it may have a little advantage to use the fact
that if $q=p^{2}$ and $\Delta(h_{P})$ belongs to $\mathbb{F}_{p}$, then
$\Delta(h_{P})\neq 0$ is equivalent to that $\Delta(h_{P})$ is nonzero square
in $\mathbb{F}_{p^{2}}$.

######  Remark 2.2.5.

The case of non-mild singularities (i.e., $C^{\prime}$ has singularities other
than mild singularities explained above) is one of our future works, see
Section 6. As in the mild case, one could also construct an algorithm to find
a reduced equation defining $C^{\prime}$ even in the non-mild case, once the
type of the singularities is known. However, the number of singular points of
$C^{\prime}$ can take any of $5$, $4$, $3$, $2$, $1$ in general. This means
that there are various kinds of singularities in the non-mild case, and thus
it would require much effort to classify their possible types.

##  3 Concrete Algorithm

Let $K$ be a finite field with characteristic $p$, and $K^{\prime}$ a finite
extension field of $K$. As it was shown in Subsection 2.1, any canonical non-
trigonal curve of genus $5$ over $K$ is realized as the normalization of a
sextic in $\mathbb{P}^{2}$. In particular, such a curve $C$ is said to be non-
special if it satisfies either of the conditions (I) or (II) stated in
Definition 2.2.1 of Subsection 2.2.

In this section, we first present a concrete algorithm (Algorithm 3.1.1 below)
to enumerate all non-special curves $C$ of genus $5$ over $K$ satisfying (I)
and $\\#C(K^{\prime})\geq N$, where $N$ is a given positive integer. Similarly
to (I), we can construct an algorithm for (II), but we omit to write down it
in this paper. After presenting the algorithm for (I), we also give some
remarks for implementation.

###  3.1 Algorithm for the case (I)

Let $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$ be a set of distinct five points in
$\mathbb{P}^{2}(L)$, where $L$ is a finite extension field of $K$. Assume that
$\mathrm{Gal}(L/K)$ stabilizes the set $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$.
Given $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$ and an integer $N\geq 1$,
Algorithm 3.1.1 below enumerates all sextic forms $F$ in $K[x,y,z]$ such that
$C^{\prime}:F=0$ is a singular (irreducible) curve of geometric genus $5$ in
$\mathbb{P}^{2}$ with
$\mathrm{Sing}(C^{\prime})=\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$ and
$\\#C(K^{\prime})\geq N$, where $C$ is the normalization of $C^{\prime}$.

######  Algorithm 3.1.1.

Input. $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$ and $N\geq 1$.

Output. A set $\mathcal{F}$ of sextic forms in $K[x,y,z]$.

  1. (1)

Set $\mathcal{F}:=\emptyset$.

  2. (2)

Construct
$F=\sum_{i=1}^{28}a_{i}x^{\alpha_{1}^{(i)}}y^{\alpha_{2}^{(i)}}z^{\alpha_{3}^{(i)}}\in
K[a_{1},\ldots,a_{28}][x,y,z]$ with

$\\{(\alpha_{1}^{(1)},\alpha_{2}^{(1)},\alpha_{3}^{(1)}),\ldots,(\alpha_{1}^{(28)},\alpha_{2}^{(28)},\alpha_{3}^{(28)})\\}=\\{(\alpha_{1},\alpha_{2},\alpha_{3})\in(\mathbb{Z}_{\geq
0})^{\oplus 3}:\alpha_{1}+\alpha_{2}+\alpha_{3}=6\\},$

where $a_{1},\ldots,a_{28}$ are indeterminates.

  3. (3)

For each $\ell\in\\{1,\ldots,5\\}$:

    1. (a)

Write $P_{\ell}=(p_{1}:p_{2}:p_{3})$ for $p_{1},p_{2},p_{3}\in L$.

    2. (b)

Let $k$ be the minimal element in $\\{1,2,3\\}$ such that $p_{k}\neq 0$, and
put $z_{k}=p_{k}$, $z_{i}=X$ and $z_{j}=Y$ for
$i,j\in\\{1,2,3\\}\smallsetminus\\{k\\}$ with $i<j$.

    3. (c)

Compute $F_{\ell}:=F(z_{1},z_{2},z_{3})\in L[a_{1},\ldots,a_{28}][X,Y]$. Let
$f_{3\ell-2}$ and $f_{3\ell-1}$ be the coefficients of $X$ and $Y$ in
$F_{\ell}$ respectively, and $f_{3\ell}$ the constant term of $F_{\ell}$ as a
polynomial in $X,Y$.

  4. (4)

Compute a basis $\\{\mathbf{b}_{1},\ldots,\mathbf{b}_{d}\\}\subset K^{\oplus
28}$ of the null-space of the linear system over $L$ defined by
$f_{t}(a_{1},\ldots,a_{28})=0$ for $1\leq t\leq 15$, where $d$ is the
dimension of the null-space. By Proposition 2.1.3 we have $d=13$.

  5. (5)

For each $(v_{1},\ldots,v_{d})\in K^{\oplus
d}\smallsetminus\\{(0,\ldots,0)\\}$ with $v_{1}\in\\{0,1\\}$:

    1. (a)

Compute $\mathbf{c}:=\sum_{j=1}^{d}v_{j}\mathbf{b}_{j}\in K^{\oplus
28}\smallsetminus\\{(0,\ldots,0)\\}$. For each $1\leq i\leq 28$, we denote by
$c_{i}$ the $i$-th entry of $\mathbf{c}$.

    2. (b)

If
$\sum_{i=1}^{28}c_{i}x^{\alpha_{1}^{(i)}}y^{\alpha_{2}^{(i)}}z^{\alpha_{3}^{(i)}}\in
K[x,y,z]$ is irreducible, and if
$\sum_{i=1}^{28}c_{i}x^{\alpha_{1}^{(i)}}y^{\alpha_{2}^{(i)}}z^{\alpha_{3}^{(i)}}=0$
in $\mathbb{P}^{2}$ has geometric genus $5$:

      1. (i)

Set
$F_{\mathbf{c}}:=\sum_{i=1}^{28}c_{i}x^{\alpha_{1}^{(i)}}y^{\alpha_{2}^{(i)}}z^{\alpha_{3}^{(i)}}$,
and let $C^{\prime}$ be the plane curve in $\mathbb{P}^{2}$ defined by
$F_{\mathbf{c}}=0$.

      2. (ii)

Compute $\\#C(K^{\prime})$ by the formula (2.2.2) given in Subsection 2.2,
where $C$ is the normalization of $C^{\prime}$.

      3. (iii)

If $\\#C(K^{\prime})\geq N$, replace $\mathcal{F}$ by
$\mathcal{F}\cup\\{F_{\mathbf{c}}\\}$.

  6. (6)

Output $\mathcal{F}$.

######  Remark 3.1.2.

  1. (1)

We can take a basis $\\{\mathbf{b}_{1},\ldots,\mathbf{b}_{d}\\}$ in Step (4)
so that $\mathbf{b}_{i}\in K^{\oplus 28}$ for all $1\leq i\leq d$, by the
following general fact: Let $E/K$ be a separable extension of fields, and $L$
the Galois closure of $E$ over $K$. If a linear system over $E$ is
$\mathrm{Gal}(L/K)$-stable, then each entry of the Echelon form of the
coefficient matrix of the system belongs to $K$.

  2. (2)

In Step (5)(b)(ii), we need to compute the discriminant $\Delta_{\ell}$ of the
degree-$2$ part of the Taylor expansion of $F_{\mathbf{c}}$ at each
$P_{\ell}$. For this, we can use $F_{\ell}$ computed in Step (3) as follows:
Let $D_{\ell}\in L[a_{1},\ldots,a_{28}]$ denote the discriminant of the
degree-$2$ part of $F_{\ell}$ as a polynomial in $X$ and $Y$. Then clearly we
have $\Delta_{\ell}=D_{\ell}(\mathbf{c})$.

###  3.2 Correctness of our algorithm and remarks for implementation

The correctness of Algorithm 3.1.1 follows mainly from the definition of the
multiplicity of a projective plane curve at a singular point: Indeed, each
polynomial $F_{\ell}$ computed in Step (3) is equal to the Taylor expansion of
$F$ at $P_{\ell}$. Since any vector $\mathbf{c}$ defining $F_{\mathbf{c}}$ in
Step (5)(b)(i) is a root of the system $f_{3\ell-2}=f_{3\ell-2}=f_{3\ell}=0$
for all $1\leq\ell\leq 5$, we have that $P_{\ell}$ is a singular point of
$C^{\prime}:F_{\mathbf{c}}=0$ with multiplicity $2$ for each $1\leq\ell\leq
5$. Conversely, we suppose that $\mathbf{c}$ is a vector in $K^{\oplus
28}\smallsetminus\\{(0,\ldots,0)\\}$ such that $F_{\mathbf{c}}=0$ is an
irreducible plane curve of geometric genus $5$ with multiplicity $2$ at the
points $P_{1},\ldots,P_{5}$. By the definition of the multiplicity of a
singularity, the vector $\mathbf{c}$ is a root of the system
$f_{3\ell-2}=f_{3\ell-1}=f_{3\ell}=0$ for all $1\leq\ell\leq 5$. As it was
described in Remark 3.1.2 (1), each entry of the Echelon form of the
coefficient matrix of the system belongs to $K$, and thus $\mathbf{c}$ is a
root of a linear system over $K$ whose null-space over $L$ is the same as that
of $f_{\ell}=f_{\ell+1}=f_{\ell+2}=0$ for all $1\leq\ell\leq 5$. Hence
$\mathbf{c}$ is written as $\mathbf{c}=\sum_{j=1}^{d}v_{j}\mathbf{b}_{j}$ for
some $(v_{1},\ldots,v_{d})\in K^{\oplus d}\smallsetminus\\{(0,\ldots,0)\\}$.
Note that it suffices to compute the part with $v_{1}\in\\{0,1\\}$, because
any scalar multiplication does not change the isomorphism class of the sextic.

We implemented Algorithm 3.1.1 and its variant for the case (II) over MAGMA
V2.25-8 [2], [3] in its 64-bit version. Details of our computational
environment will be described in Section 5. For Step (5)(b), our
implementation utilizes MAGMA’s built-in functions IsIrreducible,
GeometricGenus and Variety. In particular, we use Variety to compute
$\\#C^{\prime}(K^{\prime})=\\#V(F_{\mathbf{c}})$.

##  4 Position analysis for $\mathbb{F}_{3}$

In this section, we classify positions of five (resp. three) points in
${\mathbb{P}}^{2}$ which can be singular points of a non-special curve
$C^{\prime}$ over $\mathbb{F}_{3}$ in case (I) (resp. in case (II)), up to the
action by $\operatorname{PGL}_{3}(\mathbb{F}_{3})$. In each case, this is the
problem to enumerate the orbits of an action of a finite group on a finite
set. As this is done by a computer calculation (with MAGMA), we state only the
results.

###  4.1 Case (I): $C^{\prime}$ has five double points

The Frobenius map $\sigma$ over $\mathbb{F}_{3}$ (the map taking the third
power of each entry) makes a permutation of $\\{P_{1},\ldots,P_{5}\\}$, i.e.,

$\\{P_{1},\ldots,P_{5}\\}=\\{\sigma(P_{1}),\sigma(P_{2}),\sigma(P_{3}),\sigma(P_{4}),\sigma(P_{5})\\}.$

The patterns of the Frobenius orbits in $\\{P_{1},\ldots,P_{5}\\}$ is either
of $(1,1,1,1,1)$, $(1,1,1,2)$, $(1,2,2)$, $(1,1,3)$, $(2,3)$, $(1,4)$ and
$(5)$, where for example $(1,2,2)$ means that $\\{P_{1},\ldots,P_{5}\\}$
consists of three Frobenius orbits each of which has cardinarity $1$, $2$ and
$2$ respectively.

Case (1,1,1,1,1): This is the case where every singular point is
$\mathbb{F}_{3}$-rational. By a linear transformation of an element of
$\operatorname{PGL}_{3}(\mathbb{F}_{p})$, we may assume

$P_{1}=(1:0:0),\quad P_{2}=(0:1:0),\quad P_{3}=(0:0:1).$

A computation shows that every position of $\\{P_{1},\ldots,P_{5}\\}$ such
that any four points among them are not contained in a hyperplane is
equivalent by a linear transformation of a diagonal matrix and a permutation
of $\\{x,y,z\\}$ to either of the following two cases:

  1. (1)

$P_{4}=(1:1:0)$ and $P_{5}=(0:1:1)$,

  2. (2)

$P_{4}=(1:1:0)$ and $P_{5}=(1:2:1)$.

######  Example 4.1.1.

In characteristic $3$, the defining equation of any sextic having singular
points $P_{1}=(0:0:1)$, $P_{2}=(0:1:0)$, $P_{3}=(1:0:0)$ $P_{4}=(1:1:0)$,
$P_{5}=(0:1:1)$ is

$\displaystyle
a_{1}x^{4}y^{2}+a_{2}x^{4}yz+a_{3}x^{4}z^{2}+a_{4}x^{3}y^{3}+a_{5}x^{3}y^{2}z+a_{6}x^{3}yz^{2}+a_{7}x^{3}z^{3}$
$\displaystyle+a_{8}x^{2}y^{4}+a_{9}x^{2}y^{3}z+a_{10}x^{2}y^{2}z^{2}+a_{11}x^{2}yz^{3}+a_{12}x^{2}z^{4}+a_{13}xy^{4}z$
$\displaystyle+a_{14}xy^{3}z^{2}+a_{15}xy^{2}z^{3}+a_{16}xyz^{4}+a_{17}y^{4}z^{2}+a_{18}y^{3}z^{3}+a_{19}y^{2}z^{4}$

with

$\begin{cases}a_{1}+a_{4}+a_{8}=0,\\\ 2a_{1}+a_{8}=0,\\\
a_{2}+a_{5}+a_{9}+a_{13}=0,\end{cases}\quad\begin{cases}a_{17}+a_{18}+a_{19}=0,\\\
a_{13}+a_{14}+a_{15}+a_{16}=0,\\\ a_{17}+2a_{19}=0.\end{cases}$

The quadratic form associated to the singular point $P_{i}$ (of multiplicity
$2$) is as follows:

$\displaystyle Q_{1}$ $\displaystyle:=$ $\displaystyle
a_{12}x^{2}+a_{16}xy+a_{19}y^{2},$ $\displaystyle Q_{2}$ $\displaystyle:=$
$\displaystyle a_{8}x^{2}+a_{13}xz+a_{17}z^{2},$ $\displaystyle Q_{3}$
$\displaystyle:=$ $\displaystyle a_{1}y^{2}+a_{2}yz+a_{3}z^{2},$
$\displaystyle Q_{4}$ $\displaystyle:=$ $\displaystyle
a_{1}(y-x)^{2}+(a_{2}+2a_{5}+a_{13})(y-x)z+(a_{3}+a_{6}+a_{10}+a_{14}+a_{17})z^{2},$
$\displaystyle Q_{5}$ $\displaystyle:=$
$\displaystyle(a_{8}+a_{9}+a_{10}+a_{11}+a_{12})x^{2}+(a_{13}+2a_{14}+a_{16})x(y-z)+a_{17}(y-z)^{2}$

respectively.

Case (1,1,1,2) with linear independent $P_{1}$, $P_{2}$, $P_{3}$: We consider
the case where $\\{P_{1},\ldots,P_{5}\\}$ contains three
$\mathbb{F}_{p}$-rational points, say $P_{1},P_{2},P_{3}$, where $P_{1}$,
$P_{2}$, $P_{3}$ are linearly independent, and the other points are defined
over $\mathbb{F}_{p^{2}}$ (not over $\mathbb{F}_{p}$) and are conjugate to
each other, i.e., $(P_{4},P_{5})=(\sigma(P_{5}),\sigma(P_{4}))$. By a
transformation by an element of $\operatorname{PGL}_{3}(\mathbb{F}_{p})$, we
may assume

$P_{1}=(1:0:0),\quad P_{2}=(0:1:0),\quad P_{3}=(0:0:1).$

Let $\zeta$ be a primitive element of $\mathbb{F}_{p^{2}}$. A computation
shows that every position of $\\{P_{1},\ldots,P_{5}\\}$ such that any four
points among them are not contained in a hyperplane is equivalent by a linear
transformation of a diagonal matrix and a permutation of $\\{x,y,z\\}$ to
either of the three cases:

  1. (1)

$P_{4}=(1:\zeta^{5}:\zeta^{7})$,

  2. (2)

$P_{4}=(1:\zeta^{7}:1)$,

  3. (3)

$P_{4}=(1:\zeta^{2}:\zeta^{2})$.

with $P_{5}=\sigma(P_{4})$.

Case (1,1,1,2) with linear dependent $P_{1}$, $P_{2}$, $P_{3}$: We consider
the case where $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$ contains three
$\mathbb{F}_{p}$-rational points, say $P_{1},P_{2},P_{3}$, where $P_{1}$,
$P_{2}$, $P_{3}$ are linearly dependent, and the other points are defined over
$\mathbb{F}_{p^{2}}$ and are conjugate to each other.

By a transformation by an element of $\operatorname{PGL}_{3}(\mathbb{F}_{p})$,
we may assume

$P_{1}=(1:0:0),\quad P_{2}=(0:1:0),\quad P_{3}=(1:1:0).$

Let $\zeta$ be a primitive element of $\mathbb{F}_{p^{2}}$. We have three
equivalent classes:

  1. (1)

$P_{4}=(1:\zeta^{5}:\zeta^{7})$,

  2. (2)

$P_{4}=(1:1:\zeta^{7})$,

  3. (3)

$P_{4}=(1:\zeta^{2};\zeta^{2})$

with $P_{5}=\sigma(P_{4})$.

Case (1,2,2): We consider the case where $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$
contains one $\mathbb{F}_{p}$-rational point $P_{1}$ and two pairs
$(P_{2},P_{3})$ and $(P_{4},P_{5})$ of $\mathbb{F}_{p^{2}}$-rational points,
where $P_{3}=\sigma(P_{2})$ and $P_{5}=\sigma(P_{4})$. By a tranformation by
an element of $\operatorname{PGL}_{3}(\mathbb{F}_{p})$, we may assume

$P_{1}=(1:0:0).$

Let $\zeta$ be a primitive element of $\mathbb{F}_{p^{2}}$. We have five
equivalent classes:

  1. (1)

$(P_{2},P_{4})=((1:1:\zeta^{2}),(0:1:\zeta^{6}))$,

  2. (2)

$(P_{2},P_{4})=((1:2:\zeta^{5}),(1:\zeta^{2}:\zeta^{7}))$,

  3. (3)

$(P_{2},P_{4})=((1:\zeta:1),(1:\zeta^{7}:\zeta^{7}))$,

  4. (4)

$(P_{2},P_{4})=((1:\zeta^{2}:\zeta^{6}),(1:0:\zeta^{5}))$,

  5. (5)

$(P_{2},P_{4})=((1:0:\zeta^{2}),(1:1:\zeta^{5}))$.

Case (1,1,3): We consider the case where $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$
contains two $\mathbb{F}_{p}$-rational points $P_{1},P_{2}$ and conjugate
three points $(P_{3},P_{4},P_{5})$ of $\mathbb{F}_{p^{3}}$-rational points,
where

$(P_{3},P_{4},P_{5})=(P_{3},\sigma(P_{3}),\sigma^{2}(P_{3})).$

By a tranformation by an element of $\operatorname{PGL}_{3}(\mathbb{F}_{p})$,
we may assume

$P_{1}=(1:0:0),\quad P_{2}=(0:1:0).$

Let $\zeta$ be a primitive element of $\mathbb{F}_{p^{3}}$. We have four cases

  1. (1)

$P_{3}=(1:2:\zeta^{5})$,

  2. (2)

$P_{3}=(1:\zeta^{6}:\zeta^{25})$,

  3. (3)

$P_{3}=(1:\zeta^{17}:\zeta^{2})$,

  4. (4)

$P_{3}=(1:2:\zeta^{10})$.

Case (2,3): We consider the case where $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$
contains conjugate two points $(P_{1},P_{2})$ of $\mathbb{F}_{p^{2}}$-rational
points with $(P_{1},P_{2})=(P_{1},\sigma(P_{2}))$ and conjugate three points
$(P_{3},P_{4},P_{5})$ of $\mathbb{F}_{p^{3}}$-rational points, where
$(P_{3},P_{4},P_{5})=(P_{3},\sigma(P_{3}),\sigma^{2}(P_{3}))$. By a
tranformation by an element of $\operatorname{PGL}_{3}(\mathbb{F}_{p})$, we
may assume

$P_{1}=(1:\xi:0)$

where $\xi$ is a primitive element of $\mathbb{F}_{p^{2}}$. Let $\zeta$ be a
primitive element of $\mathbb{F}_{p^{3}}$. We have three cases:

  1. (1)

$P_{3}=(1:2:\zeta^{5})$,

  2. (2)

$P_{3}=(1:\zeta^{22}:2)$,

  3. (3)

$P_{3}=(1:\zeta^{17}:\zeta^{2})$.

Case (1,4): We consider the case where $\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}$
contains one $\mathbb{F}_{3}$-rational point $P_{1}$ and the other 4 points
are over $\mathbb{F}_{3^{4}}$ and are conjugate to each other, say

$(P_{2},P_{3},P_{4},P_{5})=(P_{2},\sigma(P_{2}),\sigma^{2}(P_{2}),\sigma^{3}(P_{2})).$

By a tranformation by an element of $\operatorname{PGL}_{3}(\mathbb{F}_{3})$,
we may assume $P_{1}=(1:0:0)$. Let $\zeta$ be a primitive element of
$\mathbb{F}_{3^{4}}$. We have five equivalent classes:

  1. (1)

$P_{2}=(1:\zeta^{75}:\zeta^{49})$,

  2. (2)

$P_{2}=(1:\zeta^{8}:\zeta^{70})$,

  3. (3)

$P_{2}=(1:\zeta^{59}:\zeta^{53})$,

  4. (4)

$P_{2}=(1:\zeta^{72}:\zeta^{29})$,

  5. (5)

$P_{2}=(1:\zeta^{5}:\zeta^{75})$.

Case (5): This is the case where singular points on $C^{\prime}$ are defined
over $\mathbb{F}_{3^{5}}$ (but not over $\mathbb{F}_{3}$). Then
$\\{P_{1},\ldots,P_{5}\\}$ consists of a single Frobenius orbit, namely

$\\{P_{1},P_{2},P_{3},P_{4},P_{5}\\}=\\{P_{1},\sigma(P_{1}),\sigma^{2}(P_{1}),\sigma^{3}(P_{1}),\sigma^{4}(P_{1})\\}.$

In this case, the five points are determined only by $P_{1}$. A computation
says that there are two cases:

  1. (1)

$P_{1}=(1:\zeta^{127}:\zeta^{143})$,

  2. (2)

$P_{1}=(1:\zeta^{218}:\zeta^{72})$

for a primitive element $\zeta$ of $\mathbb{F}_{3^{5}}$.

###  4.2 Case (II): $C^{\prime}$ has one triple point with two double points

We consider sextics $C^{\prime}$ with singular points consisting of one triple
point and two double points. The single triple point should be defined over
$\mathbb{F}_{3}$. Let $P_{1}$ be the triple point. By a tranformation by an
element of $\operatorname{PGL}_{3}(\mathbb{F}_{3})$, we may assume

$P_{1}=(0:0:1).$

The patterns of the Frobenius orbits of the two remaining double points
$\\{P_{2},P_{3}\\}$ are $(1,1)$ and $(2)$.

Case (1,1): This is the case where $C^{\prime}$ has one triple point with two
$\mathbb{F}_{3}$-rational double points. Up to actions by elements of
$\operatorname{PGL}_{3}(\mathbb{F}_{3})$ stabilizing $P_{1}$, we have a unique
case:

$P_{2}=(1:0:0),\qquad P_{3}=(0:1:0).$

Up to actions by elements of $\operatorname{PGL}_{3}(\mathbb{F}_{3})$
stabilizing $P_{1}$ and $\\{P_{2},P_{3}\\}$, we may assume that $F$ is of the
form

$F=b_{1}x^{3}z^{3}+b_{2}x^{2}yz^{3}+b_{3}x^{3}y^{3}+\text{other terms}$

with $b_{1},b_{2},b_{3}\in\\{0,1\\}$ and $(b_{1},b_{2})\neq(0,0)$.

Case (2): This is the case where $C^{\prime}$ has one triple point with two
$\mathbb{F}_{9}$-rational double points. Up to actions by elements of
$\operatorname{PGL}_{3}(\mathbb{F}_{3})$ stabilizing $P_{1}$, we have a unique
case:

$P_{2}=(1:\zeta:0),\qquad P_{3}=(1:\zeta^{3}:0),$

where $\zeta$ is the primitive element of $\mathbb{F}_{9}$ satisfing
$\zeta^{2}-\zeta-1=0$. Up to actions by elements of
$\operatorname{PGL}_{3}(\mathbb{F}_{3})$ stabilizing $P_{1}$ and
$\\{P_{2},P_{3}\\}$, we may assume that $F$ is of the form

$F=b_{1}x^{3}z^{3}+b_{2}x^{2}yz^{3}+b_{3}x^{3}y^{3}+\text{other terms}$

with $b_{1}=1$ and $b_{2},b_{3}\in\\{0,1\\}$.

##  5 Computational results with explicit equations in characteristic $3$

For $K=\mathbb{F}_{p}$ and $K^{\prime}=\mathbb{F}_{p^{2}}$ with $p=3$, we
executed Algorithm 3.1.1 for (I) and its variant for (II) over MAGMA V2.25-8
in each case given in Section 4, in order to prove Theorem 1. We choose $N=32$
as the input, since $32$ is the maximal number among the known numbers of
$\mathbb{F}_{9}$-rational points of genus-five curves over $\mathbb{F}_{9}$.
Our implementations over MAGMA were run on a PC with ubuntu 18.04.5 LTS OS at
3.50GHz quadcore CPU (Intel(R) Xeon(R) E-2224G) and 64GB RAM. It took about 58
hours in total to execute the algorithms for obtaining Theorem 1. For curves
$C$ with $\\#C(\mathbb{F}_{p^{2}})\geq 32$ listed up, we also computed their
Weil polynomials over MAGMA. We have also obtained explicit equations of non-
special curves of genus five over $\mathbb{F}_{3}$ with $32$
$\mathbb{F}_{9}$-rational points. Here, we show some examples:

  1. (1)

Case $(1,1,1,1,1)$ with $P_{5}=(1:2:1)$. The sextic

$F=x^{4}y^{2}+x^{3}y^{3}+x^{2}y^{4}+x^{4}yz+x^{3}y^{2}z+2x^{2}y^{3}z+2xy^{4}z+2x^{2}yz^{3}+xy^{2}z^{3}+x^{2}z^{4}+2xyz^{4}+y^{2}z^{4}$

has 32 $\mathbb{F}_{9}$-rational points with Weil polynomial
$(t+3)^{6}(t^{2}+2t+9)^{2}$.

  2. (2)

Case $(1,1,1,2)$ with linearly independent $P_{1},P_{2},P_{3}$ where
$P_{4}=(1:\zeta^{2}:\zeta^{2})$. The sextic

$F=x^{2}y^{4}+x^{4}yz+2y^{4}z^{2}+x^{2}z^{4}+2y^{2}z^{4}$

has 32 $\mathbb{F}_{9}$-rational points with Weil polynomial
$(t^{2}+2t+9)(t^{2}+5t+9)^{4}$.

  3. (3)

Case $(1,1,1,2)$ with linearly dependent $P_{1},P_{2},P_{3}$ where
$P_{4}=(1:\zeta^{5}:\zeta^{7})$. The sextic

$\displaystyle F$ $\displaystyle=$ $\displaystyle
x^{4}y^{2}+x^{3}y^{3}+x^{2}y^{4}+2x^{3}y^{2}z+xy^{4}z+x^{2}y^{2}z^{2}+2xy^{3}z^{2}+2x^{3}z^{3}$
$\displaystyle+2y^{3}z^{3}+x^{2}z^{4}+2xyz^{4}+2y^{2}z^{4}+z^{6}$

has 32 $\mathbb{F}_{9}$-rational points with Weil polynomial
$(t+3)^{4}(t^{2}+2t+9)(t^{2}+4t+9)^{2}$.

  4. (4)

Case $(1,2,2)$ with $P_{2}=(1:2:\zeta^{5})$ and
$P_{4}=(1:\zeta^{2}:\zeta^{7})$. The sextic

$\displaystyle F$ $\displaystyle=$ $\displaystyle
x^{4}y^{2}+2x^{3}y^{3}+2xy^{5}+2y^{6}+x^{2}y^{3}z+2y^{5}z+2x^{4}z^{2}+x^{3}yz^{2}+xy^{3}z^{2}$
$\displaystyle+2x^{3}z^{3}+x^{2}yz^{3}+xyz^{4}+y^{2}z^{4}+2xz^{5}+2yz^{5}+z^{6}$

has 32 $\mathbb{F}_{9}$-rational points with Weil polynomial
$(t+3)^{2}(t^{4}+8t^{3}+32t^{2}+72t+81)^{2}$.

  5. (5)

Case $(1,1,3)$ with $P_{3}=(1:2:\zeta^{5})$. The sextic

$\displaystyle F$ $\displaystyle=$ $\displaystyle
x^{4}y^{2}+x^{2}y^{4}+x^{4}yz+2xy^{4}z+2x^{4}z^{2}+2x^{3}yz^{2}+x^{2}y^{2}z^{2}+2xy^{3}z^{2}+2y^{4}z^{2}+2x^{3}z^{3}$
$\displaystyle+2x^{2}yz^{3}+xy^{2}z^{3}+y^{3}z^{3}+2x^{2}z^{4}+xyz^{4}+2y^{2}z^{4}+xz^{5}+2yz^{5}+2z^{6}$

has 32 $\mathbb{F}_{9}$-rational points with Weil polynomial
$(t^{2}+2t+9)(t^{2}+5t+9)^{4}$.

  6. (6)

Case $(2,3)$ with $P_{1}=(1:\xi:0)$ and $P_{3}=(1:2:\zeta^{5})$. The sextic

$\displaystyle F$ $\displaystyle=$ $\displaystyle
x^{5}y+x^{4}y^{2}+2x^{2}y^{4}+2y^{6}+x^{5}z+x^{4}yz+2x^{3}y^{2}z+2x^{3}yz^{2}+x^{2}y^{2}z^{2}+x^{3}z^{3}$
$\displaystyle+2x^{2}yz^{3}+xy^{2}z^{3}+2x^{2}z^{4}+xyz^{4}+2y^{2}z^{4}+xz^{5}+z^{6}$

has 32 $\mathbb{F}_{9}$-rational points with Weil polynomial
$(t^{2}+2t+9)(t^{2}+5t+9)^{4}$.

##  6 Concluding remarks with some open problems

We provided a new effective parameterization of the space of canonical and
non-trigonal curves of genus $5$. We realized such a curve as the
normalization of a sextic in $\mathbb{P}^{2}$. It was also proved that if the
associated sextic model has five double pounts, the dimension of the space of
such curves is $12$, which is just(!) the dimension of the moduli space of
curves of genus $5$. As an application, we also presented an algorithm to
enumerate generic curves of genus $5$ over finite fields with many rational
points. By executing the algorithm over MAGMA, we showed that the maximal
number of $\mathbb{F}_{9}$-rational points of generic curves of genus $5$ over
$\mathbb{F}_{3}$ is $32$.

Our explicit parametrization and the algorithm presented in this paper may
derive fruitful applications both for theory and computation, such as
classifying possible invariants (Hasse-Witt rank and so on) of canonical and
non-trigonal curves of genus $5$. Finally, we list some considerable open
problems:

  1. (a)

Extend the parameterization to the case where curves have more complex
singularities. Concretely, present a parameterization of non-hyperelliptic and
non-trigonal curves $C$ of genus $5$ such that the equality (2.2.1) in
Subsection 2.2 does not hold (cf. Remark 2.2.5). A more general problem is to
give an explicit model in $\mathbb{P}^{2}$ of non-hyperelliptic curves of the
other genus $\geq 4$. Cf. a parameterization of generic curves of genus $3$ is
presented in [13], where an equation with $7$ parameters (the moduli dimension
is also $6$) is given.

  2. (b)

Present methods to compute invariants of generic curves of genus $5$ such as
Cartier-Manin and Hasse-Witt matrices. If we can compute Cartier-Manin and
Hasse-Witt matrices, we can determine whether given curves are superspecial or
not, as in [7], [8], [9] and [10]. For computing Cartier-Manin matrices, we
may apply a method in [15] for a plane curve.

  3. (c)

Construct an algorithm to test whether given two generic curves of genus $5$
are isomorphic or not. Computing the automorphism group of such a curve is
also an interesting problem. Cf. for the case of genus-$4$ non-hyperelliptic
curves, the authors (resp. the authors and Senda) already presented an
algorithm for the isomorphism test (resp. for computing automorphism groups),
see [10] (resp. [11]).

  4. (d)

Improve the efficiency of Algorithm 3.1.1 and its variant for the case (II),
and then enumerate generic curves of genus $5$ with many rational points for
$p>3$. For this, it important to reduce the number of curves $C$ which we need
to search.

## References

  * [1] Arbarello, E., Cornalba, M., Griffiths, P. A. and Harris, J.: Geometry of algebraic curves. Vol. I. Grundlehren der Mathematischen Wissenschaften, 267. Springer-Verlag, New York, 1985.
  * [2] Bosma, W., Cannon, J. and Playoust, C.: The Magma algebra system. I. The user language, Journal of Symbolic Computation 24, 235–265 (1997)
  * [3] Cannon, J., et al.: Magma A Computer Algebra System, School of Mathematics and Statistics, University of Sydney, 2016. http://magma.maths.usyd.edu.au/magma/
  * [4] van der Geer, G. and van der Vlugt, M.: Tables of curves with many points, Math. Comp. 69 (2000), no. 230, 797–810.
  * [5] van der Geer, et al.: Tables of Curves with Many Points, 2009, http://www.manypoints.org, Retrieved at 13th February, 2021.
  * [6] Hartshorne, R.: Algebraic Geometry, GTM 52, Springer-Verlag (1977)
  * [7] Kudo, M. and Harashita, S.: Superspecial curves of genus $4$ in small characteristic, Finite Fields and Their Applications 45, 131–169 (2017)
  * [8] Kudo, M. and Harashita, S.: Superspecial Hyperelliptic Curves of Genus $4$ over Small Finite Fields, In: L. Budaghyan, F. Rodriguez-Henriquez (eds), Arithmetic of Finite Fields, WAIFI 2018, Lecture Notes in Computer Science, Vol. 11321, pp. 58–73, Springer, Cham, 2018, (DOI) 10.1007/978-3-030-05153-2_3.
  * [9] Kudo, M. and Harashita, S.: Superspecial trigonal curves of genus $5$. Experimental Mathematics, Published online: 16 Apr 2020. (DOI) 10.1080/10586458.2020.1723745 (preprint ver: arXiv:1804.11277 [math.AG]).
  * [10] Kudo, M. and Harashita, S.: Computational approach to enumerate non-hyperelliptic superspecial curves of genus ${\rm 4}$, Tokyo Journal of Mathematics, Vol. 43, Number 1, pp. 259–278, 2020, doi:10.3836/tjm/1502179310.
  * [11] Kudo, M., Harashita, S. and Senda, S.: Automorphism groups of superspecial curves of genus ${\rm 4}$ over $\mathbb{F}_{11}$, Journal of Pure and Applied Algebra, Vol. 224, Issue 9, 2020, (DOI) 10.1016/j.jpaa.2020.106339.
  * [12] Lauter, K. E.: Geometric methods for improving the upper bounds on the number of rational points on algebraic curves over finite fields, Journal of Algebraic Geometry 10 (2001), no. 1, 19–36.
  * [13] Lercier, R., Ritzenthaler, C., Rovetta, F. and Sijsling, J.: Parametrizing the moduli space of curves and applications to smooth plane quartics over finite fields. LMS Journal of Computation and Mathematics,17(A), 128–147, 2014. doi:10.1112/S146115701400031X
  * [14] Andrade Ramos, C. and Garzón Ramos, A.: Polynomials with a restricted range and curves with many points. Rev. Acad. Colomb. Cienc. 34 (131): 229–239, 2010.
  * [15] Stöhr, K.-O. and Voloch, J. F.: A formula for the Cartier operator on plane algebraic curves, J. Reine Angew. Math. 377 (1987), 49–64.

000 E-mail address of the first author<EMAIL_ADDRESS>
E-mail address of the second author<EMAIL_ADDRESS>


Point set registration optimizes the relative transformation
$\\{\boldsymbol{{T}}\\}$ between two point sets, by minimizing the error
between a transformed point set and the reference point set. The cost or error
function measures that error or the distance to be minimized. However,
building that function is a very critical task. Some of the algorithms use the
Euclidean distance to build the error function, such as iterative closest
point algorithm (ICP). On the other hand, many researchers use the
probabilities to build the error function, such as a Gaussian mixture model
(GMM) or normal distribution transformation (NDT) representation. Therefore,
Mahalanobis distance or L2 metric measure the error. This chapter presents a
one-dimensional synthetic scenario for the demonstration.

##  1 Synthetic Scenario -1D

[ The previous point set$\\{\mathcal{F}\\}$ ]  [The current point set
$\\{\mathcal{M}\\}$]

Figure 1: One dimensional synthetic scenario includes two-point sets
$\\{\mathcal{F},\mathcal{M}\\}$, where each point set has three targets, each
target has a distinct name and particular color. (a) represents the previous
point set$\\{\mathcal{F}\\}$. (b) represents the current point set
$\\{\mathcal{M}\\}$

The scenario incorporates two-point set
$\\{\boldsymbol{\mathcal{F}},\boldsymbol{\mathcal{M}}\\}$, where the second
point set $\\{\boldsymbol{\mathcal{M}}\\}$ is relatively translated from the
first point set $\\{\boldsymbol{\mathcal{F}}\\}$;
$\boldsymbol{\mathcal{M}}=\boldsymbol{T}(\boldsymbol{\mathcal{F}},\boldsymbol{\theta})$.
Where, $\boldsymbol{\theta}$ represents the motion parameters, and for this
scenario $\boldsymbol{\theta}=t_{x}=2$ , and for 2-D scenario with 3DOF
$\boldsymbol{\theta}=[t_{x},t_{y},\phi_{z}]$, where $t_{x}$ is the translation
along the x-axis, $t_{y}$ is the translation along the y-axis, and $\phi_{z}$
represents the rotation around the z-axis. Each point set includes three
targets $\\{\boldsymbol{A,B,C}\\}$, where each target is represented by a
different color, as shown in Fig 1. Every target is represented by a cirle and
star, where the circle is the ground truth position for the target and the
star represents the position after adding noise, all points have 0.15 standard
deviation.

Each target holds two pieces of information: the target position and the
standard deviation. Thus, each target can be represented as Gaussian
distribution
$\mathcal{N}\left(\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k}\right)$ and the
probability density function for target $k$ written as111 Although this
chapter uses a 1-D example for the explanation, the mathematical equation is
in a general form, which means for a higher dimensions.:

$\displaystyle
p(\boldsymbol{x}|\boldsymbol{\mu}_{k},\boldsymbol{\Sigma}_{k})=\frac{\exp[-\frac{1}{2}(\boldsymbol{x}-\boldsymbol{\mu}_{k})^{\mathrm{T}}\boldsymbol{\Sigma}^{-1}_{k}(\boldsymbol{x}-\boldsymbol{\mu}_{k})]}{\sqrt{(2\pi)^{\delta}|\boldsymbol{\Sigma}_{k}|}}$
(1)

However, the point set is considered as a mixture of Gaussians (all targets).
Consequently, the probability density function for each target can be written
as :

$p_{\mathrm{GMM}}(\boldsymbol{x}|\lambda)=\sum_{i=1}^{|{\cal F}|}w_{i}\cdot
p\left(\boldsymbol{x}|\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{i}\right)$ (2)

Where,
$\lambda=\left\\{w_{i},\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{i}\right\\}$
describes the GMM parameters and $w_{i}$ refers to mixture weights for each
normal distribution (component or target). ${|{\cal F}|}$ represents the
number of Gaussian components in the GMM.

##  2 Distance Metrics

This section explains two possible metrics to construct the error function:
The point-to-distribution and distribution-to-distribution. Moreover,
comparing both metrics showing the advantages and disadvantages.

### Point to Distribution

Biber in [biber2003normal] proposed this approach to achieve a 2D laser scan
matching, where he used the normal distribution transformation (NDT)to
represent the laser scan, then point-to-distribution approach to build the
cost function. Furthermore, Magnusson in [magnusson2007scan] extends the
approach; then, he proposed a 3D-NDT registration method. However, this
current work uses the GMM to represent the previous point set
$\\{\boldsymbol{\mathcal{F}}\\}$ (2), then use the Mahalanobis distance to
measure the distance between each transformed target $\boldsymbol{T}(m_{k})$
in the current point set $\\{\boldsymbol{\mathcal{M}}\\}$ and the previous
point set GMM as follows:

$\centering
p_{\mathrm{GMM}}(\boldsymbol{T}(\boldsymbol{m_{k}})|\lambda_{\mathcal{F}})=\sum_{i=1}^{|{\cal
F}|}w_{i}\cdot
p\left(\boldsymbol{T}(\boldsymbol{\mu_{k}})|\boldsymbol{\mu_{i}},\boldsymbol{\Sigma_{i}}\right)\@add@centering$
(3)

Thus, the cost function for $\boldsymbol{\theta}$ with respect to one target
$\boldsymbol{m_{k}}$ can be written as:

$\centering
f_{\mathrm{p2d}}(\boldsymbol{\theta})=p_{\mathrm{GMM}}\left(\boldsymbol{T}\left(\boldsymbol{\mu_{k}},\boldsymbol{\theta}\right)|\lambda_{\mathcal{F}}\right)\@add@centering$
(4)

Where $\\{\boldsymbol{\theta}\\}$ is the motion state, which needs to be
optimized. Consequently, for target $\boldsymbol{B}$, the cost function can be
written as:

$f_{\mathrm{p2d}}(\boldsymbol{\theta})_{\text{target}(B)}=\sum_{i=1}^{|{\cal
F}|}w_{i}\cdot
p\left(\boldsymbol{T}(\boldsymbol{\mu}_{B})|\boldsymbol{\mu_{i}},\boldsymbol{\Sigma}_{i}\right)$
(5)

The registration task translates target $\boldsymbol{B_{i+1}}$ to match all
targets in the previous point set $\\{\mathcal{F}\\}$, as depicted in Fig. 2
(a). The previous scan has three targets. Therefore, the cost surface includes
three possibilities, where at $t_{x}=2$ $B_{i+1}$ matches $B_{i}$, $t_{x}=10$
it matches $A_{i}$ and $t_{x}=12.2$ it matches $C_{i}$, as shown in Fig. 2(b).
By the same token, Fig. 3 (b) shows the cost surfaces for all targets.

[ The problem for target $\\{{B}\\}$] [ The cost surfacefor target
$\\{{B}\\}$]

Figure 2: The registration problem for target $\\{{B}\\}$.(a) represents the
aligning task for target $\\{B_{i+1}\\}$ and the previous point set
$\\{\mathcal{F}\\}$. (b) represents the cost surface for that target after the
aligning process.

### Distribution to Distribution

Several metrics are proposed to measure the distance between two probability
distribution, and Stoyanov in [stoyanov2012point] provides a review for most
of these metrics. However, most of the recent works [stoyanov2012point]
[barjenbruch2015joint] used the closed form for L2 provided by Jian in
[jian2010robust] to measure the distance between the two distributions. In
comparison to the Mahalanobis distance, L2 distance uses additionally to the
target position, the standard deviation of the targets in the current point
set. Therefore, the distance to be minimized is the distance between each
transformed target distribution and the GMM distribution for the previous
point set.

[ The cost surface for all targets p2d]  [ The cost surface for all targets
d2d]

Figure 3: The cost surface for all targets, where each target in the current
point set has three possibilities to be aligned with the previous point set.
The cost values based on the p2d approach is greater than the d2d approach
because the d2d includes the uncertainty of both points set in the cost
equation.

Consequently, the L2 distance is used to build the cost function. Assume two
distribution
$\\{\mathcal{N}_{1}(\boldsymbol{\mu_{1}},\boldsymbol{\Sigma_{1}}),\
\mathcal{N}_{2}(\boldsymbol{\mu_{2}},\boldsymbol{\Sigma_{2}})\\}$, L2 distance
can be written as :

$D_{L_{2}}(\mathcal{N}_{1},\mathcal{N}_{2})=\mathcal{N}\left(0|\boldsymbol{\mu}_{1}-\boldsymbol{\mu}_{2},\boldsymbol{\Sigma_{1}}+\boldsymbol{\Sigma_{2}}\right)$
(6)

Therefore, the cost function for $\boldsymbol{\theta}$ with respect to one
target $\boldsymbol{m_{k}}$ can be written as:

$f_{\mathrm{d2d}}(\boldsymbol{\theta})=\sum_{i=1}^{|{\cal F}|}w_{i}\cdot
p\left(0|(\boldsymbol{T}(\boldsymbol{\mu_{k}},\boldsymbol{\theta})-\boldsymbol{\mu_{i}}),(\boldsymbol{T}(\boldsymbol{\Sigma_{k}},\boldsymbol{\theta})+\boldsymbol{\Sigma_{i}})\right)$
(7)

Fig. 3 (b) shows the cost surface for all current targets based on the L2
metric, where each cost surface has three possibilities. Although the cost
surface behaves the same as the point-to-distribution approach, the cost
values are lower than in the point-to-distribution approach, which makes sense
as long as the point-to-distribution ignores the variance of the current
points, but distribution-to-distribution adds both variances.

### Discussion

Fig. 4 depicts the cost surface for the whole scenario, and the next section
introduces two methods to construct the overall cost surface. The total cost
surface for both approaches behave the same and have one global peak at the
correct aligning. Moreover, both surfaces include local peaks, which is a
well-known problem.

[Point-to-Distribution]  [Distribution-to-Distribution]

Figure 4: The overall cost surface for both approaches, where both cost
surfaces have the same behave.(a) represents the cost surface based on the
point-to-distribution approach. (b) represents the cost surface based on the
distribution-to-distribution approach.

Both cost surfaces behave the same as long as all points have the same
standard deviation, which is not the case in reality. Therefore, the scenario
is modified to have one point set with a small standard deviation, and the
second point set has a large standard deviation, to match the reality, as
shown in Fig. 5. From the cost surface in Fig. 6, it is obvious that both cost
surfaces do not behave the same. Moreover, in point-to-distribution, the cost
surface does not include a global peak. On the other hand, distribution-to-
distribution has only one global maximum beacuse it considers the noise of the
current point set in the distance metric, which is not the case in the point-
to-distribution approach.

[ The previous point set$\\{\mathcal{F}\\}$ ] [The current point
set$\\{\mathcal{M}\\}$ ]

Figure 5: One dimensional synthetic scenario, where the standard deviation for
current point set $\\{\mathcal{M}\\}$ is larger than the standard deviation
for previouse point set $\\{\mathcal{F}\\}$. (a) represents the previous point
set $\\{\mathcal{F}\\}$.(b) represents the current point $\\{\mathcal{M}\\}$.

[Point-to-Distribution] [Distribution-to-Distribution]

Figure 6:  The overall cost surface for both approaches, where both cost
surfaces does not have the same behave. (a) represents the cost surface based
on the point-to-distribution approach.(b) represents the cost surface based on
the distribution-to-distribution approach.

To sum up, the distribution-to-distribution shows a robust behavior against a
high standard deviation or more noisy point set. The following section
introduces two potential methods to fuse the cost surface for all targets.

##  3 Fusing Distance for all Targets

The previous section introduces two distance metrics to build the cost
function for each target. Besides, the distribution-to-distribution shows a
robust behavior than the point-to-distribution. Therefore, the distribution-
to-distribution strategy (7) is used to build the error metric for each
target. Two possible ways to implement the overall cost function: The summing
method and the likelihood method. The main interest here is to explain both
approaches. Furthermore, for consistent evaluation, the previous synthetic
scenario is used, as shown in Fig. 1.

### Summing Approach

The concept is to add or sum the cost for each target, where Barjenbruch in
[barjenbruch2015joint] and Stoyanov in [stoyanov2012point], as well as Biber
[biber2004probabilistic], used this approach in their implementation, as
follows:

$f_{\mathrm{d2d}}(\boldsymbol{\theta})=\sum_{k=1}^{|{\cal
M}|}D_{L_{2}}(\mathcal{N}_{GMM}(\mathcal{F}),\mathcal{N}_{k})$ (8)

$f_{\mathrm{d2d}}(\boldsymbol{\theta})=\sum_{k=1}^{|{\cal
M}|}\left(\sum_{i=1}^{|{\cal F}|}w_{i}\cdot
p\left(0|\boldsymbol{T}(\boldsymbol{m_{k}},\boldsymbol{\theta})-\boldsymbol{\mu_{i}},\boldsymbol{T}(\boldsymbol{\Sigma_{k}},\boldsymbol{\theta})+\boldsymbol{\Sigma_{i}}\right)\right)$
(9)

Fig. 7(a) shows the cost surface for the three targets, where the three cost
surfaces intersected at $t_{x}=2$. The summing approach depicted in Fig. 7(b),
where the cost surface has one global maximum and many local maxima.

### Likelihood Approach

The idea is to joint or multiply the cost for each target (7) to form the
overall cost function as follows :

$f_{\mathrm{d2d}}(\mathcal{M}|\lambda_{\mathcal{F}})=\prod_{k\in\mathcal{M}}D_{L_{2}}(\mathcal{N}_{GMM}(\mathcal{F}),\mathcal{N}_{k})$
(10)

Therefore, the cost function for $\boldsymbol{\theta}$ with respect to all
targets can be written as:

$f_{\mathrm{d2d}}(\boldsymbol{\theta})=\prod_{k\in\mathcal{M}}\left(\sum_{i=1}^{|{\cal
F}|}w_{i}\cdot
p\left(0|\boldsymbol{T}(\boldsymbol{m_{k}},\boldsymbol{\theta})-\boldsymbol{\mu_{i}},\boldsymbol{T}(\boldsymbol{\Sigma_{k}},\boldsymbol{\theta})+\boldsymbol{\Sigma_{i}}\right)\right)$
(11)

Fig. 7(c) exposes the likelihood approach, where the cost surface has only one
global maximum.

[ The cost surface for all targets ] [The summing approach ] [The likelihood
approach ]

Figure 7:  This plot exhibits the overall cost surface based on the summing
and the likelihood approach. (a) The cost surface for each target ploted in
one plot. (b) The cost surface for all targets based on the summing approach.
(c) The cost surface for all targets based on the likelihood(product)
approach. The cost surface based on the summing approach has many local peaks,
but the likelihood has only one optmal peak.

### Discussion

Both approaches have one global maximum at the correct translation
$(\boldsymbol{t_{x}=2})$. On the one hand, the cost surface based on the
summing method is smoother than the likelihood. On the other hand, the cost
surface has many local peaks, which could lead to false convergence in the
optimization step. Therefore, the initial guess is a critical aspect also for
the summing approach due to the local peaks.

Moreover, Rapp in [rapp2017probabilistic], Barjenbruch in
[barjenbruch2015joint], Stoyanov in [stoyanov2012point], as well as Biber
[biber2004probabilistic] used the summing approach to merge the cost function
for all targets with various justifications. One justification is that the
global maximum is smooth; thus, it should be easier for the optimizer to
converge in the optimal maximum. However, the summing approach shows a robust
behavior against the outlier. Hence, before discussing how the behavior is
robust against the outlier, it is intuitive to explain what does it means an
outlier and what is the outlier effect.

##  4 Outlier

An outlier is a data point, which remarkably deviates from the observation. In
our case, if a point is in one point set and does not have a correspondence in
the other set that represents an outlier, as shown in Fig. 8. This section
investigates how the outlier impacts the cost surface for each target cost
surface. Moreover, how the summing approach has a robust behavior against it
rather than the likelihood approach, as the previous works states.

### Outlier in Previous Point set

One possibility is to have a point in the previous point set, and it is not in
the current point set, as shown in Fig. 8 (a) and (b). The cost surface for
each target in the current point set will be changed because the outlier in
the point set introduces the fourth possibility, as depicted in Fig. 8 (c). on
the one hand, the summing approach introduces more local peaks. On the other
hand, the likelihood way shows a robust behavior against this type of outlier,
where the likelihood cost surface has only one global peak at the correct
translation, as shown in Fig. 9.

[The previous point set$\\{\mathcal{F}\\}$ ] [The current point
set$\\{\mathcal{M}\\}$ ] [The cost surface for all targets]

Figure 8: One dimensional synthetic scenario, where both point sets have the
same standard deviation. However,the previous point set $\\{F\\}$ have one
outlier $\\{O_{i}\\}$. The outlier introduces one more possibility in each
target cost surface.

[ The summing approach] [ The likelihood approach]

Figure 9:  This plot shows the impact of the outlier in the previous point
set. (a) Express the cost surface based on the summing method, where includes
more local peaks as the effect of the outlier. (b) Show the cost based on the
likelihood method, where it reveals a robust behavior against this type of
outlier.

### Outlier in Current Point set

[ The previous point set$\\{\mathcal{F}\\}$ ] [ The current point
set$\\{\mathcal{M}\\}$ ] [ The cost surface for all targets ]

Figure 10: One dimensional synthetic scenario, Where both point set have the
same the standard deviation. However,the current point set $\\{M\\}$ icludes
one outlier $\\{O_{i+1}\\}$. The outlier adds one more cost surface to the
overall cost surface show in black.

[ The summing approach ] [ The likelihood approach ]

Figure 11: The cost surface for all targets after adding one outlier
$\\{O_{i+1}\\}$ to the current point set $\\{M\\}$. (a) Represent the cost
surface for the summing approach, where more local peaks exist due to the
outlier. Moreover, one local peak has a large value comparing the other peaks,
but the global peak matches the correct translation. (b) Represent the
likelihood approach cost surface, where it is zero. Where the outlier cost
surface at the correct translation is zero, so the total product equals zero.

A second possibility is to have a point in the current point set and not in
the previous point set, as shown in Fig. 10. The cost surface for each target
will not change, but the outlier introduces a new cost surface. Therefore, the
outlier does not affect each target cost surface, but it affects the overall
cost surface, as depicted in Fig. 11.

In the summing approach, the outlier introduces more local peaks. Moreover, it
makes one local peak has a competitive value to the global peak. However, it
has more impact on the likelihood method, the overall cost surface becomes
zero. The likelihood multiplies the cost surface for each target. The cost
surface for the outlier has zero value at the correct transformation
$(t_{x}=2)$. Even so, Zisserman introduces the M-estimator concept to overcome
the problem. Which models the outlier as well as the inlier in the cost
function.

## Robust Cost Function

Although the outlier influenced both approaches, this work uses the likelihood
approach. This section explains one possible method to overcome the outlier
effect. Biber in [biber2004probabilistic] introduced the robust cost function
term. Zisserman in [hartley2003multiple, Appendix. 6] explains the whole idea
behind the robust cost function. The main idea is to model the outlier in the
cost function. In other words, the idea is to make the model robust against
the outlier. Zisserman introduces two ways to model the outlier:
Blake–Zisserman and corrupted Gaussian. The cost function based on the
distribution-to-distribution, and the likelihood is:

$f_{\mathrm{d2d}}(\boldsymbol{\theta})=\prod_{k\in\mathcal{M}}\left(\sum_{i=1}^{|{\cal
F}|}w_{i}\cdot
p\left(0|\boldsymbol{T}(\boldsymbol{m_{k}},\boldsymbol{\theta})-\boldsymbol{\mu_{i}},\boldsymbol{T}(\boldsymbol{\Sigma_{k}},\boldsymbol{\theta})+\boldsymbol{\Sigma_{i}}\right)\right)$
(12)

### Blake–Zisserman

[ The cost surface for all targets ] [ The likelihood approach ]

Figure 12: The cost for the problem based on the Blake–Zisserman:(a) The cost
surface for each target plotted in one plot. (b) The cost surface for all
targets based on the likelihood(product) approach.

This method models the outlier as uniform distribution, as follows:

$inlier_{component}=p\left(0|\boldsymbol{T}(\boldsymbol{m_{k}},\boldsymbol{\theta})-\boldsymbol{\mu_{i}},\boldsymbol{T}(\boldsymbol{\Sigma_{k}},\boldsymbol{\theta})+\boldsymbol{\Sigma_{i}}\right)$
(13)

$f_{\mathrm{d2d}}(\boldsymbol{\theta})=\prod_{k\in\mathcal{M}}\sum_{i=1}^{|\mathcal{F|}}w_{i}\left(inlier_{component}+outlier_{component}\right)$
(14)

Where, the $\boldsymbol{outlier_{component}}$ is a uniform distribution. Fig.
12 exhibits the cost surface for the problem after modeling the outlier as
uniform distribution, comparing to Fig. 11, the likelihood approach has one
global peak at the correct translation. Moreover, all local peaks are not
comparable to the global one, which is not the case in the summing approach.

### Corrupted Gaussian

[ The cost surface for all targets ] [ The likelihood approach ]

Figure 13: The cost surface based on the corrupted Gaussian, (a) The cost
surface for each target plotted in one plot. (b) The cost surface for all
targets based on the likelihood(product) approach.

Comparing to the previous method, this method models the outlier as a Gaussian
distribution, as follows:

$outlier_{component}=p\left(\boldsymbol{T}(\boldsymbol{m_{k}},\boldsymbol{\theta})|\boldsymbol{\mu}_{i},\boldsymbol{\Sigma}_{Outlier}\right)$
(15)

$f_{\mathrm{d2d}}(\boldsymbol{\theta})=\prod_{k\in\mathcal{M}}\sum_{i=1}^{|\mathcal{F|}}w_{i}\left(\alpha\cdot
inlier_{component}+(1-\alpha)\cdot outlier_{component}\right)$ (16)

Where, $\boldsymbol{\Sigma}_{Outlier}$ represents the covariance of the
outlier component, $\alpha$ is the inlier to outlier fraction ratio.

Fig. 13 depicts the cost surface based on the corrupted gaussian approach to
model the outlier component. The first observation is, the summing approach
cost surface behaves the same but with a change in the cost values. However,
the cost surface based on the likelihood that all local peaks are tiny
compared to the global peak. Nevertheless, modeling the outlier component’s
parameters $(\boldsymbol{{\Sigma}_{Outlier},\alpha})$ are very crucial as it
influences the cost surface.

### Discussion

The uniform distribution to model the outlier component reveals a robust
behavior against the outlier, but it violates the gaussian assumption. Which
is, the area under the curve does not equal to one. However, the corrupted
gaussian approach does not violate any Gaussian assumption and has a robust
behavior.

##  5 Clustered Points

This is a fully overlapped scenario, but some inlier points are near to each
other. The clustered points introduce multiple correct correspondences, as
this section will discuss. The clustered points can be in two shapes: the
points are very closed from each other, as shown in Fig. 14 or not completely
overlapped as shown in Fig. 15.

[ The previous point set ] [ The current point set ]

Figure 14: One dimensional synthetic scenario, where both point sets are fully
overlapped. However, this scenario includes some points are very closed from
each, which formulates a clustered scenario.

[ The previous point set ] [ The current point set ]

Figure 15: One dimensional synthetic scenario, where both point sets are fully
overlapped. However, this scenario includes some points are closed from each,
which formulates a clustered scenario.

The impact of the clustered points on the cost surface depends on how much the
points are closed to each other. Fig. 14 the clustered points are very closed
to each other; thus, the cost surface considered it as one component but with
a big impact, as shown in Fig. 16 (a). Therefore, this type of clustered
points does not have a drawback influence. Moreover, the global peak gets more
certain about the motion state, as shown in Fig. 16 (b) and (c). On the other
hand, when the points are not very closed, as shown in Fig. 15. It makes the
summing approach more sensitive to the initial guess as it introduces local
peaks near the global peak Fig. 17 (b), but it does not influence the
likelihood approach as shown in (c).

[ The cost surface for all targets ] [ The summing approach ] [ The likelihood
approach ]

Figure 16: The cost surface is based on the clustered points that are very
closed from each. (a) The clustered points are considered to be one component
with a significant impact. (b) The clustered points do not flaw the global
peak, but it makes the cost surface more certain. (c) The likelihood surface
has only one peak and gets more precise about the motion state.

[ The cost surface for all targets ] [ The summing approach ] [ The likelihood
approach ]

Figure 17:  The cost surface is based on the clustered points that are closed
but not closed like the previous example. (a) The clustered points are not
considered as one component anymore. (b) It introduces local peaks closed to
the global peak in the summing approach, which makes it more sensitive to the
initial guess. (c)It does not have a significant impact on the likelihood
approach.

## Discussion

This work proposed to use the L2 metric to build the cost function for each
target because it includes the uncertainty for both point sets: The current
and the previous point set. Moreover, it utilizes the likelihood approach to
fuse the cost surface for all targets rather than the summing approach because
the summing approach is more sensitive for the initial guess. Furthermore, it
uses the M-estimator methodology to overcome the outlier effect on the
likelihood approach. The corrupted Gaussian is used to model the outlier
component as it does not violate the Gaussian assumption.