# A family of elliptic curves with non-trivial Shafarevich-Tate group

Debopam Chakraborty  Department of Mathematics
BITS-Pilani, Hyderabad campus
Hyderabad, INDIA<EMAIL_ADDRESS>and  Vinodkumar Ghale
Department of Mathematics
BITS-Pilani, Hyderabad campus
Hyderabad, INDIA<EMAIL_ADDRESS>
###### Key words and phrases:

Elliptic curve; Selmer group; Class number; Heron triangle

###### 2020 Mathematics Subject Classification:

Primary 11G05, 11G07; Secondary 11R29, 51M04

## Abstract

Rank computation of an elliptic curve is one of the most important problems in
number theory due to its importance in various open questions in number
theory. A common way to understand this rank is via understanding the Selmer
group and the Shafarevich-Tate group of an elliptic curve. We construct a
family of Heronian elliptic curves of rank $0$ and $2$-Selmer rank $2$. This,
in turn, produces a family of elliptic curves such that the Shafarevich-Tate
group is isomorphic to the Klein four-group.

##  1\. Introduction

The rational points $E(\mathbb{Q})$ for an elliptic curve $E$ defined over
$\mathbb{Q}$ form a finitely generated abelian group by Mordell-Weil theorem.
It is a fundamental question in number theory to understand the rank
$r(E/\mathbb{Q})$ of $E(\mathbb{Q})$, also known as the algebraic or Mordell-
Weil rank. No known algorithm exists to find the rank of an elliptic curve. A
full $p$-descent method (cf. [18], Proposition X.$4$) generates an exact
sequence

$0\rightarrow E(\mathbb{Q})/pE(\mathbb{Q})\rightarrow
Sel_{p}(E/\mathbb{Q})\rightarrow\Sha(E/\mathbb{Q})[p]\rightarrow 0$

where $\text{Sel}_{p}(E/\mathbb{Q})$ denotes the $p$-Selmer group and
$\Sha(E/\mathbb{Q})$ denotes the Tate-Shafarevich group, both described in
detail for $p=2$ in the next section. The importance of the Selmer and the
Shafarevich-Tate group in the rank computation of an elliptic curve follows
from the above $p$-descent exact sequence as
$r(E/\mathbb{Q})=s_{p}(E/\mathbb{Q})-\text{dim}_{\mathbb{F}_{p}}\Sha(E/\mathbb{Q})[p]$
where
$s_{p}(E/\mathbb{Q})=\text{dim}_{\mathbb{F}_{p}}\text{Sel}_{p}(E/\mathbb{Q})-\text{dim}_{\mathbb{F}_{p}}E(\mathbb{Q})[p]$
denotes the $p$-Selmer rank of $E$.
Due to Cassels-Tate pairing (cf. [6]), the finiteness of the $p$-primary part
of $\Sha(E/\mathbb{Q})[p^{\infty}]$ would imply that $\Sha(E/\mathbb{Q})[p]$
has even $\mathbb{F}_{p}$ dimension, hence $s_{p}(E/\mathbb{Q})$ and
$r(E/\mathbb{Q})$ have the same parity. The finiteness of
$\Sha(E/\mathbb{Q})[p^{\infty}]$ implies p-Selmer rank one conjecture which
states that $r(E/\mathbb{Q})=1$ whenever $s_{p}(E/\mathbb{Q})=1$. This
conjecture has been verified for $p\geq 5$ under certain assumptions (cf.
[22], [20], [19], and [3]).
Very little is yet to be known about the $p$-Selmer rank one conjecture for
$p=2$ even though the computation of a full $2$-descent is easiest in practice
and provides the best tool as of yet to compute $r(E/\mathbb{Q})$. There has
been a growing interest in $2$-Selmer group computation for different families
of elliptic curves, as evident through the works of Klagsbrun-Mazur-Rubin in
[12], [13], and the work of Mazur-Rubin in [16].
D. R. Heath-Brown examined the size of the $2$-Selmer group of the congruent
number elliptic curve $E_{n}:y^{2}=x^{3}-n^{2}x$ in [1] and [2]. As an
appendix to [2], Monsky showed that the $2$-Selmer rank
$s_{2}(E_{n}/\mathbb{Q})=2m-\text{rank}_{\mathbb{F}_{2}}(M)$ when $m$ is the
number of distinct odd prime factors of $n$ and $M$ is a $2m\times 2m$ matrix
known as Monsky matrix. Thus the problem of finding the $2$-Selmer group for a
congruent elliptic curve becomes an intricate application of linear algebra
over $\mathbb{F}_{2}$.
An immediate generalization of the congruent number problem is the existence
of positive integers $n$, which are areas of triangles with rational sides
without the constraint of being a right-angle triangle. Such triangles are
called Heron triangles. In [10], Goins and Maddox showed the existence of an
elliptic curve similar to the congruent elliptic curve that determines whether
$n$ is the area of a Heron triangle. This triangle is known as Heron triangle
and is defined later. Even though the Monsly matrix calculation for $2$-Selmer
rank is unavailable in this setup, recently, some work towards the rank
computation of Heronian elliptic curves has been done. In [7], Dujella and
Peral showed elliptic curves of higher ranks associated with Heron triangles.
In [11], Halbeisen and Hungerbühler showed the existence of elliptic curves of
rank at least two associated with Heron triangles. In a recent work by
Chakraborty et al. [4], a family of Heronian elliptic curves with a $2$-Selmer
rank of precisely $1$ was presented.
In this article, we consider primes $p\equiv 1\pmod{8}$ and $q$ such that
$2q=p^{2}+1$. We examine the group structure of elliptic curves associated
with Heron triangles of area $p$ having one of the angles as $\theta$ such
that $\tau=\tan\frac{\theta}{2}=\frac{1}{p}$.

######  Theorem 1.1.

Let $p$ be a prime congruent to $1$ modulo $8$ and $q$ be a prime such that
$2q=p^{2}+1$. Then rank $r(E_{p})$ of the elliptic curve
$E_{p}:y^{2}=x(x-1)(x+p^{2})$ is precisely zero, and
$\Sha({E_{p}/\mathbb{Q}})[2]\cong\mathbb{Z}/2\mathbb{Z}\times\mathbb{Z}/2\mathbb{Z}$.

We note that the rank computation for the case $p\equiv 3,5,7\pmod{8}$ has
already been covered in [9] though the work did not involve the calculation of
$\Sha(E/\mathbb{Q})$. The main result of this work is the following.

##  2\. Local Solutions for the Homogeneous Spaces

A full $2$-descent method for an elliptic curve is elaborately described in
[18], Proposition X.$1.4$. In particular, for a Heronian elliptic curve a
similar background was presented in [4]. We notice that $E_{p}$ is a Heronian
elliptic curve associated to a Heron triangle of area $p$ and one of the angle
$\theta$ such that $\tan\frac{\theta}{2}=p^{-1}$.
If $\phi$ denotes the $2$-descent map, then from [18] (Proposition X.$1.4$),
we can say that $A=\\{(1,1),(-1,-1),(1,2q),(-1,-2q)\\}$ is the image of
$E_{p}(\mathbb{Q})_{\text{tors}}$ under $\phi$. Moreover, if
$(b_{1},b_{2})\in\mathbb{Q}(S,2)\times\mathbb{Q}(S,2)$ is a pair that is not
in the image of one of the three points $O,(0,0),(1,0)$ under $\phi$, where
$\mathbb{Q}(S,2)=\left\\{\pm 1,\;\pm 2,\;\pm p,\;\pm q,\;\pm 2p,\;\pm 2q,\;\pm
pq,\;\pm 2pq\right\\}$, then $(b_{1},b_{2})$ is the image of a point
$P=(x,y)\in E_{p}(\mathbb{Q})/2E_{p}(\mathbb{Q})$ if and only if the equations

(2.1) $\displaystyle b_{1}z_{1}^{2}-b_{2}z_{2}^{2}=1,$ (2.2) $\displaystyle
b_{1}z_{1}^{2}-b_{1}b_{2}z_{3}^{2}=-p^{2},$

have a solution
$(z_{1},z_{2},z_{3})\in\mathbb{Q}^{*}\times\mathbb{Q}^{*}\times\mathbb{Q}$
The smooth curves given by equations (2.1) and (2.2) are called homogeneous
space of $E_{p}$ defined over $\mathbb{Q}$. Computing
$E_{p}(\mathbb{Q})/2E_{p}(\mathbb{Q})$ boils down to determining the existence
of $\mathbb{Q}$-rational points in these spaces. The image of
$E_{p}(\mathbb{Q})/2E_{p}({\mathbb{Q}})$ under the $2$-descent map is
contained in a subgroup of $\mathbb{Q}(S,2)\times\mathbb{Q}(S,2)$ known as the
$2$-Selmer group $\mbox{Sel}_{2}(E_{p}/\mathbb{Q})$, which, as mentioned
earlier, fits into an exact sequence (see Chapter X, [18], Theorem X$.4.2$)

(2.3) $0\longrightarrow
E_{p}(\mathbb{Q})/2E_{p}(\mathbb{Q})\longrightarrow\mbox{Sel}_{2}(E_{p}/\mathbb{Q})\longrightarrow\Sha(E_{p}/\mathbb{Q})[2]\longrightarrow
0.$

The elements in $\mbox{Sel}_{2}(E_{p}/\mathbb{Q})$ correspond to the pairs
$(b_{1},b_{2})\in\mathbb{Q}(S,2)\times\mathbb{Q}(S,2)$ such that the system of
equations (2.1) and (2.2) has non-trivial local solutions in $\mathbb{Q}_{l}$
at all primes $l$ of $\mathbb{Q}$ including infinity. We note that
$\\#\,E_{p}(\mathbb{Q})/2E_{p}(\mathbb{Q})=2^{2+r(E_{p})}$ and write
$\\#\,\mbox{Sel}_{2}(E_{p}/\mathbb{Q})=2^{2+s(E_{p})}$, where $s(E_{p})$ is
referred as the $2$-Selmer rank. We have $0\leq r(E_{p})\leq s(E_{p})$ always.
We now examine the properties of the $l$-adic solutions for (2.1) and (2.2)
that are associated with the $2$-Selmer group. We use the well-known fact that
any $l$-adic number $a$ can be written as $a=l^{n}\cdot u\text{ where
}n\in\mathbb{Z}$, $u\in\mathbb{Z}_{l}^{*}$. Notice that the $l$-adic valuation
$v_{l}(a)$ of $a$ is just $n$. We first prove the following result for all odd
primes $l$.

######  Lemma 2.1.

Suppose (2.1) and (2.2) have a solution
$(z_{1},z_{2},z_{3})\in\mathbb{Q}_{l}\times\mathbb{Q}_{l}\times\mathbb{Q}_{l}$
for any prime $l$. If $v_{l}(z_{i})<0$ for any one $i\in\\{1,2,3\\}$, then
$v_{l}(z_{1})=v_{l}(z_{2})=v_{l}(z_{3})=-k<0$ for some integer $k$.

###### Proof.

Let $z_{i}=l^{k_{i}}u_{i}$, where $k_{i}\in\mathbb{Z}$ and
$u_{i}\in\mathbb{Z}_{l}^{*}$ for $i=\\{1,2,3\\}$. Then $v_{l}(z_{i})=k_{i}$
for all $i\in\\{1,2,3\\}$.

Suppose $k_{1}<0$. Then from (2.1) one can get that

$b_{1}u_{1}^{2}-b_{2}u_{2}^{2}l^{2(k_{2}-k_{1})}=l^{-2k_{1}}.$

If $k_{2}>k_{1}$, then $l^{2}$ must divide $b_{1}$, a contradiction as $b_{1}$
is square-free. Hence $k_{2}\leq k_{1}<0$. Now if $k_{2}<k_{1}<0$ then again
from (2.1) we get

$b_{1}u_{1}^{2}l^{2(k_{1}-k_{2})}-b_{2}u_{2}^{2}=l^{-2k_{2}},$

which implies $l^{2}$ must divide $b_{2}$, a contradiction again. Hence if
$k_{1}<0$, then we have $k_{1}=k_{2}=-k<0$ for some integer $k$. For
$k_{2}<0$, one similarly gets $k_{1}=k_{2}=-k<0$.
From (2.2), we have

$b_{1}u_{1}^{2}-b_{1}b_{2}u_{3}^{2}l^{2(k_{3}-k_{1})}=-p^{2}\cdot
l^{-2k_{1}}.$

If $k_{1}<0$ and $k_{3}>k_{1}$, then $l^{2}$ must divide $b_{1}$, a
contradiction as before. Hence $k_{3}\leq k_{1}<0$ if $k_{1}<0$. For
$k_{3}<k_{1}<0$, we can rewrite the above equation as

(2.4) $b_{1}u_{1}^{2}l^{2(k_{1}-k_{3})}-b_{1}b_{2}u_{3}^{2}=-p^{2}\cdot
l^{-2k_{3}},$

which implies $l^{2}$ must divide $b_{1}b_{2}$, i.e., $l=2,p$ or $q$. If
$l=p$, then from (2.4) we arrive at the contradiction that $p^{3}$ divides
$b_{1}b_{2}$ whereas $b_{1}$ and $b_{2}$ are square-free. For $l=2$ and $q$,
one can notice from (2.2) that if $k_{3}\leq-2$, then $l^{3}$ divides
$b_{1}b_{2}$, a contradiction again. This in turn implies $k_{3}=-1$ and hence
$k_{1}\geq 0$ which contradicts the assumption that $k_{1}<0$. Hence
$k_{1}<0\implies k_{3}=k_{1}$.
Now, suppose $k_{3}<0$. If $k_{1}<0$, then from the previous part we already
established $k_{1}=k_{2}=k_{3}=-k$ for some positive integer $k$. So without
loss of generality, we can assume $k_{1}\geq k_{3}$. If $k_{3}<k_{1}$ and
$k_{3}<0$, then as mentioned previously in this proof, one can get that
$b_{1}b_{2}\equiv 0\pmod{l^{2}}$ and $l=2$ or $q$. Now we subtract (2.2) from
(2.1) and observe that

$b_{1}b_{2}u_{3}^{2}-b_{2}u_{2}^{2}l^{2(k_{2}-k_{3})}=2q\cdot l^{-2k_{3}}.$

If $k_{2}>k_{3}$, we get a contradiction that $l^{3}$ divides $b_{1}b_{2}$ for
$l=2,q$. Therefore, $k_{2}\leq k_{3}<0$ but then by the first part,
$k_{1}=k_{2}\leq k_{3}$, a contradiction to the assumption $k_{1}>k_{3}$.
Hence $k_{3}<0\implies k_{1}=k_{3}$. Together, now we obtain
$k_{1}=k_{2}=k_{3}=-k<0$ for some integer $k$ if $k_{1}<0$ or $k_{2}<0$ or
$k_{3}<0$. ∎

The corollary below now follows immediately from the above lemma.

######  Corollary 2.2.

Let the system of equations (2.1) and (2.2) has a solution
$(z_{1},z_{2},z_{3})\in\mathbb{Q}_{l}\times\mathbb{Q}_{l}\times\mathbb{Q}_{l}$
for any prime $l$. Then either $v_{l}(z_{i})\geq 0$ for all $i\in\\{1,2,3\\}$,
or $v_{l}(z_{1})=v_{l}(z_{2})=v_{l}(z_{3})=-k<0$ for some positive integer
$k$.

##  3\. Bounding the Size of the $2$-Selmer Group

In this section, we bound the size of the $2$-Selmer group of the Heronian
elliptic curve $E_{p}$. We first note that it is enough to only look into one
$(b_{1},b_{2})\in[(b_{1},b_{2})]\in\text{Im}(\phi)/E_{p}(\mathbb{Q})_{\text{tors}}$.
As $E_{p}(\mathbb{Q})_{\text{tors}}=\\{(-1,-1),(1,2q),(-1,-2q),(1,1)\\}$, we
can immediately observe that without loss of generality, one can only focus on
$(b_{1},b_{2})$ such that $b_{2}$ is odd and $b_{1}$ and $b_{2}$ both positive
whenever their product $b_{1}b_{2}$ is positive. We limit the size of
$\mbox{Sel}_{2}(E_{p}/\mathbb{Q})$ by ruling out local solutions for
homogeneous spaces corresponding to certain pairs $(b_{1},b_{2})$.

######  Lemma 3.1.

Let $(b_{1},b_{2})\not\in\\{(1,1),(1,q),(p,1),(p,q)\\}$. Then the
corresponding homogeneous space can not have local solutions for all primes
$l\leq\infty$.

###### Proof.

Let the homogeneous space corresponding to
$(b_{1},b_{2})\in\mathbb{Q}(S,2)\times\mathbb{Q}(S,2)$ have real solutions.
Then $b_{1}>0$ and $b_{2}<0$ implies $-p^{2}>0$ in (2.2), which is absurd.
Similarly, $b_{1}<0$ and $b_{2}>0$ implies $1<0$ in (2.1), contradiction
again. Thus, the homogeneous space corresponding to $(b_{1},b_{2})$ has no
$l$-adic solutions for $l=\infty$ if $b_{1}b_{2}<0$.
If $\gcd(b_{1},b_{2})\equiv 0\pmod{p}$ and $v_{p}(z_{i})<0$ for any
$i\in\\{1,2,3\\}$, then from Corollary 2.2 and equation (2.2), one can get
$p^{2}$ divides $b_{1}$, a contradiction. Otherwise from Corollary 2.2 and
equation (2.1), one can get $p$ divides $1$, again a contradiction. Hence
$\gcd(b_{1},b_{2})\not\equiv 0\pmod{p}$. Moreover, if $p$ divides $b_{2}$,
then from Corollary 2.2, either $p$ divides $2q$ (subtracting (2.2) from
(2.1)) or $p$ divides both $b_{1},b_{2}$ (from (2.1)), contradiction either
way.
If $q$ divides $b_{1}$, then from Corollary 2.2 and subtracting (2.2) from
(2.1), one get that $q$ divides $b_{2}$ if $v_{q}(z_{i})\geq 0$ and so, $q$
divides $1$ from (2.1), a contradiction. Otherwise, again from Corollary 2.2
and (2.1), one gets that $q$ divides $b_{2}$ and hence from (2.2),
$b_{1}\equiv 0\pmod{q^{2}}$, a contradiction.
We now show that for the existence of local solutions everywhere, $b_{1}$
needs to be odd always. Otherwise, either $b_{1}$ even and
$z_{1},z_{3}\in\mathbb{Z}_{2}$ implies that $p^{2}\equiv 0\pmod{2}$, a
contradiction. Else, from Corollary 2.2 and (2.1), one can see that $b_{2}$ is
even, a contradiction from the assumption made above.
Now we can see for a homogeneous space corresponding to $(b_{1},b_{2})$ to
have possible local solution everywhere $b_{1}\in\\{1,p\\}$ and
$b_{2}\in\\{1,q\\}$. This concludes the proof. ∎

##  4\. Mordell-Weil Rank and ideal class group

In this section, we cover the pairs $(p,1)$ and $(1,q)$. We use the fact that
due to Hilbert’s class field theorem, the existence of an unramified abelian
extension of degree $n$ of a number field $K$ is equivalent to the class
number $h(K)\equiv 0\pmod{n}$. For brevity, we mention the original result of
Hilbert below.

######  Theorem 4.1 (Hilbert’s Class Field [17]).

The Hilbert class field $E$ of a number field $K$ is the maximal unramified
abelian extension of $K$. Its degree over $K$ equals the class number of $K$,
and $\text{Gal}(E/K)$ is canonically isomorphic to the ideal class group of
$K$.

We now state and prove that the homogeneous spaces corresponding to $(p,1)$
and $(1,q)$ have no rational solutions.

######  Lemma 4.2.

The homogeneous spaces corresponding to $(p,1)$ and $(1,q)$ have no rational
solutions.

###### Proof.

Let $z_{i}=\frac{a_{i}}{d_{i}}$ for $i=1,2,3$ is a rational solution set for
equations (2.1) and (2.2) where the rational numbers $z_{i}$ are in their
lowest form i.e. $\gcd(a_{i},d_{i})=1$ for all $i=1,2,3$. It can be shown
easily that $d_{1}^{2}=d_{2}^{2}=d_{3}^{2}=d^{2}$ for some integer $d$. So now
we have the following three equations;

(4.1) $\displaystyle b_{1}a_{1}^{2}-b_{2}a_{2}^{2}=d^{2},$ (4.2)
$\displaystyle b_{1}a_{1}^{2}-b_{1}b_{2}a_{3}^{2}=-p^{2}\cdot d^{2},$ (4.3)
$\displaystyle b_{1}b_{2}a_{3}^{2}-b_{2}a_{2}^{2}=2q\cdot d^{2},$

For the case $(p,1)$, one can trivially notice that $d$ is even; hence,
$a_{i}$ is odd for each $i=1,2,3$. Now from (4.2), $d^{2}\equiv
a_{1}^{2}-a_{3}^{2}\equiv 0\pmod{8}$, hence $d\equiv 0\pmod{4}$. A
straightforward calculation shows that there are no common odd prime factors
of $a_{1}+a_{3}$ and $a_{1}-a_{3}$. Assuming $a_{i}\geq 0$ for all $i=1,2,3$,
equation (4.2) then implies that one of the two possibilities of $a_{1}+a_{3}$
and $a_{1}-a_{3}$ is $a_{1}+a_{3}=p\cdot 2^{n_{1}}\cdot m_{1}^{2}$ and
$a_{1}-a_{3}=-2^{n_{2}}\cdot m_{2}^{2}$ where $m=m_{1}m_{2}$ is odd,
$n=n_{1}+n_{2}\geq 4$ and $d^{2}=2^{n}\cdot m^{2}$. The fact that $a_{3}$ is
odd and $2a_{3}=p\cdot 2^{n_{1}}\cdot m_{1}^{2}+2^{n_{2}}\cdot m_{2}^{2}$ now
implies that either $a_{3}=p\cdot 2^{n-2}\cdot m_{1}^{2}+m_{2}^{2}$ or
$a_{3}=p\cdot m_{1}^{2}+2^{n-2}\cdot$. In either way, $a_{3}\equiv 1\pmod{4}$.
Same is true for the case when $a_{1}+a_{3}=2^{n_{1}}\cdot m_{1}^{2},\text{
}a_{1}-a_{3}=-p\cdot 2^{n_{2}}\cdot m_{2}^{2}$.
Now if one defines $\alpha=a_{3}+d\sqrt{p}\in\mathbb{Q}(\sqrt{p})$, then from
(4.2) we get $N_{K/\mathbb{Q}}(\alpha)=a_{1}^{2}$ where
$K=\mathbb{Q}(\sqrt{p})$. Because $\gcd(a_{1},a_{3})=1$ can be proved easily,
one can also observe that $\gcd(\alpha,\bar{\alpha})=1$ in $\mathcal{O}_{K}$,
the ring of integers of $K$, where $\bar{\alpha}=a_{3}-d\sqrt{p}$. This in
turn implies that $\alpha\mathcal{O}_{K}=\mathfrak{a}^{2}$ for some ideal
$\mathfrak{a}$ which implies no finite primes except possibly primes above $2$
ramifies in $K(\sqrt{\alpha})/K$ (cf. [14], [5]). But
$\alpha=a_{3}+d\sqrt{p}\equiv 1\pmod{4}$ implies $2$ also does not ramify in
$K(\sqrt{\alpha})/K$. It is also clear that infinite primes also do not ramify
in $K(\sqrt{\alpha})/K$ as $K(\sqrt{\alpha})\subset\mathbb{R}$. Hence from
Hilbert’s class field theorem, we can conclude $K=\mathbb{Q}(\sqrt{p})$ has an
even class number whenever the homogeneous space corresponding to $(p,1)$ has
a rational solution. But it is well known that the class number of
$\mathbb{Q}(\sqrt{p})$ is always odd (cf. [8]). Hence the result follows.
We now look into the case $(b_{1},b_{2})=(1,q)$. Similarly to the previous
case, we can show that $d\equiv 0\pmod{4}$ and $a_{3}\equiv 1\pmod{4}$ again.
From (4.3), this implies $\alpha=a_{3}+d\sqrt{2}$ again gives an unramifed
abelian quadratic extension $K(\sqrt{\alpha})/K$ where
$K=\mathbb{Q}(\sqrt{2})$. Hence existence of rational solution to homogeneous
space corresponding to $(1,q)$ implies $h(\mathbb{Q}(\sqrt{2}))$ is even, a
contradiction as $h(\mathbb{Q}(\sqrt{2}))=1$. ∎

##  5\. Everywhere Local Solution

Now we prove that the homogeneous spaces corresponding to $(p,1)$ and $(1,q)$
have local solutions everywhere. To lift a solution for a polynomial $f(x)$
modulo a prime $l$ to a solution for $f(x)$ in $\mathbb{Z}_{l}$, we will need
Hensel’s lemma. For the sake of clarity we first state Hensel’s lemma below.

######  Theorem 5.1 (Hensel’s Lemma (see [21], Theorem A$.2$)).

If $f(x)$ is a polynomial with coefficients that are $l$-adic integers and
$f(x_{1})\equiv 0\pmod{l}$ for $x_{1}\in\mathbb{Z}$, then if
$f^{\prime}(x_{1})\not\equiv 0\pmod{l}$, there exists an $l$-adic integer $x$
with $x\equiv x_{1}\pmod{l}$ such that $f(x)=0$

Now we are in a position to look into the local solutions for the homogeneous
spaces corresponding to $(p,1)$ and $(1,q)$.

######  Lemma 5.2.

The homogeneous spaces corresponding to $(p,1)$ and $(1,q)$ have local
solutions everywhere for $l\leq\infty$.

###### Proof.

For $(b_{1},b_{2})=(p,1)$, first we consider $l\geq 5$, $l\neq p$. Suppose $C$
is the homogeneous space given by (2.1) and (2.2) corresponding to the pair
$(p,1)$. Then $C$ is a twist of $E_{p}$, and in particular, it has genus $1$.
By the Hasse-Weil bound, we have

$\\#C(\mathbb{F}_{l})\;\geq\;1+l-2\sqrt{l}\;\geq\;2\qquad\mbox{ for }l\geq
5,\text{ }l\neq p.$

We can choose a solution
$(z_{1},z_{2},z_{3})\in\mathbb{F}_{l}\times\mathbb{F}_{l}\times\mathbb{F}_{l}$
such that not all three of them are zero modulo $l$. Now $z_{1}\equiv
z_{2}\equiv 0\pmod{l}$ implies $l^{2}$ divides $1$ from (2.1), a
contradiction. Similarly, $z_{1}\equiv z_{3}\equiv 0\pmod{l}$ implies
$-p\equiv 0\pmod{l^{2}}$ from (2.2), contradiction again. One can now suitably
choose two of $z_{1},z_{2}$ and $z_{3}$ to convert equations (2.1) and (2.2)
into one single equation of one variable with a simple root over
$\mathbb{F}_{l}$. That common solution can be lifted to $\mathbb{Z}_{l}$ via
Hensel’s lemma.
For $l=p$, we first notice that $\big{(}\frac{-1}{p}\big{)}=1$ as $p\equiv
1\pmod{8}$. Hence there exists $a\in\mathbb{Z}$ such that
$a^{2}\equiv-1\pmod{p}$. Now fixing $z_{1}=1$ in equation (2.1) and (2.2), we
can immediately see that $z_{2}=a$ and $z_{3}=1$ are two simple roots of now
two single variable polynomials and hence can be lifted to $\mathbb{Z}_{p}$.
For $l=3$ and $p\equiv 1\pmod{3}$, using Lemma 2.1, we look into solutions for
the equations $pz_{1}^{2}-z_{2}^{2}=3^{2k}$ and $z_{1}^{2}-z_{3}^{2}=-p\cdot
3^{2k}$ in $\mathbb{Z}_{3}$. Fixing $z_{1}=1$, one can notice that
$z_{2}=z_{3}=1$ are two simple solutions and hence can be lifted to solution
in $\mathbb{Z}_{3}$ for the equations mentioned above. Diving by $3^{2k}$,
will then give rise to solutions in $\mathbb{Q}_{3}$ for equations (2.1) and
(2.2). For $p\equiv 2\pmod{3}$, fixing $z_{2}=1$ and $z_{3}=0$ in (2.1) and
(2.2) respectively will give $z_{1}=1$ as a simple solution modulo $3$ and
hence can be lifted to $\mathbb{Z}_{3}$ via Hensel’s lemma.
For the case $l=2$, just as in the beginning of the case $l=3$, using Lemma
2.1, we find solutions in $\mathbb{Z}_{2}$ for the equations
$pz_{1}^{2}-z_{2}^{2}=2^{2k}$ and $z_{1}^{2}-z_{3}^{2}=-p\cdot 2^{2k}$ such
that $k\geq 2$. Fixing $z_{2}=z_{3}=1$ gives rise to $z_{1}\equiv 1\pmod{8}$
as a solution modulo $8$ to both those equations that can be lifted to a
solution in $\mathbb{Z}_{2}$. This ensures a solution for the homogeneous
space corresponding to $(p,1)$ in $\mathbb{Q}_{2}$ also.
For the case $(b_{1},b_{2})=(1,q)$, the proof follows a very similar way as in
the case $(b_{1},b_{2})=(p,1)$. For $l\geq 5,l\neq p,q$, the homogeneous
spaces $C$ corresponding to $(1,q)$ given by equations (2.1) and (2.2) are of
genus $1$ and have solutions in $\mathbb{F}_{l}$ by Hasse-Weil bound and can
be lifted to $\mathbb{Z}_{l}$ via a similar argument used in the previous
case.
For $l=q$, because $-p^{2}\equiv 1\pmod{q}$, fixing $z_{2}=z_{3}=0$ in
equations (2.1) and (2.2) gives $z_{1}=1$ as a solution to the homogeneous
space modulo $q$ that can be lifted to $\mathbb{Z}_{l}$ via Hensel’s lemma.
For $l=p$, because $2q\equiv 1\pmod{p}$, one can notice that fixing $z_{1}=1$
and then choosing $z_{2}=1$ and $z_{3}=a$ is a solution that can be lifted to
$\mathbb{Z}_{l}$ via Hensel’s lemma where $a^{2}\equiv 2\pmod{p}$.
For the case $l=3$, fixing $z_{1}=1$ and choosing $z_{2}=z_{3}=1$ gives rise
to solutions for the equations $z_{1}^{2}-qz_{2}^{2}=3^{2k}$ and
$z_{1}^{2}-qz_{2}^{2}=-p^{2}\cdot 3^{2k}$. This solution can be lifted to
$\mathbb{Z}_{l}$ and then give solution for (2.1) and (2.2) in
$\mathbb{Q}_{l}$ as mentioned in the previous part.
For the case $l=2$, noticing the fact $p\equiv 1\pmod{8}\implies q\equiv
1\pmod{8}$, the proof follows the same way with the same choice of solutions
modulo $8$ for the case $(b_{1},b_{2})=(p,1)$. ∎

##  6\. Mordell-Weil rank and $\Sha({E_{p}/\mathbb{Q}})[2]$

We now conclude with the proof of Theorem 1.1 using the already established
results. A table is provided below in support of our result. Computations for
the table has been done in Magma [15] software.

###### Proof of Theorem 1.1.

From Lemma 3.1 and Lemma 4.2, it is now obvious that the rank of
$E_{p}:y^{2}=x(x-1)(x+p^{2})$ is at most one due to the homogeneous space
corresponding to $(p,q)$ which we are yet to rule out. But, Lemma 5.2
guarantees that both $(p,1),(1,q)\in\Sha(E_{p}/\mathbb{Q})[2]$. This, along
with the group structure of $\Sha(E_{p}/\mathbb{Q})$ implies that
$(p,q)=(p,1)\times(1,q)\in\Sha(E_{p}/\mathbb{Q})[2]$ too. Consequently,
$r(E_{p})$ is exactly zero. Moreover, due to the Cassels-Tate pairing (see
[6]), $\text{dim}_{\mathbb{F}_{2}}(\Sha(E_{p}/\mathbb{Q}))[2]$ is even which
implies
$\Sha(E_{p}/\mathbb{Q})[2]\cong\mathbb{Z}/2\mathbb{Z}\times\mathbb{Z}/2\mathbb{Z}$.
This concludes the proof of the theorem. ∎

Table 1.  Examples $p$ | Rank of $E_{p}$ | 2- Selmer Rank of $E_{p}$ | $\Sha(E_{p}/\mathbb{Q})[2]$
---|---|---|---
409 | 0 | 2 | $(\mathbb{Z}/2\mathbb{Z})^{2}$
449 | 0 | 2 | $(\mathbb{Z}/2\mathbb{Z})^{2}$
521 | 0 | 2 | $(\mathbb{Z}/2\mathbb{Z})^{2}$
569 | 0 | 2 | $(\mathbb{Z}/2\mathbb{Z})^{2}$
641 | 0 | 2 | $(\mathbb{Z}/2\mathbb{Z})^{2}$

## Acknowledgement

The first author would like to acknowledge DST-SERB for providing the grant
through the Start-Up Research Grant (SRG/2020/001937) as well as BITS-Pilani,
Hyderabad Campus, for providing amenities. The second author would like to
acknowledge the fellowship (File No:09/1026(0029)/2019-EMR-I) and amenities
provided by the Council of Scientific and Industrial Research, India (CSIR)
and BITS-Pilani, Hyderabad.

## References

  * [1] D.R. Heath-Brown, "The size of Selmer groups for the congruent number problem." Inventiones Mathematicae 111.1 (1993): 171-195.
  * [2] D.R. Heath-Brown, "The size of Selmer groups for the congruent number problem, II." Inventiones Mathematicae 118 (1994): 331-370.
  * [3] C. Francesc, and X. Wan, "Iwasawa Main Conjecture for Heegner Points: Supersingular Case." arXiv preprint arXiv:1506.02538 (2015).
  * [4] D. Chakraborty, V. Ghale, and A. Saikia, "Construction of an infinite family of elliptic curves of 2-Selmer rank 1 from heron triangles." Research in Number Theory (2022).
  * [5] D. Chakraborty, and A. Saikia, "An explicit construction for unramified quadratic extensions of biquadratic fields." Acta Arithmetica 178 (2017): 153-161.
  * [6] L. Chao, "2-Selmer groups, 2-class groups and rational points on elliptic curves." Transactions of the American Mathematical Society 371, no. 7 (2019): 4631-4653.
  * [7] A. Dujella, and J.C. Peral, "Elliptic curves coming from Heron triangles." The Rocky Mountain Journal of Mathematics 44.4 (2014): 1145-1160.
  * [8] A. Dujella, and F. Luca, "On fundamental units of real quadratic fields of class number 1." Archiv der Mathematik 113, no. 4 (2019): 349-353.
  * [9] V. Ghale, S. Das, and D. Chakraborty, "A Heron triangle and a Diophantine equation." Periodica Mathematica Hungarica (2022): 1-8.
  * [10] E.H. Goins, and D. Maddox, "Heron triangles via elliptic curves." The Rocky Mountain Journal of Mathematics (2006): 1511-1526.
  * [11] L. Halbeisen, and H. Norbert, "Heron triangles and their elliptic curves." Journal of Number Theory 213 (2020): 232-253.
  * [12] K. Zev, B. Mazur, and K. Rubin, "Disparity in Selmer ranks of quadratic twists of elliptic curves." Annals of Mathematics (2013): 287-320.
  * [13] K. Zev, B. Mazur, and K. Rubin. "A Markov model for Selmer ranks in families of twists." Compositio Mathematica 150, no. 7 (2014): 1077-1106.
  * [14] F. Lemmermeyer, "Why is the class number of $\mathbb{Q}(\root 3 \of{11})$ even?." Mathematica Bohemica 138, no. 2 (2013): 149-163.
  * [15] The MAGMA algebra system, available at http://magma.maths.usyd.edu.au/calc/
  * [16] B.Mazur, and K. Rubin. "Ranks of twists of elliptic curves and Hilbert’s tenth problem." Inventiones Mathematicae 181, no. 3 (2010): 541-575.
  * [17] J. Neukirch, “Algebraic number theory". Vol. 322. Springer Science and Business Media, 2013.
  * [18] J.H. Silverman, "The arithmetic of elliptic curves." Vol. 106. New York: Springer, 2009.
  * [19] S. Christopher, "A converse to a theorem of Gross, Zagier, and Kolyvagin." Annals of Mathematics 191, no. 2 (2020): 329-354.
  * [20] W. Xin, "Heegner point Kolyvagin system and Iwasawa main conjecture." Acta Mathematica Sinica, English Series 37, no. 1 (2021): 104-120.
  * [21] L.C. Washington, "Elliptic curves: number theory and cryptography." Chapman and Hall/CRC, 2008.
  * [22] W. Zhang, "Selmer groups and the indivisibility of Heegner points." Cambridge Journal of Mathematics 2, no. 2 (2014): 191-253.


# Deep Learning–Based Perceptual Stimulus Encoder for Bionic Vision

Lucas Relic  University of California, Santa BarbaraSanta BarbaraCAUSA
<EMAIL_ADDRESS>,  Bowen Zhang  University of California, Santa
BarbaraSanta BarbaraCAUSA<EMAIL_ADDRESS>,  Yi-Lin Tuan  University of
California, Santa BarbaraSanta BarbaraCAUSA<EMAIL_ADDRESS>and  Michael
Beyeler  University of California, Santa BarbaraSanta BarbaraCAUSA
<EMAIL_ADDRESS>
(2022)

###### Abstract.

Retinal implants have the potential to treat incurable blindness, yet the
quality of the artificial vision they produce is still rudimentary. An
outstanding challenge is identifying electrode activation patterns that lead
to intelligible visual percepts (phosphenes). Here we propose a perceptual
stimulus encoder (PSE)based on convolutional neural networks (CNNs)that is
trained in an end-to-end fashion to predict the electrode activation patterns
required to produce a desired visual percept. We demonstrate the effectiveness
of the encoder on MNIST using a psychophysically validated phosphene model
tailored to individual retinal implant users. The present work constitutes an
essential first step towards improving the quality of the artificial vision
provided by retinal implants.

deep learning, retinal implants, stimulus optimization

††journalyear: 2022††copyright: rightsretained††conference: Augmented Humans
2022; March 13–15, 2022; Kashiwa, Chiba, Japan††booktitle: Augmented Humans
2022 (AHs 2022), March 13–15, 2022, Kashiwa, Chiba, Japan††doi:
10.1145/3519391.3524034††isbn: 978-1-4503-9632-5/22/03††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Machine learning††ccs:
Computing methodologies Modeling and simulation Figure 1.  A) Simulated
prosthetic vision (SPV). Constrained by neuroanatomical and/or psychophysical
data, a phosphene model $\Psi$ (e.g., (Beyeler et al., 2019)) predicts what a
retinal implant user should “see” for any given input stimulus. The predicted
percept is typically a nonlinear continuous function of the input stimulus,
thus $\Psi$ can be approximated by a generic feedforward neural network (FNN),
$\hat{\Psi}$, which is amenable to differentiation. B) End-to-end optimization
of bionic vision. For a given target percept, a stimulus encoder based on a
convolutional neural network (CNN)is trained to predict the combination of
active electrodes that generates the percept with the smallest possible
reconstruction loss. The FNN approximator is fixed during encoder training.

##  1\. Introduction

Hereditary retinal diseases such as retinitis pigmentosa are among the leading
causes of incurable blindness in the world. Retinal implants (though
elementary) provide an improved ability to localize high-contrast objects as
well as perform basic orientation & mobility tasks (Ayton et al., 2020). These
devices electrically stimulate surviving cells in the visual pathway to evoke
visual percepts (phosphenes).

However, the quality of this artificial vision is still rudimentary, as the
visual percepts elicited by current implants are often unrecognizable
(Erickson-Davis and Korzybska, 2021). A major outstanding challenge is
identifying electrode activation patterns that lead to perceptually
intelligible phosphenes. One approach is to consider this an end-to-end
optimization problem, where a deep neural network (encoder) is trained to
predict the electrical stimulus needed to produce a desired percept (Fig. 1).

To this end, we make the following contributions:

  * •

We propose a perceptual stimulus encoder (PSE)based on convolutional neural
networks (CNNs)that is trained in an end-to-end fashion to predict the
electrode activation patterns required to produce a desired visual percept.
Importantly, the encoder is based on a psychophysically validated
computational model of bionic vision (Beyeler et al., 2019).

  * •

We demonstrate the effectiveness of the PSE on the MNIST dataset for three
different users of the Argus II Retinal Prosthesis System (Second Sight
Medical Products) (Luo and da Cruz, 2016).

##  2\. Related Work

###  2.1. Simulated Prosthetic Vision (SPV)

The goal of simulated prosthetic vision (SPV) is to predict what bionic eye
users “see” in response to electrical stimulation. To date, most SPV studies
rely on the scoreboard method, which assumes that each phosphene acts as a
small independent light source, analogous to the images projected on the light
bulb arrays of some sports stadium scoreboards (Dobelle, 2000). However,
evidence suggests that phosphenes often appear distorted (e.g., as simple
geometric shapes such as lines, wedges, and blobs) and vary drastically across
subjects and electrodes (Erickson-Davis and Korzybska, 2021). More recently,
SPV models have therefore aimed to explain phosphene appearance as a function
of neuroanatomical and psychophysical data (Beyeler et al., 2019; Granley and
Beyeler, 2021) (Fig. 1A). Open-source implementations for many of these models
are provided by _pulse2percept_ , a Python-based bionic vision simulator
(Michael Beyeler et al., 2017).

###  2.2. End-to-End Optimization of Bionic Vision

Although deep learning has previously been combined with SPV to perform image
processing on the predicted percept (e.g., (Zhao et al., 2018; Horne et al.,
2016; Han et al., 2021)), only few studies have considered the quest to
improve the quality of artificial vision as an end-to-end optimization problem
(Fig. 1B). Most notably, van Steveninck et al. (van Steveninck et al., 2020)
used a similar approach to ours by training an encoder-decoder deep neural
network with a scoreboard model in the loop. However, their approach differs
in three crucial ways: i) their phosphene model could not account for
empirical data from current retinal implant users (Beyeler et al., 2019;
Erickson-Davis and Korzybska, 2021), ii) their loss function did not consider
the performance of either encoder or phosphene model, and most importantly
iii) their decoder was in itself a deep neural network that could potentially
learn to compensate for any deficiencies in the encoder or the phosphene
model.

##  3\. Methods

Our model is illustrated in Fig. 1 and described in more detail below. We
first approximated a psychophysically validated phosphene model ($\Psi$,
(Beyeler et al., 2019)) with a generic feedforward neural network (FNN)
($\hat{\Psi}$, Fig. 1A). Once trained, the weights of the FNN Approximator
were frozen and used to train a perceptual stimulus encoder (PSE) to minimize
the reconstruction error between predicted and target percepts (Fig. 1B). The
PSE took a target image as input and returned a combination of active
electrodes as output, which were then fed into the FNN Approximator to predict
a visual percept. The pixel-wise mean squared error between predicted and
target percept served as the reconstruction error. The error was then
backpropagated via the differentiable FNN Approximator to update the weights
of the PSE. As a proof of concept, we considered the model’s ability to
predict handwritten digits from the popular MNIST dataset.

###  3.1. FNN Approximator

Our group (Beyeler et al., 2019) demonstrated through computational modeling
that phosphene shape in epiretinal implants primarily depends not just on
stimulus parameters but also on the retinal location of the stimulating
electrode. This model ($\Psi$) can be fit to individual patients; however, it
is not differentiable. We therefore approximated $\Psi$ with a single-layer
FNN ($\hat{\Psi}$) that took a $1\times 60$ vector of current amplitudes as
input (one amplitude for each electrode in the Argus II Retinal Prosthesis
System) and returned a $121\times 161$ image as output (i.e., the predicted
percept). The FNN Approximator was trained on a synthetic dataset (50,000
samples, 80-20 train-test split) generated with $\Psi$: each sample was
generated by first randomly selecting a number $N\in[1,30]$ to stimulate, then
randomly assigning a stimulation current between 1 and 10 microamps to each
electrode. After training, the weights of the FNN Approximator were frozen.

###  3.2. Perceptual Stimulus Encoder (PSE)

The PSE was a convolutional neural network (CNN) consisting of two $3\times 3$
convolutional layers (stride 1) followed by a max pooling layer after each
convolutional layer, and a fully connected layer at the end.

Rather than optimizing the PSE with a pre-trained FNN Approximator in the
loop, one might also consider training an inverse phosphene model
($\hat{\Psi^{-1}}$) to directly predict the required stimulus for a desired
percept. We therefore trained an inverse model of equal depth to the PSE as a
baseline model.

###  3.3. Simulated Bionic Eye Users

Since phosphene appearance varies drastically across patients, we followed
Beyeler et al. (Beyeler et al., 2019) to tailor the phosphene model $\Psi$ to
the individual implant setup of three Argus II patients: 12-005, 51-009, and
52-001. The setup for these patients mainly differed in the implant location
and in the values of model parameters $\rho$ (describing phosphene size) and
$\lambda$ (describing phosphene elongation) (Beyeler et al., 2019).
Consequently, we had to train three different FNN Approximators and three
different PSEs.

| Subject 12-005 | Subject 51-009 | Subject 52-001
---|---|---|---
Perceptual Stimulus Encoder (PSE) | $\mathbf{0.0317\pm 0.014}$ | $\mathbf{0.0547\pm 0.019}$ | $\mathbf{0.0311\pm 0.012}$
Inverse Phosphene Model ($\hat{\Psi}^{-1}$) | $0.0371\pm 0.016$ | $0.0718\pm 0.025$ | $0.0456\pm 0.017$
Table 1. Reconstruction error (pixel-wise mean squared error) achieved on
MNIST, reported as mean $\pm$ standard deviation across samples in the test
set. Figure 2. Representative example predictions achieved by the PSE trained
on three different phosphene models that represent three different Argus II
subjects. For each digit (left column in each panel), the corresponding
predicted percept (right column) is shown.

##  4\. Results

Results are summarized in Table 1. The best reconstruction error was achieved
for Subject 52-001 ($0.0311)$, followed by 12-005 ($0.0317$) and 51-009
($0.0547$). The relatively poor performance of the model for Subject 51-009
may be due to the fact that this subject sees very thin, elongated phosphenes
that may not be easily combined to form an MNIST digit. A deeper encoding
model may help to overcome this issue.

The PSE outperformed the inverse phosphene model in all three instances. While
and end-to-end trained inverse phosphene model might seem advantageous at
first, it has the notable drawback that the mapping from desired percept to
required stimulus may be one-to-many, which may not lend itself well to
gradient-based optimization.

Fig. 2 shows representative examples of digits with low (good) and high (bad)
reconstruction errors for all three subjects. The PSE was able to utilize the
inherent streakiness of the phosphene model to produce recognizable digits
with a small number of active electrodes. However, it is worth noting that the
digits with the lowest reconstruction error did not always look the best. The
model tended to yield poor results for thick digits, which required the
activation of a large number of electrodes and subsequently produced large
indistinct blobs. Since all phosphenes have an inherent orientation, the model
also struggled with digits whose edges ran orthogonal to that orientation.

##  5\. Conclusion

The present work constitutes an essential first step towards improving the
quality of artificial vision provided by current retinal implants. Future work
should focus on more naturalistic datasets and developing a better perceptual
error metric. The most important future contribution, however, will be to
demonstrate that the proposed approach is a viable strategy to improve the
quality of artificial vision in real bionic eye users.

###### Acknowledgements.

This work was supported by the National Institutes of Health (NIH R00
EY-029329 to MB).

## References

  * (1)
  * Ayton et al. (2020) Lauren N. Ayton, Nick Barnes, Gislin Dagnelie, Takashi Fujikado, Georges Goetz, Ralf Hornig, Bryan W. Jones, Mahiul M. K. Muqit, Daniel L. Rathbun, Katarina Stingl, James D. Weiland, and Matthew A. Petoe. 2020.  An update on retinal prostheses.  _Clinical Neurophysiology_ 131, 6 (June 2020), 1383–1398.  https://doi.org/10.1016/j.clinph.2019.11.029
  * Beyeler et al. (2019) Michael Beyeler, Devyani Nanduri, James D. Weiland, Ariel Rokem, Geoffrey M. Boynton, and Ione Fine. 2019\.  A model of ganglion axon pathways accounts for percepts elicited by retinal implants.  _Scientific Reports_ 9, 1 (2019).  https://doi.org/10.1038/s41598-019-45416-4
  * Dobelle (2000) Wm H. Dobelle. 2000\.  Artificial Vision for the Blind by Connecting a Television Camera to the Visual Cortex.  _ASAIO Journal_ 46, 1 (Feb. 2000), 3–9.
  * Erickson-Davis and Korzybska (2021) Cordelia Erickson-Davis and Helma Korzybska. 2021.  What do blind people “see” with retinal prostheses? Observations and qualitative reports of epiretinal implant users.  _PLOS ONE_ 16, 2 (Feb. 2021), e0229189.  https://doi.org/10.1371/journal.pone.0229189 Publisher: Public Library of Science.
  * Granley and Beyeler (2021) Jacob Granley and Michael Beyeler. 2021.  A Computational Model of Phosphene Appearance for Epiretinal Prostheses. 4477–4481.  https://doi.org/10.1109/EMBC46164.2021.9629663 ISSN: 2694-0604.
  * Han et al. (2021) Nicole Han, Sudhanshu Srivastava, Aiwen Xu, Devi Klein, and Michael Beyeler. 2021.  Deep Learning–Based Scene Simplification for Bionic Vision. In _Augmented Humans Conference 2021_ _(AHs’21)_. Association for Computing Machinery, New York, NY, USA, 45–54.  https://doi.org/10.1145/3458709.3458982
  * Horne et al. (2016) Lachlan Horne, Jose Alvarez, Chris McCarthy, Mathieu Salzmann, and Nick Barnes. 2016\.  Semantic labeling for prosthetic vision.  _Computer Vision and Image Understanding_ 149 (2016), 113–125.
  * Luo and da Cruz (2016) Y. H. Luo and L. da Cruz. 2016.  The Argus((R)) II Retinal Prosthesis System.  _Prog Retin Eye Res_ 50 (Jan. 2016), 89–107.  https://doi.org/10.1016/j.preteyeres.2015.09.003
  * Michael Beyeler et al. (2017) Michael Beyeler, Geoffrey M. Boynton, Ione Fine, and Ariel Rokem. 2017.  pulse2percept: A Python-based simulation framework for bionic vision. In _Proceedings of the 16th Python in Science Conference_ , Katy Huff, David Lippa, Dillon Niederhut, and M Pacer (Eds.). 81 – 88.  https://doi.org/10.25080/shinma-7f4c6e7-00c
  * van Steveninck et al. (2020) Jaap de Ruyter van Steveninck, Umut Guclu, Richard JA van Wezel, and Marcel AJ van Gerven. 2020.  End-to-end optimization of prosthetic vision.  _bioRxiv_ (2020).
  * Zhao et al. (2018) Ying Zhao, Qi Li, Donghui Wang, and Aiping Yu. 2018\.  Image Processing Strategies Based on Deep Neural Network for Simulated Prosthetic Vision. In _2018 11th International Symposium on Computational Intelligence and Design (ISCID)_ , Vol. 01. 200–203.  https://doi.org/10.1109/ISCID.2018.00052


# Angular-Based Word Meta-Embedding Learning

James O’ Neill and Danushka Bollegala
Department of Computer Science, University of Liverpool
Liverpool, L69 3BX
England
<EMAIL_ADDRESS>

(13/08/2018)

###### Abstract

Ensembling word embeddings to improve distributed word representations has
shown good success for natural language processing tasks in recent years.
These approaches either carry out straightforward mathematical operations over
a set of vectors or use unsupervised learning to find a lower-dimensional
representation. This work compares meta-embeddings trained for different
losses, namely loss functions that account for angular distance between the
reconstructed embedding and the target and those that account normalized
distances based on the vector length. We argue that meta-embeddings are better
to treat the ensemble set equally in unsupervised learning as the respective
quality of each embedding is unknown for upstream tasks prior to meta-
embedding. We show that normalization methods that account for this such as
cosine and KL-divergence objectives outperform meta-embedding trained on
standard $\ell_{1}$ and $\ell_{2}$ loss on defacto word similarity and
relatedness datasets and find it outperforms existing meta-learning
strategies.

##  1 Introduction

Meta-embeddings are a quick and useful prior step for improving word
representations in natural language learning tasks. This involves combining
several learned embeddings in a way that improve the overall input
representation. This approach is a less computationally expensive compared to
if a practitioner were to train a set of word embeddings from scratch,
particularly when considering non-sliding window methods. The most
straightforward approaches to meta-embeddings are: concatenation (CONC) and
averaging (AV). The former is limited since the dimensionality grows large for
multiple embeddings as more vectors are concatenated and the latter, while
fast, does not preserve most of the information encoded in each embedding when
taking the arithmetic mean. Although, it would seem surprising concatenating
vectors from different embedding spaces is valid, it has been shown that
Coates and Bollegala (2018) AV approximates CONC even though the embedding
spaces are different. Although, to address the loss of information when using
AV, Singular Value Decomposition has been used as a dimensionality reduction
technique to factorize the embeddings into a lower-rank approximation of the
concatenated meta-embedding set.

Linear methods include the use of a projection layer for meta-embedding (known
as 1TON) Yin and Schütze (2015), which is simply trained using an
$\ell_{2}$-based loss. Similarly, Bollegala et al. Bollegala et al. (2017) has
focused on finding a linear transformation between count-based and prediction-
based embeddings, showing that linearly transformed count-based embeddings can
be used for predictions in the localized neighborhoods in the target space.

Most recent work Bao and Bollegala (2018) has focused on the use of an
autoencoder (AE) to encode a set of $N$ pretrained embeddings using 3
different variants: (1) Decoupled Autoencoded Meta Embeddings (DAEME) that
keep activations separated for each respective embedding input during encoding
and uses a reconstruction loss for both predicted embeddings while minimizing
the loss for each respective decoded output, (2) Coupled Autoencoded Meta
Embeddings (CAEME) which instead learn to predict from a shared encoding and
(3) Averaged Autoencoded Meta-Embedding (AAME) is simply an averaging of the
embedding set as input instead of using a concatenation. This is the most
relevant work to our paper, hence, we include these 3 autoencoding schemes
along with aforementioned methods for experiments, described in Section 3. We
also include two subtle variations of the aforementioned AEs. The first
predicts a target embedding from an embedding set using the remaining
embedding set, post-learning the single hidden layer is used as the word meta-
embedding. The second method is similar except an AE is used for each input
embedding to predict the designated target embedding, followed by an averaging
over the resulting hidden layers. This alternative is described in more detail
in Section 2.

The aforementioned previously proposed unsupervised learning have a common
limitation, that is minimising the Euclidean ($\ell_{2}$) distance between
source word embeddings and their meta-embedding. Arora et al. (2016) have
shown that the $\ell_{2}$ norm of a word embedding vector is proportional to
the frequency of the corresponding word in the corpus used to learn the word
embeddings. Considering that in meta-embedding learning we use source
embeddings trained on different resources, we argue that it is more important
to preserve the semantic orientation of words, which is captured by the angle
between two word embeddings, not their length. Indeed, cosine similarity, a
popularly used measure for computing the semantic relatedness among words,
ignores the length related information. Additionally, we note the relationship
between KL-divergence and cosine similarity in the sense both methods perform
a normalization that is proportional to the semantic information. Hence, we
compare several popular measures such as MSE and MAE that use $\ell_{2}$ and
$\ell_{1}$ respectively, against KL-divergence and cosine similarity for the
purpose of learning meta-embeddings and show that the loss which accounts for
this orientation consistently outperforms the former objectives that only
consider length. We demonstrated this across multiple benchmark datasets.

##  2 Methodology

Before describing the loss functions used, we explain the aforementioned
variation on the autoencoding method and how it slightly differs from
1TON/1TON+ Yin and Schütze (2015) and standard AEs Bao and Bollegala (2018)
presented in previous work. Target Autoencoders (TAE) are defined as learning
an ensemble of nonlinear transformations between sets of bases $X_{s}$ in sets
of vector spaces
$\mathcal{X}_{S}=\\{\mathcal{X}_{1},..,\mathcal{X}_{s},..,\mathcal{X}_{N}\\}\quad
s.t\quad\mathcal{X}_{s}\in\mathbb{R}^{|v_{s}|\times d_{s}}$ to a target space
$\mathcal{X}_{t}\in\mathcal{R}^{|v_{t}|\times d_{t}}$, where
$\mathtt{f}_{w}^{(i)}:\mathcal{X}_{S}^{(i)}$ $\to$
$\mathcal{X}_{t}\quad\forall i$ is the nonlinear transformation function used
to make the mapping. Once a set of $M$ number of parameteric models
$\mathtt{f}_{w}=\\{\mathtt{f}_{w}^{(1)},\mathtt{f}_{w}^{(i)},..,\mathtt{f}_{w}^{(M)}\\}$
are trained with various objective functions to learn the mappings between the
word vectors we obtain a set of lower-dimensional target latent representation
that represents different combinations of mappings from one vector space to
another. After training, all $H$ set of latent variables
$Z_{s}=\\{z_{1},..,z_{H}\\}$ are concatenated with an autoencoded target
vector. This means thats all vector spaces have been mapped to a target space
and there hidden meta-word representations have been averaged, as illustrated
in Figure 1.

Figure 2 shows a comparison of the previous autoencoder approaches Bao and
Bollegala (2018) (left) and the alternative AE (right), where dashed lines
indicate connections during training and bold lines indicate prediction. The
Concat-AutoEncoder (CAEME) simply concatenates the embedding set into a single
vector and trains the autoencoder so to produce a lower-dimensional
representation (shown in red), while the decoupled autoencoder (DAEME) keeps
the embedding vectors separate in the encoding. In contrast the target encoder
(TAE) is similar to that of CAEME only the label is a single embedding from
the embedding set and the input are remaining embeddings from the set. After
training, TAE then concatenates the hidden layer encoding with the original
target vector. The Mean Target AutoEncoder (MTE) instead performs an averaging
over separate autoencoded representation.

Figure 1: TAE Embedding Figure 2: AE Meta-Embedding Methods

### AutoEncoder Settings

The standard Autoencoder (AE) is a 1-hidden layer AE of hidden layer dimension
$h_{d}=200$. Weights are initialized with a normal distribution, mean $\mu=0$
and standard deviation $\sigma=1$. Dropout is used with a dropout rate $p=0.2$
for all datasets. The model takes all unique vocabulary terms pertaining to
all tested word association and word similarity datasets ($n=4819$) and
performs Stochastic Gradient Descent (SGD) with batch size $\tilde{x}=32$
trained between 50 epochs for each dataset $\forall d\in\mathcal{D}$. This
results in a set of vectors $X_{j}\in\mathbb{R}^{|v_{j}|\times 200}\;\forall
j$ that are then used for finding the similarity between word pairs. The above
parameters were chosen ($h_{d}$, $\tilde{x}$ and number of epochs) over a
small grid search. As stated, we compare against previous methods Yin and
Schütze (2015); Bao and Bollegala (2018) that use $\ell_{2}$ distance, as
shown in Equation 1).

$\mathcal{L}_{\theta}(\hat{y},y)=\frac{1}{N}\sum_{i=1}^{N}(y^{(i)}-\hat{y}^{(i)})^{2}$
(1)

Similarly, the Mean Absolute Error ($\ell_{1}$ norm of difference) loss
$\frac{1}{N}\sum_{i=1}^{N}|y-\hat{y}|$ is tested. We also compare against a KL
divergence objective, as shown in Equation 2, $\hat{y}$ is the last activation
output from the log-softmax that represents $q(x)$ and the KL-divergence is
given as $KL(p|q)=\sum_{i=1}^{N}p(x_{i})\log\big{(}q(x_{i})/p(x_{i})\big{)}$.

$\mathcal{L}_{\theta}(\hat{y},y)=\frac{1}{N}\sum_{i=1}^{N}y^{(i)}\cdot\left(\log\big{(}y^{(i)}\big{)}-\hat{y}^{(i)}\right)$
(2)

Since $\tanh$ functions are used and input vectors are $\ell_{2}$ normalized
we propose a Squared Cosine Proximity (SCP) loss, shown in Equation 3 where
$m$ is the output dimensionality. This forces the optimization to tune weights
such that the rotational difference between the embedding spaces is minimized,
thus preserving semantic information in the reconstruction. In the context of
its utility for the TAE, we too want to minimize the angular difference
between corresponding vectors in different vector spaces. It is also a
suitable fit since it is a proper distance measure (i.e symmetric), unlike KL-
divergence.

$\mathcal{L}_{\theta}(\hat{y},y)=\frac{1}{N}\sum_{i=1}^{N}\Bigg{(}1-\frac{\sum_{j=1}^{m}\hat{y}_{ij}*y_{ij}}{\sqrt{\sum_{ij}^{m}\hat{y}_{ij}^{2}}\sqrt{\sum_{ij}^{m}y_{ij}^{2}}}\Bigg{)}^{2}$
(3)

The model is kept relatively simple so that the comparisons against previous
methods are directly comparable and that the performance comparison between
the proposed SCP loss and KL divergence loss against MSE and MAE is
controlled. Additionally, all comparison are that of models which are only
trained from co-occurrence statistics that are not leveraging any external
knowledge (e.g AutoExtend Rothe and Schütze (2015)).

##  3 Experiments

The following word association and word similarity datasets are used
throughout experimentation: Simlex Hill et al. (2015), WordSim-353 Finkelstein
et al. (2001), RG Rubenstein and Goodenough (1965), MTurk (MechanicalTurk-771)
Halawi et al. (2012), RareWord (RW) Luong et al. (2014) and MEN Bruni et al.
(2012). The word vectors considered in the embeddings set are
$\mathtt{skipgram}$ and $\mathtt{cbow}$ Mikolov et al. (2013), FastText
Bojanowski et al. (2016), LexVec Salle et al. (2016), Hellinger PCA (HPCA)
Lebret and Collobert (2013) and Hierarchical Document Context (HDC) Sun et al.
(2015). We now report results on the performance of meta-embedding
autoencodings with various loss functions, while also presenting target
autoencoders for combinations of word embeddings and compare against existing
current SoTA meta-embeddings.

Table 1 shows the scaled Spearman correlation test scores, where (1) shows the
original single embeddings, (2) results for standard meta-embedding approaches
that either apply a single mathematical operation or employ a linear
projection as an encoding, (3) presents the results using autoencoder schemes
by Bao and Bollegala (2018) that we have used to test the various losses, (4)
introduces TAE without concatenating the target $Y$ embedding post-training
with MSE loss and (5) shows the results of concatenating $Y$ with the lower-
dimensional (200-dimensions) vector that encodes all embeddings apart from the
target vector. Therefore reported results from (4) are of a 200d vector, while
(5) concatenates the vector leading to a vector between 300-500 dimensions
dependent on the target vector. All trained encodings from sections 3-5 are
200-dimensional vectors. Results in red shading indicate the best performing
meta-embedding for all presented approaches, while black shading indicates the
best performing meta-embedding for the respective section.

Best performing word meta-embeddings are held between concatenated
autoencoders that use the proposed Cosine-Embedding loss, while a KL-
divergence also performs well on Simlex and RareWord. Interestingly, both of
these dataset are distinct in that Simlex is the only dataset providing scores
on true similarity instead of free association, which has shown to be more
difficult for word embeddings to account for Hill et al. (2016), while
Rareword provides morphologically complex words to find similarity between.
Concretely, it would seem KL-divergence is well suited for encoding when the
word relations exhibits of a more complex or rare nature. Similarly, we find
SCP loss to achieve best results on RG and MEN, both the smallest and largest
datasets of the set. Furthermore, the TAE variant has lead to competitive
performance overall against other meta-embedding approaches and even produces
best results on WS353. Lastly, we find that HPCA embeddings are relatively
weak for word similarity.

1\. Embeddings | Simlex | WS353 | RG | MTurk | RW | MEN |
---|---|---|---|---|---|---|---
Skipgram | 44.19 | 77.17 | 76.08 | 68.15 | 49.70 | 75.85 |
FastText | 38.03 | 75.33 | 79.98 | 67.93 | 47.90 | 76.36 |
GloVe | 37.05 | 66.24 | 76.95 | 63.32 | 36.69 | 73.75 |
LexVec | 41.93 | 64.79 | 76.45 | 71.15 | 48.94 | 80.92 |
HPCA | 16.60 | 57.11 | 41.72 | 37.45 | 13.36 | 34.90 |
HDC | 40.68 | 76.81 | 80.58 | 65.76 | 46.34 | 76.03 |
2\. Standard Meta |  |  |  |  |  |  |
CONC | 42.57 | 72.13 | 81.36 | 71.88 | 49.91 | 80.33 |
SVD | 41.10 | 72.06 | 81.18 | 71.50 | 49.13 | 79.85 |
AV | 40.63 | 70.5 | 80.05 | 70.51 | 49.28 | 78.31 |
1TON | 41.30 | 70.19 | 80.20 | 71.52 | 50.80 | 80.39 |
1TON* | 41.49 | 70.60 | 78.40 | 71.44 | 50.86 | 80.18 |
3\. $\ell_{2}$-AE |  |  |  |  |  |  |
Decoupled | 42.56 | 70.62 | 82.81 | 71.16 | 50.79 | 80.33 |
Concatenated | 43.10 | 71.69 | 84.52 | 71.88 | 50.78 | 81.18 |
$\ell_{1}$-AE |  |  |  |  |  |  |
Decoupled | 43.52 | 70.30 | 82.91 | 71.43 | 51.48 | 81.16 |
Concatenated | 44.41 | 70.96 | 81.16 | 69.63 | 51.89 | 80.92 |
Cosine-AE |  |  |  |  |  |  |
Decoupled | 43.13 | 71.96 | 84.23 | 70.88 | 50.20 | 81.02 |
Concatenated | 44.85 | 72.44 | 85.41 | 70.63 | 50.74 | 81.94 |
KL-AE |  |  |  |  |  |  |
Decoupled | 44.13 | 71.96 | 84.23 | 70.88 | 50.20 | 81.02 |
Concatenated | 45.10 | 74.02 | 85.34 | 67.75 | 53.02 | 81.14 |
4\. TAE |  |  |  |  |  |  |
$\to$Skipram | 37.80 | 67.33 | 76.50 | 63.41 | 37.52 | 74.86 |
$\to$ FastText | 38.17 | 66.62 | 77.184 | 64.73 | 37.84 | 74.77 |
$\to$Glove | 39.95 | 77.14 | 81.58 | 68.82 | 47.94 | 76.67 |
$\to$ LexVec | 37.48 | 67.19 | 75.98 | 63.96 | 37.70 | 74.75 |
$\to$HPCA | 40.78 | 65.79 | 38.64 | 59.49 | 38.65 | 74.50 |
$\to$HDC | 38.15 | 66.96 | 76.62 | 63.08 | 37.53 | 76.62 |
5\. TAE +$Y$ |  |  |  |  |  |  |
$\to$ Skipram | 42.43 | 75.33 | 80.11 | 66.51 | 44.77 | 78.98 |
$\to$ FastText | 41.69 | 72.65 | 80.51 | 67.64 | 47.41 | 77.48 |
$\to$ Glove | 41.75 | 76.65 | 82.40 | 68.92 | 48.83 | 78.27 |
$\to$ LexVec | 42.85 | 73.33 | 80.97 | 69.17 | 46.71 | 79.63 |
$\to$ HPCA | 40.03 | 69.65 | 70.43 | 61.31 | 36.38 | 73.10 |
$\to$ HDC | 42.43 | 74.08 | 80.11 | 66.51 | 44.76 | 77.93 |

Table 1: Meta-Embedding Results

##  4 Conclusion

We find the meta-embeddings trained using Autoencoders with a Squared Cosine
loss and a KL-divergence loss improves performance in the majority of cases,
reinforcing the argument that accounting for angles explicitly through
normalization (log-softmax for KL) is an important criterion for encoding. It
is particularly useful for distributed word representations, since embeddings
are learned from large documents of varying length and semantics. Lastly, we
have shown its use in the context of improving meta-embeddings, although this
suggests cosine loss is also suitable for minimizing angular differences for
word embeddings, not only for meta-embeddings. Concretely, this paper has
carried out a comprehensive study of methods to embed a lower-dimensional
representation from embedding sets, while proposing losses that explicitly
keep angular information intact for meta-embeddings.

## References

  * Arora et al. (2016) Sanjeev Arora, Yuanzhi Li, Yingyu Liang, Tengyu Ma, and Andrej Risteski. 2016.  A latent variable model approach to pmi-based word embeddings.  _Transactions of the Association for Computational Linguistics_ , 4:385–399.
  * Bao and Bollegala (2018) Cong Bao and Danushka Bollegala. 2018.  Learning word meta-embeddings by autoencoding.  _COLING_.
  * Bojanowski et al. (2016) Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016.  Enriching word vectors with subword information.  _arXiv preprint arXiv:1607.04606_.
  * Bollegala et al. (2017) Danushka Bollegala, Kohei Hayashi, and Ken-ichi Kawarabayashi. 2017.  Learning linear transformations between counting-based and prediction-based word embeddings.  _PloS one_ , 12(9):e0184544.
  * Bruni et al. (2012) Elia Bruni, Gemma Boleda, Marco Baroni, and Nam-Khanh Tran. 2012.  Distributional semantics in technicolor.  In _Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1_ , pages 136–145. Association for Computational Linguistics.
  * Coates and Bollegala (2018) Joshua Coates and Danushka Bollegala. 2018.  Frustratingly easy meta-embedding–computing meta-embeddings by averaging source word embeddings.  _arXiv preprint arXiv:1804.05262_.
  * Finkelstein et al. (2001) Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001.  Placing search in context: The concept revisited.  In _Proceedings of the 10th international conference on World Wide Web_ , pages 406–414. ACM.
  * Halawi et al. (2012) Guy Halawi, Gideon Dror, Evgeniy Gabrilovich, and Yehuda Koren. 2012.  Large-scale learning of word relatedness with constraints.  In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 1406–1414. ACM.
  * Hill et al. (2015) Felix Hill, Roi Reichart, and Anna Korhonen. 2015.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_ , 41(4):665–695.
  * Hill et al. (2016) Felix Hill, Roi Reichart, and Anna Korhonen. 2016.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  _Computational Linguistics_.
  * Lebret and Collobert (2013) Rémi Lebret and Ronan Collobert. 2013.  Word emdeddings through hellinger pca.  _arXiv preprint arXiv:1312.5542_.
  * Luong et al. (2014) Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014.  Addressing the rare word problem in neural machine translation.  _arXiv preprint arXiv:1410.8206_.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Rothe and Schütze (2015) Sascha Rothe and Hinrich Schütze. 2015.  Autoextend: Extending word embeddings to embeddings for synsets and lexemes.  _arXiv preprint arXiv:1507.01127_.
  * Rubenstein and Goodenough (1965) Herbert Rubenstein and John B Goodenough. 1965.  Contextual correlates of synonymy.  _Communications of the ACM_ , 8(10):627–633.
  * Salle et al. (2016) Alexandre Salle, Marco Idiart, and Aline Villavicencio. 2016.  Matrix factorization using window sampling and negative sampling for improved word representations.  _arXiv preprint arXiv:1606.00819_.
  * Sun et al. (2015) Fei Sun, Jiafeng Guo, Yanyan Lan, Jun Xu, and Xueqi Cheng. 2015.  Learning word representations by jointly modeling syntagmatic and paradigmatic relations.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , volume 1, pages 136–145.
  * Yin and Schütze (2015) Wenpeng Yin and Hinrich Schütze. 2015.  Learning meta-embeddings by using ensembles of embedding sets.  _arXiv preprint arXiv:1508.04257_.


*OPTIONAL comment concerning the title, e.g., if a variant or an extended abstract of the paper has appeared elsewhere.

[a] [b] [c]

#  Segmentation-free Connectionist Temporal Classification loss based OCR
Model for Text Captcha Classification

Dr Vaibhav Khatavkar0000-0002-1825-0097  ,  Dr Makarand
Velankar0000-0001-8946-0350  and  Sneha Petkar 0009-0001-2783-1475  VIT Bhopal
University, Bhopal, Madhya Pradesh, India<EMAIL_ADDRESS>MKSSS’s Cummins COE, Pune, Maharashtra, India
<EMAIL_ADDRESS>GES’s Sapat COE, Nashik, Maharashtra,
India<EMAIL_ADDRESS>
###### Abstract.

Captcha are widely used to secure systems from automatic responses by
distinguishing computer responses from human responses. Text, audio, video,
picture picture-based Optical Character Recognition (OCR) are used for
creating captcha. Text-based OCR captcha are the most often used captcha which
faces issues namely, complex and distorted contents. There are attempts to
build captcha detection and classification-based systems using machine
learning and neural networks, which need to be tuned for accuracy. The
existing systems face challenges in the recognition of distorted characters,
handling variable-length captcha and finding sequential dependencies in
captcha. In this work, we propose a segmentation-free OCR model for text
captcha classification based on the connectionist temporal classification loss
technique. The proposed model is trained and tested on a publicly available
captcha dataset. The proposed model gives 99.80% character level accuracy,
while 95% word level accuracy. The accuracy of the proposed model is compared
with the state-of-the-art models and proves to be effective. The variable
length complex captcha can be thus processed with the segmentation-free
connectionist temporal classification loss technique with dependencies which
will be massively used in securing the software systems.

###### Key words and phrases:

Connectionist Temporal Classification loss based classification, OCR , Text
Capatcha, Deep Neural Network segmentation-free captcha recognition

thanks, optional.

## Introduction

In the digital world, large volumes of data are created and processed. Along
with the large volume of data, the variety of data is another challenge. The
data can be either text, image, audio, video, or a combination of text, image,
audio, and video. The processing of combined data is usually performed by
computer tasks defined as protocols and standards. Optical Character
Recognition (OCR) is one such standard process used to read text data from the
images. It has been more than a decade that researchers have developed various
techniques in OCR [Rajan A, 2021]. Researchers in [Fujitake M, 2024] have used
a transformer model in the Neural Network for encoding and decoding OCR. Still
today, OCR is applied in applications like car number plate detection [Nasim
H, 2024], cattle identification [Smink M, 2024], and text extraction from
lecture slides [Jobin K, 2024]. Optical Character Recognition (OCR) is a field
that has seen significant advancements, and various models and techniques have
been developed to tackle the challenges posed by recognizing characters in
captcha. Captcha, designed to be challenging for automated systems, often
require specialized approaches. Captcha is an acronym for Completely Automated
Turing Test to Tell Computers and Humans Apart [Hussain R, 2016, Ahn V, 2003,
Blum M, 2004, Gao H, 2017]. They are regularly used as a security measure for
web services. Captcha is an image embedded with text, audio or video data.
[Choudhary S, 2013]. The Text contents in the image are typically distorted.
The text Captcha is generally a OCR, while audio and video Captcha are non
OCR-based systems [Hasan W, 2016]. Though OCR-based Captcha is widely used,
there are major issues in them [Choudhary S, 2013]. These issues are :

  * •

Distorted contents: The captcha with text is distorted randomly. The
distortion can make the text unreadable.

  * •

Complex contents: The text captcha contains text with English digits and
characters. For a user, the English language might be complex to read. The
captcha are generated randomly, which may form a complex series of digits and
texts, making the user difficult to understand and interpret. Sometimes these
contents can be easily read by a computer program or an AI bot, making captcha
breakable.

Despite these issues, many applications still use captcha in their systems for
human user verification. The researchers in paper [Algwil A, 2023] review the
captcha along with application and classification. OCR-based captcha are used
for detection of captcha by computers. The researchers in [Sabeti V, 2024,
Kuznetsov O, 2024, Billion P, 2024] have proposed computer-based solutions to
detect OCR-based text captcha using methods like Deep Neural Networks, Machine
Learning and CNN [ref5, ref6, ref7]. The motivation behind developing an OCR
(Optical Character Recognition) model specifically designed for reading
captcha stems from several important factors related to online security, user
authentication, and the evolving landscape of automated threats. There is a
need to provide sophisticated a method to provide security to the text-based
captcha to increase its robustness against the attacks. To be effective,
captcha need to be sufficiently complex to prevent automated systems from
easily deciphering and responding to them. This complexity often involves
distorted characters, varying fonts, and other obfuscation techniques that can
be challenging for traditional OCR systems. Also, conventional OCR models,
which are designed for general text recognition, may struggle to accurately
interpret characters in captcha images. Thus, to recognize the distorted
characters efficiently, handle variable length of captcha and resolve
sequential dependencies a novel OCR based captcha detection approach is
proposed in this paper.

##  1\. Related Work

Deep learning techniques are widely used in feature extraction from the images
and has a variety of applications in image restoration [Gao H, 2013, Jin Z,
2020, Malik S, 2019] and object detection [Dong W, 2019, Ren S, 2017, Liu Y,
2018]. The high computational power of these techniques has made it a good
choice to build a captcha detection system. Despite the high computational
power, these techniques suffer due to weak feature extraction processes and
noisy input images. Thus, new techniques in deep learning are to be proposed
for captcha detection. The segmentation-based and segmentation-free are two
widely used techniques for text-based captcha detection. The segmentation-
based systems use either segmentation or character recognition algorithm
[Abdussalam A, 2019, Abdussalam A, 2018]. The segmentation algorithm is a two-
step process with the segmentation of image as first followed by character
recognition Algorithm 1. Despite being a simple two-step process, the
segmentation-based captcha detection systems have low efficiency and low
accuracy as well. To improve efficiency and accuracy, segmentation-free
captcha detection algorithms are used. The segmentation-free algorithms
recognize and classify the characters directly, i.e. without segmenting the
image into individual characters.

Algorithm 1: Segmentation Algorithm [Abdussalam A, 2019, Abdussalam A, 2018]

Step 1: Segmentation of captcha into individual characters using segmentation
module

Step 2: Individual character recognition using a character recognition module.

To train the deep learning models for captcha recognition with necessary image
features, large training datasets are required to increase the efficiency of
the algorithm. The segmentation-free models, though widely used, have complex
architectures. They also require relatively large storage resources, which
depend on the number of characters in the image. Apart from this, Captcha
often includes intentional distortions that make them inherently different
from standard text, thus requiring a specialized approach. captcha typically
involve characters that are intentionally distorted, rotated, or presented in
a way that is challenging for conventional OCR models. The OCR model needs to
be robust enough to handle these distortions and accurately recognize the
characters. captcha can have varying lengths, which means the OCR model should
be capable of handling sequences of characters with different lengths. This
requirement makes traditional fixed-size input OCR models less suitable,
necessitating the use of techniques like Connectionist Temporal Classification
(CTC) loss. Captcha images typically contain characters arranged in a
sequence, where the correct interpretation of one character depends on the
context provided by the surrounding characters. Recurrent Neural Networks
(RNNs) are particularly effective in capturing these sequential dependencies.
Thus, to recognize the distorted characters efficiently, handle variable
length of captcha and resolve sequential dependencies a novel OCR based
captcha detection approach is proposed in this paper. The template based OCR
techniques are susceptible to variations in font, size, and orientation
[Sakkatos P, 2014, Wu C, 2005, Baten R A, 2014]. They are also ineffective for
distorted or obfuscated characters in captcha. The feature-based approaches
have limited adaptability to highly distorted characters and are sensitive to
variations in scale and orientation. In the development of OCR, deep learning
models like CNN [Krizhevsky A, 2017, Jmour N, 2018], and RNN have played a
pivotal role. The CNN when used in OCR provided an excellent feature
extraction technique for image-based tasks. The OCR model with CNN can capture
hierarchical features, useful for recognizing patterns. RNN are proven to be
effective in capturing sequential dependencies in OCR. OCR with RNN are found
suitable for recognizing characters in a specific order. CRNNs integrate CNNs
and RNNs to capture both spatial and sequential information. They are
effective for recognizing characters in variable-length sequences. CNN-based
segmentation-free models for captcha detection are proposed by various
researchers in [Chen J, 2019, hu Y, 2018, Stark F, 2015]. Some segmentation-
free models combine CNN and attention-based recurrent neural network (RNN) for
captcha detection [Qing K, 2017, Zi Y, 2020, Wang P, 2020].

The Connectionist Temporal Classification (CTC) Loss for Variable-Length
Sequences is another model for OCR development. This model handles variable-
length sequences without requiring explicit alignment. It is also useful for
training models to handle the dynamic nature of captcha lengths. Therefore, in
this paper, we propose a Connectionist Temporal Classification loss based OCR
Model for Text Captcha Classification.

##  2\. Proposed System

The architecture of an OCR (Optical Character Recognition) model that
incorporates Convolutional Neural Networks (CNNs), Recurrent Neural Networks
(RNNs), and Connectionist Temporal Classification (CTC) loss is designed to
handle the challenges posed by reading captcha and is shown in Figure 1.

Figure 1. Proposed Model

The components of the proposed model with their functionalities are stated
below:

  * •

Convolutional Neural Networks (CNNs): The Convolutional Neural Network (CNN)
layers used for feature extraction from captcha images play a crucial role in
capturing relevant spatial and hierarchical features. The following outline
provides a common structure for the CNN layers used in OCR models for captcha
recognition.

  * •

Recurrent Neural Networks (RNNs): They play a crucial role in OCR models for
captcha recognition by capturing sequential dependencies in the features
extracted by Convolutional Neural Networks (CNNs). captcha often involve
characters arranged in a specific order, where the recognition of one
character depends on the context provided by the surrounding characters. The
output from the last dense layer of the CNN, representing the features
extracted from the spatial dimensions of the captcha image, is used as input
to the RNN. The RNNS then process the RNNs process the input sequence
sequentially, processing one feature at a time while maintaining hidden states
that capture information from previous steps. The sequential processing
capability of RNNs makes them suitable for tasks where the order of input
matters, as is the case in reading captcha. The hidden states of the RNN at
each time step capture relevant information about the sequential context.
These hidden states are updated and propagated through the network as each
character is processed. The output sequence from the RNN represents the
sequential encoding of features based on the context of the characters in the
captcha. Each element of the output sequence corresponds to the hidden state
at a specific time step.

  * •

Connectionist Temporal Classification (CTC) Loss: It is a technique used in
machine learning, particularly in the context of sequence-to-sequence tasks,
where the input and output sequences can have variable lengths. It is commonly
used in tasks like speech recognition, handwriting recognition, and machine
translation.
In scenarios where the input and output sequences have different lengths or
are not aligned one-to-one, traditional loss functions like mean squared error
or cross-entropy may not be suitable. This misalignment issue arises in tasks
where the length of the input sequence does not necessarily correspond to the
length of the target sequence.
CTC loss addresses this problem by allowing the model to learn alignments
between the input and output sequences automatically. It was initially
developed for speech recognition but has since been applied to various
sequence-to-sequence tasks.
The Output sequence of RNN is then passed to the CTC Loss layer, which allows
the model to produce a new output sequence that aligns with the ground truth
sequence. The CTC layer facilitates the learning of the temporal relationships
between characters and helps handle variable-length sequences. During
training, the OCR model is optimized using the CTC loss, which calculates the
difference between the predicted sequence and the ground truth sequence. The
CTC loss guides the model to learn the alignment between the input sequence
and the corresponding output sequence, handling the variability in captcha
lengths.

During inference, CTC decoding algorithms are applied to convert the model’s
output sequence into the final recognized text. Decoding methods handle
variable-length sequences and provide the predicted text by considering the
probabilities of different character alignments.

##  3\. Experiments and results

The choice and quality of the dataset play a crucial role in the training and
evaluation of an OCR model for a captcha. A well-constructed dataset that
accurately represents the challenges posed by captcha can significantly impact
the model’s performance. We used the text captcha image dataset which is
publicly available on Kaggle [Dataset, 2024]. The dataset contains 1040
samples with Text Captcha Images and the label for each sample is the name of
the file which is the text present inside the captcha, The captcha in the
dataset vary in terms of distortion, font type, background noise, and
character complexity. The character set in the captcha reflects the expected
diversity of characters in real-world scenarios. Alphanumeric characters (both
uppercase and lowercase) and special symbols commonly used in captcha are
included. There are 19 unique characters found in the dataset namely : [’2’,
’3’, ’4’, ’5’, ’6’, ’7’, ’8’, ’b’, ’c’, ’d’, ’e’, ’f’, ’g’, ’m’, ’n’, ’p’,
’w’, ’x’, ’y’]. To simulate real-world scenarios, the dataset includes a
captcha with variable-length sequences. captcha with different numbers of
characters challenges the model to handle sequences of varying lengths. The
sample data from the dataset is shown in Figure 2.

Figure 2. Sample data Figure 3. Data Pre-processing Steps

The data pre-processing involves tasks as shown in Figure 3. Captcha images
may vary in quality. Pre-processing steps such as contrast adjustment,
normalization, and noise reduction enhance the overall image quality. Image
quality enhancement also ensures uniformity in image quality, helps the model
generalize better. Normalizing pixel values to a specific range (e.g., [0, 1])
ensures consistent input to the model. While the Standardization of pixel
values (mean subtraction and scaling) may be applied to improve convergence
during training. If the dataset includes captcha with characters merged or
overlapping, a preprocessing step for character segmentation is crucial.
Proper segmentation ensures that individual characters are presented to the
OCR model for accurate recognition. Data Augmentation techniques such as
rotation, scaling, translation, and flipping are applied to artificially
increase the diversity of the dataset. Augmentation helps the model generalize
better and become more robust to variations in real-world captcha images. The
ground truth labels, representing the characters in the captcha, undergo one-
hot encoding. This encoding converts categorical labels into a binary matrix,
facilitating the training of the model with categorical cross-entropy loss.
The Balancing Classes step ensures that the dataset has a balanced
representation of different captcha types and character classes. Imbalanced
datasets can lead to biased model training, affecting the model’s performance
on under-represented classes.
After getting the preprocessed encoded balanced dataset, it is divided into
training, validation and testing dataset. The majority of the dataset is
allocated for training the OCR model. Typically, around 70-80% of the dataset
is used for training to allow the model to learn the underlying patterns in
the captcha. A smaller portion (e.g., 10-15%) is set aside for model
validation during training. The validation set helps in monitoring the model’s
performance on unseen data and prevents overfitting. A separate test set, not
seen by the model during training, is reserved for evaluating the model’s
generalization. The test set contains a diverse set of captcha, ensuring a
comprehensive assessment of the model’s performance.
The training process for an OCR model designed for reading captcha involves
several critical components, each of which contributes to the model’s
performance and generalization ability. The training process, includes
training hyper-parameters, optimizer choice, and data augmentation techniques
(image — rotation, flipping, translation, zooming, shear, brightness
adjustments, and contrast variations ). The training Steps are as shown in
figure 4.

Figure 4. Training of model

The model has been implemented in Python with libraries like TensorFlow,
Keras, OpenCV, NumPy, and nltk. The datasets available in TensorFlow_datasets
are used to load and preprocess the dataset during the training and validation
phase. The training visualization is down with the Tensorboard. These tools
provide visualization and monitoring of the training process, including
metrics, loss curves, and model architecture visualization.

Figure 5. Loss Curve

The training and validation CTC loss over epochs is shown in the graph (Figure
5). A decreasing loss indicates that the model is learning and converging.
Analyzing the results of an OCR project involves evaluating the performance of
the trained model on various metrics, understanding the challenges faced
during the project, and identifying potential improvements for the model. To
evaluate the model, we use two level accuracy metrics, namely, character level
and word level accuracy. Character level accuracy gives the accuracy of
correctly predicted characters by the model, which helps to understand how
well the model recognizes individual characters. Word level accuracy is the
accuracy of correctly predicted words by the model. Word level accuracy is a
more practical evaluation metric since OCR often recognizes words.

Type | Proposed Model Accuracy (%) | State-of-art Model Accuracy (%)
---|---|---
Character Level Accuracy | 99.80 | 98.94 [Nouri Z, 2020] , 99.70 [Jing Wang, 2019], 51.1 [Bursztein E, 2014], 51.6 [Ye G, 2020]
Word Level Accuracy | 95 | 94 [Suke Kong, 2023]

Table 1. Evaluation metric

The proposed model gives 99.80% character level accuracy, while 95% word level
accuracy. The accuracy of the proposed model is compared with the state-of-
the-art models and proves to be effective. The system has limitations due to
factors like, the limited unrepresentative training data, noisy and distorted
images, variable length sequences (though padding is used in RNN), hyper-
parameter values (learning rate, batch size, etc.) and model complexity.

##  4\. Conclusion

The OCR model demonstrated proficiency in recognizing captcha, showcasing its
ability to decipher distorted, variable-length, and complex patterns commonly
used in security measures on websites. The combination of Convolutional Neural
Networks (CNNs) for spatial feature extraction and Recurrent Neural Networks
(RNNs) for capturing sequential dependencies proved highly effective in
handling the unique challenges posed by captcha.
The use of Connectionist Temporal Classification (CTC) loss facilitated
training on variable-length sequences, allowing the model to adapt to
different captcha lengths without requiring explicit alignment information.
Data augmentation techniques, including rotation, flipping, translation, and
zooming, significantly enhanced the model’s robustness by exposing it to a
diverse range of captcha variations during training. Careful tuning of
hyperparameters, such as learning rate, batch size, and network architecture,
played a pivotal role in achieving optimal model performance during training.
The OCR model’s ability to decipher captcha contributes directly to addressing
the need for an effective solution against automated systems attempting to
bypass security measures. The proposed system is developed which can handle
distorted text, complex character combinations and detect dependencies in
character sequences. The model’s successful recognition of diverse captcha
suggests promising generalization to unseen variations, contributing to its
potential application in real-world scenarios. The findings provide a strong
foundation for further development, including exploration of attention
mechanisms, ensemble methods, and real-time application adaptation.
The possible future research and improvements to enhance the OCR model for
reading captcha is to investigate the use of GANs to generate synthetic
captcha images for augmenting the training dataset. GANs can create realistic
variations, providing additional diversity to improve the model’s
generalization. Explore transfer learning by leveraging pre-trained models on
large image datasets. Fine-tuning the OCR model on captcha data after pre-
training on general image datasets may improve convergence and performance.
Analyzing OCR model results is an iterative process that involves a
combination of quantitative metrics, qualitative assessments, and a deep
understanding of the challenges faced. Continuous refinement based on insights
gained from analysis is key to building a robust OCR system. Regularly
revisiting and updating the model based on new data and user feedback is
essential for long-term success.
Explore multimodal approaches by integrating additional information, such as
audio cues or contextual information, to enhance captcha recognition accuracy,
especially in scenarios where visual information alone may be insufficient. By
pursuing these research directions, the OCR model for reading captcha can be
enhanced in terms of accuracy, robustness, and adaptability to diverse
challenges in real-world applications.

## References

  * [Rajan A, 2021] Ranjan, A., Behera, V. N. J., & Reza, M. (2021). Ocr using computer vision and machine learning. Machine Learning Algorithms for Industrial Applications, 83-105.
  * [Fujitake M, 2024] Fujitake, M. (2024). DTrOCR: Decoder-only Transformer for Optical Character Recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 8025-8035).
  * [Nasim H, 2024] Nasim, H. I., Printia, F. J., Hasan, M., Rashid, R., Chowdhury, I. J., Mondal, J. J., … & Noor, J. (2024). Fog-Resilient Bangla Car Plate Recognition Using Dark Channel Prior and YOLO. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 1110-1119).
  * [Smink M, 2024] Smink, M., Liu, H., Döpfer, D., & Lee, Y. J. (2024). Computer Vision on the Edge: Individual Cattle Identification in Real-Time With ReadMyCow System. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 7056-7065).
  * [Jobin K, 2024] Jobin, K. V., Mishra, A., & Jawahar, C. V. (2024). Semantic Labels-Aware Transformer Model for Searching Over a Large Collection of Lecture-Slides. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 6016-6025).
  * [Algwil A, 2023] Algwil, A. M. (2023). A SURVEY ON CAPTCHA: ORIGIN, APPLICATIONS AND CLASSIFICATION. Journal of Basic Sciences, 36(1), 1-37.
  * [Hasan W, 2016] Hasan, W. K. A. (2016). A survey of current research on captcha. Int. J. Comput. Sci. Eng. Surv.(IJCSES), 7(3), 141-157.
  * [Choudhary S, 2013] Choudhary, S., Saroha, R., Dahiya, Y., & Choudhary, S. (2013). Understanding CAPTCHA: text and audio based CAPTCHA with its applications. International Journal of Advanced Research in Computer Science and Software Engineering, 3(6).
  * [Sabeti V, 2024] Sabeti, V., & Samiei, M. (2024). A comprehensive evaluation of deep learning based steganalysis performance in detecting spatial methods. Biannual Journal Monadi for Cyberspace Security (AFTA), 12(2), 42-50.
  * [Kuznetsov O, 2024] Kuznetsov, O., Frontoni, E., Romeo, L., & Rosati, R. (2024). Enhancing copy-move forgery detection through a novel CNN architecture and comprehensive dataset analysis. Multimedia Tools and Applications, 1-35.
  * [Billion P, 2024] Billion Polak, P., Prusa, J. D., & Khoshgoftaar, T. M. (2024). Low-shot learning and class imbalance: a survey. Journal of Big Data, 11
  * [Hussain R, 2016] Hussain, R.; Kumar, K.; Gao, H.; Khan, I. Recognition of merged characters in text based CAPTCHAs. In Proceedings of the 2016 3rd International Conference on Computing for Sustainable Global Development (INDIACom), New Delhi, India, 16–18 March 2016; pp. 3917–3921.
  * [Ahn V, 2003] von Ahn, L.; Blum, M.; Hopper, N.J.; Langford, J. CAPTCHA: Using Hard AI Problems for Security. In Proceedings of the Advances in Cryptology—EUROCRYPT 2003, Warsaw, Poland, 4–8 May 2003; Biham, E., Ed.; Springer: Berlin/Heidelberg, Germany, 2003; pp. 294–311.
  * [Blum M, 2004] von Ahn, L.; Blum, M.; Langford, J. Telling Humans and Computers Apart Automatically. Commun. ACM 2004, 47, 56–60.
  * [Ye G, 2018] Ye, G.; Tang, Z.; Fang, D.; Zhu, Z.; Feng, Y.; Xu, P.; Chen, X.; Wang, Z. Yet Another Text Captcha Solver: A Generative Adversarial Network Based Approach. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, Toronto, ON, Canada, 15–19 October 2018; Association for Computing Machinery: New York, NY, USA, 2018; pp. 332–348.
  * [Gao H, 2013] Gao, H.; Wang, W.; Qi, J.; Wang, X.; Liu, X.; Yan, J. The Robustness of Hollow CAPTCHAs. In Proceedings of the 2013 ACM SIGSAC Conference on Computer & Communications Security, Berlin, Germany, 4–8 November 2013; Association for Computing Machinery: New York, NY, USA, 2013; pp. 1075–1086.
  * [Gao H, 2017] Gao, H.; Tang, M.; Liu, Y.; Zhang, P.; Liu, X. Research on the Security of Microsoft’s Two-Layer Captcha. IEEE Trans. Inf. Forensics Secur. 2017, 12.
  * [Jin Z, 2020] Jin, Z.; Iqbal, M.Z.; Bobkov, D.; Zou, W.; Li, X.; Steinbach, E. A Flexible Deep CNN Framework for Image Restoration. IEEE Trans. Multimedia 2020, 22, 1055–1068.
  * [Malik S, 2019] Malik, S.; Soundararajan, R. Llrnet: A Multiscale Subband Learning Approach for Low Light Image Restoration. In Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), Taipei, Taiwan, 22–25 September 2019; pp. 779–783.
  * [Jin Z, 2020] Jin, Z.; Iqbal, M.Z.; Bobkov, D.; Zou, W.; Li, X.; Steinbach, E. A Flexible Deep CNN Framework for Image Restoration. IEEE Trans. Multimedia 2020, 22, 1055–1068.
  * [Dong W, 2019] Dong, W.; Wang, P.; Yin, W.; Shi, G.; Wu, F.; Lu, X. Denoising Prior Driven Deep Neural Network for Image Restoration. IEEE Trans. Pattern Anal. Mach. Intell. 2019, 41, 2305–2318.
  * [Redmon J, 2017] Redmon, J.; Farhadi, A. YOLO9000: Better, Faster, Stronger. In Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 21–26 July 2017; pp. 6517–6525.
  * [Ren S, 2017] Ren, S.; He, K.; Girshick, R.; Sun, J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Trans. Pattern Anal. Mach. Intell. 2017, 39, 1137–1149.
  * [Liu Y, 2018] Liu, Y. An Improved Faster R-CNN for Object Detection. In Proceedings of the 2018 11th International Symposium on Computational Intelligence and Design (ISCID), Hangzhou, China, 8–9 December 2018; Volume 2, pp. 119–123.
  * [Abdussalam A, 2019] Abdussalam, A.; Sun, S.; Fu, M.; Sun, H.; Khan, I. License Plate Segmentation Method Using Deep Learning Techniques. In Proceedings of the Signal and Information Processing, Networking and Computers; Sun, S., Ed.; Springer: Singapore, 2019; pp. 58–65.
  * [Abdussalam A, 2018] Abdussalam, A.; Sun, S.; Fu, M.; Ullah, Y.; Ali, S. Robust Model for Chinese License Plate Character Recognition Using Deep Learning Techniques. In CSPS 2018: Communications, Signal Processing, and Systems; Liang, Q., Liu, X., Na, Z., Wang, W., Mu, J., Zhang, B., Eds.; Springer: Singapore, 2020; Volume 517, pp. 121–127.
  * [Krizhevsky A, 2017] Krizhevsky, A.; Sutskever, I.; Hinton, G.E. ImageNet Classification with Deep Convolutional Neural Networks. Commun. ACM 2017, 60, 84–90.
  * [Jmour N, 2018] Jmour, N.; Zayen, S.; Abdelkrim, A. Convolutional neural networks for image classification. In Proceedings of the 2018 International Conference on Advanced Systems and Electric Technologies (IC-ASET), Hammamet, Tunisia, 22–25 March 2018; pp. 397–402.
  * [Sakkatos P, 2014] Sakkatos, P.; Theerayut, W.; Nuttapol, V.; Surapong, P. Analysis of text-based CAPTCHA images using Template Matching Correlation technique. In Proceedings of the 4th Joint International Conference on Information and Communication Technology, Electronic and Electrical Engineering (JICTEE), Chiang Rai, Thailand, 5–8 March 2014; pp. 1–5.
  * [Wu C, 2005] Wu, C.; On, L.C.; Weng, C.H.; Kuan, T.S.; Ng, K. A Macao license plate recognition system. In Proceedings of the 2005 International Conference on Machine Learning and Cybernetics, Guangzhou, China, 18–21 August 2005; Volume 7, pp. 4506–4510.
  * [Baten R A, 2014] Baten, R.A.; Omair, Z.; Sikder, U. Bangla license plate reader for metropolitan cities of Bangladesh using template matching. In Proceedings of the 8th International Conference on Electrical and Computer Engineering, Dhaka, Bangladesh, 20–22 December 2014; pp. 776–779.
  * [Chen J, 2019] Chen, J.; Luo, X.; Liu, Y.; Wang, J.; Ma, Y. Selective Learning Confusion Class for Text-Based CAPTCHA Recognition. IEEE Access 2019, 7, 22246–22259.
  * [hu Y, 2018] Hu, Y.; Chen, L.; Cheng, J. A CAPTCHA recognition technology based on deep learning. In Proceedings of the 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA), Wuhan, China, 31 May–2 June 2018; pp. 617–620.
  * [Stark F, 2015] Stark, F.; Hazırbaş, C.; Triebel, R.; Cremers, D. CAPTCHA Recognition with Active Deep Learning. In Proceedings of the German Conference on Pattern Recognition Workshop, Aachen, Germany, 7–10 October 2015.
  * [Qing K, 2017] Qing, K.; Zhang, R. A Multi-Label Neural Network Approach to Solving Connected CAPTCHAs. In Proceedings of the 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Kyoto, Japan, 9–15 November 2017; Volume 1, pp. 1313–1317.
  * [Zi Y, 2020] Zi, Y.; Gao, H.; Cheng, Z.; Liu, Y. An End-to-End Attack on Text CAPTCHAs. IEEE Trans. Inf. Forensics Secur. 2020, 15, 753–766.
  * [Wang P, 2020] Wang, P.; Gao, H.; Shi, Z.; Yuan, Z.; Hu, J. Simple and Easy: Transfer Learning-Based Attacks to Text CAPTCHA. IEEE Access 2020, 8, 59044–59058.
  * [Dataset, 2024] https://www.kaggle.com/datasets/fournierp/captcha-version-2-images
  * [Jing Wang, 2019] Jing Wang, Jiaohua Qin, Xuyu Xiang, Yun Tan, Nan Pan. CAPTCHA recognition based on deep convolutional neural network[J]. Mathematical Biosciences and Engineering, 2019, 16(5): 5851-5861. doi: 10.3934/mbe.2019292
  * [Suke Kong, 2023] Suke Kong and Yu-Chung Chang, A CAPTCHA Recognition Algorithm Based on Character Segmentation and Random Forest, J. Phys.: Conf. Ser. 2504 012036, 2023
  * [Nouri Z, 2020] Nouri, Zahra and Rezaei, Mahdi, Deep-CAPTCHA: A Deep Learning Based CAPTCHA Solver for Vulnerability Assessment (June 15, 2020). Available at SSRN: https://ssrn.com/abstract=3633354 or http://dx.doi.org/10.2139/ssrn.3633354
  * [Yusuf M O, 2023] Yusuf, M.O., Srivastava, D., Singh, D. et al. Multiview deep learning-based attack to break text-CAPTCHAs. Int. J. Mach. Learn. & Cyber. 14, 959–972 (2023). https://doi.org/10.1007/s13042-022-01675-8
  * [Bursztein E, 2014] Bursztein E, Moscicki A, Fabry C, et al (2014) Easy does it: More usable captchas. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp 2637–2646 https://link.springer.com/article/10.1007/s13042-022-01675-8#ref-CR5
  * [Ye G, 2020] Ye G, Tang Z, Fang D et al (2020) Using generative adversarial networks to break and protect text captchas. ACM Trans Privacy Secur (TOPS) 23(2):1–29