∎

11institutetext: Eric Okyere 22institutetext: Department of Basic Sciences,
University of Health and Allied Sciences, PMB 31, Ho, Volta Region, Ghana
Tel.: +233-508389028
22email<EMAIL_ADDRESS>33institutetext: F. T. Oduro, S. K. Amponsah, I.
K. Dontwi44institutetext: Department of Mathematics, Kwame Nkrumah University
of Science and Technology, Kumasi, Ghana

# Fractional Order Optimal Control Model For Malaria Infection

E. Okyere  F. T. Oduro  S. K. Amponsah  I. K. Dontwi

(July, 2016)

###### Abstract

We propose and study an optimal control model for malaria infection described
by system of fractional differential equations. The model is formulated in
terms of the left and right Caputo fractional derivatives. We determine the
necessary conditions for the optimality of the controlled dynamical system.
The forward-backward sweep method with generalized euler scheme is applied to
numerically compute the solutions of the optimality system.

Keywords: Malaria Infection, Optimal Control Model, Fractional Differential
Equations.

##  1 Introduction

Mathematical modeling of malaria infection using differential equations with
integer order and non-integer derivatives have been considered by many authors
(see, e.g, Pinto and Machado, 2013, Okyere et al., 2016, Ngonghala et al.,
2012, Ross, 1911, Abdullahi et al., 2013, Keegan and Dushoff, 2013, Chitnis et
al., 2006). A comprehensive survey on malaria infection modeling has been
captured in the work by Mandal et al. (2011). Infectious diseases prevention
and control strategies are very important factors and if properly addressed
can go a long way in reducing disease infections in a population. Optimal
control theory is one of the useful and efficient modeling tools that can help
in the modeling framework to better understand the dynamics of the disease and
suggest possible prevention and control measures. In recent times, malaria
modeling formulation based on optimal control theory with time dependent
controls has attracted much attention (see, e.g, Okosun et al., 2013, Otieno
et al., 2016, Athithan and Ghosh, 2015, Augsto et al., 2012, Mwanga et al.,
2014, Silva and Torres, 2013, Okosun and Makinde, 2014, 2011).

The paper by Lashari and Zaman (2012) proposed and numerically analyzed an
optimal control model to investigate the effectiveness of time dependent
controls on vector-borne diseases. From the numerical results, they (Lashari
and Zaman, 2012) observed that, optimal control strategies are effective in
reducing the infected populations. Lashari et al. (2012) proposed a time
dependent multiple controls to study malaria infection. The authors in
(Blayneh et al., 2009) formulated a time dependent treatment and prevention
optimal control model to study vector-borne diseases. Okosun and Makinde
(2013) presented a controlled dynamical malaria model based on non-linear
incidence. Kim et al. (2013) developed a dynamic control problem to study
malaria transmission using two time dependent control functions.

The Ebola infection that claimed many lives in Africa has also been considered
with optimal control theory (see, e.g, Rachah and Torres, 2016, Bonyah et al.,
2016, Grigorieva and Khailov, 2015). Compartmental models for infectious
diseases such as TB, HIV and Cholera with time dependent controls have been
studied (see, e.g, Moualeu et al., 2015, Magombedze et al., 2011, Bakhtiar,
2016, Choi et al., 2015).

Fractional order derivative has an important property called memory effect and
because of that, the theory and application of fractional calculus have widely
been used to model dynamical processes in the field of science, engineering
and many others (see, e.g, Petras, 2011, Podlubny, 1999). Agrawal (2004)
developed general optimal control problems that are driven by Riemann-
Liouville fractional derivatives. In another research by the same author
(Agrawal, 2008a), he formulated similar optimal control problems with Caputo
derivatives and developed a stable numerical scheme for the mathematical
model.

Several researchers have developed and analyzed optimal control problems
whereby the controlled dynamic problems are characterised by Riemann-Liouville
derivatives (see, e.g, Frederico and Torres, 2008, Tricaud and Chen, 2010,
Agrawal et al., 2010, Agrawal and Baleanu, 2007). An enzyme kinetic optimal
control problem was proposed and numerically studied by Basir et al. (2015).
The authors in (Ding et al., 2012) studeid HIV-Immune System model using
fractional optimal control. They deduced the necessary conditions for the
optimality system for a general control problem with Caputo derivatives using
the concept of calculus of variation and fractional integration by parts
formula. Necessary optimality conditions have been derived for an optimal
control problem whereby the state model had both first order derivative and
non-integer derivative (Pooseh et al., 2014).

It has been demonstrated in literature that optimal control problems are
appropriate for biological systems and other processes that follow non-linear
behavior. Therefore modeling infectious diseases using fractional derivatives
together with optimal control theory is a huge advantage. Okyere et al. (2016)
extended the malaria infection model by Tumwiine et al. (2007a, b) to include
fractional derivatives. They (Okyere et al., 2016) investigated asymptotic
analysis of the model equilibria and solved the dynamical problem with the
numerical scheme developed by Diethelm et al. (2004, 2002). In this work, we
propose a fractional optimal control model for malaria infection and
numerically simulate the solutions of the constructed control problem. We
extend the malaria epidemic model studied by the authors in (Tumwiine et al.,
2007a, b) to become an optimal control problem of which the state and costate
equations are characterised by differential equations of fractional orders.
Our model formulation is different from previous works on malaria disease
modeling since we have incorporated fractional order derivatives into the
controlled dynamical system.

The paper is organized as follows. Section 2 deals with brief review on the
mathematical formulation of fractional optimal control problem. We formulate
the controlled malaria model and deduce the necessary conditions for the
optimality of the model problem in section 3. Section 4 deals with numerical
solutions and discussions of simulations results. We then conclude the paper
in section 5.

##  2 Optimal Control Problem With Fractional Derivatives

This section deals with brief review on the mathematical formulation of
fractional optimal control model. We will then apply the modeling concept to
formulate our new malaria model in section 3. There are many types of
fractional derivatives but the most used ones in mathematical modeling and
engineering applications are Riemann-Liouville derivative and Caputo
derivative (see, e.g, Petras, 2011, Podlubny, 1999). For this work, we develop
the optimization model with Caputo fractional derivatives..

###### Definition 1

(Podlubny, 1999, Pooseh et al., 2014) The left Caputo fractional derivative is
defined as

${}_{a}^{c}D^{\alpha}_{t}w(t)=\frac{1}{\Gamma(p-\alpha)}\int_{a}^{t}\\!\frac{w^{(p)}(x)}{(t-x)^{\alpha+1-p}}\,\mathrm{d}x.$

for $p-1<\alpha<p.$

###### Definition 2

(Podlubny, 1999, Pooseh et al., 2014) The right Caputo fractional derivative
is defined as

${}_{t}^{c}D^{\alpha}_{b}w(t)=\frac{(-1)^{p}}{\Gamma(p-\alpha)}\int_{t}^{b}\\!\frac{w^{(p)}(x)}{(x-t)^{\alpha+1-p}}\,\mathrm{d}x.$

for $p-1<\alpha<p,$

where $\alpha$ is the order of the Caputo derivatives.

In the works by Agrawal (2008a, b), he formulated an optimal control problem
with Caputo fractional derivatives as follows. Find an optimal control $u$
that minimise the objective functional or performance index:

$J(u)=\int_{0}^{1}\\!W(x,u,t)\,\mathrm{d}t.$ (1)

Subject to the fractional dynamic constraint

$_{0}^{c}D^{\alpha}_{t}x(t)=M(x,u,t)$ (2)

with initial condition

$x(0)=x_{o},$ (3)

where $W(x,u,t)$ and $M(x,u,t)$ are two arbitrary functions and $x(t)$ is the
state variable (Agrawal, 2004, 2008a). By using the method of lagrange
multipliers, calculus of variation and the integration by parts formula for
fractional order derivatives and also following a similar derivation by the
same author in (Agrawal, 2004), he obtained the necessary conditions or
equations (4-6) for the fractional order controlled problem as follows:

$\displaystyle_{0}^{c}D^{\alpha}_{t}x(t)$ $\displaystyle=M(x,u,t)$ (4)

$\displaystyle_{t}^{c}D^{\alpha}_{1}x(t)$ $\displaystyle=\frac{\partial
W}{\partial x}+\lambda\frac{\partial M}{\partial x}$ (5)

$\displaystyle\frac{\partial W}{\partial u}+\lambda\frac{\partial M}{\partial
u}$ $\displaystyle=0$ (6)

with

$\begin{cases}x(0)=x_{o}&\\\ \\\ \lambda(1)=0,\end{cases}$ (7)

where $\lambda$ is the co-state variable called lagrange multiplier.

##  3 Controlled Malaria Model With Fractional Derivatives

In this section, we will use the obtained results in the optimal control
problem formulation briefly reviewed in section 2 to deduce our necessary
conditions or equations for the optimality of the optimal control malaria
model. Our model formulation seeks to find an optimal control that would
minimise the infected human populations and the cost of implementing the
optimal control and prevention strategies. In this work, we consider three
time dependent controls namely treated bednets, treatment and insecticide
spray.

The main aim in our present formulation is to find optimal control $U$ that
minimise the objective functional:

$J(U)=\int_{0}^{b}\\!W(X,U,t)\,\mathrm{d}t.$ (8)

Subject to the fractional dynamic constraints

$_{0}^{c}D^{\alpha}_{t}X(t)=M(X,U,t)$ (9)

with initial condition

$X(0)=X_{o}$ (10)

where

$\displaystyle X(t)$ $\displaystyle=(S_{H},I_{H},R_{H},S_{V},I_{V})^{T}$
$\displaystyle X(0)$
$\displaystyle=(S_{H}(0),I_{H}(0),R_{H}(0),S_{V}(0),I_{V}(0))^{T}$
$\displaystyle U(t)$ $\displaystyle=(u_{1},u_{1},u_{3})^{T}$ $\displaystyle
W(X,U,t)$
$\displaystyle=AI_{H}+\frac{d_{1}u^{2}_{1}}{2}+\frac{d_{2}u^{2}_{2}}{2}+\frac{d_{3}u^{2}_{3}}{2}$

$\displaystyle
M(X,U,t)=\left[\begin{array}[]{c}\lambda^{\alpha}_{h}N_{H}-\frac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N_{H}}+\nu^{\alpha}I_{H}+\gamma^{\alpha}R_{H}-\mu^{\alpha}_{h}S_{H}\\\
\\\
\frac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N_{H}}-\nu^{\alpha}I_{H}-(r^{\alpha}+\rho^{\alpha}u_{2})I_{H}-\delta^{\alpha}I_{H}-\mu^{\alpha}_{h}I_{H}\\\
\\\
(r^{\alpha}+\rho^{\alpha}u_{2})I_{H}-(\gamma^{\alpha}+\mu^{\alpha}_{h})R_{H}\\\
\\\
(1-u_{3})\lambda^{\alpha}_{v}N_{V}-\frac{(1-u_{1})a^{\alpha}cS_{V}I_{H}}{N_{H}}-\mu^{\alpha}_{v}S_{V}-\eta^{\alpha}u_{3}S_{V}\\\
\\\
\frac{(1-u_{1})a^{\alpha}cS_{V}I_{H}}{N_{H}}-\mu^{\alpha}_{v}I_{V}-\eta^{\alpha}u_{3}I_{V}\end{array}\right]$
(20)

The variable $X(t)$ is the state vector and $U$ is the control vector. As in
Agrawal (2004, 2008b), when $\alpha=1$, the formulated malaria controlled
dynamical model becomes classical optimal control problem for the uncontrolled
malaria model proposed and analyzed by Tumwiine et al. (2007a, b). Also note
that, when $\alpha=1$, with control functions $u_{1}=u_{2}=u_{3}=0$, then the
dynamic constraint equation (9) with the initial condition (10) becomes the
malaria infection model developed by the authors in (Tumwiine et al., 2007a,
b)

The meaning of the model variables and time dependent optimal control
functions are defined in table 1. The definitions for the model parameters can
be found in table 2

Table 1: Description of model variables and control functions. Model variables and control functions | Description
---|---
$S_{H}$ | Susceptible human class
$I_{H}$ | Infected human class
$R_{H}$ | partially immune human class
$S_{V}$ | Susceptible mosquito class
$I_{V}$ | Infected mosquito class
$N_{H}$ | Total human population
$N_{V}$ | Total mosquito population
$u_{1}$ | The control on the use of treated bednets
$u_{2}$ | The control on treatment of infected individuals
$u_{3}$ | The control on the use of insecticide spray
Table 2: Description of model parameters. Model Parameters | Description
---|---
$\lambda_{h}$ | natural birth rate of humans
$\lambda_{v}$ | natural birth rate of mosquitoes
$\mu_{h}$ | natural death rate of humans
$\mu_{v}$ | natural death rate of mosquitoes
$a$ | average daily biting rate on human by a single mosquito
$b$ | proportion of bites on human that produce an infection
$\delta$ | death rate due to the disease
$c$ | probability that a mosquito becomes infectious
$\nu$ | recovery rate of human hosts from the disease
$\gamma$ | rate of loss of immunity in human hosts
$r$ | rate at which human hosts acquire immunity
$\rho$ | rate constant due to treatment
$\eta$ | rate constant due to insecticide spray
$A,\ d_{1},\ d_{2},\ d_{3}$ | weight constants

It is important to mention that, rigorous proofs on the derivation of
necessary conditions for the optimality of various fractional dynamical
systems can be found in literature (see, e.g, Pooseh et al., 2014, Ding et
al., 2012, Agrawal, 2004). Therefore in this paper, using the ideas by the
authors in (see, e.g, Agrawal, 2008a, b, Pooseh et al., 2014, Ding et al.,
2012, Agrawal, 2004), we state the necessary conditions (21-23) for our
optimization problem as follows:

$\displaystyle_{0}^{c}D^{\alpha}_{t}X(t)$ $\displaystyle=M(X,U,t)$ (21)

$\displaystyle_{t}^{c}D^{\alpha}_{b}\lambda(t)$ $\displaystyle=\frac{\partial
W}{\partial X}+\lambda^{T}\frac{\partial M}{\partial X}$ (22)

$\displaystyle\frac{\partial W}{\partial U}+\lambda^{T}\frac{\partial
M}{\partial U}$ $\displaystyle=0$ (23)

and

$\begin{cases}X(0)=X_{o}&\\\ \lambda(b)=0\end{cases}$ (24)

where
$\lambda(t)=(\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4},\lambda_{5})^{T}$
is the co-state vector.

Now using the compact form of the necessary conditions stated above, we obtain
the optimality system for the fractional optimal control malaria model in an
expanded form as follows:

$\begin{array}[]{lllll}{}_{0}^{c}D^{\alpha}_{t}S_{H}&=&\displaystyle{\lambda^{\alpha}_{h}N_{H}-\frac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N_{H}}+\nu^{\alpha}I_{H}+\gamma^{\alpha}R_{H}-\mu^{\alpha}_{h}S_{H}},\\\\[15.0pt]
{}_{0}^{c}D^{\alpha}_{t}I_{H}&=&\displaystyle{\frac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N_{H}}-\nu^{\alpha}I_{H}-(r^{\alpha}+\rho^{\alpha}u_{2})I_{H}-\delta^{\alpha}I_{H}-\mu^{\alpha}_{h}I_{H}},\\\\[15.0pt]
{}_{0}^{c}D^{\alpha}_{t}R_{H}&=&\displaystyle{(r^{\alpha}+\rho^{\alpha}u_{2})I_{H}-(\gamma^{\alpha}+\mu^{\alpha}_{h})R_{H}},\\\\[15.0pt]
{}_{0}^{c}D^{\alpha}_{t}S_{V}&=&\displaystyle{(1-u_{3})\lambda^{\alpha}_{v}N_{V}-\frac{(1-u_{1})a^{\alpha}cS_{V}I_{H}}{N_{H}}-\mu^{\alpha}_{v}S_{V}-\eta^{\alpha}u_{3}S_{V}},\\\\[15.0pt]
{}_{0}^{c}D^{\alpha}_{t}I_{V}&=&\displaystyle{\frac{(1-u_{1})a^{\alpha}cS_{V}I_{H}}{N_{H}}-\mu^{\alpha}_{v}I_{V}-\eta^{\alpha}u_{3}I_{V}},\end{array}$
(25)

$\begin{cases}{}_{t}^{c}D^{\alpha}_{b}\lambda_{1}=&\lambda_{1}\left[\lambda^{\alpha}_{h}-\dfrac{(1-u_{1})(I_{H}+R_{H})a^{\alpha}bI_{V}}{N^{2}_{H}}-\mu^{\alpha}_{h}\right]+\lambda_{2}\left[\dfrac{(1-u_{1})(I_{H}+R_{H})a^{\alpha}bI_{V}}{N^{2}_{H}}\right]\\\
\\\
&-(\lambda_{5}-\lambda_{4})\left[\dfrac{(1-u_{1}))a^{\alpha}cS_{H}I_{H}}{N^{2}_{H}}\right]\\\
\\\
{}_{t}^{c}D^{\alpha}_{b}\lambda_{2}=&A+\lambda_{1}\left[\lambda^{\alpha}_{h}+\dfrac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N^{2}_{H}}+\nu^{\alpha}\right]-(\lambda_{4}-\lambda_{5})\left[\dfrac{(1-u_{1})(S_{H}+R_{H})a^{\alpha}cS_{V}}{N^{2}_{H}}\right]\\\
\\\
&-\lambda_{2}\left[\dfrac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N^{2}_{H}}+\nu^{\alpha}+(r^{\alpha}+\rho^{\alpha}u_{2})+\delta^{\alpha}+\mu^{\alpha}_{h}\right]+\lambda_{3}\left[r^{\alpha}+\rho^{\alpha}u_{2}\right]\\\
\\\
{}_{t}^{c}D^{\alpha}_{b}\lambda_{3}=&\lambda_{1}\left[\lambda^{\alpha}_{h}+\dfrac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N^{2}_{H}}+\gamma^{\alpha}_{h}\right]-\lambda_{2}\left[\dfrac{(1-u_{1})a^{\alpha}bS_{H}I_{V}}{N^{2}_{H}}\right]-\lambda_{3}(\gamma^{\alpha}+\mu^{\alpha}_{h})\\\
\\\
&-(\lambda_{5}-\lambda_{4})\left[\dfrac{(1-u_{1})a^{\alpha}cS_{V}I_{H}}{N^{2}_{H}}\right]\\\
\\\
{}_{t}^{c}D^{\alpha}_{b}\lambda_{4}=&\lambda_{4}\left[(1-u_{3})\lambda^{\alpha}_{v}+\dfrac{(1-u_{1})a^{\alpha}cI_{H}}{N_{H}}-\mu^{\alpha}_{v}-\eta^{\alpha}u_{3}\right]+\lambda_{5}\left[\dfrac{(1-u_{1})a^{\alpha}cI_{H}}{N_{H}}\right]\\\
\\\
{}_{t}^{c}D^{\alpha}_{b}\lambda_{5}=&(\lambda_{2}-\lambda_{1})\left[\dfrac{(1-u_{1})a^{\alpha}bS_{H}}{N_{H}}\right]+\lambda_{4}\left[(1-u_{3})\lambda^{\alpha}_{v}\right]-\lambda_{5}\left[\mu^{\alpha}_{v}+\eta^{\alpha}u_{3}\right]\end{cases}$
(26)

$\begin{cases}u_{1}=\dfrac{(\lambda_{2}-\lambda_{1})\frac{a^{\alpha}bS_{H}I_{V}}{N_{H}}+(\lambda_{5}-\lambda_{4})\frac{a^{\alpha}cS_{V}I_{H}}{N_{H}}}{B_{1}}&\\\
\\\ u_{2}=\frac{(\lambda_{2}-\lambda_{3})r^{\alpha}_{o}I_{H}}{B_{2}}&\\\ \\\
u_{3}=\frac{\lambda_{4}(\lambda^{\alpha}_{v}N_{V}+\eta^{\alpha}S_{V})+\lambda_{5}\eta^{\alpha}I_{V}}{B_{3}}\end{cases}$
(27)

$\begin{cases}S_{H}(0)=S_{H_{O}},I_{H}(0)=I_{H_{O}},R_{H}(0)=R_{H_{O}},S_{V}(0)=S_{V_{O}},I_{V}(0)=I_{V_{O}}&\\\
\\\
\lambda_{1}(b)=0,\lambda_{2}(b)=0,\lambda_{3}(b)=0,\lambda_{4}(b)=0,\lambda_{5}(b)=0,\end{cases}$
(28)

##  4 Numerical Simulations and Discussion

The book by Lenhart and Workman (2007) described the numerical solutions of
optimal control models for biological systems. In their work, they constructed
the forward-backward sweep algorithm with Runge-Kutta method to solve the
optimality system of biological models. In this work, the forward-backward
algorithm with the generalized euler method is applied to numerically compute
the solutions of the optimality system. The generalized euler method is one of
the numerical schemes for fractional order models and has recently been
applied together with the forward-backward algorithm in a TB model (Sweilam
and Al-Mekhlafi, 2016).

The following initial conditions and model parameters values are used for the
numerical simulations: $S_{H}(0)=800,\ I_{H}(0)=200,\ R_{H}(0)=20,\
S_{V}(0)=1000,\ I_{V}(0)=500$, $A=100,\ d_{1}=70,\ d_{2}=130,\ d_{3}=40,\
a=0.29,\ b=0.75,\ c=0.75,$ $r=0.05\,\gamma=0.000017,\ \rho=0.7,\ \delta=0.02,\
\nu=0.0022,\ \lambda_{h}=0.0015875,\ \lambda_{v}=0.071,\ \mu_{h}=0.00004,\
\mu_{v}=0.1429,\ \eta=0.25$ and $\alpha=1,\ 0.99,\ 0.95,\ 0.90.$

We considered the following optimal control strategies for our numerical
illustrations: Treated bednets control only ($u_{1}\neq 0,\ u_{2}=0,\
u_{3}=0$), treatment control only ($u_{1}=0,\ u_{2}\neq 0,\ u_{3}=0$),
insecticide spray control only ($u_{1}=0,\ u_{2}=0,\ u_{3}\neq 0$), treated
bednets and treatment controls ($u_{1}\neq 0,\ u_{2}\neq 0,\ u_{3}=0$),
treated bednets and insecticide spray controls ($u_{1}\neq 0,\ u_{2}=0,\
u_{3}\neq 0$), treatment and insecticide spray controls ($u_{1}=0,\ u_{2}\neq
0,\ u_{3}\neq 0$), and all the three controls ($u_{1}\neq 0,\ u_{2}\neq 0,\
u_{3}\neq 0$).

From the plots generated, it is clear that reduction of infected human and
mosquito populations occurs when $\alpha=1$ (integer order) with time
dependent controls but that of non-integer orders ($\alpha=0.99,0.95,0.90$)
with time dependent controls are much more significant.

Figure 1: Numerical solutions of infected human host with treated bednets
control for $\alpha=1,0.99,0.95,0.90$

Figure 2: Numerical solutions of infected human host with treatment control
for $\alpha=1,0.99,0.95,0.90$

Figure 3: Numerical solutions of infected human host with insecticide spray
control for $\alpha=1,0.99,0.95,0.90$

Figure 4: Numerical solutions of infected human host with treated bednets and
treatment controls for $\alpha=1,0.99,0.95,0.90$

Figure 5: Numerical solutions of infected human host with treated bednets and
insecticide spray controls for $\alpha=1,0.99,0.95,0.90$

Figure 6: Numerical solutions of infected human host with treatment and
insecticide spray controls for $\alpha=1,0.99,0.95,0.90$

Figure 7: Numerical solutions of infected human host with all the three time
dependent optimal controls (treated bednets, treatment and insecticide spray)
for $\alpha=1,0.99,0.95,0.90$

Figure 8: Numerical solutions of infected mosquitoes with treated bednets
control for $\alpha=1,0.99,0.95,0.90$

Figure 9: Numerical solutions of infected mosquitoes with treatment control
for $\alpha=1,0.99,0.95,0.90$

Figure 10: Numerical solutions of infected mosquitoes with insecticide spray
control for $\alpha=1,0.99,0.95,0.90$

Figure 11: Numerical solutions of infected mosquitoes with treated bednets and
treatment controls for $\alpha=1,0.99,0.95,0.90$

Figure 12: Numerical solutions of infected mosquitoes with treated bednets and
insecticide spray controls for $\alpha=1,0.99,0.95,0.90$

Figure 13: Numerical solutions of infected mosquitoes with treatment and
insecticide spray controls for $\alpha=1,0.99,0.95,0.90$

Figure 14: Numerical solutions of infected mosquitoes with all the three time
dependent controls (treated bednets, treatment and insecticide spray) for
$\alpha=1,0.99,0.95,0.90$

Figure 15: Optimal control function $u_{1}$ for $\alpha=1,0.99,0.95,0.90$

Figure 16: Optimal control function $u_{2}$ for $\alpha=1,0.99,0.95,0.90$

Figure 17: Optimal control function $u_{3}$ for $\alpha=1,0.99,0.95,0.90$

Figure 18: Optimal control functions $u_{1},\ u_{2}$ and $u_{3}$ for
$\alpha=1,0.99,0.95,0.90$

##  5 Conclusion

This work considered an optimal control epidemiological model for malaria
infection formulated in the sense of Caputo derivatives. The optimality of the
fractional optimal control problem was solved using the forward-backward sweep
method and the generalized euler method. We have numerically shown that when
any of the time dependent controls or any combination of them are applied, the
infected populations for both human and mosquitoes are reduced. Furthermore,
when all the time dependent controls are used in the simulations, the
reductions in the infected populations are more than any other control and
prevention strategies. Our numerical solutions for the optimal control problem
with fractional orders are much better than the integer order.

## References

  * Abdullahi et al. (2013) M. B. Abdullahi, Y. A. Hasan, and F. A. Abdullah.  A mathematical model of malaria and the effectiveness of drugs.  _Applied Mathematical sciences_ , 7:3079–3095, 2013.
  * Agrawal (2004) O. P. Agrawal.  A general formulation and solution scheme for fractional optimal control problems.  _Nonlinear Dynamics_ , 38:323–337, 2004.
  * Agrawal (2008a) O. P. Agrawal.  A formulation and numerical scheme for fractional optimal control problems.  _Journal of Vibration and Control_ , 14:1291–1299, 2008a.
  * Agrawal (2008b) O. P. Agrawal.  A quadratic numerical scheme for fractional optimal control problems.  _Journal of Dynamic Ssystems, Measurement, and Control_ , 130, 2008b.
  * Agrawal and Baleanu (2007) O. P. Agrawal and D. Baleanu.  A hamiltonian formulation and a direct numerical scheme for fractional optimal control problems.  _Journal of Vibration and Control_ , 13:1269–1281, 2007\.
  * Agrawal et al. (2010) O. P. Agrawal, O. Defterli, and D. Baleanu.  Fractional optimal control problems with several state and control variables,.  _Journal of Vibration and Control_ , 16:1967–1976, 2010\.
  * Athithan and Ghosh (2015) S. Athithan and M. Ghosh.  Stability analysis and optimal control of a malaria model with larvivorous fish as biological control agent.  _Appl. Math. Inf. Sci._ , 9:1893–1913, 2015.
  * Augsto et al. (2012) F. B. Augsto, N. Marcus, and K. O. Okosun.  Application of optimal control to the epidemiology of malaria.  _Electronic Journal of DifferentialEquations_ , 2012:1–22, 2012.
  * Bakhtiar (2016) T. Bakhtiar.  Optimal intervention strategies for cholera outbreak by education and chlorination.  _Earth and Environmental Science_ , 31, doi:10.1088/1755-1315/31/1/012022, 2016.
  * Basir et al. (2015) F. A. Basir, A. M. Elaiw, D. Kesh, and P. K. Roy.  Optimal control of a fractional-order enzyme kinetic model.  _Control and Cybernetics_ , 44:1–18, 2015.
  * Blayneh et al. (2009) K. Blayneh, Y. Cao, and H.-D. Kwon.  Optimal control of vector-borne diseases: Treatment and prevention.  _Discrete and continuous dynamical systems series B_ , 11:587–611, 2009.
  * Bonyah et al. (2016) E. Bonyah, K. Badu, and S. K. Asiedu-Addo.  Optimal control application to an ebola model.  _Asian Pacific Journal of Tropical Biomedicine_ , 6(4):283–289, 2016.
  * Chitnis et al. (2006) N. Chitnis, J. M. Cushing, and J. M. Hyman.  Bifurcation analysis of a mathematical model for malaria transmission.  _SIAM J. Appl. Math_ , 67:24–45, 2006.
  * Choi et al. (2015) S. Choi, E. Jung, and S-M. Lee.  Optimal intervention strategy for prevention tuberculosis using a smoking-tuberculosis model.  _Journal of Theoretical Biology_ , 380:256–270, 2015.
  * Diethelm et al. (2002) K. Diethelm, N. J. Ford, and A. D. Freed.  A predictor-corrector approach for the numerical solution of fractional differential equations.  _Nonlinear Dyamics_ , 29:3–22, 2002.
  * Diethelm et al. (2004) K. Diethelm, N. J. Ford, and A. D. Freed.  Detailed error analysis for a fractional adams method.  _Nonlinear Dyamics_ , 36:31–52, 2004.
  * Ding et al. (2012) Y. Ding, Z. Wang, and H. Ye.  Optimal control of a fractional-order hiv-immune system with memory.  _IEEE Trans. Contr. Sys. Techn._ , 13:763–769, 2012.
  * Frederico and Torres (2008) G. S. F. Frederico and D. F. M. Torres.  Fractional conservation laws in optimal control theory.  _Nonlinear Dynamics_ , 53:215–222, 2008.
  * Grigorieva and Khailov (2015) E. V. Grigorieva and E. N. Khailov.  Optimal intervention strategies for a seir control model of ebola epidemics.  _Mathematics_ , 3, doi:10.3390/math3040961:961–983, 2015\.
  * Keegan and Dushoff (2013) L. T. Keegan and J. Dushoff.  Population-level effects of clinical immunity to malaria.  _BMC Infectious Diseases 2013_ , 13:428, 2013.
  * Kim et al. (2013) B. N. Kim, K. Nah, C. Chu, S. U. Ryu, Y. H. Kang, and Y. Kim.  Optimal control strategy of plasmodium vivax malaria transmission in korea.  _Osong Public Health Res Perspect_ , 3:128–136, 2013.
  * Lashari and Zaman (2012) A. A. Lashari and G. Zaman.  Optimal control of a vector borne disease with horizontal transmission.  _Nonlinear Analysis: Real World Applications_ , 13:203–212, 2012.
  * Lashari et al. (2012) A. A. Lashari, S. Aly, K. Hattaf, G. Zaman, I. H. Jung, and Xue-Zhi. Li.  Presentation of malaria epidemics using multiple optimal controls.  _Journal of Applied Mathematics_ , 2012, Article ID 946504:17, 2012.
  * Lenhart and Workman (2007) S. Lenhart and J. T. Workman.  _Optimal control applied to biological models_.  London, New York: Chapman and Hall, 2007.
  * Magombedze et al. (2011) G. Magombedze, W. Gariraa, E. Mwenjeb, and C. P. Bhunua.  Optimal control for hiv-1 multi-drug therapy.  _International Journal of Computer Mathematics_ , 88:314–340, 2011.
  * Mandal et al. (2011) S. Mandal, R. R. Sarkar, and S. Sinha.  Mathematical models of malaria - a review.  _Malaria Journal_ , 10:202, 2011.
  * Moualeu et al. (2015) D. P. Moualeu, M. Weiser, R. Ehring, and P. Deuflhard.  Optimal control for a tuberculosis model with undetected cases in cameroon.  _Commun Nonlinear Sci Numer Simulatt_ , 20:986–1003, 2015\.
  * Mwanga et al. (2014) G. G. Mwanga, S. Aly, H. Haario, and B. K. Nannyonga.  Optimal control of malaria model with drug resistance in presence of parameter uncertainty.  _Applied Mathematical Sciences_ , 8:2701 – 2730, 2014.
  * Ngonghala et al. (2012) C. N. Ngonghala, G. A. Ngwa, and M. I. Teboh-Ewungkem.  Periodic oscillations and backward bifurcation in a model for the dynamics of malaria transmission.  _Mathematical Biosciences_ , 240:45–62, 2012.
  * Okosun and Makinde (2011) K. O. Okosun and O. D. Makinde.  Modelling the impact of drug resistance in malaria transmission and its optimal control analysis.  _International Journal of the Physical Sciences_ , 6:6479–6487, 2011.
  * Okosun and Makinde (2013) K. O. Okosun and O. D. Makinde.  Optimal control analysis of malaria in the presence of non-linear incidence rate.  _Appl. Comput. Math._ , 12:20–32, 2013.
  * Okosun and Makinde (2014) K. O. Okosun and O. D. Makinde.  A co-infection model of malaria and cholera diseases with optimal control.  _Mathematical Biosciences_ , 258:19–32, 2014.
  * Okosun et al. (2013) K. O. Okosun, O. Rachid, and N. Marcus.  Optimal control strategies and cost-effectiveness analysis of malaria model.  _BioSystems_ , 111:83–101, 2013.
  * Okyere et al. (2016) E. Okyere, F. T. Oduro, S. K. Amponsah, I. K. Dontwi, and N. K. Frempong.  Fractional order malaria model with temporary immunity.  _arXiv:1603.06416v1_ , 2016.
  * Otieno et al. (2016) G. Otieno, J. K. Koske, and J. M. Mutiso.  Cost effectiveness analysis of optimal malaria control strategies in kenya.  _Mathematics doi:10.3390/math4010014_ , 4, 2016.
  * Petras (2011) I. Petras.  _Fractional-Order Nonlinear Systems: Modeling, Analysis and Simulation_.  Springer, 2011.
  * Pinto and Machado (2013) C. M. A. Pinto and J. A. T. Machado.  Fractional model for malaria transmission under control strategies.  _Computers and Mathematics with Applications._ , 66:908–916, 2013.
  * Podlubny (1999) I. Podlubny.  _Fractional Differential Equations_.  Academic Press, New York, 1999.
  * Pooseh et al. (2014) S. Pooseh, R. Almeida, and D. F. M. Torres.  Fractional order optimal control problems with free terminal time.  _Journal of industrial and management optimization_ , 10:363–381, 2014.
  * Rachah and Torres (2016) A. Rachah and D. F. M. Torres.  Dynamics and optimal control of ebola transmission.  _Math.Comput.Sci._ , DOI 10.1007/s11786-016-0268-y, 2016.
  * Ross (1911) R. Ross.  The prevention of malaria.  _John Murray, London_ , 1911.
  * Silva and Torres (2013) C. J. Silva and D. F. M. Torres.  An optimal control approach to malaria prevention via insecticide-treated nets.  _Conference Papers in Mathematics_ , 2013, Article ID 658468:8, 2013.
  * Sweilam and Al-Mekhlafi (2016) N. H. Sweilam and S. M. Al-Mekhlafi.  On the optimal control for fractional multi-strain tb model.  _Optim. Control Appl. Meth. DOI: 10.1002/oca.2247_ , 2016.
  * Tricaud and Chen (2010) C. Tricaud and Y. Chen.  Time-optimal control of systems with fractional dynamics.  _International Journal of Differential Equations_ , 2010, Article ID 461048:16, 2010.
  * Tumwiine et al. (2007a) J. Tumwiine, J. Y. T. Mugisha, and L. S. Luboobi.  On oscillatory pattern of malaria dynamics in a population with temporary immunity.  _Computational and Mathematical Methods in Medicine_ , 8:191–203, 2007a.
  * Tumwiine et al. (2007b) J. Tumwiine, J. Y. T. Mugisha, and L. S. Luboobi.  A mathematical model for the dynamics of malaria in a human host and mosquito vector with temporary immunity.  _Applied Mathematics and Computation_ , 189:1953–1965, 2007b.


# Can wormholes and black holes be distinguished by magnification?

Ke Gao<EMAIL_ADDRESS>Lei-Hua Liu<EMAIL_ADDRESS>Department of Physics, College of Physics, Mechanical and Electrical
Engineering, Jishou University, Jishou 416000, China

###### Abstract

The magnification effect of wormholes and black holes has been extensively
researched. It is crucial to provide a finite distance analysis to understand
this magnification phenomenon better. In this article, the rotational Simpson-
Visser metric (RSV) is chosen as the focus of research. By calculating the
deflection of light in RSV metric, we determine the resulting magnification
effect, then applied the RSV metric to specific examples such as the Ellis-
Bronnikov wormhole, Schwarzschild black hole, and Kerr black hole (or
wormhole) to analyze the magnification. We find that Ellis-Bronnikov wormhole
only has single magnification peaks, while Kerr black hole has one to three
magnification peaks. In addition, the article’s findings suggest that the
lensing effect of the Central Black Hole of the Milky Way Galaxy exhibits
magnification of multiple peaks. However, it should be noted that these
effects are not observable from Earth.

##  I Introduction

The recent development of gravitational wave astronomy has brought a shift in
the discussion surrounding black hole mimickers from theory to empirical
observations Mazza:2021rgq ; Annulli:2021ccn ; Liu:2021aqh ; Bao:2018msr ;
Abdulxamidov:2022ofi ; Heckman:2021vzx ; Aneesh:2018hlp ; Wang:2018mlp ;
Simpson:2019cer . With the absence of a definitive quantum gravity theory
Clifton:2011jh ; Capozziello:2011et , there is an opportunity to explore
alternative phenomenologically viable scenarios using simple meta-geometries.
This approach offers a compact and accessible means of investigating such
possibilities. Simpson and Visser Mazza:2021rgq ; Simpson:2019cer ;
Lobo:2020ffi  have put forward a static and spherically symmetric metric known
as the SV metric, which provides a comprehensive and nuanced model for black
holes and wormholes. This metric smoothly transitions between different cases
and is described by the line element:

$\begin{split}ds^{2}&=-\left(1-\frac{2M}{\sqrt{r^{2}+l^{2}}}\right)dt^{2}+\left(1-\frac{2M}{\sqrt{r^{2}+l^{2}}}\right)^{-1}dr^{2}\\\
&+\left(r^{2}+l^{2}\right)\left(d\theta^{2}+\sin^{2}\theta
d\phi^{2}\right),\end{split}$ (1)

where $M\geqslant 0$ represents the ADM mass and $l\geqslant 0$ is a parameter
responsible for regularizing the central singularity. By employing the
Newman–Janis procedure Newman:1965tw , the rotating SV metric (RSV) can be
obtained:

$\begin{split}ds^{2}&=-\left(1-\frac{2M\sqrt{r^{2}+l^{2}}}{\Sigma}\right)dt^{2}+\frac{\Sigma}{\Delta}dr^{2}\\\
&+\Sigma
d\theta^{2}-\frac{4Ma\sin^{2}\theta\sqrt{r^{2}+l^{2}}}{\Sigma}dtd\phi+\frac{\chi\sin^{2}\theta}{\Sigma}d\phi^{2},\end{split}$
(2)

where $\Sigma=r^{2}+l^{2}+a^{2}\cos^{2}(\theta)$,
$\Delta=r^{2}+l^{2}+a^{2}-2M\sqrt{r^{2}+l^{2}}$, and
$\chi=(r^{2}+l^{2}+a^{2})^{2}-\Delta a^{2}\sin^{2}\theta$. Here, $a$
represents the ratio of spin angular momentum $J$ to mass $M$,
$a=\frac{J}{M}$. The RSV metric (2) reduces to the SV metric when $a=0$ and to
the Kerr metric when $l=0$. The values of $\frac{l}{M}$ and $\frac{a}{M}$
determine the conversion between black holes and wormholes in the RSV metric,
as shown in FIG. 1. These distinctive features of RSV make it a convenient
tool for studying the differences between black holes and wormholes.

Figure 1: Parameter space and corresponding spacetime structure of RSV. WoH
traversable wormhole; nWoH null WoH, i.e. one-way wormhole with the null
throat; RBH-I regular black hole with one horizon (in the $r>0$ side, plus its
mirror image in the $r<0$ side); RBH-II regular black hole with an outer and
an inner horizon (per side); eRBH extremal regular black hole (one extremal
horizon per side); nRBH null RBH-I, i.e. a regular black hole with one horizon
(per side) and a null throat. (Image referenced from Mazza:2021rgq ).

Gravitational lensing is a powerful technology for exploring the universe
Huang:2023bto ; Kelly:2023mgv ; Li:2023zrz ; Awad:2023tvu . The presence of a
lens can alter the space-time and impact the observed physical quantities such
as magnification Kelly:2023wzm ; Awad:2023tvu ; Gao:2022cds ; Sun:2022ujt ,
event rate Kelly:2023mgv ; Cai:2022kbp ; Gao:2023sla  and shadow Meng:2023wgi
; Ramadhan:2023ogm ; Xavier:2023exm . By studying these observable
measurements, we can explore the lens itself. Before our work, physicists have
extensively studied the lensing effect of black holes Delos:2023fpm ;
Lu:2022yuc ; Sun:2022ujt ; Verma:2022pym ; Gao:2022cds ; Rose:2022xdu  and
wormholes Cheng:2021hoc ; Tsukamoto:2017hva ; Sajadi:2016hko ;
Tsukamoto:2016zdu ; Safonova:2001vz ; Torres:1998xd . Their research provides
valuable insights that inspire us to search the possibility of distinguishing
wormholes and black holes based on magnification: wormholes and black holes
may exhibit different magnification patterns.

In this article, we select RSV space as the research object because it can
smoothly transform between wormholes and black holes. We calculate the
deflection angle of light in the RSV metric and study the resulting
magnification effect. We specifically discussed the magnification effect of
Ellis-Bronnikov wormhole, Schwarzschild black hole, and Kerr black hole. We
used dimensionless units to analyze the variation of magnification with lens
geometry, wormhole throat radius, ADM mass, and spin. At the end of the
article, we restored physical units to analyze the magnification of the
Central Black Hole of the Milky Way Galaxy.

The structure of this article is as follows: the first section I provides an
introduction about background space and gravitational lensing, section II
describes some geometric quantities in axisymmetric rotational spacetime,
Section III calculates the deflection angle of RSV metric, Section IV analyzes
the magnification effect of RSV associating spacetime, and Section V concludes
and provides future prospects.

##  II Spacetime

In this section, we examine various geometric quantities associated with the
metric (2). The terminology used follows from Ono:2017pie ; Ono:2018jrv ;
Ishihara:2016sfv ; Ishihara:2016vdc . For convenience, we rewrite the metric
(2) as:

$ds^{2}=-Adt^{2}+Bdr^{2}+Cd\theta^{2}-2Hdtd\phi+Dd\phi^{2},$ (3)

where $A=g_{tt}$, $B=g_{rr}$, and so on. We primarily focus on studying the
motion of photons in the metric (3). The length along the light path,
$ds^{2}=0$, is given by:

$d\ell^{2}\equiv\gamma_{ij}dx^{i}dx^{j}=\frac{B}{A}dr^{2}+\frac{C}{A}d\theta^{2}+\frac{AD+H^{2}}{A^{2}}d\phi^{2}.$
(4)

Here, $\gamma_{ij}$ defines a 3-dimensional Riemann space where the photon’s
motion is described as a trajectory in a spatial curve $\ell$. The closest
distance between a photon and the central celestial body in the lens plane is
known as the impact parameter $b$ (the definition comes from straight-line
approximation), which can be expressed as:

$b\equiv\frac{L}{E}=\frac{-H+D\frac{d\phi}{dt}}{A+H\frac{d\phi}{dt}}.$ (5)

This expression is general for axisymmetric spacetime. $L$ represents the
angular momentum of the photon and $E$ represents its energy. In terms of the
impact parameter, the orbital equation of the photon can be expressed as:

$\left(\frac{dr}{d\phi}\right)^{2}=\frac{AD+H^{2}}{B}\frac{D-2Hb-
Ab^{2}}{(H+Ab)^{2}}.$ (6)

This equation can be solved to obtain $r=F(\phi)$ using perturbational methods
(see the appendix of Gao:2023ltr ). This is an important detail for
calculating the deflection angle using the Gaussian Bonnet theory. At a zero-
order approximation, we can obtain $r=\frac{b}{\sin\phi}$. In metric (3), the
geodesic curvature can be defined as:

$\kappa=-\frac{1}{\sqrt{\gamma\gamma^{\theta\theta}}}\partial_{r}\left(\frac{H}{A}\right),$
(7)

and the Gauss curvature as:

$K=-\sqrt{\frac{A^{3}}{B(AD+H^{2})}}\partial_{r}\left[\frac{1}{2}\sqrt{\frac{A^{3}}{B(AD+H^{2})}}\partial_{r}\left(\frac{AD+H^{2}}{A^{2}}\right)\right].$
(8)

With these preparations, we can now investigate the microlensing effect of the
RSV metric.

Figure 2: Displaying the Gaussian Bonnet Integral Domain. The spin of the
celestial body will make the original geodesic no longer be the geodesic.
Figure 3: Showing the lens plane, L represents wormholes or black holes, O is
the observer, S is the source, $D_{LS},D_{L},$ and $D_{S}$ is the angular
radius distance. $b$ is the impact parameter, while others are angular
quantities.

##  III deflection angle

We consider that both the source and the observer are far away from the lens.
The deflection angle $\alpha$ is defined as the difference between the ray
directions at the source $\ell=-\infty$ and the observer $\ell=\infty$ in the
asymptotically flat regions Asada:2003nf .

$\alpha_{\text{infinity}}=e_{S}-e_{O}=-\int_{-\infty}^{\infty}d\ell\frac{de}{d\ell}.$
(9)

We adopt GW method Gibbons:2008rj ; Werner:2012rc  to calculate the deflection
angle by Gauss Bonnet’s theorem,

$\int\int_{D}KdS+\int_{\partial D}\kappa
d\ell+\sum\limits_{i}\alpha_{i}=2\pi\chi(D).$ (10)

One can choose the integral domain $D$ in Fig. 2, $SO$ is the path of light,
Besides, this Euler index $\chi$ in domain $D$ is one. Then,

$\int\int_{D}KdS+\int_{\gamma}\kappa d\ell+\int_{OS}\kappa
d\ell+\sum_{i}\alpha_{i}=2\pi.$ (11)

We can set $\gamma$ to vertically intersect extension line of $SO$ at
infinity, which means

$\sum_{i}\alpha_{i}=\frac{\pi}{2}(\infty)+\frac{\pi}{2}(\infty)=\pi.$ (12)

Then we do an integral transformation

$\kappa d\ell=\kappa\frac{d\ell}{d\phi}d\phi.$ (13)

Here $\phi$ is the angular coordinate of the center at $L$. Since we only
fixed the endpoint of $\gamma$, we can directly set up
$\kappa\frac{dt}{d\phi}=1$ on $\gamma$, so there is

$\int\int_{D}KdS+\int_{0}^{\pi+\alpha}d\phi+\int_{OS}\kappa d\ell+\pi=2\pi.$
(14)

$SO$ is a straight line (without a lens), that is, the angle that $SO$ spans
is $\pi+\alpha$, and we select the angular coordinate of point O to be 0. We
assume that both the observer and the light source are far away from the
central object for the upper limit of integration $r=\infty$. Then

$\alpha=-\int_{\phi_{O}}^{\phi_{S}}\int_{r}^{\infty}KdS+\int_{\phi_{O}}^{\phi_{S}}\kappa\psi
d\phi,$ (15)

where used $d\ell=\psi d\phi$. In the case of metric (2), We present the
results of each part of the calculation:

$\frac{1}{r}=\frac{\sin\phi}{b}+\frac{M}{b^{2}}(1+\cos^{2}\phi)+\mathcal{O}(M^{2},a^{2},l^{2},Ma),$
(16)
$K=-\frac{2M}{r^{3}}+\frac{3M^{2}}{r^{4}}-\frac{l^{2}}{r^{4}}+\mathcal{O}(l^{3},a^{3},M^{3},...),$
(17)
$dS=\left(r+\frac{l^{2}}{2r}+3M+\frac{15M^{2}}{2r}+\mathcal{O}(l^{3},a^{3},M^{3},...)\right)drd\phi,$
(18) $\kappa=\pm\frac{2aM}{r^{3}}+\mathcal{O}(l^{3},a^{3},M^{3},...),$ (19)

and

$\psi=b\csc^{2}(\phi)+\mathcal{O}(l,M,a).$ (20)

Finally, the infinite distance deflection angle of the second order is equal
to

$\alpha_{\text{infinite}}^{\text{RSV}}=\frac{4M}{b}+\frac{\pi\left(l^{2}+15M^{2}\right)}{4b^{2}}\pm\frac{4aM}{b^{2}},$
(21)

where $\pm$ comes from spin direction. For the deflection angle $\alpha$, when
$M$ is 0, it is returned to the deflection angle of the Ellis-Bronnikov
wormhole Dey:2008kn  where $l$ is wormhole throat. When $l^{2}$ equals 0, it
is returned to the Kerr black hole case Werner:2012rc . Referring to Fig. 2
and 3, we obtain a finite distance deflection angle $\alpha_{\text{finite}}$:

$\bigg{[}\frac{l^{2}+15M^{2}}{4b^{2}}\phi-\frac{2M}{b}\cos\phi\pm\frac{2aM}{b^{2}}\cos\phi\bigg{]}_{\phi=\phi_{O}}^{\phi=\phi_{S}}$
(22)

with

$\phi\big{|}_{\phi_{O}}^{\phi_{S}}=\pi-\arcsin\frac{b}{D_{LS}}-\arcsin\frac{b}{D_{L}}$
(23)

and

$\cos\phi\big{|}_{\phi_{O}}^{\phi_{S}}=-\big{(}\sqrt{1-b^{2}/D_{LS}^{2}}+\sqrt{1-b^{2}/D_{L}^{2}}\big{)}.$
(24)

These geometric relationships can also be solved by solving Eq (6), and then
the angular coordinates $\phi_{S}$ and $\phi_{O}$ corresponding to the
distances $D_{LS}$ and $D_{L}$ respectively. Furthermore, in Eq (23) and Eq
(24) we overlooked the term of $\mathcal{O}(M)$. For more details, see
Ono:2017pie .

##  IV magnification

Magnification reflects the degree to which light is distorted, which is an
observable quantity. We will study the lensing effect from this perspective.
Referring to Fig. 3, One can obtain the geometric relationship as

$\beta=\theta-\frac{D_{LS}}{D_{S}}\alpha.$ (25)

The lens potential $\Psi$ which can connect deflection angle and magnification
is defined as $\Psi=\frac{2D_{LS}}{D_{L}D_{S}}\int\Phi(\theta D_{L},x)dx,$
where $\Phi$ is Newtonian potential. The relationship between lens potential
energy and deflection angle is as follows

$\partial_{\theta}\Psi=D_{L}\partial_{b}\frac{2D_{LS}}{D_{L}D_{S}}\int\Phi(b,x)dx=\alpha.$
(26)

On the other hand, the Jacobian metric is also related to the lens potential

$A_{ij}\equiv\left(\delta_{ij}-\frac{\partial^{2}\Psi}{\partial\theta_{i}\partial\theta_{j}}\right).$
(27)

The magnification is defined as the inverse of the Jacobian determinant and
can in axisymmetric spaces be expressed as:

$\mu\equiv\frac{1}{\det
A_{ij}}=\bigg{|}\frac{\beta}{\theta}\frac{d\beta}{d\theta}\bigg{|}^{-1}.$ (28)

Based on Eq (28), (21), and (25), we could work out the magnification $\mu$ of
metric (2) associating spacetime:

$\mu_{\text{finite}}=\bigg{|}\frac{D_{L}^{2}}{b}\beta\frac{\partial}{\partial
b}\bigg{(}\frac{b}{D_{L}}-\frac{D_{LS}}{D_{S}}\alpha_{\text{finite}}\bigg{)}\bigg{|}^{-1}.$
(29)

We do not intend to discuss the issue of magnification peak, where
$\frac{d\mu}{db}=0$ and $\frac{d^{2}\mu}{db^{2}}<0$, from the analytical
expression of magnification. Directly drawing numerical graphs with limited
distance magnification is more intuitive.

Figure 4: Contour plot of magnification of RSV. The relevant parameters are
fixed as $M=0.03,\,\,l=0.05,\,\,a=0.1,\,\,b=2$. The black bar in the figure
represents the position where the magnification peak appears. Figure 5:
Contour plot of magnification of Ellis-Bronnikov wormhole.

We first consider the influence of lens geometry on magnification. (Due to the
dimensionless nature of magnification, we adopt a dimensionless
specification.) The numerical results presented in Fig. 4 demonstrate that the
distance between the source and lens has minimal influence on magnification,
while the distance between the observer and lens has a noteworthy impact. For
the interval $D_{L}\in[55,100]$, the magnification decreases to one. When the
observer is situated far from the lens $D_{L}\in[100,\infty)$ the
magnification is reduced to zero. In light of these findings, we will continue
our discussion using a fixed parameter of $D_{L}=D_{LS}=10$ in subsequent
cases.

If the ADM mass is set to zero, the RSV metric will transform into an Ellis-
Bronnikov wormhole. In this scenario, where $l$ represents the throat radius
of the wormhole, the magnification is depicted in Fig. 5. From the figure, we
can observe that the Ellis-Bronnikov wormhole exhibits a single magnification
peak. (How to determine the number of peaks? We use a straight line parallel
to the b-axis to capture the black bars in the graph, and the number of cutoff
points is the number of magnification peaks.)

Figure 6: Contour plot of magnification of Schwarzschild black hole.

When both $l$ and $a$ are set to zero, the RSV metric reverts to a
Schwarzschild black hole. In Fig. 6, it can be observed that there are three
magnification peaks within the interval $M\in[0.05,0.1]$. As we decrease the
mass to the range of $M\in[0.05,0.01]$, the three peaks coalesce into two.
Finally, when the mass is less than $0.01$, only one magnification peak
remains.

Figure 7: Kerr black hole with positive spin, in which $M=0.05$. According to
Fig. 1, when $a>M=0.05$, RSV will transform into a traversable wormhole.
Figure 8: Kerr black hole with negative spin, in which $M=0.05$.

In the case of a rotating black hole or possibly a wormhole. $l$ corresponds
to exotic matter ($T_{00}<0$), and $M$ represents conventional matter. Btween
$M$ and $l$ may has an interaction term $T_{\text{inter}}$ that corresponds to
a geometric quantity $G_{\text{inter}}$. To ensure safety, we set $l$ to zero.
it is significant to consider the direction of the black hole’s spin
determined by parameter $a$. We use the positive and negative cases of a to
represent different spin directions, as discussed in Fig. 7 and Fig. 8,
respectively. Let us first examine the scenario where the spin is positive, as
depicted in Fig. 7. In the interval of $a\in[0.00,0.05]$, two magnification
peaks can be observed. However, for $a\in[0.05,0.30]$, only one magnification
peak is present. Next, we consider the case of negative spin, as illustrated
in Fig. 8. Within the interval of $a\in[0.00,0.18]$, three magnification peaks
exist. In the interval of $a\in[0.18,0.22]$, there are two magnification
peaks. Finally, for $a\in[0.22,0.30]$, only one magnification peak is
observed.

Figure 9: Schematic diagram of the three peaks of Schwarzschild black hole,
Corresponding to the case of $M=0.08$ in Fig. 6. Here, the Schwarzschild
radius about is $0.1$ on $b$ axle.

##  V Conclusions and outlook

In this paper, we study the gravitational lensing effect of the RSV metric. We
obtained the second-order deflection angle of the RSV metric and made finite
distance corrections to it. Our analysis of magnification shows that the
distance between the lens and the observer has a significant impact on the
numerical value of magnification, which also indicates that finite distance
correction is meaningful. On this basis, Ellis-Bronnikov Wormhole only
exhibits a single peak of magnification, while Schwarzschild’s black hole
exhibits up to three magnification peaks as the ADM mass increases. Black
holes with negative spin return from three peaks to a single peak as the spin
increases, and Black holes with positive spin return from two peaks to a
single peak as the spin increases.

For the galaxy we live in, the mass of its central black hole is about four
million Solar mass $\rm M_{\odot}$. If we assume that its spin is small (Spin
slightly affect the position of the peak) and be approximated as the
Schwarzschild black hole situation, according to our calculations
$\frac{\text{four million}~{}\rm M_{\odot}}{0.08}=\frac{DL}{10}$, we will be
able to observe the phenomenon of three magnification peaks, see Fig. 9, at
distance $D_{L}=7.4\times 10^{8}~{}\rm km$ (The Schwarzschild radius is
approximately $7.8\times 10^{6}~{}\rm km$). The distance from Earth to the
center of the Milky Way galaxy is about $2.4\times 10^{16}~{}\rm km$, and the
magnification effect observed at this distance is equivalent to a small mass
situation in Fig. 6, which will converge to a single peak situation. (Not even
observable, according to Fig. 4.).

Finally, it is theoretically feasible to distinguish between wormholes and
black holes based on magnification, But it is unfeasible at far distances.

## References

## References

  * (1) J. Mazza, E. Franzin and S. Liberati, JCAP 04 (2021), 082 doi:10.1088/1475-7516/2021/04/082 [arXiv:2102.01105 [gr-qc]].
  * (2) L. Annulli, V. Cardoso and L. Gualtieri, Class. Quant. Grav. 39 (2022) no.10, 105005 doi:10.1088/1361-6382/ac6410 [arXiv:2104.11236 [gr-qc]].
  * (3) Y. T. Wang, Z. P. Li, J. Zhang, S. Y. Zhou and Y. S. Piao, Eur. Phys. J. C 78 (2018) no.6, 482 doi:10.1140/epjc/s10052-018-5974-y [arXiv:1802.02003 [gr-qc]].
  * (4) S. Aneesh, S. Bose and S. Kar, Phys. Rev. D 97 (2018) no.12, 124004 doi:10.1103/PhysRevD.97.124004 [arXiv:1803.10204 [gr-qc]].
  * (5) N. Bao, A. Chatwin-Davies, J. Pollack and G. N. Remmen, JHEP 11 (2018), 071 doi:10.1007/JHEP11(2018)071 [arXiv:1808.05963 [hep-th]].
  * (6) H. Liu, W. L. Qian, Y. Liu, J. P. Wu, B. Wang and R. H. Yue, Phys. Rev. D 104 (2021) no.4, 044012 doi:10.1103/PhysRevD.104.044012 [arXiv:2104.11912 [gr-qc]].
  * (7) F. Abdulxamidov, C. A. Benavides-Gallego, W. B. Han, J. Rayimbaev and A. Abdujabbarov, Phys. Rev. D 106 (2022) no.2, 024012 doi:10.1103/PhysRevD.106.024012 [arXiv:2205.11727 [gr-qc]].
  * (8) J. J. Heckman, A. P. Turner and X. Yu, Phys. Rev. D 105 (2022) no.8, 086021 doi:10.1103/PhysRevD.105.086021 [arXiv:2111.06404 [hep-th]].
  * (9) A. Simpson, P. Martin-Moruno and M. Visser, Class. Quant. Grav. 36 (2019) no.14, 145007 doi:10.1088/1361-6382/ab28a5 [arXiv:1902.04232 [gr-qc]].
  * (10) T. Clifton, P. G. Ferreira, A. Padilla and C. Skordis, Phys. Rept. 513 (2012), 1-189 doi:10.1016/j.physrep.2012.01.001 [arXiv:1106.2476 [astro-ph.CO]].
  * (11) S. Capozziello and M. De Laurentis, Phys. Rept. 509 (2011), 167-321 doi:10.1016/j.physrep.2011.09.003 [arXiv:1108.6266 [gr-qc]].
  * (12) F. S. N. Lobo, M. E. Rodrigues, M. V. de Sousa Silva, A. Simpson and M. Visser, Phys. Rev. D 103 (2021) no.8, 084052 doi:10.1103/PhysRevD.103.084052 [arXiv:2009.12057 [gr-qc]].
  * (13) R. Carballo-Rubio, F. Di Filippo, S. Liberati and M. Visser, Phys. Rev. D 101 (2020), 084047 doi:10.1103/PhysRevD.101.084047 [arXiv:1911.11200 [gr-qc]].
  * (14) T. Ono, A. Ishihara and H. Asada, Phys. Rev. D 96 (2017) no.10, 104037 doi:10.1103/PhysRevD.96.104037 [arXiv:1704.05615 [gr-qc]].
  * (15) T. Ono, A. Ishihara and H. Asada, Phys. Rev. D 99 (2019) no.12, 124030 doi:10.1103/PhysRevD.99.124030 [arXiv:1811.01739 [gr-qc]].
  * (16) A. Ishihara, Y. Suzuki, T. Ono and H. Asada, Phys. Rev. D 95 (2017) no.4, 044017 doi:10.1103/PhysRevD.95.044017 [arXiv:1612.04044 [gr-qc]].
  * (17) A. Ishihara, Y. Suzuki, T. Ono, T. Kitamura and H. Asada, Phys. Rev. D 94 (2016) no.8, 084015 doi:10.1103/PhysRevD.94.084015 [arXiv:1604.08308 [gr-qc]].
  * (18) X. J. Gao, X. k. Yan, Y. Yin and Y. P. Hu, Eur. Phys. J. C 83 (2023), 281 doi:10.1140/epjc/s10052-023-11414-0 [arXiv:2303.00190 [gr-qc]].
  * (19) H. Asada, M. Kasai and T. Yamamoto, Phys. Rev. D 67 (2003), 043006 doi:10.1103/PhysRevD.67.043006 [arXiv:astro-ph/0301099 [astro-ph]].
  * (20) G. W. Gibbons and M. C. Werner, Class. Quant. Grav. 25 (2008), 235009 doi:10.1088/0264-9381/25/23/235009 [arXiv:0807.0854 [gr-qc]].
  * (21) M. C. Werner, Gen. Rel. Grav. 44 (2012), 3047-3057 doi:10.1007/s10714-012-1458-9 [arXiv:1205.3876 [gr-qc]].
  * (22) T. K. Dey and S. Sen, Mod. Phys. Lett. A 23 (2008), 953-962 doi:10.1142/S0217732308025498 [arXiv:0806.4059 [gr-qc]].
  * (23) E. T. Newman and A. I. Janis, J. Math. Phys. 6 (1965), 915-917 doi:10.1063/1.1704350
  * (24) H. Li and Y. Chen, Phys. Dark Univ. 41 (2023), 101234 doi:10.1016/j.dark.2023.101234 [arXiv:2304.06529 [astro-ph.CO]].
  * (25) P. L. Kelly, S. Rodney, T. Treu, M. Oguri, W. Chen, A. Zitrin, S. Birrer, V. Bonvin, L. Dessart and J. M. Diego, et al. doi:10.1126/science.abh1322 [arXiv:2305.06367 [astro-ph.CO]].
  * (26) Y. Huang and Z. Cao, [arXiv:2306.04145 [gr-qc]].
  * (27) P. L. Kelly, S. Rodney, T. Treu, S. Birrer, V. Bonvin, L. Dessart, R. J. Foley, A. V. Filippenko, D. Gilman and S. Jha, et al. Astrophys. J. 948 (2023) no.2, 93 doi:10.3847/1538-4357/ac4ccb [arXiv:2305.06377 [astro-ph.CO]].
  * (28) P. Awad, J. H. H. Chan, M. Millon, F. Courbin and E. Paic, Astron. Astrophys. 673 (2023), A88 doi:10.1051/0004-6361/202245615 [arXiv:2304.01320 [astro-ph.GA]].
  * (29) K. Gao, L. H. Liu and M. Zhu, Phys. Dark Univ. 41 (2023), 101254 doi:10.1016/j.dark.2023.101254 [arXiv:2211.17065 [gr-qc]].
  * (30) J. Sun, X. Er and O. Y. Tsupko, Mon. Not. Roy. Astron. Soc. 520 (2023) no.1, 994-1004 doi:10.1093/mnras/stad200 [arXiv:2211.13442 [astro-ph.SR]].
  * (31) K. Gao and L. H. Liu, [arXiv:2303.11134 [gr-qc]].
  * (32) R. G. Cai, T. Chen, S. J. Wang and X. Y. Yang, JCAP 03 (2023), 043 doi:10.1088/1475-7516/2023/03/043 [arXiv:2210.02078 [astro-ph.CO]].
  * (33) M. S. Delos and G. Franciolini, Phys. Rev. D 107 (2023) no.8, 083505 doi:10.1103/PhysRevD.107.083505 [arXiv:2301.13171 [astro-ph.CO]].
  * (34) P. Lu, V. Takhistov and G. M. Fuller, Phys. Rev. Lett. 130 (2023) no.22, 22 doi:10.1103/PhysRevLett.130.221002 [arXiv:2212.00156 [astro-ph.CO]].
  * (35) H. Verma and V. Rentala, JCAP 05 (2023), 045 doi:10.1088/1475-7516/2023/05/045 [arXiv:2208.14460 [astro-ph.GA]].
  * (36) S. Rose, C. Y. Lam, J. R. Lu, M. Medford, M. W. Hosek, N. S. Abrams, E. Ramey and S. S. Vasylyev, Astrophys. J. 941 (2022) no.2, 116 doi:10.3847/1538-4357/aca09d [arXiv:2211.04471 [astro-ph.HE]].
  * (37) X. T. Cheng and Y. Xie, Phys. Rev. D 103 (2021) no.6, 064040 doi:10.1103/PhysRevD.103.064040
  * (38) N. Tsukamoto and Y. Gong, Phys. Rev. D 97 (2018) no.8, 084051 doi:10.1103/PhysRevD.97.084051 [arXiv:1711.04560 [gr-qc]].
  * (39) S. N. Sajadi and N. Riazi, Can. J. Phys. 98 (2020) no.11, 1046-1054 doi:10.1139/cjp-2019-0524 [arXiv:1611.04343 [gr-qc]].
  * (40) N. Tsukamoto and T. Harada, Phys. Rev. D 95 (2017) no.2, 024030 doi:10.1103/PhysRevD.95.024030 [arXiv:1607.01120 [gr-qc]].
  * (41) M. Safonova, D. F. Torres and G. E. Romero, Phys. Rev. D 65 (2002), 023001 doi:10.1103/PhysRevD.65.023001 [arXiv:gr-qc/0105070 [gr-qc]].
  * (42) D. F. Torres, G. E. Romero and L. A. Anchordoqui, Phys. Rev. D 58 (1998), 123001 doi:10.1103/PhysRevD.58.123001 [arXiv:astro-ph/9802106 [astro-ph]].
  * (43) S. V. Iyer and A. O. Petters, Gen. Rel. Grav. 39 (2007), 1563-1582 doi:10.1007/s10714-007-0481-8 [arXiv:gr-qc/0611086 [gr-qc]].
  * (44) C. R. Keeton and A. O. Petters, Phys. Rev. D 72 (2005), 104006 doi:10.1103/PhysRevD.72.104006 [arXiv:gr-qc/0511019 [gr-qc]].
  * (45) Y. Meng, X. M. Kuang, X. J. Wang and J. P. Wu, Phys. Lett. B 841 (2023), 137940 doi:10.1016/j.physletb.2023.137940 [arXiv:2305.04210 [gr-qc]].
  * (46) H. S. Ramadhan, M. F. Ishlah, F. P. Pratama and I. Alfredo, Eur. Phys. J. C 83 (2023) no.6, 465 doi:10.1140/epjc/s10052-023-11648-y [arXiv:2303.10921 [gr-qc]].
  * (47) S. V. M. C. B. Xavier, H. C. D. Lima, Junior. and L. C. B. Crispino, Phys. Rev. D 107 (2023) no.6, 064040 doi:10.1103/PhysRevD.107.064040 [arXiv:2303.17666 [gr-qc]].


# AiM: Taking Answers in Mind to Correct
Chinese Cloze Tests in Educational Applications

Yusen Zhang1, Zhongli Li2, Qingyu Zhou2, Ziyi Liu1∗,
Chao Li3, Mina Ma2, Yunbo Cao2, Hongzhi Liu1
1Peking University, 2Tencent Cloud Xiaowei, 3Xiaomi Group
{yusen-zhang0826@stu,lzymail<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Contribution done during internship at Tencent Cloud
Xiaowei. Qingyu Zhou is the corresponding author.

###### Abstract

To automatically correct handwritten assignments, the traditional approach is
to use an OCR model to recognize characters and compare them to answers. The
OCR model easily gets confused on recognizing handwritten Chinese characters,
and the textual information of the answers is missing during the model
inference. However, teachers always have these answers in mind to review and
correct assignments. In this paper, we focus on the Chinese cloze tests
correction and propose a multimodal approach111Our codes and data are
available at https://github.com/YusenZhang826/AiM. (named AiM). The encoded
representations of answers interact with the visual information of students’
handwriting. Instead of predicting ‘right’ or ‘wrong’, we perform the sequence
labeling on the answer text to infer which answer character differs from the
handwritten content in a fine-grained way. We take samples of OCR datasets as
the positive samples for this task, and develop a negative sample augmentation
method to scale up the training data. Experimental results show that AiM
outperforms OCR-based methods by a large margin. Extensive studies demonstrate
the effectiveness of our multimodal approach.

##  1 Introduction

Figure 1:  Task examples and correction methods. The OCR-based method
misunderstands student’s intent due to the width between “车” and “可” in
Question 2. AiM can tackle this problem since it takes both the handwriting
image and the textual answer into consideration.

The growing number of students has brought much pressure on teachers to
correct assignments manually in the educational field. Recently, Optical
Characters Recognition (OCR) based methods are widely used in several
applications222Educational applications such as http://kousuan.yuanfudao.com,
https://jiazhang.zuoyebang.com and https://www.zuoye.ai. to automatically
complete this task. The OCR model first recognizes the text in the image, and
then the OCR output is compared with the correct answer to feedback correction
results. In this pipeline method, the post-comparison mainly relies on the
pre-recognition. Without prior knowledge of answers, the OCR model easily gets
confused when recognizing handwritten Chinese characters, especially for
various handwriting styles, ligatures, and shape-similar characters. However,
most of them can be distinguished by human teachers, because they take answers
in mind at first.

In this paper, we focus on the Chinese cloze correction (CCC) and propose an
Answer-in-Mind correction model (AiM) to tackle the above problem. Figure 1
shows examples of the CCC task and the comparison of the OCR-based method with
our method. We look at Question 2. Obviously, the student knows the correct
answer and writes down “轲”, but it is recognized as “车可” by the OCR model
because of the slightly large width of the hand-written content. Consequently,
the correction result is ‘wrong’. We hypothesize that the information of
correct answers can help the neural model to understand student handwriting.
As shown in Figure 1, AiM is a multimodal model, which takes the image and the
answer text as input. Through the interaction of two modality information, our
model understands the handwritten content and feeds back the ‘right’ correct
result.

In the AiM model, the image is encoded to sequential feature representations
through Resnet He et al. (2016), where each of them represents a fixed-width
pixels block. The answer text is encoded by word embedding. Then Transformer
Vaswani et al. (2017) self-attention is adopted to compute contextual
representations for each modality. In order to fuse them, we develop a cross-
modal attention. It renders the textual representations to interact with the
visual information of students’ handwriting. On the top of AiM, instead of
predicting ‘right’ or ‘wrong’, our model performs sequence labeling on the
answer text to infer which character differs from the handwritten content.

To train AiM, we collect EinkCC, a dataset containing about 5k handwriting
images, answers, and correction results of cloze questions, from our
educational application. Teachers distribute cloze tests in our app, and
students practice on the electronic paper hardware. In addition to EinkCC, OCR
datasets can be used for this task. We take samples of OCR datasets (Liu et
al., 2011) as the positive samples, and construct negative samples by
replacing the label characters with shape-similar ones derived from an open-
sourced confusion set Wu et al. (2013). The same method also augments EinkCC
to scale up the training set.

We pretrain the image encoder in AiM to get better visual representations, and
further train AiM with the correction objective. Experimental results show
that compared with OCR-based methods Liu et al. (2020); Du et al. (2021), AiM
achieves 11% accuracy improvements. Extensive analyses verify that with the
interactions between two modalities through our attention mechanism, AiM can
understand student handwriting and ligatures, and more handwritten characters
confused by OCR can be predicted well by AiM.

The main contributions are summarized as follows: i) We propose AiM, a
multimodal model for Chinese cloze correction, to make up for shortages of
OCR-based methods. ii) We extend OCR datasets using a negative sample
augmentation method to fit this task. iii) Comprehensive experiments show that
AiM achieves better performance compared with OCR-based methods, and it’s
effective and necessary to use a multimodal approach to correct Chinese cloze
tests.

Figure 2: The architecture of AiM. The image encoder of AiM is initialized by
OCR pretraining. In AiM, the visual and textual representations are fused by
cross-modal attention.

##  2 Preliminary

###  2.1 Chinese Cloze Correction

We first give the description of the Chinese cloze correction (CCC) task in
this section. Given a handwriting image $\mathbf{I}$ and the textual answer
${\mathbf{A}=[a_{1},a_{2},...,a_{m}]}$ of the corresponding question, assuming
that the handwritten characters in the image is
${\mathbf{C}=[c_{1},c_{2},...,c_{n}]}$, the target of the task is to predict a
label $y\in\\{0,1\\}$. The $y=0$ indicates the correction result is ‘right’
(i.e. the handwritten content is a correct answer, $\mathbf{C}=\mathbf{A}$),
otherwise it is ‘wrong’.

###  2.2 Transformer

Suppose the input of Transformer Vaswani et al. (2017) is a pack of embeddings
${\bf X}^{0}=[{\bf x}_{1},{\bf x}_{2},...,{\bf x}_{|x|}].$ If we have $L$
stacked Transformer blocks, the final output is like:

${\bf X}^{l}=\text{Transformer}_{l}({\bf X}^{l-1}),l\in[1,L]$ (1)

where each block consists of a self-attention layer, a feed-forward layer,
residual connection He et al. (2016) and layer normalization.

##### Self-Attention

For the $l$-th block, the output ${\bf A}_{l}$ of a self-attention head is:

$\displaystyle{\bf Q}={\bf X}^{l-1}{\bf W}^{Q}_{l},{\bf K}={\bf X}^{l-1}{\bf
W}^{K}_{l}$ (2) $\displaystyle{\bf M}_{i,j}=\begin{cases}0,&\text{allow to
attend}\\\ -\infty,&\text{forbid to attend}\end{cases}$ $\displaystyle{\bf
A}_{l}=\text{SelfAttention}({\bf X}^{l-1})$
$\displaystyle~{}~{}~{}~{}~{}=\text{softmax}(\frac{{\bf
QK}^{\texttt{T}}}{\sqrt{d_{k}}}+{\bf M})({\bf X}^{l-1}{\bf W}^{V}_{l})$

where ${\bf W}^{Q}_{l},{\bf W}^{K}_{l},{\bf W}^{V}_{l}$ can project previous
output to queries, keys, and values, respectively. ${\bf
M}\in\mathbb{R}^{|x|\times|x|}$ is a mask matrix that controls whether two
tokens can attend each other.

##  3 Method

We first introduce the sequence labeling conversion for the CCC task in
Section 3.1. Model architecture of AiM is shown in Section 3.2 and Figure 2.
Finally, we describe the data augmentation and pretraining methods in Section
3.3.

###  3.1 Label Space

The CCC task is to feedback ‘right’ or ‘wrong’ on the handwritten content. In
this paper, we perform sequence labeling on the answer text, and the
corresponding labels are defined as:

  * •

del: the current character does not appear in the image.

  * •

add: compared to the handwritten content, one or more characters should be
inserted between the current and the next character333A placeholder <BLK> is
inserted at the beginning of the textual answers to handle the character
missing at the first position..

  * •

sub: compared to the handwritten content, the current character should be
substituted by another one.

We get these labels by calculating the edit distance between the answer and
the handwritten content444The handwritten text is annotated or taken from the
OCR dataset for calculating the label sequence of AiM..

The BIO annotation Ratinov and Roth (2009) is adopted that the label space is
{O, B-sub, I-sub, B-del, I-del, B-add}. It is similar to the label space of
the Grammatical Error Detection/Correction (GED/GEC) task, but our method
compares the answer text to the handwritten content, instead of the erroneous
sentence to the correct sentence. After this conversion, AiM is trained in a
fine-grained way. If the predicted sequence only contains ‘O’, the correction
result of AiM is ‘right’, otherwise it is ‘wrong’.

###  3.2 Model Architecture

The model architecture of AiM is shown in Figure 2. Components include: i) an
image encoder with Resnet and the self-attention mechanism to extract the
visual features, ii) a fusion module with the cross-modal attention to mine
the interactions between modalities, and iii) an output layer to predict the
label sequence.

##### Image Encoder

To understand the handwritten content in images, we follow Liu et al. (2020)
to adopt Resnet (He et al., 2016) as the image encoder. The image encoder maps
the input image $\mathbf{I}$ to a sequence of visual features
$\mathbf{H}_{v}\in\mathbb{R}^{N_{v}\times d}$ with a linear transformation:

$\begin{split}\mathbf{H^{\prime}}_{v}&=\text{Linear}\left(\text{ResNet}(\bf
I)\right)\\\
\mathbf{H}_{v}&=\mathbf{H^{\prime}}_{v}+\mathbf{P}_{v}\end{split}$ (3)

where $N_{v}=\frac{max\\_width}{32}$, ${max\\_width}$ is the maximum width of
input images and $d$ is the hidden size of the visual representation. Each
element in sequence represents a fixed-width pixels block in the image.
Besides, there is a learnable positional embedding matrix
$\mathbf{P}_{v}\in\mathbb{R}^{N_{v}\times d}$ where each row is a positional
representation for each element in $\mathbf{H}_{v}$ to capture the location
information.

We perform a padding operation on the image with extra white pixels blocks to
ensure all images have the same width. Then we adopt Transformer blocks to
capture the contextual information in visual modality and the $l$-th output
is:

$\displaystyle{\bf S}^{l}_{v}=\text{Transformer}({\bf
S}^{l-1}_{v}),l\in[1,L_{v}].$ (4)

where $L_{v}$ is the number of Transformer blocks. Notes that ${\bf
S}^{0}_{v}=\mathbf{H}_{v}$ and $\mathbf{S}_{v}=\mathbf{S}^{L_{v}}_{v}$.

##### Fusion Module

Assuming that the input answer has $N_{t}$ characters, the answer characters
are first converted to dense vectors $\mathbf{X}=[x_{1},x_{2},...,x_{N_{t}}]$
through a word embedding. The linear transformation and positional embedding
are also employed to compute the textual representation
$\mathbf{H}_{t}\in\mathbb{R}^{N_{t}\times d}$ as follows:

$\begin{split}\mathbf{H^{\prime}}_{t}&=\text{Linear}(\mathbf{X})\\\
\mathbf{H}_{t}&=\mathbf{H^{\prime}}_{t}+\mathbf{P}_{t}\end{split}$ (5)

where $\mathbf{P}_{t}$ is the positional embedding matrix.

Then our fusion block further encodes and merges the information of textual
and visual modality. As shown in Figure 2, each fusion block contains a
textual self-attention layer and a cross-modal attention layer. Self-attention
mechanism is employed to encode textual representations:

$\displaystyle{\bf S}^{\prime}_{t}=\text{SelfAttention}({\mathbf{H}_{t}})$ (6)
$\displaystyle{\bf S}_{t}=\text{LayerNorm}(\mathbf{H}_{t}+{\bf
S}^{\prime}_{t})$

To capture the interactions between them, we develop a cross-modal attention
as follows:

$\displaystyle{\bf Q}=\mathbf{S}_{t}{\bf W}^{Q},{\bf K}=\mathbf{S}_{v}{\bf
W}^{K}$ (7) $\displaystyle{\bf
M}^{f}_{i,j}=\begin{cases}-\infty,&\text{padding token or pixels block}\\\
0,&\text{otherwise}\end{cases}$ $\displaystyle{\bf
S}_{f}=\text{softmax}(\frac{{\bf QK}^{\texttt{T}}}{\sqrt{d}}+{\bf M}^{f})({\bf
S}_{v}{\bf W}^{V})$

where $\mathbf{W}^{Q}$ projects $\mathbf{S}_{t}$ to queries, $\mathbf{W}^{K}$
and $\mathbf{W}^{V}$ project $\mathbf{S}_{v}$ to keys and values, and ${\bf
M}^{f}\in\mathbb{R}^{N_{t}\times N_{v}}$ is the mask matrix to ensure only
valid tokens and pixels blocks can attend to each other. The fused
representations $\mathbf{S}_{f}$ are followed by the feed-forward layer,
residual connection, and layer normalization.

Finally, we stack our fusion blocks to obtain more informative fused features.
It can be seen that our fusion module is similar to the Transformer decoder.
Different from the cross-attention of Transformer, our cross-modal attention
merges the visual and textual information, which allows each valid answer
character to attend to all valid pixels blocks, without the order restriction
of language modeling. Besides, the Transformer decoder decodes words one by
one, but our module outputs fused features at once.

##### Output Layer

We denote the above fused features as $\mathbf{H}_{f}$. To project the hidden
representation to the space of labels, the fusion features is fed to the
fully-connected layer and a softmax function to get the final output ${{\bf
O}\in\mathbb{R}^{N_{t}\times m}}$:

${\bf O}=\text{softmax}(\mathbf{H}_{f}{\bf W}_{o})$ (8)

where ${\bf W_{o}}\in\mathbb{R}^{d\times m}$ is the weight matrix and $m$ is
the number of labels. During the training, we apply the cross-entropy function
as our correction objective and the training loss is computed as:

$\mathcal{L}=-\sum_{i=1}^{N_{t}}~{}\text{log}~{}p(i,k_{i}),~{}~{}p(i,k_{i})\in\mathbf{O}$
(9)

where $k_{i}$ is the label of the $i$-th character and $p(i,k_{i})$ is the
probability of the $i$-th character being predicted to label $k_{i}$.

###  3.3 Data Augmentation and Pretraining

Input: $\mathcal{S}=\\{({\bf I}_{i},{\bf C}_{i},{\bf A}_{i},{\bf
L}_{i},y_{i})\\}_{i=1}^{N}$, where ${\bf I}_{i}$ is the image, ${\bf C}_{i}$
is the handwritten content, ${\bf A}_{i}$ is the answer text, ${\bf L}_{i}$ is
the label sequence of AiM and $y_{i}$ is the correction result.

$D\leftarrow\mathcal{S}$

for  _$({\bf I}_{i},{\bf C}_{i},{\bf A}_{i},{\bf L}_{i},y_{i})$ in
$\mathcal{S}$_ do

repeat

$\hat{{\bf A}}_{i}\leftarrow{\bf A}_{i}$

Randomly select an index $j$ in $\hat{{\bf A}}_{i}$

$\hat{{\bf A}}_{i}[j]\leftarrow$ ${\bf A}_{i}[j]$ ’s shape-similar character

Get $\hat{{\bf L}}_{i}$ comparing ${\bf C}_{i}$ and $\hat{{\bf A}}_{i}$

$\hat{y}_{i}=1$

$D$.append ($({\bf I}_{i},{\bf C}_{i},\hat{{\bf A}}_{i},\hat{{\bf
L}}_{i},\hat{y}_{i})$)

until  _Random times_

repeat

$\hat{{\bf A}}_{i}\leftarrow{\bf A}_{i}$

Randomly select an index $j$ in $\hat{{\bf A}}_{i}$

Delete $\hat{{\bf A}}_{i}[j]$

Get $\hat{{\bf L}}_{i}$ comparing ${\bf C}_{i}$ and $\hat{{\bf A}}_{i}$

$\hat{y}_{i}=1$

$D$.append ($({\bf I}_{i},{\bf C}_{i},\hat{{\bf A}}_{i},\hat{{\bf
L}}_{i},\hat{y}_{i})$)

until  _Random times_

repeat

$\hat{{\bf A}}_{i}\leftarrow{\bf A}_{i}$

Randomly select an index $j$ in $\hat{{\bf A}}_{i}$

Insert a common Chinese character at $\hat{{\bf A}}_{i}[j]$

Get $\hat{{\bf L}}_{i}$ comparing ${\bf C}_{i}$ and $\hat{{\bf A}}_{i}$

$\hat{y}_{i}=1$

$D$.append ($({\bf I}_{i},{\bf C}_{i},\hat{{\bf A}}_{i},\hat{{\bf
L}}_{i},\hat{y}_{i})$)

until  _Random times_

Output: the augmented dataset $D$

Algorithm 1 Negative Sample Augmentation

##### Data Augmentation

The sample of the OCR dataset contains an image and the corresponding written
text, which can be easily extended for the CCC task. We directly take the
annotated text as the answer to augment our positive CCC samples. Then we
develop a negative sample augmentation method. Given a CCC sample, we keep the
image unchanged and modify the answer text. The modifications include random
character insertion, deletion and substitution. Especially for the character
substitution, we attempt to construct hard negative samples by replacing the
original character with shape-similar ones. The shape-similar characters are
derived from an open-sourced confusion set555It is taken from URL. (Wu et al.,
2013). The pseudo code of negative sample augmentation is presented in
Algorithm 1.

##### Pretraining

Before the AiM training, we first pretrain our image encoder with the OCR
objective. We follow the Liu et al. (2020) to use CTC (Graves et al., 2006) as
the loss function and apply high dropout rates after each max-pooling layer.
As shown on the left side in Figure 2, the input image is converted to a
sequence of dense representations through Resnet. A linear layer is added on
the top of Resnet to transform the space dimension to the size of the
vocabulary.

Table 1: Data statistics. “#img” means the number of handwriting images. “#sample” means the number of CCC samples. In each train or dev set, the “#sample” is larger than “#img” because of our data augmentation in Section 3.3. Dataset | Train set | Dev set | Test set
---|---|---|---
#img |  #sample |  #img |  #sample |  #img |  #sample
EinkCC |  4256 |  10610 |  - |  - |  673 |  673
HWCC |  41781 |  166984 |  10499 |  41767 |  - |  -
SynCC |  150594 |  602376 |  - |  - |  - |  -
Total |  196631 |  779970 |  10499 |  41767 |  673 |  673

##  4 Experiment

In this section, we first introduce datasets, baselines, and other details of
our experiments. Then we show experimental results and perform analyses from
different views.

###  4.1 Dataset

##### EinkCC

EinkCC is collected from our educational application. Teachers distribute
cloze tests in our app, and students practice on the e-ink display hardware.
When students finish tests, teachers can correct them and feedback to
students. Each sample in EinkCC contains a student’s handwriting image, the
answer text and the correction result marked by teachers. It mainly covers the
dictation of ancient poetry, the idiom application, and the reading
comprehension. Besides, we manually annotate the image text for the OCR
training and the AiM label sequence calculation.

##### HWCC

CASIA-HWDB Liu et al. (2011) is a benchmark for the handwritten Chinese text
recognition(HCTR) task. We select the HWDB 2.x set to be the positive CCC
samples.

##### SynCC

To further enlarge the scale of CCC datasets, we first build a synthetic OCR
dataset. The details of the data construction are presented in Appendix. Then
all OCR samples are taken as the positive CCC samples.

We split EinkCC into train and test sets, and split HWCC into train and dev
sets. All train sets and dev sets are extended by our negative sample
augmentation method of Section 3.3. The data statistics are shown in Table 1.

Dataset | OCR Model | CER
---|---|---
HWCC | PP-OCRv2 | 0.3936
CNN-CTC-CBS | 0.2943
Resnet-CTC | 0.0663
EinkCC | PP-OCRv2 | 0.3115
CNN-CTC-CBS | 0.2325
Resnet-CTC | 0.1661
Table 2: The OCR performance on dev and test sets. Dataset | Model | Sequence level | Binary level
---|---|---|---
P | R | F1 | P | R | F1 | Acc
HWCC | PP-OCRv2 | 0.3590 | 0.4661 | 0.4057 | 0.7508 | 0.9997 | 0.8576 | 0.7510
CNN-CTC-CBS | 0.4584 | 0.5563 | 0.5026 | 0.7604 | 0.9994 | 0.8637 | 0.7635
Resnet-CTC | 0.8157 | 0.8877 | 0.8502 | 0.8578 | 0.9989 | 0.9230 | 0.8751
AiM${}_{\text{wo-PT}}$ | 0.4893 | 0.5602 | 0.5223 | 0.9083 | 0.9306 | 0.9193 | 0.8571
AiM | 0.9735 | 0.9717 | 0.9726 | 0.9926 | 0.9937 | 0.9932 | 0.9898
EinkCC | PP-OCRv2 | 0.1146 | 0.6847 | 0.1964 | 0.2128 | 1.0000 | 0.3509 | 0.4502
CNN-CTC-CBS | 0.1829 | 0.8108 | 0.2985 | 0.2710 | 1.0000 | 0.4264 | 0.6003
Resnet-CTC | 0.2311 | 0.7748 | 0.3561 | 0.3322 | 0.9964 | 0.4987 | 0.7013
AiM${}_{\text{wo-PT}}$ | 0.0568 | 0.2252 | 0.0907 | 0.2025 | 0.6497 | 0.3088 | 0.5676
AiM | 0.2795 | 0.5766 | 0.3765 | 0.4262 | 0.7799 | 0.5512 | 0.8113
Table 3: Main results on dev and test sets. AiM${}_{\text{wo-PT}}$ is the AiM model without the OCR pretraining. Resnet-CTC is trained using the same data source as AiM, which is a strong OCR baseline in a fair comparison. Model | CER
---|---
Resnet-CTC${}_{\text{wo-Syn}}$ | 0.1916
Resnet-CTC${}_{\text{wo-E}}$ | 0.2995
Resnet-CTC${}_{\text{wo-H}}$ | 0.3304
Resnet-CTC | 0.1661
Table 4: Data ablation of the Resnet-CTC OCR performance on the EinkCC test
set.

###  4.2 Evaluation Metrics

We use the character error rate (CER) to evaluate the OCR performance in the
pretraining stage, and compute the following metrics to evaluate the CCC
performance:

##### Sequence level

We use the widely-used metrics, Precision ($P$), Recall ($R$), and $F1$, to
measure the quality of label sequences outputted by AiM.

##### Binary level

We transform label sequences to binary labels (‘right’ or ‘wrong’) to compute
the accuracy of binary classification. We report the $P,R,\text{and }F1$ of
negative samples (i.e. $y=1$) to evaluate model’s ability to identify wrong
answers.

###  4.3 Implementation Details and Baselines

Our training dataset $\\{({\bf I}_{i},{\bf C}_{i},{\bf A}_{i},{\bf
L}_{i},y_{i})\\}_{i=1}^{N}$ contains the image ${\bf I}_{i}$, the handwritten
content ${\bf C}_{i}$, the answer text ${\bf A}_{i}$, the label sequence of
AiM ${\bf L}_{i}$ and the correction result $y_{i}$. We pretrain the image
encoder Resnet on the subset $\\{({\bf I}_{i},{\bf C}_{i})\\}_{i=1}^{N}$, and
train the AiM model on the subset $\\{({\bf I}_{i},{\bf A}_{i},{\bf
L}_{i})\\}_{i=1}^{N}$.

In the OCR pretraining, we follow Liu et al. (2020) to set the number of
convolution blocks to {2,3,1,4}. The height of every input image is set to 128
pixels and the width is scaled to the corresponding value, so the shape of
each pixels block is $128\times 32$. The learning rate is set to 1e-3, the
batch size is 8 and the training epoch is 15.

In the AiM training, the learning rate is 1e-4, the batch size is 8 and the
training epoch is set to 14. The dimension of hidden states $d$ is 768. The
number of image encoder blocks $N_{\text{enc}}$ and fusion module blocks
$N_{\text{fus}}$ in AiM are both 2. We adopt AdamW optimizer Loshchilov and
Hutter (2018) and cosine-annealing strategy. To accelerate training, the
parameters in Resnet are frozen when training AiM. We ignore punctuation in
the text.

OCR-based methods are our baselines, where the correction results are derived
from the post-comparison between the OCR results and answers. The following
OCR models are evaluated: Resnet -CTC is the image encoder of AiM (Resnet)
trained with the CTC function on the subset $\\{({\bf I}_{i},{\bf
C}_{i})\\}_{i=1}^{N}$. PP-OCRv2 (Du et al., 2021) is an open-sourced widely-
used OCR model in Chinese recognition, and we take its text recognition server
model666https://github.com/PaddlePaddle/PaddleOCR. for evaluation. CNN-CTC-CBS
(Liu et al., 2020) is a handwritten text-line recognition model, and we take
the released well-trained model777https://github.com/intel/handwritten-
chinese-ocr-samples. for evaluation. Table 2 shows the OCR performance of
baselines on our datasets.

Model | Sequence Level
---|---
P | R | F1
AiM${}_{\text{wo-Syn}}$ | 0.0212 | 0.1712 | 0.0378
AiM${}_{\text{wo-E}}$ | 0.1857 | 0.3964 | 0.2529
AiM${}_{\text{wo-H}}$ | 0.2660 | 0.4865 | 0.3440
AiM | 0.2795 | 0.5766 | 0.3765
Table 5: Data ablation of the AiM performance at the sequence level on the
EinkCC test set.

(a) Resnet performance at binary level

(b) AiM performance at binary level

Figure 3: Data ablation of the AiM and its baseline performance at the binary
level on the EinkCC test set.

###  4.4 Results and Analyses

The main results are shown in Table 3. We evaluate OCR models at the OCR level
and the binary level. For AiM and its variants, we evaluate them at the
sequence level and the binary level. We describe our observations from the
following perspectives:

  * •

How do OCR models perform in CCC?

  * •

Does AiM improve the performance and whether the two modalities are well
fused?

  * •

Do data augmentation and pretraining work and what’s the impact of data
source?

####  4.4.1 Limitation of OCR-based Method

As shown on Table 3, the recall of OCR-based methods is extremely high (nearly
1.0), but the precision and F1 are much lower. This means that OCR models can
correct almost all wrong handwritten answers but mark many students’ right
text as wrong. Oppositely, AiM improves the precision, F1, and accuracy with
large margins.

####  4.4.2 Influence of Data Source

We conduct an ablation study on the training data. The suffix ‘-wo-E’, ‘-wo-
Syn’, ‘-wo-H’ means the model is trained without the EinkCC training set,
SynCC set and HWCC training set, respectively. The performance of Resnet and
its variants is shown in Table 4 and Figure 3(a). The CER reaches the lowest
level when Resnet is trained with all training sets, which can prove the
necessity of data extension for OCR models. Table 5 and Figure 3(b) show the
performance of AiM. Removing any training set, all metrics at sequence level
drops, as well as the binary level. It means sufficient data is valuable in
the CCC task. Meanwhile, comparing Figure 3(a) and 3(b), although the data
scale decreases, the performance of AiM is always better than Resnet, which
demonstrates the AiM is more robust and stable to data variation.

Notice that the model performance are always better on the HWCC dev set than
it on the EinkCC test set. This is because all negative samples of HWCC are
synthetic samples that are augmented in the same way. This suggests that AiM
learns the error patterns well from our negative sample augmentation.

Image: |  |
---|---|---
Handwritten: | 骆驼祥子 | 铺垫
Answer: | 骆驼祥子 | 铺垫
OCR result: | 骆马它祥子 | 铺挚
AiM output labels: | O O O O O | O O O
Correction label: | ✓ | ✓
Correcting by OCR: | ✕ | ✕
Correcting by AiM: | ✓ | ✓
Table 6: Examples on EinkCC test set. Resnet gives the wrong outputs in all examples. The mistakes made by Resnet are shown in bold with underline, while AiM predicts correctly. Notes that the first label is for the placeholder ‘¡BLK¿’. |  |  | Sequence level | Binary level
---|---|---|---|---
$N_{\text{enc}}$ | $N_{\text{fus}}$ | Text Self-Att in Fusion | P | R | F1 | P | R | F1 | Acc
1 | 1 | ✕ | 0.1150 | 0.3513 | 0.1733 | 0.3515 | 0.7099 | 0.4702 | 0.7623
1 | 1 | ✓ | 0.1773 | 0.4775 | 0.2585 | 0.3794 | 0.7908 | 0.5128 | 0.7727
2 | 2 | ✕ | 0.1235 | 0.3694 | 0.1851 | 0.3630 | 0.7161 | 0.4818 | 0.7667
2 | 2 | ✓ | 0.2795 | 0.5766 | 0.3765 | 0.4262 | 0.7799 | 0.5512 | 0.8113
3 | 3 | ✓ | 0.2501 | 0.5495 | 0.3436 | 0.3682 | 0.7398 | 0.4917 | 0.7727
Table 7: An ablation study of components in AiM on EinkCC test set.

Figure 4: Cross-modal attention visualization. The figure shows the cross-
attention scores between characters in answer and the pixels blocks in image.

####  4.4.3 Effectiveness of Multimodal

As shown in Table 3, with AiM, the recall of negative label decreases888We
perform an analysis on the recall drop in Appendix. but the precision and
F1-score increase significantly at the binary level. We give two examples on
EinkCC in Table 6 to further analyze the effectiveness of AiM. Obviously,
Resnet generates wrong outputs while AiM makes no mistake. In the first
example, Resnet considers one left-right structured character as two
independent characters, which never happens when manually correcting. In the
second example, Resnet gets confused by the shape-similar characters. AiM
predicts successfully in these examples. More predictions on the dev and test
sets are presented in Appendix to illustrate that AiM is able to identify all
situations of the mismatch between the image and answer text.

Comparing AiM and AiM${}_{\text{wo-PT}}$ in Table 3, without the pretraining,
the performance of AiM drops significantly, even worse than Resnet. It
demonstrates that visual modality features can be better understood through
pretraining.

####  4.4.4 Learning of Modal Fusion

We conduct experiments to explore the contribution of each component.
Experimental results are shown in Table 7. AiM reaches a better performance
when encoding text information with the self-attention than only using token
embeddings. This suggests that self-attention brings better textual
representation. Meanwhile, stacking each module with a proper number of layers
helps AiM capture interactions between modalities. We visualize the cross-
modal attention scores in Figure 4. We find that characters manage to attend
the most relative pixels block. For instance, in this sub-figure (a), the
character “集” appears twice in different positions, and the model captures the
context feature and both of them attend the corresponding pixels block in
order. It shows that cross-modal attention can match multimodal information
effectively.

##  5 Related Work

###  5.1 OCR Model

OCR models have evolved for a long period. Researchers usually used hybrid CNN
and RNN architectures Breuel (2017) with CTC (Connectionist temporal
classification) loss Graves et al. (2006). For handwritten Chinese text
recognition (HCTR) task, Liu et al. (2020) has achieved the state-of-the-art
performance. They use the simple end-to-end CNN-CTC method and ease the
overfitting problem with a high-rate dropout strategy.

###  5.2 Multimodal Model

Multimodal models have attracted more and more attention and have been applied
in many fields, such as visual question answering (Antol et al., 2015),
Chinese spell checking (Xu et al., 2021; Li et al., 2022) and other
applications (Toto et al., 2021; Hu et al., 2021; Aguilar et al., 2019).
Unsupervised pretraining (Kenton and Toutanova, 2019) has provided informative
representations and fine-tuning techniques (Cui et al., 2019; Li et al., 2021)
have brought further performance gains. Several large-scale pre-trained models
are utilized in multimodal models Anderson et al. (2018); Toto et al. (2021);
Xu et al. (2021). To capture multimodal features, one common method is to
concatenate encoders’ output from two sides and then feed to the downstream
multi-layer perceptrons Nie et al. (2021). While some works use attention
mechanism to get fusion representation Lu et al. (2019); Tan and Bansal
(2019); Tsai et al. (2019).

##  6 Conclusion

In this paper, we propose a multimodal model AiM to effectively correct
Chinese cloze tests. AiM employs cross-modal attention to understand the
correlation between modalities. We collect data from different sources and
develop a data augmentation method. Experiments show that AiM outperforms
traditional OCR-based methods with over 11% accuracy improvements.

## References

  * Aguilar et al. (2019) Gustavo Aguilar, Viktor Rozgić, Weiran Wang, and Chao Wang. 2019.  Multimodal and multi-view models for emotion recognition.  _arXiv preprint arXiv:1906.10198_.
  * Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018.  Bottom-up and top-down attention for image captioning and visual question answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 6077–6086.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015.  VQA: visual question answering.  _CoRR_ , abs/1505.00468.
  * Breuel (2017) Thomas M Breuel. 2017.  High performance text recognition using a hybrid convolutional-lstm implementation.  In _2017 14th IAPR international conference on document analysis and recognition (ICDAR)_ , volume 1, pages 11–16. IEEE.
  * Cui et al. (2019) Baiyun Cui, Yingming Li, Ming Chen, and Zhongfei Zhang. 2019.  Fine-tune BERT with sparse self-attention mechanism.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3548–3553, Hong Kong, China. Association for Computational Linguistics.
  * Du et al. (2021) Yuning Du, Chenxia Li, Ruoyu Guo, Cheng Cui, Weiwei Liu, Jun Zhou, Bin Lu, Yehua Yang, Qiwen Liu, Xiaoguang Hu, Dianhai Yu, and Yanjun Ma. 2021.  Pp-ocrv2: Bag of tricks for ultra lightweight OCR system.  _CoRR_ , abs/2109.03144.
  * Graves et al. (2006) Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In _Proceedings of the 23rd international conference on Machine learning_ , pages 369–376.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778.
  * Hu et al. (2021) Chuanbo Hu, Minglei Yin, Bin Liu, Xin Li, and Yanfang Ye. 2021.  Detection of illicit drug trafficking events on instagram: A deep multimodal multilabel learning approach.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 3838–3846.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Li et al. (2022) Yinghui Li, Qingyu Zhou, Yangning Li, Zhongli Li, Ruiyang Liu, Rongyi Sun, Zizhen Wang, Chao Li, Yunbo Cao, and Hai-Tao Zheng. 2022.  The past mistake is the future wisdom: Error-driven contrastive probability optimization for Chinese spell checking.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3202–3213, Dublin, Ireland. Association for Computational Linguistics.
  * Li et al. (2021) Zhongli Li, Qingyu Zhou, Chao Li, Ke Xu, and Yunbo Cao. 2021.  Improving BERT with syntax-aware local attention.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 645–653, Online. Association for Computational Linguistics.
  * Liu et al. (2020) Brian Liu, Xianchao Xu, and Yu Zhang. 2020.  Offline handwritten chinese text recognition with convolutional neural networks.  _arXiv preprint arXiv:2006.15619_.
  * Liu et al. (2011) Cheng-Lin Liu, Fei Yin, Da-Han Wang, and Qiu-Feng Wang. 2011.  Casia online and offline chinese handwriting databases.  In _2011 International Conference on Document Analysis and Recognition_ , pages 37–41. IEEE.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Decoupled weight decay regularization.  In _International Conference on Learning Representations_.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  _Advances in neural information processing systems_ , 32.
  * Nie et al. (2021) Yixin Nie, Linjie Li, Zhe Gan, Shuohang Wang, Chenguang Zhu, Michael Zeng, Zicheng Liu, Mohit Bansal, and Lijuan Wang. 2021.  Mlp architectures for vision-and-language modeling: An empirical study.  _arXiv preprint arXiv:2112.04453_.
  * Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009.  Design challenges and misconceptions in named entity recognition.  In _Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL-2009)_ , pages 147–155, Boulder, Colorado. Association for Computational Linguistics.
  * Tan and Bansal (2019) Hao Tan and Mohit Bansal. 2019.  Lxmert: Learning cross-modality encoder representations from transformers.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5100–5111.
  * Toto et al. (2021) Ermal Toto, ML Tlachac, and Elke A Rundensteiner. 2021.  Audibert: A deep transfer learning multimodal classification framework for depression screening.  In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 4145–4154.
  * Tsai et al. (2019) Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2019, page 6558. NIH Public Access.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wu et al. (2013) Shih-Hung Wu, Chao-Lin Liu, and Lung-Hao Lee. 2013.  Chinese spelling check evaluation at SIGHAN bake-off 2013.  In _Proceedings of the Seventh SIGHAN Workshop on Chinese Language Processing_ , pages 35–42, Nagoya, Japan. Asian Federation of Natural Language Processing.
  * Xu et al. (2021) Heng-Da Xu, Zhongli Li, Qingyu Zhou, Chao Li, Zizhen Wang, Yunbo Cao, Heyan Huang, and Xian-Ling Mao. 2021.  Read, listen, and see: Leveraging multimodal information helps Chinese spell checking.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 716–728, Online. Association for Computational Linguistics.

##  Appendix A Annotation Process

We present the process of annotation in our sequence labeling task in Figure
5. We put a label to individual character in textual answer referring to hand-
written content and each label stands for a kind of editing operation.

Figure 5: Examples of sequence labeling annotation. For the first sample, to
convert the answer to the hand-written content, the last character in the
answer “植” should be replaced by “直”. So the label ‘B-sub’ is annotated to “植”
and ‘O’ for the rest. For the latter one, the character “入” and “底” do not
appear in the hand-written content so the labels are both ’B-del’. Moreover,
the character “唐” should be inserted at the first position in answer for
conversion, so we put a ‘B-add’ label to the placeholder ‘¡BLK¿’.

##  Appendix B SynCC Data Construction

To build a synthetic OCR dataset, we first collect handwritten character
images from the HWDB 1.0 set of CASIA-HWDB Liu et al. (2011) and our
educational application. Then the character images are spliced together into
text-line images according to the sentences or clauses of our online essay
corpus. We also replace characters with shape-similar ones and their images to
enhance OCR models to recognize them. The synthetic OCR examples are shown on
Table 8. These samples are taken as the positive CCC samples, and negative
sample augmentation introduced in Section 3.3 is applied to construct SynCC
dataset.

Image: |
---|---
Content: | 是一个宁静的世界
Image: |
Content: | 是一个宁静的曲界
Image: |
Content: | 阴森森的面孔
Image: |
Content: | 阴林森的面孔
Image: |
Content: | 心烦意乱中又收到了一页书笔
Image: |
Content: | 心烦意乱中又政到了一页书笔
Image: |
Content: | 从此孤独的我不再孤独
Image: |
Content: | 从此似独的我不再孤独
Table 8: Synthetic OCR examples of SynCC.

##  Appendix C Recall Drop on EinkCC

Table 3 shows that AiM improves the correction precision and F1 but the recall
drops on EinkCC. We present several false-positive predictions on Table 9. All
examples are annotated as wrong by teachers but are predicted as right by AiM.
In the first example, the student writes down “斯” but there is a extremely
large margin between the left and right sides. In the rest examples, students
write down shape-similar characters but AiM can’t detect them and tend to
predict as right. Thus, even though we use a confusion set to enhance AiM to
detect shape-similar characters, there is still a lot of room for improvement
in this regard.

Image: |
---|---
Answer: | 弗洛斯特
Labels: | O O O B-sub O
Image: |
Answer: | 清
Labels: | O B-sub
Image: |
Answer: | … 已经满是桃儿 …
Labels: | … O O O B-sub O O …
Image: |
Answer: | 不畏浮云遮望眼
Labels: | O O O O O B-sub O O
Table 9: False-positive predictions on EinkCC test set. All above answer
characters are predicted to ‘O’ by AiM. The character marked by the underline
indicates why the correction result should be wrong.

##  Appendix D Predictions of AiM

Table 10 shows several cases of AiM’s predictions. AiM is able to identify all
kinds of modifications between the answer and hand-written content including
substitutions, deletions and insertions, which proves that it’s reasonable to
format the fill-in-the-blank assignments correction task to sequence labeling.
In this way, AiM can not only indicate the correctness of students’ answers,
but also can locate where the errors occur.

Image: |
---|---
Handwritten Content: | 前 认 为 的 早 5 亿 年
Correct Answer: |  麻 前 镅 认 为 的 早 5 亿 材 嚓 年
AiM Labels: | O B-del O B-del I-del O O O O O O B-del I-del O
AiM Predictions: | O B-del O B-del I-del O O O O O O B-del I-del O
Image: |
Handwritten Content: | 《 中 度 》
Correct Answer: | 《 中 庸 》
AiM Labels: | O O O B-sub O
AiM Predictions: | O O O B-sub O
Image: |
Handwritten Content: | 自 缘 身 在 最 高 层
Correct Answer: | 不 畏 浮 云 遮 望 眼
AiM Labels: | O B-sub I-sub I-sub I-sub I-sub I-sub I-sub
AiM Predictions: | O B-sub I-sub I-sub I-sub I-sub I-sub I-sub
Image: |
Handwritten Content: | 自 己 寻 找 工 作
Correct Answer: |  自 寻 工 作
AiM Labels: | O B-add B-add O O
AiM Predictions: | O B-add B-add O O
Table 10: Several outputs of AiM on dev and test sets. Some answers are
generated through our data augmentation. AiM can identify editing operations
between answers and hand-written content. Characters are shown in bold with
underline if the corresponding label is not ‘O’.