# Hybrid Event Shaping
to Stabilize Periodic Hybrid Orbits

James Zhu, Nathan J. Kong, George Council, and Aaron M. Johnson This material
is based upon work supported by the National Science Foundation under grant
#CMMI-1943900 and #ECCS-1924723 as well as the U.S. Army Research Office under
grant #W911NF-19-1-0080. The views and conclusions contained in this document
are those of the authors and should not be interpreted as representing the
official policies, either expressed or implied, of the National Science
Foundation, the Army Research Office, or the U.S. Government. The U.S.
Government is authorized to reproduce and distribute reprints for Government
purposes notwithstanding any copyright notation herein.All authors are with
the Department of Mechanical Engineering, Carnegie Mellon University,
Pittsburgh, PA, USA<EMAIL_ADDRESS>
###### Abstract

Many controllers for legged robotic systems leverage open- or closed-loop
control at discrete hybrid events to enhance stability. These controllers
appear in several well studied phenomena such as the Raibert stepping
controller, paddle juggling and swing leg retraction. This work introduces
hybrid event shaping (HES): a generalized method for analyzing and producing
stable hybrid event controllers. HES utilizes the saltation matrix, which
gives a closed-form equation for the effect that hybrid events have on
stability. We also introduce shape parameters, which are higher order terms
that can be tuned completely independently from the system dynamics to promote
stability. Optimization methods are used to produce values of these parameters
that optimize a stability measure. Hybrid event shaping captures previously
developed control methods while also producing new optimally stable
trajectories without the need for continuous-domain feedback.

##  I Introduction

In general, the walking and running gaits of legged robots are naturally
unstable and challenging to control. Hybrid systems such as these are
difficult to work with due to the discontinuities that occur in state and
dynamics at hybrid events, such as toe touchdown. These discontinuities
violate assumptions of standard controllers designed for purely continuous
systems, and work is ongoing to adapt these controllers for hybrid systems [1,
2]. One strategy for hybrid control is to cancel out the effects of hybrid
events by working with an invariant subsystem [3, 4, 5]. We propose instead
that the effects of hybrid events are valuable due to rich control properties
that can be used to stabilize trajectories of a hybrid system.

Several works have examined the utility of controlling hybrid event conditions
to improve system stability without any closed-loop continuous-domain control
[6, 7, 8]. For example, [6] found that for the paddle juggler system, paddle
acceleration at impact uniquely determines the local stability properties of a
periodic trajectory, Fig. 1. Other works [5, 9] generated open-loop swing leg
trajectories that produced deadbeat hopping of a SLIP-like system. Each of
these works found that controlling a hybrid system only at the moment of a
hybrid event is sufficient to provide stabilization. So far, however, these
results have only been produced for each specific problem structure and a
clear connection between these works has yet to be established.

In this work, we propose the concept of hybrid event shaping (HES), which
describes how hybrid event parameters can be chosen to affect the stability
properties of a periodic orbit. We also propose methods to produce values of
these hybrid event parameters to optimize a stability measure of a trajectory.
This approach is tested on both existing examples from [6, 7] as well as on a
new bipedal robot controller.

Figure 1: The paddle juggler system [6] has no control authority while the
ball is in the air. The paddle acceleration at impact determines the
convergence/divergence of the system from initial points (cyan dots) to the
final states (magenta stars) after 5 cycles. This example underscores how
hybrid event shaping can stabilize a periodic hybrid system.

##  II Preliminaries

###  II-A Hybrid system

Hybrid systems are a class of dynamical systems which exhibit both continuous
and discrete dynamics [10, 11]. Following the adaptation of [12] in [2], we
define a $C^{r}$ hybrid dynamical system for continuity class
$r\in\mathbb{N}_{>0}\cup\\{\infty,\omega\\}$ as a tuple
$\mathcal{H}:=(\mathcal{J},\mathnormal{\Gamma},\mathcal{D},\mathcal{F},\mathcal{G},\mathcal{R})$
where:

  1. 1.

$\mathcal{J}:=\\{I,J,\ldots,K\\}\subset\mathbb{N}$ is the set of discrete
modes.

  2. 2.

$\mathnormal{\Gamma}\subset\mathcal{J}\times\mathcal{J}$ is the set of
discrete transitions forming a directed graph structure over $\mathcal{J}$.

  3. 3.

$\mathcal{D}:=\amalg_{I\in\mathcal{J}}D_{I}$ is the collection of domains
where $D_{I}$ is a $C^{r}$ manifold with corners [13].

  4. 4.

$\mathcal{F}:=\amalg_{I\in\mathcal{J}}F_{I}$ is a collection of time-varying
vector fields with control inputs $F_{I}:\mathbb{R}\times D_{I}\times
U_{I}\rightarrow TD_{I}$ where $U_{I}$ is the space of allowable control
inputs.

  5. 5.

$\mathcal{G}:=\amalg_{(I,J)\in\mathnormal{\Gamma}}G_{(I,J)}(t)$ is a
collection of guards where $G_{(I,J)}(t)\subset D_{I}\times U_{I}$ for each
$(I,J)\in\mathnormal{\Gamma}$ is defined as a sublevel set of a $C^{r}$
function, i.e. $G_{(I,J)}(t)=\\{(x,u)\in D_{I}\times
U_{I}|g_{(I,J)}(t,x,u)\leq 0\\}$.

  6. 6.

$\mathcal{R}:\mathbb{R}\times\mathcal{G}\rightarrow\mathcal{D}$ is a $C^{r}$
map called the reset map that restricts as
$R_{(I,J)}:=\mathcal{R}|_{G_{(I,J)}(t)}:G_{(I,J)}(t)\rightarrow D_{J}$ for
each $(I,J)\in\mathnormal{\Gamma}$.

An execution of a hybrid system [12] begins at an initial state $x_{0}\in
D_{I}$. With a particular input $u_{I}(t,x)$, the system follows the dynamics
$F_{I}$ on $D_{I}$. If the system reaches the guard surface $G_{(I,J)}$, the
reset map $R_{(I,J)}$ is applied and the system continues in domain $D_{J}$
with the corresponding dynamics defined by $F_{J}$. An execution of a hybrid
system is defined over a hybrid time domain which is a disjoint union of
intervals $\amalg_{j\in\mathcal{N}}[\underline{t}_{j},\bar{t}_{j}]$. The flow
$\phi(t,t_{0},x_{0})$ describes how the system evolves from some initial time
$t_{0}$ and state $x_{0}$ for some length of execution time $t$.

These hybrid systems may exhibit complex behavior including sliding [14],
branching [11], and Zeno phenomena [15]. To preserve the validity of local
linearization theory, which is core to this work, we assume that transition
surfaces are isolated with transverse intersections [11, 12] and no Zeno
executions occur.

###  II-B Saltation matrix

Hybrid systems can be divided into continuous domains and discrete hybrid
events. For each of these components, the linearized variational equations
[16] describe how perturbations of state at the beginning of the phase evolve
to the end of the phase. In continuous domains, variational equations can be
derived [16] and discretized into a mapping
$A_{I}(\bar{t}_{j}-\underline{t}_{j},\underline{t}_{j},x(\underline{t}_{j}))$
equivalent to the linearized discrete dynamics matrix in $x_{k+1}\approx
A_{I}x_{k}+B_{I}u$. The variational equations of hybrid events are described
by the saltation matrix, which maps perturbations at an infinitesimal time
step before impact to an infinitesimal time step afterward [17]. This
saltation matrix, $\Xi_{(I,J)}$, describes the transition between modes $I$
and $J$ that occurs at time $\bar{t}_{i}$ with $x(\bar{t}_{i})\in G_{(I,J)}$
and with some input $u(\bar{t}_{i})$. Following the formulation from [18, 19],
the saltation matrix is,

$\Xi=D_{x}R+\frac{(F_{J}-D_{x}R\cdot F_{I}-D_{t}R)D_{x}g}{D_{t}g+D_{x}gF_{I}}$
(1)

where

$\displaystyle\Xi$
$\displaystyle:=\Xi_{(I,J)}(\bar{t}_{i},x(\bar{t}_{i}),u(\bar{t}_{i})),\;R:=R_{(I,J)}(\bar{t}_{i},x(\bar{t}_{i}),u(\bar{t}_{i})),$
$\displaystyle g$
$\displaystyle:=g_{(I,J)}(\bar{t}_{i},x(\bar{t}_{i}),u(\bar{t}_{i})),\;F_{I}:=F_{I}(\bar{t}_{i},x(\bar{t}_{i}),u(\bar{t}_{i})),$
$\displaystyle F_{J}$
$\displaystyle:=F_{J}(\underline{t}_{i+1},R_{(I,J)}(\bar{t}_{i},x(\bar{t}_{i}),u(\bar{t}_{i}))$

The saltation matrix approximates the first-order change in perturbations in
state before the $i^{th}$ hybrid event $\delta x(\bar{t}_{i})$ to
perturbations afterward $\delta x(\underline{t}_{i+1})$ [18] such that:

$\delta x(\underline{t}_{i+1})=\Xi\delta x(\bar{t}_{i})+\text{h.o.t.}$ (2)

where h.o.t. represents higher order terms.

###  II-C Periodic Stability Analysis

A dynamical system has a periodic trajectory (orbit) with period $T$ if for
some initial condition $x_{0}$, there exists a solution $\phi(t,t_{0},x_{0})$
where $\phi(t,t_{0},x_{0})=\phi(t+kT,t_{0},x_{0})$ for all $t$ and $k$ [17].
Perturbations $\delta x_{0}$ around $x_{0}$ can be mapped to perturbations
$\delta x_{T}$ after period $T$ by a linearized mapping known as the monodromy
matrix, $\Phi$ [20]:

$\Phi=\frac{\partial\phi(T+t_{0},t_{0},x_{0})}{\partial x_{0}}$ (3)

such that to the first order, $\delta x_{T}\approx\Phi\delta x_{0}$.

Following a formulation similar to [21], the monodromy matrix can be computed
by sequentially composing the linearized variational equations in each
continuous domain ($A$) and the saltation matrices ($\Xi$) at each hybrid
event. For a hybrid periodic orbit with $N$ domains, the monodromy matrix can
be formulated as:

$\displaystyle\Phi=\Xi_{(N,1)}A_{N}\dots\Xi_{(2,3)}A_{2}\Xi_{(1,2)}A_{1}$ (4)

The monodromy matrix determines local asymptotic orbital stability (which we
refer to simply as stability). For nonautonomous systems, stability is
determined by the maximum magnitude of the eigenvalues, $\max(|\lambda|)$
[17]. We refer to this as the stability measure, $\psi$, where a trajectory is
stable when $\psi<1$. Autonomous systems always have an eigenvalue that is
equal to 1 since for non-time varying dynamics, perturbations along the flow
of the orbit will by definition map back to themselves after period $T$ [17].
Assuming non-convergence in this direction is allowable, $\psi$ for autonomous
systems is based on the remaining eigenvalues.

##  III Methods

###  III-A Hybrid event shaping

Hybrid events can greatly affect the stability properties of an orbit because
of the discontinuous changes that are made to perturbations. The saltation
matrix allows for an explicit understanding of how to perform “hybrid event
shaping” (HES), i.e. choosing hybrid event parameters (such as timing, state,
input, and higher order “shape parameters”) to improve the stability of a
periodic trajectory. The key insight is that hybrid event shaping introduces a
generalizable method to stabilize hybrid systems that is independent from
continuous-domain feedback, but that can work in concert with it.

In general, the open-loop continuous variational equations of a hybrid system
are functions of initial and final time, initial state, and system dynamics.
However, it is challenging to alter any of these parameters because changes
will propagate through the rest of the trajectory and periodicity may be
violated, though we present a trajectory optimization method below that can
handle this. The saltation matrix is a function of nominal event time, state
and dynamics, but additionally may be a function of higher order shape
parameters $h$ that do not influence the dynamics of the system. These
parameters arise from the derivatives of the guards ($D_{x}g$ and $D_{t}g$)
and reset maps ($D_{x}R$ and $D_{t}R$) but are not present in the guard, reset
map, or vector field definitions themselves. Therefore, shape parameters have
absolutely no effect on the nominal trajectory and can be chosen completely
freely.

One example of a shape parameter is the angular velocity of a massless leg of
a spring-loaded inverted pendulum. Since a massless leg does not induce any
torque in the air or forces at touchdown, only the position of the leg at
touchdown affects the trajectory of the body. However, leg velocity appears in
the saltation matrix and has a significant effect on orbital stability [7].

For more complex models of robots, there may not be any physical shape
parameters that can be tuned. For example, legged robots with non-massless
legs can not vary leg velocity at impact without also changing the periodic
orbit. These cases can be handled by adjusting the trajectory at the same time
as applying HES, as we show in Sec. III-E, or by adding additional virtual
hybrid events.

###  III-B Virtual hybrid events

Certain control systems naturally have discontinuities in control inputs, such
as bang-bang control, sliding mode control, or systems with actuators that
have discretized inputs (i.e. on-off). These discontinuities in control cause
instantaneous changes in the dynamics of the system, resulting in a virtual
(as opposed to physical) hybrid event. Virtual hybrid events act no
differently than physical hybrid events and induce saltation matrices with
shape parameters to be tuned for stability. Even for systems where
discontinuous control inputs are not necessary, the addition of virtual
saltation matrices and shape parameters allows for a greater authority to
improve stability.

###  III-C Stability measure derivative

Our goal is to determine the optimal choice of hybrid event parameters that
minimizes the stability measure of a trajectory. Since directly computing
eigenvalues in closed-form is not generally feasible, one solution is to use
numerical methods to perform optimization [22]. However, this strategy becomes
untenable for high dimensional problems. With the saltation matrix available
in closed form (1), derivatives of the stability measure can be directly
computed, allowing for the use of more efficient optimization methods to
select hybrid event parameters.

Assuming that the monodromy matrix $\Phi$ depends continuously on each shape
parameter $h_{n}$, the eigenvalues of $\Phi$ are always continuous with
respect to $h_{n}$ [23]. For a diagonalizable $\Phi$, the derivative of the
eigenvalues with respect to $h_{n}$ can be computed in closed form [24].
Assume that matrix $\Phi(h_{n})$ has simple (non-repeating) eigenvalues,
$\lambda_{1},\ldots,\lambda_{N}$, and let $\boldsymbol{j}_{i}$ and
$\boldsymbol{k}_{i}$ denote the left and right eigenvectors associated with
$\lambda_{i}$. Then the derivative $\frac{d\lambda_{i}}{dh_{n}}$ is:

$\displaystyle\frac{d\lambda_{i}}{dh_{n}}=\boldsymbol{k}_{i}^{\prime}\frac{d\Phi}{dh_{n}}\boldsymbol{j}_{i}$
(5)

For matrices with eigenvalues that repeat, the derivatives of the repeated
eigenvalues can be calculated similarly with a matrix of associated
eigenvectors [24].

$\frac{d\Phi}{dh_{n}}$ can be found using the derivative product rule, which
simplifies if each shape parameter only appears in one saltation matrix. We
make this assumption here to improve computational efficiency, but it is not
required generically. Without loss of generality, take
$\frac{d\Xi_{(1,2)}}{dh_{n}}\neq 0$, so that:

$\displaystyle\frac{d\Phi}{dh_{n}}=\Xi_{(N,1)}A_{N}\dots
A_{2}\frac{d\Xi_{(1,2)}}{dh_{n}}A_{1}$ (6)

Substituting (6) into (5) allows us to compute the derivative of the stability
measure with respect to each of the shape parameters. Eq. (5) is not valid for
non-diagonalizable monodromy matrices. However, the guaranteed continuity of
the stability measure allows for finite-difference methods to be used in any
non-diagonalizable cases.

The derivative computation from (6) can be adapted for changes in $x$ and $t$
as well. Without loss of generality, consider again $\Xi_{(1,2)}$. For hybrid
event time $t_{(1,2)}$, the derivative $\frac{d\Xi_{(1,2)}}{dt_{(1,2)}}$ can
be computed in closed-form. Additionally, the derivatives
$\frac{dA_{1}}{dt_{(1,2)}}$ and $\frac{dA_{2}}{dt_{(1,2)}}$ are non-zero and
can be computed through numerical methods. The product rule expansion of
$\frac{d\Phi}{dt_{(1,2)}}$ consists of additional terms compared to (6) but
otherwise can be computed similarly. $\frac{d\Phi}{dx_{(1,2)}}$ for hybrid
event state $x_{(1,2)}$ can be computed this same way.

###  III-D Shape parameter stability optimization

Optimization techniques [25] are able to select optimal hybrid event
parameters that minimize the stability measure. Two optimization methods are
presented here: the first optimizing the shape parameters without affecting
the dynamics of the nominal orbit, and the second optimizing both the hybrid
events and periodic orbit simultaneously.

Shape parameters are powerful because they do not appear in the dynamics of
the system and have no effect on the nominal trajectory. This means that for a
given periodic trajectory, the shape parameters can be chosen freely. We use
an optimization framework to choose these shape parameters with the goal of
optimizing the stability measure $\psi(\Phi(h))$ of a trajectory,

$\begin{array}[]{cl}\underset{h_{1:M}}{\operatorname{minimize}}&\psi(\Phi(h))\end{array}$
(7)

The ability to compute derivatives of the stability measure allows for this
optimization to be more computationally efficient. The examples below show how
this optimization method is able to reproduce swing leg retraction in a one-
legged hopper system by determining optimal inputs of shape parameters to
minimize the stability measure.

###  III-E Trajectory optimization with hybrid event shaping

Some systems do not have shape parameter terms in their saltation matrices or
do not have enough to sufficiently improve stability. In these cases, we can
change the trajectory of the system itself so that the timing, state, and
input parameters of the continuous variational matrices and saltation matrices
improve stability properties. However, it must be ensured that the dynamics,
periodicity, and other constraints of the system are obeyed.

Trajectory optimization methods are a class of algorithms that aim to minimize
a cost function while satisfying a set of constraints [26]. For dynamical
systems, these costs are generally functions of states and inputs, with
constraints imposed on system dynamics and any other physical limits [27]. For
specific problems, other aspects of the system may be added into the cost or
constraint functions such as design parameters or minimizing time [28, 29].
Here we propose including the stability measure in the cost function to search
for optimally stable trajectories. Eq. (8) gives the simplest form of this
trajectory optimization problem, with periodicity and dynamics constraints
being enforced, where dynamics constraints obey continuous dynamics in each
domain and reset mappings at each hybrid event [30].

$\begin{array}[]{cl}\underset{x_{1:N},u_{1:N-1},t_{1:N-1},h_{1:M}}{\operatorname{minimize}}&\psi(\Phi(x,t,h))\\\
\text{ subject to }&x_{N}=x_{1}\\\
&x_{i+1}=\phi(t_{i+1}-t_{i},t_{i},x_{i})\end{array}$ (8)

We solve this problem using a direct collocation optimization with a multi-
phase method to handle hybrid events [26, 27, 31]. This trajectory
optimization method is capable of generating stable trajectories that would
otherwise not be intuitively apparent.

##  IV Examples and Results

Here we demonstrate how HES can improve the stability of periodic trajectories
for a variety of hybrid systems without any use of continuous-domain feedback
control. While continuous-domain feedback could be implemented into any of
these systems and should be in practice, these examples emphasize the
stabilization capabilities of HES alone.

The first two examples describe how previously discovered results, paddle
juggling [6] and swing leg retraction [7], fit into an HES framework. The
final example demonstrates how HES can be used even without any shape
parameters and how virtual hybrid events can help stabilize a more complicated
biped walking system.

###  IV-A Paddle juggler

The paddle juggler system [6], bouncing a ball with a paddle, is known to be
stabilized by impacting the ball with a paddle acceleration in a range of
negative values, (10), Fig. 1. The system state consists of the ball’s
vertical position and velocity such that $x=[x_{B},\dot{x}_{B}]^{T}$. This
periodic hybrid system can be defined with two hybrid domains (descent and
ascent) connected by two guards (impact and apex). The domain $D_{1}$
represents the ball’s aerial descent phase where $\dot{x}_{B}<0$ and $D_{2}$
represents the ball’s aerial ascent phase where $\dot{x}_{B}>0$. The guard set
$g_{(1,2)}:=x_{B}-x_{P}\leq 0$ is defined when the ball impacts the paddle,
where the paddle follows a twice differentiable trajectory $x_{P}(t)$. The
reset map $R_{(1,2)}$ is defined by a partially elastic impact law,
$R_{(1,2)}([x_{B},\dot{x}_{B}]^{T})=[x_{B},(1+\alpha)\dot{x}_{P}-\alpha\dot{x}_{B}]^{T}$,
with a coefficient of restitution $\alpha$. The continuous dynamics in both
domains follow unactuated ballistic motion:
$\mathcal{F}_{1}=\mathcal{F}_{2}=[\dot{x}_{B},-g]^{T}$, where $g$ is the
acceleration due to gravity.

Using these definitions, the saltation matrix (1) between domains $1$ and $2$
is constructed:

$\Xi_{(1,2)}=\begin{bmatrix}-\alpha&0\\\
\frac{(1+\alpha)\cdot(\ddot{x}_{P}+g)}{\dot{x}_{P}-\dot{x}_{B}}&-\alpha\end{bmatrix}$
(9)

Observe that $\ddot{x}_{P}$ appears in the saltation matrix even though it
does not appear anywhere in the definition of the guards, reset maps, or
vector fields of the system, making it a shape parameter that can be chosen
independently from the periodic orbit.

The guard set $g_{(2,1)}:=\dot{x}_{B}\leq 0$ is defined when the ball reaches
the apex of its ballistic motion. Its reset map $R_{(2,1)}$ is identity and
there is no change in dynamics. Therefore, $\Xi_{(2,1)}$ is identity and has
no effect on the variations of the system.

The continuous variational matrices of the system can be written exactly in
closed form: $A_{1}(T)=A_{2}(T)=\bigl{[}\begin{smallmatrix}1&T/2\\\
0&1\end{smallmatrix}\bigr{]}$ where $T$ is the total time spent in the air and
also the period of the system. The periodicity of the system means the ball
spends an equal time, $T/2$, ascending as descending.

The monodromy matrix, $\Phi$ is constructed by composing together the
continuous variational matrices and saltation matrices such that
$\Phi=\Xi_{(2,1)}A_{2}\Xi_{(1,2)}A_{1}$.

For a given periodic bouncing trajectory, the monodromy matrix $\Phi$ is
almost fully defined, with the exception of the shape parameter,
$\ddot{x}_{P}$ in $\Xi_{(1,2)}$. Given the 2-dimensional state space of this
problem, the eigenvalues for any given $\ddot{x}_{P}$ value can be solved for
explicitly. We can then solve exactly for where $\psi(\ddot{x}_{P})<1$,
giving:

$\displaystyle-2g\frac{1+\alpha^{2}}{(1+\alpha)^{2}}<\ddot{x}_{P}<0$ (10)

This is confirmed in [6], where the simple dynamics of the system allowed the
return map to be computed explicitly without the saltation matrix. However,
that computation is not generally possible for more complex dynamics.

###  IV-B 2D hopper

The spring-loaded inverted pendulum (SLIP) is a popular model for dynamic
legged locomotion [32, 33, 34]. This simple hopper model is effective at
capturing dynamic properties of animal and robot locomotion [35] and has been
used as a test bed for different types of hybrid controllers [36].

####  IV-B1 2D hopper hybrid model

Figure 2: SLIP-like system with actuator and damper in parallel.

Consider a point mass body with a massless leg consisting of a spring, damper,
and linear actuator all in parallel, Fig. 2. This system has two domains
(flight and stance) connected by two guards (touchdown and liftoff). The
actuator is activated while in the air to preload the spring, but immediately
releases at touchdown and provides no forces during stance. For a periodic
trajectory to occur, the actuator must preload the same amount of energy that
is dissipated by the damper during stance. The only control authority that
exists is of the leg angle in the air, which only affects the dynamics of the
body at touchdown.

During flight ($D_{1}$), the state of the hopper is represented by the
horizontal velocity, vertical position, and vertical velocity of the body:
$x=[\dot{x}_{B},y_{B},\dot{y}_{B}]^{T}$. Horizontal position is not included
because it is not periodic. During stance, the body position $x_{s}$ and
$y_{s}$ is defined with the toe at the origin. Horizontal position is added
back into the state of the hopper such that
$x=[x_{s},\dot{x}_{s},y_{s},\dot{y}_{s}]^{T}$. In flight, the dynamics of the
body follow ballistic motion, while in stance there are also forces applied by
the spring and damper.

The touchdown guard is defined by the preload length of the leg $\ell_{td}$
and angle of the leg $\theta_{td}$ such that
$g_{(1,2)}:=y_{B}-\ell_{td}\cos(\theta_{td})$. The liftoff guard is crossed
when the force exerted by the spring-damper, $F_{sd}$, becomes zero:
$g_{(2,1)}:=F_{sd}$. There is no change in physical state of the system at the
hybrid events and the reset maps only characterize the change in coordinates
between domains.

Given a set of model parameters, a trajectory from an initial condition
depends only on $\ell_{td}$ and $\theta_{td}$. $\ell_{td}$ is held fixed, but
$\theta_{td}$ is modulated from its nominal position $\overline{\theta}_{td}$
at time $\bar{t}_{td}$ in two ways. A proportional feedback term with gain K
is added to stabilize the forward velocity of the system around a nominal
$\overline{\dot{x}}$ and the angular velocity $\dot{\theta}$ of the massless
leg is also free to be chosen.

$\displaystyle\theta_{td}=\overline{\theta}_{td}+K(\dot{x}-\overline{\dot{x}})+\dot{\theta}(t-\bar{t}_{td})$
(11)

$K$ and $\dot{\theta}$ are shape parameters that are used to stabilize this
system.

####  IV-B2 2D hopper HES results

For a chosen initial apex height of $2.5$ with a forward velocity of $2$,
$\ell_{td}$ and $\overline{\theta}_{td}$ were chosen to produce a nominal
orbit, though the following results generalize for any choice of feasible
values.

With fixed touchdown parameters $[K,\dot{\theta}]=[0,0]$, the system is highly
unstable. The shape parameters $K$ and $\dot{\theta}$ can be optimized
following (7) to improve the stability of this orbit. Doing so results in
optimal shape parameters $[K,\dot{\theta}]=[0.129,-0.015]$ that stabilize the
trajectory, Table I. Setting $\dot{\theta}=-0.015\text{ rad/s}$ is a slow
retraction rate, but there exists an interval of values
$\dot{\theta}\in(-0.5892,-0.015)$ that give equivalently minimal stability
measures for a fixed $K$ value.

TABLE I: Stability measures of 2D hopper trajectories without and with optimized shape parameters. Shape parameters | K | $\dot{\theta}$ | Stability Measure
---|---|---|---
Zero | 0 | 0 | 13.756
Optimal (zero seed) | 0.129 | -0.015 | 0.948
Optimal (alternate seed) | 0.129 | -0.589 | 0.948

The results were confirmed in simulation by initializing 20 perturbed points
in a $0.1$ radius ball around the nominal initial condition. Each of these
trials was simulated for $100$ steps and the error (2-norm of the difference
in perturbed state $x$ and nominal state $\bar{x}$) at apex was recorded at
each step, Fig. 3. The zero shape parameter trajectories are not shown in the
figure as every trial diverged within just 5 steps.

Figure 3: Error of perturbed initial states for the 2D hopper asymptotically
decrease to zero. Transparent lines represent each of the 20 trials, while the
bold line represents average error at each step. Convergence is neither
monotonic nor very fast, but this is expected with asymptotic stability.

####  IV-B3 2D Hopper Discussion

The feedback term of (11) is based on the Raibert stepping controller [37],
which was utilized to great success for stabilizing early running robots.
Other works have found that this simple controller is effective on more
complex models [38].

Another stabilizing property of legged locomotion that has been studied is
swing-leg retraction [7, 22, 39]. It was noted in [7] that a 2D SLIP was able
to run stably if it impacted the ground within a range negative angular leg
velocities $\dot{\theta}$.

The results of a negative $\dot{\theta}$ and positive $K$ agree with
qualitative expectations from [7] and [37]. While a formal equivalency can not
be proven, this is significant because the HES shape parameter optimization
has no a priori knowledge nor any applied constraints that would bias its
results. HES is able to synthesize two independently generated controllers and
produce shape parameter values that stabilize an orbit. This evidence supports
the potential for HES to explore other stabilizing shape parameters that are
not as well studied.

###  IV-C Walking biped trajectory optimization

For a legged system with non-massless legs, the leg velocity shape parameter
disappears as it is no longer independent of the trajectory. Without shape
parameters, an HES trajectory optimization can choose timing, state, and input
parameters along with virtual hybrid events to discover stable orbits.

####  IV-C1 Walking biped hybrid model

In this example, we consider a fully-actuated compass walker [40] with knees,
Fig. 4. This biped model consists of two legs connected by an actuated hip
joint. Each leg is separated into two sections, the upper leg (thigh) and
lower leg (shank), which are connected by an actuated knee joint that has a
hard stop when the thigh and shank are aligned. The ankles are also actuated.

We restrict the gaits to be left-right symmetric and exclusively consist of
single stance phases. The stance leg is locked such that its shank and thigh
are aligned with each other until liftoff. There are 3 points of actuation at
the hip, swing knee, and stance ankle. The state space is defined by three
angles relative to vertical: stance leg, swing thigh, and swing shank, denoted
$(\theta_{1},\theta_{2},\theta_{3})$.

This system has two domains. $D_{1}$ is the unlocked knee domain where the
swing leg thigh and shank are able to swing freely while we enforce that
$\theta_{3}<\theta_{2}$. $D_{2}$ is the locked knee domain where the thigh and
shank are constrained to be aligned with each other ($\theta_{2}=\theta_{3}$).
In this domain, there are only two actuators because the swing knee can not
exert torque. The dynamics of this model are described in [40].

The kneestrike guard set, between the unlocked and locked knee domains, is
$g_{(1,2)}:=\theta_{3}-\theta_{2}$ and the touchdown or toestrike guard set is
$g_{(2,1)}:=\theta_{2}+\theta_{3}$. The reset maps at kneestrike and toestrike
are also computed in [40].

Figure 4: Biped walker system with kneestrike and toestrike hybrid events.

We add discrete changes in the inputs that induce virtual hybrid events to
analyze their utility in stabilizing walking trajectories. Specifically, we
choose to include 5 virtual hybrid events in $D_{1}$ and 2 more virtual hybrid
events in $D_{2}$, where the values of inputs between virtual hybrid events
are decision variables for the optimization. The virtual guard functions are
chosen such that $g_{v_{i}}:=\theta_{3}-\theta_{2}+p_{i}$ for the first 5
virtual hybrid events and $g_{v_{i}}:=\theta_{2}+\theta_{3}+p_{i}$ for the
last 2 virtual hybrid events for some offset $p_{i}$ that is also chosen by
the optimization.

A direct collocation method was used with the cost consisting of the stability
measure and a regularization on the input. Dynamics and periodicity
constraints were included along with a ground penetration constraint. The
initial conditions of the system, given as the state after touchdown, were
allowed to vary within a bounded range.

####  IV-C2 Walking biped HES results

In this experiment, three trajectories were compared to examine how HES can
generate stable trajectories and the effect that virtual hybrid events have in
further improving stability. A trajectory without HES was produced as a
control, with its objective to minimize energy expended by using just an input
regularization term in the cost. This minimum energy (ME) trajectory is
comparable to how conventional robot locomotion trajectories are generated.
Two HES trajectories were generated, one with virtual hybrid events (HES w/
VHE) and one without (HES w/o VHE).

The ME trajectory is highly unstable, while the both HES trajectories are
stable with the trade off of a higher input cost. Specifically, HES w/ VHE has
the lowest stability measure and highest energy cost, whereas HES w/o VHE was
in between for both stability and cost, Table II.

TABLE II: Stability results for the walking biped optimization. Trajectory | Stability Measure | Energy Cost
---|---|---
ME | 7.8123 | 0.985
HES w/o VHE | 0.4715 | 1.337
HES w/ VHE | 0.3266 | 3.450

The stability properties of the generated trajectories were confirmed through
simulation. 50 trials of each trajectory were initialized with perturbations
in position and velocity between $(-0.01,0.01)$. Over a sequence of 10 steps,
the state error at each step was tracked for each trial. Fig. 5 shows that
after 10 steps, the HES trajectories have nearly converged back to the nominal
trajectory whereas the ME trajectories diverge quickly. The HES w/ VHE
trajectory converges to a smaller error after 10 steps compared to the HES w/o
VHE trajectory, which supports the findings of the stability measure.

Figure 5: Error of perturbed Minimum Energy (ME), Hybrid Event Shaping without
virtual hybrid events (HES w/o VHE) and HES w/ VHE trajectories over several
steps. Bold lines show average error at each step and shaded regions indicate
$\pm 1$ standard deviation. ME trajectories becomes highly divergent within 4
steps, while both HES trajectories appear convergent after 10 steps. The
initial increase in error of the HES trajectories is allowable and is not
considered by the stability measure.

##  V Conclusion

While the idea of hybrid event control is not novel, hybrid event shaping
provides a generalized method to analyze the stability of hybrid orbits and
select hybrid event parameters to optimize stability. HES unifies results of
previous simple hybrid event controllers while also being compatible with
trajectory optimization techniques to produce stable trajectories for complex
systems. HES also improves on previous stability optimization methods by
computing the derivative of the stability measure to increase computational
efficiency. Compared to previous work, HES does not rely on prior
understanding to identify and choose stabilizing hybrid event parameters.

Throughout this paper, there was no use of continuous-domain feedback that is
commonly utilized in hybrid control. We believe that hybrid event shaping is a
piece of the puzzle that can be used in conjunction with continuous-domain
feedback to improve the success rate of robots performing dynamic behaviors in
real-world settings. In the future, we would like to synthesize HES methods
with continuous-domain feedback controllers to produce even more stable
closed-loop trajectories.

## References

  * [1] B. Altın, P. Ojaghi, and R. G. Sanfelice, “A model predictive control framework for hybrid dynamical systems,” _IFAC-PapersOnLine_ , vol. 51, no. 20, pp. 128–133, 2018.
  * [2] N. J. Kong, G. Council, and A. M. Johnson, “iLQR for piecewise-smooth hybrid dynamical systems,” in _IEEE Conference on Decision and Control_ , December 2021, to appear.
  * [3] W. Yang and M. Posa, “Impact invariant control with applications to bipedal locomotion,” _arXiv preprint arXiv:2103.06907_ , 2021.
  * [4] S. Burden, S. Revzen, and S. Sastry, “Dimension reduction near periodic orbits of hybrid systems,” in _IEEE Conference on Decision and Control_ , September 2011.
  * [5] G. Council, S. Yang, and S. Revzen, “Deadbeat control with (almost) no sensing in a hybrid model of legged locomotion,” in _International Conference on Advanced Mechatronic Systems_ , 2014, pp. 475–480.
  * [6] D. Sternad, M. Duarte, H. Katsumata, and S. Schaal, “Bouncing a ball: Tuning into dynamic stability,” _Journal of Experimental Psychology: Human Perception and Performance_ , vol. 27, pp. 1163–84, 11 2001.
  * [7] A. Seyfarth, H. Geyer, and H. Herr, “Swing-leg retraction: a simple control model for stable running,” _Journal of Experimental Biology_ , vol. 206, no. 15, pp. 2547–2555, 2003.
  * [8] M. M. Ankaralı, U. Saranlı, and A. Saranlı, “Control of underactuated planar hexapedal pronking through a dynamically embedded slip monopod,” in _IEEE International Conference on Robotics and Automation_ , 2010, pp. 4721–4727.
  * [9] K. Green, R. Hatton, and J. Hurst, “Planning for the unexpected: Explicitly optimizing motions for ground uncertainty in running,” in _IEEE International Conference on Robotics and Automation_ , May 2020.
  * [10] A. van der Schaft and J. Schumacher, “Complementarity modeling of hybrid systems,” _IEEE Transactions on Automatic Control_ , vol. 43, no. 4, 1998\.
  * [11] J. Lygeros, K. H. Johansson, S. Simic _et al._ , “Dynamical properties of hybrid automata,” _IEEE Transactions on Automatic Control_ , vol. 48, no. 1, pp. 2–17, 2003.
  * [12] A. M. Johnson, S. A. Burden, and D. E. Koditschek, “A hybrid systems model for simple manipulation and self-manipulation systems,” _The International Journal of Robotics Research_ , vol. 35, no. 11, pp. 1354–1392, 2016\.
  * [13] D. Joyce, “On manifolds with corners,” in _Advances in Geometric Analysis_ , ser. Advanced Lectures in Mathematics. International Press of Boston, Inc., 2012, vol. 21, pp. 225–258.
  * [14] M. Jeffrey, “Dynamics at a switching intersection: Hierarchy, isonomy, and multiple sliding,” _SIAM Journal on Applied Dynamical Systems_ , vol. 13, pp. 1082–1105, 07 2014.
  * [15] J. Zhang, K. H. Johansson, J. Lygeros, and S. Sastry, “Dynamical systems revisited: Hybrid systems with zeno executions,” in _Hybrid Systems: Computation and Control_. Springer Berlin Heidelberg, 2000.
  * [16] M. Hirsch, S. Smale, and R. Devaney, _Differential Equations, Dynamical Systems, and an Introduction to Chaos_. Academic Press, 2012.
  * [17] R. Leine and H. Nijmeijer, _Dynamics and Bifurcations of Non-Snooth Mechanical Systems_. Springer-Verlag Berlin Heidelberg, 2004.
  * [18] S. A. Burden, T. Libby, and S. D. Coogan, “On contraction analysis for hybrid systems,” _arXiv preprint arXiv:1811.03956_ , 2018.
  * [19] P. Müller, “Calculation of lyapunov exponents for dynamic systems with discontinuities,” _Chaos, Solitons & Fractals_, vol. 5, no. 9, pp. 1671–1681, 1995.
  * [20] X. Wang and J. K. Hale, “On monodromy matrix computation,” _Computer Methods in Applied Mechanics and Engineering_ , vol. 190, no. 18, 2001.
  * [21] H. Asahara and T. Kousaka, “Stability analysis using monodromy matrix for impacting systems,” _IEICE Transactions of Fundamentals of Electronics, Communications, and Computer Science_ , vol. E101-A, no. 6, 2018.
  * [22] J. G. D. Karssen, M. Haberland, M. Wisse, and S. Kim, “The optimal swing-leg retraction rate for running,” in _IEEE International Conference on Robotics and Automation_ , 2011, pp. 4000–4006.
  * [23] E. Tyrtyshnikov, _A Brief Introduction to Numerical Analysis_. Birkhäuser Basel, 1997.
  * [24] P. Lancaster, “On eigenvalues of matrices dependent on a parameter,” _Numerische Mathematik_ , 1964.
  * [25] J. Nocedal and S. Wright, _Numerical Optimization_. Springer, 2006.
  * [26] M. Kelly, “An introduction to trajectory optimization: How to do your own direct collocation,” _SIAM Review_ , vol. 59, no. 4, pp. 849–904, 2017.
  * [27] O. Von Stryk and R. Bulirsch, “Direct and indirect methods for trajectory optimization,” _Annals of Operations Research_ , vol. 37, pp. 357–373, 12 1992.
  * [28] C. R. Hargraves and S. W. Paris, “Direct trajectory optimization using nonlinear programming and collocation,” _Journal of Guidance, Control, and Dynamics_ , vol. 10, no. 4, pp. 338–342, 1987.
  * [29] K. Mombaur, R. Longman, H. Bock, and J. Schlöder, “Open-loop stable running,” _Robotica_ , vol. 23, pp. 21–33, 01 2005.
  * [30] O. V. Stryk, “User’s guide for DIRCOL (version 2.1): a direct collacation method for the numerical solution of optimal control problems,” in _Technische Universitat Darmstadt_ , November 1999.
  * [31] J. T. Betts, _Practical Methods for Optimal Control and Estimation Using Nonlinear Programming, Second Edition_ , 2nd ed. SIAM, 2010.
  * [32] T. A. McMahon and G. C. Cheng, “The mechanics of running: How does stiffness couple with speed?” _Journal of Biomechanics_ , vol. 23, pp. 65–78, 1990.
  * [33] P. Holmes, R. J. Full, D. E. Koditschek, and J. Guckenheimer, “The dynamics of legged locomotion: Models, analyses, and challenges,” _SIAM Review_ , vol. 48, no. 2, pp. 207–304, 2006.
  * [34] A. Seyfarth, H. Geyer, M. Günther, and R. Blickhan, “A movement criterion for running,” _Journal of Biomechanics_ , vol. 35, no. 5, pp. 649–655, 2002.
  * [35] R. J. Full and D. E. Koditschek, “Templates and anchors: neuromechanical hypotheses of legged locomotion on land,” _Journal of Experimental Biology_ , vol. 202, no. 23, pp. 3325–3332, 12 1999.
  * [36] J. Seipel and P. Holmes, “A simple model for clock-actuated legged locomotion,” _Regular and Chaotic Dynamics_ , vol. 12, no. 5, Oct. 2007\.
  * [37] M. Raibert, H. Brown, and M. Chepponis, “Experiments in balance with a 3D one-legged hopping machine,” _The International Journal of Robotics Research_ , vol. 3, no. 2, pp. 75–92, 1984.
  * [38] I. Poulakakis and J. W. Grizzle, “The spring loaded inverted pendulum as the hybrid zero dynamics of an asymmetric hopper,” _IEEE Transactions on Automatic Control_ , vol. 54, no. 8, pp. 1779–1793, 2009.
  * [39] Y. Blum, S. Lipfert, J. Rummel, and A. Seyfarth, “Swing leg control in human running,” _Bioinspiration & Biomimetics_, vol. 5, June 2010.
  * [40] V. Chen, “Passive dynamic walking with knees: A point foot model,” Master’s thesis, Massachusetts Institute of Technology, 2007.


# A Layer-Wise Tokens-to-Token Transformer Network for Improved Historical
Document Image Enhancement

Risab Biswas<EMAIL_ADDRESS>Swalpa Kumar Roy<EMAIL_ADDRESS>Umapada Pal<EMAIL_ADDRESS>Artificial Intelligence Group, Optiks
Innovations Pvt. Ltd (P360), Mumbai, Maharashtra 400066, India.  Department of
Computer Science and Engineering, Alipurduar Government Engineering and
Management College, West Bengal 736206, India.  Computer Vision and Pattern
Recognition Unit, Indian Statistical Institute, Kolkata 700108, India.

###### Abstract

Document image enhancement is a fundamental and important stage for attaining
the best performance in any document analysis assignment because there are
many degradation situations that could harm document images, making it more
difficult to recognize and analyze them. In this paper, we propose
T2T-BinFormer which is a novel document binarization encoder-decoder
architecture based on a Tokens-to-token vision transformer. Each image is
divided into a set of tokens with a defined length using the ViT model, which
is then applied several times to model the global relationship between the
tokens. However, the conventional tokenization of input data does not
adequately reflect the crucial local structure between adjacent pixels of the
input image, which results in low efficiency. Instead of using a simple ViT
and hard splitting of images for the document image enhancement task, we
employed a progressive tokenization technique to capture this local
information from an image to achieve more effective results. Experiments on
various DIBCO and H-DIBCO benchmarks demonstrate that the proposed model
outperforms the existing CNN and ViT-based state-of-the-art methods. In this
research, the primary area of examination is the application of the proposed
architecture to the task of document binarization. The source code will be
made available at https://github.com/RisabBiswas/T2T-BinFormer.

††journal: Arxiv

##  1 Introduction

Studies on document image analysis and pattern recognition place high
importance on preserving and improving the readability of document images,
particularly historical versions. Streaks, blots, artifacts, pen strokes, ink
bleed, inadequate maintenance, aging effects, warping effects, etc. can all be
different types of degradation on document images. Because of this, the
enhancement of document images remains a challenging task. These anomalies may
have a considerable influence on the subsequent downstream data processing
activities, such as optical character recognition, document layout analysis,
etc. Various thresholding techniques were used in early enhancement methods,
such as [1], [2]. Convolutional neural network (CNN) [3] based methods have
recently been popular for tackling enhancement-related subproblems. Generative
adversarial networks (GAN) [4], particularly contrastive GAN (cGAN), are also
seen as an effective way to solve image generation issues. Transformers’
recent success in natural language processing (NLP) [5] has raised interest in
the possibility of these systems to solve computer vision problems including
image recognition [6], object detection [7], the identification of handwritten
manuscripts [8], and other related problems. The proposed self-attention
mechanism in [9] facilitates the global linkages between contextual elements.
Patches of a defined size are produced by the vision transformers (ViT’s) from
an image as opposed to CNNs, which evaluate pixel arrays. ViT divides an image
into fixed-size patches e.g. ($4\times 4$), ($8\times 8$), ($16\times 16$),
etc., and embeds each of them into a latent representation (a.k.a tokens)
while also incorporating the positional information which then becomes the
input to the transformer encoder. The previously proposed method for document
image binarization using an end-to-end ViT-based architecture has achieved
significantly improved results compared to other methods, however, the
architecture fails to capture the local information from the patches.

We propose a novel document binarization framework that combines both for an
improved document binarization task that achieves significant improvement over
the existing methodologies. Our architecture draws inspiration from the idea
that taking an image’s local features into account, in conjunction with the
global contextual information, can significantly help in improving the
binarization results. Our hypothesis is that ViT’s basic tokenization of input
images using a hard split limits the model from effectively modeling the local
structure of the images and thus provides an opportunity to further enhance
the existing binarization framework based on ViT. Therefore, we incorporated
the Tokens-to-Token ViT (T2T-ViT) in the encoding stage instead of the vanilla
ViT. In this research, we backed up our hypothesis with experimental evidence
as well. The proposed approach is aimed at extracting both the global and
local features from the images which will serve as the basis to demonstrate
that the simple tokenization in ViT-based architecture fails to capture the
local knowledge from the image patches. These local features act as crucial
information to the self-attention module of the ViT. At the conclusion of the
encoding stage, the generated tokens contain both the global and local
contextual information of the image patch. The proposed paradigm achieves
state-of-the-art results in solving the document image binarization
challenges. Our research experiments illustrate the same.

In the proposed network, the degraded input image is first split into patches
and is attached to a latent token representation. Instead of the naive
tokenization used in vanilla ViT, we employed the progressive tokenization
method described in [10], also known as the Tokens-to-Token module, to combine
nearby tokens into a single token. This method may gradually shorten tokens by
modeling the local structural information of the tokens that surround them.
The decoder creates enhanced image patches after the encoding process, which
are then reassembled to make the final enhanced output image. Unlike the
previously proposed models for document image binarization using a vision
transformer, which uses a simple tokenization method, resulting in the
extraction of global contextual features, while not being able to retrieve the
intricate details that reside in the local information of the patches. The
architecture of the proposed methodology ensures that the patch’s local
structure is captured as well during the encoding stage.

The proposed network has been shown to perform better than the Vanilla ViT for
the document binarization task because the T2T block can better model the
local structure information in document images, which is very crucial for this
task. The T2T block does this by recursively aggregating neighboring tokens
into one token, such that the local structure represented by surrounding
tokens can be modeled and the length of tokens can be reduced. This results in
making it easier for the encoder block to identify the text in document
images, especially those that contain many ”white areas” (empty zones without
text). Additionally, the proposed method is more efficient than the other
Vanilla ViT-based model, requiring fewer parameters and computation resources.
This makes it a more practical choice for document image binarization from a
computation perspective as well.
The contributions of the proposed methodology are summarized below:

  * •

We introduce an end-to-end, simple, and effective binarization method using a
Tokens-to-token vision transformer. To our knowledge, this is the first
Tokens-to-Token-based ViT architecture for document binarization problems.
This architecture can also be used for other document image enhancement tasks
such as dewarping, deblurring, etc. with some minor modifications.

  * •

The tokens-to-token vision transformer produced state-of-the-art results
compared to the simple ViT-based architecture used for document image
binarization. In fact, based on our experimental results, the proposed
architecture beats the existing benchmark on a majority of the test datasets.
In the proposed architecture, instead of simply dividing the input degraded
image into patches and feeding them to the transformer, the architecture
produces tokens using a progressive tokenization technique.

  * •

We conducted extensive experiments and provided a comprehensive and in-depth
case study to show the model’s successes in solving the document image
binarization challenge from both a qualitative and quantitative standpoint on
the DIBCO and the H-DIBCO datasets.

The remaining paper is organized as follows: In Section 2 we discuss the
related research. The proposed architecture is described in Section 3. Section
4 provides the experimental results and comparisons. Finally, a comprehensive
conclusion is outlined in Section 6.

##  2 Related Works

This section will cover several binarization approaches for document images.
Both the traditional approaches and the most current advancements in this area
are covered:

###  2.1 Document Image Binarization

The earliest traditional approaches for document image binarization were based
on thresholding, some of them includes [1] and [2], [11]. Local adaptive
techniques, such as [11] perform better in situations involving poor-quality
photos. Binarization of document images may also be done using learning
algorithms like [12]. To enhance the quality of binarization, current deep-
learning networks have also been integrated with threshold-based methods.
Examples of the same are DeepOtsu [13] and SauvolaNet [14]. [15] utilizes CNNs
for solving binarization problems. [16] proposed a scalable fully
convolutional network (FCN) for document image binarization. Expanding on the
idea of using CNN-based networks, [17] describes a faster, more precise, and
more efficient technique based on U-Net. Additionally, generative models (GAN)
are widely applied to solve a variety of image restoration problems. Some of
the popular GAN-based techniques include [18], [19].

Figure 1: Architecture of the proposed network for document image
binarization. The encoding stage is shown on the left, and the decoding stage
is shown on the right. The transformer encoder block is graphically
illustrated as well.

###  2.2 Transformers for Document Image Binarization

Recent advancements in deep learning applications have been driven by
transformers. This drives numerous efforts to use them for vision tasks such
as classification [20], medical image segmentation [21], object detection
[22], drug discovery [23] and so forth. Transformers have also been utilized
for tasks like image restoration [24] and image de-warping [25]. [26] proposed
a fully transformer-based approach for document image enhancement, without the
need for any CNN. However since their approach is entirely based on
conventional ViT without any design change, it fails to capture the local
information from the patches. Extending to that, we propose T2T-BinFormer, a
more effective Tokens-to-token ViT for more effective document image
binarization.

##  3 Proposed Methodology

The proposed methodology is an efficient document image binarization framework
that employs a Tokens-to-token vision transformer (T2T-ViT) based encoder-
decoder architecture. The entire architecture is divided into two stages -
Encoding and Decoding respectively. Fig. 1 illustrates the architecture of the
proposed binarization framework. The components of the architecture are
discussed in detail from Subsection 3.1 to 3.4:

###  3.1 Patch Tokenization and Position Embedding

The input image, $\mathbf{X}\in\rm\mathbb{R}^{(H\times{W}\times{C})}$, where
$\rm H$, $\rm W$ is the height and width, respectively and $C$ be the number
of channels, here $C$ = 3, converted into tokens using T2T (Token-to-Token)
Module. We apply the same operations as mentioned in [10] for converting the
input degraded image into tokens or embeddings. Thus the input image can be
written as $\mathbf{X}\in\mathbb{R}^{(\rm H\times{W}\times{3})}$. The input
image $\mathbf{X}$ is downsampled by unfolding, leading to overlapping image
data in each token as shown in Fig. 2. After applying several T2T blocks the
input image is converted into tokens of fixed length, which is the output of
the Tokens-to-Token module, say $\mathbf{X^{\prime}}\in\mathbb{R}^{(\rm
H\times{W}\times{3})}$. These tokens include both the global and local
structural information of the image. Positional embeddings are linearly
appended to the tokens to guarantee that the pictures’ positional information
is preserved. The parameters of the learnable positional embedding vectors are
randomly initialized. The equation below illustrates the positional embeddings
for the patch representations:

$\mathbf{X^{\prime}_{token}}={\rm X^{\prime}}\oplus\rm{PE}_{Patch}$ (1)

where $\rm{PE}_{Patch}\in\mathbb{R}^{(n_{patch}\times\rm D)}$ and $\oplus$
represent the patch positional encodings and element-wise addition,
respectively. $\rm D$ represents the dimension of each patch embedding.

Figure 2: Illustration of T2T process as proposed by Yuan et al. [10]

###  3.2 Transformer Encoder Block

$\mathbf{X^{\prime}_{token}}$ shown in Eq. (1) is the input to the transformer
encoder block. There might exist one or more transformer encoder blocks. The
transformer encoder follows the identical structure as [6]. Each transformer
encoder comprises a multi-head self-attention [9] to capture the feature
dependencies along with the multi-layered perceptron (MLP) layer. The overall
tokenized matrix, $\mathbf{X^{\prime}_{token}}$ passes through a layer
normalization layer (LN) and then enters the multi-head self-attention (MSA)
block of the transformer encoder. In the proposed transformer network, we have
8 self-attention heads working in parallel inside each transformer encoder
block, this can be considered to be a hyperparameter. The objective of this
step is to capture the contextual knowledge from the image. The below equation
mathematically represents the above steps:

$\mathbf{Y_{token}}=\rm MSA(LN(\mathbf{X^{\prime}_{token}}))$ (2)
$\mathbf{Y^{\prime}_{token}}=\mathbf{Y_{token}}\oplus\rm
MLP(LN(\mathbf{Y_{token}}))$ (3)

###  3.3 Binarization using Transformer Decoder Block

At the end of Section 3.2, we have the encoded token representation
($\mathbf{Y^{\prime}_{token}}$) of the input image. This encoded vector
contains both global and local structural information about the image. This
section deals with the decoding stage of our architecture, which is also the
final step of the proposed binarization model. The encoded token is the input
to the series of transformer blocks used in the decoder stage as illustrated
in Fig. 1. The decoder processes the encoded tokens in the same way the global
transformer does in the encoding stage. These tokens are projected with a
linear layer to the required pixel values after being transmitted in the
transformer decoder blocks. As a result, each output element is converted to a
vector $\mathbf{{Y}}_{\rm pred\\_pixel}$ that represents a patch in the final
image. As the last step, the binarized patches are rearranged to construct the
original-sized output of the corresponding degraded image.

###  3.4 Loss and Optimization Function

For the training purpose, we use _Mean Square Error_ (MSE) loss function,
since MSE can handle the outliers more optimally. The ground truth (GT) image
is divided into patches and flattened into a vector. Then we calculate the MSE
loss between the output vector obtained from the decoder and the GT vector.
Let the GT vector be $\mathbf{{GT}}_{\rm patches}$. The below equation
illustrates the above calculation mathematically:

$\rm{MSE}={\frac{1}{n}}\sum_{i=1}^{n}(\mathbf{{GT}}_{patches}-\mathbf{{Y}}_{\rm
pred\\_pixel})^{2}$ (4)

For optimization, we are using Weight decay Adam Optimizer (AdamW)[27]
optimization function. This may lessen the risk of the model overfitting.

##  4 Experimental Setup

###  4.1 Dataset and Evaluation Metrics

We used a variety of DIBCO and H-DIBCO datasets for training, testing, and
validating our proposed model. We used the following DIBCO datasets: 2009
[28], 2010[29], 2011[30], 2012[31], 2013[32], 2014[33], 2018[34], and
2019[35]. These datasets contain both handwritten and machine-typed document
images. We used the ”leave-one-out strategy” for our experiment. This means
that we use all of the documents from a single (H-)DIBCO year as the training
dataset and all of the remaining documents from other years as the testing
dataset. In the following subsections, we will assess the model’s performance
both quantitatively and qualitatively. To perform a qualitative assessment of
how well the binarization model is working, we used the standard metrics such
as the peak signal-to-noise ratio (PSNR), F-Measure (FM), pseudo-F-measure
$(\rm\mathbf{F_{ps}})$, and distance reciprocal distortion (DRD).

###  4.2 Implementation Details

Each degraded image and its corresponding ground truth images are transformed
into ($256\times 256$) images during data preprocessing. These preprocessed
images are then utilized for training, testing, and validation. This resizing
is a matter of choice, which means, the original DIBCO image can be resized to
any other dimension, such as ($128\times$128) or ($512\times 512$). We also
used data augmentation to increase the data volume. This includes - flipping,
random cropping, and horizontal and vertical rotation. The internal structure
of T2T-BinFormer is described in Table 1. Patch size $\rm P$ is chosen as 16.
The model’s learning rate is 1.5e-4, eps is 1e-08, and weight decay is 0.05.
We used a 16-image batch size and trained each model iteration for 200 epochs
on a single NVIDIA RTX 4070Ti GPU. All the training and inference algorithm is
carried out in PyTorch.

Table 1: Hyper-Parameters Settings for the Proposed Model.

Block | Patch Size | #Layers | #Heads | Dim | MLP Dim
---|---|---|---|---|---
Encoder | $16\times 16$ | 12 | 8 | 768 | 2048
Decoder | $16\times 16$ | 1 | 8 | 64 | 2048

##  5 Experimental Results and Comparison

###  5.1 Quantitative Results and Comparison

After selecting the best hyperparameters and training the model, we run
experiments on a variety of datasets from both the DIBCO and H-DIBCO and
evaluate the results with those obtained using other state-of-the-art
techniques. We begin our quantitative comparative analysis with the DIBCO 2009
dataset which contains 10 sample images of both handwritten and machine-typed
types. The outcome of the proposed method and the other state-of-the-art
techniques are shown in Table 2. The performance of the proposed model is
mentioned in the last row, in bold. The findings demonstrate that, for this
dataset, the proposed model significantly outperformed all other techniques in
terms of PSNR, FM, $\rm F_{ps}$, and DRD. It also shows that the proposed
method performed better than the existing vanilla ViT-based model. Following
that, we tested our model on the DIBCO 2010, 2011, 2012, 2013, 2014, 2018, and
the most recent 2019 datasets, which include 10, 16, 14, 16, 10, 10, and 20
images, respectively. The results of these can be found in Tables 3, 4, 5, 6,
7, 8 and 9, respectively. The output tables exhibit the obtained results and
draw a comparative analysis with the previously proposed models. We can infer
that the T2T-BinFormer model gives the comparative performance in terms of
PSNR, FM, $\rm F_{ps}$, and DRD in the majority of the cases thus making it a
viable solution for document binarization task.

Table 2: Comparative results of the proposed method and other state-of-the-art
methods on the DIBCO 2009 dataset.

Method | Model | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
Bradley[36] | Thresh. | 15.21 | 80.70 | 82.10 | 0.53
Otsu[1] | Thresh. | 15.31 | 78.60 | 80.50 | 0.55
Sauvola et al.[2] | Thresh. | 16.32 | 85.00 | 89.50 | 0.50
Gatos et al.[37] | Thresh. | 17.02 | 87.27 | 89.28 | 0.50
Winner[28] | CNN | 18.66 | 91.24 | - | -
Zhao et al.[19] | GAN | 20.30 | 94.10 | 95.26 | 1.82
Kang et al.[38] | U-net CNN | 20.91 | 96.67 | - | -
SauvolaNet.[14] | GAN | 21.53 | 95.51 | 96.58 | 0.37
DocEnTr[26] | Transformer | 21.32 | 95.50 | 96.70 | 0.18
Proposed | Transformer | 22.04 | 96.20 | 97.62 | 0.22

Table 3: Comparative results of the proposed method and other state-of-the-art
methods on the DIBCO 2010 dataset.

Method | Model | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
DE-GAN[18] | GAN | 15.80 | 83.00 | 82.96 | 0.18
Sauvola et al.[2] | Thresh. | 15.86 | 74.00 | 83.39 | 0.11
Bradley[36] | Thresh. | 16.78 | 82.90 | 88.39 | 0.13
Gatos et al.[37] | Thresh. | 16.18 | 77.90 | 85.36 | 0.08
Otsu[1] | Thresh. | 17.53 | 85.50 | 90.69 | 0.15
Winner[39] | Thresh. | 19.78 | 91.50 | 93.58 | -
Xiong et al.[40] | Estimation | 20.97 | 93.73 | 95.18 | -
DocEnTr[26] | Transformer | 21.90 | 95.10 | 96.85 | 0.13
Proposed | Transformer | 23.00 | 96.17 | 97.67 | 0.22

Table 4: Comparative results of the proposed method and other state-of-the-art
methods on the DIBCO 2011 dataset.

Methods | Models | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
Sauvola et al.[2] | Thresh. | 15.60 | 82.10 | 87.70 | 8.50
Otsu[1] | Thresh. | 15.72 | 82.10 | 84.80 | 9.00
Winner[30] | CNN | 16.10 | 80.90 | - | 104.40
Zhao et al.[19] | GAN | 19.58 | 92.62 | 95.38 | 2.55
Kang et al.[38] | CNN | 19.90 | 95.50 | - | 1.80
Vo et al.[41] | DSN | 20.10 | 93.30 | 96.40 | 2.00
Sungho et al.[42] | GAN | 20.22 | 93.57 | 95.93 | 1.99
Bhunia et al.[43] | Adver. Learning | 20.40 | 93.70 | 96.80 | 1.80
SauvolaNet.[14] | GAN | 20.55 | 94.32 | 96.40 | 1.97
Dang et al.[44] | CNN | 22.09 | 95.61 | 97.34 | 1.48
DocEnTr[26] | Transformer | 20.81 | 94.37 | 96.15 | 1.63
Text-DIA[45] | Transformer | 21.29 | 95.01 | 96.86 | 1.48
Proposed | Transformer | 22.17 | 96.19 | 97.63 | 0.15

Table 5: Comparative results of the proposed method and other state-of-the-art
methods on the DIBCO 2012 dataset.

Methods | Models | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
Otsu[1] | Thresh. | 15.03 | 80.18 | 82.65 | 26.46
Sauvola et al.[2] | Thresh. | 16.71 | 82.89 | 87.95 | 6.59
Winner[31] | CNN | 20.60 | 92.53 | 96.15 | 1.55
Kang et al.[38] | CNN | 21.37 | 95.16 | 96.44 | 1.13
Xiong et al.[40] | Estimation | 21.68 | 94.26 | 95.16 | 2.08
Zhao et al.[19] | GAN | 21.91 | 94.96 | 96.15 | 1.55
Jemni et al.[46] | GAN | 22.00 | 95.18 | 94.63 | 1.62
DocEnTr[26] | Transformer | 22.29 | 95.31 | 96.29 | 1.60
Text-DIA[45] | Transformer | 23.66 | 96.52 | 97.52 | 1.10
Proposed | Transformer | 23.95 | 96.80 | 98.04 | 0.20

Table 6: Comparative results of the proposed method and other state-of-the-art
methods on the DIBCO 2013 dataset.

Methods | Models | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
Otsu[1] | Thresh. | 16.63 | 83.94 | 96.52 | 10.98
Sauvola et al.[2] | Thresh. | 16.94 | 85.02 | 89.77 | 7.58
Guo et al.[47] | Non-linear Diffusion | 17.37 | 82.35 | 85.16 | 8.09
Gatos et al.[37] | Thresh. | 17.58 | 83.47 | 87.03 | 0.17
DeepOtsu[13] | CNN | 20.65 | 93.79 | 91.90 | 2.60
Winner[32] | CNN | 20.68 | 92.12 | 94.19 | 3.10
Vo et al.[41] | Hierarchical DSN | 21.40 | 94.40 | 96.00 | 1.80
Kang et al.[38] | U-Net CNN | 22.97 | 95.88 | 96.38 | 1.46
Dang et al.[44] | CNN | 23.14 | 95.96 | 98.13 | 1.43
DocEnTr[26] | Transformer | 23.24 | 96.42 | 97.69 | 0.14
Proposed | Transformer | 23.99 | 97.10 | 98.23 | 0.07

Table 7: Comparative results of the proposed method and other state-of-the-art
methods on the DIBCO 2014 dataset.

Methods | Models | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
Sauvola et al.[2] | Thresh. | 17.48 | 83.72 | 87.49 | 5.05
Gatos et al.[37] | Thresh. | 18.24 | 85.88 | 88.07 | 0.24
Otsu[1] | Thresh. | 18.73 | 91.62 | 95.69 | 2.65
Su et al.[48] | Thresh. | 20.31 | 94.38 | 95.94 | 1.95
Zhao et al.[19] | GAN | 21.88 | 96.09 | 98.25 | 1.20
DeepOtsu[13] | CNN | 22.10 | 95.90 | 97.20 | 0.90
Howe et al.[49] | Thresh. | 22.24 | 96.49 | 97.30 | 1.08
Vo et al.[41] | Hierarchical DSN | 23.23 | 96.66 | 97.59 | 0.79
DocEnTr[26] | Transformer | 22.99 | 97.16 | 98.28 | 0.16
Proposed | Transformer | 23.48 | 97.50 | 98.50 | 0.21

Table 8: Comparative results of the proposed method and other state-of-the-art
methods on DIBCO 2018 dataset.

Methods | Models | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
Otsu[1] | Thresh. | 9.74 | 51.45 | 53.05 | 59.07
Sauvola et al.[2] | Thresh. | 13.78 | 67.81 | 74.08 | 17.69
DE-GAN[18] | GAN | 16.16 | 77.59 | 85.74 | 7.93
Kang et al.[38] | CNN | 19.39 | 89.71 | 91.62 | 2.51
Zhao et al.[19] | GAN | 18.37 | 87.73 | 90.60 | 4.58
Winner[50] | CNN | 19.11 | 88.34 | 90.24 | 4.92
DocEnTr[26] | Transformer | 19.47 | 92.53 | 95.15 | 2.37
Text-DIA[45] | Transformer | 19.95 | 91.32 | 94.44 | 3.21
Jemni et al.[46] | GAN | 20.18 | 92.41 | 94.35 | 2.60
Proposed | Transformer | 22.33 | 95.60 | 96.97 | 0.13

Table 9: Comparative results of the proposed method and other state-of-the-art
methods on DIBCO 2019 dataset.

Methods | Models | PSNR$\uparrow$ | FM$\uparrow$ | Fps$\uparrow$ | DRD$\downarrow$
---|---|---|---|---|---
Niblack[11] | Thresh. | 8.62 | 43.00 | 43.00 | 0.25
Otsu[1] | Thresh. | 9.05 | 48.00 | 48.00 | 0.29
Bradley[36] | Thresh. | 10.98 | 58.00 | 58.00 | 0.30
Sauvola et al.[2] | Thresh. | 13.11 | 65.00 | 66.00 | 0.31
Gatos et al.[37] | Thresh. | 12.81 | 65.00 | 65.00 | 0.31
DocEnTr[26] | Transformer | 13.85 | 59.00 | 60.00 | 0.30
Proposed | Transformer | 14.49 | 65.70 | 67.82 | 0.29

These improvements can be quantified as - (1) For DIBCO 2009, we notice that
the performance of the model outperforms the previous best model-based method
by 0.72 and 3.32% absolute improvement and percentage improvement respectively
on PSNR. (2) We also calculated the percentage improvement from the last best
method in terms of PSNR for other DIBCO sets. The results of which are - 5.02%
for 2010, 1.23% for 2011, 1.22% for 2012, 2.34% for 2013, 0.64% for 2014, 10%
for 2018 and 4.62% for 2019. The quantitative results produced by the model
evidently revealed that the proposed network makes good use of the Tokens-to-
Token module instead of the simple tokenization used in current transformer-
based approaches. This also helps in understanding that the local structure
captured by the proposed architecture plays a crucial role in the model’s
performance.

###  5.2 Qualitative Results and Comparison

Subsection 5.1 provided the quantitative results produced by the proposed
model and did a comparative analysis with the existing binarization
approaches. In this subsection, we will provide a qualitative evaluation. We
begin the qualitative analysis by providing an intuitive comparison between
the ground truth (GT) image and the output of the proposed model on
handwritten document images taken from different DIBCO datasets. We chose
various handwritten document images with varying types and degrees of
degradation and compared the binarized output with the GT and the original
document image. the illustration is shown in Fig. 3 and 4, respectively. Next,
we perform a qualitative analysis on samples taken from DIBCO 2009, 2010,
2011, 2012, 2013, 2018, and 2019 datasets. The comparative results are shown
in Fig. 5, 6, 7, 8, 9, 10, 11, and 12, respectively.

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |

Figure 3: Qualitative analysis on output generated by T2T-BinFormer on various
machine-typed DIBCO and H-DIBCO Images. Images from Left to Right are:
Original, Ground Truth, and Binarized output.

In conclusion, the qualitative comparison shown in Fig. 5 to 12 clearly
demonstrates the proposed model’s superiority in recovering a severely damaged
picture over conventional thresholding, CNN, GAN, and current ViT-based
approaches.

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |

Figure 4: Qualitative Analysis on Output generated by T2T-BinFormer on various
machine-typed (H-)DIBCO Images. Images from Left to Right are Original, Ground
Truth, and Binarized output.

##  6 Conclusion

In this study, we introduced a Tokens-to-Token vision transformer network for
document binarization. The Tokens-to-Token method is employed in the proposed
network as opposed to the vanilla ViT network’s straightforward tokenization.
In this study, we went into great depth on how the model extracts both global
and local feature dependencies and its impact on the results produced for the
image binarization problem. In addition to providing a detailed description of
the model’s operating principles, we graphically and mathematically
represented the complete proposed network. To the best of our knowledge,
T2T-BinFormer is the first binarization method based on ViT to employ a soft
split strategy for feature extraction as opposed to hard split patches, as
vanilla ViT does. We conducted in-depth quantitative and qualitative
experimental analysis on a variety of real-world document image binarization
datasets, each of which presented its own set of challenges. The proposed
model was successfully able to address almost all of these challenges and
produced cutting-edge results. Numerous experiments show that the
T2T-BinFormer model produces outcomes that are noticeably superior to those of
the state-of-the-art. In our upcoming work, we want to expand this network to
handle deblurring, dehazing, dewarping, and other activities related to
document image enhancement. We would also keep researching the best ways to
use ViT-based models to effectively extract the local information from an
image.

|  |  |  |
---|---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) SauvolaNet [14] |
|  |  |  |
e) Bradley [36] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer |
Figure 5: Qualitative performance of the various binarization techniques on sample no. 4 from the DIBCO 2009 dataset. |  |  |  |
---|---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) SauvolaNet [14] |
|  |  |  |
e) Bradley [36] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer |
Figure 6: Qualitative performance of the various binarization techniques on sample no. 2 from the DIBCO 2010 dataset. |  |  |  |
---|---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) SauvolaNet [14] |
|  |  |  |
e) Jemni et al. [46] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer |
Figure 7: Qualitative performance of the various binarization techniques on sample no. 6 from the DIBCO 2011 dataset. |  |  |  |
---|---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) SauvolaNet [14] |
|  |  |  |
e) Bradley [36] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer |
Figure 8: Qualitative performance of the various binarization techniques on sample no. 13 from the DIBCO 2012 dataset. |  |  |  |
---|---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) SauvolaNet [14] |
|  |  |  |
e) Bradley [36] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer |
Figure 9: Qualitative performance of the various binarization techniques on sample no. 7 from the DIBCO 2013 dataset. |  |  |  |
---|---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) SauvolaNet [14] |
|  |  |  |
e) Bradley [36] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer |
Figure 10: Qualitative performance of the various binarization techniques on sample no. 1 from the DIBCO 2018 dataset. |  |  |  |
---|---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) SauvolaNet [14] |
|  |  |  |
e) Bradley [36] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer |
Figure 11: Qualitative performance of the various binarization techniques on sample no. 6 from the DIBCO 2018 dataset. |  |  |
---|---|---|---
a) Original | b) GT | c) Otsu [1] | d) Gatos [37]
|  |  |
e) Bradley [36] | f) DE-GAN [18] | g) DocEnTr [26] | h) T2T-BinFormer
Figure 12: Qualitative performance of the various binarization techniques on
sample no. 2 from the DIBCO 2019 dataset.

## References

  * [1] N. Otsu, A threshold selection method from gray-level histograms, IEEE transactions on systems, man, and cybernetics 9 (1) (1979) 62–66.
  * [2] J. Sauvola, M. Pietikäinen, Adaptive document image binarization, Pattern recognition 33 (2) (2000) 225–236.
  * [3] Y. LeCun, Y. Bengio, et al., Convolutional networks for images, speech, and time series, The handbook of brain theory and neural networks 3361 (10) (1995) 1995.
  * [4] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, Advances in neural information processing systems 27 (2014).
  * [5] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805 (2018).
  * [6] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint arXiv:2010.11929 (2020).
  * [7] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: European conference on computer vision, Springer, 2020, pp. 213–229.
  * [8] M. Li, T. Lv, L. Cui, Y. Lu, D. Florencio, C. Zhang, Z. Li, F. Wei, Trocr: Transformer-based optical character recognition with pre-trained models, arXiv preprint arXiv:2109.10282 (2021).
  * [9] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017).
  * [10] L. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, Z.-H. Jiang, F. E. Tay, J. Feng, S. Yan, Tokens-to-token vit: Training vision transformers from scratch on imagenet, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 558–567.
  * [11] W. Niblack, An introduction to digital image processing, Strandberg Publishing Company, 1985.
  * [12] N. Papamarkos, A technique for fuzzy document binarization, in: Proceedings of the 2001 ACM Symposium on Document engineering, 2001, pp. 152–156.
  * [13] S. He, L. Schomaker, Deepotsu: Document enhancement and binarization using iterative deep learning, Pattern recognition 91 (2019) 379–390.
  * [14] D. Li, Y. Wu, Y. Zhou, Sauvolanet: learning adaptive sauvola network for degraded document binarization, in: International Conference on Document Analysis and Recognition, Springer, 2021, pp. 538–553.
  * [15] J. Pastor-Pellicer, S. España-Boquera, F. Zamora-Martínez, M. Z. Afzal, M. J. Castro-Bleda, Insights on the use of convolutional neural networks for document image binarization, in: International Work-Conference on Artificial Neural Networks, Springer, 2015, pp. 115–126.
  * [16] C. Tensmeyer, T. Martinez, Document image binarization with fully convolutional neural networks, in: 2017 14th IAPR international conference on document analysis and recognition (ICDAR), Vol. 1, IEEE, 2017, pp. 99–104.
  * [17] P. V. Bezmaternykh, D. A. Ilin, D. P. Nikolaev, U-net-bin: hacking the document image binarization contest 43 (5) (2019) 825–832.
  * [18] M. A. Souibgui, Y. Kessentini, De-gan: a conditional generative adversarial network for document enhancement, IEEE Transactions on Pattern Analysis and Machine Intelligence (2020).
  * [19] J. Zhao, C. Shi, F. Jia, Y. Wang, B. Xiao, Document image binarization with cascaded generators of conditional generative adversarial networks, Pattern Recognition 96 (2019) 106968.
  * [20] R. Biswas, A. Basu, A. Nandy, A. Deb, R. Chowdhury, D. Chanda, Identification of pathological disease in plants using deep neural networks-powered by intel® distribution of openvino™ toolkit, in: 2020 Indo–Taiwan 2nd International Conference on Computing, Analytics and Networks (Indo-Taiwan ICAN), IEEE, 2020, pp. 45–48.
  * [21] R. Biswas, Polyp-sam++: Can a text guided sam perform better for polyp segmentation? (2023).  arXiv:2308.06623.
  * [22] Z.-Q. Zhao, P. Zheng, S.-t. Xu, X. Wu, Object detection with deep learning: A review, IEEE transactions on neural networks and learning systems 30 (11) (2019) 3212–3232.
  * [23] R. Biswas, A. Basu, A. Nandy, A. Deb, K. Haque, D. Chanda, Drug discovery and drug identification using ai, in: 2020 Indo – Taiwan 2nd International Conference on Computing, Analytics and Networks (Indo-Taiwan ICAN), 2020, pp. 49–51.  doi:10.1109/Indo-TaiwanICAN48429.2020.9181309.
  * [24] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, R. Timofte, Swinir: Image restoration using swin transformer, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1833–1844.
  * [25] P. Li, J. Gu, J. Kuen, V. I. Morariu, H. Zhao, R. Jain, V. Manjunatha, H. Liu, Selfdoc: Self-supervised document representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5652–5660.
  * [26] M. A. Souibgui, S. Biswas, S. K. Jemni, Y. Kessentini, A. Fornés, J. Lladós, U. Pal, Docentr: An end-to-end document image enhancement transformer, arXiv preprint arXiv:2201.10252 (2022).
  * [27] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [28] B. Gatos, K. Ntirogiannis, I. Pratikakis, Icdar 2009 document image binarization contest (dibco 2009), in: 2009 10th International Conference on Document Analysis and Recognition, 2009, pp. 1375–1382.  doi:10.1109/ICDAR.2009.246.
  * [29] I. Pratikakis, B. Gatos, K. Ntirogiannis, H-dibco 2010 - handwritten document image binarization competition, in: 2010 12th International Conference on Frontiers in Handwriting Recognition, 2010, pp. 727–732.  doi:10.1109/ICFHR.2010.118.
  * [30] Pratikakis, Ioannis and Gatos, Basilis and Ntirogiannis, Konstantinos, Icdar 2011 document image binarization contest (dibco 2011), in: 2011 International Conference on Document Analysis and Recognition, 2011, pp. 1506–1510.  doi:10.1109/ICDAR.2011.299.
  * [31] I. Pratikakis, B. Gatos, K. Ntirogiannis, Icfhr 2012 competition on handwritten document image binarization (h-dibco 2012), in: 2012 International Conference on Frontiers in Handwriting Recognition, 2012, pp. 817–822.  doi:10.1109/ICFHR.2012.216.
  * [32] Pratikakis, Ioannis and Gatos, Basilis and Ntirogiannis, Konstantinos, Icdar 2013 document image binarization contest (dibco 2013), in: 2013 12th International Conference on Document Analysis and Recognition, 2013, pp. 1471–1476.  doi:10.1109/ICDAR.2013.219.
  * [33] K. Ntirogiannis, B. Gatos, I. Pratikakis, Icfhr2014 competition on handwritten document image binarization (h-dibco 2014), in: 2014 14th International Conference on Frontiers in Handwriting Recognition, 2014, pp. 809–813.  doi:10.1109/ICFHR.2014.141.
  * [34] I. Pratikakis, K. Zagori, P. Kaddas, B. Gatos, Icfhr 2018 competition on handwritten document image binarization (h-dibco 2018), in: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2018, pp. 489–493.  doi:10.1109/ICFHR-2018.2018.00091.
  * [35] I. Pratikakis, K. Zagoris, X. Karagiannis, L. Tsochatzidis, T. Mondal, I. Marthot-Santaniello, Icdar 2019 competition on document image binarization (dibco 2019), in: 2019 International Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 1547–1556.  doi:10.1109/ICDAR.2019.00249.
  * [36] D. Bradley, G. Roth, Adaptive thresholding using the integral image, Journal of graphics tools 12 (2) (2007) 13–21.
  * [37] B. Gatos, I. Pratikakis, S. J. Perantonis, Adaptive degraded document image binarization, Pattern recognition 39 (3) (2006) 317–327.
  * [38] S. Kang, B. K. Iwana, S. Uchida, Complex image processing with less data—document image binarization by integrating multiple pre-trained u-net modules, Pattern Recognition 109 (2021) 107577.
  * [39] B. Su, S. Lu, C. L. Tan, Binarization of historical document images using the local maximum and minimum, in: Proceedings of the 9th IAPR International Workshop on Document Analysis Systems, 2010, pp. 159–166.
  * [40] W. Xiong, L. Zhou, L. Yue, L. Li, S. Wang, An enhanced binarization framework for degraded historical document images, EURASIP Journal on Image and Video Processing 2021 (1) (2021) 1–24.
  * [41] Q. N. Vo, S. H. Kim, H. J. Yang, G. Lee, Binarization of degraded document images based on hierarchical deep supervised network, Pattern Recognition 74 (2018) 568–586.
  * [42] S. Suh, J. Kim, P. Lukowicz, Y. O. Lee, Two-stage generative adversarial networks for document image binarization with color noise and background removal, arXiv preprint arXiv:2010.10103 (2020).
  * [43] A. K. Bhunia, A. K. Bhunia, A. Sain, P. P. Roy, Improving document binarization via adversarial noise-texture augmentation, in: 2019 IEEE International Conference on Image Processing (ICIP), IEEE, 2019, pp. 2721–2725.
  * [44] Q.-V. Dang, G.-S. Lee, Document image binarization with stroke boundary feature guided network, IEEE Access 9 (2021) 36924–36936.
  * [45] M. A. Souibgui, S. Biswas, A. Mafla, A. F. Biten, A. Fornés, Y. Kessentini, J. Lladós, L. Gomez, D. Karatzas, Text-diae: Degradation invariant autoencoders for text recognition and document enhancement, arXiv preprint arXiv:2203.04814 (2022).
  * [46] S. K. Jemni, M. A. Souibgui, Y. Kessentini, A. Fornés, Enhance to read better: a multi-task adversarial network for handwritten document image enhancement, Pattern Recognition 123 (2022) 108370.
  * [47] J. Guo, C. He, X. Zhang, Nonlinear edge-preserving diffusion with adaptive source for document images binarization, Applied Mathematics and Computation 351 (2019) 8–22.
  * [48] B. Su, S. Lu, C. L. Tan, Robust document image binarization technique for degraded document images, IEEE transactions on image processing 22 (4) (2012) 1408–1417.
  * [49] N. R. Howe, A laplacian energy for document binarization, in: 2011 International Conference on Document Analysis and Recognition, 2011, pp. 6–10.  doi:10.1109/ICDAR.2011.11.
  * [50] I. Pratikakis, K. Zagoris, G. Barlas, B. Gatos, Icdar2017 competition on document image binarization (dibco 2017), in: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 01, 2017, pp. 1395–1403.  doi:10.1109/ICDAR.2017.228.