# Effect of Accretion on the evolution of Primordial Black Holes in the
context of Modified Gravity Theories

Shreya Banerjee<EMAIL_ADDRESS>Aritrya Paul<EMAIL_ADDRESS>Department of Physics, Indian Institute of Technology (Indian School of
Mines), Dhanbad, Jharkhand-826004, INDIA

###### Abstract

We investigates the effect of accretion of cosmic fluid on the evolution of
Primordial Black Holes (PBHs) within the framework of Modified gravity
theories. We consider a general form of the Hubble parameter, reflecting a
general class of modified gravity theories and bouncing models. We then study
the effect of such modified dynamics on PBH in the presence of Hawking
radiation and accretion of surrounding materials. We investigate how the
evolution of PBHs is influenced by accretion across different cosmological
eras, considering the radiation, matter, and dark energy-dominated phases like
phantom and quintessence for linear equation of state. We further incorporated
Non-linear Equations of State such as Chaplygin Gas, Modified Chaplygin gas,
Van der Waals model, Polytropic Fluid model. The study systematically analyzes
the mass variation of PBHs in the presence of such different cosmological
environments. The results will contribute to the understanding of PBH
formation and evolution in modified theory of gravity, and their possibility
of being detected with future experiments.

##  I Introduction

Observational physics is currently in its golden age. Wealth of new
observational results both at cosmological and astrophysical scales are being
uncovered. The temperature variations of the CMBR bear testimony of minute
fluctuations in the density of the primordial universe which have been
measured and verified by experiments like WMAP and PLANCK 2013ApJS..208…19H ;
Planck:2018vyg . Experiments build to probe at galactic scales have mapped the
distribution of matter and have found the presence of unknown/unexplained
stuff (hence named it as dark matter) in most parts of the galaxies. There are
a lot of differing theories about what dark matter is made of, and how we
might actually find it. The $\Lambda$CDM model is the current ‘standard model’
of Big Bang cosmology. Consensus is that it is the simplest model that can
account for the various measurements and observations relevant to cosmology.
However, as mentioned, every new theory has its own set of drawbacks and
fundamental issues which calls for further introspection. This model assumes:
GR is the correct theory of gravity on cosmological scales. The current
standard model of cosmology ($\Lambda$CDM+inflation) however faces some
serious fundamental drawbacks, unexplained issues till date. This has
intrigued scientists to probe for alternate theories of gravity, go beyond
Einstein’s GR.

Modified gravity forms a very broad class of theories that aim to overcome the
existing problems in $\Lambda$CDM model and GR. For example, the non-
renormalizability of GR, the cosmological constant problem, tension with
cosmological data, and so on. Such theories lead to improved cosmological
evolution, both at the background as well as at the perturbation levels
CANTATA:2021ktz ; Capozziello:2011et . Such gravitational modifications can be
constructed by modifying Einstein-Hilbert action of GR, resulting to theories
like $f(R)$ gravity DeFelice:2010aj ; Nojiri:2010wj ; Starobinsky:2007hu ;
Cognola:2007zu ; Zhang:2005vt ; Papanikolaou:2021uhe , Gauss-Bonnet and $f(G)$
gravity Nojiri:2005jg ; DeFelice:2008wz ; Zhao:2012vta , Lovelock gravity
Lovelock:1971yv ; Deruelle:1989fj , $f(T)$ gravity Cai:2015emx ; Chen:2010va ;
Bengochea:2010sg ; Benetti:2020hxp , $f(T,T_{G})$ gravity Kofinas:2014owa ;
Kofinas:2014daa , etc. Most of these theories lead to modification of the
evolution of the Hubble parameter from that of GR at the background level.

Primordial black holes (PBHs) are considered to form in the very early
universe out of the gravitational collapse of very high overdensity regions,
whose energy density is higher than a critical threshold Khlopov:2008qy ;
Carr:2020gox . They were introduced back in ‘70s Zeldovich:1967lct ;
1974MNRAS.168..399C ; 1975ApJ…201….1C . Remarkably, as per recent arguments,
PBHs can account for a part or the totality of the dark matter density
Chapline:1975ojl ; Belotsky:2014kca , provide an explanation for the LSS
formation through Poisson fluctuations Meszaros:1975ef ; Afshordi:2003zb ,
they can also seed the supermassive black holes residing in galactic centres
Bean:2002kx ; 1984MNRAS.206..315C . Furthermore, PBHs are associated as well
with numerous gravitational-wave (GW) signals originated from both binary
merging events and stochastic cosmology Sasaki:2018dmp ;
LISACosmologyWorkingGroup:2023njw . They have been extensively analysed in the
context of modified gravity theories Kawai:2021edk ; Yi:2022anu ;
Zhang:2021rqs  and bouncing cosmology Carr:2011hv ; Carr:2014eya ;
Quintin:2016qro ; Chen:2016kjx ; Banerjee:2022xft ; Papanikolaou:2024fzf .
Significant modifications to the perturbation power spectra have been found
due to the presence of modified background dynamics.

Long time ago Hawking showed that black holes can emit thermal radiation
through relativistic quantum effects, leading to black hole evaporation
Hawking:1975vcx . It was also found that small mass PBHs may have evaporated
through the Hawking radiation. The lifetime of such PBHs are comparatively
long, comparable to the age of the universe, owing to their small masses.
Thus, the existence of such PBHs may become possible through the detection of
the emitted Hawking radiation.

PBHs can also accrete matter from the surrounding medium which may lead to an
increase in the lifetime of these black holes. Such long surviving PBHs may
constitute a major portion of dark matter BLAIS200211 ; Blais:2002gw ;
Barrau:2003xp . Recent studies have examined the impact of phantom energy
accretion on the evolution of primordial black holes (PBHs)
jamil2011primordial . These studies indicate that the mass of PBHs decreases
over redshift, thus increasing over time. Additionally, the effects of matter,
radiation, and dark energy (both phantom and quintessence types) accretion
have been explored within the framework of modified $f(T)$ gravity theory. The
evolution of PBH mass due to the accretion of modified Chaplygin gas has also
been investigated in the context of $f(T)$ gravity theorydebnath2015evolution
.

In this paper, we investigate how the evolution of PBHs is influenced by
accretion across different cosmological eras, considering the radiation,
matter, and dark energy-dominated phases. We perform the analysis without
considering any particular model but through a specific parametrization of the
Hubble parameter that reflects the effect of modified background dynamics. By
examining a specific form of the Hubble parameter that represent a variety of
modified gravity and bouncing models and incorporating nonlinear equations of
states (EOS)sadhukhan2023non , we aim to estimate the mass evolution of PBH by
varying cosmological environments. Here we have considered Chaplygin gas
model, Modified chaplygin gas model, Van-der-waals fluid and polytropic model
for the non-linear cases.

This paper has been organised as follows. In Section II, we give a brief
overview of PBHs and the different processes through which the mass of PBH can
change. In Section III, we discuss the general methodology that we have
undertaken and evaluated the general expression for the rate of change of mass
of PBH. We then analyse the rate of change of mass in the presence of
different cosmological fluids/eras. We then extended our analysis to the case
of non-linear models like Chaplygin gas, Vander Waals, Polytropic fluid, etc.
Finally we conclude in Section IV.

##  II Overview

In this section we give a brief overview of Primordial Black Hole. It is known
that Black Holes are the final stage of evolution of a collapsing massive star
whose mass exceeds twice the mass of the sun. There are another kind of Black
Holes which are formed in early stage of the universe due to inhomogeneities
in the scalar fieldfeinstein1995scalar . Primordial Black Holes (PBHs) are
hypothetical black holes that are believed to have formed soon after the Big
Bang. In the early radiation-dominated universe, regions of critical density
containing subatomic matter could have gravitationally collapsed, resulting in
the creation of PBHs without the need for a supernova, which is the process
that forms black holes today. The masses of these primordial black holes are
much smaller than those formed by the collapse of massive stars. PBHs offer a
unique window into the early universe, providing valuable insights into its
extreme conditions and cosmological evolution. Accretion, the process by which
PBHs accumulate mass, plays a crucial role in their evolution and can
significantly influence their properties over cosmic
time.abramowicz2013foundations lasota2016black . Primordial black holes (PBHs)
can increase their mass through gravitational attraction, accreting material
from their surroundings. This accretion process is especially notable during
the early stages of PBH formation and in dense regions of the Universe where
matter is more concentrated. Additionally, the rate at which PBHs accrete mass
depends on the density and composition of the surrounding environment. The
mass of PBHs can decrease due to Hawking radiationpage2005hawking
auffinger2023primordial . Hawking radiation, a theoretical prediction by
physicist Stephen Hawking, significantly influences the evolution of
primordial black holes (PBHs) by causing their gradual mass reduction over
time. According to Hawking’s theory, particles continuously escape from the
black hole. This quantum-level process near the black hole’s boundary leads to
the emission of radiation, known as Hawking radiation. In this paper, we
investigate how the evolution of PBHs is influenced by accretion of cosmic
fluids. The rate of change of mass of PBH depends highly on the nature of
background dynamics, hence on the nature of evolution of the Hubble parameter.
For a given general form of the Hubble parameter, we estimate the mass
evolution of PBH for both linear and non-linear of equation of state. Here we
have considered Chaplygin gas model, Modified chaplygin gas model, Van-der-
waals fluid and polytropic model for the non-linear cases. The observation of
supernovae has led to the prediction that the universe is currently in a phase
of accelerated expansion.riess1998observational perlmutter2003measuring .
Understanding the fundamentals behind the current cosmic acceleration has been
a constant motivation for scientists. Various efforts have been made within
the standard cosmological model to explain the universe’s expansion
sahni2000case ; padmanabhan2003cosmological ; peebles2003cosmological . The
expansion of the universe can be understood by considering that the background
of the universe is predominantly influenced by dark energy (DE), which
currently plays a major role. Observations indicate that dark energy accounts
for approximately 70% of the total energy density of the universe, which has
negative pressureakarsu2014cosmology . In this work, we have considered
various models of dark energy namely, quintessence$(-1<w<-1/3)$Chiba:1999ka
Amendola:1999er , phantom phase$(w<-1)$caldwell2003phantom  etc. We have also
considered the late time universe when the universe was governed by
radiationlora2013pbh  and then matter eraFREESE20021 . We have predominately
investigated the effect of their presence on the net mass change of PBHs with
cosmic evolution. In the following sections, we discuss our methodologies and
results in this regard.

##  III PBH Evolution for a general Hubble Parameter

There are two primary ways in which low mass PBH can loose or gain mass. In
low mass PBH, mass loss occurs through evaporation due to Hawking Radiation.
The rate of mass loss for PBH is given by PhysRevD.13.198

$\dot{M}_{eva}=-\frac{a_{H}}{256G^{2}\pi^{3}M(t)^{2}}$ (1)

where $a_{H}$ is the black body constant.

Now if we consider the presence of cosmic fluid in and around the PBH, this
will lead to an increase in mass of the PBHs through accretion expressed as
babichev2004black ,

$\dot{M}_{accr}=4\pi AM(t)^{2}(\rho+p)$ (2)

where $A$ is the accretion efficiency and $\rho$ is the density of the cosmic
fluid and $p$ is the pressure. We at first consider linear relation between
density and pressure given by $p=w\rho$, where $w$ is the corresponding
equation of state. For radiation dominated era $(w=1/3)$, matter era $w=0$,
dark energy era $w<-1/3$ and phantom accretion $w<-1$.
So, the resulting PBH mass evolution is given by,

$\dot{M}=\dot{M}_{eva}+\dot{M}_{accr}$
$\dot{M}=-\frac{a_{H}}{256G^{2}\pi^{3}M(t)^{2}}+4\pi AM(t)^{2}(\rho+P)$ (3)

Using $a=a_{0}/(1+z)$ and choosing the present value of the scale factor
$a_{0}=1$, we have

$\frac{dz}{dt}=-H(t)(1+z)$ (4)

Without loss of generality, we consider the following parametrization of the
Hubble parameter

$H(t)=\alpha t^{\beta}$ (5)

where the values of the constant $\alpha$ and the power $\beta$ will determine
the model. They constitute our main parameters which will be constrained. This
parametrization not only apply to a class of modified gravity models cai2016f
; yang2011new ; harko2011f ; fernandes20224d ; xu2019f ; Odintsov:2018nch  but
also to a variety of bouncing models Quintin:2014oea ; Cai:2007qw ;
Quintin:2015rta ; Cai:2012va ; Singh:2022jue ; Chen:2022usd . The parameter
$\beta$ can be either integer or fraction, both positive and negative signs.
Through the following analysis, we consider all such possible cases/forms of
the Hubble parameter and see their effect on the evolution of PBH.
Eq. (4) will now be,

$\frac{dz}{dt}=-\alpha t^{\beta}(1+z)$ (6)

The mass evolution for PBH in terms of redshift parameter is now given by,

$\frac{dM}{dz}=-\frac{dM}{dt}\frac{1}{\alpha t^{\beta}(1+z)}$ (7)

By using Eq. (3), we can reconstruct the above equation as,

$\frac{dM}{dz}=-\frac{[-\frac{a_{H}}{256G^{2}\pi^{3}M^{2}}+4\pi
AM^{2}(\rho+P)]}{\alpha t^{\beta}(1+z)}$ (8)

Now, integrating Eq. (6), we get a relation between $t$ and $z$ as,

$t=\left(\frac{\beta+1}{\alpha}ln(1+z)\right)^{\frac{1}{\beta+1}}$ (9)

Substituting Eq. (9) into Eq. (8), we will get,

$\frac{dM}{dz}=-\frac{[-\frac{a_{H}}{256G^{2}\pi^{3}M^{2}}+4\pi
AM^{2}(\rho+P)]}{\alpha(1+z)[\frac{\beta+1}{\alpha}ln(1+z)]^{\frac{\beta}{\beta+1}}}$
(10)

In the following section we study the behaviour of $M$ with redshift in the
presence of different cosmic fluids.

###  III.1 Different Cosmological Era

The evolution of the universe is marked by distinct cosmological eras
characterized by the dominant form of energy. Initially, the ‘Radiation Era’
prevailed, where radiation (photons and neutrinos) dominated the energy
density of the universe. This was followed by the ‘Matter Era’, where matter
(both dark matter and baryonic matter) took over as the primary energy
component, leading to the formation of galaxies, stars, and other cosmic
structures. Currently, the universe is in the ‘Dark Energy Era’, where dark
energy, a mysterious force driving the accelerated expansion of the universe,
dominates the energy density.

####  III.1.1 Radiation era

We now consider the evolution of PBHs in the cosmological background governed
by radiation. We assume that the PBH density is low enough to ensure radiation
domination. For a PBH immersed in the radiation field, the accretion of
radiation leads to the increase of its mass. When a PBH accretes radiation
($w=1/3$), the equation governing this accretion, for a linear equation of
state, is given by,

$\dot{M}_{accr}=\frac{16}{3}\pi AM(t)^{2}\rho$ (11)

and the corresponding PBH evolution is given by,

$\dot{M}=-\frac{a_{H}}{256G^{2}\pi^{3}M(t)^{2}}+\frac{16}{3}\pi AM(t)^{2}\rho$
(12)

We know for radiation era, $\rho=\rho_{0}(1+z)^{4}$, where
$\rho_{0}=\Omega_{r}^{0}\rho_{c}$. $\Omega_{r}^{0}$ is the present value of
the radiation density parameter and $\rho_{c}$ is the critical density.
Therefore, from Eq.(10), the change of mass $M$ of PBH with respect to $z$ for
general form of Hubble parameter is given by,

$\frac{dM}{dz}=-\frac{[-\frac{a_{H}}{256G^{2}\pi^{3}M^{2}}+\frac{16}{3}\pi
AM^{2}\rho_{0}(1+z)^{4}]}{\alpha(1+z)\left[\frac{\beta+1}{\alpha}ln(1+z)\right]^{\frac{\beta}{\beta+1}}}$
(13)

In all the analyses below, we try to analyse the effect of model parameter
$\beta$ for a given value of $\alpha$.

In Fig. 1, we have plotted the variation of $M(z)$ with $z$ for different
values of the model parameter $\beta$. We have also shown plots for different
values of the accretion efficiency. As we see for positive powers of $\beta$,
$M(z)$ decreases with $z$, hence increases with time. Therefore, accreting
radiation leads to an increase in the effective mass of the PBH. The rate of
increase depends on the value of the model parameter $\beta$. Smaller the
value of $\beta$, larger is the increase in mass of PBH, for a given initial
mass. However background dynamics with negative powers of $\beta$, we see in
the last two plots of Fig. 1, the PBH mass either increase very slowly with
time or decrease with increase in time. Thus, we see that for background
dynamics giving rise to negative power dependence of $H$ on $t$, the accretion
rate is either negligible or may not result in any effective increase in the
mass of PBH. In other words, the Hawking radiation either nearly balances
accretion or overpowers it.

Figure 1: Plot of $M(z)$ vs $z$ for radiation era for different values of
$\beta$ and accretion efficiency $A$. For all the plots, the value of
$\alpha=0.01$

####  III.1.2 Matter era

In matter dominated era $(w=0)$, PBH mass can increase by absorbing
surrounding matter at a rate for linear equation of state,

$\dot{M}_{accr}=4\pi AM(t)^{2}\rho$ (14)

So, PBH evolution is given by,

$\dot{M}=-\frac{a_{H}}{256G^{2}\pi^{3}M(t)^{2}}+4\pi AM(t)^{2}\rho$ (15)

For matter dominated era, we know, $\rho=\rho_{0}(1+z)^{3}$, where
$\rho_{0}=\Omega_{m}^{0}\rho_{c}$ where $\Omega_{m}^{0}$ is the present matter
density parameter. Therefore, using Eq.(10), the change of mass $M$ with
respect to $z$ for general form of Hubble parameter is now given by,

$\frac{dM}{dz}=-\frac{[-\frac{a_{H}}{256G^{2}\pi^{3}M^{2}}+4\pi
AM^{2}\rho_{0}(1+z)^{3}]}{\alpha(1+z)[\frac{\beta+1}{\alpha}ln(1+z)]^{\frac{\beta}{\beta+1}}}$
(16)

Figure 2: Plot of $M(z)$ vs $z$ for matter era for different values of $\beta$
and accretion efficiency $A$. For all the plots, the value of $\alpha=0.01$

Similar to the radiation case, in Fig. 2, we have plotted the variation of
$M(z)$ with $z$ for different values of the model parameter $\beta$ (similar
to the ones we used for radiation era). We have also shown plots for different
values of the accretion efficiency. Once again we see for positive powers of
$\beta$, $M(z)$ decreases with $z$, hence increases with time. Therefore,
accretion of matter also leads to an increase in the effective mass of the
PBH. However, similar to the behaviour in the presence of radiation, here also
we see that for background dynamics giving rise to negative power dependence
of $H$ on $t$, the accretion rate is either negligible or may not result in
any effective increase in the mass of PBH.

####  III.1.3 Dark energy Phase

In dark energy dominated era, PBH mass changes due to dark energy at a rate,

$\dot{M}_{accr}=4\pi AM(t)^{2}(1+w)\rho_{DE}$ (17)

with

$\rho=\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}$ (18)

Here we consider two different phases, (a) phantom phase ($w<-1$) (b)
quintessence phase ($-1<w<-1/3$).

So, from Eq.(10) the change of mass $M$ with respect to $z$ for general form
of Hubble parameter is given by,

$\frac{dM}{dz}=-\frac{[-\frac{a_{H}}{256G^{2}\pi^{3}M^{2}}+4\pi
AM^{2}\rho_{0}(\frac{1}{1+z})^{-3(1+w)}]}{\alpha(1+z)\left[\frac{\beta+1}{\alpha}ln(1+z)\right]^{\frac{\beta}{\beta+1}}}$
(19)

Figure 3: Plot of $M(z)$ vs $z$ for dark energy era for both phantom
$(w=-1.5)$ and quintessence $(w=-0.5)$ phases for different values of $\beta$
and accretion efficiency $A$. For all the plots, the value of $\alpha=0.01$

In Fig. 3, we have plotted the variation of $M(z)$ with $z$ for both the
quintessence and phantom phases for $w=-0.5,-1.5$ respectively. We plotted the
graph for different values of the model parameter $\beta$ and accretion
efficiency $A$. We see that when the dark energy behaves like a phantom fluid,
the change in mass of the PBH remains either nearly constant (for positive
$\beta$) or decreases with time (for negative $\beta$). This can be explained
due to the negative pressure provided by the presence of phantom phase that
aids the mass loss due to Hawking evaporation.

In the presence of quintessence phase, the behaviour is similar to that of in
the presence of matter or radiation, i.e. the PBH mass increases with time
(hence decreases with redshift) for positive powers dependency of the Hubble
parameter on time but it can tend to decrease even in the presence of dark
energy fluids in the cases when the Hubble parameter has negative power
dependency on time.

####  III.1.4 Presence of two or more cosmic fluids

We will now consider the presence of more than one cosmic fluid like
radiation, matter and dark energy, then the PBH mass evolution in general form
is given by,

$\frac{dM}{dz}=-\frac{[-\frac{a_{H}}{256G^{2}\pi^{3}M^{2}}+4\pi
AM^{2}\sum_{i=1}^{3}(1+w_{i})\rho_{i}]}{\alpha(1+z)[\frac{\beta+1}{\alpha}ln(1+z)]^{\frac{\beta}{\beta+1}}}$
(20)

where $i$ corresponds to the different cosmological fluid, $w_{i}$ is the
corresponding equation of state and $\rho_{i}$ corresponds to the density of
fluid.

In Fig. 4, Fig. 5 and Fig. 6, we have shown the accretion plots in the
presence of two and three cosmic fluids in different combinations for certain
values of the model parameters. As we can see, the nature of the plots remain
same as before, though the rate of accretion might be more in some cases. Like
before, the negative values of $\beta$ does not lead to an increase in the
mass of PBH with time. Interestingly we see when we consider the presence of
either radiation or matter alongn with phantom fluid, for positive values of
$\beta$, the change in mass with time no longer remains nearly constant
(unlike Fig. 3a) but increases significantly (smaller the value of $\beta$,
larger is the effect), thereby showing that the effect of radiation or matter
dominates the behaviour of mass accretion onto PBH.

Figure 4: Plot of $M(z)$ vs $z$ for the presence of two cosmic fluids for
different values of $\beta$ and accretion efficiency $A$. For all the plots,
the value of $\alpha=0.1$

Figure 5: Plot of $M(z)$ vs $z$ for the presence of two cosmic fluids for
different values of $\beta$ and accretion efficiency $A$. For all the plots,
the value of $\alpha=0.1$

Figure 6: Plot of $M(z)$ vs $z$ for the presence of two cosmic fluids for
different values of $\beta$ and accretion efficiency $A$. For all the plots,
the value of $\alpha=0.1$

###  III.2 Non-Linear Equation of State

Till now our analysis was mainly based on the assumption of linear dependency
between pressure and density. In this section we extend our analysis to the
presence of different types of cosmic fluid with non-linear equation of
statesadhukhan2023non .

We know that the general expression for the rate of change of PBH mass w.r.t.
the redshift is given by Eq. (10). These new models can be used to reconstruct
different cosmological non-linear equation of state like chaplygin
gasgorini2005chaplygin , modified chaplygin gasbenaoum2012modified , Van-der
waals fluidelizalde2017cosmological jantsch2016van  and polytropic
fluidChavanis:2012uq  model with some parameter analysis. Below we discuss
such models by considering different non-linear relations between the density
and pressure of the cosmic fluid.

####  III.2.1 Chaplygin Gas Model

The Chaplygin gas model, originally proposed in cosmology to address the
problem of dark energy, has also been employed in the study of primordial
black hole (PBH) formation and evolution. Unlike the conventional perfect
fluids, the Chaplygin gas model gives a unique equation of state that allows
for negative pressure at high densities. This property is particularly
relevant in the early Universe, where extreme conditions prevail. In the
context of PBH formation, the Chaplygin gas model plays a significant role in
understanding the evolution of PBH mass. The presence of negative pressure can
lead to the amplification of density perturbations, which, under certain
conditions, can undergo gravitational collapse to form PBHs. By incorporating
the effects of the Chaplygin gas equation of state, we can investigate the
conditions under which PBHs form, and subsequent evolution.
In the present case, we study the model of a universe filled with the so
called Chaplygin gas, which is a perfect fluid characterized by the following
equation of state:

$P=-\frac{C}{\rho}$ (21)

where $C$ is a positive constant.
We aim to discuss the phantom phase with parametric analysis in this model.
This model depends on a constant $C$. For that we consider $C>0.$ From
Eq.(10), for chaplygin gas model, the evolution of PBH with respect to
redshift parameter z is given by,

$\begin{split}\frac{dM}{dz}=-\frac{1}{\alpha(1+z)\left[\frac{\beta+1}{\alpha}ln(1+z)\right]^{\frac{\beta}{\beta+1}}}\left[-\frac{a_{H}}{256\pi^{3}G^{2}M^{2}}\right.\\\
\left.+4\pi
AM^{2}\left[\rho_{0}(\frac{1}{1+z})^{-3(1+w)}-\frac{C}{\rho_{0}(\frac{1}{1+z})^{-3(1+w)}}\right]\right]\end{split}$
(22)

where we have also used Eq. (18).

Figure 7: Plot of $M(z)$ vs $z$ for Chaplygin gas model for different values
of $\beta$, $w$ while accretion efficiency $A$ remains fixed. For all the
plots, the value of $\alpha=0.5$, $C=10^{-8}$. Figure 8: Plot of $M(z)$ vs $z$
for Chaplygin gas model for different values of $\beta$, $w$ while accretion
efficiency $A$ remains fixed. For this plot, the value of $\alpha=0.5$,
$C=10^{-5}$.

In Fig. 7 and Fig. 8 we have plotted the variation of $M$ with redshift for
different values of $\beta$ for a given accretion efficiency. We have shown
how the effective mass change of PBH changes with different values of model
parameter. We have further plotted separate graphs for $w=-0.5,0,1/3,1,-1.5$.
As we can see, irrespective of the value of $w$, for certain negative values
of $\beta$ (as we have see in all the earlier cases), the PBH mass decreases
with time even in the presence of Chaplygin gas like cosmic fluid. While for
$w=0,1/3$, we can get a significant increase in the mass of PBH with time for
positive values of $\beta$, for $w=-0.5,-1.5$, the change is nearly negligible
(least being for $w=-1.5$), for $w=1$, the PBH mass will always be decreasing
with time for any value of $\beta$. Furthermore, we see for $w=1$, the PBH
mass decreases at a higher rate for earlier times, attaining a nearly constant
value as we go towards the present time. Thus, we see the dynamics is highly
dependent not only on the model parameter $\beta$ but also on the eqaution of
state parameter $w$.

####  III.2.2 Modified Chaplygin Gas Model

The modified Chaplygin gas model, an extension of the original Chaplygin gas
model, has emerged as a valuable tool in cosmology and astrophysics, including
the study of primordial black hole (PBH) formation and evolution. Unlike the
standard Chaplygin gas, which assumes a specific form for the equation of
state, the modified Chaplygin gas introduces additional parameters $A$ and
$\alpha$ that allows a greater flexibility in modeling various cosmological
scenarios. In the context of PBH formation, the modified Chaplygin gas model
gives insights into the evolution of PBH mass by accounting for non-trivial
interactions between dark energy and dark matter components.
Thus, the modified Chaplygin gas model provides a framework for investigating
the PBH mass evolution processes. So, we can reconstruct the equation of state
of Modified Chaplygin gas. For this purpose the equation of state can be taken
as,

$P=A_{1}\rho-\frac{C}{\rho^{\gamma}}$ (23)

where $A_{1}$, $C$ and $\gamma$ are constant parameters. When $C=0$ we recover
the equation of perfect fluid, i.e. $P=A_{1}\rho.$ For $A_{1}=0$ it reduces to
the generalized Chaplygin gas model. The magnitude of these parameters can
provide different cosmological phases, their evolution. Here we aim to discuss
the phantom phase with parametric analysis in this model. For that we consider
$A_{1}<0$ and $C>0$. So, from Eq.(10) and by using Eq. (18), for modified
chaplygin gas model, the evolution of PBH with respect to redshift parameter z
is given by,

$\begin{split}\frac{dM}{dz}=-\frac{1}{\alpha(1+z)\left[\frac{\beta+1}{\alpha}ln(1+z)\right]^{\frac{\beta}{\beta+1}}}\left[-\frac{a_{H}}{256\pi^{3}G^{2}M^{2}}\right.\\\
\left.+4\pi
AM^{2}\left[\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}+A_{1}\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}\right.\right.\\\
\left.\left.-\frac{C}{(\rho_{0}(\frac{1}{1+z})^{-3(1+w)})^{\gamma}}\right]\right]\end{split}$
(24)

Figure 9: Plot of $M(z)$ vs $z$ for Modified Chaplygin gas model for different
values of $\beta$ while accretion efficiency $A$ remains fixed. For all the
plots, the value of $\alpha=0.5$, $A_{1}=-10^{-5},C=10^{-8},\gamma=-0.7$
Figure 10: Plot of $M(z)$ vs $z$ for Modified Chaplygin gas model for
different values of $\beta$ while accretion efficiency $A$ remains fixed. For
this plot, the value of $\alpha=0.5$, $A_{1}=-10^{1},C=10^{-3},\gamma=-0.7$

In Fig. 9 and Fig. 10 we have plotted the variation of $M$ with redshift for
different values of $\beta$ for a given accretion efficiency. Similar to the
Chaplygin gas model, we have shown how the effective mass change of PBH
changes with different values of model parameter and different values of $w$.
As we can see, upon comparing with the Chaplygin gas model, the behaviour of
the rate of change of PBH mass with redshift w.r.t the change in model
parameter $\beta$ and $w$ follows the same pattern as the Chaplygin gas case.
The conclusions thus follow.

####  III.2.3 Van-Der-Waals Fluid model

This model can discuss the evolution of PBH with just three parameters. The
van der Waals fluid model, initially proposed to describe the behavior of real
gases, has found applications beyond its original scope, including in the
study of Black Holes. In the context of primordial black hole (PBH) formation,
the van der Waals fluid model plays a crucial role in understanding the
evolution of PBH mass. This model considers interactions among particles
within a fluid, accounting for both attractive forces (modeled by the van der
Waals term) and repulsive forces (modeled by excluded volume effects). Such
interactions become significant in the early Universe when high densities and
temperatures prevail. As the Universe cools and expands, critically dense
regions can undergo gravitational collapse, potentially leading to the
formation of PBHs. The van der Waals fluid model provides insights into the
conditions under which these collapses occur, influencing the mass
distribution of PBHs and their subsequent evolution.
So, the reconstruction of Van-Der-Waals fluid model can be done by assuming
the equation of state,

$P=A_{1}\rho+(A_{1}\beta_{1}-\gamma_{1})\rho^{2}$ (25)

This model can describe the cosmic phases and their evolution with the help of
three parameters. As we aim to discuss the phantom phase, we found some
constraints on the parameterssadhukhan2023non  which can be described as
follows, where $A_{1}<0$, $\beta_{1}>0$ and $\gamma_{1}>0$. So, from Eq. (10)
and Eq. (18), for Van-Der-Waals Fluid model, the evolution of PBH with respect
to redshift parameter z is given by,

$\begin{split}\frac{dM}{dz}=-\frac{1}{\alpha(1+z)\left[\frac{\beta+1}{\alpha}ln(1+z)\right]^{\frac{\beta}{\beta+1}}}\left[-\frac{a_{H}}{256\pi^{3}G^{2}M^{2}}\right.\\\
\left.+4\pi
AM^{2}\left[\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}+A_{1}\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}\right.\right.\\\
\left.\left.+(A_{1}\beta_{1}-\gamma_{1})\left(\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}\right)^{2}\right]\right]\end{split}$
(26)

Figure 11: Plot of $M(z)$ vs $z$ for van-der waals fluid model for different
values of $\beta$ while accretion efficiency $A$ remains fixed. For all the
plots, the value of $\alpha=0.5$, $A_{1}=-10^{-5}$, $\beta_{1}=10^{-5}$,
$\gamma_{1}=10^{-5}$ Figure 12: Plot of $M(z)$ vs $z$ for van-der waals fluid
model for different values of $\beta$ while accretion efficiency $A$ remains
fixed. For this plot, the value of $\alpha=0.5$, $A_{1}=-10^{1}$,
$\beta_{1}=10^{-5}$, $\gamma_{1}=10^{-5}$

In Fig. 11 Fig. 12 we have plotted the variation of $M$ with redshift for
different values of $\beta$ for a given accretion efficiency. We have shown
how the effective mass change of PBH changes with different values of model
parameter. We have further plotted separate graphs for different values of
$w$. As we can see, irrespective of the value of $w$, for certain negative
values of $\beta$ (as we have see in all the earlier cases), the PBH mass
decreases with time in the presence of Vander-waals fluid like cosmic fluid.
While for $w=-0.5,-1.5,0,1/3$, we can get an increase in the mass of PBH with
time for positive values of $\beta$, for $w=1$, the PBH mass will always be
decreasing with time for any value of $\beta$. One can also see that the rate
of increase of PBH mass is much more when $w=0,1/3$ and it’s very slow or
nearly constant for $w=-0.5$ and $w=-1.5$ respectively.

####  III.2.4 Polytropic Fluid model

The polytropic fluid model, widely used in astrophysics and cosmology, offers
a framework for understanding the behavior of various astrophysical systems,
including the evolution of primordial black hole (PBH) mass. In this model,
the equation of state relates the pressure of the fluid to its density by the
relation given bellow, allows a wide range of behaviors depending on the value
of the parameters. In the context of PBH formation, the polytropic fluid model
provides valuable insights into the conditions under which gravitational
collapse occurs, leading to PBH formation. By adjusting the parameters, we can
explore different scenarios of PBH formation. Thus, by employing the
polytropic fluid model, we can investigate the dynamics of PBH formation and
their implications for cosmology and astrophysics.
For the reconstruction of polytropic fluid model, the equation of state should
become,

$P=(A_{1}\beta_{1}-\gamma_{1})\rho^{2}$ (27)

By choosing the following constraints on the constants, we will be able to
discuss the phantom phasesadhukhan2023non . Where $A_{1}<0$, $\beta_{1}>0$ and
$\gamma_{1}>0$. So, from Eq.(10) and Eq. (18), for polytropic fluid model, the
evolution of PBH with respect to redshift parameter z is given by,

$\begin{split}\frac{dM}{dz}=-\frac{1}{\alpha(1+z)\left[\frac{\beta+1}{\alpha}ln(1+z)\right]^{\frac{\beta}{\beta+1}}}\left[-\frac{a_{H}}{256\pi^{3}G^{2}M^{2}}\right.\\\
\left.+4\pi
AM^{2}\left[\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}\right.\right.\\\
\left.\left.+(A_{1}\beta_{1}-\gamma_{1})\left(\rho_{0}\left(\frac{1}{1+z}\right)^{-3(1+w)}\right)^{2}\right]\right]\end{split}$
(28)

Figure 13: Plot of $M(z)$ vs $z$ for polytropic fluid for different values of
$\beta$ while accretion efficiency $A$ remains fixed. For all the plots, the
value of $\alpha=0.5$, $A_{1}=-10^{5}$, $\beta_{1}=10^{-5}$,
$\gamma_{1}=10^{-5}$ Figure 14: Plot of $M(z)$ vs $z$ for polytropic fluid for
different values of $\beta$ while accretion efficiency $A$ remains fixed. For
this plot, the value of $\alpha=0.5$, $A_{1}=-10^{5}$, $\beta_{1}=10^{-5}$,
$\gamma_{1}=10^{-5}$

In Fig. 13 and Fig. 14 we have plotted the variation of $M$ with redshift for
different values of $\beta$ for a given accretion efficiency. We have shown
how the effective mass change of PBH changes with different values of model
parameter. We have further plotted separate graphs for different values of
$w$. as we can see, irrespective of the value of $w$, for certain negative
values of $\beta$ (as we have see in all the earlier cases), the PBH mass
decreases with time in the presence of polytropic fluid like cosmic fluid.
While for $w=-0.5,-1.5,0,1/3$, we can get an increase in the mass of PBH with
time for positive values of $\beta$, for $w=1$, the PBH mass will always be
decreasing with time for any value of $\beta$. One can also see that the rate
of increase of PBH mass is much more when $w=0,1/3$ and its very slow or
nearly constant for $w=-0.5$ and $w=-1.5$ respectively.

##  IV Conclusions

In this paper we investigate the effect of accretion of cosmic fluids on the
evolution of primordial black holes within the framework of modified gravity
theories. We consider a general form of the Hubble parameter that represents a
general class of modified gravity theories and bouncing models. Our main model
parameter is defined by the power $\beta$ that represents the time dependence
of the Hubble parameter. We investigated the effect of the presence of
different cosmological era like radiation, matter, dark energy-dominated
phases like phantom and quintessence. We found that in the presence of
radiation or matter or quintessence, the mass of PBH begin to evolve from a
given mass for positive values of $\beta$, leading to longer life span of the
PBHs and making them prone to detection. The rate of mass accretion overpowers
the loss of mass due to Hawking radiation. However, for negative values of
$\beta$, we found that the PBH mass either remains constant or decreases with
time, thus indicating that the presence of such background dynamics will
result in decrease of the life span of the PBH. However, in the presence of
phantom fluid, the change in mass of the PBH remains either nearly constant
(for positive $\beta$) or decreases with time (for negative $\beta$). Since
Hawking radiation and phantom energy accretion both contribute to a decrease
in the mass of the PBH, the PBH that would be expected to decay now due to the
Hawking process would decay earlier due to the inclusion of the phantom
energy.

We then repeated the analysis for dark energy fluids with non linear equation
of state. The non linear models we considered are the Chaplygin gas, Modified
Chaplygin gas, Van der Waals and Polytropic fluid. We studied the variation of
mass of PBH with redshift with the model parameter $\beta$ as well as for
different equation of state parameter $w=-0.5,0,1/3,1,-1.5$. We found that for
all these cases, irrespective of the value of $w$, for certain negative values
of $\beta$, the PBH mass decreases with time. For $w=0,1/3$, we can get a
significant increase in the mass of PBH with time for positive values of
$\beta$. For $w=-0.5,-1.5$, the change is nearly negligible (least being for
$w=-1.5$). And for $w=1$, the PBH mass will always be decreasing with time
irrespective of the value of $\beta$.
Furthermore, we found that for $w=1$, the PBH mass decreases initially,
attaining a nearly constant value as we go towards the present time.

###### Acknowledgements.

SB and AP would like to thank the department of physics, IIT(ISM) Dhanbad for
their support during the completion of this project.

## References

  * [1] Marek A Abramowicz and P Chris Fragile.  Foundations of black hole accretion disk theory.  Living Reviews in Relativity, 16:1–88, 2013.
  * [2] N. Afshordi, P. McDonald, and D. N. Spergel.  Primordial black holes as dark matter: The Power spectrum and evaporation of early structures.  Astrophys. J. Lett., 594:L71–L74, 2003.
  * [3] N. Aghanim et al.  Planck 2018 results. VI. Cosmological parameters.  Astron. Astrophys., 641:A6, 2020.  [Erratum: Astron.Astrophys. 652, C4 (2021)].
  * [4] Özgür Akarsu, Suresh Kumar, R Myrzakulov, M Sami, and Lixin Xu.  Cosmology with hybrid expansion law: scalar field reconstruction of cosmic history and observational constraints.  Journal of Cosmology and Astroparticle Physics, 2014(01):022, 2014.
  * [5] Yashar Akrami et al.  Modified Gravity and Cosmology: An Update by the CANTATA Network.  Springer, 2021.
  * [6] Luca Amendola.  Coupled quintessence.  Phys. Rev. D, 62:043511, 2000.
  * [7] Jérémy Auffinger.  Primordial black hole constraints with hawking radiation—a review.  Progress in Particle and Nuclear Physics, page 104040, 2023.
  * [8] Eugeny Babichev, Vyacheslav Dokuchaev, and Yu Eroshenko.  Black hole mass decreasing due to phantom energy accretion.  Physical Review Letters, 93(2):021102, 2004.
  * [9] Eleni Bagui et al.  Primordial black holes and their gravitational-wave signatures.  10 2023.
  * [10] Shreya Banerjee, Theodoros Papanikolaou, and Emmanuel N. Saridakis.  Constraining F(R) bouncing cosmologies through primordial black holes.  Phys. Rev. D, 106(12):124012, 2022.
  * [11] A. Barrau, David Blais, G. Boudoul, and D. Polarski.  Peculiar relics from primordial black holes in the inflationary paradigm.  Annalen Phys., 13:115–123, 2004.
  * [12] Rachel Bean and Joao Magueijo.  Could supermassive black holes be quintessential primordial black holes?  Phys. Rev. D, 66:063505, 2002.
  * [13] K. M. Belotsky, A. D. Dmitriev, E. A. Esipova, V. A. Gani, A. V. Grobov, M. Yu. Khlopov, A. A. Kirillov, S. G. Rubin, and I. V. Svadkovsky.  Signatures of primordial black hole dark matter.  Mod. Phys. Lett. A, 29(37):1440005, 2014.
  * [14] Hachemi B Benaoum.  Modified chaplygin gas cosmology.  Advances in High Energy Physics, 2012, 2012.
  * [15] Micol Benetti, Salvatore Capozziello, and Gaetano Lambiase.  Updating constraints on f(T) teleparallel cosmology and the consistency with Big Bang Nucleosynthesis.  Mon. Not. Roy. Astron. Soc., 500(2):1795–1805, 2020.
  * [16] Gabriel R. Bengochea.  Observational information for f(T) theories and Dark Torsion.  Phys. Lett. B, 695:405–411, 2011.
  * [17] David Blais, Torsten Bringmann, Claus Kiefer, and David Polarski.  Accurate results for primordial black holes from spectra with a distinguished scale.  Phys. Rev. D, 67:024024, 2003.
  * [18] David Blais, Claus Kiefer, and David Polarski.  Can primordial black holes be a significant part of dark matter?  Physics Letters B, 535(1):11–16, 2002.
  * [19] Yi-Fu Cai, Salvatore Capozziello, Mariafelicia De Laurentis, and Emmanuel N Saridakis.  f (t) teleparallel gravity and cosmology.  Reports on Progress in Physics, 79(10):106901, 2016.
  * [20] Yi-Fu Cai, Salvatore Capozziello, Mariafelicia De Laurentis, and Emmanuel N. Saridakis.  f(T) teleparallel gravity and cosmology.  Rept. Prog. Phys., 79(10):106901, 2016.
  * [21] Yi-Fu Cai, Damien A. Easson, and Robert Brandenberger.  Towards a Nonsingular Bouncing Cosmology.  JCAP, 08:020, 2012.
  * [22] Yi-Fu Cai, Taotao Qiu, Yun-Song Piao, Mingzhe Li, and Xinmin Zhang.  Bouncing universe with quintom matter.  JHEP, 10:071, 2007.
  * [23] Robert R Caldwell, Marc Kamionkowski, and Nevin N Weinberg.  Phantom energy: dark energy with w¡- 1 causes a cosmic doomsday.  Physical review letters, 91(7):071301, 2003.
  * [24] Salvatore Capozziello and Mariafelicia De Laurentis.  Extended Theories of Gravity.  Phys. Rept., 509:167–321, 2011.
  * [25] B. J. Carr.  The primordial black hole mass spectrum.  Astrophys. J. , 201:1–19, October 1975.
  * [26] B. J. Carr and A. A. Coley.  Persistence of black holes through a cosmological bounce.  Int. J. Mod. Phys. D, 20:2733–2738, 2011.
  * [27] B. J. Carr and S. W. Hawking.  Black holes in the early Universe.  MNRAS, 168:399–416, August 1974.
  * [28] B. J. Carr and M. J. Rees.  How large were the first pregalactic objects?  MNRAS, 206:315–325, January 1984.
  * [29] Bernard Carr, Kazunori Kohri, Yuuiti Sendouda, and Jun’ichi Yokoyama.  Constraints on primordial black holes.  Rept. Prog. Phys., 84(11):116902, 2021.
  * [30] Bernard J. Carr.  Primordial Black Holes and Quantum Effects.  Springer Proc. Phys., 170:23–31, 2016.
  * [31] George F. Chapline.  Cosmological effects of primordial black holes.  Nature, 253(5489):251–252, 1975.
  * [32] Pierre-Henri Chavanis.  Models of universe with a polytropic equation of state: III. The phantom universe.  8 2012.
  * [33] Jie-Wen Chen, Junyu Liu, Hao-Lan Xu, and Yi-Fu Cai.  Tracing Primordial Black Holes in Nonsingular Bouncing Cosmology.  Phys. Lett. B, 769:561–568, 2017.
  * [34] Jie-Wen Chen, Mian Zhu, Sheng-Feng Yan, Qing-Qing Wang, and Yi-Fu Cai.  Enhance primordial black hole abundance through the non-linear processes around bounce point.  JCAP, 01:015, 2023.
  * [35] Shih-Hung Chen, James B. Dent, Sourish Dutta, and Emmanuel N. Saridakis.  Cosmological perturbations in f(T) gravity.  Phys. Rev. D, 83:023508, 2011.
  * [36] Takeshi Chiba, Takahiro Okabe, and Masahide Yamaguchi.  Kinetically driven quintessence.  Phys. Rev. D, 62:023511, 2000.
  * [37] G. Cognola, E. Elizalde, S. Nojiri, S. D. Odintsov, L. Sebastiani, and S. Zerbini.  A Class of viable modified f(R) gravities describing inflation and the onset of accelerated expansion.  Phys. Rev. D, 77:046009, 2008.
  * [38] Antonio De Felice and Shinji Tsujikawa.  Construction of cosmologically viable f(G) dark energy models.  Phys. Lett. B, 675:1–8, 2009.
  * [39] Antonio De Felice and Shinji Tsujikawa.  f(R) theories.  Living Rev. Rel., 13:3, 2010.
  * [40] Ujjal Debnath and Bikash Chandra Paul.  Evolution of primordial black hole in f (t) gravity with modified chaplygin gas.  Astrophysics and Space Science, 355:147–153, 2015.
  * [41] Nathalie Deruelle and Luis Farina-Busto.  The Lovelock Gravitational Field Equations in Cosmology.  Phys. Rev. D, 41:3696, 1990.
  * [42] E Elizalde and M Khurshudyan.  Cosmological model involving an interacting van der waals fluid.  arXiv preprint arXiv:1711.01143, 2017.
  * [43] A Feinstein, J Ibanez, and P Labraga.  Scalar field inhomogeneous cosmologies.  Journal of Mathematical Physics, 36(9):4962–4974, 1995.
  * [44] Pedro GS Fernandes, Pedro Carrilho, Timothy Clifton, and David J Mulryne.  The 4d einstein–gauss–bonnet theory of gravity: a review.  Classical and Quantum Gravity, 39(6):063001, 2022.
  * [45] Katherine Freese and Matthew Lewis.  Cardassian expansion: a model in which the universe is flat, matter dominated, and accelerating.  Physics Letters B, 540(1):1–8, 2002.
  * [46] Vittorio Gorini, Alexander Kamenshchik, Ugo Moschella, and Vincent Pasquier.  The chaplygin gas as a model for dark energy.  In The Tenth Marcel Grossmann Meeting: On Recent Developments in Theoretical and Experimental General Relativity, Gravitation and Relativistic Field Theories (In 3 Volumes), pages 840–859. World Scientific, 2005.
  * [47] Tiberiu Harko, Francisco SN Lobo, Shin’ichi Nojiri, and Sergei D Odintsov.  f (r, t) gravity.  Physical Review D, 84(2):024020, 2011.
  * [48] S. W. Hawking.  Particle Creation by Black Holes.  Commun. Math. Phys., 43:199–220, 1975.  [Erratum: Commun.Math.Phys. 46, 206 (1976)].
  * [49] G. Hinshaw, D. Larson, E. Komatsu, D. N. Spergel, C. L. Bennett, J. Dunkley, M. R. Nolta, M. Halpern, R. S. Hill, N. Odegard, L. Page, K. M. Smith, J. L. Weiland, B. Gold, N. Jarosik, A. Kogut, M. Limon, S. S. Meyer, G. S. Tucker, E. Wollack, and E. L. Wright.  Nine-year Wilkinson Microwave Anisotropy Probe (WMAP) Observations: Cosmological Parameter Results.  Apjs, 208(2):19, October 2013.
  * [50] Mubasher Jamil and Asghar Qadir.  Primordial black holes in phantom cosmology.  General Relativity and Gravitation, 43:1069–1082, 2011.
  * [51] Rudinei CS Jantsch, Marcus HB Christmann, and Gilberto M Kremer.  The van der waals fluid and its role in cosmology.  International Journal of Modern Physics D, 25(03):1650031, 2016.
  * [52] Shinsuke Kawai and Jinsu Kim.  Primordial black holes from Gauss-Bonnet-corrected single field inflation.  Phys. Rev. D, 104(8):083545, 2021.
  * [53] Maxim Yu. Khlopov.  Primordial Black Holes.  Res. Astron. Astrophys., 10:495–528, 2010.
  * [54] Georgios Kofinas and Emmanuel N. Saridakis.  Cosmological applications of $F(T,T_{G})$ gravity.  Phys. Rev. D, 90:084045, 2014.
  * [55] Georgios Kofinas and Emmanuel N. Saridakis.  Teleparallel equivalent of Gauss-Bonnet gravity and its modifications.  Phys. Rev. D, 90:084044, 2014.
  * [56] Jean-Pierre Lasota.  Black hole accretion discs.  Astrophysics of Black Holes: From Fundamental Aspects to Latest Developments, pages 1–60, 2016.
  * [57] FD Lora-Clavijo, FS Guzmán, and A Cruz-Osorio.  Pbh mass growth through radial accretion during the radiation dominated era.  Journal of Cosmology and Astroparticle Physics, 2013(12):015, 2013.
  * [58] D. Lovelock.  The Einstein tensor and its generalizations.  J. Math. Phys., 12:498–501, 1971.
  * [59] P. Meszaros.  Primeval black holes and galaxy formation.  Astron. Astrophys., 38:5–13, 1975.
  * [60] Shin’ichi Nojiri and Sergei D. Odintsov.  Modified Gauss-Bonnet theory as gravitational alternative for dark energy.  Phys. Lett. B, 631:1–6, 2005.
  * [61] Shin’ichi Nojiri and Sergei D. Odintsov.  Unified cosmic history in modified gravity: from F(R) theory to Lorentz non-invariant models.  Phys. Rept., 505:59–144, 2011.
  * [62] S. D. Odintsov, V. K. Oikonomou, and S. Banerjee.  Dynamics of inflation and dark energy from $F(R,G)$ gravity.  Nucl. Phys. B, 938:935–956, 2019.
  * [63] Thanu Padmanabhan.  Cosmological constant—the weight of the vacuum.  Physics Reports, 380(5-6):235–320, 2003.
  * [64] Don N. Page.  Particle emission rates from a black hole: Massless particles from an uncharged, nonrotating hole.  Phys. Rev. D, 13:198–206, Jan 1976.
  * [65] Don N Page.  Hawking radiation and black hole thermodynamics.  New Journal of Physics, 7(1):203, 2005.
  * [66] Theodoros Papanikolaou, Shreya Banerjee, Yi-Fu Cai, Salvatore Capozziello, and Emmanuel N. Saridakis.  Primordial black holes and induced gravitational waves in non-singular matter bouncing cosmology.  4 2024.
  * [67] Theodoros Papanikolaou, Charalampos Tzerefos, Spyros Basilakos, and Emmanuel N. Saridakis.  Scalar induced gravitational waves from primordial black hole Poisson fluctuations in f(R) gravity.  JCAP, 10:013, 2022.
  * [68] P James E Peebles and Bharat Ratra.  The cosmological constant and dark energy.  Reviews of modern physics, 75(2):559, 2003.
  * [69] Saul Perlmutter and Brian P Schmidt.  Measuring cosmology with supernovae.  Supernovae and Gamma-Ray Bursters, pages 195–217, 2003.
  * [70] Jerome Quintin and Robert H. Brandenberger.  Black hole formation in a contracting universe.  JCAP, 11:029, 2016.
  * [71] Jerome Quintin, Yi-Fu Cai, and Robert H. Brandenberger.  Matter creation in a nonsingular bouncing cosmology.  Phys. Rev. D, 90(6):063507, 2014.
  * [72] Jerome Quintin, Zeinab Sherkatghanad, Yi-Fu Cai, and Robert H. Brandenberger.  Evolution of cosmological perturbations and the production of non-Gaussianities through a nonsingular bounce: Indications for a no-go theorem in single field matter bounce cosmologies.  Phys. Rev. D, 92(6):063532, 2015.
  * [73] Adam G Riess, Alexei V Filippenko, Peter Challis, Alejandro Clocchiatti, Alan Diercks, Peter M Garnavich, Ron L Gilliland, Craig J Hogan, Saurabh Jha, Robert P Kirshner, et al.  Observational evidence from supernovae for an accelerating universe and a cosmological constant.  The astronomical journal, 116(3):1009, 1998.
  * [74] Shouvik Sadhukhan, Alokananda Kar, and Surajit Chattopadhyay.  A non-linear type equation of state and cosmic fluid dynamics.  arXiv preprint arXiv:2304.06481, 2023.
  * [75] Varun Sahni and Alexei Starobinsky.  The case for a positive cosmological $\lambda$-term.  International Journal of Modern Physics D, 9(04):373–443, 2000.
  * [76] Misao Sasaki, Teruaki Suyama, Takahiro Tanaka, and Shuichiro Yokoyama.  Primordial black holes—perspectives in gravitational wave astronomy.  Class. Quant. Grav., 35(6):063001, 2018.
  * [77] J. K. Singh, Harshna Balhara, Kazuharu Bamba, and J. Jena.  Bouncing cosmology in modified gravity with higher-order curvature terms.  JHEP, 03:191, 2023.  [Erratum: JHEP 04, 049 (2023)].
  * [78] Alexei A. Starobinsky.  Disappearing cosmological constant in f(R) gravity.  JETP Lett., 86:157–163, 2007.
  * [79] Yixin Xu, Guangjie Li, Tiberiu Harko, and Shi-Dong Liang.  f (q, t) gravity.  The European Physical Journal C, 79:1–19, 2019.
  * [80] Rong-Jia Yang.  New types of f (t) gravity.  The European Physical Journal C, 71(11):1797, 2011.
  * [81] Zhu Yi.  Primordial black holes and scalar-induced gravitational waves from the generalized Brans-Dicke theory.  JCAP, 03:048, 2023.
  * [82] Ya. B. Zel’dovich and I. D. Novikov.  The Hypothesis of Cores Retarded during Expansion and the Hot Cosmological Model.  Sov. Astron., 10:602, 1967.
  * [83] Fengge Zhang.  Primordial black holes and scalar induced gravitational waves from the E model with a Gauss-Bonnet term.  Phys. Rev. D, 105(6):063539, 2022.
  * [84] Pengjie Zhang.  Testing $f(R)$ gravity against the large scale structure of the universe.  Phys. Rev. D, 73:123504, 2006.
  * [85] Yue-Yue Zhao, Ya-Bo Wu, Jian-Bo Lu, Zhuo Zhang, Wei-Li Han, and Liang-Liang Lin.  Modified f(G) gravity models with curvature-matter coupling.  Eur. Phys. J. C, 72:1924, 2012.


# DocEnTr: An End-to-End Document Image Enhancement Transformer

Mohamed Ali Souibgui§ Computer Vision Center
Universitat Autònoma de Barcelona
Barcelona, Spain
<EMAIL_ADDRESS>Sanket Biswas§ Computer Vision Center
Universitat Autònoma de Barcelona
Barcelona, Spain
<EMAIL_ADDRESS>Sana Khamekhem Jemni§ Digital Research Center of Sfax
MIRACL Laboratory, University of Sfax
Sfax, Tunisia
<EMAIL_ADDRESS>Yousri Kessentini  Digital Research Center of Sfax
SM@RTS Laboratory
Sfax, Tunisia
<EMAIL_ADDRESS>Alicia Fornés, Josep Lladós  Computer Vision
Center, Computer Science Dept.
Universitat Autònoma de Barcelona
Barcelona, Spain
{afornes<EMAIL_ADDRESS>Umapada Pal  CVPR Unit
Indian Statistical Institute
Kolkata, India
<EMAIL_ADDRESS>
###### Abstract

Document images can be affected by many degradation scenarios, which cause
recognition and processing difficulties. In this age of digitization, it is
important to denoise them for proper usage. To address this challenge, we
present a new encoder-decoder architecture based on vision transformers to
enhance both machine-printed and handwritten document images, in an end-to-end
fashion. The encoder operates directly on the pixel patches with their
positional information without the use of any convolutional layers, while the
decoder reconstructs a clean image from the encoded patches. Conducted
experiments show a superiority of the proposed model compared to the state-of-
the-art methods on several DIBCO benchmarks. Code and models will be publicly
available at: https://github.com/dali92002/DocEnTR.

§§footnotetext: Equal contribution

##  I Introduction

The preservation and legibility of document images (especially the historical
ones) are of utmost priority for the Document Image Analysis and Recognition
(DIAR) research. Document records usually contain significant information and
in the historical cases it dates back centuries and decades [1]. The
conservation of document records can be hampered by several kinds of
degradation such as smears, stains, artefacts, pen strokes, bleed-through
effects and uneven illumination. These distortions could heavily impact the
subsequent downstream tasks for information processing, such as segmentation,
Optical Character Recognition (OCR), information spotting and layout analysis.
This manifests the need for a robust pre-processing task that denoises and
reconstructs a high-quality clean image from its already degraded counterpart.
Document Image Enhancement (DIE) aims towards restoring the quality of the
degraded document samples to yield a clear enhanced version that is locally
uniform.

In recent times, Convolutional Neural Network (CNN)-based approaches have been
widely applied to DIE related sub-tasks, like binarization [2, 3], deblurring
[4], shadow [5] and watermark removal [6], etc. Although the performance of
these models has significantly improved over classical handcrafted techniques,
they do have their own set of drawbacks. Firstly, CNNs operate on regular
grids and using the same convolutional filter to restore different regions of
a degraded document image may not be a sensible choice. Secondly, CNNs fail to
capture high-level long-range dependencies as they are more suited for
extracting low-level spatial information from images.

With the recent success of transformers in Natural Language Processing (NLP)
[7, 8], its application to computer vision problems (like image recognition
[9], object detection [10], visual question answering [11], handwritten text
recognition (HTR) [12], etc.) also started getting more prominence. The self-
attention mechanism proposed in [7] helps to capture global interactions
between contextual features. Using local information combined with the
knowledge of long-range global spatial arrangement is beneficial for an
efficient image restoration model. This local information is often encoded in
the patch content of an image and the large scale organization is contained in
the redundancy of this information across the patches of the image [13].
Contrary to CNNs, which process pixel arrays, Vision Transformers (ViTs) [9]
split an image into fixed-size patches (eg. 8x8, 16x16 etc.), they correctly
embeds each of them as latent representation, and include positional embedding
information as input to the transformer encoder. This allows to encode the
relative location of the patches, along with both local (spatial) and global
(semantic) long-range dependencies. The motivation of using ViTs for our
overall proposed baseline model is that a missing/degraded patch in the
distorted document image can be recovered from the neighbouring patches
information with the power of the multi-head self-attention in ViTs, which
quantifies pairwise global reasoning between them. Also, ViTs have been
adapted in the overall model pipeline in an encoder-decoder based setting,
inspired by the concept of denoising autoencoders [14] used in reconstruction
of corrupted input data. The encoder is mapping the degraded image patches
into latent representations, whereas the decoder is recovering a clean image
version from those encoded representations.

The overall contributions of our work can be summarized into three folds:

  * •

We introduce a simple and flexible Document image Enhancement Transformer
(DocEnTr), an end-to-end image enhancement approach, that effectively restores
and enhances a degraded document image provided as input. As far as we know,
DocEnTr is the first pure transformer-based baseline that leverages the
effectiveness of Vision Transformers (ViTs) in an encoder-decoder based
framework, without any dependency on CNNs.

  * •

We have addressed document binarization as the key problem study in this work
to investigate the power of DocEnTr architecture. Experimental evaluation
shows that DocEnTr achieves state-of-the-art results on standard document
binarization benchmarks (DIBCO), for both machine-printed and handwritten
degraded document images.

  * •

A comprehensive and intuitive case study has been dedicated in Section IV to
prove the utility of ViTs with its multi-headed self-attention mechanism in
the task of document enhancement.

The rest of this paper is organized as follows. In Section II we review the
state of the art. The Document image Enhancement Transformer (DocEnTr) is
described in Section III. Section IV contains an analysis of the extensive
experimentation that has been conducted, including different quantitative and
qualitative studies. Finally, in Section V we draw the conclusions and propose
open challenges for future research directions.

##  II Related Work

###  II-A Document Image Enhancement

This work is an application within the DIE, which has been an active filed
within the DIAR community. The first classic methods were based on
thresholding, which means finding a single (global) or multiple (local)
threshold(s) value(s) for the document. These threshold values are used to
classify the document image pixels into foreground (black) or background
(white) [15, 16]. These methods are still evolving in the recent years using
machine learning tools, for instance, with support vector machines (SVM) [17].
Later, energy based methods were introduced. These are based on tracking the
text pixels by maximizing its energy function [18], while minimizing the one
of the degraded background. However, the results using those approaches were
unsatisfactory [19].

Recently, deep learning based methods were used to tackle this problem by
learning the enhancement directly from raw data. In [20], the problem was
formulated as pixels classification. Each pixel is classified as black or
white depending on a sequence of the surrounding pixels, where a 2D Long
Short-Term Memory (LSTM) was trained for this task. This process is, of
course, time consuming. A more practical solution is to map the images from
the degraded domain to the enhanced one in an end-to-end fashion with CNN
auto-encoders. These latter, hence, were leading the recent improvements in
image denoising [21] and more particularly documents enhancement tasks, like
binarization [22, 23, 24], deblurring problems [4] and so on. Following this
strategy, a fully CNN model was proposed in [25] to binarize the degraded
document images at multiple image scales. Similarly, [2] proposed an auto-
encoder architecture that performs a cascade of pre-trained U-Net models [26]
to learn the binarization using less amount of data. Moreover, generation
models (GAN) were employed for this task to generate clean images by
conditioning on the degraded versions. These architectures are composed of a
generative model that produces a clean version of the image and a
discriminator to assess the binarization result. Both models are usually
composed of fully (or partially) CNN layers. In [6], a conditional GAN
approach was proposed for different enhancement tasks achieving good results
in document images cleaning, binarization, deblurring and dense watermarks
removal. This method was recently extended in [3] by adding a second
discriminator to assess the text readability for the goal of obtaining an
enhanced image that is clean and readable at the same time. A similar cGAN’s
based method was also proposed in [27, 28, 29, 30].

###  II-B Transformers in Vision and Image Enhancement Tasks

In the very recent years, transformers are behind the advances in deep
learning applications. Transformer based architectures firstly showed a great
success in NLP tasks [7, 8] for text translation and embedding, surpassing the
previous LSTM approaches. This motivates many works to employ them for the
vision tasks, for instance, classification [9], object detection [10],
document understanding [31, 32, 33], etc. More related to this paper,
transformers were also used for natural image restoration [34] and document
images dewarping [35]. However, the architectures that were used in these
later image and document enhancement approaches are still relying on the CNN
feature extractors before passing to the transformers stage. Also, the CNN are
used to reconstruct the output image. Contrary, what we are proposing in this
work is a fully transformer approach that attends directly to the patches on
the input images and reconstruct the pixels without the using of any CNN
layer.

##  III Method

Figure 1: Proposed model: The input image is split into patches, which are
linearly embedded, and the position information are added to them. The
resulting sequence of vectors are fed to a standard Transformer encoder to
obtain the latent representations. These representations are fed to another
Transformer representing the decoder to obtain the decoded vector, which is
linearly projected to vectors of pixels representing the output image patches.

The proposed model is a scalable auto-encoder that uses vision transformers in
its encoder and decoder parts, as illustrated in Fig 1. The degraded image is
first divided into patches before entering to the encoder part. During
encoding, the patches are mapped to a latent representation of tokens, where
each token is associated with a degraded patch. Then, the tokens are passed to
the decoder that outputs the enhanced version of patches. Unlike the CNN based
auto-encoders, which were usually employed for the document image enhancement
tasks, the transformer auto-encoder is profiting from the self attention
mechanism which gives a global information during every patch enhancement.
Both decoder and especially encoder are inspired from the vision transformer
(ViT) [9] architecture. We present more details of the model’s architecture in
what follows.

###  III-A Encoder

In the encoding stage (left part of Fig.1), given an image, we divide it into
a set of patches. Then, we embed these patches to obtain the tokens and add
their positional information. After that, a number of transformer blocks is
employed to map these tokens into the encoded latent representation. These
blocks follow the same structure as [9], composed of alternating layers of
multi-headed self-attention and multi-layered perceptron (MLP). Each of these
blocks are preceded by a LayerNorm (LN) [36], and followed by a residual
connection. The patches embedding size and the number of transformer blocks
are set depending on the model size.

###  III-B Decoder

The decoder part consists of a series of transformer blocks (having the same
number as the encoder blocks) that take as an input the sequence of outputted
tokens from the encoder. These tokens are propagated in the transformer
decoder blocks, and then projected with a linear layer to the desired pixel
values. This makes each element of the output correspond to a vector
representing a flattened patch in the output image. The ground truth pixel
values are obtained by dividing the ground truth (GT) clean image into patches
(in the same way as the input degraded image) and flattening them into
vectors. A mean squared error (MSE) loss is used between the model’s output
and the GT pixel patches to train the model.

###  III-C Model Variants

Following a similar convention as previous works [8, 9], the proposed model
configuration can be modified to produce different variants. In our
experiments we define three types of variants which are ”Small”, ”Base” and
”Large”, as enlisted in Table I. Evidently, setting a larger model require
more computational memory and training time since the number of model
parameters is increasing. Thus, a trade off between the model size and its
enhancement performance must be taken into consideration.

TABLE I: Details of our model variants Model | Layers | Dim | Attention Heads | # Parameters
---|---|---|---|---
DocEnTr-Small | 6 | 512 | 4 | 17M
DocEnTr-Base | 12 | 768 | 8 | 68M
DocEnTr-Large | 24 | 1024 | 16 | 255M

##  IV Experimental Validation

To validate our model, we use the datasets proposed in the different DIBCO and
H-DIBCO contests [37] for printed and handwritten degraded document images
binarization and compare our results with the state of the art methods. Before
these experiments, we have conducted different investigations for a proper
selection of the hyperparameters.

###  IV-A Choosing the Best Model Configuration

We begin our experiments by choosing the configuration that gives the best
performance from our model variants (Small, Base or Large). For training, each
degraded image and its GT clean one is divided into overlapped patches with
sizes $256\times 256\times 3$, the overlapping was set vertically and
horizontally by a half of the patches size (means 128). These resultant images
(patches) will be used by our models as an input and expected output (training
data). For results evaluation, and same as the usual approaches [38], we
utilize the following metrics: Peak signal-to-noise ratio (PSNR), F-Measure
(FM), pseudo-F-measure (Fps) and Distance reciprocal distortion metric (DRD).
We used in this experiment the DIBCO 2017 dataset, and the obtained results
are given in Table II. As it can be seen, a larger model gives a better result
in all the metrics, but it requires more computation resources. Thus, we
recommend using a Base model for a binarization task. Nevertheless, we will
test as well the Large version in following experiments.

TABLE II: Results of varying the model size for the DIBCO 2017 dataset. $\uparrow$: The higher the better. $\downarrow$: The lower the better. Model | PSNR $\uparrow$ | FM $\uparrow$ | Fps $\uparrow$ | DRD $\downarrow$
---|---|---|---|---
DocEnTr-Small | 18.29 | 91.06 | 93.82 | 2.78
DocEnTr-Base | 18.69 | 91.66 | 94.11 | 2.63
DocEnTr-Large | 18.85 | 92.14 | 94.58 | 2.53

Next, we do another experiment related to the input image size, and the
patches size that are used by our model. The reason behind is that having
different image size and patch size can affect the binarization since the
model is accessing to different type of information (from global to local).
The obtained results using the Base model are given in Table III. As it can be
seen, a slightly better performance is obtained using an input with the
smaller size ($256\times 256\times 3$ compared to $512\times 512\times 3$ ).
However, we can notice that the performance is highly improved when using a
smaller patch size. The reason is that, by employing a smaller patch size, we
make each patch of the image attending to more and much local patches during
the self-attention. Thus, the model is looking to more and much fine
information during the enhancement process with $8\times 8$ patch size. But,
as before, using a smaller patch size means augmenting the model parameters,
requiring more computation resources.

TABLE III: Results of varying the input and patch sizes for the DIBCO 2017 dataset Input Size | Patch Size | PSNR $\uparrow$ | FM $\uparrow$ | Fps $\uparrow$ | DRD $\downarrow$
---|---|---|---|---|---
$256\times 256\times 3$ | $8\times 8$ | 19.11 | 92.53 | 95.15 | 2.37
$256\times 256\times 3$ | $16\times 16$ | 18.69 | 91.66 | 94.11 | 2.63
$256\times 256\times 3$ | $32\times 32$ | 17.57 | 89.37 | 91.99 | 3.44
$512\times 512\times 3$ | $8\times 8$ | 18.91 | 92.2 | 94.93 | 2.45
$512\times 512\times 3$ | $16\times 16$ | 18.66 | 92.15 | 93.89 | 2.54
$512\times 512\times 3$ | $32\times 32$ | 17.27 | 89.43 | 91.51 | 3.54

###  IV-B Quantitative Evaluation

After choosing the best hyper-parameters of the model, we conduct the
experiments on the different datasets and compare our result with the related
approaches. We begin by testing with the DIBCO 2011 dataset [39]. This dataset
contains degraded document images with handwritten and printed text. For
training, we use all the images from the other DIBCO and H-DIBCO datasets
(except DIBCO 2019) and the Palm Leaf dataset [40]. These images are split
into overlapped images with size $256\times 256\times 3$ before being fed to
the model. The obtained results are given in Table IV, where we can notice a
superiority of out method compared to the different variations of the related
approaches. We choose to compare with different families of approaches:
classic thresholding and deep learning based methods (whether basing on CNN or
cGAN). Our model DocEnTr-Base{8}, which means using the Base setting with a
patch size of $8\times 8$, gives the best PSNR and DRD compared to all the
other methods. While the model DocEnTr-Large{16}, which means using the Large
setting with a patch size of $16\times 16$, leads to the second best
performance in the metrics PSNR, Fps and DRD. We note that for a computation
reason, we were not able to train the Large setting with a patch size of
$8\times 8$.

TABLE IV: Comparative results of our proposed method on DIBCO 2011 Dataset. Thresh: Thresholding, Tr: Transformers. Method | Model | PSNR $\uparrow$ | FM $\uparrow$ | Fps $\uparrow$ | DRD $\downarrow$
---|---|---|---|---|---
Otsu [15] | Thres. | 15.70 | 82.10 | – | 9.00
Savoula et al. [16] | Thres. | 15.60 | 82.10 | – | 8.50
Vo et al. [41] | CNN | 20.10 | 93.30 | – | 2.00
Kang et al [2] | CNN | 19.90 | 95.50 | – | 1.80
Tensmeyer et al [25] | CNN | 20.11 | 93.60 | 97.70 | 1.85
Zhao et al. [41] | cGAN | 20.30 | 93.80 | – | 1.80
DocEnTr-Base{8} | Tr | 20.81 | 94.37 | 96.15 | 1.63
DocEnTr-Base{16} | Tr | 20.11 | 93.48 | 96.12 | 1.93
DocEnTr-Large{16} | Tr | 20.62 | 94.24 | 96.71 | 1.69

After that, we test our model on the H-DIBCO 2012 dataset [42], which contains
degraded handwritten document images. As in the previous experiment, we use
the other datasets for training with the same split size. The obtained results
are shown in Table V, where we can notice that our model gives the best
performance in terms of PSNR and FM with the Base{8} configuration. We notice
also that the other configuration gives competitive results compared to the
other approaches.

TABLE V: Comparative results of our proposed method on H-DIBCO 2012 Dataset. Thresh: Thresholding, Tr: Transformers. Method | Model | PSNR $\uparrow$ | FM $\uparrow$ | Fps $\uparrow$ | DRD $\downarrow$
---|---|---|---|---|---
Otsu [15] | Thres. | 15.03 | 80.18 | 82.65 | 26.46
Savoula et al. [16] | Thres. | 16.71 | 82.89 | 87.95 | 6.59
Kang et al [2] | CNN | 21.37 | 95.16 | 96.44 | 1.13
Tensmeyer et al [25] | CNN | 20.60 | 92.53 | 96.67 | 2.48
Zhao et al. [41] | cGAN | 21.91 | 94.96 | 96.15 | 1.55
Jemni et al. [3] | cGAN | 22.00 | 95.18 | 94.63 | 1.62
DocEnTr-Base{8} | Tr | 22.29 | 95.31 | 96.29 | 1.60
DocEnTr-Base{16} | Tr | 21.03 | 93.31 | 94.72 | 2.31
DocEnTr-Large{16} | Tr | 22.04 | 95.09 | 96.00 | 1.64

Moreover, we tested with the more recent DIBCO 2017 dataset. In this dataset
our model achieves the best performance in all the evaluation metrics, as
presented in Table VI.

TABLE VI: Comparative results of our proposed method on DIBCO 2017 Dataset. Thresh: Thresholding, Tr: Transformers. Method | Model | PSNR $\uparrow$ | FM $\uparrow$ | Fps $\uparrow$ | DRD $\downarrow$
---|---|---|---|---|---
Otsu [15] | Thres. | 13.85 | 77.73 | 77.89 | 15.54
Savoula et al. [16] | Thres. | 14.25 | 77.11 | 84.1 | 8.85
Kang et al [2] | CNN | 15.85 | 91.57 | 93.55 | 2.92
Competition top [19] | CNN | 18.28 | 91.04 | 92.86 | 3.40
Zhao et al. [41] | cGAN | 17.83 | 90.73 | 92.58 | 3.58
Jemni et al. [3] | cGAN | 17.45 | 89.8 | 89.95 | 4.03
DocEnTr-Base{8} | Tr | 19.11 | 92.53 | 95.15 | 2.37
DocEnTr-Base{16} | Tr | 18.69 | 91.66 | 94.11 | 2.63
DocEnTr-Large{16} | Tr | 18.85 | 92.14 | 94.58 | 2.53

Lastly, we test on the H-DIBCO 2018 dataset. Here, as shown in Table VII, the
best performance is achieved by [3] basing on cGAN. Anyway, we can notice that
our model is still very competitive since it ranks second in the PSNR, FM and
Fps metrics.

TABLE VII: Comparative results of our proposed method on DIBCO 2018 Dataset. Thresh: Thresholding, Tr: Transformers. Method | Model | PSNR $\uparrow$ | FM $\uparrow$ | Fps $\uparrow$ | DRD $\downarrow$
---|---|---|---|---|---
Otsu [15] | Thres. | 9.74 | 51.45 | 53.05 | 59.07
Savoula et al. [16] | Thres. | 13.78 | 67.81 | 74.08 | 17.69
Kang et al [2] | CNN | 19.39 | 89.71 | 91.62 | 2.51
Competition top [19] | CNN | 19.11 | 88.34 | 90.24 | 4.92
Zhao et al. [41] | cGAN | 18.37 | 87.73 | 90.60 | 4.58
Jemni et al. [3] | cGAN | 20.18 | 92.41 | 94.35 | 2.60
DocEnTr-Base{8} | Tr | 19.46 | 90.59 | 93.97 | 3.35
DocEnTr-Base{16} | Tr | 19.33 | 89.97 | 93.5 | 3.68
DocEnTr-Large{16} | Tr | 19.47 | 89.21 | 92.54 | 3.96

To summarize the quantitative evaluation, we demonstrate that our model gives
good results compared to the state of the art approaches. This was shown by
obtaining the best results in most of the evaluation metrics with the H-DIBCO
2011, DIBCO 2012 and DIBCO 2017 benchmarks.

###  IV-C Qualitative Evaluation

After presenting the achieved quantitative results by our model, we present in
this subsection some qualitative results. We begin by showing the enhancing
performance of our method. This is illustrated in Fig. 2, where we compare our
binarization results with the GT clean images. As it can be seen, our model
produces highly clean images, which are very close to the optimal GT images,
reflecting the good quantitative performance that was obtained in the previous
subsection.

|  |
---|---|---
|  |
|  |
|  |
|  |
|  |
|  |
|  |
Figure 2: Qualitative results of our proposed method in binarization of some
samples from the DIBCO and H-DIBCO datasets. Images in columns are: Left:
original image, Middle: GT image, Right: Binarized image using our proposed
method.

Then, we present a quantitative comparison of our method with the related
approaches. This is shown in Fig. 3, where we can notice the superiority of
our model in recovering a highly degraded image over the classic thresholding
[15, 16], CNN [2], and cGAN [3] methods.

|
---|---
Original | Ground Truth
|
Otsu [15] | Sauvola et al. [16]
|
Jemni et al. [3] | Kang et al. [2]
|
Competition Winner | Ours
Figure 3: Qualitative results of the different binarization methods on the
sample number 12 from DIBCO 2017 Dataset.

###  IV-D Self-attention Mechanism

As we stated above, our method differs from the CNN related ones by employing
the transformers to enhance the degraded document images. The self-attention
mechanism used in the transformer blocks gives a global view to every token on
the other tokens that represents the patches within the image for a better
enhancing result. A visual illustration of the attention maps of the last
layer from the encoder is given in Fig. 4. As it can be seen, a token can
attend to all the patches within the image. In these test cases each token
(patch representation) is focusing on the text elements, while ignoring the
degraded patches. Thus, the attending patches are decoded later and projected
to pixels while taking into consideration a high-level global information from
the attended neighbouring patches that cover the full input image. We also
notice that the attention maps are mostly matching the text of the GT images,
which lead to a satisfactory binarization result that is closer to the GT.
This supports the utility of using the transformers with its powerful self-
attention mechanism in the image enhancement task. However, in other sample
cases as illustrated in Fig. 5, we observe that the attention maps are
considering some portions of the text as a background region. Hence, the
resultant enhanced image is removing foreground text because it considers it
as a background noise. This explains the failure of the self-attention
paradigm in these scenarios.

Degraded | Attention | Predicted | GT
---|---|---|---
|  |  |
|  |  |
|  |  |
|  |  |
Figure 4: Attention maps from the 2nd head of the last layer of DocEnTr{8} encoder. We display the self-attention for different (random) tokens. Degraded | Attention | Predicted | GT
---|---|---|---
|  |  |
|  |  |
Figure 5: Attention maps from the 2nd head of the last layer of DocEnTr{8}
encoder. We display the self-attention for different (random) tokens. (A
failure case).

##  V Conclusion

This paper presents a novel transformer-based architecture called DocEnTr for
document image enhancement. To the best of our knowledge, this is the first
pure transformer model addressing DIE related problems. The model captures
high-level global long-range dependencies using the self-attention mechanism
for a better performance. Quantitative and qualitative results on the DIBCO
benchmarks prove the effectiveness of DocEnTr in recovering highly degraded
document images. It is a simple and flexible framework that can also be easily
applied to enhance other kinds of degradation occurring in document images
(like blur, shadow, warps, stains etc). These aspects will be investigated in
a future work. We also wish to investigate a self-supervised learning stage
that can substantially benefit from large amounts of unlabeled data.

## Acknowledgment

This work has been partially supported by the Swedish Research Council (grant
2018-06074, DECRYPT), the Spanish projects RTI2018-095645-B-C21, the CERCA
Program / Generalitat de Catalunya, the FCT-19-15244, the Catalan projects
2017-SGR-1783, PhD Scholarship from AGAUR (2021FIB-10010) and DocPRESERV
project (Swedish STINT grant).

## References

  * [1] B. Megyesi, N. Blomqvist, and E. Pettersson, “The decode database: Collection of historical ciphers and keys,” in _The 2nd International Conference on Historical Cryptology, HistoCrypt 2019, June 23-26 2019, Mons, Belgium_ , 2019, pp. 69–78.
  * [2] S. Kang, B. K. Iwana, and S. Uchida, “Complex image processing with less data document image binarization by integrating multiple pretrained u-net modules,” _Pattern Recognition_ , vol. 109, p. 107577, 2021.
  * [3] S. K. Jemni, M. A. Souibgui, Y. Kessentini, and A. Fornés, “Enhance to read better: A multi-task adversarial network for handwritten document image enhancement,” _Pattern Recognition_ , vol. 123, p. 108370, 2022.
  * [4] M. Hradiš, J. Kotera, P. Zemcık, and F. Šroubek, “Convolutional neural networks for direct text deblurring,” in _Proceedings of BMVC_ , vol. 10, no. 2, 2015.
  * [5] B. Wang and C. L. P. Chen, “An effective background estimation method for shadows removal of document images,” in _2019 IEEE International Conference on Image Processing (ICIP)_ , 2019, pp. 3611–3615.
  * [6] M. A. Souibgui and Y. Kessentini, “De-gan: A conditional generative adversarial network for document enhancement,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2020.
  * [7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in neural information processing systems_ , 2017, pp. 5998–6008.
  * [8] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [9] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in _International Conference on Learning Representations_ , 2021.
  * [10] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _European Conference on Computer Vision_. Springer, 2020, pp. 213–229.
  * [11] A. F. Biten, R. Litman, Y. Xie, S. Appalaraju, and R. Manmatha, “Latr: Layout-aware transformer for scene-text vqa,” _arXiv preprint arXiv:2112.12494_ , 2021.
  * [12] A. C. Rouhou, M. Dhiaf, Y. Kessentini, and S. B. Salem, “Transformer-based approach for joint handwriting and named entity recognition in historical document,” _Pattern Recognition Letters_ , 2021.
  * [13] V. De Bortoli, A. Desolneux, B. Galerne, and A. Leclaire, “Patch redundancy in images: A statistical testing framework and some applications,” _SIAM Journal on Imaging Sciences_ , vol. 12, no. 2, pp. 893–926, 2019.
  * [14] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, “Extracting and composing robust features with denoising autoencoders,” in _Proceedings of the 25th international conference on Machine learning_ , 2008, pp. 1096–1103.
  * [15] N. Otsu, “A threshold selection method from gray-level histograms,” _IEEE transactions on systems, man, and cybernetics_ , vol. 9, no. 1, pp. 62–66, 1979.
  * [16] J. Sauvola and M. Pietikäinen, “Adaptive document image binarization,” _Pattern recognition_ , vol. 33, no. 2, pp. 225–236, 2000.
  * [17] W. Xiong, J. Xu, Z. Xiong, J. Wang, and M. Liu, “Degraded historical document image binarization using local features and support vector machine (svm),” _Optik_ , vol. 164, pp. 218–223, 2018.
  * [18] R. Hedjam, M. Cheriet, and M. Kalacska, “Constrained energy maximization and self-referencing method for invisible ink detection from multispectral historical document images,” in _2014 22nd International Conference on Pattern Recognition_. IEEE, 2014, pp. 3026–3031.
  * [19] I. Pratikakis, K. Zagoris, G. Barlas, B. Gatos, “Icdar 2017 competition on document image binarization (dibco 2017),” in _2017 International Conference on Document Analysis and Recognition_. IEEE, 2017, pp. 1395––1403.
  * [20] M. Z. Afzal, J. Pastor-Pellicer, F. Shafait, T. M. Breuel, A. Dengel, and M. Liwicki, “Document image binarization using lstm: A sequence learning approach,” in _Proceedings of the 3rd international workshop on historical document imaging and processing_ , 2015, pp. 79–84.
  * [21] X.-J. Mao, C. Shen, and Y.-B. Yang, “Image restoration using very deep convolutional encoder-decoder networks with symmetric skip connections,” in _Proceedings of the 30th International Conference on Neural Information Processing Systems_ , 2016, pp. 2810–2818.
  * [22] K. G. Lore, A. Akintayo, and S. Sarkar, “Llnet: A deep autoencoder approach to natural low-light image enhancement,” _Pattern Recognition_ , vol. 61, pp. 650–662, 2017.
  * [23] J. Calvo-Zaragoza and A.-J. Gallego, “A selectional auto-encoder approach for document image binarization,” _Pattern Recognition_ , vol. 86, pp. 37–47, 2019.
  * [24] Y. Akbari, S. Al-Maadeed, and K. Adam, “Binarization of degraded document images using convolutional neural networks and wavelet-based multichannel images,” _IEEE Access_ , vol. 8, pp. 153 517–153 534, 2020.
  * [25] C. Tensmeyer and T. Martinez, “Document image binarization with fully convolutional neural networks,” in _2017 14th IAPR international conference on document analysis and recognition (ICDAR)_ , vol. 1. IEEE, 2017, pp. 99–104.
  * [26] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [27] J. Zhao, C. Shi, F. Jia, Y. Wang, and B. Xiao, “Document image binarization with cascaded generators of conditional generative adversarial networks,” _Pattern Recognition_ , vol. 96, p. 106968, 2019.
  * [28] A. K. Bhunia, A. K. Bhunia, A. Sain, and P. P. Roy, “Improving document binarization via adversarial noise-texture augmentation,” in _2019 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2019, pp. 2721–2725.
  * [29] M. O. Tamrin, M. El-Amine Ech-Cherif, and M. Cheriet, “A two-stage unsupervised deep learning framework for degradation removal in ancient documents,” in _Pattern Recognition. ICPR International Workshops and Challenges_. Springer International Publishing, 2021, pp. 292–303.
  * [30] M. A. Souibgui, Y. Kessentini, and A. Fornés, “A conditional gan based approach for distorted camera captured documents recovery,” in _Mediterranean Conference on Pattern Recognition and Artificial Intelligence_. Springer, 2020.
  * [31] Y. Xu, Y. Xu, T. Lv, L. Cui, F. Wei, G. Wang, Y. Lu, D. Florencio, C. Zhang, W. Che _et al._ , “Layoutlmv2: Multi-modal pre-training for visually-rich document understanding,” _arXiv preprint arXiv:2012.14740_ , 2020.
  * [32] S. Appalaraju, B. Jasani, B. U. Kota, Y. Xie, and R. Manmatha, “Docformer: End-to-end transformer for document understanding,” _ICCV_ , 2021.
  * [33] P. Li, J. Gu, J. Kuen, V. I. Morariu, H. Zhao, R. Jain, V. Manjunatha, and H. Liu, “Selfdoc: Self-supervised document representation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 5652–5660.
  * [34] J. Liang, J. Cao, G. Sun, K. Zhang, L. Van Gool, and R. Timofte, “Swinir: Image restoration using swin transformer,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 1833–1844.
  * [35] H. Feng, Y. Wang, W. Zhou, J. Deng, and H. Li, “Doctr: Document image transformer for geometric unwarping and illumination correction,” _arXiv preprint arXiv:2110.12942_ , 2021.
  * [36] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” _arXiv preprint arXiv:1607.06450_ , 2016.
  * [37] I. Pratikakis, K. Zagori, P. Kaddas, and B. Gatos, “Icfhr 2018 competition on handwritten document image binarization (h-dibco 2018),” in _2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR)_ , 2018, pp. 489–493.
  * [38] I. Pratikakis, B. Gatos, K. Ntirogiannis, “H-dibco 2010 - handwritten docu- ment image binarization competition,” in _International Conference on Frontiers in Handwriting Recognition_. IEEE, 2010, pp. 727––732.
  * [39] K. N. I. Pratikakis, B. Gatos, “Icdar 2011 document image binariza- tion contest (dibco 2011),” in _2011 International Conference on Document Analysis and Recognition_ , 2011, p. 1506–1510.
  * [40] J.-C. Burie, M. Coustaty, S. Hadi, M. W. A. Kesiman, J.-M. Ogier, E. Paulus, K. Sok, I. M. G. Sunarya, and D. Valy, “Icfhr2016 competition on the analysis of handwritten text in images of balinese palm leaf manuscripts,” in _2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR)_. IEEE, 2016, pp. 596–601.
  * [41] Q.N. Vo, S.H. Kim, H.J. Yang, G. Lee, “Binarization of degraded document images based on hierarchical deep supervised network,” _Pattern Recognition_ , vol. 74, pp. 568––586, 2018.
  * [42] I. Pratikakis, B. Gatos and K. Ntirogiannis, “ ICFHR 2012 competition on handwritten document image binarization (H-DIBCO 2012),” in _Proceedings of the International Conference on Frontiers in Handwriting Recognition_. IEEE, 2012, pp. 817––822.