# Regular black holes and gravitational particle-like solutions in generic
DHOST theories

Olaf Baake†‡, Christos Charmousis♯, Mokhtar Hassaine† and Miguel San Juan†
†Instituto de Matemáticas, Universidad de Talca, Casilla 747, Talca 3460000,
Chile,
‡Centro de Estudios Cientificos (CECs), Av. Arturo Part 514, Valdivia 5090000,
Chile,
♯Université Paris-Saclay, CNRS/IN2P3, IJCLab, 91405 Orsay, France

###### Abstract

We construct regular, asymptotically flat black holes of higher order scalar
tensor (DHOST) theories, which are obtained by making use of a generalized
Kerr-Schild solution generating method. The solutions depend on a mass
integration constant, admit a smooth core of chosen regularity, and
generically have an inner and outer event horizon. In particular, below a
certain mass threshold, we find massive, horizonless, particle-like solutions.
We scan through possible observational signatures ranging from weak to strong
gravity and study the thermodynamics of our regular solutions, comparing them,
when possible, to General Relativity black holes and their thermodynamic laws.

##  1 Introduction

It is an undeniable fact, whose origin goes back to the Schwarzschild
solution, that the notion of a black hole is intimately linked to the concept
of spacetime singularities. In fact, it is well-known that, under certain
energy conditions, classical solutions of general relativity exhibit
singularities as a direct consequence of the so-called singularity theorems
[1, 2]. The appearance of singularities is essentially due to the classical
character of the theory of general relativity, and a quantum theory of gravity
may be expected to cure such pathologies.

However, in the absence of a complete theory of quantum gravity one can search
for black hole spacetimes with a global structure similar to the well-known
solutions (like the Schwarzschild or Reissner-Nordström solutions), but in
which the central singularity is absent. Such solutions are commonly known as
regular black holes. These ideas originate from the pioneering works of
Sakharov [3], Gilner [4] and also Bardeen [5] who presented the first example
of a regular black hole as an ad-hoc metric (not originating from an action).
A physical construction of the Bardeen metric as a solution of a given action
was finally proposed much later in [6]. There the authors showed that the
Bardeen metric can be obtained from the Einstein equations with a non-linear
magnetic source. Although the Bardeen metric was the first example of a
regular spacetime, the first exact regular black hole solution of a given
theory was found by Ayón-Beato and Garcia [7] for the Einstein equations
coupled to a specific non-linear electrodynamic source.

Models involving non-linear electrodynamics have been fruitful in the
construction of regular solutions, see e.g. Refs. [8], and also Ref. [9] for a
review. Many of these regular black holes present a de Sitter core at the
origin, and their regularity is quantified by a regularizing parameter
identified with a non-linear electrodynamic charge. It is also important to
stress that the parameter of regularization is not a constant of integration
but is rather an input of the matter action. This observation has important
consequences, for example on the thermodynamic properties of these regular
solutions. Indeed, depending on whether the regularizing parameter is
considered as a varying parameter or not, the thermodynamic properties may be
different. In order to illustrate this fact, one can note that for the Bardeen
regular black hole, the one-quarter area law of the entropy is usually
violated [10] when considering a non varying magnetic charge, while this
”universal” law can be restored by promoting the magnetic charge as a variable
parameter [11]. Note that in certain non-minimally coupled Lagrangians
analytic regular solutions were found where the mass and the charge truly are
integration constants [12].

In this work we will construct regular black hole solutions, which are
asymptotically very similar to Schwarzschild, without the need of introducing
an additional regularization parameter inherent to the action. For these black
holes, regularity will not be enforced by the fine tuning of some action
parameter, it will rather be achieved due to the functional form of the
regularizing function appearing in the solutions. In other words, the fall-off
of the mass term of our solutions turns out to be an analytic function with a
de Sitter core at the origin as a consequence of the field equations. The
degree of regularity and its strength are monitored by two parameters, one
fixing the core to be de Sitter or higher and one fixing the strength of the
higher order term against the mass of the black hole. The regular black holes
found here are exact solutions of scalar tensor theories beyond those
initially proposed by Horndeski [13]. The regularizing function sets, as one
would expect, the scalar degree of freedom without any fine tuning of the
theory.

The scalar tensor theories have higher than second derivative equations of
motion and are (still) free of Ostrogradski type pathologies [14, 15]. These
general Lagrangians have been dubbed Degenerate Higher Order Scalar Tensor
(DHOST) or Extended Scalar Tensor (EST) theories [14, 15, 16], and are widely
studied in the literature (see for example [17, 18, 19, 20] for their compact
objects and [21] for a review). More precisely, we will consider the following
class of shift symmetric and parity preserving DHOST theories that contain up
to second order covariant derivatives of the scalar field (in the action),

$\displaystyle S[g,\phi]=\int d^{4}x\sqrt{-g}$
$\displaystyle\Big{[}K(X)+G(X)R+A_{1}(X)\left[\phi_{\mu\nu}\phi^{\mu\nu}-(\Box\phi)^{2}\right]+A_{3}(X)\Box\phi\,\phi^{\mu}\phi_{\mu\nu}\phi^{\nu}$
(1.1)
$\displaystyle+A_{4}(X)\phi^{\mu}\phi_{\mu\nu}\phi^{\nu\rho}\phi_{\rho}+A_{5}(X)\left(\phi^{\mu}\phi_{\mu\nu}\phi^{\nu}\right)^{2}\Big{]},$

where the coupling functions $K,G,A_{1},A_{3},A_{4}$ and $A_{5}$ depend only
on the kinetic term of the scalar field $X=g^{\mu\nu}\phi_{\mu}\phi_{\nu}$,
and where $\phi_{\mu}=\partial_{\mu}\phi$ and
$\phi_{\mu\nu}=\nabla_{\mu}\nabla_{\nu}\phi$. The coupling functions $A_{4}$
and $A_{5}$ are chosen to satisfy

$\displaystyle A_{4}$ $\displaystyle=$
$\displaystyle\frac{1}{8(G-XA_{1})^{2}}\left\\{4G\left[3(-A_{1}+2G_{X})^{2}-2A_{3}G\right]-A_{3}X^{2}(16A_{1}G_{X}+A_{3}G)\right.$
$\displaystyle\left.+4X\left[-3A_{2}A_{3}G+16A_{1}^{2}G_{X}-16A_{1}G_{X}^{2}-4A_{1}^{3}+2A_{3}GG_{X}\right]\right\\},$
$\displaystyle A_{5}$ $\displaystyle=$
$\displaystyle\frac{1}{8(G-XA_{1})^{2}}(2A_{1}-XA_{3}-4G_{X})\left(A_{1}(2A_{1}+3XA_{3}-4G_{X})-4A_{3}G\right)$
(1.2)

in order to ensure the absence of Ostrogradski ghosts [14, 15]. Recently, it
has been shown that regular black hole solutions for this class of theories
can be constructed (including the well known cases of Bardeen [5] or Hayward
metrics [22]), see Ref. [23]. The algorithm of construction is a byproduct of
extending the Kerr-Schild solution generating method to scalar tensor
theories. The key point in extending this well known method from GR is the
assumption that the kinetic term of the scalar field remains unchanged under
the static (usual) Kerr-Schild transformation. Another crucial observation
that we make here is that regular black holes cannot belong to Horndeski
theory. We will see that the theories involving regular black holes correspond
to a conformal and disformal map originating from Horndeski theory and
ultimately belong to a pure DHOST theory. We will trace the reason for this to
the recent interesting work discussing singularities in scalar tensor theories
[24].

We would like to note that although the kinetic term of the scalar field will
be assumed to be only depending on the radial coordinate, this does not
exclude the fact that the scalar field can depend linearly, for example, on
the time coordinate, i.e. $\phi(t,r)=\alpha t+\psi(r)$ where $\alpha$ is a
constant. This possibility is attributed to the higher order nature of DHOST
theory, and the shift invariance symmetry of the scalar field. The scalar time
dependence was first used in [25] and has been found recently to be related to
the geodesics of spacetime [19] whenever the kinetic term $X$ is constant. In
fact, in the case of higher order scalar tensor theories, examples of compact
objects with a linear time dependent scalar field have been found, see e.g.
[25, 26, 27, 28]. In particular, stationary solutions, which are distinctively
different from the Kerr spacetime [29], have been recently constructed.

In our search for regular black holes we will focus on a static scalar field
where $X$ will not be a constant function. This is a crucial requirement as
$X$ will also play the role of the regularizing function smoothing out the
geometry near the origin. Once we obtain our regular solution we will discuss
its most important properties. We will then proceed to study its possible
observational characteristics scanning from weaker to stronger gravity
effects.

The plan of the paper is organized as follows. In the next section, we will
explicitly write the field equations associated to the variation of the DHOST
action (1.1-1). The key steps of the Kerr-Schild solution generating method
[23] will also be outlined, in order to explicitly construct a family of
regular asymptotically flat black holes, that are solutions of some specific
DHOST action (1.1-1) with coupling functions specified in the Appendix. We
will analyze the solutions and discuss the leading Post-Newtonian parameters,
precession effects and null geodesics, scanning through observable signatures.
In Sec. $3$, the thermodynamic analysis of these regular solutions will be
carried out through the Euclidean method, and we will show that the regularity
condition of the solutions is incompatible with the area law of the entropy.
In spite of this, the first law of thermodynamics is shown to hold for the
regular solutions. Our conclusions will be presented in Sec. $4$.

##  2 Field equations and construction of regular black holes

We will be dealing with a four-dimensional scalar tensor theory described by
the metric $g$ and a single scalar field $\phi$ whose dynamics is governed by
the action (1.1) and whose coupling functions $A_{4}$ and $A_{5}$ are given by
(1). We will focus on static metrics with a scalar field such that its
standard kinetic term $X=g^{\mu\nu}\phi_{\mu}\phi_{\nu}$ only depends on the
radial coordinate $r$, i.e.

$\displaystyle
ds^{2}=-h(r)\,dt^{2}+\frac{dr^{2}}{f(r)}+r^{2}\left(d\theta^{2}+\sin(\theta)^{2}d\varphi^{2}\right),\qquad
X=g^{\mu\nu}\phi_{\mu}\phi_{\nu}:=X(r).$ (2.3)

For this ansatz, the field equations associated with the DHOST action (1.1-1)
are conveniently written as

$\displaystyle
X[2(A_{1}G)_{X}+GA_{3}]+r^{2}\left[(K{\mathcal{H}})_{X}+\frac{3}{4}K{\mathcal{B}}\right]=0,$
(2.4a)
$\displaystyle-3({\mathcal{B}}rX^{\prime})^{2}+8({\mathcal{B}}rX^{\prime}){\mathcal{H}}\left(\frac{rh^{\prime}}{h}+4\right)-32{\mathcal{H}}\left[\frac{Kr^{2}+2G}{f}+2{\mathcal{H}}\left(\frac{rh^{\prime}}{h}+1\right)\right]=0,$
(2.4b) $\displaystyle
r^{2}(16{\mathcal{B}}_{X}{\mathcal{H}}+3{\mathcal{B}}^{2})X^{\prime
2}+8{\mathcal{H}}X^{\prime}r\left({\mathcal{B}}r\frac{f^{\prime}}{f}-16{\mathcal{H}}_{X}\right)+16r^{2}{\mathcal{H}}{\mathcal{B}}X^{\prime\prime}$
$\displaystyle-64{\mathcal{H}}^{2}\left[\left(\frac{rf^{\prime}}{f}+1\right)+\frac{2G+r^{2}K}{2f{\mathcal{H}}}\right]=0,$
(2.4c)

where $(\;)^{\prime}$ denotes the derivative with respect to the radial
coordinate, $r$, while subscript $X$ denotes the derivation with respect to
the kinetic term $X$. To simplify the notation, we have defined the auxiliary
functions of the action,

$\displaystyle{\mathcal{H}}(X)=A_{1}(X)\,X-G(X),\qquad{\mathcal{B}}(X)=A_{3}(X)\,X+4G_{X}(X)-2A_{1}(X),$
$\displaystyle{\mathcal{Z}}(X)=A_{3}(X)+A_{4}(X)+X\,A_{5}(X).$ (2.5)

Another interesting note is the Horndeski limit [13] and the beyond Horndeski
limit [30, 31] of our general DHOST theory equations. Indeed, (quartic)
Horndeski theory, parameterized by $G_{4}=G$ is attained with
$2G_{X}=A_{1}=-A_{2}$ and $A_{3}=0$, while quartic beyond Horndeski is given
by $2G_{X}-XF=A_{1}=-A_{2}$ and $A_{3}=-2F$. The function $F$ is the quartic
beyond Horndeski term which is in a one to one correspondence with the
disformal transformation, mapping Horndeski to beyond Horndeski theory (see
for example the nice analysis in [15]). In particular, we note that in both
cases of quadratic Horndeski and beyond Horndeski we have ${\mathcal{B}}=0$,
which means that ${\mathcal{B}}$ in our field equations represents the
conformal transformation mapping beyond Horndeski to pure DHOST theory. We
will come back to this observation in a moment.

In order to be self-contained, we will briefly recall the procedure described
in [23] which allows the construction of regular black hole solutions from
simple seed configurations. The first step is to look for a simple seed
solution of the field equations (which does not describe a black hole) and
schematically represent it by

$\displaystyle
ds_{0}^{2}=-h_{0}(r)dt^{2}+\frac{dr^{2}}{f_{0}(r)}+r^{2}\left(d\theta^{2}+\sin(\theta)^{2}d\varphi^{2}\right),\qquad
X_{0}=g_{(0)}^{\mu\nu}\phi^{(0)}_{\mu}\phi^{(0)}_{\nu}:=X_{0}(r).$ (2.6)

Now, as shown in Ref. [23], the equations of motion (2.4) are invariant under
a Kerr-Schild transformation of the metric, provided that the kinetic term of
the scalar field is left invariant. More precisely, it is straightforward to
see that the equations (2.4) are invariant under the following simultaneous
transformations

$\displaystyle h_{0}(r)\to h_{0}(r)-2\mu\frac{m(r)}{r},\qquad
f_{0}(r)\to\frac{f_{0}(r)}{h_{0}(r)}\left(h_{0}(r)-2\mu\frac{m(r)}{r}\right),\quad\mbox{with}\,\,\,\,\,m(r)=e^{\frac{3}{8}\int
dX\frac{{\mathcal{B}}(X)}{{\mathcal{H}}(X)}},$ (2.7)

and $X$ remains unchanged, i. e. $X_{0}(r)=X(r)$. Here $\mu$ is a constant
that will be shown to be proportional to the mass of the resulting solution.
Our second step is to use this Kerr-Schild symmetry (2.7) to deduce that the
configuration given by,

$\displaystyle
ds^{2}=-\left(h_{0}(r)-2\mu\,\frac{m(r)}{r}\right)dt^{2}+\frac{h_{0}(r)\,dr^{2}}{f_{0}(r)\left(h_{0}(r)-2\mu\,\frac{m(r)}{r}\right)}+r^{2}\left(d\theta^{2}+\sin(\theta)^{2}d\varphi^{2}\right),$
$\displaystyle X(r)=g^{\mu\nu}\phi_{\mu}\phi_{\nu}=X_{0}(r),$ (2.8)

will satisfy the same equations as those satisfied by the simple seed solution
(2.6), provided that the mass function $m(r)$ is given by

$\displaystyle m(r)=e^{\frac{3}{8}\int
dX\frac{{\mathcal{B}}(X)}{{\mathcal{H}}(X)}}.$ (2.9)

Note that in order for the mass term to be non trivial (i.e. with a non-
Newtonian fall-off) we need to venture outside of beyond Horndeski theory,
where ${\mathcal{B}}\neq 0$. According to the observation made in the previous
paragraph, ${\mathcal{B}}$ is related to the conformal degree of freedom for
pure DHOST theory. This leads us to the conclusion that we must have a
combined disformal and conformal transformation of Horndeski theory to have
any hope of constructing a regular solution. The regular solutions are
crucially situated in higher order DHOST theory-not in Horndeski or beyond
Horndeski theory.

To keep things simple we make the following working hypothesis [23]

$\frac{3{\mathcal{B}}}{8{\mathcal{H}}}=\frac{1}{X}\Longrightarrow m(r)=X(r),$
(2.10)

Hence, starting from a seed metric, the “choice” of the mass function $m(r)$,
or equivalently of the seed kinetic term (2.10) will be key in order to ensure
the regularity of the final (massive) configuration (2.8) at the origin and at
infinity. Moreover, once we fix the expression of $X_{0}(r)$ as an invertible
function, we will be able to specify the corresponding DHOST theory (1.1-1),
that is to determine the functions $K,G,A_{1}$ and $A_{3}$ (as functions of
$X$ only) [23]. For example, in the asymptotically flat case with a seed
metric $f_{0}=h_{0}=1$, the regularity at the origin will be ensured if
$m(r)=\mathcal{O}(r^{3})$. Indeed, in this case the solution is shown to
exhibit a de Sitter core at the origin, ensuring that any invariant
constructed out of the Riemann tensor will be regular at the origin. Given
these preliminary requirements we see that it is essential to be in the
context of DHOST theory, in order to find regular black holes in accordance
with the discussion and findings in [24]. Hence, regular black holes are
necessarily solutions of a pure DHOST theory. In other words, such regular
solutions would be images of the mapping of a combined conformal and disformal
transformation of a Horndeski solution.

###  2.1 Asymptotically flat regular black holes

We will first focus on the construction of asymptotically regular black holes
with a flat seed metric given by $h_{0}=f_{0}=1$. In this case, following the
results obtained in Ref. [23], one can easily express ${\mathcal{H}}$ and $G$
as

$\displaystyle{\mathcal{H}}=\frac{1}{X\left(\frac{rX^{\prime}}{3X}-1\right)},\qquad
G=\frac{1}{X}\left(1-\frac{rX^{\prime}}{X}\right)-\frac{Kr^{2}}{2}.$

Now, in order to get the coupling function $K$, we first write

$\displaystyle
A_{3}=-\frac{4G_{X}}{X}+\frac{2A_{1}}{X}+\frac{8{\mathcal{H}}}{3X^{2}},\qquad
A_{1}=\frac{{\mathcal{H}}+G}{X}$ (2.11)

and then inserting the expressions (2.11) into Eq.(2.4a), we obtain, after
some algebraic manipulations,

$2({\mathcal{H}}G)_{X}+r^{2}(K{\mathcal{H}})_{X}+\frac{2{\mathcal{H}}}{X}\left(\frac{4}{3}G+Kr^{2}\right)=0.$
(2.12)

Finally, the coupling function $K$ is shown to be given by

$\displaystyle
K=-{\frac{2\left[3X\left(rX^{\prime\prime}+2X^{\prime}\right)+r^{2}X^{-1}X^{\prime
3}-7rX^{\prime 2}\right]}{rX\left(rX^{\prime}-3X\right)^{2}}}.$

We are now ready to construct an explicit family of regular black hole
solutions. We will opt for a (seed) kinetic term,

$\displaystyle X(r)=X_{0}(r)=\frac{2}{\pi}\arctan(\frac{\pi
r^{p}}{2\sigma^{p-1}}).$ (2.13)

The function $X$ depends on the integer $p$ and the bookkeeping parameter
$\sigma$. In particular, the limiting case $\sigma\to 0$ gives us the usual
Schwarzschild case. Our choice is motivated from three essential requirements
emanating from the resulting metric function, $h(r)=1-\frac{2\mu X(r)}{r}$ :

  * •

First of all, for $r$ close to the origin we have,

$h(r)=1-2\mu\left(\frac{r}{\sigma}\right)^{p-1}+O(r^{3p-1}),$ (2.14)

and hence, as shown below for $p\geq 3$, $\sigma\neq 0$ , the final metric
will be regular at the origin. The de Sitter core is attained for $p=3$, and
increasing regularity from there on for $p>3$.

  * •

Secondly, $X$ asymptotes unity for large $r$, and as such gives for $h$ a
similar behavior at asymptotic infinity to the Schwarzschild solution. We
have,

$h(r)=1-\frac{2\mu}{r}+\frac{8\mu\sigma^{p-1}}{\pi^{2}r^{p+1}}+O(r^{3p+1}),$
(2.15)

  * •

Last but not least, the function $X(r)$ is bijective for our coordinate range
$r\in[0,\infty[$.

Using the latter property one can see that the seed configuration,
$h_{0}=f_{0}=1$, with a kinetic term given by (2.13), is a solution of the
DHOST action (1.1-1) with coupling functions reported in the Appendix.
Crucially, the action functionals are only functions of $X$, and the theory
parameters, $\sigma$ and $p$. The power, $p$, fixes the solution’s core
regularity at the origin. Once $p$ is fixed, the solution is regular without
any fine-tuning of the parameter $\sigma$, which has been inserted so as to
track down differences from GR at $\sigma\to 0$. Using therefore the
generalized Kerr-Schild transformation, one determines that the solution given
by

$\displaystyle ds^{2}=-\left(1-\frac{4\mu\arctan(\frac{\pi
r^{p}}{2\sigma^{p-1}})}{r\pi}\right)dt^{2}+\frac{dr^{2}}{\left(1-\frac{4\mu\arctan(\frac{\pi
r^{p}}{2\sigma^{p-1}})}{r\pi}\right)}+r^{2}\left(d\theta^{2}+\sin(\theta)^{2}d\varphi^{2}\right),$
$\displaystyle X(r)=\frac{2}{\pi}\arctan(\frac{\pi r^{p}}{2\sigma^{p-1}}),$
(2.16)

satisfies the field equations of the DHOST action (1.1-1) with coupling
functions given in the Appendix, which has been additionally verified by
inserting this solution directly into the equations of motion.

Figure 1: Metric function $g_{00}$ for $p=3$ and $2\sigma^{2}=\pi$. The inner
and outer horizons correspond to the roots of the function, while for smaller
masses than $\mu_{ext}$ (blue dotted curve) the solution has no horizon.

Let us now make some comments on the properties of (2.16). First of all, for
$p>0$, the metric solution will behave asymptotically $(r\to\infty)$ as the
Schwarzschild spacetime. For $\mu>0$ and $p>0$, the metric solution has an
inner and an outer event horizon as we see from the plot in Fig. 1. The outer
horizon is an event and Killing horizon (for the Killing vector
$\partial_{t}$), which is manifest by preforming the usual Eddington-
Finkestein coordinate transformation. The inner horizon is a Cauchy horizon
for any timelike hypersurface situated in the exterior spacetime where
$\partial_{t}$ is timelike. The solution has a central curvature singularity
for $0<p<3$. However, for $p=3$, the metric solution (2.16) is regular with a
de Sitter core, while for $p>3$, the family of solutions are again regular
black holes with an increasingly regular core [32]. The region internal to the
inner horizon is spacelike and completely regular at the origin. Setting $p=3$
for definiteness and $2\sigma^{2}=\pi$ we find that for $\mu_{ext}\sim 1.13$
we have an extremal black hole. For $\mu_{ext}\leq\mu$ we have a sequence of
regular black holes whereas for smaller masses than $\mu_{ext}$ we have a
regular solution without horizon; spacetime is curved but not sufficiently in
order to create an event horizon. These solutions are gravitational particle-
like solutions akin to dark matter, provided they are stable.

We now proceed to scan, starting from weak up to strong gravity, the possible
notable differences of our regular solution, as compared to standard GR. We do
not aim to be extensive here, we rather give a first approach that is useful
for future studies. Let us first seek the leading PPN parameters of this
solution in order to effectively see how it compares with GR. In order to do
this we effectively find a Cartesian distance coordinate
$\rho=\sqrt{x^{2}+y^{2}+z^{2}}$ where $(x,y,z)$ are harmonic coordinates
suited for a Newtonian gauge. As an example take $p=3$ whereupon we get,

$r=\rho+M-\frac{4\mu\sigma^{2}}{\rho^{3}}+O(1/\rho^{4}).$ (2.17)

This coordinate system is harmonic for large distances compared to the size of
the outer event horizon. Furthermore, to leading order, it agrees with the
harmonic radial coordinate of Schwarzschild (see [33] for clarification on
coordinate issues in higher PN calculations). Such distances of the order of
some $1400$ Schwarzschild radii correspond to the orbits of stars like S2
orbiting Sgr*A. Using these coordinates we can quite easily obtain the leading
(see for example [34]) PN parameters, $\beta=\gamma=1$, which end up identical
to GR for $p\geq 3$.

We can try to go a step further and evaluate directly the precession of a star
like S2 orbiting the massive compact object identified with Sgr A* (see [35]
and references within). Star S2 orbits the central, regular for our purposes,
black hole, following timelike geodesics at the equator $\theta=\pi/2$. Using
the Killing symmetries for rest energy per unit rest mass $E$ and angular
momentum per unit rest mass $L$ we have the standard relations,

$E=h(r)\frac{dt}{d\tau},\qquad L=r^{2}\frac{d\phi}{d\tau},$ (2.18)

where $\tau$ is the geodesic parameter. Transforming to $u=1/r$ coordinates
and using the above, it is straightforward to obtain the Binet’s modified
equation governing the trajectory of $S2$,

$\frac{d^{2}u}{d\phi^{2}}+u=\frac{\mu}{L^{2}}(uX_{u}+X)+3\mu u^{2}X+\mu
u^{3}X_{u},$ (2.19)

where now $u$ is a function the angular coordinate $\phi$. The above equation
gives us precisely the GR case of Schwarzschild for $X=1$. Binet’s original
equation, valid for the Newtonian limit, is obtained if we take $X=1$ and we
additionally neglect the higher order $3\mu u^{2}$ term. This orbital equation
is valid for any regular black hole we choose in the face of $X$ and for
classical precession tests of solar system planets. As an example, we can set
$p=3$ for our regular solution and Taylor expand for small $u$ (or large $r$),

$X=1-\frac{4\sigma^{2}}{\pi^{2}}u^{3}+O(u^{9}).$ (2.20)

We get the approximate equation,

$\frac{d^{2}u}{d\phi^{2}}+u=\frac{\mu}{L^{2}}+\frac{\epsilon
L^{2}}{\mu}u^{2}-\frac{16\sigma^{2}\epsilon}{3\mu\pi^{2}}u^{3}+O(u^{5}).$
(2.21)

Here we have introduced $\epsilon=\frac{3\mu^{2}}{L^{2}}$ as our small111In
our geometrized units we have $G=c^{2}=1$ and therefore $\mu(cm)=0.742\times
10^{-28}\frac{cm}{g}\mu(g)$. dimensionless parameter [36]. We are using the
same expansion parameter as for the case of Schwarzschild as we want to point
out the difference with the case of GR. Now expanding $u=u_{0}+\epsilon
u_{1}$, we obtain to zeroth order the elliptic Kepler trajectory
$u_{0}=\frac{\mu}{L^{2}}(1+e\cos\phi)$, where $e$ is the eccentricity. To
linear order in $\epsilon$, keeping only the term with growing contribution we
find at the end,

$u\sim\frac{\mu}{L^{2}}\Big{[}1+e\cos[\phi(1-\epsilon f_{SP})]\Big{]},$ (2.22)

where $f_{SP}=1-8\frac{\mu\sigma^{2}}{L^{4}\pi^{2}}(1+\frac{e^{2}}{4})$
denotes our correction beyond the GR $f_{SP}=1$ value. Constraints from
GRAVITY place $f_{SP}\sim 1.1\pm 0.2$ which in turn constrains our action
parameter $\sigma$. Note however, that given our expansion in $\epsilon$ we
are assuming that our parameter $\sigma^{2}$ is big enough so as to be of the
same order as the Schwarzschild correction. If we adapt our calculation to the
orbit characteristics of the S2 star orbit there will be fine-tuning involved.
Generically $f_{SP}=1$ since $\beta=\gamma=1$ for our background. A similar
calculation can be undertaken using null geodesics for time delay effects akin
to pulsars for example (see the review by Johannsen [37]).

A last interesting point is to consider our solution in the strong field
regime. For our generic purposes we will pursue here the light trajectories of
photons or massless particles such as neutrinos in presence of our regular
black hole. Again we follow the standard text book procedure for equatorial
geodesics but now we focus on light rays, defining $b=L/E$, the apparent
impact parameter, for an observer in the asymptotically flat region. The
parameter $b$ can vary up to the closest distance photons get to the black
hole without being necessarily eaten up by the gravitational well of the black
hole. The geodesic equation takes a familiar (particle in a potential) form,

$\frac{1}{2}\left(\frac{dr}{d\tilde{\tau}}\right)^{2}+\frac{h(r)}{2r^{2}}=\frac{1}{2b^{2}},$
(2.23)

where we have rescaled $\tilde{\tau}=L\tau$. Therefore the effective potential
takes the form,

$V_{eff}=\frac{1}{2r^{2}}\left(1-\frac{2\mu}{r}X(r)\right),$ (2.24)

and critical light rings occur at the zeroes of $V^{\prime}_{eff}=0$ which are
the zeroes of the equation,

$r+\mu X^{\prime}-3\mu X=0.$ (2.25)

The effective potential and its derivative are depicted in figures 2 and 3
respectively. Note the familiar light ring solution at $r_{R}=3\mu$ for
Schwarzschild when we set $X=1$. Once we have a zero of (2.25), $r=r_{R}$ we
get the maximal impact parameter using (2.23),

$b_{crit}=\frac{r_{R}}{\sqrt{h(r_{R})}}.$ (2.26)

The critical impact factor can be as well formulated as

$\displaystyle
b_{crit}=b_{\tiny{\mbox{Schwar.}}}\frac{\left(X(r_{R})-\frac{1}{3}X^{\prime}(r_{R})\right)^{\frac{3}{2}}}{\sqrt{X(r_{R})-X^{\prime}(r_{R})}}=b_{\tiny{\mbox{Schwar.}}}\left(\frac{r_{R}}{3\mu}\right)\sqrt{\frac{4\sigma^{4}+\pi^{2}r_{R}^{6}}{\pi^{2}r_{R}^{6}-24\mu
r_{R}\sigma^{2}+4\sigma^{4}}},$ (2.27)

where the impact factor for the Schwarzschild solution is given by
$b_{\tiny{\mbox{Schwar.}}}=3^{3/2}\mu$. It is easy to see that

$\left(\frac{r_{R}}{3\mu}\right)b_{\tiny{\mbox{Schwar.}}}\leq
b_{crit}\leq\left(\frac{r_{R}^{2}}{3\mu}\right)\sqrt{\frac{\pi}{\pi
r_{R}^{2}-6\mu}}$

and the lower bound is achieved for $\sigma=0$ (the Schwarzschild limit) and
at the limit $\sigma\to\infty$, corresponding to the flat limit.

Figure 2: Effective potential (2.24) for different values of $\sigma$ our
theory parameter. In particular, $\sigma=0$ corresponds to the effective
potential of the Schwarzschild solution for which $X=1$. Varying $\sigma>0$
changes the root of the potential and a non-zero value actually changes the
singularity to a minimum. Increasing the value of $\sigma$ further can even
remove the root corresponding to the absence of an event horizon altogether.
The height of the potential maximum marks $1/b^{2}_{crit}$ for each curve of
the potential. Figure 3: Derivative of the effective potential. One can see a
small but finite shift of its root, $r_{R}$, for different values of $\sigma$
as a decreasing function of $\sigma$.

The determination of the light ring sets the size of the black hole shadow.
The Event Horizon Telescope (EHT) has obtained the first image of the
supermassive M87 black hole. For M87 the size of the shadow was used as a test
for GR, estimating the black hole mass [38], [39] and comparing to the
independent calculation for M87’s mass given by stellar dynamics [40]. There
are a number of caveats with this calculation as a test of GR that have
primarily to do with the little knowledge of the illuminating accretion flow
for M87 or the sheer mass of the object (see in particular the critical
analysis presented in [41]). Rather than putting in the numbers we will choose
here to sketch the different cases for our regular solution as opposed to
Schwarzschild. For definiteness let us fix the mass of the black hole to
$\mu=1$ and vary the theory parameter $\sigma$ instead, in order to see how
the characteristics of the effective potential change as we sweep through our
theory. Indeed we find that for $0<\sigma<\sigma_{ext}$ our effective
potential always has a photon ring (outside of the event horizon) and as
$\sigma$ is increased we have $r_{R}^{\sigma}<3$, the GR photon ring case. At
the same time, increasing $\sigma$, the height of the potential maximum
increases and therefore the critical impact parameter
$b_{crit}^{\sigma}<b_{crit}^{0}$ is always below the Schwarzschild one (again
see [41]). Note also that once $\sigma>0$ we always have a minimum of the
potential. This scheme continues until we arrive at $\sigma_{ext}$, the case
where (for unit mass) we have an extremal black hole. Beyond this point there
is no event horizon anymore, for $\mu=1$, and our theories present now two
visible critical points, one stable and one unstable. For a region of impact
parameters in between the critical values of the potential, we have bound
light orbits for local light sources at $r<3$ or so. This is a distinctive
feature of the particle-like solutions and is something that differentiates
them from the regular black hole case. Furthermore, note that photons starting
out from infinity can probe into the gravitational solution to all distances.
Therefore, for $\sigma>\sigma_{ext}$ there is no longer a central shadow, but
rather enhanced light rings very close to the $r=0$ center. In summary, for
each given theory (where $p$ and $\sigma$ are fixed) we will have particle-
like solutions for $\mu<\mu_{ext}$ and regular black holes for
$\mu>\mu_{ext}$.

##  3 Thermodynamics of asymptotically flat regular black holes with a scalar
field source

We now turn to the study of the thermodynamic properties of the regular class
of black hole solutions (2.16). The thermodynamics of regular solutions is one
of the aspects that is widely studied in the literature, see e.g. [42]. We
start by pointing out a difference of our DHOST solution in comparison to
regular black holes with non-linear electrodynamics. In the latter case the
regularization parameter is actually part of the theory, and is usually
associated with a magnetic charge. This means that the latter solution exists
for a fixed value of the magnetic charge, and that to change this value
corresponds to changing the theory. A direct consequence of this is that the
regularization parameter cannot be considered as a variable parameter, and
hence must not appear in the equation of the first law of thermodynamics. This
aspect obscures the thermodynamic interpretation of regular solutions. On the
contrary in our case, the regularity of the solution (2.16) is not inherent to
the presence of our action bookkeeping parameter $\sigma$, but rather in the
presence of the regularizing arctangent function rendering the metric function
smooth at the origin. In addition, as it can be seen in Eq. (2.16), the
regularizing function comes with a constant $\mu$ which is an integration
constant, and hence its interpretation as a thermodynamical variable is not
ambiguous.

The thermodynamic analysis of the regular solution (2.16) will be carried out
with the Euclidean approach in which the partition function is identified with
the Euclidean path integral in the saddle point around the classical solution.
In practice, we consider a mini superspace with the following ansatz

$\displaystyle
ds^{2}=N(r)^{2}f(r)d\tau^{2}+\frac{dr^{2}}{f(r)}+r^{2}d\Sigma_{2}^{2},\qquad\phi=\phi(r),$
(3.28)

where $\tau$ (in this section) is the Euclidean (periodic) time with
$0<\tau\leq\beta$ and, where $\beta$ is the inverse of the temperature

$\displaystyle\beta^{-1}=T=\frac{1}{4\pi}\,N(r)f^{\prime}(r)|_{r_{h}},$ (3.29)

with $r_{h}$ being the radius of the horizon. In the mini superspace defined
by the ansatz (3.28), the Euclidean action $I_{E}$ (using the proper
normalization factor) reads

$\displaystyle I_{E}=-\frac{1}{4}\beta\int N\left[\left({\cal P}-2{\cal
Q}^{\prime}\right)f-{\cal Q}\,f^{\prime}+2G+r^{2}K\right]+B_{E},$ (3.30)

where ${\cal H},{\cal B}$ and ${\cal Z}$ are given in (2.5), and where for
simplicity we have defined,

$\displaystyle{\cal Q}=\frac{\cal B}{4}r^{2}X^{\prime}-2r{\cal H},\qquad{\cal
P}=rX^{\prime}{\cal B}+\frac{r^{2}}{4}(X^{\prime})^{2}{\cal Z}-2{\cal H}.$
(3.31)

In the Euclidean action (3.30), the term $B_{E}$ is an appropriate boundary
term ensuring that the solution corresponds to an extremum of the action, and
at the same time it codifies all the thermodynamic properties. After some
algebraic manipulations we get,

$\displaystyle B_{E}=\frac{\beta}{4}\lim_{r\to\infty}\left\\{\frac{N(r){\cal
Q}(r)X(r)}{r}\right\\}\,\mu-\pi\int{\cal Q}(r_{h})dr_{h}.$ (3.32)

On the other hand, since the Euclidean action is related to the Gibbs free
energy ${\cal G}$ through

$I_{E}=\beta\,{\cal G}=\beta\,{\cal M}-{\cal S},$

one can easily read off the expressions of the mass ${\cal M}$ and of the
entropy ${\cal S}$ from the boundary term,

$\displaystyle{\cal M}=\frac{1}{4}\lim_{r\to\infty}\left\\{\frac{N(r){\cal
Q}(r)X(r)}{r}\right\\}\,\mu,\qquad{\cal S}=\pi\int{\cal Q}(r_{h})dr_{h}.$
(3.33)

For the specific regular black hole solution (2.16), these expressions reduce
to

$\displaystyle{\cal M}=\frac{1}{6}\frac{r_{h}}{\arctan\left(\frac{1}{2}\pi
r_{h}^{p}\sigma^{1-p}\right)},\qquad{\cal S}=\frac{2}{3}\int\frac{\pi
r_{h}}{\arctan\left(\frac{1}{2}\pi r_{h}^{p}\sigma^{1-p}\right)}dr_{h},$
(3.34)

while the temperature is given by

$\displaystyle T=\frac{1}{4\pi
r_{h}}\left(1-\frac{2\pi\sigma^{p-1}pr_{h}^{p}}{\left(\pi^{2}r_{h}^{2p}+4\sigma^{2p-2}\right)\arctan\left(\frac{1}{2}\pi
r_{h}^{p}\sigma^{1-p}\right)}\right).$

It is clear from these relations that the mass and the entropy of the regular
solution are positive, and although we do not have a closed form of the
entropy we can nonetheless verify the validity of the first law $d{\cal
M}=T\,d{\cal S}$. We also note that the entropy of the regular solution does
not satisfy the area law. In fact, from the generic expression as obtained in
(3.33), the only way for the entropy to satisfy the area law is that the
function ${\cal Q}$, as defined in (3.31), must be proportional to ${\cal
Q}(r)\propto r$. However, it is a simple matter to check that the solutions of
the field equations given by (2.4), and for an ansatz of the form (2.8) will
necessarily imply that

${\cal Q}(r)\propto\frac{r}{X(r)},$

and, consequently the entropy will be proportional to one-quarter of the area
only for a constant kinetic term. On the other hand, our analysis shows that a
constant kinetic term is incompatible with the regularity of the solution.
Hence, we deduce that for the DHOST theories considered here the regularity of
the solutions fitting our ansatz (2.8) will not be compatible with the one-
quarter area law for the entropy. This is not uncommon for modified gravity
theories and is understood geometrically in certain cases such as Einstein-
Gauss-Bonnet theory (see for example [43]).

Thermodynamic stability of the regular solution is addressed by computing the
heat capacity $C_{H}=T\frac{\partial{\cal S}}{\partial T}$. From this
definition it becomes clear that the heat capacity will provide information
about the thermal stability with respect to the temperature fluctuations, and
that a positive heat capacity is a necessary condition to ensure the local
stability of the system. Also, the critical hypersurfaces, that is those where
$C_{H}$ vanishes or diverges, will correspond to the extrema of the
temperature with respect to the entropy. For technical reasons, it is more
convenient to express the heat capacity as

$C_{H}=T\frac{\partial{\cal S}}{\partial T}=T\left(\frac{\partial{\cal
S}}{\partial r_{h}}\right)\left(\frac{\partial T}{\partial
r_{h}}\right)^{-1},$

and, for the regular black hole solution (2.16) we get

$\displaystyle C_{\mathrm{H}}=\frac{2\pi
Cr_{h}^{2}\left(r_{h}^{2p}+\frac{4}{\pi^{2}}\sigma^{2p-2}\right)\left[\left(r_{h}^{2p}+\frac{4}{\pi^{2}}\sigma^{2p-2}\right)\arctan\left(\frac{1}{2}\pi
r_{h}^{p}\sigma^{1-p}\right)-\frac{2}{\pi}\sigma^{p-1}pr_{h}^{p}\right]}{\mathcal{C}},\qquad$

with

$\displaystyle\mathcal{C}$ $\displaystyle=$ $\displaystyle
3\arctan\left(\frac{1}{2}\pi
r_{h}^{p}\sigma^{1-p}\right)\left[\frac{2}{\pi}\sigma^{p-1}p\left(\frac{4}{\pi^{2}}\sigma^{2p-2}(p-1)-(p+1)r_{h}^{2p}\right)r_{h}^{p}+\left(r_{h}^{2p}+\frac{4}{\pi^{2}}\sigma^{2p-2}\right)^{2}\arctan\left(\frac{1}{2}\pi
r_{h}^{p}\sigma^{1-p}\right)\right]$
$\displaystyle-\frac{12}{\pi^{2}}\sigma^{2p-2}p^{2}r_{h}^{2p}.$

Due to its lengthy form it is insightful to plot the heat capacities. The heat
capacities are shown in figure 4, where we have excluded the part that
corresponds to negative temperatures (akin to the presence of an internal
horizon). From this picture, one can see that only small black holes are
locally stable and a critical hypersurface will emerge at some positive radius
revealing the existence of a second order phase transition, as it is the case
for the non-linear electrodynamical regular black holes, see e. g. [42].

Figure 4: Heat capacity of the (4.38) black hole for different values of $p$
and $\sigma$ such that $2\sigma^{p-1}=\pi$ starting at $r_{\mathrm{Extremal}}$
respectively. Note that these correspond to different theories. There is a
second order phase transition at $r_{\mathrm{PT}}$. The asymptotic behavior is
like $\propto-r^{2}$ at infinity. Setting $p=0$ corresponds to the
Schwarzschild solution, which has no phase transition.

Before closing this section, we would like to address the following question:
for the DHOST theory as defined in Appendix, does there exist another
solution, and if so, would this allow for a thermodynamic stability comparison
of the two solutions? In order to answer this question, we notice that the
first equation (2.4a) gives,

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\frac{16\left[\frac{2}{\pi}\sigma^{p-1}\sin\left(\frac{\pi}{2}X\right)\right]^{-\frac{2}{p}}}{3\pi^{2}\left[\frac{2}{\pi}p\sin\left(\frac{\pi}{2}X\right)-6X\right]^{4}X}\left[-r^{2}\cos\left(\frac{\pi}{2}X\right)^{\frac{2}{p}}+\left(\frac{2}{\pi}\sigma^{p-1}\sin\left(\frac{\pi}{2}X\right)\right)^{\frac{2}{p}}\right]F[X],$
(3.35)

with $F[X]$ being an algebraic equation in $X$ given by

$\displaystyle F[X]$ $\displaystyle=$ $\displaystyle
72X^{2}\left[p^{2}\cos\left(2\pi X\right)-p\cos\left(\pi
X\right)-2\right]-\frac{32}{\pi^{2}}p^{2}\sin^{2}\left(\pi
X\right)\left[p\cos\left(\pi X\right)-4\right]$ $\displaystyle+$
$\displaystyle\frac{12}{\pi}pX\sin\left(\pi X\right)\left[p^{2}\cos\left(2\pi
X\right)+3p^{2}-26p\cos\left(\pi X\right)+26\right].$

From this it is easy to see that there are only two possibilities: either $X$
is given by the previous form (2.16), or $X$ is a constant solving the
constraint $F[X]=0$. On the other hand, taking the difference between
(2.4b-2.4c) yields $f(r)=h(r)$, so in the first case we end up with the
regular black hole. After some straightforward computations, we can establish
that only the DHOST theory defined in the Appendix with $p=1$ will admit two
different solutions, and one of these is a stealth Schwarzschild black hole
configuration given by

$\displaystyle h(r)=f(r)=1-\frac{\mu}{r},\qquad X=1+2n,$ (3.36)

where $n$ is an integer number. The thermodynamic quantities of this stealth
solution are given by

$\displaystyle{\cal M}=\frac{r_{h}}{3\pi},\quad{\cal
S}=\frac{2}{3}r_{h}^{2},\quad T=\frac{1}{4\pi r_{h}},\quad
C_{\mathrm{H}}=-\frac{4}{3}r_{h}^{2},$ (3.37)

and as stressed before the entropy satisfies the area law because of the
constant value of the kinetic term (3.36). The comparison of the respective
heat capacities can be seen in Figure 5. We can now compare the
$\arctan-$solution (2.16) for $p=1$ with the stealth solution (3.36). Using
the free energy, defined as ${\cal F}={\cal M}-T{\cal S}$, one can calculate
the difference of the respective solutions at equal temperatures

$\displaystyle\Delta{\cal F}$ $\displaystyle=$ $\displaystyle
F_{\mathrm{regular}}-F_{\mathrm{stealth}}=T\int{\cal F}(r_{h})dr_{h},$
$\displaystyle{\cal F}(r)$ $\displaystyle=$
$\displaystyle\frac{r\left[-4\left(r^{2}+1\right)\arctan(r)^{2}+\pi\left(r^{2}+1\right)\arctan(r)-\pi
r\right]\left[-2r^{3}\arctan(r)-r^{2}+\left(r^{2}+1\right)^{2}\arctan(r)^{2}\right]}{\arctan(r)\left[\left(r^{2}+1\right)\arctan(r)-r\right]^{3}}$

It is easy to notice that the integrand ${\cal F}(r)$, goes to $+\infty$ for
$r\rightarrow 0$ and to $-\infty$ for $r\rightarrow\infty$. Hence, one would
expect the stealth solution to be thermodynamically favored for small $r_{h}$,
and there is the possibility that this changes for sufficiently large $r_{h}$.
However, because of its lengthy integral form it is not possible to make any
exact statements about this.

Figure 5: Heat capacity of the (4.38) black hole for $p=1$ and the stealth
Schwarzschild solution. This time they correspond to the same theories, even
though their behaviour looks identical to before. Further the temperature is
positive everywhere, so there is no extremal value of $r$ and the heat
capacities can be plotted from $r=0$.

##  4 Conclusions

Making use of a generalized Kerr-Schild solution generating method, as
described in [23], we have constructed a family of regular black holes, namely
solutions without curvature singularities. They are characterized by the
presence of an arctangent regularizing function, and are regular solutions of
specific higher-order scalar tensor theories known as DHOST theories. The
solutions are asymptotically flat and are accompanied by a regular scalar
field. They are characterized by a de Sitter or, increasingly regular core,
inner and outer event horizons and particle-like regular solutions. The latter
appear depending on a certain theory strength parameter $\sigma$ (related to
the mass) and could have a distinct phenomenology as compared to black holes
due to the absence of the horizon. Indeed we examined a number of observable
consequences of our solutions ranging from weaker to stronger gravity: from
the leading post-Newtonnian Eddington parameters to leading precession effects
up to enhanced geodesic light rings. It would be interesting to go beyond our
initial calculations and check for example echoes of our particle-like
solutions as predicted in [44]. Very recent similar studies have shown such
effects in the case of Einstein-Gauss-Bonnet theories [45] and it would be
interesting to apply known methods for our analytic explicit solutions.

Our regular black hole solutions differ from existing models of regular
solutions in several ways. First of all, it is important to stress that the
DHOST models for which regular black holes exist are not finetuned by some
regularizing parameter, which is usually the case for regular black holes.
Regularity of the solution is achieved directly by the form of the kinetic
$X(r)$ function. As a direct consequence the regular solutions (once
regularity of the core is fixed) only depend on a unique integration constant,
mass and a bookkeeping parameter $\sigma$ which measures the magnitude of the
higher order effects (the limiting case $\sigma\to 0$ gives GR). This is a
major difference with respect to the regular black holes of non-linear
electrodynamic models, since in those cases the mass, as well as the
regularizing parameter (usually associated to a magnetic charge), are part of
the non-linear electrodynamic Lagrangian. In the present case, the regular
solutions only depend on a unique integration constant, which is shown to be
proportional to the mass. We also note that the ”usual” area law for the
entropy is not compatible with the regularity of our solution (2.8-2.9-2.10)
and this is due to the theory’s modified nature of gravity. This is quite
common and understood in certain cases due to the higher order nature of the
theory (see for example [43]). In spite of the violation of the area law, we
have shown that the first law of thermodynamics is always satisfied. The
regular black hole solutions have a mass fall-off of the form
$\frac{\arctan(r^{p})}{r}$, where $p>0$ is a parameter of the theory. Note
that examples of black hole solutions with such regular terms at the origin
have been encountered [46] as AdS solitons. We have seen that the small
regular black holes are thermodynamically stable since their heat capacity
turns out to be positive and for the range of values of the parameter ensuring
the regularity solution, we have observed the existence of second order phase
transitions for all our regular black holes.

It would be interesting to question if regularity of such solutions in DHOST
theories persists once these are rotating. Given the recent progress in this
direction [29] there may be hope in such a direction, even analytically.
Furthermore, it would be an interesting first step to extend regular solutions
to the presence of a time dependent scalar field in order to understand how
the picture of geodesics is altered with regularity. These are some of the
possible directions in this exciting field that we hope to pursue in the near
future.

## Acknowledgements

We would like to thank Tim Anson, Eloy Ayón-Beato, Eugeny Babichev, Thanasis
Bakopoulos, Alessandro Fabbri, Panagiota Kanti, Antoine Lehébel and Georgios
Pappas for many enlightening discussions. The authors also gratefully
acknowledge the kind support of the PROGRAMA DE COOPERACIÓN CIENTÍFICA ECOSud-
CONICYT 180011/C18U04. OB is funded by the PhD scholarship of the University
of Talca. The work of MH has been partially supported by FONDECYT grant
1210889. The work of MSJ is funded by the National Agency for Research and
Development (ANID) / Scholarship Program/ DOCTORADO BECA NACIONAL/ 2019 -
21192009

## Appendix: DHOST models for the regular solution (2.16)

Along the lines of [23], one can show that the DHOST action defined by

$\displaystyle{\cal H}(X)=-\frac{2}{3\pi X-p\sin(\pi X)},$ $\displaystyle
G(X)=\frac{p^{2}\sin(2\pi X)-8p\sin(\pi X)+6\pi X}{\left(p\sin(\pi X)-3\pi
X\right)^{2}},$ $\displaystyle A_{1}(X)=\frac{2p\sin(\pi X)(p\cos(\pi
X)-3)}{X(p\sin(\pi X)-3\pi X)^{2}},$ $\displaystyle
K(X)=\frac{p\sin(\frac{\pi}{2}X)^{\frac{p-2}{p}}\cos(\frac{\pi}{2}X)^{\frac{p+2}{p}}\left(B^{2}p^{2}\cos(2\pi
X)-B^{2}p^{2}-24pX^{2}\cos(\pi X)+28BpX\sin(\pi
X)-24X^{2}\right)}{3X^{2}A^{\frac{2}{p}}(p\sin(\pi X)-3\pi X)^{2}},$

and

$A_{3}(X)=\frac{B\left(2p^{2}\left(5B^{2}+144X^{2}\right)\cos(2\pi
X)+3p\left(B^{2}p^{2}-192X^{2}\right)\cos(\pi X)-3B^{2}p^{3}\cos(3\pi
X)-10B^{2}p^{2}+24BpX\sin(\pi X)\left(-23p\cos(\pi
X)+2p^{2}+43\right)-288X^{2}\right)}{3X^{2}\left(Bp\sin(\pi
X)-6X\right)^{3}},\\\ $

where $A=\frac{2\sigma^{p-1}}{\pi}$ and $B=\frac{2}{\pi}$ and $\sigma$ an
unspecified constant, admits the following regular black hole solution

$\displaystyle ds^{2}=-\left(1-\frac{2\mu\arctan\left(\frac{\pi
r^{p}}{2\sigma^{p-1}}\right)}{\pi
r}\right)dt^{2}+\frac{dr^{2}}{\left(1-\frac{2\mu\arctan\left(\frac{\pi
r^{p}}{2\sigma^{p-1}}\right)}{\pi r}\right)}+r^{2}(d\theta^{2}+\sin^{2}\theta
d\varphi^{2}),$ $\displaystyle X(r)=\frac{2}{\pi}\arctan\left(\frac{\pi
r^{p}}{2\sigma^{p-1}}\right).$ (4.38)

## References

  * [1] R. Penrose, Phys. Rev. Lett. 14 (1965), 57-59, doi: 10.1103/PhysRevLett.14.57
  * [2] S. W. Hawking and R. Penrose, Proc. Roy. Soc. Lond. A 314 (1970), 529-548, doi: 10.1098/rspa.1970.0021
  * [3] A. D. Sakharov, Zh. Eksp. Teor. Fiz. 49, no. 1, 345 [Sov. Phys. JETP 22, 241 (1966)]
  * [4] E. Gliner, Sov. Phys. JETP 22, 378 (1966).
  * [5] Bardeen, J., in Proceedings of GR5, Tiflis, U.S.S.R. (1968).
  * [6] E. Ayon-Beato and A. Garcia, Phys. Lett. B 493 (2000), 149-152, doi: 10.1016/S0370-2693(00)01125-4
  * [7] E. Ayon-Beato and A. Garcia, Phys. Rev. Lett. 80, 5056 (1998), doi: 10.1103/PhysRevLett.80.5056
  * [8] E. Ayon-Beato and A. Garcia, Phys. Lett. B 464 (1999), 25, doi: 10.1016/S0370-2693(99)01038-2; E. Ayon-Beato and A. Garcia, Gen. Rel. Grav. 37 (2005), 635, doi: 10.1007/s10714-005-0050-y; K. A. Bronnikov, Phys. Rev. D 63 (2001), 044005, doi: 10.1103/PhysRevD.63.044005; I. Dymnikova, Class. Quant. Grav. 21 (2004), 4417-4429, doi: 10.1088/0264-9381/21/18/009; L. Balart and E. C. Vagenas, Phys. Rev. D 90 (2014) no.12, 124045, doi: 10.1103/PhysRevD.90.124045
  * [9] S. Ansoldi, Spherical black holes with regular center: A Review of existing models including a recent realization with Gaussian sources, [arXiv:0802.0330 [gr-qc]]
  * [10] M. Sharif and W. Javed, J. Korean Phys. Soc. 57 (2010), 217-222, doi: 10.3938/jkps.57.217
  * [11] M. S. Ma and R. Zhao, Class. Quant. Grav. 31 (2014), 245014, doi: 10.1088/0264-9381/31/24/245014
  * [12] P. A. Cano and Á. Murcia, Class. Quant. Grav. 38, no.7, 075014 (2021) doi:10.1088/1361-6382/abd923 [arXiv:2006.15149 [hep-th]]. P. A. Cano and Á. Murcia, JHEP 10, 125 (2020) doi:10.1007/JHEP10(2020)125 [arXiv:2007.04331 [hep-th]].
  * [13] G. W. Horndeski, Int. J. Theor. Phys. 10, 363 (1974), doi: 10.1007/BF01807638
  * [14] D. Langlois and K. Noui, JCAP 1602 (2016) 034, doi: 10.1088/1475-7516/2016/02/034; D. Langlois and K. Noui, JCAP 1607, 016 (2016), doi: 10.1088/1475-7516/2016/07/016
  * [15] M. Crisostomi, M. Hull, K. Koyama and G. Tasinato, JCAP 03, 038 (2016), doi: 10.1088/1475-7516/2016/03/038; M. Crisostomi, K. Koyama and G. Tasinato, JCAP 1604, 044 (2016), doi: 10.1088/1475-7516/2016/04/044
  * [16] H. Motohashi, K. Noui, T. Suyama, M. Yamaguchi and D. Langlois, JCAP 07, 033 (2016), doi: 10.1088/1475-7516/2016/07/033; J. Ben Achour, M. Crisostomi, K. Koyama, D. Langlois, K. Noui and G. Tasinato, JHEP 1612 (2016) 100, doi: 10.1007/JHEP12(2016)100
  * [17] E. Babichev, C. Charmousis and A. Lehébel, JCAP 1704 (2017) 027, doi: 10.1088/1475-7516/2017/04/027; E. Babichev and A. Lehébel, JCAP 12 (2018), 027, doi: 10.1088/1475-7516/2018/12/027; J. Chagoya and G. Tasinato, JCAP 1808, 006 (2018), doi: 10.1088/1475-7516/2018/08/006; T. Kobayashi and T. Hiramatsu, Phys. Rev. D 97 (2018) no.10, 104012, doi: 10.1103/PhysRevD.97.104012; M. Minamitsuji and J. Edholm, Phys. Rev. D 101, no. 4, 044034 (2020), doi: 10.1103/PhysRevD.101.044034
  * [18] E. Babichev, C. Charmousis, G. Esposito-Farèse and A. Lehébel, Phys. Rev. Lett. 120, no. 24, 241101 (2018), doi: 10.1103/PhysRevLett.120.241101; E. Babichev, C. Charmousis, G. Esposito-Farèse and A. Lehébel, Phys. Rev. D 98 (2018) no.10, 104050, doi: 10.1103/PhysRevD.98.104050
  * [19] C. Charmousis, M. Crisostomi, R. Gregory and N. Stergioulas, Phys. Rev. D 100, no.8, 084020 (2019), doi: 10.1103/PhysRevD.100.084020;
  * [20] E. Babichev and G. Esposito-Farese, Phys. Rev. D 95 (2017) no.2, 024020 doi:10.1103/PhysRevD.95.024020 [arXiv:1609.09798 [gr-qc]].
  * [21] T. Kobayashi, Rept. Prog. Phys. 82 (2019) no.8, 086901, doi: 10.1088/1361-6633/ab2429
  * [22] S. A. Hayward, Phys. Rev. Lett. 96, 031103 (2006), doi: 10.1103/PhysRevLett.96.031103
  * [23] E. Babichev, C. Charmousis, A. Cisterna and M. Hassaine, JCAP 06, 049 (2020) , doi: 10.1088/1475-7516/2020/06/049
  * [24] G. Domènech, A. Naruko, M. Sasaki and C. Wetterich, Int. J. Mod. Phys. D 29, no.03, 2050026 (2020), doi: 10.1142/S0218271820500261
  * [25] E. Babichev and C. Charmousis, JHEP 1408, 106 (2014), doi: 10.1007/JHEP08(2014)106
  * [26] T. Kobayashi and N. Tanahashi, PTEP 2014, 073E02 (2014), doi:10.1093/ptep/ptu096
  * [27] M. Bravo-Gaete and M. Hassaine, Phys. Rev. D 89, 104028 (2014) doi: 10.1103/PhysRevD.89.104028 [arXiv:1312.7736 [hep-th]]. A. Cisterna, T. Delsate and M. Rinaldi, Phys. Rev. D 92 (2015) no.4, 044050 doi: 10.1103/PhysRevD.92.044050 [arXiv:1504.05189 [gr-qc]]. O. Baake, M. Bravo-Gaete and M. Hassaine, Phys. Rev. D 102 (2020) no.2, 024088 doi: 10.1103/PhysRevD.102.024088 A. Anabalon, A. Cisterna and J. Oliva, Phys. Rev. D 89, 084050 (2014) doi: 10.1103/PhysRevD.89.084050 [arXiv:1312.3597 [gr-qc]]. C. Charmousis and D. Iosifidis, J. Phys. Conf. Ser. 600 (2015), 012003 doi: 10.1088/1742-6596/600/1/012003 [arXiv:1501.05167 [gr-qc]].
  * [28] A. Lehébel, “Compact astrophysical objects in modified gravity,” [arXiv:1810.04434 [gr-qc]].
  * [29] T. Anson, E. Babichev, C. Charmousis and M. Hassaine, JHEP 01, 018 (2021), doi: 10.1007/JHEP01(2021)018; J. Ben Achour, H. Liu, H. Motohashi, S. Mukohyama and K. Noui, JCAP 11 (2020), 001, doi: 10.1088/1475-7516/2020/11/001; T. Anson, E. Babichev and C. Charmousis, [arXiv:2103.05490 [gr-qc]]
  * [30] J. Gleyzes, D. Langlois, F. Piazza and F. Vernizzi, JCAP 02, 018 (2015), doi: 10.1088/1475-7516/2015/02/018
  * [31] J. Gleyzes, D. Langlois, F. Piazza and F. Vernizzi, Phys. Rev. Lett. 114, no.21, 211101 (2015), doi: 10.1103/PhysRevLett.114.211101
  * [32] A. Simpson and M. Visser, Universe 6 (2019) no.1, 8, doi: 10.3390/universe6010008
  * [33] A. Tucker and C. M. Will, Class. Quant. Grav. 36 (2019) no.11, 115001 doi: 10.1088/1361-6382/ab1c53 [arXiv:1809.05163 [gr-qc]].
  * [34] C. M. Will, Living Rev. Rel. 17 (2014), 4, doi: 10.12942/lrr-2014-4
  * [35] R. Abuter et al. [GRAVITY], Astron. Astrophys. 636, L5 (2020), doi: 10.1051/0004-6361/202037813
  * [36] Ray d’Inverno, Introducing Einstein’s Relativity, Clarendon Press, Oxford 1992.
  * [37] T. Johannsen, Class. Quant. Grav. 33, no.11, 113001 (2016), doi: 10.1088/0264-9381/33/11/113001
  * [38] K. Akiyama et al. [Event Horizon Telescope], Astrophys. J. Lett. 875 (2019) no.1, L6, doi: 10.3847/2041-8213/ab1141
  * [39] D. Psaltis et al. [Event Horizon Telescope], Phys. Rev. Lett. 125 (2020) no.14, 141104, doi: 10.1103/PhysRevLett.125.141104
  * [40] K. Gebhardt, J. Adams, D. Richstone, T. R. Lauer, S. M. Faber, K. Gultekin, J. Murphy and S. Tremaine, Astrophys. J. 729 (2011), 119, doi: 10.1088/0004-637X/729/2/119
  * [41] K. Glampedakis and G. Pappas, [arXiv:2102.13573 [gr-qc]].
  * [42] Y. S. Myung, Y. W. Kim and Y. J. Park, Phys. Lett. B 659 (2008), 832-838, doi: 10.1016/j.physletb.2007.12.007; Y. S. Myung, Y. W. Kim and Y. J. Park, Gen. Rel. Grav. 41 (2009), 1051-1067, doi: 10.1007/s10714-008-0690-9; L. Balart, Phys. Lett. B 687 (2010), 280-285, doi: 10.1016/j.physletb.2010.03.056; R. Tharanath, J. Suresh and V. C. Kuriakose, Gen. Rel. Grav. 47 (2015) no.4, 46, doi: 10.1007/s10714-015-1884-6; S. H. Hendi, S. N. Sajadi and M. Khademi, doi: 10.1103/PhysRevD.103.064016, [arXiv:2006.11575 [gr-qc]]
  * [43] C. Charmousis, Lect. Notes Phys. 769 (2009), 299-346, doi: 10.1007/978-3-540-88460-6_8 [arXiv:0805.0568 [gr-qc]]
  * [44] V. Cardoso, E. Franzin and P. Pani, Phys. Rev. Lett. 116 (2016) no.17, 171101 [erratum: Phys. Rev. Lett. 117 (2016) no.8, 089902], doi: 10.1103/PhysRevLett.116.171101; V. Cardoso, S. Hopper, C. F. B. Macedo, C. Palenzuela and P. Pani, Phys. Rev. D 94 (2016) no.8, 084031, doi: 10.1103/PhysRevD.94.084031; V. Cardoso and P. Pani, Nature Astron. 1 (2017) no.9, 586-591, doi: 10.1038/s41550-017-0225-y
  * [45] B. Kleihaus, J. Kunz and P. Kanti, Phys. Rev. D 102 (2020) no.2, 024070, doi: 10.1103/PhysRevD.102.024070
  * [46] E. Babichev, C. Charmousis and M. Hassaine, JHEP 05 (2017), 114 doi: 10.1007/JHEP05(2017)114 [arXiv:1703.07676 [gr-qc]].


[name=authors]

11institutetext: 1Yonsei University, 2Adobe

# Error Compensation Framework for Flow-Guided Video Inpainting

Jaeyeon Kang 11 Seoung Wug Oh 22 Seon Joo Kim 11

###### Abstract

The key to video inpainting is to use correlation information from as many
reference frames as possible. Existing flow-based propagation methods split
the video synthesis process into multiple steps: flow completion
$\rightarrow{}$ pixel propagation $\rightarrow{}$ synthesis. However, there is
a significant drawback that the errors in each step continue to accumulate and
amplify in the next step. To this end, we propose an Error Compensation
Framework for Flow-guided Video Inpainting (ECFVI), which takes advantage of
the flow-based method and offsets its weaknesses. We address the weakness with
the newly designed flow completion module and the error compensation network
that exploits the error guidance map. Our approach greatly improves the
temporal consistency and the visual quality of the completed videos.
Experimental results show the superior performance of our proposed method with
the speed up of $\times{6}$, compared to the state-of-the-art methods. In
addition, we present a new benchmark dataset for evaluation by supplementing
the weaknesses of existing test datasets.

###### Keywords:

Video Inpainting, Object Removal, Video Restoration

##  1 Introduction

Video inpainting is the task of filling missing regions in frames with
realistic content. Its applications cover object removal,
watermark/logo/subtitle removal, and even corrupted video restoration. It is
challenging as the hole regions should be filled with synthesized contents
that are unnoticeable with the surrounding background and temporally
consistent.

Existing video inpainting methods usually extract useful information from
reference frames and merge them to fill the target frame. The key is to use as
many frames as possible since most correlated pixels are somewhere in other
frames. The common pipeline for video inpainting can be categorized into two
groups: 1) direct synthesis, 2) flow-based propagation.

In general, direct synthesis methods adopt convolution-based [3, 4, 7, 23] and
attention-based [22, 13] networks. They usually take corrupted frames and
corresponding pixel-wise masks as input and directly output completed frames.
However, due to the computational complexity, these methods have relatively
small temporal windows resulting in only a few reference frames to be used (up
to 10 frames). Serious temporal inconsistencies can occur if some key frames
that contain unique pixel values and textures needed to fill the hole are not
selected. Fig. 1(a) illustrates failure cases of [22, 13] when an occluded
object is not properly included in the small reference frame set.

Flow-based methods [20, 6] split the synthesis process as into three steps:
flow completion, pixel propagation, and synthesis. During the flow completion,
the corrupted optical flow maps are processed by inpainting the holes within
the flow maps. In the pixel propagation step, pixel values from reference
frames are propagated to the holes with the guidance of the inpainted flows.
Finally, in the synthesis step, only the remaining holes are synthesized using
an image inpainting method. Flow-based approaches show better long-term
temporal consistency compared to direct synthesis methods as they explicitly
propagate pixel values from other frames using flows.

Corrupted frame |  STTN[22] |  FuseFormer[13] |  ECFVI(Ours)
---|---|---|---

(a) Small temporal window issue in direct synthesis methods

Corrupted frame |  DFC-Net[20] |  FGVC[6] |  ECFVI(Ours)
---|---|---|---

(b) Pixel misalignment and brightness inconsistency issues in flow-based
methods

Figure 1: Qualitative comparison with other video inpainting methods on object
removal scenarios. (a) The rear camel is originally shown in all frames, but
it disappears in the result of [22, 23] due to the small temporal window. (b)
Wrong pixels are propagated, and the brightness inconsistency issue occurred
in [20, 6].

Nevertheless, there are some drawbacks in flow-based methods. (1) Flow
completion. In flow inpainting, it is crucial to infer spatio-temporal
relationships from other corrupted flows. However, even adjacent flows can
have different values and directions due to inaccurate flow estimation or non-
linear motion between frames, resulting in temporally inconsistent flows.
Also, without taking the underlying video contents, errors from the flow
estimator like From corrupted in Fig. 2 cannot be handled. (2) Pixel
propagation. Simply copy-and-pasting propagated pixels causes pixel
misalignment and brightness inconsistency. As the pixel values/textures can be
conveyed from distant frames, there are often perspective mismatch such as
scale and shape. In addition, the brightness inconsistency may arise due to
the changes in the lighting condition or camera exposure in the given video
sequence. Models will output visual artifacts without the proper compensation
for these errors. While existing methods attempt to address these issues with
checking flow consistency [20, 6] and poisson blending [6], visual artifacts
still remain as shown in Fig. 1(b).

To this end, we propose a simple yet effective Error Completion Framework for
Flow-guided Video Inpainting (ECFVI) that addresses both drawbacks. We follow
the three steps in flow-based methods (i.e., flow completion, pixel
propagation, synthesis) but significantly improve the first two steps by
correcting errors, preventing the error accumulation. For the flow completion,
we make the flow inpainting be aware of RGB values. Hallucinating missing flow
values based on the underlying video contents can infer temporally coherent
flows. Specifically, we first roughly complete RGB pixels on holes,
considering the local temporal relationship. Then, the locally coherent RGB
inpainting results guide the flow inpainting. Note that the local RGB
inpainting here is only used for guiding the flow, and the actual inpainting
results are obtained in later steps. For the pixel propagation step, we
introduce additional compensation network to detect and correct the
misalignment and brightness inconsistency errors from the pixel propagation.
The network utilizes the error map outside the hole (called error guidance
map) to predict errors inside the hole. Such carefully designed components can
prevent errors from accumulating at the following stages, which significantly
improves the visual quality and temporal consistency of completed videos.

For quantitative evaluation, many video inpainting methods construct their own
datasets by randomly masking regions on a video. They usually set the unmasked
video as the ground-truth and evaluate their models. However, when the
randomly generated mask covers an entire object like in Fig. 6(a), a model
will output object-removed results which are different from the unmasked
video. Evaluation values from this case are not proportional to human
perception. Therefore, we propose a new benchmark dataset by using an object
segmentation algorithm [5] so that the mask only partially covers the object.
The results on our dataset are identical to the unmasked video, which can
provide comparative analysis into video inpainting methods.

In summary, the contributions of the paper are as follows:

  * •

We propose a simple yet effective way to compensate for the limitations of the
flow-based video inpainting. Specifically, we improve the flow completion with
RGB-awareness and propose to compensate errors in pixel propagation.

  * •

Our method outperforms previous state-of-the-art video inpainting methods in
terms of PSNR/SSIM and visual quality. Compared with the state-of-the-art in
flow-based method FGVC [6], we produce better results while reducing the
computation time (×6 faster).

  * •

We provide a new benchmark dataset for quantitative evaluation of video
inpainting, which can be very useful for evaluating future work on this topic
by providing a common ground.

##  2 Related Work

###  2.1 Direct synthesis methods

From the success of deep learning, many deep-based video inpainting methods
have emerged. [17, 10, 3] proposed to use 3D encoder-decoder networks for
enhancing the efficiency and the temporal consistency. [22, 13] exploited
attention-based methods, which use transformer modules for matching similar
patches on inter-intra frames. Zou et al. [23] used the optical flow to
progressively merge target frames with reference frames to enrich feature
representations on temporal relationships. Due to the memory constraints,
above methods can only refer to a few frames. On the other hand, Oh et al.
[14] exploited a non-local pixel matching to have a global temporal window by
using the memory network. Although they refer to all frames, corruption of
information occurs because the frames are continuously referenced in an
implicit way. While Lee et al. [12] designed a network to take information
from long-distance frames with global affine matrices, it cannot handle non-
rigid complex motions.

Corrupted frame |  From original [20, 6] |  From corrupted |  Ground-truth flow
---|---|---|---

Figure 2:  This figure shows the corrupted flows with different settings. In
[20, 6], they take original frames as input to the flow estimator and remove
the values in the hole. However, if the original frames are unavailable,
errors can occur when the corrupted frames are used as input. The flow values
near the hole are different from the ground-truth flow.

###  2.2 Flow-based methods

Huang et al. [8] adopted spatial patch matching and flow-based method but
suffered from mismatching issues caused by corrupted flows. Instead, [19, 6]
first inpainted the corrupted flows and propagated pixels along their flow
trajectories. Despite the success, they still have some limitations to produce
better results: First, to estimate the corrupted flows, they entered original
frames to the flow estimator and removed the values with masks. This procedure
works well in object removal scenarios where the original frames exist. The
problem is when the original frames are not available as in video restoration
scenarios. If the flow estimator takes the corrupted frames as input, the
masks can interfere with estimating the motion of objects in the video (Fig.
2). The errors in corrupted flows will affect the flow completion stage,
resulting in incorrect flow values. Second, it is difficult to use spatio-
temporal information between the corrupted flows for flow completion as
mentioned in previous section.

Wrong estimation from the flow completion would lead to the pixel misalignment
issue. Both methods checked the flow consistency and only propagated
trustworthy pixels to deal with the issue, but the consistency is not well
preserved as the completed flow itself is incorrect. Our framework focuses on
offsetting these limitations of flow-based methods.

##  3 Method

Let $X=[x_{1},x_{2},...,x_{T}]$ be a set of corrupted video frames of length
$T$. $M=[m_{1},m_{2},...,m_{T}]$ denotes the corresponding frame-wise masks,
where the spatial resolution is the same as $X$. For each mask, value $0$
indicates valid regions, and $1$ indicates hole (missing) regions. Video
inpainting aims to predict the original or object removed video
$\hat{Y}=[\hat{y}_{1},\hat{y}_{2},...,\hat{y}_{T}]$ given $X$ and $M$ as
inputs. We call the frame to be inpainted as the target frame and other frames
as reference frames.

Figure 3: Overview of our Error Compensation Framework for Flow-guided Video
Inpainting (ECFVI). At first, we estimate all completed flows between adjacent
frames using our flow completion module. With the guidance of the completed
flows, we iteratively propagate and compensate pixels from reference frames to
fill target holes. One iteration of the process is shown in Fig. 4. The
remaining holes are further synthesized with the existing video inpainting
method.

###  3.1 Overview

We illustrate our video inpainting framework in Fig. 3. The overall procedure
can be summarized as follows. (1) Flow completion with RGB guidance:  Given
the corrupted frames, we first generate locally coherent frames. Then, we
estimate complete flows between adjacent frames using our flow estimator (Sec.
3.2). Before the propagation, we estimate bi-directional completed flows
between all frames. (2) Pixel Propagation with Error Compensation:  With the
guidance of completed flows, we propagate valid pixels from reference frames
to target holes. In this process, we prevent the errors from propagating to
the following stages using our error compensation network (Sec 3.3). In the
order close to the target frame, we iterate propagation and compensation
procedures until the hole regions are entirely removed or all frames are
referenced. (3) Synthesis:  If there are remaining holes to fill, we
synthesize using an existing video inpainting method. This case usually occurs
due to occlusion, where some pixels cannot be found in any other frames. For
the synthesis, we used FuseFormer [13] with the weights from their website.

###  3.2 Flow completion with RGB guidance

Previous methods [20, 6] first estimate corrupted flows with the flow
estimator [9, 16] and complete flow values with their methods. However,
without considering the original video contents, there is no ability to handle
the errors from the corrupted flows. Therefore, we design our flow completion
module to be aware of RGB values and directly output completed flows from the
flow estimator.

A simple approach would be to have the flow estimator take the corrupted
frames and complete the flow values. However, since the flow estimator [9, 16]
iteratively takes two frames to estimate entire flows, completed flows are
created without considering spatio-temporal information of other flows. Also,
it is more challenging to run flow estimation and completion simultaneously.
Therefore, we roughly complete RGB pixels through local neighboring frames
(larger than two frames) and pass the results to our flow estimator. This can
further exploit temporal information for estimating completed flows. Here, our
flow estimator takes locally coherent frames $\bar{x}$ from a local temporal
network $LTN$. Note that the intermediate frames $\bar{x}$ are only used for
guiding the flow completion.

We design the local temporal network $LTN$ to consist of an encoder, multiple
spatio-temporal transformer layers [22, 13] and a decoder. We first extract
features of each input from the encoder and exploit transformer layers to
merge the information from the reference in the deep encoding space. The
decoder takes the output of the transformer layers to reconstruct locally
coherent frames $\bar{x}$ as follows:

$\displaystyle\bar{x}_{i}=LTN(x_{i-N:i+N},m_{i-N:i+N}),$ (1)

where $N$ is the temporal radius and $N=5$ is used in our experiment. Then, we
estimate a completed flow between adjacent frames as follows:

$\displaystyle\tilde{f}_{t\rightarrow{t+1}}=F(\bar{x}_{t},\bar{x}_{t+1},m_{t},m_{t+1}),$
(2)

where $F$ is our flow estimator. Backward flow $\tilde{f}_{t\rightarrow{t-1}}$
can be estimated in the same manner. For classifying the regions where the
original and coarsely completed content exist, we use masks $m$ as additional
input. The flow estimator is initialized with pretrained weights from RAFT
[16] except for the first layer to take the masks as additional input. We
jointly train our local temporal network and flow estimator as follows:

$\displaystyle\mathcal{L}=\mathcal{L}_{rec}+\lambda_{flow}\mathcal{L}_{flow},$
(3)

where $\mathcal{L}_{rec}$ is the reconstruction L1 loss defined as
$||\bar{x}-{y}||_{1}$. $\lambda_{flow}$ is a coefficient for the loss terms
and we set it to $2$ in, our experiment. For the flow loss
$\mathcal{L}_{flow}$, inspired by [15], we employ hard example mining
mechanism to weigh more on the difficult areas as follows:

$\displaystyle\mathcal{L}_{flow}=||{h}_{t}\odot{}(\tilde{f}_{t\rightarrow{t+1}}-f_{t\rightarrow{t+1}})||_{1},$
(4)

where $\odot{}$ is element-wise multiplication and ground-truth
$f_{t\rightarrow{t+1}}$ is calculated from original frames $Y$. We set the
hard mining weight ${h}_{t}$ with $(1+|\bar{x}_{t}-y_{t}|)^{2}$. It encourages
the model to implicitly deal with errors from the local temporal network. More
details of our network are shown in the supplementary material.

This design choice has advantages in two aspects. First, it alleviates the
problems of corrupted flows mentioned in Sec.2.2. Second, with the strong
initialization of pretrained weights and no need to inpaint flow values from
scratch, it can be trained much easier. Although there still could be errors
from the two networks, we can further compensate it in the next stage.

Figure 4:  For simplicity, we show the propagation and compensation stages
from one reference frame $x_{j}$ to the target frame $x_{i}$. We dilate the
original hole region to get the overfilled frame $\tilde{x}^{d}_{i}$. Pixels
corresponding to the blue region are further propagated from the aligned
reference frame. The filled frame $\tilde{x}_{i}$ and overfilled frame
$\tilde{x}^{d}_{i}$ are used to estimate the error guidance map $e_{i}$. Note
that the green and blue regions correspond to propagation and error mask
respectively.

###  3.3 Pixel Propagation with Error Compensation

The next step, pixel propagation with error compensation, is illustrated in
Fig. 4. With the guidance of the completed flows, we iterate propagation and
compensation steps so that the errors do not accumulate or amplify in the next
iteration.

#### Propagation.

Let $x_{i}$ and $x_{j}$ be the target and the reference frame respectively.
With the completed flow $\tilde{f}_{i\rightarrow{j}}$, only the valid pixels
in reference frame $x_{j}$ are forwarded to target holes by using backward
warping function $w$.

$\displaystyle{m}^{p}_{i}=m_{i}\odot{}(1-w(m_{j},\tilde{f}_{i\rightarrow{j}})),$
(5)
$\displaystyle\tilde{x}_{i}=x_{i}+{m}^{p}_{i}\odot{}w(x_{j},\tilde{f}_{i\rightarrow{j}}),$

where $\tilde{x}_{i}$ is the filled target frame. To find the regions where
the matched and valid regions are in the reference frame, we warp the
reference mask $m_{j}$ and get valid regions
$1-w(m_{j},\tilde{f}_{i\rightarrow{j}})$ in the warped reference frame.
$m^{p}_{i}$ denotes a propagation mask where the propagated pixels exist,
shown by the green region in Fig. 4. This mask is used at the compensation
stage to know where to compensate for errors. After the propagation, there may
be a remaining mask $m^{r}_{i}$, where $m^{r}_{i}=m_{i}-{m}^{p}_{i}$. The
remaining mask is further used for the next propagation or input for the
synthesis network.

Target frame $x_{i}$ |  Filled frame $\tilde{x}_{i}$ |  Error guidance
---|---|---
map $e_{i}$ |  Estimated error $\tilde{e}_{i}$ |  Compensated
frame $\tilde{y}_{i}$ |  |

Figure 5:  Problems from the propagation stage and corresponding error maps.
From top to bottom: no issue, brightness inconsistency, pixel misalignment,
and combined issues. The estimated errors and compensated frames are computed
using our method.

#### Compensation.

As shown in Fig. 5, we observe that directly propagating pixels from the
reference frame may cause misalignment or brightness inconsistency issues. To
remedy these issues, one solution is taking filled frame $\tilde{x}_{i}$ and
corresponding mask $m^{p}_{i}$ as input and processing them through a GAN
framework. However, it is difficult to train such a network because it cannot
easily detect what types of problem occurred.

We approach this problem by introducing the error guidance map $e_{i}$ as
input to our network. To know what is wrong with the propagation, we need
valid (ground-truth) values. Since only regions outside the holes in the given
corrupted frame have valid values, we propagate more pixels by dilating the
original hole regions like in Fig. 4. Then, we calculate the errors on
enlarged parts, which is the error guidance map $e_{i}$. This process assumes
that the propagated pixels on the enlarged and the propagated regions have
similar error tendencies. We set the dilation factor as 17 pixels in our
experiment.

We set dilated mask as $m^{d}_{i}$ and overfilled frame as
$\tilde{x}^{d}_{i}$. The error guidance map $e_{i}$ and the corresponding
error regions (mask) $m^{e}_{i}$ are computed as follows:

$\begin{gathered}m^{e}_{i}=(m^{d}_{i}-m_{i})\odot{}(1-w(m_{j},\tilde{f}_{i\rightarrow{j}})),\\\
e_{i}=m^{e}_{i}\odot{}(\tilde{x}^{d}_{i}-\tilde{x}_{i}).\end{gathered}$ (6)

In Fig. 5, this error guidance map intuitively shows the problems of pixel
propagation. If it has no issue, it will be black (zero). On the other hand,
if it has a pixel misalignment issue, it will result in the edges being
visible in the error map. This additional information helps our network be
more aware of the problems from the propagation stage.

We design our error compensation network with a similar structure as our local
temporal network $LTN$, but with different input and output settings:

$\displaystyle\tilde{e}_{i}=ECN(\tilde{x}^{d}_{i-N:i+N},{e}_{i-N:i+N},m^{[e,p,r]}_{i-N:i+N}),$
(7)

where $ECN$ is our error compensation network and $m^{[e,p,r]}$ denotes error,
propagation, remaining mask respectively. For simplicity, we will omit the
index from now on. $\tilde{x}^{d},{e},m^{e}$ are concatenated in the channel
dimension and forwarded to our encoder, and each of the two masks
$m^{p},m^{r}$ are used at the transformer layers. Note that the overfilled
frames $\tilde{x}^{d}$ are used instead of filled frames $\tilde{x}$ as input.
It makes our network to learn the relationship between overfilled frame
$\tilde{x}^{d}$ and the error guidance maps $e$ on the error regions $m^{e}$.
Since the propagated regions have some structure information, based on the
relationship, our network can estimate error values on the regions $m^{p}$
with accuracy. A final compensated frame is
$\tilde{y}=\tilde{x}^{d}+\tilde{e}$. The results and corresponding errors are
shown in Fig. 5.

The loss to train our error compensation network consists of reconstruction
loss $\mathcal{L}_{rec}$ and adversarial loss $\mathcal{L}_{adv}$:

$\displaystyle\mathcal{L}=\mathcal{L}_{rec}+\lambda_{adv}\mathcal{L}_{adv},$
(8)

where $\lambda_{adv}$ is a coefficient for the loss terms and we set $0.01$ in
our experiment. $\mathcal{L}_{rec}$ is a L1 loss between the ground-truth
frame ${y}_{i}$ and compensated output $\tilde{y}_{i}$. For
$\mathcal{L}_{adv}$, we use the model from Temporal PatchGAN (T-PatchGAN) [3].
Further details on our network are included in the supplementary material.

##  4 Experiments

###  4.1 Training Details

Our whole networks are trained on Youtube-VOS [19] dataset. It consists of
4,453 videos, which are split into 3,471/474/508 for training, validation and
testing respectively. We randomly select $256\times{}256$ frames and free-form
masks from [22] as input. To train our compensation network, we freeze the
weights of flow completion. And for handling the brightness inconsistency
issue, we further modify the brightness, hue and saturation of reference
frames. Adam optimizer [11] is used where the learning rate starts with 1e-4
and is divided by 2 every 40,000 iterations. For each module, the total
iteration is 120,000 with mini-batch size 4, and it takes about 2 days using
two NVIDIA RTX 2080 TI GPUs.

Original frame |  Corrupted frame |  ECFVI(Ours)
---|---|---

(a) Object removal scenario

(b) Video restoration scenario

Figure 6:  Two common scenarios for video inpainting: object removal and video
restoration. The gray region in the corrupted frame denotes a mask. In the
object removal scenario, the result and the original frame are different. Only
the video restoration scenario should be used for qualitative evaluation.

###  4.2 Youtube-VI Dataset

Most previous works created their own random masks and measured the
performance on Youtube-VOS [19] and DAVIS [2] datasets. However, when a
randomly generated mask coincidentally covers an entire object, a model will
create the object-removed result like in Fig. 6(a). The result is reasonable,
but it is far from the original frame. Only the case in Fig. 6(b) should be
used for quantitative evaluation. Also, if there is no motion in the mask and
its vicinity, it is not suitable for video inpainting, which cannot trace any
pixels from other frames.

TSAM [23] publicly provides frame-wise masks for the FVI [21] dataset. It
consists of 100 scenes, each with 15 frames. For each scene, three types of
masks are provided, some of which are shown in Fig. 7. While the masks cover
diverse scenarios, 15 frames in a video are still limited compared to real-
world videos. Also, the problems mentioned above on the mask still exist.

To this end, we design the new Youtube Video Inpainting (Youtube-VI) dataset,
which will be publicly available. 100 scenes are selected in order of name
from the beginning of Youtube-VOS dataset, and each scene has 50 frames. We
generated two types of masks, which are shown in Fig. 7 and described as
follows:

(1) Moving mask. We follow the same rules for generating free-form mask in
STTN [22]. But we use a video segmentation network STCN [5] so that an object
is not fully covered. With the help of STCN, we randomly generate one moving
mask that partially overlaps with objects. Then we modify the mask again if
the scene and mask do not look visually proper.

(2) Stationary mask. This case is usually used to cover the logo/subtitle
removal scenarios. We find that using free-form masks is unsuitable since
there may be no motion near the mask. Instead, we use 5x4 small square masks
like FGVC [6].

(a) Circle |  (b) Curve |  (c) Free-form |  (d) Stationary
---|---|---|---

Figure 7:  Four different types of mask. In FVI [23] dataset, they use (a),(b)
as a moving mask and (c) as a stationary mask. In our Youtube-VI dataset, (c)
and (d) is used as a moving mask and stationary mask respectively.

###  4.3 Comparisons

We quantitatively compare with others on published FVI [23] and our new
dataset for video restoration scenarios. Like [22, 23, 13], we use DAVIS [2]
dataset for qualitative comparison in object removal scenarios. DAVIS dataset
consists of 150 videos in total, in which 90 videos are annotated with the
pixel-wise object masks. In addition, DAVIS-shadow dataset [8] where the mask
covers both an object and its shadows are used for more visual comparison.

DFC-Net [20] and FGVC [6] only take original frames as input to the flow
estimator from their published code. It is acceptable in object removal
scenarios, but it is not fair to evaluate quantitative performance where the
original frame is equal to ground-truth. Therefore, we also experiment with
corrupted frames as input, and denote them as DFC-Net* and FGVC* respectively.

|  FVI [23] | Youtube-VI |
---|---|---|---
| Stationary Mask | Circle Mask | Curve Mask | Stationary Mask | Moving Mask |
|  PSNR |  SSIM |  PSNR |  SSIM |  PSNR |  SSIM |  PSNR |  SSIM |  PSNR |  SSIM |  Time(s)
DFC-Net [20] |  29.20 |  0.9632 |  24.77 |  0.9219 |  28.03 |  0.9436 |  29.03 |  0.9609 |  32.94 |  0.9771 |  95
DFC-Net* [20] |  28.51 |  0.9609 |  24.12 |  0.9141 |  26.60 |  0.9295 |  27.73 |  0.9536 |  32.14 |  0.9747 |  95
OPN [14] |  31.59 |  0.9689 |  26.37 |  0.9264 |  29.79 |  0.9519 |  31.91 |  0.9721 |  33.56 |  0.9767 |  154
CPN [12] |  31.76 |  0.9693 |  26.90 |  0.9315 |  31.09 |  0.9664 |  31.32 |  0.9687 |  34.15 |  0.9783 |  24
STTN [22] |  32.18 |  0.9699 |  27.04 |  0.9288 |  30.01 |  0.9515 |  31.96 |  0.9716 |  34.66 |  0.9789 |  17
FGVC [6] |  30.88 |  0.9690 |  26.14 |  0.9369 |  31.50 |  0.9676 |  32.25 |  0.9750 |  35.68 |  0.9841 |  734
FGVC* [6] |  30.60 |  0.9682 |  25.55 |  0.9302 |  29.90 |  0.9565 |  31.21 |  0.9701 |  34.05 |  0.9795 |  734
TSAM [23] |  31.74 |  0.9695 |  26.66 |  0.9250 |  31.18 |  0.9578 |  - |  - |  - |  - |  -
FuseFormer [13] |  33.19 |  0.9733 |  27.85 |  0.9368 |  30.85 |  0.9577 |  33.04 |  0.9755 |  35.29 |  0.9806 |  18
ECFVI(Ours) |  33.27 |  0.9749 |  28.24 |  0.9469 |  31.15 |  0.9638 |  33.53 |  0.9795 |  36.80 |  0.9860 |  121
Table 1:  Quantitative evaluation on FVI and Youtube-VI datasets. Best values
are shown in bold and second best values are underlined. For PSNR and SSIM,
the higher is better. Missing entries indicate the method impossible to run at
those settings.

#### Quantitative Results.

We evaluate video inpainting quality on PSNR and SSIM. In Table 1, we achieve
the best performance on PSNR and SSIM except for the curve mask on FVI
dataset. It is worth noting that the curve mask is a particular case in real-
world scenarios. As we assume our error guidance map for common scenarios, the
error values in dilated regions can be overlapped, which worsens our results
in such thin and overlapped masks. On the other hand, on the moving and
stationary mask, our framework clearly outperforms by a large margin compared
to DFC-Net [20] and FGVC [6], where the flow-based propagation is used. Such
results show that our overall framework plays critical role in addressing the
limitations of existing methods. More comparisons on other metrics, Video-
based Fréchet Inception Distance (VFID) [18] and optical flow based warping
error (EWarp) [18], are shown in the supplementary material.

We measure the execution time on 864x480 resolution for inpainting 50 frames
to compare the efficiency. Since we design our entire components with deep
frameworks, ours is $\times{6}$ faster compared to FGVC [6] where the
optimization-based methods are used.

Corrupted frame |  STTN [22] |  FGVC [6] |  FuseFomer [13] |  ECFVI(Ours)
---|---|---|---|---

Figure 8:  Qualitative results compared with [22, 6, 13]. The temporal
profiles of the red scan line are shown below the results. Best viewed in
zoom. Figure 9:  A user study on DAVIS [2] object removal. The lower is
better.

#### Qualitative Results.

In Fig. 8, we show our model’s qualitative results compared with other
methods, including STTN [22], FGVC [6] and FuseFormer [13]. To visualize the
temporal consistency, we show the temporal profile [1] of the resulting videos
below the completed frames. Sharp and smooth edges in the temporal profile
indicate that the video has much less flickering. As can be seen, our method
shows more visually pleasing results compared to others.

For a more comprehensive comparison, we conducted a user study to subjectively
compare our method against others on the DAVIS dataset through Amazon
Mechanical Turk. The experiments were performed on 30 videos that excludes
easy (Tennis, Flamingo, etc.) and difficult (India, Drone, etc.) cases where
every methods failed. The videos like Fig. 8 were shown to 20 participants. We
unlabeled and shuffled the order in all videos, and asked to rank the results
from 1 to 4 (1 is the highest preference). As shown in Fig. 9, the results of
our model showed higher preference over others, supporting that our approach
can produce more visually pleasing outputs.

(a) Flow completion | Stationary Mask | Moving Mask |
---|---|---|---
| Flow-EPE | Flow-EPE | Time(s)
DFC-Net/* [19] | 0.32/0.89 | 0.38/0.86 | 43
FGVC/* [6] | 0.12/0.57 | 0.28/0.80 | 412
FF [13] \+ RAFT [16] | 0.49 | 0.80 | 52
$w/o$ $LTN$ | 0.71 | 0.83 | 24
$w/o$ Hard | 0.40 | 0.58 | 52
Ours | 0.33 | 0.49 | 52

(b) Video inpainting | Stationary Mask | Moving Mask
---|---|---
|  PSNR |  SSIM |  PSNR |  SSIM
FGVC* + Comp |  32.87 |  0.9762 |  35.59 |  0.9823
GT flow + Comp |  33.64 |  0.9796 |  37.17 |  0.9865
$w/o$ Comp |  30.05 |  0.9691 |  34.04 |  0.9810
$w/o$ Guide |  30.48 |  0.9692 |  33.81 |  0.9797
Ours |  33.53 |  0.9795 |  36.80 |  0.9860

Table 2: Ablation studies on Youtube-VI dataset. For Flow-EPE, the lower is
better. FGVC/* means FGVC and FGVC* respectively.

###  4.4 Ablation Study

#### Effectiveness of Flow Completion.

In Table 2(a), we show the flow end-point-error (EPE) metric to compare our
flow completion with others. We set the estimated flows from RAFT [16] as
pseudo ground-truth flows. In the table, FGVC/* means FGVC and FGVC*
respectively. In DFC-Net [20] and FGVC [6], the performance is significantly
reduced when the corrupted frames are used as input to the flow estimator. It
verifies that the errors in corrupted flows prevent estimating correct flows.
In the same setting where the corrupted frames are used, ours achieves the
best performance.

In addition, we conduct a series of ablation studies to analyze the
effectiveness of our flow completion module. First, we sequentially run
pretrained FuseFormer [13] followed by original RAFT [16] (FF + RAFT). To
demonstrate the effectiveness of local temporal network $LTN$, we show the
results only using the flow estimator ($w/o$ $LTN$). The flow estimator is
trained with the same setting as ours. In addition, we train our flow
estimator with equal weights in Eq. (4) instead of $h$ ($w/o$ Hard). Such
results verify that our jointly trained flow completion module can deal with
the errors from the local temporal network $LTN$ and produce more accurate
completed flows.

#### Effectiveness of Compensation.

In Table 2(b), we show the performance of the error compensation network by
replacing our flow completion with FGVC’s method (FGVC* + Comp). Although the
flow completion from FGVC* is worse than ours, it outperforms the full
procedure of FGVC*. We also show the pseudo ground-truth flow from RAFT [16]
(GT flow + Comp). These two experiments demonstrate our error compensation
network is robust to errors from the flow completion. Also, we test our model
without compensation stage ($w/o$ Comp) and without error guidance map ($w/o$
Guide). In $w/o$ Guide experiment, we use naive GAN where filled frames with
propagation masks and remaining masks are only used as input. The results show
severe performance drops compared to our method. These two experiments
demonstrate that our compensation network using the error guidance map is
necessary to handle the errors from the previous stages.

##  5 Conclusion

In this paper, we have proposed a simple yet effective video inpainting
framework, which takes advantage of the flow-based methods while compensating
for its shortcomings. We can propagate valid pixels with our flow completion
and error compensation network, showing high-quality completed videos.
Especially with the help of the error guidance map, we can prevent the errors
from accumulating and amplifying at the following stages. We show that our
method achieves state-of-the-art performance and demonstrate the benefits of
our framework through extensive experiments. We also provide a new benchmark
dataset, which provides comparative analysis into video inpainting methods.

#### Acknowledgement

This work was supported by Institute of Information & Communications
Technology Planning & Evaluation (IITP) grant funded by the Korea government
(MSIT) (No. 2014-3-00123, Development of High Performance Visual BigData
Discovery Platform for LargeScale Realtime Data Analysis), and No.
2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei
University).

## References

  * [1] Caballero, J., Ledig, C., Aitken, A., Acosta, A., Totz, J., Wang, Z., Shi, W.: Real-time video super-resolution with spatio-temporal networks and motion compensation. In: CVPR. pp. 4778–4787 (2017)
  * [2] Caelles, S., Montes, A., Maninis, K.K., Chen, Y., Van Gool, L., Perazzi, F., Pont-Tuset, J.: The 2018 davis challenge on video object segmentation. arXiv preprint arXiv:1803.00557 (2018)
  * [3] Chang, Y.L., Liu, Z.Y., Lee, K.Y., Hsu, W.: Free-form video inpainting with 3d gated convolution and temporal patchgan. In: ICCV. pp. 9066–9075 (2019)
  * [4] Chang, Y.L., Liu, Z.Y., Lee, K.Y., Hsu, W.: Learnable gated temporal shift module for deep video inpainting”. BMVC (2019)
  * [5] Cheng, H.K., Tai, Y.W., Tang, C.K.: Rethinking space-time networks with improved memory coverage for efficient video object segmentation. In: NeurIPS (2021)
  * [6] Gao, C., Saraf, A., Huang, J.B., Kopf, J.: Flow-edge guided video completion. In: ECCV. pp. 713–729. Springer (2020)
  * [7] Hu, Y.T., Wang, H., Ballas, N., Grauman, K., Schwing, A.G.: Proposal-based video completion. In: ECCV. pp. 38–54. Springer (2020)
  * [8] Huang, J.B., Kang, S.B., Ahuja, N., Kopf, J.: Temporally coherent completion of dynamic video. ACM Transactions on Graphics (TOG) 35(6), 1–11 (2016)
  * [9] Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: CVPR. pp. 2462–2470 (2017)
  * [10] Kim, D., Woo, S., Lee, J.Y., Kweon, I.S.: Deep video inpainting. In: CVPR. pp. 5792–5801 (2019)
  * [11] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [12] Lee, S., Oh, S.W., Won, D., Kim, S.J.: Copy-and-paste networks for deep video inpainting. In: ICCV. pp. 4413–4421 (2019)
  * [13] Liu, R., Deng, H., Huang, Y., Shi, X., Lu, L., Sun, W., Wang, X., Dai, J., Li, H.: Fuseformer: Fusing fine-grained information in transformers for video inpainting. In: ICCV. pp. 14040–14049 (2021)
  * [14] Oh, S.W., Lee, S., Lee, J.Y., Kim, S.J.: Onion-peel networks for deep video completion. In: ICCV. pp. 4403–4412 (2019)
  * [15] Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: CVPR. pp. 761–769 (2016)
  * [16] Teed, Z., Deng, J.: Raft: Recurrent all-pairs field transforms for optical flow. In: European conference on computer vision. pp. 402–419. Springer (2020)
  * [17] Wang, C., Huang, H., Han, X., Wang, J.: Video inpainting by jointly learning temporal structure and spatial details. In: AAAI. vol. 33, pp. 5232–5239 (2019)
  * [18] Wang, T.C., Liu, M.Y., Zhu, J.Y., Liu, G., Tao, A., Kautz, J., Catanzaro, B.: Video-to-video synthesis. arXiv preprint arXiv:1808.06601 (2018)
  * [19] Xu, N., Yang, L., Fan, Y., Yang, J., Yue, D., Liang, Y., Price, B., Cohen, S., Huang, T.: Youtube-vos: Sequence-to-sequence video object segmentation. In: ECCV. pp. 585–601 (2018)
  * [20] Xu, R., Li, X., Zhou, B., Loy, C.C.: Deep flow-guided video inpainting. In: CVPR. pp. 3723–3732 (2019)
  * [21] Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.S.: Free-form image inpainting with gated convolution. In: ICCV. pp. 4471–4480 (2019)
  * [22] Zeng, Y., Fu, J., Chao, H.: Learning joint spatial-temporal transformations for video inpainting. In: ECCV. pp. 528–543. Springer (2020)
  * [23] Zou, X., Yang, L., Liu, D., Lee, Y.J.: Progressive temporal feature alignment network for video inpainting. In: CVPR. pp. 16448–16457 (2021)