#  $(2+1)$-Dimensional Black Holes in $f(R,\phi)$ Gravity

Thanasis Karakasis<EMAIL_ADDRESS>Physics Division, National
Technical University of Athens, 15780 Zografou Campus, Athens, Greece.
Eleftherios Papantonopoulos<EMAIL_ADDRESS>Physics Division, National
Technical University of Athens, 15780 Zografou Campus, Athens, Greece.  Zi-Yu
Tang<EMAIL_ADDRESS>School of Fundamental Physics and Mathematical
Sciences, Hangzhou Institute for Advanced Study, UCAS, Hangzhou 310024, China
School of Physical Sciences, University of Chinese Academy of Sciences,
Beijing 100049, China  Bin Wang<EMAIL_ADDRESS>School of Aeronautics and
Astronautics, Shanghai Jiao Tong University, Shanghai 200240, China  Center
for Gravitation and Cosmology, College of Physical Science and Technology,
Yangzhou University, Yangzhou 225009, China

###### Abstract

We consider a $f(R)$ gravity theory in $(2+1)$-dimensions with a self-
interacting scalar field non-minimally coupled to gravity. Without specifying
the form of the $f(R)$ function, solving the field equations we find that the
Ricci scalar receives a non-linear correction term which breaks the conformal
invariance and leads to a massless black hole solution. When the non-linear
term decouples, we get a well known hairy black hole solution with the scalar
field conformally coupled to gravity. We also find that the entropy of our
black hole may be higher than the corresponding conformal black hole which
indicates that our solution may be thermodynamically preferred.

###### Contents

  1. I Introduction
  2. II Conformal $(2+1)$-dimensional black hole
  3. III $f(R)$ Gravity Black Hole Solution
  4. IV Thermodynamics
    1. IV.1 Hawking temperature
    2. IV.2 Entropy
    3. IV.3 Conserved Mass
  5. V Conclusions

##  I Introduction

In $(2+1)$-dimensions one of the first exact black hole solutions of General
Relativity (GR) is the Bañados, Teitelboim, and Zanelli (BTZ) black hole [1,
2]. It is known that in $(2+1)$-dimensions, the Weyl tensor vanishes by
definition and in the absence of matter we have $R_{\mu\nu}=0$, therefore we
cannot have a black hole solution. To evade this problem, a negative
cosmological constant was introduced in the theory allowing black hole
solutions to exist in a spacetime with negative constant curvature.
Introducing a linear Maxwell field, a black hole solution can also be obtained
with a dynamical Ricci scalar, while a conformally invariant non-linear
electromagnetic field $\left(F_{\mu\nu}F^{\mu\nu}\right)^{3/4}$ gives a black
hole solution with constant Ricci scalar [3].

After the discovery of the BTZ black hole, several $(2+1)$-dimensional black
hole solutions have been obtained coupled to a scalar field. In [4, 5] black
holes with an everywhere regular scalar field have been investigated. Then
more $(2+1)$-dimensional hairy black hole solutions were discussed [6, 7, 8,
9, 10, 11, 12, 13]. In [14] asymptotically AdS $(2+1)$-dimensional black hole
solutions with conformally coupled scalar and Abelian gauge fields were found.
In [15] and [16] $(2+1)$-dimensional charged black holes with scalar hair were
derived, where the scalar potential is not fixed ad hoc but instead derived
from Einstein’s equations. In [17] exact $(2+1)$-dimensional black holes with
a non-minimally coupled scalar field were discussed, where the arbitrary
coupling constant breaks the conformal invariance. In [18] and [19], static
black holes in $(2+1)$-dimensional dilaton gravity and modifications of the
BTZ black hole by a dilaton/scalar were investigated. Recently, regular black
hole solutions with a real scalar field coupled to the Maxwell field via a
duality transformation were constructed [20] and $(2+1)$-dimensional regular
black holes with nonlinear electrodynamics sources were studied in [21, 22,
23].

In four dimensions one of the first hairy black holes was derived by
Bocharova, Bronnikov and Melnikov and independently by Bekenstein, called BBMB
black hole [24]. The BBMB action consists of GR and a conformally coupled
scalar field. The metric has the form of an extremal Reissner-Nördstrom
spacetime, however the scalar field is divergent at the horizon and it is
found to be unstable under scalar perturbations [25]. A cosmological constant
can make the scalar field regular at horizon [26] but the solution is still
unstable [27]. Later the MTZ black hole was considered with a scalar potential
[28], where the scalar field is finite at horizon and the spacetime is
hyperbolic. Interestingly, a charged black hole solution with a scalar field
minimally coupled to gravity was found [29], in which the mass of the black
hole is zero due to the cancellation of the contributions of the scalar field
and the gravitational field. In this solution the self-interacting potential
of the scalar field breaks the conformal invariance of the MTZ black hole. A
new class of exact hairy black hole solutions was discussed in [30, 31]. In
[32, 33] neutral and charged black holes with a minimally coupled scalar field
have also been found. In [34, 35, 36] black holes with non-minimal derivative
coupling were studied. Black holes in bi-scalar extensions of Horndeski
theories were found in [37], hairy black hole solutions with time dependent
scalar field were discussed in [38] and in [39] hairy black holes in disformal
scalar-tensor gravity theories were discussed. The stability of black holes
with non-minimally coupled scalar hair was studied in [40].

The $f(R)$ theories were mainly introduced in cosmology in an attempt to
describe the early and late cosmological history of our Universe [41, 42]. The
consistency with GR imposed constraints on the choices of the $f(R)$ models
[43]. In these theories black hole solutions were found, that deviate from the
GR black holes or they possess new properties that are distinguishable from
the known GR solutions. Static and spherically symmetric black hole solutions
were derived in $(3+1)$-dimensions and $(2+1)$-dimensions [44, 45, 46, 47,
48], while in [49, 50, 51, 52, 53, 54, 55, 56, 57, 62] charged and rotating
solutions were found and thin-shells surrounding $f(R)$ black holes with
dynamical scalar curvature were discussed [58]. Static and spherically
symmetric black hole solutions were investigated with constant curvature, with
and without electric charge and cosmological constant in [59, 60, 61]. Scalar
fields have been recently introduced as matter fields in the context of
$f(R)$, exact black hole solutions have been found and the corresponding
physical properties have been investigated in [63, 64, 65], while the
stability of $f(R)$ black holes was also considered in [66]. The possibility
of dressing the black hole with a gravitational hair has also been discussed
in [68, 67].

In this work, we generalize the $(2+1)$-dimensional conformal black hole
solution [4] in $f(R)$ gravity. Although our study shows that the conformal
invariance has to be broken due to the nonlinear term of the $f(R)$ model, an
exact black hole solution can be solved analytically, while the $f(R)$ theory
can be obtained asymptotically. When the nonlinear term of the $f(R)$ model
decouples, the solution reduces to the conformally dressed black hole with
$f(R)=R+2l^{-2}$ (the abbreviation ”GR black hole/solution/case” will always
refer to the conformally dressed $(2+1)$-dimensional black hole [4]) .
Besides, the entropy of the $f(R)$ black hole may be higher than the
corresponding conformal black hole, which indicates our solution may be
thermodynamically preferred.

The paper is organized as follows. In Section II we review the known
conformally dressed black hole solution [4] for comparison. In Section III we
generalize the conformally dressed black hole in $f(R)$ gravity. An exact
black hole solution is obtained with the $f(R)$ function being solved from the
equations asymptotically, while the conformal invariance is broken due to the
nonlinear term of the $f(R)$ model. We show that the $f(R)$ model is free of
ghosts and tachyonic instabilities. In Section IV we study the thermodynamics
of the solution and find that the solution is thermodynamically preferred for
most cases. Finally, in Section V we conclude.

##  II Conformal $(2+1)$-dimensional black hole

We begin with a brief review of the conformally dressed black hole derived in
[4]. The action of the theory consists of the Ricci scalar, a negative
cosmological constant, and a conformally coupled scalar field, namely

$S=\frac{1}{2}\int
d^{3}x\sqrt{-g}\Big{\\{}\frac{R+2l^{-2}}{\kappa}-\partial_{\mu}\phi\partial^{\mu}\phi-\frac{1}{8}R\phi^{2}\Big{\\}}~{},$
(1)

where we will use $\kappa=8\pi G=1$ for simplicity throughout the paper. By
variation one can obtain the Einstein equation and the Klein-Gordon equation

$\displaystyle G_{\mu\nu}-g_{\mu\nu}l^{-2}=T_{\mu\nu}~{},$ (2)
$\displaystyle\Box\phi-\cfrac{1}{8}R\phi=0~{},$ (3)

where $G_{\mu\nu}\equiv R_{\mu\nu}-\frac{1}{2}g_{\mu\nu}R$ and the energy-
momentum tensor is given by

$T_{\mu\nu}=\partial_{\mu}\phi\partial_{\nu}\phi-\frac{1}{2}g_{\mu\nu}\partial^{\alpha}\phi\partial_{\alpha}\phi+\frac{1}{8}\Big{(}g_{\mu\nu}\Box-\nabla_{\mu}\nabla_{\nu}+G_{\mu\nu}\Big{)}\phi^{2}~{}.$
(4)

One can prove that by virtue of equation (3) the energy-momentum tensor is
traceless so that we have a constant Ricci scalar

$R=-\frac{6}{l^{2}}~{}.$ (5)

We assume the metric ansatz with $g_{tt}g_{rr}=-1$

$ds^{2}=-b(r)dt^{2}+b(r)^{-1}dr^{2}+r^{2}d\theta^{2}~{},$ (6)

where $b(r)$ is the only degree of freedom and it can be obtained from the
Ricci scalar (5)

$b(r)=\frac{r^{2}}{l^{2}}-\frac{c_{1}}{r}+c_{2}~{},$ (7)

while from the $tt$ and $rr$ components of the Einstein equation we can get
the scalar field

$\phi(r)=\frac{1}{\sqrt{c_{3}r+c_{4}}}~{}.$ (8)

Substituting them into the $\theta\theta$ component of the Einstein equation
and together with the Klein-Gordon equation (3) we have

$\displaystyle b(r)=\frac{r^{2}}{l^{2}}+\frac{B^{2}(-2B-3r)}{l^{2}r}~{},$ (9)
$\displaystyle\phi(r)=\sqrt{\frac{8B}{r+B}}~{},$ (10)

where $B=c_{4}/c_{3}>0$.

This solution is regular for any positive $r$, except for a singularity at the
origin $r=0$, as can be seen from the divergence of the Kretschmann scalar
$K\equiv R_{\mu\nu\sigma\rho}R^{\mu\nu\sigma\rho}$. The metric function $b(r)$
has only one root which gives the radius of the event horizon $r_{h}=2B$. The
scalar field does not diverge at the event horizon $r_{h}$ like the BBMB black
hole [24] because of the presence of a negative cosmological constant.

The Hawking temperature [78] is given by the Euclidean trick ($t\to-i\tau$)

$T_{H}=\frac{b^{\prime}(r_{h})}{4\pi}=\frac{9B}{8\pi l^{2}}~{}.$ (11)

Using Wald’s formula [69], the entropy at the event horizon can also be
obtained as

$S=\frac{\mathcal{A}}{4}\left(1-\frac{1}{8}\phi(r_{h})^{2}\right)=\frac{\pi
r_{h}}{3}=\frac{2\pi B}{3}~{},$ (12)

where $\mathcal{A}=2\pi r_{h}$ is the horizon area.

The entropy acquires another term, besides the GR one, that depends on the
scalar field, which comes from the non-minimal coupling between matter and
curvature. As a result, the entropy is smaller than the corresponding BTZ
black hole entropy [1] which is $\pi r_{h}/2$, but is positive and finite,
while the entropy of the BBMB black hole is infinite due to the divergence of
the scalar field at the event horizon. The conserved black hole mass can be
obtained by using the first law of thermodynamics

$dM=TdS\rightarrow M=\int
T(r_{h})S^{\prime}(r_{h})dr_{h}=\frac{3r_{h}^{2}}{32l^{2}}=\frac{3B^{2}}{8l^{2}}~{}.$
(13)

All thermodynamic quantities grow with the increase of the scalar charge $B$
(or $r_{h}$), in agreement with those obtained from the Hamiltonian formalism
[4].

The scalar field dresses the black hole with a secondary scalar hair, since
its charge $B$ is not an independent conserved quantity, as it is related to
the conserved mass of the black hole.

##  III $f(R)$ Gravity Black Hole Solution

In this Section we extend the conformal black hole solution [4] described in
the previous Section in $f(R)$ gravity by replacing the Einstein-Hilbert term
$R$ with the $f(R)$ function and endowing the scalar field with a self-
interacting potential $V(\phi)$ in the action

$S=\frac{1}{2}\int
d^{3}x\sqrt{-g}\Big{\\{}f(R)-\partial_{\mu}\phi\partial^{\mu}\phi-\cfrac{1}{8}R\phi^{2}-2V(\phi)\Big{\\}}~{},$
(14)

The field equations that arise from this action are

$\displaystyle I_{\mu\nu}\equiv$ $\displaystyle
f_{R}R_{\mu\nu}-\cfrac{1}{2}g_{\mu\nu}f(R)+g_{\mu\nu}\Box
f_{R}-\nabla_{\mu}\nabla_{\nu}f_{R}=T_{\mu\nu}~{},$ (16)
$\displaystyle\Box\phi-\cfrac{1}{8}R\phi-V^{\prime}(\phi)=0~{},$

where $f_{R}\equiv\cfrac{df(R)}{dR}$ and the energy momentum tensor becomes

$T_{\mu\nu}=\partial_{\mu}\phi\partial_{\nu}\phi-\frac{1}{2}g_{\mu\nu}\partial^{\alpha}\phi\partial_{\alpha}\phi+\frac{1}{8}\Big{(}g_{\mu\nu}\Box-\nabla_{\mu}\nabla_{\nu}+G_{\mu\nu}\Big{)}\phi^{2}-g_{\mu\nu}V(\phi)~{}.$
(17)

The trace of Einstein equation (16) gives

$I^{\mu}_{\,\mu}\equiv 2f_{R}R-3f(R)+4\Box
f_{R}=\phi\Box\phi-R\phi^{2}/8-6V(\phi)~{}.$ (18)

Assuming the same metric ansatz (6), the $tt,rr,\theta\theta$ components of
the Einstein equation and the Klein-Gordon equation take the form

$\displaystyle
tt:2r\left(b^{\prime}\left(\phi\phi^{\prime}-4f^{\prime}_{R}\right)+4f_{R}b^{\prime\prime}-2b\left(4f^{\prime\prime}_{R}+\phi^{\prime
2}-\phi\phi^{\prime\prime}\right)+4f-8V\right)+b^{\prime}\left(8f_{R}+\phi^{2}\right)-16bf^{\prime}_{R}+4b\phi\phi^{\prime}=0~{},$
(19) $\displaystyle
rr:2r\left(b^{\prime}\left(\phi\phi^{\prime}-4f^{\prime}_{R}\right)+4f_{R}b^{\prime\prime}+4b\phi^{\prime
2}+4f-8V\right)+b^{\prime}\left(8f_{R}+\phi^{2}\right)-16bf^{\prime}_{R}+4b\phi\phi^{\prime}=0~{},$
(20)
$\displaystyle\theta\theta:r\left(4b^{\prime}\left(\phi\phi^{\prime}-4f^{\prime}_{R}\right)+\phi^{2}b^{\prime\prime}-4b\left(4f^{\prime\prime}_{R}+\phi^{\prime
2}-\phi\phi^{\prime\prime}\right)+8f-16V\right)+16f_{R}b^{\prime}=0~{},$ (21)
$\displaystyle\text{KG:
}b^{\prime}(r)\phi^{\prime}(r)+\frac{\phi(r)\left(2b^{\prime}(r)+rb^{\prime\prime}(r)\right)}{8r}+\frac{b(r)\phi^{\prime}(r)}{r}+b(r)\phi^{\prime\prime}(r)-\frac{V^{\prime}(r)}{\phi^{\prime}(r)}=0~{},$
(22)

also the trace (18) becomes

$I^{\mu}_{\,\mu}\equiv-32rb^{\prime}f^{\prime}_{R}+32f_{R}b^{\prime}+16rf_{R}b^{\prime\prime}+8r\phi
b^{\prime}\phi^{\prime}+2\phi^{2}b^{\prime}+r\phi^{2}b^{\prime\prime}-32bf^{\prime}_{R}-32rbf^{\prime\prime}_{R}+8b\phi\phi^{\prime}+8rb\phi\phi^{\prime\prime}+24rf-48rV=0~{}.$
(23)

The Klein-Gordon equation can be obtained by taking the covariant derivative
of Einstein’s equation [64]. Therefore, we have a system of three independent
equations with four unknown functions: the $f(R)$ function, the potential
$V(\phi)$, the scalar field $\phi(r)$ and the metric function $b(r)$. We will
leave the potential undetermined and solve it from the field equations. We
will then check the trace of the energy-momentum tensor. A vanishing trace
will indicate that the matter field is conformally coupled to gravity and a
scale (if any) is counterbalanced in the action. From equations (19) and (20)
we can obtain the relation between the gravitational function $f_{R}(r)$ and
the scalar field $\phi(r)$

$4f^{\prime\prime}_{R}(r)+3\phi^{\prime}(r)^{2}-\phi(r)\phi^{\prime\prime}(r)=0~{}.$
(24)

We can immediately integrate this equation for $f_{R}(r)$

$f_{R}(r)=s+\alpha
r+\int\int\frac{1}{4}\left(\phi(r)\phi^{\prime\prime}(r)-3\phi^{\prime}(r)^{2}\right)drdr~{},$
(25)

where $s$ and $\alpha$ are constants of integration. The constant $s$ is the
coefficient of the Einstein-Hilbert term, $\alpha$ is related to geometric
corrections to Einstein gravity that are encoded in $f(R)$ theories and the
last term is generated from the scalar field. It shows that the scalar field
gives an immediate modification to the $f(R)$ model if the integrand does not
equal zero. To simplify the equations we consider the integrand to be
vanishing, i.e. $f_{R}^{\prime\prime}(r)=0$, which gives the profile of the
scalar field as $\phi(r)=\sqrt{A/(r+B)}$ and $f_{R}(r)=s+\alpha r$. Also, in
order to make it comparable with the GR case [4] we use $A=8B$ and $s=1$, then
the scalar field becomes same with (10) and

$f_{R}(r)=1+\alpha r~{}.$ (26)

We can immediately integrate $f_{R}(r)$ with respect to Ricci scalar to obtain
the general form of the $f(R)$ theory

$f_{R}(r)=1+\alpha r\to f(R)=R+\alpha\int^{R}r(R)dR+C~{},$ (27)

where $C$ is an integration constant with the unit $[L]^{-2}$, related to the
cosmological constant. This expression shows that a geometric correction term
appears in addition to the Einstein-Hilbert term, while the scalar field does
not appear immediately in the $f(R)$ model as happens in [65].

Then we can solve the metric function as

$b(r)=-\frac{3B^{2}}{l^{2}(\alpha B+1)^{2}}-\frac{2B^{3}}{l^{2}r(\alpha
B+1)}+\frac{6\alpha B^{2}r}{l^{2}(\alpha
B+1)^{3}}+r^{2}\left(\frac{1}{l^{2}}+\frac{6\alpha^{2}B^{2}}{l^{2}(\alpha
B+1)^{4}}\ln\left(\frac{r}{\alpha l(B+r)+l}\right)\right)~{},$ (28)

where $l$ is the AdS radius that appears as an integration constant. We can
see that the metric function is well behaved for any $r>0$ if we constrain the
parameters $B,\alpha$ to be positive. For this reason we will impose that
$\alpha,B>0$. At large distances, the metric function asymptotes to

$b(r\to\infty)\sim-\Lambda_{\text{eff}}r^{2}-\frac{2B^{2}}{\alpha
l^{2}r}+\mathcal{O}(r^{-2})~{},$ (29)

where the effective cosmological constant that the $f(R)$ theory and the non-
minimal coupling generate is given by

$\Lambda_{\text{eff}}=-\left(\frac{1}{l^{2}}-\frac{6\alpha^{2}B^{2}\ln(\alpha
l)}{l^{2}(\alpha B+1)^{4}}\right)~{}.$ (30)

For vanishing scalar charge $B$, we obtain pure AdS spacetime and we will also
consider that $1-6\alpha^{2}B^{2}\ln(\alpha l)/(\alpha B+1)^{4}>0$ in order to
have an asymptotically AdS spacetime, so we can compare our solution with [4].
Now, we can obtain the potential from the Klein-Gordon equation

$V(r)=\frac{B^{3}}{l^{2}(\alpha
B+1)^{4}}\Bigg{(}\frac{6\alpha^{2}\left(B^{2}+\alpha(B+r)^{3}\right)}{(B+r)^{3}}\ln\left(\frac{r}{\alpha
l(B+r)+l}\right)+\frac{3\left(\alpha^{2}-\alpha^{4}B^{2}\right)}{B+r}+\frac{3\alpha^{3}(\alpha
B+1)^{2}}{\alpha(B+r)+1}\\\ +\frac{6\alpha^{2}B(\alpha
B+1)}{(B+r)^{2}}+\frac{\alpha B(\alpha B+1)(\alpha B(\alpha
B+5)-2)}{(B+r)^{3}}+6\alpha^{3}\ln\alpha l\Bigg{)}~{},$ (31)

which vanishes at spatial infinity and as a function of $\phi$ reads

$V(\phi)=\frac{\alpha B}{512l^{2}(\alpha B+1)^{4}\left(8\alpha
B+\phi^{2}\right)}\Bigg{(}\phi^{2}(\alpha
B+1)(3072\alpha^{2}B^{2}+\phi^{6}(\alpha B(\alpha B+5)-2)+8\alpha
B\phi^{4}(\alpha B+1)(\alpha B+4)\\\ +192\alpha B\phi^{2}(\alpha B+1))+6\alpha
B\left(8\alpha B+\phi^{2}\right)\left(\left(512\alpha
B+\phi^{6}\right)\ln\left(\frac{B\left(-\phi^{2}+8\right)}{l\left(8\alpha
B+\phi^{2}\right)}\right)+512\alpha B\ln\left(\alpha
l\right)\right)\Bigg{)}~{}.$ (32)

The Ricci scalar can be obtained from the metric function

$R(r)=-\frac{6}{l^{2}}-\frac{6\alpha B^{2}\left(\alpha\left(2\alpha
B^{2}+9\alpha Br+4B+6\alpha r^{2}+9r\right)+2\right)}{l^{2}r(\alpha
B+1)^{3}(\alpha(B+r)+1)^{2}}-\frac{36\alpha^{2}B^{2}}{l^{2}(\alpha
B+1)^{4}}\ln\left(\frac{r}{\alpha l(B+r)+l}\right)~{},$ (33)

while the function $f(r)$ yields

$f(r)=-\frac{4}{l^{2}}+\frac{6\alpha B^{2}\left(\alpha\left(\alpha
B^{2}(3\alpha r-2)+B(\alpha r(2\alpha r-3)-4)-2r(2\alpha
r+3)\right)-2\right)}{l^{2}r(\alpha B+1)^{3}(\alpha(B+r)+1)^{2}}+\\\
\frac{12\alpha^{2}B^{2}}{l^{2}(\alpha B+1)^{4}}\left((\alpha
B-2)\ln\left(\frac{r}{\alpha l(B+r)+l}\right)+\alpha B\ln(\alpha
l)\right)~{},$ (34)

The Ricci scalar is divergent at origin and related to the AdS scale at
infinity. The Kretschmann scalar behaves as

$R^{\alpha\beta\gamma\delta}R_{\alpha\beta\gamma\delta}(r\to
0)\sim\frac{24B^{6}}{l^{4}r^{6}(\alpha
B+1)^{2}}+\mathcal{O}\left(\frac{1}{r^{4}}\right)~{},$ (35)

near the origin, which is also divergent at $r=0$, indicating a physical
singularity. It is clear that we cannot invert the Ricci scalar and solve it
for $r$, to substitute back to the $f(r)$ function in order to obtain $f(R)$.
However, we can use asymptotics to have a feeling of the curvature model at
the origin and at infinity. The asymptotic expressions of the Ricci scalar
near the origin and at infinity are respectively

$\displaystyle R(r\to 0)$ $\displaystyle\sim$ $\displaystyle-\frac{12\alpha
B^{2}}{l^{2}r(\alpha B+1)^{3}}+\mathcal{O}(\ln(r))~{},$ (36) $\displaystyle
R(r\to\infty)$ $\displaystyle\sim$ $\displaystyle
6\Lambda_{\text{eff}}-\frac{3B^{2}}{\alpha^{2}l^{2}r^{4}}+\mathcal{O}(r^{-5})~{},$
(37)

so the $f(R)$ function near the origin and at large distances yields
respectively

$\displaystyle f(R(r\to 0))$ $\displaystyle\sim$ $\displaystyle
R-\frac{12\alpha^{2}B^{2}}{l^{2}(\alpha B+1)^{3}}\ln(R)~{},$ (38)
$\displaystyle f(R(r\to\infty))$ $\displaystyle\sim$ $\displaystyle
R-\frac{4B\left(6\Lambda_{\text{eff}}-R\right){}^{3/4}}{3^{3/4}\sqrt{\alpha
Bl}}~{},$ (39)

up to a constant of integration. The argument of the $\ln$ tern is not
dimensionless, but this expression is an approximation.

In FIG.1 we plot $f(R(r))$ as a function of $R(r)$ to see how our $f(R)$
deviates from the GR case $f(R)=R+2l^{-2}$. We can see that for stronger
$\alpha$, our theory deviates more from GR.

Figure 1: $f(R(r))$ as a function of $R(r)$ for different values of $\alpha$,
where we have set $B=l=1$.

To check if the resultant $f(R,\phi)$ theory is free of ghost and tachyonic
instabilities [41, 70, 71, 72, 73] we need to confirm if the following
relations hold respectively

$\displaystyle f_{R_{\text{total}}}>0\to
f_{R_{\text{total}}}=f_{R_{\text{gravity}}}+f_{R_{\text{matter}}}=1+\alpha
r-\frac{1}{8}\phi(r)^{2}=+1+\alpha
r-\frac{B}{B+r}=r\left(\alpha+\frac{1}{B+r}\right)>0~{},$ (40) $\displaystyle
f_{RR}(r)>0\to
f_{RR}(r)=\frac{f_{R}^{\prime}(r)_{\text{total}}}{R^{\prime}(r)}=\frac{l^{2}r^{2}(\alpha(B+r)+1)^{3}\left(\alpha(B+r)^{2}+B\right)}{12\alpha
B^{2}(B+r)^{2}}>0~{}.$ (41)

The two above relations hold for $B>0$ and $\alpha>0$ and in this case the
resultant $f(R,\phi)$ theory is free of ghosts and avoids the tachyonic
instability. The fact that $f_{R_{\text{total}}}>0$ also ensures that the
entropy is positive [74, 75, 76, 77] and our solutions may possess a higher
entropy than the corresponding GR counterpart [4] for a fixed horizon value,
meaning that the $f(R)$ black holes may be thermodynamically preferred over
the GR one [4].

Expanding the metric function $b(r)$, $f(r)$ and $R(r)$ near $\alpha\to 0$ we
find that

$\displaystyle
b(r)=\frac{-3B^{2}r-2B^{3}+r^{3}}{l^{2}r}+\frac{2\alpha\left(3B^{2}r^{2}+3B^{3}r+B^{4}\right)}{l^{2}r}+\mathcal{O}\left(\alpha^{2}\right)~{},$
(42) $\displaystyle f(r)=-\frac{4}{l^{2}}-\frac{12\alpha
B^{2}}{l^{2}r}+\mathcal{O}\left(\alpha^{2}\right)~{},$ (43) $\displaystyle
R(r)=-\frac{6}{l^{2}}-\frac{12\alpha
B^{2}}{l^{2}r}+\mathcal{O}\left(\alpha^{2}\right)~{},$ (44)

where, as expected, at zeroth order we obtain the GR black hole [4] and the
curvature functions $f(r),R(r)$ become dynamical due to the gravitational
scale $\alpha$. The trace of the resultant energy-momentum tensor is dynamical

$T_{\mu}^{~{}\mu}=-\frac{3\alpha^{2}B^{3}}{2r(\alpha B+1)^{4}(\alpha
l(B+r)+l)^{2}}\Bigg{(}(\alpha B+1)(\alpha(2\alpha B^{2}+B(9\alpha
r+4)+3r(2\alpha r+3))+2)\\\ +6\alpha r(\alpha(B+r)+1)^{2}(\ln(\frac{r}{\alpha
l(B+r)+l})+\ln(\alpha l))\Bigg{)}~{},$ (45)

which indicates that the theory is not conformally invariant. This scale is
the geometric correction parameter $\alpha$. For vanishing $\alpha$, the trace
of the energy momentum tensor vanishes as expected since $\alpha\to 0$ gives
the GR case [4]. We present some plots of
$b(r),R(r),f(r),V(r),T^{~{}\mu}_{\mu}$ in Fig. 2, in order to better
understand our solution alongside the $\alpha=0$ case which corresponds to GR
[4]. We can see that the modified gravity parameter affects the dynamics of
the curvature related functions, while the GR black hole [4] admits a larger
horizon radius in comparison with the modified gravity one. We also plot the
horizon radius as a function of $\alpha$. The fact that the larger the
deviation from the GR solution [4] is, the smaller the horizon radius becomes,
is in agreement with the $(3+1)$-dimensional case [64]. Next, we will briefly
discuss scalar perturbations of the obtained spacetime. For this reason, we
consider a massless test scalar field $\phi_{0}$ that satisfies its equation
of motion [17],

$\Box\phi_{0}=0~{}.$ (46)

Transforming the scalar field as
$\phi_{0}=r^{-1/2}\varphi_{0}e^{-i\omega_{0}t}$, the Klein-Gordon equation
takes the form of a Schrodinger-like one

$\cfrac{d^{2}\varphi_{0}}{dr^{2}_{*}}+(\omega_{0}^{2}-V_{\text{eff}})\varphi_{0}=0~{},$
(47)

where we expressed this equation using the tortoise coordinate $r_{*}=\int
drb(r)^{-1}$. The resulting effective potential is complicated, however its
asymptotic expression is

$V_{\text{eff}}(r\to\infty)\sim\frac{3\Lambda^{2}_{\text{eff}}r^{2}}{4}+\mathcal{O}\left(\frac{1}{r^{2}}\right)~{},$
(48)

meaning that there is an AdS boundary at infinity constraining the matter
fields, regardless of the modified gravity parameter $\alpha$ and the effect
it has on $\Lambda_{\text{eff}}$. We checked that, the inclusion of a mass
term for the test scalar field does not change the behavior of the resulting
effective potential at large distances. Also, no potential well is formed near
the horizon of the black hole for both the massive and massless case, as can
be confirmed from FIG. 3, where we plot the effective potential of the
massless case, meaning that the test scalar particles are not trapped near the
black hole, so, as a result, the spacetime is stable under massless and
massive scalar perturbations.

Figure 2: The functions $b(r),V(r),R(r),f(r)$ and $T^{~{}\mu}_{\mu}(r)$ are
plotted versus $r$ with different values of $\alpha$, while in the last panel,
the radius of the event horizon $r_{h}$ is plotted as a function of $\alpha$.
In all the figures we have set $B=l=1$. Figure 3: The effective potential
$V_{\text{eff}}(r)$ for the massless test scalar particles as a function of
$r$ for different values of $\alpha$, where we have set $B=l=1$.

##  IV Thermodynamics

In this section we will study the thermodynamics of the extended black hole
solution in $f(R)$ gravity, including Hawking temperature, entropy and the
conserved mass.

###  IV.1 Hawking temperature

The Hawking temperature can be calculated as

$T_{H}=\frac{b^{\prime}(r_{h})}{4\pi}=\frac{3B^{2}(B+r_{h})}{2\pi
l^{2}r_{h}^{2}(\alpha B+\alpha r_{h}+1)}~{},$ (49)

where the relation $b(r_{h})=0$ has been used. As expected, it can reduce to
the Hawking temperature (11) in conformal dressed black hole case [4] when
$\alpha\to 0$.

In FIG. 4, we plot the Hawking temperature $T_{H}$ as a function of $\alpha$.
With the increase of $\alpha$, the Hawking temperature of the black hole first
decreases slightly, then grows up to a maximum, finally descends until
approaching zero which can be seen from the expression (49).

Figure 4: The Hawking temperature is plotted as a function of $\alpha$, where
we have set $B=l=1$.

###  IV.2 Entropy

Using Wald’s formula [69, 79], we can calculate the entropy of the black hole
in $f(R)$ gravity with a non-minimal coupling as

$S=-\frac{1}{4}\int
d\theta\sqrt{r_{h}^{2}}\left(\frac{\partial\mathcal{L}}{\partial
R_{\alpha\beta\gamma\delta}}\right)\Big{|}_{r=r_{h}}\hat{\varepsilon}_{\alpha\beta}\hat{\varepsilon}_{\gamma\delta}~{},$
(50)

where $\hat{\varepsilon}_{\alpha\beta}$ is the binormal to the horizon surface
[80], $\mathcal{L}$ is the Lagrangian of the theory, and

$\frac{\partial\mathcal{L}}{\partial
R_{\alpha\beta\gamma\delta}}\Big{|}_{r=r_{h}}=\frac{1}{2}\left(\frac{f_{R}(r_{h})}{2}-\frac{1}{16}\phi(r_{h})^{2}\right)\left(g^{\alpha\gamma}g^{\beta\delta}-g^{\beta\gamma}g^{\alpha\delta}\right)~{}.$
(51)

Finally the formula of the entropy for our theory can be obtained

$S=\pi
r_{h}\left(\frac{f_{R}(r_{h})}{2}-\frac{1}{16}\phi(r_{h})^{2}\right)=\frac{\mathcal{A}}{4}f_{R_{\text{total}}}(r_{h})~{}.$
(52)

Substituting the explicit expression for $f_{R_{\text{total}}}$, we have

$S=\frac{1}{2}\pi r_{h}\left(1+\alpha r_{h}-\frac{B}{B+r_{h}}\right)~{},$ (53)

In fact, here $r_{h}$ is also changing with the choices of $B$, $l$ and
$\alpha$. One might deduce that since $\alpha>0$, the $f(R)$ black holes have
higher entropy than the conformal ones [4]. However, we have to keep in mind
that the conformal case [4] has a larger radius for the event horizon as can
be seen from the metric function $b(r)$ in Fig. 2. Using the relations (53)
and $b(r_{h})=0$, we can plot the entropy at the event horizon as a function
of $\alpha$ in FIG. 5.

Figure 5:  The entropy $S$ at the event horizon is plotted as a function of
$\alpha$, where we have set $B=l=1$.

With the increase of $\alpha$, the entropy first decreases a little bit to a
minimum value, then grows up always. Therefore, for most values of $\alpha$,
the entropy of the $f(R)$ black hole is higher than the corresponding
conformal $(2+1)$-dimensional black hole [4], indicating that our solution is
thermodynamically preferred for most cases. It is worth to mention that the
conformal case ($\alpha=0$) is a local maximum of the entropy with respect to
$\alpha$.

###  IV.3 Conserved Mass

For a $D$-dimensional spacetime manifold $\mathcal{M}$, which is topologically
the product of a spacelike hypersurface and a real line interval
$\Sigma\times\mathcal{I}$, the total quasi-local energy is defined as [81, 82]

$E=\int_{\mathcal{B}}d^{D-2}x\sqrt{\sigma}\varepsilon~{},$ (54)

where $\mathcal{B}\equiv\partial\Sigma$ is the $(D-2)$-dimensional boundary,
$\sigma$ is the determinant of the induced metric $\sigma_{ab}$ on $B$, and
$\varepsilon$ is the energy density.

The boundary $\partial\mathcal{M}$ consists of initial and final spacelike
hypersurfaces $t^{\prime}$ and $t^{\prime\prime}$ respectively, and a timelike
hypersurface $\mathcal{T}=\mathcal{B}\times\mathcal{I}$ joining them. The
$(D-1)$-metric $\gamma_{ij}$ on $\mathcal{T}$ can be written according to the
ADM decomposition as

$\gamma_{ij}dx^{i}dx^{j}=-N^{2}dt^{2}+\sigma_{ab}\left(dx^{a}+V^{a}dt\right)\left(dx^{b}+V^{b}dt\right)~{}.$
(55)

The conserved charge associated with a Killing vector field $\xi^{i}$ is
defined as [81, 82]

$Q_{\xi}=\int_{\mathcal{B}}d^{D-2}x\sqrt{\sigma}\left(\varepsilon
u^{i}+j^{i}\right)\xi_{i}~{},$ (56)

where $u^{i}$ is the unit normal to spacelike hypersurfaces $t^{\prime}$ or
$t^{\prime\prime}$, and $j^{i}$ is the momentum density.

We first calculate the quasi-local energy inside the spacelike hypersurface
$r=r_{0}=const.$

$E=\int_{\mathcal{B}}d^{D-2}x\sqrt{\sigma}\varepsilon=\int_{\mathcal{B}}d^{D-2}x\sqrt{\sigma}\left(k-\varepsilon_{0}\right)~{},$
(57)

where $k=-\sqrt{b(r_{0})}/r_{0}$ is the trace of the extrinsic curvature and
$\epsilon_{0}$ is the vacuum energy density.

For $r_{0}\to\infty$, we have the global quasi-local energy

$E(r_{0})=-\frac{2\pi
r_{0}}{l}\sqrt{\frac{6\alpha^{2}B^{2}\ln\left(\frac{1}{\alpha
l}\right)}{(\alpha B+1)^{4}}+1}-2\pi
r_{0}\varepsilon_{0}(r_{0})+\mathcal{O}\left(\frac{1}{r_{0}^{2}}\right)~{}.$
(58)

To make it finite, the vacuum energy density has to be

$\varepsilon_{0}(r_{0})=-\frac{1}{l}\sqrt{\frac{6\alpha^{2}B^{2}\ln\left(\frac{1}{\alpha
l}\right)}{(\alpha B+1)^{4}}+1}~{},$ (59)

then the global quasi-local energy becomes zero.

The conserved mass can be further calculated as

$\displaystyle M$ $\displaystyle=$
$\displaystyle-\int_{\mathcal{B}}d^{D-2}x\sqrt{\sigma}\varepsilon
u_{i}\xi^{i}$ (60) $\displaystyle=$
$\displaystyle\lim_{r_{0}\to\infty}E(r_{0})\sqrt{b(r_{0})}$ $\displaystyle=$
$\displaystyle\lim_{r_{0}\to\infty}\frac{2\pi B^{2}}{\alpha
l^{2}r_{0}}-\frac{3\pi
B^{2}}{2r_{0}^{2}\alpha^{2}l^{2}}+\mathcal{O}\left(\frac{1}{r_{0}^{4}}\right)$
$\displaystyle=$ $\displaystyle 0~{},$

which, however, turns out to be zero.

The fact that the conserved mass is zero has its root in the $f(R)$ theory. It
is known that the conserved mass is related to the constant term in the metric
function that survives in the asymptotic expansion at infinity when one is
dealing with (A)dS spacetime in $(2+1)$ dimensions. We can split the metric
function in two parts. A part that is not completely supported by the
gravitational scale $\alpha$ denoted by $b(r)_{GR,\alpha,\phi}$ and a part
that is completely supported by $\alpha$, i.e, when we turn off $\alpha$ these
terms will vanish, denoted $b(r)_{\alpha,\phi}$. We have
$b(r)=b(r)_{\alpha,\phi}+b(r)_{GR,\alpha,\phi}$, where

$\displaystyle b(r)_{GR,\alpha,\phi}=-\frac{3B^{2}}{l^{2}(\alpha
B+1)^{2}}-\frac{2B^{3}}{l^{2}r(\alpha B+1)}+\frac{r^{2}}{l^{2}}~{},$ (61)
$\displaystyle b(r)_{\alpha,\phi}=\frac{6\alpha B^{2}r}{l^{2}(\alpha
B+1)^{3}}+r^{2}\frac{6\alpha^{2}B^{2}}{l^{2}(\alpha
B+1)^{4}}\ln\left(\frac{r}{\alpha l(B+r)+l}\right)~{}.$ (62)

It is clear that by setting $\alpha=0$ in $b(r)_{\alpha,\phi}$, the term will
vanish, while $b(r)_{GR,\alpha,\phi}$ will yield the conformal black hole
solution [4]. It can be seen that the $b(r)_{GR,\alpha,\phi}$ part contains a
term that is related to the mass of the black hole

$M_{GR,\alpha,\phi}=\frac{3B^{2}}{l^{2}(\alpha B+1)^{2}}~{},$ (63)

while expanding the $b(r)_{\alpha,\phi}$ term at infinity, we find that the
constant term will be related to the mass of the black hole reads

$M_{\alpha,\phi}=-\frac{3B^{2}}{l^{2}(\alpha B+1)^{2}}~{},$ (64)

which is the opposite of the mass term the $b(r)_{GR,\alpha,\phi}$ term
generates. Hence the term that exists because of the $f(R)$ function in the
metric (28), $b(r)_{\alpha,\phi}$ yields a massless black hole, and one can
argue that the $f(R)$ theory that satisfies $f_{R}(r)=s+\alpha r$ yields black
holes with no mass. In fact, if one ignores the scalar field and considers
only

$S=\int d^{3}{x}\sqrt{-g}f(R)~{},$ (65)

with our metric ansatz (6) the field equations will naturally yield
$f_{R}(r)=s+\alpha r$, where a logarithmic term that depends on $\alpha$ will
cancel the mass the other terms generate yielding massless black holes. For
this reason, a more general metric ansatz has to be considered that will yield
different profiles for $f_{R}(r)$, as is indeed recently done in [57].
However, in our case, since we are interested in comparing the $f(R)$ black
hole with the GR one [4], we cannot consider a more general metric ansatz, as
the metric specifies the form of the scalar field, which further specifies
$f_{R}(r)$ as can be seen in (25).

The parameter $\alpha$ which provides a gravitational correction term to the
Ricci scalar term in our $f(R)$ theory, breaks the conformal invariance of the
GR case presented in [4]. In the case of GR [4] the mass of the black hole
depends on the scalar charge $B$. In our theory in the metric function (6)
both the gravitational parameter and the scalar charge are present and except
the mass term there is another term which is proportional to $r^{2}$ which
appears in the metric function because of the presence of both the scalar
field and the gravitational scale $\alpha$. Considering the expansions of
$b(r)_{\alpha,\phi},b(r)_{GR,\alpha,\phi}$ at infinity in (61) and (62) we can
say that the massless black hole is a result of the cancellation from the
scalar field and the gravitational field contributions. A similar behavior was
found in [29]. Breaking the conformal invariance of the action of the MTZ
black hole in the Einstein frame through a particular scalar potential, a
massless black hole was found and this was attributed to the cancellation of
gravitational and scalar field contributions.

##  V Conclusions

In this work, we considered $f(R)$ gravity theory and matter in the form of a
self-interacting, non-minimally coupled scalar field. Solving the field
equations we found that $f_{R}(r)=\frac{df(R)}{dR}=1+\alpha r$ where $\alpha$
is a non-linear correction term of the Ricci scalar $R$, having dimensions of
inverse length. If $\alpha=0$ we go back to GR recovering the theory of a
conformally coupled scalar field to gravity, discussed in [4]. The parameter
$\alpha$ introduces a gravitational scale that breaks the conformal
invariance. Calculating the exact forms of the derivatives of $f(R)$ function
we deduced that $f_{R_{\text{total}}}>0$ and $f_{RR}>0$ which makes our theory
free of ghost and tachyonic instabilities. We also calculated the conserved
mass of the black hole and interestingly we found that the black hole is
massless due to the cancellation of gravitational and scalar field
contributions to the mass term. We attributed this effect to the breaking of
the conformal invariance due to the presence of the gravitational parameter
$\alpha$.

We also studied the thermodynamics of the extended black hole solution in
$f(R)$ gravity, including Hawking temperature and the entropy. With the
increase of $\alpha$, the Hawking temperature of the black hole first
decreases slightly, then grows up to a maximum, finally descends until
approaching zero, while the entropy first decreases to a minimum value, then
grows up with the increase of $\alpha$. Besides, the entropy of the black hole
is higher than the corresponding conformal $(2+1)$-dimensional black hole [4]
for most values of $\alpha$, indicating that our solution is thermodynamically
preferred for most cases. We also briefly discussed the stability of the
obtained spacetime under massive and massless scalar perturbations and deduced
that the obtained solution is stable under both types of perturbations.

A possible extension of this work is to perform a detailed thermodynamical
analysis to examine the validity of the first law of thermodynamics, as well
as the thermodynamical stability and possible phase transitions of the
obtained black hole solution. One can also introduce a linear Maxwell field in
the action and study the interplay of the gravitational parameter $\alpha$ and
the charge $Q$ on the conformal invariance of the theory and see their effects
on the black hole solution. With the addition of electric charge one can also
study possible thermodynamical critical behaviors, pointing out how the
gravitational scale $\alpha$ affects thermodynamics. Rotating solutions might
also be considered. The properties of the resultant conformal field theory
could also be studied. The stability of the obtained spacetime may be
investigated, as well as the geodesic motion of particles around the black
hole solution and how the gravitational parameter $\alpha$ affects the motion.

## References

  * [1] M. Banados, C. Teitelboim and J. Zanelli, “The Black hole in three-dimensional space-time,” Phys. Rev. Lett. 69, 1849-1851 (1992) [arXiv:hep-th/9204099 [hep-th]].
  * [2] M. Banados, M. Henneaux, C. Teitelboim and J. Zanelli, “Geometry of the (2+1) black hole,” Phys. Rev. D 48, 1506-1525 (1993) [erratum: Phys. Rev. D 88, 069902 (2013)] [arXiv:gr-qc/9302012 [gr-qc]].
  * [3] M. Cataldo, N. Cruz, S. del Campo and A. Garcia, “(2+1)-dimensional black hole with Coulomb - like field,” Phys. Lett. B 484, 154 (2000) [arXiv:hep-th/0008138 [hep-th]].
  * [4] C. Martínez and J. Zanelli, “Conformally dressed black hole in (2+1)-dimensions,” Phys. Rev. D 54, 3830 (1996) [gr-qc/9604021].
  * [5] M. Henneaux, C. Martínez, R. Troncoso and J. Zanelli, “Black holes and asymptotics of 2+1 gravity coupled to a scalar field,” Phys. Rev. D 65, 104007 (2002) [hep-th/0201170].
  * [6] F. Correa, C. Martínez and R. Troncoso, “Scalar solitons and the microscopic entropy of hairy black holes in three dimensions,” JHEP 1101, 034 (2011). [arXiv:1010.1259 [hep-th]].
  * [7] F. Correa, C. Martínez and R. Troncoso, “Hairy Black Hole Entropy and the Role of Solitons in Three Dimensions,” JHEP 1202, 136 (2012). [arXiv:1112.6198 [hep-th]].
  * [8] F. Correa, A. Faúndez and C. Martínez, “Rotating hairy black hole and its microscopic entropy in three spacetime dimensions,” Phys. Rev. D 87, 027502 (2013) [arXiv:1211.4878 [hep-th]].
  * [9] O. Baake, M. F. Bravo Gaete and M. Hassaine, “Spinning black holes for generalized scalar tensor theories in three dimensions,” Phys. Rev. D 102, no.2, 024088 (2020) [arXiv:2005.10869 [hep-th]].
  * [10] M. Bravo-Gaete and M. Hassaine, “Thermodynamics of a BTZ black hole solution with an Horndeski source,” Phys. Rev. D 90, no.2, 024008 (2014) [arXiv:1405.4935 [hep-th]].
  * [11] M. Natsuume and T. Okamura, “Entropy for asymptotically AdS(3) black holes,” Phys. Rev. D 62, 064027 (2000) [hep-th/9911062].
  * [12] J. Aparício, D. Grumiller, E. Lopez, I. Papadimitriou and S. Stricker, “Bootstrapping gravity solutions,” JHEP 1305, 128 (2013) [arXiv:1212.3609 [hep-th]].
  * [13] W. Xu, L. Zhao and D. -C. Zou, “Three dimensional rotating hairy black holes, asymptotics and thermodynamics,” arXiv:1406.7153 [gr-qc].
  * [14] M. Cardenas, O. Fuentealba and C. Martínez, “Three-dimensional black holes with conformally coupled scalar and gauge fields,” Phys. Rev. D 90, no.12, 124072 (2014) [arXiv:1408.1401 [hep-th]].
  * [15] W. Xu and D. C. Zou, “$(2+1)$ -Dimensional charged black holes with scalar hair in Einstein–Power–Maxwell Theory,” Gen. Rel. Grav. 49, no.6, 73 (2017) [arXiv:1408.1998 [hep-th]].
  * [16] W. Xu and L. Zhao, “Charged black hole with a scalar hair in (2+1) dimensions,” Phys. Rev. D 87, no.12, 124008 (2013) [arXiv:1305.5446 [gr-qc]].
  * [17] Z. Y. Tang, Y. C. Ong, B. Wang and E. Papantonopoulos, “General black hole solutions in ( 2+1 )-dimensions with a scalar field nonminimally coupled to gravity,” Phys. Rev. D 100, no.2, 024003 (2019) [arXiv:1901.07310 [gr-qc]].
  * [18] K. C. K. Chan and R. B. Mann, “Static charged black holes in (2+1)-dimensional dilaton gravity,” Phys. Rev. D 50, 6385 (1994) [erratum: Phys. Rev. D 52, 2600 (1995)] [arXiv:gr-qc/9404040 [gr-qc]].
  * [19] K. C. K. Chan, “Modifications of the BTZ black hole by a dilaton / scalar,” Phys. Rev. D 55, 3564-3574 (1997) [arXiv:gr-qc/9603038 [gr-qc]].
  * [20] P. Bueno, P. A. Cano, J. Moreno and G. van der Velde, “Regular black holes in three dimensions,” Phys. Rev. D 104, no.2, L021501 (2021) [arXiv:2104.10172 [gr-qc]].
  * [21] M. Cataldo and A. Garcia, “Regular (2+1)-dimensional black holes within nonlinear electrodynamics,” Phys. Rev. D 61, 084003 (2000) [arXiv:hep-th/0004177 [hep-th]].
  * [22] S. Habib Mazharimousavi, M. Halilsoy and T. Tahamtan, “Regular charged black hole construction in 2+1 -dimensions,” Phys. Lett. A 376, 893-898 (2012) [arXiv:1107.0242 [gr-qc]].
  * [23] Y. He and M. S. Ma, “$(2+1)$-dimensional regular black holes with nonlinear electrodynamics sources,” Phys. Lett. B 774, 229-234 (2017) [arXiv:1709.09473 [gr-qc]].
  * [24] N. Bocharova, K. Bronnikov and V. Melnikov, Vestn. Mosk. Univ. Fiz. Astron. 6, 706 (1970);
J. D. Bekenstein, Annals Phys. 82, 535 (1974);
J. D. Bekenstein, “Black Holes With Scalar Charge,” Annals Phys. 91, 75
(1975).

  * [25] K. A. Bronnikov and Y. N. Kireyev, “Instability of black holes with scalar charge,” Phys. Lett. A 67, 95 (1978).
  * [26] C. Martinez, R. Troncoso and J. Zanelli, “De Sitter black hole with a conformally coupled scalar field in four-dimensions,” Phys. Rev. D 67, 024008 (2003) [arXiv:hep-th/0205319 [hep-th]].
  * [27] T. J. T. Harper, P. A. Thomas, E. Winstanley and P. M. Young, “Instability of a four-dimensional de Sitter black hole with a conformally coupled scalar field,” Phys. Rev. D 70, 064023 (2004) [arXiv:gr-qc/0312104 [gr-qc]].
  * [28] C. Martinez, R. Troncoso and J. Zanelli, “Exact black hole solution with a minimally coupled scalar field,” Phys. Rev. D 70, 084035 (2004) [arXiv:hep-th/0406111 [hep-th]].
  * [29] C. Martinez and R. Troncoso, “Electrically charged black hole with scalar hair,” Phys. Rev. D 74, 064007 (2006) [arXiv:hep-th/0606130 [hep-th]].
  * [30] T. Kolyvaris, G. Koutsoumbas, E. Papantonopoulos and G. Siopsis, “A New Class of Exact Hairy Black Hole Solutions,” Gen. Rel. Grav. 43, 163-180 (2011) [arXiv:0911.1711 [hep-th]].
  * [31] T. Kolyvaris, G. Koutsoumbas, E. Papantonopoulos and G. Siopsis, Class. Quant. Grav. 29, 205011 (2012) [arXiv:1111.0263 [gr-qc]].
  * [32] P. A. González, E. Papantonopoulos, J. Saavedra and Y. Vásquez, “Four-Dimensional Asymptotically AdS Black Holes with Scalar Hair,” JHEP 12, 021 (2013) [arXiv:1309.2161 [gr-qc]].
  * [33] P. A. González, E. Papantonopoulos, J. Saavedra and Y. Vásquez, “Extremal Hairy Black Holes,” JHEP 11, 011 (2014) [arXiv:1408.7009 [gr-qc]].
  * [34] M. Rinaldi, “Black holes with non-minimal derivative coupling,” Phys. Rev. D 86, 084048 (2012) [arXiv:1208.0103 [gr-qc]].
  * [35] A. Anabalon, A. Cisterna and J. Oliva, “Asymptotically locally AdS and flat black holes in Horndeski theory,” Phys. Rev. D 89, 084050 (2014) [arXiv:1312.3597 [gr-qc]].
  * [36] M. Minamitsuji, “Solutions in the scalar-tensor theory with nonminimal derivative coupling,” Phys. Rev. D 89, 064017 (2014) [arXiv:1312.3759 [gr-qc]].
  * [37] C. Charmousis, T. Kolyvaris, E. Papantonopoulos and M. Tsoukalas, “Black Holes in Bi-scalar Extensions of Horndeski Theories,” JHEP 07, 085 (2014) [arXiv:1404.1024 [gr-qc]].
  * [38] E. Babichev and C. Charmousis, “Dressing a black hole with a time-dependent Galileon,” JHEP 08, 106 (2014) [arXiv:1312.3204 [gr-qc]].
  * [39] C. Erices, P. Filis and E. Papantonopoulos, “Hairy black holes in disformal scalar-tensor gravity theories,” Phys. Rev. D 104, no.2, 024031 (2021) [arXiv:2104.05644 [gr-qc]].
  * [40] N. Chatzifotis, C. Vlachos, K. Destounis and E. Papantonopoulos, “Stability of black holes with non-minimally coupled scalar hair,” [arXiv:2109.02678 [gr-qc]].
  * [41] A. De Felice and S. Tsujikawa, “f(R) theories,” Living Rev. Rel. 13, 3 (2010), [arXiv:1002.4928 [gr-qc]].
  * [42] A. A. Starobinsky, “A New Type of Isotropic Cosmological Models Without Singularity,” Phys. Lett. B 91 (1980) 99 [Phys. Lett. 91B (1980) 99] [Adv. Ser. Astrophys. Cosmol. 3 (1987) 130].
  * [43] S. Capozziello, O. Farooq, O. Luongo and B. Ratra, “Cosmographic bounds on the cosmological deceleration-acceleration transition redshift in $f(\mathfrak{R})$ gravity,” Phys. Rev. D 90, no.4, 044016 (2014) [arXiv:1403.1421 [gr-qc]].
  * [44] L. Sebastiani and S. Zerbini, “Static Spherically Symmetric Solutions in F(R) Gravity,” Eur. Phys. J. C 71, 1591 (2011) [arXiv:1012.5230 [gr-qc]].
  * [45] T. Multamaki and I. Vilja, “Spherically symmetric solutions of modified field equations in f(R) theories of gravity,” Phys. Rev. D 74, 064022 (2006) [arXiv:astro-ph/0606373 [astro-ph]].
  * [46] Z. Amirabi, M. Halilsoy and S. Habib Mazharimousavi, “Generation of spherically symmetric metrics in f(R) gravity,” Eur. Phys. J. C 76, no.6, 338 (2016) [arXiv:1509.06967 [gr-qc]].
  * [47] S. H. Hendi, “(2+1)-Dimensional Solutions in $F(R)$ Gravity,” Int. J. Theor. Phys. 53, no.12, 4170-4181 (2014) [arXiv:1410.7527 [gr-qc]].
  * [48] S. H. Hendi, B. Eslam Panah and R. Saffari, “Exact solutions of three-dimensional black holes: Einstein gravity versus $F(R)$ gravity,” Int. J. Mod. Phys. D 23, no.11, 1450088 (2014) [arXiv:1408.5570 [hep-th]].
  * [49] T. Multamaki and I. Vilja, “Static spherically symmetric perfect fluid solutions in f(R) theories of gravity,” Phys. Rev. D 76, 064021 (2007) [arXiv:astro-ph/0612775 [astro-ph]].
  * [50] G. G. L. Nashed and E. N. Saridakis, “New rotating black holes in nonlinear Maxwell $f({\mathcal{R}})$ gravity,” Phys. Rev. D 102, no.12, 124072 (2020) [arXiv:2010.10422 [gr-qc]].
  * [51] G. G. L. Nashed and S. Nojiri, “Analytic charged BHs in f(R) gravity,” Phys. Lett. B 820, 136475 (2021) [arXiv:2010.04701 [hep-th]].
  * [52] G. G. L. Nashed and K. Bamba, “Charged spherically symmetric Taub–NUT black hole solutions in $f(R)$ gravity,” PTEP 2020, no.4, 043E05 (2020) [arXiv:1902.08020 [gr-qc]].
  * [53] G. G. L. Nashed and S. Capozziello, “Charged spherically symmetric black holes in $f(R)$ gravity and their stability analysis,” Phys. Rev. D 99, no.10, 104018 (2019) [arXiv:1902.06783 [gr-qc]].
  * [54] R. A. Hurtado and R. Arenas, “Spherically symmetric and static solutions in $f(R)$ gravity coupled with electromagnetic fields,” Phys. Rev. D 102, no.10, 104019 (2020) [arXiv:2002.06059 [gr-qc]].
  * [55] J. A. R. Cembranos, A. de la Cruz-Dombriz and P. Jimeno Romero, “Kerr-Newman black holes in $f(R)$ theories,” Int. J. Geom. Meth. Mod. Phys. 11 (2014) 1450001 [arXiv:1109.4519 [gr-qc]].
  * [56] S. C. Jaryal and A. Chatterjee, “Gravitationally collapsing stars in $f(R)$ gravity,” Eur. Phys. J. C 81, no.4, 273 (2021) [arXiv:2102.08717 [gr-qc]].
  * [57] G. G. L. Nashed and A. Sheykhi, “New black hole solutions in three-dimensional $\mathit{f(R)}$ gravity,” [arXiv:2110.02071 [physics.gen-ph]].
  * [58] E. F. Eiroa and G. Figueroa Aguirre, “Thin shells in F(R) gravity with non-constant scalar curvature,” Eur. Phys. J. Plus 135, no.10, 774 (2020) [arXiv:2005.10694 [gr-qc]].
  * [59] A. de la Cruz-Dombriz, A. Dobado and A. L. Maroto, “Black Holes in f(R) theories,” Phys. Rev. D 80 (2009) 124011 Erratum: [Phys. Rev. D 83 (2011) 029903] [arXiv:0907.3872 [gr-qc]].
  * [60] S. H. Hendi, B. Eslam Panah and S. M. Mousavi, “Some exact solutions of F(R) gravity with charged (a)dS black hole interpretation,” Gen. Rel. Grav. 44, 835 (2012) [arXiv:1102.0089 [hep-th]].
  * [61] E. F. Eiroa and G. Figueroa-Aguirre, “Thin shells in ( 2+1 )-dimensional $F(R)$ gravity,” Phys. Rev. D 103, no.4, 044011 (2021) [arXiv:2011.14952 [gr-qc]].
  * [62] Z. Y. Tang, B. Wang and E. Papantonopoulos, “Exact charged black hole solutions in $D$-dimensions in $f(R)$ gravity,” Eur. Phys. J. C 81, no.4, 346 (2021) [arXiv:1911.06988 [gr-qc]].
  * [63] Z. Y. Tang, B. Wang, T. Karakasis and E. Papantonopoulos, “Curvature scalarization of black holes in f(R) gravity,” Phys. Rev. D 104, no.6, 064017 (2021) [arXiv:2008.13318 [gr-qc]].
  * [64] T. Karakasis, E. Papantonopoulos, Z. Y. Tang and B. Wang, “Exact Black Hole Solutions with a Conformally Coupled Scalar Field and Dynamic Ricci Curvature in $f(R)$ Gravity Theories,” Eur. Phys. J. C 81, 897 (2021) [arXiv:2103.14141 [gr-qc]].
  * [65] T. Karakasis, E. Papantonopoulos, Z. Y. Tang and B. Wang, “Black holes of (2+1)-dimensional $f(R)$ gravity coupled to a scalar field,” Phys. Rev. D 103, no.6, 064063 (2021) [arXiv:2101.06410 [gr-qc]].
  * [66] A. Aragón, P. A. González, E. Papantonopoulos and Y. Vásquez, “Quasinormal modes and their anomalous behavior for black holes in $f(R)$ gravity,” Eur. Phys. J. C 81, no.5, 407 (2021) [arXiv:2005.11179 [gr-qc]].
  * [67] P. Cañate, L. G. Jaime and M. Salgado, “Spherically symmetric black holes in $f(R)$ gravity: Is geometric scalar hair supported ?,” Class. Quant. Grav. 33, no.15, 155005 (2016) [arXiv:1509.01664 [gr-qc]].
  * [68] P. Cañate, “A no-hair theorem for black holes in $f(R)$ gravity,” Class. Quant. Grav. 35, no.2, 025018 (2018)
  * [69] R. M. Wald, “Black hole entropy is the Noether charge,” Phys. Rev. D 48, no.8, R3427-R3431 (1993) [arXiv:gr-qc/9307038 [gr-qc]].
  * [70] O. Bertolami and M. C. Sequeira, “Energy Conditions and Stability in f(R) theories of gravity with non-minimal coupling to matter,” Phys. Rev. D 79, 104010 (2009) [arXiv:0903.4540 [gr-qc]].
  * [71] V. Faraoni, “f(R) gravity: Successes and challenges,” [arXiv:0810.2602 [gr-qc]].
  * [72] V. Faraoni, “Matter instability in modified gravity,” Phys. Rev. D 74, 104017 (2006) [arXiv:astro-ph/0610734 [astro-ph]].
  * [73] L. Pogosian and A. Silvestri, “The pattern of growth in viable f(R) cosmologies,” Phys. Rev. D 77, 023503 (2008) [erratum: Phys. Rev. D 81, 049901 (2010)] [arXiv:0709.0296 [astro-ph]].
  * [74] Y. Zheng and R. J. Yang, “Horizon thermodynamics in $f(R)$ theory,” Eur. Phys. J. C 78, no.8, 682 (2018) [arXiv:1806.09858 [gr-qc]].
  * [75] U. Camci, “Three-dimensional black holes via Noether symmetries,” Phys. Rev. D 103, no.2, 024001 (2021) [arXiv:2012.06064 [gr-qc]].
  * [76] M. Akbar and R. G. Cai, “Thermodynamic Behavior of Field Equations for f(R) Gravity,” Phys. Lett. B 648, 243-248 (2007) [arXiv:gr-qc/0612089 [gr-qc]].
  * [77] C. Zhu and R. J. Yang, “Horizon Thermodynamics in $D$-Dimensional $f(R)$ Black Hole,” Entropy 22, no.11, 1246 (2020) [arXiv:2102.01475 [gr-qc]].
  * [78] S. W. Hawking, “Particle Creation by Black Holes,” Commun. Math. Phys. 43, 199-220 (1975) [erratum: Commun. Math. Phys. 46, 206 (1976)]
  * [79] V. Iyer and R. M. Wald, “Some properties of Noether charge and a proposal for dynamical black hole entropy,” Phys. Rev. D 50, 846-864 (1994) [arXiv:gr-qc/9403028 [gr-qc]].
  * [80] S. Dutta and R. Gopakumar, “On Euclidean and Noetherian entropies in AdS space,” Phys. Rev. D 74, 044007 (2006) [arXiv:hep-th/0604070 [hep-th]].
  * [81] J. D. Brown and J. W. York, Jr., “Quasilocal energy and conserved charges derived from the gravitational action,” Phys. Rev. D 47 (1993), 1407-1419 [arXiv:gr-qc/9209012 [gr-qc]].
  * [82] J. D. Brown, J. Creighton and R. B. Mann, “Temperature, energy and heat capacity of asymptotically anti-de Sitter black holes,” Phys. Rev. D 50 (1994), 6394-6403 [arXiv:gr-qc/9405007 [gr-qc]].


# Reproducibility Companion Paper: Describing Subjective Experiment
Consistency by $p$-Value P–P Plot

Jakub Nawała<EMAIL_ADDRESS>0000-0002-5671-3726 ,  Lucjan Janowski
0000-0002-3151-2944 AGH University of Science and Technology, Institute of
TelecommunicationsKrakówPoland ,  Bogdan Ćmiel  AGH University of Science and
Technology, Department of Mathematical Analysis, Computational Mathematics and
Probability MethodsKrakówPoland ,  Krzysztof Rusek  0000-0003-4336-7841 AGH
University of Science and Technology, Institute of
TelecommunicationsKrakówPoland ,  Marc A. Kastner  0000-0002-9193-5973
National Institute of InformaticsTokyoJapan and  Jan Zahálka
0000-0002-6743-3607 Czech Technical University in PraguePragueCzech Republic

(2021)

###### Abstract.

In this paper we reproduce experimental results presented in our earlier work
titled “Describing Subjective Experiment Consistency by $p$-Value P–P Plot”
that was presented in the course of the 28th ACM International Conference on
Multimedia. The paper aims at verifying the soundness of our prior results and
helping others understand our software framework. We present artifacts that
help reproduce tables, figures and all the data derived from raw subjective
responses that were included in our earlier work. Using the artifacts we show
that our results are reproducible. We invite everyone to use our software
framework for subjective responses analyses going beyond reproducibility
efforts.

Quality of Experience; Subjective Experiment; Consistency; Reproducibility;
P–P Plot; Subjective Data

††copyright: rightsretained††ccs: Human-centered computing User models††ccs:
Human-centered computing User studies††ccs: Social and professional topics
Quality assurance††journalyear: 2021††conference: Proceedings of the 29th ACM
International Conference on Multimedia; October 20–24, 2021; Virtual Event,
China.††booktitle: Proceedings of the 29th ACM International Conference on
Multimedia (MM ’21), October 20–24, 2021, Virtual Event, China††doi:
10.1145/3474085.3477935††isbn: 978-1-4503-8651-7/21/10

##  1\. Artifacts Organisation

The artifacts are available for download from the following GitHub repository:
https://github.com/Qub3k/subjective-exp-consistency-check (Nawała et al.,
2021). Its file structure is presented in Fig. 1.

The reproduce.py script is the entry point, serving as a master script
governing the reproducibility process. The script’s help message is designed
so as to provide sufficient information necessary to understand framework’s
operation. The following snippet shows how to invoke the reproduce.py’s help
message using the command-line interface (CLI).

⬇

$ python3 reproduce.py -h

Should the help message be insufficient, the reader is encouraged to take a
look at the README.md file. More specifically, its “Reproducibility” section
provides further guidance on how to use the framework. At last, since the
framework is entirely open-sourced, its operation can be investigated by
looking at the source code.

.

├── .gitignore

├── G_test_on_real_data.py

├── G_test_results.csv

├── LICENSE

├── README.md

├── _logger.py

├── bootstrap.py

├── figures/

├── friendly_gsd.py

├── gsd.py

├── gsd_prob_grid.pkl

├── hdtv1_exp1_scores_pp_plot_ready.csv

├── p_value_pp_plot_in-depth_tutorial.ipynb

├── probability_grid_estimation.py

├── qnormal.py

├── qnormal_prob_grid.pkl

├── reproduce.py

├── reproducibility/

├── requirements.txt

├── subjective_quality_datasets.csv

└── typical_vs_atypical.csv

Figure 1. Structure of the GitHub repository.

Another two important files in the repo are: (i)
subjective_quality_datasets.csv and (ii) G_test_results.csv. The former one
includes raw subjective data that is processed in the original paper (Nawała
et al., 2020). The most important output of this processing is the
G_test_results.csv file. It includes results of running computationally
intensive bootstrapped version of the G-test of goodness-of-fit (cf. Fig. 2
from (Nawała et al., 2020)). Effectively, recreating these results is the most
significant part of the reproducibility efforts.

##  2\. Setup and Execution

We recommend to create a separate Python virtual environment and install there
all the dependencies listed in the requirements.txt file. Importantly, Python
version 3.7 or newer is required. When the required packages are installed the
reproduce.py script can be run through the CLI, as shown below.

⬇

$ python3 reproduce.py [-h] [-n N] {scenario}

The {scenario} place-holder identifies the execution scenario. As of the time
of writing this paper there are five such scenarios. They are identified by
subsequent integers from the range 1–5. The following list provides details on
each scenario.

  1. (1)

Reproduce the original experiments using existing G-test results (i.e., the
G_test_results.csv file). This executes immediately, but in principle does not
reproduce the most important piece of the data presented in the original
paper.

  2. (2)

Reproduce only these G-test results that are necessary to draw Fig. 3 from the
original paper. This already takes a significant amount of time to run
(approximately nine days according to our internal tests). This scenario makes
sense since Fig. 3 is a central part of the discussion presented in the
original paper.

  3. (3)

Reproduce all G-test results. This scenario reproduces from the scratch all
the results presented in the original paper. As such this is the most
important scenario of the reproducibility framework. However, according to our
internal tests, this takes around 21 days to run.

  4. (4)

Run the G-test for randomly selected N stimuli (cf. the -n N argument of the
call to reproduce.py above). This scenario can be used to quickly check the
correctness of N randomly selected results from the G_test_results.csv file.
We note that according to our internal tests processing one stimulus takes
about four to seven minutes.

  5. (5)

Reproduce the probability grids of the Generalised Score Distribution (GSD)
and Qunatized Normal (QNormal) models (cf. Sec. GSD Parameters Estimation in
(Nawała et al., 2020)). Those grids are internally used when running the
G-test.

Since our framework is implemented in Python in its entirety, it can be run on
any platform.111We confirmed framework’s operation on the three popular
operating systems: Windows 10, Mac OS 10.15 and Ubuntu Linux 18.04.05. All
execution times mentioned in this paper were measured using the following
hardware setup: Intel Core i3-8130U CPU, 16 GB of 2400 MHz RAM and 256 GB SSD
disk (Lenovo LENSE30256GMSP34MEAT3TA).

###  2.1. Batch Processing Capability

Reproducing complete G-test results takes a significant amount of time when
done on a single machine. Thus, we make available the batch processing
friendly variation of the G-test running framework. It can be used to run
multiple parallel instances of the G-test, each running on a different chunk
of the input data. Crucially, the reproduce.py script does not support batch
processing, as this would greatly complicate its structure. Instead, the
friendly_gsd.py script must be used. Still, both reproduce.py and
friendly_gsd.py scripts will produce the same output. For more information we
refer the reader to the “Batch Processing” section of the README.md file.

##  3\. Experiments

Execution scenarios 1, 2 and 3 (cf. Sec 2) reproduce all tables and figures
presented in the original paper. However, only scenario 3 reproduces entirely
the data used to generate figures and tables. The other two scenarios either
use the data processing outputs from the original paper (scenario 1) or
reproduce only a part of the data (scenario 2).

Since the G-test running framework internally uses pre-calculated probability
grid of the GSD model, to achieve the complete reproducibility (i.e., being
able to achieve the same results when being provided only with raw subjective
data contained in the subjective_quality_datasets.csv file) one has to run
execution scenario 5 as well.

All in all, both scenario 3 and scenario 5 must be executed to check results
reproducibility. The snippet below shows two calls to the reproduce.py script
that fulfil this goal.

⬇

$ python3 reproduce.py 3

$ python3 reproduce.py 5

The first call produces four types of output: (i) CSV files with reproduced
tables contents, (ii) PDF files with reproduced figures, (iii) a CSV file with
G-test results and (iv) tables contents written to the standard output. Fig. 2
shows files that are created as a result of this call. Significantly, the
reproduced figures have the same formatting as the one used in the original
paper (cf. Fig. 3).

We also note here that the G-test used in the framework has randomness built
into it. Thus, reproduced results will not exactly match the ones generated
for the purposes of the original paper. This is because we use the
bootstrapped version of the G-test that internally generates 10,000 synthetic
random samples based on each observed sample. We refer readers interested into
more details on the topic to section “In-depth Tutorial about Generating
$p$-Value P–P Plots for Your Subjective Data” of the README.md file in the
GitHub repository.

p-value_pp-plot_fig_one_a.pdf

p-value_pp-plot_fig_one_b.pdf

table_one_score_distribution.csv

table_two_pvals.csv

p-value_pp-plot_HDTV1_fig_three.pdf

p-value_pp-plot_ITS4S2_fig_three.pdf

p-value_pp-plot_ITS4S_AGH_fig_three.pdf

p-value_pp-plot_AGH_NTIA_fig_three.pdf

table_three_five_lowest_pvalue_res_its4s_agh.csv

table_four_five_lowest_pvalue_res_its4s2.csv

G_test_on_subjective_quality_datasets_chunk000_of_001.csv

Figure 2. Files generated as a result of running execution scenario 3. Figure
3. Reproduced Fig. 1b from the original paper. Please note that the formatting
of the figure is the same as in the original paper.

Running scenario 5 (the second call from the snippet above) creates two files
only: (i) reproduced_gsd_prob_grid.pkl and (ii)
reproduced_qnormal_prob_grid.pkl. They are pickled Python objects and more
specifically, pickled Pandas DataFrames. They can be manually compared with
the corresponding pickle files from the original paper: gsd_prob_grid.pkl and
qnormal_prob_grid.pkl.

We note that our framework reproduces Fig. 1a and 1b from the original paper
using ready-made CSV files.222The CSV files are available in the
reproducibility folder in the GitHub repository. This is because the two plots
use synthetic data. Differently put, the plots were not generated from
subjective responses gathered during any real-life subjective experiment.

##  4\. Reproducibility Efforts

The code is open-source, well readable, and sufficiently commented. All
results of the original paper are easily reproducible, directly generating the
figures used in the original paper.

Since the submitted software was of high quality in the first version already,
the reproducibility review has mostly consisted of minor fixes and ease-of-use
improvements. Firstly, the review process resulted in the batch processing
mode of the software being more accessible to the user. This is essential, as
the sequential mode runs for days to reproduce all results. Secondly, the
authors have fixed the random stimuli scenario that did not fail gracefully
when the -n parameter was omitted (now the default is 3 stimuli). Finally, the
authors have been very responsive not only to the reviewer comments, but also
to general GitHub user comments.

All of the above aspects lead us to believe this is a software worthy of the
reproducibility badge.

##  5\. Invitation

Although this paper focuses on reproducibility, our GitHub repository (Nawała
et al., 2021) was created to help others use our framework in future analyses
as well. We invite everyone, who has at hand a data set of subjective
responses, to use the framework. It can test how well the GSD models
subjective responses distribution and provide insights into subjective
experiment consistency. For more details we refer the reader to our original
paper (Nawała et al., 2020) and the README.md file in the repository.

###### Acknowledgements.

This work was supported by the Polish Ministry of Science and Higher Education
with the subvention funds of the Faculty of Computer Science, Electronics and
Telecommunications of AGH University and by the PL-Grid Infrastructure.
Furthermore, the research leading to these results has received funding from
the Norwegian Financial Mechanism 2014–2021 under project 2019/34/H/ST6/00599.
We also kindly acknowledge the help of Franz Hahn, who identified and helped
us resolve an issue with our source code.

## References

  * (1)
  * Nawała et al. (2020) Jakub Nawała, Lucjan Janowski, Bogdan Ćmiel, and Krzysztof Rusek. 2020.  Describing Subjective Experiment Consistency by p-Value P-P Plot. In _Proceedings of the 28th ACM International Conference on Multimedia_. ACM, New York, NY, USA, 852–861.  https://doi.org/10.1145/3394171.3413749
  * Nawała et al. (2021) Jakub Nawała, Lucjan Janowski, Bogdan Ćmiel, and Krzysztof Rusek. 2021.  _subjective-exp-consistency-check GitHub Repository_.  https://github.com/Qub3k/subjective-exp-consistency-check