January 2024 IPMU24-0001 version 2

Improved model of large-field inflation with primordial
black hole production in Starobinsky-like supergravity

Ryotaro Ishikawa a and Sergei V. Ketov a,b,c

a Department of Physics, Tokyo Metropolitan University

1-1 Minami-ohsawa, Hachioji-shi, Tokyo 192-0397, Japan

b Research School of High-Energy Physics, Tomsk Polytechnic University

Tomsk 634028, Russian Federation

c Kavli Institute for the Physics and Mathematics of the Universe (WPI)

The University of Tokyo Institutes for Advanced Study, Kashiwa 277-8583, Japan

<EMAIL_ADDRESS><EMAIL_ADDRESS>
Abstract

A viable model of large-field (chaotic) inflation with efficient production of
primordial black holes is proposed in Starobinsky-like (modified) supergravity
leading to the ”no-scale-type” Kähler potential and the Wess-Zumino-type
(”renormalizable”) superpotential. The cosmological tilts are in good (within
$1\sigma$) agreement with Planck measurements of the cosmic microwave
background radiation. In addition, the power spectrum of scalar perturbations
has a large peak at smaller scales, which leads to a production of primordial
black holes from gravitational collapse of large perturbations with the masses
about $10^{17}$ g. The masses are beyond the Hawking (black hole) evaporation
limit of $10^{15}$ g, so that those primordial black holes may be viewed as
viable candidates for a significant part or the whole of the current dark
matter. The parameters of the superpotential were fine-tuned for those
purposes, while the cubic term in the superpotential is essential whereas the
quadratic term should vanish. The vacuum after inflation (relevant to
reheating) is Minkowskian. The energy density fraction of the gravitational
waves induced by the production of primordial black holes and their frequency
were also calculated in the second order with respect to perturbations.

##  1 Introduction

Starobinsky model of inflation [1] based on modified gravity with the higher
spacetime scalar curvature against the Einstein-Hilbert action remains popular
today despite its origin in 1980. It is because of its conceptual
attractiveness (using only gravitational interactions consistent with General
Relativity), excellent agreement with recent measurements of Cosmic Microwave
Background (CMB) radiation [2, 3], and its simplicity (having only one free
parameter that becomes fixed by the known amplitude of CMB scalar
perturbations).

The Starobinsky model is distinguished in modified gravity because the
$R^{2}$-gravity action (with a positive dimensionless coefficient in four
spacetime dimensions) dominating over the Einstein-Hilbert action during
inflation is the only ghost-free action in a quadratically generated gravity,
being also scale-invariant, which is the origin of flatness of the effective
inflaton scalar potential during slow-roll inflation. The inflaton scalar in
the Starobinsky model is a physical mode of the higher-derivative gravity,
which becomes manifest after a transformation of the Starobinsky gravity to
the Einstein frame (scalar-tensor gravity). Hence, Starobinsky’s inflaton can
be viewed as the Nambu-Goldstone mode associated with spontaneous breaking of
scale-invariance of the $R^{2}$ gravity during inflation [4]. It implies the
$R^{2}$-term with a proper coefficient must be present in any viable model of
inflation based on modified gravity. The magnitude of the inflationary scale
predicted by the Starobinsky model is approximately five orders lower than the
Planck scale that is the ultra-violet cutoff scale in the Starobinsky model,
which assures stability of inflation against quantum gravity corrections, see
recent reviews [5, 6, 7] for details.

The Hubble value $H\sim{\cal O}(10^{14})$ GeV during Starobinsky inflation is
far beyond the electro-weak scale. On the Hubble scale gravity becomes truly
important, strongly coupled and (presumably) unified with other fundamental
physical interactions based on supergravity [8]. Viable extensions of the
Starobinsky model of inflation in modified gravity to modified supergravity
were pioneered in Refs. [9, 10, 11, 12]. Those supergravity models can be
upgraded in order to include a formation of Primordial Black Holes (PBH)
during Starobinsky inflation, while those PBH may be an essential part (or the
whole) of the current dark matter (DM), see Refs. [13, 14] for recent reviews.
In particular, the Cecotti-Kallosh supergravity model [12] of inflation in the
Starobinsky-like supergravity was generalized in Ref. [15] by adding a
holomorphic function to the superpotential, which led to an efficient
production of PBH with the masses beyond the Hawking (black hole) evaporation
limit of $10^{15}$ g, though in marginal agreement with the Planck-measured
CMB tilt of scalar perturbations within $3\sigma$ but outside $1\sigma$
accuracy. On the other hand, when demanding good (within $1\sigma$) agreement
with CMB, the masses of produced PBH appeared to be under the Hawking limit in
the supergravity model of Ref. [15] and, hence, those PBH would not survive at
present. In this paper we found a remedy to this problem.

The paper is organized as follows. The relevant background information
available in the literature is summarized in Section 2 that represents our
setup. Our new model is introduced in Section 3. Our results for inflation and
primordial black holes production are given in Section 4. The gravitational
waves induced by the PBH production are studied in Section 5. Our conclusion
is Section 6.

Our paper is based on a different ansatz for the superpotential when compared
to that of Ref. [15], and our new results are not an extension or upgrade of
the previously obtained ones, though we used the same methods. We skipped a
historical overview and many standard equations describing inflation and PBH
together with observational constraints on them because of numerous
comprehensive reviews available in the literature, and focused on our new
results with a limited list of references. We use the units with
$\hbar=c=M_{\rm Pl}=1$ throughout this paper.

##  2 Setup and background state-of-the-art

Modified gravity theories are generally-covariant non-perturbative extensions
of Einstein-Hilbert (EH) gravity theory by the higher-order terms. The higher
order terms are irrelevant in the Solar system but are relevant in the high-
curvature regimes (inflation, black holes) and for large cosmological
distances (dark energy). Any modified gravity Lagrangian has the higher-
derivatives and generically suffers from Ostrogradsky instability and ghosts.
However, there are exceptions. In a quadratically generated gravity with
respect to the spacetime curvature, the only ghost-free term is given by the
Ricci scalar squared, $R^{2}$, with a positive coefficient. It leads to the
Starobinsky model of modified gravity with the action

$S_{\rm
Star.}={{1}\over{2}}\int\mathrm{d}^{4}x\sqrt{-g}\left(R+{{1}\over{6M^{2}}}R^{2}\right)=\int\mathrm{d}^{4}x\sqrt{-g}~{}F(R)~{},$
(1)

having the only parameter $M\approx 1.3\cdot 10^{-5}$ whose value is fixed by
the CMB amplitude (WMAP normalization). 111We use the spacetime signature
$(-,+,+,+,)$.

In the high-curvature regime relevant to inflation, the EH term can be ignored
and the pure $R^{2}$-action becomes scale-invariant. The Starobinsky gravity
has the special attractor solution in the Friedmann-Lemaitre-Robinson-Walker
(FLRW) universe with the Hubble function [1]

$H(t)\approx\left({{M}\over{6}}\right)^{2}(t_{\rm end}-t)~{},$ (2)

for $M(t_{\rm end}-t)\gg 0$. This solution spontaneously breaks the scale
invariance of the $R^{2}$-gravity and, hence, implies the existence of the
associated Nambu-Goldstone boson called scalaron. Scalaron can be revealed by
rewriting the Starobinsky action to the quintessence form by the field
redefinition (see, e.g., Ref. [16])

$\varphi=\sqrt{{{3}\over{2}}}\ln F^{\prime}(\chi)\quad{\rm and}\quad
g_{\mu\nu}\to 2F^{\prime}(\chi)g_{\mu\nu}~{},\quad\chi=R~{},$ (3)

which leads to

$S[g_{\mu\nu},\varphi]={{1}\over{2}}\int\mathrm{d}^{4}x\sqrt{-g}R-\int\mathrm{d}^{4}x\sqrt{-g}\left[{\textstyle{1\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{2}$}}}}g^{\mu\nu}\partial_{\mu}\varphi\partial_{\nu}\varphi+V(\varphi)\right]~{},$

with the potential

$V(\varphi)={{3}\over{4}}M^{2}\left[1-\exp\left(-\sqrt{{\textstyle{2\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}}\varphi\right)\right]^{2}~{}.$ (4)

This potential is suitable for describing slow-roll inflation with scalaron
$\varphi$ as the canonical inflaton of mass $M$ due to the infinite plateau
with a positive height, and leads to the well-known predictions for the
cosmological tilt $n_{s}$ of scalar perturbations and the tensor-to-scalar
ratio $r$ for CMB,

$n_{s}\approx 1-{{2}\over{N}}~{},\quad r\approx{{12}\over{N^{2}}}~{}~{},$ (5)

which fit current CMB measurements [2, 3]

$n_{s}=0.9649\pm 0.0042~{}(68\%~{}{\rm CL})\quad{\rm and}\quad
r<0.032~{}(95\%~{}{\rm CL})~{},$ (6)

for the e-foldings number $N=55\pm 10$, defined by $N=\int^{t_{\rm
end}}_{t}H(t)dt$.

A locally supersymmetric extension of the action (1) is called Starobinsky
(modified) supergravity. It has only the first and second powers of $R$ but
also includes other fields of the supergravity multiplet, see Ref. [17] for
its general action. Similarly to Eq. (3), the Starobinsky supergravity action
can be transformed to the (dual or equivalent) standard Einstein supergravity
action that has only the first power of $R$ and is coupled to two chiral
matter supermultiplets [18]. Those chiral matter supermultiplets are called
$T$ that includes a complex physical scalar field (real inflaton and its
pseudo-scalar superpartner called sinflaton), and $C$ that includes goldstino
arising due to spontaneous supersymmetry breaking during inflation. Goldstino
is eaten up by gravitino that becomes massive (super-Higgs effect). The Kähler
potential and the superpotential of the matter $(T,C)$ have the special (”no-
scale”) structure that can be used for describing Starobinsky inflation [8].
Further generalizations of the Kähler potential and the superpotential,
keeping the no-scale structure and describing Starobinsky inflation in
supergravity, are called Starobinsky-like in the literature. PBH production
during inflation in the Starobinsky (modified) supergravity was investigated
in Refs. [19, 20]. PBH production in the (modified) Cecotti-Kallosh model [12]
of inflation in Starobinsky-like supergravity was initiated in Ref. [15].

The Starobinsky-like Einstein supergravity studied in Ref. [15] has the
following ”no-scale” Kähler potential and the superpotential:

$K=-3\ln\left(T+\bar{T}-|C|^{2}+\zeta{{|C|^{4}}\over{T+\bar{T}}}\right)~{},\quad
W=MC(T-1)+g(T),$ (7)

which include the new real parameter $\zeta$ and arbitrary holomorphic
function $g(T)$. The model of Ref. [12] arises in the case of $g(T)=0$. The
case of $\zeta=g(T)=0$ defines the ”no-scale” structure. However, it does not
describe viable inflation because of an instability in the space of scalars
(too small $N$) so that the $\zeta$-dependent term is required for Starobinsky
inflation near $C=0$ with the effective inflaton mass $M$. Therefore, it is
possible to set $C=0$ in equations of motion, which greatly simplifies
calculations of inflationary dynamics and PBH production.

The main idea of Ref. [15] was to find an analytic function $g(T)$ in the
superpotential, in order to describe PBH production during Starobinsky
inflation by demanding the existence of the Ultra-Slow-Roll (USR) phase
between two Slow-Roll (SR) phases of inflation, which leads to a large peak in
the power spectrum of density perturbations that later collapse to PBH. In our
models (7), the USR phase arises via the so-called ”iso-curvature pumping
mechanism” due to tachyonic instabilities in multi-field inflation, producing
large iso-curvature perturbations that act as the source of large density
perturbations that later gravitationally collapse into PBH, see Refs. [21, 22]
for details.

It is worth mentioning that the existence of an USR phase in the supergravity
model (7) is by no means guaranteed. First, one has to suppress extra scalars
in order to get single-field SR inflation (this problem was solved in Ref.
[15]). Second, one has to get good agreement (within $1\sigma$) with the
Planck data in Eq. (5). Third, one has to produce a large peak in the power
spectrum, which would generate PBH with masses beyond the Hawking evaporation
limit of $10^{15}$ g, so that those PBH would survive in the current universe
and may form dark matter. Third, as was demonstrated in Ref. [15], a linear
combination of linear and exponential functions $g$ of $T$ does not do the job
for any choice of the parameters. It was the reason why a quadratic polynomial
$g(T)$ was considered in Ref. [15] where it was found that fine-tuning of the
parameters in a quadratic superpotential does lead to viable inflation and PBH
production beyond the Hawking limit, though in agreement with the Planck-
measured $n_{s}$ within $3\sigma$ but outside $1\sigma$. It follows from this
fact that the model proposed and studied in Ref. [15] may be easily falsified
by more precise measurements of the $n_{s}$ tilt in the near future.

Required calculations and scanning of the parameter space in the models (7)
can only be performed numerically by using Mathematica and a dedicated code on
the case-by-case basis. We used the transport method and the open code
provided in Ref. [23].

It is more practical (for numerical calculations) to make Cayley
transformation from the complex variable $T$ defined in the upper half-plane
to another complex variable $Z$ defined on Poincaré disk, and redefine $C$ in
favor of $S$ as follows:

$T={{1+Z}\over{1-Z}},\quad C={{\sqrt{2}S}\over{1-Z}}~{}~{}.$ (8)

The inverse transformation reads

$Z={{T-1}\over{1+T}},\quad S={{\sqrt{2}C}\over{1+T}}~{}~{}.$ (9)

The scalar part of the Lagrangian in the model (7) with $C=S=0$ is given by

$\mathcal{L}=-{{3}\over{\left(1-|Z|^{2}\right)^{2}}}\partial_{\mu}Z\partial^{\mu}\bar{Z}-V$
(10)

with the scalar potential

$\displaystyle V=$ $\displaystyle\
{\textstyle{M^{2}\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}{{|Z|^{2}\left|1-Z\right|^{2}}\over{\left(1-|Z|^{2}\right)^{2}}}+{\textstyle{1\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{24}$}}}}{{|1-Z|^{6}}\over{1-|Z|^{2}}}\left|{{dg}\over{dZ}}\right|^{2}-{\textstyle{1\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{8}$}}}}{{|1-Z|^{4}}\over{\left(1-|Z|^{2}\right)^{2}}}\left[(1-Z)^{2}{{dg}\over{dZ}}\bar{g}+(1-\bar{Z})^{2}{{d\bar{g}}\over{d\bar{Z}}}g\right]$
(11)

in terms of complex inflaton $Z$ and its complex conjugate $\bar{Z}$, with a
holomorphic function $g(Z)$ and its conjugate $\bar{g}(\bar{Z})$. The
canonical inflaton $\varphi$ appears in the parametrization

$Z=re^{i\theta}={\rm{tanh}}{{\varphi}\over{\sqrt{6}}}\,e^{i\theta}$ (12)

because the kinetic term in this parametrization takes the form

$\mathcal{L}=-{\textstyle{1\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{2}$}}}}(\partial_{\mu}\varphi)^{2}-{\textstyle{3\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{4}$}}}}{\rm{sinh}}^{2}{{2\varphi}\over{\sqrt{6}}}(\partial_{\mu}\theta)^{2}$
(13)

with sinflaton $\theta$. The two-dimensional target (field) space of this non-
linear sigma-model has hyperbolic geometry with a constant negative curvature
equal to $-4/3$.

As was demonstrated in Ref. [15] in the case of $g=0$, sinflaton $\theta$ is
stabilized during SR inflation because of its heavy mass beyond the Hubble
value, so that sinflaton dynamics and related iso-curvature perturbations can
be ignored during SR inflation. However, it does not apply to the USR phase
needed for PBH production, because of a tachyonic instability in the sinflaton
direction for special choices of $g(Z)$ function, see next Sections. Given
$\theta=g(Z)=0$ the scalar potential (11) in the parametrization (12) reduces
to the Starobinsky potential (4), see Section 2 of Ref. [15] for a detailed
derivation.

When the function $g\neq 0$, both scalars should be taken into account. The
equations of motion in the flat FLRW universe are

$\displaystyle
0=\ddot{\varphi}+3H\dot{\varphi}-\sqrt{{\textstyle{3\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{8}$}}}}}\sinh\left(\sqrt{{\textstyle{8\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}}\varphi\right)\dot{\theta}^{2}+\partial_{\varphi}V~{}~{},$
$\displaystyle
0=\ddot{\theta}+3H\dot{\theta}+2\sqrt{{\textstyle{2\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}}{\mathrm{coth}}{\left(\sqrt{{\textstyle{2\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}}\varphi\right)}\dot{\varphi}\dot{\theta}+{\textstyle{2\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}\mathrm{csch}^{2}\left(\sqrt{{\textstyle{2\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}}\varphi\right)\partial_{\theta}V~{}~{},$
(14) $\displaystyle 0={\textstyle{1\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{2}$}}}}\dot{\varphi}^{2}+{\textstyle{3\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{4}$}}}}\sinh^{2}\left(\sqrt{{\textstyle{2\over\vphantom{2}\smash{\raise
0.60275pt\hbox{$\scriptstyle{3}$}}}}}\varphi\right)\dot{\theta}^{2}-3H^{2}+V~{}~{},$

with the potential

$\displaystyle V=$
$\displaystyle{{M^{2}}\over{3}}{{r^{2}(1-2r\cos\theta+r^{2})}\over{(1-r^{2})^{2}}}+{{1}\over{24}}{{(1-2r\cos\theta+r^{2})^{3}}\over{1-r^{2}}}\left|{{\mathrm{d}g}\over{\mathrm{d}Z}}\right|^{2}$
$\displaystyle-{{1}\over{8}}{{(1-2r\cos\theta+r^{2})^{2}}\over{(1-r^{2})^{2}}}\left[(1-re^{i\theta})^{2}{{\mathrm{d}g}\over{\mathrm{d}Z}}\bar{g}+(1-re^{-i\theta})^{2}{{\mathrm{d}\bar{g}}\over{\mathrm{d}\bar{Z}}}g\right]~{}~{}.$
(15)

It is straightforward to derive the equations for perturbations by varying the
equations of motion with respect to $\varphi$ and $\theta$ with the potential
(11). However, those equations are long and not very illuminating, so that we
do not quote them here.

##  3 New ansatz for $g$-function

In this paper we investigate the Starobinsky-like supergravity models (7) with
more general $g$-functions having the form

$\displaystyle g(Z)=M(g_{0}+g_{1}Z+g_{2}Z^{2}+g_{3}Z^{3})$ (16)

and four parameters $(g_{0},g_{1},g_{2},g_{3})$. Equation (16) is the most
general renormalizable (Wess-Zumino) superpotential. Unlike Ref. [15], we
assume $g_{3}\neq 0$. Actually, the latter condition leads to a rather
complicated scalar potential from Eqs. (11) and (2).

The parameter $g_{0}$ is fixed in terms of the other parameters by demanding a
Minkowski vacuum in the potential, which amounts to solving the equations

$\displaystyle\partial_{\varphi}V(g_{0})=\partial_{\theta}V(g_{0})=V(g_{0})=0~{}.$
(17)

Next, we impose the desired inflationary dynamics by demanding the effective
single-field Starobinsky inflation during slow-roll, a saddle point in the
potential below the first inflationary plateau and a sharp turn of the
inflationary trajectory after the saddle point toward another plateau, in
order to get three-phase (SR-USR-SR) inflation and PBH production during the
USR phase. All that leads to restrictions on the free parameters
$(g_{1},g_{2},g_{3})$, which can be satisfied without significant fine-tuning.
We illustrate those features by a sample solution to the equations of motion
(14), which is shown in Fig. 1. The realized scenario is as follows: (i)
during the first SR phase of inflation driven by scalaron $\varphi$, sinflaton
$\theta$ is suppressed, (ii) when approaching the saddle point, the effective
mass squared of sinflaton becomes negative, which signals a tachyonic
instability leading to large iso-curvature perturbations fueling large density
perturbations during the USR phase, (iii) after the USR phase, the second SR
phase of single-field inflation begins with the sinflaton as the driver having
the effective positive mass squared. The double SR inflation implies the
existence of two plateaus in the Hubble function $H(t)$.

(a) |  (b)
---|---
Figure 1: The shape of the sample potential with the inflationary trajectory
(a), and the corresponding solution to the equations of motion with the
parameters $(g_{0},g_{1},g_{2},g_{3})\approx(14.3,-21,0,16)$ and the initial
conditions $\varphi_{0}=5.5,~{}\theta_{0}=0$ with $\theta^{\prime}=1.4\times
10^{-6}$ (b). The blue part of the trajectory corresponds to the first SR
stage of the inflation and the red part corresponds to the second SR stage of
inflation in (a). The USR phase between the SR phases is short (a few
e-folds), being almost invisible on these plots. The blue line describes
$\varphi$-evolution and the red line describes $\theta$-evolution in (b).

The SR multi-field inflation parameters $\epsilon$ and $\eta_{\Sigma\Sigma}$
are defined by [23, 24]

$\displaystyle\epsilon=-{{\dot{H}}\over{H^{2}}}~{},\quad\eta_{\Sigma\Sigma}=\left|{{\mathcal{M}_{B}^{A}\Sigma_{A}\Sigma^{B}}\over{V}}\right|~{},\quad\phi^{A}=\\{\varphi,\theta\\}~{},\quad
A,B,\ldots=1,2~{},$ (18) $\displaystyle{\rm
where}\quad\Sigma^{A}={{\dot{\phi}^{A}}\over{|\dot{\phi}|}}~{},\quad\mathcal{M}_{B}^{A}=G^{AC}\nabla_{B}\partial_{C}V-\mathrm{R}^{A}_{~{}CDB}\dot{\phi}^{C}\dot{\phi}^{D}~{},$
(19)

in terms of the metric $G_{AB}$ in the scalar field space with the related
Christoffel symbols $\Gamma^{A}_{BC}$ and the Riemann-Christoffel curvature
$\mathrm{R}^{A}_{~{}CDB}$, see Eq. (13). The iso-curvature parameter is
defined by [23, 24]

$\displaystyle\eta_{\Omega\Omega}={{\mathcal{M}_{B}^{A}\Omega_{A}\Omega^{B}}\over{V}}~{},\quad\mathrm{where}\quad\Omega^{A}={{{\omega}^{A}}\over{|{\omega}|}}~{},\quad\omega=\dot{\Sigma^{A}}+\Gamma^{A}_{BC}\Sigma^{B}\dot{\phi}^{C}~{}.$
(20)

The Hubble function during SR inflation is determined by the potential via
Friedmann equation. The CMB tilts (the scalar spectral index $n_{s}$ and the
tensor-to-scalar ratio $r$) are simply related to the SR parameters in the
first order with respect to perturbations as

$\displaystyle n_{s}=1-6\epsilon+2\eta_{\Sigma\Sigma}\quad{\rm and}\quad r\leq
16\epsilon~{},$ (21)

which are to be evaluated at the CMB pivot scale
$k_{*}=0.05~{}\mathrm{Mpc}^{-1}$.

The USR phase appears when $\epsilon$ becomes very small$,\epsilon\to+0$,
which leads to a large enhancement (peak) in the power spectrum of scalar
perturbations. The SR conditions are violated in the USR phase with
$\eta_{\Sigma\Sigma}\equiv\eta>1$.

The start of the second SR stage of inflation is determined by the time when
the parameter $\eta$ goes back to one (or when the parameter $\epsilon$ first
reaches its maximum), while the end of the second stage is determined by the
time when the parameter $\epsilon$ reaches one first. We set the total
duration of inflation to $70$ e-folds. The target value of the second SR stage
of inflation is chosen to be about 20 e-folds.

We used Mathematica for numerically computing the scalar potential and getting
solutions to the equations of motion by randomly taking initial conditions for
the scalars and scanning the parameter space $(g_{1},g_{2},g_{3})$. The
standard transport method and the Mathematica package (code) of Ref. [23] were
used to numerically calculate the power spectrum of perturbations in our
models. We refer to Ref. [23] for more details.

A dependence of our solutions upon initial conditions on $\varphi$ was found
to be weak, which was not surprising because the Starobinsky inflationary
solution is an attractor. Contrary to that, a dependence of our solutions upon
initial conditions on sinflaton $\theta$ was found to be strong because
initial conditions $\theta_{0}$ and ${\theta^{\prime}}_{0}$ determine the
value of the velocity $\theta^{\prime}$ at the saddle point, while the
$\theta^{\prime}$ appears to be the key parameter for the power spectrum
enhancement. 222The primes denote the time derivatives here. When approaching
the saddle point, the velocity $\theta^{\prime}$ becomes very small, being out
of control, so that we consider its value as an additional (quantum)
parameter, similarly to a ’kick’ caused by quantum diffusion [25].

In addition, we had to avoid approaching $\theta=0$ in our calculations
because of failure of numerical calculations of the power spectrum.
Presumably, it was caused by the presence of resonances on sub-horizon scales,
when a massive scalar oscillates at the bottom of the potential [26, 27, 28].
Because of that complication, the power spectrum was computed separately for
each stage of inflation by using independent initial conditions, see Section 6
of Ref. [15] for more details.

##  4 Results

The most time-consuming part of our investigation was scanning the parameter
space $(g_{1},g_{2},g_{3})$ in a search for proper potential and deriving the
power spectrum and CMB observables $n_{s}$ and $r$ for some small values of
$\theta^{\prime}$ at the saddle point. As a result, we found the parameters
$g_{1}$ and $g_{3}$ have to be non-vanishing and fine-tuned, whereas the
parameter $g_{2}$ should essentially vanish. Actually, the parameters $g_{1}$
and $g_{3}$ were also related to each other due to the required shape of the
potential, so that in practice we had to increase both $g_{1}$ and $g_{3}$ and
cancel the negative effect of $g_{2}$ by a positive impact of $g_{1}$ during
animation of the potential by varying the parameters. Our best findings are
collected in Table 1.

$g_{1}$ | $g_{2}$ | $g_{3}$ | $g_{0}$ | $n_{s}$ | $r$ | $\theta^{\prime}$ | $\Delta N_{2}$ | $M_{PBH}~{}(g)$
---|---|---|---|---|---|---|---|---
$-20.0$ | $0$ | $15.0$ | $13.9$ | $0.96081$ | $0.004085$ | $4.5\times 10^{-5}$ | $19.7$ | $1.3\times 10^{17}$
$-20.4$ | $0$ | $15.4$ | $14.1$ | $0.96079$ | $0.004078$ | $1.1\times 10^{-5}$ | $19.9$ | $2.1\times 10^{17}$
$-20.0$ | $0.2$ | $15.0$ | $13.8$ | $0.96088$ | $0.004067$ | $1.1\times 10^{-5}$ | $19.7$ | $1.4\times 10^{17}$
$-20.6$ | $0$ | $15.6$ | $14.2$ | $0.96081$ | $0.004072$ | $5.5\times 10^{-6}$ | $19.9$ | $2.2\times 10^{17}$
$-20.8$ | $0$ | $15.8$ | $14.2$ | $0.96092$ | $0.004046$ | $2.9\times 10^{-6}$ | $19.8$ | $1.8\times 10^{17}$
$-21.0$ | $0$ | $16.0$ | $14.3$ | $0.96089$ | $0.004046$ | $1.4\times 10^{-6}$ | $19.9$ | $2.2\times 10^{17}$
Table 1: The values of the parameters $(g_{1},g_{2},g_{3},g_{0})$ and the
corresponding CMB observables $(n_{s},r)$, with a small kick $\theta^{\prime}$
at the saddle point, the duration $\Delta N_{2}$ of the second SR phase of
inflation, and the resulting PBH masses. The initial conditions are chosen to
be $\varphi_{0}=5.5$ and $\theta_{0}=0$ with the vanishing velocities. The
pivot scale $k_{*}=0.05~{}\mathrm{Mpc}^{-1}$ corresponds to $63.79$ e-folds,
while the total duration of inflation is 70 e-folds.

The shape of the scalar potential and the inflationary trajectory with the
fine-tuned parameters $(g_{1},g_{2},g_{3})=(-21,0,16)$ are the same as in Fig.
1. The value of the potential height in the saddle point is $V/M^{2}\approx
0.05$. The Hubble function with two plateaus corresponding to two SR phases of
inflation is given in Fig. 2. The inflaton mass $M$ is approximately
$0.8\times 10^{-5}$, and the CMB scales are around $2.0\times 10^{-9}$.

Figure 2: The plot of the Hubble function for the parameters $g_{1}=-21$,
$g_{2}=0$ and $g_{3}=16$. The blue shaded region shows the first SR stage of
inflation and the red shaded region shows the second SR stage of inflation.

The SR parameters $\epsilon$ and $\eta$ are displayed in Fig. 3 where the plot
(a) shows the presence of the USR phase because of very small values of
$\epsilon$ of the order $10^{-8}$, whereas the plot (b) demonstrates a
violation of the SR conditions in the USR phase because of $\eta>1$. The
values of $\eta_{\mathrm{CMB}}$ at CMB scales are around $0.015$ and increase
to $0.2$ at the end of the first SR stage of inflation. The values of
$\eta_{\mathrm{USR}}$ during the USR phase are between $3$ and $4$. The
duration of the USR phase is $5.5$ e-folds.

(a) |  (b)
---|---
Figure 3: The plots of the SR parameters $\epsilon$ in (a) and $\eta$ in (b)
for $g_{1}=-21$, $g_{2}=0$ and $g_{3}=16$. The blue shaded region shows the
first SR stage of inflation and the red shaded region shows the second SR
stage of inflation.

The derived values of the CMB tilt $n_{s}$ in Table 1 are within the $1\sigma$
range of the Planck-observed value in Eq. (6), while the derived values of the
tensor-to-scalar ratio $r$ in Table 1 are well below the current observational
bound in Eq. (6).

The power spectrum is given in Fig. 4 for the fine-tuned parameters collected
in Table 1. The peaks in the power spectrum have heights between $10^{-2}$ and
$10^{-3}$ which are about 7 to 6 orders of the magnitude higher than the CMB
level of the order $10^{-9}$ (on the left of the plot), respectively.

Figure 4: The power spectrum of scalar perturbations for the parameter sets in
Table. 1.

Given the power spectrum, the masses of PBH originating from large over-
densities can be estimated as follows [29]:

$M_{\mathrm{PBH}}\simeq{{1}\over{H(t_{*})}}\mathrm{exp}\left[2(N_{\mathrm{end}}-N_{*})+\int_{t_{*}}^{t_{\mathrm{exit}}}\epsilon(t)H(t)dt\right]~{},$
(22)

where $t_{*}$ is the time when the first SR stage of inflation ends and
$t_{\mathrm{exit}}$ is the time when the CMB pivot scale
$k_{*}=0.05~{}\mathrm{Mpc}^{-1}$ exits the horizon. The leading term in the
exponential is given by $2\Delta N_{2}=2(N_{\mathrm{end}}-N_{*})$, whereas the
second term in the exponential is sub-leading. The value of $\Delta N_{2}$ is
close to $20$ in our model, which results in the PBH masses of the order
$10^{17}$ g given in Table 1.

The dependence of $\Delta N_{2}$ upon $\theta^{\prime}$ is shown in Fig. 5.

Figure 5: The impact of quantum kick $\theta^{\prime}$ on $\Delta N_{2}$ for
$g_{1}=-21$, $g_{2}=0$ and $g_{3}=16$.

The power spectrum enhancement (large peak) describes large scalar
perturbations near the critical point, which are essentially independent upon
initial conditions but are sensitive to the kick velocity $\theta^{\prime}$ at
the saddle point.

We compared our result for the peak in the power spectrum of Fig. 4 to the
log-normal fit of the peak [30, 31, 32],

$\mathcal{P}_{\zeta}(k)={{\mathcal{A}_{\zeta}}\over{\sqrt{2\pi}\Delta}}\mathrm{exp}\left[-{{\mathrm{ln}^{2}(k/k_{p})}\over{2\Delta^{2}}}\right],$
(23)

which has only two adjustable parameters, the amplitude $\mathcal{A}_{\zeta}$
and the width $\Delta$ of the peak at the location scale $k_{p}$. A comparison
of the power spectrum and the log-normal fit is shown in Fig. 6.

Figure 6: The plot of the power spectrum (blue) in the case
$(g_{1},g_{2},g_{3})=(-21,0,16)$, against the log-normal fit (red) with the
amplitude $\mathcal{A}_{\zeta}=0.01$ and the width $\Delta$=1.5.

The standard (Press-Schechter) formalism [33] can be used to estimate the PBH
fraction of DM in our model by using the power spectrum shown in Fig. 4 with
the parameters given in Table 1. The relevant equations can be found in Ref.
[34], see also Section 6 of Ref. [35]. We found that the PBH-DM fraction is
very sensitive to the values of the parameters, while it is possible to get
the whole (100%) PBH DM with the density threshold for PBH formation
$\delta_{c}\approx 0.55$ and the peak height $P_{\zeta,{\rm max.}}\approx
0.05$. It is worth mentioning that even a small contribution of PBH to DM can
play an important cosmological role [13].

##  5 Gravitational waves induced by PBH production

PBH production leads to induced gravitational waves (GW) that are different
from primordial GW caused by inflation. The current energy density fraction of
the PBH induced GW can be computed in the second order with respect to
perturbations, and the result reads [36, 37]

$\displaystyle\Omega_{\mathrm{GW}}(k)=$
$\displaystyle{{c_{g}\Omega_{r,0}}\over{36}}\int^{{{1}\over{\sqrt{3}}}}_{0}\,\mathrm{d}d\int^{\infty}_{{{1}\over{\sqrt{3}}}}\,\mathrm{d}s\left[{{(d^{2}-1/3)(s^{2}-1/3)}\over{s^{2}+d^{2}}}\right]^{2}\times$
$\displaystyle\mathcal{P}_{\zeta}\left({{k\sqrt{3}}\over{2}}(s+d)\right)\mathcal{P}_{\zeta}\left({{k\sqrt{3}}\over{2}}(s-d)\right)\left[\mathcal{I}_{c}(d,s)^{2}+\mathcal{I}_{s}(d,s)^{2}\right]~{},$
(24)

where the functions $\mathcal{I}_{c}(d,s)$ and $\mathcal{I}_{s}(d,s)$ are
given by

$\displaystyle\mathcal{I}_{c}(d,s)$
$\displaystyle=-36\pi{{(s^{2}+d^{2}-2)^{2}}\over{(s^{2}-d^{2})^{3}}}\theta(s-1)~{},$
$\displaystyle\mathcal{I}_{s}(d,s)$
$\displaystyle=-36{{s^{2}+d^{2}-2}\over{(s^{2}-d^{2})^{2}}}\left[{{s^{2}+d^{2}-2}\over{(s^{2}-d^{2})}}\mathrm{ln}\left|{{d^{2}-1}\over{s^{2}-1}}\right|+2\right]~{},$
(25)

with $\Omega_{r,0}\sim 8.6\times 10^{-5}$ being the current energy density
fraction of radiation. The $\theta(s-1)$ is the step function and
$c_{g}\approx 0.4$.

A numerical calculation of Eq. (24) with our power spectrum in Fig. 4 yields a
simple result

$\displaystyle\Omega_{\mathrm{GW}}(k)\approx 3\cdot
10^{-5}\mathcal{P}^{2}_{\zeta}(k)$ (26)

that agrees with the rough estimate given in Ref. [30]. The induced GW
frequencies are related to the PBH masses as [38]

$f_{p}\approx
5.7\left({{M_{\odot}}\over{M_{\mathrm{PBH}}}}\right)^{1/2}10^{-9}~{}\mathrm{Hz},$
(27)

where $M_{\odot}\approx 2\cdot 10^{33}$ g is the mass of the Sun. In our
models, the PBH masses are around $10^{17}g$, which results in the GW
frequency $f_{p}\approx 0.6$ Hz that is much higher than the GW frequencies
between 3 and 400 nHz detected by NANOGrav [39]. As was argued in Ref. [40], a
simultaneous detection of three GW peaks with nHz, Hz and kHz frequencies may
be a signature of Starobinsky-like supergravity.

##  6 Conclusion

The high scale of Starobinsky inflation naturally implies its local
supersymmetrization in the context of four-dimensional supergravity, while
high-scale supersymmetry breaking avoids the Large Hadron Collider constraints
on (low-scale) supersymmetry. It is well known that Starobinsky inflation can
be realized in the standard (Einstein) supergravity coupled to matter, see
e.g., Ref. [8], while it is also possible in the modified Starobinsky
supergravity [14] without significant fine-tuning.

More efforts are needed to realize efficient production of PBH with the masses
beyond the Hawking evaporation limit during inflation in good agreement with
CMB observations because of common tension with the well-measured value of the
CMB tilt $n_{s}$ of scalar perturbations. The necessity to produce a
significant enhancement of the power spectrum of scalar perturbations during
inflation usually drives the predicted value of $n_{s}$ below its Planck-
measured value, thus demanding fine-tuning of the parameters in the potential.
We do not think that fine-tuning itself is necessarily a problem because both
inflation and (presumably) PBH production were unique (non-reproducible)
events in the Universe.

A very existence of viable inflation with efficient PBH production in the
Starobinsky-like supergravity models is not guaranteed because of the
constraints imposed by local supersymmetry. For instance, as was shown in Ref.
[15], any linear combination of holomoprphic quadratic polynomials and
exponentials of $Z$ in the superpotential cannot solve the task within
$1\sigma$ agreement with the known value of $n_{s}$. Nevertheless, we
demonstrated in this paper that it is possible by exploiting the most general
(cubic) Wess-Zumino-type (renormalizable) superpotentials. The $g$-function
used in Ref. [15] was

$g(Z)=M(g_{0}+g_{1}Z+g_{2}Z^{2})~{}.$ (28)

The best choice of the coefficients $(g_{0},g_{1},g_{2})\approx(1,-1,2)$ was
shown to lead to the PBH masses beyond $10^{15}$ g within $3\sigma$ (but
outside $1\sigma$) agreement with the $n_{s}$ value in Eq. (6), whereas
demanding $1\sigma$ agreement with the CMB tilt $n_{s}$ was shown to drive the
PBH masses below $10^{15}$ g [15]. In this paper, we found that the quadratic
term should be absent, $g_{2}=0$, and the cubic term is essential for success,
$g_{3}\neq 0$, see Table 1.

The issue of ultra-violet (UV) completion of our supergravity models is beyond
the scope of this investigation. To this end, we confine ourselves to a few
comments. Though very little is known for sure about quantum gravity, when
assuming the quantum gravity scale to be given by the Planck scale, the
Starobinsky model and its supergravity extensions are on the safe side against
quantum gravity corrections because the scale of Starobinsky inflation is well
below the Planck scale that is simultaneously the UV-cutoff scale for the
Starobinsky gravity (1). When assuming the Swampland conjectures [41] that
imply a significantly lower quantum gravity scale, quantum gravity may have
significant impact on our models so that their UV embedding is needed. It was
recently argued in the literature [42, 43] that it is difficult to realize
Starobinsky inflation in superstring theory, both as regards Starobinsky
gravity (1) and the Starobinsky potential (4), as well as to reconcile the
Starobinsky model (1) with the Swampland program.

We think that inflation has happened, and we know that black holes exist,
though we do not know whether some of them are primordial or not. In any case,
inflation, supersymmetry and PBH genesis are the important windows into very
high energy physics, whose investigation put constraints on our understanding
of the origin of the Universe.

## Acknowledgements

RI and SVK were supported by Tokyo Metropolitan University. SVK was also
supported by the World Premier International Research Center Initiative (WPI
Initiative), MEXT, Japan, the Japanese Society for Promotion of Science under
the grant No. 22K03624, and the Development Program
Priority-2030-NIP/EB-004-375-2023 of Tomsk Polytechnic University.

One of the authors (SVK) is grateful to Ignatios Antoniadis, Shuntaro Aoki,
Guillem Domenech, David I. Kaiser, Florian Kühnel, Kazunori Kohri, Edward W.
Kolb, Theodoros Papanikolaou, Takahiro Terada, Burt Ovrut, Shi Pi, Misao
Sasaki and Alexei Starobinsky for discussions and correspondence.

This paper is devoted to memory of late Alexei Starobinsky, one of the
greatest cosmologists of all times.

## References

  * [1] A. A. Starobinsky, “A new type of isotropic cosmological models without singularity,” Phys. Lett. B 91 no. 1, (1980) 99 – 102.
  * [2] BICEP, Keck Collaboration, P. A. R. Ade et al., “Improved Constraints on Primordial Gravitational Waves using Planck, WMAP, and BICEP/Keck Observations through the 2018 Observing Season,” Phys. Rev. Lett. 127 no. 15, (2021) 151301, arXiv:2110.00483 [astro-ph.CO].
  * [3] M. Tristram et al., “Improved limits on the tensor-to-scalar ratio using BICEP and Planck data,” Phys. Rev. D 105 no. 8, (2022) 083524, arXiv:2112.07961 [astro-ph.CO].
  * [4] S. V. Ketov and A. A. Starobinsky, “Inflation and non-minimal scalar-curvature coupling in gravity and supergravity,” JCAP 08 (2012) 022, arXiv:1203.0805 [hep-th].
  * [5] S. V. Ketov, “Supergravity and Early Universe: the Meeting Point of Cosmology and High-Energy Physics,” Int. J. Mod. Phys. A 28 (2013) 1330021, arXiv:1201.2239 [hep-th].
  * [6] S. V. Ketov, “On the equivalence of Starobinsky and Higgs inflationary models in gravity and supergravity,” J. Phys. A 53 no. 8, (2020) 084001, arXiv:1911.01008 [hep-th].
  * [7] S. V. Ketov, “Multi-Field versus Single-Field in the Supergravity Models of Inflation and Primordial Black Holes,” Universe 7 no. 5, (2021) 115.
  * [8] J. Ellis, D. V. Nanopoulos, and K. A. Olive, “No-Scale Supergravity Realization of the Starobinsky Model of Inflation,” Phys. Rev. Lett. 111 (2013) 111301, arXiv:1305.1247 [hep-th]. [Erratum: Phys.Rev.Lett. 111, 129902 (2013)].
  * [9] S. J. Gates, Jr. and S. V. Ketov, “Superstring-inspired supergravity as the universal source of inflation and quintessence,” Phys. Lett. B 674 (2009) 59–63, arXiv:0901.2467 [hep-th].
  * [10] S. V. Ketov and A. A. Starobinsky, “Embedding $(R+R^{2})$-Inflation into Supergravity,” Phys. Rev. D 83 (2011) 063512, arXiv:1011.0240 [hep-th].
  * [11] F. Farakos, A. Kehagias, and A. Riotto, “On the Starobinsky Model of Inflation from Supergravity,” Nucl. Phys. B 876 (2013) 187–200, arXiv:1307.1137 [hep-th].
  * [12] S. Cecotti and R. Kallosh, “Cosmological Attractor Models and Higher Curvature Supergravity,” JHEP 05 (2014) 114, arXiv:1403.2932 [hep-th].
  * [13] B. Carr, K. Kohri, Y. Sendouda, and J. Yokoyama, “Constraints on primordial black holes,” Rept. Prog. Phys. 84 no. 11, (2021) 116902, arXiv:2002.12778 [astro-ph.CO].
  * [14] S. V. Ketov, Inflationary Cosmology from Supergravity. In: Bambi, C., Modesto, L., Shapiro, I. (eds) Handbook of Quantum Gravity. Springer, Singapore (2023); https://doi.org/10.1007/978-981-19-3079-9.
  * [15] S. Aoki, R. Ishikawa, and S. V. Ketov, “Pole inflation and primordial black holes formation in Starobinsky-like supergravity,” Class. Quant. Grav. 40 no. 6, (2023) 065002, arXiv:2210.10348 [hep-th].
  * [16] K.-i. Maeda, “Towards the Einstein-Hilbert Action via Conformal Transformation,” Phys. Rev. D 39 (1989) 3159.
  * [17] S. V. Ketov and T. Terada, “Old-minimal supergravity models of inflation,” JHEP 12 (2013) 040, arXiv:1309.7494 [hep-th].
  * [18] S. Cecotti, “Higher derivative supergravity is equivalent to standard supergravity coupled to matter. 1.,” Phys. Lett. B 190 (1987) 86–92.
  * [19] Y. Aldabergenov, A. Addazi, and S. V. Ketov, “Primordial black holes from modified supergravity,” Eur. Phys. J. C 80 no. 10, (2020) 917, arXiv:2006.16641 [hep-th].
  * [20] R. Ishikawa and S. V. Ketov, “Exploring the parameter space of modified supergravity for double inflation and primordial black hole formation,” Class. Quant. Grav. 39 no. 1, (2022) 015016, arXiv:2108.04408 [astro-ph.CO].
  * [21] A. Gundhi and C. F. Steinwachs, “Scalaron-Higgs inflation,” Nucl. Phys. B 954 (2020) 114989, arXiv:1810.10546 [hep-th].
  * [22] A. Gundhi, S. V. Ketov, and C. F. Steinwachs, “Primordial black hole dark matter in dilaton-extended two-field Starobinsky inflation,” Phys. Rev. D 103 no. 8, (2021) 083518, arXiv:2011.05999 [hep-th].
  * [23] M. Dias, J. Frazer, and D. Seery, “Computing observables in curved multifield models of inflation—A guide (with code) to the transport method,” JCAP 12 (2015) 030, arXiv:1502.03125 [astro-ph.CO].
  * [24] S. R. Geller, W. Qin, E. McDonough, and D. I. Kaiser, “Primordial black holes from multifield inflation with nonminimal couplings,” Phys. Rev. D 106 no. 6, (2022) 063535, arXiv:2205.04471 [hep-th].
  * [25] C. Pattison, V. Vennin, D. Wands, and H. Assadullahi, “Ultra-slow-roll inflation with quantum diffusion,” JCAP 04 (2021) 080, arXiv:2101.05741 [astro-ph.CO].
  * [26] C. Pahud, M. Kamionkowski, and A. R. Liddle, “Oscillations in the inflaton potential?,” Phys. Rev. D 79 (2009) 083503, arXiv:0807.0322 [astro-ph].
  * [27] J. Chluba, J. Hamann, and S. P. Patil, “Features and New Physical Scales in Primordial Observables: Theory and Observation,” Int. J. Mod. Phys. D 24 no. 10, (2015) 1530023, arXiv:1505.01834 [astro-ph.CO].
  * [28] J. Fumagalli, S. e. Renaux-Petel, and L. T. Witkowski, “Resonant features in the stochastic gravitational wave background,” JCAP 08 (2021) 059, arXiv:2105.06481 [astro-ph.CO].
  * [29] S. Pi, Y.-l. Zhang, Q.-G. Huang, and M. Sasaki, “Scalaron from $R^{2}$-gravity as a heavy field,” JCAP 05 (2018) 042, arXiv:1712.09896 [astro-ph.CO].
  * [30] S. Pi and M. Sasaki, “Gravitational Waves Induced by Scalar Perturbations with a Lognormal Peak,” JCAP 09 (2020) 037, arXiv:2005.12306 [gr-qc].
  * [31] G. Domènech, “Scalar Induced Gravitational Waves Review,” Universe 7 no. 11, (2021) 398, arXiv:2109.01398 [gr-qc].
  * [32] D. Frolovsky and S. V. Ketov, “Fitting Power Spectrum of Scalar Perturbations for Primordial Black Hole Production during Inflation,” Astronomy 2 no. 1, (2023) 47–57, arXiv:2302.06153 [astro-ph.CO].
  * [33] W. H. Press and P. Schechter, “Formation of galaxies and clusters of galaxies by selfsimilar gravitational condensation,” Astrophys. J. 187 (1974) 425–438.
  * [34] K. Inomata, M. Kawasaki, K. Mukaida, Y. Tada, and T. T. Yanagida, “Inflationary Primordial Black Holes as All Dark Matter,” Phys. Rev. D 96 no. 4, (2017) 043504, arXiv:1701.02544 [astro-ph.CO].
  * [35] Y. Aldabergenov, A. Addazi, and S. V. Ketov, “Inflation, SUSY breaking, and primordial black holes in modified supergravity coupled to chiral matter,” Eur. Phys. J. C 82 no. 8, (2022) 681, arXiv:2206.02601 [astro-ph.CO].
  * [36] J. R. Espinosa, D. Racco, and A. Riotto, “A Cosmological Signature of the SM Higgs Instability: Gravitational Waves,” JCAP 09 (2018) 012, arXiv:1804.07732 [hep-ph].
  * [37] K. Kohri and T. Terada, “Semianalytic calculation of gravitational wave spectrum nonlinearly induced from primordial curvature perturbations,” Phys. Rev. D 97 no. 12, (2018) 123532, arXiv:1804.08577 [gr-qc].
  * [38] V. De Luca, G. Franciolini, and A. Riotto, “NANOGrav Data Hints at Primordial Black Holes as Dark Matter,” Phys. Rev. Lett. 126 no. 4, (2021) 041303, arXiv:2009.08268 [astro-ph.CO].
  * [39] NANOGrav Collaboration, G. Agazie et al., “The NANOGrav 15 yr Data Set: Evidence for a Gravitational-wave Background,” Astrophys. J. Lett. 951 no. 1, (2023) L8, arXiv:2306.16213 [astro-ph.HE].
  * [40] S. Basilakos, D. V. Nanopoulos, T. Papanikolaou, E. N. Saridakis, and C. Tzerefos, “Gravitational wave signatures of no-scale Supergravity in NANOGrav and beyond,” arXiv:2307.08601 [hep-th].
  * [41] E. Palti, “The Swampland: Introduction and Review,” Fortsch. Phys. 67 no. 6, (2019) 1900037, arXiv:1903.06239 [hep-th].
  * [42] M. Brinkmann, M. Cicoli, and P. Zito, “Starobinsky inflation from string theory?,” JHEP 09 (2023) 038, arXiv:2305.05703 [hep-th].
  * [43] D. Lust, J. Masias, B. Muntz, and M. Scalisi, “Starobinsky Inflation in the Swampland,” arXiv:2312.13210 [hep-th].


Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

CLEF 2021 – Conference and Labs of the Evaluation Forum, September 21–24,
2021, Bucharest, Romania

#  Retrieving Comparative Arguments using
Ensemble Methods and Neural Information Retrieval

Viktoriia Chekalina  Alexander Panchenko  Skolkovo Institute of Science and
Technology, Moscow, Russian Federation  Philips Innovation Lab Rus, Moscow,
Russian Federation

(2021)

#  Notebook for the Touche Lab on Argument Retrieval at CLEF 2021

Viktoriia Chekalina  Alexander Panchenko  Skolkovo Institute of Science and
Technology, Moscow, Russian Federation  Philips Innovation Lab Rus, Moscow,
Russian Federation

(2021)

###### Abstract

In this paper, we present a submission to the Touché lab’s Task 2 on Argument
Retrieval for Comparative Questions [1, 2]. Our team Katana supplies several
approaches based on decision tree ensembles algorithms to rank comparative
documents in accordance with their relevance and argumentative support. We use
PyTerrier [3] library to apply ensembles models to a ranking problem,
considering statistical text features and features based on comparative
structures. We also employ large contextualized language modelling techniques,
such as BERT [4], to solve the proposed ranking task. To merge this technique
with ranking modelling, we leverage neural ranking library OpenNIR [5].

Our systems substantially outperforming the proposed baseline and scored first
in relevance and second in quality according to the official metrics of the
competition (for measure NDCG@5 score). Presented models could help to improve
the performance of processing comparative queries in information retrieval and
dialogue systems.

###### keywords:

comparative argument retrievalnatural language processing neural information
retrieval

##  1 Introduction

On a daily basis, people face the problem of choosing between two entities -
which phone is more reliable, which juice contains less sugar, which hotel is
better for a holiday. Domain-specific comparison systems, like WolframAlpha or
Diffen, solve this problem partly and rely on structured data, which limits
the number of cases it can be used.

On the other hand, the Web contains a vast number of opinions and objective
arguments that can facilitate the comparative decision-making process. It
creates the need of developing an open-domain general system that could
process such information. The issue is to retrieve from a set of documents
relevant, supportive and credible arguments. The aim of the proposed work is
to retrieve from ClueWeb12 111http://lemurproject.org/clueweb12 corpus
documents and re-rank them, considering argumentation for or against one
option or the other.

The contribution of our work is the following: we are first to use ensemble
methods based on mixed statistical and comparative features to the document
ranking; we are first to use neural information retrieval approach to the task
of argument retrieval; we propose a model outperforming the baseline and
yielding the first and the second-best result according to the relevance and
quality metric, respectively.

##  2 Related work

The most relevant to this work is the previous shared task Touche 2020 [6]. 17
participants took part in the competition and submitted 41 runs. Various
approaches were tested by these participants, including methods based on
extraction of structures corresponding to claims and premises, assessing
argument quality, representation of documents by language models, expansion of
the query by similar words. The ranking function from search engine ChatNoir
[7] based on BM25F [8] approach was used as a baseline.

Only a few of the submitted solutions can slightly improve the baseline. The
best overall approach in the previous competition was the method based on
query extension and reranking documents by relevance, credibility, and
supportive quality [9].

This work is based on our run submitted in the previous version of the Touche
shared task [10]. In this work, we used a pre-trained language model to find
relevance between the query and document. Extraction of comparative structures
and counting the number of comparative sentences in a document help us to
assess the quality of relevant arguments.

Therefore, the problem of argument retrieval arises in other scenarios.
Comparative argumentation machine CAM [11] retrieves comparative sentences
with respect to accepted objects and comparison aspects. The paper [12]
explores the influence of context on an argument detecting system and proves
the performance increasing related to it.

##  3 Datasets and experimental design

###  3.1 Datasets

The organizers provided 50 comparative questions (topics), for which we should
obtain documents containing convincing arguments for or against one or another
option. Topics for the competition are available online.
222https://webis.de/events/touche-21/shared-task-2.html

In addition, 50 topics and corresponding relevance annotations of the previous
year’s competition [13] were given for supervised learning. These documents
were also retrieved from ChatNoir and ranked manually to 0 (not relevant), 1
(relevant) or 2 (highly relevant) scores. We use this data to train and set up
models based on the decision trees and fine-tune the BERT ranker. Besides,
last year’s teams submissions were available too.

Unfortunately, this data is not insufficient for fitting large supervised
ranking models, for example, based on the BERT technique. In this case we use
adjacent question-answering dataset called Antique [14]. This dataset consists
of the questions and answers of Yahoo! Webscope L6 and contains 2,626 open-
domain non-factoid questions and 34,011 manual relevance annotations.

The example of query and ranked answers are in Table 6, Table 7 in Appendix A.
It might be noticed that Antique dataset has a different set of ranking scores
- 0, 1, 2 instead of 1, 2, 3, 4 - so we rewrite Antique ranks in accordance
with the following mapping 1$\rightarrow$0, 2$\rightarrow$1, 3$\rightarrow$1,
4$\rightarrow$2.

###  3.2 Evaluation setup

We use every topic as a query in ChatNoir [7] search engine and extract up to
100 unique documents from the ClueWeb12 corpus. We clean documents’ bodies
from HTML tags and markups and ranked them using one of the developed
approaches described below.

As auxiliary data, the organizers provided the topics of the previous year’s
competition. For each proposed topic, a set of documents from ChatNoir was
retrieved and labelled as described above. We use this data to train
developing models and valid composing approaches. In the validation phase, we
split the ranked data into 40 topics in train and 10 in validation.

In the run phase, we execute produced solutions on web evaluation platform
Tira [15]. In this stage to fit the model we use ranked data from the previous
year entirely and predict rank for current proposed topics. The runs were
evaluated using the NDCG metrics based on the human judgements of the
submitted runs. Retrieved documents were judged in accordance with two
criteria: (i) document relevance, (ii) whether sufficient argumentative
support is provided [16].

##  4 Document ranking using ensembles of trees

In this section, we use ensembles of trees as a supervised machine learning
technique to solve ranking problems. We choose either pointwise regression
tree algorithms, like Random Forest, or boosted tree algorithms like XGBoost
and LightGBM. In the cases of LightGBM model we employ LambdaMART [17]
objective. It combines cost function derived from minimizing the number of
inversions in ranking (LambdaRank [18]) and objective for building gradient
boosted decision trees (MART [19]). We use PyTerrier platform for information
retrieval.333https://pyterrier.readthedocs.io/en/latest/index.html It
simplifies the extraction of the text features and allows expressing retrieval
experiments [20].

###  4.1 Feature extraction

For our ranking ML methods, we use features that came from 3 origins described
below: (i) ranking features extracted by PyTerrier, (ii) specific comparative
features, (iii) score from ChatNoir system based on custom BM25 scoring
function.444https://www.elastic.co/guide/en/elasticsearch/reference/current/index-
modules-similarity.html

####  4.1.1 Features extracted by PyTerrier

PyTerrier provides measure of matching query-document texts by several models.
Among these models are statistical measures (TF-IDF), mesures based on
language models (Heimstra, Diriclet), measures based on occurrence of a
document depending on the fields that the term occurs in (BM25F, PLF). The
list of all possible models are available at the cite
555http://terrier.org/docs/current/javadoc/org/terrier/matching/models/package-
summary.html. Among these varieties we have chosen BM25, Heimstra, DFIC, DPH,
TF-IDF, DiricletLM, PL2 for our exploration.

Table 1: Results on validation set for text features in PyTerrier models. Method -Score | NDCG@5
---|---
BM25 | 0.3637
Heimstra | 0.3616
DFIC | 0.3642
DPH | 0.3110
TF-IDF | 0.3637
DiricletLM | 0.3307
PL2 | 0.3703

We applied each of the selected methods sequentially and independently to the
training set, ranked documents by the obtained scores and evaluated the
ranking on the validation set. The result of these tests is in Table 1. We
have chosen 3 methods with the most promising results, and these 3 methods
combine 3 features.

####  4.1.2 Comparative features

We focus not only on finding high relevant documents as on finding documents
with a comparison of one object relative to another. The work [21] assumes
that the comparative issue can be represented by comparative structures -
objects for comparison, comparative aspects and predicates. We take the
sequence-labelling model suggested in the cited paper and applied it to the
query. It helps us to define objects for comparison for every topic. Then we
apply the model to document and get a comparative feature set.

The feature is_retrieved describes are there any comparative structures in the
document at all. Characteristic objs_score defines how many objects from query
are found in document (0, 1 or 2). Feature asp_pred_score is counted in the
following way: if at least one object from a query is in the document, every
word in the document labelled as an aspect or predicate increases the score to
0.5. Finally, we combined defined features with scores obtained from the
ChatNoir system, and a resulting feature vector for pair query-document is
{score_pl2, score_tf, score_bm, score_dfic, baseline_scores, is_retrieved,
ap_score, objs_score}.

###  4.2 Models

####  4.2.1 Random Forest

We use the Random Forest model imported from Sklearn and wrapped by the
PyTerrier pipeline. To find the best setup, we vary the number of estimators
from 10 to 150, the value 20 gives the best valid score NDCG@5 0.408.

####  4.2.2 XGBoost

We also wrapped gradient boosting library from Sklearn to PyTerrier class and
tune hyperparameters by setting the learning rate from $1{\rm e}{-4}$ to 0.1
and max_depth from 4 to 16. The best setup is learning rate $0.01$, max_depth
$6$ and gives NDCG@5 0.547.

####  4.2.3 LightGBM

In the case of LightGBM, we vary the number of leaves from 8 to 20 and the
learning rate from 0.001 to 0.1. The best configuration with num_leaves = 15
and learning rate = 0.1 gives 0.579 score.

The feature importance of the resulting model is in Table 2. It can be seen
that the most significant feature is the score retrieved from the ChatNoir,
then there is a Divergence from Independence based on Chi-square [22] and the
existence of comparison objects in the document.

##  5 Document ranking using neural information retrieval based on BERT

Contextualized language models such as BERT can be much more efficient for
ranking tasks because they contain vast relationships between language units.
In the proposed work we use a reranking model from OpenNIR [5] based on
“Vanilla” Transformer architecture.

###  5.1 Text representation

BERT receives a query and document and processes it jointly. A distinctive
feature of the BERT reranker is injection token similarity matrices on each
layer, which considerably improves performance [23].

Table 2: Feature importance in the proposed LightGBM model Feature | Pl2 | TF-IDF | BM25 | Dfic | ChatNoir | has comp | objs_score | asp_pred
---|---|---|---|---|---|---|---|---
Importance | 1.76 | 1.19 | 1.51 | 2.3 | 20.8 | 0 | 1.66 | 1.51

###  5.2 Training process

First, we pretrain this reranker on the Antique dataset. We clean this dataset
from incorrect symbols and makeups. We also left from the dataset documents of
length more than 300 characters, since the length of the ChatNoir retrieves
usually does not exceed 300. The training process lasted for 500 epochs with
0.001 learning rate and 56 objects in every batch. Finally, our model gives
NDCG@5 0.3362 on a validation set. We fine-tune the model on 40 train topics
from the previous year for 50 epochs with the same configuration. Fine-tuning
increased the score on validation up to 0.412.

##  6 Results

###  6.1 Results on validation set

Table 3: Results on validation set. Method | NDCG@5 | Time, ms
---|---|---
Random Forest | 0.408 | 127.168
XGBoost | 0.547 | 128.848
LightGBM | 0.572 | 131.244
Bert Ranker | 0.412 | 1560.947
Baseline’20 | 0.534 | -

The result for every proposed approach obtained on the validation part of data
from the previous year competition is in Table 3. We also evaluate the
previous year’s baseline on the validation set. The best scores come from the
LightGBM model, which also outperforms the baseline. XGBoost has fewer scores,
Random Forest as a simple algorithm has the smallest score. Bert overtakes
Random Forest a little.

In the right column, we also added the time required to train each model. It
can be seen that the ensemble-based models have approximately the same time
complexity, while the Bert requires much more time to train.

###  6.2 Results on test set

For final testing, the retrieved documents were labelled manually with a score
from 0 to 3. Judgment was carried out in two independent criteria: the
relevance of the document to the given topic and the quality of the text.
Quality criterion includes good language styling, easy reading and proper
sentence structure, the absence of typos and alliteration.

For each criterion, a separate file with the assessor’s scores is available.
The results of two evaluations are presented in the Table 5 and Table 5. The
runs of our team Katana have the best result between all teams in terms of
relevance and the second result in terms of the text quality.

As in the validation set, XGBoost and LightGBM give the best performance. It
is well explained, since the loss of these models based on the ranking quality
functions, NDCG in the XGBoost case and LambdaMART in the LightGBM case. The
first model describes relevance a bit better (0.489) and has first place among
the whole participant. For quality, conversely, LightGBM is better. It
archives 0.684 and takes second place in a quality table, slightly
surrendering to Top 1. The random forest method has scores just below the
baseline in both cases. It can be explained by a more elementary algorithm for
building an ensemble. Bert gives a quite good result for quality and weak for
relevance. Perhaps the data from the adjacent task (factoid QA) used for the
training is the reason for not a very accurate solution.

Table 4: NDCG@5 scores on runs for relevance for Katana
team, baseline and Top-2 approach  Method | NDCG@5
---|---
Random Forest | 0.393
XGBoost (Top 1) | 0.489
LightGBM | 0.460
Bert Ranker | 0.091
ChatNoir baseline | 0.422
Thor team (Top 2) | 0.478

Table 5: NDCG@5 scores on runs for quality for Katana
team, baseline and Top-1 approach Method | NDCG@5
---|---
Random Forest | 0.630
XGBoost | 0.675
LightGBM (Top 2) | 0.684
Bert Ranker | 0.466
ChatNoir baseline | 0.636
Rayla team (Top 1) | 0.688

##  7 Conclusion

In this paper, we present our solution to the Argument retrieval shared task.
We pay attention to ensembles methods and use statistic approaches, language
modelling and comparative structure extraction to retrieve features for it. We
also use a neural reranker based on the Bert technique to use information from
a contextualized model in our task.

The best results were obtained by gradient boosting methods, training on
ranking cost function: XGBoost and LightGBM. The proposed approaches
outperform baseline and take first and second places in relevance and quality
ranking, respectively. Bert contextualized model shows the need for large
learning data.

###### Acknowledgements.

This work is partially supported by the project “ACQuA: Answering Comparative
Questions with Arguments” (grants BI 1544/7-1 and HA 5851/2-1) as part of the
priority program “RATIO: Robust Argumentation Machines” (SPP 1999). We thank
Maik Frobe for providing the support of the software runs in the TIRA system.

## Appendix A: Examples of training data

Table 6: Example of query and document with different relevance in Touche task dataset Query |  Document |  Rank
---|---|---
What is better for the environment, a real or a fake Christmas tree? |  Disease and condition content is reviewed by our medical review board real or artificial? There is so much confusing information out there about which is better for your health and the environment. |  2
You may think you’re saving a tree, but the plastic alternative has problems too. Which is “greener” an artificial Christmas tree or a real one? |  1
This entry is part 25 of 103 in the series eco-friendly friday november 28th’s tip christmas trees: stuck between choosing a real Christmas tree or a fake one? |  0
Table 7: Example of query and document with different relevance in Antique dataset Query |  Document |  Rank
---|---|---
Why do we put the letter k on the words knife and knob, knee? |  They are saxon words. Knife would have been pronounced ker-niff. |  4
As a guess I would say that historically “kn” would have been pronounced differently to “n” and that time has altered the way the words are pronounced. |  3
Because English is a funny language. |  2
|  I don’t really (k)now! |  1

## Appendix B: Examples of ranking results

In this appendix you can find examples Top-3 ranked documents in accordance to
LightGBM and Baseline approaches.

Table 8: Example of documents with the different relevance to query “Is
admission rate in Stanford higher than that of MIT ?” Is admission rate in
Stanford higher than that of MIT ?
---
LightGBM Top-3 |  Baseline Top-3
1\. Stanford and Harvard have a similar admissions rate of about 7%. MIT comes with a somewhat greater rate of success admitting just under 10% or 1742 for the class of 2015. Harvard, Stanford and MIT are global leaders in culture, commerce and governmental policies. |  1\. Stanford and Harvard have a similar admissions rate of about 7%. MIT comes with a somewhat greater rate of success admitting just under 10% or 1742 for the class of 2015. Harvard, Stanford and MIT are global leaders in culture, commerce and governmental policies
2\. For more than a decade, i have served as an admissions officer for MIT. In that time, i’ve read more than 10,000 applications and have watched thousands of new students enter MIT. It is a privilege to work at the most dynamic and exciting university in the world. |  2\. For more than a decade, i have served as an admissions officer for MIT. In that time, i’ve read more than 10,000 applications and have watched thousands of new students enter MIT. It is a privilege to work at the most dynamic and exciting university in the world.
3\. Our primary enhancement was targeted at families earning less than $75,000 — making mit tuition free and eliminating |  3\. All of this factual information, plus a lot of other detail, can be found in the mit admissions literature. In fact, this year, mit will award $74 million in undergraduate aid.
Table 9: Example documents with the different relevance to query “Which
smartphone has a better battery life: Xperia or iPhone ?” Which smartphone has
a better battery life: Xperia or iPhone ?
---
LightGBM Top-3 |  Baseline Top-3
1\. 1. The power saver app that will turn down settings when battery life is low to get as much juice out of the battery as possible. Sony has set the benchmark with its 12 megapixel camera inside the Xperia S. |  1\. The iPhone 4 is apple’s thinnest smartphone yet, but offers a much better screen, faster processor, video calling, and many other enhancements.
2\. How to increase the battery life of apple’s iPhone 4s many of those with an iphone 4s have complaints about the battery life. Apple has acknowledged these problems, and is working to fix them. |  2\. Sony Xperia’s review: an above average smartphone gizmotraker’, as far as battery life is concerned, it last about 7 hr 30 min in talktime, 450 hrs in standby.
3\. Sony Ericsson includes an 8gb card in the sales package the Sony Ericsson Xperia arc s has below average battery life. Most users will get around 24 hours of life out of the Xperia. X27’s 1600mah battery before it needs a recharge, but heavy users may need an injection of power before then. |  3\. How to increase the battery life of Apple’s Iphone 4s many of those with an iphone 4s have complaints about the battery life. Apple has acknowledged these problems, and is working to fix them.

## References

  * Bondarenko et al. [2021a] A. Bondarenko, L. Gienapp, M. Fröbe, M. Beloucif, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, M. Hagen,  Overview of Touché 2021: Argument Retrieval,  in: D. Hiemstra, M.-F. Moens, J. Mothe, R. Perego, M. Potthast, F. Sebastiani (Eds.), Advances in Information Retrieval. 43rd European Conference on IR Research (ECIR 2021), volume 12036 of Lecture Notes in Computer Science, Springer, Berlin Heidelberg New York, 2021a, pp. 574–582. URL: https://link.springer.com/chapter/10.1007/978-3-030-72240-1_67. doi:10.1007/978-3-030-72240-1_67.
  * Bondarenko et al. [2021b] A. Bondarenko, L. Gienapp, M. Fröbe, M. Beloucif, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, M. Hagen,  Overview of Touché 2021: Argument Retrieval,  in: G. Faggioli, N. Ferro, A. Joly, M. Maistro, F. Piroi (Eds.), Working Notes Papers of the CLEF 2021 Evaluation Labs, CEUR Workshop Proceedings, 2021b.
  * MacAvaney et al. [2021] S. MacAvaney, C. Macdonald, N. Tonellotto,  IR from Bag-of-words to BERT and Beyond through Practical Experiments: An ECIR 2021 tutorial with PyTerrier and OpenNIR,  in: Proceedings of the 43rd European Conference on Information Retrieval Research, 2021, pp. 728–730.
  * Devlin et al. [2019] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova,  BERT: Pre-training of deep bidirectional transformers for language understanding,  in: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp. 4171–4186. URL: https://www.aclweb.org/anthology/N19-1423. doi:10.18653/v1/N19-1423.
  * MacAvaney [2020] S. MacAvaney,  OpenNIR: A Complete Neural Ad-Hoc Ranking Pipeline,  in: J. Caverlee, X. B. Hu, M. Lalmas, W. Wang (Eds.), WSDM ’20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020, ACM, 2020, pp. 845–848. URL: https://doi.org/10.1145/3336191.3371864. doi:10.1145/3336191.3371864.
  * Bondarenko et al. [2020] A. Bondarenko, M. Fröbe, M. Beloucif, L. Gienapp, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, M. Hagen, Overview of Touché 2020: Argument Retrieval, 2020, pp. 384–395. doi:10.1007/978-3-030-58219-7_26.
  * Potthast et al. [2012] M. Potthast, M. Hagen, B. Stein, J. Graßegger, M. Michel, M. Tippmann, C. Welsch,  ChatNoir: A Search Engine for the ClueWeb09 Corpus,  in: B. Hersh, J. Callan, Y. Maarek, M. Sanderson (Eds.), 35th International ACM Conference on Research and Development in Information Retrieval (SIGIR 2012), ACM, 2012, p. 1004. doi:10.1145/2348283.2348429.
  * Robertson et al. [2004] S. E. Robertson, H. Zaragoza, M. J. Taylor,  Simple BM25 extension to multiple weighted fields,  in: D. A. Grossman, L. Gravano, C. Zhai, O. Herzog, D. A. Evans (Eds.), Proceedings of the 2004 ACM CIKM International Conference on Information and Knowledge Management, Washington, DC, USA, November 8-13, 2004, ACM, 2004, pp. 42–49. URL: https://doi.org/10.1145/1031171.1031181. doi:10.1145/1031171.1031181.
  * Abye et al. [2020] T. Abye, T. Sager, A. J. Triebel,  An open-domain web search engine for answering comparative questions,  in: L. Cappellato, C. Eickhoff, N. Ferro, A. Névéol (Eds.), Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020, volume 2696 of CEUR Workshop Proceedings, CEUR-WS.org, 2020. URL: http://ceur-ws.org/Vol-2696/paper_130.pdf.
  * Chekalina and Panchenko [2020] V. Chekalina, A. Panchenko,  Retrieving comparative arguments using deep pre-trained language models and NLU,  in: L. Cappellato, C. Eickhoff, N. Ferro, A. Névéol (Eds.), Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020, volume 2696 of CEUR Workshop Proceedings, CEUR-WS.org, 2020. URL: http://ceur-ws.org/Vol-2696/paper_210.pdf.
  * Schildwächter et al. [2019] M. Schildwächter, A. Bondarenko, J. Zenker, M. Hagen, C. Biemann, A. Panchenko,  Answering comparative questions: Better than ten-blue-links?,  in: L. Azzopardi, M. Halvey, I. Ruthven, H. Joho, V. Murdock, P. Qvarfordt (Eds.), Proceedings of the 2019 Conference on Human Information Interaction and Retrieval, CHIIR 2019, Glasgow, Scotland, UK, March 10-14, 2019, ACM, 2019, pp. 361–365. URL: https://doi.org/10.1145/3295750.3298916. doi:10.1145/3295750.3298916.
  * Fromm et al. [2019] M. Fromm, E. Faerman, T. Seidl,  TACAM: topic and context aware argument mining (2019) 99–106. URL: https://doi.org/10.1145/3350546.3352506. doi:10.1145/3350546.3352506.
  * Bondarenko et al. [2020] A. Bondarenko, M. Fröbe, M. Beloucif, L. Gienapp, Y. Ajjour, A. Panchenko, C. Biemann, B. Stein, H. Wachsmuth, M. Potthast, M. Hagen,  Overview of Touché 2020: Argument Retrieval,  in: L. Cappellato, C. Eickhoff, N. Ferro, A. Névéol (Eds.), Working Notes Papers of the CLEF 2020 Evaluation Labs, volume 2696 of CEUR Workshop Proceedings, 2020. URL: http://ceur-ws.org/Vol-2696/.
  * Hashemi et al. [2020] H. Hashemi, M. Aliannejadi, H. Zamani, W. B. Croft,  ANTIQUE: A non-factoid question answering benchmark 12036 (2020) 166–173. URL: https://doi.org/10.1007/978-3-030-45442-5_21. doi:10.1007/978-3-030-45442-5_21.
  * Potthast et al. [2019] M. Potthast, T. Gollub, M. Wiegmann, B. Stein,  TIRA Integrated Research Architecture,  in: N. Ferro, C. Peters (Eds.), Information Retrieval Evaluation in a Changing World, The Information Retrieval Series, Springer, Berlin Heidelberg New York, 2019. doi:10.1007/978-3-030-22948-1_5.
  * Braunstain et al. [2016] L. Braunstain, O. Kurland, D. Carmel, I. Szpektor, A. Shtok,  Supporting human answers for advice-seeking questions in CQA sites,  in: N. Ferro, F. Crestani, M. Moens, J. Mothe, F. Silvestri, G. M. D. Nunzio, C. Hauff, G. Silvello (Eds.), Advances in Information Retrieval - 38th European Conference on IR Research, ECIR 2016, Padua, Italy, March 20-23, 2016. Proceedings, volume 9626 of Lecture Notes in Computer Science, Springer, 2016, pp. 129–141. URL: https://doi.org/10.1007/978-3-319-30671-1_10. doi:10.1007/978-3-319-30671-1_10.
  * Wu et al. [2010] Q. Wu, C. J. Burges, K. M. Svore, J. Gao,  Adapting bboosting for information retrieval measures,  Information Retrieval 13 (2010) 254–270. URL: https://www.microsoft.com/en-us/research/publication/adapting-boosting-for-information-retrieval-measures/.
  * Burges et al. [2007] C. Burges, R. Ragno, Q. Le,  Learning to Rank with Nonsmooth Cost Functions,  in: B. Schölkopf, J. Platt, T. Hoffman (Eds.), Advances in Neural Information Processing Systems, volume 19, MIT Press, 2007. URL: https://proceedings.neurips.cc/paper/2006/file/af44c4c56f385c43f2529f9b1b018f6a-Paper.pdf.
  * Friedman [2002] J. Friedman,  Stochastic Gradient Boosting,  Computational Statistics & Data Analysis 38 (2002) 367–378. doi:10.1016/S0167-9473(01)00065-2.
  * Macdonald and Tonellotto [2020] C. Macdonald, N. Tonellotto,  Declarative Experimentation in Information Retrieval using PyTerrier,  in: K. Balog, V. Setty, C. Lioma, Y. Liu, M. Zhang, K. Berberich (Eds.), ICTIR ’20: The 2020 ACM SIGIR International Conference on the Theory of Information Retrieval, Virtual Event, Norway, September 14-17, 2020, ACM, 2020, pp. 161–168. URL: https://dl.acm.org/doi/10.1145/3409256.3409829.
  * Chekalina et al. [2021] V. Chekalina, A. Bondarenko, C. Biemann, M. Beloucif, V. Logacheva, A. Panchenko,  Which is better for deep learning: Python or MATLAB? Answering Comparative Questions in Natural Language,  in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, Association for Computational Linguistics, Online, 2021, pp. 302–311. URL: https://www.aclweb.org/anthology/2021.eacl-demos.36.
  * Kocabas et al. [2013] I. Kocabas, B. Dincer, B. Karaoğlan,  A Nonparametric Term Weighting Method for Information Retrieval Based on Measuring the Divergence from Independence,  Information Retrieval (2013) 1–24. doi:10.1007/s10791-013-9225-4.
  * MacAvaney et al. [2019] S. MacAvaney, A. Yates, A. Cohan, N. Goharian,  CEDR: Contextualized Embeddings for Document Ranking (2019) 1101–1104. URL: https://doi.org/10.1145/3331184.3331317. doi:10.1145/3331184.3331317.