*mps*

# Effective potential of scalar-tensor gravity with quartic self-interaction
of scalar field

A. Arbuzov<EMAIL_ADDRESS>Bogoliubov Laboratory of Theoretical Physics,
JINR, Dubna, 141980, Russia  Dubna State University, Universitetskaya str. 19,
Dubna, 141982, Russia  B. Latosh<EMAIL_ADDRESS>Bogoliubov Laboratory of
Theoretical Physics, JINR, Dubna, 141980, Russia  Dubna State University,
Universitetskaya str. 19, Dubna, 141982, Russia  A. Nikitenko
<EMAIL_ADDRESS>Bogoliubov Laboratory of Theoretical Physics, JINR,
Dubna, 141980, Russia  Dubna State University, Universitetskaya str. 19,
Dubna, 141982, Russia

###### Abstract

One-loop effective potential of scalar-tensor gravity with a quartic scalar
field self-interaction is evaluated up to first post-Minkowskian order. The
potential develops an instability in the strong field regime which is expected
from an effective theory. Depending on model parameters the instability region
can be exponentially far in a strong field region. Possible applications of
the model for inflationary scenarios are highlighted. It is shown that the
model can enter the slow-roll regime with a certain set of parameters.

##  1 Introduction

Scalar-tensor models of gravity occupy a special place in the landscape of
modified gravity. They provide a minimal modification of general relativity
yet have a vast phenomenology. Valuable examples of scalar-tensor gravity are
provided by inflation theories. A scalar field can drive inflation if it
either has a potential of a special form or a non-minimal coupling with
gravity [1, 2, 3, 4, 5].

Two following models should be highlighted due to their specific features. The
first one is the Starobinsky inflation [1]. This model extends general
relativity with a curvature squared term $R^{2}$ which drives inflation. The
term is generated at the one-loop level within quantum theory111At the one
loop level the following operators are generated $R^{2}$, $R_{\mu\nu}^{2}$,
$R_{\mu\nu\alpha\beta}^{2}$, $\square R$. The latter two shall be omitted as
they are reduced to a complete derivative in four dimensions. $R_{\mu\nu}^{2}$
term can be brought to the Weyl tensor squared which is conformally invariant.
Therefore it is irrelevant for Friedmann cosmologies as the Friedmann
spacetime is conformally flat. so it can be naturally included in a low-energy
effective action. Thus one can argue that the Starobinsky model describes
inflation as an effect dynamically generated at the loop level. Lastly, the
Starobinsky inflation, together with any $f(R)$-gravity, can be mapped on a
scalar-tensor gravity [6, 7, 8, 9, 10, 11, 12]. The second model worth to be
mentioned is the specific Horndeski model with the following kinetic coupling
$G^{\mu\nu}\nabla_{\mu}\phi\nabla_{\nu}\phi$ where $G^{\mu\nu}$ is the
Einstein tensor [13, 14, 3]. It has certain similarities with the Starobinsky
inflation. Firstly, this kinetic coupling can drive inflation [3, 4, 15].
Secondly, such a non-minimal coupling is also dynamically induced at the one-
loop level [16].

These models show that it is important to study quantum effects in scalar-
tensor gravity as they may provide natural candidates for inflationary models.
In particular, in paper [17] one-loop effective potentials of some simplest
scalar-tensor models were studied. It was found that they generate effective
potentials which have areas with small slow-roll parameters therefore they may
provide viable inflationary scenarios.

In this paper we study a more sophisticated scalar-tensor model and evaluate
its effective potential. In the previous paper [17] only models without scalar
field self-interaction were considered for the sake of simplicity. Here we
study the model given by the following action:

$\displaystyle\mathcal{A}=\int
d^{4}x\sqrt{-g}\,\left[-\cfrac{2}{\kappa^{2}}\,R-\cfrac{1}{2}\,\phi\,(\square+m^{2})\,\phi-\cfrac{\lambda}{4!}\,\phi^{4}\right].$
(1)

Here $\lambda$ is a dimensionless coupling constant chosen to be positive for
the sake of stability; $\kappa$ is related to the Newton constant $G_{N}$:

$\displaystyle\kappa^{2}=32\pi G_{N}.$ (2)

We also only consider the case with a non-broken symmetry $(m^{2}>0)$. The
broken symmetry case requires a separate treatment and we will address this
issue elsewhere.

There are several reasons to study this model. Firstly, the model serves as a
natural extension of the previous studies [18, 17, 16]. In particular, it
would be possible to separate an influence of the scalar field self-
interaction on an effective potential. Secondly, the model has only two
dimensional parameters which are the Planck scale $\kappa^{-1}$ and the scalar
field mass $m$. The scalar field self-interaction coupling $\lambda$ is
dimensionless. This allows one to negate the issue of a mass scale hierarchy
as it is natural to assume that the scalar field mass $m$ should be much
smaller that the Planck mass. Thirdly, in a flat spacetime the scalar sector
of model (1) is renormalizable although gravity ruins this feature, as we will
see. The effective potential technique provides a framework capable to soften
some renormalization issues and to consistently study quantum effects
associated with gravity. Finally, model (1) can serve as an analogy to the
Higgs sector of the Standard Model. It may provide a useful insight in Higgs
inflation scenarios [19, 20, 21, 22].

This paper is organized as follows. In the next section we briefly discuss the
effective potential technique and apply it for model (1). We discuss the
structure of the leading gravitational corrections and show that certain
corrections can be resummed in an analogy to the well-known case [18]. In
Section 3 we discuss features of the potential and its possible
implementations for inflation. In Appendix A we give a more detailed
description of the calculation of the effective potential.

##  2 Effective potential

The effective potential technique is well-known within quantum field theory
and very well presented in [23]. Because of this we will not discuss it in
detail. However, a few comments on its implementation for gravity is due.

We constraint ourselves only to perturbative quantum gravity. Therefore we
assume that the complete spacetime metric $g_{\mu\nu}$ is composed of a flat
background $\eta_{\mu\nu}$ and small perturbations $h_{\mu\nu}$:

$\displaystyle g_{\mu\nu}=\eta_{\mu\nu}+\kappa\,h_{\mu\nu}.$ (3)

The action describing gravity within perturbative approach shall be expanded
in an infinite series with respect to $h_{\mu\nu}$. Therefore the complete
action will contain an infinite number of terms describing $h_{\mu\nu}$ self-
interactions and $h_{\mu\nu}$ couplings to matter. Not all of these terms are
relevant in the effective potential context. Terms describing interactions of
perturbations $h_{\mu\nu}$ can be omitted completely as they do not contribute
to the effective potential of the scalar field at the one-loop level.
Similarly, at the one-loop level among all terms describing matter
gravitational coupling only terms linear in $\kappa$ are relevant. This allows
one to reduce the infinite perturbative expansion of an action to an
expression with a finite number of terms. Finally, the gravitational field
gauge should be fixed. We use the following gauge-fixing term which allows one
to decouple Faddeev-Popov ghosts:

$\displaystyle\mathcal{A}_{\text{gf}}$ $\displaystyle=\int
d^{4}x\left(\partial_{\mu}h^{\mu\nu}-\cfrac{1}{2}\,\partial^{\mu}h\right)^{2}.$
(4)

Therefore within the perturbative approach action (1) is reduced to the
following form:

$\displaystyle\begin{split}\mathcal{A}+\mathcal{A}_{\text{gf}}=&\int
d^{4}x\Bigg{[}-\cfrac{1}{2}\,h^{\mu\nu}\left(\cfrac{1}{2}\,C_{\mu\nu\alpha\beta}\square\right)h^{\alpha\beta}-\cfrac{1}{2}\,\phi\,(\square+m^{2})\phi\\\
&-\cfrac{\lambda}{4!}\,\phi^{4}-\cfrac{\kappa}{4}\,h^{\mu\nu}\left[C_{\mu\nu\alpha\beta}\,\partial^{\alpha}\phi\,\partial^{\beta}\phi+\eta_{\mu\nu}\,m^{2}\,\phi^{2}\right]+\cfrac{\kappa}{2}\,h^{\mu\nu}\,\eta_{\mu\nu}\cfrac{\lambda}{4!}\,\phi^{4}\Bigg{]}+\text{irrelevant
terms}.\end{split}$ (5)

Here tensor $C$ is defined as follows:

$\displaystyle
C_{\mu\nu\alpha\beta}=\eta_{\mu\alpha}\eta_{\nu\beta}+\eta_{\mu\beta}\eta_{\nu\alpha}-\eta_{\mu\nu}\eta_{\alpha\beta}\,.$
(6)

This action generates the following set of tree-level Feynman rules:

$\displaystyle\begin{gathered}\fmffile{pic_{A}01}\fmfgraph*(30,30)\fmfleft{L}\fmfright{R}\fmf{dashes}{L,R}\end{gathered}$
$\displaystyle=i\,\cfrac{1}{k^{2}-m^{2}}\,,$
$\displaystyle\begin{gathered}\fmffile{pic_{A}02}\fmfgraph*(30,30)\fmfleft{L}\fmfright{R}\fmf{dbl_{w}iggly}{L,R}\fmflabel{$\mu\nu$}{L}\fmflabel{$\alpha\beta$}{R}\end{gathered}$
$\displaystyle\hskip 19.91684pt=i\,\cfrac{\,C_{\mu\nu\alpha\beta}}{k^{2}}\,,$
$\displaystyle\begin{gathered}\fmffile{pic_{A}03}\fmfgraph*(30,30)\fmfleft{L1,L2}\fmfright{R1,R2}\fmf{dashes}{L1,V,L2}\fmf{dashes}{R1,V,R2}\fmfdot{V}\end{gathered}=-i\,\cfrac{\lambda}{4!}\,,$
(7)
$\displaystyle\begin{gathered}\fmffile{pic_{A}04}\fmfgraph*(30,30)\fmfleft{L}\fmfright{R1,R2}\fmf{dbl_{w}iggly}{L,V}\fmf{dashes}{R1,V,R2}\fmflabel{$\mu\nu$}{L}\fmflabel{$p$}{R1}\fmflabel{$q$}{R2}\fmfdot{V}\end{gathered}\hskip
14.22636pt$
$\displaystyle=i\,\cfrac{\kappa}{4}\,\left(C_{\mu\nu\alpha\beta}\,p^{\alpha}\,q^{\beta}-\eta_{\mu\nu}\,m^{2}\right),$
$\displaystyle\begin{gathered}\fmffile{pic_{A}05}\fmfgraph*(40,40)\fmftop{T}\fmfleft{L1,L2}\fmfright{R1,R2}\fmf{dbl_{w}iggly,tension=.3}{T,V}\fmf{dashes}{L1,V,L2}\fmf{dashes}{R1,V,R2}\fmflabel{$\mu\nu$}{T}\fmfdot{V}\end{gathered}=-i\,\cfrac{\kappa}{2}\,\eta_{\mu\nu}\,\cfrac{\lambda}{4!}\,.$
(9)

We only consider the case with $m^{2}>0$ corresponding to a non-broken
symmetry of the original action (1). There are both physical and technical
considerations motivating such a decision. First and foremost, if the symmetry
of the original action (1) is spontaneously broken and the scalar field
potential has a well-known form
$V(\phi)=\lambda\,\left(\phi^{2}-\phi_{0}^{2}\right)^{2}$, then one shall use
different dynamical variables. The scalar field variable $\phi$ describes
perturbations about a false vacuum which is unstable. Because of this the
corresponding effective potential will be also unstable222In matter of fact,
the evaluated expression for the effective potential (32) becomes non-
Hermitian if $m^{2}<0$. which is a mere reflection of the poor choice of
dynamical variables. Naturally, one shall use variables that describe scalar
field perturbations about the true ground state which, in turn, will spawn a
new set of Feynman rules that shall be used to evaluate the corresponding
effective potential. Consequently, the broken symmetry case presents an
independent problem which shall be addressed elsewhere.

From the physical point of view, the broken symmetry case also presents a
separate problem. It is expected to find a broken symmetry in a low energy
regime where a scalar field experience small perturbations about the true
vacuum state. The non-broken symmetry case, on the other hand, is expected to
describe a high energy regime when a scalar field experience strong
excitations. The later case is more suitable for the inflation paradigm as at
the beginning of inflation a scalar field is expected to be highly excited.
Because of these reasons within this paper we only consider the non-broken
symmetry case.

Let us turn to a discussion of the effective potential. To find the one-loop
scalar field effective potential it is required, firstly, to find all
$n$-point connected Green functions $G^{c}_{n}(x_{1},\cdots,x_{n})$; secondly,
to obtain all $n$-point vertex functions $\Gamma_{n}(x_{1},\cdots,x_{n})$ from
the corresponding Green functions via an amputation of external lines;
thirdly, to calculate the effective action $\Gamma$ as the following series
[23]:

$\displaystyle\Gamma=\sum\limits_{n=1}^{\infty}\,\cfrac{1}{n!}\,\int
d^{4}x_{1}\cdots
d^{4}x_{n}\,\Gamma_{n}(x_{1},\cdots,x_{n})\,\varphi(x_{1})\cdots\varphi(x_{n}).$
(10)

Here $\varphi(x)=\langle 0\lvert\widehat{\phi}\rvert 0\rangle$ is the
expectation value of the quantum scalar field $\phi$. This shows that an
effective potential is generated by one-loop one-particle irreducible
diagrams. In turn, a few topologically equivalent diagrams contribute to each
connected Green function so the vertex functions receive an additional
“combinatorial” factor that partially cancels $1/n!$ in each term of the
effective action expansion.

In this section we will not discuss the calculation of these “combinatorial”
factors but rather focus on the renormalization of the potential and its
features. A more detailed discussion of the calculations is given in Appendix
A.

The model (1) is well studied without gravity. We will refer to this case as
to the leading order (LO) contribution as it is of the order
$\mathcal{O}(\kappa^{0})$. All corrections associated with quantum gravity
appear at least at $\mathcal{O}(\kappa^{2})$ level and will provide next-to-
leading order (NLO) contribution. At the leading order the effective potential
is generated by the following series:

$\displaystyle\begin{gathered}\fmffile{pic_{S}1_{0}1}\fmfgraph*(40,40)\fmfleft{L}\fmfright{R}\fmf{dashes}{L,V,R}\fmf{dashes,right=1}{V,V}\fmfdot{V}\end{gathered}+\begin{gathered}\fmffile{pic_{S}1_{0}2}\fmfgraph*(40,40)\fmfleft{L1,L2}\fmfright{R1,R2}\fmf{dashes}{L1,VL,L2}\fmf{dashes}{R1,VR,R2}\fmf{dashes,right=1,tension=.3}{VL,VR,VL}\fmfdot{VL,VR}\end{gathered}+\begin{gathered}\fmffile{pic_{S}1_{0}3}\fmfgraph*(40,40)\fmfleft{L1,L2,L3,L4}\fmfright{R1,R2,R3,R4}\fmftop{T1,T2,T3,T4}\fmfbottom{B1,B2,B3,B4}\fmf{dashes,tension=.4}{V1,V2,V3,V1}\fmf{dashes}{L2,V1,B2}\fmf{dashes}{B3,V2,R2}\fmf{dashes}{T2,V3,T3}\fmfdot{V1,V2,V3}\end{gathered}+\cdots\to\Delta
V_{\text{eff,LO}}.$ (13)

The expression for the effective potential reads

$\displaystyle\Delta V_{\text{eff,LO}}=$
$\displaystyle\cfrac{1}{2}\,\int\cfrac{d^{4}_{E}k}{(2\pi)^{4}}\,\ln\left[1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}\right]$
(14)
$\displaystyle\begin{split}=&\,\cfrac{m^{2}}{2}\,\varphi^{2}\,\left(\cfrac{\lambda}{16\pi^{2}}\right)\left[\cfrac{1}{d-4}-\cfrac{3}{4}+\cfrac{1}{2}\,\gamma-\cfrac{3}{2}\,\ln
2\pi-\cfrac{1}{2}\,\ln
2+\cfrac{1}{2}\ln\cfrac{\frac{\lambda}{2}\varphi^{2}+m^{2}}{\mu^{2}}\right]\\\
&+\cfrac{\lambda}{4!}\,\varphi^{4}\,\left(\cfrac{3\lambda}{16\pi^{2}}\right)\left[\cfrac{1}{d-4}-\cfrac{3}{4}+\cfrac{1}{2}\,\gamma-\cfrac{3}{2}\,\ln
2\pi-\cfrac{1}{2}\,\ln
2+\cfrac{1}{2}\ln\cfrac{\frac{\lambda}{2}\varphi^{2}+m^{2}}{\mu^{2}}\right]\\\
&+\cfrac{m^{4}}{64\pi^{2}}\,\ln\left[1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{m^{2}}\right]+\mathcal{O}(d-4).\end{split}$
(15)

Within this paper we used dimensional regularization in $d$ dimensions with
$\mu$ being the renormalization scale. Unlike the massless case studied in
[18], potential (14) is regular in $\varphi\to 0$ limit due to the presence of
a non-vanishing scalar field mass.

Next-to-leading order corrections to potential (14) are controllable. Firstly,
the one-loop effective potential is generated only by one-loop one-particle
irreducible diagrams with external lines being all scalars carrying vanishing
momenta. Because of this only a few terms from the infinite expansion of (1)
in a series with respect to $h_{\mu\nu}$ are relevant. Namely, terms
describing self-interactions of $h_{\mu\nu}$ do not contribute to the
potential. Secondly, among all terms describing gravitational coupling of the
scalar field only terms linear in $\kappa$ are relevant. The other terms
either contribute at higher orders of $\kappa$ or at the higher loop orders.
Finally, because of the structure of the one-loop one-particle irreducible
diagrams virtual gravitons (associated with $h_{\mu\nu}$) can only appear in
the loop. These considerations allow one to split all possible one-loop one-
particle irreducible diagrams in three separate series. These three series
contributing at next-to-leading order to the effective potential read

$\displaystyle\begin{gathered}\fmffile{pic_{S}2_{0}1}\fmfgraph(50,50)\fmfleft{L}\fmfright{R}\fmf{dashes,tension=3}{L,V}\fmf{dashes,left=1}{V,VR}\fmf{dbl_{w}iggly,right=1}{V,VR}\fmf{dashes,tension=3}{VR,R}\fmfdot{V,VR}\end{gathered}+\begin{gathered}\fmffile{pic_{S}2_{0}2}\fmfgraph(50,50)\fmfleft{L}\fmfright{R}\fmftop{T0,T1,T2,T3,T4}\fmfbottom{B}\fmf{dashes,tension=3}{L,V}\fmf{phantom}{T2,VT}\fmf{phantom,left=1}{V,VR}\fmf{dbl_{w}iggly,right=1}{V,VR}\fmf{dashes,tension=3}{VR,R}\fmf{phantom}{T2,VT}\fmf{phantom}{VT,B}\fmffreeze\fmf{dashes}{T1,VT,T3}\fmf{dashes}{V,VT,VR}\fmfdot{V,VR,VT}\end{gathered}+\begin{gathered}\fmffile{pic_{S}2_{0}3}\fmfgraph(80,40)\fmfleft{L}\fmfright{R}\fmftop{T1,T2,T3,T4,T5,T6}\fmfbottom{B1,B2,B3,B4}\fmf{phantom,tension=3}{L,VL}\fmf{phantom,tension=3}{R,VR}\fmf{phantom}{VL,VR}\fmffreeze\fmf{phantom}{T2,VT1,T3}\fmf{phantom}{T4,VT2,T5}\fmf{phantom}{VT1,B2}\fmf{phantom}{VT2,B3}\fmf{phantom}{VT1,VT2}\fmffreeze\fmfdot{VL,VR,VT1,VT2}\fmf{dashes}{L,VL}\fmf{dashes}{R,VR}\fmf{dashes}{T2,VT1}\fmf{dashes}{T3,VT1}\fmf{dashes}{T4,VT2}\fmf{dashes}{T5,VT2}\fmf{dashes}{VL,VT1,VT2,VR}\fmf{dbl_{w}iggly,right=.4}{VL,VR}\end{gathered}+\cdots\to\Delta
V_{\text{eff,NLO,1}}\,,$ (19)
$\displaystyle\begin{gathered}\fmffile{pic_{S}3_{0}1}\fmfgraph(50,50)\fmfleft{L0,L1,L2,L3,L4}\fmfright{R}\fmf{dashes}{L1,V}\fmf{dashes}{L2,V}\fmf{dashes}{L3,V}\fmf{dashes,left=1}{V,VR}\fmf{dbl_{w}iggly,right=1}{V,VR}\fmf{dashes,tension=2}{VR,R}\fmfdot{V,VR}\end{gathered}+\begin{gathered}\fmffile{pic_{S}3_{0}2}\fmfgraph(50,50)\fmfleft{L0,L1,L2,L3,L4}\fmfright{R}\fmftop{T0,T1,T2,T3,T4}\fmfbottom{B}\fmf{dashes}{L1,V}\fmf{dashes}{L2,V}\fmf{dashes}{L3,V}\fmf{phantom}{T2,VT}\fmf{phantom,left=1}{V,VR}\fmf{dbl_{w}iggly,right=1}{V,VR}\fmf{dashes,tension=2.7}{VR,R}\fmf{phantom}{T2,VT}\fmf{phantom}{VT,B}\fmffreeze\fmf{dashes}{T1,VT,T3}\fmf{dashes}{V,VT,VR}\fmfdot{V,VR,VT}\end{gathered}+\begin{gathered}\fmffile{pic_{S}3_{0}3}\fmfgraph(80,40)\fmfleft{L0,L1,L2,L3,L4}\fmfright{R}\fmftop{T1,T2,T3,T4,T5,T6}\fmfbottom{B1,B2,B3,B4}\fmf{phantom,tension=3}{L2,VL}\fmf{phantom,tension=3}{VR,R}\fmf{phantom}{VL,VR}\fmf{dashes}{T2,VT1}\fmf{dashes}{T3,VT1}\fmf{dashes}{T4,VT2}\fmf{dashes}{T5,VT2}\fmf{phantom}{VT1,B2}\fmf{phantom}{VT2,B3}\fmf{phantom}{VT1,VT2}\fmffreeze\fmf{dashes}{L1,VL}\fmf{dashes}{L2,VL}\fmf{dashes}{L3,VL}\fmf{dashes}{VR,R}\fmf{dbl_{w}iggly,right=.5}{VL,VR}\fmfdot{VL,VR,VT1,VT2}\fmf{dashes}{VL,VT1,VT2,VR}\end{gathered}+\cdots\to\Delta
V_{\text{eff,NLO,2}}\,.$ (23)
$\displaystyle\begin{gathered}\fmffile{pic_{S}4_{0}1}\fmfgraph(50,50)\fmfleft{L0,L1,L2,L3,L4}\fmfright{R0,R1,R2,R3,R4}\fmf{dashes}{L1,V}\fmf{dashes}{L2,V}\fmf{dashes}{L3,V}\fmf{dashes,left=1}{V,VR}\fmf{dbl_{w}iggly,right=1}{V,VR}\fmf{dashes}{VR,R1}\fmf{dashes}{VR,R2}\fmf{dashes}{VR,R3}\fmfdot{V,VR}\end{gathered}+\begin{gathered}\fmffile{pic_{S}4_{0}2}\fmfgraph(50,50)\fmfleft{L0,L1,L2,L3,L4}\fmfright{R0,R1,R2,R3,R4}\fmftop{T0,T1,T2,T3,T4}\fmfbottom{B}\fmf{dashes}{L1,V}\fmf{dashes}{L2,V}\fmf{dashes}{L3,V}\fmf{phantom}{T2,VT}\fmf{phantom,left=1}{V,VR}\fmf{dbl_{w}iggly,right=1}{V,VR}\fmf{dashes}{VR,R1}\fmf{dashes}{VR,R2}\fmf{dashes}{VR,R3}\fmf{phantom}{T2,VT}\fmf{phantom}{VT,B}\fmffreeze\fmf{dashes}{T1,VT,T3}\fmf{dashes}{V,VT,VR}\fmfdot{V,VR,VT}\end{gathered}+\begin{gathered}\fmffile{pic_{S}4_{0}3}\fmfgraph(80,40)\fmfleft{L0,L1,L2,L3,L4}\fmfright{R0,R1,R2,R3,R4}\fmftop{T1,T2,T3,T4,T5,T6}\fmfbottom{B1,B2,B3,B4}\fmf{phantom,tension=3}{L2,VL}\fmf{phantom,tension=3}{VR,R2}\fmf{phantom}{VL,VR}\fmf{dashes}{T2,VT1}\fmf{dashes}{T3,VT1}\fmf{dashes}{T4,VT2}\fmf{dashes}{T5,VT2}\fmf{phantom}{VT1,B2}\fmf{phantom}{VT2,B3}\fmf{phantom}{VT1,VT2}\fmffreeze\fmf{dashes}{L1,VL}\fmf{dashes}{L2,VL}\fmf{dashes}{L3,VL}\fmf{dashes}{VR,R1}\fmf{dashes}{VR,R2}\fmf{dashes}{VR,R3}\fmf{dbl_{w}iggly,right=.5}{VL,VR}\fmfdot{VL,VR,VT1,VT2}\fmf{dashes}{VL,VT1,VT2,VR}\end{gathered}+\cdots\to\Delta
V_{\text{eff,NLO,3}}\,.$ (27)

In series (19) we collect all contributions which are spawned by the
gravitational interaction with the scalar field kinetic energy, described by
term $\sqrt{-g}\,g^{\mu\nu}\nabla_{\mu}\phi\nabla_{\nu}\phi$. To be exact, the
first term of series (19) is generated only by the gravitational scalar self-
energy which is due to its kinetic energy. The other terms of the series only
introduce contributions related with the scalar field self-interaction. In
series (23) we collect terms that are generated both by kinetic and potential
energy. The potential energy, in turn, is described by term
$\sqrt{-g}\phi^{4}$. Finally, in series (27) we collect contributions that are
generated by pure potential energy. The corresponding contributions to the
effective potential read

$\displaystyle\begin{split}\Delta
V_{\text{eff,NLO,1}}=&\cfrac{\kappa^{2}m^{4}}{\lambda}\,\int\cfrac{d^{4}_{E}k}{(2\pi)^{4}}\,\cfrac{1}{k^{2}}\,\cfrac{\cfrac{\frac{\lambda}{2}\varphi^{2}}{k^{2}+m^{2}}}{1+\cfrac{\frac{\lambda}{2}\varphi^{2}}{k^{2}+m^{2}}}\\\
=&\cfrac{m^{2}}{2}\,\varphi^{2}\,\cfrac{\kappa^{2}m^{2}}{8\pi^{2}}\,\left[-\cfrac{1}{d-4}+\cfrac{1}{2}\,-\cfrac{1}{2}\,\gamma+\cfrac{3}{2}\,\ln
2\pi+\cfrac{1}{2}\,\ln
2-\cfrac{1}{2}\,\ln\cfrac{\frac{\lambda}{2}\,\varphi^{2}+m^{2}}{\mu^{2}}\right]+\mathcal{O}(d-4)\,,\end{split}$
(28) $\displaystyle\begin{split}\Delta
V_{\text{eff,NLO,2}}=&\cfrac{\kappa^{2}m^{2}}{3}\,\varphi^{2}\int\cfrac{d^{4}_{E}k}{(2\pi)^{4}}\,\cfrac{1}{k^{2}}\,\cfrac{\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}}{1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}}\\\
=&\cfrac{\lambda}{4!}\,\varphi^{4}\,\cfrac{\kappa^{2}m^{2}}{2\pi^{2}}\left[-\cfrac{1}{d-4}+\cfrac{1}{2}\,-\cfrac{1}{2}\,\gamma+\cfrac{3}{2}\,\ln
2\pi+\cfrac{1}{2}\,\ln
2-\cfrac{1}{2}\,\ln\cfrac{\frac{\lambda}{2}\,\varphi^{2}+m^{2}}{\mu^{2}}\right]+\mathcal{O}(d-4),\end{split}$
(29) $\displaystyle\begin{split}\Delta
V_{\text{eff,NLO,3}}=&\cfrac{\kappa^{2}\,\lambda}{36}\,\varphi^{4}\int\cfrac{d^{4}_{E}k}{(2\pi)^{4}}\,\cfrac{1}{k^{2}}\,\cfrac{\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}}{1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}}\\\
=&\cfrac{\kappa^{2}}{6!}\,\varphi^{6}\,\cfrac{5\lambda^{2}}{4\pi^{2}}\,\left[-\cfrac{1}{d-4}+\cfrac{1}{2}\,-\cfrac{1}{2}\,\gamma+\cfrac{3}{2}\,\ln
2\pi+\cfrac{1}{2}\,\ln
2-\cfrac{1}{2}\,\ln\cfrac{\frac{\lambda}{2}\,\varphi^{2}+m^{2}}{\mu^{2}}\right]+\mathcal{O}(d-4).\end{split}$
(30)

It can be seen that these contributions share the expression in square
brackets, which is not a coincidence. From the form of the corresponding
diagrams (19), (23), and (27) it can be clearly seen that these series share
the divergent structure.

The 1-loop contribution to the effective potential of model (1) evaluated up
to $\mathcal{O}(\kappa^{2})$ order reads

$\displaystyle\Delta V_{\text{eff}}=$
$\displaystyle\cfrac{1}{2}\,\int\cfrac{d^{4}_{E}k}{(2\pi)^{4}}\,\ln\left[1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}\right]+\cfrac{\kappa^{2}m^{2}}{\lambda}\left(m^{2}+\cfrac{\lambda}{3}\,\varphi^{2}+\cfrac{\lambda^{2}\,\varphi^{4}}{36\,m^{2}}\right)\,\int\cfrac{d^{4}_{E}k}{(2\pi)^{4}}\,\cfrac{1}{k^{2}}\,\cfrac{\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}}{1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}+m^{2}}}+\mathcal{O}(\kappa^{4})$
(31)
$\displaystyle\begin{split}=&\cfrac{m^{4}}{64\pi^{2}}\ln\left[1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{m^{2}}\right]+\cfrac{m^{2}}{2}\,\varphi^{2}\left[\left(\cfrac{\lambda}{16\pi^{2}}-\cfrac{\kappa^{2}m^{2}}{8\pi^{2}}\right)\left(\cfrac{1}{d-4}+\cfrac{1}{2}\,\ln\cfrac{\frac{\lambda}{2}\,\varphi^{2}+m^{2}}{\mu^{2}}\right)+\mathcal{F}_{1}\right]\\\
&+\cfrac{\lambda}{4!}\,\varphi^{4}\left[\left(\cfrac{3\lambda}{16\pi^{2}}-\cfrac{\kappa^{2}m^{2}}{2\pi^{2}}\right)\left(\cfrac{1}{d-4}+\cfrac{1}{2}\,\ln\cfrac{\frac{\lambda}{2}\,\varphi^{2}+m^{2}}{\mu^{2}}\right)+\mathcal{F}_{2}\right]\\\
&+\cfrac{\kappa^{2}}{6!}\,\varphi^{6}\left[-\left(\cfrac{5\lambda^{2}}{4\pi^{2}}\right)\left(\cfrac{1}{d-4}+\cfrac{1}{2}\,\ln\cfrac{\frac{\lambda}{2}\,\varphi^{2}+m^{2}}{\mu^{2}}\right)+\mathcal{F}_{3}\right]+\mathcal{O}(\kappa^{4})\end{split}$
(32)

with $\mathcal{F}_{1}$, $\mathcal{F}_{2}$, and $\mathcal{F}_{3}$ being finite
parts defined as follows:

$\displaystyle\begin{split}\mathcal{F}_{1}=&\cfrac{\lambda}{16\pi^{2}}\left\\{-\cfrac{3}{4}\,+\cfrac{1}{2}\,\gamma-\cfrac{3}{2}\,\ln(2\pi)-\cfrac{1}{2}\,\ln(2)\right\\}+\cfrac{\kappa^{2}m^{2}}{8\pi^{2}}\left\\{\cfrac{1}{2}-\cfrac{1}{2}\,\gamma+\cfrac{3}{2}\ln(2\pi)+\cfrac{1}{2}\,\ln(2)\right\\}\,,\\\
\mathcal{F}_{2}=&\cfrac{3\,\lambda}{16\pi^{2}}\left\\{-\cfrac{3}{4}\,+\cfrac{1}{2}\,\gamma-\cfrac{3}{2}\,\ln(2\pi)-\cfrac{1}{2}\,\ln(2)\right\\}+\cfrac{\kappa^{2}m^{2}}{2\pi^{2}}\left\\{\cfrac{1}{2}-\cfrac{1}{2}\,\gamma+\cfrac{3}{2}\ln(2\pi)+\cfrac{1}{2}\,\ln(2)\right\\}\,,\\\
\mathcal{F}_{3}=&\cfrac{5\lambda^{2}}{4\pi^{2}}\,\left\\{\cfrac{1}{2}\,-\cfrac{1}{2}\,\gamma+\cfrac{3}{2}\,\ln
2\pi+\cfrac{1}{2}\,\ln 2\right\\}\,.\end{split}$ (33)

Expression (32) has a finite number of divergences, namely, only
$\varphi^{2}$, $\varphi^{4}$, and $\varphi^{6}$ terms shall be renormalized.
Unlike the flat spacetime case [18], new interaction $\varphi^{6}$ is
generated and it should be renormalized. The corresponding renormalization
problems are softened within the effective theory approach. The generation of
new non-renormalizable interactions at the loop level is expected as the
theory contains quantum gravity which by itself is non-renormalizable.
However, the effective potential (and the effective action) drives macroscopic
degrees of freedom $\varphi=\langle 0\lvert\phi\rvert 0\rangle$ and it is not
renormalized in the same sense as the microscopic action does. One shall
choose a normalization point $\varphi_{0}$ and renormalize all divergent
interactions on the values empirically measured in this point. But this
procedure do not impose any constraints on fundamental physics, it only
imposes the initial conditions taken from experiment.

We choose $\varphi_{0}=0$ as the normalization point. It can be done because
of the presence of a non-vanishing scalar field mass. Let us note that with
such a choice of the normalization point one shall pay a special attention to
$m\to 0$ limit. In that limit $\varphi_{0}=0$ can no longer be a suitable
normalization point, so the whole effective potential can become singular.
Further, such a choice of a normalization point corresponds to a physical
setup where the scalar field mass and self-interaction couplings are measured
in the low-energy regime. We normalize the effective potential on the observed
scalar field mass $m$, observed four-particle coupling $\lambda$, and observed
six-particle coupling $g$ with an introduction of the following counter-term:

$\displaystyle\begin{split}\Delta
V_{\text{c.t.}}=&\cfrac{g}{6!}\,\varphi^{6}-\cfrac{m^{2}}{2}\,\varphi^{2}\left[\left(\cfrac{\lambda}{16\pi^{2}}-\cfrac{\kappa^{2}m^{2}}{8\pi^{2}}\right)\left(\cfrac{1}{d-4}+\cfrac{1}{2}\,\ln\cfrac{m^{2}}{\mu^{2}}\right)+\mathcal{F}_{1}+\cfrac{\lambda}{64\pi^{2}}\right]\\\
&-\cfrac{\lambda}{4!}\,\varphi^{4}\left[\left(\cfrac{3\lambda}{16\pi^{2}}-\cfrac{\kappa^{2}m^{2}}{2\pi^{2}}\right)\left(\cfrac{1}{d-4}+\cfrac{1}{2}\,\ln\cfrac{m^{2}}{\mu^{2}}\right)+\mathcal{F}_{2}+\cfrac{3}{64\pi^{2}}\left(3\lambda-8\,\kappa^{2}m^{2}\right)\right]\\\
&-\cfrac{\kappa^{2}}{6!}\,\varphi^{6}\left[-\left(\cfrac{5\lambda^{2}}{4\pi^{2}}\right)\left(\cfrac{1}{d-4}+\cfrac{1}{2}\,\ln\cfrac{m^{2}}{\mu^{2}}\right)+\mathcal{F}_{3}\right]\,.\end{split}$
(34)

The renormalized effective potential reads

$\displaystyle\begin{split}V_{\text{eff,ren}}=&\ln\left[1+\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{m^{2}}\right]\left\\{\cfrac{m^{4}}{64\pi^{2}}+\cfrac{m^{2}}{2}\,\varphi^{2}\,\cfrac{\lambda-2m^{2}\kappa^{2}}{32\pi^{2}}+\cfrac{\lambda}{4!}\,\varphi^{4}\,\cfrac{3\lambda-8m^{2}\kappa^{2}}{32\pi^{2}}-\cfrac{\kappa^{2}}{6!}\,\varphi^{6}\,\cfrac{5\lambda^{2}}{8\pi^{2}}\right\\}\\\
&+\cfrac{m^{2}}{2}\,\left(1-\cfrac{\lambda}{64\pi^{2}}\right)\,\varphi^{2}+\cfrac{\lambda}{4!}\,\left(1-\cfrac{3(3\lambda-8\kappa^{2}m^{2})}{64\pi^{2}}\right)\,\varphi^{4}+\cfrac{g}{6!}\,\varphi^{6}\left(1-\cfrac{15\,\lambda^{2}}{32\pi^{2}}\,\cfrac{\lambda-2m^{2}\kappa^{2}}{g\,m^{2}}\right)\end{split}$
(35) $\displaystyle=$
$\displaystyle\cfrac{m^{2}}{2}\,\varphi^{2}+\cfrac{\lambda}{4!}\,\varphi^{4}+\cfrac{g}{6!}\,\varphi^{6}-\cfrac{\lambda^{3}\left(3\lambda+16\kappa^{2}m^{2}\right)}{36864\pi^{2}m^{4}}\,\varphi^{8}+\mathcal{O}\left(\varphi^{10}\right).$
(36)

The following comments on the renormalization shall be given. Firstly, we
shall note that in the most general case one can distinguish the microscopic
action mass $m$ and coupling $\lambda$ from the effective action (effective
potential) mass $m_{0}$ and couplings $\lambda_{0}$, $g_{0}$. We believe that
such an approach shall not be used in our case as it is physically irrelevant.
The symmetry of the original model is not broken therefore both $\phi$ and
$\varphi=\langle 0\lvert\widehat{\phi}\rvert 0\rangle$ describe scalar field
excitations about the same state. Therefore there is no reason to believe that
the effective action will develop parameters different from $m$ and $\lambda$.
The new scalar field interaction $\varphi^{6}$, on the other hand, is
generated dynamically by the effective theory. Because of this we can only
normalize the corresponding coupling to the value $g$ that supposed to be
measured in the low energy limit. Therefore the renormalized effective
potential has only four free parameters $m$, $\lambda$, $g$, and $\kappa$
which are measure in normalization point $\varphi_{0}=0$.

Secondly, we shall draw attention to $m\to 0$ limit. As it was noted above,
$\varphi_{0}=0$ is a suitable renormalization point if and only if the scalar
field has a non-vanishing mass $m\not=0$. Therefore, if one takes $m\to 0$
limit they should choose a different normalization point and introduce a
different counter-term. It can be clearly seen that before the renormalization
effective potential (32) remains regular in $m\to 0$ limit while the counter-
term (34) is not. Therefore the absence of a direct $m\to 0$ limit in (35) is
due to the choice of the normalization point only.

To proceed with the study we shall note that expression (35) contains four
free parameters which are the scalar field mass $m$, scalar field quartic
coupling $\lambda$, gravitational coupling $\kappa$, and scalar field
dimensional coupling $g$. It is more convenient to analyze this expression in
the scalar field mass units:

$\displaystyle\begin{split}\widetilde{V}_{\text{eff,ren}}=&\ln\left[1+\cfrac{\lambda}{2}\,\tilde{\varphi}^{2}\right]\left\\{\cfrac{1}{64\pi^{2}}+\cfrac{1}{2}\,\tilde{\varphi}^{2}\,\cfrac{\lambda-2\zeta^{2}}{32\pi^{2}}+\cfrac{\lambda}{4!}\,\tilde{\varphi}^{4}\,\cfrac{3\lambda-8\,\zeta^{2}}{32\pi^{2}}-\cfrac{1}{6!}\,\tilde{\varphi}^{6}\,\cfrac{5\lambda^{2}\,\zeta^{2}}{8\pi^{2}}\right\\}\\\
&+\cfrac{1}{2}\,\left(1-\cfrac{\lambda}{64\pi^{2}}\right)\,\tilde{\varphi}^{2}+\cfrac{\lambda}{4!}\,\left(1-\cfrac{3(3\lambda-8\,\zeta^{2})}{64\pi^{2}}\right)\,\tilde{\varphi}^{4}+\cfrac{\tilde{g}}{6!}\,\tilde{\varphi}^{6}\left(1-\cfrac{15\,\lambda^{2}}{32\pi^{2}}\,\cfrac{\lambda-2\zeta^{2}}{\tilde{g}}\right)\,.\end{split}$
(37)

Here $V_{\text{eff,ren}}=m^{4}\,\widetilde{V}_{\text{eff,ren}}$,
$\varphi=m\,\widetilde{\varphi}$, $\zeta=\kappa\,m$, $\tilde{g}=g\,m^{2}$. We
naturally assume that $\zeta\ll 1$ and the scalar field mass is much smaller
that the Planck scale.

First and foremost, let us address the asymptotic of effective potential (37).
In the weak field region $\tilde{\varphi}\sim 0$, the potential is regular due
to the presence of a non-vanishing mass:

$\displaystyle\widetilde{V}_{\text{eff,ren}}=\cfrac{1}{2}\,\tilde{\varphi}^{2}+\cfrac{\lambda}{4!}\,\tilde{\varphi}^{4}+\cfrac{\tilde{g}}{6!}\,\tilde{\varphi}^{6}+\mathcal{O}(\tilde{\varphi}^{8})\,.$
(38)

The first two terms of the expansion always have positive coefficients while
$\tilde{g}$ may be negative. This shows that the potential may have a local
maximum and an instability region. The existence of an instability region do
not pose a serious problem within an effective theory as the region can lie
outside the theory applicability domain.

The effective potential does develop an instability in the strong field region
($\tilde{\varphi}\to\infty$). In the strong field regime the leading
contribution to the effective potential reads

$\displaystyle\widetilde{V}_{\text{eff,ren}}\overset{\tilde{\varphi}\sim\infty}{\sim}\cfrac{1}{6!}\,\tilde{\varphi}^{6}\left\\{\tilde{g}\left(1-\cfrac{15\lambda^{2}}{32\pi^{2}}\,\cfrac{\lambda-2\,\zeta^{2}}{\tilde{g}}\right)-\cfrac{5\lambda^{2}\,\zeta^{2}}{8\pi^{2}}\ln\left[1+\cfrac{\lambda}{2}\,\tilde{\varphi}^{2}\right]\right\\}\,.$
(39)

The $\log$-term coefficient of this expression is always negative thus the
potential always approaches $-\infty$ in $\tilde{\varphi}\to\infty$ limit. At
the same time the first term in the brackets can be both negative and
positive. If the term is negative, then (39) is negative even in the weak
field regime. Consequently, the potential develops a local maximum that marks
the stable region in a vicinity of $\tilde{\varphi}=0$. On the contrary, if
the term is positive, then (39) is positive in the weak field region and
becomes negative only at $\tilde{\varphi}\gg 1$. The potential, in tern,
develops a local maximum far in the strong field region.

The field value $\tilde{\varphi}_{\text{crit}}$ where (39) becomes negative we
call the critical value. The value can be derived from (39):

$\displaystyle\left(\tilde{\varphi}_{\text{crit}}\right)^{2}=\cfrac{2}{\lambda}\left[\exp\left[\cfrac{8\pi^{2}\,\tilde{g}}{5\,\lambda^{2}\zeta^{2}}\left(1-\cfrac{15\lambda^{2}}{32\pi^{2}}\,\cfrac{\lambda-2\,\zeta^{2}}{\tilde{g}}\right)\right]-1\right]\,.$
(40)

The critical field value is real if the following condition is met

$\displaystyle\cfrac{8\pi^{2}\,\tilde{g}}{5\,\lambda^{2}\zeta^{2}}\left(1-\cfrac{15\lambda^{2}}{32\pi^{2}}\,\cfrac{\lambda-2\,\zeta^{2}}{\tilde{g}}\right)<0\,.$
(41)

This condition is satisfied if either of the following two conditions is
fulfilled:

$\displaystyle
0<\tilde{g}<\cfrac{15\lambda^{2}}{32\pi^{2}}\,(\lambda-2\zeta^{2})\,,$ (42)
$\displaystyle\tilde{g}<0\text{ and
}\tilde{g}<\cfrac{15\lambda^{2}}{32\pi^{2}}\,(\lambda-2\zeta^{2})\,.$ (43)

These results show the following behavior of the effective potential. In the
weak field region the effective potential growth with $\tilde{\varphi}$. It
growth to the local maximum point $\tilde{\varphi}_{\text{max}}$ and then
strives to $-\infty$. The position of the local maximum is related with the
strong field asymptotic. If either (42) or (43) is met, then the local maximum
$\tilde{\varphi}_{\text{max}}$ exists in the weak field region. This is due to
the fact that contribution (39) is negative even at $\tilde{\varphi}\sim 0$.
If conditions (42) and (43) are not hold, then the local maximum is expected
to appear about $\tilde{\varphi}_{\text{crit}}$ (40) which is exponential in
model parameters and resides far in the strong field region.

Because of the complicated form of potential (37), it is more suitable to
study it numerically. We present an explicit form of the effective potential
(37) with $\lambda=10^{-3}$, $\zeta=10^{-6}$, and $\tilde{g}=-1$ in Figure 1.
With such parameters condition (43) is fulfilled and the local maximum appears
in the weak field region $\tilde{\varphi}_{\text{max}}\simeq 3.31126$.

We would like to demonstrate a plot showing the exponential behavior of the
local maximum but this is impossible as $\tilde{\varphi}_{\text{crit}}$ grows
extremely rapidly. Let us take parameter $\lambda=10^{-3}$, $\zeta=10^{-6}$
from the previous case for the sake of illustration. We shall consider
$\tilde{g}>0$ case which does not satisfy (42). Condition (42) puts the
following constraint on $\tilde{g}$:

$\displaystyle\tilde{g}<\left.\cfrac{15\lambda^{2}}{32\pi^{2}}\,(\lambda-2\zeta^{2})\right|_{\lambda=10^{-3},\,\zeta=10^{-6}}\sim
4.74\times 10^{-11}\,.$ (44)

If we set $\tilde{g}=10^{-10}$, which is by a single order of amplitude bigger
then the critical value, then the critical field value
$\tilde{\varphi}_{\text{crit}}$ will be separated from the weak field region
by many orders of magnitude:

$\displaystyle\tilde{\varphi}_{\text{crit}}\Big{|}_{\lambda=10^{-3},\,\zeta=10^{-6}}\sim
2.8\times 10^{360089499}\,.$ (45)

Finally, if condition (41) is met then the position of the local maximum also
growth as the model approaches strong interaction regime ($\lambda\to 1$). We
present plots showing such a behavior in Figure 2 and Figure 3.

These results are important in the context of possible implications for
inflation. As the potential (35) has an instability region, the local maximum
marks the applicability region for the model. Initial conditions for
inflation, in turn, shall also be taken from
$\tilde{\varphi}\leq\tilde{\varphi}_{\text{max}}$. If condition (41) is
fulfilled, then $\tilde{\varphi}_{\text{max}}$ can be calculated and direct
constraints on the initial conditions compatible with the effective model can
be established. If condition (41) is not fulfilled, then
$\tilde{\varphi}_{\text{max}}$ is exponentially big and resides far beyond the
Planck region. Therefore in that case no reasonable constraints on the initial
conditions can be imposed.

The complete analysis of inflation scenarios described by the discussed
effective model lies beyond the scope of this paper and will be presented
elsewhere. However, simple numerical estimates provide a way to show that the
model has at least one area of parameters where slow-roll conditions are
satisfied. Let us focus on $\tilde{g}<0$ region. Slow-roll parameters
evaluated via the scalar field effective potential (35) read

$\displaystyle\varepsilon$
$\displaystyle=\cfrac{2}{3}\,\cfrac{1}{\kappa^{2}}\,\left(\cfrac{1}{V_{\text{eff,ren}}}\,\cfrac{d}{d\varphi}\,V_{\text{eff,ren}}\right)^{2}\,,$
$\displaystyle\eta$
$\displaystyle=\cfrac{4}{3}\,\cfrac{1}{\kappa^{2}}\,\cfrac{1}{V_{\text{eff,ren}}}\,\cfrac{d^{2}}{d\varphi^{2}}\,V_{\text{eff,ren}}\,.$
(46)

In Figure 4 we present plots which show that for a certain range of parameters
the effective potential enters an area where $\varepsilon<1$ and $\eta<1$.
This area corresponds to $\tilde{\varphi}\sim\tilde{\varphi}_{\text{max}}$,
therefore the models indeed has room for slow-roll inflationary scenarios.

##  3 Discussion and conclusions

In this paper we studied a simple scalar-tensor model (1) and the effective
potential generated by it. One-loop contributions to the effective potential
up to $\mathcal{O}(\kappa^{2})$ order is given by expression (32). The
effective potential is regular in $\varphi\to 0$ limit therefore we choose the
normalization point $\varphi_{0}=0$ and renormalize the effective potential
with counter-term (34). The renormalized effective potential is given by (35).
It contains a new non-renormalizable effective $\varphi^{6}$ interaction
dynamically generated at the one-loop level. The interaction is normalized at
its observed value in the low-energy limit.

The effective potential (35) always has an instability region. If condition
(41) is satisfied, then the instability region exists in the weak field
region. If condition (41) is not satisfied, then the instability region exists
exponentially far in the strong field regime. The instability region naturally
marks the effective model applicability domain. Finally, despite the fact that
the potential is unstable, it does not generate new non-trivial minima in the
stability region.

The main result of this paper is the renormalized effective potential (35)
while its applications shall be studied elsewhere. One of the main directions
of further studies will be implementation of (35) for inflation. In Figure 4
we present plots which explicitly show that it is possible to find a set of
model parameters which guarantee the existence of a region with small slow-
roll parameters. This fact shows that there is room for a standard slow-roll
inflation within this model. A more detailed analysis of inflationary
scenarios within this model will be performed in a forthcoming paper.

## Acknowledgment

The work (B.L.) was supported by the Foundation for the Advancement of
Theoretical Physics and Mathematics “BASIS”.

##  Appendix A Calculation of effective potentials

In order to calculate an effective potential one shall calculate connected
Green functions, recover vertex functions, and re-sum a series generating the
effective potential. The technique is well described in [23] and we will not
discus it in detail. However, the calculation of one-loop connected Green
functions, which is the corner stone of the technique, must be discussed.

We use the continual integral technique to evaluate connected Green functions.
An $n$-point one-loop connected Green function $G^{c}_{n}(x_{1},\cdots,x_{n})$
is given by the following expression [23]:

$\displaystyle\begin{split}i\,G^{c}_{N}(x_{1},\cdots,x_{N})=&\exp\left[i\,\mathcal{A}_{\text{int}}\left(\cfrac{\delta}{\delta(iJ)}\,,\,\cfrac{\delta}{\delta(iJ^{\mu\nu})}\right)\right]\\\
&\times\left(\prod\limits_{i=1}^{n}\cfrac{\delta}{\delta(iJ(x_{i}))}\right)\left\\{\,\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\,\exp\left[\cfrac{i}{2}\,J^{\mu\nu}\cdot\mathcal{G}_{\mu\nu\alpha\beta}\cdot
J^{\alpha\beta}\right]\right\\}\Bigg{|}_{J,J^{\mu\nu}=0}^{\text{connected}}.\end{split}$
(47)

Here $\mathcal{A}_{\text{int}}(\phi,h_{\mu\nu})$ is the part of action (1)
describing interaction; $\mathcal{G}$ and $\mathcal{G}_{\mu\nu\alpha\beta}$
are Green functions

$\displaystyle\mathcal{G}(x)$
$\displaystyle=\int\cfrac{d^{4}k}{(2\pi)^{4}}\,\cfrac{1}{-k^{2}+m^{2}}\,e^{-ikx}\,,$
$\displaystyle\mathcal{G}_{\mu\nu\alpha\beta}(x)$
$\displaystyle=\cfrac{1}{2}\,C_{\mu\nu\alpha\beta}\,\int\cfrac{d^{4}k}{(2\pi)^{4}}\,\cfrac{1}{-k^{2}}\,e^{-ikx}\,;$
(48)

dots denote scalar products defined as follows:

$\displaystyle J\cdot\mathcal{G}\cdot J$ $\displaystyle=\int
d^{4}x\,d^{4}y\,J(x)\,\mathcal{G}(x-y)\,J(y)\,,$
$\displaystyle(\mathcal{G}\cdot J)(x)$ $\displaystyle=\int
d^{4}y\,\mathcal{G}(x-y)\,J(y)\,;$ (49)

$J$, $J^{\mu\nu}$ are external currents. The notations on the right hand side
of (47) note that we only account for contributions corresponding to connected
diagrams with vanishing external currents. Expression (47) can be simplified.
Within the original expression (47) variational derivatives associated with an
external field act on the exponent containing the scalar field Green function.
Such an expression does contain contributions corresponding to both connected
and disconnected diagrams. To exclude irrelevant contribution one shall only
take the term that explicitly account for connected contributions which reads

$\displaystyle\begin{split}i\,G^{c}_{N}(x_{1},\cdots,x_{N})=&\exp\left[i\,\mathcal{A}_{\text{int}}\left(\cfrac{\delta}{\delta(iJ)}\,\cfrac{\delta}{\delta(iJ^{\mu\nu})}\right)\right]\\\
&\times\left\\{\left(\prod\limits_{i=1}^{n}(\mathcal{G}\cdot
J)(x_{i})\right)\,\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\,\exp\left[\cfrac{i}{2}\,J^{\mu\nu}\cdot\mathcal{G}_{\mu\nu\alpha\beta}\cdot
J^{\alpha\beta}\right]\right\\}\Bigg{|}_{J,J^{\mu\nu}=0}^{\text{connected}}.\end{split}$
(50)

Here each variational derivative associated with an external line acted on
each own Green function effectively spawning external lines. The remaining
variation derivatives to be evaluated can only spawn internal lines or
connections to external lines.

Within the discussed model at the leading order a $2N$-point connected Green
functions read

$\displaystyle
i\,G_{2N}^{c}(x_{1},\cdots,x_{2N})=\exp\left[-i\,\cfrac{\lambda}{4!}\int
d^{4}\zeta\,\left(\cfrac{\delta}{\delta(iJ(\zeta))}\right)^{4}\right]\,\prod\limits_{i=1}^{2N}\cfrac{\delta}{\delta(iJ(x_{i}))}\,\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\Bigg{|}_{J=0}^{\text{connected}}$ (51)
$\displaystyle\begin{split}&=\cfrac{1}{N!}\,\left(-i\,\cfrac{\lambda}{4!}\right)^{N}\begin{pmatrix}4\\\
2,2\end{pmatrix}^{N}\,\int d^{4}\zeta_{1}\cdots
d^{4}\zeta_{N}\left\\{\left(\prod\limits_{i=1}^{N}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{2}\right)\left(\prod\limits_{j=1}^{N}(\mathcal{G}\cdot
J)(x_{j})\right)\right\\}\\\
&\times\left\\{\left(\prod\limits_{n=1}^{N}\left(\cfrac{\delta}{\delta(iJ(\zeta_{n}))}\right)^{2}\right)\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\right\\}\Bigg{|}_{J=0}^{\text{connected}}.\end{split}$ (52)

Here factor $1/N!$ is due to the exponent expansion;

$\displaystyle\begin{pmatrix}4\\\ 2,2\end{pmatrix}=\cfrac{4!}{2!2!}$ (53)

is the multinomial coefficient that appears as we calculated derivatives of a
product. The first term in curve brackets

$\displaystyle\left(\prod\limits_{i=1}^{N}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{2}\right)\left(\prod\limits_{j=1}^{N}(\mathcal{G}\cdot
J)(x_{j})\right)$ (54)

corresponds to external lines connected to internal interaction vertices. This
contribution contains $(2N)!$ term which is the number of ways by which $2N$
external lines can be connected to an internal one-loop diagram. After an
amputation of external lines all these contributions become equivalent so at
the level of vertex functions contribution of this term is reduced to a
$(2N)!$ factor. The second curve bracket term

$\displaystyle\left\\{\left(\prod\limits_{n=1}^{N}\left(\cfrac{\delta}{\delta(iJ(\zeta_{n}))}\right)^{2}\right)\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\right\\}\Bigg{|}_{J=0}^{\text{connected}}$ (55)

describes scalar field propagators connecting internal interaction vertices.
It contains many term but all of them are equivalent due to variable
redefinition. After a suitable redefinition of variables, the contribution can
be brought to the following form:

$\displaystyle\left\\{\left(\prod\limits_{n=1}^{N}\left(\cfrac{\delta}{\delta(iJ(\zeta_{n}))}\right)^{2}\right)\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\right\\}\Bigg{|}_{J=0}^{\text{connected}}=(N-1)!\,2^{N-1}\,\cfrac{1}{i}\,\mathcal{G}(\zeta_{1}-\zeta_{2})\cdots\cfrac{1}{i}\,\mathcal{G}(\zeta_{N}-\zeta_{1}).$
(56)

Such a $2N$-point connected Green function generates the following
contribution to the effective action $\Gamma$:

$\displaystyle\begin{split}&\int d^{4}\zeta_{1}\cdots
d^{4}\zeta_{2N}\,\cfrac{1}{(2N)!}\,\Gamma_{2N}(\zeta_{1},\cdots,\zeta_{2N})\,\varphi(\zeta_{1})\cdots\varphi(\zeta_{2N})\\\
&=\int
d^{4}x\,\cfrac{\varphi(x)^{2N}}{(2N)!}\,\cfrac{1}{N!}\,\left(-\cfrac{\lambda}{4!}\,\cfrac{4!}{2!}\right)^{N}\,\cfrac{1}{2^{N}}\,(2N)!\,(N-1)!\,2^{N-1}\,(-i)\,\int\cfrac{d^{4}k}{(2\pi)^{4}}\left(\cfrac{1}{-k^{2}+m^{2}}\right)^{N}\end{split}$
(57) $\displaystyle=\int
d^{4}x(-i)\int\cfrac{d^{4}k}{(2\pi)^{4}}\,\cfrac{1}{2N}\,\left(\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}-m^{2}}\right)^{N}\,.$
(58)

This expression (up to the inclusion of a non-vanishing mass term) matches the
well-known result [18].

Green functions that generate series (19) are defined similarly to the
previous case:

$\displaystyle\begin{split}&i\,G_{2N}^{c}\\!=\\!\cfrac{1}{2!}\int
d^{4}\\!z_{1}d^{4}\\!z_{2}\left(-i\,\cfrac{\kappa}{4}\,m^{2}\right)^{2}\eta^{\mu\nu}\eta^{\alpha\beta}\,\cfrac{\delta}{\delta(iJ^{\mu\nu}(z_{1}))}\,\cfrac{\delta}{\delta(iJ^{\alpha\beta}(z_{2}))}\left(\cfrac{\delta}{\delta(iJ(z_{1}))}\right)^{2}\left(\cfrac{\delta}{\delta(iJ(z_{2}))}\right)^{2}\\\
&\times\cfrac{\left(-i\,\cfrac{\lambda}{4!}\right)^{N-1}}{(N-1)!}\int\left(\prod\limits_{n=1}^{N-1}d^{4}\zeta_{n}\,\left(\cfrac{\delta}{\delta(iJ(\zeta_{n}))}\right)^{4}\right)\,\left\\{\prod\limits_{i=1}^{2N}(\mathcal{G}\cdot
J)(x_{i})\,e^{\frac{i}{2}\,J\cdot\mathcal{G}\cdot
J}\,e^{\frac{i}{2}\,J^{\mu\nu}\cdot\mathcal{G}_{\mu\nu\alpha\beta}\cdot
J^{\alpha\beta}}\right\\}\Bigg{|}_{J,J^{\mu\nu}=0}^{\text{connected}}\,.\end{split}$
(59)

Scalar quartic interaction and scalar-graviton interaction can be placed in
two independent exponents as $\delta/\delta(iJ)$ commutes with
$\delta/\delta(iJ^{\mu\nu})$. Factor $1/2!$ is due to an expansion of the
exponent describing gravitational interaction. Factor $1/(N-1)!$ is due to an
expansion of scalar field interaction exponent. This expression can be
simplified in a similar way:

$\displaystyle\begin{split}i\,G_{2N}^{c}=&\cfrac{1}{2!}\int
d^{4}\\!z_{1}d^{4}\\!z_{2}\left(-i\,\cfrac{\kappa}{4}\,m^{2}\right)^{2}\,\eta^{\mu\nu}\,\eta^{\alpha\beta}\,\cfrac{1}{i}\,\mathcal{G}_{\mu\nu\alpha\beta}(z_{1}-z_{2})\left(\cfrac{\delta}{\delta(iJ(z_{1}))}\right)^{2}\left(\cfrac{\delta}{\delta(iJ(z_{2}))}\right)^{2}\\\
&\times\cfrac{\left(-i\cfrac{\lambda}{4!}\right)^{N-1}}{(N-1)!}\int\left(\prod\limits_{n=1}^{N-1}d^{4}\zeta_{n}\,\left(\cfrac{\delta}{\delta(iJ(\zeta_{n}))}\right)^{4}\right)\,\left\\{\prod\limits_{i=1}^{2N}(\mathcal{G}\cdot
J)(x_{i})\,\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\right\\}\Bigg{|}_{J=0}^{\text{connected}}\,.\end{split}$ (60)
$\displaystyle\begin{split}=&\cfrac{1}{2!}\int
d^{4}\\!z_{1}\,d^{4}\\!z_{2}\,\left(-i\,\cfrac{\kappa}{4}\,m^{2}\right)^{2}\,\cfrac{\left(-i\cfrac{\lambda}{4!}\right)^{N-1}}{(N-1)!}\begin{pmatrix}2\\\
1,1\end{pmatrix}^{2}\\!\begin{pmatrix}4\\\
2,2\end{pmatrix}^{N-1}\,\int\left(\prod\limits_{n=1}^{N-1}d^{4}\\!\zeta_{n}\right)\\\
&\times\left\\{\prod\limits_{i=1}^{2}\,\cfrac{\delta}{\delta(iJ(z_{i}))}\,\prod\limits_{i=1}^{N-1}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{2}\,\left(\prod\limits_{i=1}^{2N}(\mathcal{G}\cdot
J)(x_{i})\right)\right\\}\\\
&\times\left\\{\eta^{\mu\nu}\,\eta^{\alpha\beta}\,\cfrac{1}{i}\,\mathcal{G}_{\mu\nu\alpha\beta}(z_{1}-z_{2})\prod\limits_{i=1}^{2}\,\cfrac{\delta}{\delta(iJ(z_{i}))}\,\prod\limits_{i=1}^{N-1}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{2}\,\times\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\right\\}\Bigg{|}_{J=0}^{\text{connected}}\,.\end{split}$ (61)

At the level of the effective action this contribution can be simplified in
exactly the same way as the LO contribution. The first expression in curved
brackets describes all possible ways to connect $2N$ external lines to an
internal structure. At the level of the effective action this contribution is
reduced to $(2N)!$ factor. The second expression in curved brackets describes
all ways to arrange internal lines. In full analogy with the previous case all
term in this expression match up to a redefinition of integration variable.
The corresponding contribution will generate factor $(N-1)!2^{N-1}$ in the
effective action.The corresponding part of the effective action reads:

$\displaystyle\begin{split}&\int d^{4}\zeta_{1}\cdots
d^{4}\zeta_{2N}\,\cfrac{1}{(2N)!}\,\Gamma_{2N}(\zeta_{1},\cdots,\zeta_{2N})\,\varphi(\zeta_{1})\cdots\varphi(\zeta_{2N})\\\
&=\int
d^{4}x(-i)\cfrac{\kappa^{2}m^{4}}{\lambda}\int\cfrac{d^{4}k}{(2\pi)^{4}}\,\cfrac{1}{-k^{2}}\,\left(\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}-m^{2}}\right)^{N}\,.\end{split}$
(62)

Similar calculations are performed for series (23). Green functions are given
by the following expression (with $N\geq 2$):

$\displaystyle\begin{split}iG_{2N}^{c}=&\int
d^{4}\\!z_{1}\\!\left(-i\cfrac{\kappa}{2}\,\cfrac{\lambda}{4!}\right)\eta^{\mu\nu}\cfrac{\delta}{\delta(iJ^{\mu\nu}(z_{1}))}\left(\cfrac{\delta}{\delta(iJ(z_{1}))}\right)^{4}\int
d^{4}\\!z_{2}\left(-i\cfrac{\kappa}{4}\right)\,m^{2}\eta^{\alpha\beta}\cfrac{\delta}{\delta(iJ^{\alpha\beta}(z_{2}))}\left(\cfrac{\delta}{\delta(iJ(z_{2}))}\right)^{2}\\\
&\int\left(\prod\limits_{i=1}^{N-2}d^{4}\zeta_{i}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{4}\right)\left(-i\cfrac{\lambda}{4!}\right)^{N-2}\,\cfrac{1}{(N-2)!}\left[\prod\limits_{j=1}^{2N}(\mathcal{G}\cdot
J)(x_{j})\,e^{\frac{i}{2}\,J\cdot\mathcal{G}\cdot
J}\,e^{\frac{i}{2}\,J^{\mu\nu}\cdot\mathcal{G}_{\mu\nu\alpha\beta}\cdot
J^{\alpha\beta}}\right]\end{split}$ (63)

The same considerations allow one to bring the expression to the following
form:

$\displaystyle\begin{split}iG_{2N}^{c}=&\left(-i\cfrac{\kappa}{2}\,\cfrac{\lambda}{4!}\right)\left(-i\cfrac{\kappa}{4}\right)\,m^{2}\,\cfrac{1}{(N-2)!}\,\left(-i\cfrac{\lambda}{4!}\right)^{N-2}\begin{pmatrix}4\\\
3,1\end{pmatrix}\begin{pmatrix}2\\\ 1,1\end{pmatrix}\begin{pmatrix}4\\\
2,2\end{pmatrix}^{N-2}\int
d^{4}\\!z_{1}d^{4}\\!z_{2}\prod\limits_{n=1}^{N-2}d^{4}\\!\zeta_{n}\\\
&\times\eta^{\mu\nu}\cfrac{1}{i}\,\mathcal{G}_{\mu\nu\alpha\beta}(z_{1}-z_{2})\eta^{\alpha\beta}\,\left\\{\left(\cfrac{\delta}{\delta(iJ(z_{1}))}\right)^{3}\cfrac{\delta}{\delta(iJ(z_{2}))}\prod\limits_{i=1}^{N-2}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{2}\prod\limits_{j=1}^{2N}(\mathcal{G}\cdot
J)(x_{j})\right\\}\\\
&\times\left\\{\eta^{\mu\nu}\,\cfrac{1}{i}\,\mathcal{G}_{\mu\nu\alpha\beta}(z_{1}-z_{2})\eta^{\alpha\beta}\cfrac{\delta}{\delta(iJ(z_{1}))}\cfrac{\delta}{\delta(iJ(z_{2}))}\prod\limits_{l=1}^{N_{2}}\left(\cfrac{\delta}{\delta(iJ(\zeta_{l}))}\right)^{2}\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\right\\}\Bigg{|}_{J=0}^{\text{connected}}.\end{split}$ (64)

In full analogy with the previous cases the first term in curved brackets
describes external scalar lines and at the level of an effective action
reduces to a factor $(2N)!$. The second term describes the structure of
internal lines connecting interaction vertices and at the level of effective
action it is reduced to factor $(N-2)!2^{N-2}$. The factor depends on $N-2$ as
the Green function $G_{2N}^{c}$ contains only $N-2$ quartic scalar interaction
vertices. Finally, such a Green function generates the following contribution
to the effective action:

$\displaystyle\begin{split}&\int d^{4}\zeta_{1}\cdots
d^{4}\zeta_{2N}\,\cfrac{1}{(2N)!}\,\Gamma_{2N}(\zeta_{1},\cdots,\zeta_{2N})\,\varphi(\zeta_{1})\cdots\varphi(\zeta_{2N})\\\
&=\int
d^{4}x\,(-i)\int\cfrac{d^{4}k}{(2\pi)^{4}}\,\cfrac{\kappa^{2}m^{2}}{3}\,\varphi^{2}\,\cfrac{1}{-k^{2}}\left(\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}-m^{2}}\right)^{N-1}.\end{split}$
(65)

Finally, the same actions shall be done with series (27). The green function
generating the series read ($N\geq 3$):

$\displaystyle\begin{split}i\,G_{2N}=&\cfrac{1}{2}\,\int
d^{4}\\!z_{2}d^{4}\\!z_{2}\left(-i\,\cfrac{\kappa}{2}\,\cfrac{\lambda}{4!}\right)^{2}\,\eta^{\mu\nu}\eta^{\alpha\beta}\,\cfrac{\delta}{\delta(iJ^{\mu\nu}(z_{1}))}\cfrac{\delta}{\delta(iJ^{\alpha\beta}(z_{2}))}\left(\cfrac{\delta}{\delta(iJ(z_{1}))}\right)^{4}\left(\cfrac{\delta}{\delta(iJ(z_{2}))}\right)^{4}\\\
&\cfrac{1}{(N-3)!}\,\left(-i\cfrac{\lambda}{4!}\right)^{N-3}\int\left(\prod\limits_{i=1}^{N-3}d^{4}\\!\zeta_{i}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{4}\right)\left[\prod\limits_{j=1}^{2N}\,(\mathcal{G}\cdot
J)(x_{j})\,e^{\frac{i}{2}\,J\cdot\mathcal{G}\cdot
J}\,e^{\frac{i}{2}\,J^{\mu\nu}\cdot\mathcal{G}_{\mu\nu\alpha\beta}\cdot
J^{\alpha\beta}}\right]\end{split}$ (66)
$\displaystyle\begin{split}=&\cfrac{1}{2}\left(-i\,\cfrac{\lambda}{4!}\right)^{N-1}\left(\cfrac{\kappa}{2}\right)^{2}\,\cfrac{1}{(N-3)!}\begin{pmatrix}4\\\
3,1\end{pmatrix}^{2}\begin{pmatrix}4\\\ 2,2\end{pmatrix}^{N-3}\int
d^{4}\\!z_{1}d^{4}\\!z_{2}\prod\limits_{i=1}^{N-3}d^{4}\zeta_{i}\\\
&\times\eta^{\mu\nu}\cfrac{1}{i}\,\mathcal{G}_{\mu\nu\alpha\beta}(z_{1}-z_{2})\eta^{\alpha\beta}\left\\{\left(\cfrac{\delta}{\delta(iJ(z_{1}))}\right)^{3}\left(\cfrac{\delta}{\delta(iJ(z_{2}))}\right)^{3}\prod\limits_{i=1}^{N-3}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{2}\prod\limits_{j=1}^{2N}(\mathcal{G}\cdot
J)(x_{j})\right\\}\\\
&\left\\{\cfrac{\delta}{\delta(iJ(z_{1}))}\cfrac{\delta}{\delta(iJ(z_{2}))}\prod\limits_{i=1}^{N-3}\left(\cfrac{\delta}{\delta(iJ(\zeta_{i}))}\right)^{2}\,\exp\left[\cfrac{i}{2}\,J\cdot\mathcal{G}\cdot
J\right]\right\\}\,.\end{split}$ (67)

This expression, in turn, provides the following contribution to the effective
action:

$\displaystyle\begin{split}&\int d^{4}\zeta_{1}\cdots
d^{4}\zeta_{2N}\,\cfrac{1}{(2N)!}\,\Gamma_{2N}(\zeta_{1},\cdots,\zeta_{2N})\,\varphi(\zeta_{1})\cdots\varphi(\zeta_{2N})\\\
&=\int
d^{4}x\,(-i)\,\int\cfrac{d^{4}k}{(2\pi)^{4}}\,\cfrac{\kappa^{2}\lambda\,\varphi^{4}}{36}\,\cfrac{1}{-k^{2}}\,\left(\cfrac{\frac{\lambda}{2}\,\varphi^{2}}{k^{2}-m^{2}}\right)^{N-2}\,.\end{split}$
(68)

## References

  * [1] Alexei A. Starobinsky.  A New Type of Isotropic Cosmological Models Without Singularity.  Phys. Lett., B91:99–102, 1980.  doi:10.1016/0370-2693(80)90670-X.
  * [2] Andrei D. Linde.  Chaotic Inflation.  Phys. Lett., 129B:177–181, 1983.  doi:10.1016/0370-2693(83)90837-7.
  * [3] Tsutomu Kobayashi, Masahide Yamaguchi, and Jun’ichi Yokoyama.  Generalized G-inflation: Inflation with the most general second-order field equations.  Prog. Theor. Phys., 126:511–529, 2011.  arXiv:1105.5723, doi:10.1143/PTP.126.511.
  * [4] Alexei A. Starobinsky, Sergey V. Sushkov, and Mikhail S. Volkov.  The screening Horndeski cosmologies.  JCAP, 1606(06):007, 2016.  arXiv:1604.06085, doi:10.1088/1475-7516/2016/06/007.
  * [5] Dmitry S. Gorbunov and Valery A. Rubakov.  Introduction to the theory of the early universe: Cosmological perturbations and inflationary theory.  2011\.  URL: http://www.DESY.eblib.com/patron/FullRecord.aspx?p=737613, doi:10.1142/7874.
  * [6] R. H. Dicke.  Mach’s principle and invariance under transformation of units.  Phys. Rev., 125:2163–2167, 1962.  doi:10.1103/PhysRev.125.2163.
  * [7] Kei-ichi Maeda.  Towards the Einstein-Hilbert Action via Conformal Transformation.  Phys. Rev. D, 39:3159, 1989.  doi:10.1103/PhysRevD.39.3159.
  * [8] David Wands.  Extended gravity theories and the Einstein-Hilbert action.  Class. Quant. Grav., 11:269–280, 1994.  arXiv:gr-qc/9307034, doi:10.1088/0264-9381/11/1/025.
  * [9] Guido Magnano and Leszek M. Sokolowski.  On physical equivalence between nonlinear gravity theories and a general relativistic selfgravitating scalar field.  Phys. Rev. D, 50:5039–5059, 1994.  arXiv:gr-qc/9312008, doi:10.1103/PhysRevD.50.5039.
  * [10] Valerio Faraoni, Edgard Gunzig, and Pasquale Nardone.  Conformal transformations in classical gravitational theories and in cosmology.  Fund. Cosmic Phys., 20:121, 1999.  arXiv:gr-qc/9811047.
  * [11] Antonio De Felice and Shinji Tsujikawa.  f(R) theories.  Living Rev. Rel., 13:3, 2010.  arXiv:1002.4928, doi:10.12942/lrr-2010-3.
  * [12] Xavier Calmet and Boris Latosh.  The Spectrum of Quantum Gravity.  Phys. Part. Nucl. Lett., 16(6):656–661, 2019.  arXiv:1907.10024, doi:10.1134/S1547477119060426.
  * [13] Gregory Walter Horndeski.  Second-order scalar-tensor field equations in a four-dimensional space.  Int. J. Theor. Phys., 10:363–384, 1974.  doi:10.1007/BF01807638.
  * [14] G. W. Horndeski.  Conservation of Charge and the Einstein-Maxwell Field Equations.  J. Math. Phys., 17:1980–1987, 1976.  doi:10.1063/1.522837.
  * [15] N. Avdeev and A. Toporensky.  On viability of inflation in non-minimal kinetic coupling theory.  2 2021.  arXiv:2103.00556.
  * [16] Boris Latosh.  One-loop effective scalar-tensor gravity.  Eur. Phys. J. C, 80(9):845, 2020.  arXiv:2004.00927, doi:10.1140/epjc/s10052-020-8371-2.
  * [17] Andrej Arbuzov and Boris Latosh.  Effective potential of scalar-tensor gravity.  Class. Quant. Grav., 38(1):015012, 2021.  arXiv:2007.06306, doi:10.1088/1361-6382/abc572.
  * [18] Sidney R. Coleman and Erick J. Weinberg.  Radiative Corrections as the Origin of Spontaneous Symmetry Breaking.  Phys. Rev., D7:1888–1910, 1973.  doi:10.1103/PhysRevD.7.1888.
  * [19] Fedor L. Bezrukov and Mikhail Shaposhnikov.  The Standard Model Higgs boson as the inflaton.  Phys. Lett. B, 659:703–706, 2008.  arXiv:0710.3755, doi:10.1016/j.physletb.2007.11.072.
  * [20] A. O. Barvinsky, A. Yu. Kamenshchik, and A. A. Starobinsky.  Inflation scenario via the Standard Model Higgs boson and LHC.  JCAP, 11:021, 2008.  arXiv:0809.2104, doi:10.1088/1475-7516/2008/11/021.
  * [21] Y. Akrami et al.  Planck 2018 results. X. Constraints on inflation.  Astron. Astrophys., 641:A10, 2020.  arXiv:1807.06211, doi:10.1051/0004-6361/201833887.
  * [22] F. Bezrukov, A. Magnin, M. Shaposhnikov, and S. Sibiryakov.  Higgs inflation: consistency and generalisations.  JHEP, 01:016, 2011.  arXiv:1008.5157, doi:10.1007/JHEP01(2011)016.
  * [23] I. L. Buchbinder, S. D. Odintsov, and I. L. Shapiro.  Effective action in quantum gravity.  1992\.

Figure 1: Effective potential (37) with $\lambda=10^{-3}$, $\zeta=10^{-6}$,
$\tilde{g}=-1$. The local maximum is $\tilde{\varphi}_{\text{max}}\simeq
3.31126$.

Figure 2: The dependence of the position of the local maximum of effective
potential (37) with respect to $\tilde{g}$

Figure 3: The dependence of the position of the local maximum of effective
potential (37) with respect to $\tilde{g}$ and $\lambda$. $\zeta=10^{-6}$.

Figure 4: Slow-roll parameters for the effective model. Lines start at
infinity with $\tilde{\varphi}=0$ and strive to $\varepsilon=0$, $\eta=0$
region with $\tilde{\varphi}\to\tilde{\varphi}_{\text{max}}$.


††thanks: Current affiliation††thanks: Current affiliation

# Continuous-variable quantum tomography of high-amplitude states

Ekaterina Fedotova<EMAIL_ADDRESS>Institute of Science and Technology
Austria, am Campus 1, 3400 Klosterneuburg, Austria  Nikolai Kuznetsov
Institute of Physics, Swiss Federal Institute of Technology Lausanne (EPFL),
CH-1015 Lausanne, Switzerland  Egor Tiunov  Quantum Research Center,
Technology Innovation Institute, Abu Dhabi, UAE  A. E. Ulanov  Deutsches
Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany  A. I.
Lvovsky<EMAIL_ADDRESS>Clarendon Laboratory, University of
Oxford, Parks Road, Oxford OX1 3PU, UK

###### Abstract

Quantum state tomography is an essential component of modern quantum
technology. In application to continuous-variable harmonic-oscillator systems,
such as the electromagnetic field, existing tomography methods typically
reconstruct the state in discrete bases, and are hence limited to states with
relatively low amplitudes and energies. Here we overcome this limitation by
utilizing a feed-forward neural network to obtain the density matrix directly
in the continuous position basis. An important benefit of our approach is the
ability to choose specific regions in the phase space for detailed
reconstruction. This results in relatively slow scaling of the amount of
resources required for the reconstruction with the state amplitude, and hence
allows us to dramatically increase the range of amplitudes accessible with our
method.

##  I Introduction

The ongoing rapid development of quantum technology results in increased
complexity of various quantum states we operate with [1, 2, 3, 4, 5, 6, 7, 8,
9, 10, 11]. This, in turn, raises the requirements to quantum state tomography
(QST) — the technique for reconstruction of the quantum state of a system from
measurements [12, 13, 14, 15, 16]. The higher dimension of the Hilbert space
can exponentially enlarge the amount of data required for a QST quorum, as
well as the amount of processing power required to restore the state. These
problems are often solved using machine learning and neural networks, which
allow QST to be carried out quickly and efficiently [17, 18, 19, 20, 21, 22].

A convenient choice of state representation for the QST with neural networks
is a basis of discrete variables [19, 17]. A discrete basis is often suitable
for systems naturally described in continuous-variable (CV) bases, such as the
harmonic oscillator. Indeed, for a long time, QST of harmonic oscillator
states in the optical, microwave and mechanical domains has been done in the
Fock basis [23, 18], even when the measurements were performed in the
continuous quadrature basis. We note that QST of harmonic oscillators
historically began in the CV basis [24] with the filtered back-projection
algorithm; however, this approach later became unpopular because of its
relative computational complexity and unphysical artefacts in the
reconstructed state [23].

However, the choice of the Fock basis for restoring the states of a harmonic
oscillator is optimal only when the state under consideration has a relatively
small amplitude. This is because the number of Fock terms needed to represent
a state grows quadratically with its amplitude. Moreover, the superexponential
factor of $(2^{n}n!)^{-1/2}$ present in the $n$-photon Fock state
wavefunction, which enters the QST algorithm, complicates the calculations for
high photon numbers. This is why the reconstruction space in the currently
published results on QST of a harmonic oscillator in the Fock basis is limited
to the subspace of 30 photons [18], up to our knowledge.

In this paper, we develop a new approach to QST of a harmonic oscillator with
a neural network where both the measurement and the reconstruction are in the
CV domain. Specifically, we reconstruct a density matrix $\rho(X,X^{\prime})$
in the position basis on a predefined finite coordinate grid. This enables us
to directly obtain density matrices of states with arbitrarily high
amplitudes, overcoming computational issues. Our method allows us to on those
areas of the phase space that are relevant to the state in question. For
example, the wave function of a coherent state $\ket{\alpha}$ takes nonzero
values in a narrow region around $X_{\alpha}=\alpha\sqrt{2}$ and we can query
the network with the position values mainly around $X_{\alpha}$, thereby
ensuring that it is aware of the density matrix structure around this region
particularly well. No _a priori_ information about the state is needed since
localized regions of interest can be inferred directly from the measurement
data. After the training, the neural network will correctly interpolate the
values of the density matrix corresponding to the coordinates between the grid
nodes [25, 26].

We anticipate our approach to be particularly useful for optical analogues of
Schrödinger’s cats, i.e. superpositions of coherent states of different
amplitudes and/or phases, which find broad application in CV quantum
information processing [6, 27, 28]. Similarly to coherent states, cat states
are well-localized in the quadrature space, making them amenable to our
method. We demonstrate our method to perform QST of cat states with amplitudes
up to $\alpha=40$, which in the Fock basis would require a reconstruction
space with up to $\sim 1800$ photons.

Figure 1: CV QST via a fully connected feed-forward neural network. Input
position value pairs $(X,X^{\prime})$ within the regions of interests (shaded
grey) are processed with the NN into real and imaginary values of a matrix
$A(X,X^{\prime})$, which is then transformed into a physically valid density
matrix via the Cholesky decomposition (3). The NN is trained using the
backpropagation algorithm to maximize the log-likelihood of the measured
quadrature set.

##  II Concept

We specialize to optical homodyne tomography [16, 15], which measures samples
of generalized quadrature
$\hat{X}_{\theta}=\hat{X}\cos\theta+\hat{P}\sin\theta$, where $\theta$ is the
phase of the local oscillator and can be controlled in the experiment (our
treatment is straightforwardly generalized to CV-QST based on other types of
measurement). $N$ homodyne measurements produce the set of amplitude and phase
pairs $\set{X_{\theta_{n}},\theta_{n}}_{n=1}^{N}$. We utilise likelihood
maximization approach with the log-likelihood functional which in our case is
defined as

$\mathbf{L}=-\sum_{n}\ln\left[P(X_{\theta_{n}},\theta_{n})\right],$ (1)

where $P(X_{\theta},\theta)$ is the probability density of obtaining a
measurement result $X_{\theta}$ for the local oscillator phase equal to
$\theta$. Minimization of $\mathbf{L}$ corresponds to finding the density
matrix that maximizes the probability to get the specified quadrature
distribution. This probability density is given (see Refs. [29] and Appendix)
by

$\displaystyle P(X_{\theta},\theta)$
$\displaystyle=\frac{1}{2\pi|\sin\theta|}\iint\rho(x,x^{\prime})\cdot$ (2a)
$\displaystyle\cdot\exp\left[-i\dfrac{x-x^{\prime}}{\sin\theta}\left(X_{\theta}-\cos\theta\dfrac{x+x^{\prime}}{2}\right)\right]dxdx^{\prime}.$
where the density matrix $\rho(x,x^{\prime})$ in the position basis is known
from the neural network output.

The above equation can give rise to numerical instabilities for small $\theta$
due to the denominator containing $\sin\theta$. To avoid the problem, we use
Eq. (2a) only for the quadratures with $|\sin\theta|\geq 1/\sqrt{2}$. When
$|\sin\theta|<1/\sqrt{2}$, we first apply the Fourier transform to compute the
density matrix $\rho(p,p^{\prime})$ in the momentum basis, and then find the
probabilities according to

$\displaystyle P(X_{\theta},\theta)$
$\displaystyle=\dfrac{1}{2\pi|\cos\theta|}\iint\rho(p,p^{\prime})\cdot$ (2b)
$\displaystyle\cdot\exp\left[-i\dfrac{p-p^{\prime}}{\cos\theta}\left(X_{\theta}+\sin\theta\dfrac{p+p^{\prime}}{2}\right)\right]\text{d}p\text{d}p^{\prime}.$

Figure 2: Wigner functions reconstructed from the experimental data with the
MaxLik method and our neural network for a single photon state (a), displaced
single photon state (b), engineered Fock superpositions up to the two-photon
level (c) and optical Schrödinger’s cat (d). Efficiency correction is applied
to each state (see Table I). Mutual fidelity between each pair of states
exceeds $0.994$.

To reconstruct the density matrix, we exploit a fully connected feed-forward
neural network (NN) [30] based on freely accessible PyTorch libraries. The
reconstruction process is shown in Fig. 1. NN takes a pair of coordinates as
input and outputs a single complex number $A(X,X^{\prime})$, which is
connected to the density matrix via the Cholesky decomposition [31]

$\rho(X,X^{\prime})=\dfrac{A^{\dagger}A}{{\rm Tr}(A^{\dagger}A)}.$ (3)

The motivation for this intermediate step is to ensure that the output density
matrix is Hermitian, semipositive definite and normalized [32].

In order to compute the density matrix, the NN is applied in sequence to all
pairs $\left(X,X^{\prime}\right)$ from a predefined grid. When all
$A(X,X^{\prime})$ are known, $\rho(X,X^{\prime})$ is calculated via Eq. (3).
To train the NN, we evaluate the loss functional $\mathbf{L}$, and iteratively
apply backpropagation to update the NN parameters.

To choose the grids for the position and momentum quadratures, we inspect the
experimental data for $\theta\approx 0$ and $\theta\approx\pi/2$ and find the
regions where the measured samples are localized. The grids must cover these
regions. The grid period ($\delta X,\delta P$) is chosen to ensure correct
Fourier transform. That is, large values of momentum quadratures present in
the state imply that the density matrix in the position space undergoes fast
oscillations, and vice versa. The grid must be sufficiently frequent to
capture these oscillations. We observed correct reconstruction by setting the
grid periods according to

$\displaystyle\delta X$ $\displaystyle\lesssim P_{\max}^{-1};$
$\displaystyle\delta P$ $\displaystyle\lesssim X_{\max}^{-1},$

where $X_{\max}$ and $P_{\max}$ are the highest quadrature values observed in
the measurement.

##  III Results

First, we test our method on several experimentally acquired sets of
measurements that correspond to prepared quantum optical states containing
only a few photons. The experimental apparatus is described e.g. in Refs. [33,
34]. We compare our technique with the discrete iterative likelihood-
maximization algorithm (MaxLik) [23], which reconstructs the state in the Fock
basis. We apply correction for linear losses as described in the Appendix. The
reconstruction NN featured three hidden layers, each containing 100 units. For
the NN training, the grid in the position and momentum spaces is chosen to
cover the interval $P,X\in[-4,4]$ with 80 equidistant intervals.

The states are listed in Table I and the results of the reconstruction are
shown in Fig. 2. In all cases, the mutual fidelity between the density
matrices obtained with the two methods exceeds $0.994$. To evaluate the
fidelity more precisely, we exploit the interpolation capability of the NN to
predict the density matrix values over a more frequent grid than during the
training. Specifically, the grid contains 400 equidistant position values over
the interval $[-4,4]$.

Table 1: Experimental states reconstructed in Fig. 2. The reconstruction involves correction for linear losses corresponding to the efficiency $\eta$. The fidelity shown is between the reconstructions via the NN and MaxLik [23] techniques. Fig. |  State | $\eta$ | Fidelity
---|---|---|---
2(a) |  single-photon | 0.56 | $>0.999$
2(b) |  displaced single-photon | 0.56 | $>0.999$
2(c) |  normalized superposition of Fock states $a_{0}\ket{0}+a_{1}\ket{1}+a_{2}\ket{2}$ with $a_{0}:a_{1}:a_{2}\sim-0.76:0.49:0.42$ [33] | 0.56 | $0.998$
2(d) |  normalized superposition of coherent states $\ket{\alpha}+\ket{-\alpha}$ with $|\alpha|=1.85$ squeezed by $3$ dB [34] | 0.62 | $0.994$
Figure 3: QST of Schrödinger’s cat states $\ket{\alpha}-\ket{-\alpha}$ with
MaxLik and CN NN from simulated quadrature data sets. (a): Simulated
quadrature data for $\alpha=5,40$ and $N=5\cdot 10^{3},20\cdot 10^{3}$ (top
and bottom panels, respectively). Bottom right insets show the zoomed-in
quadrature samples in the intersection region. Top right insets present ideal
probability distributions from the same regions, exhibiting fringe patterns.
(b): Real parts of the reconstructed density matrices for the data in (a).
(c): Reconstructed Wigner functions for the same states. The inset in the
bottom panel shows a fringe pattern in the central peak. Grey rectangles show
the reconstruction regions. d) Fidelity as a function of the amplitude
$\alpha$ for a fixed number of quadrature measurements ($5\cdot 10^{3}$ and
$20\cdot 10^{3}$). (e): dependence of the fidelity on the number of quadrature
measurements for fixed amplitudes.  Figure 4: Various reconstructed states.
(a): Two-headed Schrödinger’s cat with imaginary amplitude $\alpha=13i$. A
checkerboard pattern is observed in the position-basis density matrix. (b):
Three-headed cat state with amplitude $\alpha=13$. The Wigner function
contains three Gaussian peaks and three oscillating regions between each pair
of Gaussian peaks. (c): approximate Gottesman-Kitaev-Preskill state. Grey
rectangles in the Wigner function plots show the reconstruction regions.

To explore our method further and demonstrate the performance of the CV NN QST
approach in its full glory, we simulate quadrature measurement data sets for a
variety of high-amplitude states. In Fig. 3 we show Schrödinger’s cat states
$\ket{\alpha}-\ket{-\alpha}$ with different real amplitudes $\alpha$. These
states’ Wigner functions exhibit an oscillating pattern near the phase space
origin [Fig. 3(c)]. The density matrix in the coordinate representation
consists of two positive and two negative Gaussian peaks as shown in Fig.
3(b). In the momentum basis, the density matrix exhibits a rapidly oscillating
pattern with a Gaussian envelope centered around $p=0,p^{\prime}=0$.

We simulated quadrature measurement datasets with sizes ranging from $5\cdot
10^{3}$ to $10^{5}$ for cat amplitudes up to $\alpha=40$. The same NN as in
the previous section was used. The grid in the position space contains $360$
values distributed evenly over the two intervals
$[\pm\alpha\sqrt{2}-4.5,\pm\alpha\sqrt{2}+4.5]$, i.e. in the vicinity of the
expected Gaussian peaks. In the momentum basis, the grid is in the interval
$[-5,5]$, also with 360 equidistant values.

Figures 3(d,e) demonstrate the fidelity between the reconstructed and true cat
states as functions of the amplitude and number of quadrature measurements.
The value of each point is estimated using 5 sets of synthetically generated
measurements. As expected, the fidelity increases with the number of
quadratures acquired and decreases with the cat amplitude. Again, we compare
the NN method with MaxLik. The latter method works with cat amplitudes up to
$\alpha=10$. Higher amplitudes are inaccessible for MaxLik because of the
aforementioned factor of $(2^{n}n!)^{-1/2}$ in the position-basis wavefunction
of a Fock state $\ket{n}$. For instance, to represent a cat state with the
amplitude $\alpha=10$, the reconstruction Fock basis must include elements up
to at least $n=170$, for which this numerical factor is as low as $10^{-179}$,
requiring very high bit depth for precise calculation. This issue does not
arise in the continuous-variable basis, making our approach advantageous for
high-amplitude states.

To illustrate the challenge of cat state reconstruction, we refer to Fig.
3(a), which shows the simulated phase-dependent quadrature data. Visible
fringes at the intersection of the two sinusoidal envelopes is the feature
that distinguishes a cat state from an incoherent mixture of two coherent
states. With growing amplitudes, the phase regions in which these
intersections are present reduce in width. At the same time, the frequency of
the fringes increases. This implies the requirement for larger quadrature
sample sizes to make this feature statistically significant. At the same time,
increasingly finer grids in the momentum space are needed, as well as high
homodyne photodetection efficiencies. These factors make the required data
acquisition and reconstruction process increasingly complicated and time-
intensive.

For this reason, the magnitude of the off-diagonal peaks in the density matrix
[blue peaks in Fig. 3(b)], as well as the amplitude of the interference fringe
pattern in the Wigner function, degrade with the growing amplitude is. It is
remarkable, however, that although the fringe pattern is not visible in the
dataset of Fig. 3(a,bottom), the NN is able to make it out, reconstructing the
cat state with a fidelity of $0.87$ with pronounced coherence between the
peaks [bottom panels in Fig. 3(b,c)] in spite of a relatively small number of
quadrature samples ($2\cdot 10^{4}$).

Finally, we perform tomography of more complex states relevant to continuous-
variable quantum information processing (Fig. 4 ). We consider the following
states.

  * •

Two-headed Schrödinger’s cats with imaginary amplitudes [(Fig. 4(a)]. The
state is analogous to that studied above, but the position and momentum are
exchanged — so the NN needs to predict a density matrix with fast
oscillations. The NN has four hidden layers with 100 units each. The
activation functions are $\sin(\cdot)$ in the first two layers and
$\tanh(\cdot.)$ for the second two layers. The sinusoidal activation function
is helpful in modeling the oscillating behavior of the wavefunction. High-
quality reconstruction demonstrates that our algorithm’s capabilities are
insensitive to the choice of the zero phase reference point.

  * •

Three-headed cat states
$\ket{\alpha}+\ket{\alpha\cdot\exp{(i2\pi/3)}}+\ket{\alpha\cdot\exp{(-i2\pi/3)}}$
containing three peaks with evenly distributed phases [(Fig. 4(b)]. We used
the same NN architecture as for the above two-headed state.

  * •

Gottesman-Kitaev-Preskill state [35], which is of importance for quantum
computing [36, 37] [(Fig. 4(c)]. Ideally, the wavefunction of this state is an
infinite array of delta-functions, but the approximate wavefunction relevant
to experiments is an array of Gaussian peaks under a broader Gaussian
envelope. The specific state used in our experiment has the wavefunction

$\psi_{GKP}(x)\sim\sum_{s=-\infty}^{\infty}e^{-2\pi(sk)^{2}}e^{-\frac{(x-2s\sqrt{\pi})^{2}}{\Delta^{2}}}$

where $\Delta=k=1/4$. The NN has four hidden layers with 100 units each. The
activation functions are $\sin(\cdot)$ in the third layer and $\tanh(\cdot)$
for the other layers.

The reconstruction grid parameters and fidelities are given in Table II. As
evidenced Although some artefacts are present, our NN accurately reconstructs
all of these states.

Table 2: Reconstruction grids and fidelities of complex state QST (Fig. 4). The fidelities obtained with our method ($F_{\rm NN}$) and MaxLik ($F_{\rm MaxLik}$) are shown. State | position grid | momentum grid | $F_{\rm NN}$ | $F_{\rm MaxLik}$
---|---|---|---|---
Two-headed cat, $\alpha=13i$ |  $[-5,5]$, 500 pts |  $[-23,23]$, 500 pts | 0.98 | N/A
Three-headed cat, $\alpha=13$ |  $[-13,-5]\cup[14,23]$, 300 pts |  $[-23,23]$, 300 pts | 0.91 | N/A
Gottesman-Kitaev-Preskill |  $[-14,14]$, 600 pts |  $[-14,14]$, 600 pts | 0.92 | 0.87

##  IV Conclusion

We demonstrate a new approach to quantum tomography based on the
representation of a density matrix in a continuous-variable basis by a feed-
forward neural network. The symbiosis of a continuous-variable basis and NN as
universal approximator allows us to overcome the limitations on reconstructing
quantum states with high amplitudes and/or photon numbers. We believe our
method to be especially valuable in the context of quantum state engineering
in superconducting circuits, which are known to be capable of producing high-
amplitude continuous-variable states with high efficiencies [38].

A particularly useful feature of our NN-based method is its ability to
reconstruct states in the predefined regions of the coordinate space. However,
this feature cannot be taken advantage of if the state is not well-localized
in the phase space. In this case, the reconstruction would require a
coordinate grid that is broad and dense at the same time, and, consequently,
more computing power.

Further development might be towards utilisation of more sophisticated
architectures of deep learning models to represent multi-mode states and
states with complex spatial structure.

##  V Appendix

###  V.1 Quadrature probability distribution

Here we present the alternative analytical derivation of the quadrature
probability density function (2a), which might be found simpler than the
original derivation by Man’ko et al. [39, 40, 29].

First, consider the overlap between a quadrature state
$\ket{X_{\theta},\theta}$ and a position state $\ket{x}$. The state
$\ket{X_{\theta},\theta}$ is the eigenstate of the position operator in the
rotated frame characterized by angle $\theta$ with eigenvalue $X_{\theta}$:

$(\hat{X}\cos\theta+\hat{P}\sin\theta)\ket{X_{\theta},\theta}=X_{\theta}\ket{X_{\theta},\theta}.$

In the position basis we can express the action of the momentum operator as
$-i\dfrac{d}{dx}$, so the above equation becomes

$\dfrac{d}{dx}\braket{x}{X_{\theta},\theta}=\dfrac{i}{\sin\theta}\left(X_{\theta}-x\cos\theta\right)\braket{x}{X_{\theta},\theta}.$

The solution of this differential equation is

$\braket{x}{X_{\theta},\theta}=C\exp\left(i\frac{x}{\sin\theta}X_{\theta}-i\tan\theta\frac{x^{2}}{2}\right),$
(4)

with $C=1/\sqrt{2\pi|\sin\theta|}$ being a normalization factor obtained from

$\braket{X^{\prime}_{\theta},\theta}{X_{\theta},\theta}=\delta(X_{\theta}-X^{\prime}_{\theta}).$
(5)

Finally, the probability density $P(X_{\theta},\theta)$ of observing the
particular quadrature $X_{\theta}$ for a density matrix
$\hat{\rho}(x,x^{\prime})$ is

$P(X_{\theta},\theta)=\iint\braket{X_{\theta},\theta}{x}\rho(x,x^{\prime})\braket{x^{\prime}}{X_{\theta},\theta}dxdx^{\prime}.$

Substituting $\braket{x}{X_{\theta},\theta}$ from Eqs. (4) and (5), we obtain
the final result (2a):

$\displaystyle P(X_{\theta},\theta)$
$\displaystyle=\frac{1}{2\pi|\sin\theta|}\iint\rho(x,x^{\prime})\cdot$
$\displaystyle\cdot\exp\left[-i\dfrac{x-x^{\prime}}{\sin\theta}\left(X_{\theta}-\cos\theta\dfrac{x+x^{\prime}}{2}\right)\right]dxdx^{\prime}.$

###  V.2 Correction for losses

Here we derive the formalism tthat would enable our QST algorithm to correct
for the effect of losses and inefficient photodetection that may affect
quadrature measurements. In presence of these effects, the probability
distribution for the measured quadrature $X_{\theta}$ is [24]

$\begin{gathered}P(X_{\theta},\theta,\eta)=\dfrac{1}{\sqrt{\pi(1-\eta)}}\cdot\\\
\cdot\int
P(Q_{\theta},\theta)\exp\left[-\dfrac{\eta}{1-\eta}\left(Q_{\theta}-\dfrac{X_{\theta}}{\sqrt{\eta}}\right)^{2}\right]dQ_{\theta},\end{gathered}$
(6)

where $P(Q_{\theta},\theta)$ is the quadrature probability density in the
absence of losses given by Eq. (2a). After some simplifications, we obtain

$\begin{gathered}P(X_{\theta},\theta,\eta)=\dfrac{1}{2\pi\sqrt{\eta}|\sin\theta|}\iint\rho(x,x^{\prime})\cdot\\\
\cdot\exp\left\\{-\dfrac{x-x^{\prime}}{\sin\theta}\left[\dfrac{(x-x^{\prime})(1-\eta)}{4\eta\sin\theta}\right.\right.+\\\
+\left.\left.i\left(\dfrac{X_{\theta}}{\sqrt{\eta}}-\cos\theta\dfrac{x+x^{\prime}}{2}\right)\right]\right\\}dxdx^{\prime}.\end{gathered}$
(7)

A similar expression can be derived for the momentum basis. These expressions
are then used in place of Eqs. (2a) and (2b) to compute the likelihood (1).
Remarkably, Eq. (7) contains integration over as many variables as the
lossless integral (2a). This is in contrast to the iterative algorithm [23],
which requires additional summation over all basis elements.

## Author Contributions

ET came up with the project; EF and NK built NNs and performed simulations;
EF, AU and NK collected the data; all authors prepared the manuscript; ET and
AL supervised the project.

## Acknowledgments

The authors thank V.I. Man’ko for a fruitful discussion.

## Data availability

The code and experimental data used for tomography within this work will be
available upon reasonable request after publication of this preprint.

## Disclosures

All authors declare no conflict of interest.

## References

  * Eaton  _et al._ [2022] M. Eaton, C. González-Arciniegas, R. N. Alexander, N. C. Menicucci, and O. Pfister, Measurement-based generation and preservation of cat and grid states within a continuous-variable cluster state, Quantum 6, 769 (2022).
  * Nehra  _et al._ [2021] R. Nehra, M. Eaton, O. Pfister, and A. Marandi, All-optical quantum state engineering for rotation-symmetric bosonic states, arXiv preprint arXiv:2105.11035 (2021).
  * Ourjoumtsev  _et al._ [2006] A. Ourjoumtsev, R. Tualle-Brouri, J. Laurat, and P. Grangier, Generating optical schrodinger kittens for quantum information processing, Science 312, 83 (2006).
  * Eaton  _et al._ [2019] M. Eaton, R. Nehra, and O. Pfister, Non-gaussian and gottesman–kitaev–preskill state preparation by photon catalysis, New Journal of Physics 21, 113034 (2019).
  * Dakna  _et al._ [1998] M. Dakna, L. Knöll, and D.-G. Welsch, Quantum state engineering using conditional measurement on a beam splitter, The European Physical Journal D-Atomic, Molecular, Optical and Plasma Physics 3, 295 (1998).
  * Thekkadath  _et al._ [2020] G. Thekkadath, B. Bell, I. A. Walmsley, and A. Lvovsky, Engineering schrödinger cat states with a photonic even-parity detector, Quantum 4, 239 (2020).
  * Takase  _et al._ [2021] K. Takase, J.-i. Yoshikawa, W. Asavanant, M. Endo, and A. Furusawa, Generation of optical schrödinger cat states by generalized photon subtraction, Physical Review A 103, 013710 (2021).
  * Asavanant  _et al._ [2021] W. Asavanant, K. Takase, K. Fukui, M. Endo, J.-i. Yoshikawa, and A. Furusawa, Wave-function engineering via conditional quantum teleportation with a non-gaussian entanglement resource, Physical Review A 103, 043701 (2021).
  * Asavanant  _et al._ [2017] W. Asavanant, K. Nakashima, Y. Shiozawa, J.-I. Yoshikawa, and A. Furusawa, Generation of highly pure schrödinger’s cat states and real-time quadrature measurements via optical filtering, Optics Express 25, 32227 (2017).
  * Asavanant  _et al._ [2019] W. Asavanant, Y. Shiozawa, S. Yokoyama, B. Charoensombutamon, H. Emura, R. N. Alexander, S. Takeda, J.-i. Yoshikawa, N. C. Menicucci, H. Yonezawa, _et al._ , Generation of time-domain-multiplexed two-dimensional cluster state, Science 366, 373 (2019).
  * Larsen  _et al._ [2019] M. V. Larsen, X. Guo, C. R. Breum, J. S. Neergaard-Nielsen, and U. L. Andersen, Deterministic generation of a two-dimensional cluster state, Science 366, 369 (2019).
  * Paris and Sacchi [2003] M. G. Paris and M. F. Sacchi, Quantum tomography, Advances in Imaging and Electron Physics 128, 206 (2003).
  * D’Ariano and Presti [2001] G. D’Ariano and P. L. Presti, Quantum tomography for measuring experimentally the matrix elements of an arbitrary quantum operation, Physical review letters 86, 4195 (2001).
  * Banaszek  _et al._ [2013] K. Banaszek, M. Cramer, and D. Gross, Focus on quantum tomography, New Journal of Physics 15, 125020 (2013).
  * Lvovsky and Raymer [2009] A. I. Lvovsky and M. G. Raymer, Continuous-variable optical quantum-state tomography, Reviews of modern physics 81, 299 (2009).
  * Smithey  _et al._ [1993] D. Smithey, M. Beck, M. G. Raymer, and A. Faridani, Measurement of the wigner distribution and the density matrix of a light mode using optical homodyne tomography: Application to squeezed states and the vacuum, Physical review letters 70, 1244 (1993).
  * Ahmed  _et al._ [2021] S. Ahmed, C. S. Muñoz, F. Nori, and A. F. Kockum, Quantum state tomography with conditional generative adversarial networks, Physical Review Letters 127, 140502 (2021).
  * Tiunov  _et al._ [2020] E. S. Tiunov, V. Tiunova, A. E. Ulanov, A. Lvovsky, and A. K. Fedorov, Experimental quantum homodyne tomography via machine learning, Optica 7, 448 (2020).
  * Torlai  _et al._ [2018] G. Torlai, G. Mazzola, J. Carrasquilla, M. Troyer, R. Melko, and G. Carleo, Neural-network quantum state tomography, Nature Physics 14, 447 (2018).
  * Carrasquilla  _et al._ [2019] J. Carrasquilla, G. Torlai, R. G. Melko, and L. Aolita, Reconstructing quantum states with generative models, Nature Machine Intelligence 1, 155 (2019).
  * Kurmapu  _et al._ [2022] M. K. Kurmapu, V. Tiunova, E. Tiunov, M. Ringbauer, C. Maier, R. Blatt, T. Monz, A. K. Fedorov, and A. Lvovsky, Reconstructing complex states of a 20-qubit quantum simulator, arXiv preprint arXiv:2208.04862 (2022).
  * Zhu  _et al._ [2022] Y. Zhu, Y.-D. Wu, G. Bai, D.-S. Wang, Y. Wang, and G. Chiribella, Flexible learning of quantum states with generative query neural networks, Nature Communications 13, 1 (2022).
  * Lvovsky [2004] A. I. Lvovsky, Iterative maximum-likelihood reconstruction in quantum homodyne tomography, Journal of Optics B: Quantum and Semiclassical Optics 6, S556 (2004).
  * Leonhardt [1997] U. Leonhardt,  _Measuring the quantum state of light_, Vol. 22 (Cambridge university press, 1997).
  * Cybenko [1989] G. Cybenko, Approximation by superpositions of a sigmoidal function, Mathematics of control, signals and systems 2, 303 (1989).
  * Hornik [1991] K. Hornik, Approximation capabilities of multilayer feedforward networks, Neural networks 4, 251 (1991).
  * Ofek  _et al._ [2016] N. Ofek, A. Petrenko, R. Heeres, P. Reinhold, Z. Leghtas, B. Vlastakis, Y. Liu, L. Frunzio, S. Girvin, L. Jiang, _et al._ , Extending the lifetime of a quantum bit with error correction in superconducting circuits, Nature 536, 441 (2016).
  * Lvovsky  _et al._ [2020] A. Lvovsky, P. Grangier, A. Ourjoumtsev, V. Parigi, M. Sasaki, and R. Tualle-Brouri, Production and applications of non-gaussian quantum states of light, arXiv preprint arXiv:2006.16985 (2020).
  * Man’ko  _et al._ [1999] V. Man’ko, M. Moshinsky, and A. Sharma, Diffraction in time in terms of wigner distributions and tomographic probabilities, Physical Review A 59, 1809 (1999).
  * Svozil  _et al._ [1997] D. Svozil, V. Kvasnicka, and J. Pospichal, Introduction to multi-layer feed-forward neural networks, Chemometrics and intelligent laboratory systems 39, 43 (1997).
  * Higham [1990] N. J. Higham, Analysis of the cholesky decomposition of a semi-definite matrix, University of Manchester eprint (1990).
  * Blum [2012] K. Blum,  _Density matrix theory and applications_, Vol. 64 (Springer Science & Business Media, 2012).
  * Bimbard  _et al._ [2010] E. Bimbard, N. Jain, A. MacRae, and A. Lvovsky, Quantum-optical state engineering up to the two-photon level, Nature Photonics 4, 243 (2010).
  * Sychev  _et al._ [2017] D. V. Sychev, A. E. Ulanov, A. A. Pushkina, M. W. Richards, I. A. Fedorov, and A. I. Lvovsky, Enlargement of optical schrödinger’s cat states, Nature Photonics 11, 379 (2017).
  * Gottesman  _et al._ [2001] D. Gottesman, A. Kitaev, and J. Preskill, Encoding a qubit in an oscillator, Physical Review A 64, 012310 (2001).
  * Glancy and Knill [2006] S. Glancy and E. Knill, Error analysis for encoding a qubit in an oscillator, Physical Review A 73, 012325 (2006).
  * Glancy  _et al._ [2009] S. C. Glancy, H. M. Vasconcelos, L. Sanz, _et al._ , Generation of gkp states with optical states, nist.gov (2009).
  * Vlastakis  _et al._ [2013] B. Vlastakis, G. Kirchmair, Z. Leghtas, S. E. Nigg, L. Frunzio, S. M. Girvin, M. Mirrahimi, M. H. Devoret, and R. J. Schoelkopf, Deterministically encoding quantum information using 100-photon schrödinger cat states, Science 342, 607 (2013).
  * Mancini  _et al._ [1996] S. Mancini, V. Man’Ko, and P. Tombesi, Symplectic tomography as classical approach to quantum systems, Physics Letters A 213, 1 (1996).
  * Man’ko  _et al._ [1998] V. Man’ko, L. Rosa, and P. Vitale, Time-dependent invariants and green functions in the probability representation of quantum mechanics, Physical Review A 57, 3291 (1998).