#  Non-thermal Higgs Spectrum in Reheating Epoch:
Primordial Condensate vs. Stochastic Fluctuation

Kunio Kaneta and Kin-ya Oda

∗†Department of Mathematics, Tokyo Woman’s Christian University
Tokyo 167-8585, Japan  E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Since electroweak symmetry is generally broken during inflation, the Standard
Model Higgs field can become supermassive even after the end of inflation. In
this paper, we study the non-thermal phase space distribution of the Higgs
field during reheating, focusing in particular on two different contributions:
primordial condensate and stochastic fluctuations. We obtain their analytic
formulae, which agree with the previous numerical result. As a possible
consequence of the non-thermal Higgs spectrum, we discuss perturbative Higgs
decay during reheating for the case it is kinematically allowed. We find that
the soft-relativistic and hard spectra are dominant in the decay rate of the
stochastic fluctuation and that the primordial condensate and stochastic
fluctuations decay almost at the same time.

##  1 Introduction

Despite the discovery of the Higgs boson at the Large Hadron Collider at CERN
[1, 2], the least understood in the Standard Model of particle physics is the
Higgs sector. This is simply because, in the absence of sufficient
experimental data, many possibilities can fit into the theory of electroweak
symmetry breaking, which necessarily include cosmological implications; see
e.g. Ref. [3] for a recent review. For instance, electroweak phase transition
in the early universe has been extensively studied and many connections to
other subjects including baryogenesis [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
15, 16, 17, 18, 19, 20] and gravitational waves [21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34] have been found. The Higgs sector may also be
responsible for producing dark matter, known as the Higgs portal scenario [35,
36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48]; see also Ref. [49] for a
pedagogical introduction on particle dark matter.

The recent development of observation technologies allows us to address the
cosmological history at much earlier times than when the electroweak phase
transition takes place. The inflationary scenario is one of the main targets
of the recent cosmological/astrophysical observations; see e.g. Refs. [50, 51,
52, 53, 54, 55, 56] for reviews. Indeed, cosmic microwave background (CMB)
observation has set constraints on the scalar and tensor power spectra, by
which many inflationary models have already been ruled out [57]. In such early
times during the inflation epoch, the Higgs field may play an important role.
One such example is the Higgs inflation where the Standard Model Higgs is
identified as the inflaton [58, 59]; see also Refs. [60, 61, 62, 63, 64, 65,
66, 67, 68, 69] for related topics and a review.

In the case that the Higgs itself is not the inflaton and the inflation is
driven by another field, the Higgs is regarded as a light spectator field
during inflation/reheating. In this case, the Higgs vacuum stability during
inflation and reheating is an important subject. The mass measurement of the
Higgs and top quark has brought up an indication that the vacuum becomes
unstable at high energies, whose cosmological implications have also been
extensively studied [70, 71, 72, 73, 74, 75, 76, 77]. The Higgs field may also
affect in many ways the temperature evolution after the end of inflation. For
instance, the highest temperature of the universe is affected by the
gravitational production of the Higgs during reheating [78, 79] or preheating
through thermalization of weak gauge bosons produced by the oscillation of
primordial Higgs condensate [80, 81, 82, 83].

In discussing the cosmological role of the Higgs, especially during and right
after inflation, there are two distinctive contributions: the primordial
condensate and the stochastic fluctuations. Sometimes their difference is not
well recognized or even mixed up. This is partly because when one does not
need to care about the momentum distribution of the primordial Higgs field,
such refined separation is not necessary. However, when one needs to care,
e.g. in considering the decay of the Higgs, then a closer look at the
difference is needed.

In the present work, we focus on the non-thermal phase space distribution
arising from two different origins, namely, the primordial Higgs condensate
and the stochastic fluctuation. These are both seen as a displacement of the
Higgs field value during inflation, hence are hardly distinguishable from each
other in the field space. On the other hand, they have different phase space
distributions after the end of inflation. We will explicitly show such a non-
thermal spectrum of the Standard Model Higgs and, as a possible implication,
will discuss their subsequent decay.

The paper is organized as follows. In Sec. 2, we first present an explicit
model of the Higgs and inflaton sectors to set up the stage for our
discussion. We explain the dynamics of the Higgs during inflation in Sec. 3,
where we first clarify what the primordial condensate means and then briefly
review the stochastic fluctuations. The non-thermal Higgs spectrum after the
end of inflation is discussed in Sec. 4, which is then applied to the non-
thermal Higgs decay in Sec. 5. The summary of our findings is given in Sec. 6.

##  2 Higgs and inflation sectors

Before discussing the detail of the Higgs dynamics during inflation, we make
our framework explicit in this section, including the Higgs and inflaton
sectors. While we will consider the Higgs sector in its conventional form (the
quadratic and quartic terms in the potential), the choice of the inflationary
sector is arbitrary to some extent. Since what we will argue in the subsequent
sections does not quite depend on the inflationary models, our choice here
should be considered as an example for the sake of concreteness.

###  2.1 Set-up

The relevant part of the action is given by

$\displaystyle S$ $\displaystyle=\int d^{4}x\sqrt{-g}\left({\cal L}_{\rm
grav}+{\cal L}_{\rm inf}+{\cal L}_{\rm Higgs}\right),$ (1)

where ${\cal L}_{\rm grav}$, ${\cal L}_{\rm inf}$, and ${\cal L}_{\rm Higgs}$
are the Lagrangian densities for the Einstein gravity, inflaton, and Higgs
sectors, respectively. With the reduced Planck mass $M_{\text{P}}\simeq
2.4\times 10^{18}$ GeV, the gravity Lagrangian is given by

$\displaystyle{\cal L}_{\rm grav}$
$\displaystyle=-\frac{M_{\text{P}}^{2}}{2}R,$ (2)

where $R$ is the Ricci scalar. We do not specify the inflaton sector ${\cal
L}_{\rm inf}$ at the moment, but the slow-roll inflation is assumed. The Higgs
sector is given by

$\displaystyle{\cal L}_{\rm Higgs}$
$\displaystyle=g^{\mu\nu}\left(D_{\mu}\Phi\right)^{\dagger}\left(D_{\nu}\Phi\right)-V(\Phi),$
(3)

where $\Phi$ denotes the Higgs doublet and

$\displaystyle V(\Phi)$
$\displaystyle=-\mu^{2}|\Phi|^{2}+\lambda\left(\Phi^{\dagger}\Phi\right)^{2}-\xi\left|\Phi\right|^{2}R.$
(4)

We take the Friedmann-Lemaître-Robertson-Walker metric $g_{\mu\nu}={\rm
diag}\\!\left(1,-a^{2},-a^{2},-a^{2}\right)$ with the scale factor $a$. The
Higgs non-minimal coupling to $R$ is also included in the potential. Note that
such a non-minimal coupling affects the kinetic term of the Higgs when we turn
to the Einstein frame. However, as we will see, only the case
$\xi\left|\Phi\right|^{2}\ll M_{\text{P}}^{2}$ becomes relevant, and thus we
may effectively regard the given $\Phi$ as defined in the Einstein frame and
the non-minimal coupling may be perturbatively treated as an effective mass to
the Higgs field (especially during inflation).

To illustrate how $\Phi$ evolves during inflation, we simplify the setup by
replacing $\Phi\to\chi/\sqrt{2}$ with $\chi$ a real scalar. The extension to
the doublet is straightforward and will be seen later. The equation of motion
for $\chi$ during inflation is given by

$\displaystyle\ddot{\chi}+3H\dot{\chi}-a^{-2}\vec{\nabla}^{2}\chi+\partial_{\chi}V(\chi)=0,$
(5)

where the dot and $\partial_{\chi}$ denote the derivatives with respect to the
comoving time $t$ and the field $\chi$, respectively, and
$\vec{\nabla}^{2}=\sum_{i=1}^{3}\partial_{i}^{2}$ is the Laplacian with
respect to the comoving coordinates. The scalar potential in the present case
is given by

$\displaystyle V(\chi)$
$\displaystyle=-\frac{\mu^{2}}{2}\chi^{2}+\frac{\lambda}{4}\chi^{4}-\frac{\xi}{2}\chi^{2}R.$
(6)

Unlike the free scalar case, the quartic coupling induces an effective mass,
which we will take a closer look at based on the stochastic approach.

###  2.2 Inflationary model

Before getting into the detail about the Higgs dynamics during inflation, we
clarify the setup of the inflaton sector. Although our analysis does not quite
depend on the choice of the inflation model, for the sake of definiteness, we
consider the T-model [84] whose inflaton potential is given by

$\displaystyle
V(\phi)=6\lambda_{\phi}M_{\text{P}}^{4}\tanh^{2}\left(\frac{\phi}{\sqrt{6}M_{\text{P}}}\right),$
(7)

where the potential coupling $\lambda_{\phi}$ is determined by observation.
The amplitude of the curvature power spectrum, $A_{\text{S}}$, determines the
inflationary scale as

$\displaystyle\lambda_{\phi}$
$\displaystyle\simeq\frac{18\pi^{2}A_{\text{S}}}{6N^{2}_{\rm CMB}},$ (8)

where $N=\ln(a_{\text{e}}/a)$ is the number of e-folds measured from the end
of inflation at $a=a_{\text{e}}$, and $N_{\rm CMB}$ is the time $a=a_{\rm
CMB}$ when the CMB scale exits the horizon. In the following, we use
$\ln(10^{9}A_{\text{S}})=3.044$ measured at the pivot scale $k_{\rm CMB}=0.05$
Mpc-1 and assume $N_{\rm CMB}=55$, so $\lambda=2.1\times 10^{-11}$. Note that
the inflaton starts oscillating about $\phi=0$ after the end of inflation,
during which the inflaton potential is approximately quadratic:

$\displaystyle V(\phi\ll\phi)$
$\displaystyle\simeq\frac{1}{2}m_{\phi}^{2}\phi^{2},$ (9)

where $m_{\phi}=\sqrt{2\lambda_{\phi}}M_{\text{P}}\simeq 1.5\times 10^{13}$
GeV.

##  3 Higgs dynamics during inflation

Our focus is on the evolution of the Standard Model Higgs field during
inflation, where the Higgs is considered as a spectator field whose energy
density is assumed to be subdominant compared with that of the inflaton.

###  3.1 Primordial condensate

We first consider the dynamics of the what-we-call _primordial condensate_ ,
a.k.a. misalignment contribution, where the initial field value of $\chi$ is
set at some time during or before inflation. In particular, we define the
primordial condensate in a narrow sense by considering that the contribution
arises only from the zero momentum state.

Note that we may also consider non-zero amplitudes (and initial phases
depending on the wave number) for higher momentum states, i.e., a finite
initial number of the corresponding particle. On the other hand, as we will
see shortly, the higher momentum states may acquire a non-zero variance
through stochastic fluctuations even when the initial field value (and/or
phase) is zero, i.e., when there are no initial particles, or equivalently
when we start from the Bunch-Davies vacuum [85, 86]. So, the initial
displacement would add up to the stochastic fluctuation in the end, if it may
be regarded as a small correction to the Bunch-Davies vacuum. Notice however
that if the initial amplitude is much greater than the stochastic fluctuation,
the Fokker-Planck equation for a Gaussian noise may not be a good
approximation any further, since the noise term may be affected by the
primordial contribution, and hence deviates from Gaussian distribution,111 In
such a case, the initial condition may not be close to the Bunch-Davies
vacuum. See e.g. Ref. [87] for deviations from the Bunch-Davies vacuum.  which
is beyond the scope of our present work. Therefore in the following argument,
we constrain ourselves to the case where only the zero momentum state may form
the primordial condensate.222 This assumption is justified when e-folding from
an “initial” time $a_{\rm ini}$ to $a_{\rm CMB}$ is sufficiently large,
$\ln(a_{\rm CMB}/a_{\rm ini})\gtrsim\ln(M_{\text{P}}/H_{\rm inf})$ barring the
existence of primordial condensates that have wave number larger than
$M_{\text{P}}$. Without such trans-Plackian modes, all the primordial
condensates are red-shifted to sufficiently low momenta that can be
collectively approximated by the zero momentum mode as in our treatment.

For the zero-momentum mode $\chi$, the equation of motion is given by

$\displaystyle\ddot{\chi}+3H\dot{\chi}+\partial_{\chi}V(\chi)=0.$ (10)

Here, we approximate the Hubble parameter during inflation as a constant such
that $H_{\rm inf}$ is the same as the one at the end of inflation
$a=a_{\text{e}}$ i.e. $H_{\rm inf}=H(a=a_{\text{e}})\equiv H_{\text{e}}$.
Supposing that $\chi$ has a vanishing velocity $\dot{\chi}=0$ at an initial
time, there are two cases for the solution of Eq. (10). If the potential term
is negligibly small compared with the Hubble friction term, then
$\chi(t)=\chi_{0}$, where $\chi_{0}$ is a constant field value set at an
initial time, so the Higgs field value remains constant during inflation. On
the other hand, if the potential term dominates over the Hubble friction term,
then

$\displaystyle\chi(t)\simeq\chi_{0}\ {\rm
sn}\\!\left(K\\!\left(-1\right)-\sqrt{\lambda/2}\,\chi_{0}t,\ -1\right),$ (11)

where ${\rm sn}(u,m)$ is the elliptic sine and $K\\!\left(n\right)$ is the
complete elliptic integral of the first kind: $K\\!\left(-1\right)\simeq
1.31$. We see that $\chi(t)$ oscillates during inflation while the Hubble
friction still damp the amplitude. Therefore, at some time when the Hubble
drag becomes efficient, $\chi(t)$ settles down to a certain constant value.

Figure 1: Evolution of the primordial condensate for three different initial
amplitudes as a function of $N=\ln(a_{\text{e}}/a)$. The squared amplitude is
normalized by $H_{\text{e}}=m_{\phi}$. The time flows from right to left. The
case $\chi_{0}=10H_{\text{e}}$ shows the initial settle down to the constant
value, whereas the other two cases correspond to the dominance of the Hubble
friction term from the beginning; see main text.

Figure 1 shows the evolution of the primordial condensate for the different
choices of the initial amplitude. It shows that a non-zero initial amplitude
ends up with a constant final value during inflation if the inflation lasts
long enough, as expected. After the end of inflation, $\chi(t)$ starts
oscillating when the Hubble friction drops below the potential term.

###  3.2 Stochastic fluctuations

In addition to the primordial condensate, the stochastic fluctuations, which
always exist even if the initial amplitude of the Higgs is zero, also affect
the Higgs field value during inflation. We briefly review how the Higgs field
acquires an effective vacuum expectation value and hence mass during
inflation, based on the stochastic approach [88, 89].333 See e.g. Refs. [90,
91, 92, 93, 94, 95] for related topics.

The stochastic approach begins by dividing the field into two pieces, one of
which resides in the infrared (IR) domain, and the other in the ultraviolet
(UV) domain. This can be done in practice by introducing a cutoff scale
$k_{\rm cut}$ in the comoving momentum space.444 Note that physical momentum
$p$ is related to comoving momentum $k$ through $p=k/a$.  It is convenient to
discuss with the mode function $\chi_{\vec{k}}$ (or $u_{k}$) defined by

$\displaystyle\chi(t,\vec{x})$
$\displaystyle=\int\frac{d^{3}k}{(2\pi)^{3/2}}e^{i\vec{k}\cdot\vec{x}}\chi_{\vec{k}}(t),$
(12)

where

$\displaystyle\chi_{\vec{k}}(t)$
$\displaystyle=a_{\vec{k}}u_{k}(t)+a^{\dagger}_{-\vec{k}}u^{*}_{k}(t),$ (13)

in which $a_{\vec{k}}$ and $a^{\dagger}_{-\vec{k}}$ are the annihilation and
creation operators. In the same manner, we define the conjugate momentum by
$\pi\equiv\dot{\chi}$ whose mode function is
$\pi_{\vec{k}}(t)=a_{\vec{k}}v_{k}(t)+a^{\dagger}_{\vec{k}}v^{*}_{k}(t)$. The
splitting can be realized by introducing a window function $W_{k}(t)$ such
that

$\displaystyle u_{k}(t)=u_{k<k_{\rm cut}}(t)+u_{k>k_{\rm cut}}(t),$ (14)

where

$\displaystyle u_{k>k_{\rm cut}}$
$\displaystyle\equiv\left[1-W_{k}(t)\right]u_{k},$ (15) $\displaystyle
u_{k<k_{\rm cut}}$ $\displaystyle\equiv W_{k}(t)u_{k}.$ (16)

A conventional choice for $W_{k}(t)$ is given by the step function as

$\displaystyle W_{k}(t)$ $\displaystyle=\theta\\!\left(\epsilon
a(t)H-k\right),$ (17)

where $\epsilon\ll 1$ is an arbitrary constant, corresponding to $k_{\rm
cut}=\epsilon aH$.

Denoting the IR contributions of $\chi$ and $\pi$ as

$\displaystyle\overline{\chi}$
$\displaystyle\equiv\int\frac{d^{3}k}{(2\pi)^{3/2}}e^{i\vec{k}\cdot\vec{x}}W_{k}\chi_{\vec{k}},$
(18) $\displaystyle\overline{\pi}$
$\displaystyle\equiv\int\frac{d^{3}k}{(2\pi)^{3/2}}e^{i\vec{k}\cdot\vec{x}}W_{k}\pi_{\vec{k}},$
(19)

we see that

$\displaystyle\dot{\overline{\chi}}-\overline{\pi}=f^{\chi},$ (20)

which is non-zero, where

$\displaystyle f^{\chi}(t,\vec{x})$
$\displaystyle\equiv\int\frac{d^{3}k}{(2\pi)^{3/2}}e^{i\vec{k}\cdot\vec{x}}\dot{W}_{k}\chi_{k}.$
(21)

When $H^{2}\gg\partial_{\chi}^{2}V$, the IR modes ($k<k_{\rm cut}$) are in the
slow-roll regime,555 See e.g. Ref. [96].  and thus, we obtain the relation
among the IR-only quantities:

$\displaystyle\overline{\pi}$
$\displaystyle\simeq-\frac{\partial_{\overline{\chi}}V(\overline{\chi})}{3H}.$
(22)

Plugging Eq. (22) into Eq. (20), we end up with

$\displaystyle\dot{\overline{\chi}}$
$\displaystyle\simeq-\frac{\partial_{\overline{\chi}}V(\overline{\chi})}{3H}+f^{\chi}.$
(23)

The appearance of non-zero $f^{\chi}$ indicates that UV modes keep flowing
into the IR domain during inflation, caused by the constant (Hubble) horizon
size of the (quasi) de Sitter spacetime.

It is important that Eq. (23) may be regarded as the Langevin equation for
$\overline{\chi}$ with a random noise $f^{\chi}$. Furthermore, the
distribution of the noise is Gaussian as the variance is given by
$\left\langle
f^{\chi}(t_{1},\vec{x})f^{\chi}(t_{2},\vec{x})\right\rangle=\left(H_{\text{e}}^{3}/4\pi^{2}\right)\delta(t_{1}-t_{2})$
when $\chi_{k}$ in Eq. (21) follows the equation of motion for a free massless
state, whose mode functions are given by
$u_{k}=-(H/\sqrt{2k})\left(\eta-i/k\right)e^{-ik\eta}$ for the conformal time
$\eta=-1/aH$.666 For reader’s ease, mass dimensions are $[a]=0$, $[\eta]=-1$,
$[u_{k}]=-1/2$, $[a_{\vec{k}}]=-3/2$, $[W_{k}]=0$, $[\dot{W}_{k}]=1$,
$[\chi_{k}]=-2$, $[\chi]=1$, $[\overline{\chi}]=1$, and $[f^{\chi}]=2$.
Therefore the distribution of $\overline{\chi}$ in the field space,
$P(\overline{\chi},t)$, follows the Fokker-Planck equation given by777 See
e.g. Chapter 4 in Ref. [97].

$\displaystyle\dot{P}(\overline{\chi},t)$
$\displaystyle=\frac{1}{2}\partial_{\overline{\chi}}\left(\frac{H_{\text{e}}^{3}}{8\pi^{2}}\partial_{\overline{\chi}}+\frac{\partial_{\overline{\chi}}V(\overline{\chi})}{3H_{\text{e}}}\right)P(\overline{\chi},t).$
(24)

Supposing that the duration of the inflation is long enough so that
$\overline{\chi}$ reaches the equilibrium state in the field space,888 This is
indeed the case if the number of e-folds for the duration of the inflation
satisfies $N\gtrsim 1/\sqrt{\lambda}$ [98].  we can readily find the solution
of Eq. (24) as

$\displaystyle P(\overline{\chi})$
$\displaystyle=N\exp\left[-\frac{8\pi^{2}V(\overline{\chi})}{3H_{\text{e}}^{4}}\right],$
(25)

where the normalization factor $N$ is determined by $\int
d\overline{\chi}P(\overline{\chi})=1$. For instance, when the potential energy
is dominated by the quartic coupling,
$V(\overline{\chi})\simeq(\lambda/4)\overline{\chi}^{4}$, we find

$\displaystyle\langle\overline{\chi}^{2}\rangle$
$\displaystyle=\frac{\sqrt{3}}{\Gamma^{2}(\frac{1}{4})}\frac{H_{\text{e}}^{2}}{\sqrt{\lambda}}\simeq
0.132\times\frac{H_{\text{e}}^{2}}{\sqrt{\lambda}}.$ (26)

Through the non-zero variance of the IR modes, $\chi$ acquires an effective
mass for the separation
$\chi^{2}=\langle\overline{\chi}^{2}\rangle+\delta\chi^{2}$ from the quartic
coupling, namely, $m_{\chi,{\rm
eff}}^{2}\simeq\lambda\langle\overline{\chi}^{2}\rangle$.

The Standard Model Higgs may obtain the non-zero variance in the same manner,
except that $\Phi$ has four real scalar degrees of freedom. Therefore, we find

$\displaystyle\langle\overline{\Phi}^{\dagger}\overline{\Phi}\rangle$
$\displaystyle=\sqrt{\frac{3}{8\pi}}\frac{H_{\text{e}}^{2}}{\sqrt{\lambda}}\left(1-(\pi-4)\sqrt{\frac{2\pi}{3}}\frac{\mu_{H}^{2}}{H_{\text{e}}^{2}}+\dots\right),$
(27)

where $\overline{\Phi}$ denotes the IR modes of $\Phi$, and
$\mu_{H}^{2}\equiv-\mu^{2}+6\xi H_{\text{e}}^{2}$ with
$R=-12H_{\text{e}}^{2}$, which we have assumed $|\mu_{H}^{2}|\ll
H_{\text{e}}^{2}$. When the Higgs potential is dominated by the quartic
coupling during inflation, the effective mass $m_{h,{\rm eff}}$ is given by

$\displaystyle m_{h,{\rm
eff}}^{2}=\sqrt{\frac{3\lambda}{8\pi}}H_{\text{e}}^{2}$ (28)

for every single real scalar in $\Phi$. In the same manner, the total energy
density becomes

$\displaystyle\rho_{h}$ $\displaystyle\simeq\left\langle
V(\overline{\Phi})\right\rangle=\frac{3}{8\pi^{2}}H_{\text{e}}^{4}.$ (29)

Finally, it should be stressed that the stochastic fluctuation is induced by
the gravitational effect and thus always exists even in the absence of the
primordial Higgs condensate.

##  4 Higgs spectrum after inflation

We turn our discussion to the phase space distribution of the Higgs after the
end of inflation. In the following argument, we suppose that the
thermalization of the Higgs takes a very long time such that it does not
complete and that the non-thermal spectrum persists. This setup is justified
e.g. when the reheating temperature is sufficiently low compared with the
effective Higgs mass, which can easily be realized unless $\lambda$ is
extremely small.

The phase space distribution for the primordial condensate is particularly
simple, which is by definition given by

$\displaystyle f_{\rm cond}(p,t)$ $\displaystyle=n_{\rm
cond}\left(2\pi\right)^{3}\delta^{3}(\vec{p}),$ (30)

where $p$ is the physical momentum, and the normalization $n_{\rm cond}$ is
determined by

$\displaystyle n_{\rm cond}$
$\displaystyle=\int\frac{d^{3}p}{(2\pi)^{3}}f_{\rm
cond}(p,t)=\frac{\rho_{h,{\rm cond}}(t)}{m_{h,{\rm eff}}}.$ (31)

By denoting the initial squared amplitude of the Higgs by $|\Phi_{0}|^{2}$,
the effective mass is given by $m_{h,{\rm eff}}^{2}=\lambda|\Phi_{0}|^{2}$,
and the initial energy density $\rho_{h,0}$ is determined accordingly. Note
that $\rho_{h,{\rm cond}}$ keeps to be $\rho_{h,{\rm cond}}\sim\rho_{h,0}$
until $a_{\rm osc}$ that satisfies $H(a=a_{\rm osc})\simeq m_{h,{\rm eff}}$.
Afterward, $\rho_{h,{\rm cond}}\simeq\rho_{h,0}(a/a_{\rm osc})^{-4}$ assuming
that the quartic term in the potential dominates during the Higgs oscillation.

It becomes more involved to take into account other Higgs spectra than the
primordial condensate. As we will discuss in the following, there are three
distinctive regimes in the phase space density for the stochastic fluctuation.

###  4.1 Hard spectrum from inflaton scattering

When $m_{h,{\rm eff}}<m_{\phi}$, the Higgs quanta may be produced through the
inflaton oscillation after the end of inflation. This process can produce only
the Higgs with $k>m_{\phi}$ for the kinematical reason [99]. In terms of
physical momentum, we have

$\displaystyle f_{\rm hard}(p,t)$
$\displaystyle\simeq\frac{9\pi}{64}\left(\frac{H(t)}{m_{\phi}}\right)^{3}\left(\frac{m_{\phi}}{p}\right)^{9/2}\theta\\!\left(m_{\phi}-p\right)\theta\\!\left(p-m_{\phi}a_{\text{e}}/a\right),$
(32)

where we have used $H(t)=H_{\text{e}}(a/a_{\text{e}})^{-3/2}$; see also Ref.
[100]. Note that the first step function ensures the energy conservation in
the process of $\phi\phi\to\chi\chi$, and the second bounds the spectrum from
below by imposing the maximal amount of the redshift of the physical momentum.

###  4.2 Soft spectra, relativistic and non-relativistic

The Higgs spectrum with $k<m_{\phi}$ is produced via the phase transition from
de Sitter (inflation) to matter domination (inflaton oscillation). The soft
spectrum depends on $m_{h,{\rm eff}}$ though the behavior of the effective
frequency which is, during reheating, given by

$\displaystyle\omega_{k}^{2}$
$\displaystyle=k^{2}+\frac{H_{\text{e}}^{4}m_{h,{\rm
eff}}^{2}}{16}(\eta-\overline{\eta}_{\text{e}})^{4}-\frac{2}{(\eta-\overline{\eta}_{\text{e}})^{2}},$
(33)

where $\overline{\eta}_{\text{e}}=\eta_{\text{e}}-2/a_{\text{e}}H_{\text{e}}$
[99]. Writing $(\eta-\overline{\eta}_{\text{e}})^{2}\equiv y$ and
$H_{\text{e}}^{4}m_{h,{\rm eff}}^{2}/16\equiv M^{6}/3$, we solve

$\displaystyle k^{2}+\frac{M^{6}}{3}y^{2}-\frac{2}{y}=0$ (34)

for $y$, yielding

$\displaystyle
y=\frac{-\left(k/M\right)^{2}+\left[3+\sqrt{9+\left(k/M\right)^{6}}\right]^{2/3}}{M^{2}\left[3+\sqrt{9+\left(k/M\right)^{6}}\right]^{1/3}}\equiv
y_{\rm entry}.$ (35)

Notice that a given mode $k$ enters the horizon at $y=y_{\rm entry}$. For
$k\gg M$, the horizon entry happens at $y_{\rm entry}\sim
k^{-2}\Rightarrow\eta_{\rm entry}\sim 1/k$, which is approximately the
massless case. On the other hand, by keep decreasing $k$ we observe that at
some point, $y_{\rm entry}$ becomes independent from $k$, indicating that the
spectrum becomes non-relativistic and proportional to $k^{-3}$. The transition
from relativistic to non-relativistic occurs at $k=k_{*}$ which satisfies

$\displaystyle y_{\rm entry}(k_{*}\gg
M)\simeq\frac{2}{k_{*}^{2}}-\frac{8}{3}\frac{M^{6}}{k_{*}^{8}}$
$\displaystyle=0$ $\displaystyle\Rightarrow$ $\displaystyle k_{*}$
$\displaystyle=\left(\frac{4}{3}\right)^{1/6}M=\left(\frac{H_{\text{e}}^{2}m_{h,{\rm
eff}}}{2}\right)^{1/3}.$ (36)

The soft spectrum has two regimes: relativistic ($k>k_{*}$) and non-
relativistic ($k<k_{*}$).

For the relativistic soft spectrum, we may neglect the Higgs mass effect, and
thus the phase space distribution is approximately given by the massless case:

$\displaystyle f_{\rm soft,R}(p,t)$
$\displaystyle\simeq\frac{9}{64}\left(\frac{H(t)}{p}\right)^{6}\left(\frac{a}{a_{\text{e}}}\right)^{3}\theta\\!\left(m_{\phi}a_{\text{e}}/a-p\right)\theta\\!\left(p-k_{*}a_{\text{e}}/a\right).$
(37)

For $k<k_{*}$, we obtain

$\displaystyle f_{\rm soft,NR}(p,t)$
$\displaystyle\simeq\frac{9}{32}\left(\frac{H}{m_{h,{\rm
eff}}}\right)\left(\frac{H}{p}\right)^{3}\left(\frac{a}{a_{\text{e}}}\right)^{-3}\theta\\!\left(k_{*}a_{\text{e}}/a-p\right).$
(38)

Therefore the total phase space distribution is given by the sum of Eqs. (32),
(37), and (38):

$\displaystyle f(p,t)$ $\displaystyle=f_{\rm hard}(p,t)+f_{\rm
soft,R}(p,t)+f_{\rm soft,NR}(p,t).$ (39)

Figure 2: Comparison of non-thermal Higgs spectrum between numerical result
[99] and the analytic estimate.

Figure 2 shows the total phase space distribution for $\chi$ as a function of
the comoving momentum $k=ap$. The red solid line in the figure is the
numerical result taken from Ref. [99], and the black dashed, dotted, dot-
dashed lines are analytical estimates given by Eq. (39). Note that to fit the
numerical result by the analytic estimate, we have taken
$H_{\text{e}}=m_{\phi}/1.25$ which reflects the uncertainty that the end of
inflation is not abrupt at $H=H_{\text{e}}$, rather it is smooth with time.
The choice of $m_{\chi}/m_{\phi}=0.001$ corresponds to $\lambda\simeq
10^{-11}$, though our results do not depend on the specific value of $\lambda$
as long as $m_{h,{\rm eff}}<m_{\phi}$ is satisfied.

##  5 Non-thermal Higgs decay

As an implication of the non-thermal Higgs spectrum, we consider the
perturbative decay of the Higgs into the Standard Model particles. This can be
possible whenever $m_{h,{\rm eff}}$ is larger than the mass of the decay
products. In such kinematic reasoning, one should keep in mind that the decay
products may also acquire their mass from the effective Higgs vacuum
expectation value. Therefore, for instance, the (perturbative) decay into the
weak gauge bosons does not occur when $\sqrt{\lambda}\lesssim g$, where $g$ is
the SU(2) gauge coupling. This is normally the case if we extrapolate the
renormalization group running of the couplings up to the inflationary scale
within the Standard Model [80, 81].999 See also, e.g., Refs. [82, 83] for
implications of non-perturbative Higgs decay. The Higgs energy density,
however, cannot dissipate at all through the parametric resonance, since the
instability terminates when the energy density of the produced particles
reaches the Higgs energy density. Thus the complete dissipation of the energy
density needs a perturbative decay [101].  However, such running is affected
by, for instance, the error in the top quark mass measurements and/or the
appearance of new physics at certain high scales [102]. Therefore, we take
rather an agnostic approach to the choice of $\lambda$, regarding it as a free
parameter at high scales.

In the same spirit, we take the Higgs decay width $\Gamma_{h}$ as a free
parameter and will keep track of how the Higgs decays. To do this, instead of
looking at the decay on a mode-by-mode basis, we consider the integrated
Boltzmann equation given by

$\displaystyle\dot{n}_{h}+3Hn_{h}=R_{h\to AB},$ (40)

where the decay rate $R_{h\to AB}$ is given by

$\displaystyle R_{h\to AB}$
$\displaystyle=-\int\frac{d^{3}p_{h}}{\left(2\pi\right)^{3}2p_{h}^{0}}\frac{d^{3}p_{A}}{\left(2\pi\right)^{3}2p_{A}^{0}}\frac{d^{3}p_{B}}{\left(2\pi\right)^{3}2p_{B}^{0}}\left|{\cal
M}_{h\to
AB}\right|^{2}f_{h}\\!\left(p_{h},t\right)\left(2\pi\right)^{4}\delta^{4}(p_{h}-p_{A}-p_{B})$
$\displaystyle=-\gamma\Gamma_{h},$ (41)

in which

$\displaystyle\gamma$
$\displaystyle\equiv\int\frac{d^{3}p}{\left(2\pi\right)^{3}}\frac{m_{h,{\rm
eff}}}{\sqrt{m_{h,{\rm eff}}^{2}+p^{2}}}f_{h}(p,t),$ (42)

with $f_{h}(p,t)=4f(p,t)$ by taking four degrees of freedom in $\Phi$ into
account, or $f_{h}(p,t)=f_{\rm cond}(p,t)$ for the primordial
condensate.101010 All the four modes acquire the effective mass since it comes
from the quartic coupling.

In the case of the primordial condensate (30), the Higgs decay is particularly
easy. By using $\rho_{h}=m_{h,{\rm eff}}n_{h}$ from Eq. (31), we may rewrite
the Boltzmann equation as

$\displaystyle\dot{\rho}_{h}+4H\rho_{h}=-\Gamma_{h}\rho_{h}$ (43)

after the onset of the Higgs oscillation, approximating that the quartic
coupling dominates in the potential during the oscillation [103]. The solution
of Eq. (43) is given by

$\displaystyle\rho_{h}(t)$ $\displaystyle=\rho_{h,0}\left(\frac{a}{a_{\rm
osc}}\right)^{-4}e^{-\Gamma_{h}t},$ (44)

and thus the decay time can readily be estimated as111111 Here, we have
neglected the time dependence in $\Gamma_{h}$, though the effective Higgs mass
may potentially be time-dependent during the Higgs oscillation epoch. To see
this effect, a more careful study of every possible decay channel is needed,
which is left for future work. For a related study, see Refs. [103, 104].

$\displaystyle t_{\rm dec}$ $\displaystyle=\Gamma_{h}^{-1}$
$\displaystyle\Rightarrow$ $\displaystyle\frac{a_{\rm dec}}{a_{\text{e}}}$
$\displaystyle\simeq\left(\frac{3}{2}\frac{H_{\text{e}}}{\Gamma_{h}}\right)^{2/3},$
(45)

where we have used
$t=2/3H\simeq\left(2/3H_{\text{e}}\right)\left(a/a_{\text{e}}\right)^{3/2}$.

Figure 3: Normalized decay rate of each spectrum (hard, soft relativistic,
soft non-relativistic) as a function of the scale factor $a$, with a choice of
parameters $\lambda=0.01$ and $H_{\text{e}}=m_{\phi}$.

The decay of the stochastic fluctuation is more involved, compared with the
primordial condensate. Figure 3 shows each contribution of the spectrum in the
decay rate (normalized by $\Gamma_{h}H_{\text{e}}^{3}$) as a function of the
scale factor $a$ with $\lambda=0.01$ and $H_{\text{e}}=m_{\phi}$. As seen in
the figure, although $\gamma_{\rm soft,NR}$ is the greatest contribution at
early times, the other two soon dominate and can be the main contribution in
the Higgs decay. Depending on the value of $\lambda$, they can be comparable,
and in the large-$a$ limit we find

$\displaystyle\gamma_{\rm hard}$
$\displaystyle\simeq\frac{3H_{\text{e}}^{3}}{32\pi}\left(\frac{a}{a_{\text{e}}}\right)^{-3},$
(46) $\displaystyle\gamma_{\rm soft,R}$
$\displaystyle\simeq\frac{3}{32\pi^{2}}\frac{H_{\text{e}}^{4}}{m_{h,{\rm
eff}}}\left(1-\frac{m_{h,{\rm
eff}}H_{\text{e}}^{2}}{2m_{\phi}^{3}}\right)\left(\frac{a}{a_{\text{e}}}\right)^{-3},$
(47)

where we have used Eq. (36) for $\gamma_{\rm soft,R}$. Note that the soft non-
relativistic contribution takes a simple form

$\displaystyle\gamma_{\rm soft,NR}$
$\displaystyle=\frac{9}{32\pi^{2}}\frac{H_{\text{e}}^{4}}{m_{h,{\rm
eff}}}\left(\frac{a}{a_{\text{e}}}\right)^{-9}\left[\sinh^{-1}\\!\left(\frac{am_{h,{\rm
eff}}}{k_{\rm IR}}\right)-\sinh^{-1}\\!\left(\frac{am_{h,{\rm
eff}}}{k_{*}}\right)\right],$ (48)

where $k_{\rm IR}$ is an IR cutoff which we take $k_{\rm IR}=0.05$ Mpc-1 in
the figure. A slight change in $k_{\rm IR}$, however, does not change the
order of magnitude of the decay rate, since it affects the result only
logarithmically.

Since $\gamma_{\rm hard}+\gamma_{\rm soft,R}\propto a^{-3}$, we write
$\gamma\simeq\overline{\gamma}\left(a/a_{\text{e}}\right)^{-3}$. Then, Eq.
(40) can be solved to find

$\displaystyle n_{h}(a)$
$\displaystyle=n_{h}(a_{\text{e}})\left(\frac{a}{a_{\text{e}}}\right)^{-3}-\frac{2}{3}\frac{\overline{\gamma}\Gamma_{h}}{H_{\text{e}}}\left[\left(\frac{a}{a_{\text{e}}}\right)^{-3/2}-\left(\frac{a}{a_{\text{e}}}\right)^{-3}\right],$
(49)

where, from Eq. (29), we have

$\displaystyle n_{h}(a_{\text{e}})$
$\displaystyle=\frac{3}{8\pi^{2}}\frac{H_{\text{e}}^{4}}{m_{h,{\rm eff}}}.$
(50)

Supposing that $\gamma\sim\gamma_{\rm
soft,R}\Rightarrow\overline{\gamma}\simeq\left(3/32\pi^{2}\right)\left(H_{\text{e}}^{4}/m_{h,{\rm
eff}}\right)$ and requiring $n_{h}(a_{\rm dec})=0$, we find the decay time as

$\displaystyle\frac{a_{\rm dec}}{a_{\text{e}}}$
$\displaystyle\simeq\left(6\frac{H_{\text{e}}}{\Gamma_{h}}\right)^{2/3}.$ (51)

It is interesting that this is almost the same as the decay time of the
primordial condensate given in Eq. (45).

##  6 Summary

The cosmological role of the Higgs, especially during and immediately after
inflation, has two distinctive contributions, one being the primordial
condensate and the other being the stochastic fluctuations. However, the
difference between the two is not well recognized and sometimes confused. This
is partly because such an elaborate separation is not necessary when one does
not need to care about the momentum distribution of the primordial Higgs
field. However, if, for example, the Higgs decay is taken into account, this
difference needs to be looked at more closely.

In this paper, we focused on non-thermal phase space distribution arising from
these two different sources: primordial Higgs condensate and stochastic
fluctuation. Both of these components are considered as displacement of the
Higgs field value during inflation and are therefore hardly distinguishable in
the field space. On the other hand, once inflation is over, these components
will have different phase space distributions.

The primordial condensate can be viewed as the initial field strength of the
Higgs field. It persists during inflation and starts oscillating when the
Hubble rate becomes equal to the condensate effective frequency (or mass)
after the end of inflation. The stochastic fluctuation originates from the UV
modes flowing into the IR regime. It is a secular effect if the Higgs was
massless, namely, the variance of the field grows forever. However, the
existence of the potential including the self-interaction works as a drift
term that tries to restore the original position of the field, namely, the
origin. The balance between these two effects leaves the finite variance for
the Higgs field during inflation, by which the Higgs spectrum after inflation
is affected.

The primordial Higgs condensate has the delta function-like shape of the
spectrum localized at the zero momentum state. On the other hand, the
stochastic fluctuation has a nontrivial phase space distribution which
consists of three different regimes depending on the comoving momentum $k$,
namely, $k>m_{\phi}$, $k_{*}<k<m_{\phi}$, and $k<k_{*}$. We derived an
analytic estimate and showed that the estimate is in good agreement with our
previous numerical result.

As an implication for the non-thermal Higgs spectrum, we discussed the Higgs
decay after the inflation, where we assumed that the perturbative decay is
always possible, while the Higgs remains non-thermal until it decays. We found
that the decay times of the primordial condensate and the stochastic
fluctuation almost coincide with each other, namely, both contributions decay
away almost at the same time.

Finally, we emphasize that implication of the non-thermal Higgs spectrum is
not limited to Higgs decays. Since the Higgs sector is linked to new physics
in many ways, the non-thermal Higgs spectrum can have many opportunities to be
relevant to Higgs cosmology.

## Acknowledgements

The authors would like to thank Sung-Mook Lee and Kenji Nishiwaki for the
helpful discussion at the early stages of the project. This work is in part
supported by the JSPS KAKENHI Grant Nos. 19H01899 (KK and KO) and 21H01107
(KO). KK acknowledges support by Institut Pascal at Université Paris-Saclay
during the Paris-Saclay Astroparticle Symposium 2021, with the support of the
P2IO Laboratory of Excellence (program “Investissements d’avenir”
ANR-11-IDEX-0003-01 Paris-Saclay and ANR-10-LABX-0038), the P2I axis of the
Graduate School Physics of Université Paris-Saclay, as well as IJCLab, CEA,
IPhT, APPEC, the IN2P3 master projet UCMN and EuCAPT.

## References

  * [1] ATLAS collaboration, _Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC_ , _Phys. Lett. B_ 716 (2012) 1 [1207.7214].
  * [2] CMS collaboration, _Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC_ , _Phys. Lett. B_ 716 (2012) 30 [1207.7235].
  * [3] O. Lebedev, _The Higgs portal to cosmology_ , _Prog. Part. Nucl. Phys._ 120 (2021) 103881 [2104.03342].
  * [4] V.A. Kuzmin, V.A. Rubakov and M.E. Shaposhnikov, _On the Anomalous Electroweak Baryon Number Nonconservation in the Early Universe_ , _Phys. Lett. B_ 155 (1985) 36.
  * [5] N. Turok and J. Zadrozny, _Dynamical generation of baryons at the electroweak transition_ , _Phys. Rev. Lett._ 65 (1990) 2331.
  * [6] A.I. Bochkarev, S.V. Kuzmin and M.E. Shaposhnikov, _Electroweak baryogenesis and the Higgs boson mass problem_ , _Phys. Lett. B_ 244 (1990) 275.
  * [7] N. Turok and J. Zadrozny, _Electroweak baryogenesis in the two doublet model_ , _Nucl. Phys. B_ 358 (1991) 471.
  * [8] A.E. Nelson, D.B. Kaplan and A.G. Cohen, _Why there is something rather than nothing: Matter from weak interactions_ , _Nucl. Phys. B_ 373 (1992) 453.
  * [9] N. Turok and J. Zadrozny, _Phase transitions in the two doublet model_ , _Nucl. Phys. B_ 369 (1992) 729.
  * [10] A.G. Cohen, D.B. Kaplan and A.E. Nelson, _Progress in electroweak baryogenesis_ , _Ann. Rev. Nucl. Part. Sci._ 43 (1993) 27 [hep-ph/9302210].
  * [11] K. Funakubo, A. Kakuto, S. Otsuki, K. Takenaga and F. Toyoda, _CP violating profile of the electroweak bubble wall_ , _Prog. Theor. Phys._ 94 (1995) 845 [hep-ph/9507452].
  * [12] J.M. Cline, K. Kainulainen and A.P. Vischer, _Dynamics of two Higgs doublet CP violation and baryogenesis at the electroweak phase transition_ , _Phys. Rev. D_ 54 (1996) 2451 [hep-ph/9506284].
  * [13] V.A. Rubakov and M.E. Shaposhnikov, _Electroweak baryon number nonconservation in the early universe and in high-energy collisions_ , _Usp. Fiz. Nauk_ 166 (1996) 493 [hep-ph/9603208].
  * [14] K. Funakubo, _CP violation and baryogenesis at the electroweak phase transition_ , _Prog. Theor. Phys._ 96 (1996) 475 [hep-ph/9608358].
  * [15] M. Trodden, _Electroweak baryogenesis_ , _Rev. Mod. Phys._ 71 (1999) 1463 [hep-ph/9803479].
  * [16] L. Fromme, S.J. Huber and M. Seniuch, _Baryogenesis in the two-Higgs doublet model_ , _JHEP_ 11 (2006) 038 [hep-ph/0605242].
  * [17] M. Aoki, S. Kanemura and O. Seto, _Neutrino mass, Dark Matter and Baryon Asymmetry via TeV-Scale Physics without Fine-Tuning_ , _Phys. Rev. Lett._ 102 (2009) 051805 [0807.0361].
  * [18] H.-K. Guo, Y.-Y. Li, T. Liu, M. Ramsey-Musolf and J. Shu, _Lepton-Flavored Electroweak Baryogenesis_ , _Phys. Rev. D_ 96 (2017) 115034 [1609.09849].
  * [19] K. Enomoto, S. Kanemura and Y. Mura, _Electroweak baryogenesis in aligned two Higgs doublet models_ , _JHEP_ 01 (2022) 104 [2111.13079].
  * [20] K. Enomoto, S. Kanemura and Y. Mura, _New benchmark scenarios of electroweak baryogenesis in aligned two Higgs double models_ , _JHEP_ 09 (2022) 121 [2207.00060].
  * [21] C. Grojean and G. Servant, _Gravitational Waves from Phase Transitions at the Electroweak Scale and Beyond_ , _Phys. Rev. D_ 75 (2007) 043507 [hep-ph/0607107].
  * [22] C. Caprini, R. Durrer and G. Servant, _Gravitational wave generation from bubble collisions in first-order phase transitions: An analytic approach_ , _Phys. Rev. D_ 77 (2008) 124015 [0711.2593].
  * [23] S.J. Huber and T. Konstandin, _Gravitational Wave Production by Collisions: More Bubbles_ , _JCAP_ 09 (2008) 022 [0806.1828].
  * [24] J.R. Espinosa, T. Konstandin, J.M. No and G. Servant, _Energy Budget of Cosmological First-order Phase Transitions_ , _JCAP_ 06 (2010) 028 [1004.4187].
  * [25] J.M. No, _Large Gravitational Wave Background Signals in Electroweak Baryogenesis Scenarios_ , _Phys. Rev. D_ 84 (2011) 124025 [1103.2159].
  * [26] P. Binetruy, A. Bohe, C. Caprini and J.-F. Dufaux, _Cosmological Backgrounds of Gravitational Waves and eLISA/NGO: Phase Transitions, Cosmic Strings and Other Sources_ , _JCAP_ 06 (2012) 027 [1201.0983].
  * [27] Y. Kikuta, K. Kohri and E. So, _Detecting the relic gravitational wave from the electroweak phase transition at SKA_ , 1405.4166.
  * [28] M. Kakizaki, S. Kanemura and T. Matsui, _Gravitational waves as a probe of extended scalar sectors with the first order electroweak phase transition_ , _Phys. Rev. D_ 92 (2015) 115007 [1509.08394].
  * [29] C. Caprini et al., _Science with the space-based interferometer eLISA. II: Gravitational waves from cosmological phase transitions_ , _JCAP_ 04 (2016) 001 [1512.06239].
  * [30] G.C. Dorsch, S.J. Huber, T. Konstandin and J.M. No, _A Second Higgs Doublet in the Early Universe: Baryogenesis and Gravitational Waves_ , _JCAP_ 05 (2017) 052 [1611.05874].
  * [31] C. Caprini and D.G. Figueroa, _Cosmological Backgrounds of Gravitational Waves_ , _Class. Quant. Grav._ 35 (2018) 163001 [1801.04268].
  * [32] J. Ellis, M. Lewicki and J.M. No, _On the Maximal Strength of a First-Order Electroweak Phase Transition and its Gravitational Wave Signal_ , _JCAP_ 04 (2019) 003 [1809.08242].
  * [33] C. Caprini et al., _Detecting gravitational waves from cosmological phase transitions with LISA: an update_ , _JCAP_ 03 (2020) 024 [1910.13125].
  * [34] R. Zhou and L. Bian, _Gravitational wave and electroweak baryogenesis with two Higgs doublet models_ , _Phys. Lett. B_ 829 (2022) 137105 [2001.01237].
  * [35] V. Silveira and A. Zee, _SCALAR PHANTOMS_ , _Phys. Lett. B_ 161 (1985) 136.
  * [36] J. McDonald, _Gauge singlet scalars as cold dark matter_ , _Phys. Rev. D_ 50 (1994) 3637 [hep-ph/0702143].
  * [37] C.P. Burgess, M. Pospelov and T. ter Veldhuis, _The Minimal model of nonbaryonic dark matter: A Singlet scalar_ , _Nucl. Phys. B_ 619 (2001) 709 [hep-ph/0011335].
  * [38] H. Davoudiasl, R. Kitano, T. Li and H. Murayama, _The New minimal standard model_ , _Phys. Lett. B_ 609 (2005) 117 [hep-ph/0405097].
  * [39] Y. Mambrini, _Higgs searches and singlet scalar dark matter: Combined constraints from XENON 100 and the LHC_ , _Phys. Rev. D_ 84 (2011) 115017 [1108.0671].
  * [40] S. Baek, P. Ko and W.-I. Park, _Search for the Higgs portal to a singlet fermionic dark matter at the LHC_ , _JHEP_ 02 (2012) 047 [1112.1847].
  * [41] A. Djouadi, O. Lebedev, Y. Mambrini and J. Quevillon, _Implications of LHC searches for Higgs–portal dark matter_ , _Phys. Lett. B_ 709 (2012) 65 [1112.3299].
  * [42] O. Lebedev, H.M. Lee and Y. Mambrini, _Vector Higgs-portal dark matter and the invisible Higgs_ , _Phys. Lett. B_ 707 (2012) 570 [1111.4482].
  * [43] A. Djouadi, A. Falkowski, Y. Mambrini and J. Quevillon, _Direct Detection of Higgs-Portal Dark Matter at the LHC_ , _Eur. Phys. J. C_ 73 (2013) 2455 [1205.3169].
  * [44] S. Baek, P. Ko, W.-I. Park and E. Senaha, _Vacuum structure and stability of a singlet fermion dark matter model with a singlet scalar messenger_ , _JHEP_ 11 (2012) 116 [1209.4163].
  * [45] S. Baek, P. Ko, W.-I. Park and E. Senaha, _Higgs Portal Vector Dark Matter : Revisited_ , _JHEP_ 05 (2013) 036 [1212.2131].
  * [46] H. Han and S. Zheng, _New Constraints on Higgs-portal Scalar Dark Matter_ , _JHEP_ 12 (2015) 044 [1509.01765].
  * [47] J.A. Casas, D.G. Cerdeño, J.M. Moreno and J. Quilis, _Reopening the Higgs portal for single scalar dark matter_ , _JHEP_ 05 (2017) 036 [1701.08134].
  * [48] J. Ellis, A. Fowlie, L. Marzola and M. Raidal, _Statistical Analyses of Higgs- and Z-Portal Dark Matter Models_ , _Phys. Rev. D_ 97 (2018) 115014 [1711.09912].
  * [49] Y. Mambrini, _Particles in the dark universe_ , Springer (2021), 10.1007/978-3-030-78139-2.
  * [50] K.A. Olive, _Inflation_ , _Phys. Rept._ 190 (1990) 307.
  * [51] A.D. Linde, _Particle physics and inflationary cosmology_ , vol. 5 of _Contemp. Concepts Phys._ , Harwood (1990), [hep-th/0503203].
  * [52] D.H. Lyth and A. Riotto, _Particle physics models of inflation and the cosmological density perturbation_ , _Phys. Rept._ 314 (1999) 1 [hep-ph/9807278].
  * [53] A.D. Linde, _Inflationary cosmology_ , _Phys. Rept._ 333 (2000) 575.
  * [54] J. Martin, C. Ringeval and V. Vennin, _Encyclopædia Inflationaris_ , _Phys. Dark Univ._ 5-6 (2014) 75 [1303.3787].
  * [55] J. Martin, C. Ringeval, R. Trotta and V. Vennin, _The Best Inflationary Models After Planck_ , _JCAP_ 03 (2014) 039 [1312.3529].
  * [56] J. Martin, _The Observational Status of Cosmic Inflation after Planck_ , _Astrophys. Space Sci. Proc._ 45 (2016) 41 [1502.05733].
  * [57] Planck collaboration, _Planck 2018 results. X. Constraints on inflation_ , _Astron. Astrophys._ 641 (2020) A10 [1807.06211].
  * [58] D.S. Salopek, J.R. Bond and J.M. Bardeen, _Designing Density Fluctuation Spectra in Inflation_ , _Phys. Rev. D_ 40 (1989) 1753.
  * [59] F.L. Bezrukov and M. Shaposhnikov, _The Standard Model Higgs boson as the inflaton_ , _Phys. Lett. B_ 659 (2008) 703 [0710.3755].
  * [60] F. Bezrukov, A. Magnin, M. Shaposhnikov and S. Sibiryakov, _Higgs inflation: consistency and generalisations_ , _JHEP_ 01 (2011) 016 [1008.5157].
  * [61] Y. Hamada, H. Kawai, K.-y. Oda and S.C. Park, _Higgs Inflation is Still Alive after the Results from BICEP2_ , _Phys. Rev. Lett._ 112 (2014) 241301 [1403.5043].
  * [62] F. Bezrukov and M. Shaposhnikov, _Higgs inflation at the critical point_ , _Phys. Lett. B_ 734 (2014) 249 [1403.6078].
  * [63] Y. Hamada, K.-y. Oda and F. Takahashi, _Topological Higgs inflation: Origin of Standard Model criticality_ , _Phys. Rev. D_ 90 (2014) 097301 [1408.5556].
  * [64] Y. Hamada, H. Kawai and K.-y. Oda, _Eternal Higgs inflation and the cosmological constant problem_ , _Phys. Rev. D_ 92 (2015) 045009 [1501.04455].
  * [65] Y. Hamada, H. Kawai, Y. Nakanishi and K.-y. Oda, _Meaning of the field dependence of the renormalization scale in Higgs inflation_ , _Phys. Rev. D_ 95 (2017) 103524 [1610.05885].
  * [66] R. Jinno, K. Kaneta and K.-y. Oda, _Hill-climbing Higgs inflation_ , _Phys. Rev. D_ 97 (2018) 023523 [1705.03696].
  * [67] R. Jinno, M. Kubota, K.-y. Oda and S.C. Park, _Higgs inflation in metric and Palatini formalisms: Required suppression of higher dimensional operators_ , _JCAP_ 03 (2020) 063 [1904.05699].
  * [68] S.M. Lee, K.-y. Oda and S.C. Park, _Spontaneous Leptogenesis in Higgs Inflation_ , _JHEP_ 03 (2021) 083 [2010.07563].
  * [69] J. Rubio, _Higgs inflation_ , _Front. Astron. Space Sci._ 5 (2019) 50 [1807.02376].
  * [70] J.R. Espinosa, G.F. Giudice, E. Morgante, A. Riotto, L. Senatore, A. Strumia et al., _The cosmological Higgstory of the vacuum instability_ , _JHEP_ 09 (2015) 174 [1505.04825].
  * [71] M. Herranen, T. Markkanen, S. Nurmi and A. Rajantie, _Spacetime curvature and Higgs stability after inflation_ , _Phys. Rev. Lett._ 115 (2015) 241301 [1506.04065].
  * [72] K. Kohri and H. Matsui, _Higgs vacuum metastability in primordial inflation, preheating, and reheating_ , _Phys. Rev. D_ 94 (2016) 103509 [1602.02100].
  * [73] W.E. East, J. Kearney, B. Shakya, H. Yoo and K.M. Zurek, _Spacetime Dynamics of a Higgs Vacuum Instability During Inflation_ , _Phys. Rev. D_ 95 (2017) 023526 [1607.00381].
  * [74] K. Enqvist, M. Karciauskas, O. Lebedev, S. Rusak and M. Zatta, _Postinflationary vacuum instability and Higgs-inflaton couplings_ , _JCAP_ 11 (2016) 025 [1608.08848].
  * [75] A. Joti, A. Katsis, D. Loupas, A. Salvio, A. Strumia, N. Tetradis et al., _(Higgs) vacuum decay during inflation_ , _JHEP_ 07 (2017) 058 [1706.00792].
  * [76] J.R. Espinosa, D. Racco and A. Riotto, _A Cosmological Signature of the SM Higgs Instability: Gravitational Waves_ , _JCAP_ 09 (2018) 012 [1804.07732].
  * [77] T. Markkanen, A. Rajantie and S. Stopyra, _Cosmological Aspects of Higgs Vacuum Metastability_ , _Front. Astron. Space Sci._ 5 (2018) 40 [1809.06923].
  * [78] S. Clery, Y. Mambrini, K.A. Olive and S. Verner, _Gravitational portals in the early Universe_ , _Phys. Rev. D_ 105 (2022) 075005 [2112.15214].
  * [79] S. Clery, Y. Mambrini, K.A. Olive, A. Shkerin and S. Verner, _Gravitational portals with nonminimal couplings_ , _Phys. Rev. D_ 105 (2022) 095042 [2203.02004].
  * [80] K. Enqvist, T. Meriniemi and S. Nurmi, _Generation of the Higgs Condensate and Its Decay after Inflation_ , _JCAP_ 10 (2013) 057 [1306.4511].
  * [81] K. Enqvist, T. Meriniemi and S. Nurmi, _Higgs Dynamics during Inflation_ , _JCAP_ 07 (2014) 025 [1404.3699].
  * [82] D.G. Figueroa, J. Garcia-Bellido and F. Torrenti, _Decay of the standard model Higgs field after inflation_ , _Phys. Rev. D_ 92 (2015) 083511 [1504.04600].
  * [83] S. Passaglia, W. Hu, A.J. Long and D. Zegeye, _Achieving the highest temperature during reheating with the Higgs condensate_ , _Phys. Rev. D_ 104 (2021) 083540 [2108.00962].
  * [84] R. Kallosh and A. Linde, _Universality Class in Conformal Inflation_ , _JCAP_ 07 (2013) 002 [1306.5220].
  * [85] N.A. Chernikov and E.A. Tagirov, _Quantum theory of scalar fields in de Sitter space-time_ , _Ann. Inst. H. Poincare Phys. Theor. A_ 9 (1968) 109.
  * [86] T.S. Bunch and P.C.W. Davies, _Quantum Field Theory in de Sitter Space: Renormalization by Point Splitting_ , _Proc. Roy. Soc. Lond. A_ 360 (1978) 117.
  * [87] S. Kundu, _Inflation with General Initial Conditions for Scalar Perturbations_ , _JCAP_ 02 (2012) 005 [1110.4688].
  * [88] A.A. Starobinsky, _STOCHASTIC DE SITTER (INFLATIONARY) STAGE IN THE EARLY UNIVERSE_ , _Lect. Notes Phys._ 246 (1986) 107.
  * [89] A.A. Starobinsky and J. Yokoyama, _Equilibrium state of a selfinteracting scalar field in the De Sitter background_ , _Phys. Rev. D_ 50 (1994) 6357 [astro-ph/9407016].
  * [90] T. Fujita and I. Obata, _Does anisotropic inflation produce a small statistical anisotropy?_ , _JCAP_ 01 (2018) 049 [1711.11539].
  * [91] L. Pinol, S. Renaux-Petel and Y. Tada, _Inflationary stochastic anomalies_ , _Class. Quant. Grav._ 36 (2019) 07LT01 [1806.10126].
  * [92] L. Pinol, S. Renaux-Petel and Y. Tada, _A manifestly covariant theory of multifield stochastic inflation in phase space: solving the discretisation ambiguity in stochastic inflation_ , _JCAP_ 04 (2021) 048 [2008.07497].
  * [93] A. Cable and A. Rajantie, _Free scalar correlators in de Sitter space via the stochastic approach beyond the slow-roll approximation_ , _Phys. Rev. D_ 104 (2021) 103511 [2011.00907].
  * [94] T. Fujita, K. Mukaida and Y. Tada, _Stochastic formalism for U(1) gauge fields in axion inflation_ , _JCAP_ 12 (2022) 026 [2206.12218].
  * [95] M. Honda, R. Jinno, L. Pinol and K. Tokeshi, _Borel resummation of secular divergences in stochastic inflation_ , 2304.02592.
  * [96] T. Chiba, _Slow-Roll Thawing Quintessence_ , _Phys. Rev. D_ 79 (2009) 083517 [0902.4037].
  * [97] J. Zinn-Justin, _Quantum field theory and critical phenomena_ , vol. 77 of _International Series of Monographs on Physics_ , Oxford University Press (4, 2021).
  * [98] R.J. Hardwick, V. Vennin, C.T. Byrnes, J. Torrado and D. Wands, _The stochastic spectator_ , _JCAP_ 10 (2017) 018 [1701.06473].
  * [99] K. Kaneta, S.M. Lee and K. ya Oda, _Boltzmann or bogoliubov? approaches compared in gravitational particle production_ , 2206.10929.
  * [100] M.A.G. Garcia, K. Kaneta, Y. Mambrini, K.A. Olive and S. Verner, _Freeze-in from preheating_ , _Journal of Cosmology and Astroparticle Physics_ 2022 (2021) 016 [2109.13280].
  * [101] L. Kofman, A.D. Linde and A.A. Starobinsky, _Towards the theory of reheating after inflation_ , _Phys. Rev. D_ 56 (1997) 3258 [hep-ph/9704452].
  * [102] Y. Hamada, H. Kawai, Y. Nakanishi and K.-y. Oda, _Cosmological implications of Standard Model criticality and Higgs inflation_ , _Nucl. Phys. B_ 953 (2020) 114946 [1709.09350].
  * [103] M.A.G. Garcia, K. Kaneta, Y. Mambrini and K.A. Olive, _Inflaton Oscillations and Post-Inflationary Reheating_ , _JCAP_ 04 (2021) 012 [2012.10756].
  * [104] M.A.G. Garcia, K. Kaneta, Y. Mambrini and K.A. Olive, _Reheating and Post-inflationary Production of Dark Matter_ , _Phys. Rev. D_ 101 (2020) 123507 [2004.08404].


# Text-Guided Mask-Free Local Image Retouching

###### Abstract

In the realm of multi-modality, text-guided image retouching techniques
emerged with the advent of deep learning. Most currently available text-guided
methods, however, rely on object-level supervision to confine the region of
interest that may be updated. This not only makes it more challenging to
develop these algorithms but also limits how widely deep learning can be used
for image retouching. In this paper, we offer a text-guided mask-free image
retouching approach that yields consistent results to address this concern.
Specifically, we propose a two-stage mask-free training paradigm tailored for
text-guided image retouching tasks. In the first stage, an unified mask is
proposed according to the query description, and then several candidate images
are generated with the provided mask and the conditional description based on
diffusion model. Extensive experiments have shown that our method can produce
high-quality images based on spoken language.

Index Terms—  text guided, mask free, image retouching

##  1 Introduction

With the advent of the internet and the rise in popularity of numerous
photography tools, it is now easier than ever before to obtain enormous
photos. In recent years, image retouching has attracted a significant deal of
attention due to the need of modifying photographs to accommodate a variety of
scenarios. Creating satisfying photographs using conventional techniques,
however, often necessitates a substantial amount of talent and labor.
Therefore, it would be convenient enough if we could modify photographs with
language-specific descriptions. In this work, we focus on retouching images
with language guidance.

Fig. 1: Applications of our method: from left to right are object removal,
object replacement, and background replacement.

Approaches for text-guided image retouching tasks can be easily classified
into two categories: mask-based and mask-free methods, depending on whether
the mask is used. Paint By Word [1] is the first zero-shot solution to solve
local image retouching. Following this, several diffusion-based models have
manifested themselves with increasing attention, such as Blended Diffusion
[2], Glide [3], and Blended Latent Diffusion [4]. However, it can be
challenging to create an appropriate mask for each image that requires
manipulation, particularly when the shape of the mask is complex. Therefore,
researchers attempt to pay more attention to mask-free techniques, which are
more practical. Mask-free methods can address local image retouching without
mask guidance, like VQGAN-CLIP [5], CLIP-Guided Diffusion [6], DiffusionCLIP
[7], and DiffuseIT [8]. However, these algorithms often suffer from the
drawbacks of skewed object recognition and incomplete preservation of the
original backdrop.

To cope with the above issues, we propose a two-stage framework named text-
guided mask-free local image retouching, as shown in Fig 1, which converts a
specific part of the image that matches the query to a target object described
in the text. In the first stage, our method will propose an object mask
according to the query. In the second step, given the input image, the
suggested mask, and the text description, our algorithm will generate several
candidate images, which will then be assessed to yield the final retouched
image. Compared with past approaches, our method can perform local image
editing without masking regions of interest. In addition, we also present the
multi-modality quality assessment strategy for quantifying the quality of
model outputs by picking the best one among candidate ones based on cross-
model alignment and image quality assessment.

We summarize the contributions as follows:

  * •

We propose a two-stage mask-free training paradigm tailored for text-guided
image retouching tasks. In the first stage, an object mask is proposed
according to the query, and several candidate images are generated based on
the proposed mask and the text description. Candidate images are then
evaluated to produce the final retouched image.

  * •

A location-aware refinement module is introduced in our framework, where users
can select one entity from all entities by describing it in the text.
Moreover, we also propose a multi-modal quality assessment module in which
several candidate images can be assessed to help produce high-quality and
semantically consistent retouched images.

  * •

We conducted extensive experiments on ITMSCOCO and ITFlickr which yield
promising visual results.

##  2 Related Work

###  2.1 Text-guided Image Retouching

The pioneering work [9] introduced a deep generative model that can generate
images from natural language descriptions. Later, another work [10] handled
image retouching by developing an effective GAN architecture conditioned on
text embeddings. Recent advances are fueled by autoregressive transformer
models that utilize VQ-VAE [11] to alleviate the problem of an unaffordable
amount of computation for large models. Diffusion models have been proven to
be able to generate high-quality images, especially when natural language
descriptions are given for diversity. Recent works [12, 3] explored diffusion
models for text-guided image retouching and achieved promising results with
efficient computations. However, these approaches mainly specialize in global
image retouching, ignoring the manipulation of a user-defined region of an
image, which is a common case in our daily life.

Region-based text-guided image retouching focuses on modifying a specific
region of an image while preserving the rest of it. Paint By Word [1]
generates a realistic painting of the synthesized image based on user-provided
masks and words. Furthermore, Blended Diffusion [2] was proposed for region-
based editing of generic natural images instead of synthesized ones, and
language descriptions are not restricted to a specific domain. However, the
methods mentioned above require masks to guide where to edit for region-based
image retouching. To tackle this problem, our framework utilizes a
segmentation model to generate masks based on queries that are more convenient
for users to provide.

###  2.2 Diffusion Models

Diffusion probabilistic models [13] have recently been shown to produce high-
quality images while offering sufficient image generation diversity. Ho et al.
[14] firstly proposed to synthesize images based on diffusion models. To take
more control of the generation process, SDEdit [12] was presented that keeps
low-frequency information by terminating the diffusion process early before it
turns into a Gaussian distribution. To further improve the quality of
generated images, Dhariwal et al. [15] presented a trade-off between quality
and diversity by adding attention pooling as a classifier. It beats some
famous GAN-based image-generation methods and shows the powerful generative
ability of diffusion models. At the same time, multi-modal vision and language
models trained on several large-scale datasets, such as CILP, promote the
development of diffusion models by adding auxiliary information to the
denoising process. For example, Blended Diffusion [2] utilizes CLIP to modify
parts of the image based on language and mask guidance. Different from Blended
Diffusion, DiffusionCLIP modifies the diffusion model to match up CLIP by
introducing a new loss function, which improves the diversity of the generated
effect and relieves the performance of the model collapse. Compared with
classifier guidance and CLIP guidance, classifier-free guidance [16] can
acquire more accurate and better results because it can indicate an
unconditional gradient estimation model and a conditional gradient estimation
model at the same time in the same model. Benefiting from this, several huge
diffusion models like Glide [3], DALLE2 [17], Imagen [18] are proposed and
make amazing results.

Fig. 2: The overview of our framework. It comprises two parts: image
retouching and mask generation. Based on the query, mask generation tries to
create the mask based on the query description. Image retouching focuses on
altering the image based on the provided conditional text and picking the best
of several proposals via the multi-modality quality assessment module.

##  3 Method

###  3.1 Overview

Let $I\in\mathbb{R}^{H\times W\times 3}$ be the input image, $\mathbf{q}$ be
the query description and $\mathbf{v}$ be the conditional text, respectively.
The goal of the proposed framework attempts to retouch $I$ at the local region
of interest based on $\mathbf{q}$ and $\mathbf{v}$ to produce a high-quality
natural image.

To achieve this, we draw inspiration from entity segmentation [19] to segment
all visual entities $I_{i},i\in\left\\{0,...,n-1\right\\}$ in an image based
on mask potentials $M_{i}$ without considering semantic category labels. Each
visual entity $I_{i}$ is generated as follows:

$I_{i}=I\odot M_{i}$ (1)

Where $\odot$ denotes an element-wise multiplication operation.

Then, we adopt pre-trained CLIP models to predict the correct pairings of a
batch of (image, text) examples. Assuming that $\mathbf{E}_{T}$ and
$\mathbf{E}_{I}$ are text encoder and image encoder, respectively. For each
entry in multi-modal embedding space spanning by visual entities and query
descriptions, we compute the correlation $\mathcal{C}_{i}$ as:

$\mathcal{C}_{i}=\mathbf{E}_{I}(I_{i})\cdot\mathbf{E}_{T}(\mathbf{q})$ (2)

Notably that we consider all query words as one instance to form single
embedding $T_{e}=\mathbf{E}_{T}(\mathbf{q})$, as illustrated in Fig 2. To
determine the regions that ought to be edited, the mask $\mathbf{M}_{S}$ with
high correlation confidence is selected. Formally,
$\mathbf{M}_{S}=\bigcup_{i}M_{i},~{}st.~{}\mathcal{C}_{i}\geq\tau$. In
practice, fixing threshold $\tau$ does not work well in real sceneries, and we
introduce an adaptive thresholding strategy inspired by [20]. More details can
be referred at Sec 3.2.2.

Once we obtain the accurate regions of interest, the input image $I$, and the
conditional text $\mathbf{v}$ as well as detected areas $\mathbf{M}_{S}$ are
fed into a diffusion model to generate retouched proposals
$\mathbf{P}_{k},k\in\\{0,...,m-1\\}$. $m$ is the number of proposals. We
finally attach a multi-modality quality assessment module to produce high-
quality and semantic-aware results based on proposed Cross-Model Alignment
(CMA) and Image-Quality Assessment (IQA) strategies. The subsequent sections
will detail the above components.

###  3.2 Mask Generation

Unlike existing popular local text-guided image retouching techniques, our
model will construct the mask based on the query to determine where to modify.
In particular, it can generate accurate masks in relatively simple scenarios,
however, it fails when it comes to more complex situations. In this section,
we attempt to generate more precise masks based on the query description. We
will outline them in detail below.

####  3.2.1 Entity-Based Instance Segmentation

The entity-based instance segmentation module is built upon ES [19], a novel
open-world entity segmentation task to investigate the feasibility of
convolutional center-based representation to segment things and stuff in a
unified manner. The introduced global kernel bank and overlap suppression aim
to exploit the requirements of ES in order to improve the segmentation quality
of predicted entity masks. As a result, the mask potentials $M_{i}$ and visual
entities $I_{i}$ are obtained for subsequent processing.

####  3.2.2 Text-Image Alignment

We leverage the CLIP [21] strategy to model cross-modality relations for query
description $\mathbf{q}$ and visual entities $I_{i}$. As stated in eq 2,
$\tau$ plays an important role in obtaining precise regions. Thus, we adopt an
adaptive thresholding strategy based on sampling over cumulative correlation
confidence. Specifically, we first sort the correlation values in
$\mathcal{C}_{i},i\in\\{0,...,n-1\\}$ from high to low and calculate the
cumulative confidence. Then the adaptive threshold $\tau$ is obtained based on
the inverse transform sampling over the maximal contribution step.

####  3.2.3 Location-Aware Refinement

The above text-image alignment procedure can work well in most cases, but it
may fail when descriptions contain location constraints. To tackle this issue,
we propose a location-aware refinement module that amends the mask region as
expected. We simply split the image space into nine regions evenly and bag
each detected visual entity into the corresponding location. Recall that we
separate the regions and background based on the correlation confidence
$\mathcal{C}$, and we proceed to perform a location-aware refinement operation
by restricting the orientation.

###  3.3 Text-Guided Image Retouching

After obtaining regions $\mathbf{M}_{S}$ correlated with query description
$\mathbf{q}$, we further retouch the image $I$ with conditional text
$\mathbf{v}$ with provided mask $\mathbf{M}_{S}$. The input image $I$ and the
conditional text $\mathbf{v}$ will be compressed into the same latent space,
respectively. Formally, denoting $z_{0}$ as the initial latent vector, we
utilize the denoising network $\epsilon_{\theta}(z_{t},t,\mathbf{v})$ to
reconstruct $z$ from the Gaussian distribution $\mathcal{N}(0,I)$. Thus, we
can simplify the corresponding objective as:

$\mathcal{L}:=\mathbb{E}_{z,\epsilon\sim\mathcal{N}(0,I),t}\left[\left\|\epsilon-\epsilon_{\theta}(z_{t},t,\mathbf{v})\right\|\right]$
(3)

For timestep $t$, the denoising result $z^{{}^{\prime}}_{t}$ is further
updated with mask $\mathbf{M}_{S}$ as:

$z^{{}^{\prime}}_{t}=z_{t}\odot(1-\mathbf{M}_{S})+z^{{}^{\prime}}_{t}\odot\mathbf{M}_{S}$
(4)

Eq 4 indicates that we keep the original background while learning text-guided
visual entities based on the conditional text $\mathbf{v}$. The last
$z^{{}^{\prime}}_{t}$ is put forward into a decoder network to generate
plausible proposals $\mathbf{P}_{k},k\in\\{0,...,m-1\\}$.

###  3.4 Multi-Modality Quality Assessment

Randomization is an essential factor of the denoising procedure, which assures
the variety of the diffusion model, yet it may also lead to undesirable
outcomes. To address this issue, we present a multi-modality quality
assessment module for selecting the best result from several candidates. In
this module, we examine the quality of the output from two perspectives:
cross-model assessment and image quality assessment.

####  3.4.1 Cross-Model Alignment

In practice, we employ CLIP to measure the consistency between proposals
$\mathbf{P}_{k}$ and the conditional text $\mathbf{v}$ as described in section
3.2.2. The cross-model confidence $\mathcal{C}^{\prime}_{k}$ for proposal
$\mathbf{P}_{k}$ is then defined as:

$\mathcal{C}^{\prime}_{k}=\mathbf{E}_{I}(\mathbf{P}_{k})\cdot\mathbf{E}_{T}(\mathbf{v})$
(5)

Noted that $\mathcal{C}^{\prime}_{k}$ is normalized to $[0,1]$. Recall that
$\mathbf{E}_{T}$ and $\mathbf{E}_{I}$ are text encoder and image encoder
before.

####  3.4.2 Image Quality Assessment

In addition to cross-modality alignment, we shall also pay attention to the
visual quality of the generated result. Therefore, we introduce an image
quality assessment module to enable high-quality output images. Formally, we
define the image quality confidence score $\mathcal{S}_{k}$ as follows:

$\mathcal{S}_{k}=\frac{1}{H*W}\sum_{i}\sum_{j}\|I(i,j)-\mathbf{P}_{k}(i,j)\|$
(6)

Finally, we choose the best suitable proposal $\mathbf{P}_{k^{\prime}}$ that
fulfills:

$k^{\prime}:=\operatorname*{arg\,max}_{k}(\mathcal{C}^{\prime}_{k}-\alpha\cdot\mathcal{S}_{k})$
(7)

Where $\alpha$ is the hyperparameter, and we empirically set $\alpha=5.0$.
Experiments indicate they can increase the overall quality of image retouching
results with semantic consistency.

##  4 Experiment

###  4.1 Experimental Settings

We follow [22] to generate ITMSCOCO and ITFlickr by selecting 5000 text-image
pairs from MSCOCO [23] and Fliker30K [24], respectively. Furthermore, SSIM,
PSNR, FID, MSE, and LPIPS are utilized as metrics to quantify the quality of
retouching results.

###  4.2 Compared to Existing Methods

Fig. 3: After each image was assessed by 20 participants, we computed the mean
scores of Background / Structure, Text Alignment and Realism) for 20 generated
images.

To evaluate the performance of our method, we compare our results against the
following state-of-the-art baseline models: Glide, Blended Latent Diffusion,
VQGAN-CLIP, CLIP-Guided Diffusion, and DiffuseIT. Experiments show that our
model outperforms the above methods qualitatively and quantitatively.

####  4.2.1 Qualitative Comparison

As shown in Fig 7, we compare our results with two mask-based methods and
three mask-free methods. The masks for the mask-based methods are generated by
our method, which shows our superiority that we needn’t provide the mask for
the regions of interest. As for mask-free methods, the main drawback of them
is that they couldn’t preserve the whole background well, which may lead to
the failure of retouching, besides, these methods may fail when handling
complex situations like Column 7 of Fig 7. As a result, our method brings the
best of both worlds and obtains persuasive results.

####  4.2.2 Quantitative Comparison

In the realm of text-guided image retouching, it’s difficult to measure the
performance of models by metrics quantitatively. Therefore, we introduce a
user study, as depicted in Table 1 and Fig. 3. The study participants were
instructed to evaluate each result based on (1) the preservation of the
background and structure relative to the source image, (2) alignment with the
corresponding text, and (3) the realism of the output. A total of 20
participants took part in the study and evaluated 20 randomly selected
examples for each method. The examples were presented in random order. For
each example, the participants rated the quality on a scale of 0 to 5, with
higher ratings indicating better performance.

Table 1: The quantitative statistical rating of generated results. $\uparrow$
means that the higher the value, the better. Red indicates the best scores,
while blue indicates the second-best results.

Methods | Background / | Text Alignment ($\uparrow$) | Realism ($\uparrow$)
---|---|---|---
Structure ($\uparrow$)
Glide [3] | 4.36±0.59 | 2.97±1.16 | 3.47±0.9
Blended [4] | 4.19±0.64 | 3.29±1.16 | 3.23±0.95
VQGAN [5] | 0.93±0.28 | 2.25±0.73 | 1.27±0.58
Clip-guided [6] | 1.68±1.5 | 2.25±1.11 | 2.35±1.35
DiffuseIT [8] | 2.58±0.94 | 2.66±0.87 | 2.5±0.96
Ours | 4.52±0.3 | 4.21±0.53 | 4.03±0.65

###  4.3 Ablation Studies

In this section, we conduct ablation studies to verify the effectiveness of
the mask generation and image retouching modules.

####  4.3.1 Mask generation

Adaptive Threshold As illustrated in Fig 4, the empirically rough threshold
often cause the loss of visual entities when complicated scenarios are
encountered. While our adaptive threshold exhibits much more characteristics
that capture the regions of interest well.

Fig. 4: Adaptive threshold. We empirically set the threshold to $\tau$ to 0.2
for the baseline.

Location-Aware Refinement We demonstrate the effectiveness of the proposed
location-aware refinement in Fig 5 and most common positional restraints are
well retained in real cases.

Fig. 5: Visual entities with orientation.

####  4.3.2 Multi-Modality Quality Assessment

CMA vs. IQA.

As can be seen in Tab 2, we evaluate the CMA and IQA modules with the mean
values of all validation images. The quantitative results just reveal that the
IQA component plays an essential role in the production of high-quality
images, while the qualitative visual results in Fig 6 show us the significant
involvement of CMA to yield semantic-aware targets.

Table 2: CMA vs. IQA. We carry out comparative experiments on the ITMSCOCO
datasets. The max timestep is 200, and the proposal number $m$ equals 4.

CMA | IQA | SSIM($\uparrow$) | PSNR($\uparrow$) | FID($\downarrow$) | MSE($\downarrow$) | LPIPS($\downarrow$)
---|---|---|---|---|---|---
$\times$ | $\times$ | 0.7042 | 17.7389 | 16.7268 | 0.0218 | 0.2673
$\surd$ | $\times$ | 0.7121 | 17.7823 | 14.5790 | 0.0191 | 0.2608
$\times$ | $\surd$ | 0.7218 | 18.6803 | 12.7913 | 0.0160 | 0.2499
$\surd$ | $\surd$ | 0.7125 | 17.8141 | 14.5512 | 0.0190 | 0.2605

Fig. 6: The effectiveness of CMA and IQA.
$\mathbf{P}_{1},\mathbf{P}_{2},\mathbf{P}_{3},\mathbf{P}_{4}$ are ranked only
based on CMA strategy, while
$\mathbf{P}_{3},\mathbf{P}_{4},\mathbf{P}_{2},\mathbf{P}_{1}$ are ranked by
the IQA module. After combining the CMA and IQA module, the final result
changed to $\mathbf{P}_{3},\mathbf{P}_{2},\mathbf{P}_{1},\mathbf{P}_{4}$. Fig.
7: Qualitative comparison of text-guided image retouching on MSCOCO dataset.
We choose two mask-based methods and three mask-free methods. Our approach
generates realistic samples based on the conditional text with better
perceptual quality than the baselines.

##  5 Conclusion

This study offers a two-stage framework for text-guided mask-free local image
retouching. This framework transforms a specified region of the image that
fits the query into a text-described target object. Extensive experiments
demonstrate the superiority of the proposed approach against existing methods.

## Acknowledgments

This work is partially supported by the National Natural Science Foundation of
China (Grant No. 62106235), by the Exploratory Research Project of Zhejiang
Lab(2022PG0AN01), by the Zhejiang Provincial Natural Science Foundation of
China (LQ21F020003).

## References

  * [1] David Bau, Alex Andonian, Audrey Cui, YeonHwan Park, Ali Jahanian, Aude Oliva, and Antonio Torralba,  “Paint by word,”  arXiv preprint arXiv:2103.10951, 2021.
  * [2] Omri Avrahami, Dani Lischinski, and Ohad Fried,  “Blended diffusion for text-driven editing of natural images,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 18208–18218.
  * [3] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen,  “Glide: Towards photorealistic image generation and editing with text-guided diffusion models,”  arXiv preprint arXiv:2112.10741, 2021.
  * [4] Omri Avrahami, Ohad Fried, and Dani Lischinski,  “Blended latent diffusion,”  arXiv preprint arXiv:2206.02779, 2022.
  * [5] Katherine Crowson, Stella Biderman, Daniel Kornis, Dashiell Stander, Eric Hallahan, Louis Castricato, and Edward Raff,  “Vqgan-clip: Open domain image generation and editing with natural language guidance,”  in European Conference on Computer Vision. Springer, 2022, pp. 88–105.
  * [6] Katherine Crowson,  “Clip-guided diffusion,” https://github.com/afiaka87/clip-guided-diffusion.
  * [7] Gwanghyun Kim, Taesung Kwon, and Jong Chul Ye,  “Diffusionclip: Text-guided diffusion models for robust image manipulation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2426–2435.
  * [8] Gihyun Kwon and Jong Chul Ye,  “Diffusion-based image translation using disentangled style and content representation,”  arXiv preprint arXiv:2209.15264, 2022.
  * [9] Elman Mansimov, Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov,  “Generating images from captions with attention,”  arXiv preprint arXiv:1511.02793, 2015.
  * [10] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee,  “Generative adversarial text to image synthesis,”  in International conference on machine learning. PMLR, 2016, pp. 1060–1069.
  * [11] Aaron Van Den Oord, Oriol Vinyals, et al.,  “Neural discrete representation learning,”  Advances in neural information processing systems, vol. 30, 2017\.
  * [12] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon,  “Sdedit: Guided image synthesis and editing with stochastic differential equations,”  in International Conference on Learning Representations, 2021.
  * [13] Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli,  “Deep unsupervised learning using nonequilibrium thermodynamics,”  in International Conference on Machine Learning. PMLR, 2015, pp. 2256–2265.
  * [14] Jonathan Ho, Ajay Jain, and Pieter Abbeel,  “Denoising diffusion probabilistic models,”  Advances in Neural Information Processing Systems, vol. 33, pp. 6840–6851, 2020.
  * [15] Prafulla Dhariwal and Alexander Nichol,  “Diffusion models beat gans on image synthesis,”  Advances in Neural Information Processing Systems, vol. 34, pp. 8780–8794, 2021.
  * [16] Jonathan Ho and Tim Salimans,  “Classifier-free diffusion guidance,”  arXiv preprint arXiv:2207.12598, 2022.
  * [17] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen,  “Hierarchical text-conditional image generation with clip latents,”  arXiv preprint arXiv:2204.06125, 2022.
  * [18] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, et al.,  “Photorealistic text-to-image diffusion models with deep language understanding,”  arXiv preprint arXiv:2205.11487, 2022.
  * [19] Lu Qi, Jason Kuen, Yi Wang, Jiuxiang Gu, Hengshuang Zhao, Zhe Lin, Philip Torr, and Jiaya Jia,  “Open-world entity segmentation,”  arXiv preprint arXiv:2107.14228, 2021.
  * [20] Hui Su, Yue Ye, Zhiwei Chen, Mingli Song, and Lechao Cheng,  “Re-attention transformer for weakly supervised object localization,”  arXiv preprint arXiv:2208.01838, 2022.
  * [21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.,  “Learning transferable visual models from natural language supervision,”  in International Conference on Machine Learning. PMLR, 2021, pp. 8748–8763.
  * [22] Ruotian Luo, Brian Price, Scott Cohen, and Gregory Shakhnarovich,  “Discriminability objective for training descriptive captions,”  arXiv preprint arXiv:1803.04376, 2018.
  * [23] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick,  “Microsoft coco: Common objects in context,”  in European conference on computer vision. Springer, 2014, pp. 740–755.
  * [24] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier,  “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,”  Transactions of the Association for Computational Linguistics, vol. 2, pp. 67–78, 2014.