11institutetext: Lampros Gavalakis 22institutetext: Department of Engineering,
University of Cambridge, Trumpington Street, Cambridge CB2 1PZ, U.K. 22email:
<EMAIL_ADDRESS>33institutetext: Ioannis Kontoyiannis 44institutetext:
Statistical Laboratory, DPMMS, University of Cambridge, Centre for
Mathematical Sciences, Wilberforce Road, Cambridge CB3 0WB, U.K. 44email:
<EMAIL_ADDRESS>
# Information in probability:
Another information-theoretic proof of a finite de Finetti theorem

Lampros Gavalakis and Ioannis Kontoyiannis

###### Abstract

We recall some of the history of the information-theoretic approach to
deriving core results in probability theory and indicate parts of the recent
resurgence of interest in this area with current progress along several
interesting directions. Then we give a new information-theoretic proof of a
finite version of de Finetti’s classical representation theorem for finite-
valued random variables. We derive an upper bound on the relative entropy
between the distribution of the first $k$ in a sequence of $n$ exchangeable
random variables, and an appropriate mixture over product distributions. The
mixing measure is characterised as the law of the empirical measure of the
original sequence, and de Finetti’s result is recovered as a corollary. The
proof is nicely motivated by the Gibbs conditioning principle in connection
with statistical mechanics, and it follows along an appealing sequence of
steps. The technical estimates required for these steps are obtained via the
use of a collection of combinatorial tools known within information theory as
‘the method of types.’

##  1 Entropy and information in probability

Shannon’s landmark 1948 paper shannon:48  founded the field of information
theory and ignited the fuse that led to much of the subsequent explosive
development of communications theory and engineering in the 20th century. At
the same time, it also led to a wave of applications of information theory to
numerous other branches of science. Of those, some, e.g. those in
bioinformatics and neuroscience, were successful, while some others, despite
Shannon’s “bandwagon” warning shannon:56 , much less so.

Within mathematics, information-theoretic ideas have had a major impact along
several directions, perhaps most notably (although certainly not exclusively)
in connection with probability theory. For our present purposes, the most
relevant line of work is based on the idea of utilising information-theoretic
tools and ideas in order to prove core probabilistic results. Over the past 55
years, a great number of such proofs have appeared. These are often
accompanied by new interpretations and rich intuition, thus providing new ways
of understanding why fundamental probabilistic theorems are true, and
sometimes also giving stronger versions of the original results.

In the rest of this introduction we describe some of the main landmarks along
this path, and we indicate directions of current and likely near-future
activity. This brief survey is necessarily incomplete and biased, due to our
own subjective taste and bounded knowledge. Then in Section 2 we state and
prove a new finite version of de Finetti’s classical representation theorem
for finite-valued exchangeable random variables.

The first appearance of information-theoretic ideas in the proof of a
genuinely probabilistic result was in 1958, when Hájek hajek:58a ; hajek:58b
proved that the laws $\mu$ and $\nu$ of any two Gaussian processes are either
absolutely continuous with respect to each other, or singular. Hájek exploited
the implications of $D(\mu\|\nu)+D(\nu\|\mu)$ being finite or infinite, where
$D(\mu\|\nu)$ denotes the relative entropy or Kullback-Leibler divergence
between $\mu$ and $\nu$,

$\displaystyle
D(\mu\|\nu):=\begin{cases}\int\log\frac{d\mu}{d\nu}\,d\mu,&\mbox{if}\;\frac{d\mu}{d\nu}\;\mbox{exists}\\\
+\infty,&\mbox{otherwise.}\end{cases}$ (1)

[Throughout, log denotes the natural logarithm.] In the same year, Kolmogorov
kolmogorov:58  introduced entropy in ergodic theory. He provided a way to
calculate the entropy of a transformation to conclude that Bernoulli shifts of
different entropies are not metrically isomorphic. The importance of entropy
in ergodic theory was also highlighted more than a decade later, when Ornstein
ornstein:70a ; ornstein:70b ; ornstein:70c  proved that Bernoulli shifts with
the same entropy are necessarily isomorphic.

The following year, 1959, Linnik linnik:59  gave an information-theoretic
proof of the central limit theorem (CLT), showing that the law of the
standardised sum $S_{n}=(1/\sigma\sqrt{n})\sum_{i=1}^{n}X_{i}$ of $n$
independent and identically distributed (i.i.d.) random variables
$X_{1},\ldots,X_{n}$ with variance $\sigma^{2}$ (or, more generally, of
independent random variables satisfying the Lindeberg condition) converges in
distribution to a Gaussian. Linnik’s connection between the CLT and
information-theoretic ideas was the first in a long series of works, along a
path that remains active until today. Indeed, in a sequence of papers,
including works by Shimizu shimizu:75 , Brown brown:82 , Barron barron:clt ,
Johnson johnson:00 , Artstein et al. artstein:04 , Tulino and Verdú tulino:06
, and Madiman and Barron madiman:07 , it was shown that the differential
entropy $h(S_{n})$ of the standardised sums in fact increases with $n$, and
its limiting value is the entropy $h(Z)$ of the standard Gaussian $Z\sim
N(0,1)$. This monotonic convergence in combination with the fact that the
Gaussian has maximum entropy among all random variables with variance
$\sigma^{2}$, presents an appealing analogy between the CLT and the second law
of thermodynamics.

In the above discussion, the differential entropy of a continuous random
variable $X$ with density $f$ is given by $h(X)=h(f)=-\int f\log f$, and the
relative entropy between two probability measures $\mu,\nu$ on ${\mathbb{R}}$
with densities $f,g$ is $D(\mu\|\nu)=D(f\|g)=\int f\log(f/g)$. A simple
computation shows that the “entropic CLT” just described can equivalently be
stated as, $D(f_{n}\|\phi)\downarrow 0$ as $n\to\infty$, where $f_{n}$ is the
density of $S_{n}$ and $\phi$ the standard normal density. This convergence in
the sense of relative entropy implies convergence in $L^{1}$: Pinsker’s
inequality, established by Csiszár csiszar:67 , Kullback kullback:67  and
Kemperman kemperman:69 , states that:

$D(\mu\|\nu)\geq\frac{1}{2\log{2}}\|\mu-\nu\|_{1}^{2}.$ (2)

Subsequent work along these lines includes Carlen and Soffer’s dynamical
systems approach carlen:91 , Johnson’s convergence to Haar measure on compact
groups johnson:00b , Johnson and Barron’s rates of convergence in the entropic
CLT johnson-barron:04 , Bubeck and Ganguly’s entropic CLT for Wishart random
matrices bubeck:18 , and, most recently, an information-theoretic CLT for
discrete random variables gavalakis-arxiv:clt .

A year after Linnik’s paper made the first entropy-CLT connection, in 1960,
Rényi renyi:61  examined the convergence of Markov chains to equilibrium from
an information-theoretic point of view, thus initiating another path of
information-theoretic investigation in probability. Rényi showed that the
relative entropy $D(P_{n}\|\pi)$ between the time-$n$ distribution $P_{n}$ of
a finite-state chain with an all-positive transition matrix and its unique
invariant distribution $\pi$, decreases to zero as $n\to\infty$. Similar and
slightly more general results were established independently by Csiszár
csiszar:63  in 1963, who also employed Rényi’s notion of $f$-divergence, an
important generalisation of relative entropy. In the same year, Kendall
kendall:63  extended Rényi’s techniques and results, to include certain
countable state space chains. A significant advance came with Fritz’s 1973
work fritz:73 , where he studied the asymptotic behavior of reversible Markov
kernels and established their weak convergence to equilibrium. Barron in 2000
barron:isit00  extended Fritz’s result to convergence in relative entropy, and
in 2009 Harremoës and Holst harremoes:09  used ideas related to information
projections to further extend and generalise those earlier results.

The problem of Poisson approximation and convergence was first examined
through the lens of information theory around 20 years ago, leading to a
development analogous to that of the entropic CLT. Harremoës in 2001
identified the Poisson as the maximum entropy distribution among all laws that
arise from sums of independent Bernoulli random variables with a fixed mean
harremoes:01 . This characterisation was extended in 2007 by Johnson
johnson:07  to the class of ultra log-concave laws on the nonnegative
integers. Meanwhile, in 2005 Kontoyiannis et al. konto-H-J:05  derived
convergence results and nonasymptotic Poisson approximation bounds using
entropy-theoretic methods. Interestingly, some of those results were based, in
part, on a discrete modified logarithmic Sobolev inequality for the entropy
established by Bobkov and Ledoux bobkov-ledoux:98 . In a related direction,
Harremoës et al. harremoesJK:07 ; konto-H-J:10  obtained Poisson approximation
results under the thinning operation.

A similar program was carried out in the case of compound Poisson
approximation. Compound Poisson laws on the integers were again given a
natural maximum-entropy interpretation by Johnson et al. jKm:13  and Yu yu:09
, and compound Poisson approximation bounds and convergence results were
established via information-theoretic techniques by Madiman et al.
madimanJK:07  and Barbour et al. Kcompound:10 . Interestingly, in some cases
the resulting nonasymptotic bounds give the best results to date.

The method of types and large deviations.  Suppose $\\{X_{n}\\}$ are i.i.d.
random variables with common probability mass function (PMF) $Q$ on a finite
alphabet $A$ of size $m=|A|$. The type $\hat{P}_{n}$ of a string
$x_{1}^{n}=(x_{1},\ldots,x_{n})\in A^{n}$ is simply the empirical PMF induced
by $x_{1}^{n}$ on $A$. Let ${\cal P}_{n}$ denote the collection of all
$n$-types on $A$, namely, all PMFs that arise as types of strings of length
$n$. Then, e.g., we have the obvious bound,

$\displaystyle|{\cal P}_{n}|\leq(n+1)^{m},$ (3)

and direct computation also shows that, for any $x_{1}^{n}\in A^{n}$,

$\displaystyle Q^{n}(x_{1}^{n})=e^{-n[H(\hat{P}_{n})+D(\hat{P}_{n}\|Q)]}.$ (4)

Here, $H(P):=-\sum_{x\in A}P(x)\log P(x)$ is the (discrete Shannon) entropy of
a PMF $P$ on $A$, and the definition (1) of the relative entropy $D(P\|Q)$
between two PMFs $P,Q$ on the same discrete alphabet becomes
$D(P\|Q)=\sum_{x\in A}P(x)\log[P(x)/Q(x)]$. Slightly more involved
calculations lead to interesting and useful bounds. For example, for an
$n$-type $P$, let $T(P)$ denote the type class of $P$, consisting of all
$x_{1}^{n}\in A^{n}$ with type $P$. Then the cardinality and probability of
$T(P)$ satisfy,

$\displaystyle(n+1)^{-m}e^{nH(P)}$ $\displaystyle\leq$
$\displaystyle|T(P)|\;\leq\;e^{nH(P)}$ (5)
$\displaystyle(n+1)^{-m}e^{-nD(P\|Q)}$ $\displaystyle\leq$ $\displaystyle
Q^{n}(T(P))\;\leq\;e^{-nD(P\|Q)}.$ (6)

The method of types is a collection of combinatorial estimates for
probabilities associated with discrete i.i.d. random variables and memoryless
channels, of which the examples in (3)–(6) above are the starting point. Based
in part on preliminary ideas of Wolfowitz wolfowitz:61book , the method of
types was fully developed in 1981 by Csiszár and Körner csiszar:book . As
described in Csiszár’s review csiszar:98 , the method of types has been
employed very widely and with great success in numerous information-theoretic
problems arising from different communication-theoretic scenarios.

Based in part on the method of types, and also building on ideas from related
work by Groenebook et at. GOR:79 , Csiszár was able to establish a series of
important results in large deviations. In 1975 csiszar:75  he identified the
exponent in Sanov’s theorem sanov:57  as an extremum of relative entropies,
and in 1984 csiszar:84  he proved a general, strong version of Sanov’s
theorem, by a combination of the method of types, discretisation arguments,
and a general Pythagorean inequality for the relative entropy established by
Topsøe topsoe:79 . He also gave a simpler proof along the same lines in his
2006 paper csiszar:06 .

Moreover, in the same paper csiszar:84  Csiszár established a version of the
Gibbs conditioning principle (also know as the conditional limit theorem)
using the same tools. This was further extended by Csiszár et al. in 1987
csiszar-cover-choi:87  to the case of Markov conditioning, and by Algoet et
al. in 1992 algoet:92  to Markov types.

The method of types and the Gibbs conditioning principle will both play an
important role in our proof of the finite de Finetti theorem in Section 2.

Exchangeability.  Suppose $\\{X_{n}\\}$ are i.i.d. random variables, and let
$\mathcal{E}$ denote the exchangeable $\sigma$-algebra, that is, the
sub-$\sigma$-algebra of $\sigma(\\{X_{n}\\})$ that consisting of those events
that are invariant under finite permutations of the indices in the sequence
$\\{X_{n}\\}$. In 2000, O’Connell oconnell:TR  gave a beautiful, elementary
information-theoretic proof the Hewitt-Savage 0-1 law hewitt-savage:55 :
$\mathcal{E}$ is trivial, in that all events in ${\cal E}$ have probability
either zero ore one.

Another aspect of exchangeability comes up in connection with de Finetti’s
theorem. Let $\\{X_{n}\\}$ be an exchangeable sequence of random variables
with values in the same finite alphabet $A$. Here, exchangeability means that,
for any $n$ and any permutation $\pi$ on $\\{1,2,\dots,n\\}$, the distribution
of the random variables $(X_{\pi(1)},X_{\pi(2)},\ldots,X_{\pi(n)})$ is the
same as that of $(X_{1},X_{2},\ldots,X_{n})$. De Finetti’s theorem
definetti:31 ; definetti:37  states that $\\{X_{n}\\}$ is exchangeable if and
only if it is a mixture of i.i.d. sequences, that is, if and only if there is
a measure $\bar{\mu}$ on the simplex ${\cal P}$ of probability distributions
on $A$, such that, for any $k\geq 1$ and any
$x_{1}^{k}=(x_{1},\ldots,x_{k})\in A^{k}$,

$\displaystyle\mathbb{P}(X_{1}^{k}=x_{1}^{k})=M_{\bar{\mu},k}(x_{1}^{k}):=\int_{\cal
P}Q^{k}(x_{1}^{k})d\bar{\mu}(Q).$ (7)

De Finetti’s theorem plays an important role in the foundations of subjective
probability and Bayesian statistics, see, e.g., the discussions in diaconis:77
; bayarri:04 . But arguments about its practical relevance are limited by the
fact that, as is well known diaconis-freedman:80b , the representation (7)
fails in general if it is only assumed that a finite collection of random
variables $(X_{1},\ldots,X_{n})$ is exchangeable for some fixed $n$.
Nevertheless, approximate versions of (7) remain valid in this case
diaconis:77 ; diaconis-freedman:80b . Such a ‘finite’ version of de Finetti’s
theorem for binary random variables was recently established in gavalakis:21 ,
using information-theoretic ideas: It was shown that there is a mixing measure
$\mu$ on ${\cal P}$ such that, for any $1\leq k\leq n$, the distribution
$P_{k}$ of $(X_{1},\ldots,X_{k})$ is close to $M_{\mu,k}$ in the precise sense
that:

$D(P_{k}\|M_{\mu,k})\leq\frac{5k^{2}\log{n}}{n-k}.$ (8)

A different information-theoretic proof of a different finite version of de
Finetti’s theorem is given in Section 2.

Further connections.  There are numerous other directions along which
information-theoretic methods have been employed to establish either known or
new probabilistic results. We briefly mention only a few more from the long
list of relevant works, some of which go beyond probability theory. The
interested reader may also consult Barron’s reviews of information-theoretic
proofs and connections with statistics and learning barron:97 ; barron:isit00
, Csiszár’s review of information-theoretic methods in probability csiszar:97
, and Johnson’s text johnson:book .

A natural and powerful connection has been drawn between information theory
and concentration of measure inequalities, through what has come to be known
as the entropy method. Often attributed Herbst davies-simon:84 , the entropy
method was primarily developed by Ledoux ledoux:96 ; ledoux:97 ; ledoux:book .
Marton’s 1996 work marton:96  had an early and significant influence in this
direction as well. Entropy also appears naturally in connection with related
work on transportation theory bobkov:99 ; villani:book . Book-length accounts
of measure concentration and related inequalities, including the entropy
method, are given in boucheron:book  and raginsky:book .

A fascinating and multifaceted series of connections between information-
theoretic ideas and functional inequalities started with Shannon’s entropy
power inequality (EPI), stated in his original 1948 paper shannon:48  and
later proved by Stam stam:59  and Blachman blachman:65 . Much of the relevant
literature up to 1991 is summarised in Dembo et al.’s review dembo-cover-
thomas:91 , including the connection with Gross’ celebrated Gaussian
logarithmic Sobolev inequality gross:75 . This paper also contains an early
discussion of the strong ties between entropy inequalities and high-
dimensional convex geometry, starting with Costa and Cover’s 1984 observation
costa-cover:84  that the Brunn-Minkowski inequality can be viewed as a special
case of a generalised EPI.

Building on the technical ideas of Stam and Blachman, Bakry and Émery in a
very influential 1985 paper bakry-emery:85  derived an important
representation of the derivative of the relative entropy $D(P_{t}\|Q_{t})$ of
the time-$t$ distributions of a diffusion with different initial conditions.
Under appropriate assumptions, strong connections were established with
logarithmic Sobolev inequalities, generalising the earlier connection between
the EPI and Gross’ Gaussian logarithmic Sobolev inequality, and facilitating
the study of the long-term behaviour of the underlying diffusion. An important
observation, independently re-discovered by Barron barron:97 , is that this
derivative can be expressed as a “relative” Fisher information, which also
admits an interpretation as a minimum mean squared error. This interpretation
had been promoted earlier in work by Brown, see e.g. brown:86 , and it was re-
framed in more information theoretic terms by Guo et al. in 2005 guo:05 ,
leading to a variety of subsequent developments.

More recently, a remarkable equivalence between the subadditivity property of
entropy and the classical Brascamp-Lieb inequality was pointed out by Carlen
and Cordero-Erausquin carlen:09 , and a unified information-theoretic
treatment was given by Liu et al. courtade:16 . In yet another direction, Tao
in 2010 tao:10  developed a series of discrete entropy inequalities motivated
by sumset and inverse sumset bounds in additive combinatorics, also leading to
a discrete version of the EPI. More recent work in this direction includes
KM:14 ; KM:16 .

Finally, we mention that a natural analog of the entropy in free probability
was introduced in a series of papers by Voiculescu voiculescu:I ;
voiculescu:II ; voiculescu:III ; voiculescu:IV , where several properties of
the free entropy are established, including a free version of the EPI. In
related work, convergence results to maximum free entropy distributions is
considered by Johnson in (johnson:book, , Chapter 8).

##  2 Information-theoretic proof of a finite de Finetti theorem

Suppose $X_{1}^{n}=(X_{1},\ldots,X_{n})$, for some fixed $n$, are
exchangeable, discrete random variables, with values in a finite alphabet $A$
of $m=|A|$ elements. Let $\hat{P}_{X_{1}^{n}}$ denote the (random) type of
$X_{1}^{n}$, and let the measure $\mu=\mu_{n}$ denote the law of
$\hat{P}_{X_{1}^{n}}$ on the probability simplex ${\cal P}$. In this section
we provide an information-theoretic proof of the following:

######  Theorem 2.1 (Finite de Finetti theorem)

For any $1\leq k\leq n$, let $P_{k}$ denote the distribution of
$X_{1}^{k}=(X_{1},\ldots,X_{k})$ and $M_{\mu,k}$ denote the mixture-
of-i.i.d.s:

$M_{\mu,k}(x_{1}^{k})=\int_{\cal P}Q^{k}(x_{1}^{k})d\mu(Q),\qquad x_{1}^{k}\in
A^{k}.$

For any $1\leq k\leq(n/100)^{1/3}$, we have,

$D(P_{k}\|M_{\mu,k})\leq\epsilon(n,k):=2\delta+ke^{-\frac{n}{k}\delta}\Bigl{(}\frac{n}{k}+1\Bigr{)}^{2m^{k}}\log
n,$ (9)

with
$\alpha=\alpha_{n,k}=\big{[}\frac{2k}{\sqrt{n}}\big{(}\frac{1+2k}{\sqrt{n}}+1\big{)}\big{]}^{1/2}$
and $\delta=\delta_{n,k}=\alpha\log(m^{k}/\alpha)$.

Before giving the proof of the theorem, some remarks are in order:

  1. 1.

It can be seen from (9) that, if $k$ stays bounded as $n\to\infty$, then:

$\epsilon(n,k)=O(\delta_{n,k})=O\Bigl{(}\Bigl{(}\frac{k}{\sqrt{n}}\Bigr{)}^{1/2}\log{\frac{n}{k}}\Bigr{)}\to
0.$

Moreover, in order for $\epsilon(n,k)$ to vanish, $k$ can grow at most
logarithmically with $n$. This is, at least asymptotically, weaker than the
bound (8) given in gavalakis:21  for the binary case $m=2$. What’s more, the
proof of (9) given below is longer and more involved that the corresponding
proof of (8) in gavalakis:21 . So why bother? The reason is that the proof
given here follows a completely different information-theoretic path than that
in gavalakis:21 , and that path consists of an appealing sequence of steps
making interesting connections. So we first present a heuristic outline, and
then give the actual proof. In fact, as will be seen from the proof
(especially Lemma 1), it is easy to improve the bound $\epsilon(n,k)$, but our
purpose here is to illustrate the ideas rather than to obtain optimal results.

  2. 2.

We have cheated slightly in the statement of the theorem, in that the proof
below is only given for the case when $n$ is a multiple of $k$. However, this
is only a minor technical inconvenience; for example, we can replace $n$ with
an integer multiple of $k$ which is no less than $n-k$, leading to the same
bound with $\epsilon(n-k,k)$ in place of $\epsilon(n,k)$.

  3. 3.

Fe Finetti’s original theorem (7) easily follows from (9) by an application of
Pinsker’s inequality (2) and a standard weak convergence argument.

Heuristic proof of de Finetti’s theorem (7).

Step 1: Since the sequence $\\{X_{n}\\}$ is exchangeable it is also
stationary, therefore, by the ergodic theorem $\hat{P}_{X_{1}^{n}}$ converges
as $n\to\infty$ a.s. to a (random) $P$ on $A$. Let $\bar{\mu}$ denote the law
of $P$, and let $\\{Y_{n}\\}$ be i.i.d. random variables uniformly distributed
on $A$. Then, by exchangeability, we clearly have for any $n$, any $k\leq n$,
any $n$-type $Q_{n}$, and any $a_{1}^{k}\in A^{k}$,

$\displaystyle\mathbb{P}(X_{1}^{k}=a_{1}^{k}|\hat{P}_{X_{1}^{n}}=Q_{n})=\mathbb{P}(Y_{1}^{k}=a_{1}^{k}|\hat{P}_{Y_{1}^{n}}=Q_{n}).$
(10)

Step 2: Choose and fix any one of the almost all realisations $\\{Q_{n}\\}$
along which $\hat{P}_{X_{1}^{n}}$ converges to some $Q$ as $n\to\infty$. By
(10) and symmetry we have,

$\mathbb{P}(X_{1}=a|\hat{P}_{X_{1}^{n}}=Q_{n})={\mathbb{E}}\Big{(}{\mathbb{I}}_{\\{Y_{1}=a\\}}\Big{|}\hat{P}_{Y_{1}^{n}}=Q_{n}\Big{)}={\mathbb{E}}\Big{(}\frac{1}{n}\sum_{i=1}^{n}{\mathbb{I}}_{\\{Y_{i}=a\\}}\Big{|}\hat{P}_{Y_{1}^{n}}=Q_{n}\Big{)},$

so that,

$\mathbb{P}(X_{1}=a|\hat{P}_{X_{1}^{n}}=Q_{n})={\mathbb{E}}\Big{(}\hat{P}_{Y_{1}^{n}}(a)\Big{|}\hat{P}_{Y_{1}^{n}}=Q_{n}\Big{)}\\\
=Q_{n}(a),$

for any $a\in A$, and letting $n\to\infty$ yields,

$\displaystyle\lim_{n\to\infty}\mathbb{P}(X_{1}=a|\hat{P}_{Y_{1}^{n}}=Q_{n})=Q(a).$
(11)

Step 3: Next we generalise (11) to blocks of random variables. As before,
choose and fix any one of the almost all realisations $\\{Q_{n}\\}$ of the
random $\hat{P}_{X_{1}^{n}}$ such that $Q_{n}\to$ some $Q$ as $n\to\infty$.
Define a new sequence of i.i.d. random variables $Z_{n}=(Y_{2n-1},Y_{2n})$,
$n\geq 1,$ so that each $Z_{n}$ is uniformly distributed on $A\times A$. From
(10), taking $k=2$ and an arbitrary even $n=2\ell$,

$\displaystyle\mathbb{P}\big{(}(X_{1},X_{2})=(a_{1},a_{2})\big{|}\hat{P}_{X_{1}^{2\ell}}=Q_{2\ell}\big{)}=\mathbb{P}\big{(}Z_{1}=(a_{1},a_{2})\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E(Q_{2\ell})\big{)},$ (12)

where $E(Q)$ denotes the set of probability distributions $W$ on $A\times A$
with the property that the average of the two marginals $W_{1}$ and $W_{2}$ of
$W$ equals $Q$,

$\displaystyle E(Q)=\Big{\\{}W\;\mbox{on}\;A\times
A\,:\,\frac{W_{1}+W_{2}}{2}=Q\Big{\\}}.$

If we write $U$ for the uniform distribution on $A\times A$, it is easy to
check that the distribution $W_{\ell}^{*}$ that uniquely achieves the
$\min_{W\in E(Q)}D(W\|U)$ is simply $Q\times Q$. At this point, we would wish
to apply the conditional limit theorem cover:book2  to the i.i.d. process
$\\{Z_{n}\\}$, to obtain that,

$\displaystyle\lim_{\ell\to\infty}\mathbb{P}\big{(}Z_{1}=(a_{1},a_{2})\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E(Q_{2\ell})\big{)}$ $\displaystyle=$
$\displaystyle\lim_{\ell\to\infty}W_{\ell}^{*}(a_{1},a_{2})$ $\displaystyle=$
$\displaystyle\lim_{\ell\to\infty}Q_{2\ell}(a_{1})Q_{2\ell}(a_{2})$
$\displaystyle=$ $\displaystyle Q(a_{1})Q(a_{2}),$

and combining this with (12) would yield:

$\mathbb{P}\big{(}(X_{1},X_{2})=(a_{1},a_{2})\big{|}\hat{P}_{X_{1}^{2\ell}}=Q_{2\ell}\big{)}\to
Q(a_{1})Q(a_{2}),\;\;\;\ell\to\infty.$

The same argument can be used without difficulty to show that for any $k\geq
1$ and any $a_{1}^{k}\in A^{k}$,

$\displaystyle\lim_{\ell\to\infty}\mathbb{P}(X_{1}^{k}=a_{1}^{k}|\hat{P}_{X_{1}^{k\ell}}=Q_{k\ell})=Q^{k}(a_{1}^{k}).$
(13)

Step 4: Since (13) holds for almost every sequence $\\{Q_{n}\\}$, letting
$\ell\to\infty$, by the bounded converge theorem we have,

$\mathbb{P}(X_{1}^{k}=a_{1}^{k})={\mathbb{E}}\Big{(}\mathbb{P}(X_{1}^{k}=a_{1}^{k}|\hat{P}_{X_{1}^{k\ell}})\Big{)}\to\int_{\cal
P}Q^{k}(a_{1}^{k})d\bar{\mu}(Q),$

as required. $\Box$

The only problem with the above argument is that the set $E(Q)$ has an empty
interior so that the conditional limit theorem is not directly applicable.
Nevertheless, in the next section where we take a finite-$n$ approach, we are
able to ‘imitate’ the proof of the conditional limit theorem and replace the
step where the non-empty interior assumption is used with a different
argument.

###  2.1 Proof

Recall the notation and terminology for types described in the Introduction.
Let $\mu=\mu_{n}$ denote the law of $\hat{P}_{X_{1}^{n}}$ on ${\cal P}$, and
let $\\{Y_{n}\\}$ be i.i.d. random variables uniformly distributed on $A$. For
any $k\leq n$, any $n$-type $Q_{n}$, and any $a_{1}^{k}\in A^{k}$,

$\displaystyle\mathbb{P}(X_{1}^{k}=a_{1}^{k}|\hat{P}_{X_{1}^{n}}=Q_{n})=\mathbb{P}(Y_{1}^{k}=a_{1}^{k}|\hat{P}_{Y_{1}^{n}}=Q_{n}).$

Foe $k=1$ and any $a\in A$, by symmetry we have,

$\displaystyle\mathbb{P}(X_{1}=a|\hat{P}_{X_{1}^{n}})\;=\;\hat{P}_{X_{1}^{n}}(a),$

and taking the expectation of both sides with respect to $\mu$ shows that in
fact $P_{1}=M_{\mu,1}$.

For general $1\leq k\leq n$ with $n=k\ell$, for any $n$-type $Q$ we have,

$\displaystyle\mathbb{P}(X^{k}_{1}=a_{1}^{k}|\hat{P}_{X_{1}^{k\ell}}=Q)$
$\displaystyle=$
$\displaystyle\mathbb{P}(Z_{1}=a_{1}^{k}|\hat{P}_{Z_{1}^{\ell}}\in E_{k}(Q))$
$\displaystyle=$
$\displaystyle{\mathbb{E}}\big{(}\hat{P}_{Z_{1}^{\ell}}(a_{1}^{k})\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\big{)},$

where now $\\{Z_{n}\\}$ is a sequence of i.i.d. random variables uniformly
distributed on $A^{k}$, and $E_{k}(Q)$ consists of all probability
distributions $W$ on $A^{k}$ with the property that the average of the $k$
one-dimensional marginals of $W$ equals $Q$. Taking expectations with respect
to $\hat{P}_{X_{1}^{k\ell}}=\hat{P}_{X_{1}^{n}}\sim\mu=\mu_{n}$,

$\displaystyle\mathbb{P}(X_{1}^{k}=a_{1}^{k})$ $\displaystyle=$
$\displaystyle\int{\mathbb{E}}\big{(}\hat{P}_{Z_{1}^{\ell}}(a_{1}^{k})\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\big{)}d\mu(Q),$

and by the joint convexity of relative entropy,

$\displaystyle D(P_{k}\|M_{\mu,k})$ $\displaystyle=$ $\displaystyle
D\Big{(}\int{\mathbb{E}}\big{(}\hat{P}_{Z_{1}^{\ell}}\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\big{)}d\mu(Q)\Big{\|}\int Q^{k}\;d\mu(Q)\Big{)}$ (14)
$\displaystyle\leq$ $\displaystyle\int
D\Big{(}{\mathbb{E}}\big{(}\hat{P}_{Z_{1}^{\ell}}\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\big{)}\Big{\|}Q^{k}\Big{)}d\mu(Q)$ $\displaystyle\leq$
$\displaystyle\int{\mathbb{E}}\Big{(}D(\hat{P}_{Z_{1}^{\ell}}\|Q^{k})\,\Big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\Big{)}d\mu(Q).$

We will obtain an explicit bound for the relative entropy in (14). First, we
construct a joint $\ell$-type $W$ with desirable properties. Let ${\cal
P}_{\ell}$ denote the set of $\ell$-types on $A^{k}$.

###### Lemma 1

For any $\ell>k\geq 1$ and any $n$-type $Q$, there is a $W\in
E_{k}(Q)\cap\mathcal{P}_{\ell}$ with:

$\max_{a_{1}^{k}}|W(a_{1}^{k})-Q^{k}(a_{1}^{k})|\leq
M:=\Bigg{[}\frac{2}{\ell}+\frac{4k}{\ell}+2\sqrt{\frac{k}{\ell}}\Bigg{]}^{1/2}.$

Moreover, for $2\leq k\leq\sqrt{\ell}/10,$

$|H(W)-H(Q^{k})|\leq-M\log\frac{M}{m^{k}}.$

###### Proof

Let $x_{1}^{k\ell}\in A^{k\ell}$ have type $Q$, let $V_{1}^{k\ell}$ be a
random permutation of $x_{1}^{k\ell}$, and let $\widehat{W}$ denote its
(random) $\ell$-type. Obviously we have that $\widehat{W}\in E_{k}(Q)$ by
construction, and we will also show that $\widehat{W}$ satisfies the statement
of the lemma with positive probability. Taking any $k\leq\ell$ and $\gamma>0$
arbitrary,

$\displaystyle\mathbb{P}\Big{(}\max_{a_{1}^{k}}$
$\displaystyle\,|\widehat{W}(a_{1}^{k})-Q^{k}(a_{1}^{k})|>\gamma\Big{)}$
$\displaystyle\leq\sum_{a_{1}^{k}}\mathbb{P}\Big{(}|\widehat{W}(a_{1}^{k})-Q^{k}(a_{1}^{k})|>\gamma\Big{)}$
$\displaystyle\leq\sum_{a_{1}^{k}}\gamma^{-2}\mathbb{E}\Big{[}\Big{(}\widehat{W}(a_{1}^{k})-Q^{k}(a_{1}^{k})\Big{)}^{2}\Big{]}$
$\displaystyle=\gamma^{-2}\sum_{a_{1}^{k}}\Big{[}\rho_{2}(a_{1}^{k})-2Q^{k}(a_{1}^{k})\rho_{1}(a_{1}^{k})+Q^{k}(a_{1}^{k})^{2}\Big{]},$
(15)

where $\rho_{2}(a_{1}^{k})=\mathbb{E}\big{[}\widehat{W}(a_{1}^{k})^{2}\big{]}$
and $\rho_{1}(a_{1}^{k})=\mathbb{E}\big{[}\widehat{W}(a_{1}^{k})\big{]}$.

Now we find appropriate bounds so that the above probability is $<1$. To get
an upper bound on $\rho_{1}(a_{1}^{k})$ for some fixed $a_{1}^{k}$ note that,

$\rho_{1}(a_{1}^{k})=\mathbb{P}(V_{1}^{k}=a_{1}^{k})\leq\prod_{i=1}^{k}\frac{n(a_{i})}{\ell
k-i+1},$

where $n(a_{i})$ is the number of appearances of $a_{i}$ in $x_{1}^{k\ell}$,
and hence,

$\displaystyle\rho_{1}(a_{1}^{k})$ $\displaystyle\leq$
$\displaystyle\prod_{i=1}^{k}\frac{n(a_{i})}{\ell k}\frac{\ell k}{\ell k-i+1}$
$\displaystyle\leq$ $\displaystyle Q^{k}(a_{1}^{k})\Big{(}\frac{\ell k}{\ell
k-k}\Big{)}^{k}$ $\displaystyle\leq$ $\displaystyle
Q^{k}(a_{1}^{k})\Big{(}1-\frac{1}{\ell}\Big{)}^{-k}$ $\displaystyle\leq$
$\displaystyle Q^{k}(a_{1}^{k})(1+k/\ell),$

since $(1-x)^{-k}\leq 1+kx$ for $x\in[0,1)$. Similarly, writing
$a_{1}^{k}*a_{1}^{k}$ for the concatenation of $a_{1}^{k}$ with itself, we can
estimate,

$\mathbb{P}(V_{1}^{2k}=a_{1}^{k}*a_{1}^{k})\leq
Q^{k}(a_{1}^{k})^{2}(1+4k/\ell),$

so that,

$\displaystyle\rho_{2}(a_{1}^{k})$ $\displaystyle=$
$\displaystyle\ell\frac{1}{\ell^{2}}\rho_{1}(a_{1}^{k})+\ell(\ell-1)\frac{1}{\ell^{2}}\mathbb{P}(V_{1}^{2k}=a_{1}^{k}*a_{1}^{k})$
(16) $\displaystyle\leq$
$\displaystyle(1/\ell)(1+k/\ell)Q^{k}(a_{1}^{k})+(1+4k/\ell)Q^{k}(a_{1}^{k})^{2}$
$\displaystyle\leq$
$\displaystyle(2/\ell)Q^{k}(a_{1}^{k})+(1+4k/\ell)Q^{k}(a_{1}^{k})^{2}.$

Substituting the bound (16) in (15) we have,

$\displaystyle\mathbb{P}\Big{(}\max_{a_{1}^{k}}|W(a_{1}^{k})-Q^{k}(a_{1}^{k})|>\gamma\Big{)}$
$\displaystyle\leq\gamma^{-2}\sum_{a_{1}^{k}}Q^{k}(a_{1}^{k})\Big{[}\frac{2}{\ell}+\Big{(}2+\frac{4k}{\ell}\Big{)}Q^{k}(a_{1}^{k})-2\rho_{1}(a_{1}^{k})\Big{]}$
$\displaystyle\leq\gamma^{-2}\max_{a_{1}^{k}}\Big{[}\frac{2}{\ell}+\Big{(}2+\frac{4k}{\ell}\Big{)}Q^{k}(a_{1}^{k})-2\rho_{1}(a_{1}^{k})\Big{]}.$
(17)

Finally, we get a lower bound on $\rho_{1}(a_{1}^{k})$. In the case where for
some $\beta>0$ (to be chosen later), $Q(a_{i})>\beta$ for all $a_{i}$, we
have,

$\rho_{1}(a_{1}^{k})\geq\prod_{i=1}^{k}\frac{n(a_{i})-i+1}{\ell
k-i+1}\geq\prod_{i=1}^{k}\frac{n(a_{i})-k+1}{\ell k}\geq
Q^{k}(a_{1}^{k})\prod_{i=1}^{k}\Big{(}1-\frac{1}{\ell Q(a_{i})}\Big{)},$

so that,

$\displaystyle\rho_{1}(a_{1}^{k})>Q^{k}(a_{1}^{k})\Big{(}1-\frac{1}{\ell\beta}\Big{)}^{k}\geq
Q^{k}(a_{1}^{k})\Big{(}1-\frac{k}{\ell\beta}\Big{)},$ (18)

assuming $\beta\geq 1/2\ell,$ since $(1-x)^{k}\geq 1-kx$ for all $k\geq 1$ and
all $x\in[0,2]$. For all such $a_{1}^{k}$, using (18) we can bound the
expression in the maximum in (17) by

$\Big{[}\frac{2}{\ell}+\Big{(}2+\frac{4k}{\ell}\Big{)}Q^{k}(a_{1}^{k})-2\rho_{1}(a_{1}^{k})\Big{]}<\frac{2}{\ell}+\frac{4k}{\ell}+\frac{2k}{\ell\beta}.$

And in the case when at least one $a_{i}$ has $Q(a_{i})\leq\beta$, simply
omitting the negative term and noting that $Q^{k}(a_{1}^{k})\leq\beta$ we
bound the same term above by,

$\Big{[}\frac{2}{\ell}+\Big{(}2+\frac{4k}{\ell}\Big{)}Q^{k}(a_{1}^{k})\Big{]}\leq\frac{2}{\ell}+\Big{(}2+\frac{4k}{\ell}\Big{)}\beta.$

Combining the last three bounds,

$\mathbb{P}\Big{(}\max_{a_{1}^{k}}|W(a_{1}^{k})-Q^{k}(a_{1}^{k})|>\gamma\Big{)}\leq\gamma^{-2}\Big{[}\frac{2}{\ell}+\max\Big{\\{}\frac{2k}{\ell}\big{(}2+\frac{1}{\beta}\big{)},\;\big{(}2+\frac{4k}{\ell}\big{)}\beta\Big{\\}}\Big{]},$

where the inequality is strict when the first term dominates the maximum. To
obtain a good bound we take for $\beta$ a value approximately equal to the
minimiser of the above expression: We set $\beta^{*}=\sqrt{k/\ell}$. Note that
for this $\beta^{*}$ it can be easily verified that the first term strictly
dominates the maximum, giving,

$\mathbb{P}\Big{(}\max_{a_{1}^{k}}|W(a_{1}^{k})-Q^{k}(a_{1}^{k})|>\gamma\Big{)}<\gamma^{-2}\Big{[}\frac{2}{\ell}+\frac{4k}{\ell}+2\sqrt{\frac{k}{\ell}}\Big{]},$

and taking $\gamma=M$ as in the lemma, completes the proof of the first
statement.

For the second part, noting that for $2\leq k\leq\sqrt{\ell}/10$ we have
$M<1/2$, the result follows from (csiszar:book, , Lemma 2.7).

Next we obtain an upper bound on the conditional expectation in (14).

###### Lemma 2

Suppose $n=\ell k$, with $2\leq k\leq\sqrt{\ell}/10$. For any $n$-type $Q$ we
have:

$\mathbb{E}\Big{(}D(\hat{P}_{Z_{1}^{\ell}}\|Q^{k})\,\Big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\Big{)}\leq\epsilon(n,k).$

###### Proof

We follow the same steps as in the proof of the conditional limit theorem in
cover:book2 . Recall that if we write $U_{k}$ for the uniform distribution on
$A^{k}$, then the $W_{k}^{*}$ that uniquely achieves $D^{*}=\min_{W\in
E_{k}(Q)}D(W\|U_{k})$ is $W_{k}^{*}=Q^{k}$. We partition $E_{k}(Q)$ into
$B_{2\delta}$ and $C=E_{k}(Q)-B_{2\delta}$, where $B_{2\delta}=\\{W\in
E_{k}(Q)\,:\,D(W\|U_{k})\leq D^{*}+2\delta\\}$, with $\delta=\delta_{n,k}$.
Then, writing $\nu_{\ell}$ for the distribution of $\hat{P}_{Z_{1}^{\ell}}$,

$\displaystyle\nu_{\ell}(C|E_{k}(Q))=\frac{\nu_{\ell}(C\cap
E_{k}(Q))}{\nu_{\ell}(E_{k}(Q))}\leq\frac{\nu_{\ell}(C)}{\nu_{\ell}(B_{2\delta})}.$

Next we bound the above numerator and denominator. For the numerator, writing
again ${\cal P}_{\ell}$ for the set of $\ell$-types on $A^{k}$,

$\displaystyle\nu_{\ell}(C)$ $\;\stackrel{{\scriptstyle(a)}}{{=}}\;$
$\displaystyle\sum_{W\in C\cap\mathcal{P}_{\ell}}U_{k}^{\ell}(T(W))$
$\;\stackrel{{\scriptstyle(b)}}{{\leq}}\;$ $\displaystyle\sum_{W\in
C\cap\mathcal{P}_{\ell}}e^{-\ell D(W\|U_{k})}$
$\;\stackrel{{\scriptstyle(c)}}{{\leq}}\;$
$\displaystyle|E_{k}(Q)\cap\mathcal{P}_{\ell}|e^{-\ell(D^{*}+2\delta)}$
$\;\stackrel{{\scriptstyle(d)}}{{\leq}}\;$
$\displaystyle(\ell+1)^{m^{k}}e^{-\ell(D^{*}+2\delta)}$

where $T(W)$ in $(a)$ denotes the type class of all strings of length $\ell$
in $A^{k}$ with type $W$, $(b)$ is a standard property cover:book2 , $(c)$
follows from the definition of $C$ and the fact that
$E_{k}(Q)\cap\mathcal{P}_{\ell}\subset E_{k}(Q)$, and $(d)$ follows from the
standard observation that
$|E_{k}(Q)\cap\mathcal{P}_{\ell}|\leq|\mathcal{P}_{\ell}|\leq(\ell+1)^{m^{k}}$.
Similarly, letting $W_{0}$ denote the type from Lemma 1,

$\displaystyle\nu_{\ell}(B_{2\delta})$ $\displaystyle\geq$
$\displaystyle\nu_{\ell}(B_{\delta})$ $\displaystyle=$
$\displaystyle\sum_{W\in B_{\delta}\cap\mathcal{P}_{\ell}}U_{k}^{\ell}(T(W))$
$\displaystyle\geq$ $\displaystyle U_{k}^{\ell}(T(W_{0}))$ $\displaystyle\geq$
$\displaystyle(\ell+1)^{-m^{k}}e^{-\ell D(W_{0}\|U)}$ $\displaystyle\geq$
$\displaystyle(\ell+1)^{-m^{k}}e^{-\ell(D^{*}+\delta)}.$

Combining these bounds, we obtain,

$\mathbb{P}\big{(}\hat{P}_{Z_{1}^{\ell}}\in C\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\big{)}\leq(\ell+1)^{2m^{k}}e^{-\ell\delta},$

or,

$\displaystyle\mathbb{P}\Bigl{(}D(\hat{P}_{Z_{1}^{\ell}}\|U_{k})>D^{*}+2\delta\Big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\Bigr{)}\leq(\ell+1)^{2m^{k}}e^{-\ell\delta}.$

Since the set $E_{k}(Q)$ is closed and convex, we may apply the Pythagorean
identity for relative entropy cover:book2  to conclude that:

$\displaystyle\mathbb{P}\Bigl{(}D(\hat{P}_{Z_{1}^{\ell}}\|Q^{k})>2\delta|\hat{P}_{Z_{1}^{\ell}}\in
E_{k}(Q)\Bigr{)}\leq(\ell+1)^{2m^{k}}e^{-\ell\delta}.$

Thus,

$\mathrm{e}{D(\hat{P}_{Z_{1}^{\ell}}\|Q^{k})\big{|}\hat{P}_{Z_{1}^{\ell}}\in
E_{\ell}(Q)}\leq(\ell+1)^{2m^{k}}e^{-\ell\delta}\max_{P\in
E_{k}(Q)}{D(P\|Q^{k})}+2\delta.$

The claimed bound now follows by Lemma 3 on taking $\ell=k/n$.

###### Lemma 3

For any $n$-type $Q$, $\max_{W\in E_{k}(Q)}D(W\|Q^{k})\leq k\log{n}$.

###### Proof

If $a_{1}^{k}\in A^{k}$ is such that
$Q^{k}(a_{1}^{k})=\prod_{i=1}^{k}{Q(a_{i})}=0$, then $Q(a_{i_{0}})=0$ for some
$i_{0}$. Since $Q(a_{i_{0}})=\frac{1}{k}\sum_{j=1}^{k}{W_{j}(a_{i_{0}})}$, we
must have $W_{1}(a_{i_{0}})=\cdots=W_{k}(a_{i_{0}})=0$, which implies that
$W(a_{1}^{k})=0.$

On the other hand, if $Q^{k}(a_{1}^{k})>0$ then
$Q^{k}(a_{1}^{k})\geq\frac{1}{n^{k}}$. Thus, for any $W\in E_{k}(Q)$,

$\displaystyle D(W\|Q^{k})$ $\displaystyle=$ $\displaystyle\sum_{a_{1}^{k}\in
A^{k}}W(a_{1}^{k})\log\frac{W(a_{1}^{k})}{Q^{k}(a_{1}^{k})}$
$\displaystyle\leq$ $\displaystyle\sum_{a_{1}^{k}\in
A^{k}}W(a_{1}^{k})\log\frac{W(a_{1}^{k})}{({\frac{1}{n}})^{k}}$
$\displaystyle=$ $\displaystyle k\log n-H(W)$ $\displaystyle\leq$
$\displaystyle k\log n,$

as required.

Theorem 2.1 follows from (14) combined with Lemma 2.

###### Acknowledgements.

We wish to thank Andrew Barron for pointing out several useful references for
the historical development outlined in our Introduction.

## References

  * [1] P.H. Algoet and B.H. Marcus.  Large deviation theorems for empirical types of Markov chains constrained to thin sets.  IEEE Trans. Inform. Theory, 38(4):1276–1291, July 1992.
  * [2] S. Artstein, K. Ball, F. Barthe, and A. Naor.  Solution of Shannon’s problem on the monotonicity of entropy.  J. Amer. Math. Soc., 17(4):975–982, 2004.
  * [3] D. Bakry and M. Émery.  Diffusions hypercontractives.  In Seminaire de probabilités XIX 1983/84, pages 177–206. Springer, 1985.
  * [4] A.D. Barbour, O. Johnson, I. Kontoyiannis, and M. Madiman.  Compound Poisson approximation via information functionals.  Electron. J. Probab, 15:1344–1369, 2010.
  * [5] A.R. Barron.  Entropy and the central limit theorem.  Ann. Probab., 14(1):336–342, January 1986.
  * [6] A.R. Barron.  Information theory in probability, statistics, learning, and neural nets.  In Y. Freund and R.E. Schapire, editors, Proceedings of the Tenth Annual Conference on Computational Learning Theory (COLT), Nashville, Tennessee, July 1997.  Available at: http://www.stat.yale.edu/$\sim$arb4/publications_files/COLT97.pdf.
  * [7] A.R. Barron.  Limits of information, Markov chains, and projection.  In 2000 IEEE International Symposium on Information Theory (ISIT), Sorrento, Italy, June 2000.
  * [8] M. Bayarri and J.O. Berger.  The interplay of Bayesian and frequentist analysis.  Statistical Science, 19(1):58–80, 2004.
  * [9] N.M. Blachman.  The convolution inequality for entropy powers.  IEEE Trans. Inform. Theory, 11(2):267–271, April 1965.
  * [10] S.G. Bobkov and F. Götze.  Exponential integrability and transportation cost related to logarithmic Sobolev inequalities.  J. Funct. Anal., 163(1):1–28, 1999.
  * [11] S.G. Bobkov and M. Ledoux.  On modified logarithmic Sobolev inequalities for Bernoulli and Poisson measures.  J. Funct. Anal., 156(2):347–365, 1998.
  * [12] S. Boucheron, G. Lugosi, and P. Massart.  Concentration inequalities: A nonasymptotic theory of independence.  Oxford University Press, Oxford, U.K., 2013.
  * [13] L.D. Brown.  A proof of the central limit theorem motivated by the Cramér-Rao inequality.  In Statistics and Probability: Essays in Honor of C.R. Rao, pages 141–148. North-Holland, Amsterdam, 1982.
  * [14] L.D. Brown.  Fundamentals of statistical exponential families: With applications in statistical decision theory, volume 9 of IMS Lecture Notes Monograph Series.  Institute of Mathematical Statistics, Hayward, CA, 1986.
  * [15] S. Bubeck and S. Ganguly.  Entropic CLT and phase transition in high-dimensional Wishart matrices.  International Mathematics Research Notices, 2018(2):588–606, 2018\.
  * [16] E.A Carlen and D. Cordero-Erausquin.  Subadditivity of the entropy and its relation to Brascamp-Lieb type inequalities.  Geometric & Functional Analysis, 19(2):373–405, 2009.
  * [17] E.A. Carlen and A. Soffer.  Entropy production by block variable summation and central limit theorems.  Comm. Math. Phys., 140(2):339–371, 1991.
  * [18] M.H.M. Costa and T.M. Cover.  On the similarity of the entropy power inequality and the Brunn-Minkowski inequality.  IEEE Trans. Inform. Theory, 30(6):837–839, November 1984.
  * [19] T.M. Cover and J.A. Thomas.  Elements of information theory.  J. Wiley & Sons, New York, second edition, 2012.
  * [20] I. Csiszár.  Eine informationstheoretische ungleichung und ihre anwendung auf den beweis der ergodizität von Markoffschen ketten.  Publ. Math. Inst. Hungar. Acad. Sci., 8:85–108, 1963.
  * [21] I. Csiszár.  Information-type measures of difference of probability distributions and indirect observations.  Studia Sci. Math. Hungar., 2:299–318, 1967.
  * [22] I. Csiszár.  ${I}$-divergence geometry of probability distributions and minimization problems.  Ann. Probab., 3(1):146–158, February 1975.
  * [23] I. Csiszár.  Sanov property, generalized ${I}$-projection and a conditional limit theorem.  Ann. Probab., 12(3):768–793, August 1984.
  * [24] I Csiszár.  Information theoretic methods in probability and statistics.  In 1997 IEEE International Symposium on Information Theory (ISIT), Ulm, Germany, June 1997.
  * [25] I. Csiszár.  The method of types.  IEEE Trans. Inform. Theory, 44(6):2505–2523, October 1998.  Information theory: 1948–1998.
  * [26] I. Csiszár.  A simple proof of Sanov’s theorem.  Bulletin of the Brazilian Mathematical Society, 37(4), 2006.
  * [27] I. Csiszár, T.M. Cover, and B.S. Choi.  Conditional limit theorems under Markov conditioning.  IEEE Trans. Inform. Theory, 33(6):788–801, November 1987.
  * [28] I. Csiszár and J. Körner.  Information theory: Coding theorems for discrete memoryless systems.  Academic Press, New York, 1981.
  * [29] E.B. Davies and B. Simon.  Ultracontractivity and the heat kernel for Schrödinger operators and Dirichlet Laplacians.  J. Funct. Anal., 59(2):335–395, 1984.
  * [30] B. De Finetti.  Sul significato soggettivo della probabilita.  Fundamenta Mathematicae, 17(1):298–329, 1931.
  * [31] B. De Finetti.  La prévision: ses lois logiques, ses sources subjectives.  Ann. Inst. Henri Poincaré, 7(1):1–68, 1937.
  * [32] A. Dembo, T.M. Cover, and J.A. Thomas.  Information-theoretic inequalities.  IEEE Trans. Inform. Theory, 37(6):1501–1518, November 1991.
  * [33] P. Diaconis.  Finite forms of de Finetti’s theorem on exchangeability.  Synthese, 36(2):271–281, 1977.
  * [34] P. Diaconis and D.A. Freedman.  Finite exchangeable sequences.  Ann. Probab., 8(4):745–764, 1980.
  * [35] J. Fritz.  An information-theoretical proof of limit theorems for reversible Markov processes.  In Transactions of the Sixth Prague Conference on Information Theory, Statistical Decision Functions, Random Processes (Tech. Univ., Prague, 1971; dedicated to the memory of Antonín Špaček), pages 183–197. Academia, Prague, 1973.
  * [36] L. Gavalakis and I. Kontoyiannis.  Entropy and the discrete central limit theorem.  arXiv e-prints, 2106.00514 [math.PR], June 2021.
  * [37] L. Gavalakis and I. Kontoyiannis.  An information-theoretic proof of a finite de Finetti theorem.  Electron. Comm. Probab., 26:1–5, 2021.
  * [38] P. Groeneboom, J. Oosterhoff, and F.H. Ruymgaart.  Large deviation theorems for empirical probability measures.  Ann. Probab., 7(4):553–586, August 1979.
  * [39] L. Gross.  Logarithmic Sobolev inequalities.  Amer. J. Math., 97(4):1061–1083, 1975.
  * [40] D. Guo, S. Shamai, and S. Verdú.  Mutual information and minimum mean-square error in Gaussian channels.  IEEE Trans. Inform. Theory, 51(4):1261–1282, April 2005.
  * [41] J. Hájek.  On a property of normal distributions of any stochastic process.  Czechoslovak Math. J., 8(83):610–618, 1958.
  * [42] J. Hájek.  A property of J-divergences of marginal probability distributions.  Czechoslovak Math. J., 8(3):460–463, 1958.
  * [43] P. Harremoës.  Binomial and Poisson distributions as maximum entropy distributions.  IEEE Trans. Inform. Theory, 47(5):2039–2041, July 2001.
  * [44] P. Harremoës and K.K. Holst.  Convergence of Markov chains in information divergence.  J. Theoret. Probab., 22(1):186–202, March 2009.
  * [45] P. Harremoës, O. Johnson, and I. Kontoyiannis.  Thinning and the law of small numbers.  In 2007 IEEE International Symposium on Information Theory (ISIT), pages 1491–1495, Nice, France, June 2007.
  * [46] P. Harremoës, O. Johnson, and I. Kontoyiannis.  Thinning, entropy, and the law of thin numbers.  IEEE Trans. Inform. Theory, 56(9):4228–4244, 2010.
  * [47] E. Hewitt and L.J. Savage.  Symmetric measures on Cartesian products.  Trans. Amer. Math. Soc., 80(2):470–501, 1955.
  * [48] O. Johnson.  Entropy inequalities and the central limit theorem.  Stoch. Proc. Appl., 88(2):291–304, 2000.
  * [49] O. Johnson.  Information theory and the central limit theorem.  Imperial College Press, London, U.K., 2004.
  * [50] O. Johnson.  Log-concavity and the maximum entropy property of the Poisson distribution.  Stoch. Proc. Appl., 117(6):791–802, June 2007.
  * [51] O. Johnson and A.R. Barron.  Fisher information inequalities and the central limit theorem.  Probab. Theory Related Fields, 129(3):391–409, 2004.
  * [52] O. Johnson, I. Kontoyiannis, and M. Madiman.  Log-concavity, ultra-log-concavity, and a maximum entropy property of discrete compound Poisson measures.  Discrete Applied Mathematics, 161(9):1232–1250, 2013.
  * [53] O. Johnson and Yu.M. Suhov.  Entropy and convergence on compact groups.  J. Theoret. Probab., 13(3):843–857, 2000.
  * [54] J.H.B. Kemperman.  On the optimum rate of transmitting information.  In Probability and information theory, pages 126–169. Springer-Verlag, 1969.
  * [55] D.G. Kendall.  Information theory and the limit-theorem for Markov chains and processes with a countable infinity of states.  Ann. Inst. Statist. Math., 15(1):137–143, May 1963.
  * [56] A.N. Kolmogorov.  A new metric invariant of transitive dynamical systems and Lebesgue space automorphisms.  In Dokl. Acad. Sci. USSR, volume 119, pages 861–864, 1958.
  * [57] I. Kontoyiannis, P. Harremoës, and O. Johnson.  Entropy and the law of small numbers.  IEEE Trans. Inform. Theory, 51(2):466–472, February 2005.
  * [58] I. Kontoyiannis and M. Madiman.  Sumset and inverse sumset inequalities for differential entropy and mutual information.  IEEE Trans. Inform. Theory, 60(8):4503–4514, August 2014.
  * [59] S. Kullback.  A lower bound for discrimination information in terms of variation.  IEEE Trans. Inform. Theory, 13(1):126–127, January 1967.
  * [60] M. Ledoux.  Isoperimetry and Gaussian analysis.  In Lectures on probability theory and statistics, volume 1648 of Lecture Notes in Mathematics, pages 165–294. Springer, Berlin, 1996\.
  * [61] M. Ledoux.  On Talagrand’s deviation inequalities for product measures.  ESAIM Probab. Statist., 1:63–87 (electronic), 1997.
  * [62] M. Ledoux.  The concentration of measure phenomenon.  American Mathematical Society, Providence, RI, 2001.
  * [63] Ju.V. Linnik.  An information-theoretic proof of the central limit theorem with Lindeberg conditions.  Theory Probab. Appl., 4:288–299, 1959.
  * [64] J. Liu, T.A. Courtade, P. Cuff, and S. Verdú.  Brascamp-Lieb inequality and its reverse: An information theoretic view.  In 2016 IEEE International Symposium on Information Theory (ISIT), pages 1048–1052, Barcelona, Spain, July 2016.
  * [65] M. Madiman and A.R Barron.  Generalized entropy power inequalities and monotonicity properties of information.  IEEE Trans. Inform. Theory, 53(7):2317–2329, July 2007.
  * [66] M. Madiman, O. Johnson, and I. Kontoyiannis.  Fisher information, compound Poisson approximation, and the Poisson channel.  In 2007 IEEE International Symposium on Information Theory (ISIT), pages 976–980, Nice, France, June 2007.
  * [67] M. Madiman and I. Kontoyiannis.  Entropy bounds on abelian groups and the Ruzsa divergence.  IEEE Trans. Inform. Theory, 64(1):77–92, January 2016.
  * [68] K. Marton.  Bounding $\overline{d}$-distance by informational divergence: A method to prove measure concentration.  Ann. Probab., 24(2):857–866, April 1996.
  * [69] N. O’Connell.  Information-theoretic proof of the Hewitt-Savage zero-one law.  Technical report, Hewlett-Packard Laboratories, Bristol, U.K., June 2000\.
  * [70] D.S. Ornstein.  Bernoulli shifts with the same entropy are isomorphic.  Advances in Mathematics, 4(3):337–352, 1970.
  * [71] D.S. Ornstein.  Imbedding Bernoulli shifts in flows.  In Contributions to ergodic theory and probability, pages 178–218. Springer, 1970.
  * [72] D.S. Ornstein.  Two Bernoulli shifts with infinite entropy are isomorphic.  Advances in Mathematics, 5(3):339–348, 1970.
  * [73] M. Raginsky and I. Sason.  Concentration of Measure Inequalities in Information Theory, Communications and Coding.  Foundations and Trends in Communications and Information Theory. NOW Publishers, Boston, MA, 2018.
  * [74] A. Rényi.  On measures of entropy and information.  In Proc. 4th Berkeley Sympos. Math. Statist. and Prob., Vol. I, pages 547–561. Univ. California Press, Berkeley, Calif., 1961.
  * [75] I.N. Sanov.  On the probability of large deviations of random variables.  Mat. Sb., 42:11–44, 1957.  English translation in Sel. Transl. Math. Statist. Probab. (1961) 1, 213-244.
  * [76] C.E. Shannon.  A mathematical theory of communication.  Bell System Tech. J., 27(3):379–423, 623–656, 1948.
  * [77] C.E Shannon.  The bandwagon.  IRE Trans. Inform. Theory, 2(1):3, March 1956.
  * [78] R. Shimizu.  On Fisher’s amount of information for location family.  In G.P. Patil, S. Kotz, and J.K. Ord, editors, A Modern Course on Statistical Distributions in Scientific Work, pages 305–312. Springer, Dordrecth, Netherlands, 1975.
  * [79] A.J. Stam.  Some inequalities satisfied by the quantities of information of Fisher and Shannon.  Information and Control, 2(2):101–112, 1959.
  * [80] T. Tao.  Sumset and inverse sumset theory for Shannon entropy.  Combinatorics, Probability and Computing, 19(4):603–639, July 2010\.
  * [81] F. Topsøe.  Information-theoretical optimization techniques.  Kybernetika, 15(1):8–27, 1979.
  * [82] A.M. Tulino and S. Verdú.  Monotonic decrease of the non-Gaussianness of the sum of independent random variables: A simple proof.  IEEE Trans. Inform. Theory, 52(9):4295–4297, September 2006.
  * [83] C. Villani.  Optimal transport: Old and new.  Springer, Berlin, 2009.
  * [84] D. Voiculescu.  The analogues of entropy and of Fisher’s information measure in free probability theory, I.  Comm. Math. Phys., 155(1):71–92, 1993.
  * [85] D. Voiculescu.  The analogues of entropy and of Fisher’s information measure in free probability theory, II.  Inventiones Mathematicae, 118(1):411–440, 1994.
  * [86] D. Voiculescu.  The analogues of entropy and of Fisher’s information measure in free probability theory III: The absence of Cartan subalgebras.  Geometric & Functional Analysis, 6(1):172–199, 1996.
  * [87] D. Voiculescu.  The analogues of entropy and of Fisher’s information measure in free probability theory, IV: Maximum entropy and freeness, in free probability theory.  Fields Inst. Commun., 12:293–302, 1997.
  * [88] J. Wolfowitz.  Coding theorems of information theory.  Springer, Berlin, 1961.
  * [89] Y. Yu.  On the entropy of compound distributions on nonnegative integers.  IEEE Trans. Inform. Theory, 55(8):3645–3650, August 2009.


# Two pseudo-Goldstone bosons in the Gildener-Weinberg model

Parsa Ghorbani  Physics Department, Faculty of Science, Ferdowsi University of
Mashhad, Iran

###### Abstract

In a dimensionless multi-scalar extension of the Standard Model, Gildener and
Weinberg assumed that along a flat direction, there is only one classically
massless scalar, known as the scalon, which acquires mass through radiative
corrections à la Coleman-Weinberg, while all other scalars remain heavy. In
this paper, by introducing a toy model with four scalar degrees of freedom, we
demonstrate the existence of two scalons along a specific flat direction that
we construct. We present the effective potential for the model and provide the
masses of the heavy scalars and two radiatively light pseudo-Goldstone bosons.

Gildener and Weinberg (GW) [1] came up with a generic dimensionless multi-
scalar model as an extension to the Standard Model Higgs potential for which
the theory undergoes a dynamical electroweak symmetry breaking through the
radiative corrections à la Coleman-Weinberg [2].

The approach of GW is that a quartic multi-scalar potential
$V=\lambda_{ijkl}\phi_{i}\phi_{j}\phi_{k}\phi_{l}$ can be written as
$V=\phi^{4}\lambda_{ijkl}N_{i}N_{j}N_{k}N_{l}$ where the scalar fields
$\phi_{i}$ are replaced by $\phi_{i}=\phi N_{i}$ with $\phi$ and $N_{i}$ being
respectively the radial field and any arbitrary direction in the field
configuration space. Then at a given mass scale $\Lambda$, a flat direction
${\bf N=n}$ for which ${\bf n}.{\bf n}=1$, is found such that the potential
and its minimum vanish; i.e. $\lambda_{ijkl}n_{i}n_{j}n_{k}n_{l}=0$ and
$\lambda_{ijkl}n_{j}n_{k}n_{l}=0$. According to Coleman-Weinberg mechanism,
loop quantum corrections to the effective potential lead to the generation of
non-zero vacuum expectation value (VEV) for the radial field $\phi$ giving VEV
to all scalars through $\braket{\phi_{i}}=\braket{\phi}n_{i}$. Whether or not
a scalar field in the model takes a VEV, depends on the choice of the flat
direction. This non-zero $\braket{\phi}$ breaks symmetries of the underlying
theory. Gildener and Weinberg assume, without providing any proof, that at the
tree level there is only one massless scalar (regardless of the number of
scalars). They term this scalar the ”scalon” which gains a slight mass due to
radiative corrections. All other scalars remain massive.

In this paper we want to question the uniqueness of the scalon in scale
invariant multi-scalar potentials. We will argue that in the GW approach, it
is not necessary to be restricted to only one radial field or flat direction.
In fact, instead of having a single flat direction $n_{i}$ with
$n_{i}n_{i}=1$, we can identify two flat directions, $n_{j}$ and $l_{k}$, such
that $n_{j}n_{j}+l_{k}l_{k}=1$. Essentially, this implies that rather than
replacing $\phi_{i}$ by $\phi N_{i}$ for all scalar fields in the potential,
we opt for two radial fields, say $\rho$ and $\eta$. For a subset of scalars,
we have $\phi_{j}=\rho N_{j}$, and for the remaining ones, $\phi_{k}=\eta
L_{k}$, where $N_{j}$ and $L_{k}$ represent two arbitrary directions in the
field space. Having made this adjustment in terms of the radial fields $\rho$
and $\eta$, the potential takes the general form as follows

$V=\rho^{4}\lambda^{(1)}_{ijkl}N_{i}N_{j}N_{k}N_{l}+\eta^{4}\lambda^{(2)}_{ijkl}L_{i}L_{j}L_{k}L_{l}+\rho^{2}\eta^{2}\lambda^{(3)}_{ijkl}N_{i}N_{j}L_{k}L_{l}\,.$
(1)

Note that the original GW conditions must still be satisfied; that is along
the flat directions one must have

$V_{\text{tr}}({\bf N,L})\bigg{|}_{\tiny\begin{matrix}{\bf N=n}\\\ {\bf
L=l}\end{matrix}}=\frac{\partial V_{\text{tr}}({\bf N,L})}{\partial
N_{i}}\bigg{|}_{\tiny\begin{matrix}{\bf N=n}\\\ {\bf
L=l}\end{matrix}}=\frac{\partial V_{\text{tr}}({\bf N,L})}{\partial
L_{i}}\bigg{|}_{\tiny\begin{matrix}{\bf N=n}\\\ {\bf L=l}\end{matrix}}=0.$ (2)

To employ the GW approach with one scalon, let’s examine a straightforward
$\mathbb{Z}_{2}$ symmetric potential featuring two real scalars. The tree-
level potential is given by

$V(h_{1},h_{2})=\frac{1}{4}\lambda_{1}h_{1}^{4}+\frac{1}{2}\lambda_{12}h_{1}^{2}h_{2}^{2}+\frac{1}{4}\lambda_{2}h_{2}^{4}\,.$
(3)

This structure could represent a scale-invariant real singlet scalar extension
to the Higgs potential

$V(H_{1},h_{2})=\lambda_{1}\left(H_{1}^{\dagger}H_{1}\right)^{2}+\lambda_{12}\left(H_{1}^{\dagger}H_{1}\right)h_{2}^{2}+\frac{1}{4}\lambda_{2}h_{2}^{4}\,,$
(4)

where $H_{1}^{\dagger}=(0,h_{1})/\sqrt{2}$ denotes the Higgs doublet in the
unitary gauge. However, our intention is not to complicate the model with
internal symmetries that are broken after the fields acquire VEV. Instead, we
focus solely on the potential presented in Eq. (3) involving two singlet
scalars. The flat direction $(n_{1},n_{2})$ is determined by the conditions

$\lambda_{12}^{2}=\lambda_{1}\lambda_{2},~{}~{}~{}\frac{\braket{h_{1}}^{2}}{\braket{h_{2}}^{2}}\equiv\frac{n_{1}^{2}}{n_{2}^{2}}=-\frac{\lambda_{12}}{\lambda_{1}},~{}~{}~{}\lambda_{12}<0.$
(5)

with $n_{1}^{2}+n_{2}^{2}=1$. This is equivalent to having

$\displaystyle\lambda_{1}=-\lambda_{12}\frac{\braket{h_{2}}^{2}}{\braket{h_{1}}^{2}}\,,$
(6a)
$\displaystyle\lambda_{2}=-\lambda_{12}\frac{\braket{h_{1}}^{2}}{\braket{h_{2}}^{2}}\,.$
(6b)

with $\lambda_{12}<0$. Note that the condition $\lambda_{12}<0$ is needed,
along with $\lambda_{1}>0$ and $\lambda_{2}>0$, for the potential to be
bounded from below along the flat direction. The potential in terms of a
single radial field is

$V_{\text{tr}}({\bf
N})=\rho^{4}\left(\frac{1}{4}\lambda_{1}N_{1}^{4}+\frac{1}{4}\lambda_{2}N_{2}^{4}+\frac{1}{2}\lambda_{12}N_{1}^{2}N_{2}^{2}\right).$
(7)

Along the flat direction given by Eq. (5) or Eq. (6), the tree-level potential
in Eq. (7) satisfies the conditions

$V_{\text{tr}}({\bf N})\bigg{|}_{{\bf N=n}}=\frac{\partial V_{\text{tr}}({\bf
N})}{\partial N_{i}}\bigg{|}_{{\bf N=n}}=0\,.$ (8)

The eigenvalues of the mass matrix are

$m_{1}^{2}=0,~{}~{}~{}m_{2}^{2}=-\lambda_{12}\left(\braket{h_{1}}^{2}+\braket{h_{2}}^{2}\right)\,.$
(9)

As anticipated by the GW scheme, there exists one massless scalon at the
classical level, while the other state is massive. The corresponding effective
potential, using dimensional regularization in the $\overline{\text{MS}}$
scheme for the one-loop contribution, is given by [3, 4],

$V_{\text{eff}}({\bf n})\equiv V_{\text{tr}}({\bf n})+V_{\text{1-loop}}({\bf
n})=B({\bf
n})\rho^{4}\left(\log\frac{\rho^{2}}{\braket{\rho}^{2}}-\frac{1}{2}\right),$
(10)

with

$B({\bf n})=\frac{m_{2}^{4}}{64\pi^{2}\braket{\rho}^{4}}.$ (11)

This leads to radiative correction to the scalon mass as

$\delta m_{1}^{2}=\frac{m_{2}^{4}}{8\pi^{2}\braket{\rho}^{2}},$ (12)

where $m_{2}^{2}$ is known from Eq. (9). Identifying the flat direction with
one scalon is simple for a potential with two real scalars, but it becomes
increasingly intricate as the number of scalars grows [5]. A pertinent
question is whether we can discover two flat directions corresponding to two
scalons, giving rise to two pseudo-Goldstone bosons due to quantum
corrections. To address this question, we examine a toy model featuring four
real singlet scalar degrees of freedom,

$\begin{split}V_{\text{tr}}=&\lambda_{0}h_{1}h_{2}h_{3}h_{4}+\frac{1}{4}\lambda_{1}h_{1}^{4}+\frac{1}{4}\lambda_{2}h_{2}^{4}+\frac{1}{4}\lambda_{3}h_{3}^{4}+\frac{1}{4}\lambda_{4}h_{4}^{4}\\\
&+\frac{1}{4}\lambda_{12}h_{1}^{2}h_{2}^{2}+\frac{1}{4}\lambda_{13}h_{1}^{2}h_{3}^{2}+\frac{1}{4}\lambda_{14}h_{1}^{2}h_{4}^{2}+\frac{1}{4}\lambda_{23}h_{2}^{2}h_{3}^{2}+\frac{1}{4}\lambda_{24}h_{2}^{2}h_{4}^{2}+\frac{1}{4}\lambda_{34}h_{3}^{2}h_{4}^{2},\end{split}$
(13)

with the discrete $\mathbb{Z}_{2}\times\mathbb{Z}_{2}$ symmetry

$h_{1}\to-h_{1},~{}~{}~{}h_{2}\to h_{2},~{}~{}~{}h_{3}\to-
h_{3},~{}~{}~{}h_{4}\to h_{4}.$ (14)

Note the presence of an ad-hoc quad-linear term in the potential. The linear
and cubic terms are absent because of the $\mathbb{Z}_{2}$ symmetry. Once
more, it’s essential to clarify that our focus here does not extend to the
internal symmetries of the underlying theory. However, should one desire to
align this model with a framework beyond the Standard Model, consider, for
instance, a potential comprising two Higgs doublets and two real singlet
scalars (For scale-invariant scenarios of 2HDM, refer to [6, 7, 8])

$\begin{split}V_{\text{tr}}=&\lambda_{0}(H_{1}^{\dagger}H_{2}+H_{2}^{\dagger}H_{1})h_{3}h_{4}+\lambda_{1}(H_{1}^{\dagger}H_{1})^{2}+\lambda_{2}(H_{2}^{\dagger}H_{2})^{2}+\frac{1}{4}\lambda_{3}h_{3}^{4}+\frac{1}{4}\lambda_{4}h_{4}^{4}\\\
&+\lambda_{12}(H_{1}^{\dagger}H_{1})(H_{2}^{\dagger}H_{2})+\frac{1}{2}\lambda_{13}(H_{1}^{\dagger}H_{1})h_{3}^{2}+\frac{1}{2}\lambda_{14}(H_{1}^{\dagger}H_{1})h_{4}^{2}\\\
&+\frac{1}{2}\lambda_{23}(H_{2}^{\dagger}H_{2})h_{3}^{2}+\frac{1}{2}\lambda_{24}(H_{2}^{\dagger}H_{2})h_{4}^{2}+\frac{1}{4}\lambda_{34}h_{3}^{2}h_{4}^{2},\end{split}$
(15)

where all the couplings are real. In this context, where $H_{1}$ and $H_{2}$
represent the doublets, and $h_{3}$ and $h_{4}$ denote the singlets, the
discrete symmetry outlined in Eq. (14) manifests as a mechanism to prevent
Flavor-Changing Neutral Currents (FCNCs).

If we were to follow the GW prescription in search of the flat direction
$(n_{1},n_{2},n_{3},n_{4})$ that satisfies the conditions in Eq. (8), the
solution would be extremely challenging, if not unattainable. Instead,
following the approach outlined above, at a given scale $\Lambda$ we define
two flat directions $(n_{1},n_{2})$ and $(l_{1},l_{2})$ that satisfy the
conditions in Eq. (2). The result is the conditions

Here, $\braket{h_{i}}$ is defined in terms of the radial fields $\rho$ or
$\eta$ as follows

$\displaystyle\braket{h_{1}}$ $\displaystyle=\braket{\rho}n_{1},$ (17)
$\displaystyle\braket{h_{2}}$ $\displaystyle=\braket{\rho}n_{2},$
$\displaystyle\braket{h_{3}}$ $\displaystyle=\braket{\eta}l_{1},$
$\displaystyle\braket{h_{4}}$ $\displaystyle=\braket{\eta}l_{2}.$

The tree-level potential, expressed in terms of the radial fields along the
arbitrary directions ${\bf N}$ and ${\bf L}$ as explained above, is given by

$\begin{split}V_{\text{tr}}=&\rho^{4}\left(\frac{1}{4}\lambda_{1}N_{1}^{4}+\frac{1}{4}\lambda_{2}N_{2}^{4}+\frac{1}{4}\lambda_{12}N_{1}^{2}N_{2}^{2}\right)\\\
&+\eta^{4}\left(\frac{1}{4}\lambda_{3}L_{1}^{4}+\frac{1}{4}\lambda_{4}L_{2}^{4}+\frac{1}{4}\lambda_{34}L_{1}^{2}L_{2}^{2}\right)\\\
&+\rho^{2}\eta^{2}\left(\lambda_{0}N_{1}N_{2}L_{1}L_{2}+\frac{1}{4}\lambda_{13}N_{1}^{2}N_{1}^{2}+\frac{1}{4}\lambda_{14}N_{1}^{2}L_{2}^{2}+\frac{1}{4}\lambda_{23}N_{2}^{2}L_{1}^{2}+\frac{1}{4}\lambda_{24}N_{2}^{2}L_{2}^{2}\right).\end{split}$
(18)

The solutions for the flat directions ${\bf n}$ and ${\bf l}$ that satisfy Eq.
(16), which itself meets the conditions of Eq. (2) for the tree-level
potential in Eq. (18) are

$n_{1}^{2}=\frac{\lambda_{12}}{\lambda_{12}-2\lambda_{1}},~{}n_{2}^{2}=\frac{2\lambda_{1}}{2\lambda_{1}-\lambda_{12}},~{}l_{1}^{2}=\frac{\lambda_{34}}{\lambda_{34}-2\lambda_{3}},~{}l_{2}^{2}=\frac{2\lambda_{3}}{2\lambda_{3}-\lambda_{34}}.$
(19)

Considering the aforementioned flatness conditions, the requirement for the
potential to be bounded from below is simplified to
$\lambda_{0}<0,~{}\lambda_{12}<0$ and $\lambda_{34}<0$. Alternatively, the
flatness condition can be expressed as

$\begin{split}\lambda_{12}^{2}=4\lambda_{1}\lambda_{2},\\\
\lambda_{34}^{2}=4\lambda_{3}\lambda_{4},\\\
\lambda_{13}\lambda_{24}=\lambda_{14}\lambda_{23}.\end{split}$ (20)

It can be shown that the eigenvalues of the mass matrix are

$\begin{split}&m_{1}^{2}=0,\\\ &m_{2}^{2}=0,\\\
&m_{3}^{2}=-\left(\braket{h_{1}}^{2}+\braket{h_{2}}^{2}\right)\left(\frac{\braket{h_{3}}\braket{h_{4}}}{\braket{h_{1}}\braket{h_{2}}}\lambda_{0}+\lambda_{12}\right),\\\
&m_{4}^{2}=-\left(\braket{h_{3}}^{2}+\braket{h_{4}}^{2}\right)\left(\frac{\braket{h_{1}}\braket{h_{2}}}{\braket{h_{3}}\braket{h_{4}}}\lambda_{0}+\lambda_{34}\right).\end{split}$
(21)

As we asserted earlier, it is clear that there exist two classically massless
states along the flat directions, while two other scalars are heavy. Radiative
corrections should give small mass corrections to two scalons.

In general, the radial fields acquire non-zero VEVs, $\braket{\rho}$ and
$\braket{\phi}$, introducing two distinct scales into the theory. The
calculation of the effective potential, therefore, differs from the GW
approach, which involves only one radial field, and consequently, only one
scale. When dealing with the renormalization of a theory with different
scales, caution is necessary concerning perturbativity, as these scales may be
significantly separated. The intricacies of multi-scale renormalization were
elucidated by Einhorn and Jones in [9] (refer also to [10] for multi-scale
renormalization involving two fields). To streamline the computations, we
assume that the two scales in our model consistently remain within the
perturbativity regime. Therefore, we can express the one-loop effective
potential in terms of only one scale

$V_{\text{eff}}^{\text{1-loop}}(\rho{\bf n},\eta{\bf l})=A({\bf
n})\rho^{4}+B({\bf n})\rho^{4}\log\frac{\rho^{2}}{\Lambda^{2}}+A^{\prime}({\bf
l})\eta^{4}+B^{\prime}({\bf l})\eta^{4}\log\frac{\eta^{2}}{\Lambda^{2}}\,.$
(22)

where the coefficients are given by

$\begin{split}&A({\bf
n})=\frac{1}{64\pi^{2}\braket{\rho^{4}}}\sum_{i=3,4}m_{i}^{4}\left(-\frac{3}{2}+\log\frac{m_{i}^{2}}{\braket{\rho}^{2}}\right)\\\
&B({\bf n})=\frac{1}{64\pi^{2}\braket{\rho^{4}}}\sum_{i=3,4}m_{i}^{4}\\\
&A^{\prime}({\bf
l})=\frac{1}{64\pi^{2}\braket{\eta^{4}}}\sum_{i=3,4}m_{i}^{4}\left(-\frac{3}{2}+\log\frac{m_{i}^{2}}{\braket{\eta}^{2}}\right)\\\
&B^{\prime}({\bf
l})=\frac{1}{64\pi^{2}\braket{\eta^{4}}}\sum_{i=3,4}m_{i}^{4}\\\ \end{split}$
(23)

The one-loop effective potential needs to fulfill the tadpole conditions along
two directions,

$\frac{\partial
V_{\text{eff}}^{\text{1-loop}}}{\partial\rho}\bigg{|}_{\rho=\braket{\rho}}=\frac{\partial
V_{\text{eff}}^{\text{1-loop}}}{\partial\eta}\bigg{|}_{\eta=\braket{\eta}}=0\,.$
(24)

These conditions determine the scale $\Lambda$ as

$\Lambda=\braket{\rho}\exp\left(\frac{A}{2B}+\frac{1}{4}\right)=\braket{\eta}\exp\left(\frac{A^{\prime}}{2B^{\prime}}+\frac{1}{4}\right),$
(25)

leading to the following effective potential

$V_{\text{eff}}^{\text{1-loop}}=B({\bf
n})\rho^{4}\left(\log\frac{\rho^{2}}{\braket{\rho}^{2}}-\frac{1}{2}\right)+B^{\prime}({\bf
l})\eta^{4}\left(\log\frac{\eta^{2}}{\braket{\eta}^{2}}-\frac{1}{2}\right).$
(26)

Following the procedures outlined in [3, 4], one can determine the masses of
the two scalons resulting from the aforementioned effective potential,

$\delta
m_{1}^{2}=\frac{1}{8\pi^{2}\braket{\rho}^{2}}\sum_{i=3,4}m_{i}^{4},~{}~{}~{}~{}~{}\delta
m_{2}^{2}=\frac{1}{8\pi^{2}\braket{\eta}^{2}}\sum_{i=3,4}m_{i}^{4},$ (27)

where $m_{3}^{2}$ and $m_{4}^{2}$ are given by Eq. (21).

Therefore, we have shown that having at least four scalar fields in the
theory, it becomes feasible to construct two flat directions, paving the way
for two classically massless scalons. However, incorporating more than two
scalons in a renormalizable theory within the framework of the general GW
potential is not possible.

## References

  * [1] Eldad Gildener and Steven Weinberg.  Symmetry Breaking and Scalar Bosons.  Phys. Rev. D, 13:3333, 1976.
  * [2] Sidney R. Coleman and Erick J. Weinberg.  Radiative Corrections as the Origin of Spontaneous Symmetry Breaking.  Phys. Rev. D, 7:1888–1910, 1973.
  * [3] Lisa Alexander-Nunneley and Apostolos Pilaftsis.  The Minimal Scale Invariant Extension of the Standard Model.  JHEP, 09:021, 2010.
  * [4] Karim Ghorbani and Hossein Ghorbani.  Scalar Dark Matter in Scale Invariant Standard Model.  JHEP, 04:024, 2016.
  * [5] Kristjan Kannike, Aleksei Kubarski, and Luca Marzola.  Geometry of Flat Directions in Scale-Invariant Potentials.  Phys. Rev. D, 99(11):115034, 2019.
  * [6] Jae Sik Lee and Apostolos Pilaftsis.  Radiative Corrections to Scalar Masses and Mixing in a Scale Invariant Two Higgs Doublet Model.  Phys. Rev. D, 86:035004, 2012.
  * [7] Alexis D. Plascencia.  Classical scale invariance in the inert doublet model.  JHEP, 09:026, 2015.
  * [8] Alexandros Karam and Kyriakos Tamvakis.  Dark matter and neutrino masses from a scale-invariant multi-Higgs portal.  Phys. Rev. D, 92(7):075010, 2015.
  * [9] Martin B. Einhorn and D. R. Timothy Jones.  A NEW RENORMALIZATION GROUP APPROACH TO MULTISCALE PROBLEMS.  Nucl. Phys. B, 230:261–272, 1984.
  * [10] C. Ford and C. Wiesendanger.  Multiscale renormalization.  Phys. Lett. B, 398:342–346, 1997.