# Lower Bounds on the Bayesian Risk
via Information Measures

Amedeo Roberto Esposito, Adrien Vandenbroucque, Michael Gastpar Institute of
Science and Technology Austria<EMAIL_ADDRESS>Ecole
Polytechnique Fédérale de Lausanne<EMAIL_ADDRESS>Ecole
Polytechnique Fédérale de Lausanne<EMAIL_ADDRESS>
###### Abstract

This paper focuses on parameter estimation and introduces a new method for
lower bounding the Bayesian risk. The method allows for the use of virtually
_any_ information measure, including Rényi’s $\alpha$, $\varphi$-Divergences,
and Sibson’s $\alpha$-Mutual Information. The approach considers divergences
as functionals of measures and exploits the duality between spaces of measures
and spaces of functions. In particular, we show that one can lower bound the
risk with any information measure by upper bounding its dual via Markov’s
inequality. We are thus able to provide estimator-independent impossibility
results thanks to the Data-Processing Inequalities that divergences satisfy.
The results are then applied to settings of interest involving both discrete
and continuous parameters, including the “Hide-and-Seek” problem, and compared
to the state-of-the-art techniques. An important observation is that the
behaviour of the lower bound in the number of samples is influenced by the
choice of the information measure. We leverage this by introducing a new
divergence inspired by the “Hockey-Stick” Divergence, which is demonstrated
empirically to provide the largest lower-bound across all considered settings.
If the observations are subject to privatisation, stronger impossibility
results can be obtained via Strong Data-Processing Inequalities. The paper
also discusses some generalisations and alternative directions.

###### Contents

  1. 1 Introduction
    1. 1.1 Overview of the document
    2. 1.2 Related Work
    3. 1.3 Problem Setting
  2. 2 Preliminaries
    1. 2.1 Information Measures
      1. 2.1.1 Rényi’s $\alpha$-Divergence
      2. 2.1.2 Maximal Leakage
      3. 2.1.3 $\varphi$-Mutual Information
    2. 2.2 (Strong) Data-Processing Inequalities
    3. 2.3 Variational Repesentations and Functional Inequalities
  3. 3 Main Results: Lower Bounds on the Risk
    1. 3.1 Leveraging SDPIs
    2. 3.2 Conditioning
  4. 4 Examples of application
    1. 4.1 Bernoulli Bias
    2. 4.2 Noisy Bernoulli Bias
    3. 4.3 Gaussian prior with Gaussian noise (and absolute error)
    4. 4.4 “Hide-and-seek” problem
  5. A Comparison with similar approaches
  6. B Proof of Section 3
    1. B.1 Proof of Theorem 3
    2. B.2 Proof of Theorem 4
    3. B.3 Proof of Corollary 3
    4. B.4 Proof of Corollary 4
  7. C Computations for Section 4
    1. C.1 Maximisation over $\rho$
    2. C.2  Section 4.1
      1. C.2.1 Maximal Leakage
      2. C.2.2 Sibson’s $\alpha$-Mutual Information
      3. C.2.3 Hellinger $p$
    3. C.3  Section 4.3
      1. C.3.1 Hellinger $p$
  8. D Other approaches
    1. D.1 Inverting the roles
    2. D.2 Lower-Bounding the Risk Directly

##  1 Introduction

In this work,111. This article was presented in part at the 2021 and 2022 IEEE
International Symposia on Information Theory we consider the problem of
parameter estimation in a Bayesian setting. More precisely, we propose an
approach to _lower-bounding_ the Bayesian risk leveraging most information
measures present in the literature. We look at the problem through an
information-theoretic lens, similarly to [1]. We thus treat the parameter to
be estimated as a message sent through a channel. This allows us to include
frameworks where, in a distributed fashion, $m$ processors observe noisy
samples of this parameter. The processors will then send a version of their
observations to a central node. The central node will then proceed to estimate
the parameter. We thus shift the focus from the estimation problem to the
computation of two main quantities (which we render as independent of the
estimator as possible):

  1. 1.

an information measure (e.g., Sibsons’s $\alpha$-Mutual Information,
$\varphi$-Mutual Information, etc.);

  2. 2.

a functional of the probability of some event under independence (e.g., a
small-ball probability [1]);

The main tools utilised rely on Legendre-Fenchel duality and they allow us to
introduce bounds involving Rényi’s, $\varphi$-Divergences and Sibson’s
$\alpha$-Mutual Information. An advantage of using this type of bounds is that
one can render the functional in Item 2 (e.g., the small-ball probability)
independent of the specific estimator. Similarly, the information measure can
also be rendered independent of the estimator via Data-Processing
Inequalities. Therefore, these lower bounds can be applied to any standard
estimation framework regardless of the specific choice of the estimator. More
details on the formal framework that we adopted can be found in Section 1.3.

It is important to notice that, although the problem can be interpreted as a
transmission problem, a fundamental difference is that the size of the
quantised messages may not grow with the number of samples. This might render
the reconstruction of the samples impossible but the estimation of the
parameter may remain feasible [1]. Our main focus will not be on asymptotic
results but rather on finite number of samples lower bounds.

###  1.1 Overview of the document

Following the Introduction, the paper will be broken in four main sections:

  * •

Section 2: Preliminaries, in which we will define the information measures of
interest as well as describe the theoretical framework leveraged to provide
the bounds;

  * •

Section 3: Main Bounds, in which (making use of the famework described in
Section 2) we propose a variety of lower bounds on the Bayesian Risk involving
a variety of information measures, in particular:

    * –

Sibson’s $\alpha$-Mutual Information and Maximal Leakage (Theorem 3);

    * –

$\varphi$-Mutual Information (Theorem 4). In particular:

      * *

Hellinger $p$-Divergence (Corollary 3);

      * *

Rényi’s $\alpha$-Divergence (Remark 6);

      * *

a generalisation of the “Hockey-Stick” Divergence, $E_{\gamma}$ (Corollary 4);

  * •

Section 4: Examples, in which we apply the bounds proposed in Section 3 to a
variety of classical and less classical settings:

    * –

estimation of the bias of a Bernoulli random variable (see Section 4.1);

    * –

estimation of the bias of a Bernoulli random variable after injection of
additional noise (e.g., observing privatised samples, see Section 4.2);

    * –

estimation of the mean of a Gaussian random variable (with Gaussian prior, see
Section 4.3);

    * –

lower-bound on the minimax risk for the “Hide-and-Seek” problem [2] (see
Section 4.4).

For each of the problems we derive bounds involving a variety of information
measures and we compare said bounds among themselves and with respect to
relevant bounds in the literature as well;

  * •

throughout the document we also consider further generalisations, in which we
propose a variety of ways of extending/tightening/altering the results we
proposed in Section 3. In particular, one can provide new bounds:

    * –

conditioning on an additional (and cleverly constructed) random variable (see
Section 3.2);

    * –

leveraging the asymmetry of some information measures (see Section D.1);

    * –

ignoring the probabilities and lower-bounding the risk directly (see Section
D.2).

###  1.2 Related Work

The problem of parameter estimation has been extensively studied over the
years, with many contributions coming from a variety of fields. Relevant
literature, mostly leveraging the Van Trees Inequality (and the quadratic
risk) can be found in [3, 4, 5, 6, 7]. Moreover, a survey of early work in
this area (mainly focusing on asymptotic settings) can be found in [8]. More
recent but important advances are instead due to [9, 10, 2]. Closely connected
to this work is [1]. The approach is quite similar, with the main difference
that we employ a family of bounds involving a variety of divergences while [1]
relies solely on Mutual Information and the Kullback-Leibler Divergence.
Related is also [11], where the authors use the so-called
$E_{\gamma}$-Divergence to provide a lower-bound on the Bayesian Risk. A
similar approach was also undertaken in [12]. The authors focused on the
notion of $\varphi$-informativity (cf. [13]) and leveraged the Data-Processing
inequality similarly to [14, Theorem 3]. A more thorough discussion of the
differences between this work and [12] can be found in Appendix A.

###  1.3 Problem Setting

Let $\mathcal{W}$ denote the parameter space and assume that we have access to
a prior distribution over $\mathcal{W}$, $\mathcal{P}_{W}$. Suppose that we
observe $W\sim\mathcal{P}_{W}$ through the family of distributions
$\mathcal{P}=\\{\mathcal{P}_{X|W=w}:w\in\mathcal{W}\\}.$ Given a function
$\phi:\mathcal{X}\to\hat{\mathcal{W}}$, one can then estimate $W$ from
$X\sim\mathcal{P}_{X|W}$ via $\phi(X)=\hat{W}$. Let us denote with
$\ell:\mathcal{W}\times\hat{\mathcal{W}}\to\mathbb{R}^{+}$ a loss function,
the Bayesian risk is defined as:

$R_{B}=\inf_{\phi}\,\mathcal{P}_{W\hat{W}}(\ell(W,\phi(X)))=\inf_{\phi}\,\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W})).$
(1)

Our purpose is to lower-bound $R_{B}$ using a variety of information measures.
With this drive and leveraging various tools that stem from Legendre-Fenchel
duality, one can connect the expected value of $\ell$ under the joint
$\mathcal{P}_{W\hat{W}}$ to

  * •

the expected value of the same function under the product of the marginals
($\mathcal{P}_{W}\mathcal{P}_{\hat{W}}$) or a “small-ball probability” ;

  * •

an information measure (quantifying how “far” the joint is from the product of
the marginals).

This will allow us to render the lower-bound as independent as possible from
the specific choice of the estimator $\phi$. More precisely, our desideratum
will be a lower-bound of the following form:

$R_{B}\geq\upvarpi\left(\frac{d\mathcal{P}_{W\hat{W}}}{d\mathcal{P}_{W}\mathcal{P}_{\hat{W}}}\right)\upvartheta(\mathcal{P}_{W}\mathcal{P}_{\hat{W}},\ell),$
(2)

with, once again, the purpose of then rendering the right-hand side of
Equation 2 as independent as possible of the estimator $\phi$. Let us denote
with
$L_{W}(\hat{W},\rho)=\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\ell(W,\hat{W})<\rho)$,
a functional $\upvartheta$ of particular interest to us is the one that leads
to (a function of) the so-called small-ball probability

$L_{W}(\rho)=\sup_{\hat{w}\in\mathcal{\hat{W}}}L_{W}(\hat{w},\rho)=\sup_{\hat{w}\in\mathcal{\hat{W}}}\mathcal{P}_{W}(\ell(W,\hat{w})<\rho).$
(3)

More generally, the choice of $\upvartheta$ will depend on the choice of
$\upvarpi$ and vice versa. In the following sections, we will explore
different choices of said functionals that lead to interesting results in the
field.

##  2 Preliminaries

In this section, we will define the main objects utilised throughout the
document and define the relevant notation. We will mainly adopt a measure-
theoretic framework. Given a measurable space $(\mathcal{X},\mathcal{F})$ and
two measures $\mu,\nu$ which render it a measure space, if $\nu$ is absolutely
continuous with respect to $\mu$ (denoted with $\nu\ll\mu$) then we will
represent with $\frac{d\nu}{d\mu}$ the Radon-Nikodym derivative of $\nu$ with
respect to $\mu$. Given a (measurable) function $f:\mathcal{X}\to\mathbb{R}$
and a measure $\mu$ we denote with $\mu(f)=\langle f,\mu\rangle=\int fd\mu$
i.e., the Lebesgue integral of $f$ with respect to the measure $\mu$
represents a bilinear inner-product which will characterise a pairing between
a (properly defined) space of functions and a (properly defined) space of
measures. Once such a pairing is set, one can then proceed onto defining the
Legendre-Fenchel transform connecting functionals acting on measures to
functionals acting on functions. More formally, let $C_{b}(\mathcal{X})$
denote the space of continuous and bounded functions defined on $\mathcal{X}$
and $\mathcal{M}(\mathcal{X})$ the set of Radon measures defined on the same
space, then one has that $\mathcal{M}(\mathcal{X})$ and $C_{b}(\mathcal{X})$
are in separating duality through the bilinear mapping
$\langle\cdot,\cdot\rangle$ defined above (see [15]). Thus, given a functional
$\psi:C_{b}(\mathcal{X})\to\mathbb{R}$ one can define its Legendre-Fenchel
dual as follows:

$\psi^{\star}(\mu)=\sup_{f\in C_{b}(\mathcal{X})}\langle
f,\mu\rangle-\psi(f).$ (4)

Another connection between the spaces of interest comes from considering a
norm on a space and the corresponding dual norm on the dual space, i.e., given
a norm acting on $\mathcal{X}$, $\left\lVert\cdot\right\rVert$ and a pairing
between two spaces $(\mathcal{X},\mathcal{Y})$, one can construct a norm on
$\mathcal{Y}$ as follows:

$\left\lVert h\right\rVert_{\star}=\sup_{f:\left\lVert f\right\rVert\leq
1}|\langle h,f\rangle|.$ (5)

For the purpose of this paper, we will essentially interpret the expected
value $\mathcal{P}_{W\hat{W}}(\ell)$ as
$\langle\mathcal{P}_{W\hat{W}},\ell\rangle$. Once this simple observation is
made, these tools will allow us to connect functionals of measure (e.g.,
information measures) to functionals of the loss (e.g., small-ball
probabilities).

###  2.1 Information Measures

We will now continue introducing the information measures that we need in
order to provide the main results.

####  2.1.1 Rényi’s $\alpha$-Divergence

Introduced by Rényi as a generalization of KL-divergence, $\alpha$-divergence
has found many applications ranging from hypothesis testing to guessing and
several other statistical inference problems [16]. Indeed, it has several
useful operational interpretations (e.g., hypothesis testing, and the cut-off
rate in block coding [16, 17]). It can be defined as follows [16].

######  Definition 1.

Let $(\Omega,\mathcal{F},\mathcal{P}),(\Omega,\mathcal{F},\mathcal{Q})$ be two
probability spaces. Let $\alpha>0$ be a positive real number different from
$1$. Consider a measure $\mu$ such that $\mathcal{P}\ll\mu$ and
$\mathcal{Q}\ll\mu$ (such a measure always exists, e.g.
$\mu=(\mathcal{P}+\mathcal{Q})/2$)) and denote with $p,q$ the densities of
$\mathcal{P},\mathcal{Q}$ with respect to $\mu$. The $\alpha$-Divergence of
$\mathcal{P}$ from $\mathcal{Q}$ is defined as follows:

$\displaystyle D_{\alpha}(\mathcal{P}\|\mathcal{Q})=\frac{1}{\alpha-1}\log\int
p^{\alpha}q^{1-\alpha}d\mu.$ (6)

######  Remark 1.

The definition is independent of the chosen measure $\mu$. It is indeed
possible to show that $\int
p^{\alpha}q^{1-\alpha}d\mu=\int\left(\frac{q}{p}\right)^{1-\alpha}d\mathcal{P}$,
and that whenever $\mathcal{P}\ll\mathcal{Q}$ or $0<\alpha<1,$ we have $\int
p^{\alpha}q^{1-\alpha}d\mu=\int\left(\frac{p}{q}\right)^{\alpha}d\mathcal{Q}$,
see [16].

It can be shown that if $\alpha>1$ and $\mathcal{P}\not\ll\mathcal{Q}$ then
$D_{\alpha}(\mathcal{P}\|\mathcal{Q})=\infty$. The behavior of the measure for
$\alpha\in\\{0,1,\infty\\}$ can be defined by continuity. In general, one has
that $D_{1}(\mathcal{P}\|\mathcal{Q})=D(\mathcal{P}\|\mathcal{Q})$, where
$D(\mathcal{P}\|\mathcal{Q})$ denotes the Kullback-Leibler divergence between
$\mathcal{P}$ and $\mathcal{Q}$. However, if
$D(\mathcal{P}\|\mathcal{Q})=\infty$ or there exists $\beta$ such that
$D_{\beta}(\mathcal{P}\|\mathcal{Q})<\infty$ then $\lim_{\alpha\downarrow
1}D_{\alpha}(\mathcal{P}\|Q)=D(\mathcal{P}\|\mathcal{Q})$ [16, Theorem 5]. For
an extensive treatment of $\alpha$-divergences and their properties we refer
the reader to [16]. Starting from Rényi’s Divergence and the geometric
averaging that it involves, Sibson built the notion of Information Radius
[18]. A deconstructed and generalised version of the Information Radius leads
us to the following definition of a generalisation of Shannon’s Mutual
Information [19]:

######  Definition 2.

Let $X$ and $Y$ be two random variables jointly distributed according to
$\mathcal{P}_{XY}$, and with marginal distributions $\mathcal{P}_{X}$ and
$\mathcal{P}_{Y}$, respectively. For $\alpha>0$, the Sibson’s mutual
information of order $\alpha$ between $X$ and $Y$ is defined as:

$\displaystyle
I_{\alpha}(X,Y)=\min_{\mathcal{Q}_{Y}}D_{\alpha}(\mathcal{P}_{XY}\|\mathcal{P}_{X}\mathcal{Q}_{Y}).$
(7)

The following, alternative formulation is also useful [19]:

$\displaystyle I_{\alpha}(X,Y)$
$\displaystyle=\frac{\alpha}{\alpha-1}\log\mathbb{E}\left[\mathbb{E}^{\frac{1}{\alpha}}\left[\left(\frac{\mathcal{P}_{Y|X}}{\mathcal{P}_{Y}}\right)^{\alpha}\bigg{|}Y\right]\right]$
(8)
$\displaystyle=D_{\alpha}(\mathcal{P}_{XY}\|\mathcal{P}_{X}\mathcal{P}_{Y})-D_{\alpha}(\mathcal{P}_{Y_{\alpha}}\|\mathcal{P}_{Y}),$
(9)

where $\mathcal{P}_{Y_{\alpha}}$ is the measure minimizing (7). In analogy
with the limiting behavior of $\alpha$-Divergence we have that
$\lim_{\alpha\to 1}I_{\alpha}(X,Y)=I(X;Y)$, where $I(X;Y)$ represents the
Mutual Information between $X$ and $Y$. When $\alpha\to\infty$ we retrieve the
following object:

$I_{\infty}(X,Y)=\log\mathbb{E}_{\mathcal{P}_{Y}}\left[\sup_{x:\mathcal{P}_{X}(x)>0}\frac{\mathcal{P}_{XY}(x,Y)}{\mathcal{P}_{X}(x)\mathcal{P}_{Y}(Y)}\right].$

To conclude, let us list some of the properties of the measure:

######  Proposition 1 ([19]).

  1. 1.

Data-Processing Inequality: given $\alpha>0$,
$I_{\alpha}(X,Z)\leq\min\\{I_{\alpha}(X,Y),I_{\alpha}(Y,Z)\\}$ if the Markov
Chain $X-Y-Z$ holds;

  2. 2.

$I_{\alpha}(X,Y)\geq 0$ with equality iff $X$ and $Y$ are independent;

  3. 3.

Let $\alpha_{1}\leq\alpha_{2}$ then $I_{\alpha_{1}}(X,Y)\leq
I_{\alpha_{2}}(X,Y)$;

  4. 4.

Let $\alpha\in(0,1)\cup(1,\infty)$, for a given $\mathcal{P}_{X}$,
$\frac{1}{\alpha-1}\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(X,Y)\right)$ is
convex in $\mathcal{P}_{Y|X}$;

  5. 5.

$I_{\alpha}(X,Y)\leq\min\\{\log|X|,\log|Y|\\}$;

For an extensive treatment of Sibson’s $\alpha$-MI we refer the reader to
[19].

####  2.1.2 Maximal Leakage

A particularly relevant dependence measure, strongly connected to Sibson’s
Mutual Information is the maximal leakage, denoted by
$\mathcal{L}\left(X\\!\\!\to\\!\\!Y\right).$ It was introduced as a way of
measuring the leakage of information from $X$ to $Y$, hence the following
definition:

######  Definition 3 (Def. 1 of [20]).

Given a joint distribution $\mathcal{P}_{XY}$ on finite alphabets
$\mathcal{X}$ and $\mathcal{Y}$, the maximal leakage from $X$ to $Y$ is
defined as:

$\mathcal{L}\left(X\\!\\!\to\\!\\!Y\right)=\sup_{\begin{subarray}{c}U-X-Y-\hat{U}\end{subarray}}\log\frac{\mathbb{P}(\\{U=\hat{U}\\})}{\max_{u\in\mathcal{U}}\mathbb{P}_{U}(\\{u\\})},$
(10)

where $U$ and $\hat{U}$ take values in the same finite, but arbitrary,
alphabet.

It is shown in [20, Theorem 1] that, for finite alphabets:

$\displaystyle\mathcal{L}\left(X\\!\\!\to\\!\\!Y\right)=\log\sum_{y\in\mathcal{Y}}\max_{\begin{subarray}{c}x\in\mathcal{X}\end{subarray}:\mathcal{P}_{X}(x)>0}\mathcal{P}_{Y|X}(y|x).$
(11)

If $X$ and $Y$ have a jointly continuous pdf $f_{XY}(x,y)$, we get [20,
Corollary 4]:

$\displaystyle\mathcal{L}\left(X\\!\\!\to\\!\\!Y\right)=\log\int_{\mathbb{R}}\sup_{x:f_{X}(x)>0}f_{Y|X}(y|x)dy.$
(12)

One can show that $\mathcal{L}\left(X\\!\\!\to\\!\\!Y\right)=I_{\infty}(X;Y)$
i.e., Maximal Leakage corresponds to the Sibson’s Mutual Information of order
infinity. This allows the measure to retain the properties listed in
Proposition 1, furthermore:

######  Lemma 1 ([20]).

For any joint distribution $\mathcal{P}_{XY}$ on finite alphabets
$\mathcal{X}$ and $\mathcal{Y}$,
$\mathcal{L}\left(X\\!\\!\to\\!\\!Y\right)\geq I(X;Y)$.

Another relevant notion is Conditional Maximal Leakage:

######  Definition 4 (Conditional Maximal Leakage [20]).

Given a joint distribution $\mathcal{P}_{XYZ}$ on alphabets
$\mathcal{X},\mathcal{Y},\text{ and }\mathcal{Z}$, define:

$\mathcal{L}\left(X\\!\\!\to\\!\\!Y|Z\right)=\sup_{\begin{subarray}{c}U:U-X-Y|Z\end{subarray}}\log\frac{\mathbb{P}(\\{U=\hat{U}(Y,Z)\\})}{\mathbb{P}(\\{U=\tilde{U}(Z)\\})},$
(13)

where $U$ takes value in an arbitrary finite alphabet and we consider
$\hat{U},\tilde{U}$ to be the optimal estimators of $U$ given $(Y,Z)$ and $Z$,
respectively.

Again, it is shown in [20] that for discrete random variables $X,Y,Z$:

$\mathcal{L}\left(X\\!\\!\to\\!\\!Y|Z\right)=\log\max_{\begin{subarray}{c}z:\mathcal{P}_{Z}(z)>0\end{subarray}}\sum_{y}\max_{\begin{subarray}{c}x:\mathcal{P}_{X|Z}(x|z)\end{subarray}>0}\mathcal{P}_{Y|XZ}(y|xz),$

and

$\mathcal{L}\left(X\\!\\!\to\\!\\!(Y,Z)\right)\leq\mathcal{L}\left(X\\!\\!\to\\!\\!Y\right)+\mathcal{L}\left(X\\!\\!\to\\!\\!Z|Y\right).$
(14)

####  2.1.3 $\varphi$-Mutual Information

Another generalization of the KL-Divergence can be obtained by considering a
generic convex function $\varphi:\mathbb{R}^{+}\to\mathbb{R}$, usually with
the simple constraint that $\varphi(1)=0$. The constraint can be ignored as
long as $\varphi(1)<+\infty$ by simply considering a new mapping
$\tilde{\varphi}(x)=\varphi(x)-\varphi(1)$.

######  Definition 5.

Let $(\Omega,\mathcal{F},\mathcal{P}),(\Omega,\mathcal{F},\mathcal{Q})$ be two
probability spaces. Let $\varphi:\mathbb{R}^{+}\to\mathbb{R}$ be a convex
function. Consider a measure $\mu$ such that $\mathcal{P}\ll\mu$ and
$\mathcal{Q}\ll\mu$. Denoting with $p,q$ the densities of the measures with
respect to $\mu$, the $\varphi$-Divergence of $\mathcal{P}$ from $\mathcal{Q}$
is defined as follows:

$\displaystyle D_{\varphi}(\mathcal{P}\|\mathcal{Q})=\int
q\varphi\left(\frac{p}{q}\right)d\mu.$ (15)

Despite the fact that the definition uses $\mu$ and the densities with respect
to this measure, it is possible to show that $\varphi$-Divergences are
actually independent from the dominating measure [21]. Indeed, when absolute
continuity between $\mathcal{P},\mathcal{Q}$ holds, i.e.
$\mathcal{P}\ll\mathcal{Q}$, an assumption we will often use, we retrieve the
following [21]:

$D_{\varphi}(\mathcal{P}\|\mathcal{Q})=\int\varphi\left(\frac{d\mathcal{P}}{d\mathcal{Q}}\right)d\mathcal{Q}.$
(16)

Denoting with $\mathcal{F}_{X}$ the $\sigma$-field generated from the random
variable $X$, (i.e., $\sigma(X)$), $\varphi$-Mutual Information is defined as
follows:

######  Definition 6.

Let $X$ and $Y$ be two random variables jointly distributed according to
$\mathcal{P}_{XY}$ over the measurable space
$(\mathcal{X}\times\mathcal{Y},\mathcal{F}_{XY})$. Let
$(\mathcal{X},\mathcal{F}_{X},\mathcal{P}_{X}),(\mathcal{Y},\mathcal{F}_{Y},\mathcal{P}_{Y})$
be the corresponding probability spaces induced by the marginals. Let
$\varphi:\mathbb{R}^{+}\to\mathbb{R}$ be a convex function such that
$\varphi(1)=0$. The $\varphi$-Mutual Information between $X$ and $Y$ is
defined as:

$I_{\varphi}(X,Y)=D_{\varphi}(\mathcal{P}_{XY}\|\mathcal{P}_{X}\mathcal{P}_{Y}).$
(17)

If $\mathcal{P}_{XY}\ll\mathcal{P}_{X}\mathcal{P}_{Y}$ we have that:

$I_{\varphi}(X,Y)=\int\varphi\left(\frac{d\mathcal{P}_{XY}}{d\mathcal{P}_{X}\mathcal{P}_{Y}}\right)d\mathcal{P}_{X}\mathcal{P}_{Y}.$
(18)

It is possible to see that, if $\varphi$ satisfies $\varphi(1)=0$ and it is
strictly convex at $1$, then $I_{\varphi}(X,Y)=0$ if and only if $X$ and $Y$
are independent [21]. This generalisation includes the Kullback-Leibler
Divergence (by simply setting $\varphi(t)=t\log(t)$) and allows to retrieve
$\alpha$-Divergences through a one-to-one mapping. Other meaningful examples
are: the Total Variation distance ($\varphi(t)=\frac{1}{2}|t-1|$), the
Hellinger distance ($\varphi(t)=(\sqrt{t}-1)^{2}$), Pearson’s
$\chi^{2}$-divergence ($\varphi(t)=(t-1)^{2}$), etc. Exploiting a bound
involving $I_{\varphi}(X,Y)$ for a broad enough set of functions $\varphi$
allows to differently measure the dependence between $X$ and $Y$. This allows
us to provide bounds that are tailored to the specific problem at hand and, as
we will see, to provide bounds that are tighter with respect to the ones
leveraging the well-known Mutual Information.

###  2.2 (Strong) Data-Processing Inequalities

An important property that $\varphi$-Divergences share is the Data-Processing
Inequality (DPI) i.e., given two measures $\mu,\nu$ and a Markov Kernel $K$,
one has that for every convex $\varphi$

$D_{\varphi}(\nu K\|\mu K)\leq D_{\varphi}(\nu\|\mu).$ (19)

This property holds as well for Rényi’s $\alpha$-Divergences, despite them not
being a $\varphi$-Divergence [16, Theorem 9]. DPIs represent a powerful and
widely used tool to derive bounds in the field, so much so that a large body
of work has focused on tightening said inequalities. In particular, in many
settings of interest, given a reference measure $\mu$ one can show that
$D_{\varphi}(\nu K\|\mu K)$ is strictly smaller than $D_{\varphi}(\nu\|\mu)$
unless $\nu=\mu$ and the characterisation of the ratio between these two
quantities for Markov Kernels of interest gave rise to the concept and study
of “strong Data-Processing Inequalities”. Formally,

######  Definition 7 ([22, Definition 3.1]).

Given a probability measure $\mu$, a Markov Kernel $K$ and a convex function
$\varphi,$ we say that $K$ satisfies a $\varphi$-type strong data-processing
inequality (SDPI) at $\mu$ with constant $c\in[0,1)$ if, for all $\nu\ll\mu$
one has that

$D_{\varphi}(\nu K\|\mu K)\leq c\cdot D_{\varphi}(\nu\|\mu).$ (20)

In order to characterise the tightest such constant $c$, let us define the
following objects:

$\displaystyle\eta_{\varphi}(\mu,K)$
$\displaystyle=\sup_{\nu\neq\mu}\frac{D_{\varphi}(\nu K\|\mu
K)}{D_{\varphi}(\nu\|\mu)}$ $\displaystyle\eta_{\varphi}(K)$
$\displaystyle=\sup_{\mu}\eta_{\varphi}(\mu,K).$

Said quantities are generally hard to compute for a given Markov Kernel $K$
and functional $\varphi$, however a variety of bounds is present in the
literature (see [22]). In particular, given any convex $\varphi$ it is
possible to show the following result:

######  Lemma 2 ([23, Theorem 4.1]).

Let $K:\mathcal{F}\times\Omega\to[0,1]$ represent a Markov Kernel, and let
$\varphi$ be a convex functional such that $\varphi(1)=0$, one has that

$\eta_{\varphi}(K)\leq\vartheta(K),$ (21)

where $\vartheta(K)$ represents the Dobrushin contraction coefficient of $K$
i.e.,

$\vartheta(K)=\max_{x,\hat{x}\in\Omega}TV(K(\cdot|x),K(\cdot|\hat{x})).$ (22)

Despite the generality of the result one can show that Equation 21 does not
hold true for Rényi’s Divergences:

######  Example 1.

Let $\mu=(1/2,1/2)$ and $K=\text{BSC}(\lambda)$ with $\lambda<\frac{1}{2}$.
Then $\eta_{TV}(K)=\vartheta(K)=(1-2\lambda)$. Consider now
$D_{\alpha}(K(\cdot|0)\|\mu)=D_{\alpha}(\delta_{0}K\|\mu
K)=\frac{1}{\alpha-1}\log(2^{1-\alpha}(\lambda^{\alpha}+(1-\lambda)^{\alpha}))$.
Moreover, $D_{\alpha}(\delta_{0}\|\mu)=-\log(2)$. If $\lambda=0.2$ and
$\alpha=6$ one has that

$\eta_{D_{\alpha}}(K)>\frac{D_{\alpha}(\delta_{0}K\|\mu
K)}{D_{\alpha}(\delta_{0}\|\mu)}=0.6138>\vartheta(K)=0.6,$ (23)

and the gap increases with $\alpha.$

Moreover, a universal lower-bound is known as well, in case of three times
differentiable functions $\varphi$ with $\varphi^{\prime\prime}(1)>0$ [22,
Theorem 3.3]:

$\eta_{\varphi}(K)\geq\eta_{\chi^{2}}(K)=\sup_{\mu}\left(\sup_{f,g}\mu
K(f\cdot g)\right)^{2}=\sup_{\mu}S^{2}(\mu,K),$ (24)

where the term on the right-hand side of Equation 24 is also known in the
literature as “Maximal Correlation”. For functions $\varphi$ which are
operator convex, the inequality in Equation 24 is actually an equality.
Examples of operator convex functions are: $x\log(x)$, $x^{p}$ with $1\leq
p\leq 2$, and so on. This is particularly advantageous because it allows us to
leverage tensorisation properties of SDPI that are known to hold for the
Kullback-Leibler Divergences even for other divergences (e.g., Hellinger $p$
with $p\leq 2$). More precisely, one can say the following: if $\varphi$ is
operator convex and the channel considered is an $n$-fold tensor-product
$K^{\otimes n}$ then the following holds true [22, Corollary 3.1],[24,
Equation (62)]:

$\eta_{\varphi}(K^{\otimes n})\leq 1-(1-\eta_{\varphi}(K))^{n}.$ (25)

###  2.3 Variational Repesentations and Functional Inequalities

A re-interpretation of the comments stated in Section 2 and tailored to
divergences leads us to the main technical tools that will be used through the
document: “variational representations” and functional inequalities. The main
starting point will be looking at divergences as functionals acting on the
first measure i.e., $D_{\varphi}(\cdot\|\mu)=\psi_{\mu}(\cdot)$. Once this is
established, most variational representations are instances of Legendre-
Fenchel duality as stated in Equation 4. The most well-known is certainly the
Donsker-Varadhan representation of the Kullback-Leibler Divergence, that
states the following [25]:

$D(\nu\|\mu)=\sup_{f\in
B(\mathcal{X})}\langle\nu,f\rangle-\log\left(\mu(\exp(f))\right),$ (26)

where $B(\mathcal{X})$ denotes the space of bounded and measurable real-valued
functions defined on $\mathcal{X}$. Equation 26 characterises the Kullback-
Leibler divergence as the Legendre-Fenchel dual of the functional
$\log(\mu(\exp(f)))=\vartheta_{\mu}(f)$ i.e.,
$D(\cdot\|\mu)=\vartheta_{\mu}^{\star}(\cdot).$ Similar variational
representation can be found for large families of Divergences, like Rényi’s
Divergences [26, 27] and $\varphi$-Divergences [28]. We will now lay the
ground-work in order to state the variational representation for
$\varphi$-Divergences as it represents a meaningful tool for the scope of this
work. In particular, let $F(\mathcal{X})$ be an arbitrary family of real-
valued functions defined on $\mathcal{X}$ and denote with
$\mathcal{M}_{1}(\mathcal{X})$ the space of probability measures over
$\mathcal{X}$. Denote with $\langle F(\mathcal{X})\cup B(\mathcal{X})\rangle$
the linear span of $F(\mathcal{X})\cup B(\mathcal{X})$. Moreover, consider the
following sets:

$\mathcal{M}_{1}^{F}(\mathcal{X})=\left\\{\nu\in\mathcal{M}_{1}(\mathcal{X}):\int|f|d\nu<\infty\text{
for }f\in F(\mathcal{X})\right\\},$

and

$\mathcal{M}^{F}(\mathcal{X})=\left\\{\nu\in\mathcal{M}(\mathcal{X}):\int|f|d|\nu|\text{}<\infty\text{
for }f\in F(\mathcal{X})\right\\}.$

If $F(\mathcal{X})=B(\mathcal{X})$ then
$\mathcal{M}_{1}^{F}(\mathcal{X})=\mathcal{M}_{1}(\mathcal{X})$ and
$\mathcal{M}^{F}(\mathcal{X})=\mathcal{M}(\mathcal{X})$. Denote with
$\tau_{F}$ the weakest topology on $\mathcal{M}^{F}(\mathcal{X})$ such that
all mappings $\nu\to\nu(f)$ are continuous when $f\in\langle
F(\mathcal{X})\cup B(\mathcal{X})\rangle$ and with $\tau_{M}$ the weakest
topology on $\langle F(\mathcal{X})\cup B(X)\rangle$ such that all mappings
$f\to\nu(f)$ are continuous when $\nu\in\mathcal{M}^{F}(\mathcal{X})$. One can
then show the following result

######  Proposition 2 ([28, Proposition 2.1]).

The space $\mathcal{M}^{F}(\mathcal{X})$ equipped with the $\tau_{F}$-topology
and the space $\langle F(\mathcal{X})\cup B(\mathcal{X})\rangle$ equipped with
the $\tau_{M}$ are locally convex topological vector spaces and are the
topological dual of each other.

$D_{\varphi}(\cdot\|\mu)=\psi_{\mu}(\cdot)$ is thus a convex and lower semi-
continuous mapping with respect to $\tau_{F}$ [28, Proposition 2.2] and it is
possible to characterise its variational representation, bridging us between
the two spaces $\mathcal{M}^{F}(\mathcal{X})$ and $\langle F(\mathcal{X})\cup
B(\mathcal{X})\rangle$. For the additional technical condition required on
$\varphi$ (i.e, guaranteeing the uniqueness of the dual optimal solution the
reader is referred to [28]).

######  Theorem 1 ([28, Proposition 4.2 and Theorem 4.3]).

Let $\varphi$ be a strictly convex functional and let
$\mu\in\mathcal{M}(\mathcal{X})$. One has that for every
$\nu\in\mathcal{M}^{F}(\mathcal{X})$:

$D_{\varphi}(\nu\|\mu)=\sup_{f\in\langle F(\mathcal{X})\cup
B(\mathcal{X})\rangle}\nu(f)-\mu(\varphi^{\star}(f)),$ (27)

where $\varphi^{\star}$ denotes the Legendre-Fenchel dual of $\varphi$.
Moreover, one has that for a given $f\in\langle F(\mathcal{X})\cup
B(\mathcal{X})\rangle$:

$\mu(\varphi^{\star}(f))=\sup_{\nu\in\mathcal{M}^{F}(\mathcal{X})}\nu(f)-D_{\varphi}(\nu\|\mu).$
(28)

######  Remark 2 (Expected values, divergences and duality).

Through Equation 27, given a measure $\mu$, one can connect the expected value
of any function $f\in\langle F(\mathcal{X})\cup B(\mathcal{X})\rangle$ under
any measure $\nu\ll\mu$ (i.e., $\nu(f)$) to the divergence
$D_{\varphi}(\nu\|\mu)$. The behavior of the third actor in Equation 27, the
dual of $D_{\varphi}(\cdot\|\mu)$, is crucial in order to obtain bounds. For
instance, when $f$ is the indicator function of an event, one can explicitly
compute the dual (and then retrieve a family of Fano-like inequalities
involving arbitrary divergences, see [14], [29, Chapter 3]). When $f$ is _not_
an indicator function, one cannot typically compute the dual explicitly and
has to upper-bound it leveraging properties of $\mu$ and $f$. In this work, to
provide such an upper bound on the dual, we will make use of Markov’s
inequality. This takes us back to indicator functions for which we can
completely characterise the dual. For technical details see Section B.2. This
pattern is fundamental whenever one is trying to relate (via upper or lower
bounds) the expected value of a function to some divergence/entropy (see [29,
Chapter 2]).

The other main tool that will be utilised is Hölder’s inequality. In
particular, there is a connection between Rényi’s $\alpha$-information measure
and $L^{\alpha}$-norms of the Radon-Nikodym derivative. The results we are
about to provide are all a consequence of one or multiple applications of
Hölder’s inequality333 Hölder’s inequality can itself be seen as functional
inequality stemming from Equation 5, considering the usual pairing and, as a
starting norm, the regular $L^{\alpha}$-norm but also as a consequence of
Equation 4 with $\psi(x)=\frac{|x|^{\alpha}}{\alpha}$. For details see [29,
Section 1.3.1]..

######  Theorem 2 ([29]).

Let
$(\mathcal{X}\times\mathcal{Y},\mathcal{F},\mathcal{P}_{XY}),(\mathcal{X}\times\mathcal{Y},\mathcal{F},\mathcal{P}_{X}\mathcal{P}_{Y})$
be two probability spaces, and assume that
$\mathcal{P}_{XY}\ll\mathcal{P}_{X}\mathcal{P}_{Y}$. Given an
$\mathcal{F}$-measurable function
$f:\mathcal{X}\times\mathcal{Y}\to\mathbb{R}^{+}$, then,

$\displaystyle\mathcal{P}_{XY}(f)$ $\displaystyle\leq\left\lVert\left\lVert
f\right\rVert_{L^{\beta}(\mathcal{P}_{X})}\right\rVert_{L^{\beta^{\prime}}(\mathcal{P}_{X})}\cdot\left\lVert\left\lVert\frac{d\mathcal{P}_{XY}}{d\mathcal{P}_{X}\mathcal{P}_{Y}}\right\rVert_{L^{\alpha}(\mathcal{P}_{X})}\right\rVert_{L^{\alpha^{\prime}}(\mathcal{P}_{Y})}$
(29)

where $\beta,\alpha,\beta^{\prime},\alpha^{\prime}$ are such that
$1=\frac{1}{\alpha}+\frac{1}{\beta}=\frac{1}{\alpha^{\prime}}+\frac{1}{\beta^{\prime}}$.
Given a measurable function $g$, $\left\lVert g\right\rVert_{L^{\alpha}(\mu)}$
denotes the $\alpha$-Norm of $g$ under $\mu$ i.e., $\left(\int
g^{\alpha}d\mu\right)^{\frac{1}{\alpha}}.$

######  Remark 3.

Theorem 2 (and corresponding generalisations including Orlicz and Amemiya
norms) has already appeared in [14] in a slightly less general form, and in
[29] in a variety of forms. It has been re-stated here for ease of reference.

Theorem 2 provides multiple degrees of freedom:

  * •

the parameters characterising the norms: $\alpha,\alpha^{\prime}$;

  * •

the (positive-valued) function $f$.

Three choices of the above are meaningful to us:

  1. 1.

$\alpha^{\prime}=\alpha$, which makes Rényi’s Divergence of order $\alpha$
appear on the right-hand side of Equation 29 (as a norm of the Radon-Nikodym
derivative);

  2. 2.

$\alpha^{\prime}\to 1$, which makes Sibson’s Mutual Information of order
$\alpha$ appear on the right-hand side of Equation 29;

  3. 3.

$f=\mathbbm{1}_{E}$, which allows us to relate the probability of the same
event under the joint and a function of the product of the marginals (and an
information measure).

The choices described in Item 1 and Item 3 give rise to the following
corollary:

######  Corollary 1 ([14, Corollary 6]).

Given $E\in\mathcal{F}$ and $\alpha>1$, we have that:

$\mathcal{P}_{XY}(E)\leq(\mathcal{P}_{X}\mathcal{P}_{Y}(E))^{\frac{\alpha-1}{\alpha}}\exp\left(\frac{\alpha-1}{\alpha}D_{\alpha}(\mathcal{P}_{XY}\|\mathcal{P}_{X}\mathcal{P}_{Y})\right),$
(30)

while the choices described in Item 2 and Item 3 give rise to the following
corollary:

######  Corollary 2 ([14, Corollary 1]).

Given $E\in\mathcal{F}$, we have that:

$\displaystyle\mathcal{P}_{XY}(E)$
$\displaystyle\leq\left(\operatorname*{ess\,sup}_{\mathcal{P}_{y}}\mathcal{P}_{X}(E_{Y})\right)^{1/\beta}\cdot\mathcal{P}_{Y}\left(\mathcal{P}_{X}^{1/\alpha}\left(\left(\frac{d\mathcal{P}_{XY}}{d\mathcal{P}_{X}\mathcal{P}_{Y}}\right)^{\alpha}\right)\right)$
(31)
$\displaystyle=\left(\operatorname*{ess\,sup}_{\mathcal{P}_{y}}\mathcal{P}_{X}(E_{Y})\right)^{1/\beta}\cdot\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(X,Y)\right),$
(32)

where $I_{\alpha}(X,Y)$ is the Sibson mutual information of order $\alpha$,
[19]. Moreover, $\alpha$ and $\beta$ are such that
$\frac{1}{\alpha}+\frac{1}{\beta}=1$.

##  3 Main Results: Lower Bounds on the Risk

The very first result one can provide stems from an immediate application of
Corollary 2 in conjunction with Markov’s Inequality:

######  Theorem 3.

Consider the Bayesian framework described in Section 1.3. The following must
hold for every $\alpha>1$ and $\rho>0$:

$R_{B}\geq\rho\left(1-\exp\left(\frac{\alpha-1}{\alpha}\left(I_{\alpha}(W,X)+\log(L_{W}(\rho))\right)\right)\right).$
(33)

Moreover, taking the limit of $\alpha\to\infty$ one recovers the following:

$R_{B}\geq\sup_{\rho>0}\rho\left(1-\exp\left(\mathcal{L}\left(W\\!\\!\to\\!\\!X\right)+\log(L_{W}(\rho))\right)\right).$
(34)

The proof can be found in Section B.1. Two remarks are in order:

  * •

It is important to notice that the behaviour of Equation 33 is fundamentally
different from [1, Theorem 1]. In [1, Theorem 1] the dependence is linear with
respect to the Mutual Information and logarithmic in $L_{W}(\rho)$ while in
Theorem 3 there is an exponential dependence on $I_{\alpha}$ and linear in
$L_{W}(\rho).$

  * •

Theorem 3 introduces a new parameter $\alpha>1$ to optimise over. The presence
of $\alpha$ leads to a trade-off between the two quantities for a given
$\rho$, $I_{\alpha}(W,X)$ and $L_{W}(\rho)$:
$\frac{\alpha-1}{\alpha}I_{\alpha}(W,X)$ will increase with $\alpha$ whereas
$L_{W}(\rho)^{\frac{\alpha-1}{\alpha}}$ will decrease with $\alpha$.

An interesting characteristic of Equation 34 is that
$\mathcal{L}\left(W\\!\\!\to\\!\\!X\right)$ depends on $W$ only through the
support. This allows us to provide, essentially for free, an even more general
lower-bound on the risk. Indeed, ignoring $L_{W}(\rho)$ for a moment, for a
fixed family of $\mathcal{P}_{X|W}$,
$\mathcal{L}\left(W\\!\\!\to\\!\\!X\right)$ has the same value regardless of
$\mathcal{P}_{W}$ (as long as the support of $W$ remains the same). We can
also walk the same path undertaken in [14] and derive a variety of lower
bounds involving a variety of information measures.

######  Theorem 4.

Consider the Bayesian framework described in Section 1.3. Let
$\varphi:[0,+\infty)\to\mathbb{R}$ be a monotone, strictly convex function and
suppose that the generalised inverse, defined as $\varphi^{-1}(y)=\inf\\{t\geq
0:\varphi(t)>y\\}$, exists. Then for every $\rho>0$ and every estimator
$\hat{W}$, if $\varphi$ is non-decreasing one has the following

$\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\rho\left(1-L_{W}(\hat{W},\rho)\cdot\varphi^{-1}\left(\frac{I_{\varphi}(W,\hat{W})+(1-L_{W}(\hat{W},\rho))\cdot\varphi^{\star}(0)}{L_{W}(\hat{W},\rho)}\right)\right),$
(35)

while if $\varphi$ is non-increasing one recovers the following:

$\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\rho\left(1-L_{W}(\hat{W},\rho)\right)\cdot\varphi^{-1}\left(\frac{I_{\varphi}(W,\hat{W})+L_{W}(\hat{W},\rho)\cdot\varphi^{\star}(0)}{1-L_{W}(\hat{W},\rho)}\right).$
(36)

The proof can be found in Section B.2.

######  Remark 4 (Recovering Mutual Information).

A natural question is whether Theorem 4 also includes Shannon’s Mutual
Information (and, consequently, the results in [1]). Selecting
$\varphi(x)=x\log x$ is problematic as the function is non-monotonic and its
inverse would not have a closed-form expression one could leverage in
Equations 35 to 36. However, following the same steps undertaken in Section
B.2, with $\varphi(x)=x\log(x)$, but with a different choice of
$f=-\tilde{\lambda}\ell-\log\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\exp(-\tilde{\lambda}\ell))+1$
and
$\tilde{\lambda}=-\frac{1}{\rho}\log\left(\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})\right)$,
Equation 120 does lead to [1, Theorem 1].

######  Remark 5 (Simplification).

If $\varphi^{\star}(0)\leq 0$, the expressions in Equations 35 and 36 can be
respectively simplified as follows, if $\varphi$ is non-decreasing:

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\rho\left(1-L_{W}(\hat{W},\rho)\cdot\varphi^{-1}\left(\frac{I_{\varphi}(W,\hat{W})}{L_{W}(\hat{W},\rho)}\right)\right),$
(37)

while if $\varphi$ is non-increasing:

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\rho\left(1-L_{W}(\hat{W},\rho)\right)\cdot\varphi^{-1}\left(\frac{I_{\varphi}(W,\hat{W})}{1-L_{W}(\hat{W},\rho)}\right).$
(38)

The assumption $\varphi^{\star}(0)\leq 0$ holds indeed true in a variety of
cases (cf. Corollary 3).

Although Theorem 4 represents a quite general result, in order to apply it to
the Bayesian Risk setting (and provide an estimator-independent lower-bound)
one has to carefully select $\varphi$. In particular, one has to render the
right-hand side of Equation 35 (or Equation 37) independent of
$\hat{W}=\phi(X)$. In order to do that, the following two quantities need to
be rendered independent of $\hat{W}$:

  1. 1.

The information measure (e.g., through the data-processing inequality
$I_{\varphi}(W,\hat{W})\leq I_{\varphi}(W,X)$);

  2. 2.

The quantity $L_{W}(\hat{W},\rho)$ (which can be easily upper-bounded in the
following way:
$L_{W}(\hat{W},\rho)\leq\sup_{\hat{w}}L_{W}(\hat{w},\rho)=L_{W}(\rho)$).

For simplicity, consider Equation 37 and introduce the following object

$G_{\varphi}(I_{\varphi},L_{W})=L_{W}(\hat{W},\rho)\cdot\varphi^{-1}\left(\frac{I_{\varphi}(W,\hat{W})}{L_{W}(\hat{W},\rho)}\right).$
(39)

To use the two inequalities just stated in Item 1) and Item 2), one needs that
for a given choice of $\varphi$, $G_{\varphi}(I_{\varphi},L_{W})$ is
increasing in $I_{\varphi}$ for a given value of $L_{W}$ and vice versa. This
allows us to further lower bound Equation 37 and render the quantity
independent of the specific choice of $\phi$. Analogously, one can state
similar assumptions in order to apply the same reasoning to Equation 38.
Hence, starting from Equation 1 one can provide a lower bound on the risk
$R_{B}$ that is independent of $\phi$.
Let us now look at some specific choices of $\varphi$ such that $G_{\varphi}$
satisfies the desired properties and, thus, for which a bound on the Bayesian
risk can be retrieved.

######  Corollary 3.

Consider the Bayesian framework described in Section 1.3. The following must
hold for every $p>1$ and $\rho>0$:

$R_{B}\geq\rho\left(1-L_{W}(\rho)^{\frac{p-1}{p}}\cdot\left((p-1)\mathcal{H}_{p}(W,X)+1\right)^{\frac{1}{p}}\right).$
(40)

The proof of Corollary 3 is in Section B.3. Restricting the choice of
$\varphi$ to this family of polynomials one can thus state the following
lower-bound on the risk:

$R_{B}\geq\sup_{\rho>0}\sup_{p>1}\rho\\!\left(1-L_{W}(\rho)^{\frac{p-1}{p}}\cdot\left((p-1)\mathcal{H}_{p}(W,\hat{W})+1\right)^{\frac{1}{p}}\right).$
(41)

######  Remark 6.

Using the one-to-one mapping connecting Hellinger divergences and Rényi’s
$\alpha$-Divergence, the bound above can be re-written as follows:

$\displaystyle
R_{B}\geq\sup_{\rho>0}\sup_{\alpha>1}\rho\left(1-L_{W}(\rho)^{\frac{\alpha-1}{\alpha}}\cdot\exp\left(\frac{\alpha-1}{\alpha}D_{\alpha}(\mathcal{P}_{W\hat{W}}\|\mathcal{P}_{W}\mathcal{P}_{\hat{W}})\right)\right).$
(42)

Moreover, via this relationship one can see that, for a given $\alpha>1$:

$\displaystyle\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X^{n})\right)$
$\displaystyle=\exp\left(\frac{\alpha-1}{\alpha}\inf_{\mathcal{Q}_{X^{n}}}D_{\alpha}(\mathcal{P}_{WX^{n}}\|\mathcal{P}_{W}\mathcal{Q}_{X^{n}})\right)$
(43)
$\displaystyle\leq\exp\left(\frac{\alpha-1}{\alpha}D_{\alpha}(\mathcal{P}_{WX^{n}}\|\mathcal{P}_{W}\mathcal{P}_{X^{n}})\right)$
(44)
$\displaystyle=\left(\mathcal{H}_{\alpha}(W,X^{n})\right)^{\frac{1}{\alpha}}.$
(45)

Consequently, given the similarity between Equation 40 and Equation 33 one can
easily see that leveraging $I_{\alpha}$, when possible, provides a larger
lower-bound than the Hellinger divergence. However, as we will see, in a
variety of settings, the Hellinger $\alpha$-divergence can be computed
explicitly while $I_{\alpha}$ cannot. For this reason, we will often leverage
the Hellinger divergence to provide closed-form lower bounds on the Risk.

Clearly, a number of results can be derived from Theorem 4. Each of them with
potentially interesting applications in specific Bayesian Estimation settings.
In this work, we will mostly focus on Sibson’s $\alpha$-Mutual Information and
Hellinger $p$-Divergences. In the spirit of leveraging the generality of
Theorem 4, we also provide a bound involving a novel information measure
$E_{\gamma,\zeta}$, strongly inspired by the so-called $E_{\gamma}$-Divergence
[30, Equation (66)], [31, Page 2314], also known in the literature as the
Hockey-stick Divergence and whose application in this framework has been
explored in [11]. Its definition is the following:

######  Definition 8.

Let $(\Omega,\mathcal{F})$ be a measurable space and let $\mu$ and $\nu$ be
two probability measures defined on the space. Denote with
$\varphi_{\gamma,\zeta}(x)=\max\\{0,\zeta
x-\gamma\\}-\max\\{0,\zeta-\gamma\\}$ with $\zeta>0$ and $\gamma\geq 0$. The
function $\varphi_{\gamma,\zeta}(x)$ is convex, increasing and is such that
$\varphi_{\gamma,\zeta}(1)=0$. Assume that $\nu\ll\mu$, then define the
following object:

$E_{\gamma,\zeta}(\nu\|\mu)=D_{\varphi_{\gamma,\zeta}}(\nu\|\mu).$ (46)

Moreover, whenever $\nu=\mathcal{P}_{XY}$ and
$\mu=\mathcal{P}_{X}\mathcal{P}_{Y}$ we denote (with a slight abuse of
notation) $E_{\gamma,\zeta}(\mathcal{P}_{XY}\|\mathcal{P}_{XY})$ with
$E_{\gamma,\zeta}(X,Y).$ If $\zeta=1$ then one recovers the usual
$E_{\gamma}$-divergence.

Leveraging it, one can provide the following result in this framework:

######  Corollary 4.

Consider the Bayesian framework described in Section 1.3. The following must
hold for every $\zeta>0$, $\gamma\geq 0$, and $\rho>0$:

$R_{B}\geq\rho\left(1-\frac{E_{\gamma,\zeta}(W,\hat{W})+\gamma
L_{W}(\rho)+\max\\{0,\zeta-\gamma\\}}{\zeta}\right).$ (47)

One can thus retrieve the following lower-bound on the risk:

$R_{B}\geq\sup_{\rho>0}\sup_{\zeta>0,\gamma\geq
0}\rho\left(1-\frac{E_{\gamma,\zeta}(W,\hat{W})+\gamma
L_{W}(\rho)+\max\\{0,\zeta-\gamma\\}}{\zeta}\right).$ (48)

The proof is in Section B.4.

######  Remark 7.

Setting $\zeta=1$ in Equation 47 one recovers [11, Remark 1]. In fact, by
introducing an additional degree of freedom through the $\zeta$ parameter in
Equation 48, the resulting lower-bound can only be tighter than [11, Remark
1].

Using these results one can provide meaningful lower bounds on the Risk in a
variety of settings of interest, as we will see in Section 4. We will now
explore a few natural extensions over the framework just presented that follow
from either a slight change of perspective or from a slight alteration of the
observation model considered.

###  3.1 Leveraging SDPIs

A key step in the results proved here (as well as in [1]) consists of
leveraging the Markov Chain $W-X-\hat{W}$ along with DPI as follows:
$I_{\varphi}(W,\hat{W})\leq I_{\varphi}(W,X^{n})$. In case more information is
available with respect to the kernel linking $X^{n}$ and $\hat{W}$ (i.e., more
information on how the estimation is executed) then one can leverage the
corresponding SDPI inequality as follows:

$I_{\varphi}(W,\hat{W})\leq
I_{\varphi}(W,X^{n})\eta_{\varphi}(\mathcal{P}_{X^{n}},\mathcal{P}_{\hat{W}|X^{n}})\leq
I_{\varphi}(W,X^{n})\eta_{\varphi}(\mathcal{P}_{\hat{W}|X^{n}}),$

potentially providing a refinement over the results that can be advanced.
Moreover, in a variety of settings one does not have direct access to samples
(but rather access to noisy copies or privatised versions). In this case, one
can provide refinements over the bounds via SDPI’s and, consequently, provide
results tailored to the type of noise that has been injected. Consider the
following setting in which the samples $X^{n}$ generated from $W$ are not
directly observed but rather a sequence $Z^{n}$, where $Z^{n}$ is a
noisy/privatised version of $X^{n}$ obtained through the sequence of Markov
Kernel $K_{1},\ldots,K_{n}$ where for every $i\geq 1$,
$K_{i}:\mathcal{F}\times\mathcal{X}\to[0,1]$ and
$\mathcal{P}_{Z_{i}}=\mathcal{P}_{X_{i}}K_{i}$. The goal is to estimate $W$
from the sequence $Z^{n}$ via an estimator
$\psi:\mathcal{Z}^{n}\to\mathcal{W}$. Given a loss function
$\ell:\mathcal{W}\times\hat{\mathcal{W}}\to\mathbb{R}^{+}$, the noisy Bayesian
Risk is thus defined as

$R_{B}^{\text{noisy}}=\inf_{\psi}\mathcal{P}_{WZ}(\ell(W,\psi(Z^{n})))=\inf_{\psi}\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W})),$
(49)

and it can be lower-bounded similarly to the non-private/noisy case via
Theorem 4. For simplicity of exposition, we will consider a single kernel $K$
and, consequently, one has that $Z^{n}$ is obtained from $X^{n}$ through the
tensor-product $K^{\otimes n}$. In this case, one has the Markov chain
$W-X^{n}-Z^{n}-\hat{W}$ and can leverage SDPI twice as follows:

$I_{\varphi}(W,\hat{W})\leq\eta_{\varphi}(\mathcal{P}_{W|Z^{n}})I_{\varphi}(W,Z^{n})\leq\eta_{\varphi}(\mathcal{P}_{W|Z^{n}})\eta_{\varphi}(\mathcal{P}_{X^{n}},K^{\otimes
n})I_{\varphi}(W,X^{n}),$ (50)

and consequently state the following result

######  Corollary 5.

Consider the private Bayesian framework considered above. Denote with $Z^{n}$
the private samples obtained from $X^{n}$ through the tensor product of a
kernel $K$. Let $\varphi:[0,+\infty)\to\mathbb{R}$ be a monotone convex
function and suppose that the generalised inverse, defined as
$\varphi^{-1}(y)=\inf\\{t\geq 0:\varphi(t)>y\\}$, exists. Assume as well that
the function $G_{\varphi}$ defined in Equation 39 is non-decreasing in both
arguments. Then, for every estimator $\hat{W}$, if $\varphi$ is non-decreasing
one has the following

$\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\sup_{\rho>0}\rho\left(1-L_{W}(\hat{W},\rho)\cdot\varphi^{-1}\left(\frac{\eta_{\varphi}(\mathcal{P}_{\hat{W}|Z^{n}})\eta_{\varphi}(\mathcal{P}_{X^{n}},K^{\otimes
n})I_{\varphi}(W,X^{n})+(1-L_{W}(\rho))\varphi^{\star}(0)}{L_{W}(\rho)}\right)\right),$
(51)

whereas if $\varphi$ is non-increasing one recovers the following:

$\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\sup_{\rho>0}\rho\left(1-L_{W}(\hat{W},\rho)\right)\cdot\varphi^{-1}\left(\frac{\eta_{\varphi}(\mathcal{P}_{\hat{W}|Z^{n}})\eta_{\varphi}(\mathcal{P}_{X^{n}},K^{\otimes
n})I_{\varphi}(W,X^{n})+L_{W}(\rho)\cdot\varphi^{\star}(0)}{1-L_{W}(\rho)}\right).$
(52)

######  Remark 8.

Notice that in case one does _not_ estimate from noisy observations $Z^{n}$
then one can still consider the same setting as in Corollary 5 with $K$
representing the kernel associated to the identity mapping i.e.,
$K(y|x)=\delta_{x}(y)$. In this case one has that
$\mathcal{P}_{X^{n}}=\mathcal{P}_{Z^{n}}$,
$\eta_{\varphi}(\mathcal{P}_{X^{n}},K^{\otimes n})=1$ and, consequently,
$\eta_{\varphi}(\mathcal{P}_{\hat{W}|Z^{n}})=\eta_{\varphi}(\mathcal{P}_{\hat{W}|X^{n}})$.
Hence, Corollary 5 boils down to a refinement of Theorem 4 without any
additional noise injection.

######  Remark 9.

In case one is considering $E_{\gamma,\zeta}$ with $\zeta=1$, then the
corresponding contraction parameter has been analysed in [11], where
contraction has been shown to be equivalent to Local Differential-Privacy
(LDP). I.e., a kernel $K$ is said to be $(\epsilon,\delta)$ “Locally-
Differentially private” (LDP), if:

$\max_{E\in\mathcal{F},x,\hat{x}}|K(E|x)-e^{\epsilon}K(E|\hat{x})|\leq\delta.$
(53)

In this case, one has that $\eta_{\varphi_{e^{\epsilon}}}(K)\leq\delta$.
Moreover, due to [11, Lemma 2] one has that if $K$ is $(\epsilon,\delta)$-LDP
then for every $\varphi$ on has that
$\eta_{\varphi}(K)\leq(1-(1-\delta)e^{-\epsilon}).$ It is unclear, however,
whether there are settings in which said upper-bound is more convenient than
others. Indeed, one also has that for every convex function $\varphi$

$\eta_{\varphi}(K)\leq\eta_{TV}(K)<(1-(1-\delta)e^{-\epsilon}),$ (54)

and, for many channels, $\eta_{TV}(K)$ is relatively easy to compute.
Moreover, even $\eta_{TV}(K)$ tends to be quite larger than the effective
contraction coefficient of the divergence at hand: e.g., if
$K=\text{BSC}(\lambda_{\epsilon})$ with
$\lambda_{\epsilon}=\frac{1}{1+e^{\epsilon}}$ then $K$ is $(\epsilon,0)$-LDP
(cf. [11, Example 1]) and if $\varphi$ is operator-convex (e.g.,
$\varphi(x)=x\log x$ or $\frac{x^{p}-1}{p-1}$with $1<p\leq 2$, etc.) then [22,
Corollary 3.1]:

$\eta_{\varphi}(K)=\left(1-\frac{2}{1+e^{\epsilon}}\right)^{2}\ll\left|1-\frac{2}{1+e^{\epsilon}}\right|<(1-e^{-\epsilon}).$
(55)

An important feature of Corollary 5 is that for a subclass of functions
$\varphi$ one can leverage tensorisation properties of $\eta_{\varphi}$.
Indeed, if $\varphi$ satisfies the conditions of [22, Theorem 3.9] then one
has that

$\eta_{\varphi}(\mu^{\otimes n},K^{\otimes n})=\eta_{\varphi}(\mu,K).$ (56)

Moreover, if $\varphi$ is operator convex, then one can say the following:

$\eta_{\varphi}(K^{\otimes n})=\eta_{\chi^{2}}(K^{\otimes n})\leq
1-(1-\eta_{\varphi}(K))^{n}.$ (57)

Both the results are true for instance, for $\varphi(x)=(x^{p}-1)/(p-1)$ with
$1\leq p\leq 2$ (but the assumptions of [22, Theorem 3.9] are violated if
$p>2$). This means that one can leverage Equations 56 to 57 for the Hellinger
divergence $\mathcal{H}_{p}$ with $1\leq p\leq 2$.

###  3.2 Conditioning

Following the approach undertaken in [1], it is also possible to propose a
conditional version of the theorems proposed above, in order to retrieve
tighter bounds. For this to happen one needs a definition of conditional
information measures. For $\varphi$–Divergences the choice would typically
fall on objects of the following form

$I_{\varphi}(X,Y|Z)=D_{\varphi}(\mathcal{P}_{XYZ}\|\mathcal{P}_{Z}\mathcal{P}_{X|Z}\mathcal{P}_{Y|Z}).$
(58)

As for Sibson’s $I_{\alpha}$, the matter becomes slightly more complicated as
$I_{\alpha}(X,Y)=\min_{\mathcal{Q}_{Y}}D_{\alpha}(\mathcal{P}_{XY}\|\mathcal{P}_{X}\mathcal{Q}_{Y})$.
In the case of three random variables, it is unclear which factorisation of
the joint and which minimisation to consider. Indeed, it has been shown in
[32] that several definitions of conditional $I_{\alpha}$ can be proposed,
depending on the operational meaning and corresponding probability bound one
needs. In this subsection, we will consider the following conditional version
of $I_{\alpha}$:

$I^{Y|Z}_{\alpha}(X,Y|Z)=\min_{\mathcal{Q}_{Y|Z}}D_{\alpha}(\mathcal{P}_{XYZ}\|\mathcal{P}_{X|Z}\mathcal{Q}_{Y|Z}\mathcal{P}_{Z}).$
(59)

The choice of this specific definition is necessary in order to provide a
conditional version of Theorem 3 and Equation 34 similar to [1, Theorem 1, Eq.
(5)]. Leveraging said definition and the fact that:

$I^{Y|Z}_{\alpha}(X,Y|Z)\xrightarrow[]{\alpha\to\infty}\mathcal{L}\left(X\\!\\!\to\\!\\!Y|Z\right)$

one can thus give a conditional version of Theorem 3 and Equation 34,
introducing the following notion of conditional small-ball probability,
$L_{W|U}(U,\rho)=\sup_{\hat{w}\in\hat{\mathcal{W}}}\mathcal{P}_{W|U}(\ell(W,\hat{w})<\rho)$:

######  Theorem 5.

Consider the Bayesian framework described in Section 1.3,

$R_{B}\geq\sup_{\mathcal{P}_{U|W,X}}\sup_{\rho>0,\alpha\geq
1}\rho\left(1-\exp\left(\frac{\alpha-1}{\alpha}(I_{\alpha}(W,X|U)+\log(\mathcal{P}_{U}(L_{W|U}(U,\rho)))\right)\right),$
(60)

Moreover, taking the limit of $\alpha\to\infty$ one has:

$R_{B}\geq\sup_{\mathcal{P}_{U|W,X}}\sup_{\rho>0}\rho\left(1-\exp\left(\mathcal{L}\left(W\\!\\!\to\\!\\!X|U\right)+\log(\mathcal{P}_{U}(L_{W|U}(U,\rho)))\right)\right).$
(61)

###### Proof.

For the selected choice of conditional Sibson Mutual Information (see Equation
59) one has that

$I_{\alpha}(W,\hat{W}|U)=\frac{\alpha}{\alpha-1}\log\left\lVert\left\lVert\left\lVert\frac{d\mathcal{P}_{W,\hat{W},U}}{d\mathcal{P}_{U}\mathcal{P}_{\hat{W}|U}\mathcal{P}_{W|U}}\right\rVert_{L^{\alpha}(\mathcal{P}_{W|U})}\right\rVert_{L^{1}(\mathcal{P}_{\hat{W}|U})}\right\rVert_{L^{\alpha}(\mathcal{P}_{U})}.$
(62)

Consequently, one can prove via Hölder’s inequality a result analogous to
Theorem 2 (cf. [29, Theorem 17]) which implies then, selecting
$f=\mathbbm{1}_{\\{\ell(W,\hat{W})<\rho\\}}$, the following for every
$\rho>0,\alpha\geq 1$ and every $U$

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W})<\rho)$
$\displaystyle=\mathcal{P}_{W\hat{W}U}(\ell(W,\hat{W})<\rho)$ (63)
$\displaystyle\leq\mathcal{P}_{U}^{\frac{\alpha-1}{\alpha}}\left(\operatorname*{ess\,sup}_{\mathcal{P}_{\hat{W}|U}}\mathcal{P}_{W|U}(\ell(W,\hat{W})\leq\rho)\right)\cdot\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,\hat{W}|U)\right).$
(64)
$\displaystyle\leq\mathcal{P}_{U}^{\frac{\alpha-1}{\alpha}}\left(L_{W|U}(U,\rho)\right)\cdot\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X|U)\right)$
(65)
$\displaystyle=\exp\left(\frac{\alpha-1}{\alpha}\left(I_{\alpha}(W,X|U)+\log(\mathcal{P}_{U}(L_{W|U}(U,\rho)))\right)\right)$
(66)

The statement of the theorem then follows from the same sequence of steps
(involving Markov’s inequality) that led to Theorem 3. Moreover, starting from
Equation 62 and taking the limit of $\alpha\to+\infty$ one recovers the
following:

$I_{\infty}(W,X|U)=\log\operatorname*{ess\,sup}_{\mathcal{P}_{U}}\left\lVert\operatorname*{ess\,sup}_{\mathcal{P}_{W|U}}\frac{d\mathcal{P}_{W,X,U}}{d\mathcal{P}_{U}\mathcal{P}_{X|U}\mathcal{P}_{W|U}}\right\rVert_{L^{1}(\mathcal{P}_{X|U})},$
(67)

which can be seen as being equal to
$\mathcal{L}\left(W\\!\\!\to\\!\\!X|U\right)$ (cf. [20, Section III.E]). ∎

The main idea behind using conditional Mutual Information, as presented in
[1], is that by choosing an appropriate $U$ it is possible to control the
growth of $I(W;X|U)$ and obtain tighter bounds in some cases. In particular,
consider the sequence of $n$ samples $X^{n}$. If the family
$\mathcal{P}=\\{\mathcal{P}_{X|W=w}:w\in\mathcal{W}\\}$ is a subset of a
finite-dimensional exponential family and $W$ has a density supported on a
compact subset of $\mathbb{R}^{d}$, choosing $U$ to be a conditionally
independent copy $\hat{X^{n}}$ of $X^{n}$ (given $W$) the mutual information
$I(W;X^{n}|\hat{X}^{n})$ will converge to a constant as $n$ grows (rather than
grow with $n$) [1]. This property seems to be specific to Shannon’s Mutual
Information. In the examples considered below, there does not appear to be a
suitable $U$ that tightens the bounds further for the divergences considered.
Nonetheless, we state the result as it may be of interest in other settings.

##  4 Examples of application

In this section, we apply the results presented in the previous section to
four estimation settings. The first three are classical settings, while the
fourth comes from a distributed estimation setting:

  * •

estimation of the mean of a Bernoulli random variable with parameter $W$,
where $W$ is assumed to be uniform between $(0,1)$;

  * •

the same setting as above with the difference that one does not observes the
samples $X^{n}$ directly but rather a noisy/privatised version $Z^{n}$, where
each $Z_{i}$ is assumed to be the outcome of $X_{i}$ after being passed
through a Binary Symmetric Channel with parameter $\lambda$ (BSC($\lambda$));

  * •

estimation of the mean of a Gaussian random variable with different prior
distributions over the mean;

  * •

identification of the biased random variable in a $d$-dimensional vector in a
distributed fashion (cf., the “Hide-and-seek” problem advanced in [2]).

The loss function for the first three cases will be the $L^{1}$-distance while
for the fourth one, we will consider the $0-1$ loss. For the first three
cases, the maximisation over $\rho$ in the lower bounds is carried out
analytically (details in Section C.1).

###  4.1 Bernoulli Bias

######  Example 2.

Suppose that $W\sim\mathcal{U}([0,1])$ and that for each $i\in[n]$,
$X_{i}|W=w\sim\text{Ber}(w)$. Also, assume that $\ell(w,\hat{w})=|w-\hat{w}|$.

Using the sample mean estimator i.e.,
$\hat{W}=\frac{1}{n}\sum_{i=1}^{n}X_{i}$, one has that (cf. [1, Equation
(20)]):

$R_{B}\leq\frac{1}{\sqrt{6n}}.$ (68)

Let us now lower-bound the risk in this setting. First, we find that

$L_{W}(\rho)=\sup_{\hat{w}}\mathcal{P}_{W}(|W-\hat{w}|<\rho)=2\rho.$ (69)

To obtain a lower-bound involving Maximal Leakage, one can see that (details
in Section C.2.1)

$\mathcal{L}\left(W\\!\\!\to\\!\\!X^{n}\right)\leq\log\left(2+\sqrt{\frac{\pi
n}{2}}\right).$ (70)

Substituting Equation 70 in Equation 34, along with $L_{W}(\rho)=2\rho$,
provides us with the following lower-bound on the risk:

$\displaystyle R_{B}$
$\displaystyle\geq\sup_{\rho>0}\rho\left(1-\exp\left(\mathcal{L}\left(W\\!\\!\to\\!\\!X^{n}\right)\right)L_{W}(\rho)\right)$
(71) $\displaystyle\geq\sup_{\rho>0}\rho\left(1-\left(2+\sqrt{\frac{\pi
n}{2}}\right)2\rho\right).$ (72)

The quantity in Equation 72 is a concave function of $\rho$ and thus we can
maximise it. In particular, the maximiser is
$\hat{\rho}=\frac{1}{4\left(2+\sqrt{\frac{\pi n}{2}}\right)}$ and plugging it
in Equation 72 one gets the following:

$R_{B}\geq\frac{1}{8\left(2+\sqrt{\frac{\pi n}{2}}\right)},$ (73)

which, for $n$ large enough (i.e., $n\geq 127/\pi\approx 41$), can be further
lower-bounded as follows

$R_{B}\geq\frac{1}{5\sqrt{2\pi n}}.$

Surprisingly, Maximal Leakage already offers a lower-bound that matches the
upper-bound up to a constant (cf. Equation 68) without any extra machinery.
Equation 73 provides a larger lower-bound than the one provided using Mutual
Information (cf. [1, Corollary 2]) for $n\geq 1$. Moreover, the proof in [1]
needs a more complicated setting involving a conditioning with respect to an
independent copy of $X^{n}$ and can only provide an asymptotic lower bound on
the risk of $1/(16\sqrt{2\pi n})$ (that thus, only holds for $n$ large
enough).
On the contrary, given the closed-form expression, Maximal Leakage can be
quite easy to compute or upper-bound. Moreover, the information measure
depends on $\mathcal{P}_{W}$ only through the support. This means that if one
has access to an upper-bound on $L_{W}(\rho)$ that does not employ any
knowledge of $\mathcal{P}_{W}$ except for the support (e.g., if $W$ were to be
discrete, an upper-bound of $1$ over the probability mass function could
suffice) the resulting lower-bound on the risk (in this example), would apply
to any $W$ whose support is the interval $[0,1]$.
One can also provide a more general lower-bound involving $I_{\alpha}$.
Indeed, one has that (details in Section C.2.2), in this setting:

$\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X^{n})\right)=\sum_{k=0}^{n}\binom{n}{k}\left(\frac{\Gamma(k\alpha+1)\Gamma((n-k)\alpha+1)}{\Gamma(n\alpha+2)}\right)^{\frac{1}{\alpha}}.$
(74)

Plugging Equation 74 in Equation 33 one obtains the following lower-bound on
the risk:

$R_{B}\geq\sup_{\rho>0}\sup_{\alpha>1}\rho\left(1-(2\rho)^{\frac{\alpha-1}{\alpha}}\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X^{n})\right)\right).$
(75)

The lower-bound in Equation 75 can clearly only improve the one provided in
Equation 72, as
$\mathcal{L}\left(W\\!\\!\to\\!\\!X^{n}\right)=I_{\infty}(W,X^{n})>I_{\alpha}(W,X^{n})$
for every $\alpha<\infty$. However, differently from Equation 72, it does not
admit a closed-form expression and needs to be computed numerically in order
to assess how far it is from the upper-bound. Similarly, one could try to
employ a lower-bound that includes Hellinger$-p$ Divergences. The lower-bound
on the risk induced by Corollary 3 is given by

$R_{B}\geq\sup_{\rho>0}\sup_{p>1}\rho\left(1-(2\rho)^{\frac{p-1}{p}}\cdot\left(\mathcal{H}_{p}(W,X^{n})\right)^{\frac{1}{p}}\right).$
(76)

The expressions represented in Equation 76 and Equation 75 are extremely
similar. Thus, one can easily argue that Equation 75 will always provide a
larger lower-bound than Equation 76 and indeed this is confirmed in the
simulations. It is also true that, for some values of $p$, one can actually
provide nice closed-form expressions for the lower-bound provided by Equation
76. Indeed, in general, one has that (details in Section C.2.3):

$((p-1)\mathcal{H}_{p}(W,X^{n})+1)=(n+1)^{p-1}\sum_{k=0}^{n}\binom{n}{k}^{p}\frac{\Gamma(kp+1)\Gamma((n-k)p+1)}{\Gamma(np+2)},$
(77)

Then, with $p=2$ one recovers (details in Section C.2.3):

$\mathcal{H}_{2}(W,X^{n})+1=\chi^{2}(W,X^{n})=\frac{n+1}{2n+1}\cdot\frac{4^{n}}{\binom{2n}{n}}\leq\frac{16\sqrt{\pi
n}}{21}.$ (78)

Hence, specialising Equation (76) to $p=2$ leads us to:

$R_{B}\geq\sup_{\rho>0}\rho\left(1-\sqrt{2\rho(\chi^{2}(W,X^{n})+1)}\right).$
(79)

Solving then the maximisation over $\rho$ and using Equation 78 one can
conclude that:

$\displaystyle R_{B}$
$\displaystyle\geq\frac{2}{27}\cdot\frac{1}{\chi^{2}(W,X^{n})+1}\geq\frac{7}{72\sqrt{\pi
n}}.$ (80)

Notice that Equation 80 also matches the upper-bound up to a constant and,
similarly to Maximal Leakage, tightens the result in [1, Corollary 2] while
not requiring that $n\to\infty$.

######  Remark 10.

Stirling’s approximation yields $(\chi^{2}(W,X^{n})+1)\sim\frac{\sqrt{\pi
n}}{2}$ when $n$ is large. This implies that, for $n$ large, one can show that
$R_{B}\gtrsim\frac{4}{27\sqrt{\pi n}}$, thus leading to a slight improvement
over Equation 80.

To conclude, one can apply the same steps with the
$E_{\gamma,\zeta}$-Divergence. The lower-bound on the risk one can retrieve
via Corollary 4 is the following:n this example can thus be expressed as

$\displaystyle R_{B}$
$\displaystyle\geq\sup_{\zeta,\gamma}\sup_{\rho>0}\rho\left(1-\frac{\left(E_{\gamma,\zeta}(W,X^{n})+2\rho\gamma\right)}{\zeta}\right)$
(81) $\displaystyle=\sup_{\zeta,\gamma}\frac{(\zeta-
E_{\gamma,\zeta}(W,X^{n}))^{2}}{8\gamma\zeta}.$ (82)

The lower-bound in Equation 82 can be empirically seen to be the best among
the ones presented so far (thus beating Hellinger, $I_{\alpha}$ and,
consequently, Maximal Leakage and Mutual Information). A direct comparison
between the bounds provided here and those already present in the literature
can be seen in Figure 1(a) and Figure 1(b). The lower bounds are computed as a
function of the number of samples $n$, which we consider to be in the range
$\\{1,\dots,50\\}$. The figure shows that all the divergences we considered in
this work provide a larger (and thus, tighter) lower-bound on the Bayesian
risk when compared with results that stem from using Shannon’s Mutual
Information (cf. [1, Corollary 2]). In particular, the lower-bound involving
the $E_{\gamma,\zeta}$-Mutual Information represents the largest among the
ones we consider. Given the lack of a closed-form expression for
$E_{\gamma,\zeta}$ in this example, the quantity in Equation 82 was computed
numerically. Moreover, in order to verify whether the behaviour (and ordering)
of the lower bounds in Figure 1(a) was determined by the specific choices of
the parameters $p,\gamma,\zeta$ and $\alpha$, in Figure 1(b) the lower bounds
on the risk have also been numerically optimised over the respective
parameters $p,\gamma,\zeta,\alpha$. As Figure 1(b) shows, the lower-bound
provided by $E_{\gamma,\zeta}$ remains the best. Notice that the lower-bound
involving Mutual Information has no parameter to optimise over (other than
$\rho$). Maximal Leakage does not provide the best bound, but it possesses the
interesting characteristic of depending on $\mathcal{P}_{W}$ only through the
support, thus leading to potential applicability in a variety of settings in
which $\mathcal{P}_{W}$ is not accessible. In contrast, Mutual Information,
the Hellinger Divergence and the $E_{\gamma,\zeta}$-Divergence all require to
know $\mathcal{P}_{W}$. The lower bounds on the Risk in this Example can thus
be summarised as follows:

######  Corollary 6.

Consider the setting of Example 2 one has that

$\displaystyle R_{B}\geq\max\left\\{\max_{\zeta>0,\gamma\geq
0}\left\\{\frac{(\zeta-
E_{\gamma,\zeta}(W,X^{n}))^{2}}{8\gamma\zeta}\right\\},\max_{\alpha>1}\left\\{\left(\frac{(2\alpha-1)}{2\alpha}\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X^{n})\right)\right)^{-\frac{\alpha}{\alpha-1}}\frac{(\alpha-1)}{(2\alpha-1)}\right\\}\right\\}.$
(83)

(a)

(b)

Figure 1: Comparison of various bounds for Example 2 with and without
(numerical) optimisation of parameters.

###  4.2 Noisy Bernoulli Bias

Assume, like in Section 4.1, that $W$ is uniform on the $[0,1]$ interval,
$X_{i}\sim\text{Ber}(W)$. In line with the discussion above, suppose that one
observes noisy copies of $X_{i}$’s denoted with $Z_{i}$’s, where $Z_{i}$ is
the outcome of $X_{i}$ after being passed through a Binary Symmetric Channel
with parameter $\lambda$ ($K$=BSC($\lambda$)). The purpose is to estimate $W$
through a function of $Z_{1},\ldots,Z_{n}$ i.e., $\hat{W}=\psi(Z^{n})$. One
thus has the following Markov Chain $W-X^{n}-Z^{n}-\hat{W}$. In order to
lower-bound the Bayesian risk in this setting, one can use Corollary 5. In
particular, given the additional injection of noise, it is to be expected that
one has a stronger impossibility result with respect to the non-noisy version.
This is reflected in the computations below. Let us restrict ourselves to
$\varphi$-Divergences as one can then leverage the results in the literature
on SDPI constants for the channel considered here. In particular, if one
considers Hellinger divergences then the following can be said [22, Corollary
3.1], [33]:

$\displaystyle\eta_{p}(K)$ $\displaystyle=(1-2\lambda)^{2}\text{ if }1\leq
p\leq 2$ (84) $\displaystyle\eta_{p}(K)$ $\displaystyle\leq|1-2\lambda|\hskip
6.99997pt\text{ if }p>2.$ (85)

Moreover, if $p\leq 2$ then one can leverage tensorisation properties of SDPI-
coefficients (see Section 2.2) and the following can be said

$\eta_{p}(\mathcal{P}_{X^{n}},K^{\otimes
n})=\eta_{p}(\mathcal{P}_{X},K)\leq\eta_{p}(K)=(1-2\lambda)^{2}\text{ if
}1\leq p\leq 2$ (86)

In this setting, one has that the Hellinger divergence
$\mathcal{H}_{p}(W,X^{n})$ is given in Equation 77. With $p=2$ and without
making any assumption on $\mathcal{P}_{\hat{W}|Z^{n}}$, one can leverage
Corollary 5 and retrieve the following closed-form expression for the Risk in
this setting:

$\displaystyle R^{\text{noisy}}$
$\displaystyle\geq\sup_{\rho>0}\rho\left(1-\sqrt{2\rho((1-2\lambda)^{2}\chi^{2}(W,X^{n})+1)}\right)$
(87) $\displaystyle=\frac{2}{27}\frac{1}{(1-2\lambda)^{2}\chi^{2}(W,X^{n})+1}$
(88) $\displaystyle\geq\frac{2}{27}\frac{1}{(1-2\lambda)^{2}\frac{16\sqrt{\pi
n}}{21}+1}.$ (89)

Clearly, the denominator in Equation 89 is smaller than the one in Equation 80
thus yielding a larger lower-bound on the risk, this is depicted in Figure 2
for the case $\lambda=0.25$.

Figure 2: Comparison of the lower bounds in Equation 80 and Equation 89 for
the noisy Bernoulli bias setting described in Section 4.2 with $\lambda=0.25$.
Solid lines mean that the corresponding lower-bound is the largest.

###  4.3 Gaussian prior with Gaussian noise (and absolute error)

Another classical and interesting setting is given by the following example:

######  Example 3.

Assume that $W\sim N(0,\sigma^{2}_{W})$ and that for $i\in[n]$,
$X_{i}=W+Z_{i}$ where $Z_{i}\sim N(0,\sigma^{2})$. Assume also that the loss
is s.t. $\ell(w,\hat{w})=|w-\hat{w}|$.

Using the sample mean estimator one has that:

$R_{B}\leq\sqrt{\frac{\sigma^{2}_{W}}{1+n\sigma^{2}_{W}/\sigma^{2}}}.$ (90)

Moreover, given that $\ell(w,\hat{w})=|w-\hat{w}|$ it is also possible to show
that:

$L_{W}(\rho)\leq\left(\sup_{w\in\mathbb{R}}\mathcal{P}_{W}(w)\right)\left(\int_{-\rho}^{\rho}1du\right)\leq\rho\sqrt{\frac{2}{\sigma^{2}_{W}\pi}}.$
(91)

In this setting, $\mathcal{L}\left(W\\!\\!\to\\!\\!X^{n}\right)$ is infinite.
However, $I_{\alpha}(W,X^{n})$ is finite for every $\alpha<+\infty$. One can
thus provide a lower bound on the Risk, resorting to $I_{\alpha}$ via Equation
33. Given that the empirical mean is a sufficient statistic for $W$ in this
case, one has that [19, Example 5]:

$I_{\alpha}(W,X^{n})=I_{\alpha}\left(W,\frac{1}{n}\sum_{i=1}^{n}X_{i}\right)=\frac{1}{2}\log\left(1+\alpha
n\frac{\sigma^{2}_{W}}{\sigma^{2}}\right).$ (92)

These considerations imply that :

$\displaystyle R_{B}$
$\displaystyle\geq\sup_{\alpha>1}\sup_{\rho>0}\rho\left(1-\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X^{n})\right)\left(\rho\sqrt{\frac{2}{\sigma^{2}_{W}\pi}}\right)^{\frac{\alpha-1}{\alpha}}\right)$
(93)
$\displaystyle=\sup_{\alpha>1}\sup_{\rho>0}\rho\left(1-\left(\rho\sqrt{\left(1+\alpha
n\frac{\sigma^{2}_{W}}{\sigma^{2}}\right)\frac{2}{\sigma^{2}_{W}\pi}}\right)^{\frac{\alpha-1}{\alpha}}\right)$
(94)
$\displaystyle=\sup_{\alpha>1}\frac{1}{(\beta+1)}\left(\frac{\beta}{\beta+1}\right)^{\beta}\left(\sqrt{\left(1+\alpha
n\frac{\sigma^{2}_{W}}{\sigma^{2}}\right)\frac{2}{\sigma^{2}_{W}\pi}}\right)^{-\frac{1}{\beta}},$
(95)

remembering that $\beta=\frac{\alpha}{\alpha-1}.$ Stepping away from Sibson’s
$\alpha$-Mutual Information one can look at Hellinger $p$-Divergences and
$E_{\gamma,\zeta}$ once again. In particular, one has that for $p>1$ (details
in Section C.3.1):

$\mathcal{H}_{p}(W,X)=\left(\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{p}}{1+(2-p)p\frac{\sigma_{W}^{2}}{\sigma^{2}}}\right)^{\frac{1}{2}}.$
(96)

Thus, the family of bounds provided by Corollary 3 can be expressed as follows

$\displaystyle R$
$\displaystyle\geq\sup_{p>1}\sup_{\rho>0}\rho\left(1-\left(\frac{2\rho}{\sqrt{2\pi\sigma_{W}^{2}}}\right)^{\frac{p-1}{p}}\mathcal{H}^{\frac{1}{p}}_{p}(W,X^{n})\right)$
(97)
$\displaystyle=\sup_{p>1}\sup_{\rho>0}\rho\left(1-\left(\frac{2\rho}{\sqrt{2\pi\sigma_{W}^{2}}}\right)^{\frac{p-1}{p}}\left(\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{p}}{1+(2-p)p\frac{\sigma_{W}^{2}}{\sigma^{2}}}\right)^{\frac{1}{2p}}\right)$
(98)
$\displaystyle=\sup_{p>1}\frac{1}{q+1}\left(\frac{q}{q+1}\right)^{q}\left(\left(\frac{2}{\sqrt{2\pi\sigma_{W}^{2}}}\right)^{\frac{p-1}{p}}\left(\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{p}}{1+(2-p)p\frac{\sigma_{W}^{2}}{\sigma^{2}}}\right)^{\frac{1}{2p}}\right)^{-\frac{1}{q}},$
(99)

where $q$ represents the Hölder’s conjugate with respect to $p$, i.e.,
$q=\frac{p}{p-1}$.

In particular, setting $p=3/2$ one obtains:

$\mathcal{H}_{3/2}(W,X)=\sqrt{\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{3}{2}}}{1+\frac{3\sigma_{W}^{2}}{4\sigma^{2}}}},$
(100)

leading us to a lower bound on the Bayesian risk given by:

$R_{B}\geq\frac{81\sqrt{2\pi}}{2048}\sqrt{\frac{\sigma_{W}^{2}}{1+n\frac{\sigma_{W}^{2}}{\sigma^{2}}}}.$
(101)

Similarly to the previous example, one has that Equation 101 matches the
upper-bound up to a constant factor, and provides a strengthening of the
bounds obtained in [1, Corollary 1]. Repeating the analysis with the
$E_{\gamma,\zeta}$-Divergence, one obtains the following:

$\displaystyle R_{B}$
$\displaystyle\geq\sup_{\rho>0}\rho\left(1-\frac{\left(E_{\gamma,\zeta}(W,X^{n})+\frac{2\rho\gamma}{\sqrt{2\sigma^{2}_{W}\pi}}\right)}{\zeta}\right)$
(102) $\displaystyle=\frac{\sqrt{2\sigma^{2}_{W}\pi}\left(\zeta-
E_{\gamma,\zeta}(W,X^{n})\right)^{2}}{8\gamma\zeta}.$ (103)

(a)

(b)

Figure 3: Comparison of various bounds for Example 3 with and without
(numerical) optimisation of parameters.

Like in Example 2, one can numerically evaluate Equation 103 and compare it
with Equation 95, Equation 101 and [1, Equation (16)]. Figure 3(a) and Figure
3(b) show the resulting lower bounds as a function of the number of samples
$n$. One can observe similar behaviors when comparing with the results from
previous example: the bounds retrieved through the $\mathcal{H}_{p}$\- and
$E_{\gamma,\zeta}$-Divergences are both able to improve on the lower-bound
relying on Shannon’s Mutual Information. Once again, $E_{\gamma,\zeta}$, (cf.
Equation 103) provides the largest lower-bound, while Sibson’s $\alpha$-Mutual
Information is still able to provide a stronger result than Equation 99.
Similarly to before, one can also numerically optimise the bounds with respect
to the corresponding parameters $\alpha>1,p>1,\zeta>0$ and $\gamma\geq 0$ and
the resulting comparison is depicted in Figure 3(b).

###  4.4 “Hide-and-seek” problem

To conclude, let us consider next a $d$-dimensional distributed estimation
problem, known as the “Hide-and-seek” problem. It has been first presented in
[2] and also studied in [1].

######  Example 4.

Consider a family of distributions
$\mathcal{P}=\\{\mathcal{P}_{w}:w=1,\ldots,d\\}$ on $\\{0,1\\}^{d}$. Under
$\mathcal{P}_{w}$, the $w$-th coordinate of the random vector
$X\in\\{0,1\\}^{d}$ has bias $\frac{1}{2}+\theta$ while the other coordinates
of $X$ are independently drawn from Ber$(1/2)$. For $i=1,\ldots,m$, the $i$-th
processor observes $n$ samples $X^{n}_{i}$ drawn independently from
$\mathcal{P}_{W}$, and sends a $b$-bits message
$Y_{i}=\varphi(X_{i}^{n},Y^{i-1})$ to the estimator. The estimator computes
$\hat{W}=\psi(Y^{m})$ from the received messages. The risk in this example is
defined as:

$R_{M}=\inf_{\varphi^{m},\psi}\max_{w\in[d]}\mathbb{P}[W\neq\hat{W}].$ (104)

A lower-bound for $R_{M}$ derived in [2] is as follows:

$R_{M}\geq 1-\left(\frac{3}{d}+5\sqrt{\min\left\\{\frac{10\theta
nmb}{d},mn\theta^{2}\right\\}}\right)$ (105)

and only holds for $0\leq\theta\leq 1/(4n).$ Additionally, in [1] a quite
different lower-bound has been proposed:

$R_{M}\geq 1-\frac{1}{\log
d}\min\left\\{\left[1-\left(\frac{1-2\theta}{1+2\theta}\right)^{n}\right]mb+1,\min(4mn\theta^{2},\log
d)+1\right\\},$ (106)

and it holds for $0\leq\theta\leq 1/2.$ Let us now use a naïve approach with
Maximal Leakage. We have that $W-X^{n\times m}-Y^{m}-\hat{W}$ forms a Markov
Chain. Thus,

$\mathcal{L}\left(W\\!\\!\to\\!\\!\hat{W}\right)\leq\min\left\\{\mathcal{L}\left(W\\!\\!\to\\!\\!X^{n\times
m}\right),\mathcal{L}\left(W\\!\\!\to\\!\\!Y^{m}\right)\right\\}.$

We also have that $\mathcal{L}\left(W\\!\\!\to\\!\\!Y^{m}\right)\leq mb$ and
that:

$\displaystyle\mathcal{L}(W\to X^{n\times m})$ $\displaystyle\leq
nm\mathcal{L}(W\to X)$ (107)
$\displaystyle=nm\log\sum_{x}\max_{w}\mathcal{P}_{X|W=w}(x)$ (108)
$\displaystyle\leq
nm\log\sum_{x}\left(\frac{1}{2}\right)^{d-1}\left(\frac{1}{2}+\theta\right)$
(109) $\displaystyle=nm\log(2^{d}(2^{-d}+2^{-d+1}\theta))$ (110)
$\displaystyle=nm\log(1+2\theta),$ (111)

Hence:

$\mathcal{L}\left(W\\!\\!\to\\!\\!\hat{W}\right)\leq\min(nm\log(1+2\theta),\log
d,mb).$ (112)

Using Equation 112 in Equation 34 we get the following:

$\mathbb{P}(\\{\hat{W}\neq W\\})\geq 1-\frac{\exp(\text{min}\\{mb,\log
d,nm\log(1+2\theta)\\})}{d}.$ (113)

Notice that Equation 113 is such that the right-hand side is always greater or
equal than $0$. Indeed, assuming $d$ to be fixed and letting $n$ and $m$ grow,
we have that the minimum is achieved by $\log d$, and in that case, we have
$\mathbb{P}(\\{\hat{W}\neq W\\})\geq 0$. Here, the difference in behaviour of
Equation 34 with respect to [1, Theorem 1] is pivotal. Let us now compare the
results on a common setting. The setting chosen in [1], where
$d=512,b=3d,m=10$ and $\theta=1/(4n)$ does not represent a choice of
parameters where Equation 113 is interesting. Indeed, for large enough $n$,
$nm\log(1+2\theta)=nm\log(1+1/2n)\approx m/2$ and, as a consequence, the
expression will converge to a constant determined by the minimum between
$mb,\log d,m/2$. Furthermore, both Equation 105 and Equation 106 have a term
that depends on $mn\theta^{2}$ which, for $\theta=1/(4n)$, will decay with
$n$. Thus, choosing $\theta\sim n^{-p}$ with $p>1$ represents an interesting
setting for the bound in Equation 113, as the plots in Figure 4(a) and Figure
4(b) show.

(a)

(b)

(c)

(d)

Figure 4: Behaviour of Equation 113 for various values of $\rho$. Solid lines
mean that the corresponding lower-bound is the largest.

Thanks to the different behaviour of Equation 113 (reaching $1$ exponentially
fast) we can see a much sharper jump towards $1$ with respect to Equation 106,
which instead plateaus below $1$, and with respect to Equation 105 that
reaches $1$ more slowly. The growth towards $1$ of Equation 113 becomes even
sharper with larger $p$ and converges towards a specific behaviour at
$p\approx 2$. Increasing $p$ any further does not alter the behaviour of the
bound meaningfully. As for the behaviour of the bound for fixed $\theta$, if
$\theta=0.01$. then Equation 113 provides a larger lower-bound only for
$n<25$. If the parameter is brought down to $\theta=0.0001$ then Equation 113
is larger than Equation 106 for all $n$ but only larger than Equation 105 for
$n<1850$. Regardless of the considerations related to the specific settings,
it is interesting how a very simple application of Equation 34 can provide a
tighter lower-bound. Moreover, in the proof of Equation 106 in [1], in order
to compute $I(W;X)$ an assumption on the distribution of $W$ was necessary and
the choice fell on $W$ uniform on $[d]$. With Maximal Leakage,
$\mathcal{L}\left(W\\!\\!\to\\!\\!X\right)$ does not depend on the specific
distribution over $W$, rendering the bound potentially more general. Other
divergences could be explored in this setting as well. However, one in general
does not have a chain rule for any other $\varphi$-Divergence (or Sibson’s
$\alpha$-Mutual Information with $\alpha<+\infty$) which is a fundamental step
in the proof for Maximal Leakage (cf. Equation 107). Moreover, some assumption
(or maximisation over) $\mathcal{P}_{W}$ would be necessary. In general, some
additional machinery would be required in order to employ them in this
setting. Given that this is outside of the scope of this work, these
approaches will not be explored in this document.

##  Appendix A Comparison with similar approaches

An approach closely connected to the one proposed in here is in [12]. The
authors therein focused on the notion of $\varphi$-informativity [13] and
leveraged the Data-Processing inequality of the information measure. In
particular, $\varphi$-informativities can potentially lead to tighter results
than the $\varphi$-Mutual Information considered in this work. Similarly to
Sibson’s $\alpha$-Mutual Information, they are defined as follows:

$\hat{I}_{\varphi}(X,Y)=\inf_{\mathcal{Q}_{Y}}D_{\varphi}(\mathcal{P}_{XY}\|\mathcal{P}_{X}\mathcal{Q}_{Y})\leq
I_{\varphi}(X,Y).$ (114)

Given that the minimum-achieving distribution, $\mathcal{Q}_{Y}^{\star}$, is
guaranteed to exist in Equation 114 (see [13]), one can see that
$\hat{I}_{\varphi}(X,Y)=D_{\varphi}(\mathcal{P}_{XY}\|\mathcal{P}_{X}\mathcal{Q}_{Y}^{\star})$.
Consequently, the same steps followed in the proof of Theorem 4 can be
undertaken in order to reach a similar result involving $\hat{I}_{\varphi}$
and $\mathcal{P}_{X}\mathcal{Q}_{Y}^{\star}$ rather than $I_{\varphi}$ and
$\mathcal{P}_{X}\mathcal{P}_{Y}$. However, except in some specific settings,
the minimum-achieving distribution in Equation 114 does not necessarily admit
a closed-form expression [13]. As a consequence, the corresponding
$\varphi$-Informativity does not admit a closed-form expression. Moreover,
another step the authors leveraged in order to achieve [12, Theorem 3.2], is
the inversion of the resulting binary divergence, leading to a bound which can
rarely be expressed in closed form and can only be computed numerically. While
a direct comparison between the two approaches would be hard, some
similarities are present and hint at the fact that [12, Theorem 3.2] is
tighter than Theorem 4. Indeed, an alternative proof for Theorem 4 also stems
from leveraging the DPI of $I_{\varphi}$ (cf. [14, Theorem 3]). However,
additional steps are introduced in order to get a closed-form lower bound. Our
analysis is designed to retrieve a large family of results which are amenable
to analysis and interpretable. This allows us to retrieve lower bounds in
closed-form expressions that can be seen to match the upper bounds, up to a
constant, in a variety of settings. From a more conceptual standpoint, one
could see [14, Theorem 3] (and, consequently, Theorem 4) as a generalisation
of Hölder’s444Selecting $\varphi(x)=x^{p}$ specialises Theorem 4 to Corollary
3 which can also be proven as an application of Hölder’s inequality followed
by Markov’s inequality, cf. [14, Corollary 6]. inequality to arbitrary convex
functionals. This generalisation, which in turn can be seen as a
generalisation of Fano’s inequality for $\varphi$-Mutual Information, allows
us to also encompass divergences from the Rényi’s family and Sibson’s
$\alpha$-Mutual Information, which are not $\varphi$-Divergences and are thus
excluded from [12]. To conclude, let us highlight that our approach, which
leverages duality, allows us to provide a single analysis for every type of
loss and does not require a separate treatise for $0-1$ losses and more
general losses. Consequently, the two approaches for general losses are
different and hard to compare.

##  Appendix B Proof of Section 3

###  B.1 Proof of Theorem 3

###### Proof.

We have that

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W})<\rho)$
$\displaystyle\leq\left(\sup_{\hat{w}\in\mathcal{\hat{W}}}\mathcal{P}_{W}(\ell(W,\hat{w})<\rho)\right)^{\frac{\alpha-1}{\alpha}}\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,\hat{W})\right)$
(115)
$\displaystyle=\exp\left(\frac{\alpha-1}{\alpha}\left(I_{\alpha}(W,\hat{W})+\log(L_{W}(\rho))\right)\right)$
(116)
$\displaystyle\leq\exp\left(\frac{\alpha-1}{\alpha}\left(I_{\alpha}(W,X)+\log(L_{W}(\rho))\right)\right).$
(117)

Equation 115 follows from Corollary 2, Equation 117 follows from the Data-
Processing Inequality for $I_{\alpha}$ and the Markov Chain $W-X-Y-\hat{W}$.
Moreover, using Markov’s inequality one has that

$\displaystyle R_{B}$
$\displaystyle\geq\rho\cdot\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W})\geq\rho)$
(118)
$\displaystyle=\rho\cdot(1-\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W})<\rho)).$
(119)

The statement follows from lower bounding Equation 119 using Equation 117. ∎

###  B.2 Proof of Theorem 4

###### Proof.

From the variational representation for $\varphi$-Divergences (see Equation
27), given $\mathcal{P}_{W\hat{W}}$, for every function $f$ in the respective
space (defined in Theorem 1) one has that:

$I_{\varphi}(W,\hat{W})=D_{\varphi}(\mathcal{P}_{W\hat{W}}\|\mathcal{P}_{W}\mathcal{P}_{\hat{W}})\geq\mathcal{P}_{W\hat{W}}(f)-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\varphi^{\star}(f)).$
(120)

Equation 120 allows us to relate the expected value of any function
$f:\mathcal{W}\times\mathcal{\hat{W}}$ under the joint with
$I_{\varphi}(W,\hat{W})$ and the corresponding Legendre-Fenchel dual. Our
purpose is to provide a lower-bound on the expected loss $\ell$ hence we will
select $f=\tilde{\lambda}(\rho-\ell)$ with $\rho,\tilde{\lambda}>0$. Moreover,
given the non-negativity of $\ell$ one can also see that
$\ell\geq\rho\mathbbm{1}_{\\{\ell\geq\rho\\}}$ (i.e., Markov’s Inequality in
its functional form). Hence, plugging our choice of $f$ in Equation 120 the
following chain of inequality follows:

$\displaystyle\tilde{\lambda}\mathcal{P}_{W\hat{W}}(\ell)$
$\displaystyle\geq\tilde{\lambda}\rho-
I_{\varphi}(W,\hat{W})-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\varphi^{\star}(\tilde{\lambda}(\rho-\ell))$
(121) $\displaystyle\geq\tilde{\lambda}\rho-
I_{\varphi}(W,\hat{W})-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\varphi^{\star}(\tilde{\lambda}\rho(1-\mathbbm{1}_{\\{\ell\geq\rho\\}})))$
(122) $\displaystyle=\tilde{\lambda}\rho-
I_{\varphi}(W,\hat{W})-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\varphi^{\star}(\tilde{\lambda}\rho\mathbbm{1}_{\\{\ell<\rho\\}}))$
(123) $\displaystyle=\tilde{\lambda}\rho-
I_{\varphi}(W,\hat{W})-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})\cdot\varphi^{\star}(\tilde{\lambda}\rho)-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell\geq\rho\\})\cdot\varphi^{\star}(0),$
(124)

where Equation 122 follows by the monotonicity of $\varphi^{\star}$ which can
be seen as stemming from the strict convexity and the monotonicity of
$\varphi$. Indeed, if $\varphi$ is strictly convex then
${\varphi^{\star}}^{\prime}(t)={\varphi^{\prime}}^{-1}(t)$ for every
$t\in\text{Im}(\varphi^{\prime})$ [34, Theorem 26.5]. Since $\varphi$ is
monotone non-decreasing on the positive axis, one has that
$\varphi^{\prime}(t)\geq 0$ on $[0,+\infty]$. Accordingly, the inverse of
$\varphi^{\prime}$ will also be non-negative on $[0,+\infty]$, which implies
the non-negativity of ${\varphi^{\star}}^{\prime}$ and, therefore, the
monotonicity of $\varphi^{\star}$. A similar argument shows the monotonicity
of $\varphi^{\star}$ when $\varphi$ is non-increasing. Then, dividing both
sides by $\tilde{\lambda}$ and selecting
$\tilde{\lambda}=\frac{1}{\rho}\lambda$ with $\lambda>0$ one recovers the
following:

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell)$
$\displaystyle\geq\sup_{\lambda>0}\rho\left(1-\frac{I_{\varphi}(W,\hat{W})+\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell\geq\rho\\})\cdot\varphi^{\star}(0)+\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})\cdot\varphi^{\star}(\lambda)}{\lambda}\right)$
(125)
$\displaystyle=\rho\left(1-\inf_{\lambda>0}\frac{I_{\varphi}(W,\hat{W})+\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell\geq\rho\\})\cdot\varphi^{\star}(0)+\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})\cdot\varphi^{\star}(\lambda)}{\lambda}\right)$
(126)
$\displaystyle=\rho\left(1-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})\inf_{\lambda>0}\frac{\frac{I_{\varphi}(W,\hat{W})+\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell\geq\rho\\})\cdot\varphi^{\star}(0)}{\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})}+\varphi^{\star}(\lambda)}{\lambda}\right)$
(127)
$\displaystyle=\rho\left(1-\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})\varphi^{-1}\left(\frac{I_{\varphi}(W,\hat{W})+\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell\geq\rho\\})\cdot\varphi^{\star}(0)}{\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(\\{\ell<\rho\\})}\right)\right),$
(128)

where Equation 128 follows from the same argument as in [29, Theorem 13].
Equation 122 is the step of the proof which is relevant with respect to Remark
2. In particular, the choice of $f$, along with the non-decreasability of
$\varphi$ allowed us to leverage the functional form of Markov’s inequality
and, consequently, to upper-bound the dual of
$D_{\varphi}(\cdot\|\mathcal{P}_{W}\mathcal{P}_{\hat{W}})$. Upper-bounding the
dual is crucial in order to achieve a bound of the form of Equation 128. In
order to prove the result for $\varphi$ non-increasing one has to select
$f=-\tilde{\lambda}\ell$ leverage Markov’s inequality and select
$\tilde{\lambda}=-\frac{1}{\rho}\lambda$ with $\lambda<0$. The result then
follows from the same argument as in [29, Theorem 13] i.e., from selecting
$\lambda={\varphi^{\prime}}(\varphi^{-1}(c))$ with
$c=\frac{I_{\varphi}(W,\hat{W})+\varphi^{\star}(0)\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(E)}{\mathcal{P}_{W}\mathcal{P}_{\hat{W}}(E^{c})}$
and $E=\\{\ell<\rho\\}$. ∎

###  B.3 Proof of Corollary 3

###### Proof.

The statement follows from Theorem 4 with $\varphi(x)=\frac{x^{p}-1}{p-1}$.
Hence, for every estimator $\hat{W}=\phi(X^{n})$,

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W})\geq\rho)$
$\displaystyle\leq
L_{W}(\hat{W},\rho)\cdot\varphi^{-1}\left(\frac{I_{\varphi}(W,\hat{W})+(1-L_{W}(\hat{W},\rho))\cdot\varphi^{\star}(0)}{L_{W}(\hat{W},\rho)}\right)$
(129) $\displaystyle\leq
L_{W}(\rho)^{\frac{p-1}{p}}\left((p-1)\mathcal{H}_{p}(W,X)+1\right)^{\frac{1}{p}},$
(130)

where Equation 130 follows from the data-processing inequality for
$\varphi$-divergences. One thus retrieves that for every estimator $\hat{W}$

$\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\rho\left(1-L_{W}(\rho)^{\frac{p-1}{p}}\left((p-1)\mathcal{H}_{p}(W,X)+1\right)^{\frac{1}{p}}\right).$
(131)

Since the right-hand side of Equation 131 is independent of $\hat{W}=\phi(X)$
one can use it to lower-bound the risk $R$. ∎

###  B.4 Proof of Corollary 4

###### Proof.

Let $\varphi(x)=\max\\{0,\zeta x-\gamma\\}-\max\\{0,\zeta-\gamma\\}$ in
Theorem 4, along with the fact that
$\varphi^{-1}(y)=\frac{y+\max\\{0,\zeta-\gamma\\}+\gamma}{\zeta}$ for $y>0$
and $\varphi^{\star}(0)=\max\\{0,\zeta-\gamma\\}$ one has that for every
estimator $\hat{W}=\phi(X^{n})$,

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))$
$\displaystyle\geq\rho\left(1-\frac{E_{\gamma,\zeta}(W,\hat{W})+\gamma
L_{W}(\hat{W},\rho)+\max\\{0,\zeta-\gamma\\}}{\zeta}\right)$ (132)
$\displaystyle\geq\rho\left(1-\frac{E_{\gamma,\zeta}(W,X)+\gamma
L_{W}(\rho)+\max\\{0,\zeta-\gamma\\}}{\zeta}\right).$ (133)

Since Equation 133 is independent of $\hat{W}=\phi(X)$ one can use it to
lower-bound the risk $R$. ∎

##  Appendix C Computations for Section 4

###  C.1 Maximisation over $\rho$

The bounds considered in the first three examples have the following form

$\sup_{\rho>0}\rho(1-c\rho^{t}-b),$ (134)

for some $c,t,b\geq 0$. Letting $h(\rho):=\rho(1-c\rho^{t}-b)$, the optimal
value $\rho_{\star}$ is found by setting $h^{\prime}(\rho_{\star})=0$, which
yields

$1-(t+1)c\rho_{\star}^{t}-b=0\iff\rho_{\star}=\left(\frac{1-b}{(t+1)c}\right)^{\frac{1}{t}}.$
(135)

Since $h^{\prime\prime}(\rho_{\star})=-t(t+1)c\rho_{\star}^{t-1}\leq 0$, this
ensures $\rho_{\star}$ is a maximum. Substituting $\rho^{\star}$ back in
Equation 134, we can express the lower bound as

$\sup_{\rho>0}\rho(1-c\rho^{t}-b)=\frac{t}{c^{\frac{1}{t}}}\left(\frac{1-b}{t+1}\right)^{1+\frac{1}{t}}.$
(136)

###  C.2  Section 4.1

####  C.2.1 Maximal Leakage

In this setting one has that

$\mathcal{P}_{X^{n}|W=w}(x^{n})=w^{k}(1-w)^{(n-k)}$

where $k=\sum_{i=1}^{n}$ is the hamming weight of $x^{n}$. As per assumption,
$\mathcal{P}_{W}(w)=1$ if $0\leq w\leq 1$ and, consequently, one has that

$\mathcal{P}_{W|X^{n}=x^{n}}(w)=(n+1)\binom{n}{k}(1-w)^{n-k}w^{k}.$

One can thus compute Maximal Leakage in this setting:

$\displaystyle\mathcal{L}\left(W\\!\\!\to\\!\\!X^{n}\right)$
$\displaystyle=\log\sum_{x^{n}}\max_{w}\mathcal{P}_{X^{n}|W=w}(x^{n})$ (137)
$\displaystyle=\log\sum_{k=0}^{n}\binom{n}{k}\max_{w}w^{k}(1-w)^{n-k}$ (138)
$\displaystyle=\log\sum_{k=0}^{n}\binom{n}{k}\left(\frac{k}{n}\right)^{k}\left(1-\frac{k}{n}\right)^{n-k}$
(139) $\displaystyle\leq\log\left(2+\sum_{k=1}^{n-1}\sqrt{\frac{n}{2\pi
k(n-k)}}\right)$ (140) $\displaystyle\leq\log\left(2+\sqrt{\frac{\pi
n}{2}}\right).$ (141)

####  C.2.2 Sibson’s $\alpha$-Mutual Information

For Sibson’s $\alpha$-Mutual Information with $\alpha>1$, one has that:

$\displaystyle\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X^{n}))\right)$
$\displaystyle=\mathbb{E}\left[\mathbb{E}^{\frac{1}{\alpha}}\left[\left(\frac{\mathcal{P}_{X^{n}|W}}{\mathcal{P}_{X^{n}}}\right)^{\alpha}\bigg{|}X^{n}\right]\right]$
(142)
$\displaystyle=\sum_{x^{n}}\mathcal{P}_{X^{n}}(x^{n})\left(\int_{0}^{1}\mathcal{P}_{W}(w)\left(\frac{\mathcal{P}_{W|X^{n}=x^{n}}(w)}{\mathcal{P}_{W}(w)}\right)^{\alpha}dw\right)^{\frac{1}{\alpha}}$
(143)
$\displaystyle=\sum_{x^{n}}\mathcal{P}_{X^{n}}(x^{n})\left(\int_{0}^{1}\left(\mathcal{P}_{X^{n}|W=w}(x^{n})\right)^{\alpha}dw\right)^{\frac{1}{\alpha}}$
(144)
$\displaystyle=\sum_{k=0}^{n}\binom{n}{k}\frac{1}{(n+1)\binom{n}{k}}\left(\int_{0}^{1}\left((n+1)\binom{n}{k}w^{k}(1-w)^{n-k}\right)^{\alpha}dw\right)^{\frac{1}{\alpha}}$
(145)
$\displaystyle=\sum_{k=0}^{n}\binom{n}{k}\left(\int_{0}^{1}\left(w^{k}(1-w)^{n-k}\right)^{\alpha}dw\right)^{\frac{1}{\alpha}}$
(146)
$\displaystyle=\sum_{k=0}^{n}\binom{n}{k}\left(\frac{\Gamma(k\alpha+1)\Gamma((n-k)\alpha+1)}{\Gamma(n\alpha+2)}\right)^{\frac{1}{\alpha}},$
(147)

where Equation 147 uses the identity relating the Beta function with the Gamma
function i.e.,

$\mathrm{Beta}(x,y)=\int_{0}^{1}w^{x-1}(1-w)^{y-1}dw=\frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)},$
(148)

so that

$\int_{0}^{1}w^{k\alpha}(1-w)^{(n-k)\alpha}dw=\frac{\Gamma(k\alpha+1)\Gamma((n-k)\alpha+1)}{\Gamma(n\alpha+2)}.$
(149)

####  C.2.3 Hellinger $p$

For the Hellinger$-p$ divergence with $p>1$, one has that:

$\displaystyle((p-1)\mathcal{H}_{p}(W,X^{n})+1)$
$\displaystyle=\left\lVert\frac{d\mathcal{P}_{WX^{n}}}{d\mathcal{P}_{W}\mathcal{P}_{X^{n}}}\right\rVert_{L_{p}(\mathcal{P}_{W}\mathcal{P}_{X^{n}})}^{p}$
(150)
$\displaystyle=\sum_{x^{n}}\mathcal{P}_{X^{n}}(x^{n})\int_{0}^{1}\mathcal{P}_{W}(w)\left(\frac{\mathcal{P}_{W|X^{n}=x^{n}}(w)}{\mathcal{P}_{W}(w)}\right)^{p}dw$
(151)
$\displaystyle=\sum_{x^{n}}\mathcal{P}_{X^{n}}(x^{n})\int_{0}^{1}\left(\mathcal{P}_{X^{n}|W=w}(x^{n})\right)^{p}dw$
(152)
$\displaystyle=\sum_{k=0}^{n}\binom{n}{k}\frac{1}{(n+1)\binom{n}{k}}\int_{0}^{1}\left((n+1)\binom{n}{k}w^{k}(1-w)^{n-k}\right)^{p}dw$
(153)
$\displaystyle=(n+1)^{p-1}\sum_{k=0}^{n}\binom{n}{k}^{p}\int_{0}^{1}\left(w^{k}(1-w)^{(n-k)}\right)^{p}dw$
(154)
$\displaystyle=(n+1)^{p-1}\sum_{k=0}^{n}\binom{n}{k}^{p}\frac{\Gamma(kp+1)\Gamma((n-k)p+1)}{\Gamma(np+2)},$
(155)

where Equation 155 follows from Equation 149. For the special case $p=2$, we
get

$\displaystyle\chi^{2}(W,X^{n})+1$
$\displaystyle=(n+1)\sum_{k=0}^{n}\binom{n}{k}^{2}\frac{(2k)!(2(n-k))!}{(2n+1)!}$
(156)
$\displaystyle=\frac{n+1}{(2n+1)}\sum_{k=0}^{n}\frac{(n!)^{2}(2k)!(2(n-k))!}{(k!)^{2}((n-k)!)^{2}(2n)!}$
(157)
$\displaystyle=\frac{n+1}{(2n+1)\binom{2n}{n}}\sum_{k=0}^{n}\binom{2k}{k}\binom{2(n-k)}{n-k}$
(158) $\displaystyle=\frac{n+1}{2n+1}\cdot\frac{4^{n}}{\binom{2n}{n}},$ (159)

where in Equation 159 we use the result in [35, Eq. (5.39), p.187] stating
that $\sum_{k=0}^{n}\binom{2k}{k}\binom{2(n-k)}{n-k}=4^{n}$.

###  C.3  Section 4.3

####  C.3.1 Hellinger $p$

For the Hellinger$-p$ divergence with $p>1$, one has that:

$\displaystyle((p-1)\mathcal{H}_{p}(W,X^{n})+1)$
$\displaystyle=\left\lVert\frac{d\mathcal{P}_{WX^{n}}}{d\mathcal{P}_{W}\mathcal{P}_{X^{n}}}\right\rVert_{L_{p}(\mathcal{P}_{W}\mathcal{P}_{X^{n}})}^{p}$
(160)
$\displaystyle=\int_{\mathbb{R}}\int_{\mathbb{R}}\mathcal{P}_{W}(w)\mathcal{P}_{X}(x)\left(\frac{\mathcal{P}_{X|W=w}(x)}{\mathcal{P}_{X}(x)}\right)^{p}dwdx$
(161)
$\displaystyle=\int_{\mathbb{R}}\mathcal{P}_{X}(x)^{1-p}\int_{\mathbb{R}}\mathcal{P}_{W}(w)\mathcal{P}_{X|W=w}(x)^{p}dwdx.$
(162)

Focusing on the innermost integral (which we denote as $I_{p}(x)$), one has

$\displaystyle I_{p}(x)$
$\displaystyle\vcentcolon=\int_{\mathbb{R}}\mathcal{P}_{{W}}({w})\mathcal{P}_{{X}|{W}={w}}({x})^{p}d{w}$
(163)
$\displaystyle=\left(\frac{(2\pi\sigma^{2})^{-p}}{2\pi\sigma_{W}^{2}}\right)^{\frac{1}{2}}\int_{\mathbb{R}}e^{-\frac{w^{2}}{2\sigma_{W}^{2}}-\frac{p(w-x)^{2}}{2\sigma^{2}}}d{w}$
(164)
$\displaystyle=\left(\frac{(2\pi\sigma^{2})^{-p}}{2\pi\sigma_{W}^{2}}\right)^{\frac{1}{2}}\int_{\mathbb{R}}e^{-\frac{1}{2\sigma^{2}}\left(px^{2}-2p{x}{w}+\left(\frac{\sigma^{2}}{\sigma_{W}^{2}}+p\right)w^{2}\right)}d{w}$
(165)
$\displaystyle=\left(\frac{(2\pi\sigma^{2})^{-p}}{2\pi\sigma_{W}^{2}}\right)^{\frac{1}{2}}e^{\frac{-p\cdot
x^{2}}{2\sigma^{2}}}\int_{\mathbb{R}}e^{-\frac{1}{2\sigma^{2}}\left(-2p{x}{w}+\left(\frac{\sigma^{2}}{\sigma_{W}^{2}}+p\right)w^{2}\right)}d{w}.$
(166)

Adding and subtracting $cx^{2}$ with
$c=-p\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{-1}$ in the exponent
inside the integral in Equation 166 leads to

$\displaystyle I_{p}(x)$
$\displaystyle=\left(\frac{(2\pi\sigma^{2})^{-p}}{2\pi\sigma_{W}^{2}}\right)^{\frac{1}{2}}e^{{\frac{c{x}^{2}}{2\sigma^{2}}}}\int_{\mathbb{R}}e^{-\frac{\frac{\sigma^{2}}{\sigma_{W}^{2}}+p}{2\sigma^{2}}\left({w}-\sqrt{\frac{p+c}{\frac{\sigma^{2}}{\sigma_{W}^{2}}+p}}{x}\right)^{2}}d{w}$
(167)
$\displaystyle=\left(\frac{(2\pi\sigma^{2})^{-p}}{2\pi\sigma_{W}^{2}}\right)^{\frac{1}{2}}\exp\left(-{\frac{p{x}^{2}}{2\sigma^{2}\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)}}\right)\left(2\pi\frac{\sigma^{2}}{\frac{\sigma^{2}}{\sigma_{W}^{2}}+p}\right)^{\frac{1}{2}}$
(168)
$\displaystyle=\left(2\pi\sigma^{2}\right)^{-\frac{p}{2}}\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{-\frac{1}{2}}\exp\left(-{\frac{p{x}^{2}}{2\sigma^{2}\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)}}\right).$
(169)

Finally, plugging the value of $I_{p}$ back in (162), we retrieve that:

$\displaystyle(p-1)\mathcal{H}_{p}(W,X)+1$
$\displaystyle=\int_{\mathbb{R}}\mathcal{P}_{{X}}({x})^{1-p}\frac{1}{(2\pi\sigma^{2})^{\frac{p}{2}}}e^{-\frac{p{x}^{2}}{2\left(\sigma^{2}+p\sigma_{W}^{2}\right)}}\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{-\frac{1}{2}}d{x}$
(170)
$\displaystyle=\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{d(p-1)}{2}}}{(2\pi\sigma^{2})^{\frac{1}{2}}\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{1}{2}}}\int_{\mathbb{R}}e^{\frac{(p-1){x}^{2}}{2\left(\sigma^{2}+\sigma_{W}^{2}\right)}-\frac{p{x}^{2}}{2\left(\sigma^{2}+p\sigma_{W}^{2}\right)}}d{x}$
(171)
$\displaystyle=\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{(p-1)}{2}}}{(2\pi\sigma^{2})^{\frac{1}{2}}\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{1}{2}}}\int_{\mathbb{R}}e^{-\frac{{x}^{2}}{2}\left(\frac{1-p}{\sigma^{2}+\sigma_{W}^{2}}+\frac{p}{\sigma^{2}+p\sigma_{W}^{2}}\right)}d{x}$
(172)
$\displaystyle=\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{(p-1)}{2}}}{(2\pi\sigma^{2})^{\frac{1}{2}}\left(1+p\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{1}{2}}}\left(\frac{2\pi}{\frac{1-p}{\sigma^{2}+\sigma_{W}^{2}}+\frac{p}{\sigma^{2}+p\sigma_{W}^{2}}}\right)^{\frac{1}{2}}$
(173)
$\displaystyle=\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{\frac{(p-1)}{2}}}{\left(\sigma^{2}+p\sigma_{W}^{2}\right)^{\frac{1}{2}}}\left(\frac{1}{\frac{1-p}{\sigma^{2}+\sigma_{W}^{2}}+\frac{p}{\sigma^{2}+p\sigma_{W}^{2}}}\right)^{\frac{1}{2}}$
(174)
$\displaystyle=\left(\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{p-1}}{\frac{(1-p)\left(\sigma^{2}+p\sigma_{W}^{2}\right)}{\sigma^{2}+\sigma_{W}^{2}}+p}\right)^{\frac{1}{2}}$
(175)
$\displaystyle=\left(\frac{\left(1+\frac{\sigma_{W}^{2}}{\sigma^{2}}\right)^{p}}{1+(2-p)p\frac{\sigma_{W}^{2}}{\sigma^{2}}}\right)^{\frac{1}{2}}.$
(176)

##  Appendix D Other approaches

###  D.1 Inverting the roles

The Sibson’s $\alpha$-mutual information is an asymmetric quantity. A natural
question is: can one provide a result similar to Theorem 3 involving
$I_{\alpha}(X,W)$ instead? Indeed, by inverting the roles of $W$ and
$\hat{W},$ such a bound can be given but it will involve the small ball
probability for $\hat{W}$ i.e.,

$\displaystyle L_{\hat{W}}(\rho)$
$\displaystyle=\sup_{w}\mathcal{P}_{\hat{W}}(\ell(w,\hat{W})\geq\rho).$ (177)

This quantity hinges on the marginal distribution of $\hat{W},$ which, in
turn, depends on the estimator used. In terms of $L_{\hat{W}}(\rho),$ one can
give the following general bound:

######  Lemma 3.

Consider the Bayesian framework described in Section 1.3. The following holds
for every $\alpha>1$ and $\rho>0$:

$R_{B}\geq\rho\left(1-\exp\left(\frac{\alpha-1}{\alpha}\left(I_{\alpha}(X,W)+\log(L_{\hat{W}}(\rho))\right)\right)\right).$
(178)

Moreover, taking the limit of $\alpha\to\infty$ one has:

$R_{B}\geq\rho\left(1-\exp\left(\mathcal{L}\left(X\\!\\!\to\\!\\!W\right)+\log(L_{\hat{W}}(\rho))\right)\right).$
(179)

To apply this lemma in concrete cases, one needs to compute or upper bound the
small ball probability $L_{\hat{W}}(\rho).$ Leveraging basic properties of the
estimator, one can sometimes bound it. For example, if the estimator is a
linear function of the noisy observations one can leverage results related to
Lévy’s concentration functions of sums of independent random variables. E.g.,
if $Y_{1},\ldots,Y_{m}$ are uncorrelated and have log-concave distributions,
then for every $\rho\geq 0$ [36, Theorem 1.1],

$L_{\sum_{i}^{m}Y_{i}}(\rho)\leq\frac{2\rho}{\sqrt{\text{Var}(\sum_{i=1}^{m}Y_{i})+\rho^{2}/3}}=\frac{2\rho}{\sqrt{m\text{Var}(Y_{1})+\rho^{2}/3}}.$
(180)

More general statements can be made, assuming
$\phi(Y^{m})=\sum_{i=1}^{m}a_{i}Y_{i}$ under different constraints over
$a_{i}$ [37]. To appreciate the promise of this approach, let us also discuss
the behaviors of $I_{\alpha}(W,X)$ and $I_{\alpha}(X,W)$. More specifically,
let us consider again the “Hide-and-Seek” problem. Assuming, as in [1, Example
12], that $\mathcal{P}_{W}$ is uniform over $[d]$, one has that

$\mathcal{L}\left(X^{n\times
m}\\!\\!\to\\!\\!W\right)=\log\frac{d(1/2+\rho)}{(d-1)(1/2-\rho)+(1/2+\rho)}=\log\kappa(d,\rho)<\log
d.$ (181)

In case $\rho$ and $d$ are constant and the estimator $\phi$ is a linear
combination of the observations, using Equation 180 in Lemma 3 one gets:

$R_{B}\geq\rho\left(1-\frac{\kappa(d,\rho)2\rho}{\sqrt{m\text{Var}(Y_{1})+\rho^{2}/3}}\right).$
(182)

This lower bound approaches $\rho$ as $m$ grows, rather than providing the
trivial lower bound of $0$, as it happens in Equation 113.
The assumptions required, along with the need of specifying a prior over $W$,
clearly restrict the domain of applicability of Lemma 3 with respect to
Theorem 3 and Equation 34. However, this approach can provide results in
settings where Theorem 3 and Equation 34 become vacuous.

###  D.2 Lower-Bounding the Risk Directly

An alternative route can be undertaken that does not use Markov’s inequality
as a first step and can possibly lead to tighter bounds. Since our purpose is
to provide lower bounds on the Risk (essentially, an inner-product between the
joint measure of the parameter and the estimation and the loss function,
$\langle\mathcal{P}_{W\hat{W}},\ell\rangle$) one can also consider the
application of reverse Hölder’s inequality in order to directly lower-bound
the Risk. Consider $\alpha<1$, the following result can be easily proven:

######  Corollary 7.

Consider the Bayesian framework described in Section 1.3. The following must
hold for every $\alpha,\alpha^{\prime}<1$

$\displaystyle\mathcal{P}_{W,\hat{W}}(\ell)\geq$
$\displaystyle\mathcal{P}_{\hat{W}}^{\frac{1}{\beta^{\prime}}}\left(\mathcal{P}_{W}^{\frac{\beta^{\prime}}{\beta}}\left(\ell^{\beta}\right)\right)\cdot\mathcal{P}_{\hat{W}}^{\frac{1}{\alpha^{\prime}}}\left(\mathcal{P}_{W}^{\frac{\alpha^{\prime}}{\alpha}}\left(\left(\frac{d\mathcal{P}_{W\hat{W}}}{d\mathcal{P}_{W}\mathcal{P}_{\hat{W}}}\right)^{\alpha}\right)\right),\leavevmode\nobreak\
$ (183)

where
$\frac{1}{\alpha}+\frac{1}{\beta}=1=\frac{1}{\alpha^{\prime}}+\frac{1}{\beta^{\prime}}$
and $\alpha,\alpha^{\prime}<1$. Moreover, if one takes the limit of
$\alpha^{\prime}\to 1^{-}$, which implies $\beta^{\prime}\to-\infty$, then one
recovers the following with $0<\alpha<1$:

$R_{B}\geq\operatorname*{ess\,inf}_{\mathcal{P}_{\hat{W}}}\left(\mathcal{P}_{W}^{\frac{1}{\beta}}\left(\ell(W,\hat{W})^{\beta}\right)\right)\cdot\exp\left(\frac{\alpha-1}{\alpha}I_{\alpha}(W,X)\right).$
(184)

###### Proof.

The proof of Equation 183 follows from the same proof of Theorem 2 (cf. [29,
Theorem 15]) with $f=\ell$ but using reverse Hölder’s inequality rather than
regular Hölder’s inequality. Considering the limit of $\alpha^{\prime}\to
1^{-}$ in Equation 183 one recovers the following:

$\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq\operatorname*{ess\,inf}_{\mathcal{P}_{\hat{W}}}\left(\mathcal{P}_{W}^{\frac{1}{\beta}}\left(\ell(W,\hat{W})^{\beta}\right)\right)\cdot\exp\left(\operatorname{sign}(\alpha)\cdot\frac{\alpha-1}{\alpha}I_{\alpha}(W,\hat{W})\right)$
(185)

Now, if $0<\alpha<1$ then $\frac{1}{\beta}<0$. By the Data-Processing
Inequality for $I_{\alpha}$ with $0<\alpha<1$ (along with the negativity of
$\frac{1}{\beta}$) one has that

$\displaystyle\exp\left(\operatorname{sign}(\alpha)\cdot\frac{\alpha-1}{\alpha}I_{\alpha}(W,\hat{W})\right)$
$\displaystyle=\exp\left(\frac{1}{\beta}I_{\alpha}(W,\hat{W})\right)\geq\exp\left(\frac{1}{\beta}I_{\alpha}(W,X)\right).$
(186)

The lower-bound on the Risk follows by noticing that the right-hand side
Equation 185 can be rendered independent of $\hat{W}$ for every $\alpha<1$
(i.e., it will only depend on the support of $\hat{W}$ through the
$\operatorname*{ess\,inf}$) via Equation 186. ∎

######  Remark 11 (Extending to $\alpha<0$).

One could also extend the result to $\alpha<0$ (which implies $0<\beta<1$),
however, this would lead to a notion of $I_{\alpha}$ for $\alpha<0$ (cf. [38])
which is outside the scope of this work. However, in that case one would have
the following interesting limiting behavior when $\alpha\to-\infty$:

$\displaystyle\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))$
$\displaystyle\geq\left(\operatorname*{ess\,inf}_{\mathcal{P}_{\hat{W}}}\mathcal{P}_{W}\left(\ell(W,\hat{w})\right)\right)\left(\int_{\mathcal{\hat{W}}}\operatorname*{ess\,inf}_{\mathcal{P}_{W}}\mathcal{P}_{\hat{W}|W}\right)$
(187)
$\displaystyle=\left(\operatorname*{ess\,inf}_{\mathcal{P}_{\hat{W}}}\mathcal{P}_{W}\left(\ell(W,\hat{w})\right)\right)\exp\left(-\mathcal{L}^{c}(W\\!\\!\to\\!\\!{\hat{W}})\right),$
(188)

where $\mathcal{L}^{c}(W\\!\\!\to\\!\\!{\hat{W}})$ represents maximal cost-
leakage [20, Definition 11].

Corollary 7 is different from the results presented in the previous section.
While in Section 3 the only dependence on $\ell$ was through the small-ball
probability, in Corollary 7 one is required to have access to the expected
value of the $\beta$-th moments of $\ell$ with respect to $\mathcal{P}_{X}$.
Such an object may not be as easy to bound as the small-ball probability.

######  Remark 12.

If $W=\hat{W}$ then $\ell(W,\hat{W})=0$ and $I_{\alpha}(W,\hat{W})=0$. If
$\,0<\alpha<1$, given that $\beta<0$, one recovers the following lower-bound
on the risk, which matches with our intuition:
$\mathcal{P}_{W\hat{W}}(\ell(W,\hat{W}))\geq 0.$

The main difference with the results presented in Section 3 is that there is
no small-ball probability involved and it is thus required to have access to
an object of the form
$\min_{\hat{w}}\left\lVert\ell(W,\hat{w})\right\rVert_{L^{\beta}(\mathcal{P}_{W})}$
(where we are abusing the notation as for $\beta<0$ it is not a norm), which
might be harder to compute than $L_{W}(\rho)$.

## References

  * [1] A. Xu and M. Raginsky, “Information-theoretic lower bounds on bayes risk in decentralized estimation,” _IEEE Transactions on Information Theory_ , vol. 63, no. 3, pp. 1580–1600, 2017.
  * [2] O. Shamir, “Fundamental limits of online and distributed algorithms for statistical learning and estimation,” in _Advances in Neural Information Processing Systems_ , Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, Eds., vol. 27. Curran Associates, Inc., 2014, pp. 163–171.
  * [3] H. L. V. Trees, _Detection, Estimation, and Modulation Theory_. John Wiley & Sons, Ltd, 2001. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/0471221082
  * [4] M. Sato and M. Akahira, “An information inequality for the bayes risk,” _The Annals of Statistics_ , vol. 24, no. 5, pp. 2288–2295, 1996. [Online]. Available: http://www.jstor.org/stable/2242654
  * [5] L. D. Brown and L. Gajek, “Information inequalities for the bayes risk,” _The Annals of Statistics_ , vol. 18, no. 4, pp. 1578–1594, 1990. [Online]. Available: http://www.jstor.org/stable/2241876
  * [6] H. L. Van Trees and K. L. Bell, _Bounds on the Bayes and Minimax Risk for Signal Parameter Estimation_ , 2007, pp. 329–337.
  * [7] L. Brown and R. Liu, “Bounds on the bayes and minimax risk for signal parameter estimation,” _IEEE Transactions on Information Theory_ , vol. 39, no. 4, pp. 1386–1394, 1993.
  * [8] Te Sun Han and S. Amari, “Statistical inference under multiterminal data compression,” _IEEE Transactions on Information Theory_ , vol. 44, no. 6, pp. 2300–2324, 1998.
  * [9] Y. Zhang, J. Duchi, M. I. Jordan, and M. J. Wainwright, “Information-theoretic lower bounds for distributed statistical estimation with communication constraints,” in _Advances in Neural Information Processing Systems_ , C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger, Eds., vol. 26. Curran Associates, Inc., 2013.
  * [10] J. C. Duchi and M. J. Wainwright, “Distance-based and continuum fano inequalities with applications to statistical estimation,” _CoRR_ , vol. abs/1311.2669, 2013. [Online]. Available: http://arxiv.org/abs/1311.2669
  * [11] S. Asoodeh, M. Aliakbarpour, and F. P. Calmon, “Local differential privacy is equivalent to contraction of an $f$-divergence,” in _2021 IEEE International Symposium on Information Theory (ISIT)_ , 2021, pp. 545–550.
  * [12] X. Chen, A. Guntuboyina, and Y. Zhang, “On bayes risk lower bounds,” _J. Mach. Learn. Res._ , vol. 17, no. 1, p. 7687–7744, jan 2016.
  * [13] I. Csiszár, “A class of measures of informativity of observation channels,” _Periodica Mathematica Hungarica_ , vol. 2, pp. 191–213, 1972\.
  * [14] A. R. Esposito, M. Gastpar, and I. Issa, “Generalization error bounds via rényi-, f-divergences and maximal leakage,” _IEEE Transactions on Information Theory_ , vol. 67, no. 8, pp. 4986–5004, 2021.
  * [15] F. Rassoul-Agha and T. Seppäläinen, _A course on large deviations with an introduction to Gibbs measures_ , 05 2015.
  * [16] T. van Erven and P. Harremoës, “Rényi divergence and Kullback-Leibler divergence,” _IEEE Trans. Inf. Theory_ , vol. 60, no. 7, pp. 3797–3820, July 2014.
  * [17] I. Csiszar, “Generalized cutoff rates and Rényi’s information measures,” _IEEE Transactions on Information Theory_ , vol. 41, no. 1, pp. 26–34, Jan 1995.
  * [18] R. Sibson, “Information radius,” _Z. Wahrscheinlichkeitstheorie verw Gebiete 14_ , pp. 149–160, 1969.
  * [19] S. Verdú, “$\alpha$-mutual information,” in _2015 Information Theory and Applications Workshop, ITA 2015, San Diego, CA, USA, February 1-6, 2015_ , 2015, pp. 1–6.
  * [20] I. Issa, A. B. Wagner, and S. Kamath, “An operational approach to information leakage,” _IEEE Transactions on Information Theory_ , vol. 66, no. 3, pp. 1625–1657, 2020.
  * [21] F. Liese and I. Vajda, “On divergences and informations in statistics and information theory,” _IEEE Trans. Inf. Theor._ , vol. 52, no. 10, pp. 4394–4412, 2006. [Online]. Available: http://dx.doi.org/10.1109/TIT.2006.881731
  * [22] M. Raginsky, “Strong data processing inequalities and $\phi$-Sobolev inequalities for discrete channels,” _IEEE Transactions on Information Theory_ , vol. 62, no. 6, pp. 3355–3389, 2016.
  * [23] J. E. Cohen, Y. Iwasa, G. Rautu, M. Beth Ruskai, E. Seneta, and G. Zbaganu, “Relative entropy under mappings by stochastic matrices,” _Linear Algebra and its Applications_ , vol. 179, pp. 211–235, 1993. [Online]. Available: https://www.sciencedirect.com/science/article/pii/002437959390331H
  * [24] A. Makur and L. Zheng, “Comparison of contraction coefficients for f-divergences,” _Problems of Information Transmission_ , vol. 56, p. 103–156, 2020.
  * [25] S. Varadhan, _Large Deviations and Applications_ , 1984.
  * [26] V. Anantharam, “A variational characterization of rényi divergences,” _CoRR_ , vol. abs/1701.07796, 2017. [Online]. Available: http://arxiv.org/abs/1701.07796
  * [27] J. Birrell, P. Dupuis, M. A. Katsoulakis, L. Rey-Bellet, and J. Wang, “Variational representations and neural network estimation of rényi divergences,” 2021.
  * [28] M. Broniatowski and A. Keziou, “Minimization of divergences on sets of signed measures,” _Studia Scientiarum Mathematicarum Hungarica_ , vol. 43, no. 4, pp. 403–442, 2006.
  * [29] A. R. Esposito, “A functional perspective on information measures,” p. 170, 2022\. [Online]. Available: http://infoscience.epfl.ch/record/294547
  * [30] I. Sason and S. Verdú, “$f$ -divergence inequalities,” _IEEE Transactions on Information Theory_ , vol. 62, no. 11, pp. 5973–6006, 2016.
  * [31] Y. Polyanskiy, H. V. Poor, and S. Verdu, “Channel coding rate in the finite blocklength regime,” _IEEE Transactions on Information Theory_ , vol. 56, no. 5, pp. 2307–2359, 2010.
  * [32] A. R. Esposito, D. Wu, and M. Gastpar, “On conditional sibson’s $\alpha$-mutual information,” in _2021 IEEE International Symposium on Information Theory (ISIT)_ , 2021, pp. 1796–1801.
  * [33] J. E. Cohen, Y. Iwasa, G. Rautu, M. Beth Ruskai, E. Seneta, and G. Zbaganu, “Relative entropy under mappings by stochastic matrices,” _Linear Algebra and its Applications_ , vol. 179, pp. 211–235, 1993. [Online]. Available: https://www.sciencedirect.com/science/article/pii/002437959390331H
  * [34] R. T. Rockafellar, _Convex analysis_ , ser. Princeton Mathematical Series. Princeton, N. J.: Princeton University Press, 1970.
  * [35] R. L. Graham, D. E. Knuth, and O. Patashnik, _Concrete Mathematics: A Foundation for Computer Science_. Reading: Addison-Wesley, 1989.
  * [36] S. G. Bobkov and G. P. Chistyakov, “On concentration functions of random variables,” _Journal of Theoretical Probability volume_ , vol. 28, 2015.
  * [37] H. H. Nguyen and V. H. Vu, _Small Ball Probability, Inverse Theorems, and Applications_. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 409–463.
  * [38] A. R. Esposito, A. Vandenbroucque, and M. Gastpar, “On sibson’s $\upalpha$-mutual information,” in _2022 IEEE International Symposium on Information Theory (ISIT)_. IEEE, Jun. 2022. [Online]. Available: https://doi.org/10.1109/isit50566.2022.9834428