∎

11institutetext: N. Balakrishnan, 22institutetext: McMaster University,
Hamilton, Ontario, Canada
22email<EMAIL_ADDRESS>33institutetext: F. Buono 44institutetext:
Università degli Studi di Napoli Federico II, Naples, Italy
44email<EMAIL_ADDRESS>55institutetext: M. Longobardi
66institutetext: Università degli Studi di Napoli Federico II, Naples, Italy
66email<EMAIL_ADDRESS>
# On cumulative entropies in terms of moments of order statistics

Narayanaswamy Balakrishnan  Francesco Buono  Maria Longobardi

(Received: date / Accepted: date)

###### Abstract

In this paper relations among some kinds of cumulative entropies and moments
of order statistics are presented. By using some characterizations and the
symmetry of a non negative and absolutely continuous random variable $X$,
lower and upper bounds for entropies are obtained and examples are given.

AMS Subject Classification: 60E15. 62N05, 94A17

###### Keywords:

Cumulative Entropies Order Statistics Moments

##  1 Introduction

In reliability theory, to describe and study the information related to a non-
negative absolutely continuous random variable $X$ we use the Shannon entropy,
or differential entropy, of $X$, defined by (Shannon, 1948)

$H(X)=-\mathbb{E}[\log(X)]=-\int_{0}^{+\infty}f(x)\log f(x)\mathrm{d}x,$

where $\log$ is the natural logarithm and $f$ is the probability density
function (pdf) of $X$. In the following we use $F$ and $\overline{F}$ to
indicate the cumulative distribution function (cdf) and the survival function
(sf) of $X$, respectively.

In the literature, there are several different versions of entropy, each one
suitable for a specific situation. Rao et al., 2004, introduced the Cumulative
Residual Entropy (CRE) of $X$ as

$\displaystyle\mathcal{E}(X)$ $\displaystyle=$
$\displaystyle-\int_{0}^{+\infty}\overline{F}(x)\log\overline{F}(x)\mathrm{d}x.$

Di Crescenzo and Longobardi, 2009, introduced the Cumulative Entropy (CE) of
$X$ as

$\displaystyle\mathcal{CE}(X)$ $\displaystyle=$
$\displaystyle-\int_{0}^{+\infty}F(x)\log F(x)\mathrm{d}x.$ (1)

This information measure is suitable to measure information when uncertainty
is related to the past, a dual concept of the cumulative residual entropy
which relates to uncertainty on the future lifetime of a system.

Mirali et al., 2016, introduced the Weighted Cumulative Residual Entropy
(WCRE) of $X$ as

$\displaystyle\mathcal{E}^{w}(X)$ $\displaystyle=$
$\displaystyle-\int_{0}^{+\infty}x(1-F(x))\log(1-F(x))\mathrm{d}x.$

Mirali and Baratpour, 2017, introduced the Weighted Cumulative Entropy (WCE)
of $X$ as

$\displaystyle\mathcal{CE}^{w}(X)$ $\displaystyle=$
$\displaystyle-\int_{0}^{+\infty}xF(x)\log F(x)\mathrm{d}x.$

Recently, various authors have discussed different versions of entropy and
their applications (see, for instance, cali, CaLoNa, CaLoPsa, Lo2014).

The paper is organized as follows. In Section 2, we study relations among some
kinds of entropies and moments of order statistics and present various
examples. In Section 3, bounds are given by using also some characterizations
and properties (as the symmetry) of the random variable $X$, some examples and
bounds for known distributions are given.

##  2 A relation among entropies and order statistics

We recall that, if we have $n$ i.i.d. random variables
$X_{1},X_{2},\dots,X_{n}$, we can introduce the order statistics $X_{k:n}$,
$k=1,\dots,n$. The $k$-th order statistic is equal to the $k$-th smallest
value from the sample. We know that the cdf of $X_{k:n}$ can be given in terms
of the cdf of the parent distribution; in fact

$F_{k:n}(x)=\sum_{j=k}^{n}\binom{n}{j}[F(x)]^{j}[1-F(x)]^{n-j},$

whereas the pdf of $X_{k:n}$ is expressed as

$f_{k:n}(x)=\binom{n}{k}k[F(x)]^{k-1}[1-F(x)]^{n-k}f(x).$

Choosing $k=1$ and $k=n$ we get the smallest and the largest order statistic,
respectively. Their cdf and pdf are given by

$\displaystyle F_{1:n}(x)=1-[1-F(x)]^{n}\ \ \ \ \
f_{1:n}(x)=n[1-F(x)]^{n-1}f(x)$ $\displaystyle F_{n:n}(x)=[F(x)]^{n}\ \ \ \ \
\ \ \ \ \ \ \ \ \ \ f_{n:n}(x)=n[F(x)]^{n-1}f(x).$

###  2.1 Cumulative residual entropy

The Cumulative Residual Entropy (CRE) of $X$ can be written also in terms of
order statistics, that is

$\displaystyle\mathcal{E}(X)$ $\displaystyle=$
$\displaystyle-\int_{0}^{+\infty}(1-F(x))\log(1-F(x))\mathrm{d}x$ (2)
$\displaystyle=$
$\displaystyle-x(1-F(x))\log(1-F(x))\big{|}_{0}^{+\infty}-\int_{0}^{+\infty}x\log(1-F(x))f(x)\mathrm{d}x-$
$\displaystyle\int_{0}^{+\infty}xf(x)\mathrm{d}x$ $\displaystyle=$
$\displaystyle\int_{0}^{+\infty}x[-\log(1-F(x))]f(x)\mathrm{d}x-\mathbb{E}(X)$
$\displaystyle=$
$\displaystyle\int_{0}^{+\infty}x\left[\sum_{n=1}^{+\infty}\frac{F(x)^{n}}{n}\right]f(x)\mathrm{d}x-\mathbb{E}(X)$
$\displaystyle=$
$\displaystyle\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\mu_{n+1:n+1}-\mathbb{E}(X),$

where $\mathbb{E}(X)$ is the expectation or mean of $X$, and $\mu_{n+1:n+1}$
the mean of the largest order statistic in a sample of size $n+1$ from $F$,
provided that $\lim_{x\to+\infty}-x(1-F(x))\log(1-F(x))=0$.

We note that (2) can be rewritten as

$\mathcal{E}(X)=\sum_{n=1}^{+\infty}\left(\frac{1}{n}-\frac{1}{n+1}\right)\mu_{n+1:n+1}-\mathbb{E}(X).$
(3)

###### Example 1

Consider the standard exponential distribution with pdf
$f(x)=\mathrm{e}^{-x}$, $x>0$. Then, it is known that

$\mathbb{E}(X)=1\ \mbox{ and }\
\mathbb{E}(X_{n:n})=1+\frac{1}{2}+\dots+\frac{1}{n}.$

Then, from (3), we readily have

$\displaystyle\mathcal{E}(X)$ $\displaystyle=$
$\displaystyle\left(1-\frac{1}{2}\right)\mu_{2:2}+\left(\frac{1}{2}-\frac{1}{3}\right)\mu_{3:3}+\left(\frac{1}{3}-\frac{1}{4}\right)\mu_{4:4}+\dots-\mathbb{E}(X)$
$\displaystyle=$
$\displaystyle\left(\mu_{2:2}-\mathbb{E}(X)\right)+\frac{1}{2}\left(\mu_{3:3}-\mu_{2:2}\right)+\frac{1}{3}\left(\mu_{4:4}-\mu_{3:3}\right)+\dots$
$\displaystyle=$ $\displaystyle\frac{1}{2}+\frac{1}{2\cdot 3}+\frac{1}{3\cdot
4}+\dots$ $\displaystyle=$
$\displaystyle\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}=1.$

###### Example 2

Consider the standard uniform distribution with pdf $f(x)=1$, $0<x<1$. Then,
it is known that

$\mathbb{E}(X)=\frac{1}{2}\ \mbox{ and }\ \mathbb{E}(X_{n:n})=\frac{n}{n+1}.$

So, from (2), we readily find

$\displaystyle\mathcal{E}(X)$ $\displaystyle=$
$\displaystyle\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\
\frac{n+1}{n+2}-\frac{1}{2}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{n=1}^{+\infty}\left(\frac{1}{n}-\frac{1}{n+2}\right)-\frac{1}{2}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\left(1+\frac{1}{2}\right)-\frac{1}{2}=\frac{1}{4}.$

###  2.2 Cumulative entropy

The Cumulative Entropy (CE) of $X$ can be rewritten in terms of the mean of
the minimum order statistic; integrating by parts (1)

$\displaystyle\mathcal{CE}(X)$ $\displaystyle=$ $\displaystyle-xF(x)\log
F(x)\big{|}_{0}^{+\infty}+\int_{0}^{+\infty}x\log
F(x)f(x)\mathrm{d}x+\int_{0}^{+\infty}xf(x)\mathrm{d}x$ (4) $\displaystyle=$
$\displaystyle\int_{0}^{+\infty}x\log[1-(1-F(x))]f(x)\mathrm{d}x+\mathbb{E}(X)$
$\displaystyle=$
$\displaystyle-\int_{0}^{+\infty}x\sum_{n=1}^{+\infty}\frac{(1-F(x))^{n}}{n}f(x)\mathrm{d}x+\mathbb{E}(X)$
$\displaystyle=$
$\displaystyle-\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\mu_{1:n+1}+\mathbb{E}(X),$

where $\mu_{1:n+1}$ is the mean of the smallest order statistic from a sample
of size $n+1$ from $F$, provided that $\lim_{x\to+\infty}-xF(x)\log F(x)=0$.

We note that (4) can be rewritten as

$\mathcal{CE}(X)=-\sum_{n=1}^{+\infty}\left(\frac{1}{n}-\frac{1}{n+1}\right)\mu_{1:n+1}+\mathbb{E}(X).$
(5)

###### Example 3

For the standard exponential distribution, it is known that

$\mathbb{\mu}_{1:n}=\frac{1}{n},$

and so from (5), we readily have

$\displaystyle\mathcal{CE}(X)$ $\displaystyle=$
$\displaystyle-\sum_{n=1}^{+\infty}\left(\frac{1}{n}-\frac{1}{n+1}\right)\frac{1}{n+1}+1$
$\displaystyle=$
$\displaystyle-\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}+\sum_{n=1}^{+\infty}\frac{1}{(n+1)^{2}}+1$
$\displaystyle=$ $\displaystyle\frac{\pi^{2}}{6}-1,$

by the use of Euler’s identity.

###### Example 4

For the standard uniform distribution, using the fact that

$\mathbb{\mu}_{1:n}=\frac{1}{n+1},$

we obtain from (4) that

$\displaystyle\mathcal{CE}(X)$ $\displaystyle=$
$\displaystyle-\sum_{n=1}^{+\infty}\frac{1}{n(n+1)(n+2)}+\frac{1}{2}$
$\displaystyle=$
$\displaystyle\frac{1}{2}-\frac{1}{2}\sum_{n=1}^{+\infty}\frac{1}{n}+\sum_{n=1}^{+\infty}\frac{1}{n+1}-\frac{1}{2}\sum_{n=1}^{+\infty}\frac{1}{n+2}$
$\displaystyle=$ $\displaystyle\frac{1}{4}$

by the use of Euler’s identity.

###### Remark 1

If the random variable $X$ has finite mean $\mu$ and is symmetrically
distributed about $\mu$, then we know

$\mu_{n:n}-\mu=\mu-\mu_{1:n},$

and so the symmetry property of $\mathcal{CE}$ readily follows.

###  2.3 Weighted cumulative entropies

In the same way the Weighted Cumulative Residual Entropy (WCRE) of $X$ can be
expressed as

$\displaystyle\mathcal{E}^{w}(X)$ $\displaystyle=$
$\displaystyle-\frac{x^{2}}{2}(1-F(x))\log(1-F(x))\big{|}_{0}^{+\infty}-\frac{1}{2}\int_{0}^{+\infty}x^{2}\log(1-F(x))f(x)\mathrm{d}x$
(6) $\displaystyle-\frac{1}{2}\int_{0}^{+\infty}x^{2}f(x)\mathrm{d}x$
$\displaystyle=$
$\displaystyle\frac{1}{2}\int_{0}^{+\infty}x^{2}[-\log(1-F(x))]f(x)\mathrm{d}x-\frac{1}{2}\mathbb{E}(X^{2})$
$\displaystyle=$
$\displaystyle\frac{1}{2}\int_{0}^{+\infty}x^{2}\left[\sum_{n=1}^{+\infty}\frac{F(x)^{n}}{n}\right]f(x)\mathrm{d}x-\frac{1}{2}\mathbb{E}(X^{2})$
$\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\mu^{(2)}_{n+1:n+1}-\frac{1}{2}\mathbb{E}(X^{2}),$

where $\mu^{(2)}_{n+1:n+1}$ is the second moment of the largest order
statistic in a sample of size $n+1$, provided that
$\lim_{x\to+\infty}-\frac{x^{2}}{2}(1-F(x))\log(1-F(x))=0$.

###### Example 5

For the standard uniform distribution, using the fact that

$\mu^{(2)}_{n+1:n+1}=\frac{n+1}{n+3}\ \mbox{ and }\
\mathbb{E}(X^{2})=\frac{1}{3},$

we obtain from (6) that

$\displaystyle\mathcal{E}^{w}(X)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\
\frac{n+1}{n+3}-\frac{1}{6}$ $\displaystyle=$
$\displaystyle\frac{1}{6}\sum_{n=1}^{+\infty}\left(\frac{1}{n}-\frac{1}{n+3}\right)-\frac{1}{6}$
$\displaystyle=$
$\displaystyle\frac{1}{6}\left(1+\frac{1}{2}+\frac{1}{3}\right)-\frac{1}{6}=\frac{5}{36}.$

Moreover, we can derive the Weighted Cumulative Entropy (WCE) of $X$ in terms
of the second moment of the minimum order statistic in the following way

$\displaystyle\mathcal{CE}^{w}(X)$ $\displaystyle=$
$\displaystyle-\frac{x^{2}}{2}F(x)\log
F(x)\big{|}_{0}^{+\infty}+\frac{1}{2}\int_{0}^{+\infty}x^{2}\log
F(x)f(x)\mathrm{d}x+$ (7)
$\displaystyle\frac{1}{2}\int_{0}^{+\infty}x^{2}f(x)\mathrm{d}x$
$\displaystyle=$
$\displaystyle\frac{1}{2}\int_{0}^{+\infty}x^{2}\log[1-(1-F(x))]f(x)\mathrm{d}x+\frac{1}{2}\mathbb{E}(X^{2})$
$\displaystyle=$
$\displaystyle-\frac{1}{2}\int_{0}^{+\infty}x^{2}\sum_{n=1}^{+\infty}\frac{(1-F(x))^{n}}{n}f(x)\mathrm{d}x+\frac{1}{2}\mathbb{E}(X^{2})$
$\displaystyle=$
$\displaystyle-\frac{1}{2}\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\mu^{(2)}_{1:n+1}+\frac{1}{2}\mathbb{E}(X^{2}),$

where $\mu^{(2)}_{1:n+1}$ is the second moment of the smallest order statistic
in a sample of size $n+1$, provided that
$\lim_{x\to+\infty}-\frac{x^{2}}{2}F(x)\log F(x)=0$.

##  3 Bounds

Let us consider a sample with parent distribution $X$ such that
$\mathbb{E}(X)=0$ and $\mathbb{E}(X^{2})=1$. Hartley and David, 1954, and
Gumbel, 1954, have shown that

$\mu_{n:n}\leq\frac{n-1}{\sqrt{2n-1}}.$

We relate $\mu_{n:n}$ with the mean of the largest statistic order from the
standard distribution. In fact, by normalizing the random variable $X$ with
mean $\mu$ and variance $\sigma^{2}$ we get

$Z=\frac{X-\mu}{\sigma}.$

Hence, the cdf $F_{Z}$ is given in terms of the cdf $F_{X}$ by

$F_{Z}(x)=F_{X}(\sigma x+\mu).$

Then, cdf and pdf of the largest order statistic in a sample of size $n$ are

$F_{Z_{n:n}}(x)=F_{X}^{n}(\sigma x+\mu),\ \ \ \ \
f_{Z_{n:n}}(x)=nF_{X}^{n-1}(\sigma x+\mu)f_{X}(\sigma x+\mu)\sigma.$

The mean of $X_{n:n}$ is given by

$\mu_{n:n}=\mathbb{E}(X_{n:n})=n\int_{0}^{+\infty}xF_{X}^{n-1}(x)f_{X}(x)\mathrm{d}x.$

The mean of the largest statistic order from $Z$ is given by

$\displaystyle\mathbb{E}(Z_{n:n})$ $\displaystyle=$ $\displaystyle
n\sigma\int_{-\frac{\mu}{\sigma}}^{+\infty}xF_{X}^{n-1}(\sigma
x+\mu)f_{X}(\sigma x+\mu)\mathrm{d}x$ $\displaystyle=$ $\displaystyle
n\int_{0}^{+\infty}\frac{x-\mu}{\sigma}F_{X}^{n-1}(x)f_{X}(x)\mathrm{d}x$
$\displaystyle=$
$\displaystyle\frac{n}{\sigma}\int_{0}^{+\infty}xF_{X}^{n-1}(x)f_{X}(x)\mathrm{d}x-\frac{n\mu}{\sigma}\int_{0}^{+\infty}F_{X}^{n-1}(x)f_{X}(x)\mathrm{d}x$
$\displaystyle=$ $\displaystyle\frac{\mu_{n:n}-\mu}{\sigma}.$

Using the Hartley-David-Gumbel bound for a non-negative parent distribution
with mean $\mu$ and variance $\sigma^{2}$, we get

$\mu_{n:n}=\sigma\mathbb{E}(Z_{n:n})+\mu\leq\sigma\frac{n-1}{\sqrt{2n-1}}+\mu.$
(8)

###### Theorem 3.1

Let $X$ be a non-negative random variable with mean $\mu$ and variance
$\sigma^{2}$. Then, we obtain an upper bound for the CRE of $X$

$\mathcal{E}(X)\leq\sum_{n=1}^{+\infty}\frac{\sigma}{(n+1)\sqrt{2n+1}}\simeq
1.21\ \sigma.$ (9)

###### Proof

From (2) and (8) we get

$\displaystyle\mathcal{E}(X)$ $\displaystyle=$
$\displaystyle\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\mu_{n+1:n+1}-\mathbb{E}(X)$
$\displaystyle\leq$
$\displaystyle\sum_{n=1}^{+\infty}\frac{1}{n(n+1)}\left(\sigma\frac{n}{\sqrt{2n+1}}+\mathbb{E}(X)\right)-\mathbb{E}(X)$
$\displaystyle=$
$\displaystyle\sum_{n=1}^{+\infty}\frac{\sigma}{(n+1)\sqrt{2n+1}}\simeq 1.21\
\sigma,$

i.e., the upper bound given in (9)

###### Remark 2

Since $X$ is non-negative we have that $\mu_{n+1:n+1}\geq 0$, for all
$n\in\mathbb{N}$. For this reason, using finite series approximations we get
lower bounds for $\mathcal{E}(X)$:

$\mathcal{E}(X)\geq\sum_{n=1}^{m}\frac{1}{n(n+1)}\mu_{n+1:n+1}-\mathbb{E}(X),$

for all $m\in\mathbb{N}$.

###### Remark 3

Since $X$ is non-negative we have that $\mu_{1:n+1}\geq 0$, for all
$n\in\mathbb{N}$. For this reason, using finite series approximations we get
upper bounds for $\mathcal{CE}(X)$:

$\mathcal{CE}(X)\leq-\sum_{n=1}^{m}\frac{1}{n(n+1)}\mu_{1:n+1}+\mathbb{E}(X),$

for all $m\in\mathbb{N}$.

###### Theorem 3.2

Let $X$ be DFR (decreasing failure rate). Then, we have the following lower
bound for $\mathcal{CE}(X)$

$\mathcal{CE}(X)\geq\mathbb{E}(X)-\frac{\sqrt{\mathbb{E}(X^{2})}}{\sqrt{2}}\left(2-\frac{\pi^{2}}{6}\right).$
(10)