# Nonlocal quantum state ensembles and quantum data hiding

Donghoon Ha  Department of Applied Mathematics and Institute of Natural
Sciences, Kyung Hee University, Yongin 17104, Republic of Korea  Jeong San Kim
<EMAIL_ADDRESS>Department of Applied Mathematics and Institute of Natural
Sciences, Kyung Hee University, Yongin 17104, Republic of Korea

###### Abstract

We consider the discrimination of bipartite quantum states and establish a
relation between nonlocal quantum state ensemble and quantum data hiding
processing. Using a bound on optimal local discrimination of bipartite quantum
states, we provide a sufficient condition for a bipartite quantum state
ensemble to be used to construct a quantum data-hiding scheme. Our results are
illustrated by examples in multidimensional bipartite quantum systems.

##  I Introduction

Quantum nonlocality is a key feature of quantum information processing and
quantum communication[1, 2, 3, 4]. Quantum entanglement is a well-known
quantum nonlocal correlation, which can be used as crucial resources for
quantum key distribution and quantum teleportation[5, 6, 7]. Quantum steering
is another quantum nonlocality, which can be used in various quantum
information processing tasks such as quantum cryptography, randomness
certification, and quantum channel discrimination[4]. Recently, it was shown
that the nonlocality of quantum coherence can provide advantages for coherence
steerability[8, 9, 10].

Quantum nonlocal phenomenon can also arise in discriminating multipartite
quantum states. In general, orthogonal quantum states can be discriminated
with certainty, whereas nonorthogonal quantum states cannot[11, 12, 13, 14].
However, there exist some multipartite orthogonal quantum states that cannot
be perfectly discriminated only by _local operations and classical
communication_(LOCC)[15, 16, 17]. Moreover, there exist some multipartite
nonorthogonal quantum states that cannot be optimally discriminated only by
LOCC[19, 20, 18].

The nonlocality of multipartite quantum state discrimination can be useful in
quantum communications[21, 22]. Quantum data hiding is one of the typical
applications using the nonlocality of multipartite quantum state
discrimination[21, 22]. The first quantum data-hiding scheme was shown through
the nonlocal quantum state ensemble consisting of Bell states[21]; a classical
bit is globally accessible but not locally, and the bit is perfectly hidden
asymptotically. There also have been several quantum data-hiding schemes
introduced using some sets of quantum states[23, 24, 25, 26, 27]. However, it
is still natural to ask which kind of nonlocal quantum state ensemble can be
used in quantum data-hiding schemes.

Here, we consider the discrimination of bipartite quantum states and establish
a relation between nonlocal quantum state ensemble and quantum data hiding
processing. Based on a bound on optimal local discrimination of bipartite
quantum states, we provide a sufficient condition for a bipartite quantum
state ensemble to be used to construct a quantum data-hiding scheme. Using our
quantum data-hiding scheme, a classical data of arbitrary size can be
perfectly hidden, asymptotically, if the users are restricted to LOCC
measurements, whereas the data can be completely recovered if they can
collaborate to use global measurements. Our results are illustrated by
examples in multidimensional bipartite quantum systems.

This paper is organized as follows. In Sec. II, we review the definitions and
properties of minimum-error discrimination in bipartite quantum systems. In
Sec. III, we recall a bound on local minimum-error discrimination, and provide
some properties which are useful in quantum data hiding. In Sec. IV, we
provide a sufficient condition for a bipartite quantum state ensemble to be
used to construct a quantum data-hiding scheme. Our results are illustrated by
examples in multidimensional bipartite quantum systems. In Sec. V, we
summarize our results with future works.

##  II Minimum-error discrimination of bipartite quantum states

For bipartite quantum systems held by two parties, Alice and Bob, a quantum
state is described by a density operator $\rho$, that is, a positive-
semidefinite operator $\rho\succeq 0$ with unit trace
$\operatorname{Tr}\rho=1$, acting on a bipartite Hilbert space
$\mathcal{H}=\mathbb{C}^{d_{\rm A}}\otimes\mathbb{C}^{d_{\rm B}}$. A
measurement is represented by a positive operator-valued measure
$\\{M_{i}\\}_{i}$, that is, a set of positive-semidefinite operators
$M_{i}\succeq 0$ on $\mathcal{H}$ satisfying the completeness relation
$\sum_{i}M_{i}=\mathbbm{1}$, where $\mathbbm{1}$ is the identity operator on
$\mathcal{H}$. When a measurement $\\{M_{i}\\}_{i}$ is performed to a quantum
state $\rho$, the probability of obtaining the measurement outcome with
respect to $M_{j}$ is $\operatorname{Tr}(\rho M_{j})$.

In this section, we consider the situation of discriminating $n$ bipartite
quantum states $\rho_{0},\ldots,\rho_{n-1}$ where the state $\rho_{i}$ is
prepared with the probability $\eta_{i}\in[0,1]$ for $i=0,\ldots,n-1$. We
denote this situation as the _quantum state discrimination of the ensemble_ ,

$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}.$ (1)

In discriminating the states from the ensemble $\mathcal{E}$ using a
measurement $\mathcal{M}=\\{M_{i}\\}_{i=0}^{n-1}$, we decide that the prepared
state is $\rho_{i}$ if the measurement outcome is from $M_{i}$. In this case,
the average probability of correctly guessing the prepared state is

$\sum_{i=0}^{n-1}\eta_{i}\operatorname{Tr}(\rho_{i}M_{i}).$ (2)

The _minimum-error discrimination_ of $\mathcal{E}$ is to achieve the optimal
success probability

$p_{\rm
G}(\mathcal{E})=\max_{\mathcal{M}}\sum_{i=0}^{n-1}\eta_{i}\operatorname{Tr}(\rho_{i}M_{i}),$
(3)

where the maximum is taken over all possible measurements[28, 29].

A measurement is called a _LOCC measurement_ if it can be realized by LOCC.
When the available measurements are limited to LOCC measurements, we denote
the maximum success probability by

$p_{\rm L}(\mathcal{E})=\max_{\begin{subarray}{c}\rm LOCC\\\
\mathcal{M}\end{subarray}}\sum_{i=0}^{n-1}\eta_{i}\operatorname{Tr}(\rho_{i}M_{i}).$
(4)

From the definitions of $p_{\rm G}(\mathcal{E})$ and $p_{\rm L}(\mathcal{E})$
in Eqs. (3) and (4), respectively, $p_{\rm G}(\mathcal{E})$ is obviously an
upper bound of $p_{\rm L}(\mathcal{E})$, therefore

$p_{\rm L}(\mathcal{E})\leqslant p_{\rm G}(\mathcal{E}).$ (5)

Moreover, deciding the prepared state as the state with largest prior
probability is obviously a LOCC measurement, thus we have

$\max\\{\eta_{0},\ldots,\eta_{n-1}\\}\leqslant p_{\rm L}(\mathcal{E}).$ (6)

##  III Bounds of local minimum-error discrimination

For a bipartite quantum state ensemble $\mathcal{E}$ in Eq. (1), we first
consider an upper bound of $p_{\rm L}(\mathcal{E})$ in Eq. (4) based on
_partial transposition_ of the states. We further provide some properties of
the upper bound which are useful in quantum data hiding.

######  Definition 1.

A Hermitian operator $E$ on a bipartite Hilbert space $\mathcal{H}$ is called
_positive partial transpose_(PPT) if $E^{\operatorname{PT}}\succeq 0$, where
the superscript $\operatorname{PT}$ is to indicate the partial
transposition[30, 31]. Otherwise, $E$ is called _negative partial
transpose_(NPT).

For an ensemble $\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$, let us
consider

$q_{\rm
G}(\mathcal{E})=\max_{\mathcal{M}}\sum_{i=0}^{n-1}\eta_{i}\operatorname{Tr}(\rho_{i}^{\operatorname{PT}}M_{i}),$
(7)

where the maximum is taken over all possible measurements. This quantity is
known as an upper bound of $p_{\rm L}(\mathcal{E})$, that is,

$p_{\rm L}(\mathcal{E})\leqslant q_{\rm G}(\mathcal{E})$ (8)

for any bipartite quantum state ensemble $\mathcal{E}$[32]. The following
proposition provides a necessary and sufficient condition of a measurement
realizing $q_{\rm G}(\mathcal{E})$[32].

######  Proposition 1.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ and a measurement
$\mathcal{M}=\\{M_{i}\\}_{i=0}^{n-1}$, $\mathcal{M}$ realizes $q_{\rm
G}(\mathcal{E})$ if and only if

$\sum_{j=0}^{n-1}\eta_{j}\rho_{j}^{\operatorname{PT}}M_{j}-\eta_{i}\rho_{i}^{\operatorname{PT}}\succeq
0$ (9)

for each $i=0,\ldots,n-1$.

Although Proposition 1 provides a necessary and sufficient condition of the
optimal measurement realizing $q_{\rm G}(\mathcal{E})$, the definition of
$q_{\rm G}(\mathcal{E})$ in Eq. (7) is based on the optimization over all
possible measurements. However, the following theorem provides the closed-form
expression of $q_{\rm G}(\mathcal{E})$ if the ensemble consists of two states.

######  Theorem 1.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{1}$,

$q_{\rm
G}(\mathcal{E})=\frac{1}{2}+\frac{1}{2}\operatorname{Tr}|\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}}|,$
(10)

where $|E|$ is the positive square root of $E^{\dagger}E$, that is,

$|E|=\sqrt{E^{\dagger}E}.$ (11)

###### Proof.

Let $M_{0}$ and $M_{1}$ be the projection operators onto the non-negative and
negative eigenspaces of the Hermitian operator
$\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}}$,
respectively, and consider the measurement
$\mathcal{M}=\\{M_{i}\\}_{i=0}^{1}$. From the completeness
$M_{0}+M_{1}=\mathbbm{1}$, we have

$\displaystyle\sum_{j=0}^{1}\eta_{j}\rho_{j}^{\operatorname{PT}}M_{j}-\eta_{i}\rho_{i}^{\operatorname{PT}}$
(12) $\displaystyle=$
$\displaystyle\eta_{0}\rho_{0}^{\operatorname{PT}}M_{0}+\eta_{1}\rho_{1}^{\operatorname{PT}}M_{1}-\eta_{i}\rho_{i}^{\operatorname{PT}}(M_{0}+M_{1})$
$\displaystyle=$
$\displaystyle(\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{i}\rho_{i}^{\operatorname{PT}})M_{0}+(\eta_{1}\rho_{1}^{\operatorname{PT}}-\eta_{i}\rho_{i}^{\operatorname{PT}})M_{1}$
$\displaystyle=$
$\displaystyle(\eta_{1-i}\rho_{1-i}^{\operatorname{PT}}-\eta_{i}\rho_{i}^{\operatorname{PT}})M_{1-i}$

for each $i=0,1$. By using the notation

$E^{(\pm)}=\frac{1}{2}(|E|\pm E)$ (13)

together with the definition of $\mathcal{M}$, the last term of Eq. (12) can
be rewritten as

$\displaystyle(\eta_{1-i}\rho_{1-i}^{\operatorname{PT}}-\eta_{i}\rho_{i}^{\operatorname{PT}})M_{1-i}$
(14) $\displaystyle=$
$\displaystyle(\eta_{1-i}\rho_{1-i}^{\operatorname{PT}}-\eta_{i}\rho_{i}^{\operatorname{PT}})^{(+)}\succeq
0,$

where the positivity is from $E^{(+)}\succeq 0$ for any Hermitian operator
$E$. Equations (12) and (14) imply that $\mathcal{M}$ satisfies the optimality
condition in Proposition 1.

Thus, we have

$\displaystyle q_{\rm G}(\mathcal{E})$ $\displaystyle=$
$\displaystyle\sum_{i=0}^{1}\eta_{i}\operatorname{Tr}(\rho_{i}^{\operatorname{PT}}M_{i})$
(15) $\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{i=0}^{1}\operatorname{Tr}\Big{(}\sum_{j=0}^{1}\eta_{j}\rho_{j}^{\operatorname{PT}}M_{j}-\eta_{i}\rho_{i}^{\operatorname{PT}}+\eta_{i}\rho_{i}^{\operatorname{PT}}\Big{)}$
$\displaystyle=$
$\displaystyle\frac{1}{2}+\frac{1}{2}\sum_{i=0}^{1}\operatorname{Tr}(\eta_{1-i}\rho_{1-i}^{\operatorname{PT}}-\eta_{i}\rho_{i}^{\operatorname{PT}})^{(+)}$
$\displaystyle=$
$\displaystyle\frac{1}{2}+\frac{1}{2}\operatorname{Tr}|\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}}|,$

where the first equality is from Proposition 1, the third equality is due to
Eqs. (12) and (14) together with $\eta_{0}+\eta_{1}=1$, and the last equality
is from $E^{(-)}=(-E)^{(+)}$ and $|E|=E^{(+)}+E^{(-)}$ for any Hermitian
operator $E$. ∎

For an ensemble $\mathcal{E}$ with an arbitrary number of states, the
following theorem provides an upper bound of $q_{\rm G}(\mathcal{E})$.

######  Theorem 2.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ and a Hermitian operator
$H$, if

$H-\eta_{i}\rho_{i}^{\operatorname{PT}}\succeq 0$ (16)

for each $i=0,\ldots,n-1$, then the trace of $H$ is an upper bound of $q_{\rm
G}(\mathcal{E})$, that is,

$\operatorname{Tr}H\geqslant q_{\rm G}(\mathcal{E}).$ (17)

###### Proof.

For the optimal measurement $\mathcal{M}=\\{M_{i}\\}_{i=0}^{n-1}$ providing
$q_{\rm G}(\mathcal{E})$, we have

$\operatorname{Tr}H-q_{\rm
G}(\mathcal{E})=\sum_{i=0}^{n-1}\operatorname{Tr}[(H-\eta_{i}\rho_{i}^{\operatorname{PT}})M_{i}]\geqslant
0,$ (18)

where the equality is from $\sum_{i=0}^{n-1}M_{i}=\mathbbm{1}$ and the
definition of $q_{\rm G}(\mathcal{E})$ in Eq. (7) and the inequality is by
Condition (16) and $M_{i}\succeq 0$ for all $i=0,\ldots,n-1$. Thus,
$\operatorname{Tr}H$ is an upper bound of $q_{\rm G}(\mathcal{E})$. ∎

##  IV Quantum data hiding

In this section, we provide a sufficient condition for a bipartite quantum
state ensemble to be used to construct a quantum data-hiding scheme. To
analyze how reliably the data is hidden, we first introduce the notion of
_multi-fold ensemble_.

###  IV.1 Multi-fold ensemble

For positive integers $n$ and $L$, we denote $\mathbb{Z}_{n}$ as the set of
all integers from $0$ to $n-1$ and $\mathbb{Z}_{n}^{L}$ as the Cartesian
product of $L$ copies of $\mathbb{Z}_{n}$. For the ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ in Eq. (1) and each vector

$\vec{c}=(c_{1},\ldots,c_{L})\in\mathbb{Z}_{n}^{L},$ (19)

we define

$\eta_{\vec{c}}=\prod_{l=1}^{L}\eta_{c_{l}},~{}\rho_{\vec{c}}=\bigotimes_{l=1}^{L}\rho_{c_{l}}$
(20)

where $\rho_{c_{l}}$ is the state in the ensemble $\mathcal{E}$ whose index is
the $l$th coordinate of the vector $\vec{c}$, and $\eta_{c_{l}}$ is the
corresponding probability for $l=1,\ldots,L$. We further use
$\mathcal{E}^{\otimes L}$ to denote the _$L$ -fold ensemble_ of $\mathcal{E}$,
that is,

$\mathcal{E}^{\otimes
L}=\\{\eta_{\vec{c}},\rho_{\vec{c}}\\}_{\vec{c}\in\mathbb{Z}_{n}^{L}}.$ (21)

For each $\vec{c}\in\mathbb{Z}_{n}^{L}$, we define $\omega_{n}(\vec{c})$ as
the modulo-$n$ summation of all entries in $\vec{c}=(c_{1},\ldots,c_{L})$,
that is,

$\omega_{n}(\vec{c})=\sum_{l=1}^{L}c_{l}~{}\mathrm{mod}~{}n.$ (22)

Clearly, we have $\omega_{n}(\vec{c})\in\mathbb{Z}_{n}$ for any
$\vec{c}\in\mathbb{Z}_{n}^{L}$.

For the state $\rho_{\vec{c}}$ prepared from the ensemble
$\mathcal{E}^{\otimes L}$, let us consider the situation of guessing
$\omega_{n}(\vec{c})$ of the prepared state $\rho_{\vec{c}}$ from
$\mathcal{E}^{\otimes L}$. This situation is equivalent to discriminating the
states $\rho_{0}^{(L)},\ldots,\rho_{n-1}^{(L)}$ prepared with the
probabilities $\eta_{0}^{(L)},\ldots,\eta_{n-1}^{(L)}$, respectively, where

$\displaystyle\eta_{i}^{(L)}=\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}\eta_{\vec{c}},~{}\rho_{i}^{(L)}=\frac{1}{\eta_{i}^{(L)}}\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}\eta_{\vec{c}}\rho_{\vec{c}}.$ (23)

In other words, quantum state discrimination of the ensemble

$\mathcal{E}^{(L)}=\\{\eta_{i}^{(L)},\rho_{i}^{(L)}\\}_{i=0}^{n-1}.$ (24)

For a given two-state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{1}$, the following theorem gives
the closed-form expression of $q_{\rm G}(\mathcal{E}^{(L)})$. The proof of
Theorem 3 is given in Appendix A.

######  Theorem 3.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{1}$ and any positive integer $L$,

$\displaystyle q_{\rm G}(\mathcal{E}^{(L)})$ $\displaystyle=$
$\displaystyle\frac{1}{2}+\frac{1}{2}(\operatorname{Tr}|\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}}|)^{L}$
(25) $\displaystyle=$ $\displaystyle\frac{1}{2}+\frac{1}{2}(2\cdot q_{\rm
G}(\mathcal{E})-1)^{L}.$

Moreover, we have

$p_{\rm L}(\mathcal{E}^{(L)})=q_{\rm G}(\mathcal{E}^{(L)})$ (26)

if $p_{\rm L}(\mathcal{E})=q_{\rm G}(\mathcal{E})$.

######  Example 1.

For a NPT quantum state $\sigma$, let us consider the two-state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{1}$ consisting of

$\displaystyle\eta_{0}=\frac{\operatorname{Tr}|\sigma^{\operatorname{PT}}|+1}{2\operatorname{Tr}|\sigma^{\operatorname{PT}}|},$
$\displaystyle~{}~{}\rho_{0}=\frac{|\sigma^{\operatorname{PT}}|+\sigma^{\operatorname{PT}}}{\operatorname{Tr}|\sigma^{\operatorname{PT}}|+1},$
$\displaystyle\eta_{1}=\frac{\operatorname{Tr}|\sigma^{\operatorname{PT}}|-1}{2\operatorname{Tr}|\sigma^{\operatorname{PT}}|},$
$\displaystyle~{}~{}\rho_{1}=\frac{|\sigma^{\operatorname{PT}}|-\sigma^{\operatorname{PT}}}{\operatorname{Tr}|\sigma^{\operatorname{PT}}|-1}.$
(27)

We first note that $\operatorname{Tr}|\sigma^{\operatorname{PT}}|>1$ because
$\sigma$ is a NPT state.

It is straightforward to verify

$\operatorname{Tr}|\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}}|=\frac{1}{\operatorname{Tr}|\sigma^{\operatorname{PT}}|}.$
(28)

From Theorem 1 together with Eqs. (1) and (28), we have

$q_{\rm
G}(\mathcal{E})=\frac{\operatorname{Tr}|\sigma^{\operatorname{PT}}|+1}{2\operatorname{Tr}|\sigma^{\operatorname{PT}}|}=\eta_{0}.$
(29)

We also note that

$\eta_{0}\leqslant p_{\rm L}(\mathcal{E})\leqslant q_{\rm
G}(\mathcal{E})=\eta_{0},$ (30)

where the first inequality is by Inequality (6), the second inequality is due
to Inequality (8), and the last equality is from Eq. (29). Thus, we have

$p_{\rm L}(\mathcal{E})=q_{\rm G}(\mathcal{E})=\eta_{0}.$ (31)

From Theorem 3 together with Eqs. (28) and (31), we have

$p_{\rm L}(\mathcal{E}^{(L)})=q_{\rm
G}(\mathcal{E}^{(L)})=\frac{1}{2}+\frac{1}{2(\operatorname{Tr}|\sigma^{\operatorname{PT}}|)^{L}}$
(32)

for any positive integer $L$.

Theorem 3 provides the closed-form expression of $q_{\rm
G}(\mathcal{E}^{(L)})$ for the case of two-state ensemble $\mathcal{E}$. The
following theorem provides an upper bound of $q_{\rm G}(\mathcal{E}^{(L)})$
for a bipartite quantum state ensemble $\mathcal{E}$ with an arbitrary number
of states. The proof of Theorem 4 is given in Appendix B.

######  Theorem 4.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ and any positive integer
$L$, we have

$q_{\rm G}(\mathcal{E}^{(L)})\leqslant\frac{1}{n}+\frac{n-1}{n}(n\cdot q_{\rm
G}(\mathcal{E})-1)^{L}.$ (33)

Due to Theorem 4, we have

$\displaystyle\frac{1}{n}$ $\displaystyle\leqslant$ $\displaystyle p_{\rm
L}(\mathcal{E}^{(L)})\leqslant q_{\rm G}(\mathcal{E}^{(L)})$ (34)
$\displaystyle\leqslant$ $\displaystyle\frac{1}{n}+\frac{n-1}{n}(n\cdot q_{\rm
G}(\mathcal{E})-1)^{L},$

where the first inequality is due to Inequality (6), the second inequality is
by Inequality (8), and the last inequality is from Theorem 4. For the case of
$q_{\rm G}(\mathcal{E})<\frac{2}{n}$, the rightmost-hand side of Inequality
(34) decreases to $\frac{1}{n}$ as $L$ is getting larger. Thus, we have the
following corollary.

######  Corollary 1.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ with $q_{\rm
G}(\mathcal{E})<\frac{2}{n}$,

$\lim_{L\rightarrow\infty}p_{\rm L}(\mathcal{E}^{(L)})=\frac{1}{n}.$ (35)

We further note that the rightmost-hand side of Inequality (34) decreases to
$\frac{1}{n}$ exponentially fast with respect to $L$. Thus, the convergence of
$p_{\rm L}(\mathcal{E}^{(L)})$ to $\frac{1}{n}$ is also exponentially fast
with respect to $L$.

######  Example 2.

For positive integers $d,m,n$ with $d\geqslant 2$ and $2^{m-1}<n\leqslant
2^{m}$, let us consider the $d^{m}\otimes d^{m}$ quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ consisting of $m$-fold
tensor products of $d\otimes d$ quantum states
$(\mathbbm{1}\pm\mathcal{F})/(d^{2}\pm d)$,

$\displaystyle\eta_{i}$ $\displaystyle=$
$\displaystyle\frac{1}{\mathcal{N}}\prod_{k=1}^{m}(d^{2}+(-1)^{b_{k}(i)}d),$
$\displaystyle\rho_{i}$ $\displaystyle=$
$\displaystyle\bigotimes_{k=1}^{m}\frac{\mathbbm{1}+(-1)^{b_{k}(i)}\mathcal{F}}{d^{2}+(-1)^{b_{k}(i)}d}$
(36)

where $b_{k}(i)$ is the $k$th digit in the $m$-digit binary representation of
$i$,

$\sum_{k=1}^{m}b_{k}(i)\cdot 2^{k-1}=i,$ (37)

$\mathcal{F}$ is the flip operator in two-qudit system,

$\mathcal{F}=\sum_{i,j=0}^{d-1}\ket{ij}\\!\bra{ji},~{}$ (38)

and $\mathcal{N}$ is the normalization factor,

$\mathcal{N}=\sum_{i=0}^{n-1}\prod_{k=1}^{m}(d^{2}+(-1)^{b_{k}(i)}d).$ (39)

Note that $(\mathbbm{1}+\mathcal{F})/(d^{2}+d)$ and
$(\mathbbm{1}-\mathcal{F})/(d^{2}-d)$ are PPT and NPT Werner states in two-
qudit system, respectively[33].

It is straightforward to verify

$\displaystyle(\mathbbm{1}+\mathcal{F})^{\operatorname{PT}}=s_{0}^{(0)}\Pi_{0}+s_{1}^{(0)}\Pi_{1},$
$\displaystyle(\mathbbm{1}-\mathcal{F})^{\operatorname{PT}}=s_{0}^{(1)}\Pi_{0}+s_{1}^{(1)}\Pi_{1},$
$\displaystyle s_{0}^{(0)}=1+d,~{}s_{1}^{(0)}=1,$ $\displaystyle
s_{0}^{(1)}=1-d,~{}s_{1}^{(1)}=1$ (40)

where $\Pi_{0}$ and $\Pi_{1}$ are mutually orthogonal projection operators
such that

$\Pi_{0}=\frac{1}{d}\sum_{i,j=0}^{d-1}\ket{ii}\\!\bra{jj},~{}\Pi_{1}=\mathbbm{1}-\frac{1}{d}\sum_{i,j=0}^{d-1}\ket{ii}\\!\bra{jj}.$
(41)

For each $i=0,\ldots,n-1$, we can see from Eq. (40) that

$\eta_{i}\rho_{i}^{\operatorname{PT}}=\frac{1}{\mathcal{N}}\sum_{\vec{a}\in\mathbb{Z}_{2}^{m}}s_{\vec{a}}^{(i)}\Pi_{\vec{a}}$
(42)

where

$s_{\vec{a}}^{(i)}=\prod_{k=1}^{m}s_{a_{1}}^{(b_{k}(i))},\,\Pi_{\vec{a}}=\bigotimes_{k=1}^{m}\Pi_{a_{k}},\,\vec{a}=(a_{1},\ldots,a_{m}).$
(43)

To obtain $q_{\rm G}(\mathcal{E})$, we use the measurement
$\\{M_{i}\\}_{i=0}^{n-1}$ with $M_{0}=\mathbbm{1}^{\otimes m}$. For each
$i=0,\ldots,n-1$, we have

$\displaystyle\sum_{j=0}^{n-1}\eta_{j}\rho_{j}^{\operatorname{PT}}M_{j}-\eta_{i}\rho_{i}^{\operatorname{PT}}$
(44) $\displaystyle=$
$\displaystyle\frac{1}{\mathcal{N}}\sum_{\vec{a}\in\mathbb{Z}_{2}^{m}}(s_{\vec{a}}^{(0)}-s_{\vec{a}}^{(i)})\Pi_{\vec{a}}\succeq
0,$

where the equality is from Eq. (42) and the positivity is due to
$s_{\vec{a}}^{(0)}\geqslant s_{\vec{a}}^{(i)}$ for all $i=0,\ldots,n-1$ and
all $\vec{a}\in\mathbb{Z}_{2}^{m}$. Thus, Proposition 1 leads us to

$q_{\rm
G}(\mathcal{E})=\sum_{i=0}^{n-1}\eta_{i}\operatorname{Tr}(\rho_{i}^{\operatorname{PT}}M_{i})=\eta_{0},$
(45)

which implies $p_{\rm L}(\mathcal{E})=q_{\rm G}(\mathcal{E})$ because

$\eta_{0}\leqslant p_{\rm L}(\mathcal{E})\leqslant q_{\rm
G}(\mathcal{E})=\eta_{0},$ (46)

where the first inequality is due to Inequality (6), the second inequality is
from Inequality (8), and the equality is by Eq. (45).

We also note that

$\displaystyle\eta_{0}=\frac{(d^{2}+d)^{m}}{\mathcal{N}}$ $\displaystyle<$
$\displaystyle\frac{(d^{2}+d)^{m}}{n(d^{2}-d)^{m}}$ (47) $\displaystyle=$
$\displaystyle\frac{1}{n}\Big{(}1+\frac{2}{d-1}\Big{)}^{m},$

where the inequality is because

$\prod_{k=1}^{m}(d^{2}+(-1)^{b_{k}(i)}d)>(d^{2}-d)^{m}$ (48)

for any $i\in\\{0,\ldots,n-1\\}$ with $i\neq 2^{m}-1$. From Eq. (45) and
Inequality (47), we have

$q_{\rm G}(\mathcal{E})<\frac{1}{n}\Big{(}1+\frac{2}{d-1}\Big{)}^{m}.$ (49)

Thus, we have $q_{\rm G}(\mathcal{E})<\frac{2}{n}$ for

$d\geqslant\frac{\sqrt[m]{2}+1}{\sqrt[m]{2}-1}.$ (50)

In this case, Corollary 1 tells us that $p_{\rm L}(\mathcal{E}^{(L)})$
converges to $\tfrac{1}{n}$ as $L$ is getting larger. For the ensemble
$\mathcal{E}$ in Eq. (36) with $(m,n,d)=(2,3,6)$, $(3,6,9)$, and $(4,9,12)$,
Fig. 1 illustrates the decreasing of an upper bound of $p_{\rm
L}(\mathcal{E}^{(L)})$, the right-hand side of Inequality (33), with respect
to $L$.

Figure 1: Decreasing of the upper bound of $p_{\rm L}(\mathcal{E}^{(L)})$ in
Inequality (33) for the ensemble $\mathcal{E}$ of Example 2 with
$(m,n,d)=(2,3,6)$, $(3,6,9)$, and $(4,9,12)$. The upper bounds decrease to
$\frac{1}{n}$ exponentially fast with respect to $L$.

###  IV.2 Quantum data hiding

In information theory, data hiding is the protection of a data by sharing it
to multiple users so that the data only becomes accessible as a consequence of
their cooperative action. By using particular bipartite quantum states, there
have been several quantum data-hiding schemes proposed to share a data between
two parties, Alice and Bob[21, 22, 23, 24, 25, 26, 27]. If Alice and Bob can
only perform LOCC measurements, they can only obtain arbitrarily little
information about the hidden data. They can access the data only by global
measurements, which require either a quantum channel, shared quantum
entanglement, or direct interaction between them.

Figure 2: Data-hiding scheme based on a bipartite orthogonal quantum state
ensemble $\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ with $q_{\rm
G}(\mathcal{E})<\frac{2}{n}$ (Step 1). (a) Hider first prepares the state
$\rho_{\vec{c}}$ with the corresponding probability $\eta_{\vec{c}}$ from the
$L$-fold ensemble of $\mathcal{E}$. (b) Hider sends $\rho_{\vec{c}}$ to Alice
and Bob.

Here, we establish a relation between a bipartite quantum state ensemble and
its role in quantum data hiding. For a given bipartite quantum state ensemble
$\mathcal{E}$, we first provide a condition in terms of $p_{\rm
G}(\mathcal{E})$ and $q_{\rm G}(\mathcal{E})$ defined in Eqs. (3) and (7),
respectively. We show that $\mathcal{E}$ can be used to construct a quantum
data hiding scheme if $\mathcal{E}$ satisfies the condition. The data is
globally accessible and arbitrary little information about the data can be
obtained only by LOCC measurements. In other words, the data is perfectly
hidden asymptotically.

For any integer $n\geqslant 2$, let us consider a bipartite quantum state
ensemble $\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ satisfying the
following condition

$p_{\rm G}(\mathcal{E})=1,~{}~{}q_{\rm G}(\mathcal{E})<\frac{2}{n}.$ (51)

By using the ensemble $\mathcal{E}$, we construct a quantum data-hiding scheme
that can hide an $n$-ary classical data, that is, one in $\\{0,\ldots,n-1\\}$.

Hider, the third party, first choose a bipartite quantum state $\rho_{c_{1}}$
from $\mathcal{E}$ with the corresponding probability $\eta_{c_{1}}$ for
$c_{1}\in\mathbb{Z}_{n}$, and shares it to Alice and Bob. The hider repeats
this process $L$ times, so that $\rho_{c_{l}}$ is chosen with probability
$\eta_{c_{l}}$ in the $l$th repetition for $l=1,\ldots,L$. This whole
procedure is equivalent to the situation that the hider first prepares the
state

$\rho_{\vec{c}}=\rho_{c_{1}}\otimes\cdots\otimes\rho_{c_{L}}$ (52)

from the $L$-fold ensemble $\mathcal{E}^{\otimes L}$ in Eq. (21) with the
corresponding probability $\eta_{\vec{c}}$ for
$\vec{c}=(c_{1},\ldots,c_{L})\in\mathbb{Z}_{n}^{L}$, and sends it to Alice and
Bob. This step is illustrated in Fig. 2.

To hide a classical data $x\in\mathbb{Z}_{n}$, the hider broadcasts the
classical information $z$ to Alice and Bob,

$z=x\oplus y$ (53)

where $\oplus$ is the modulo-$n$ addition and

$y=\omega_{n}(\vec{c})$ (54)

for the prepared state $\rho_{\vec{c}}$ in Eq. (52). We illustrate this step
in Fig. 3.

As the information $z$ in Eq. (53) was broadcasted to Alice and Bob, guessing
the data $x$ is equivalent to guessing the data $y$. Moreover, guessing the
data $y$ in Eq. (54) is equivalent to discriminating the states in Eq. (23)
due to the argument in the paragraph containing Eqs. (22) and (23). Thus, the
maximum average probability of correctly guessing the data $x$ is equal to the
maximum average probability of discriminating the states from the ensemble
$\mathcal{E}^{(L)}$ in Eq. (24), that is, $p_{\rm G}(\mathcal{E}^{(L)})$.
Similarly, the maximum average probability of correctly guessing the data $x$
only using LOCC measurements becomes $p_{\rm L}(\mathcal{E}^{(L)})$.

Figure 3: Data-hiding scheme based on a bipartite orthogonal quantum state
ensemble $\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ with $q_{\rm
G}(\mathcal{E})<\frac{2}{n}$ (Step 2). To hide the classical data
$x\in\\{0,\ldots,n-1\\}$, hider broadcasts $x\oplus y$ to Alice and Bob, where
$y$ is the modulo-$n$ summation of all entries in $\vec{c}$ for the prepared
state $\rho_{\vec{c}}$.

The condition $p_{\rm G}(\mathcal{E})=1$ in Eq. (51) means the states
$\rho_{i}$’s of $\mathcal{E}$ are mutually orthogonal. In this case, the
states $\rho_{\vec{c}}$’s of $\mathcal{E}^{\otimes L}$ in Eq. (20) are
mutually orthogonal, which also implies the mutual orthogonality of
$\rho_{i}^{(L)}$’s of $\mathcal{E}^{(L)}$ in Eq. (23). Thus, we have

$p_{\rm G}(\mathcal{E}^{(L)})=1.$ (55)

In other words, the data $x$ is accessible when Alice and Bob can collaborate
to use global measurements.

On the other hand, the condition $q_{\rm G}(\mathcal{E})<\frac{2}{n}$ in Eq.
(51) and Corollary 1 imply that $p_{\rm L}(\mathcal{E}^{(L)})$ can be
arbitrarily close to $\frac{1}{n}$ by choosing an appropriately large $L$.
Thus, the globally accessible data $x$ is perfectly hidden asymptotically if
Alice and Bob can only use LOCC measurements.

Let us recall the two-state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{1}$ in Example 1. Because
$|E|^{2}=E^{2}$ for any Hermitian operator $E$, we can easily check that
$\rho_{0}$ and $\rho_{1}$ in Eq. (1) are orthogonal, therefore the condition
$p_{\rm G}(\mathcal{E})=1$ in Eq. (51) holds. Moreover, the condition $q_{\rm
G}(\mathcal{E})<1$ in Eq. (51) holds due to Eq. (31). Thus, the ensemble
$\mathcal{E}$ of Example 1 can be used to construct a data-hiding scheme that
hides one classical bit.

Let us also recall the bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ in Example 2. Since
$\mathbbm{1}+\mathcal{F}$ and $\mathbbm{1}-\mathcal{F}$ are orthogonal, the
states of $\mathcal{E}$ in Eq. (36) are mutually orthogonal, therefore the
condition $p_{\rm G}(\mathcal{E})=1$ in Eq. (51) holds. From Inequality (49),
the condition $q_{\rm G}(\mathcal{E})<\frac{2}{n}$ in Eq. (51) also holds if
Inequality (50) is satisfied. Thus, the ensemble $\mathcal{E}$ of Example 2
can be used to construct a data-hiding scheme that hides an $n$-ary classical
data.

##  V Discussion

We have considered bipartite quantum state discrimination and established a
relation between nonlocal quantum state ensemble and quantum data hiding
processing. Using the bound $q_{\rm G}(\mathcal{E})$ on local minimum-error
discrimination of bipartite quantum states, we have provided a sufficient
condition for a bipartite orthogonal quantum state ensemble to be used to
construct a data-hiding scheme; to make a classical data globally accessible
but not locally, and the data is perfectly hidden asymptotically(Corollary 1).
Our results have been illustrated by examples in multidimensional bipartite
quantum systems(Examples 1 and 2).

We also remark that our results can be used to hide a classical data by
directly encoding it into a bipartite quantum state. For any integers
$n\geqslant 2$, $L\geqslant 1$ and a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ satisfying Condition (51),
let us consider the $n$ bipartite quantum states
$\rho_{0}^{(L)},\ldots,\rho_{n-1}^{(L)}$ in Eq. (23).

To hide an $n$-ary classical data $x\in\mathbb{Z}_{n}$, hider sends the state
$\rho_{x}^{(L)}$ to Alice and Bob. This situation is illustrated in Fig. 4,
which is equivalent to discrimination of the states
$\rho_{0}^{(L)},\ldots,\rho_{n-1}^{(L)}$ with the same prior probability
$\frac{1}{n}$. From the argument before Eq. (55), the condition $p_{\rm
G}(\mathcal{E})=1$ in Eq. (51) means that the states
$\rho_{0}^{(L)},\ldots,\rho_{n-1}^{(L)}$ are mutually orthogonal. Thus, we
have

$p_{\rm G}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})=1.$ (56)

In other words, the data $x$ is accessible when Alice and Bob can collaborate
to use global measurements.

Figure 4: Data-hiding scheme using $n$ bipartite quantum states
$\rho_{0}^{(L)},\ldots,\rho_{n-1}^{(L)}$ defined from a bipartite orthogonal
quantum state ensemble $\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ with
$q_{\rm G}(\mathcal{E})<\frac{2}{n}$. To hide the classical data
$x\in\\{0,\ldots,n-1\\}$, hider sends the bipartite quantum state
$\rho_{x}^{(L)}$ to Alice and Bob.

On the other hand, from Theorem 5 in Appendix C, the condition $q_{\rm
G}(\mathcal{E})<\frac{2}{n}$ in Eq. (51) implies that $p_{\rm
L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})$ can be arbitrarily close to
$\frac{1}{n}$ by choosing an appropriately large $L$. In other words, the
globally accessible data $x$ is perfectly hidden asymptotically if Alice and
Bob can only use LOCC measurements. Thus, the $n$ quantum states
$\rho_{0}^{(L)},\ldots,\rho_{n-1}^{(L)}$ can be used to construct a data-
hiding scheme that hide $n$-ary classical data by directly encoding them into
the quantum states.

Our results tells us that any bipartite orthogonal quantum state ensemble
$\mathcal{E}$ satisfying $q_{\rm G}(\mathcal{E})<\frac{2}{n}$ is useful in
quantum data hiding. Therefore, it is natural to ask whether $q_{\rm
G}(\mathcal{E})<\frac{2}{n}$ is also necessary for $\mathcal{E}$ to be used in
quantum data hiding. This question is related to the research on the
operational meaning of $q_{\rm G}(\mathcal{E})$ in terms of quantum data
hiding. It is also an interesting future work to extend our data-hiding scheme
to multi-party quantum systems with more than two users.

## Acknowledgments

This work was supported by a National Research Foundation of Korea(NRF) grant
funded by the Korean government(Ministry of Science and
ICT)(No.NRF2023R1A2C1007039).

##  Appendix A Proof of Theorem 3

###### Proof.

From the definitions of $\eta_{i}^{(L)}$ and $\rho_{i}^{(L)}$ in Eq. (23), we
can see that

$\displaystyle\eta_{i}^{(L)}\rho_{i}^{(L)}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\big{[}(\eta_{0}\rho_{0}+\eta_{1}\rho_{1})^{\otimes
L}$ (57) $\displaystyle+(-1)^{i}(\eta_{0}\rho_{0}-\eta_{1}\rho_{1})^{\otimes
L}\big{]}$

for each $i=0,1$, which implies

$\displaystyle\eta_{0}^{(L)}\rho_{0}^{(L)\operatorname{PT}}-\eta_{1}^{(L)}\rho_{1}^{(L)\operatorname{PT}}=(\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}})^{\otimes
L}.$ (58)

For the two-state ensemble
$\mathcal{E}^{(L)}=\\{\eta_{i}^{(L)},\rho_{i}^{(L)}\\}_{i=0}^{1}$, we have

$\displaystyle q_{\rm G}(\mathcal{E}^{(L)})$ $\displaystyle=$
$\displaystyle\frac{1}{2}+\frac{1}{2}\operatorname{Tr}\big{|}\eta_{0}^{(L)}\rho_{0}^{(L)\operatorname{PT}}-\eta_{1}^{(L)}\rho_{1}^{(L)\operatorname{PT}}\big{|}$
(59) $\displaystyle=$
$\displaystyle\frac{1}{2}+\frac{1}{2}\operatorname{Tr}\big{|}(\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}})^{\otimes
L}\big{|}$ $\displaystyle=$
$\displaystyle\frac{1}{2}+\frac{1}{2}\big{(}\operatorname{Tr}\big{|}\eta_{0}\rho_{0}^{\operatorname{PT}}-\eta_{1}\rho_{1}^{\operatorname{PT}}\big{|}\big{)}^{L},$

where the first equality is from Theorem 1, the second equality is by Eq.
(58), and the last equality is due to $|E^{\otimes L}|=|E|^{\otimes L}$ for
any Hermitian operator $E$.

Now, let us assume $p_{\rm L}(\mathcal{E})=q_{\rm G}(\mathcal{E})$. To prove
Eq. (26), it is enough to show

$p_{\rm L}(\mathcal{E}^{(L)})\geqslant q_{\rm G}(\mathcal{E}^{(L)})$ (60)

because $p_{\rm L}(\mathcal{E}^{(L)})\leqslant q_{\rm G}(\mathcal{E}^{(L)})$
due to Inequality (8).

Let $\mathcal{M}=\\{M_{i}\\}_{i=0}^{1}$ be a LOCC measurement realizing
$p_{\rm L}(\mathcal{E})$, and consider the measurement
$\mathcal{M}^{(L)}=\\{M_{i}^{(L)}\\}_{i=0}^{1}$ with

$M_{i}^{(L)}=\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{2}^{L}\\\
\omega_{2}(\vec{c})=i\end{subarray}}\bigotimes_{l=1}^{L}M_{c_{l}}.$ (61)

Note that $\mathcal{M}^{(L)}$ is a LOCC measurement because it can be realized
by performing the same LOCC measurement $\mathcal{M}$ for each of $L$
subsystems.

From the definition of $\omega_{2}(\vec{c})$ in Eq. (22), we have

$M_{i}^{(L)}=\frac{1}{2}\big{[}(M_{0}+M_{1})^{\otimes
L}+(-1)^{i}(M_{0}-M_{1})^{\otimes L}\big{]}$ (62)

for all $i=0,1$. Now, we note that the success probability of discriminating
the states from $\mathcal{E}^{(L)}$ using $\mathcal{M}^{(L)}$ is equal to
$q_{\rm G}(\mathcal{E}^{(L)})$ because

$\displaystyle\sum_{i=0}^{1}\eta_{i}^{(L)}\operatorname{Tr}(\rho_{i}^{(L)}M_{i}^{(L)})$
(63) $\displaystyle=$
$\displaystyle\frac{1}{2}\operatorname{Tr}\Big{[}(\eta_{0}\rho_{0}+\eta_{1}\rho_{1})^{\otimes
L}(M_{0}+M_{1})^{\otimes L}\Big{]}$
$\displaystyle+\frac{1}{2}\operatorname{Tr}\Big{[}(\eta_{0}\rho_{0}-\eta_{1}\rho_{1})^{\otimes
L}(M_{0}-M_{1})^{\otimes L}\Big{]}$ $\displaystyle=$
$\displaystyle\frac{1}{2}\operatorname{Tr}\Big{[}\sum_{i=0}^{1}\eta_{i}\rho_{i}\Big{]}^{\otimes
L}+\frac{1}{2}\operatorname{Tr}\Big{[}\sum_{i=0}^{1}\eta_{i}\rho_{i}(2M_{i}-\mathbbm{1})\Big{]}^{\otimes
L}$ $\displaystyle=$
$\displaystyle\frac{1}{2}+\frac{1}{2}\Big{[}2\sum_{i=0}^{1}\eta_{i}\operatorname{Tr}(\rho_{i}M_{i})-1\Big{]}^{L}$
$\displaystyle=$ $\displaystyle\frac{1}{2}+\frac{1}{2}\big{(}2q_{\rm
G}(\mathcal{E})-1\big{)}^{L}=q_{\rm G}(\mathcal{E}^{(L)}),$

where the first equality is from Eqs. (57) and (62), the second equality is
due to $M_{0}+M_{1}=\mathbbm{1}$, the third equality is by
$\sum_{i=0}^{1}\operatorname{Tr}(\eta_{i}\rho_{i})=1$, the fourth equality
follows from the assumption of $\mathcal{M}$ and $p_{\rm
L}(\mathcal{E})=q_{\rm G}(\mathcal{E})$, and the last equality is from Theorem
1 together with Eq. (59). The success probability in Eq. (63) is obviously a
lower bound of $p_{\rm L}(\mathcal{E}^{(L)})$, therefore Inequality (60)
holds. ∎

##  Appendix B Proof of Theorem 4

Let $\mathcal{M}=\\{M_{i}\\}_{i=0}^{n-1}$ be a measurement providing $q_{\rm
G}(\mathcal{E})$, and consider

$\Gamma=\sum_{i=0}^{n-1}\eta_{i}\rho_{i}^{\operatorname{PT}},~{}H=\sum_{i=0}^{n-1}\eta_{i}\rho_{i}^{\operatorname{PT}}M_{i}.$
(64)

From Proposition 1, we have

$H-\eta_{i}\rho_{i}^{\operatorname{PT}}=\sum_{j=0}^{n-1}\eta_{j}\rho_{j}^{\operatorname{PT}}M_{j}-\eta_{i}\rho_{i}^{\operatorname{PT}}\succeq
0$ (65)

for all $i=0,\ldots,n-1$. Let

$W_{i}=H-\eta_{i}\rho_{i}^{\operatorname{PT}}\succeq 0$ (66)

for each $i=0,\ldots,n-1$, and define the positive semidefinite operator

$W_{\vec{c}}=\bigotimes_{l=1}^{L}W_{c_{l}}\succeq 0$ (67)

for each $\vec{c}=(c_{1},\ldots,c_{L})\in\mathbb{Z}_{n}^{L}$ with the $l$th
coordinate $c_{l}$.

We first note that for each $i=0,\ldots,n-1$,

$\displaystyle\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}W_{\vec{c}}+\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})\neq i\end{subarray}}W_{\vec{c}}$ $\displaystyle=$
$\displaystyle\Big{(}\sum_{j=0}^{n-1}W_{j}\Big{)}^{\otimes L}$ (68)
$\displaystyle=$ $\displaystyle(nH-\Gamma)^{\otimes L},$

where the first equality is from the definition of $W_{\vec{c}}$ in Eq. (67)
and the last equality is by Eqs. (64) and (66).

######  Lemma 1.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ and for any positive integer
$L$,

$\displaystyle\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}W_{\vec{c}}-(-1)^{L}\eta_{i}^{(L)}\rho_{i}^{(L)\operatorname{PT}}$
$\displaystyle=\frac{1}{n}(nH-\Gamma)^{\otimes
L}-(-1)^{L}\frac{1}{n}\Gamma^{\otimes L}.$ (69)

where $H$ and $\Gamma$ are in Eq. (64), $W_{\vec{c}}$ is in Eq. (67), and
$\eta_{i}^{(L)}$ and $\rho_{i}^{(L)}$ are in Eq. (23).

###### Proof.

We first remark the following relation

$\displaystyle\bigotimes_{l=1}^{L}A_{l}-\bigotimes_{l=1}^{L}B_{l}$ (70)
$\displaystyle=$ $\displaystyle\sum_{k=1}^{L}A_{1}\otimes\cdots\otimes
A_{L-k}\otimes(A_{L-k+1}-B_{L-k+1})$ $\displaystyle\otimes
B_{L-k+2}\otimes\cdots\otimes B_{L}$

for any operators $A_{l}$’s and $B_{l}$’s. For each $i=0,\ldots,n-1$, we can
see that

$\displaystyle\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}W_{\vec{c}}-(-1)^{L}\eta_{i}^{(L)}\rho_{i}^{(L)\operatorname{PT}}$
$\displaystyle=$
$\displaystyle\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}\Bigg{[}\bigotimes_{l=1}^{L}W_{c_{l}}-\bigotimes_{l=1}^{L}(-\eta_{c_{l}}\rho_{c_{l}}^{\operatorname{PT}})\Bigg{]}$
$\displaystyle=$
$\displaystyle\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}\sum_{k=1}^{L}W_{c_{1}}\otimes\cdots\otimes
W_{c_{L-k}}\otimes H$
$\displaystyle\otimes(-\eta_{c_{L-k+2}}\rho_{c_{L-k+2}}^{\operatorname{PT}})\otimes\cdots\otimes(-\eta_{c_{L}}\rho_{c_{L}}^{\operatorname{PT}})$
$\displaystyle=$
$\displaystyle\sum_{k=1}^{L}(\sum_{j\in\mathbb{Z}_{n}}W_{j})^{\otimes
L-k}\otimes
H\otimes\Big{[}\sum_{j\in\mathbb{Z}_{n}}(-\eta_{j}\rho_{j}^{\operatorname{PT}})\Big{]}^{\otimes
k-1},$

where the first equality is due to Eqs. (23) and (67), the second equality is
by Eq. (70) together with Eq. (66). The last equality of Eq. (B) holds because

$\displaystyle\\{\vec{c}\in\mathbb{Z}_{n}^{L}\,|\,\omega_{n}(\vec{c})=i\\}$
(72) $\displaystyle=$
$\displaystyle\\{(c_{1},\ldots,c_{L})\,|\,c_{1}\in\mathbb{Z}_{n},\ldots,c_{L-k}\in\mathbb{Z}_{n},$
$\displaystyle c_{L-k+2}\in\mathbb{Z}_{n},\ldots,c_{L}\in\mathbb{Z}_{n},$
$\displaystyle c_{L-k+1}=i-\mbox{$\sum_{l=1,l\neq
L-k+1}^{L}c_{l}$}~{}\mbox{mod}~{}n\\}$

for all $i=0,\ldots,n-1$ and all $k=1,\ldots,L$.

The last term of Eq. (B) can be rewritten as

$\displaystyle\sum_{k=1}^{L}(\sum_{j\in\mathbb{Z}_{n}}W_{j})^{\otimes
L-k}\otimes
H\otimes\Big{[}\sum_{j\in\mathbb{Z}_{n}}(-\eta_{j}\rho_{j}^{\operatorname{PT}})\Big{]}^{\otimes
k-1}$ $\displaystyle=$ $\displaystyle\sum_{k=1}^{L}(nH-\Gamma)^{\otimes
L-k}\otimes H\otimes(-\Gamma)^{\otimes k-1}$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{k=1}^{L}(nH-\Gamma)^{\otimes
L-k}\otimes(nH-\Gamma-(-\Gamma))\otimes(-\Gamma)^{\otimes k-1}$
$\displaystyle=$ $\displaystyle\frac{1}{n}\Big{[}(nH-\Gamma)^{\otimes
L}-(-\Gamma)^{\otimes L}\Big{]}.$ (73)

where the first equality is from Eqs. (64) and (66) and the last equality is
due to Eq. (70). Equations (B) and (73) lead us to Eq. (69). ∎

Now, let us consider

$H^{(L)}=\begin{dcases}\tfrac{1}{n}\Gamma^{\otimes
L}+\tfrac{1}{n}(nH-\Gamma)^{\otimes L},&L:\mathrm{odd},\\\
\tfrac{1}{n}\Gamma^{\otimes L}+\tfrac{n-1}{n}(nH-\Gamma)^{\otimes
L},&L:\mathrm{even}.\end{dcases}$ (74)

From Lemma 1 together with Eq. (68), we have

$H^{(L)}-\eta_{i}^{(L)}\rho_{i}^{(L)\operatorname{PT}}=\begin{dcases}\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})=i\end{subarray}}W_{\vec{c}},&L:\mathrm{odd}\\\
\sum_{\begin{subarray}{c}\vec{c}\in\mathbb{Z}_{n}^{L}\\\
\omega_{n}(\vec{c})\neq
i\end{subarray}}W_{\vec{c}},&L:\mathrm{even}\end{dcases}$ (75)

for all $i=0,\ldots,n-1$.

Due to the non-negativity of $W_{\vec{c}}$ in (67), Eq. (75) implies

$H^{(L)}-\eta_{i}^{(L)}\rho_{i}^{(L)\operatorname{PT}}\succeq 0.$ (76)

In other words, $H^{(L)}$ satisfies the condition of Theorem 2 for the
ensemble $\mathcal{E}^{(L)}$ in Eq. (24), therefore its trace provides an
upper bound of $q_{\rm G}(\mathcal{E}^{(L)})$,

$q_{\rm G}(\mathcal{E}^{(L)})\leqslant\operatorname{Tr}H^{(L)}.$ (77)

As $\operatorname{Tr}\Gamma=1$ and $\operatorname{Tr}H=q_{\rm
G}(\mathcal{E})$, we have

$\displaystyle\operatorname{Tr}H^{(L)}$ $\displaystyle=$
$\displaystyle\begin{dcases}\tfrac{1}{n}+\tfrac{1}{n}(n\cdot q_{\rm
G}(\mathcal{E})-1)^{L},&L:\mathrm{odd}\\\ \tfrac{1}{n}+\tfrac{n-1}{n}(n\cdot
q_{\rm G}(\mathcal{E})-1)^{L},&L:\mathrm{even}\end{dcases}$ (78)
$\displaystyle\leqslant$ $\displaystyle\frac{1}{n}+\frac{n-1}{n}(n\cdot q_{\rm
G}(\mathcal{E})-1)^{L},$

where the equality is due to Eq. (74) and the inequality is from the non-
negativity of $n\cdot q_{\rm G}(\mathcal{E})-1$. Inequalities (77) and (78)
lead us to Inequality (33), which completes the proof.

##  Appendix C Convergence of $p_{\rm
L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})$

######  Theorem 5.

For a bipartite quantum state ensemble
$\mathcal{E}=\\{\eta_{i},\rho_{i}\\}_{i=0}^{n-1}$ with $q_{\rm
G}(\mathcal{E})<\frac{2}{n}$,

$\lim_{L\rightarrow\infty}p_{\rm
L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})=\frac{1}{n}.$ (79)

###### Proof.

For the quantum states $\rho_{0}^{(L)},\ldots,\rho_{n-1}^{(L)}$ in Eq. (23),
we will show that

$\displaystyle p_{\rm L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})$ (80)
$\displaystyle\leqslant$
$\displaystyle\frac{1}{n}+\frac{(n-1)(n^{2}-n+2)}{2n}(n\cdot q_{\rm
G}(\mathcal{E})-1)^{L}.$

Because $q_{\rm G}(\mathcal{E})<\frac{2}{n}$, the right-hand side of
Inequality (80) converges to $\tfrac{1}{n}$ as $L$ is getting larger.
Moreover, $p_{\rm L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})$ is
bounded below by $\frac{1}{n}$ from Inequality (6). Therefore, Inequality (80)
leads us to Eq. (79).

Let $\mathcal{M}=\\{M_{i}\\}_{i=0}^{n-1}$ be a LOCC measurement realizing
$p_{\rm L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})$. From the
definition of $p_{\rm L}(\mathcal{E}^{(L)})$ in Eq. (4), we have

$\displaystyle p_{\rm L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})-p_{\rm
L}(\mathcal{E}^{(L)})$ (81) $\displaystyle=$
$\displaystyle\sum_{i=0}^{n-1}\frac{1}{n}\operatorname{Tr}(\rho_{i}^{(L)}M_{i})-p_{\rm
L}(\mathcal{E}^{(L)})$ $\displaystyle\leqslant$
$\displaystyle\sum_{i=0}^{n-1}\Big{(}\frac{1}{n}-\eta_{i}^{(L)}\Big{)}\operatorname{Tr}(\rho_{i}^{(L)}M_{i}).$

Since $0\leqslant\operatorname{Tr}(\rho_{i}^{(L)}M_{i})\leqslant 1$ for any
$i\in\\{0,\ldots,n-1\\}$ and $r\leqslant\frac{1}{2}(|r|+r)$ for any real
number $r$, we have

$\displaystyle\sum_{i=0}^{n-1}\Big{(}\frac{1}{n}-\eta_{i}^{(L)}\Big{)}\operatorname{Tr}(\rho_{i}^{(L)}M_{i})$
(82) $\displaystyle\leqslant$
$\displaystyle\sum_{i=0}^{n-1}\frac{1}{2}\Bigg{[}\Big{|}\frac{1}{n}-\eta_{i}^{(L)}\Big{|}+\Big{(}\frac{1}{n}-\eta_{i}^{(L)}\Big{)}\Bigg{]}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{i=0}^{n-1}\Big{|}\frac{1}{n}-\eta_{i}^{(L)}\Big{|},$

where the last equality holds from $\sum_{i=0}^{n-1}\eta_{i}^{(L)}=1$. Thus,
we have

$\displaystyle p_{\rm L}(\\{\tfrac{1}{n},\rho_{i}^{(L)}\\}_{i=0}^{n-1})$
$\displaystyle\leqslant$ $\displaystyle p_{\rm
L}(\mathcal{E}^{(L)})+\frac{1}{2}\sum_{i=0}^{n-1}\Big{|}\frac{1}{n}-\eta_{i}^{(L)}\Big{|}$
$\displaystyle\leqslant$ $\displaystyle q_{\rm
G}(\mathcal{E}^{(L)})+\frac{1}{2}\sum_{i=0}^{n-1}\Big{|}\frac{1}{n}-\eta_{i}^{(L)}\Big{|}$
$\displaystyle\leqslant$ $\displaystyle\frac{1}{n}+\frac{n-1}{n}(n\cdot q_{\rm
G}(\mathcal{E})-1)^{L}+\frac{1}{2}\sum_{i=0}^{n-1}\Big{|}\frac{1}{n}-\eta_{i}^{(L)}\Big{|},$

where the first inequality is due to Inequalities (81) and (82), the second
inequality is by Inequality (8), and the last inequality is from Theorem 4.

To obtain an upper bound of
$\frac{1}{2}\sum_{i=0}^{n-1}|\frac{1}{n}-\eta_{i}^{(L)}|$, let us first
consider

$\displaystyle\max\\{\eta_{0}^{(L)},\ldots,\eta_{n-1}^{(L)}\\}$ (84)
$\displaystyle\leqslant$ $\displaystyle p_{\rm L}(\mathcal{E}^{(L)})\leqslant
q_{\rm G}(\mathcal{E}^{(L)})$ $\displaystyle\leqslant$
$\displaystyle\frac{1}{n}+\frac{n-1}{n}(n\cdot q_{\rm G}(\mathcal{E})-1)^{L},$

where the first inequality is by Inequality (6), the second inequality is due
to Inequality (8), and the last inequality is from Theorem 4. Moreover, we
have

$\displaystyle\min\\{\eta_{0}^{(L)},\ldots,\eta_{n-1}^{(L)}\\}$ (85)
$\displaystyle\geqslant$ $\displaystyle
1-(n-1)\max\\{\eta_{0}^{(L)},\ldots,\eta_{n-1}^{(L)}\\}$
$\displaystyle\geqslant$ $\displaystyle\frac{1}{n}-\frac{(n-1)^{2}}{n}(n\cdot
q_{\rm G}(\mathcal{E})-1)^{L},$

where the first inequality is because $\\{\eta_{i}^{(L)}\\}_{i=0}^{n-1}$ is a
probability distribution, and the second inequality follows from Eq. (84).

From Inequalities (84) and (85), we have

$\Big{|}\frac{1}{n}-\eta_{i}^{(L)}\Big{|}\leqslant\frac{(n-1)^{2}}{n}(n\cdot
q_{\rm G}(\mathcal{E})-1)^{L}$ (86)

for each $i=0,\ldots,n-1$, which implies

$\frac{1}{2}\sum_{i=0}^{n-1}\Big{|}\frac{1}{n}-\eta_{i}^{(L)}\Big{|}\leqslant\frac{(n-1)^{2}}{2}(n\cdot
q_{\rm G}(\mathcal{E})-1)^{L}.$ (87)

Inequalities (C) and (87) lead us to Inequality (80). ∎

## References

  * [1] R. Horodecki, P. Horodecki, M. Horodecki, and K. Horodecki, Quantum entanglement, Rev. Mod. Phys. 81, 865 (2009).
  * [2] A. M. Childs, D. Leung, L. Mančinska, and M. Ozols, A framework for bounding nonlocality of state discrimination, Commun. Math. Phys. 323, 1121 (2013).
  * [3] N. Brunner, D. Cavalcanti, S. Pironio, V. Scarani, and S. Wehner, Bell nonlocality, Rev. Mod. Phys. 86, 419 (2014).
  * [4] R. Uola, A. C. S. Costa, H. C. Nguyen, and O. Gühne, Quantum steering, Rev. Mod. Phys. 92, 015001 (2020).
  * [5] A. K. Ekert, Quantum cryptography based on Bell’s theorem, Phys. Rev. Lett. 67, 661 (1991).
  * [6] C. H. Bennett, G. Brassard, C. Crépeau, R. Jozsa, A. Peres, and W. K. Wootters, Teleporting an unknown quantum state via dual classical and Einstein-Podolsky-Rosen channels, Phys. Rev. Lett. 70, 1895 (1993).
  * [7] E. Chitambar and G. Gour, Quantum resource theories, Rev. Mod. Phys. 91, 025001 (2019).
  * [8] A. Streltsov, G. Adesso, and M. B. Plenio, Colloquium: Quantum coherence as a resource, Rev. Mod. Phys. 89, 041003 (2017).
  * [9] D. Mondal, T. Pramanik, and A. K. Pati, Nonlocal advantage of quantum coherence, Phys. Rev. A 95, 010301 (2017).
  * [10] M.-L. Hu and H. Fan, Nonlocal advantage of quantum coherence in high-dimensional states, Phys. Rev. A 98, 022312 (2018).
  * [11] A. Chefles, Quantum state discrimination, Contemp. Phys. 41, 401 (2000).
  * [12] J. A. Bergou, Quantum state discrimination and selected applications, J. Phys.: Conf. Ser. 84, 012001 (2007).
  * [13] S. M. Barnett and S. Croke, Quantum state discrimination, Adv. Opt. Photonics 1, 238 (2009).
  * [14] J. Bae and L.-C. Kwek, Quantum state discrimination and its applications, J. Phys. A: Math. Theor. 48, 083001 (2015).
  * [15] C. H. Bennett, D. P. DiVincenzo, C. A. Fuchs, T. Mor, E. Rains, P. W. Shor, J. A. Smolin, and W. K. Wootters, Quantum nonlocality without entanglement, Phys. Rev. A 59, 1070 (1999).
  * [16] S. Ghosh, G. Kar, A. Roy, A. Sen(De), and U. Sen, Distinguishability of Bell states, Phys. Rev. Lett. 87, 277902 (2001).
  * [17] J. Walgate and L. Hardy, Nonlocality, asymmetry, and distinguishing bipartite states, Phys. Rev. Lett. 89, 147901 (2002).
  * [18] E. Chitambar, D. Leung, L. Mančinska, M. Ozols, and A. Winter, Everything you always wanted to know about LOCC (but were afraid to ask), Commun. Math. Phys. 328, 303 (2014).
  * [19] A. Peres and W. K. Wootters, Optimal detection of quantum information, Phys. Rev. Lett. 66, 1119 (1991).
  * [20] E. Chitambar and M.-H. Hsieh, Revisiting the optimal detection of quantum information, Phys. Rev. A 88, 020302(R) (2013).
  * [21] B. M. Terhal, D. P. DiVincenzo, and D. W. Leung, Hiding bits in bell states, Phys. Rev. Lett. 86, 5807 (2001).
  * [22] D. P. DiVincenzo, D. W. Leung, and B. M. Terhal, Quantum data hiding, IEEE Trans. Inf. Theory 48, 580 (2002).
  * [23] T. Eggeling and R. F. Werner, Hiding classical data in multipartite quantum states, Phys. Rev. Lett. 89, 097905 (2002).
  * [24] T. Morimae, Security of Terhal-DiVincenzo-Leung quantum-data-hiding protocol against inconclusive positive-operator-valued measure, Phys. Rev. A 87, 054302 (2013).
  * [25] M. Piani, V. Narasimhachar, and J. Calsamiglia, Quantumness of correlations, quantumness of ensembles and quantum data hiding, New. J. Phys. 16, 113001 (2014).
  * [26] L. Peng and F. Liu, Quantum advantage of resources in data hiding, Quantum Inf. Process. 20, 279 (2021).
  * [27] L. Lami, Quantum data hiding with continuous-variable systems, Phys. Rev. A 104, 052428 (2021).
  * [28] C. W. Helstrom, Quantum detection and estimation theory, J. Stat. Phys. 1, 231 (1969).
  * [29] J. Bae, Structure of minimum-error quantum state discrimination, New J. Phys. 15, 073037 (2013).
  * [30] A. Peres, Separability criterion for density matrices, Phys. Rev. Lett. 77, 1413 (1996).
  * [31] PPT property does not depend on the choice of basis or the subsystem to be transposed. For simplicity, we consider the standard basis and Bob’s system throughout this paper.
  * [32] D. Ha and J. S. Kim, Bound on local minimum-error discrimination of bipartite quantum states, Phys. Rev. A 105, 032421 (2022).
  * [33] G. Vidal and R. F. Werner, Computable measure of entanglement, Phys. Rev. A 65, 032314 (2002).


# The Pump Scheduling Problem:
A Real-World Scenario for Reinforcement Learning

Henrique Donâncio
Normandie Université, INSA Rouen
LITIS
Rouen, France
<EMAIL_ADDRESS>
Laurent Vercouter
Normandie Université, INSA Rouen
LITIS
Rouen, France
<EMAIL_ADDRESS>
Harald Roclawski
Technical University of Kaiserslautern
SAM
Kaiserlaustern, Germany
<EMAIL_ADDRESS>

###### Abstract

Deep Reinforcement Learning (DRL) has achieved remarkable success in scenarios
such as games and has emerged as a potential solution for control tasks. That
is due to its ability to leverage scalability and handle complex dynamics.
However, few works have targeted environments grounded in real-world settings.
Indeed, real-world scenarios can be challenging, especially when faced with
the high dimensionality of the state space and unknown reward function. We
release a testbed consisting of an environment simulator and demonstrations of
human operation concerning pump scheduling of a real-world water distribution
facility to facilitate research. The pump scheduling problem can be viewed as
a decision process to decide when to operate pumps to supply water while
limiting electricity consumption and meeting system constraints. To provide a
starting point, we release a well-documented codebase, present an overview of
some challenges that can be addressed and provide a baseline representation of
the problem. The code and dataset are available at
https://gitlab.com/hdonancio/pumpscheduling

##  1 Introduction

The pump scheduling problem is a decision process to decide when to operate
pumps where the objective is to supply water while limiting electricity
consumption and meeting safety constraints. The strategies adopted can vary
according to the particularities of the water distribution system. For
example, in some locations, the price of electricity may have different
tariffs throughout the day. In this case, storage tanks can supply water while
the pumps operate in off-peak time windows to reduce costs [1]. Moreover, the
scheduling strategies must avoid switching the pump operation too frequently
to protect the assets and provide water exchange in the tanks to preserve its
quality. Some works that address the pump scheduling problem use methods like
linear optimization, branch-and-bound, and genetic algorithms [2, 3, 4, 1, 5,
6]. However, some of these methods are limited to small water networks due to
their computational complexity. In other cases, the scheduling considers many
steps ahead with decisions based on observed patterns, not being able to
handle unexpected situations. To overcome this, in previous work [7] we have
shown that Deep Reinforcement Learning (DRL) has the potential to provide a
data-driven solution that can be robust and scalable.

Reinforcement Learning (RL) [8] is a decision-making process where the agent
learns to interact with the environment aiming to maximize returns. Recent
works include applications in autonomous driving [9], robotics [10, 11], video
games [12, 13, 14, 15], dialogue assistance [16], among others [17].
Currently, most of the research utilizes virtual environments such as Arcade
Learning Environment (ALE) [18] and DeepMind Control Suite [19], where
typically a reward function and the state representation are available.
However, in real-world problems, the agent usually has no indication of how
efficient are the actions it performs in the environment. In other cases, the
rewards are sparse and binary, only perceived when the agent achieves the task
accomplishment [20]. Besides this, most real-world scenarios have complex
dynamics with tasks consisting of multiple steps and a high-dimensional state
space [11]. However, few works have focused on problems that can be shared and
reproduced among the community and grounded in real-world settings. Some
examples include robotics applications, where the limitation is having access
to the assets.

We release in this work a simulator and a database related to pump operation
in a real-world water distribution system. The dataset has been gathered
through sensors over three years in 1-min timesteps from the water
distribution facility controlled by humans. Moreover, we detail the operation
constraints given by specialists and discuss the current strategy applied to
the system. Finally, we point out some of the challenges that can be addressed
and establish a baseline for the state representation and reward function
engineering. Our goal is that this testbed can represent a benchmark with
grounded real-world settings for RL branches, such as Learning from
Demonstrations (LfD), (safe) exploration, inverse RL, and state learning
representation, among others. We summarize the main contributions of this work
below:

  * •

We release a RL testbed grounded in real-world settings based on a water
distribution facility, containing a simulator of the system and a dataset of
human demonstrations collected over three years in 1-minute timesteps.

  * •

We point out the characteristics of the water system such as constraints and
the current scheduling strategy being applied to control it and provide a
baseline for problem representation as a Partially Observable Markov Decision
Process (POMDP).

##  2 Water Distribution System

Figure 1: Water distribution system overview. The system has four pumps with
fixed speed (ON/OFF) and two elevated tanks.

Figure 1 shows an overview of the water distribution system considered in this
work. The system collects and treats the raw water from wells before storing
it in a reservoir. In the water utility, four distribution pumps (NP1 to NP4)
of different sizes are available for pumping the water through the system
network into two storage tanks. The start/stop control operates the pumps.
Speed or throttle control are not utilized. At most, only one pump can be
running, and parallel operation is usually not applied. However, it is
possible in case of exceptionally high water demand. A decision process has to
decide on the operation of the most suitable pump regarding the water demand
forecast, energy consumption, water quality, security of supply, and
operational reliability. The tanks are located approximately 47m above the
pump station. Both storage containers have identical dimensions and can be
treated as a single tank with a storage volume of 16000m³. The maximum water
level is 10m. Thus, the geodetic height between pumps and tank is [47, 57]m.

In the water distribution system, the human operation (behavioral policy)
follows a strategy to handle the pump schedule. Figure 2 shows the effects of
this behavioral policy. As shown in Figures 2, 2, 2, the operation fills the
tank to a high level before the peak in water consumption (see Figures 2, 2,
2), and then they let it decrease along the day to provide water exchange and
keep water’s quality. A safety operation must guarantee the tank level with at
least 3m filled once it allows the system operators to handle unexpected
situations. Figures 2, 2, 2, shows the average daily pump switch per month.
Each pump switch is considered either ON to OFF or OFF to ON, counting +1.
Thus, we could say that the current pump operation generally uses each pump at
most once a day. Although it is difficult to measure the impact of a strategy
in preserving the system’s assets, the idea is to minimize the amount of
switching and provide a distribution pumps usage. Finally, in Figures 2, 2, 2
we show the electricity consumption where given the pumps settings, we can see
the prioritization of the use of pump NP2.

##  3 A Partially Observable Markov Decision Process for the Pump Scheduling
Problem

The RL problem statement can be formalized as a POMDP [8], defined by the
tuple ($\mathcal{S}$, $\mathcal{A}$, $\mathcal{P}$, $\mathcal{R}$, $\Omega$,
$\mathcal{O}$, $\gamma$), where:

  * •

$\mathcal{S}$ is the state space;

  * •

$\mathcal{A}$ is the action space;

  * •

$\mathcal{P}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\mapsto[0,1]$ is the
transition probability for being in some state $s\in\mathcal{S}$, perform an
action $a\in\mathcal{A}$ and reach a state $s^{\prime}\in\mathcal{S}$

  * •

$\mathcal{R}:\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ is the reward
function;

  * •

$\mathcal{O}:\mathcal{S}\times\mathcal{A}\times\Omega\mapsto[0,1]$ is the
probability to receive an observation $o\in\Omega$ about the next state
$s^{\prime}$ ;

  * •

$\Omega$ denotes the observation space;

  * •

$\gamma\in[0,1]$ is the discount factor that balances the relevance of
immediate reward over rewards in the future;

The objective in the PO(MDP) is to learn a policy $\pi:S\mapsto A$ aiming
maximize the expected discount rewards
$J(\pi)=\mathbb{E}[\sum_{t=1}^{T}\gamma^{t}r(s_{t},a_{t})]$. In previous work
[7], we cast the pump scheduling problem for a water system as an episodic
POMDP. We consider the daily cycle pattern observed in the water consumption
to define the episode length. Thus, each episode has a length of 1440
timesteps or one day of operation. The state/observation space and reward were
partially based on data that sensors could gather, such as the tank level,
water consumption, flow rate $Q$, and hydraulic head $H$. The action space
$\mathcal{A}$ represents the discrete set of pumps (NP1 to NP4) and the option
to turn all of them off (NOP). These pumps have different flow rates $Q$,
which leads to distinct electricity consumption $kW$. Thus, the decision
process consists of defining a policy that meets the water demand while
limiting electricity consumption and satisfying safety constraints. Below, we
review the state-action space and the proposed reward function.

State/Observation: The states/observation is represented by the
$tank~{}level_{(t)}\in[47,57]$ and a given $water~{}consumption_{(t)}$ for a
$time~{}of~{}day~{}(t)$, the respective $month$, the $action\in\mathcal{A}$ at
$t-1$, the cumulative $time~{}running\in[0,1440]$ of pumps along with an
episode, and a binary value called $water~{}quality$ that indicates if during
the episode the $tank~{}level$ achieved a level lower than 53m.

The state’s representation is responsible for providing information to achieve
the desired behavior through the reward function presented next. Thus, the
features were selected aiming to:

  * •

Provide information regarding the system’s current state, such as tank level
and water consumption;

  * •

Understand the cyclical behaviour of water consumption along the hours of the
day in different seasons of the year;

  * •

Lead the agent to avoid switching the pump operation as well as make a balance
of their use;

  * •

Provide information that could lead to preserving the water quality;

Action: The action space corresponds to the pumps available and the option to
turn them off, such that $\mathcal{A}=\\{$NP1, NP2, NP3, NP4, NPO$\\}$. These
pumps have different sizes which corresponds to distinct $Q$ and $kW$, being
NP1 > NP2 > NP3 > NP4. The action NOP has $Q,kW=0$.

Reward function: We defined two dense reward functions through Eq. 1 and 2,
where $P$ correspond to the cumulative time along an episode that some pump is
running. These reward functions intends to balance the sub-goals: (i) enhance
the efficiency of the pump operation; (ii) maintain safe levels of tank
operation; (iii) penalize switches and provide distribution of the use of the
pumps; (iv) maintain the water quality. We present in the Algorithms 1 and 2
how these reward are calculated.

Although we do not assume that these dense reward functions induce optimal
policies, handcrafting them requires a considerable engineering burden [21].
To overcome this and address new challenges whenever the case, we suggest
future research consider mimicking the behavioral policy assuming a goal-
oriented approach. The goals, for instance, could be designed to meet certain
tank levels by associating them with time windows, targeting the behavior
observed in Figures 2 2 2.

$r_{t}=e^{1/(-Q_{t}/kW_{t})}-B*\psi+\log(1/(P+\omega))$ (1)

$r_{t}=-e^{(-1/kW_{t})}-B*\psi+\log(1/(P+\omega))$ (2)

Algorithm 1 Reward Function Eq. 1

$\psi\leftarrow 10$

if $action_{(t-1)}=action_{(t)}$ or $time~{}running(action_{(t)})=0$ or
$action_{(t)}=$ NOP then

$\omega\leftarrow 1$

else

$\omega\leftarrow 30$ $\triangleright$ Penalize pump switch

end if

if $tank~{}level_{(t)}>49$ and $tank~{}level_{(t)}<50$ then

$B\leftarrow||tank~{}level_{(t)}-50||$

else if $tank~{}level_{(t)}\leq 49$ or $tank~{}level_{(t)}=57$ then

$B\leftarrow 1$

else if $tank~{}level_{(t)}\geq 50$ and $tank~{}level_{(t)}<53$ and not
$water~{}quality$ then

$B\leftarrow-1$

else

$B\leftarrow 0$

end if

if $action_{(t)}~{}\neq$ NOP then

$r_{(t)}\leftarrow
e^{(-Q_{t}/kW_{t})}-B\times\psi+\log(1/(time~{}running(action_{(t)}+\omega))$

else

$r_{(t)}\leftarrow-B\times\psi+\log(1/(time~{}running(action_{(t)})+\omega))$

end if

Algorithm 2 Reward Function Eq. 2

$\psi\leftarrow 10$

if $action_{(t-1)}=action_{(t)}$ or $time~{}running(action_{(t)})=0$ then

$\omega\leftarrow 1$

else

$\omega\leftarrow 30$ $\triangleright$ Penalize action switch

end if

if $tank~{}level_{(t)}>49$ and $tank~{}level_{(t)}<50$ then

$B\leftarrow||tank~{}level_{(t)}-50||$

else if $tank~{}level_{(t)}\leq 49$ or $tank~{}level_{(t)}=57$ then

$B\leftarrow 1$

else if $tank~{}level_{(t)}\geq 50$ and $tank~{}level_{(t)}<53$ and not
$water~{}quality$ then

$B\leftarrow-1$

else

$B\leftarrow 0$

end if

if $action_{(t)}~{}\neq~{}NOP$ then

$r_{(t)}\leftarrow-e^{(-1/kW_{t})}-B\times\psi+\log(1/(time~{}running(action_{(t)})+\omega))$

else

$r_{(t)}\leftarrow-B\times\psi+\log(1/(time~{}running(action_{(t)})+\omega))$

end if

Figure 2: Overview of human operation for the water system. 2 2 2 show the
water comsumption pattern. 2 2 2 show the tank level profile for the adopted
strategy. 2 2 2 are the pump switch daily average and 2 2 2 are the average
electricity consumption per month.

##  4 Casting the Water Distribution System Simulator into RL Settings

An RL agent, after applying an action $a$ in the state $s$, receives a reward
$r$ and reaches a new state $s^{\prime}$, what is called transition
$\tau=<s,a,r,s^{\prime}>$. Figure 3 presents how we model this dynamic for the
pump scheduling problem using the simulator. First, real water consumption
data at a given time $t$ is necessary as input to the simulator since it can
not be generated through new interactions. After that, we need to define an
initial condition for the tank level, among other features that compose the
state $s$ representation. Given a state $s$, we can then apply an action to
that state. In the codebase provided, we select the action looking at the real
data, although any other policy could be used. A pump $p$ is ON if its value
$kW_{p}(t)\neq 0$ in the dataset. If for all pumps $p$ the values $kW=0$, no
pump is in operation, and the resulting action is NOP.

Figure 3: The dynamics of an MDP and the RL learning process through the water
distribution system simulator.

Once an action $a$ is applied to the state $s$, the simulator outputs the
values of $Q_{p}$, $kW_{p}$, and $H_{p}$ for the respective timestep $t$ (see
details in A.2). These values can then used to calculate a reward $r(t)$.
Furthermore, a new tank level is observed at $t+1$, corresponding to state
$s^{\prime}$. Thus we have a loop in which transitions are generated and
stored in the Experience Replay [22], implemented with prioritization
mechanism [23]. Finally, we can use these experiences to learn a new policy
$\pi_{a}$, different from the behavioral policy $\pi_{b}$ which generated the
dataset. To do so, we provide an implementation of Random Ensemble Mixture
(REM) [24] as a learning algorithm. REM is dropout inspired convex combination
of $K$ parametric Q-functions aiming to mitigate overestimation in offline RL
settings. The underlying idea is for each policy update, generate a set of $k$
random weights $\alpha$ such that
$Q(s,a)^{k}_{\theta}=\\{\alpha_{1}Q(s,a)_{\theta_{1}}+\alpha_{2}Q(s,a)_{\theta_{2}}+\alpha_{3}Q(s,a)_{\theta_{3}}+...+\alpha_{k}Q(s,a)_{\theta_{k}}\\}$
and $\sum_{k}\alpha_{k}=1$.

##  5 Addressing Challenges

This section lists some challenges that can be addressed using the
simulator/dataset.

Modeling the problem. The pump scheduling problem does not have a prior state
representation or a single specified reward function. For this reason, state
representation learning [25, 26, 27] can explore more informative and low
dimensional state representations for efficient policy learning. The same
applies to reward engineering. Specifying a reward function is difficult once
it must encapsulate and balance multiple goals that might be contradictory. A
way to overcome this is through inverse RL [28] approaches to extract a reward
function from the demonstrations. Moreover, we presented a suggestion to model
the problem as a goal/task-oriented with sparse rewards to allow the
evaluation of approaches [20] that deal with the lack of reward signals.

Reset mechanism. We have modeled the pump scheduling as an episodic POMDP
where the agent interacts for a limited number of timesteps. Therefore, our
configuration is reset-free, which means no mechanism is applied to start an
episode under a known distribution of initial states. In other words, it means
that the last step of an episode is the initial one for the next episode.
Thus, a suitable policy must be robust to handle the variability of initial
conditions, providing dynamism challenge [29].

An exploration challenge. One of the constraints set for controlling the
system is to avoid switching the pump operation too frequently to protect the
assets. Thus, exhaustive exploration strategies such as epsilon-greedy [8] are
contradictory to the idea of performing the same action selection over many
timesteps and may prematurely converge to suboptimal policies. Furthermore,
approaches that attempt to cover unknown areas of state space can lead to
meaningless states that do not meet the safety constraints. Thus, the pump
scheduling problem can be a setting for evaluating exploration strategies that
aim to enhance sample efficiency or limit state visitation to those that meet
system safety constraints [30].

Learning from demonstrations. Some real-world settings may restrict the
learning process to logged data. This constraint is due to the difficulty of
creating accurate simulators or even safety constraints. For example, an
autonomous vehicle cannot learn by interacting with the environment through
trial-error, both for safety and cost reasons. For this, offline RL [17]
approaches such as BCQ [31, 32], REM [24], Conservative Q-learning [33], and
Averaged-DQN [34] were designed to increase the efficiency of exploiting
previously collected data, mainly tackling the overestimation phenomena [35].
This data can be, for instance, obtained by human operation in the real-world
system, as in the present case. In [7], we show that a large dataset is
capable of producing policies through offline approaches that can outperform
the behavioral policy. Thus, the present dataset can serve as a benchmark for
offline RL approaches. Finally, these expert demonstrations can guide the
agent’s actions through imitation learning [36], used to build models [37,
38], or boost the learning process [11].

Continuous action-space. The set of actions of the water distribution system
is discrete. There is a finite set of pumps that operate with a fixed speed
beside the option to turn all of them off. Although not explored in previous
works, the simulator can adapt to variable speed pump(s). Then, approaches
suitable for continuous action space, such as actor-critic [8], can be
evaluated while exploring the environment.

##  6 Related Works

We discuss in this section some related approaches found in the literature
regarding applied RL to control and pump scheduling optimization.

Control using Reinforcement Learning. In [39] the authors use Deep Recurrent
Q-Networks [40] in the scenario of a smart grid. The objective is to develop a
pricing strategy to maximize broker’s (agents) profits. It is proposed a
reward shaping strategy once customers are clustered according to their
consumption patterns and managed by their respective sub-brokers. Then, a
mechanism for credit assignment was necessary to indicate the contribution of
each sub-broker to the global return. Wei and colleagues [41] proposes a data-
driven solution to control HVAC (heating, ventilation and air conditioning)
system. The objective is to control the environment temperature by handling
numerous disturbances with real-time data. The possible actions are a discrete
set of airflow rates divided by building zone. Then, the approach split each
zone controlling it by distinct artificial neural networks. Sivakumar et al.
[42] propose a network control strategy that acts asynchronously with the
environment. The delay, which authors call by $\delta$, corresponds to the
policy lookup time when selecting an action. Meanwhile, the network proceeds
transmitting data in the interval [t, t + $\delta$]. Thus, the transitions are
not only dependent on the state and action at some timestep $t$ but also the
previous action at $t-1$. In [43] the problem of congestion control in
networks is also addressed, and a testbed was released. In [44], the authors
applied RL to a flight controller of a stratospheric balloon which must handle
a partially observable environment and continuous interaction. Finally,
Degrave et al. [45] use RL for nuclear fusion control, which achieved a
variety of plasma configurations.

Pump Scheduling. In [4], the authors adopted a branch-and-bound algorithm that
interacts with the hydraulic simulator EPANET 111https://www.epa.gov/water-
research/epanet to evaluate decisions at each timestep corresponding to 1 hour
in a horizon length of 24 hours. The proposed algorithm aims to find a
solution with minimum electricity consumption. Menke and colleagues in [6]
also used a branch-and-bound approach through an algorithm with several steps
for the branching procedure. The objective function adopted has contact with
the presented in this work. The authors consider pumps with fixed speed
(ON/OFF), intending to minimize the electricity consumption meanwhile
penalizing switches in the pump operation. In [46], the authors also applied
DRL to control a real-world scenario of a wastewater treatment plant.
Differently from our study case, in their system, the electricity price has
different tariffs throughout the day, which add one more constraint to the
proposed strategy.

This section presents some control-related scenarios where the approach used
was RL-based. Indeed, RL has already shown remarkable success in applications
such as games and is beginning to show potential for sensitive real-world
applications such as nuclear fusion. Afterward, we discuss some approaches for
the specific scenario of pump scheduling. We believe that a data-driven
solution based on RL can overcome some limitations presented in previous
works, such as robustness and scalability. This work distinguishes itself from
the others by releasing a testbed aiming RL approaches for the pump scheduling
problem.

##  7 Conclusion

We release a testbed for RL where logged data and a water distribution system
simulator are available. This data was gathered through sensors in the real-
world operation controlled by humans and the simulator validated by hydraulic
specialists. Later, we present some statistics on the real-world water supply
and cast the pump scheduling problem in the RL settings. To that end, we
proposed POMDP for the problem, with the state-action space representation and
the reward function. We combine this representation in the simulator,
providing an end-to-end process to learn new policies from the logged data.
Finally, we list some challenges that could be addressed in future works using
the tools and data that have been made available. Thus, we hope to facilitate
research and bridge the gap between RL and real-world applications.

## Acknowledgments and Disclosure of Funding

The authors would like to acknowledge that Fig. 1 is based on an image
originally created by Thomas Krätzig. The authors also acknowledge FAPEMIG,
Federal Ministry of Education and Research of Germany, Agence Nationale de la
Recherce de France, and Fonds de la Recherce Scientifique Belge, for funding
this research by the project IoT.H2O (ANR-18-IC4W-0003) on the IC4Water JPI
call.

## References

  * [1] Antonio Candelieri, Raffaele Perego, and Francesco Archetti.  Bayesian optimization of pump operations in water distribution systems.  Journal of Global Optimization, 71(1):213–235, 2018.
  * [2] Sanda-Carmen Georgescu and Andrei-Mugur Georgescu.  Pumping station scheduling for water distribution networks in epanet.  UPB Sci. Bull, Series D, 77(2):235–246, 2015.
  * [3] Tiago Luna, João Ribau, David Figueiredo, and Rita Alves.  Improving energy efficiency in water supply systems with pump scheduling optimization.  Journal of cleaner production, 213:342–356, 2019.
  * [4] Luis Henrique Magalhães Costa, Bruno de Athayde Prata, Helena M Ramos, and Marco Aurélio Holanda de Castro.  A branch-and-bound algorithm for optimal pump scheduling in water distribution networks.  Water resources management, 30(3):1037–1052, 2016.
  * [5] Folorunso Taliha Abiodun and Fatimah Sham Ismail.  Pump scheduling optimization model for water supply system using awga.  In 2013 IEEE Symposium on Computers Informatics (ISCI), pages 12–17, 2013.
  * [6] Ruben Menke, Edo Abraham, Panos Parpas, and Ivan Stoianov.  Exploring optimal pump scheduling in water distribution networks with branch and bound methods.  Water Resources Management, 30(14):5333–5349, 2016.
  * [7] Henrique Donâncio and Laurent Vercouter.  Safety through intrinsically motivated imitation learning.  In Proceedings of the Adaptive and Learning Agents Workshop (ALA 2022), 2022.
  * [8] Richard S. Sutton and Andrew G. Barto.  Reinforcement Learning: An Introduction.  A Bradford Book, Cambridge, MA, USA, 2018.
  * [9] Haochen Liu, Zhiyu Huang, and Chen Lv.  Improved deep reinforcement learning with expert demonstrations for urban autonomous driving.  CoRR, abs/2102.09243, 2021.
  * [10] Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine.  Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.  In 2017 IEEE International Conference on Robotics and Automation (ICRA), page 3389–3396. IEEE Press, 2017.
  * [11] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel.  Overcoming exploration in reinforcement learning with demonstrations.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6292–6299, 2018.
  * [12] Guillaume Lample and Devendra Singh Chaplot.  Playing fps games with deep reinforcement learning.  In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI’17, page 2140–2146. AAAI Press, 2017.
  * [13] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller.  Playing atari with deep reinforcement learning.  CoRR, abs/1312.5602, 2013.
  * [14] Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al.  Outracing champion gran turismo drivers with deep reinforcement learning.  Nature, 602(7896):223–228, 2022.
  * [15] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  Nature, 575(7782):350–354, 2019.
  * [16] Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind W. Picard.  Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.  CoRR, abs/1907.00456, 2019.
  * [17] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu.  Offline reinforcement learning: Tutorial, review, and perspectives on open problems.  CoRR, abs/2005.01643, 2020.
  * [18] Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, and Michael Bowling.  Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents.  J. Artif. Intell. Res., 61:523–562, 2018.
  * [19] Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq, Timothy P. Lillicrap, and Martin A. Riedmiller.  Deepmind control suite.  CoRR, abs/1801.00690, 2018.
  * [20] Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAI Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * [21] Conor F. Hayes, Roxana Radulescu, Eugenio Bargiacchi, Johan Källström, Matthew Macfarlane, Mathieu Reymond, Timothy Verstraeten, Luisa M. Zintgraf, Richard Dazeley, Fredrik Heintz, Enda Howley, Athirai A. Irissappane, Patrick Mannion, Ann Nowé, Gabriel de Oliveira Ramos, Marcello Restelli, Peter Vamplew, and Diederik M. Roijers.  A practical guide to multi-objective reinforcement learning and planning.  CoRR, abs/2103.09568, 2021.
  * [22] Long Ji Lin.  Self-improving reactive agents based on reinforcement learning, planning and teaching.  Mach. Learn., 8:293–321, 1992.
  * [23] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  In 4th International Conference on Learning Representations, ICLR 2016, 2016.
  * [24] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  An optimistic perspective on offline reinforcement learning.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 104–114. PMLR, 13–18 Jul 2020\.
  * [25] Timothée Lesort, Natalia Díaz Rodríguez, Jean-François Goudou, and David Filliat.  State representation learning for control: An overview.  CoRR, abs/1802.04181, 2018.
  * [26] Changmin Yu, Dong Li, Jianye HAO, Jun Wang, and Neil Burgess.  Learning state representations via retracing in reinforcement learning.  In 10th International Conference on Learning Representations, ICLR 2022, 2022.
  * [27] Astrid Merckling, Alexandre Coninx, Loic Cressot, Stephane Doncieux, and Nicolas Perrin.  State representation learning from demonstration.  In Machine Learning, Optimization, and Data Science, pages 304–315, Cham, 2020. Springer International Publishing.
  * [28] Pieter Abbeel and Andrew Y. Ng.  Apprenticeship learning via inverse reinforcement learning.  In Proceedings of the twenty-first international conference on Machine learning, ICML ’04, New York, NY, USA, 2004. Association for Computing Machinery.
  * [29] John D. Co-Reyes, Suvansh Sanjeev, Glen Berseth, Abhishek Gupta, and Sergey Levine.  Ecological reinforcement learning.  CoRR, abs/2006.12478, 2020.
  * [30] Javier García and Fernando Fernández.  A comprehensive survey on safe reinforcement learning.  Journal of Machine Learning Research, 16(42):1437–1480, 2015.
  * [31] Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau.  Benchmarking batch deep reinforcement learning algorithms.  arXiv preprint arXiv:1910.01708, 2019.
  * [32] Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2052–2062. PMLR, 09–15 Jun 2019.
  * [33] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1179–1191. Curran Associates, Inc., 2020.
  * [34] Oron Anschel, Nir Baram, and Nahum Shimkin.  Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning.  In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 176–185. JMLR.org, 2017.
  * [35] Georg Ostrovski, Pablo Samuel Castro, and Will Dabney.  The difficulty of passive learning in deep reinforcement learning.  In Advances in Neural Information Processing Systems, volume 34, 2021.
  * [36] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne.  Imitation learning: A survey of learning methods.  ACM Comput. Surv., 50(2), apr 2017.
  * [37] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma.  Mopo: Model-based offline policy optimization.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14129–14142. Curran Associates, Inc., 2020.
  * [38] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims.  Morel: Model-based offline reinforcement learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21810–21823. Curran Associates, Inc., 2020.
  * [39] Yaodong Yang, Jianye Hao, Mingyang Sun, Zan Wang, Changjie Fan, and Goran Strbac.  Recurrent deep multiagent q-learning for autonomous brokers in smart grid.  In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 569–575. International Joint Conferences on Artificial Intelligence Organization, 7 2018\.
  * [40] Matthew Hausknecht and Peter Stone.  Deep recurrent q-learning for partially observable mdps.  In AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents (AAAI-SDMIA15), November 2015.
  * [41] Tianshu Wei, Yanzhi Wang, and Qi Zhu.  Deep reinforcement learning for building hvac control.  In Proceedings of the 54th Annual Design Automation Conference 2017, DAC ’17, New York, NY, USA, 2017. Association for Computing Machinery.
  * [42] Viswanath Sivakumar, Tim Rocktäschel, Alexander H. Miller, Heinrich Küttler, Nantas Nardelli, Mike Rabbat, Joelle Pineau, and Sebastian Riedel.  MVFST-RL: an asynchronous RL framework for congestion control with delayed actions.  CoRR, abs/1910.04054, 2019.
  * [43] Nathan Jay, Noga Rotman, Brighten Godfrey, Michael Schapira, and Aviv Tamar.  A deep reinforcement learning perspective on internet congestion control.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 3050–3059. PMLR, 09–15 Jun 2019.
  * [44] Marc G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Subhodeep Moitra, Sameera S Ponda, and Ziyu Wang.  Autonomous navigation of stratospheric balloons using reinforcement learning.  Nature, 588(7836):77–82, 2020.
  * [45] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al.  Magnetic control of tokamak plasmas through deep reinforcement learning.  Nature, 602(7897):414–419, 2022.
  * [46] Giup Seo, Seungwook Yoon, Myungsun Kim, Changho Mun, and Euiseok Hwang.  Deep reinforcement learning-based smart joint control scheme for on/off pumping systems in wastewater treatment plants.  IEEE Access, 9:95360–95371, 2021.
  * [47] Hellmann Dieter-Heinz.  Lecture Notes on Centrifugal Pumps and Compressors.  Technical University of Kaiserslautern, 2004.
  * [48] Carl Pfleiderer and Hartwig Petermann.  Strömungsmaschinen.  Springer-Verlag Berlin Heidelberg, 2005.

##  Appendix A Appendix

###  A.1 Electricity consumption (kW) using the simulator

Figure 4 shows the electricity consumption of the measured (real world) data
and the simulator. Note that the range of values between the measured and
simulated data differ. The reason is due to not taking into account the pump
efficiency in Equation 3. Thus, we obtain the values for measured data by
taking the hydraulic power ($P_{h}$), and dividing it by the efficiency
$\eta$:

$P_{h}(kW)=Q\rho gh/(3.6~{}10^{6}),\textrm{where}$ (3)

  * •

$Q$ is the flow $(m^{3}/h)$

  * •

$\rho$ is the density of fluid $(kg/m^{3})$

  * •

$g$ is the acceleration of gravity $(9.81m/s^{2})$

  * •

$h$ is the differential head (m)

Figure 4: Average electricity consumption calculated using measured data 4 and
simulator 4.

###  A.2 The simulator program

The simulator was developed for investigating the operating behavior of
distribution pumps in a real water network in Germany. The calculation of the
operating point of a pump is based on intersecting the characteristic curve of
a pump (hydraulic head vs. flow rate) with the system curve.

The system curve describes the amount of pressure or hydraulic head which is
necessary to deliver a certain flow into a system. For the system curve one
can distinguish between a static and a dynamic part (Figure 5). The static
part is independent of the flow rate caused by a difference in geodetic
height, e.g. when liquid is pumped into an elevated storage tank, or by a
static pressure difference. The dynamic part is caused by hydraulic losses in
the piping system and increase with the second power of the flow rate. The
operating point of the pump is the intersection of the system curve and the
hydraulic head curve of the pump.

For the water system according to Figure 1 the system curve is not constant.
The static part depends on the current tank level and to a smaller amount on
the water demand. The slope of the system curve depends on the hydraulic
losses in the piping system and in case of a water system of the current water
demand. For a high water demand, the pressure in the system is smaller and
therefore the slope of the system curves becomes smaller, too. In cases of a
small water demand, the slope is greater, because of the higher pressure in
the system. Based on measurements, the relationship of tank level, water
demand static head and slope of the system curve were derived.

With the simulator the pump operation of an arbitrary time period can be
calculated. At the beginning, the expected water demand has to be defined for
each time step. Also, the initial tank level and the pump schedule have to be
specified. For the simulation, the current system curve is calculated based on
current tank level and water demand. At each time step it is checked whether a
pump is in operation. If this is the case, the operating point of the pump is
calculated by intersecting the pump curve with the system curve. In case the
flow rate of the pump is larger than the water demand, the tank level is
increased accordingly. On the contrary, when the flow rate is smaller than the
water demand, the tank level is decreased. With the new tank level and the
water demand of the next time step, a new system curve is calculated for the
next time step. This procedure is continued in a loop for the specified time
period.

###  A.3 Extending the simulator for applying speed control to pumps

Speed control is an efficient method for controlling the operating point of a
pump. In this case, the frequency of the motor is changed by a frequency
converter. By the change in speed, the characteristic curves of the pump for
hydraulic head vs. flow rate and efficiency vs. flow rate are changed. The
change in flow rate and hydraulic head can be calculated by means of the
affinity laws (4 and 5) where the flow rate (Q) changes proportional with the
speed (n) and the hydraulic head (H) changes proportional to the second power
of the speed.

$Q\propto n$ (4) $H\propto n^{2}$ (5)

With equations 4 and 5 it can be shown that the hydraulic head changes with
the second power of the flow rate (6) where $Q_{1}$ and $H_{1}$ are flow rate
and hydraulic head on the pump curve for the initial speed.

$H_{x}=\left(\frac{H_{1}}{Q_{1}^{2}}\right)*Q_{x}^{2}$ (6)

So, all points of a pump curve are moved on quadratic curves through the
origin of the diagram (Figure 5).

Figure 5: Influence of speed control on pump curve 5 [47] and relationship of
pump curve and system curve 5.

Changing the speed, the efficiency curve is also modified. In case the point
on the efficiency curve is identical to the point of best efficiency, it will
also be the point of best efficiency at any other speed. All other points on
the efficiency curve are shifted on a quadratic curve and maintain their
position relative to the point of best efficiency. The maximum value of the
efficiency remains nearly constant. Effects of the change in Reynolds-number
can be included by empirical equations according to Ackeret [48] for example:

$\frac{\left(1-\eta_{1,opt}\right)}{\left(1-\eta_{2,opt}\right)}=\left(1-V\right)+V*\left(\frac{Re_{2}}{Re_{1}}\right)^{(\frac{1}{\alpha})}$
(7)

The change in operating point of the pump depends on the shape of the system
curve. For systems where the static part is zero, the system curve is
identical to the affinity parabola. In this case it is possible that a pump is
always operating at the point of best efficiency when the speed is changed.
Figure 6 shows the situation for a non-zero static part. In this case, the
intersection of pump curve and system curve is not a point on the parabola
based on the affinity laws and the pump is operating with a smaller efficiency
under part-load conditions when the speed is reduced.

Figure 6: Influence of speed control on operating point.

The function for speed control can be included in the main code of the
simulator. With this function it is possible to calculate the pump curve every
time the speed has changed.