# Privacy Signaling Games with Binary Alphabets

Photios A. Stavrou1, Serkan Sarıtaş2 and Mikael Skoglund3 *This work was
funded in part by the Swedish research council under contract
2019-03606.1Photios A. Stavrou is with the Communication Systems Department at
EURECOM, Campus SophiaTech, 06904, France<EMAIL_ADDRESS>Sarıtaş is with the Department of Electrical and Electronics Engineering,
Middle East Technical University, 06800, Ankara, Turkey<EMAIL_ADDRESS>Mikael Skoglund is with the Division of Information Science and Engineering,
KTH Royal Institute of Technology, SE-10044, Stockholm, Sweden.
<EMAIL_ADDRESS>
###### Abstract

In this paper, we consider a privacy signaling game problem for binary
alphabets and single-bit transmission where a transmitter has a pair of
messages, one of which is a casual message that needs to be conveyed, whereas
the other message contains sensitive data and needs to be protected. The
receiver wishes to estimate both messages to acquire as much information as
possible. For this setup, we study the interactions between the transmitter
and the receiver with non-aligned information-theoretic objectives (modeled by
mutual information and hamming distance) due to the privacy concerns of the
transmitter. We derive conditions under which Nash and/or Stackelberg
equilibria exist and identify the optimal responses of the encoder and
decoders strategies for each type of game. One particularly surprising result
is that when both types of equilibria exist, they admit the same encoding and
decoding strategies. We corroborate our analysis with simulation studies.

##  I Introduction

Decision-making is pivotal for a wide range of real-world applications such as
social networks, networked control systems, smart grids, and recommendation
systems. In these applications, usually, several customers (users) in a
network may share extensive amounts of information with some service provider
(i.e., utility company) because the latter wishes to know as much as possible
about the service offered at the customer to improve the quality of service.
However, this may come with a price as sometimes the users in the network may
be prone to network-based attacks from malicious elements aiming to steal some
sensitive information. Therefore, the users, in addition to the continuous
improvement of the quality of service offered by a provider, wish to maintain
a certain level of privacy. A type of privacy objective can be assumed when
the information transmitted by some user to the service provider may be
correlated with certain private information they want to protect. For example,
in smart grids, the smart meter provides real-time information on energy
supplies from the energy provider on the demands of the consumer (user), which
can be utilized for unauthorized purposes, e.g., to infer the private
information of the consumer, such as their habits and behaviors, see, e.g.,
[1, 2]. Identifying privacy-preserving mechanisms or approaches under various
contexts related to information theory and control applications can be found
in an anthology of papers, for instance, in [3, 4, 5, 6, 7, 8].

###  I-A Motivational Example

Consider the scenario illustrated in Fig.1. In that scenario, a smart house is
illustrated in which a smart meter records the energy consumption and
exchanges consumption data with energy suppliers, which can be used for
monitoring and billing. Evidently, the existence of home residents and the
electricity usage recorded by the smart meter at home are, in practice,
correlated. Nevertheless, the presence of the house residents at home should
be kept secret to possible outsiders (burglars, adversaries, etc.) whereas, at
the same time, the energy consumption should be available to the electricity
service providers. Therefore, the smart meter should be designed so that the
service providers can access the electricity usage data, whereas the outsiders
should not be able to deduce if the residents are home or not by checking the
information from the smart meter.

Figure 1: Motivational example.

###  I-B Literature review

The studies on cheap talk and signaling games were initiated by Crawford and
Sobel in their seminal work[9], and found applications in various topics,
e.g., in networked systems [10, 11], recommendation systems [12, 13], and
economics [14, 15]. Starting with [16], there are many studies that consider
the Stackelberg equilibrium of signaling games; an incomplete list includes
[17, 18, 19, 20, 21, 22, 23, 24, 25, 26] (see also the references therein).
Many of these works assume that the non-alignment between the objective
functions of the encoder and the decoder is a function of a Gaussian random
variable (RV) correlated with the Gaussian source and hidden from the decoder
(unlike the original case where it is fixed and commonly known by the encoder
and the decoder [9], which is also studied in [17, 21, 24]). Nash and
Stackelberg equilibria of signaling games are investigated in [25] when there
is a mismatch in priors of players. We refer to [15, 17, 21] for more
discussion on the literature and some extensions (including Nash equilibrium
analyses and multi-stage extensions) on cheap talk and signaling games.

In the context of strategic information transmission, several works consider
the scenario where the sender takes the privacy of certain information into
account by deploying a suitable privacy measure under either the Nash or
Stackelberg criteria. For instance, in [27], a communication scenario between
a sender and a receiver is investigated using the Stackelberg equilibrium. A
family of nontrivial equilibria, in which the communicated messages carry
information, is constructed, and its properties are studied. In [28], the
authors study a Stackelberg game where the utility measure for the public
parameter is quadratic and the privacy measure is entropy-based. Additional
results therein include characterizations of the equilibrium under noisy and
noiseless communication scenarios and analysis of the corresponding coding
policies. In [29], the effect of privacy via a Nash game is studied between a
sender and a receiver. As a measure of merit, the authors use mutual
information between the private information and the communicated message to
quantify the amount of the leaked information. For discrete RVs, they provide
a numerical algorithm to find an equilibrium, whereas for Gaussian RVs, a
bound on the estimation error is provided, and affine policies are shown to
achieve this bound. In [30], a privacy-signaling game problem for the setup in
Fig. 2 is considered in which a transmitter with privacy concerns observes a
pair of correlated random vectors which are modeled as jointly Gaussian. Among
other results, it is shown that a payoff dominant Nash equilibrium among all
admissible policies is attained by a set of explicitly characterized linear
policies and coincides with a Stackelberg equilibrium. Regarding the state of
the art privacy metrics, we refer to [31] for a selection of over eighty
privacy metrics and their categorization. Herein, we consider one of the
discussed metrics therein, namely, Hamming distance.

In this paper, we consider the scenario that was first introduced in [30]. In
this setup, a transmitter has a pair of messages, one of which is a casual
message that needs to be conveyed, whereas the other message contains
sensitive data and needs to be protected. On the other hand, the receiver
wishes to estimate both messages with the aim of acquiring as much information
as possible. For this setup, we study the interactions between the transmitter
and the receiver whose objectives are not-aligned due to the privacy concerns
of the transmitter in a game-theoretic framework. However, in contrast to [30]
that deals with jointly Gaussian random vectors (of possibly different
lengths) and linear policies, here we deal with binary alphabets and consider
different objectives for our single-bit transmitter and receiver.

###  I-C Contributions

The main contributions of this paper can be summarized as follows:

  * (i)

We model a binary privacy signaling game, assuming a single-bit transmission,
between an encoder and a decoder in which the encoder aims to hide one of two
correlated binary RVs (i.e., private message) and to transmit the other (i.e.,
public message) while the decoder’s goal is to learn about both of the RVs as
much as possible. We use mutual information as a metric to measure the
information exchange, a Hamming distortion to measure the level of privacy,
and a weighting coefficient that determines the importance of privacy from the
perspective of the encoder.

  * (ii)

We characterize the objective functions of the encoder and the decoder in
terms of priors and strategies (see Lemma III.1), derive the best response of
the encoder (resp. decoder) for a given decoder (resp. encoder) (see Lemmas
III.2 and III.3). Then, using these best response maps, we characterize the
Stackelberg and Nash equilibria (see Theorems III.1 and III.2, respectively).

  * (iii)

We show that under certain conditions on the source prior probabilities, Nash
and Stackelberg equilibria exist and coincide. Otherwise, there does not exist
a Nash equilibrium, and the privacy coefficient only affects the Stackelberg
equilibrium. In particular, for large privacy coefficients, the encoder may
even prefer to hide information about the public message.

##  II Problem formulation and preliminaries

In this paper, we consider the scenario illustrated in Fig. 2 that was first
introduced in [30].

Figure 2: Our setup.

We assume that the transmitter encodes a pair of correlated random variables
$({\bf x},{\bf y})\in{\cal X}\times{\cal Y},\;{\cal X}={\cal Y}=\\{0,1\\}$
into ${\bf z}\in{\cal Z}=\\{0,1\\}$ using an encoding function denoted by
${z}=\gamma^{e}(x,y)$ and the receiver wants to decode both messages based on
the observation ${\bf z}=z$. Note that the transmitter desires to transmit
information about ${\bf y}$ and sees ${\bf x}$ as a private parameter that
needs to be hidden from the receiver. In contrast, the receiver wants to
accurately estimate both public and private messages given the observation
${\bf z}=z$. We denote the decoding functions for estimating ${\bf x}$ and
${\bf y}$ by $\hat{x}=\gamma^{d_{\bf x}}({z})$ and $\hat{y}=\gamma^{d_{\bf
y}}({z})$, respectively.

Since the transmitter needs to encode two messages generated by the joint
distribution of (x, y), i.e., ${\bf P}(x,y)$, it means that hiding ${\bf x}$
or transmitting ${\bf y}$ are somehow inter-dependent actions. Since our
scenario is for binary alphabets, in the sequel, we will denote the joint
distribution or probability mass function of $({\bf x},{\bf y})$ to be given
by the following column stochastic matrix:

$\displaystyle{\bf P}(x,y)=\begin{bmatrix}{\bf P}(x=0,y=0)\\\ {\bf
P}(x=0,y=1)\\\ {\bf P}(x=1,y=0)\\\ {\bf
P}(x=1,y=1)\end{bmatrix}=\begin{bmatrix}a\\\ b\\\ c\\\ d\end{bmatrix},$ (1)

where $d\triangleq{1-(a+b+c)}$ with
$(a,b,c,d)\in[0,1]\times[0,1]\times[0,1]\times[0,1]$ (also denoted for
simplicity $[0,1]^{4}$). The objective of the transmitter is to maximize the
public information ${\bf y}$ for the receiver and at the same time to hide as
much as possible the sensitive information ${\bf x}$. These can be cast by the
following objective function

$\displaystyle J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})=I({\bf
y};\hat{\bf y})+\rho{\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\},$ (2)

which is to be maximized by the encoder, where $I({\bf y};\hat{\bf y})$ is the
mutual information between ${\bf y}$ and $\hat{\bf y}$ [32], $\rho>0$ is a
weighting coefficient that determines the level of desired privacy of ${\bf
x}$, ${\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$ is some loss function which for
this paper is assumed to be modeled by Hamming distortion, i.e.,

$\displaystyle d_{H}$ $\displaystyle({\bf x},\hat{\bf x})=\begin{cases}1&{\bf
x}\neq\hat{\bf x}\\\ 0&{\bf x}=\hat{\bf x}\end{cases},$ (3)

responsible to capture the privacy term ${\bf x}$. The objective of the
receiver is to maximize the information of both public information ${\bf y}$
and sensitive information ${\bf x}$. This can be cast by the following
objective function

$\displaystyle J^{d}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})=I({\bf
y};\hat{\bf y})-{\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}\,,$ (4)

which is to be maximized by the decoder. Since the costs of the encoder and
the decoder are not aligned, the problem is studied under a game theoretic
framework, and Stackelberg and Nash equilibria are investigated. In the Nash
(simultaneous-move) game, the encoder and the decoder announce their
strategies at the same time. More precisely, suppose that the set of possible
strategies at the encoder is denoted by $\Gamma^{e}$ and those at the decoders
by $\Gamma^{d_{\bf y}}$ and $\Gamma^{d_{\bf x}}$, respectively, such that
$\gamma^{e}\in\Gamma^{e}$, $\gamma^{d_{\bf y}}\in\Gamma^{d_{\bf y}}$,
$\gamma^{d_{\bf x}}\in\Gamma^{d_{\bf x}}$. Then, a triplet of policies
$(\gamma^{e,*},\gamma^{d_{\bf y},*},\gamma^{d_{\bf x},*})$ is said to be a
Nash equilibrium [33] if

$\displaystyle\begin{split}J^{e}(\gamma^{e,*},\gamma^{d_{\bf
y},*},\gamma^{d_{\bf x},*})&\geq J^{e}(\gamma^{e},\gamma^{d_{\bf
y},*},\gamma^{d_{\bf x},*}),\quad\forall\gamma^{e}\in\Gamma^{e}\,,\\\
J^{d}(\gamma^{e,*},\gamma^{d_{\bf y},*},\gamma^{d_{\bf x},*})&\geq
J^{d}(\gamma^{e,*},\gamma^{d_{\bf y}},\gamma^{d_{\bf
x}})\quad\forall\gamma^{d_{\bf y}}\in\Gamma^{d_{\bf y}},\gamma^{d_{\bf
x}}\in\Gamma^{d_{\bf x}}\,.\end{split}$ (5)

As observed in (5), none of the players prefer to change their optimal
strategies at the equilibrium, i.e., there is no profitable unilateral
deviation from any of the players. In the Stackelberg game, the leader
(encoder) commits to a particular policy and announces it to the follower
(decoder). The decoder takes its optimal action upon observing the encoder’s
committed strategy. More precisely, a triplet of strategies
$(\gamma^{e,*},\gamma^{d_{\bf y},*},\gamma^{d_{\bf x},*})$ is said to be a
Stackelberg equilibrium [33] if

$\displaystyle\begin{split}J^{e}(\gamma^{e,*},\gamma^{d_{\bf
y},*}(\gamma^{e,*}),\gamma^{d_{\bf x},*}(\gamma^{e,*}))&\geq
J^{e}(\gamma^{e},\gamma^{d_{\bf y},*}(\gamma^{e}),\gamma^{d_{\bf
x},*}(\gamma^{e})),\quad\forall\gamma^{e}\in\Gamma^{e}\,,\\\ \text{where
}&(\gamma^{d_{\bf y},*}(\gamma^{e}),\gamma^{d_{\bf x},*}(\gamma^{e}))\text{
satisfy}\\\ J^{d}(\gamma^{e},\gamma^{d_{\bf y},*}(\gamma^{e}),\gamma^{d_{\bf
x},*}(\gamma^{e}))&\geq J^{d}(\gamma^{e},\gamma^{d_{\bf
y}}(\gamma^{e}),\gamma^{d_{\bf x}}(\gamma^{e}))\quad\forall\gamma^{d_{\bf
y}}\in\Gamma^{d_{\bf y}},\gamma^{d_{\bf x}}\in\Gamma^{d_{\bf
x}}\,.\end{split}$

Note that the follower (decoder) takes its action after observing the strategy
$\gamma^{e}$ of the leader (encoder), thus the strategies $(\gamma^{d_{\bf
y}}(\gamma^{e}),\gamma^{d_{\bf x}}(\gamma^{e}))$ of the decoder are a function
of $\gamma^{e}$.

##  III Main Results

Before we start with our main results, we first introduce the general
structure of the “stochastic” encoder and decoder policies for our setup. In
particular, the encoder is given by the transition matrix

$\displaystyle{\bf
P}^{e}(z|x,y)=\begin{bmatrix}\kappa_{1}&\kappa_{2}&\kappa_{3}&\kappa_{4}\\\
1-\kappa_{1}&1-\kappa_{2}&1-\kappa_{3}&1-\kappa_{4}\end{bmatrix},$ (6)

where $(\kappa_{1},\kappa_{2},\kappa_{3},\kappa_{4})\in[0,1]^{4}$, whereas the
transition matrices at the decoder are given by the column stochastic matrices

$\displaystyle{\bf P}^{d_{\bf
y}}(\hat{y}|z)=\begin{bmatrix}\delta_{1}&\delta_{2}\\\
1-\delta_{1}&1-\delta_{2}\end{bmatrix},$ (7) $\displaystyle{\bf P}^{d_{\bf
x}}(\hat{x}|z)=\begin{bmatrix}\epsilon_{1}&\epsilon_{2}\\\
1-\epsilon_{1}&1-\epsilon_{2}\end{bmatrix},$ (8)

where $(\delta_{1},\delta_{2},\epsilon_{1},\epsilon_{2})\in[0,1]^{4}$. To
derive our main results, we make use of the following assumption.

###### Assumption III.1

(Structural assumption on (6)) Restrict the information structure in (6) to
one that $\kappa_{3}=1-\kappa_{2}$ and $\kappa_{4}=1-\kappa_{1}$.

###### Remark III.1

(Comments on Assumption III.1) By putting such a restriction (i.e., a
“symmetric” encoder assumption) on $\kappa_{3}$ and $\kappa_{4}$, we prevent
infinitely many quadruples $(\kappa_{1},\kappa_{2},\kappa_{3},\kappa_{4})$
resulting in essentially equivalent encoders with respect to performance.
Furthermore, after eliminating redundant quadruples by Assumption III.1, it is
possible to obtain the (joint) convexity of $I({\bf y};\hat{\bf y})$ with
respect to $\kappa_{1}$ and $\kappa_{2}$ in Appendix B. Otherwise, i.e.,
without Assumption III.1, there is no conclusion on the (joint)
convexity/concavity of $I({\bf y};\hat{\bf y})$ with respect to the quadruple
$(\kappa_{1},\kappa_{2},\kappa_{3},\kappa_{4})$.

Next, we prove a lemma that reformulates the objective functions of (2), (4).
We note that this lemma holds even if Assumption III.1 does not hold.

###### Lemma III.1

(Characterization) For the information structure of the stochastic encoder and
decoder in (6)-(8), the objective functions in (2), (4) can be characterized
as follows

$\displaystyle J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})=I({\bf
y};\hat{\bf y})+\rho{\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\},$ (9)
$\displaystyle J^{d}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})=I({\bf
y};\hat{\bf y})-{\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\},$ (10)

where111The logarithms are taken with base two throughout the paper.

$\displaystyle I({\bf y};\hat{\bf
y})=H_{b}(q_{1})+H_{b}(P_{1}+P_{2})+P_{1}\log(P_{1})+P_{2}\log(P_{2})+(q_{1}-P_{1})\log(q_{1}-P_{1})+(1-q_{1}-P_{2})\log(1-q_{1}-P_{2})\,,$
(11) $\displaystyle{\bf E}\\{d_{H}({\bf x},\hat{\bf
x})\\}=a(1-n_{1})+b(1-n_{2})+cn_{3}+dn_{4},$ (12)

with $H_{b}(p)$ denoting the binary entropy function, i.e.,
$H_{b}(p)\triangleq-p\log p-(1-p)\log(1-p)$, $q_{1}\triangleq a+c$,
$P_{1}\triangleq at_{1}+ct_{3}$, $P_{2}\triangleq bt_{2}+dt_{4}$,
$t_{i}\triangleq\delta_{1}\kappa_{i}+\delta_{2}(1-\kappa_{i})$ and
$n_{i}\triangleq\epsilon_{1}\kappa_{i}+\epsilon_{2}(1-\kappa_{i})$ for
$i=1,2,3,4$.

###### Proof:

See Appendix A. ∎

After formulating the objectives of the encoder and the decoder, next, we
characterize their optimal strategies, in particular, their best responses for
any other given strategy.

###### Lemma III.2

(Best Response: Encoder) Suppose that Assumption III.1 holds. Then, for given
decoder strategies $\gamma^{d_{\bf x}}$ and $\gamma^{d_{\bf y}}$, the
objective function of the encoder in (9) is a jointly convex function of the
pair $(\kappa_{1},\kappa_{2})$, and the maximum is achieved at one of the
extreme points, i.e., $\kappa_{1}\kappa_{2}=\\{00,01,10,11\\}$.

###### Proof:

See Appendix B. ∎

###### Lemma III.3

(Best Response: Decoder) Suppose that Assumption III.1 holds. Then the
following hold.

  * (i)

For a given encoder strategy $\gamma^{e}$, $I({\bf y};\hat{\bf y})$ in (11) is
a jointly convex function of the pair $(\delta_{1},\delta_{2})$, and the
maximum is achieved either when $\delta_{1}\delta_{2}=01$ or
$\delta_{1}\delta_{2}=10$.

  * (ii)

For a given encoder strategy $\gamma^{e}$, the average distortion ${\bf
E}\\{d_{H}({\bf x},\hat{\bf x})\\}$ in (12) is minimized using the decoder
strategy characterized in Table I, where
$\theta\triangleq\kappa_{1}(a+d)+\kappa_{2}(b+c)$.

TABLE I: Optimal decoder strategy to minimize the average distortion. Condition | $\epsilon_{1}$ | $\epsilon_{2}$ | ${\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$
---|---|---|---
$a+b\leq\theta\leq c+d$ | $0$ | $0$ | $a+b$
$\theta\leq a+b\,,\,\theta\leq c+d$ | $0$ | $1$ | $\theta$
$\theta\geq a+b\,,\,\theta\geq c+d$ | $1$ | $0$ | $1-\theta$
$a+b\geq\theta\geq c+d$ | $1$ | $1$ | $c+d$

###### Proof:

See Appendix C. ∎

Next, we proceed to derive conditions for which Nash and/or Stackelberg
equilibria exist together with their corresponding optimal strategies.

###### Theorem III.1

(Stackelberg) Suppose that Assumption III.1 holds and
$\min\\{a+b,c+d,a+d,b+c\\}$ is $a+b$ or $c+d$. Then, the following strategies
form a Stackelberg equilibrium:

$\displaystyle\begin{split}\kappa_{1}\kappa_{2}&=01\text{ or
}\kappa_{1}\kappa_{2}=10~{}~{}\text{(Encoder)}\\\
\delta_{1}\delta_{2}&=01\text{ or
}\delta_{1}\delta_{2}=10~{}~{}\text{(Decoder-$\hat{\bf y}$)}\\\
\epsilon_{1}\epsilon_{2}&=\begin{cases}00\text{ if }a+b\leq c+d\\\ 11\text{ if
}a+b\geq c+d\end{cases}.~{}~{}\text{(Decoder-$\hat{\bf x}$)}\end{split}$ (13)

Otherwise, if $\min\\{a+b,c+d,a+d,b+c\\}$ is $a+d$ or $b+c$, then, for
sufficiently small $\rho$, the equilibrium strategies of (13) are still valid.
In contrast, for sufficiently large $\rho$, the decoder strategies in (13) are
still the same and the optimum encoder strategy lies at the boundary of the
$\kappa_{1}\kappa_{2}$ region which satisfy either $a+b\leq\theta\leq c+d$ or
$a+b\geq\theta\geq c+d$, where
$\theta\triangleq\kappa_{1}(a+d)+\kappa_{2}(b+c)$ (defined as before).

###### Proof:

See Appendix D. ∎

###### Theorem III.2

(Nash) Suppose that Assumption III.1 holds and $\min\\{a+b,c+d,a+d,b+c\\}$ is
$a+b$ or $c+d$. Then, the same strategies as in (13) form a Nash equilibrium.
Otherwise, there does not exist a Nash equilibrium.222We exclude the trivial
case of equal priors $a=b=c=d=0.25$ in which any strategy pair ends up an
equilibrium.

###### Proof:

See Appendix E. ∎

Next we give some technical comments related to our results in Theorems III.1,
III.2.

###### Remark III.2

(Technical comments) (TC1) When in Theorems III.1, III.2,
$\min\\{a+b,c+d,a+d,b+c\\}$ is $a+b$ or $c+d$, the optimal encoder selects
either $\kappa_{1}\kappa_{2}=01$ or $\kappa_{1}\kappa_{2}=10$, which
correspond to sending information only about ${\bf y}$ (e.g.,
$\kappa_{1}\kappa_{2}=01$ implies ${\bf P}^{e}(z|x,y)={\bf P}^{e}(z|y)$). In
this case, since the received message ${\bf z}$ does not contain any direct
information about ${\bf x}$, the decoder-$\hat{\bf x}$ uses only priors of
${\bf P}(x)$ and achieves the average distortion ${\bf E}\\{d_{H}({\bf
x},\hat{\bf x})\\}=\min\\{a+b,c+d\\}$. (TC2) If $\min\\{a+b,c+d,a+d,b+c\\}$ is
$a+d$ or $b+c$ and the encoder still uses $\kappa_{1}\kappa_{2}=01$ or
$\kappa_{1}\kappa_{2}=10$, then, the decoder makes use of the conditional
probability ${\bf P}(x|y)$ (since ${\bf z}$ is directly related to ${\bf y}$),
which further means that the average distortion ${\bf E}\\{d_{H}({\bf
x},\hat{\bf x})\\}=\min\\{a+d,b+c\\}$. Hence in order to increase the privacy
level (i.e., increase the average distortion to $\min\\{a+b,c+d\\}$), the
encoder uses different strategies that result in smaller value of mutual
information (see Fig. 4). To make this point clear, we note that the
strategies in (13) result in $I({\bf y};\hat{\bf y})=H_{b}(q_{1})$ and ${\bf
E}\\{d_{H}({\bf x},\hat{\bf x})\\}=\min\\{a+b,c+d\\}$. However, for large
privacy weighting coefficient $\rho$, as shown in Theorem III.1, the average
distortion does not change, and the mutual information, as a function of the
pair $(\kappa_{1},\kappa_{2})$, can be characterized as in (11) with
$P_{1}=a\kappa_{1}+c(1-\kappa_{2})$ and $P_{2}=b\kappa_{2}+d(1-\kappa_{1})$.
The resulting mutual information value will be less than $H_{b}(q_{1})$, which
means that the encoder ventures to send less information about the public
message to be able to hide information about the private message.

###### Remark III.3

(Connection to similar work) In [30], a similar setup is considered in which
the random sources are jointly Gaussian, and the squared error is utilized as
a privacy and information exchange metric. Similar to our result, it is shown
that Stackelberg and payoff dominant Nash equilibria coincide. However, due to
the difference between our source assumption (i.e., binary), information
exchange metric (i.e., mutual information), and privacy metric (i.e., Hamming
distortion), we have some cases under which Stackelberg equilibria exist, but
there is no Nash equilibrium.

##  IV Numerical Results

In this section, we validate our theoretical results via simulations. For all
simulations, we let $a=0.3$, $b=0.1$, $c=0.2$, and $d=1-a-b-c=0.4$. We start
by validating the best responses of the players as follows:

  * (i)

First we let $\rho=1$. Then, for given decoder strategies
$(\delta_{1},\delta_{2})\in[0,1]^{2}$ and
$(\epsilon_{1},\epsilon_{2})\in[0,1]^{2}$, we calculate corresponding encoder
costs by (9) for every possible symmetric encoder actions
$\gamma^{e}=(\kappa_{1},\kappa_{2})$, to find the optimal one, i.e., the
maximizer. We repeat this process for every $\delta_{1}$, $\delta_{2}$,
$\epsilon_{1}$, and $\epsilon_{2}$, which can take one of the $20$ evenly
spaced values between $0$ and $1$. As illustrated in Fig. 3(a), for $16\times
10^{4}$ different combinations, $\kappa_{1}\kappa_{2}$ takes only four
different values, which are $\delta_{1}\delta_{2}=00$,
$\delta_{1}\delta_{2}=01$, $\delta_{1}\delta_{2}=10$, or
$\delta_{1}\delta_{2}=11$. Thus, the best response of the encoder stated in
Lemma III.2 is proved numerically, too. Note that, in our simulations, when
there are multiple optima, the encoder selects any of them randomly.

  * (ii)

For a given encoder strategy $(\kappa_{1},\kappa_{2})\in[0,1]^{2}$, we
calculate corresponding decoder costs by (10) (indeed, only the mutual
information $I({\bf y};\hat{\bf y})$ part) for every possible decoder actions
$\gamma^{d_{\bf y}}=(\delta_{1},\delta_{2})$, to find the optimal one, i.e.,
the maximizer. We repeat this process for every $\kappa_{1}$ and $\kappa_{2}$,
which can take one of the $100$ evenly spaced values between $0$ and $1$. As
illustrated in Fig. 3(b), for $10^{4}$ different $\kappa_{1}\kappa_{2}$
values, $\delta_{1}\delta_{2}$ takes only two different values, which are
$\delta_{1}\delta_{2}=01$ or $\delta_{1}\delta_{2}=10$. Thus, the best
response of the decoder stated in Lemma III.3.(i) is proved numerically, too.
Note that, in our simulations, when there are multiple optima, the decoder
selects any of them randomly. Thus, always optimal actions
$\delta_{1}\delta_{2}=01$ or $\delta_{1}\delta_{2}=10$ are selected
approximately equal number of times.

  * (iii)

Similar to the previous analysis, for a given encoder strategy
$(\kappa_{1},\kappa_{2})\in[0,1]^{2}$, now we calculate corresponding decoder
costs by (10) (indeed, only the average distortion ${\bf E}\\{d_{H}({\bf
x},\hat{\bf x})\\}$ part) for every possible decoder actions $\gamma^{d_{\bf
x}}=(\epsilon_{1},\epsilon_{2})$, to find the optimal one, i.e., the
minimizer. We repeat this process for every $\kappa_{1}$ and $\kappa_{2}$, and
obtain Fig. 3(c). The optimal actions are $\epsilon_{1}\epsilon_{2}=00$,
$\epsilon_{1}\epsilon_{2}=01$ or $\epsilon_{1}\epsilon_{2}=10$. Thus, the best
response of the decoder stated in Lemma III.3.(ii) is proved numerically, too.
Note that, by Table I, $\epsilon_{1}\epsilon_{2}$ cannot be $11$ since
$a+b\geq c+d$ is not satisfied for our selection.

(a) The distribution of the best response of the encoder
$\gamma^{e}=(\kappa_{1},\kappa_{2})$ for given decoder actions. As it can be
seen, $\kappa_{1}\kappa_{2}$ can only be any of $00$, $01$, $10$, and $11$.

(b) The distribution of the best response of the decoder $\gamma^{d_{\bf
y}}=(\delta_{1},\delta_{2})$ for given encoder actions. As it can be seen,
$\delta_{1}\delta_{2}$ is either $01$ or $10$.

(c) The best response of the decoder $\gamma^{d_{\bf
x}}=(\epsilon_{1},\epsilon_{2})$ for given encoder actions. As it can be seen,
$\epsilon_{1}\epsilon_{2}$ can only be any of $00$, $01$, and $10$.

Figure 3: Simulation results on the best responses of the players.

After getting the best response maps of the players, we can utilize these
results to obtain the Stackelberg equilibrium. In Fig. 4, we plot $I({\bf
y};\hat{\bf y})$ and ${\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$ as a function
of the encoder strategy $(\kappa_{1},\kappa_{2})$ for given decoder
strategies. In particular, Fig. 4(a) illustrates the best response of the
decoder-$\hat{\bf y}$ due to a Stackelberg assumption, i.e.,
$\delta_{1}\delta_{2}=01$ or $\delta_{1}\delta_{2}=10$ (via Lemma III.3), and
the maximum $I({\bf y};\hat{\bf y})$ is achieved when
$\kappa_{1}\kappa_{2}=01$ or $\kappa_{1}\kappa_{2}=10$ (see Theorem III.1). In
Fig. 4(b), via Lemma III.3, the best response of the decoder-$\hat{\bf x}$ is
considered due to a Stackelberg assumption. Since
$\min\\{a+b,c+d,a+d,b+c\\}=b+c$, $\kappa_{1}\kappa_{2}=01$ or
$\kappa_{1}\kappa_{2}=10$ is not in the optimal region to maximize ${\bf
E}\\{d_{H}({\bf x},\hat{\bf x})\\}$ (see Theorem III.1 and Remark III.2). The
effect on this confusion can be observed for large value of $\rho$. Indeed,
for small enough privacy weighting coefficient $\rho$, as it can be seen in
Fig. 5(a), the maximizers of $I({\bf y};\hat{\bf y})$ are still the maximizers
of $J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})$ in (9). On the
other hand, for large enough privacy weighting coefficient $\rho$, ${\bf
E}\\{d_{H}({\bf x},\hat{\bf x})\\}$ gets more dominant in
$J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})$ in (9). This case is
illustrated in Fig. 5(b).

(a)

(b)

Figure 4: $I({\bf y};\hat{\bf y})$ and ${\bf E}\\{d_{H}({\bf x},\hat{\bf
x})\\}$ as a function of the encoder strategy $(\kappa_{1},\kappa_{2})$ to
analyze the Stackelberg equilibrium.

(a) $\rho=2$.

(b) $\rho=20$.

Figure 5: $J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})$ as a
function of the encoder strategy $(\kappa_{1},\kappa_{2})$ to analyze the
Stackelberg equilibrium. As it can be seen, for small $\rho$,
$\kappa_{1}\kappa_{2}=01$ and $\kappa_{1}\kappa_{2}=10$ are still optimal,
whereas, the encoder selects intermediate $\kappa_{1}$ and $\kappa_{2}$ values
as $\rho$ gets larger.

##  V Conclusion and Future Research

In this paper, we studied Nash and Stackelberg equilibria of privacy signaling
games with binary alphabets with single-bit transmission between an encoder
and a decoder with misaligned objectives. We derived the conditions under
which Nash and/or Stackelberg equilibria exist.

Our model has several possible interesting extensions. The most important
question that needs to be answered is the extension of the framework beyond
the single-bit transmission, that is to say, the transmitted messages are
random vectors. Another interesting extension would be to consider scenarios
with alternative objective functions and privacy criteria (e.g., log-loss
function).

##  Appendix A Proof of Lemma III.1

From the information structure of the stochastic encoder and decoder in
(6)-(8) the constraint sets in (9) and (10) are clear. Hence it suffices to
characterize only the mutual information and the loss function. First, we
characterize the mutual information between ${\bf y}$ and $\hat{\bf y}$ which
is given by

$\displaystyle I({\bf y};\hat{\bf y})=\sum_{({\bf y},\hat{\bf
y})\in\\{0,1\\}^{2}}\log\left({{\bf P}(\hat{y}|y)\over{\bf
P}(\hat{y})}\right){\bf P}(\hat{y}|y){\bf P}(y).$ (14)

To do it, we need to compute

$\displaystyle{\bf P}(\hat{y}|y)=\sum_{({\bf z},{\bf x})\in\\{0,1\\}^{2}}{\bf
P}(\hat{y},z,x|y)=\sum_{({\bf z},{\bf x})\in\\{0,1\\}^{2}}{\bf P}^{d_{\bf
y}}(\hat{y}|z){\bf P}^{e}(z|x,y){\bf P}(x|y),$ (15) $\displaystyle{\bf
P}(\hat{y})=\sum_{{\bf y}\in\\{0,1\\}}{\bf P}(\hat{y}|y){\bf P}(y),$ (16)
$\displaystyle{\bf P}({y})=\sum_{{\bf x}\in\\{0,1\\}}{\bf P}(x,y),$ (17)
$\displaystyle{\bf P}(x|y)=\frac{{\bf P}(x,y)}{{\bf P}(y)}.$ (18)

Observe that given (1), then in (17) we obtain

$\displaystyle{\bf P}(y)=\begin{bmatrix}q_{1}\\\ 1-q_{1}\end{bmatrix},$ (19)

where $q_{1}\triangleq{a+c}$. This in turn implies when substituted in (18)
that

$\displaystyle{\bf P}(x|y)=\begin{bmatrix}\frac{a}{q_{1}}&\frac{b}{1-q_{1}}\\\
\frac{c}{q_{1}}&\frac{d}{1-q_{1}}\end{bmatrix}.$ (20)

Substituting (20) in (15), and using (6), (7) we obtain

$\displaystyle{\bf
P}(\hat{y}|y)=\begin{bmatrix}\frac{at_{1}+ct_{3}}{q_{1}}&\frac{bt_{2}+dt_{4}}{1-q_{1}}\\\
\frac{a(1-t_{1})+c(1-t_{3})}{q_{1}}&\frac{b(1-t_{2})+d(1-t_{4})}{1-q_{1}}\end{bmatrix},$
(21)

where ${t_{i}=\delta_{1}\kappa_{i}+\delta_{2}(1-\kappa_{i})},~{}i=1,2,3,4$.
Finally, using (21) and (19) in (16) we obtain

$\displaystyle{\bf P}(\hat{y})=\begin{bmatrix}at_{1}+bt_{2}+ct_{3}+dt_{4}\\\
1-(at_{1}+bt_{2}+ct_{3}+dt_{4})\end{bmatrix}.$ (22)

The characterization of the mutual information in (11) is obtained by
substituting (19), (21), (22), in (15). Next we proceed to characterize the
loss function (i.e., Hamming distance in (2), (4)). Observe that

$\displaystyle{\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$
$\displaystyle=\sum_{({\bf x},\hat{\bf
x})\in\\{0,1\\}^{2}}d_{H}(x,\hat{x}){\bf P}(x,\hat{x})=\sum_{({\bf x},\hat{\bf
x},{\bf y},{\bf z})\in\\{0,1\\}^{2}}d_{H}(x,\hat{x}){\bf P}(x,\hat{x},y,z)$
$\displaystyle=\sum_{({\bf x},\hat{\bf x},{\bf y},{\bf
z})\in\\{0,1\\}^{2}}d_{H}(x,\hat{x}){\bf P}^{d_{\bf x}}(\hat{x}|z){\bf
P}^{e}(z|x,y){\bf P}(x,y)=\eqref{eq:hammingLoss}.$ (23)

This completes the derivation.

##  Appendix B Proof of Lemma III.2

Observe the following analysis on mutual information in (11) with respect to
$P_{1}$ and $P_{2}$:

$\displaystyle{\partial I({\bf y};\hat{\bf y})\over\partial P_{1}}$
$\displaystyle=-\log(P_{1}+P_{2})+\log(1-(P_{1}+P_{2}))+\log(P_{1})-\log(q_{1}-P_{1})\,,$
$\displaystyle{\partial I({\bf y};\hat{\bf y})\over\partial P_{2}}$
$\displaystyle=-\log(P_{1}+P_{2})+\log(1-(P_{1}+P_{2}))+\log(P_{2})-\log(1-q_{1}-P_{2})\,.$

Then, the Hessian of $I({\bf y};\hat{\bf y})$ with respect to $P_{1}$ and
$P_{2}$ becomes

$\displaystyle\operatorname{Hess}\,I({\bf y};\hat{\bf y})={\partial^{2}I({\bf
y};\hat{\bf y})\over\partial P_{i}\partial
P_{j}}={1\over\ln(2)}\begin{bmatrix}-A+B&-A\\\ -A&-A+C\end{bmatrix}\,,$ (24)

where $A\triangleq{1\over P_{1}+P_{2}}+{1\over 1-(P_{1}+P_{2})}$,
$B\triangleq{1\over P_{1}}+{1\over q_{1}-P_{1}}$, and $C\triangleq{1\over
P_{2}}+{1\over 1-q_{1}-P_{2}}$. Note that, by the definitions in Lemma III.1,
we have $t_{i}\in[0,1]$, which implies $P_{1}\leq q_{1}$ and $P_{2}\leq
1-q_{1}$. Therefore, $B>A$ and $C>A$ hold, i.e., the Hessian matrix has
positive diagonals. Furthermore, since $|\operatorname{Hess}I({\bf y};\hat{\bf
y})|>0$, Hessian matrix has two positive eigenvalues, i.e., it is a positive
definite matrix. This proves the (joint) convexity of $I({\bf y};\hat{\bf y})$
with respect to $P_{1}$ and $P_{2}$.

Now consider this analysis with respect to $\kappa_{1}$ and $\kappa_{2}$. By
using the definitions of the parameters, the Jacobian of $P_{1}$ and $P_{2}$
with respect to $\kappa_{1}$ and $\kappa_{2}$ can be calculated as

$\displaystyle J\triangleq\begin{bmatrix}{\partial
P_{1}\over\partial\kappa_{1}}&{\partial P_{1}\over\partial\kappa_{2}}\\\
{\partial P_{2}\over\partial\kappa_{1}}&{\partial
P_{2}\over\partial\kappa_{2}}\end{bmatrix}=(\delta_{1}-\delta_{2})\begin{bmatrix}a&-c\\\
-d&b\end{bmatrix}\,,$ (25)

and corresponding Hessian becomes

$\displaystyle\widetilde{\operatorname{Hess}}\,I({\bf y};\hat{\bf
y})={\partial^{2}I({\bf y};\hat{\bf
y})\over\partial\kappa_{i}\partial\kappa_{j}}=J^{T}\operatorname{Hess}\,I({\bf
y};\hat{\bf y})\,J\,.$ (26)

Here, by utilizing the analysis of $\operatorname{Hess}\,I({\bf y};\hat{\bf
y})$, it can be similarly shown that $\widetilde{\operatorname{Hess}}\,I({\bf
y};\hat{\bf y})$ has also positive diagonals and determinant, thus a positive
definite matrix, which shows the (joint) convexity of $I({\bf y};\hat{\bf y})$
with respect to $\kappa_{1}$ and $\kappa_{2}$.

Regarding the distortion part, it can be seen that the Hamming distortion in
(12) is an affine function of $\kappa_{1}$ and $\kappa_{2}$. Consequently,
$I({\bf y};\hat{\bf y})+\rho{\bf E}\\{d({\bf x},\hat{\bf x})\\}$ is a
(jointly) convex function of $\kappa_{1}$ and $\kappa_{2}$ since the first
term is convex and the second term is affine in $\kappa_{1}$ and $\kappa_{2}$.

As a result, for given decoder strategies $\gamma^{d_{\bf x}}$ and
$\gamma^{d_{\bf y}}$, the encoder chooses its strategy $\gamma^{e}$ as one of
its extreme points $\kappa_{1}\kappa_{2}=\\{00,01,10,11\\}$ which results in
the maximum $J^{e}$. This completes the proof.

##  Appendix C Proof of Lemma III.3

Suppose that Assumption III.1 holds. Then we obtain the following.

  * (i)

Using a similar approach in the proof of Lemma III.2, first the Jacobian of
$P_{1}$ and $P_{2}$ with respect to $\delta_{1}$ and $\delta_{2}$, and then
the Hessian of $I({\bf y};\hat{\bf y})$ with respect to $\delta_{1}$ and
$\delta_{2}$ can be calculated, and it can be shown that $I({\bf y};\hat{\bf
y})$ is a (jointly) convex function of $\delta_{1}$ and $\delta_{2}$. Then,
the decoder $\gamma^{d_{\bf y}}$ selects extreme points of $\delta_{1}$ and
$\delta_{2}$ in order to maximize $I({\bf y};\hat{\bf y})$. In particular, to
find the best response of the decoder $\gamma^{d_{\bf y}}$, it would suffice
to compare the values of $I({\bf y};\hat{\bf y})$ at four different
$\delta_{1}\delta_{2}$ pairs, i.e., $00$, $01$, $10$, and $11$, tabulated as
follows.

$\delta_{1}\delta_{2}$ | $t_{i}$ | $P_{1}$ | $P_{2}$
---|---|---|---
$00$ | $0$ | $0$ | $0$
$01$ | $1-{\kappa_{i}}$ | $a(1-\kappa_{1})+c\kappa_{2}$ | $b(1-\kappa_{2})+d\kappa_{1}$
$10$ | ${\kappa_{i}}$ | $a\kappa_{1}+c(1-\kappa_{2})$ | $b\kappa_{2}+d(1-\kappa_{1})$
$11$ | $0$ | $a+c$ | $b+d$

From (11), it can be seen that $\delta_{1}\delta_{2}=00$ and
$\delta_{1}\delta_{2}=11$ result in the same $I({\bf y};\hat{\bf y})$; and
similarly, $\delta_{1}\delta_{2}=01$ and $\delta_{1}\delta_{2}=10$ result in
the same $I({\bf y};\hat{\bf y})$. Note that, for $\delta_{1}\delta_{2}=00$,
we obtain $I({\bf y};\hat{\bf y})=0$, and since the mutual information is a
non-negative quantity, the best response of the decoder $\gamma^{d_{\bf y}}$
is $\delta_{1}\delta_{2}=01$ or $\delta_{1}\delta_{2}=10$.

  * (ii)

For a given encoder strategy ${\kappa_{i}}$, the decoder $\gamma^{d_{\bf x}}$
tries to minimize (12). We can write (12) in a different way as follows:

$\displaystyle{\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$
$\displaystyle=a(1-n_{1})+b(1-n_{2})+cn_{3}+dn_{4}$
$\displaystyle=a+b+\epsilon_{1}(-a\kappa_{1}-b\kappa_{2}+c\kappa_{3}+d\kappa_{4})+\epsilon_{2}(-a(1-\kappa_{1})-b(1-\kappa_{2})+c(1-\kappa_{3})+d(1-\kappa_{4}))$
$\displaystyle=a+b+\epsilon_{1}(c+d-\kappa_{1}(a+d)-\kappa_{2}(b+c))+\epsilon_{2}(-a-b+\kappa_{1}(a+d)+\kappa_{2}(b+c)).$

Thus, the optimal decoder strategy and corresponding distortion can be
characterized as in Table I. From there, it can be seen that the decoder
chooses its strategy $\gamma^{d_{\bf x}}$ so that

$\displaystyle{\bf E}\\{d_{H}({\bf x},\hat{\bf
x})\\}=\min\\{a+b,c+d,\theta,1-\theta\\}\,.$ (27)

This completes the proof.

##  Appendix D Proof of Theorem III.1

Suppose that Assumption III.1 holds. Then, for a given encoder action
$\kappa_{1}\kappa_{2}$, the best response of the decoder $\gamma^{d_{\bf y}}$
is $\delta_{1}\delta_{2}=01$ (or equivalently $\delta_{1}\delta_{2}=10$).
Then, the corresponding mutual information $I({\bf y};\hat{\bf y})$ becomes

$\displaystyle I({\bf y};\hat{\bf y})$
$\displaystyle=H_{b}(a+c)+H_{b}(a+b)+a\log(a)+b\log(b)+c\log(c)+d\log(d)\leq
H_{b}(a+c),$ (28)

for $\kappa_{1}\kappa_{2}=00$ or $\kappa_{1}\kappa_{2}=11$, and

$\displaystyle I({\bf y};\hat{\bf y})$ $\displaystyle=H_{b}(a+c),$ (29)

for $\kappa_{1}\kappa_{2}=01$ or $\kappa_{1}\kappa_{2}=10$. Thus, considering
only the mutual information part, the optimal encoder strategy $\gamma^{e}$ is
either $\kappa_{1}\kappa_{2}=01$ or $\kappa_{1}\kappa_{2}=10$.

Regarding the distortion part, the encoder aims to make it as great as
possible, more precisely, the encoder tries to
$\max_{\theta}\min\\{a+b,c+d,\theta,1-\theta\\}$. Note that if
$\min\\{a+b,1-(a+b),\theta,1-\theta\\}$ is $\theta$ or $1-\theta$, it is
possible to obtain greater $\min\\{a+b,c+d,\theta,1-\theta\\}$ by increasing
or decreasing $\theta$, respectively. Thus, we obtain
$\max_{\theta}\min\\{a+b,c+d,\theta,1-\theta\\}=\min\\{a+b,c+d\\}$. Here, we
can have two different conditions.

  1. (C1)

If $\min\\{a+b,c+d,a+d,b+c\\}$ is $a+b$ or $c+d$, then for
$\kappa_{1}\kappa_{2}=01\Rightarrow\theta=b+c$ or
$\kappa_{1}\kappa_{2}=10\Rightarrow\theta=a+d$, and we obtain ${\bf
E}\\{d_{H}({\bf x},\hat{\bf
x})\\}=\min\\{a+b,c+d,\theta,1-\theta\\}=\min\\{a+b,c+d\\}$, which is optimal
for the encoder. Thus, both the mutual information part $I({\bf y};\hat{\bf
y})$ and the distortion part ${\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$ are
maximized when $\kappa_{1}\kappa_{2}=01$ or $\kappa_{1}\kappa_{2}=10$.
Therefore, in this case, Stackelberg equilibrium strategies are given by (13).

  2. (C2)

If $\min\\{a+b,c+d,a+d,b+c\\}$ is $a+d$ or $b+c$, then for
$\kappa_{1}\kappa_{2}=01\Rightarrow\theta=b+c$ or
$\kappa_{1}\kappa_{2}=10\Rightarrow\theta=a+d$, and we get ${\bf
E}\\{d_{H}({\bf x},\hat{\bf
x})\\}=\min\\{a+b,c+d,\theta,1-\theta\\}=\min\\{\theta,1-\theta\\}$, which is
not an optimal choice for the encoder. The encoder selects its strategy so
that either $a+b\leq\theta\leq c+d$ or $a+b\geq\theta\geq c+d$ hold depending
on the prior. Therefore, even though $\kappa_{1}\kappa_{2}=01$ and
$\kappa_{1}\kappa_{2}=10$ maximize $I({\bf y};\hat{\bf y})$, they are not
optimal for maximizing ${\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$, thus the
optimal encoder strategy which maximizes $J^{e}(\gamma^{e},\gamma^{d_{\bf
x}},\gamma^{d_{\bf y}})=I({\bf y};\hat{\bf y})+\rho{\bf E}\\{d_{H}({\bf
x},\hat{\bf x})\\}$ may not be these strategies. Even though for sufficiently
small $\rho$ values, $\kappa_{1}\kappa_{2}=01$ and $\kappa_{1}\kappa_{2}=10$
characterizes the Stackelberg equilibrium, for large values of $\rho$,
equilibrium strategies change. For sufficiently large $\rho$ values, the
distortion (i.e., privacy) part becomes more dominant and the optimum encoder
strategies are designed so that either $a+b\leq\theta\leq c+d$ or
$a+b\geq\theta\geq c+d$ holds. In particular, for the privacy part, there are
infinitely many equivalent encoder strategies $\kappa_{1}\kappa_{2}$ which
satisfy either $a+b\leq\theta\leq c+d$ or $a+b\geq\theta\geq c+d$ resulting in
the same distortion, where $\theta\triangleq\kappa_{1}(a+d)+\kappa_{2}(b+c)$
as defined earlier. Then, due to the convexity of mutual information in
$\kappa_{1}\kappa_{2}$, the optimum encoder strategy lies at the boundary of
the $\kappa_{1}\kappa_{2}$ region (which satisfy either $a+b\leq\theta\leq
c+d$ or $a+b\geq\theta\geq c+d$).

This completes the proof.

##  Appendix E Proof of Theorem III.2

Suppose that Assumption III.1 holds. Then, recall that the best responses of
the players are as follows

  * •

for a given decoder strategy, the optimal encoder selects one of
$\kappa_{1}\kappa_{2}=\\{00,01,10,11\\}$.

  * •

for a given encoder strategy, the optimal decoder-$\hat{\bf y}$ selects either
$\delta_{1}\delta_{2}=01$ or $\delta_{1}\delta_{2}=10$, which result in
identical $I({\bf y};\hat{\bf y})$.

  * •

for a given encoder strategy, the optimal decoder-$\hat{\bf x}$ selects one of
$\epsilon_{1}\epsilon_{2}=\\{00,01,10,11\\}$.

Therefore, the game can be expressed in a normal form with
$(\kappa_{1}\kappa_{2},\epsilon_{1}\epsilon_{2})$ (which correspond to the
rows for the encoder strategies and the columns for the decoder-$\hat{\bf x}$
strategies, respectively) and corresponding costs333Note that comparing
$J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})$ and
$J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})-H_{b}(q_{1})$ does
not affect the equilibrium, and $M$ represents a non-positive value
$M\triangleq H_{b}(a+b)+a\log(a)+b\log(b)+c\log(c)+d\log(d)$.
$J^{e}(\gamma^{e},\gamma^{d_{\bf x}},\gamma^{d_{\bf y}})-H_{b}(q_{1})$ and
${\bf E}\\{d_{H}({\bf x},\hat{\bf x})\\}$.

| $00$ | $01$ | $10$ | $11$
---|---|---|---|---
$00$ | $(M+\rho(a+b),a+b)$ | $(M,0)$ | $(M+\rho,1)$ | $(M+\rho(c+d),c+d)$
$01$ | $(\rho(a+b),a+b)$ | $(\rho(b+c),b+c)$ | $(\rho(a+d),a+d)$ | $(\rho(c+d),c+d)$
$10$ | $(\rho(a+b),a+b)$ | $(\rho(a+d),a+d)$ | $(\rho(b+c),b+c)$ | $(\rho(c+d),c+d)$
$11$ | $(M+\rho(a+b),a+b)$ | $(M+\rho,1)$ | $(M,0)$ | $(M+\rho(c+d),c+d)$

Here, it can be seen that,

  * •

if $\min\\{a+b,c+d,a+d,b+c\\}=a+b$, then
$(\kappa_{1}\kappa_{2}=01,\epsilon_{1}\epsilon_{2}=00)$ and
$(\kappa_{1}\kappa_{2}=10,\epsilon_{1}\epsilon_{2}=00)$ are equilibrium
points,

  * •

if $\min\\{a+b,c+d,a+d,b+c\\}=c+d$, then
$(\kappa_{1}\kappa_{2}=01,\epsilon_{1}\epsilon_{2}=11)$ and
$(\kappa_{1}\kappa_{2}=10,\epsilon_{1}\epsilon_{2}=11)$ are equilibrium
points,

  * •

otherwise, a pure Nash equilibrium does not exist (excluding the trivial case
of equal priors $a=b=c=d=0.25$ in which any strategy pair ends up an
equilibrium).

This completes the proof.

## References

  * [1] P. McDaniel and S. McLaughlin, “Security and privacy challenges in the smart grid,” _IEEE Security Privacy_ , vol. 7, no. 3, pp. 75–77, 2009.
  * [2] S. Finster and I. Baumgart, “Privacy-aware smart metering: A survey,” _IEEE Communications Surveys Tutorials_ , vol. 17, no. 2, pp. 1088–1101, 2015\.
  * [3] J. Le Ny and G. J. Pappas, “Differentially private filtering,” _IEEE Transactions on Automatic Control_ , vol. 59, no. 2, pp. 341–354, 2014.
  * [4] J. Gómez-Vilardebó and D. Gündüz, “Smart meter privacy for multiple users in the presence of an alternative energy source,” _IEEE Transactions on Information Forensics and Security_ , vol. 10, no. 1, pp. 132–141, 2015.
  * [5] Z. Li, T. J. Oechtering, and D. Gündüz, “Privacy against a hypothesis testing adversary,” _IEEE Transactions on Information Forensics and Security_ , vol. 14, no. 6, pp. 1567–1581, 2019.
  * [6] E. Nekouei, T. Tanaka, M. Skoglund, and K. H. Johansson, “Information-theoretic approaches to privacy in estimation and control,” _Annual Reviews in Control_ , vol. 47, pp. 412–422, 2019.
  * [7] Y. Lu and M. Zhu, “On privacy preserving data release of linear dynamic networks,” _Automatica_ , vol. 115, p. 108839, 2020.
  * [8] B. Cavarec, P. A. Stavrou, M. Bengtsson, and M. Skoglund, “Designing privacy filters for hidden Markov processes,” in _European Control Conference (ECC)_ , 2021.
  * [9] V. P. Crawford and J. Sobel, “Strategic information transmission,” _Econometrica_ , vol. 50, pp. 1431–1451, 1982.
  * [10] I. Shames, A. M. H. Teixeira, H. Sandberg, and K. H. Johansson, “Agents misbehaving in a network: a vice or a virtue?” _IEEE Network_ , vol. 26, no. 3, pp. 35–40, May 2012.
  * [11] B. Larrousse, O. Beaude, and S. Lasaulce, “Crawford-Sobel meet Lloyd-Max on the grid,” in _IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , May 2014, pp. 6127–6131.
  * [12] J. Miklós-Thal and H. Schumacher, “The value of recommendations,” _Games and Economic Behavior_ , vol. 79, pp. 132–147, 2013.
  * [13] O. Ben-Porat and M. Tennenholtz, “A game-theoretic approach to recommendation systems with strategic content providers,” in _International Conference on Neural Information Processing Systems (NeurIPS)_ , 2018, p. 1118–1128.
  * [14] J. G. Riley, “Silver signals: Twenty-five years of screening and signaling,” _Journal of Economic Literature_ , vol. 39, no. 2, pp. 432–478, June 2001\.
  * [15] J. Sobel, “Signaling games,” in _Encyclopedia of Complexity and Systems Science_ , R. A. Meyers, Ed. Springer New York, 2009, pp. 8125–8139.
  * [16] E. Kamenica and M. Gentzkow, “Bayesian persuasion,” _American Economic Review_ , vol. 101, no. 6, pp. 2590–2615, Oct. 2011.
  * [17] S. Sarıtaş, S. Yüksel, and S. Gezici, “Quadratic multi-dimensional signaling games and affine equilibria,” _IEEE Transactions on Automatic Control_ , vol. 62, no. 2, pp. 605–619, Feb. 2017.
  * [18] F. Farokhi, A. M. H. Teixeira, and C. Langbort, “Estimation with strategic sensors,” _IEEE Transactions on Automatic Control_ , vol. 62, no. 2, pp. 724–739, Feb. 2017.
  * [19] E. Akyol, C. Langbort, and T. Başar, “Information-theoretic approach to strategic communication as a hierarchical game,” _Proceedings of the IEEE_ , vol. 105, no. 2, pp. 205–218, Feb. 2017.
  * [20] M. O. Sayin, E. Akyol, and T. Başar, “Hierarchical multistage Gaussian signaling games in noncooperative communication and control systems,” _Automatica_ , vol. 107, pp. 9–20, 2019.
  * [21] S. Sarıtaş, S. Yüksel, and S. Gezici, “Dynamic signaling games with quadratic criteria under Nash and Stackelberg equilibria,” _Automatica_ , vol. 115, p. 108883, May 2020.
  * [22] M. le Treust and T. Tomala, “Strategic communication with decoder side information,” in _IEEE International Symposium on Information Theory (ISIT)_ , 2021, pp. 2696–2701.
  * [23] M. L. Treust and T. Tomala, “Persuasion with limited communication capacity,” _Journal of Economic Theory_ , vol. 184, p. 104940, 2019.
  * [24] S. Sarıtaş, G. Dán, and H. Sandberg, “Passive fault-tolerant estimation under strategic adversarial bias,” in _American Control Conference (ACC)_ , 2020, pp. 4644–4651.
  * [25] E. Kazıklı, S. Sarıtaş, S. Gezici, and S. Yüksel, “Quadratic signaling with prior mismatch at an encoder and decoder: Equilibria, continuity and robustness properties,” _IEEE Transactions on Automatic Control_ , pp. 1–1, 2022.
  * [26] S. Sarıtaş, P. A. Stavrou, R. Thobaben, and M. Skoglund, “Quadratic signaling games with channel combining ratio,” in _IEEE International Symposium on Information Theory (ISIT)_ , 2021, pp. 2690–2695.
  * [27] F. Farokhi, H. Sandberg, I. Shames, and M. Cantoni, “Quadratic Gaussian privacy games,” in _54th IEEE Conference on Decision and Control (CDC)_ , 2015, pp. 4505–4510.
  * [28] E. Akyol, C. Langbort, and T. Başar, “Privacy constrained information processing,” in _54th IEEE Conference on Decision and Control (CDC)_ , 2015, pp. 4511–4516.
  * [29] F. Farokhi and G. Nair, “Privacy-constrained communication,” _IFAC-PapersOnLine_ , vol. 49, no. 22, pp. 43–48, 2016.
  * [30] E. Kazikli, S. Gezici, and S. Yüksel, “Quadratic privacy-signaling games and the MMSE information bottleneck problem for Gaussian sources,” _arxiv.org_ , vol. abs/2005.05743v3, 2022. [Online]. Available: https://arxiv.org/abs/2005.05743v3
  * [31] I. Wagner and D. Eckhoff, “Technical privacy metrics: A systematic survey,” _ACM Comput. Surv._ , vol. 51, no. 3, June 2018.
  * [32] T. M. Cover and J. A. Thomas, _Elements of Information Theory_ , 2nd ed. John Wiley & Sons, Inc., Hoboken, New Jersey, 2006.
  * [33] T. Başar and G. J. Olsder, _Dynamic Noncooperative Game Theory_. Philadelphia, PA: SIAM Classics in Applied Mathematics, 1999.


# Reverberant Sound Localization with a Robot Head
Based on Direct-Path Relative Transfer Function

Xiaofei Li†, Laurent Girin†,‡,#, Fabien Badeig† and Radu Horaud†
†INRIA Grenoble Rhone-Alpes, ‡GIPSA-LAB, #Univ. Grenoble Alpes This research
has received funding from the EU-FP7 STREP project EARS (#609465).

###### Abstract

This paper addresses the problem of sound-source localization (SSL) with a
robot head, which remains a challenge in real-world environments. In
particular we are interested in locating speech sources, as they are of high
interest for human-robot interaction. The microphone-pair response
corresponding to the direct-path sound propagation is a function of the source
direction. In practice, this response is contaminated by noise and
reverberations. The direct-path relative transfer function (DP-RTF) is defined
as the ratio between the direct-path acoustic transfer function (ATF) of the
two microphones, and it is an important feature for SSL. We propose a method
to estimate the DP-RTF from noisy and reverberant signals in the short-time
Fourier transform (STFT) domain. First, the convolutive transfer function
(CTF) approximation is adopted to accurately represent the impulse response of
the microphone array, and the first coefficient of the CTF is mainly composed
of the direct-path ATF. At each frequency, the frame-wise speech auto- and
cross-power spectral density (PSD) are obtained by spectral subtraction. Then
a set of linear equations is constructed by the speech auto- and cross-PSD of
multiple frames, in which the DP-RTF is an unknown variable, and is estimated
by solving the equations. Finally, the estimated DP-RTFs are concatenated
across frequencies and used as a feature vector for SSL. Experiments with a
robot, placed in various reverberant environments, show that the proposed
method outperforms two state-of-the-art methods.

##  I Introduction

Sound source localization (SSL) is a crucial methodology for robot audition.
This paper addresses the problem of real-world SSL using a microphone array
embedded into a robot head. The NAO robot (version 5) is used in this paper,
whose head and its four embedded microphones are shown on Fig. 1.

Microphone-array processing SSL techniques are widely adopted for robot
audition, e.g., [1, 2, 3, 4, 5, 6]. These techniques generally need a large
number of microphones and high computational cost. The time difference of
arrival (TDOA) techniques [7, 8] are suitable if fewer microphones are
available, however they are generally applied to a free-field setup, in which
the TDOA is frequency-independent. We address SSL in the more general case,
namely when the source-to-sensor sound propagation is affected by the robot’s
head and torso, e.g., binaural audition [9, 10], as well as by the room
acoustics [11], and these effects are frequency-dependent [12].

As shown in Fig. 1, four microphones are embedded in NAO’s head. The two most
discriminative microphone pairs in terms of SSL, i.e., the two cross
microphone pairs (A-C and B-D) are used in this paper. The acoustic features
are extracted separately from these two microphone pairs, and then these
pairwise features are combined together. Two interaural cues, the interaural
time (or phase) difference (ITD or IPD) and the interaural level difference
(ILD), are widely used for SSL. When computed using the STFT, the ILD and IPD
correspond to the magnitude and phase of a two-channel relative transfer
function (RTF), which is the ratio between the ATFs of the two microphones
[13]. The interaural cues, or equivalently the two-channel RTF, that
correspond to the direct-path sound propagation are a function of the source
direction, which is to be estimated from noisy and reverberant sensor signals,
as they are available in a real environment.

Techniques have been proposed to identify the RTF in noisy environments, such
as a speech non-stationary estimator [13], an RTF identification method based
on speech presence probability and spectral subtraction [14], and an RTF
estimator based on segmental PSD matrix subtraction [15]. In these RTF
estimators, the multiplicative transfer function (MTF) approximation [16] is
assumed. This approximation is justified only when the length of the room
impulse response is shorter than the length of the STFT window, which is
rarely the case in realistic acoustic setups. Moreover, the RTF estimated
above is the ratio between two ATFs that include the reverberations and hence
it is poorly suitable for SSL in echoic environments.

Figure 1: The version 5 of the NAO head has four microphones, namely A, B, C,
and D. This robot-head configuration is used in our experiments to illustrate
the proposed SSL method.

Techniques have been proposed to extract the interaural cues that correspond
to the direct-path sound propagation, e.g. based on the detection of time
frames with less reverberations. The precedence effect [17] is widely modeled
for SSL [18, 19], which relies on the principle that the onset frame is
dominated by the direct-path wavefront [20, 21]. In the STFT domain, the
coherence test [22] and the direct-path dominance test [23] are proposed to
detect the frames dominated by one active source (namely only the direct-path
propagation), from which reliable localization cues are estimated. However, in
practice, there are always reflection components in the frames selected by
these algorithms due to the inaccurate model or an improper decision
threshold.

In this paper we propose a direct-path RTF estimator suitable for the
localization of a single speech source in the real world. We build on the
crossband filter proposed in [24] (actually a simplified CTF approximation
proposed in [25]) for system identification. This filter accurately
characterizes the impulse response in the STFT domain by a convolutive
transfer function instead of the MTF approximation. The first coefficient of
the CTF at different frequencies represents the STFT of the first segment of
the channel impulse response, which is composed of the impulse response of the
direct-path propagation and possibly a few reflections. Therefore, we refer to
the first coefficient of the CTF as the direct-path ATF, and the ratio between
the coefficients from the two channels as the direct-path RTF (DP-RTF). For
the noise-free case, inspired by [26], based on the relation of the CTFs
between the two channels, we construct a set of linear equations using the
auto- and cross-power spectral density (PSD) of the speech signal received by
the microphones.

At each frequency, the DP-RTF is the unknown variable of the linear equations,
and can be estimated from these equations using the least square estimator.
However, in practice, the sensor signals are always contaminated by noise. The
speech PSD constructing the linear equations can be obtained by subtracting
the noise PSD from the sensor signal PSD. Finally, the estimated DP-RTFs are
concatenated over microphone pairs and frequencies, and mapped to the source
direction space using the probabilistic piecewise affine mapping model [27].
Experiments, conducted in various real-world environments, show the
effectiveness of the proposed method.

The remainder of this paper is organized as follows. Section II formulates the
sensor signals based on the crossband filter. Section III presents the DP-RTF
estimator. In Section IV, the SSL algorithm based on the probabilistic
piecewise affine mapping model is described. Experimental results are
presented in Section V, and Section VI draws some conclusions.

##  II Signal Formulation Based on Crossband Filter

In this work, we process microphone pairs separately. Thence, without loss of
generality, only the sensor signals of one microphone pair are defined in this
section, analyzed in section III, and the acoustic features of several
microphone pairs will be combined for SSL in Section IV.

Let us consider a non-stationary source signal, e.g., a speech source $s(n)$
in the time domain. In a noise-free environment, the microphone-pair signals
are

$\displaystyle x(n)=a(n)*s(n),\quad y(n)=b(n)*s(n),$ (1)

where $*$ denotes convolution, $a(n)$ and $b(n)$ are the room impulse
responses from the source to the first and second microphone, respectively.
Let $T$ denote the length of $a(n)$ and $b(n)$. Applying the STFT, based on
the MTF approximation, microphone signal $x(n)$ is approximated in the time-
frequency (TF) domain as $x_{p,k}=s_{p,k}a_{k}$, where $x_{p,k}$ and $s_{p,k}$
are the STFT of the corresponding signals, $p$ and $k$ are the indexes of time
frame and frequency bin, respectively. Let $N$ denote the length of the STFT
window (frame). This MTF approximation is only valid when the impulse response
length $T$ is lower than $N$. For a non-stationary acoustic signal, such as
speech, a small length $N$ (around 20 ms) is typically chosen to assume
‘local’ stationarity, i.e. in each frame. Therefore the MTF approximation is
questionable in a, possibly strongly, reverberant environment with a long room
impulse response.

To address this problem, the crossband filter was introduced in [24] to
represent a linear system in the STFT domain more accurately. Let
$\tilde{\omega}(n)$ and $\omega(n)$ denote the analysis and synthesis STFT
windows respectively, and let $L$ denote the frame step. The crossband filter
model consists of representing the STFT coefficient $x_{p,k}$ as a summation
of multiple convolutions across frequency bands. A CTF approximation is
further introduced in [25] to simplify the analysis, i.e. using only band-to-
band filters as

$\displaystyle x_{p,k}$
$\displaystyle=\sum_{p^{\prime}=0}^{Q_{k}-1}s_{p-p^{\prime},k}a_{p^{\prime},k}=s_{p,k}*a_{p,k},$
(2)

where convolution is applied to the time variable $p$. The frequency dependent
CTF length $Q_{k}$ is related to the reverberation at the $k$th frequency
band, which will be discussed in section V. The TF-domain impulse response
$a_{p^{\prime},k}$ is related to the time-domain impulse response $a(n)$ by:

$\displaystyle a_{p^{\prime},k}={a(n)*\zeta_{k}(n)}|_{n=p^{\prime}L},$ (3)

which represents the convolution with respect to the time index $n$ evaluated
at frame steps, with

$\displaystyle\zeta_{k}(n)=e^{j\frac{2\pi}{N}kn}\sum_{m}\tilde{\omega}(m)\omega(n+m).$
(4)

In the next section, the CTF formalism is used to extract the impulse response
of the direct-path propagation.

##  III Direct-Path Relative Transfer Function

###  III-A Definition of DP-ATF and DP-RTF Based on CTF

In the CTF approximation (2), using (3) and (4) at $p^{\prime}=0$, the first
coefficient of $a_{p^{\prime},k}$ can be derived as

$\displaystyle
a_{0,k}={a(n)*\zeta_{k,k}(n)}|_{n=0}=\sum\nolimits_{t=0}^{N-1}a(t)\nu(t)e^{-j\frac{2\pi}{N}kt},$
(5)

where

$\nu(t)=\begin{cases}\sum_{m=0}^{N}\tilde{\omega}(m)\omega(m-t)&\mbox{if
}1-N\leq t\leq N-1,\\\ 0,&\mbox{otherwise.}\end{cases}$

Therefore, $a_{0,k}$ can be interpreted as the $k$-th Fourier coefficient of
the impulse response segment $a(n)|_{n=0}^{N-1}$ (windowed by
$\nu(t)|_{n=0}^{N-1}$). In the sense of transfer function identification,
without loss of generality, we assume that the room impulse response $a(n)$
begins with the impulse response of the direct-path sound propagation. If the
frame length $N$ is properly chosen, $a(n)|_{n=0}^{N-1}$ is composed of the
impulse responses of the direct-path propagation and a few reflections.
Particularly, if the initial time delay gap (ITDG) is large compared to the
frame length $N$, $a(n)|_{n=0}^{N-1}$ is mainly composed of the direct-path
impulse response. Thence we refer to $a_{0,k}$ as the direct-path ATF.

Similarly, the CTF approximation of $y_{p,k}$ is written as

$\displaystyle y_{p,k}=s_{p,k}*b_{p,k},$ (6)

and $b_{0,k}$ is assumed to represent the direct-path ATF from the source to
the second microphone. By definition, DP-RTF is given by:
$\frac{b_{0,k}}{a_{0,k}}$. Let us remind that this DP-RTF is a relevant cue
for SSL.

###  III-B DP-RTF Estimation

Since both channels are assumed to follow the CTF model, we can write:

$\displaystyle x_{p,k}*b_{p,k}=s_{p,k}*a_{p,k}*b_{p,k}=y_{p,k}*a_{p,k}.$ (7)

In [26], this relation is proposed in time domain for TDOA estimation. Eq.(7)
can be written in vector form as

$\displaystyle\mathbf{x}_{p,k}^{\top}\mathbf{b}_{k}=\mathbf{y}_{p,k}^{\top}\mathbf{a}_{k}$
(8)

where ⊤ denotes vector or matrix transpose, and

$\displaystyle\mathbf{x}_{p,k}=[x_{p,k},x_{p-1,k},\dots,x_{p-Q_{k}+1,k}]^{\top},$
$\displaystyle\mathbf{y}_{p,k}=[y_{p,k},y_{p-1,k},\dots,y_{p-Q_{k}+1,k}]^{\top},$
$\displaystyle\mathbf{b}_{k}=[b_{0,k},b_{1,k},\dots,b_{Q_{k}-1,k}]^{\top},$
$\displaystyle\mathbf{a}_{k}=[a_{0,k},a_{1,k},\dots,a_{Q_{k}-1,k}]^{\top}.$
(9)

Dividing both sides of (8) by $a_{0,k}$ and reorganizing the terms, we can
write:

$\displaystyle y_{p,k}=\mathbf{z}_{p,k}^{\top}\mathbf{g}_{k},$ (10)

where

$\displaystyle\mathbf{z}_{p,k}=[x_{p,k},\dots,x_{p-Q_{k}+1,k},y_{p-1,k},\dots,y_{p-Q_{k}+1,k}]^{\top}$
$\displaystyle\mathbf{g}_{k}=\left[\frac{b_{0,k}}{a_{0,k}},\dots,\frac{b_{Q_{k}-1,k}}{a_{0,k}},-\frac{a_{1,k}}{a_{0,k}},\dots,-\frac{a_{Q_{k}-1,k}}{a_{0,k}}\right]^{\top}.$
(11)

We see that the DP-RTF appears as the first entry of $\mathbf{g}_{k}$. Hence,
in the following, we base the estimation of the DP-RTF on the construction of
$y_{p,k}$ and $\mathbf{z}_{p,k}$ statistics. More specifically, multiplying
both sides of (10) by $y_{p,k}^{*}$ (∗ denotes complex conjugation) and taking
the expectation (denoted by $E\\{\cdot\\}$), we obtain:

$\displaystyle\phi_{yy}(p,k)=\varphi_{zy}^{\top}(p,k)\mathbf{g}_{k},$ (12)

where $\phi_{yy}(p,k)=E\\{y_{p,k}y_{p,k}^{*}\\}$ is the PSD of $y(p,k)$, and

$\displaystyle\varphi_{zy}(p,k)$
$\displaystyle=[E\\{x_{p,k}y_{p,k}^{*}\\},\dots,E\\{x_{p-Q_{k}+1,k}y_{p,k}^{*}\\},$
$\displaystyle
E\\{y_{p-1,k}y_{p,k}^{*}\\},\dots,E\\{y_{p-Q_{k}+1,k}y_{p,k}^{*}\\}]^{\top}$
(13)

is a vector which is composed of cross-PSD terms between the elements of
$\mathbf{z}_{p,k}$ and $y_{p,k}$. In practice, these auto- and cross-PSD terms
can be estimated by averaging the corresponding spectra over a number $D$ of
frames, i.e.:

$\displaystyle\hat{\phi}_{yy}(p,k)=\frac{1}{D}\sum_{d=0}^{D-1}y_{p-d,k}y_{p-d,k}$
(14)

The elements in $\varphi_{zy}(p,k)$ can be estimated by using the same
principle. Consequently, (12) is approximated as

$\displaystyle\hat{\phi}_{yy}(p,k)=\hat{\varphi}_{zy}^{\top}(p,k)\mathbf{g}_{k}.$
(15)

In this equation, the speech PSD $\hat{\phi}_{yy}(p,k)$ and
$\hat{\varphi}_{zy}^{\top}(p,k)$ can be obtained from the noise-free sensor
signals. However in the real world, the PSD of speech signals are deteriorated
by noise.

###  III-C Speech PSD Estimate in the Presence of Noise

Noise signals are added into the sensor signals in (1) as

$\displaystyle\tilde{x}(n)=x(n)+u(n)=a(n)*s(n)+u(n),$
$\displaystyle\tilde{y}(n)=y(n)+v(n)=b(n)*s(n)+v(n),$ (16)

where $u(n)$ and $v(n)$ are the noise signals in two sensors, respectively,
which are supposed to be stationary and uncorrelated to the speech signal
$s(n)$.

Applying the STFT to the sensor signals in (III-C):
$\tilde{x}_{p,k}=x_{p,k}+u_{p,k}$ and $\tilde{y}_{p,k}=y_{p,k}+v_{p,k}$,
respectively, in which each quantity is the STFT coefficient of its
corresponding time domain signal. Similar to ${\mathbf{z}}_{p,k}$, we define

$\displaystyle\tilde{\mathbf{z}}_{p,k}$
$\displaystyle=[\tilde{x}_{p,k},\dots,\tilde{x}_{p-Q_{k}+1,k},\tilde{y}_{p-1,k},\dots,\tilde{y}_{p-Q_{k}+1,k}]^{\top}$
$\displaystyle=\mathbf{z}_{p,k}+\mathbf{w}_{p,k}$ (17)

where

$\displaystyle\mathbf{w}_{p,k}=[u_{p,k},\dots,u_{p-Q_{k}+1,k},v_{p-1,k},\dots,v_{p-Q_{k}+1,k}]^{\top}.$
(18)

We define the PSD of $\tilde{y}_{p,k}$ as $\phi_{\tilde{y}\tilde{y}}(p,k)$. We
also define the PSD vector $\varphi_{\tilde{z}\tilde{y}}(p,k)$, which is
composed of the auto- or cross-PSD between the elements of
$\tilde{\mathbf{z}}_{p,k}$ and $\tilde{y}_{p,k}$. Following the principle in
(14), by averaging the auto or cross spectra of multiple frames, these PSDs
can be estimated using the STFT coefficients of input signals as
$\hat{\phi}_{\tilde{y}\tilde{y}}(p,k)$ and
$\hat{\varphi}_{\tilde{z}\tilde{y}}(p,k)$. Because the speech and noise
signals are not correlated, they can be represented as

$\displaystyle\hat{\phi}_{\tilde{y}\tilde{y}}(p,k)=\hat{\phi}_{yy}(p,k)+\hat{\phi}_{vv}(p,k)$
$\displaystyle\hat{\varphi}_{\tilde{z}\tilde{y}}(p,k)=\hat{\varphi}_{zy}(p,k)+\hat{\varphi}_{wv}(p,k)$
(19)

where $\hat{\phi}_{vv}(p,k)$ is an estimation of the PSD of $v_{p,k}$, the PSD
vector $\hat{\varphi}_{wv}(p,k)$ is composed of the estimated auto or cross
PSD between the elements of ${\mathbf{w}}_{p,k}$ and ${v}_{p,k}$. The auto-
and cross-PSD of noise can be subtracted by using the noise estimator [28] or
the inter-frame spectral subtraction technique [15]. In this work, for
simplicity, we assume that noise is stationary (for example, the robot’s ego-
noise), and the noise-only signal is available, from which the noise PSD
${\phi}_{vv}(p,k)$ and ${\varphi}_{wv}(p,k)$ can be computed in advance.
Consequently, we approximately compute the speech PSD as

$\displaystyle\hat{\phi}_{yy}(p,k)$
$\displaystyle\approx\hat{\phi}_{\tilde{y}\tilde{y}}(p,k)-{\phi}_{vv}(p,k)$
$\displaystyle\hat{\varphi}_{zy}(p,k)$
$\displaystyle\approx\hat{\varphi}_{\tilde{z}\tilde{y}}(p,k)-{\varphi}_{wv}(p,k).$
(20)

Because of the temporal sparsity of the speech signal, parts of the frames are
dominated by noise, which should be disregarded for DP-RTF estimation. Thence
we define the frame index set that comprises the frames with considerable
speech power as

$\displaystyle\mathbf{p}_{k}=\\{p\ |\
\hat{\phi}_{\tilde{y}\tilde{y}}(p,k)>\gamma{\phi}_{vv}(p,k)\\},$ (21)

where $\gamma$ is a power threshold. Let $P_{k}=|\mathbf{p}_{k}|$ denote the
cardinal of $\mathbf{p}_{k}$.

###  III-D Direct-Path Relative Transfer Function Estimation

Based on the speech PSD estimated in (III-C), by concatenating across frames,
(15) could be written in matrix form

$\displaystyle\hat{\Phi}_{yy}(k)=\hat{\Psi}_{zy}(k)\mathbf{g}_{k}.$ (22)

where

$\displaystyle\hat{\Phi}_{yy}(k)=[\dots,\hat{\phi}_{yy}(p,k),\dots]^{\top},\quad
p\in\mathbf{p}_{k}$
$\displaystyle\hat{\Psi}_{zy}(k)=[\dots,\hat{\varphi}_{zy}(p,k),\dots]^{\top},\quad
p\in\mathbf{p}_{k}$

are $P_{k}\times 1$ vector, $P_{k}\times(2Q_{k}-1)$ matrix, respectively.

A least-square (LS) solution to (22) is given as

$\displaystyle\hat{\mathbf{g}}_{k}=(\hat{\Psi}_{zy}^{H}(k)\hat{\Psi}_{zy}(k))^{-1}\hat{\Psi}_{zy}(k)\hat{\Phi}_{yy}(k)$
(23)

where H denotes matrix conjugate transpose, -1 denotes maxtrix inverse. The
first element of $\hat{\mathbf{g}}_{k}$ is denoted as $\hat{g}_{k}$, which is
an estimation of DP-RTF $\frac{b_{0,k}}{a_{0,k}}$.

##  IV Sound Source Localization Method

The amplitude and the phase of DP-RTF is equivalent to the IPD and ILD
interaural cues corresponding to the direct-path propagation. As discussed in
[29, 30], when the reference transfer function $a_{0,k}$ is much smaller than
$b_{0,k}$, the amplitude ratio estimation is sensitive to the noise in the
reference channel. Therefore, we normalize $\hat{g}_{k}$ as

$\displaystyle\hat{c}_{k}=\frac{\hat{g}_{k}}{\sqrt{|\hat{g}_{k}|^{2}+1}}.$
(24)

It is clear that the phase is retained, and the amplitude is normalized as
$0<|\hat{c}_{k}|<1$.

The quantity $\hat{c}_{k}$ is the estimated DP-RTF for one microphone pair,
where the index of microphone pair is omitted. Concatenating the estimated DP-
RTF of microphone pairs A-C and B-D, yields
$\hat{\mathbf{c}}_{k}=[\hat{c}_{k,AC},\hat{c}_{k,BD}]^{\top}$.111For NAO
version 5, a total of six microphone pairs are available. However, experiments
show that it is sufficient to consider two microphone pairs. Then,
concatenating $\hat{\mathbf{c}}_{k}$ across frequencies, we obtain a global
feature vector in ${\mathbb{C}}^{2K}$:

$\hat{\mathbf{c}}=[\hat{\mathbf{c}}_{0}^{\top},\dots,\hat{\mathbf{c}}_{k}^{\top},\dots,\hat{\mathbf{c}}_{K-1}^{\top}]^{\top},$
(25)

where $K$ denotes the number of frequencies involved in SSL.

To map the high-dimensional feature vector $\hat{\mathbf{c}}$ to a low-
dimensional source direction $\mathbf{o}\in\mathbb{R}^{O}$ ($O$ denote the
dimension of source direction), we adopt the regression method proposed in
[27]. Briefly, a probabilistic piecewise-linear regression
$f:\mathbb{C}^{2K}\rightarrow\mathbb{R}^{O}$ is learned from a training
dataset $\\{\mathbf{c}_{i},\mathbf{o}_{i}\\}_{i=1}^{I}$, where
$\mathbf{c}_{i}$ is a feature vector and $\mathbf{o}_{i}$ is the corresponding
sound-source direction. Then, for a test DP-RTF feature vector
$\hat{\mathbf{c}}$ extracted from the microphone signals, the source direction
is predicted with $\hat{\mathbf{o}}=f(\hat{\mathbf{c}})$.

Due to the sparsity of speech signals in the STFT domain, it is possible that
there are only a few significant speech frames at frequency $k$ for one
microphone pair, especially in the case of low SNR. In other words, $P_{k}$
could be small, which makes the estimated $\hat{c}_{k}$ unreliable. To
disregard the unreliable $\hat{c}_{k}$ in the regression procedure, we
introduce a missing data indicator vector $\mathbf{h}\in\mathbb{R}^{2K}$. If
the matrix $\hat{\Psi}_{zy}(k)$ in (22) is underdetermined, i.e.,
$P_{k}<2Q_{k}-1$, its corresponding element in $\mathbf{h}$ is set to 0, and 1
otherwise. The regression method that we use [27] makes use of such an
indicator vector $\mathbf{h}$ and the element in $\hat{\mathbf{c}}$ with a 0
indicator is disregarded. The revised prediction is
$\hat{\mathbf{o}}=f(\hat{\mathbf{c}},\mathbf{h})$.

##  V Experiments with the Nao Robot

In this section several experiments using the NAO robot (version 5) are
conducted in various real-world environments. From Fig. 1, one can see that
four microphones are nearly coplanar, and that the angle between the
microphone plane and the horizontal plane is small. The microphones are close
to the head’s fan (the circular ear in Fig. 1), thence the microphone
recording include ego-noise due to the fan. As mentioned in [31], the fan
noise is stationary and spatially correlated. In addition, its spectral energy
mainly concentrates in a frequency range of up to 4 kHz, thence the recorded
speech signal will be contaminated by the fan noise significantly.

###  V-A The Datasets

The data are recorded in four real world environments: meeting room,
laboratory, office, e.g., Fig. 2, and cafeteria, whose reverberation time
$T_{60}$ are approximately 1.04 s, 0.52 s, 0.47 s and 0.24 s, respectively.

Figure 2: A typical audio-only localization experiment in the office
environment. The robot turns its head towards the speaking person shown on the
screen (please see the supplementary video).

Two test datasets are recorded in these environments:

  * •

The _Audio-only_ dataset: in the laboratory, the speech recording from the
TIMIT dataset [32] are emitted by a loudspeaker. Two groups of data are
recorded with a fixed robot-to-source distance of 1.1 m and 2.5 m,
respectively. Besides $T_{60}$, ITDG and direct-to-reverberation ratio (DRR)
are also important to measure the intensity of the reverberation. In general,
the larger the robot-to-source distance the less ITDG and DRR. Obviously, the
two cross microphone pairs allow a $360^{\circ}$ azimuth localization.
However, because of the limitation of NAO’s head joint, NAO’s head can not
rotate in a $360^{\circ}$ azimuth range. Thence, for each group, 174 sounds
are emitted from directions uniformly distributed in the range $-120^{\circ}$
to $120^{\circ}$ (azimuth), and $-15^{\circ}$ to 25∘ (elevation).

  * •

The _Audio-visual_ dataset: Fig. 1 shows the NAO head camera, with a field-of-
view of 60.97${}^{\circ}\times$47.64∘; speech sounds are emitted by a
loudspeaker lying in the camera’s field of view. The image resolution is of
640$\times$480 pixels, so 1∘ of azimuth/elevation corresponds to about 10.5
horizontal/vertical pixels. For this dataset, the source direction corresponds
to a pixel in the image. The ground-truth source direction is obtained by
localizing in the image the visual marker fixed on the loudspeaker. Four
groups of data are recorded in four rooms, respectively. For each group, about
230 sounds are emitted from directions uniformly distributed in the the camera
field-of-view. As an example, Fig. 3 illustrates the 228 directions shown as
blue dots in the image plane. The robot-to-source distance is approximately
fixed as 2 m in this dataset.

Figure 3: The audio-visual training dataset contains sound sources emitted by
a loud-speaker that correspond to sound directions materialized by image
locations (marked as blue circles).

In both of these two datasets, the external noise is much lower than the fan
noise, thence noise in the recorded signal is almost composed of the fan
noise. The signal to noise ratios (SNR) are approximately 14 dB, 11 dB for
_Audio-only_ dataset with 1.1 m and 2.5 m robot-to-source distance,
respectively, and 2 dB for _audio-visual_ dataset 222Note that the loudspeaker
volume is different for two datasets.. As mentioned in Section III-C, the fan
noise PSD ${\phi}_{vv}(p,k)$ and ${\varphi}_{wv}(p,k)$ are precomputed.

The training dataset $\\{\mathbf{c}_{i},\mathbf{o}_{i}\\}_{i=1}^{I}$ for
_Audio-only_ experiments is generated by the anechoic head-related impulse
responses (HRIR) of 1002 directions uniformly distributed in the same range as
the test dataset. The training dataset for _Audio-visual_ experiments is
generated by the HRIR of 378 directions uniformly distributed in the camera
field-of-view. The anechoic HRIR is obtained by truncating the room impulse
response before the first reflection. White Gaussian noise (WGN) signals are
emitted from each direction, and the cross-correlation between the microphone
signal and source WGN signal gives the room impulse response of each
direction.

###  V-B Parameter Setup

The sampling rate of the microphone signals is 16 kHz. The window length of
STFT is 16 ms (256 samples) with 8 ms overlap (128 samples). Only the
frequency band from 300 Hz to 4 kHz is taken into account for speech source
localization, i.e., the corresponding frequency bins are from 5 to 63, so the
number of frequencies is $K$=59. The number of frames $D$ for PSD estimation
is set to 25 (0.2 s). The power threshold $\gamma$ is set to 1.8. We set the
length of CTF $Q_{k}$ to be equal for all the frequency bins for simplicity,
and denote it as $Q$, which is set to $0.25T_{60}$.

###  V-C Method Comparison

The crucial point of binaural localization is to extract the reliable binaural
cues from the noisy and reverberant sensor signals. Two state-of-the-art
binaural feature estimation methods with good capability to reduce noise or
reverberations are tested for comparison.

  * •

A variation of the unbiased RTF estimator proposed in [14], in which the MTF
approximation is adopted. The noise PSD is recursively estimated in the
original work, while is more accurately precomputed using the noise-only
signal in this work. We refer to this method as RTF-MTF.

  * •

The coherence test (CT) method in [22]. The coherence test is used for
searching the rank-1 time-frequency bins, which are supposed to be dominated
by one active source. In this work, it is adopted for single speaker
localization, in which one active source denotes the direct-path source
signal. The TF bins that involve considerable reflections have low coherence.
We first detect the maximum coherence over all the frames at each frequency
bin, and then set the coherence test threshold for each frequency bin to 0.9
times its maximum coherence. In our experiments, this threshold achieves the
best performance. The covariance matrix is estimated by taking a 120 ms (15
adjacent frames) averaging. The auto and cross PSD of all the frames that have
a coherence larger than the threshold are applied the spectral subtraction
with the same principle in (III-C), and then are averaged over frames for
acoustic feature extraction. We refer to this method as RTF-CT.

  * •

In addition, a conventional beamforming SSL method: the steered-response power
(SRP) utilizing the phase transform (PHAT) [33, 34] is also tested. The source
directions in the training set of the proposed method are taken as the
steering directions, and their HRIRs are taken as the steering vector.

###  V-D Localization Results with the Audio-Only Dataset

Our experiments on _Audio-only_ dataset show that, in the elevation range
$[-15^{\circ}\,25^{\circ}]$, the elevation localization results are completely
unreliable for all the three methods. This can be easily explained by the fact
that the angle between the microphone plane and the horizontal plane is small,
hence the microphone array has a low resolution for the elevation direction.
Therefore, in Fig. 4, we only present the azimuth localization results.

(a)

(b)

Figure 4: Localization results for _Audio-only_ dataset. (a) 1.1 m robot-to-
source distance. (b) 2.5 m robot-to-source distance. The elevations of
multiple source directions corresponding to each azimuth uniformly distribute
from -15∘ to 25∘.

From Fig. 4-(a), we observe that both the proposed method and the RTF-MTF and
RTF-CT methods work well in the azimuth range $[-50^{\circ},\ 50^{\circ}]$.
The proposed method achieves slightly better results in this range. The
performance drops drastically for the source directions out of this range.
This indicates that the NAO’s microphone array has a better localization
capability for the azimuth range $[-50^{\circ},\ 50^{\circ}]$. From the
results for the azimuth range $[-120^{\circ},\ -50^{\circ}]$ and
$[50^{\circ},\ 120^{\circ}]$, it can be seen that RTF-MTF has the largest
localization error and many localization outliers caused by the
reverberations. By selecting frames that involve less reverberations, RTF-CT
performs better than RTF-MTF, evidently, which can be observed from the fact
that RTF-CT has less outliers than RTF-MTF. However, it is difficult to
automatically set a coherence test threshold that could perfectly select the
desired frames. Many frames that have a coherence larger than the threshold
include reflections. Therefore, RTF-CT also has a relatively large
localization error and some localization outliers. There are also many
outliers for SRP-PHAT, which indicates that the steered response power is
influenced by the reverberation. The proposed method achieves the best
performance by properly extracting the direct-path RTF.

Fig. 4-(b) shows the localization results for the data with 2.5 m robot-to-
source distance. Compared to the robot-to-source distance of 1.1 m, both ITDG
and DRR are smaller. Consequently, the performance degrades for both the
proposed method and the two state-of-the-art methods compared to Fig. 4-(a).
The reasons for this degradation are the followings: for both RTF-MTF and RTF-
CT the reflections are large relative to the direct-path impulse response,
which makes the feature estimated from the reverberated signals more different
than the feature corresponding to the direct-path propagation. In addition,
concerning RTF-CT, the early reflection is closer to the direct-path impulse
response, which makes less reverberation-free TF bins to be available. SRP-
PHAT also has more outliers than the case in Fig. 4-(a) due to the lower DRR.
For the proposed method, (i) the early reflections in the impulse response
segment $a(n)|_{n=0}^{N}$ increase and (ii) in vector $\mathbf{g}_{k}$, the
DP-RTF $\frac{b_{0,k}}{a_{0,k}}$ plays a more unimportant role relative to the
other elements with the decreasing of DRR, which makes the DP-RTF estimation
error larger. We can see that the proposed method still achieves the best
performance, and most of its localization results are reliable.

###  V-E Localization Results with the Audio-Visual Dataset

The source directions of _audio-visual_ dataset distribute in the camera
field-of-view, which is a small range in front of NAO’s head (azimuth range
$[-30.5^{\circ},\ 30.5^{\circ}]$). As shown in Fig. 4, good azimuth
localization results are obtained in this range. Table I shows the
localization error for both the azimuth (Azi.) and elevation (Ele.)
directions. The localization error is computed by averaging all the absolute
errors between the localized directions and their corresponding ground truth
(in degrees).

Figure 5: Overview of the proposed distributed architecture that allows fast
development of interactive applications using the humanoid robot NAO [35].

It can be seen that the elevation errors are always much bigger than the
azimuth errors, due to the low elevation resolution of the microphone array
that we already mentioned. In the cafeteria, the reverberation time $T_{60}$
is 0.24 s, generally speaking, which is a low reverberation time. The RTF-MTF
and RTF-CT methods yields performance comparable with the proposed method in
the cafeteria environment. The reason is: the MTF approximation is relatively
proper for this case, while the proposed method has a higher model complexity
which needs more reliable data. In the office and laboratory, the
reverberation times are larger, so the MTF approximation is not accurate
anymore. As a result, Table I shows that the proposed method achieves
evidently better performance than the two other methods in the office and
laboratory environments. The performance of RTF-MTF is even better than RTF-
CT, the reason is probably that the coherence test doesn’t work well under low
SNR conditions (the SNR is about 2 dB). In the meeting room, the reverberation
time is high (1.04 s). SRP-PHAT achieves the worst performance due to the
intense noise, especially the noise is spatially correlated. The proposed
method still evidently performs better than the other methods. These further
validates that the proposed method is more efficient in reverberant
environments.

| Cafeteria | Office | Laboratory | Meeting room
---|---|---|---|---
Methods | Azi. | Ele. | Azi. | Ele. | Azi. | Ele. | Azi. | Ele.
RTF-MTF | 0.45 | 1.57 | 0.62 | 2.14 | 1.44 | 2.31 | 1.87 | 3.66
RTF-CT | 0.44 | 1.50 | 0.64 | 2.25 | 1.61 | 2.36 | 1.77 | 3.44
SRP-PHAT | 0.77 | 1.95 | 1.03 | 2.80 | 1.41 | 3.33 | 2.04 | 3.52
Proposed | 0.47 | 1.47 | 0.55 | 1.87 | 0.82 | 1.84 | 0.95 | 2.12
TABLE I: Localization error (degrees) for the _audio-visual_ dataset. The best
results are shown in bold.

###  V-F Software Architecture

Ideally, one would like to implement the SSL method just presented using the
embedded computing resources available with a robot such as the NAO companion
humanoid. However, NAO like any other commercially available robot, has two
limitations. Firstly, the on-board computing resources are restricted which
implies that it is difficult to implement sophisticated audio signal
processing and analysis algorithms needed by SSL in particular and by robot
audition in general. Secondly, robot programming implies the development of
embedded software modules and libraries, which is a difficult task in its own
right necessitating specialized knowledge.

We have developed a distributed software architecture that attempts to
overcome these two limitations and which allows fast experimental validation
of proof-of-concept demonstrators [35]. Broadly speaking, NAO’s on-board
computing resources are networked with external (or remote) computing
resources. The latter is a computer platform (laptop or desktop) with its
CPU’s, GPU’s, memory, operating system, libraries, software packages, internet
access, etc. This configuration enables easy and fast development in Matlab,
C, C++, Python, etc. Moreover, it allows the user to combine on-board
libraries (motion control, face detection, etc.) with external toolboxes, such
as Matlab’s signal processing toolbox.

An overview of the proposed software architecture is shown on Fig. 5. Data
coming from NAO (motor positions, images, microphone signals, or data produced
by on-board computing modules) are fed into the external computer. Conversely,
the latter can control the robot. Currently we developed three internal-to-
remote interfaces: vision, audio, and locomotion. The role of these interfaces
is twofold: (i) to feed the data into a memory space that is subsequently
shared with existing software modules or with modules under development and
(ii) to send back to the robot commands generated by the external software
modules. Although these modules may be developed in a variety of programming
languages, special emphasis was put to allow integration with the Matlab
programming environment.

The proposed SSL method is implemented in Matlab, which offers the possibility
to use Matlab’s signal processing toolbox, e.g., the STFT. The Matlab computer
vision toolbox is used for image processing. The on-board robot controller is
invoked to rotate the robot head in the direction of the detected sound
source.

##  VI Conclusions

We have proposed a direct-path RTF estimator for SSL, and tested it on NAO
robot. Instead of the MTF approximation, the method takes the CTF
approximation, which is more precise when the impulse response is too long.
Compared with the conventional RTF, the ratio between two direct-path ATFs is
more reliable for SSL. Because the trainning dataset is generated using the
anechoic HRIR, the SSL module can operate for various room configurations,
which is important for robot audition. Experiments have shown that the
proposed method performs well for azimuth localization under difficult
acoustic conditions, however poorly for elevation localization because of the
microphone geometry of NAO robot head version 5. Thence, for the next version
of NAO, a more reasonable microphone topology is expected.

## References

  * [1] S. Argentieri and P. Danes, “Broadband variations of the MUSIC high-resolution method for sound source localization in robotics,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2009–2014, IEEE, 2007.
  * [2] K. Nakamura, K. Nakadai, and G. Ince, “Real-time super-resolution sound source localization for robots,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 694–699, IEEE, 2012.
  * [3] J.-M. Valin, F. Michaud, B. Hadjou, and J. Rouat, “Localization of simultaneous moving sound sources for mobile robot using a frequency-domain steered beamformer approach,” in IEEE International Conference on Robotics and Automation, vol. 1, pp. 1033–1038, IEEE, 2004.
  * [4] J.-M. Valin, F. Michaud, and J. Rouat, “Robust localization and tracking of simultaneous moving sound sources using beamforming and particle filtering,” Robotics and Autonomous Systems, vol. 55, no. 3, pp. 216–228, 2007.
  * [5] Y. Sasaki, N. Hatao, K. Yoshii, and S. Kagami, “Nested iGMM recognition and multiple hypothesis tracking of moving sound sources for mobile robot audition,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 3930–3936, IEEE, 2013.
  * [6] R. Gomez, K. Nakamura, T. Mizumoto, and K. Nakadai, “Temporal smearing compensation in reverberant environment for speech-based human-robot interaction,” in Robotics and Automation (ICRA), 2015 IEEE International Conference on, pp. 3347–3353, IEEE, 2015.
  * [7] A. Badali, J.-M. Valin, F. Michaud, and P. Aarabi, “Evaluating real-time audio localization algorithms for artificial audition in robotics,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 2033–2038, IEEE, 2009.
  * [8] X. Alameda-Pineda and R. Horaud, “A geometric approach to sound source localization from time-delay estimates,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 22, no. 6, pp. 1082–1095, 2014.
  * [9] J. Hornstein, M. Lopes, J. S. Victor, and F. Lacerda, “Sound localization for humanoid robots-building audio-motor maps based on the HRTF,” in IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1170–1176, IEEE, 2006.
  * [10] M. Raspaud, H. Viste, and G. Evangelista, “Binaural source localization by joint estimation of ILD and ITD,” Audio, Speech, and Language Processing, IEEE Transactions on, vol. 18, no. 1, pp. 68–77, 2010.
  * [11] A. Deleforge, F. Forbes, and R. Horaud, “Acoustic space learning for sound-source separation and localization on binaural manifolds,” International Journal of Neural Systems, vol. 25, no. 1, 2015.
  * [12] J. Blauert, Spatial hearing: the psychophysics of human sound localization.  MIT press, 1997.
  * [13] S. Gannot, D. Burshtein, and E. Weinstein, “Signal enhancement using beamforming and nonstationarity with applications to speech,” Signal Processing, IEEE Transactions on, vol. 49, no. 8, pp. 1614–1626, 2001.
  * [14] I. Cohen, “Relative transfer function identification using speech signals,” IEEE Transactions on Speech and Audio Processing, vol. 12, no. 5, pp. 451–459, 2004.
  * [15] X. Li, L. Girin, R. Horaud, and S. Gannot, “Estimation of relative transfer function in the presence of stationary noise based on segmental power spectral density matrix subtraction,” in 40th IEEE International Conference on Acoustics, Speech and Signal Processing, 2015.
  * [16] Y. Avargel and I. Cohen, “On multiplicative transfer function approximation in the short-time Fourier transform domain,” IEEE Signal Processing Letters, vol. 14, no. 5, pp. 337–340, 2007.
  * [17] R. Y. Litovsky, H. S. Colburn, W. A. Yost, and S. J. Guzman, “The precedence effect,” The Journal of the Acoustical Society of America, vol. 106, no. 4, pp. 1633–1654, 1999.
  * [18] D. Bechler and K. Kroschel, “Reliability criteria evaluation for TDOA estimates in a variety of real environments,” in IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 4, pp. 981–985, IEEE, 2005.
  * [19] C. Hummersone, R. Mason, and T. Brookes, “A comparison of computational precedence models for source separation in reverberant environments,” Journal of the Audio Engineering Society, vol. 61, no. 7/8, pp. 508–520, 2013\.
  * [20] T. May, S. Van De Par, and A. Kohlrausch, “A probabilistic model for robust localization based on a binaural auditory front-end,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 19, no. 1, pp. 1–13, 2011.
  * [21] J. Woodruff and D. Wang, “Binaural localization of multiple sources in reverberant and noisy environments,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 5, pp. 1503–1512, 2012.
  * [22] S. Mohan, M. E. Lockwood, M. L. Kramer, and D. L. Jones, “Localization of multiple acoustic sources with small arrays using a coherence test,” The Journal of the Acoustical Society of America, vol. 123, no. 4, pp. 2136–2147, 2008.
  * [23] O. Nadiri and B. Rafaely, “Localization of multiple speakers under high reverberation using a spherical microphone array and the direct-path dominance test,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 22, no. 10, pp. 1494–1505, 2014.
  * [24] Y. Avargel and I. Cohen, “System identification in the short-time Fourier transform domain with crossband filtering,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 4, pp. 1305–1319, 2007.
  * [25] R. Talmon, I. Cohen, and S. Gannot, “Relative transfer function identification using convolutive transfer function approximation,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 17, no. 4, pp. 546–555, 2009\.
  * [26] J. Benesty, “Adaptive eigenvalue decomposition algorithm for passive acoustic source localization,” The Journal of the Acoustical Society of America, vol. 107, no. 1, pp. 384–391, 2000.
  * [27] A. Deleforge, R. Horaud, Y. Y. Schechner, and L. Girin, “Co-localization of audio sources in images using binaural features and locally-linear regression,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 23, no. 4, pp. 718–731, 2015.
  * [28] I. Cohen and B. Berdugo, “Speech enhancement for non-stationary noise environments,” Signal processing, vol. 81, no. 11, pp. 2403–2418, 2001\.
  * [29] S. Araki, H. Sawada, R. Mukai, and S. Makino, “Underdetermined blind sparse source separation for arbitrarily arranged multiple sensors,” Signal Processing, vol. 87, no. 8, pp. 1833–1847, 2007.
  * [30] X. Li, R. Horaud, L. Girin, and S. Gannot, “Local relative transfer function for sound source localization,” in The European Signal Processing Conference, 2015.
  * [31] H. W. Loellmann, H. Barfuss, A. Deleforge, S. Meier, and W. Kellermann, “Challenges in acoustic signal enhancement for human-robot communication,” in Proceedings of Speech Communication, pp. 1–4, VDE, 2014.
  * [32] J. S. Garofolo et al., “Getting started with the DARPA TIMIT CD-ROM: An acoustic phonetic continuous speech database,” National Institute of Standards and Technology (NIST), Gaithersburgh, MD, vol. 107, 1988.
  * [33] J. H. DiBiase, H. F. Silverman, and M. S. Brandstein, “Robust localization in reverberant rooms,” in Microphone Arrays, pp. 157–180, Springer, 2001\.
  * [34] H. Do, H. F. Silverman, and Y. Yu, “A real-time SRP-PHAT source location implementation using stochastic region contraction (SRC) on a large-aperture microphone array,” in Acoustics, Speech and Signal Processing, 2007. ICASSP 2007. IEEE International Conference on, vol. 1, pp. I–121, IEEE, 2007.
  * [35] F. Badeig, Q. Pelorson, S. Arias, V. Drouard, I. D. Gebru, X. Li, G. Evangelidis, and R. Horaud, “A distributed architecture for interacting with NAO,” in ACM International Conference on Multimodal Interaction, (Seattle, WA), November 2015.