# Few-Shot Visual Grounding
for Natural Human-Robot Interaction

Giorgos Tziafas  Department of Artificial Intelligence
University of Groningen
<EMAIL_ADDRESS>Hamidreza Kasaei  Department of Artificial
Intelligence
University of Groningen
<EMAIL_ADDRESS>
###### Abstract

Natural Human-Robot Interaction (HRI) is one of the key components for service
robots to be able to work in human-centric environments. In such dynamic
environments, the robot needs to understand the intention of the user to
accomplish a task successfully. Towards addressing this point, we propose a
software architecture that segments a target object from a crowded scene,
indicated verbally by a human user. At the core of our system, we employ a
multi-modal deep neural network for visual grounding. Unlike most grounding
methods that tackle the challenge using pre-trained object detectors via a
two-stepped process, we develop a single stage zero-shot model that is able to
provide predictions in unseen data. We evaluate the performance of the
proposed model on real RGB-D data collected from public scene datasets.
Experimental results showed that the proposed model performs well in terms of
accuracy and speed, while showcasing robustness to variation in the natural
language input.

##  I Introduction

Humans have the cognitive capacity to process multi-modal data (e.g. vision,
language) and make cross-references between parts of the two modalities in
real-time effortlessly. They are also very capable of identifying such cross-
references in degenerate cases where one modality suffers from noise. However,
this is not the case in robotics, where most commonly the visual perception
and HRI modules are treated separately. In this regime, when verbal input is
given to the robot (e.g. in the context of grasping: “Grasp the mug”) the
referring phrases that correspond to objects to be segmented (aka grounded,
e.g. “Mug”) must be predefined explicitly and hard-coded in the agents
behaviour. As a result, the agent is unable to comprehend variants of the
predefined object category from the verbal input as it is often the case in
real-world scenarios, where objects might be referred by their visual
attributes or spatial relation to another object (e.g. “Grasp the red mug” or
“Grasp the mug that is next to the laptop” in the case of multiple mug objects
within view). A more natural HRI setting would require a bridging between the
two modules, allowing the model to interpret ambiguous verbal references and
generalize over unseen object categories. This paper proposes such a bridging
that tackles the problem by employing an end-to-end multi-modal deep learning
model, able to make predictions for image-phrase pairs that were never seen in
the training data.

The task at hand is described in literature as single-query visual grounding
or referring expression comprehension. In this task, given an input image and
a natural language phrase referring to a specific entity, the model has to
localize the entity inside the frame (see Fig. 1). Several research has been
done to address this task, both solely and as a proxy task for visual
question-answering (VQA) and visual reasoning downstream applications. Section
II provides a brief overview of such research directions. In this work, we aim
to design a grounding agent that can be utilized in a practical HRI scenario,
such as grasping household objects based on verbal commands given by a human
supervisor. To be efficient for this application, our agent needs to be able
to infer groundings at high speed. Also, when a broad variation of object
categories, as well as robustness to the phrasing of the referring expressions
describing them, is desired, the system should be able to provide predictions
for object - phrase instances that it has never encountered during training.
These considerations guide our choice of deep learning model and the
architecture of the implemented system, both of which are described in Section
III.

|  |
---|---|---
barbecue | ice cream | beer
|  |
soda can | bowl | bottle

Figure 1: Examples of grounding predictions made by our model for real (top)
and simulated (bottom) scenes along with the corresponding input phrase
queries.

We explore the robustness of the proposed approach to variations of the
natural language input by probing it in test time with extensive captions of
multiple levels of reference, namely: a) multiple instances of the same entity
(plural) or b) a specific instance of an entity appeared multiple times in the
scene by referring to its visual attribute (color) or some other property
(e.g. gender in the case of humans). Additionally, as most benchmarks for
visual grounding evaluate on domain-agnostic datasets, in this work we opt to
explore the grounding models performance on an HRI-oriented data domain, by
evaluating its zero/few-shot performance in two tiny-scale datasets of image-
query pairs respecting the assumed HRI scenario mentioned above. The image
samples are collected from sub-sets of two RGB-D datasets, namely: RGBD-Scenes
[26], used most popularly for 3D vision learning and the Objects Cluttered
Indoors Dataset (OCID) [1], used for manipulation planning. Details about the
evaluation data and some results are presented in section IV. Conclusion and
some interesting potential future research directions are given in section V.

##  II Related work

Visual grounding architectures typically use object detectors such as Faster-
RCNN [22] as a pre-processing step to extract bounding box proposals and
corresponding object features, thus limiting the categories that the model is
able to learn to the ones pre-specified by the pre-trained detector [23].
Refined versions use cross-modal self-attention mechanisms to capture long
range dependencies in the two modalities [29]. More recent approaches extract
scene graphs from images-text and tackle grounding as a structure prediction
task where visual and textual Graph Neural Networks (GNNs) are used to
contextualize each modalitys representations and a graph similarity metric is
introduced to prune the two graphs appropriately [13].

Another family of state-of-the-art models is that of pre-trained
representation learners, such as VisualBERT [8] and ViLBERT [14]. Such models
are very parameter-heavy, trained on massive-scale datasets with self-
supervised objectives, such as language modeling, masked region modeling,
word-region alignment and image-text matching. They often also apply a multi-
task setting [15], in which supervision from multiple tasks-domains is
injected during self-supervised pre-training. Single-task performance is then
benefited from attempting to solve all tasks at once. On the downside, the
enormous scale of these models grant them potentially inefficient for real-
time application.

Several robotics-grounded works attempt to explicitly model natural language
input for HRI applications, either as a purely textual or as an end-to-end
multi-modal system. In some works, the system maps input natural language
instructions into action sequences for task handling [16, 27]. Special care is
given so that the model handles variations of natural language input and is
able to generalize over unseen environment and task settings. Paul et. al.
[17] implemented a probabilistic model that parses natural language input into
object/region proposals as well as motion constraints by incorporating notions
of abstract spatial concepts. Shridhar et. al. [25] presented an HRI-applied
multi-modal system for visual grounding, where the model was able to handle
variations of natural language input (for situations of multiple object
instances) by interacting with the user through an ambiguity-resolution
question, generated by an auto-regressive RNN which is trained jointly with
the grounding network. One drawback of their work is that they employed a two-
stage approach for providing groundings, intensifying computational
requirements in inference time. In this work, we attempt to address this
limitation by employing a one-stage zero-shot model.

##  III System Architecture

In this section we present the visual grounding architecture that is developed
as well as the technical details of the overall architecture of the system.

###  III-A Learning Model

The deep learning model that is chosen as a baseline for our implementation is
the Zero-Shot Grounding (ZSG) network [23]. Similar to single-shot
architectures employed in object localization, such as [12], the proposed
model generates bounding box proposals that refer to the input query based
entirely on the size of the input image. As a result, an end-to-end trainable
image encoder for capturing image representations can replace pre-trained
object descriptors. For each image-query input pair, the model generates a set
of bounding box proposals $B=\\{b_{1},b_{2},...,b_{N}\\}$ and outputs the best
candidate box $b_{i}$, as well as four regression parameters
$\\{x_{1},y_{1},x_{2},y_{2}\\}$ that correspond to the updated top-left and
bottom-right box coordinates in the input image frame. Since this is an end-
to-end architecture, the visual features extracted by this model are
independent of the trained object intra-class variance. Therefore, the
proposed method departs from the limitations posed by other pre-trained visual
grounding systems, and is suitable for real-time applications due to its
single stage design and computational efficiency, especially during inference.

The model consists of five main components, including: (1) a language module
that encodes the query phrase into a continuous vector space, (2) a visual
module that extracts multiple image feature maps, (3) an anchor generator for
proposing multiple scale bounding boxes, (4) a multi-modal fusion scheme for
injecting all features into a single representation and a linear layer that
predicts the most likely box proposal, (5) as well as the array of regression
parameters for its fixed coordinates. For coherence purposes, we provide a
brief overview of these modules in the following sub-sections [23].

####  III-A1 Language module

This module consists of an embedding layer followed by a recurrent neural
encoder for encoding the input query phrase. The embedding layer is
responsible for mapping each word $W_{i}$ in our vocabulary to a dense vector
$\vec{w_{i}}\in\mathbb{R}^{d_{w}}$. The encoder is a uni-layered bi-
directional LSTM architecture [24] that processes sequentially the entire
input word vector sequence $\\{\vec{w_{i}}\\},i=1,...,T,$ in both directions
(start to end and vice-versa) and at each step outputs a hidden state vector
$\vec{h_{i}}\in\mathbb{R}^{2d_{w}}$ that is informed by the context of the
entire phrase in both directions at this point. The encoding that we use for
representing the entire phrase is the hidden state vector $\vec{h_{T}}$
produced in the last time step.

Alternatively, we implement phrase contextualization using a single
Transformer Encoder layer [28] and aggregate the hidden representations
$\mathbf{H}\in\mathbb{R}^{T\times d_{w}}$ for the entire sequence with average
pooling.

####  III-A2 Visual module

This module consists of a deep Convolutional Neural Network (CNN) that learns
how to represent the input 2D image into a dense feature map. This is a
standard CNN architecture used for object recognition without the linear
layers that are used for cross-entropy classification. In our version, $K$
feature maps $\mathbf{V_{j}}\in\mathbb{R}^{d_{v}\times d_{v}},j=1,...,K,$ are
extracted at different resolutions. The model used for encoding is a ResNet-50
[4], augmented into the Feature Pyramid Network (FPN) architecture [10] for
extracting multi-scale hierarchical feature maps. The feature maps are
normalised across the channel dimension.

####  III-A3 Anchor generation

The first step is to form a grid. For each cell of the grid, anchors of
different shapes are proposed. The convention of the grid is (-1,-1) to (1,1),
similar to the one defined for bounding boxes. Anchors are defined in terms of
ratios and scales. The ratio is the ratio of the height of an anchor box to
its width. The scale is used to calculate the unit of the height and width of
each anchor box. Given a scale ($s$) and a ratio ($r$), the aspect is
[$s\sqrt{r}$, $\frac{s}{\sqrt{r}}$]. These aspects are then resized according
to the size of each cell of the grid. The center of each anchor box is
considered the centre of the grid cell.

####  III-A4 Multi-modal fusion

The language feature vector extracted by the neural encoder is expanded to fit
the dimensions of the $K$ extracted visual feature maps and it is concatenated
along the channel dimension of each feature map, so that
$\mathbf{H_{T}}\in\mathbb{R}^{d_{v}\times
d_{w}},\;\mathbf{H_{T}}=\begin{pmatrix}\vec{h_{t}}&\vec{h_{t}}&...&\vec{h_{t}}\end{pmatrix}^{T}$.
The generated anchor box centres are also appended at each cell of the feature
maps. The resulting multi-modal feature representations
$\mathbf{M_{j}}\in\mathbb{R}^{d_{m}\times d_{m}}$ are then given for
$j=1,...,K$ by:

$\mathbf{M_{j}}(x,y)=\mathbf{V_{j}}(x,y)\;;\;\mathbf{H_{T}}(x,y)\;;\;\frac{c_{x}}{W}\;;\;\frac{c_{y}}{H}$

where ; operator denotes the concatenation, $c_{x},$ and $c_{y}$ represent the
center locations of the normalised feature maps at each $(x,y)$ location. The
initial size of the input image is represented by $W$, and $H$. The scaling
operation is performed in order to aid location-based grounding, when input
query phrases contain location information, providing functionality for making
spatial references between recognised objects in the input image frame.

####  III-A5 Anchor matching

Each generated anchor proposal of different size is matched to every cell of
the produced multi-modal feature maps. For each box, a linear layer maps the
multi-modal features into a 5-dimensional vector containing the prediction
score (confidence) and the regression box parameters that update the
coordinates to bound the referenced object tightly.

###  III-B Online Object Segmentation

Figure 2: A schematic of the implemented ROS system. The inference pipeline is
implemented through an action client that given an input RGB frame and a
caption sends a request to our network and outputs the coordinates of the most
probable bounding box. A 3D Extraction node extracts a PointCloud by combining
the RGB box with the corresponding depth box from the registered depth frame.

In this section, we describe the design of the implemented software
architecture, shown in Fig. 2. Our system receives a stream of RGB-D topics
produced real-time from a depth sensor along with a natural language caption
given by a human user and produces an RGB image with the predicted bounding
box drawn into the frame, as well as a PointCloud containing 3D information of
the segmented object. The images on the RGB-D buffer are refreshed with each
new pair and only when a caption is given the last entry is grabbed and used
as input to the network.

For real-time visual grounding, we have implemented an action server that
functions as the communication side between the sensor data and the
implemented ZSG network. At its spawn, the server loads all the necessary
utilities (word embeddings etc.) and loads a pre-trained instance of the ZSG
network. When a caption is buffered, a client request for a grounding
inference is published. The action server appropriately pre-processes the
input raw RGB frame in order to enhance it’s contrast, by applying Contrast
Limited Adaptive Histogram Equalisation (CLAHE) [30] in the YUV colourspace.
The query phrase is embedded and the resulting image-query pair is passed as
input to the network for a single forward pass. After inference, some post-
processing steps for drawing the predicted box and segmenting the foreground
for 3D extraction are applied and the resulting frames are published. The
server responses are executed in $>30$Hz, even without utilizing GPU-enabled
acceleration like during training (tested in an Intel Xeon E5-2680 v3 @
2.50GHz CPU).

##  IV Experimental Results

###  IV-A Training

We trained our model using two benchmark visual grounding datasets. The first
training dataset was Flickr30k Entities [20], comprised of $30\,000$ annotated
images associated with five sentences, each having multiple entity queries
referring to the image with an average of $3.6$ queries per sentence. As a
result, each sentence has been split to provide more grounding samples for the
respective image. The annotations are bounding boxes containing the category
labels of the grounded entity. The second dataset was the RefClef strain of
the ReferIt dataset [6], itself being a subset of Imageclef [3], enumerating
approximately $20\,000$ images with a total of $85k$ single-query caption
sentences. For both datasets, we used the same training-validation-test splits
as in [5].

We considered two options for embedding input language semantics into word
vectors for English, namely: (i) pre-trained GloVe embeddings [18] and (ii)
the last hidden layers representations of a pre-trained BERT-BASE [9],
followed by a trainable linear layer down-scaling the vectors to desired
dimensionality. For extracting visual features we utilized a RetinaNet [11]
network with a ResNet-50 [4] backbone. The image samples were resized to
$300\times 300$ and the word embeddings size as well as the hidden size of the
bi-LSTM contextualizing the input phrase was set to $300$, so that both
feature vectors could be appended to fit the dimensionality of the multi-modal
feature maps. The input query phrases were padded to a maximum length of $50$
words per phrase. Following [23], a total of $9$ candidate anchor proposals of
different sizes is generated.

For formulating the supervision signal as well as quantifying the evaluation
performance, we utilised the Intersection over Union (IoU) metric, calculated
as the total overlapping area between the proposed box and the ground truth
box. Following the original implementation, we used an IoU threshold of 0.5,
meaning that only proposals that fit the ground truth box’s area over 50% are
considered candidate:

$g_{b_{i}}\doteq 1\cdot[IoU(b_{i},gt)\geq 0.5]\;+\;0\cdot[IoU(b_{i},gt)<0.5],$
$G\doteq\\{b_{i}\;|\;g_{b_{i}}=1\\}$

where $B=\\{b_{i},\;i=1,...,9\\}$ represents the set of all proposals and $gt$
states the ground truth box. The set $G$ is the collection of all candidate
proposals $(g_{b_{1}}=1)$. The training loss is then calculated as the sum of
the focal loss $L_{F}$ described in [11] (applied for deciding the prediction
score of the i-th proposal $p_{b_{i}}$ for its binary classification as
foreground that possibly contains the query vs. background), with parameters
$\alpha=0.25,\;\gamma=2$ and the smooth-L1 loss $L_{S}$ [21] (used for
regressing the parameters $r_{b_{i}}$ of a tighter matching bounding box for
the most probable proposal):

$L=\frac{1}{\left|G\right|}\sum_{i=1}^{\left|B\right|}L_{F}(p_{b_{i}},g_{b_{i}})+\frac{1}{\left|G\right|}\sum_{i=1}^{\left|B\right|}g_{b_{i}}\cdot
L_{S}(r_{b_{i}},gt)$

The reported accuracies are measured as the total average of correctly
classified samples with an IoU $>0.5$. The end-to-end model was trained for a
total of $12$ hours for each dataset (around $7-8$ epochs), where the
validation loss had already saturated. We utilised the Adam optimizer [7] with
a learning rate of $10^{-4}$ and a weight decay of $10^{-4}$ for
regularisation. The model was trained on a NVidia Tesla v100 GPU node for
parallel processing of batches of $128$ image-query pairs. All code is written
in Python using the PyTorch deep learning framework. Training results compared
with several baselines are reported in Table I.

Method | Flickr30k | RefClef
---|---|---
QRC⋆ [2] | 60.21% | 44.10%
CITE⋆ [19] | 61.89% | 34.13%
QRG⋆ [2] | 60.1% | -
ZSG [23] | 63.39% |  58.64%
VilBERT [15] | 64.61% | -
ZSG (Glove-LSTM) | 62.73% | 53.44%
ZSG (Glove-TRM) | 61.74% | 50.41%
ZSG (BERT-LSTM) | 63.09% |  54.21%
ZSG (BERT-TRM) | 62.13% | 52.12%

TABLE I:  Best top-1 accuracies @IoU = $0.5$ of the implemented model variants
after 12 hours of training in both datasets. Results compared to original
implementation and other state-of-the-art methods. Methods with $\star$
further fine-tune their network on the entities of the Flickr30k dataset.

Even though the pre-trained BERT language module provides marginally better
embeddings for visual grounding, its massive scale sets a huge computational
bottleneck to the system. Therefore, we implement our agent based on the pre-
trained Glove model.

###  IV-B Evaluation

In order to evaluate the performance of the proposed system, we conducted
multiple experiments, aiming to test not only the predictive accuracy but also
the generalisation potential of the proposed model in variants of the standard
input query phrases. In this vein, we performed online interactive sessions
with our implemented agent, during which humans provide captions for arbitrary
images and evaluate the system’s predicted segmentation qualitatively.
Different depicted objects are queried in free linguistic fashion, also
including ambiguous query phrases (plural, referring to a visual cue etc.)
that challenge our system in regards to the natural HRI element that we strive
for. Figure 3 demonstrates some examples of such experiments and a live
recorded demo of a session is available online at https://youtu.be/kgQgaghf71o

---
(a) Correctly segmented objects
(b) Correctly segmented objects with low number of occurrences in training
data
(c) Incorrectly segmented objects
Figure 3: Several examples of correct and some of incorrect bounding box
predictions for image-query pairs. Queries are from left to right: (a) ice
cream, hat, guitar, rope, (b) microscope, necklace, barbecue, roulette, (c)
cup of coffee, watermelon, beer, bottle.

Real-time sensory data streamed from a depth sensor, often suffer from various
forms of noise compared to the digitally pre-processed images of the training
data. To investigate the performance of the proposed system in sensor data, we
collected two small-scale datasets of pre-recorded RGB-D images. The scenes
included in our samples respect the assumed HRI scenario, i.e., they always
comprise of a collection of objects in arbitrary spatial arrangements on top
of a planar surface in clear front view of the RGB-D sensor.

Dataset | #Img. | avg.QPI | #Samp. | Mult.Inst. (%)
---|---|---|---|---
RGBD-Scenes | 67 | 3.13 | 210 | 17.97%
OCID | 18 | 2.94 | 53 | 22.22%

TABLE II: Total number of image samples. average number of query phrases per
image, total number of samples (query-phrase pairs) and percentage of samples
that include multiple instances of the same object category.

The first dataset is a sub-set of the RGBD-Scenes dataset [26], which has a
total of $67$ RGB-D pairs depicting $22$ different typical household object
instances from $6$ different categories (bowl, cap, cereal box, coffee mug,
flashlight, soda can) in $8$ different scenes (including multiple desk
settings, kitchen, meeting room, small and large table). Bounding box
annotations come packed with the dataset. We appropriately sub-sampled the raw
training data to include scenes that only contain different object categories
or the same objects but viewed from a different angle, resulting in different
spatial relationships between them. The second dataset is a curated sub-set of
OCID [1], containing a total of $89$ different object categories in two scene
settings (table and floor) and two different camera placements (front and top
view). As this dataset is originally intended for learning motion planning of
a robotic actuator in scenes with highly cluttered objects, we filter the raw
data to include only samples with none to minimal cluttering. Statistics about
the collected RGB-D datasets are described in Table II.

We again qualitatively evaluated and reported accuracy scores representing the
zero-shot performance of our system in both domains. We performed preliminary
experiments for evaluating the domain adaptation potential of our system,
where we fine-tuned our model in one RGB-D-scene (e.g. desk) and evaluated in
another (e.g. kitchen). By this we aimed to simulate the fine-tuning step that
a potential user of our system can employ after gathering minimal samples from
the specific object catalogue / environment they wish their agent to operate.
Fine-tuning had been implemented identically to the training process with the
exception of using a batch size of $32$. The results are presented in Table
III and examples of real-time inferences are shown in Figure 4.

---
(a) Correctly segmented objects from RGBD-Scenes
(b) Correctly segmented objects with relative cluttered environments
(c) Correctly segmented objects in synthetic data simulated in Gazebo
Figure 4: Several examples of bounding box predictions for real-time RGB-D
sensory data from real/synthetic scenes. Queries are from left to right: (a)
cereal box, paper, bowl, hat, (b) envelope, tortellini, soda can, bowl, (c)
bowl, bottle, box, cup.

| OCID | RGB-D | RGB-D | RGB-D
---|---|---|---|---
Model | (full) | (full) | (kitchen) | (tables)
ZSG-Flickr30k | 32.13% | 34.19% | 32.50% | 34.13%
ZSG-RefClef | 31.37% | 33.50% | 36.35% | 34.13%
+desk | - | - | 49.49% | 50.15%
+desk+meeting | - | - | 55.90% | 58.20%

TABLE III: Evaluation results for both pre-trained models in HRI-specific
data, collected from sub-sets of the RGBD-scenes and OCID datasets. The $+$
rows denote few-shotting results of the ZSG-Flickr30k model in specific scenes
after fine-tuning in similar examples from different scenes

As expected, we observe a performance drop in the sensory input datasets. In
such data, images suffer from potentially high amount of noise (reflections,
bad illumination etc.) resulting in noisy visual representations. At the same
time, potential cluttering of objects in real-life scenarios also degrades the
quality of the proposal generation. These limitations grant our model
practically unreliable without any fine-tuning, as we observe by the poor
zero-shot performance (below $50\%$). However, the few-shotting experiments
suggest a significant performance boost even with minimal fine-tuning.
Moreover, we observe that due to its single-stage nature, our model is capable
of unravelling ambiguities of reference in the input phrase. Specifically,
there are non-sensory examples where the model responds to use of plural, use
of definitive pronouns, as well as visual cues, e.g. colour (see Fig. 5).

---
(a) Comprehension of plural
(b) Comprehension of pronouns
(c) Comprehension of visual cues (colour)
Figure 5: The system is capable of generalising over variants of the natural
language input (plural, pronouns, visual cues). Queries are from left to
right: (a) sandal(s), human(s) (b) his/the girls face, his/her shoulder (c)
green/black top, blonde/brunette girl.

##  V Conclusion

In this work, we employ a state-of-the-art multi-modal deep learning model and
develop an HRI agent for real-time visual grounding. The focus of this work is
not on improving the zero-shot model but rather exploiting its single-stage
nature, which allows for predictions in unseen data, in order to achieve
interaction with humans in dynamic environments (potentially unknown objects)
and in natural fashion (robustness to ambiguities of reference). The segmented
3D information of the object can serve as a perceptive utility assisting in
navigation, planning, and manipulation actions. We test our methodology by
gathering RGB-D data from two robotics-oriented datasets and perform several
rounds of experiments to evaluate the systems performance in both zero-shot
and few-shot scenarios. Experimental results showed that our approach works
well in the fine-tuning scenario, both in terms of accuracy and speed. In the
continuation of this work, we would like to investigate the possibility of
ameliorating the low quality of visual representations in sensory images by
applying Synthetic2Real Domain Adaptation, in effect generating synthetic
scenes similar to the setup of our HRI scenario and utilizing them as a fine-
tuning resource. Another interesting direction is the replacement of the
current scene-level visual module with an object-level one, as bounding box
extraction can be managed in an RGB-D sensor with the aid of depth data.

## References

  * [1] International Conference on Robotics and Automation, ICRA 2019, Montreal, QC, Canada, May 20-24, 2019. IEEE, 2019.
  * [2] K. Chen, R. Kovvuri, and R. Nevatia.  Query-guided regression network with context policy for phrase grounding.  CoRR, abs/1708.01676, 2017.
  * [3] H. J. Escalante, C. A. Hernández, J. A. Gonzalez, A. López-López, M. Montes, E. F. Morales, L. Enrique Sucar, L. Villaseñor, and M. Grubinger.  The segmented and annotated IAPR TC-12 benchmark.  Comput. Vis. Image Underst., 114(4):419–428, Apr. 2010.
  * [4] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [5] R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell.  Natural language object retrieval.  CoRR, abs/1511.04164, 2015.
  * [6] S. Kazemzadeh, V. Ordonez, M. Matten, and T. L. Berg.  Referit game: Referring to objects in photographs of natural scenes.  In EMNLP, 2014.
  * [7] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization, 2014.  cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
  * [8] L. H. Li, M. Yatskar, D. Yin, C. Hsieh, and K. Chang.  Visualbert: A simple and performant baseline for vision and language.  CoRR, abs/1908.03557, 2019.
  * [9] L. H. Li, M. Yatskar, D. Yin, C. Hsieh, and K. Chang.  Visualbert: A simple and performant baseline for vision and language.  CoRR, abs/1908.03557, 2019.
  * [10] T. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie.  Feature pyramid networks for object detection.  CoRR, abs/1612.03144, 2016.
  * [11] T. Lin, P. Goyal, R. Girshick, K. He, and P. DollÃ¡r.  Focal loss for dense object detection.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2999–3007, Oct 2017.
  * [12] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C. Fu, and A. C. Berg.  SSD: single shot multibox detector.  CoRR, abs/1512.02325, 2015.
  * [13] Y. Liu, B. Wan, X. Zhu, and X. He.  Learning cross-modal context graph for visual grounding.  CoRR, abs/1911.09042, 2019.
  * [14] J. Lu, D. Batra, D. Parikh, and S. Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  CoRR, abs/1908.02265, 2019.
  * [15] J. Lu, V. Goswami, M. Rohrbach, D. Parikh, and S. Lee.  12-in-1: Multi-task vision and language representation learning.  CoRR, abs/1912.02315, 2019.
  * [16] D. Misra, J. Sung, K. Lee, and A. Saxena.  Tell me dave: Context-sensitive grounding of natural language to manipulation instructions.  The International Journal of Robotics Research, 35:281 – 300, 2016\.
  * [17] R. Paul, J. Arkin, N. Roy, and T. M. Howard.  Efficient grounding of abstract spatial concepts for natural language interaction with robot manipulators.  In Robotics: Science and Systems, 2016.
  * [18] J. Pennington, R. Socher, and C. D. Manning.  Glove: Global vectors for word representation.  In In EMNLP, 2014.
  * [19] B. A. Plummer, P. Kordas, M. H. Kiapour, S. Zheng, R. Piramuthu, and S. Lazebnik.  Conditional image-text embedding networks.  CoRR, abs/1711.08389, 2017.
  * [20] B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  IJCV, 123(1):74–93, 2017.
  * [21] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 91–99. Curran Associates, Inc., 2015.
  * [22] S. Ren, K. He, R. B. Girshick, and J. Sun.  Faster R-CNN: towards real-time object detection with region proposal networks.  CoRR, abs/1506.01497, 2015.
  * [23] A. Sadhu, K. Chen, and R. Nevatia.  Zero-shot grounding of objects from natural language queries.  In Proceedings of the IEEE International Conference on Computer Vision, pages 4694–4703, 2019.
  * [24] M. Schuster and K. Paliwal.  Bidirectional recurrent neural networks.  Trans. Sig. Proc., 45(11):2673–2681, Nov. 1997.
  * [25] M. Shridhar and D. Hsu.  Interactive visual grounding of referring expressions for human-robot interaction.  CoRR, abs/1806.03831, 2018.
  * [26] S. Song, S. P. Lichtenberg, and J. Xiao.  Sun rgb-d: A rgb-d scene understanding benchmark suite.  In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 567–576, 2015.
  * [27] J. Thomason, A. Padmakumar, J. Sinapov, N. Walker, Y. Jiang, H. Yedidsion, J. W. Hart, P. Stone, and R. J. Mooney.  Improving grounded natural language understanding through human-robot dialog.  CoRR, abs/1903.00122, 2019.
  * [28] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin.  Attention is all you need.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc., 2017\.
  * [29] L. Ye, M. Rochan, Z. Liu, and Y. Wang.  Cross-modal self-attention network for referring image segmentation.  CoRR, abs/1904.04745, 2019.
  * [30] K. Zuiderveld.  Graphics gems iv.  chapter Contrast Limited Adaptive Histogram Equalization, pages 474–485. Academic Press Professional, Inc., San Diego, CA, USA, 1994.


# Audio scene monitoring using redundant un-localized microphone arrays

Peter Gerstoft, Yihan Hu, Chaitanya Patil, Ardel Alegre, Michael J. Bianco,
Yoav Freund, and François Grondin,  P. Gerstoft, Y. Hu, C. Patil, A. Alegre,
M.J. Bianco and Y. Freund are with University of California San Diego, La
Jolla, CA 92093-0238,USA, http://noiselab.ucsd.eduF. Grondin is with
Université de Sherbrooke, Sherbrooke, Québec, Canada

###### Abstract

We present a system for localizing sound sources in a room with several
microphone arrays. Unlike most existing approaches, the positions of the
arrays in space are assumed to be unknown. Each circular array performs
direction of arrival (DOA) estimation independently. The DOAs are then fed to
a fusion center where they are concatenated and used to perform the
localization based on two proposed methods, which require only few labeled
source locations for calibration. The first proposed method is based on
principal component analysis (PCA) of the observed DOA and does not require
any calibration. The array cluster can then perform localization on a manifold
defined by the PCA of concatenated DOAs over time. The second proposed method
performs localization using an affine transformation between the DOA vectors
and the room manifold. The PCA approach has fewer requirements on the training
sequence, but is less robust to missing DOAs from one of the arrays. The
approach is demonstrated with a set of five 8-microphone circular arrays,
placed at unknown fixed locations in an office. Both the PCA approach and the
direct approach can easily map out a rectangle based on a few calibration
points with similar accuracy as calibration points. The methods demonstrated
here provide a step towards monitoring activities in a smart home and require
little installation effort as the array locations are not needed.

###### Index Terms:

Smart homes, circular microphone arrays, sound localization, self-calibration.

##  I Introduction

Microphone arrays, in the form of smart speakers, have become an affordable
household item. As a result, these systems are ubiquitous, and there may be
many microphone arrays in a single room. Most audio array systems require
knowledge of the relative locations and orientations of the arrays. We instead
use a few source calibration points with known relative locations, which are
easier to implement. By using redundant arrays, we obtain higher accuracy and
are less concerned with array placement.

In this paper we describe localization approaches which use several microphone
arrays of unknown locations. The DOAs from each array are then collected
through WiFi to a central fusion center where they are concatenated to form a
global DOA vector. Based on the global DOA vector we perform localization in a
room with two proposed methods: a subspace method based on PCA and an affine
mapping-based approach.

We first consider localization using principal component analysis (PCA) to
obtain a low-dimensional mapping of the DOA vector observations over time and
space. This PCA approach does not require knowledge of calibration source
location, just an association between PCA coordinates and room coordinates.
The mapping requires the relative location of calibration points, say corner
of a table. While this approach does not provide physical dimensions, it can
give relations between locations in a room.

We also consider a localization approach based on affine mapping between the
DOA vector and room coordinates. This mapping is more robust to missing array
DOAs than the PCA approach, as it is based on a physical mapping to room
coordinates. To train the mapping it also requires relative location of
calibration points.

For our experiments, we have five arrays, placed at fixed unknown locations in
a reverberant office environment. The arrays are connected to a Raspberry Pi
3, which runs the Open embeddeD Audition System (ODAS) software, [1, 2] and
can track the directions of arrival (DOAs) for up to four sound sources
simultaneously. This open-source framework is appealing as it allows on-board
DOA estimation for each microphone array. We use this existing system to
observe the sound sources and compute their DOA relative to the array.

Such a setup could be useful for improving monitoring of sound events in a
smart home, using existing arrays [3]. Since the DOA processing is performed
on-board, this helps preserve privacy, as audio in this approach is not share
over the network. This could be used for fall monitoring [4] and daily events
classification [5]. This system could also be combined to sound scene
classification, e.g., [6, 7].

###  I-A Related work

Alternative methods would combine the whole array stream of data at the fusion
center by using maximum likelihood beamforming for multiple arrays [8, 9] and
localize the arrays in an ad hoc network [10]. In our approach, where the DOAs
are processed at the individual arrays, we benefit from transmitting just the
DOA stream. This requires less bandwidth than using the raw array output. In
terms of privacy this is a huge improvement as no signals are transmitted to
the fusion center. In a fusion center multiple DOAs can be used for
localization based on known array positions[11].

Resolving closely spaced acoustic sources when the noise power is varying in
space and time has been studied extensively. As part of ODAS, we rely on the
low-complexity Steered-Response Power Phase Transform (SRP-PHAT) to estimate
DOAs [12, 1]. Many other methods could be used such as adaptive beamformers
[13] or compressive DOA estimation [14, 15, 16, 17]. Sparse Bayesian learning
(SBL) [15, 18, 19, 20, 21] could also be used. Multiple Signal Classification
(MUSIC) [22] is also common for localization in noisy environments [23, 24].
DOA estimation could also rely on machine leaning [25, 26]. Since each
circular array has well-defined geometry classical array processing seems more
appropriate, and machine learning could be used in the combination of the
arrays. More features than just DOA could be beneficial for localization [27],
e.g., echoes [28]. As a low-complexity approach, just three microphones have
been used [29].

In the future, more advanced machine learning (ML)[30] could help with this
task as it already did in related localization problems. In particular, ML
approaches might better-address the inherent non-linearity in sound
localization. Neural network classifier [31, 32, 25, 33] or semi supervised
learning[34, 35, 36] would be of first interest.

Many indoor localization systems have been proposed [37]. Sound source
localization (SSL) also has widespread applications in human-robot interaction
[38], ocean acoustics [39], teleconferencing [40], drone localization [41],
and automatic speech recognition [42]. These have been designed for various
applications as indoor navigation, communication and health. In general, they
involve a first step where receiver geometric configuration such as distances
and angles are measured. In the second step, the target is located using the
measured data. Our proposed system replaces the first step with a
training/calibration step.

DOAs are the only features used here. Other systems also use travel time
difference of arrival between the arrays [43]. This requires a precise
relative clock for each array system, maybe requiring more hardware. The
actual system implemented here cost about 200 USD for one array. The proposed
system is demonstrated with active sources at well-known points. This serves
to better validate the approach.

Figure 1: The Matrix Creator array is equipped with 8 microphones (circled in
red), disposed on a 10 cm diameter circle. A unit vector in the positive
$z$-plane points in the direction of the DOA (in orange, denoted by
$\mathbf{d}$).

##  II Processing localization

We assume a room equipped with $M$ time-synchronized circular microphone
arrays, which provides the DOAs based on a local coordinate system. The DOA
for each array lies on a virtual unit half-sphere in the positive $z$-plane
(see Fig. 1), and is defined as ${\bf d}=[d_{x}\ d_{y}\ d_{z}]^{T}$, where
$d_{x}^{2}+d_{y}^{2}+d_{z}^{2}=1$ and $d_{z}\geq 0$. Our development and
processing use the output DOA vectors of the time-synchronized arrays as they
arrive at the fusion center.

From the M arrays we obtain one $3M$ dimensional DOA vector ${\bf d}=[{\bf
d}_{1}^{T}\dots\bf d_{M}^{T}]^{T}$ from one sound source. The array positions
are unknown. Thus, to perform source localization we first need to do training
based on previous $L$ observations of the ${\bf d}$ vector, collected into a
${3M\times L}$ matrix $\bf D$, called calibration DOAs. We propose two
methods:
Sec. II-A method relies on observing $\bf D$ and then performing principal
component analysis (PCA) on this matrix. Thus, we localize the source in a PCA
space. The location of the sources generating the $\bf D$ matrix can be
unknown.
Sec. II-B–II-D approach assumes an affine transformation between ${3M}$
dimensional ${\bf d}$-vector and the local $N=3$ dimensional room.

For the first approach we need all (or the same) arrays observing the sound
source of interest, but no calibration sources are needed. For the second
approach we need a set of calibration sources (more than 3).

###  II-A PCA

Performing PCA on the observed DOA vectors gives ${\bf D}={\bf
U}{\bf\Lambda}{\bf V}^{T}$, with ${\bf U},{\bf V}$ the left and right singular
vectors and ${\bf\Lambda}$ the singular values. Assuming the relation between
DOA and spatial location is approximately linear (3) with N (2 or 3) spatial
coordinates, the $J$ first (2 or 3) singular DOA vectors should be sufficient
to describe the location. Throughout the rest of the paper, we use $J=2$ first
components, as all sources are mainly in the 2-D horizontal plane.

Decomposing the $i$th DOA observation ${\bf d}_{i}$ with only the first $J=2$
singular vectors used, the reduced ${\bf U}_{J}=[{\bf u}_{1}\,{\bf u}_{2}]$.

${\bf d}_{i}\approx\sum_{j=1}^{J}a_{ij}{\bf u}_{j}={\bf U}_{J}{\bf
a}_{i},\qquad a_{ij}={\bf d}_{i}^{T}{\bf u}_{j}~{},$ (1)

where ${\bf a}_{i}=[a_{i1}\,a_{i2}]^{T}$ are the coefficients of the two
singular vectors that define the DOA vector ${\bf d}_{i}$. Since physically
there should only be 2–3 large components, finding the decomposition in PCA
space defines an unknown room position.

In the above processing, it is assumed that all arrays observe all the
sources. However, in a real system there will always be missing observations
due to malfunctioning arrays, nature of the room setup, source directivity or
weak sources. This can happen in either the calibration or the mapping part of
the experiments.

For mapping to PCA coordinates, omitting the missing array is easy, but the
sound source will then map to a different location in PCA space. Thus, for
sources evolving sequentially in time it is preferred to only use the arrays
that is continuously tracking the source.

###  II-B Affine mapping to room manifold

In general, the mapping between $N$-dimensional room observations
$\mathbf{r}\in\mathbb{R}^{N}$ ($N$ is 2 or 3) and $3M$-dimensional DOA vector
$\mathbf{d}$ is non-linear and unknown

${\bf r}={\bf f}({\bf d}).$ (2)

The non-linear variation of the DOA vector $\bf d$ vs spatial location for one
array is indicated in Fig. 2. We assume a linear mapping (affine
transformation) might be sufficient for this mapping of a room. This was
observed to work well for the DOA variation in the room used here, see Sec.
IV. The affine transformation is

${\bf r}={\bf r}_{0}+{\bf B}{\bf d}$ (3)

where ${\bf r}_{0}\in\mathbb{R}^{N}$ is the offset and ${\bf
B}\in\mathbb{R}^{N\times 3M}$ are the linear coefficients. Both ${\bf r}_{0}$
and ${\bf B}$ are determined in Sec. II-C by performing recording at $K$
locations ${\bf r}_{k}$.

Typically, a weak source is not observed on all arrays, giving a DOA
observation $\bf d$ with non-active elements, we then retain the active
elements in ${\bf d}_{a}$ produced by the list of active arrays $I_{a}$. From
the calibration DOA matrix $\bf D$ we only use the active arrays $I_{a}$. For
each DOA observation ${\bf d}$ and $I_{a}$, we then determine the ${\bf\sf
B}(I_{a})$ function with only the entries of the calibration DOA $\bf D$
corresponding to active arrays $I_{a}$ and use this to determine the $\bf B$
matrix for just $I_{a}$. This $\bf B$ matrix is then multiplied with the
active DOAs ${\bf d}_{a}$. The non-active elements are not used in the
mapping. This gives

${\bf r}={\bf r}_{0}+{\bf\sf B}(I_{a})\,{\bf d}_{a}~{}.$ (4)

For each case of active DOAs, the $\bf B$ matrix is determined as in Sec.
II-C.

###  II-C Linear mapping calibration

We perform a sound source recording at $K$ locations, for location ${\bf
r}_{k}$ we have $L_{k}$ observations ${\bf D}_{k}=[{\bf d}_{1}\dots\bf
d_{L_{k}}]$. For all $L=\sum_{k=1}^{K}L_{k}$ observations, we obtain the
${3M\times L}$ matrix of DOA observations ${\bf D}=[{\bf D}_{1}\,\dots\bf
D_{K}]$ generated from a source with corresponding location matrix

${\bf R}=[{\bf r}_{1}{\bf i}^{T}_{L_{1}}\dots\bf r_{K}{\bf
i}^{T}_{L_{K}}]\in\mathbb{R}^{N\times L}$ (5)

where ${\bf i}_{L_{j}}$ is a vector of length $L_{j}$ with all ones.

Defining the cost function (using the trance operator Tr and Frobenius norm
$\|\cdot\|_{f}$)

$\displaystyle\phi$ $\displaystyle=$ $\displaystyle\|{\bf R}-({\bf r}_{0}{\bf
i}^{T}_{L}+{\bf B}{\bf D})\|_{F}^{2}=\mbox{Tr}[{\bf E}^{T}{\bf E}]$
$\displaystyle{\bf E}$ $\displaystyle=$ $\displaystyle{\bf R}-({\bf r}_{0}{\bf
i}^{T}_{L}+{\bf B}{\bf D})~{}.$ (6)

Differentiating the cost function

$\displaystyle\frac{\partial\phi}{\partial{\bf E}}$ $\displaystyle=2{\bf E}$
(7) $\displaystyle\frac{\partial\phi}{\partial{\bf r}_{0}}$
$\displaystyle=-2[{\bf R}-({\bf r}_{0}{\bf i}^{T}_{L}+{\bf B}{\bf D})]{\bf
i}_{L}$ (8) $\displaystyle\frac{\partial\phi}{\partial{\bf B}}$
$\displaystyle=-2{\bf E}{\bf D}^{T}~{}.$ (9)

Forcing (8) to zero gives ( $\bar{\cdot}$ is the mean operator over
observations $L$)

${\bf r}_{0}=\frac{1}{L}[{\bf R}-{\bf B}{\bf D}]{\bf
i}_{L}=\mathop{\overline{{\bf r}-{\bf B}{\bf d}}}=\mathop{\overline{\bf
r}}-{\bf B}\mathop{\overline{\bf d}}$ (10)

Forcing (9) to zero gives

$\displaystyle{\bf E}{\bf D}^{T}$ $\displaystyle={\bf 0}$ $\displaystyle{\bf
B}{\bf D}{\bf D}^{T}$ $\displaystyle=[{\bf R}-{\bf r_{0}}{\bf i}^{T}_{L}]{\bf
D}^{T}$ $\displaystyle{\bf B}{\bf D}{\bf D}^{T}$ $\displaystyle=[{\bf
R}-(\mathop{\overline{\bf r}}-{\bf B}\mathop{\overline{\bf d}}){\bf
i}^{T}_{L}]{\bf D}^{T}$ $\displaystyle{\bf B}({\bf D}-\mathop{\overline{\bf
d}}{\bf i}^{T}_{L}){\bf D}^{T}$ $\displaystyle=[{\bf R}-\mathop{\overline{\bf
r}}{\bf i}^{T}_{L}]{\bf D}^{T}$ $\displaystyle{\bf B}$ $\displaystyle=[{\bf
R}-\mathop{\overline{\bf r}}{\bf i}^{T}_{L}]{\bf D}^{T}[({\bf
D}-\mathop{\overline{\bf d}}{\bf i}^{T}_{L}){\bf D}^{T}]^{+}~{},$ (11)

where + denotes the Moore-Penrose pseudo-inverse. Thus, the mapping is
determined from (11). The number of calibration points needed depends on the
uniqueness of the points in spatial and DOA space. In general, for an
$N$-dimensional room space, $N+1$ calibration points spanning the space are
sufficient. Using just 2 calibration points maps all observations to a line
between the points and 3 calibration point is sufficient spanning a 2D plane.

###  II-D Disturbances from reference point

For a known reference point between a DOA vector and room location $({\bf
d}_{i},{\bf r}_{i})$, an approximate mapping from a measurement ${\bf d}_{n}$
to a new room location ${\bf r}_{n}$ is for a given $\bf B$ matrix:

$\displaystyle{\bf r}_{n}-{\bf r}_{i}=\frac{\partial{\bf r}}{\partial{\bf
d}}\Delta{\bf d}={\bf B}({\bf d}_{n}-{\bf d}_{i})~{}.$ (12)

For a given $\bf B$ matrix learned though the calibration points, we do the
mapping by selecting a reference point $({\bf d}_{i},{\bf r}_{i})$ close to
the observed ${\bf d}_{n}$ and use (12).

Similarly, for a known DOA PCA components ${\bf a}_{i}$ (as in (1)) and room
location ${\bf r}_{i}$ and given a PCA observation ${\bf a}_{n}$ the new room
location ${\bf r}_{n}$ is

${\bf r}_{n}-{\bf r}_{i}={\bf B}({\bf d}_{n}-{\bf d}_{i})={\bf B}{\bf U}({\bf
a}_{n}-{\bf a}_{i})={\bf C}({\bf a}_{n}-{\bf a}_{i})~{},$ (13)

and ${\bf C}={\bf B}{\bf U}$ describes the mapping from the DOA PCA components
to spatial location. From (13) it is clear that there are an approximately a
linear relation between PCA space and location.

###  II-E Unknown calibration source location

The PCA method comes handy when it is impractical to measure the exact
positions of sound sources for calibration. Thus, we perform the $K$ source
experiments with unknown locations and compute the singular vectors from these
observations to enable PCA. Then for a new source we perform localization in
this PCA space and describe the solution in term of proximity to the $K$
sources in PCA space.

Figure 2: For each frame in time, ODAS selects four potential DOAs (out of the
1321 points) corresponding to the maximum power of SRP-PHAT, and then filters
each DOA with a Kalman filter. ODAS can track up to four simultaneous sources,
but in this work only the loudest source is extracted.

a)
b)

Figure 3: (a) Photo of the five ODAS-arrays mounted in ceiling lamps (array-2,
array-3, array-4) and on walls (array-0, array-1). (b) Layout of room with the
five arrays (red $\otimes$), the 11 calibration speaker points (blue
$\bullet$) and coordinate system with north and $y$ direction aligned.

##  III Experimental design

###  III-A Circular microphone array

We used arrays that are part of development kit called Matrix
Creator111https://www.matrix.one/products/creator with software
(ODAS222http://odas.io) that computes the DOA using the Steered response power
with phase transform (SRP-PHAT)[12, 1] as summarized below as background
information.

Each array consists of 8 microphones, uniformly placed in circle with 10 cm
diameter, and connected to a Raspberry Pi 3
device333https://www.raspberrypi.org/products/raspberry-pi-3-model-b/. Each
device runs its own instance of the ODAS framework, which outputs one DOA with
a 3D unit vector in the array’s local coordinate system, see Fig. 1.

To represent potential DOAs, the unit half sphere is discretized using 1321
points obtained by subdividing a 20-sided convex polyhedron [1], giving a
solid angle of approximately $3^{\circ}$ between grid points. A set of time
difference of arrival (TDOAs) matches each DOA based on the microphone array
geometry and speed of sound (assuming plane waves). For each hypothetical DOA
$i$, ODAS computes SRP-PHAT, with power $E_{i}$[12]:

$E_{i}=\sum_{p=1}^{7}\sum_{q=p+1}^{8}\frac{1}{N}\sum_{k=0}^{N-1}{\frac{X_{p}[k]X^{*}_{q}[k]}{|X_{p}[k]||X_{q}[k]|}\exp{\left(j\frac{2\pi\tau_{i,p,q}k}{N}\right)}},$
(14)

where the superscript ∗ denotes the complex conjugate and $X_{p}[k]$ is the
short-time Fourier transform (STFT) frame, $p$ is the microphone index and $k$
the frequency bin index. Each frame consists of $N$ samples, and the
expression $\tau_{i,p,q}$ is the TDOA at point $i$ between microphones $p$ and
$q$. SRP-PHAT benefits from the low-complexity of the Fast Fourier Transform
(FFT) and makes sound source localization robust to indoor reverberation. At
each time step, ODAS returns up to four potential DOAs corresponding to the
points with maximum power, as shown in (14).

ODAS[1] relies on a tracking module to improve the stability and resolution of
DOA estimation. Speech sources include silence periods, which should be
accounted for while tracking sound sources over time. Robust detection and
tracking therefore becomes an important feature. ODAS provides the DOA of the
four sources with highest SRP-PHAT power (14), and then use Kalman filters to
track the DOAs over time. For this application, we restrict ODAS to track one
source.

The multi-channel raw audio is sampled at 44,100 samples/s from the Matrix
Creator array, resampled by ODAS at 16,000 samples/s, which then returns an
updated DOA estimation with an 8 ms time resolution to the fusion center.
Figure 2 shows the ODAS pipeline for the tracked DOA.

###  III-B Array setup

An office room was equipped with $M=5$ arrays, see Fig. 3a for locations of
the three arrays mounted in ceiling and the two on the wall, and Fig. 3b for
general layout of the room with arrays and calibration points. For
convenience, we ordered the arrays with the local $y$-component pointing up
(for array-0, array-1) or in the global $y$-direction (North) for array-2,
array-3, and array-5). The array locations and headings are assumed unknown.
The DOA for the $M$ arrays are concatenated into one $3M$ dimensional vector
${\bf d}=[{\bf d}_{1}^{T}\dots\bf d_{M}^{T}]^{T}$.

###  III-C Processing chain

Each array processes the data independently on a Raspberry Pi 3, which is
synced to a common clock. The Raspberry Pi then sends the DOAs to a fusion
center, where all DOAs arrays are stored in a database after careful time
alignment. All data in the next sections were extracted from this database.

Figure 4: DOAs measured from the five arrays (array-0 though array-4), here
shown for measurement along the table-edge between the four calibration-points
CP 1-2-3-4 (vertical line).  Figure 5: Based on the calibration matrix $\bf D$
with 15 rows a) Normalized singular values (amplitudes as a fraction of sum of
amplitudes) and First $J$=2 DOA left singular vectors with 15 elements based
on points from (b) table (calibration points 1–6), (c) chairs (calibration
points 7–11), and (d) chair+table (calibration points 1–11). On the horizontal
axis, each array shows 3 points corresponding to the $x,y,z$ component. Apart
from sign change, the singular vectors appear similar. Note the effect of
array orientation on sub-vectors of singular vector. Figure 6: Estimated $\bf
B$ matrix (11) based on (a) table (calibration points 1–6), (b) chair
(calibration points 7–11), and (c) chair+table (calibration points 1–11). On
the horizontal axis, each array shows 3 points corresponding to the $x,y,z$
component.  Figure 7: Affine mapping (3) using estimated $\bf B$ matrix from
calibration points on (a) table (point 1–6), (b) chair (point 7–11), and (c)
chair+table (point 1–11).

##  IV Experimental demonstration

The approach consists of a calibration and a mapping step. In the calibration
step, either the $\bf B$ matrix is created based on sounds from known
calibration locations, or the SVD is determined using available DOAs with no
need for knowing the source locations. In the mapping step we used the
observed DOA vector to map to either room coordinates (3) or PCA coordinates
(1).

###  IV-A Setup

In order to maximally activate DOA localization on each array, an
omnidirectional loudspeaker (10 cm long) playing a female voice is used. The
reverberation time in the office was measured as RT60=0.3 s. ODAS can track up
to four simultaneous sound sources, but we limit it to one in these
experiments. The ODAS sound source tracking module is also tuned to track both
static and moving sources.

An office room is equipped with $M=5$ arrays (see Fig. 3 for locations). Three
arrays are mounted on the ceiling (array-2, array-3, and array-4), array-0 is
placed on south facing wall and array-1 is installed on west facing wall. Note
that the processing is independent on knowing array locations or orientations.
For better physical understanding, we order the arrays with the local
$y$-component pointing up for array-0, array-1, and North for array-2,
array-3, and array-4.

The global $x$-axis (East) corresponds to the negative local $x$-axis for
(array-0, array-2, array-3, and array-4) and $z$-axis for array-1. The global
$y$-axis (North) corresponds to the positive local $y$-axis for (array-1,
array-2, array-3, and array-4) and $z$-axis for array-0. These relations are
for interpretation of results only, and are not used for mapping.

Each Matrix Creator array is connected to a Raspberry Pi 3 (RP3) single-board
computer which provide a DOA estimate every 8 ms. The local clock of each RP3
is synchronized to a local time server at ntp.ucsd.edu over office WIFI
network using the Network Time Protocol (NTP) [44]. When left running
continuously, each local NTP client on RP3 was observed to maintain
synchronization within 10 ms, but most much less. Each array generates a time-
stamped digital summary of detected DOAs once every 64 ms. These Summaries are
stored on local memory of the array and are pushed to the cloud every five
minutes.

Every five minutes new data is down-loaded from the cloud to our fusion center
and loaded into a relational database. The data is then sorted and joined
based on the time stamp. The result is a chronological table which contains
the DOAs from all arrays synchronized using the timestamps. This table can be
queried to get the full information for any period of time.

ODAS tracks the loudest sound source and transfers the DOAs to the fusion
center. At the fusion center, we bin the DOAs using a 64 ms time window. The
average of all DOAs from each array is used in this 64 ms window and stored in
the database. The 64 ms resolution is sufficient as the sound sources move
slowly. Fig. 4 shows the handheld loudspeaker being moved manually along the
table edge.

We use 11 calibration points, see Fig. 3, 6 on table corners (points 1–6) and
5 on chairs (points 7–11) as shown in Fig. 3 placed at heights, 78 cm (table)
and 50 cm (chair).

###  IV-B Calibration

The loudspeaker is placed at each calibration point for 30 s. For these points
we record a 3-component DOA from each of the 5 arrays, giving a $3\times M=15$
element DOA-vector $\bf d$ every 0.064 s (about 450 for each point). Based on
these records in $\bf D$, we extract singular vectors corresponding to
measurements points on chair, table and chair+table, see Fig. 5. Since all
measurements were performed in nearly a horizontal plane, the first two PCA
components carry most of the energy, see Fig. 5a. Focusing on chair+table in
Fig. 5d, the first singular vector has large amplitudes, 0.5, for
$x$-component of (array-0 array-2, array-3, and array-4), all pointing
westward. Thus, an increase in 1st PCA component correspond to a more westerly
source. The second singular vector has large amplitudes, 0.5, for
$y$-component of (array-2, array-3, and array-4) and $x$-component of array-1.
An increase in the 2nd PCA component correspond to a more northerly source.

From these point measurements, the $\bf B$-matrix (11) is extracted Fig. 6, we
split them into batches of chair, table, and chair+table. For all 3 mappings
the $y$-component of array-2 is strongest for bottom row, corresponding to the
y-component of the room. This indicates that an increase in y-component for
array-2 gives a larger global y-component. Focusing on chair+table in Fig. 6c
a larger y-component of array-2 gives a larger global y-component.

Although the true mapping is non-linear, it is possible to find a linear
mapping for the 5 points (Fig. 7a, table), 6 points (Fig. 7a, chair), and 11
points (Fig. 7a). For each point, 450 DOA vectors were recorded with small
fluctuations around the true DOA values. The fluctuations are from the Kalman
filter output of the fixed sources. These fluctuations cause a cloud around
each point when applying the estimated $\bf B$ matrix. The noise cloud near
each point increases as more calibration points are used in Fig. 7c. This is
because as number of points increases, due to non-linearity the mapping on
these points become noisier.

Figure 8: Affine mapping (3) based on sliding source along table edge (points
1-2-3-4) using point 1 as reference and (b) long-table edge (points 5-6) using
point 6 as reference. The estimated $\bf B$ matrix is based on table (point
1–6), chair (point 7–11), and chair+table (point 1–11).  Figure 9: PCA method
projections, $\bf a$, (1) for sliding source along table edge (points
1-2-3-4). The singular vectors are determined based on (a) table (point 1–6),
(b) chair (point 7–11), and (c) chair+table (point 1–11). The PCA of the mean
DOA for the 11 calibration points are also shown. Since the singular vectors
differ for the three set of measurements, their mapping is different.  Figure
10: Mapping (4) based on sliding source along the rectangle of table edge
(points 1-2-3-4) with (a) All 5 arrays working or (b) 1, (c) 2, (d) 3 (using 2
arrays), (e) 4 arrays missing (using 1 array). In (b–e) all combinations of
arrays are considered.

###  IV-C Mapping

We slide the loudspeaker along the table edge (Fig. 4) forming ideally a
rectangle (point 1–4) or a line (point 5–6) to demonstrate the method, see
Fig. 8. First, a $\bf B$-matrix computed using 2 calibration points was used
and we confirmed that in such a case, the mapping was just a projection in the
direction of these two points (not shown). All three $\bf B$-matrices
determined in Fig. 6 were used. The $\bf B$ matrix based on chair points (blue
line) gives the largest error by drifting away from initial point 1 and 6.
This is because chair $\bf B$ matrix is determined at different height and
larger horizontal spread, However, for all mappings it is easy to recognize
the rectangle and the line.

###  IV-D PCA mapping

In an actual room setup, it might be impractical to measure the calibration
source locations. We could potentially learn a room from ambient noise DOAs,
and then do a PCA mapping to the first $J=2$ components. For comparison here
we use the DOAs from the calibration points as illustrated in Fig. 9. The
singular vectors are different for the 3 sets of observation points, see Fig.
5, but in each of the PCA projections the shape of the rectangular table and
table edge are recognized. In Fig. 9, the PCA component changes due to changes
in the magnitude of the singular vectors entries but the major change is due
to sign change of the singular vectors.

The rectangle on the table is 0.76 m $\times$ 1.37 m. Thus, based on Fig. 9c,
the long side of the table is 1.1 units, a change of PCA coordinate,
corresponds roughly to a spatial change of 1.37/1.1=1.25 m.

Figure 11: PCA for sliding source along the table edge (points 1-2-3-4) with
all arrays (blue) or with one array dropped for the whole experiment giving
one rectangle for each combination of 4 arrays. When one array is dropped the
DOAs for that array is set to 0 and the PCA for the full array is used. It is
seen that when one array is dropped the rectangle maps to other region in PCA
space, but can still be identified.  Figure 12: Mapping for sliding source
along the table edge (points 1-2-3-4) with for each observation one of the
five arrays dropping with 50% chance. (a) Mapping (4), and (b) simple PCA.
When mapping to room locations 9a) it is feasible to see the location of the
rectangle.

###  IV-E Missing arrays

A weak source can easily cause an array to miss a source, in this subsection
we assume that all sources for calibration are sufficiently strong. During the
mapping step, the setup could be such that a sound source does not activate
the DOA localization for all arrays. This will give an undefined DOA vector
for that array. In this section the ${\bf B}$-matrix and PCA are determined by
the calibration points on the table (points 1–4).

For the PCA mapping, it is problematic to drop an array as either the PCA has
to be recalculated without the array or the missing DOA from that array should
be estimated. However, a simple solution is to omit that part of the DOA
vector, but still use the same SVD vector based on all arrays. Fig. 10 shows
how this affects the estimation for the whole rectangle. Depending on which
array is dropped it projects to quite different area of the PCA. Thus, if an
array is dropped, it does not give clear results for one observation. However,
Fig. 10 shows that if we drop an entire array in computing the SVD with the
calibration points and for the PCA, the metasurface shows a consistent result
with the rectangle of sources clearly mapped.

The non-linear mapping (4) maps to the domain and therefore shows good results
as long as there are sufficient arrays to perform the mapping, see Fig. 11.
Since all calibration points and measurements are in a horizontal plane, even
just one array (Fig. 11e) can do well in the mapping provided its DOA are
nearly perpendicular to the plane of motion of the sound source. Here, the
ceiling arrays (array-2, array-3, array-4) have their DOAs approximately
perpendicular to the plane of the motion of the sound source. For one array,
the arrays on the walls (array-0, array-1) have DOAs to close to parallel with
the source plane, and hence do not give good result. There is significant
stability improvement in increasing to two arrays (Fig. 11d).

Figure 12 illustrate the difference in (a) affine mapping (4) and (b) PCA
mapping for each observation 50% chance one array is dropped, which array is
dropped is decided uniformly. In this figure the ${\bf B}$-matrix is
determined by the calibration points on the table (points 1–4). Similar to the
above results, the figure shows that simple PCA is noisy with large
uncertainty.

##  V Conclusion

Sound source localization in rooms with redundant un-localized arrays are
discussed. Each array performs the direction of arrival (DOA) estimation
independently and feed it to a cluster where the DOA from all arrays are
concatenated. The DOAs can then be used for source localization, either in
principal component space or directly mapping to the room.

The method was demonstrated with five circular microphone arrays at un-
measured locations in an office. Both methods work well for tracking a source
in a room. The direct mapping to room coordinates is more robust to array
failures. The methods demonstrated here provides a step towards monitoring
activities in smart home with little installation effort.

## Acknowledgements

We appreciate initial assistance of Brian Wang and Yifan Wu.

## References

  * [1] F. Grondin and F. Michaud.  Lightweight and optimized sound source localization and tracking methods for open and closed microphone array configurations.  Robot. Autonom. Syst., 113:63–80, 2019.
  * [2] F. Grondin, D. Létourneau, F. Ferland, V. Rousseau, and F. Michaud.  The ManyEars open framework.  Autonom. Robots, 34(3):217–232, 2013.
  * [3] C. Debes, A. Merentitis, S. Sukhanov, M. Niessen, N. Frangiadakis, and A. Bauer.  Monitoring activities of daily living in smart homes: Understanding human behavior.  IEEE Signal Process. Mag., 33(2):81–94, 2016.
  * [4] A Fleury, M. Vacher, and N. Noury.  SVM-based multimodal classification of activities of daily living in health smart homes: sensors, algorithms, and first experimental results.  IEEE Trans. Inform. Techn. biomedicine, 14(2):274–283, 2009\.
  * [5] T Virtanen, M.D Plumbley, and D. Ellis.  Computational analysis of sound scenes and events.  Springer, 2018.
  * [6] R. Serizel, N. Turpault, A. Shah, and J. Salamon.  Sound event detection in synthetic domestic environments.  In Proc. of IEEE ICASSP, pages 86–90, 2020.
  * [7] T.K. Chan and C. S. Chin.  A comprehensive review of polyphonic sound event detection.  IEEE Access, 8:103339–103373, 2020.
  * [8] C. Zhang, D. Florencio, D. E. Ba, and Z. Zhang.  Maximum likelihood sound source localization and beamforming for directional microphone arrays in distributed meetings.  IEEE Trans. Multimed., 10(3):538–548, 2008.
  * [9] P. Aarabi.  The fusion of distributed microphone arrays for sound localization.  EURASIP J. Adv. Signal Proces., 2003(4):860465, 2003.
  * [10] N.D. Gaubitch, W.B. Kleijn, and R. Heusdens.  Auto-localization in ad-hoc microphone arrays.  In Proc. of IEEE ICASSP, pages 106–110, 2013.
  * [11] M. Cobos, F. Antonacci, A. Alexandridis, A. Mouchtaris, and B. Lee.  A survey of sound source localization methods in wireless acoustic sensor networks.  Wirel. Commun. Mob. Com., 2017, 2017.
  * [12] M.S. Brandstein and H.F. Silverman.  A robust method for speech signal time-delay estimation in reverberant rooms.  In Proc. of the IEEE ICASSP, volume 1, pages 375–378, 1997.
  * [13] H.L. Van Trees.  Optimum Array Processing, chapter 1–10.  Wiley-Interscience, New York, 2002.
  * [14] A. Xenaki, P. Gerstoft, and K. Mosegaard.  Compressive beamforming.  J. Acoust. Soc. Am., 136(1):260–271, 2014.
  * [15] D.P. Wipf and B.D. Rao.  An empirical Bayesian strategy for solving the simultaneous sparse approximation problem.  IEEE Trans. Signal Process., 55(7):3704–3716, 2007.
  * [16] P. Gerstoft, A. Xenaki, and C. F. Mecklenbräuker.  Multiple and single snapshot compressive beamforming.  J. Acoust. Soc. Am., 138(4):2003–2014, 2015.
  * [17] P. Gerstoft, C.F. Mecklenbräuker, W. Seong, and M.J. Bianco.  Introduction to compressive sensing in acoustics.  J. Acoust. Soc. Am., 143(6):3731–3736, 2018.
  * [18] P. Gerstoft, C. F. Mecklenbräuker, A. Xenaki, and S. Nannuru.  Multisnapshot sparse Bayesian learning for DOA.  IEEE Signal Process. Lett., 23(10):1469–1473, 2016.
  * [19] S. Nannuru, K.L. Gemba, P. Gerstoft, W.S. Hodgkiss, and C.F. Mecklenbräuker.  Sparse Bayesian learning with multiple dictionaries.  Signal Process., 159:159–170, 2019.
  * [20] P. Gerstoft, S. Nannuru, C.F. Mecklenbräuker, and G. Leus.  DOA estimation in heteroscedastic noise.  Signal Process., 161:63–73, 2019.
  * [21] K.L. Gemba, S. Nannuru, and P. Gerstoft.  Robust ocean acoustic localization with sparse Bayesian learning.  IEEE J Sel. Topics Signal Process., 13:49–60, 2019.
  * [22] R. Schmidt.  Multiple emitter location and signal parameter estimation.  IEEE Trans Ant. Prop., 34(3):276–280, 1986.
  * [23] K. Nakamura, K. Nakadai, F. Asano, and G. Ince.  Intelligent sound source localization and its application to multimodal human tracking.  In Proc. of IEEE/RSJ IROS, pages 143–148, 2011.
  * [24] K. Nakamura, K. Nakadai, , and G. Ince.  Real-time super-resolution sound source localization for robots.  In Proc. of IEEE/RSJ IROS, pages 694–699, 2012.
  * [25] E. Ozanich, P. Gerstoft, and H. Niu.  A feedforward neural network for direction-of-arrival estimation.  J. Acoust. Soc. Am., 147(3):2035–2048, 2020.
  * [26] A. Barthelme and W. Utschick.  A machine learning approach to DoA estimation and model order selection for antenna arrays with subarray sampling.  arXiv preprint arXiv:2009.12858, 2020.
  * [27] Florian Meyer, Bernhard Etzlinger, Zhenyu Liu, Franz Hlawatsch, and Moe Z Win.  A scalable algorithm for network localization and synchronization.  IEEE Internet of Things Journal, 5(6):4714–4727, 2018.
  * [28] D. Di Carlo, A. Deleforge, and N. Bertin.  Mirage: 2D source localization using microphone pair augmentation with echoes.  In Proc. of IEEE ICASSP, pages 775–779, 2019.
  * [29] C. Rascon, G. Fuentes, and I. Meza.  Lightweight multi-doa tracking of mobile speech sources.  EURASIP, 2015(1):1–16, 2015.
  * [30] M.J. Bianco, P. Gerstoft, J. Traer, E. Ozanich, M.A. Roch, S. Gannot, and C.-A. Deledalle.  Machine learning in acoustics: Theory and applications.  J. Acoust. Soc. Am., 146(5):3590–3628, 2019.
  * [31] S. Chakrabarty and E. A. P. Habets.  Multi-speaker doa estimation using deep convolutional networks trained with noise signals.  IEEE J. Sel. Topics Signal Process., 13(1):8–21, March 2019.
  * [32] S. Adavanne, A. Politis, J. Nikunen, and T. Virtanen.  Sound event localization and detection of overlapping sources using convolutional recurrent neural networks.  IEEE J. Sel. Topics Signal Process., 2019.
  * [33] Y. Wu, R Ayyalasomayajula, M. J. Bianco, D. Bharadia, and P. Gerstoft.  Blind sound source localization based on deep learning.  in press ICASSP, arXiv preprint arXiv:2010.14420, 2020.
  * [34] M.J. Bianco, S. Gannot, and P. Gerstoft.  Semi-supervised source localization with deep generative modeling.  IEEE MLSP, 2020.
  * [35] M. J. Bianco, S. Gannot, E. Fernandez-Grande, and P. Gerstoft.  Semi-supervised source localization in reverberant environments with deep generative modeling.  arXiv preprint arXiv:2101.10636, 2021.
  * [36] Y. Hu, P. N. Samarasinghe, S. Gannot, and T. D. Abhayapala.  Semi-supervised multiple source localization using relative harmonic coefficients under noisy and reverberant environments.  IEEE/ACM Trans. Audio, Speech, Lang. Process, 28:3108–3123, 2020\.
  * [37] M. Liu, L Cheng, K. Qian, J. Wang, J. Wang, and Y. Liu.  Indoor acoustic localization: a survey.  Human-centric Comp. Inf. Sci., 10(1):2, 2020.
  * [38] K. Nakadai, T. Lourens, H. G. Okuno, and H. Kitano.  Active audition for humanoid.  In Proc. Natl. Conf. Artif. Intell., pages 832–839, 2000.
  * [39] K.L. Gemba, S. Nannuru, and P. Gerstoft.  Robust ocean acoustic localization with sparse bayesian learning.  IEEE J. Sel. Topics Signal Process., 13(1):49–60, 2019.
  * [40] S. Zhao, S. Ahmed, Y. Liang, K. Rupnow, D. Chen, and D.L. Jones.  A real-time 3D sound localization system with miniature microphone array for virtual reality.  In Proc. of IEEE ICIEA, pages 1853–1857, 2012.
  * [41] J.-S. Lauzon, F. Grondin, D. Létourneau, A. Lussier Desbiens, and F. Michaud.  Localization of RW-UAVs using particle filtering over distributed microphone arrays.  In Proc. of IEEE/RSJ IROS, pages 2479–2484, 2017.
  * [42] X. Xiao, S. Zhao, D.H.H. Nguyen, X. Zhong, D.L. Jones, E.S. Chng, and H. Li.  The NTU-ADSC systems for reverberation challenge 2014.  In Proc. of REVERB Challenge Workshop, 2014.
  * [43] X. Dang and H. Zhu.  A feature-based data association method for multiple acoustic source localization in a distributed microphone array.  J. Acoust. Soc. Am., 149(1):612–628, 2021.
  * [44] D.L. Mills.  On the accuracy and stability of clocks synchronized by the network time protocol in the internet system.  ACM SIGCOMM Computer Comm. Review, 20(1):65–75, 1989.