# Evaluating Deep vs. Wide & Deep Learners as Contextual Bandits for
Personalized Email Promo Recommendations

Aleksey A. Kocherzhenko
<EMAIL_ADDRESS>
Nirmal Sobha Kartha
<EMAIL_ADDRESS>
Tengfei Li
<EMAIL_ADDRESS>Hsin-Yi (Jenny) Shih
<EMAIL_ADDRESS>Marko Mandic
<EMAIL_ADDRESS>Mike Fuller
<EMAIL_ADDRESS>Arshak Navruzyan
<EMAIL_ADDRESS>LaunchPad.AI, Honolulu, HI, USA

###### Abstract

Personalization enables businesses to learn customer preferences from past
interactions and thus to target individual customers with more relevant
content. We consider the problem of predicting the optimal promotional offer
for a given customer out of several options as a contextual bandit problem.
Identifying information for the customer and/or the campaign can be used to
deduce unknown customer/campaign features that improve optimal offer
prediction. Using a generated synthetic email promo dataset, we demonstrate
similar prediction accuracies for (a) a wide and deep network that takes
identifying information (or other categorical features) as input to the wide
part and (b) a deep-only neural network that includes embeddings of
categorical features in the input. Improvements in accuracy from including
categorical features depends on the variability of the unknown numerical
features for each category. We also show that selecting options using upper
confidence bound or Thompson sampling, approximated via Monte Carlo dropout
layers in the wide and deep models, slightly improves model performance.

_Keywords_ Contextual Bandit, Wide & Deep Learning, Recommender Systems

##  1 Introduction

A personalized recommendation model can be conceptualized as a ranking system
that learns from customers’ past engagements and interactions. Pu et al.
(2011)Polatidis and Georgiadis (2013). Recommendation systems are a critical
component of the digital marketing industry, where they are used for matching
digital ads to customer preferences. In 2019, this industry was valued at 43.8
billion USD, with an expected compound annual growth rate of 17.4% for
2020–2027 Research ([n.d.]).

Deep neural networks are extensively used in personalized recommendation
systems. Zhang et al. (2019) However, although they are good at generalization
(prediction for previously unseen feature combinations), they struggle with
memorization (prediction based on co-occurrences of previously observed
categorical feature values) Cheng et al. (2016). Wide and deep networks that
use wide linear models to memorize sparse feature cross-products and deep
neural networks to generalise to unseen feature values help solve this problem
Zhang et al. (2019).

Recommendation models are typically trained in an online setting, so as to
learn from shifting user preferences. However, because predictions aid
decision-making in the production context and influence customer decisions,
they can create positive feedback loops that alter the underlying data
generating process and result in a selection bias for the input data Chaney et
al. (2017). Causal inference Bottou et al. (2013) and bandit theory Li et al.
(2010) can be used to account for such algorithmic bias.

The current paper makes the following contributions:

  * •

We propose a strategy for generating synthetic email promo datasets with
clearly defined measures for unknown deterministic features and feature
randomness.

  * •

We evaluate the performance of the wide & deep, wide-only and deep-only
architectures.

  * •

We show that bandit algorithms that use upper confidence bound (UCB) Agrawal
and Goyal (2012) and Thompson sampling (TS)Auer et al. (2002) for action
selection slightly improves model performance.

Our code is publicly available for use by future researchers.

##  2 Problem Formulation

A business regularly runs marketing campaigns and for each campaign needs to
decide which of several promotional offers might most appeal to each customer.
We assume that the preferences of each customer are entirely determined by
that customer’s personal characteristics, as well as those of the offers:
i.e., if we knew everything about a customer and all available offers, we
could predict that customer’s preferred offer with certainty. Then, there
exists a generally non-linear function $f$ that deterministically maps
$\mathbf{C_{j}}$ and $\mathbf{P_{k}}$, complete features of the
$j^{\mathrm{th}}$ customer and $k^{\mathrm{th}}$ campaign respectively, to one
of $N$ available promotional offers, $\mathbb{A}=\left\\{1,...,N\right\\}$:

$f\left(\mathbf{C}_{j},\mathbf{P}_{k}\right)=a\in\mathbb{A}$ (1)

The number of features that completely describe a customer (or a promotional
campaign) is enormous, so only a _subset_ of features will ever be known. If
$L$ features are known for each customer and $M$ features are known for each
campaign, the projections of $\mathbf{C}_{j}$ and $\mathbf{P}_{k}$ onto the
sub-spaces of _known_ customer and campaign features are an $L$-dimensional
vector and an $M$-dimensional vector, respectively. We denote corresponding
vectors in the full feature space, with non-zero components representing the
_known_ features as $\mathbf{c}_{j}^{L}$ and $\mathbf{p}_{k}^{M}$. We denote
vectors in the full feature space that represent the $L^{\prime}$ _unknown_
customer features and $M^{\prime}$ _unknown_ campaign features as
$\mathbf{c}_{j}^{L^{\prime}}$ and $\mathbf{p}_{k}^{M^{\prime}}$, respectively.
Then,

$\mathbf{C}_{j}=\mathbf{c}_{j}^{L}+\mathbf{c}_{j}^{L^{\prime}},\;\;\;\;\;\mathbf{P}_{k}=\mathbf{p}_{k}^{M}+\mathbf{p}_{k}^{M^{\prime}}.$
(2)

Given unique IDs $j$ and $k$ for each customer and promotional campaign
respectively, there exists a _one-to-one_ mapping between each ID and the
corresponding complete feature vector: $j\leftrightarrow\mathbf{C}_{j}$,
$k\leftrightarrow\mathbf{P}_{k}$. The existence of such mappings requires the
existence of mappings of IDs to _each of the components_ of the corresponding
vectors (but not vice versa). Therefore, there must exist functions

$g:~{}g\left(j\right)=\mathbf{c}_{j}^{L^{\prime}},\;\;\;\;\;h:~{}h\left(k\right)=\mathbf{p}_{k}^{M^{\prime}},$
(3)

that are generally nonlinear and not necessarily invertible.

Introducing Eqs. (2–3) into Eq. (1), we find:

$f\left(\mathbf{C}_{j},\mathbf{P}_{k}\right)=f\left[\mathbf{c}_{j}^{L}+g\left(j\right),\mathbf{p}_{k}^{M}+h\left(k\right)\right]=F\left(\mathbf{c}_{j}^{L},\mathbf{p}_{k}^{M},j,k\right)=a,$
(4)

where $F$ is some new nonlinear function. In the limit when no customer or
promotional campaign features are known, but it is known that all relevant
features are uniquely defined by the customer ID and the campaign ID, Eq. (4)
reduces to

$F\left(j,k\right)=a.$ (5)

Optimal offers for previously unseen customers and promotional campaigns can
be learned more quickly based on similarity to previously seen customers and
promotional campaigns (generalization). The better the vectors
$\mathbf{c}_{j}^{L}$ and $\mathbf{p}_{k}^{M}$ approximate the vectors
$\mathbf{C}_{j}$ and $\mathbf{P}_{k}$, and the better the space of all
customer and promotional campaign features is sampled, the more accurately the
optimal offer for any new customer-promotional campaign pair can be inferred.
Observing optimal offers for _specific_ customer-promotional campaign pairs
allows refining predictions for the rare cases when subtle differences between
similar customer-promotional campaign pairs lead to different optimal offers.
If similarity between different customer-promotional campaign pairs cannot be
established _a priori_ , as is the case for Eq. (5), no generalization is
possible and the optimal offer for each customer-campaign pair must be
memorized individually, making the training process inefficient. With partial
information about customer and promotional campaign features, memorization and
generalization can be combined to enhance prediction accuracy and training
efficiency.

##  3 Related Work

The idea of combining wide linear models with deep neural networks for
building robust recommendation systems that can memorize as well as generalize
was first proposed in Ref. Cheng et al. (2016), refining the idea of
factorization machines Rendle (2012). The system was deployed and evaluated on
the Google Play app store, which yielded a positive app acquisition gain of
3.9% in an online setting.

Personalized dynamic recommendation has been reframed in a contextual bandit
setting to mitigate the cold-start problem and algorithmic bias Li et al.
(2010). This paper also developed the LinUCB algorithm to efficiently estimate
confidence bounds in closed form. Because computing the actual posterior
distribution of the bandit problem is usually intractable, Refs. Dielman et
al. (1994) and Gal and Ghahramani (2016) proposed sampling techniques that use
Monte Carlo (MC) dropout to approximate the posterior distribution. Both UCB
and TS on this approximated distribution perform well and are computationally
efficient. Empirical evaluation of many techniques for approximating TS and
the best practices for designing such approximations are discussed in Ref.
Riquelme et al. (2018). Recently, combining bandit algorithms with deep neural
networks in an ad recommendation system on a proprietary dataset has also been
explored Guo et al. (2020).

##  4 Methodology

###  4.1 Generalizing via Embeddings

Embeddings map categorical variables to vectors of pre-defined dimensionality
and allow reducing dimensionality and encoding similarities in the data.
Typically randomly initialized, they are then learned in a relevant supervised
machine learning task. Including an $L_{\mathrm{e}}$-dimensional embedding
vector for the user ID $j$ and an $M_{\mathrm{e}}$-dimensional embedding
vector for the campaign ID $k$ as input to a deep neural network allows
training them, over many occurrences of individual customers and promotional
campaigns in the dataset.

The resulting embeddings may capture unknown customer and promotional campaign
features that may be important for selecting the optimal offer $a$. In
practice, a customer is represented by the _concatenation_ of the known
customer feature vector and the customer ID embedding vector. Then, by
construction, $\mathbf{c}_{j}^{L}$ and the vector
$\mathbf{c}_{j}^{L_{\mathrm{e}}}$ that represents embedding in the full
customer feature space span orthogonal sub-spaces and are linearly
independent. Consequently,
$\mathbf{c}_{j}^{L}+\mathbf{c}_{j}^{L_{\mathrm{e}}}$ spans an
$\left(L+L_{\mathrm{e}}\right)$-dimensional subspace of the total customer
feature space and can, in principle, approximate the complete feature vector
$\mathbf{C}_{j}$ better than $\mathbf{c}_{j}^{L}$ that spans only an
$L$-dimensional sub-space. Similar reasoning applies to using an embedding
vector $\mathbf{p}_{k}^{M_{\mathrm{e}}}$ for campaign IDs.

If $\mathbf{c}_{j}^{L}$ and $\mathbf{p}_{k}^{M}$ are treated as fixed
constraints, optimizing $\mathbf{c}_{j}^{L}+\mathbf{c}_{j}^{L_{\mathrm{e}}}$
and $\mathbf{p}_{k}^{M}+\mathbf{p}_{k}^{M_{\mathrm{e}}}$ only changes the
embedding vectors. Because these vectors and the known feature vectors are
linearly independent, training the embeddings does not double-count the known
features, but rather augments them. Consequently, a nonlinear function

$\widetilde{f}_{\mathrm{e}}\left(\mathbf{c}_{j}^{L}+\mathbf{c}_{j}^{L_{\mathrm{e}}},~{}\mathbf{p}_{k}^{M}+\mathbf{p}_{k}^{M_{\mathrm{e}}}\right)=\widetilde{a}_{\mathrm{e}}\in\mathbb{A}$
(6)

that is learned using both the known feature vectors and the embedding vectors
should return better predictions than the nonlinear function learned only
using known feature vectors.

###  4.2 Memorizing Cross-Products

Instead of embeddings, one can also use cross-products of customer and
campaign IDs Cheng et al. (2016) (treated as categorical features) in the wide
part of a wide & deep neural network. We must explicitly make the assumption
that the optimal offer is specific to every customer-promotional campaign
_pair_ , similarly to Eq. (5).

If sufficient data for _all_ possible user ID-campaign ID pairs is available,
a wide linear model is sufficient and adding a deep neural network provides no
benefit for predicting the optimal offer. However, it is impossible to train
the model on all possible user ID-campaign ID pairs if new customers are
continuously added and new promotional campaigns are regularly run.
Furthermore, if there is _variability_ in customer or campaign features with a
given user or campaign ID, then such variability cannot captured by $F$ in Eq.
(5).

In section 4.1, the known customer and promotional campaign features were
augmented with user ID and campaign ID embeddings that capture otherwise
unknown customer and promotional campaign features, once sufficiently trained.
In this approach, cross-products of user IDs and campaign IDs are input to a
wide linear model, $l\left(j\times k\right)$, whereas the known customer and
promotional campaign feature vectors are input to a deep nonlinear model,
$\widetilde{f}_{\mathrm{d}}\left(\mathbf{c}_{j}^{L},\mathbf{p}_{k}^{M}\right)$,
in the same way as in Eq. (6). The outputs of the linear and nonlinear
transformations are then passed to the output layer:

$\sigma\left[\widetilde{f}_{\mathrm{d}}\left(\mathbf{c}_{j}^{L},\mathbf{p}_{k}^{M}\right),l\left(j\times
k\right)\right]=\widetilde{a}_{\mathrm{dw}}\in\mathbb{A},$ (7)

where $\sigma$ is a nonlinear activation function.

###  4.3 Generated Dataset

The architectures discussed in Sections 4.1 and 4.2 represent different
approximations of Eq. (4). We hypothesize that, given sufficient training
data, both should result in similar prediction accuracy. To test this
hypothesis, we generated a synthetic dataset of 10 million email promo
campaign instances with the following features for each sample: user/customer
ID; customer features 1 and 2; hidden customer feature; campaign ID; campaign
features 1 and 2; hidden campaign feature.

The dataset contains 1,000 distinct customers, each identified by a unique ID
and seen a random number of times, with occurrences randomly distributed
throughout the dataset. The relative frequency of seeing a specific customer
is sampled from a uniform distribution on the interval (0, 1); the minimum and
maximum numbers of times that any single customer is seen are 17 and 20,004,
respectively. Every customer is described by three features: customer features
1 and 2, as well as a hidden customer feature. For a given customer, the mean
values and the standard deviations for each of these features are constant.
The mean values of customer features 1 and 2 are sampled from a uniform
distribution on the interval (–0.5, 0.5). The standard deviations for these
features are sampled from uniform distributions on the intervals (0, 0.2) and
(0, 0.3), respectively. The mean value of the hidden customer feature is
sampled from a uniform distribution on the interval (–0.2, 0.2), and the
standard deviation is sampled from a uniform distribution on the interval (0,
0.1). The values of all customer features vary from sample to sample and are
sampled from normal distributions with fixed means and standard deviations for
each customer. The numbers of samples with each combination of customer
features 1 and 2 are shown in Fig.1A; the numbers of samples with each value
of the hidden customer feature are shown in Fig.1B.

The samples are evenly distributed among 100 promotional campaigns, each
identified by a unique campaign ID. Each promotional campaign consists of
100,000 samples, and the promotional campaigns are listed sequentially. Every
promotional campaign is described by three features: campaign features 1 and
2, and hidden campaign feature. For a given promotional campaign, the mean
values and the standard deviations for each of these features are constant.
The mean values for campaign feature 1 and campaign feature 2 are sampled from
a uniform distribution on the interval (–0.5, 0.5). The standard deviations
for these features are sampled from uniform distributions on the intervals (0,
0.1) and (0, 0.05), respectively. The mean value for the hidden campaign
feature is sampled from a uniform distribution on the interval (–0.15, 0.15),
and the standard deviation is sampled from a uniform distribution on the
interval (0, 0.05). The values of all promotional campaign features for
individual samples in the dataset are sampled in the same way as for customer
features. The numbers of samples with each combination of campaign feature 1
and campaign feature 2 are shown in Fig. 1C; the numbers of samples with each
value of the hidden campaign feature are shown in Fig. 1D.

Figure 1: Distributions of customer (top) and campaign (bottom) features in
the dataset. Left - combinations of known feature values; right - hidden
feature value.

As evident from Fig. 1, the customer feature space is sampled better than the
campaign feature space because there are 1,000 distinct customers (compared to
100 distinct campaigns) and larger standard deviations. There is a one-to-one
correspondence between the user ID and the _mean values_ and _standard
deviations_ for each customer feature (likewise for the promotional campaign
features). The actual feature values describing a specific occurrence of a
given customer or promotional campaign are _sampled_ from more or less narrow
normal distributions, with fixed defined means and standard deviations. This
ensures variation in the customer features between different samples with the
same user ID, and in the promotional campaign features between different
samples with the same campaign ID. This construction reflects the fact that,
in practice, the values of many customer and promotional campaign features may
vary somewhat between appearances of a customer or a promotional campaign in
the dataset due to unaccounted for factors that are external to the
customer–promotional campaign system and reflect its interaction with the
broader “environment".

The three customer features for the $n$th sample in the dataset can be
represented by a three-dimensional customer feature vector, $\mathbf{c}_{n}$,
and the three promotional campaign features for the $n$th sample can be
represented by a three-dimensional promotional campaign feature vector,
$\mathbf{p}_{n}$, where the sample number $n=1,...,10,000,000$ (these vectors
are schematically illustrated in Fig. 2).

Figure 2: Schematic representation of customer feature vector $\mathbf{c}_{n}$
(left) and campaign feature vector $\mathbf{p}_{n}$ (right). Dashed black
boxes show the range of possible feature combinations. Dashed colored boxes
illustrate the range of feature variability for a given customer or campaign.
Dashed lines show projections of the complete feature vector.

The optimal offer for the $n$-th sample is defined by

$a_{n}=\left\\{\begin{array}[]{ll}1,&\mathrm{if}~{}\mathbf{c}_{n}\uparrow\uparrow\mathbf{p}_{n},\\\
\mathrm{ceil}\left[\frac{10}{\pi}\arccos\left(\frac{\mathbf{c}_{n}\cdot\mathbf{p}_{n}}{\left|\mathbf{c}_{n}\right|\left|\mathbf{p}_{n}\right|}\right)\right],&\mathrm{otherwise},\end{array}\right.$
(8)

a piecewise continuous nonlinear function of the angle between the vectors
$\mathbf{c}_{n}$ and $\mathbf{p}_{n}$ that can take 10 values,
$a_{n}\in\left\\{1,...,10\right\\}$. For two samples $n$ and $n^{\prime}$,
where the customers and promotional campaigns have similar features, the
optimal offers defined by Eq. (8) will be the same, $a_{n}=a_{n^{\prime}}$,
unless the angles between their customer and promotional campaign feature
vectors happen to be separated by a point of discontinuity (in which case
$\left|a_{n}-a_{n^{\prime}}\right|=1$).

All models in our experiment were trained on 60% of the generated dataset,
with 20% used for validation and 20% used for testing; the data was split into
the training, validation, and test sets _randomly_. For comparison, a
different split of the dataset was also used for training and testing two of
the models: the _first_ 80% of the data was randomly split into the training
and validation datasets at a 3:1 ratio, and the _remaining_ 20% was used for
testing.

###  4.4 Model Architectures

This paper focuses on 3 network architectures: a wide model, a deep model, and
a wide & deep model.

Figure 3: Architectures used for the wide (A), deep (B) and wide & deep models
(C). Italic numbers in square brackets indicate model versions with different
inputs (See Table 1).

The wide model (Fig. 3A) took as input 100,000 cross-products between the 100
distinct user IDs and the 1,000 distinct campaign IDs in the dataset. This
input was passed to a single linear transformation and a softmax function to
select one of the ten possible offers. This model had 1 million trainable
parameters.

The deep model (Fig. 3B) used a dense neural network with three hidden layers
(with 512, 256, and 128 nodes and ReLU activation functions), followed by a
softmax output layer to select among the ten possible offers. This model was
trained with four different sets of features in the input:

  1. 1.

16-dimensional embeddings for user IDs and 7-dimensional embeddings for
campaign IDs;

  2. 2.

customer features 1 and 2 and campaign features 1 and 2;

  3. 3.

customer features 1 and 2, campaign features 1 and 2, as well as
16-dimensional embeddings for user IDs and 7-dimensional embeddings for
campaign IDs;

  4. 4.

customer features 1 and 2, campaign features 1 and 2, as well as the hidden
customer and campaign features.

Even with the largest input dimensions (version 3), this model had under 200
thousand trainable parameters (considerably fewer than the wide model). We
found that increasing or decreasing the number of nodes in all hidden layers
by a factor of 2 had no statistically significant effect on the resulting
prediction accuracy.

For the wide and deep model (Fig. 3C), the cross-products between user IDs and
campaign IDs (wide input) were first concatenated with the output of the third
hidden layer of the deep model and then passed to the output softmax layer.
Only customer features 1 % 2 and campaign features 1 & 2 were used as input
for the deep model.

TensorFlow Abadi et al. (2015) implementations of these models were trained
using categorical cross-entropy loss and the Adam optimizer with a learning
rate of $10^{-3}$. Early stopping with a patience of 5 epochs (deep, wide) or
10 epochs (wide & deep) was applied. Varying the patience between 5 and 20
epochs had no significant effect on the resulting accuracy. Early stopping
occurred after 16 epochs for the wide model, after 46, 33, 41, and 14 epochs
respectively, for the four sets of inputs to the deep model, and after 108
epochs for the wide & deep model.

###  4.5 Bandit Implementation

A contextual bandits predicts an action that should be taken for a specific
context in order to maximize a reward. In selecting an action, the bandit
algorithm balances _exploitation_ of prior knowledge about the rewards
associated with various possible actions and _exploration_ of actions that
have not been tried enough times to confidently predict their associated
rewards.

In the scenario considered in this paper, contextual bandits are applied to a
multi-class classification problem, where the objective is to find the mapping
of the customer and campaign features that serve as the context onto the
optimal action, in accordance with Eq. (4). The probabilities of each action
being optimal are returned by the softmax output layer of a neural network
with one of the architectures described in Section 4.4. We treat these
probabilities as the normalized rewards associated with each action.

We consider two popular bandit algorithms, Thompson sampling (TS) and upper
confidence bound (UCB), and compare their performance to that of a neural
network that does not use a bandit algorithm. The TS algorithm randomly
samples a single value from the posterior distribution of the rewards
associated with each action, then selects the action with the highest sampled
reward. The UCB algorithm optimistically selects the action with the highest
upper confidence bound for its reward value. The confidence bounds depend on
the expected rewards associated with each action and on their uncertainties
that can be estimated from the posterior distributions of rewards.

The posterior distributions of rewards are approximated using the Monte Carlo
(MC) dropout method Gal and Ghahramani (2016). The reward distributions are
approximated by making multiple forward passes through a neural network with
MC dropout at inference time. To improve computational inefficiency, MC
dropout (with dropout probability $p=0.3$) is only applied in a “multi-head”
layer that immediately precedes the softmax output layer Guo et al. (2020).
Thus, only one forward pass through the computation graph up to the dropout
layer is necessary, and multiple passes from the dropout layer onward can be
executed in parallel.

To select an action using the TS algorithm, we perform a single pass through
the network with MC dropout and pick the action with the largest sampled
value. In the UCB algorithm at 95% confidence level, we perform 100 forward
passes starting at the MC dropout layer and approximate the upper confidence
bound for each action by the $5^{\mathrm{th}}$-largest value among 100
samples.

##  5 Results

Table 1 shows the accuracy of the optimal offer predictions for all models,
trained and tested on random subsets of our dataset.

Table 1: Performance Comparison: Wide vs Deep vs Wide & Deep architectures Model |  Deep input |  Wide input | Accuracy, %
---|---|---|---
Train |  Valid |  Test
Random |  None |  None |  10.0 |  10.0 |  10.0
Wide |  None |  user ID $\times$ campaign ID |  35.2 |  33.3 |  33.3
Deep |  1User & campaign ID embeddings |  None |  27.6 |  27.0 |  26.9
|  2Customer features 1&2,
Campaign features 1&2 |  68.0 |  68.1 |  68.0
|  3Customer features 1&2,
Campaign features 1&2,
User & campaign ID embeddings |  81.3 |  80.9 |  80.8
|  4Customer features 1&2,
Campaign features 1&2,
Hidden User & Campaign features |  97.7 |  98.0 |  97.2
Wide & Deep |  Customer features 1&2,
Campaign features 1&2 |  user ID $\times$ campaign ID |  80.2 |  77.3 |  77.3

As expected, random guessing results in 10% accuracy for predicting the
optimal offer out of 10 possible offers. Also as expected, the prediction
accuracy for the deep model that takes the complete customer and promotional
campaign feature vectors as input approaches 100% (it is 97.2% for the test
set and could potentially be further improved by providing additional training
data, using a larger neural network, adjusting the learning rate, etc.).

Note that for all subsequent models, hidden customer and campaign features are
_not used_ : only customer features 1 and 2, campaign features 1 and 2, and/or
the user and campaign IDs are used for predicting the optimal offer. Thus, the
models were deliberately provided with incomplete information:

  * •

Numerical customer and campaign features supplied projections of the complete
customer and campaign feature vectors onto the “known" customer and campaign
feature subspaces.

  * •

In accordance with Eq. (3), the user and campaign IDs can be mapped to the
_mean value_ and _standard deviation_ that define the distribution of the
hidden customer and campaign features for a specific customer and a specific
promotional campaign. However, the user and campaign IDs _cannot_ be
deterministically mapped to the values of the hidden customer and campaign
features _for a specific sample_ , because these features are _randomly
sampled_ from a normal distribution with a fixed mean and a fixed standard
deviation.

Therefore, even with the user and campaign IDs available, the information that
is provided to the models is incomplete.

If only identifying information for the customer (user ID) and for the
promotional campaign (campaign ID) were known, any inference of the optimal
offer would be based on memorization. The wide-only model with the user ID and
campaign ID cross products as input correctly predicts the optimal offer in
about a third of cases, while the deep-only model with only the user ID and
campaign ID embeddings as input correctly predicts the optimal offer in just
over a quarter of all cases (see Table 1). The differences in accuracy between
the two models are not due to difference in the number of trainable
parameters, as varying layer sizes for the deep model did not significantly
affect prediction accuracy. These differences are also not due to insufficient
training data, as training both models on smaller subsets did not
significantly change accuracies either. Therefore, the wide model outperforms
the deep model when it comes to pure memorization, at least for the embedding
sizes used for the user ID and the campaign ID. This difference in accuracy
may be due to more efficient memorization for the wide model, compared to a
deep model, where the user ID and the campaign ID embeddings are randomly
initialized and then adjusted every time a user ID or campaign ID value is
encountered in the training data.

Note that the poor performance of both the wide and the deep models trained
using only the user and campaign IDs is largely determined by the relatively
broad distributions of the customer (campaign) features for a given user ID
(campaign ID). For instance, reducing the maximum standard deviations for
customer feature 1, customer feature 2, and the hidden customer feature from
0.2, 0.3, and 0.1 respectively, to 0.05, 0.1, and 0.05 respectively, without
changing the campaign feature distributions, improves the performance of the
deep model from 26.9% to 62.1%. If the customer and campaign features were
completely determined by the user and campaign IDs, in accordance with Eq.
(3), then near-perfect prediction accuracy could, in principle, be achievable
for both the wide and deep models, provided that they were sufficiently
trained.

Table 2: Performance Comparison: Different Feature Distributions for Training/Validation and Test Sets Model |  Deep input |  Wide input | Accuracy, %
---|---|---|---
Train |  Valid |  Test
Deep |  Customer features 1&2,
Campaign features 1&2,
User & campaign ID embeddings |  None |  81.9 |  81.2 |  58.1
Wide & Deep |  Customer features 1&2,
Campaign features 1&2 |  user ID $\times$ campaign ID |  80.0 |  77.6 |  63.2

If only the projections of the complete customer and campaign feature vectors
onto the known customer and campaign feature subspaces (customer features 1
and 2, campaign features 1 and 2) were known, then the accuracy of the optimal
offer predictions would largely be determined by how well these projections
approximate the complete feature vectors. In the limit of the hidden customer
and hidden campaign features being zero for all samples, customer features 1
and 2 and campaign features 1 and 2 would provide complete information about
the customer and the promotional campaign. Consequently, near-perfect accuracy
for this model could be expected, for a sufficiently large and sufficiently
trained neural network. When the deep model only takes as input customer
features 1 and 2 and campaign features 1 and 2 in our generated dataset that
has relatively large hidden customer and campaign features, it can predict the
optimal offer in over two thirds of cases (see Table 1). Further improving the
prediction accuracy requires accounting for the hidden customer and campaign
features that can, to an extent, be inferred from the user and campaign IDs.
The precision of this inference is limited by the variability in the hidden
features for a given customer (promotional campaign).

Predictably, the highest accuracy of the optimal offer predictions were
achieved when both the identifying information for the customer and the
promotional campaign, as well as the projections of the complete customer and
campaign feature vectors onto the known customer and campaign feature
subspaces were provided to the machine learning models. The deep model that
took customer features 1 and 2, campaign features 1 and 2, as well as the user
ID and campaign ID embeddings as input achieved an accuracy of 80.8% on the
test set. The wide & deep model that took customer features 1 and 2 and
campaign features 1 and 2 as input to the deep part, as well as user
ID–campaign ID cross-products as input to the wide part, achieved an accuracy
of 77.3% on the test set. Interestingly, the deep-only model slightly
outperformed the wide & deep model, even though the wide model that took user
ID–campaign ID cross-products as input outperformed the deep model that took
only user ID and campaign ID embeddings as input. Examining the confusion
matrices for the predicted optimal offer, $\widetilde{a}\in\mathbb{A}$, vs.
the actual optimal offer, $a\in\mathbb{A}$, suggests that the performance of
both models is largely limited by the randomness in the hidden customer and
campaign features: $\left|a-\widetilde{a}\right|\leq 1$ for over 95% of the
predictions.

It should be noted that the reasonably high accuracies for the deep-only and
the wide & deep models that took customer features 1 and 2, campaign features
1 and 2, as well as the user and campaign IDs as input were achieved because
the training, validation, and test sets were _randomly sampled_ from the
generated dataset. Consequently, the training, validation, and test sets all
contained samples with (nearly) all possible user ID–campaign ID combinations.
Randomly splitting the _first_ 80% of the data between the training and
validation sets at a 3:1 ratio and using the _remaining_ 20% as the test set
leads to significantly worse prediction accuracies on the test set for both
the deep-only and the wide & deep models (Table 2). In this case, the model is
trained and tested on _different_ identifying information (different
campaigns), making memorization useless, and, in fact, counterproductive: the
accuracies on the test set for both the deep-only and the wide & deep models
turn out to be considerably lower than the accuracy for the deep model that
just uses customer features 1 and 2 and campaign features 1 and 2 as input
(Table 1). In practice, this result means that considering a cross-product of
a user ID with a campaign _type_ , for which certain features are similar
across multiple campaigns, is likely to lead to enhanced performance of a
machine learning model. Conversely, using a cross-product of a user ID with a
campain ID that will not re-occur in the data to which the model will
eventually be applied may be detrimental to the model’s prediction accuracy.

We evaluated the benefit of using the implementations of UCB and TS, outlined
in Section 4.5, on the output of the wide & deep model for our dataset (see
Table 3). Both bandit algorithms show a slight improvement in accuracy
compared to the wide & deep model without explicit exploration. Our
implemention of UCB consistently quantified uncertainty and predicted optimal
offers better than TS, but was considerably more computationally expensive
with, on average, an order of magnitude longer runtimes.

Table 3: Performance Comparison: Bandit Algorithms Model | Accuracy, %
---|---
Train | Valid | Test
No Bandit | 80.2 | 77.3 | 77.3
TS | 81.1 | 80.3 | 79.3
UCB | 81.6 | 80.8 | 80.2

##  6 Conclusions and Future Work

Identifying features, such as user or campaign IDs, can significantly improve
predictions of machine learning models, especially when properties of the
customer and of the promotional campaign are uniquely determined by their
identities. However, variability in features that correspond to the same ID
reduces the predictive power of identifying features (or other categorical
features that characterize the properties of individual customers or
campaigns).

If the optimal offer is, to some extent, determined by the cross-product of
two or more categorical features, then any improvements in the accuracy from
using these features in a model hinges on the following requirement: the data
used to train the model should contain the same _combinations_ of the
categorical feature values as the test set. Otherwise, using cross-products of
categorical features may be detrimental to the model’s accuracy on the test
set.

Cross-products of relevant categorical features can be provided as input to
the wide part of a wide & deep neural network. Alternatively, embeddings for
individual categorical features can be included in the input for a deep neural
network. In the setting we explored, both models showed similar prediction
accuracies and computational costs. These results may be dataset dependent, so
it would be of interest to study how the accuracies and computational costs
for both models depend on the total numbers of numerical and categorical
features, the variability between features for a given customer and/or
promotional campaign, the number of offers per campaign, the embedding sizes
for categorical features, etc.

A wide & deep model requires more careful feature engineering than a deep
model with embeddings for categorical features. Specifically, it requires
explicitly identifying the cross-products of categorical features that affect
the optimal offer. Conversely, when using embeddings in a deep model, any
interplay between categorical features is implicit in the trained network
parameters. If the relevant cross-products are easily identifiable, using a
wide & deep model may be the better option. Otherwise, particularly for
datasets with many categorical features, a deep model with embeddings may
require less extensive exploratory data analysis.

Our network approximations of the TS and UCB bandit algorithms conferred an
increase in accuracy of 1–3% for predicting the optimal offer. However, the
computational cost of bandit algorithms may complicate their adoption in
online settings. Because in online settings a sound exploration strategy can
limit algorithmic bias, we intend to explore possible solutions in future
research.

## References

  * (1)
  * Abadi et al. (2015) Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2015\.  TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems.  https://www.tensorflow.org/ Software available from tensorflow.org.
  * Agrawal and Goyal (2012) Shipra Agrawal and Navin Goyal. 2012.  Thompson Sampling for Contextual Bandits with Linear Payoffs.  _CoRR_ abs/1209.3352 (2012).  arXiv:1209.3352 http://arxiv.org/abs/1209.3352
  * Auer et al. (2002) Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. 2002\.  _Machine Learning_ 47, 2/3 (2002), 235–256.  https://doi.org/10.1023/a:1013689704352
  * Bottou et al. (2013) Léon Bottou, Jonas Peters, Joaquin Quiñonero-Candela, Denis X. Charles, D. Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. 2013\.  Counterfactual Reasoning and Learning Systems: The Example of Computational Advertising.  _Journal of Machine Learning Research_ 14, 65 (2013), 3207–3260.  http://jmlr.org/papers/v14/bottou13a.html
  * Chaney et al. (2017) Allison J. B. Chaney, Brandon M. Stewart, and Barbara E. Engelhardt. 2017.  How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility.  _CoRR_ abs/1710.11214 (2017).  arXiv:1710.11214 http://arxiv.org/abs/1710.11214
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016\.  Wide and Deep Learning for Recommender Systems.  arXiv:1606.07792 [cs.LG]
  * Dielman et al. (1994) Terry Dielman, Cynthia Lowry, and Roger Pfaffenberger. 1994\.  A comparison of quantile estimators.  _Communications in Statistics - Simulation and Computation_ 23, 2 (Jan. 1994), 355–371.  https://doi.org/10.1080/03610919408813175
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning.  arXiv:1506.02142 [stat.ML]
  * Guo et al. (2020) Dalin Guo, Sofia Ira Ktena, Ferenc Huszar, Pranay Kumar Myana, Wenzhe Shi, and Alykhan Tejani. 2020.  Deep Bayesian Bandits: Exploring in Online Personalized Recommendations.  arXiv:2008.00727 [cs.LG]
  * Li et al. (2010) Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. 2010\.  A contextual-bandit approach to personalized news article recommendation.  _Proceedings of the 19th international conference on World wide web - WWW ’10_ (2010).  https://doi.org/10.1145/1772690.1772758
  * Polatidis and Georgiadis (2013) Nikolaos Polatidis and C. Georgiadis. 2013.  Recommender Systems: The Importance of Personalization in E-Business Environments.  _Int. J. E Entrepreneurship Innov._ 4 (2013), 32–46.
  * Pu et al. (2011) Pearl Pu, Li Chen, and Rong Hu. 2011.  A user-centric evaluation framework for recommender systems. In _Proceedings of the fifth ACM conference on Recommender systems - RecSys '11_. ACM Press.  https://doi.org/10.1145/2043932.2043962
  * Rendle (2012) Steffen Rendle. 2012\.  Factorization Machines with libFM.  _ACM Transactions on Intelligent Systems and Technology_ 3, 3 (May 2012), 1–22.  https://doi.org/10.1145/2168752.2168771
  * Research ([n.d.]) Grand View Research. [n.d.].  Digital Marketing Software Market Size, Share and Trends Analysis Report By Solution (Content Management, Campaign Management), By Service, By Deployment, By Enterprise Size, By End-use, By Region, And Segment Forecasts, 2020 - 2027.  Available at https://www.grandviewresearch.com/industry-analysis/digital-marketing-software-dms-market (2021/05/22).
  * Riquelme et al. (2018) Carlos Riquelme, George Tucker, and Jasper Snoek. 2018\.  Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling.  arXiv:1802.09127 [stat.ML]
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019\.  Deep Learning Based Recommender System.  _Comput. Surveys_ 52, 1 (Feb. 2019), 1–38.  https://doi.org/10.1145/3285029

## Appendix: Online Resources

The code used in this paper is publicly available at
https://github.com/fellowship/deep-and-wide-bandit.

The dataset used can be found at https://www.kaggle.com/
alekseykocherzhenko/deep-with-embeddings-vs-deep-and-wide.


# MuMMER: Socially Intelligent Human-Robot Interaction in Public Spaces

Mary Ellen Foster,1 Bart Craenen,1 Amol Deshmukh,1
Oliver Lemon,2 Emanuele Bastianelli,2 Christian Dondrup,2 Ioannis
Papaioannou,2 Andrea Vanzo,2
Jean-Marc Odobez,3 Olivier Canévet,3 Yuanzhouhan Cao,3 Weipeng He,3
Angel Martínez-González,3 Petr Motlicek,3 Rémy Siegfried,3
Rachid Alami,4 Kathleen Belhassein,4 Guilhem Buisan,4 Aurélie Clodic,4
Amandine Mayima,4
Yoan Sallami,4 Guillaume Sarthou,4 Phani-Teja Singamaneni,4 Jules Waldhart,4
Alexandre Mazel,5 Maxime Caniot,5
Marketta Niemelä,6 Päivi Heikkilä,6 Hanna Lammi,6 Antti Tammela6
1 University of Glasgow, Glasgow, UK 2 Heriot-Watt University, Edinburgh, UK
3 Idiap Research Institute, Martigny, Switzerland 4 LAAS-CNRS, Toulouse,
France
5 SoftBank Robotics Europe, Paris, France 6 VTT Technical Research Centre of
Finland, Tampere, Finland
<EMAIL_ADDRESS>
###### Abstract

In the EU-funded MuMMER project, we have developed a social robot designed to
interact naturally and flexibly with users in public spaces such as a shopping
mall. We present the latest version of the robot system developed during the
project. This system encompasses audio-visual sensing, social signal
processing, conversational interaction, perspective taking, geometric
reasoning, and motion planning. It successfully combines all these components
in an overarching framework using the Robot Operating System (ROS) and has
been deployed to a shopping mall in Finland interacting with customers. In
this paper, we describe the system components, their interplay, and the
resulting robot behaviours and scenarios provided at the shopping mall.

## Introduction

In the EU-funded MuMMER project (http://mummer-project.eu/), we have developed
a socially intelligent interactive robot designed to interact with the general
public in open spaces, using SoftBank Robotics’ Pepper humanoid robot as the
primary platform (?). The MuMMER system provides an entertaining and engaging
experience to enrich a human-robot interaction. Crucially, our robot exhibits
behaviour that is socially appropriate and engaging by combining speech-based
conversational interaction with non-verbal communication, and motion planning.
To support this behaviour, we have developed and integrated new methods from
audiovisual scene processing, social-signal processing, conversational AI,
perspective taking, and geometric reasoning.

Figure 1: The MuMMER robot system interacting with a customer in the Ideapark
shopping mall, September 2018.

The primary MuMMER deployment location is Ideapark, a large shopping mall in
Lempäälä, Finland. The MuMMER robot system has been taken to the shopping mall
several times for short-term co-design activities with the mall customers and
retailers (?; ?); the full robot system has been deployed for short periods in
the mall in September 2018 (Figure 1), May 2019, and June 2019, and has been
installed for a long-term, three-month deployment as of September 2019.

The demo system supports a range of behaviours covering a variety of
functional and entertainment tasks that are appropriate for a shopping-mall
setting, including guidance to various locations within the mall, small-talk,
and playing quiz games with customers. The activities during the deployment
have included a number of data collection studies with real users: recording
of customer interaction with the robot in guidance situations, sound
localisation and automatic speech recognition in the noisy mall environment,
and tests for AI-based conversation and localisation and navigation based on a
partial 3D model of the mall and a complete semantic model.

In the remainder of this paper, we outline the technical contributions in each
of the main MuMMER component areas: audiovisual sensing, social signal
processing, conversational interaction, human-aware robot motion planning,
knowledge representation and decision. At the end, we describe the details of
the deployed robot system.

## Audio-visual sensing

For MuMMER, the main task of audio-visual perception is sensing people in
general – that is, maintaining a representation of the persons around the
robot, with a dedicated attention to people susceptible of interacting with
it, or those who are (or have been) interacting with it. This requires several
audio-visual algorithms to detect, track, re-identify people, and detect their
non-verbal behaviors and activities, and also predict their
position/behaviours even when they are not seen. At the same time, the
representation of people needs to be defined and shared with other modules
which are responsible for inferring other knowledge about people (for
instance, to define a person’s goal in the interaction).

Figure 2: The perception systems tracks and re-identifies people leaving the
field, and extracts other features: speaker turns, head pose, visual focus of
attention and nods.

For visual tracking, we first detect the person with the convolutional pose
machines (CPM) (?) which provide accurate locations of the body joints (nose,
eyes, shoulders, etc.). The output of the algorithm is almost perfect when
people are in the foreground of the image and up to 3 of 5 meters, depending
on the resolution. This is our use-case definition of an entertainment robot
in a shopping mall. On top of the CPM, we use
OpenHeadPose111https://gitlab.idiap.ch/software/openheadpose (?), which makes
use of the heap maps of the CPM to estimate the head pose of the person. Then,
we perform head pose tracking (?) to maintain a consistent identity across
adjacent frames (Figure 2). The head pose tracker is represented by a particle
filter mainly based on color and face cues. As faces are tracked, we store
OpenFace features (?) which are computed on an aligned face. When a new
tracklet is created, the OpenFace features of the new tracklet are compared
with the features previously accumulated, and the new tracklet is re-assigned
the identity of the one which had the more votes.

For sound localization , we use a multi-task neural network (NN) which jointly
performs speech/non-speech detection and sound source localisation (?; ?)
applied on top of the 4-channel microphone array (embedded on the robot). The
NN uses as an input a 4-channel audio transformed into a frequency domain, and
it outputs the likelihood values for the two tasks. Thanks to a semi-automated
and synthetic data collection procedure taking advantage of the robotic
platform as well as the use of a weak supervision learning approach, it is
possible to quickly collect data to learn the models for a new sensor (?). The
fusion between the visual and audio parts is done by assigning the detected
speech to the person who is standing in the given direction.

Finally, although a close range (up to 1.2m) gaze sensing module is available
and can be applied for one selected person using a self-calibrated approach
(?), as a compromise between computation and robustness, we instead compute
the visual focus of attention of each person based on the head pose (?). The
algorithm can reliably estimate the object the person is looking at (either
the robot, the other persons, the targets, the shops, or the tablet embedded
at the robot) which is a preliminary step to identify the addressee, and is
also used in the context of perspective taking to determine whether the human
has looked in the direction where the robot pointed (?).

## Social signal processing

Figure 3: Social state estimator visualiser output, displaying all relevant information for fine-tuning. |  |
---|---|---
|  |
|  |
|  |
Figure 4: Gesture variations. Each row shows the same gesture using different
parameters.

For Social Signal Processing, we focus on two primary tasks: fusing the
provided audio-visual sensing data for social state estimation, and
synthesising appropriate social signals for the robot to use when
communicating with users. While detecting, tracking, (re)identifying users, as
well as detecting their primary non-verbal behaviours and activities provide
the basic signals, the multi-modal fusion of these signals allows for a more
accurate and deeper understanding of the underlying social state, including
gaining personality impressions from the user. The estimated social state is
then made available to inform planning of the robot’s subsequent actions; who
and how to converse with the users of the robot; and, how the robot is to move
and behave (gestures) in the presence of the user(s).

### Social state estimation

On the fusion side, the main function of the social state estimator is to
determine which user the robot should initiate interaction with. We used the
underlying assumption that the robot should initiate interaction with the user
perceived to be the most willing to interact; which we took to be the user
paying the most attention to the robot. We assume that the user paying the
most attention to the robot is the user that is looking most directly at the
robot, and who is most closely situated near the robot.

To this end, the social state estimator aggregates audio-visual sensing data
about the head pose of users, whether the users are looking at the robot
and/or the screen on the robot, and the distance between the users’ head and
the robot. The head pose data of the users are used to calculate the
(Euclidean) distance between the head pose of the users and three centroids
derived from clustering/classifying previously recorded lab and deployment
data. This distance is then normalised to a value between zero and one, and
used as a probability. The distances between the users and the robot are used
as a penalty, and normalised between zero and one as a probability in such a
way that users further away from the robot are penalised more than users
closer to the robot.

This results in four probabilities, two taken directly from the audio-visual
sensing data, and two derived from it. These four probabilities are then fused
into one _attention_ probability by calculating their (weighted) average.
Choosing which user the robot should interact with is done by comparing the
attention probability against a configurable minimum attention probability
threshold, and then selecting the user with the highest attention probability.

To prevent immediately re-initiating an interaction with a user that the robot
has just interacted with, the social state estimator also monitors the actions
of the planning and dialogue components. The social state estimator then
maintains a list of the users that the robot is, and has been, interacting
with, and applies a penalty to their attention probability while they are
interacting and for a short time afterwards.

The social state estimator is fully configurable by a set of parameters, with
the initial parameter settings determined from extensive recorded lab data.
The parameters were further fine-tuned during deployment to provide the most
accurate and applicable social state estimates. To facilitate fine-tuning the
parameters of social state estimator, a visualiser is provided to display all
relevant features (Figure 3).

### Social signal generation

For the synthesis side, a repertoire of non-verbal social signals, including
gestures and sounds, has been developed for the robot, available to be used in
conjunction with moving and interacting with the users. The non-verbal
behaviour of an embodied agent is at least as communicative as its verbal
behaviour (?), and in a noisy environment such as a mall it may even be more
important, so understanding and controlling the robot’s non-verbal signals is
crucial. Examples of some robot gesture variants are shown in Figure 4.

In a series of perception experiments, we have examined how manipulating
gesture parameters affect users’ subjective responses to the robot as well as
their perception of the robot’s personality. These studies have found several
clear relationships: for example, manipulating the amplitude and speed
significantly affected users’ perception of the Extraversion and Neuroticism
of the robot, while the attributed personality also affected users’ subjective
reactions to the robot (?). In addition, it was found that while the majority
of users preferred a robot that they perceived to have a similar personality
to their own, a significant minority preferred a robot whose personality was
perceived to be different than their own (?).

We are currently integrating a finer-grained method of gesture control based
on sentiment (?), as well as a set of affectively generated artificial sounds
(?), with the goal of further enhancing the robot’s expressiveness.

## Conversational interaction

The MuMMER system focuses on enabling an agent to combine a task-based
dialogue system with chat-style open-domain social interaction, to fulfil the
required tasks while at the same time being natural, entertaining, and
engaging to interact with. The presented work is based on the “Alana”
conversational framework, a finalist of the Amazon Alexa Challenge in both
2017 (?) and 2018 (?). Alana was initially developed for the Amazon Echo as an
open-domain social chatbot. For the needs of this project, Alana acts as the
core module for every dialogue interaction with the user from every other
module. This means that whenever a module requires to either verbally notify
the user or get the user’s feedback, Alana will handle this task. In this way
the conversation throughout the interaction will be more contextually
relevant, and easier to maintain. Since the robot needs to engage in social
dialogue as well as to complete tasks, Alana was enriched with so-called task
bots to conversationally execute and monitor behaviours on a physical agent
(Figure 5) (?).

In order to enable the functionality described above, a new Natural Language
Understanding (NLU) module, HERMIT NLU, has been implemented and integrated
into the Alana system, which is able to deal with social chit-chat but also
extract the necessary information from commands to start tasks. HERMIT NLU (?)
is thus used to decide if the task bot is triggered and to extract the
required parameters for tasks, such as the name of the shop someone is looking
for. While standard chatbots mostly rely on NLU that works on shallow semantic
representations (e.g., intents + slots), task-based applications require
richer characterisations. In line with (?), we promote the idea that the
user’s intent can be represented through the combination of existing theories,
capturing different dimensions of the overall problem, namely Dialogue Acts
and Frame Semantics. Existing approaches to NLU for dialogue systems are based
on formal languages designed around the targeted domain. However, it has been
widely demonstrated that the generalisation capability of statistical-based
approaches is more robust towards lexical and domain variability (?). We thus
use a deep learning architecture based on a hierarchy of self-attention
mechanisms and BiLSTM encoders followed by CRF tagging layers to perform
multi-task learning over the aforementioned semantic dimensions (?). The
system effectively learns how to predict Dialogue Acts, Frames, and Frame
Elements in a sequence labelling scheme, starting from a corpus of annotated
sentences which we are currently developing.

Figure 5: Architecture of the Dialogue system. The blue parts on the left
represent the task management and execution system, the green parts on the
right represent Alana as the dialogue system. The Bot Ensemble contains social
chat bots and the task bot which is able to trigger tasks and handle
communication between the task and the user. The yellow middle part is the
task specific dialogue management system (Arbiter), Text-To-Speech (TTS) and
Automatic Speech Recognition (ASR).

After a task has been identified, executing it on a robot usually includes
physical actions that require a finite amount of time to complete and are not
instantaneous such as dialogue actions. While the robot is executing such an
action, the user might want to continue the conversation, or give new
instructions. In order to be able to support such a multi-threaded dialogue
management of interleaving tasks with general chitchat and other tasks, we
build on the ideas presented in (?). To this end, the execution system
introduced in (?) has been extended to use so-called recipes that define
dialogue and physical actions to execute in order to achieve the given goal
(?). The execution framework described therein has been redesigned to support
multi-threaded execution and an arbitration process has been put in place to
manage the currently running tasks on the execution side and in the Alana
system. This lets tasks be started, stopped, and paused at any time, with
appropriate feedback to the user. If a task has been suspended by another
action, it will be resumed after the new action finishes and any open
questions will be re-raised to prompt the user.

## Route guidance supervision

One of the core tasks for the MuMMER robot in the mall is the guiding of users
to specific locations in the mall, by pointing at places and explaining the
route to the wanted location. The task is triggered when a human asks for a
location. A supervision system, based on Jason (?), a BDI agent-oriented
framework handles the execution through Jason reactive plans. Throughout the
task, the robot supervises the execution and, depending what goes wrong, the
robot has multiple possible responses. For example, it is able to handle
nominal scenarios of route guidance while being able to take into account
contingencies such as the human lack of visibility of the direction, his/her
ability or not to take the stairs, his/her understanding of the message, etc.
Finally, if at some point, the human is not perceived during a certain time,
the robot ends the task, assuming that the human has left.

### Route computing and route verbalization

The entire description of the route, from the search for the best route to get
to the final destination to the verbalization of this route, is based on the
SSR (Semantic Spatial Representation) (?). This representation is used to
describe the topology of an indoor environment as well as semantic information
(type of stores or items sold by stores) in a single ontology. This ontology
is managed by Ontologenius (?), a lightweight open-source ROS-compatible
package which stores semantic knowledge, reasons with it, and shares that
information to all the other system components.

### Geometric reasoning

Geometric reasoning uses Underworlds (?; ?), a lightweight framework for
_cascading spatio-temporal situation assessment_ in robotics. It represents
the environment as real-time distributed data structures, containing scene
graph (for representation of 3D geometries). Underworlds supports _cascading_
representations: the environment is viewed as a set of _worlds_ that can each
have different spatial granularities, and may inherit from each other. It also
provides a set of high-level client libraries and tools to introspect and
manipulate the environment models. Based on a 3D model of the mall (Figure 6),
it maintains what the robot knows about the scene as well as alternative world
states. These states represent the estimation of the human’s beliefs about the
scene It also provides the symbolic relations among entities with stamped
predicates (e.g. [$isInsideArea(person,area)$] or [$isSpeakingTo(X,Y)$] when
$X$ speaks and looks at $Y$ (given by perception)).

Figure 6: Visualization of the visibility grids of a landmark on the 3D model
of the central square of the Ideapark shopping center.

### Motion planning

The navigation of the robot is implemented using the ROS navigation stack,
with navfn as the global planner and a Timed Elastic Band (TEB) (?) planner as
the local planner. For MuMMER, the local planner was modified in order to
accommodate humans into planning inspired from (?), resulting a new planner
called and this new planner called Social TEB (S-TEB). This algorithm is able
to plan and execute trajectories while ensuring satisfaction of robot
kinematics constraints, avoiding static and moving non-human obstacles and
planning navigation solutions respecting social constraints with humans
perceived. The planner ensures the safety of humans by re-planning a local
plan at each control loop.

### SVP planner

Although the target robot location in the mall is in a large square, several
elements of the environment can block the visibility of important landmarks
for the proper understanding of the route to take. The purpose of the SVP
(Shared Visual Perspective) planner (?) is therefore to try to find a position
where the human will have to go in order to observe an element of the
environment such as a passage, a staircase or a store. To do this, a
visibility grid is computed for each possible landmark, as shown on figure 6.
Having determine a good position for the human, the planer also allows to
determine the good position for the robot so as to have a human-robot-landmark
conformation allowing both to point the landmark and to look at the human.

## Deployment

A long-term deployment (three months from September 2019) will allow the study
of the customer behaviours around a helpful and entertaining robot over an
extended period of time. This section gives details on how the hardware and
software that are being employed in the final deployment, as well as the
scenarios that are supported.

### Setup

The fully integrated MuMMER system consists of several hardware components to
allow the computation to be performed on the appropriate platforms.

The robot we are using is an updated, custom version of the Pepper platform,
which is equipped with an Intel D435 camera and a NVIDIA Jetson TX2 in
addition to the traditional sensors that are found on the previous versions of
the robot. We use the Robot Operating System (ROS) to enable the communication
between the processing nodes. All the streams (audio, video, robot states) are
sent to a remote laptop which performs all the computation. The laptop has a
NVIDIA RTX 2080 graphics card (for the deep learning part) and 12 CPU cores.
The perception algorithms process the Intel images at a resolution of
$320\times 180$ for the detection and tracking parts, and at a resolution of
$640\times 360$ for the re-identification part, which enables fast tracking
and a good re-identification quality with OpenFace. The 4 microphone streams
are processed at a frequency of $16000$ Hz, and the full perception system
delivers the output at 10 fps.

To transcribe the user’s speech signal we use the Google Automatic Speech
Recognition (ASR) API222https://cloud.google.com/speech-to-text/ which
receives an enhanced audio signal from a delay and sum beamformer based on the
location of the speaking person determined by the audio-visual sensing. A
dedicated ROS node streams the audio to the ASR which in real-time returns an
incrementally updated string transcribing the utterance. Using silence to mark
the end of speech, this transcription is enhanced using the context of the
sentence to provide for a more coherent result. Finally, the text output of
the Google ASR is sent to the Alana framework to perform the dialogue task,
through the arbitration module as explained above.

The system is deployed in two languages, English and Finnish, though due to
the vast linguistic differences between the two languages, the two versions
have been kept separated, and the whole interaction can either be in one or
the other. Due to the complexity of the NLU module in the Finnish version of
the system, the user’s utterance is being translated into English using Google
Translate API333https://cloud.google.com/translate/. The result of this
translation is sent to the Alana conversational framework and goes through the
NLU pipeline described in detail in (?). In the English version of the system,
Alana then returns the reply to be verbalised. Due to the relatively poor
performance of Google translate when it comes to translating English into
Finnish (as remarked upon by our Finnish partners), the Finnish version of
Alana has a much reduced set of bots in its ensemble (see Figure 5). These
bots mainly return answers based on templates that have been translated into
Finnish beforehand.

### Scenarios deployed

As a proof of concept, a real-time autonomous system has been built to
integrate all the components described in the sections above. The following
types of interactions can be triggered by the user:

Chat

The staple of the interaction is social dialogue (?). During all other modes
of interaction, the user can always default to simply chat to the robot
irrespective of whether it is currently executing a goal/task (e.g. the user
requires guidance to a specific shop) or not. For example the user might
approach the robot and start discussing various topics. At specific points
throughout this conversation, the system might explain its capabilities to the
user in order to recover from a conversational stalemate or to simply make
them aware of the fact that it can also be helpful in finding your way around
the mall (see below).

Quiz

In this scenario, multiple choice questions are asked by the robot, and the
human replies by stating the number of the answer they think is correct.

Route Description (Dialogue only)

When the human asks how to get to a specific shop, the robot gives him/her the
route description. In this most “simple” form, the system uses only verbal
interaction. This means that especially the route description is merely
presented as a string of synthesised text.

Route Guidance (Dialogue + Pointing)

In this version, the robot guides the human to specific locations in the mall,
by pointing at places and explaining the route to the wanted location. To do
so, the robot first computes positions so that the human will be able to see
what the robot is pointing for him/her. Then, the robot navigates to its
position (this part is optional), expecting that the human will join it once
it stopped and checks the human’s visibility. Then, the robot explains to the
human how to reach the destination. According to a human-human guidance study
(?), the robot points, first at the location direction and then points at the
access point (a corridor, stairs or an escalator) to go through to reach the
location. While pointing, the robot verbalizes the route. Finally, the robot
checks that the human knows how to reach the goal and leaves open the
possibility to repeat if needed. All along the task, the robot supervises the
task and adapts accordingly.

All these modes of interaction can be interleaved at the user’s discretion.
This means, for example, that during the quiz the user could revert to social
dialogue. If they do so, the system might occasionally try to bring them back
to the quiz by re-raising the last question. The same holds true if the person
chooses to abandon a route guidance task before it was finished.

## Conclusions

The MuMMER project has built a fully autonomous entertainment robot to perform
HRI scenarios in a shopping mall, in which the main goal is to have
entertainment interaction (quiz, chat), as well as route guidance. The system
is real-time, by leveraging the heavy deep learning computation on a remote
laptop, the ASR on the Google platform, and the Alana conversational AI system
on a remote server. This system enables a natural interaction with the
participants; it has been tested and was tested in real conditions for several
short sessions, and as of September 2019 is fully deployed for a three-month
long-term user study.

Further work for large scale deployment could include some software
optimizations to run more components on the robot itself, and to reduce the
lag which sometime exists between the human speech and the robot reply.

## Acknowledgements

This research has been partially funded by the European Union’s Horizon 2020
research and innovation program under grant agreement no. 688147 (MuMMER,
http://mummer-project.eu/).

## References

  * [Amos, Ludwiczuk, and Satyanarayanan 2016] Amos, B.; Ludwiczuk, B.; and Satyanarayanan, M.  2016\.  Openface: A general-purpose face recognition library with mobile applications.  Technical report, CMU-CS-16-118, CMU School of Computer Science.
  * [Bastianelli et al. 2016] Bastianelli, E.; Croce, D.; Vanzo, A.; Basili, R.; and Nardi, D.  2016\.  A discriminative approach to grounded spoken language understanding in interactive robotics.  In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, IJCAI’16, 2747–2753.
  * [Belhassein et al. 2017] Belhassein, K.; Clodic, A.; Cochet, H.; Niemelä, M.; Heikkilä, P.; Lammi, H.; and Tammela, A.  2017\.  Human-Human Guidance Study.  Technical Report 17596, LAAS.
  * [Bordini, Hübner, and Wooldridge 2007] Bordini, R. H.; Hübner, J. F.; and Wooldridge, M.  2007\.  Programming Multi-Agent Systems in AgentSpeak Using Jason (Wiley Series in Agent Technology).  USA: John Wiley & Sons, Inc.
  * [Cao et al. 2017] Cao, Z.; Simon, T.; Wei, S.-E.; and Sheikh, Y.  2017\.  Realtime multi-person 2d pose estimation using part affinity fields.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 7291–7299.
  * [Cao, Canévet, and Odobez 2018] Cao, Y.; Canévet, O.; and Odobez, J.-M.  2018\.  Leveraging convolutional pose machines for fast and accurate head pose estimation.  In IEEE/RSJ International Conference on Intelligent Robots and Systems.
  * [Craenen et al. 2018a] Craenen, B.; Deshmukh, A.; Foster, M. E.; and Vinciarelli, A.  2018a.  Do we really like robots that match our personality? the case of big-five traits, godspeed scores and robotic gestures.  In Proceedings of the 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).
  * [Craenen et al. 2018b] Craenen, B. G.; Deshmukh, A.; Foster, M. E.; and Vinciarelli, A.  2018b.  Shaping gestures to shape personalities: The relationship between gesture parameters, attributed personality traits, and Godspeed scores.  In Proceedings of the 27th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN), 699–704.
  * [Curry et al. 2018] Curry, A. C.; Papaioannou, I.; Suglia, A.; Agarwal, S.; Shalyminov, I.; Xu, X.; Dušek, O.; Eshghi, A.; Konstas, I.; Rieser, V.; et al.  2018\.  Alana v2: Entertaining and informative open-domain social dialogue using ontologies and entity linking.  Alexa Prize Proceedings.
  * [Deshmukh, Foster, and Mazel 2019] Deshmukh, A.; Foster, M. E.; and Mazel, A.  2019\.  Contextual non-verbal behaviour generation for humanoid robot using text sentiment.  In Proceedings of the 28th IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN).
  * [Dinarelli et al. 2009] Dinarelli, M.; Quarteroni, S.; Tonelli, S.; Moschitti, A.; and Riccardi, G.  2009\.  Annotating spoken dialogs: From speech segments to dialog acts and frame semantics.  In Proceedings of SRSL 2009, the 2nd Workshop on Semantic Representation of Spoken Language, 34–41.
  * [Dondrup et al. 2017] Dondrup, C.; Papaioannou, I.; Novikova, J.; and Lemon, O.  2017\.  Introducing a ROS based planning and execution framework for human-robot interaction.  In Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents, ISIAA 2017, 27–28.
  * [Foster et al. 2016] Foster, M. E.; Alami, R.; Gestranius, O.; Lemon, O.; Niemelä, M.; Odobez, J.-M.; and Pandey, A. K.  2016\.  The MuMMER project: Engaging human-robot interaction in real-world public spaces.  In Social Robotics, 753–763.  Cham: Springer International Publishing.
  * [Hastie et al. 2016] Hastie, H.; Dente, P.; Küster, D.; and Kappas, A.  2016\.  Sound emblems for affective multimodal output of a robotic tutor: A perception study.  In Proceedings of the 18th ACM International Conference on Multimodal Interaction, 256–260.
  * [He, Motlicek, and Odobez 2018a] He, W.; Motlicek, P.; and Odobez, J.  2018a.  Deep neural networks for multiple speaker detection and localization.  In 2018 IEEE International Conference on Robotics and Automation (ICRA), 74–79.
  * [He, Motlicek, and Odobez 2018b] He, W.; Motlicek, P.; and Odobez, J.-M.  2018b.  Joint localization and classification of multiple sound sources using a multi-task neural network.  In Proceedings of Interspeech 2018, 312–316.
  * [He, Motlicek, and Odobez 2019] He, W.; Motlicek, P.; and Odobez, J.  2019\.  Adaptation of multiple sound source localization neural networks with weak supervision and domain-adversarial training.  In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 770–774.
  * [Heikkilä et al. 2019] Heikkilä, P.; Niemelä, M.; Belhassein, K.; Sarthou, G.; Tammela, A.; Clodic, A.; and Alami, R.  2019\.  Should a robot guide like a human? a qualitative four-phase study of a shopping mall robot.  In International Conference on Social Robotics (ICSR).
  * [Heikkilä, Lammi, and Belhassein 2018] Heikkilä, P.; Lammi, H.; and Belhassein, K.  2018\.  Where can I find a pharmacy? - human-driven design of a service robot’s guidance behaviour.  In Proceedings of PubRob 2018.
  * [Khalidov and Odobez 2017] Khalidov, V., and Odobez, J.-M.  2017\.  Real-time multiple head tracking using texture and colour cues.  Idiap-RR Idiap-RR-02-2017, Idiap.
  * [Khambhaita and Alami 2017] Khambhaita, H., and Alami, R.  2017\.  Viewing Robot Navigation in Human Environment as a Cooperative Activity.  In International Symposium on Robotics Research (ISSR 2017), 18p.
  * [Lemaignan et al. 2018] Lemaignan, S.; Sallami, Y.; Wallbridge, C.; Clodic, A.; Belpaeme, T.; and Alami, R.  2018\.  UNDERWORLDS: Cascading Situation Assessment for Robots.  In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems.
  * [Lemon et al. 2002] Lemon, O.; Gruenstein, A.; Battle, A.; and Peters, S.  2002\.  Multi-tasking and collaborative activities in dialogue systems.  In Proceedings of the 3rd SIGdial Workshop on Discourse and Dialogue - Volume 2, 113–124.
  * [Papaioannou et al. 2017] Papaioannou, I.; Cercas Curry, A.; Part, J.; Shalyminov, I.; Xinnuo, X.; Yu, Y.; Dusek, O.; Rieser, V.; and Lemon, O.  2017\.  Alana: Social dialogue using an ensemble model and a ranker trained on user feedback.  In 2017 Alexa Prize Proceedings.
  * [Papaioannou, Dondrup, and Lemon 2018] Papaioannou, I.; Dondrup, C.; and Lemon, O.  2018\.  Human-robot interaction requires more than slot filling-multi-threaded dialogue for collaborative tasks and social conversation.  In FAIM/ISCA Workshop on Artificial Intelligence for Multimodal Human Robot Interaction, 61–64.
  * [Rastogi, Gupta, and Hakkani-Tur 2018] Rastogi, A.; Gupta, R.; and Hakkani-Tur, D.  2018\.  Multi-task learning for joint language understanding and dialogue state tracking.  In Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, 376–384.
  * [Rösmann, Hoffmann, and Bertram 2017] Rösmann, C.; Hoffmann, F.; and Bertram, T.  2017\.  Integrated online trajectory planning and optimization in distinctive topologies.  Robotics and Autonomous Systems 88:142–153.
  * [Sallami et al. 2019] Sallami, Y.; Lemaignan, S.; Clodic, A.; and Alami, R.  2019\.  Simulation-based physics reasoning for consistent scene estimation in an hri context.  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2019).  To appear.
  * [Sarthou, Alami, and Clodic 2019a] Sarthou, G.; Alami, R.; and Clodic, A.  2019a.  Ontologenius : A long-term semantic memory for robotic agents.  In RO-MAN 2019.
  * [Sarthou, Alami, and Clodic 2019b] Sarthou, G.; Alami, R.; and Clodic, A.  2019b.  Semantic Spatial Representation: a unique representation of an environment based on an ontology for robotic applications.  In SpLU-RoboNLP 2019, 50 – 60.
  * [Sheikhi and Odobez 2015] Sheikhi, S., and Odobez, J.  2015\.  Combining dynamic head pose and gaze mapping with the robot conversational sftate or attention recognition in human-robot interactions.  Pattern Recognition Letters 66:81–90.
  * [Siegfried, Yu, and Odobez 2017] Siegfried, R.; Yu, Y.; and Odobez, J.-M.  2017\.  Towards the use of social interaction conventions as prior for gaze model adaptation.  In 19th ACM International Conference on Multimodal Interaction (ICMI).
  * [Vanzo, Bastianelli, and Lemon 2019] Vanzo, A.; Bastianelli, E.; and Lemon, O.  2019\.  Hierarchical multi-task natural language understanding for cross-domain conversational ai: HERMIT NLU.  In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, to appear.  Stockholm, Sweden: Association for Computational Linguistics.
  * [Vinciarelli, Pantic, and Bourlard 2009] Vinciarelli, A.; Pantic, M.; and Bourlard, H.  2009\.  Social signal processing: Survey of an emerging domain.  Image and Vision Computing 27(12):1743–1759.
  * [Waldhart, Clodic, and Alami 2019] Waldhart, J.; Clodic, A.; and Alami, R.  2019\.  Reasoning on Shared Visual Perspective to Improve Route Directions.  In 2019 28th IEEE International Conference on Robot & Human Interactive Communication.