# Learning on the Job: Long-Term Behavioural Adaptation in Human-Robot
Interactions

Francesco Del Duchetto and Marc Hanheide This work was supported by the
Lincolnshire County Council.Authors are with Lincoln Center for Autonomous
Systems (L-CAS), School of Computer Science, University of Lincoln, United
Kingdom. Emails: {fdelduchetto<EMAIL_ADDRESS>
###### Abstract

In this work, we propose a framework for allowing autonomous robots deployed
for extended periods of time in public spaces to adapt their own behaviour
online from user interactions. The robot behaviour planning is embedded in a
Reinforcement Learning (RL) framework, where the objective is maximising the
level of overall user engagement during the interactions. We use the Upper-
Confidence-Bound Value-Iteration (UCBVI) algorithm, which gives a helpful way
of managing the exploration-exploitation trade-off for real-time interactions.
An engagement model trained end-to-end generates the reward function in real-
time during policy execution. We test this approach in a public museum in
Lincoln (UK), where the robot is deployed as a tour guide for the visitors.
Results show that after a couple of months of exploration, the robot policy
learned to maintain the engagement of users for longer, with an increase of
22.8% over the initial static policy in the number of items visited during the
tour and a 30% increase in the probability of completing the tour. This work
is a promising step toward behavioural adaptation in long-term scenarios for
robotics applications in social settings.

##  I INTRODUCTION

The ability to maintain user engagement during an interaction is an essential
ability for a robot designed to be deployed in a social scenario. In order to
do so, a robot should be able to assess the users’ state and to learn from
experience how the actions it takes affect that same users’ state and the
ongoing interaction. Reinforcement Learning (RL) techniques are a special case
of Machine Learning algorithms that deals exactly with those scenarios where
the “goodness” of the actions an agent can take is not known in advance, and
exploration is required. The goal is to find the best sequence of actions to
maximise a certain objective, which is manifested through rewards. However
difficult this may seem for scenarios where the goal is well defined, it
becomes even more challenging for social scenarios where the objective is
expressed by the users’ internal state and can only be estimated from sensors
or proxy variables (like the duration of the interaction).

In this paper, we approach this problem by enabling our robot Lindsey to
estimate the users’ engagement during the interaction and by allowing it to
learn, through RL, the actions to maximise such engagement. We build on our
previous work [1] where we described and analysed the long-term deployment,
which is still ongoing with the present work, of our robot in a public museum
where it serves as a tour guide to the visitors. For detecting the users’
engagement we use our regression model, proposed in [2], which provides a
single scalar engagement from standard video streams obtained from the point
of view of the interacting robot. Figure 1 shows Lindsey the robot in the
museum while interacting with the users and depicts the concept for our
proposed learning framework.

By attempting to learn in a long-term scenario using only the users’
engagement as guidance, we provide a proof-of-concept for a framework to
enable behavioural adaptation in social robots. Given that the learning is
guided by the users’ state, manifested by their expressed engagement, this
will allow the robot to adapt to the users’ preferences, which cannot be known
or programmed in advance. Finally, with our proposed framework we set out to
test the hypotheses of whether _the engagement value estimated during the
interactions a sufficient feedback signal for driving the in-situ learning of
the robot social behaviours_ ; and if _it leads to a more sustained
interaction with the users_.

Figure 1: A picture of Lindsey in the archaeological gallery of The Collection
museum during a tour guide interaction. The robot learning framework detects
user engagement online and optimises its interaction policy for maximising
such overall engagement using Reinforcement Learning.

##  II RELATED WORK

###  II-A Long-Term Deployment in Public Spaces

Previous work have addressed the issues of deploying robots in public
environments for long periods of time taking into account the necessity of
interacting with people. Past long-term deployments in public spaces [3, 4]
have identified the interaction with humans a necessity for recovering for
failures and performing tasks that the robot was not able to. Similarly, in a
deployment with the SCITOS G5 robot, that travelled more than 160 km within
the STRANDS project [5], the authors report the need for a way to manage
failures and to have a better understanding of human activities. Hanheide et
al. [6] propose a spatio-temporal model to learn when, where and how users
interacted with the robot info-terminal during a long-term deployment. They
found they could improve the efficiency and usefulness of the system by
proposing the right content at the right time and place. Building from these
works, we designed a system that is able to interact autonomously in a public
environment for years while learning from the interactions with humans. A
survey on long-term interaction between users and robots [7] raises the issue
that memory and adaptation remains nearly unexplored in the field. Similarly,
Kunze et al. [8] explores the state of the art on Artificial Intelligence (AI)
techniques for long-term autonomy recognising that interactions in this
context can be exploited to improve a robot behaviour. With the learning
algorithm described in this work, we lay the foundations for a robot framework
that allows exploiting the human feedback during interaction in order to
optimise its social abilities.

Previous works featured a robot deployed in a museum environment. The robot
Rhino [9] was deployed in a museum in Germany for 6 days guiding hundreds of
visitors. At the time, the main issues and the focuses of the work were
navigation and obstacle avoidance. The Minerva robot [10] traversed more than
44 km and interacted with more than 50k people. Moreover, it was able to
display mood (i.e. happy or angry) and, more importantly for our work, used an
RL approach to learn the best actions to engage visitors. In [11] four robots
were deployed over five years. Focusing on interactivity and education they
learned that long and non-interactive presentations are guaranteed to drive
audience away.

In the present work, building on our ongoing long-term study of a museum robot
[1], we plan to address some of these issues by enabling the robot with the
ability to directly optimise the users’ engagement during the interaction
through RL.

###  II-B Behavioural Adaptation in Social Settings

Previous work has shown how it is possible incorporate the users feedback,
with a focus here on user engagement, for modifying the robot behaviour or
influencing the user own engagement. Sidner et al. [12] explored how the use
of gazing and gestures affects positively the user perception of the robot,
increasing their engagement. Similarly, Holroyd [13] defines policies with the
goal of increasing user engagement and shows that the robot equipped with
these policies is perceived to be more human-like, to behave more fluently and
that users reciprocate more robot cues. Recent works aimed at learning these
social behaviors typically use (Deep) RL techniques to exploit the real-world
interaction experiences a robot can collect. Qureshi et al. [14, 15] proposed
end-to-end models to teach a robot the most appropriate action for approaching
humans and starting an interaction. The reward signal was triggered by
successful/unsuccessful handshakes. Lathuilière et al. [16] uses Deep RL to
learn a gaze policy from an intrinsic reward function based on the audiovisual
position of people with respect to the robot camera field of view. Gao et al.
[17] learns a robot policy for approaching groups of people by maximising a
group formation score and minimising the displacement of other participants in
the group when the robot approaches. Also in a museum context, Meng et al.
[18] propose an RL approach where they use the users’ engagement during group
interactions with an interactive sculpture as the reward to learn engaging
interactive behaviours.

In this work, we use a state-of-the-art RL approach [19] in order to improve
our robot’s social behavior, in particular its choice of actions during the
guided tours. We employ the human engagement level as a robot internal reward
to maximise.

##  III PRELIMINARIES

###  III-A Lindsey the Robot

Lindsey, the robot used in this long-term study, is a Scitos G5 robot
manufactured by MetraLabs GmbH. In order to sense the environment, the robot
has a laser scanner with 270∘ scan angle on its base and an Asus xtion depth
camera mounted on a pan-tilt unit above his head. The interactions with the
visitors are mediated through a touch screen, two speakers, a microphone and a
head with two eyes that can move with five degrees of freedom to provide
human-like expressions. To ensure safe operations in public environments the
robot is equipped with an array of bumpers around the circular base with
sensors to detect collisions and two easily reachable emergency buttons that,
when activated, cuts the power to the motors.

The software framework is based on ROS and uses STRANDS project [5] core
modules for topological navigation, people tracking, task scheduling and data
collection.

###  III-B The Museum Scenario

Figure 2: A representation of the archaeological gallery with markers to
identify the position of items in the robot tours.

The scenario of this project is The Collection
museum111https://www.thecollectionmuseum.com/robot-at-the-collection in
Lincoln, UK. The museum is freely accessible to all members of the public 5
days a week, although it used to be open 7 days a week before the COVID
lockdown, from 10AM to 4PM. Lindsey, the robot, is deployed as a tour guide in
the archaeological section of the museum which displays local findings dating
from the stone age to the early modern age. Working with the museum’s staff
four different guided tours have been devised for the robot. Figure 2 shows
the position of the items visited by the robot during the tours. Different
color markers correspond to items in different tours. Figure 3 depicts the
items shown in each tour.

The robot is free to roam around during idle times in search for people to
interact with. When people enter the gallery and go close to the robot they
are greeted by it. Using the robot’s touchscreen they can choose to start one
of the tours or to being guided to the location of a specific item and
receiving a description of it. For the purpose of this study only the guided
tour interactions are considered. The museum visitors are not instructed how
to behave with the robot and what to expect during the interaction, therefore
the interactions themselves are unstructured with users being free to
interrupt the tours or to just leave at any moment. Moreover, both the robot
and the users move in the environment during the guided tours making the robot
perception of users from the on-board cameras often limited.

Figure 3: The sequence of items in each robot tour.

###  III-C Analysis of the Robot Tours

In a previous analysis of the interactions with our tour guide robot [1] we
report that Lindsey has been very popular among the museum’s visitors,
totalling more than 2300 guided tours in just 103 days of operations. However,
observing the duration of these tours one realises that most interactions are
actually very short and that only about 18% of them are completed by the user.
These observations evidences the need for adaptation and motivate us to study,
with the present work, how the users’ feedback can be taken into account in
such long-term scenario to improve the robot ability to maintain engagement
over time.

##  IV METHODOLOGY

In this study we combine together several methodologies in order to have a
unified framework that allows the robot to explore and learn online without
the need of having separate phases for data collection and learning.

###  IV-A States & Actions Specification

The robot behaviours are specified as conditional plans using the Petri net
plans (PNP) formalism [20]. The formalism facilitates action abstraction a
reusability in the specification of plans, moreover it allows to monitor the
execution and deal with failure situations during execution with _Execution
Rules_ (ER) [21]. Each action used in this study is a conditional sequence of
lower-level actions with ERs for allowing robust execution. More information
about how robustness during autonomous operations is assured can be found in
our previous work [1]. The duration of each action can vary from seconds to
minutes and it is not known in advance, with some requiring navigating some
distance around the museum or describing items with different amount of words.
Moreover, each action is interruptible and can potentially terminate the tour
before the end, for example in case of failure or users stopping the guided
tour.

TABLE I: Action space with constraints over possible action successors. The successors in grey are conditional on that action not being executed already in the tour and those underlined are only valid for the death tour. Id | Action | Action Successors
---|---|---
0 | doNothing | 0,1
1 | describeTour | 2,3,4,5,6,7
2 | gotoExhibit_1 | 8
3 | gotoExhibit_2 | 8
4 | gotoExhibit_3 | 8
5 | gotoExhibit_4 | 8
6 | gotoExhibit_5 | 8
7 | gotoExhibit_6 | 8
8 | describeExhibit |  2,3,4,5,6,7,9,10
9 | describeMoreExhibit |  2,3,4,5,6,7,10
10 | endTour | 0

Figure 4: Examples of continuous engagement predictions from the robot head
camera using model from [2]. Engagement prediction value is LOW in the left
frame (the girl is looking elsewhere), MEDIUM in the center (man is taking a
picture of the robot) and HIGH on the right (man looking at robot screen).
Written consent was obtained from the users for taking and reusing these
pictures.

The scenario requires that certain actions are performed exclusively in a
specific sequence. For example, it would be wrong to start describing an
exhibit’s item before going to its location or describing the theme tour only
at the end of the interaction. Moreover, the gotoExhibit_* actions cannot be
executed more than once because it wouldn’t make sense to bring people to the
same item in the museum multiple times during a tour. To enforce these
constraints at execution time, we created subsets of all possible actions
$C_{s}\subset A$ $\forall$ $s\in S$ from which the policy can choose the next
action. The action space with the associated successive actions constraints is
showed in Table I.

The state vector used for learning is composed as follows

$s=\langle v_{1},v_{2},v_{3},v_{4},v_{5},v_{6},n,a_{p},e,t\rangle$

where $v_{k}\in{0,1}$ for $k=1,...,6$ indicates whether the $k^{th}$ item in
the tour has been visited,
$n\in\\{\texttt{death},\texttt{tools},\texttt{religion},\texttt{art}\\}$ is
the name of the tour, $a_{p}\in A$ is the previous action,
$e\in\\{\texttt{LOW},\texttt{MEDIUM},\texttt{HIGH}\\}$ is the average
engagement level during the execution of $a_{p}$, and
$t\in\\{\texttt{none},\texttt{ended},\texttt{stopped},\texttt{abandoned}\\}$
indicates the terminal state.

###  IV-B Engagement Model

The engagement of users who interact with Lindsey at the museum is detected
using the engagement model presented in [2], which runs in real-time on the
robot’s GPU. The model proposed comprises and end-to-end regression model,
taking the camera feed as input and providing a holistic engagement measure in
the interval [0,1] as output for each image frame. It has been validated on
HRI data sets as an accurate measure of engagement. Figure 4 depicts 3 frames
of a scene during a tour with the continuous engagement signal provided by the
model. For the purpose of this work, we want to use the engagement of users as
an evaluation of the robot behaviours to drive learning. Therefore, in our RL
scenario the reward observed at each state is derived from the engagement
collected during the execution of the last action. The engagement scalar
values are provided by the model at a frequency of about 1 Hz, they are then
averaged over the entire duration of the action and discretised in the
variables LOW, for values in $(0,\frac{1}{3}]$, MEDIUM, for values in
$(\frac{1}{3},\frac{2}{3}]$, or HIGH, for values in $(\frac{2}{3},1]$.
Finally, the reward function, given the state $s$, is

$r(s)\sim\begin{cases}\mathcal{U}(0,\frac{1}{3})&\text{if $e=$ {LOW}},\\\
\mathcal{U}(\frac{1}{3},\frac{2}{3})&\text{if $e=$ {MEDIUM}},\\\
\mathcal{U}(\frac{2}{3},1)&\text{if $e=$ {HIGH}}.\end{cases}$

We provide the scalar value of engagement as reward in order to guide the
learned policy toward actions that are expected to elicit higher future
engagement in the users. Also, given that $r(s)>0$ $\forall\,s\in S$ there is
an implicit effect of favouring longer tours.

###  IV-C Behaviour Adaptation with UCBVI

When training an agent to learn in an unknown environment, the _exploration-
exploitation dilemma_ quickly arises where the agent needs to decide if to
keep exploring new actions or executing the one that has returned the highest
rewards so far. This trade-off is particularly relevant in robotic
applications, like ours, where the amount of exploration one can perform is
costly in terms of time and resources and limited by the dynamics of the real
world. In this work, for online learning, we use a Reinforcement Learning
algorithm based on the “optimism in the face of uncertainty” principle. This
principle prescribes that when the model of the environment is uncertain, one
should consider the best possible world; if the model is correct, you have _no
regrets_ (exploitation); otherwise, you have effectively learned something new
about the world (exploration). This general principle gives an almost optimal
solution for the stochastic multi-armed bandit problem [22] and episodic [19]
and ergodic [23] RL problems.

In this work, we are dealing with an episodic RL problem where every guided
tour given by the robot is an episode. For improving the robot policy we use
the Upper Confidence Bound Value Iteration (UCBVI) algorithm[19] which allows
to balance the exploration of novel state-action pairs with the exploration of
already explored ones. In out setting this is particularly important because
we have a part of the state-action space which was explored very well during
the initial deployment with the static policy. UCBVI gives a natural way of
incorporating previously accumulated data without biasing the policy to the
point that it doesn’t explore new actions. Algorithm 1 outlines our
implementation of the algorithm. As in standard _model based_ Value Iteration,
the algorithm is composed of 3 phases. First, the value function is computed
from the model and an initial policy is generated, then the policy is used for
acting in the environment while collecting a new episode, finally the model is
improved based on the updated dataset of episodes. The loop repeats, in our
case, for every new guided tour that is requested by the users.

UCBVI favours exploration of novel state-action pairs thanks to a bonus
function that increases the $Q$-value for pairs that have been scarcely
visited. The bonus decreases toward zero as we obtain more data and the
$Q$-value tends to the real value. Algorithm 2 shows the bonus, inspired by
the Chernoff-Hoeffding’s bonus proposed in [19]. The speed with which the
bonus tends to zero is set by the free variable $\sigma$. For this work, it
was set to $\sigma=20$ in order to allow a relatively fast exploration of new
state-action pairs in our real-world scenario, where obtaining new episodes is
difficult, but it can be set lower in situations where performing exploration
is easier, for example simulations.

Data: $S$, $A$, $H$, $C_{s}$ $\forall$ $s\in S$

$Q_{h}(s,a)\leftarrow 0$ $\forall\,\,(s,a)\in S\times A$ and $h=1,...,H$;

$N_{h}(s,a,s^{\prime})\leftarrow 0$ $\forall\,\,(s,a,s^{\prime})\in S\times
A\times S$ and $h=1,...,H$;

$\mathbb{D}\leftarrow\emptyset$;

foreach  _episode_ do

// Policy optimisation

for  _$h=H,...,1$_ do

for  _$(s,a)\in S\times A$_ do

$b_{h}(s,a)\leftarrow\texttt{bonus}(h,s,a)$;

$Q_{h}(s,a)\leftarrow
r(s)+b_{h}(s,a)+\mathbf{E}_{s^{\prime}\sim\hat{p}_{h}(\cdot\mid
s,a)}[V_{h+1}(s^{\prime})]$;

$V_{h}(s)\leftarrow\min{\\{H-h,\max_{a\in A}Q_{h}(s,a)\\}}$;

end for

end for

$\pi_{h}(s)\leftarrow\text{arg\,max}_{a\in C_{s}}Q_{h}(s,a)$ $\forall\,\,s\in
S$;

// Policy execution

Observe $s_{1}$;

for  _$h=1,...,H$_ do

Execute $a_{h}\leftarrow\pi_{h}(s_{h})$;

Observe $r_{h}$ and $s_{h+1}$;

Add $(s_{h},a_{h},s_{h+1})$ to $\mathbb{D}$

end for

// Model update

$N_{h}(s,a,s^{\prime})\leftarrow\mid\\{(s_{h},a_{h},s_{h+1})\in\mathbb{D}:(s_{h},a_{h},s_{h+1})=(s,a,s^{\prime})\\}\mid$
$\forall\,\,(s,a,s^{\prime},h)$;

$\hat{p}_{h}(s^{\prime}\mid s,a)\leftarrow\frac{N_{h}(s,a,s^{\prime})}{\mid
N_{h}(s,a,\cdot)\mid}$ $\forall\,\,(s,a,s^{\prime},h)$;

end foreach

Algorithm 1 Online learning algorithm

Data: $N_{h}(s,a,s^{\prime})$, $H$

$N\leftarrow\mid N_{h}(s,a,\cdot)\mid$;

$b\leftarrow\sqrt{\frac{1}{N}}+\frac{H-h}{\sigma\cdot N}$;

return $\min\\{b,H-h\\}$

Algorithm 2 bonus

In our implementation, is important to notice that the $Q$-function defined
favours the exploration of poorly explored areas over other areas that have
never been observed. The effect is that the policy sticks to choosing the
same, poorly explored, actions over consecutive episodes and starts exploring
a new one only after a certain number of visitations have been done, rather
than continuously selecting different actions to explore. This effect, which
we found quite useful to maintain a more consistent behaviour over time for
our on-line application, could be eliminated by _optimistically_ initialising
$Q_{h}(s,a)=H-h$ $\forall\,\,(s,a)\in S\times A$ and $h=1,...,H$.

##  V EXPERIMENTAL VALIDATION

In order to validate the hypothesis that optimising a social robot policy to
maximise the user engagement leads to more sustained human-robot interactions
we have performed a long-term study in the museum with our tour guide robot
Lindsey. Since 2019, Lindsey has been delivering the guided tours depicted in
Figure 3 to the visitors in a ”static” way. With its static policy, during
each tour the robot would guide the users to the exhibits always in the same
order and the choice of whether to give a more detailed description about each
item was left to the user. Motivated by the fact that users’ displaying
different perceived engagement have different willingness to continue the
interaction [24], as shown also by our data in Figure 8(a) where for example a
perceived low engagement at the first stop of the tour means a 20% decrease in
chances of users continuing the interaction, we attempt to employ the users
engagement value as a factor to select different actions during the
interactions.

In our learning scenario, we allow the policy to: 1) choose freely the order
of items visited in the tours; and 2) decide whether to provide additional
information about each specific item, rather than asking the user. The model
of our scenario, namely the visitation frequency table
$N_{h}(s,a,s^{\prime})$, is initialised with the transitions from the
thousands of episodes collected with the static policy. During the learning
phase, our _optimistic_ learning algorithm implemented takes care of directing
the exploration toward the other less explored areas. The reward function is
computed on-line from the engagement values detected by our engagement model,
as described in Section IV-B. The robot deployment, interrupted in 2020
because of COVID lockdown, was resumed in December 2021 and the learning phase
began. During the following 8 weeks the robot behaviour was driven by our
online learning algorithm which explored the different actions it could now
take, while exploiting the model to increase users’ willingness to continue.

In order to assess more properly the effect of our learned policy on the tour
success and on users’ engagement we perform a 2 week verification phase,
during which the robot behaviour is again driven by the original static
policy. This validation was necessary to observe whether the effect over our
results during the learning phase was actually caused by our learned policy or
by other spurious effects, like people willing to spend more time at the
museum after COVID lockdowns or face masks wearing.

###  V-A Performances of Learned Policy

Figure 5: (a) Number of guided tours delivered per week, (b) average number of
stops visited in the tours, (c) rate of tours completed to the end, (d)
average scalar engagement level during tours (e) average change in engagement
binned value (i.e. from HIGH, MEDIUM, LOW) between each pair of consecutive
actions. Missing time ranges are caused by the robot not being operational or
museum closures.

In this section we report the results of our evaluation to assess the
performances of the learned robot behaviour. In Figure 5 we report the
evolution of different metrics over the entire deployment (omitting the
periods during which the robot was not operational). The values shown are
collected and averaged per-week with the number of tours for each data point
reported in Figure 5(a) for significance. During the whole deployment period
the museum scenario did not change in any major way to affect the robot tour
guides. Similarly, the robot structure and touch screen interface had only
minor interventions like the replacement of an RGB-D sensor. Data reported for
the weeks from (2019, 21) to (2020, 10) and after (2022, 5) are results of the
”static” tours, while from (2021, 50) to (2022, 5) are caused by the learned
policy.

In Figure 5(b) we report the average number of stops visited (exhibit
described) in the tours. During the learning phase we observe an increase of
22.8% over the previous period, with the verification phase bringing back the
values in line with the previous data for the static tour. Similar results are
obtained for the tour success rate, i.e. the rate of tours that terminate
after visiting all the stops, with an overall increase of over 30% with
respect to the static tour as reported in Figure 5(c). Notice that the nominal
number of items is 6 for the Death tour and 5 for the rest, therefore a lower
number of items means that the users have stopped or abandoned the interaction
before the end. These results together suggest that our learned policy can
indeed lead users to keep a more sustained engagement during the interactions
in the museum, confirming the hypotheses which motivates this work. In
addition, we study the average users’ engagement detected by our model during
the whole duration of the tours and the average change in engagement over each
consecutive actions executed. The plots in Figure 5(d)-(e) suggest that during
the learning phase there was a general decrease of detected engagement with
respect to the previous and the verification periods. Also, we can observe
that the learned policy had the effect of maintaining the users’ engagement
stable over time almost 50% of the times, differently from the previous period
where there was an equal chance of increasing, decreasing or maintaining
stable the engagement.

Figure 6: Different actions chosen by the learned policy for the tour art at
different levels of engagement. Engagement values are red for LOW, blue for
MEDIUM and green for HIGH.

###  V-B Analysis of Learned Tours

In this section we analyse how the guided tour has changed after the learning
phase, with respect to the original static tour. One of the effect of allowing
the learning algorithm to explore different actions was that now the robot
could scramble the sequence of the items in the tour, guiding people to
different places in order. In Figure 6 we show, as an example of this change,
how the final learned policy would conduct the art tour at different level of
engagements detected during execution. The figure evidences how different
users’ engagement levels generates different tours. Additionally, the
different engagements had the effect of changing the amount of information
given to the users at each exhibit. In Figure 7 we report how many times an
additional description is given to user, by performing the describeMoreExhibit
action, for each stop number of the tour at different engagement levels. We
observe that the robot learned to give it more often at the beginning of the
tour, while for later steps only when the engagement is MEDIUM or HIGH.

Finally, to verify whether a different engagement level correlates with a
different willingness to stay in the interaction we study how many times a
tour is stopped or abandoned varying the engagement value at each tour stop.
The results in Figure 8 shows that users are more probable to disengage in the
earlier stops, after the initial tour description, and when they show an
already LOW engagement level, consistent with our intuition that poorly
engaged users are less willing to continue an interaction in the first place.
We observe an overall decrease in disengagement when using the learned policy
compared to the static policy, but no other significant difference is present
between the two conditions.

Figure 7: Percentage of times additional information is provided to the user
at different steps of the tour and for different values of engagement. (a) For
static tour, the additional information is requested by the user; (b) the
learned policy decides when to give more information.

Figure 8: Percentage of times the tour is continued, rather than stopped or
abandoned, at different number of items visited in the tour and for different
values of engagement. (a) Static tour, (b) learned tour. Stop 0 corresponds to
the describeTour action, before having reached any exhibit.

###  V-C State-Action Exploration

Figure 9: Exploration of the state-action space of the policy per week. (a)
Number of newly explored state-action pairs, (b) cumulative exploration.

The UCBVI algorithm used in this work gives a simple yet efficient way of
exploring new unseen states while exploiting the best performing actions, as
described in Section IV-C. In this section we study how the algorithm has
effectively explored the state-action space in our scenario.

Figure 9 shows the amount of exploration that the robot has done, reported per
week. As per our expectations, when using the static tour little to no
exploration is performed, given that the robot is always choosing the same
actions at the same point in the tour. Note that in this condition a little
amount of exploration is still ongoing, but it is entirely governed by the
users’ behaviour with their choices during the tour, i.e. asking for more
information or not, and by manifesting different levels of engagement. The
adoption of our learning algorithm has brought an initial week of high
exploration of many new state-action transitions that were previously
forbidden and kept a steady exploration rate in the subsequent weeks. The
exploration is smaller after the initial period given that the policy starts
to exploit the new actions that have resulted being most promising in terms of
future users’ engagement, as evidenced also by the analysis of the learning
outcomes in Sections V-B and V-A.

The exploration rate achieved as of the beginning of 2022 is only about 13% of
the entire state-action space. We expect that as the learning phase is
resumed, after the verification period, with more exploration the robot will
achieve higher performances in terms of being able to retain users interest
over time in the tours. However, we hypothesise that the more we continue to
act and explore, the more will be difficult to find new states to explore. As
it was mentioned in the previous paragraph, some parts of the state-action
space are not directly explorable by the robot but they can only be observed
indirectly after the user behaviour allows so. To validate this hypothesis, we
have performed a simulation of the next 5 weeks into the future where we keep
using and improving our learned policy and we simulate the next observed
states by sampling uniformly from the set of all the possible that could be
observed at each point in the tour. The simulated exploration, shown in Figure
9, is an upper-bound of the empirical exploration we can obtain in real life
because it does not simulate the possibility of _stopping_ or _abandoning_ the
interaction, which would ultimately prevent exploring further into the tour
sequence. However, the data shows that as the time passes the amount of states
that can be explored by the algorithm decreases.

##  VI DISCUSSION & CONCLUSIONS

In this work, we proposed an Reinforcement Learning approach to in-situ
behavioural adaptation for social robots. The framework exploit a long-term
museum deployment of a tour guide robot for learning, by experience, what are
the best actions the robot should perform during the guided tour to sustaining
the users interaction with the robot for longer. The user engagement is
detected from the robot own camera in real-time to be used both as a part of
the state, to plan actions that are more appropriate for the displayed
engagement, and as a continuous reward to the robots actions, to encourage it
executing actions that lead to eliciting higher future engagement. With our
experimental validations, we observe that the adaptation framework leads the
robot to perform longer tours and the users to stop, or abandon, the
interaction less frequently than before with a static policy. Even though only
a fraction of the entire state-action space has been explored, complete
exploration cannot be practically achieved in our real-world scenario. The
UCBVI algorithm implemented is capable of exploiting the most promising
actions explored so far, hence maintaining good overall performances, while
exploring new states and actions, as confirmed by our empirical results.

Our study present results that span over a period of almost 3 years, including
the COVID pandemic, which could very well encompass many factors influencing
the users’ behaviour over time. In fact, when reporting the users’ detected
engagement, we observe an overall decrease since the museum reopening after
lockdown which could be caused by mask wearing affecting the ability of our
model to detect engagement. However, the improved results of our learned
policy over the static one, even confirmed by the two-weeks verification
period, shows that the overall learning framework maximising detected
engagement is able to produce more sustained human-robot interactions.

A natural extension of this work is to study how our learning framework would
allow the adaptation of the robot behaviour over time as people’s preferences
change. The UCBVI bonus of the present algorithm favours exploration of poorly
explored areas, but it would not recognise whether the estimation of well
explored areas is still relevant, or it needs more experience caused by a
shift in users’ behaviour. A general idea to integrate this would be to have
the bonus factor in the divergence between the values of newly explored state-
action pairs and their estimation from past data.

## References

  * [1] F. Del Duchetto, P. Baxter, and M. Hanheide, “Lindsey the tour guide robot-usage patterns in a museum long-term deployment,” in _International Conference on Robot & Human Interactive Communication (RO-MAN)_. IEEE, 2019.
  * [2] F. Del Duchetto, P. Baxter, and M. Hanheide, “Are You Still With Me? Continuous Engagement Assessment From a Robot’s Point of View,” _Frontiers in Robotics and AI_ , vol. 7, sep 2020.
  * [3] W. Meeussen, E. Marder-Eppstein, K. Watts, and B. P. Gerkey, “Long term autonomy in office environments,” in _ALONE Workshop, In Proceedings of Robotics: Science and Systems (RSS’11), Los Angeles, USA_ , 2011.
  * [4] J. Biswas and M. Veloso, “The 1,000-km challenge: Insights and quantitative and qualitative results,” _IEEE Intelligent Systems_ , vol. 31, no. 3, pp. 86–96, 2016.
  * [5] N. Hawes, C. Burbridge, F. Jovan, L. Kunze, B. Lacerda, L. Mudrova, J. Young, J. Wyatt, D. Hebesberger, T. Kortner, _et al._ , “The strands project: Long-term autonomy in everyday environments,” _IEEE Robotics & Automation Magazine_, vol. 24, no. 3, pp. 146–156, 2017.
  * [6] M. Hanheide, D. Hebesberger, and T. Krajník, “The when, where, and how: An adaptive robotic info-terminal for care home residents,” in _Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_. ACM, 2017, pp. 341–349.
  * [7] I. Leite, C. Martinho, and A. Paiva, “Social robots for long-term interaction: a survey,” _International Journal of Social Robotics_ , vol. 5, no. 2, pp. 291–308, 2013.
  * [8] L. Kunze, N. Hawes, T. Duckett, M. Hanheide, and T. Krajník, “Artificial intelligence for long-term robot autonomy: A survey,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 4023–4030, 2018.
  * [9] W. Burgard, A. B. Cremers, D. Fox, D. Hähnel, G. Lakemeyer, D. Schulz, W. Steiner, and S. Thrun, “The interactive museum tour-guide robot,” in _Aaai/iaai_ , 1998, pp. 11–18.
  * [10] S. Thrun, M. Bennewitz, W. Burgard, A. B. Cremers, F. Dellaert, D. Fox, D. Hahnel, C. Rosenberg, N. Roy, J. Schulte, _et al._ , “Minerva: A second-generation museum tour-guide robot,” in _Proceedings 1999 IEEE International Conference on Robotics and Automation (Cat. No. 99CH36288C)_ , vol. 3. IEEE, 1999.
  * [11] I. R. Nourbakhsh, C. Kunz, and T. Willeke, “The mobot museum robot installations: A five year experiment,” in _Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003)(Cat. No. 03CH37453)_ , vol. 4. IEEE, 2003, pp. 3636–3641.
  * [12] C. L. Sidner, C. Lee, C. D. Kidd, N. Lesh, and C. Rich, “Explorations in engagement for humans and robots,” _Artificial Intelligence_ , vol. 166, no. 1-2, pp. 140–164, 2005.
  * [13] A. Holroyd, “Generating engagement behaviors in human-robot interaction,” Master’s thesis, Worcester Polytechnic Institute, 2011.
  * [14] A. H. Qureshi, Y. Nakamura, Y. Yoshikawa, and H. Ishiguro, “Robot gains social intelligence through multimodal deep reinforcement learning,” in _2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)_. IEEE, 2016, pp. 745–751.
  * [15] ——, “Show, attend and interact: Perceivable human-robot social interaction through neural attention q-network,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2017, pp. 1639–1645.
  * [16] S. Lathuilière, B. Massé, P. Mesejo, and R. Horaud, “Deep reinforcement learning for audio-visual gaze control,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 1555–1562.
  * [17] Y. Gao, F. Yang, M. Frisk, D. Hernandez, C. Peters, and G. Castellano, “Learning socially appropriate robot approaching behavior toward groups using deep reinforcement learning,” in _International Conference on Robot & Human Interactive Communication (RO-MAN)_. New Delhi: IEEE, 2019.
  * [18] L. Meng, D. Lin, A. Francey, R. Gorbet, P. Beesley, and D. Kulić, “Learning to Engage with Interactive Systems,” _ACM Transactions on Human-Robot Interaction_ , vol. 10, no. 1, 2020.
  * [19] M. G. Azar, I. Osband, and R. Munos, “Minimax regret bounds for reinforcement learning,” in _ICML_ , 2017.
  * [20] V. A. Ziparo, L. Iocchi, D. Nardi, P. F. Palamara, and H. Costelha, “Petri net plans: a formal model for representation and execution of multi-robot plans,” in _Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 1_. International Foundation for Autonomous Agents and Multiagent Systems, 2008, pp. 79–86.
  * [21] L. Iocchi, L. Jeanpierre, M. Lazaro, and A.-I. Mouaddib, “A practical framework for robust decision-theoretic planning and execution for service robots,” _Proceedings of the International Conference on Automated Planning and Scheduling_ , vol. 26, no. 1, pp. 486–494, Mar. 2016. [Online]. Available: https://ojs.aaai.org/index.php/ICAPS/article/view/13790
  * [22] S. Bubeck and N. Cesa-Bianchi, “Regret analysis of stochastic and nonstochastic multi-armed bandit problems,” _Foundations and Trends in Machine Learning_ , vol. 5, no. 1, pp. 1–122, 2012.
  * [23] T. Jaksch, R. Ortner, and P. Auer, “Near-optimal regret bounds for reinforcement learning,” in _J. Mach. Learn. Res._ , 2008.
  * [24] C. Peters, C. Pelachaud, E. Bevacqua, M. Mancini, and I. Poggi, “A model of attention and interest using gaze behavior,” in _International Workshop on Intelligent Virtual Agents_. Springer, 2005, pp. 229–240.


11institutetext: DEIM, Rovira i Virgili University, 43007 Tarragona, Spain.
22institutetext: ETSEQ, Rovira i Virgili University, 43007 Tarragona, Spain.
33institutetext: Department of Mathematics, University of Barcelona, 08007
Barcelona, Spain.

# MACNet: Multi-scale Atrous Convolution Networks for Food Places
Classification in Egocentric Photo-streams

Md. Mostafa Kamal Sarker 11Corresponding Author:
<EMAIL_ADDRESS>Author:
<EMAIL_ADDRESS>Hatem A. Rashwan 11 Estefania Talavera 33 Syeda
Furruka Banu 22 Petia Radeva 33 Domenec Puig 11

###### Abstract

First-person (wearable) camera continually captures unscripted interactions of
the camera user with objects, people, and scenes reflecting his personal and
relational tendencies. One of the preferences of people is their interaction
with food events. The regulation of food intake and its duration has a great
importance to protect against diseases. Consequently, this work aims to
develop a smart model that is able to determine the recurrences of a person on
food places during a day. This model is based on a deep end-to-end model for
automatic food places recognition by analyzing egocentric photo-streams. In
this paper, we apply multi-scale Atrous convolution networks to extract the
key features related to food places of the input images. The proposed model is
evaluated on an in-house private dataset called “EgoFoodPlaces”. Experimental
results shows promising results of food places classification recognition in
egocentric photo-streams.

###### Keywords:

deep learning, food pattern classification, egocentric photo-streams, visual
lifelogging

##  1 Introduction

The interest at lifelogging devices, such as first-person (wearable) cameras,
being able to collect daily user information is recently increased. These
cameras capable of frequently capturing images that record visual information
of our daily life known as “visual lifelogging” in order to create a visual
diary with activities of first-person life with unprecedented details [1].
Since, the wearable camera can collect a huge number of images by non-stop
image collection capacity (1-4 per minute, 1K-3K per day and 500K-1000K per
year). The analysis of these egocentric photo-streams (images) can improve the
people lifestyle; by analyzing social pattern characterization [2] and social
interactions [3], as well as generating storytelling of first-person days [1].
In addition, the analysis of these images can greatly affect on human
behaviors, habits, and even health [4]. One of the personal tendencies of
people is food events that can badly affected on their health. For instance,
some people can eat more if they see and senses (e.g. smell) food that
constantly feel them hungry immediately [5, 6]. Thus, monitoring and
determining the duration of food intakes will help to improve people food
behaviour.

The motivation behind this research is twofold. Firstly, using a wearable
camera is to capture images related to food places, where the users are
engaged within foods (see Fig. 1). Consequently, these images of visual
lifelogging can give a unique opportunity to work on food pattern analysis
from the first-person viewpoint. Secondly, the analysis of everyday
information (entering, leaving and stay time, see in Fig. 2) of visited food
places can enable a novel health care application that can help to analyze the
food eating patterns of people and prevent the diseases related to food, like
obesity, diabetes and heart diseases.

Figure 1: Examples of images of food places from an in-house private
EgoFoodPlaces dataset. EgoFoodPlaces is captured by 12 different users in
different food places using the Narrative Clip camera. EgoFoodPlaces is
employed to evaluate the proposed MACNet model for food places recognition.
Figure 2: Examples of commonly spending time in food places everyday.

Early work of places or scene recognition in conventional images was mainly
motivated by two large scale places or scene datasets, i.e., Places2 [7] and
SUN397 [8]) with millions of labeled images. The semantic classes of these
datasets are defined by their labels by representing the entry-level of an
environment. The images of datasets were collected form the internet with a
large diversity. However, the two datasets failed to record the real
involvement of first-person with food environment and the characterization of
the first-person activity. In turn, wearable cameras can able to capture the
scenes from a more intimate perspective by its ego-vision system. Thus, we
built a new in-house private dataset, so-called “EgoFoodPlaces”, with details
involvement information of places that can help to classify the food places or
environment to solve the first-person food pattern characterization.

With diversity of food places (cafeterias, bars, restaurants, ..etc.)
traditional methods of feature extraction (e.g., HOG and SIFT) and
classification (e.g., Support Vector Machine (SVM) and Neural Network (NN))
[9] are not sufficient to deal with this complex problem of food places
recognition. Thus, this paper aims to use deep learning models (e.g.,
Convolutional Neural Network (CNN)) that will help us to automatically select
and extract key features and also to construct new ones for different food
places. One of recent architectures of deep networks used for classification
and segmentation tasks is Atrous Convolution Networks proposed in [10]. That
networks can encode contextual information by using filters or pooling
operations at multiple rates with different sizes of neighbourhoods. Thus, in
this paper, we propose to use these networks in our deep model to improve the
classification rate with ResNet networks. In addition to detect important
structures as well as small details of the input images, we rescale the input
images in a multi-scale space (i.e., a pyramid of images with different
resolutions).

The main contributions of this work is summarized as follows:

  * $\bullet$

Introduce a new dataset developed by lifelogging camera for food places
classification, named EgoFoodPlaces.

  * $\bullet$

Proposed a new deep network architecture based on multi-scale Atrous
convolutional networks [10] for improving classification rate of food places
in egocentric photo-streams.

The paper is organized as follows. Section 2 explores the proposed approach.
In turn, Section 3 describes our in-house dataset and demonstrate the
experimental results and discussions. Finally, conclusion and future work are
shown in section 4.

##  2 Proposed Approach

The proposed deep model, MACNet, is based on multi-scale Atrous convolutional
networks for extracting the key patterns of food places in the input
egocentric photo-streams. The multi-scale features are used to fine tuning
four layers of a pre-trained ResNet-101 model as shown in figure 3.

The input images are scaled to five resolutions (i.e., the original size and
four different resolutions) as shown in figure 3. The five images with
different resolutions feeds to Atrous convolutional networks [10]. In MACNet,
five blocks of Atrous convolutional network with three different rates per
block are used to extract the key features of an input image. Atrous
convolutional network allows us to explicitly extract features with different
scales. In addition, it adjusts filter’s size with the rate value in order to
capture multi-scale information, generalizes standard convolution operation.
We used $3\times 3$ kernels in all blocks with different rate values set to 1,
2 and 3. More details about these networks presented in [11] and [10].

Following, four pre-trained ResNet-101 blocks are then used to extract $256$,
$512$, $1024$ and $2048$ feature maps, respectively as shown in figure 3. The
four ResNet-101 layers are with stride $2.0$. Thus, the final output size of
the last ResNet block is $1/16$ of the input image size. Indeed, each ResNet
is corresponding to a resolution level in the image pyramid. Each output of
the five Atrous network blocks is followed by a pointwise convolution (i.e.,
$1\times 1$ convolution) to reduce the computation complexity and the number
of channels to be compatible with the input channels accepted by the
corresponding ResNet layer. All Atrous convolutional networks and $1\times 1$
convolution are randomly initialized.

The output of the fourth ResNet layer feeds to a fully connected layer with
$1024$ neurons followed by another fully connected layer with $512$ neurons. A
dropout function with $0.5$ is used for reducing overfitting in the two fully
connected layers. A ReLU function is also used as an activation function for
the first fully connected layer. In turn, a softmax function (i.e., normalized
exponential function) is finally utilized as a logistic function for producing
the final probability of the input image to each class. The two fully
connected layers are randomly initialized.

Figure 3: Architecture of our proposed model for food places classification.

In this paper, we constructed new dataset from egocentric images. In our
dataset “EgoFoodPlaces”, the numbers of instances with different labels are
very unbalanced. To deal with this issue, we used a weighted categorical
cross-entropy function. For the cross-entropy with a multi-class
classification, we calculate a separate loss for each class label and then
then sum the result as:

$\ell_{i}=-\sum_{j=1}^{N}w_{y_{j}}y_{j}\log(\hat{y_{j}}),$ (1)

where $N$ is the number of instances, $y_{j}$ is the actual label of the
$j_{t}h$ instance, $\hat{y_{j}}$ is the prediction score, and $w_{y_{j}}$, the
loss weight of the label $y_{j}$, is defined as:

$w_{y_{j}}=1-\frac{N_{y_{j}}}{N},$ (2)

where $N_{y_{j}}$ refers to the number of instances per label $y_{j}$.

##  3 Experimental Results

###  3.1 Datasets

In this work we introduce to “EgoFoodPlaces”, a new egocentric photo-streams
dataset that devolved by $12$ users by using wearable camera (narrative clip 2
111http://getnarrative.com/, which has an image resolution of 720p and 1080p
by a 8-megapixel camera with an 86-degree field of view and capable of record
about $4,000$ photos or $80$ minutes of 1080p video at 30fps). Figure 1 shows
some example images from the EgoFoodPlaces dataset. It is composed by
egocentric photo-streams describing the users daily food related activities
(preparing, eating, buying, etc). Some images are used in our data,
EgoFoodPlaces, from the EDUB-Seg dataset [12].

The first-person used the camera fixed to his chest from morning to night
before sleeping. Figure 2 shows the day hours for capturing the images. Every
frames of a photo-stream is recording first-person activities, which is very
helpful to analyze different pattern of first-person lifestyle. However, the
captured images have different challenges, such as background variation,
lighting change, and handling objects sometimes occluded during the photo-
stream. In addition, the constructed dataset has unbalanced classes. However,
it is not possible to make it as a balanced dataset by reducing images from
other classes, since some classes have very small number of images. The
classes with few images are related to some food places that do not have rich
visual information (e.g., candy store) or the users do not spend much time at
there (e.g., butchers shop). In turn, we have very large number of images are
of visited places with rich visual information that refer to daily contexts
(e.g., kitchen, supermarket), or of places, where we send more times (e.g.,
restaurant). We labelled our dataset manually by taking the reference labels
related to food places from the Places2 dataset [7]. Since, some of the
classes related to food places from the Places2 dataset [7] are not available
(e.g., beer garden). Therefore, we excluded these classes from EgoFoodPlaces.

Twenty-two classes of food places are described in our dataset as shown in
Table 1. We have split EgoFoodPlaces into three sets: train, validation and
test. The images of each set were not randomly choose to avoid of taking
similar images from the same events. Thus, we split the dataset based on food
event information. The events represents the entry and exit image frame from
the places visited. This can make the dataset more robust to train and
validate our model.

Table 1: The distribution of images per class in the EgoFoodPlaces dataset.

| Train | Val | Test |  | Train | Val | Test
---|---|---|---|---|---|---|---
Classes | images | events | images | events | images | events | Classes | images | events | images | events | images | events
bakery_shop | 96 | 15 | 15 | 3 | 28 | 4 | food_court | 161 | 6 | 37 | 2 | 06 | 1
banquet_hall | 203 | 1 | 52 | 1 | 96 | 1 | ice_cream_parlor | 70 | 4 | 12 | 2 | 25 | 1
bar | 1121 | 23 | 137 | 5 | 374 | 6 | kitchen | 2701 | 81 | 389 | 13 | 743 | 23
beer_hall | 296 | 1 | 62 | 1 | 318 | 1 | market_indoor | 644 | 15 | 97 | 3 | 163 | 4
butchers_shop | 251 | 4 | 11 | 1 | 15 | 1 | market_outdoor | 1271 | 11 | 13 | 2 | 104 | 3
cafeteria | 1238 | 23 | 141 | 5 | 310 | 6 | picnic_area | 659 | 4 | 89 | 2 | 173 | 1
candy_store | 172 | 4 | 26 | 2 | 55 | 1 | pizzeria | 1022 | 3 | 125 | 1 | 265 | 1
coffee_shop | 1662 | 29 | 210 | 5 | 441 | 8 | pub_indoor | 342 | 7 | 60 | 1 | 109 | 2
delicatessen | 652 | 6 | 29 | 2 | 05 | 1 | restaurant | 4198 | 29 | 481 | 5 | 1044 | 8
dining_room | 2481 | 73 | 326 | 12 | 832 | 21 | supermarket | 3019 | 70 | 477 | 10 | 827 | 20
fastfood_restaurant | 858 | 14 | 102 | 2 | 217 | 4 | sushi_bar | 1151 | 7 | 195 | 1 | 296 | 2

###  3.2 Experimental Setup

The proposed model is implemented on PyTorch[13]: an open source deep learning
library. For the optimization method, we used the Stochastic Gradient Descent
(SGD) [14] with momentum of 0.9 and weight decay of 0.0005. For adjusting
learning rate depending on first and second order moments of the gradient, we
used a “step” learning rate policy [15] and we selected a base learning rate
of 0.001 and the step is 20. In order to increase the number of images related
to a class having few images, we used data augmentation. For data
augmentation, we performed random crop, image brightness and contrast change
with 0.2 and 0.1, respectively. We also use random affine transform between
the angle of -20 and 20, image translation of 0.5, random scale between 0.5
and 1.0, and random rotation of 10 degrees. The optimized batchsize is set to
$32$ for training and the number of epochs is set to $100$. All the
experiments are executed on NVIDIA TITAN X with 12GB memory taking around 20
hours to train the network. All these parameters are used for all tested
methods in our experiments.

###  3.3 Evaluation metrics

Since the constructed dataset, EgoFoodPlaces, is highly imbalanced, the
classification performance of all tested methods was assessed by not only
using the accuracy, but also using other three evolution measures: precision,
recall, and F1-score.

###  3.4 Comparison with classification methods

Three different CNN architectures, specifically the VGG16, InceptionV3, and
ResNet 50, are used in a comparison to assess our proposed model, MACNet.

VGG-16: We fine-tuned a VGG-16 network proposed in [16] in the all 16 layers
were back-propagated, and the SGD optimization method used.

ResNet50: The ResNet50 network proposed in [17] was fine-tuned and was
optimized using SGD.

InceptionV3: The InceptionV3 network proposed in [11] was also fine-tuned with
SGD as an optimization method.

###  3.5 Results and Discussions

We compared the performances of VGG16, ResNet50 and InceptionV3 to our
proposed model, MACNet as shown in Table 2. MCANet yielded an average of
Precision of $72\%$, Recall of $60\%$ and F1-score of $65\%$ with the
validation set, and about $70\%$, $57\%$ and $63\%$, respectively with the
test set. Our experiments demonstrated that the food places classification
scores obtained with MACNet are better than the scores of the three test
models on both validation and test set. However, InceptionV3 provided
acceptable results with around $61\%$, $50\%$ and $55$ with both validation
and test sets. In turn, VGG16 yielded the worst scores among the four tested
method. This means that the MACNet based on multi-scale Atrous convolutional
networks can be able to improve the classification of food places in
egocentric photo-images.

Table 2: The average Precision, Recall and F1-score of both validation and
test sets of the EgoFoodPlaces dataset with VGG16, ResNet50, InceptionV3 and
the proposed MACNet model.

Models | Validation | Test
---|---|---
Precision | Recall |  $F_{1}$ score | Precision | Recall |  $F_{1}$ score
VGG16 | 38.12 | 25.06 | 30.24 | 36.46 | 24.85 | 29.55
ResNet50 | 61.30 | 49.04 | 54.48 | 59.07 | 47.44 | 52.62
InceptionV3 | 63.91 | 52.13 | 57.42 | 61.39 | 50.51 | 55.42
MACNet | 72.33 | 59.53 | 65.37 | 69.54 | 57.19 | 62.76

Furthermore, the Top-1 and Top-5 accuracy of the three test models, VGG16,
ResNet50 and InceptionV3, and the proposed MACNet model are shown in Table 3.
As shown, for the validation set, MACNet yielded more than a $10\%$
improvement in Top-1 accuracy with respect to the VGG-16 model, and around a
$4\%$ improvment with respect to both ResNet50 and InceptionV3 models.
Regarding to the test set, MACNet lead to a $3\%$ improvement to the three
tested model.

Table 3: The average Top-1 and Top-5 accuracy of both validation and test sets
of the EgoFoodPlaces dataset with VGG16, ResNet50, InceptionV3 and the
proposed MACNet model.

Models | Validation | Test
---|---|---
Top-1 | Top-5 | Top-1 | Top-5
VGG16 | 53.93 | 83.98 | 49.20 | 81.07
ResNet50 | 61.31 | 85.48 | 55.38 | 84.95
InceptionV3 | 60.82 | 88.22 | 54.76 | 85.60
MACNet | 64.80 | 90.70 | 58.47 | 86.78

Figure 4: The resulted of F1-score of the (Top) validation set (down) test set
of the EgoFoodPlaces dataset with three methods VGG16, ResNet50, InceptionV3
and the proposed MACNet model.

Figure 4 shows the F1-score per class with the four tested methods over 22
classes of the validation and test set of EgoFoodPlaces. In the most classes
(e.g., dining room, sushi bar, ice cream, coffee shop and food court), MACNet
yielded a significant improvement of F1-score. In some cases (e.g., hall bar
and pub indoor), ResNet50 provided better results than the other methods. In
turn, VGG16 can classify the food places in the EgoFoodPlaces better then the
other tested methods, such as picnic area and bakery shop. While, InceptionV3
did not outperform the other methods per class, however its average F1-score
is better than VGG16 and ResNet50 and less than MACNet. Note that the zero
values of F1-score shown in figure 4 are related to the classes that have few
images per class.

Moreover, the improvement over the overlapping classes can also be seen on the
confusion matrices shown in figure 5. This means that the multi-scale Atrous
convolutional networks improved the food places classification belonging to
classes that score similar probabilities.

Figure 5: The confusion matrix of the (top) validation set and (down) test set
of the EgoFoodPlaces dataset with three methods VGG16, ResNet50, InceptionV3
and the proposed MACNet model.

##  4 Conclusions

In this paper, we proposed a new architecture of deep model, MACNet for food
places recognition from egocentric photo-streams. MACNet is based on multi-
scale Atrous convoluitonal networks that fusing with four pre-tranied layers
of ResNet-101 and two fully connected layers. MACNet extracting features of
different resolutions of an input image of first-person images. In addition,
we constructed an in-house private egocentric photo-streams dataset containing
22 classes of food places, named EgoFoodPlaces. Experimental results on this
dataset demonstrated that the proposed approach achieve better performances
than a three common architecture of classification methods, VGG16, ResNet50
and InceptionV3. The proposed method achieved an overall Top-5 accuracy around
$86.78\%$ over the test set of EgoFoodPlaces. Future work aims to use the
MACNet model with a complete framework for people food behaviour.

## References

  * [1] Bolanos, M., Dimiccoli, M., Radeva, P.:  Toward storytelling from visual lifelogging: An overview.  IEEE Transactions on Human-Machine Systems 47(1) (2017) 77–90
  * [2] Aghaei, M., Dimiccoli, M., Ferrer, C.C., Radeva, P.:  Towards social pattern characterization in egocentric photo-streams.  Computer Vision and Image Understanding (2018)
  * [3] Aghaei, M., Dimiccoli, M., Radeva, P.:  Towards social interaction detection in egocentric photo-streams.  In: Eighth International Conference on Machine Vision (ICMV 2015). Volume 9875., International Society for Optics and Photonics (2015) 987514
  * [4] Grimm, E.R., Steinle, N.I.:  Genetics of eating behavior: established and emerging concepts.  Nutrition reviews 69(1) (2011) 52–60
  * [5] Kemps, E., Tiggemann, M., Hollitt, S.:  Exposure to television food advertising primes food-related cognitions and triggers motivation to eat.  Psychology & health 29(10) (2014) 1192–1205
  * [6] de Wijk, R.A., Polet, I.A., Boek, W., Coenraad, S., Bult, J.H.:  Food aroma affects bite size.  Flavour 1(1) (2012) 3
  * [7] Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.:  Learning deep features for scene recognition using places database.  In: Advances in neural information processing systems. (2014) 487–495
  * [8] Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.:  Sun database: Large-scale scene recognition from abbey to zoo.  In: Computer vision and pattern recognition (CVPR), 2010 IEEE conference on, IEEE (2010) 3485–3492
  * [9] Moran, T.H., Gao, S.:  Looking for food in all the right places?  Cell metabolism 3(4) (2006) 233–234
  * [10] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.:  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40(4) (2018) 834–848
  * [11] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.:  Rethinking atrous convolution for semantic image segmentation.  arXiv preprint arXiv:1706.05587 (2017)
  * [12] Dimiccoli, M., Bolaños, M., Talavera, E., Aghaei, M., Nikolov, S.G., Radeva, P.:  Sr-clustering: Semantic regularized clustering for egocentric photo streams segmentation.  Computer Vision and Image Understanding 155 (2017) 55–69
  * [13] Paszke, A., Gross, S., Chintala, S., Chanan, G.:  Pytorch (2017)
  * [14] Gulcehre, C., Sotelo, J., Moczulski, M., Bengio, Y.:  A robust adaptive stochastic gradient method for deep learning.  In: Neural Networks (IJCNN), 2017 International Joint Conference on, IEEE (2017) 125–132
  * [15] Sebag, A., Schoenauer, M., Sebag, M.:  Stochastic gradient descent: Going as fast as possible but not faster.  In: OPTML 2017: 10th NIPS Workshop on Optimization for Machine Learning. (2017)
  * [16] Schüssler-Fiorenza Rose, S.M., Stineman, M.G., Pan, Q., Bogner, H., Kurichi, J.E., Streim, J.E., Xie, D.:  Potentially avoidable hospitalizations among people at different activity of daily living limitation stages.  Health services research 52(1) (2017) 132–155
  * [17] He, K., Zhang, X., Ren, S., Sun, J.:  Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition. (2016) 770–778


# Kandinsky: an Improved Text-to-Image Synthesis with
Image Prior and Latent Diffusion

Anton Razzhigaev1,2, Arseniy Shakhmatov3, Anastasia Maltseva3, Vladimir
Arkhipkin3,
Igor Pavlov3, Ilya Ryabov3, Angelina Kuts 3, Alexander Panchenko2,1,
Andrey Kuznetsov3,1, and Denis Dimitrov3,1
1AIRI, 2Skoltech, 3Sber AI
{razzhigaev, kuznetsov<EMAIL_ADDRESS>

###### Abstract

Text-to-image generation is a significant domain in modern computer vision and
has achieved substantial improvements through the evolution of generative
architectures. Among these, there are diffusion-based models that have
demonstrated essential quality enhancements. These models are generally split
into two categories: pixel-level and latent-level approaches. We present
Kandinsky111The system is named after Wassily Kandinsky, a famous painter and
an art theorist., a novel exploration of latent diffusion architecture,
combining the principles of the image prior models with latent diffusion
techniques. The image prior model is trained separately to map text embeddings
to image embeddings of CLIP. Another distinct feature of the proposed model is
the modified MoVQ implementation, which serves as the image autoencoder
component. Overall, the designed model contains 3.3B parameters. We also
deployed a user-friendly demo system that supports diverse generative modes
such as text-to-image generation, image fusion, text and image fusion, image
variations generation, and text-guided inpainting/outpainting. Additionally,
we released the source code and checkpoints for the Kandinsky models.
Experimental evaluations demonstrate a FID score of 8.03 on the COCO-30K
dataset, marking our model as the top open-source performer in terms of
measurable image generation quality.

##  1 Introduction

Figure 1: Image prior scheme and inference regimes of the Kandinsky model.

In quite a short period of time, generative abilities of text-to-image models
have improved substantially, providing users with photorealistic quality, near
real-time inference speed, a great number of applications and features,
including simple easy-to-use web-based platforms and sophisticated AI graphics
editors.

This paper presents our unique investigation of latent diffusion architecture
design, offering a fresh and innovative perspective on this dynamic field of
study. First, we describe the new architecture of Kandinsky and its details.
The demo system with implemented features of the model is also described.
Second, we show the experiments, carried out in terms of image generation
quality and come up with the highest FID score among existing open-source
models. Additionally, we present the rigorous ablation study of prior setups
that we conducted, enabling us to carefully analyze and evaluate various
configurations to arrive at the most effective and refined model design.

Our contributions are as follows:

  * •

We present the first text-to-image architecture designed using a combination
of image prior and latent diffusion.

  * •

We demonstrate experimental results comparable to the state-of-the-art (SotA)
models such as Stable Diffusion, IF, and DALL-E 2, in terms of FID metric and
achieve the SotA score among all existing open source models.

  * •

We provide a software implementation of the proposed state-of-the-art method
for text-to-image generation, and release pre-trained models, which is unique
among the top-performing methods. Apache 2.0 license makes it possible to use
the model for both non-commercial and commercial
purposes.222https://github.com/ai-forever/Kandinsky-2
333https://huggingface.co/kandinsky-community

  * •

We create a web image editor application that can be used for interactive
generation of images by text prompts (English and Russian languages are
supported) on the basis of the proposed method, and provides
inpainting/outpainting functionality.444https://fusionbrain.ai/en/editor The
video demonstration is available on
YouTube.555https://www.youtube.com/watch?v=c7zHPc59cWU

##  2 Related Work

Early text-to-image generative models, such as DALL-E Ramesh et al. (2021) and
CogView Ding et al. (2021), or later Parti Yu et al. (2022) employed
autoregressive approaches but often suffered from significant content-level
artifacts. This led to the development of a new breed of models that utilized
the diffusion process to enhance image quality. Diffusion-based models, such
as DALL-E 2 Ramesh et al. (2022), Imagen Saharia et al. (2022b), and Stable
Diffusion666https://github.com/CompVis/stable-diffusion, have since become
cornerstones in this domain. These models are typically divided into pixel-
level Ramesh et al. (2022); Saharia et al. (2022b) and latent-level Rombach et
al. (2022) approaches.

This surge of interest has led to the design of innovative approaches and
architectures, paving the way for numerous applications based on open-source
generative models, such as DreamBooth Ruiz et al. (2023) and DreamPose Karras
et al. (2023). These applications exploit image generation techniques to offer
remarkable features, further fueling the popularity and the rapid development
of diffusion-based image generation approaches.

This enabled a wide array of applications like 3D object synthesis Poole et
al. (2023); Tang et al. (2023); Lin et al. (2022); Chen et al. (2023), video
generation Ho et al. (2022b); Luo et al. (2023); Ho et al. (2022a); Singer et
al. (2023); Blattmann et al. (2023); Esser et al. (2023), controllable image
editing Hertz et al. (2023); Parmar et al. (2023); Liew et al. (2022); Mou et
al. (2023); Lu et al. (2023), and more, which are now at the forefront of this
domain.

Figure 2: Examples of inference regimes using Kandinsky model.

Diffusion models achieve state-of-the-art results in image generation task
both unconditional Ho et al. (2020); Nichol and Dhariwal (2021) and
conditional Peebles and Xie (2022). They beat GANs Goodfellow et al. (2014) by
generating images with better scores of fidelity and diversity without
adversarial training Dhariwal and Nichol (2021). Diffusion models also show
the best performance in various image processing tasks like inpainting,
outpainting, and super-resolution Batzolis et al. (2021); Saharia et al.
(2022a).

Text-to-image diffusion models have become a popular research direction due to
the high performance of diffusion models and the ability to simply integrate
text conditions with the classifier-free guidance algorithm Ho and Salimans
(2022). Early models like GLIDE Nichol et al. (2022), Imagen Saharia et al.
(2022b), DALL-E 2 Ramesh et al. (2022) and eDiff-I Balaji et al. (2022)
generate low-resolution image in pixel space and then upsample it with another
super-resolution diffusion models. They are also using different text
encoders, large language model T5 Raffel et al. (2020) in Imagen, CLIP Radford
et al. (2021) in GLIDE and DALL-E 2.

##  3 Demo System

We implemented a set of user-oriented solutions where Kandinsky model is
embedded as a core imaging service. It has been done due to a variety of
inference regimes, some of which need specific front-end features to perform
properly. Overall, we implemented two main inference resources: Telegram bot
and FusionBrain website.

FusionBrain represents a web-based image editor with such features as loading
and saving images, sliding location window, erasing tools, zooming in/out,
various styles selector, etc. (cf. Figure 3). In terms of image generation,
the three following options are implemented on this side:

  * •

text-to-image generation – user inputs a text prompt in Russian or English,
then selects an aspect-ratio from the list (9:16, 2:3, 1:1, 16:9, 3:2), and
the system generates an image;

  * •

inpainting – using the specific erasing tool, user can remove any arbitrary
input image part and fill it, guided by a text prompt or without any guidance;

  * •

outpainting – input image can be extended with a sliding window that can be
used as a mask for the following generation (if the window intersects any
imaged area, then the empty window part is generated with or without text
prompt guidance).

Inpainting and outpainting options are the main image editing features of the
model. Architectural details about these generation types can also be found in
Figure 1.

Figure 3: Kandinsky web interface for “a corgi gliding on the wave”:
generation (left) and in/outpainting (right).

Telegram bot contains the following image generation features (cf. Figure 2):

  * •

text-to-image generation;

  * •

image and text fusion – user inputs an image and a text prompt to create a new
image guided by this prompt;

  * •

image fusion – user inputs an image as the main one and another ’guiding’
image, and the system generates their fusion;

  * •

image variations – user inputs an image, and the system generates several new
images similar to the input one.

##  4 Kandinsky Architecture

In our work, we opted to deliver state-of-the-art text-to-image synthesis. In
the initial stages of our research, we experimented with multilingual text
encoders, such as mT5 Xue et al. (2021), XLMR Conneau et al. (2020), XLMR-
CLIP777https://github.com/FreddeFrallan/Multilingual-CLIP, to facilitate
robust multilingual text-to-image generation. However, we discovered that
using the CLIP-image embeddings instead of standalone text encoders resulted
in improved image quality. As a result, we adopted an image prior approach,
utilizing diffusion and linear mappings between text and image embedding
spaces of CLIP, while keeping additional conditioning with XLMR text
embeddings. That is why Kandinsky uses two text encoders: CLIP-text with image
prior mapping and XLMR. We have set these encoders to be frozen during the
training phase.

The significant factor that influenced our design choice was the efficiency of
training latent diffusion models, as compared to pixel-level diffusion models
Rombach et al. (2022). This led us to focus our efforts on the latent
diffusion architecture. Our model essentially comprises three stages: text
encoding, embedding mapping (image prior), and latent diffusion.

The construction of our model involves three primary steps: text encoding,
embedding mapping (image prior), and latent diffusion. At the embedding
mapping step, which we also refer to as the image prior, we use the
transformer-encoder model. This model was trained from scratch with a
diffusion process on text and image embeddings provided by the CLIP-ViT-L14
model. A noteworthy feature in our training process is the use of element-wise
normalization of visual embeddings. This normalization is based on full-
dataset statistics and leads to faster convergence of the diffusion process.
We implemented inverse normalization to revert to the original CLIP-image
embedding space in the inference stage.

Table 1: Proposed architecture comparison by FID on COCO-30K validation set on $256\times 256$ resolution. * For the IF model we reported reproduced results on COCO-30K, but authors provide FID of 7.19. Model | FID-30K
---|---
Open Sourced Techologies
Kandinsky (Ours) | 8.03
Stable Diffusion 2.1 (2022) 888https://github.com/Stability-AI/stablediffusion | 8.59
GLIDE 0 Nichol et al. (2022) | 12.24
IF* (2023) 0 | 15.10
Kandinsky 1.0 (2022) 999https://github.com/ai-forever/ru-dalle | 15.40
ruDALL-E Malevich (2022) 0 | 20.00
GLIGEN 101010https://github.com/gligen/GLIGEN Li et al. (2023) | 21.04
Proprietary Technologies
eDiff-I Balaji et al. (2022) | 6.95
Imagen Saharia et al. (2022b) | 7.27
GigaGAN Kang et al. (2023) | 9.09
DALL-E 2 Ramesh et al. (2022) | 10.39
DALL-E Ramesh et al. (2021) | 17.89

The image prior model is trained on text and image embeddings, provided by the
CLIP models. We conducted a series of experiments and ablation studies on the
specific architecture design of the image prior model (Table 3, Figure 6). The
model with the best human evaluation score is based on a 1D-diffusion and
standard transformer-encoder with the following parameters: num_layers=20,
num_heads=32, and hidden_size=2048.

The latent diffusion part employs a UNet model along with a custom pre-trained
autoencoder. Our diffusion model uses a combination of multiple condition
signals: CLIP-image embeddings, CLIP-text embeddings, and XLMR-CLIP text
embeddings. CLIP-image and XLMR-CLIP embeddings are merged and utilized as an
input to the latent diffusion process. Also, we conditioned the diffusion
process on these embeddings by adding all of them to the time-embedding.
Notably, we did not skip the quantization step of the autoencoder during
diffusion inference as it leads to an increase in the diversity and the
quality of generated images (cf. Figure 4). In total, our model comprises 3.3
B parameters (Table 2).

Table 2: Kandinsky model parameters. Architecture part | Params | Freeze
---|---|---
Diffusion Mapping | 1B | False
CLIP image encoder (ViT-L14) | 427M | True
CLIP text encoder | 340M | True
Text encoder (XLM-R-L) | 560M | True
Latent Diffusion UNet | 1.22B | False
MoVQ image autoencoder | 67M | True

We observed that the image decoding was our main bottleneck in terms of
generated image quality; hence, we developed a Sber-MoVQGAN, our custom
implementation of MoVQGAN Zheng et al. (2022) with minor modifications. We
trained this autoencoder on the LAION HighRes dataset Schuhmann et al. (2022),
obtaining the SotA results in image reconstruction. We released the weights
and code for these models under an open source
licence111111https://github.com/ai-forever/MoVQGAN. The comparison of our
autoencoder with competitors can be found in Table LABEL:movq.

Table 3: Ablation study: FID on COCO-30K validation set on $256\times 256$ resolution. Setup | FID-30K | CLIP
---|---|---
Diffusion prior with quantization | 9.86 | 0.287
Diffusion prior w/o quantization | 9.87 | 0.286
Linear prior | 8.03 | 0.261
Residual prior | 8.61 | 0.249
No prior | 25.92 | 0.256

##  5 Experiments

We sought to evaluate and refine the performance of our proposed latent
diffusion architecture in our experimental analysis. To this end, we employed
automatic metrics, specifically FID-CLIP curves on the COCO-30K dataset, to
obtain the optimal guidance-scale value and compare Kandinsky with competitors
(cf. Figure 4). Furthermore, we conducted investigations with various image
prior setups, exploring the impact of different configurations on the
performance. These setups included: no prior, utilizing text embeddings
directly; linear prior, implementing one linear layer; ResNet prior,
consisting of 18 residual MLP blocks; and transformer diffusion prior.

An essential aspect of our experiments was the exploration of the effect of
latent quantization within the MoVQ autoencoder. We examined the outputs with
latent quantization, both enabled and disabled, to better comprehend its
influence on image generation quality.

Figure 4: CLIP-FID curves for different setups.

To ensure a comprehensive evaluation, we also included an assessment of the IF
model 121212https://github.com/deep-floyd/IF, which is the closest open-source
competitor to our proposed model. For this purpose, we computed FID scores for
the IF model 131313https://github.com/mseitzer/pytorch-fid (Table 1).

However, we acknowledged the limitations of automatic metrics that become
obvious when it comes to capturing user experience nuances. Hence, in addition
to the FID-CLIP curves, we conducted a blind human evaluation to obtain
insightful feedback and validate the quality of the generated images from the
perspective of human perception based on the DrawBench dataset Saharia et al.
(2022b).

The combination of automatic metrics and human evaluation provides a
comprehensive assessment of Kandinsky performance, enabling us to make
informed decisions about the effectiveness and usability of our proposed image
prior to design.

Figure 5: Image generation results with prompt "astronaut riding a horse" for original image prior and linear prior trained on 500 pairs of images with cats. Figure 6: Human evaluation: competitors vs Kandinsky with diffusion prior on Drawbench. The total count of votes is 5000. Table 4: Sber-MoVQGAN comparison with competitors on ImageNet dataset. Model | Latent size | Num Z | Train steps | FID $\downarrow$ | SSIM $\uparrow$ | PSNR $\uparrow$ | L1 $\downarrow$
---|---|---|---|---|---|---|---
ViT-VQGAN* | 32x32 | 8192 | 500,000 | 1.28 | - | - | -
RQ-VAE* | 8x8x16 | 16384 | 10 epochs | 1.83 | - | - | -
Mo-VQGAN* | 16x16x4 | 1024 | 40 epochs | 1.12 | 0.673 | 22.42 | -
VQ CompVis | 32x32 | 16384 | 971,043 | 1.34 | 0.650 | 23.85 | 0.0533
KL CompVis | 32x32 | - | 246,803 | 0.968 | 0.692 | 25.11 | 0.0474
Sber-VQGAN | 32x32 | 8192 | 1 epoch | 1.44 | 0.682 | 24.31 | 0.0503
Sber-MoVQGAN 67M | 32x32 | 1024 | 5,000,000 | 1.34 | 0.704 | 25.68 | 0.0451
Sber-MoVQGAN 67M | 32x32 | 16384 | 2,000,000 | 0.965 | 0.725 | 26.45 | 0.0415
Sber-MoVQGAN 102M | 32x32 | 16384 | 2,360,000 | 0.776 | 0.737 | 26.89 | 0.0398
Sber-MoVQGAN 270M | 32x32 | 16384 | 1,330,000 | 0.686 | 0.741 | 27.04 | 0.0393

##  6 Results

Our experiments and evaluations have showcased the capabilities of Kandinsky
architecture in text-to-image synthesis. Kandinsky achieved the FID score of
8.03 on the COCO-30K validation set at a resolution of 256×256, which puts it
in close competition with the state-of-the-art models, and among the top
performers within open-source systems. Our methodical ablation studies further
dissected the performance of different configurations: quantization of latent
codes in MoVQ slightly improves the quality of images (FID 9.86 vs 9.87). The
best CLIP score and human-eval score are obtained by diffusion prior.

The best FID score is achieved using Linear Prior. This configuration stands
out with the best FID score of 8.03. It is an intriguing outcome: the simplest
linear mapping showcased the best FID, suggesting that there might exist a
linear relationship between visual and textual embedding vector spaces. To
further scrutinize this hypothesis, we trained a linear mapping on a subset of
500 cat images and termed it the "cat prior". Astonishingly, this mapping
displayed high proficiency (cf. Figure 5).

##  7 Conclusion

We presented Kandinsky, a system for various image generation and processing
tasks based on a novel latent diffusion model. Our model yielded the SotA
results among open-sourced systems. Additionally, we provided an extensive
ablation study of an image prior to design choices. Our system is equipped
with free-to-use interfaces in the form of Web application and Telegram
messenger bot. The pre-trained models are available on Hugging Face, and the
source code is released under a permissive license enabling various, including
commercial, applications of the developed technology.

In future research, our goal is to investigate the potential of the latest
image encoders. We plan to explore the development of more efficient UNet
architectures for text-to-image tasks and focus on improving the understanding
of textual prompts. Additionally, we aim to experiment with generating images
at higher resolutions and to investigate new features extending the model:
local image editing by a text prompt, attention reweighting, physics-based
generation control, etc. The robustness against generating abusive content
remains a crucial concern, warranting the exploration of real-time moderation
layers or robust classifiers to mitigate undesirable, e.g. toxic or abusive,
outputs.

##  8 Limitations

The current system produces images that appear natural, however, additional
research can be conducted to (1) enhance the semantic coherence between the
input text and the generated image, and (2) to improve the absolute values of
FID and image quality based on human evaluations.

##  9 Ethical Considerations

We performed multiple efforts to ensure that the generated images do not
contain harmful, offensive, or abusive content by (1) cleansing the training
dataset from samples that were marked to be harmful/offensive/abusive, and (2)
detecting abusive textual prompts.

While obvious queries, according to our tests, almost never generate abusive
content, technically it is not guaranteed that certain carefully engineered
prompts may not yield undesirable content. We, therefore, recommend using an
additional layer of classifiers, depending on the application, which would
filter out the undesired content and/or use image/representation
transformation methods tailored to a given application.

## Acknowledgements

As usual, we would like to thank the anonymous reviewers for their useful
comments. We would also like to thank Sergey Markov and his team for helpful
feedback and discussions, for collaboration in multimodal dataset collecting,
labelling and processing.

## References

  * Balaji et al. (2022) Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. 2022.  ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers.
  * Batzolis et al. (2021) Georgios Batzolis, Jan Stanczuk, Carola-Bibiane Schönlieb, and Christian Etmann. 2021.  Conditional image generation with score-based diffusion models.  _CoRR_ , abs/2111.13606.
  * Blattmann et al. (2023) Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. 2023.  Align your latents: High-resolution video synthesis with latent diffusion models.  _CoRR_ , abs/2304.08818.
  * Chen et al. (2023) Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. 2023.  Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation.  _CoRR_ , abs/2303.13873.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8440–8451. Association for Computational Linguistics.
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Quinn Nichol. 2021.  Diffusion models beat gans on image synthesis.  In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pages 8780–8794.
  * Ding et al. (2021) Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. 2021.  Cogview: Mastering text-to-image generation via transformers.  In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pages 19822–19835.
  * Esser et al. (2023) Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. 2023.  Structure and content-guided video synthesis with diffusion models.  _CoRR_ , abs/2302.03011.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014.  Generative adversarial nets.  In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 2672–2680.
  * Hertz et al. (2023) Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. 2023.  Prompt-to-prompt image editing with cross-attention control.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Ho et al. (2022a) Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey A. Gritsenko, Diederik P. Kingma, Ben Poole, Mohammad Norouzi, David J. Fleet, and Tim Salimans. 2022a.  Imagen video: High definition video generation with diffusion models.  _CoRR_ , abs/2210.02303.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel. 2020.  Denoising diffusion probabilistic models.  In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
  * Ho and Salimans (2022) Jonathan Ho and Tim Salimans. 2022.  Classifier-free diffusion guidance.  volume abs/2207.12598.
  * Ho et al. (2022b) Jonathan Ho, Tim Salimans, Alexey A. Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. 2022b.  Video diffusion models.  In _NeurIPS_.
  * Kang et al. (2023) Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. 2023.  Scaling up gans for text-to-image synthesis.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 10124–10134. IEEE.
  * Karras et al. (2023) Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. 2023.  Dreampose: Fashion image-to-video synthesis via stable diffusion.
  * Li et al. (2023) Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. 2023.  GLIGEN: open-set grounded text-to-image generation.  _CoRR_ , abs/2301.07093.
  * Liew et al. (2022) Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. 2022.  Magicmix: Semantic mixing with diffusion models.  _CoRR_ , abs/2210.16056.
  * Lin et al. (2022) Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. 2022.  Magic3d: High-resolution text-to-3d content creation.  _CoRR_ , abs/2211.10440.
  * Lu et al. (2023) Shilin Lu, Yanzhu Liu, and Adams Wai-Kin Kong. 2023.  TF-ICON: diffusion-based training-free cross-domain image composition.  _CoRR_ , abs/2307.12493.
  * Luo et al. (2023) Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. 2023.  Videofusion: Decomposed diffusion models for high-quality video generation.  _CoRR_ , abs/2303.08320.
  * Mou et al. (2023) Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. 2023.  Dragondiffusion: Enabling drag-style manipulation on diffusion models.  _CoRR_ , abs/2307.02421.
  * Nichol and Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal. 2021.  Improved denoising diffusion probabilistic models.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8162–8171. PMLR.
  * Nichol et al. (2022) Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. 2022.  GLIDE: towards photorealistic image generation and editing with text-guided diffusion models.  In _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , pages 16784–16804. PMLR.
  * Parmar et al. (2023) Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. 2023.  Zero-shot image-to-image translation.  In _ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH 2023, Los Angeles, CA, USA, August 6-10, 2023_ , pages 11:1–11:11. ACM.
  * Peebles and Xie (2022) William Peebles and Saining Xie. 2022.  Scalable diffusion models with transformers.  _CoRR_ , abs/2212.09748.
  * Poole et al. (2023) Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. 2023.  Dreamfusion: Text-to-3d using 2d diffusion.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022.  Hierarchical text-conditional image generation with CLIP latents.
  * Ramesh et al. (2021) Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021.  Zero-shot text-to-image generation.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8821–8831. PMLR.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022_ , pages 10674–10685. IEEE.
  * Ruiz et al. (2023) Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. 2023.  Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023_ , pages 22500–22510. IEEE.
  * Saharia et al. (2022a) Chitwan Saharia, William Chan, Huiwen Chang, Chris A. Lee, Jonathan Ho, Tim Salimans, David J. Fleet, and Mohammad Norouzi. 2022a.  Palette: Image-to-image diffusion models.  In _SIGGRAPH ’22: Special Interest Group on Computer Graphics and Interactive Techniques Conference, Vancouver, BC, Canada, August 7 - 11, 2022_ , pages 15:1–15:10. ACM.
  * Saharia et al. (2022b) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. 2022b.  Photorealistic text-to-image diffusion models with deep language understanding.
  * Schuhmann et al. (2022) Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 2022.  LAION-5B: an open large-scale dataset for training next generation image-text models.  In _NeurIPS_.
  * Singer et al. (2023) Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. 2023.  Make-a-video: Text-to-video generation without text-video data.  In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
  * Tang et al. (2023) Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and Dong Chen. 2023.  Make-it-3d: High-fidelity 3d creation from A single image with diffusion prior.  _CoRR_ , abs/2303.14184.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mt5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021_ , pages 483–498. Association for Computational Linguistics.
  * Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022.  Scaling autoregressive models for content-rich text-to-image generation.  _Trans. Mach. Learn. Res._ , 2022.
  * Zheng et al. (2022) Chuanxia Zheng, Tung-Long Vuong, Jianfei Cai, and Dinh Phung. 2022.  Movq: Modulating quantized vectors for high-fidelity image generation.  In _NeurIPS_.

##  Appendix A Additional generation examples