# Jaco: An Offline Running Privacy-aware Voice Assistant

Daniel Bermuth  ISSE
University of Augsburg
Augsburg, Germany
<EMAIL_ADDRESS>Alexander Poeppel  ISSE
University of Augsburg
Augsburg, Germany
<EMAIL_ADDRESS>Wolfgang Reif  ISSE
University of Augsburg
Augsburg, Germany
<EMAIL_ADDRESS>
###### Abstract

With the recent advance in speech technology, smart voice assistants have been
improved and are now used by many people. But often these assistants are
running online as a cloud service and are not always known for a good
protection of users’ privacy. This paper presents the architecture of a novel
voice assistant, called Jaco, with the following features: (a) It can run
completely offline, even on low resource devices like a RaspberryPi. (b)
Through a skill concept it can be easily extended. (c) The architectural focus
is on protecting users’ privacy, but without restricting capabilities for
developers. (d) It supports multiple languages. (e) It is competitive with
other voice assistant solutions. In this respect the assistant combines and
extends the advantages of other approaches.

###### Index Terms:

multilingual smart voice assistant; human-computer interaction; offline voice
assistant

##  I Introduction

Smart voice assistants have greatly improved over the last years and are now
helping in various tasks in many households. Often these assistants are
running online as a cloud service and are not always known for a good
protection of users’ privacy. On the other hand, some approaches exist which
try to improve the privacy aspect. One important concept in this regard is
that the assistant can run completely offline. In typical cloud solutions
users have no guarantee that the voice commands sent to online servers are
handled safely there. The only option would be to fully trust the cloud
provider.

Amazon’s Alexa [1] is a widely used voice assistant, which can understand
multiple languages. It is normally shipped with specialized hardware (the Echo
speakers) and offers a large skill store with both free and commercial skills.
Snips [2] was a voice assistant, capable of running offline, and still
achieving a high recognition accuracy. It had a small skill store where hobby
developers could share their skills. The company behind Snips was bought by
Sonos and the possibility to create own assistants was removed. Mycroft [3] is
a fully open source assistant and has, like Snips, a focus on preserving
privacy. For speech recognition Mycroft uses Google’s speech-to-text service.
Rhasspy [4] combines different services into a voice assistant capable of
running offline. The project is focused on creating a voice interface for home
automation software like Home Assistant. Unlike the other alternatives it does
not have a skill store for which developers can build specialized skills in
order to share them with others.

Besides the above mentioned assistants, several solutions exist that are
specialized on the speech to intent extraction, which is the most important
part of an assistant, regarding its command recognition performance. Google’s
DialogFlow, Microsofts’s LUIS and IBM’s Watson are cloud-based solutions and
Picovoice offers an offline running alternative with Rhino [5].

This paper presents a novel voice assistant that combines the benefits of the
aforementioned approaches. The main advantages of Jaco are: (a) It can run
completely offline, in contrast to Snips where only the usage was offline, but
the training of the assistant had to be done online, similar to the cloud-
based training of Rhino. Mycroft uses the online speech recognition service
from Google, and Alexa runs completely in the cloud. (b) By adding skills to
the assistant it can be easily extended with new features or integrated into
other frameworks like the Robot Operating System [6]. (c) While keeping
architectural focus on protecting users’ privacy, the developers are not
restricted in using all of the host device’s resources. (d) It supports
multiple languages, which currently are German, English, Spanish and French,
and can easily be extended with new ones. (e) It is competitive with other
voice assistant solutions and outperforms them in various benchmarks.

The complete assistant and the benchmark code can be found at:
https://gitlab.com/Jaco-Assistant

##  II Architecture

The general architecture of a typical voice assistant is shown in Figure 1.
Usually an interaction starts with a user speaking a keyword (here: computer)
that triggers the assistant to listen to the following spoken request. A
Speech-to-Text module transcribes the request and the transcription is sent to
a Natural-Language-Understanding module, which extracts the useful information
from the full sentence. The extracted intent and the entities are then handled
by a skill, which performs the appropriate action and informs the user upon
success, by sending a textual answer to a Text-to-Speech service, which
answers the user.

Figure 1: General architecture of a voice assistant. Figure 2: Process of a
voice interaction using Jaco.

The architecture of the proposed assistant Jaco is presented in greater detail
in Figure 2. The general architecture is split into two parts, one or multiple
lightweight Satellites, which can be installed in different rooms, and a
single central node, which handles the main processing tasks. The example
consists of two different Satellites (Alpha, Beta) and the Master. Both
satellites have a microphone from which the Audio Streamer reads the audio
input and streams it to the Wake Word modules. If they detect the wake word in
the stream, they send a message to the Dialog Manager. After checking which
Satellite detected the wake word first, the Dialog Manager sends an activation
request to the Speech to Text module. The transcription of the speech is then
started and continues until the user stops speaking. Afterwards the detected
text is sent to the Nlu Parser which extracts the intent and sends it back to
the Dialog Manager. From there it is sent to the skill which handles the
detected intent type. The skill can run various actions (like turning on the
light in the lab) and responds with a text message. The Dialog Manager then
forwards the message to the Text to Speech module. There the text is converted
into an audio file which afterwards is sent to the Satellite specified in the
message. The Audio Streamer then plays the file to the user through the
speaker.

All the modules are container-based and run in their own individual container,
which has the advantage of a reduced potential of conflicts with programs
already installed on the host system. All module requirements can also be
preinstalled into the containers, which greatly improves the installation
time, as users can download usage-ready container images, instead of building
them on their own device (which would take about $12\text{\,}\mathrm{h}$ on a
RaspberryPi-4).

For the wake-word recognition Porcupine [7] is used. The speech-to-text module
uses the pretrained models from Scribosermo [8], which are designed to run in
real-time on a RaspberryPi. Scribosermo was developed for usage with Jaco and
the source code can be found in the assistant’s repository group. The NLU-
parsing is done with Rasa [9] and Duckling [10], and the text-to-speech
generation uses picotts [11]. Communication between the modules is executed
over MQTT. The solutions were selected because they can run completely offline
on a RaspberryPi. An advantage of the module based approach of Jaco is that
the assistant doesn’t rely on specific solutions, so they can easily be
replaced by any other service. This for example allows other researchers to
replace specific modules with their current research, with the benefit that
they can test it in a complete workflow.

Two of the modules, the Nlu Parser and the Speech to Text module, require
additional training, depending on the skills installed by a user. For the
speech recognition, the acoustic model, which maps an audio input to character
possibilities, differs only between different languages, but a customized
n-gram language model (LM) is created, which rescores the character
predictions, using the example requests included in the skill files. The NLU
model also has to be retrained on the skill’s example requests. In both cases
a script collects the command examples from the installed skills and extends
them with automatically generated examples. This is done by inserting
different entity values into the example intents. Training both models usually
takes a few seconds on a computer and a few minutes on a RaspberryPi, but this
depends on the installed skills. Adding a new language would only require a
general purpose STT model for this language, as well as a TTS model, the rest
of the system is language independent.

##  III Skills

One of the most important features of a smart voice assistant is the
possibility to add new capabilities in form of user created skills. The
architecture of Jaco was designed in a way that this is very easy to achieve,
and, in contrast to the other assistants, the skills can also access all the
hardware resources of the executing device.

To create a skill, a developer first has to define possible user requests. The
syntax for the dialog examples is designed in a way that makes them easily
readable when they are inspected in the skill’s git repository. In the
following example, the sample request would only be displayed as “Book
(me$|$us) a flight from Augsburg to Berlin”, with the two cities highlighted
as links, which are referring to the city.txt file. The roles start and
destination, which are needed for a later distinction of the two cities in the
action code, are separated with a question mark so that the names of the roles
are ignored if a user clicks on the link. Words in parentheses define
alternatives or synonyms. About ten example sentences per intent is a good
amount to start with, but using more can slightly improve accuracy.

⬇

========================== city.txt ========================

Augsburg

(New York|N Y)->New York

Berlin

=========================== nlu.md =========================

## lookup:city

city.txt

## intent:book_flight

- Book (me|us) a flight from [Augsburg](city.txt?start) \

to [Berlin](city.txt?destination)

Besides that, the developer also has to create a python script which can
handle the incoming requests. To simplify development, all specialized message
interactions are handled by the jacolib library. After creating a skill it can
be shared with other users, by adding it to a skill store.

⬇

========================= action.py ========================

from jacolib import assistant

assist = assistant.Assistant()

def callback_book(msg):

locs = assist.extract_entities(msg, "myskill-book_flight")

locs = [lc["value"] for lc in locs]

if "munich" in locs:

r = "that wouldn’t be wise"

else:

r = "ok boss"

assist.publish_answer(r, msg["satellite"])

assist.add_topic_callback("book_flight", callback_book)

assist.run()

##  IV Improving privacy

As mentioned before, the most important aspect regarding privacy is that the
complete assistant can run entirely offline. In contrast to Snips, which
required training the assistant on an online server before it could be
downloaded, the training is executed completely offline on device as well.
Running everything on the user’s local device ensures that users don’t have to
trust their cloud provider that voice commands or accidentally recorded sounds
(if the wake-word was triggered through a false positive) are handled safely
there.

Besides that, multiple features were implemented to protect privacy when
third-party skills are used, without restricting the possible use-cases too
much. An important part of this is that all skills must include a
configuration file which implements a simple permission system. In the config
file all communication topics the skill wants to access have to be listed. By
design all MQTT-topics are automatically encrypted to ensure that a skill can
not read topics of other skills or the main modules. With the listing in the
config file a user can easily see which information the skill wants to read,
without restricting a skill’s capabilities if they require reading other
topics. For example, a weather skill should normally not require to read the
microphone recording stream, but a music playing skill might use this data to
automatically adjust the playback volume to background noises. In the config
file the developer also has to state if the skill needs internet access and if
it runs an action. An example of a skill without an action is one that
contains only specific dialog examples for sharing them with other skill
developers.

⬇

========================= config.yaml ======================

system:

has_action: true

extra_container_flags: ""

needs_internet_access: true

topics_read:

- "book_flight"

topics_write:

- "Jaco/Skills/SayText"

Unlike other assistants, where skills can not access the hardware resources of
the executing device directly (Alexa) or require extra setup steps (Snips and
Mycroft), skills for Jaco can fully access the device resources. To improve
the security of this feature, all skills are executed inside containers. This
also allows the automatic installation of arbitrary additional software
without the problem of creating software conflicts on the host device. In some
cases the containers require additional runtime flags to access specific
resources, which have to be added in the configuration, too.

An example would be a skill that wants to control GPIO-pins on a Raspberry Pi.
With Snips this could be achieved too, but the user had to install all
required system libraries (like WiringPi) and do the setup himself as the
skill could only install new python libraries. With Jaco the skill developer
can automate this through the skill container which then is granted access to
the necessary resources. Another example could be a skill that works as
interface for ROS where the skill runs the multi-step installation
automatically in its container. By allowing this skill to listen to system
topics, the interface can optionally be used to pause robot movements to
reduce disturbing noises while the user or system is speaking. A demo for both
skills can be found in the skill store.

If users want to install shared third-party skills from the official skill
store, they can directly see if skills request possibly dangerous access
permissions (Figure 3).

Figure 3: Store with permission notifications of the selected skill.

The store entry also includes a link to the git repository from where the
skill is downloaded later, so that a user can easily inspect the source code.
Using the official store is not mandatory, a skill’s code also can be shared
through any other provider. The store can run completely offline in the
browser after opening the website or can be rebuilt and executed from the
source release.

##  V Benchmarks

To compare the performance relative to other Speech and Language Understanding
(SLU) solutions, different benchmarks were performed. The first benchmark was
published by Picovoice [12] and consists of 620 commands of different people
ordering coffee in English. The audio is mixed with different volume levels of
background noise from cafe and kitchen environments. An example command would
be: “i’d like a [medium roast] [large] [mocha] with [lots of cream] and [a
little bit of brown sugar]”. A command is correctly detected if the intent, as
well as all the slots, could be retrieved by the assistant. The results are
shown in Figure 4. The benchmark shows that Jaco outperforms most other
solutions in medium and low noise settings.

Figure 4: Benchmark coffee orders with noisy backgrounds. The results of
DialogFlow, Watson, Luis and Rhino have been taken from [12].

Another often used capability of smart voice assistants is controlling lights
in different rooms. This was tested with the SmartLights benchmark from Snips
[13]. It consists of 1660 requests which are split into five partitions for a
5-fold training of the LM and NLU components. A sample command could be:
“please change the [bedroom] lights to [red]” or “i’d like the [living room]
lights to be at [twelve] percent”. Jaco could outperform all the other
comparable solutions (Table I).

Some notes on the execution: For Alexa, a new skill had to be created for each
of the five folds. Houndify [14] was tested because it claims “to deliver
unprecedented speed and accuracy” but did not provide any quantifiable
evidence for this claim. However it was only possible to use the pretrained
domain for smart light controls. Commands such as “it’s too dark in here”,
which should trigger the SwitchLightOn intent, were often not understood. For
a better comparison the Word-Error-Rate (WER) was measured as well. Houndify
and Jaco both had a WER of $10.8\%$, for Alexa no method for returning the
transcriptions was found.

TABLE I: Benchmark smart light control commands. | Accuracy | WER
---|---|---
Google [13] | $0.793$ | $-$
Snips [13] | $0.842$ | $-$
Alexa | $0.792$ | $-$
Houndify | $0.545$ | $0.108$
Jaco | 0.854 | $0.108$
AT-AT [15] | $0.849$ | $-$
SynSLU [16] | $0.714$ | $-$

The last benchmark (Table II) tests the performance of reacting to music
player commands in English as well as in French. The benchmark is from Snips
[13], too, and is the only one that could be found that includes a language
other than English. It has the difficulty of containing many artist or music
tracks with uncommon names in the commands, like “play music by [a boogie wit
da hoodie]” or “I’d like to listen to [Kinokoteikoku]”. In the English
benchmark none of the solutions performed very well. The authors of the Snips
benchmark did precompute pronunciation mappings for the artist names, which
was not done for Jaco. While benchmarking Alexa it could be noticed that
stylized artist names like ”Bonez MC” often were automatically matched to the
spelling of the originating words (”Bones MC”) and are therefore classified as
incorrect slot extractions.

In the French benchmark Alexa performed surprisingly well, the spelling
correction issue did not occur anymore. Snips performed well, too, the
generated pronunciations were automatically mapped to a French spelling. Jaco
had great problems with recognizing many of the artists’ names. Something like
a pronunciation map for the names could help here, but was not implemented.
But it is generally possible to ship such a pronunciation map with a skill, if
the skill author chooses to create one.

TABLE II: Benchmark music wishes in English and French. Accurracy: | English | French
---|---|---
Snips [13] | 0.687 | $0.751$
Google [13] | $0.478$ | $0.423$
Jaco | $0.627$ | $0.480$
Alexa | $0.455$ | 0.889

##  VI Conclusion

In this paper the full-featured voice assistant Jaco was presented. The
assistant combines and extends the advantages of other approaches. It is
competitive with other solutions and can run on single-board computers like a
RaspberryPi. The assistant has a strong focus on protecting users’ privacy,
and runs completely offline, so no user interactions are shared with any other
service.

## References

  * [1] Amazon, “Alexa,” 2021, [accessed 23-March-2021]. [Online]. Available: https://developer.amazon.com/en-US/alexa
  * [2] A. Coucke, A. Saade, A. Ball, T. Bluche, A. Caulier, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone, T. Lavril _et al._ , “Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces,” _arXiv preprint arXiv:1805.10190_ , 2018.
  * [3] Mycroft AI, “Mycroft,” 2021, [accessed 17-June-2021]. [Online]. Available: https://mycroft.ai/
  * [4] M. Hansen, “Rhasspy,” 2021, [accessed 08-October-2021]. [Online]. Available: https://rhasspy.readthedocs.io/en/latest/
  * [5] Picovoice, “Rhino,” 2021, [accessed 11-October-2021]. [Online]. Available: https://picovoice.ai/platform/rhino/
  * [6] Quigley, Morgan and Conley, Ken and Gerkey, Brian and Faust, Josh and Foote, Tully and Leibs, Jeremy and Wheeler, Rob and Ng, Andrew Y and others, “ROS: an open-source Robot Operating System,” in _ICRA workshop on open source software_ , vol. 3, no. 3.2. Kobe, Japan, 2009, p. 5.
  * [7] Picovoice, “Porcupine,” 2021, [accessed 23-March-2021]. [Online]. Available: https://picovoice.ai/platform/porcupine/
  * [8] D. Bermuth, A. Poeppel, and W. Reif, “Scribosermo: Fast Speech-to-Text models for German and other Languages,” _arXiv preprint arXiv:2110.07982_ , 2021\.
  * [9] Rasa Technologies Inc, “Rasa,” 2021, [accessed 23-March-2021]. [Online]. Available: https://rasa.com/
  * [10] Facebook Inc, “Duckling,” 2021, [accessed 09-June-2021]. [Online]. Available: https://github.com/facebook/duckling
  * [11] naggety, “picotts,” 2021, [accessed 23-March-2021]. [Online]. Available: https://github.com/naggety/picotts
  * [12] Picovoice, “Barrista Benchmark,” 2021, [accessed 4-June-2021]. [Online]. Available: https://github.com/Picovoice/speech-to-intent-benchmark
  * [13] A. Saade, A. Coucke, A. Caulier, J. Dureau, A. Ball, T. Bluche, D. Leroy, C. Doumouro, T. Gisselbrecht, F. Caltagirone _et al._ , “Spoken language understanding on the edge,” _arXiv preprint arXiv:1810.12735_ , 2018.
  * [14] SoundHound Inc., “Houndify,” 2021, [accessed 31-March-2021]. [Online]. Available: https://www.houndify.com/
  * [15] S. Rongali, B. Liu, L. Cai, K. Arkoudas, C. Su, and W. Hamza, “Exploring Transfer Learning For End-to-End Spoken Language Understanding,” _arXiv preprint arXiv:2012.08549_ , 2020.
  * [16] L. Lugosch, B. H. Meyer, D. Nowrouzezahrai, and M. Ravanelli, “Using speech synthesis to train end-to-end spoken language understanding models,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 8499–8503.


# Understanding the World to Solve Social Dilemmas using Multi-Agent
Reinforcement Learning

Manuel Sebastián Ríos & Nicanor Quijano
Department of Electrical and Electronic Engineering
Universidad de los Andes
Bogotá, Colombia
{ms.rios10<EMAIL_ADDRESS>
&Luis Felipe Giraldo
Department of Biomedical Engineering
Universidad de los Andes
Bogotá, Colombia
<EMAIL_ADDRESS>
We thank Google DeepMind and CINFONIA for partially funding this project
through the scholarship programme.

###### Abstract

Social dilemmas are situations where groups of individuals can benefit from
mutual cooperation but conflicting interests impede them from doing so. This
type of situations resembles many of humanity’s most critical challenges, and
discovering mechanisms that facilitate the emergence of cooperative behaviors
is still an open problem. In this paper, we study the behavior of self-
interested rational agents that learn world models in a multi-agent
reinforcement learning (RL) setting and that coexist in environments where
social dilemmas can arise. Our simulation results show that groups of agents
endowed with world models outperform all the other tested ones when dealing
with scenarios where social dilemmas can arise. We exploit the world model
architecture to qualitatively assess the learnt dynamics and confirm that each
agent’s world model is capable to encode information of the behavior of the
changing environment and the other agent’s actions. This is the first work
that shows that world models facilitate the emergence of complex coordinated
behaviors that enable interacting agents to “understand” both environmental
and social dynamics.

##  1 Introduction

Social dilemmas are situations where a group of individuals can benefit from
the cooperativeness of its members, but they are tempted to act selfishly to
satisfy their individual interests Komorita (2019). This conflict of interest
is common among many of humanity’s most critical challenges that include
global warming, pandemic preparedness, and inequality Dafoe et al. (2020).
Understanding which mechanisms foster the emergence of cooperative behaviors
that aid communities to solve social dilemmas is an important scientific
question. Game theory has traditionally used matrix games to model social
dilemmas. However, recent works have suggested to extend these matrix games
into more dynamic and complex environments that are typically implemented in
video game-like scenarios Leibo et al. (2017). In this new paradigm, multi-
agent RL-based algorithms have been used to model the decision-making of self-
interested agents, showing how a variety of collective behaviors can emerge
from these simulated environments Zheng et al. (2022). These behaviors have
also shed light on some relevant human traits that can potentially encourage
cooperation Hughes et al. (2018) Song et al. (2022).

Despite that these recent works have focused their efforts on using model-free
RL algorithms in multi-agent systems, theoretical analyses in social
psychology have suggested that, in a group, an individual’s actions are driven
by his personal qualities and his own understanding of the social and changing
environment they are in Forsyth (2018). Thus, we hypothesize that learning
world models is key in multi-agent RL to study the emergent behaviors of
agents that are in environments where social dilemmas arise, as world models
enable each agent to “understand” the dynamics of a changing environment that
involves social interactions. RL algorithms based on world models have gained
special attention from the research community in the context of visual control
and robotics Ha & Schmidhuber (2018) Hafner et al. (2020). These model-based
algorithms aim to build abstract, compact, low-dimensional representations
that embed the environment’s dynamics. Some authors consider that world models
constitute the basis of the common sense and are essential for building
autonomous machine intelligence LeCun (2022). To our knowledge, there are no
reported works that study the social dynamics of RL agents that learn world
models.

We simulated the common-pool resource appropriation problem as a case of
study. In this setting, a group of individuals simultaneously exploit a common
resource, and it is impossible for them to exclude other individuals from
using it Perolat et al. (2017). Generally, the resource is non-renewable or
takes considerable time to renew. Therefore, exploitation decreases the
available amount of resources for other individuals. A sustainable community
must act in a coordinated manner, avoiding complete resource depletion.
Failing to do so is commonly known as the tragedy of the commons. Our results
show that world model-based RL algorithms outperform all other methods in the
simulated scenarios. While the model-free algorithms fail in the task by
continuously falling into the tragedy of the commons, world model-based
algorithms are able to find sustainable consumption strategies. Additionally,
the use of world models allows us to qualitatively assess the learnt dynamics.
In this case, we show that the world model is able to encode both
environmental and social dynamics. These results are consistent with
theoretical models of groups in social psychology Forsyth (2018) and support
current trends in artificial intelligence research LeCun (2022).

##  2 Related Work

Figure 1: Testing environment with two independent self-interested agents and
a single apple patch.

Social sequential dilemmas (SSDs) Leibo et al. (2017) can be considered as the
first framework that extends social dilemmas from the classical matrix game
perspective to video game-like 2D simulations. SSDs maintain the mixed
motivation structure in matrix games but additionally seek to better capture
some crucial aspects of real social dilemmas that include: i) social dilemmas
are temporally extended; ii) cooperation and deflection are labels that should
be assigned to policies instead of atomic actions; and iii) deciding whether
to cooperate or deflect is done quasi-simultaneously and based only on partial
information from the environment and other individuals’ actions. On the other
hand, finding optimal behaviors in these scenarios is computationally
expensive, and requires considering the agent’s high dimensional observation
space. This impedes the use of classical optimization or learning techniques.
Therefore, the authors propose to use reinforcement learning algorithms to
train self-interested agents in these simulated scenarios.

SSDs inspired several works that aimed to mimic some specific human traits to
improve social capabilities in simulated populations using model-free
reinforcement learning algorithms. For instance, Hughes et al. (2018) showed
that modeling envy and guilt will promote the emergence of cooperative
behaviors. Also, Jaques et al. (2019) shows that rewarding agents for having
causal influence over other agents’ actions promotes the emergence of
coordinated behaviors, and Ndousse et al. (2021) show that simulated agents
can benefit from the presence of expert agents to achieve complex behaviors
that con not be obtained from single agent training. These video game-like 2D
environments have also been used to simulate complex economic interactions.
The work in Zheng et al. (2022) simulated a small group of self-interested
agents that aimed to increase their individual income by trading, collecting,
and exploiting resources. Additionally, the authors trained a planner agent to
design optimal taxation policies seeking to increase the population’s welfare
and productivity. Remarkably, the taxation policy that emerged from this
simulation outperformed many human-crafted policies.

These promising results have encouraged researchers to develop evaluation
protocols and testing suits Leibo et al. (2021) Johanson et al. (2022). Some
researchers have also coined the term Cooperative AI Dafoe et al. (2020) to
the study of those mechanisms that make possible the emergence of cooperation
in AI-based systems, and have made an effort to identify open problems and
challenges in the field. The latter aims to guide future research in the use
of AI to solve problems of cooperation in both simulated and real scenarios.

##  3 Experimental Setup

Figure 2: Performance of populations trained with different algorithms on the
two agent environment shown in Fig. 1. The performance is measured using an
efficiency metric that represents the population’s per-capita consumption. The
shadowed area represents the standard deviation of the population performance
over multiple experiment runs, and the solid line represents the mean
performance. The dashed line represents the expected efficiency of a
population of random agents.

We developed our testing environment based on DeepMind’s Melting Pot suit
Leibo et al. (2021). Inspired by Perolat, et al. Perolat et al. (2017), we
simulated the common-pool resource appropriation problem using the grid-world
shown in Fig. 1. In this environment, agents receive a positive reward for
each apple consumed, and the apple’s regrowth probability is directly
proportional to the number of uneaten apples in a predefined radius.
Therefore, agents must coordinate to keep at least one apple on the patch to
avoid complete depletion. Agents can also use a laser beam to temporally
remove other agents from the environment. Unlike previous works, our
environments use a smaller regrowth probability and a lower apples-to-agents
ratio to make the environment a lot more challenging to deal with. Agents have
partial observations of their environment, where they observe a small portion
of the environment centered on their positions. agents must learn optimal
policies directly from raw images.

We trained independent agents using DreamerV2 Hafner et al. (2020), which is a
world model-based reinforcement learning algorithm that uses a recurrent
state-space model Hafner et al. (2019) to learn the environment’s dynamics.
DreamerV2 encodes these dynamics in a sparse low dimensional discrete latent
state and learns optimal behavior policies by using these representations
instead of the real environment’s observations. We compare the performance of
populations trained with DreamerV2 with populations of agents that use both
off-policy (Recurrent Replay Distributed DQN R2D2) and on-policy (Proximal
Policy Optimization PPO) RL algorithms. It is important to note that, unlike
DreamerV2, both PPO and R2D2 do not use world models. To ensure a fair
comparison, we trained all algorithms for 2000 episodes, following the
parameters proposed by the authors in the original implementations or the
parameters used in the libraries employed. We use the population’s efficiency
as a performance metric. Given a population of $N$ agents and their respective
sum of rewards $R_{i}$, the efficiency metric can be computed as:

$\text{Efficiency}=\frac{\sum_{i=1}^{N}R_{i}}{N}.$

Intuitively, the efficiency represents the population’s per-capita
consumption. Fig. 2 depicts the performance of all the tested populations. Our
results clearly show how the agents trained using DreamerV2 outperform both
PPO and R2D2. We provide videos of the learnt policies for all the trained
agents (supporting video) and all the source code used to conduct this
research 111https://github.com/ManuelRios18/Commons-Tragedy.

One of the key features of DreamerV2 is that each agent after the learning
process is able to predict hypothetical future state sequences from a single
initial observed state. That is, the world model. Therefore, it is possible to
qualitatively assess the agent’s learnt behaviors by predicting state
sequences from key initial observations. Fig. 3 shows the final predicted
states computed from initial states with different densities of uneaten apples
and the presence of other agents. These results suggest that the algorithm is
able to properly encode the environment’s dynamics. The apples’ density in the
final state is directly proportional to the density in the initial state.
Additionally, DreamerV2 is also able to model other agents’ behavior by
predicting their intentions of consuming the apples. Moreover, the model also
predicts future attacks to the other agents, proving that the model
understands that it is beneficial to reduce the effective population, given
that this allows the agent to consume the apples without taking the risk of
being taken out of the environment.

Figure 3: Initial and final states sampled from predicted trajectories when:
(a) there is a high density of uneaten apples in the initial states; (b)
uneaten apples are scarce in the initial states; (c) there are interactions
with the other agent (the yellow shaded area in front of the agent is the
laser beam). $\Delta t$ denotes the number of states between the initial and
final states. Figure 4: t-SNE projection of different latent states, and the
observation sequence used to compute some of these latent states. Each state
is colored according to its corresponding state value.

Fig. 4 shows the t-SNE projection of many latent states obtained with a
trained agent. The colors represent the state value, which can be interpreted
as a scalar number that indicates how good the agent is to be in a given state
in terms of the expected sum of rewards. Our results show that the learnt
model groups together states that share similar state values and also
environmental and social dynamics. For instance, the cluster located in the
upper middle section of Fig. 4 is composed of observation sequences where the
agents interacted with each other. The risk of being too close to the other
agent, the low apple density, and receiving a direct attack may explain the
low value assigned to this set of latent states.

In the case of the common-pool appropriation problem, understanding the
environmental dynamics helps the agents to avoid eating the last apple in the
patch. As shown in our supporting videos, when there is a single apple left,
the agents patiently wait for the other apples to grow back and never
completely deplete the patch. Fig. 3 shows that DreamerV2 excels at modeling
other agents’ intentions. The predicted trajectories show that the other
agents are perceived as a threat that must be attacked. Moreover, some
predicted states show these agents in the middle of the map consuming the
apples. Generally, the other agent is represented as a blurry patch over many
adjacent cells showing that the model captures the uncertainty associated with
the other’s actions. Additionally, our t-SNE projection shows that the states
are clustered based on both the presence of the other agents and the current
apples’ density. This suggests that both social and environmental dynamics
shape the learnt policies.

##  4 Discussion

The results presented in this study suggest that world models can considerably
ease the emergence of coordinated behaviors in self-interested individuals.
The fact that the world models encode social and environmental dynamics and
that the agents exploit this information to compute sustainable behavior
policies, is consistent with theoretical models in social psychology and
current artificial intelligence research directions LeCun (2022).
Understanding both the environmental dynamics and the intentions of the other
agents is considered one of the key elements in cooperative intelligence Dafoe
et al. (2020) and was crucial for the emergence of these complex coordinated
behaviors.

Also, we highlight the use of discrete representations to encode the world’s
dynamics. Ma et al. (2022) argue that parsimony is a cornerstone for the
emergence of intelligence, and Gomez et al. (2022) showed the benefits of
using discrete representations to facilitate the ability to generalize to
novel situations. In this case, using discrete and sparse arrays as latent
states of the world models ensures compactness and simplicity, potentially
allowing agents to model more challenging social scenarios.

We consider that this is a promising research direction that can lead to
intelligent systems to aid decision-makers in complex real-world challenges
that involve social dynamics.

## References

  * Dafoe et al. (2020) Allan Dafoe, Edward Hughes, Yoram Bachrach, Tantum Collins, Kevin R McKee, Joel Z Leibo, Kate Larson, and Thore Graepel.  Open problems in cooperative ai.  _arXiv preprint arXiv:2012.08630_ , 2020.
  * Forsyth (2018) Donelson R Forsyth.  _Group dynamics_.  Cengage Learning, 2018.
  * Gomez et al. (2022) Diego Gomez, Nicanor Quijano, and Luis Felipe Giraldo.  Information optimization and transferable state abstractions in deep reinforcement learning.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Ha & Schmidhuber (2018) David Ha and Jürgen Schmidhuber.  Recurrent world models facilitate policy evolution.  _Advances in neural information processing systems_ , 31, 2018.
  * Hafner et al. (2019) Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels.  In _International conference on machine learning_ , pp. 2555–2565. PMLR, 2019.
  * Hafner et al. (2020) Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba.  Mastering atari with discrete world models.  _arXiv preprint arXiv:2010.02193_ , 2020.
  * Hughes et al. (2018) Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Antonio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al.  Inequity aversion improves cooperation in intertemporal social dilemmas.  _Advances in neural information processing systems_ , 31, 2018.
  * Jaques et al. (2019) Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas.  Social influence as intrinsic motivation for multi-agent deep reinforcement learning.  In _International conference on machine learning_ , pp. 3040–3049. PMLR, 2019.
  * Johanson et al. (2022) Michael Bradley Johanson, Edward Hughes, Finbarr Timbers, and Joel Z Leibo.  Emergent bartering behaviour in multi-agent reinforcement learning.  _arXiv preprint arXiv:2205.06760_ , 2022.
  * Komorita (2019) Samuel S Komorita.  _Social dilemmas_.  Routledge, 2019.
  * LeCun (2022) Yann LeCun.  A path towards autonomous machine intelligence.  _preprint posted on openreview_ , 2022.
  * Leibo et al. (2017) Joel Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel.  Multi-agent reinforcement learning in sequential social dilemmas.  05 2017.
  * Leibo et al. (2021) Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel.  Scalable evaluation of multi-agent reinforcement learning with melting pot.  In _International Conference on Machine Learning_ , pp. 6187–6199. PMLR, 2021.
  * Ma et al. (2022) Yi Ma, Doris Tsao, and Heung-Yeung Shum.  On the principles of parsimony and self-consistency for the emergence of intelligence.  _Frontiers of Information Technology & Electronic Engineering_, 23(9):1298–1323, 2022.
  * Ndousse et al. (2021) Kamal K Ndousse, Douglas Eck, Sergey Levine, and Natasha Jaques.  Emergent social learning via multi-agent reinforcement learning.  In _International Conference on Machine Learning_ , pp. 7991–8004. PMLR, 2021.
  * Perolat et al. (2017) Julien Perolat, Joel Z Leibo, Vinicius Zambaldi, Charles Beattie, Karl Tuyls, and Thore Graepel.  A multi-agent reinforcement learning model of common-pool resource appropriation.  _Advances in neural information processing systems_ , 30, 2017.
  * Song et al. (2022) Zhao Song, Hao Guo, Danyang Jia, Matjaž Perc, Xuelong Li, and Zhen Wang.  Reinforcement learning facilitates an optimal interaction intensity for cooperation.  _Neurocomputing_ , 513:104–113, 2022.
  * Zheng et al. (2022) Stephan Zheng, Alexander Trott, Sunil Srinivasa, David C Parkes, and Richard Socher.  The ai economist: Taxation policy design via two-level deep multiagent reinforcement learning.  _Science advances_ , 8(18):eabk2607, 2022.