# Learning Multi-Object Symbols for Manipulation with Attentive Deep Effect
Predictors

Alper Ahmetoglu1, Erhan Oztop2,3, and Emre Ugur1 1Department of Computer
Engineering, Bogazici University
<EMAIL_ADDRESS>of Computer Science,
Ozyegin University<EMAIL_ADDRESS>SISReC, Osaka University

###### Abstract

In this paper, we propose a concept learning architecture that enables a robot
to build symbols through self-exploration by interacting with a varying number
of objects. Our aim is to allow a robot to learn concepts without constraints,
such as a fixed number of interacted objects or pre-defined symbolic
structures. As such, the sought architecture should be able to build symbols
for objects such as single objects that can be grasped, object stacks that
cannot be grasped together, or other composite dynamic structures. Towards
this end, we propose a novel architecture, a self-attentive predictive
encoder-decoder network with binary activation layers. We show the validity of
the proposed network through a robotic manipulation setup involving a varying
number of rigid objects. The continuous sensorimotor experience of the robot
is used by the proposed network to form effect predictors and symbolic
structures that describe the interaction of the robot in a discrete way. We
showed that the robot acquired reasoning capabilities to encode interaction
dynamics of a varying number of objects in different configurations using the
discovered symbols. For example, the robot could reason that (possible
multiple numbers of) objects on top of another object would move together if
the object below is moved by the robot. We also showed that the discovered
symbols can be used for planning to reach goals by training a higher-level
neural network that makes pure symbolic reasoning.

##  I INTRODUCTION

If a task of a robotic manipulator can be described in symbolic form, then the
problem of obtaining a plan to achieve a task goal can be solved by reverting
to search techniques from classical AI [1]. In well-defined environments, a
symbolic system describing the interactions of a robot with its environment
can be designed manually. However, for unknown, unstructured and/or changing
environments, it is desirable that the robot itself discovers the symbolic
structures that are useful in reasoning and planning. As such, there is
considerable literature on how the continuous sensorimotor experience of a
robotic system can be converted to a set of symbolic interaction experiences
[2]. Although the state-of-the-art in this frontier is progressing rapidly,
currently, there is no learning system that can form symbols for a varying
number of objects in one learning session. In our previous work [3], we have
shown that the symbols formed with single object interactions can be used to
bootstrap new symbol or rule formation while interacting with two objects.
However, the transition from a single object to two objects required the
construction of a new neural network. Furthermore, it was not possible to
learn symbols from interactions that involve a varying number of objects, some
of which may affect the action execution and others not.

In this study, we aim to remove these limitations by acquiring a single neural
system in order to discover symbols based on the sensorimotor data generated
by the robot when it interacts with multiple objects with arbitrary
multiplicity. To this end, we propose a deep neural architecture that includes
self-attentive layers [4] with binary latent representations. We show the
validity of the proposed architecture in a simulated manipulation scenario
where a robotic arm interacts with object(s) while building symbolic
representations. Most importantly, our system not only learns object-specific
symbols but also learns multi-object symbols that are formed on-the-fly by
automatically processing the related object symbols through the self-attention
mechanism. After learning, we investigated the formed symbols and observed
that they are effective in making effect predictions. We showed that the
learned symbols enable reasoning capabilities with multiple objects that may
influence the interaction dynamics in various ways. We finally showed that the
discovered symbols can be used for planning to reach desired goals, such as
building desired structures with multiple objects via pure symbolic reasoning.

Figure 1: Attentive DeepSym architecture.

##  II RELATED WORK

Early symbol grounding studies in robotics (e.g., [5, 6]) assumed the
existence of manually defined symbols that were effective in plan generation.
These studies collected data from interactions of agents and robots, and
learned sensor to symbol mappings in order to ground the pre-defined symbols
in the sensorimotor experience of the robot. In these studies, transition
rules, which are connected by symbolic preconditions and effects were defined,
and the continuous experience of the robot were used to map the manually
defined symbolic predicates to the continuous perceptual space of the robots.
Recently, [7] proposed a deep neural network architecture that was based on
Convolutional Variational Auto-Encoders to discover visual features that well-
suit for pre-defined recognition and interaction tasks. [8] used Multi-modal
Latent Dirichlet Allocation (MLDA) to learn the mapping between multi-modal
sensory experience and preconditions and post-conditions of actions of a
robot. We argue that pre-defining symbols in unknown and changing environments
is not possible, and as stated by [9] symbols should rather “be formed in
relation to the experience of agents, through their perceptual/motor
apparatuses, in their world and linked to their goals and actions”.

Unsupervised discovery of discrete symbols and rule learning from the
continuous sensorimotor experience of embodied agents has been recently
studied in robotics in order to equip robots with advanced reasoning and
planning capabilities [2, 10]. [11] investigated the discovery of sub-symbolic
neural activations that facilitate resource economy and fast learning in skill
transfer but did not address high-level reasoning with discrete symbols. [12,
13] discovered symbols that were directly used as predicates in precondition
and post-condition fields of action descriptors, which were represented in
Problem Domain Definition Language (PDDL). This encoding allowed for making
deterministic and probabilistic plans in 2-dimensional agent environments. The
same architecture was extended to a real-world robotic environment in [14],
where symbols that represent absolute global states were learned and used for
planning. [15], on the other hand, learned egocentric symbolic representations
that enabled the agents to transfer the previously learned symbols to novel
environments directly. Effect clustering techniques and SVM classifiers were
used to discretize the continuous sensorimotor experience of the agents in
these works. Whereas the previous work addressed learning symbols from given
skills, [16] learned a set of skills from a set of symbolic predicates and a
collection of demonstrations. [17, 18] discovered discrete symbols and used
these symbols in order to generate PDDL rules for planning by again combining
effect clustering techniques to find discrete effect categories and SVM
classifiers to discretize continuous object feature space. These studies used
ad-hoc combinations of several machine learning methods. On the other, [3]
provided a more generic symbol formation engine, which used a novel deep
network architecture that runs at the pixel level, and relied on purely
predictive mechanisms in forming symbols instead of unsupervised clustering
techniques. They used an effect predictor encoder-decoder network that took
the object image and action as input, and exploited a binary bottleneck layer
to automatically form object categories. Similar to this work, [19, 20, 21]
also exploited deep neural networks with binary bottleneck units to find
discrete state and effect symbols and achieve plan generation using these
symbols. They realized symbolic planning in purely visual environments such as
2D puzzles, and they did not address actions and interactions of embodied
agents. While the majority of the previous work formed symbols to auto-encode
the continuous state or make effective one-step-ahead effect predictions, [22]
proposed an architecture that learns state and action symbols that were
explicitly optimized for effective and efficient multi-step planning.
Different from our work, they assumed already defined goal predicates and a
collection of demonstrations that enabled the robot to achieve the given
goals.

Our ultimate aim with this model is to learn symbolic representations for
objects by predicting the generated effect similar to the DeepSym architecture
[3]. The main difference from DeepSym is that the proposed architecture can
model the interaction between a varying number of objects by allowing symbols
to interact with each other via self-attention layers [4]. This not only
allows object symbols to gather information from each other but also allows
the architecture to be applicable to any number of interactions. For example,
while the effect (in our case, the displacement of the object) of a push
action to an object would require the symbol of the object (together with the
action vector), the effect of stacking an object on top of another one would
automatically require symbols of multiple objects. In the DeepSym
architecture, this was only possible by manually concatenating cropped images
of a fixed number of objects, which is not a scalable and general approach.

##  III METHOD

(a) The experiment setup.

(b) Example exploration.

Figure 2: The experiment setup. Six possible pick and release locations are
shown in purple in the synthetic camera image in (a). An example exploration
with three objects is shown in (b).

The proposed architecture is shown in Fig. 1. This architecture predicts the
effects of an executed action at a given state. The architecture consists of
an encoder $f(x)$ with a binarized bottleneck layer, a decoder $g(x)$, and an
attentive module $a(x)$. We assume that we have an image segmentation module
for cropping and tracking objects through several timesteps. While this is a
strong assumption, state-of-the-art segmentation methods in computer vision
can segment and track objects in complex environments [23, 24].

In the first part, the high-dimensional continuous state $x$ is first
partitioned into segments $(x_{1},x_{2},\dots,x_{k})$ using the segmentation
module. As we expect to obtain relational symbols that potentially encode
information related to several objects, the relative positions of the objects
are important and need to be preserved while processing image segments. In
order not to lose the location information, here we concatenate the location
of each pixel in the cropped image as two additional channels. Then, each
segmented image ($x_{i}$) is given as input to the encoder. The encoder
consists of several convolutional layers for processing high-dimensional
visual input. For binarization at the last layer of the encoder, we use the
Gumbel-sigmoid function [25, 26] for gradient backpropagation. In the end, the
encoder outputs a binary vector $z_{i}$ for each segment $x_{i}$. These binary
vectors ($z_{i}$) are expected to encode symbols that can encode one or more
objects and that are effective in predicting the effects of actions and,
therefore, multi-step planning.

Each action is represented with a one-hot binary vector. Actions are assumed
to be high-level parameterized motion primitives. The action vector $a$ is
concatenated with each $z_{i}$ separately. Concatenated vectors
$((z_{1},a),(z_{2},a),\dots,(z_{k},a))$ are given as input to the attentive
module. The attentive module consists of several self-attentive layers [4]
which allows the generated symbols $(z_{1},z_{2},\dots,z_{k})$ to interact
with each other to generate $(h_{1},h_{2},\dots,h_{k})$ that should hold
relational information for accurate effect prediction. Note that we assumed
the existence of action primitives in this work. These primitives can be
learned with different motion primitive methods [27, 28, 29] and transferred
from previous stages of development as we previously showed [30].

The self-attention operation allows each symbol to attend to other symbols to
form a new representation. Since the whole model is trained in an end-to-end
fashion, symbols are formed in such a way that they not only define the
characteristics of the cropped object but also contain information about
relations with other objects. For example, if a long stick is picked and
released to a different location, and if there is a small cube on top of the
stick, then the position of the cube changes as well. To accurately predict
such relational effects, the model needs to encode information regarding
relations between objects. Self-attention seamlessly combines this information
using query-key-based attention operation proposed in [4].

As the last step, the decoder function predicts the generated effect for each
segment. In our case, we represent the effect eff$(x_{i})$ as the position
displacement of the $i$th object after the action. In our experiments, we
directly retrieve the position displacement from the simulator. However, one
can use a more generic effect such as the change in pixel values as in [3].
The learned symbols heavily depend on the effect representation, and we treat
effect representation as a separate problem. This work, instead, focuses on
creating an architecture that can learn symbols for a varying number of inputs
and predict action effects that would require processing relational
information between objects.

##  IV EXPERIMENTS

###  IV-A Experiment Setup

The experiment setup is shown in Fig. 2. This is a tabletop environment where
a UR10 robot arm picks up and releases objects at six pre-defined locations
shown in Fig. 2(a). Given six pick-up and six release locations, the total
number of high-level actions is 36. An example sequence of action executions
is shown in Fig. 2(b). The environment is initialized with one to three
objects initially located at the row closer to the robot body (see the top
three purple dots in Fig. 2(a)). The robot perceives its environment with a
depth camera located on top of the table. The camera takes $256\times 256$
pixels depth image. We assume that the robot has a segmentation module that
can crop and track the movement of objects. This can be achieved with state-
of-the-art slot-based models [23, 24]. In this work, we use the segments
provided by the simulator. An example crop is shown in Fig. 1 (i). All crops
are $64\times 64$ pixels. Finally, in order to preserve object locations, the
$x$ and $y$ locations of each pixel in the cropped image are concatenated as
two additional channels.

The robot collects the interaction data set as follows. The environment is
initiated with $k\in\\{1,2,3\\}$ objects where $k$ is set randomly. The
initial depth image $x_{i}$ of the environment together with its segmentation
$s_{i}$ is recorded before taking any action. Then, a random action $a_{i}$ is
executed and position displacements of $k$ objects $e_{i}=([\Delta
x_{1},\Delta y_{1},\Delta z_{1}],\dots,[\Delta x_{k},\Delta y_{k},\Delta
z_{k}])_{i}$ are recorded as the generated effects. Here, $x$ and $y$
represent the pixel coordinates of the object’s center of mass, and $z$
represents the depth value obtained from the depth camera for the
corresponding object. Note that the length of $e_{i}$ depends on the number of
objects. In total, 12,000 $(x_{i},s_{i},a_{i},e_{i})$ tuples are recorded as
the interaction data set. We use 10,000 samples for training, 1,000 samples
for validation, and the remaining 1,000 samples for testing.

We train the architecture in Fig. 1 to predict the generated effects $e_{i}$
of an action $a_{i}$. The encoder $f(x)$ consists of four convolutional layers
with 64, 128, 256, and 512 filters. Convolutional layers are similar to that
of DCGAN [31] with a kernel size of four, a stride of two, and a padding of
one. After convolutions, we take an average across height and width
dimensions, and project this fixed-size vector into an 8-dimensional vector.
Lastly, we binarize the activations using Gumbel-sigmoid function for
backpropagation [26, 25]. The self-attention module $a(x)$ is a transformer
with four transformer encoder layers [4]. We use the default settings in
PyTorch for transformer layers [32]. The decoder $g(x)$ is a multi-layer
perceptron (MLP) with three hidden layers, each containing 256 units. We use
batch-normalization [33] in the encoder and the decoder to increase the
convergence speed. We train all modules in an end-to-end fashion with Adam
optimizer [34].

Figure 3: Effect prediction for different states.

###  IV-B Effect Prediction

Our system predicts the continuous effect, i.e., continuous pixel position and
depth changes of objects, from learned discrete symbolic activations. It is
neither our aim nor possible to minimize the prediction error to zero, yet we
need to make sure that our system can make discover symbols that can make
fairly good effect predictions. Therefore, we analyzed the prediction error.
After training, the mean effect prediction errors on the test set are 8.6mm,
15mm, and 6mm for $x$, $y$, and $z$ dimensions, respectively. For comparison,
the average changes of the corresponding dimensions of objects in the effect
set are 50mm, 68mm, and 10mm. As a result, we can conclude that our system
discovered symbolic representations that are effective in predicting effects
of actions on single or multiple objects.

TABLE I: Effect predictions for example cases in Fig. 3. Units are in millimeters. | Green Cube | Orange Stick | Red Cube
---|---|---|---
Predictions
Case 1 | (3, 6, -2) | (-2, 192, -2) | (4, -9, -1)
Case 2 | (3, 8, 0) | (-17, 196, 1) | (-10, 180, -5)
Case 3 | (-3, 210, -45) | (4, 159, 8) | (-30, 131, -20)
Ground truth
Case 1 | (0, 0, 0) | (0, 180, 0) | (0, 0, 0)
Case 2 | (0, 0, 0) | (0, 180, 0) | (-1, 176, 0)
Case 3 | (102, 177, -75) | (3, 180, 0) | (2, 176, 0)

Next, we investigate whether our system discovered symbols that automatically
include information from action-relevant objects in multi-object settings, and
model interaction dynamics of these objects. For this, we created a scenario
where the action applied to an object is the same, but the other objects in
the environment are arranged in different ways such that different effects are
expected to be obtained and are expected to be correctly predicted by our
system. The action is to pick up a long stick from one position and release it
to another position, and two other objects are placed in different
configurations, as shown in Fig. 3. In Fig. 3, three different example
interactions experienced by the robot were shown in three columns. The initial
snapshot of each interaction is shown in the upper row, and the final snapshot
after action execution is shown in the bottom row. In all three cases, the
robot picks up the orange stick, moves it to one position right, and releases
it (see the bottom-right purple position in Fig. 2(a)). Due to different
initial configurations of the red and green blocks, different effects are
(expected to be) observed.

The effects predicted by our system along with ground-truth effect values are
provided in Table I for each interaction (case). The results show us that our
system was able to model relational information between objects and therefore
made correct predictions. For example, in Case 1, the system correctly
predicted that only the position of the stick changes. In Case 2, the position
of the red cube also changes along with the stick. And finally, in Case 3, the
positions of both cubes change along with the stick (in the same direction).
This shows that our system learned symbols that enable it to make high-level
reasoning that involves multiple objects, such as “objects on top of another
object will move together with the object below”. We conclude that self-
attention in transformer layers indeed helps the binary symbols to interact
with each other to predict the correct effect. In the previous work, DeepSym
[3], the modeling of such interactions was only possible at the input level by
manually concatenating the necessary object crops with a pre-defined number of
objects. Here, due to the self-attention layers, the model discovers symbols
that automatically use the binary activations of the related objects and
disregards the activations of unrelated objects. This is a clear advantage
when compared with the DeepSym architecture.

Figure 4: The average depth images for each symbol sorted by their activation
count. Only the symbols that are activated more than 100 times out of 28,607
samples (the number of objects in 10,000 samples) are shown in the figure. The
symbol and its activation count are reported on the left, and the average
x-axis and y-axis pixel locations are reported on the top of each subfigure.
The color bar shows the normalized depth value. Figure 5: Example images that
activate the symbol ‘00011011’. The common feature in these examples is that
the object is occluded by another object. Notice that this symbol is activated
for both sticks and cubes. Since the robot cannot grasp the occluded object
and therefore cannot change its position. This symbol can rather be
interpreted as ‘not graspable’.

###  IV-C Learned symbols

The number of binary units in the post-encode bottleneck layer is set to 8
units; therefore, there can be at most 256 unique symbols. As we analyzed the
activation frequency of these symbols, we observed that the most frequently
activated 35 symbols covers 95% of the training set. The prototypical values,
i.e., the average states from samples that activate these 35 specific object
symbols, are shown in Fig. 4. As shown, these object symbols encode the
location of the object, the depth of the object, and the occlusion
information.

While a number of symbols are activated only for sticks or only for cubes,
others are activated for both cubes and sticks. For example, the symbol in the
second row and the first column is activated for both cubes and sticks (see
samples in Fig. 5). This makes sense since the generated effect is the same in
both cases; the robot cannot grasp the occluded object, and therefore, cannot
change its position. We can interpret this as a ‘not graspable’ symbol.
However, note that there is not a specific symbol for ‘not graspable stick’ or
‘not graspable cube’ since it does not bring any additional advantage in terms
of effect prediction accuracy; the model can already predict the generated
effect (which is 0mm, 0mm, 0mm on average) without differentiating these two
objects. We can conclude that our system learned a minimal set of symbols that
is required to make predictions and reasoning, and these symbols were not only
determined by the available objects in the environment but also by the action
capabilities of the robot.

###  IV-D Planning with Discovered Symbols

In this section, we investigate the suitability of symbolic representations in
making multi-step plans. Although our system can predict the effects on
objects given the tabletop image and the action, here, we aim to acquire a
full symbolic reasoning capability by learning and exploiting a transition
model that predicts the next symbolic state given the current symbolic state
and the action. For this, we train a separate network $m(z)$ that predicts the
next symbolic state given the current symbolic state and the action (see Fig.
6) using the discovered symbolic representations that were presented in the
previous sections. This network consists of two dense layers followed by a
single self-attention layer and another two dense layers. We train the network
with binary cross entropy loss for each dimension.

Figure 6: Symbolic forward module. Given the symbolic representation of the
current state, this module directly predicts the symbolic representation of
the next state.

Such a network, after training, allows us to search for a goal symbolic state
using any tree search algorithm [1]. Fig. 7 shows a plan generated and
executed in order to achieve a composite structure given as a goal. As shown,
the learned symbolic transition model could be used to generate feasible plans
and, when executed, could achieve the desired goal, indicating that multi-step
predictions in the generated long horizon plan were correct. This example
shows that the discovered symbols can be used for training a forward symbol
prediction model that allows achieving goals via tree search.

Figure 7: An example plan for an arbitrary goal state. In the fourth action,
the robot tries to pick and release from the top-right position to the bottom-
left position, which does not interact with any object.

##  V CONCLUSION

We proposed and implemented a predictive encoder-decoder network that utilized
a binary bottleneck layer and, importantly, a self-attention mechanism in
order to discover symbols relevant for interacting with the environment of the
robot that hosts a varying number of objects. Through experiments with a
simulated manipulator robot, we showed that the robot acquired reasoning
capabilities to encode interaction dynamics of a varying number of multiple
objects in different configurations using the discovered symbols. For example,
when queried, the robot could reason that (possible multiple numbers of)
objects that are on top of another object would move together if the object
below is picked-up, and the objects around would not move. We showed that
these reasoning capabilities were acquired by learning a minimal set of
symbols that are optimized for effect prediction in the ecological niche of
the robot. We also showed that the discovered symbols can be used for planning
by training a higher-level neural network that makes pure symbolic reasoning.
In order to acquire optimal plan generation capability, we plan to incorporate
effective and efficient plan generation requirement in the symbol discovery
loop as suggested by [22]. We also plan to transfer the learned symbolic
reasoning capabilities to the real robot via domain randomization techniques.

## ACKNOWLEDGMENT

This research was supported by TUBITAK (The Scientific and Technological
Research Council of Turkey) ARDEB 1001 program (project number: 120E274) and
by the BAGEP Award of the Science Academy. The numerical calculations reported
in this paper were partially performed at TUBITAK ULAKBIM, High Performance
and Grid Computing Center (TRUBA resources). Additional support is provided by
the International Joint Research Promotion Program, Osaka University under the
project “Developmentally and biologically realistic modeling of perspective
invariant action understanding” and the Japan Society for the Promotion of
Science, Grant-in-Aid for Scientific Research - project number 22H03670.

## References

  * [1] S. J. Russell, _Artificial intelligence a modern approach_. Pearson Education, Inc., 2010.
  * [2] T. Taniguchi, E. Ugur, M. Hoffmann, L. Jamone, T. Nagai, B. Rosman, T. Matsuka, N. Iwahashi, E. Oztop, J. Piater, _et al._ , “Symbol emergence in cognitive developmental systems: a survey,” _IEEE Transactions on Cognitive and Developmental Systems_ , 2019.
  * [3] A. Ahmetoglu, M. Y. Seker, J. Piater, E. Oztop, and E. Ugur, “Deepsym: Deep symbol generation and rule learning from unsupervised continuous robot interaction for planning,” _arXiv preprint arXiv:2012.02532_ , 2020.
  * [4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [5] K. Mourao, R. P. Petrick, and M. Steedman, “Using kernel perceptrons to learn action effects for planning,” in _International Conference on Cognitive Systems (CogSys 2008)_. Citeseer, 2008, pp. 45–50.
  * [6] F. Wörgötter, A. Agostini, N. Krüger, N. Shylo, and B. Porr, “Cognitive agents—a procedural perspective relying on the predictability of object-action-complexes (oacs),” _Robotics and Autonomous Systems_ , vol. 57, no. 4, pp. 420–432, 2009.
  * [7] A. Dehban, S. Zhang, N. Cauli, L. Jamone, and J. Santos-Victor, “Learning deep features for robotic inference from physical interactions,” _IEEE Transactions on Cognitive and Developmental Systems_ , 2022.
  * [8] F. S. Lay, A. S. Bauer, A. Albu-Schäffer, F. Stulp, and D. Leidner, “Unsupervised symbol emergence for supervised autonomy using multi-modal latent dirichlet allocations,” _Advanced Robotics_ , vol. 36, no. 1-2, pp. 71–84, 2022.
  * [9] R. Sun, “Symbol grounding: a new look at an old idea,” _Philosophical Psychology_ , vol. 13, no. 2, pp. 149–172, 2000.
  * [10] G. Konidaris, “On the necessity of abstraction,” _Current Opinion in Behavioral Sciences_ , vol. 29, pp. 1–7, 2019.
  * [11] A. Ahmetoglu, E. Ugur, M. Asada, and E. Oztop, “High-level features for resource economy and fast learning in skill transfer,” _Advanced Robotics_ , vol. 36, no. 5-6, pp. 291–303, 2022.
  * [12] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez, “Constructing symbolic representations for high-level planning,” in _28th AAAI Conf. on AI_ , 2014\.
  * [13] G. Konidaris, L. Kaelbling, and T. Lozano-Perez, “Symbol acquisition for probabilistic high-level planning,” in _International Joint Conference on Artificial Intelligence_ , 2015.
  * [14] G. Konidaris, L. P. Kaelbling, and T. Lozano-Perez, “From skills to symbols: Learning symbolic representations for abstract high-level planning,” _Journal of Artificial Intelligence Research_ , vol. 61, pp. 215–289, 2018\.
  * [15] S. James, B. Rosman, and G. Konidaris, “Learning portable representations for high-level planning,” _arXiv preprint arXiv:1905.12006_ , 2019.
  * [16] T. Silver, A. Athalye, J. B. Tenenbaum, T. Lozano-Perez, and L. P. Kaelbling, “Learning neuro-symbolic skills for bilevel planning,” _arXiv preprint arXiv:2206.10680_ , 2022.
  * [17] E. Ugur and J. Piater, “Bottom-up learning of object categories, action effects and logical rules: From continuous manipulative exploration to symbolic planning,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 2627–2633.
  * [18] ——, “Refining discovered symbols with multi-step interaction experience,” in _2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids)_. IEEE, 2015, pp. 1007–1012.
  * [19] M. Asai and A. Fukunaga, “Classical planning in deep latent space: Bridging the subsymbolic-symbolic boundary,” _arXiv preprint arXiv:1705.00154_ , 2017\.
  * [20] M. Asai and C. Muise, “Learning neural-symbolic descriptive planning models via cube-space priors: The voyage home (to strips),” _arXiv preprint arXiv:2004.12850_ , 2020.
  * [21] M. Asai, H. Kajino, A. Fukunaga, and C. Muise, “Classical planning in deep latent space,” _arXiv preprint arXiv:2107.00110_ , 2021.
  * [22] T. Silver, R. Chitnis, N. Kumar, W. McClinton, T. Lozano-Perez, L. P. Kaelbling, and J. Tenenbaum, “Inventing relational state and action abstractions for effective and efficient bilevel planning,” _arXiv preprint arXiv:2203.09634_ , 2022.
  * [23] F. Locatello, D. Weissenborn, T. Unterthiner, A. Mahendran, G. Heigold, J. Uszkoreit, A. Dosovitskiy, and T. Kipf, “Object-centric learning with slot attention,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 11 525–11 538, 2020.
  * [24] G. F. Elsayed, A. Mahendran, S. van Steenkiste, K. Greff, M. C. Mozer, and T. Kipf, “Savi++: Towards end-to-end object-centric learning from real-world videos,” _arXiv preprint arXiv:2206.07764_ , 2022.
  * [25] C. J. Maddison, A. Mnih, and Y. W. Teh, “The concrete distribution: A continuous relaxation of discrete random variables,” _arXiv preprint arXiv:1611.00712_ , 2016.
  * [26] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization with gumbel-softmax,” _arXiv preprint arXiv:1611.01144_ , 2016.
  * [27] A. Paraschos, C. Daniel, J. R. Peters, and G. Neumann, “Probabilistic movement primitives,” _Advances in neural information processing systems_ , vol. 26, 2013.
  * [28] S. Schaal, “Dynamic movement primitives-a framework for motor control in humans and humanoid robotics,” in _Adaptive motion of animals and machines_. Springer, 2006, pp. 261–280.
  * [29] M. Y. Seker, M. Imre, J. H. Piater, and E. Ugur, “Conditional neural movement primitives.” in _Robotics: Science and Systems_ , vol. 10, 2019.
  * [30] E. Ugur, Y. Nagai, E. Sahin, and E. Oztop, “Staged development of robot skills: Behavior formation, affordance learning and imitation with motionese,” _IEEE Transactions on Autonomous Mental Development_ , vol. 7, no. 2, pp. 119–139, 2015.
  * [31] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” _arXiv preprint arXiv:1511.06434_ , 2015.
  * [32] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.
  * [33] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” _arXiv preprint arXiv:1502.03167_ , 2015.
  * [34] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.


2024

1]Department of Data Science and Artificial Intelligence, Monash University,
Melbourne, Victoria, Australia 2]Department of Human-Centred Computing, Monash
University, Melbourne, Victoria, Australia

# Human Brain Exhibits Distinct Patterns When Listening to Fake Versus Real
Audio: Preliminary Evidence

Mahsa Salehi<EMAIL_ADDRESS>Kalin Stefanov
<EMAIL_ADDRESS>Ehsan Shareghi<EMAIL_ADDRESS>[  [

###### Abstract

In this paper we study the variations in human brain activity when listening
to real and fake audio. Our preliminary results suggest that the
representations learned by a state-of-the-art deepfake audio detection
algorithm, do not exhibit clear distinct patterns between real and fake audio.
In contrast, human brain activity, as measured by EEG, displays distinct
patterns when individuals are exposed to fake versus real audio. This
preliminary evidence enables future research directions in areas such as
deepfake audio detection.

###### keywords:

deepfake audio, human cognition, time series analysis, EEG

##  1 Introduction

Increasingly powerful deep learning algorithms accompanied by the rapid
advances in computing power have enabled the generation of hyper-realistic
synthetic media commonly referred to as _deepfakes_. Deepfakes include all
forms of digital content - video, text, images, and audio - that have been
either manipulated or created from scratch using deep learning algorithms to
primarily mislead, deceive or influence audiences UNI . In addition, access to
deepfake generation technologies has become widespread, and the technologies
are easy to use - deepfake generation technologies have been democratised
Weaponised . However, deepfake generation technologies provide new tools to
cybercriminals with broad impact in cyber-enabled crime. For example, audio
generation and in particular voice cloning (creating a synthesised voice from
a small audio sample of an authentic voice) can be used in sophisticated
attacks. In 2019, criminals used voice cloning technology to impersonate an
executive’s voice in the first reported use of deepfakes in a cybercrime
operation, deceiving the CEO of a United Kingdom energy firm into transferring
$240,000 Wallstreet . In 2020, cybercriminals cloned the voice of a company
director in the United Arab Emirates to steal as much as $35 million 35Mil .

In the past 5 years, we have witnessed a dramatic increase in the number of
publications per year that address issues related to deepfake content
detection - from less than $250$ publications in $2018$ to more than $3250$ in
$2022$ 111app.dimensions.ai. In $2022$, at the IEEE International Conference
on Acoustics, Speech and Signal Processing, the first Audio Deep Synthesis
Detection Challenge was launched to motivate the development of deepfake audio
detection algorithms ADD2022 . The body of research on deepfake detection is
driven by datasets created with different deepfake generation techniques.
Consequently, most deepfake detection methods are designed, developed, and
evaluated on a subset of manipulations, and there is no guarantee that they
will generalise to unseen classes of manipulations. The nature of voice
cloning-enabled crime (_i.e.,_ the fact that victims have expectations for the
voice qualities from prior experiences) motivates formulating the problem of
cloned voice detection as anomaly detection. Anomaly detection is a data
mining task and is the process of finding rare events (in this context, the
cloned voice) in a dataset which are significantly different from normal
events (in this context, the real voice). Like deepfake detection, one of the
challenges in detecting anomalies is that there are many types of anomalies
which makes the process of detecting anomalies hard and most existing anomaly
detection methods are likely to fail in generalising to novel anomalous
events.

Despite the many remarkable successes of machine learning (ML), there is still
a gap between human and machine learning. Evidence for this gap is that
patterns of errors made by today’s machine learning algorithms are different
from the patterns of humans performing the same tasks. While the human brain
has been a source of inspiration for machine learning, little effort has been
made to directly use data collected from brain activities as a guide for
machine learning algorithms fong2018 . To address the generalisation problem
outlined above, this paper proposes anomaly detectors (_i.e.,_ cloned voice
detectors) that are guided by human brain activities (_i.e., EEG_). The
hypothesis is that the neural activity carries information for the concept of
anomaly (_i.e.,_ “meta level”) which is agnostic to the exact type of the
anomaly (_i.e.,_ “surface level”). This hypothesis is supported by a recent
study suggesting that we can reliably decode deepfake content using human
neural activity moshel2022you .

##  2 Method

###  2.1 Problem Formulation

Given a labeled audio time series dataset $A$ and an equivalent labeled EEG
time series dataset $E$ of a human subject listening to $A$, the problem is
how …

###  2.2 Data Collection

#### Real Audio Data

We collected audio data with ethics clearance for deepfake audio production.
We hired 20 native English-speaking actors (10 males, 10 females) between the
age of 20-35. We prepared 10 conversational scripts with controlled topics for
producing 10 single-turn (i.e., monologue) audios. Each script roughly took 3
minutes to read. We required to collect the data ourselves (as opposed to
using publicly available alternatives) to have full control over topics,
prosodic features and have the ethical consents to release this unique dataset
paired with its equivalent audio (see next paragraph) to the research
community.

#### Deepfake Audio Generation.

The total duration of our mixed audio file is 44 minutes and 26 seconds
including the pauses -silent segments (refereed to as “silence audio”) that
were added before and after each script within the audio file. The duration of
the silence audio is 1 minute and the reason why we added them is to give the
human participants an opportunity to rest during the EEG data collection
phase.

#### EEG Data.

We collected brain EEG from the participants who listen to mixed audio (audio
that contains both read and fake) and are asked to detect deepfake segments
and explain their rationale. The data collection sessions took place in the
Monash Biomedical Imaging (MBI) center in Monash University. Participants were
asked to close their eyes throughout the experiment including the very first 3
minutes baseline recording at the beginning of the experiments. We have used a
64-channel Easycap EEG recording caps following the 10-20 system. Fig. 1 the
electrode layout. The recording frequency of the EEG caps is 5000 Hz.

Figure 1: Electrode layout of 64-channel Easycap

We hired 2 English-speaking participants to record EEG and the corresponding
reaction (press a button) when a deepfake was detected. The participants will
be instructed not to be concerned with fact checking of statements. To the
best of our knowledge there is no public EEG dataset thare are paired with
mixed audio like ours. Our dataset will be unique for its controlled
construction, scale, and more importantly for containing information along 3
axes: original and deepfake audios, EEG, and participant labeling. This will
not only serve the basis of our work but will also be supporting further
research in this underexplored space in ML, psychology and neurosciences. Note
there are two possibilities - one is that the humans can detect the deepfakes,
in which case the EEG can help understand how they detect them. Another is
that the EEG shows different brain activity when confronted with deepfakes,
but the person involved does not identify the speech as deepfake consciously.

###  2.3 Audio Preprocessing

###  2.4 EEG Preprocessing

In this section we explain standard EEG preprocessing steps bigdely2015prep
we took to prepare EEG data for analysis.

#### Baseline correction.

We collected 3 minutes of EEG recording without any stimuli (playing audio) at
the beginning of our EEG data collection experiment as a baseline. In order to
only analysis the effect of our stimuli, we correct our recordings using this
baseline data. We used mean baseline correction as suggested in the literature
bigdely2015prep  which is basically subtracting the baseline mean from the raw
EEG. A sample snapshot of our raw EEG is shown in Fig. 2. As can be seen the
raw EEG is noisy, containing different types of artifacts such as muscle
movement. A standard technique used to remove artifacts is Independent
Component Analysis (ICA).

Figure 2: 1-second Raw EEG data of Subject 1

#### Filtering to remove slow drifts.

We filter the data to remove low-frequency drifts, which can negatively affect
the quality of the ICA fit. The slow drifts are problematic because they
reduce the independence of the assumed-to-be-independent sources (e.g., during
a slow upward drift, the neural, heartbeat, blink, and other muscular sources
will all tend to have higher values), making it harder for the ICA algorithm
to find an accurate solution. To this end we apply a 0.5-80 Hz bandpass
filter.

#### Re-referencing.

We apply common average re-reference to ensure that all channels contribute
equally towards the reduction in channel amplitude, minimizing bias towards
specific brain regions. In order to obtain the voltage signal of each channel
we use the average of the left and right mastoid.

#### Artifact removal.

We applied ICA with 20 components to remove potential artifacts in EEG. After
decomposition, we then applied widely used automatic ICA component labeling
algorithm called ICLabel pion2019iclabel , which will assign a probability
value for each component being one of the following categories: muscle
artifact, heart beat, line noise, channel noise, eye blink and other. Note our
data does not contain eye blink artifact as the participants were asked to
close their eyes. ICLabel produces predicted probability values for each of
these categories. We visually inspected all the components and the non-
artifact components (“Brain” and “Other” categories) were reasonable Fig. 3
shows an example of a muscle category that was removed. “Other” is a catch-all
that for non-classifiable components. We will stay on the side of caution and
assume we cannot blindly remove these. After removing the artifact components
we reconstructed the EEG back to its original space. We then resampled our EEG
data to 256 Hz to be compatible with our audio recordings.

Figure 3: An example of a muscle artifact component in Subject 1

#### Segmentation.

We segment the processed EEG recording into segments of time series of 0.5
second windows. Each time series window overlaps 50% with its before and after
segment. Since the frequency rate is 256 Hz, the number of data points in each
time series is 128. We denote a EEG segment as
$E_{i}=\\{e_{1},e_{2},\ldots,e_{L}\\}$ where $L=128$ is the length of the time
series and $e_{j}\in\mathbb{R}^{d}$, where $d=64$ is the number of channels.
Finally, the segments in which the silence audio is played are removed. This
results in a a total of 7548 samples of time series ($E_{i}$s).

Table 1: Description of dataset used in our experiments. Dataset | #Train | # Test | Length | # Channel | # Class
---|---|---|---|---|---
EEG- Subject 1 | 6038 | 1510 | 128 | 64 | 2

##  3 Experimental Results

Our experiments are divided to two categories: 1) We apply statistical tools
to visualise our Audio and EEG datasets, and 2) We apply time series
classification tools to classify fake and real classes using our EEG dataset.
We used data from one of our subjects (Subject 1) in this section. The details
of this dataset is presented in Table 1.

Figure 4: Visualisation of our pilot study

#### Highest Density Regions Visualisation

We visualize the Highest Density Regions (HDR;hyndman1996computing ) of the
samples of EEG time series ($E_{i}$s), as well the HDR of the representations
of audios from the classification head of the CITE state-of-the-art audio
deepfake detection system using the Mapper222https://github.com/scikit-
tda/kepler-mapper algorithm singh2007topological . The HDR cuts the overall
density space to create latent spaces that contain above a threshold
probability mass (i.e., minimum ditance $\geq 0.65$ ). The Mapper’s output is
represented as a graph in which each component corresponds to a cluster of
points that together form a high density region. The connectivity of the graph
reflects some topoligical characteristics of the representation space, with
darker colors (i.e., dark purples reflecting higher density). Blue represents
fake and purple represents real. Fig. 4 demonstrates the Mapper output on the
audio representations versus EEG representations. The observation highlights a
much more clear separation of fake and real signals in the EEG space compared
to the audio’s. This is an strong indication that EEG’s representation of
deepfake audio could offer a more discriminative representation of data.

#### Brain Activity Classification.

In the experiments we use the EEG dataset of $E_{i}$ samples of subject 1 tp
train and test a classifier. We used the state-of-the-art time series
classification method called ConvTran foumani2024improving  to classify fake
and real classes using our EEG dataset. Convtran uses CNN and Transformer to
classify samples of time series taking into account the order of data points
in each $E_{i}$ effectively. A two-tower network each using ConvTran is used.
The first tower processes the raw data $E_{i}$, while the second tower
processes the FFT of the input, i.e., $FFT(E_{i})$. This tower extracts the
frequency features including frequency bands such as alpha and gamma. The
outputs of both towers are concatenated for classification.

The train and test split is done in two ways. In our first method we selected
$E_{i}$ EEG time series randomly, see Table 1 for the train and test split.
The results of In the second experiment we used the first 80% of the $E_{i}$s
in the data collection session for train and the rest for test. This is a
harder task given that the content of the audio was changing throughout the
data collection. The results are shown in Tables 3 and LABEL:Tab:result2. They
are very promising and suggesting that we can learn the patterns of fake and
real only suing EEG data of human listeners listening to real and fake audio.

Table 2: EEG classification results on Subject 1- random train/test split Class | Precision | Recall | F1-score
---|---|---|---
Real class | 0.999 | 0.999 | 0.999
Fake class | 0.961 | 0.980 | 0.970
Table 3: EEG classification results on Subject 1- ordered train/test split Class | Precision | Recall | F1-score
---|---|---|---
Real class | 0.982 | 0.935 | 0.958
Fake class | 0.125 | 0.356 | 0.185

## References

  *   * (1) Anand, A., Bianco, B.: The 2021 Innovations Dialogue Conference Report: Deepfakes, Trust and International Security. United Nations Institute for Disarmament Research. https://unidir.org/sites/default/files/2021-12/UNIDIR_2021_Innovations_Dialogue.pdf. Accessed: 2024-01-31 (2021)
  * (2) Smith, H., Mansted, K.: Weaponised Deep Fakes - National Security and Democracy. https://www.aspi.org.au/report/weaponised-deep-fakes. Accessed: 2024-01-31 (2020)
  * (3) Stupp, C.: Fraudsters Used AI to Mimic CEO’s Voice in Unusual Cybercrime Case. https://www.wsj.com/articles/fraudsters-use-ai-to-mimic-ceos-voice-in-unusual-cybercrime-case-11567157402. 24-01-31 (2019)
  * (4) Brewster, T.: Fraudsters Cloned Company Director’s Voice In $35 Million Bank Heist. https://www.forbes.com/sites/thomasbrewster/2021/10/14/huge-bank-fraud-uses-deep-fake-voice-tech-to-steal-millions. 24-01-31 (2021)
  * (5) Tao, J., Li, H., Yi, J.: ADD 2022: The First Audio Deep Synthesis Detection Challenge. http://addchallenge.cn/add2022. 24-01-31 (2022)
  * (6) Fong, R.C., Scheirer, W.J., Cox, D.D.: Using human brain activity to guide machine learning. Scientific Reports 8(1), 5397 (2018). https://doi.org/10.1038/s41598-018-23618-6
  * (7) Moshel, M.L., Robinson, A.K., Carlson, T.A., Grootswagers, T.: Are you for real? Decoding realistic AI-generated faces from neural activity. Vision Research 199, 108079 (2022)
  * (8) Bigdely-Shamlo, N., Mullen, T., Kothe, C., Su, K.-M., Robbins, K.A.: The PREP pipeline: Standardized preprocessing for large-scale EEG analysis. Frontiers in Neuroinformatics 9, 16 (2015)
  * (9) Pion-Tonachini, L., Kreutz-Delgado, K., Makeig, S.: ICLabel: An automated electroencephalographic independent component classifier, dataset, and website. NeuroImage 198, 181–197 (2019)
  * (10) Hyndman, R.: Computing and graphing highest density regions. The American Statistician 50(2), 120–126 (1996)
  * (11) Singh, G., Mémoli, F., Carlsson, G.: Topological methods for the analysis of high dimensional data sets and 3d object recognition. Eurographics 2 (2007)
  * (12) Foumani, N., Tan, C.W., Webb, G.I., Salehi, M.: Improving position encoding of transformers for multivariate time series classification. Data Mining and Knowledge Discovery 38(1), 22–48 (2024)