# Predictive Event Segmentation and Representation with Neural Networks: A
Self-Supervised Model Assessed by Psychological Experiments

Hamit Basgol1 Inci Ayhan2 Emre Ugur3 Department of Computer Science1 at the
University of Tübingen, GermanyDepartment of Psychology2,Department of
Computer Engineering3 at Boğaziçi University, Turkey

###### Abstract

People segment complex, ever-changing and continuous experience into basic,
stable and discrete spatio-temporal experience units, called events. Event
segmentation literature investigates the mechanisms that allow people to
extract these units from the continuous experience. Aiming to shed light on
event segmentation ability, event segmentation theory points out that people
predict ongoing activities and observe prediction error signals in order to
find event boundaries that keep events apart. In this study, we investigated
the mechanism giving rise to this ability by a computational model and
accompanying psychological experiments. Inspired from the principles of event
segmentation theory and predictive processing, we introduced a semi-
mechanistic model of event segmentation, learning, and representation. This
model consists of feed-forward neural networks that predict the sensory signal
in the next time-step in order to represent different events, and a cognitive
model that regulates these neural networks on the basis of their prediction
errors. In order to verify the ability of our model in segmenting experience
into spatio-temporal units, learning them during passive observation, and
representing them in its internal representational space, we prepared a video
that depicts natural human behaviors represented by point-light displays. We
compared event segmentation behaviors of human participants and our model with
this video in two hierarchical event segmentation levels. By using point-
biserial correlation technique, we demonstrated that event segmentation
decisions of our model correlated with the responses of participants.
Moreover, by approximating internal representation space of participants by a
similarity-based technique, we showed that our model formed a similar internal
representation space with those of participants. Our results suggests that our
model that tracks the prediction error signals can produce human-like event
segmentation decisions and event representations. Finally, we discussed our
contribution to the literature of event cognition and our understanding of how
event segmentation is implemented in the brain.

###### keywords:

event segmentation, point-light displays, predictive processing, self-
supervision

11footnotetext: Hamit Basgol is a PhD Student in the Department of Computer
Science at the University of Tübingen, Tübingen, Germany (e-mail:
hamitbasgol@gmail.com). The study is conducted when Hamit Basgol is a master’s
student in Cognitive Science Department at Bogazici University.22footnotetext:
Inci Ayhan is with Department of Psychology in Bogazici University, Istanbul,
Turkey (e-mail: inci.ayhan@boun.edu.tr)33footnotetext: Emre Ugur is with
Department of Computer Engineering in Bogazici University, Istanbul, Turkey
(e-mail<EMAIL_ADDRESS>
##  1 Introduction

Humans segment continuous information stream into event units to show robust,
adaptive, and intelligent behavior, which is called event segmentation (Zacks
., 2007; Zacks Swallow, 2007; Zacks, 2020; Richmond Zacks, 2017). In the
recent years, a growing number of computational models were proposed to
capture how humans segment events in order to utilize their continuous
experience (Franklin ., 2020; Gumbsch ., 2016, 2017; Reynolds ., 2007; Metcalf
Leake, 2017). Despite their essential contribution to the understanding of the
event segmentation ability, these models have demonstrated certain
limitations. Namely they were not capable of segmenting events in varying
lengths (Reynolds ., 2007; Metcalf Leake, 2017), they used datasets that
involved abrupt transitions between naturalistic action sequences (Reynolds .,
2007; Metcalf Leake, 2017) and they included robotic models that did not aim
for capturing human event segmentation decisions (Gumbsch ., 2016, 2017). In
fact, to the best of our knowledge, there has been only one study, where
authors compared the performance of their model to the human event
segmentation decisions (Franklin ., 2020).

In this study, we aim at addressing these limitations by a novel computational
model, which we built upon three main elements: (1) the event segmentation
theory (Zacks ., 2007), (2) the predictive processing of Wiese Metzinger
(2017); Clark (2013), and (3) the robotic model of Gumbsch . (2017), the
contributions of which will be highlighted throughout the paper. In this
study, we showed that a self-supervised and semi-mechanistic model monitoring
prediction error signals could produce multimodal event segments in varying
lengths and store the knowledge of events in activations of neural networks.
Moreover, we compared the segmentation and representation results of our model
with those of humans to reveal their similarities and differences. We believe
that our model presents a fruitful approach to modeling event segmentation and
integrating event knowledge into a wide range of perceptual and cognitive
processes.

The introduction of the paper is organized as follows: Firstly, we introduce
the event segmentation theory and the importance of the prediction error
signals for the event segmentation. Secondly, we review the computational
models of event segmentation and identify their limitations. Finally, we
explain the methodology of our current study, its results and conclusions.

###  1.1 Event segmentation theory and prediction error

Early studies of event segmentation were conducted by Newtson (1973) using a
unitization paradigm, where participants were asked to watch a movie and
segment it into meaningful units. The results of Newston’s study demonstrated
that a substantial agreement across participants on the segmentation
locations, which happened to be persistent in time. Subsequent research
verified these findings and opened up the possibilities of investigating the
role of events in human cognition (Zacks Swallow, 2007; Zacks, 2020). The
locations at which participants segment a continuous information stream (e.g.,
a movie) are termed as event boundaries, which are the positions in time that
show perceptual changes in spatial locations, movements, relative distances
between agents, or goals (Kurby Zacks, 2008; Hard ., 2011; Zacks ., 2007;
Zacks, 2020; Cutting ., 2012; Hard ., 2006; Huff ., 2014; Newtson ., 1977;
Zacks ., 2010; Cutting, 2014).

Events are known to be hierarchically structured (Zacks, 2020). People can
detect smallest (fine-grained) and largest events (coarse-grained) (Newtson,
1973; Hard ., 2006, 2011; Zacks Swallow, 2007; Zacks, 2020; Zacks ., 2001),
when they are instructed to do so. Research with functional magnetic resonance
neuroimaging (fMRI) suggests that hierarchical segmentation is an automatic
process (Zacks ., 2001; Speer ., 2007) such that while observing a movie or
reading a story, the brain selectively responds to the fine- and coarse-
grained event boundaries. Hard . (2011), for example, demonstrated that
changes at event boundaries are more numerous than other parts of an activity;
moreover, they particularly peak at coarse-grained boundaries. The strong
relationship between both types of change, namely the sensory (fine-grained)
and the conceptual (coarse-grained) change, suggests that events are segmented
based on the perceptual cycle formed by the bottom-up processing of sensory
features and the top-down processing of conceptual knowledge (Neisser, 1976;
Zacks ., 2007; Zacks, 2020).

A computational model or a theory of event segmentation should explain at
least two basic properties of event segmentation. The first one is how
locations of event boundaries are detected and the second one is how event
segmentation operation is conducted in different hierarchies. Event
segmentation theory (EST) proposes an account for both of these properties.
According to the EST, people constantly make perceptual predictions by event
models in the working memory (Zacks ., 2007; Reynolds ., 2007). The event
boundary is formed when the current event model cannot capture the current
situation, in other words, when the corresponding prediction error signal
follows a transient increase. In such situations, the system triggers another
event model to predict the following sensory input. Thus, a strategy that is
based on monitoring the prediction error signals, might correspond to the
basic mechanism behind the event segmentation ability. Indeed, the EST and the
role of prediction error signals in the event segmentation were supported by
many studies (Zacks ., 2011; Eisenberg ., 2018; Hard ., 2011; Gumbsch ., 2016,
2017; Franklin ., 2020; Reynolds ., 2007; Stawarczyk ., 2021), despite
exceptions (Shin DuBrow, 2021; O’Reilly, 2013). Along with its focus on the
prediction error signals for event boundary detection, EST also suggests that
people might make predictions by events in multiple timescales simultaneously
and sensitivity differences between events to incoming prediction error
signals might determine their lengths or positions in the hierarchy. For
example, an event model might be sensitive to minor prediction errors compared
to another (Zacks Swallow, 2007), and this sensitivity difference might make
the former shorter than the latter.

Due to this mechanism, computational models in the literature have been mostly
inspired from the EST (Franklin ., 2020; Gumbsch ., 2016, 2017; Reynolds .,
2007; Metcalf Leake, 2017). All these models, on the other hand, come up with
certain limitations, which will be the topic of the next sub-section.

###  1.2 Computational models of event segmentation

Several important computational models have been proposed in the literature
with different limitations (Franklin ., 2020; Gumbsch ., 2016, 2017; Reynolds
., 2007). For example, Reynolds . (2007) utilized a set of sequence models for
the segmentation of human behaviors. Despite the success of the model in
detecting event boundaries, hierarchical segmentation of events in varying
granularities was not addressed. At the same time, behavioral sequences that
were used for training the model involved abrupt and unnatural transitions.
Metcalf Leake (2017) enhanced this model by a reinforcement learning agent.
Although these two models suggest that monitoring prediction error signals is
an effective strategy for the event segmentation, they did not address
hierarchical segmentation of events.

Gumbsch . (2016, 2017) developed a robotic model that chunks sensory-motor
information flow into parts. The model represents events by linear models,
which encode different sensory dimensions and predict sensory signal in the
next time-step. The linear models are regulated by a cognitive model at a
higher level based on the prediction errors of the lower-level linear models.
From this perspective, whereas the cognitive model resembles to the mechanism
proposed by the EST, linear models correspond to the working memory
representations. As has been addressed in Gumbsch . (2019), however, since
linear models encoding sensory dimensions are disconnected from one another,
Gumbsch et al.’s model is not capable of discovering multi-modal associations
between sensory modalities in a particular event structure. Besides this
limitation, these models are robotic models that assume the involvement of an
active agent. However, the event segmentation literature is based largely on
the unitization paradigm in which participants observe events passively and
press a button to separate them from one another (Newtson, 1973; Newtson
Engquist, 1976; Newtson ., 1977; Zacks ., 2007; Zacks, 2020; Hard Tversky,
2003; Hard ., 2006; Zacks ., 2001).

Lastly, Franklin . (2020) developed an inclusive model of event cognition,
which considers various domains such as event memorization, segmentation,
retrieval, and inference. To the best of our knowledge, this is the only study
that used naturalistic videos and considered the human-level event
segmentation performance for the model validation, even though the received
correlation between the performances of the model and ground-truth data is
open to improvement.

Overall, the computational models of event segmentation suggest that the EST
presents a plausible mechanism for the event segmentation task. For addressing
the missing points in the literature and testing the suggested mechanism by
the EST for more than one granularity level, we developed a novel
computational model for event segmentation. In addition to event segmentation
capability, our model could form event representations.

###  1.3 Event representations

Representations are mental objects with semantic properties (Pitt, 2020). To
express the strength of the relationships, a representational space can be
formed by taking the pairwise distance between all representations (Shepard
Arabie, 1979; Shepard, 1980, 1987). This two-way relationship makes the
similarity a valuable metric to reveal how the system organizes knowledge as
representations form the basis of categorization and generalization. One aim
of the artificial intelligence is to learn valuable and representative
information from the data (Bengio ., 2014). Multi-layer perceptrons (i.e.,
deep neural networks) can learn distributed and semantically meaningful
representations (Bengio ., 2014; Urban Gates, 2021). The similarity between
representations (i.e., semantic relationships between represented entities) of
a deep learning model can be found by the Euclidean distance or cosine
similarity. For example, the semantic relationship between words and sentences
(Mikolov ., 2013; Rogers McClelland, 2005), objects (Deselaers Ferrari, 2011),
scenes (Eslami ., 2018), and episodes (Rothfuss ., 2018) can be captured with
the help of representations learned by a deep learning system. Since
representations give researchers a gist about how humans organize knowledge,
generalize between instances, and make analogical transfers (Blough, 2001;
Nosofsky, 1992; Shepard, 1987, 1980; Tversky, 1977), they have a fundamental
place in cognitive science. As could be expected, researchers exploited human
similarity judgments to achieve human mental representations (Shepard, 1980,
1987; Shepard Arabie, 1979). The role of representations and similarity
judgments in artificial intelligence and cognitive science suggest that they
might provide a basis for comparing people and machines. In fact, recent
research provides excellent examples of this comparison (Peterson ., 2018;
Hebart ., 2020).

Event representation literature is very rich and represents a diverse set of
studies (Blom ., 2020; Schütz-Bosbach Prinz, 2007; Day Bartels, 2008; Wang .,
2017; Sheldon El-Asmar, 2018; Fivush ., 1992; Kominsky ., 2021). In the
context of computational modeling, recent studies use (Shen ., 2020) and learn
(Dias Dimiccoli, 2018) event representations. In contrast, despite the
interest received by event representations, event similarity judgments is a
concealed area under the action similarity judgments (Tarhan ., 2020; Tarhan
Konkle, 2018). In our work, utilizing this possibility, we compare the event
representations of our computational model and participants by exploiting
event similarity judgments.

###  1.4 Our contribution

In this study, inspired from the EST (Zacks ., 2007), the predictive
processing (Wiese Metzinger, 2017; Clark, 2013), and the Gumbsch’s robotic
model (Gumbsch ., 2016, 2017), we developed a novel computational model for
event segmentation. Our model consists of multi-layer perceptrons (i.e., event
models) that are managed by a cognitive mechanism and consequently,
determining the event boundaries. As our contribution to the literature, (1)
our model is capable of learning to represent and predict multi-modal event
segments with sensory associations in passive observation unlike the models
developed by (Gumbsch ., 2016, 2017) which segment unimodal events based on
actions in a simulation environment. (2) With the help of a parameter,
changing sensitivies of event models to prediction error signals, our model
can also segment events in varying granularities, which was not addressed by
Reynolds . (2007) and Metcalf Leake (2017). (3) Moreover, segmentation and
representation capabilities of our model were tested by ground-truth data
received from psychological experiments.

A multi-layer perceptron is a plain deep neural network which consists of an
input layer, an intermediate (hidden) layer or layers, and an output layer.
The network learns the relationship between inputs and outputs by updating
weights in each iteration. Thanks to the hidden units, multi-layer perceptrons
can classify complex patterns (Lippmann, 1989), approximating non-linear
functions (Hornik ., 1989). Moreover, the knowledge developed throughout the
training is stored in weights and what is learned by the model can be explored
by analyzing the representations of the network (Peterson ., 2018; Hebart .,
2020; Fleming Storrs, 2019). The use of deep neural networks in cognitive
science and artificial intelligence has a long history and had an important
role in the emergence of the connectionist framework (Rumelhart ., 1986). The
effect of connectionist framework, in other words deep learning models, on
cognitive sciences still persists and leads to revolutionary results in a wide
variety of domains such as perception (Krizhevsky ., 2012; Russakovsky .,
2015; He ., 2016; Spoerer ., 2017; Fleming Storrs, 2019), linguistics (Radford
., 2017; Wu ., 2016; Floridi Chiriatti, 2020), developmental psychology (Orhan
., 2020), and cognitive neuroscience (Yamins DiCarlo, 2016; Khaligh-Razavi
Kriegeskorte, 2014; Tripp, 2017).

We used multi-layer perceptrons as a member of deep neural networks to
represent events. A deep neural network model can be trained in several ways:
supervised, unsupervised and self-supervised way. In the supervised learning,
models receive outputs (e.g., categories for object identification) of inputs
(e.g., images) from huge labelled datasets (Krizhevsky ., 2012; Russakovsky .,
2015). Despite its success in a range of domains such as object identification
(Krizhevsky ., 2012; Russakovsky ., 2015), supervised learning is criticized
being inconsistent with how humans actually learn. Humans learn new concepts
and abilities with little supervision without the requirement of hand-crafted
labels (Vinyals ., 2016). The dependency of supervised learning on labels
leads to researchers investigate other learning possibilities such as
unsupervised and self-supervised learning that do not require explicit labels.
In the unsupervised learning, the network learns how to represent data
efficiently by not relying on labels, but rather by capturing the high-order
statistics of the dataset (Fleming Storrs, 2019). On the other hand, in the
self-supervised learning, labels are substituted by the information in the
input data so much so that rather than mapping inputs to the hand-crafted
labels, the network learns to predict selected parts of the data (e.g.,
predicting the next sequence of a video or a certain part of an image) and
generates representations by this way (Liu ., 2021).

In the context of event segmentation, using a supervised model receiving event
boundary locations from a supervisor or instructor would not be natural
because infants mostly learn new abilities without supervision. Therefore, our
model is self-supervised as it learns to make prediction in a segment and
detects event boundaries from the data without the need of human-crafted
labels (Liu ., 2021).

Despite the contribution of deep neural networks to the progress in a wide
range of areas from linguistics to neuroscience (Krizhevsky ., 2012;
Russakovsky ., 2015; He ., 2016; Spoerer ., 2017; Fleming Storrs, 2019;
Radford ., 2017; Wu ., 2016; Floridi Chiriatti, 2020; Orhan ., 2020; Yamins
DiCarlo, 2016; Khaligh-Razavi Kriegeskorte, 2014; Tripp, 2017), they have an
important limitation, namely explainability. Deep neural networks are black-
box models whose functioning is not explicit, which is a crucial property
especially for the domains where understanding the decisions of networks is
critical (e.g., medical decision making) or in scientific practice (e.g., why
the network decides this way and what this decision says for the problem in
question). Even though it uses multi-layer perceptrons for representing the
event segments, our model is not an end-to-end black box model (Rudin Radin,
2019); rather, it involves an easily understandable and trackable white-box
model that regulates multi-layer perceptrons. From this perspective, our model
is a semi-mechanistic model, incorporating the capabilities of white-box and
black-box models, giving researchers a chance to benefit from the power of
neural networks without fully sacrificing the explainability.

Our proposed model is both self-supervised and semi-mechanistic. By tracking
the prediction error signals, (1) the model produces multimodal event segments
in varying hierarchies via passive observation with the help of multi-layer
perceptrons unlike the models developed by (Gumbsch ., 2016, 2017), (2) With
the help of a parameter, changing sensitivities of event models to prediction
error signals, our model can produce event segments in varying granularities,
which was not addressed by (Reynolds ., 2007; Metcalf Leake, 2017). (3)
Moreover, not only did we study the activations in the layers to have an
insight with respect to the functioning, we also compared the performance of
our model to that of the human observers in order to assess its capabilities.
We received a higher point-biserial correlation score than the existing score
in the literature (Franklin ., 2020). More specifically,

  * •

we prepared two videos depicting naturalistic human behaviors that are
represented as point-light displays (PLDs) (Johansson, 1973). PLDs depict
biological movements by several points that are placed on the positions
corresponding to joints. From their movements, people can recognize human
movements, emotions, actions, and gender (Alaerts ., 2011; Troje Basbaum,
2008). The ability to perceive PLDs emerges very early in human life
(Bertenthal ., 1987; Fox McDaniel, 1982) and human brain imaging studies
(Grossman ., 2000; Krakowski ., 2011; Pavlova ., 2004; Michels ., 2009;
Peuskens ., 2005) suggest that biological motion processing are specialized.
Since they can provide an easily manipulable stimulus for the psychological
experiments and present an opportunity to reduce data dimension and processing
time for computational experiments.

  * •

By using the dataset prepared from PLDs, we assessed the smallest (fine-
grained) and largest (coarse-grained) event segmentation performances of our
model. Presenting the same videos to the participants in an online
psychological experiment, we collected the behavioral event boundary
judgments, to which we then compared the performances of our model by a point-
biserial correlation.

  * •

We proposed a new validation technique inspired from the literature on the
concept emergence in deep neural networks (Peterson ., 2018) for investigating
whether event segments that are produced by our model is meaningful and that
neural networks capture human-like event representation space. For
approximating event representations of participants, in another online
experiment, we asked participants the perceived similarities between different
events to turn this into an event representation space. We then compared the
internal event representation space of our model with that of the
participants. The techniques by which we approximated event representations of
our model will be further explained in the method section.

Our results suggested that the proposed model received a considerable point-
biserial correlation score ($r_{pb}$) with the event segmentation decisions of
participants, namely 0.254 and 0.196 for the fine- and the coarse-grained
segmentation, respectively. Moreover, the correlation between the internal
representation spaces of the participants and our model resulted in $r$ of
0.435 and 0.614 for fine-grained and coarse-grained segmentation. These
results, overall, propose that (1) a model guided by prediction error signals
can capture event segmentation decisions of human participants, (2) multi-
layer perceptrons can successfully capture the internal representation space,
and (3) events can be segmented to one another by a self-supervised predictive
model that does not receive event boundary locations as human-crafted labels.

##  2 General Method

In this section, we first illustrate our proposed computational model for the
event segmentation and then explain how we prepared the dataset for the
psychological and computational experiments.

###  2.1 The overview of the proposed computational model

The model we propose is built on top of the computational architecture
proposed by (Gumbsch ., 2016, 2017) for the robotic event segmentation. We
reformulated the model in question (Gumbsch ., 2016, 2017). In general (1) we
represented event models as multi-layer perceptrons capable of representing
non-linear relationships in the data. In this way, our model can now
approximate complex sequences of human actions and learn associations between
possible modalities. Additionally, (2) we excluded the role of action (i.e.,
motor modality) in the event formation to extend the model to passive
segmentation.

The overview of the proposed model and how the proposed model is tested are
given in Figure 1. The model represents events as predictive models that
predict sensory information at the next timestep and receive a prediction
error signal (A, F). Prediction error signals are accumulated and used for
calculating a surprise threshold (B), whose tolerance, therefore granularity,
is weighted by a parameter called $\theta$ (C). If the surprise threshold is
exceeded, the searching period starts. From this period, the best event model
is returned (D). After each to be chunked event ends, all activated events
during the last training step are trained (E). For subsequent analyses, event
segmentation results are calculated (G) and compared with human performance
(I). For further investigations, a hierarchical neural network is trained to
capture the relationship between events (H), and its results are compared with
human performance (J).

Figure 1: The overview of the study(A) In the online prediction phase, the
current event model makes a prediction, on the basis of which the current
prediction error is calculated. (B) At the same time, for each model, a
surprise threshold is computed. If the prediction error exceeds the surprise
threshold, the system enters into a search period for spotting the best event
model. (C) The parameter $\theta$ determines the tolerance of event models,
and therefore, their granularities. (D) In the search period, all event models
are trained for the next n timesteps which returns the best event model. (E)
After each epoch, the system moves into the replay phase in which all event
models are trained for the timesteps for which they are active. (F) Each event
model receives a body posture represented by the PLDs and tries to predict the
body posture at the next timestep. Event models are multi-layer perceptrons
whose functioning is not explicitly tractable. (G) After the training is over,
the segmentation results are calculated, and (I) they are compared with human
performance obtained from the online psychological experiments. (H) In order
to investigate whether segmented events are meaningful and their semantic
relationships can be captured by a hierarchical neural network, we received
representations of each timestep and reduced their dimensionality. (J)
Similarity judgments extracted from these representations are then compared
with those of participants.

####  2.1.1 Online prediction

Our model continuously makes predictions by using an active event model
${M_{t}}$ at the time point ${t}$ (see Figure 1A). $M_{t}$ predicts the next
sensory observation by

$S^{\prime}_{t+1}=S_{t}+\Delta S^{\prime}_{t+1}$

Making predictions until receiving the next surprise signal, the active model
${M_{t}}$ receives sensory inputs, computes and stores prediction error
signals, and learns to make predictions via backpropagation. At each
prediction ${M_{t}}$ makes, the system calculates a dynamically changing
surprise threshold. When the prediction error encountered by the ${M_{t}}$
exceeds the surprise threshold, the system enters into a search period to
start using the best event model for predicting the following sensory input.

####  2.1.2 Surprise threshold and search period

The surprise threshold $\Phi_{M}$ is calculated by the rolling mean of the
stored prediction errors $\overline{e}_{M}$ and of the variance $\sigma_{M}$
with a window (w) (see Figure 1B). The confidence parameter $\theta$ affects
the tolerance of event models to the incoming errors and, thereby, controls
the coarseness/length of the events to be segmented (see Figure 1C).
$\Phi_{M}$ is calculated by

$\Phi_{M}=\overline{e}_{M}+\theta*\sigma_{M}$

If the prediction error of an event model exceeds the threshold $\Phi_{M}$,
the system moves into a search period (see Figure 1D). Each search period
starts with generating a new potential event model having random weights.
Then, all event models in the system are trained for a rehearsal duration that
indicates the number of training epochs. Following this training period, the
best event model - the event model that receives the least mean squared error
- is selected, and the effect of training on event models, except the best
model, is ignored. Sometimes, the newly generated event model does not receive
the minimum mean squared error, in this case, the new event model is removed
from the long term memory.

During the search period, for each event, a different training set is
generated to avoid catastrophic forgetting. For the prediction of the future,
the next n timesteps in the sequence should be learned. Meanwhile, event
models should not forget their history. Thereby, we sample a distinct training
dataset for each event model from the combination of the next n timeteps and
their histories.

####  2.1.3 Memory range and replay

At each search period an extensive search period takes place, which increases
the training time as a function of the number of event models in the system.
For this reason, at the end of each training epoch, we determine the
unutilized event models to remove them when they are not utilized for n epochs
(i.e., memory range).

In order to foster memory consolidation, avoid catastrophic forgetting, and
reduce the time spent for the training, we introduce a replay period in which
each event model is trained by their histories (see Figure 1E). The replay
period is a biologically plausible phenomenon observed in the hippocampal
regions of the brain, which is thought to be related to the memory
consolidation (Ólafsdóttir ., 2018). It was also used for stabilizing the
training regimes of the reinforcement learning agents (Andrychowicz ., 2017).

Up to this point, we portrayed the formalization of the computational model.
In the following section, we detail the preparation of the PLDs, which we used
as the main stimuli for the psychological and computational experiments.

###  2.2 Dataset and videos

Figure 2: Two examples of PLDs are given. Both displays show a type of walking

For the preparation of the dataset from PLDs, we selected twelve natural human
behaviors (such as walking, jumping, picking an object, sitting on a chair,
and searching for an object) from the KIT Motion-Language Dataset (Plappert .,
2016) (see Figure 2). The selected complex human behaviors were added back-to-
back by an interpolation method (see Supplementary Material). All behaviors
were represented in a PLDs format using the X and Y dimensions of 14 markers.

From the behaviors selected, we prepared two datasets/videos -normal and
noisy- involving the same behaviors but in varying degrees of sensory change.
By these datasets/videos, we assessed our model by examining its segmentation
performance in comparison to the behavioral performance of humans,

####  2.2.1 Noisy dataset

We produced the noisy dataset by exploiting the inherent relationship between
the change and event segmentation (Kurby Zacks, 2008; Hard ., 2011; Zacks .,
2007; Zacks, 2020; Cutting ., 2012; Hard ., 2006; Huff ., 2014; Newtson .,
1977; Zacks ., 2010). It is known that the event boundaries are the points
where the change is notable the most and that the change is maximal
particularly at the coarse-grained event boundaries (Hard ., 2011). That is, a
reduction in change must hinder the potential timesteps that separate events
from one another.

In order to prepare the noisy dataset, we applied Gaussian noise to the
temporal signal of the normal dataset (window: 40, standard deviation: 10),
and thereby, removed its fine movement dynamics and made it look much more
fluent than the normal video. From normal and noisy datasets, we created two
16 Hz and 267-second videos for the psychological experiments. The videos can
be found here 111https://youtu.be/L2oCNEvzdrU. For the computational
experiments, we applied a min-max normalization operation to these datasets
and reduced them to 4 Hz (1076 timesteps in total).

##  3 Event segmentation experiments

Humans can segment the ongoing activity into events in varying lengths. In the
unitization paradigm, participants are asked to detect the shortest (fine-
grained) and the largest natural/meaningful events (coarse-grained) (Newtson,
1973; Zacks Swallow, 2007; Zacks ., 2007; Hard ., 2011). We conducted a
unitization-based online psychological experiment that has two experimental
conditions: event granularity (fine-grained and coarse-grained segmentation)
and sensory reliability (normal and noisy videos). The number and positions of
the event boundaries were determined to be the dependent variables.

In addition, we tested whether our computational model (1) captured the human
event segmentation decisions and (2) showed a bias against the noisy video
(i.e., the reduction of the rate of change) similar to that was expected in
the performance of the human participants.

###  3.1 Participants and procedure

Nineteen participants (9 female, mean age 25) were recruited for a within-
subject design. They were primarily undergraduate or graduate university
students and were given the chance of a gift voucher lottery as an incentive.
The study was approved by the Ethics Committee for the Master and PhD Theses
in Social Sciences and Humanities (SOBETİK) at Boğaziçi University.

We reached the participants via the internet and, upon their agreement, sent
them a link of the online experiment (i.e., Pavlovia) developed by Psychopy3
(Peirce ., 2019). Each participant segmented first the normal and then the
noisy video. The experiment started with a segmentation granularity
instruction which asked participants to make either a fine- or a coarse-
grained segmentation by pressing the space button and the level of granularity
was counterbalanced. For each type of segmentation granularity, participants
observed the same video twice to segment the video either in the shortest or
the longest possible way.

We coded each observation by a label for maintaining simplicity, namely Fine
1, Fine 2, Coarse 1, and Coarse 2. For example, Fine 1 denotes the first
observation of the fine-grained segmentation level. Throughout our analyses we
considered second observations as they were considered to be more reliable due
to practice effect.

We trained twelve computational models for each possible combination of our
independent variables. Please see the supplementary materials for the
hyperparameter selection.

###  3.2 Results

One challenge posed by the online psychological experiments is maintaining the
data reliability (Gosling Mason, 2015). For this reason, we applied a number
of control measures to ensure the quality of the data before applying the
statistical tests. Firstly, we checked whether participants understand the
instruction of sensory granularity by comparing their number of responses in
coarse to those in fine segmentation condition. We excluded two participants
as they had more number of responses in the Coarse 2 than the Fine 2 in the
normal video segmentation. Secondly, we calculated the intra-participant
correlations of the fine- and coarse-grained segmentation responses. This
procedure resulted in the exclusion of three more participants (see
Supplementary Material). At the end, 14 participants remained, whose data were
considered for the statistical analysis.

####  3.2.1 The reduction in sensory change reduces the number of event
boundaries

For analyzing the effect of the rate of change in fine- and coarse-grained
event segmentation decisions, we used Friedman’s ANOVA. Analysis revealed that
the number of responses differed significantly across groups (X2(7) = 80.93, p
= .000). Wilcoxon signed-rank was applied along with Holm–Bonferroni method to
adjust the p values to avoid inflating false positives. As a result, it was
found that participants perceived a smaller number of event boundaries in the
noisy (M = 32.71, Md = 17.5, SEM = 4.44) than in the normal video (M = 41.91,
Md = 22.5, SEM = 5.95), W = 879.0, p = 0.021, r = 0.191. The trend was similar
for both fine-grained and coarse-grained segmentations. For the fine-grained
segmentation, the normal video (M = 68.14, Md = 43, SEM = 9.56) received more
response than the noisy video (M = 52.50, Md = 43.5, SEM = 6.98), W = 288.5, p
= 0.016, r = 0.31. Similar to the fine-grained, the coarse-grained
segmentation level showed the same trend, normal (M = 15.67, Md = 16, SEM =
1.49) and noisy video (M = 12.92, Md = 10.5, SEM = 1.66), W = 203.5, p =
0.018, r = 0.33, with a slightly higher effect size. These results verified
that the reduction in sensory change in fact reduced the number of event
boundaries, as expected.

####  3.2.2 Fine-grained event boundaries are more correlated with sensory
change than coarse-grained event boundaries

Aiming to reveal the relationship between sensory changes in point locations
in two dimensions and event segmentation decisions of participants as a
function of time, we generated histograms by grouping the responses of the
participants into the 0.25-second bins and quantified changes as the absolute
sum differences between the X and Y dimensions of points. Considering a
possible lag between event boundary responses and changes in events, which may
result from the decision-making process or the motor response latencies, we
computed the correlations by shifting the response distributions backward in
time. The relationship between the sensory change and the event segmentation
are given in Figure 3. Figure 3 shows that whereas the absolute sensory change
is correlated with the rate of perceived fine-grained event boundaries; it is
uncorrelated with the coarse-grained event boundaries.

Figure 3: Event boundary responses and the degrees of change are given. Since
participants detect event boundaries by pressing a button, there might be
latencies in the responses. For this reason, response probabilities were
calculated by event histograms and gradually shifted backward in time. For all
the time shifts, the correlation between the response of probabilities of the
participants and the absolute sensory change was computed. Results showed that
fine-grained segmentation are correlated with the absolute sensory change,
whereas coarse-grained segmentation decisions seem to be uncorrelated.

####  3.2.3 Event boundaries of the model correlate with those of participants

We received segmentation decisions of our computational model (Figure 1G). A
video displaying fine- and coarse-grained segmentation instances can be found
here 111https://youtu.be/SDQpmdehG8o. After receiving the results, we compared
them with the data of participants by point-biserial correlation (Figure 1I).
Point-biserial correlation is a suitable technique for assessing the
correlation between the event boundary responses (discrete) and the event
boundary histograms (continuous) (Franklin ., 2020). For this reason, we
generated a 1-second event boundary histogram for each condition and assessed
event boundary responses of each participant and each model by this histogram.
For testing whether the results of our model perform better than the chance
level, we devised two control models: random and change models. While the
former selects event boundaries uniformly by giving each timestep the same
weight, the latter selects timesteps by different weights corresponding to
their respective absolute sensory changes.

Directly comparing the performances of humans and computational models may
fail to reveal the actual match because models can respond to an event change
right away while humans show response latencies. Due to this fact, for a
thorough comparison, we assessed the performance of models by shifting their
responses in time. In details, we calculated the point-biserial correlation
values with the temporal shifts up to 2 seconds (8 frames) (see Figure 4).

Figure 4: Time-dependent correlations of the computational models with the
ground-truth data are given. In the unitization paradigm, participants might
show a response latency. Considering this fact, we shifted the responses of
the models forward in time before computing the point-biserial correlations.
Our model reached better correlation scores with the human performance than
the control models.

For the fine-grained event segmentation, our model reached its highest
performance at a 0.5-second shift with an r value of 0.254 when the mean r of
the ground-truth data was 0.394. The coarse-grained event segmentation
performance of our model was maximized after a 1-second shift with an r value
of 0.196, when the mean of participants was rmean = 0.366. Although this is a
significant difference, the particular model receiving maximum correlation
scores (rmax = 0.256) showed a comparable performance to those of participants
(z = -1.40, p = 0.158, two-tailed). In general, these results showed that
segments produced by our model correlate with those produced by human
observers.

####  3.2.4 Event boundaries produced by the computational model correlate
with sensory change

We also investigated the relationship between the sensory change and the
responses of the computational models as showed in Figure 5. Results showed
that the fine-grained segmentation decisions of the computational models were
more accountable by the absolute sensory change than its coarse-grained
segmentation decisions. In this sense, this pattern is roughly analogous to
the asymmetry emerging in the responses of the participants (see Figure 3).

Figure 5: The relationship between the responses of the proposed model and the
change is given. Time-dependent correlations between the absolute sensory
change and the responses of the computational models for the fine-grained and
coarse-grained segmentation. Coarse-grained segmentation decisions of the
model were less correlated with the absolute change than its fine-grained
segmentation decisions. In comparison to the participants, the computational
model reacts to the change immediately.

####  3.2.5 Sensory noise does not bias the computational model similar to
participants

We have so far showed that, for the normal video, event segments produced by
our model correlate with those produced by human observers. We further tested
our model to check whether it would be biased by the quality of sensory input,
in other words, by the reduction in the sensory change. In participants’ data,
the number of segments produced for the noisy video was less than those
produced for the normal video. Figure 6 shows the number of event boundaries
produced by the participants and our computational model. The model did not
show a similar bias, although the event boundary decisions results were
comparable.

Figure 6: The figure shows the difference between the mean number of event
boundaries detected in the normal and noisy videos by the participants and
computational models. The computational model did not show a similar bias in
the fine-grained segmentation, but the results were comparable in the coarse-
grained segmentation.

In the first part of this study, we investigated how the reduction of sensory
change affects event boundaries detected by participants. Results showed that
fine-grained event boundaries of participants correlated with sensory change,
whereas coarse-grained ones did not (Section 3.2.2). Additionally, when the
sensory information was degraded, the number of fine-grained event boundaries
detected by participants were reduced. Then, we assessed our computational
model and found out that decisions of our model were correlated with
participants and its fine-grained segmentation decisions showed higher
correlation with sensory change than its coarse-grained segmentation
decisions. Lastly, we checked whether our model was biased by degraded sensory
change similar to participants by comparing their number of event boundaries
in normal and noisy videos. Our model did detect higher number of fine-grained
event boundaries for the normal video, which was contrary to our expectations
(Section 3.2.5).

Overall, in this section, we showed that our model captures the essence of
event segmentation by exhibiting similar event segmentation decisions except
showing a bias to the noise in sensory information. In the second part, we
assessed the internal event representation space and the corresponding
similarity judgments of the model with those of humans.

##  4 Event representation experiments

The aim of this section is to analyze and compare the event representations
formed by the human participants and our computational model. However,
representations formed by humans are not directly accessible. Therefore, using
the observed PLD streams and the corresponding ground truth event labels, two
metrics are calculated in human experiments:

  * •

Pairwise Event Distance in Humans (PED-H): Assuming that events with similar
representations are reported to be similar by humans, we utilize similarity
judgements of the participants to collect pairwise event distances in humans.

  * •

Estimated Event Representation in Humans (EER-H): Given pairwise distances
between each event, we estimate the position of each event in a reconstructed
representation space using non-linear dimensionality transformation
techniques. This transformation places each event instance in a 2-D event
representation space such that the pairwise distances between events are
preserved.

Taking into account the metrics obtained in human experiments above, the
following two metrics, which serve as the counterparts in our computational
model, are used:

  * •

Pairwise Event Distance in Computational Model (PED-CM): Recall that each
event in our model is represented with a feed-forward neural network (FFNN).
The pairwise distances between events are calculated by taking the distances
between the outputs of the corresponding event models.

  * •

Estimated Event Representation in Computational Model (EER-CM): We train an
event model classifier, which is another neural network that maps the PLD
sensory data stream to the corresponding event model id ($M_{j}$). Here, the
neural activations, which are generated by the PLD data streams, encode the
representations of the corresponding events. These activations are mapped into
2-D event representation space using dimensionality reduction techniques. The
pairwise distances between events are calculated by taking the distances
between their locations in 2-D event representation.

Finally, we propose two additional baselines to assess metrics defined for the
computational model: PED-CM and EER-CM. These metrics are not computed from
the behavior of the computational model or the similarity judgments of humans.
They are computed based on the ground-truth data, in other words, the visual
appearance of PLDs streams. We expect this assessment to show whether the
defined metrics and, therefore, our computational model capture distances
between and representations of events similar to or better than ground-truth
data.

  * •

Pairwise Event Distances based on Visual Appearances (PED-VA): The pairwise
distances between events are estimated based on their visual appearances.

  * •

Pairwise Event Distance based on Changes in Visual Appearances (PED-CVA): PED-
VA is based locations of dots, which are necessarily dependent on previous dot
configurations in an event. Thus, for estimating distances between events
without the bias of previous dot locations, we estimated distances by changes
in visual appearances of events.

In the rest of this section, we first describe the details of the human
similarity judgment experiments, next provide the computation details of the
metrics introduced above, and finally give the results.

###  4.1 Procedure

####  4.1.1 Event Similarity Judgment Experiment with Human Participants

We devised an online psychological experiment based on the pairwise similarity
judgment paradigm (Shepard, 1980, 1987; Shepard Arabie, 1979) (see Figure 1J).
In this paradigm, participants are asked to report the similarity between two
items, and the obtained similarity matrices are transformed to a
representation space by multidimensional scaling (Shepard, 1980).

In our experiment, participants were presented with the videos side-by-side
and had a chance to play videos using the K and L buttons (K for left, L for
right) of the keyboard. They reported the degree of similarity of event videos
in both fine- and coarse-grained levels using a continuous ranking slide.
Psychological experiments were designed by Psychopy3 (Peirce ., 2019), and
online experiments were conducted via Pavlovia.

Since the number of segments increases the number of possible comparisons, we
reduced the number of selected events by sampling the ones that were also
segmented by our computational model (see Section 4.1.2 and Supplementary
Material for details). This sampling schema yielded 21 videos for the fine-
and 12 videos for the coarse-grained segmentation. In order to further
decrease the number of comparisons, we randomly selected 7 fine-grained and 4
coarse-grained event videos for each participant. This reduced the number of
comparisons to 49 for the fine- and 16 coarse-grained videos.

Forty-two participants (32 female, mean age 21) were recruited for the study
from the Research Participation System of Boğaziçi University. Other details
were same as those in the event segmentation experiment (Section 3.1).

This event similarity judgment experiment provided us with PED-H, the
similarity scores between the events in both fine- and coarse-grained levels.
We calculated EER-H metrics from PED-H by multi-dimensional scaling in the
scikit-learn package (Pedregosa ., 2011).

####  4.1.2 Event Similarity Judgment Extraction from the Computational Model

We defined two metrics for comparing representations of computational model
with those of participants, namely PED-CM and EER-CM.

The PED-CM is calculated by taking the distance between the outputs of the
corresponding event models. Recall that each event model in our computational
model is represented with a learned feed-forward neural network which predicts
the PLD configuration in the next time step given the displayed PLD
configurations. If the learned models are similar, then we expect similar
outputs, i.e. prediction trajectories, from those models. Therefore, given
arbitrary PLD streams, the distance between the prediction trajectories of two
event models provides a measure for their pairwise similarity. In order to
calculate the distance between two prediction trajectories, Dynamic Time
Warping (DTW) method (Giorgino, 2009) is used (Supplementary Material for
details).

EER-CM, on the other hand, is calculated by training a neural network
classifier that takes PLD data stream as input and predicts the corresponding
event model id as output. The neural activations of this classifier encodes
the predicted event by our computational model (see Figure 1H). These
activations are mapped to 2-D representation space using dimensionality
reduction methods such as PCA (Principal Component Analysis) (Wold ., 1987) or
t-SNE (T-Distributed Stochastic Neighbor) (Van der Maaten Hinton, 2008).
Finally, the mean location of the points that belong to the same event in the
2-D representation space is found and assigned as the representative position
for the corresponding event. The distance between events, on the other hand,
are found by calculating the Euclidean distance between these representative
positions. Due to the two dimensionality reduction methods, we received two
different metrics for EER-CM by using PCA (EER-CM/PCA) and using t-SNE (EER-
CM/t-SNE).

####  4.1.3 Similarity Judgments based on Visual Appearances

We invented two additional baselines for the evaluation of similarity
judgments and estimated event representations of the computational model,
namely PED-VA and PED-CVA. These techniques are used to estimate pairwise
visual similarities between events. PED-VA estimates visual similarities based
on actual trajectories of events; on the other hand, PED-CVA uses changes in
trajectories of events. In order to calculate the distance between actual
trajectories and their changes, Dynamic Time Warping (DTW) method (Giorgino,
2009) is used.

In this part of the study, we explained how we extracted several metrics for
the participants, the computational model, and baselines. Our aim is to find
out (1) whether similarity is a meaningful metric for events and (2) whether
our model captures event similarity judgments, in other words, estimated event
representations of participants. In particular, the performance of our model
in this task would be a sign for validating our model.

###  4.2 Results

In this section, we compared subjective similarity judgments of the
participants with the metrics we calculated for the computational model and
the baseline. In particular, first we assessed the correlation between the
similarity judgment of participants, namely PED-H, with those of the
computational model (PED-CM and EER-CM) and the baselines (PED-PLDs and PED-
CPLDs) in Section 4.2.2. Next, we visually compared event representations of
participants (EER-H) with representations estimated from the computational
model (EER-CM) in Section 4.2.3.

####  4.2.1 Event similarity judgments of participants show strong correlation
to their group

We averaged out the subjective similarity judgments of the participants to
compute the group similarity judgments (PED-H) and compared the performances
of the individual participants with the group decisions by using Pearson
correlation coefficients. This investigation (1) revealed correlation
distributions of the participants and (2) provided a basis for assessing the
representation discovery techniques. Results showed that that the mean Pearson
correlation coefficients of the participants to their group were 0.90 and 0.92
for the fine- and coarse-grained event comparisons, respectively.

####  4.2.2 Event similarity judgments of the computational model and
participants correlate each other

After receiving the group similarity judgments for event pairs (PED-H) from
the event similarity judgment experiment, we compared them to similarity
judgments we computed for the computational model and the baselines by Pearson
correlation. Recall our metrics for the computational model, namely PED-CM and
EER-CM, for the representation discovery. We received two results from the
EER-CM because of using two-dimensionality reduction techniques (EER-CM/PCA
and EER-CM/t-SNE) and one result from the PED-CM. For the baselines, we
received two results; one for the PED-PLDs and one for the PED-CPLDs.

Figure 7 displays the correlations of these metrics with group decisions. It
can be seen from the figure that the EER-CM/t-SNE was better than other
techniques and baselines (r = 0.435 for fine-grained and r = 0.614 for coarse-
grained). Statistical tests revealed that correlation scores of EER-CM/t-SNE
(r = 0.618) with PED-H was higher than EER-CM/PCA (r = 0.393), (z = 2.579, p =
0.004) for coarse-grained units. Moreover, the same also applied for fine-
grained units where EER-CM/t-SNE (r = 0.449) was significantly higher than
EER-CM/PCA (r = 0.339), (z = 1.92, p = 0.027).

These results suggested that the metric estimating event representations in
the computational model by a nonlinear transformation technique (EER-CM/t-SNE)
showed better results than the other techniques, even superior to baselines
estimating pairwise event distances based on visual appearances (PED-VA and
PED-CVA). That is, metrics based on visual appearances cannot capture human
similarity judgments (PED-H) well. We showed that, for capturing human
similarity judgments, a higher-level mechanism (i.e., event model classifier)
mapping sensory information (i.e., PLDs in the current case) to event units
seems necessary. In fact, in the literature, a higher-level mechanism
capturing the relationship between event pairs is proposed (Aslin, 2017;
Levine ., 2019).

Although a higher-level mechanism for capturing similarity judgments might be
necessary, it is apparent that it seems it is not sufficient. Recall that EER-
CM/PCa computes similarities between events based on the activations of a
higher-level mechanism. Despite this, EER-CM/PCA does not correlate well with
human similarity judgments (PED-H), and it seems inferior to baselines based
on visual appearances (PED-VA and PED-CVA). The dimensionality technique is
the only difference between EER-CM/t-SNE, showing superior performance by
using a nonlinear dimensionality reduction technique (Van der Maaten Hinton,
2008), and EER-CM/PCA, showing inferior performance by using a linear
dimensionality reduction technique (Wold ., 1987). Superiority of EER-CM/t-SNE
over EER-CM/PCA suggests that the former can represent the relationships of
events in higher dimensions on a 2-D event representation space by nonlinear
transformations. Thus, we can conclude that the higher-dimensional space
formed by the event model classifier correlates with those of humans.

Figure 7: Correlations of the representation discovery techniques and
baselines with the human data are given. The best technique was EER-CM/t-SNE
receiving an outstanding performance compared to the baseline models, namely
PED-PLDs and PED-CPLDs. Even though the correlation scores were not
approximated perfectly, they were comparable to the human performance, namely
0.90 and 0.92 for the fine- and coarse-grained event comparisons,
respectively.

So far, we have demonstrated that (1) people can judge similarities between
events, and (2) our model shows remarkable correlation scores with the human
similarity judgments.

For a visual exploration of event representations beyond correlation scores,
we compared EER-CM/t-SNE with EER-H on a self-organizing map (SOM) (Kohonen,
1990). SOM is an unsupervised machine learning technique that produces the
low-dimensional representation of data. In the literature, SOM is used is used
for action recognition (Huang Wu, 2010; Gharaee, 2021), classification
(Wünstel ., 2000) and analysis (Bauer Schöllhorn, 1997). Moreover, compared to
t-SNE and PCA, SOM preserves topological relationships in the data (Dou .,
2021), which makes it suitable for the visualization of events represented by
EER-CM/t-SNE on a 2-D space.

####  4.2.3 Self-organizing maps reveal visual similarities between
represented events

SOM is an unsupervised learning technique that extracts the structure of the
data by preserving topological relationships (Dou ., 2021; Kohonen, 1990).
Upon training, SOM forms a topographic map according to the patterns in the
input data and represents statistical patterns by the distances between
elements (Kohonen, 1990). In this part, we used SOM models to visually explore
event representations of the computational model and the participants on the
same 2-D space.

To visualize fine- and coarse-grained events on a 2-D space (Dou ., 2021), we
trained two SOM models for both the computational model and the participants.
SOM models trained for participants received their input from the PED-H that
is gathered from the event similarity judgment experiment, whereas those of
the computational model received their input from EER-CM/t-SNE as it reached
the best correlation score. We thereby trained SOM models by representing each
event by its perceived (i.e., the participants) or estimated (i.e., the
computational model) distance to other events in the dataset.

As a result of the training, we obtained locations of events for the
computational model and the participants on different 2-D spaces. To make the
comparison more accessible, we mapped the event locations of the computational
model to the 2-D space of participants. For mapping, we used the orthogonal
Procrustes method (Schönemann, 1966) in Scipy (Virtanen ., 2020) since it
preserves pairwise distances between events on a 2-D space.

The resulting 2-D space representing the fine-grained events was given in
Figure 8. The hexagons represent neurons of the SOM model on which events of
the computational model and the participants are located, and colors of
hexagons represent their distances from surrounding neurons (i.e., hexagons).
Events are labeled by a letter and two numbers, such as M.0.7. The letter
shows whether the event is represented by the model (M) or humans (H), the
first number indicates the event model used for the event, and the second
number shows the order of the segment that particular event model was used
for. For a visual comparison, we illustrated PLDs streams corresponding to
events and organized them in different boxes. For example, Figure 8A includes
two events labeled as 10.0 and 10.1, which are represented by neighboring
neurons. The spatial closeness on the 2D space represents the degree of
similarity. Throughout this section we will evaluate whether this similarity
corresponds to the visual and the semantic similarities between event
segments.

It can be seen that the representations of the computational model and
participants were well-matched to one another (see event segments coded as
10.0 and 10.1 in Figure 8-A, and 7.0 and 7.1 in Figure 8-B), aligning with the
resulting Pearson correlation coefficient (r = 0.435). Moreover, SOM displayed
the topological relationships between the represented units. For example,
Figure 8-A represents walking, Figure 8-B walking and leaning down, Figure 8-C
a complete bending (i.e., crouching down), Figure 8-D and Figure 8-E various
kinds of hand movements.

Despite matches between representations of the computational model and the
participants, there are several disagreements (see event segment coded as 0.1
(see Figure 8-D and Figure 8-E). The computational model considered 0.1 more
similar to 2.0 and 19.0, whereas participants perceived it as more like 20.0.
Interestingly, the neuron representing M.0.1 is distant from surrounding
neurons (see the color of the hexagon), which implies that the computational
model could not estimate the similarity of M.0.1 with other events properly
and assigned it to an individual neuron.

Figure 8: SOM results for the fine-grained event representations are given.
Event segments, in general, are represented by the same or neighborhood
neurons (see 10.0 and 10.1, 7-0 and 7.1). Moreover, SOM extracted the
topology, as expected. For example, (A) represents walking, (B) walking and
leaning down, (C) a complete bending (i.e., crouching down), (D), and (E) hand
movements. Despite this, there are several deviations. For instance (D and E),
the computational model considered 0-1 more similar to 2.0 and 19.0, whereas
humans thought it was more like 20.0. Lines connecting the points are pictured
for visualization purposes.

Representations produced for the coarse-grained event segments were given in
Figure 9. Similar to the fine-grained event segments, representations of the
computational model and participants overlapped one another (see M.0.0 and
H.0.0 and Figure 9-A). The overlap of representations is consistent with the
degree of correlation scores received in similarity judgments (r = 0.614).
Since the coarse-grained event units represent rich and diverse movement
sequences, interpreting their topological relationships is rather hard.

Event units may share similar sequences, and the computational model and the
participants seem to use those similarities in their judgments. For instance,
Figure 9-B displays three event segments: 0.3, 7.0, and 8.0. These event
segments differ from one another, but they involve a similar sequence, a
bending movement. The participants and the computational model might have
spotted the shared movement between sequences and considered them similar (see
M.7.0 and H.7.0 and compare them with M.8.0). This implies a shared strategy
between the model and the participants.

Figure 9-C displays a crucial difference between the model and participants
(see 2.0 and 2.1). Participants found 2.0 and 2.1 similar (see H.2.0 and
H.2.1) as they involve nearly the same movement sequence. On the other hand,
the model could not perceive them as similar (see M.2.0 and M.2.1). We think
this might have been because 2.1 and 2.0 differed in one movement sequence,
namely, raising both hands together, which took place in 2.1 but not in 2.0.
It is natural that participants found raising one or two hands similar, but
the model could not detect this fact. Although our model has a high
correlation with human similarity judgments (r = 0.614) and produces
overlapping representations on 2-D space, it fails to capture the relationship
between events in certain cases because of failing to conceptualize the
relationship between body parts (i.e., left and right hand) or actions.

In this part, we visually compared representations of the computational model
and the participants to reveal their similarities and differences. In general,
our model produced representations that overlap with those of participants.
However, we detected several cases of failure due to not being able to
estimate the similarity of an event and conceptualize the semantic
relationships between body parts or actions. In the next section, we will
discuss our results in the light of event segmentation and representation
literature and present possible improvements over the proposed model.

Figure 9: SOM results for the coarse-grained event representations are given.
(A) shows an instance, (B) shows the relationships between three coarse-
grained events, namely 0.3, 7.0, and 8.0. They share a similar movement, which
can be defined as bending. On the other hand, (C) shows a divergence between
humans and the computational model. Humans, but not the computational model,
found 2.0 (raising one hand) and 2.1 (raising two hands together) similar.
Lines connecting the points are pictured for visualization purposes.

##  5 Discussion

In this study, we developed a computational model of event segmentation,
learning, and representation by considering the limitations in the literature.
In the literature, several studies used stimuli which have discontinuities and
unnatural abruptions (Reynolds ., 2007; Metcalf Leake, 2017), some only worked
in the context of interaction (Gumbsch ., 2016, 2017, 2019), and the majority
of studies did not validate their model by psychological experiments (Gumbsch
., 2016, 2017, 2019; Reynolds ., 2007; Metcalf Leake, 2017). To the best of
our knowledge, the only model validated by the ground-truth data was tested
only for one granularity level (Franklin ., 2020). We addressed these
limitations by developing a simple semi-mechanistic and self-supervised
computational model and demonstrated that our model can produce multimodal
segments in varying hierarchies.

###  5.1 Humans and computers differ in terms of their temporal dynamics

One interesting output of this study is that humans and computers are
different in terms of their temporal dynamics, and researchers investigating
the cognitive and perceptual abilities should be cautious about these
differences (Funke ., 2021). In this study, we observed a delay in the event
segmentation responses of participants, which might have occurred due to the
decisional or motor processing delays. For a safe and sound comparison between
the proposed model and the ground-truth data, we shifted the responses of the
model in time. Our analysis showed that the proposed model received comparable
mean correlation scores with participants, namely 0.254 and 0.196 for the
fine- and coarse-grained segmentations, respectively, where the corresponding
values received by the best models were 0.32 and 0.256. Steady improvement of
the correlation scores as the time shifted forward implied that the comparison
between humans and computational models is not straightforward. Nonetheless,
shifting responses in time was an approximation; thus, further computational
models can benefit from other paradigms of event segmentation; for example,
the dwell-time paradigm can address to this limitation (Hard ., 2011). In the
dwell-time paradigm, participants are presented with a video frame by frame
and are asked to use a button for passing into the next frame. By considering
the time spent for each frame, an event boundary histogram can be extracted
without or with considerably less delay (Hard ., 2011). In comparison to the
study of Franklin . (2020), which is the only study which validates the model
with human response, our model received a higher correlation score. However,
while Franklin . (2020) used naturalistic videos directly, here, we used PLDs
extracted from natural videos. Despite this fact, our simple model extracting
the event boundaries by tracking the prediction error signals can also be
extended to the naturalistic videos.

###  5.2 Changes predict fine- but not coarse-grained event segmentation
responses

We conducted a series of comprehensive correlational analyses to reveal the
relationship between the change and the event segmentation performances of the
participants and our computational model (see Figure 3). In general, we showed
that the fine-grained segmentation decisions of humans were partly driven by
the absolute sensory change. In contrast, their coarse-grained segmentation
decisions seemed to be independent from the change in the sensory information.
Analyzing the relationship between the event segmentation decisions of
participants and the sensory change gave us a chance to interpret the
capabilities of our model. Figure 3 shows that the fine-grained segmentation
decisions of our model are largely correlated with the absolute sensory
change. On the other hand, Figure 5 shows that the effect of absolute sensory
change on the decisions of our model is reduced for the coarse-grained
segmentation. This implies the possibility of a weak top-down effect developed
by our model.

In the second condition of our event segmentation experiments, where the
videos to be segmented are noisy, we benefited from the well-known
relationship between the change and the event segmentation. We expected that
the reduction in sensory change and removing the fine-grained dynamics would
affect the event segmentation decisions of our model in line with those of the
participants. By verifying the relationship between the event segmentation and
the sensory change, participants produced coarser segments, as expected (see
Figure 6). When we tested our model trained on the normal video for the noisy
video, the number of boundaries did not show the bias we expected. We think
that the noise applied to the dataset might have increased the prediction
error signals observed by the model and led to the higher number of detected
event boundaries. Nevertheless, correlation scores of the computational model
for the noisy video segmentation were still better than the control models
(see Supplementary Material).

###  5.3 Our results conform to the conceptualization of uncertainty framework

The event segmentation ability of our model can be explained within the
context of the uncertainty framework (Dayan Yu, 2003, 2006). According to this
framework, a mental model can encounter two types of uncertainties:
uncertainties because of the known unreliability of the predictive
relationships and uncertainties above and beyond them, requiring a model
change (Dayan Yu, 2003, 2006; Payzan-LeNestour ., 2013; Zhao ., 2019; Yu
Dayan, 2005). Expected uncertainty is related to the inherent stochasticity of
the world that is expected from the perspective of the model. If an
uncertainty is expected, one does not have to reset its internal models since
they work for the current context. When a change is not explainable by the
model, on the other hand, this is an unexpected uncertainty as the current
model is not able to represent the current context and thus, the internal
models are better change. In our proposed model, the threshold values of each
event model determine their degrees of uncertainty or their representation
powers for the current context. Prediction errors below this value are
considered to be explainable within the dynamics of the model and therefore
those changes form the expected uncertainty. If the observed changes are
beyond the prediction capacity of the current model, in other words the
current prediction error exceeds its threshold, those changes form the
unexpected uncertainty and trigger a model reset. It has been observed that
the unexpected uncertainties suppress the top-down expectations and increase
the importance of newly acquired information (Dayan Yu, 2003, 2006; Farashahi
., 2017; Soltani Izquierdo, 2019; Yu Dayan, 2005). Our model works similarly.
When it observes an unexpected uncertainty, it suppresses the current event
model and enters into a search period for learning a new sequence.

###  5.4 Event representations reveal perception of events

We developed and employed a novel validation standard for our model based on
event representations. For this aim, we conducted an online pairwise
similarity judgment experiment and compared the similarity judgments of the
participants for the segmented events with those of the computational model.
We demonstrated that people were very reliable to the group decision in their
similarity judgments, even when the events were displayed as PLDs (Johansson,
1973). In particular, mean correlation scores of each participant with all
participants were 0.90 and 0.92 for the fine- and coarse-grained events,
respectively. When we extracted the representations of the participants via
various representation discovery techniques we developed, namely EER-CM/t-SNE,
our model also received considerable correlation scores with the similarity
judgments of the participants (0.435 and 0.614 for fine and coarse,
respectively). Moreover, representations discovered by the EER-CM/t-SNE
outperformed all the baseline models, directly using the information coming
from the ground-truth data rather than the activations of a neural network. We
believe that this finding is in line with the studies comparing the
representations of neural networks and humans, showing their similarities, and
using neural networks as a tool for capturing mental representations in
neuroscience (Khaligh-Razavi Kriegeskorte, 2014; Mur ., 2013; Tripp, 2017) and
psychology (Lake ., 2015; Dubey ., 2015; Kubilius ., 2016; Hebart ., 2020). As
a necessary reminder, EER-CM/t-SNE is partially dependent on the event
segmentation mechanism of our model. The technique uses activations of a
neural network model which receives various segmented events and discovers
semantic relationship between those instances. This simple framework, namely
segmenting event instances by monitoring the prediction errors and relating
segments via an hierarchical mechanism implies the presence of a hierarchical
(possibly recurrent) system in the event cognition. In fact, studies suggest
that infants can find higher-level units based on the structural relationship
between lower-level units (Levine ., 2019; Aslin, 2017).

Despite its success in capturing the human event similarity judgments, EER-
CM/t-SNE and EER-CM/PCA have an importation limitation. Basically, those
models map each timestep to an event instance and, by this way, form the
representation of each timestep as the training unfolds. Representations of
each timesteps (frame-based representation) of a particular event are averaged
out to extract a representation for that event (event-based representation).
Although this technique is simple and understandable, the mean operation,
namely averaging out, assigns the same credit for all timesteps in an event.
However, this may not be the most suitable operation as humans remember (i.e.,
give an information processing advantage) the first and the end of a sequence
better than its midpoint, known as the serial position effect (Jonides .,
2008; Murdock, 1962). Further research might consider developing a method that
aggregates frame-based representations based on the saliencies of the
individual frames.

###  5.5 Limitations

Our model has a number of limitations. First of all, event models in our
framework are distinct from one another, even though human actions are
groupable. With the aim of addressing this issue, we connected different event
segments by several representation discovery techniques. Further research can
investigate how the relationship between the event instances can be utilized
for achieving more robust event segmentation. Secondly, although our model can
segment events in different lengths, it is not able to capture the part-whole
relationship and use this information for the hierarchical segmentation. Even
though we tried to exploit the event similarity judgments to capture the
hierarchical organization of events, it was not successful. Recent research
shows the possibility that hierarchical segmentation requires tracking higher-
order statistical regularities between the event units (Franklin ., 2020;
Schapiro ., 2013). Subsequent research can extend the proposed model in this
way by learning temporal regularities between the event labels by a
hierarchical recurrent neural network model. Further research can exploit the
latent representation space proposed in this study for building a top-down
signal, regulating behaviors of the event models. Such a latent representation
space can be exploited to capture the event relationships, learn the new
events faster, and segment events into parts more reliably.

At the first step of modeling event segmentation, we represented events by
PLDs, which are utilized in event segmentation literature (Reynolds ., 2007;
Metcalf Leake, 2017). Further research can use naturalistic videos -rather
than PLDs- to capture event segmentation abilities in certain event types,
including human-object, human-human interactions, and scene-related changes.
Since the current model trains all event models to find the next event model
in the search period, using raw naturalistic videos may increase the
computational load. Nevertheless, the current model can be scaled by an
already trained object identification model such as AlexNet (Krizhevsky .,
2012) or a dimensionality reduction algorithm such as PCA or autoencoders to
reduce data dimensionality to achieve human-level event segmentation
performance in the naturalistic videos. In the case of using sequences of
images (i.e., videos) for representing events, convolutional neural networks
(CNNs) might be more advantageous than the multi-layer perceptrons (Goodfellow
., 2016; Haushofer ., 2008; Kriegeskorte Mur, 2012).

Despite these limitations, our model allows researchers to integrate various
systems in a simple manner for exploring the prediction-error-based event
segmentation and other perceptual and cognitive processes. This advantage is
due to the semi-mechanistic nature of the model. For example, one can enrich
the current model by defining the adaptive learning rates by comparing the
thresholds of models and received prediction errors, and therefore, explore
the effects of expected and unexpected uncertainties on learning (Dayan Yu,
2003, 2006; Farashahi ., 2017; Soltani Izquierdo, 2019; Yu Dayan, 2005).
Another possibility is to target prediction error signals and investigate
their role in certain perceptual and cognitive processes. For instance, one
can extend our model into a time perception model, which aggregates prediction
error signals for estimating the duration of an event (Basgol ., 2022; Fountas
., 2022).

###  5.6 Conclusion

In this study, we developed a computational model of event segmentation,
learning, and representation. Our model represents events by multi-layer
perceptrons and manages them by monitoring their prediction error signals. By
this way, our model can produce multimodal event segments in varying
hierarchies via passive observation. We compared the capabilities of our model
by event segmentation and representation experiments.

  * •

We evaluated fine-grained and coarse-grained event segmentation performances
of the model with two videos, representing various human behaviors by PLDs
(Johansson, 1973). We then compared its performance by online psychological
experiments. We showed that the proposed model received 0.254 and 0.196 point-
biserial correlation scores for the fine- and coarse-grained segmentation,
respectively.

  * •

With the aim of examining whether the event segments that are produced by our
model are interpretable and extractable from the activations of neural
networks, we proposed a new validation technique inspired from the literature
(Peterson ., 2018). We approximated the event representations of our model by
several techniques of participants by an event similarity judgment experiment
and extracted their event representation spaces. We compared the event
representation spaces of our model with those of the participants and received
the correlation scores of 0.435 and 0.614 for the fine- and coarse-grained
segments, respectively.

  * •

Our results suggest that a model based on monitoring the prediction error
signals can capture the event segmentation decisions of humans, and multi-
layer perceptrons can capture the internal representation space.

  * •

From a broader perspective, our results confirm an already held theory, event
segmentation theory (Zacks Swallow, 2007), and computational and robotic
models in the literature (Gumbsch ., 2016, 2017, 2019; Reynolds ., 2007;
Metcalf Leake, 2017). Moreover, they imply a hierarchical mechanism relating
event segments (Levine ., 2019; Aslin, 2017) and present the possibility of
tracking higher-order statistical regularities between event segments in a
more opaque way (Franklin ., 2020; Schapiro ., 2013). The present model shows
how perceptual and cognitive abilities can be modeled with maintaining
interpretability and without the need of human-crafted labels.

##  6 Acknowledgement

This work was supported in part by the Bogazici Research Projects (BAP),
Project titled “A Computational Model of Event Learning and Segmentation:
Event Granularity, Sensory Reliability and Expectation,” under Grant 20A01P2.

## References

  * Alaerts . (2011) Alaerts2011ActionAlaerts, K., Nackaerts, E., Meyns, P., Swinnen, SP. Wenderoth, N.  2011\.  Action and emotion recognition from point light displays: an investigation of gender differences Action and emotion recognition from point light displays: an investigation of gender differences.  PLoS ONE66e20989.  10.1371/journal.pone.0020989
  * Andrychowicz . (2017) Andrychowicz2017HindsightAndrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P.Zaremba, W.  2017\.  Hindsight Experience Replay Hindsight experience replay.  I. Guyon . (), Advances in Neural Information Processing Systems Advances in neural information processing systems ( 30).  Curran Associates, Inc.
  * Aslin (2017) aslin2017statisticalAslin, RN.  2017\.  Statistical learning: a powerful mechanism that operates by mere exposure Statistical learning: a powerful mechanism that operates by mere exposure.  Wiley Interdisciplinary Reviews: Cognitive Science81-2e1373.  10.1002/wcs.1373
  * Basgol . (2022) Basgol2021TimePerceptionBasgol, H., Ayhan, I. Ugur, E.  2022\.  Time perception: A review on psychological, computational, and robotic models Time perception: A review on psychological, computational, and robotic models.  IEEE Transactions on Cognitive and Developmental Systems142301-315.  10.1109/TCDS.2021.3059045
  * Bauer Schöllhorn (1997) bauer1997selfBauer, HU. Schöllhorn, W.  1997\.  Self-organizing maps for the analysis of complex movement patterns Self-organizing maps for the analysis of complex movement patterns.  Neural Processing Letters53193–199.  10.1023/A:1009646811510
  * Bengio . (2014) Bengio2014RepresentationBengio, Y., Courville, A. Vincent, P.  2014\.  Representation learning: A review and new perspectives Representation learning: A review and new perspectives.  IEEE Transactions on Pattern Analysis and Machine Intelligence3581798 – 1828.  10.1109/TPAMI.2013.50
  * Bertenthal . (1987) bertenthal1987perceptionBertenthal, BI., Proffitt, DR. Kramer, SJ.  1987\.  Perception of biomechanical motions by infants: Implementation of various processing constraints Perception of biomechanical motions by infants: Implementation of various processing constraints.  Journal of Experimental Psychology: Human Perception and Performance134577.  10.1037//0096-1523.13.4.577
  * Blom . (2020) Blom2020PredictionsBlom, T., Feuerriegel, D., Johnson, P., Bode, S. Hogendoorn, H.  2020\.  Predictions drive neural representations of visual events ahead of incoming sensory information Predictions drive neural representations of visual events ahead of incoming sensory information.  Proceedings of the National Academy of Sciences117137510–7515.  10.1073/pnas.1917777117
  * Blough (2001) Blough2001PerceptionBlough, D.  2001\.  The perception of similarity The perception of similarity.  RG. Cook (), Avian visual cognition Avian visual cognition (1 , 6, 23–25).  Comparative Cognition Press.  www.pigeon.psy.tufts.edu/avc/dblough/
  * Clark (2013) Clark2013WhateverClark, A.  2013\.  Whatever next? Predictive brains, situated agents, and the future of cognitive science Whatever next? Predictive brains, situated agents, and the future of cognitive science.  Behavioral and Brain Sciences363181–204.  10.1017/s0140525x12000477
  * Cutting (2014) cutting2014eventCutting, JE.  2014\.  Event segmentation and seven types of narrative discontinuity in popular movies Event segmentation and seven types of narrative discontinuity in popular movies.  Acta Psychologica14969–77.  10.1016/j.actpsy.2014.03.003
  * Cutting . (2012) Cutting2012PerceivingCutting, JE., Brunick, KL. Candan, A.  2012\.  Perceiving event dynamics and parsing Hollywood films Perceiving event dynamics and parsing Hollywood films.  Journal of Experimental Psychology: Human Perception and Performance3861476–1490.  10.1037/a0027737
  * Day Bartels (2008) Day2008RepresentationDay, SB. Bartels, DM.  2008\.  Representation over time: The effects of temporal distance on similarity Representation over time: The effects of temporal distance on similarity.  Cognition10631504–1513.  10.1016/j.cognition.2007.05.013
  * Dayan Yu (2003) dayan2003uncertaintyDayan, P. Yu, A.  2003\.  Uncertainty and learning Uncertainty and learning.  IETE Journal of Research492-3171–181.  10.1080/03772063.2003.11416335
  * Dayan Yu (2006) dayan2006phasicDayan, P. Yu, A.  2006\.  Phasic norepinephrine: a neural interrupt signal for unexpected events Phasic norepinephrine: a neural interrupt signal for unexpected events.  Network: Computation in Neural Systems174335–350.  10.1080/09548980601004024
  * Deselaers Ferrari (2011) Deselaers2011VisualDeselaers, T. Ferrari, V.  2011\.  Visual and semantic similarity in ImageNet Visual and semantic similarity in ImageNet.  Computer vision and pattern recognition Computer vision and pattern recognition ( 1777–1784).  IEEE.  10.1109/cvpr.2011.5995474
  * Dias Dimiccoli (2018) Dias2019LearningDias, C. Dimiccoli, M.  2018\.  Learning Event Representations by Encoding the Temporal Context Learning Event Representations by Encoding the Temporal Context.  Proceedings of the European Conference on Computer Vision (ECCV) Proceedings of the european conference on computer vision (eccv) ( 11131 LNCS, 587–596).  Springer Verlag.  10.1007/978-3-030-11015-4_44
  * Dou . (2021) dou2021vDou, H., Xu, B., Shen, F. Zhao, J.  2021\.  V-SOINN: A topology preserving visualization method for multidimensional data V-soinn: A topology preserving visualization method for multidimensional data.  Neurocomputing449280–289.  10.1016/j.neucom.2021.03.113
  * Dubey . (2015) dubey2015makesDubey, R., Peterson, J., Khosla, A., Yang, MH. Ghanem, B.  2015\.  What Makes an Object Memorable? What makes an object memorable?  Proceedings of the IEEE International Conference on Computer Vision Proceedings of the ieee international conference on computer vision ( 1089–1097).
  * Eisenberg . (2018) Eisenberg2018DynamicPredictionEisenberg, ML., Zacks, JM. Flores, S.  2018\.  Dynamic prediction during perception of everyday events Dynamic prediction during perception of everyday events.  Cognitive Research: Principles and Implications311–12.  10.1186/s41235-018-0146-z
  * Eslami . (2018) Eslami2018NeuralEslami, SMA., Jimenez Rezende, D., Besse, F., Viola, F., Morcos, AS., Garnelo, M.Hassabis, D.  2018\.  Neural scene representation and rendering Neural scene representation and rendering.  Science36063941204–1210.  10.1126/science.aar6170
  * Farashahi . (2017) farashahi2017metaplasticityFarashahi, S., Donahue, CH., Khorsand, P., Seo, H., Lee, D. Soltani, A.  2017\.  Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty Metaplasticity as a neural substrate for adaptive learning and choice under uncertainty.  Neuron942401–414.  10.1016/j.neuron.2017.03.044
  * Fivush . (1992) Fivush1992StructureFivush, R., Kuebli, J. Clubb, PA.  1992\.  The structure of events and event Representations: A developmental analysis The structure of events and event Representations: A developmental analysis.  Child Development631188.  10.2307/1130912
  * Fleming Storrs (2019) fleming2019learningFleming, RW. Storrs, KR.  2019\.  Learning to see stuff Learning to see stuff.  Current Opinion in Behavioral Sciences30100–108.  10.1016/j.cobeha.2019.07.004
  * Floridi Chiriatti (2020) floridi2020gptFloridi, L. Chiriatti, M.  2020\.  GPT-3: Its nature, scope, limits, and consequences Gpt-3: Its nature, scope, limits, and consequences.  Minds and Machines304681–694.  10.1007/s11023-020-09548-1
  * Fountas . (2022) Fountas2020PredictiveProcessingFountas, Z., Sylaidi, A., Nikiforou, K., Seth, AK., Shanahan, M. Roseboom, W.  2022\.  A predictive processing model of episodic memory and time perception A predictive processing model of episodic memory and time perception.  Neural Computation3471501–1544.  10.1162/neco_a_01514
  * Fox McDaniel (1982) fox1982perceptionFox, R. McDaniel, C.  1982\.  The perception of biological motion by human infants The perception of biological motion by human infants.  Science2184571486–487.  10.1126/science.7123249
  * Franklin . (2020) Franklin2020StructuredFranklin, NT., Norman, KA., Ranganath, C., Zacks, JM. Gershman, SJ.  2020\.  Structured event memory: A neuro-symbolic model of event cognition Structured event memory: A neuro-symbolic model of event cognition.  Psychological Review1273327–361.  10.1037/rev0000177
  * Funke . (2021) Funke2021FivePointsFunke, CM., Borowski, J., Stosio, K., Brendel, W., Wallis, TS. Bethge, M.  2021\.  Five points to check when comparing visual perception in humans and machines Five points to check when comparing visual perception in humans and machines.  Journal of Vision21316–16.
  * Gharaee (2021) gharaee2021onlineGharaee, Z.  2021\.  Online recognition of unsegmented actions with hierarchical SOM architecture Online recognition of unsegmented actions with hierarchical som architecture.  Cognitive Processing22177–91.  10.1007/s10339-020-00986-4
  * Giorgino (2009) Giorgino2009ComputingGiorgino, T.  2009\.  Computing and visualizing dynamic time warping alignments in R: The dtw package Computing and visualizing dynamic time warping alignments in R: The dtw package.  Journal of Statistical Software3171–24.  10.18637/jss.v031.i07
  * Goodfellow . (2016) Goodfellow2016DeepGoodfellow, I., Bengio, Y. Courville, A.  2016\.  Deep Learning Deep Learning (1 ).  MIT Press.  https://www.deeplearningbook.org
  * Gosling Mason (2015) Gosling2015InternetGosling, SD. Mason, W.  2015\.  Internet research in psychology Internet research in psychology.  Annual Review of Psychology66877–902.  10.1146/annurev-psych-010814-015321
  * Grossman . (2000) grossman2000brainGrossman, E., Donnelly, M., Price, R., Pickens, D., Morgan, V., Neighbor, G. Blake, R.  2000\.  Brain areas involved in perception of biological motion Brain areas involved in perception of biological motion.  Journal of Cognitive Neuroscience125711–720.  10.1162/089892900562417
  * Gumbsch . (2019) Gumbsch2019AutonomousGumbsch, C., Butz, MV. Martius, G.  2019\.  Autonomous identification and goal-directed invocation of event-predictive behavioral primitives Autonomous identification and goal-directed invocation of event-predictive behavioral primitives.  IEEE Transactions on Cognitive and Developmental Systems111–1.  10.1109/TCDS.2019.2925890
  * Gumbsch . (2016) Gumbsch2016LearningGumbsch, C., Kneissler, J. Butz, MV.  2016\.  Learning Behavior-Grounded Event Segmentations Learning Behavior-Grounded Event Segmentations.  Proceedings of the 38th Annual Meeting of the Cognitive Science Society Proceedings of the 38th annual meeting of the cognitive science society ( 1787–1792).  AustinCurran Associates, Inc.
  * Gumbsch . (2017) Gumbsch2017ComputationalModelDynGumbsch, C., Otte, S. Butz, MV.  2017\.  A Computational Model for the Dynamical Learning of Event Taxonomies A Computational Model for the Dynamical Learning of Event Taxonomies.  Proceedings of the 39th Annual Meeting of the Cognitive Science Society Proceedings of the 39th annual meeting of the cognitive science society ( 452–457).  London, UKCurran Associates, Inc.
  * Hard . (2011) Hard2011ShapeHard, BM., Recchia, G. Tversky, B.  2011\.  The shape of action The shape of action.  Journal of Experimental Psychology: General1404586–604.  10.1037/a0024310
  * Hard Tversky (2003) Hard2003SegmentingHard, BM. Tversky, B.  2003\.  Segmenting Ambiguous Events Segmenting Ambiguous Events.  Proceedings of the 25th Annual Meeting of the Cognitive Science Society Proceedings of the 25th annual meeting of the cognitive science society ( 25, 781–786).  BostonCurran Associates, Inc.
  * Hard . (2006) Hard2006MakingHard, BM., Tversky, B. Lang, DS.  2006\.  Making sense of abstract events: Building event schemas Making sense of abstract events: Building event schemas.  Memory & Cognition3461221–1235.  10.3758/bf03193267
  * Haushofer . (2008) Haushofer2008MultivariateHaushofer, J., Livingstone, MS. Kanwisher, N.  2008\.  Multivariate patterns in object-selective cortex dissociate perceptual and physical shape similarity Multivariate patterns in object-selective cortex dissociate perceptual and physical shape similarity.  PLoS Biology671459–1467.  10.1371/journal.pbio.0060187
  * He . (2016) he2016deepHe, K., Zhang, X., Ren, S. Sun, J.  2016\.  Deep Residual Learning for Image Recognition Deep residual learning for image recognition.  2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016 ieee conference on computer vision and pattern recognition (cvpr) ( 770-778).  10.1109/CVPR.2016.90
  * Hebart . (2020) Hebart2020RevealingHebart, MN., Zheng, CY., Pereira, F. Baker, CI.  2020\.  Revealing the multidimensional mental representations of natural objects underlying human similarity judgements Revealing the multidimensional mental representations of natural objects underlying human similarity judgements.  Nature Human Behaviour4111173–1185.  10.1038/s41562-020-00951-3
  * Hornik . (1989) hornik1989multilayerHornik, K., Stinchcombe, M. White, H.  1989\.  Multilayer feedforward networks are universal approximators Multilayer feedforward networks are universal approximators.  Neural Networks25359–366.  10.1016/0893-6080(89)90020-8
  * Huang Wu (2010) huang2010humanHuang, W. Wu, QJ.  2010\.  Human Action Recognition based on Self Organizing Map Human action recognition based on self organizing map.  2010 IEEE International Conference on Acoustics, Speech and Signal Processing 2010 ieee international conference on acoustics, speech and signal processing ( 2130–2133).
  * Huff . (2014) huff2014changesHuff, M., Meitz, TG. Papenmeier, F.  2014\.  Changes in situation models modulate processes of event perception in audiovisual narratives Changes in situation models modulate processes of event perception in audiovisual narratives.  Journal of Experimental Psychology: Learning Memory and Cognition4051377–1388.  10.1037/a0036780
  * Johansson (1973) Johansson1973VisualPerceptionJohansson, G.  1973\.  Visual perception of biological motion and a model for its analysis Visual perception of biological motion and a model for its analysis.  Perception & Psychophysics142201–211.  10.3758/bf03212378
  * Jonides . (2008) Jonides2008MindJonides, J., Lewis, RL., Nee, DE., Lustig, CA., Berman, MG. Moore, KS.  2008\.  The mind and brain of short-term memory The mind and brain of short-term memory.  Annual Review of Psychology591193–224.  10.1146/annurev.psych.59.103006.093615
  * Khaligh-Razavi Kriegeskorte (2014) khaligh2014deepKhaligh-Razavi, SM. Kriegeskorte, N.  2014\.  Deep supervised, but not unsupervised, models may explain IT cortical representation Deep supervised, but not unsupervised, models may explain it cortical representation.  PLoS Computational Biology1011e1003915.  10.1371/journal.pcbi.1003915
  * Kohonen (1990) Kohonen1990SelfOrganizingKohonen, T.  1990\.  The self-organizing map The self-organizing map.  Proceedings of the IEEE7891464-1480.  10.1109/5.58325
  * Kominsky . (2021) Kominsky2020CausalityKominsky, JF., Baker, L., Keil, FC. Strickland, B.  2021\.  Causality and continuity close the gaps in event representations Causality and continuity close the gaps in event representations.  Memory & Cognition493518–531.  10.3758/s13421-020-01102-9
  * Krakowski . (2011) krakowski2011neurophysiologyKrakowski, AI., Ross, LA., Snyder, AC., Sehatpour, P., Kelly, SP. Foxe, JJ.  2011\.  The neurophysiology of human biological motion processing: a high-density electrical mapping study The neurophysiology of human biological motion processing: a high-density electrical mapping study.  NeuroImage561373–383.  10.1016/j.neuroimage.2011.01.058
  * Kriegeskorte Mur (2012) Kriegeskorte2012InverseKriegeskorte, N. Mur, M.  2012\.  Inverse MDS: Inferring dissimilarity structure from multiple item arrangements Inverse MDS: Inferring dissimilarity structure from multiple item arrangements.  Frontiers in Psychology31245.  10.3389/fpsyg.2012.00245
  * Krizhevsky . (2012) krizhevsky2012imagenetKrizhevsky, A., Sutskever, I. Hinton, GE.  2012\.  ImageNet Classification with Deep Convolutional Neural Networks Imagenet classification with deep convolutional neural networks.  F. Pereira, C. Burges, L. Bottou K. Weinberger (), Advances in Neural Information Processing Systems Advances in neural information processing systems ( 25).  Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2012/file/ c399862d3b9d6b76c8436e924a68c45b-Paper.pdf
  * Kubilius . (2016) kubilius2016deepKubilius, J., Bracci, S. Op de Beeck, HP.  2016\.  Deep neural networks as a computational model for human shape sensitivity Deep neural networks as a computational model for human shape sensitivity.  PLoS Computational Biology124e1004896.  10.1371/journal.pcbi.1004896
  * Kurby Zacks (2008) Kurby2008SegmentationKurby, CA. Zacks, JM.  2008\.  Segmentation in the perception and memory of events Segmentation in the perception and memory of events.  Trends in Cognitive Sciences12272–79.  10.1016/j.tics.2007.11.004
  * Lake . (2015) lake2015deepLake, BM., Zaremba, W., Fergus, R. Gureckis, TM.  2015\.  Deep Neural Networks Predict Category Typicality Ratings for Images Deep neural networks predict category typicality ratings for images.  Proceedings of the 37th Annual Conference of the Cognitive Science Society. Proceedings of the 37th annual conference of the cognitive science society.  Curran Associates, Inc.
  * Levine . (2019) levine2019findingLevine, D., Buchsbaum, D., Hirsh-Pasek, K. Golinkoff, RM.  2019\.  Finding events in a continuous world: A developmental account Finding events in a continuous world: A developmental account.  Developmental Psychobiology613376–389.  10.1002/dev.21804
  * Lippmann (1989) lippmann1989patternLippmann, RP.  1989\.  Pattern classification using neural networks Pattern classification using neural networks.  IEEE Communications Magazine271147-50.  10.1109/35.41401
  * Liu . (2021) liu2021selfLiu, X., Zhang, F., Hou, Z., Mian, L., Wang, Z., Zhang, J. Tang, J.  2021\.  Self-supervised learning: Generative or contrastive Self-supervised learning: Generative or contrastive.  IEEE Transactions on Knowledge and Data Engineering1-1.  10.1109/TKDE.2021.3090866
  * Metcalf Leake (2017) Metcalf2017ModellingMetcalf, K. Leake, DB.  2017\.  Modelling Unsupervised Event Segmentation: Learning Event Boundaries from Prediction Errors Modelling Unsupervised Event Segmentation: Learning Event Boundaries from Prediction Errors.  EJ. Davelaar (), Proceedings of the 39th Annual Conference of the Cognitive Science Society Proceedings of the 39th annual conference of the cognitive science society ( 2717–2722).  Curran Associates, Inc.
  * Michels . (2009) michels2009brainMichels, L., Kleiser, R., de Lussanet, MH., Seitz, RJ. Lappe, M.  2009\.  Brain activity for peripheral biological motion in the posterior superior temporal gyrus and the fusiform gyrus: Dependence on visual hemifield and view orientation Brain activity for peripheral biological motion in the posterior superior temporal gyrus and the fusiform gyrus: Dependence on visual hemifield and view orientation.  NeuroImage451151–159.  10.1016/j.neuroimage.2008.10.063
  * Mikolov . (2013) Mikolov2013DistributedMikolov, T., Sutskever, I., Chen, K., Corrado, G. Dean, J.  2013\.  Distributed Representations of Words and Phrases and their Compositionality Distributed Representations of Words and Phrases and their Compositionality.  ( 3111–3119).  Curran Associates.
  * Mur . (2013) mur2013humanMur, M., Meys, M., Bodurka, J., Goebel, R., Bandettini, PA. Kriegeskorte, N.  2013\.  Human object-similarity judgments reflect and transcend the primate-IT object representation Human object-similarity judgments reflect and transcend the primate-it object representation.  Frontiers in Psychology4128.  10.3389/fpsyg.2013.00128
  * Murdock (1962) Murdock1962SerialMurdock, BB.  1962\.  The serial position effect of free recall The serial position effect of free recall.  Journal of Experimental Psychology645482–488.  10.1037/h0045106
  * Neisser (1976) Neisser1976CognitionNeisser, U.  1976\.  Cognition and reality: Principles and implications of cognitive psychology Cognition and reality: Principles and implications of cognitive psychology (1 ).  New YorkW. H. Freeman.
  * Newtson (1973) Newtson1973AttributionNewtson, D.  1973\.  Attribution and the unit of perception of ongoing behavior Attribution and the unit of perception of ongoing behavior.  Journal of Personality and Social Psychology28128–38.  10.1037/h0035584
  * Newtson Engquist (1976) Newtson1976PerceptualNewtson, D. Engquist, G.  1976\.  The perceptual organization of ongoing behavior The perceptual organization of ongoing behavior.  Journal of Experimental Social Psychology125436–450.  10.1016/0022-1031(76)90076-7
  * Newtson . (1977) Newtson1977ObjNewtson, D., Engquist, GA. Bois, J.  1977\.  The objective basis of behavior units The objective basis of behavior units.  Journal of Personality and Social Psychology3512847–862.  10.1037/0022-3514.35.12.847
  * Nosofsky (1992) Nosofsky1992SimilarityNosofsky, RM.  1992\.  Similarity scaling and cognitive process models Similarity scaling and cognitive process models.  Annual Review of Psychology43125–53.  10.1146/annurev.ps.43.020192.000325
  * Ólafsdóttir . (2018) Olafsdottir2018RoleÓlafsdóttir, HF., Bush, D. Barry, C.  2018\.  The role of hippocampal replay in memory and planning The role of hippocampal replay in memory and planning.  Current Biology28137–50.  10.1016/j.cub.2017.10.073
  * O’Reilly (2013) OReilly2013MakingO’Reilly, JX.  2013\.  Making predictions in a changing world-inference, uncertainty, and learning Making predictions in a changing world-inference, uncertainty, and learning.  Frontiers in Neuroscience77105.  10.3389/fnins.2013.00105
  * Orhan . (2020) orhan2020selfOrhan, AE., Gupta, VV. Lake, BM.  2020\.  Self-supervised learning through the eyes of a child Self-supervised learning through the eyes of a child.  arXiv preprint arXiv:2007.16189.
  * Pavlova . (2004) pavlova2004dissociablePavlova, M., Lutzenberger, W., Sokolov, A. Birbaumer, N.  2004\.  Dissociable cortical processing of recognizable and non-recognizable biological movement: analysing gamma MEG activity Dissociable cortical processing of recognizable and non-recognizable biological movement: analysing gamma meg activity.  Cerebral Cortex142181–188.  10.1093/cercor/bhg117
  * Payzan-LeNestour . (2013) payzan2013neuralPayzan-LeNestour, E., Dunne, S., Bossaerts, P. O’Doherty, JP.  2013\.  The neural representation of unexpected uncertainty during value-based decision making The neural representation of unexpected uncertainty during value-based decision making.  Neuron791191–201.  10.1016/j.neuron.2013.04.037
  * Pedregosa . (2011) scikit-learnPedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O.Duchesnay, E.  2011\.  Scikit-learn: Machine Learning in Python Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research122825–2830.
  * Peirce . (2019) Peirce2019PsychopyPeirce, J., Gray, JR., Simpson, S., Macaskill, M., Höchenberger, R., Sogo, H.Lindeløv, JK.  2019\.  PsychoPy2: Experiments in behavior made easy PsychoPy2: Experiments in behavior made easy.  Behavior Research Methods511195–203.  10.3758/s13428-018-01193-y
  * Peterson . (2018) Peterson2018EvaluatingPeterson, JC., Abbott, JT. Griffiths, TL.  2018\.  Evaluating (and improving) the correspondence between deep neural networks and human representations Evaluating (and improving) the correspondence between deep neural networks and human representations.  Cognitive Science4282648–2669.  10.1111/cogs.12670
  * Peuskens . (2005) peuskens2005specificityPeuskens, H., Vanrie, J., Verfaillie, K. Orban, GA.  2005\.  Specificity of regions processing biological motion Specificity of regions processing biological motion.  European Journal of Neuroscience21102864–2875.  10.1111/j.1460-9568.2005.04106.x
  * Pitt (2020) Pitt2020MentalRepresentationPitt, D.  2020\.  Mental representation Mental representation.  EN. Zalta (), The Stanford encyclopedia of philosophy The Stanford encyclopedia of philosophy (Spring 202 ).  Metaphysics Research Lab, Stanford University.  https://plato.stanford.edu/ archives/spr2020/entries/mental-representation/.
  * Plappert . (2016) Plappert2016KITPlappert, M., Mandery, C. Asfour, T.  2016\.  The KIT motion-language dataset The KIT motion-language dataset.  Big Data44236–252.  http://arxiv.org/pdf/1607.03827  10.1089/big.2016.0028
  * Radford . (2017) radford2017learningRadford, A., Jozefowicz, R. Sutskever, I.  2017\.  Learning to generate reviews and discovering sentiment Learning to generate reviews and discovering sentiment.  arXiv preprint arXiv:1704.01444.
  * Reynolds . (2007) Reynolds2007ComputationalReynolds, JR., Zacks, JM. Braver, TS.  2007\.  A computational model of event segmentation from perceptual prediction A computational model of event segmentation from perceptual prediction.  Cognitive Science314613–643.  10.1080/15326900701399913
  * Richmond Zacks (2017) Richmond2017ConstructingRichmond, LL. Zacks, JM.  2017\.  Constructing experience: Event models from perception to action Constructing experience: Event models from perception to action.  Trends in Cognitive Sciences2112962–980.  10.1016/j.tics.2017.08.005
  * Rogers McClelland (2005) Rogers2005ParallelRogers, TT. McClelland, JL.  2005\.  A parallel distributed processing approach to semantic cognition: Applications to conceptual development A parallel distributed processing approach to semantic cognition: Applications to conceptual development (DH. Gershkoff-Stowe, L., Rakison, ).  New JerseyLawrence Erlbaum Associates Publishers.
  * Rothfuss . (2018) Rothfuss2018DeepEpisodicRothfuss, J., Ferreira, F., Aksoy, EE., Zhou, Y. Asfour, T.  2018\.  Deep episodic memory: Encoding, recalling, and predicting episodic experiences for robot action execution Deep episodic memory: Encoding, recalling, and predicting episodic experiences for robot action execution.  IEEE Robotics and Automation Letters344007–4014.  10.1109/LRA.2018.2860057
  * Rudin Radin (2019) rudin2019weRudin, C. Radin, J.  2019\.  Why are we using black box models in AI when we don’t need to? A lesson from an explainable AI competition Why are we using black box models in ai when we don’t need to? a lesson from an explainable ai competition.  Harvard Data Science Review12.  10.1162/99608f92.5a8a3a3d
  * Rumelhart . (1986) rumelhart1986learningRumelhart, DE., Hinton, GE. Williams, RJ.  1986\.  Learning representations by back-propagating errors Learning representations by back-propagating errors.  Nature3236088533–536.  10.1038/323533a0
  * Russakovsky . (2015) russakovsky2015imagenetRussakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S.others  2015\.  Imagenet large scale visual recognition challenge Imagenet large scale visual recognition challenge.  International Journal of Computer Vision1153211–252.  10.1007/s11263-015-0816-y
  * Schapiro . (2013) Schapiro2013NeuralSchapiro, AC., Rogers, TT., Cordova, NI., Turk-Browne, NB. Botvinick, MM.  2013\.  Neural representations of events arise from temporal community structure Neural representations of events arise from temporal community structure.  Nature Neuroscience164486–492.  10.1038/nn.3331
  * Schönemann (1966) Schonemann1966GeneralizedSolutionSchönemann, PH.  1966\.  A generalized solution of the orthogonal procrustes problem A generalized solution of the orthogonal procrustes problem.  Psychometrika3111–10.  10.1007/BF02289451
  * Schütz-Bosbach Prinz (2007) Schutz-Bosbach2007ProspectiveCodingSchütz-Bosbach, S. Prinz, W.  2007\.  Prospective coding in event representation Prospective coding in event representation.  Cognitive Processing8293–102.  10.1007/s10339-007-0167-x
  * Sheldon El-Asmar (2018) Sheldon2018CognitiveToolsSheldon, S. El-Asmar, N.  2018\.  The cognitive tools that support mentally constructing event and scene representations The cognitive tools that support mentally constructing event and scene representations.  Memory266858–868.  10.1080/09658211.2017.1417440
  * Shen . (2020) Shen2020StudyShen, J., Fu, C., Deng, X. Ino, F.  2020\.  A Study on Training Story Generation Models Based on Event Representations A Study on Training Story Generation Models Based on Event Representations.  3rd International Conference on Artificial Intelligence and Big Data, ICAIBD 3rd international conference on artificial intelligence and big data, icaibd ( 210–214).  Institute of Electrical and Electronics Engineers Inc.  10.1109/ICAIBD49809.2020.9137439
  * Shepard (1980) Shepard1980MultidimensionalShepard, RN.  1980\.  Multidimensional scaling, tree-fitting, and clustering Multidimensional scaling, tree-fitting, and clustering.  Science2104468390–398.  10.1126/science.210.4468.390
  * Shepard (1987) Shepard1987TowardShepard, RN.  1987\.  Toward a universal law of generalization for psychological science Toward a universal law of generalization for psychological science.  Science23748201317–1323.  10.1126/science.3629243
  * Shepard Arabie (1979) Shepard1979AdditiveShepard, RN. Arabie, P.  1979\.  Additive clustering: Representation of similarities as combinations of discrete overlapping properties Additive clustering: Representation of similarities as combinations of discrete overlapping properties.  Psychological Review86287.  10.1037/0033-295X.86.2.87
  * Shin DuBrow (2021) Shin2021StructuringShin, YS. DuBrow, S.  2021\.  Structuring memory through inference-based event segmentation Structuring memory through inference-based event segmentation.  Topics in Cognitive Science131106–127.  10.1111/tops.12505
  * Soltani Izquierdo (2019) soltani2019adaptiveSoltani, A. Izquierdo, A.  2019\.  Adaptive learning under expected and unexpected uncertainty Adaptive learning under expected and unexpected uncertainty.  Nature Reviews Neuroscience2010635–644.  10.1038/s41583-019-0180-y
  * Speer . (2007) Speer2007HumanBrainSpeer, NK., Zacks, JM. Reynolds, JR.  2007\.  Human brain activity time-locked to narrative event boundaries Human brain activity time-locked to narrative event boundaries.  Psychological Science185449–455.  10.1111/j.1467-9280.2007.01920.x
  * Spoerer . (2017) spoerer2017recurrentSpoerer, CJ., McClure, P. Kriegeskorte, N.  2017\.  Recurrent convolutional neural networks: a better model of biological object recognition Recurrent convolutional neural networks: a better model of biological object recognition.  Frontiers in Psychology81551.  10.3389/fpsyg.2017.01551
  * Stawarczyk . (2021) Stawarczyk2021EventStawarczyk, D., Bezdek, MA. Zacks, JM.  2021\.  Event representations and predictive processing: The role of the midline default network core Event representations and predictive processing: The role of the midline default network core.  Topics in Cognitive Science131164–186.  10.1111/tops.12450
  * Tarhan . (2020) Tarhan2020SemanticTarhan, L., de Freitas, J., Alvarez, G. Konkle, T.  2020\.  Semantic embeddings of verbal descriptions predict action similarity judgments Semantic embeddings of verbal descriptions predict action similarity judgments.  Journal of Vision20111241.  10.1167/jov.20.11.1241
  * Tarhan Konkle (2018) Tarhan2018HighLevelTarhan, L. Konkle, T.  2018\.  High-Level Features Organize Perceived Action Similarities High-Level Features Organize Perceived Action Similarities.  2018 Conference on Cognitive Computational Neuroscience 2018 conference on cognitive computational neuroscience ( 5–8).  Philadelphia.  10.32470/CCN.2018.1120-0
  * Tripp (2017) tripp2017similaritiesTripp, BP.  2017\.  Similarities and Differences between Stimulus Tuning in the Inferotemporal Visual Cortex and Convolutional Networks Similarities and differences between stimulus tuning in the inferotemporal visual cortex and convolutional networks.  International Joint Conference on Neural Networks (IJCNN, 2017) International joint conference on neural networks (ijcnn, 2017) ( 3551–3560).  10.1109/IJCNN.2017.7966303
  * Troje Basbaum (2008) Troje2008BiologicalMotionTroje, NF. Basbaum, A.  2008\.  Biological motion perception Biological motion perception.  B. Fritzsch (), The senses: A comprehensive reference The senses: A comprehensive reference (2 , 2, 231–238).  CambridgeAcademic Press.
  * Tversky (1977) Tversky1977FeaturesTversky, A.  1977\.  Features of similarity Features of similarity.  Psychological Review844327–352.  10.1037/0033-295X.84.4.327
  * Urban Gates (2021) Urban2019DeepLearningUrban, CJ. Gates, K.  2021\.  Deep learning: A primer for psychologists Deep learning: A primer for psychologists.  Psychological Methods266.  10.1037/met0000374
  * Van der Maaten Hinton (2008) van2008visualizingVan der Maaten, L. Hinton, G.  2008\.  Visualizing data using t-SNE Visualizing data using t-sne.  Journal of Machine Learning Research911.
  * Vinyals . (2016) vinyals2016matchingVinyals, O., Blundell, C., Lillicrap, T., Kavukcuoglu, K. Wierstra, D.  2016\.  Matching Networks for One Shot Learning Matching networks for one shot learning.  D. Lee, M. Sugiyama, U. Luxburg, I. Guyon R. Garnett (), Advances in Neural Information Processing Systems Advances in neural information processing systems ( 29).  Curran Associates, Inc.
  * Virtanen . (2020) 2020SciPy-NMethVirtanen, P., Gommers, R., Oliphant, TE., Haberland, M., Reddy, T., Cournapeau, D.SciPy 1.0 Contributors  2020\.  SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.  Nature Methods17261–272.  10.1038/s41592-019-0686-2
  * Wang . (2017) Wang2017PredictingWang, J., Cherkassky, VL. Just, MA.  2017\.  Predicting the brain activation pattern associated with the propositional content of a sentence: Modeling neural representations of events and states Predicting the brain activation pattern associated with the propositional content of a sentence: Modeling neural representations of events and states.  Human Brain Mapping38104865–4881.  10.1002/hbm.23692
  * Wiese Metzinger (2017) Wiese2017VanillaWiese, W. Metzinger, T.  2017\.  Vanilla PP for philosophers: A primer on predictive processing Vanilla PP for philosophers: A primer on predictive processing.  Philosophy and Predictive Processing1–18.  10.15502/9783958573024
  * Wold . (1987) wold1987principalWold, S., Esbensen, K. Geladi, P.  1987\.  Principal component analysis Principal component analysis.  Chemometrics and Intelligent Laboratory Systems21-337–52.  10.1016/0169-7439(87)80084-9
  * Wu . (2016) wu2016googleWu, Y., Schuster, M., Chen, Z., Le, QV., Norouzi, M., Macherey, W.others  2016\.  Google’s neural machine translation system: Bridging the gap between human and machine translation Google’s neural machine translation system: Bridging the gap between human and machine translation.  arXiv preprint arXiv:1609.08144.
  * Wünstel . (2000) wunstel2000behaviorWünstel, M., Polani, D., Uthmann, T. Perl, J.  2000\.  Behavior classification with self-organizing maps Behavior classification with self-organizing maps.  Robot Soccer World Cup Robot soccer world cup ( 108–118).
  * Yamins DiCarlo (2016) yamins2016usingYamins, DL. DiCarlo, JJ.  2016\.  Using goal-driven deep learning models to understand sensory cortex Using goal-driven deep learning models to understand sensory cortex.  Nature Neuroscience193356–365.  10.1038/nn.4244
  * Yu Dayan (2005) angela2005uncertaintyYu, A. Dayan, P.  2005\.  Uncertainty, neuromodulation, and attention Uncertainty, neuromodulation, and attention.  Neuron464681–692.  10.1016/j.neuron.2005.04.026
  * Zacks (2020) Zacks2020EventPerceptionZacks, JM.  2020\.  Event perception and memory Event perception and memory.  Annual Review of Psychology711165–191.  10.1146/annurev-psych-010419-051101
  * Zacks . (2001) Zacks2001HumanBrainZacks, JM., Braver, TS., Sheridan, MA., Donaldson, DI., Snyder, AZ., Ollinger, JM.Raichle, ME.  2001\.  Human brain activity time-locked to perceptual event boundaries Human brain activity time-locked to perceptual event boundaries.  Nature Neuroscience41651–655.  https://doi.org/10.1038/88486
  * Zacks . (2011) Zacks2011PredictionZacks, JM., Kurby, CA., Eisenberg, ML. Haroutunian, N.  2011\.  Prediction error associated with the perceptual segmentation of naturalistic events Prediction error associated with the perceptual segmentation of naturalistic events.  Journal of Cognitive Neuroscience23124057–4066.  10.1162/jocn_a_00078
  * Zacks . (2007) Zacks2007EventPerceptionZacks, JM., Speer, NK., Swallow, KM., Braver, TS. Reynolds, JR.  2007\.  Event perception: A mind-brain perspective Event perception: A mind-brain perspective.  Psychological Bulletin1332273–293.  10.1037/0033-2909.133.2.273
  * Zacks . (2010) Zacks2010BrainZacks, JM., Speer, NK., Swallow, KM. Maley, CJ.  2010\.  The brain’s cutting-room floor: Segmentation of narrative cinema The brain’s cutting-room floor: Segmentation of narrative cinema.  Frontiers in Human Neuroscience41168.  10.3389/fnhum.2010.00168
  * Zacks Swallow (2007) Zacks2007EventSegmentationZacks, JM. Swallow, KM.  2007\.  Event segmentation Event segmentation.  Current Directions in Psychological Science16280–84.  10.1111/j.1467-8721.2007.00480.x
  * Zhao . (2019) zhao2019pupilZhao, S., Chait, M., Dick, F., Dayan, P., Furukawa, S. Liao, HI.  2019\.  Pupil-linked phasic arousal evoked by violation but not emergence of regularity within rapid sound sequences Pupil-linked phasic arousal evoked by violation but not emergence of regularity within rapid sound sequences.  Nature Communications1011–16.  10.1038/s41467-019-12048-1