# Simulation and time series analysis of responsive active Brownian particles
(rABPs) with memory

Maximilian R. Bailey,a,∗ Fabio Grillo, and Lucio Isaa,∗
aLaboratory for Soft Materials and Interfaces, Department of Materials, ETH
Zürich, Switzerland
E-mail<EMAIL_ADDRESS>
E-mail<EMAIL_ADDRESS>
###### Abstract

To realise the goals of active matter at the micro- and nano-scale, the next
generation of microrobots must be capable of autonomously sensing and
responding to their environment to carry out pre-programmed tasks. Memory
effects are proposed to have a significant effect on the dynamics of
responsive robotic systems, drawing parallels to strategies used in nature
across all length-scales. Inspired by the integral feedback control mechanism
by which E. Coli are proposed to sense their environment, we develop a
numerical responsive active Brownian particle (rABP) model in which the rABPs
continuously react to changes in the physical parameters dictated by their
local environment. The resulting generated time series are then used to
classify and characterise their response, leading to the identification of
conditional heteroscedasticity in their physics. We then train recurrent
neural networks (RNNs) capable of quantitatively describing the responsiveness
of rABPs using their 2-D trajectories. We believe that our proposed strategy
to determine the parameters governing the dynamics of rABPs can be applied to
guide the design of microrobots with physical intelligence encoded during
their fabrication.

##  1 Introduction

Inspired in part by science fiction, there is a widespread interest in
miniaturising systems capable of autonomously responding to their environment
to perform useful functions Bechinger _et al._ (2016); Ebbens (2016); Tsang
_et al._ (2020). Despite the manifold benefits that micro- and nano-robotics
could provide to applications such as biomedicine Li _et al._ (2017); Soto _et
al._ (2020), there are still significant hurdles that must first be overcome
in their design and fabrication. In particular, the “intelligence" of the
current state of art of synthetic microrobots - where the machine is capable
of autonomously sensing the environment and then performing programmed
computations to achieve human objectives - is notably lacking. This, in large
part, is attributable to the difficulty in introducing sophisticated hardware
at the small time-scales of interest for micro- and nanorobotics Sitti (2021).
In fact, most “microrobotic" systems to date typically consist of a
“microswimmers + computer" type configuration, where the tasks of sensing and
adaptation are externalised - e.g. via light microscopy and particle tracking
algorithms coupled to manipulation of external fields Frangipane _et al._
(2018); Arlt _et al._ (2018, 2019); Koumakis _et al._ (2019); Khadka _et al._
(2018); Lavergne _et al._ (2019); Sprenger _et al._ (2020); Muiños-Landin _et
al._ (2020); Fernandez-Rodriguez _et al._ (2020); Wang _et al._ (2023); Chen
_et al._ (2024)

To overcome this “intelligence gap", there is a growing interest in encoding
basic “physical intelligence" Sitti (2021) into synthetic active materials
during their fabrication. Specifically, stimuli-responsive materials such as
thermoresponsive poly-NIPAM (pNIPAM) can be utilised to provide microswimmers
with the ability to perceive their environment and respond in a programmed
manner Alvarez _et al._ (2021); van Kesteren _et al._ (2023). An important
design parameter in such systems is the presence of “memory" Sitti (2021),
which has been shown to influence the complexity of behaviours which can be
achieved with responsive active systems Mijalkov _et al._ (2016); Leyman _et
al._ (2018); Muiños-Landin _et al._ (2020); Sprenger _et al._ (2022)

In particular, one can draw inspiration from nature, where living systems at
various length-scales, including in the micro- and nano-regime, have developed
much of the computational and physical machinery desirable for autonomous
applications Barkai and Leibler (1997); Yi _et al._ (2000); Vladimirov and
Sourjik (2009); Berdahl _et al._ (2013); Aquino _et al._ (2014); Lozano and
Bechinger (2019); Gosztolai and Barahona (2020). Memory plays an important
role in the ability of micro-organisms to perform “useful" tasks such as
painting Frangipane _et al._ (2018); Arlt _et al._ (2018); Koumakis _et al._
(2019), and memory has more generally been proposed to be a fundamental aspect
of molecular machinery, enhancing the ability of bacteria to sense Berg and
Purcell (1977); Aquino _et al._ (2014) and chemotaxis Berg (1975); Yi _et al._
(2000); Skoge _et al._ (2014); Gosztolai and Barahona (2020). To date, most
studies with synthetic/numerical systems have relied on discrete time delays
to encode memory Fernandez-Rodriguez _et al._ (2020); Muiños-Landin _et al._
(2020); Holubec _et al._ (2021); Wang _et al._ (2023), which, while effective,
relies on a feedback system controlled by an external computer. However,
encoding such capacity via physical intelligence is highly challenging.
Nevertheless, the manner by which micro-organisms encode memory, for example
via integral control type systems with an effectively delayed response Yi _et
al._ (2000); Aquino _et al._ (2014); Gosztolai and Barahona (2020), could
provide instructive for the design of responsive microrobots.

In the current societal context, one cannot discuss the role of intelligence
without also referencing the looming specter of artificial intelligence (AI).
With respect to micro- and nanorobotic systems, AI has found application in
particle tracking Midtvedt _et al._ (2021, 2022); Bailey _et al._ (2022);
Pineda _et al._ (2023), estimating parameters of their physics Bo _et al._
(2019); Argun _et al._ (2020, 2021); Gentili and Volpe (2021); Muñoz-Gil _et
al._ (2021); Ruiz-Garcia _et al._ (2022), developing swimming strategies Tsang
_et al._ (2020); Qin _et al._ (2023), and determining optimal navigation paths
Muiños-Landin _et al._ (2020); Nasiri and Liebchen (2022); Mo _et al._ (2023);
Tovey _et al._ (2023). In particular, “deep learning" strategies excel at
extracting underlying representations of complex data - otherwise intractable
using standard statistical tools - due to their intrinsic ability to model
hierarchies and non-linearities present within the datasets. As dynamical
systems, studying micro- and nanorobots often involves the analysis of their
generated time series data to obtain a greater understanding of their physical
properties. Of the various deep-learning architectures, recurrent neural
networks (RNNs) are often the best suited to this task due to their ability to
capture sequential dependencies and thus account for the “history" or "memory"
of the data Lipton _et al._ (2015). This property has been exploited in
several recent studies to determine the governing physics of a range of micro-
scale systems with significant success Bo _et al._ (2019); Argun _et al._
(2020); Gentili and Volpe (2021); Muñoz-Gil _et al._ (2021).

To assist the design of micro- and nanorobotic systems which can adapt to the
complex environment through which they locomote, we develop a numerical scheme
of responsive active Brownian particles (rABPs) whose continuous response to
their 2-D environment intrinsically imparts memory of their internal system
onto their dynamics. Specifically, we build upon a previous numerical model
where the rABP responds via an aphysical zero-order-hold (ZOH) delay to a
checkerboard environment Fernandez-Rodriguez _et al._ (2020), to motion
through a double sinusoid, where the response of the rABP is governed by an
integral-control type feedback inspired by the mechanisms which govern
chemotaxis in bacterial systems Barkai and Leibler (1997); Yi _et al._ (2000).
Moving from a discrete feedback in a discretised environment to a continuous
response in a differentiable environment not only represents a more realistic
system, but also generates time series data which can be used for statistical
analysis and to train RNNs which extract the physics governing the response of
our rABPs to their local environment. Our numerical simulation and time series
analysis therefore provides a framework by which RNNs can be trained to
extract physical parameters from trajectories of real experimental systems, in
turn providing insights into the responsiveness and suitability of different
materials to impart physical intelligence onto active materials depending on
the application desired.

##  2 Numerical method

Our numerical model builds upon the previous system developed in Fernandez-
Rodriguez _et al._ (2020). In brief, we simulate the dynamics of the
responsive active Brownian particles (rABPs) by solving the following
equations of motion:

$\displaystyle m\ddot{x}=$
$\displaystyle\,f_{x}(\bm{\mathrm{r}},\tau,\theta)-\gamma_{T}\dot{x}+\sqrt{2k_{B}T\gamma_{T}}\eta_{x}(t)$
$\displaystyle m\ddot{y}=$
$\displaystyle\,f_{y}(\bm{\mathrm{r}},\tau,\theta)-\gamma_{T}\dot{y}+\sqrt{2k_{B}T\gamma_{T}}\eta_{y}(t)$
$\displaystyle I\ddot{\theta}=$
$\displaystyle\,\gamma_{R}(\bm{\mathrm{r}},\tau)\dot{\theta}+\sqrt{2k_{B}T\gamma_{R}(\bm{\mathrm{r}},\tau)}\eta_{\theta}(t)$
(1)

where $m$ and $I$ are the mass and the moment of inertia of the rABPs (density
$\rho=2500$ kg/m3 and $R=0.9e^{-6}\mu$m, for correspondence to typical
experimental realisation of active colloids).
$f_{x}(\bm{\mathrm{r}},\tau,\theta)$ and $f_{x}(\bm{\mathrm{r}},\tau,\theta)$
are the x and y components of the active force acting on the rABPs, but in
contrast to Fernandez-Rodriguez _et al._ (2020), also has a dependence on the
response variable $\tau$ and the xy- coordinates of the particles due to the
positional dependence of its velocity $V(\bm{\mathrm{r}},\tau)$.
$f_{x}(\bm{\mathrm{r}},\tau,\theta)$ and $f_{x}(\bm{\mathrm{r}},\tau,\theta)$
are thus given by $V(\bm{\mathrm{r}},\tau)\text{cos($\theta$)}\gamma_{T}$ and
$V(\bm{\mathrm{r}},\tau)\text{sin($\theta$)}\gamma_{T}$ respectively, where
$\gamma_{T}$ is the friction coefficient for translational motion.
$\gamma_{R}(\bm{\mathrm{r}},\tau)$ is the state-dependent friction coefficient
for the rABPs’ rotational dynamics, while $\eta_{x}(t)$, $\eta_{y}(t)$, and
$\eta_{\theta}(t)$ are uncorrelated random numbers satisfying the conditions:

$\displaystyle\langle\eta_{x}\rangle=$
$\displaystyle\,\langle\eta_{y}\rangle=\,\langle\eta_{\theta}\rangle=0$
$\displaystyle\langle\eta_{x}^{2}\rangle=$
$\displaystyle\,\langle\eta_{y}^{2}\rangle=\,\langle\eta_{\theta}^{2}\rangle=1$
(2)

As in Fernandez-Rodriguez _et al._ (2020), we solve Eq. 1 in the underdamped
limit for faster convergence, using a Verlet-type integration scheme proposed
by Gronbech-Jensen and Farago applying the It$\hat{\text{o}}$ convention
Grønbech-Jensen and Farago (2013).

The value of the position-dependent rotational diffusion coefficient $D_{R}$
($D_{R}(\bm{\mathrm{r}})=k_{B}T/\gamma_{R}(\bm{\mathrm{r}})$) and its velocity
$V$ are set by the rABPs spatial position
$\bm{\mathrm{r}}=[x(\text{t}),y(\text{t})]$, via the following double
sinusoidal mapping:

$\displaystyle F(\bm{\mathrm{r}})=$
$\displaystyle\frac{F_{min}-F_{max}}{2}\left[1+\text{sin}\left(\frac{\pi
x}{L}+\frac{\pi}{2}\right)\text{sin}\left(\frac{\pi
y}{L}+\frac{\pi}{2}\right)\right]+F_{max}$ (3)

where $F$ is the general form of the physical quantity determined by the
spatially varying environment (e.g. $D_{R}(\bm{\mathrm{r}})$,
$V(\bm{\mathrm{r}})$ \- see Figure 1), and $F_{min}$ and $F_{max}$ determine
the range of values that the mapping can take.

The value input into the rABP’s governing equations of motion Eq. 1 at time t,
$\hat{F}_{t=\text{t}}$, is determined by both its position $\bm{\mathrm{r}}$
via $F_{t=\text{t}}(\bm{\mathrm{r}})$, and by its “memory" of its previous
values $\hat{F}_{t=\text{t}-\Delta\text{t}}$. The importance of the rABPs
history to its physics is thus set by $\tau$, which determines the
responsiveness of the moving rABP to $F(\bm{\mathrm{r}})$ via the response
function $\frac{d\hat{F}}{dt}$ (see Figure 1 and the main text for more
details). As with the environment, the response function $\frac{d\hat{F}}{dt}$
can take any form, however we have studied the simplest form of an integral-
control feedback system, given by:

$\displaystyle\frac{d\hat{F}}{dt}=$
$\displaystyle-\frac{1}{\tau}\left(\hat{F}-F(\bm{\mathrm{r}})\right)$ (4)

where $\tau$ determines the rate at which the current value of the rABP,
$\hat{F}$, is able to “match" the value set by its spatial position
$F(\bm{\mathrm{r}})$. The time-scale of the simulations is set via $L/V$, the
simulation time taken for a ballistically moving particle to travel the
distance of one half period of the sinusoidal mapping (for example, $L/V=10$
represents 10 simulated seconds). The simulation is incremented in 0.001
$\Delta t$ time steps, excepting for $\tau=1e-3$ \- where $\Delta t=$ 0.0001
s, while the trajectories are sub-sampled as in experiment.

Figure 1: Schematic of the numerical simulations of a responsive active
Brownian particle (rABP) moving in a spatially fluctuating environment of some
variable $F(\overrightarrow{R})$. The particle is initialised at the origin at
$t=0$ (red disc), and moves during a simulation time-step $\Delta t$ to it’s
new position $\overrightarrow{r}$ (blue disc). At $t=0$, the particle’s
dynamics $\hat{F}_{t=0}$ are determined completely by its position
$F(\overrightarrow{0})$. However, the update of the internal particle dynamics
at $t=\Delta t$, $\hat{F}_{t=\Delta t}$, is non-Markovian and governed by the
response function $d\hat{F}/dt$.

##  3 Results & Discussion

###  3.1 Localisation of a responsive active Brownian particle (rABP) in a
double sinusoidal mapping of $D_{R}$

We begin our study by evaluating the localisation of 1000 non-interacting
rABPs moving through an environment characterised by a spatial variation in
the set rotational diffusion coefficient, $D_{R}(\bm{\mathrm{r}})$, where the
mapping is given by Eq. 3. As in Fernandez-Rodriguez _et al._ (2020), we vary
the values of $D_{R}$ via 3 orders of magnitude from $D_{R,L}\approx 0.01$ to
$D_{R,H}\approx 10$ (see Figure 2a). However, here we implement a continuous
integral-control type feedback to govern the response of the rABP to its
environment, rather than the discrete time-feedback loop previously studied.
In this case, the value of $\tau$ governs the rate $d\hat{D}_{R}/dt$ of the
response to changes in the external environment, rather than defining a
discrete-time after which the value of $\hat{D}_{R}$ is updated. With respect
to Eq. 4, the value $\hat{D}_{R}-D_{R}(\bm{\mathrm{r}})$ can be thought as the
error between the system target state $D_{R}(\bm{\mathrm{r}})$ and its current
state $\hat{D}_{R}$, similar to integral feedback control, a response observed
in a range of biological systems including those governing bacterial
chemotaxis Barkai and Leibler (1997); Yi _et al._ (2000).

We find that rABPs continuously responding to a differentiable environment
will display qualitatively similar localisation behaviour as that previously
reported for such particles with a discrete-time response to a checkerboard
pattern Fernandez-Rodriguez _et al._ (2020). Specifically, for different $L/V$
landscapes, where $V$ here is a constant at $5\mu$ms-1, we note the existence
of an optimal response time $\tau$ for which the localisation of rABPs in
$D_{R,H}$ regions, $\eta=N_{H}/(N_{L}+N_{H})$ \- where $N_{L}$ and $N_{H}$ are
the number of rABPs in the $D_{R,L}$ and $D_{R,H}$ regions respectively - is
maximised (see Figure 2b). Very low values of $\tau$ correspond to a near-
instantaneous update of $\hat{D}_{R}$, and therefore no localisation is
observed as previously described in Fernandez-Rodriguez _et al._ (2020).
Likewise, at very high values of $\tau$ the rABP is effectively “blind" to the
spatial variations in $D_{R}(\bm{\mathrm{r}})$, and the distribution reflects
the original initialisation of the particles in their environment.

Figure 2: Localisation of rABPs in a spatially varying environment where
$D_{R}(\bm{\mathrm{r}})$ varies between 0.01 and 10 according to a double
sinusoidal mapping (in x and y). a) Schematic representation of an rABP (blue
sphere) moving in the spatially varying $D_{R}(\bm{\mathrm{r}})$, ranging from
values of $D_{R,min}$ to $D_{R,max}$ with period $2L$. b) Varying the extent
of localisation in high $D_{R}$ regions ($D_{R,H}$) with increasing response
variable $\tau$ for a given environment time-scale of fluctuation $L/V=10$. At
intermediate response times $\tau/(L/\langle V\rangle)=0.3$, localisation is
maximised in $D_{R,H}$ regions, as illustrated by a heat map of particle
positions. For small and large response times, the extent of localisation is
significantly reduced, indicating the existence of an optimal $\tau$. c)
Extent of particle localisation $\eta$ in $D_{R,H}$ regions, defined as
$D_{R}(\bm{\mathrm{r}})\geq(D_{R,max}+D_{R,min})/2$, as a function of response
time for different system time-scales $L/V$. d) The response time at which
localisation is optimised, $\tau_{\eta=\text{argmax}(\eta)}$, increases
linearly with the system time-scale $L/V$, while the maximum localisation
$\eta_{max}$ begins to saturate around $L/V=10$.

The reduction in argmax($\eta$) compared to Fernandez-Rodriguez _et al._
(2020) is largely attributed to the smooth gradient in
$D_{R}(\bm{\mathrm{r}})$, which, coupled to the gradual response of the rABPs
to their environment (rather than an instantaneous update after a discrete
delay-time), provides the particles with a greater probability of escaping a
high $D_{R}$ ($D_{R,H}$) region into a low $D_{R}$ one ($D_{R,L}$). More
specifically, particles cannot ballistically enter $D_{R,H}$ regions in the
absence of a ZOH-delay type feedback response Fernandez-Rodriguez _et al._
(2020), as their $\hat{D}_{R}$ value will continuously rise, increasing the
likelihood that they are trapped diffusively near the edges, or are able to
orient away with intermediate $\hat{D}_{R}$ values and escape into $D_{R,L}$
regions. This is also reflected in the heat maps of the particle position (see
Figure 2c) for low and intermediate $\tau/(L/V)$, where we observe the absence
of particles in the central parts of the $D_{R,H}$ regions, indicating the
inability of particles to penetrate deeply within the high $D_{R}$ regions.

Notably, as $\tau$ increases further beyond the optimal localisation time
(e.g. $\tau/\frac{L}{V}>0.3$), this distinction is blurred as the rABPs no
longer update sufficiently quickly to sense the spatial variations in
$D_{R}(\bm{\mathrm{r}})$ and thus are able to travel further into the
$D_{R,H}$ regions before re-orienting (see Figure 2c, $\tau/(L/V)\geq 3$). As
a further result of the continuous update, and unlike the system studied in
Fernandez-Rodriguez _et al._ (2020), the maximum value of $\eta$ attained is
not a linear function of $L/V$ but instead saturates around $L/V=10$ (see
Figure 2d). This again arises due to the continuous mapping in $D_{R}$,
preventing the distinct delineation between “diffusive" ($D_{R,H}$) and
“ballistic" ($D_{R,L}$) regions at steady state, as previously argued in
Fernandez-Rodriguez _et al._ (2020). Nevertheless, we note that the lag time
at which $\eta$ is maximised, $\tau_{\eta=\text{argmax}(\eta)}$, is linear
with $L/V$, which we attribute to the need for rABPs to be less responsive to
their environment to ensure that they penetrate sufficiently far into the
$D_{R,H}$ domain, and thus minimise the probability of re-orienting and
escaping into a $D_{R,L}$ region.

###  3.2 Statistical analysis of rABP time series with auto-regressive models

As a consequence of studying rABPs which now continuously respond to a
differentiable spatial mapping of their physical parameters, we now obtain two
additional time series’, 1) the particle’s “set" value by the environment,
$F(\bm{\mathrm{r}})$ (here $D_{R}(\bm{\mathrm{r}})$), and 2) the value
experienced by the particle as a function of its memory,
$\hat{F}_{t=\text{t}}$ (here $\hat{D}_{R}$). We see clearly the effect of
increasing $\tau$ on the ability of the particle to respond to its environment
in Figure 3a, which progressively smooths and delays the response
$\hat{D}_{R}$ to the environment value $D_{R}(\bm{\mathrm{r}})$, drawing
parallels to an integral-feedback controller Barkai and Leibler (1997); Yi _et
al._ (2000).

To study the generated time series, we first turn to a frequently invoked,
parsimonious family of parametric models used to describe stationary time
series and make short-term forecasting predictions, namely derivatives of the
auto-regressive-integrated-moving-average (ARIMA) process Makridakis and Hibon
(1997). Such models approximate a time series as a combination of its $p$
previous values and/or $q$ error values ($p$ AR and $q$ MA terms respectively,
see Eq. 5), while the $I$ refers to the number of differencing steps required
to obtain a stationary process. Specifically, we concentrate on the time
series of $D_{R}(\bm{\mathrm{r}})$ experienced by the rABP for
$\tau/(L/V)=0.3$, ($\hat{D}_{R,\tau=3}$), previously identified as the value
for which $\eta$ is near its maximum.

$\displaystyle X_{t}=$
$\displaystyle\,\epsilon_{t}+\sum_{i=1}^{p}\phi_{i}X_{t-i}+\sum_{j=1}^{q}\theta_{j}\epsilon_{t-j}$
(5)

where $\epsilon_{t}$ represents the white-noise error (or innovation) at time
$t$, $\phi_{i}$ denotes the $i^{th}$ auto-regressive parameter relating the
value of time series at $t$, $X_{t}$, to its previous value $X_{t-i}$, and
$\theta_{j}$ is the $j^{th}$ fitting parameter of the moving-average (MA)
process describing the dependence of the current value $X_{t}$ on its error at
time $t-j$, $\epsilon_{t-j}$.

Preliminary investigations showed that standard ARIMA models were not capable
of adequately capturing the dynamics of $\hat{D}_{R}$, with the best fits
obtained using a ARIMA($p$=5, $I$=2, $q$=0) model for $\hat{D}_{R,\tau=3}$
displaying relatively weak time dependency on its previous values.
Importantly, the squared standardised residuals of the model showed
significant signs of autocorrelation (see Figure 3d), a key marker for
conditional heterocedasticity Tse (2002). Conditional heteroscedasticity
refers to situations where the non-constant variance observed is itself a
function of previous variance terms. Common examples are to be found in
econometrics, e.g. the case of volatility clustering, with a common example
being that shocks in a share price are more likely to be followed by further
volatility (high variance). The most widespread approach to treat conditional
heteroscedasticity in a time series is to use the autoregressive-conditional-
heteroscedasticity (ARCH) family of models Engle (1982). From comparing the
Akaike information criterion (AIC) of different generalised-ARCH (GARCH)
models, the exponential-GARCH (EGARCH) model was found to provide the best
results, and removed the presence of autocorrelation in the squared
standardised residuals (see Figure 3e). The EGARCH model extends GARCH
processes by allowing for asymmetric effects in the volatility of a time
series, a feature commonly found in the stock market for which the model was
proposed (shares are expected to be more volatile after bad news than good
news Nelson (1991)). In the context of our rABPs, the classification of
$\hat{D}_{R}$ as an ARIMA-EGARCH process can be rationalised by considering
the asymmetric variance with which a ballistic (low $D_{R}$) and diffusive
(high $D_{R}$) will change their spatial co-ordinates and thus internal
$\hat{D}_{R}$ value (large $\Delta D_{R}(\bm{\mathrm{r}})$, low $\Delta
D_{R}(\bm{\mathrm{r}})$ respectively). Although providing some useful insight
into the dynamics of an rABP’s response, such models are nevertheless
inadequate for a more quantitative description of its motion-determining
parameters.

Figure 3: The continuous response of the simulated rABP - contrasting with the
discrete zero-order-hold (ZOH) delay investigated in Fernandez-Rodriguez _et
al._ (2020) \- allows the characterisation of its dynamics using time series
analysis techniques. a-c) As $\tau$ increases, the ability of the particle to
update it’s internal $\hat{D}_{R}$ term to follow $D_{R}(\bm{\mathrm{r}})$
decreases, and there is a smoothing and delay effect reminiscent of an
integral feedback controller. d) The auto-correlation function (ACF) of the
normalised squared-residuals after fitting an auto-regressive ARIMA(5,2,0)
model to the time series $\hat{D}_{R,\tau=3}$ indicates the presence of
conditional heteroscedasticity in the time series of $\hat{D}_{R,\tau=3}$.
Inset: the time series of the normalised residuals. e) Both the Akaike and
Bayesian information criteria (AIC, BIC respectively) indicate that
$\hat{D}_{R,\tau=3}$ is best modelled as an ARIMA(5,2,0)-EGARCH(1,1) process
amongst the conditional heteroscedasticity models tested. The previously
detected auto-correlation of the normalised squared-residuals when fitting
only an ARIMA model also disappears. Inset: the time series of the normalised
residuals. Note the absence of large fluctuations, contrasting to the case
where the conditional heteroscedasticity was not accounted for. Here,
$L/V=10.$

###  3.3 Characterising the response of rABPs with recurrent neural networks
(RNNs)

####  3.3.1 Predicting $\tau$ for rABPS moving in a double sinusoidal mapping
of $D_{R}$

In the last few years, recurrent neural networks (RNNs) of various forms have
demonstrated significant successes in the quantification of anomalously
diffusing processes Bo _et al._ (2019); Argun _et al._ (2021); Gentili and
Volpe (2021); Muñoz-Gil _et al._ (2021). We are therefore motivated by the
question of whether such networks can also be utilised to extract meaningful
information on the underlying physics governing the motion of our rABPs.
Specifically, the responsiveness of the rABPs to their spatially varying
environment is determined by $\tau$ (see Eq. 4), which in turn dictates the
importance of their history and thus their overall dynamics (see Figure 2). We
note that in Figures 3a-c, a clear increase in the smoothing-and-delaying of
$\hat{D}_{R}$ can be observed as a function of $\tau$. We thus hypothesise
that it is possible to extract information on $\tau$ by examining the
differences between $F_{t=\text{t}}(\bm{\mathrm{r}})$ and
$\hat{F}_{t=\text{t}}$ (beginning with the case of $D_{R}$). Specifically, we
investigate the ability of gated-recurrent-units (GRUs) Cho _et al._ (2014);
Chung _et al._ (2014) \- a specific type of RNNs which performs especially
well at capturing short-term dependencies in time series (as identified with
the ARIMA-GARCH family of models) - to estimate $\tau$ from the available time
series.

To generate sufficient training data for the recurrent neural networks, we
simulate 10 rABPs for 748 equally log-spaced $\tau$ values between 0.01 and 10
(i.e. spanning 3 orders of magnitude) for $L/V=10$, noting that the number of
trajectories used is orders of magnitude less than that studied in previous
works Bo _et al._ (2019). We begin by sub-sampling our simulated trajectories
every 50 time steps for 1200 data points, corresponding to an video
acquisition rate of 20 frames-per-second (FPS) for a 60s active matter
experiment. The raw input data of particle $i$ consists of the time series
$D_{R,i}(\bm{\mathrm{r}})$ and $\hat{D}_{R,i}$, while the label for regression
is $\tau_{i}$, which governs the response of the rABP to its environment. As
the values for $D_{R}$ and $\tau$ take positive values spanning 3 orders of
magnitude, we take the common logarithm (log10) before differencing the time
series inputs to remove underlying trends, and then re-scale the values to
take near integer values. We then reshape the trajectories from $[N\times
2]\rightarrow[N/2\times 4]$, i.e. treating 2 sequential time steps as a single
data point, which we found to improve the model fitting Argun _et al._ (2020).
To construct and train our deep learning architecture, we make use of the
keras API Chollet _et al._ (2015), which has a range of in-built features
suitable for network training. Our sequential network consists of 2 pre-
processing layers of multilayer perceptrons (MLPs), which feed into 8 GRU
layers, both of which have 32 hidden units, before a final regression layer
with a single output to estimate $\hat{\tau}$. We use the in-built Adam
optimiser with a learning rate of 0.001, noting that we performed minimal
hyper-parameter tuning (as we were not overly interested with obtaining the
smallest loss possible). We trained our network with a 70:15:15
train:validation:test split, using the mean-squared error on the log-
transformed labels as the performance metric. To integrate our pre-processing
steps with the data input into the network, we used a Deeptrack 2.0 Midtvedt
_et al._ (2021) generator function to continuously sample and feed the
training data into our network architecture batch-wise.

Given the relative simplicity of our pre-processing steps and network
architecture, as well as the little hyperparameter tuning performed, the
ability of our network to predict $\tau$ from the provided time series of
$D_{R}(\bm{\mathrm{r}})$ and $\hat{D}_{R}$ is impressive (see Figure 4).
Across the range of $\tau$ values investigated, the network estimates
$\hat{\tau}$ mostly within 10% of the true value (see Figure 4c), with minimal
structure in the residuals. We note that for the lowest and highest values of
$\tau$, the network struggles more, which reflects the almost instantaneous
changes in $\hat{D}_{R}$ for very small $\tau$ (where the two time series
almost completely overlap), and the nearly imperceptible differences in
$\hat{D}_{R}$ for large $\tau$. In particular, the performance at the lowest
$\tau$ values is particularly surprising given the sampling rate which is in
fact larger than the smallest values of $\tau$ studied ($\Delta t=0.05$s,
while $\tau=0.01$s).

Figure 4: Fitting the response value $\tau_{i}$ of different rABPs from their
true rotational diffusion $\hat{D}_{R,i}$ and the value set by their position
$D_{R,i}(\bm{\mathrm{r}})$ using recurrent neural networks (RNNs) where the
system time-scale is given by $L/\langle V\rangle=10$s. a) The values of
$\tau$ span 3 orders of magnitude, and we therefore apply a $log_{10}$
transform to the labels before fitting. b) Reverting the fitted values
$\hat{\tau}$ and $\tau$ to their original scale. c) We observe little
structure in the normalised residuals of the fits, noting that at the highest
and lowest $\tau$ values studied, the network performs worse. 748 different
$\tau$ values with 10 different particle realisations for each $\tau$ were
simulated to obtain the data for model training and evaluation.

####  3.3.2 Predicting $\tau$ for rABPS moving in a double sinusoidal mapping
of $V$

The fitting procedure outlined here relies on both $D_{R,i}(\bm{\mathrm{r}})$
and $\hat{D}_{R,i}$ to be accessible parameters to predict $\tau$ for the
$i^{\text{th}}$ rABP. $D_{R,i}(\bm{\mathrm{r}})$ can be readily extracted from
experiment if the spatial mapping of the spatially varying variable is known,
e.g. due to its external imposition on the particles Fernandez-Rodriguez _et
al._ (2020). In contrast, $\hat{D}_{R,i}$ cannot be easily determined from
particle coordinates as for the environmental set-point. This lies in the
difficulty of obtaining accurate estimates of $D_{R}$ from ABP dynamics, which
is typically extracted from their mean-squared-displacement (MSD) Bailey _et
al._ (2022). In particular, the problem of poor statistics and small $\Delta
D_{R}$ is likely to be a significant challenge when attempting to determine a
dynamic $\hat{D}_{R}$ from particle trajectories. A notable exception may be
the case of the “raspberry" active particles studied in Niggel _et al._
(2023), where the rotations can be explicitly determined, and therefore the
value of $\hat{D}_{R}$ could be evaluated at shorter time-scales.
Nevertheless, it remains questionable whether the approach outlined above
using RNNs to estimate $\tau$ from $D_{R,i}(\bm{\mathrm{r}})$ and
$\hat{D}_{R,i}$ would find broader application in systems currently studied.

Nevertheless, we do expect that the approach outlined here could be used for
the spatial mapping of certain particle properties. For example, the
translational velocity of active particles can be readily extracted from their
trajectories, without requiring the fitting of the full ABP MSD equation.
Indeed, most experimental systems with an externally imposed variation in the
dynamics of active materials typically focus on modulating their speed, often
via light patterns Lozano _et al._ (2016); Lozano and Bechinger (2019);
Frangipane _et al._ (2018); Arlt _et al._ (2018, 2019); Koumakis _et al._
(2019); Caprini _et al._ (2022). Motivated by these considerations, we
therefore evaluate the ability of our numerical model to capture the dynamics
of rABPs locomoting through a double sinusoidal mapping of $V$, and the
ability of RNNs to extract details of their underlying response. For
physically relevant values, we modulate the velocity of the rABPs from
$V_{min}=1\mu$ms-1 to $V_{max}=10\mu$ms-1, corresponding to active Péclet
numbers in the range [7.6, 75.8], where $Pe=V/\sqrt{D_{T}D_{R}}$.

Figure 5: Implementing velocity $V$ as the response parameter of the rABPs,
which varies between $V_{min}=1\mu ms^{-1}$ and $V_{max}=10\mu$ms-1. a)
Particle localisation $\eta$ in low $V(\bm{\mathrm{r}})$ ($V_{L}$) regions,
defined as $V(\bm{\mathrm{r}})<(V_{max}+V_{min})/2$, as a function of response
time for different system time-scales $L/\langle V\rangle$. b) Fitting the
response time $\tau$ from their true particle velocity $\hat{V}$ and the value
set by their position $V(\bm{\mathrm{r}})$ using the gated recurrent unit
(GRU) network architecture developed for $D_{R}$, where the system time-scale
is again given by $L/\langle V\rangle=10$s. c) As in Figure 4c, we observe
minimal structure in the normalised residuals of the fits, with poorer network
performance for the lowest $\tau$ values. 748 different $\tau$ values with 10
different particle realisations for each $\tau$ were simulated to obtain the
data for model training and evaluation.

Unlike the systems with a spatial mapping of $D_{R}$, we find that even with
instantaneous localisation ($\tau\rightarrow 0$) we observe some degree of
localisation of rABPs in low velocity regions $V_{L}<\langle V\rangle$ (see
Figure 5a). This can be rationalised by considering the relation
$V_{L}\rho_{L}=V_{H}\rho_{H}=C$ at steady state, where $C$ is some constant, a
property of out-of-equilibrium systems (forbidden to those at thermal
equilibrium) theoretically predicted for a range of systems Schnitzer (1993);
Tailleur and Cates (2008); Stenhammar _et al._ (2016); Caprini _et al._ (2022)
and demonstrated with swimming bacteria by Arlt et al. Arlt _et al._ (2019).
Considering the mean velocities in the $V_{L}$ and $V_{H}$ regions (3.25
$\mu$ms-1 and 7.75 $\mu$ms-1 respectively), from this relation we expect a
localisation $\eta\approx 70\%$ which is indeed what we observe (see Figure
5a). Otherwise, we again note a similar behaviour as observed for $D_{R}$,
with a non-monotonic relationship between $\tau$ and $\eta$ as a function of
$L/\langle V\rangle$. In particular, at high $\tau$ we once more recover an
equal distribution in the $V_{L}$ and $V_{H}$ regions as the particles are
unable to respond to changes in the environment. Interestingly, we note that
non-gaussian localisation effects only become apparent for $\tau>1$, which we
expect to be a result of the much shallower gradient in $V$ compared to the
case of $D_{R}$ ($D_{R,max}/D_{R,min}=100\cdot V_{max}/V_{min}$), and the
resulting need for less-responsive particles for localisation effects to
emerge.

We then evaluate the ability of our previously developed RNN to estimate
$\hat{\tau_{i}}$ from $\hat{V_{i}}$ and $V_{i}(\bm{\mathrm{r}})$ for
$L/\langle V\rangle=10$. We retain the architecture and pre-processing steps
previously developed for analysing $D_{R}$ time series, but no longer take the
common logarithm of the raw input data, as $V$ now only varies over 1 order of
magnitude. We find that the RNN performs equally well, if not better, for
particles moving in a landscape of $V.$ In particular, we see that the network
appears to predict higher values of $\tau$ better than for $D_{R}$, which may
result from the more direct influence that velocity has on the rABP dynamics
and the resulting variations in $V_{i}(\bm{\mathrm{r}})$. Nevertheless, we
note here that, like for $D_{R}$, in this approach we require direct access to
$\hat{V_{i}}$, the “true" particle velocity. In the absence of noise, i.e.
where Brownian motion due to thermal fluctuations are negligible (i.e.
$D_{T},D_{R}\rightarrow 0$), this value could indeed be directly extracted
from the instantaneous displacements of the particle,
$\hat{V_{i}}=V_{0,i}=\sqrt{\Delta x_{i}^{2}+\Delta y_{i}^{2}}/\Delta t$, where
$\Delta x_{i}$ and $\Delta y_{i}$ denote the motion of the $i^{th}$ particle
in x and y respectively over time-step $\Delta t$. Unfortunately, in
experiment one is typically plagued by the infernal nuisance that is noise
Muiños-Landin _et al._ (2020); Tovey _et al._ (2023), and it is at the very
least necessary to account for Brownian effects on the particle dynamics. We
therefore now investigate whether it is possible to extract $\tau$ from
experimentally accessible parameters, i.e. from the trajectory of a rABP.

As before, we investigate the time series of rABPs generated for $L/\langle
V\rangle=10$. However, instead of using $\hat{V_{i}}$ directly from the
simulations, we instead now attempt to extract an approximation from the
instantaneous velocity of the particles, $V_{0,i}=\sqrt{\Delta
x_{i}^{2}+\Delta y_{i}^{2}}/\Delta t$. We note here that the determination of
$V_{0,i}$ over two time steps requires that we do not use the final data point
of $V_{i}(\bm{\mathrm{r}})$. Immediately, the problems associated with working
with experimental trajectories subject to noise and sampling effects become
apparent (see Figure 6a). The values determined directly, $V_{0,raw}$, are too
noisy to provide a faithful approximation of the underlying velocity
$\hat{V_{i}}$, preventing model training. More specifically, the large
differences between subsequent measurements of $V_{0,raw}$ is exacerbated by
the differencing step required in pre-processing, creating a uninformative
time series from which the RNN must learn. The issues associated with noise
can be somewhat mitigated by increasing the sampling time, $\Delta t$, so as
to increase the importance of the persistent motion $V$ relative to the
thermal diffusion ($V\times\sqrt{D_{T}/\Delta t}>1$), albeit at the cost of
the time resolution. For $V_{min}=1\mu$s-1 and $D_{T}=0.137\mu$m2s-1, as in
the simulations, we therefore set $\Delta t=0.1$s to ensure that the we study
time-scales where the ballistic motion of the particles outweighs the
contribution of diffusion. As we still require 1200 data points for analysis
with the recurrent network, this doubles the effective experimental time. An
equivalent active matter experiment would therefore require a trajectory of 2
minutes at 10 FPS, which is reasonable.

Figure 6: Investigating the ability of our GRU architecture to extract $\tau$
from particle trajectories, where the system time-scale is given by $L/\langle
V\rangle=10$. a) Translational and rotational diffusion, as well as sub-
sampling the trajectories, introduce noise into the instantaneous velocity
extracted from trajectories, $V_{0,raw}$ (here sub-sampled every 0.1 s). A
Butterworth filter is applied to extract the de-noised signal $V_{0,smooth}$,
which better approximates the underlying generating function
$V(\bm{\mathrm{r}})$. b) With the denoised time series $V_{0,smooth}$, a
binary classifier can be constructed using the same GRU architecture which is
able to classify with high accuracy between $\tau\geq 1$ and $\tau<1$. The
classifier was trained and evaluated on simulated data with 748 different
$\tau$ values with 10 different particle realisations for each $\tau$. c)
Fitting the response time $\tau$ from experimentally accessible data, for
meaningful values of $\tau$ given $(V_{max}/V_{min})/(L/\langle V\rangle)=1$
($\tau\geq 1$). For smaller $\tau$ values, the response is effectively
instantaneous given the small gradient in $V$. d) For $1\leq\tau\leq 10$, the
normalised residuals obtained have minimal structure. We note that the effect
of experimental “noise" widens the variance in $\hat{\tau}$. In c-d), we
trained the network on 1000 realisations of $\tau$, log-spaced between 1 and
10.

It can be seen in Figure 6a that this sub-sampling rate still provides a large
degree of noise in the directly extracted $V_{0,raw}$. Further increasing
$\Delta t$ was found not to significantly affect the issue, as the dynamics
between time-steps then became increasingly important (with a continuously
fluctuating $V(\bm{\mathrm{r}})$). We therefore used standard signal
processing techniques, namely using the transfer function coefficients of a
4th order Butterworth filter (using the MATLAB in-built function butter
Butterworth (1930)) with a zero-phase digital filter (using the MATLAB in-
built function filtfilt Gustafsson (1996)) applied to the noisy $V_{0,raw}$ to
get the smoothed time series $V_{0,smooth}$. As we show in Figure 6a, the
denoised signal well approximates the “true" velocity generating the motion
observed, $\hat{V}_{i}$. Despite the ability to denoise the instantaneous
velocity signal, and obtain reasonable approximations for $\hat{V}$, initial
investigations indicated the inability to accurately estimate small values of
$\tau$, which we attribute to the still prevalent effects of noise Muiños-
Landin _et al._ (2020); Tovey _et al._ (2023). Re-examining Figure 5a, we
again note the threshold $\tau>1$ to observe enhanced localisation effects for
all $L/\langle V\rangle$. We suspect that the shallow gradient in $V$
($(V_{max}/V_{min})/(L/V)=1$s-1) is hidden by the stochastic noise present in
the simulation, such that for small $\tau$ \- where the update in velocity is
near-instantaneous - changes in velocity due to the landscape or because the
effects of diffusion cannot be effectively decoupled. This would in turn
explain why our RNNs are not able to satisfactorily train on real
“experimental" data for $\tau<1$, as the noise from thermal diffusion is of
the same order as the particle acceleration resulting from the change in its
position.

To overcome this issue and still enable the prediction of reasonable $\tau$
values over which non-gaussian behaviours such as localisation are to be
expected, we therefore propose a two-step approach to characterising the
dynamics of rABPs. In the first step, we train the same RNN architecture, but
now implement a binary classifier in the final layer using binary cross-
entropy as the loss metric. More specifically, we label all values $\tau<1$ as
0, and all values of $\tau\geq 1$ as 1, and are thereby able to obtain a very
high accuracy on the test dataset of 0.973. Values of $\tau<1$ are assumed to
be effectively instantaneous, which is a reasonable assumption given for the
Péclet number of the rABPs studied, where localisation only emerges for finite
response times of $\tau>1$. We then proceed to train our RNN architecture on
the now experimentally relevant range of $1\leq\tau\leq 10$, and are able to
obtain good estimates with minimal structures in the residuals (see Figures
6c-d). Notably, the width of the normalised errors has approximately doubled
to 20%, however, the effects of thermal noise on reducing the accuracy of
neural networks is to be expected Muiños-Landin _et al._ (2020), and we
believe that the values for $\hat{\tau}$ we obtain with our approach are
satisfactory.

##  4 Conclusions

We have demonstrated that our numerical model of responsive active Brownian
particles (rABPs) can implement feedback strategies inspired from nature, and
thereby introduce memory into the system. These memory effects in turn result
in out-of-equilibrium dynamics such as enhanced localisation in more realistic
spatial environments than those previously studied. Our approach can also be
extended to include temporal fluctuations, enabling the modelling of e.g.
active agents in complex dynamic environments such as travelling waves Lozano
and Bechinger (2019); Skoge _et al._ (2014). Importantly, the continuous
response of our rABPs provides us with time series of their dynamics, which
not only enables the classification of their response via statistical
analytical tools, but also opens the door to using recurrent neural networks
(RNNs) to characterise their behaviour in a non-trivial manner. Specifically,
RNNs provide us with insights into the physical parameters which govern the
response of ABPs to their environment. By performing experiments with various
microswimmer systems responding to externally imposed variations in their
velocity (or other directed propulsion strategies, e.g. angular velocity), we
expect that it will be possible to determine the responsiveness of different
materials to their environment. Given the importance of memory effects for
active materials, our proposed strategy could thus be used to guide the design
of physically intelligent microrobots capable of autonomously detecting to
their surroundings and performing pre-programmed tasks, helping realise yet
outstanding challenges within the field.

Acknowledgements The authors would like to thank Dr. G Volpe, J Pineda, H. K.
Moberg, and B. Midtvedt for insightful discussions and instruction. This
project has received funding from the European Research Council (ERC) under
the European Union’s Horizon 2020 research and innovation program grant
agreement No. 101001514.

Author Contribution Statement \- Author contributions are defined based on the
CRediT (Contributor Roles Taxonomy). Conceptualization: M.R.B., F.G., L.I.
Discussions: M.R.B., F.G., L.I. Formal Analysis: M.R.B. Funding acquisition:
L.I. Investigation: M.R.B. Methodology: M.R.B. Software: M.R.B., F.G.
Supervision: L.I. Validation: M.R.B. Visualization: M.R.B. Writing - original
draft: M.R.B. Writing - review and editing: M.R.B., L.I.

## References

  * Bechinger _et al._ (2016) C. Bechinger, R. Di Leonardo, H. Löwen, C. Reichhardt, G. Volpe and G. Volpe, _Reviews of Modern Physics_ , 2016, 88, 045006.
  * Ebbens (2016) S. Ebbens, _Current Opinion in Colloid & Interface Science_, 2016, 21, 14–23.
  * Tsang _et al._ (2020) A. C. H. Tsang, E. Demir, Y. Ding and O. S. Pak, _Advanced Intelligent Systems_ , 2020, 2, 1900137.
  * Li _et al._ (2017) J. Li, B. Esteban-Fernández de Ávila, W. Gao, L. Zhang and J. Wang, _Science robotics_ , 2017, 2, eaam6431.
  * Soto _et al._ (2020) F. Soto, J. Wang, R. Ahmed and U. Demirci, _Advanced science_ , 2020, 7, 2002203.
  * Sitti (2021) M. Sitti, _Extreme Mechanics Letters_ , 2021, 46, 101340.
  * Frangipane _et al._ (2018) G. Frangipane, D. Dell’Arciprete, S. Petracchini, C. Maggi, F. Saglimbeni, S. Bianchi, G. Vizsnyiczai, M. L. Bernardini and R. Di Leonardo, _Elife_ , 2018, 7, e36608.
  * Arlt _et al._ (2018) J. Arlt, V. A. Martinez, A. Dawson, T. Pilizota and W. C. Poon, _Nature Communications_ , 2018, 9, 1–7.
  * Arlt _et al._ (2019) J. Arlt, V. A. Martinez, A. Dawson, T. Pilizota and W. C. Poon, _Nature Communications_ , 2019, 10, 1–7.
  * Koumakis _et al._ (2019) N. Koumakis, A. T. Brown, J. Arlt, S. E. Griffiths, V. A. Martinez and W. C. Poon, _Soft Matter_ , 2019, 15, 7026–7032.
  * Khadka _et al._ (2018) U. Khadka, V. Holubec, H. Yang and F. Cichos, _Nature Communications_ , 2018, 9, 1–9.
  * Lavergne _et al._ (2019) F. A. Lavergne, H. Wendehenne, T. Bäuerle and C. Bechinger, _Science_ , 2019, 364, 70–74.
  * Sprenger _et al._ (2020) A. R. Sprenger, M. A. Fernandez-Rodriguez, L. Alvarez, L. Isa, R. Wittkowski and H. Löwen, _Langmuir_ , 2020, 36, 7066–7073.
  * Muiños-Landin _et al._ (2020) S. Muiños-Landin, A. Fischer, V. Holubec and F. Cichos, _Science Robotics_ , 2020, 6, 1–9.
  * Fernandez-Rodriguez _et al._ (2020) M. A. Fernandez-Rodriguez, F. Grillo, L. Alvarez, M. Rathlef, I. Buttinoni, G. Volpe and L. Isa, _Nature Communications_ , 2020, 11, 1–10.
  * Wang _et al._ (2023) X. Wang, P. C. Chen, K. Kroy, V. Holubec and F. Cichos, _Nature Communications_ , 2023, 14, 1–9.
  * Chen _et al._ (2024) Z. Chen, H. Ding, P. S. Kollipara, J. Li and Y. Zheng, _Advanced Materials_ , 2024, 36, 2304759.
  * Alvarez _et al._ (2021) L. Alvarez, M. A. Fernandez-Rodriguez, A. Alegria, S. Arrese-Igor, K. Zhao, M. Kröger and L. Isa, _Nature Communications_ , 2021, 12, 4762.
  * van Kesteren _et al._ (2023) S. van Kesteren, L. Alvarez, S. Arrese-Igor, A. Alegria and L. Isa, _Proceedings of the National Academy of Sciences of the United States of America_ , 2023, 120, 1–9.
  * Mijalkov _et al._ (2016) M. Mijalkov, A. McDaniel, J. Wehr and G. Volpe, _Physical Review X_ , 2016, 6, 1–16.
  * Leyman _et al._ (2018) M. Leyman, F. Ogemark, J. Wehr and G. Volpe, _Physical Review E_ , 2018, 98, 052606.
  * Sprenger _et al._ (2022) A. R. Sprenger, C. Bair and H. Löwen, _Physical Review E_ , 2022, 105, 1–8.
  * Barkai and Leibler (1997) N. Barkai and S. Leibler, _Nature_ , 1997, 387, 913–917.
  * Yi _et al._ (2000) T. M. Yi, Y. Huang, M. I. Simon and J. Doyle, _Proceedings of the National Academy of Sciences of the United States of America_ , 2000, 97, 4649–4653.
  * Vladimirov and Sourjik (2009) N. Vladimirov and V. Sourjik, _Chemotaxis: How bacteria use memory_ , 2009.
  * Berdahl _et al._ (2013) A. Berdahl, C. J. Torney, C. C. Ioannou, J. J. Faria and I. D. Couzin, _Science_ , 2013, 339, 574–576.
  * Aquino _et al._ (2014) G. Aquino, L. Tweedy, D. Heinrich and R. G. Endres, _Scientific Reports_ , 2014, 4, 1–9.
  * Lozano and Bechinger (2019) C. Lozano and C. Bechinger, _Nature communications_ , 2019, 10, 2495.
  * Gosztolai and Barahona (2020) A. Gosztolai and M. Barahona, _Communications Physics_ , 2020, 3, 1–10.
  * Berg and Purcell (1977) H. C. Berg and E. M. Purcell, _Biophysical Journal_ , 1977, 20, 193–219.
  * Berg (1975) H. C. Berg, _Annual review of biophysics and bioengineering_ , Annual Reviews 4139 El Camino Way, PO Box 10139, Palo Alto, CA 94303-0139, USA, Berlin, Heidelberg, 1975, vol. 4, pp. 119–136.
  * Skoge _et al._ (2014) M. Skoge, H. Yue, M. Erickstad, A. Bae, H. Levine, A. Groisman, W. F. Loomis and W.-J. Rappel, _Proceedings of the National Academy of Sciences_ , 2014, 111, 14448–14453.
  * Holubec _et al._ (2021) V. Holubec, D. Geiss, S. A. Loos, K. Kroy and F. Cichos, _Physical Review Letters_ , 2021, 127, 258001.
  * Midtvedt _et al._ (2021) B. Midtvedt, S. Helgadottir, A. Argun, J. Pineda, D. Midtvedt and G. Volpe, _Applied Physics Reviews_ , 2021, 8, 011310.
  * Midtvedt _et al._ (2022) B. Midtvedt, J. Pineda, F. Skärberg, E. Olsén, H. Bachimanchi, E. Wesén, E. K. Esbjörner, E. Selander, F. Höök, D. Midtvedt _et al._ , _Nature communications_ , 2022, 13, 7492.
  * Bailey _et al._ (2022) M. R. Bailey, F. Grillo and L. Isa, _Soft Matter_ , 2022, 18, 7291–7300.
  * Pineda _et al._ (2023) J. Pineda, B. Midtvedt, H. Bachimanchi, S. Noé, D. Midtvedt, G. Volpe and C. Manzo, _Nature Machine Intelligence_ , 2023, 5, 71–82.
  * Bo _et al._ (2019) S. Bo, F. Schmidt, R. Eichhorn and G. Volpe, _Physical Review E_ , 2019, 100, 1–6.
  * Argun _et al._ (2020) A. Argun, T. Thalheim, S. Bo, F. Cichos and G. Volpe, _Applied Physics Reviews_ , 2020, 7, 041404.
  * Argun _et al._ (2021) A. Argun, G. Volpe and S. Bo, _Journal of Physics A: Mathematical and Theoretical_ , 2021, 54, 294003.
  * Gentili and Volpe (2021) A. Gentili and G. Volpe, _Journal of Physics A: Mathematical and Theoretical_ , 2021, 54, 314003.
  * Muñoz-Gil _et al._ (2021) G. Muñoz-Gil, G. Volpe, M. A. Garcia-March, E. Aghion, A. Argun, C. B. Hong, T. Bland, S. Bo, J. A. Conejero, N. Firbas _et al._ , _Nature communications_ , 2021, 12, 6253.
  * Ruiz-Garcia _et al._ (2022) M. Ruiz-Garcia, C. Gutierrez, L. C. Alexander, D. G. Aarts, L. Ghiringhelli and C. Valeriani, _arXiv preprint arXiv:2203.14846_ , 2022, 1–14.
  * Tsang _et al._ (2020) A. C. H. Tsang, P. W. Tong, S. Nallan and O. S. Pak, _Physical Review Fluids_ , 2020, 5, 074101.
  * Qin _et al._ (2023) K. Qin, Z. Zou, L. Zhu and O. S. Pak, _Physics of Fluids_ , 2023, 35, 032003.
  * Nasiri and Liebchen (2022) M. Nasiri and B. Liebchen, _New Journal of Physics_ , 2022, 24, 073042.
  * Mo _et al._ (2023) C. Mo, G. Li and X. Bian, _Frontiers in Physics_ , 2023, 11, 1–17.
  * Tovey _et al._ (2023) S. Tovey, D. Zimmer, C. Lohrmann, T. Merkt, S. Koppenhoefer, V.-L. Heuthe, C. Bechinger and C. Holm, _arXiv preprint arXiv:2307.00994_ , 2023, 1–12.
  * Lipton _et al._ (2015) Z. C. Lipton, J. Berkowitz and C. Elkan, _arXiv preprint arXiv:1506.00019_ , 2015, 1–38.
  * Grønbech-Jensen and Farago (2013) N. Grønbech-Jensen and O. Farago, _Molecular Physics_ , 2013, 111, 983–991.
  * Makridakis and Hibon (1997) S. Makridakis and M. Hibon, _Journal of Forecasting_ , 1997, 16, 147–163.
  * Tse (2002) Y. K. Tse, _The Econometrics Journal_ , 2002, 5, 358–373.
  * Engle (1982) R. F. Engle, _Econometrica_ , 1982, 50, 987.
  * Nelson (1991) D. B. Nelson, _Econometrica_ , 1991, 59, 347.
  * Cho _et al._ (2014) K. Cho, B. Van Merriënboer, D. Bahdanau and Y. Bengio, _arXiv preprint arXiv:1409.1259_ , 2014, 1–9.
  * Chung _et al._ (2014) J. Chung, C. Gulcehre, K. Cho and Y. Bengio, _arXiv preprint arXiv:1412.3555_ , 2014, 1–9.
  * Chollet _et al._ (2015) F. Chollet _et al._ , _Keras_ , https://keras.io, 2015.
  * Bailey _et al._ (2022) M. R. Bailey, A. R. Sprenger, F. Grillo, H. Löwen and L. Isa, _Physical Review E_ , 2022, 106, L052602.
  * Niggel _et al._ (2023) V. Niggel, M. R. Bailey, C. van Baalen, N. Zosso and L. Isa, _Soft Matter_ , 2023, 19, 3069–3079.
  * Lozano _et al._ (2016) C. Lozano, B. Ten Hagen, H. Löwen and C. Bechinger, _Nature communications_ , 2016, 7, 12828.
  * Caprini _et al._ (2022) L. Caprini, U. Marini Bettolo Marconi, R. Wittmann and H. Löwen, _Soft Matter_ , 2022, 18, 1412–1422.
  * Schnitzer (1993) M. J. Schnitzer, _Physical Review E_ , 1993, 48, 2553–2568.
  * Tailleur and Cates (2008) J. Tailleur and M. E. Cates, _Physical review letters_ , 2008, 100, 218103.
  * Stenhammar _et al._ (2016) J. Stenhammar, R. Wittkowski, D. Marenduzzo and M. E. Cates, _Science Advances_ , 2016, 2, 1–6.
  * Butterworth (1930) S. Butterworth, _Wireless Eng_ , 1930, 7, 536.
  * Gustafsson (1996) F. Gustafsson, _IEEE Transactions on Signal Processing_ , 1996, 44, 988–992.


# Memories in the Making: Predicting Video Memorability with Encoding Phase
EEG

Lorin Sweeney, Graham Healy and Alan F. Smeaton<EMAIL_ADDRESS>Insight SFI Research Centre for Data Analytics,
Dublin City University, Glasnevin, Dublin 9Ireland

(2023)

###### Abstract.

In a world of ephemeral moments, our brain diligently sieves through a cascade
of experiences, like a skilled gold prospector searching for precious nuggets
amidst the river’s relentless flow. This study delves into the elusive ”moment
of memorability”—a fleeting, yet vital instant where experiences are
prioritised for consolidation in our memory. By transforming subjects’
encoding phase electroencephalography (EEG) signals into the visual domain
using scaleograms and leveraging deep learning techniques, we investigate the
neural signatures that underpin this moment, with the aim of predicting
subject-specific recognition of video. Our findings not only support the
involvement of theta band (4-8Hz) oscillations over the right temporal lobe in
the encoding of declarative memory, but also support the existence of a
distinct moment of memorability, akin to the gold nuggets that define our
personal river of experiences.

EEG, Video Memorability, Deep Learning

††journalyear: 2023††copyright: rightsretained††conference: International
Conference on Content-based Multimedia Indexing; September 20–22, 2023;
Orleans, France††booktitle: International Conference on Content-based
Multimedia Indexing (CBMI 2023), September 20–22, 2023, Orleans, France††ccs:
Information systems Video search††ccs: Information systems Multimedia
information systems

##  1\. Introduction

In a world awash with fleeting moments, the river of time relentlessly
splashes us with the experience of being. Amidst this mercurial torrent of
sensory droplets—each vying for a place in the precious annals of our
memory—our brain stands as the vigilant gatekeeper, painstakingly managing the
flow of water and deciding which droplets will reach the reservoir of our
memory. However, our reservoir—like any storage system—is subject to
constraints of capacity and encoding efficiency. We accordingly posit that a
critical moment of memorability should exist, an ephemeral yet potent point in
time which captures the essence of an experience, and assigns it a
“remembering priority”, which will ultimately determine its fate within the
annals of our memory when consolidation comes around.

In this paper, we explore the predictive power of encoding phase
electroencephalography (EEG) signals, recorded from subjects during video
stimulus presentation, to predict subject specific recognition upon subsequent
(24–72 hours later) re-viewing. We transform the EEG signals into the visual
domain by turning them into scaleograms with a continuous wavelet function,
which allows us to avail of state-of-the-art visual deep learning techniques.
By leveraging temporal and spatial information contained within the EEG data,
we position ourselves to capture the moment of memorability—a moment of
encoding that corresponds to a remembering moment. We hypothesise that the
neural signals recorded during this moment of memorability will differ from
those recorded during forgettable moments, and that these differences can be
used to predict whether a given subject will remember a given video. We
employed a two factor study design—comparing subject-independent (SI) and
subject-dependent (SD) training approaches, and comparing single electrode and
composite 28 electrode scaleogram images—in order to evaluate the
generalisability of our approach and whether theta band (4–8Hz) activity over
the right temporal lobe (channel P8 being the closest available electrode
location to this brain region in this dataset), which has been implicated in
memory formation (Osipova et al., 2006), leads to more accurate predictions.

##  2\. Related Work

Although the nature and constitution of people’s memories remains elusive, and
our understanding of what makes one thing more/less memorable than another is
still nascent, combining computational (e.g., machine learning) and
neurophysiological (e.g., electroencephalography; EEG) tools to investigate
the mechanisms of memory has the potential to offer insights that would be
otherwise unobtainable. While EEG is not a tool that can directly explain the
factors that make an experience more/less memorable, it can help us trim the
umbral undergrowth surrounding the subject, and offer a potential leap forward
in our understanding of the interplay between the mechanisms of memory and
memorability.

The use of EEG has proven to be an effective tool in the investigation of the
neural mechanisms that underpin memory formation and recall, providing insight
into the timing and general location of these processes (Sanquist et al.,
1980; Karis et al., 1984; Klimesch, 1999; Noh et al., 2014). Even though the
application of machine learning to EEG is an active area of interest—enabling
the classification of various cognitive states and processes, such as emotion
(Wang et al., 2014); mental tasks, e.g., relaxation, counting, multiplication,
word generation (Liang et al., 2006; Lin and Hsieh, 2009); sleep stages
(Ebrahimi et al., 2008); and the automation, or augmentation of neurological
diagnostics (Ieracitano et al., 2020; Lehmann et al., 2007; Hosseinifard et
al., 2013; Engemann et al., 2018)—the use of EEG to predict visual
memorability has been limited to static stimuli such as images (Jo and Jeong,
2020), leaving video entirely unexplored.

##  3\. Dataset & Methodology

EEGMem (Sweeney et al., 2022) is an extension of the Memento10k (Newman et
al., 2020) video memorability dataset. It is comprised of encoding phase EEG
recordings captured from 12 participants as they watched a subset of the
Memento10k videos, and was collected as part of the 2022 MediaEval Predicting
Video Memorability task (Sweeney et al., 2022). EEGMem data collection was
split into two phases: an encoding phase, during which participant EEG was
recorded while watching a continuous stream of 1,000 videos, and an online
recognition phase, which took place 24 to 72 hours after the encoding phase,
and during which they re-watched those 1,000 videos mixed with an additional
1,000 unseen Memento10k videos, responding with the keyboard press if they
recognised a video from the encoding phase, thus providing binary annotations.

###  3.1. Pre-processing

The raw EEG data was first referenced with a common average, and band-pass
filtered between 0.1–30Hz (N.B., this does not affect the time-frequency
resolution of the EEG signal, which was sampled at 1000Hz). Independent
Component Analysis (ICA) was then used to remove artefacts and reject trials
using subject-specific thresholds. Binary annotations from the recognition
phase were used to label subject specific encoding phase EEG trials and their
associated Memento10k videos as “True” for correctly remembered videos (true
positive), and “False” for all other outcomes.

###  3.2. Feature-extraction

(a) 28-channel composite

(b) single channel

Figure 1. Examples of the two types of scaleogram images.

With the goal of leveraging the state-of-the-art Vision Transformer (ViT)
architecture, we turned the filtered EEG data into one of two kinds of
scaleogram with a Morlet wavelet function—one with all 28 electrode channels,
and the other with a single channel (Figure 1). The choice of Morlet wavelet
scaleograms was motivated by its superior time-frequency resolution, and our
desire evaluate the efficacy of the theta band of frequencies (4–8Hz) over the
right temporal lobe (electrode P8) as a predictive feature of video
memorability. Single channel scaleograms were generated with 20 linearly
spaced frequencies from 4 to 8Hz, while each channel in the 28 channel
scaleograms were generated with 8 linearly spaced frequencies from 3 to 17Hz.
The number of cycles used for both sets of frequencies were determined by a
logarithmic function that balances the time-frequency resolution trade-off.
For both scaleogram types we use a baseline period of -0.25 to 0 seconds and
apply z-score normalisation, and then average across trials and visualised for
the time period -0.5 to 3 seconds (the full duration of the video). The final
28 channel scaleogram image is a 4 by 7 grid of individual scaleograms, one
for each channel.

###  3.3. Fine-tuning ViT Models

The ViT (Dosovitskiy et al., 2020) architecture was chosen due to its lower
image-specific inductive bias and its current state-of-the-art status in image
recognition tasks. We fine-tuned (i.e., no layers were frozen, a new task
specific head was created, and the model is re-trained) a total of 52 distinct
ViT-large models pre-trained on the ImageNet-21k dataset. Each model was fine-
tuned on one of four types of data: 28 Channel, Fp1, P8, and Frames (using the
first, middle, and last frame). The fine-tuning procedure was executed under
two distinct training categories: Subject-Independent (SI) and Subject-
Dependent (SD), with 12 SI models (one per subject), and 4 SD models trained
for each data type. SI models were fine-tuned on all subjects aside from one,
in a leave-one-out fashion, and tested on a stratified test set consisting of
15% of the left out subject’s data. SD models were fine-tuned on all subject
data aside from a combined test set composed of each subject’s stratified test
set. All models were trained using the AdamW optimiser with a learning rate of
2e-5, a cosine learning rate scheduler with a 0.1 weight decay, and both early
stopping and a dropout rate of 0.65 were used to mitigate overfitting.

##  4\. Results

Table 1. AUC ROC scores for all models. (Subject Mean excludes Frames column data). | 28 Channel |  | Fp1 |  | P8 |  | Frames |  |
---|---|---|---|---|---|---|---|---|---
SID | SI | SD |  | SI | SD |  | SI | SD |  | SI | SD |  | Subject Mean
S-2 | 0.445 | 0.540 |  | 0.488 | 0.509 |  | 0.558 | 0.572 |  | 0.487 | 0.477 |  | 0.518
S-4 | 0.482 | 0.466 |  | 0.483 | 0.470 |  | 0.634 | 0.577 |  | 0.472 | 0.496 |  | 0.518
S-9 | 0.437 | 0.530 |  | 0.459 | 0.478 |  | 0.569 | 0.613 |  | 0.498 | 0.501 |  | 0.514
S-10 | 0.462 | 0.457 |  | 0.489 | 0.502 |  | 0.568 | 0.578 |  | 0.515 | 0.483 |  | 0.509
S-13 | 0.430 | 0.474 |  | 0.464 | 0.508 |  | 0.574 | 0.596 |  | 0.492 | 0.500 |  | 0.508
S-16 | 0.383 | 0.523 |  | 0.497 | 0.528 |  | 0.534 | 0.564 |  | 0.489 | 0.482 |  | 0.503
S-19 | 0.393 | 0.405 |  | 0.509 | 0.516 |  | 0.578 | 0.636 |  | 0.514 | 0.488 |  | 0.506
S-30 | 0.388 | 0.441 |  | 0.331 | 0.349 |  | 0.340 | 0.436 |  | 0.502 | 0.492 |  | 0.381
S-31 | 0.626 | 0.655 |  | 0.521 | 0.556 |  | 0.510 | 0.520 |  | 0.479 | 0.506 |  | 0.565
S-36 | 0.696 | 0.540 |  | 0.599 | 0.532 |  | 0.707 | 0.564 |  | 0.511 | 0.481 |  | 0.606
S-37 | 0.562 | 0.323 |  | 0.531 | 0.401 |  | 0.614 | 0.632 |  | 0.503 | 0.486 |  | 0.510
S-41 | 0.566 | 0.386 |  | 0.501 | 0.343 |  | 0.567 | 0.538 |  | 0.500 | 0.506 |  | 0.484
Mean | 0.489 | 0.478 |  | 0.489 | 0.474 |  | 0.563 | 0.567 |  | 0.497 | 0.492 |  |
Figure 2. Raincloud plot comparing 28 channel & P8 channel scaleogram trained
models. Figure 3. Raincloud plots for each subject plotting the difference in
ground-truth memorability scores (x-axis) for remembred (tp) and forgotten
(tn) videos. The colour and size of each dot is proportional to the subjects
response reaction time when viewing the video for the second time.

Model performance was evaluated using the Area Under the Receiver Operating
Characteristic Curve (AUC-ROC) as there were only two prediction
classes—remembered and forgotten. Individual classification results for each
subject (column headed SID) are displayed in Table 1, for both the SI and SD
models. This shows that our ViT models trained on channel P8 scaleograms are
the best performing, with a mean AUC-ROC of 0.567 for SD trained models, and a
mean AUC-ROC of 0.563 for SI trained models. A set of ViT models were trained
on a random channel’s (Fp1) scaleograms in order to account for the
possibility that data resolution was the factor driving the 28 channel and P8
single channel performance difference, which the results suggest was not the
case. An additional set of ViT models (Frames) were trained, which were
trained on video frame data rather than EEG data, the results of which were
random, with no discernible difference between SI and SD training, nor notable
difference in subject performance, which is logically coherent with the fact
that video frame data is not influenced by the subject viewing it.

Our repeated-measures design (each participant is in both subject-dependent
and subject-independent conditions, and their data contribute to both
scaleogram types), allowed us to perform a paired t-test, finding that the
AUC-ROC scores for models trained with P8 channel scaleogram images are
significantly (t = 3.243, p = 0.0036) greater than those trained with 28
channel scaleogram images, adding weight the hypothesis that theta band
(4–8Hz) oscillations over the right temporal lobe (channel P8), predict
encoding of declarative memory (Osipova et al., 2006). Figure 2 and Figure 4
provide visual illustrations of the difference between P8 and 28 channel
trained scaleogram images. Although we did not find a significant interaction
between SD and SI training, we can see that more points in Figure 2 lie below
the line of best fit, indicating that more models trained on SD performed
better than their SI trained counterparts, which makes sense from a training
perspective, as EEG data can be highly subject specific, and not having any
subject training examples can hinder prediction performance. Figure 2
additionally highlights the impact of 28 channel vs channel P8 scalograms on a
per subject basis, with a green line indicating that either SD or SI AUC ROC
scores improved and red lines indicating that they both worsened.

Subjects S-30 and S-36 sit on two ends of the performance spectrum, with S-30
producing the worst performances across the board (an average AUC ROC of
0.381), and S-36 producing the highest average AUC ROC of 0.606 and the
highest absolute AUC ROC of 0.707. While both subjects boast an inordinately
high imbalance (¿90%) in their ground truth responses— subject S30 responding
“not seen” for 1,828 videos out of 2,000, and subject S36 doing so for 1,871
videos—subject S30’s results are likely a reflection of the quality of their
response data as a large portion of their response reaction times consistently
rested within the 1 - 1.5 second mark. This suggests a more rhythmic rather
than innate and earnest nature to their responses, whereas subject S36’s
average response reaction times were more varied and typically given ¡1 second
after stimulus onset.

Figure 3 shows the per-subject distributional differences in video ground-
truth memorability (population level) scores for remembered (tp) and forgotten
(fn) videos. A series of statistical tests were carried out in order to assess
the relationship between response reaction times and memorability scores.

###  4.1. Statistical Analysis

First, we conducted a Pearson correlation analysis between response times and
ground-truth memorability scores for each video across all subjects. Our
findings revealed no statistically significant correlations between these
variables for any subject, suggesting that no clear linear relationship exists
between response times and memorability scores. Then we performed two per
subject independent t-tests with Bonferroni correction, testing the
differences between mean response reaction times and mean video memorability
scores for remembered and forgotten videos. For mean response reaction times,
we found significant differences for subjects 10, 13, 16, 19, 30, 36, 37, 9,
and 2, where the mean response reaction times for remembered videos were
consistently higher. For subjects 41 and 4, we found the inverse. Subject 31
was the only subject with no statistically significant difference in their
mean response reaction times. For ground-truth memorability scores, we found
that only subjects 2 and 4’s results showed significant differences in mean
memorability scores. For subject 2, the mean memorability score was higher for
remembered videos (M=0.842) compared to not remembered videos (M=0.771),
t=6.801, p=3.33e-11. Likewise, for subject 4, the mean memorability score was
higher for remembered videos (M=0.809) compared to not remembered videos
(M=0.780), t=2.891, p=0.00395. Given that only subjects 2 and 4 demonstrated a
significant difference, it highlights the fact that the influence of
memorability on long-term video recognition might be inconsistent across
individuals, or potentially affected by individual differences or data quality
variations. Further investigation is required to determine if there is a
causal relationship or if other factors might be influencing the observed
differences.

Figure 4. AUC-ROC scatter plot for scaleogram comparison. Subject specific P8
vs 28 Channel model performance difference is illustrated with coloured
lines—green indicating P8 improvement, and red indicating 28 Channel
improvement.

##  5\. Conclusion

The results of our research support both the hypothesis that theta band
(4–8Hz) oscillations over the right temporal lobe (channel P8) are involved in
the encoding of declarative memory, and the potential existence of a distinct
moment of memorability that can be leveraged to predict subsequent subject-
specific recognition. Although we did not find a significant interaction
between subject-dependent (SD) and subject-independent (SI) training
approaches, the better performance of several SD trained models suggests the
potential importance of subject-specific EEG data for prediction. This
highlights the need for further investigation into the adaptability and
generalisability of models across different individuals.

One limitation of the current study is the reliance on binary ground truth
responses, which may not fully capture the intricacies of memory formation and
recall. Additionally, the presence of volume conduction in EEG measures
warrants further exploration of methods to mitigate or account for its impact
on the results. While the study has unveiled promising avenues for further
exploration, demonstrating the feasibility of using EEG signals and deep
learning techniques to predict video memorability, it also raises new
questions and challenges for future research.

###### Acknowledgements.

This publication has emanated from research partly supported by Science
Foundation Ireland (SFI) under Grant Number SFI/12/RC/ 2289_P2 (Insight SFI
Research Centre for Data Analytics).

## References

  * (1)
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Ebrahimi et al. (2008) Farideh Ebrahimi, Mohammad Mikaeili, Edson Estrada, and Homer Nazeran. 2008.  Automatic sleep stage classification based on EEG signals by using neural networks and wavelet packet coefficients. In _2008 30th Annual International Conference of the IEEE Engineering in Medicine and Biology Society_. IEEE, 1151–1154.
  * Engemann et al. (2018) Denis A Engemann, Federico Raimondo, Jean-Rémi King, Benjamin Rohaut, Gilles Louppe, Frédéric Faugeras, Jitka Annen, Helena Cassol, Olivia Gosseries, Diego Fernandez-Slezak, et al. 2018\.  Robust EEG-based cross-site and cross-protocol classification of states of consciousness.  _Brain_ 141, 11 (2018), 3179–3192.
  * Hosseinifard et al. (2013) Behshad Hosseinifard, Mohammad Hassan Moradi, and Reza Rostami. 2013.  Classifying depression patients and normal subjects using machine learning techniques and nonlinear features from EEG signal.  _Computer methods and programs in biomedicine_ 109, 3 (2013), 339–345.
  * Ieracitano et al. (2020) Cosimo Ieracitano, Nadia Mammone, Amir Hussain, and Francesco C Morabito. 2020.  A novel multi-modal machine learning based approach for automatic classification of EEG recordings in dementia.  _Neural Networks_ 123 (2020), 176–190.
  * Jo and Jeong (2020) Sang-Yeong Jo and Jin-Woo Jeong. 2020.  Prediction of visual memorability with EEG signals: A comparative study.  _Sensors_ 20, 9 (2020), 2694.
  * Karis et al. (1984) Demetrios Karis, Monica Fabiani, and Emanuel Donchin. 1984\.  “P300” and memory: Individual differences in the von Restorff effect.  _Cognitive Psychology_ 16, 2 (1984), 177–216.
  * Klimesch (1999) Wolfgang Klimesch. 1999\.  EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis.  _Brain research reviews_ 29, 2-3 (1999), 169–195.
  * Lehmann et al. (2007) Christoph Lehmann, Thomas Koenig, Vesna Jelic, Leslie Prichep, Roy E John, Lars-Olof Wahlund, Yadolah Dodge, and Thomas Dierks. 2007\.  Application and comparison of classification algorithms for recognition of Alzheimer’s disease in electrical brain activity (EEG).  _Journal of neuroscience methods_ 161, 2 (2007), 342–350.
  * Liang et al. (2006) Nan-Ying Liang, Paramasivan Saratchandran, Guang-Bin Huang, and Narasimhan Sundararajan. 2006\.  Classification of mental tasks from EEG signals using extreme learning machine.  _International journal of neural systems_ 16, 01 (2006), 29–38.
  * Lin and Hsieh (2009) Cheng-Jian Lin and Ming-Hua Hsieh. 2009.  Classification of mental task from EEG data using neural networks based on particle swarm optimization.  _Neurocomputing_ 72, 4-6 (2009), 1121–1130.
  * Newman et al. (2020) Anelise Newman, Camilo Fosco, Vincent Casser, Allen Lee, Barry McNamara, and Aude Oliva. 2020\.  Multimodal Memorability: Modeling Effects of Semantics and Decay on Video Memorability. In _Computer Vision – ECCV 2020_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 223–240.
  * Noh et al. (2014) Eunho Noh, Grit Herzmann, Tim Curran, and Virginia R de Sa. 2014\.  Using single-trial EEG to predict and analyze subsequent memory.  _NeuroImage_ 84 (2014), 712–723.
  * Osipova et al. (2006) Daria Osipova, Atsuko Takashima, Robert Oostenveld, Guillén Fernández, Eric Maris, and Ole Jensen. 2006.  Theta and gamma oscillations predict encoding and retrieval of declarative memory.  _Journal of neuroscience_ 26, 28 (2006), 7523–7531.
  * Sanquist et al. (1980) Thomas F Sanquist, John W Rohrbaugh, Karl Syndulko, and Donald B Lindsley. 1980.  Electrocortical signs of levels of processing: Perceptual analysis and recognition memory.  _Psychophysiology_ 17, 6 (1980), 568–576.
  * Sweeney et al. (2022) Lorin Sweeney, Mihai Gabriel Constantin, Claire-Hélène Demarty, Camilo Fosco, Alba García Seco de Herrera, Sebastian Halder, Graham Healy, Bogdan Ionescu, Ana Matran-Fernandez, Alan F. Smeaton, and Mushfika Sultana. 2022.  Overview of The MediaEval 2022 Predicting Video Memorability Task. In _MediaEval Multimedia Benchmark Workshop Working Notes_.
  * Wang et al. (2014) Xiao-Wei Wang, Dan Nie, and Bao-Liang Lu. 2014.  Emotional state classification from EEG data using machine learning approach.  _Neurocomputing_ 129 (2014), 94–106.