# On learning spatial sequences with the movement of attention

V.M. Osaulenko
Igor Sikorsky Kyiv Polytechnic Institute
Kyiv, Ukraine
<EMAIL_ADDRESS>
Use footnote for providing further information about the author (webpage,
alternative address)— _not_ for acknowledging funding agencies.

###### Abstract

In this paper we start with a simple question, how is it possible that humans
can recognize different movements over skin with only a prior visual
experience of them? Or in general, what is the representation of spatial
sequences that are invariant to scale, rotation, and translation across
different modalities? To answer, we rethink the mathematical representation of
spatial sequences, argue against the minimum description length principle, and
focus on the movements of attention. We advance the idea that spatial
sequences must be represented on different levels of abstraction, this adds
redundancy but is necessary for recognition and generalization. To address the
open question of how these abstractions are formed we propose two hypotheses:
the first invites exploring selectionism learning, instead of finding
parameters in some models; the second proposes to find new data structures,
not neural network architectures, to efficiently store and operate over
redundant features to be further selected. Movements of attention are central
to human cognition and lessons should be applied to new better learning
algorithms.

_Keywords_ spatial sequence $\cdot$ sequence learning $\cdot$ movement of
attention $\cdot$ selectionism

##  1 Introduction

This paper deals with the problem of learning spatial sequences like the
movements of objects. In contrast to temporal sequences, like music or text,
spatial sequences have distance and direction relations for any two elements.
Interestingly, there are no senses for distances and directions but biological
organisms receive this information with a sequential, not parallel, process of
eye saccades and covert movements of attention.

Such a sequential manner drastically differs from many computational
approaches, like recurrent neural networks [10] that treat the spatial
sequence as temporal. Also, convolutional neural networks despite showing
great performance in computer vision tasks have problems with geometry-related
tasks [29, 28]. That is why practical applications of SLAM in robotics often
use hybrid approaches [22], combining neural networks with precise algorithms
for spatial data.

In recent studies, it was shown that humans leverage spatial relationships for
sequence learning [33, 1]. We discover regularities on multiple nested levels
similar to the language structure. Also, we chunk long sequences into the
composite to reduce the working memory load and improve the prediction of the
next locations. All this is done in an unsupervised manner, with the proposed
goal to reduce the minimum description length (MDL). The brain-inspired
mechanisms still have a lot to share to improve the current algorithms. For
example, recently it was proposed [5] to return to the active vision approach
for robotics perception that computes sequentially where to look for the most
informative places.

The goal of this paper is to highlight the main challenges in spatial sequence
learning and bring new or revive old insights and ideas by analyzing human
perception and mathematical representation of spatial sequences. The paper
proceeds with a thought experiment that addresses the invariant recognition of
spatial sequences. Next, we clarify what sequence learning is and how spatial
sequences are different. Further, we argue that most mathematical forms of
sequences should be changed to better reflect invariances. Thus, we favor the
instruction form, that links with the previous research on MDL representation.
However, we provide arguments that MDL is not compatible with making
abstractions, redundancy should be added to receive generalization. In the
last section, we address the movement of attention both in space and scale.
Unlike temporal sequences, spatial relationships alleviate the problem of
long-range sequence dependencies and are represented on different abstraction
levels.

In the end, we propose two hypotheses, on how to proceed with brain-inspired
algorithms for spatial sequence learning. First, invites exploring
selectionism learning, instead of finding parameters in some models. The
second addresses the problem of the exponential explosion of features to be
selected, with the idea to explore new data structures, that are not
necessarily neural network based.

##  2 Thought experiment

Imagine a person that looks at a screen where the black dot moves in three
sequential patterns (a right triangle, letter N, and letter U). Afterward, an
experimenter draws these patterns over the skin of a person’s arm while the
person’s eyes are closed (see fig. 1a). Everyday experience tells that,
trivially, the person can recognize such simple sequences over skin
flawlessly. We can make a number of observations and conclusions from the
experiment.

1\. Movement of attention. Even though visual information is processed in the
visual cortex and the skin perception is somatosensory, none of them are
responsible for sequence recognition and prediction alone. Otherwise, transfer
of recognition between modalities would not be possible. One could argue that
both visual and somatosensory cortices are topographically organized, and
serve as a saliency map, thus both elicit similar movements of attention.
However, drawing patterns on two arms stacked together also gives correct
recognition, even though the activation is discontinuous in the somatosensory
cortex. Therefore, both sequences are encoded in the same neural structure
(possibly that represents the space around the body), which gives a similar
continuous movement of attention in both tasks.

2\. Invariance. Recognition does not depend on the location where the pattern
was drawn on the body. The arm, leg, or tongue does not matter as long as the
movements are made sufficiently large to differentiate. The same is true for
orientation and the scale of the sequences.

3\. Working memory. Awareness of the location of the beginning of the sequence
and its corners persists even after the stimulus presentation. Spatial
sequences leave a lasting activity in contrast to pure temporal sequences like
music or language. Therefore, the order of the sequence has less influence on
the recognition.

4\. Multiple predictions. Before the sequence is finished, once it is
recognized, it can be predicted. In case of ambiguity, multiple predictions
are made. However, this relies on memory, for a more complex stimuli
prediction would be harder.

5\. Abstraction. Recognition does not depend on noisy variations like the
length or angle of the segments of patterns, although variations are
noticeable. Therefore, the sequence is represented in an abstract way while
being able to differentiate different instances.

This thought experiment is an instance of what is known in the literature as a
cross-modal transfer of knowledge. Multiple experiments showed that learning
in one modality enhances or transfers to another, like for visual-auditory
[37], visual-haptic [36], or even in motor sequences effector1-effector2
(movement of fingers and elbows in [4]). Based on this, the multisensory
hypothesis states that an abstract amodal representation of objects must be
created that is responsible for such a transfer. Also, this idea of amodal
abstractions becomes more salient in light of the recent trend for multimodal
models [12].

All these observations form the necessary features of any algorithm designed
for spatial sequence learning. Note, spatial sequence processing is closely
related to space representation in the entorhinal-hippocampal network, where
grid cells, place cells, and many others were found, see review [34]. However,
in this paper, we avoid biological realism and focus on a mathematical
representation of spatial sequences and the computational problems of
recreating these observations. But at first, we briefly overview the task of
sequence learning.

Figure 1:  a)Illustration of a thought experiment. Three spatial sequences are
presented visually but can be recognized with somatosensory input b)Summary of
the main sequence learning problems

##  3 Background in sequence learning

A discrete sequence $s_{1:N}=\\{s_{i}\in A,i=1:N\\}$ of length $N$ is an
ordered collection of elements from some set $A$. Individual elements are not
comparable, but the sequence induces the temporal structure, like $s_{i}$ and
$s_{i+1}$ are adjacent in time. Usually, the temporal structure is described
by a joint probability distribution over all elements $p(s_{1:N})$. The next
categorization is based on [31], with some modification and extension.

Sequence learning includes many related tasks over sequences. The central is a
sequence prediction, where given a sequence $s_{1:n}$ the task is to predict
the next $k$ elements $s_{n+1:n+k}$. This is closely related to pattern
completion, with the difference that the task includes outputting missing
elements in the middle or at the beginning of a sequence. Sequence recognition
deals with the problem of assigning the correct label to a sequence
$s_{1:n}->l$, where the label can be just a number. In online sequence
recognition the output is a distribution of possible labels after the
presentation of elements one by one. Sequence generation is a reverse problem
of retrieving the sequence from a label ($l->s_{1:m}$). We can use online
recognition and generation to solve the prediction task. If recognition
outputs multiple hypotheses, then multiple predictions are made. Sequence-to-
sequence learning links one sequence to another $s^{1}_{1:n}->s^{2}_{1:m}$,
which is common in practical applications, like text translation. Great
importance has a sequence clustering task with the goal to find groups of
similar sequences, especially for sequences with complex structure, noise, and
variations. Clustering reduces dimensions and is also known as finding the
prototype or forming the abstraction. Closely related to clustering is a
sequence compression task. Lossless compression reduces the redundancy of
temporal structure but does not lose information and is widely used for text
or code compression. Lossy sequence compression discards less useful
information based on some criteria, thus compresses more but cannot restore
the original sequence.

To reduce dimensionality it is essential to utilize the structure of the
sequence. The sequence has a compositional structure if it is formed from
other sequences $s_{1:n}=s^{1}_{1:k}s^{2}_{1:m}$. Note, any sequence can be
split into subsequences arbitrarily, however, a good division utilize the
statistics where the resulted subsequences appear in many other sequences. The
sequence has a hierarchical structure if it has a recursive compositional
structure, where the subsequences are composed of other subsequences. See
fig.1b which illustrates major sequence learning tasks together. One of the
prominent ideas is that by finding the minimum description length (MDL) of the
sequence (that is related to Kolmogorov complexity) the algorithm should
uncover the hierarchical and compositional structure. However, no such
algorithm exists to this day, also with some notable attempts [8].

This paper considers spatial sequences where each element belongs to
k-dimensional space $r_{1:N}=\\{(r_{i}):r_{i}\in R^{k},i=1:N\\}$. Usually, the
space is Euclidian with $k=2$ or $k=3$, and two elements are related via
distance vector ($r_{i}=r_{j}+d_{ij}$). Often, such sequences are described in
a continuous space as a parametrized function known as a curve. All previous
sequence learning tasks are applicable to spatial sequences as well. However,
because elements have a distance relation the solutions to prediction,
recognition, and generation tasks might leverage spatial structure. For
example, two vertical lines of equal length can have different locations, thus
different sequences of numbers. Still, they have the same pairwise differences
and thus are more similar than two random lines. Note, movement trajectories
like in the thought experiment are spatial sequences.

Sequence learning arises in many domains and applications. In neuroscience, as
a sequential activation of spiking neurons, or chunking the perception into
the sequence of events. Recent work suggests that cells that encode places,
boundaries, and landmarks arise from a pure sequence learning [25]. In machine
learning, the statistical approach is used to learn sequences on large
datasets. Particularly, the LSTM and transformer models with many different
variations are applied to text, sounds, or even images. Reinforcement learning
deals with sequential decision-making and can be reframed as a large sequence
learning task [16]. Sequence prediction is at the core of algorithmic
information theory where the prediction heavily depends on the sequence
complexity [14]. In control theory, a fusion of visual and inertial
information to track the robot’s position in space can also be treated as
sequence-to-sequence learning [3]. Note, that all these fields deal with
slightly different flavors of sequence learning, whether it is the difference
in elements (text, sounds, spatial locations), discrete or continuous case,
large or small database, statistical or one-shot learning. Therefore, all have
different mathematical descriptions and algorithms.

##  4 Spatial sequence representations and their problems

###  4.1 Linear movement

Returning to the thought experiment, the three figures consist of three linear
segments. To encode a line segment it is enough to specify the position of two
endpoints $(p_{0},p_{1})$. The most common representation of a segment is
parametric: $p(t)=p_{0}+(p_{1}-p_{0})t,t\in[0,1]$ where a vector $p(t)$
encodes all segment points, see fig. 2a. To represent the segment in hardware
it should be discretized, either by selecting a number of points $N$ or a
discretization step of parameter $t$: $\Delta t=L/N$, where $L$ is the segment
length. In any case, for two dimensions the result is a sequence of numbers,
like $s_{1}=[(1,2),(2,4)...(100,200)]$, that can be visualized in a 2D figure,
like in figure 2a.

The problem with this representation is that Euclidean transformation
(translation, rotation, and scaling) leads to a completely different sequence
of numbers. But we would like to know that the two segments are actually the
same, but moved. In computer vision, this problem of alignment between two
sets of points is known as point set registration and classically is solved
with iterative algorithms (like the iterative closest point). However, humans
use another strategy, they extract invariant features from two sequences and
find correspondences in feature space. Many deep learning architectures also
learn features and then search for correspondences, however, they require a
lot of training data and poorly generalize. See a recent review for different
approaches to point set registration [13].

Also, humans do not use end-to-end learning but rather perform unsupervised
feature extraction, where features can be in the form of subprograms [20]. For
example, we could write the segment of a line in the differential form
$\frac{dp(t)}{dt}=q,p(0)=p_{0},t\in[0,L]$, where $q$ is a unit vector in the
direction from $p_{0}$ to $p_{1}$. Or in discretized version
$p_{i+1}=p_{i}+q\dot{\Delta}t,i=0:L/\Delta t$, that can be interpreted as an
instruction: “start at point $p_{0}$ and repeat a small movement in the
direction $q$ until the length L is reached”. Another conceptually different
instruction is “to repeat the movement until the position is close enough to
the endpoint $p_{1}$”. The first case requires measuring the length. The
second needs to perceive the current location globally and compare it with the
stored coordinate. In any case, the two translated lines have different
locations, but the instruction for change remains the same. As was suggested,
humans may widely use instructions for geometry processing dubbed “language of
geometry” [2].

The weak spot of the instruction form is that given two line segments it is
unknown if they intersect. Both instructions should be executed, intermediate
results saved, and then checked for collisions. This is in contrast to visual
imagination, where all points of two segments are constantly present. Or
another weakness is the length of a segment. What units to use? Meters?
Interestingly, people do not imagine a linear segment with a certain length,
it has direction but not size. The size appears as a ratio to other objects,
therefore next we consider several linear segments.

###  4.2 Piecewise linear movement

Two connected line segments can be represented as the coordinates of three key
points (A, B, C) see fig. 2b. But again, spatial transformation changes the
sequence of numbers completely. A better way to represent segments as an
initial position plus a sequence of turning angles and lengths, $p_{0}$ plus
$(\phi_{0},L_{0}),(\phi_{1},L_{1})$ (see a fundamental theorem of space
curves). Here $\phi_{0}$ is an angle with a horizontal axis, and $\phi_{1}$ is
relative to the first segment. Furthermore, to fix the problem of measuring
length, we can specify not length but the ratio of segments length:
$(\phi_{1},a_{1})$, where $a_{1}=L_{1}/L_{0}$ and $L_{0}$ becomes some
arbitrary length. Under rotation, turning angles do not change (except the
first one), and the ratio stays the same under scaling.

This representation is similar to what people use in navigation, they
construct landmarks and learn instructions for going from one place to the
next. However, consider fig. 2c, two sequences are very similar in space but
have different numbers of keypoint and slight variations that make comparison
with turning angles representation complicated. So this chain-like
relationship is not a robust strategy and representation should include
relations with other distant elements on different scales.

Figure 2:  a)Parametric representation of a linear segment. b)Representation
of multiple segments with length and turning angles. c) Illustration that two
similar spatial sequences may have very different representations. d) Bezier
curve with handles. e) A curve as a shape with endpoints plus extremum point
representation. f) Fitting points with two Bezier curves (shown with different
thicknesses). g) Two different sequences are the same at a higher abstraction
level.

###  4.3 Curves

In differential geometry, a curve is a continuous map from some interval
$(a,b)$ to n-dimensional space $p:(a,b)\in R->R^{n}$. The well known example
is a 2d circle $p(t)=(r\cos t,r\sin t),t\in[0,2\pi)$. Similar to a linear
segment, in computers, the compact functional form must be converted to a
sequence of numbers.

In practical applications like vector computer graphics, complex curves
usually are approximated with the composition of Bezier curves. They are easy
to manipulate, resolution-free, and have a compact functional representation
compared to pixels in raster graphics. 2D Bezier curve compresses a sequence
of point positions to just 8 numbers that describe four control points, see
fig. 2d. Many more representations of curves exist (see a book chapter [9])
but all trade computation over memory, that is store a long sequence of
numbers as a function with a much shorter sequence of parameters.

But again, under transformation, noise, and variations these parameters
change, and no invariances in the parameter space are preserved. Consider in
contrast instruction representation of turning angles, similar to piecewise
linear segment. The circle can be represented as: “1. start at some position;
2. move in some direction by a small amount; 3. turn by a small angle; 4.
Repeat 2 and 3 until return to a starting position”. This instruction holds
for all circles at any position and any scale, with the only difference in the
amount of movement.

Experiments in human perception suggest that sensory inputs of the movement
trajectories are also encoded in a compact form similar to programs [1]. This
observation is linked with the minimum description length (MDL) principle,
that neural networks encode the sequence with the shortest instruction, thus
becoming more energy-efficient in terms of neural activation. However, as we
show next, shorter is not necessarily better.

###  4.4 Real spatial sequences

The key distinction of real spatial sequences, like movements on the skin,
striding pedestrians, or swinging of a falling leaf is that they are perceived
in a form of raw data as the sequence of numbers (or receptor activation).
Instead of two points for a line, or parameters of the curve, we have just a
sequence of numbers.

The statistical approach proposes to use a parametric model and to optimize
some criteria to fit parameters to the data. With this method, we can
approximate the sequence of numbers with a composite Bezier curve, splines, or
some other parametric function, like an artificial neural network. See for
example fig.2f, where points are fitted with two Bezier curves, which makes a
compact representation. However, similar to images, where objects will always
produce variations in pixel values, real spatial sequences never repeat
exactly the same trajectory. Thus, curve fitting is challenged with the same
problem of generalization, how one model is applicable to different sequences.
A huge amount of research is dedicated to this topic, where bias-variance
tradeoff takes the central place. A good example of finding this tradeoff for
curve fitting with B-splines is presented in the work [7].

Recently, Vapnik, one of the founders of the statistical learning theory,
introduced the notion of statistical invariants (as derived from predicates,
see [32]), special equalities that link observations and the hypothesis of the
model, like the parameters for the Bezier curve. These invariants are
different from features from classic learning, they reduce the space of
possible hypotheses, while more features increase the hypothesis space. With
good invariants, learning is faster, requires less data, and gives better
generalization. But finding these invariants Vapnik calls the “intelligent
part of the learning problem”, in contrast to finding parameters to fit the
data.

Such invariants may be a symmetrical relationship, or some other rules that
link the features, like the relative position of eyes and a nose for a face.
In previous sections, it was shown that the instructions for curves that
follow the movement of attention can provide the invariance for translation,
rotation, and scaling. Perhaps, the movement of attention can help with
finding good rules that explain and predict the sequence. However, in
addition, four issues need to be addressed in the next section.

##  5 Movement of attention

###  5.1 Movement in space

Consider again the thought experiment. The sequence of positions is perceived
not element by element like music, but as a lasting perception of the overall
shape. Most likely, frontoparietal connections are responsible for this, where
the frontal cortex act as a working memory (with complex nested tree-like
structure) for elements of the sequence represented in the parietal cortex,
see more at [33].

Sequence learning applied to temporal sequences usually faces the problem of
long-range dependencies, because of exponentially increased dimensionality of
possible combinations (state space). The temporal-to-spatial recoding for
spatial sequences changes the problem of long-range dependencies to the search
for relationships in the spatial domain. The movement of attention between
persistent salient features gives the distance relationships and can provide
additional invariances like symmetry or specific spatial patterns like the
corners of a triangle. With the movements, we can approximate a curve as a
collection of three features: the position and direction of two endpoints plus
the position of the most distant point from the line that connects the
endpoints (see fig.2e). This represents a 1d curve more like a 2d shape and
makes the task of spatial sequence learning similar to active vision, where
the image is processed not as a whole but as a sequence of features from the
eye movements.

In addition, the spatial representation explicitly represents line-line
intersection, which is not possible with the instruction form. This follows
everyday experience, draw two lines on the skin, and you know if they
intersect. Furthermore, the intersection point becomes salient and acts as a
new feature to describe the composite shape. The same is true for visual
imagination, mental movements of two line segments create at some moment an
intersection, and we can estimate the relative distance from the intersection
to the endpoint or any other spatial relationships.

It is far from clear how exactly this space representation is formed on a
neural level. Perhaps, something functionally similar to grid cells, place
cells, distance cells, etc. that are found in the hippocampus and entorhinal
network is present in the neocortex (in our case most likely parietal), as was
hypothesized in [21]. Or perhaps the entorhinal cortex represents not just
locations in space, but visual space as well, following some evidence provided
in [19, 17]. Nevertheless, it feels that mental travel in different locations
is different from a mental rotation of a triangle. So it is likely that a
dedicated network should exist, but the neural mechanism for encoding space
and visual-object space might be similar.

###  5.2 Movement is scale

If the analogy with the encoding space in the hippocampus is correct then we
can make parallels with the scale as well. The dorsal part of the hippocampus
encodes place on a small scale, like rooms, and the ventral part on a larger
scale, like streets [11]. This dorsoventral longitudinal axis forms a scale
axis. We may hypothesize that similarly a spatial sequence is encoded in
different scales with varying accuracy of details. Two features, like corners,
might be treated separately on one scale, but on the other as one feature.
This directly relates to grouping by spatial proximity from Gestalt
principles. Consider two different curves (fig.2g) that perceptually look
similar. This similarity arises because the sequence representations are the
same on the level of some neural populations responsible for encoding data at
some scale. This hypothesized axis of scale for curves would encode data in a
coarse-to-fine manner.

Unfortunately, this mechanism is not found, and no computational algorithms
can demonstrate such a bottom-up, unsupervised coarse-to-fine representation
of spatial sequences yet. In 1984, this idea was demonstrated for 1d data,
with the solution in a form of a tree where the deeper level encodes more fine
features [35]. Subsequent works on 2d data on images were not as successful
but resulted in other algorithms known as active contours or snakes [18].

Representation of spatial sequences on different scales results in different
levels of abstraction. However, this forms a redundant representation, since
the lowest level contains all information. This deviates from the conclusions
made in recent paper [1], that humans represent sequences following the MDL
principle. It is possible that their conclusion is valid for studied spatial
sequences of locations on the octagon. But we argue, that in order to compare
different sequences and be able to generalize, redundant representation on
multiple levels of abstractions is necessary.

Similar confusion is present in the compression community, where the primary
goal is to achieve lossless compression that later can always be made coarse,
see M.Hutter web page. But finding similarity and patterns in a compressed
representation is not trivial [23].

Representation on multiple scales may be an instance of a balance in memory-
computation trade-off where more memory (needed for redundant representation)
is traded for less computation (to compare sequences and find similarities for
generalization). Classical point set registration algorithms do not use
redundant memory for intermediate features and instead use more computations.
Humans, most likely, have multiple representations that exist simultaneously,
and attention selects the necessary level of description for a given task. So
attention moves not just in space but in scale, group close features together,
or differentiate more complex to composite.

Figure 3: Illustration of the difference between parametric learning and
selectionism learning. Top: the hypotheses space is fixed and learning is a
search for the best model. Bottom:

###  5.3 Movement in hierarchy

Temporal-to-spatial recoding makes it easier to form spatial relationships
with temporarily distant features. However, if the spatial sequence becomes
too long, with too many features, the capacity of working memory in the
frontal cortex is not enough to sustain spatial representation. The well-known
strategy to fix this is to chunk, or combine, features into another composite
feature that requires less working memory [26].

This changes the mathematical representation of the sequence to a recursive
form $U=\\{(s_{i},r_{i}):s_{i}\in U,r_{i}\in R^{k},i=1:N\\}$ as a collection
of features $s_{i}$ at some locations where each $s_{i}$ can be as well
composed of feature-location pairs. Thus, a numerical sequence of positions is
replaced with a sequence of symbols at some positions. Moreover, each
recursive level may be represented in a local reference frame, very similar to
the architecture widely adopted in computer graphics.

An interesting option is that the parietal cortex stores these symbols as
small chunks of typical patterns of the movement of attention. Given that the
parietal cortex has a similar structure to other parts, like visual or
auditory, and we know that later store composite sensory features, it is
reasonable to assume that the parietal cortex does the same with the inputs of
the movements of attention. And possibly its role not directly in attentional
control, as suggested before [30], but in attentional pattern recognition and
completion. Moreover, similarly to the movement in scale, attention moves
across a hierarchy of composite features to find the appropriate level for
some task. How it is different from movement in scale is unclear, maybe the
same mechanism is used with the exception that on a larger scale, details are
not lost but compressed in the form of symbols.

###  5.4 Origin of the movements

Sequence learning is based on the ability to compare sequences. In a simple
case prediction or recognition can be done with a nearest-neighbor search. But
how to form a good feature space to make comparisons, especially without large
datasets and classical supervised or self-supervised setup?

We propose two hypotheses that might lead to an answer.

Hypothesis 1 (Selection). Good features, invariants, instructions, and levels
of abstraction can be received by a selection process from a large dynamic set
of possible candidates.

This hypothesis is illustrated in fig.3, where learning is not a movement in a
fixed parametric space to minimize some error function, but a generation and
selection of the best hypotheses that explain and predict the data. An example
of such a generation process is multiple saccades of the eye that scan the
scene to receive distance and direction relationships. We hypothesize that
similarly the movements of attention are made in an excessive manner but in a
cognitive space, which also includes movement in scale, that groups or
differentiates features. Relationships from movements that reoccur or
correlate across different inputs are reinforced and less likely to be erased.
The selected relationships might form good compact representations for solving
sequence learning tasks, particularly to perform comparison and sustain
generalization. Such extensive search and elimination might be the mechanism
for finding statistical invariants introduced by Vapnik.

This hypothesis is based on two previous ideas. The first is selectionism,
where the large population diversity increases the probability to have
individuals with some useful qualities that will not be filtered by an
external selection process. It has many realizations in other fields, like
evolution or optimization, and was promoted to explain brain functioning by
neural group selection, known as "Neural Darwinism" [6]. The proposed
mechanisms of polychronization [15] in spiking neural networks turned out to
be wrong and cannot stably implement such selection. Still, a modern view on
cell assemblies formation with multiple inhibitory and disinhibitory circuits
[27, 24] might form a basis with sufficient memory capacity for such
redundancy and selectivity.

The second idea is from the models of computer vision before the deep learning
surge (pre-2012). Many wrote about what deep learning brought new, but few
mention what was unfortunately forgotten. The formation of saliency maps from
the first principles of information theory and the selection of features with
further comparison were good brain-inspired ideas. However, they lost the race
in the benchmark metrics, and instead of further research were overshadowed by
the deep learning approach.

A straightforward implementation of the proposed selection hypothesis using
the pre-2012 approach fails. It quickly runs into computational limitations
since the number of possible relationships grows exponentially. Therefore to
select better relationships given a finite amount of memory we need to order
locations and scales where to attend by importance. Also, other heuristics
will be useful, like the composition of features based on some criteria (see
Gestalt principles). However, selectionism does not propose to return to
handcrafted features but to use lessons from a previous learning decade and
utilize huge computational resources for generating and selecting good
features. But to overcome exponential explosion we make a second hypothesis.

Hypothesis 2 (Nonparametric data structure). An efficient data structure that
compactly stores features and their relationships and allows for quick search,
composition, and comparison exists and is not based on neural networks.

This hypothesis is based on the analogy to algorithms and data structures for
string processing (like radix tree, suffix tree, etc). They provide an
inspiring example, where the search among a billion strings requires examining
only around 30 bits. String symbols should not be fundamentally different from
features or even instructions and reduction in computation should apply here
as well.

Such reduction is widespread in biological networks. On a local level, some
inhibitory neurons target specifically axon hillock (spike generation site) to
prevent neurons from sending action potential. What is not sent usually may
not be computed. On a system level, the activation function of every neuron in
a network is not calculated, but only of certain areas selected by the
movements of attention.

These two hypotheses invite deviating from the common approach of finding
parameters that fit the data according to some loss function. And to explore
nonparametric learning procedures based on selectionism with redundant
movements of attention.

##  6 Conclusions

In this paper, with the help of a thought experiment, we highlighted some
properties of human spatial sequence perception. We argued that the invariance
to transformation arises from the movement of attention, which is similar to
the instruction form for spatial sequence representation. Unlike temporal
data, spatial data leave a persistent activation due to the working memory.
This temporal-to-spatial recoding alleviates the problem of long-range
dependencies and helps to uncover good features, like symmetry. The movement
of attention in scale is like the movement in a coarse-to-fine axis that
reduces the complexity of a sequence but increases redundancy. This is in
contrast to the MDL principle, but we argue that redundancy is inevitable in
order to have multiple levels of abstraction that are the basis for
generalization.

We provided two hypotheses on how to make and organize the right movements of
attention. Instead of finding iteratively the better solution in some
parametric model, selectionism learning forms as many possible relationships
and features with more data, with further elimination of the less useful. To
deal with computational complexity the second hypothesis purposes that some
efficient data structures could store different features with a quick search
and low memory. Still, many unresolved questions remain, like how to precisely
define the usefulness of features without the error function, or what is a
good problem setup to test the proposed hypotheses.

Overall, the topics mentioned in this paper like composition, hierarchy,
generalization, and multiple levels of abstraction are well known and eagerly
discussed in the literature, (see recent review [34]), but no agreement is
made on how to approach these problems. This paper proposes to explore the
nonparametric selectionism approach to learning.

## References

  * [1] F. Al Roumi, S. Marti, L. Wang, M. Amalric, and S. Dehaene.  Mental compression of spatial sequences in human working memory using numerical and geometrical primitives.  Neuron, 109(16):2627–2639.e4, 2021.
  * [2] M. Amalric, L. Wang, P. Pica, S. Figueira, M. Sigman, and S. Dehaene.  The language of geometry: Fast comprehension of geometrical primitives and rules in human adults and preschoolers.  PLoS computational biology, 13(1):e1005273, 2017.
  * [3] R. Clark, S. Wang, H. Wen, A. Markham, and N. Trigoni.  Vinet: Visual-inertial odometry as a sequence-to-sequence learning problem.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
  * [4] A. Cohen, R. I. Ivry, and S. W. Keele.  Attention and structure in sequence learning.  Journal of Experimental Psychology: Learning, Memory, and Cognition, 16(1):17–30, 1990.
  * [5] A. J. Davison.  Futuremapping: The computational structure of spatial ai systems.  arXiv preprint arXiv:1803.11288, 2018.
  * [6] G. M. Edelman.  Neural Darwinism: The theory of neuronal group selection. Basic books, 1987.
  * [7] M. A. Figueiredo, J. M. Leitão, and A. K. Jain.  Unsupervised contour representation and estimation using B-splines and a minimum description length criterion.  IEEE Transactions on Image Processing, 9(6):1075–1087, 2000.
  * [8] A. Franz, V. Gogulya, and M. Löffler.  William: A monolithic approach to agi.  In Artificial General Intelligence: 12th International Conference, AGI 2019, Shenzhen, China, August 6–9, 2019, Proceedings 12, pages 44–58. Springer, 2019.
  * [9] M. Gleicher.  Curves.  In Fundamentals of Computer Graphics, pages 383–428. AK Peters/CRC Press, 2021.
  * [10] A. Graves.  Generating sequences with recurrent neural networks.  arXiv preprint arXiv:1308.0850, 2013.
  * [11] B. Harland, M. Contreras, and J.-M. Fellous.  A role for the longitudinal axis of the hippocampus in multiscale representations of large and complex spatial environments and mnemonic hierarchies.  In A. Stuchlik, editor, The Hippocampus, chapter 4. IntechOpen, Rijeka, 2017.
  * [12] S. Huang, L. Dong, W. Wang, Y. Hao, S. Singhal, S. Ma, T. Lv, L. Cui, O. K. Mohammed, Q. Liu, K. Aggarwal, Z. Chi, J. Bjorck, V. Chaudhary, S. Som, X. Song, and F. Wei.  Language is not all you need: Aligning perception with language models, 2023.
  * [13] X. Huang, G. Mei, J. Zhang, and R. Abbas.  A comprehensive survey on point cloud registration.  arXiv preprint arXiv:2103.02690, 2021.
  * [14] M. Hutter.  Algorithmic information theory.  Scholarpedia, 2(3):2519, 2007.  revision #186543.
  * [15] E. M. Izhikevich.  Polychronization: computation with spikes.  Neural computation, 18(2):245–282, 2006.
  * [16] M. Janner, Q. Li, and S. Levine.  Reinforcement learning as one big sequence modeling problem.  In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021\.
  * [17] J. B. Julian, A. T. Keinath, G. Frazzetta, and R. A. Epstein.  Human entorhinal cortex represents visual space using a boundary-anchored grid.  Nature neuroscience, 21(2):191–194, 2018.
  * [18] M. Kass, A. Witkin, and D. Terzopoulos.  Snakes: Active contour models.  International journal of computer vision, 1(4):321–331, 1988.
  * [19] N. J. Killian, M. J. Jutras, and E. A. Buffalo.  A map of visual space in the primate entorhinal cortex.  Nature, 491(7426):761–764, 2012.
  * [20] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum.  Human-level concept learning through probabilistic program induction.  Science, 350(6266):1332–1338, 2015.
  * [21] M. Lewis, S. Purdy, S. Ahmad, and J. Hawkins.  Locations in the neocortex: a theory of sensorimotor object recognition using cortical grid cells.  Frontiers in neural circuits, 13:22, 2019.
  * [22] A. Macario Barros, M. Michel, Y. Moline, G. Corre, and F. Carrel.  A Comprehensive Survey of Visual SLAM Algorithms.  Robotics, 11(1), 2022.
  * [23] S. P. Mishra, C. G. Singh, and R. Prasad.  A review on compressed pattern matching.  Perspectives in Science, 8:727–729, 2016.
  * [24] V. Osaulenko and D. Ulianych.  Formation of cell assemblies with iterative winners-take-all computation and excitation-inhibition balance.  arXiv preprint arXiv:2108.00706, 2021.
  * [25] R. V. Raju, J. S. Guntupalli, G. Zhou, M. Lázaro-Gredilla, and D. George.  Space is a latent sequence: Structured sequence learning as a unified theory of representation in the hippocampus.  arXiv preprint arXiv:2212.01508, 2022.
  * [26] P. Ramkumar, D. E. Acuna, M. Berniker, S. T. Grafton, R. S. Turner, and K. P. Kording.  Chunking as the result of an efficiency computation trade-off.  Nature communications, 7(1):12176, 2016.
  * [27] S. Sadeh and C. Clopath.  Excitatory-inhibitory balance modulates the formation and dynamics of neuronal assemblies in cortical networks.  Science advances, 7(45):eabg8411, 2021.
  * [28] P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Hammarstrand, F. Kahl, et al.  Back to the feature: Learning robust camera localization from pixels to pose.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3247–3257, 2021.
  * [29] T. Sattler, Q. Zhou, M. Pollefeys, and L. Leal-Taixe.  Understanding the limitations of cnn-based absolute camera pose regression.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3302–3312, 2019.
  * [30] S. Shomstein.  Cognitive functions of the posterior parietal cortex: top-down and bottom-up attentional control.  Frontiers in integrative neuroscience, 6:38, 2012.
  * [31] R. Sun and C. L. Giles.  Sequence learning: From recognition and prediction to sequential decision making.  IEEE Intelligent Systems, 16(4):67–70, 2001.
  * [32] V. Vapnik and R. Izmailov.  Complete statistical theory of learning: learning using statistical invariants.  In Conformal and Probabilistic Prediction and Applications, pages 4–40. PMLR, 2020.
  * [33] L. Wang, M. Amalric, W. Fang, X. Jiang, C. Pallier, S. Figueira, M. Sigman, and S. Dehaene.  Representation of spatial sequences using nested rules in human prefrontal cortex.  NeuroImage, 186:245–255, 2019.
  * [34] J. C. Whittington, D. McCaffary, J. J. Bakermans, and T. E. Behrens.  How to build a cognitive map.  Nature Neuroscience, 25(10):1257–1272, 2022.
  * [35] A. Witkin.  Scale-space filtering: A new approach to multi-scale description.  In ICASSP’84. IEEE International Conference on Acoustics, Speech, and Signal Processing, volume 9, pages 150–153. IEEE, 1984.
  * [36] I. Yildirim and R. A. Jacobs.  Transfer of object category knowledge across visual and haptic modalities: Experimental and computational studies.  Cognition, 126(2):135–148, 2013.
  * [37] I. Yildirim and R. A. Jacobs.  Learning multisensory representations for auditory-visual transfer of sequence category knowledge: a probabilistic language of thought approach.  Psychonomic Bulletin and Review, 22(3):673–686, 2015.


# comparison of user models based on GMM-UBM and i-vectors for speech,
handwriting, and gait assessment of Parkinson’s disease patients

###### Abstract

Parkinson’s disease is a neurodegenerative disorder characterized by the
presence of different motor impairments. Information from speech, handwriting,
and gait signals have been considered to evaluate the neurological state of
the patients. On the other hand, user models based on Gaussian mixture models
- universal background models (GMM-UBM) and i-vectors are considered the
state-of-the-art in biometric applications like speaker verification because
they are able to model specific speaker traits. This study introduces the use
of GMM-UBM and i-vectors to evaluate the neurological state of Parkinson’s
patients using information from speech, handwriting, and gait. The results
show the importance of different feature sets from each type of signal in the
assessment of the neurological state of the patients.

Index Terms—  Parkinson’s disease, GMM-UBM, i-vectors, gait analysis,
handwriting analysis, speech analysis.

##  1 Introduction

Parkinson’s disease (PD) is a neurological disorder characterized by the
progressive loss of dopaminergic neurons in the midbrain, producing several
motor and non-motor impairments [1]. PD affects all of the sub-systems
involved in motor activities like speech production, walking, or handwriting.
The severity of the motor symptoms is evaluated with the third section of the
movement disorder society - unified Parkinson’s disease rating scale (MDS-
UPDRS-III) [2]. The assessment requires the patient to be present at the
clinic, which is expensive and time-consuming because several limitations,
including the availability of neurologist and the reduced mobility of
patients. The evaluation of motor symptoms is crucial for clinicians to make
decisions about the medication or therapy for the patients [3]. The analysis
of signals such as gait, handwriting, and speech helps to assess the motor
symptoms of patients, providing objective information to clinicians to make
timely decisions about the treatment.

Several studies have analyzed different signals such as speech, gait, and
handwriting to monitor the neurological state of the PD patients. Speech was
considered in [4] to predict the MDS-UPDRS-III score of 61 PD patients using
spectral and glottal features. The authors computed the Hausdorff distance
between a speaker from the test set and the speakers in the training set. The
neurological state of the patients was predicted with a Pearson’s correlation
of up to 0.58. In [5] the authors predicted the MDS-UPDRS-III score of 35 PD
patients with features based on articulation and prosody analyses, and a
Gaussian staircase regression. The authors reported moderate Spearman’s
correlations ($\rho$=0.42). Handwriting was considered in [6], to predict the
H&Y score of 33 PD patients using kinematic features and a regression based on
gradient-boosting trees. The H&Y score was predicted with an equal error rate
of 12.5%. Finally, regarding gait features, in [7] the authors predicted a
lower limbs subscore of the MDS-UPDRS-III from 19 PD patients, using several
harmonic and non-linear features, and a support vector regression (SVR)
algorithm. The subscore for lower limbs was predicted with an intra-class
correlation coefficient of 0.78.

According to the literature, most of the related works consider only one
modality. Multimodal analyses, i.e., considering information from different
sensors, have not been extensively studied. In [8] the authors combined
information from statistical and spectral features extracted from handwriting
and gait signals. The fusion of features improved the accuracy of the
classification between PD and healthy control (HC) subjects. Previous studies
[9, 10] suggested that the combination of modalities also improved the
accuracy in the prediction of the neurological state of the patients. This
study proposes the use of different features extracted from speech,
handwriting, and gait to evaluate the neurological state of PD patients. The
prediction is performed with user models based on Gaussian mixture models -
universal background models (GMM-UBM) and i-vectors. To the best of our
knowledge, this is one of the few studies for multimodal analysis of PD
patients, and the first one that considers multimodal user models to evaluate
the neurological state of the patients.

##  2 Methods

The methods used in this study are summarized in Figure 1. Speech,
handwriting, and gait signals are characterized using different feature
extraction strategies. Then, data from HC subjects are used to train user
models based on GMM-UBM and i-vector systems. For the case of the GMM-UBM,
data from PD patients were used to adapt the UBMs into GMMs, creating a
specific GMM for each patient. On the other hand, for the i-vector modeling, a
reference i-vector was created with data from HC subjects with similar age and
gender of the patients, thus i-vectors extracted from the patients can be
compared with a personalized reference model. Finally, distance measures are
computed between the reference models and those adapted/extracted from the PD
patients. The computed distance is correlated with the neurological state of
the patients based on the MDS-UPDRS-III scale.

Fig. 1: General methodology followed in this study.

###  2.1 Speech features

Phonation: these features model abnormal patterns in the vocal fold vibration.
Phonation features are extracted from the voiced segments. The feature set
includes descriptors computed for 40 ms frames of speech, including jitter,
shimmer, amplitude perturbation quotient, pitch perturbation quotient, the
first and second derivatives of the fundamental frequency
$F\raisebox{-1.72218pt}{\scriptsize 0}$, and the log-energy [11].

Articulation: these features model aspects related to the movements of limbs
involved in the speech production. The features considered the energy content
in onset segments [11]. The onset detection is based on the computation of
$F\raisebox{-1.72218pt}{\scriptsize 0}$. Once the border between unvoiced and
voiced segments is detected, 40 ms of the signal are taken to the left and to
the right, forming a segment with 80 ms length. The spectrum of the onset is
distributed into 22 critical bands according to the Bark scale, and the Bark-
band energies (BBE) are calculated. 12 MFCCs and their first two derivatives
are also computed in the transitions to complete the feature set.

Prosody:  for these features, the log-$F\raisebox{-1.72218pt}{\scriptsize 0}$
and the log-energy contours of the voiced segments were approximated using
Lagrange polynomials of order $P=5$. A 13-dimensional feature vector is formed
by concatenating the six coefficients computed from the
log-$F\raisebox{-1.72218pt}{\scriptsize 0}$ and the log-energy contours, in
addition to the duration of the voiced segment [12]. The aim of these features
is to model speech symptoms such as monotonicity ad mono-loudness in the
patients.

Phonological:  these features are represented by a vector with interpretable
information about the placement and manner of articulation. The different
phonemes of the Spanish language are grouped into 18 phonological posteriors.
The phonological posteriors were computed with a bank of parallel recurrent
neural networks to estimate the probability of occurrence of a specific
phonological class [13].

###  2.2 Handwriting features

Handwriting features are based on the trajectory of the strokes in vertical,
horizontal, radial, and angular positions. We computed the velocity and
acceleration of the strokes in the different axes, in addition to the pressure
of the pen, the azimuth angle, the altitude angle, and their derivatives.
Finally, we considered features based on the in-air movement before the
participant put the pen on the tablet’s surface. Additional information of the
features can be found in [14].

###  2.3 Gait features

Harmonic: these features model the spectral wealth and the harmonic structure
of the gait signals obtained from the inertial sensors. We computed the
continuous wavelet transform with a Gaussian wavelet. The feature set is
formed with the energy content in 8 frequency bands from the scalogram, three
spectral centroids, the energy in the in the 1st, 2nd, and 3rd quartiles of
the spectrum, the energy content in the locomotor band (0.5–3 Hz), the energy
content in the freeze band (3–8 Hz), and the freeze index, which is the ratio
between the energy in the locomotor and freeze bands [15, 16].

Non-linear:  gait is a complex and non-linear activity that can be modeled
with non-linear dynamics features. The first step to extract those features is
the phase space reconstruction, according to the Taken’s theorem. Different
features can be extracted from the reconstructed phase space to assess the
complexity and stability of the walking process. The extracted features
include the correlation dimension, the largest Lyapunov exponent, the Hurst
exponent, the detrended fluctuation analysis, the sample entropy, and the
Lempel-Ziv complexity [17].

###  2.4 User models based on GMM-UBM

GMM-UBM systems were proposed recently to quantify the disease progression of
PD patients [18]. We propose to extend the idea to multimodal GMM-UBM systems.
The main hypothesis is that speech, handwriting, or gait impairments of PD
patients can be modeled by comparing a GMM adapted for a patient with a
reference model created with recordings from HC subjects. GMMs represent the
distribution of feature vectors extracted from the different signals from a
single PD patient. When the GMM is trained using features extracted from a
large sample of subjects, the resulting model is a UBM. The model for each PD
patient is derived from the UBM by adapting its parameters following a maximum
a posteriori process. Then, the neurological state of the patients is
estimated by comparing the adapted model with the UBM using a distance
measure. We use the Bhattacharyya distance, which considers differences in the
mean vectors and covariances matrices between the UBM and the user model [19].

###  2.5 User models based on i-vectors

I-vectors are used to transform the original feature space into a low-
dimensional representation called total variability space via joint factor
analysis [20]. For speech signals, such a space models the inter- and intra-
speaker variability, in addition to channel effects. For this study we aim to
capture changes in speech, handwriting, and gait due to the disease [21].
I-vectors have been considered previously to model handwriting [22] and gait
data [23]. Similar to the GMM-UBM systems, we train the i-vector extractor
with data from HC subjects, and compute a reference i-vector to represent
healthy speech, handwriting, or gait. Then, we extract i-vectors from PD
patients, and compute the cosine distance between the patient i-vector and the
reference.

##  3 Data

We considered an extended version of the PC-GITA corpus [24]. This version
contains speech, handwriting, and gait signals, collected from 106 PD patients
and 87 HC subjects. All of the subjects are Colombian Spanish native speakers.
The patients were labeled according to the MDS-UPDRS-III scale. Table 1
summarizes clinical and demographic aspects of the participants included in
the corpus.

Table 1: Clinical and demographic information of the subjects. [F/M]:
Female/Male. Average(Standard deviation). TD: Time since diagnosis. TD and age
are given in years.

| PD patients | HC subjects
---|---|---
Gender [F/M] | 49/57 | 43/44
Age [F/M] | 60.9(11.2)/64.7(9.4) | 61.4(9.8)/64.9(10.5)
TD [F/M] | 15.5(14.5)/8.1(5.9) | –
MDS–UPDRS–III [F/M] | 36.2(18.1)/36.3(18.9) | –

Speech signals were recorded with a sampling frequency of 16 kHz and 16-bit
resolution. The same speech tasks recorded in the PC-GITA corpus [24], except
for the isolated words, are included in this extended version. Handwriting
data consist of online drawings captured with a tablet Wacom cintiq 13-HD with
a sampling frequency of 180 Hz. The tablet captures six different signals:
x-position, y-position, in-air movement, azimuth, altitude, and pressure. The
subjects performed a total of 14 exercises divided into writing and drawing
tasks. Additional information about the handwriting exercises can be found in
[10]. Gait signals were captured with the eGaIT system, which consists of a
3D-accelerometer (range $\pm$6g) and a 3D gyroscope (range $\pm$500∘/s)
attached to the external side (at the ankle level) of the shoes [25]. Data
from both feet were captured with a sampling frequency of 100 Hz and 12-bit
resolution. The exercises included 20 meters walking with a stop after 10
meters, 40 meters walking with a stop every 10 meters, _heel-toe tapping_ ,
and the _time up and go_ test.

##  4 Experiments and results

Data from HC subjects were used to train the UBMs and the i-vector extractors.
For the GMM-UBM system, data from PD patients were used to adapt the UBMs into
GMMs. The Bhattacharya distance is used to compare the GMM and the UBM. For
the i-vectors, a reference was created by averaging the i-vectors extracted
from HC subjects that have same gender and similar age of the patients (in a
range of $\pm$ 2 years). I-vectors extracted from PD patients are compared to
the reference i-vector using the cosine distance. The computed distances are
correlated with the MDS-UPDRS-III score of the patients.

###  4.1 User models from different modalities

The correlation between the neurological state of the patients and the user
models based on GMM-UBM and i-vectors is shown in Table 2 for the speech,
handwriting, and gait features. The results indicate that for gait and speech
signals, the user models based on GMM-UBM systems are more accurate than the
i-vectors. This can be explained because the distance between the adapted GMM
and the UBM considers more information about the statistical distribution of
the population than for the case of i-vectors, where the reference for healthy
subjects is reduced to a single vector. A “strong” correlation is obtained
with the harmonic features (gait analysis) modeled with the GMM-UBM system
($\rho$=0.619). This is expected because most of the items used by the
neurologist in the MDS-UPDRS-III are based on the movement of the lower limbs.
For handwriting features, “weak” correlations are obtained both with the GMM-
UBM and the i-vector systems. The correlations obtained with speech features
are not robust to model the general neurological state of the patients. This
result can be explained because the MDS-UPDRS-III is a complete neurological
scale that consider speech impairments in only one of the 33 items of the
total scale [2].

Table 2: Correlation between the MDS-UPDRS-III scale of PD patients and the
user models based on GMM-UBM and i-vectors extracted using different feature
sets. r: Pearson’s correlation coefficient, $\rho$: Spearman’s correlation
coefficient.

|  | GMM-UBM | I-vector
---|---|---|---
Modality | Features | r | $\mathbf{\rho}$ | r | $\mathbf{\rho}$
Gait | Harmonic | 0.436 | 0.619 | 0.301 | 0.354
Non-linear | 0.262 | 0.312 | 0.345 | 0.381
Handwriting | Kinematic | 0.339 | 0.261 | 0.237 | 0.346
Speech | Phonation | 0.176 | 0.198 | 0.279 | 0.244
Articulation | 0.179 | 0.195 | 0.225 | 0.213
Prosody | 0.067 | 0.202 | 0.191 | 0.190
Phonological | 0.266 | 0.298 | 0.212 | 0.197

###  4.2 Multimodal user models

The user models extracted from all feature sets using the GMM-UBM system were
combined by concatenating the distance between the user model and the UBM for
each feature set. A linear regression was trained with the matrix of distances
to predict the MDS-UPDRS-III scale. The model was trained following a leave
one subject out cross-validation strategy. The results of the fusion are shown
in Table 3. The Spearman’s correlation increases by 2.4%, absolute with
respect to the one obtained only with the harmonic features. Additional
regression algorithms based on random forest regression or SVRs were
considered, however, they overfitted the test set, predicting only the mean
value of the total scale.

Table 3: Prediction of the MDS-UPDRS-III combining information of the user
models based on GMM-UBM. r: Pearson’s correlation coefficient, $\rho$:
Spearman’s correlation coefficient. MAE: median absolute error.

| $\rho$ | p-val | r | p-val | MAE
---|---|---|---|---|---
Fusion of features | 0.634 | $\ll 10^{-10}$ | 0.516 | $\ll 10^{-7}$ | 10.5

Figure 2 shows the error in the prediction of the MDS-UPDRS-III score of the
patients. Most of the patients are in initial or intermediate state of the
disease (10$<$MDS-UPDRS-III$<$50), and they were predicted with the same
distribution. The outlayer in the top of the figure corresponds to the eldest
patient in the corpus. The patient has a score of 53, and it was predicted
with a score of 78. Although the intermediate value of the MDS-UPDRS-III score
of the patient, his MDS-UPDRS-speech item is 4, i.e., the patient was
completely unable to speak, which highly affected his speech features.

Fig. 2: Prediction of the MDS-UPDRS-III score using multimodal user models
based on GMM-UBM systems.

Figure 3 shows the contribution of each feature set to the multimodal user
model. Each bar indicates the coefficient for the linear regression associated
to each feature set. Harmonic features were the most important for the
multimodal model, followed by prosody and articulation features, which have
shown to be the most important features to evaluate the dysarthria associated
to PD [9]. Handwriting features were less important than expected; however,
this fact can be explained because the extracted features are based on a
standard kinematic analysis that might not be completely related to the
symptoms associated with PD. The results for handwriting could be improved
with a feature set more related to the handwriting impairments of the
patients, like those based on a neuromotor analysis [26].

Fig. 3: Contrbution of each feature set to the multimodal user model system.

##  5 Conclusion

The present study compared user models based on GMM-UBM and i-vectors to
evaluate the neurological state of PD patients using information from speech,
handwriting, and gait. Different features were extracted from each bio-signal
to model different dimensions of PD symptoms. Gait features were the most
accurate to model the general neurological state of the patients, however, the
combination of different bio-signals improved the correlation of the proposed
method. In addition, user models based on GMM-UBM were more accurate than
those based on i-vectors. Better results could be obtained with the i-vector
system if more training data from HC subjects were available to create the
reference model, especially for handwriting and gait. Further studies will
consider additional features to model other aspects of PD symptoms, especially
from handwriting signals. At the same time, additional models based on
representation learning, and additional fusion methods can be considered to
evaluate the neurological state of the patients.

##  6 Acknowledgments

This project received funding from the EU Horizon 2020 research and innovation
programme under the Marie Sklodowska-Curie Grant Agreement No. 766287, and
from CODI from University of Antioquia by Grant No. 2017–15530.

## References

  * [1] O. Hornykiewicz,  “Biochemical aspects of Parkinson’s disease,”  Neurology, vol. 51, no. 2, pp. S2–S9, 1998.
  * [2] C. Goetz et al.,  “Movement Disorder Society-sponsored revision of the Unified Parkinson’s Disease Rating Scale (MDS-UPDRS): Scale presentation and clinimetric testing results,”  Movement Disorders, vol. 23, no. 15, pp. 2129–2170, 2008.
  * [3] S. Patel et al.,  “Home monitoring of patients with Parkinson’s disease via wearable technology and a web-based application,”  in EMBC, 2010, pp. 4411–4414.
  * [4] M. Tu, V. Berisha, and J. Liss,  “Objective assessment of pathological speech using distribution regression,”  in Proc. ICASSP. IEEE, 2017, pp. 5050–5054.
  * [5] K. M. Smith et al.,  “Vocal markers of motor, cognitive, and depressive symptoms in Parkinson’s disease,”  in Proc. ACCI. IEEE, 2017, pp. 71–78.
  * [6] J. Mucha et al.,  “Identification and Monitoring of Parkinson’s Disease Dysgraphia Based on Fractional-Order Derivatives of Online Handwriting,”  Applied Sciences, vol. 8, no. 12, pp. 1–18, 2018.
  * [7] S. Aghanavesi et al.,  “Motion sensor-based assessment of Parkinson’s disease motor symptoms during leg agility tests: results from levodopa challenge,”  Journal of Biomed. and Health Inf. (in press), 2019.
  * [8] J. Barth, M. Sünkel, et al.,  “Combined analysis of sensor data from hand and gait motor function improves automatic recognition of parkinson’s disease,”  in Proc. EMBC, 2012, pp. 5122–5125.
  * [9] J. C. Vásquez-Correa, J. R. Orozco-Arroyave, et al.,  “Multi-view representation learning via GCCA for multimodal analysis of parkinson’s disease,”  in Proc. ICASSP. IEEE, 2017, pp. 2966–2970.
  * [10] J. C. Vasquez-Correa, T. Arias-Vergara, et al.,  “Multimodal assessment of parkinson’s disease: A deep learning approach,”  IEEE Journal of Biomed. and Health Inf., vol. 23, no. 4, pp. 1618–1630, 2019.
  * [11] J. R. Orozco-Arroyave et al.,  “Neurospeech: An open-source software for parkinson’s speech analysis,”  Digital Signal Processing, vol. 77, pp. 207–221, 2018.
  * [12] N. Dehak et al.,  “Modeling Prosodic Features With Joint Factor Analysis for Speaker Verification,”  IEEE Trans. on Audio, Speech and Language Processing, vol. 15, no. 7, pp. 2095–2103, 2007.
  * [13] J. C. Vásquez-Correa et al.,  “Phonet: A tool based on gated recurrent neural networks to extract phonological posteriors from speech,”  in INTERSPEECH, 2019, pp. 549–553.
  * [14] C. D. Rios-Urrego et al.,  “Analysis and evaluation of handwriting in patients with parkinson’s disease using kinematic, geometrical, and non-linear features,”  Computer methods and programs in biomedicine, vol. 173, pp. 43–52, 2019.
  * [15] H. Zach et al.,  “Identifying freezing of gait in Parkinson’s disease during freezing provoking tasks using waist-mounted accelerometry,”  Parkinsonism & related disorders, vol. 21, no. 11, pp. 1362–1366, 2015.
  * [16] S. Rezvanian et al.,  “Towards real-time detection of freezing of gait using wavelet transform on wireless accelerometer data,”  Sensors, vol. 16, no. 4, pp. 475, 2016.
  * [17] P. A. Pérez-Toro, J. C. Vásquez-Correa, et al.,  “A non-linear dynamics approach to classify gait signals of patients with parkinson’s disease,”  in Workshop on Engineering Applications, 2018, pp. 268–278.
  * [18] T. Arias-Vergara et al.,  “Speaker models for monitoring parkinson’s disease progression considering different communication channels and acoustic conditions,”  Speech Communication, vol. 101, pp. 11–25, 2018.
  * [19] C. H. You, K. A. Lee, and H. Li,  “Gmm-svm kernel with a bhattacharyya-based distance for speaker recognition,”  IEEE Trans.s on Audio, Speech, and Language Processing, vol. 18, no. 6, pp. 1300–1312, 2010.
  * [20] N. Dehak et al.,  “Front-End Factor Analysis for Speaker Verification,”  IEEE Trans. on Audio, Speech, and Language Proc., vol. 19, no. 4, pp. 788–798, may 2011.
  * [21] N. Garcia et al.,  “Multimodal i-vectors to detect and evaluate parkinson’s disease,”  Proc. INTERSPEECH 2018, pp. 2349–2353, 2018.
  * [22] V. Christlein,  Handwriting Analysis with Focus on Writer Identification and Writer Retrieval,  Logos-Verlag, 1st edition, 2018.
  * [23] R. San-Segundo et al.,  “I-vector analysis for gait-based person identification using smartphone inertial signals,”  Perv. and Mobile Comp., vol. 38, pp. 140–153, 2017.
  * [24] J. R. Orozco-Arroyave et al.,  “New Spanish speech corpus database for the analysis of people suffering from Parkinson’s disease,”  in LREC, 2014, pp. 342–347.
  * [25] J. Barth et al.,  “Biometric and mobile gait analysis for early diagnosis and therapy monitoring in Parkinson’s disease,”  in Proc. EMBC, 2011, pp. 868–871.
  * [26] D. Impedovo,  “Velocity-based signal features for the assessment of parkinsonian handwriting,”  IEEE Signal Processing Letters, vol. 26, no. 4, pp. 632–636, 2019\.


# Benchmarking Robustness to Text-Guided Corruptions

Mohammadreza Mofayezi , Yasamin Medghalchi11footnotemark: 1
Sharif University of Technology
indicates equal contribution

###### Abstract

This study investigates the robustness of image classifiers to text-guided
corruptions. We utilize diffusion models to edit images to different domains.
Unlike other works that use synthetic or hand-picked data for benchmarking, we
use diffusion models as they are generative models capable of learning to edit
images while preserving their semantic content. Thus, the corruptions will be
more realistic and the comparison will be more informative. Also, there is no
need for manual labeling and we can create large-scale benchmarks with less
effort. We define a prompt hierarchy based on the original ImageNet hierarchy
to apply edits in different domains. As well as introducing a new benchmark we
try to investigate the robustness of different vision models. The results of
this study demonstrate that the performance of image classifiers decreases
significantly in different language-based corruptions and edit domains. We
also observe that convolutional models are more robust than transformer
architectures. Additionally, we see that common data augmentation techniques
can improve the performance on both the original data and the edited images.
The findings of this research can help improve the design of image classifiers
and contribute to the development of more robust machine learning systems. The
code for generating the benchmark is available at
https://github.com/ckoorosh/RobuText.

##  1 Introduction

Image classifiers are widely used in various applications such as object
recognition, medical diagnosis, and autonomous driving. These systems are
designed to classify images accurately and reliably, which requires them to be
robust to various sources of noise and corruption [21, 9, 1].

However, recent studies have shown that image classifiers can be vulnerable to
small corruptions, where subtle changes to an image can significantly impact
the classifier’s performance [9]. This issue has raised concerns about the
robustness of image classifiers and their reliability in real-world scenarios.
Also, current benchmarks and training datasets do not cover all possible real-
world situations such as weather changes, color and texture variation, or
context changes [27].

In this paper, we investigate the robustness of image classifiers to text-
guided corruptions using diffusion models to create image edits. Diffusion
models are a type of generative model that can learn to edit images while
preserving their semantic content. We use these models to create various text-
guided corruptions in different domains and evaluate the performance of
different image classifiers under these corruptions.

Figure 1: Current image classifiers are not robust to domain changes. We can
see that when we evaluate these models on edited images the performance drops
significantly. For this, we introduce a new benchmark that includes text-
guided edited images in 5 different domains to assess and improve models’
performance.

The main objective of this study is to conduct a comprehensive benchmark of
the performance of image classifiers under various language-based corruptions.
We evaluate the classifiers’ performance in different domains. The results of
this study will provide insights into the robustness of image classifiers to
language-based corruptions and identify the limitations of existing image
classification systems. We observe that convolutional models are more robust
than transformers in different domains. We can also see that increasing the
complexity usually improves robustness. Moreover, we check the effect of the
different domains of prompts for editing on the robustness of classifiers. The
results show that their accuracies fall more significantly in the drawing
domain in comparison to others. Figuring out which domain of prompts is tough
for handling for classifiers can give the hint for designing a new
augmentation technique based on language-based models.

The remainder of this paper is organized as follows. Section 2 reviews related
work in the field of image classification and robustness to corruption.
Section 3 describes the methodology used to create the benchmark in this
study, including the dataset and the prompts hierarchy used to create
language-based corruptions and the image classifiers evaluated. Section 4
presents the experimental results and analyses the performance of different
image classifiers under various language-based corruptions. Finally, Section 5
concludes the paper, and Section 6 discusses future directions for research in
this field.

Our contributions are in summary: 1) Creating a novel text-guided benchmark
for evaluating the robustness of different vision models. 2) Using five
domains as a testbed to evaluate image classifier performance, to gain a
better understanding of where the models struggle most, and 3) Studying the
effect of two common data augmentation techniques on robustness in different
domains.

##  2 Related Work

Diffusion Models. Diffusion models are new generative models which produce
images based on a forward and reverse process. In the forward process, we add
a gaussian noise with certain variance and zero mean to the image in each
step. The noisy image in step $t$, $x_{t}$, is produced as follow:

$x_{t}=\sqrt{\alpha_{t}}x_{0}+\sqrt{1-\alpha_{t}}\epsilon,\quad\epsilon~{}\sim\mathcal{N}(0,1).$
(1)

$x_{0}$ is a clean image and $\epsilon$ is an additive noise. $\alpha_{t}$ is
defined as :

$\alpha_{t}=\prod_{s=1}^{t}1-\beta_{s}.$ (2)

$\beta_{t}$ can be a learned or fixed variance scheduler. In the training
procedure, the model tries to predict the noise added in the forward process.
In the reverse process, the predicted noise is subtracted from the noisy image
in each step [13].

Null-text Inversion. Null-text Inversion model [20] is a text-guided diffusion
model that can edit and manipulate an image using text only. This model needs
source and target prompts to produce an edited image. This model has two main
contributions which are pivotal inversion for diffusion models and null-text
optimization.

Most inversion works use random noise vectors for optimization while null-text
performs local optimization and uses a pivotal noise vector instead, which
illustrates more efficient inversion. Since they utilize Stable Diffusion
[23], they require applying classifier-free guidance which has a trade-off
between editability and being a good approximation. As a result, the guidance
scale is fixed to 1 as their pivot trajectory and the optimization starts
around it with a guidance scale larger than 1.

Null-text is inspired by the prompt-to-prompt model [12]. Instead of using
default embedding for null text, they optimize it for each input image that
can be used for multiple editing scenarios. This process is applied for each
time step, $t$, and an optimized result of the previous step, $t+1$, is used
as a starting point [20].

Benchmarking Robustness. Recent work focuses on two approaches for creating
robustness benchmarks: Synthetic data which means utilizing synthetic images
to test the robustness of neural networks and Hand-collected data which means
focusing on collecting real-world images.

Recently, there has been considerable effort devoted to testing the robustness
of neural networks by employing synthetic images. One such example is
ImageNet-C [9], which assesses how neural networks perform on images with
synthetic noise such as Gaussian noise, motion blur, zoom blur, and JPEG
compression. To accomplish this, the standard ImageNet [2] validation set is
modified with these noises. In addition to perturbing images with these
processing pipeline noises, other research [5] has evaluated the shape and
texture bias of deep neural networks by using images with artificially
overwritten textures. Augmenting images with style transfer [4, 5] or by
taking a linear combination of augmented images and the original images [10]
has been found to be a beneficial way of enhancing the robustness to these
synthetic image noises or texture alterations. Nevertheless, these evaluations
are limited because synthetic image perturbations are unable to accurately
replicate real-world scenarios. [27] demonstrates that style transfer and
strong augmentation are not helpful in dealing with changes in shape and pose.

Other recent studies have focused on using real-world images to evaluate the
performance of deep neural network (DNN) models, as distribution shifts in
these images can have a significant impact on the robustness of these models.
For example, ImageNet-V2 [22], which gathered images from Flickr to create a
new test set for ImageNet [2], found that DNN model performance decreased when
faced with these real-world images. Similarly, ImageNet-A [11] utilized an
adversarial filtration technique to collect a new test set of images using
ResNet-50 [6] that caused the performance to drop significantly when
transferred to other architectures. However, ImageNet-A [11] was unable to
isolate the specific factors causing the decrease in performance. More
recently, ImageNet-R [8] collected four out-of-distribution testing benchmarks
that incorporated shifts in texture, geo-location, camera parameters, and
blur, and found that no single technique could improve model performance
across all these factors. Other benchmarks have also been developed to
evaluate how well DNN models can learn invariant features from unbalanced
datasets [25], as well as benchmarks composed of real-world image shifts [16].

##  3 Text-guided Robustness Benchmark

We utilize the null-text inversion [20] model to edit ImageNet [2] images with
our prompts. Our text-guided benchmark is constructed by edited images in
different domains. These domains are Drawing, Weather, Color, Texture, and
Context.

We will introduce the prompts hierarchy of each domain in the next section.

###  3.1 Dataset

ImageNet [2] is one of the largest benchmarks for image classification. The
ImageNet-1K dataset consists of 1000 object classes with 50K validation
images. The classes are based on a hierarchical structure provided by WordNet.
Each meaningful word or phrase in Wordnet, called ”synset”, has about a
thousand images on average in the ImageNet dataset. For the benchmark, the
validation set of the ILSVRC-2012 dataset is used.

###  3.2 ImageNet Hierarchy

First, we needed to divide the ImageNet classes into more general sub-classes
and define prompts for each subclass differently to generate more meaningful
images. For instance, changing a texture to metal makes sense for tools not
for animals. The super-class and sub-classes are shown in Table 1.

Super Class | Sub-class | Index
---|---|---
Organism | Animal | 1
Plant | 2
Person | 3
Artifact | Vehicle | 4
Furniture | 5
Tool | 6
Food | 7
Geological Formation | Structure | 8
Landscape | 9
Table 1: The ImageNet [2] hierarchy used for defining the sub-classes.

###  3.3 Prompt Hierarchy

One of the main problems in recent text-guided models is that all target
prompts cannot be applied to whole images; so, it is necessary to find prompts
that have good results on most of the images in that specific subclass. To
overcome this problem, we introduce a set of hand-engineered prompts for
applying the edits to the images. To have a sight of what happened to each
image and be able to analyze the impact of each category of prompts, we define
5 domains for editing prompts. It is noteworthy to mention that if the editing
prompts can not be performed, the image doesn’t convert to any damaged or
different image. This point is vital for using this process without prompt
engineering that can assure users that the originality of images is kept safe.
The full prompt hierarchy is shown in Table 2.

Domain | Prompt | Sub-class
---|---|---
Drawing | A watercolor painting of a [class] | 1-2-3-4-5-6-7-8-9
An antique sketch of a [class] | 1-2-3-4-5-6-7-8-9
A pencil drawing of a [class] | 1-2-3-4-5-6-7-8-9
A sketch with crayon of a [class] | 1-2-3-4-5-6-7-8-9
A [class] in Monet style | 1-2-3-4-5-6-7-8-9
A [class] in starry night style | 1-2-3-4-5-6-7-8-9
Weather | A [class] in the snow | 1-2-3-4-5-6-7-8-9
A [class] on ice | 1
A [class] in the fog | 1-2-3-4-5-6-7-8-9
A [class] in the rain | 1-2-3-4-5-6-7-8-9
Color | A [color name] [class] | 1-2-3-4-5-6-7-8-9
Texture | A metal [class] | 5-6
A wooden [class] | 5-6-8
A glass [class] | 8
A brick [class] | 8
A golden [class] | 8
Context | A [class] on the water | 1-2
A [class] on the moon | 1-2-3-4-6
A [class] on the mars | 1-2-3-4-6
A [class] in the desert | 1-2-3-4-6
A [class] on the cloth | 2-7
A [class] on the glass | 2-7

Table 2: The prompt hierarchy for editing images in the five mentioned
domains. In each of the five domains, we introduce specific prompts for
different subclasses.

##  4 Experiments

We feed the edited images to multiple classifiers to calculate their error in
finding the correct classes. Since we want to have an interpretation of which
model is more robust or to which prompt they are sensitive, we get the results
of each classifier on some random images of each super-class with a random
prompt of each class of prompts.

Model | Original | Color | Context | Drawing | Weather | Texture
---|---|---|---|---|---|---
AlexNet [17] | 55.6 | 37.9 | 44.1 | 27.4 | 40.3 | 37.1
SqueezeNet v1.0 [15] | 58.0 | 38.8 | 46.3 | 26.6 | 43.8 | 34.9
SqueezeNet v1.1 [15] | 58.5 | 40.8 | 45.7 | 27.0 | 44.5 | 37.1
VGG-11 [24] | 66.2 | 48.2 | 53.6 | 31.5 | 51.3 | 46.6
VGG-19 [24] | 70.0 | 52.3 | 55.7 | 34.6 | 53.2 | 47.7
VGG-19+BN [24] | 70.2 | 53.9 | 58.9 | 38.2 | 56.5 | 50.9
DenseNet-121 [14] | 72.1 | 53.9 | 62.0 | 41.9 | 58.4 | 51.4
DenseNet-169 [14] | 73.6 | 56.3 | 63.1 | 44.3 | 57.7 | 52.0
DenseNet-201 [14] | 75.2 | 57.5 | 63.7 | 44.7 | 60.2 | 52.6
ResNet-18 [7] | 67.9 | 51.5 | 56.5 | 40.7 | 54.1 | 46.9
ResNet-34 [7] | 72.1 | 52.9 | 59.1 | 40.7 | 57.4 | 50.9
ResNet-50 [7] | 73.3 | 55.0 | 60.7 | 40.0 | 58.1 | 50.3
ResNet-101 [7] | 79.5 | 61.1 | 69.7 | 50.0 | 66.9 | 61.1
ResNet-152 [7] | 81.4 | 63.5 | 70.0 | 49.8 | 68.8 | 62.3
ResNeXt-50 [26] | 80.1 | 62.6 | 67.9 | 48.6 | 64.2 | 60.3
ResNeXt-101 [26] | 80.1 | 64.9 | 70.9 | 53.2 | 67.0 | 61.7
ResNeXt-101 64x4d [26] | 81.2 | 65.7 | 70.5 | 52.0 | 68.2 | 64.9
ViT-B/16 [3] | 80.2 | 61.1 | 69.3 | 50.1 | 67.4 | 56.6
ViT-B/32 [3] | 74.1 | 57.8 | 64.5 | 48.3 | 60.6 | 55.1
ViT-L/16 [3] | 78.7 | 60.9 | 64.3 | 47.9 | 62.0 | 55.7
ViT-L/32 [3] | 77.3 | 60.6 | 66.0 | 50.2 | 62.1 | 56.6
ConvNeXt-B [19] | 82.1 | 65.7 | 72.4 | 50.8 | 69.3 | 63.1
Swin-B [18] | 82.5 | 63.7 | 66.5 | 47.7 | 66.9 | 60.3
Swin-B v2 [18] | 82.6 | 64.5 | 69.3 | 47.7 | 67.9 | 58.9
Table 3: Performance of different vision models for image classification on
different domains. We report the top-1 accuracy on the original data and the
edited images in different domains.

Experimental Setup. We pick some classes from each super-class mentioned in
Table 1 and for each class, we choose 10 images randomly. In total, we have
1000 images for this experiment. Since we select them randomly, we can assume
that their results are a good representative of a whole test set. We pick a
randomly suitable prompt from each domain mentioned in Table 2 for each image.

Baselines. Different image classifiers used in this study are based on
convolutional, and transformer layers. In the following, we intend to compare
the results of AlexNet [17], SqueezeNet [15], VGG [24], DenseNet [14], ResNet
[7], ResNeXt [26], ViT [3], ConvNeXt [19], and Swin-Transformer [18]. For
SqueezeNet [15], we use both version 1.0 and 1.1 of the model. (SqueezeNet
v1.1 model requires 2.4x less computation than SqueezeNet v1.0 without
diminishing accuracy.) For VGGNets [24], we chose VGG-11, VGG-19, and VGG-19
with batch normalization. For ResNeXt [26], we evaluate models with 50 and 101
layers and also the model with increased cardinality and width named
ResNeXt-101 64x4d. And finally, we chose the base and large models of ViT [3]
and the base model for ConvNeXt [19], and Swin-Transformer [18]. We evaluate
the performance of classifiers using the Top-1 Accuracy.

###  4.1 Effect of Architecture on Robustness

Although the Swin-Transformer models [18] have a better performance on the
original data, the ConvNeXt [19] model, the ResNeXt [26] models, and deep
ResNet [7] models perform better on edited images.

In VGGNets [24], we can see that increasing the layers results in more
accuracy, and using batch normalization makes the model more robust. In
ResNets [7], increasing the layers makes the model more accurate and also more
robust to different domains. In ResNeXts [26], the model with increased
cardinality and width (ResNeXt-101 64x4d) is more robust to texture and
weather edits. Vision Transformers [3] have competitive accuracy on original
data but they are less robust than other architectures, especially in color
and texture changes.

Model | Original | Color | Context | Drawing | Weather | Texture
---|---|---|---|---|---|---
ResNet-50 [7] | 73.3 | 55.0 | 60.7 | 40.0 | 58.1 | 50.3
ResNet-50 [7] \+ Style Transfer [5] | 71.5 | 55.4 | 62.9 | 43.6 | 60.1 | 51.7
ResNet-50 [7] \+ AugMix [10] | 75.6 | 57.7 | 64.5 | 48 | 62.4 | 55.1
Table 4: Effect of data augmentation techniques on robustness. We can see that
the style transfer method [5] has improved robustness in the drawing domain
more significantly. AugMix [10] has improved the performance both on the
original data and on the edited data. Figure 2: Robustness of various image
classifiers in different domains. We can see that all models struggle in the
’Drawing’ domain. We define robustness as the relative accuracy on the edited
data and the original data, divided by the accuracy on the original data.

###  4.2 Effect of Data Augmentation on Robustness

Previous research has shown that data augmentation techniques can improve the
robustness of image classifiers [5, 10]. Stylizing images with artistic
textures [5] and blending the original image with a highly augmented one
(known as AugMix) [10] are examples of such data augmentation techniques.

We can see the effect of training a ResNet-50 [7] with both methods in Table
4. Using style transfer as augmentation improves the overall performance in
different domains and the effect is most in the ’Drawing’ domain. AugMix [10],
has a better performance in all of the domains including the original data.

###  4.3 Effect of Domains on Robustness

In this section, we aim to observe for each class what kind of manipulation is
riskier in the image classification task. According to Figure 2, all domains
decrease the accuracy of classifiers but the drawing domain is the most
difficult domain for classifiers. This interpretation is advantageous for
designing a new augmentation technique based on drawing domains and also
comparing the performance of various classifiers in this tough domain in the
object detection task. Another interesting point is that although different
models have dissimilar accuracy results, they have the same behavior about
different domains, and the order of hardness of domains is approximately alike
for all of them.

###  4.4 Ablation Study

We also evaluated the models’ performance on the animal subclass of the
ImageNet-1K dataset. The results are provided in Table 5. We chose 10 classes
from the subclass and 10 images per class.

In ResNets [7], increasing the layers makes the model more accurate and robust
to different domains. Similar to Table 3, we can see that batch normalization
has made VGGNet more robust.

Model | Original | Color | Context | Drawing | Weather
---|---|---|---|---|---
AlexNet [17] | 60 | 40 | 42 | 26 | 51
SqueezeNet v1.0 [15] | 60 | 37 | 57 | 36 | 55
SqueezeNet v1.1 [15] | 64 | 39 | 52 | 30 | 54
VGG-11 [24] | 80 | 60 | 57 | 39 | 71
VGG-19 [24] | 82 | 58 | 62 | 39 | 69
VGG-19+BN [24] | 82 | 63 | 72 | 45 | 72
DenseNet-121 [14] | 85 | 59 | 71 | 45 | 70
DenseNet-169 [14] | 87 | 58 | 77 | 52 | 77
DenseNet-201 [14] | 83 | 62 | 71 | 47 | 67
ResNet-18 [7] | 76 | 52 | 63 | 42 | 65
ResNet-34 [7] | 78 | 52 | 65 | 41 | 67
ResNet-50 [7] | 85 | 60 | 68 | 50 | 68
ResNet-101 [7] | 88 | 70 | 81 | 63 | 75
ResNet-152 [7] | 89 | 72 | 78 | 63 | 79
ResNeXt-50 [26] | 88 | 68 | 79 | 56 | 78
ResNeXt-101 [26] | 88 | 79 | 79 | 64 | 82
ResNeXt-101 64x4d [26] | 92 | 76 | 78 | 64 | 84
ViT-B/16 [3] | 83 | 68 | 76 | 52 | 69
ViT-B/32 [3] | 81 | 62 | 68 | 53 | 65
ViT-L/16 [3] | 89 | 65 | 79 | 57 | 72
ViT-L/32 [3] | 81 | 69 | 74 | 57 | 69
ConvNeXt-B [19] | 88 | 76 | 82 | 64 | 81
Swin-B [18] | 90 | 72 | 81 | 53 | 75
Swin-B v2 [18] | 90 | 77 | 80 | 56 | 79

Table 5: Performance of different vision models for image classification on
the animal subset. We report the top-1 accuracy on the original data and the
edited images in different domains. Figure 3: Visualization of edited images
with different prompts in the drawing, weather, color, texture, and context
domains. With our benchmark vision models can be further evaluated with more
realistic and diverse data.

##  5 Conclusion

In this study, we provide an automated process to edit images with language-
based diffusion models which have a great performance in generating images.
First, we divide the ImageNet classes into some super classes based on the
WordNet hierarchy and we assigned some prompts for each subclass in 5 domains.
Second, we compare the robustness of different classification baselines with
this newly edited benchmark, which illustrates that convolutional models are
more robust. Furthermore, we compare the results of different domains of
prompts and figure out that the drawing domain is the hardest editing domain
for networks to handle. As our final step, we can mention that the data
augmentation technique can improve the performance on both the new corrupted
and the original images. This study can be a first step for benchmarking
different test sets in a fully automated way for any dataset. Moreover, it can
give an impressive insight into the weakness of models in diverse domains.

##  6 Future Work

This manipulation procedure with text-guided models can be added to the
training process as a new automated augmentation technique. Since previous
research [10, 5] shows the effectiveness of augmented techniques, we expect
that augmentation with language-based models can outperform the previous
methods. We highly recommend optimizing prompts for each image as an
adversarial attack on the network. As a result, we can assign the best prompts
for each image in a fully automated way to fool the model.

## References

  * [1] Aharon Azulay and Yair Weiss.  Why do deep convolutional networks generalize so poorly to small image transformations?  arXiv preprint arXiv:1805.12177, 2018.
  * [2] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [3] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  ICLR, 2021.
  * [4] Leon A Gatys, Alexander S Ecker, and Matthias Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2414–2423, 2016.
  * [5] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel.  Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness.  In International Conference on Learning Representations, 2018.
  * [6] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [8] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al.  The many faces of robustness: A critical analysis of out-of-distribution generalization.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8349, 2021.
  * [9] Dan Hendrycks and Thomas Dietterich.  Benchmarking neural network robustness to common corruptions and perturbations.  Proceedings of the International Conference on Learning Representations, 2019.
  * [10] Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan.  Augmix: A simple data processing method to improve robustness and uncertainty.  In International Conference on Learning Representations, 2019.
  * [11] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song.  Natural adversarial examples.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15262–15271, June 2021.
  * [12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Prompt-to-prompt image editing with cross attention control.  arXiv preprint arXiv:2208.01626, 2022.
  * [13] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  arXiv preprint arxiv:2006.11239, 2020.
  * [14] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [15] Forrest N. Iandola, Song Han, Matthew W. Moskewicz, Khalid Ashraf, William J. Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and $<$0.5mb model size.  arXiv:1602.07360, 2016.
  * [16] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In International Conference on Machine Learning, pages 5637–5664. PMLR, 2021.
  * [17] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in Neural Information Processing Systems, volume 25, 2012.
  * [18] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
  * [19] Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie.  A convnet for the 2020s.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [20] Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or.  Null-text inversion for editing real images using guided diffusion models.  arXiv preprint arXiv:2211.09794, 2022.
  * [21] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do cifar-10 classifiers generalize to cifar-10?  arXiv preprint arXiv:1806.00451, 2018.
  * [22] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In International conference on machine learning, pages 5389–5400. PMLR, 2019.
  * [23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models, 2021.
  * [24] Andrew Zisserman Simonyan, Karen.  Very deep convolutional networks for large-scale image recognition.  arXiv:1409.1556, 2014.
  * [25] Kaihua Tang, Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang.  Invariant feature learning for generalized long-tailed classification.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 709–726, 2022.
  * [26] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He.  Aggregated residual transformations for deep neural networks.  arXiv preprint arXiv:1611.05431, 2016.
  * [27] Bingchen Zhao, Shaozuo Yu, Wufei Ma, Mingxin Yu, Shenxiao Mei, Angtian Wang, Ju He, Alan Yuille, and Adam Kortylewski.  Ood-cv: A benchmark for robustness to out-of-distribution shifts of individual nuisances in natural images.  Proceedings of the European Conference on Computer Vision (ECCV), 2022.