# Exploring the effects of robotic design on learning and neural control

Joshua Paul Powers

###### Abstract

The ongoing deep learning revolution has allowed computers to outclass humans
in various games and perceive features imperceptible to humans during
classification tasks. Current machine learning techniques have clearly
distinguished themselves in specialized tasks. However, we have yet to see
robots capable of performing multiple tasks at an expert level. Most work in
this field is focused on the development of more sophisticated learning
algorithms for a robot’s controller given a largely static and presupposed
robotic design. By focusing on the development of robotic bodies, rather than
neural controllers, I have discovered that robots can be designed such that
they overcome many of the current pitfalls encountered by neural controllers
in multitask settings. Through this discovery, I also present novel metrics to
explicitly measure the learning ability of a robotic design and its resistance
to common problems such as catastrophic interference.

Traditionally, the physical robot design requires human engineers to plan
every aspect of the system, which is expensive and often relies on human
intuition. In contrast, within the field of evolutionary robotics,
evolutionary algorithms are used to automatically create optimized designs,
however, such designs are often still limited in their ability to perform in a
multitask setting. The metrics created and presented here give a novel path to
automated design that allow evolved robots to synergize with their controller
to improve the computational efficiency of their learning while overcoming
catastrophic interference.

Overall, this dissertation intimates the ability to automatically design
robots that are more general purpose than current robots and that can perform
various tasks while requiring less computation.

###### Abstract

Many organisms, including various species of spiders and caterpillars, change
their shape to switch gaits and adapt to different environments. Recent
technological advances, ranging from stretchable circuits to highly deformable
soft robots, have begun to make shape-changing robots a possibility. However,
it is currently unclear how and when shape change should occur, and what
capabilities could be gained, leading to a wide range of unsolved design and
control problems. To begin addressing these questions, here we simulate,
design, and build a soft robot that utilizes shape change to achieve
locomotion over both a flat and inclined surface. Modeling this robot in
simulation, we explore its capabilities in two environments and demonstrate
the existence of environment-specific shapes and gaits that successfully
transfer to the physical hardware. We found that the shape-changing robot
traverses these environments better than an equivalent but non-morphing robot,
in simulation and reality.

### Citations

Material from this dissertation has been published in the following form:

Powers. J., Kriegman. S., and Bongard. J.. “The effects of morphology and
fitness on catastrophic interference,” in Artificial Life Conference
Proceedings. MIT Press, 2018, pp. 606-613.

Powers. J., Grindle. R., Kriegman. S., Frati. L., Cheney. N., and Bongard. J..
“Morphology dictates learnability in neural controllers,” in Artificial Life
Conference Proceedings. MIT Press One Rogers Street, Cambridge, MA 02142-1209
USA journals-info mit, 2020, pp. 52-59.

Powers. J., Grindle. R., Frati. L., and Bongard. J.. “A good body is all you
need: avoiding catastrophic interference via agent architecture search,” in
Online digital archive for electronic preprints of scientific papers (ArXiv)
2021.

Shah. D.*, Powers. J.*, Tilton. L., Kriegman. S., Bongard. J., and Kramer-
Bottiglio. R.. “A soft robot that adapts to environments through shape
change,” Nature Machine Intelligence, vol. 3, no. 1, pp. 51-59, 2021. *These
authors contributed equally

### Acknowledgements

Thanks to my advisor Josh Bongard, my exemplary mentor and teacher, and to the
other member of my committee Mathew Weston, Safwan Wshah, and Joe Near for
their support. Thanks to my family who make all of my work worth it, and more
especially my wife Nina, for her constant love and unwavering support.

### Table of Contents

toc

### List of Figures

lof

### List of Tables

lot

## Chapter 1 Introduction

Current machine learning algorithms should not be underestimated; they have
drastically changed the research landscape in various fields such as vision
[1, 2, 3, 4], natural language processing [5, 6, 7], and control [8, 9, 10].
While neural networks and deep learning (the techniques used in these
advancements) are behind this progress, most of these techniques were
developed in the past and have only now come to fruition. The perception,
which is a simple binary model of a biological neuron was developed as early
as 1957 [11], from there research preceded to sigmoid neurons (a continuous
value, non-linear model of biological neurons) and back propagation [12, 13,
14, 15]. Eventually, work was also done to improve neural architectures by
creating neural networks with many layers and introducing new types of neural
network layers such as convolutions layers (principally used in vision
research today) [16, 17] and recurrent network layers (principally used in
natural language processing). Only recently have these neural techniques
supplanted traditional state of the art methods, largely due to advancements
in computer hardware and data availability [18]. This brief history of deep
learning and neural networks also highlights that the key focus of research in
the past and present is neurocentric: focused on improving neuron models,
neural architectures, and training algorithms.

Today, neural-network-based machine learning continues to set new benchmarks
and achieve superhuman performance in many tasks, and the spread of domains
subject to such advancements continues to broaden. However, one area where
machine learning adoption has been slow, and in some cases non-existent, are
domains which require multitask learning and/or multiple behaviors [19]; this
is especially true in robotics.

(A) Catastrophic Forgetting (B) Catastrophic Interference

Figure 1.1: (A) a representation of the occurrence of catastrophic forgetting
during training: task A is learned first and then when task B is learned the
network ends in a local optima that no longer solves task A. (B) a
representation of the occurrence of catastrophic interference during training:
during simultaneous training on both tasks the robot falls into a local optima
that solves only one task. (Both) $\theta$: A two dimensional representation
of the complete parameter space of a neural networks weights.
$\theta^{*}_{A}$: The network weights that produce optimal performance on a
given task A. $\theta^{*}_{B}$: The network weights that produce optimal
performance on a given task B. $\theta^{*}$: The set of network weights that
allow a single neural network to perform optimally on all tasks. The red
trajectory shows the actual trajectory that occurs when a network experiences
a catastrophic failure during training, while the green arrows show the
desired training trajectory. In both cases the failure occurs because the
network find a local optima that is not in $\theta^{*}$. This figure has been
adapted from [20]. Figure 1.2: Due in part to the design of the muscular and
skeletal systems, a human performs a similar gait pattern across various
terrains. However, current bi-pedal rigid robots must adapt their gait for
even the smallest of terrain changes. In essence the human form allows for the
training of a single walking strategy for a variety of circumstances and thus
reduces the required computational effort (compared to a robot) to discover
the function for walking in multiple domains.

### 1.1 The Problem

Given the current computational capabilities, robots in simulation can often
be trained to perform a single task very well, but when that same robot is
trained to perform a new task, the robot performs increasingly poor on the
former task, or more often, forgets its prior training completely [21]. A
robot may also be trained on multiple desired tasks simultaneously in which
case training steps that results in improvement in one task often cause
greater reduction in the others resulting in a robot that learns to specialize
in only one of the tasks. These crippling phenomenon are known as catastrophic
forgetting and catastrophic interference(see Fig. 1.1). Thus, currently we see
industrial and consumer robots performing only specialized tasks. While there
have been many proposed methods for attempting to overcome these problems, few
have been successful beyond a few tasks [19, 20] and often perform much worse
than specialist robots on the same tasks. Furthermore, training such neural
networks requires impressive amounts of computational costs of up to
$12,976,128 [22, 8] for the training of a single agent on a single task. Thus,
current strategies of machine learning also lack the ability to scale down to
the computational restrictions on common robot platforms.

As in other fields where neural networks have been successful, most research
on methods to combat catastrophic interference are neurocentric: focused on
the development of better machine learning algorithms and improvements to
neural architectures. One line of research attempts to link individual
specialist networks in an effort to produce a generalist whole [23, 24]. Such
methodologies often suffer from requiring extremely complex networks, massive
amounts of computing power, and excessively long training periods, while being
very sensitive to small changes in algorithmic parameters [24]. Other methods
attempt to hide the problem by producing robots that learn new or forgotten
tasks rapidly enough that they appear to know how to perform all tasks
simultaneously [25]; and one of the most famous methods freezes sections of
the neural network when training on a new task to prevent the overwriting of
previously learned information [20]. However, these methods similarly suffer
from requiring a robot to learn a large amount of prior information, besides
being even more complex and computing intensive.

Much of the motivation for these neurocentric based resolutions stems from the
_Universal Approximation Theorem_ [26], that shows that a neural network is
capable of approximating any function, assuming that the function exists. For
example, if there exists a function that can turn written English words into
spoken English sounds (which we assume exists since humans do this every day),
then a neural network is capable of approximating that function if we can just
find the correct parameters. In robotics the functions we approximate
generally involve taking sensor data (sight, touch, sounds) and converting
them to robotic motor commands that achieve a task. For instance, we may
design a bipedal robot that needs to learn a function to identify a specific
object and then output motor commands to walk toward it. We first assume that
such a function exists (which is likely since humans can perform this task)
and then train a network to solve the problem. However, if we expand the
problem such that the robot must perform this task over multiple terrains,
this could prove quite difficult and is even a challenge for the most
sophisticated bipedal robots today even using the best neural network
techniques. One way humans may overcome this problem is possibly due to the
design of the muscular and skeletal systems (see Fig. 1.2). The human foot,
unlike a rigid robot foot, is complaint against an uneven surface and adapts
such that walking on rough terrain is essentially the same as walking on
smooth terrain [27]. Thus, humans use a similar gait pattern across various
terrains, where current bipedal rigid robots must adapt their gait for even
the smallest of terrain changes. The human form allows for the training of a
single walking strategy for a variety of circumstances and thus reduces the
required computational effort (compared to a robot) to discover the function
for walking in multiple domains. Thus in robotics, the robot design in which a
neural network is embodied, is likely to have a measurable impact on the
ability to approximate the function required to complete the desired task.
Research that is focused on the synergy between AI and robotic design often
falls in the category of evolutionary robotics.

### 1.2 Evolutionary Robotics

Evolutionary robotics is an embodied approach to AI in which both a robots’
design and controller are often optimized simultaneously using an evolutionary
algorithm [28]. This is done by optimizing the parameters of a robot against a
behavioral goal (e.g. move in a direction). The parameters of the robot
usually include the weights of a neural network as well as parameters that
define aspects of the robot’s design such as physical structure, sensor number
and placement, motor strength, etc. Thus, research in this field has often
incorporated methods beyond the architecture and training of neural networks
to overcome the hurtles faced by traditional machine learning approaches [28].

The first artificial evolution of a robots design and controller was done in
1994 by Karl Sims [29], creating simulated robots that displayed lifelike
forms and behaviors. This has since inspired others to utilize the principles
of embodiment to build upon his work such as [30] which optimized both the
body and control of robot that could then be 3D printed, and [31] who used
these principles to design simulated soft robots. However, all of these
experiments focused solely on a simple locomotion task. They also focused
mainly on developing new methods in the evolution of body design rather than
looking at the effects of those designs on problems typically faced in neural
control. An example of recent work that does apply these principles to the
domain of machine learning is [32] which evolved robot designs that were
modular to varying environments with a net effect of reducing the overall
training time required by the neural network.

The principles of evolutionary robotics are considered most beneficial when we
encounter initially unintuitive problems. In this case, using evolutionary
algorithms can give new insight to a problem as they tend to ignore human
bias, thus generating unique solutions that otherwise may typically be
disregarded[33, 34]. While, this evolutionary process almost always takes
place in simulation, this knowledge can then be used to increase our capacity
to build real robots with more desirable characteristics. As mentioned,
identifying scalable methods for creating more general, multi-task robots is
currently one such unintuitive space and is thus the task to which
evolutionary robotics is applied in this dissertation.

Figure 1.3: Ways in which the single task of phototaxis (locomotion toward a
light source) may be broken down into multiple tasks. (A) We can break
phototaxis down into it’s parts such as (1) identifying the direction of the
light, (2) orienting towards the light (3) locomotion toward the light. (B) We
can break the phototaxis task down by environment. (1) locomote toward the
light when it is on the left (2) locomote toward the light when it is on the
right. In this dissertation I use the latter method. In the context of
catastrophic interference, this means that a robot that is simultaneously
trained to walk toward a light source at at multiple different locations but
fails to learn to walk toward a light source in at least one of the locations
would be subject to this catastrophic interference

### 1.3 Description of Tasks and Environments

Often in robotics we describe a specific task (e.g., locomotion) or
environment (e.g., flat ground, incline ground) in which we would like our
robot to be successful, and the description is mostly accepted. However, when
we move to multiple tasks or environments there are often disputations as to
what constitutes a single task or environment. For example in this
dissertation, much of the work is explored under phototaxis, that is, getting
a robot to locomote toward a light source. This could be considered as a
single task in and of itself, or could be broken down as multiple tasks in
different ways as shown in Fig. 1.3. One way is to break the task down into
its parts, such as, identifying the direction of light and locomotion in a
given direction. Another way would be to consider each different location of
the light source in relationship to the robot to be a separate task (e.g.,
move toward the light when it is on the left, move toward the light when it is
on the right). In both cases, the combination of these sub tasks would then
constitute a single larger task of general phototaxis. Thus, it is not
uncommon for researchers in the field to disagree on what is a multi-task
robot depending on how one defines a task.

In this dissertation I consider each placement of the light to be a different
task. In the context of catastrophic interference, this means that a robot
that is simultaneously trained to walk toward a light source at at multiple
different locations but fails to learn to walk toward a light source in at
least one of the locations would be subject to this catastrophic interference.
While there may be neural centric methods to make robots overcome this
particular problem instance and perform general phototaxis, in this
dissertation, I reduce the learning method and network size of the robots such
that they do experience catastrophic interference. The focus of this
dissertation is on alternate embodied methods to overcome this phenomenon that
can synergize rather than compete with traditional methods on this problem.
Thus, the description of phototaxis as a multitask problem is justified in
this sense.

Figure 1.4: Two sin waves with the same frequency but different amplitude and
phase; if the waves both represent the values of the sensor of a robot over
time, the robot would be considered homeostatic in relation to this sensor.
The chief concern for homeostasis in this dissertation is thus overall signal
pattern.

### 1.4 Homeostasis

Throughout this dissertation, I will create metrics to explain how certain
robot designs affect the loss landscape of neural control policies associated
with that design. I also develop a theory to explain why those designs have
certain effects on that landscape. The main theory brought forth is that
robots that are resistant to catastrophic interference exhibit designs that
allow for sensor homeostasis.

Homeostasis as a principle component of intelligence has a rich history in the
literature and is built principally on the ideas presented English
Cyberneticist and Psychiatrist Ashby [35]. For Ashby, homeostasis is a base
principle in intelligence and the ability of a living or artificial system to
adapt to a continuously changing environment [36], specifically he designed a
famous system called the homeostat to demonstrate these principles. Later,
Dyke and Harvey would conclude that understanding the mechanisms of
homeostasis is necessary for understanding real or artificial life [37, 38].
In all this work, researchers found that intelligent systems are created with,
among other goals, a desire to maintain a consistent internal state. These
ideas have been used to motivate the creation of neural networks capable of
dynamically changing their connections by means of plasticity rules [39, 40].
However, this work did not extend these ideas to the body design of robots
used in these studies.

In the context of phototaxis, homeostasis means that as a robot locomotes
toward light sources placed at different locations, the time series data of
its light sensor values rapidly converges to the same values across these
trials. That is, the robot “sees” these seemingly different environments as
the same. This description of homeostasis can be seen in Fig. 1.4. Internal
homeostasis has been postulated as a possible biological evolutionary goal
[35, 41] and has seen considerable focus as a reward method for improving the
behaviors in reinforcement learning [42, 43] as well. In a previous example,
we described how humans maintain a similar gait across a variety of terrains.
In this sense, each terrain is internally experienced in much the same way.
Thus, according to the previous work cited [36, 27, 41], some evolutionary
pressure may have favored functional anatomy in the human foot that
mechanically “erased” slight differences in terrain, thereby obviating the
need for neural adaptation of gait over such terrains. We can see many such
homeostatic patterns throughout biology as various levels: many animals build
structures (beaver damn, human home, etc) to provide consistent environmental
conditions, and single cell organisms clump together in colonies such that
internal cells experience minimal variation in their environment [44].
However, work to date has not used homeostasis as a tool for optimization and
analysis in the context of catastrophic interference. Thus, In this
dissertation we will focus on measuring sensor homeostasis as an analysis tool
to describe robot behavior and designs. Doing so will allow for the future use
of homeostasis as a fitness objective which may guide the automatic design of
robot structures that are both performant and resistant to catastrophic
interference.

### 1.5 Contribution Outline

In my work, I have taken a radically different morphological approach to the
multitask learning problems facing neural networks in robotics. Rather than
work on improving neural training algorithms or architectures, my research
focuses on the how different aspects of a robot’s design can facilitate faster
training and the likely hood of achieving multitask behavior. Overall this
dissertation introduces novel metrics that quantify why certain designs
improve neural performance and how these principles can be applied to the
design of other robots.

Chapter 2 demonstrates the learning differences between three equally capable
robot morphologies in phototaxis. Additionally, I introduce novel metrics to
quantify catastrophic interference and definitively show that morphology is
the principle factor in determining the amount of interference a robot
experiences during training. This chapter, however, only provides speculation
as to why but lays the foundation for embodiment as an important aspect of the
catastrophic interference problem. Consequently, Chapter 3 develops a
theoretical foundation for measuring a robot design’s sample efficiency
($M_{L}$) and resistance to catastrophic interference ($M_{CF}$) that also
serve as an explanation for why embodiment is such an important component of
neural training. Chapter 4 tests the theoretical work from Chapter 3 across
various robot designs and training algorithms. I show that $M_{L}$ and
$M_{CF}$ are predicative of a design’s real world performance. Furthermore, I
demonstrate that optimizing for the design of variations in a robot’s body
along with a robot’s controller leads to the creation of more general purpose
robots that have the added benefit of requiring reduced training times. In
this chapter I also show that optimizations naturally follow a homeostatic
design gradient, that is, the robotic designs are more homeostatic in later
generations than in earlier ones. This last finding provides an explanation
for the improved performance of certain robot designs. By focusing on
advancements in robot design rather than neural controllers, my work is
synergistic rather than competitive with traditional machine learning research
in this area. Thus, this work intimates that algorithms which automatically
detect and exploit complementary features of the brain and the body will
enhance current state-of-the-art techniques in the field.

While currently most of my work is conducted in simulation, I have also begun
to validate some of this work in physical systems. Thus in Chapter 5, I
showcase a simulated soft robot that can change its shape (or design) to
operate in different environments. Using an evolutionary algorithm, the shapes
and controllers for the different environments are automatically determined to
allow for successful locomotion.

## Chapter 2 The effects of morphology and fitness on catastrophic
interference

### 2.1 Abstract

Catastrophic interference occurs when an agent improves in one training
instance but becomes worse in other instances. Many methods intended to combat
interference have been reported in the literature that modulate properties of
a neural controller, such as synaptic plasticity or modularity. Here, we
demonstrate that adjustments to the body of the agent, or the way its
performance is measured, can also reduce catastrophic interference without
requiring changes to the controller. Additionally, we introduce new metrics to
quantify catastrophic interference. We do not show that our approach
outperforms others on benchmark tests. Instead, by more precisely measuring
interactions between morphology, fitness, and interference, we demonstrate
that embodiment is an important aspect of this problem. Furthermore,
considerations into morphology and fitness can combine with, rather than
compete with, existing methods for combating catastrophic interference.

### 2.2 Introduction

Currently, a popular method for realizing intelligent machines is to optimize
the parameters of fixed-architecture deep neural networks [45]. However,
increasing interest is coming to bear on optimizing the cognitive architecture
of such networks as well [46]. Indeed, investigations into the evolution of
cognitive architectures has long been a target of study [47, 48, 49] in the
evolutionary computation community.

It follows from this, if dealing with robots, that optimizing body plan
influences sensory repercussions of action, which in turn will influence which
cognitive architecture will facilitate learning for a given task. To begin
investigations into this last observation, here we investigate how the choice
of robot morphology and fitness affect one specific aspect of neural networks:
their ability to resist catastrophic interference. We employ an evolutionary
robotics approach to investigate this question.

#### 2.2.1 Evolutionary robotics

Since its beginnings, many investigators in the field of evolutionary robotics
[50, 51, 52] have used evolutionary algorithms to optimize both the body plan
and neural controllers of robots [53, 54, 55]. Here we show that indeed the
choice of body plan can influence the efficacy of training neural controllers:
some body plans enable greater resistance to catastrophic interference.

(a) (b) (c)

Figure 2.1:  Three classes of phototaxic robots—(a) legged, (b) wheeled, and
(c) whegged—and their environments were simulated using Pyrosim
(ccappelle.github.io/pyrosim). Each robot has eight degrees of freedom, as
depicted by the black and white arrows which indicate the axis (straight) and
direction (curved) of rotation for a particular hinge-joint (a, c) or wheel
(b, c).

#### 2.2.2 Catastrophic interference

It has been acknowledged since the early days of neural network research that
catastrophic interference [21], also known as catastrophic forgetting [56,
19], is a major challenge to training them effectively. Even in the most
common forms of network training such as the backpropagation of error, there
is no guarantee that reducing the network’s error on the current training
sample does not increase error on the other training samples.

For these reasons, much effort has been expended to combat this challenge. One
family of solutions involves constructing modular networks [57, 58, 59, 60,
61, 62] in which different modules deal with different subsets of the training
set. In such networks, changes to one module may result in improved
performance for the training subset associated with that module without
disrupting performance on other subsets. Such modularity has indeed been
demonstrated to minimize catastrophic interference [63, 64, 65, 66].

Related to this concept of modularity are networks in which some subsets of
the network that have a large impact on the current training set are made less
resistant to change during subsequent training [20, 67]. The remaining parts
of the network remain adaptive, and thus able to deal with new training
instances without disrupting behavior on previous instances.

The drawback of these approaches however is that network size tends to
increase with the amount of training data, because new modules must be
implicitly or explicitly added for new training data.

Another guard against catastrophic interference is to reduce the magnitude of
behavioral impact after some change is made during training. The intuition
here is that small changes to network behavior may increase the likelihood of
local improvements for new training instances while minimizing or nullifying
performance decreases on the previous training set.

In evolutionary methods, one way of reducing behavioral impacts is to
dynamically tune mutation rates [68] and/or crossover events [69]. A recent
approach demonstrated for neuroevolution is to dynamically tune individual
synaptic weights proportionally to their impact on the network’s behavior
[70]. In the genetic programming community, semantic variation operators have
been reported [71, 72]. These operators take into account the semantics of
subtrees or individual tree nodes, and attempt to replace them with new
genetic material that exhibits similar semantics.

In this work we show that the body plan itself as well as the manner in which
fitness is defined can buffer the behavioral impact of mutations such that the
embodied agent’s behavior can improve in one environment without adversely
impacting its performance in another environment in which it is already
proficient without using any methods for specifically preventing interference
in the controller.

#### 2.2.3 Embodied impacts on neural properties

Besides catastrophic interference, it has been shown elsewhere that embodiment
can influence the positive or negative aspects of neural networks. For
instance, work in morphological computation has shown that a good choice of
morphology can allow for simplified neural networks (e.g. [73]).

Morphology may also render a robot more robust to external environmental
perturbation [74] or internal changes to the neural controller [75]. Some body
plans also facilitate or obstruct the discovery of modular neural networks
[76, 77]. In this work we introduce a heretofore unexplored aspect of the
interaction between body plan and neural control of embodied agents: how the
choice of body plan may render the neural controller more or less resistant to
catastrophic interference.

The next section describes our methodology for demonstrating this phenomenon;
the following sections provide results from evolving neural controllers in
different body plans; the final sections provide some discussion as to how and
why this phenomenon arises and concludes with avenues for future study.

### 2.3 Methods

#### 2.3.1 The robots

Three types of robots were used in this study (Fig. 2.1). All are variations
on a standard radially symmetric quadrupedal form used in other evolutionary
robotics studies [78, 79, 80]. The robots differ in their differential use of
legs and wheels.111github.com/jpp46/ALIFE2018 contains the source code
necessary for reproducing the results reported in this here. Combining wheels
and legs in different ways is itself an active area of study in robotics [81,
82, 83].

##### The legged robot

This robot consists of a body and four legs attached to the body by a joint.
Each joint rotates $180\degree$ through the plane defined by the two cylinders
comprising that leg. Each leg consists of two limbs bent at $90\degree$
attached by a joint which also rotates $180\degree$ through the plane defined
by the upper and lower legs (Fig. 2.1a). Each of the resulting eight joints
are actuated using position control: a value arriving from the motor neuron
attached to the joint is treated as a desired angle. The length of each upper
leg, lower leg, and the two sides of the main body are 0.3 units of length
long (the physics engine is agnostic to the length scale).

##### The wheeled robot

This robot freezes the previously mentioned joints at their initial angles and
cannot change these positions. In addition, each leg has a wheel attached on
the end that rotates through the plane defined by its upper and lower leg
components. These wheels are attached to the legs by axles which rotate
through the sagittal plane (Fig. 2.1b). This allows the wheels to change two
directions of their rotation, acting as caster wheels. Values received at each
of the two motors controlling the wheel treat the incoming values as desired
angular velocity. In the spirit of keeping the robots as similar as possible,
the motor neurons innervate the wheels output as desired angular velocities in
$[-90\degree/s,+90\degree/s]$ to match the legged robot where motor neurons
dictate desired angles in $[-90\degree,+90\degree]$.

##### The whegged robot

This robot combines features of the previous two robots. In this robot, only
the joints connecting the two leg limbs are frozen and can not change from
their initial $90\degree$ angle. The joints joining the legs to the body are
the same as the first robot: position controlled, actuated one-degree-of-
freedom rotational joints. The robot also has wheels on the legs, but they do
not act as caster wheels (Fig. 2.1c). This robot combines position control at
the four shoulder joints and velocity control at the four wheels.

##### The sensors

Each robot contains a light sensor $\ell$ that responds to light as a float
value according to the inverse square law for light propagation:
$\ell=1/d^{2}$, where $d$ is the distance from a light source. Occlusion is
not simulated in the light sensor: if an object is between the sensor and the
light, there is no change in sensor value. Each robot contains a single binary
touch sensor in each leg. These four sensors read $+1$ when in contact with
the ground, and $-1$ otherwise.

#### 2.3.2 The controllers

The controller for each robot is a neural net with $5$ input neurons, one for
each sensor, and $8$ output neurons, one for each motor. The neural net has no
hidden layers and is fully connected. The connection weights are captured by a
$5\times 8$ matrix, which also represents the genome of a robot. The weights
of the matrix are constrained to values in the interval $[-1,1]$. The update
function for a neuron during simulation follows this function:

$m_{i}^{(t)}=\tanh\left[m_{i}^{(t-1)}+\tau_{i}\sum_{s=1}^{5}w_{s_{i}}m_{s}^{(t)}\right]$
(2.1)

where $m_{i}^{(t)}$ denotes the value of the $i^{th}$ motor neuron at the
current time step, $m_{i}^{(t-1)}$ is a momentum term that guards against
‘jitter’ (high-speed and continuous reversals in the angular velocity of a
joint), $\tau_{i}$ is a time constant that can strengthen or weaken the
influence of sensation on the $i^{th}$ motor neuron relative to its momentum,
and $w_{s_{i}}$ is the weight of the synapse connecting the $s^{th}$ sensor
neuron to the $i^{th}$ motor neuron. In order to ensure that random
controllers produce diverse yet not overly-energetic motion, all $\tau_{i}$
were set to 0.3 via empirical investigation.

#### 2.3.3 The task environment

Robots are evolved to perform phototaxis: minimizing the distance between
themselves and a light source in their environment. Each robot is exposed to
all training environments. Each environment consists of a light source that is
30 body lengths away from the robot (equivalent to nine units of distance).
The robot starts at coordinate space $(0,0)$ and the light sources are placed
at coordinate space $(0,9)$, $(0,-9)$, $(9,0)$, $(-9,0)$ for environments $1$,
$2$, $3$, and $4$ respectively. The robots are evaluated for $1000$ time steps
in each environment using a fixed time step of 0.05 seconds.

#### 2.3.4 The fitness functions

The fitness function applied to a single environment is the value of the
robot’s light sensor at the end of an evaluation period ($t=1000$). We combine
fitness values drawn from multiple environments in three different ways:

$F_{\Sigma}\doteq\sum_{i=1}^{n}f_{i},\qquad
F_{\Pi}\doteq\prod_{i=1}^{n}f_{i},\qquad F_{\min}\doteq\min_{i=1}^{n}f_{i}.$
(2.2)

where $f_{i}$ is the individual’s fitness in environment $i\in(1,n)$.

#### 2.3.5 The evolutionary algorithm

We employed a simple parallel hill climber to nine different experimental
treatments as shown:

| Legged | Wheeled | Whegged
---|---|---|---
$\sum$ | Treatment 1 | Treatment 2 | Treatment 3
$\prod$ | Treatment 4 | Treatment 5 | Treatment 6
$\min$ | Treatment 7 | Treatment 8 | Treatment 9

For each treatment, 30 independent evolutionary runs were conducted each with
a population size of $S=100$. Each individual $p$ was encoded as a $5\times 8$
matrix of synaptic weights. Thus population $P$ is represented by a $100\times
5\times 8$ tensor. We used a mutation strength of $m=0.05$, and conducted each
run for $G=3000$ generations. At each generation $g$ we took the current
population $P_{g}$ and generated a new population $P_{(g+1)}$ by mutation such
that:

$P_{(g+1)}=\mathcal{N}\left(P_{g},m\right)$ (2.3)

Where $\mathcal{N}\left(\mu,\sigma\right)$ is the standard normal
distribution. We then updated the new population using:

$p_{g+1}=\begin{cases}p_{g+1},\;F\left(p_{g+1}\right)>F\left(p_{g}\right)\\\
p_{g},\;\text{otherwise}\end{cases}$ (2.4)

where $p$ denotes an individual in the population at generation $g$ and the
fitness function $F$ is determined by the treatment as defined in Eq. 2.2. As
shown, each child in the evolutionary algorithm only competes with its direct
parent, creating $S$ individual climbers.

#### 2.3.6 Measuring catastrophic interference

Catastrophic interference, in its simplest formulation, occurs when an
improvement in one environment incurs reduced performance in one or more other
environments. In an evolutionary setting, catastrophic interference can be
measured at the highest temporal resolution by considering mutations: the
change in performance between a parent and child for each environment
experienced by both agents (Fig. 2.2).

For the purposes of analysis, we only track performance changes between
mutations where overall fitness increased (we do not record deleterious or
stagnant mutations). Also we only record and perform analysis on the change in
Euclidean distance of the robot from the light source (blue and red bars in
Fig. 2.2). This provides a more intuitive understanding of changes in
performance and allows comparison between any 2 treatments regardless of the
fitness function used in that treatment.

We define a function $D$ on an individual such that it returns a vector
$[x_{1},x_{2},...,x_{n}]$, where $x_{i}$ is the distance from the light source
at the end of simulation in environment $i$. As shown in Fig. 2.2, fitness and
distance are inversely correlated therefore we record change in distance after
every successful mutation as:

$\Delta D=-\left[D\left(p_{(g+1)}\right)-D\left(p_{g}\right)\right].$ (2.5)

We negate this difference, so that a positive increase in fitness in an
environment causes an increase in the corresponding component of $\Delta D$.
We apply four metrics $(M_{1},M_{2},M_{3},M_{4})$ to this data ($\Delta D$) to
measure the overall effect of catastrophic interference in each each of the
treatments.

Figure 2.2:  Example mutations that (a) are deleterious, (b) result in
catastrophic forgetting, and (c) avoid catastrophic forgetting. The smaller
the distance from the light source (blue, red), the higher the fitness
(inverse square law).

#####  $M_{1}$: Average Worst Absolute Distance

For this metric, we take the distances from the light sources for the champion
from each run, for each treatment. We here define a run champion such that:

$\displaystyle C=\min_{P_{g}}\left[\max D(p_{g})\right],\;\text{where
}\;g=3000.$ (2.6)

Recall that distance is measured from the light source so higher is worse.
Therefore a champion $C$ is the one with the lowest distance in it’s worst
environment. We compute $C$ for each run and take the average across all runs,
for each treatment:

$M_{1}=\frac{1}{N}\sum_{i=1}^{N=30}C_{i}$ (2.7)

If this number is particularly high for a given treatment then it either
failed to evolve robots that behaved well in at least one environment, a sign
of catastrophic interference. If this number is sufficiently low for a given
treatment, it was able to evolve agents that solved all environments, and at
worst suffered mild amounts of interference.

#####  $M_{2}$: Average $\theta$ from $V$

If a treatment is able to avoid evolving specialists by producing children
that enjoy increases in fitness in all environments (Fig. 2.2c), the most
beneficial mutation possible would be represented by the vector
$V=[x_{1},x_{2},...,x_{n}]$ where all elements are equal,
$x_{1}=x_{2}=...=x_{n}$. To compute $M_{2}$ we take all of the distance
vectors from the mutations of a champion $C$ and record the cosine angle
difference between $\Delta D$ and $V$. If the angle is less than or equal to
$45\degree$, then that vector, although perhaps biased slightly toward one or
the other environment, nevertheless represents a mutation that avoided
catastrophic interference. For this metric, we again record all such
beneficial mutations within the lineage of a run that produced that run’s
champion. This can be represented as:

$\displaystyle\theta^{\prime}$ $\displaystyle=$
$\displaystyle\frac{1}{A_{i}}\sum_{a=1}^{A_{i}}\arccos\frac{\Delta
D_{a}\cdot(V)}{|\Delta D_{a}||V|}$ (2.8) $\displaystyle\theta_{i}$
$\displaystyle=$
$\displaystyle\begin{cases}\theta^{\prime}:\theta^{\prime}_{i}\leq
180\degree\\\ |\theta^{\prime}-360\degree|:\text{otherwise}\\\ \end{cases}$
(2.9) $\displaystyle M_{2}$ $\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N=30}\theta_{i}$ (2.10)

where $A_{i}$ is the number of recorded $\Delta D$’s, $\theta^{\prime}$
denotes the angle between the $\Delta D$ and $V$, and $\theta_{i}$ represents
the average amount of catastrophic interference that individuals in the
lineage of the champion from run $i$ experienced (lower $\theta_{i}$
represents less catastrophic interference). $M_{2}$ thus represents the
average amount of catastrophic interference experienced by the run champion
lineages in a treatment. If $M_{2}$ is low for a given treatment, then that
treatment can be considered to be resistant to interference.

#####  $M_{3}$: Average $\Delta D$ Length

Again we take the vector $\Delta D$ of a beneficial mutation that occurred
within the ancestral lines of run champions. We then perform a similar method
as in Eq. 2.8 on the length $|\Delta D|$ as shown:

$\displaystyle L_{i}$ $\displaystyle=$
$\displaystyle\frac{1}{A_{i}}\sum_{a=1}^{A_{i}}|\Delta D_{a}|$ (2.11)
$\displaystyle M_{3}$ $\displaystyle=$
$\displaystyle\frac{1}{N}\sum_{i=1}^{N=30}\Delta L_{i}$ (2.12)

$M_{3}$ represents the average magnitude of improvement made during mutations
by the run champion lineages in a treatment. This metric only makes sense in
conjunction with $M_{2}$. Consider a treatment with small $M_{2}$ yet also
small $M_{3}$. The treatment will suffer less from catastrophic interference,
but is insignificant since the mutations yield insignificant improvements.
However, if $M_{2}$ is small and $M_{3}$ is large for a given treatment, that
treatment not only yielded mutations that avoided catastrophic interference
but also exhibited high evolvability.

#####  $M_{4}$: Average Percentage of Points in Quadrant $I$

Even though we attempt to measure catastrophic interference among just
beneficial mutations, it is possible for a beneficial mutation to make a
sufficiently large improvement in fitness in one environment even though there
is some degradation in performance in the other environment (e.g. Fig. 2.2b).
This equates to a $\Delta D$ where one of its elements is negative. When
plotted as a point for the case of 2 environments the point falls in the upper
left or lower right quadrant of a scatter plot. Thus, to buttress our
measurement of catastrophic interference we devised a fourth metric, which is
simply the fraction of beneficial mutations within run champions’ ancestral
lines where all elements of $\Delta D$ are positive. In the case of 2
environments they fall within the upper right quadrant (Quadrant $I$). This
can be seen visually in Fig. 2.4.

### 2.4 Results

We analyze the performance and relative amounts of catastrophic interference
for all nine treatments across $M_{1-4}$. We use the Mann-Whitney U test (with
Bonferroni correction for eighteen comparisons) to indicate statistical
significance at the $p=0.05$ level. The metrics generally show that there is
an interaction between fitness function and morphology. If there was no
interaction, an entire row (morphology does not matter) or an entire column
(fitness function does not matter) would not be significantly different.

Table 2.1:  Mean of $M_{1}$ across treatments. Arrows indicate statistical
significance between adjacent cells.

$D$ | Legged |  | Wheeled |  | Whegged |
---|---|---|---|---|---|---
$\sum$ | 8.181 |  | 8.386 | $\operatorname*{\leftrightarrow}$ | 5.915 | $\operatorname*{\leftrightarrow}$
| $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |
$\prod$ | 6.530 | $\operatorname*{\leftrightarrow}$ | 3.593 | $\operatorname*{\leftrightarrow}$ | 1.234 | $\operatorname*{\leftrightarrow}$
$\min$ | 5.827 | $\operatorname*{\leftrightarrow}$ | 3.575 | $\operatorname*{\leftrightarrow}$ | 1.296 | $\operatorname*{\leftrightarrow}$

#### 2.4.1 $M_{1}$: Average Worst Absolute Distance

As can be seen in Table 2.1, evolutionary performance generally improves
moving from top to bottom row-wise and left to right column-wise. This metric
shows that the whegged robot significantly outperformed the other two robots
and that the min function is generally a better fitness function for this
task. Interestingly, however, the treatments under the product and min fitness
functions aren’t significantly different. This suggests morphology might have
a greater impact on this metric.

Table 2.2:  Mean of $M_{2}$ across treatments. Arrows indicate statistical
significance between adjacent cells.

$\theta$ | Legged |  | Wheeled |  | Whegged |
---|---|---|---|---|---|---
$\sum$ | 71.035 |  | 73.362 |  | 72.640 |
$\prod$ | 66.292 |  | 71.090 |  | 66.895 |
| $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |
$\min$ | 57.843 |  | 58.226 | $\operatorname*{\leftrightarrow}$ | 48.844 | $\operatorname*{\leftrightarrow}$
| $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |

#### 2.4.2 $M_{2}$: Average $\theta$ from $V$

We now wish to investigate whether the greater evolvability seen for the
whegged robot under the product and min fitness functions is a result of those
treatments being better able to resist catastrophic interference. Table 2.2
shows that the whegged robot with the min fitness function achieves beneficial
mutations that yield improvements in both environments, or least only slight
decreases in fitness in one of them evidenced by their higher relative
proximity to $V$.

Table 2.3:  Mean of $M_{3}$ across treatments. Arrows indicate statistical
significance between adjacent cells.

$|\Delta D|$ | Legged |  | Wheeled |  | Whegged |
---|---|---|---|---|---|---
$\sum$ | 1.734 | $\operatorname*{\leftrightarrow}$ | 3.209 |  | 4.971 | $\operatorname*{\leftrightarrow}$
$\prod$ | 1.889 | $\operatorname*{\leftrightarrow}$ | 3.372 | $\operatorname*{\leftrightarrow}$ | 5.341 | $\operatorname*{\leftrightarrow}$
| $\operatorname*{\updownarrow}$ |  |  |  | $\operatorname*{\updownarrow}$ |
$\min$ | 1.164 | $\operatorname*{\leftrightarrow}$ | 2.411 |  | 2.001 | $\operatorname*{\leftrightarrow}$
| $\operatorname*{\updownarrow}$ |  |  |  | $\operatorname*{\updownarrow}$ |

#### 2.4.3 $M_{3}$: Average $\Delta D$ Length

Even though the whegged robot with the min fitness function may yield
evolutionary improvements in both environments after a single mutation, those
increases in fitness may be very small and thus not contribute to the observed
evolvability in that treatment. If so, one would expect $M_{3}$ to be very low
for this treatment. However, as Table 2.3 reports, this is not the case: the
legged robot with the min fitness function has the lowest $M_{3}$ value, and
the value for the whegged robot with the min fitness function is significantly
higher.

Table 2.4:  Mean of $M_{4}$ across treatments. Arrows indicate statistical
significance between adjacent cells.

% in $I$ | Legged |  | Wheeled |  | Whegged |
---|---|---|---|---|---|---
$\sum$ | 25.655 |  | 22.903 |  | 30.824 |
$\prod$ | 22.973 |  | 22.167 |  | 26.981 |
| $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |
$\min$ | 49.519 |  | 48.674 | $\operatorname*{\leftrightarrow}$ | 54.332 | $\operatorname*{\leftrightarrow}$
| $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |  | $\operatorname*{\updownarrow}$ |

#### 2.4.4 $M_{4}$: Average Percentage of Points in Quadrant $I$

Using this metric we can conclude that the min function, regardless of
morphology, seems to better force mutations to results in less catastrophic
interference, at least according to this particular metric (Table 2.4). We can
conclude this because each $M_{4}$ value in the min row is significantly
higher than the two $M_{4}$ values in the two entries above it.

#### 2.4.5 Performance in four environments.

The same general pattern held when we scaled our approach from two to four
environments. Namely, the pairwise comparisons that were significant (at the
0.05 level) in two environments remained so in four environments. However,
while the whegged robot with the product and min fitness functions similarly
outperformed the other treatments, they could not solve all four environments.

Figure 2.3:  Change in fitness ($\Delta D$, as defined in Eq.2.5), in two
environments, for the three robots (Fig. 2.1) and three fitness metrics (Eq.
2.2), colored by the generation of the mutation. Dots in the upper-right
quadrants of each robot-fitness cell represent beneficial changes in both
environments; these mutations avoided catastrophic interference. Dots in the
upper-left and lower-right quadrants of each cell are mutations that were
beneficial in one environment but deleterious in the other; these changes
caused catastrophic interference. We did not record mutations that were
deleterious in both environments (lower-left quadrants).  Figure 2.4:  The
same as Fig 2.3 , but only for the run champs.  Figure 2.5:  A tracing of a
typical whegged robot (blue) and legged robot (red) trained in two
environments, under the min fitness function. The light source is first placed
at (9, 0), and then at (-9, 0). Video is available at youtu.be/uWy33A5HZGM .

### 2.5 Discussion

As shown by Figures 2.4 and 2.3 as we change the morphology from legged to
whegged the robots demonstrate increased evolvability. Thus the fitness
landscape allows for larger jumps towards the optima. This includes those
jumps that avoid catastrophic interference altogether: mutations visualized by
points in Figs. 2.4 and 2.3 that lie in the upper right quadrant.

In conjunction, as we change the fitness function from sum to min, we see the
spread of points in Figures 2.4 and 2.3 condense toward the origin. When
combined with the whegged robot, we see a significant improvement in the
metrics we used to measure catastrophic interference. It appears that it is
the combination of correct fitness function (min) with the correct morphology
(whegged) that resists catastrophic interference: changes in morphology and
fitness alone are not sufficient. We hypothesize that this greater resistance
to catastrophic interference is what enables the whegged robot, under the min
fitness function, to achieve higher fitness within environments and consistent
fitness across environments.

One objection to this hypothesis could attribute the performance of the
whegged robot to the increased speed allowed for by wheels. We do not feel
that this is valid for two reasons: the wheeled robot also has wheels and does
not achieve the same level of performance, and the evaluation time of a
simulation was set such that all morphologies are able to reach the light
source before the end of simulation. Indeed we observed that all morphologies
reached and waited at the light source when trained against a single
environment.

In observing the behavior of the robots we noticed a pattern among whegged
robots that could account for their resistance to interference. Whegged robots
move very rapidly in a circular pattern during the initial time steps of a
simulation which may allow them to ‘sidestep’ catastrophic interference by
rapidly turning unfamiliar environments into familiar ones. An example is
shown in Fig. 2.5: the rotationally symmetric trajectories of the blue whegged
robot indicates it has recognized two versions of the same environment. The
red legged robot does not: its two trajectories are different, and take longer
to diverge. The wheeled and legged robot both seem to have much more
difficulty in turning.

### 2.6 Conclusions

This work suggests not only that the particular choice of morphology and
fitness function for embodied agents can affect their resistance to
catastrophic interference, but the very fact that the agent has a body can
help. In effect, an agent can use its body to move in such a way that a
seemingly different training instance converges sensorially to a familiar
instance. The implication of this is that the very phenomenon of catastrophic
interference itself may be to some degree a false problem arising from
investigations using non-embodied systems: Since such systems do not have
control over their input, they cannot align objects of interest in different
training instances and thus reduce catastrophic interference.

A simple example may suffice here: a human face that appears in two different
locations in an image may be difficult for a non-embodied learner to
recognize, unless there is a large amount of training data that contains
diversity along that feature (face position). In contrast, an embodied agent
equipped with a camera that experiences the same two stimuli may learn to move
such that the face is centered in its field of view. Furthermore, it may be
that different types of embodied agents may more easily discover and perform
this centering. Finally, such appropriately embodied agents may thus be able
to generalize about faces regardless of position using less training instances
than the non-embodied agent because of this ability. However, whether this
latter system is indeed more scalable in this way compared to an equivalent
non-embodied system remains as future work.

## Chapter 3 Morphology dictates learnability in neural controllers

### 3.1 Abstract

Catastrophic forgetting continues to severely restrict the learnability of
controllers suitable for multiple task environments. Efforts to combat
catastrophic forgetting reported in the literature to date have focused on how
control systems can be updated more rapidly, hastening their adjustment from
good initial settings to new environments, or more circumspectly, suppressing
their ability to overfit to any one environment. When using robots, the
environment includes the robot’s own body, its shape and material properties,
and how its actuators and sensors are distributed along its mechanical
structure. Here we demonstrate for the first time how one such design decision
(sensor placement) can alter the landscape of the loss function itself, either
expanding or shrinking the weight manifolds containing suitable controllers
for each individual task, thus increasing or decreasing their probability of
overlap across tasks, and thus reducing or inducing the potential for
catastrophic forgetting.

### 3.2 Introduction

It has been shown in various single-task settings how an appropriate robot
design can simplify the control problem [84, 85, 86, 87, 88, 89], but because
these robots were restricted to a single training environment, they did not
suffer catastrophic forgetting.

Catastrophic forgetting is a major and unsolved challenge in the machine
learning literature [56, 19, 20, 90]. Regardless of learning algorithm or task
domain, a neural network trained to perform task A and then challenged with
learning task B as well usually forgets A at the same rate as it learns B.
Such interference can also occur when an agent attempts to learn tasks A and B
simultaneously if gradients of improvement in A lead away from those of B.

In a multitask setting, [91] recently demonstrated that certain body plans
suffer catastrophic forgetting, while others do not. It was hypothesized that
a robot with the right morphology could in some cases alias separate tasks:
certain designs are able to move in such a way that a seemingly different
training instance converges sensorially to a familiar instance. However, this
conjecture was not isolated and tested. Likewise, the relationship between the
body and the loss landscape was not investigated.

In this paper, we provide a more thorough investigation on the role of
embodiment in catastrophic forgetting based on the assumption that in order to
avoid catastrophic forgetting, there must exist a set of control parameters
that are adequately performant across multiple task environments
simultaneously. Since a robot’s mechanical design can influence the set of
controller parameters suitable for each individual task environment, we here
test the hypothesis that a specific physical property of the robot’s
design|namely, the location of its sensors along its body|can help or hinder
continual learning by allowing for more or less overlap in suitable parameter
settings across multiple task environments.

Using a simple yet embodied agent as our model, we analytically and
empirically investigate how sensor location affects the weight manifolds of
the neural controller over multiple tasks. We show how morphological
optimization often results in asymmetrical and unintuitive sensor arrangements
with much more potential to allow learning algorithms to avoid catastrophic
forgetting than more intuitive, symmetrical designs. Thus, human designer
bias, while often useful, can sometimes inadvertently increase the likelihood
of catastrophic forgetting during learning. This suggests that we should
scrutinize our prior assumptions about the body plan of robots challenged with
continual learning, and where possible replace them with end-to-end data-
driven design automation.

### 3.3 Methods

A B C

Figure 3.1: Modeling the robot. A: The effect of lateral and contralateral
synaptic connections (adopted from [92]). B: The theoretical model with sensor
positions determined by $\ell_{1}$ and $\ell_{2}$. C: The simulated robot with
two light sensors (red), two motorized wheels (black), and a passive, anterior
castor wheel for balance (gray). The robot is drawn (A-C) with symmetrical,
anteriormost sensor placement, which we refer to in this paper as the
“canonical design”.

#### 3.3.1 The robot

The robot has a square frame, two separately-driven wheels, and two infrared
sensors (Fig. 3.1). The sensors detect light according to the inverse square
law, i.e., $1/d^{2}$, where $d$ is the distance from the light source;
occlusion was not modeled. The motors driving the wheels are contralaterally
connected to the sensors by weighted synapses yielding two trainable
parameters $w_{1},w_{2}\in[-1.0,1.0]$.

We here consider change to a single, isolated morphological attribute: the
physical location of the two sensors, which can be placed anywhere on the
dorsal surface of the robot’s square body. The location of the $i$-th sensor
$\ell_{i}$ can be described by its Cartesian coordinates $\ell_{i}=(x,y)$,
where $x,y\in[-0.5,0.5]$, and (0, 0) denotes the center of the body (Fig.
3.1B).

The effect of sensor location $\ell_{i}$ can be measured with respect to the
space, denoted $\theta$, of possible synapse weight pairs ($w_{1}$, $w_{2}$).
Since we cannot perform an exhaustive sweep over the infinitude of possible
sensor positions, we discretized each dimension of $\ell_{i}$ into nine
uniformly-spaced bins. Because sensors are varied in two dimensions ($x$ and
$y$) there are $9^{2}=81$ possible locations for each sensor; and because
there are two such sensors, the space $\theta$ is discretized into a 81-by-81
uniformly-spaced grid, thus yielding a searchable space of 6561 possible robot
designs.

For each of the 6561 designs, we conducted another sweep over the synapse
weights $(w_{1},w_{2})$, likewise discretizing each weight into 121 evenly-
space values, yielding $121^{2}=14641$ different weight configurations.
Finally, for each of the $6561\times 14641=96059601$ evaluated combinations of
sensor locations and weight values, we analyzed the robot analytically using
differential equations and empirically using a physics engine. These
discretizations were chosen to be as small as possible within the limit of our
computational resources and time.

#### 3.3.2 The task environments

The task is phototaxis in four environments, which differ in their position of
the light source in relation to the robot. The light source is placed at polar
coordinates $(r,\varphi)$ where
$\varphi\in\\{45^{\circ},135^{\circ},225^{\circ},315^{\circ}\\}$ and $r$ is a
fixed distance. A controller was considered successful for a given environment
if the robot comes within 0.2 cm of the light source at any time during its
evaluation period.

While there is of course a general strategy that solves the task for all
environments (follow the light), the easiest gradients to follow in the loss
landscape are initially those which produce forward locomotion in a single
direction and cause the robot to ignore the light. This is because, from the
robot’s perspective, due to the inverse square law of light decay, improving
its ability to move in the one environment with least loss earns quadratically
more reward than improvements to locomotion in any of the other three
environments in which the robot is less proficient. This causes the
catastrophic forgetting experienced by neural learning algorithms.

#### 3.3.3 The metrics

We here define two metrics: $M_{L}$ and $M_{CF}$, that are measured over $k=4$
environments. These metrics measure how a robot design impacts the weight
space of the controller and consequently measure how amenable to learning a
robot would have been if the controller were to be learned with a standard
learning algorithm rather than found by grid search. $M_{L}$ measures
controller learnability: how easy it would be to learn a generalist
controller. $M_{CF}$ measures resistance to catastrophic forgetting: the
probability that a environment-specific controller will generalize to other
environment.

Figure 3.2: A general example of overlapped binary success matrices for some
tasks A and B. Each element represents a different controller. Generalist
controllers sit inside the intersection $\theta^{*}$ of successful
environment-specific controllers $\theta_{k}^{*}$.

For each mechanical design $(\ell_{1},\ell_{2})$, we expect some optimal
manifolds $\theta_{k}^{*}$ in the space of control parameters $(w_{1},w_{2})$
to succeed for a specific environment $k$. For a controller to be successful
in multiple environments, it must reside within the intersection of
environment-specific manifolds, $theta^{*}$, on the loss surface. Thus, the
likelihood of finding a generalist controller|its learnability|will be
proportional to the size of the intersection ($M_{L}$). Likewise, a
controller’s potential to resist catastrophic forgetting ($M_{CF}$) will be
proportional to the ratio of generalist controllers (those successful in all
four environments) to specialists (those successful in at least one
environment).

Given a design $(\ell_{1},\ell_{2})$ and environment $k$, a binary success
matrix $S^{k}(\ell_{1},\ell_{2})$ is constructed such that each element
$S^{k}_{i,j}(\ell_{1},\ell_{2})$ is either 1 (success) or 0 (failure). By
overlapping the success matrices for a fixed design across the four
environments, we can visualize the manifolds $\theta_{k}^{*}$ where
$k\in\\{1,2,3,4\\}$ for the robot (Fig. 3.2).

We define the overlap $\mathcal{O}$ as a element-wise sum of the success
matrices over each environment $k$:

$\mathcal{O}=\sum_{k=1}^{4}S^{k}(\ell_{1},\ell_{2}).$ (3.1)

The learnability metric is simply the proportion of 4s (where a 4 represents
success in all 4 environments) in the overlapped success matrices to the
entire matrix space:

$M_{L}=\frac{g_{4}(\mathcal{O})}{n^{2}},$ (3.2)

where $g_{k}$ is a function that counts the total elements of a matrix with
value equal to $k$ and $n$ is the square dimension of the matrix defined by
the discrete parameter sweep.

Resistance to catastrophic forgetting is measured by:

$M_{CF}=\begin{cases}0\;\text{ if }\mathcal{O}\text{ is a null matrix},\\\
g_{4}(\mathcal{O})\left[\sum_{k=1}^{4}g_{k}(\mathcal{O})\right]^{-1}\;\text{otherwise}.\end{cases}$
(3.3)

which is the number of control parameters that solved all four environments
divided by the number of control parameters that solved at least one.

#### 3.3.4 The theoretical model.

The location and orientation of the robot can be defined by a system of
differential equations, where the change in position and orientation is
determined by the change in light captured by two sensors. Ignoring deviations
from the idealized environment, such as sensor noise and friction, the rate of
angular and linear velocities will be proportional to a linear combination of
the sensor values.

Let $\alpha(t)$ be the angle of the robot at time $t$, where $\alpha=0$
denotes the positive $x$ direction, and $\phi(t)=(x(t),y(t))$ be the position
of the robot in the world, then if the robot is located at the origin and
facing east ($\alpha=0$), its two light sensors are located exactly at
$\ell_{1}$ and $\ell_{2}$, and they each capture a some amount of light
$s_{1}(t)$ and $s_{2}(t)$, respectively.

Hence the absolute position of the $i$-th sensor is
$\phi(t)+R_{\alpha}\ell_{i}^{\hskip 3.0ptT}$, where

$R_{\alpha}=\begin{bmatrix}\cos\alpha&-\sin\alpha\\\ \sin\alpha&\cos\alpha\\\
\end{bmatrix}$ (3.4)

is the two-dimensional counterclockwise rotation matrix (in the amount
$\alpha$).

If we formulate the problem such that it is the robot’s initial position and
heading that is adjusted in each environment, instead of the position of the
light source, we can assume that the source is always at the origin. Then, the
distance of $\ell_{i}$ from the light source is given by:
$\|\phi(t)+R_{\alpha}\ell_{i}^{\hskip 3.0ptT}\|$. And since the intensity of
light is inversely proportional to the square of the distance, the sensor
values are given by:

$s_{i}(t)=c\cdot\|\phi(t)^{T}+R_{\alpha}{\ell_{i}}^{T}\|^{-2},$ (3.5)

where $c$ is a constant that we set equal to one.

Assuming the robot turns based on the difference between the sensor values
(with weights applied), the velocity of the robot is the average of the two
sensor values. Thus, the following system of equations determines the location
and orientation of the robot:

$\begin{cases}\dot{x}=v(t)\cos\alpha\\\ \dot{y}=v(t)\sin\alpha\\\
\dot{\alpha}=w_{1}s_{1}(t)-w_{2}s_{2}(t),\\\ \end{cases}$ (3.6)

where $v$ is the velocity of the robot given by
$2v(t)=w_{1}s_{1}(t)+w_{2}s_{2}(t).$

#### 3.3.5 The empirical model.

Because our theoretical model is highly abstracted from the real world and
built on a number of assumptions (no friction, motor limits, collisions, etc.)
which may potentially affect the robot’s behavior, we also empirically test
our claims by simulating the robots inside a physics engine.

The robot is simulated using Open Dynamics Engine (Fig. 3.1C). Just like the
theoretical model, the simulated robot contains two light sensors, which
innervate two motorized, spherical wheels (each with a single axis of
rotation), which are attached midway along the sides of a $1\times 1\times
0.13$ cm box. Additionally, an anterior passive castor wheel was added for
balance. Finally, a light source is simulated on the floor of the environment
at polar coordinates $(r,\alpha)$ as a fixed sphere with radius 0.2 cm. In
simulation, the behavior of a robot in a given environment is taken to be
successful if it collides with the light source at anytime during an
evaluation period of 2500 time steps ($dt=0.05$) or 125 seconds.

In order to replicate the baseline behavior of the canonical robot design it
was necessary to pre-optimize various physical attributes of the robot’s body,
including the mass of each component, the radii of the wheels, and the maximum
torque, speed, and target actuation rate. A multiobjective optimization
algorithm [93] was used to find a base morphology, with the sensors fixed in
the canonical position, that was both performant and stable. The first
objective was to maximize the performance of the robot (distance from the
light source), summed across all the four environments. The second objective
was to minimize the sum of the maximum torque, speed and target actuation
rate. This second objective is used to avoid both simulator instability and
behavior that is unlikely to transfer to reality.

After discovering a good base morphology, we performed the nested grid search
described in §3.3.1, for sensor locations ($\ell_{1}$, $\ell_{2}$) and weights
($w_{1},w_{2}$).

A B C

Figure 3.3: The best designs under the theoretical model according to
controller learnability (A; Metric $M_{L}$) and resistance to catastrophic
forgetting (B; Metric $M_{CF}$). Under the empirical model, the design with
the highest controller learnability was also the most resistant to
catastrophic forgetting (C). Although the design space we swept over contains
many symmetrical sensor arrangements, and most real robots utilize symmetrical
sensor distributions, the best designs are notably asymmetrical.  Figure 3.4:
Successful trajectories with canonical (symmetrical) sensor location under the
theoretical model. With canonical sensor placement $\\{\ell_{1}=(0.5,0.5)$,
$\ell_{2}=(0.5,-0.5)\\}$ (Fig. 3.1B), only 57 of the $121^{2}$ evaluated
controllers (0.4%) were successful all four environments. A: The trajectories
generated by one of the successful controllers $(w_{1},w_{2})=(0.6,0.98)$.
This controller initially generated phototaxis, but passed through the light
source and continued to move away from it. B: The trajectories generated by
another successful controller $(w_{1},w_{2})=(0.77,0.77)$. This controller
continuously spirals about the light source. The light source is drawn once at
the origin, and the initial positions/orientations of the robot relative to
the it are superimposed for the four environments.  Figure 3.5: Trajectories
of the design with maximal controller learnability, as measured by metric
$M_{1}$ (Eq. 3.2) under the theoretical model. With sensor locations
$\ell_{1}=(-0.5,-0.25)$ and $\ell_{2}=(0.5,0.25)$ (Fig. 3.3A), 2255 of the
$121^{2}$ evaluated controllers (15.4%) were successful all four environments.
A: The trajectories generated by one of the successful controllers,
parameterized by weights $(w_{1},w_{2})=(-0.85,0.82)$. B: The trajectories
with weights $(-0.8,0.6)$. C: The trajectories with weights $(-0.28,0.37)$.
The axes are equivalent to those in Fig. 3.4.

### 3.4 Results

#### 3.4.1 Theoretical results.

We employed SciPy (scipy.integrate.odeint) for numerical integration of the
robot’s location and orientation (Eq. 3.6), for $10^{5}$ timesteps.

For each evaluated mechanical design and controller (sensor locations and
synapse weights), the robot’s trajectory is computed in each of the four
environments defined in §3.3.2. As in the empirical model, if robot’s
trajectory comes within 0.2 units of the light source, the robot is determined
to have succeeded in that environment. Otherwise, it is determined to have
failed.

The mechanical design sketched in Fig. 3.3A (and its mirror image when
reflected about the sagittal plane) had the highest controller learnability
score, with $M_{L}=0.286$. However it did score the best in resistant to
catastrophic forgetting: the proportion of resistant to nonresistant
controllers for that design was $M_{CF}=0.636$, whereas several other found
designs had full resistance $M_{CF}=1$. But those with a perfect ratio
$M_{CF}=1$ had much smaller optimal weight manifold: the highest learnability
score achieved by this group was $M_{L}=0.206$. In other words, while all the
successful environment-specific controllers for these designs generalize
across all four environments, the manifold containing them is much smaller and
thus would be more difficult to find if controller parameters were to be
optimized by learning.

The canonical design had a much lower controller learnability ($M_{L}=0.049$)
and resistance to catastrophic forgetting ($M_{CF}=0.24$), than many found
asymmetrical designs.

For both the canonical, symmetrical design (Fig. 3.4) and the design with the
highest controller learnability score (Fig. 3.5) there are initial conditions
that generate persistent phototaxis: the robot moves toward the light source
and remains near it. However, whereas 35 of the 35 found phototaxing
controllers for the found design remain in the neighborhood of the source,
only 2 of the 6 found controllers for the canonical design do so. Some initial
conditions of the canonical design initially produce phototaxis, but the
design passes through the source and then continues to move away from it (Fig.
3.4A). This was not observed to occur with the “optimized” designs.

Figure 3.6: Measuring successful multitask learning. The distribution of
metrics $M_{L}$ and $M_{CF}$, for all evaluated designs, in the theoretical
(A, B), empirical (C, D). Metric $M_{L}$ (Eq. 3.2) indicates controller
learnability: the proportion of controllers deemed successful in all four
environments, for a given design. Metric $M_{CF}$ (Eq. 3.3) indicates
resistance to catastrophic forgetting: the ratio of the number of controllers
successful in all environments, over the number successful in at least one. If
no controllers are successful for a given design, $M_{CF}=0$.

#### 3.4.2 Empirical results.

As with the theoretical model the empirical model showed that non-intuitive
asymmetrical designs scored higher in learnability and in resistance to
catastrophic forgetting. However unlike the theoretical model one design
performed the best on both metrics.

The found asymmetrical design shown in Fig. 3.3C had both the highest
generalist controller learnability ($M_{L}=0.0039$) and resistance to
catastrophic forgetting ($M_{CF}=0.038$). Overall, there were 57 generalist
phototaxing controllers found (out of 14641 evaluated; 0.389%) for this
design, compared to only one generalist phototaxing controller found (0.0068%)
for the canonical, symmetrical design. The controller learnability of the
canonical design was thus $M_{L}=0.000068$; and its resistance to catastrophic
forgetting was $M_{CF}=0.00052$. Thus, the found asymmetrical design has both
higher controller learnability and resistance to catastrophic forgetting.

Figure 3.7: Measuring learnability and forgetting. For both the theoretical
(A-C) and empirical models (D-F), we performed a 121-by-121 grid search of
controller weights (14641 unique controllers) nested within a 81-by-81 grid
search for sensor locations (6561 unique designs). The controller space its
mapped for the canonical, symmetrical design (A, D), the design with highest
controller learnability ($M_{L}$; Eq. 3.2) (B, E), and the design most
resistant to catastrophic forgetting ($M_{CF}$; Eq. 3.3) (C, F). Under the
controller sweep on (D-F), the design with the highest controller learnability
also had the greatest resistance to forgetting, so E and are identical. Each
pixel represents a different controller $(w_{1},w_{2})$ for the given design,
and is colored by the number of environments that the combination successfully
exhibited phototaxis (i.e., the overlapped binary success matrices, defined by
Eq. 3.1). Under both the theoretical and empirical models, the unintuitive
asymmetrical designs (B, E) were found to have higher controller learnability
and greater resistance to forgetting in their landscape than their respective
canonical design (A, D) as measured by the number pixels in the heatmap that
are successful in all four environments (cyan). Likewise, the asymmetrical
designs (C, F) had higher resistance to catastrophic forgetting as measured by
the number of cyan pixels to non-blue pixels.

#### 3.4.3 Overview.

In Fig. 3.7 the successes of weight manifolds for all of these design in both
the theoretical and empirical model can be seen in detail, where cyan
represents weight assignments that succeed in all for environments for a given
design. These weight manifolds show clearly that in this case the weight
assignments for the asymmetrical would be much easier to find by a learning
algorithm while the canonical design is akin to looking for a needle in a
haystack.

Fig. 3.6 plots the frequency of metrics $M_{L}$ and $M_{CF}$ (Eqs. 3.2 and
3.3, respectively) within each bin of the grid search. This again shows how
there are many designs (including intuitive symmetric ones) that score poorly
on $M_{L}$ and $M_{CF}$ while there are relatively few designs that perform
well. Thus a given design has a drastic effect on the theoretical learnability
of a robots controller parameters.

### 3.5 Discussion

In this paper, we considered a simple robot and task in order to sample the
entire loss landscape of the weight manifold at a relatively high resolution.
While we haven’t tested these robot with any specific learning algorithm, our
results suggest that changes in one element of a robot’s design (sensor
location) can fundamentally alter the loss surface, thus influencing the
controller’s learnability, and resistance to catastrophic forgetting. More
specifically, by changing sensor location, we observed changes in the number
and placement along the loss surface of control parameters suitable for
individual environments, as well in how these optimal yet environment-specific
parameters overlapped across different environments to produce generalist
controllers which resist catastrophic forgetting. However, we acknowledge that
this work mainly builds a theoretical foundation and that our metrics need to
be tested against existing methods for learning.

Previous efforts to avoid catastrophic forgetting have relied almost
exclusively on increased control complexity. Most were focused on making
changes to small subsets of neural network weights [20, 90, 94, 95, 96, 97,
98, 99]. Others have attempted to sidestep the problem by learning good
initial weights such that they can be quickly updated when switching between
tasks [25, 100]. We have shown here that, in theory, regardless of the
algorithm used it is also possible to alleviate catastrophic forgetting by
changing aspects of the robot’s design, without increasing control complexity,
but doing so can be non-intuitive.

We found that even the seemingly trivial case of phototaxis with contralateral
connections described by [92] can require morphological tuning to work as
expected in a single simulated environment, and that, when challenged to
perform in additional environments, other adjustments in morphology,
specifically to sensor location, could either suppress or multiply the
potential for catastrophic forgetting by expanding or shrinking the overlap of
performant controller settings for that body plan across different task
environments.

The physical location of sensors is thus a relevant property of robots that is
nevertheless abstracted away in the (mostly disembodied) systems that address
catastrophic forgetting reported in the literature to date. While sensor
location could in principle be dynamically controlled via a lattice of sensors
[101] or adjustable antenna [102], change in (and rational control over) other
morphological attributes|such as geometry [88], material properties [103], or
the number and placement of actuators [30]|is much more difficult in practice,
and such design elements are almost always presupposed and fixed prior to
training [31].

However, unless experimental proof is obtained in the real world, this theory
will remain speculation. In fact it is possible that the proposed empirical
model using rigid body physics was more disconnected from reality than our
theoretical model. The simulated wheels, for instance, have just a single
point of contact with the ground. A more realistic surface contact geometry
might completely change the optimal sensor locations, but there’s also reason
to believe that the loss surface manifolds containing adequate controllers for
a compliant body could be larger than those of a rigid body [88, 73], further
increasing the probability of overlap across tasks.

In the limit, machines with the right morphology may use a single controller
to accomplish a set of tasks that appear disparate to a robot with a different
body plan. For example, a granular jamming gripper [86] need not precisely
control the placement of each joint around differently shaped objects: a
single policy (vacuum air, hold, relax) works regardless of object shape.
However, this control policy is exceedingly simple. The degree to which
morphology influences learnability in more complex robots, task environments
and behaviors has yet to be investigated, but will be the focus of future
work.

In this work, two control- and two morphology parameters were optimized. In
future work we will investigate whether co-optimizing the morphology and
control parameters confers greater overall learnability on the robot compared
to a robot with a fixed morphology and four control parameters. This will help
determine whether a poorly chosen mechanical design can be compensated for by
increased control complexity.

## Chapter 4 A good body is all you need: avoiding catastrophic interference
via agent architecture search

### 4.1 Abstract

In robotics, catastrophic interference continues to restrain policy training
across environments. Efforts to combat catastrophic interference to date focus
on novel neural architectures or training methods, with a recent emphasis on
policies with good initial settings that facilitate training in new
environments. However, none of these methods to date have taken into account
how the physical architecture of the robot can obstruct or facilitate
catastrophic interference, just as the choice of neural architecture can. In
previous work we have shown how aspects of a robot’s physical structure
(specifically, sensor placement) can facilitate policy learning by increasing
the fraction of optimal policies for a given physical structure. Here we show
for the first time that this proxy measure of catastrophic interference
correlates with sample efficiency across several search methods, proving that
favorable loss landscapes can be induced by the correct choice of physical
structure. We show that such structures can be found via co-
optimization—optimization of a robot’s structure and control policy
simultaneously—yielding catastrophic interference resistant robot structures
and policies, and that this is more efficient than control policy optimization
alone. Finally, we show that such structures exhibit sensor homeostasis across
environments and introduce this as the mechanism by which certain robots
overcome catastrophic interference.

### 4.2 Introduction

Catastrophic interference is a phenomenon that occurs when training policies
on multiple environments simultaneously: a training step that results in
improvement in one environment causes a greater reduction in performance in
other environments. This comes with various consequences, one of which are
policies that specialize to only one of the environments. This problem is
often related to the problem of catastrophic forgetting [56, 19, 20, 90],
however in catastrophic forgetting environments are trained sequentially,
where in catastrophic interference they are trained simultaneously as stated.

Many of the methods used to combat catastrophic intereference rely solely on
advances to neural architecture or training methods [104]. Most are focused on
making changes to small subsets of neural network weights [20, 90, 94, 95, 96,
97, 98, 99]. Others have attempted to sidestep the problem by learning good
initial weights such that they can be quickly updated when switching between
tasks [25, 100].

None of these methods however consider the effects that the architecture of
the robot housing the policy has on this phenomenon. The robot’s body is often
considered a part of the environment [105]. From this perspective, it is the
only part of the task environment that is changeable prior to or during
training. Consequently, it has been shown in various task settings that an
appropriate robot design can simplify the problem of learning a sufficient
control policy [84, 85, 86, 87, 88, 89]. However, so far these findings have
been restricted to examples with a single training environment; they did not
need to overcome multi-domain problems such as catastrophic interference. In a
multitask setting, [91] recently demonstrated that certain body plans suffer
catastrophic interference, while others do not. It was hypothesized there that
a robot with an appropriate design could in some cases alias separate tasks:
certain designs are able to move so that seemingly different environments
converge sensorially to a common and familiar observation. This is not unlike
tilting one’s head to recognize a familiar face in a rotated image. However,
this conjecture was not isolated and tested. Likewise, the relationship
between the body and the loss landscape was not investigated.

Here, we provide a more thorough investigation of the role of embodiment in
catastrophic interference, based on the assumption that in order to avoid
catastrophic interference, there must exist a set of policy parameters that
yield adequate performance across several task environments. Since a robot’s
mechanical design can change which sets of policy parameters are appropriate
for each individual task environment, we here test the hypothesis that a
specific physical property of the robot’s design—sensor distribution—can help
or hinder learning by inducing more or less overlap in specialized policy
parameter sets across multiple task environments. These regions of overlap
thus correspond to sets of general purpose policy parameters.

We first define metrics to explicitly measure the distribution and number of
specialized and general policy parameter sets for a given robot design. We
then tested whether those metrics actually correlate with sample efficiency
for different training algorithms, and found that they do for all of the
tested algorithms. We show how robot designs that facilitate learning can be
found by co-optimizing policy and sensor distribution parameters. We found
that co-optimization results in significantly better sample efficiency than
policy optimization alone. Lastly, we show that the designs found during co-
optimization follow a sensor homeostasis gradient. This leads to the
hypotheses that part of what allows an agent’s body to facilitate policy
search is its ability to facilitate sensor homeostasis, and that including
homeostasis in future loss functions may further improve sample efficiency in
multi domain policy and agent design training.

### 4.3 Methods

#### 4.3.1 Experimental Overview

A minimal agent, task, and environment were employed to test the hypothesis
that co-optimizing policy and structure parameters increases sample efficiency
during training: agents are trained to approach a light source, and they are
placed in environments with differing light source locations.

The robot is modeled as having a square frame (edge length $0.5m$) with two
separately-driven wheels and two infrared sensors. A baseline, hand designed
robot can be seen in Fig. 4.1. The sensors detect light according to the
inverse square law, i.e., $1/d^{2}$, where $d$ is the distance from the light
source and are modeled without occlusion. The motors driving the wheels are
contralaterally connected to the sensors by weighted synapses yielding two
trainable parameters $w_{1},w_{2}\in[-1.0,1.0]$.

Figure 4.1: The robot is modeled as having a square frame with two separately-
driven wheels and two infrared sensors located on the dorsal surface of the
frame at locations defined by two vectors emanating from the center of the
frame $\ell_{1}$ and $\ell_{2}$. The vector $v$ shows the forward direction.
This design is a hand designed baseline based on a Braitenberg Vehicle [92].

In this paper we explore change to a single design attribute: the physical
location of the two sensors, which can be placed anywhere on the dorsal
surface of the robot’s square body. The location of the $i$-th sensor
$\ell_{i}$ can be described by its Cartesian coordinates $\ell_{i}=(x,y)$,
where $x,y\in[-0.5,0.5]$, and (0, 0) denotes the center of the body (Fig.
4.1).

If we define $\theta$ as the space of all possible synapse weight pairs
($w_{1}$, $w_{2}$), we can investigate how a given sensor distribution
($\ell_{1}$, $\ell_{2}$) affects $\theta$. Since we cannot perform an
exhaustive sweep over the infinitude of possible sensor positions, we
discretize each dimension of $\ell_{i}$ into nine uniformly-spaced bins.
Because sensors are varied in two dimensions ($x$ and $y$) there are
$9^{2}=81$ possible locations for each sensor. Since there are two sensors,
and each can be in 81 possible discrete locations, the total design space
consists of $81^{2}=6561$ possible robot designs.

For each of these 6561 designs, we conducted another sweep over the synapse
weights $(w_{1},w_{2})$, likewise discretizing each weight into 121 evenly-
spaced values within $[-1,1]$, yielding $121^{2}=14641$ different weight
controllers per robot design. For each of the $6561\times 14641=96059601$
robot design and policy pairs, we evaluated the simulated robot for a
phototaxis task across four environments as described below in §4.3.2. Doing a
grid search as described creates an exhaustive investigation of the weight
space ($\theta$) for each robot design, thus allowing an analysis as described
in §4.3.3 to measure the predicted ability of a robot design to facilitate
policy training. We then applied four different training algorithms to each
robot design (see §4.3.4) to test for any correlation between the utility of a
given design predicted by our metrics and actual performance from training
policies on it. Finally, we compare policy optimization using the default
design against simultaneous co-optimization of design and policy (see §4.3.5),
and analyze the resulting optimized designs.

All of the experiments were run in parallel on a cpu computing cluster that is
powdered by $95\%$ renewable energy. In total the experiments took $564,786$
cpu hours.

#### 4.3.2 Simulating the robot.

The robot is trained on a phototaxis task across four environments: the robot
should move toward a light source in each environment. In each environment a
light is placed in the center of an $(x,y)$ plane and the robot is placed
diagonally at a distance of $8$ body lengths away from the light source;
specifically at positions $(d,d)$, $(d,-d)$, $(-d,d)$, $(-d,-d)$ where
$d=\frac{4}{\sqrt{2}}$ constitutes the four differing environments. Success in
an environment is determined by how close the robot comes to the light source.
We assume the light source has a radius ($r=0.075m$) that determines its
physical size, and maximum loss is defined to be reached if the robot touches
the light source during the course of its evaluation.

Given the robot’s and environment’s simplicity, rather than using a simulated
robot and task environment, the location and orientation of the robot in the
plane can be defined by a system of differential equations, where the change
in position and orientation is determined by the change in light captured by
two sensors. Ignoring deviations from the idealized environment, such as
sensor noise and friction, the rate of angular and linear velocities will be
proportional to a linear combination of the sensor values.

Let $\alpha(t)$ denote the angle of the robot at time $t$, where $\alpha=0$
denotes the positive $x$ direction, and $\phi(t)=(x(t),y(t))$ denote the
position of the robot in the world. If the robot is located at the origin and
facing east ($\alpha=0$), its two light sensors are located exactly at
$\ell_{1}$ and $\ell_{2}$, and they each capture some amount of light
$s_{1}(t)$ and $s_{2}(t)$, respectively.

Hence the absolute position of the $i$-th sensor is
$\phi(t)+R_{\alpha}\ell_{i}^{\hskip 3.0ptT}$, where

$R_{\alpha}=\begin{bmatrix}\cos\alpha&-\sin\alpha\\\ \sin\alpha&\cos\alpha\\\
\end{bmatrix}$ (4.1)

is the two-dimensional counterclockwise rotation matrix (in the amount
$\alpha$).

Since we formulate the problem such that it is the robot’s initial position
and heading that is adjusted in each environment instead of the position of
the light source, we can assume that the light source is always at the origin.
Then, the distance of $\ell_{i}$ from the light source is given by:
$\|\phi(t)+R_{\alpha}\ell_{i}^{\hskip 3.0ptT}\|$. And since the intensity of
light is inversely proportional to the square of the distance, the sensor
values are given by:

$s_{i}(t)=\|\phi(t)^{T}+R_{\alpha}{\ell_{i}}^{T}\|^{-2},$ (4.2)

The robot turns based on the difference between the sensor values multiplied
by the two neural network weights, the velocity of the robot is equal to the
average of the two sensor values. Thus, the following determines the location
and orientation of the robot:

$\begin{cases}\dot{x}=v(t)\cos\alpha\\\ \dot{y}=v(t)\sin\alpha\\\
\dot{\alpha}=w_{1}s_{1}(t)-w_{2}s_{2}(t),\\\ \end{cases}$ (4.3)

where $v$ is the velocity of the robot given by
$2v(t)=w_{1}s_{1}(t)+w_{2}s_{2}(t).$

#### 4.3.3 Metrics

We here define two metrics $M_{L}$ and $M_{CI}$ that are measured over the
$K=4$ environments. These metrics measure how a robot design impacts the
weight space of the control policy and consequently provide a prediction of
sample efficiency, if policy training were performed using that design.
$M_{L}$ measures controller learnability: how easy it would be to learn a
generalist controller. $M_{CI}$ measures resistance to catastrophic
interference: the probability that starting training with an environment-
specific controller will result in discovery of a generalist controller.

Figure 4.2: An example of overlapped binary success matrices for two
environments A and B. Each element represents a different set of policy
parameters. Generalist controllers lie within the intersection $\theta^{*}$ of
successful environment-specific controllers $\theta_{k}^{*}$.

Given a design $(\ell_{1},\ell_{2})$ and environment $k$, a binary success
matrix $S^{(k)}(\ell_{1},\ell_{2})$ is constructed such that each element
$S^{(k)}_{i,j}(\ell_{1},\ell_{2})$ is either 1 (the light source was touched)
or 0 (it was not) and each element corresponds to particular values of weights
$w_{1}$ and $w_{2}$. By overlapping the success matrices for a fixed design
across the four environments, we can visualize the manifolds $\theta_{k}^{*}$
where $k\in\\{1,2,3,4\\}$ for the robot (Fig. 2.2).

We define the overlap $\mathcal{O}$ as an element-wise sum of the success
matrices over each environment $k$:

$\mathcal{O}=\sum_{k=1}^{K}S^{(k)}(\ell_{1},\ell_{2}).$ (4.4)

The learnability metric is simply the proportion of $|K|=4$ values, which
denote success in all environments, in the overlapped success matrices,
compared to the entire matrix space:

$M_{L}=\frac{g_{|K|}(\mathcal{O})}{n^{2}},$ (4.5)

where $g_{k}$ is a function that counts the total elements of a matrix with
value equal to $k$ and $n$ is the square dimension of the matrix defined by
the discrete policy parameter set sweep.

Resistance to catastrophic interference is measured by:

$M_{CI}=\begin{cases}0\;\text{ if }\mathcal{O}\text{ is a null matrix},\\\
g_{|K|}(\mathcal{O})\left[\sum_{k=1}^{K}g_{k}(\mathcal{O})\right]^{-1}\;\text{otherwise}.\end{cases}$
(4.6)

which is the number of policy parameter sets that solved all $K$ environments
divided by the number of sets that solved at least one.

#### 4.3.4 Training

We used four different training methods to tune the weights of the control
policies: random search, generating set search, separable natural evolution
strategies (NES), and differential evolution (DE). These methods were chosen
because they encompass different paradigms in optimization with the first
(random search) serving as a baseline. Generating set search is considered a
powerful derivative free algorithm from the direct search paradigm [106].
Natural evolution strategies is another derivative free method loosely based
on biological evolution [107]. Differential evolution is also based on
biological evolution but is generally considered to be in a different class of
evolutionary based optimizers than NES [108]. Thus, together, these methods
give motivation that the results from this research extend beyond the actual
methods used111All of the search methods were used as is from
https://github.com/robertfeldt/BlackBoxOptim.jl using the default hyper
parameters from the code base..

We applied each training method to each of the $6561$ robot designs, five
times using different initial random policies. For each method the loss
function was computed as the sum of the minimum distance from the light source
achieved in every environment. During training we also kept track of the total
number of environments in which the robot was successful. We then compute the
average number of evaluations it took to achieve success in all environments
for a given training method and plot them against the designs’ metrics $M_{L}$
and $M_{CI}$ to measure the Pearson correlation and corresponding p-value. A
strong negative correlation is desirable here: designs with higher metrics
require a lower number of evaluations to achieve optimality. This would
indicate that good designs predicted by the metrics yielded significantly
better sample efficiency.

#### 4.3.5 Co-Optimization

To simultaneously optimize both the controller and sensor placement of the
robot we use the Borg Multiobjective Evolutionary Algorithm (Borg MOEA) [109].
As before the goal of the algorithm is to produce a policy that works in all
four environments in as few evaluations as possible. However rather than
optimize directly on the sum of score in each environment we treat each of
these scores as it’s own objective, and pick the best solution at the end
based on the aggregate sum of the objectives. The algorithm works on 6
trainable parameters: the 2 values for the first light sensor vector, the 2
values for the other light sensor vector, and the 2 weight values of the
controller.

We compare this to the same algorithm, but fix the design of the robot to the
baseline design in Fig. 4.1. In this baseline the algorithm only optimizes 2
trainable parameters which are the 2 weights of the controller.

We run each algorithm over 30 different random seeds and take the average
number of evaluations to required to achieve success in all four environments.
We then run a Mann Whitney U Test to compare if there were any significant
differences between the two methods and report a corresponding p-value.

Lastly, we analyze the sensor values experienced by designs found over the
course of co-optimization with dynamic time warping (DTW) [110]. The dynamic
time warping algorithm is a state-of-the-art way to measure the similarity
between two signals by determining the cost to align the signals via
stretching and/or shifting. For each environment we record a vector of the
light sensor signals experienced (four in total). For one of the light sensor
we compute the average DTW distance between all combinations of experienced
signals (six comparisons) to get a signal score for the light sensor, we then
do the same for the other light sensor. We average both of those scores to get
the aggregate DTW distance for the design, the lower this number is the more
similar the light sensor signals were between environments.

Figure 4.3: Examples of different robot designs with corresponding policy
trace, success matrices, and metrics. The vectors $\ell_{1}$ and $\ell_{2}$
show the placement of the light sensors for a specific design. The colors in
the second column show the trace of the $(x,y)$ coordinates of the robot in
the four different starting positions as it moves toward the light source
represented by the yellow point. The last column shows the weight assignments
that solve different environments; red assignments solve no environments,
while any other color solves at least one, and blue assignments solve all four
environments. (A) The baseline morphology as shown in Fig. 4.1. (B) The design
with the best learnability metric ($M_{L}$). (C) The design with the best
resistance to catastrophic forgetting ($M_{CI}$) Figure 4.4: The correlation
between design learnability and resistance to catastrophic interference. Each
point is a design with its learnability on the x-axis and resistance on the
y-axis. The two variables are strongly correlated in this case. The density of
points is more concentrated near the origin showing that high scoring designs
are relatively rare.

Random Search

Generating Set Search

Natural Evolution Strategy

Differential Evolution

Figure 4.5: For each design we plot sample efficiency against the design’s
learnability ($M_{L}$) and resistance to interference ($M_{CI}$) scores across
the search methods. In each case there is a significant negative correlation
demonstrating that a design’s score is predictive of the required effort to
train a policy for that design.

Figure 4.6: (Left) Average number of environments a policy is successful in
(Average Success) as a function of evaluation time, for policy optimization
using the baseline design (red) and co-optimization of design and control
(blue). Co-optimization was significantly better that controller optimization
alone. (Right) The average Dynamic Time Warping (DTW) distance between the
robots’ sensors in all of the environments for the robot designs found during
co-optimization. As optimization progresses, designs progress from those that
experience environments differently (high DTW) to those that experience
differing environments more similarly (low DTW). Shading in both panels
reports the $95\%$ confidence interval.

### 4.4 Results

We employed the DifferentialEquations.jl package [111] for numerical
integration of the robot’s location and orientation using standard Euler
Integration with a time step of $0.1$ for $10^{5}$ time steps.

For each evaluated mechanical design and policy (sensor locations and synapse
weights), the robot’s trajectory is computed in each of the four environments
defined in §4.3.2. Again, if robot’s trajectory comes within 0.075 meters of
the light source, the robot is determined to have succeeded in that
environment. Otherwise, it is determined to have failed.

The mechanical design sketched in Fig. 4.3B had the highest learnability
score, with $M_{L}=0.1206$. A very similar design (Fig. 4.3C) scored the best
in resistant to catastrophic interference: the proportion of resistant to
nonresistant policies for that design was $M_{CI}=0.6479$. In general the best
policies for both metrics featured a specific similar design feature of
asymmetric placement of the light sensors. The baseline design (Fig. 4.3A) had
a much lower learnability ($M_{L}=0.0018$) and resistance to catastrophic
interference ($M_{CF}=0.0142$) then the best found asymmetrical designs.

The worst designs for both metrics have scores of zero and feature designs
where the lights sensors are both placed in the same location or extremely
close to one another. This makes particular sense because the closer the light
sensors are the less differential there is between signals thus making it
difficult or impossible (in the case of overlapping placement) for the robot
to distinguish between different environments with out some sore of behavioral
memory mechanism such, as a recurrent network.

In Fig. 4.3 the successes matrices of weight manifolds for all the mentioned
designs can be seen in detail, where dark blue represents weight assignments
that succeed in all for environments for a given design. These weight
manifolds show clearly that in this case the optimal weight assignments for
the asymmetrical designs should be much easier to find during training, while
the baseline design is akin to looking for a needle in a haystack. This also
shows that for this specific robot it appears a high score in one metric
correlates with the others. We specifically tested this by plotting the metric
scores for all of the designs along with the correlation between the two
metrics. As shown in Fig. 4.4 there is a significant positive correlation
($p<0.001$) between the metrics, thus giving an explanation for why the best
designs under both metrics are so similar.

We can also see that based on the frequency of metrics $M_{L}$ and $M_{CF}$
that there are many designs (including intuitive symmetric ones) that score
poorly on $M_{L}$ and $M_{CF}$ while there are relatively few designs that
perform well.

When we conducted the training of the policies for each of these designs as
described in §4.3.4, we found that a design’s score over the metrics was
significantly correlated with the number of evaluations required to solve the
phototaxis task for all four environments. This can be seen in Fig. 4.5. On
the x-axis the metric is plotted and the y-axis is the average number of
evaluations required to achieve success in all four environments, thus we see
a strong negative correlation across the methods with a p-value less that
$0.001$, meaning that a design’s metric score was predictive of the required
effort to train that design on the task.

Lastly, we compared co-optimization of the policy and robot design as
described in §4.3.5 versus policy optimization of the baseline design. As can
be seen in Fig. 4.6, co-optimization significantly outperformed ($p<0.001)$
policy optimization alone, despite increased complexity in the dimension of
optimizable parameters. In fact, in the number of evaluations attempted,
optimizing the policy for the baseline design never achieved competence in all
four environments, and thus, that design ultimately succumbed catastrophic
interference. We further analyzed the designs found by co-optimization (Fig.
4.6) and discovered that they progressively move from designs with high DTW
distance scores to ones with low DTW scores. Thus without any guidance, the
optimization naturally found a homeostatic design (one that sensorally
experiences differing environments similarly) or a design which was best
suited to optimization in multiple environments.

### 4.5 Discussion

Here, a simple robot and task were employed to investigate the entire loss
landscape of the weight manifold at high resolution. In our sweep across
policy parameter sets for many robot designs, the results suggested that
changes in one element of a robot’s design (sensor distribution) greatly
altered the loss surface. We hypothesized that this is likely to influence
sample efficiency during policy optimization for a given design, and that
design’s resistance to catastrophic interference. This hypothesis was
strengthened by the observation that changing sensor location induced changes
in the number and placement across the loss surface of specialized parameter
sets optimal for one environment, as well as the size of specialist overlap
regions containing optimal generalized parameter sets that confer resistance
to catastrophic interference.

We confirmed this hypothesis for four orthogonal training methods: we found
that a robot design’s learnability and resistance scores correlated with its
sample efficiency, for this phototaxis task.

Lastly, we demonstrated that these high scoring designs could be found via
simultaneous co-optimization of robot design and control, and that doing so
was significantly more sample efficient than optimizing just the policy for a
manually designed baseline robot. We concluded with evidence showing sensor
homeostasis increased as robot designs improved, even though the loss function
does not reward homeostasis. This suggests a causal link between agent design
and sample efficient policy search: some designs better enable a robot to move
such that observations converge across seemingly different environments, and
allow it to perform the same, appropriate actions when aligned appropriately
in each environment.

Even in the seemingly trivial case of phototaxis with contralateral
connections on a minimal robot as first described by [92], we found that when
challenged to perform in multiple environments, adjustments in design,
specifically to sensor location, could either suppress or exacerbate
catastrophic interference by expanding or shrinking the overlap of optimal
policy parameters for that design across different task environments.

The physical location of sensors is thus a relevant property of robots
nevertheless abstracted away in the mostly disembodied systems that address
catastrophic interference reported in the literature to date. While sensor
location could in principle be dynamically controlled via a lattice of sensors
[101] or adjustable antenna [102], change in (and rational control over) other
design attributes such as geometry [88], material properties [103], or the
number and placement of actuators [30] is much more difficult in practice, and
such design elements are almost always presupposed and fixed prior to training
[31].

Overall we found that machines with the right design may use a single policy
to succeed at a set of tasks that appear different to a robot with an ill
suited body plan. The degree to which design influences learnability in more
complex robots, task environments and behaviors has yet to be investigated,
but will be the focus of future work.

## Chapter 5 A soft robot that adapts to environments through shape change

### 5.1 Introduction

Nature provides several examples of organisms that utilize shape change as a
means of operating in challenging, dynamic environments. For example, the
spider Araneus Rechenbergi [112, 113] and the caterpillar of the Mother-of-
Pearl Moth (Pleurotya ruralis) [114] transition from walking gaits to rolling
in an attempt to escape predation. Across larger time scales, caterpillar-to-
butterfly metamorphosis enables land to air transitions, while mobile to
sessile metamorphosis, as observed in sea squirts, is accompanied by radical
morphological change. Inspired by such change, engineers have created
caterpillar-like rolling [115], modular [116, 117, 118], tensegrity [119,
120], plant-like growing [121], and origami [122, 123] robots that are capable
of some degree of shape change. However, progress toward robots which
dynamically adapt their resting shape to attain different modes of locomotion
is still limited. Further, design of such robots and their controllers is
still a manually intensive process.

Despite the growing recognition of the importance of morphology and embodiment
on enabling intelligent behavior in robots [124], most previous studies have
approached the challenge of operating in multiple environments primarily
through the design of appropriate control strategies. For example, engineers
have created robots which can adapt their gaits to locomote over different
types of terrain [125, 126, 127], transition from water to land [128, 129],
and transition from air to ground [130, 131, 132]. Other research has
considered how control policies should change in response to changing loading
conditions [133, 134], or where the robot’s body was damaged [78, 135, 136].
Algorithms have also been proposed to exploit gait changes that result from
changing the relative location of modules and actuators [137], or tuning
mechanical parameters, such as stiffness [138]. In such approaches, the
resting dimensions of the robot’s components remained constant. These robots
could not, for instance, actively switch their body shape between a
quadrupedal form and a rolling-optimized shape.

The emerging field of soft robotics holds promise for building shape-changing
machines [139]. For example, one robot switched between spherical and
cylindrical shapes using an external magnetic field, which could potentially
be useful for navigating internal organs such as the esophagus and stomach
[140]. Robotic skins wrapped around sculptable materials were shown to morph
between radially-symmetric shapes such as cylinders and dumbbells to use
shape-change as a way to avoid obstacles [141]. Lee et al. proposed a hybrid
soft-hard robot that could enlarge its wheels and climb onto step-like
platforms [142]. A simulated soft robot was evolved to automatically regain
locomotion capability after unanticipated damage, by deforming the shape of
its remnant structure [88]. With the exception of the study by Kriegman et al.
[88], control strategies and metamorphosis were manually programmed into the
robots, thereby limiting such robots to shapes and controllers that human
intuition is capable of designing. However, there may exist non-intuitive
shape-behavior pairings that yield improved task performance in a given
environment. Furthermore, manufacturing physical robots is time-consuming and
expensive relative to robot simulators such as VoxCad [143], yet discovering
viable shape-behavior pairs and transferring simulated robots to functioning
physical hardware remains a challenge. Although many simulation to reality
(“sim2real”) methods have been reported [144, 30, 78, 147, 135, 148, 149, 150,
151, 145, 146], none have documented the transfer from simulation to reality
of shape-changing robots.

To test whether situations exist where shape change improves a robot’s overall
average locomotion speed within a set of environments more effectively than
control adaptations, here we present a robot which actively controls its shape
to locomote in two different environments: flat and inclined surfaces (Fig.
5.1). The robot had an internal bladder, which it could inflate/deflate to
change shape, and a single set of external inflatable bladders which could be
used for locomotion. Depending on the core’s shape, the actuators created
different motions, which could allow the robot to develop new gaits and gain
access to additional environments. Within a soft multi-material simulator, an
iterative “hill-climbing” algorithm [152] generated multiple shapes and
controllers for the robot, then automatically modified the robots’ shapes and
controllers to discover new locomotion strategies. No shape-controller pairs
were found that could locomote efficiently in both environments. However, even
relatively small changes in shape could be paired with control policy
adaptations to achieve locomotion within the two environments. In flat and
even slightly inclined environments, the robot’s fastest strategy was to
inflate and roll. At slopes above a critical transition angle, we could
increase the robot’s speed by flattening it and programming it with an
inchworm gait to exhibit an inchworm gait. A physical robot was then designed
and manufactured to achieve similar shape-changing ability and gaits (Fig.
5.2). When placed in real-world analogs of the two simulated environments, the
physical robot was able to change shape to locomote with two distinct
environmentally-effective gaits, demonstrating that shape change is a
physically realistic adaptation strategy for robots.

Figure 5.1: Shape change can result in faster locomotion speeds than control
adaptation, when a robot must operate in multiple environments. a, Using
inflatable external bladders, rolling was the most effective gait on flat
ground. b, Rolling was ineffective on the inclined surface. We discovered a
flat shape (achieved by deflating the inner bladder; c) and crawling gait (d)
that allowed the robot to succeed in this environment. e,f, After we
discovered these strategies in simulation, we transferred these strategies for
rolling (e) and inchworm motion (f) to real hardware. Scale bars, 5 cm. Figure
5.2: Simulation revealed successful shapes and controllers, which we attempted
to realize in hardware. Sets consisting of a shape, an orientation, and a
controller were generated for the robot in simulation. Each numbered sub-panel
depicts a single automatically generated parameter set. After running
simulations to determine the speed of each set, some were deemed too slow,
while successful (relatively quick) sets were used to design a single physical
robot that could reproduce the shapes and gaits found in simulation for both
environments. During prototyping, actuator limits were measured and
incorporated into the simulator to improve the accuracy of the simulation.

This work points toward the creation of a pipeline that takes as input a
desired objective within specified environments, automatically searches in
simulation for appropriate shape and control policy pairs for each
environment, and then searches for transformations between the most successful
shapes. If transformations between successful shapes can be be found, those
shape/behavior pairs are output as instructions for designing the
metamorphosing physical machine. Here, we demonstrate that at least some
shape/behavior pairs, as well as changes between shapes, can be transferred to
reality. Thus, this work represents an important step toward an end-to-end
pipeline for shape-changing soft robots that meet the demands of dynamic,
real-world environments.

### 5.2 Results

#### 5.2.1 The simulated robot.

We sought to automate search for efficient robot shapes and control policies
in simulation, to test our hypothesis that shape and controller adaptation can
improve locomotion speeds across changing environments more effectively when
given a fixed amount of computational resources, as compared to controller
adaption only. To verify that multiple locomotion gaits were possible with the
proposed robot design, we first used our intuition to create two hand-designed
shape and control policies: one for rolling while inflated in a cylindrical
shape (Fig. 5.1a), and the other for inchworm motion while flattened (Fig.
5.1d). Briefly, the rolling gait consisted of inflating the trailing-edge
bladder to tip the robot forward, then inflating one actuator at a time in
sequence. The hand-designed “inchworm” gait consisted of inflating the four
upward-facing bladders simultaneously to bend the robot in an arc. We then
performed three pairs of experiments in simulation. Within each pair, the
first experiment automatically sought robot parameters for flat ground; the
second experiment sought parameters for the inclined plane. Each successive
pair of experiments allowed the optimization routine to control an additional
set of the robot’s parameters, allowing us to measure the marginal benefit of
adapting each parameter set when given identical computational resources
(summarized in Table 5.1 and Fig. 5.3). The three free parameter sets of our
shape-changing robot are shape, orientation relative to the contour (equal-
elevation) lines of the environment, and control policy (Fig. 5.3a). This
sequence of experiments sought to determine whether optimization could find
successful parameter sets in a high-dimensional search space, while also
attempting to determine to what degree shape change was necessary and
beneficial.

In all experiments, fitness was computed as the distance the simulated robot
traveled in a desired direction for a fixed time period. To facilitate
comparison with speeds achieved by the physical robot, the average speed of
the simulated robot, when using hand-designed or evolved shapes and policies,
was computed and is reported herein in body lengths/second (BL/s) attained
over flat ground or uphill, depending on the current environment of interest,
during a fixed period of time.

Parameters for the simulation were initialized based on observations of
previous robots [141, 153], and adjusted to reduce the simulation-to-reality
gap after preliminary tests with physical hardware (see Methods for additional
details). The results reported here are for the final simulations that led to
the functional physically-realized robot and gaits.

Table 5.1: Simulation results, reported as the mean and maximum velocity attained for each test condition. The simulator is deterministic, so no mean is reported for the hand-designed gaits (since they will always yield identical locomotion speed). Shape-change allowed the robot to switch between dissimilar locomotion gaits, outperforming the benchmark policies. Combined maximum was determined by averaging the maximum speed attainable in both environments. All values have units of body-lengths per second (BL/s). Free parameters | Flat ground | Hill |
---|---|---|---
Mean | Max | Mean | Max | Combined max
Orientation, Shape, Control | 0.0791 | 0.1888 | 0.0023 | 0.0158 | 0.1023
Shape, Control | 0.0945 | 0.2000 | -0.0032 | -0.0004 | 0.0998
Control | 0.0921 | 0.1576 | -0.0087 | -0.0023 | 0.0777
Hand-designed rolling | N/A | 0.1753 | N/A | -0.5743 | -0.1995
Hand-designed inching | N/A | 0.0603 | N/A | 0.0246 | 0.0424

In the first pair of experiments, we sought to discover whether optimization
could find any viable controllers within a constrained optimization space,
which was known to contain the viable hand-designed controllers. Solving this
initial challenge served to test the pipeline prior to attempting to search in
the full search space, which has the potential to have more local minima. The
shape and orientation were fixed (flat and oriented length-wise,
$\theta=90{{}^{\circ}}$, for the inclined surface, cylindrical and oriented
width-wise, $\theta=0{{}^{\circ}}$ for the flat surface). In the second pair
of experiments, the algorithm was allowed to simultaneously search for an
optimal shape and controller pair. Finally, in the third pair of experiments,
all three parameter sets were open to optimization in both environments,
allowing optimization the maximum freedom to produce novel shapes,
orientations, and controllers for locomoting in the two different
environments. For each experiment, we ran 60 independent “hill-climbers”
(instantiations of the “hill-climbing” search algorithm [152], not to be
confused with a robot that climbs a hill) for 200 generations, thus resulting
in identical resource allocation for each experiment (Fig. 5.3b-c). In
addition, we ran a control experiment in which we fixed the shape of the robot
to be fully inflated and oriented width-wise ($\theta=0{{}^{\circ}}$) for the
inclined surface, to determine whether shape-change was necessary.

Figure 5.3:  Automated search attempting to improve gaits in both
environments. a, For each simulation, the algorithm could adjust the
orientation, shape, and/or controller of the robot. Orientation ($\theta$) was
measured by the angle between the robot’s leading edge and a constant-
elevation line on the surface. Shape was parameterized as the inner bladder’s
pressure, resulting in a family of shapes between the cylinder and flat shape
shown. Control of each actuator was parameterized as the number of timesteps
until its first actuation ($\phi$) and the number of timesteps between
actuations ($f$). Here, we show an example controller for the eight main
bladders, with green shaded squares illustrating inflation and white squares
showing deflation. b, Results on a flat surface and (c) on an inclined
surface. Shaded regions represent one standard deviation about the mean (solid
line), while dashed lines represent maximum fitness. The legend indicates
which parameters were to open to optimization, the others being held constant.

When shape and orientation were set as fixed parameters, optimization found a
rolling gait for the inflated shape on flat ground (max fitness 0.1576 BL/s;
Table 1), but it was unable to discover a successful gait for the hill when in
the deflated shape (max fitness -0.0023 BL/s). We were, however, able to
manually program an inchworm gait (0.0246 BL/s) that enabled the deflated
robot to climb the incline (Table 5.1). For reference, other robots that
exclusively utilize inchworm gaits have widely varied speeds, ranging from
0.013 BL/s (for a 226 mm-long robot) [154]to 0.16 BL/s (for a centimeter-scale
robot) [155].

For the second pair of experiments (orientation fixed), the best robots
produced inflated shapes that rolled over flat ground (max fitness 0.2000
BL/s), and similarly failed to produce inchworm motion on the inclined ground
(max fitness -0.0004 BL/s).

In the last pair of experiments (all parameters open), the algorithm again
discovered that cylindrical rolling robots were the most effective over a flat
surface. However, over the inclined surface, the optimization algorithm found
designs with a inflated shape capable of shuffling up the hill when oriented
at an angle (max fitness 0.0158 BL/s). Using this strategy, the robot achieved
combined locomotion of 0.1023 BL/s, outperforming the hand-designed strategy
of using crawling on inclines and rolling on flat ground (combined max of
0.0998 BL/s). The robot was able to inflate two of its external bladders to
balance itself like a table, increasing stability between the robot and the
ground, while the non-standard orientation reduced the amount of gravitational
force opposing the direction of motion, thereby requiring less propulsive
force and reducing the likelihood of the robot rolling back down the hill.
However, this behavior is physically unrealistic. Thus, the best strategy for
moving up the incline remains the hand-designed flattened shape which
traverses the hill using an inchworm-like gait. For the successful
evolutionary trials, the policies found were less finely tuned than those that
were hand-designed. Thus, even in the highly optimized rolling behaviors,
there were occasional counterproductive or superfluous actuations (see
Supplementary Movie S1). Such unhelpful motions could likely be reduced via
further optimization and by adding a fitness penalty for the number of
actuators used per time step.

A similar trend is shown in Fig. 5.3b-c, where the best combined robot for
each environment was discovered by the pair of experiments in which the hill
climbing algorithm had the most control over the optimization of the robot,
despite the larger number of trainable parameters, and thus an increased
likelihood of getting stuck in a local minimum. Additionally, the population
of simulated robots continued to exhibit similar (and often superior) mean
performance compared to the control-only experiments (Fig. 5.3). These
observations suggest that the robots avoided local minima, and that more
parameters should be mutable during automated design of shape-changing robots.