# Forecaster: Towards Temporally Abstract
Tree-Search Planning from Pixels

Thomas Jiralerspong
McGill University, Mila
Montreal, Canada
<EMAIL_ADDRESS>
&Flemming Kondrup∗
McGill University, Mila
Montreal, Canada
<EMAIL_ADDRESS>
&Doina Precup
McGill University, Mila
Montreal, Canada
<EMAIL_ADDRESS>
&Khimya Khetarpal
Mila
<EMAIL_ADDRESS>
Equal contribution, order was attributed alphabetically.

###### Abstract

The ability to plan at many different levels of abstraction enables agents to
envision the long-term repercussions of their decisions and thus enables
sample-efficient learning. This becomes particularly beneficial in complex
environments from high-dimensional state space such as pixels, where the goal
is distant and the reward sparse. We introduce Forecaster, a deep hierarchical
reinforcement learning approach which plans over high-level goals leveraging a
temporally abstract world model. Forecaster learns an abstract model of its
environment by modelling the transitions dynamics at an abstract level and
training a world model on such transition. It then uses this world model to
choose optimal high-level goals through a tree-search planning procedure. It
additionally trains a low-level policy that learns to reach those goals. Our
method not only captures building world models with longer horizons, but also,
planning with such models in downstream tasks. We empirically demonstrate
Forecaster’s potential in both single-task learning and generalization to new
tasks in the AntMaze domain.

##  1 Introduction

Many day-to-day applications such as navigating in a maze like environment,
picking and placing objects in a kitchen, etc. require decision making at many
different levels – from high level steps to fine grained motor control.
Despite significant advancements in deep reinforcement learning, artificial
agents lack the general ability to plan and learn at multiple time scales.
Such an ability can be very useful for speeding up learning, ensuring
robustness and building prior knowledge into AI systems.

When confronted with decision-making, agents with a limited planning horizon
may tend to prioritize immediate rewards over sustainable success, potentially
leading to detrimental outcomes. Hence, one could contend that a fundamental
aspect of intelligence lies in the capacity to imagine the enduring
ramifications of one’s actions and employ this insight to assess whether a
specific behavior is conducive to meeting the desired long-term outcomes. An
agent capable of acquiring a model of its surroundings can, therefore,
strategize across multiple potential courses of action and ascertain which
choices align with its overarching objectives. Analogous to one-step model of
an action, this work focuses on developing a temporally abstract world model –
a model of temporally extended actions (referred to as options). Planning with
such a model has the potential to facilitate generalization from a limited
number of examples and acquire skills that can be applied to solve new unseen
tasks.

Hierarchical Reinforcement Learning (HRL) offers a paradigm for autonomously
decompose a sequential decision making problem into a hierarchy of subtasks.
Feudal learning separates decision making into two processes, where a manager
selects high-level options and a worker learns to execute those tasks [2, 11].
This proficiency to make high-level decisions proves immensely advantageous in
intricate control problems that span a vast number of time steps, primarily by
effectively managing long-term credit assignment. Nevertheless, two pivotal
challenges persist: 1) the efficient implementation of this capability in
pixel-based environments and 2) the strategic selection of these high-level
objectives by the manager.

Our work proposes Forecaster, a deep model-based reinforcement learning
approach which optimizes long-term performance in complex visual environments.
Forecaster learns a temporally abstract model to predict the consequences of
choosing a given high-level goal and therefore evaluate long-term success.
Forecaster then uses this model to plan over options and pick high-level goals
than are expected to be durably beneficial. Specifically, we develop a tree-
search based approach to plan over these high-level goals and determine
favorable behaviors. We demonstrate that Forecaster not only shows promise in
single task learning, but also, the ability to generalize to similar but new
unseen environments. In summary, our contributions are as follows:

  * •

We propose Forecaster, a practical implementation of a tree-search based deep
model-based hierarchical reinforcement learning agent that can learn an
abstract extended world model from pixels, in conjunction with planning over
high level goals.

  * •

We evaluate our method on AntMaze domain providing evidence for the benefits
of Forecaster in both single task learning and generalization to new tasks.

##  2 Preliminaries

In reinforcement learning (RL), the goal of the agent is to maximize rewards
received within the environment. RL problems are often characterized by a
Markov Decision Process (MDP), which is defined by a tuple
$(\mathcal{S},\mathcal{A},P,r,\gamma)$, where $\mathcal{S}$ is the state
space, $\mathcal{A}$ the action space, $P$ the transition function defining
the probability of arriving at a given state $s_{t+1}$ after taking action
$a_{t}$ from state $s_{t}$, $r$ the reward function defining the expected
reward received after taking action $a_{t}$ from state $s_{t}$ and
$\gamma\in(0,1)$ the discount factor. At a given step $t$ of an episode, the
agent takes an action $a_{t}\in\mathcal{A}$ by following its policy $\pi$ and
transitions from $s_{t}\in\mathcal{S}$ to $s_{t+1}\in\mathcal{S}$. Over this
transition, the agent gains a reward $r_{t}$. The goal of the agent is to
learn a policy that maximizes the cumulative discounted return
$\sum_{t=0}^{T}\gamma^{t}r_{t}$ received over an episode of $T$ timesteps.

To account for actions of variable length, the concept of _options_ was
introduced [10]. An option is commonly defined by the tuple
($\mathcal{I},\pi,\beta$), where $\mathcal{I}$ is the set of states where the
option can be initiated, $\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ is
the policy that the agent follows within the option and
$\beta:\mathcal{S}^{+}\rightarrow[0,1]$ is the termination policy which
defines where the option will end. At any given state $s\in\mathcal{S}$, we
can define $\mathcal{O}_{s}$ the set of available options. Consequently,
$\mathcal{O}=\bigcup_{s\in\mathcal{S}}\mathcal{O}_{s}$ denotes the set of all
options. We can then define a policy over options
$\mu:\mathcal{S}\times\mathcal{O}\rightarrow[0,1]$ which at time $t$ will
choose an option $o\in\mathcal{O}_{s_{t}}$ according to the probability
distribution $\mu\left(s_{t},\cdot\right)$.

##  3 Related Work

A limited number of studies have effectively showcased the acquisition of
hierarchical behaviors directly from pixel data, without relying on domain-
specific knowledge. Some of these works include HSD-3 [3], HAC [8], FuN [11]
and Director [4]. The latter harnesses explicit representation learning and
hierarchical exploration, allowing it to achieve high performance across a
wide array of tasks albeit in a single task setting. While Director
demonstrates impressive performance within individual environments, it still
relies on substantial data to acquire proficiency when faced with a new task.
In contrast, we leverage a temporally-abstract world model to plan at a high
level, and show that this approach allows for better generalization to
downstream tasks.

Another line of work in this space combines a world model with tree search
(e.g. Muzero [9]), but primarily considers a world model for primitive actions
so it can only perform tree search over these primitive actions. In contrast,
our work leverages option models ([10, 6]) and learns an extended world model,
which allows us to plan over extended actions, with every extended action
bringing us $k$ timesteps into the future. This allows us to plan much farther
into the future.

##  4 Forecaster: An Approach for Temporally Abstract Tree-Search Planning

Figure 1: General workflow of Forecaster - The manager picks high-level goals
by planning using the extended world model and the worker learns to achieve
them. The extended world model is trained jointly with the manager, worker,
and primitive world model.

Forecaster is a deep model-based hierarchical reinforcement learning algorithm
that learns to plan ahead and maximize long-term success in complex pixel
environments. As shown in Figure 1, Forecaster uses a manager-worker dynamic,
where the former picks high-level goals and the latter learns to achieve them.
In addition, Forecaster learns temporally abstract models of its environment.
This enables it to plan across diverse potential trajectories and select the
one that is anticipated to yield optimal performance in tasks with long-term
horizons. All of Forecaster’s elements are optimized during learning through
one gradient steps every fixed number of environment steps.

###  4.1 Manager-Worker Dynamic

We adopt the general framework of Director [4] for manager-worker dynamics in
pixel environments. We here provide a brief review of this framework, and of
the adaptations made. Director consists of:

  * •

The primitive world model PlaNet [5] which aims to learn a model of the
environment. It is made of four neural networks: the representation model
$\operatorname{repr}_{\theta}\left(s_{t}\mid s_{t-1},a_{t-1},x_{t}\right)$,
the decoder $\operatorname{rec}_{\theta}\left(s_{t}\right)\approx x_{t}$, the
dynamics $dyn_{\theta}\left(s_{t}\mid s_{t-1},a_{t-1}\right)$ and the reward
predictor $\operatorname{rew}_{\theta}\left(s_{t+1}\right)\approx r_{t}$.

  * •

The goal autoencoder which permits to translate between a state $s_{t}$ and
its abstract representation $z$. It is made of the goal encoder
$\operatorname{enc}_{\phi}\left(z\mid s_{t}\right)$ and the goal decoder
$\operatorname{dec}_{\phi}(z)\approx s_{t}$.

  * •

The manager $\operatorname{mgr}_{\psi}\left(z\mid s_{t}\right)$ and worker
$\mathrm{wkr}_{\xi}\left(a_{t}\mid s_{t},g\right)$ which respectively pick a
new goal arbitrarily every K = 8 steps and learn to execute that goal.

In Forecaster, we build on top of this framework to improve high-level
decision making. Our agent learns a temporally abstract world model, which our
manager employs for the purposes of planning.

###  4.2 Temporally Abstract World Models

To estimate the consequences of picking a high level action, we introduce
temporally abstract models. These are built upon the Recurrent State Space
Model (RSSM) PlaNet framework [5] with the key difference that they permit
predictions over high-level temporally abstract goals rather than primitive
actions. Specifically, our model can be represented as
$f(s_{t},g)->(s_{t+k},r)$, where $s_{t}$ is the current state, $s_{t+k}$ is
the state the agent is predicted to arrive at after aiming to reach the goal
$g$, and $r$ is the associated reward. As such, these models help our manager
estimate where it will arrive and how much reward will be received if it picks
a given goal.

###  4.3 Tree-Search Planning over Options

Figure 2: Instead of directly picking an option based on expected return, our
manager builds a tree of possible path by first picking X goals, leverages the
world model to predict the consequences of following that goal for K steps,
and repeats this process $m$ times.

Forecaster leverages its temporally abstract world model to plan over high-
level goals for long-term success in pixel environments. Whenever our manager
is to pick a new high-level goal, it first builds a tree and imagines the
long-term effects of each of the possible paths it could choose from (see
Figure 2). It then computes the reward associated with each path and picks the
first option expected to lead to high long-term reward.

###  4.4 Algorithmic summary

We present a detailed description of our approach in Algorithm 1. We note that
the key ideas in Forecaster that are distinct from Director are highlighted in
blue.

Initialize replay buffers and neural networks.

while  __not converged_ _ do

Update model state $s_{t}\sim$ repr($s_{t}|s_{t-1},a_{t-1},x_{t}$).

if  _t mod 8 = 0_ then

_ _ _# Tree-Search Planning_ __

 _Sample X goals_ $z_{1},...z_{X}\sim$ mgr($z|s_{t})$, decode into goals
$g_{i}=dec(z_{i})$ and have the manager imagine itself at each $g_{i}$. Repeat
from each child node $m$ times to build the full tree. Compute the expected
reward of each path and pick the first goal in the path leading to the highest
reward as $g$.

end if

Sample action $a_{t}\sim wkr(a_{t}|s_{t},g)$

Send action to environment and observe $r_{t}$ and $x_{t+1}$

Add transition $(x_{t},a_{t},r_{t},x_{t+1})$ to replay buffer

if  _t+1 mod 8 = 0_ then

_ _ _# Every 8 steps, add the trajectory to the extended buffer_ __

 _Add trajectory_ $(x_{t-7},x_{t+1},g,r)$ to extended replay buffer, where $g$
is the current goal and $r$ the cumulative reward from $t-7$ to $t$.

end if

if  _t mod C = 0_ then

_ _ _# Update the abstract world model_ __

 _Draw trajectory batch_ $(x,x^{\prime},g,r)$ from extended buffer where $x$
is the starting state, $x^{\prime}$ is the final state, $g$ the goal, $r$ the
cumulative reward. Update abstract world model on trajectory batch.

end if

if  _t mod 16 = 0_ then

_# Update the primitive world model, the autoencoder and the manager and
worker networks_

Draw sequence batch $(x,a,r,x^{\prime})$ from replay buffer

Update world model on batch and get states s

Update goal autoencoder on s

Imagine trajectory $\\{(\hat{s},\hat{a},\hat{g},\hat{z})\\}$ under model and
policies starting at s, compute expected rewards and update manager and worker

end if

end while

Algorithm 1 Forecaster

##  5 Experiments

We implemented Forecaster on top of Director [4], reusing its default
hyperparameters. Director uses the same manager-worker approach but is limited
as it does not involve the notion of options, of the abstract world model, of
the tree-search planning or the goal-conditioning.

Environments. We evaluate Forecaster on one benchmarking environment[4]:
Egocentric Ant Maze: The agent controls a quadruped robot which navigates
through a 3D maze and is controlled through joint torques. The agent has a
first-person camera and proprioceptive inputs, and only receives a reward if
it reaches the terminal reward.

Figure 3: Performance of Forecaster and Director on Antmaze environments.
Forecaster reaches competitive performance in Ant Maze S and higher sample
efficiency in Ant Maze M.

Single Task. We first train both Forecaster and Director from scratch on
Egocentric Ant Maze. In Figure 3 we see that Forecaster learns as well as
Director on Small Ant Maze and learns more sample efficiently when compared to
Director on Medium Ant Maze. We hypothesize that the increase in sample
efficiency is only visible in Medium Ant Maze because Small Ant Maze is too
simple of an environment for Forecaster’s planning to begin to be useful.

Generalization. Next, we train Forecaster on Egocentric Small Ant Maze and
compare loading the pre-trained worker, manager and extended world model vs.
only loading the pre-trained worker and manager vs. training from scratch on
Ant Maze M. In Figure 2 we report that loading the extended world model
trained on Small Ant Maze results in improved sample efficiency when compared
to only loading the worker and manager trained on Small Ant Maze. Forecaster
is able to generalize its extended world model from one environment to
another, showcasing its ability to plan across different environments after
only being trained in one.

Figure 4: Generalization ability of Forecaster and baselines. The agents were
trained on Ant Maze S and then evaluated on Ant Maze M. Leveraging the
extended world model for tree-search planning results in improved sample
efficiency when compared to only loading the worker and manager.

##  6 Discussion

In this work, we proposed a method to demonstrate the benefits of building an
abstract world model with tree-search planning from high-dimensional input
space such as pixel-based environments. It’s noteworthy that the gains we’ve
observed are a result of a limited hyper-parameter search, which is quite
promising. Future research focusing on exploring variable tree sizes is likely
to yield even better performance enhancements. While investigating larger tree
structures may incur increased computational costs, affordances [6] may
provide solutions to the large branching factor [7].

Furthermore, in the context of high-level decision-making, an open research
questions is to relax the assumption of changing goals at fixed intervals.
This would enable the agent to adapt dynamically to the environment,
potentially improving performance on tasks that demand precise timing and
stitching. Leveraging the proposed ideas in conjunction with large models such
as Transformer XL [1] is an interesting direction for future work with
potential to impact even larger scale.

## References

  * [1] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context, 2019.
  * [2] Peter Dayan and Geoffrey E. Hinton.  Feudal reinforcement learning.  In NIPS, 1992.
  * [3] Jonas Gehring, Gabriel Synnaeve, Andreas Krause, and Nicolas Usunier.  Hierarchical skills for efficient exploration, 2021.
  * [4] Danijar Hafner, Kuang-Huei Lee, Ian Fischer, and Pieter Abbeel.  Deep hierarchical planning from pixels, 2022.
  * [5] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson.  Learning latent dynamics for planning from pixels, 2018.
  * [6] Khimya Khetarpal, Zafarali Ahmed, Gheorghe Comanici, and Doina Precup.  Temporally abstract partial models, 2021.
  * [7] Khimya Khetarpal and Doina Precup.  Learning options with interest functions.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9955–9956, 2019.
  * [8] Andrew Levy, George Konidaris, Robert Platt, and Kate Saenko.  Learning multi-level hierarchies with hindsight, 2019.
  * [9] Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver.  Mastering atari, go, chess and shogi by planning with a learned model.  Nature, 588(7839):604–609, dec 2020.
  * [10] Richard S. Sutton, Doina Precup, and Satinder Singh.  Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning.  Artificial Intelligence, 112(1):181–211, 1999.
  * [11] Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu.  Feudal networks for hierarchical reinforcement learning, 2017.


# Planar Modeling and Sim-to-Real of a Tethered Multimaterial Soft Swimmer
Driven by Peano-HASELs

Stephan-Daniel Gravert∗, Mike Y. Michelis∗, Simon Rogler, Dario Tscholl,
Thomas Buchner,
and Robert K. Katzschmann *These authors contributed equally to this work.All
authors are with the Soft Robotics Lab, Department of Mechanical and Process
Engineering, ETH Zurich, Tannenstrasse 3, 8092 Zurich, Switzerland {sgravert,
michelism, srogler, dtscholl, tbuchner<EMAIL_ADDRESS>
###### Abstract

Soft robotics has the potential to revolutionize robotic locomotion, in
particular, soft robotic swimmers offer a minimally invasive and adaptive
solution to explore and preserve our oceans. Unfortunately, current soft
robotic swimmers are vastly inferior to evolved biological swimmers,
especially in terms of controllability, efficiency, maneuverability, and
longevity. Additionally, the tedious iterative fabrication and empirical
testing required to design soft robots has hindered their optimization. In
this work, we tackle this challenge by providing an efficient and
straightforward pipeline for designing and fabricating soft robotic swimmers
equipped with electrostatic actuation. We streamline the process to allow for
rapid additive manufacturing, and show how a differentiable simulation can be
used to match a simplified model to the real deformation of a robotic swimmer.
We perform several experiments with the fabricated swimmer by varying the
voltage and actuation frequency of the swimmer’s antagonistic muscles. We show
how the voltage and frequency vary the locomotion speed of the swimmer while
moving in liquid oil and observe a clear optimum in forward swimming speed.
The differentiable simulation model we propose has various downstream
applications, such as control and shape optimization of the swimmer;
optimization results can be directly mapped back to the real robot through our
sim-to-real matching.

##  I INTRODUCTION

###  I-A Motivation

Robots truly useful to humans must integrate into our world by safely
performing tasks in close proximity to living beings. Ideally, robots
seamlessly integrate into our daily lives and assist us without any
possibility of causing physical harm. While rigid robots are powerful and fast
[1, 2], they lack dexterity and cannot adapt well to their environment; thus,
they can be dangerous to humans. In contrast, soft robots [3] have a partially
compliant and multifunctional composition and the potential to self-heal,
biodegrade, and interact safely with their environment. Therefore, soft robots
will likely be the only robots to fully integrate and perform smoothly in the
real world [4]. Soft robots are inspired by animals’ ability to deform, adapt,
and survive in complex environments [5, 6].

Indeed, technologies enabling robotic movement based on pneumatic, magnetic,
thermal, and electrostatic actuation mechanisms [6, 7, 8] have been
investigated with the aim of creating soft robots with the ability to move,
adapt, and interact with complex and unpredictable natural environments in the
same way as animals [9, 10, 11, 12]. The replication of animals’ movement
abilities in robots would revolutionize robotic locomotion, and bring us
closer to safe robot companions. However, no computational tool to date can
model and optimize soft robot performance.

Figure 1: (A) Comparison of our electrostatically-actuated swimmer with
simulation, actuated at 5.5kV in both simulation and reality. (B) Comparison
of actuator position in simulation versus reality. In simulation we depict the
vertices, which are colored according to which type of element they belong to:
red is upper muscle, blue is lower muscle, black is spine elements, and grey
is soft material in the swimmer body.

Soft robots have attracted particular attention for their potential
applications within fluidic environments. For instance, soft robots moving
within aquatic environments can serve for naturalistic sampling, geographical
mapping, and exploration of oceans both on earth and in outer space [9, 10,
11, 12]. Oceans are filled with objects and living entities still to be
discovered as only 20% of it been mapped to date [13]. Soft robotic swimmers
offer a minimally invasive, biodegradable and affordable solution for
exploration of this ‘final frontier’ on earth. Biomimetic soft robots could
integrate into an ecosystem to explore, observe, and preserve biodiversity
[14], natural resources [15], and new habitats [16]. Soft robots could
potentially observe vulnerable populations in protected areas where only
minimally-invasive biomimetic [17, 18] and biodegradable [19] entities are
permitted. To therefore develop efficient soft robots for these challenges, we
need effective strategies that can quickly prototype and optimize robots for
their target objective.

###  I-B Approach

We propose an electrostatically-actuated soft robotic swimmer using
Hydraulically amplified self-healing electrostatic (HASEL) actuators,
manufactured using rapid multimaterial 3D-printing technology, and modeled in
a differentiable simulation using simplified muscle models that close the sim-
to-real gap through gradient-based optimization. HASELs are a promising soft
actuator technology due to their muscle-like performance. Combined with soft-
rigid printing for a robot “body”, we can create soft robots in a swift
manner. Since it is often tedious to optimize for control and shape of the
resulting robot, we integrate our model into a differentiable simulation with
simulated muscle models that match the real actuation behavior such that it
can be used in future downstream gradient-based optimization tasks.

###  I-C Contributions

We developed (1) a 2.5D (2D shape extruded with constant height profile) Peano
HASEL soft robotics swimmer using (2) a rapid multimaterial manufacturing
approach for soft robotic systems, and (3) present a Finite Element Method
(FEM) model for HASEL actuators that matches a simplified muscle model in a
differentiable simulation with reality (sim-to-real) through gradient-based
system identification.

##  II Related Works

###  II-A Soft Swimmer Muscles

Several pioneering works have demonstrated the ability of fish-like swimming
[20, 21]. Advances in biomimetic movement, mimicking and interacting with real
fish [22, 23] has been utilized further. Fluidic actuators have been used in
almost all recent fish designs. In particular, hydraulic actuators have been
proposed [24], which benefit from many different design and fabrication
techniques like soft lithography [25], retractable pin casting [26] or
monolithic casting with a lost-wax fabrication. Multiple swimmers using
electrostatic type actuators have also been shown before [27, 28]. Actuators
such as dielectric elastomer actuators (DEAs) offer large actuation strains
with high efficiency [29]. Because DEAs are usually in a stacked configuration
with very thin dielectric and stretchable electrode layers, they are prone to
electrical breakdowns and fatigue [30]. Over the last few years HASELs have
shown potential in many different applications [31]. HASELs have high power
density and are efficient, especially when recuperating high voltage power
supplies are used. They are able to exert high forces [32] and specific torque
outperforming comparable electrical servo motors [33]. Furthermore, HASELs
offer fast actuation speeds making them well-suited for use in soft robotic
systems [34].

###  II-B Soft Robot Simulation

Because soft robotic systems are continuously deforming structures, FEM is
usually the only simulation method that will accurately represent the dynamics
of the robot [35, 36, 37]. Due to the computational complexity of FEM,
simplifying assumptions are often used in practice, such as Piecewise Constant
Curvature models [38], which work for only a limited range of soft robot
morphologies. On the other hand, data-driven approaches have the potential to
work for a wider variety of robots, however, usually require specific
experimental measurements to fit the robot [39, 40, 41].

In the soft robotics community, a popular framework for accurate FEM
simulations is the Simulation Open Framework Architecture (SOFA) [42]. This
framework is able to model highly deformable objects, and has been used for
reduced-order models [43, 44, 36]. In recent years, differentiable simulations
have gained interest in the robotics community. Differentiable simulators
allow for computation of gradients of any variable (state, model, or control)
in a system with respect to a simulation objective, e.g., object velocity,
force, distance to target. Several downstream soft robotic applications are
possible due to this differentiable property, including system identification
[45], trajectory optimization [46], motion control [47], and shape
optimization [48]. Transferring these simulations to real robots often
struggle with the so-called “sim-to-real gap”, which has been for instance
overcome with system identification to correct the simulated system to match
the real world parameters [49, 50, 51]. These methods for sim-to-real were
either only shown on non-actuated systems, or only on fluidic actuation, which
is drastically different in terms of dynamics from electrostatic actuators.

##  III METHODOLOGY

###  III-A Fish design and swim environment

The swimmer’s profile is generated using a parametric polynomial adapted from
[52]. The fish measures $190\text{\,}\mathrm{mm}$ from fin to head, where the
fin is $30\text{\,}\mathrm{mm}$ long and the solid head is
$52\text{\,}\mathrm{mm}$ long. The maximum width of the fish is
$21\text{\,}\mathrm{mm}$. The outer structure of the swimmer is 3D printed
with a multimaterial printer described in detail in Chapter III-B. The HASEL
fish swims in canola oil because the electrostatic actuators are currently not
insulated for ease of manufacturing and simulation; oil has a lower Reynolds
number than water and fluids with less turbulence will be easier to simulate.
The fish is connected to two custom $5\text{\,}\mathrm{W}$ high voltage
($10\text{\,}\mathrm{kV}$) power supplies [53] controlled with an Arduino
through a very thin conductive and electrically insulated yarn (Datastretch
from Muca).

###  III-B Multi-Material Printing

The structure of the fish is 3D printed on a tool-changing CoreXY FDM printer
(adapted Rat Rig V-Core 3) using two different types of extruders. The printer
can switch between two E3D Hemera
extruders111https://e3d-online.com/products/e3d-hemera-direct-kit-1-75mm for
filaments and two V4 pellet extruders222https://mahor.xyz/producto/v4-pellet-
extruder/. The body of the fish is printed using a polypropylene (PP) (Shore
hardness 72D) filament for the semi-rigid parts and styrene-ethylene-butylene-
styrene (SEBS) pellets (Shore hardness 18A) for the soft parts. Additionally,
polylactic acid (PLA) filament is used to print a stabilization wall,
strengthening the SEBS parts and provides rigidity for tall prints. A 3D
printing adhesive from Magigoo is used to ensure adhesion of PP and SEBS on
the printbed. Figure 2 shows the fish structure directly after printing with
the stabilization structure and skirts still attached.

Figure 2: Fish structure directly after printing. The long continuous
polypropylene (PP) wall (spine of the fish) requires brims to reduce warping.
The polylactic acid (PLA) wall mechanically stabilizes the styrene-ethylene-
butylene-styrene (SEBS) soft shell during printing.

###  III-C HASEL Manufacturing

We adapt the method shown in [53] to manufacture the Peano HASEL actuators. We
use a commercially available 3D printer (Prusa MK3S) to heat-seal the outer
shell (Mylar 850, Petroplast GmbH) of the actuator. In contrast to [53] we do
not use a soldering iron for the automated heat-sealing. Instead, we extrude
one layer of filament onto a protective Kapton sheet (Kapton
$25\text{\,}\mathrm{\SIUnitSymbolMicro m}$ from RS Online) to evenly transfer
heat to the Mylar films below. This avoids the need for a neoprene rubber
sheet and lubrication, and makes scratching of the dielectric films less
likely. Additionally, this method does not require any adaptions to the 3D
printer. The actuator design shown on Figure 3 was created in Fusion 360
v2.0.11415 and exported to PrusaSlicer v2.3.1. All lines are
$1.5\text{\,}\mathrm{mm}$ wide and are heat-sealed twice to make the seals
more durable. Electrodes are made from the carbon black based color Electrodag
502 (Gloor Instruments GmbH), which is airbrushed onto the actuator. To define
the electrode area, masks are 3D printed and clamped onto the actuator using
binder clips. The HASELs are filled with oil through a
$1.5\text{\,}\mathrm{mm}$ wide filling port, which is added to the sealing
line (Figure 3). The amount of filling liquid is chosen lower compared to
other works [30] at approximately 75% of the theoretical maximum cylindrical
volume of the pouch because of the low width of the actuator pouches. This
volume of dielectric oil is injected into the sample using a
$1\text{\,}\mathrm{mL}$ syringe, minimising the presence of air bubbles.
Lastly, the filling ports are heat-sealed with a soldering iron.

Figure 3: Finished HASEL actuator overlaid with the seal design. Each chamber
has a separate filling port which is sealed with a soldering iron after
filling. Two mounting holes are placed on either side to secure the actuator
in the fish.

###  III-D Peano HASEL characterization

The vertical displacement of the weights is measured using a Baumer
OM70-11216521 distance measurement sensor. A Trek 20/20C-HS high voltage
amplifier is used to generate the high voltage signal. The low voltage input
signal for the amplifier is generated by a MyDAQ from National Instruments
controlled by custom LabView code. The program creates a bipolar voltage curve
for the amplifier and reads the high voltage level and distance back from the
high voltage amplifier and the distance measurement sensor. The force/strain
performance Figure 8 of the actuators is generated by lifting weights three
times using a bipolar voltage signal (frequency $0.1\text{\,}\mathrm{Hz}$,
duty cycle 50%) and averaging the resulting distances.

###  III-E Simulated Swimmer Modeling

We modeled the swimmer as a filled shape using the 5th-order parametric
polynomial defined for carangiforms [54], the same shape we used for the
printed swimmer. After measuring the dimensions of the real swimmer, we chose
the parametric height and length of the swimmer to be $15\text{\,}\mathrm{mm}$
and $160\text{\,}\mathrm{mm}$ respectively. We modified this shape by adding a
more rigid spine with thickness $1\text{\,}\mathrm{mm}$ that extends into a
tail that is $30\text{\,}\mathrm{mm}$ long. During meshing, we increased the
meshing resolution near the spine within the body, not the tail, so we
achieved a mean spine meshing error (distance of elements that should not
belong to the spine) of $0.07\text{\,}\mathrm{mm}$. This resulted in a mesh
with 1396 vertices shown in Figure 1.

The materials we used for the 3D-printed swimmer are SEBS for the soft white
material, with Young’s Modulus $0.65\text{\,}\mathrm{MPa}$ and density
$940\text{\,}\mathrm{kg}\text{\,}{\mathrm{m}}^{-3}$, and PP for the rigid
black material, with Young’s Modulus $1.33\text{\,}\mathrm{GPa}$ and density
$920\text{\,}\mathrm{kg}\text{\,}{\mathrm{m}}^{-3}$. We decided to lower the
stiffness of both materials since the structure in simulation is a filled body
instead of the thin-walled geometry in the real world. We modelled the
stiffness of the SEBS and PP materials with $65\text{\,}\mathrm{kPa}$ and
$0.13\text{\,}\mathrm{MPa}$, respectively. We approximate the density of both
materials with $900\text{\,}\mathrm{kg}\text{\,}{\mathrm{m}}^{-3}$. We assume
that both are incompressible materials, and hence selected a Poisson’s Ratio
of $0.45$. The reasoning why we model the structure as a filled body, is due
to how we define muscles. We opted for modeling the upper and lower halves of
the swimmer to be contractile muscles using simplified “Muscle Models” (the
contractile deformation can be seen in Figure 1), as defined in [55, 37, 51].
These simplified artificial muscles are an efficient alternative to simulating
the accurate thin-wall geometry and electrostatic actuation of our swimmer,
and we show that using the differentiability of the soft-body simulation we
can accurately match simulation and reality. We match the artificial muscle
models, i.e., the contractile actuation parameters, to data collected from the
real swimmer. We will further describe this process in the next two sections
and utilize the differentiability offered by our choice of differentiable
simulator “DiffPD” [37].

(a)

(b)

(c)

Figure 4: Tracking of marker points on the real fish and computing the bending
angle of the swimmer. (a) shows the relaxed state, and (b) the deformed state
where the top muscle is actuated at $5.5\text{\,}\mathrm{kV}$, and (c) shows
two periods of HASEL actuation at $0.5\text{\,}\mathrm{Hz}$. Figure 5: Sloped
box actuation function for the simplified muscles. Top (blue) and bottom
(orange) muscles are contracted sequentially. Contraction occurs with an
actuation below 1.

###  III-F Data Collection

To close the sim-to-real gap, we collected data from the real HASEL swimmer by
actuating it using (a) varying voltages, to match the static behavior, and (b)
different frequencies, to match the dynamic behavior. We collected data for
voltages in $[$3.5$,$4.5$,$5.5$]$$\text{\,}\mathrm{kV}$ and frequencies in
$[$0.5$,$2.0$,$3.5$]$$\text{\,}\mathrm{Hz}$, resulting in a total of 9
datapoints for system identification. We captured the movement of the real
swimmer using a camera recording from a top-down perspective at
$$1920$\times$1080$$ resolution with a framerate of $120\text{\,}\mathrm{Hz}$.
We placed three blue circles on the swimmer: one on the front of its head, one
in the middle of the swimmer, and lastly one on the tip of its tail.

We used the CSRT algorithm [56] to track the three marker points on the
swimmer, and cut the video to the first frame where the upper muscle starts
moving and the last frame after 2 periods have passed since the movement
begins. We manually selected the bounding boxes for these keypoints in the
starting frame, and tracked these three points over the 2 periods. We then
computed the sine of the angle $\theta$ spanned by the vector pointing from
head to middle $\vec{v}_{HM}$ and the vector pointing from middle to tail
$\vec{v}_{MT}$ with
$\sin{\theta}=(\vec{v}_{HM}\times\vec{v}_{MT})/(\|\vec{v}_{HM}\|\,\|\vec{v}_{MT}\|)$.
This bending angle will be invariant to the global motion of the swimmer,
since the entire body may move due to friction with the underlying surface.
Additionally, the bending angle removes the need for us to map between pixel
coordinates and physical units. The sine of the angle will be positive when
the top muscle is contracting, while it will be negative for the bottom
muscle. We notice that in the real world the swimmer is slightly bent in its
relaxed state, hence while processing the data, we subtract the average of the
all bending angles to assure an as symmetric as possible behavior, such that
the simulation can fit it more accurately, since it is by design symmetric.
Lastly we store $\sin{\theta}$ in CSV files that we can use for system
identification. Figure 4 shows how the captured data.

###  III-G Simulation System Identification

The data collected from the previous section is now compared to the simulation
results, and the simulated swimmer is optimized to match reality as best as
possible. We can define the exact same three keypoints on the simulated
structure, and compute their angle using the same vector directions as defined
previously for the real swimmer. The parameters we optimize for the simulated
swimmer are (a) muscle actuation amplitude and (b) slope of the box actuation
signal seen in Figure 5. Since our simulation is differentiable, we can
optimize w.r.t. these parameters given a loss objective. Our objective will be
a mean squared error of simulated angle to real angle at every frame $i$ of
the simulation
$\frac{1}{N}\sum_{i=1}^{N}\|\theta_{sim}(i)-\theta_{real}(i)\|^{2}$. We use
Adam optimizer with a learning rate of $0.05$ for the amplitude and $0.005$
for slope.

We optimized with the following assumptions: (a) given the same voltage of the
real data, the muscle actuation amplitude should be the same for all actuation
frequencies, and (b) the actuation slope will be the same for all samples and
characterizes the HASEL swimmer (HASEL actuation + material behavior) itself.
We verified the success of our system identification by testing on unseen
data, namely on $2.5\text{\,}\mathrm{Hz}$ at $4.25\text{\,}\mathrm{kV}$,
$4.8\text{\,}\mathrm{kV}$, and $5.2\text{\,}\mathrm{kV}$.

Figure 6: Speed of the swimmer in an oil tank actuated with different
frequencies and voltages. At $7\text{\,}\mathrm{Hz}$ there is an optimum for
locomotion speed across all voltages.

##  IV RESULTS

We submerged our swimmer into a bath of canola oil and measured the locomotion
speed at different frequencies and voltages. Figure 6 shows a local optimum
for fish speed at $7\text{\,}\mathrm{Hz}$ which applies to all voltages. Above
$5.5\text{\,}\mathrm{kV}$ breakdown was likely. Figure 7 shows a typical
locomotion behavior of the fish with a slight turning angle because of
imperfections in actuator production and fish manufacturing. This turning
angle can be compensated for by applying different voltages to each HASEL
muscle.

Figure 7: A time series of still images of the 3D printed fish swimming in oil
at $0\text{\,}\mathrm{s}$, $10\text{\,}\mathrm{s}$, and
$20\text{\,}\mathrm{s}$. The swimmer traversed $10\text{\,}\mathrm{cm}$ within
the $20\text{\,}\mathrm{s}$ experiment. The current position of the tip of the
tail fin is marked with a blue label. In the bottom image, the tail fin’s
maximal positions are visualized with blue lines.

We investigated the performance of the Peano HASELs used for the fish. The
resulting force/strain values are shown in Figure 8 Our actuators are strong
enough to actuate the fish but significantly underperform the analytical model
given by [32] for Peano HASELs with a width of $18\text{\,}\mathrm{mm}$. Our
guess is that for small widths the model predictions are not accurate because
edge effects dominate the zipping behavior.

Figure 8: Force/strain performance of HASEL actuators used for actuation of
the swimmer.

###  IV-A Sim-to-Real Performance

We optimized the muscle actuation amplitude and slope as described in Section
III-G, and found a resulting mean absolute angle error $\varepsilon_{\theta}=$
$1.553\text{\times}{10}^{-02}$ for the nine data points acquired during
training. Note that the mean absolute angle error is defined by
$\varepsilon_{\theta}=\frac{1}{N}\sum_{i=1}^{N}|\theta_{sim}(i)-\theta_{real}(i)|$.
We can show generalization of our muscle model to unseen scenarios by
validating our optimized result for amplitude and slope on three voltages at a
fixed frequency, all unseen to the optimization problem. We interpolate the
muscle actuation from our previous data, and plug in the frequency since we
assume this to be consistent between simulation and reality. The result can be
seen in Figure 9, and we computed a mean absolute angle error of
$\varepsilon_{\theta}=$ $3.103\text{\times}{10}^{-02}$. Both in amplitude and
frequency this shows a reasonable generalizability that we can in future work
use in downstream tasks that require the differentiability this model offers,
such as control or shape optimization [48, 57].

(a) Voltage at $4.25\text{\,}\mathrm{kV}$

(b) Voltage at $5.2\text{\,}\mathrm{kV}$

Figure 9: The simulation results (solid blue) and real bending angle (dashed
orange). We show two periods of HASEL actuation at $2.5\text{\,}\mathrm{Hz}$
for every unseen test voltage.

##  V DISCUSSION AND FUTURE WORK

With this work, we have shown the feasibility of using HASEL actuators as
muscles for underwater robotic systems. The rapid multimaterial 3D printing
approach simplifies prototyping different soft robot designs, and we have
shown that simplified muscle models can be used to model HASELs for FEM
simulations. We observe that the HASEL force-strain curve lends itself well to
the placement and geometry in the fish. Bending the spine out of its initial
straight position requires the strongest force. This equals the maximum force
available at low strains. We have tested our soft robotic swimmer for various
voltages and frequencies, and believe that as a next step control optimization
would be greatly beneficial for showing that optimized, fast, efficient
swimmer designs are possible.

Our work exhibits several limitations, for example, the number of datapoints
was too limited to assure a general model for the sim-to-real mapping. We
could consider applying more sophisticated methods such as multilayer
perceptrons from machine learning for learning this relationship when enough
data is given. We also noted that the dynamic response, especially the
damping, of DiffPD does not entirely correspond to the deformation behavior of
the real swimmer. This can be compensated for explicitly [51], or we could
create more complex models for our real structure.

For future work we aim to create an untethered version of the fish,
incorporating new dielectric shell materials to lower the required voltages
and therefore the size of internal power supplies. As a next step we also want
to insulate the fish against water to make it useful for real-world
applications. We also wish to optimize the fish movement and design submerged
in fluid. Existing works often assume simplified hydrodynamics to allow fluid-
structure interaction for underwater swimmers [57, 58]. As a next step, we
will combine the differentiable soft body simulation used in this work with a
learned hydrodynamics model [59] to assure differentiability through the
entire fluid-solid coupling. As far as we are aware, no one has applied soft
robot optimization on real robots yet using complete, realistic hydrodynamics
to describe a differentiable, multiphysical simulation.

## ACKNOWLEDGMENT

We would like to thank Barnabas Gavin Cangan for his insightful comments
during marker tracking, and Oliver Fischer for his help polishing visuals.

## References

  * [1] S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, and R. Tedrake, “Optimization-based locomotion planning, estimation, and control design for the atlas humanoid robot,” _Auton. Robots_ , vol. 40, no. 3, pp. 429–455, 2016, publisher: Springer.
  * [2] T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” _Science Robotics_ , vol. 7, no. 62, p. eabk2822, 2022.
  * [3] D. Rus and M. T. Tolley, “Design, fabrication and control of soft robots,” _Nature_ , vol. 521, no. 7553, pp. 467–475, May 2015, number: 7553 Publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/nature14543
  * [4] E. W. Hawkes, C. Majidi, and M. T. Tolley, “Hard questions for soft robotics,” _Science Robotics_ , vol. 6, no. 53, Apr. 2021, publisher: Science Robotics Section: Viewpoint. [Online]. Available: https://robotics.sciencemag.org/content/6/53/eabg6049
  * [5] S. A. Morin, R. F. Shepherd, S. W. Kwok, A. A. Stokes, A. Nemiroski, and G. M. Whitesides, “Camouflage and display for soft machines,” _Science_ , vol. 337, no. 6096, pp. 828–832, 2012, publisher: American Association for the Advancement of Science.
  * [6] A. Miriyev, K. Stack, and H. Lipson, “Soft material for soft actuators,” _Nature Communications_ , vol. 8, no. 1, p. 596, Sep. 2017, number: 1 Publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/s41467-017-00685-3
  * [7] E. Sachyani Keneth, R. Lieberman, M. Rednor, G. Scalet, F. Auricchio, and S. Magdassi, “Multi-Material 3D Printed Shape Memory Polymer with Tunable Melting and Glass Transition Temperature Activated by Heat or Light,” _Polymers_ , vol. 12, no. 3, p. 710, Mar. 2020, number: 3 Publisher: Multidisciplinary Digital Publishing Institute. [Online]. Available: https://www.mdpi.com/2073-4360/12/3/710
  * [8] M. Ilami, H. Bagheri, R. Ahmed, E. O. Skowronek, and H. Marvi, “Materials, Actuators, and Sensors for Soft Bioinspired Robots,” _Advanced Materials_ , p. 2003139, 2020, publisher: Wiley Online Library.
  * [9] F. Berlinger, M. Saadat, H. Haj-Hariri, G. V. Lauder, and R. Nagpal, “Fish-like three-dimensional swimming with an autonomous, multi-fin, and biomimetic robot,” _Bioinspiration & Biomimetics_, vol. 16, no. 2, p. 026018, Feb. 2021, publisher: IOP Publishing. [Online]. Available: https://doi.org/10.1088/1748-3190/abd013
  * [10] D. Costa, G. Palmieri, M.-C. Palpacelli, D. Scaradozzi, and M. Callegari, “Design of a Carangiform Swimming Robot through a Multiphysics Simulation Environment,” _Biomimetics_ , vol. 5, no. 4, p. 46, Dec. 2020, number: 4 Publisher: Multidisciplinary Digital Publishing Institute. [Online]. Available: https://www.mdpi.com/2313-7673/5/4/46
  * [11] G. Li, X. Chen, F. Zhou, Y. Liang, Y. Xiao, X. Cao, Z. Zhang, M. Zhang, B. Wu, S. Yin, Y. Xu, H. Fan, Z. Chen, W. Song, W. Yang, B. Pan, J. Hou, W. Zou, S. He, X. Yang, G. Mao, Z. Jia, H. Zhou, T. Li, S. Qu, Z. Xu, Z. Huang, Y. Luo, T. Xie, J. Gu, S. Zhu, and W. Yang, “Self-powered soft robot in the Mariana Trench,” _Nature_ , vol. 591, no. 7848, pp. 66–71, Mar. 2021, number: 7848 Publisher: Nature Publishing Group. [Online]. Available: https://www.nature.com/articles/s41586-020-03153-z
  * [12] D. Scaradozzi, G. Palmieri, D. Costa, and A. Pinelli, “BCF swimming locomotion for autonomous underwater robots: a review and a novel solution to improve control and efficiency,” _Ocean Engineering_ , vol. 130, pp. 437–453, Jan. 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0029801816305613
  * [13] L. Trethewey, “Earth’s final frontier: the global race to map the entire ocean floor,” Jun. 2020, section: Environment. [Online]. Available: http://www.theguardian.com/environment/2020/jun/30/earths-final-frontier-the-global-race-to-map-the-entire-ocean-floor
  * [14] J. Krause, A. F. T. Winfield, and J.-L. Deneubourg, “Interactive robots in experimental biology,” _Trends Ecol. Evol._ , vol. 26, no. 7, pp. 369–375, Jul. 2011.
  * [15] U. C. Marx, J. Roles, and B. Hankamer, “Sargassum blooms in the Atlantic Ocean – From a burden to an asset,” _Algal Research_ , vol. 54, p. 102188, Apr. 2021\. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2211926421000072
  * [16] J. E. A. Gomez Jr., “The size of cities: A synthesis of multi-disciplinary perspectives on the global megalopolis,” _Progress in Planning_ , vol. 116, pp. 1–29, Aug. 2017. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0305900616300113
  * [17] S. Marras and M. Porfiri, “Fish and robots swimming together: attraction towards the robot demands biomimetic locomotion,” _Journal of The Royal Society …_ , vol. 9, no. 73, pp. 1856–1868, Aug. 2012.
  * [18] S. Butail, N. Abaid, S. Macrì, and M. Porfiri, “Fish–Robot Interactions: Robot Fish in Animal Behavioral Studies,” in _Robot Fish_ , ser. Springer Tracts in Mechanical Engineering. Springer, Berlin, Heidelberg, 2015, pp. 359–377.
  * [19] S. Aracri, F. Giorgio-Serchi, G. Suaria, M. E. Sayed, M. P. Nemitz, S. Mahon, and A. A. Stokes, “Soft Robots for Ocean Exploration and Offshore Operations: A Perspective,” _Soft Robotics_ , Jan. 2021, publisher: Mary Ann Liebert, Inc., publishers. [Online]. Available: https://www.liebertpub.com/doi/full/10.1089/soro.2020.0011
  * [20] J. Yu, M. Tan, S. Wang, and E. Chen, “Development of a biomimetic robotic fish and its control algorithm,” _IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics)_ , vol. 34, no. 4, pp. 1798–1810, 2004.
  * [21] J. M. Anderson and N. K. Chhabra, “Maneuvering and stability performance of a robotic tuna1,” in _Integrative and comparative biology_ , 2002.
  * [22] A. D. Marchese, C. D. Onal, and D. Rus, “Autonomous soft robotic fish capable of escape maneuvers using fluidic elastomer actuators,” _Soft Robotics_ , vol. 1, no. 1, pp. 75–87, 2014, pMID: 27625912. [Online]. Available: https://doi.org/10.1089/soro.2013.0009
  * [23] S. Marras and M. Porfiri, “Fish and robots swimming together: Attraction towards the robot demands biomimetic locomotion,” _Journal of the Royal Society, Interface / the Royal Society_ , vol. 9, pp. 1856–68, 02 2012.
  * [24] R. K. Katzschmann, J. DelPreto, R. MacCurdy, and D. Rus, “Exploration of underwater life with an acoustically controlled soft robotic fish,” _Science Robotics_ , vol. 3, no. 16, p. 3449, 3 2018. [Online]. Available: http://robotics.sciencemag.org/
  * [25] Y. Xia and G. M. Whitesides, “Soft lithography,” _Annual Review of Materials Science_ , vol. 28, no. 1, pp. 153–184, 1998. [Online]. Available: https://doi.org/10.1146/annurev.matsci.28.1.153
  * [26] A. D. Marchese, R. K. Katzschmann, and D. Rus, “A recipe for soft fluidic elastomer robots,” _Soft Robotics_ , vol. 2, no. 1, pp. 7–25, 2015, pMID: 27625913. [Online]. Available: https://doi.org/10.1089/soro.2014.0022
  * [27] W. Tang, Y. Lin, C. Zhang, Y. Liang, J. Wang, W. Wang, C. Ji, M. Zhou, H. Yang, and J. Zou, “Self-contained soft electrofluidic actuators,” _Science Advances_ , vol. 7, no. 34, p. eabf8080, 2021. [Online]. Available: https://www.science.org/doi/abs/10.1126/sciadv.abf8080
  * [28] J. Shintake, V. Cacucciolo, H. Shea, and D. Floreano, “Soft biomimetic fish robot made of dielectric elastomer actuators,” _Soft Robotics_ , vol. 5, no. 4, pp. 466–474, 2018, pMID: 29957131. [Online]. Available: https://doi.org/10.1089/soro.2017.0062
  * [29] R. Pelrine, R. Kornbluh, Q. Pei, and J. Joseph, “High-speed electrically actuated elastomers with strain greater than 100%,” _Science (New York, N.Y.)_ , vol. 287, no. 5454, p. 836–839, 2000.
  * [30] N. Kellaris, V. G. Venkata, G. M. Smith, S. K. Mitchell, and C. Keplinger, “Peano-hasel actuators: Muscle-mimetic, electrohydraulic transducers that linearly contract on activation,” _Science Robotics_ , vol. 3, no. 14, p. eaar3276, 2018. [Online]. Available: https://www.science.org/doi/abs/10.1126/scirobotics.aar3276
  * [31] P. Rothemund, N. Kellaris, S. K. Mitchell, E. Acome, and C. Keplinger, “Hasel artificial muscles for a new generation of lifelike robots—recent progress and future opportunities,” _Advanced Materials_ , vol. 33, no. 19, p. 2003375, 2021. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/adma.202003375
  * [32] N. Kellaris, V. G. Venkata, P. Rothemund, and C. Keplinger, “An analytical model for the design of peano-hasel actuators with drastically improved performance,” _Extreme Mechanics Letters_ , vol. 29, p. 100449, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2352431619300173
  * [33] N. Kellaris, P. Rothemund, Y. Zeng, S. K. Mitchell, G. M. Smith, K. Jayaram, and C. Keplinger, “Spider-inspired electrohydraulic actuators for fast, soft-actuated joints,” _Advanced Science_ , vol. 8, no. 14, p. 2100916, 2021\. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/advs.202100916
  * [34] Z. Yoder, N. Kellaris, C. Chase-Markopoulou, D. Ricken, S. K. Mitchell, M. B. Emmett, R. F. f. Weir, J. Segil, and C. Keplinger, “Design of a high-speed prosthetic finger driven by peano-hasel actuators,” _Frontiers in Robotics and AI_ , vol. 7, 2020. [Online]. Available: https://www.frontiersin.org/article/10.3389/frobt.2020.586216
  * [35] C. Duriez, E. Coevoet, F. Largilliere, T. Morales-Bieze, Z. Zhang, M. Sanz-Lopez, B. Carrez, D. Marchal, O. Goury, and J. Dequidt, “Framework for online simulation of soft robots with optimization-based inverse model,” in _2016 IEEE International Conference on Simulation, Modeling, and Programming for Autonomous Robots, SIMPAR 2016_. Institute of Electrical and Electronics Engineers Inc., 2 2017, pp. 111–118.
  * [36] S. Tonkens, J. Lorenzetti, and M. Pavone, “Soft robot optimal control via reduced order finite element models,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_ , 2021, pp. 12 010–12 016.
  * [37] T. Du, K. Wu, P. Ma, S. Wah, A. Spielberg, D. Rus, and W. Matusik, “Diffpd: Differentiable projective dynamics,” _ACM Transactions on Graphics_ , vol. 41, no. 2, nov 2021.
  * [38] I. Robert J. Webster and B. A. Jones, “Design and kinematic modeling of constant curvature continuum robots: A review,” _The International Journal of Robotics Research_ , vol. 29, no. 13, pp. 1661–1683, 2010.
  * [39] D. Bruder, B. Gillespie, C. David Remy, and R. Vasudevan, “Modeling and control of soft robots using the koopman operator and model predictive control,” _arXiv: 1902.02827_ , 2019.
  * [40] D. A. Haggerty, M. J. Banks, P. C. Curtis, I. Mezić, and E. W. Hawkes, “Modeling, reduction, and control of a helically actuated inertial soft robotic arm via the koopman operator,” _arXiv: 2011.07939_ , 2020.
  * [41] M. Han, J. Euler-Rolle, and R. K. Katzschmann, “Desko: Stability-assured robust control with a deep stochastic koopman operator,” in _International Conference on Learning Representations_ , 2021.
  * [42] F. Faure, C. Duriez, H. Delingette, J. Allard, B. Gilles, S. Marchesseau, H. Talbot, H. Courtecuisse, G. Bousquet, I. Peterlik, and S. Cotin, “SOFA: A Multi-Model Framework for Interactive Physical Simulation,” in _Soft Tissue Biomechanical Modeling for Computer Assisted Surgery_ , ser. Studies in Mechanobiology, Tissue Engineering and Biomaterials, Y. Payan, Ed. Springer, Jun. 2012, vol. 11, pp. 283–321. [Online]. Available: https://hal.inria.fr/hal-00681539
  * [43] O. Goury and C. Duriez, “Fast, generic, and reliable control and simulation of soft robots using model order reduction,” _IEEE Transactions on Robotics_ , vol. 34, no. 6, pp. 1565–1576, 12 2018.
  * [44] R. K. Katzschmann, M. Thieffry, O. Goury, A. Kruszewski, T.-M. Guerra, C. Duriez, and D. Rus, “Dynamically closed-loop controlled soft robotic arm using a reduced order finite element model with state observer,” in _2019 2nd IEEE International Conference on Soft Robotics (RoboSoft)_. IEEE, 2019, pp. 717–724.
  * [45] Y. Hu, J. Liu, A. Spielberg, J. B. Tenenbaum, W. T. Freeman, J. Wu, D. Rus, and W. Matusik, “Chainqueen: A real-time differentiable physical simulator for soft robotics,” _Proceedings - IEEE International Conference on Robotics and Automation_ , vol. 2019-May, pp. 6265–6271, 5 2019.
  * [46] M. Geilinger, D. Hahn, J. Zehnder, M. Bächer, B. Thomaszewski, and S. Coros, “Add: Analytically differentiable dynamics for multi-body systemswith frictional contact,” _ACM Transactions on Graphics (TOG)_ , vol. 39, no. 6, p. 15, 11 2020.
  * [47] Y.-L. Qiao, J. Liang, V. Koltun, and M. C. Lin, “Differentiable Simulation of Soft Multi-body Systems,” in _Conference on Neural Information Processing Systems (NeurIPS)_ , 2021.
  * [48] P. Ma, T. Du, J. Z. Zhang, K. Wu, A. Spielberg, R. K. Katzschmann, and W. Matusik, “DiffAqua: A Differentiable Computational Design Pipeline for Soft Underwater Swimmers with Shape Interpolation,” in _arXiv:2104.00837 [cs]_ , Apr. 2021, arXiv: 2104.00837. [Online]. Available: http://arxiv.org/abs/2104.00837
  * [49] D. Hahn, P. Banzet, J. M. Bern, and S. Coros, “Real2sim: Visco-elastic parameter estimation from dynamic motion,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 6, 11 2019.
  * [50] J. Z. Zhang, Y. Zhang, P. Ma, E. Nava, T. Du, P. Arm, W. Matusik, and R. K. Katzschmann, “Learning Material Parameters and Hydrodynamics of Soft Robotic Fish via Differentiable Simulation,” _arXiv:2109.14855 [cs]_ , Sep. 2021, arXiv: 2109.14855. [Online]. Available: http://arxiv.org/abs/2109.14855
  * [51] M. Dubied, M. Y. Michelis, A. Spielberg, and R. K. Katzschmann, “Sim-to-real for soft robots using differentiable fem: Recipes for meshing, damping, and actuation,” _IEEE Robotics and Automation Letters_ , pp. 1–1, 2022.
  * [52] M. Curatolo and L. Teresi, “The virtual aquarium: simulations of fish swimming,” in _Proc. European COMSOL Conference_ , 2015.
  * [53] S. K. Mitchell, X. Wang, E. Acome, T. Martin, K. Ly, N. Kellaris, V. G. Venkata, and C. Keplinger, “An easy-to-implement toolkit to create versatile and high-performance hasel actuators for untethered soft robots,” _Advanced Science_ , vol. 6, no. 14, p. 1900178, 2019. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/advs.201900178
  * [54] M. Curatolo and L. Teresi, “The virtual aquarium: simulations of fish swimming,” in _Proc. European COMSOL Conference_ , 2015.
  * [55] S. Min, J. Won, S. Lee, J. Park, and J. Lee, “Softcon: Simulation and control of soft-bodied animals with biomimetic actuators,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 6, pp. 1–12, 2019.
  * [56] A. Lukezic, T. Vojir, L. ˇCehovin Zajc, J. Matas, and M. Kristan, “Discriminative correlation filter with channel and spatial reliability,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6309–6318.
  * [57] T. Du, J. Hughes, S. Wah, W. Matusik, and D. Rus, “Underwater soft robot modeling and control with differentiable simulation,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4994–5001, 2021.
  * [58] N. Obayashi, C. Bosio, and J. Hughes, “Soft passive swimmer optimization: From simulation to reality using data-driven transformation,” in _5th IEEE-RAS International Conference on Soft Robotics (RoboSoft 2022)_ , no. CONF, 2022.
  * [59] N. Wandel, M. Weinmann, and R. Klein, “Learning Incompressible Fluid Dynamics from Scratch – Towards Fast, Differentiable Fluid Models that Generalize,” in _International Conference on Learning Representations (ICLR 2021)_ , Mar. 2021, arXiv: 2006.08762. [Online]. Available: http://arxiv.org/abs/2006.08762