#  SRL-VIC: A Variable Stiffness-based Safe Reinforcement Learning
for Contact-rich Robotic Tasks

Heng Zhang1,2, Gokhan Solak1, Gustavo J. G. Lahr1, Arash Ajoudani1 Manuscript
received: January 11, 2024; Revised March 28, 2024; Accepted April 24,
2024.This paper was recommended for publication by Editor Jens Kober upon
evaluation of the Associate Editor and Reviewers’ comments. This work was
supported part by the European Union’s Horizon 2020 research and innovation
programme SOPHIA (GA 871237) and CONCERT (GA 101016007).1 Human-Robot
Interfaces and Interaction Lab, Istituto Italiano di Tecnologia, Genoa, Italy.
e-mails<EMAIL_ADDRESS>Ph.D. program of national interest in Robotics and
Intelligent Machines (DRIM) and Università di Genova, Genoa, Italy.Digital
Object Identifier (DOI): see top of this page.

###### Abstract

Reinforcement learning (RL) has emerged as a promising paradigm in complex and
continuous robotic tasks, however, safe exploration has been one of the main
challenges, especially in contact-rich manipulation tasks in unstructured
environments. Focusing on this issue, we propose SRL-VIC: a model-free safe RL
framework combined with a variable impedance controller (VIC). Specifically,
safety critic and recovery policy networks are pre-trained where safety critic
evaluates the safety of the next action using a risk value before it is
executed and the recovery policy suggests a corrective action if the risk
value is high. Furthermore, the policies are updated online where the task
policy not only achieves the task but also modulates the stiffness parameters
to keep a safe and compliant profile. A set of experiments in contact-rich
maze tasks demonstrate that our framework outperforms the baselines (without
the recovery mechanism and without the VIC), yielding a good trade-off between
efficient task accomplishment and safety guarantee. We show our policy trained
on simulation can be deployed on a physical robot without fine-tuning,
achieving successful task completion with robustness and generalization. The
video is available at https://youtu.be/ksWXR3vByoQ.

###### Index Terms:

Reinforcement Learning, Compliance and Impedance Control, Robotics and
Automation in Construction

##  I INTRODUCTION

Exploring unstructured environments is a key skill for future robotics, where
the robots will be employed in unknown and potentially hostile surroundings.
The inherent physical interactions in such scenarios pose risks to robots,
humans, and the environment. These risks are increased in the context of
automated learning and exploration, e.g., in RL. Indeed, ensuring the safety
of RL in tasks involving rich contact remains an ongoing challenge, especially
in unstructured environments.

We study the problem of safe RL on a maze exploration scenario as an
illustrative example of contact-rich tasks. This scenario is inspired by the
common construction task of installing electricity cables within walls, as
depicted in Fig. 1. Traditionally, these tasks are done by workers and are
time-consuming and labor-intensive. These characteristics indicate high
potential for robotic automation, but raise several challenges: 1) The
environment is unknown to the robot; 2) Visual perception may be difficult to
deploy, thus physical contact is essential; 3) Frequent contact with the
environment causes safety issues.

Figure 1: Maze exploration is an unstructured contact-rich task that has
practical applications such as cable-lying in walls. It also has similarities
to search missions in dark and narrow environments. The agent does not have
access to vision, thus it should navigate using only the contact information.

To this end, we abstracted the cable-lying task into a robot exploration task
since all the slots in the wall are composed of two basic shapes (turns and
straight sections) as shown in Fig. 1. Then, we created an experimental setup
that includes a 7-DoF robot arm and a maze with obstacles as shown in Fig. 1.

In tasks involving rich contact, the robot must engage in continuous
interaction with the environment, presenting a notable safety risk that could
result in damage to either the robot itself or objects in its surroundings.
Impedance control is widely used to alleviate such hazards during physical
robot interaction [1]. Moreover, variable impedance control (VIC) methods
enable adapting impedance parameters to the changing conditions of the task or
the surrounding environment [2, 3]. The robot can trade between compliance and
accuracy to maximize task performance while staying safe. Nevertheless,
determining the adjustment of these parameters in VIC, particularly in
unstructured environments with dynamic obstacles, remains a challenging
question.

Learning-based approaches became a powerful way to address this challenge,
with a wide variety of methods for tuning impedance parameters [2]. In
previous works, such parameters were learned from human (or expert)
demonstrations using Dynamic movement primitives (DMPs) [4], inverse
reinforcement learning (IRL) [5] and skill priors [6]; however, providing
demonstrations in advance is relatively time-consuming and labor-intensive.
Also, it can be impractical to provide expert demonstrations in some tasks
where the environment is unstructured or potentially hostile.

In summarizing the current state-of-the-art research on contact-rich tasks,
two notable approaches emerge: 1) Some studies employ VIC to ensure safe
performance while utilizing RL for learning impedance parameters [6][7]; 2)
Other works leverage the predictive capabilities of neural networks to
anticipate the safety value of future actions, allowing for informed decisions
on whether to pursue a safer course of action in advance [8][9]. Each of these
ideas has its own advantage: the former is to adjust the behaviour of the
robot reactively based on the observed contact forces, while the latter is to
proactively select a safer robot behaviour by predicting the safety of future
actions, therefore avoiding actions that would generate dangerous amounts of
contact force. They both improve the safety of robots when performing contact-
rich tasks to some extent.

Compared with other previous work, in this paper, we take a step towards
combining the advantages of an RL-based VIC approach with the advantages of
using a predictive safe RL approach. We introduce a new safe-RL framework
incorporating VIC for contact-rich tasks. Following the recovery RL (RRL)
method [8], during the pre-training phase, we train a Safety-Critic network
using procedurally collected offline data, which can predict a risk value
based on the current state and the next action. Meanwhile, we pre-train a
recovery policy network, which can sample a safe action if the risk value
exceeds the safety threshold. In the online training phase, a task policy is
trained with the goal of maximizing the reward. Unlike previous works, we do
not use expert demonstrations to accelerate task policy training, given that
the environment is unknown to the robot.

We compare the proposed framework with scenarios where VIC is absent and where
a recovery mechanism is not implemented. The results of our simulation
experiments indicate that the combination of VIC and recovery mechanism
improves the safety and efficiency of the solution.

To summarize, the main contributions of our work are listed as follows:

1) We present a safe RL framework with VIC for contact-rich tasks without
prior knowledge of the environment. Prior to action execution, the safety
critic predicts the safety of the action. The recovery policy is activated for
risky situations. During the action execution, the contact force is regulated
with the help of VIC to enhance safety.

2) We developed a contact-rich maze exploration task in simulations, and
empirically demonstrated the proposed framework outperforms baselines.

3) We demonstrate that our policy trained on simulation can be deployed on a
physical robot without fine-tuning, successfully achieving the tasks even with
different type of obstacles, maze sizes and shapes, and flange sizes.

##  II Related work

###  II-A Contact-rich robotic tasks

The solutions for contact-rich tasks have undergone a shift from contact
modeling analysis [10] to learning-based approaches [11] [12]. However, safety
issues have always been an inescapable aspect in such contact-rich tasks.

###  II-B Variable impedance control for contact-rich tasks

Implementing VIC enables a robot to exhibit a certain degree of compliance,
contributing to the achievement of safe behavior in contact-rich tasks.
However, it is a challenge to pre-define the impedance parameters according to
different tasks or different stages of the task. To solve this problem, Yang
et al. [13] utilized muscle surface electromyography (sEMG) signals to extract
the demonstrator’s variable stiffness which were passed to the robot. In this
type of method, however, specialized equipment is required. Lee et al. [14]
proposed a stiffness modulation method that achieved safe and robust
performance in a door opening task, while it focused on stiffness in joint
space. A self-turning impedance was proposed in [3], which ensures compliant
contact and low contact forces based on two planning strategies. However, the
direction of motion of the robot will not change unless large contact forces
are generated, which means that the actual trajectory executed by the robot
may not be optimal with respect to energy and time. Furthermore, some scholars
[6, 15, 16] learn different impedance parameters from collected expert
demonstration trajectories. However, it is difficult to demonstrate
trajectories for tasks like ours, i.e., blind maze exploration. Moreover, some
works address it by utilizing the impedance properties of the human body. Yang
et al. [13, 17] transferred the stiffness profile from the upper limb of the
human to a humanoid dual-arm robot for contact-rich tasks.

Therefore, these methods of obtaining variable impedance parameters either
need demonstrations or require special devices or are specific to a single
task.

###  II-C Safe RL for contact-rich tasks

RL-based works focus on contact-rich tasks can be divided into two groups
depending on when contact safety is increased. On the one hand, a safe
controller with variable stiffness can be learned via RL. Specifically,
Martin-Martin et al. [7] compared the effect of different action spaces in RL
with variable impedance control in end-effector space for contact-rich tasks,
such as surface-wiping and door-opening. Some work [4, 5, 6] leveraged
demonstrations to learn variable impedance parameters as part of the action
space. On the other hand, a safe action or policy can be learned with RL to
significantly enhance safety during the training stage. For instance, MoPA-RL
[18] augmented the action space with long-horizon planning to improve safety.
Safety critic network can be trained that generate a risk value to help task
policy sample a conservative or relatively safe action, works such as RRL [8]
and CSC [19].

Figure 2: The proposed framework: we combine the recovery-based safe RL
approach [8] with VIC to solve the contact-rich maze exploration task. We
first use an automated procedure to collect the offline data and pre-train our
safety critic and recovery policy. Then, we train all learning components
using online data. The risk value $\epsilon_{risk}$ is used to activate either
the task or recovery policy. The action $a_{t}$ is chosen by the activated
component, and it is fed to the VIC. Our action includes a relative position
change and desired stiffness vector $\\{K_{x},K_{y}\\}$.

However, in contrast to these, our proposed framework combines the strengths
of both. We improve safety in terms of both impedance parameters in the
controller and safety prediction during training. Furthermore, unlike most of
the existing methods, we do not use expert demonstrations for task learning.
Our method uses a scripted procedure to demonstrate only the concept of
collision constraints which is a more immediate and simpler concept compared
to a long-term task, hence easier to program.

##  III Approach

We introduce the details of our approach in this section. In Section III-A,
the fundamentals of impedance control in task space are presented, which is
our downstream controller. Then, we describe the proposed RRL-based VIC in
Section III-B. A diagram of the proposed framework is shown in Fig. 2.

###  III-A Impedance control in task space

The robotic manipulator rigid body model in joint space may be written as

$\boldsymbol{M}(\boldsymbol{q})\ddot{\boldsymbol{q}}+\boldsymbol{C}(\boldsymbol{q},\dot{\boldsymbol{q}})\dot{\boldsymbol{q}}+\boldsymbol{g}(\boldsymbol{q})=\boldsymbol{\tau}+\boldsymbol{\tau}_{ext},$
(1)

where
$\boldsymbol{q},\dot{\boldsymbol{q}},\ddot{\boldsymbol{q}}\in\mathbb{R}^{n}$
are the position, velocity, and acceleration vectors in joint space,
respectively, and $n$ is the number of DoF.
$\boldsymbol{M}(\boldsymbol{q})\in\mathbb{R}^{n\times n}$ is the joint space
inertia matrix,
$\boldsymbol{C}(\boldsymbol{q},\dot{\boldsymbol{q}})\in\mathbb{R}^{n}$ the
Coriolis term, $\boldsymbol{g}(\boldsymbol{q})\in\mathbb{R}^{n}$ the gravity
term, and $\boldsymbol{\tau}\in\mathbb{R}^{n}$ is the actuation torques. The
external torques generated by the interaction forces are given by
$\boldsymbol{\tau}_{ext}=\boldsymbol{J}^{T}(\boldsymbol{q})\boldsymbol{F}_{ext}$,
being $\boldsymbol{J}^{T}(\boldsymbol{q})\in\mathbb{R}^{n\times 6}$ the
Jacobian transpose matrix.

The actuation torques are chosen to compensate for the Coriolis and
gravitational vectors, plus to deal with the external forces risen by contact
with the environment through an impedance model in task space:

$\boldsymbol{F}_{ext}=\boldsymbol{D}^{d}\dot{\tilde{\boldsymbol{x}}}+\boldsymbol{K}^{d}\tilde{\boldsymbol{x}},$
(2)

where $\tilde{\boldsymbol{x}}$ is the Cartesian pose error given by the
difference between the desired Cartesian pose
$\boldsymbol{x}_{d}\in\mathbb{R}^{6}$ and the actual pose
$\boldsymbol{x}\in\mathbb{R}^{6}$. Accordingly, $\dot{\tilde{\boldsymbol{x}}}$
is the velocity error between the desired and actual end-effector’s velocity,
$\dot{\boldsymbol{x}},\dot{\boldsymbol{x}}_{d}\in\mathbb{R}^{6}$,
respectively. The desired stiffness matrix
$\boldsymbol{K}^{d}\in\mathbb{R}^{6\times 6}$ is a diagonal matrix with
variable terms as
$\boldsymbol{K}^{d}$=$diag\\{K_{x},K_{y},K_{z},0.15K_{x},0.15K_{y},0\\}$,
where the rotation elements for $[K_{x},K_{y}]$ are set as 0.15 times of
$[K_{x},K_{y}]$ inspired by [3] to reduce and simplify the parameters tuning.
We set the rotational z-axis stiffness to zero, to keep it indifferent w.r.t.
the end-effector roll. The desired Cartesian damping matrix
$\boldsymbol{D}^{d}\in\mathbb{R}^{6\times 6}$ is also diagonal and its terms
are defined by the double diagonal rule, i.e.,
$\boldsymbol{D}_{i}^{d}(t)=2\zeta\sqrt{\boldsymbol{K}_{i}^{d}}$ where $i$
represents the linear or rotational components in Task Space. We use
$\zeta{=}0.707$ in our experiments. The multi-dimensional stiffness can be
represented with a stiffness ellipsoid having different compliance in
different dimensions [3].

###  III-B Safe RL for contact-rich tasks

We adopt the RRL framework [8] to actively predict safety hazards and take
corrective actions. The original RRL was not specifically designed for
contact-rich tasks. In [8] contact was avoided, on the contrary, in our work
continuous contact is required but possibly unsafe. Thus, we introduce
impedance control in the task space to better handle the contact-rich aspect
of the problem. Specifically, we add the Cartesian impedance parameters to the
action space in RL.

As shown in Fig. 2, there are three components in our framework: Firstly,
similar to RRL, a safety critic network and recovery policy are trained with
the pre-collected offline data. Then, our task policy was trained online with
the maze exploration experiments. The output of the task policy is given to
the variable impedance controller which controls the robot to achieve the
desired actions.

In this framework, the problem is defined as a constrained Markov decision
process (CMDP) problem [20]. A tuple
$(\mathcal{S},\mathcal{A},R,P,\gamma,\mu,\mathcal{C})$ denotes a CMDP, where
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function,
$P$ is the state transition probability, $\gamma\in(0,1)$ is a reward discount
factor, $\mu$ is the starting state distribution and
$\mathcal{C}=\left\\{\left(c_{i}:\mathcal{S}\rightarrow\\{0,1\\},\chi_{i}\in\mathbb{R}\right)\mid
i\in\mathbb{Z}\right\\}$ denote safety constraints that the agent must
satisfy. In our task, we set $\gamma$ as 0.9 and we define the other
components as detailed below.

  * •

State Space $\mathcal{S}$. To make our framework more general to contact-rich
exploration tasks instead of this specific maze, we differentiate the state
space across different networks: it is 6-dimensional vector for safety critic
network and recovery policy including 6 force/torque
[$F_{x},F_{y},F_{z},T_{x},T_{y},T_{z}$] values measured from a F/T sensor. We
do not give the position as an input to the safety network, so that it does
not memorise the locations of the walls. While for task policy, we add the
position [$P_{x},P_{y},P_{z}$] of the end-effector, obtaining a 9-dimensional
vector as state space so that the agent remembers the position when finding
its way to the goal.

  * •

Action Space $\mathcal{A}$. We combine 2 impedance parameters $[K_{x},K_{y}]$
with 2 end-effector displacement values $[\Delta P_{x},\Delta P_{y}]$ which
consist of 4-dimensional vector as action space. Note that here the
displacement values $\Delta P$ are not absolute positions in the world
coordinate but the desired change w.r.t. the end-effector position. The sample
range of $\Delta P$ is [-0.03, 0.03] m and the impedance values
$[K_{x},K_{y}]$ are within the sample range of [300, 1000] N/m. We project the
stiffness values onto the $\Delta P$ vector components as
$[K_{x}^{*},K_{y}^{*}]$ in the controller:

$\displaystyle K_{x}^{*}=$ $\displaystyle|K_{x}\cdot\Delta
P_{x}/\left\|\Delta\boldsymbol{P}\right\||+|K_{y}\cdot-\Delta
P_{y}/\left\|\Delta\boldsymbol{P}\right\||$ (3) $\displaystyle K_{y}^{*}=$
$\displaystyle|K_{x}\cdot\Delta
P_{y}/\left\|\Delta\boldsymbol{P}\right\||+|K_{y}\cdot\Delta
P_{x}/\left\|\Delta\boldsymbol{P}\right\||$

  * •

Reward Function $R$. We use a negative Euclidean distance between the current
position $P_{cur}$ of the end-effector and the goal position $P_{goal}$
multiplied by a constant $c_{pos}{=}100$. We introduce a penalty for high-
force collisions $r_{col}{=}{-}250$ and a penalty for leaving from the
entrance $r_{ent}{=}{-}500$ to satisfy our task requirements. Also we add a
bonus for reaching the goal $r_{goal}{=}1000$ to increase the speed of
convergence.

  * •

Safety Constraints $\mathcal{C}$. The force constraint requires the measured
force magnitude to stay below a threshold.

####  III-B1 Safety critic and recovery policy

Based on [21], our safety critic learn a critic function $Q^{\pi}_{risk}$:

$\displaystyle
Q_{\text{risk}}^{\pi}\left(s_{t},a_{t}\right)=\mathbb{E}_{\pi}\left[\sum_{t^{\prime}=t}^{\infty}\gamma_{\text{risk
}}^{t^{\prime}-t}c_{t^{\prime}}\mid s_{t},a_{t}\right]$ (4)
$\displaystyle\quad=c_{t}+\left(1-c_{t}\right)\gamma_{\text{risk
}}\mathbb{E}_{\pi}\left[Q_{\text{risk }}^{\pi}\left(s_{t+1},a_{t+1}\right)\mid
s_{t},a_{t}\right]$

where $c_{t}=1$ denotes constraint violation, and $c_{t}=0$ indicates safety
in state $s_{t}$. $Q^{\pi}_{risk}$ is trained by minimizing a MSE loss
function:

$\displaystyle J_{\text{risk
}}\left(s_{t},a_{t},s_{t+1};\phi\right)=\frac{1}{2}\biggl{(}\hat{Q}_{\phi,\text{risk}}^{\pi}\left(s_{t},a_{t}\right)-$
(5)
$\displaystyle\left(c_{t}+\left(1-c_{t}\right)\gamma_{\text{risk}}\underset{a_{t+1}\sim\pi\left(\cdot\mid
s_{t+1}\right)}{E}\left[\hat{Q}_{\phi,\text{risk}}^{\pi}\left(s_{t+1},a_{t+1}\right)\right]\right)\biggr{)}^{2}$

Similar to [8], we use an off-policy DDPG [22] algorithm to learn recovery
policy $\pi_{rec}$ by performing gradient descent on the safety critic network
$\hat{Q}_{\phi,\text{ risk }}^{\pi}\left(s,\pi_{\text{rec}}(s)\right)$.

####  III-B2 Pre-training

By providing a set of offline data for the agent, we expect our agent to learn
the relationship of safety with contact force and motion. It is possible to
avoid catastrophic collisions caused by unexpected actions during online
training by learning the implicit relationship between the two in advance.
Details of the data collection procedure are described in Section IV-B1.
Unlike online training where the agent searches for actions that maximise the
reward, in this stage, we run a scripted behavior to collect examples of
collisions without any reward. This approach reduces the problem complexity by
separating task learning and constraint learning which are otherwise
interlaced.

We do not pre-train the task policy, because this requires recording complex
expert demonstrations which are costly and difficult to obtain for a blind
maze exploration task.

We pre-train the safety critic network and recovery policy with the offline
data which are collected in simulation. Specifically, We use 256 as batch size
according to the number of hidden units in the network and update the safety
critic network and recovery policy 10000 times in the pre-training stage.

####  III-B3 Task policy

The aim of the task policy is to finish the task and obtain maximum rewards
within the constraints of safety. We train the task policy by the SAC
algorithm [23], which is based on the idea of maximum entropy. The main
difference from other RL algorithms is that SAC maximizes the entropy while
optimizing the policy for higher cumulative returns. For our task, the benefit
of introducing entropy into the learning stage is the increased randomization.
Therefore, the agent can sufficiently explore the state space to avoid getting
stuck in the local optimum. Our task policy works in complementarity with the
recovery network described in III-B1. Action is selected as follows depending
on the risk value:

$a_{t}=\begin{cases}a_{t}^{\pi_{\text{task
}}}&{Q}_{\left(s_{t},a_{t}^{\pi_{\text{task }}}\right)}\leq\epsilon_{risk}\\\
a_{t}^{\pi_{\mathrm{rec}}}&{Q}_{\left(s_{t},a_{t}^{\pi_{\text{task
}}}\right)}>\epsilon_{risk}\end{cases}$ (6)

where $a_{t}^{\pi_{\text{task }}}\sim\pi_{\text{task }}\left(\cdot\mid
s_{t}\right)$ and $a_{t}^{\pi_{\text{rec }}}\sim\pi_{\text{rec
}}\left(\cdot\mid s_{t}\right)$. Different from task policy, the recovery
policy is trained to minimize the $\hat{Q}_{\phi,\text{ risk
}}^{\pi}\left(s_{t},a_{t}\right)$ so that it can provide a safe action
$a_{t}^{\pi_{\text{rec }}}$ with small risk value i.e.
$Q_{\left(s_{t},a_{t}^{\pi_{\text{r}ec}}\right)}<\epsilon_{risk}$.

####  III-B4 Online training

All of the policy and safety critic networks are updated online, where task
policy $\pi_{task}$ is only trained in the online RL stage while recovery
policy $\pi_{rec}$ and safety critic are also pre-trained with offline data.

In every single step of training, we update the replay buffer with online data
where each one is denoted as a tuple $[s_{t},a_{t},r_{t},s_{t+1},done]$. We
set 500 as horizon for each episode, in each transition, actions are sampled
from the policies and are executed continuously until one of the termination
conditions is met. The termination conditions are: violating a constraint,
leaving the maze from the entrance, achieving success, and reaching the
maximum step count. The networks are updated online after collecting each
observation.

##  IV Evaluation

In the following evaluation experiments, we aim to study whether the proposed
framework is: 1) safer; 2) more efficient, in comparison to our baselines,
which are standard RL with VIC, and RRL with constant stiffness ($K{=}300$ and
$1000$ N/m). We label the baselines in this order as Std_RL-VIC, SRL-K300,
SRL-K1000. In Std_RL-VIC, we use the same RL algorithm without the recovery
mechanism to highlight the added benefit of our framework that combines VIC
with RRL. The other baselines SRL-K300 and SRL-K1000 are used to evaluate the
benefit of the VIC. Safety can be judged by the absence of excessively high
interaction forces, whereas efficiency can be defined as attaining high task
successes and reward with less training.

###  IV-A Experiment setup

We evaluate the proposed framework on a maze exploration task using the 7-DoF
Franka Emika Panda arm in Mujoco simulation environment as shown in Fig. 1. We
use a peg-shaped flange mounted on the end-effector of the robot with the
diameter of 30 mm and the length of 55 mm. The maze channel is 50 mm in width
70.35 cm in length. Because there is a small space for the robot to explore
width-wise, we can treat it also as an extended peg-in-hole task. The goal for
the agent is to reach from entrance to exit without constraint violations. We
adhere to safety level 1 [24], in which the agent will receive a penalty when
it violates the constraints, and the current training episode will be
terminated. 33footnotetext: The orange circle denotes goal point of the task,
the cyan circles and yellow arrows along the trajectory represent the
stiffness ellipsoids and the motion directions respectively.

This problem combines two challenges of finding the way to exit without
vision, and staying safe by avoiding high contact forces. It creates a dilemma
as the contacts are absolutely necessary to navigate while they are also
risky. Furthermore, we add movable obstacles in the maze that requires the
robot to push harder. Thus, resorting to a conservative policy cannot solve
the problem. The agent needs to learn both the safe behaviour and the task
achievement simultaneously.

We implement the obstacles as three heavy balls in simulation, and a pile of
bolts in real-world. We position the obstacles in the middle of the maze.

We choose MuJoCo simulator (version 2.3.3) for its favorable multi-contact
physics computation, suitable for contact-rich tasks. However, this required
us to split the maze mesh into smaller convex components because non-convex
objects are not allowed in MuJoCo. We used V-HACD44footnotetext:
https://github.com/kmammou/v-hacd library to decompose our maze into 128
convex primitive shapes and assembled them into the same maze.

\begin{overpic}[width=433.62pt]{figs/training_figs/trajs_in_training1_update.pdf}
\put(4.0,62.0){\small\color[rgb]{0,.5,.5}{episode 10}}
\put(4.0,47.0){\small\color[rgb]{0,.5,.5}{episode 40}}
\put(4.0,28.0){\small\color[rgb]{0,.5,.5}{episode 210}}
\put(4.0,12.0){\small\color[rgb]{0,.5,.5}{episode 1500}} \end{overpic} Figure
3: Trajectories of the end-effector during different stages of training, in
top-down viewIV-A.

The safe recovery RL part of our work is implemented based on the RRL
codebase555https://github.com/abalakrishna123/recovery-rl. We follow the same
hyperparameter procedure, except for: $10$ training updates per simulation
step, 0.65 as $\gamma\\_risk$, 0.7 as $\epsilon\\_risk$ and 500 as the maximum
number of actions for each episode. Another difference from RRL is that we
normalize the actions of the RL model and denormalize them before applying
them in the robot controller, for faster convergence.

###  IV-B Simulation experiments

We first collect offline data to pre-train the safety critic and the recovery
policy. Then, we train all approaches online and compare their performances.

####  IV-B1 Offline data collection

In order to teach the notion of risk, we generate random states and actions
that are likely to cause constraint violation. The robot moves to one of the
six predefined points inside the maze, then moves in a random direction, which
is likely to collide with the maze walls. The number of transitions are split
equally between each point. In order to make the collected data more diverse,
we add noise to the coordinates of each point, so that we have many different
starting positions. Then, we sample a move direction as an angle between
$0^{\circ}$ and $360^{\circ}$ and a move size in the action range. Lastly, the
robot is commanded to move in that direction and the transition is stored as a
tuple $(s_{t},a_{t},c_{t},s_{t+1},done)$. Offline data for pretraining are
given $40018$ transitions of data, $185$ of these transitions contain
constraint-violating states.

Figure 4: Learning curves for maze exploration. Our framework (SRL-VIC)
outperforms others, particularly in terms of ratio of successes/violation and
cumulative violations, i.e., our framework is safer than others. Furthermore,
it learns faster, achieving success at an earlier stage. The large deviation
area in the ratio of successes/violations comes from the division operation;
the cumulative successes and violations do not exhibit large variation.

####  IV-B2 Online training

We train each setup online for 3 runs with 1500 episodes in each run. In each
episode, the robot needs to move to the start point and then begin exploring
the maze, until it meets termination conditions. We train the policy using 1
chunk of 20 cores and one GPU (NVIDIA Tesla V100 16Gb) from the cluster
Franklin High Performance Computing (HPC)666We gratefully acknowledge the Data
Science and Computation Facility and its Support Team at Fondazione Istituto
Italiano di Tecnologia.. The average training time for 3 runs of each
experiment SRL-VIC, Std_RL-VIC, SRL-K1000, SRL-K300 is 14.47, 7.32, 9.08,
68.28 hours respectively. The time gets shorter when episodes terminate early
by constraint violation or task success.

Fig. 3 shows the evolution of the robot’s behaviour with increasing episodes
of online learning. In the beginning, the agent fails because it violates the
constraints while interacting with the obstacles. The agent learns to complete
the task in later episodes. The stiffness ellipsoids indicate that the agent
learns to adapt the stiffness to different conditions such as obstacle or wall
contact. We observe that the trajectory becomes smoother and the robot
finishes the task faster with further training.

####  IV-B3 Results

We compare the proposed framework with baselines in four different
perspectives: ratio of successes/violations, reward, cumulative task
successes, cumulative violations. Specifically, cumulative task successes
indicate the effectiveness of task completion; while cumulative violations
show the number of constraint violations that indicates the safety; ratio of
successes/violations show the trade-off between task completion and safety
guarantee. The results of the simulation training are presented in Fig. 4.

In terms of safety, our method achieves minimal cumulative violations apart
from SRL-K300. Although SRL-K300 has the minimum cumulative violations among
all setups, it gets stuck in the obstacle area and cannot move forward. Thus,
our method has an overwhelming advantage in the ratio of successes/violations
(Fig. 4). We also tested the final learned model of SRL-VIC 100 times in
simulation and observed that it was successful in all cases without any
violation.

\begin{overpic}[trim=0.0pt 0.0pt 0.0pt
85.35826pt,width=407.60385pt]{figs/training_figs/K300FailedwithObstacles2.pdf}
\put(6.0,47.0){\small\color[rgb]{0,.5,.5}{SRL-K300}}
\put(6.0,16.0){\small\color[rgb]{0,.5,.5}{SRL-VIC}} \end{overpic} Figure 5:
Different behaviours when encountering obstacles at training episode 1000IV-A.
Top (SRL-K300): The robot cannot move forward and fails due to low stiffness.
Bottom (SRL-VIC): VIC approach switches to high stiffness when there are some
obstacles on the way. The method learns to safely push the obstacles without
exceeding the force threshold.

Our method achieves the best results also in means of task successes (Fig. 4).
Although Std_RL-VIC performed fairly well in means of reward and cumulative
task successes, our framework improves it further. The difference is even
stronger in the success/violation ratio results. Meanwhile, our method
significantly outperforms the other recovery-based methods (SRL-K300 and
SRL-K1000). SRL-K300 cannot obtain high performance because it is not stiff
enough to deal with obstacles. Fig. 5 shows the advantage of our method over
SRL-K300 when encountering obstacles. Please note the increased size of the
stiffness ellipsoids when the robot makes contact with the obstacles.

---
Figure 6:  Relation between the risk value and state/action values in the
offline test. Top: The relationship of the risk $\epsilon_{risk}$ to stiffness
$\left\|\boldsymbol{K}\right\|$ and desired move
$\left\|\Delta\boldsymbol{P}\right\|$ values. Bottom: Plot of
$\epsilon_{risk}$ and contact force $\left\|\boldsymbol{F}\right\|$. The force
magnitude is scaled down by $1/100$.  \begin{overpic}[trim=0.0pt 270.30118pt
0.0pt 28.45274pt,width=420.61192pt]{figs/real_exp_snapshot4_new.pdf}
\put(1.5,22.0){\small\hbox{\pagecolor{white}a}}
\put(26.0,21.7){\small\hbox{\pagecolor{white}b}}
\put(51.0,22.0){\small\hbox{\pagecolor{white}c}}
\put(75.5,21.7){\small\hbox{\pagecolor{white}d}}
\put(1.5,10.0){\small\hbox{\pagecolor{white}e}}
\put(26.0,9.7){\small\hbox{\pagecolor{white}f}}
\put(51.0,10.0){\small\hbox{\pagecolor{white}g}}
\put(75.5,9.7){\small\hbox{\pagecolor{white}h}} \end{overpic} Figure 7: (a)
Maze shape-1, without obstacles. The yellow overlay shows the original maze
from the simulation. (b) The robot meets the pile of bolts halfway. (c) The
obstacles create congestion at the curve. (d) The SRL-VIC policy overcomes the
obstacles by increasing the stiffness. (e, f) The complex task scenario with
small wall segments at new positions. (g) Maze shape-2. (h) Maze shape-3,
failed at red X marker.

Regarding speed, SRL-VIC has to take relatively conservative actions in unsafe
situations. Therefore, it achieved the task in slightly longer times compared
to SRL-K1000 and Std_RL-VIC, but it was still faster than K300.

The results strongly demonstrate that our method provides a good trade-off
between safety and task performance by combining the benefits of VIC and SRL.
The fact that Std_RL-VIC is the second-best method, performing better than the
other baselines of SRL with constant stiffness values, confirms the importance
of VIC in dealing with contact-rich tasks. The VIC allows switching between
the low-stiffness behaviour and high-stiffness behaviour under different
conditions to use the advantage of both. We also observe that it stays stiffer
along the motion direction while it stays more compliant laterally. However,
the proposed method improves the performance even further as the safety critic
provides a predictive mechanism to avoid dangerous situations before getting
too close. It decreases the violations, thus allows the task policy to explore
more states without failing. Additionally, the recovery policy takes the robot
away from the risky state, making it harder to get stuck.

####  IV-B4 Safety critic behaviour

In order to understand the behaviour of our safety critic network, we
visualise the relationship between the predicted risk value and the
state/action values. The robot was commanded to repetitively apply a random
action 200 times; starting on a random point (with 0.5 cm deviation), and
moving in random direction (in [$0^{\circ},360^{\circ}$]) with random move
size and stiffness. We recorded the risk value produced for each of these
cases and plotted them in Fig. 6.

As seen in the figure, higher $\Delta\boldsymbol{P}$ and $\boldsymbol{K}$
values are correlated with higher $\epsilon_{risk}$. This matches the human
intuition that stronger actions are riskier. We also see that the model is
sensitive to the contact force, as the the spikes in $\boldsymbol{F}$ are
aligned with the spikes in $\epsilon_{risk}$. However, it does not merely
follow the force, as there is a more complex relationship between the
state/action values and risk. These outcomes confirm that the system acts
reasonably.

\begin{overpic}[trim=14.22636pt 113.81102pt 85.35826pt
213.39566pt,width=385.92152pt]{figs/trajs_with_ellipses6.pdf}
\put(0.5,93.0){\small\color[rgb]{0,0,0}{(a)}}
\put(0.5,81.0){\small\color[rgb]{0,0,0}{(b)}}
\put(0.5,67.0){\small\color[rgb]{0,0,0}{(c)}}
\put(0.5,51.0){\small\color[rgb]{0,0,0}{(d)}}
\put(0.5,39.0){\small\color[rgb]{0,0,0}{(e)}}
\put(0.5,22.0){\small\color[rgb]{0,0,0}{(f)}} \end{overpic} Figure 8:
Trajectories of the end-effector during executing task in the real worldIV-A.
(a-c) Maze shape-1 (Fig.7.a): (a) no obstacle, (b) small obstacles, (c) mixed
obstacles. (d, e) Complex task scenario with discrete walls, (f) Maze shape-2
(Fig.7.g).

###  IV-C Real-world experiments

We trained the policy in Mujoco for the contact-rich maze-exploration task and
deployed it on a physical 7-DOF Franka robot arm without any fine-tuning. In
the setup, we use Robot Operating System (ROS) where RL policy is a separate
ROS node communicating with the physical robot.

Our policy succeeded in 4 out of 5 trials at first. In the failed case, the
robot got stuck at a turning point. It was not highly robust to the real-world
environment because our policy was trained with the observations in
simulation. This is because of the physical dynamics differences and the
imprecision of the F/T sensing in simulation. For this reason, we retrained
the policy by adding Ornstein-Uhlenbeck(OU) noise in state observations. OU
noise is a type of correlated noise often used in RL for continuous control
tasks which helps the agent explore more systematically than purely random
noise. The OU process is defined by a differential equation and has a mean-
reverting property, which can be useful for stabilizing exploration. The
success rate increased to 6/6 after retraining. Fig. 7 shows the deployment on
a physical robot. We put bolts of different size as dynamic obstacles (50
M6$\times$30 for small obstacles, 100 M6$\times$8 and 50 M6$\times$30 for
mixed obstacles). A video of the experiments is available at
https://youtu.be/ksWXR3vByoQ.

---
Figure 9: Force ($F_{y}$), move size ($\Delta P_{x}$), and stiffness ($K_{y}$)
values during the real-robot experiments plotted against the y- position. The
pink highlight shows the approximate placement of obstacles.

#### Results

The SRL-VIC policy transfers successfully from simulation to real world. The
transfer success was increased by the addition of state randomization in the
simulation training. Apart from the physical dynamics and sensing differences
between the simulation and the real world, there were also task-level
differences, namely, the shape of the maze and the character of the obstacles.

In the real-world experiments, we tested the generalization capability of our
safety model by changing the position of the maze curves and placing wall
segments at new positions as shown in Fig. 7. The maze length varied between
simulation (70.35 cm) and the real-world (84.62 cm). Additionally, the nature
of the obstacles differed from those present in the simulation. The learned
policy successfully completed most of these cases, suggesting that the safety
mechanism generalizes to different wall positions. The agent can also complete
the Maze shape-1 when tested by changing the flange diameter (1, 2, 3 and 4
cm). Our task policy is trained on a single maze shape; thus, it could not
generalize to the case shown in Fig. 7.h. It still did not violate a
constraint, but it could not find the exit. Dealing with more diverse shapes
of mazes requires training with more diverse shapes as well, however, we do
not focus on task generalization in this work.

The end-effector trajectories in real-robot experiments are shown in Fig. 8.
The policy adapts the stiffness values effectively by becoming stiff in the
goal direction and staying compliant laterally. It also employs distinct
strategies for encounters with walls at curves and collisions with movable
obstacles. For the latter, it adopts a larger, rounded stiffness ellipse to
facilitate pushing through in all directions.

We also plot the measured force ($F_{y}$), stiffness $K_{y}$ and move size
($\Delta P_{x}$) values w.r.t. y-position in Fig. 9, as y is the main axis
towards the goal. The stiffness adaptation policy becomes more evident in
these plots, comparing the no obstacle case to the others. The $2^{nd}$ and
$3^{rd}$ cases have larger stiffness in the obstacle area as they observe
higher force, while the $1^{st}$ case increases stiffness only at the curves,
while touching the wall. Another difference is seen in the $\Delta P_{x}$
action, as the $2^{nd}$ and $3^{rd}$ cases move laterally to open the way
through the obstacles, while the $1^{st}$ maintains its x-position.

##  V Conclusion and future work

We presented a framework that integrates the strengths of a safe RL method
with VIC. This combination addresses the dual challenges of ensuring safety
and maintaining continuous physical contact. Experiments in contact-rich maze
tasks demonstrate that our framework achieved a good trade-off between task
accomplishment and collision-based safety, and outperformed Std_RL-VIC. The
inclusion of the VIC adds increased adaptability, creating a better trade-off
between safety and success in comparison to the constant impedance approaches.
The trained model can transfer to real-world, and also the safety model can
generalize to different obstacles and wall positions. In the future, we will
evaluate our framework in more generalized task scenarios and introduce model-
based RL into our framework to improve the safety and robustness of our
proposed framework in unknown environments.

## References

  * [1] A. Ajoudani, A. M. Zanchettin, S. Ivaldi, A. Albu-Schäffer, K. Kosuge, and O. Khatib, “Progress and prospects of the human-robot collaboration,” _Autonomous Robots_ , vol. 42, no. 5, pp. 957–975, 2018.
  * [2] F. J. Abu-Dakka and M. Saveriano, “Variable impedance control and learning—a review,” _Frontiers in Robotics and AI_ , vol. 7, p. 590681, 2020.
  * [3] Y. Kato, P. Balatti, J. M. Gandarias, M. Leonori, T. Tsuji, and A. Ajoudani, “A self-tuning impedance-based interaction planner for robotic haptic exploration,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 9461–9468, 2022.
  * [4] C. Chang, K. Haninger, Y. Shi, C. Yuan, Z. Chen, and J. Zhang, “Impedance adaptation by reinforcement learning with contact dynamic movement primitives,” in _2022 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM)_. IEEE, 2022, pp. 1185–1191.
  * [5] X. Zhang, L. Sun, Z. Kuang, and M. Tomizuka, “Learning variable impedance control via inverse reinforcement learning for force-related tasks,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 2225–2232, 2021.
  * [6] Q. Yang, A. Dürr, E. A. Topp, J. A. Stork, and T. Stoyanov, “Variable impedance skill learning for contact-rich manipulation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 8391–8398, 2022.
  * [7] R. Martin-Martin, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg, “Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 1010–1017.
  * [8] B. Thananjeyan, A. Balakrishna, S. Nair, M. Luo, K. Srinivasan, M. Hwang, J. E. Gonzalez, J. Ibarz, C. Finn, and K. Goldberg, “Recovery rl: Safe reinforcement learning with learned recovery zones,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4915–4922, 2021.
  * [9] S. Kim, J. Kwon, T. Lee, Y. Park, and J. Perez, “Safety-aware unsupervised skill discovery,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 894–900.
  * [10] H. Qiao, B. Dalay, and R. Parkin, “Robotic peg-hole insertion operations using a six-component force sensor,” _Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science_ , vol. 207, no. 5, pp. 289–306, 1993.
  * [11] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,” _IEEE Transactions on Robotics_ , vol. 36, no. 3, pp. 582–596, 2020.
  * [12] G. Schoettler, A. Nair, J. A. Ojea, S. Levine, and E. Solowjow, “Meta-reinforcement learning for robotic industrial insertion tasks,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 9728–9735.
  * [13] C. Yang, P. Liang, Z. Li, A. Ajoudani, C.-Y. Su, and A. Bicchi, “Teaching by demonstration on dual-arm robot using variable stiffness transferring,” in _2015 IEEE International Conference on Robotics and Biomimetics (ROBIO)_ , 2015, pp. 1202–1208.
  * [14] J. Lee, A. Ajoudani, E. M. Hoffman, A. Rocchi, A. Settimi, M. Ferrati, A. Bicchi, N. G. Tsagarakis, and D. G. Caldwell, “Upper-body impedance control with variable stiffness for a door opening task,” in _2014 IEEE-RAS International Conference on Humanoid Robots_ , 2014, pp. 713–719.
  * [15] D. Mitrovic, S. Klanke, and S. Vijayakumar, “Learning impedance control of antagonistic systems based on stochastic optimization principles,” _The International Journal of Robotics Research_ , vol. 30, no. 5, pp. 556–573, 2011.
  * [16] Y. Li, G. Ganesh, N. Jarrassé, S. Haddadin, A. Albu-Schaeffer, and E. Burdet, “Force, impedance, and trajectory learning for contact tooling and haptic identification,” _IEEE Transactions on Robotics_ , vol. 34, no. 5, pp. 1170–1182, 2018.
  * [17] C. Yang, C. Zeng, C. Fang, W. He, and Z. Li, “A dmps-based framework for robot learning and generalization of humanlike variable impedance skills,” _IEEE/ASME Transactions on Mechatronics_ , vol. 23, no. 3, pp. 1193–1203, 2018.
  * [18] J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. Sukhatme, J. Lim, and P. Englert, “Motion planner augmented reinforcement learning for robot manipulation in obstructed environments,” in _Conference on Robot Learning_. PMLR, 2021, pp. 589–603.
  * [19] H. Bharadhwaj, A. Kumar, N. Rhinehart, S. Levine, F. Shkurti, and A. Garg, “Conservative safety critics for exploration,” _arXiv preprint arXiv:2010.14497_ , 2020.
  * [20] E. Altman, _Constrained Markov decision processes_. Routledge, 1995.
  * [21] K. Srinivasan, B. Eysenbach, S. Ha, J. Tan, and C. Finn, “Learning to be safe: Deep rl with a safety critic,” _arXiv preprint arXiv:2010.14603_ , 2020.
  * [22] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [23] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in _International conference on machine learning_. PMLR, 2018, pp. 1861–1870.
  * [24] L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, “Safe learning in robotics: From learning-based control to safe reinforcement learning,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 5, pp. 411–444, 2022.


# Design and Experimental Investigation of Trikarenos:
A Fault-Tolerant 28nm RISC-V-based SoC

Michael Rogenmoser, Philip Wiese, Bruno Endres Forlin, Frank K. Gürkaynak,
Paolo Rech, Alessandra Menicucci, Marco Ottavi, Luca Benini

35-words abstract: We present a fault-tolerant by-design RISC-V SoC and
experimentally assess it under atmospheric neutrons and
$200\text{\,}\mathrm{MeV}$ protons. The dedicated ECC and Triple-Core Lockstep
countermeasures correct most errors, guaranteeing a device cross-section lower
than $5.36\text{\times}{10}^{-12}\text{\,}{\mathrm{cm}}^{2}$.

Corresponding and Presenting Author: Michael Rogenmoser; Email:
<EMAIL_ADDRESS>Telephone: +41 44 632 54 33; Address: ETZ J78,
Gloriastrasse 35, 8092 Zürich, Switzerland;

Contributing Authors:

Michael Rogenmoser, Philip Wiese, and Frank K. Gürkaynak are with the
Integrated Systems Laboratory at ETH Zurich, Switzerland

Bruno Endres Forlin is with the Computer Architecture for Embedded Systems of
the University of Twente, the Netherlands

Paolo Rech is with the Department of Industrial Engineering of the University
of Trento, Italy

Alessandra Menicucci is with the Department of Space Engineering at the Delft
University of Technology, the Netherlands

Marco Ottavi is with the Computer Architecture for Embedded Systems of the
University of Twente, the Netherlands and the Department of Electronic
Engineering at the University of Rome, Tor Vergata, Italy

Luca Benini is with the Integrated Systems Laboratory at ETH Zurich,
Switzerland and the Department of Electrical, Electronic, and Information
Engineering at the University of Bologna, Italy

Session Preference: F - Radiation Hardening Techniques

Presentation Preference: Oral

##  I Introduction

In the automotive and space domains, the reliability of semiconductor chips
and systems-on-chip (SoCs) is critical. The space sector faces challenges due
to high radiation levels causing single event effects (SEEs) , which can
disrupt the normal functioning of electronic components. For the automotive
industry, on the other hand, reliability is a major intrinsic concern since
atmospheric neutrons can cause single event upsets (SEUs) , leading to crashes
or unexpected behaviors. Various strategies are employed to mitigate these
risks. At the technology level, radiation hardened by design (RHBD)
technologies or specialized cells can enhance resilience. Alternatively,
architectural solutions such as dual modular redundancy (DMR) , triple modular
redundancy (TMR) , or error correction codes (ECC) can be implemented to mask
faults and ensure a safe operation in critical applications.

The RISC-V architecture is rapidly gaining traction in safety-critical
applications, thanks to the openness of the instruction set architecture (ISA)
and the availability of open processor IPs [1]. As a key advantage, the RISC-V
flexibility allows customization and improvement of the ISA and the underlying
hardware, tuning them to meet specific (reliability) requirements.
Consequently, the RISC-V ecosystem is being increasingly adopted in
applications where system reliability is non-negotiable [2].

Within this context, our paper focuses on the experimental evaluation of
Trikarenos [3], a SoC designed to address the reliability requirements of both
automotive and space applications. Trikarenos leverages the flexibility of the
RISC-V architecture, integrating a triple-core lockstep (TCLS) system
alongside ECC and scrubbing in memory. This work investigates the hardening
by-design features of Trikarenos under neutron and proton radiation,
evaluating its resilience for automotive and space contexts. Specifically, our
contributions include:

  * •

The measurement of static random-access memory (SRAM) vulnerability within
Trikarenos, focusing on the effectiveness of ECC protection and scrubbing
techniques.

  * •

An evaluation of the TCLS mechanism, demonstrating its capacity to maintain
the functionality of the processor in the event of a fault.

  * •

A performance assessment of Trikarenos in various radiation environments,
including an atmospheric setting dominated by neutrons and an orbital
environment characterized by protons.

##  II SoC Architecture Description

Figure 1: Trikarenos SoC architectural block diagram (left) and annotated die
shot (right).

The Trikarenos architecture, presented in [3], implements an adaptation of the
PULPissimo, a RISC-V-based microcontroller design. As shown in Figure 1,
Trikarenos includes 3 physically separated Ibex cores connected to a low
latency interconnect exposing $256\text{\,}\mathrm{KiB}$ of SRAM memory.
Furthermore, the SoC features a boot memory, timers, and peripherals such as
general purpose input/outputs (GPIOs) , UART and QSPI with direct memory
access (DMA) , and a JTAG debug unit for programming.

To enable tolerance to SEUs at an architectural level, we augment Trikarenos’
processing cores with on-demand redundancy grouping (ODRG) [4], a configurable
TCLS mechanism. While the system allows the cores to operate independently for
increased performance, this investigation focuses on the locked reliability
mode, allowing for the detection and correction of any mismatch in the cores’
operation. The SRAM memory, occupying the majority of the design area, is
protected with a single error correction, double error detection (SECDED)
Hsiao code, storing 39 bits for each 32-bit word, effectively containing
$2\,555\,904\text{\,}\mathrm{b}\mathrm{i}\mathrm{t}\mathrm{s}$ for the
$256\text{\,}\mathrm{kB}$ memory. To ensure minimal performance impact and
avoid error accumulation, the mechanism supports an efficient read-modify-
write architecture for sub-word writes and features a configurable scrubber,
continuously scanning the memory banks and correcting any detected single
bitflip.

To track the SoC ’s status, the architecture features a variety of telemetry
registers counting errors in the system. This allows efficient readout of
errors in the memory banks detected on access and detected by the scrubber,
both for correctable and uncorrectable faults.

Trikarenos is implemented in TSMC 28HPC+, a 28nm bulk process that has been
shown to be tolerant to total ionizing dose (TID) effects [5], targeting
operating frequencies up to $250\text{\,}\mathrm{MHz}$ (at
$0.9\text{\,}\mathrm{V}$), occupying $0.7\text{\,}{\mathrm{mm}}^{2}$ logic
area in the $2\text{\,}{\mathrm{mm}}^{2}$ design shown in Figure 1. In the
experimental setup, Trikarenos operates at $125\text{\,}\mathrm{MHz}$ and
$0.9\text{\,}\mathrm{V}$.

##  III Evaluation Methodology

###  III-A Testing Facilities

We conducted experiments at the ChipIR and HollandPTC facilities to assess the
sensitivity of Trikarenos to neutron and proton radiation. The ChipIR
beamline, located at the Rutherford Appleton Laboratories in the United
Kingdom, is designed to replicate the energy spectrum of atmospheric neutrons
[6]. On the other hand, HollandPTC, located in Delft, The Netherlands, is
equipped with a fixed horizontal proton beamline, delivering a beam with
nominal energies ranging from $70\text{\,}\mathrm{MeV}$ up to
$240\text{\,}\mathrm{MeV}$, and its Cyclotron can deliver nominal beam
currents from $1\text{\,}\mathrm{nA}$ up to $800\text{\,}\mathrm{nA}$ and a
maximum flux of
$1.13\text{\times}{10}^{09}\text{\,}\mathrm{p}\text{\,}{\mathrm{cm}}^{-2}\text{\,}{\mathrm{s}}^{-1}$
[7].

###  III-B Error model

Our experiments are focused on analyzing SEEs in the RISC-V-based Trikarenos
SoC . For the TSMC 28nm technology, other research has already investigated
the TID effects [5], showing extreme tolerance levels.

We investigate the error mechanisms listed in Table I, which are traceable
with the SoC telemetry registers and given test applications.

TABLE I: Observable error types and classification Observed Error | Class
---|---
Single SRAM errors (scrubber) | Correctable
Multiple SRAM errors (on access & scrubber) | Functional Error
TCLS errors | Correctable
Miscalculations (silent data corruption (SDC) ) | Functional Error
System crashes | Functional Error
Single event latchup (SEL) | Total Loss

While most error sources can be traced back to SEUs or multiple/accumulation
thereof, system crashes can have different causes. They can be traced back to
a single event functional interrupt (SEFI) or a SEU affecting an unprotected,
critical component, such as the system’s interconnect. While some GPIO signals
and routines are designed to assist in differentiating these faults, exact
cause identification remains difficult.

Finally, most data corruption will be detected by one of the two mitigation
mechanisms. ECC protection in the SRAM memory will catch an error affecting
stored data, and the TCLS mechanism will detect data corruption in the cores.
For a particle strike in the cores, it is more than likely that only a single
core will be affected due to their separation, thus resulting in different
outputs from the cores and an error in the voter while storing the result back
in memory.

###  III-C Experimental Setup

Figure 2: Annotated image of testing setup at HollandPTC (top) and schematic
of the testing setup (bottom), showing the proton beam, the device under test
(DUT), and the test harness.

The test setup, shown in Figure 2, comprised a dedicated testing printed
circuit board (PCB) containing the device under test (DUT) and a test harness,
a monitoring device with an interface board. A Raspberry Pi (RPi) single-board
computer served as the primary control and monitoring system for the entire
test suite. The interface board supplied the necessary
$1.8\text{\,}\mathrm{V}$ I/O and $0.9\text{\,}\mathrm{V}$ core voltages for
the Trikarenos SoC , in addition to incorporating a level shifter, dual clock
generation circuits, and several status LEDs for real-time feedback and
diagnostics. This setup controlled the power supply to the DUT , managed
binary loading into the SRAM via JTAG, and monitored GPIOs pins while also
interfacing with the DUT via JTAG and UART for data acquisition.

The test application running on the DUT was carefully designed to emulate a
realistic application and maximize the propagation probability and
observability of bitflips inside the core complex and memory. To emulate a
realistic application, Trikarenos executed the Coremark benchmark,
implementing a variety of workloads to test a processor core. The final
version used for the proton tests augmented this with a section continuously
executing simple not operations on the processor core’s registers filled with
a checkerboard pattern and periodically saving these values to memory. Saving
the register values makes use of the TCLS voters to detect any errors. This
maximized the exposure time of the vulnerable components, allowing a possible
mismatch between the cores in TCLS to trigger an error.

For testing, the DUT application facilitated real-time error monitoring by
periodically transmitting the status of memory-mapped registers counting
errors through the UART and JTAG interfaces. The test harness acquired this
transmitted data, additionally monitoring other GPIO signals designed to
indicate other system failures. Finally, in the event of critical, non-
recoverable errors, the monitoring system would halt the cores and dump all
relevant registers and memory areas for subsequent analysis. This
comprehensive approach, combined with multiple redundant recovery mechanisms
and extensive logging strategies, equipped the setup for effective long-term
and unsupervised radiation testing.

Data integrity and logging were critical concerns addressed in our setup. All
test data was logged locally on NAND flash memory using an SD card. To ensure
a proper backup in case of SD card corruption, the data, along with RPi ’s
system logs to assist in tracing errors, were transmitted to a computer
outside the beam room over ethernet for later processing. To ensure the
detection of latch-up events, a current monitor and a current-limited power
switch were added to the test harness to prevent permanent damage to the DUT .

##  IV Experimental Results

Throughout all our tests, we did not observe any uncorrectable SRAM errors,
miscalculations, nor SELs indicated by a change in power consumption.
Therefore, the results below only differentiate correctable SRAM errors,
correctable TCLS errors, and uncorrectable errors resulting in a system crash.
Because some errors are rare or possibly even zero, we calculate upper and
lower estimates of the cross-section assuming a Poisson distribution and use
the relationship between the cumulative distribution function of the Possion
and chi-squared distribution [8]. To calculate these estimates $\sigma$ we use

$\frac{1}{2F}\chi^{2}\left(\frac{\alpha}{2},2N\right)<\sigma<\frac{1}{2F}\chi^{2}\left(1-\frac{\alpha}{2},2(N+1)\right)$
(1)

where $\chi^{2}(p,n)$ is the quantile function of the chi-squared distribution
with $n$ degrees of freedom, $\alpha$ describes the $100(1-\alpha)$ percent
confidence interval, $F$ is the total exposed fluence, and $N$ is the number
of detected errors. All calculations below use a $95\text{\,}\mathrm{\char
37\relax}$ confidence interval.

To differentiate between errors attributed to the test setup and those
inherent to the DUT , we conducted a detailed analysis of the system logs from
the RPi . Our evaluation identified that most errors originating from the
setup involved SD card-related timeouts and kernel segmentation faults.

Figure 3: Visualization of induced errors and flux rates during both radiation
tests of Trikarenos. We show the cross-sections, flux rates, TCLS errors, and
system crashes over time. We use an atmospheric-like neutron beam [6] and
$200\text{\,}\mathrm{MeV}$ protons. For the proton tests, no TID-related
degradation was observed.

###  IV-A Neutron Tests

TABLE II: Cross-section results of Trikarenos under neutron beam for a fluence
of
$6.88\text{\times}{10}^{11}\text{\,}\mathrm{n}\mathrm{e}\mathrm{u}\mathrm{t}\mathrm{r}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{2}}$

Component | Error Count | Errors / $\mathrm{bit}$ | Cross-Section†
---|---|---|---
Single SRAM errors | 18786 | $7.35\text{\times}{10}^{-3}\text{\,}$ | $2.77\text{\times}{10}^{-08}\text{\,}{\mathrm{cm}}^{2}$
TCLS errors | 13 | - | $3.23\text{\times}{10}^{-11}\text{\,}{\mathrm{cm}}^{2}$
System crashes | 0 | - | $5.36\text{\times}{10}^{-12}\text{\,}{\mathrm{cm}}^{2}$

  * †

Upper-bound with a $95\text{\,}\mathrm{\char 37\relax}$ confidence interval &
Poisson distribution. This assumes one additional fault, i.e., device failure
after the tests.

At the ChipIR facility [6], Trikarenos was exposed to a total fluence of
$6.88\text{\times}{10}^{11}\text{\,}\mathrm{n}\text{\,}{\mathrm{cm}}^{-2}$.
While active, the beam produced an average neutron flux of
$4.90\text{\times}{10}^{06}\text{\,}\mathrm{n}\text{\,}{\mathrm{cm}}^{-2}\text{\,}{\mathrm{s}}^{-1}$
with an energy spectrum representing that found for atmospheric neutrons at
ground level up to $800\text{\,}\mathrm{MeV}$. The measurements are shown in
Figure 3, and the errors and estimated cross-sections are summarized in Table
II.

####  IV-A1 Memory Errors

During the neutron exposure, we observed a total of
$18\,786\text{\,}\mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}\mathrm{s}$
for a median error rate of
$492\text{\,}\mathrm{error}\text{\,}{\mathrm{h}}^{-1}$. All of these errors
were single errors, detected and corrected by the scrubber checking the
hardware ECC encoding. Using Equation 1 allows us to estimate the cross-
section of the memory to $2.75(0.02)e-08\text{\,}{\mathrm{cm}}^{2}$. With
$2\,555\,904\text{\,}\mathrm{bit}$s, we measured an average bit error rate of
$1.92\text{\times}{10}^{-04}\text{\,}\mathrm{error}\text{\,}{\mathrm{bit}}^{-1}\text{\,}{\mathrm{h}}^{-1}$
for a cross-section per bit of
$1.08(0.01)e-14\text{\,}{\mathrm{cm}}^{2}\text{\,}{\mathrm{bit}}^{-1}$. This
rate is around $11\text{\,}\mathrm{\char 37\relax}$ lower compared to the
findings in [9], which evaluated the $14\text{\,}\mathrm{MeV}$ neutron
sensitivity of an SRAM manufactured in the same technology.

####  IV-A2 Core Errors

During the experiment, we measured $13\text{\,}$ correctable TCLS errors. We
estimate the cross-section of recoverable TCLS events to be
$2.55(0.68)e-11\text{\,}{\mathrm{cm}}^{2}$. No unrecoverable faults were
measured. Thus, we can only estimate the upper-bound cross-section of the
unrecoverable events to
$5.36\text{\times}{10}^{-12}\text{\,}{\mathrm{cm}}^{2}$.

To contextualize, we consider a terrestrial environment to provide estimations
for mean time to failure (MTTF) . Assuming an integral flux density of
$20\text{\,}\mathrm{n}\text{\,}{\mathrm{cm}}^{-2}\text{\,}{\mathrm{h}}^{-1}$
above $1\text{\,}\mathrm{MeV}$ corresponding to the ground radiation level in
New York [10], the minimum MTTF for a terrestrial application with an active
TCLS recovery mechanism is above $1.06\text{\,}$ million years.

###  IV-B Proton Tests

TABLE III: Cross-section results of Trikarenos under proton beam for a fluence
of
$2.91\text{\times}{10}^{12}\text{\,}\mathrm{p}\mathrm{r}\mathrm{o}\mathrm{t}\mathrm{o}\mathrm{n}\mathrm{s}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{2}}$

Component | Error count | Errors / $\mathrm{bit}$ | Cross-Section†
---|---|---|---
Single SRAM error | 8249 | $3.23\text{\times}{10}^{-3}\text{\,}$ | $2.89\text{\times}{10}^{-09}\text{\,}{\mathrm{cm}}^{2}$
TCLS errors | 11 | - | $6.76\text{\times}{10}^{-12}\text{\,}{\mathrm{cm}}^{2}$
System crashes | 1 | - | $1.91\text{\times}{10}^{-12}\text{\,}{\mathrm{cm}}^{2}$

  * †

Upper-bound with a $95\text{\,}\mathrm{\char 37\relax}$ confidence interval &
Poisson distribution. This assumes one additional fault, i.e., device failure
after the tests.

Trikarenos was exposed to a total fluence of
$2.91\text{\times}{10}^{12}\text{\,}\mathrm{p}\text{\,}{\mathrm{cm}}^{-2}$
over $55\text{\,}\mathrm{min}$ of active beam time, a notably high value. This
value is a direct result of our stress testing of the system with the highest
flux available in the facility. We tested a range of beam currents for
$200\text{\,}\mathrm{MeV}$ proton with the flux rate extracted from
calibration data provided by the facility operator. Figure 3 displays the
measurements, and Table III presents the observed errors alongside estimated
cross-sections.

####  IV-B1 Memory Errors

In total, we measured
$8249\text{\,}\mathrm{e}\mathrm{r}\mathrm{r}\mathrm{o}\mathrm{r}\mathrm{s}$
for a median error rate of
$9180\text{\,}\mathrm{error}\text{\,}{\mathrm{h}}^{-1}$, all of which were
corrected by the scrubbers, leading to SRAM cross-section
$2.86(0.03)e-09\text{\,}{\mathrm{cm}}^{2}$. This results in a cross-section
per bit of
$1.12(0.01)e-15\text{\,}{\mathrm{cm}}^{2}\text{\,}{\mathrm{bit}}^{-1}$.

####  IV-B2 Core Errors

During the experiment, we measured $11\text{\,}$ recoverable and $1\text{\,}$
unrecoverable errors. Our data indicates that an unresponsive debug module
caused the unrecoverable error, an element of the SoC currently not protected.
We estimate the cross-section of recoverable TCLS events to
$5.25(1.51)e-12\text{\,}{\mathrm{cm}}^{2}$ and the upper bound for
unrecoverable faults to
$1.91\text{\times}{10}^{-12}\text{\,}{\mathrm{cm}}^{2}$. Using this, we can
calculate the minimum mean fluence to failure (MFTF) of the system with the
TCLS recovery mechanism to
$5.23\text{\times}{10}^{11}\text{\,}\mathrm{p}\text{\,}{\mathrm{cm}}^{-2}$.

####  IV-B3 Other

Throughout the measurements with protons, Trikarenos received a total ionizing
dose of more than $1.77\text{\times}{10}^{3}\text{\,}\mathrm{Gy}$. No
degradation in performance or TID-related errors were observed. Furthermore,
the observed current consumption remained constant within noise-induced
bounds, indicating no SELs.

###  IV-C Discussion

####  IV-C1 Memory Errors

While the application running may have an impact on the TCLS and total fault
rate, the SRAM fault rate checked by the scrubber should not be affected.
Comparing neutron and proton fault rates, we observe a $>9\times$ higher
cross-section of the SRAM for neutrons. A precise cause for this requires
further investigation but could potentially be attributed to variations in
particle types, energy levels, and differences in the devices used across
tests. However, this provides an initial foundation for the validity of
neutron beam experiments to evaluate the radiation tolerance of new devices,
even in other domains.

With SRAM cross-sections of
$1.08(0.01)e-14\text{\,}{\mathrm{cm}}^{2}\text{\,}{\mathrm{bit}}^{-1}$ for
neutrons and
$1.12(0.01)e-15\text{\,}{\mathrm{cm}}^{2}\text{\,}{\mathrm{bit}}^{-1}$ for
protons, we can appropriately tune the scrub rate of the device. Trikarenos’
scrubber was set to the maximum possible rate for the experiments to ensure
all errors were caught without accumulation. This can be reduced to limit
power consumption. With individual scrubbers for each of the 8 memory banks,
each scrubber is responsible for 8192 words. To ensure the entire memory space
is checked between the mean failure rate, the scrubber needs to complete its
memory check every $408\text{\,}\mathrm{ms}$ for the proton test condition,
leading to a word check per bank every $6225\text{\,}$ cycles.

####  IV-C2 Core Errors

In the tested operation mode, Trikarenos detects all corruptions within the
cores, both those that may lead to system corruption due to an erroneous
branch, corrupted pointer or misdecoded instruction, and data corruption.

Operating in Trikarenos’ independent mode, a performance improvement of up to
$2.96\times$ with three cores in parallel or a 35% reduction in core power
consumption with a single core is possible. In this independent mode, however,
erroneous control flow may lead to stalls that can be caught by a watchdog
timer, or SDC can lead to erroneous data. Assuming all faults in a core lead
to a system crash, the device cross-section in parallel operation increases by
at least $3.5\times$ from $2.55(0.68)e-11\text{\,}{\mathrm{cm}}^{2}$ for
neutrons and by at least $2.14\times$ from
$5.64(1.56)e-12\text{\,}{\mathrm{cm}}^{2}$ for protons. For single-core
operation, assuming an equal distribution of errors in the cores, the cross-
section improvement is at least $1.12\times$ from
$1.08(0.48)e-11\text{\,}{\mathrm{cm}}^{2}$ for neutrons. Further measurements
can help narrow down the improvement, likely showing a significant increase in
the reliability factor. Without SRAM protection, the improvement is far more
significant, even considering that only a subsection of the memory is used and
that some data corruption may not cause a system crash.

The TCLS recovery routine, requiring around 600 cycles or
$4.8\text{\,}\mathrm{\SIUnitSymbolMicro s}$ at $125\text{\,}\mathrm{MHz}$, has
a negligible impact on performance for the observed error rates.

####  IV-C3 Other

Both the neutron and proton experiments also revealed vulnerabilities in the
telemetry registers, showing flips of specific bits reflecting immediate,
immense increases in measured error counts. While not critical for performance
or operations, such registers or similar SoC control registers may be worth
protecting in future designs.

Finally, while unrecoverable errors were very rare in our investigations, it
was shown that they are still possible, requiring watchdog circuitry or
further protection in future designs. For the Trikarenos SoC , their root
cause analysis requires further investigation due to their rarity.

##  V Conclusion

Exposing the Trikarenos SoC to neutrons and protons, we observe most errors
affecting the SRAM for a cross-section per bit of
$1.08(0.01)e-14\text{\,}{\mathrm{cm}}^{2}\text{\,}{\mathrm{bit}}^{-1}$ and
$1.12(0.01)e-15\text{\,}{\mathrm{cm}}^{2}\text{\,}{\mathrm{bit}}^{-1}$ for
atmospheric neutrons and $200\text{\,}\mathrm{MeV}$ protons, respectively,
protected by ECC, thus, not affecting the system’s function. Trikarenos’ TCLS
mechanism corrects most of the remaining errors, reducing the cross-section by
at least $3.5\times$ from $2.55(0.68)e-11\text{\,}{\mathrm{cm}}^{2}$ to below
$5.36\text{\times}{10}^{-12}\text{\,}{\mathrm{cm}}^{2}$, showing TCLS ’
efficacy and Trikarenos’ validity for use in critical environments.

## References

  * [1] Stefano Di Mascio et al.  “Open-source IP cores for space: A processor-level perspective on soft errors in the RISC-V era”  In _Computer Science Review_ 39, 2021, pp. 100349  DOI: 10.1016/j.cosrev.2020.100349
  * [2] A. Walsemann et al.  “STRV — a radiation hard RISC-V microprocessor for high-energy physics applications”  In _Journal of Instrumentation_ 18.02, 2023, pp. C02032  DOI: 10.1088/1748-0221/18/02/C02032
  * [3] Michael Rogenmoser et al.  “Trikarenos: A Fault-Tolerant RISC-V-based Microcontroller for CubeSats in 28nm”  In _IEEE International Conference on Electronics, Circuits and Systems (ICECS)_ , 2023, pp. 1–4  DOI: 10.1109/ICECS58634.2023.10382727
  * [4] Michael Rogenmoser et al.  “Hybrid Modular Redundancy: Exploring Modular Redundancy Approaches in RISC-V Multi-Core Computing Clusters for Reliable Processing in Space” Just Accepted  In _ACM Transactions on Cyber-Physical Systems_ , 2023  DOI: 10.1145/3635161
  * [5] G. Borghello et al.  “Total ionizing dose effects on ring-oscillators and SRAMs in a commercial 28 nm CMOS technology”  In _Journal of Instrumentation_ 18.02, 2023, pp. C02003  DOI: 10.1088/1748-0221/18/02/C02003
  * [6] Carlo Cazzaniga et al.  “Progress of the Scientific Commissioning of a fast neutron beamline for Chip Irradiation”  In _Journal of Physics: Conference Series_ 1021, 2018, pp. 012037  DOI: 10.1088/1742-6596/1021/1/012037
  * [7] M. Rovituso et al.  “Characterisation of the HollandPTC R&D proton beamline for physics and radiobiology studies”  In _arXiv.org_ , 2023  URL: https://arxiv.org/abs/2302.09943v1
  * [8] Norman L. Johnson et al.  “Poisson Distribution”  In _Univariate Discrete Distributions_ John Wiley & Sons, Ltd, 2005, pp. 176  DOI: 10.1002/0471715816.ch4
  * [9] Vanessa Vargas et al.  “Radiation Experiments on a 28 nm Single-Chip Many-Core Processor and SEU Error-Rate Prediction” Conference Name: IEEE Transactions on Nuclear Science  In _IEEE Transactions on Nuclear Science_ 64.1, 2017, pp. 483–490  DOI: 10.1109/TNS.2016.2638081
  * [10] M.S. Gordon et al.  “Measurement of the flux and energy spectrum of cosmic-ray induced neutrons on the ground”  In _IEEE Transactions on Nuclear Science_ 51.6, 2004, pp. 3427–3434  DOI: 10.1109/TNS.2004.839134