# Enhancing Hardware Fault Tolerance in Machines with Reinforcement Learning
Policy Gradient Algorithms

Sheila Schoepp<EMAIL_ADDRESS>Mehran Taghian<EMAIL_ADDRESS>Shotaro Miwa<EMAIL_ADDRESS>Yoshihiro Mitsuka
<EMAIL_ADDRESS>Shadan Golestan
<EMAIL_ADDRESS>Osmar R. Zaïane<EMAIL_ADDRESS>
###### Abstract

Industry is rapidly moving towards fully autonomous and interconnected systems
that can detect and adapt to changing conditions, including machine hardware
faults. Traditional methods for adding hardware fault tolerance to machines
involve duplicating components and algorithmically reconfiguring a machine’s
processes when a fault occurs. However, the growing interest in reinforcement
learning-based robotic control offers a new perspective on achieving hardware
fault tolerance. However, limited research has explored the potential of these
approaches for hardware fault tolerance in machines. This paper investigates
the potential of two state-of-the-art reinforcement learning algorithms,
Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), to enhance
hardware fault tolerance into machines. We assess the performance of these
algorithms in two OpenAI Gym simulated environments, Ant-v2 and FetchReach-v1.
Robot models in these environments are subjected to six simulated hardware
faults. Additionally, we conduct an ablation study to determine the optimal
method for transferring an agent’s knowledge, acquired through learning in a
normal (pre-fault) environment, to a (post-)fault environment in a continual
learning setting. Our results demonstrate that reinforcement learning-based
approaches can enhance hardware fault tolerance in simulated machines, with
adaptation occurring within minutes. Specifically, PPO exhibits the fastest
adaptation when retaining the knowledge within its models, while SAC performs
best when discarding all acquired knowledge. Overall, this study highlights
the potential of reinforcement learning-based approaches, such as PPO and SAC,
for hardware fault tolerance in machines. These findings pave the way for the
development of robust and adaptive machines capable of effectively operating
in real-world scenarios.

###### keywords:

Hardware faults , Fault tolerance , Reinforcement learning , Policy gradient
algorithms , Real-time fault recovery

††journal: Engineering Applications of Artificial Intelligence

[a]organization=Department of Computing Science, University of
Alberta,addressline=8900 114 St NW, city=Edmonton, state=Alberta, postcode=T6G
2S4, country=Canada

[b]organization=Alberta Machine Intelligence Institute,addressline=10065
Jasper Ave #1101, city=Edmonton, state=Alberta, postcode=T5J 3B1,
country=Canada

[c]organization=Advanced Technology R&D Center, Mitsubishi Electric
Corporation,addressline=8-1-1,Tsukaguchi-honmachi, city=Amagasaki-shi,
state=Hyogo, postcode=661-8661, country=Japan

[d]organization=Information Technology R&D Center, Mitsubishi Electric
Corporation,addressline=5-1-1, Ofuna, city=Kamakura-shi, state=Kanagawa,
postcode=247-8501, country=Japan

##  1 Introduction

Automation is revolutionizing industries, enhancing productivity and
efficiency (Haleem et al., 2021; Javaid et al., 2021; Vaidya et al., 2018).
Automation is achieved by integrating machines equipped with sensors, such as
robots, with Artificial Intelligence (AI)-powered analytical systems (agents),
enabling real-time data collection and analysis. These agents make the
critical decisions necessary for automation, including facilitating the real-
time identification and diagnosing of hardware faults in machines through
fault detection and fault diagnosis, respectively (Riazi et al., 2019; Zhang
et al., 2017; Samanta and Nataraj, 2009). While fault detection and fault
diagnosis are essential processes that inform on faults, they do not address
adaptation to hardware faults. To succeed in real-world deployments, it is
crucial for machines to adapt to unexpected events, such as hardware faults
(Chen et al., 2023; Schrum and Gombolay, 2019). As such, enhancing hardware
fault tolerance in machines is imperative.

An established method of enhancing hardware fault tolerance in machines is
through redundancy, where critical hardware components are duplicated to
mitigate the risk of failure (Guiochet et al., 2017; Visinsky et al., 1994,
1991). However, this approach has significant drawbacks, including increased
machine size, weight, power consumption, and financial costs (Dubrova, 2013).
Moreover, retrofitting existing machines with redundant components is often
impossible. Therefore, exploring alternative approaches that do not rely on
redundancy provides a significant advantage in enhancing hardware fault
tolerance in machines.

Taking inspiration from nature, one can imagine an agent adapting a machine’s
behaviour in response to a hardware fault. Animals, for example, demonstrate
an extraordinary ability to adapt. Animals can modify their gait in the
presence of an injured limb, using non-injured limbs in a compensatory manner
(Fuchs et al., 2014; Jarvis et al., 2013). An established approach to
incorporate adaptable, compensatory behaviours into machines is through
algorithmic reconfiguration. In algorithmic reconfiguration, an agent adjusts
the underlying algorithms that govern hardware usage within a machine,
enabling adaptation to changes in hardware conditions (Guiochet et al., 2017;
Visinsky et al., 1994, 1991). Algorithmic reconfiguration may entail altering
the algorithm’s hyperparameter settings, adjusting the algorithm’s model
architecture, or even switching to an entirely different algorithm.

In essence, algorithmic reconfiguration is closely linked to continual
learning. With continual learning, when a machine faces evolving hardware
conditions throughout its lifetime, it must adapt to succeed in the changed
conditions. Adaptation in continual learning involves leveraging the
continuous flow of data to modify aspects of the algorithm, such as the models
parameters and/or the storage contents. To accelerate the adaptation process,
previously acquired knowledge can be strategically transferred to the new task
(Chen and Liu, 2018).

In nature, adaptation to physical changes is often achieved through a trial-
and-error approach, enhanced by knowledge acquired from past experiences.
Continual Reinforcement Learning (CRL) (Abel et al., 2024; Sutton and Barto,
2018; Xu and Zhu, 2018) is an ideal strategy for an agent tasked with adapting
a machine’s behaviour to unexpected events, such as hardware faults. CRL
typically entails an agent continuously interacting with a non-stationary
environment, learning from feedback (trial-and-error), and strategically
transferring acquired knowledge.

Figure 1: Overview of our study. A machine (robot) encounters a fault at
$t{=}t^{*}$. We explore approaches for transferring knowledge acquired by the
agent learning in a normal environment up to $t{=}t^{*}$
(${\mathcal{K}}_{t{=}t^{*}}$), thereby constructing a prior for the agent in a
fault environment.

Despite CRL being a valuable strategy for enhancing hardware fault tolerance
in machines, it remains relatively unexplored. In this study, we examine how
CRL can enhance hardware fault tolerance in simulated machines. Figure 1
outlines our study’s approach, where knowledge acquired by an agent learning
in a normal environment (${\mathcal{K}}_{t{=}T}$) is selectively transferred
and adapted by the same agent in a changed, fault environment. Specifically,
we study the impact of retaining or discarding components of the agent’s
acquired knowledge, such as the learned model parameters and the storage
contents, at the onset of a fault. We examine how retaining and fine-tuning,
or discarding, the model parameters learned in a normal environment affects
adaptation in a fault environment. Additionally, we explore the effect of
retaining or discarding the storage contents, which, if retained, may
initially serve as a proxy for a model of the normal environment. Our
hypothesis is that ${\mathcal{K}}_{t{=}t^{*}}$ contains useful information
that can enable faster adaptation to hardware faults. We empirically evaluate
the effectiveness of two state-of-the-art Reinforcement Learning (RL)
algorithms, Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC), in
improving hardware fault tolerance in machines. For our evaluation, we use
widely accepted benchmarks from the OpenAI Gym toolkit, namely Ant-v2 and
FetchReach-v1 (Brockman et al., 2016).

To the best of our knowledge, this study is the first to comprehensively
examine the performance of PPO and SAC in the context of hardware fault
enhancement in machines. Our findings provide valuable insights into
optimizing knowledge transfer in a CRL and hardware fault problem setting,
thereby accelerating machine adaptation.

In the subsequent sections, we outline the framework and findings of our
study. In Section 2, we review the foundational knowledge and literature
related to our study. In Section 3, we describe the methods applied in our
study. Within Section 4, we present our results. In Section 5, we discuss the
interpretation of our findings and outline potential directions for future
work. Lastly, in Section 6, we summarize the outcomes of our work and
highlight their broader significance.

##  2 Background and Related Work

This section serves as a foundation for understanding the research context and
examines the existing literature relevant to our study. In Section 2.1, we
provide a comprehensive overview of key concepts and techniques in RL. In
Section 2.2, we review notable works to provide a comprehensive understanding
of the research landscape. By presenting this background information and
discussing relevant works, we aim to establish a solid foundation for our
research and highlight the unique contributions of our study in this field.

###  2.1 Reinforcement Learning

RL is a subfield of machine learning that enables agents to autonomously learn
by making sequential decisions through interactions with their environment.
Sequential decision-making problems are typically modelled as a Markov
Decision Process (MDP). An MDP is defined by the tuple
$(\mathcal{S},\mathcal{A},R,P,\gamma)$, where $\mathcal{S}$ denotes the state
space; $\mathcal{A}$ denotes the action space;
$R:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function;
$P(s_{t+1}|s_{t},a_{t})$ represents the MDP dynamics, which is the probability
of transitioning to state $s_{t+1}$ after taking action $a_{t}$ in state
$s_{t}$, where $s_{t},s_{t+1}\in\mathcal{S}$, $a_{t}\in\mathcal{A}$; and
$\gamma\in[0,1]$ denotes the discount rate (Sutton and Barto, 2018).

Interaction between the agent and the environment occurs over a series of
discrete time steps, represented by $t=0,1,2,...,H$, where $H$ is the horizon,
which may be infinite. During each interaction, the environment provides the
agent with information on its current state $s_{t}$. Guided by its policy,
represented as the function $\pi:\mathcal{S}\rightarrow\mathcal{A}$, the agent
selects and executes an action $a_{t}\in\mathcal{A}(s_{t})$ for the next time
step. Subsequently, the agent receives feedback in the form of a numerical
reward $r_{t+1}\in\mathcal{R}$, and the environment updates the agent on its
new state $s_{t+1}$. The sequence of states and actions forms a trajectory,
denoted by $\tau$. The primary objective of the agent is to maximize the
expected discounted cumulative reward received in a trajectory $\tau$, known
as the expected discounted return, denoted by
$J(\pi)=\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{H}\gamma^{t}R(s_{t},a_{t})]$.
This objective guides the agent’s learning, as it continually seeks to select
actions that result in higher rewards. The optimization problem that aims to
maximize the objective function $J(\pi)$ can be expressed as:

$\pi^{*}=\operatorname*{arg\,max}_{\pi}J(\pi),$ (1)

where $\pi^{*}$ is the optimal policy.

####  2.1.1 Policy Gradient Methods

Policy gradient methods optimize the policy directly, as described by Equation
1. The policy is represented as a parameterized function, i.e., a neural
network, denoted as $\pi_{\theta}(\cdot|s)$, where $\theta\in\mathbb{R}^{d}$
represents the policy’s $d$-dimensional parameters. In policy gradient
methods, the policy parameters are evaluated using a performance measure and
are iteratively updated using gradient ascent. This process improves the
policy by identifying parameter values that yield higher expected returns.

####  2.1.2 Proximal Policy Optimization (PPO)

PPO is an on-policy algorithm designed to limit the magnitude of policy
updates, thus ensuring stable and gradual changes during training (Schulman et
al., 2017). PPO has proven to be an effective algorithm for robotic
applications (Kuo and Chen, 2023).

A PPO agent collects experiences — namely, a collection of trajectories —
through interactions with the environment, while following the current policy.
The collected experiences are stored in the agent’s memory ${\mathcal{M}}$.
Once $\mathcal{M}$ is full, these trajectories are used to compute the loss
using a surrogate objective function, $L^{CLIP}(\theta)$, which approximates
policy improvement and guides the policy update from the current policy
$\pi_{\theta_{old}}$ to an updated candidate policy $\pi_{\theta}$. To prevent
drastic policy changes that could destabilize learning, $L^{CLIP}(\theta)$
incorporates a clipping mechanism, constraining the updated candidate policy
to be closely aligned with the current policy. The parameters $\theta$ are
optimized over one or more epochs, using mini-batches of experiences sampled
from $\mathcal{M}$, according to the following equation:

$\theta^{*}=\operatorname*{arg\,max}_{\theta}L^{CLIP}(\theta)$ (2)

After completing the update, the experiences stored in $\mathcal{M}$ are
discarded. The algorithm then collects new experiences using the updated
policy and stores them in $\mathcal{M}$. This iterative process allows PPO to
refine the policy gradually and converge towards an optimal or near-optimal
solution. We denote the solution at convergence, in the normal environment, as
$\theta_{PPO}$. For more details on PPO, we refer readers to the original
paper (Schulman et al., 2017).

####  2.1.3 Soft Actor-Critic (SAC)

SAC is an off-policy algorithm that is well-suited for real-world problems,
owing to its low sample complexity and its robustness to various
hyperparameter settings (Haarnoja et al., 2018, 2019). A distinguishing
feature of SAC is its use of an entropy regularization term in its objective
function, denoted as $J^{\prime}(\pi_{\theta}(\cdot|s))$. The entropy
regularization term encourages exploration by favouring stochastic policies. A
temperature coefficient for the entropy regularization term adjusts the
balance between entropy and expected returns. SAC incorporates an automatic
adjustment mechanism for this coefficient, achieved by imposing a lower-bound
average policy entropy constraint during the optimization process. This
mechanism enables SAC to strike a delicate balance between exploration and
exploitation, allowing the agent to learn a policy that effectively adapts to
diverse environmental conditions while maximizing expected returns.

To facilitate the learning process, SAC uses a replay buffer, denoted as
$\mathcal{B}$, to retain past experiences. Once $\mathcal{B}$ reaches a
specified minimum capacity, batches of experiences — specifically, past agent-
environment interactions — are uniformly sampled from $\mathcal{B}$ at each
time step and used to update the parameters. The parameters are optimized
using:

$\theta^{*}=\operatorname*{arg\,max}_{\theta}J^{\prime}(\pi_{\theta}(\cdot|s))$
(3)

Following each update, a new experience is generated under the updated policy
and is added to the replay buffer $\mathcal{B}$, replacing the oldest stored
experience.

It is worth noting that each experience stored in the replay buffer is
generated under a different policy. The ability to reuse past experiences,
even when generated under different policies, is what enables SAC to be highly
sample-efficient, requiring fewer interactions with the environment compared
to on-policy algorithms, such as PPO.

We refer to the SAC solution at convergence, in the normal environment, as
$\theta_{SAC}$. For more details on SAC, we refer readers to the original
paper (Haarnoja et al., 2018).

###  2.2 Related Works

####  2.2.1 Collection of Pre-Trained Policies

The algorithm developed by Raileanu et al. (2020), known as Policy-Dynamics
Value Functions (PD-VF), demonstrated rapid adaptability to previously unseen
environments with changed dynamics. The methodology used in this work involved
training a set of policies correspondingly with trajectories collected within
a set of environments, each characterized by its unique dynamics. At test
time, when faced with an unseen environment, each policy from the set was
evaluated in the test environment for a small number of time steps. The policy
that exhibited the highest performance in the test environment was considered
to be the adapted policy. To evaluate the effectiveness of their approach,
Raileanu et al. (2020) applied PD-VF to four Ant-Legs environments, where
modifications were made to the length of the Ant-v2 robot model’s linkages.

A limitation of this work is that, unlike online learning algorithms such as
SAC and PPO, PD-VF does not learn a policy in real-time. Rather, it selects
the best-performing policy from a collection of pre-trained policies. This
approach deprives improvement of a transferred policy through interaction with
an environment. In contrast, SAC and PPO are online learning algorithms
capable of adapting a transferred policy. Furthermore, a competing algorithm,
PPO all, was trained in a multi-task setting, where negative transfer can
hinder performance if the training set tasks are too dissimilar (Rosenstein et
al., 2005), rendering this method of limited practical value in fault
adaptation domain, where fault tasks may be drastically different from normal
ones. In our study, we train PPO on a single task, demonstrating that the
learned, high-performing policy is both transferable and quickly adaptable to
unseen tasks with similar structure, and outperforming the reported
performance of PD-VF on comparable tasks.

####  2.2.2 Meta-Reinforcement Learning

Nagabandi et al. (2018) introduced two variants of meta-RL algorithms, namely
a Gradient-Based Adaptive Learner (GrBAL) and a Recurrence-Based Adaptive
Learner (ReBAL), which demonstrated superior speed and sample efficiency in
adapting to hardware faults in robots. These two algorithms learn a model of
the environment dynamics, enabling rapid adaptation to hardware faults with as
little as 1.5 - 3 hours of real-world experience. Nevertheless, it is worth
noting that a third model-based RL algorithm, not belonging to the meta-RL
class, performed comparably, underscoring the sample efficiency of model-based
approaches. The evaluation of the algorithms was conducted on an Ant: crippled
leg task, which involved damaging a single leg of the Ant-v2 robot model.

A drawback of the algorithms proposed by Nagabandi et al. (2018) is that
machine faults can arise unexpectedly, exhibiting diverse and unpredictable
characteristics. Meta-RL algorithms, such as GrBAL and ReBAL, require prior
knowledge of the fault type to define a task distribution customized for a
particular fault, which limits their practicality. In contrast, PPO and SAC
have the advantage of adapting without any prior knowledge of the fault type.
This flexibility enables them to effectively handle a wide range of faults
without the need for specialized pre-training. Furthermore, we show that PPO
and SAC outperform GrBAL and ReBAL on comparable tasks, emphasizing the
tendency of model-based approaches to achieve suboptimal asymptotic
performance when compared to model-free approaches.

####  2.2.3 Pre-Processing In Simulation

Cully et al. (2014) used a robot simulator to construct a low-dimensional
behaviour performance map. In the event of a fault, a real-world robot would
take a trial-and-error approach to fault adaptation, iteratively selecting and
executing the projected best-performing behaviour from the behaviour
performance map, and subsequently updating the map with the real-world
performance. This iterative procedure would continue until a high-performing
behaviour, surpassing a predefined threshold, was identified. Notably, this
approach demonstrated significant success, achieving recovery within minutes
in a low-dimensional parameter search space.

This approach, however, had certain limitations. Firstly, it relied on the
availability of a simulator to generate a behaviour-performance map, which
limited its practicality for robots lacking simulators. Furthermore, this
method faced challenges when confronted with a high-dimensional parameter
search space. Our study addresses these limitations by offering broad
applicability to all robots without the need for prior computation in
simulation. Furthermore, we demonstrate the effectiveness of adaptation using
the PPO and SAC algorithms, even within high-dimensional parameter spaces, by
utilizing raw sensor output data to select continuous actions.

####  2.2.4 A Policy for Each Actuator

Huang et al. (2020) introduced a decentralized approach to learning called
Shared Modular Policies (SMP), where each component of a robot, e.g.,
actuator, was treated as an individual module. Each module had its own policy,
which was instantiated from a global policy (i.e., a single, reusable neural
network). The input to each policy included raw data from the module’s local
sensors. A system of message passing allowed for communication between
modules. SMP was trained in a multi-task learning setting, where each task
involved a robot model with a unique arrangement of linkages and/or joints.

Huang et al. (2020) evaluated SMP in the OpenAI Gym environments Walker2D-v2,
Humanoid-v2, Hopper-v2, and HalfCheetah-v2. Although SMP was shown to perform
well on simple tasks, it faced difficulties on more complex tasks (e.g.,
Humanoid-v2). The performance of SMP was compared to a baseline, where a
centralized policy was trained using Twin Delayed DDPG (TD3) (Fujimoto et al.,
2018) in a multi-task setting, highlighting the effectiveness of SMP. Notably,
previous work has shown that training a centralized policy in a multi-task
setting, as was done with TD3, can lead to performance issues due to negative
transfer from vastly different tasks (Rosenstein et al., 2005).

In our study, we employ a centralized approach to learning, using a single
policy to control all actuators in the robot model, within a single-task
setting. This approach to learning demonstrates high performance within a
high-dimensional state space effectively capturing interactions and
interdependencies among actuators, and is shown to be transferable and quickly
adaptable to new, unseen environments. Unlike SMP, our transferred policy does
not require significant pre-training in a multi-task setting. Instead, it is
trained using experiences collected under normal conditions, emphasizing its
practicality and efficiency.

##  3 Methodology

In this section, we describe the experimental procedures used in our study
exploring the potential of PPO and SAC in enhancing hardware fault tolerance
in simulated machines. Our methodology consists of three phases: (1) learning
a task in a normal (pre-fault) environment, (2) introducing a hardware fault,
and (3) continuing learning the task from Phase 1 in a (post-)fault
environment.

In our study, we use two OpenAI Gym environments, Ant-v2 and FetchReach-v1,
simulated using the MuJoCo 2.0 physics engine (Todorov et al., 2012).

###  3.1 Experimental Phases

In this section, we describe the three phases of our experiments.

####  3.1.1 Phase 1

Phase 1 involves having an agent learn a task in a standard OpenAI Gym
environment for $t^{*}$ time steps, using either PPO or SAC. In the OpenAI Gym
Ant-v2 environment, the task requires an agent to learn a gait that maximizes
the forward propulsion speed of a four-legged ant. In the OpenAI Gym
FetchReach-v1 environment, the task involves an agent learning to move the end
effector (or closed gripper) of a robot arm to a randomly generated, three-
dimensional goal position as quickly and accurately as possible. We refer to
the standard OpenAI Gym environment as a normal (pre-fault) environment.

At the end of Phase 1, the knowledge attained by the agent in the normal
environment can be represented as
${\mathcal{K}}_{t=t^{*}}{=}\\{\theta_{PPO},{\mathcal{M}}\\}$ and
${\mathcal{K}}_{t=t^{*}}{=}\\{\theta_{SAC},{\mathcal{B}}\\}$, respectively.

####  3.1.2 Phase 2

In Phase 2, a hardware fault is introduced into the environment. This
modification aims to replicate a real-world hardware fault for experimental
purposes. The hardware faults we replicate are described in detail in Section
3.2.

####  3.1.3 Phase 3

Phase 3 involves the continuation of learning the task with a robot model
experiencing the hardware fault introduced in Phase 2. We refer to the
modified OpenAI Gym environment as a fault environment.

At the start of Phase 3, the agent’s knowledge from Phase 1,
${\mathcal{K}}_{t=t^{*}}$, is transferred111We assume that a fault is
detected.. To investigate knowledge transfer in the context of continual
learning, we conduct an ablation study with four different approaches, as
shown in Table 1.

| RL Algorithm
---|---
Knowledge Transfer Approach |  PPO |  SAC
Approach 1 |  $\text{retain}~{}\theta_{PPO},\text{retain}~{}\mathcal{M}$ |  $\text{retain}~{}\theta_{SAC},\text{retain}~{}\mathcal{B}$
Approach 2 |  $\text{retain}~{}\theta_{PPO},\text{discard}~{}\mathcal{M}$ |  $\text{retain}~{}\theta_{SAC},\text{discard}~{}\mathcal{B}$
Approach 3 |  $\text{discard}~{}\theta_{PPO},\text{retain}~{}\mathcal{M}$ |  $\text{discard}~{}\theta_{SAC},\text{retain}~{}\mathcal{B}$
Approach 4 (baseline) |  $\text{discard}~{}\theta_{PPO},\text{discard}~{}\mathcal{M}$ |  $\text{discard}~{}\theta_{SAC},\text{discard}~{}\mathcal{B}$
Table 1: The four knowledge transfer approaches in our ablation study.

###  3.2 Hardware Faults

In this section, we detail the specific hardware faults introduced in our
experiments.

####  3.2.1 Ant-v2

In the Ant-v2 environment, we introduce a fault to the right rear leg of the
ant. This leg is one of two legs primarily responsible for generating the
forward pushing motion of the ant, while the other two legs primarily
contribute to stability, as evidenced by numerous runs conducted within the
normal environment. By introducing a fault to the right rear leg, the agent’s
learned policy is no longer effective — its ability to effectively propel the
ant forward is impeded. Consequently, the agent experiences difficulty in
achieving the desired task of maximizing the ant’s forward speed.

We implement four distinct faults in separate instances of the Ant-v2
environment, as shown in Figure 2. Each fault independently alters the
underlying dynamics of the environment, thereby impacting the ant’s locomotion
capabilities. The following paragraphs describe each fault type in detail.

(a) Hip ROM Restriction

(b) Ankle ROM Restriction

(c) Broken, Severed Linkage

(d) Broken, Unsevered Linkage

Figure 2: The four distinct faults introduced to separate instances of the
Ant-v2 environment. Links and/or joints affected by a fault are indicated in
red.

##### Reduced Range of Motion

In real-world machines, the presence of external or internal debris can
disrupt the gear mechanics within a joint, limiting its Range of Motion (ROM).
To simulate this fault, we consider two cases that restrict the ROM of a joint
in the ant’s right rear leg: (1) restricting the hip joint motion to $[-5,5]$
degrees (Figure 2(a)), and (2) restricting the ankle joint motion to $[65,70]$
degrees (Figure 2(b))222The original ranges of hip joint motion and ankle
joint motion are $[-30,30]$ and $[30,70]$ degrees, respectively..

##### Broken Limb

Real-world robot linkages are susceptible to damage from physical impacts
(Steinbauer-Wagner, 2012). A linkage, the rigid section between the joints on
a robot, can sustain damage in such a way that it can become broken and (1)
dissociated from the robot frame, or (2) partially dissociated from the robot
frame, with the linkage held together by a weakened, unbroken section or inner
connector cables. In our experiments, we consider these two types of faults.
To simulate the former (i.e., a broken, severed linkage, as shown in Figure
2(c)), we reduce the length of the lower link in the ant’s right rear leg by
half. To simulate the latter (i.e., a broken, unsevered linkage, as shown in
Figure 2(d)), we also reduce the length of the lower link in the ant’s right
rear leg by half. However, at the end of the modified link, we add an
unactuated hinge joint with a range of $[-180,180]$ degrees and a link with
the same shape and length as the removed section. We then adjust our
observation vector to exclude entries that report data for the added joint and
link, ensuring it matches the dimensionality of the unmodified Ant-v2
environment observation vector.

####  3.2.2 FetchReach-v1

In the FetchReach-v1 environment, we introduce a fault to the shoulder lift
joint in one instance and to the elbow flex joint in another. We choose these
two joints because they are crucial for the robot arm to succeed in its task.
Introducing a fault to either joint causes the the robot arm to experience
inaccuracies, resulting in delays in the robot arm reaching the goal position
with its end effector.

(a) Frozen Shoulder Lift Position Sensor

(b) Slippery Elbow Flex Joint

Figure 3: FetchReach-v1 faults. Links and/or joints affected by a fault are
indicated in red.

##### Frozen Position Sensor

A frozen sensor is a common type of sensor fault. A frozen sensor consistently
reports a constant value that may not accurately reflect the physical property
being measured (Visinsky et al., 1991). The first fault in FetchReach-v1,
depicted in Figure 3(a), is a frozen shoulder lift position sensor. To
simulate this fault, we modify the robot’s shoulder lift position sensor so
that it continuously reports a position of $-1.5$ radians. This inaccurately
reported position subsequently affects the computed position of the robot’s
end effector, which is calculated using forward kinematics within MuJoCo. As a
result, this fault impacts the observation obtained in FetchReach-v1. Notably,
this fault does not alter the underlying dynamics of the environment itself.
Rather, it creates a discrepancy because the observation no longer accurately
represents the true state of the robot, with one entry being noisy.

##### Position Slippage

In mechanical gears, gear teeth can break, leading to operational issues as
the gear slips to the next non-broken tooth, causing inaccurate gear
movements. Our second fault in FetchReach-v1, depicted in Figure 3(b), is a
slippery elbow flex joint. This fault causes the joint to move further than
commanded by a constant value when an action command is given. Specifically,
if the joint position is expected to change by $x$ radians, this fault causes
the joint position to move by $x+c$ radians, where $c$ is a constant noise
value of $0.05$ radians.

###  3.3 Evaluation

For each experiment, we conduct 30 runs using seeds ranging from 0 to 29. All
experiments assess learning progress through policy evaluation rollouts. At
predefined intervals, learning is temporarily paused and the current policy is
evaluated for 10 episodes. The reported metric for our experiments is the
average return in a policy evaluation rollout, calculated by averaging across
30 runs. Additionally, we compute 95% confidence intervals using a
t-distribution and indicate them on our plots with shaded regions.

In our primary experiments, we ensure both the normal and fault environments
are allocated the same learning durations, allowing the algorithms to reach
convergence. In the Ant-v2 environment, the learning duration for the PPO
algorithm is set at 600 million time steps, with an evaluation frequency of 3
million time steps. For the SAC algorithm, the learning duration is 20 million
time steps, with evaluations every 100,000 time steps. In the FetchReach-v1
environment, PPO is trained for 6 million time steps with an evaluation
frequency of 30,000 time steps, while SAC trains for 2 million time steps,
with evaluations every 10,000 time steps.

To highlight the early adaptation capabilities of each algorithm, we conduct a
secondary set of experiments in the fault environments, adjusting both the
learning durations and evaluation frequencies. In the Ant-v2 fault
environments, both the PPO and SAC algorithms have learning duration of
300,000 time steps, with evaluations conducted every 10,000 steps. Similarly,
in the FetchReach-v2 fault environments, the learning duration for both
algorithms is set at 30,000 time steps, with evaluations every 10,000 steps.

###  3.4 Algorithm Implementation

In this section, we present a concise overview of our implementation of two RL
algorithms: Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC). We
describe the neural network architectures for each algorithm, including the
number of layers and nodes. Additionally, we highlight specific implementation
choices such as the use of code-level optimizations.

####  3.4.1 Proximal Policy Optimization

Our implementation of PPO uses a single neural network comprising a policy
network and a value network. Both internal networks are designed as
feedforward networks with two hidden layers, each consisting of 64 nodes and
using hyperbolic tangent activation functions. The policy network incorporates
a learnable parameter known as the log standard deviation, which is initially
set to $0$. The Adam optimizer (Kingma and Ba, 2017) is used for optimization.
To improve the performance of PPO, we incorporate code-level optimizations
recommended by Engstrom et al. (2020), including linear learning rate decay,
use of a generalized advantage estimator (Mnih et al., 2016), orthogonal
initializations (Saxe et al., 2013), and hyperbolic tangent activation
functions. In the Ant-v2 environment, we deviate from the standard linear
learning rate decay rate of $1/\text{\\#updates}$. Instead, we decay at a rate
of $0.25/\text{\\#updates}$, which leads to improved convergence. Notably,
prior to learning in the fault environment, we reset PPO’s learning rate to
its initial value. This reset allows the algorithm to adapt to the changed
environment.

####  3.4.2 Soft Actor-Critic

Our implementation of SAC uses two twinned soft Q-networks and a policy
network. All networks are feedforward networks with two hidden layers, each
consisting of 256 nodes and using Rectified Linear Unit (ReLU) activations.
Each layer is initialized using Xavier uniform initialization (Glorot and
Bengio, 2010). Optimization is performed using the Adam optimizer (Kingma and
Ba, 2017).

Hyperparameter |  Possible Values |  Ant-v2 |  FetchReach-v1
---|---|---|---
number of samples |  $\\{32,33,34,\dots,5000\\}$ |  |
mini-batch size |  $\\{4,8,16,\dots,8192\\}$ |  $1024$ |  $8$
epochs |  $\\{3,4,5,\dots,30\\}$ |  $5$ |  $24$
epsilon ($\epsilon$) |  $\\{0.1,0.2,0.3\\}$ |  $0.3$ |  $0.3$
gamma ($\gamma$) |  $[0.8,0.9997]$ |  $0.9839$ |  $0.848$
GAE lambda |  $[0.9,1.0]$ |  $0.911$ |  $0.9327$
value function loss coefficient ($c_{1}$) |  $\\{0.5,1.0\\}$ |  $1.0$ |  $1.0$
policy entropy coefficient ($c_{2}$) |  $[0,0.01]$ |  $0.0019$ |  $0.0007$
learning rate |  $[0.000005,0.006]$ |  $0.000123$ |  $0.000275$
Table 2: Possible values for the hyperparameters of PPO.

####  3.4.3 Hyperparameter Optimization

Real-world machines are typically optimized to perform effectively under
normal conditions. When a fault occurs, conducting a new hyperparameter search
tailored to the changed conditions can be time-consuming and computationally
demanding. Therefore, we focus our hyperparameter optimization exclusively on
learning the OpenAI Gym task in a normal environment (i.e., with a normal
robot model).

Hyperparameter |  Possible Values |  Ant-v2 |  FetchReach-v1
---|---|---|---
replay buffer size |  $\\{10k,100k,500k,1000k\\}$ |  500k |  10k
batch size |  $\\{16,64,256,512\\}$ |  512 |  512
target smoothing coefficient ($\tau$) |  $[0.0001,0.1]$ |  0.0721 |  0.0877
gamma ($\gamma$) |  $[0.8,0.9997]$ |  0.8097 |  0.9646
learning rate |  $[0.000005,0.006]$ |  0.001738 |  0.001092
Table 3: Possible values for the hyperparameters of SAC.

To optimize the hyperparameters of each algorithm, we adopt a random search
strategy. We employ this strategy by randomly selecting values for each
hyperparameter from either a specified continuous interval $[a,b]$ or a
specified set ${x_{1},x_{2},\dots}$. To ensure thorough exploration, we
evaluate a minimum of 80 unique hyperparameter configurations for each
algorithm. This is achieved by varying the Numpy random seed values from 0 to
100. To mitigate the impact of random fluctuations and obtain a reliable
estimate of the algorithm’s performance, we conduct 10 runs for each
hyperparameter configuration using seed values ranging from 0 to 9. Tables 2
and 3 provide an overview of the possible values used to randomly draw
hyperparameters for PPO and SAC, respectively.

To identify the best hyperparameter configuration for each algorithm, we
employ a two-step process. We first determine the hyperparameter
configuration(s) that achieved the highest asymptotic performance. The highest
asymptotic performance is defined as the highest average return in the final
10 policy evaluation rollouts, averaged across 30 runs. In cases where
multiple hyperparameter configurations demonstrate comparable highest
asymptotic performance, we further evaluate them based on the speed of initial
learning. This involves assessing how quickly the average return improves over
time.

Consequently, the best hyperparameter configuration is determined by
considering the highest asymptotic performance and, if necessary for
exclusion, the best initial performance. By following this methodology, we aim
to identify the hyperparameter configuration that maximizes the algorithm’s
overall performance in terms of achieving high average returns and efficient
learning.

Tables 2 and 3 summarize the best performing hyperparameters for each standard
OpenAI Gym environment (i.e., normal environment).

##  4 Results

In this section, we address four distinct focal points of investigation. In
Section 4.1, we examine whether PPO and SAC exhibit adaptation to hardware
faults. In Section 4.2, we analyze the differences in PPO and SAC in
performance, sample efficiency, and adaptation speed in a fault environment.
In Section 4.3, we assess the performance disparities among the four knowledge
transfer approaches in a CRL and hardware fault setting. Finally, in Section
4.4, we compare PPO and SAC with previous strategies for hardware fault
adaptation.

(a) No Adaptation

(b) Adaptation

Figure 4: The state-visitation probability distribution of each joint in the
Ant-v2 Ankle ROM Restriction fault environment is examined under two
scenarios: (a) immediately after transferring all knowledge acquired in the
normal environment — the model parameters and the memory (PPO) or replay
buffer (SAC) — with no adaptation within the fault environment, and (b) after
transferring all knowledge acquired in the normal environment, allowing for a
period of adaptation (fine-tuning) within the fault environment. We observe a
shift in the state-visitation distribution due to policy adaptation.

###  4.1 Adaptation to Hardware Faults

To demonstrate the adaptation of an agent to a hardware fault, we assess its
ability to learn compensatory behaviours through model (e.g., policy)
adaptation. A change in the agent’s policy should alter the state-visitation
distribution within the environment. To gather data for this evaluation, we
run a specified algorithm and policy in the fault environment for 100
episodes, recording the joint angles visited throughout.

In Figure 4, heatmaps depict the observed shift in the distribution of visited
joint angles in the Ankle ROM Restriction fault environment. Each heat map
shows the probability of visiting angles within the normalized range for each
joint with the specified algorithm and policy. For each algorithm, we compare:
(a) the knowledge acquired in the normal environment and transferred to the
fault environment without adaptation and (b) the knowledge acquired in the
normal environment, transferred to the fault environment, and subsequently
adapted (fine-tuned) within the fault environment. For both PPO and SAC, the
transferred and adapted policy shows a reduction in how often the hip_4 joint
reaches the upper end of its range when compared to the transferred and
unadapted policy. Additionally, we observe variations in how frequently the
other joints reach the upper or lower limits of their respective ranges.

(a) Hip ROM Restriction

(b) Ankle ROM Restriction

(c) Broken, Severed Limb

(d) Broken, Unsevered Limb

Figure 5: Learning curves depicting the performance of PPO and SAC in four
Ant-v2 fault environments, with the four knowledge transfer approaches. The
average return is plotted against the number of training steps in millions,
illustrating the adaptation and learning efficiency of each approach.

###  4.2 Performance, Sample Efficiency, and Real-Time Speed

(a) Frozen Shoulder Lift Position Sensor

(b) Elbow Flex Position Slippage

Figure 6: Learning curves depicting the performance of PPO and SAC in two
FetchReach-v1 fault environments, with four knowledge transfer approaches:
retaining or discarding the model parameters $\theta$ and the replay buffer
$\mathcal{B}$. The average return is plotted against the number of training
steps in millions, illustrating the adaptation and learning efficiency of each
approach.

In our second comparative evaluation, we examine the performance, sample
efficiency, and real-time speed of the adaptation process with PPO and SAC in
the six fault environments. Figures 5 and 6 illustrate the average return
achieved by each of the four knowledge transfer options with respect to the
number of real experiences (i.e., time steps) in the fault environments.
Figure 5 depicts the real-time equivalent of 10 hours (PPO) and 4.8 hours
(SAC) of learning in the four Ant-v2 fault environments, while Figure 6
depicts the real-time equivalent of 1.1 hours (PPO) and 0.6 hours (SAC) of
learning in the two FetchReach-v1 fault environments.

For our evaluation, we focus on each algorithm’s best knowledge transfer
approach in each fault environment. We define each algorithm’s best knowledge
transfer approach as the approach exhibiting the highest performance
throughout the entire learning process (depicted in Figures 5 and 6). For
example, in the Hip ROM Restriction fault environment shown in Figure 5(a),
the best knowledge transfer approach for PPO is to retain the model parameters
$\theta$, noting that discarding or retaining the memory $\mathcal{M}$ has
little impact on performance; whereas for SAC, the best knowledge transfer
approach is to retain the model parameters $\theta$ and the replay buffer
$\mathcal{B}$.

When comparing each algorithm’s best knowledge transfer approach, our results
show that PPO significantly outperforms SAC in the four Ant-v2 fault
environments depicted in Figure 5. Notably, PPO converges significantly faster
than SAC in real-time; within 1 million time steps (or 2.4 hours real-time),
SAC reaches a performance that is 10-40% less than that reached with PPO after
3 million time steps (or 30 minutes real-time). In the FetchReach-v1 fault
environments depicted in Figure 6, the differences in performance between PPO
and SAC are comparable. PPO attains near-asymptotic performance in both fault
environments within 100,000 time steps (or 11 minutes real-time), while SAC
reaches similar, near-asymptotic performance in fewer than 50,000 time steps
(or 9 minutes real-time).

Our findings demonstrate that PPO exhibits a comparable or faster real-time
adaptation to faults when evaluated against SAC. Furthermore, PPO achieves
higher performance in the high-dimensional Ant-v2 fault environments and near-
equivalent performance in the low-dimensional FetchReach-v1 fault
environments. However, due to its on-policy nature, PPO requires 2 to 3 times
more real experiences to reach near-asymptotic performance, indicating its
lower sample efficiency compared to SAC.

###  4.3 Transfer of Task Knowledge

(a) No Adaptation (0 Time Steps)

(b) Partial Adaptation (300,000 Time Steps)

Figure 7: Early performance in the Ant-v2 fault environments. The asymptotic
performance of the baseline (Approach 4) in each fault environment is
indicated by a purple dashed line. With PPO, retaining the model parameters
consistently leads to the best early performance. With SAC, discarding all
knowledge generally results in the best early performance.

In our third comparative evaluation, we investigate the performance of the
four knowledge transfer approaches in a CRL and hardware fault adaptation
setting. Our baseline (Approach 4) involves no knowledge transfer; tuples
$\\{\theta_{PPO},{\mathcal{M}}\\}$ and $\\{\theta_{SAC},{\mathcal{B}}\\}$ are
discarded for PPO and SAC, respectively. Consequently, the policy is trained
with data solely collected in each fault environment and is unbiased to the
normal environment.

(a) No Adaptation (0 Time Steps)

(b) Partial Adaptation (30,000 Time Steps)

Figure 8: Early performance in the FetchReach-v1 fault environments. The
asymptotic performance of the baseline (Approach 4) in each fault environment
is indicated by a purple dashed line.

Figures 7 and 8 show the early performance of each knowledge transfer approach
in the Ant-v2 and FetchReach-v1 fault environments at two critical points in
the learning process: (1) immediately after the onset of a fault, where
knowledge from the normal environment has been transferred (and discarded, if
applicable), but no learning in the fault environment has occurred; and (2)
300,000 time steps (Ant-v2) and 30,000 time steps (FetchReach-v1) after the
onset of a fault, chosen to highlight the notable differences in early
performance among the four knowledge transfer approaches in each fault
environment. The asymptotic performance of the baseline in each fault
environment is indicated by a black dashed line.

Our results show that, with PPO, retaining and fine-tuning the model
parameters $\theta_{PPO}$ generally leads to the best early performance in the
fault environments, as the knowledge contained within the models facilitates
rapid adaptation. However, we observe an exception with the Frozen Shoulder
Lift Position Sensor fault (depicted in Figure 8). In this particular case,
retaining the model parameters $\theta_{PPO}$ leads to sub-optimal performance
with PPO. The Frozen Shoulder Lift Position Sensor fault is unique as it is
the only fault that does not alter the dynamics of the environment. Rather, an
observation in this fault environment includes a noisy element that
inaccurately reports the position of the Shoulder Lift joint.

Additionally, we find that whether the memory $\mathcal{M}$ is retained or
discarded with PPO has minimal impact on performance in the fault
environments. In our experiments, the memory capacity is small and all
experiences are discarded from the memory after a single update. Consequently,
the old experiences retained in the memory have minimal influence on
adaptation.

Furthermore, in Figures 5 and 6, we observe that for PPO, the variability in
asymptotic performance across all four knowledge transfer options is slight in
most fault environments.

According to Figure 7, with SAC, we observe that our baseline (Approach 4)
generally leads to the best early performance, often outperforming the three
other knowledge transfer approaches. The second-best knowledge transfer
approach for SAC, in terms of early performance, is the retention of all
knowledge (i.e., the model parameters $\theta_{SAC}$ and the replay buffer
contents $\mathcal{B}$). Similar to PPO, the retained model parameters
$\theta_{SAC}$ contain knowledge that facilitates adaptation to the new
environment. However, in contrast to PPO, we find that retaining or disposing
of the contents of the replay buffer $\mathcal{B}$ significantly impacts the
performance of SAC. For example, when the model parameters $\theta_{SAC}$ are
retained and fine-tuned, disposing of the replay buffer $\mathcal{B}$
substantially degrades the performance of SAC (as depicted in Figure 7(b)).

In Figure 5, our results show significant variability in the asymptotic
performance of SAC across all four knowledge transfer approaches in the high-
dimensional Ant-v2 fault environments. Conversely, Figure 6 illustrates that,
in the low-dimensional FetchReach-v1 fault environments, the variability in
SAC’s performance is less pronounced. These results suggests that, when using
SAC, the manner in which knowledge is transferred to the fault environment
depends heavily on the task at hand.

###  4.4 Comparison With Prior Fault Adaptation Approaches

In our fourth comparative evaluation, we compare the performance of PPO and
SAC in our four Ant-v2 fault environments (shown in Figure 7) to the
performance of three meta-RL algorithms in an Ant: crippled leg environment
(Nagabandi et al., 2018) and the performance of PD-VF in four Ant-Legs
environments (Raileanu et al., 2020).

In the meta-RL experiments conducted by Nagabandi et al. (2018), the early
performance of two proposed meta-RL algorithms, namely GrBAL and ReBAL,
reached an average return of 430 in their Ant: crippled leg environment. The
authors do not provide information on the asymptotic performance of these two
algorithms. However, they reported the asymptotic performance of a third meta-
RL algorithm, Model-Agnostic Meta-Learning (MAML), in the same environment as
710.

In the PD-VF experiments conducted by Raileanu et al. (2020), the performance
of PD-VF in their four Ant-Legs environments ranged from 200 to 350,
consistently outperforming PPO all. PPO all, trained in a multi-task learning
setting, achieved a performance in the range of 120 to 320. Meanwhile, PPO
env, trained exclusively in the test environment, achieved a performance in
the range of 374 to 862, consistently outperforming PD-VF.

In our Ant-v2 fault environments, both PPO and SAC surpass the performance of
these prior methods. When the model parameters are retained, PPO achieves an
average return in the range of 934 to 2919 with no adaptation and 3006 to 4681
within 300,000 time steps. When discarding all knowledge, SAC achieves an
average return in the range of 1506 to 2273 within 300,000 time steps.
Notably, PPO, with no adaptation, attains a performance 2 to 7 times higher
than that reported for GrBAL and ReBAL, and 1.3 to 4 times higher than the
asymptotic performance of MAML. After some adaptation, PPO achieves an early
performance 7 to 11 times higher than the early performance reported for GrBAL
and ReBAL, and 4 to 7 times higher than the asymptotic performance reported
for MAML. Similarly, SAC, after adaptation, achieves an early performance 3 to
5 times higher than GrBAL and ReBAL, and 2 to 3 times higher than the
asymptotic performance of MAML. Furthermore, when the models are retained,
PPO, with no adaptation, also outperforms PPO env. It is worth noting that
other factors, such as the implementation of PPO and the disparity between the
fault environments, may additionally contribute to these differences in
performance.

While a more in-depth comparison of these methods is required, our preliminary
comparative evaluation indicates that PPO and SAC achieve competitive
performance compared to previous related work, especially when an appropriate
knowledge transfer method is chosen. This comparison suggests the potential of
more sophisticated knowledge transfer methods for off-the-shelf RL algorithms
in the context of machine fault adaptation.

##  5 Discussion and Future Work

The goal of this paper is to explore the effectiveness of transferring
knowledge acquired by an RL agent in a normal environment, where the agent
learns to perform a task with a fully-functioning robot model, to a fault
environment, where the agent must adapt to complete the same task with a robot
model experiencing a hardware fault. Four transfer learning approaches are
examined. Tables 4 and 5 summarize our findings regarding the early adaptation
speed of three knowledge transfer approaches relative to our baseline
(Approach 4) by calculating the percentage of time steps saved. For this
calculation, we first determine the number of time steps required for the
baseline to reach the lower bound of its 95% confidence interval for
performance (i.e., average return) at 300,000 time steps, which we represent
as $t_{baseline}$. We then determine the number of time steps required for
each of the three approaches to reach the lower bound of the baseline’s 95%
confidence interval for performance at 300,000 time steps, which we denote as
$t_{approach}$. To compute the percentage of time steps saved, we use the
formula:

$\text{\%}=100-\left(\frac{t_{approach}}{t_{baseline}}\times 100\right).$ (4)

Method | Ant-v2 Fault Environments
---|---
Algorithm |  Knowledge Transfer Approach |  Hip ROM Restriction |  Ankle ROM Restriction |  Broken, Severed Limb |  Broken, Unsevered Limb
PPO |  $\text{retain}~{}\theta_{PPO}\\\ \text{retain}~{}\mathcal{M}$ |  100.00% |  100.00% |  100.00% |  100.00%
$\text{retain}~{}\theta_{PPO}\\\ \text{discard}~{}\mathcal{M}$ |  100.00% |  100.00% |  100.00% |  100.00%
$\text{discard}~{}\theta_{PPO}\\\ \text{retain}~{}\mathcal{M}$ |  19.23% |  1.85% |  7.27% |  23.33%
SAC |  $\text{retain}~{}\theta_{SAC}\\\ \text{retain}~{}\mathcal{B}$ |  100.00% |  — |  — |  —
$\text{retain}~{}\theta_{SAC}\\\ \text{discard}~{}\mathcal{B}$ |  100.00% |  — |  — |  —
$\text{discard}~{}\theta_{SAC}\\\ \text{retain}~{}\mathcal{B}$ |  — |  — |  — |  —
Table 4: Adaptation Speed Analysis for Ant-v2: We assess the efficiency of three knowledge transfer approaches relative to the baseline (Approach 4) by measuring how quickly each approach reaches the lower bound of the baseline’s 95% confidence interval for performance at 300,000 time steps. We quantify this efficiency by calculating the percentage of time steps saved by each approach relative to our baseline using Equation 4. A dashed line indicates cases where an approach does not reach the lower bound of the 95% confidence interval for performance of the baseline, signifying its failure to reach the benchmark within the assessed time frame. The baseline efficiency, which is not shown, is 0% for all fault environments. Method | FetchReach-v1 Fault Environments
---|---
Algorithm |  Knowledge Transfer Approach |  Frozen Shoulder Lift Position Sensor |  Elbow Flex Position Slippage
PPO |  $\text{retain}~{}\theta_{PPO}\\\ \text{retain}~{}\mathcal{M}$ |  — |  68.00%
$\text{retain}~{}\theta_{PPO}\\\ \text{discard}~{}\mathcal{M}$ |  — |  68.00%
$\text{discard}~{}\theta_{PPO}\\\ \text{retain}~{}\mathcal{M}$ |  — |  -8.00%
SAC |  $\text{retain}~{}\theta_{SAC}\\\ \text{retain}~{}\mathcal{B}$ |  — |  92.59%
$\text{retain}~{}\theta_{SAC}\\\ \text{discard}~{}\mathcal{B}$ |  — |  96.30%
$\text{discard}~{}\theta_{SAC}\\\ \text{retain}~{}\mathcal{B}$ |  20.00% |  22.22%
Table 5: Adaptation Speed Analysis in FetchReach-v1: We assess the efficiency
of three knowledge transfer approaches relative to the baseline (Approach 4)
by measuring how quickly each approach reaches the lower bound of the
baseline’s 95% confidence interval for performance at 300,000 time steps. We
quantify this efficiency by calculating the percentage of time steps saved by
each approach relative to our baseline using Equation 4. A dashed line
indicates cases where an approach does not reach the lower bound of the 95%
confidence interval for performance of the baseline, signifying its failure to
reach the benchmark within the assessed time frame. Negative values indicate
an approach that takes longer than the baseline to reach the lower bound of
the 95% confidence interval for performance of the baseline. The baseline
efficiency, which is not shown, is 0% for all fault environments.

An analysis of Figures 7 and 8, as well the data presented in Tables 4 and 5,
leads us to conclude that the knowledge learned in the normal environment
indeed contains useful information that, in most cases, accelerates adaptation
in the fault environments.

Table 4 shows that, in the high-dimensional Ant-v2 fault environments, PPO
adapts faster when retaining (and fine-tuning) the model parameters
$\theta_{PPO}$, learned in the normal environment, than when discarding them.
Similarly, Table 5 shows that, when facing an Elbow Flex Position Slippage
fault in the low-dimensional FetchReach-v1 environment, retaining (and fine-
tuning) the model parameters $\theta_{PPO}$ is more advantageous than
discarding them. On the other hand, when facing a Frozen Shoulder Lift
Position Sensor fault, PPO performs best when the model parameters
$\theta_{PPO}$ are discarded. This fault introduces noise and unreliability in
the measurement of the Shoulder Lift joint’s position, resulting in
inaccuracies in the state representation that the agent relies on for making
decisions.

According to Figure 7, in the high-dimensional Ant-v2 environment, SAC
performs better when both the model parameters $\theta_{SAC}$ and replay
buffer $\mathcal{B}$ are discarded compared to when they are retained. This
improvement may be attributed to SAC’s optimization process, which includes an
entropy regularization term designed to encourage exploration that is balanced
against the rewards received for actions. While mechanism promotes diverse
action selection, it may result in increased exploitation of highly rewarding
actions. In the normal environment, SAC may have learned a policy that
exploited actions not suitable for the fault environments, resulting in sub-
optimal performance.

For SAC, retaining all knowledge acquired in the normal environment ranks as
the second-best knowledge transfer approach. Similar to PPO, the retained
model parameters $\theta_{SAC}$ contain knowledge that facilitates adaptation
to the fault environment. In contrast to PPO, the decision to retain or
discard the replay buffer $\mathcal{B}$ significantly impacts the performance
of SAC. When the model parameters $\theta_{SAC}$ are retained and fine-tuned,
discarding the replay buffer $\mathcal{B}$ degrades the performance of SAC
(see Figures 7(b) and 8(b)). This observation is consistent with the
challenges posed by distribution shift in offline learning methods, where
fine-tuning a policy learned offline can lead to instability and potentially
degraded performance when encountering online, out-of-distribution experiences
(Lee et al., 2021). With SAC, only one new experience is collected at each
time step. Thus, at the start of learning in the fault environment,
encountering online, out-of-distribution experiences is infrequent when the
replay buffer $\mathcal{B}$ is retained. Consequently, retaining the replay
buffer $\mathcal{B}$ enhances the stability and performance of SAC during
fine-tuning. Notably, in the Ant-v2 fault environments (Figure 7(b)), SAC’s
replay buffer is large, containing 500,000 experiences, leading to a
significant improvement in performance in the fault environments. Conversely,
in the FetchReach-v1 fault environments (Figure 8(b)), SAC’s replay buffer has
a smaller capacity limited to 10,000 experiences, limiting the beneficial
effect.

Building on these insights, our future research will focus on further
harnessing the potential of knowledge transfer methods. By examining the
characteristics of environments and the nature of faults, we aim to develop
more advanced, generalized knowledge transfer methods. Our goal is to explore
how the characteristics of environments and faults can be used to enhance the
design of knowledge transfer methods. Following the methodology proposed by
Chen et al. (2023), these advanced methods would selectively transfer the most
useful knowledge, determining what to transfer based on the similarities
between the current environments and those previously encountered. For
example, if an agent encounters an environment (e.g., fault) similar to one it
had previously successfully adapted to, transferring the specific knowledge
that proved most effective for that type of environment could be more
beneficial than transferring general knowledge. This approach is expected to
lead to more consistent performance across various scenarios.

We also emphasize that ensuring safety during and after adaptation to the
fault environment is essential. Current methods may not guarantee safe
behaviour, especially when encountering unforeseen faults. Thus, the challenge
of learning in safety-critical environments remains an open research avenue,
as evidenced by several works focusing on safe adaptation Zhang et al. (2020);
Peng et al. (2022). While our current work concentrates on enhancing the
performance of state-of-the-art RL algorithms using knowledge transfer
methods, we recognize the importance of safe adaptation in robotic
applications. Our future work will explore how fault adaptation techniques can
be designed with a focus on safety in various environments, including those
involving other agents (human or robotic), delicate obstacles, and scenarios
where a robot may risk self-harm.

##  6 Conclusion

In this study, we have showcased the effectiveness of two RL algorithms, PPO
and SAC, in enhancing hardware fault tolerance in simulated machines. We
conducted an ablation study to determine the most optimal approach for
knowledge transfer in a continual learning and hardware fault adaptation
setting.

Our findings indicate that transferring the models learned in the normal
environment to the fault environment with PPO is generally the most effective
knowledge transfer option. This approach yields the fastest performance
improvement with the fewest real experiences. Remarkably, the performance
achieved with PPO, after a brief period of adaptation, surpasses that of meta-
RL algorithms, in a comparable broken limb environment (Nagabandi et al.,
2018).

For SAC, our results demonstrate that discarding all knowledge acquired in the
normal environment and learning in the fault environment from scratch is
generally the preferred approach. Furthermore, the transfer of specific
components of the algorithm’s knowledge, such as the models and replay buffer
contents, had a significant impact on performance.

In conclusion, our study highlights the potential of RL algorithms as a
promising solution for enhancing hardware fault tolerance in real-world
machines. This solution holds promise for various industries, where it can
mitigate the consequences of minor fault-related events, thereby minimizing
productivity losses.

## Acknowledgements

We express our sincere gratitude to Mitsubishi Electric Corporation for their
generous support, which made this work possible. Their assistance has been
invaluable, and we are deeply grateful for the insightful feedback and
constructive input they have provided throughout the project.

## References

  * Abel et al. (2024) Abel, D., Barreto, A., Van Roy, B., Precup, D., van Hasselt, H.P., Singh, S., 2024.  A definition of continual reinforcement learning.  Advances in Neural Information Processing Systems 36.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., Zaremba, W., 2016.  Openai gym.  arXiv preprint arXiv:1606.01540 .
  * Chen et al. (2023) Chen, A.S., Chada, G., Smith, L., Sharma, A., Fu, Z., Levine, S., Finn, C., 2023.  Adapt on-the-go: Behavior modulation for single-life robot deployment.  arXiv preprint arXiv:2311.01059 .
  * Chen and Liu (2018) Chen, Z., Liu, B., 2018.  Lifelong machine learning.  Synthesis Lectures on Artificial Intelligence and Machine Learning 12, 1–207.
  * Cully et al. (2014) Cully, A., Clune, J., Mouret, J.B., 2014.  Robots that can adapt like natural animals.  arXiv preprint arXiv:1407.3501 .
  * Dubrova (2013) Dubrova, E., 2013.  Fault-Tolerant Design.  Springer, New York, NY.
  * Engstrom et al. (2020) Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., Madry, A., 2020.  Implementation matters in deep policy gradients: A case study on ppo and trpo.  arXiv preprint arXiv:2005.12729 .
  * Fuchs et al. (2014) Fuchs, A., Goldner, B., Nolte, I., Schilling, N., 2014.  Ground reaction force adaptations to tripedal locomotion in dogs.  The Veterinary Journal 201, 307–315.
  * Fujimoto et al. (2018) Fujimoto, S., Hoof, H., Meger, D., 2018.  Addressing function approximation error in actor-critic methods, in: International conference on machine learning, PMLR. pp. 1587–1596.
  * Glorot and Bengio (2010) Glorot, X., Bengio, Y., 2010.  Understanding the difficulty of training deep feedforward neural networks, in: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, PMLR, Sardinia, Italy. pp. 249–256.
  * Guiochet et al. (2017) Guiochet, J., Machin, M., Waeselynck, H., 2017.  Safety-critical advanced robots: A survey.  Robotics and Autonomous Systems 94, 43–52.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Abbeel, P., Levine, S., 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: Proceedings of the 35th International Conference on Machine Learning, PMLR, Stockholm, Sweden. pp. 1861–1870.
  * Haarnoja et al. (2019) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S., 2019.  Soft actor-critic algorithms and applications.  arXiv preprint arXiv:1812.05905 .
  * Haleem et al. (2021) Haleem, A., Javaid, M., Singh, R.P., Rab, S., Suman, R., 2021.  Hyperautomation for the enhancement of automation in industries.  Sensors International 2, 100124.
  * Huang et al. (2020) Huang, W., Mordatch, I., Pathak, D., 2020.  One policy to control them all: Shared modular policies for agent-agnostic control, in: Proceedings of the 37th International Conference on Machine Learning, PMLR, Virtual. pp. 4455–4464.
  * Jarvis et al. (2013) Jarvis, S.L., Worley, D.R., Hogy, S.M., Hill, A.E., Haussler, K.K., Reiser, R.F., 2013.  Kinematic and kinetic analysis of dogs during trotting after amputation of a thoracic limb.  American Journal of Veterinary Research 74, 1155–1163.
  * Javaid et al. (2021) Javaid, M., Haleem, A., Singh, R.P., Suman, R., 2021.  Substantial capabilities of robotics in enhancing industry 4.0 implementation.  Cognitive Robotics 1, 58–75.
  * Kingma and Ba (2017) Kingma, D.P., Ba, J., 2017.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Kuo and Chen (2023) Kuo, P.H., Chen, K.L., 2023.  Two-stage fuzzy object grasping controller for a humanoid robot with proximal policy optimization.  Engineering Applications of Artificial Intelligence 125, 106694.
  * Lee et al. (2021) Lee, S., Seo, Y., Lee, K., Abbeel, P., Shin, J., 2021.  Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.  arXiv preprint arXiv:2107.00591 .
  * Mnih et al. (2016) Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K., 2016.  Asynchronous methods for deep reinforcement learning, in: Proceedings of The 33rd International Conference on Machine Learning, PMLR, New York, NY. pp. 1928–1937.
  * Nagabandi et al. (2018) Nagabandi, A., Clavera, I., Liu, S., Fearing, R.S., Abbeel, P., Levine, S., Finn, C., 2018.  Learning to adapt in dynamic, real-world environments through meta-reinforcement learning.  arXiv preprint arXiv:1803.11347 .
  * Peng et al. (2022) Peng, Z., Li, Q., Liu, C., Zhou, B., 2022.  Safe driving via expert guided policy optimization, in: Conference on Robot Learning, PMLR. pp. 1554–1563.
  * Raileanu et al. (2020) Raileanu, R., Goldstein, M., Szlam, A., Fergus, R., 2020.  Fast adaptation via policy-dynamics value functions.  arXiv preprint arXiv:2007.02879 .
  * Riazi et al. (2019) Riazi, M., Zaïane, O., Takeuchi, T., Maltais, A., Günther, J., Lipsett, M., 2019.  Detecting the onset of machine failure using anomaly detection methods, in: Proceedings of the 21st International Conference on Big Data Analytics and Knowledge Discovery, Springer, Linz, Austria. pp. 3–12.
  * Rosenstein et al. (2005) Rosenstein, M.T., Marx, Z., Kaelbling, L.P., Dietterich, T.G., 2005.  To transfer or not to transfer, in: Proceedings of the NIPS 2005 workshop on transfer learning, NIPS, Whistler, British Columbia. pp. 1–4.
  * Samanta and Nataraj (2009) Samanta, B., Nataraj, C., 2009.  Use of particle swarm optimization for machinery fault detection.  Engineering Applications of Artificial Intelligence 22, 308–316.
  * Saxe et al. (2013) Saxe, A.M., McClelland, J.L., Ganguli, S., 2013.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks.  arXiv preprint arXiv:1312.6120 .
  * Schrum and Gombolay (2019) Schrum, M.L., Gombolay, M.C., 2019.  When your robot breaks: Active learning during plant failure.  IEEE Robotics and Automation Letters 5, 438–445.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., 2017.  Proximal policy optimization algorithms.  arXiv preprint arXiv:1707.06347 .
  * Steinbauer-Wagner (2012) Steinbauer-Wagner, G., 2012.  A survey about faults of robots used in robocup, in: Proceedings of the 16th RoboCup International Symposium, Springer, Mexico City, Mexico. pp. 344–355.
  * Sutton and Barto (2018) Sutton, R.S., Barto, A.G., 2018.  Reinforcement Learning: An Introduction (2nd Edition).  MIT press, Cambridge, MA.
  * Todorov et al. (2012) Todorov, E., Erez, T., Tassa, Y., 2012.  Mujoco: A physics engine for model-based control, in: Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, Algarve, Portugal. pp. 5026–5033.
  * Vaidya et al. (2018) Vaidya, S., Ambad, P., Bhosle, S., 2018.  Industry 4.0–a glimpse.  Procedia manufacturing 20, 233–238.
  * Visinsky et al. (1994) Visinsky, M.L., Cavallaro, J.R., Walker, I.D., 1994.  Robotic fault detection and fault tolerance: A survey.  Reliability Engineering & System Safety 46, 139–158.
  * Visinsky et al. (1991) Visinsky, M.L., Walker, I.D., Cavallaro, J.R., 1991.  Fault detection and fault tolerance in robotics, in: Proceedings of the 1991 NASA Space Operations, Applications, and Research Symposium, NASA, Houston, TX. pp. 262–271.
  * Xu and Zhu (2018) Xu, J., Zhu, Z., 2018.  Reinforced continual learning.  arXiv preprint arXiv:1805.12369 .
  * Zhang et al. (2020) Zhang, J., Cheung, B., Finn, C., Levine, S., Jayaraman, D., 2020.  Cautious adaptation for reinforcement learning in safety-critical settings, in: International Conference on Machine Learning, PMLR. pp. 11055–11065.
  * Zhang et al. (2017) Zhang, W., Peng, G., Li, C., Chen, Y., Zhang, Z., 2017.  A new deep learning model for fault diagnosis with good anti-noise and domain adaptation ability on raw vibration signals.  Sensors 17, 425.