#  Fault Identification Enhancement with Reinforcement Learning (FIERL)

Valentina Zaccaria*, Davide Sartor*, Simone Del Favero and Gian Antonio Susto
V. Zaccaria and D. Sartor equally contributed to this work. All authors are
with the Department of Information Engineering, University of Padua, Padua PD
35131, Italy. This study was partially carried out within the MICS (Made in
Italy – Circular and Sustainable) Extended Partnership and received funding
from Next-GenerationEU (Italian PNRR – M4 C2, Invest 1.3 – D.D. 1551.11-10-
2022, PE00000004).

###### Abstract

This letter presents a novel approach in the field of Active Fault Detection
(AFD), by explicitly separating the task into two parts: Passive Fault
Detection (PFD) and control input design. This formulation is very general,
and most existing AFD literature can be viewed through this lens. By
recognizing this separation, PFD methods can be leveraged to provide
components that make efficient use of the available information, while the
control input is designed in order to optimize the gathering of information.
The core contribution of this work is FIERL, a general simulation-based
approach for the design of such control strategies, using Constrained
Reinforcement Learning (CRL) to optimize the performance of arbitrary passive
detectors. The control policy is learned without the need of knowing the
passive detector inner workings, making FIERL broadly applicable. However, it
is especially useful when paired with the design of an efficient passive
component. Unlike most AFD approaches, FIERL can handle fairly complex
scenarios such as continuous sets of fault modes. The effectiveness of FIERL
is tested on a benchmark problem for actuator fault diagnosis, where FIERL is
shown to be fairly robust, being able to generalize to fault dynamics not seen
in training.

##  I Introduction

In automated systems, faults can arise for a wide variety of reasons,
resulting in disruptions of the smooth operation of the process and in safety
hazards. In this scenario, the need for reliable Fault Detection and
Identification (FDI) modules has arisen. Indeed, detecting faults at the
earliest stage enables the implementation of an appropriate recovery strategy,
preventing the propagation of faults and the potential subsequent failure of
the system, thereby avoiding severe safety, economic or environmental
consequences.

FDI methods can be broadly categorized into two main groups: Passive Fault
Detection (PFD) and Active Fault Detection (AFD), depending on whether or not
interactions with the system are possible. In PFD methods, process data are
collected and analyzed without interfering with the system and anomalous
system behaviour is identified by relying on nominal input-output relations,
which can be either derived from a model or learned from data. Model-based PFD
is extensively explored in the literature [1], including a variety of well-
established approaches, most notably residual- or observer-based strategies
[2]. Most recently, data-driven methods have also been proposed, achieving
successful outcomes in various fields [3, 4].
However, in some scenarios PFD may yield suboptimal diagnosis performances.
For instance, this can occur in the presence of incipient faults, that may be
masked by noise in the real system, or due to the effect of closed-loop
feedback controllers that tend to compensate, to some degree, for system
deviations caused by faults.

These issues can be mitigated with the use of AFD approaches. Such
methodologies involve the design of a minimally intrusive input, called
auxiliary input, that is added to the primary control action. This can improve
the diagnosis by gathering more informative data, but may result in a
degradation in the control performance [5]. This trade-off is better handled
by integrated Active Fault Detection and Control (AFDC) techniques that
jointly design the primary control and the auxiliary input, at the cost of
significantly increased complexity.

Both AFD and AFDC techniques commonly assume the set of fault modes to be
finite, in order to make the problem tractable. In this case, the goal is to
achieve separation of the different fault dynamics. This can be theoretically
guaranteed by assuming deterministic and bounded disturbances [6, 7, 8], but
it is possible to model more realistic processes, by considering stochastic
disturbances and moving to probabilistic objectives, like minimizing the
Bayesian risk of model misdiagnosis [9, 10], or maximizing a statistical
distance measure of the predicted output distributions of different fault
modes [11, 12, 13, 14]. The aforementioned formulations usually involve nested
optimization problems, whose solution via traditional methods require
significant simplifications or heavily restricting conditions, especially in
the case of AFDC. For this reason, recent studies have started to investigate
the use of Reinforcement Learning (RL) for solving these optimization
problems, with promising results [15, 16, 17, 18].

In this work, we introduce FIERL, a general framework for RL-based AFD. In
FIERL, the AFD (and AFDC) paradigm is reformulated by explicitly disentangling
two key tasks: the first is a PFD step, which handles the diagnosis itself,
the second is the input design, that aims to maximize the performance of the
PFD component. While most AFD techniques in the literature could be viewed
through this lens, when using RL this separation becomes particularly useful
to recognize. In this case the PFD component can be treated as a generic black
box, allowing for seamless integration of any PFD strategy into the
optimization loop. This not only makes FIERL broadly applicable, but also
allows it to tackle complex diagnostic scenarios (i.e. continuous spectrum of
faults). Moreover, FIERL can naturally be applied to integrated AFDC, which is
therefore the main focus of this work. However, the discussion could be easily
adapted to AFD without control.

This letter unfolds as follows: Section II covers the preliminary notions
required, Section III briefly describes the basics of constrained RL up to the
motivation and derivation of the policy optimization algorithm considered in
this work. Section IV describes in detail a general formulation of the
proposed strategy, while section V discusses an example application
considering the class of linear systems affected by actuator faults. Finally,
section VI provides some experimental results, evaluating the performance
improvements of using RL over a naive approach, followed by brief closing
remarks in VII, discussing in detail the advantages of the proposed approach
as well as its limitations.

Figure 1: FIERL: depiction of the offline policy training phase (left) and of
the control flow during deployment (right).

##  II Preliminaries

###  II-A System Modelling and Notation

Consider generic time-varying faults $z_{t}\in Z$ that affect the dynamics of
a discrete-time stochastic system:

$\displaystyle x_{t+1}$ $\displaystyle=g(z_{t},x_{t},u_{t},w_{t}),$ (1)
$\displaystyle y_{t}$ $\displaystyle=h(z_{t},x_{t},u_{t},v_{t}),$
$\displaystyle z_{t+1}$ $\displaystyle=l(z_{t},x_{t},u_{t}),$

where ${x_{t}\in\mathbb{R}^{n_{x}}}$, ${y_{t}\in\mathbb{R}^{n_{y}}}$ and
${u_{t}\in\mathbb{R}^{n_{u}}}$ represent respectively the system state, output
and input at time $t$,
${v_{t}\in\mathbb{R}^{n_{v}}\sim\mathcal{N}(\mu_{v},\Sigma_{v})}$ and
${w_{t}\in\mathbb{R}^{n_{w}}\sim\mathcal{N}(\mu_{w},\Sigma_{w})}$ represent
independent stationary Gaussian measurement and process noises, the functions
$g(\cdot)$ and $h(\cdot)$ describe, respectively, the system dynamics and
model output and notably depend on the fault $z_{t}$. The fault free mode is
given by a specific element ${z^{0}\in Z}$ and the fault evolution dynamics
are described by the stochastic function $l(\cdot)$.

In our example application, we consider linear systems with actuator faults,
where Eq. 1 is reduced to:

$\displaystyle x_{t+1}$
$\displaystyle=Ax_{t}+B\operatorname{diag}(z_{t})u_{t}+w_{t}$ (2)
$\displaystyle y_{t}$ $\displaystyle=Cx_{t}+v_{t}$ $\displaystyle z_{t+1}$
$\displaystyle=z_{t}+\xi_{t}$

where $A$, $B$ and $C$ are known real-valued matrices and each component of
$z_{t}\in\mathbb{R}^{n_{u}}$ models partial faults of the corresponding
actuator. Fault evolution is independent of $x_{t}$ and $u_{t}$, and is
characterized by the random variable $\xi_{t}$.

###  II-B Reinforcement Learning

In basic Reinforcement Learning (RL) formulation, an agent interacts with its
environment, trying to maximize some notion of cumulative reward. Basic
sequential decision-making problems of this kind are usually modelled as
Markov Decision Process (MDP). The structural components of an MDP are fully
represented by the tuple
${\left\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}\right\rangle}$,
where the state space $S$ and action space $A$ are arbitrary sets. The
environment state ${S_{t}\in\mathcal{S}}$ follows a discrete-time stochastic
process, defined by the state transition function
${\mathcal{P}\left(s,a,s^{\prime}\right)=\mathbb{P}\left[S_{t+1}=s^{\prime}\mid
S_{t}=s,A_{t}=a\right]}$ which depends on the action ${A_{t}\in\mathcal{A}}$
performed by the agent. Finally, the reward function
${\mathcal{R}(s,a,s^{\prime})}$ is used to measure the quality of a given
state transition. The behaviour of the agent is generally characterized by a
stochastic policy, a function
${\pi:\mathcal{S}\rightarrow\mathbf{P}(\mathcal{A})}$ that specifies the
probability that a given action will be selected given the environment state.
The realization of an action state-trajectory
${\tau=(s_{0},a_{0},s_{1},a_{1},...)}$ obtained by following policy $\pi$ is
commonly denoted by ${\tau\sim\pi}$. The return (cumulative reward) represents
the total reward (discounted by ${\gamma\in(0,1]}$) accumulated by the agent
over the trajectory $\tau$. The goal of RL is to find the optimal policy
amongst a set of admissible ones $\Pi$:

$\pi^{*}=\arg\max_{\pi\in\Pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(s_{t},a_{t},s_{t+1})\right].$
(3)

Many common algorithms also make use of the value function
$V^{\mathcal{R}}_{\pi}$, the action-value function $Q^{\mathcal{R}}_{\pi}$ and
the advantage function $A^{\mathcal{R}}_{\pi}$ defined as:

$\displaystyle
V^{\mathcal{R}}_{\pi}(s)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(s_{t},a_{t},s_{t+1})\mid
s_{0}=s\right],$ (4) $\displaystyle
Q^{\mathcal{R}}_{\pi}(s,a)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}(s_{t},a_{t},s_{t+1})\mid
s_{0}=s,a_{0}=a\right],$ $\displaystyle
A^{\mathcal{R}}_{\pi}(s,a)=Q_{\pi}(s,a)-V_{\pi}(s).$

##  III Constrained Reinforcement Learning

Constrained Reinforcement Learning (CRL) is the general problem of training an
RL agent under specific constraints, usually with the purpose of satisfying
them throughout both training and testing stages, ensuring that the agent
operates within predefined boundaries. CRL can also be used to handle
conflicting objectives, by framing one of the two as a constraint and
optimizing for the other.

The de-facto standard to characterize CRL is the Constrained Markov Decision
Process (CMDP) [19]. A CMDP extends a MDP including a set of $m$ cost
functions
${C_{i}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}}$.
Costs have the same structure as rewards, therefore one can define the
expected return of $C_{i}$, defining the expected constraint return (or
$C_{i}$-return):

$J_{C_{i}}(\tau)=\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}C_{i}(s_{t},a_{t},s_{t+1})\right].$
(5)

The optimal policy in a CMDP is obtained from the constrained optimization
problem:

$\pi^{*}=\underset{\begin{subarray}{c}\pi:J_{C_{i}}(\pi)\leq d_{i},\\\
i=1,\ldots,m\end{subarray}}{\arg\max}J_{\mathcal{R}}(\pi),$ (6)

where $d_{i}$ are hyper-parameters representing individual constraints. This
is equivalent to the optimal policy of the simple MDP, where the set of
feasible policies is restricted to:

$\Pi_{C}=\\{\pi:J_{C_{i}}(\pi)\leq d_{i},i=1,...,m\\}.$ (7)

###  III-A Constrained Policy Optimization

Constrained Policy Optimization (CPO) [20] is a general-purpose local policy
search algorithm for CRL. It has theoretical guarantees for near-constraint
satisfaction throughout all training and it has demonstrated to be effective
in learning neural network policies for complex control tasks with high-
dimensional constraints. One of the main advantages of CPO is that it
generally outperforms other methods like primal-dual optimization [21] on
enforcing constraints without significantly compromising performance with
respect to returns [20]. CPO shows to outperform fixed-penalty methods that
incorporate constraints directly in the reward function. CPO is a local policy
search method and as such, given a parametrization $\theta$ of the policy
space, it aims to find an approximation to the optimal policy by iterated
optimization of a surrogate objective $J(\pi)$ within a local region of radius
$\delta$ in the policy parameter space defined by some distance measure $D$:

$\displaystyle\pi_{\theta_{k+1}}=$ $\displaystyle\arg\max_{\pi\in\Pi}J(\pi).$
(8) s.t. $\displaystyle D(\pi,\pi_{\theta_{k}})\leq\delta$

When dealing with CMDP, the policy domain $\Pi$ is replaced by the set of
feasible policies $\Pi_{C}$. Unfortunately, assessing which policies lie in
$\Pi_{C}$ requires precise off-policy evaluation, making the local update
impractical. To overcome this issue, CPO uses a principled approximation of
the objective and the constraints that depends on the state distribution under
the current policy (denoted $s\sim\pi_{\theta_{k}}$) and can therefore be
estimated using on-policy samples, as follows

$\displaystyle\pi_{\theta_{k+1}}=$
$\displaystyle\arg\max_{\pi\in\Pi}\quad\mathbb{E}_{\begin{subarray}{c}s\sim\pi_{\theta_{k}}\\\
a\sim\pi_{\theta_{k}}\end{subarray}}\left[A^{\mathcal{R}}_{\pi_{\theta_{k}}}(s,a)\right].$
(9) s.t. $\displaystyle
J_{C_{i}}(\pi_{\theta_{k}})+\dfrac{1}{1-\gamma}\mathbb{E}_{\begin{subarray}{c}s\sim\pi_{\theta_{k}}\\\
a\sim\pi_{\theta_{k}}\end{subarray}}\left[A^{C_{i}}_{\pi_{\theta_{k}}}(s,a)\right]\leq
d_{i}$
$\displaystyle\mathbb{E}_{s\sim\pi_{k}}\left[D_{KL}(\pi_{\theta}(s)\|\pi_{\theta_{k}}(s))\right]\leq\delta$

The use of a constraint based on the Kullback-Leibler-divergence ${D}_{KL}$
[22] makes CPO a trust region method, allowing for theoretical guarantees on
monotonic performance improvement and approximate satisfaction of constraints.
These aspects are discussed in detail in [20]. Solving directly the CPO update
in Eq. 9 is intractable when dealing with thousands of parameters. An
efficient implementation relies on a linearization of both the objective and
the cost constraint around the current policy $\pi_{\theta_{k}}$ while the KL-
divergence is approximated by second-order expansion.

Due to all these approximations, the resulting iterates might be unfeasible
and therefore the implementation of a suitable recovery strategy is necessary.
All the technical implementation details are described in [20].

##  IV Fault Identification Enhancement with Reinforcement Learning

In this work the AFD paradigm is reinterpreted as the enhancement of PFD
approaches, by designing a feasible control strategy that optimizes the
passive component accuracy. Indeed, AFD is composed of two distinct tasks: the
selection of a control action, and the computation of the fault detector
output signal $z_{t}^{est}$. Deriving the optimal detector output, based on
the available information, is essentially a PFD task. This is why most AFD
(and by extension AFDC) techniques, which aim to optimize some metric related
to fault diagnosis accuracy, often employ nested optimization loops. For
instance, for many set-separation based approaches the inner optimization
performs template matching which can be thought of as an implicit PFD step.

In FIERL, we explicitly disentangles these two tasks, relying on existing
approaches for the PFD part, and focusing on the input design, using RL to
find a control strategy that improves the accuracy of the selected PFD module.
In particular, we propose the use of CRL to handle the conflicting objectives
of diagnosis accuracy and control quality. Instead of solving a dual-objective
optimization problem, we directly optimize for the accuracy of the diagnosis,
under some constraints on control performance. The rationale being that fixing
the desired control performance bounds via the hyper-parameters $d_{i}$ is
generally more intuitive than balancing the conflicting objectives directly.
From these considerations, the AFD problem can be formulated as a CMDP where:

  * •

The environment state should contain $x$, $z$ and the passive component
estimates $x^{est}_{t}$ and $z^{est}_{t}$ for system state and current fault.
In the case of AFDC, information related to the control task, like the
reference signal $y^{ref}_{t}$ or system output $y_{t}$, should also be
included.

  * •

The agent action $a_{t}$ can be either the control input $u_{t}$ in the case
of integrated AFDC, or an auxiliary control signal in the case of AFD without
control;

  * •

The reward function is defined in terms of metric of choice $\mathcal{M}$,
that expresses the instantaneous adherence of the fault estimate signal
$z^{est}_{t}$ to the true fault $z_{t}$.
${\mathcal{R}(s_{t},a_{t},s_{t+1})=\mathcal{M}(z^{est}_{t},z_{t})}$

  * •

The cost function is used to enforce the desired guarantees on control. In
this work we focus on enforcing adherence of the system output $y_{t}$ to a
reference signal $y_{t}^{ref}$, using a distance metric of choice
$\mathcal{D}$.
${\mathcal{C}(s_{t},a_{t},s_{t+1})=\mathcal{D}(y^{ref}_{t},y_{t})}$.

Therefore, the optimal control policy derived in 6, becomes:

$\displaystyle\pi_{\theta^{*}}=$
$\displaystyle\arg\max_{\pi_{\theta}}\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{M}(z^{est}_{t},z_{t})\right].$
(10)
$\displaystyle\text{s.t.}\quad\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t=0}^{\infty}\gamma_{c}^{t}\mathcal{D}(y_{t},y^{ref}_{t})\right]\leq
d$

Remark: the inclusion of the system state $x$ and the true fault $z$ is needed
since $s_{t}$ ought to be a sufficient statistic for the future evolution of
the environment. However, this information is not available to the agent.
Therefore, during both training and deployment phases, we operate with a
masked partial state $s^{\prime}_{t}$ excluding these elements. Fig. 1
illustrates both training and deployment setups.

##  V Actuator Faults in Linear Systems

###  V-A Passive Component Derivation

For systems modelled by Eq. 2, assuming $A$, $B$ and $u_{t}$ to be known, a
passive observer can be derived, which provides estimates for $x_{t}$ and
$z_{t}$ at time $\tau$ of the form:

$\displaystyle
x^{est}_{t,\tau}\sim\mathcal{N}\left(\mu_{x_{t}}^{(\tau)},\Sigma_{x_{t}}^{(\tau)}\right)$
(11) $\displaystyle
z^{est}_{t,\tau}\sim\mathcal{N}\left(\mu_{z_{t}}^{(\tau)},\Sigma_{z_{t}}^{(\tau)}\right).$

Update rules for distribution parameters are derived in closed form using
Bayesian inference. If omitted, assume $\tau=t$.
Consider $B^{*}_{t}=B\operatorname{diag}(u_{t})$ and Eq. 2 expressed as:

$x_{t+1}=Ax_{t}+B^{*}_{t}z_{t}+w_{t}.$ (12)

Direct substitution of the estimates at time $t$ in Eq. 12 yields:

$\displaystyle\mu_{x_{t+1}}^{(t)}$
$\displaystyle=A\,\mu_{x_{t}}^{(t)}+B^{*}_{t}\,\mu_{z_{{t}}}^{(t)}$ (13)
$\displaystyle\Sigma_{x_{t+1}}^{(t)}$
$\displaystyle=A\,\Sigma_{x_{t}}^{(t)}\,A^{T}+B^{*}_{t}\,\Sigma_{z_{t}}^{(t)}\,{B^{*}_{t}}^{T}+\Sigma_{w}.$

To account for $y_{t+1}$, the density term $p(x_{t+1}\mid y_{t+1})$ can be
directly evaluated applying Bayes’ theorem, resulting in:

$\displaystyle\Sigma_{x_{t+1}}^{(t+1)}$
$\displaystyle=\left((\Sigma_{x_{t+1}}^{(t)})^{-1}+C^{T}\Sigma^{-1}_{v}C\right)^{-1}$
(14) $\displaystyle\mu_{x_{t+1}}^{(t+1)}$
$\displaystyle=\Sigma_{x_{t+1}}^{t+1}\left((\Sigma^{(t)}_{x_{t+1}})^{-1}\mu_{x_{t+1}}^{(t)}+C^{T}\Sigma^{-1}_{v}\,{y}_{t+1}\right).$

Applying Woodbury matrix identity [23] and performing some algebraic
manipulations, a more numerically stable set of equations can be obtained:

$\displaystyle K_{x}(t)$
$\displaystyle=\Sigma_{x_{t+1}}^{(t)}C^{T}\left(\Sigma_{v}+C\Sigma_{x_{t+1}}^{(t)}C^{T}\right)^{-1}$
(15) $\displaystyle\Sigma_{x_{t+1}}$
$\displaystyle=\left(I-K_{x}(t)\;C\right)\Sigma_{x_{t+1}}^{(t)}$
$\displaystyle\mu_{x_{t+1}}$
$\displaystyle=\mu_{x_{t+1}}^{(t)}+K_{x}(t)\left({y}_{t+1}-C\mu_{x_{t+1}}^{(t)}\right).$

The procedure up to this point is equivalent to the derivation of a standard
Kalman filter, where the uncertainty in the fault vector $z_{t}$ adds to the
existing process noise.

Parameter update rules for the fault estimate can also be derived from Eq. 12.
Substituting best estimates for all terms that do not depend on $z_{t}$
yields:

$B^{*}_{t}\,z_{t}\sim
N\left(\mu_{x_{t+1}}^{(t+1)}-A\,\mu_{x_{t}}^{(t)}\,,\Sigma_{x_{t+1}}^{(t+1)}+A\Sigma_{x_{t}}^{(t)}A^{T}+\Sigma_{w}\right)$
(16)

This represents a way to measure $z_{t}$, analogously to how the term
$Cx_{t}\sim N({y}_{t}\,,\Sigma_{w})$ represents a measurement for $x_{t}$.
Therefore, using the previous results in Eq. 15 and substituting the correct
mean and variances:

$\displaystyle K_{z}(t)$
$\displaystyle=\Sigma_{z_{t}}^{(t)}{B^{*}_{t}}^{T}\left(\Sigma_{x_{t+1}}^{(t+1)}+\Sigma_{x_{t+1}}^{(t)}\right)^{-1}$
(17) $\displaystyle\Sigma_{z_{t}}^{(t+1)}$
$\displaystyle=(I-K_{z}(t)B^{*}_{t})\Sigma_{z_{t}}^{(t)}$
$\displaystyle\mu_{z_{t}}^{(t+1)}$
$\displaystyle=\mu_{z_{t}}^{(t)}+K_{z}(t)(\mu_{x_{t+1}}^{(t+1)}-\mu_{x_{t+1}}^{(t)}).$

Note that this improves the estimate for the parameter at time $t$, using
information obtained from $y_{t+1}$ at the following time step. To obtain
$z_{t+1}^{(t+1)}$, some assumptions on the fault evolution need to be made. In
the case of
$\xi_{t}\sim\mathcal{N}\left(\mu_{\xi_{t}},\Sigma_{\xi_{t}}\right)$ the
parameter updates become:

$\Sigma_{z_{t+1}}^{(t+1)}=\Sigma_{z_{t}}^{(t+1)}+\Sigma_{\xi_{t}},\;\mu_{z_{t+1}}^{(t+1)}=\mu_{z_{t}}^{(t+1)}+\mu_{\xi_{t}}.$
(18)

Notably, when $\mu_{\xi_{t}}=0$, the update reduces to adding a bias term to
the estimate covariance. This simple addition produces remarkably stable
filters that can perform reasonably even if the fault dynamics do not match
the assumptions (i.e. jump processes) as shown in VI.

###  V-B FIERL application

Since the considered passive component provides estimates in the form of
Gaussian distributions, the selected reward metric is the negative expected
squared norm of the fault estimation error, which admits the closed form:

$\displaystyle\mathcal{R}(s_{t},a_{t},s_{t+1})$
$\displaystyle=-\mathbb{E}_{z\sim z^{est}_{t}}\left(\lVert
z^{true}_{t}-z\rVert^{2}_{2}\right)$
$\displaystyle=-\operatorname{trace}(\Sigma_{z_{t}})-\rVert
z_{t}^{true}-\mu_{z_{t}}\rVert^{2}_{2}.$

The cost function counts the number of times the adherence to the reference
exceeds a desired tolerance $\Delta y^{max}$:

$C(s_{t},a_{t},s_{t+1})=\begin{cases}0&\text{if }\lVert
y_{t}-y^{ref}_{t}\rVert_{\infty}\leq\Delta y^{max}\\\ 1&\text{if }\lVert
y_{t}-y^{ref}_{t}\rVert_{\infty}>\Delta y^{max}\end{cases}.$ (19)

The full state vector incorporates the state and fault estimate distribution
parameters, the system output and reference:

$s^{\prime}_{t}=\begin{bmatrix}{\mu}_{x_{t}}&\operatorname{triu}({\Sigma}_{x_{t}})&{\mu}_{z_{t}}&\operatorname{triu}({\Sigma}_{z_{t}})&y^{ref}_{t}&y_{t}\end{bmatrix}.$

Therefore the optimal policy becomes:

$\displaystyle\pi_{\theta^{*}}=$
$\displaystyle\arg\max_{\pi_{\theta}}\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t=0}^{\infty}-\gamma^{t}\;\mathbb{E}_{z\sim
z^{est}_{t}}\left(\lVert z^{true}_{t}-z\rVert^{2}_{2}\right)\right].$ (20)
s.t.
$\displaystyle\mathbb{E}_{\tau\sim\pi_{\theta}}\left[\sum_{t=0}^{\infty}\gamma_{c}^{t}\;\mathbf{1}_{(\Delta
y^{max},\infty)}\left(\lVert
y_{t}-y^{ref}_{t}\rVert_{\infty}\right)\right]\leq d$

In this case the parameter $d$ represents the allowed tracking bound violation
rate. For $d=0$ it must be enforced at all times.

##  VI Case Study

Figure 2:  Experimental results for a random testing episode. The top two
panels show the evolution of fault estimate distributions for FIERL compared
to the naive approach. The fault estimate obtained with the RL policy promptly
approaches the true fault values after jumps, and generally it tends to be
more accurate, while also maintaining a tighter distribution.
Because of the passive component design, single bursty actions provide more
information on the fault mode than applying the equivalent control over a
longer period of time. This is in accordance with the RL policy behaviour,
depicted in the fourth panel, where a large action is followed by a sequence
of small negative ones in order to meet the tracking performance requirements
(shown in panel 3). Figure 3: Diagnosis performance as a function of tracking
threshold $\Delta y^{max}$. Each boxplot represents the average return
distribution in test episodes for FIERL with the corresponding control
requirements. The color-map shows the probability of violating the constraints
purely due drifts caused by process noise. As $\Delta y^{max}$ shrinks, the
diagnosis performance quickly degrades, and the training becomes unstable.
This is because the starting policy, no matter how conservatively it is
initialized, is almost certainly unfeasible, which is needed to ensure
convergence bounds [20].

In this section FIERL is applied to the three-tank benchmark system. The
system model, in the form of Eq. 2 is derived in the Appendix of [24]. The
system state represents the liquid levels in the tanks, and system matrices
are obtained through linearization around the equilibrium point
${x^{*}=[0.489,0.2332,0.3611]^{T}}$ meters discretized using a sampling time
of $T_{S}=0.1$ seconds. The covariance for i.i.d process and measurement noise
are, respectively, ${\Sigma_{w}=10^{-8}}$ and ${\Sigma_{v}=10^{-6}}$. The
control input ${u_{t}\in\mathbb{R}^{2}}$ is limited to ${U=[-0.002,0.02]}$.
The system output is the first and the second components of the state. This
choice is driven by the absence of direct control over the third component.

In order to assess its effectiveness, FIERL policy is compared to the naive
approach of a propoprtional controller where the control input is subject to
random perturbation ${\Delta u^{P}_{t}\sim\mathcal{U}([-K_{p},K_{p}])}$. The
gain $K_{p}$ is tuned in order to achieve the best diagnosis performance while
respecting the same tracking constraints of the RL policy. This simple
approach is surprisingly effective, guaranteeing reasonable control
performance thanks to the proportional controller, while acquiring
significantly more information on the fault state. In fact, if the domain of
the input were symmetric (as it was originally presented in [24]), the naive
approach would be a remarkably close approximation of the optimal policy,
making it difficult to appreciate the FIERL approach potential. This is the
primary reason for the modification.

The RL agent is trained for 1000 policy updates, each one performed using data
collected across 90 short episodes of length $N_{D}=40$ steps (4 seconds). At
each episode the true fault vector, which remains constant for the entire
duration, is sampled uniformly in ${[0,1]\times[0,1]}$, while the true initial
liquid level is sampled uniformly in the ball of radius $\Delta y^{max}=0.1$
centered in $x^{*}$. The cost constraints is set to ${d=6=0.15N_{D}}$. The
passive component is initialized with a wide uninformative prior
${z^{est}_{0}\sim\mathcal{N}((0.5,0.5),\operatorname{diag}(1,1))}$ and the
bias term of the estimated covariance is $\Sigma_{\omega_{t}}=10^{-3}$.

The testing setup consists of long episodes with a variable number of steps
ranging from 90 to 180 (instead of 40 used in training). The true fault
trajectory is assumed to change abruptly, re-sampling the fault vector at
random time steps, guaranteeing that each fault persists for at least 30
steps, allowing the module enough time to produce a reliable estimate. For the
RL policy, averaging over 10000 test episodes yields an expected return per
step of ${R_{RL}=(-1.458\pm 0.562)\cdot 10^{-2}}$, and an expected C-return
per step of ${C_{RL}=(1.178\pm 1.145)\cdot 10^{-1}}$. For the proportional
controller the expected returns are significantly lower at ${R_{P}=(-3.892\pm
1.477)\cdot 10^{-2}}$, ${C_{P}=(0.030\pm 2.18)\cdot 10^{-3}}$.

Finally Fig. 2 illustrates a single test episode in detail, while Fig. 3
describes the trade-off between tracking constraints and diagnosis
performance.

Code of the experiments is available at:
https://github.com/davidesartor/FIERL.

##  VII CONCLUSIONS

This work introduces a general application of CRL to enhance the performance
of passive diagnosis modules. The integration of FIERL with the application-
specific PFD component derived in section V-A highlights several important
features of the method. Most importantly, it does not require perfect
knowledge of the system state, relying on state estimates instead.
Additionally, unlike the majority of existing literature, this strategy can
effectively handle a continuous spectrum of faults or, more in general, any
stochastic fault dynamics that can be effectively simulated. Furthermore,
FIERL can, in principle, be directly applied to non-linear time varying
systems. This might result in more complicated PFD components, but would not
affect the RL loop. Finally, FIERL is robust to unseen fault dynamics, as
shown in section VI where, even by training on stationary faults only,
experimental results prove its effectiveness in handling transient faults.
This is particularly important when considering the possibility of continuous
monitoring.

Conversely, RL is computationally intensive, demanding accurate simulations
(and accurate system models) to ensure the correct dynamics are learned.
Moreover, determining when convergence has been achieved throughout the entire
environment state domain is not trivial nor is it possible to ensure full
convergence is always reached. Theoretical guarantees can be obtained by
assuming perfect knowledge of the value function, which is unrealistic for
practical applications [25]. Lastly, a noteworthy observation pertains to the
cost function. The one used in VI aims to limit the average number of
constraint violations to a maximum of $d$ times. However, this is only
enforced on average across multiple episodes. This might result in a objective
miss-specification, since the optimal policy might significantly over-violate
the constraint in certain domain regions, offset by others where the violation
rate is almost zero.

Directions for further investigation include the replacement of the indicator
function with a cost that discourages multiple constraint violation in the
same episode and testing on more challenging scenarios such as non-linear or
over-actuated system, and to more generic fault dynamics.

## References

  * [1] S. X. Ding, “Fault diagnosis in feedback control systems and fault-tolerant architecture,” _Model-Based Fault Diagnosis Techniques: Design Schemes, Algorithms and Tools_ , pp. 471–489, 2013.
  * [2] R. J. Patton and J. Chen, “Observer-based fault detection and isolation: Robustness and applications,” _Control Engineering Practice_ , vol. 5, no. 5, pp. 671–682, 1997.
  * [3] N. Md Nor, C. R. Che Hassan, and M. A. Hussain, “A review of data-driven fault detection and diagnosis methods: Applications in chemical process systems,” _Reviews in Chemical Engineering_ , vol. 36, no. 4, pp. 513–553, 2020.
  * [4] D. Gonzalez-Jimenez, J. Del-Olmo, J. Poza, F. Garramiola, and P. Madina, “Data-driven fault diagnosis for electric drives: A review,” _Sensors_ , vol. 21, no. 12, p. 4024, 2021.
  * [5] M. Forouzanfar and M. J. Khosrowjerdi, “A constrained optimization approach to integrated active fault detection and control,” _Iranian Journal of Science and Technology, Transactions of Electrical Engineering_ , vol. 41, pp. 229–240, 2017.
  * [6] G. R. Marseglia and D. M. Raimondo, “Active fault diagnosis: A multi-parametric approach,” _Automatica_ , vol. 79, pp. 223–230, 2017.
  * [7] F. Xu, “Observer-based asymptotic active fault diagnosis: A two-layer optimization framework,” _Automatica_ , vol. 128, 2021.
  * [8] J. Tan, S. Olaru, M. M. Seron, and F. Xu, “Set-based guaranteed active fault diagnosis for lpv systems with unknown bounded uncertainties,” _Automatica_ , vol. 128, p. 109602, 2021.
  * [9] J. Paulson, M. Martin-Casas, and A. Mesbah, “Input design for online fault diagnosis of nonlinear systems with stochastic uncertainty,” _Industrial & Engineering Chemistry Research_, 2017.
  * [10] I. Punčochář, J. Široký, and M. Šimandl, “Constrained active fault detection and control,” _IEEE Trans. on Automatic Control_ , 2015.
  * [11] J. A. Paulson, T. A. N. Heirung, R. D. Braatz, and A. Mesbah, “Closed-loop active fault diagnosis for stochastic linear systems,” in _2018 Annual American Control Conference (ACC)_ , 2018.
  * [12] T. A. N. Heirung and A. Mesbah, “Stochastic nonlinear model predictive control with active model discrimination: A closed-loop fault diagnosis application,” _IFAC-PapersOnLine_ , 2017.
  * [13] F. Xu, S. Olaru, V. Puig, C. Ocampo-Martinez, and S.-I. Niculescu, “Sensor-fault tolerance using robust mpc with set-based state estimation and active fault isolation,” _International Journal of Robust and Nonlinear Control_ , vol. 27, no. 8, pp. 1260–1283, 2017.
  * [14] J. Tan, F. Xu, Y. Wang, and B. Liang, “Input design for integrated active fault diagnosis and output tracking control,” _Automatica_ , 2022.
  * [15] F. Jia and X. He, “Auxiliary input design for active fault detection via deep reinforcement learning,” in _CAA SAFEPROCESS_ , 2021.
  * [16] L. Král and I. Punčochář, “Policy search for active fault diagnosis with partially observable state,” _International Journal of Adaptive Control and Signal Processing_ , vol. 36, no. 9, pp. 2190–2216, 2022.
  * [17] Z. Yan, F. Xu, J. Tan, H. Liu, and B. Liang, “Reinforcement learning-based integrated active fault diagnosis and tracking control,” _ISA transactions_ , vol. 132, pp. 364–376, 2023.
  * [18] J. Škach, I. Punčochář, and F. L. Lewis, “Optimal active fault diagnosis by temporal-difference learning,” in _IEEE CDC_ , 2016.
  * [19] E. Altman, _Constrained Markov Decision Processes_. Chapman and Hall, 1999.
  * [20] J. Achiam, D. Held, A. Tamar, and P. Abbeel, “Constrained policy optimization,” in _ICML_ , 2017, pp. 22–31.
  * [21] Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone, “Risk-constrained reinforcement learning with percentile risk criteria,” _The Journal of Machine Learning Res._ , vol. 18(1), pp. 6070–6120, 2017.
  * [22] S. Kullback and R. A. Leibler, “On information and sufficiency,” _The annals of mathematical statistics_ , vol. 22, no. 1, pp. 79–86, 1951.
  * [23] M. A. Woodbury, _Inverting modified matrices_. Department of Statistics, Princeton University, 1950.
  * [24] Z. Zhang and X. He, “Fault-structure-based active fault diagnosis: A geometric observer approach,” _Energies_ , vol. 13(17), p. 4475, 2020.
  * [25] G. Neu, A. Jonsson, and V. Gómez, “A unified view of entropy-regularized markov decision processes,” _ArXiv_ , 2017.

  *[FIERL]: Fault Identification Enhancement with Reinforcement Learning
  *[AFD]: Active Fault Detection
  *[PFD]: Passive Fault Detection
  *[CRL]: Constrained Reinforcement Learning
  *[FDI]: Fault Detection and Identification
  *[AFDC]: Active Fault Detection and Control
  *[RL]: Reinforcement Learning
  *[MDP]: Markov Decision Process
  *[CMDP]: Constrained Markov Decision Process
  *[CPO]: Constrained Policy Optimization