# Application of twin delayed deep deterministic policy gradient learning for
the control of transesterification process

###### Abstract

Persistent depletion of fossil fuels has encouraged mankind to look for
alternatives fuels that are renewable and environment-friendly. One of the
promising and renewable alternatives to fossil fuels is bio-diesel produced by
means of batch transesterification process. Control of batch
transesterification process is difficult due to its complex and non-linear
dynamics. It is expected that some of these challenges can be addressed by
developing control strategies which directly interact with the process and
learning from the experiences. To achieve the same, this study explores the
feasibility of reinforcement learning (RL) based control of batch
transesterification process. In particular, the present study exploits the
application of twin delayed deep deterministic policy gradient (TD3) based RL
for the continuous control of batch transesterification process.This results
showcase that TD3 based controller is able to control batch
transesterification process and can be promising direction towards the goal of
artificial intelligence based control in process industries.

Tanuja Joshi, Shikhar Makker, Hariprasad Kodamana,
Department of Chemical Engineering,
Indian Institute of Technology Delhi,
Hauz Khas, New Delhi.
<EMAIL_ADDRESS>

Harikumar Kandnath,
International Institute of Information Technology,
Hyderabad.
<EMAIL_ADDRESS>

_K_ eywords Reinforcement learning $\cdot$ deep Q-learning $\cdot$ deep
deterministic policy gradient $\cdot$ batch transesterification

##  1 Introduction

Gradual depletion of fossil fuels over the past has mandated the need for
alternative eco-friendly fuels owing to environmental sustainability and low
carbon emissions. One of the promising and renewable alternatives to diesel-
fuel that has been gaining much attention is the bio-diesel. It is obtained
from various plant-based sources such as vegetable oil, soyabean oil, palm
oil, waste cooking oil, animal fats, etc. [1, 2]. The process of the
production of bio-diesel, transesterification, wherein triglycerides from the
fatty acids react with alcohols in the presence of a catalyst to produce
glycerol, and methyl esters [3]. Industrial-scale production of bio-diesel is
carried out in both continuous stirred tank reactors and batch reactors.
However, studies have emphasized the usefulness of batch process because of
various advantages like low capital cost, raw material cost and flexibility in
operation [4].

The operation of the transesterification process grapples with several
challenges due to the unique characteristics of the batch process, such as
non-linearity due to the temperature-dependent kinetics, time-varying dynamics
and a broad range of operating conditions. Therefore, the control and
optimisation of the batch transesterification process is a very challenging
task. Reaction kinetics suggests that biodiesel production is affected mainly
by four factors: reaction time, the molar ratio, catalyst, and the reaction
temperature [5]. The reaction temperature has a significant effect on the
biodiesel yield and is; therefore, the primary control variable in the
existing literature on the control of the batch process [6, 7, 8]. Hence, one
of the primary goal of the control of transesterification is to track the
desired the temperature trajectory, optimally. In the end, the yield of the
product depends mostly on the choice of the controller and its capability and
the tuning parameters. Even if the controller is appropriately tuned for an
operating condition, the drifts in process parameters can significantly
deteriorate the controller performance over time.

To overcome some of these issues, advanced control strategies such as model
predictive control (MPC), iterative learning control (ILC), non-linear MPC
(NMPC), are presented in the literature for the control of
transesterification. [9, 10, 11, 12, 13, 14, 15, 7, 16, 17]. The performance
of a model-based controller is heavily dependent on the accuracy of the
underlying model of the process. Even the slightest inaccuracies in the model
will result in plant-model mismatch and, consequently, inaccurate prediction
of the concentration profiles of the species from the model. Further,
optimization-based control algorithms are computationally demanding as the
computation of the optimal input sequence involve on-line optimization at each
time step. Despite the advances in numerical methods and computational
hardware, this is still a challenging task for complex non-linear, high-
dimensional dynamical systems. Hence, there is much incentive if a control
strategy can directly interact with process trajectory and provide a control
solution for on-line course correction.

To this extent, it is useful to explore the feasibility of reinforcement
learning (RL) as a potential paradigm for the control of transesterification
process. Contrary to classical controllers, RL based controllers do not
require a process model or control law; instead, they learn the dynamics of
the process by directly interacting with the operational environment [18, 19].
As a result, the controller performance will not be contingent on having a
pressing requirement of a high fidelity model of the process. Further, as
opposed to the traditional controllers, the RL based controller learns from
experiences and past history and thus improving the control policy at every
step. As an earlier proposition in this direction, approximate dynamic
programming(ADP) approach that learns the optimal ’cost-to-go’ function has
been proposed for the optimal control of non-linear systems [20, 21, 22].
Following this, an ADP based ’Q-learning’ approach that learns the optimal
policy using value iteration in a model-less manner has also been proposed
[20]. Unlike the traditional applications of ‘Q-learning’, the state-space and
action-space is continuous for process control applications. This poses a
major limitation to the application of conventional Q-learning methods in this
context. For systems with continuous state-space, Mhin et al. combined
Q-learning with deep neural network for function approximation and developed
deep Q network (DQN) [23]. Inspired from these, some of the recent works
related to RL applications in chemical processes have attempted to optimize
the control policy by applying Q-learning and deep Q- learning for
applications such as chromatography and polymerization [24, 25].

Although the advent of DQN based RL was a major break-through in applications
to systems with continuous state-space, their utility to systems with
continuous action-space was still limited. Another approach to solving the RL
problem is by employing policy gradient (PG), where instead of evaluating the
value functions to find an optimal policy, the optimal policy is evaluated
directly. This approach is particularly well-suited to deal with problems
where both the state space and the action space are continuous. For instance,
Petsagkourakis et al. applied the PG algorithm to find the optimal policy for
a batch bioprocess by using principles of transfer learning [26]. However, PG
methods suffer from the drawback of noisy gradients due to high variance
leading to slow convergence. Actor-critic Methods reduces this low variance
gradient estimates by exploiting a critic network and has been the widely used
framework for dealing with continuous action spaces. DDPG is one of the actor-
critic algorithms that has celebrated a huge success [27]. Indeed, DDPG was
the first efficient algorithm used to solve high dimensional continuous
control tasks. It effectively combines the architecture of DQN and
deterministic policy gradient (DPG). Recent works have shown the application
of DDPG algorithm for chemical process control as well. For example, Maet al.
have proposed a DDPG based controller for the control of semi-batch
polymerization [28], Spielberg et al. has applied the DDPG algorithm for SISO
and MIMO linear systems [29, 30]. For a detailed review of the application of
RL for chemical control problems, the readers are referred to [31, 19, 32, 33]

Fujimoto et al. [26] have shown that the DDPG algorithm suffers from a
critical issue of overestimation of network bias due to function approximation
error and leads to sub-optimal policy. The authors have also provided an
approach to address the function approximation error in actor-critic methods
and termed the new algorithm as Twin Delayed Deep Deterministic Policy
Gradient (TD3). Some recent work involving the application of TD3 algorithm
involves motion planning of robot manipulators, half- cheetah robot as an
intelligent agent to run across a field, etc. [27, 28]. Inspiring from these,
this paper presents a comparative simulation study of the two actor-critic
methods, namely TD3 and DDPG, for the optimal control of reactor temperature
in the batch transesterification processes. Specifically, we propose a control
strategy based on TD3, for the control of reactor temperature in batch
Transesterification processes. We also propose novel reward functions
structure analogous to PI and PID controllers to aid the agent in better
learning. We have also compared the results of actor-critic methods with RL
algorithms with discrete action spaces. To the best our knowledge (i) TD3
algorithm has not been applied for the control of the chemical process, and
(ii) such a comprehensive study involving the comparison of RL based
algorithms with continuous and discrete action spaces has also not been
reported in literature in the context of the process control. The analysis
show the advantages of the TD3 algorithm in terms of tracking performance,
control effort and computational cost when compared to the well known DDPG
algorithm in controlling transesterification process.

The rest of the paper is divided into the following sections. Section 2
presents the background of RL useful for developing a TD3 based controller.
Section 3 explain the TD3 algorithm for the control of transesterification
process. Section 4 discusses the methodology adopted for implementing our
algorithm. Section 5 shows the application of the proposed controller and
important results for the control of batch-transesterification process with
continuous and discrete action spaces. Section 6 draws conclusive remarks from
the study.

##  2 Background and motivation

###  2.1 Q- learning and deep Q-learning

A standard RL framework consists of three main components, namely the agent,
the environment ($E$) and the reward ($r$) [29]. At each time step $t$, the
agent performs an action $a_{t}$ based on the state $s_{t}$ of the environment
and receives a scalar reward $r_{t}$, as a result, the environment moves to a
new state $s_{t+1}$. The objective of the RL problem is to find the sequence
of control actions $\mathcal{A}:=\\{a_{0},\,a_{1},\,a_{2},\dots\\}$ to
maximise the expected discounted reward as given below:

$\arg\underset{\mathcal{A}}{\max}\,\,\,\mathbb{E}~{}[R_{t}:=\sum\limits_{k=0}^{\infty}\gamma^{k}{r_{t+k}}]$
(1)

where $0<\,\gamma\,<1$ is the discount factor and $\mathbb{E}$ denotes the
expectation operator which is applied to the discounted reward due to the
stochastic nature of the process dynamics.However, explicit solution of Eq.(1)
is tedious to obtain. Q-learning is an iterative algorithm to solve the RL
problem over a finite set of actions. The Q-value of a state-action pair,
$Q_{\pi}(s_{t},a_{t})$ is the expected return after performing an action
$a_{t}$ at a state $s_{t}$ following a policy
$\pi:\mathcal{S}\rightarrow\mathcal{A}$ :

$Q_{\pi}(s_{t},a_{t})=\mathbb{E}_{\pi}\Big{[}\sum\limits_{k=0}^{\infty}\gamma^{k}{r_{t+k}}|s_{t}=s,a_{t}=a\Big{]}$
(2)

where $\mathcal{S}=\\{s_{0},s_{1},s_{2},\dots\\}$. The objective of the
Q-learning is to find the optimal policy ($\pi^{*}$) by learning the optimal
Q-value, $Q^{*}(s_{t},a_{t})$ which is the maximum expected return achievable
by any policy for a state-action pair. The optimal Q-value,
$Q^{*}(s_{t},a_{t})$ must satisfy the Bellman optimality equation [30] given
as :

$Q^{*}(s_{t},a_{t})=\mathbb{E}[r_{t}+\gamma\max_{a_{t+1}}Q^{*}(s_{t+1},a_{t+1}|s_{t}=s,a_{t}=a)]$
(3)

where $s_{t+1}$ and $a_{t+1}$ are the state and action at the next time step.
The Q-learning algorithm iteratively update the Q-value for each state-action
pair until the Q-function converges to the optimal Q-function. This is known
as value iteration and is given as:

$Q(s_{t},a_{t})\leftarrow
Q(s_{t},a_{t})+\alpha(r_{t}+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1})-Q(s_{t},a_{t}))$
(4)

Major encumbrances of traditional Q-learning are: (i) its application is
limited only to the problems with discrete state and actions spaces; (ii) the
computational difficulty faced while dealing with large state space owing to
the large size of the Q-matrix. The former problem can be circumvented by
employing a function approximator for modelling the relation between Q-value
and state-action pairs. Deep Q-learning [23] is an RL framework wherein a Deep
neural network (DNN) is used as function for approximating the optimal
Q-values. Since the goal is to find an optimal policy via learning an optimal
Q-value, the DNN is termed as the policy network. A separate target network is
used to estimate the target Q-value for the policy network. The weights of the
target network are updated periodically with the updated weights of the policy
network to make the learning stable. To address the problem of correlated
sequences, DQN uses a replay buffer or experience replay memory which has a
pre-defined capacity where all the past experiences are stored as the
following transition tuple
($s:=s_{t},a:=a_{t},s^{\prime}:=s_{t+1},a^{\prime}:=a_{t+1})$). The DQN uses
past experiences to train the policy network by selecting suitable mini-
batches from the replay buffer. The state is given as the input to the policy
network and the the network outputs the Q-value corresponding to all possible
actions in the action space. The loss is calculated as the mean square error
(MSE) between the current Q-value and the target Q-value as given in Eq. (5):

$\displaystyle Loss$
$\displaystyle=\mathbb{E}\big{[}\big{(}Q^{*}(s,a)-Q(s,a)\big{)}^{2}\big{]}$
(5)
$\displaystyle=\mathbb{E}\big{[}\big{(}r+\gamma\max_{a^{\prime}}Q_{\phi,T}(s^{\prime},a^{\prime})-Q_{\phi}(s,a)\big{)}^{2}\big{]}$
(6)

where $\phi$ represents the parameters of the network.

###  2.2 Actor-Critic Algorithms for RL

The applications of RL algorithms such as Q-learning and DQN is limited only
to problems with discrete action spaces. Policy-based methods provides an
alternative solution for continuous stochastic environments by directly
optimizing the policy by taking the gradient of the objective function with
respect to the policy parameter $\theta$:

$\displaystyle\nabla_{\theta}(J(\pi_{\theta}))=\displaystyle\nabla_{\theta}\big{(}\mathop{\mathbb{E}}_{\tau\sim\pi_{\theta}}[R(\tau)]\big{)}=\mathop{\mathbb{E}}_{\tau\sim\pi_{\theta}}\left[\Big{(}\sum_{t=0}^{T}\nabla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\Big{)}R(\tau)\right]$
(7)

where $\pi_{\theta}$ is the stochastic parameterized policy and $R(\tau)$ is
the return obtained from the trajectory
$\tau=\\{s_{0},a_{0},s_{1},a_{1},\dots\\}$. The architecture of the actor-
critic algorithms is based on policy-gradient, making them amenable for
continuous action spaces [31]. Policy-based (actor) methods suffer from the
drawback of high-variance estimates of the gradient and lead to slow learning.
The value-based (critic) methods are an indirect method for optimizing the
policy by optimizing the value function. Actor-critic algorithms combine the
advantages of both actor-only (policy-based) and critic-only (value-based)
methods and learn optimal estimate of both policy and value function. In the
actor-critic methods, policy dictates the action based on the current state,
and the critic evaluates the action taken by the actor based on the value
function estimate. The parameterized policy is then updated using the value
function using the gradient ascent for improving the performance.Deterministic
policy gradient (DPG) proposed by Silver et al. is an actor-critic off-policy
algorithm use for continuous action spaces [32]. It uses the expected gradient
of the Q-value function to evaluate the gradient of the objective function
with respect to parameter $\theta$ to find the optimal policy as given below:

$\displaystyle\nabla_{\theta}J(\mu_{\theta})=\displaystyle\mathop{\mathbb{E}}[\nabla_{\theta}Q_{\mu}(s,a)|_{a=\mu_{\theta}(s)}]$
(8)

where $\mu$ is the deterministic policy. However, they have only investigated
the performance using linear function approximators to evaluate an unbiased
estimate of the Q-value. DPG algorithm is further extended by Lillicrap et al.
to Deep deterministic policy gradient (DDPG) [33] by employing DQN as a non-
linear function approximator for the estimation of Q-values. DDPG incorporate
the merits of experience replay buffer and target networks to learn stable and
robust Q-values. In the actor part of DDPG architecture, rather than directly
copying the weights of the policy network in the target network, the target
network weights are allowed to slowly track the policy network weights to
improve the stability.The critic part of the DDPG uses a regular DQN to find
the estimate of Q-value by minimising the loss function. Both the value-based
method and actor-critic methods suffer from the problem of overestimation
bias. This problem comes due to the maximisation term in the target action-
value function of the loss function. Since the agent is unaware of the
environment in the beginning, it needs to first estimate $Q(s,a)$ and then
update them further for learning an optimal policy. Since the estimates of
$Q(s,a)$ are likely to be very noisy, evaluating the maxima over value
function does not guarantee the selection of the optimal action. If the target
itself is prone to error, then the value estimates are overestimated, and this
bias is then propagated further through the Bellman equation during the
update. Double Q-learning or Double DQN [34] was an attempt to provide
solution to this problem in the value-based framework, where actions are
discrete, by separating the action selection step and the estimation of the
value of the action step. For example, Double DQN proposed by Hasselt et al.
[35] uses two estimates of the Q-value: firstly, an online network is used for
the selection of action that gives the maximum Q-value, and, secondly, a
target network estimates the Q-value based on this action. Fujimoto et al.
[26] have proved that this overestimation-bias problem also exists in an
actor-critic setting which leads to the selection of sub-optimal actions,
resulting in poor policy updates. They have addressed this problem by
introducing a variant of DQN in the actor-critic framework; the twin delayed
deep deterministic policy gradient (TD3).

TD3 algorithm is an extension of the DDPG algorithm with the following
modifications to address some of the lacunae of DDPG. (i) To address the
overestimation bias problem, the concept of clipped double Q-learning is used
wherein two Q-values are learned, and the minimum of them is used for the
approximation of the target Q-value. Thus, TD3 has two critic networks and
corresponding critic target networks reflecting the ’twin’ term in its name.
(ii) To reduce the high variance and noisy gradients while minimising the
error per update, target networks are used to reduce error propagation by
delaying the policy update until the convergence of the Q-value. This results
in a less frequent policy network updates than the critic network updates.
(iii) To reduce the variance in the target action values, target policy
smoothing is performed by regularisation technique where clipped noise is
added to the target action obtained from the policy.

##  3 TD3 for transesterifcaton - the methodology

The TD3 algorithm is applied for the control of Batch transesterification
process to achieve the desired reactor temperature ($T_{ref}$). The
mathematical model of the process is assumed as the environment. The model
includes the kinetic model and the energy balance equations and are adapted
from [7, 6]. The details of the model and the kinetic equations are omitted
here for brevity. We propose a two-stage framework for developing a TD3 based
controller for the control of batch transesterification process as shown in
Fig. 1. The learning of the agent (controller) includes both offline learning
and online learning steps. The agent performs the offline learning by
interacting with the process model of the environment.

We propose to use a concept similar to that of transfer learning [36] to adapt
the trained actor and the critic networks from offline learning step in the
online learning stage, enabling a warm startup. This is expected to reduce the
number of episodes required by the agent in the online stage to achieve
convergence, making it suitable to apply in real-time. The online learning
starts with the trained actor and critic model and data obtained during the
offline learning stage serves as the historical data for the agent in online
learning. The historical data contains tuples of state ($s$), action ($a$),
reward ($r$), next state ($s^{\prime}$) which is used as the replay memory.
When TD3 agent works in a closed-loop fashion, the actor model receives the
initial state, $s$ from the plant and outputs the action, $a$. This action is
then injected into the plant, which then reaches a new state, $s^{\prime}$ and
outputs the reward, $r$ for taking the selected action. We get a transition
tuple, and we add it in the experience replay memory ($E2$). Training of the
actor and the critic networks happens as per TD3 algorithm steps. After the
training happens, the actor-network outputs the action $a^{\prime}$ for the
new state $s^{\prime}$, which is again injected into the plant. These steps
take place in an iterative fashion until the first batch run $(b_{i})$
completes. The updated model obtained for the $b_{i}$ batch is used for the
initialisation of networks at the start time $(t_{start})$ of the $b_{i+1}$
batch run. The whole process repeats for multiple batch runs. In this way, the
agent is able to learn from the environment and thus achieve convergence.
Figure 1 shows the schematic of the framework of the TD3 based controller for
the control of batch transesterification process. The following subsections
contain elaborate discussions on the reward selection procedure, offline
learning, and online learning for batch transesterification process.

###  3.1 Reward function

Reward function is an imperative constituent in the RL as the agent learns in
the direction of increasing the reward.We have considered two reward functions
inspiring the PI and PID control laws functional structure as illustrated
below to accommodate the time varying nature of the error profiles:

  1. 1.

Reward Function 1 (PI)

$\displaystyle
r(t)=\begin{cases}&\big{(}c_{1}^{\Romannum{1}}+c_{2}^{\Romannum{1}}g(f(e(t)))\\\
&+c_{3}^{\Romannum{1}}g(\sum_{t=0}^{(k-1)}f(e(t)))\big{)}~{}~{}~{}~{}\text{if
${f(e(t))<thr_{1}}$}\\\
&c_{4}^{\Romannum{1}}+c_{5}^{\Romannum{1}}g(\sum_{t=0}^{(k-1)}f(e(t)))~{}~{}~{}~{}\text{otherwise}\end{cases}$
(9)

In Eq. (9), $c_{1}^{\Romannum{1}},\dots,c_{5}^{\Romannum{1}}$ are suitable
tuning parameters, $f(.)$ and $g(.)$ are suitable function of error
$e(t):x(t)-x_{ref}\in\mathbb{R}^{n}$, and $thr_{1}$ is a suitably chosen
threshold parameter. For instance, the penalty function $f(.)$ may take
structural forms like $||e(t)||_{w,p}$, $w$ weighted $p$ norm of the vector
space of $e$, while $g$ is a suitable operator that converts penalty to
reward. In this reward formulation, instead of calculating the reward only
based on the current error $e(t)$, we have considered the effect of historical
error profiles also. The idea is basically to minimise the error that is being
accumulated over time in the past. Since this is inspired by the principle of
a proportional-integral (PI) controller, so we are naming it as PI Reward
Function.

  2. 2.

Reward Function 2 (PID)

$\displaystyle
r(t)=\begin{cases}&\big{(}c_{1}^{\Romannum{2}}+c_{2}^{\Romannum{2}}g(f(e(t))+c_{3}^{\Romannum{2}}g(\sum_{t=0}^{(k-1)}f(e(t))\\\
&+c_{4}^{\Romannum{2}}g(c_{5}^{\Romannum{2}}+f(\Delta
e(t))\big{)}~{}~{}~{}~{}\text{if $f(e(t))<thr_{2}$}\\\
&c_{6}^{\Romannum{2}}+c_{7}^{\Romannum{2}}(\sum_{t=0}^{(k-1)}f(e(t))\\\
&+c_{8}^{\Romannum{2}}g(c_{9}^{\Romannum{2}}+f(\Delta
e(t)))~{}~{}~{}~{}otherwise\end{cases}$ (10)

Here, in Eq. (10), there is an additional term compared to Eq. (9) $\Delta
e(t):=e(t)-e(t-1)$. This particular term helps to bring in the information
regarding the rate of change of error profile in the reward calculation. The
parameters $c_{1}^{\Romannum{2}},\dots,c_{9}^{\Romannum{2}}$ and the threshold
parameter $thr_{2}$ are to be tuned appropriately. Hence, we call this a
proportional integral derivative (PID) reward function. In the next
subsection, we proceed to illustrate simulation steps required for the
implementation of the TD3 based controller.

Figure 1: Schematic of the TD3 based controller for batch transesterification

###  3.2 TD3 : offline learning

The detailed steps for offline learning of TD3 algorithm employed in our study
are given below:

  1. 1.

Build the actor network and the corresponding ‘target actor network’ with
parameters $\phi_{\mathbf{A}_{0}}$ and $\phi_{\mathbf{A}_{0},\mathbf{T}}$,
where $\mathbf{T}$, $\mathbf{A}$ denote the target network and actor network,
respectively. Initialize the target network as
$\phi_{\mathbf{A}_{0},\mathbf{T}}$ $\rightarrow$ $\phi_{\mathbf{A}_{0}}$.

  2. 2.

Build two critic networks and initialise them with parameters
$\phi_{\mathbf{C}_{1_{0}}}$ and $\phi_{\mathbf{C}_{2_{0}}}$ and the
corresponding ’target critic networks’ with parameters
$\phi_{\mathbf{C}_{1_{0}},\mathbf{T}}$ and
$\phi_{\mathbf{C}_{2_{0}},\mathbf{T}}$, where $\mathbf{C}$ denotes the critic
network. Initialize the target network network as
$\phi_{\mathbf{C}_{1_{0}},\mathbf{T}}$ $\rightarrow$
$\phi_{\mathbf{C}_{1_{0}}}$ and $\phi_{\mathbf{C}_{2_{0}},\mathbf{T}}$
$\rightarrow$ $\phi_{\mathbf{C}_{2_{0}}}$.

  3. 3.

Initialise the experience replay buffer ($E1$) set with a defined cardinally,
say $M$.

  4. 4.

Observe the initial state, $s$ and select action $a$ from the actor network
with noise added to the action .

$a=clip(\mu_{\phi_{\mathbf{A}_{0}}}(s)+\epsilon,a_{min},a_{max})$ (11)

where $a_{min}$ and $a_{max}$ represent the upper and lower bounds of action,
respectively, $\mu_{\phi_{\mathbf{A}_{0}}}$ is the parametrized deterministic
policy and $\epsilon\sim\mathcal{N}(0,\,\sigma^{2})$, with a suitably chosen
$\sigma$.

  5. 5.

Execute the action, $a$ by injecting it into the plant model and obtain reward
$r:=r(t)$ and new state $s^{\prime}$.

  6. 6.

Add the obtained tuple of state, action, reward ,next state
$(s,a,r:=r(t),s^{\prime})$ in the replay buffer ($E1$).

  7. 7.

Train both actor and critic according to Step 6 -Step 13 as detailed in the
online learning algorithm (section 3.3) for $n$ desired episodes until the
convergence.

  8. 8.

Save the updated actor and critics models after the training completed. Let
the updated actor network parameters and the corresponding $\mathbf{T}$
parameters be $\phi_{\mathbf{A}}$ and $\phi_{\mathbf{A},\mathbf{T}}$,
respectively. Similarly, the updated critic model parameters are denoted by
$\phi_{\mathbf{C}_{1}}$ and $\phi_{\mathbf{C}_{2}}$ and the corresponding
’target critic networks’ is denoted with parameters
$\phi_{\mathbf{C}_{1},\mathbf{T}}$ and $\phi_{\mathbf{C}_{2},\mathbf{T}}$
respectively. These networks are termed as the trained models in the
subsequent steps.

###  3.3 TD3 : online learning

Step 1 Use the trained actor and critic model obtained from offline learning
as the actor network and the critic network for the true-plant.

Step 2 Initialise the experience replay buffer ($E2$) with a suitable
cardinally and add tuples obtained from the offline learning ($E1$). Each
tuple is composed of a state, action, reward and new state i.e
$(s,a,r,s^{\prime})$.

Step 3 Observe the initial state $s$ and select action $a$ from the actor
network and add Gaussian noise as a way of exploration during training. Also,
clip the action, $a$ between the action range, as follows:

$a=clip(\mu_{\phi_{A}}(s)+\epsilon,a_{min},a_{max})$ (12)

where $\epsilon\sim\mathcal{N}(0,\,\sigma^{2})$ with a suitable exploration
noise variance $\sigma^{2}$ .

Step 4 Execute the action, $a$ by injecting it into the true-process and
obtain reward $r$ and new state $s^{\prime}$.

Step 5 Add the obtained tuple of state, action, reward,next state
$(s,a,r,s^{\prime})$ in the replay buffer ($E2$).

Step 6 Sample a batch of transitions from the experience replay $E2$.

Step 7 Target actor network outputs the deterministic action $\tilde{a}$ for
state $s^{\prime}$, subsequently, a clipped noise added to this action. The
action values are further clipped to ensure that they are in the valid action
range.

$\tilde{a}=clip(\mu_{\phi_{A,T}}(s^{\prime})+clip(\epsilon,-c,c),a_{min},a_{max})$
(13)

where $\epsilon\sim\mathcal{N}(0,\sigma)$; $c$ and $\sigma$ are the noise clip
and the policy noise variance, respectively.

Step 8 The state $s^{\prime}$ and the target action $\tilde{a}$ is given as
input to the target Q-network to estimate the target Q-value,
$Q_{\phi_{\mathbf{C}_{1},\mathbf{T}}}(s^{\prime},\tilde{a})$ and
$Q_{\phi_{\mathbf{C}_{2},\mathbf{T}}}(s^{\prime},\tilde{a})$. Select the
minimum of the two Q-values to calculate the target value ($TV$) given as:

$TV=r+\gamma\min Q_{\phi_{\mathbf{C}_{i},\mathbf{T}}}(s^{\prime},\tilde{a})$
(14)

Step 9 Estimate the Q- value for the state-action pair $(s,a)$,
$Q_{\phi_{\mathbf{C}_{1}}}(s,a)$ and $Q_{\phi_{\mathbf{C}_{2}}}(s,a)$, and
calculate the Loss (equation (5)), as below,

$\displaystyle Loss1$ $\displaystyle=MSE(Q_{\phi_{\mathbf{C}_{1}}}(s,a),TV)$
(15) $\displaystyle=\mathbb{E}[(Q_{\phi_{\mathbf{C}_{1}}}(s,a)-TV)^{2}]$ (16)
$\displaystyle Loss2$ $\displaystyle=MSE(Q_{\phi_{\mathbf{C}_{2}}}(s,a),TV)$
(17) $\displaystyle=\mathbb{E}[(Q_{\phi_{\mathbf{C}_{2}}}(s,a)-TV)^{2}]$ (18)
$Loss=MSE(Q_{\phi_{\mathbf{C}_{1}}}(s,a),TV)+MSE(Q_{\phi_{\mathbf{C}_{2}}}(s,a),TV)$
(19)

Step 10 Update the Q-value by backpropogating the loss and update the critic
network parameters $\phi_{\mathbf{C}_{1}}$ and $\phi_{\mathbf{C}_{2}}$ by
stochastic gradient descent using a suitable optimizer.

Step 11 For every two iterations update the actor network by doing gradient
ascent on the Q-value of the first critic network,
$\nabla_{\phi_{\mathbf{A}}}Q_{\phi_{\mathbf{C}_{1}}}(s,\mu_{\phi_{\mathbf{A}}}(s))$.

Step 12 Update the weights of the critic target

for i= 1,2
$\displaystyle\phi_{\mathbf{C}_{i},\mathbf{T}}\leftarrow\tau\phi_{\mathbf{C}_{i}}+(1-\tau)\phi_{\mathbf{C}_{i},\mathbf{T}}$
(20) end

and actor target

$\phi_{\mathbf{A},\mathbf{T}}\leftarrow\tau\phi_{\mathbf{A}}+(1-\tau)\phi_{\mathbf{A},\mathbf{T}}$
(21)

by Polyak averaging, where $\tau\in[0,1]$ is a suitable target update rate.

Step 13 Obtain the new state, $s\rightarrow s^{\prime}$ & Repeat from Step 3
until the batch process completes $t=t_{end}$ where $t_{end}$ is the end time
of the batch.

Step 14 Repeat for Steps 1-13 for subsequent batches ( Note: Now Step 1 is to
use the updated model obtained from the previous batch for the initialisation
of networks at the start time of the next batch run instead of the model
obtained from offline learning). Also in Step 2, the $E2$ contains tuples
obtained from the previous batch run.

##  4 Results and Discussion

This section discusses the numerical simulation results for the application of
TD3 based controller to batch transesterification process. We have compared
the performance of TD3 with respect to other continuous RL algorithms such as
DDPG, and dicrete action-space RL algorithms such as DQN, and Q-learning with
Gaussian process (GP) as the function approximator. The above-mentioned
algorithms are trained using two types of reward functions that resemble PI
reward and PID reward, respectively. We have also introduced batch-to-batch
variations and evaluated the comparative performance of both TD3 and DDPG
algorithms in this section.

###  4.1 Experimental Details of training the agent

Herein, we present the training details of RL based agent used for the control
of the batch transesterification problem. A neural network consisting of 2
hidden layers with 400 and 300 hidden nodes, respectively, is used for both
actor-network and the twin critic networks. Rectified Linear Unit (ReLU) is
used as the activation function between each hidden layer for both actor and
critics. Further, a linear activation function is used for the output in the
actor-network. The network parameters are updated using the ADAM optimiser for
both the actor and critics network. Both the state and action are given as
input to the critic network to estimate the Q-value.

The implementation of the algorithm was done in Python 3.7.4, and the neural
network framework is constructed in PyTorch (for continuous action space) and
Keras (for discrete action space) API. The mathematical model of the
transesterification process was simulated in Matlab and integrated to Python
via Matlab engine. Table 1 lists the hyper-parameters used for the
implementation of the TD3 algorithm.

In this study, the penalty function $f(.)$ in the reward formulation (as
discussed in Section 3) is taken as the absolute value of error, $\left\lvert
e(t)\right\rvert$ i.e., $\left\lvert T_{r}(k)-T_{ref}\right\rvert$ and $g(.)$
is the inverse operator ($g(.)=\frac{1}{f(.)}$). Here $T_{r}$ is the Reactor
temperature and $T_{ref}$ is the desired temperature. The threshold parameters
$thr_{1}$ and $thr_{2}$ are fixed to be 5 in the present study. For the PI
reward the constant values $c_{1}^{\Romannum{1}}\dots c_{5}^{\Romannum{1}}$
are taken as $0,10,100,0.05,100$ and for the PID reward the constant values
from $c_{1}^{\Romannum{2}}\dots c_{9}^{\Romannum{2}}$ are taken as
$0,10,100,1,1,0.05,100,1,1$ respectively.

Table 1: Hyperparameters for TD3 algorithm Hyperparameters | Value
---|---
Discount factor | 0.99
Policy noise | 0.2
Exploration noise | 0.1
Clippd noise | 0.5
Actor Learning Rate | 10e-3
Critic Learning Rate | 10e-3
Target Update Rate | 0.005
Policy frequency | 2

###  4.2 Tracking Performance and performance during batch-to-batch variation

Table 2 compares the average RMSE values for four different algorithms,
namely, TD3 and DDPG, DQN, and GP, respectively. Neural network and Gaussian
process regression (GPR) are the candidate for function approximator in DQN
and GP, respectively. Further, the results are compared for the two types of
reward functions considered, namely, PI and PID, as discussed in Section 3.
Here, the RMSE values reported are the average of the last four batches for a
total of 10 batches. It can be seen that the TD3 based controller has the
lowest RMSE of 1.1526 and 1.1738 for PID and PI reward functions,
respectively, as compared to DDPG and other discrete action space algorithms.
Additionally, it can be seen that the PID reward function is a better choice
for reward function due to their low RMSE for all the four algorithms.

Table 2: RMSE comparison of four different RL algorithms for batch transesterification process Reward | Continuous Action | Discrete Action
---|---|---
TD3 | DDPG | DQN | GP
PI | 1.1738 | 1.2524 | 1.3189 | 1.4026
PID | 1.1526 | 1.1758 | 1.2826 | 1.2835

Figure 2 and Figure 3 shows the comparison of the control performance with
reactor temperature ($T_{r}$) with respect to time. Figure 2 compares the
control performance of continuous action space algorithms, namely, TD3 vs DDPG
for both the PID and PI reward in subplots (a) and (b), respectively.
Similarly, the performance of the discrete action space algorithms, namely,
DQN and GP, for both the reward functions are shown in Figure 3. It can be
seen from the plots that the reactor temperature profile closely follows the
target value for TD3 controller for the PID reward function. The results
clearly conclude that that the proposed TD3 based controller is capable to
learn from the given environment and control the system by achieving the
desired set-point of temperature $(T_{ref})$ which is 345K.

(a) Continuous Action (PID)

(b) Continuous Action (PI)

Figure 2: Comparison of control Performance of a) TD3 vs. DDPG for PID reward
and b) TD3 vs. DDPG for PI reward

(a) Discrete Action(PID)

(b) Discrete Action(PI)

Figure 3:  Comparison of control Performance of a) DQN vs. GP for PID reward
b) DQN vs. GP for PI reward

Since the steady-state error with PID reward function is better than the PI
reward function for all the four algorithms, the subsequent analysis considers
only the PID reward function. It can be seen that the variability in control
actions are more for the discrete case as compared to the continuous action
space. Table 3 reports the average value of standard deviation (SD) of control
actions across the last four episodes. The results show that the SD values for
the TD3 and DDPG are comparable while the input fluctuations are more for the
discrete case. It is worth noting that we have constrained the action space
between 330-350 K for continuous action space algorithms. However, we have
observed that the discrete action space algorithms are unable to honour this
constraint due to limited discrete action options. Hence, we have relaxed the
constraint space to 330-360 K for discrete action space algorithms, with an
interval of 0.75 K. Figure 4 (a) shows the control action plots for all the
four algorithms for the PID reward. Further, the control effort is calculated
and compared for TD3 and DDPG by taking the average of integral of the square
of the control actions and the obtained values are reported in Table 4.

We have also compared the results of reward vs time plots for TD3 and DDPG.
Here, in order to make the contrast visible, we have evaluated the inverse of
the reward function, the penalty, and plotted the penalty values vs time as
shown in Figure 4 (b). Figure 4(b) shows that DDPG has a higher value of
penalty as compared to TD3 algorithm, indicating low rewards obtained in
comparison with TD3. These results reinforce that TD3 is better than DDPG for
control application of batch transesterification processes.

Further, to test the efficacy of the TD3 controller, we have introduced batch-
to-batch variations in the simulations. Batch-to-batch variation may occur due
to slight perturbations in the process parameters and changes in the
environmental conditions during a batch run. For the batch transesterification
process, we have introduced batch-to-variation by randomly changing the rate
constant, $k_{c}$, by changing the pre-exponential factor ($k_{o}$) having a
variance of $10\%$ in each batch. The tracking trajectory plot presented in
Figure 5 clearly shows that the TD3 based controller is able to reach the set-
point in the presence of batch-to-batch variations and thus we achieve the
desired control performance. The average RMSE is 1.1620 and 1.2133 for TD3 and
DDPG, respectively, for batch-to-batch variations, once again indicating the
advantages of TD3 over DDPG for controlling the batch transesterification
process.

(a) Comparison of control inputs for four different algorithms

(b) Comparison of Penalty for TD3 and DDPG algorithm

Figure 4: Comparison of (a) control Performances of all approaches for PID reward (b) Comparison of Penalty for TD3 and DDPG for PID reward Figure 5: Comparison of control performance of TD3 vs. DDPG for PID reward(for batch-to-batch variation) Table 3: Variability in Control Action for four differnt RL algorithm for the PID reward  Algorithm | SD
---|---
TD3 | 3.2430
DDPG | 3.2642
DQN | 11.1897
GP | 9.1718
Table 4: Comparison of Control effort for continuous Action space algorithms  Algorithm | Control Effort
---|---
TD3 | 118416.7
DDPG | 118680.3

##  5 Conclusion

This paper presented a controller based on the TD3 RL algorithm for the
control of batch transesterification process. The reactor temperature is
considered the state (control variable) to be controlled, and the jacket inlet
temperature is taken as action (control input). It was observed that the
controller is able to learn the optimal policy and achieve the desired reactor
temperature by implementing appropriate control actions. We also formulated
reward functions taking inspiration from the functional structure of PI and
PID controller, by incorporating the historical errors and showed that it
helps the agent to better learn about the process. The results indicate that
TD3 shows better convergence as compared to continuous action-space algorithm
DDPG and other algorithms such as DQN and GP that are applicable to discrete
action spaces. In summary, TD3 based RL controller is able to learn and
intervene the process operation and control the process operation efficiently
and can be used as potential framework for complex non-linear systems where
both the state and the action space are continuous.

## References

  * [1] Khalid Khalizani, Khalid Khalisanni, et al.  Transesterification of palm oil for the production of biodiesel.  American Journal of Applied Sciences, 8(8):804–809, 2011.
  * [2] Xuejun Liu, Huayang He, Yujun Wang, Shenlin Zhu, and Xianglan Piao.  Transesterification of soybean oil to biodiesel using cao as a solid base catalyst.  Fuel, 87(2):216–221, 2008.
  * [3] Junzo Otera.  Transesterification.  Chemical reviews, 93(4):1449–1470, 1993.
  * [4] Mehdi Hosseini, Ali Mohammad Nikbakht, and Meisam Tabatabaei.  Biodiesel production in batch tank reactor equipped to helical ribbon-like agitator.  Modern Applied Science, 6(3):40, 2012.
  * [5] BEHP Freedman, EH Pryde, and TL Mounts.  Variables affecting the yields of fatty esters from transesterified vegetable oils.  Journal of the American Oil Chemists Society, 61(10):1638–1643, 1984.
  * [6] A Chanpirak and W Weerachaipichasgul.  Improvement of biodiesel production in batch transesterification process.  In Proceedings of the International MultiConference of Engineers and Computer Scientists, II, volume 5, 2017.
  * [7] Pahola T Benavides and Urmila Diwekar.  Optimal control of biodiesel production in a batch reactor: Part ii: Stochastic control.  Fuel, 94:218–226, 2012.
  * [8] Riju De, Sharad Bhartiya, and Yogendra Shastri.  Dynamic optimization of a batch transesterification process for biodiesel production.  2016 Indian Control Conference, ICC 2016 - Proceedings, (Icc):117–122, 2016.
  * [9] Alejandro G Marchetti, Grégory François, Timm Faulwasser, and Dominique Bonvin.  Modifier adaptation for real-time optimization—methods and applications.  Processes, 4(4):55, 2016.
  * [10] Ali Mesbah.  Stochastic model predictive control: An overview and perspectives for future research.  IEEE Control Systems Magazine, 36(6):30–44, 2016.
  * [11] Jay H Lee and Kwang S Lee.  Iterative learning control applied to batch processes: An overview.  Control Engineering Practice, 15(10):1306–1318, 2007.
  * [12] Richard Kern and Yogendra Shastri.  Advanced control with parameter estimation of batch transesterification reactor.  Journal of Process Control, 33:127–139, 2015.
  * [13] Farouq S Mjalli and Mohamed Azlan Hussain.  Approximate predictive versus self-tuning adaptive control strategies of biodiesel reactors.  Industrial & engineering chemistry research, 48(24):11034–11047, 2009.
  * [14] Farouq S Mjalli, L Kim San, K Chai Yin, and M Azlan Hussain.  Dynamics and control of a biodiesel transesterification reactor.  Chemical Engineering & Technology: Industrial Chemistry-Plant Equipment-Process Engineering-Biotechnology, 32(1):13–26, 2009.
  * [15] Ho Yong Kuen, Farouq S Mjalli, and Yeoh Hak Koon.  Recursive least squares-based adaptive control of a biodiesel transesterification reactor.  Industrial & engineering chemistry research, 49(22):11434–11442, 2010.
  * [16] Ana SR Brásio, Andrey Romanenko, João Leal, Lino O Santos, and Natércia CP Fernandes.  Nonlinear model predictive control of biodiesel production via transesterification of used vegetable oils.  Journal of Process Control, 23(10):1471–1479, 2013.
  * [17] Ana SR Brásio, Andrey Romanenko, Natércia CP Fernandes, and Lino O Santos.  First principle modeling and predictive control of a continuous biodiesel plant.  Journal of Process Control, 47:11–21, 2016.
  * [18] Steven Spielberg, Aditya Tulsyan, Nathan P Lawrence, Philip D Loewen, and R Bhushan Gopaluni.  Deep reinforcement learning for process control: A primer for beginners.  arXiv preprint arXiv:2004.05490, 2020.
  * [19] Rui Nian, Jinfeng Liu, and Biao Huang.  A review On reinforcement learning: Introduction and applications in industrial process control, 8 2020.
  * [20] Jong Min Lee and Jay H Lee.  Approximate dynamic programming-based approaches for input–output data-driven control of nonlinear processes.  Automatica, 41(7):1281–1288, 2005.
  * [21] Jong Min Lee.  A Study on Architecture, Algorithms, and Applications of Approximate Dynamic Programming Based Approach to Optimal Control.  Technical report, 2004.
  * [22] Jay H. Lee and Weechin Wong.  Approximate dynamic programming approach for process control.  In Journal of Process Control, volume 20, pages 1038–1048, 10 2010\.
  * [23] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  nature, 518(7540):529–533, 2015.
  * [24] Saxena Nikita, Anamika Tiwari, Deepak Sonawat, Hariprasad Kodamana, and Anurag S Rathore.  Reinforcement learning based optimization of process chromatography for continuous processing of biopharmaceuticals.  Chemical Engineering Science, 230:116171, 2021.
  * [25] Vikas Singh and Hariprasad Kodamana.  Reinforcement learning based control of batch polymerisation processes.  IFAC-PapersOnLine, 53(1):667–672, 2020.
  * [26] Scott Fujimoto, Herke Hoof, and David Meger.  Addressing function approximation error in actor-critic methods.  In International Conference on Machine Learning, pages 1587–1596. PMLR, 2018.
  * [27] MyeongSeop Kim, Dong-Ki Han, Jae-Han Park, and Jung-Su Kim.  Motion planning of robot manipulators for a smoother path using a twin delayed deep deterministic policy gradient with hindsight experience replay.  Applied Sciences, 10(2):575, 2020.
  * [28] Stephen Dankwa and Wenfeng Zheng.  Modeling a continuous locomotion behavior of an intelligent agent using deep reinforcement technique.  In 2019 IEEE 2nd International Conference on Computer and Communication Engineering Technology (CCET), pages 172–175. IEEE, 2019.
  * [29] Richard S Sutton and Andrew G Barto.  Reinforcement learning: an introduction.  2018\.
  * [30] EN Barron and H Ishii.  The bellman equation for minimizing the maximum cost.  Nonlinear Analysis: Theory, Methods & Applications, 13(9):1067–1090, 1989.
  * [31] Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al.  Policy gradient methods for reinforcement learning with function approximation.  In NIPs, volume 99, pages 1057–1063. Citeseer, 1999.
  * [32] David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller.  Deterministic policy gradient algorithms.  In International conference on machine learning, pages 387–395. PMLR, 2014.
  * [33] Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  arXiv preprint arXiv:1509.02971, 2015.
  * [34] Hado Hasselt.  Double q-learning.  Advances in neural information processing systems, 23:2613–2621, 2010.
  * [35] Hado Van Hasselt, Arthur Guez, and David Silver.  Deep reinforcement learning with double q-learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
  * [36] Lisa Torrey and Jude Shavlik.  Transfer learning.  In Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pages 242–264. IGI global, 2010\.