# Design and Comparison of Reward Functions in Reinforcement Learning for
Energy Management of Sensor Nodes

Yohann Rioual, Yannick Le Moullec, Johann Laurent, Muhidul Islam Khan, and
Jean-Philippe Diguet Y. Rioual and J. Laurent are with Lab-STICC, University
Bretagne Sud, F-56100 Lorient, France, e-mail: {firstname.lastname}@univ-
ubs.frY. Le Moullec and M. Islam Khan are with Thomas Johann Seebeck
Department of Electronics, Tallinn University of Technology, Ehitajate tee 5,
19086 Tallinn, Estonia<EMAIL_ADDRESS><EMAIL_ADDRESS>Diguet is with IRL CNRS CROSSING, Adelaide, SA, AustraliaThis paper is an
extended version of our paper published in 2018 16th Biennial Baltic
Electronics Conference (BEC).

###### Abstract

Interest in remote monitoring has grown thanks to recent advancements in
Internet-of-Things (IoT) paradigms. New applications have emerged, using small
devices called sensor nodes capable of collecting data from the environment
and processing it. However, more and more data are processed and transmitted
with longer operational periods. At the same, the battery technologies have
not improved fast enough to cope with these increasing needs. This makes the
energy consumption issue increasingly challenging and thus, miniaturized
energy harvesting devices have emerged to complement traditional energy
sources. Nevertheless, the harvested energy fluctuates significantly during
the node operation, increasing uncertainty in actually available energy
resources. Recently, approaches in energy management have been developed, in
particular using reinforcement learning approaches. However, in reinforcement
learning, the algorithm’s performance relies greatly on the reward function.
In this paper, we present two contributions. First, we explore five different
reward functions (R1–R5) to identify the most suitable variables to use in
such functions to obtain the desired behaviour. Experiments were conducted
using the Q-learning algorithm to adjust the energy consumption depending on
the energy harvested. Results with the five reward functions illustrate how
the choice thereof impacts the energy consumption of the node. Secondly, we
propose two additional reward functions (R6 and R7) able to find the
compromise between energy consumption and a node performance using a non-fixed
balancing parameter. Our simulation results show that the proposed reward
functions (R6 and R7) adjust the node’s performance depending on the battery
level and reduce the learning time.

###### Index Terms:

reinforcement learning, IoT, WBANs, reward function, Q-learning, energy
management

##  I Introduction

The rapidly growing interest for physiological sensors and progress in low-
power integrated circuits and wireless communication have enabled a new
generation of wireless sensor networks (WSN). WSNs consist of a number of
communicating nodes, they can be fitted on the human body to monitor
physiological parameters of the wearer or environmental variables. Advances in
microelectronics led to significant miniaturization of the sensors; however,
battery technologies have not improved at the same rate [1]. Consequently, the
storage of energy is a bottleneck for the deployment of such sensors.

To minimize the battery’s size, an increasingly popular approach is to harvest
energy from the environment [2]. Miniaturised energy harvesting technologies
cannot harvest a lot of energy (see Table I), but they can be used as
complements to the battery. However, such energy sources vary greatly over
time and bring uncertainties in the system energy resources. Reinforcement
Learning (RL) algorithms have acquired a certain popularity in recent years
for energy management ([3], [4]). Indeed, they can handle such uncertainty in
energy sources and appear to be a valid solution for energy management. They
adapt the node’s behaviour by promoting good decisions by means of a reward
function [5].

TABLE I:  Power density of energy harvesting technologies [6]

Harvesting technologies | Power density
---|---
Solar cell (outdoors at noon) | $15~{}$\mathrm{m}\mathrm{W}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{2}}$$
Wind flow (at $5~{}$\mathrm{m}\mathrm{/}\mathrm{s}$$) | $16.2~{}$\mathrm{\SIUnitSymbolMicro}\mathrm{W}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{3}}$$
Vibration (Piezoelectric – shoe insert) | $330~{}$\mathrm{\SIUnitSymbolMicro}\mathrm{W}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{3}}$$
Vibration (electromagnetic conversion at $52~{}$\mathrm{Hz}$$) | $306~{}$\mathrm{\SIUnitSymbolMicro}\mathrm{W}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{3}}$$
Thermo-electric ($5~{}$\mathrm{\SIUnitSymbolCelsius}$$ gradient) | $40~{}$\mathrm{\SIUnitSymbolMicro}\mathrm{W}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{3}}$$
Acoustic noise ($100~{}$\mathrm{dB}$$) | $960~{}$\mathrm{n}\mathrm{W}\mathrm{/}\mathrm{c}\mathrm{m}\mathrm{{}^{3}}$$

The choice of an appropriate reward function is challenging. Since this
function determines the behaviour of the system, choosing it is an essential
task for the system designer. Still, the literature on this topic rarely
discusses the choice of the reward function.

Thus, in this paper, we explore the influence of different parameters on the
reward function performance with a popular RL algorithm, i.e. Q-learning. This
paper complements the work presented in [7], we present two different
contributions:

  * $\bullet$

a comparison of five different reward functions (R1–R5) to identify the most
suitable variables to design a function able to adjust correctly the energy
consumption depending of the harvested energy. This version of the paper
provides some additional details as compared to [7].

  * $\bullet$

using the previous results, we expand [7] with the detailed design of two
additional reward functions (R6 and R7) which find a compromise between the
energy consumption and node’s performance depending on the battery level with
a reduced learning time.

The remainder of this paper is structured as follows. Section II discusses the
related work. Section III introduces the RL mechanism and presents the
Q-learning algorithm used in this study. Section IV presents a body sensor
node as use case and experimental results wherein five reward functions are
evaluated and compared. Then, Section V presents a piecewise reward function
which adjusts the energy consumption of a sensor node according to the battery
state of charge. Section VI presents a generalization of the previous
function. Lastly, Section VII summarizes and concludes this paper.

##  II Related work

Works on energy management in sensor networks such as WBANs typically focus on
the consumption of the radio, often disregarding the power consumption of the
other parts of the system. In cases where energy harvesting technologies are
employed, related works have proposed adaptive protocols that deal with the
challenge of providing the required quality of service under the uncertainty
of the energy input provided by a human activity. [8] presents a method for
adapting the transmission of data according to the energy harvested. The
measured data is stored in a queue and it is transmitted if there is enough
available energy. The old data that lost their validity, determined by the
medical application requirements, are deleted from the queue to avoid having
to transmit it and preventing a buffer overflow. If the queue is full then the
oldest data is deleted. The authors also propose an algorithm that determines,
during the communication, how many packets will be transmitted according to
the available energy and the state of the queue.

To extend a sensor node’s lifespan in WBANs, there also exist different energy
management methods. This includes adapting the node’s energy consumption to
the activity of the wearer. To this end, [9] proposes a classifier to identify
the activity of a wearer using data collected from an accelerometer and adapts
the operating policy accordingly. In the literature, there is a variety of
classifiers, so the authors compare five different approaches (Nearest
Neighbour, Support Vector Machines, Naïve Bayes, Linear Discriminant and
Decision Tree). Their comparison criterion is to minimize the trade-off
between computational cost, power consumption and recognition performance. The
decision tree method corresponds the most to the application requirements and
detects the activity of the wearer with an accuracy of $98\%$. The output of
the classifier is used to change the node’s energy policy according the
activity and thus to adapt the energy consumption. The authors tested it in a
use case scenario with continuous monitoring and feedback exercises. Their
results show a five-time increase in battery lifetime.

Since RL is an approach to take decisions under uncertainty, it is suitable
for energy management in systems where energy harvesting technologies are
used. For example, the authors of [10] have developed a power control approach
in WBANs based on RL. This approach mitigates the interference due to wireless
communication in the network, which provides a substantial saving in energy
consumption per transmitted bit. However, this energy management approach only
focuses on the wireless communication and not on the other parts of the WBAN
node. Communications represent an important part of the energy consumption in
WBANs, nevertheless recent trends increase the computational load, such as
data-preprocessing or edge computing on the node to reduce the energy consumed
by communication, which increases the energy consumed by sampling and
processing.

The energy management of a sensor node is a complex problem. On the one hand,
the aim is to reduce the energy consumption, on the other hand the performance
must be maximized or satisfied depending on the application constraints. A
solution to this emerges with Multi-Objective Reinforcement Learning (MORL)
[11]. The aim of MORL is to solve problems where there are multiple
conflicting objectives. The naive approach consists of a look-up table for
each objective, the choice being made on the action that maximizes the sum of
the different Q-values.

There also exist different non-naive algorithms such as weighted sum [12] and
W-learning [13] for single policy approaches, but they cannot express exactly
the preferences of the designer. If the preferences must be expressed more
exactly, there exist Analytic Hierarchy Process (AHP) [14], ranking [15] and
geometric [16] approaches but they need prior knowledge of the problem domain.
For a multi-policy approach, there exist the convex hull algorithm [17].
However, some researchers argue that explicit multi-objective modelling is not
necessary, and that a scalar reward function is adequate [18]. For instance
Sutton’s reward hypothesis, states “that all of what we mean by goals and
purposes can be well thought of as maximization of the expected value of the
cumulative sum of a received scalar signal (reward)”. The implication is that
a multi-objective Markov Decision Process can always be converted into a
single-objective one. We will therefore focus on designing reward functions
that take into account different parameters in a single objective, i.e.,
reduction in energy consumption.

There are few papers about WBANs that deal with harvesting energy and RL for
the energy management of the entire node ([4], [19], [20]). The process of
designing a reward function is not explained in most papers; for instance [10]
shows good results with one reward function without explaining if they tried
different reward functions. The rest of the papers apply different RL
approaches to various problem and do not discuss the construction of the
reward function they used. The definition of a reward function is important
and depends on the application. The following section introduces the
reinforcement learning and one algorithm in particular, the Q-learning.

##  III Reinforcement Learning

In this section, we give an overview of RL. Then we present the selected
Markov Decision Process (MDP) used to deal with the energy management process.
And to conclude this section, we introduce the selected RL algorithm, i.e.
Q-learning.

###  III-A Overview of Reinforcement Learning

RL is a formal framework that models sequential decision problems [18], in
which an agent learns to make better decisions by interacting with the
environment (Figure 1). When the agent performs an action, the state changes
and the agent receives a feedback called a reward, which indicates the quality
of the transition. The agent’s goal is to maximize its total reward over the
long-term.

Figure 1: Interaction between an agent and its environment

There is a trade-off between exploration and exploitation in RL. During the
exploration, the agent chooses an action randomly to find out the utility of
that action, whereas during the exploitation, the action is chosen based on
the previously learned utility of the actions. The exploration probability is
given according to Equation 1 [21]:

$\epsilon=min(\epsilon_{max},\epsilon_{min}+k\times(S_{max}-S)/S_{max})$ (1)

where $\epsilon_{max}$ and $\epsilon_{min}$ denote upper and lower limits of
the exploration factor, respectively, $S_{max}$ represents the size of the
state space, and $S$ represents the current number of states already visited.
At each time step, the system computes $\epsilon$ and generates a random
number in the interval $[0,1]$. If the selected random number is at most
$\epsilon$, the system chooses a uniformly random action (exploration),
otherwise it chooses the best one using $Q$-values (exploitation).

###  III-B Markov Decision Process for Energy Management

The energy management problem can be modelled as a Markov Decision Process
(MDP). An MDP provides a theoretical framework for modelling decision making
in situations where outcomes are partly random and partly under the control of
a decision maker. An MDP is formally defined as a n-tuple
$\left<\mathcal{S},\mathcal{A},\mathcal{T},\mathcal{R}\right>$ where
$\mathcal{S}$ is a state space, $\mathcal{A}$ a set of possible actions,
$\mathcal{T}:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$
are the transitions’ probabilities between states
$\left(\mathcal{T}(s,a,s^{\prime})=p(s^{\prime}|a,s)\right.$ is the
probability to reach state $\left.s^{\prime}\text{ starting from }s\text{
after taking an action }a\right)$, and
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is a reward
signal.

The transitions’ probabilities between the states $T$ are unknown, so we use a
model-free algorithm. Model-free algorithms work even when we do not have a
precise model of the environment. These algorithms primarily rely on learning
algorithms such as Q-learning, which is described in the next section.

###  III-C Q-learning algorithm

In this section, we present the Q-Learning algorithm [22]. The Q-learning
algorithm is widely used thanks to its ease of implementation yet effective
performance, and its convergence is proven. We use Algorithm 1 for the energy
management of a sensor node in combination with the MDP presented above.

Algorithm 1 Q-learning algorithm

Initialise $Q(s,a)$ arbitrarily

The agent observes the initial state $s_{0}$

for each decision epochs do

Choose an action $a$ from state $s$ using policy derived from $Q$

Take action $a$, observe the new state $s^{\prime}$ and the associated reward
$r$

Update of the related $Q$-value:
$Q(s,a)\leftarrow
Q(s,a)+\alpha\left(r+\gamma\max\limits_{a^{\prime}}Q(s^{\prime},a^{\prime})-Q(s,a)\right)$

$s\leftarrow s^{\prime}$

end for

#### Learning rate $\alpha$

The learning rate $\alpha$ determines how fast the new information will
replace the old one. A factor of 0 would not teach the agent in question
anything, whereas a factor of 1 would only teach the agent the latest
information. In our work, we slowly decrease the learning rate $\alpha$ in
such a way that it reflects the degree to which a state-action pair has been
chosen in the past (Equation 2).

$\alpha=\frac{\zeta}{visited(s,a)}$ (2)

where $\zeta$ is a positive constant and $visited(s,a)$ represents the visited
state-action pairs so far [23].

#### Discount factor $\gamma$

The discount factor $\gamma$ determines the importance of future rewards. A
discount factor of 0 would make the agent myopic by considering only current
rewards, while a factor close to 1 would also involve more distant rewards. If
the discount factor is close or equal to 1, the value of Q may never converge.

Using the MDP, we aim to identify the best variables to use in the reward
function to adapt the energy consumption according to the energy we can
harvest. In the following section, we present some results and identify those
variables.

##  IV Energy management of a body sensor node

In this section, we test five reward functions (R1 to R5) to identify the most
valuable one for the energy management of a body sensor node. First, we
present the use case and the decision process, then we present results of the
node’s behaviour.

###  IV-A Use case

In this work, the objective is to manage the energy consumption of a sensor
node fitted on a human chest to monitor the cardiac activity for non-medical
application (Figure 2). The heart rate is measured for 10 seconds. Then, data
are sent immediately to a smartphone to be processed. The smartphone is used
as a gateway and communicates with the node using a Bluetooth Low Energy (BLE)
transceiver. The node does not continuously monitor the cardiac activity, i.e.
after each measurement it enters a sleep mode to minimize energy consumption.
The period between each measurement is variable and lasts from 10 to 60
minutes.

Figure 2: Sensors node fitted on a chest to monitor the heart beat

The sensor node is equipped with an optical heart rate detection sensor, a
low-power micro-controller unit (MCU), a BLE transceiver and a battery with a
capacity of $100~{}$\mathrm{m}\mathrm{A}\\!\mathrm{h}$$. The energy
consumption of each component is summarized in Table II. The energy
consumption of the MCU depends on the processor clock frequency. Here, the
maximum frequency is $32~{}$\mathrm{MHz}$$, corresponding to a maximum current
consumption of $7.2~{}$\mathrm{mA}$$. As can see in Table II, the
communication unit (BLE transmitter) in active mode consumes more than the two
other components combined.

TABLE II: Node components and respective current consumption

Component | Active mode | Sleep mode
---|---|---
Heart rate sensor | $1.6~{}$\mathrm{mA}$$ | $0.12~{}$\mathrm{mA}$$
Micro-controller | $225~{}$\mathrm{\SIUnitSymbolMicro A}$/$\mathrm{MHz}$$ | $0.93~{}$\mathrm{\SIUnitSymbolMicro A}$$
BLE transmitter | $10.5~{}$\mathrm{mA}$$ |  $0~{}$\mathrm{\SIUnitSymbolMicro A}$$ (turned off)

To increase the amount of energy available, a kinetic motion energy harvester
is added to the battery. This energy harvester is presented in [24]. The
harvested energy is low to power entirely a node, but it still can extend the
node’s lifespan; Table III shows how much power can be harvested according to
the activity of the wearer. These data are extracted from [24].

TABLE III: Kinetic motion’s harvested power for three different activities

Activity | Power harvested
---|---
relaxing |  $2.4$ $\mathrm{\SIUnitSymbolMicro W}$
walk |  $180.3$ $\mathrm{\SIUnitSymbolMicro W}$
run |  $678.3$ $\mathrm{\SIUnitSymbolMicro W}$

We use the dominant frequency of motion, $F_{m}$, to identify which activity
is performed by the wearer. We obtain $F_{m}$ by determining the maximum
spectral component of the Fourier Transform of the acceleration $a(t)$. Since
the harvested energy is uncertain, we use an RL approach to manage the node’s
consumption by adjusting its sleep duration and its processor’s clock
frequency.

###  IV-B Decision Process

For this use case, we define a set of actions with different processor
frequencies ($F_{p}$) and periods between each measurement ($P_{s}$) (Table
IV). For instance, Action $1$ has a processor frequency of
$32~{}$\mathrm{MHz}$$ and a measurement every minute, whereas Action 3 has a
processor frequency of $4~{}$\mathrm{MHz}$$ and a measurement every 5 minutes.
Thus, Action 1 consumes more current (and thus more energy) than Action 3. All
actions have different energy consumption levels since they depend on the
processor’s frequency in active mode and its consumption in sleep mode (see
the third row in Table IV).

TABLE IV: Set of actions with different processor frequencies ($F_{p}$) and
periods between each measurement ($P_{s}$), and the associated average current
consumption

Action | $F_{p}$ | $P_{s}$ | Average current consumption
---|---|---|---
1 |  $32$ $\mathrm{MHz}$ |  $1$ $\mathrm{min}$ |  $0.6278$ $\mathrm{mA}$
2 |  $4$ $\mathrm{MHz}$ |  $1$ $\mathrm{min}$ |  $0.4873$ $\mathrm{mA}$
3 |  $4$ $\mathrm{MHz}$ |  $5$ $\mathrm{min}$ |  $0.2292$ $\mathrm{mA}$
4 |  $4$ $\mathrm{MHz}$ |  $20$ $\mathrm{min}$ |  $0.2044$ $\mathrm{mA}$
5 |  $1$ $\mathrm{MHz}$ |  $60$ $\mathrm{min}$ |  $0.1926$ $\mathrm{mA}$

Our state space is divided into three different states corresponding to the
activity of the wearer (Table III). We use the dominant frequency motion
$F_{m}$ which is correlated with the energy we harvest to consider our state.
A high value of $F_{m}$ corresponds to more energy being harvested and a low
value of $F_{m}$ corresponds to less energy being harvested. The state is
determined with the value of $F_{m}$ and corresponds to an activity. The
activity can be considered high (i.e. running) if $F_{m}>2~{}$\mathrm{Hz}$$ ,
moderate (i.e. walking) if $2~{}$\mathrm{Hz}$\geq F_{m}>1~{}$\mathrm{Hz}$$ or
low (i.e. relaxing) if $F_{m}\leq 1~{}$\mathrm{Hz}$$.

###  IV-C Simulation results

First of all, it should be noted that the harvesting capabilities of the
kinetic motion harvester are not sufficient to fully recharge the sensor
node’s battery. Thus, we seek and expect to reduce the node’s consumption when
the harvested energy is low. We test five different reward functions to
identify which parameters have an appropriate influence on our system’s
behaviour. To avoid divergence in the Q-values, the values of the different
reward function are limited to [-1, 1].

There are different constraints when designing the system and most of them are
conflicting; for instance keeping the sleep period as short as possible while
also reducing energy consumption. The main purpose of the RL algorithm is to
find the equilibrium point to respect these constraints. To this end, the
first and second reward functions use a parameter $\beta$ to balance the
equilibrium point according to what is considered most important between
performance and battery level [21].

The first reward function (R1) seeks to balance the conflicting objectives
between the sleep duration $P_{s}$ and the energy consumption of the sensor
node. $B_{r}(t)$ is the residual energy in the battery’s node at time $t$.

$R=\beta*\frac{\min(P_{s})}{P_{s}}+(1-\beta)*(B_{r}(t)-B_{r}(t-1))$ (R1)

The second reward function (R2) is similar to the first one but instead of
using the energy consumption, it only uses the residual energy of the
battery’s node at time $t$.

$R=\beta*\frac{\min(P_{s})}{P_{s}}+(1-\beta)*\frac{B_{r}(t)}{B_{max}}$ (R2)

The third reward function (R3) does not consider the sleep duration $P_{s}$
but only the energy consumption. The objective is to find the less consuming
operating mode without taking care of the performance.

$R=B_{r}(t)-B_{r}(t-1)$ (R3)

Finding the right setting for $\beta$ is not trivial, that is why the fourth
reward function (R4) uses the product of the sleep duration $P_{s}$ and the
residual energy $B_{r}(t)$. Indeed, the result is maximal when both values are
maximum.

$R=\frac{\min(P_{s})}{P_{s}}\times B_{r}(t)$ (R4)

We want to adapt the energy consumption to the activity of the wearer. So, in
the fifth reward function (R5), we use the dominant motion frequency $F_{m}$
which determines the activity. The aim is to minimize the difference between
the normalized $F_{m}$ and the energy consumption; a cosine function restricts
the reward to $[-1,1]$. The reward is maximized when the difference is close
to $0$. Moreover, this reward function eliminates the $\beta$ parameter that
is not trivial to adjust. $\mathcal{N}$ is the normalization function.

$R=\cos\left(\frac{\mathcal{N}(F_{m})-(B_{r}(t)-B_{r}(t-1))}{2}\right)$ (R5)

We simulate a week-long node deployment to observe the evolution of the
battery’s charge level. The activity changes every 30 minutes, and our agent
chooses an action (Table IV) every 20 minutes. Figure 3 shows the average
energy consumption of the node according to the activity identified with the
dominant frequency of motion, $F_{m}$. The parameter $\beta$ is fixed at $0.3$
since our primary goal is to adapt the node’s consumption, i.e. we give more
importance to the energy factor. The results show that the choice of the
reward function has a significant impact on the average current consumption.
While some reward functions yield the expected behaviour, others adapt poorly
to the wearer’s activity and others do not yield the correct behaviour at all,
as discussed in what follows.

Figure 3:  Normalised average energy consumption of the node according to 1)
the activity of the wearer and 2) the reward function used within the
Q-learning algorithm. R1, R2 and R5 behave as expected since they allow the
node to consume more when more energy is harvested.

The expected behaviour of the node is to adjust its energy’s consumption
depending on the harvested energy. Thus, during a physical effort, when the
harvested energy is high, the node realizes more measurements. A second
objective is that the node needs to survive at least a week to reduce the
number of human intervention on the node. This second objective is achieved
for all the reward functions. Based on Figure 3, our observations are as
follows:

$\blacktriangleright$ Reward function R1 computes the reward using the sleep
time and the energy consumption of the node. This function produces a maximal
value when the sleep time and the energy consumption are low. It successfully
adapts the energy consumption according to the activity, increasing the node’s
consumption when the harvested energy increases. A drawback is how to select
the most appropriate value for the $\beta$ parameter.

$\blacktriangleright$ Reward function R2 computes the reward with the sleep
time $P_{s}$ and the battery’s residual energy. In the same way as the reward
function R1, it successfully adapts the energy consumption according to the
activity, achieving lowest energy consumption than the reward function R1 in 2
activities (walking and running). Furthermore both rewards share the same
drawback, the choice of the value of the $\beta$ parameter.

$\blacktriangleright$ Reward function R3 computes the reward only using the
node’s consumption. It fails to adapt the node’s behaviour according to the
harvested energy. It does not make any difference between the activities;
however, it succeeds to minimize the energy consumption. At the end of the
simulation, the battery charge is still above $75\%$ .

$\blacktriangleright$ Reward function R4 computes the reward with the product
of the $P_{s}$ and the battery’s residual energy. This reward does not have a
parameter to tune, and it is easy to compute. Unfortunately, it fails to
adjust the node’s consumption according to the harvested energy. The reward
function increases the node’s consumption when the wearer is relaxing (i.e.
the energy harvested is low), decreases it when the wearer is walking and then
increases the node’s consumption when the wearer is running (i.e. the energy
harvested is maximal). This is obviously not desired and the reward function
(R4) is more influenced by the sleep time $P_{s}$ than by the consumption of
the sensor node.

$\blacktriangleright$ Reward function R5 computes the reward with the
normalized value of the dominant frequency of motion $F_{m}$ and the node’s
consumption. The reward is maximal when the difference between the energy
consumption and the normalized dominant frequency of motion is close to $0$.
The function R5 fulfills the different objectives we had set at the beginning.
There is not a parameter to tune, and so, this reward function can be used
easily in this application. Nevertheless, the absence of this parameter makes
this function less relevant for other applications with different
requirements.

TABLE V: Evaluation of the different reward functions

Reward function | Configurable | Energy consumption | Compliance with the objectives
---|---|---|---
R1 | $\bigstar\bigstar\bigstar$ | $\bigstar\bigstar$ | $\bigstar\bigstar\bigstar$
R2 | $\bigstar\bigstar\bigstar$ | $\bigstar\bigstar$ | $\bigstar\bigstar\bigstar$
R3 | – | $\bigstar\bigstar\bigstar$ | –
R4 | – | $\bigstar$ | –
R5 | – | $\bigstar\bigstar$ | $\bigstar\bigstar\bigstar$

The overall evaluation of the five reward functions is summarized in Table V.
The reward functions R1 and R2 allows regulating the importance given to the
energy consumption according to the application requirements by increasing or
decreasing the value of $\beta$, whereas the reward functions (R3) and (R4)
are not relevant to adapt correctly the energy in a sensor node. The correct
behaviour of the node can be obtained by using a $\beta$ parameter to balance
energy consumption and performance (R1, R2). However, it is necessary to
adjust this parameter. Using the dominant motion frequency in R5 removes this
parameter and still achieves the right behaviour. However, this reward
function is less modular. It allows adapting the energy consumption according
to the activity but does not take the sleep duration into account.

IoT sensors are also used in other applications such as monitoring
environmental conditions. For this kind of application, we could prefer to
reduce performance when the battery is low and then increase it after it has
recharged. In the following section, we present different designs for reward
functions to balance the performance according to the residual energy in the
battery. For this, we use a monitoring application based on a marine buoy.

##  V Design of a Piecewise Reward Function

As seen in the previous section, the use of a balancing parameter allows a
designer to design a reward function that complies with the objectives.
Moreover, the use of a balancing parameter makes the reward function
configurable, either to maximize a performance parameter or to preserve the
battery’s energy.

The balancing parameter is selected using experience of the designer and it is
fixed. We propose a reward function in which the configuration changes
depending on the battery level (Equation R6).

$R=\left\\{\begin{array}[]{cl}F_{s}\times\rho_{1}+B\times(1-\rho_{1})&B\geq
75\%\\\ F_{s}\times\rho_{2}+B\times(1-\rho_{2})&75\%>B\geq 50\%\\\
F_{s}\times\rho_{3}+B\times(1-\rho_{3})&50\%>B\geq 25\%\\\
F_{s}\times\rho_{4}+B\times(1-\rho_{4})&\text{otherwise}\end{array}\right.$
(R6)

where $1\geq\rho_{1}>\rho_{2}>\rho_{3}>\rho_{4}\geq 0$ are balancing
parameters, $F_{s}$ the sampling frequency and $B$ is the charge of the
battery.

When the battery is fully charged, it is useless to preserve it and it is
possible to maximize the performance. Whereas, when the battery is discharged,
it becomes really important to restrict the energy consumption in order to
recharge the battery and extend the node’s lifetime. A difficulty with this
reward function is its adjustment. In Equation R6, the battery level is
divided into four equal-sized parts; however, this may vary according to the
application or the node. Moreover, the parameters $\rho_{1,2,3,4}$ must be
selected and a poor choice will make the reward function less effective or
even not effective at all.

To evaluate the proposed reward function, a simulation is conducted for the
deployment of a marine buoy near Lorient, France, for 21 days. The buoy is
equipped with two sensors (Table VI), a 3D anemometer and an atmospheric
sensor, to monitor environmental conditions. The sensors have different
energetic behaviours and the buoy should be deployed for as long as possible.
To complete the battery, the buoy is equipped with two small solar panels. To
avoid collision with ships, a beacon light flashes for $500~{}$\mathrm{ms}$$
every four seconds when the brightness is low.

TABLE VI:  Buoy components

Components | Device
---|---
3D Anemometer | WindMaster HS
Atmospheric sensor | YOUNG61302L
Processor | Cortex-M4 MCU
Radio transceiver | CC1000
Energy harvester | Power
Solar panels | $2\times 10~{}$\mathrm{W}$$
Battery capacity | 5200 $\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$

The marine buoy use case is preferred instead of the body sensor node use
case. Indeed, the marine buoy has more energy harvesting capabilities and
battery capacities than the body sensor node. The $Q$-learning algorithm is
applied with the proposed reward function (Eq. R6). The value of the learning
rate $\alpha$ is computed using the same equation as previously (Eq. 2), and
we set the value of the discount factor $\gamma$ to $0.8$. The agent chooses a
new action every $30$ minutes to let the battery state of charge change.

Several experiments have been conducted to find the most suitable values for
the parameters $\rho_{1,2,3,4}$. We found out that the best values are:
$\rho_{1}=1$, $\rho_{2}=0.6$, $\rho_{3}=0.3$, $\rho_{4}=0$. Thus, when the
battery is more than $75\%$ charged, the reward is computed only with the
frequency of measurements. When the battery level is between $75\%$ and
$50\%$, the reward is computed with both frequency and battery levels, but the
frequency is given more weight. It is the opposite when the battery level is
between $50\%$ and $25\%$, the reward function starts to preserve the
battery’s energy when the energy harvested is not sufficient to recharge it.
If the battery charge level decreases below $25\%$, the reward is computed
only with the battery level in order to preserve the node’s energy. The
results of a simulation using these values for the reward function (R6) are
presented in Figure 4.

Figure 4:  Evolution of the battery charge level and frequency measurements of
sensors using the $Q$-learning algorithm with the reward function of Equation
R6. Battery capacity=$5.2~{}$\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$$

At the beginning of the deployment (1), the agent has no prior information
about its environment and takes random actions to explore it. When the battery
level decreases (2), the agent adapts the frequency of measurements and finds
a behaviour which adjusts the frequency according to the daily variation in
the battery charge level. When the battery charge level increases around day 6
(3), the agent has not enough information in the new state to find a correct
behaviour. And when the battery decreases again (4), the first knowledge
learned has been replaced with the new information; the agent lost the
behaviour which adjusts the frequency of measurements according to the daily
variation. Nevertheless, at the end of the simulation (5) around the $16^{th}$
day, the agent’s behaviour adapts correctly the frequency to the variation,
this behaviour receives enough rewards to be reinforced.

Since the battery does not decrease enough, the agent never explores the
environment when the battery level is critical. So, a second experiment is
conducted where the battery capacity is reduced to
$3.2~{}$\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$$ instead of
$5.2~{}$\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$$. The result of this simulation
is shown in Figure 5.

Figure 5:  Evolution of the battery charge level and measurements frequency of
sensors using the $Q$-learning algorithm with the reward function R6. Battery
capacity=$3.2~{}$\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$$

At the beginning of the simulation (1), the agent has still no prior
information about its environment. Then, it successfully adapts to the daily
variations in the battery charge level (2). When the battery level decreases
below the $50\%$ of charge level (3), the agent decreases the frequency of
measurements more than during the previous simulation. The results show that
the proposed reward function is suitable for preserving the battery and for
maximizing the performance when the battery is recharged. However, the agent’s
behaviour does not show an adaptation as good as previously. This change in
behaviour is due to the exploration.

Nevertheless, a difficulty with this reward function is to determine the
values of the $\rho$ parameters. The adjustment of the parameters is based on
the expertise of the designer or is determined empirically. To obtain the best
behaviour with the use case, the parameter $\rho$ was determined empirically,
but it can be different according to the application and the prevalence of the
performance or the energy. For such an approach to be accepted, it is
important to avoid adding extra-parameters to tune. In such case the reward
function becomes more complex and the behaviour of the agent becomes less
predictable. Thus, in the following section, we present an improvement of the
proposed reward function where the parameters $\rho_{1,2,3,4}$ are removed and
the different levels too. This improvement eliminates the adjustment step,
making it easier to use the new reward function. Furthermore, the performance
of the new reward function still remains similar to that of the previous one.

##  VI Continuous Reward Function to Balance the Performance and the Energy
Consumption

The selection of the different variables used in a RL approach is time-
consuming and add more complexity. So, the designer needs to reduce the number
of parameters to be tuned in the reward function. The previous reward function
is efficient and increases the learning speed as compared to the reward
function presented in the previous section (R6). The values of the different
parameters can be adjusted to correspond perfectly to the desired behaviour
for a given application. Nevertheless, most sensor nodes are used for
environmental monitoring applications, and the main objective is to extend the
node’s lifetime. So, in this section, we present a reward function that
reinforces the same behaviour as the previous one, but without any parameter
to tune.

Indeed, while experimenting different values for $\rho$, we observe that this
parameter’s value varies as the battery level. Using this observation, we
design a new reward function without parameter to tune (Eq. R7) to balance the
battery charge level and the performance:

$R=F_{s}\times B+B\times(1-B)$ (R7)

where $F_{s}$ is the frequency of the measurements and $B$ the battery level.
The parameters $\rho_{1,2,3,4}$ have been replaced by the value of the battery
charge level.

This reward function is a generalization of the previous one. The reward is
computed mainly with the frequency of measurements when the battery is fully
charged, and mainly with the battery level when the battery level is low.
Thus, this proposed reward function requires no additional parameter to
adjust.

We conducted a simulation on the marine buoy use case. We simulated the
deployment of the buoy during a period of three weeks near Lorient, France. We
applied the $Q$-learning algorithm with the same value for $\alpha$ (Eq. 2)
and $\gamma=0.8$.

Figure 6:  Evolution of the battery charge level and measurements frequency of
sensors using the $Q$-learning algorithm with the reward function in Eq. R7.
Battery capacity=$5.2~{}$\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$$

The simulation results (Fig. 6) show that the agent adapts correctly the
measurements’ frequency to the battery’s behaviour. At the beginning of the
deployment (1), the battery level decreases quickly and the agent adjusts
almost immediately the frequency of measurements (2). Then, when the battery
level increases (4) or decreases (3) due to the harvested energy, the agent
reacts and adapts the frequency of measurements accordingly. The frequency of
measurements is maximum when the battery is fully charged. Before the end of
the simulation, the agent achieves the correct behaviour. The agent is able to
adjust the frequency of measurements to the battery charge in less than three
days, when it needs more than two weeks to adapt in the previous experiments.

The proposed reward function is able to improve greatly the learning speed,
the learning time is reduced by $81\%$. Furthermore, it adjusts the balance
between the performance and the battery level according to the battery level
without any balancing parameter. A second experiment is done with a smaller
battery in the same way as for the previous reward function. The results of
this experiment are shown in Figure 7.

Figure 7:  Evolution of the battery charge level and measurements frequency of
sensors using the $Q$-learning algorithm with the reward function in Eq. R7.
Battery capacity=$3.2~{}$\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$$

At the beginning of the deployment (1), the agent has no prior information
about the environment. However, when the battery charge level decreases (2)
the agent correctly adjusts the frequency of measurements, even if the daily
variation are not respected (3). Then, the battery charge level increases due
to the energy harvested and the agent increases the frequency as well. At the
end of the simulation (4), the agent seems to respect the daily variation in
the battery. To confirm the convergence of $Q$-values with the proposed reward
function, we conducted a slightly longer simulation of 24 days, shown in
Figure 8.

Figure 8:  Evolution of the battery charge level and measurements frequency of
sensors using the $Q$-learning algorithm with the reward function R7. Battery
capacity=$3.2~{}$\mathrm{mA}\text{\,}{\mathrm{h}}^{-1}$$, but number of
days=24.

The simulation results shown in Figure 8 confirms the convergence of the
$Q$-values. Furthermore, the agent ends up complying with the daily variation.
The differences between the two simulations are due to the exploration of the
environment. During the exploration, the agent takes random actions but the
agent’s policy converges to the same behaviour.

When compared to the results of the previous reward function (R6, Figures 4
and 5), we observe that the new proposed reward function adapts more
efficiently the measurements’ frequency to the battery charge level. This
reward function improves the performance when the battery charge level is
high. Another important point is that the reward functions in Eq. R6 and Eq.
R7 are both suitable regardless of the battery capacity. The system is seen by
the agent as a black box, it does not need to know the different components.

##  VII Conclusions and Perspectives

The use of energy harvesting technology creates a need for new energy
management algorithms. The approach using reinforcement learning is an
interesting solution for this type of problem. However, there are many
obstacles to overcome such as the dilemma between exploration and
exploitation, the choice of the learning rate value, and the definition of the
reward function.

The reward function influences the behaviour of the node. Its design is an
important part of the work and must be explained. In this paper, we experiment
different reward functions in a series of simulation to identify the best
design. We find out that including a balancing parameter to adjust the trade-
off between performance and energy consumption is the best solution.

These generic reward functions can achieve the same performance as a specific
reward function.

We conducted a series of experiments to design a reward function able to
balance energy consumption and performance. We found out that the most
effective reward functions (R1, R2) take into account the balance with a
$\beta$ parameter. They succeed in adjusting the energy consumption with the
energy harvested. Another effective reward function (R5) takes only into
account the dominant frequency of motion. The obtained behaviour adapts to
energy harvesting; however, it does not allow increasing the importance of
performance on energy consumption.

The other tested reward functions (R3, R4) did not allow a good energy
management. Either the reward function did not make a connection between the
node’s consumption and the energy harvested (in particular R3), or the reward
function did a connection but failed to choose less consuming actions when the
harvested energy is low (especially R4).

The design of the reward functions R1 and R2 is interesting. We propose a
modification to take account into the importance of energy consumption
depending on the battery state of charge using several balancing parameters.
The results show that the node adjust more efficiently its behaviour. Noticing
that the value of the balancing parameters is close the battery state of
charge, we propose two reward functions (R6 and R7) without fixed balancing
parameter which perform even better than the previous ones.

In a real application, the sensor nodes have more settings to adjust and
present more parameters to balance (e.g. several sensors with different
importance, several concurrent applications running on the same node), which
might require adaptation of the reward functions. The Q-learning algorithm is
not efficient for large Markov Decision Process. Indeed, it uses a look up
table and needs to explore several times each state-action pair. Nevertheless,
its neural version (Deep Q-learning) is able to scale for large environment.
Future work includes reward functions taking into account a maximum of
different parameters to adjust energy consumption according to the harvested
energy and performance requirements. We also plan to use Deep Q-learning to
cope with larger Markov Decision Processes.

## References

  * [1] X. Yuan, H. Liu, and J. Zhang, _Lithium-ion batteries: advanced materials and technologies_. CRC press, 2011.
  * [2] G. Zhou, L. Huang, W. Li, and Z. Zhu, “Harvesting ambient environmental energy for wireless sensor networks: a survey,” _Journal of Sensors_ , vol. 2014, 2014.
  * [3] S. Shresthamali, M. Kondo, and H. Nakamura, “Adaptive power management in solar energy harvesting sensor node using reinforcement learning,” _ACM Transactions on Embedded Computing Systems (TECS)_ , vol. 16, no. 5s, p. 181, 2017\.
  * [4] T. Ahmed and Y. Le Moullec, “A qos optimization approach in cognitive body area networks for healthcare applications,” _Sensors_ , vol. 17, no. 4, p. 780, 2017.
  * [5] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” _Journal of artificial intelligence research_ , vol. 4, pp. 237–285, 1996.
  * [6] S. Kim, R. Vyas, J. Bito, K. Niotaki, A. Collado, A. Georgiadis, and M. M. Tentzeris, “Ambient rf energy-harvesting technologies for self-sustainable standalone wireless sensor platforms,” _Proceedings of the IEEE_ , vol. 102, no. 11, pp. 1649–1666, 2014.
  * [7] Y. Rioual, Y. Le Moullec, J. Laurent, M. I. Khan, and J.-P. Diguet, “Reward function evaluation in a reinforcement learning approach for energy management,” in _2018 16th Biennial Baltic Electronics Conference (BEC)_. IEEE, 2018, pp. 1–4.
  * [8] E. Ibarra, A. Antonopoulos, E. Kartsakli, J. J. Rodrigues, and C. Verikoukis, “Qos-aware energy management in body sensor nodes powered by human energy harvesting,” _IEEE Sensors Journal_ , vol. 16, no. 2, pp. 542–549, 2016\.
  * [9] F. Casamassima, E. Farella, and L. Benini, “Context aware power management for motion-sensing body area network nodes,” in _Proceedings of the conference on Design, Automation & Test in Europe_. European Design and Automation Association, 2014, p. 170.
  * [10] R. Kazemi, R. Vesilo, and E. Dutkiewicz, “Dynamic power control in Wireless Body Area Networks using reinforcement learning with approximation,” _2011 IEEE 22nd International Symposium on Personal, Indoor and Mobile Radio Communications_ , pp. 2203–2208, 2011.
  * [11] C. Liu, X. Xu, and D. Hu, “Multiobjective reinforcement learning: A comprehensive overview,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , vol. 45, no. 3, pp. 385–398, 2015.
  * [12] J. Karlsson, “Learning to solve multiple goals,” Ph.D. dissertation, Citeseer, 1997.
  * [13] M. Humphrys, “Action selection methods using reinforcement learning,” _From Animals to Animats_ , vol. 4, pp. 135–144, 1996.
  * [14] Y. Zhao, Q. Chen, and W. Hu, “Multi-objective reinforcement learning algorithm for mosdmp in unknown environment,” in _2010 8th World Congress on Intelligent Control and Automation_. IEEE, 2010, pp. 3190–3194.
  * [15] P. Vamplew, R. Dazeley, A. Berry, R. Issabekov, and E. Dekker, “Empirical evaluation methods for multiobjective reinforcement learning algorithms,” _Machine learning_ , vol. 84, no. 1-2, pp. 51–80, 2011.
  * [16] S. Mannor and N. Shimkin, “A geometric approach to multi-criterion reinforcement learning,” _Journal of machine learning research_ , vol. 5, no. Apr, pp. 325–360, 2004.
  * [17] L. Barrett and S. Narayanan, “Learning all optimal policies with multiple criteria,” in _Proceedings of the 25th international conference on Machine learning_. ACM, 2008, pp. 41–47.
  * [18] R. S. Sutton and A. G. Barto, _Reinforcement learning: An introduction_. MIT press Cambridge, 1998, vol. 1, no. 1.
  * [19] G. Chen, Y. Zhan, G. Sheng, L. Xiao, and Y. Wang, “Reinforcement learning-based sensor access control for wbans,” _IEEE Access_ , vol. 7, pp. 8483–8494, 2019.
  * [20] J. Jagannath, N. Polosky, A. Jagannath, F. Restuccia, and T. Melodia, “Machine learning for wireless communications in the internet of things: A comprehensive survey,” _arXiv preprint arXiv:1901.07947_ , 2019.
  * [21] M. I. Khan and B. Rinner, “Energy-aware task scheduling in wireless sensor networks based on cooperative reinforcement learning,” in _IEEE International Conference on Communications Workshops (ICC), 2014_. IEEE, 2014, pp. 871–877.
  * [22] C. J. Watkins and P. Dayan, “Q-learning,” _Machine learning_ , vol. 8, no. 3-4, pp. 279–292, 1992.
  * [23] R. A. C. Bianchi, C. H. C. Ribeiro, and A. H. R. Costa, “Heuristically accelerated q–learning: A new approach to speed up reinforcement learning,” in _Advances in Artificial Intelligence – SBIA 2004_. Springer Berlin Heidelberg, 2004, pp. 245–254.
  * [24] M. Gorlatova, J. Sarik, G. Grebla, M. Cong, I. Kymissis, and G. Zussman, “Movers and shakers: Kinetic energy harvesting for the internet of things,” in _ACM SIGMETRICS Performance Evaluation Review_ , vol. 42, no. 1. ACM, 2014, pp. 407–419.