# Learning to Optimize Energy Efficiency in Energy Harvesting Wireless Sensor
Networks

Debamita Ghosh, Manjesh K. Hanawal and Nikola Zlatanov Debamita Ghosh is Ph.D.
student in the Dept. of IEOR, IITB-Monash Research Academy, IIT Bombay,
India.Manjesh K. Hanawal is with the Dept. of IEOR, IIT Bombay, India.Nikola
Zlatanov is with the Dept. of ECSE, Monash University, Australia.

###### Abstract

We study wireless power transmission by an energy source to multiple energy
harvesting nodes with the aim to maximize the energy efficiency. The source
transmits energy to the nodes using one of the available power levels in each
time slot and the nodes transmit information back to the energy source using
the harvested energy. The source does not have any channel state information
and it only knows whether a received codeword from a given node was
successfully decoded or not. With this limited information, the source has to
learn the optimal power level that maximizes the energy efficiency of the
network. We model the problem as a stochastic Multi-Armed Bandits problem and
develop an Upper Confidence Bound based algorithm, which learns the optimal
transmit power of the energy source that maximizes the energy efficiency.
Numerical results validate the performance guarantees of the proposed
algorithm and show significant gains compared to the benchmark schemes.

###### Index Terms:

Multi-Armed Bandits, Energy Harvesting, Wirelessly Powered Communication
Network, Energy Efficiency

##  I Introduction

Wirelessly powered communication networks (WPCNs), where nodes can perform
Energy Harvesting (EH), have recently drawn an upsurge in interest as a
possible technology for next generation wireless networks. EH helps to enhance
the life of wireless nodes by continuously charging their batteries either by
scavenging energy from the ambient atmosphere or from a dedicated energy
source. When a dedicated energy source is used for energy transmission, the
amount of energy it emits has to be appropriately calibrated. In the one hand,
high-power emissions may result in energy wastage due to the fundamental
limitations of the amount of energy EH devices can harvest. On the other hand,
low-power emissions may result in EH devices not being able to harvest enough
energy. It is thus important that the energy source transmits with the right
amount of power. As a result, the metric Energy Efficiency (EE), measured in
bits-per-joule, has drawn considerable attention as an essential performance
metric for optimizing the transmit power in WPCNs. Motivated by this, our goal
in this paper is to develop an algorithm that learns the optimal power level
of an energy source that maximizes the EE in WPCNs.

Several authors have investigated EE of various WPCNs. For example, the
authors in [1] study the maximization of EE in multi-users multiple-input-
single-output systems with Simultaneous Wireless Information and Power
Transfer (SWIPT). The authors in [2] study EE of Orthogonal Frequency Division
Multiple Acesss (OFDMA) systems employing SWIPT. The authors in [3], maximize
the weighted sum of the user’s EEs by jointly optimizing the used power for
downlink wireless energy transmission (WET) and for uplink wireless
information transmission (WIT). Similar joint transmitter and receiver
optimization of EE of OFDMA systems taking into account the effects of active
subcarriers and users is studied in [4]. The authors in [5] study the EE of
SWIPT in mobile wireless sensor networks considering the minimum individual
and system data rate requirements of the receivers, the minimum required power
transfer, and the maximum system power consumption.

The works [1]-[5], as well as others in the literature, study EE in WPCNs
assuming that the channel state information (CSI) is known at the receivers
and/or the transmitters. However, acquiring CSI consumes energy, which
diminishes the EE of the system. As a result, we propose a learning algorithm
that maximizes the EE of the system without any CSI knowledge. Specifically,
we consider a WPCN where the energy source transmits energy to EH nodes and
the nodes use the harvested energy to transmit information back to the energy
source at a fixed rate. The energy source then tries to decode the received
codewords and observes which nodes were successful in their transmission.
Using this limited information regarding the successful decoding of received
codewords, the source learns which power level to use in order to maximise the
EE of the system. To this end, we model the problem as a stochastic multi-
armed bandits problem and develop a learning algorithm to optimize EE.

Learning algorithms in WPCN were investigated recently in [6] and [7]. The
authors in [6] propose reinforcement learning (RL) to investigate the tradeoff
between EE and quality of service parameters, such as transmission delay, in
device-to-device communications underlaying cellular networks. The authors in
[7] propose a RL based offloading policy for EH IoT devices to select the edge
devices for computationally intensive applications. These learning algorithms
require that the devices either have CSI or know additional information such
as the available battery power, tasks which are hard to realize in practice
and also that diminish the EE. n this paper, we adopt the multi-armed bandit
framework for solving the EE problem since it enables us to establish
theoretical guarantees on the performance of our algorithm. Thereby, we
propose an Upper Confidence Bound (UCB) based learning algorithm using which
the energy source maximizes the EE without any CSI and only based on whether a
received codeword was successfully decoded or not. To the best of our
knowledge, maximizing the EE of a WPCN by optimizing the power selection
strategy without CSI knowledge has not been studied previously.

##  II System Model

We consider a system model comprised of an energy source, referred to as the
source, and $k$ EH wireless nodes, referred to as the nodes. The source and
the nodes employ frequency-division duplex (FDD)111Alternatively, we can also
assume that the source and the nodes employ time-division duplex (TDD) on
$k+1$ time slots in order to avoid interference. on $k+1$ dedicated channels
where each channel occupies a different frequency band. Specifically, the
source radiates energy on its dedicated channel, which is then harvested by
the nodes and used to transmit information back to the source on their
respective dedicated channels, cf. Fig. 1. Note that since FDD is used,
interference does not occur. The source also serves as a receiver of
information. We assume that the dedicated channels are flat-fading channels
with channel gains that are constant in each time slot and change
independently from one time slot to the next. Let $G_{j}(t)$ denote the
complex-valued channel gain between the source and node $j$ in time slot $t$
on the channel dedicated for energy transmission, where
$G_{j}(t)\sim\mathbb{CN}\\{0,\sigma_{G_{j}}^{2}\\}$, i.e., complex Gaussian
distributed with mean zero and variance $\sigma^{2}_{G_{j}}$. Let $H_{j}(t)$
denote the complex-valued channel gain between node $j$ and the source in time
slot $t$ on the channel dedicated for information transmission by node $j$,
where $H_{j}(t)\sim\mathbb{CN}\\{0,\sigma_{H_{j}}^{2}\\}$. We also assume that
the received signal at the source is corrupted by complex zero-mean additive
white Gaussian noise (AWGN) with variance $\sigma^{2}$.

The source has a set of $m$ powers222The proposed algorithm works also with a
continuum of power, provided we quantize the continuous power into discrete
power. Then, the proposed algorithm will learn the optimal discrete power that
is close to the optimal continuous power. denoted by
$\mathcal{P}=\\{p_{1},p_{2},\dots,p_{m}\\}$. In each time slot $t$, the source
can select any power in $\mathcal{P}$ to transmit energy. Let
$P(t)\in\mathcal{P}$ denote the power selected in time slot $t$. The amount of
energy harvested by node $j$ in time slot $t$ from power $P(t)$, denoted by
$E_{j}(t)$, is given by

$\displaystyle E_{j}(t)=\min\\{b_{\max},\max\left(0,\lambda
P(t)|G_{j}(t)|^{2}-p_{\min}\right)\\},$ (1)

where $p_{\min}$ is the minimum required power for each node to operate,
$0\leq\lambda<1$ is the EH inefficiency coefficient, and $b_{\max}$ is the
maximum capacity of the battery of the nodes. We assume that the nodes use the
entire energy harvested in time slot $t-1$ for information transmission in the
next time slot $t$. Each node transmits information using a capacity-achieving
codeword with rate $r_{0}$. The received rate at the source from node $j$ in
time slot $t$ is denoted by $R_{j}(t)$ and is given by

$\displaystyle R_{j}(t)$ $\displaystyle=O_{j}(t)r_{0},$ (2)

where $O_{j}(t)$ is a binary variable that models success or failure of the
decoding of the codeword of node $j$ at the source, given as

$\displaystyle O_{j}(t)$
$\displaystyle=\left\\{\begin{array}[]{ll}1&\textrm{if
}\log_{2}\left(1+\frac{E_{j}(t-1)|H_{j}(t)|^{2}}{\sigma^{2}}\right)>r_{0}\\\
0&\textrm{otherwise}.\end{array}\right.$ (5)

Using $R_{j}(t),\forall j$, we can define the weighted sum rate received at
the source in time slot $t$ as

$\displaystyle R_{sum}(t)=\sum\limits_{j=1}^{k}\omega_{j}R_{j}(t),$ (6)

where $0\leq\omega_{j}\leq 1,\forall j$, and $\sum_{j=1}^{k}\omega_{j}=1$. The
weights $\omega_{j},\forall j$, are used to give different priorities to the
users.

Figure 1: Energy Harvesting setup. Refer [8] for a detailed architecture.

##  III Problem Formulation

In the following, we formulate the problem and develop a learning algorithm to
maximize the EE in the WPCN.

###  III-A Problem Setting

As we increase the transmit power of the source, the amount of energy
harvested by the nodes increases only to a certain point and then saturates
due to (1). As a result, further increase in the transmit power of the source
will not increase the weighted sum rate received at the source given by (6),
thereby leading to a wastage of energy. Hence, there must be some optimal
power level which maximizes the weighted sum rate per energy spent. To
investigate this point, we define the EE as a ratio between the weighted sum
rate received at the source and source’s transmit power over $n$ time slots
given as

$\displaystyle
EE(n)=\frac{1}{n}\sum_{t=1}^{n}\sum_{j=1}^{k}\frac{\omega_{j}R_{j}(t+1)}{P(t)}.$
(7)

Our goal is to identify a fixed power level that maximizes the expected EE,
i.e., we solve the following optimization problem

$\displaystyle\arg\max\limits_{p\in\mathcal{P}}\mathbb{E}\left[EE(n)\mid
P(t)=p,\;\forall\;t\right].$ (8)

The rationale for seeking a fixed power level in all time slots is because the
source does not have CSI and hence adapting the power levels in each time slot
is not useful. The only information available to the source is whether a
received codeword from a given node was successfully decoded or not. Based on
this limited information, we derive a strategy for the source to learn the
optimal power that maximizes the expected value of the EE defined by (8).

###  III-B Notation

We refer to each power in the set $\mathcal{P}$ simply by its index. Let
$\boldsymbol{R}(t)=\\{R_{1}(t),R_{2}(t),...,R_{k}(t)\\}$ denote the vector of
received rates at the source in time slot $t$, where $R_{j}(t)$ is the
received rate from node $j$, given by (2). Note that the vector
$\boldsymbol{R}(t)$ depends on the energy harvested by the nodes in the
previous time slot $t-1$. At $t=1$, $R_{j}(1)=0,\forall j$, as the nodes do
not have any energy harvested from $t=0$. Let the history at the source in
time slot $t$ be defined as
$\mathcal{H}(t):=\left\\{(P(1),\boldsymbol{R}(1)),(P(2),\boldsymbol{R}(2)),..,(P(t),\boldsymbol{R}(t))\right\\}$
with $\mathcal{H}(0)=\phi$. The power selection policy of the source denoted
by $\Pi=\left\\{\pi(t)\right\\}_{t=1}^{\infty}$ is a sequence of maps
$\pi(t):\mathcal{H}(t-1)\rightarrow\mathcal{P}$. Our goal is to find a policy
that maximizes the expected EE of the system given by (8). To this end, we set
up the problem as a Multi-Armed Bandits (MAB) problem, where the source is the
learner and the set of transmit powers at the source constitute the arms.

Let $R_{j}^{i}(t)$ denote the rate received at the source from node $j$ in
time slot $t$, when the transmit power of the source in time slot $t-1$ was
$P(t-1)=i$. For each node $j$ and power $i$, the random process
$\left\\{R_{j}^{i}(t)\right\\}_{t\geq 2}$ is an independent and identically
distributed (i.i.d.) process with a mean denoted by $\mu_{ij}$ and given by
$\mathbb{E}\left[R_{j}(t+1)|P(t)=i\right]=\mu_{ij}$. We denote the power that
maximizes the expected EE by $i^{*}$ and denote the corresponding optimal
expected EE by $\mu_{i^{*}}$, where

$\displaystyle i^{*}=\operatorname*{arg\,max}_{i\in
m}\sum_{j=1}^{k}\frac{\omega_{j}\mu_{ij}}{p_{i}}\quad\text{and}\quad\mu_{i^{*}}=\sum_{j=1}^{k}\frac{\omega_{j}\mu_{i^{*}j}}{p_{i^{*}}}.$
(9)

We define the expected regret that evaluates the performance gap between the
proposed strategy and the optimal power selection strategy on average over $n$
time slots as
$\mathcal{R}_{n}=n\mu_{i^{*}}-\sum_{t=1}^{n}\sum_{j=1}^{k}\omega_{j}\mathbb{E}\left[\frac{\mu_{I_{t}j}}{p_{I_{t}}}\right]$,
where $I_{t}$ is the power index selected in time slot $t$ and
$p_{I_{t}}=P(t)$ is the associated transmit power. Hence, maximizing the
average EE is equivalent to designing a learning algorithm that minimises the
expected regret by learning the optimal power level at the source.

###  III-C The Proposed Algorithm

We develop an algorithm for optimal power selection using the well known UCB
approach [9]. We begin by giving a concentration bound on the sum of weighted
random variables that will be used in the algorithm. Let
$\hat{\mu}_{ij}^{(s)}$ denote the empirical mean obtained from $s$ i.i.d.
samples of the received rate at the source from node $j$ using the energy
harvested from power $i$, given by
$\hat{\mu}_{ij}^{(s)}=\frac{1}{s}\sum_{t=1}^{s}R_{j}^{i}(t)$. Since
$R_{j}^{i}(t)\in\\{0,r_{0}\\}$, by Hoeffding’s Lemma [10, eq. (2.2)] there
exits a convex function $\psi:\mathbb{R}\rightarrow\mathbb{R}_{+}$ such that
for all $a>0$,

$\displaystyle\mathbb{E}\left[\exp(a(\mu_{ij}-R_{j}^{i}))\right]\leq\exp(\psi(a)),$
(10)

where the convex function $\psi(a)=a^{2}r_{0}^{2}/8$.

Using (10) we get the following concentration bound on the weighted sum rate
received at the source.

######  Lemma 1.

For any $s$, we have

$\displaystyle\mathbb{P}\left\\{\sum\limits_{j=1}^{k}\omega_{j}\mu_{ij}-\sum\limits_{j=1}^{k}\omega_{j}\hat{\mu}_{ij}^{(s)}>\epsilon\right\\}\leq\exp(-s\psi^{*}(\epsilon)),$

where
$\psi^{*}(\epsilon)=\sup_{a\in\mathbb{R}}(\alpha\epsilon-\sum_{j=1}^{k}\omega_{j}^{2}\psi(a))$
is the (weighted) Legendre-Fenchel transform of $\psi(\alpha)$.

Proof: Please refer to Appendix A.

Lemma 1 states that with probability $1-\delta$,
$\sum_{j=1}^{k}\omega_{j}\hat{\mu}^{(s)}_{ij}+(\psi^{*})^{-1}\left(\frac{1}{s}\log\frac{1}{\delta}\right)>\sum_{j=1}^{k}\omega_{j}\mu_{ij}$
holds $\forall i\in m.$ Using this lemma we define UCB for each power $i$ in
time slot $t>m$ as

$\displaystyle\mbox{UCB}_{i}(t)=\sum\limits_{j=1}^{k}\omega_{j}\hat{\mu}_{ij}(t-1)+(\psi^{*})^{-1}\left(\frac{\alpha\ln
t}{N_{i}(t-1)}\right),$ (11)

where $\alpha$ is an input parameter to the algorithm,
$N_{i}(t)=\sum_{s=1}^{t}\mathbb{I}\\{I_{s}=i\\}$ denotes the number of times
the power $i$ is selected till time slot $t$ and
$(\psi^{*})^{-1}\left(\frac{\alpha\ln
t}{N_{i}(t-1)}\right)=r_{0}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}}$. The larger the value of
$\alpha$, more is the exploration. We propose a UCB based algorithm whose
pseudo-code is given in Algorithm: UCB-EH, which works as follows. The UCB-EH
algorithm takes $m$ powers, $k$ nodes, and $\alpha$ as inputs, where $\alpha$
is a constant to be specified later. In the first $m$ time slots, each power
is selected in a round-robin fashion. In each of the subsequent time slot $t$,
the UCB value of power $i$ is calculated as per (11). The source then selects
the power index $i$ which maximizes the ratio $\text{UCB}_{i}(t)/p_{i}$. The
maximizing power index is denoted by $I_{t}$ and the transmit power by
$p_{I_{t}}$. For the selected $I_{t}$, the source then observes the received
rates $R_{j}^{I_{t}}(t)$ and updates the empirical means
$\hat{\mu}_{I_{t}j}(t)$ for all of the nodes.

UCB-EH

1: Input: $m,k,\alpha\text{ and }\omega_{j},\forall j$

2: Select each transmit power once in first $m$ time slots

3: Update $\hat{\mu}_{ij}$ for all $i\in m,j\in k$

4: for $t=m+1,m+2,\ldots,n$ do

5: For each power $i\in m$ calculate
$\displaystyle\text{UCB}_{i}(t)\leftarrow\sum\limits_{j=1}^{k}\omega_{j}\hat{\mu}_{ij}(t-1)+r_{0}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}}$

6: Set
$I_{t}\leftarrow\operatorname*{arg\,max}\limits_{i}\frac{\text{UCB}_{i}(t)}{p_{i}}$
and transmit power $p_{I_{t}}$

7: Observe received rate $R_{j}^{I_{t}}(t)$ for all $j\in k$

8: Update the estimates of $\hat{\mu}_{I_{t}j}(t)$ for all $j\in k$

9: end for

###  III-D Regret Analysis

Now we give a theoretical guarantee on the performance of the proposed UCB-EH
algorithm.

######  Theorem 1.

Let $\Delta_{i}=\mu_{i^{*}}-\sum_{j=1}^{k}(\omega_{j}\mu_{ij}/p_{i})$ be the
sub-optimality gap of power $i$ and $\Delta=\min\limits_{i\neq
i^{*}}\Delta_{i}$. Then, the regret of UCB-EH after $n$ rounds is upper
bounded as

$\displaystyle\mathcal{R}_{n}$ $\displaystyle\leq
6r_{o}^{2}\sum\limits_{i:\Delta_{i}>0}\frac{\ln
n\sum_{j=1}^{k}\omega_{j}^{2}}{p_{i}^{2}\Delta_{i}}+\sum\limits_{i:\Delta_{i}>0}\left(\frac{\pi^{2}}{3}+1\right)\Delta_{i},$
(12)

i.e., the regret is of the order $O\left(\frac{m\ln
n\sum_{j=1}^{k}\omega_{j}^{2}}{\Delta}\right)$. Further, the problem
independent bound is given as $\mathcal{R}_{n}\leq O\left(\sqrt{nm\ln
n\sum_{j=1}^{k}\omega_{j}^{2}}\right).$

Proof: Please refer to Appendix A.

As seen from Theorem 1, the proposed UCB-EH algorithm is asymptotically
optimal and thereby maximizes the expected EE. Moreover, the complexity of the
proposed algorithm grows linearly with the number of power levels $m$.

##  IV Numerical Results

Assuming Rayleigh fading, $|G_{j}(t-1)|^{2}$ and $|H_{j}(t)|^{2}$ in
$E_{j}(t-1)$ in (1) are exponentially distributed RVs with means
$2\sigma_{G_{j}}^{2}$ and $2\sigma_{H_{j}}^{2}$, respectively. The variances
$\sigma_{G_{j}}^{2}$ and $\sigma_{H_{j}}^{2}$ are obtained using the free-
space path loss model as $\sigma_{x_{j}}^{2}=\frac{1}{2}\left(\frac{c}{4\pi
f_{x_{j}}}\right)^{2}d^{-\gamma}_{j},x\in\\{G,H\\}$, where $c$ denotes the
speed of light, $f_{x_{j}}$ for $x\in\\{G,H\\}$ is the carrier frequency of
the signal to/from the node $j$, $d_{j}$ is the length of the link between the
source and node $j$, and $\gamma$ is the path loss exponent. Moreover, the
noise power $\sigma^{2}$ in (5) is given by $\sigma^{2}=W\sigma_{0}^{2}$,
where $W$ is the bandwidth and $\sigma_{0}^{2}$ is the noise power per Hz. For
our numerical simulations, we assume a bandwidth $W=100$ kHz and noise power
per Hz $\sigma^{2}_{0}=-170$ dBm/Hz, that makes the total noise power
$\sigma^{2}=-120$ dBm. We assume carrier frequencies as $f_{G_{j}}=2.4$ GHz
and $f_{H_{j}}=2.4+(10^{-3}\times j)$ GHz, the length $d_{j}=10+(3\times j)$
m, $\omega_{j}=1/k$ for $j=1,2,..k$. Furthermore, we consider the path loss
exponent $\gamma$ = 2.5, the EH inefficiency coefficient $\lambda=0.5$, the
minimum power $p_{\min}=-60$ dBm, the maximum battery capacity
$b_{\max}\geq-40$ dBm, and $\alpha=3$. For our EH setup, we consider the set
of transmit powers as $\mathcal{P}=\\{0,1,2,\dots,30\\}$ in dBm.

We empirically evaluate the performance of our proposed algorithm UCB-EH in
terms of average EE and compare it with a benchmark scheme without CSI. In the
benchmark scheme without CSI the source always transmits with the maximum
power to charge the EH nodes, i.e., it transmits with power $30$ dBm. We
define the oracle scheme as the scheme based on the optimal policy which
maximizes the energy efficiency of the system. The aim is our algorithm to
learn the oracle scheme with time. We measure the performance of our algorithm
for $T=10^{4}$ time slots and the results are averaged over $1000$
repetitions.

In Fig. 2, we show the EE achieved with the proposed algorithm, the oracle
scheme, and the benchmark scheme without CSI for different numbers of nodes
$k=\\{4,8,12\\}$ and $r_{0}=0.1$ bpcu as a function of the number of time
slots $n$. We find that the EE decreases as the number of nodes increases for
a fixed number of transmit powers, as expected. As Fig. 2 shows, our algorithm
achieves an EE that converges toward the EE achieved with the oracle.
Moreover, our algorithm significantly outperforms the benchmark scheme without
CSI for any $k$ and for any time slot.

Figure 2: Energy Efficiency v/s Time Slots.

In Fig. 3, we show the EE achieved with the proposed algorithm, the oracle
scheme, and the benchmark scheme without CSI as a function of the rate $r_{0}$
when we have $k=5$ nodes. Initially, as we increase the value of $r_{0}$, more
number of nodes can successfully transmit a codeword with rate $r_{0}$ to the
source for a fixed transmit power level at the source. As a result, the
expected EE of the system increases and reaches its maximum point. But as
$r_{0}$ increases further, more number of nodes fail to successfully transmit
a codeword with rate $r_{0}$ to the source which will decrease the EE of the
system. As Fig. 3 shows, our algorithm achieves an EE which is $52\%$ higher
than the benchmark scheme without CSI and just $9\%$ lower than the oracle
scheme. Moreover, to get the system’s highest EE, our algorithm and the oracle
scheme need a rate of $0.75$ bpcu, while the benchmark scheme without CSI
requires $2$ bpcu to get the system’s highest EE.

Figure 3: Energy Efficiency v/s Rate $r_{0}$ (in bpcu).

In Fig. 4, we compare the EE of our scheme with a benchmark scheme that has
CSI knowledge of all channels for $k=8$. For a fair comparison, we have
included in the benchmark scheme the cost of acquiring CSI. As can be seen
from Fig. 4, the benchmark scheme with CSI outperforms our algorithm only if
the CSI cost is very small, i.e., less than -60 dBm in this example, which is
not possible in practice since CSI estimation consumes a fair bit of energy.

Figure 4: Energy Efficiency with and without CSI.

##  V Conclusion

We investigated the problem of optimal power selection at a dedicated source
that wirelessly charges EH nodes in a WPCN, for improving the EE of the system
without any CSI knowledge. To address this problem, we proposed an online
learning approach based on the theory of multi-armed bandit that selects the
optimal transmit power which maximizes the EE of the system. The simulation
results shows that the proposed algorithm learns the optimal power and it
significantly outperforms the benchmark schemes.

## Appendix A

###### Proof of Lemma 1.

Let $\hat{\mu}_{ij}^{(s)}$ denote the empirical mean based on $s$ i.i.d.
samples of node $j$ using the energy harvested from power $i$. Using Markov’s
inequality for some $a>0$, we obtain

$\displaystyle\mathbb{P}\left\\{\sum\limits_{j=1}^{k}\omega_{j}\mu_{ij}-\sum\limits_{j=1}^{k}\omega_{j}\hat{\mu}_{ij}^{(s)}>\epsilon\right\\}$
$\displaystyle=e^{-(as\epsilon)}\prod_{j=1}^{k}\prod_{t=1}^{s}\mathbb{E}\left[\exp\left\\{a\omega_{j}(\mu_{i,j}-R^{i}_{j}(t))\right\\}\right]$
(13) $\displaystyle\leq
e^{-(as\epsilon)}\prod_{j=1}^{k}\prod_{t=1}^{s}\exp\\{\psi(a\omega_{j})\\}\leq\exp\\{-s\psi^{*}(\epsilon)\\}.$
(14)

As $\mathbb{R}_{j}^{i}(t)$ are i.i.d. across time slots and nodes, we obtain
(13). We insert (10) and $\psi(a\omega_{j})=\omega_{j}^{2}\psi(a)$ in (13) to
get (14). ∎

###### Proof of Theorem 1.

Without loss of generality, let $i^{*}=1$ be the optimal index of the power.
Hence, the optimal power is $p_{1}$. Recall that by convention
$\Delta_{i}=\sum_{j=1}^{k}\omega_{j}\left(\frac{\mu_{1j}}{p_{1}}-\frac{\mu_{ij}}{p_{i}}\right)$.
If $I_{t}=i$ is a suboptimal power selected in time slot $t$, then at least
one of the following equations must be true

$\displaystyle\sum\limits_{j=1}^{k}\omega_{j}\hat{\mu}_{1j}(t-1)+r_{0}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}}\leq\sum\limits_{j=1}^{k}\omega_{j}\mu_{1j},$
(15)
$\displaystyle\sum\limits_{j=1}^{k}\omega_{j}\hat{\mu}_{ij}(t-1)-r_{0}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}}\geq\sum\limits_{j=1}^{k}\omega_{j}\mu_{ij},$
(16)
$\displaystyle\qquad\qquad\qquad\Delta_{i}<\frac{2r_{0}}{p_{i}}\sqrt{\frac{\alpha\ln
n\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}}.$ (17)

Let us assume that all three conditions are false. Then, we have

$\displaystyle\sum\limits_{j=1}^{k}\frac{\omega_{j}\hat{\mu}_{1j}(t-1)}{p_{1}}+\frac{r_{0}}{p_{1}}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}}\geq\sum\limits_{j=1}^{k}\frac{\omega_{j}\mu_{1j}}{p_{1}}$
(18a) $\displaystyle\geq\frac{2r_{0}}{p_{i}}\sqrt{\frac{\alpha\ln
n\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}}+\sum\limits_{j=1}^{k}\frac{\omega_{j}\mu_{ij}}{p_{i}}$
(18b)
$\displaystyle\geq\sum\limits_{j=1}^{k}\frac{\omega_{j}\hat{\mu}_{ij}(t-1)}{p_{i}}+\frac{r_{0}}{p_{i}}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2N_{i}(t-1)}},$ (18c)

where (18a) follows by negating (15), (18b) follows by the definition of
$\Delta_{i}$ and by using the negation of (17), and (18c) follows by applying
the negation of (16) and $t\leq n$ in (18b). This contradicts the hypothesis
that $I_{t}=i$.

Hence, if $I_{t}=i$ then $\\{N_{i}(t)\leq u\\}$ where we consider
$u=\left\lceil\frac{2r_{0}^{2}\alpha\ln
n\sum_{j=1}^{k}\omega_{j}^{2}}{p_{i}^{2}\Delta_{i}^{2}}\right\rceil$. We know
$N_{i}(n)=\sum\limits_{t=1}^{n}\mathbb{I}\left\\{I_{t}=i\right\\}$. Therefore,

$\displaystyle\mathbb{E}\left[N_{i}(n)\right]=\mathbb{E}\left[\sum\limits_{t=1}^{n}\mathbb{I}\left\\{I_{t}=i\right\\}\right]\leq
u+\sum\limits_{t=1}^{n}\mathbb{P}\left\\{I_{t}=i,N_{i}(t)>u\right\\}.$ We know
that if (17) is false and $I_{t}=i$, then (15) or (16) must hold. So we get
$\displaystyle\mathbb{E}\left[N_{i}(n)\right]\leq
u+\sum\limits_{t=1}^{n}\mathbb{P}\left\\{\text{\eqref{eqn:UnderEstimate}) or
\eqref{eqn:OverEstimate} must hold}\right\\}.$ By applying the union bound, we
get $\displaystyle\mathbb{E}\left[N_{i}(n)\right]\leq
u+\sum\limits_{t=1}^{n}[\mathbb{P}\left\\{\eqref{eqn:UnderEstimate}\mbox{
holds}\right\\}+\mathbb{P}\left\\{\eqref{eqn:OverEstimate}\mbox{
holds}\right\\}],$ (19)

where

$\displaystyle\sum\limits_{t=1}^{n}\mathbb{P}\left\\{(\ref{eqn:UnderEstimate})\mbox{
holds}\right\\}\leq\sum\limits_{t=1}^{n}\mathbb{P}\Biggl{\\{}\exists
s\in\\{1,2,..,t\\};\sum\limits_{j=1}^{k}\omega_{j}\hat{\mu}_{1j}^{(s)}$
$\displaystyle+r_{0}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2s}}\leq\sum\limits_{j=1}^{k}\omega_{j}\mu_{1j}\Biggr{\\}}$
$\displaystyle\leq\sum\limits_{t=1}^{n}\sum\limits_{s=1}^{t}\mathbb{P}\left\\{\sum\limits_{j=1}^{k}\omega_{j}\left(\mu_{1j}-\hat{\mu}_{1j}^{(s)}\right)\geq
r_{0}\sqrt{\frac{\alpha\ln t\sum_{j=1}^{k}\omega_{j}^{2}}{2s}}\right\\}.$
$\displaystyle\leq\sum\limits_{t=1}^{n}\sum\limits_{s=1}^{t-1}e^{-s\psi^{*}\left(r_{0}\sqrt{\frac{\alpha\ln
t\sum_{j=1}^{k}\omega_{j}^{2}}{2s}}\right)}\qquad\text{By Lemma
\ref{lem:ConcentrationBound1}}$
$\displaystyle\leq\sum\limits_{t=1}^{n}\sum\limits_{s=1}^{t-1}e^{-\alpha\ln
t}\qquad\qquad\text{as
$\psi^{*}(\epsilon)=\frac{2\epsilon^{2}}{r_{0}^{2}\sum_{j=1}^{k}\omega_{j}^{2}}$}$
$\displaystyle\leq\sum\limits_{t=1}^{n}\sum\limits_{s=1}^{t}\frac{1}{t^{\alpha}}\leq\sum\limits_{t=1}^{\infty}\frac{1}{t^{\alpha-1}}.$
$\displaystyle\text{Considering $\alpha=3$, we get
}\sum\limits_{t=1}^{n}\mathbb{P}\left\\{\eqref{eqn:UnderEstimate}\mbox{
holds}\right\\}\leq\frac{\pi^{2}}{6}.$ (20) $\displaystyle\text{Similarly, we
get
\quad}\sum\limits_{t=1}^{n}\mathbb{P}\left\\{\eqref{eqn:OverEstimate}\mbox{
holds}\right\\}\leq\frac{\pi^{2}}{6}.$ (21) By inserting $u$, (20), and (21)
into (19), we get
$\displaystyle\mathbb{E}\left[N_{i}(n)\right]\leq\frac{6r_{0}^{2}\ln
n\sum_{j=1}^{k}\omega_{j}^{2}}{p_{i}^{2}\Delta_{i}^{2}}+\left(\frac{\pi^{2}}{3}+1\right).$
(22) By the Regret Decomposition Lemma [9], we have
$\displaystyle\mathcal{R}_{n}=\sum\limits_{i:\Delta_{i}>0}\Delta_{i}\mathbb{E}\left[N_{i}(n)\right].$
(23) $\displaystyle\text{Inserting \eqref{eqn:SampleBound} into
\eqref{eqn:regret decomposition} we get the desired bound in
\eqref{regret:Thm1}}.\qed$

## References

  * [1] Q. Shi, C. Peng, W. Xu, M. Hong, and Y. Cai, “Energy efficiency optimization for MISO SWIPT systems with zero-forcing beamforming,” _IEEE Transactions on Signal Processing_ , vol. 64, no. 4, pp. 842–854, 2015.
  * [2] D. W. K. Ng, E. S. Lo, and R. Schober, “Wireless information and power transfer: Energy efficiency optimization in OFDMA systems,” _IEEE Transactions on Wireless Communications_ , vol. 12, no. 12, pp. 6352–6370, 2013\.
  * [3] Q. Wu, W. Chen, D. W. K. Ng, J. Li, and R. Schober, “User-centric energy efficiency maximization for wireless powered communications,” _IEEE Transactions on Wireless Communications_ , vol. 15, no. 10, pp. 6898–6912, 2016\.
  * [4] Q. Wu, W. Chen, M. Tao, J. Li, H. Tang, and J. Wu, “Resource allocation for joint transmitter and receiver energy efficiency maximization in downlink OFDMA systems,” _IEEE Transactions on Communications_ , vol. 63, no. 2, pp. 416–430, 2014.
  * [5] S. Guo, Y. Shi, Y. Yang, and B. Xiao, “Energy efficiency maximization in mobile wireless energy harvesting sensor networks,” _IEEE Transactions on Mobile Computing_ , vol. 17, no. 7, pp. 1524–1537, 2017.
  * [6] Y. Luo, M. Zeng, and H. Jiang, “Learning to tradeoff between energy efficiency and delay in energy harvesting-powered D2D communication: A distributed experience-sharing algorithm,” _IEEE Internet of Things Journal_ , vol. 6, no. 3, pp. 5585–5594, 2019.
  * [7] M. Min, L. Xiao, Y. Chen, P. Cheng, D. Wu, and W. Zhuang, “Learning-based computation offloading for IoT devices with energy harvesting,” _IEEE Transactions on Vehicular Technology_ , vol. 68, no. 2, pp. 1930–1941, 2019.
  * [8] X. Lu, P. Wang, D. Niyato, D. I. Kim, and Z. Han, “Wireless networks with RF energy harvesting: A contemporary survey,” _IEEE Communications Surveys & Tutorials_, vol. 17, no. 2, pp. 757–789, 2014.
  * [9] T. Lattimore and C. Szepesvári, _Bandit algorithms_. Cambridge University Press, 2020.
  * [10] S. Bubeck, N. Cesa-Bianchi _et al._ , “Regret Analysis of Stochastic and Nonstochastic Multi-Armed Bandit Problems,” _Foundations and Trends® in Machine Learning_ , vol. 5, no. 1, pp. 1–122, 2012\.


11institutetext: Radboud University, Nijmegen, The Netherlands 11email:
<EMAIL_ADDRESS>
22institutetext: IKERLAN Research Center, Mondragon/Arrasate, Spain
22email<EMAIL_ADDRESS>
# Playing with blocks

Toward re-usable deep learning models for side-channel profiled attacks

Servio Paguada ID 1122 Lejla Batina ID 11 Ileana Buhan 11 Igor Armendariz ID
11

###### Abstract

This paper introduces a deep learning modular network for side-channel
analysis. Our deep learning approach features the capability to exchange part
of it (modules) with others networks. We aim to introduce reusable trained
modules into side-channel analysis instead of building architectures for each
evaluation, reducing the body of work when conducting those. Our experiments
demonstrate that our architecture feasibly assesses a side-channel evaluation
suggesting that learning transferability is possible with the network we
propose in this paper.

###### Keywords:

side-channel analysis modular network deep learning autoencoders transfer
learning

##  1 Introduction

In the side-channel analysis (SCA) field of research, deep learning models (DL
models) are powerful tools to evaluate the implementation of secure
algorithms. Unfortunately, despite the significant accomplishments by using
deep learning models, many challenges remains.

When evaluating a secure implementation of an IoT device, for example, it is
challenging to develop a deep learning classifier that feasibly assesses the
resilience of the devices. Electronic noise as countermeasure and
desynchronization are specific challenges during the evaluation. Indeed, a
noisy signal intrinsically suggests dealing with high-dimensional signals. For
instance, targeting a modern System-on-Chip with high clock frequencies
requires increasing the sampling resolution; as consequence, the side-channel
information required for the evaluation contains leakage traces with several
irrelevant features (sample points). Then, noise filters and feature
engineering as pre-processing steps are being reconsidered as tools to deal
with those challenges [22, 19, 27, 16, 13, 11].

This paper proposes a new technique to overcome those challenges and
introduces a novel approach that uses a deep learning classifier whose part of
its architecture allows to be re-used in other models that use the same
approach. By featuring the exchangeable modules, we can re-use networks for
different SCA evaluations, reducing the body of work of deriving models each
time. The suggested architecture comprises coupled modules, and those modules
have specific tasks to deal with the challenges of an SCA evaluation. We call
this approach DL-SCA modular network. Precisely, an autoencoder and a
convolution base classifier are the two modules that we suggest in this paper.

An autoencoder can effectively deal with the problem of high dimensionality
and the problem of noise. An autoencoder comprises two parts by itself; the
encoder and decoder. The encoder ends to an embedding where high dimensional
leakage traces are transformed into lower dimensional version of them. Because
of it, autoencoder are learning algorithms used in pre-processing steps i.e
feature extraction [15, 19, 7].

The classifier module serves two objctives; (i) the classification required
for the SCA evaluation and (ii) to regularize the autoencoder. As we explain
in further sections of this paper, autoencoders might fail to compress the
samples taken from the device under test; so penalizing it with a
regularization might correct it toward better performance.

Our experiment uses datasets with desyncronization and countermeasures. After
proving the effectiveness of the DL-SCA modular network, we perform a second
set of experiments where we exchange the modules between modular networks. Our
results show that transferability is feasible and applicable to side-channel
analysis. The contributions of this paper are as follow:

  * •

We introduce an approach called DL-SCA modular network and deep learning
architecture featuring the exchange of modules through models. We provide the
implementation details of the architecture, as well as the hyperparameter to
take into account in the design to avoid pitfalls.

  * •

We present a training strategy based on sharing weight technique and early
stopping policy for a seamlessly adoption of our approach in current SCA
evaluations.

  * •

We elaborate experiments that demonstrate the effectiveness of re-using
modules through modular networks, using different ”sharing” protocols based on
non-trainable layers.

The rest of the paper is organized as follows: Sect. 2 details theoretical
aspects of the topics used for this work. Related works are discussed in Sect.
3. Sect. 4 provides information about datasets used in the experiments. Sect.
5 discuss the main contribution of this paper. Sect. 6 and Sect. 7 discuss the
experiments. While, Sect. 8 concludes the paper.

##  2 Background

###  2.1 Profiled attack

A side-channel attack requires a leakage model to attack the sensitive
information contained in a target device. A leakage model refers to a function
($\delta$) that models the leak of sensitive information. Using a leakage
model, an adversary can steal the secret key from a device that implements a
cryptographic algorithm. The expression (1) is an example of leakage model
used to attack a cryptographic implementation of AES111Or any other
cryptographic primitive with non-linear functions. In that leakage model, $p$
is the publicly available data i.e. the plaintext and $k^{*}$ is the secret
key.

$\delta=\text{S-box}(p\mathbin{\oplus}k^{*})$ (1)

The adversary measures the power consumption222Being the most traditional
measurement used in SCA, others are Electromagnetic Emanation, heat, sound,
and a few more. when the device inputs the AES algorithm with $p$ random
values from the keyspace $\mathcal{K}=\\{0,\cdots 255\\}$, drawing several
leakage traces (a.k.a power traces). With enough leakage traces, the adversary
can find a correlation between the power consumption and the inputs $p$ of the
leakage model; consequently, he can infer the key $k^{*}$.

The previous paragraph just describes a traditional non-profiled side-channel
attack over a crypto primitive. However, to understand how a profiled side-
channel attack works we have to explain its differences from a non-profiled
attack. A profiled type of attack born with the idea of training classifiers
to distinguish the outputs of a leakage model; so that the attack splits into
two phases; (i) to train a classifier (profiling phase) and (ii) to perform
the attack (attack phase).

The first phase comprises applying the corresponding leakage model to a clone
device (a.k.a. profile device), collecting from it the leakage traces forming
a set of profiling traces ($\mathcal{X}$) used to train the classifier. During
the attack phase, a set of attacks traces from the actual device is collected
and used for the trained classifier to compute probabilities, then a key
recovery process takes place using an algorithm called guessing entropy that
we will explain in brief.

Template attack and machine learning are two techniques to build a classifier
to evaluate side-channel attacks [7, 12]. It is well-known that to come up
with a classifier for SCA evaluation is not straightforward. Indeed, to reduce
the body work of building classifier anytime an SCA evaluation is required is
the motivation for the proposal of this paper. We propose a deep learning-
based model whose architecture allows the model to exchange classifiers with
other deep learning model aimed to conduct SCA evaluation over a different
device; and in the following, we address the necessary aspects that support
the theory about this approach.

###  2.2 Guessing entropy (GE)

GE is the average rank of the correct key byte value $k^{*}$ in a key guessing
vector $\mathbf{g}$, over all the set $\mathcal{K}$ of key candidates $k$
[26]. Formally denoted as $\textrm{GE}=rank_{k^{*}}(\mathbf{g})$, where
$rank_{k}(\mathbf{g})\in\\{0,\dots,|\mathcal{K}|-1\\}$, and the key guessing
vector is defined as:
$\mathbf{g}=sort(\mathbf{E}[log\mathbf{P_{r}}(\operatorname{\mathbf{t}}_{i};\operatorname{\mathcal{C}_{\delta}^{\theta}})])$.
$\mathbf{P_{r}}(\operatorname{\mathbf{t}}_{i};\operatorname{\mathcal{C}_{\delta}^{\theta}})$
is the input vector of probabilities $\mathbf{p}_{i,j}$ from a classifier
(usually aimed for key recovery task) given a leakage trace
$\operatorname{\mathbf{t}}_{i}$. After applying the expectation $\mathbf{E}$
per multiples experiments of $\mathbf{P_{r}}$, the sort function orders the
resultant vector $\mathbf{g}$ in decreasing order. The element
$g_{0}\in\mathbf{g}$ corresponds to the most likely key candidate, while
$g_{|\mathcal{K}|-1}\in\mathbf{g}$ is the less likely one.

###  2.3 Deep learning base profiled attacks

A deep learning classifier outputs a vector of probabilities fed into the
guessing entropy (GE) metric to compute the rank of the key ($k^{*}$). We
denote a deep learning model $\operatorname{\mathcal{C}_{\delta}^{\theta}}$
for profiled attacks as a classifier $\mathcal{C}$ with a vector of parameters
$\theta\in\mathbb{R}^{n}$ aimed to distinguish leakage traces labeled using a
leakage model $\delta$. Having labeled leakage traces means that our learning
approach is supervised learning [10] which represents one of the most feasible
ways to leverage the learning of a deep learning classifier.

Despite several deep learning architectures, CNNs based models are the
preferred architecture to use in profiled attacks. The convolutional part
plays an essential role when leakage traces are desynchronized. The deep
learning model we propose uses a specific type of convolution, called dilated
convolution [18] for boosting the feature extraction capability of the layer
(see sub-section 2.5).

###  2.4 Feature extraction

A feature extraction process applies a transformation (linear or non-linear)
to a space of observations resulting in a new space mapped by the
transformation. Formally, given profiling set $\mathcal{X}$ of $N$ leakage
traces and each trace comprises $m$ features (or sample points). Feature
extraction applies a function $\digamma$ to the profiling set $\mathcal{X}$
mapping a new profiling set $\mathcal{Y}$ whose elements have fewer dimensions
of the corresponding elements in $\mathcal{X}$; precisely, $\digamma$ is an
application such as $\digamma\colon\mathcal{X}\longmapsto\mathcal{Y}$, and
$\mathcal{X}\in\mathbb{R}^{m},\>\mathcal{Y}\in\mathbb{R}^{n}$ such that $n<m$.

This transformation aims to derive new features ($\mathcal{Y}$) to leverage
the performance of a classifier, for instance. Theoretically, features in
$\mathcal{Y}$ contain the “transformed” information that best represents the
ground truth of $\mathcal{X}$, in the SCA case, it is the leakage of the
sensitive information. In simple words, the intensity of the valuable
information gets emphasized while the irrelevant information (non-correlated
information) has little to no influence in the new space.

However, it is not straightforward to come up with a transformation $\digamma$
that indeed emphasized the side-channel information. A transformation that
goes wrong discards a lot of useful information, and it happens when
$\digamma$ cannot keep the variance that distinguishes a leakage trace from
another; as consequence, $\mathcal{Y}$ is made of several collapsed traces
becoming useless for classification purposes. In section 5, we will discuss
how our proposed method implements regularization to avoid transformations
that collapse the $\mathcal{Y}$ space.

Function $\digamma$ can be inferred directly from $\mathcal{X}$. For instance,
Principal Components Analysis (PCA) [9] or Linear Discriminant Analysis (LDA)
[2] are two algorithms to build linear base $\digamma$ functions for feature
extraction. However, PCA and LDA are highly sensitive to desynchronization
because of their “per feature” process, meaning they find a relation by
correlating the same positioning feature through samples. So that, when the
samples have a spatial disruption, the relation gets reduced, requiring more
samples.

###  2.5 Autoencoders

An autoencoder is a learning algorithm useful to infer $\digamma$; contrary to
PCA and LDA, an autoencoder can infer a non-linear transformation due to the
non-linear activation functions in its architecture. Moreover, when the
autoencoder architecture comprises convolution layers, it handles the spatial
disruption better than PCA and LDA.

An autoencoder consists of two parts; (i) an encoder $\varphi$ and (ii) a
decoder $\psi$. Let us define a leakage trace
$\operatorname{\mathbf{t}}_{i}\in\mathcal{X}$, and its dimension being denoted
as $dim(\operatorname{\mathbf{t}}_{i})=m$. The encoder outputs a new trace
$\operatorname{\operatorname{\mathbf{t}}^{\prime}}_{i}$ with
$dim(\operatorname{\operatorname{\mathbf{t}}^{\prime}}_{i})<dim(\operatorname{\mathbf{t}}_{i})$
(see expression (2)). At the other side of the autoencoder, the decoder tries
to reconstruct $\operatorname{\mathbf{t}}_{i}$ but it is able to re-build an
approximation $\operatorname{\tilde{\operatorname{\mathbf{t}}}}_{i}$ only;
consequently, one can understand that an autoencoder learns by minimizing the
difference between $\operatorname{\mathbf{t}}_{i}$ and
$\operatorname{\tilde{\operatorname{\mathbf{t}}}}_{i}$ (as we will see in
expression 6).

$\operatorname{\operatorname{\mathbf{t}}^{\prime}}_{i}=\varphi(\operatorname{\mathbf{t}}_{i}),\quad\operatorname{\tilde{\operatorname{\mathbf{t}}}}_{i}=\psi(\operatorname{\operatorname{\mathbf{t}}^{\prime}}_{i})$
(2)

From a functional perspective, the encoder maps $\mathcal{X}$ to an embedding
space denoted by $\mathbf{Z}$ (i.e.
$\varphi\colon\mathcal{X}\mapsto\mathbf{Z}$), the embedding $\mathbf{Z}$ is
usually called latent space, code, latent code or hidden code. Further,
$\mathbf{Z}$ is the space result of the transformation applied by the encoder.
According to the discussion in the previous sub-section, $\mathbf{Z}$ is the
resultant space of a feature extraction process i.e. $\mathcal{Y}$. Likewise,
the decoder maps $\mathbf{Z}$ to $\tilde{\mathcal{X}}$ (i.e.
$\psi\colon\mathbf{Z}\mapsto\tilde{\mathcal{X}}$), where
$\operatorname{\tilde{\operatorname{\mathbf{t}}}}\in\tilde{\mathcal{X}}$. The
expressions (3) and (4) formalize these two mappings;

$\displaystyle\mathbf{Z}=\varphi(\mathcal{X})$ $\displaystyle=$
$\displaystyle\sigma(W_{enc}\,\mathcal{X}+b)$ (3)
$\displaystyle\tilde{\mathcal{X}}=\psi(\mathbf{Z})$ $\displaystyle=$
$\displaystyle\sigma(W_{dec}\,\mathbf{Z}+b^{\prime})$ (4)

Function $\sigma$ denoted a non-linear activation function. An encoder is
parameterized by a weight matrix $W_{enc}\in\mathbb{R}^{m\times n}$ and a bias
vector $b\in\mathbb{R}^{n}$; likewise, a decoder is parameterized by a weight
matrix $W_{dec}\in\mathbb{R}^{n\times m}$ and a bias vector
$b^{\prime}\in\mathbb{R}^{m}$ (see Fig. 1). Training an autoencoder implies
finding a vector of parameters $\theta=(W_{enc},W_{dec},b,b^{\prime})$ that
minimize a loss function $\mathcal{L}$ such as;

$\Theta=\min_{\theta}\mathcal{L}(\operatorname{\mathbf{t}},\operatorname{\tilde{\operatorname{\mathbf{t}}}})=\min_{\theta}\mathcal{L}(\operatorname{\mathbf{t}},\psi(\varphi(\operatorname{\mathbf{t}})))$
(5)

As we said, autoencoder learns by minimizing the difference between
$\operatorname{\mathbf{t}}_{i}$ and
$\operatorname{\tilde{\operatorname{\mathbf{t}}}}_{i}$; so that, the Mean
Square Error (MSE) is a loss function commonly used;

$\mathcal{L}_{\textrm{MSE}}=\mathcal{L}(\operatorname{\mathbf{t}},\operatorname{\tilde{\operatorname{\mathbf{t}}}})=\frac{1}{m}\sum^{m}_{i=1}{(\operatorname{\mathbf{t}}[i]-\operatorname{\tilde{\operatorname{\mathbf{t}}}}[i])^{2}}$
(6)

Figure 1: Typically autoencoders are symmetrical models, meaning that both
parts of the encoder and decoder resemble each other. During training, the
encoder trains to code the original signal to a Latent space, ideally this
code from the features that better represent the characteristic of the
original signal. From there, the decoder re-constructs as much as possible the
original signal.

##### Convolution layer architecture

Autoencoders are built using either fully connected layers or convolutional
layers. The latter makes the autoencoder inherit the spatial invariant
robustness property, which is useful when leakage traces are desynchronized;
our autoencoder uses dilated convolution layers.

A convolution layer consists of kernels that essentially are matrices; then,
to dilate a convolution layer consists of inserting zeros into its kernels,
meaning to separate the matrices’ elements using zeros, expanding their
receptive field333Called it to those kernel elements which are not zero.
According to [18] a dilated kernel allows convolutions base classifiers to
combine spread features that contain the leakage information, at the same
time, avoiding irrelevant features that might lay in between.

Let us consider the expression in (2.5) showing a regular convolution where a
leakage trace $\operatorname{\mathbf{t}}_{i}$ is multiplied by the kernel
$\mathbf{q}$ whose length is denoted by $l_{\mathbf{q}}$. If we displace the
leakage trace $\operatorname{\mathbf{t}}_{i}$ from right to left, a single
feature $t$ of $\operatorname{\mathbf{t}}_{i}$ is multiplied $l_{\mathbf{q}}$
times. If $l_{\mathbf{q}}$ is large, then $t$ might be excessively used during
the operation. According to [29], this excessive use of $t$ may decrease the
convolution effectiveness. Notice that if $l_{\mathbf{q}}$ increases aiming to
use further spread features, it also increases the times $t$ is used. By using
dilated convolutions, one can avoid this downside.

$\displaystyle(\operatorname{\mathbf{t}}\circledast\>\mathbf{q})[t]$
$\displaystyle{}={}$
$\displaystyle\sum^{\infty}_{n=-\infty}{\operatorname{\mathbf{t}}[t-n]\cdot\mathbf{q}[t]}$
$\displaystyle{}={}$ $\displaystyle\cdots+$
$\displaystyle\underbrace{(\operatorname{\mathbf{t}}[t-n_{i}]\cdot\mathbf{q}[n_{i}])+(\operatorname{\mathbf{t}}[t-n_{i+1}]\cdot\mathbf{q}[n_{i+1}])}_{\text{$l_{\mathbf{q}}\>$times}}$
$\displaystyle+\cdots$

The expression in (2.5) shows a dilated kernel with one zero inserted between
its elements. Notice that when the convolution is performed, the feature $t$
alternates being multiplied or not by a zero; consequently, it reduces the
times the operation uses the feature $t$.

$\displaystyle(\operatorname{\mathbf{t}}\circledast\>\mathbf{q}_{d})[t]$
$\displaystyle{}={}$
$\displaystyle\sum^{\infty}_{n=-\infty}{\operatorname{\mathbf{t}}[t-n]\cdot\mathbf{q}_{d}[t]}$
$\displaystyle{}={}$ $\displaystyle\cdots+$
$\displaystyle\left.\begin{array}[]{l}(\operatorname{\mathbf{t}}[t-n_{i}]\cdot
0)+\\\ (\operatorname{\mathbf{t}}[t-n_{i+1}]\cdot\mathbf{q}_{d}[n_{i+1}])+\\\
(\operatorname{\mathbf{t}}[t-n_{i+2}]\cdot 0)+\\\
(\operatorname{\mathbf{t}}[t-n_{i+3}]\cdot\mathbf{q}_{d}[n_{i+3}])\end{array}\,\right\\}\hat{l_{\mathbf{q}}}\>\text{times}$
$\displaystyle+\cdots$

The hyperparameter dilatation rate ($dr$) controls the number of zeros
inserted. When a kernel is dilated its receptive field is modified by the
relation;

$\hat{l_{\mathbf{q}}}={l_{\mathbf{q}}+(l_{\mathbf{q}}-1)(dr-1)}$ (13)

In this way, the receptive field increases by modifying either the length of
the kernel or the dilatation rate, letting the user regularize the convolution
operation.

##  3 Related work

While few works in SCA discuss an approach of architecture transferability
with reusable modules, several works have discussed feature reduction for SCA.
Cagli et al. in [6, 5, 4] discussed application of traditional feature
reduction methods using PCA [9], LDA [2], and its kernel base variant Kernel
PCA and KDA. Picek et al. [21] published results using same methods as [5].
However, authors in [21] used an approach that combined feature extraction and
feature selection; precisely, PCA and LDA combined with SOST and SOSD, they
called it hybrid feature selection methods.

Intrinsically, any work that uses the same feature reduction techniques aims
to downsample the signal by taking it to a new space (latent space). However,
these approaches consider only linear base feature reduction disregarding the
more powerful non-linear version of it; it is likely, that this situation may
be a consequence of advertising CNNs as built-in feature extraction deep
learning models. Hence, very few works have addressed non-linear methods for
SCA evaluation. One of those few works are, for instance, Paguada et al. [19],
and Yang et al. [28]; similar to us, those works used autoencoders toward
inferring a non-linear function to pre-process leakage traces in a fashion
that overcome linear methods.

While those two works are the closest one we can relate with, to our best
knowledge, there is no previous work on side-channel analysis that suggest a
deep learning approach based on modules; featuring to share modules between
models.

##  4 ASCAD fixed and random datasets

ASCAD dataset444This dataset is publicly available at
https://github.com/ANSSI-FR/ASCAD was introduced in [23]. The leakage traces
were collected from an Atmega8515 8-bit microcontroller. The cryptographic
algorithm implemented is AES-128 protected using masking countermeasure [8,
3].

The dataset has two versions, traces collected with fixed key encryption
$k_{f}$ and traces collected with random key encryption $k_{r}$ (plaintext is
always random), while the target byte of the secret key in both cases is the
third one. We named these versions as $\text{ASCAD}^{f}$ and
$\text{ASCAD}^{r}$ respectively. Due to these key characteristics,
$\text{ASCAD}^{r}$ is more challenging and more realistic than
$\text{ASCAD}^{f}$ when conducting an SCA evaluation over them. TABLE 1
contains a summary of main characteristics of these two datasets.

$\text{ASCAD}^{f}$ | $\text{ASCAD}^{r}$
---|---
Profiling_traces | $50\,000$ | Profiling_traces | $200\,000$
Attack_traces | $10\,000$ | Attack_traces | $100\,000$
dim($\operatorname{\mathbf{t}}_{i}$) | $700$ | dim($\operatorname{\mathbf{t}}_{i}$) | $1\,400$
Table 1: Cardinalities of the ASCAD datasets. Since their goal is to be used
for benchmarking profiled attacks the leakage traces are grouped in
profiling_traces and the attack_traces sets.

Leakage traces in each version are desynchronized according to a threshold
value that moves traces around the $x$-axis, being frequently used threshold
values of 0, 50, and 100. Then, to make clear distinctions when exchanging the
modules between modular networks, we add to the name the threshold value, for
instance, $\text{ASCAD}^{r}\>desync50$.

##  5 DL-SCA modular network architecture

This section explains the details about the architecture of the DL-SCA modular
network; further, we describe the strategy to train it.

Since we are using autoencoders; then, our suggested DL-SCA modular network
comprises three main modules; an encoder, a decoder, and a classifier (see
Fig. 2). Particularly, we will group the encoder and decoder into a single
module called a downsampler. The downsampler has two goals; (i) to extract
meaningful features by reducing the noise in the leakage traces and (ii) to
downsample them. Now, the classifier is in charge of evaluating those
extracted features as a classification problem.

It is worth mentioning that once the DL-SCA modular network is trained, we
discard the decoder of the downsampler, and we only use the encoder and
classifier to perform the SCA evaluation. Due to this, we elaborate a training
strategy to monitor only those two parts of the model; we will elaborate this
late in this section.

The goal of both modules might be apparent; however, the downsampler has an
implicit objective. To achieve compatibility with as many classifiers as
possible, we should use a downsampler to fix the classifier input. Precisely,
we downsample the leakage traces to a fixed length; then, when we re-use the
classifier with another downsampler, this latter fixes its output to match the
classifier’s input. By doing this, we fulfill the first step of re-usability.
We demonstrate this in the experimental section of this paper.

Training a DL-SCA modular network architecture requires a loss function for
the decoder and another for the classifier. The decoder’s loss function
($\mathcal{L}_{MSE}$) was discussed in the sub-section 2.5. We introduce the
classifier loss function.

Figure 2: DL-SCA modular network architecture illustration. The encoder, with
its embedding layer and the classifier ensemble the final model used to
perform the attack.

###  5.1 Classifier loss function

As we said, a classifier outputs a vector of probabilities used as input for
the guessing entropy. So, for the classifier to output this vector, it must be
trained using a cross-entropy ($CE$) loss function. In supervised profiled
side-channel attacks the leakage traces are labeled by the output of a leakage
model (see expression (1)). Further, the classifier learns by minimizing its
error in predicting the label of each trace.

To explain this better, let us consider the expression (14). The space
$\hat{\mathcal{K}}$ corresponds to a batch of key candidates or labels, each
one of the labels in $\hat{\mathcal{K}}$ represents a trace. For instance, let
us take $\delta_{i}\in\hat{\mathcal{K}}$ as one of those labels, we say that
$\delta_{i}$ is the ground truth while $\sigma(\delta_{i})$ is the output
score a neural network computed555Often $\sigma$ is the softmax activation
function for multi-class classification.

$\mathcal{L}_{CE}=-\sum_{i}^{\hat{\mathcal{K}}}\delta_{i}\cdot\log(\sigma(\delta_{i}))$
(14)

During training, this loss function computes the error in the prediction made
by the classifier; consequently, the weights of the classifier are updated
toward achieving a prediction with highest accuracy possible.

Clearly, we use a classifier with the same purpose as in a common profiled
side-channel evaluation. However, the feature in using a classifier in our
approach is to add a regularization term to the downsampler. Precisely, the
supervised classifier adds an extra penalization to the downsampler with
regard to the featuring space $\mathcal{Y}$ the downsampler is building up;
leading the whole network toward better performance. It is impossible to
feature this with a self-supervised downsampler trained separately as authors
did in [19].

The arrangement depicted in Fig. 2 suggests that both the classifier and
decoder attach to the encoder. Consequently, when training the modular
network, the classifier feed-forwards the downsampled traces from the
embedding and back-forwards its loss. Meanwhile, the decoder trains its
reconstruction capability that additionally penalizes the encoder. These two
losses resemble a double voting system that the encoder uses to leverage
learning.

Now, notice that because the activation functions are non-linear, the
classifier acts as a non-linear regularizer for the embedding space.
Consequently, the decoder takes the regularization effect as small
perturbations in that space; those perturbations challenge the decoder in
reconstructing the original traces as it understands that those are small
errors in its reconstruction. Contrary to the approach in [19], training
jointly the autoencoder and the classifier produces an embedding likely to
learn positive features. Due to the regularization factor, less correlated
features are emphasized over highly correlated noisy features.

###  5.2 Analogy with linear regularized autoencoder

Autoencoders aim to be imperfect models; so, when training an autoencoder we
must avoid an architecture that ends with a model called “identity function”.
When this phenomenon happens, the autoencoder will just copy the data from the
input to the output. One way to avoid this is by using a undercomplete
architecture, which refers to the embedding we discussed early; further, the
deepest an autoencoder is the stronger becomes to avoid ending as an identity
function. However, doing this carelessly might reduce the network performance
as the model becomes supra-complex, so that, we cannot rely on it repeatedly.

Applying a regularizer to the latent space is another alternative. Regularized
autoencoder proved overcoming normal autoencoders when leveraging meaningful
features in the embedding. A linear regularizer applies to the latent space an
extra penalization. The embedding neurons fire the additional penalization to
the decoder added to its loss function as small epsilons of error. This latter
advocates the disruption by training its neurons to reconstruct the original
data; ignoring that it is being fooled by the regularizer, so its learning is
actually “imperfect” [10].

A drawback of using linear regularizer is precisely its nature. A linear
regularizer applies the penalization linearly to all the embedding neurons,
there is no a criterion that controls the magnitude each neuron should receive
based on its contribution to the loss function; as eventually, the decoder
starts copying the input as it is.

In a DL-SCA modular network, the classifier acts as a regularizer;
nonetheless, the regularization is based on non-linearity since the classifier
is a non-linear function. The non-linear activation functions used in the
classifier receive their input from the embedding neurons; once the classifier
does the back-propagation it applies an epsilon value according to their
contribution to the classification. Once again the decoder interprets those as
small errors, but now facing a more advance regularization.

Both linear and non-linear regularizers require a value to control the
intensity of the penalization. For our non-linear regularizer, this value is a
parameter $\gamma\,\in\>]0,1]\subset\mathbb{R}$ multiplied by the loss
function’s result.

###  5.3 DL-SCA modular network loss function

Now that we know the two losses required by our architecture as well as the
hyperparameter to control their intensity, we have the expression (15) that
defines the loss function for a DL-SCA modular network architecture.

$\mathcal{L}_{\text{DL-SCA modular
network}}=\gamma\cdot\mathcal{L}_{CE}+\omega\cdot\mathcal{L}_{\textrm{MSE}}$
(15)

Notice that there is an $\omega$ parameter for $\mathcal{L}_{\textrm{MSE}}$
that works exactly as $\gamma$. We fix $\omega=1$, because our goal is to
control the regularization and not the reconstruction.

###  5.4 Training strategy for a DL-SCA modular network

Recently, authors from [20] published an early stopping framework to monitor
the state of a deep learning model during its training preventing it from
getting overfit/underfit. Overfitting/underfitting is a phenomenon that might
happen during training; it represents the state when a deep learning network
cannot generalize beyond its training set.

The framework computes the guessing entropy at the end of each epoch basing
the stopping criterion on the whole guessing entropy vector, considering when
the guessing entropy converges, and how many traces keep the guessing entropy
in the state of convergence, proving to overcome existing frameworks (more
details can be found in the original paper [20]). We use this early stopping
to elaborate a training strategy for our DL-SCA modular network.

##### Training strategy

We know that an early stopping framework stops the training of a deep learning
model when it meets conditions established using a metric, e.g., the accuracy
of the model. Typically, these frameworks evaluate the entire model. In
contrast, we need the framework to consider just the encoder and the
classifier as they are the parts used in the SCA evaluation. The framework
from [20] is a “typical” framework, so it monitors the whole deep learning
model.

We modified the suggested framework to receive a truncated model which
comprises just the modules of interest (encoder and classifier). To apply this
modification is effortless when using the weight sharing technique [30]. In
this technique two or more neural networks share the references to some
specific layers, all of those networks can update the weights of those layers;
however, in our particular case the original networks updates the weights,
while the truncate model monitors the state (of those weights) of the encoder
and classifier.

Precisely, we set a truncated model (see Fig. 3) to reference those modules
and to be only evaluated (not trained) by the early stopping framework. Then,
it stops training when the encoder generates features that makes the
classifier to achieve an expected performance, in this case an expected
guessing entropy convergence. The modification works since the framework uses
the truncated model as the predictor, and its output serves as the input to
compute the guessing entropy.

In the first experimental results section of this paper, we show the training
strategy outcome using surface plots. Notice that we did not stop the network
training, so the surface plots correspond to the entire training process, our
goal is to show that an SCA-DL modular network does not rely on an early
stopping framework.

Figure 3: The truncated model shares weights with the DL-SCA modular network;
while the latter is training, the former updates his weights. The early
stopping framework uses the truncated model to compute the guessing entropy at
the end of each epoch, and it stops the training when it meets the conditions.

##  6 Experimental results of training modules

In this section, we discuss the results of using our proposed approach over
ASCAD datasets — $\text{ASCAD}^{f}\>all\>desync$ and
$\text{ASCAD}^{r}\>all\>desync$.

We organize the experimental results as two different use cases where a DL-SCA
modular network analyzes; (i) ASCAD fixed key dataset and (ii) ASCAD random
key dataset. We accomplish two goals with these uses cases; (i) to show the
feasibility when an SCA evaluation uses our architecture to attack an specific
dataset, and (ii) to create a scenario where we demonstrate the feasibility of
sharing modules. The strategy is applicable to real evaluations; the derived
modular network evaluates a first dataset; consequently, a second modular
network could evaluate another dataset borrowing a module from a previous
modular network.

In our particular case, our experiments use two datasets that share the same
source of data; precisely, both datasets were composed with leakage traces
from the same microcontroller (Atmega8515 8-bit). We aim for performing
experiments when the source of data is uncommon between both datasets as
future works.

Notice, we used the same model for all levels of desynchronization, meaning
that additional effort in finding neural network architectures for specific
noisy scenarios is not required.

###  6.1 $\text{ASCAD}^{f}$ all desync use case

The TABLE 2 summarizes the hyperparameters of the modular network architecture
to evaluate $\text{ASCAD}^{f}\>all\>desync$.

####  6.1.1 Network’s architecture

We set the architecture by following the discussion in Sect. 2.5; the first
convolutional block uses dilated convolutions to avoid any useless features
that might reduce the model’s performance. We dilate the convolutions at the
first convolutional block because it is where we deal with the original
version of the trace. Further, we add convolutional blocks to the encoder
following the rules applied for VGG [25] base deep learning
architectures666VGG base architectures increase their number of kernels in
convolutional layers by the power of 2.

ENCODER
---
Layer type | Details
1\. Conv | # kernels: 32, kernel size: 64, SeLU
| dr: 3, kernel init: He_Uniform
2\. Batch Norm |
3\. Pooling | Average, # kernels: 2, stride: 2
4\. Conv | # kernels: 64, kernel size: 25, SeLU
| kernel init: He_Uniform
5\. Batch Norm |
6\. Pooling | Average, # kernels: 25, stride: 25
7\. Conv | # kernels: 128, kernel size: 3, SeLU
| kernel init: He_Uniform
8\. Batch Norm |
9\. Pooling | Average, # kernels: 5, stride: 5
10\. Flatten |
11\. Dense | 300 Units (Latent space)
DECODER
Layer type | Details
1\. ConvTranspose | # kernels: 128, kernel size: 3, SeLU
| stride: 7, kernel init: He_Uniform
2\. Batch Norm |
3\. ConvTranspose | # kernels: 64, kernel size: 25, SeLU
| stride: 25, kernel init: He_Uniform
4\. Batch Norm |
5\. ConvTranspose | # kernels: 32, kernel size: 64, SeLU
| stride: 2, kernel init: He_Uniform
6\. Batch Norm |
7\. ConvTranspose | # kernels: 1, kernel size: 1, Sigmoid
| stride: 1, kernel init: He_Uniform
CLASSIFIER
Layer type | Details
1\. Conv | # kernels: 4, kernel size: 1, SeLU
| dr: 3, kernel init: He_Uniform
2\. Batch Norm |
3\. Pooling | Average, # kernels: 2, stride: 2
4\. Flatten |
5\. Dense | 10 (units), SeLU
6\. Dense | 10 (units), SeLU
7\. Dense | 10 (units), SeLU
8\. Dense | 256 (units), Softmax
Table 2: DL-SCA modular network architecture to use in experiments with
$\text{ASCAD}^{f}$ all desynchronizations levels

The decoder mirrors the encoder, as our downsampler uses symmetric
autoencoders. For the decoder to up-sample, namely to reconstruct the actual
length of the trace, it uses transpose convolutions. As known, matrix
multiplication is not commutative, and we cannot achieve the same output in
respective convolutional blocks. Consequently, we have to tune the
hyperparameters in the decoder’s convolution layers. For instance, let us take
the third encoder’s convolutional block that uses a stride value of $5$, its
corresponding decoder’s transpose convolutional block is the first one but it
uses stride value of $7$. By doing this, we fix the output of the decoder to
meet the original trace dimension.

Figure 4: The training process of the modular network for $\text{ASCAD}^{f}$
datasets. The surface represents the values of the guessing entropy during a
chosen number of epochs. Stopping condition success GE=0.

####  6.1.2 Latent space hyperparameters

With regard to latent space units and $\gamma$ value. We perform a grid search
for the best number of units in the latent space, using the values 100, 200,
300, 400, and 560. Further, we know that the parameter $\gamma$ relates
strictly with the number of latent units; consequently, to find the value of
$\gamma$ we create combinations using the latent space values and values of
$\gamma$ as $\\{1e^{-3},1e^{-6},1e^{-9}\\}$. It turned out that the best
combinations was 300, and $1e^{-3}$ for latent space units and the $\gamma$
parameter, respectively.

Regarding the classifier module, bear in mind that we are only interested in
its classification performance and not too much in its ability to filter out
unnecessary features of the leakage traces, so we use a shallow architecture
since it will deal with already filtered features.

####  6.1.3 Training strategy and results

As we said, to train a modular network, we use the early stopping framework
from [20]. To show that our suggested architecture does not rely on the
framework, we did not stop the training after the mentioned framework finds
the best learning state. Further, we will use this outcome in the next section
to discuss the result of the reusing modules experiment. Fig. 4 depicts the
training process when our modular network evaluates $\text{ASCAD}^{f}$
datasets. As we expected, the training outcome differs according to the level
of desynchronization; regardless, our modular network achieved a zero
convergent guessing entropy for all desynchronization levels. A view of the
attack performance is depicted in Fig. 5.

Figure 5: Guessing entropy over $\text{ASCAD}^{f}$ all levels of
desynchronization

###  6.2 $\text{ASCAD}^{r}$ all desync use case

Figure 6: Guessing entropy over $\text{ASCAD}^{r}$ all levels of
desynchronization Figure 7: The training process of the modular network for
$\text{ASCAD}^{r}$ datasets. The surface represents the values of the guessing
entropy during a chosen number of epochs.

####  6.2.1 Network’s architecture

Regarding this dataset, our strategy was to keep the same modular network as
the previous use case to reuse as much as possible an already worked model and
see how it performs. After experimenting, we noticed that the downsampler
module required an additional convolutional block —identical to the third
convolutional block of the decoder— without pooling layer. Consequently, the
decoder should also have the corresponding transpose convolutional block.

####  6.2.2 Latent space hyperparameters, training strategy, and results

We keep the same classifier as in the previous use case because we have the
same number of latent units. In our particular case, to keep the same latent
units is convenient because we aim for exchanging a trained classifier in the
following experiments to evaluate the modular re-usability. Fig. 7 depicts the
training process of guessing entropy by epochs for $\text{ASCAD}^{r}$ dataset.
In this case, we observe that the performance of our modular network slightly
decreases, which is expected since the dataset has a higher level of noise
than the previous. Even though, we achieve good guessing entropy convergence
as depicts Fig. 6. Finally, we compare our experimental results with
previously reported results over the same datasets. TABLE 3 gathers this
information.

| $\text{ASCAD}^{f}$ | $\text{ASCAD}^{r}$
---|---|---
| desync level | desync level
Reference | 0 | 50 | 100 | 0 | 50 | 100
From [19] | 791 | - | - | - | - | 1500
From [18] | 178 | - | - | 550 | - | 3 000
From [29] | 191 | 244 | 270 | - | - | -
This work | 148 | 231 | 236 | 210 | 407 | 1092
Table 3: Best guessing entropy convergence comparison

##  7 Module re-usability experimental results

This section presents the results of module re-usability. We show that another
non-trained DL-SCA modular network can reuse the modules of a DL-SCA modular
network. We use the DL-SCA-based module networks trained in the previous
section to prove it.

###  7.1 Analyzing transferability

We aim to show how “transferable” is the knowledge of a classifier module. We
have six modular network —meaning six classifiers— trained with three
different datasets —three on $\text{ASCAD}^{f}$ and three on
$\text{ASCAD}^{r}$. Further, due to the number of latent units (300) we used,
all classifiers are interchangeable without performing additional downsampling
operations to fix their inputs. For our experiments, we took the classifier
from the DL-SCA modular network of $\text{ASCAD}^{f}\>desync50$ to share with
all the downsampler from $\text{ASCAD}^{r}$. We considered it sufficient for
proving our claim about “module re-usability”. We chose
$\text{ASCAD}^{f}\longmapsto\text{ASCAD}^{r}$ direction because it represents
the complex direction —from fixed key to random key.

| Original |  Gradient
---|---|---
Shared classifier | downsampler & classifier |  operation
| $\text{ASCAD}^{r}\>desync0$ |  heatmap
|  gradient vis
| $\text{ASCAD}^{r}\>desync50$ |  heatmap
|  gradient vis
| $\text{ASCAD}^{r}\>desync100$ |  heatmap
$\text{ASCAD}^{f}\>desync50$ |  gradient vis
Table 4: Summary of the similarity analysis between
$\text{ASCAD}^{f}\>desync50$ classifier and $\text{ASCAD}^{r}\>all\>desync$
classifiers

We inspect the transferability of the $\text{ASCAD}^{f}\>desync50$ classifier
by conducting a similarity analysis using gradient activation operations. In
particular, we use heatmaps and gradient visualization to compare how the
neurons’ of the classifier are activated by the data outputted from the
downsamplers.

We perform this analysis by locking specific layers of the classifier to
identify how transferable those layers are. Precisely, we choose convolutional
block (Conv) layers and fully connected block (FC) layers and lock them by
turns to evaluate them separately. A heatmap allows us to inspect the
convolutional layers of the classifier, while gradient visualization helps us
analyze how both Conv and FC perform with the different datasets.

TABLE 4 summarizes the similarity analysis we are going to perform using the
classifier $\text{ASCAD}^{f}\>desync50$, the $\text{ASCAD}^{r}$ datasets, and
the gradient activation operations. Fig. 8 depicts the first convolutional
layer heatmaps from $\text{ASCAD}^{f}\>desync50$ classifier and
$\text{ASCAD}^{r}\>all\>desync$ classifiers ($desync0$, $desync50$, and
$desync100$).

For these particular experiments, all $\text{ASCAD}^{r}$ classifiers share
similarities with the $\text{ASCAD}^{f}\>desync50$ classifier in how their
convolutional layer neurons’ get stimulated. According to our assumptions, it
indicates that the weights of those layers might be transferable. This claim
is experimentally demonstrated later in the final experiments.

Figure 8: Comparison between heatmaps of the $\text{ASCAD}^{f}\>desync50$
classifier and classifiers from all the $\text{ASCAD}^{r}$ datasets. Notice
how $\text{ASCAD}^{f}\>desync50$ heatmap resembles all other heatmaps. It
indicates that $\text{ASCAD}^{f}\>desync50$ classifier’s convolutional layer
fires its neurons according the data received.

Although the magnitude of the $\text{ASCAD}^{f}\>desync50$ classifier’s
heatmap is higher than any other heatmap from $\text{ASCAD}^{r}$ classifiers,
it does not represent a drawback to the transferability. We could have gotten
the same magnitudes if we had normalized the weights applying constraints in
the architecture, though a similarity analysis does not need to do this.

We use gradient visualization to inspect the classifiers’ fully connected
block (FC). The output of that operation indicates which input features are
the most meaningful for the classification. The gradient visualization uses
the loss function of a trained classifier to conduct backpropagation,
collecting the information about those neurons that emphasize the performance.
Further, when it reaches the input layer, it points out which features are
connected to those neurons, indicating the meaningful features [14, 24, 1].
Fig. 9 depicts the result of gradient visualization operation.

Figure 9: Comparison between gradient activation per sample of the
$\text{ASCAD}^{f}\>desync50$ classifier and classifiers from all the
$\text{ASCAD}^{r}\>all\>desync$.

Notice that gradient visualization shows less intuition than heatmap. As a
workaround, we apply a Dynamic Time Warping (DTW) [17] to visualize the
similarities between gradient visualization signals.

According to this experiment, two phenomena happen; (i) the meaningful
features are displaced according to each classifier, or/and (ii) the
meaningful features are less intense in magnitude. These phenomena could
represent an issue. For instance, let us take the $\text{ASCAD}^{r}\>desync0$
classifier, notice the displacement because the $\text{ASCAD}^{f}\>desync50$
interprets that the meaningful features localize differently. Further, those
features have an even lower magnitude in contrast to those supposedly being
the lowest(see points from 0 to 30 in Fig.9 top plot).

This analysis gives us the intuition that we will need to retrain the
classifier; nevertheless, the reader might remember that the classifier is
just a part of a bigger model. The downsampler will leverage its learning
according to the limitation imposed by the classifier.

###  7.2 Playing with blocks

Let us suppose we have trained a DL-SCA modular network using a former
dataset; then, we have the opportunity to evaluate another dataset. We could
use the classifier module of the first network to evaluate it. In this
hypothetical scenario, the first dataset is played by $\text{ASCAD}^{f}$ and
the second one by the $\text{ASCAD}^{r}$ dataset.

To experimentally evaluate if we need to re-train some or all parts of the
classifier, we perform experiments locking the blocks of the classifier to
restrict them from getting trained. In the previous sub-section, we inspected
the blocks of the classifier (Conv and FC), and we observed some similarities
in its neurons’ weights. Now, we are going to evaluate the performance of the
whole modular network when its classifier module has the following locks:

  * •

Convolutional block

  * •

Fully-connected block

  * •

Both blocks

We will refer to these as “sharing protocols”. We find out which could be the
best sharing protocol for these particular modular networks by locking the
blocks. TABLE 5 summarizes the combination of locks and dataset where the
shared classifier will be used.

Classifier | Re-used in
---|---
Shared classifier |  Lock use case |  Dataset & filter
|  Convlock | $\text{ASCAD}^{r}desync0$
|  Bothlock
|  FClock
|  Convlock | $\text{ASCAD}^{r}desync50$
|  Bothlock
|  FClock
|  Convlock | $\text{ASCAD}^{r}desync100$
|  Bothlock
$\text{ASCAD}^{f}\>desync50$ |  FClock
Table 5: Combination of sharing protocols used for the
$\text{ASCAD}^{f}\>desync50$ classifier

We previously said that the chosen classifier $\text{ASCAD}^{f}\>desync50$
will tackle a more complex dataset —the $\text{ASCAD}^{r}\>desync100$. Now, by
evaluating the $\text{ASCAD}^{r}\>desync0$ dataset; then, we will cover the
scenario where the shared classifier comes from a more complex dataset. Still,
bear in mind that it is in terms of desynchronization because it does not come
from a complex dataset in terms of its secret key’s nature —from random to
fixed key, for example. So, we rate the “experience” of the classifier as
medium level of experience.

Figure 10: The training results of the knowledge transferability experiments.
Through the columns lies the levels of desynctronization [0, 50, 100]; while
through the rows, lies the different block lock cases —ConvLock, BothLock, and
FCLock.

Due to space constraints, we did not perform an inter-classifier sharing and a
no-block lock sharing protocol; furthermore, we claim that the sharings
addressed in our experiments represent the difficult one, being enough to
prove our contribution. However, we let those experiments and further
combinations of sharing protocol for future works. Fig. 10 depicts the
training process of all chosen sharing protocols. It is worthy of mentioning
that we did not change the loss intensity parameter ($\gamma$), reducing the
effort in tuning the modular network.

For this experiments, we trained the modular networks using the early stopping
framework from [20]. Contrary we did in the previous section, we do stop the
training when the policy finds out the best learning state. We can now know
the number of epochs required to achieve good performance.

Figure 11: Best guessing entropy results from all sharing protocols.

Generally, all sharing protocols perform well if we contrast the training
process of Fig. 10 and Fig. 5. Nevertheless, the sharing protocols that worked
best are the fully-connected block, both blocks, and the convolutional block
lock.

Observe that for the fully-connected block lock, $\text{ASCAD}^{r}\>desync0$
has a convergent guessing entropy after 9 epochs, $\text{ASCAD}^{r}\>desync50$
at 65 epochs, and $\text{ASCAD}^{r}\>desync100$ took the whole training
process (100 epochs); even thought, it achieves good performance. Both blocks
lock cases seem to require more epoch or the convergence is roughly achieved,
$\text{ASCAD}^{r}\>desync0$ and $\text{ASCAD}^{r}\>desync50$, for instance.
Finally, we notice that the convolutional block lock converges after several
more epochs than the previous locks. In this case,
$\text{ASCAD}^{r}\>desync100$ did not converge within $1\,000$ leakage traces.
We summarize in Fig. 11 the best guessing entropy from all combination of
locks.

###  7.3 Discussion

Using a shared classifier instead of a non-trained modular network, we have
reduced the training time and the effort in tuning hyperparameters while
evaluating the leakage of a dataset with good results. Since we locked some
blocks and the whole classifier, we reduced the number of neurons to train;
consequently, the training time is reduced since the number of operations per
neuron is less than a non-trained modular network. As we do not have to tune
the hyperparameter of a classifier, then we do not spend time in it. Further,
we are confident that the classifier has a high probability of working since
it already has previous “experience”. We demonstrated the latter by actually
achieving good results.

Clearly, some initial effort has to be made. For instance, we were tuning the
latent space and losses intensity hyperparameters. Coming up with an initial
deep learning modular network could be challenging, but it is an equivalent
effort in finding several small deep learning models for different datasets.
Finally, bear in mind that by saying that a classifier has previous
experience, we do not claim that it will work flawlessly. As we said, the
experience of a shared classifier represents a neurons’ weights initializer.
So instead of randomly initializing the weights using any well-known function
—he uniform, for instance—; we start from a state leveraged by a previous
worked learning. We have proved, experimentally, that it has good results.

##  8 Conclusion

We introduced the DL-SCA modular network approach to conducting SCA evaluation
reusing modules from previously trained modular networks. A DL-SCA modular
network consists of two main modules; a downsampler and a classifier. We
demonstrate that modules from a modular network can be detached and attached
to other modular networks and conduct an efficient SCA evaluation. The
strategy is to use a classifier with good performance and reuse it to conduct
another evaluation in a different dataset.

Our experiments demonstrate that it is not mandatory to re-train a classifier
module to effectively evaluate the aimed dataset, regardless of whether the
source classifier has been trained with a dataset with a lower noise level. We
systematically lock the layers of the classifier to restrict them from getting
trained, replicating different sharing protocols to evaluate the effectiveness
of our approach.

As we said in the paper, we aim to work with more sharing protocols and
improve the performance of our modular network in future works by using other
types of deep learning architecture for the downsampler. Furthermore, we look
for applying methodologies that might help tun the hyperparameters of a
modular network.

## Acknowledgment

Acknowledge might change once this paper gets accepted.

## References

  * [1] Ancona, M., Ceolini, E., Öztireli, C., Gross, M.: Towards better understanding of gradient-based attribution methods for deep neural networks. arXiv preprint arXiv:1711.06104 (2017)
  * [2] Blei, D.M., Ng, A.Y., Jordan, M.I.: Latent dirichlet allocation. the Journal of machine Learning research 3, 993–1022 (2003)
  * [3] Blömer, J., Guajardo, J., Krummel, V.: Provably Secure Masking of AES. In: Selected Areas in Cryptography, pp. 69–83. Springer Berlin Heidelberg (2004)
  * [4] Cagli, E.: Feature Extraction for Side-Channel Attacks. Ph.D. thesis, Sorbonne université (2018)
  * [5] Cagli, E., Dumas, C., Prouff, E.: Enhancing dimensionality reduction methods for side-channel attacks. In: International Conference on Smart Card Research and Advanced Applications. pp. 15–33. Springer (2015)
  * [6] Cagli, E., Dumas, C., Prouff, E.: Kernel Discriminant Analysis for Information Extraction in the Presence of Masking. In: International Conference on Smart Card Research and Advanced Applications. pp. 1–22. Springer (2016)
  * [7] Choudary, M.O., Kuhn, M.G.: Efficient, Portable Template Attacks. IEEE Transactions on Information Forensics and Security 13(2), 490–501 (Feb 2018)
  * [8] Daemen, J., Rijmen, V.: The Design of Rijndael. Springer-Verlag, Berlin, Heidelberg (2002)
  * [9] Dunteman, G.H.: Principal components analysis. No. 69, Sage (1989)
  * [10] Goodfellow, I., Bengio, Y., Courville, A.: Deep learning. MIT press (2016), http://www.deeplearningbook.org
  * [11] Kong, Y., Mukhtar, N.: On Features Suitable for Power Analysis — Filtering the Contributing Features for Symmetric Key Recovery. 2018 6th International Symposium on Digital Forensic and Security (ISDFS) pp. 1–6 (2018)
  * [12] Lerman, L., Poussier, R., Markowitch, O., Standaert, F.X.: Template Attacks Versus Machine Learning Revisited and The Curse of Dimensionality in Side-Channel Analysis: extended version. Journal of Cryptographic Engineering 8(4), 301–313 (apr 2017)
  * [13] Mangard, S., Oswald, E., Popp, T.: Power Analysis Attacks: Revealing the Secrets of Smart Cards, vol. 31. Springer Science & Business Media (2008)
  * [14] Masure, L., Dumas, C., Prouff, E.: Gradient visualization for general characterization in profiling attacks. In: International Workshop on Constructive Side-Channel Analysis and Secure Design. pp. 145–167. Springer (2019)
  * [15] Mukhtar, N., Fournaris, A., Khan, T.M., Dimopoulos, C., Kong, Y.: Improved Hybrid Approach for Side-Channel Analysis Using Efficient Convolutional Neural Network and Dimensionality Reduction. IEEE Access 8, 184298–184311 (2020)
  * [16] Mukhtar, N., Papachristodoulou, L., Fournaris, A.P., Batina, L., Kong, Y.: Machine-Learning assisted Side-Channel Attacks on RNS-based Elliptic Curve Implementations using Hybrid Feature Engineering. Cryptology ePrint Archive, Report 2020/1065 (2020), https://ia.cr/2020/1065
  * [17] Müller, M.: Dynamic time warping. Information retrieval for music and motion pp. 69–84 (2007)
  * [18] Paguada, S., Armendariz, I.: The Forgotten Hyperparameter: Introducing Dilated Convolution for Boosting CNN-Based Side-Channel Attacks. In: International Conference on Applied Cryptography and Network Security. pp. 217–236. Springer (2020)
  * [19] Paguada, S., Batina, L., Armendariz, I.: Toward Practical Autoencoder-based Side-channel Analysis Evaluations. Computer Networks p. 108230 (2021). https://doi.org/https://doi.org/10.1016/j.comnet.2021.108230, https://www.sciencedirect.com/science/article/pii/S1389128621002735
  * [20] Paguada, S., Batina, L., Buhan, I., Armendariz, I.: Being Patient and Persistent: Optimizing An Early Stopping Strategy for Deep Learning in Profiled Attacks. arXiv e-prints pp. arXiv–2111 (2021)
  * [21] Picek, S., Heuser, A., Jovic, A., Batina, L.: A systematic evaluation of profiling through focused feature selection. IEEE Transactions on Very Large Scale Integration (VLSI) Systems 27(12), 2802–2815 (2019)
  * [22] Picek, S., Heuser, A., Jovic, A., Batina, L., Legay, A.: The secrets of profiling for side-channel analysis: feature selection matters. IACR Cryptol. ePrint Arch. 2017, 1110 (2017)
  * [23] Prouff, E., Strullu, R., Benadjila, R., Cagli, E., Canovas, C.: Study of Deep Learning Techniques for Side-Channel Analysis and Introduction to ASCAD Database. IACR Cryptology ePrint Archive 2018, 53 (2018)
  * [24] Shrikumar, A., Greenside, P., Shcherbina, A., Kundaje, A.: Not just a black box: Learning important features through propagating activation differences. arXiv preprint arXiv:1605.01713 (2016)
  * [25] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [26] Standaert, F.X., Malkin, T.G., Yung, M.: A Unified Framework for the Analysis of Side-Channel Key Recovery Attacks. In: Joux, A. (ed.) Advances in Cryptology - EUROCRYPT 2009. pp. 443–461. Springer Berlin Heidelberg, Berlin, Heidelberg (2009)
  * [27] Wu, L., Picek, S.: Remove Some Noise: On Pre-processing of Side-channel Measurements with Autoencoders. IACR Transactions on Cryptographic Hardware and Embedded Systems 2020(4), 389–415 (Aug 2020). https://doi.org/10.13154/tches.v2020.i4.389-415
  * [28] Yang, G., Li, H., Ming, J., Zhou, Y.: CDAE: Towards Empowering Denoising in Sidechannel Analysis. In: International Conference on Information and Communications Security. pp. 269–286 (2020)
  * [29] Zaid, G., Bossuet, L., Habrard, A., Venelli, A.: Methodology for Efficient CNN Architectures in Profiling Attacks. IACR Transactions on Cryptographic Hardware and Embedded Systems Volume 2020, Issue 1– (2019)
  * [30] Zhang, D., Wang, H., Figueiredo, M., Balzano, L.: Learning to share: Simultaneous parameter tying and sparsification in deep learning. In: International Conference on Learning Representations (2018)