marginparsep has been altered.
topmargin has been altered.
marginparwidth has been altered.
marginparpush has been altered.
The page layout violates the ICML style. Please do not change the page layout,
or include packages like geometry, savetrees, or fullpage, which change it for
you. We’re not able to reliably undo arbitrary changes to the style. Please
remove the offending package(s), or layout-changing commands and try again.

Factors for the Generalisation of Identity Relations by Neural Networks

Anonymous Authors1  and

††footnotetext:  1Anonymous Institution, Anonymous City, Anonymous Region,
Anonymous Country. Correspondence to: Anonymous Author
<EMAIL_ADDRESS>
Preliminary work. Under review by the ICML 2019 Workshop “Understanding and
Improving Generalization in Deep Learning”. Do not distribute.

###### Abstract

Many researchers implicitly assume that neural networks learn arbitrary
relations and generalise them to new unseen data. It has been shown recently,
however, that the generalisation of feed-forward networks fails for identity
relations. The proposed solution for this problem is to create an inductive
bias with Differential Rectifier (DR) units. In this work we explore various
factors in the neural network architecture and learning process whether they
make a difference to the generalisation on equality detection of Neural
Networks without and and with DR units in early and mid fusion architectures.

We find in experiments with synthetic data effects of the number of hidden
layers, the activation function and the data representation. The training set
size in relation to the total possible set of vectors also makes a difference.
However, the accuracy never exceeds 61% without DR units at 50% chance level.
DR units improve generalisation in all tasks and lead to almost perfect test
accuracy in the Mid Fusion setting. Thus, DR units seem to be a promising
approach for creating generalisation abilities that standard networks lack.

##  1 Introduction

Humans are very effective in learning abstract patterns that don’t depend on
the values of items but only on their relations, such as the identity of
pattern components, and applying them to new stimuli, often after very brief
exposure to a sequence of patterns Marcus et al. (1999). There, it was also
shown that recurrent neural networks do not generalise identity rules. This
has recently been confirmed for the most common neural network types and the
‘Relation Based Patterns’ (RBP) solution, based on ‘Differentiator-Rectifier’
(DR) units has been introduced in Weyde & Kopparti (2018a). The more
fundamental problem of learning equality relations has identified and a
solution with DR units has been proposed in Weyde & Kopparti (2018b).

In this work, we study the generalisation behaviour of neural networks,
especially feed-forward neural networks (FFNN) on the equality detection task.
FFNNs are universal approximators Leshno et al. (1993) and thus the modelling
of equality between binary vectors is in their hypothesis space. The failure
to generalise is therefore in the learning process and we investigate the
effect of the learning parameters on the generalisation of FFNNs with and
without DR units.

The remainder of this paper is organised as follows: Section 2 presents a
literature review, including a description of the RBP/DR structures and the
task of abstract rule learning. Section 3 presents the experimental results
and discussion, and the conclusions of this paper follow in Section 4.

##  2 Related work

The question of systematic learning of abstractions by neural networks has
been posed already in Fodor & Pylyshyn (1988). In a well-known study a
recurrent neural network failed to distinguish abstract patterns, based on
equality relations between sequence elements, although seven-month-old infants
could distinguish them after a few minutes of exposure (Marcus et al., 1999).
This was followed by an exchange on rule learning by neural networks and in
human language acquisition with different approaches to training and designing
neural networks (Elman, 1999; Altmann & Dienes, 1999; Shultz & Bale, 2001;
Vilcu & Hadley, 2001; 2005; Shultz & Bale, 2006; Vilcu & Hadley, 2005; Shastri
& Chang, 1999; Dominey & Ramus, 2000; Alhama & Zuidema, 2016).

A more specific problem of learning equality relations was posed in Marcus
(2001) by showing that neural network learning of equality on even numbers in
binary representation does not generalise to odd numbers. Mitchell et al.
(2018) addressed this problem with different approaches as an example for
extrapolation and inductive biases for machine learning in natural language
processing. Inductive biases can be realised in a number of ways and have
attracted increased interest recently. They have been applied, for instance,
to spatial reasoning Hamrick et al. (2018), to arithmetic tasks Trask et al.
(2018) and to learning from few examples Snell et al. (2017), suggesting a
number of potential benefits.

RBP and DR units have been introduced in Weyde & Kopparti (2018a) and Weyde &
Kopparti (2018b) as a way to create an inductive bias for learning identity
relations. DR units compare two input values by calculating the absolute
difference: $f(x,y)=|x-y|$. Pairs of vectors are compared with one DR unit for
every vector dimension with weights from the inputs to the DR units fixed at
$1$. There are two ways of adding DR units to a standard neural network. In
Early Fusion, DR units are concatenated to input units, and in Mid Fusion they
are concatenated to the first hidden layer. The DR units have activation $0$
for equal input and positive values otherwise. Learning the suitable summation
weights for the DRs is sufficient for creating a generalisable equality
detector, so that they make this learning task easier. They have been shown to
have a positive effect on real-life tasks in Weyde & Kopparti (2018a) and no
adverse effects were found on other tasks in Weyde & Kopparti (2018b).

##  3 Experiments and Results

The task here is a simple one: detecting if two binary vectors $v_{1},v_{2}$
with $n$ dimensions are equal. $v_{1}$ and $v_{2}$ are concatenated as input
to a neural network. We extend the work in Weyde & Kopparti (2018b) by
experimenting with various factors to determine their effect on
generalisation.

Our plain FFNN is a network with one hidden layer and ReLU activation.
Throughout our experiments, the network is trained for 20 epochs, which led to
convergence in all cases and 100% average training accuracy. We run 10
simulations for each configuration and round accuracy averages. We test the
significance of differences between models with a Wilcoxon Signed Rank Test
with threshold $p=.05$, over the results of the simulations, which does not
assume normal distribution and tests for different medians.

###  3.1 Variations of the data size

#### Vector dimensionality

We generate pairs of random binary vectors with dimensionality $n$ between 2
and 100 as shown in table 1. For $n<10$ we use all the possible binary vectors
to generate equal pairs, i.e. $2^{n}$ equal pairs, and random vectors for the
unequal pairs. For $n\leq{}10$ we use a random, class balanced selection of
10000 vectors. We randomly generate the same number of unequal vector pairs.
Then we use stratified sampling to split the data 75:25 into train and test
set. The results are shown in Table 1.

We see that the standard FFNNs barely exceed chance level (50%). The early
fusion model improves results, but never reaches full generalisation. The Mid
Fusion reaches close to perfect test performance. We tested for significant
differences between the lowest dimensionalities ($n=2,3$) vs the highest
($n=90,100$), with a sample size of 20 each. We found significant differences
only for the DR Early and Mid Fusion ($p<.01$), where accuracy is better for
lower dimensionalities, but not for the plain FFNN.

Vector | Plain | DR Early | DR Mid
---|---|---|---
Dimension | FFNN | Fusion | Fusion
n=2 | 52% | 66% | 100%
n=3 | 55% | 65% | 100%
n=5 | 51% | 67% | 100%
n=10 | 51% | 65% | 100%
n=20 | 49% | 63% | 100%
n=30 | 50% | 65% | 100%
n=40 | 50% | 64% | 100%
n=50 | 51% | 65% | 100%
n=60 | 48% | 64% | 100%
n=70 | 50% | 62% | 100%
n=80 | 50% | 63% | 100%
n=90 | 49% | 62% | 100%
n=100 | 50% | 64% | 100%
SD | 1.59 | 1.23 | 0.07
Table 1: Accuracy of the different network types on pairs of vectors of
different dimensions. The joint train and test data covers all possible equal
vector pairs for $n<10$, and a random, class-balanced selection of 10000
vector pairs where $n\geq 10$. The standard deviation (SD) is given in
percentage points.

#### Training data size

We study here how much the performance depends on the training data size. For
this, we vary only the training data size and keep the test set and all other
parameters constant. We use training data sizes of 1% to 50% (in relation to
the totally available data as defined above) and the accuracy achieved in
various conditions is plotted in Figure 1. The Mid Fusion network reaches 100%
accuracy from 10% data size on while the FFNN shows only small learning
effects.

Figure 1: Accuracy of plain FFNN and DR variants for 10-dimensional binary
vectors when varying the distributions of training data from 1% to 50%,
keeping the testing data fixed at 50%.

In order to further study the effect of increased training data we also use
higher proportions of the total data as training data, which means that we
have to reduce the test data size. We vary the train/test split from 75/25% of
training and testing data up to 95/5% on pairs of 10-dimensional vectors. The
results are given in Figure 2, showing gradual improvements of the Early
Fusion DR and plain FFNN models, but even with 95% of all possible
combinations in the training set the accuracy never exceeds 72% and 68%,
respectively.

Figure 2: Accuracy of plain FFNN and DR variants for 10-dimensional vectors
when varying the size of training data set from 75% to 95% of the total, with
the remaining data used for testing.

#### Vector coverage

A possible hypothesis for the results of the FFNN is that the coverage of the
vectors in the training set plays a role. We do not share vectors in equal
pairs between training and test set, as it would mean to train on the test
data. Instead we create a training set that contains in its unequal pairs all
vectors that appear in the test set. The results are shown in column a) of
Table 2. We also created a training set where each vector appeared as above,
but in two pairs, one in position $v_{1}$ and once in $v_{2}$. The results are
shown in column b) of Table 2. The results in both cases are similar to those
without this additional coverage in Table 1.

Type | Plain | DR Early | DR Mid
---|---|---|---
| FFNN | Fusion | Fusion
a) one | 50% (1.56) | 75% (1.17) | 100% (0.03)
b) both | 52% (1.56) | 87% (1.14) | 100% (0.04)
Table 2: Test set accuracy (standard deviation) of plain FFNNs and with DR
units for test set vectors appearing in training in a) one position, or b)
both positions for $n=10$. Accuracies for a) and b) are not significantly
different to the case of random vectors for plain FFNNs, but the improvement
is significant for Early and Mid Fusion DR models.

###  3.2 Variations of the network architecture

#### Network depth

Given the general success of deep networks, we hypothesised that depth might
help in learning a generalisable solution and thus tested networks with
greater numbers of hidden layers. In the mid fusion architecture, the DR units
are always concatenated to the first layer that is connected to the inputs.
The results show some improvement, but not to a large extent, as we increase
the number of hidden layers as given in Table 3. The difference between the
shallow networks (1 and 2 layers) and the deep networks (4 and 5 layers) is
statistically significant ($p<.01$). This is an interesting result, as the
additional depth is not necessary for representing a generalisable solution,
but does help somewhat to find such a solution. It is not obvious, why deeper
networks learn somewhat more generalisable solutions, given that Early Fusion
models, where there is one more layer between the DR units and the outputs,
are doing worse than the Mid Fusion models.

Number of | Plain | DR Early | DR Mid
---|---|---|---
hidden layers | FFNN | Fusion | Fusion
h=1 | 55% | 65% | 100%
h=2 | 57% | 69% | 100%
h=3 | 58% | 69% | 100%
h=4 | 60% | 72% | 100%
h=5 | 61% | 73% | 100%
SD | 1.57 | 1.59 | 0.05
Table 3: Test accuracy for different numbers of hidden layers with 10 neurons
each, for vector dimension n=3. Deeper networks are significantly better than
shallower ones (see text for details).

#### Hidden layer width

Based on the positive effect of a larger model with more parameters, we also
evaluated the performance of the network using a single hidden layer and
varying the number of neurons in that layer. We considered hidden layer size
of 10 to 100, again with and without DR units, the results are tabulated in
Table 4. The observation here is that the larger models do not improve in
performance when width is changed instead of depth. There is no significant
difference between the two smallest networks ($h_{n}=10,20$) and the largest
($h_{n}=80,100$) for Plain FFNNs, while for the DR models the smaller networks
perform significantly better.

Hidden layer | Plain | DR Early | DR Mid
---|---|---|---
size | FFNN | Fusion | Fusion
$h_{n}=10$ | 50% | 65% | 100%
$h_{n}=20$ | 49% | 63% | 100%
$h_{n}=30$ | 51% | 61% | 100%
$h_{n}=40$ | 45% | 65% | 100%
$h_{n}=50$ | 47% | 65% | 100%
$h_{n}=80$ | 51% | 62% | 100%
$h_{n}=100$ | 50% | 64% | 100%
SD | 1.62 | 1.23 | 0.04
Table 4: Accuracy of different network types for n=3. The networks contain a
single hidden layer of variable size $h_{n}$.

###  3.3 Other factors

#### Activation function

Another approach to change the learning behaviour is using different
activation functions in the hidden layer. For n=3, we evaluated the networks
with ReLU, Sigmoid and Tanh activations. The results are given in Table 5 and
they show that the type of activation in the hidden layer has small positive
effect in the overall accuracy, which is significant for all models.

Activation | Plain | DR Early | DR Mid
---|---|---|---
function | FFNN | Fusion | Fusion
ReLu | 55% (1.38) | 65% (1.23) | 100% (0.05)
Sigmoid | 58% (1.28) | 69% (1.18) | 100% (0.03)
Tanh | 58% (1.29) | 69% (1.17) | 100% (0.03)
Table 5: Test accuracy (standard deviation) of the network for different
activation functions for vector dimension n=3

#### Data representation

We replace 0;1 with -1;1, like in (Courbariaux et al., 2016), but only applied
to the inputs. The results of the accuracy for n=3 with and without DR units
are given in Table 6. We see an improvement, it is relatively small but
significant across all models.

Type | Plain | DR Early | DR Mid
---|---|---|---
| FFNN | Fusion | Fusion
a) 0/1 | 55% (1.23) | 65% (1.09) | 100% (0.03)
b) -1/1 | 58% (1.19) | 69% (1.07) | 100% (0.03)
Table 6: Accuracy of the network for data representation. a) standard 0/1
representation and b) ‘sign’ -1/1 representation for vector dimension n=3.

We also tested a FFNN with 5 hidden layers, sigmoid activation function and
-1/1 representation. Apparently the gains do not accumulate, as the test set
accuracy was only 58%.

###  3.4 Other tasks

Given the positive results for the DR mid fusion architecture for vector
pairs, we want to test whether the DR units have an effect, possibly negative,
on other learning tasks. We test a) comparison of the two vectors in the pair,
b) checking if the digit sum is $\geq 3$, The results are shown in Table 7. In
both a) and b) we see, that the performance is actually not hindered but
helped by the DR units.

We also test two tasks the DR architecture was not designed for: c) digit
reversal ($v_{1}=flip(v_{2})$) and d) parity checking (digit sum mod 2). The
DR units organised per corresponding input neurons do not deliver a perfect
solution here, but still lead to better results than a plain FFNN. The
differences of Plain FFNN vs Early Fusion and Early vs Mid Fusion DR are
statistically significant for all tasks.

Task | Plain | DR Early | DR Mid
---|---|---|---
| FFNN | Fusion | Fusion
a) | 75% (1.08) | 92% (0.94) | 100% (0.03)
b) | 77% (1.05) | 82% (1.03) | 100% (0.03)
c) | 50% (1.62) | 55% (1.53) | 58% (1.50)
d) | 51% (1.62) | 55% (1.53) | 62% (1.45)
Table 7: Test set accuracy (standard deviation) of FFNN without and with DR
units for a) numeric comparison ($v_{1}\geq v_{2}$), b) digit sum $\geq 3$, c)
inversion of digit order and d) parity check.

##  4 Conclusions

In this study we examined the lack of generalisation of identity rules by
feed-forward neural networks and the effect of various factors in network
architecture and learning method. While data dimensionality, hidden layer size
and vector coverage had no influence, the data representation, activation
function and network depth do lead to some improvements. Including DR units
leads to substantial improvements. DR Mid Fusion reaches almost perfect
generalisation, even from small amounts of training data, in all variants of
identity detection.

Identity is a fundamental task that many practitioners expect neural networks
to solve. We therefore believe it is important to investigate the design of
more techniques for creating and controlling inductive biases in neural
network learning, as we find that even relatively a simple task like learning
identity rules requires them for good generalisation.

We see two sets of general questions that these results raise. First: Why do
FFNNs not learn to generalise vector identity? Which other relations do neural
networks not learn? What does that mean for more complex tasks? Second: What
kinds of inductive biases should we design and how can we implement them for
more complex tasks?

## References

  * Alhama & Zuidema (2016) Alhama, R. G. and Zuidema, W.  Pre-wiring and pre-training: What does a neural network need to learn truly general identity rules.  _CoCo at NIPS_ , 2016.
  * Altmann & Dienes (1999) Altmann, G. and Dienes, Z.  Technical comment on rule learning by seven-month-old infants and neural networks.  _In Science_ , 284(5416)):875–875, 1999.
  * Courbariaux et al. (2016) Courbariaux, M., Hubara, I., Soudry, D., El-Yaniv, R., and Bengio, Y.  Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1.  _arXiv preprint arXiv:1602.02830_ , 2016.
  * Dominey & Ramus (2000) Dominey, P. and Ramus, F.  Neural network processing of natural language: Isensitivity to serial, temporal and abstract structure of language in the infant.  _Language and Cognitive Processes_ , pp. 15(1),87–127, 2000.
  * Elman (1999) Elman, J.  Generalization, rules, and neural networks: A simulation of marcus et. al.  _https://crl.ucsd.edu/ elman/Papers/MVRVsimulation.html_ , 1999.
  * Fodor & Pylyshyn (1988) Fodor, J. A. and Pylyshyn, Z. W.  Connectionism and cognitive architecture: A critical analysis.  _Cognition_ , 28(1-2):3–71, 1988.
  * Hamrick et al. (2018) Hamrick, J. B., Allen, K. R., Bapst, V., Zhu, T., McKee, K. R., Tenenbaum, J. B., and Battaglia, P. W.  Relational inductive bias for physical construction in humans and machines.  _arXiv preprint arXiv:1806.01203_ , 2018.
  * Leshno et al. (1993) Leshno, M., Lin, V. Y., Pinkus, A., and Schocken, S.  Multilayer feedforward networks with a nonpolynomial activation function can approximate any function.  _Neural networks_ , 6(6):861–867, 1993.
  * Marcus (2001) Marcus, G. F.  The algebraic mind: Integrating connectionism and cognitive science.  _Cambridge MIT Press_ , 2001.
  * Marcus et al. (1999) Marcus, G. F., Vijayan, S., Rao, S., and Vishton, P.  Rule learning by seven-month-old infants.  _Science, 283_ , 5398:77–80, 1999.
  * Mitchell et al. (2018) Mitchell, J., Minervini, P., Stenetorp, P., and Riedel, S.  Extrapolation in nlp.  _arXiv:1805.06648_ , 2018.
  * Shastri & Chang (1999) Shastri and Chang.  A spatiotemporal connectionist model of algebraic rule-learning.  _International Computer Science Institute_ , pp. TR–99–011, 1999\.
  * Shultz & Bale (2001) Shultz, T. R. and Bale, A. C.  Neural network simulation of infant familiarization to artificial sentences: Rule-like behavior without explicit rules and variables.  _Infancy, 2:4, 501-536, DOI: 10.1207/S15327078IN020407_ , 2001.
  * Shultz & Bale (2006) Shultz, T. R. and Bale, A. C.  Neural networks discover a near-identity relation to distinguish simple syntactic forms.  _Minds and Machines_ , 16(2):107–139, 2006.
  * Snell et al. (2017) Snell, J., Swersky, K., and Zemel, R.  Prototypical networks for few-shot learning.  In _Advances in Neural Information Processing Systems_ , pp. 4077–4087, 2017.
  * Trask et al. (2018) Trask, A., Hill, F., Reed, S. E., Rae, J., Dyer, C., and Blunsom, P.  Neural arithmetic logic units.  In _Advances in Neural Information Processing Systems_ , pp. 8046–8055, 2018.
  * Vilcu & Hadley (2001) Vilcu, M. and Hadley, R. F.  Generalization in simple recurrent networks.  In _Proceedings of the Annual Meeting of the Cognitive Science Society_ , volume 23, 2001.
  * Vilcu & Hadley (2005) Vilcu, M. and Hadley, R. F.  Two apparent ‘counterexamples’ to marcus: A closer look.  _Minds and Machines_ , 15(3-4):359–382, 2005\.
  * Weyde & Kopparti (2018a) Weyde, T. and Kopparti, R.  Modeling identity rules with neural networks.  _https://arxiv.org/abs/1812.02616_ , 2018a.
  * Weyde & Kopparti (2018b) Weyde, T. and Kopparti, R.  Feed-forward neural networks need inductive bias to learn equality relations.  2018b.


# Securing IoT Communication using Physical Sensor Data - Graph Layer Security
with Federated Multi-Agent Deep Reinforcement Learning

Liang Wang, Zhuangkun Wei and Weisi Guo {liang.wang.133, zhuangkun.wei,
<EMAIL_ADDRESS>School of Aerospace, Transport and Manufacturing,
Cranfield University, Milton Keynes, UK

###### Abstract

Internet-of-Things (IoT) devices are often used to transmit physical sensor
data over digital wireless channels. Traditional Physical Layer Security
(PLS)-based cryptography approaches rely on accurate channel estimation and
information exchange for key generation, which irrevocably ties key quality
with digital channel estimation quality. Recently, we proposed a new concept
called Graph Layer Security (GLS), where digital keys are derived from
physical sensor readings. The sensor readings between legitimate users are
correlated through a common background infrastructure environment (e.g., a
common water distribution network or electric grid). The challenge for GLS has
been how to achieve distributed key generation. This paper presents a
Federated multi-agent Deep reinforcement learning-assisted Distributed Key
generation scheme (FD2K), which fully exploits the common features of physical
dynamics to establish secret key between legitimate users. We present for the
first time initial experimental results of GLS with federated learning,
achieving considerable security performance in terms of key agreement rate
(KAR), and key randomness.

###### Index Terms:

Graph Layer Security, Multi-agent, Deep Reinforcement Learning, Distributed
Key Generation, Federated learning, and Feature Extraction.

##  I Introduction

Recently, with the advances in wireless sensor networks and embedded systems,
our life has been revolutionized by a variety of Internet-of-Things (IoT)
devices, which provide ubiquitous connections among environments. More and
more IoT-based applications have been implemented in healthcare and smart
cities. For instance, in a complex networked system, such as an oil, gas, or
water network, the maintenance team can have 24/7 monitoring of the conditions
of the network by deploying multiple embedded IoT devices. However, as the
data obtained by IoT devices are normally related to privacy, sensitivity, and
confidentially, which are easily captured by adversaries or eavesdroppers, the
research challenge lies in how to secure the data through wireless
communication.

###  I-A Physical Layer Security

The traditional method for securing the data is to apply a complexity-based
cryptography scheme, which, however, does not have an information theory-based
secrecy guarantee, and is therefore vulnerable to post-quantum threat. In
order to achieve the theoretical secure capacity, physical layer security
(PLS)-based key generation has been viewed as an efficient technology, which
extracts common randomness of features to generate secret keys between two
legitimate users, i.e, Alice and Bob [1]. Specifically, Alice and Bob keep
measuring their common channel through a key generation protocol including
channel probing, quantization, information reconciliation, and privacy
amplification, and finally generate symmetric cipher keys at both ends [2].
Current PLS-based key generation schemes leverage to extract common features
from channel state information (CSI). In [3], the authors were the first to
propose a sufficient key generation algorithm in WIFI system, in which both
the received signal strength indicator (RSSI) and the peak of channel impulse
response (CIR) were considered. In order to break the barrier that RSSI and
the peak of CIR can only provide limited information from wireless channels,
it was found that there are more CSI about multiple sub-carriers of orthogonal
frequency-division multiplexing (OFDM) [4, 5]. In [6], Zhang  _et al._
proposed a key generation scheme that exploits the subcarriers’ channel
response over time in OFDM system. In [7], Xi  _et al._ proposed a key
generation scheme named KEEP, which utilized a validation recombination
mechanism to generate keys from CSI measurements of all subcarriers. In [8],
the authors conducted extensive experiments about the different configurations
of long-range communication technology (LoRA) on the key generation
performance. In [9], Zhang  _et al._ concluded that the movement of the LoRA
devices has a significant influence on the received power in a large-scale
environment and they proposed a differential value-based key generation
scheme. Other key generation schemes have been extensively studied for various
wireless scenarios, e.g., the fifth-generation (5G), and wireless body area
network (WBAN). In [10], the features of the virtual angle of arrival (AoA)
and angle of departure (AoD) were utilized to generate keys in the millimetre
wave multiple-input and multiple-output (mmWave MIMO) system and the authors
analyzed the performance. In [11], Zhang  _et al._ found that the interpulse
intervals (IPIs) between two adjacent peaks of heartbeat signals can be a good
random source for key generation in WBAN and they proposed a key establishment
protocol, which utilized the electrocardiography (ECG) and
photoplethysmography (PPG) signals to secure communications.

Although PLS-based key generation schemes have been widely applied in a
variety of wireless networks, it is still not suitable for IoT-embedded
networks. On the one hand, PLS assumes that the wireless channel between the
legitimates must contain enough characteristics in terms of reciprocity,
dynamics, and uniqueness [12]. That is to say, in order to establish reliable
keys, IoT devices should acquire reasonable signal processing capability and
rely on the high communication signal-to-noise ratio (SNR). On the other hand,
most IoT devices are normally buried underground, which means that reasonable
SNR can not be obtained. Therefore, PLS is not suitable when channel
estimation is not reliable.

###  I-B Graph Layer Security

In order to solve the issue that PLS requires a high SNR for channel
estimation, Graph Layer Security (GLS) [13] has become a promising approach.
Generally, in networked systems, the embedded physical dynamics can be
described by a common continuity equation, such as Navier Stokes for flow, and
Nonlinear Schrodinger for optical transmission. The IoT devices deployed in
networked systems can commonly detect the physical dynamics with high accuracy
and they do not suffer from high SNR for channel estimation. However, the
proposed scheme in [13] utilized the overall physical dynamics among the
networked system to generate the global keys for information
encryption/decryption between legitimate users. The global keys could be
easily captured by adversaries and the overall key generation system is not
fully distributed. Motivated by the previous work in GLS and in order to make
the generated keys robust against attacks from adversaries, we summarize the
main contributions of this paper as follows:

  * •

We propose a FD2K scheme in which IoT devices can generate cipher keys based
on their locally detected physical dynamics for information encryption and
decryption.

  * •

Considering the situation that physical dynamics of IoT devices stored locally
could be easily detected by the adversary, we utilize the technique of
federated learning during the training process to eliminate the process of
data collection, therefore, the privacy and security of the physical dynamics
are protected.

  * •

The multi-agent deep reinforcement learning technique is applied to generate
keys within the framework of centralized training and decentralized execution.
Thus, the overall key generation system can be fully distributed when
deploying.

The rest of the paper is organized as follows. The system model is described
in Section II. Then, we introduce the proposed FD2K scheme in Section III. The
simulation results are analyzed in Section IV. Finally, the conclusion is
given in Section V.

##  II System Model

Figure 1: Overall system model.

We describe the system model in Fig. 1, where two legitimate users, namely
Alice (A) and Bob (B), are placed to constantly monitor the physical signals
of a networked system with equipped IoT devices. The key generation scheme
consists of 6 stages, namely \small{1}⃝ signal acquisition, \small{2}⃝ common
feature extraction, \small{3}⃝ key generation, \small{4}⃝ information
encryption, \small{5}⃝ wireless transmission, and \small{6}⃝ information
decryption. Note that the networked system can be a water, oil, or gas network
and the physical signals can be pressure, flow rate, or temperature. For
simplicity, in this paper, we consider the water distribution network as the
networked system and water pressure as the physical signals.

We assume the overall process lasts $T$ time steps (TSs), in each of which,
both Alice and Bob have $M$ detected physical values. Therefore, the physical
signals detected in TS $t$ can be expressed as

$\displaystyle\bm{P^{i}_{t}}=[p^{i}_{t,1},...,p^{i}_{t,m}...,p^{i}_{t,M}],$
(1)

where $i=\\{A,B\\}$, $p^{i}_{t,m}$ is the $m$-th signal value obtained by
Alice or Bob in TS $t$.

Then, Alice and Bob can extract common features of obtained physical signals
to generate cipher keys. Note that in this paper, we assume the key generation
rate (KGR) is $1$, meaning the keys’ length is $M$. We denote the keys
generated in TS $t$ as

$\displaystyle\bm{K^{i}_{t}}=[k^{i}_{t,1},...,k^{i}_{t,m}...,k^{i}_{t,M}],$
(2)

where $i=\\{A,B\\}$, $k^{i}_{t,m}\in\\{0,1\\}$ denotes the $m$-th binary bit
of keys generated by Alice or Bob.

After that, one legitimate user (e.g., Alice) can use his/her generated cipher
keys to encrypt the information and then transmit the encrypted information
through a wireless channel to another (e.g., Bob), with the same process for
information decryption. As a result, secure communication between Alice and
Bob is conducted.

The challenge of key generation in this system model lies in how to generate
cipher keys in a distributed manner and protect the local physical signals
simultaneously. More precisely, during the process of key generation, Alice
and Bob have to fully exploit the common features of physical signals between
each other without any information exchange. Besides, they have to generate
well-qualified keys that satisfy the performance metrics, which we will
introduce in Section IV.

##  III Proposed FD2K Scheme

Figure 2: Structure of proposed FD2K scheme.

In this section, we introduce our proposed FD2K scheme. First of all, the
overall structure of FD2K is shown in Fig. 2, which can be described within
the framework of multi-agent deep reinforcement learning (MADRL) and is based
on the partially observable Markov decision process (POMDP). Specifically, the
networked system can be viewed as an environment. Alice and Bob are controlled
by its dedicated agent, which equips with two deep neural networks (DNNs),
namely actor
$a^{A}_{t}=\pi^{A}(o^{A}_{t}|\theta^{A}_{\pi}),a^{B}_{t}=\pi^{B}(o^{B}_{t}|\theta^{B}_{\pi})$
for policy approximation and critic
$Q^{A}(s_{t},a_{t}|\theta^{A}_{Q}),Q^{B}(s_{t},a_{t}|\theta^{B}_{Q})$, where
$t$ means the index of time step; $o^{A}_{t},o^{B}_{t}$ denote the local
observations of Alice and Bob; $a^{A}_{t},a^{B}_{t}$ are the local actions
generated actor networks, followed by policy functions
$\pi^{A}(\cdot|\theta^{A}_{\pi}),\pi^{B}(\cdot|\theta^{B}_{\pi})$ with network
parameters $\theta^{A}_{\pi},\theta^{B}_{\pi}$;
$s_{t}\triangleq\\{o^{A}_{t},o^{B}_{t}\\},a_{t}\triangleq\\{a^{A}_{t},a^{B}_{t}\\}$
are the state and action assembled by all agents;
$Q^{A}(\cdot|\theta^{A}_{Q}),Q^{B}(\cdot|\theta^{B}_{Q})$ are the action-value
functions with network parameters $\theta^{A}_{Q},\theta^{B}_{Q}$. Note that
both actor and critic network has their own target network with the same
structure, which can be expressed as
$\pi^{A^{\prime}}(\cdot|\theta^{A^{\prime}}_{\pi}),\pi^{B^{\prime}}(\cdot|\theta^{B^{\prime}}_{\pi}),Q^{A^{\prime}}(\cdot|\theta^{A^{\prime}}_{Q}),Q^{B^{\prime}}(\cdot|\theta^{B^{\prime}}_{Q})$
with parameters
$\theta^{A^{\prime}}_{\pi},\theta^{B^{\prime}}_{\pi},\theta^{A^{\prime}}_{Q},\theta^{B^{\prime}}_{Q}$
respectively. During the training process, the agents of Alice and Bob keep
sending local observations and local actions to the environment. Then, they
will receive the state, action and the environment will transfer to the next
state. The agents learn to train the actor, critic networks from exploitation
and exploration until the convergence is obtained, aiming at maximizing the
accumulated reward
$R_{t}=\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t}r_{t^{\prime}}$, where
$\gamma\in[0,1]$; $r_{t^{\prime}}$ is the reward function in TS $t^{\prime}$.

The training process of the traditional DRL-based system is implemented in a
simulator that requires the established training data based on previously
obtained physical signals, which have to be collected remotely and could be
known by Eve. In order to protect the security and privacy of the local
physical signals of Alice and Bob in the training process, federated learning
(FL) [14] is applied to eliminate the process of data collection. More
precisely, in the framework of the proposed scheme, a controller is deployed
in the proposed scheme. It first initializes a global actor network and
downloads it to Alice and Bob. Then, Alice and Bob take local training for
certain epochs and upload their local actor network models to the controller,
which takes aggregation and obtains a new actor network. Both Alice and Bob
repeat the process of downloading, local training, and uploading. The
controller repeats aggregation until the network is adequately trained.
Therefore, it is concluded that during the training process, the training data
of agents are stored locally and only the network model is
uploaded/downloaded, which can help to protect privacy and security.

Algorithm 1 Proposed FD2K

1: Controller initializes global actor network and downloads it to Alice
$\pi^{A}(\cdot|\theta^{A}_{\pi})$ and Bob $\pi^{B}(\cdot|\theta^{B}_{\pi})$;

2: Initialize critic networks
$Q^{A}(\cdot|\theta^{A}_{Q}),Q^{B}(\cdot|\theta^{B}_{Q})$ with parameters
$\theta^{A}_{Q}$, $\theta^{B}_{Q}$;

3: Initialize target networks:
$\theta^{i^{\prime}}_{\pi}\leftarrow\theta^{i}_{\pi},i\in\\{A,B\\}$,$\theta^{i^{\prime}}_{Q}\leftarrow\theta^{i}_{Q},i\in\\{A,B\\}$;

4: Initialize experience replay memory with the size of $N$;

5: for epoch $e=1$ to $e^{\textit{max}}$ do

6: Initialize $o^{A}_{t}$, $o^{B}_{t}$;

7: for TS t=1 to T do

8: Obtain $s_{t}\triangleq\\{o^{A}_{t},o^{B}_{t}\\}$;

9: Obtain local actions $a^{A}_{t}$, $a^{B}_{t}$ according to:
$a^{A}_{t}=\pi^{A}(o^{A}_{t}|\theta^{A}_{\pi})+\epsilon$ and
$a^{B}_{t}=\pi^{B}(o^{B}_{t}|\theta^{B}_{\pi})+\epsilon$;

10: Execute local actions $a^{A}_{t}$, $a^{B}_{t}$;

11: Obtain action $a_{t}\triangleq\\{a^{A}_{t},a^{B}_{t}\\}$;

12: Generate keys according to Algorithm 2;

13: Calculate reward function $r_{t}$;

14: Transfer the environment to $s_{t+1}\triangleq\\{o^{A}_{t+1}$,
$o^{B}_{t+1}\\}$;

15: Store transition $\\{s_{t},a_{t},r_{t},s_{t+1}\\}$ into experience replay
memory;

16: if learning process starts then

17: Random sample $n$ transitions from experience replay memory;

18: Update critic networks of Alice and Bob according to Eq. 4;

19: Update actor networks of Alice and Bob according to Eq. 3;

20: Update target networks with the rate $\rho$:
$\theta^{i^{\prime}}_{\pi}\leftarrow\rho\theta^{i}_{\pi}+(1-\rho)\theta^{i^{\prime}}_{\pi},i\in\\{A,B\\}$,$\theta^{i^{\prime}}_{Q}\leftarrow\rho\theta^{i}_{Q}+(1-\rho)\theta^{i^{\prime}}_{Q},i\in\\{A,B\\}$;

21: end if

22: end for

23: if $e$ mod $E==0$ then

24: Upload actor networks of Alice and Bob to the controller for federated
updating;

25: end if

26: end for

We further provide the overall pseudo code of the proposed FD2K scheme in
Algorithm 1. First of all, the controller initializes a global actor network
and downloads it to Alice and Bob. Then, Alice and Bob initialize their critic
and target networks. In Line 4, in order to avoid divergence and achieve
stable convergence performance, experience replay memory [15] with the size of
$N$ is deployed and initialized for storing the previous transitions. During
the training process, Alice and Bob initialize their local observations in the
first TS. Then, in each TS, their local observations are assembled to state
$s_{t}$ and they obtain their local actions by actor networks. Note that in
Line 9, an action noise $\epsilon$ is used for a better exploration and it
decays with a fixed rate. After that, Alice and Bob take common features of
physical signals to generate keys according to the obtained actions. More
details about the process of key generation are explained in Algorithm 2.
Then, the reward function is calculated with the given generated keys of Alice
and Bob in Eq. 7. From Line 14, the environment is transferred to the next
state and the transition is stored in the experience replay memory. When the
learning process starts, $n$ transitions are randomly sampled from the memory
to train the networks. Specifically, the actor networks of Alice and Bob are
trained by the following policy gradient method [16], whose equation can be
described as

$\displaystyle\triangledown_{\theta^{i}_{\pi}}J\approx\mathbb{E}\big{[}\triangledown_{\theta^{i}_{\pi}}\pi^{i}\big{(}o^{i}_{t}|\theta^{i}_{\pi}\big{)}\triangledown_{a^{i}_{t}}Q^{i}(s_{t},a_{t}|\theta^{i}_{Q})\big{]},$
(3)

where $i=\\{A,B\\}$. The critic networks of Alice and Bod are trained by the
loss function [17], which is:

$\displaystyle L(\theta^{i}_{Q})=\mathbb{E}\big{[}r_{t}+\gamma
Q^{i^{\prime}}(s_{t+1},a_{t+1}|\theta^{i^{\prime}}_{Q})-Q^{i}(s_{t},a_{t}|\theta^{i}_{Q})\big{]},$
(4)

where $i=\\{A,B\\}$ and $a_{t+1}$ is the action of next state.

Then, the target networks are updated as well at a rate of $\rho$. Finally,
from Line 23, both Alice and Bob upload their actor networks to the controller
and the aggregation process is carried out to obtain a new network. In this
paper, as the FedAvg [18] is applied, the aggregation process can be expressed
as:

$\displaystyle\bar{\theta_{\pi}}=\frac{\theta^{A}_{\pi}+\theta^{B}_{\pi}}{2},$
(5)

where $\bar{\theta_{\pi}}$ denotes the parameters of new network.

Therefore, we define the local observations, and local actions of Alice and
Bob as follows:

  * •

Local observation: we define the signal values detected by Alice and Bob in TS
$t$ as their observations, which are $o^{A}_{t}=\bm{P^{A}_{t}}$ and
$o^{B}_{t}=\bm{P^{B}_{t}}$.

  * •

Local action: in our proposed FD2K scheme, the local actions are used to
extract common features of the physical signals of Alice and Bob for key
generation. Specifically, let us denote two vectors
$\bm{A^{A}_{t}}\in\mathbb{R}^{M},\bm{A^{B}_{t}}\in\mathbb{R}^{M}$ for Alice
and Bob. For $m$-th values in $\bm{A^{A}_{t}}$ and $\bm{A^{B}_{t}}$, they
have:

$\displaystyle\bm{A^{i}_{t}}(m)=\begin{cases}1,&\text{if
$a^{i}_{t}(m)\geq\lambda$},\\\ 0,&\text{otherwise},\end{cases}$ (6)

where $\lambda$ is the threshold, $i\in\\{A,B\\}$. Specifically, if
$\bm{A^{A}_{t}}(m)=1$ or $\bm{A^{B}_{t}}(m)=1$, it means the $m$-th signal
value of $\bm{P^{A}_{t}}$ or $\bm{P^{B}_{t}}$ has a common feature that can be
used to generate keys.

  * •

Reward: we define the reward function as follows:

$\displaystyle r_{t}=$ $\displaystyle
1-\frac{\sum_{m=1}^{M}|\bm{K^{A}_{t}}(m)-\bm{K^{B}_{t}}(m)|}{M}$ (7)
$\displaystyle+\phi\frac{\sum_{m=1}^{M}\big{(}\bm{A^{A}_{t}}(m)+\bm{A^{B}_{t}}(m)\big{)}}{2M},$

where $1-\frac{\sum_{m=1}^{M}|\bm{K^{A}_{t}}(m)-\bm{K^{B}_{t}}(m)|}{M}$ is the
key agreement rate (KAR), which is one of the performance metrics of key
generation.
$\frac{\sum_{m=1}^{M}\big{(}\bm{A^{A}_{t}}(m)+\bm{A^{B}_{t}}(m)\big{)}}{2M}$
is the average percentage of common features applied to generate keys. $\phi$
is expressed as

$\displaystyle\phi=\begin{cases}1,&\text{if
$1-\frac{\sum_{m=1}^{M}|\bm{K^{A}_{t}}(m)-\bm{K^{B}_{t}}(m)|}{M}=1$},\\\
0,&\text{otherwise},\end{cases}$ (8)

Overall, the reward function is designed to guarantee that generated keys can
satisfy the KAR and fully utilize the common features of physical signals.

Algorithm 2 Key Generation

1: Obtain $\bm{P^{A}_{t}},\bm{P^{B}_{t}},\bm{A^{A}_{t}},\bm{A^{B}_{t}}$;

2: Initialize $\bm{K^{A}_{t}},\bm{K^{B}_{t}}$;

3: for m=1 : M do

4: for $i\in\\{A,B\\}$ do

5: if $\bm{A^{i}_{t}}(m)==1$ and $\bm{P^{i}_{t}}(m)\geq\bm{P^{i}_{t}}(m-1)$
then

6: $\bm{K^{i}_{t}}(m)\leftarrow 1$;

7: else

8: $\bm{K^{i}_{t}}(m)\leftarrow 0$ ;

9: end if

10: end for

11: end for

12: return $\bm{K^{A}_{t}}$ and $\bm{K^{B}_{t}}$;

We further give the overall algorithm design for key generation in Algorithm
2. Note that in this paper, the differentiation of physical signals is
captured as the common feature. Specifically,
$\bm{P^{A}_{t}},\bm{P^{B}_{t}},\bm{A^{A}_{t}},\bm{A^{B}_{t}}$ are first
obtained and $\bm{K^{A}_{t}},\bm{K^{B}_{t}}$ are initialized. Then, the
feature extraction for key generation is started. Specifically, for Alice, if
$\bm{A^{A}_{t}}(m)==1$ and $\bm{P^{A}_{t}}(m)\geq\bm{P^{A}_{t}}(m-1)$, we have
$\bm{K^{A}_{t}}(m)\leftarrow 1$, otherwise, we have
$\bm{K^{A}_{t}}(m)\leftarrow 0$. Similarly, Bob has the same key generation
scheme. Finally, the generated keys of Alice and Bob are obtained.

##  IV Experimental Results

Figure 3: The structure of the water distribution network.

In this section, we conduct experiments to analyze the performance of the
proposed FD2K scheme in a simulated water distribution network, which is
designed by an open-sourced simulation software - EPANET [19]. Specifically,
it is based on a small region of the Cherry Hill/Brushy Plains distribution
system and has a total demand of 0.90 million gallons per day (MGD), one tank
and 6.8 miles of pipe. The overall structure is shown in Fig. 3, in which we
choose Node 18 and Node 8 as Alice and Bob. We define the water pressures of
Alice and Bob as the physical signals and we record the previous 2 hours’
water pressure for training. The snapshot of the water pressures of Alice and
Bob is shown in Fig. 4.

Figure 4: The physical signals detected by Alice and Bob.

The proposed FD2K algorithm is implemented by Python 3.7 and Tensorflow 2.3.0.
Both the actor and critic networks of Alice and Bob have four connected hidden
layers, each of which has 100 neurons and $elu(\cdot)$ is applied as the
activation function. The activation function of the output layer of actor
networks is $sigmoid(\cdot)$. The learning rate of actor networks is $10^{-4}$
and the learning rate of critic networks is $10^{-3}$. The AdamOptimizer [20]
is applied. More parameters can be found in Table 1.

TABLE I: Simulation Parameters Notation | Description
---|---
M | 20
T | 19
$\gamma$ | 0.99
$n$ | 128
$N$ | 10000
$e^{max}$ | 3000
$\epsilon$ | 0.4
$\rho$ | 0.01
$E$ | 5
$\lambda$ | 0.5

In order to evaluate the performance of the proposed FD2K scheme, we introduce
the evaluation metrics for generated keys as follows:

  * •

Key Agreement Rate (KAR): KAR represents the percentage of the same key bits
between Alice and Bob, whose equation is described as a part of Eq. 7.

  * •

Randomness: as the keys are generated for encryption/decryption algorithms,
such as Advanced Encryption Standard (AES), and Data Encryption Standard
(DES). It is widely acknowledged that the keys should be random to avoid
force-brute attacks. National Institute of Standards and Technology (NIST)
[21] is proved to be an efficient random test suite to evaluate the randomness
of both true and pseudo random number generators. Specifically, given the
generated keys, each test of NIST will return a p-value. If the p-value is
larger than the threshold, such as 0.01, we can confirm the generated keys are
considered to pass the test.

  * •

Uniqueness: uniqueness is defined to evaluate whether the physical signals can
represent the different patterns and are different from each other. In this
paper, the uniqueness is guaranteed by the different physical signals from
Alice, Bob and Eve.

Figure 5: Accumulated reward versus training epochs.

First, we depict the accumulated reward of the proposed FD2K scheme in the
training process in Fig. 5. Specifically, it is observed that the training
curve remains below $0.8$ at the beginning. This is because the networks of
the proposed FD2K are not updated and some poor attempts, which lead to lower
rewards are explored by the agents of Alice and Bob. After that, the training
curve starts to increase consistently and finally keeps stable above $1.6$,
one plausible explanation is that the experience replay memory has enough
transitions and the training process starts. The networks of Alice and Bob are
trained to capture the common features of physical signals between each other
for key generation.

Figure 6: Comparison of KAR of Alice-Bob and Alice-Eve.

After the training stage, the networks of Alice and Bob are saved for
evaluation. Note that our proposed FD2K scheme is based on the framework of
centralized training and decentralized execution, both Alice and Bob enable to
generate keys with their locally observed physical signals without any
information/key exchange. As a result, the process of key generation can be
fully distributed. We evaluate the performance of keys generated by FD2K in
terms of KAR in Fig. 6. Considering the situation that most IoT devices
embedded in networked systems in the real world are normally buried
underground, the physical signals are extremely difficult to be obtained by
the adversary, namely Eve, or easily to be found if inserting extra IoT
devices. Additionally, in PLS, most of the digital attacks are based on the
assumption that the IoT devices detached in the networked system are viewed as
legitimate users, which means Eve can not insert any IoT device or can not
obtain the perfect physical signals. Therefore, we consider a practical case
that the actor network of Bob is attacked and known by Eve during the
federated learning but it can not perfectly obtain the physical signals of
Bob. Eve can capture the wireless signals from Alice and generate keys with
different physical signals for information decryption. From Fig. 6, we can
conclude that the average KAR of keys generated between Alice and Bob
consistently outperforms the average KAR between Alice and Eve. Precisely, the
generated keys between Alice and Bob have at least $32\%$ performance
improvement when the index of TSs is 3 and have at most $46\%$ performance
improvement when the index of TSs is 17, compared with the generated keys
between Alice and Eve.

TABLE II: p-values of NIST Test NIST Test | p-value
---|---
Monobit Frequency | 0.184077
Runs | 0.272715
Discrete Fourier Transform | 0.520943
Non Overlapping Template | 0.535998
Approximate Entropy | 0.545495
Cumulative Sums | 0.395994
Random Excursion | 0.394067
Random Excursion Variant | 0.600110

Finally, we evaluate the randomness of keys generated by our proposed scheme
in Table II with Python implementation of the NIST test suite. It is clear to
see that all p-values of keys are larger than $0.01$ and all generated keys
pass the NIST test, which concludes that FD2K is efficient to extract common
features of physical signals for key generation.

##  V Conclusion

In this paper, we have proposed a distributed key generation scheme, which
fully exploits the common features of physical dynamics embedded in networked
systems for key generation. The proposed key generation scheme is based on
multi-agent deep reinforcement learning approach, which enables legitimate
users to generate keys in a distributed manner. Besides, the federated
learning technique is applied to protect the privacy and security of local
training data. Experimental results conclude that the proposed key generation
scheme has efficient superiority for capturing common features between
different physical signals and the generated keys have considerable
performance improvement in terms of KAR, randomness, etc.

##  VI Acknowledgement

This work has been supported by the PETRAS National Centre of Excellence for
IoT Systems Cybersecurity, which has been funded by the UK EPSRC under grant
number EP/S035362/1.

## References

  * [1] U. Maurer, “Secret key agreement by public discussion from common information,” IEEE Transactions on Information Theory, vol. 39, no. 3, pp. 733–742, 1993.
  * [2] J. Zhang, B. He, T. Q. Duong, and R. Woods, “On the key generation from correlated wireless channels,” IEEE Communications Letters, vol. 21, no. 4, pp. 961–964, 2017.
  * [3] S. Mathur, W. Trappe, N. Mandayam, C. Ye, and A. Reznik, “Radio-telepathy: Extracting a secret key from an unauthenticated wireless channel,” in Proceedings of the 14th ACM International Conference on Mobile Computing and Networking, MobiCom ’08, (New York, NY, USA), p. 128–139, Association for Computing Machinery, 2008.
  * [4] Y. Liu, S. C. Draper, and A. M. Sayeed, “Exploiting channel diversity in secret key generation from multipath fading randomness,” IEEE Transactions on Information Forensics and Security, vol. 7, no. 5, pp. 1484–1497, 2012.
  * [5] D. Halperin, W. Hu, A. Sheth, and D. Wetherall, “Tool release: Gathering 802.11 n traces with channel state information,” ACM SIGCOMM computer communication review, vol. 41, no. 1, pp. 53–53, 2011.
  * [6] J. Zhang, A. Marshall, R. Woods, and T. Q. Duong, “Secure key generation from ofdm subcarriers’ channel responses,” in 2014 IEEE Globecom Workshops (GC Wkshps), pp. 1302–1307, 2014.
  * [7] W. Xi, X.-Y. Li, C. Qian, J. Han, S. Tang, J. Zhao, and K. Zhao, “Keep: Fast secret key extraction protocol for d2d communication,” in 2014 IEEE 22nd International Symposium of Quality of Service (IWQoS), pp. 350–359, 2014\.
  * [8] H. Ruotsalainen, J. Zhang, and S. Grebeniuk, “Experimental investigation on wireless key generation for low-power wide-area networks,” IEEE Internet of Things Journal, vol. 7, no. 3, pp. 1745–1755, 2019.
  * [9] J. Zhang, A. Marshall, and L. Hanzo, “Channel-envelope differencing eliminates secret key correlation: Lora-based key generation in low power wide area networks,” IEEE Transactions on Vehicular Technology, vol. 67, no. 12, pp. 12462–12466, 2018.
  * [10] L. Jiao, J. Tang, and K. Zeng, “Physical layer key generation using virtual aoa and aod of mmwave massive mimo channel,” in 2018 IEEE Conference on Communications and Network Security (CNS), pp. 1–9, 2018.
  * [11] J. Zhang, Y. Zheng, W. Xu, and Y. Chen, “H2k: A heartbeat-based key generation framework for ecg and ppg signals,” IEEE Transactions on Mobile Computing, 2021.
  * [12] X. Fang, N. Zhang, S. Zhang, D. Chen, X. Sha, and X. Shen, “On physical layer security: Weighted fractional fourier transform based user cooperation,” IEEE Transactions on Wireless Communications, vol. 16, no. 8, pp. 5498–5510, 2017.
  * [13] Z. Wei, L. Wang, S. C. Sun, B. Li, and W. Guo, “Graph layer security: Encrypting information via common networked physics,” Sensors, vol. 22, no. 10, 2022.
  * [14] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019.
  * [15] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015.
  * [16] R. S. Sutton, D. McAllester, S. Singh, and Y. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in Proceedings of the 12th International Conference on Neural Information Processing Systems, NIPS’99, (Cambridge, MA, USA), p. 1057–1063, MIT Press, 1999.
  * [17] R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” Neural Information Processing Systems (NIPS), 2017.
  * [18] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, pp. 1273–1282, PMLR, 2017.
  * [19] L. A. Rossman et al., “EPANET 2: users manual,” 2000.
  * [20] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [21] L. Bassham, A. Rukhin, J. Soto, J. Nechvatal, M. Smid, S. Leigh, M. Levenson, M. Vangel, N. Heckert, and D. Banks, “A statistical test suite for random and pseudorandom number generators for cryptographic applications,” 2010-09-16 2010.


# Brief technical note on linearizing recurrent neural networks (RNNs) before
vs after the pointwise nonlinearity

Marino Pagan  Simons Initiative for the Developing Brain, University of
Edinburgh, Edinburgh, UK  Adrian Valente  Laboratoire de Neurosciences
Cognitives et Computationnelles, INSERM U960, Ecole Normale Superieure - PSL
Research University, 75005 Paris, France  Srdjan Ostojic  Laboratoire de
Neurosciences Cognitives et Computationnelles, INSERM U960, Ecole Normale
Superieure - PSL Research University, 75005 Paris, France  equal Contribution
Carlos D. Brody  Howard Hughes Medical Institute and Princeton Neuroscience
Institute, Princeton University, Princeton NJ, USA  equal Contribution

(February 2023)

###### Abstract

Linearization of the dynamics of recurrent neural networks (RNNs) is often
used to study their properties. The same RNN dynamics can be written in terms
of the “activations” (the net inputs to each unit, before its pointwise
nonlinearity) or in terms of the “activities” (the output of each unit, after
its pointwise nonlinearity); the two corresponding linearizations are
different from each other. This brief and informal technical note describes
the relationship between the two linearizations, between the left and right
eigenvectors of their dynamics matrices, and shows that some context-dependent
effects are readily apparent under linearization of activity dynamics but not
linearization of activation dynamics.

###### Contents

  1. 1 Introduction
  2. 2 Two linearizations for the same discrete-time RNN Dynamics
  3. 3 Left and right eigenvectors of the dynamics matrices
  4. 4 Linearizations and context-dependence of input vectors
  5. 5 Conclusion

##  1 Introduction

Recurrent neural network (RNN) dynamics can be equivalently expressed in two
different forms [3]. One form describes the dynamics of the net input, or
“activations” of the units, usually interpreted as the membrane potential of
biological neurons. A second form describes dynamics in terms of the output,
i.e. “activity” or ”rate” of the units, often thought of as comparable to
spiking rates of biological neurons. A pointwise nonlinearity relates the two,
with the activity being the result of the nonlinearity after it is applied to
the activation.

Linearization of dynamics is often used to study the properties of dynamical
systems. But when considering an RNN, should one linearize the activity
dynamics? Or the activation dynamics? The two linearizations produce different
linear equations. What is the difference between them and what is the
relationship between the two? Do some conclusions depend on which
linearization is chosen?

This document explores these questions, and makes the relationship between the
two linearizations explicit. The two are related by a simple diagonal linear
transform that depends on the gains of each unit.

We additionally briefly consider effects of the two linearizations when
considering context-dependent networks [2, 4], in which each “context” is
defined by a constant vector of inputs to each unit, and point out that a
modulation by context of the linearized inputs to the RNN is observable only
in the activity space linearization, not in the activation space
linearization.

##  2 Two linearizations for the same discrete-time RNN Dynamics

Consider the standard recurrent neural network equations

$\displaystyle\hat{\mathbf{x}}^{k+1}$ $\displaystyle=$ $\displaystyle
W\hat{\mathbf{r}}^{k}+\mathbf{u}^{k}$ $\displaystyle\hat{\mathbf{r}}^{k+1}$
$\displaystyle=$ $\displaystyle{\rm g}(\hat{\mathbf{x}}^{k+1})$ (1)

where $\hat{\mathbf{x}}^{k}$ represents the vector of unit activations at
timepoint $k$, ${\rm g}()$ is a differentiable and invertible pointwise
nonlinear function such as $\tanh()$, $\hat{\mathbf{r}}^{k}$ is the vector of
unit activities at timepoint $k$, $W$ is a square matrix representing
recurrent connection weights, and $\mathbf{u}^{k}$ is a vector of external
inputs at timepoint $k$.

The dynamics of (1) can be rewritten entirely in terms of $\bf\hat{x}$. As we
do that, let us define the vector-valued dynamics function ${\mathbf{F}}_{x}$:

$\mathbf{F}_{x}(\hat{\mathbf{x}},\mathbf{u})=W{\rm
g}(\hat{\mathbf{x}})+\mathbf{u},$ (2)

so that

$\displaystyle\hat{\mathbf{x}}^{k+1}$ $\displaystyle=$
$\displaystyle\mathbf{F}_{x}(\hat{\mathbf{x}}^{k},\mathbf{u}^{k}).$ (3)

Similarly, we can define the dynamics function ${\mathbf{F}}_{r}$

$\mathbf{F}_{r}(\hat{\mathbf{r}},\mathbf{u})\;=\;{\rm
g}(W\hat{\mathbf{r}}+\mathbf{u})$ (4)

and rewrite the dynamics (1) entirely in terms of $\bf\hat{r}$,

$\displaystyle\hat{\mathbf{r}}^{k+1}$ $\displaystyle=$
$\displaystyle\mathbf{F}_{r}(\hat{\mathbf{r}}^{k},\mathbf{u}^{k})$ (5)

We will consider the effects of linearizing around a fixed point when the
dynamics are written in terms of ${\mathbf{F}}_{x}$ versus when they are
written in terms of ${\mathbf{F}}_{r}$.

To begin, consider a point specified by

$\displaystyle\hat{\mathbf{x}}_{0}$ $\displaystyle\mathbf{u}_{0}$
$\displaystyle=$ $\displaystyle\mathbf{0}$ $\displaystyle\hat{\mathbf{r}}_{0}$
$\displaystyle=$ $\displaystyle{\rm g}(\hat{\mathbf{x}}_{0})$ (6)

which we choose to be a fixed point of the dynamics (1), i.e., it is such that

$\displaystyle\hat{\mathbf{x}}_{0}$ $\displaystyle=$
$\displaystyle\mathbf{F}_{x}(\hat{\mathbf{x}}_{0},\mathbf{u}_{0}).$ (7)

Linearizing ${\mathbf{F}}_{x}$ around that fixed point, we obtain

$\displaystyle\hat{\mathbf{x}}^{k+1}={\mathbf{F}}_{x}(\hat{\mathbf{x}}^{k},\mathbf{u}^{k})\approx\;{\mathbf{F}}_{x}(\hat{\mathbf{x}}_{0})\;+\;\frac{\partial{\mathbf{F}}_{x}}{\partial\hat{\mathbf{x}}}(\hat{\mathbf{x}}^{k}-\hat{\mathbf{x}}_{0})\;+\;\frac{\partial{\mathbf{F}}_{x}}{\partial\mathbf{u}}\mathbf{u}^{k}$
(8)

Inserting Eq. 7, we obtain

$\displaystyle\hat{\mathbf{x}}^{k+1}-\hat{\mathbf{x}}_{0}=\frac{\partial{\mathbf{F}}_{x}}{\partial\hat{\mathbf{x}}}(\hat{\mathbf{x}}^{k}-\hat{\mathbf{x}}_{0})\;+\;\frac{\partial{\mathbf{F}}_{x}}{\partial\mathbf{u}}\mathbf{u}^{k}.$
(9)

Changing variables to

$\displaystyle\mathbf{x}^{k}=\hat{\mathbf{x}}^{k}-\hat{\mathbf{x}}_{0},$ (10)

we arrive at

$\displaystyle\mathbf{x}^{k+1}=\frac{\partial{\mathbf{F}}_{x}}{\partial\hat{\mathbf{x}}}\mathbf{x}^{k}\;+\;\frac{\partial{\mathbf{F}}_{x}}{\partial\mathbf{u}}\mathbf{u}^{k}.$
(11)

In index notation, the two matrices involved in (11) are

$\displaystyle\left[\frac{\partial{\mathbf{F}}_{x}}{\partial\hat{\mathbf{x}}}\right]_{ij}$
$\displaystyle=$ $\displaystyle W_{ij}{\rm g}^{\prime}(\hat{x}_{0j})$ (12)
$\displaystyle\left[\frac{\partial{\mathbf{F}}_{x}}{\partial\mathbf{u}}\right]_{ij}$
$\displaystyle=$ $\displaystyle\delta_{ij}$ (13)

Let us define a diagonal matrix $D$, i.e., with zeros on all the non-
diagonals, and entries along the diagonal that are each a function of the
$j^{\rm th}$ element of $\hat{\mathbf{x}}_{0}$ :

$\displaystyle D_{jj}={\rm g}^{\prime}(\hat{x}_{0j}).$ (14)

Since its elements are the gains of ${\rm g}$ for each element of
$\hat{x}_{0}$, we will call this matrix the diagonal gain matrix $D$. Then, in
matrix notation, we can use $D$ to rewrite the linearized dynamics (11) as

$\displaystyle\mathbf{x}^{k+1}$ $\displaystyle=$ $\displaystyle
WD\,\mathbf{x}^{k}+\mathbf{u}^{k}$ (15)

The second linearization is obtained by differentiating equation (5) with
respect to $\hat{\mathbf{r}}$ and changing variables to

$\displaystyle\mathbf{r}^{k}=\hat{\mathbf{r}}^{k}-\hat{\mathbf{r}}_{0}.$ (16)

This requires the derivatives

$\displaystyle\left[\frac{\partial{\mathbf{F}}_{r}}{\partial\hat{\mathbf{r}}}\right]_{ij}$
$\displaystyle=$ $\displaystyle{\rm g}^{\prime}(\hat{x}_{0i})W_{ij}$ (17)
$\displaystyle\left[\frac{\partial{\mathbf{F}}_{r}}{\partial\mathbf{u}}\right]_{ij}$
$\displaystyle=$ $\displaystyle{\rm g}^{\prime}(\hat{x}_{0i})\delta_{ij}$ (18)

which we rewrite in matrix notation as

$\displaystyle\mathbf{r}^{k+1}$ $\displaystyle=$ $\displaystyle
DW\mathbf{r}^{k}+D\mathbf{u}^{k}$ (19)

The two linear dynamical systems (15) and (19) might appear at first sight to
be quite disparate. $DW$ represents a scaling of the rows of $W$ by the
diagonal elements of $D$, while $WD$ represents a scaling of the columns of
$W$ by the diagonal elements of $D$. The results of the two scalings could be
quite different, suggesting that different conclusions might be drawn from
analyzing $WD$ versus analyzing $DW$, even though they are both linearizations
of the same dynamics around the same fixed point.

But this is not the case. The two equations describe dynamics in terms of
different variables, $\mathbf{x}$ and $\mathbf{r}$ , but are in fact
intimately related. If we express the dynamics in terms of the same variable,
the two different linearizations lead to identical trajectories.

To relate the variables $\mathbf{x}$ and $\mathbf{r}$,consider a linearization
of ${\rm g}()$ around $\hat{\mathbf{x}}_{0}$ so that

$\displaystyle\hat{\mathbf{r}}$ $\displaystyle\approx$ $\displaystyle{\rm
g}(\hat{\mathbf{x}}_{0})+{\rm
g}^{\prime}(\hat{\mathbf{x}}_{0})(\hat{\mathbf{x}}-\hat{\mathbf{x}}_{0})$ (20)

Then, given that $\hat{\mathbf{r}}_{0}={\rm g}(\hat{\mathbf{x}}_{0})$, and
using the variable changes (10) and (16), we can find the map relating
$\mathbf{r}~{}$and $\mathbf{x}$:

$\displaystyle\mathbf{r}$ $\displaystyle\approx$ $\displaystyle{\rm
g}^{\prime}(\hat{\mathbf{x}}_{0})\,\mathbf{x}\;=\;D\mathbf{x}$ (21)

This makes it plain that the two equations (15) and (19) are equivalent,
related through the map in (21). That is, we can take equation (15), multiply
it on the left by the gain matrix $D$, and obtain equation (19):

$\displaystyle\mathbf{x}^{k+1}$ $\displaystyle=$ $\displaystyle
WD\,\mathbf{x}^{k}+\mathbf{u}^{k}$ $\displaystyle D\mathbf{x}^{k+1}$
$\displaystyle=$ $\displaystyle DWD\,\mathbf{x}^{k}+D\mathbf{u}^{k}$
$\displaystyle\mathbf{r}^{k+1}$ $\displaystyle=$ $\displaystyle
DW\,\mathbf{r}^{k}+D\mathbf{u}^{k}$ (22)

This means that if we take a trajectory of points $\mathbf{x}^{k}$ produced by
the linearization of ${\mathbf{F}}_{x}$ in (15), and map each $\mathbf{x}^{k}$
onto its corresponding $\mathbf{r}^{k}$ using (21), we will get exactly the
set of $\mathbf{r}^{k}$ that the linearization of ${\mathbf{F}}_{r}$ in (19)
would have produced. The two linearizations describe the same trajectories and
thus the same dynamics, albeit mapped onto each other through $D$, as in (21).

##  3 Left and right eigenvectors of the dynamics matrices

As we have described, (15) and (19) are two views of the same dynamical
trajectories. But they have different linearized dynamics matrices,
respectively $WD$ and $DW$, which in general have different
eigendecompositions. The right and left eigenvectors of linearized dynamics
matrices determine many features of the dynamics, but as shown above, the
dynamics are independent of the chosen linearization. This suggests that the
eigendecompositions of the two matrices should be closely related, and here we
show that indeed they are.

Let $W$ be a square matrix and $D$ be a diagonal matrix of the same size as
$W$.

Let $\mathbf{s}_{r}^{T}$ be a left eigenvector of matrix $DW$, with
corresponding eigenvalue $\lambda$. In other words,

$\displaystyle\mathbf{s}_{r}^{T}DW=\lambda\mathbf{s}_{r}^{T}$ (23)

Multiplying on the right by $D$ we obtain

$\displaystyle\mathbf{s}_{r}^{T}DWD=\lambda\mathbf{s}_{r}^{T}D$ (24)

which means that the vector $\mathbf{s}_{r}^{T}D$ is a left eigenvector of the
matrix $WD$, with eigenvalue $\lambda$.

In other words,

If $\mathbf{s}_{r}^{T}$ is a left eigenvector of $DW$ with eigenvalue
$\lambda$, then $\displaystyle\mathbf{s}_{x}^{T}=\mathbf{s}_{r}^{T}D$ (25) is
a corresponding left eigenvector of $WD$, also with eigenvalue $\lambda$.

Similarly, let $\boldsymbol{\rho}_{x}$ be a right eigenvector of $WD$, with
eigenvalue $\lambda$. That is,

$\displaystyle WD\boldsymbol{\rho}_{x}=\lambda\boldsymbol{\rho}_{x}$ (26)

Multiplying on the left by $D$ we obtain

$\displaystyle DWD\boldsymbol{\rho}_{x}=\lambda D\boldsymbol{\rho}_{x}$ (27)

which means that the vector $D\boldsymbol{\rho}_{x}$ is a right eigenvector of
the matrix $DW$, with eigenvalue $\lambda$.

In other words,

If $\boldsymbol{\rho}_{r}$ is a right eigenvector of $DW$ with eigenvalue
$\lambda$, then
$\displaystyle\boldsymbol{\rho}_{x}=D^{-1}\boldsymbol{\rho}_{r}$ (28) is a
corresponding right eigenvector of $WD$, also with eigenvalue $\lambda$.

These relationships imply that the dot product between left and right
eigenvectors is preserved:

$\displaystyle\mathbf{s}_{x}^{T}\cdot\boldsymbol{\rho}_{x}$ $\displaystyle=$
$\displaystyle=$ $\displaystyle\mathbf{s}_{r}^{T}D\cdot
D^{-1}\boldsymbol{\rho}_{r}$ $\displaystyle=$
$\displaystyle\mathbf{s}_{r}^{T}\cdot\boldsymbol{\rho}_{r}$

Note that, except for the case when W is rank 1, the relationship between the
eigenvectors of $W$ and the eigenvectors of $WD$ or $DW$ is in general non-
trivial.

##  4 Linearizations and context-dependence of input vectors

Any given RNN will be defined by its weight matrix $W$, and trajectories on it
will be induced by inputs $\mathbf{u}^{k}$, where $k$ indexes timepoints. We
define $\mathbf{u}^{k}=0$ for $k<0$, and consider the case where the network
is simulated over multiple different “runs” or “trials”, each of which begins
at a timepoint $k<<0$, and evolves to some timepoint $k>0$. Let us now
consider a situation in which there are additional inputs to the units of the
network, constant in time during each run, but potentially different across
different runs. That is, during each run $R$, the dynamical equations are

$\displaystyle\hat{\mathbf{x}}^{k+1}$ $\displaystyle=$ $\displaystyle
W\hat{\mathbf{r}}^{k}+\mathbf{u}^{k}+\mathbf{c}_{R}$
$\displaystyle\hat{\mathbf{r}}^{k+1}$ $\displaystyle=$ $\displaystyle{\rm
g}(\hat{\mathbf{x}}^{k+1})$ (29)

The inputs $\mathbf{c}_{R}$ define what we will call context R.

Let us further suppose that before timepoint $k=0$ of each run in context $R$,
and before any inputs $\mathbf{u}~{}$are non-zero in that run, the network has
settled into a fixed-point determined by $\mathbf{c}_{R}$. This fixed-point
will be such that

$\displaystyle\hat{\mathbf{x}}_{0}^{R}$ $\displaystyle=$ $\displaystyle W{\rm
g}(\hat{\mathbf{x}}_{0}(\mathbf{c}_{R}))+\mathbf{c}_{R}$ (30)

and will have a corresponding gain matrix $D_{R}$ whose diagonal entries are
the elements of ${\rm g}^{\prime}(\hat{\mathbf{x}}_{0}(\mathbf{c}_{R})).$

Following (15) and (19), let us define the linearization of the network for
context $R$ to be the linear network with dynamics

$\displaystyle\mathbf{x}^{k+1}$ $\displaystyle=$ $\displaystyle
WD_{R}\,\mathbf{x}^{k}+\mathbf{u}^{k}$ $\displaystyle{\rm and}$
$\displaystyle\mathbf{r}^{k+1}$ $\displaystyle=$ $\displaystyle
D_{R}W\,\mathbf{r}^{k}+D_{R}\mathbf{u}^{k}$ (32)

Differences between two contexts $A$ and $B$ in how a network behaves will
then correspond to different instantiations of the network, one determined by
the gain matrix $D_{A}$, the other by the gain matrix $D_{B}$.

Notice that context-dependent modulation of the linearized input
$\mathbf{u}~{}$is observable only in the activity space linearization (32)
(where the linearized input is $D_{R}\mathbf{u},~{}$and thus depends on the
gain matrix $D_{R}$). In the activation space linearization (4), the
linearized input is always $\mathbf{u},~{}$independent of $D_{R}$.

Context-dependent input modulation of recurrent networks with a fixed input
vector $\mathbf{u}~{}$is studied, for example, in [1], who utilize activity
space linearization (32) for this purpose: the linearized inputs
$D_{R}\mathbf{u}~{}$depend on context through $D_{R}$. In contrast, [2] used
activation space linearization (4) when studying context dependence of RNN
dynamics with fixed input vectors, and therefore did not study context-
dependent input modulation.

##  5 Conclusion

In a recurrent neural network, the linear dynamics that result from
linearization in activation space, and those that result from linearization in
activity space, are different. Nevertheless, the two linear dynamics describe
the same underlying trajectories, albeit mapped onto each other through a
scaling given by the gain of each of the network’s units.

Despite this close relationship between the two linearizations, the two are
not interchangeable. In particular, context-dependent modulations of external
inputs that follow from context-dependent changes in unit gains are directly
observable as input modulations in the activity space linearization, but not
in the activation space linearization.

## References

  * [1] Niru Maheswaranathan and David Sussillo  “How recurrent networks implement contextual processing in sentiment analysis”, 2020  arXiv:2004.08013 [cs.CL]
  * [2] Valerio Mante, David Sussillo, Krishna V Shenoy and William T Newsome  “Context-dependent computation by recurrent dynamics in prefrontal cortex”  In _Nature_ 503.7474, 2013, pp. 78–84
  * [3] Kenneth D Miller and Francesco Fumarola  “Mathematical equivalence of two common forms of firing rate models of neural networks”  In _Neural Comput._ 24.1, 2012, pp. 25–31
  * [4] Marino Pagan et al.  “A new theoretical framework jointly explains behavioral and neural variability across subjects performing flexible decision-making”  In _bioRxiv_ , 2022, pp. 2022.11.28.518207