# Memoria: Hebbian Memory Architecture
for Human-Like Sequential Processing

Sangjun Park & JinYeong Bak
Department of Computer Science and Engineering
Sungkyunkwan University, Seoul, South Korea
{psj8252<EMAIL_ADDRESS>

###### Abstract

Transformers have demonstrated their success in various domains and tasks.
However, Transformers struggle with long input sequences due to their limited
capacity. While one solution is to increase input length, endlessly stretching
the length is unrealistic. Furthermore, humans selectively remember and use
only relevant information from inputs, unlike Transformers which process all
raw data from start to end. We introduce Memoria, a general memory network
that applies Hebbian theory which is a major theory explaining human memory
formulation to enhance long-term dependencies in neural networks. Memoria
stores and retrieves information called engram at multiple memory levels of
working memory, short-term memory, and long-term memory, using connection
weights that change according to Hebb’s rule. Through experiments with popular
Transformer-based models like BERT and GPT, we present that Memoria
significantly improves the ability to consider long-term dependencies in
various tasks. Results show that Memoria outperformed existing methodologies
in sorting and language modeling, and long text classification.

##  1 Introduction

Humans possess an incredible ability to retain relevant details over extended
periods. Humans extract major information from the flood of data, classify
this information into long-term and short-term memory based on importance and
utility, retrieve helpful information when needed, and gradually forget
useless and unemployed information (Nairne & Pandeirada, 2008; Atkinson &
Shiffrin, 1968a; Craik & Lockhart, 1972; Atkinson & Shiffrin, 1968b; Waugh &
Norman, 1965; Brown, 1958; Underwood & Postman, 1960). This memorization is a
fundamental skill for humans that is essential for learning and completing
various tasks. Even when reading a book, we can form a condensed understanding
of prior occurrences, such as the characters and plot progresses, despite
passing many pages or chapters. Memorization is also associated with problem-
solving or language skills since it permits individuals to apply previously
learned information to solve novel issues.

Hebbian theory (Hebb, 1949) of human memorization is a prominent theory that
explains how the brain forms connections between neurons to store and retrieve
information. This theory proposes that when two neurons are repeatedly
activated at the same time, the connections between them become strengthened.
This phenomenon is commonly referred to as the “fire together, wire together”
principle. The more frequently the neurons fire together, the stronger the
connection becomes, which results in more robust and stable memory formation.

Memorization is critical for neural networks to perform well on a wide range
of tasks, such as language modeling and long-document classification. To solve
these problems successfully, models must remember long-term dependencies in
the data, such as the context of a sentence or the relationships between
pronouns in text. Transformer (Vaswani et al., 2017) has found extensive use
in diverse domains and tasks (Devlin et al., 2019; Radford et al., 2018; Brown
et al., 2020; Lewis et al., 2020). Self-attention mechanism, which is the key
component of Transformer, facilitates the fusion of information from all
sequence elements into the comprehensive contextual representation of whole
sequence.

However, the downside of Transformer model is that, unlike recurrent neural
networks (Rumelhart & McClelland, 1987; Hochreiter & Schmidhuber, 1997; Chung
et al., 2014), it requires the entire sequential data at the same time. Most
publicly available Transformer-based models are pre-trained with a limited
context length, and dealing with long length data is generally difficult due
to the time and space complexity of $O(L^{2})$ where $L$ is the input length
(Vaswani et al., 2017). Moreover, it is significantly different from the
mechanisms of human memory.

We propose a Hebbian memory architecture, Memoria, which grants memory
management capabilities to deep neural network. Memoria is a separate module
that can be used with various sequence processing models. It stores the
information processed by the neural network as three level memories according
to the Multi-Store model (Atkinson & Shiffrin, 1968b): working memory, short-
term memory, and long-term memory, and retrieves it as necessary. This process
is quite similar to the way of humans. Each piece of information, called an
engram, is connected to one another, and the connection weight changes
according to Hebb’s rule (Hebb, 1949). We evaluated Memoria with the most
widely used Transformer-based encoder and decoder models, such as BERT and GPT
(Devlin et al., 2019; Brown et al., 2020). As a result, we confirmed that
Memoria enhances the ability to consider long-term dependencies in sorting,
language modeling, and text classification tasks. The implementation of
Memoria and experiment code are available on Github.111The code is available
at https://github.com/cosmoquester/memoria.

Contributions

1\. We designed Memoria, an independent memory module that satisfies Hebbian
learning rule which is the core theory of human memory construction, by
applying various theories related to memorization and forgetting.

2\. We developed effective strategies to integrate Memoria with diverse
Transformer-based models, including BERT and GPT while taking into account the
properties of their architectures.

3\. We show that Memoria outperforms other existing methodologies in language
modeling, sorting, and text classification for long sequences through
extensive experiments.

##  2 Related Work

Memory-augmented neural networks have a rich history in the field of machine
learning. Recurrent Neural Networks (RNNs) (Rumelhart & McClelland, 1987;
Hochreiter & Schmidhuber, 1997; Chung et al., 2014) were introduced as a
neural network architecture capable of processing sequential data with memory.
Neural Turing Machines (NTMs) (Graves et al., 2014) have a storage system for
vector representations that can be accessed using an attention mechanism. NTMs
were further developed into Differentiable Neural Computer (DNC) (Graves et
al., 2016) and Sparse DNC (Rae et al., 2016). Transformer model (Vaswani et
al., 2017) has gained popularity for its ability to achieve state-of-the-art
results on various domains, especially natural language processing. However,
Transformer suffers from a limitation in processing long sequences due to its
quadratic time and space complexity (Vaswani et al., 2017).

To address this limitation, two major approaches have been proposed. Firstly,
the sparse attention approach uses various techniques such as local attention,
reversible layers, and hashing to reduce the computational cost of attention
while maintaining the ability to model long-range dependencies. The models
like Longformer (Beltagy et al., 2020), BigBird (Zaheer et al., 2020), and
Reformer (Kitaev et al., 2020) adopted the approach. However, this approach
still has the limitation of processing only a restricted size of consecutive
inputs, even though it has capability to handle longer lengths with the same
amount of resources. The second approach is segmentation and recurrent
processing, which includes models such as Transformer-XL (Dai et al., 2019),
Compressive Transformer (Rae et al., 2020), $\infty$-Transformer (Martins et
al., 2021), Memory Transformer (Burtsev & Sapunov, 2020). Recurrent Memory
Transformer (Bulatov et al., 2022) focused on using the small number of memory
tokens for efficiency, and Memorizing Transformers (Wu et al., 2022) attmepted
to use $k$-NN cache as memory. These models split inputs into multiple
segments and incorporate them to better maintain long-term dependencies in
sequential data. However, these methods have a drawback in that, no matter how
significant the past information may be, it inevitably becomes diluted
gradually. While Memoria follows the second approach, Memoria preserves
crucial past information ensuring that the information remains unchanged, just
as it was initially accessed if the information is important enough.

Hebbian theory (Hebb, 1949) is a principle in neuroscience that states, “fire
together, wire together”, meaning that if two neurons are activated at the
same time, the connection between them will strengthen. According to Hebb, an
engram is a representation of a memory in the brain, consisting of a group of
neurons and their connections that are activated together during the encoding
of a memory. In recent years, there has been growing interest in applying
Hebbian learning to deep learning (Kuriscak et al., 2015; Movellan, 1991;
Journé et al., 2023). These works have shown promising results and highlight
the potential for Hebbian learning to improve the unsupervised learning of
deep neural networks. We adopted the concept of Hebbian engrams for Memoria.

Hebbian learning rule (Caporale & Dan, 2008; Gerstner & Kistler, 2002; Song et
al., 2000), a specific mathematical formulation of a fundamental principle in
neuroscience, describes how synapses between neurons can be modified based on
their activity and enables the network to learn and memorize information over
the time by changing their connections in response to input patterns. Gerstner
& Kistler (2002) suggested six important aspects, which are locality,
cooperativity, synaptic depression, boundedness, competition, and long-term
stability, for the formulation of a useful plasticity model. We manifest that
Memoria satisfies all the six of attributes. (See Appendix A for details.)

Memoria categorize memories into three levels according to the Multi-Store
model (Atkinson & Shiffrin, 1968b), using the term working memory instead of
sensory memory. Furthermore, to account for forgetting in short-term memory,
we applied the displacement mechanism (Waugh & Norman, 1965), which replaces
old information with new information when the short-term memory is full. For
forgetting in both short-term and long-term memory, we incorporated the
concept of trace decay theory (Brown, 1958; Peterson & Peterson, 1959), which
suggested that memories that are not actively recalled gradually fade away.

##  3 Memoria

There are three stages of utilizing Memoria. First stage is remind stage, in
which it uses working memory to remind the engrams from short-term memory and
long-term memory. Second stage is exploit stage, where a model uses the
reminded engrams to solve the task. Last stage is memorize & forget. In this
stage, all reminded engrams get more lifespan depending on the usefulness of
each engram and all the engrams will lose their lifespan by one. We provided
the visualizations of changes of connection in Appendix D to help understand
these processes.

###  3.1 Component

Figure 1: Working Memory retains the most recent information, while Short-term
Memory also holds a fixed number of recent engrams but its size can be
adjusted. The number of engrams in Long-term Memory is not predetermined. The
arrows in the diagram represent the connections between each engram.

Memoria has three types of memory, working memory (WM), short-term memory
(STM), and long-term memory (LTM). Engrams, which are the smallest unit of
memory information, constitute each memory. These engrams have their own
lifespan and are eliminated when their lifespan reaches zero. Figure 1 shows
the structure of the three types of memory.

#### Memory Encoder

A memory encoder $f_{e}$ is needed to transform the input $X_{t}$ at a
particular time step. The design of the memory encoder can vary, and $X_{t}$
could be defined as the input for a task-solving model, model hidden states of
previous time step, or other values. The output of $f_{e}$ is a set of engrams
$M=\\{e_{1},e_{2},\dots,e_{N}\\}$.

#### Working Memory

Working memory $M_{wm}$ corresponds to human sensory memory. It represents the
most recent memory and serves as a reference to retrieve associated engrams
from short-term and long-term memory. Working memory uses a queue structure
with a fixed size, which is equivalent to the memory length of a single time
step. After every time step, the working memory is updated.

#### Short-term Memory

Short-term memory $M_{stm}$, like human short-term memory, holds recent
information. Engrams that were previously in working memory are transferred to
short-term memory after one time step. Similar to working memory, short-term
memory employs a queue data structure with a fixed size, which can be defined
as a parameter.

#### Long-term Memory

Long-term memory $M_{ltm}$ is equivalent to human long-term memory and has the
capacity to store an indefinite number of engrams. Engrams that were dequeued
from short-term memory are transferred to long-term memory.

#### Memory Graph

Engrams in any memory can be linked together, represented as a directed
weighted graph data structure, where each vertex corresponds to an engram. A
directed edge weight $E_{i\rightarrow j}$ denotes the empirical conditional
probability that the engram $e_{j}$ will be reminded when the engram $e_{i}$
is reminded, with $M^{rem}$ representing the set of all reminded engrams. This
empirical probability can be calculated by dividing the number of times
$e_{i}$ and $e_{j}$ were reminded together by the number of times $e_{i}$ was
reminded. $Count_{i,j}$ represents the number of times $e_{i}$ and $e_{j}$
were reminded together. The edge is utilized to search for engrams in the
long-term memory and its weight is adjusted based on the “fire together, wire
together” principle.

$\displaystyle E_{i\rightarrow j}$ $\displaystyle=P(e_{j}\in M^{rem}\mid
e_{i}\in M^{rem})$ $\displaystyle=\frac{Count_{i,j}}{Count_{i,i}}$

###  3.2 Remind

Remind is the process to remind engrams from short-term and long-term memory.
Figure 2 shows entire reminding process.

Figure 2: Remind process.

  1. 1.

Using the encoder function $f_{e}$ with input $X$, create engrams $M_{wm}$ and
put into the working memory. All the engrams in the working memory will have
the same initial lifespan.

$M_{wm}=f_{e}(X)=\\{e_{wm,1},e_{wm,2},\dots,e_{wm,N_{wm}}\\}$

  2. 2.

By utilizing the correlation function $f_{c}$, calculate the correlation
weight $C_{stm}$ for each $e_{stm,i}$ within the short-term memory $M_{stm}$
by averaging all the correlation weights for the engram. The distance function
$f_{d}$ used is L2 distance. Here, $i$ represents the index of $M_{stm}$ and
$j$ represents the index of $M_{wm}$.

$\displaystyle f_{c}(e_{i},e_{j})$
$\displaystyle=\mathrm{exp}(-(f_{d}(e_{i},e_{j}))^{2})$ $\displaystyle
C_{stm,i}$
$\displaystyle=\frac{1}{N_{wm}}\sum_{j=1}^{N_{wm}}f_{c}(e_{stm,i},e_{wm,j})$

  3. 3.

Select only the top $N_{stm}^{rem}$ number of engrams with $C_{stm}$ values to
remind. Denote the selected engrams as $M_{stm}^{rem}$.

  4. 4.

For each $e_{i}\in M_{stm}^{rem}$, select an engram in $M_{ltm}$ having
highest edge weight from $e_{i}$. Denote the selected engrams as
$M_{ltm}^{init}$.

$\displaystyle M_{ltm}^{init}=\operatorname*{arg\,max}_{e_{j}\in
M_{ltm}}E_{i\rightarrow j},\ \textrm{where }e_{i}\in M_{stm}^{rem}$

  5. 5.

Using the engrams $M_{ltm}^{init}$ as a starting point, traverse the $M_{ltm}$
graph using the depth-first search (DFS) algorithm with a search depth of
$N_{depth}$. The exploration direction should be based on the edge weight,
toward the highest edge weight. Gather all the unique engrams that were
encountered during the search, including $M_{ltm}^{init}$, and refer to them
as $M_{ltm}^{found}$.

$\displaystyle M_{ltm}^{0}=M_{ltm}^{init}$ $\displaystyle
M_{ltm}^{k}=\operatorname*{arg\,max}_{e_{j}\in M_{ltm}}E_{i\rightarrow
j},\textrm{ where }e_{i}\in M_{ltm}^{k-1},\ e_{j}\notin M_{ltm}^{found,k-1}$
$\displaystyle M_{ltm}^{found,k}=\bigcup\limits_{l=0}^{k}M_{ltm}^{l}$
$\displaystyle M_{ltm}^{found}=M_{ltm}^{found,N_{depth}}$

  6. 6.

Calculate correlation weight $C_{ltm}$ from $M_{wm}$ for $M_{ltm}^{found}$ and
select top $N_{ltm}^{rem}$ number of engrams like STM. Denote the engrams as
$M_{ltm}^{rem}$.

  7. 7.

Use $M_{wm},M_{stm}^{rem},M_{ltm}^{rem}$ as activated memory.

$\displaystyle M^{rem}$ $\displaystyle=M_{stm}^{rem}\cup M_{ltm}^{rem}$
$\displaystyle M^{act}$ $\displaystyle=M_{wm}\cup M^{rem}$

Figure 3: All the engrams in working memory and reminded engrams are connected
more. The reminded engrams gain lifespan depending on the contribution. End-
of-life engrams are removed like $e_{ltm,7}$. The engrams in the gray area
refer to activated engrams $M^{act}$.

###  3.3 Exploit

Exploit all the engrams reminded to aid in solving the task and evaluate each
engram’s contribution towards the solving. A cross-attention mechanism is
applied to use information of the engrams. After self-attention layer, the
working memory engrams are attended to first, followed by the short-term and
long-term memory engrams, using the exactly same cross-attention layer. The
average attention weight $w_{i}$ for each engram $e_{i}$ is regarded as its
contribution towards the solution.

###  3.4 Memorize & Forget

There are two important principles for memorizing. First, useful engrams
should be long-lived. Second, related engrams should be strongly connected
together. These principles are applied in memorize stage as follows. Figure 3
shows overall process in this stage.

  1. 1.

Increase $Count_{i,j}$ by one for all engrams in $M^{act}$, which is the
number of times $e_{i}$ and $e_{j}$ reminded together.

$\displaystyle\mathcal{N}=\\{1,2,\dots,|M^{act}|\\}$ $\displaystyle
Count_{i,j}:=Count_{i,j}+1,\forall i,j\in\mathcal{N}$

  2. 2.

Increase lifespan of reminded engrams by the increment ${Inc}_{i}$ for the
engram $e_{i}$. ${Inc}_{i}$ is calculated as follows where $\alpha$ is
hyperparameter meaning lifepspan extend scale. If $\alpha$ is 1.0, each engram
$e\in M^{rem}$ gets lifespan 1.0 on average.

$\displaystyle{Inc}_{i}=\frac{w_{i}}{\sum_{k=1}^{|M^{rem}|}w_{k}}\times|M^{rem}|\times\alpha$

  3. 3.

Decrease lifespan of all engrams by 1.0.

  4. 4.

Remove engrams having a lifespan of 0 or less.

  5. 5.

Move $e_{wm}$ into STM.

  6. 6.

Move oldest engrams from STM by the number exceeding capacity into LTM.

##  4 Experiments

We experimented how well Memoria maintains long-term connections in various
tasks using Transformer (Vaswani et al., 2017) architecture. We integrated
Memoria with Transformer by appending encoder-decoder attention over memory
engrams, but the method to create engrams is a little different depending on
the architecture. We provide figures with descriptions representing how to
apply Memoria to Transformer in Appendix C. The first task is sorting. Martins
et al. (2021) evaluated the model’s ability to remember long-term information
about the occurrence of numbers by generating a sorted sequence of numbers
based on their frequency of occurrence. In the second group of experiments, we
focused on language modeling task for token-level on WikiText-103 (Merity et
al., 2017) and PG-19 (Rae et al., 2020), and character-level on enwik8
(Mahoney, 2006). For the Wikitext-103 dataset, since the word-level dataset
contains <unk> in the texts, the raw dataset was used. Similar to Martins et
al. (2021), only the first 2,000 books of the training dataset were used for
PG-19. We compared Memoria with other models such as Transformer (Brown et
al., 2020), Transformer-XL (Dai et al., 2019), Compressive Transformer (Rae et
al., 2020), and $\infty$-former (Martins et al., 2021). Lastly, we conducted a
classification task on the long document classification dataset, Hyperpartisan
(Kiesel et al., 2019).

###  4.1 Sorting

As Memoria is an independent module that enhances long-term dependencies, in
order to apply Memoria to Transformer, we needed to define the memory encoder
$f_{e}$ and a method that utilizes the reminded engram data. We used the
attention-based abstractor as $f_{e}$ and the last hidden state of the
previous time step of the model as $X_{t}$. The hidden states $h_{t-1}$ of
previous time step used as $X_{t}$. The three values of $Q$, $W_{k}$, and
$W_{v}$ are trainable parameters. FFN is feed-forward network as same in
Transformer (Vaswani et al., 2017). The number of working memory engrams
$N_{wm}$ is determined by the number of query $Q$, so the number of query is
hyperparameter.

$\displaystyle X_{t}$ $\displaystyle=h_{t-1}$ $\displaystyle f_{e}(X_{t})$
$\displaystyle=\textrm{Abstract}(X_{t})$
$\displaystyle=\textrm{FFN}(\textrm{Attention}(Q,W_{k}X,W_{v}X))$
$\displaystyle=\textrm{FFN}(\textrm{Attention}(Q,W_{k}h_{t-1},W_{v}h_{t-1}))$
$\displaystyle=\textrm{FFN}(\textrm{softmax}(QW_{k}h_{t-1})W_{v}h_{t-1})$
$\displaystyle=M_{wm}$

This task is about taking a sequence of numbers and outputting the numbers in
descending order of their frequency of occurrence (Martins et al., 2021). The
vocabulary consists of 20 number tokens, and we experimented with sequences of
various lengths ranging from 1K to 32K,222We used the script of
$\infty$-former at https://github.com/deep-spin/infinite-
former/blob/main/sorting/generate_data.py to generate dataset. with segment
lengths of 256, 512, and 1024. We compared the Transformer-XL, Compressive
Transformer, $\infty$-former, and Memoria Transformer.

Figure 4: Results of sorting task. Memoria shows more robust performance than
other baselines as the input sequence length increases. The entire raw scores
are specified in Table 4.

Figure 4 demonstrates the performance in the sorting task as sequence length
increases for each segment length. The memory length was set to the same value
as the segment length. Generally, as the sequence length increased, the
performance tended to decrease because longer context information needs to be
maintained. Compared to the other two models, Memoria exhibited the least
performance degradation as the sequence length increased, showcasing its
ability to maintain long-term memory for preserving extended context. (See
Section B.1 for details on hyperparameters.)

###  4.2 Language Modeling

In language modeling as well, Memoria was applied to the Transformer
architecture using the same approach as in the sorting task. We trained
various models of Transformer, Transformer-XL, Compressive Transformer,
$\infty$-former, and Memoria Transformer from scratch. As publicly available
pre-trained models were trained on different datasets and parameters, we
conducted this experiment by training the model from scratch. We experimented
pre-trained language models with Memoria and showed the results in Section
B.2. We utilized GPT-2 architecture for the implementation of Transformer. We
chose hyperparameters of 12 layers and 768 dimensions. The pre-trained GPT-2
tokenizer was used for all token-level experiments. We set the segment length
as 150 tokens for token-level experiments and 512 for character-level
experiments following the Bulatov et al. (2022). (See Section B.2 for details
on hyperparameters.)

Table 1 shows the results. All other models demonstrated improved performance
compared to Transformer. Among them, Memoria Transformer achieved the best
performance on all three datasets. This result demonstrates that Memoria has
better performance not only compared to Transformer but also to existing
competitors that models long-term dependency. Moreover, since Memoria is an
independent module, it can be used in conjunction with other techniques if
desired.

Table 1: Language Modeling Performance. Perplexity (PPL) is shown for Wikitext-103 and PG-19, while bits-per-character (BPC) is shown for Enwik8. All of them had the same memory length as the segment length, and Wikitext-103 and PG-19 use 150 while Enwik8 used 512. Memoria outperformed Transformer and other baselines that consider long-term dependency.  Model |  Wikitext-103 (PPL) |  PG-19 (PPL) |  Enwik8 (BPC)
---|---|---|---
Transformer |  26.755 |  31.631 |  1.28
Transformer-XL |  24.543 |  29.945 |  1.19
Compressive Transformer |  24.794 |  29.603 |  1.16
$\infty$-former |  24.685 |  29.154 |  1.21
Memoria Transformer |  23.471 |  29.149 |  1.16
Table 2: Perplexity with smaller segment length of 50. Memoria outperformed other baselines in the shorter context and memory setting.  Model [Memory Length] |  Wikitext-103
---|---
Transformer |  39.287
Transformer-XL [50] |  31.459
Compressive Transf. [50] |  31.644
$\infty$-former [50] |  31.790
Memoria Transformer [48] |  30.007

Table 2 presents the performance measurement in a case where the length of
each segment was decreased to 50 tokens, aiming to handle longer long-term
dependencies by increasing the number of segments. When comparing the results
in Table 1, it is evident that there is a significantly larger performance gap
between plain Transformer and the memory utilization models. Even in
situations where longer long-term dependencies need to be considered, Memoria
demonstrated the best performance.

Figure 5: Average age of engrams in LTM per step. The age of engrams in the
long-term memory being recalled gradually increased as steps pass by.

We validated whether Memoria effectively utilizes long-term memory. Figure 5
shows the average age of reminded engrams in long-term memory at each step on
test dataset. The age represents the number of steps that have passed since
the engram was created. If the model only refers to the most recent engram in
long-term memory, it would not correctly serve as a long-term memory, and the
age of reminded engrams remain constant on the graph. On the contrary, if the
model can refer to past information continuously, the past information will
gradually age more, leading to an increase in the average age of reminded
engrams over time. The graph indicates that as the step increases, the average
age also increases, demonstrating the ability of Memoria to refer to important
past information even after a significant number of time steps.

###  4.3 Classification

Utilizing the information from the current time step could lead to causal
leakage in language modeling so previous time steps were used as working
memory instead. However, with masked language models such as BERT, it is
possible to use the information from the current time step as working memory
without causing causal leakage. The memory encoder $f_{e}$ utilized the hidden
states $h_{t}^{l}$ memory representation. Here, $t$ denotes the current time
step, and $l$ represents the memory layer index. Memory is obtained from the
hidden state of the BERT layer $l$ with abstractor, then working memory
engrams and reminded engrams are utilized in the subsequent layers using
cross-attention mechanisms.

Table 3: Text classification performance on Hyperpartisan. The metrics are average macro F1-score and accuracy of five runs. We reported validation and test set results because of data distribution discrepancy. Memoria RoBERTa achieves the highest performance in the models.  Model [Input Length + Memory Length] |  Validation |  Test
---|---|---
F1 |  Acc. |  F1 |  Acc.
BERT [512] |  75.62 |  76.77 |  91.67 |  93.05
RoBERTa [512] |  82.96 |  84.06 |  95.24 |  95.38
Bigbird [4096] |  81.22 |  82.81 |  93.24 |  93.54
Longformer [4096] |  78.33 |  79.69 |  94.56 |  94.77
Memoria BERT [512 + 192] |  78.24 |  80.00 |  94.59 |  94.77
Memoria RoBERTa [512 + 192] |  86.39 |  87.19 |  96.51 |  96.62

Hyperpartisan has been a widely used news classification dataset for long
document classification task. To validate the effectiveness of Memoria in
encoder-based architectures, we applied Memoria to BERT and roBERTa and we
compared its performance on the Hyperpartisan dataset. Already pretrained
models were used to be finetuned for all the classification experiments. The
size of models was 12-layer base-sized.

Table 3 presents the classification performance of models. It is evident that
Memoria applied models show meaningful performance gain compared to the plain
models, although it is not easy to compare the performance of different base
pre-trained models directly. Memoria RoBERTa achieved the highest score of all
metrics. (See Section B.3 for details on hyperparameters.)

##  5 Conclusion and Future Work

We propose Memoria as a general memory network that follows Hebb’s rule, which
explains how humans form memories. Memoria is a seperate module that learns
the strength of the connection between different engrams according to the
utility of the connections. Memoria replicates the human process of encoding
information, selectively remembering, and forgetting. We applied Memoria to
the widely used Transformer-based neural network and demonstrate its strong
performance compared to other methodologies in tasks of sorting, language
modeling, and classification. Memoria demonstrates the potential to
revolutionize the way deep neural networks process and retain information,
opening avenues for improved performance in a wide range of tasks that rely on
long-term dependencies.

While Memoria strives to actively incorporate the structure and mechanisms of
human memory, there are still discrepancies in many aspects. We categorized
memories into three types using the Multi-store model (Atkinson & Shiffrin,
1968b), but the Levels of Processing theory (Craik & Lockhart, 1972) proposed
a more continuous structure of memory based on the depth of processing rather
than discrete categories. Additionally, we only utilized trace decay (Brown,
1958; Peterson & Peterson, 1959) and displacement (Waugh & Norman, 1965) as
mechanisms of forgetting, but Interference theory (Underwood & Postman, 1960)
suggests that interference effect between existing memories and new
information are significant forgetting mechanisms in long-term memory. Our
future research will incorporate these mechanisms enabling neural networks to
better reflect the ways human memory operates.

## Reproducibility

The structure of Memoria is described in detail in the main text Section 3. We
provided the architectural details of Memoria Transforemr and Memoria BERT in
Appendix C. Additionally, all the code used for the experiments are publicly
available.

The core module, Memoria, has been implemented as an independent Python
package, allowing future researchers to install Memoria using pip and utilize
it for their research. The model settings can be found in the main text of the
paper Section 4.2 for language modeling, Section 4.1 for sorting, and Section
4.3 for classification.

The hyperparameters used during training are all specified in Appendix B. To
ensure reproducibility, we fixed random seeds for all the experiments. The
datasets were also loaded through libraries in the code and were preprocessed,
so except for the sorting task that requires data generation, this paper and
source code will be enough to reproduce our experimental results.

## References

  * Atkinson & Shiffrin (1968a) R.C. Atkinson and R.M. Shiffrin.  Human memory: A proposed system and its control processes11this research was supported by the national aeronautics and space administration, grant no. ngr-05-020-036. the authors are indebted to w. k. estes and g. h. bower who provided many valuable suggestions and comments at various stages of the work. special credit is due j. w. brelsford who was instrumental in carrying out the research discussed in section iv and whose overall contributions are too numerous to report in detail. we should also like to thank those co-workers who carried out a number of the experiments discussed in the latter half of the paper; rather than list them here, each will be acknowledged at the appropriate place.  volume 2 of _Psychology of Learning and Motivation_ , pp. 89–195. Academic Press, 1968a.  doi: https://doi.org/10.1016/S0079-7421(08)60422-3.  URL https://www.sciencedirect.com/science/article/pii/S0079742108604223.
  * Atkinson & Shiffrin (1968b) R.C. Atkinson and R.M. Shiffrin.  Human memory: A proposed system and its control processes.  volume 2 of _Psychology of Learning and Motivation_ , pp. 89–195. Academic Press, 1968b.  doi: https://doi.org/10.1016/S0079-7421(08)60422-3.  URL https://www.sciencedirect.com/science/article/pii/S0079742108604223.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan.  Longformer: The long-document transformer.  _CoRR_ , abs/2004.05150, 2020.  URL https://arxiv.org/abs/2004.05150.
  * Brown (1958) John Brown.  Some tests of the decay theory of immediate memory.  _Quarterly Journal of Experimental Psychology_ , 10(1):12–21, 1958.  doi: 10.1080/17470215808416249.  URL https://doi.org/10.1080/17470215808416249.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * Bulatov et al. (2022) Aydar Bulatov, Yury Kuratov, and Mikhail Burtsev.  Recurrent memory transformer.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 11079–11091. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/47e288629a6996a17ce50b90a056a0e1-Paper-Conference.pdf.
  * Burtsev & Sapunov (2020) Mikhail S. Burtsev and Grigory V. Sapunov.  Memory transformer.  _CoRR_ , abs/2006.11527, 2020.  URL https://arxiv.org/abs/2006.11527.
  * Caporale & Dan (2008) Natalia Caporale and Yang Dan.  Spike timing-dependent plasticity: a hebbian learning rule.  _Annu Rev Neurosci_ , 31:25–46, 2008.
  * Chung et al. (2014) Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _CoRR_ , abs/1412.3555, 2014.  URL http://arxiv.org/abs/1412.3555.
  * Craik & Lockhart (1972) Fergus I.M. Craik and Robert S. Lockhart.  Levels of processing: A framework for memory research.  _Journal of Verbal Learning and Verbal Behavior_ , 11(6):671–684, 1972.  ISSN 0022-5371.  doi: https://doi.org/10.1016/S0022-5371(72)80001-X.  URL https://www.sciencedirect.com/science/article/pii/S002253717280001X.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  _ACL 2019 - 57th Annual Meeting of the Association for Computational Linguistics, Proceedings of the Conference_ , pp. 2978–2988, 1 2019.  doi: 10.48550/arxiv.1901.02860.  URL https://arxiv.org/abs/1901.02860v3.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pp. 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/N19-1423.  URL https://aclanthology.org/N19-1423.
  * Gerstner & Kistler (2002) Wulfram Gerstner and Werner M. Kistler.  Mathematical formulations of hebbian learning.  _Biological Cybernetics_ , 87(5):404–415, 2002\.  doi: 10.1007/s00422-002-0353-y.  URL https://doi.org/10.1007/s00422-002-0353-y.
  * Graves et al. (2014) Alex Graves, Greg Wayne, and Ivo Danihelka.  Neural turing machines.  _CoRR_ , abs/1410.5401, 2014.  URL http://arxiv.org/abs/1410.5401.
  * Graves et al. (2016) Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, Adrià Puigdomènech Badia, Karl Moritz Hermann, Yori Zwols, Georg Ostrovski, Adam Cain, Helen King, Christopher Summerfield, Phil Blunsom, Koray Kavukcuoglu, and Demis Hassabis.  Hybrid computing using a neural network with dynamic external memory.  _Nature_ , 538(7626):471–476, October 2016.  ISSN 00280836.  URL http://dx.doi.org/10.1038/nature20101.
  * Hebb (1949) D O Hebb.  _The organization of behavior_.  1949\.
  * Hochreiter & Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural Computation_ , 9:1735–1780, 11 1997.  ISSN 08997667.  doi: 10.1162/NECO.1997.9.8.1735.  URL https://www.researchgate.net/publication/13853244_Long_Short-term_Memory.
  * Journé et al. (2023) Adrien Journé, Hector Garcia Rodriguez, Qinghai Guo, and Timoleon Moraitis.  Hebbian deep learning without feedback.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=8gd4M-_Rj1.
  * Kiesel et al. (2019) Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast.  SemEval-2019 task 4: Hyperpartisan news detection.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pp. 829–839, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.  doi: 10.18653/v1/S19-2145.  URL https://aclanthology.org/S19-2145.
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  _CoRR_ , abs/2001.04451, 2020.  URL https://arxiv.org/abs/2001.04451.
  * Kuriscak et al. (2015) Eduard Kuriscak, Petr Marsalek, Julius Stroffek, and Peter G. Toth.  Biological context of hebb learning in artificial neural networks, a review.  _Neurocomputing_ , 152:27–35, 2015.  ISSN 0925-2312.  doi: https://doi.org/10.1016/j.neucom.2014.11.022.  URL https://www.sciencedirect.com/science/article/pii/S0925231214015239.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pp. 7871–7880, Online, July 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.acl-main.703.  URL https://aclanthology.org/2020.acl-main.703.
  * Mahoney (2006) Matt Mahoney.  Large text compression benchmark, 2006.  URL http://www.mattmahoney.net/dc/text.html.
  * Martins et al. (2021) Pedro Henrique Martins, Zita Marinho, and André F. T. Martins.  $\infty$-former: Infinite memory transformer.  9 2021.  doi: 10.48550/arxiv.2109.00301.  URL https://arxiv.org/abs/2109.00301v3.
  * Merity et al. (2017) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher.  Pointer sentinel mixture models.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net, 2017.  URL https://openreview.net/forum?id=Byj72udxe.
  * Movellan (1991) Javier R. Movellan.  Contrastive hebbian learning in the continuous hopfield model.  In David S. Touretzky, Jeffrey L. Elman, Terrence J. Sejnowski, and Geoffrey E. Hinton (eds.), _Connectionist Models_ , pp. 10–17. Morgan Kaufmann, 1991.  ISBN 978-1-4832-1448-1.  doi: https://doi.org/10.1016/B978-1-4832-1448-1.50007-X.  URL https://www.sciencedirect.com/science/article/pii/B978148321448150007X.
  * Nairne & Pandeirada (2008) James S. Nairne and Josefa N.S. Pandeirada.  Adaptive memory: Remembering with a stone-age brain.  _Current Directions in Psychological Science_ , 17(4):239–243, 2008.  doi: 10.1111/j.1467-8721.2008.00582.x.  URL https://doi.org/10.1111/j.1467-8721.2008.00582.x.
  * Peterson & Peterson (1959) L R Peterson and M J Peterson.  Short-term retention of individual verbal items.  _J Exp Psychol_ , 58:193–198, September 1959.
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.  URL https://www.cs.ubc.ca/~amuham01/LING530/papers/radford2018improving.pdf.
  * Rae et al. (2016) Jack Rae, Jonathan J Hunt, Ivo Danihelka, Timothy Harley, Andrew W Senior, Gregory Wayne, Alex Graves, and Timothy Lillicrap.  Scaling memory-augmented neural networks with sparse reads and writes.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 29. Curran Associates, Inc., 2016.  URL https://proceedings.neurips.cc/paper_files/paper/2016/file/3fab5890d8113d0b5a4178201dc842ad-Paper.pdf.
  * Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap.  Compressive transformers for long-range sequence modelling.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=SylKikSYDH.
  * Rumelhart & McClelland (1987) David E. Rumelhart and James L. McClelland.  _Learning Internal Representations by Error Propagation_ , pp. 318–362.  1987\.
  * Song et al. (2000) Sen Song, Kenneth D. Miller, and L. F. Abbott.  Competitive hebbian learning through spike-timing-dependent synaptic plasticity.  _Nature Neuroscience_ , 3(9):919–926, 2000.  doi: 10.1038/78829.  URL https://doi.org/10.1038/78829.
  * Underwood & Postman (1960) B J Underwood and L Postman.  Extraexperimental sources of interference in forgeting.  _Psychol Rev_ , 67:73–95, March 1960.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * Waugh & Norman (1965) Nancy C. Waugh and Donald A. Norman.  Primary memory.  _Psychological Review_ , 72(2):89–104, 1965.  doi: 10.1037/h0021797.
  * Wu et al. (2022) Yuhuai Wu, Markus Norman Rabe, DeLesley Hutchins, and Christian Szegedy.  Memorizing transformers.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=TrjbxzRcnf-.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed.  Big bird: Transformers for longer sequences.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 17283–17297. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/c8512d142a2d849725f31a9a7a361ab9-Paper.pdf.

##  Appendix A Hebbian attributes for Memoria

Gerstner & Kistler (2002) suggested six attributes of useful plasticity model
for Hebbian learning as follows. Memoria meets these attributes.

#### Locality

The learning rule for the synapse $E_{i\rightarrow j}$ connecting neuron $j$
to neuron $i$ should depend only on the activity of $j$ and $i$ and not on the
state of other neurons $k\neq i,j$.

$\displaystyle E_{i\rightarrow j}$
$\displaystyle=\frac{Count_{i,j}}{Count_{i,i}}$

By definition, Memoria meets locality because it depends on only the count of
$i,j$.

#### Cooperativity

Hebb’s formulation ‘takes part in firing it’ implies that an increase in
weight requires both the pre- and the postsynaptic neuron to be active.

$\displaystyle E_{i\rightarrow j}\propto Count_{i,j}$

Since $E_{i\rightarrow j}$ is proportional to $Count_{i,j}$ and $Count_{i,i}$
never decreases, it only increases when $e_{i}$ and $e_{j}$ fire (reminded)
together.

#### Synaptic depression

A mechanism for decreasing weights is a necessary requirement for any useful
learning rule. There are three engrams $e_{i},e_{j},e_{k}$. $E_{i\rightarrow
j}$ decreased when $e_{i}$ and $e_{k}$ fire together while $e_{j}$ does not.
The superscript $pre$ means the value before firing of $e_{i}$ and $e_{j}$ and
$post$ means the value after firing.

$\displaystyle E_{i\rightarrow j}^{pre}$
$\displaystyle=\frac{Count_{i,j}^{pre}}{Count_{i,i}^{pre}}$ $\displaystyle
Count_{i,k}^{post}$ $\displaystyle=Count_{i,k}^{pre}+1$ $\displaystyle
Count_{i,i}^{post}$ $\displaystyle=Count_{i,i}^{pre}+1$ $\displaystyle
E_{i\rightarrow j}^{post}$
$\displaystyle=\frac{Count_{i,j}^{post}}{Count_{i,i}^{post}}$
$\displaystyle=\frac{Count_{i,j}^{pre}}{Count_{i,i}^{pre}+1}$
$\displaystyle<E_{i\rightarrow j}^{pre}$

#### Boundedness

In realistic rules, weights should remain bounded in a specific range.
$E_{i\rightarrow j}$ must be between 0 and 1 because it is probability.

$\displaystyle E_{i\rightarrow j}=P(e_{j}\in M^{rem}\mid e_{i}\in M^{rem})$
$\displaystyle 0\leq P(e_{j}\in M^{rem}\mid e_{i}\in M^{rem})\leq 1$

#### Competition

The growth of some weights comes at the cost of a decrease in others. The
increase of $E_{i\rightarrow j}$ require the increase of $Count_{i,j}$ and
$Count_{i,i}$. The increase of $Count_{i,i}$ reduce all the weight
$E_{i\rightarrow k}$, for $k\neq j$.

#### Long-term stability

In adaptive systems, it is important to ensure that previously acquired
knowledge is not forgotten. In Memoria, $E_{i\rightarrow j}$ is always the
result of learning from all past examples because $Count$ is cumulative.

##  Appendix B Training details and additional results

For all experiments, the Adam optimizer and linear scheduler with warm up were
used, and the gradient clipping was set to a norm of 1.0. One or more NIVIDIA
A100 or A6000 GPUs were used for training.

###  B.1 Sorting

For all sorting experiments, a batch size of 32, a warmup rate of 0.06, a
learning rate 2e-4, and an epoch 5 were used for 80,000 train examples. A
memory length was configured to match the segment length. The experiments were
conducted on datasets with lengths ranging from 1000 to 32,000. Each example
on the datasets was divided into segments of lengths 256, 512, and 1024. For
each segment length, combinations of sequence lengths and segment lengths were
constructed by varying the number of segments, which were set to 4, 8, 16, and
32. The model configuration used was 5 layer, 4 head, embedding dimension of
512 Transformer. The compression rate is 4 and the ratio of normal memory and
compressed memory is one-to-one for Compressive Transformer.

Memoria parameters used in the experiment were as follows: an initial lifespan
of 5, a lifespan extension scale $\alpha$ of 8, a long-term memory search
depth $N_{depth}$ of 10 in all cases. Other parameters are adjusted
proportionally to the segment length. the number of working memories $N_{wm}$
set to 1/8 of the segment length, the number of reminded engrams in short-term
memory $N_{stm}^{rem}$ set to 1/4 of the segment length, the number of remind
engrams in long-term memory $N_{ltm}^{rem}$ set to 5/8 of the segment length,
and a capacity of short-term memory set to half of the segment length. The sum
of $N_{wm}$, $N_{stm}^{rem}$, and $N_{ltm}^{rem}$ is equal to the segment
length.

Table 4 shows the all score of models in sorting task. The metric is accuracy.
For the convenience of comparison, we marked the number of segments instead of
the total sequence length of each dataset. The sequence length can be obtained
by multiplying the number of segments by segment length. Memoria Transformer
proves its robustness for long-term dependency compared to the other models,
especially as the number of segments increases.

Table 4: Accuracy in sorting task. When the segments increase, Memoria outperms other baselines. Model |  Segments |  Segment Length
---|---|---
256 |  512 |  1024
Transformer-XL |  4 |  74.66 |  60.46 |  68.86
Compressive Transformer |  4 |  64.38 |  64.57 |  79.51
$\infty$-former |  4 |  84.49 |  83.75 |  84.28
Memoria Transformer |  4 |  80.42 |  80.99 |  82.27
Transformer-XL |  8 |  36.24 |  37.41 |  40.09
Compressive Transformer |  8 |  56.88 |  49.58 |  71.84
$\infty$-former |  8 |  70.21 |  75.55 |  74.34
Memoria Transformer |  8 |  70.84 |  74.47 |  74.08
Transformer-XL |  16 |  32.75 |  34.59 |  35.06
Compressive Transformer |  16 |  35.57 |  37.69 |  44.03
$\infty$-former |  16 |  53.61 |  53.61 |  47.31
Memoria Transformer |  16 |  63.99 |  64.50 |  66.58
Transformer-XL |  32 |  32.24 |  32.76 |  33.87
Compressive Transformer |  32 |  32.68 |  33.15 |  35.07
$\infty$-former |  32 |  34.36 |  36.41 |  39.71
Memoria Transformer |  32 |  50.08 |  56.48 |  63.42

###  B.2 Language Modeling

For all language modeling experiments, a batch size of 8 and a warmup rate of
0.06 were used. The model configuration used the settings of GPT-2 small by
default. The Wikitext-103 and PG-19 datasets were trained for 3 epochs, while
the Enwik8 dataset was trained for 20 epochs. GPT-2 tokenizer was used for all
datasets except Enwik8, which was trained at the character level using 204
characters. The default learning rate was 2e-4, but in cases where convergence
was challenging, 1e-4 was used. However, for experiments fine-tuning
pretrained models, a learning rate of 5e-5 was used. In the experiments
conducted on the Wikitext-103 dataset using Transformer-XL and on the PG-19
dataset using $\infty$-former, as well as the experiment with reduced segment
length to 50, both Memoria Transformer and Transformer-XL were trained with a
learning rate of 1e-4. The memory length was set to be the same or similar to
the segment length. The compression rate is 4 and the ratio of normal memory
and compressed memory is one-to-one for Compressive Transformer.

Memoria parameters were set as follows: initial lifespan of 9, lifespan extend
scale $\alpha$ of 8, and long-term memory search depth $N_{depth}$ of 10.
Furthermore, to prevent potential interference with the learning process, we
periodically reset all memory in Memoria every 500 steps during training (1500
steps for enwik8 dataset). This was done to avoid referencing memory generated
at stages where learning was insufficient, as it could impede the training
progress. For the Wikitext-103 and PG-19 datasets, the number of working
memories $N_{wm}$, the number of reminded engrams in short-term memory
$N_{stm}^{rem}$, and the number of remind engrams in long-term memory
$N_{ltm}^{rem}$ were all set to 50, and a capacity of short-term memory was
set to 400. For the Enwik8 dataset, $N_{wm}$, $N_{stm}^{rem}$ and
$N_{ltm}^{rem}$ were set to 170, and a capacity of short-term memory was set
to 1360. When training on the Wikitext-103 dataset with a reduced segment
length of 50, $N_{wm}$, $N_{stm}^{rem}$, and $N_{ltm}^{rem}$ were all set to
16, and the short-term memory capacity was set to 128.

Table 5: Finetuning performance on Wikitext-103. Model |  Wikitext-103
---|---
GPT-2 |  20.498
Memoria GPT-2 |  18.986
GPT-2 Large |  15.332
Memoria GPT-2 Large |  13.227
GPT-2 XL |  15.254
Memoria GPT-2 XL |  13.241

To verify whether Memoria can consider long-term context even when finetuning
a pretrained model, we measured performance on Wikitext-103 dataset by
finetuning Memoria GPT-2. The architecture of Memoria GPT-2 is same as Memoria
Transformer. The results are Table 5. Memoria GPT-2 showed significantly
better performance than GPT-2. This result suggests that Memoria can be
combined with various pre-trained models to increase long-term dependencies.
Furthermore, as the use of pretrained large language models (LLMs) has become
prevalent, we conducted experiments to verify whether Memoria can be applied
in conjunction with LLMs. We performed experiments using large and xl sized
models, and successfully achieved performance improvements when applying
Memoria to even larger pretrained models. This demonstrates the potential for
LLMs to benefit from considering longer contexts with the help of Memoria.

###  B.3 Classification

All hyperpartisan text classification experiments were conducted with a batch
size of 16, a learning rate of 5e-5, and a warmup rate of 0.1. The models were
trained for 20 epochs. For BERT, the experiment utilized the pretrained bert-
base-uncased model. As for Longformer, the base model was used in the
experiment.

Memoria parameters used in the experiment were as follows: an initial lifespan
of 12, a lifespan extension scale $\alpha$ of 8, a long-term memory search
depth $N_{depth}$ of 10, the number of working memories $N_{wm}$ set to 64,
the number of reminded engrams in short-term memory $N_{stm}^{rem}$, and the
number of remind engrams in long-term memory $N_{ltm}^{rem}$ both set to 64, a
capacity of short-term memory of 128, and the memory layer index set to 9.
This means that the output of the 10th layer is used as memory, and it is
referenced in the remaining 2 layers of the model.

##  Appendix C Memoria applied Transformers

###  C.1 Memoria Transformer

Figure 6: Architecture of Memoria Transformer. $t$ represents the current time
step, and $x$ is the input embedding. The residual network and layer
normalization are omitted for clarity. Memoria Transformer creates engrams
from the previous time step output $h_{t-1}$ and reminds engrams from short-
term and long-term memory. Memoria Transformer exploits the engrams with cross
attention. Memory Attention, depicted as two blocks in the diagram, is
actually a single layer that shares the same weights.

###  C.2 BERT with Memoria

Figure 7: Architecture of BERT with Memoria. $t$ represents the current time
step, and $x$ is the input embedding. The residual network and layer
normalization are omitted for clarity. In BERT, unlike in GPT-2, engrams are
created using information from the current time step. $l$ represents the
memory layer index, and from layer 1 to layer $l$, each layer is identical to
a regular BERT layer. Using the output $h_{t}^{l}$ from layer $l$, engrams are
created and reminded from short-term and long-term memory. These engrams are
then utilized in the subsequent layers (after layer $l$) through cross
attention. Memory Attention blocks, depicted as two blocks in the diagram,
actually share the same weights.

##  Appendix D Visualization of Memoria

(a) $t=20$

(b) $t=60$

(c) $t=130$

Figure 8: Changes in engrams of Memoria over time. The dots represent engrams,
and the lines represent connections between engrams. $t$ is the time step.
Engrams on right side are created more recently. Only the connections with
high weights are shown for clarity. As we expected, the engrams gradually fade
away. However, we can see that some important engrams still remains for a
longer duration.

(a) $t=87$

(b) $t=105$

(c) $t=122$

Figure 9: Changes in engrams of Memoria over time when Memoria see same data
twice. The lower half of each image represents the engrams generated when
observing at first, while the upper half represents the engrams generated when
observing at second. Thus, the dots in the same vertical column represent
engrams created from the same data. Engrams from the same data exhibit a
generally stronger connectivity.


# Smart Finite State Devices:
A Modeling Framework for Demand Response Technologies

Konstantin Turitsyn, Scott Backhaus, Maxim Ananyev and Michael Chertkov The
work of MC at LANL was carried out under the auspices of the National Nuclear
Security Administration of the U.S. Department of Energy at Los Alamos
National Laboratory under Contract No. DE-AC52-06NA25396.K. Turitsyn is with
MIT, Mechanical Engineering, Cambridge, MA 02139<EMAIL_ADDRESS>Backhaus
is with MPA Division at LANL, Los Alamos, NM 87545<EMAIL_ADDRESS>Ananyev
is with New Economic School, Moscow, Russia<EMAIL_ADDRESS>Chertkov is with Theory Division & Center for Nonlinear Studies at LANL, Los
Alamos, NM 87545 and also with New Mexico Consortium, Los Alamos, NM 87544
<EMAIL_ADDRESS>
###### Abstract

We introduce and analyze Markov Decision Process (MDP) machines to model
individual devices which are expected to participate in future demand-response
markets on distribution grids. We differentiate devices into the following
four types: (a) optional loads that can be shed, e.g. light dimming; (b)
deferrable loads that can be delayed, e.g. dishwashers; (c) controllable loads
with inertia, e.g. thermostatically-controlled loads, whose task is to
maintain an auxiliary characteristic (temperature) within pre-defined margins;
and (d) storage devices that can alternate between charging and generating.
Our analysis of the devices seeks to find their optimal price-taking control
strategy under a given stochastic model of the distribution market.

##  I Introduction

Automated demand response is often used to manage electrical load during
critical system peaks[1, 2]. During a typical event as the system approaches
peak load, signaling from the utility results in automated customer load
curtailment for a given period of time to avoid overstressing the grid.
Although this type of load control is useful for maintaining system security,
automated demand response must evolve further to meet the coming challenge of
integrating time-intermittent renewables such as wind or photovoltaic
generation. When these resources achieve high penetration and their temporal
fluctuations exceed a level that can be economically mitigated by the
remaining flexible traditional generation (e.g. combustion gas turbines),
automated demand response will play a large role in maintaining the balance
between generation and load. To fill this role, automated demand response must
go beyond today’s peak-shaving capability

To follow intermittent generation, automated demand response must be bi-
directional control, i.e. it should provide for controlled increases and
decreases in load. The response must also be predictable and preferably non-
hysteretic, otherwise the load-generation imbalance may actually be
exacerbated. Predictability would be highly valued by third party companies
that aggregate loads into a pool of demand response resources. Finally,
whatever control methodology is implemented, it must also be stable and not
exhibit temporal oscillations. There are several factors that make achieving
these demand response goals challenging: the different options for demand
response signal, the uncertainty of the aggregate response to that signal, and
the inhomogeneity of the underlying ensemble of loads.

The demand response control signal could take several forms: direct load
control where some number of loads could be disabled via a utility-controlled
switch[3, 4]; end-use parameter control where an ensemble of loads can be
controlled by modifying the set point of the end-use controller, e.g. a
thermostat temperature set point[5, 6, 7]; or indirect control via energy
pricing in either a price taking (open loop) or auction (closed loop)[8]
setting. Today’s automated demand response for peak-shaving is a form of
direct load control which could be adapted and refined for the type of
operation we desire, however, it is difficult to assess the impact of demand
response on the end user because loads are simply disabled and re-enabled with
little concern for the current state of the end use. Direct control is
feasible for a relatively small number of large loads because the
communication overhead is not extreme. Individual direct control of a large
number of small loads would potentially overburden a communication system,
however, “ensemble” control using a single parameter for control has been
proposed, e.g. set point control for thermostatic loads[5, 6, 7] and
connection rate control for electric vehicle charging[9]. However, in these
control models, the underlying loads are assumed to be homogeneous (all of the
same type), which is advantageous because it allows for a quantifiable measure
of the end use impacts and customer discomfort, e.g. increasing all cooling
thermostat set points by 1${}^{o}F$ will generate a decrease in load with a
known end-use impact.

To control a large ensemble of inhomogenous loads with a single demand
response signal requires a quantity that applies to all loads, i.e. energy
pricing [10]. When given access to energy prices, consumers (or automated
controllers acting on their behalf) can make their own local decisions about
whether to consume or not. These local decisions open up new possibilities and
also create problems. The customer is now enabled to automatically modify and
perhaps optimize his consumption of energy to maximize his own welfare, which
is a combination of his total energy costs and the completion of the load’s
end use function. However, without an understanding of how consumers respond
to energy prices, the fidelity of the control allowed by the direct or
ensemble control schemes described above is lost. Retail-level double auction
markets[8] are an effective way of making demand response via pricing a
closed-loop control system, however, a logical outcome of these markets would
be locational prices potentially driven distribution system constraints making
the regulatory implementation troublesome. In contrast, a model where retail
customers are price takers may avoid some regulatory issues, however, price
taking is in essence a form of open loop control which then requires an
understanding of how the aggregate load on the system will respond to price.

Our goal in this initial work is to layout the computational framework for
discovering the end-use response to these price-taking “open-loop” control
systems. We develop state models for several different loads and subject them
to a stochastic price signal that represents how energy prices might behave in
an grid with a large amount of time-intermittent generation. We analyze the
response of these smart loads using a Markov Decision Process (MDP) to
optimize the welfare of the end user. Human owners of the devices have the
ability to program the devices in accordance to their strategies and
preferences, for instance by adjusting their willingness to sacrifice comfort
in exchange for savings on electricity costs. Otherwise, most of the time we
assume that the devices operate automatically in accordance to some optimal
algorithm that was either preprogrammed by their owners, discovered via
adaptive learning[11], or programmed by a third-party aggregator. The
resulting load end-use policies can then be turned around to predict the
effect of a change in prices on electrical load. Our long term strategic
intention is to analyze the aggregated network effect on power flows of many
independent customers and design optimal strategies for both consumers and the
power operator. However, the prime focus of our first publication on the
subject is less ambitious. We focus here on description of different load
models and analyze the optimal behavior of individual consumers.

The material in the manuscript is organized as follows. We formulate our main
assumptions and introduce the general MDP framework in Section II. Models of
four different devices (optional, deferable and control loads and storage
devices) are introduced in Section III. Our enabling simulation example of a
control load (smart thermostat) is presented in Section IV. We summarize our
main results and discuss a path forward in Section V.

##  II Setting the Problems

###  II-A Basic Assumptions

Future distribution networks are expected to show complex, collective behavior
originating from competitive interaction of individual players of the
following three types:

  * •

Market operator, having full or partial control over the signals sent to
devices/customers. The most direct signal is energy price. The operator may
also provide subsidies and incentives or impose penalties, however in this
manuscript, we will mainly focus on direct price control.

  * •

Human customers/owners, who are able to reprogram smart-devices or override
their actions.

  * •

Smart devices, capable of making decisions about their operations. The devices
are semi-automatic, i.e. pre-programmed to respond to the signal on a short
time scale (measured in seconds-to-minutes) in a specific way, however the
owner of the device may also choose to change the strategy on a longer time-
scale (days or weeks). We model the smart devices as finite state machines
using a Markov Decision Process (MDP) framework. At the beginning of each
interval, a device decides how to change its state based on the current price.
Each change comes with a reward expressing actual transactions between the
provider and the consumer and the level of consumer satisfaction with the
decision. We assume that smart devices are selfish and not collaborative, each
optimizing its own reward.

In this manuscript we restrict our attention to a simple price-taking strategy
of consumer behavior, deferring analysis of more elaborate game-theoretic
interactions between the operator and the individual customers to further
publications.

We model the external states (that include electricity price, weather, and
human behavior) as a stochastic, Markov Chain process, $\\{s^{(e)}(t)\\}$. At
the beginning of the time interval, $t$, the variable describing these factors
is set to $s^{(e)}_{t}$ and changes during the next time step to
$s^{(e)}_{t+1}$ with the transition probability
$T(s^{(e)}_{t+1}|s^{(e)}_{t})$. The transition probabilities are assumed to be
known to the device and statistically stationary, i.e. independent of $t$.
(The later assumption can be easily relaxed to account for natural cycles and
various external factors.) The probability, $p(s^{(e)};t)$, to observe the
external state, $s^{(e)}(t)=s^{(e)}$, at the time $t$, follows the standard
Markov chain equation

$\displaystyle
p(s^{(e)};t+1)=\sum_{s_{t}^{(e)}}T(s^{(e)},s_{t}^{(e)})p(s_{t}^{(e)}|t).$ (1)

We also assume that the Markov chain (1) is ergodic and converges after a
finite transient to the statistically stationary distribution:
$p(s^{(e)};t+1)=p(s^{(e)};t)=p(s^{(e)})$. In the simulation tests that follow
we will restrict ourselves to $s^{(e)}$ drawn from a finite set $S^{(e)}$.

###  II-B General Markov Decision Process Framework

Here we adopt the standard (Markov Decision Process) MDP approach [12, 13, 14]
to the problem of interest: description of smart devices responding to the
external (exogenous) Markov process $\\{s^{(e)}(t)\\}$. MDPs provide a
mathematical framework for modeling decision-making in situations where
outcomes are partly random and partly under the control of a decision maker.
Formally, the MDP is a 4-tuple, $(S,A,P(\cdot,\cdot),R(\cdot,\cdot))$, where

  * •

$S$ is the finite set of states, in our case a direct product of the machine
states set $S^{(m)}$, and the externality state set $S^{(e)}$,
$S=S^{(m)}\otimes S^{(e)}$.

  * •

$A$ is a finite set of actions. $A_{s}$ is the finite set of actions available
from state $s\in S$. Within our framework we model only the decisions made by
the machine, so the set $A$ consists only of actions associated with the
machine, $A=A^{(m)}$.

  * •

$P_{a}(s,s^{\prime})=\Pr(s_{t+1}=s^{\prime}\mid s_{t}=s,\,a_{t}=a)$ is the
probability that action $a$ chosen while in state $s=(s^{(m)},s^{(e)})$ at
time $t$ will lead to state $s^{\prime}$ at time $t+1$. The probabilistic
description of the transition allows to account for stochastic nature of the
price fluctuations as well as for the randomness in the dynamics of the smart
devices.

  * •

$R_{a}(s,s^{\prime})$ is the reward associated with the transition $s\to
s^{\prime}$ if the action $a$ was chosen. In our models, the reward will
reflect the price paid for electricity consumption associated with the
transition as well as the level of discomfort related to the event.

In the most simple setting analyzed in this work, the behavior of the device
is modeled via the policy function $\pi(s):S\to A$ that determines the action
chosen by the device for a given state: $a_{t}=\pi(s_{t})$. More general
formulations that include randomized decision making process, are not
considered in this paper. Our smart device models seek to operate with the
policy, $\pi(s)$, that maximizes over actions the expectation value of the
total discounted reward,
$\sum_{t=0}^{\infty}\gamma^{t}R_{a_{t}}(s_{t},s_{t+1})$ over the Markov
process, $P_{a}(\cdot,\cdot)$, where $0<\gamma\leq 1$, is the discount rate.
There are numerous algorithms used for optimizing the policies. In our work we
use the algorithms implemented in MDP Matlab toolbox [14].

##  III Models of Devices

The specifics of our MDP setting are to be described below for four examples
of loads. Note that these examples are meant to illustrate the power of the
framework and its applicability to ”smart grid” problems. In this first paper,
we do not aim to make the examples realistic. Instead, we focus on the
qualitative features of the loads.The states and actions associated with the
devices are illustrated in the diagrams shown in Figs. 1-4. For simplicity, we
ignore the external part of the state $s^{(e)}$ in these diagrams. Full
diagrams can be produced by taking the Kronecker product of transition graphs
associated with the device and the external factors. In our diagrams, the
states are marked by squares and actions are marked by dashed circles.
Transitions from states to actions and actions to states are marked by dashed
and solid arrows, respectively.

Figure 1: MDP diagram for the model of optional load. See text for
explanations.

###  III-A Optional Loads

A smart device described by an “optional load” pattern can operate in two
regimes, at full and limited capacity. An example of such load is a light that
can be automatically dimmed if the electricity price becomes too high (see
Fig. 1). To simplify the mathematical notations, we denote the states of the
machine $s^{(m)}$ by $x$. The machine can be in either of the two states:
$x=0$ and $x=1$, shown as $Idle$ and $Active$ in the diagram (1) respectively.
In the $x=0$ state the machine does not operate (the lights are off). In the
$x=1$ state, the machine is active and the lights are shining at the full
brightness, or are dimmed. Actions of the device are $a_{0}=\mbox{pass}$,
$a_{1}=\mbox{full}$ or $a_{2}=\mbox{shed}$. The $a_{0}$ action represents the
process of waiting for the external signal of switching on the device. If no
external external signal (requesting switching on) appears, the system returns
to the $x=0$ state, otherwise it moves to the $x=1$ state. When the device is
active (in the $x=1$ state), it has two options: operate at full capacity,
corresponding to the action $a_{1}$, or shed the load (dim the lights),
corresponding to action $a_{2}$. Turning the device on or off is an
externality dependent on a human. We assume that the external/human action is
random, with the probability of turning the device ON and turning the device
OFF being $\rho_{ON}$ and $\rho_{OFF}$ respectively. (For simplicity, we
assume that the OFF signal may arrive only by the end of the time interval.)
Assuming additionally that the transition probabilities do not not depend on
the device actions, we arrive to the following expression for the transition
kernel:

$\displaystyle
P_{pass}(s,s^{\prime})=T(c^{\prime}|c)\left[\rho_{ON}\delta_{x^{\prime},1}+(1-\rho_{ON})\delta_{x^{\prime},0}\right],$
(2) $\displaystyle
P_{full,shed}(s,s^{\prime})=T(c^{\prime}|c)\left[\rho_{OFF}\delta_{x^{\prime},0}+(1-\rho_{OFF})\delta_{x^{\prime},1}\right],$
(3)

where $\delta_{x_{1},x_{2}}$ is the Kronecker symbol: it is unity if
$x_{1}=x_{2}$ and zero otherwise.

There is no reward associated with either outcome of the $a_{0}=\mbox{pass}$
action, however, the other two actions ($a_{1}$ and $a_{2}$) result in a
reward consisting of two contributions. First is the price paid to the
electricity provider, $E_{full,shed}c$, where $c(t)$ is the cost of
electricity (considered as a component of $s^{(e)}$) and $E_{full,shed}$ is
the amount of energy consumed during the time interval which depends on
whether the lights are fully on or dimmed. ( Here, $E_{full}>E_{shed}>0$ and
both values do not depend on the resulting state of the device). Second, the
reward function accounts for a subjective level of comfort associated with the
$a_{1,2}$ actions: $C_{full,shed}$. The discomfort of the light dimming is
accounted by choosing $C_{full}>C_{shed}$. Summarizing, the cumulative reward
function in this model of the optional load becomes

$\displaystyle R_{pass}(s,s^{\prime})=0,$ (4) $\displaystyle
R_{full}(s,s^{\prime})=C_{full}-E_{full}c,$ (5) $\displaystyle
R_{shed}(s,s^{\prime})=C_{shed}-E_{shed}c.$ (6)

Obviously, our model of optional loads is an oversimplification because there
are a variety of additional effects which may also be important in practice,
however, all these can be readily expressed within the MDP framework. For
example, one may need to limit the wear and tear on the device, thus
encouraging (via a proper reward) minimization of switching. (To account for
this effect would require splitting the $Active$ state in the model explained
above into two states $Active-Full$ and $Active-Shed$.)

###  III-B Deferable Loads

Figure 2: MDP diagram for the model of deferable load. See text for
explanations.

Our second example model is a deferable load, i.e. a load whose operation can
be delayed without causing a major consumer discomfort. Practical examples
include dishwashing machines or some maintenance jobs like disk
defragmentation on a computer. A simple model of such a device, shown in Fig.
2, has two states: $x=0$ ($Idle$) when no work is required and $x=1$
($Waiting$) when a job has been requested and the machine is waiting for the
right moment (optimal in terms of the cost) to execute it. As in the previous
model, the only action of the machine in the $Idle$ state is $a_{0}$ ($Pass$),
however, in the $Waiting$ state, there are two possible actions: $a_{1}=Wait$
results in waiting for possible drop of the electricity price and $a_{2}=Work$
results in immediate execution of the job. The transition kernel for the model
is

$\displaystyle
P_{Pass}(s,s^{\prime})=T(c^{\prime}|c)\left[\rho_{ON}\delta_{x^{\prime},1}+(1-\rho_{ON})\delta_{x^{\prime},0}\right],$
(7) $\displaystyle
P_{Wait}(s,s^{\prime})=T(c^{\prime}|c)\delta_{x^{\prime},1},$ (8)
$\displaystyle P_{Work}(s,s^{\prime})=T(c^{\prime}|c)\delta_{x^{\prime},0},$
(9)

where $\rho_{ON}$ is the probability of an exogeneous job request. In this
model, there is no reward for choosing the $a_{0}=Pass$ action. The reward for
the $a_{2}=Work$ action is equal to minus the price paid for the electricity,
$R_{Work}(s,s^{\prime})=-E*c$, and the reward for the $a_{1}=Wait$ action
represents the level of discomfort associated with the delay,
$R_{Wait}=C_{delay}<0$. As in the model of optional loads, $E$ and $C_{delay}$
are constants parameters.

###  III-C Control Loads

Figure 3: MDP diagram for the model of controllable load. See text for
explanations.

A very important class of devices that will likely play a key role in future
demand response technologies are machines tasked to maintain a prescribed
level of physical characteristics of some system. For example, thermostats are
tasked with keeping the temperature in a building within acceptable bounds.
Other examples of the control devices are water heaters, electric ovens,
ventilation systems, CPU coolers etc.

In our enabling, proof-of-principle model of the control load, we consider a
thermostat responsible for temperature control in a residential home. The
state of the device is fully characterized by temperature which can take three
possible values: $x=0,1,2$ corresponding to $Low,Medium,High$ temperatures,
respectively. Each temperature is assumed to be operationally acceptable. For
simplicity, we assume that the thermostat uses an electric heater to modify
the temperature (i.e. the outside temperature is low). The device can choose
between the following three actions. $a_{0}=Cool$ leaves the heater idle for
the forthcoming interval. Since there is some base consumption associated with
the thermostat operation we assume that $E_{Cool}>0$. The next action,
$a_{1}=Keep$, maintains the temperature at the current level and requires some
energy for heater operation: $E_{Keep}>E_{cool}>0$. Finally, $a_{2}=Heat$
corresponds to intensive heating that raises the temperature and requires the
largest amount of energy $E_{Heat}$, and $E_{Heat}>E_{Keep}>E_{Cool}=0$. Our
thermostat state diagram, shown in Fig. (3), assumes that the dynamics of the
thermostat are deterministic, and the resulting state depends only on the
action chosen. The transition probabilities of the thermostat MDP is

$\displaystyle P_{Heat}(s,s^{\prime})=T(c^{\prime}|c)\delta_{x^{\prime},x+1},$
(10) $\displaystyle
P_{Keep}(s,s^{\prime})=T(c^{\prime}|c)\delta_{x^{\prime},x},$ (11)
$\displaystyle P_{Cool}(s,s^{\prime})=T(c^{\prime}|c)\delta_{x^{\prime},x-1}.$
(12)

Assuming that all levels of temperature are equally comfortable, the reward
function depends only on the price and energy consumption associated with the
action,

$\displaystyle R_{Cool,Keep,Heat}(s,s^{\prime})=-cE_{Cool,Keep,Heat}.$ (13)

Our basic model can be generalized to account for different comfort levels of
different states, the possibility for the owner to override an action,
variations of the outside temperature, etc.

###  III-D Storage loads

Figure 4: MDP diagram for the model of storage. See text for explanations.

The number of devices with rechargeable batteries is expected to increase
dramatically in the coming years. Currently, these are mostly laptops,
uninterruptable power supplies, etc. In addition, a significant number of
large-scale batteries will be added to the grid most likely via the
anticipated Plug-in Hybrid Electric Vehicles (PHEV) potentially enabled with
Vehicle-to-Grid (V2G) capability. Storage devices, illustrated with the MDP in
Fig. (4), share some similarity with the controlled loads discussed in the
previous Subsection, but they are also different in two aspects. First,
users/owners wants their devices to be charged which leads to a level of
discomfort if the devices are not fully charged. Second, and probably most
significantly, storage devices such as PHEVs are disconnected from the grid
when in use. Having PHEVs in mind, we propose the following model of (mobile)
storage. The system can be in either of the three states, the $x=0=Unplugged$
state (which is similar to the Idle state in the models of Optional and
Deferable loads discussed above), the $x=1=Partially$ state where the storage
is partially charged, and the $x=2=Full$ state where the device is fully
charged.

The four available actions are: $a_{0}=Pass$ when the device is in the
unplugged state, the $a_{1}=Keep$ action possible when the initial state is
$x=1=Partially$ or $x=2=Full$, the $a_{2}=Charge$ action available from the
$x=1=Partially$ state which transitions to the $x=2=Full$ state, and, finally,
the $a_{3}=Discharge$ action, that is an inverse of the $a_{2}$ one, available
from the $x=2=Full$ state resulting in the $x=1=State$. Except for
$a_{0}=Pass$, all these actions can be interrupted by transitioning at the end
of the time interval to the $x=0=Unplugged$ state. As in previous sections, we
assume that the unplugging happens at the end of a time interval. Assuming the
device can be unplugged with the probability $\rho_{OFF}$ and that it can be
reconnected to the grid with the probability $\rho_{ON}$, we arrive at the
following expressions for the transition probability:

$\displaystyle
P_{Pass}(s,s^{\prime})=T(c^{\prime}|c)\left[\rho_{ON}\delta_{x^{\prime},1}+(1-\rho_{ON})\delta_{x^{\prime},0}\right],$
(14) $\displaystyle
P_{Keep}(s,s^{\prime})=T(c^{\prime}|c)\left[\rho_{OFF}\delta_{x^{\prime},0}+(1-\rho_{OFF})\delta_{x,x^{\prime}}\right],$
(15) $\displaystyle
P_{Charge}(s,s^{\prime})=T(c^{\prime}|c)\left[\rho_{OFF}\delta_{x^{\prime},0}+(1-\rho_{OFF})\delta_{x^{\prime},2}\right],$
(16) $\displaystyle
P_{Discharge}(s,s^{\prime})=T(c^{\prime}|c)\left[\rho_{OFF}\delta_{x^{\prime},0}+(1-\rho_{OFF})\delta_{x^{\prime},1}\right].$
(17)

The reward function accounts for the following effects. First, the
$a_{1}=Keep$ action has the cost associated with keeping the battery charged,
$E_{Keep}(x)$, naturally dependent on the state,
$E_{Keep}(2)>E_{Keep}(1)>E_{Keep}(0)=0$. Second, the $a_{2}=Charge$ action
requires $E_{Charge}$ of energy while the $a_{3}=Discharge$ action generates
the $E_{Discharge}<0$ of energy, both nonzero only if the resulting state is
not the $x=0=Unplugged$. Therefore, all the “active” actions,
$Keep,Charge,Discharge$, contribute the reward function in accordance with the
energy price, $c^{\prime}E_{\dots}$. Finally, we also assign an additional
negative reward, $C_{Unplug}<0$, accounting for the discomfort (to the human)
associated with being in the $x=0=Unplugged$ state. The resulting reward
function is

$\displaystyle R_{Pass}(s,s^{\prime})=0,$ (18) $\displaystyle
R_{Keep}(s,s^{\prime})=C_{Unplug}\delta_{x^{\prime},0}\delta_{x,1}-cE_{Keep}(x),$
(19) $\displaystyle R_{Charge}(s,s^{\prime})=-cE_{Charge},$ (20)
$\displaystyle
R_{Discharge}(s,s^{\prime})=C_{Unplug}\delta_{x^{\prime},0}-cE_{Discharge}.$
(21)

##  IV Simulations

In order to illustrate the capabilities of the proposed framework, we consider
a simple model of the control load, describing a smart thermostat,
characterized by $N_{T}=10$ levels of the temperature parameter $T$. At every
moment of time the thermostat can choose to raise, lower or keep the same
temperature. The raise and lower options are not available at the highest and
lowest possible temperatures, respectively. The energy consumption associated
with the actions is given by $E_{Keep}=1.0$, $E_{Cool}=0.1$ and
$E_{Heat}=2.1$, respectively, in some normalized energy units. This choice of
energies discourages the system from switching the heater too often: although
the combinations $Heat+Cool$ and $Keep+Keep$ lead to the same temperature
levels, the latter action is preferable as it consumes less energy.

Variations in price are modeled by a Markov chain of $N_{P}=5$ equidistant
levels with the minimum and maximum corresponding to $1.0$ and $2.0$ price
units, respectively. At each time interval, the price either increases with
probability $T(c+1|c)=0.5$ by $1$ level, decreases with probability
$T(c-1|c)=0.3$ by $1$ level, or stays the same. The resulting stationary
probability distribution $p(c)$ is shown in the Figure 5. It is skewed towards
the higher price, mimicking the effect of intermittent renewable generators
that occasionally provide excess power to the grid, thus leading to rapid dips
in the price.

Figure 5: Probability distribution of electricity price in the model example.

The reward function (13) is fully determined by the total cost of energy
consumed by the thermostat within the given time-interval. Our MDP model
imposes upper and lower bounds on the temperature, and we assume that there is
no additional discomfort associated with the variations of temperature between
these bounds, i.e. all of the $N_{T}$ temperature levels are equally
comfortable for the consumer.

Figure 6: Visualization of the policy found as a result of optimization.

This system was analyzed with the Matlab MDP package [14] where we used
different algorithms to verify the stability of the results. The resulting
optimal policy (for the range of parameters tested) is illustrated in 6. As
expected, the thermostat chooses the $Heat$ action when the price is low and
decides to $Cool$ when the price is high; a set of actions that lead to the
skewed probability distribution of temperatures shown in Figure 7. One finds
that the thermostat spends most of the time performing $Keep$ in the low
temperature state waiting for the price to drop.

Figure 7: Probability distribution of temperature levels observed at the
optimal policy.

Perhaps, the most interesting feature of the MDP model is the relation between
consumption and price. We define the expected demand as the average energy
demand for a given price

$\langle
E|c\rangle=\frac{\sum_{x}E_{\pi(x,c)}P_{st}(x,c)}{\sum_{x}P_{st}(x,c)},$ (22)

where $P_{st}(x,c)$ is the stationary joint distribution function of the
temperature and price at the optimal strategy. Dependence of the consumption
on the price for our choice of the parameters is shown in Figure 8, thus
illustrating that variations in price indeed produce demand response. An
interesting feature is that the demand curve is not monotonic. At low
temperatures, the energy consumption shows a slight increase with the price; a
surprising behavior related to saturation of the demand. When the electricity
price decreases gradually from high to low levels, there is a high probability
that the thermostat will reach the highest level of temperature before the
price reaches the lowest level. In this case, the demand will be lower at the
smallest price levels as there will be no unsatisfied demand left in the
system to capitalize on the lowest price. From the economic viewpoint, it is
important to note that this non-monotonicity of the demand curve reflects the
adaptive nature of the MDP algorithm: the smart devices adjust to fluctuations
in price, thus making it more difficult for the electricity providers to
exploit the non-monotonic demand curve for making profit.

Figure 8: Demand of the smart thermostats.

Another interesting result found in our simulations is an increase in average
consumption of the smart (policy optimized) thermostat when compared to its
non-smart counterpart, where the latter is defined as the one ignoring price
fluctuations and sticking to the $Keep$ action. For the set of parameters
chosen in the test case, we observed that the average level of consumption in
the optimal case is $1.03$, i.e. it is $3\%$ higher than in the naive
strategy, an effect associated with the additional penalty (in energy) imposed
on the $Heat$ and $Cool$ actions.

It is also instructive to evaluate savings of the consumer. The average value
of the reward associated with the optimal policy is equal to $-1.6722$, which
should be compared with the reward of $-1.73$ generated by its non-smart
counterpart. Since the reward reflects the customer’s cost of electricity, we
conclude that the customer saves about $3\%$ on the electricity costs
associated with the thermostat. The lower total energy costs for higher energy
consumption was also seen in a related “smart-device” demonstration
project[8].

Note, that the quantitative conclusions drawn and numbers presented above were
meant to illustrate the questions the MDP approach can resolve. The
conclusions and the numbers do not represent any real device as the parameters
used were not justified by actual data.

##  V Discussions, Conclusions and Path Forward

To conclude, we have presented a novel modeling framework to analyze future
demand response technologies. The main novel aspect of our approach lies in
the capability of the framework to describe behavior of the smart devices
under varying/fluctuating electricity prices. To achieve this goal, we modeled
the devices as rational agents which seek to maximize a predefined reward
function associated with its actions. In general, the reward function includes
the price paid for the electricity consumption and the level of owner
discomfort associated with the choices made by the device. At the mathematical
level, the system can be described via Markov Decision Processes that have
been extensively studied over the last 50 years. Utilizing the MDP approach,
we showed that a great variety of practical devices can be described within
the same framework by simply changing the set of device states, actions and
reward functions. Specifically, we identified four main device categories and
proposed simple MDP models for each of them. These four categories include
optional loads (like light dimming), deferrable loads (like dishwashing),
control loads (thermostats and ventilation systems), and finally storage loads
(charging of batteries).

To illustrate the approach we experimented with a simple model of a smart
heating thermostat. The MDP-optimized policy of the thermostat followed the
expected pattern: it chooses to not heat or keep the temperature stationary at
high prices and prefers to heat when the price is low. This policy resulted in
$3\%$ of savings in the price paid for electricity, but at the same time led
to the total of $3\%$ increase in the consumption level due to the energy
costs associated with the thermostat actions. The resulting demand curve
showed a noticeable amount of elasticity, thus meeting the main objective of
the demand response technology.

There are many relevant aspects of the model that we did not discuss in the
manuscript. We briefly list some of these and future research challenges and
direction.

  * •

_Learning algorithms_. In our model we assumed that smart devices have an
accurate model of stochastic dynamics for external factors (such as price for
electricity), and use this model to find the optimal policy. In reality,
however, this model is not known ab initio and has to be learned from the
observations. Moreover, one can expect that the dynamics of external factors
will be highly non-stationary (i.e. the transition matrix
$T(s^{(e)}_{t+1}|s^{(e)}_{t+1})$ will have an explicit dependence on time).
Therefore, the optimal policy has to be constantly adapted to the varying
dynamics of the external factors. Of a special practical interest is the
generalization of the framework to almost periodic processes, reflecting
natural daily/weekly/yearly cycles in the electricity consumption.

  * •

_Price-setting policies_. We did not discuss the price setting policies above,
assuming that the policies are given/pre-defined. However, the electricity
providers might adjust their policies to consumer response. As the electricity
providers pursue their own goals, this setting essentially becomes game-
theoretic and as such it requires more sophisticated approaches for analysis.
Another extension of the model is to introduce auction-based price-setting
schemes, such as in the Olympic Peninsula project [8]. This setting can be
naturally incorporated in the same framework, although the modification may
require simultaneous modeling of multiple (ensemble of) devices.

  * •

_Time delays_. Another aspect of the real world not incorporated in our
analysis concerns the separation of the time scales associated with operations
of the device and intervals of the price variations. Multiple time-scale can
be naturally incorporated in the framework by introducing additional states of
the device. These modifications will certainly affect final answer for the
optimal policy, and the resulting demand curve. However, accurate
characterization of the multi-scale behavior will be a challenging task,
requiring analysis of nonlinear response functions and dynamical description
of the underlying non-Markovian processes.

##  VI Acknowledgements

We are thankful to the participants of the “Optimization and Control for Smart
Grids” LDRD DR project at Los Alamos and Smart Grid Seminar Series at
CNLS/LANL for multiple fruitful discussions.

## References

  * [1] N. Motegi, M. A. Piette, W. D., S. Kiliccote, and P. Xu, “Introduction to commercial building control strategies and techniques for demand response,” LBNL Report Number 59975, Tech. Rep., 2007. [Online]. Available: http://gaia.lbl.gov/btech/papers/59975.pdf
  * [2] “U.s. department of energy, “benefits of demand response in electricity markets and recommendations for achieving them”,” U.S. DOE, Tech. Report, 2006\.
  * [3] S. S. Oren and S. A. Smith, “Design and management of curtailable electricity service to reduce annual peaks,” _OPERATIONS RESEARCH_ , vol. 40, no. 2, pp. 213–228, 1992. [Online]. Available: http://or.journal.informs.org/cgi/content/abstract/40/2/213
  * [4] R. Baldick, S. Kolos, and S. Tompaidis, “Interruptible electricity contracts from an electricity retailer’s point of view: Valuation and optimal interruption,” _OPERATIONS RESEARCH_ , vol. 54, no. 4, pp. 627–642, 2006\. [Online]. Available: http://or.journal.informs.org/cgi/content/abstract/54/4/627
  * [5] D. S. Callaway, “Tapping the energy storage potential in electric loads to deliver load following and regulation, with application to wind energy,” _Energy Conversion and Management_ , vol. 50, no. 5, pp. 1389 – 1400, 2009\. [Online]. Available: http://www.sciencedirect.com/science/article/B6V2P-4VS9KPY-1/2/32649b4a9a6779a2cea84379a7c1f9a6
  * [6] D. Callaway and I. Hiskens, “Achieving controllability of electric loads,” _Proceedings of the IEEE_ , vol. 99, no. 1, pp. 184 –199, 2011.
  * [7] S. Kundu, N. Sinitsyn, S. Backhaus, and I. Hiskens, “Modeling and control of thermostatically controlled loads,” in _17th Power Systems Computation Conference_ , 2011, submitted.
  * [8] D. Hammerstrom and et al, “Pacific northwest gridwise testbed demonstration project:part i. olympic peninsuila project,” PNNL-17167, Tech. Rep., 2007. [Online]. Available: http://gridwise.pnl.gov/docs/op˙project˙final˙report˙pnnl17167.pdf
  * [9] K. Turitsyn, N. Sinitsyn, S. Backhaus, and M. Chertkov, “Robust broadcast-communication control of electric vehicle charging,” in _Smart Grid Communications (SmartGridComm), 2010 First IEEE International Conference on_ , 2010, pp. 203 –207.
  * [10] F. Schweppe, R. Tabors, J. Kirtley, H. Outhred, F. Pickel, and A. Cox, “Homeostatic utility control,” _Power Apparatus and Systems, IEEE Transactions on_ , vol. PAS-99, no. 3, pp. 1151 –1163, May 1980.
  * [11] D. O’Neill, M. Levorato, A. Goldsmith, and U. Mitra, “Residential demand response using reinforcement learning,” in _Smart Grid Communications (SmartGridComm), 2010 First IEEE International Conference on_ , 2010, pp. 409 –414.
  * [12] R. E. Bellman, “A markovian decision process,” _Jounral of Mathematics and Mechanics_ , vol. 6, 1957.
  * [13] M. L. Putterman, _Markov Decision Processes. Discrete Stochastic. Dynamic Programming._ Wiley-Interscience, 2005\.
  * [14] “Markov decision process (mdp) toolbox for matlab.” [Online]. Available: http://www.cs.ubc.ca/~murphyk/Software/MDP/mdp.html