11institutetext: SaluteDevices, Russia
11email: {aakapitanov, kskvanchiani, aonagaev<EMAIL_ADDRESS>
# Slovo: Russian Sign Language Dataset

Alexander Kapitanov  Karina Kvanchiani  Alexander Nagaev  Elizaveta Petrova

###### Abstract

One of the main challenges of the sign language recognition task is the
difficulty of collecting a suitable dataset due to the gap between deaf and
hearing society. In addition, the sign language in each country differs
significantly, which obliges the creation of new data for each of them. This
paper presents the Russian Sign Language (RSL) video dataset Slovo, produced
using crowdsourcing platforms. The dataset contains 20,000 FullHD recordings,
divided into 1,000 classes of RSL gestures received by 194 signers. We also
provide the entire dataset creation pipeline, from data collection to video
annotation, with the following demo application. Several neural networks are
trained and evaluated on the Slovo to demonstrate its teaching ability.
Proposed data and pre-trained models are publicly
available111https://github.com/hukenovs/slovo
222https://gitlab.aicloud.sbercloud.ru/rndcv/slovo.

###### Keywords:

Sign Language Video Dataset Gesture Recognition Data Creating Pipeline Human
Computer Interaction.

##  1 Introduction

While the contemporary world is developing rapidly with the advent of high-end
technologies, some parts of society are out of their scope. One such part is
the deaf community, which still struggles in many situations and can be
misunderstood in some extreme cases. For example, some hospitals still do not
have a sign language interpreter on staff, and therefore the interaction
between deaf people and healthcare providers is complex, which prevents timely
assistance. A similar problem exists in structures such as banks, government
institutions, airports, public places, and others, significantly complicating
their everyday life. Moreover, many consequences of deafness as social
isolation, an education gap with the hearing population, and difficulties in
finding employment, also negatively affect the life of this community. Sign
Language Recognition (SLR) systems have the potential to simplify these
processes by, for example, developing a sign language learning app [4][5] or
embedding a feature in video conferencing apps. Also, such technology can
accomplish more transparent communication between people with different
hearing and speaking abilities and be integrated into human-computer
interaction systems [14], allowing deaf individuals access to information and
services easier and helping to overcome barriers in education [2] and
employment.

Figure 1: RSL signs "at eight fifteen" (left top), "appetite" (left bottom),
"yellow" (right top), and "this" (right bottom).

SLR is a field of study that should accurately convert sign language gestures
from video footage into textual representation. This task is indispensable but
daunting due to the tangle and rapid nature of sign language, which entails
intricate hand gestures, body postures, and facial expressions.

The complexity of data collection is the major problem of SLR due to a gap
between the deaf and hearing community. Adding to this the need for a
different sign language for each country and significant language differences
within even one country, Russian SLR system developers face the challenge of
data absence. Furthermore, existing RSL datasets have few samples or must be
sufficiently diverse across subjects, which is necessary to train a robust
model. This paper presents two main contributions to simplify the solution of
sign language recognition:

  * •

We provide a pipeline for creating a video dataset consisting of three main
steps: video collection, video validation, and time interval annotation.
Crowdsourcing platforms were used throughout the pipeline to increase the
number of signers and improve the dataset’s quality. We apply some exam tasks
to signers for the most correctly executed gestures and add a quality check to
the validation step to extract inappropriate videos. In the third step, all
videos were marked by the start and end time of the gesture and then cropped.

  * •

We release the Russian Sign Language dataset, Slovo, which can become the
basis for this area. It consists of 20,000 FullHD videos from 194 signers and
is divided into 1.000 classes of glosses from the RSL without words shown by
dactyl (finger-spelling) or compound gestures. All signers recorded videos
mostly in their homes or office in front of a laptop or smartphone camera.
Each video, whose length varies from 1 second to 4, was cropped using two
timestamps (start and end), contributing to the production of the "nothing"
class. Added class corresponds to the video’s parts, where the signer is
preparing to perform the gesture or has already performed it.

##  2 Related Work

This paper solely focuses on comparing our findings with the Russian Sign
Language. It would be inappropriate to compare RSL datasets with SL datasets
in foreign languages as they all differ significantly in structure. However,
we provide an overview of datasets in other languages to show the specifics
and features of this domain. Table 1 encompasses the prevalent datasets having
significant data volumes. Since the collection step is the main problem of SL
dataset creation and one of our contributions, we describe the main ways to
collect it.

Table 1: The main characteristics of the reviewed SL datasets. In addition, we
added a column named "Method" with the collection way information since it
says a lot about the videos. "Manually" means that the dataset’s authors
recorded the video and could influence the scenes, gestures, and recording.
"Download" means that the authors downloaded videos from third-party resources
that sometimes guarantee the high accuracy of the gestures shown. "Crowd."
means that the authors used crowdsourcing platforms to collect videos.
"Subset" means that RWTH-BOSTON-400 was created from another dataset BU ASL
[1].

Dataset | Classes | Videos | Signers | Resolution | Language | Method
---|---|---|---|---|---|---
GSLC, 2007 [8] | 20 | 840 | 6 | 848 × 480 | Greek | manually
RWTH-BOSTON-400, 2008 [7] | 400 | 843 | 5 | 648 × 484 | American | subset
RWTH-PHOENIX-Weather, 2015 [15] | 1,066 | 8,257 | 9 | 210 × 260 | Germany | download
LSE-Sign, 2016 [10] | 2,400 | 2,400 | 2 | FullHD | Spanish | manually
LSA64, 2016 [20] | 64 | 3,200 | 10 | FullHD | Argentinian | manually
MS-ASL, 2018 [12] | 1,000 | 25,513 | 222 | varying | American | download
WLASL2000, 2020 [16] | 2,000 | 21,083 | 8 | varying | American | download
AUTSL, 2020 [21] | 226 | 38,336 | 43 | 512 × 512 | Turkish | manually
TheRuSLan, 2020 [13] | 164 | 13 | 13 | FullHD | Russian | manually
K-RSL, 2020 [11] | 600 | 28,250 | 10 | FullHD | Kazakh-Russian | manually
RSL, 2021 [9] | 1,000 | 35,000 | 5 | FullHD | Russian | manually
FluentSigners-50, 2022 [19] | 278 | 43,250 | 50 | varying | Kazakh-Russian | crowd.
Slovo, 2023 (ours) | 1,000 | 20,000 | 194 | HD / FullHD | Russian | crowd.

###  2.1 Sign Language Datasets in Russian Domain.

There are four more widespread RSL datasets. The first, TheRuSLan [13], is
composed of a total of 164 gestures, primarily related to the supermarket
theme. A group of 13 signers was involved in the video collection, where each
signer produced a unique recording with an average duration is 36 minutes. All
signers come from different parts of the country, which generates variability
within a class due to various dialects. The authors also proposed subtitles
for each sample, which indicate the specific signs class. The second,
FluentSigners-50 [19], were created with the help of 6 natives, who chose
frequently used signs, produced the templates for them, and wrote the
instruction for signers. All signers came from different Kazakhstan regions,
making the dataset a high degree of linguistics. Heterogeneity in the signer’s
age, skin color, clothes, variable background, and lighting make the dataset
immensely diverse. The videos are in a total of 43 hours of labeled trimmed
materials. The third, K-RSL [11], contains 4 subsets of phrases from a
linguistic point of view: question-statement pairs, signs of emotion,
emotional question-statement pairs, and phonologically similar signs. It was
divided into 600 glosses with 28,250 examples (3.6 per gloss on average). Ten
signers recorded K-RSL, the first 5 are professional SL interpreters, and the
other 5 are deaf native signers. The last, RSL [9], consists of two sets of
gestures obtained from an online dictionary. Each gesture was repeated by the
signer at least 5 times. All signers are dressed in black suits against a
clean background, which makes the dataset visually monotonous. All videos are
marked with additional classes named "start gesture" and "end gesture";
suggestions include an additional class named "transition".

###  2.2 Others Sign Language Datasets.

Since RSL is significantly different from other sign languages, we describe
only notable SL datasets, comparing them according to the main specific
features of the domain. Many of reviewed datasets are not diverse in signers:
RWTH-BOSTON-400 [7] were recorded by only 4 speakers, LSE-Sign [10] – by two
sign language natives, LSA64 [20] – by 10 non-expert signers, and GSLC – by 6.
Besides, LSE-Sign was recorded within one week to minimize the diversity of
the signer’s appearance. Others tried to make more heterogeneous datasets. MS-
ASL [12] and WLASL2000 [16] are the most extensive publicly available ASL
datasets, and their videos were produced by 222 and 47 signers, respectively.
The AUTSL dataset [21] was recorded with 20 backgrounds, including indoor and
outdoor scenes and with different angles. The reviewed datasets differ in the
goal of creating and choosing the domain of signs. For example, The RWTH-
PHOENIXWeather corpus [15] contains SL recordings from the German TV station
PHOENIX. Low resolution and frame rate are due to the TV station’s broadcast
method. The more typical variant to choose a sign basket is to collect it from
the frequently daily-used signs like in AUTSL [21].

###  2.3 Sign Language Dataset Collection.

The main problem of dataset creation for SLR is video collection because it is
challenging to find sign language experts. The need for diversity in signers
makes this task even more problematic. The choice of sign basket is the other
significant problem because natural and sign languages are significantly
different. We reviewed ways to collect sign language videos and some
annotation details. This overview was divided into three groups by collection
methods for convenience.

Manually recorded videos. One of the main ways to collect videos for sign
language recognition is to produce them manually with a camera. Kagirov et al.
[13] used the MS Kinect 2.0 device to record video in 3D with a depth map to
create the TheRusLan dataset. The Turkish Sign Language dataset, AUTSL [21],
was collected for real-life scenarios by the same camera. To make the model
robust to scenes, 20 different backgrounds, including dynamic, various
lighting conditions, from artificial light to sunlight, and different field-
of-views were used to create AUTSL. The authors choose the frequently used
signs; some are compound signs formed by simultaneously making two consecutive
signs. The videos were performed by 43 different signers, where 60% are
students of the TSL course, 18% are persons who know TSL (instructors and
translators), 15% are trained signers by the AUTSL dataset, and others
related. The Argentinian Sign Language dataset, LSA64 [20], was recorded by a
Sony HDR-CX240 camera in two different scene conditions: outdoor and indoor.
The authors simplified the hand segmentation task with fluorescent-colored
gloves. Signers wore black clothes against a white wall’s backdrop for more
accurate hand extraction.

Downloaded videos. Another way to collect samples is to download from news or
educational video sources. It has the advantage of correctly matching video
and signs since sign language experts checked the content. For the WLASL
dataset, the authors [16] chose multiple education SL websites as suitable
video sources. The authors filtered samples by signs and leaved videos
containing words only. Such meta information as a temporal boundary
(manually), body bounding box (by YOLOv3), signer identifiers (by FaceNet with
a manual check), and sign dialect/variation annotations (manually) were given
in the WLASL dataset. Annotators of sign dialects were not native sign
languages: they received training to understand SL specifics and, with a
designed interface, compare signs from two videos displayed simultaneously.
Samples for the MS-ASL dataset [12] were downloaded from video-sharing
platforms to communicate and study ASL. Since videos are recorded and uploaded
by ASL students and teachers, they differ by background, lighting,
positioning, and dialect. Such platforms accompany the video with subtitles,
which authors processed by OCR. Face detection and recognition are integral
parts of sample preprocessing in cases where videos were taken from websites,
and the authors included them in their dataset creation pipeline. After,
manual video trimming was applied to separate signs from pauses between them.

Kind of crowdsourcing. Mukushev, Medet, et al. [19] choose a more complicated
but effective way to collect videos for SLR. Their dataset, FluentSigners-50,
was created with six professional SL interpreters. They developed a sign
basket including commonly useful phrases and sentences in the deaf community.
Other signers were invited by interpreters and use SL daily, and the
subsequent distribution of signers by SL use was obtained: 32 deaf, 6 hard of
hearing, 3 hearing SODA (Sibling of a Deaf Adult), and 9 hearing CODA (Child
of Deaf Adults). They used instructions and templates from interpreters to
repeat the KRSL sentences.

##  3 Dataset Creation

The following part provides details about our data collection pipeline. It
consists of 3 main stages: (1) video collection, (2) video validation, and (3)
and gesture time interval annotation. We used two crowdsourcing platforms:
Yandex Toloka333https://platform.toloka.ai/ for data mining and ABC
Elementary444https://elementary.activebc.ru for the validation and the
annotation so that different users are involved in recording and verifying
videos. In addition, before each stage, crowdworkers must pass a mandatory RSL
exam555RSL exam aims to reveal the knowledge of the language, as it has
specific features. with a score of at least 80% before being granted access to
assignments (see Figure 2). These two nuances allow us to get a better and
unbiased assessment of the correctness of the videos.

Figure 2: Crowdsourcing pipeline: collection, validation, and annotation. Each
stage used its own rules, but the exam was the same.

1\. Video Collection. The essential part of dataset creation started with
designing a sign basket. We paid attention to choosing frequently used words
and sentences in everyday life, and in the end, it turned out to be 1,000
glosses. We chose words related to commonly used topics such as food, animals,
emotions, colors, time, etc. After, we asked the crowdworkers from Yandex
Toloka to record a short video of themselves singing a specific word in RSL
based on the provided template. Video templates were taken from SpreadTheSign
website666https://www.spreadthesign.com/ru.ru/search/, a project of the
European Sign Language Center association.

2\. Video Validation. Correctly signing the gesture can be challenging for
people not fluent in sign language, so we added the validation stage on the
ABC Elementary platform in our dataset-creating pipeline. Workers were asked
to check if the gesture was performed correctly. Each video was checked at
least by three different workers. If they disagree, another marker
participated in the validation of such a video, so up to 5 markers on the
video could be repeated. If most workers mark the video as invalid, it is
rejected; otherwise, it is accepted and passed to the next stage. After the
validation, we left videos with a short edge of at least 720 pixels and
converted them to a 30 FPS rate.

3\. Time Interval Annotation. Collected videos may contain uninformative
frames at the beginning and the end of the video, where workers turn the
camera on and off and prepare to show the gesture. Therefore, annotating the
gesture’s start and end time on the video is necessary. The crowdworkers from
ABC Elementary were asked to indicate the time interval with a gesture. Since
our dataset contains glosses and phrases, some videos may have several
gestures. In this case, workers should tag them together as a full gloss. Each
video was annotated by three different crowdworkers.

We developed the aggregation algorithm to get the average over the responses
time interval (Figure 3). First, we split the beginning and end timestamps
into different groups and then independently calculated distances between all
points in each group. Then, if the maximum distance is less than 30 frames, we
find the average value of each group and assume them to be the final pair
(begin, end). Otherwise, video with annotations that do not fit this condition
was not taken into the dataset.

After cutting off the gestures, we had the cuts at the beginning and the end
of the video where "no gesture" is shown, and we decided to use them as zero-
class objects in training to predict the absence of action.

Figure 3: Time intervals aggregation pipeline. At first, we collected from 3
to 5 time intervals and created two lists with beginning and end timesteps
from each. After, if the maximum distance between points in each list is less
than 30 frames, we averaged points of start and end. Otherwise, the
annotations were rejected.

Dataset Post-processing. While reviewing the collected videos, we noticed that
some users show gestures significantly slower than others. This circumstance
leads to inhomogeneous video length within the same class: the duration of the
same gesture varied by more than five times, complicating the classification
of such data. To make our dataset more homogenous, we decided to calculate the
distribution of video lengths for each class and speed up those videos that
are slower than the average value by more than 30 frames. As a result, 347
videos from 270 classes were sped up by an average of 1.7 times.

##  4 Dataset Description

Dataset Content. Our dataset is approximately 16 GB – it contains 20,400
videos of 1,001 classes representing frequently used glosses and short phrases
in Russian Sign Language, including alphabet and numerals. The dataset does
not include fingerspelling words, i.e., words spelled letter by letter using
dactylology. In addition, our dataset includes 400 samples of a special “no
event” class where the subject is not signing any gestures. About 86% of the
videos are oriented vertically, 13% are oriented horizontally, and 1% are in
square format. To the best of our knowledge, 194 crowdworkers participated in
the video recording for our dataset, making it the most subject-diverse RSL
dataset and the second among all sign language datasets (see Table 1 for more
details). The dataset was collected mainly indoors and varied in scenes and
lighting conditions.

Video Quality. The videos were recorded primarily in HD and FullHD formats
(Figure 4d). The number of frames distribution is also shown in Figure 4a, the
average video length is 1.67 seconds, and the overall duration of the dataset
is about 19.81 hours.

Data Splitting. The data was split into training (75%) and test (25%) sets,
containing 15 and 5 video samples for each class, respectively. The numbers of
subjects in training and test sets are equal 112 and 174, respectively. Note
that groups of subjects in these two sets intersect; however, we tried to
minimize it by filling out the test set with inactive users (see Figure 4b-c).

Figure 4: Video length, resolution and user’s splitting analysis. (a) Videos’
number of frames distribution divided into sets, (b) distribution of recorded
video by users in train, and (c) test, (d) video resolution ratio.

##  5 Experiments

Models. Addressing the challenge of recognizing sign language necessitates the
utilization of formidable and expansive models endowed with the capacity to
analyze video data. We utilized a group of 3 pre-trained on Kinetics dataset
models: ResNet3D-50 as a baseline, Swin-large [18] both with I3D [6] head, and
MViTv2-small [17] with MViT-Head. The SLR task requires lightweight models for
real-time use; however, Swin-small showed poor results and was replaced by
Swin-large. The MViTv2 and Swin demonstrate state-of-the-art performance in
various image recognition benchmarks and can accurately analyze video input.

Data Pre-processing. The samples were resized to (300, 300) first; after,
augmentations, different for each model, were applied to the resized samples.
MViTv2-small and Swin-large trained with a horizontal flip and sharpness
augmentations, whereas ResNet3D – with the same horizontal flip, salt random
noise, and color jitter. Horizontal flip augmentation is used to bring the
data distribution to the real because RSL signs do not change the meaning of
mirror reflection. Finally, the videos were padded to (300, 300) and randomly
cropped to 224.

Implementation Details. Several sampling strategies were tested with a
different number of frames from [16, 32, 48] and a frame interval from 1 to 4.
We also checked models trained on 64 frames, but it generated poor results.
Frame intervals are limited to 4 because skipping more frames in the SLR task
can miss important information about the sign. We trained all 36 models over
120 epochs with a learning rate of 0.001, AdamW optimizer employing betas
(0.9, 0.999), and weight decay of 0.05. Two schedulers – LinearLR from 0 to 10
epochs and CosineAnnealingLR from begin until the end of training – were used
to optimize the Swin-large training process. Only LinearLR from 0 to 120
epochs were used for MViTv2-small.

Figure 5: Mean accuracy is achieved by each model on the Slovo with different
sampling strategies. Note that the graphs have various scales depending on the
order of the metrics.

Results. Since each gesture corresponds to 20 video samples, we validated the
models on the test set. Figure 5 shows the results of each chosen model
separately with a different number of frames and frame interval. We can
observe that MViTv2-small vastly outperforms other models due to its video
purpose. Thirty-two frames with a frame interval of 2 are enough for
MViTv2-small to be most effective.

##  6 Discussion and Future Work

Shortly, we plan to expand our dataset: increase the number of classes and the
number of examples per class and collect not only words but also phrases,
n-grams, and sentences. Furthermore, since SL is more expressive than regular
ones: not only hand gestures are used to indicate words, but also facial
expressions, articulation, and body posture, multimodal models can potentially
improve SLR results. For example, authors from [21] mentioned that some words
might have the same hand gesture and differ only in facial expressions. In
addition, we annotated our samples with MediaPipe [3] to see if this makes it
easier to recognize signs.

There are two promising directions for future research: text-based sign
language generation and speech synthesizing. We can achieve the first task by
training a generative video model using RSL datasets. This model will
translate Russian text into a sign language speech video. The second has the
potential facilitating interaction between SLR service and users.

Finally, the most global future challenge is to create a multilingual sign
language model which can help deaf people all over the world communicate
easily and share their experiences. It could be an essential step towards
breaking the social isolation of the deaf community and making the world more
inclusive.

##  7 Conclusion

In this paper, we proposed a new Russian Sign Language dataset Slovo and a
pipeline for creating diverse video data despite a specific domain. The Slovo
is divided into 1,000 classes, each corresponding to 20 videos from 194
signers. It can favorably influence the development of sign language
recognition in the Russian domain. Besides, several models were trained and
evaluated on Slovo to demonstrate the learning ability of the dataset. We plan
to expand the dataset by adding new words and sentences in future versions.
The current dataset, pre-trained models, and demo are publicly available in
the repository.

## References

  * [1] Bu asl corpus as elan. http://asl.cs.depaul.edu/corpus/index.html
  * [2] The Importance of Sign Language for Deaf Education and Sign Technology. https://inlnk.ru/RjpjRB (2012)
  * [3] Mediapipe hands. https://solutions.mediapipe.dev/hands (2019)
  * [4] Google - American Sign Language Fingerspelling Recognition. https://www.kaggle.com/competitions/asl-fingerspelling (2023)
  * [5] Google - Isolated Sign Language Recognition. https://www.kaggle.com/competitions/asl-signs (2023)
  * [6] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)
  * [7] Dreuw, P., Neidle, C., Athitsos, V., Sclaroff, S., Ney, H.: Benchmark databases for video-based automatic sign language recognition. In: LREC (2008)
  * [8] Efthimiou, E., Fotinea, S.E.: Gslc: creation and annotation of a greek sign language corpus for hci. In: Universal Acess in Human Computer Interaction. Coping with Diversity: 4th International Conference on Universal Access in Human-Computer Interaction, UAHCI 2007, Held as Part of HCI International 2007, Beijing, China, July 22-27, 2007, Proceedings, Part I 4. pp. 657–666. Springer (2007)
  * [9] Grif, M., Elakkiya, R., Prikhodko, A., Bakaev, M., Rajalakshmi, E.: Raspoznavanie recognition of russian and indian sign languages based on machine learning. Analysis and Data Processing Systems 3(83), 53–74 (2021)
  * [10] Gutierrez-Sigut, E., Costello, B., Baus, C., Carreiras, M.: Lse-sign: A lexical database for spanish sign language. Behavior Research Methods 48, 123–137 (2016)
  * [11] Imashev, A., Mukushev, M., Kimmelman, V., Sandygulova, A.: K-rsl: a corpus for linguistic understanding, visual evaluation, and recognition of sign languages. In: Proceedings of the 24th Conference on Computational Natural Language Learning. Association for Computational Linguistics (2020)
  * [12] Joze, H.R.V., Koller, O.: Ms-asl: A large-scale data set and benchmark for understanding american sign language. arXiv preprint arXiv:1812.01053 (2018)
  * [13] Kagirov, I., Ivanko, D., Ryumin, D., Axyonov, A., Karpov, A.: Theruslan: Database of russian sign language. In: Proceedings of the Twelfth Language Resources and Evaluation Conference. pp. 6079–6085 (2020)
  * [14] Kenshimov, C., Buribayev, Z., Amirgaliyev, Y., Ataniyazova, A., Aitimov, A.: Sign language dactyl recognition based on machine learning algorithms. Eastern-European Journal of Enterprise Technologies 4(2), 112 (2021)
  * [15] Koller, O., Forster, J., Ney, H.: Continuous sign language recognition: Towards large vocabulary statistical recognition systems handling multiple signers. Computer Vision and Image Understanding 141, 108–125 (2015)
  * [16] Li, D., Rodriguez, C., Yu, X., Li, H.: Word-level deep sign language recognition from video: A new large-scale dataset and methods comparison. In: Proceedings of the IEEE/CVF winter conference on applications of computer vision. pp. 1459–1469 (2020)
  * [17] Li, Y., Wu, C.Y., Fan, H., Mangalam, K., Xiong, B., Malik, J., Feichtenhofer, C.: Mvitv2: Improved multiscale vision transformers for classification and detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4804–4814 (2022)
  * [18] Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3202–3211 (2022)
  * [19] Mukushev, M., Ubingazhibov, A., Kydyrbekova, A., Imashev, A., Kimmelman, V., Sandygulova, A.: Fluentsigners-50: A signer independent benchmark dataset for sign language processing. Plos one 17(9), e0273649 (2022)
  * [20] Ronchetti, F., Quiroga, F., Estrebou, C.A., Lanzarini, L.C., Rosete, A.: Lsa64: an argentinian sign language dataset. In: XXII Congreso Argentino de Ciencias de la Computación (CACIC 2016). (2016)
  * [21] Sincan, O.M., Keles, H.Y.: Autsl: A large scale multi-modal turkish sign language dataset and baseline methods. IEEE Access 8, 181340–181355 (2020)


Entropic Associative Memory for Manuscript Symbols

Rafael Morales1, Noé Hernández 2,3, Ricardo Cruz2,4, Victor D. Cruz3, Luis A.
Pineda2,3,*

1 Universidad de Guadalajara, SUV, Guadalajara, 44130, México.

2 Universidad Nacional Autónoma de México (UNAM), IIMAS, México City, 04510,
México.

3 Posgrado en Ciencia e Ingeniería de la Computación, UNAM, México City,
04510, México.

4 Catedrático CONACyT, México City, México.

*<EMAIL_ADDRESS>
## Abstract

Manuscript symbols can be stored, recognized and retrieved from an entropic
digital memory that is associative and distributed but yet declarative; memory
retrieval is a constructive operation, memory cues to objects not contained in
the memory are rejected directly without search, and memory operations can be
performed through parallel computations. Manuscript symbols, both letters and
numerals, are represented in Associative Memory Registers that have an
associated entropy. The memory recognition operation obeys an entropy trade-
off between precision and recall, and the entropy level impacts on the quality
of the objects recovered through the memory retrieval operation. The present
proposal is contrasted in several dimensions with neural networks models of
associative memory. We discuss the operational characteristics of the entropic
associative memory for retrieving objects with both complete and incomplete
information, such as severe occlusions. The experiments reported in this paper
add evidence on the potential of this framework for developing practical
applications and computational models of natural memory.

##  1 Functional Description

In this paper we present a set of experiments for registering, recognizing and
recovering representations of calligraphic letters, both capital and lower
case, and numerals, using the Entropic Associative Memory (EAM), with
satisfactory results. These experiments add evidence on the viability of the
EAM model and its associated theory for developing computational models of
natural memories as well as practical applications. The formal specification
of the EAM system is introduced in our previous work [1]. Here we provide an
intuitive description to make the present paper self-contained. An EAM system
contains a set of Associative Memory Registers (AMRs) consisting of standard
tables, where columns are interpreted as the attributes or characteristics of
the stored objects, the rows as their potential discrete values, and marked
cells at the intersections specify the values of the attributes. Individual
objects are represented as functions from attributes to values, and classes
are represented through the superposition in the table of the functions
representing the individuals in the class. Hence, classes are relations from
attributes to values. Here we refer to the attributes in the columns and the
values in the rows as the _arguments_ and the _values_ respectively.

Each AMR of size $n\times m$ has an associated auxiliary register defined as a
table of the same dimension that is used to place the memory cue for the
memory register and recognition operations, designated here as $\lambda$ and
$\eta$, and also the objects extracted from the AMR by the memory retrieval
operation, designated as $\beta$. The $\lambda$-operation is defined as the
logical inclusive disjunction between the value of each cell of the auxiliary
register, and the value of the corresponding cell of the AMR, for all cells,
and writing up the result on the AMR. The operation can be interpreted
diagrammatically as overlapping the content of the auxiliary register –the
cue– on the AMR proper –the memory. Figure 1 (a) shows the input of an object
on an empty AMR and Figure 1 (b) illustrates the union of two functions
producing a distributed representation which includes the functions input
explicitly —i.e.,
$\\{(a_{1},v_{1}),(a_{2},v_{2}),(a_{3},v_{4}),(a_{4},v_{7})\\}$ and
$\\{(a_{1},v_{3}),(a_{2},v_{2}),(a_{3},v_{6}),(a_{4},v_{7})\\}$— and two
additional functions which are produced as collateral effects of the
$\lambda$-operation —i.e.,
$\\{(a_{1},v_{1}),(a_{2},v_{2}),(a_{3},v_{6}),(a_{4},v_{7})\\}$ and
$\\{(a_{1},v_{3}),(a_{2},v_{2}),(a_{3},v_{4}),(a_{4},v_{7})\\}$. The novel
objects can be seen as the gain or generalization of the $\lambda$-operation.
These representations are distributed because the relation between the cells
and the represented objects are _many-to-many_ ; i.e., each marked cell can
contribute to the representation of different units of content –the
represented objects– and each represented objects can share memory cells with
other represented objects [2].

Fig 1: Memory register operation

The $\eta$-operation is defined as the logical material implication between
the corresponding cells of the auxiliary register and the AMR, the antecedent
and the consequent, respectively. This operation is interpreted as the
inclusion of the cue in the AMR, and fails whenever the auxiliary register has
at least one cell _on_ but its corresponding cell in the AMR is _off_. This
is, if the cue is included in the memory, the value of $\eta$ is true and
false otherwise, as shown in Figures 2 (a) and (b), respectively.

Fig 2: Memory recognition operation

The $\beta$-operation, in turn, extracts the representation of an object if
the $\eta$-operation is successful for the given cue. To this effect, a value
for each attribute is selected randomly out of the set of values that are on
in the corresponding column using a triangular probability distribution with
the cue as mode. In the present experiment the lower $v_{l}$ and upper $v_{u}$
values are chosen such that the range $[v_{l},v_{u}]$ is the group of values
set on in the surroundings of the cue —that is, either $l=1$ or the value
$v_{l-1}$ is off in the column, and $u=m$ or the value $v_{u+1}$ is off in the
column. In the basic case, the object retrieved is the cue exactly, as shown
in Figure 3 (a); however, the retrieved object may be a previously stored
object that is associated to the cue, or a novel object constructed on the
basis of the cue, as illustrated in Figures 3 (b) and (c), respectively.

Fig 3: Memory retrieval operation

AMRs have an associated entropy, which is defined as the average indeterminacy
of the distributed representation that it holds. Let $\mu_{i}$ be the number
of values assigned to the argument $a_{i}$ in the relation $r$ held in an AMR;
let $\nu_{i}$ = $1/\mu_{i}$ and $n$ the number of arguments in the relation’s
domain. In case there are columns that have no marks, i.e., the relation is
partial, we define $\nu_{i}=1$ for all $a_{i}$ that has no value assigned in
$r$, as such a fact is fully determined. The _computational entropy_ $e(r)$
–or the entropy of a relation– is defined here as:

$e(r)=-\frac{1}{n}\sum_{i=1}^{n}\log_{2}(\nu_{i}).$

The representation containing a single function is fully determined, hence its
entropy is zero.

The number of functions or patterns stored in an AMR of size $n\times m$ with
entropy $e$ is $(2^{e})^{n}$ or $2^{en}$, out of the maximum AMR’s capacity of
$m^{n}$. For instance, the entropy of the abstraction produced through the
$\lambda$-operation in Figure 1 (b) is $e=1/2$; the number of stored
functions, including the two input explicitly and the ones that are produced
as side effects of the $\lambda$-operation, is $2^{(1/2)4}=4$, out of the
maximum capacity of $7^{4}=2401$. The novel functions represent potential
reconstructions that depend on the interactions between stored objects. The
space of potential objects is huge and can play a role in imagination and
creativity.

##  2 Analysis and Synthesis

Modality specific images are input and stored in modality specific buffers,
such as standard pixels buffers for storing pictures. In the present model
such concrete representations are translated into abstract amodal
representations in the form of attribute-value structures or functions, which
are the objects registered, recognized and retrieved in AMRs. The concrete and
abstract representations of the same object stand in a _one-to-one_ relation
to each other; the memory regions allocating these objects are mutually
exclusive and constitute _local_ representations[2]. However, when such
individual objects are input into AMRs, their representations are overlapped,
their identity is left indeterminate and the relation between the memory units
–i.e., the cells in the table– and the corresponding functions or units of
content is _many-to-many_ , and such representations are _distributed_[2].

The use of the present associative memory system requires an analysis module
for mapping concrete into abstracts representations, for memory register and
recognition, and a synthesis module for performing the inverse mapping. The
bidirectional mapping is performed through an autoencoder[3, 4] including an
encoder and a decoder[5]. These two components are modular –independent– and
implement the analysis and synthesis modules, respectively, as illustrated in
Figure 4.

Fig 4: Training the Analysis and Synthesis modules

The encoder in the experiments presented below in Section 3 is implemented
with a VGG5-like neural network with ten convolutional layers [6], and the
decoder with a transposed convolutional neural network with four layers. The
classifier is a fully connected neural network (FCNN) with two layers, mapping
sets of features output by the encoder into their corresponding $c$ classes,
and its purpose is to moderate the autencoder’s training process, so that the
abstract representations are satisfactory both for classification and decoding
purposes. The architectures for the neural networks were chosen so that they
were simple but functional.

The input to the encoder consists of $p$ features, that correspond to the
information in the input buffer, as illustrated in Figure 4. The encoder maps
such concrete representation into $n$ outputs, which constitutes the input to
the AMRs directly –i.e., a function of $n$ arguments with their corresponding
values. This set of features is also the input to the classifier, which
associates one of $c$ categories to its input, and the decoder, that computes
an approximation of the inverse function of the encoder, and renders $p$
features, regenerating the concrete image. The encoder, the decoder and the
classifier are trained simultaneously in a supervised manner by standard back-
propagation, and the latter is removed once the autoencoder has been trained.
Autoencoders were originally proposed to reduce the dimensionality of the
data[3, 4], and the present use constitutes a novel application of such
technology.

The $n$ outputs of the encoder are floating point values which are converted
into integers in the range $[1,m]$ by rounding the results of a linear
transformation, producing the discrete value of the corresponding argument.
The resulting discrete function is input into the corresponding AMR in the
memory register operation or is used as the cue to the memory recognition or
retrieval operation. There are $c$ associative memories, one per class in the
dataset. If the cue is accepted the corresponding object is retrieved as well
as its class; otherwise a conventional rejection code is returned with the
class _unknown_. If the cue is accepted by more than one AMR the class with
the smallest entropy is selected. This operation is performed by the _entropy-
based filter_.

##  3 A Visual Memory for Hand Written Manuscript Symbols

The associative memory system was tested in previous work[1] through the
construction of a visual memory for storing and retrieving distributed
representations of hand written digits from “0” to “9”. The system was built
and tested using the MNIST data-set available at
http://yann.lecun.com/exdb/mnist/. In the present work we add evidence of the
potential of the framework for the construction of practical applications. We
use now the EMNIST [7] database including manuscript capital and lower case
letters, and also the ten digits, increasing the number of classes in relation
to MNIST from 10 to 62; however, capital and lower case letters with very
similar visual shapes are further merged, rendering 47 classes. Here we define
the alphabet EMNIST-47, as shown in Figure 5. As can be seen, there are eleven
lower case letters that can be clearly distinguished from their upper case
counterparts –i.e., classes 36 to 46– and have independent entries. For the
experiment 1 described in Section 3.1 we used the EMNIST Balanced segment
including $2,800$ instances of each class.

Fig 5: EMNIST-47 Alphabet

In previous work we also showed that an Associative Memory Register can hold
the representation of more than one class and yet the system has a
satisfactory performance. Here we capitalize such functionality and also model
a memory in which capital and lower case letters of the same type that have
very different visual shapes are held in the same memory register. As a
result, classes 36 to 46 of EMNIST-47 are dropped, and the total number of
classes is reduced to 36. This second alphabet is referred to as EMNIST-36,
and it is shown in Figure 6.

Fig 6: EMNIST-36 Alphabet

The corpus was partitioned for the experiments in both settings in three
disjoint sets. The partitions were rotated through a standard 10-fold cross-
validation procedure. The partitions names and the assigned amount of data are
as follows:

  * •

Training Corpus (_TrainCorpus_): For training the analysis and synthesis
modules (57 %).

  * •

Remembered Corpus (_RemCorpus_): For filling in the Associative Memory
Registers (33 %).

  * •

Test Corpus (_TestCorpus_): For testing (10 %).

The functional neural network architectures and the optimal value for the
number of inputs to the AMRs –i.e., the parameter $n$– was determined using
the training corpus through preliminary experiments. The integer powers of 2
were explored and $n$ was set to $64$ for all the experiments. Next the
training corpus was used for training the neural networks –the encoder, the
decoder, and the classifier, assembled into a single neural network with one
input and two output channels– for all the tend folds. Then the encoder was
used to process the remembered and test corpora to produce the corresponding
sets of abstract representations. These were in turn input to the
corresponding AMRs to perform the experiments.

Developing on our previous work, the entropic associative memory for
manuscript symbols was tested with four experiments as follows:

  1. 1.

Experiment 1: Define an associative memory system including an AMR for holding
the distributed representation of each one of the forty seven manuscript
symbols of EMNIST-47. Determine the recognition precision and recall of the
individual AMRs, and of the overall system, for AMRs of size $n\times 2^{m}$.
The experiment was performed for $0<=m<=9$. Identify the parameter $m$ of the
AMRs with satisfactory performance. Finally, determine the precision and
recall of the memory recognition when the AMRs contain different amounts of
remembered instances and, consequently, different levels of entropy.

  2. 2.

Experiment 2: Determine the value of $m$ of AMRs holding the distributed
representation of capital and lower case letters of EMNIST-36. Determine the
recognition precision and recall of the individual AMRs and of the overall
system, for $0<=m<=9$ as before. Determine the precision and recall of the
memory recognition when the AMRs contain different levels of entropy, as in
experiment 1.

  3. 3.

Experiment 3: Retrieve objects out of a cue for different levels of entropy
and generate their corresponding images –with the best AMRs found in
experiment 1. Assess the similarity between the cue and the recovered object
at different levels of entropy.

  4. 4.

Experiment 4: Retrieve letters and digits of significantly occluded objects
with the same AMR size used in Experiment 3. Assess the precision and recall
of the memory retrieval operation and the quality of the generated images.

The source code for replicating the experiments, including the detailed
results and the specifications of the hardware used, are available in Github
at https://github.com/eam-experiments/EMNIST.

###  3.1 Experiment 1

Compute the characteristics of AMR of size $64\times 2^{m}$ for $0\leq m\leq
9$:

  1. 1.

Register the totality of _RemCorpus_ in their corresponding register through
the $Memory\\_Register$ operation;

  2. 2.

Test the recognition performance of all the instances of the test corpus
through the $Memory\\_Recognize$ operation;

  3. 3.

Compute the average precision, recall and entropy of individual memories.

  4. 4.

For each instance of the test corpus recover a unique object by the
$Memory\\_Retrieve$ operation or reject the cue; compute the average precision
and recall of the integrated system when this choice has been made.

  5. 5.

Select the parameter $m$ with the best trade-off between precision and recall.
Determine the performance of the system with such memory size $n\times 2^{m}$,
for different amounts of the _RemCorpus_ and entropy levels.

The average precision, recall, and entropy of the AMRs, across the ten-fold
cross-validation experiment is shown in Figure 7 (a). Precision and recall are
computed per AMR in the standard way as follows: $Precision=TP/(TP+FP)$;
$Recall=TP/(TP+FN)$, where $TP$, $FP$ and $FN$ stand for true positives, false
positives and false negatives, respectively. As can be seen, precision is very
low for small values of $m$ but grows with the number of rows, i.e., $2^{m}$,
but recall is very high for low values of $m$, as the information is confused
when the number of rows is very low, and most instances are accepted. However,
when the value of $m$ is increased, the grid is made finer, true instances are
missed, and the recall is lowered. The optimal value of $m$ is a trade-off
between the precision and recall graphs. Figure 7 (a) shows that there is a
good compromise at $m=6$ and $m=7$, i.e., for $64$ and $128$ rows. The figure
also shows the entropy at the bottom bar. As can be seen its value is
increased almost linearly with the AMRs size, starting from $0$ for $1$ row,
and is maximal for $512$ rows.

Fig 7: Results of Experiment 1 using EMNIST-47

Figure 7 (b) shows the precision and the recall as a function of the number of
rows of the system as a whole. In this case if an instance is rejected by all
AMRs, it counts as a false negative for the memory system, and lowers the
total recall. In case the instance is accepted by at least one AMR, the one
with the lowest entropy is chosen; if it is of the wrong class, it is a false
negative for the right class and a false positive for the accepting class, and
increases by one the count of both the false positives and false negatives of
the system as a whole. Figure 7 (b) shows that precision and recall have a
very similar pattern, as both grow from a very low to a higher value according
to the increase of the number of rows. However, when the grid is too fine,
true instances are reject and recall starts to decrease as before. Figures 7
(a) and (b) are coherent, and both show that there is a good compromise at
$64$ and $128$ rows. Figure 7 (c) shows the average number of accepting AMRs
for each instance per AMR size. As can be seen, this number goes from $47$ for
AMRs with one row to $1$ for AMRs with 32 or more rows. This effect is further
illustrated in Figure 7 (d).

The overall purpose of experiment 1 was to investigate the performance of AMRs
with satisfactory operational characteristics in relation to its entropy or
information content. These are the AMRs with sizes $64\times 64$ and $64\times
128$. Figures 7 (e) and 7 (g) show the respective average performance of these
AMRs when they are filled up with varying proportions of the _RemCorpus_ –1%,
2%, 4%, 8%, 16%, 32%, 64% and 100%– using the best AMRs size; and Figures 7
(f) and 7 (h) show the performance of the system as whole for the
corresponding register sizes, as before.

These latter figures show another aspect of the entropy trade-off. If the
number of remembered instances is very low, the entropy is also very low –in
the limiting case, if there is only one instance, the entropy is zero. In this
condition there are very few objects stored in the AMRs, and if the cue to the
memory recognition operation is accepted, precision is very high; however,
even cues that are very similar to the stored objects will be rejected; hence
recall is very low. Precision decreases slightly with the increase of
remembered information and the consequent entropy increase, but recall grows
quite rapidly, until a satisfactory compromise between precision and recall is
reached. However, if the entropy is increased even further the precision
starts to lower but the recall continues to grow. In this case the AMRs are
saturated, most instances are accepted and recall is very high but precision
lowers significantly.

Figure 7 (e) shows that the best trade-off between precision and recall for
AMRs of size $64\times 64$ occurs when the percentage of the remembered corpus
included in the memory is $64\%$ where the corresponding graphs intersect. The
graphs vary slightly for AMRs of size $64\times 128$ where they do not
intersect, and the best performance occurs when the totality of the remembered
corpus is used, as shown in Figure 7 (g). The performance of the system as a
whole, shown respectively in Figures 7 (f) and (h), shows that the precision
remains high when the totality of the remembered corpus is used, but the
recall is lower and the graphs do not intersect. To asses the performance of
the system it also has to be considered the cost of the memory resource which
is doubled for the largest register.

###  3.2 Experiment 2

This experiment shows that an AMR can hold the distributed representation of
objects of different classes, adding evidence on previous work. In this case,
the EMNIST-36 alphabet is used. The procedure is the same used above in
experiment 1. The results are shown in the corresponding graphs in Figure 8.
These show that the performance of the two settings are analogous, with the
only difference that the entropy of the AMRs holding capital and lower case
letters is slightly larger than the entropy of the corresponding AMR holding
only one class.

Experiments 1 and 2 show that the performance of the systems is mostly similar
for AMRs of sizes $64\times 64$ and $64\times 128$; they also show that the
system’s performance is similar for the alphabets EMNIST-47 and EMNIST-36. The
latter is more economical –as it abstracts over lower case and capital letters
and uses only 36 AMRs– hence we use AMRs of size $64\times 64$ with the
EMNIST-36 for investigating further the quality of the memory system in
experiments 3 and 4.

Fig 8: Results of Experiment 2 using EMNIST-36

###  3.3 Experiment 3

This experiment assesses the performance of the memory retrieval operation,
using the configuration and entropy levels in Figures 8 (e) and (f). This
operation is constructive in opposition to photography memories, and all
recovered objects are produced by the $\beta$-operation as explained above in
Section 1 and illustrated in Figure 3. We study three conditions for memory
retrieval, which are illustrated in the tables of Figures 9, 10 and 11. There
is a column for every type of symbol of the EMNIST-36 alphabet. The different
rows show the symbol that is recovered from the memory using the same cue at
the different amounts of corpus and levels of entropy. White cells indicate
that the memory cue was rejected at the corresponding entropy level. The top
row shows the cue to the memory retrieval operation. The second row shows the
symbol that is produced by the autoencoder feeding the output of the encoder
directly into the decoder, i.e., bypassing the memory. As the decoder ideally
computes the inverse function that is computed by the encoder, the symbols in
this row should be exact copies of corresponding cues in the top row. However,
the decoder computes only an approximation and the symbols are slightly
different. The encoder and decoder produced always an object, which is the
most proximate to the cue, as neural networks never reject an object, despite
that the quality of the cue may be very poor; hence the rendered object may be
of a wrong class. The 3rd to the 10th row show the symbols recovered at the
eight different levels of entropy of the _RemCorpus_. We study three
conditions of the memory retrieval operation, as follows:

Fig 9: EMNIST-36 Symbols recovered with high quality cues as a function of the
entropy Fig 10: EMNIST-36 Symbols recovered with moderate quality cues as a
function of the entropy Fig 11: EMNIST-36 Symbols recovered with very poor
cues as a function of the entropy

  1. 1.

The cue to the memory retrieval operation is accepted at the lowest possible
level of entropy. In this condition it is expected that the symbol will be
recovered at all higher levels of entropy too, which it is in fact the case as
shown in Figure 9. The table shows that the recovered object is quite similar
to the cue at most entropy levels, although its quality diminishes slightly at
higher levels. This condition illustrates cues that are recovered with high
precision, but there are very few objects stored in the memory and recall is
very low. Intuitively, the contribution of the cue to the construction of the
retrieved object is very high, hence the cues and the corresponding retrieved
objects are very similar and have “good quality”.

  2. 2.

The cue is accepted at a moderate level of entropy –at about $4\%$ of the
remembered corpus or at an entropy level between $3$ and $4$– but is rejected
at very low levels, as illustrated in Figure 10. In this condition both the
cue and the memory content contribute to the shape of the retrieved object
depending of the entropy level: the higher the entropy the less the impact of
the cue on the shape of constructed object and vice versa. However, as there
are more objects stored in the memory, the entropy and the recall are higher,
and more cues are likely to be recovered.

  3. 3.

The cue is accepted at the highest entropy levels or rejected, as illustrated
in Figure 11. In this condition the cues are far from representative instances
of the class. If accepted, the content of the memory may contribute much more
to the construction of the retrieved object than the cue itself; the
reconstruction may be very noisy; the quality of the recovered objects may be
poor or very poor; and the cue and the retrieved object may be quite
different. Precision is very low and the retrieved object may be of the wrong
class.

###  3.4 Experiment 4

In this experiment, we investigated the memory retrieval operation using
severely occluded cues. In this condition, cues should be rejected at most
entropy levels, as they are very different of the corresponding stored
objects. This is reflected in the recall of the $\eta$-operation, and
consequently of the $\beta$-operation, which is very low. However, there may
be situations in which recovering tentative objects may be useful if they can
be further processed in relation to contextual information, and the right
object may still be recovered, as is common in visual interpretation. Despite
severe occlusions, the visible part of the cue has some amount of structure
that is reflected in its abstract representation, and recognition failure may
be due to the rejection of a small number of features, as shown in our
previous work [1]. The definition of the $\eta$-operation states that all $64$
features of the abstract representation of the cue must be included in the AMR
of the corresponding class, which is a very strong condition. A means to
increase the recall is to relax the test and allow that memory retrieval is
successful if a small number of features may fail.

We tested the memory retrieval operation when the bottom-half of the
manuscript symbols were occluded, and also when the symbols were occluded by
horizontal bars covering more than half of the total area in which the symbol
is placed. The results are shown in Figures 12 and 13, respectively. Figures
(a) at the top in both occlusion conditions show the objects recovered with no
tolerance at all levels of entropy, and Figures (b), (c) and (d) show the
relaxation of $1$, $2$ and $3$ out of the $64$ total features, respectively.
As expected, the recall is increased but the precision is lowered according to
the amount of relaxation. As the recall is very low in all these conditions
only a small amount of symbols are recovered. In the none-relaxation condition
most cells are blanks at low entropy levels and there are columns with only
blank cells. The relaxation of one feature shows that objects are recovered at
lower levels of entropy, and that the blanks are filled-up from bottom to top;
this tendency is continued with the relaxation of two and three features as
shown in (c) and (d) in both figures, respectively.

Fig 12: Memory retrieval with occlusions of $50\%$ at the bottom with
relaxations of $1$, $2$ and $3$ features out of $64$ Fig 13: Memory retrieval
occluding the cues with horizontal bars and relaxing $1$, $2$ and $3$ features
out of $64$

##  4 Experimental Setting

The programming for all the experiments was carried out in Python 3.8 on the
Anaconda distribution. The neural networks were implemented with TensorFlow
GPU 2.4.1, and most of the graphs were produced using Matplotlib. The
experiments were run on an Alienware Aurora R5 with an Intel Core i7-6700
Processor, 64 GBytes of RAM and an NVIDIA GeForce GTX 1080 graphics card.

##  5 EAM versus Neural Networks models of Associative Memory

Figures 1 to 3 illustrate that EAM resembles natural memories[8, 9] in a
number of putative properties, and diverges from the proposals developed
within the neural networks paradigm [10, 11, 12, 13, 14, 15, 16], that were
consolidated with Hopfield’s model[17], and subsequent work [18, 19, 20, 21,
22, 23]. The basic difference is that we conceive a memory as a functional
system using a declarative representational format in which recollections are
registered, recognized and recovered on the basis of a cue, which corresponds
to the intuitive notion of “remembering”, in opposition to neural networks
models that conceive memory as a dynamical system in which input patterns are
used to “train” a neural network in a “learning stage”; such that the stored
patterns can be recovered out of complete or incomplete input patterns in a
“use” or “test stage”. Although all neural networks models use numerical
matrix representations, they can be divided intuitively into those that focus
on the memory as a dynamical system, as Hopfield’s[17] and Kosko’s[18]
original models; those that focus on the algebraic properties of the matrices
representing such networks, such as morphological models[19, 20, 24] and
related work[25]; and those inspired on natural neurons directly, such as
dendritic associative memories[26]. The difference between the present
proposal and the neural networks models shows up in several specific
characteristics of the systems, as elaborated below.

###  5.1 Representational format

AMRs in the EAM model hold distributed representations that are produced out
of the logical inclusive disjunction of discrete finite functions representing
individual objects in the table format, and conform directly to the criterion
stated by Hinton[2] for distributed representations, i.e., that units of
memory stand in a _many-to-many_ relation to the units of content, as opposed
to local representations where such relation is _one-to-one_. This is, it is
possible to tell explicitly which units of memory contribute to the
representation of an individual object, and which memory cells are shared by a
number of units of content or represented objects. The representations stored
in the AMRs are manipulated directly by the memory register, memory
recognition and memory retrieval operations, and the representation is
declarative. This contrasts with memories developed in the neural networks
paradigm, in which the memories are numerical matrices of weights produced
through standard matrix operations; and such representations are better
thought of as sub-symbolic or non-declarative.

###  5.2 Productivity

A salient property of distributed representations is that they “contain” not
only the objects used in the training process, but also additional similar
objects that are produced as a side-effect of the operations through which the
representation is created[2]. This generalization property is essential to
neural networks devices, such as classifiers trained with standard back-
propagation, that are constructed with a training corpus that should be
different from the testing corpus, and can classify unseen inputs in the test
stage. Associative memories within the neural networks paradigm are trained
through a forward process, such that each input pattern should correspond
ideally to a minimum of the energy function associated to the network, and can
be retrieved explicitly. Although there may by additional local minima, these
are considered errors due to side-effects of the training, that should be
avoided. The EAM model, on its part, do produce a potential number of novel
patterns due to the productivity of the $\lambda$-operation, which define the
space of potential reconstructions; hence, has the generalization property,
but in a manner that is coherent with the retrieval of explicitly stored
patterns.

###  5.3 Constructive recovery

Associative memories developed within the neural networks paradigm need to be
provided with the full set of patterns that are codified in the weight matrix,
and the retrieve operation, either on the basis or complete or incomplete
inputs, produces the stored patterns exactly; for this reason, such memories
are reproductive or “photographic”, alike to standard RAM memories, and can be
better considered pattern matching machines. This contrasts with EAM, where
the objects retrieved out of a complete or a partial cue through the
$\beta$-operation are constructions always. A recovered object may be
identical but also similar to the cue, and even an “imaged object”, resembling
better the intuitions reported by Bartlett in his seminal work, and also the
common intuition about remembering objects in natural memory.

###  5.4 Kinds of associations

Neural networks models of associative memory are classified as _auto-
associative_ and _hetero-associative_. The former are systems in which the cue
is the pattern to be recovered itself, either in full or a segment of it, and
possibly with noise. Hence, the images codified and stored in the memory can
be recovered with partial information, e.g., when the image is partially
occluded or noisy. Hopfield’s[17] original model is the paradigmatic case of
auto-associative memories. Hetero-associative memories store a relation
between two patterns, e.g., the pair $(a,b)$, that represent associated or
similar objects, possibly of different classes. In these models, $b$ can be
recovered using the pattern $a$ or a partial or noisy version of it and vice
versa, and the memory is said to be “bidirectional”. A paradigmatic case of
this model is the so-call BAM model[18]. These models require that all the
associations are included in the training set; hence, associations cannot be
established dynamically on the basis of the content of the memory at an
arbitrary state and a novel input. In contrast, the distinction between auto
and hetero associative is not basic in the EAM model. In the case study
presented in Section 3 the auto-associative property is illustrated with the
recovery of calligraphic letters and numerals out of complete and severely
occluded cues; but, in addition, the bidirectional hetero-associative property
is illustrated with the recovery of a capital letter using a lower-case letter
as the cue and vice versa, which are stochastic processes. In EAM all patterns
are input independently and the associations are established dynamically; this
is due to the representational format, the definition of the
$\lambda$-operation, and the stochastic aspect of the $\beta$-operation;
unlike hetero-associative models, in which associations are established
explicitly.

###  5.5 Memory rejection

The EAM model defines an explicit reject operation through the logical
material implication without search, implementing the true or strong negation.
This property is not addressed explicitly in neural networks models of
associative memory. Such models are oriented to recover the pattern that is
most similar to the input. For instance, Hopfield’s and Kosko’s models use a
recurrent search process that converges to the pattern in the memory that is
the most similar to the input pattern. However, if the input is partial the
system may converge to the wrong pattern, and the object recovered would be a
false positive. It is also possible that the process does not converge in a
reasonable amount of time, and the process needs to be interrupted. In this
situation it is possible to return the current hypothesis of the search
process, which may be a false positive; an alternative may be to state that
the pattern was not found, implementing a form of negation as failure[24].
These problems are avoided in systems that do not search and provide the
answer through an analytical process that always terminate. Such is the case
of morphological hetero-associative memories[19, 20, 22] and related work[25]
which are also bidirectional. However, the full set of associations must be
given in advance to train the network, and such systems do not contemplate
that an input may not be included in the memory; hence, whenever this is the
case, provide the most similar object, which is a false positive by necessity.
More generally, the assumption that the input must converge to a known object
presupposes that all of the patterns in the domain of interest are included in
the training set, or that the information about the world is complete, so to
approximate to the most similar object is valid. This conforms to the so-
called Closed-World Assumption (CWA) in its most general form.

The EAM model, on its part, do have an explicit reject capability and
implements the true or strong negation, and assumes incomplete information.
This is an essential property of a natural memory, because its purpose is
precisely to learn or acquire novel information. Natural memory is also
oriented to the interaction with the environment, which makes untenable the
adoption of the CWA, and also that the memory should be trained in full
whenever a new bit of information is acquired.

###  5.6 Parallelism

The three memory operations of EAM can be computed through parallel
computations between corresponding cells of the AMR and its associated
auxiliary register in two or three computing steps, i.e., the operation proper
and updating the AMR and its auxiliary register, if the appropriate hardware
is provided. Furthermore, a cue may be used to select a large number of AMRs
through a BUS simultaneously. Parallel computing in neural networks models, on
its part, is achieved through the parallelization of matrix operations, as any
standard matrix parallel computing, with the corresponding demands of RAM and
time.

###  5.7 Energy function

Neural networks models of associative memory have an associated energy
function which determines the search path from the high to the low-energy
nodes, that correspond to incomplete cues memories and stored patterns,
respectively. The definition of such function is essential for models in the
dynamical system’s paradigm. The EAM model, on its part, does not require of
such notion, because it is not construed as a dynamical system.

###  5.8 Entropy

The EAM model defines the computational entropy, which is a measure of the
indeterminacy of the representation contained in an AMR. Operational AMRs obey
an entropy trade-off such that there is an interval of entropy values, where
the entropy is not too low nor too high, where the $\eta$ and
$\beta$-operations have a good compromise between precision and recall in
relation to a cue, as is shown in the case study in Section 3. The neural
networks models hold fully determined patterns, do not involve a notion of
indeterminacy, and do not use an operational notion of entropy.

###  5.9 Capacity

The storage capacity of Hopfield’s memories is about $0.15$ patterns in
relation to the number of nodes. This has been extended significantly with
approaches that use a more general energy function[27, 19, 20, 21, 24, 23] and
the promise of these systems is that they have a very large storing capacity
using moderate physical memory resources. This is also the case in the present
proposal. For instance, the AMRs of size $64\times 64$ –i.e., 4096 bits– can
hold up to $64^{64}$ patterns. This capacity is modulated by the amount of
information stored in AMRs, which should be not too low and not to high,
obeying the entropy trade-off, that determines their operational capacity. The
number of functions contained in an AMR with an entropy $e$ and $n$ arguments,
including the ones registered explicitly and the ones that are due to the
side-effects of the $\lambda$-operation, is $2^{en}$, as specified in Section
1. For instance, Figure 8 (e) shows that $e=4.5$ for an AMR of size $64\times
64$ filled up with $772$ functions, i.e., an average of $22.62$ cells set on
per column. The number of functions is obtained considering that the AMR
includes $64\%$ of the _RemCorpus_ , which is the $33\%$ of the full EMNIST-36
corpus, although the corpus is not balanced and $772$ is the average number of
functions in the $36$ AMRs. Hence, the AMR includes $22.62^{64}$ functions
representing actual and potential calligraphic symbols, with satisfactory
precision and recall. As can be seen the number of functions input explicitly
is very small in relation to the number of potential objects stored in the
AMR, proving a very large space for the construction of novel objects and
“imagination”.

###  5.10 Quantitative Comparisons

The present discussion shows that there are several qualitative dimensions
that should be considered for comparing the present proposal with associative
memories developed within the neural networks paradigm. One preliminary
consideration is that artificial neural networks are often used to model
functions that differ from memory, such as perception and action. However,
these are often labeled as “memories”, for instance in neuroscience
studies[28]. However, for something to be a memory at the functional level[29]
the objects registered should be stored in a format that allows that such
objects can be retrieved or remembered later. In the present approach we use
neural networks to implement perception and action, but these are different
from the memory itself, both structurally and functionally. Of course, the
performance of the memory system depends on the performance of the networks
implementing the analysis and synthesis modules, but comparisons to asses the
present proposal must be with systems that implement associative memories at
the functional level. In particular comparing the functional with the
algorithmic or implementation level in Mars’s sense[29] is a category mistake.
Hence, neural networks as well as other kinds of machine learning mechanisms
implementing classifiers, filters, controllers, etc., should not be considered
memories. An example is the use of celullar automata to classify the digits of
the MNIST corpus[30]: this system can tell the class of a numeral but cannot
use its full or incomplete image as a cue to retrieving the digit. Conversely,
a memory system can be used as a classifier, as Krotov and Hopfield’s dense
associative memories[27] for classifying the MNIST corpus too, but in such
model, the memory system is used at an algorithmic level to support
classification at the functional level, reversing somehow Marr’s hierarchy of
system levels.

Comparisons also face the problem that most proposals are tested with a corpus
suited for their particular goals; so corpora designed for a particular model
or application cannot be used directly in other settings. In particular we
considered associative memories that use MNIST or EMNIST, such as the spiking
Neural Networks developed by Hu et al.[26]. However, the experiments are
limited to a very small set of cases, and cannot be compared directly with
ours. We also reviewed the survey of systems using MNIST and EMNIST for
handwritten character recognition [31], but associative memories are not even
mentioned, and we were unable to find a single associative memory using the
EMNIST corpus for direct comparisons.

The most similar study that we found was Krotov and Hopfied’s dense
associative memories[27] for classifying the MNIST corpus. We adapted such
approach using the code available at
https://github.com/DimaKrotov/Dense_Associative_Memory to make a quantitative
comparison with EAM, but testing with the EMNIST-36 corpus instead. Dense
associative memories extend Hopfield’s original model using a new energy
function with a much larger number of local minima and a much larger storing
capacity. For their experiments the corpus was partitioned in the training and
testing corpora, which are mutually exclusive, as is done in standard
classifiers. The training corpus was used to train the memory and the test
corpus to asses the classification performance. For our experiment we used the
112,800 and 18,800 instances images of the training and test corpora of
EMNIST-36, respectively; the corresponding sets of MNIST consists of 60,000
and 10,000 instances, respectively. We computed the precision of the training
and test corpora using their methodology, rendering $0.7957$ and $0.7813$,
respectively. These results can be contrasted with our experiment 3.2 in
Figure 8 (f) using the totality of remembered corpus, where precision is
$0.8868$ and recall is $0.8361$, which improves over the figures obtained
using Krotov and Hopfield’s code. However, it should also be considered that
the classification rate for this task using standard classifiers may be much
higher; in particular our classifier in Figure 4 renders an accuracy of $0.9$,
and suggests that an associative memory system is not the best tool for
performing classification.

This comparison should also be placed in the larger perspective of the nature
and assumptions of the two kind of memory systems. Hopfiels’ model is the
paradigmatic model of auto-associative memories, and the memory is
reproductive. Consequently, in the classification exercise, all of the cues in
the test corpus fall into the most proximate local minima but never select the
right object, which is not in the training set, hence in the memory, and
should count as false positives. Conversely, as Hopefiel’s memory always
selects the most proximate object, there are no rejections. EAM, in contrast,
assumes that cues are never contained in the memory exactly, unless the
entropy is very low, the precision very high, and the recall very low, which
is an exceptional situation, and even then the retrieved object is a
construction that may differ slightly from the cue. This highlights that
Hopfield’s memory is a pattern matching machine at the computational or
functional level in Marr’s sense[29] that is used pragmatically as a memory,
while EAM aims to model natural memory at such system level, in a manner
coherent with the algorithmic and implementational levels.

Comparison with other associative memories within the neural networks
paradigm, such as Associative Long-Short Term Memories [32], could be made
–although neither their corpus nor their code are available– but in any case,
considerations similar to the ones made above for dense associative memories,
would have to be taken into account. Most generally, EAM uses a declarative
format holding a distributed representation that is manipulated symbolically,
and differs from the conception of associative memory as a dynamical system
with an energy function, and a direct comparison between these two approaches
constitutes a category mistake.

##  6 Summary and Discussion

The present memory system is associative because the AMRs are accessed through
their contents, which are codified as discrete functions represented in a
tabular format. The concrete representations of manuscript symbols are placed
on modality specific input and output buffers, and these are translated to
their abstract representations, which are input to the AMRs through the
encoder. The output of the AMR is an abstract representation of the same kind
that is fed into the decoder to generate the corresponding concrete
representation.

EAM holds distributed representations because the individual instances of the
stored objects are represented as functions that are overlapped within the
corresponding AMR. Hence the cells of the register’s table can contribute to
the representation of more than one object –all the objects whose
representation share the same value for the same argument– and the
representation of an object –a function– shares memory cells with the
representations of other objects, and the relation between memory units and
their contents is _many-to-many_ in opposition of local representations in
which such relation is _one-to-one_ [2].

The memory register operation is conceptualized as an abstraction between the
representation of the object to be stored and the content of the memory, and
is implemented with the standard inclusive logical disjunction between the
arguments and values of the object to be stored and its corresponding
arguments and values in the memory, which can be construed as the micro-
features of the representations.

The memory recognition operation is conceptualized as the inclusion relation
of the object to be recognized in relation to the content of the memory, and
is implemented through the logical material implication between the
corresponding micro-features of the object to be recognized and the content of
the memory. Failure in the recognition test implements a strong negation
directly, without search.

The memory retrieval operation is conceptualized as a constructive operation
that renders a novel object always. The retrieval operation is conditioned by
the recognition test, and the rendered object is built by selecting randomly
the values associated to all the arguments of the function representing the
retrieved object, using a distribution centered on the memory cue.

The objects of computing in the three memory operations are the micro-features
constituting the representations. The computations can be performed in
parallel in a natural way taking very few computing steps, if the appropriate
hardware is made available.

The memory operations conform to the main properties of human memory that
emerged from the paradigmatic studies carried on by Bartlett[8]; and differ
fundamentally from the corresponding operations in memories developed within
the neural networks paradigm, as was intensively discussed in Section 5.

AMRs hold relations that have a certain amount of indeterminacy. A function is
a fully determined relation and its entropy is zero. The entropy is related to
the number of objects stored in the memory; when there are few, the entropy is
very low, and successful retrieval operations are very precise but recall is
very low; conversely, when the number of stored objects is high, so is the
entropy, and recall may be very high, but precision decreases significantly.
The entropy depends on the interactions between such functions, i.e., the
number of arguments that share the same value for a number of functions: the
greater the interactions the lower of the entropy. More generally, the
operational capacity of AMRs conforms to an entropy trade-off, to the effect
that the objects recovered through the memory retrieval operation in relation
to the memory cue are “photographic reproductions”, “appropriate
reconstructions”, “imaged objects” and noise, according to the increase of the
entropy, from very low to very high, respectively. The trade-off between
precision and recall is satisfactory for moderate entropy values.

The purpose of experiments 1 and 2 was to identify the size of operational
AMRs for the application domain. Our previous work with digits showed that
AMRs with $64$ features –the cardinality of the domain of the stored
functions– offer a good compromise between performance and memory size and,
consequently, cost[1]. Further experiments showed that this choice is
appropriate for the present case study too. Figures 7 and 8 show the
performance of the memory for different sizes or discrete levels of the
functions’ codomains. This choice is also essential for the system performance
and its cost. The tests were performed with ten values of the parameter $m$,
where the number of rows of the AMRs is $2^{m}$. Figures 7 and 8 from (a) to
(d) in experiments 1 and 2 show that there is a good compromise between
precision and recall at $m=6$ and $m=7$ with entropy from $4.6$ to $5.4$ in
both experiments.

Then, we used the test corpus to measure the performance of the system when
the operational registers have different amounts of information. Figures 7 and
8 (e) and (f) show the performance of the AMRs with size $64\times 64$ for the
corpora $EMNIST-47$ and $EMNIST-36$, respectively; and (g) and (h) the
corresponding performance of the AMRs of size $64\times 128$. As can be seen,
a satisfactory performance is achieved with a significant amount of memorized
information.

AMRs can hold the representation of more than one class, with the only penalty
of an increase of the entropy, as shown in our previous work[1]. Here we
capitalize such property to represent capital and lower case letters in the
same AMRs, reducing the number of registers from $47$ to $36$. As can be seen
in the figures, the reduction in the number of classes compensated the entropy
increase of the abstracted classes, and the performance of the system using
the EMNIST-47 and the EMNIST-36 is practically the same. Also, the precision
and recall are very similar when the AMRs are filled up with a substantial
amount of corpus, but considering that the memory cost is twice as much for
registers of larger size i.e., $64\times 128$, we selected the EMNIST-36
corpus and alphabet, with AMRs of size $64\times 64$ for the memory retrieval
experiments.

Although the focus of the experiments presented in this paper is the auto-
associative aspect of EAM, the corpus and the experiments do show hetero-
associations between capital and their corresponding lower-case letter
symbols. However, EAM cannot be compared directly with systems in which the
associations are stated explicitly beforehand, such as BAM and morphological
models, because in the present proposal all inputs are independent, and the
associations are established dynamical as side-effects of the
$\lambda$-operation, the diagrammatic representation of functions and
relations, and the stochastic aspect of the $\beta$-operation. For the moment,
the hetero-associative aspect of EAM is left for further work.

Experiment 3 confirmed our previous memory retrieval experiments using memory
cues that are recovered at very low entropy, moderate entropy but not at lower
entropy, and recovered only at high entropy or not recovered at all, as shown
in Figures 9, 10 and 11, respectively, that confirms the entropy trade-all:
the reproductions of cues recovered at very low entropy have high quality, but
there are very few objects in the memory; cues recovered at moderate entropy
can be reconstructed flexibly and with a reasonable quality; and cues that are
only recovered at high entropy or not recovered at all correspond to objects
of bad quality or that are not included in the memory at all.

However, when the output of the encoder is fed into the decoder directly, an
object is rendered always, independently of its quality. If the cue is poor or
very poor, the shape rendered by the decoder is due to the prototype patterns
codified implicitly in the networks implementing the analysis and synthesis
modules.

Experiment 4 addresses recovering objects with incomplete information. In this
case, with severely occluded objects. This task has been an important
motivation for auto-associative memories. In most realistic situations, where
objects are seen from different orientations and distances, and there is
usually noise, such as occlusions, poor lighting conditions, or impaired
vision, the cue is incomplete, and it has to be approximated to the object
recovered in the memory. If the cue is too poor the retrieved object may be
the wrong one, and the agent would have no means to realize such a failure,
unless the object is subjected to further contextual interpretation.
Furthermore, if the cue is not included in the memory it should be rejected
directly, as in EAM, in opposition to models that adopt the Closed-World
Assumption, and produced the most similar object, which in this case is a
false positive.

Incomplete information of the cue is reflected in the values of a small number
of features of the function representing the object, which are the cause of
the rejection by the recognition test[1]. Hence, our strategy to recover
objects with incomplete cues consists simply of relaxing the recognition test.
Figures 12 and 13 show the performance of the retrieval operation with very
severely occluded objects in two settings –covering the bottom-half of the
figure and placing the figure behind wide bars. These conditions are very hard
to interpret even for people, and constitute a very strong test for memory
systems in general.

The results show another aspect of the entropy trade-off. Most objects
severely occluded are likely to be rejected at low entropy levels, as shown in
the tables (a) in Figures 12 and 13. However, a small relaxation of $1$, $2$
and $3$ features increases the recall proportionally, and objects may be
recovered at lower levels of entropy, as shown in Figures 12 and 13 (b), (c)
and (d), reducing the gap of blanks at low entropy levels. As before, the
price to pay for the increase of the recall is a decrease of the precision,
and the objects recovered may be of the wrong class. Nevertheless, having some
candidate interpretations that can be processed further in relation to a
context is better than having none.

The present research introduces a novel application to autoencoders developed
within the deep-neural networks paradigm. Such systems were introduced to
reduce the number of dimensions of large features spaces, but not to produce
abstract representations of concrete objects and vice versa, as is performed
by the analysis and synthesis modules of the EAM architecture, respectively.
The construction of this architecture requires making such modules
independent, so the output of the encoder and the input to the decoder, which
are the objects of the memory operations, are independent.

Finally, the present experiments show that the entropic associative memory has
a satisfactory performance for storing, recognizing and retrieving manuscript
letters and digits; and can potentially be used in practical applications. The
present theory and experiments also show a novel conception of memory that
differs in several respects from the neural networks paradigm. In particular,
it uses a more transparent notion of distributed representation; makes an
explicit use of the entropy; and shows that memory conforms to the entropy
trade-off. The memory operations use the logical disjunction, the material
implication, and the strong negation directly, which operate on the micro-
features of representations, and can be computed in parallel in a very reduced
number of steps if the appropriate hardware is provided. This suggests the
conjecture that such logical functions have their roots on the basic
operations of associative memory. The random element suggests that
imagination, creativity and free will have their roots, at least in part, in
associative memory too. The memory is constructive as opposed to reproductive,
and resembles better the properties of natural memory within constructivist
approaches to knowledge and learning.

##  7 Acknowledgments

Luis A. Pineda acknowledges the partial support of grant PAPIIT-UNAM IN112819,
Mexico.

## References

  * 1. Pineda LA, Fuentes G, Morales R.  An entropic associative memory.  Scientific Reports. 2021;11:6948.
  * 2. Hinton GE, McClelland JL, Rumelhart DE.  Distributed Representations (Chapter 3).  In: Rumelhart DE, McClelland JL, editors. Parallel Distributed Processing, Explorations in the Microstructure of Cognition, Vol.1: Foundations. Cambridge, Mass.: The MIT Press; 1986.
  * 3. Hinton GE, Salakhutdinov RR.  Reducing the Dimensionality of Data with Neural Networks.  Science. 2006;313(5786):504–507.  doi:10.1126/science.1125249.
  * 4. Masci J, Meier U, Cireşan D, Schmidhuber J.  Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction.  In: Honkela T, Duch W, Girolami M, Kaski S, editors. Artificial Neural Networks and Machine Learning – ICANN 2011. vol. 6791 of Lecture Notes in Computer Science. Springer; 2011. p. 52–59.
  * 5. LeCun Y, Bengio Y, Hinton G.  Deep Learning.  Nature. 2015;521(11):436–444.  doi:10.1038/nature14539.
  * 6. Kweon M. MNIST Competition Tensorflow KR Group; 2017.  Available from: https://github.com/kkweon/mnist-competition.
  * 7. Cohen G, Afshar S, Tapson J, van Schaik A.  EMNIST: an extension of MNIST to handwritten letters.  arXiv preprint arXiv:170205373. 2017;.
  * 8. Bartlett FC.  Remembering: A study in experimental and social psychology.  Cambridge University Press; 1932.
  * 9. Kosslyn SM, Thomson WL, Ganis G.  The Case for Mental Imagery.  Oxford University Press; 2006.
  * 10. Steinbuch K.  Die Lernmatrix.  Kybernetik. 1961;1(1):36–45.
  * 11. Willshaw DJ, Buneman OP, Longuet-Higgins HC.  Non-Holographic Associative Memory.  Nature. 1969;222:960–962.
  * 12. Kohonen T.  Correlation Matrix Memories.  Computers, IEEE Transactions on. 1972;C-21:353 – 359.  doi:https://doi.org/10.1109/TC.1972.5008975.
  * 13. Anderson JR, Bower GH.  Human Associative Memory: A Brief Edition.  Lawrence Erlbaum Associates, Publishers, Hillsdale, New Jersey; 1980.
  * 14. Palm G.  On associative memory.  Biological Cybernetics. 1980;36(1):19–36.  doi:10.1007/BF00337019.
  * 15. Aleksander I, Thomas W, Bowden P.  WISARD, a radical new step forward in image recognition.  Sensor Rev. 1984;4(3):120–124.
  * 16. Aleksander I.  An introduction to neural computing.  London: Chapman and Hall; 1990.
  * 17. Hopfield JJ.  Neural networks and physical systems with emergent collective computational abilities.  Proceedings of the National Academy of Sciences of the USA. 1982;79(8):2554–2558.
  * 18. Kosko B.  Bidirectional associative memories.  IEEE Transactions on Systems, man, and Cybernetics. 1988;18(1):49–60.
  * 19. Ritter GX, Sussner P, DiazdeLeon JL.  Morphological Associative Memories.  IEEE Transaction on Neural Networks. 1998;9(2):281–293.
  * 20. Ritter GX, DiazdeLeon JL, Sussner P.  Morphological Bidirectional Associative Memories.  Neural Networks. 1999;12:851–867.
  * 21. Sussner P, Valle ME.  Implicative Fuzzy Associative Memories.  IEEE Transactions on Fuzzy Systems. 2006;14(6):793–807.
  * 22. Sussner P, Schuster T.  Interval-valued fuzzy morphological associative memories: Some theoretical aspects and applications.  Information Sciences. 2018;438:127–144.  doi:doi.org/10.1016/j.ins.2018.01.042.
  * 23. Ramsauer H, Schäfl B, Lehner J, Seidl P, Widrich M, Adler T, et al.  Hopfield Networks is All You Need.  arXiv preprint. 2020;.
  * 24. Ritter G, Chyzhyk D, Urcid G, Graña M.  A Novel Lattice Associative Memory Based on Dendritic Computing.  vol. 7209; 2012. p. 491–502.
  * 25. Yañez-Marquez C, Lopez-Yañes I, Aldape-Lopez M, Camacho-Nieto O, Argüelles-Cruz AJ, Villuendas-Rey Y.  Theoretical Foundations for the Alpha-Beta Associative Memories: 10 Years of Derived Extensions, Models, and Applications.  Neural Processing Letters. 2018;48:811–847.  doi:https://doi.org/10.1007/s11063-017-9768-2.
  * 26. He H, Shang Y, Yang X, Di Y, Lin J, Zhu Y, et al.  Constructing an Associative Memory System Using Spiking Neural Network.  Frontiers in Neuroscience. 2019;13:650.  doi:10.3389/fnins.2019.00650.
  * 27. Krotov D, Hopfield JJ.  Dense Associative Memory for Pattern Recognition.  In: Advances in Neural Information Processing Systems. vol. 29; 2016. p. 1172–1180.
  * 28. Tan SZK, Du R, Perucho JAU, Chopra SS, Vardhanabhuti V, Lim LW.  Dropout in Neural Networks Simulates the Paradoxical Effects of Deep Brain Stimulation on Memory.  bioRxiv. 2020;doi:10.1101/2020.05.01.073486.
  * 29. Marr D.  Vision: A Computational Investigation into the Human Representation and Processing of Visual Information.  New York, NY, USA: Henry Holt and Co., Inc.; 1982.
  * 30. Benoso B, Yañez-Marquez C.  Alpha-Beta Cellular Automata.  Computacion y Sistemas. 2012;16:471–479.
  * 31. Baldominos A, Saez Y, Isasi P.  A Survey of Handwritten Character Recognition with MNIST and EMNIST.  Applied Sciences. 2019;2019:3169.  doi:10.3390/app9153169.
  * 32. Danihelka I, Wayne G, Uria B, Kalchbrenner N, Graves A.  Associative Long Short-Term Memory.  In: Proceedings of The 33rd International Conference on Machine Learning. PMLR; 2016. p. 1986–1994.