# Representational Tenets for Memory Athletics

Kevin Schmidt1*, Othalia Larue2, Ray Kulhanek2, Dylan Flaute3, Razvan
Veliche4,
Christian Manasseh5, Nelson Dellis, Scott Clouse1, Jared Culbertson1, Steve
Rogers1
1Air Force Research Laboratory 2Parallax Advanced Research 3Jacobs
4Keystone Strategy 5Mobius Logic 0pt
<EMAIL_ADDRESS>

###### Abstract

We describe the current state of world-class memory competitions, including
the methods used to prepare for and compete in memory competitions, based on
the subjective report of World Memory Championship Grandmaster and co-author
Nelson Dellis. We then explore the reported experiences through the lens of
the Simulated, Situated, and Structurally coherent Qualia (S3Q) theory of
consciousness, in order to propose a set of experiments to help further
understand the boundaries of expert memory performance.

##  1 Introduction and Background

Analyzing expert cognitive performance on tasks that involve resource-limited
constraints allows researchers to explore the boundaries of computational
cognitive models. This approach dates back at least to the classic works of
Herbert Simon and Allen Newell, who advanced theories of Artificial
Intelligence (AI) through symbolic information processing models of expertise
(Chase and Simon, 1973; Newell and Rosenbloom, 1981). Newer models of
expertise (as summarized in Gobet et al. (2015)) have provided additional
fidelity and richer explanations, but settings such as memory athletics still
provide opportunities for refining these models and potentially guiding novel
AI representations.

The purpose of this note is twofold. First, this work describes the current
state of world-class memory competitions, based primarily on the subjective
report from World Memory Championship Grandmaster and co-author Nelson Dellis
(ND). We discuss the principal method used to prepare for and compete in
competitions, the Method of Loci, as well as the current neuroscientific
projection of this method onto the multiple memory systems in the brain.
Secondly, this manuscript will explore the phenomenological aspects of this
extreme memory experience through the lens of the Simulated, Situated, and
Structurally coherent Qualia (S3Q) representational framework of consciousness
(Schmidt et al., 2021) and show how this framework suggests novel experiments
to further understand the boundaries of expert memory performance. This paper
is written to be accessible to those in cognitive neuroscience who are
interested in the particular memory mechanisms described, but also to those in
AI who are interested in implementing related systems. Previous work has
demonstrated the value of syntheses of neuroscience and AI. For example,
artificial neural networks have a tendency to forget previously learned
information when learning new tasks (catastrophic forgetting). The multiple
memory systems within the brain help to limit the disruption of existing
structures when learning new information (McClelland et al., 1995);
understanding these systems might therefore inspire new artificial neural
structures that are more resilient to catastrophic forgetting.

###  1.1 Speed Cards

The operating characteristics of these multiple memory systems can be
described by assessing elite memory athletes competing in memory competitions.
“Speed Cards” is one of the most common events in memory competitions, and
will be used as an example throughout this paper. Competitors are given up to
five minutes to memorize a shuffled standard 52-card deck. They are then given
a new deck in factory order that must be placed in the same sequence as the
first deck. If at least one competitor reproduces the deck perfectly, the
winner is the competitor who took the least time to memorize the deck
(reconstruction time is not included). If all decks contain errors, a judge
compares the original and reconstructed deck one card at a time, and the score
is determined by the number of correct cards before the first mistake (World
Memory Sports Council, 2013). Speed Cards world record holder Shijir-Erdene
Bat-Enkh has accomplished a perfect deck in 12.74 seconds. ND, who achieved a
grandmaster ranking at the 2012 World Memory Championships (World Memory
Sports Council, 2022), achieved a time of 40.65 seconds, offering a comparable
case study for understanding how such memory achievements are possible.

###  1.2 Expertise and Multiple Memory Systems

Successful performance in this task is thought to require a complex
interaction between the two types of long-term memory in the human brain. The
first depends on a localized biological structure called the hippocampus,
which is needed to consciously experience past episodes from our lives
(Squire, 1992; Squire and Dede, 2015). This episodic representation can be
investigated through the Simulated, Situated, and Structurally coherent Qualia
(S3Q) representational framework of consciousness, which describes qualia as
the fundamental units composing an internally generated spatiotemporal and
causal sensorimotor world model (simulation) defined by relationships with
other qualia (situatedness) and grounded through awareness of the environment
(structural coherence) (Schmidt et al., 2021). The central tenet of the S3Q
model is a prioritization of stability, consistency, and usefulness of the
representation over fidelity with the environment; the functional utility of
this being the regularization capability of constructing a world model while
constraining representational complexity with mechanisms of parsimony. The S3Q
framework provides a way to express and explore the experiential aspects and
underlying mechanisms of conscious memory by deconstructing the phenomenal
experience and characterizing the foundational representation.

In contrast to the characteristics of the conscious qualia-based memory
representation, there are other forms of memory in the brain which are
evolutionarily older and reflect a principle of neuroplasticity that operates
nonconsciously, whereby the long term memory network comes to reflect the
statistical regularities of the environment (Reber, 2013; Squire, 1992).
Through repeated activation, the landscape of the neural network is adapted to
the regularities of experience to allow more efficient processing, such as
that seen in memory athletes, and much like current machine learning tools.
Nonconscious statistical learning takes place automatically whereas conscious
learning is highly regularized (Schneider and Shiffrin, 1977). The allocation
dynamics of conscious processing resources in the brain are well characterized
by multiple resource theory (Wickens, 2008), but a deeper understanding is
needed of how these limited conscious resources interact with implicit,
nonconscious representations to achieve optimal performance during memory
competitions.

Both of these systems learn in parallel, with the conscious learning system
being important for one-shot learning of object locations, while the mature
implicit neural circuits allow for rapid memory retrieval and fluid pattern-
matching decision making (Chase and Simon, 1973; Klein et al., 2010). For
example, chess masters are exceptionally good at recognizing patterns of
meaningful positions of pieces on a chess board and can instantly store in
memory entire chess board arrangements (Chase and Simon, 1973). In chess, a
sufficiently regular state-space exists such that regular patterns of play
have come to be reflected in experts’ long term memory through chunked
retrieval structures, allowing them to instantly match large knowledge
structures in long-term memory to the patterns on the board (Gobet and Simon,
1998). A pattern matching process thus allows the honed neural representation
of the expert chess player to rapidly connect the positions on the board to an
advantageous play, similar to the performance seen in expert radiological
diagnosis (Kundel and Nodine, 1975; Lesgold et al., 1988; Evans et al., 2013).
Through practice, the sets and repetitions have tuned the neural network in
the brains of experts such that statistical patterns in the arrangement of the
pieces on the chess board are quickly recognized as known patterns in the
expert’s brain (Chase and Simon, 1973), allowing them to efficiently respond
appropriately. These pattern-matching feats of chess masters are resilient to
interruptions, signifying that they are drawing on long-term memory for their
performance gains, and not short-term working memory, which is known to be
affected by interruptions (Charness, 1976). These same processes and effects
are seen with memory athletes in card memorization, as will be described
below.

###  1.3 Cognitive Models of Expertise

Earlier models of expertise emphasized short-term memory limits, proposing
chunking models of learning. These theories estimated the limits of this
working memory store at seven, or more recently, four chunks (Miller, 1956;
Cowan, 2001). Many of these theories explicitly declared that all learning is
acquired through limited capacity short-term memory chunks (Chase and Simon,
1973). Specifically, earlier theories of expertise made the assumption that
there was only one learning system: initially memories are encoded consciously
and these representations are later transformed into automatic processes
(Anderson, 1982). However, this approach is not able to account for the rapid
acquisition of long-term memory seen in expert workers. For example, a case
study of an expert restaurant waiter documented the ability to remember the
spatial layout of a table of customers (always broken down into tables of four
customers maximum), and then chunk the information of their orders spatially
by the customer’s seat at the table, then by food order (e.g., meat,
starches), etc. (Ericsson and Polson, 1988). The expert waiter is still only
able to hold four chunks of information in mind at a time, yet the structure
of that information is nested hierarchically, which is thought to allow
retrieval structures to access and organize many bits of information directly
into long-term memory, all while still being nested within the limited
chunking mechanism.

Without a direct route into long-term memory (i.e., without two systems),
earlier chunking theories could not account for the waiter being able to
remember many orders over an extended time amid distractions. Later models
(e.g., Maddox and Ashby, 2004; Ashby and Maddox, 2005) offer a parallel,
implicit way into the brain which can be described through mechanisms called
_templates_ (Ericsson and Kintsch, 1995) by which pointers in working memory
and retrieval structures in long-term memory are used to interface with chunks
of related information. Thus, specific _slots_ are created that hold typed
variables that can be rapidly written to during learning by experts in their
area of expertise (see the example of Person-Action-Object templates below).
While the working memory pointers are thought to be represented (in part) in
the hippocampus, the structure of the information chunks is thought to be
represented through statistical mechanisms in the neocortex unless in
interaction with the medial temporal lobe learning system. Thus, there are
parallel representations of learned information: one can be developed
nonconsciously and is distributed pervasively throughout the brain through
mechanisms such as Hebbian statistical neuroplasticity, whereas the other
requires the localized medial temporal lobe system to interface with this
underlying neuronal structure to allow for conscious access of the learned
content, through mechanisms such as pattern completion inferencing.

EPAM (Elementary Perceiver and Memoriser (Feigenbaum and Simon, 1984)) and
CHREST (Chunk Hierarchy and REtrieval Structures (Gobet and Jackson, 2002))
are instantiations and implementations of Template Theory in which chunks are
computationally defined as nodes in a discrimination network that act as an
index to long-term memory. Influenced by the earlier EPAM model, an early
theory and cognitive architecture created by Herbert Simon, modeling how
humans learn to recognise patterns, and how learning affects recall and
categorization behavior, CHREST originally modeled chess expertise. Nodes
(chunks) in the discrimination network act as an index to long-term memory.
This discrimination network, a branching tree structure, is modeled as a
hierarchical sequence of perceptual tests which classifies the features and
leads to an existing leaf node in the long-term memory network which returns
the system’s knowledge in connection to the stimulus. During classification,
if a difference is identified in the current discrimination network,
information is added to the node (if there is familiarity) or a new node is
created under the previous parent node. Short-term memory contains references
to information held in long-term memory. Frequently occurring patterns in the
environment allow for chunks to evolve into complex data structures, called
templates: a slotted schema that can contain both variable and fixed
information. Templates support expertise since they can hold variables for the
rapid acquisition and integration of new information into the long-term memory
network.

##  2 Subjective Report

Having provided the neuroscientific basis for current understanding of memory
management in the brain, in this section a subjective report is given
primarily by ND, capturing experiences in both memory training and
participation in memory competitions. The Method of Loci111This technique is
known by several other names, including Memory Palace, Mind Palace, Roman Room
Method, Journey Method, and the Peg Method. While variations occur in the way
that the method is presented and taught, the fundamental concept of linking a
mental image with a spatial location is common across all formats. technique
is a mnemonic method in which a practitioner visualizes traversing sites in a
spatial location (a “Memory Palace”), and the items to be remembered are
“placed” at the sites in that location (Yates, 1966). More specifically for
Speed Cards, many memory athletes use a mnemonic system involving using a
Person-Action-Object (PAO) triple encoding to identify three cards with one
experienced image. The athletes then link these images via an ordered memory
palace to store and retrieve those images in a desired sequence. This is the
technique used by Speed Cards world record holder Shijir-Erdene Bat-Enkh
(Dellé, 2017), as well as by ND.

###  2.1 Summarizing the Strategy

Specific mnemonic systems have been developed for abstract memorization tasks
and there are several such systems for memorizing cards within the broad
umbrella of the Method of Loci. When applied to Speed Cards, the common goal
across these systems is to turn each card in a 52 card deck into something
memorable. Former memory champion Dominic O’Brien developed one such system
called the Dominic System (O’Brien, 2013).

Figure 1: Person-Action-Object Method of Loci: Before the competition, the
memory athlete memorizes a mapping from each card onto the corresponding
person, action, and object (top-left) and a sequence of locations within the
memory palace (bottom-left). During competition, the memory athlete memorizes
the deck by mapping each triple of cards onto the corresponding person,
action, and object (in that order), and placing the resulting scene in the
next location of the memory palace (top-right). Then during recall, they
traverse the memory palace in order, observe the memorized scene at each
location, and map it back to the cards (lower-right). (Map excerpt: Olaus,
1572)

The method consists of two components. The first is a _memory palace_ : a pre-
memorized familiar location containing several sub-locations (at least 17, for
the Dominic System). For example, ND uses his high school, where the sub-
locations are specific locations throughout the school (e.g., the entrance to
the school, specific classrooms, and the gymnasium). The memory athlete
visualizes themselves visiting each sub-location in a fixed order (ND starts
at the entrance to the school, then proceeds in sequence to certain
classrooms) and observing a scene placed there (Figure 1, bottom-left).

The scenes themselves are constructed using the Person-Action-Object (PAO)
mapping. Each card is associated with an arbitrary, but distinctive person.
For each person, an action and an object that relate to the person are then
chosen. For example, in ND’s mapping (Figure 1, top-left), the five of clubs
could correspond to:

  * •

_Eric Clapton_ (person)

  * •

_playing_ an instrument (action)

  * •

a _guitar_ (object)

Both the memory palace and PAO mapping are fixed, and will be memorized before
the competition.

When presented with a deck to memorize at the competition, cards are memorized
in sequences of three by taking the person associated with the first card, the
action associated with the second card, and the object associated with the
third card. This visualization of the person doing the action to or with the
object is then embedded in the next location of the memory palace. For
example, ND maps the sequence  onto _C-3PO_ ( person) _playing_ ( action) a
_pizza_ ( object) (Figure 1, top-right). During recall, the athlete visits
each location in turn, observes the scene, and recovers the card sequence that
it encodes (Figure 1, bottom-right). The problem of recalling 52 cards is thus
reduced to recalling 17 unique scenes. (Further detail is provided in Appendix
A).

###  2.2 Distinctiveness of Representation

Memory athletes may try to reduce the difficulty of discriminating the items
in their memory palaces by ensuring the PAO associations are sufficiently
distinct. When a memory athlete has two people that could be associated with
the same action, the memory athlete may try to choose a different action for
each person that would be a stronger association. For example, ND uses Kobe
Bryant and Shaquille O’Neal, both basketball players, in his PAO scheme. While
Kobe Bryant and Shaquille O’Neal could both be shooting a basketball, ND
instead associates Bryant with dribbling and O’Neal with dunking. ND reports
that this aids in reducing the memorization time for the PAO mapping.

###  2.3 Qualitative Experiences During Training and Testing

During the memorization phase, ND is aware of looking at the three cards one-
by-one rather than looking at all three cards at once, but perceives a single
composite image associated with the three cards. ND compares it to reading a
sentence: he reports that, when encoding a triple of cards in a certain
location, he does not think about the previous three cards or next three cards
at all. In a competition, after the memorization phase and stopping his timer,
ND will use the remaining time to review the memory palace. He reports that
this review is crucial. He traverses the memory palace from the most recent
spot to the first spot (i.e., backwards from the order he memorizes the
cards). He finds that, if he can go through the palace once, he can think
about something else (or even have a conversation) for the rest of the time
and still recall the information.

ND reports that he spent time making imagery vivid when he was a novice, but,
given the time pressure of competition settings, the amount of detail in the
imagery must drop. This happened automatically, not intentionally, for him. He
gives the example of his father: ND used to take time to visualize his
father’s hair, hear his accent, and figure his height based on the location,
but now, ND mainly imagines the “essence” of his father’s hair. He reports
that the parts he does not focus on are blurry, fuzzy, and grayscale while the
things he does focus on have some color. An interesting example of this is the
Ace of Spades. The person for this card is Arnold Schwarzenegger. ND reports
that ND does not see Schwarzenegger at all and only hears Schwarzenegger’s
voice saying a particular thing. When creating the PAO image, the action is
performed by a “blob.”

##  3 S3Q-Guided Analysis and Hypotheses

In order to push the boundaries of our understanding of memory athletes and
memory systems in general, we give both a characterization of the neural
mechanisms used in the Method of Loci as well as a methodology for using
current cognitive modeling tools to explore these mechanisms. Finally, we
propose two hypotheses that highlight the potential for how the S3Q theory of
consciousness could provide meaningful guidance toward novel understanding of
expert memory performance.

###  3.1 Neural Characterization

We hypothesize that high repetitions practice will lead to a decrease in the
network activation strength needed to maintain performance over time, while
the conscious learning mechanism will be indexed by a positive activation
signal for successful memory. In popular computational models of expertise
(Gobet and Simon, 2000), it takes 250 ms to write a chunk into a template
slot. Fellner et al. (2016) hypothesize that the memory palace is represented
via $\sim$250 ms theta oscillations of place cell activity in a hippocampal
template. Higher frequency alpha, beta, and gamma ($\sim$10-50 Hz) rhythms
could represent the imagined items translated from the physical cards and
chunked into an event. It is hypothesized that the PAO chunk is either placed
into working memory representation via these theta rhythms, or if templates
have been developed, directly into a slot in long term memory represented by
higher frequency rhythms. During competition, synchronous high frequency
oscillations in the posterior cortex are here hypothesized to ride theta
oscillations directly into slots generated by the hippocampus and surrounding
cortex, binding chunks of PAO items to the spatial context map loci. Neuronal
sequences generated by the hippocampus are hypothesized to act as working
memory pointers that interface with retrieval structures represented in
cortical neural networks that have been honed through extensive practice.
Repetitive experience automatically activates Hebbian neuroplasticity
mechanisms (“neurons that fire together, wire together”) to optimize activity
based on statistical demands of the memory event (Hebb, 2005), creating
templates in the hippocampus and neocortex that are hypothesized to achieve
efficiency through statistical prediction that optimizes the trajectory of the
oscillation towards improved performance. After repeated practice mentally
navigating a memory palace, a template is formed such that forward prediction
of sequential loci via pattern completion facilitates transition between event
boundaries (DuBrow and Davachi, 2013).

Similarly, a PAO template allows efficient transition between variable
categories represented in distant cortical regions for the oscillations,
chunking the imagined qualitative features matched to sensory cue input
between variable categories represented in distant cortical regions. These
processing efficiencies allow the integration of more features into the same
limited bandwidth theta cycle. The implication is that the frequency bandwidth
of the 4-chunk conscious, working memory-based learning mechanism is the same
for experts and novices ($\sim$250 ms), but within their domain of expertise,
memory athletes are better able to access hierarchical long term memory chunks
through efficient retrieval structures honing slots into nested oscillations
by high repetition neuroplasticity learning mechanisms. This trainable
capacity can be operationalized through an increase in theta-gamma cross-
frequency coupling during the memory competition, for example, or a decrease
in high frequency power and amplitude as a function of practice. These
parallel mechanisms allow athletes to achieve increased qualitative capacity
with practice, through an interaction of S3Q resource allocation honed for
efficiency by the statistical demands of the task.

###  3.2 Cognitive Modeling for Memory Experimentation

As described in the introduction, there are distinct learning systems enabling
expert performance during competitive memory athletics. The first is a
conscious representation that flexibly balances S3Q resource tenets for the
Method of Loci during the deck of cards competition. In contrast, a parallel
nonconscious system emerges through practice to adapt neural circuits for
efficient processing at the cost of flexibility. These two learning systems
interact for elite memory performance, providing a clear demonstration that
performance is not fixed, but matures with practice to provide a more rich
conscious experience.

Although other cognitive architectures (SOAR, ICARUS) could be used for this
model, we leveraged some of our previous modeling work in the ACT-R cognitive
architecture which relies on the same architectural mechanisms we will use in
this model: production compilation and the core-affect module (Larue et al.,
2017, 2018). ACT-R is a theory of human cognition and a computational
cognitive architecture composed of different modules (i.e., specialized
processing units) that interact via their associated buffers to complete a
cognitive task (Anderson, 2013). The distinction between two systems,
procedural memory (rules) and declarative memory (chunks), is a strong
foundational principle. An estimate of _human_ real-time processing is
associated with rule-firing (250 ms); similarly, a time estimate, a function
of recency and frequency of the use of that specific memory, is associated
with memory retrieval operations.

In ACT-R, the transition from novice to expert has been modeled as skill
learning through production compilation for specialization (Taatgen and
Anderson, 2002). Production compilation is the learning mechanism that
supports the transition from novice to expert. Through production compilation,
two existing rules (from procedural memory) can be combined into one new rule
if they are fired sequentially frequently. This transforms task-specific
declarative knowledge and general production rules into task-specific
production rules. Chunk retrievals from declarative memory that were occurring
in each of the rules will be substituted by constants into the new rule (one
rule fires faster than two).

A model of memory athletics would be decomposed in four phases: (1) Initialize
the model, (2) Build PAO of a deck’s face and number cards, (3)
Rehearsal/Practice, and (4) Competition.

During this learning phase, we emulate the knowledge a participant would have
prior to building a PAO and which will be necessary to build a PAO. In future
models, initial declarative memory chunks could be initialized with an open
database (ConceptNet (Speer et al., 2017) \+ NRC emotion database (Mohammad
and Turney, 2013)), providing a semantic network of chunks and NRC database
for arousal and valence values of words. Learning the palace occurs through
predictive learning. A palace is initialized as a sequence of chunks
(locations within the palace) from a preset map, and situating (i.e. locating
elements) in this memory palace will enable the memorization of a sequence of
elements during competition. Through multiple simulations and learning by
comparison with the “real world,” the palace “geography” will be reinforced
and lead to production compilation (i.e., from a procedure retrieving elements
explicitly from declarative memory to procedures including
implicit/overlearned knowledge).

In the second phase, to build a PAO narrative of face cards of a deck
(following the procedure described by ND), the associate-face-cards rule
creates an association with the elements with the highest arousal. For
example, when a retrieval request will be made for the king, the PAO image
with the highest activation and arousal among those with some semantic
connection to “king” (e.g., father) will be retrieved and a production rule
will link the card to the word. In S3Q terms, elements with the highest
arousal are the one that have been the most simulated and are the fastest to
retrieve (maximizing the feeling of rightness). Simulation in the ACT-R model
is supported by consecutive open-retrieval requests to declarative memory
elements (i.e., chunks), retrieval requests with no constraints on chunk
values but constraints on activation parameters of arousal/core-affect value
(i.e. to ensure the retrieval of elements with a similar emotional intensity).
To consolidate those memory chunks, the model will go through a “dreaming”
phase (Juvina et al., 2018): a dreaming model will do “open-retrievals” (i.e.
retrieval requests to declarative memory with no constraints on specific
value) which will reinforce chunks with the highest activation (i.e. most
recently and/or frequently activated chunks). This mirrors the way in which
the qualia vocabulary underpinning the S3Q model is created and modified. The
repetition of this simulation and dreaming in this phase will lead to
production compilation of relationships in PAOs. Through production
compilation, a model instance might no longer need to retrieve PAO chunks from
declarative memory.

Following ND’s verbal report of the task (i.e., face and number cards are
encoded in two separate steps), to encode other types of cards, a retrieve-
else rule will exclude retrieval of existing face card associations and loop
to associate any number-cards to a new “person”. The retrieval request will
include characteristics of the card, and through pattern matching, the highest
arousal “person” chunk will be retrieved, and an object and action will be
retrieved in subsequent rules to be associated to the person in a final rule.

During the third phase, rehearsal/practice, the model’s instance will test the
strength of its associations for each card through free-recall rules. We
hypothesize that the differentiation between each type of action/object for
each person will occur naturally as “testing” the memory will result through
practice in the elimination of pairings which lead to confusion, maintaining
structural coherence. More specifically, structural coherence is gained by
eliminating interferences. The structural coherence of a simulation is
maximized when this simulation can not be confused with another one (i.e., in
this case structural coherence = more distinctiveness). Simulation will lead
to retaining the representation with the highest Feeling of Rightness (FOR)
(i.e., the most consistent response modeled as the most easily retrieved) and
the least interference. This unique representation will be a stable,
consistent and useful representation.

During the fourth phase, the competition, the model will leverage the mind
palace it has built to temporarily memorize new elements. The palace will
facilitate sequence learning: transition from room to room will have been
learnt through production compilation, and placing the characters which have
been rehearsed in every room rather than new characters will facilitate
retrieval: rehearsed characters and action are associated with a high feeling
of rightness (faster retrieval); placing them in different rooms of the palace
will allow for memorizing their sequence (the sequence of rooms is present in
implicit memory and will make for a faster use for encoding the sequence of
cards). We hypothesize that an accelerated retrieval is supported by the
compiled productions involved during competition, allowing for the retrieval
of association between characters/actions and rooms.

###  3.3 Future Work

In this section we give two examples of planned experiments that would improve
our understanding of memory systems. The first shows how we can use the S3Q
perspective to make testable predictions about the impacts of modifications to
the Method of Loci strategy on memory capacity and failure rates. The second
tests the flexibility of the encoding scheme in order to tease out the complex
relationship between the neural mechanisms involved in the repetition-based
learning of PAO templates and the mapping to the phenomenal experience.

####  3.3.1 S3Q Flexibilities for Optimized Method of Loci

The mainstream view is that consciousness consists of a small number of high
resolution items, specifying that this representation is limited by a number
of fixed-capacity pointers (Luck and Vogel, 1997). Alternatively, an S3Q two-
system model requires a flexible resource allocation policy with a capacity
that is not fixed, but results from a complex interaction with complementary
nonconscious knowledge structuring (McClelland et al., 1995). For example,
there are separate pools of conscious resources for object and location
process codes in the brain, suggesting the possibility that PAO chunks and
memory palace representations flexibly use separate resources (Baddeley, 1986;
Wickens and Liu, 1988). Auditory perception resources are also separable from
visual, suggesting the use of auditory features to improve encoding efficiency
(Wickens, 2008). This modality could be beneficial for distinctiveness,
although this feature would ultimately compete for limited conscious
resources. Alvarez and Cavanagh (2004) found that estimated working memory
capacity of research participants in basic laboratory paradigms decreased as a
function of the difficulty of discriminating different items, with maximally
discriminable stimuli having a maximum capacity of approximately 4.5 for the
simplest items. It is expected that a 4-item mnemonic system might improve
Speed Cards performance, and leveraging the auditory modality could help this
system push the limits of memory.

S3Q theory suggests the generated memory palace mnemonic system must suitably
balance the number of items being processed with the relationships between
them during the Speed Cards competition as more items per chunk will come at
the cost of distinctiveness in representation, and therefore memory. With
practice, it is expected this can become ultimately distinct. For example, a
Person-Action-Object system is expected to be a slower strategy for encoding a
deck of cards, yet more accurate than a Person-Action-Object-Sound system222
There may be additional limitations that arise here in finding 52 distinctive
sounds to make this proposed method tractable., which would be predicted to
have the capacity to encode more cards per working memory cycle, but at the
risk of increased memory failures. Targeted Memory Reactivation (TMR) offers a
cutting-edge technique to test this (Rudoy et al., 2009). It has been
demonstrated that pairing a tone with a learning event, and then replaying
that tone during sleep improves memory (Rudoy et al., 2009). When these tones
are cued during slow wave sleep, techniques such as Representation Similarity
Analysis (Kriegeskorte et al., 2008) can reveal that pattern of electrical
activity in the neural structure that was seen during learning is reactivated
during spindle events between the hippocampus and neocortex. Memories are
actively labile during sleep and can be actively manipulated as they are
malleable. This technique could be employed in competitive memory training
research protocols to accelerate practice as well as test the hypothesis that
these different modalities are separate resources in the single bound
conscious representation.

####  3.3.2 Implicit Learning is Rapid but Inflexible

We hypothesize that experts demonstrate operating characteristics of
representations with matured implicit circuits, such as affording rapid
response at the cost of flexibility. Thus, we propose that the physical card
is automatically mapped to an image of the person, action, or object in the
memory palace space of experts (i.e. card image $\to$ C3P0 image), while in
novices there are intermediate steps (e.g., card image $\to$ verbal “three of
clubs” $\to$ C3P0 image). To test this, athletes can be presented with a
series of deck memorization tasks, in which experimenters vary the deck image
to disrupt the hypothesized implicit mapping from physical card image to
simulated PAO image. For example, variations of card images can be used:
traditional English-style cards, cards in which the suit colors are replaced
(e.g. blue diamonds), or cards in which the name of the card is written in
English text, e.g. “three of clubs.” Should the direct visual mapping exist,
we predict that experts will rapidly memorize the English-style cards, yet be
slower for other deck images 333These predictions are in-line with ND’s
reported experiences using similar decks (e.g., a “JUMBO” Bicycle deck rather
than a standard deck) which does not impact memorization times versus decks
with more distinctiveness (e.g., a french deck with the letters “V,” “D,” and
“R” in place of the “J,” “Q,” and “K” in an english deck), where there is a
noticeable hesitation. due to the distance in the perceptual space of
situations. On the other hand, if experts are mapping to the card name, the
verbally descriptive cards should yield the fastest times, as would be
predicted in novices – which offers a within-subjects design to see how this
flexibility and expertise change as a function of practice repetitions.

##  4 Conclusions

There are two types of learning and memory, and they interact to produce
expert performance. One type utilizes relatively fixed circuits honed for
rapid application of expertise, but rendered inflexible in transfer to novel
conditions. The other type balances S3Q to achieve flexible one shot learning.
The capacity of this second system is not fixed, but results from a balance of
S3Q resources interacting with the neuroplasticity mechanics of parallel
implicit representations. The theoretical position taken here aligns with the
assertion that working memory mechanisms affording one-shot learning are thus
not of a fixed capacity, but are optimized in interaction with implicit
mechanisms through practice. While memory athletes are pushing the limits of
the method of loci through high repetitions training of chunked information,
it will be important for generated mnemonic systems to balance the number of
items being represented with the distinctiveness of the features based on the
limited conscious resources of the brain. State-of-the-art memory techniques,
such as TMR, offer novel approaches to testing these theories while optimizing
memory performance for expert competition. The important future questions
still remain: in what ways are these multiple memory systems representations
similar and dissimilar, and what are the affordances each provide to both
biological and artificially intelligent implementations.

##  Appendix Appendix A The Dominic System

The overall approach of the Dominic System is to assign visuals, also known as
“attributes,” to each card. This system assigns each of the 40 non-face cards
a unique pair of letters where the first letter represents the number (e.g., 1
= A, 2 = B, 3 = C, …) and the other letter represents the suit (H = Hearts, D
= Diamonds, C = Clubs, S = Spades). (Traditionally, no mapping is given for
the twelve face cards since these are usually easier to visualize due to the
intrinsic semantics as described below.) Next, a person (or animal, fictional
character, etc.) who has those initials is chosen. For example, the 5 of Clubs
is assigned the letter pair (E, C), and ND chose Eric Clapton. The face cards
are handled differently. They are chosen to be intuitive based on the suit and
face. For example, consider the King of Hearts. The heart suit reminds ND of
family and the “king” of his family is his father, thus ND choses his father
for the King of Hearts. This gives a card $\to$ person mapping. It is
important that this mapping associates a distinctive person with each card.
That is, once a person is associated with one card, that person (or
subjectively similar people) should not be associated with another card. To
memorize a deck of cards with this mapping, memory athletes would need a
memory palace with 52 positions. The first card is mapped to its person and a
mental image of the person is stored in the first position of the memory
palace. This repeats for the whole deck of cards. Once the deck of cards is
stored in the memory palace, memory athletes recall the deck by traversing the
memory palace and reversing the mapping at each location to recall the card in
that position.

This limited system has the disadvantage of requiring 52 positions in the
memory palace, but the full PAO system only requires 17 (or optionally 18).
The system starts with the person mappings assigned to each card. Then, an
action and an object are assigned to each card that corresponds naturally with
the person already assigned to that card. The action and object are chosen to
correspond with the person to make learning the mapping easier. For an example
of an action and object, recall that the person assigned to the 5 of Clubs is
_Eric Clapton_. The action is chosen to be _playing_ an instrument, and the
object is a _guitar_. It is important that the action and object are both
unique. That is, no action or object may be assigned to two different cards.
At this point, a card $\to$ (person, action, object), or PAO, mapping has been
constructed for all 52 cards.444A complete PAO list for a 52 card deck used by
ND can be found in his recent book (Dellis, 2018).

To memorize a deck of cards with this mapping, memory athletes use a memory
palace with 17 (or optionally 18) positions. The first card is mapped to the
first card’s person, the second card is mapped to the second card’s action,
and the third card is mapped to the third card’s object. A mental image is
formed of the person associated with the first card doing the action
associated with the second card to the object associated with the third card.
(For instance,  would be _C-3PO_ _playing_ a _pizza_ as a guitar.) That image
is then placed in the first location of the memory palace, which has been
previously constructed. Then the fourth, fifth, and sixth cards are likewise
mapped to an image and that image is stored in the second position of the
palace. This repeats for the first 51 cards/17 positions. The final card is
remembered without the memory palace and, if forgotten, deduced by process of
elimination (or stored in the 18th position). To recall the deck of cards,
memory athletes traverse the memory palace. In order to construct the memory
palace itself, the memory athlete chooses a well-known real world place (ND
uses places such as his home or high school campus), and rehearses a specific
set of locations in those places which will serve as the locations at which
visual items will be stored. For example, a memory palace through a high
school might start at the entrance to the school, then proceed to a hallway
just inside the door, and then to a classroom nearby with the distinct
locations connected by a rehearsed path through the palace.

Memory athletes do not need to follow the system for choosing people exactly
as described. To enable faster learning, it is often better to choose
something intuitive for number cards, as is done for face cards, so that the
mapping is easier to remember. In this way, less time is spent thinking of who
a card maps to. For example, ND struggled to come up with a sufficiently
intuitive mapping for the 4 of Spades. The 4 of Spades maps to (D, S). This
reminded ND of the Nintendo DS and, in turn, Super Mario. So, the person for
the 4 of Spades is Mario.

In order to make the mappings intuitive, the PAO associations may evolve over
time. Early on, some associations may be found to be weak (e.g., hard to
visualize or hard to recall). These can be replaced. Likewise, memory athletes
may find new things they find memorable that they want to include in their PAO
mappings. Eventually, mapping a card to its associations and vice versa
becomes automatic. Memory athletes look at a card and automatically think of
the person associated with it. For Speed Cards, this is critical because the
speed of memorization often comes down to how quickly memory athletes can
encode the cards.

It should be noted that, while the technique has been described as primarily
visual, it does not need to be. Memory athletes may also leverage auditory
imagination, somatosensory imagination, etc. ND reports using non-visual
imagery in his novice stage to make the mental image as rich as possible. For
example, ND would try to hear particular accents. At the expert level, ND
reports using the sound of Arnold Schwarzenegger’s voice as the main
representation for Schwarzenegger as a person in his scheme. ND reports that
using sound as the main representation for a person is uncommon in his
encodings and Schwarzenegger may be unique in this respect. And, ND reports
using auditory imagery for some actions, like shooting.

## References

  * Alvarez and Cavanagh [2004] George A. Alvarez and Patrick Cavanagh.  The capacity of visual short-term memory is set both by visual information load and by number of objects.  _Psychological science_ , 15(2):106–111, 2004\.
  * Anderson [1982] John R. Anderson.  Acquisition of cognitive skill.  _Psychological review_ , 89(4):369, 1982.
  * Anderson [2013] John R. Anderson.  _The architecture of cognition_.  Psychology Press, 2013.
  * Ashby and Maddox [2005] F. Gregory Ashby and W. Todd Maddox.  Human category learning.  _Annual review of psychology_ , 56(1):149–178, 2005.
  * Baddeley [1986] Alan Baddeley.  _Working Memory_.  Larendon Press/Oxford University Press, 1986.
  * Charness [1976] Neil Charness.  Memory for chess positions: Resistance to interference.  _Journal of Experimental Psychology: Human Learning and Memory_ , 2(6):641–653, 1976.
  * Chase and Simon [1973] William G. Chase and Herbert A. Simon.  Perception in chess.  _Cognitive psychology_ , 4(1):55–81, 1973.
  * Cowan [2001] Nelson Cowan.  The magical number 4 in short-term memory: A reconsideration of mental storage capacity.  _Behavioral and brain sciences_ , 24(1):87–114, 2001.
  * Dellé [2017] Florian Dellé.  Interview with Shijir-Erdene Bat-Enkh.  http://web.archive.org/web/20171006112442/https://memory-sports.com/blog/interviews/interview-with-shijir-erdene-bat-enkh/, 2017\.
  * Dellis [2018] Nelson Dellis.  _Remember It! The Names of People You Meet, All of Your Passwords, Where You Left Your Keys, and Everything Else You Tend to Forget_.  Abrams Image, 2018.
  * DuBrow and Davachi [2013] Sarah DuBrow and Lila Davachi.  The influence of context boundaries on memory for the sequential order of events.  _Journal of Experimental Psychology: General_ , 142(4):1277, 2013.
  * Ericsson and Kintsch [1995] K. Anders Ericsson and Walter Kintsch.  Long-term working memory.  _Psychological review_ , 102(2):211, 1995.
  * Ericsson and Polson [1988] K. Anders Ericsson and Peter G. Polson.  A cognitive analysis of exceptional memory for restaurant orders.  _The nature of expertise_ , 1:23–70, 1988.
  * Evans et al. [2013] Karla K. Evans, Diane Georgian-Smith, Rosemary Tambouret, Robyn L. Birdwell, and Jeremy M. Wolfe.  The gist of the abnormal: Above-chance medical decision making in the blink of an eye.  _Psychonomic bulletin & review_, 20(6):1170–1175, 2013.
  * Feigenbaum and Simon [1984] Edward A. Feigenbaum and Herbert A. Simon.  Epam-like models of recognition and learning.  _Cognitive Science_ , 8(4):305–336, 1984.
  * Fellner et al. [2016] Marie-Christin Fellner, Gregor Volberg, Maria Wimber, Markus Goldhacker, Mark W. Greenlee, and Simon Hanslmayr.  Spatial mnemonic encoding: theta power decreases and medial temporal lobe bold increases co-occur during the usage of the method of loci.  _ENEURO_ , 3(6), 2016.
  * Gobet and Jackson [2002] Fernand Gobet and Samuel Jackson.  In search of templates.  _Cognitive Systems Research_ , 3(1):35–44, 2002\.
  * Gobet and Simon [1998] Fernand Gobet and Herbert A. Simon.  Expert chess memory: Revisiting the chunking hypothesis.  _Memory_ , 6(3):225–255, 1998.
  * Gobet and Simon [2000] Fernand Gobet and Herbert A. Simon.  Five seconds or sixty? presentation time in expert memory.  _Cognitive science_ , 24(4):651–682, 2000.
  * Gobet et al. [2015] Fernand Gobet, Peter CR Lane, and Martyn Lloyd-Kelly.  Chunks, schemata, and retrieval structures: Past and current computational models, 2015.
  * Hebb [2005] Donald Olding Hebb.  _The organization of behavior: A neuropsychological theory_.  Psychology Press, 2005.
  * Juvina et al. [2018] Ion Juvina, Othalia Larue, and Alexander Hough.  Modeling valuation and core affect in a cognitive architecture: The impact of valence and arousal on memory and decision-making.  _Cognitive Systems Research_ , 48:4–24, 2018.
  * Klein et al. [2010] Gary Klein, Roberta Calderwood, and Anne Clinton-Cirocco.  Rapid decision making on the fire ground: The original study plus a postscript.  _Journal of cognitive engineering and decision making_ , 4(3):186–209, 2010.
  * Kriegeskorte et al. [2008] Nikolaus Kriegeskorte, Marieke Mur, and Peter A. Bandettini.  Representational similarity analysis-connecting the branches of systems neuroscience.  _Frontiers in systems neuroscience_ , page 4, 2008.
  * Kundel and Nodine [1975] Harold L. Kundel and Calvin F. Nodine.  Interpreting chest radiographs without visual search.  _Radiology_ , 116(3):527–532, 1975.
  * Larue et al. [2017] Othalia Larue, Alexander Hough, and Ion Juvina.  A core-affect model of decision making in simple and complex tasks.  In _CogSci_ , 2017.
  * Larue et al. [2018] Othalia Larue, Alexander Hough, and Ion Juvina.  A cognitive model of switching between reflective and reactive decision making in the wason task.  In _Proceedings of the Sixteenth International Conference on Cognitive Modeling_ , pages 55–60, 2018.
  * Lesgold et al. [1988] Alan Lesgold, Harriet Rubinson, Paul Feltovich, Robert Glaser, Dale Klopfer, and Yen Wang.  Expertise in a complex skill: Diagnosing x-ray pictures.  _The nature of expertise_ , pages 311–342, 1988.
  * Luck and Vogel [1997] Steven J. Luck and Edward K. Vogel.  The capacity of visual working memory for features and conjunctions.  _Nature_ , 390(6657):279–281, 1997.
  * Maddox and Ashby [2004] W. Todd Maddox and F. Gregory Ashby.  Dissociating explicit and procedural-learning based systems of perceptual category learning.  _Behavioural processes_ , 66(3):309–332, 2004\.
  * McClelland et al. [1995] James L. McClelland, Bruce L. McNaughton, and Randall C. O’Reilly.  Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory.  _Psychological review_ , 102(3):419, 1995.
  * Miller [1956] George A. Miller.  The magical number seven, plus or minus two: Some limits on our capacity for processing information.  _Psychological review_ , 63(2):81, 1956.
  * Mohammad and Turney [2013] Saif M Mohammad and Peter D Turney.  Crowdsourcing a word–emotion association lexicon.  _Computational intelligence_ , 29(3):436–465, 2013.
  * Newell and Rosenbloom [1981] Allen Newell and P. Rosenbloom.  Mechanisms of skill acquisition.  _Cognitive skills and their acquisition_ , 1981.
  * O’Brien [2013] Dominic O’Brien.  _How to pass exams: accelerate your learning, memorize key facts, revise effectively_.  Watkins Media Limited, 2013.
  * Olaus [1572] Magnus Olaus.  Carta marina.  https://www.loc.gov/resource/gdcwdl.wdl_03037/, 1572.
  * Reber [2013] Paul J. Reber.  The neural basis of implicit learning and memory: A review of neuropsychological and neuroimaging research.  _Neuropsychologia_ , 51(10):2026–2042, 2013.
  * Rudoy et al. [2009] John D. Rudoy, Joel L. Voss, Carmen E. Westerberg, and Ken A. Paller.  Strengthening individual memories by reactivating them during sleep.  _Science_ , 326(5956):1079–1079, 2009.
  * Schmidt et al. [2021] Kevin Schmidt, Jared Culbertson, Chad Cox, H. S. Clouse, O. Larue, M. Molineaux, and S. Rogers.  What is it like to be a bot: Simulated, situated, structurally coherent qualia (s3q) theory of consciousness.  _arXiv preprint arXiv:2103.12638_ , 2021.
  * Schneider and Shiffrin [1977] Walter Schneider and Richard M. Shiffrin.  Controlled and automatic human information processing: I. detection, search, and attention.  _Psychological review_ , 84(1):1, 1977.
  * Speer et al. [2017] Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_ , 2017\.
  * Squire [1992] Larry R. Squire.  Declarative and nondeclarative memory: Multiple brain systems supporting learning and memory.  _Journal of cognitive neuroscience_ , 4(3):232–243, 1992.
  * Squire and Dede [2015] Larry R. Squire and Adam J. O. Dede.  Conscious and unconscious memory systems.  _Cold Spring Harbor perspectives in biology_ , 7(3):a021667, 2015.
  * Taatgen and Anderson [2002] Niels A. Taatgen and John R. Anderson.  Why do children learn to say “broke”? a model of learning the past tense without feedback.  _Cognition_ , 86(2):123–155, 2002.
  * Wickens [2008] Christopher D. Wickens.  Multiple resources and mental workload.  _Human factors_ , 50(3):449–455, 2008.
  * Wickens and Liu [1988] Christopher D. Wickens and Yili Liu.  Codes and modalities in multiple resources: A success and a qualification.  _Human factors_ , 30(5):599–616, 1988.
  * World Memory Sports Council [2013] World Memory Sports Council.  _The official competitor and arbiter handbook for the mind sport of memory_.  Filament Publishing, http://www.gomsa.global/wp-content/uploads/2018/01/Competitors-and-Arbiters-Handbook.pdf, 2013\.
  * World Memory Sports Council [2022] World Memory Sports Council.  World memory sports council official statistics and world rankings.  http://www.world-memory-statistics.co.uk/home.php, 2022.
  * Yates [1966] Frances A. Yates.  _The Art of Memory_.  University of Chicago Press, 1966.


# Improving Speech Prosody of Audiobook Text-to-Speech Synthesis with Acoustic
and Textual Contexts

###### Abstract

We present a multi-speaker Japanese audiobook text-to-speech (TTS) system that
leverages multimodal context information of preceding acoustic context and
bilateral textual context to improve the prosody of synthetic speech. Previous
work either uses unilateral or single-modality context, which does not fully
represent the context information. The proposed method uses an acoustic
context encoder and a textual context encoder to aggregate context information
and feeds it to the TTS model, which enables the model to predict context-
dependent prosody. We conducted comprehensive objective and subjective
evaluations on a multi-speaker Japanese audiobook dataset. Experimental
results demonstrate that the proposed method significantly outperforms two
previous works. Additionally, we present insights about the different choices
of context - modalities, lateral information and length - for audiobook TTS
that have never been discussed in the literature before.

Index Terms—  text-to-speech synthesis, TTS, audiobook, speech prosody,
context modeling

##  1 Introduction

Recent text-to-speech (TTS) systems based on deep neural networks (DNNs) have
been able to synthesize natural read-out speech [1, 2, 3]. However, how to
synthesize speech with a lot of prosody variations like audiobooks remains
unsolved. Synthesizing such speech requires the system to not only transform
linguistic information but also para-/non-linguistic information such as
emotions, and intentions from text to speech [4, 5, 6]. Speech prosody in
audiobooks produced by professional speakers depends on several factors
including characteristics, context, and styles (narration or dialogue) [7].
Among these factors, context, either acoustic or textual, is popularly
utilized in the literature to improve the prosody of audiobook TTS [8, 9, 10,
11]. This is because (1) consecutive utterances always have sequential
relations in audiobooks; (2) unlike characteristics and other factors, context
requires no additional cost to get.

However, existing work either (1) uses only single-modality context or (2)
uses unilateral context which cannot fully leverage the power of context
information. Gallegos et al. first proposed to use acoustic context to improve
the prosody of audiobook TTS [8]. In their following work, they further used
acoustic and textual contexts in audiobook TTS [9], but only preceding
contexts were used. Xu et al. first proposed to use pretrained bidirectional
encoder representations from transformers (BERT) [12] to encode preceding and
succeeding sentences to incorporate textual context information in audiobook
TTS [10]. Nakata et al. also used BERT embeddings, but only in an implicit way
by encoding the target sentences with the bilateral context [11]. All of these
works only used textual context with one or two sentences without trying
longer context.

In this paper, we present a multi-speaker Japanese audiobook TTS system that
fully and explicitly utilizes preceding acoustic context and bilateral textual
context to improve the prosody of synthetic speech. The proposed method first
uses an acoustic context encoder (ACE) [8] to encode preceding mel-
spectrograms as acoustic context representations. Moreover, we propose a
textual context encoder (TCE) based on attention mechanisms to extract textual
context representations from BERT embeddings of bilateral textual context.
These context representations are then fed to a multi-speaker TTS model to
guide it to synthesize the target utterance with appropriate prosody. We
conducted comprehensive experiments with both objective and subjective
evaluations to verify the effectiveness of the proposed method. We further
compared how different modalities, laterals, and lengths of context influence
the prosody of audiobook TTS, which was never studied in previous work. Our
contributions are summarized as follows:

  * •

We propose a multi-speaker Japanese audiobook TTS system with acoustic and
textual context encoding mechanisms.

  * •

We conduct comprehensive objective and subjective experiments to show the
effectiveness of the proposed method.

  * •

We further conduct experiments to find the best combination of context
modalities, laterals, and lengths for audiobook TTS.

Audio samples are publicly available111https://aria-k-
alethia.github.io/2022rat-demo/.

##  2 Related Work

Methods of audiobook TTS can be roughly grouped into two categories: single-
sentence and multi-sentence methods. Single-sentence methods synthesize one
utterance at a time, and improves speech prosody by feeding auxiliary features
like context [8, 9, 10], emotions [13], and features extracted by DNNs like
variational autoencoder [14, 15] and global style tokens (GST) [16].
Sufficient and correct prosody information should be maintained to guarantee
satisfactory speech quality in such methods.

Multi-sentence methods, on the other hand, synthesize multiple sentences at a
time. The length of the target sequence ranges from three sentences [17] to a
whole paragraph [18]. While such methods are potentially better than single-
sentence methods, it requires more memory and sophisticated training process.

The proposed method in this work is a single-sentence method utilizing both
acoustic and textual contexts.

##  3 Proposed Method

Fig. 1: Architecture of the proposed method.

The general architecture of the proposed method is illustrated in Figure 1.
The model contains three components: an improved multi-speaker FastSpeech2 [3]
for mel-spectrograms synthesis, an acoustic context encoder and a textual
context encoder for context encoding. We introduce these components separately
in the following sections.

###  3.1 Improved multi-speaker FastSpeech2

We follow the original FastSpeech2 [3] to construct the TTS model that
contains a phoneme encoder, a variance adaptor, and a mel-spectrograms
decoder. To adapt the model to a multi-speaker setting, we first create a
look-up embedding table for speakers. The speaker embedding is summed to the
output of the phoneme encoder and fed to the following modules to generate
mel-spectrograms for the corresponding speaker. Second, we also use speaker-
dependent pitch normalization [19] to disentangle speaker information from
pitch values. Specifically, for a pitch value $p_{s}$ of speaker $s$ of a
voiced frame, we normalize it to $\bar{p}_{s}$ by:
$\bar{p}_{s}=\frac{p_{s}-\mu_{s}}{\sigma_{s}},$ where $\mu_{s},\sigma_{s}$ are
the mean and standard deviation values of the pitch of the speaker $s$,
respectively.

We also notice that the utterances of audiobooks are longer than utterances in
read-out corpus, but the absolute positional encoding [20] used in the
original FastSpeech2 cannot well handle long sequences. To solve this problem,
we replace the absolute positional encoding with relative positional encoding
[21] so that the model can handle sequences with any length. We follow the
previous work [21] and add relative positional embeddings to the attention
layers of both the encoder and the decoder. The clipping distance is set to
$4$ so that the model can capture relative position differences within this
value. For more details please refer to the original paper.

Finally, the original FastSpeech2 uses a length regulator in the variance
adaptor before the pitch and energy predictors so that the model outputs
frame-level pitch and energy, which makes the synthesized speech unstable.
Therefore we move the length regulator after the two predictors to learn
phoneme-level pitch and energy. In our preliminary experiments, we found all
of the aforementioned modifications could improve the overall performance.

The deterministic nature of the above model makes it difficult to synthesize
speech with various prosodies. Therefore, the proposed method further uses two
context encoders to incorporate context information into the model.

###  3.2 Acoustic context encoding

Fig. 2: Architecture of TCE. FC denotes fully connected layer.

In audiobooks it can be assumed that the prosodies of consecutive utterances
have minor differences, hence using acoustic context can intuitively make
consecutive utterances more coherent in single-sentence audiobook TTS methods.
To this end, the proposed method uses ACE to encode acoustic context. Since
during inference, only preceding utterances are available, ACE only encodes
preceding acoustic context. Supposing the index of the target utterance for
synthesis is $N$, we use GST as the implementation of ACE to extract a fixed-
length acoustic context vector from the $(N-1)$-th mel-spectrogram. The
context vector is summed to the output of the phoneme encoder to assist the
synthesis of the $N$-th mel-spectrogram. Note that, although during training
we use the ground truth (GT) $(N-1)$-th mel-spectrogram as the input of ACE,
during inference we use the synthesized one as the input.

Following previous work [8], we set an extra next-prediction task for ACE. As
Figure 1 illustrates, we use an acoustic encoder (AE) to extract a fixed-
length vector from the $N$-th mel-spectrogram. Here AE is implemented as
another GST module that has different parameters from the one of ACE. We then
minimize the L1 distance between the two vectors extracted by ACE and AE.
While previous work doesn’t explain why the extra task is effective [8], we
suppose this is because the extra task forces ACE to learn a corresponding
relation between the $(N-1)$-th and the $N$-th mel-spectrograms. During
training, we add the L1 loss term to the loss function of the TTS model and
jointly train the whole model.

###  3.3 Textual context encoding

Textual context is also an informative source for producing appropriate
prosody. Phrases like “a man/woman says”, and “happily/sadly” in the contexts
can be useful cues to predict the prosody of the target sentence. Therefore we
propose a textual context encoder (TCE) to incorporate textual context
information in the model. The architecture of TCE is illustrated in Figure 2.
First, the bilateral context and the target text are fed to a pretrained BERT
model to extract word embeddings. Note that, different from most previous
works [9, 10, 11], we set the context length of each lateral to $k$ characters
instead of setting it to a certain number of sentences, by which we can easily
evaluate model performance with different context lengths. The word embeddings
of the target sentence are then fed to a gated recurrent unit (GRU) [22], and
the last hidden state of GRU is used as the sentence embedding of the target
sentence. Next, the sentence embedding is used as a query vector in two
attention modules with the word embeddings of preceding and succeeding
contexts as keys and values to extract bilateral context vectors. Finally, the
two context vectors and the sentence embedding are concatenated together and
fed to a fully connected (FC) layer to get the textual context vector. This
vector is then summed to the output of the phoneme encoder to incorporate
textual context information in the TTS model.

##  4 Experiments

###  4.1 Setup

We used J-MAC, a multi-speaker Japanese audiobook corpus produced by
professional speakers, as the dataset [23]. We selected $9$ speakers from
J-MAC who at least have $3$ books to ensure each speaker has sufficient data
for training. We then randomly picked up gender-balanced $6$ speakers from the
$9$ speakers as the test speakers and excluded one audiobook for each test
speaker from the training set as the test set. The final training set
contained about 7 hours of audio data. For the BERT model in TCE, we used
”bert-base-japanese-v2”222https://huggingface.co/cl-tohoku/bert-base-
japanese-v2 pretrained on $4$GB Japanese Wikipedia data. We used the output of
the last layer of the BERT model as the word embeddings used in TCE.
OpenJTalk333https://open-jtalk.sp.nitech.ac.jp/ was used to convert Japanese
characters to phonemes. For the forced alignment we used Julius [24] to get
the duration of each phoneme. The pitch values were extracted by the WORLD
vocoder [25]. We used the pretrained “UNIVERSAL_V1” HiFi-GAN
model444https://github.com/jik876/hifi-gan to convert mel-spectrograms into
time-domain waveforms.

For the TTS model we used the same parameter setting as the one of the
previous work [3] except for the modifications mentioned in Section 3.1. The
dimension of the speaker embedding was set to $256$. In ACE, the GST token
number was set to $10$. Following the original work [26], we used multi-head
attention with $8$ heads to improve the robustness. In TCE, the number of
hidden units in GRU was set to $256$. The dimension of both the acoustic and
the textual context vectors was set to $256$ so that they could be summed to
the output of the phoneme encoder.

During training, the batch size was set to $32$. We used Adam [27] as the
optimizer, with a scheduled learning rate proposed in [20]. However, for the
fine-tuning of the pre-trained BERT model in TCE, we set the learning rate to
$10^{-7}$. The combined model converged in around $200$k steps.

We trained several variations of the proposed model to study how different
modalities, laterals, and lengths of context influence the prosody of the
synthesized speech. We denote the proposed method as ATCE-{pre., suc., bi},
where the suffix represents textual context laterals (pre. for preceding, suc.
for succeeding, and bi for bilateral context). We also trained the proposed
models without ACE, which are denoted by TCE-{pre., suc., bi}. We used two
previous methods as the baselines. The first baseline, denoted by ACE, uses
ACE to utilize acoustic context [8]. The second baseline uses one-sentence
bilateral textual context implicitly by feeding the target sentence with one-
sentence bilateral context to the BERT model but only inputting the word
embeddings of the target sentence to the TTS model [11]. Since the original
method of Nakata et al. didn’t use FastSpeech2 to synthesize mel-spectrograms,
we adapted it to the proposed method by (1) changing the context length to one
sentence and (2) only inputting the target sentence embedding to the TTS
model.

###  4.2 Objective evaluations

####  4.2.1 Metrics

In the objective evaluations we used several metrics to evaluate the synthetic
speech:

  * •

Character error rate (CER) computed using Vosk Japanese speech recognition
API555https://github.com/alphacep/vosk-api.

  * •

Mel-cepstral distortion (MCD) computed with dynamic time warping (DTW).

  * •

F0 root mean square error (F0-RMSE) computed with DTW.

  * •

Gross Pitch Error (GPE) represents the proportion of voiced frames whose
relative pitch error is higher than a certain threshold ($20\%$ in this work).

  * •

Accuracy of speaker classification (ACC) computed by a speaker classifier
trained on the training set.

Here CER and MCD measure the general speech quality, ACC measures the speaker
similarity, F0-RMSE and GPE measure the performance on speech prosody.

####  4.2.2 Textual context length

Table 1: Results of objective evaluation for the proposed ATCE-bi model with different context length $k$. Bold indicates the best score. $k$ |  CER($\downarrow$) |  MCD($\downarrow$) |  F0-RMSE($\downarrow$) |  GPE($\downarrow$) |  ACC($\uparrow$)
---|---|---|---|---|---
$16$ | $0.206$ | $7.69$ | $28.60$ | $13.49$ | $99.18$
$32$ | $0.206$ | $7.68$ | $28.33$ | $13.21$ | $99.32$
$64$ | $0.208$ | $7.72$ | $\mathbf{28.19}$ | $\mathbf{12.90}$ | $98.77$
$128$ | $0.207$ | $7.64$ | $28.56$ | $13.51$ | $98.77$
$128\rightarrow 64$ | $0.206$ | $7.65$ | $28.43$ | $13.23$ | $98.91$

We first evaluated model performances with different textual context length
$k$. We trained ATCE-bi with $k$ in $\\{16,32,64,128\\}$. The result is shown
in Table 1. First, all models have similar CER, MCD, and ACC, which is natural
since the proposed method only focuses on prosody. Second, the best
performance is obtained when $k=64$, which demonstrates that increasing
textual context length can improve speech prosody, but when the length is too
long ($k=128$), the performance degrades. We suppose this is because textual
context with a long distance to the target sentence is not relevant for
predicting the target prosody. To verify this hypothesis, we also set $k=64$
in the $k=128$ model for inference. As expected, the performance increases
slightly in this case ($128\rightarrow 64$) in Table 1), which implies the
correctness of the hypothesis. Given that the average sentence length of the
corpus is $27$ characters, our results suggest that the best textual context
length is about $2\text{-}3$ sentences from either lateral of the target
sentence.

####  4.2.3 Context modalities and laterals

Table 2: Results of objective evaluation for models with different types of context. Bold indicates the best score. ∗ represents significant improvement over the baseline methods with p-value $<0.05$. Model |  CER($\downarrow$) |  MCD($\downarrow$) |  F0-RMSE($\downarrow$) |  GPE($\downarrow$) |  ACC($\uparrow$)
---|---|---|---|---|---
GT | $0.192$ | N/A | N/A | N/A | $96.19$
HiFi-GAN | $0.208$ | $4.25$ | $14.03$ | $2.2$ | $98.64$
ACE | $0.207$ | $7.85$ | $29.05$ | $14.16$ | $98.37$
Nakata et al. | $0.206$ | $7.70$ | $28.89$ | $13.21$ | $98.64$
TCE-pre. | $0.206$ | $7.71$ | $28.19$ | $13.00$ | $98.64$
TCE-suc. | $0.206$ | $7.70$ | $28.82$ | $13.18$ | $99.05$
TCE-bi | $0.207$ | $7.69$ | $28.84$ | $12.94$ | $98.50$
ATCE-pre. | $0.205$ | $7.68$ | $28.86$ | $13.11$ | $99.18$
ATCE-suc. | $0.209$ | $\mathbf{7.66}$ | $\mathbf{27.93^{*}}$ | $\mathbf{12.57^{*}}$ | $98.91$
ATCE-bi | $0.208$ | $7.72$ | $28.19$ | $12.90$ | $98.77$

We then evaluated model performances with different context modalities and
laterals. All of the models were trained with $k=64$ obtained in the previous
section. The result is shown in Table 2. First, the proposed ATCE-* models
outperform all the baseline models, which demonstrates the effectiveness of
the proposed method. Second, all TCE-* models outperform ACE, which
demonstrates the effectiveness of introducing textual context. Third, all
ATCE-* models except ATCE-pre. have better performance than the corresponding
TCE-* models, which demonstrates the necessity of combining both acoustic and
textual context information for synthesis. Finally, to our surprise, ATCE-suc.
using succeeding textual context obtains the best performance and has better
performance than ATCE-bi using bilateral textual context. We suppose this is
because the information overlapping between the $(N-1)$-th mel-spectrogram and
the preceding texts makes succeeding textual context more informative for the
model. Therefore in such case using preceding acoustic and bilateral textual
contexts together is probably not beneficial and can even confuse the TTS
model. This hypothesis can also be verified by comparing the performances of
ATCE-pre. and TCE-pre..

We also notice that the accuracy of speaker classification of the GT model is
the worst among all models. We believe this is because the speakers usually
change their voices to act different characters in the audiobooks, which makes
it difficult to recognize the speaker identity.

####  4.2.4 Prosody under different contexts

Fig. 3: Pitch contours of the same utterance synthesized using the ATCE-bi
model with $2$ random contexts. “Predicted” represents the one synthesized
with the correct context.

Finally, we verified whether the proposed method actually learned to predict
context-dependent prosody. We selected an utterance from the test set and
synthesized it using the ATCE-bi model with random contexts. We observed that
the model could synthesize the same text with different tensions. We selected
typical examples and visualized their pitch contours in Figure 3. It can be
seen that the prosody varies a lot with different contexts, which proves that
the proposed method can predict context-dependent speech prosody.

###  4.3 Subjective evaluations

Table 3: Results of MOS evaluation. Bold indicates the best method without overlapping $95$% confidence interval. Model | MOS
---|---
ACE | $3.26$
Nakata et al. | $3.30$
ATCE-suc. | $\mathbf{3.38}$
ATCE-bi | $3.35$

####  4.3.1 Multi-sentence speech naturalness MOS test

In the subjective evaluations, we first conducted a standard $5$-scale mean
opinion score (MOS) test. We fine-tuned the HiFi-GAN vocoder on the training
set for $3000$ epochs with an initial learning rate $10^{-5}$. Following
previous work [23], we conducted a five-sentence MOS test, in which the
listeners rate the naturalness of an audio including five consecutive
utterances. In this test we selected the two baselines and ATCE-{suc., bi.}
with the best objective performance obtained in the previous section to
evaluate. For each test speaker we synthesized $10$ five-sentence audios, in
which we inserted a $0.5$ second pause after each sentence. The duration of
each audio ranges from $40$ seconds to $1$ minute. We used
Lancers666https://www.lancers.jp/, a Japanese crowd-sourcing platform, to
conduct the test. $32$ listeners joined in the test. Each listener rated $30$
audios with $5$ dummy samples at the beginning whose ratings were not counted
in the final result. Each audio had $3$ answers on average.

The result is shown in Table 3. It can be seen that all proposed models
outperform the two baseline models, and ATCE-suc. obtains the best
performance, which is consistent with the results of the objective
evaluations. This again demonstrates the effectiveness of the proposed method.

####  4.3.2 Preference AB test

Next we conducted a preference AB test using the same five-sentence audios
synthesized in the previous section. We selected two AB pairs: (ATCE-suc.,
ACE), (ATCE-suc, Nakata et al.). $40$ listeners participated in the test. Each
listener rated $10$ pairs, in which $5$ pairs have the same audios but
different orders from the rest $5$ audios. Each pair had $3$ answers on
average.

Table 4: Results of AB preference evaluation. Bold indicates the best method with p-value $<0.05$. Method A | Score A | Score B | Method B
---|---|---|---
ATCE-suc. | $\mathbf{0.615}$ | $0.385$ | ACE
ATCE-suc. | $\mathbf{0.545}$ | $0.455$ | Nakata et al.

The result is shown in Table 4. It can be seen that the proposed ATCE-suc.
method significantly outperforms the two baselines, which is consistent with
the results obtained in the previous section. All in all, the proposed method
utilizing informative acoustic and textual contexts obtains the best
performance in all evaluations.

##  5 Conclusions

This paper presented a Japanese multi-speaker audiobook TTS system that fully
and explicitly utilized preceding acoustic context and bilateral textual
context to improve the prosody of the synthetic speech. Experimental results
demonstrated that the proposed method significantly outperformed two previous
work in both objective and subjective evaluations. We also found it was
helpful to use multimodal contexts and the optimal textual length was about
$2\text{-}3$ sentences. These results can potentially shed light on future
researches in this field. Future work could be extending ACE and TCE to frame-
level resolution.

## References

  * [1] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.,  “Natural tts synthesis by conditioning wavenet on mel spectrogram predictions,”  in Proc. ICASSP. IEEE, 2018, pp. 4779–4783.
  * [2] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu,  “Fastspeech: Fast, robust and controllable text to speech,”  Proc. NeurIPS, vol. 32, 2019.
  * [3] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu,  “Fastspeech 2: Fast and high-quality end-to-end text to speech,”  arXiv preprint arXiv:2006.04558, 2020.
  * [4] Paul Taylor,  Text-to-speech synthesis,  Cambridge university press, 2009.
  * [5] Björn Schuller, Stefan Steidl, Anton Batliner, Felix Burkhardt, Laurence Devillers, Christian MüLler, and Shrikanth Narayanan,  “Paralinguistics in speech and language—state-of-the-art and the challenge,”  Computer Speech & Language, vol. 27, no. 1, pp. 4–39, 2013.
  * [6] Jennifer Cole,  “Prosody in context: A review,”  Language, Cognition and Neuroscience, vol. 30, no. 1-2, pp. 1–31, 2015.
  * [7] King Simon, Crumlish Jane, Martin Amy, and Wihlborg Lovisa,  “The blizzard challenge 2018,”  in Proc. Blizzard Challenge workshop, 2018.
  * [8] Pilar Oplustil-Gallegos and Simon King,  “Using previous acoustic context to improve text-to-speech synthesis,”  arXiv preprint arXiv:2012.03763, 2020.
  * [9] Pilar Oplustil Gallegos, Johannah O’Mahony, and Simon King,  “Comparing acoustic and textual representations of previous linguistic context for improving text-to-speech,”  in The 11th ISCA Speech Synthesis Workshop (SSW11). ISCA, 2021, pp. 205–210.
  * [10] Guanghui Xu, Wei Song, Zhengchen Zhang, Chao Zhang, Xiaodong He, and Bowen Zhou,  “Improving prosody modelling with cross-utterance bert embeddings for end-to-end speech synthesis,”  in Proc. ICASSP. IEEE, 2021, pp. 6079–6083.
  * [11] Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, Naoko Tanji, Yusuke Ijima, Ryo Masumura, and Hiroshi Saruwatari,  “Audiobook speech synthesis conditioned by cross-sentence context-aware word embeddings,”  in Proc. 11th ISCA Speech Synthesis Workshop (SSW 11), 2021, pp. 211–215.
  * [12] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova,  “Bert: Pre-training of deep bidirectional transformers for language understanding,”  arXiv preprint arXiv:1810.04805, 2018.
  * [13] Junjie Pan, Lin Wu, Xiang Yin, Pengfei Wu, Chenchang Xu, and Zejun Ma,  “A chapter-wise understanding system for text-to-speech in chinese novels,”  in Proc. ICASSP. IEEE, 2021, pp. 6069–6073.
  * [14] Wataru Nakata, Tomoki Koriyama, Shinnosuke Takamichi, Yuki Saito, Yusuke Ijima, Ryo Masumura, and Hiroshi Saruwatari,  “Predicting VQVAE-based Character Acting Style from Quotation-Annotated Text for Audiobook Speech Synthesis,”  Proc. Interspeech, pp. 4551–4555, 2022.
  * [15] Ning-Qian Wu, Zhao-Ci Liu, and Zhen-Hua Ling,  “Discourse-level prosody modeling with a variational autoencoder for non-autoregressive expressive speech synthesis,”  in Proc. ICASSP. IEEE, 2022, pp. 7592–7596.
  * [16] Daisy Stanton, Yuxuan Wang, and RJ Skerry-Ryan,  “Predicting expressive speaking style from text in end-to-end speech synthesis,”  in 2018 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2018, pp. 595–602.
  * [17] Peter Makarov, Ammar Abbas, Mateusz Łajszczak, Arnaud Joly, Sri Karlapati, Alexis Moinet, Thomas Drugman, and Penny Karanasou,  “Simple and effective multi-sentence tts with expressive and coherent prosody,”  arXiv preprint arXiv:2206.14643, 2022.
  * [18] Liumeng Xue, Frank K Soong, Shaofei Zhang, and Lei Xie,  “Paratts: Learning linguistic and prosodic cross-sentence information in paragraph-based tts,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 30, pp. 2854–2864, 2022.
  * [19] Eugene Kharitonov, Ann Lee, Adam Polyak, Yossi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Morgane Riviere, Abdelrahman Mohamed, Emmanuel Dupoux, et al.,  “Text-free prosody-aware generative spoken language modeling,”  in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 8666–8681.
  * [20] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,”  Proc. NeurIPS, vol. 30, 2017.
  * [21] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani,  “Self-attention with relative position representations,”  arXiv preprint arXiv:1803.02155, 2018.
  * [22] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio,  “Empirical evaluation of gated recurrent neural networks on sequence modeling,”  in NIPS 2014 Workshop on Deep Learning, 2014.
  * [23] Shinnosuke Takamichi, Wataru Nakata, Naoko Tanji, and Hiroshi Saruwatari,  “J-mac: Japanese multi-speaker audiobook corpus for speech synthesis,”  arXiv preprint arXiv:2201.10896, 2022.
  * [24] Akinobu Lee, Tatsuya Kawahara, and Kiyohiro Shikano,  “Julius—an open source real-time large vocabulary recognition engine,”  2001\.
  * [25] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa,  “World: a vocoder-based high-quality speech synthesis system for real-time applications,”  IEICE TRANSACTIONS on Information and Systems, vol. 99, no. 7, pp. 1877–1884, 2016.
  * [26] Yuxuan Wang, Daisy Stanton, Yu Zhang, RJ-Skerry Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Ye Jia, Fei Ren, and Rif A Saurous,  “Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis,”  in Proc. ICML. PMLR, 2018, pp. 5180–5189.
  * [27] Diederik P Kingma and Jimmy Ba,  “Adam: A method for stochastic optimization,”  arXiv preprint arXiv:1412.6980, 2014.