ifaamas [AAMAS ’21]Proc. of the 20th International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2021)May 3–7, 2021OnlineU. Endriss, A.
Nowé, F. Dignum, A. Lomuscio (eds.) 2021 2021 28 i3ai.org

# Cognitive Homeostatic Agents

Blue Sky Ideas Track

Amol Kelkar<EMAIL_ADDRESS>
###### Abstract.

Human brain has been used as an inspiration for building autonomous agents,
but it is not obvious what level of computational description of the brain one
should use. This has led to overly opinionated symbolic approaches and overly
unstructured connectionist approaches. We propose that using homeostasis as
the computational description provides a good compromise. Similar to how
physiological homeostasis is the regulation of certain homeostatic variables,
cognition can be interpreted as the regulation of certain ’cognitive
homeostatic variables’. We present an outline of a Cognitive Homeostatic
Agent, built as a hierarchy of physiological and cognitive homeostatic
subsystems and describe structures and processes to guide future exploration.
We expect this to be a fruitful line of investigation towards building
sophisticated artificial agents that can act flexibly in complex environments,
and produce behaviors indicating planning, thinking and feelings.

###### Key words and phrases:

artificial agent; homeostasis; cognitive homeostasis; hierarchical
reinforcement learning; predictive coding; free energy principle; cognitive
homeostatic agent; robotic control; lifelong learning; online learning;
emotions; rewards; predictive processing; hippocampus; engrams

##  1\. Introduction

The principle goal of artificial agents is intelligent behaviorRussell Stuart
and Norvig (2009). Early research towards building Artificial Intelligent (AI)
systems viewed human cognition as analogous to symbolic computation in digital
computersGarson (1991). Later, connectionist approaches used artificial neural
networks and attempted to implement cognitive features by mapping them to
functions, taking advantage of the universal approximation theoremCsáji et al.
(2001). Recently, deep neural networks, reinforcement learning, meta-learning
and related research areas have achieved great success in many areas of AI,
such as playing console gamesMnih et al. (2013), robotic object
manipulationOpenAI et al. (2019), natural language understanding and
generationBrown et al. (2020), and so on. Despite these advances, many
challenges remainDulac-Arnold et al. (2019) for building robust artificial
agents that can act in real world environments, interacting with humans and
other agents. First, such agents must adapt continuously to changes in their
environment, which requires online and lifelong learning. That remains
difficultChen and Liu (2018) for current deep learning systems for reasons
such as catastrophic forgettingMcCloskey and Cohen (1989). Second, current
artificial agents are driven by arbitrarily defined external rewards, or
rewards derived from unsatisfactory signals such as novelty. Third, affects
are important signalsScheutz (2012) that may help the agents deal with complex
interactions with their environment, other agents and with humans. This area
remains under-explored. Lastly, mortality and the life regulation to postpone
death are some of the fundamental driving forces behind human creativity,
industriousness and morality. It is unclear whether artificial agents would
achieve those attributes without the meta objective of self-preservation Man
and Damasio (2019).

The computational description of the brain that symbolic and connectionist
approaches use are over and under constrained, respectively. To investigate
what an appropriate level of computational description would be, we go back to
the first principles and explore how intelligence evolved and operates in
living organisms.

##  2\. Homeostasis and Cognition

Adaptive behavior is a hallmark of all living organisms. Single-celled
organisms such as bacteria can move towards food and away from noxious
chemicalsAlon et al. (1999), and routinely make complex decisions as part of
bacterial coloniesRoss-Gillespie and Kümmerli (2014). Plants share
resourcesSimard et al. (1997) with each other through networks of fungi.
Social insects lead complex, socially co-dependent livesOster and Wilson
(1978). Birds can countEmery (2006) and dogs can understand many verbal
commandsCoren (1994). Octopuses can solve puzzles and remember large
underwater terrains using landmarksGodfrey-Smith (2016). It is evident that
there is a continuum of cognitive capabilities across living organisms. Humans
have exceptionally sophisticated versions of some of these capabilities, but
the underlying physiological processes seem to be mostly unremarkable compared
to other evolutionarily proximate animal species. Building intelligent agents
based on our shared evolutionary inheritance is an under-explored approach,
which we focus on in this article.

Adaptive behavior of living organisms is grounded on an internal reward system
Deci and Ryan (2010); Barto (2013) that is tuned to their homeostatic
imperatives Morville et al. (2018). Deviation (or the prediction of deviation)
of a homeostatic variable has a positive or negative valenceJoffily and
Coricelli (2013) based on whether the deviation is towards or away from a
homeostatic target state, i.e. whether the deviation is beneficial or harmful
for the organism. For example, on a cold winter night, core body temperature
starts to fall, moving away from nominal body temperature, which feels bad.
Wearing a sweater feels good because it helps move body temperature closer
towards the target value. These valences lead to affects and feelingsCraig and
Craig (2002), and drive behavior. Can homeostasis then be a good candidate
computational description for building artificial agents that can act in
complex environments in a grounded manner?

###  2.1. (Re)defining Homeostasis

Although the term homeostasis is often used to represent a narrow set of
reactionary behaviors that move an organism towards a fixed set point,
CannonCannon (1929) originally intended the term to represent myriad
physiological strategies to sense, maintain and defend regulated variables, as
”wisdom of the body”. This broader usage includes allostasisSterling (1988),
the learned anticipatory responses and non-stationary target states, and other
related mechanisms such as predictive homeostasis Moore-Ede (1986) . We use
the term in its broadest scope, as explored under the free energy
principleFriston (2013), where homeostasis includes all processes that keeps
an organism within a relatively small set of favourable states that are
conducive to its ongoing existenceCorcoran and Hohwy (2017). Under this
definition, homeostasis not only regulates near term physiological states of
the organism and survival, but also regulates well-being and long term
flourishing.

###  2.2. Hierarchical Homeostatic Systems

Each organism is built as a hierarchy of homeostatic systemsFriston (2013);
Kirchhoff et al. (2018); Connolly and van Deventer (2017) such as membranes,
sub-cellular organelles, cells, organs and organ systems. Such hierarchies of
homeostatic systems remain stable only under specific conditions. At every
level in the hierarchy, a controller subsystem must act to ensure their own
homeostasis, the homeostasis of all subordinate subsystems and the homeostasis
of the combined system. To achieve this, controllers need to have their own
homeostatic variables (”hvars”), and hvars to represent the homeostatic state
of subordinate subsystems, conveyed by bottom-up, interoceptive signals from
those subsystems. Top-down signals allow a controller to influence homeostasis
of subordinate subsystems. These top-down signals convey the expectations of
future interoceptive signalsPaulus and Stein (2006), which imply changes to
steady state, i.e. target values, of certain hvars for the subsystems. The
changed hvar targets prompt the subsystems to initiate appropriate actions or
generate expectations for their own subsystems. This activity cascadeFriston
(2019) results in the engagement of the entire hierarchy towards meeting the
top-down expectations. For example, when bacteria or viruses invade the body
and cause tissue injury, one of the immune system’s responses is to produce
pyrogensKluger (1991). These chemicals are carried by the blood to the brain,
where they affect the functioning of the hypothalamus by effectively
increasing the target value for the core body temperature hvar. In response,
the hypothalamus raises the body’s temperature above the normal range, thereby
causing a fever.

Signalling among subsystems is critical for a hierarchical homeostatic system
to operate. Many organisms and bodily subsystems use chemical messengersTaylor
et al. (2010). It is difficult to convey different types of messages over long
distances quickly and at precise targets using chemical signalling. On the
other hand, neurons carry signals relatively rapidly and precisely and
networks of neurons are capable of performing computations. This allows a
nervous system based homeostatic controller to receive homeostatic state
updates from a large number of subsystems and send back control signals to
many subsystems, resulting in a fine grained mechanism to maintain
homeostasis. This provides a clue why, in animals species, the nervous system
took up the role as the highest level homeostatic controller.

###  2.3. Brain as a Homeostatic Controller

Over evolutionary time scales, nervous systems have proven to be highly
advantageous. Neural networks can represent homeostatic state information and
other useful sensory information, and complex computations can be performed
across signals from different modalities to produce highly adaptive control
signals. With a nervous system, organisms can optimize actions over longer
time horizons, which sometimes requires ignoring or acting against immediate
homeostatic drives.

This ability of a nervous system based controller to potentially drive
subsystems against their homeostatic imperatives may prove dangerous for the
organism. First, subsystems may be driven to states where they are no longer
able to maintain their homeostasis, leading to organ failure. Second, the
nervous system operates on partial information and may make decisions that are
not optimal. Third, the nervous system’s control algorithms, either
genetically specified or learned, may be insufficient to handle the current
situation. For a stable homeostatic system hierarchy, it is imperative that
controllers must treat bottom-up signals conveying subsystem homeostatic
state, such as a painful sensation, as valenced, i.e. conveying a ”good” or
”bad” feeling. The controllers must consider these signals as ”privileged”,
compared to other signals that may carry miscellaneous information.
Controllers must pay a high metabolic cost if they choose to ignore these
privileged signals. For example, ignoring hunger while voluntary fasting
requires constant upkeep of the resolve to fast. Note that although privileged
signals are present at every level in the hierarchy, they may become ”felt”
only at certain systems in the brain.

##  3\. Cognitive Homeostasis

Cognitive science typically makes a sharp distinction between mental activity
like planning and activity for physiological homeostasis. We propose that what
appears as resisting homeostatic drives during planning is in reality
following homeostasis of other, higher order variables.

The brain is organized roughly as a hierarchy of subsystemsHilgetag and Goulas
(2020). One question relevant for this discussion is whether the brain
represents a single homeostatic system or a hierarchy of homeostatic
subsystems. Note that a single homeostatic system can be organized internally
as a hierarchy of subsystems, but in that case, the subsystems do not
represent homeostatic variables of their own and thus act merely as functional
modules. Looking at what hvars the brain represents will help answer this
question.

###  3.1. Cognitive Homeostatic Variables

As a homeostatic regulator, the brain must represent the physiological hvars
such as the core body temperature, sodium levels, blood pressure, heart rate
and so on. Basic emotionsCeleghin et al. (2017) such as fear, anger, joy,
sadness, surprise and disgust can be interpreted as a set of hvars because the
brain attempts to regulate them. For example, fear is the expectation of
future harm and can be interpreted as an hvar whose steady state is the
expectation of ongoing safety. Interpreting emotions as hvars is appropriate
because emotions carry valence and by definition, valence is available only
when a quantity affects homeostasis of the organism. More complex attributes
such as the desire for well-being is also a homeostatic variable, because it
has a valence specific to an individual. Long term goals are hvars because
achieving those goals has a positive valence. Habits are hvars. Even short
term goals such as raising a hand can be cast as a temporary hvar. Successful
completion of every thought has a positive valence, while jumbled, incoherent
thoughts that do not lead to a resolution have a negative valence. As a
result, each thought can also be interpreted as a process that leads to the
target state of a temporary hvar. Indeed, all facets of cognition that have
valence can be cast as homeostasis via corresponding hvars. We call these
presumed hvars Cognitive Homeostatic Variables (”cognitive hvars”).

Cognitive hvars may be fleeting like those for thoughts, short term like those
for intentional actions, long term like a new year’s resolution, or lifelong
like career goals or social standing. They may be genetically specified, such
as parenting instincts, or learned, such as habits.

###  3.2. Why Cognitive Homeostasis?

Why re-frame entire cognition as homeostasis? First, doing so gives a
normative view of the behavior of all organisms, from single celled organism
to humans, and potentially all self-organized systems such as societies and
bacterial colonies. Second, it allows for the prioritization across
physiological drives and cognitive goals. For example, conflicting goals such
as energy homeostasis (hunger) and the desire to fast (not eat) can influence
behavior from different levels in the homeostatic hierarchy, and actions
result based on the relative influence of the two hvars and their interplay
with other hvars in the system. Third, evolution often results in the reuse of
components and patterns. In this case, it is conceivable that the neural
patterns useful for physiological homeostasis were repeated for cognitive
homeostasis as well, which presents an approach to investigate the evolution
of nervous systems. Forth, this unification opens up the possibility of new
ways of explaining free-will, self-hood and consciousness.

###  3.3. Modelling Cognitive HVars

Can we find a common computational description of how hvars are implemented in
biological systems, so it can be applied in the construction of artificial
agents? In nature, the mechanisms used for homeostasis vary substantially. For
example, in the mitochondrial membrane, proton gradient across the membrane is
maintained and is thus a homeostatic variable. When proton gradient shifts
away from desired levels, molecular machines called proton pumps are triggered
to restore the gradient. Locusts possess genetically specified visual neurons
(the descending contralateral movement detectors, DCMDs) that detect predatory
birds’ looming motionSanter et al. (2012) and trigger escape jumps. In this
case, the hvar is implemented as specific afferent connectivity pattern of the
DCMDs. Core body temperature hvar of endothermic animals is implemented as a
system of sensors, neural and chemical signalling and several populations of
neurons.

All these mechanisms can be summarized computationally as a set of desirable
trajectories described using partial stateSenft et al. (2017) of the system. A
trajectory is a sequence of changes described using certain partial state of
the system over $\tau$ time steps, like a short animation clip. An hvar
comprises the set of such trajectories that use the same set of partial state
features. For example, thermoregulation hvar is a set of trajectories where
starting state indicates a system that is either too cold or too warm, and
ending state is where the system is closer to the optimal temperature. Other
state features of the system, say blood glucose level, are not relevant to
describe these trajectories, thus ”partial state” above.

In other words, an hvar is a human-understandable interpretation of a set of
trajectories. In a cognitive homeostatic system, we may not have
interpretation for all trajectories. For this reason, when building such a
system, we work directly with trajectories and not hvars.

##  4\. Cognitive Homeostatic Agents

We propose building Cognitive Homeostatic Agents, artificial agents that are
based on the Cognitive Homeostasis principles.

Figure 1. Temporal Trace Trajectory. (A) shows a signal generated from a
signal source such as a sensory modality. (B) shows temporal trace, i.e.
motion blurred, version of the signal. Temporal Trace Trajectory is a $\tau$
time step sequence of the signal’s temporal trace. When storing in memory,
trajectories can be memorized using a partial state space, indicated by the
opaque areas in (C).

The system is organized as a hierarchy-like graph of homeostatic subsystems,
with each of those organized as a hierarchy-like graph of either homeostatic
or non-homeostatic subsystems. All subsystems are connected to orchestrate a
form of predictive processingKeller and Mrsic-Flogel (2018).

Each subsystem contains a memory bank that stores a set of events in the form
of temporal trace trajectories (see figure 1). A set of trajectories may
represent a homeostatic variable. For instance, the set of trajectories that
show core body temperature going from too warm or too cold to nominal
temperature represent the core body temperature hvar. Temporal traces are used
as a means of encoding time. They are computed as a decaying running sum of
signals with a decay factor $\delta$. Higher levels in the hierarchy generally
use temporal traces with smaller $\delta$ resulting in longer motion trails
and trajectories of longer time duration $\tau$ compared to lower levels.

Discussion of specific organization of individual subsystems and modules is
beyond of scope of this article.

###  4.1. System Operation

The system is bootstrapped by adding a set of trajectories to represent known
hvars at various levels in the system hierarchy. Over time, other subsystems
learn by remembering trajectories that result from the interaction between
sensory signals and the bootstrapped trajectories. Once such trajectories are
memorized, they may represent new hvars in corresponding subsystems. Thus, the
entire system gets wired with higher level hvars that are grounded to the
bootstrap hvars.

The system operation depends on predictive processingKeller and Mrsic-Flogel
(2018), evidence accumulationShadlen and Shohamy (2016) and top-down signal
generationAdams et al. (2013). Further investigation is needed to identify how
a system based on a hierarchy of autoassociative memory modules may produce
these characteristics. Here we present a sketch of a plausible, although
incomplete, mechanism. In a given homeostatic subsystem, at a given moment in
time, select a trajectory matching current conditions. Replay this winning
trajectory through top-down signalling over next $T$ time steps. If multiple
trajectories are matched with no clear winner, then replay top few matching
trajectories over the next few cycles of $T$ time steps each. Repeat till a
clear winner emerges by adjusting trajectory strength temporarily through a
short term plasticity mechanism.

Confident recall of a higher level trajectory represents an instantiated goal.
When such a trajectory is replayed to downstream subsystems, it encourages the
whole hierarchy to act according to that goal. Such long term goal
trajectories represent cognitive hvars and thus, planned behavior can be
interpreted as reflexive behavior corresponding to these higher level
cognitive hvars, blurring the boundaries between adaptive, reflexive behavior
and intentional, planned behavior.

Cycling through potential future trajectories is seen in the theta band
activation of rat hippocampusKay et al. (2020). This approach also aligns with
evidence accumulationShadlen and Shohamy (2016) and action planningMartiny-
Huenger et al. (2017) through simulated playback from memory. What modules in
the system participate in this mechanism remains an open question.

Unlike reinforcement learning, we propose that using an explicit reward signal
is not necessary. The more homeostatically helpful a particular trajectory,
the stronger it is imprinted in the memory. High utility trajectories exert a
stronger influence on downstream subsystem behavior when replayed. Note that
cognitive homeostasis allows hierarchical reinforcement learningBarto and
Mahadevan (2003) to emerge within certain subsystems. If a subsystem learns a
cognitive hvar that represents an RL-like reward, trajectories for such hvars
would be action policies that maximize corresponding rewards.

###  4.2. Implementation

Here we do not propose a specific implementation strategy. Instead, we lay out
important characteristics that an implementation must have. The system 1)
should preferentially remember favorable trajectories, 2) is able to
autoassociativelyGritsenko et al. (2017) recall and replay matching
trajectories based on the current state of the system, 3) is wired to perform
predictive processingKeller and Mrsic-Flogel (2018), 4) has an attention-like
mechanisms to identify, instantiate and eventually discard goal trajectories
and corresponding cognitive homeostatic variables and 5) has the ability to
learn using sequential, non I.I.D., data continuously without catastrophic
forgetting, such as using meta-learning approachesJaved and White (2019) to
continual learning.

##  5\. Concluding remarks

In this article, we described Cognitive Homeostasis, where cognition is
interpreted as a homeostatic process, operating over Cognitive Homeostatic
Variables. We then described Cognitive Homeostatic Agents that are built as
homeostatic hierarchies based on the Cognitive Homeostasis principles.

Much work is needed to explore various aspects of this approach. Is temporal
trace trajectory the appropriate unit of memory? What computational framework
is suitable for implementing autoassociative trajectory memory? Does the
playback of future trajectories use a synchronized clock across various
modules? How to integrate bottom-up sensory evidence, the winning trajectories
being played back and top-down expectations? How are transient cognitive hvars
instantiated? How would planning, curiosity, creativity, emotions and other
higher cognitive features emerge? Is this approach biologically plausible, and
if so, what are the predictions of this approach that can be investigated
through cognitive neuroscience and psychology? Can Cognitive Homeostasis
provide normative mechanisms for episodic memoryTulving (1983) and episodic-
like memoryClayton and Dickinson (1998)? What does this approach imply for
free-will, self-hood and consciousness? Also, a mathematical description based
on the Free Energy PrincipleKarl (2012) formulation would provide formal
grounding for cognitive homeostasis. Finally, implementing cognitive
homeostatic agents would provide an empirical validation of this approach.

We are cautiously optimistic that this approach would lead to a fruitful line
of investigation towards building sophisticated artificial agents. Some of the
reasons for our optimism are - 1) Temporal trace trajectories as the unit of
memory aligns with engram as a neural memory unitJosselyn and Tonegawa (2020).
2) Mechanism for sampling alternate futuresChadwick et al. (2016) as a means
of planning may enable planning in physical and conceptualConstantinescu et
al. (2016) spaces. 3) All reward-like signals in the proposed system are
internal and grounded on the initial set of homeostatic variables, which
obviates the need to define explicit rewards or arbitrary mechanisms to derive
rewards. 4) We hypothesize that symbols and symbolic manipulation may emerge
through the same mechanisms that instantiate and manage cognitive hvars. 5)
our approach produces a sparse factor graphBengio (2017) over events that may
result in causal reasoning capabilities. 6) Each homeostatic subsystem behaves
in ”System 1” Kahneman (2011) mode when prediction errors are small. Otherwise
it behaves in ”System 2” Kahneman (2011) mode, where alternate futures need to
be sampled over an extended time period in an attempt to minimize prediction
errors. 7) Using homeostasis provides a natural and grounded approach to
feelings and emotions.

##  6\. Acknowledgments

We would like to thank the careful and constructive anonymous reviewers, as
well as Lance Hughes for comments and constructive criticism. This work is
supported by i3ai.org.

## References

  * (1)
  * Adams et al. (2013) Rick A. Adams, Stewart Shipp, and Karl J. Friston. 2013\.  Predictions not commands: Active inference in the motor system.  _Brain Structure and Function_ 218, 3 (2013), 611–643.  https://doi.org/10.1007/s00429-012-0475-5
  * Alon et al. (1999) Uri Alon, Michael G Surette, Naama Barkai, and Stanislas Leibler. 1999.  Robustness in bacterial chemotaxis.  _Nature_ 397, 6715 (1999), 168–171.
  * Barto (2013) Andrew G Barto. 2013\.  Intrinsic motivation and reinforcement learning.  In _Intrinsically motivated learning in natural and artificial systems_. Springer, 17–47.
  * Barto and Mahadevan (2003) Andrew G Barto and Sridhar Mahadevan. 2003.  Recent advances in hierarchical reinforcement learning.  _Discrete event dynamic systems_ 13, 1 (2003), 41–77.
  * Bengio (2017) Yoshua Bengio. 2017\.  The consciousness prior.  _arXiv preprint arXiv:1709.08568_ (2017).
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners.  arXiv:2005.14165 [cs.CL]
  * Cannon (1929) Walter B. Cannon. 1929\.  Organization for Physiological Homeostasis.  _Physiological Reviews_ 9, 3 (1929), 399–431.
  * Celeghin et al. (2017) Alessia Celeghin, Matteo Diano, Arianna Bagnis, Marco Viola, and Marco Tamietto. 2017.  Basic Emotions in Human Neuroscience: Neuroimaging and Beyond.  _Frontiers in Psychology_ 8 (2017), 1432.  https://doi.org/10.3389/fpsyg.2017.01432
  * Chadwick et al. (2016) Angus Chadwick, Mark CW van Rossum, and Matthew F Nolan. 2016\.  Flexible theta sequence compression mediated via phase precessing interneurons.  _Elife_ 5 (2016), e20349.
  * Chen and Liu (2018) Zhiyuan Chen and Bing Liu. 2018.  Lifelong machine learning.  _Synthesis Lectures on Artificial Intelligence and Machine Learning_ 12, 3 (2018), 1–207.
  * Clayton and Dickinson (1998) Nicola S Clayton and Anthony Dickinson. 1998.  Episodic-like memory during cache recovery by scrub jays.  _Nature_ 395, 6699 (1998), 272–274.
  * Connolly and van Deventer (2017) Patrick Connolly and Vasi van Deventer. 2017.  Hierarchical recursive organization and the free energy principle: From biological self-organization to the psychoanalytic mind.  _Frontiers in Psychology_ 8, SEP (2017).  https://doi.org/10.3389/fpsyg.2017.01695
  * Constantinescu et al. (2016) Alexandra O Constantinescu, J. X. OReilly, and T. E. J. Behrens. 2016\.  Organizing conceptual knowledge in humans with a gridlike code.  _Science_ 352, 6292 (jun 2016), 1464–1468.  https://doi.org/10.1126/science.aaf0941
  * Corcoran and Hohwy (2017) Andrew Corcoran and Jakob Hohwy. 2017.  Allostasis, interoception, and the free energy principle: Feeling our way forward.  (2017), 1–16.  https://doi.org/10.31234/osf.io/zbqnx
  * Coren (1994) Stanley Coren. 1994\.  _The intelligence of dogs: Canine consciousness and capabilities_.  Free Press New York.
  * Craig and Craig (2002) a. D. Craig and a. D. Craig. 2002.  How do you feel? Interoception: the sense of the physiological condition of the body.  _Nature Reviews. Neuroscience_ 3, 8 (2002), 655–666.  https://doi.org/10.1038/nrn894
  * Csáji et al. (2001) Balázs Csanád Csáji et al. 2001\.  Approximation with artificial neural networks.  _Faculty of Sciences, Etvs Lornd University, Hungary_ 24, 48 (2001), 7\.
  * Deci and Ryan (2010) Edward L Deci and Richard M Ryan. 2010.  Intrinsic motivation.  _The corsini encyclopedia of psychology_ (2010), 1–2.
  * Dulac-Arnold et al. (2019) Gabriel Dulac-Arnold, Daniel Mankowitz, and Todd Hester. 2019\.  Challenges of real-world reinforcement learning.  _arXiv preprint arXiv:1904.12901_ (2019).
  * Emery (2006) Nathan J Emery. 2006\.  Cognitive ornithology: the evolution of avian intelligence.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 361, 1465 (2006), 23–43.
  * Friston (2013) Karl Friston. 2013\.  Life as we know it.  _Journal of the Royal Society Interface_ 10, 86 (2013).  https://doi.org/10.1098/rsif.2013.0475
  * Friston (2019) Karl J Friston. 2019\.  Waves of prediction.  _PLoS biology_ 17, 10 (2019), e3000426.
  * Garson (1991) James W. Garson. 1991\.  _What Connectionists Cannot Do: The Threat to Classical AI_.  Springer Netherlands, Dordrecht, 113–142.  https://doi.org/10.1007/978-94-011-3524-5_6
  * Godfrey-Smith (2016) Peter Godfrey-Smith. 2016\.  _Other minds: The octopus, the sea, and the deep origins of consciousness_.  Farrar, Straus and Giroux.
  * Gritsenko et al. (2017) Vladimir I Gritsenko, Dmitri A Rachkovskij, Alexander A Frolov, Ross Gayler, Denis Kleyko, and Evgeny Osipov. 2017.  Neural distributed autoassociative memories: A survey.  _arXiv preprint arXiv:1709.00848_ (2017).
  * Hilgetag and Goulas (2020) Claus C. Hilgetag and Alexandros Goulas. 2020.  ‘Hierarchy’ in the organization of brain networks.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 375, 1796 (2020).  https://doi.org/10.1098/rstb.2019.0319
  * Javed and White (2019) Khurram Javed and Martha White. 2019.  Meta-learning representations for continual learning.  _arXiv preprint arXiv:1905.12588_ (2019).
  * Joffily and Coricelli (2013) Mateus Joffily and Giorgio Coricelli. 2013.  Emotional Valence and the Free-Energy Principle.  _PLoS Computational Biology_ 9, 6 (2013).  https://doi.org/10.1371/journal.pcbi.1003094
  * Josselyn and Tonegawa (2020) Sheena A. Josselyn and Susumu Tonegawa. 2020.  Memory engrams: Recalling the past and imagining the future.  _Science_ 367, 6473 (2020).  https://doi.org/10.1126/science.aaw4325 arXiv:https://science.sciencemag.org/content/367/6473/eaaw4325.full.pdf
  * Kahneman (2011) Daniel Kahneman. 2011\.  _Thinking, fast and slow_.  Macmillan.
  * Karl (2012) Friston Karl. 2012\.  A Free Energy Principle for Biological Systems.  (2012), 2100–2121.  https://doi.org/10.3390/e14112100
  * Kay et al. (2020) Kenneth Kay, Jason E. Chung, Marielena Sosa, Jonathan S. Schor, Mattias P. Karlsson, Margaret C. Larkin, Daniel F. Liu, and Loren M. Frank. 2020.  Constant Sub-second Cycling between Representations of Possible Futures in the Hippocampus.  _Cell_ 180, 3 (2020), 552–567.e25.  https://doi.org/10.1016/j.cell.2020.01.014
  * Keller and Mrsic-Flogel (2018) Georg B. Keller and Thomas D. Mrsic-Flogel. 2018.  Predictive Processing: A Canonical Cortical Computation.  _Neuron_ 100, 2 (2018), 424 – 435.  https://doi.org/10.1016/j.neuron.2018.10.003
  * Kirchhoff et al. (2018) Michael Kirchhoff, Thomas Parr, Ensor Palacios, Karl Friston, and Julian Kiverstein. 2018.  The markov blankets of life: Autonomy, active inference and the free energy principle.  _Journal of the Royal Society Interface_ 15, 138 (2018).  https://doi.org/10.1098/rsif.2017.0792
  * Kluger (1991) Matthew J. Kluger. 1991\.  Fever: Role of Pyrogens and Cryogens.  _Physiological Reviews_ 71, 1 (1991), 93–127.  https://doi.org/10.1152/physrev.1991.71.1.93 arXiv:https://doi.org/10.1152/physrev.1991.71.1.93  PMID: 1986393.
  * Man and Damasio (2019) Kingson Man and Antonio Damasio. 2019.  Homeostasis and soft robotics in the design of feeling machines.  _Nature Machine Intelligence_ 1, 10 (2019), 446–452.  https://doi.org/10.1038/s42256-019-0103-7
  * Martiny-Huenger et al. (2017) Torsten Martiny-Huenger, Sarah E. Martiny, Elizabeth J. Parks-Stamm, Elisa Pfeiffer, and Peter M. Gollwitzer. 2017.  From conscious thought to automatic action: A simulation account of action planning.  _Journal of Experimental Psychology: General_ 146, 10 (2017), 1513–1525.  https://doi.org/10.1037/xge0000344
  * McCloskey and Cohen (1989) Michael McCloskey and Neal J Cohen. 1989.  Catastrophic interference in connectionist networks: The sequential learning problem.  In _Psychology of learning and motivation_. Vol. 24. Elsevier, 109–165.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ (2013).
  * Moore-Ede (1986) M. C. MC Moore-Ede. 1986\.  Physiology of the circadian timing system: Predictive versus reactive homeostasis.  _The American Journal of Physiology_ 250, 5 (may 1986), 737–752.  https://doi.org/10.1152/ajpregu.1986.250.5.R737
  * Morville et al. (2018) Tobias Morville, Karl Friston, Denis Burdakov, Hartwig R. Siebner, and Oliver J. Hulme. 2018\.  The Homeostatic Logic of Reward.  _bioRxiv_ (2018), 242974\.  https://doi.org/10.1101/242974
  * OpenAI et al. (2019) OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, Jonas Schneider, Nikolas Tezak, Jerry Tworek, Peter Welinder, Lilian Weng, Qiming Yuan, Wojciech Zaremba, and Lei Zhang. 2019.  Solving Rubik’s Cube with a Robot Hand.  arXiv:1910.07113 [cs.LG]
  * Oster and Wilson (1978) George F Oster and Edward O Wilson. 1978.  _Caste and ecology in the social insects_.  Princeton University Press.
  * Paulus and Stein (2006) Martin P Paulus and Murray B Stein. 2006.  An insular view of anxiety.  _Biological psychiatry_ 60, 4 (2006), 383–387.
  * Ross-Gillespie and Kümmerli (2014) Adin Ross-Gillespie and Rolf Kümmerli. 2014.  Collective decision-making in microbes.  _Frontiers in microbiology_ 5 (2014), 54.
  * Russell Stuart and Norvig (2009) J Russell Stuart and Peter Norvig. 2009.  _Artificial intelligence: a modern approach_.  Prentice Hall.
  * Santer et al. (2012) Roger D. Santer, F. Claire Rind, and Peter J. Simmons. 2012\.  Predator versus Prey: Locust Looming-Detector Neuron and Behavioural Responses to Stimuli Representing Attacking Bird Predators.  _PLoS ONE_ 7, 11 (2012).  https://doi.org/10.1371/journal.pone.0050146
  * Scheutz (2012) Matthias Scheutz. 2012\.  The affect dilemma for artificial agents: should we develop affective artificial agents?  _IEEE Transactions on Affective Computing_ 3, 4 (2012), 424–433.
  * Senft et al. (2017) Emmanuel Senft, Séverin Lemaignan, Paul Baxter, and Tony Belpaeme. 2017.  Toward supervised reinforcement learning with partial states for social HRI.  (2017).
  * Shadlen and Shohamy (2016) Michael N N. Shadlen and Daphna Shohamy. 2016.  Decision Making and Sequential Sampling from Memory.  _Neuron_ 90, 5 (2016), 927–939.  https://doi.org/10.1016/j.neuron.2016.04.036
  * Simard et al. (1997) Suzanne W Simard, David A Perry, Melanie D Jones, David D Myrold, Daniel M Durall, and Randy Molina. 1997\.  Net transfer of carbon between ectomycorrhizal tree species in the field.  _Nature_ 388, 6642 (1997), 579–582.
  * Sterling (1988) Peter Sterling. 1988\.  Allostasis: a new paradigm to explain arousal pathology.  _Handbook of life stress, cognition and health_ (1988).
  * Taylor et al. (2010) Ann Gill Taylor, Lisa E Goehler, Daniel I Galper, Kim E Innes, and Cheryl Bourguignon. 2010.  Top-down and bottom-up mechanisms in mind-body medicine: development of an integrative framework for psychophysiological research.  _Explore_ 6, 1 (2010), 29–41.
  * Tulving (1983) Endel Tulving. 1983\.  Elements of episodic memory.  (1983).


SEM/XRD Inspection of the thermally treated niobium samples with SEM in
secondary electron mode did not reveal precipitates of new phases compared to
the untreated surface (see Figure fig:SEM_annealed). A few organic particles
from the contact with environment as well as calcium oxide that were also
traced by XPS were found. According to the EDX elemental mapping, some areas
of the BCPed and annealed at 200230 samples contained trace amount of
phosphorous (P/Nb=0.01, $E_{acc}$=310) and fluorine (F/Nb=0.006-0.1), while in
400-sample no fluorine could be detected by EDX. No P-containing particles
were found. X-ray diffraction analysis did not reveal any reflexes other than
niobium (not shown).

[width=]figures/SEM_annealed_v2.png

Figure : SEM images of the annealed niobium surface: (a) 200°C/11.5h, (b)
230°C/15h, (c) 300°C/3h, (d) 400°C/3h.

##  Initial State: Niobium after BCP

The survey low-resolution EDCs (pass energy 50, $h\nu$=1000) of the niobium
samples after BCP demonstrate classical features associated with niobium,
oxygen, carbon, and fluorine (see Figure ).

[width=]figures/XPSSurveySpectra300C_v1.png

Figure : Survey spectra ($h\nu$=1000) collected from the niobium sample after
BCP (black), annealing at 300 °C (red), and subsequent air exposure (blue).

The typical high-resolution EDCs ($h\nu$=1000) for the regions of interest of
a niobium sample after the chemical etching are shown in Figure (a-f).

[width=]figures/XPS_typical_v1.png

Figure : Typical high-resolution EDCs of P 2p (a), C 1s (b), O 1s (c), N 1s
(d), F 1s (e), and Nb 3d (f) of niobium after the BCP-treatment before the
thermal anneal collected at $h\nu$=1000. Inset: EDC of N 1s collected at
$h\nu$=480.

A weak peak P 2$p_{3/2}$ at BE=134.25 (Figure (a)) was detected which
according to rumble1992nist, siow2018xps may correspond to phosphate (PO4)– or
metaphosphate (PO3)–. The roughly estimated position of P 2$s$ peak in the
survey EDCs (not visible in Figure ) at 191.7 according to wagner2003nist may
be a characteristic of (PO3)–. According to our data, BE of EDC of P 2$p$ is
close to valence five, so phosphorus is most likely in P+5-state. From these
data it is hard to distinguish whether it is a salt, a functional group or an
anion. Several compounds originating from the contact of niobium or its oxides
with phosphoric acid may fulfil these criteria. Among the most probable ones
is (H2PO4)– anion which may be incorporated into the oxide films typically
anodic ones jouve1986x, bokii1989x, or being adsorbed on the surface
francisco2004surface, pavan2005adsorption. Niobium phosphate NbOPO4 was also
reported okazaki1993surface. Since the intensity of P 2$p$ peak is very weak,
and it rises as the energy of the primary X-ray beam decreases, we deem that
it may rather correspond to the ionic species adsorbed to niobium surface
rather than being incorporated into the niobium-oxide film. Trace amount of
silicon (atomic ratio Si/Nb = 0.01-0.02) in the state of organic silicon
(BE=102.3) was also detected (see Figure ).

[width=]figures/XPS_Si2p_detailed_v2.png

Figure : Si 2p ($h\nu$=480) before and after annealing of Nb at 300 (a) and
400 (b).

The C 1$s$ core-level is represented by carbonaceous adsorbates, a
contamination layer commonly found at the surfaces of solids after the air
exposure (Figure (b)). It was fitted with four contributions. The most intense
is located at 285.1285.2 and interpreted as the non-graphitic C–C and C–H
bonds. The peaks of lower intensity may be assigned to C-OH, C-O-C (286.7),
C=O (288.2), and O-C=O (289.5). O 1$s$ is fitted with three peaks located at
530.9, 531.8 and 533.1 (Figure (c)). The first is the most intense and is
usually interpreted as O-2 ions in niobium oxides while the second peak is
assigned to low-coordination-number O-1 ions that are compensating for
deficiencies in the oxide subsurface dupin2000systematic. The feature at
around 533.1 is associated with weakly adsorbed species such as aliphatic
C-O-C, C-OH, etc. and possibly (H2PO4)– beamson1992high. The intensity of N
1$s$ was very weak at $h\nu$=1000 (Figure (d)). Two broad peaks could
presumably be fitted at around 400 and 395.7 that could be related to the
nitrogen containing species adsorbed at the surface inhomogeneities after
exposure to the ambient environment and to nitrogen in Nb lattice,
respectively. For more information, the EDC of N 1$s$ taken at $h\nu$=480 is
presented in which only a peak at 400.78 from adsorbates is present (Figure
(d), inset). At this photon energy the probing depth is much smaller
($\approx$2.25) so nitrogen in niobium might not be traced. F 1$s$ is
represented by a single peak located at BE=685.15(4) confirming the chemical
bonding of fluorine with niobium (Figure (e)) luo2008hydrogen,
mandula2017electrochemical. A hardly noticeable shoulder at 686.8687.0
appeared during measurements at room temperature (which will be discussed
later.) The Nb 3$d$ spectrum ($h\nu$=1000) with the fitted components typical
for all niobium samples treated with BCP is presented in Figure (f). Peak
maxima of the components, as well as the respective FWHMs estimated by the
procedure described in the previous section are summarized in Table . The
relative percentage areas of the fitted peaks are summarized in Table .

Table : Binding energy shifts with respect to Nb0, $\Delta$BE, and the full width at half-maximums (FWHMs) of the fitted components of Nb 3d core-level obtained for the initial niobium samples. Oxidation state |  Nb+5 |  Nb+4 |  Nb+3 |  Nb+2 |  Nb+1 |  Nb+0.6 |  Nb+0.4 |  Nb0
---|---|---|---|---|---|---|---|---
BE shift, eV | 5.95±0.05 | 4.14±0.07 | 3.08±0.02 | 1.91±0.01 | 1.02±0.01 | 0.51±0.03 | 0.31±0.02 | 202.00±0.01
FWHM, eV | 1.18±0.07 | 1.16±0.00 | 1.07±0.00 | 1.13±0.03 | 0.75±0.00 | 0.68±0.04 | 0.54±0.03 | 0.47±0.00
|  |  |  |  |  |  |  |
Table : Relative area of the fitted components of Nb 3d core-level
($h\nu$=1000) of niobium before and after the respective anneal.

Treatment | Nb+5 | Nb+4 | Nb+3 | Nb+2 | Nb+1 | Nb+0.6 | Nb+0.4 | Nb0 | Nb-C | Nb-O-F | Nb-O-F (II)
---|---|---|---|---|---|---|---|---|---|---|---
200 anneal
Initial | 67.36±0.36 | 1.61±0.30 | 0 | 6.61±0.47 | 0.48±0.26 | 0.01±0.01 | 13.58±0.39 | 10.35±0.44 | 0 | 0 | 0
3h | 51.21±0.73 | 12.70±0.60 | 0 | 7.25±0.49 | 0.23±0.45 | 1.28±0.37 | 14.23±0.49 | 13.11±0.59 | 0 | 0 | 0
11.5h | 43.52±0.61 | 16.74±0.30 | 0.53±0.51 | 9.85±0.49 | 0.31±0.36 | 0.28±0.23 | 12.99±0.27 | 15.77±0.30 | 0 | 0 | 0
230 anneal Nb 3d core level was measured at hv = 900 eV
Initial | 68.67±0.53 | 0.94±0.46 | 0 | 6.23±0.67 | 2.11±0.09 | 0 | 12.70±0.87 | 9.35±0.90 | 0 | 0 | 0
3h | 30.76±0.76 | 13.60±0.48 | 13.32±0.88 | 7.52±0.71 | 1.64±0.28 | 0 | 14.32±0.67 | 15.50±0.82 | 0 | 0 | 3.34±0.36
15h | 16.88±0.47 | 11.54±0.30 | 14.17±0.20 | 16.37±0.60 | 2.32±0.14 | 0 | 17.27±0.42 | 15.66±0.49 | 1.72±0.17 | 0 | 4.06±0.16
Air | 77.14±0.31 | 3.03±0.30 | 0 | 3.19±0.22 | 1.36±0.14 | 2.30±0.08 | 7.60±0.09 | 3.35±0.11 | 1.45±0.12 | 0 | 0.57±0.16
300 anneal
Initial | 65.94±0.76 | 1.11±0.67 | 0 | 6.07±0.76 | 3.33±0.61 | 0.01±0.01 | 15.94±0.76 | 7.61±0.58 | 0 | 0 | 0
3h | 5.42±0.60 | 1.93±0.54 | 1.39±0.97 | 18.88±0.67 | 13.85±0.69 | 4.70±0.65 | 22.28±0.73 | 17.39±1.03 | 6.82±0.52 | 2.71±0.54 | 4.62±0.45
Air | 71.18±0.37 | 6.26±0.34 | 0±0 | 3.67±0.26 | 0.80±0.23 | 1.87±0.15 | 10.70±0.13 | 1.73±0.12 | 2.09±0.05 | 0.57±0.02 | 1.13±0.17
400 anneal
Initial | 75.67±0.43 | 0.67±0.38 | 0 | 4.89±0.30 | 0.09±0.17 | 0 | 14.58±0.30 | 4.10±0.33 | 0 | 0 | 0
3h | 0 | 0 | 0 | 4.89±0.31 | 9.76±0.42 | 16.21±0.22 | 18.53±0.41 | 34.11±0.45 | 11.57±0.40 | 1.71±0.27 | 3.21±0.37
Air | 59.49±0.23 | 3.64±0.25 | 0±0 | 12.34±0.41 | 0.98±0.16 | 0.25±0.18 | 10.12±0.26 | 6.95±0.14 | 6.24±0.12 | 0 | 0
|  |  |  |  |  |  |  |  |  |  |

The EDC of Nb 3$d$ collected from the BCPed niobium was dominated by the Nb+5
state (Nb2O5) with minor contributions from lower oxides. At that stage it has
been assumed that fluorine may be distributed within the native oxide layer as
Nb2O5-xFx or/and located under the pentoxide. Analysis of the data presented
in Table  shows, that in the initial niobium samples the area of Nb+0.6 peak
component is approximately zero, and it is getting pronounced upon baking.
Thus, it exists only in the niobium subjected to the thermal treatment, and is
the highest for 400-baked Nb. It was reported that oxidation of niobium at
400500 proceeds via creation of Nb6O phase chang1969phase which is close to
Nb+0.4 state. Alternatively, it can also be assumed that Nb+0.6 may refer to a
niobium bond with the element other than oxygen (for example, fluorine
interstices) which is created during anneal. From the data of Table  one can
approximately estimate the atomic concentration of oxygen ions found in the
state of niobium oxides (Nb+5, Nb+4, Nb+3, Nb+2) and interstitials (the
remaining states) within the XPS-information depth. For calculation, the
number of associated oxygen ions in each oxidation state per Nb-atom was
considered as well as the Nb+5 peak area was corrected (scaled by 0.5) as
Nb2O5 contains two atoms of niobium. The average values obtained for BCP-ed
niobium are 58.6 and 5.3 for niobium oxides and interstitials, respectively.
These values were used for the calculation of oxygen-concentration depth
profiles. Upon analysis of the room-temperature Nb 3$d$ core-levels of the
BCPed niobium samples collected from the same location it was revealed that
Nb+4 state was initially absent and it appeared subsequently while the other
core-levels were recorded. At that, a hardly pronounced shoulder within F 1$s$
at 686.8 appeared which is interpreted as C-F bond nikolenko2002xps. Thus, the
formation of Nb+4 state at room temperature is most likely associated with the
formation of vacancies in the pentoxide due to F-ions transition to an
adsorbed state on the sample surface and interaction with carbonaceous layer
at the niobium surface.

##  Heat-Treated Niobium: Nd 3d Core-Level.

The Nb 3$d$ spectra with the fitted components of the samples baked at
200°C/11.5h, 230°C/15h, 300°C/3h, and 400°C/3h are presented in Figure (a). As
an example, the evolution of Nb 3$d$ region measured in-situ during the heat
treatment at 300 is demonstrated in Figure (b).

[width=]figures/XPS_Nb3d_v1.png

Figure : (a) The fitted EDCs of Nb 3d region ($h\nu$=1000) of niobium after
the respective thermal anneal at 200 °C/11.5 h, 230 °C/15 h, 300 °C/3 h, 400
°C/3 h. (b) Evolution of high-resolution EDCs of Nb 3d region during 300 °C
anneal. The spectra are unaltered experimental EDCs with a subtracted Shirley
background and non-normalized intensity. The FWHMs of doublets corresponding
to niobium of higher oxidation states (Nb+5 and Nb+4) increased during baking.
For example, the FWHM of Nb+5 increased from 1.20 to 1.84 after 3 and to 2.10
after 11.5 of baking at 200 ($h\nu=\SI{1000}{\electronvolt}$). For 230, the
FWHM of Nb+5 changed from 1.14 to 2.90 (3) and to 3.1 (15)
($h\nu=\SI{900}{\electronvolt}$). Similar tendency was observed for the FWHM
of Nb+4. On the other hand, the FWHM of a newly created bond, like for example
Nb+3, had a larger value ($\approx$1.41.5) as compared to the average FWHMs of
the fitted components within Nb 3$d$ core-level measured at room-temperature
(see Table ). The broadening of spectral lines can be related to the defect
structure of the crystal lattice and intermediate valence states
gopel1984surface. The relative percentage areas of the fitted peaks after 3
anneal are summarized in Table . The data for 200°C- and 230°C-samples that
were annealed for longer time are also presented. During baking at 200 changes
occurred in Nb-O system within the high oxidation states, Nb+5 and Nb+4.
Particularly, the relative area of Nb+5 decreased from 67% to 51% after 3 of
baking and to 43% after 11.5. At that, the relative area of Nb+4 increased
from 1.6 to 12.7% after 3 and to 16.7% after 11.5. After 3 baking, the
contribution of Nb+2 and Nb0 increased insignificantly. After 11.5, the
surface was represented mainly by Nb+5 with nearly equal contributions of
Nb+4, Nb+0.4, Nb0 (within 13–17%) and slightly less Nb+2 (9.8%) (Figure (a,
200). At 230 the changes in Nb-O system become more evident. The percentage of
Nb+5 drops from 69% to 31% in 3, and to 17% in 15. The percentage of Nb+4 is
similar to 200-sample, and maintained at 11–13% during baking. Additionally,
Nb+3 bond appears and remains at 13–14% after both 3 and 15 baking time. The
percentage of Nb+2 after 3 was comparable (7.5%) to 200/3, but finally
increased to 16%. The content of Nb+1 and Nb+0.6 was small or close to zero in
the both samples at various baking times. The quantity of Nb+0.4 changed
insignificantly with respect to the initial values of $\approx$13% in the both
cases (see Table ). Thus, after 15 at 230 the surface is composed almost
equally of Nb+5, Nb+3, Nb+2, Nb+0.4, Nb0 and slightly less Nb+4 (Figure (a,
230)). For the niobium annealed at 300/3, Nb+5, Nb+4, Nb+3 chemical states
almost completely reduced (Figure (b)), and the percentage of the remaining
chemical states was distributed in the descending order as Nb+0.4(22%),
Nb+2(19%), Nb0(17%), Nb+1(14%) and Nb+0.6 (5%) (Figure (a, 300)). At 400, the
oxides Nb+5, Nb+4 and Nb+3 vanished, and the percentage of chemical states
within the Nb 3$d$ core-level was dominated by Nb+0.4(19%) and Nb+0.6 (16%)
followed by Nb+1 (10%) and Nb+2(5%) (Figure (a, 400)). The maximal
contribution of metallic Nb0 was retained within the information depth as
compared to the other samples. This might be related to a relatively high
diffusion rate of oxygen interstices in Nb-lattice. The calculated atomic
concentrations of oxygen ions found in the state of niobium oxides and
interstitials for the baked niobium are presented in Table . Thus, the number
of interstitials in the vicinity of oxide layer approximately doubled at
300400 as compared to the BCPed niobium. For the prolonged treatments at
200230, these numbers are compared to the initial ones. Table : The
concentration of oxygen, at%, in niobium oxides and interstitial phases within
the XPS-information depth in the annealed Nb samples calculated using the
fitted components of Nb 3d core level measured at $h\nu$=1000. The values for
the BCPed Nb are shown in the brackets for comparison.

Treatment 200°C/11.5h 230°C/15h* 300°C/3h 400°C/3h Oxygen in NbOx
55.71±(58.64±) 47.19±(58.86±) 24.52±(57.50±) 4.66±(61.86±) Oxygen in Nb
lattice 3.52±(4.27±) 4.80±(5.19±) 11.62±(6.75±) 11.86±(4.54±) *Nb 3d core
level was measured at $h\nu=\SI{900}{\electronvolt}$.

The changes of the relative percentage areas of the fitted peaks during these
thermal treatments are demonstrated in Figure (a-d).

[width=]figures/Components_area_v1.png

Figure : The relative peak areas of the fitted components during the
respective thermal anneal: (a) 200°C/11.5h, $h\nu$=1000; (b) 230°C/15h,
$h\nu$=900 (c) 300°C/3h, $h\nu$=900; (d) 400°C/3h, $h\nu$=900.

At 200 the conversion of Nb+5 to Nb+4 occurs synchronously and gradually
(Figure (a)); Nb+2 and Nb+0.4 rise slowly; Nb+3, Nb+1 and Nb+0.6 remain close
to zero. At 230 reduction of Nb+5 proceeds much faster and a maximum of Nb+4
percentage (17%) was observed at the time when the area of Nb+5 was at about
40–45% (Figure (b)). Furthermore, at that moment the formation of Nb+3 bonds
starts. The percentages of Nb+4, Nb+3, Nb+2, Nb+0.4, Nb0 did not change
significantly during the whole duration of baking. Approximately at the same
time a new bond is formed which was assigned to Nb-F bond (associated with the
changes within F 1$s$, which will be described later). At 300, Nb+4 and Nb+3
states appeared during the heating-up stage and vanished by the end of baking.
The Nb+2 state was at its maximum (29%) when the baking temperature was
reached and it slightly dropped (22%) during baking (Figure (c)). The Nb+1
percentage raised at the time when Nb+5 vanished. Noteworthy that Nb+1
chemical state was not pronounced at 200230. Contribution of Nb+0.4 increased
insignificantly and stayed at $\approx$20%. At 400, the Nb+5–Nb+1 states were
reduced during the heating stage. The Nb+0.4 state behaved similar to 200300,
and it raised as the baking temperature increased (200 °C: 14%, 230 °C: 16%,
300 °C-400 °C: 18-23 °C). The Nb+0.6 state appeared simultaneously with Nb+1,
reached 18% and, unlike Nb+1, did not disappear during baking (Figure (d)).
However, the Nb+1 state appears upon cooling to room temperature. Comparing
the Nb-O states upon 300 and 400 anneals, in the first case the Nb+2 state
(corresponding to NbO oxide) still remained on the surface while in the latter
only the chemical states corresponding to oxygen interstices were left. The
maxima in the time dependence of the relative peak areas of the respective
chemical states were observed during the heating-up, i.e. when the baking
temperature was not reached. Since the similar maxima were observed in the
kinetics of the niobium oxide reduction at constant temperatures
king1990kinetic, it can be concluded that the maxima are inherent to the
kinetics of the formation and dissolution processes of niobium oxides.

##  Heat-treated Niobium: Impurities.

Analysis of C 1$s$, N 1$s$, P 2$p$ and Si $2p$ core levels showed no evidence
of chemical interaction of these elements with niobium at 200. The intensity
of Si 2$p$ peak attributed to organic silicon diminished upon baking at
200300, and completely vanished after 400 bake out (Figure (a,b)).

###  Phosphorus

From the analysis of EDC of P 2$p$ (Figure (a)) it follows that the
phosphorous-containing species were stable at 200, while partially (230, 300)
or completely (400) reacted with niobium at higher temperatures transforming
for the most part to niobium phosphide characterized by P-3 (BE=128.50) and
also P-2 states (129.15), as well as a free-state phosphorus, P0 (130.25),
upon the respective anneal. Thus, at temperatures above 230 the chemisorption
process of phosphorus was facilitated. At 200 the quantity of P+3 depleted
(probably by desorption) as was determined at $h\nu=\SI{480}{\electronvolt}$
(Figure (a)) which was not evident at $h\nu=\SI{1000}{\electronvolt}$ owing to
a lower surface sensitivity and small concentration of phosphorus.

[width=]figures/XPS_OCFP_detailed_v3.png

Figure : EDCs of O 1s, C 1s, F 1s ($h\nu$=1000) and P 2p ($h\nu$=480) at the
end of the respective baking: 200 °C/11.5h, 230°C/15h, 300°C/3h and 400°C/3h.
The EDCs are normalized to the area of Nb 3d core-level.

###  Fluorine

The position of the main F 1$s$ peak shifts to lower BEs by 0, 0.1, 0.2 and
0.6 for niobium baked at 200/11.5, 230/15, 300/3 and 400/3 indicating the
change of the chemical state of fluorine during baking. Additionally, a weak
shoulder emerges at $\approx$687 (Figure (b)). The fitting procedure of F 1$s$
revealed four components positioned at 684.55, 685.15, 686.70, and 688.1. In
general, metal fluorides reveal peaks in the range of 684685.5 while C-F, O-F,
and F-F reveal the peaks at 686689 nikolenko2002xps, tressaud1996nature,
ninomiya1985reaction. Therefore, the component at 685.15 that is present in
the initial samples and looses its intensity upon baking, has been interpreted
as the Nb-F-O state. The component at 684.55, on the contrary, increases and
is associated with the Nb-F bonds. The peaks at 686.70 and 688.1 can be
assigned to the adsorbed fluorine species possibly bonded to oxygen or
hydrogen since no peaks associated with fluorine have been observed within C
1$s$. During baking at 230400 new doublets emerged within Nb 3$d_{5/2}$
positioned at around 203.63 and 204.30 ($\Delta$BE equals 1.63 and 2.30
respectively). We suppose that they could be the Nb-O-F and Nb-F states (the
estimated quantity of phosphorus is too small to originate sufficient peak-
components within Nb 3$d$).

###  Carbon

At 200/11.5 the maximum of the C 1$s$ peak was found to be shifted to lower BE
upon baking, i.e. a new component at 284.5 emerged. The peak may be referred
to the adsorbed to niobium carbon atoms that are formed upon decomposition of
C-C or C-O molecules as a preceding state of carbon prior to the formation of
niobium carbide (Figure (c)). Alternatively, the peak at 284.5 may be refereed
to the carbonaceous species produced by the X-ray beam during measurements.
After 15 at 230 a small shoulder at $\approx$282.9 could be additionally
resolved within C 1$s$ when the spectrum was left for a longer time for
accumulation of the signal after the baking (Figure (c)). It could be assigned
to the formation of Nb-C or Nb-C-O bonds (because of the small percentage, it
was not included in the fitting model for building the plot in Figure (b), but
in the analysis of the chemical composition after the baking). At 300 and at
400 the Nb-C peak was located at $\approx$282.65. Within the Nb 3$d$ the peak
at approximately 203.37 was assigned to Nb-C. It has been established that the
interaction of niobium with carbon and other impurities occurs rapidly when
the percentage of Nb+5 within Nb 3$d$ is below 30% for 230, 300 and 400 baked
samples, which equals to approximately 1 thickness of Nb2O5.

###  Oxygen

It was identified that at 300 and 400, the maximum of the O 1$s$ peak shifts
to lower BE which is associated with the emerging of a new component at 530.3
(Figure (d)). It is more intense at 400 and gets more pronounced at lower
photon energies of X-ray beam which implies it originates from the very
surface. This peak may be associated with the formation of Nb-O-C bonds or be
a characteristic feature of the niobium oxides with lower oxidation numbers.

##  Quantitative Analysis

The near-surface distribution of elements was estimated in two different ways.
In the first, the data collected at $h\nu$=900 or 1000 were used. In this case
the areas of the peaks of the respective EDCs were corrected as described in
nanotechnology2005ea. The advantage of this method is a larger information
depth. Despite it may have a substantial error it can be used for comparison
of the samples. In the second approach, the EDCs were acquired at various
$h\nu$ providing the constant kinetic energy of photoelectrons and thus the
equal information depth for all elements girard2012chemical. The concentration
of elements was calculated according to the following expression:
$n_{i}=\frac{S_{i}}{\sigma_{i}F_{i}\displaystyle\sum_{j=1}^{k}\frac{S_{j}}{\sigma_{j}F_{j}}},$
(1) where $n_{i}$ is the atomic concentration of an element $i$, $S_{i}$ is
the area of EDC originating from a particular core level $j$ of an element
$i$, $\sigma_{i}$ is the photoionization cross section of the particular core
level of element $i$, $F_{i}$ is the photon flux at a particular $h\nu$. The
first approach is of particular interest for the initial samples, as they have
a carbonaceous adsorbate layer on the surface and a thicker niobium pentoxide.
The second method of a constant kinetic energy was used additionally to
analyze the samples annealed at 300 and 400. In the both approaches the
effects related to the angular distribution of the photoelectrons
($L(\beta,\gamma)\approx 1$) were minimized by the setup configuration. The
inhomogeneity of the element distribution over depth was not considered. An
estimate of the near-surface composition of niobium before and after the
baking by the first and second approaches is presented in Tables
LABEL:tab:XPS_concentration_1000ev and LABEL:tab:XPS_concentration_CKE,
respectively. The information on the percentage of elements that are
chemically bound to Nb is also presented. It allows to write the surface
composition of the initial and baked samples estimated according to the first
approach in the following form. NbO1.99F0.20N0.04 initial and NbO1.64F0.3
baked at 200. NbO1.73F0.20 initial and NbO1.03C0.015F0.35P0.006 baked at 230.
NbO1.91F0.16N0.04 initial and NbO0.56C0.056F0.22P0.009N0.019 baked at 300.
NbO2.10F0.17N0.03 initial and NbO0.245C0.127F0.107P0.023N0.030 baked at 400.
By the second approach we get the following. NbO1.38F0.079N0.007 initial and
NbO0.506C0.052F0.111P0.018N0.006 baked at 300. NbO1.62F0.09N0.005 initial and
NbO0.248C0.128F0.05P0.037N0.006 baked at 400. The nitrogen interstitials were
also considered. Comparing the unbaked niobium samples, the estimated amounts
of O and F are smaller when they are measured at equal kinetic energy which is
primarily related to a smaller information depth (2.3 at $h\nu$=480 against
4.6 at $h\nu$=1000 in Nb2O5), and as a consequence to a presumable probing of
surface adsorbates that are always present at the surface prior to a heat-
treatment. For the baked samples consistent results were obtained for oxygen
and carbon. However, the second approach resulted in a larger amount of
phosphorus and a smaller amount of fluorine which may be related to the non-
homogeneous distribution of these elements over the depth. Table : Approximate
distribution of elements within the near-surface region of the samples,
($h\nu$=1000). For the niobium baked at 230 and 300, the amount of the
detected fluorine at the surface after the air oxidation significantly
diminished as compared to the just-baked niobium surface ($\approx$ 10 times),
and for the 400 sample it was not detectable in our experimental conditions
(Figure LABEL:fig:XPS_after_aird”). The possible reason of the depletion of
fluorine in the oxidized samples is the high enthalpy of formation of niobium
pentoxide (so fluorine remains in niobium under the oxide layer) and the
instability of the formed fluorides in an air environment.

###  Carbon

After the air exposure, the Nb-C/Nb ratio remained the same in the 230 sample,
while it decreased about 10 times for the 300 sample, and only 1.3 times for
the 400 sample (Table LABEL:tab:XPS_concentration_1000ev). Thus, the amount of
the reacted carbon within the XPS information depth in the 400 sample after
the air exposure exceeded the analogous for the 300 sample by 20 times, while
right after the baking they differed only 2.3 times. One could suggest that at
300 a portion of the detected Nb-C bonds was also shared with oxygen as Nb-
C-O, and upon air oxidation a portion of these oxygen atoms participated in
the formation of niobium oxide.

###  Phosphorus

In the 230 and 300 baked samples both the free (P0) and the bound phosphorus
(P-2, P-3) that were formed during the baking transformed to the P+4 state
(BE(P $2p_{3/2}$)=134.4) upon air exposure. Thus, no phosphides were detected
in these samples. In the 400, the P-3 state still remained along with the P+4
state. The total P/Nb ratio slightly decreased as compared to the unbaked
samples: from 0.05 to 0.04 and from 0.09 to 0.07 for 230 and 400 samples,
respectively. It was not obvious for 300 (P/Nb=0.02 before and after the air
exposure), since the quantity of this element was relatively small in the
initial sample. Also, a small non-uniformity in the distribution of P over the
surface could be present. It was checked whether the P-containing particle
agglomerates could influence the measured photoelectron current between
different measurement runs. For this purpose several areas have been tested by
XPS. The SEM/EDX inspection also has not revealed such particles. For example,
for the 400 sample the EDX mapping in several locations revealed the P/Nb-
ratio of 0.12–0.02 at accelerating voltages in the range 310, correspondingly.

###  Oxygen

The amount of oxygen in the samples is determined substantially by the formed
surface oxide layer. The calculated thicknesses of the top Nb2O5 layer using
the data collected at $h\nu=\SI{1000}{\electronvolt}$ for 230, 300 and
400-baked niobium followed by air exposure are presented in Table
LABEL:tab:Pentoxide_thickness. These values correlate with the amount of
carbon chemically bound to niobium, i.e. the more Nb-C percentage, the smaller
Nb+5 and larger Nb+2 contributions. Fluorine effect was hard to establish.

##  Kinetics of Oxide Reduction

The process of niobium oxide reduction via oxygen dissolution into niobium was
studied by the analysis of the ratio of peak areas corresponding to oxygen
which is bound to niobium (fitted within O 1$s$) and the total area of niobium
(fitted within Nb 3$d$) as a function of annealing time (Figure (a)).

[width=]figures/O_to_Nb_v3.png

Figure : Kinetics of the oxide-layer reduction process: (a) the time
dependence of the normalized ratio of O 1s and Nb 3d peak area at various
temperatures. (b, c) Fits to the kinetic models for the first- and second-
order reactions, respectively. Arrhenius plots created using the calculated
rate constants are shown in the insets. For the analysis, only the XPS data
collected at the temperature of baking were accounted (the data gathered
during the heating-up stage were not used). Since the oxide layer was
completely decomposed during the heating-up before reaching 400, this
temperature was not considered. In the studied temperature range of 200300,
the data fit both the kinetics of the first- and second-order reactions. The
corresponding data plotted in coordinates $ln\frac{C}{C_{O}}$ and
$\frac{C_{O}-C}{C}$ against time $t$ are shown in Figures (b) and (c),
respectively. Probably, since the oxide decomposition started in some cases
earlier than the target temperature was reached, the deviation from these
models is observed in the beginning. The dependencies $lnk(T^{-1})$ for the
baking experiments at temperatures 200300 are shown in the insets of Figures
(b, c). It follows that the rate constants for the first- and second-order
reactions are determined as $k_{1}=33.96exp({\frac{-63.62\pm 2.49}{RT}})$ and
$k_{2}=0.85exp({\frac{-68.07\pm 4.97}{RT}})$. These relations represent the
Arrhenius equations with the activation energy, $E_{a}$, in . The obtained
values of $E_{a}$ (63.6 and 68.1) are much smaller as compared to 174 obtained
in king1990kinetic and 119.9 lechner2021rf, but similar to 58.2 in
veit2019oxygen. For comparison, in Figure  the temperature dependence of the
rate constants for the niobium oxide dissolution reaction of the first,
$k_{1}$, and second, $k_{2}$, orders obtained in the present work are plotted
together with the published values.

##  The Oxygen Concentration Depth Profile

Knowing the rate constants for oxide reduction it is possible to calculate the
oxygen-concentration depth profiles in niobium.

###  200-300°C

In the cases of annealing at temperatures 200300 and the durations applied,
the following solution of the second Fick’s law can be applied (“the
continuous plane source”) jaeger1959conduction (p.262, eq.8):
$C=\frac{1}{\sqrt{\pi
D}}\int_{0}^{t}\frac{e^{-x^{2}}}{4Dt}\frac{\phi(t)dt}{\sqrt{t}},$ (2) where
$D$ is the diffusion coefficient, , $t$ is the duration of baking, $\phi(t)$
is the rate at which the diffusing species are liberated from the surface
plane, i.e. the rate of oxide reduction. As compared to jaeger1959conduction,
“2” is excluded from the denominator as Equation 2 is applied for a semi-
infinite body. At the duration of baking used in our experiments $\phi(t)=v$,
constant, so the Equation 2 becomes: $C=v\sqrt{\frac{t}{\pi
D}}exp({\frac{-x^{2}}{4Dt}})-\frac{vx}{2D}erfc(\frac{x}{2\sqrt{Dt}}),$ (3) The
oxygen diffusion coefficient was taken from pick1982depth and is shown in
Figure  for comparison to other referenced values. The results of calculation
of the depth profiles of oxygen under the surface oxides using both $k_{1}$
and $k_{2}$ are presented in Figure . The curves were calculated using
$v_{1}=k_{1}C_{0}$ and $v_{2}=k_{2}C_{0}^{2}$, where $C_{0}$ is the average
concentration of oxygen in the oxide layer among the studied BCPed Nb samples.
The contribution from the 5-oxygen enriched layer below the oxide layer was
also accounted and is negligible. Oxygen profile caused by the oxide only
(without contribution of the oxygen-enriched layer) is shown in Figure  with a
dashed line for the 300 sample. The calculated amount of liberated oxygen from
the oxide agrees with the XPS data (Table ). With $k_{1}$ a better agreement
was obtained. The obtained curves represent the depth profiles of oxygen under
the remaining surface oxides.

[width=]figures/Diff_coef_v2.png

Figure : Temperature dependence of the rate constants for the niobium oxide
reduction of the first, $k_{1}$, and the second order, $k_{2}$, obtained in
the present work in comparison with the published values lechner2021rf,
king1990kinetic, as well as the oxygen diffusion coefficients, $D_{0}$, from
different references lechner2021rf, pick1982depth, powers1959diffusion,
ang1953activation.

[width=]figures/ConcentrProf_v1.png

Figure : The calculated concentration profiles of oxygen in niobium upon
respective anneal using the obtained $k_{1}$ and $k_{2}$ and $D_{0}$ from
pick1982depth, ang1953activation. Dashed lines: oxygen was liberated from the
oxides (Nb2O5 / NbO2 / Nb2O3 / NbO) only, $k_{1}$(dash-dot: $k_{2}$) was used.
Solid lines: the O-interstitial concentration enrichment (average value of 5.3
at%) under the oxides in BCPed Nb is additionally considered, $k_{1}$(dot-dot:
$k_{2}$) was used.

###  400°C

The oxide-dissolution process was completed during the heating-up before 400
was reached. No gain in the O/Nb ratio was observed during 3 baking time (O/Nb
= 0.25). A slight increase in the Nb-C/Nb ratio was observed both during the
heating-up stage and constant temperature baking (Figure (a)).

[width=]figures/O_profiles_v2.png

Figure : (a) The Nb-C/Nb ratio estimated by fitting the peak area within C 1s
and Nb 3d ($h\nu$=900) during the thermal anneal at 400. (b) The hypothetical
oxygen concentration profile at 400 if the partial oxygen pressure was
6e-48e-4mbar. At 400 an oxygen-solute atom may travel within the Nb lattice at
a rate of 40, which is about 6 times faster than at 300. One could assume that
the surface-concentration level of oxygen is kept constant since its diffusion
into the bulk is compensated by absorbing atoms from the ambient environment.
However, considering the UHV condition this option can hardly be the case. It
is possible to roughly estimate the maximal gain in solute concentration of
gas atoms, $\Delta C$, which depends on the number of gas molecules striking
the surface per unit time. According to fromm1980hydrogen: $\Delta
C\approx\frac{1}{2}\frac{pt}{d},$ (4) where $\Delta C$ is the change of the
concentration of dissolved species in , $p$ is the residual pressure of
reactive gases in ¯, $t$ is the heat treatment duration in , and $d$ is the
thickness of the sample in . Thus, at a pressure of 3.5e-9¯ of oxygen-
containing species in a vacuum chamber at 400, the maximal $\Delta C$ after 3
is estimated to be 6.75e-5. The real $\Delta C$ is usually smaller since the
formulae does not account for surface reactions. The obtained value of $\Delta
C$ is an order of magnitude smaller than the bulk oxygen concentration of
niobium with $RRR$=300 which is below 6.78e-4 (i.e. below 10 ppm by weight
according to material specifications) and thus is negligible in our case.
However, if the base pressure was higher, $\Delta C$ would rise and the
constant surface concentration mode described by
$C=C_{s}erfc(\frac{x}{2\sqrt{Dt}})$ (where $C_{s}$ is surface concentration)
could be realized during baking. This situation would have taken place at a
base oxygen pressure of 6e-48e-4mbar considering the estimated surface
concentration of $C_{0}$ = 15.49 (11.86) at% (if the oxygen sustained both in
oxides and interstitials (only interstitials) is counted). This hypothetical
case is demonstrated in Figure (b). Since $\Delta C$ is quite low, the oxide
dissolution rate and the oxygen-diffusion coefficient are high, so that one
can disregard the kinetics of oxide dissolution and consider solving the
diffusion equation for the infinitely thin layer: $C=M\frac{1}{\sqrt{\pi
Dt}}exp(\frac{-x^{2}}{4Dt}),$ (5) where $M$ is the number of diffusing species
per unit area. The calculated surface concentration of $\approx$0.02 is much
lower than the measured by XPS (Figure ). This is explained by the presence of
a thin oxygen-segregation layer with the oxide precipitates (in our case,
trace amount of NbO and the interstices), typical for exothermal metal-gas
system such as Nb-O at the temperatures under investigation horz1983role,
which is being probed by XPS.