# Incentive Schemes for Rollup Validators

Akaki Mamageishvili and Edward W. Felten
Offchain Labs, Inc

###### Abstract

We design and analyze attention games that incentivize validators to check
computation results. We show that no pure strategy Nash equilibrium of the
game without outside parties exists by a simple argument. We then proceed to
calculate the security of the system in the mixed Nash equilibrium, as a
function of the number of validators and their stake sizes. Our results
provide lower and upper bounds on the optimal number of validators. More
concretely, a minimal feasible number of validators minimizes the probability
of failure. The framework also allows to calculate optimum stake sizes,
depending on the target function. In the end, we discuss optimal design of
rewards by the protocol for validators.

##  1 Introduction

Decentralized systems, such as blockchains, rely on validators to compute and
verify the state of the system. Validators are rational players that provide a
service when the right incentives are in place: they are rewarded for correct
behavior and punished for misbehavior. Blockchains have successfully
implemented these incentives. The validators stake some amount of (blockchain
native) tokens and earn rewards if they provide validation service. If they
misbehave, there are two ways they get punished. First, directly, by slashing
stakes. Second, indirectly: if the system loses security, it causes the
validator stake to be devalued or even stolen.

There are still many research directions to explore. We study a particular
setting in this paper, specifically, a setting where a single honest and
active validator is necessary for the security of a system, as for example
described in [5]. We ask a question exactly how many validators are required
for a high enough security level of the system, and what their stake sizes
should be. Our model and insights are general enough to be applied to a broad
class of decentralized systems, but they are best suited to optimistic rollup
protocols. To scale the base layer protocol, computationally heavy execution
is delegated from the base layer to layer two protocols, called rollup
protocols. In case of optimistic rollups, only claims on intermediate states
are made on the base layer. This is where active validation becomes important.
If one party, typically one of the validators, wrongly claims a state of the
system, other validators are supposed to challenge and prove that the claim is
wrong. A single honest validator is enough to disprove a wrong claim, and
therefore, guarantee the security of the protocol.

In the first part of the paper, we study a game where validators obtain
rewards only when they find false claims made by other (malicious) validators.
In the context of optimistic rollups, only one honest validator which checks
computation results is enough to detect and prevent malicious attackers’
efforts. The malicious asserter can make a false claim that allows all assets
at the rollup to be stolen if all validators fail to challenge it. This is why
long periods for a challenge are given. Successful attacks are subtler on the
base blockchains, as they also devalue stolen assets if they are in the native
tokens, however, in the rollups the deployed assets are not native and
therefore, not devalued on the base layer.

We first consider a game in which the behavior of players is completely based
on economic incentives. In particular, there are no social norms and the
payoffs of the players are not affected by outside parties, e.g., the system
designer does not inject money into the system. The reward a validator might
receive is paid by another validator that made a false claim about the system
state. This way the game is similar to the so-called multi-player zero-sum
game. The difference with the zero-sum game, however, is that if the malicious
validator succeeds with a false claim, it may obtain a much higher payoff than
what the rest of the validators lose.

In this model, we argue that a pure strategy equilibrium does not exist, and
therefore, we focus on mixed strategy equilibrium solution concept. In this
solution, we calculate the probability of the system failure, which should
serve as an upper bound on the system security expected if there are only
rational players. In reality, we expect the system to be more secure because
there are altruistic players and some players may also care about their
reputation.

In the second part, we allow rewards to validators without detecting false
claims. To achieve this, validators are required to post their checking
results on-chain with some probability. To be able to figure out if posting is
needed, validators need to compute the state of the system fully. If the
probability of posting is high enough, validators are incentivized to do
computation all the time, hence, pure strategy equilibrium is obtained. They
are rewarded when they correctly post about the state of the world, the amount
which compensates for their past checking efforts as well.

### Related Literature

The question of correctly incentivizing validators in blockchain settings is
not recent, see [7], where the problem is referred to as the verifier’s
dilemma. In parallel research, [6] studies the security of optimistic rollups
in a very similar model as ours. The differences are the absence of deposits
of the validators, equivalent to our definition of a silent validator and
focusing on 2 players, a representative attacker, an Asserter in our case, and
a representative validator. A closely related paper to ours is [8]. The
difference between our approaches is that in [8] validators (data providers)
are not explicitly rewarded for their service. Instead, the authors look at
only punishing strategies to incentivize the right behavior. Also, they
consider pure strategy Nash equilibria solutions. [3] studies a similar
question of how to design a bug-searching committee. The paper studies the
question of the optimal number of searchers and reward policies, given a fixed
budget. [2] studies blockchain security through the lenses of game theory, in
particular, how to design a sharing of validation rewards. However, in this
model, the costs of validation differ across validators. Another related paper
is [4], where costs of validation differ, and the authors look at the problem
of delegating validation, a common way of earning through rewards. Game theory
and security literature have a big intersection. For research relating these
two in the context of multi-party computation settings as well, see [1].

##  2 Model

We first consider the simplest case of two validators, which we sometimes
refer to as players. One of them is playing the role of the Asserter and the
other playing the role of the Checker. This setting corresponds to the system
with two validators. Most of the parameters are defined in the game with two
players, denoted by $\mathcal{G}_{2}$ and later used for a more general
setting with $n$ players. The roles of players are specified in the following:

  * •

The Asserter makes a claim about the state of a system that is either true or
false. The state of the system is a result of a computation of a state
transition function, that is fed a stream of incoming transactions.

  * •

The Checker can check the claim made by the Asserter, at some computational
cost. The Checker has the option to challenge the claim.

The Asserter’s action set consists of false and true claims. The Checker’s
action set consists of checking and not checking.

The following is a list of the parameters of the game that define the final
utilities of the players.

  * •

$C$: the cost of checking. Here we refer to the cost of checking each claim
about the state of the system. Claims arrive at a regular rate. We ignore the
sunk costs of setting up the validator node. After such a node is set up, the
validator that runs a node pays only server and maintenance costs. Node
software that does a check of the claim is typically available for validators
of the rollup systems. Therefore, it is safe to assume that they incur
homogenous costs of checking.

  * •

$R$: the deposit of the Asserter. In case of false claim discovery, $R$ is
also the Checker’s reward. In practice, only a fraction of the reward is given
to a Checker and the rest is burnt. This is done in order to avoid situations
where both validators are controlled by the same malicious player. In such a
case this malicious player does not lose any deposit by rising a false
assertion and losing the challenge to itself 111Such delay is an attack on its
own.. However, in the analysis, we assume that validators are not controlled
by the same player and the mechanism is budget balanced: what one player loses
goes to another player. The analysis does not change and results are the same
asymptotically (and qualitatively) if we assume burning some fraction of the
Asserter’s deposit.

  * •

$L$: the Checker’s loss if cheating goes undetected. Note that $L$ does not
have to be equal to $R$. It can be a stake validators have locked for being a
validator. However, by the design of a challenge mechanism, $L$ is at most
$R$.

  * •

$U$: is the Asserter’s gain if cheating goes undetected. It can be thought of
as the full assets locked in the protocol, sometimes referred to as total
value locked (TVL). This interpretation of the value is most suited to rollup
protocols.

strategy |  false ($\pi$) |  true ($1-\pi$)
---|---|---
check ($\alpha$) |  $R-C,-R$ |  $-C,0$
don’t check ($1-\alpha$) |  $-L$, $U$ |  $0$,$0$

The table above gives a game $\mathcal{G}_{2}$ in a bimatrix format. The first
number is the utility (payoff) of the Checker, the second number is the
utility of the Asserter. The Checker is sometimes referred to as a row player
and the Asserter – a column player.

We make a mild assumption on the parameters:

######  Assumption 1.

Assume that the Checker does not have a dominant strategy, that is, $R-C>-L$,
equivalent to $C<R+L$.

The condition is intuitive: if the cost of checking is too high, the Checker
never checks. A common goal is to minimize the probability of a system
failure, denoted by $F(C,L,R,U)$. That is, the probability that a false claim
is introduced by the Asserter and the Checker does not check it. Note that the
Asserter does not have a dominant strategy, given the assumption. Since the
Checker does not have a dominant strategy, we can only have a totally mixed
equilibrium game. A similar observation is made in [6]. However, in the
totally mixed equilibrium solution of [6], in our case, because of the
assumptions, the probability of a successful attack is very low, but the
damage is huge, while in the case of [6], both probability and damage are
moderate. We assume that if an attack is detected, then the Asserter gets
nothing, while in [6] gives the same $\pi$ payoff in both cases. In the case
of detection, the payoff is negative because of a larger deposit slashed,
while in our case it is just negated deposit, $-R$. Second, we assume that in
case of a successful attack, the attacker can in principle steal all assets,
while [6] assumes that the $\pi$ is a moderate value.

We compute probabilities in the mixed equilibrium solution in the following.
Consider mixed strategies for both players. For the Asserter, strategies can
be characterized by the probability $\pi$ that the chosen action (claim) is
false. With probability $1-\pi$, the claim is true. For the Checker, the mixed
strategy is characterized by the probability $\alpha$ that the Checker checks.
With probability $1-\alpha$, the Checker does not check.

######  Proposition 1.

The probability of failure is increasing in the cost of checking $C$ and
decreases in $U$.

###### Proof.

We can now calculate equilibrium probabilities $\alpha$ and $\pi$, using
indifference conditions. The indifference condition for the Checker is that
the expected utility of playing ”check” is equal to the expected utility of
playing ”don’t check”. That is,

$\pi(R-C)+(1-\pi)(-C)=\pi(-L),$ (1)

equivalent to

$\pi=\frac{C}{R+L}.$ (2)

The indifference condition for the Asserter is that the expected utility of
playing ”false” is equal to the expected utility of playing ”true”. That is,

$0\alpha+0(1-\alpha)=(-R)\alpha+U(1-\alpha),$ (3)

equivalent to

$\alpha=\frac{U}{R+U}.$ (4)

By plugging in the equilibrium values of $\pi$ and $\alpha$, we obtain:

$F(C,L,R,U):=\pi(1-\alpha)=\frac{CR}{(R+L)(R+U)}.$ (5)

It is easy to see that $F(C,L,R,U)$ is decreasing in increasing $U$.

∎

If the rollup has a higher value, it is less likely to fail in the
equilibrium. The explanation is simple: a higher value of the rollup protocol
makes it more attractive for a malicious Asserter to try to make a false claim
to transfer all value to itself, but this on the other hand gives more
motivation to the Checker to check, as it is earning on finding a false claim.
In light of this, another value of interest is the expected loss of the system
$F(C,L,R,U)U$. This value is increasing in $U$, and converges to
$\frac{CR}{R+L}$ as $U$ tends to infinity.

Note that, by (4), $\alpha$ is decreasing in increasing $R$. This sounds
counterintuitive – a larger reward for the checker discovering a false claim
makes it less likely that the checker will check for a false claim. Even
though higher $R$ should increase the incentive of the Checker to check, and
all else being equal it does, it also decreases the probability $\pi$ that the
Asserter introduces a false claim, which on its own decreases the incentive
for the Checker to check. That is, the recommendation is that increasing $R$
is not the solution to maximize checking probability. Consider a derivative of
$F(C,L,R,U)$ as a function of $R$.

$\frac{dF}{dR}=\frac{C(R+L)(R+U)-CR(R+L+R+U)}{(R+L)^{2}(R+U)^{2}}=\frac{CUL-
CR^{2}}{(R+U)^{2}(R+L)^{2}}.$ (6)

Solving $\frac{dF}{dR}=0$ gives $R=\sqrt{UL}$. Therefore, when
$R<R^{*}:=\sqrt{UL}$, LHS of (6) is positive and when $R>R^{*}$, then it is
negative. That is, the probability is decreasing in $R$ above $R^{*}$, and
should be taken as high as possible.

Note a few observations on $F$, from the formula (5). First, $F$ is minimized
at $R=0$. It implies that a false claim comes for free, therefore, the
Asserter tries a false claim all the time, and the Checker checks all the
time, as the cost of checking is less than the punishment for not checking:
$C<L$. This would be a desirable solution, but each false claim delays
finality, and therefore, harms the system. Therefore, in optimizing the
parameter sets, we also care about $\pi$, which is maximized by taking $R=0$.
One immediate takeaway from the (2) formula is that if we want to minimize
$\pi$, we need to increase $L$ and $R$.

For decreasing $\pi$, we need to increase $L$, equivalent to disincentivizing
the Checker to stay idle, causing the Asserter to introduce a false claim less
often, and to decrease $R$, equivalent to incentivizing the Asserter to
introduce the false claim more often and, therefore, causing the Checker to
check more often.

Note that $\alpha=1$ for any $R$ is achievable if the Asserter commits to
introduce the false claim with a certain minimum probability:
$\pi>\frac{C}{R+L}$. However, this can not be sustained in equilibrium: the
Checker always checks as it has positive utility, while the Asserter has
strictly negative utility: $-\pi\cdot R$. It is not rational for the Asserter.
It can only be supported as a solution if, for example, the protocol designer
plays the role of the Asserter and posts wrong claims with a probability more
than the bound $\frac{C}{R+L}$.

One potential goal a system designer can have is to optimize social welfare,
in which the costs of validating and a fraction of the stakes are subtracted
from the success probability times TVL. The first cost is obvious - the cost
of (duplicate) checking is lost for the validator. The second, opportunity
cost, is incurred by the validators by staking their assets in the validation
system instead of earning interest outside. The system designer has to
minimize the following target function:

$M:=fU\pi+U\pi(1-\alpha)+\alpha C+r(L+R),$ (7)

where $f$ denotes the relative loss of the system when there is a delay and
$r$ is a potential return on investment outside the system. $f$ is typically
assumed to be a low number, say $10^{-2}$. Plugging in the values for $\pi$
and $\alpha$ gives an equivalent equation to (7):

$M=f\frac{C}{R+L}+\frac{C}{R+L}\frac{R}{R+U}=\frac{C}{R+L}\left(f+\frac{R}{R+U}\right)+\frac{U}{U+R}C+r(L+R).$

The optimum values of $R$ and $L$ minimizing $M$ can be computed by taking
partial derivatives of $M$ with respect to $R$ and $L$.

###  2.1 Extension to $n+1$ validators

In this section, we assume that there are $n+1$ validators. One of the
validators is an Asserter, in each round. We want to incentivize the
validators to check claims often enough. In each round, the Asserter makes a
claim, and validators can check it. If they check and find the false claim,
they do not get slashed if they post a challenge to the false claim. If they
check and find out that the claim is true, they do not need to post anything.
That is, not posting anything can mean two things: the validator checked and
found out the claim is true, or the validator did not check and that is why
there is no post. There are two ways to implement the payoff to the players in
the protocol:

  1. 1.

Simultaneous: every validator posts, if they want to post, at the end of a
predefined time interval. This approach is simpler to analyze, and for the
players, it is simpler to make a decision.

  2. 2.

Sequential: The validators see what other validators have done so far. If
nobody posts anything this may motivate them not to post anything, but that
increases the chances that someone will post a fraud-proof at the last second
and slash the silent validators.

For simplicity, we focus on the simultaneous model in this paper. That is,
validators see only at the end of the round how many posted checks. Having
homogenous costs of checking among validators is a natural assumption in the
setting of rollups, as there is available software for running a validator
node and standard hardware requirements.

If no validator detects a false claim, the Asserter proceeds with the false
claim, and all validators are punished by losing all their deposited stake –
$L$ – and the Asserter can steal all value on the chain, giving it payoff $U$.

We consider a fully mixed symmetric equilibrium of this game. Similarly to the
case with two players, the probability that each validator checks is denoted
by $\alpha$, and the probability the Asserter claims a false claim is denoted
by $\pi$. The timeline of the events is the following:

  * •

If $m$ out of $n$ validators find a false claim and post about it, they are
paid equally: $\frac{R}{m}$.

  * •

the other $n-m$ validators are slashed $s_{w}$, which we assume to be (much)
smaller than $L$.

The probability that at least one out of $n$ validators will check is equal to

$P_{s,\alpha}(n):=1-(1-\alpha)^{n}.$

Note that in this definition $\alpha$ is an independent parameter, however, in
the equilibrium it depends on $n$. Similarly to (1), we derive the
indifference condition for the validator. Shortly, it is EU[check] = EU[don’t
check], where EU[X] stands for expected utility from taking an action $X$. The
condition can be translated as:

$\displaystyle\pi\left(\sum_{i=0}^{n-1}{n-1\choose
i}\alpha^{i}(1-\alpha)^{n-1-i}\frac{R}{i+1}\right)+(1-\pi)0-C=$ (8)
$\displaystyle\pi(P_{s,\alpha}(n-1)(-s_{w})+(1-P_{s,\alpha}(n-1))(-L))+(1-\pi)0.$
(9)

The first summand on the left-hand side (LHS) represents the product of the
probability that the claim is false, the probability that $i$ other validators
check and (expected) rewards $R/(i+1)$, as there are $i+1$ validators finding
the false claim.

The second summand on the LHS is a product of the probability that the claim
is true with $0$, while the last represents the minus cost of checking.

The first summand of the right-hand side (RHS) represents the product of the
probability that the claim is false, the product of the probability that
someone else checks with $-s_{w}$.

The second summand of RHS is a product of the probability that nobody checks
with $-L$, while the third summand is a product of the probability that the
claim is true with $0$.

The indifference condition can be further simplified to:

$\displaystyle\pi\left(\sum_{i=0}^{n-1}{n-1\choose
i}\alpha^{i}(1-\alpha)^{n-1-i}\frac{R}{i+1}\right)-C=$
$\displaystyle\pi(P_{s,\alpha}(n-1)(-s_{w})+(1-P_{s,\alpha}(n-1))(-L)).$ (10)

First, the following claim, obtained in [3], holds.

######  Lemma 1.

For $n\in\mathbb{N}$ and $x\neq 0$,

$\sum_{k=0}^{n}{n\choose
k}\frac{x^{k}y^{n-k}}{k+1}=\frac{1}{n+1}\frac{(x+y)^{n+1}-y^{n+1}}{x}.$ (11)

The lemma implies simplification:

$\sum_{i=0}^{n-1}{n-1\choose
i}\alpha^{i}(1-\alpha)^{n-1-i}\frac{R}{i+1}=R\frac{1-(1-\alpha)^{n}}{n\alpha}.$
(12)

Similarly to (3), the indifference condition of the Asserter is EU[false
claim] = EU[true claim]. The LHS is the sum of the product of the probability
that some validator checks with $-R$ and the product of the probability that
no validator checks with $U$. The RHS is equal to $0$. The condition
simplifies to:

$(1-(1-\alpha)^{n})R=(1-\alpha)^{n}U.$ (13)

From the condition, we obtain a solution for $\alpha$:

$\alpha=1-\frac{R}{R+U}^{1/n}.$

RHS of (12) further simplifies to:

$R\frac{1-(1-\alpha)^{n}}{n\alpha}=\frac{UR}{(R+U)n\alpha}.$

Plugging in $\alpha$ in the equation (2.1) and taking into account (12)
derives $\pi$:

$\pi=\frac{C}{\frac{UR}{(U+R)n\alpha}+P_{s,\alpha}(n-1)s_{w}+(1-P_{s,\alpha}(n-1))L}.$
(14)

The main value of interest as in the case with 2 players is $P_{s,\alpha}(n)$.
It is obtained by solving the Asserter’s indifference condition (13) and is
equal to $\frac{U}{R+U}$. That is, it does not depend on the number of
validators $n$. The second value of interest is $\pi$, as
$\pi(1-P_{s,\alpha}(n))$ is the probability that the false claim will go
through unnoticed. Note that $\pi$ is decreasing in increasing $s_{w}$ and
$L$. That is, for decreasing the probability that the Asserter is introducing
the false claim, and therefore, decreasing the total probability the false
claim goes through unnoticed, we need to increase the slashed stakes of the
validators. All else being equal, we obtain the following result:

######  Proposition 2.

$\pi$ is increasing with increasing $n$.

This result helps to find out the optimum number of validators, in particular,
it suggests to rollup systems that $n$ should be as low as possible. On the
other hand, $n>1$ might still be needed because, for example, some validators
are not online for some time.

Suppose $t$ validators go offline for a technical reason. We calculate the
probability that the system still functions in equilibrium. It is equal to
$P_{s,\alpha}(n-t)$. In the following, we give a numerical example.

######  Example 1.

In this example, we consider realistic values of parameters. Suppose $C$ is
normalized to $1$ (dollar), which is a reasonable approximation of one round
computation costs, $U=10^{9}$ corresponding to the TVL, $R=10^{6}$
corresponding to a stake that an asserting validator needs to commit and
$L=10^{5}$ corresponding to a stake an active validator needs to commit. The
probability of failure is minimized when $n=1$ and it is equal to
$\frac{10^{6}}{(10^{6}+10^{5})(10^{9}+10^{6})}\approx 10^{-9}$. Now assume
that $n=12$. Then the probability each validator checks, $\alpha$, is
approximately equal to $0.4377$, and the probability of a false claim is
approximately equal to $3.448\cdot 10^{-6}$ and the probability of failure
equals to $3.445\cdot 10^{-6}$. When $t=2$, the probability of failure equals
to $0.343\cdot 10^{-6}$. The driving force of these (good) results is that
$\pi$ is very low. The other multiplier, the probability that one of the
validators will check, has a lower effect on the result. The following table
shows approximate values of $\alpha$ and $\pi$ for $n\leq 12$. Note that the
probability all validators will fail to catch a false claim, in that case, is
$(1-\alpha)^{n}=\frac{R}{U+R}$, independent of $n$, and approximately equals
to $10^{-3}$.

$n$ | $1$ | $2$ | $3$ | $4$ | $5$ | $6$ | $7$ | $8$ | $9$ | $10$ | $11$ | $12$
---|---|---|---|---|---|---|---|---|---|---|---|---
$\alpha$ | $0.999$ | $0.968$ | $0.900$ | $0.822$ | $0.748$ | $0.683$ | $0.627$ | $0.578$ | $0.535$ | $0.498$ | $0.466$ | $0.437$
$\pi$ | 9.1e-7 | 1.9e-6 | 2.7e-6 | 3.3e-6 | 3.7e-6 | 4.1e-6 | 4.4e-6 | 4.6e-6 | 4.8e-6 | 5e-6 | 5.1e-6 | 5.3e-6
Table 1: Equilibrium probabilities as a function of $n$.

###  2.2 Silent validators

In this section, we assume the existence of ”silent” validators. They do not
stake anything, unlike (active) validators considered so far, but can access
the base layer contract after each claim and challenge the (false) claim of
the Asserter, in case active validators did not do so 222The role of a silent
validator can be played by the Asserter as well. That is, instead of stealing
all the assets in the system, it may only collect $nL$ and allow the system to
survive.. A successful claim by a silent validator allows it to collect all
stakes – $nL$ and the Asserter’s deposit $R$. This gives more incentive to the
staked validators to check.

To get an intuition, we start with the smallest instance. Assume that there is
one active and one silent validator. The indifference condition of the active
validator stays the same as in the case without silent validators, as the
active validator loses its deposit if the Asserter’s claim is false. The
indifference condition of the silent validator, on the other hand, is:

$\pi(1-\alpha)R=C.$ (15)

The expected gains for the silent validator is $(1-\alpha)R$, as the active
validator finds the false claim with probability $\alpha$.

Plugging in $\pi=\frac{C}{R+L}$ in (15) gives a contradiction, the LHS is
always lower than the RHS. This implies that the silent validator never checks
in the equilibrium. The same holds even if we add the active validator’s
deposit to the reward of a silent validator. The indifference condition in
this case becomes:

$\pi(1-\alpha)(R+L)=C.$ (16)

However, the mechanics change when we consider more than $1$ active validator.
Suppose there are $m$ silent validators. The indifference condition of such a
validator is different from the active validator, as it does not risk losing
stake if it does not check. On the other hand, if the silent validator checks
while no staked validator does, it will be rewarded both by staked validator
stakes – $nL$ – and the dishonest Asserter’s stake $R$. Silent validators have
the same cost of checking, $C$. By a similar argument as with only active
validators, it is easy to show that there is no pure Nash equilibrium solution
to the game. The proof is by contradiction: if one of the validator types
checked with certainty, it would make malicious Asserter not make a false
claim, causing validators not to check. Therefore, we again consider a totally
mixed Nash equilibrium solution. The probability that the silent validator
plays the checking strategy is $\beta$. Then, the indifference condition for
the silent validator is:

$C=\pi(1-(1-\alpha)^{n})\sum_{i=0}^{m-1}\left({m-1\choose
i}\beta^{i}(1-\beta^{m-1-i})\frac{R+nL}{i+1}\right).$ (17)

The indifference condition for active validator is the same as (2.1), as for
this type of validator it does not matter what silent validators will do. If
there is a false claim and active validators do not find it, they will lose
all their stakes. For completeness, we state the condition here:

$\pi\left(\sum_{i=0}^{n-1}{n-1\choose
i}\alpha^{i}(1-\alpha)^{n-1-i}\frac{R}{i+1}\right)-C=\pi(P_{s,\alpha}(n-1)(-s_{w})+(1-P_{s,\alpha}(n-1))(-L)).$
(18)

The indifference condition for the malicious Asserter is:

$(1-(1-\alpha)^{n})(1-(1-\beta)^{m})R=((1-\alpha)^{n}+(1-\beta)^{m}-(1-\alpha)^{n}(1-\beta)^{m})U.$
(19)

Analyzing indifference conditions (17), (18) and (19) gives conditions on the
parameters when totally mixed equilibrium of the game exists. Consider, for
example, $m=2$ and $n=1$. The indifference conditions become:

$C=\pi\alpha(R+2L),C=\pi(R+L),(2\alpha-\alpha^{2})\beta
R=(1-(2\alpha-\alpha^{2})\beta)U.$

This solves $\alpha=\frac{R+L}{R+2L}$,
$\beta=\frac{U}{(2\alpha-\alpha^{2})(R+U)}$ and $\pi=\frac{C}{R+L}$. That is,
$\alpha$ needs to be high enough, to make sure that $\beta$ is smaller than
$1$. When $\alpha$ is low enough, then it must be that $\beta=1$. That is,
silent validators always check.

##  3 Protocol level incentives

In this section, we ask the question of how to reward validators for checking
(and posting about) the true claim 333For a similar discussion for Ethereum
validator incentivization see
https://dankradfeist.de/ethereum/2021/09/30/proofs-of-custody.html.. The post,
a transaction to a smart contract at the base layer network does not need to
include proof. This approach allows obtaining a pure strategy equilibrium, in
which all validators check with certainty. This guarantees that a false claim
is found with probability one. However, it comes at the cost of adding new
functionality to the protocol, which is usually not desirable.

Similar to the previous section, there are $n+1$ validators. The probability
that each one needs to post about checking the state of the world is denoted
by $P$. Validators need to take a decision whether to check or not before they
find out whether they need to post about the result. In case they fail to post
when they are required to post, they have slashed their stakes $L$. The cost
of checking is $C$, as before. The cost of posting is $c$, which is typically
assumed to be less than $C$. The opportunity cost of staking on the platform
is $rL$ in each round. Therefore, $r$ denotes the return on investment outside
the system in one round. The payment validators receive for posting the right
outcome is denoted by $p$. Then, the expected payoff is equal to $-C+P(-c+p)$
when the validator checks, or $P(-L)$ when the validator does not check. To
guarantee that the validator checks in equilibrium, the expected payoff of
checking needs to be larger than the expected payoff of not checking:

$-C+P(-c+p)>P(-L).$

This gives a condition on $P$, namely $P>\frac{C}{p-c+L}=:\pi_{l}$.

The expected budget of the protocol per round is equal to $nPp$, which is
lower bounded by $n\pi_{l}p=\frac{nCp}{p-c+L}$. Note that taking a high enough
$L$ lowers the expected cost of the system to guarantee incentive
compatibility (IC), but it increases the cost to guarantee individual
rationality (IR). The latter means that validators want to be a part of the
system in the first place, instead of staying away and obtaining zero utility.
To guarantee IR, we need to offset the opportunity cost the validator incurs
by staking $L$, which equals $rL$. Since by IC, the validator always checks,
we need that $-C+P(-c+p)>rL$, that is, $P>\frac{C+rL}{p-c}=:\pi_{r}$. This
simplifies to the condition that $\pi$ needs to be larger than
$\max(\pi_{c},\pi_{r})$. The minimum value of $\pi$ is achieved at the
minimax. One function is decreasing in $L$, another is increasing. The minimax
is achieved when they are equal. That is,

$\frac{C+rL}{p-c}=\frac{C}{p-c+L},$

implying $L=\frac{rc-rp-C}{r}$ (which is negative, therefore, not possible) or
$L=0$. This optimization is done for fixed $p$. We can minimize $P$ and $Pp$
over $p$ as well. Plugging in $L=0$ into the formula of $P$ gives:
$P=\frac{C}{p-c}$. It is minimized for $p$ as large as possible. Similarly,
$pP=\frac{Cp}{p-c}$ is minimized for $p$ as large as possible. The value
approaches $C$, which is intuitive:

  1. 1.

the system pays exactly the cost of checking on average,

  2. 2.

it checks the validators with very low probability,

  3. 3.

when they are checked - the system pays a very large amount $p$.

Unless there is some cost associated with high payment $p$ for the protocol to
upper bound it, this is an optimal strategy. However, such costs are obvious.
The protocol cannot invest an arbitrarily high amount at once in rewarding
validators.

### Implementation

We present the implementation of attaching a message of checking and posting
with some probability to an assertion. The sampling can be done on the
protocol level, by referring to state-relevant hash values. Suppose the
Asserter is making a claim about the value of $f(x)$ for some function $f$
which is common knowledge, and a value $x$ which varies across different runs
of the protocol. We want to pose a randomly generated challenge to the
validator, such that the checker must know $f(x)$ in order to respond
correctly to the challenge. Then we can punish the validator for responding
incorrectly.

The validator has a private key $k$, with a corresponding public key $g^{k}$
which is common knowledge ($g$ is a suitable generator of a group where the
Diffie-Hellman problem is hard). To issue a challenge for the computation of
$f(x)$, the asserter generates a random value $r$, then publishes $(x,g^{r})$
as a challenge. A validator who has private key $k$ should respond to the
challenge by posting a tiny transaction on-chain if and only if
$H(g^{rk},f(x))<T$, where $H$ is a suitable hash function and $T$ is a
suitably chosen threshold value.

Note that only the Asserter (who knows $r$) and the validator (who knows $k$)
will be able to compute the hash because they are the only two parties who can
compute $g^{rk}$. Note also that computing the hash requires knowledge of
$f(x)$. After the validator(s) have had a window of time to post their
response(s) to the challenge, the Asserter can post its claimed $f(x)$ which
will be subject to challenge if any validator disagrees with it.

At this time, the Asserter can accuse any validator who responded incorrectly:
the Asserter must publish $r$ to substantiate its accusation. If the
Asserter’s claimed value of $f(x)$ is later confirmed, a smart contract can
verify the accusation and punish the misbehaving validator (if the Asserter’s
claimed $f(x)$ is rejected, the Asserter’s accusation is ignored).

If any funds are seized from validators, the Asserter gets half of the seized
funds and the remainder is burned. One way to build this in the rollup is to
have assertions, rather than revealing the state root $f(x)$. Instead, include
an attention challenge $(x,g^{r},H(x,g^{r},f(x)))$, which is also a binding
commitment to $f(x)$, and only reveal $f(x)$ when there is a challenge, or
when the assertion is confirmed. Validators could self-identify and stake, and
they would have until the confirmation time of the assertion to post their
response to the attention challenge.

##  4 Conclusions and future work

We initiate a study of the optimal number of validators and their stake sizes
in the rollup protocols. The main result is that for higher system security
guarantees, the cost of checking should be low, TVL should be high and the
number of validators should be as low as possible. We also derive optimal
validation and assertion deposits in the equilibrium. Future avenues of
research include weighted staking. Even if such staking is not allowed, if one
validator creates multiple identities, but checks only once, it results in
weighted staking. Such validator’s indifference condition is different from
the others, as it has invested $kL$ tokens, where $k$ is the number of copies
it created.

## References

  * [1] Gilad Asharov, Ran Canetti, and Carmit Hazay.  Towards a game theoretic view of secure computation.  In Kenneth G. Paterson, editor, Advances in Cryptology - EUROCRYPT 2011 - 30th Annual International Conference on the Theory and Applications of Cryptographic Techniques, Tallinn, Estonia, May 15-19, 2011. Proceedings, volume 6632 of Lecture Notes in Computer Science, pages 426–445. Springer, 2011.
  * [2] Lars Brünjes, Aggelos Kiayias, Elias Koutsoupias, and Aikaterini-Panagiota Stouka.  Reward sharing schemes for stake pools.  In IEEE European Symposium on Security and Privacy, EuroS&P 2020, Genoa, Italy, September 7-11, 2020, pages 256–275. IEEE, 2020.
  * [3] Hans Gersbach, Akaki Mamageishvili, and Fikri Pitsuwan.  Decentralized attack search and the design of bug bounty schemes.  CoRR, abs/2304.00077, 2023.
  * [4] Hans Gersbach, Akaki Mamageishvili, and Manvir Schneider.  Staking pools on blockchains.  CoRR, abs/2203.05838, 2022.
  * [5] Harry A. Kalodner, Steven Goldfeder, Xiaoqi Chen, S. Matthew Weinberg, and Edward W. Felten.  Arbitrum: Scalable, private smart contracts.  In William Enck and Adrienne Porter Felt, editors, 27th USENIX Security Symposium, USENIX Security 2018, Baltimore, MD, USA, August 15-17, 2018, pages 1353–1370. USENIX Association, 2018.
  * [6] Jiasun Li.  On the security of optimistic blockchain mechanisms.  Available at SSRN 4499357, 2023.
  * [7] Loi Luu, Jason Teutsch, Raghav Kulkarni, and Prateek Saxena.  Demystifying incentives in the consensus computer.  In Indrajit Ray, Ninghui Li, and Christopher Kruegel, editors, Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, CO, USA, October 12-16, 2015, pages 706–719. ACM, 2015.
  * [8] Ertem Nusret Tas and Dan Boneh.  Cryptoeconomic security for data availability committees.  Forthcoming at Financial Cryptography, 2023.


# Incremental procedural and sensorimotor learning in cognitive humanoid
robots

Leonardo de Lellis Rossi, Letícia Mara Berto, Eric Rohmer, Paula Paro Costa,
Ricardo Ribeiro Gudwin, Esther Luna Colombini and Alexandre da Silva Simões
This work was developed within the scope of PPI-Softex with support from MCTI
through the Technical Cooperation Term [01245.013778/2020-21].A. S. Simões and
L. L. Rossi are with Dept. of Control and Automation Engineering (DECA),
Institute of Science and Technology (ICT), Campus Sorocaba, Universidade
Estadual Paulista (Unesp)R. R. Gudwin, E. Rohmer, P. P. Costa and L. L. Rossi
are with the Dept. of Computer Engineering and Industrial Automation (DCA),
School of Electrical and Computer Engineering (FEEC), University of Campinas,
BrazilE. L. Colombini and L. M. Berto are with the Laboratory of Robotics and
Cognitive Systems (LaRoCS), Institute of Computing, University of Campinas,
BrazilAll authors are with the Artificial Intelligence and Cognitive
Architectures Hub (H.IAAC), University of Campinas, Brazil

###### Abstract

The ability to automatically learn movements and behaviors of increasing
complexity is a long-term goal in autonomous systems. Indeed, this is a very
complex problem that involves understanding how knowledge is acquired and
reused by humans as well as proposing mechanisms that allow artificial agents
to reuse previous knowledge. Inspired by Jean Piaget’s theory’s first three
sensorimotor substages, this work presents a cognitive agent based on CONAIM
(_Conscious Attention-Based Integrated Model_) that can learn procedures
incrementally. Throughout the paper, we show the cognitive functions required
in each substage and how adding new functions helps address tasks previously
unsolved by the agent. Experiments were conducted with a humanoid robot in a
simulated environment modeled with the Cognitive Systems Toolkit (CST)
performing an object tracking task. The system is modeled using a single
procedural learning mechanism based on Reinforcement Learning. The increasing
agent’s cognitive complexity is managed by adding new terms to the reward
function for each learning phase. Results show that this approach is capable
of solving complex tasks incrementally.

###### Index Terms:

Cognitive Robotics, Cognitive Architectures, Reinforcement Learning,
Incremental Learning, Developmental Robotics.

##  I Introduction

Advancements in artificial intelligence and robotics increased the interest in
introducing robots into daily activities that involve interaction with other
agents, both robots and humans. These robots should operate autonomously in
complex, partially unknown, unpredictable, and unstructured scenarios, making
pre-programming impossible and requiring robots to have a superior capability
to perform tasks. This challenge raises questions such as how to incorporate
new knowledge and skills through interactions with the world, resulting in the
research area of Cognitive Robotics. Cognitive Robotics is intrinsically
related to Cognitive Architectures (CA), which represent comprehensive
computer models providing theoretical frameworks to work with cognitive
processes searching for complex behavior.

Cognitive architectures are systems that can reason in different domains,
develop different views, adapt to new situations, and reflect on themselves
[1, 2]. They are general control systems inspired by scientific theories
developed to explain cognition in humans and other animals, comprising modules
responsible for implementing different cognitive abilities, such as
perception, attention, memory, reasoning, and learning.

Inspired by how humans build knowledge through interactions with the world,
cognitive architecture researchers seek to reproduce this behavior with
artificial creatures [3]. However, the development of cognitive skills in
machines requires the coordination of complex mechanisms that depend on each
other. According to Piaget [4], the process of developing these skills is
incremental and evolutionary.

In this work, a cognitive agent based on the CONAIM model (_Conscious
Attention-Based Integrated Model_) [2] was proposed and implemented with the
_Cognitive Systems Toolkit_ [3]. A humanoid robot was designed to
incrementally learn procedures to perform object tracking experiments inspired
by the first three sensorimotor substages of Jean Piaget’s Theory [4].
Throughout the work, we present the cognitive functions necessary to form
circular reactions in each substage using a Reinforcement Learning (RL) [5]
environment and how new functions can be added to the reward function allowing
the agent to solve complex tasks, previously unresolved.

As the main contributions of this work, we can list the following:

  1. 1.

The proposition of a cognitive architecture based on CONAIM with attention,
memories, and learning modules focused on sensorimotor and procedural
learning;

  2. 2.

The design and implementation of CONAIM’s top-down pathway in CST that can be
incorporated into any agent implemented with CST;

  3. 3.

The design and implementation of a single procedural learning mechanism in CST
that can incrementally learn and reuse schemas for the first three
sensorimotor substages of Piaget’s Theory;

  4. 4.

The modeling of a set of environments for sensorimotor experiments for the
movements learning in humanoid robots;

  5. 5.

The implementation and evaluation of sensorimotor experiments for object
tracking in the first three sensorimotor substages of Piaget’s Theory as
proposed by [6].

The code used to implement the architecture is available at:
https://github.com/CST-Group/cst.

##  II Cognitive Architectures

Cognitive architectures are systems that can reason in different domains,
develop different views, adapt to new situations, and reflect on themselves
[7]. They are general control systems inspired by the cognition of humans and
other animals, comprising modules responsible for implementing different
cognitive abilities, such as perception, attention, memory, reasoning, and
learning [3].

A cognitive architecture plays the role of cognition in computational
modeling, making explicit the set of processes and assumptions on which this
cognitive model is based [8]. It consists of processing units that represent,
extract, select and combine knowledge and memories to produce behavior [9, 10,
11].

Next, the main aspects of the reference cognitive architecture used in this
work and the computational toolkit used to program the cognitive agent are
described. Both were proposed in previous work by the working group.

###  II-A CONAIM

The CONAIM model (_Conscious Attention-Based Integrated Model_) [2, 12, 13] is
a formal attention-based model for machine consciousness. CONAIM incorporates
several relevant aspects for a cognitive agent (memories, body schema,
motivation, attention, among others) and is capable of dealing with multiple
sensory systems, multiple processes of feature extraction, decision making,
and learning [13]. The model provides a consciousness-based agent that
performs calculations on attention-directed schemas, significantly reducing
the space of the model’s input dimensions. In its cognitive cycle, top-down
and bottom-up mechanisms are used. During the bottom-up cycle, the sensors
provide external data. The data is stored in the sensory memory to form
feature, attentional and salience maps [13]. The complete modeling is detailed
in [13]. In the top-down cycle, the attentional modulation of the system will
depend on the global state of attention (for example, the robot’s battery
level). It will also depend on the agent’s objective, and the attentional
dynamics of the current state [14].

###  II-B CST

The _Cognitive Systems Toolkit_ [3] is an open-source Java-based [15] toolkit
for building cognitive architectures. The core of CST consists of a set of
basic concepts that can be generalized within any cognitive architecture
built. CST tools allow the creation of multi-agent systems running
asynchronously and in parallel. The CST architecture is codelet oriented.
Codelets are small pieces of code, implemented as asynchronous functions, and
that run in parallel with simple and defined tasks [3]. A memory object (MO)
is a signal or representation used, with other MOs, by codelets to store and
access data [3].

##  III Developmental Robotics

The development of artificial agents with autonomy, adaptive behavior and
incremental learning capabilities are research goals in Cognitive Robotics and
Developmental Robotics (DevRobotics) [16, 17]. The area emerged due to the
need for robots to perform tasks that require comparable levels of human
intelligence in complex and unpredictable environments involving adaptation
and evolution [18]. The models and experiments in the area are inspired by the
principles and mechanisms of development observed in early life, involving
robots performing the same cognitive abilities as children, as in the
experiments proposed by Jean Piaget.

###  III-A Piaget’s Theory

A relevant concept in Piaget’s theory [4] are the schemas, which represent
networks of mental structures that help remember specific concepts and
understand the environment. When simple mental processes become more
sophisticated, new schemas are developed, and behavior becomes more complex
and suited to the environment [19].

In mental development, according to Piaget, adaptation – or learning – is the
tendency to adjust mental processes to the environment by changing cognitive
structures [4]. The adaptation process involves balancing the processes of
assimilation and accommodation. Assimilation and Accommodation are
inseparable, complementary and simultaneous processes [4]. Assimilation is the
creation of new schemas following the same cycle or sequence of existing
schemas for interpreting experiences and making decisions. Accommodation is
the complementary process that involves altering existing schemas as a result
of new information acquired through assimilation. Assimilation can originate
circular reactions, repetitions of cycles acquired or in the process of
acquisition [4, 20]. The circular reaction results from the assimilation of an
interesting result unknown to the subject, which was produced by the
rediscovery or repetition of the action. Circular reactions can be primary,
secondary, or tertiary:

  * •

Primary Circular Reactions are behaviors derived from reflexes, activities of
the body itself that form new schemes through the coordination of the senses;

  * •

Secondary Circular Reactions are derived from intentional behaviors that
direct interest to external outcomes rather than the baby’s body;

  * •

Tertiary Circular Reactions are the subject’s effort to seek new experiences.

###  III-B Sensorimotor Experiments for Incremental Learning

Experiments with cognitive architectures in the field of DevRobotics are
heavily based on theories of childhood development. However, a common point in
these experiments is the lack of standardization to conduct and evaluate agent
development. To help resolve these issues, we proposed in previous work [6] a
set of incremental experiments for the robotic scenario according to Piaget’s
sensorimotor stages of development, along with the expected results. These
experiments are based on Piaget’s studies [4] on the sensorimotor period, on
the Bayley Child Development Scale [21], in different scenarios described in
the literature for the assessment of learning development in infants [22, 23,
24], and in the parameters and levels of ConsScale [25]. The computational
scenarios focused on incorporating human behaviors in robots and evaluating
their cognitive development. Experiments are classified according to the type
of skill to be learned by the agent in many scenarios.

##  IV Proposed Approach

In the current work, we addressed a subset of the theoretical scenarios
proposed by [6], focusing on building an agent to perform exclusively those
experiments related to object tracking task, as shown in Table I. In those
experiments, we expect a robot to incrementally learn the skills of tracking
objects in a scene using RGB-D sensors. The goal is to investigate incremental
computational processes that can allow robots to learn intentional
sensorimotor schemas from an initially unintentional perspective, that is,
computational processes able to model circular reactions using procedural
memory elements.

Task: Tracking objects using a vision sensor
---
Sub-stage |  Experiment |  Expected results |  Details
1 |  Robot placed in front of an object at close distance |  Staring at the object |  Object properties:  • position:  inside the agent’s visual field • movement:  fixed • color: primary Agent’s visual acuity: 20/400 Agent’s FOV: small
1 |  Robot placed in front of an object at close distance |  Track object with the look |  Object properties:  • position:  inside the agent’s visual field • movement:  moving slowly • color: primary Agent’s visual acuity: 20/400 Agent’s FOV: small
1 |  Robot placed in front of an object at close distance |  Do not follow the object with the look |  Object properties:  • position:  leaving the agent’s visual field • movement:  moving slowly • color: primary Agent’s visual acuity: 20/400 Agent’s FOV: small
2 |  Robot placed in front of an object at medium distance |  Track object with the look |  Object properties:  • position:  inside the agent’s visual field • movement:  moving at medium speed • color: primary Agent’s visual acuity: 20/100 Agent’s FOV: increased FOV
2 |  Robot placed in front of an object at medium distance |  Track object with the look |  Object properties:  • position:  outside the agent’s visual field • movement:  fixed position • color: primary Agent’s visual acuity: 20/100 Agent’s FOV: increased FOV
2 |  Robot placed in front of an object at medium distance |  Accompany object while looking within visual field. Does not follow outside it |  Object properties:  • position:  entering or leaving the agent’s visual field • movement:  medium speed • color: primary Agent’s visual acuity: 20/100 Agent’s FOV: increased FOV
3 |  Robot placed in front of an object at high distance |  Track object with the look |  Object properties:  • position:  inside the visual field • movement:  medium speed • color: primary Agent’s visual acuity: 20/20 Agent’s FOV: increased FOV
3 |  Robot placed in front of an object at high distance |  Accompany object with the look (turns the head to continue accompanying the object) |  Object properties:  • position:  leaving or entering the visual field • movement:  medium speed • color: primary Agent’s visual acuity: 20/20 Agent’s FOV: increased FOV

TABLE I: Subset of experiments and expected results for a cognitive robot
sensorimotor learning in a visual object tracking task. Adapted from [6].

The experiments proposed for object tracking describe what capabilities are
expected from the agent in this task at each developmental stage and what it
should not accomplish. These experiments are designed for scenarios with
increasing complexity. It also defines which sensors should be employed to
achieve such capabilities and any sensor limitations. For example, it
considers the child’s visual acuity development when vision is employed.
Objects are positioned at different distances from the robot but are perceived
according to the visual acuity compatible with the presented by a baby at the
specified sensorimotor substage. We can expect a less precise associated
behavior when considering a less accurate sensor, as discussed later in the
experiments. The agent’s visual acuity and its distance to the objects are
variable in the experiments. We assess the development level of the robot by
verifying if it can achieve the expected result, as described in Table I.

##  V Methodology

###  V-A Robots and Environment

The humanoid robot Marta was adopted in our experiments. Marta is a teen-size
1.1m tall female robot designed and built by our workgroup. Marta has 25
degrees of freedom, and it’s head – particularly relevant in the present work
– can perform _pitch_ and _yaw_ movements. The robot was equipped with an
RGB-D camera on its head, inspecting the world in four distinct channels of
color (R, G and B) and distance information (D). The robot was controlled by a
cognitive system detailed in the next sections. Several simulated scenes were
also created in CoppeliaSim for the experiments. In the scenes, Marta is
sitting in a small space with a wide view of its surroundings. An arena was
delimited outside this first space, and colored blocks (blue and green) were
randomly distributed. A second robot, a red Pioneer P3DX, randomly navigates
the arena as a mobile distractor. This robot was modeled with reactive
behaviors using the Braitenberg Algorithm [26]. Both robots and the
environment are shown in Figure 1.

(a) (b) (c) (d)

Figure 1: Simulation environment. From Left to Right. a) Marta robot equipped
with an RGB-D camera; b) Environment with distributed colored blocks and a
Pioneer P3DX robot acting as a distractor. Marta’s view of the scene is shown
at left. (c) Agent’s degrees of freedom. Motors used in red. (d) Division of
the agent’s visual space for the virtual actuator (named as ”eye”, ”look” or
”fovea”).

###  V-B Robot control overview

Table II summarizes the key aspects of the humanoid robot control strategies
adopted in the proposed approach, focusing on Piaget’s schemas created or
utilized along developmental sensorimotor stages. It emphasizes the cognitive
system functions present, the configuration of the robot’s sensors and
actuators, and the Reinforcement Learning reward strategy adopted. We will
address these aspects in the following sections.

Substage |  Circular Piaget’s schemas |  Cognitive system functions |  Vision resolution |  Actions and resolution |  RL Reward strategy
---|---|---|---|---|---
1 |  Use of Reflexes: creation of the first schemas by exercising reflexive behaviors |  Standard |  Low |  10 (motor and virtual), low resolution |  $\bullet$ Stimulate new procedural memories
$\bullet$ Stimulate synchronization between movement and bottom-up attention
focus
$\bullet$ Punish failures
2 |  Primary Circular Reactions: creation and adaptation of schemas based on functions, but not intentions |  Standard + motivation |  Medium |  10 (motor and virtual), high resolution |  $\bullet$ Stimulate new procedural memories
$\bullet$ Stimulate synchronization between movement and bottom-up attention
focus
$\bullet$ Punish failures
3 |  Secondary Circular Reactions: creation and adaptation of schemas to form plans, objectives, and intentions |  Standard + motivation |  High |  17 (motor, virtual, attentional), high resolution |  $\bullet$ Stimulate new procedural memories
$\bullet$ Stimulate synchronization between movement and bottom-up attention
focus
$\bullet$ Stimulate synchronization between movement and top-down attention
(intention)
$\bullet$ Punish failures
TABLE II: Overview of the approach: schemas, vision resolution, actions and
resolutions, cognitive system and reward strategy adopted in each substage in
the experiments.

###  V-C Cognitive model

The cognitive system of the humanoid Marta was modeled according to CONAIM
[2]. A schematic of the adopted cognitive model is shown in Figure 2. The
first level of architecture is the attentional system [14], responsible for
collecting data from the environment, compressing this information and
selecting the most relevant points in the scene. The system’s inputs receive
multiple sensory information from the four camera channels (R, G, B, D) with a
previously configured resolution. These observations generate four distinct
_bottom-up_ feature maps ($\mathcal{F}_{R}$, $\mathcal{F}_{G}$,
$\mathcal{F}_{B}$, $\mathcal{F}_{D}$), which carry information about the most
discrepant signals in each channel. Three _top-down_ feature maps
($\mathcal{F}_{color}$, $\mathcal{F}_{dist}$, $\mathcal{F}_{reg}$) were also
adopted, which can emphasize particular aspects of color, distance and region
of the input data according to agent’s goals (if applicable). All feature maps
are weighted and combined into a single Combined Feature Map ($\mathcal{C}$)
that carries the information of the most relevant information considering all
input data. The Attentional Map ($\mathcal{M}$), which carries the information
about the attentional focus at time $t-1$ modulated by inhibition of return
(IOR) effects over previously selected points, is combined to $\mathcal{C}$
and produces the Salience Map ($\mathcal{L}$), which contains the most
relevant points of the visual field at present $t$.

(a) (b) (c)

Figure 2: Schematic model of the cognitive-attentive system adopted. a) Full
view; b) Details of the Cognitive System in 1st Substage, with some of the
modules ($\mathbb{MO},\mathbb{VO},\mathbb{MV},\mathbb{G},\textbf{t}$, _top-
down_) disabled (painted in grey); c) Details of the Cognitive System in 2nd
Substage and 3rd Substage.

As a result, all exogenous stimuli will compete against each other to be the
_winner_ of this bottom-up competitive process, i.e., the most relevant point
of the scene. Besides the _bottom-up_ process, any particular feature can also
be emphasized via an endogenous _top-down_ process. Via the top-down pathway,
a specific scene region or a desired color can receive the attentional focus
depending on the agent’s goal.

The second level of the architecture is the cognitive system, responsible for
modulating the relationship between the robot and the environment, as well as
for the cognitive evolution of the agent. In the present experiments, we
considered only some modules of CONAIM [2] cognitive system. These modules
were activated incrementally during successive experiments. Each
procedure/schema $m_{p}\in M_{P}$ represents learned knowledge stored in
procedural memory $\mathbb{M_{P}}$. Initially, the working memory
($\mathbb{M_{W}}$) receives the salience map ($\mathcal{L}$) – used as a state
in reinforcement learning – emerging from the attentional system. As the agent
does not know the beginning, a new procedure $m_{p}$ is created in
$\mathbb{M_{P}}$, and the cognitive agent can gradually learn something about
it. If the agent has some prior knowledge stored in $\mathbb{M_{P}}$ that fits
the current state, an _recall_ of procedures ($\mathbb{R_{P}}$) takes place.
Decision Maker ($\mathbb{D}$) will consider this knowledge. In some
experiments, a set of motivations $mv_{i}$ $\in\mathbb{M_{V}}$ was also
modeled to explore the use of new actions in some states. The volition
$\mathbb{VO}$ is the function responsible for transforming the agent’s
motivations into tasks that the decision process will also consider. A
procedural learning function ($\mathbb{L_{P}}$) is responsible for creating or
updating the content of $m_{p}\in M_{P}$, in this case acting respectively in
an analogous way to assimilation and accommodation in Piaget’s theory. The
cognitive model was fully implemented in _Java_ using CST [3]. Figure 3 shows
the implementation scheme of the CONAIM+CST architecture for the proposed
incremental learning.

Figure 3: Implementation scheme of the CONAIM+CST architecture with the robot
Marta that receives attentional stimuli _bottom-up_ and _top-down_.

###  V-D Reinforcement Learning (RL) in substages

Due to the trial and error nature of learning at early child developmental
stages [27], we adopted RL as the primary paradigm to learn state-action
pairs, that is, the agent’s procedures in the procedural memory. This section
details (1) states, (2) actions, and (3) learning in this approach.

####  V-D1 States

Before we can compute the agent State ($\mathbb{S}$), the input of our RL
algorithm, we must go back to the sensors data observation ($\mathcal{O}$).
Our approach gradually increases the robot’s visual acuity, refining the
agent’s perception of the world. Three different image resolutions were
adopted among the experiments: 64x64 pixels (1st Substage), 128x128 pixels
(2nd Substage), and 256x256 pixels (3rd Substage). The _bottom-up_ maps of the
RGB-D channels ($\mathcal{F_{R}}$, $\mathcal{F_{G}}$, $\mathcal{F_{B}}$ and
$\mathcal{F_{D}}$) were computed using an _average pool_ over the observation
of each color map at time $t$, and then the difference between each region
mean and the image mean. Since the resolution of the image changes among the
experiments, we computed the necessary size of the kernel and stride to reduce
the feature map to a final size of 16x16. In other words, the most discrepant
elements at each map at each time $t$ are highlighted. _Top-down Feature Maps_
($\mathcal{F}_{color}$, $\mathcal{F}_{dist}$ and $\mathcal{F}_{reg}$) allow
the agent to target its attention to desired elements deliberately. We compare
each pixel value to a particular (color, distance, or spatial region) goal to
build these maps. The closer these elements are to the target values according
to predefined percentage ranges, the higher the map activation in that region.
Our experiments adopted 20%, 40%, 60%, and 80% proximity ranges, respectively.
The corresponding attentional values are $1$, $0.75$, $0.5$, and $0.25$.
Particularly, in the _regions top-down Feature Map_ ($\mathcal{F}_{reg}$), the
visual space was divided into $5$ distinct regions, as shown in Figure 1 (d).
These regions define particular regions of interest for the agent. The
_Combined Feature Map_ ($\mathcal{C}$) computes an element-wise weighted mean
of the $i$ enabled _Features Maps_. An element-wise multiplication of the
_Attentional Map_ ($\mathcal{M}$) and ($\mathcal{C}$) results in the _Salience
Map_ ($\mathcal{L}$). Finally, we compute the State ($\mathbb{S}$) vector that
will be used as input for the learning algorithm. $\mathbb{S}$ is computed in
the _Working Memory_ ($\mathbb{M_{W}}$) using a _MaxPool_ operator with a 4x4
kernel and a 4 stride over the Salience Map ($\mathcal{L}$), generating a 4x4
matrix with a 2-level discretization per element obtained with a threshold.
This process results in a state vector of size 65.536 ($2^{16}$).

####  V-D2 Actions

In analogy to the typical actions performed by children in each substage, the
robot was allowed to perform 17 possible actions ($\mathbb{A}$), divided into
three groups: motor, virtual and attentional actions. The Motor Actions
($\mathbb{A}_{m}$) are the actions on the physical actuators on the robot’s
neck, capable of turning the head motors pitch and yaw. Virtual Actions
($\mathbb{A}_{v}$) are internal to the agent and simulate eye movement. The
virtual actuator selects a point in the visual space where the agent focuses
(eye). Attention Actions ($\mathbb{A}_{a}$) are divided into two subgroups.
The first group of actions involves directing the robot’s head toward the most
salient point in the image (winner). The second group refers to _top-down_
actions. It can emphasize specific colors, distances, or regions in resource
maps. Some of these possible actions have been enabled or disabled for each of
the three sets of experiments. Figure 4 presents the actions available to each
distinct substage and experiment.

Figure 4: Possible actions for the cognitive robot in experiments for 1st
Substage, 2nd Substage and 3rd Substage. Motor actions ($\mathbb{A}_{m}$): 1.
No-action; 2-5. Move neck pitch/yaw actuators with low discretization or Move
neck pitch/yaw actuators with high discretization; Virtual actions
($\mathbb{A}_{v}$): 8-10. Move virtual actuators (eyes) to particular image
zones; Attentional actions ($\mathbb{A}_{a}$): 11-14. Move neck pitch/yaw
actuators towards the attentional stimulus; _Top-down Attentional Actions_ :
15. Emphasize stimuli of a particular color; 16. Emphasize stimuli at a
particular distance; 17. Emphasize stimuli in a particular region of the
space.

####  V-D3 Learning

The _Learning Proccess_ ($\mathbb{L_{P}}$) has a central role in the current
investigation. We selected a _Reinforcement Learning_ (RL) algorithm, the
Q-learning [5], for the cognitive agent’s learning. The memory elements
$m_{p}\in\mathbb{M_{P}}$ were modeled as _QTables_ capable of storing State-
Action pairs ($\mathbb{S}\rightarrow\mathbb{A}$) for particular procedures.
The states ($\mathbb{S}$) were modeled from the saliency maps ($\mathcal{L}$)
that represent the environment. Reinforcement positively rewards the robot if
there is space-time synchronization between the visual stimulus (most salient
point of the image) and the robot’s current focus (motor or virtual). There is
no reward if there is no such timing, and the reward is strongly negative if
the robot loses balance. The learning mechanism remains unchanged during all
experiments. Figure 5 details the reinforcement policy for some states of the
agent.

Figure 5: Reinforcement for the cognitive robot depends on the current state
($s_{i}$) and previous action. Reinforcement for motor actions (Rma): The
system is positively reinforced if the direction the robot’s head moved
matches the emergence of a visual stimulus; The system receives no
reinforcement if there is no such space-time synchronicity; The system is
strongly negatively reinforced if the robot loses its balance; Reinforcement
for virtual actions (Rva): The system is positively reinforced if the
direction in which the virtual actuator (eye) moved matches the emergence of a
visual stimulus; The system receives no reinforcement if there is no such
spatial-temporal synchronicity.

###  V-E Proposed Experiments

Three sets of experiments (EXP-01/03) were proposed based on the scenarios to
assess the agent’s ability to learn to track other agents or objects, proposed
in section IV [6]. During training, for each episode, we initialize the agent
in a fixed position in the environment, adding random noise to its actuators.

The episode ends when one of the following conditions is reached: i) the agent
reached the maximum number of steps/actions; ii) the robot falls over or
exceeds the limits of its motorized actuators, or iii) the robot has no
saliences for several iterations. The following parameters were adopted:

  * •

Simulation parameters. Maximum number of episodes: 200; Maximum number of
steps: 500; Maximum number of iterations without bosses: 5;

  * •

Learning parameters. A $\epsilon-$greedy policy was employed with an
exploitation rate starting at 0.95 and linearly decaying to 0 in the last
episode. Learning rate $\alpha$: 0.9; Time discount rate $\gamma$: 0.99;

  * •

Visual acuity / Vision sensor resolutions. 64x64 for $1^{st}$ substage;
128x128 for $2^{nd}$ substage; 256x256 for $3^{rd}$ substage;

  * •

Rewards. +1 reward for each new data inserted in procedural memory; +1 for
holding or directing an actuator (motor or virtual) to the attention winner;
-10 if the agent falls or exceeds the limits of physical actuators, or if the
agent has no saliences in its attentional cycle for several iterations; and,
only for $3^{rd}$ substage, +1 when the agent identifies regions in which a
certain desired characteristic is highlighted according to the _top-down_
process.

Table III details the experiments carried out, the cognitive system, available
modules, functions and learning.

EXP |  Substage |  Vision resolution |  Bottom-up features |  Memories |  Actions |  Cognitive functions |  RL Reward strategy
---|---|---|---|---|---|---|---
01 |  1 |  64x64 pixels |  RGBD |  $\bullet$ Short-term: Sensory Memory ($\mathbb{M_{S}}$) and Working Memory ($\mathbb{M_{W}}$)
$\bullet$ Long-term: Procedural Memory ($\mathbb{M_{P}}$) |  $\bullet$ Motor actions ($\mathbb{A}_{m}$): stay focused; move physical actuators
$\bullet$ Virtual action ($\mathbb{A}_{v}$): move virtual actuator
$\bullet$ Attentional actions ($\mathbb{A}_{a}$): - |  $\bullet$ Procedural Learning ($\mathbb{L_{P}}$): Q-Learning
$\bullet$ Motivation ($\mathbb{MO}$): - |  The agent receives a +1 reward for each new data inserted into the procedural memory and also when it directs its face (physical) or eyes (virtual) to the winner of the attentional process. The agent receives a -10 reward if it falls, exceeds the actuator limits, or does not have saliences in its attentional cycle for a number of iterations
02 |  2 |  128x128 pixels |  RGBD |  $\bullet$ Short-term: Sensory Memory ($\mathbb{M_{S}}$) and Working Memory ($\mathbb{M_{W}}$)
$\bullet$ Long-term: Procedural Memory ($\mathbb{M_{P}}$) |  $\bullet$ Motor actions ($\mathbb{A}_{m}$): stay focused; move physical actuators
$\bullet$ Virtual action ($\mathbb{A}_{v}$): move virtual actuator
$\bullet$ Attentional actions ($\mathbb{A}_{a}$): - |  $\bullet$ Procedural Learning ($\mathbb{L_{P}}$): Q-Learning
$\bullet$ Motivation ($\mathbb{MO}$): Motivation modifies the curiosity drive’s activation. This driver incentives actions not tried in a particular state. Hence, encouraging new sensorimotor discoveries |  same as previous
03 |  3 |  256x256 pixels |  RGBD |  $\bullet$ Short-term: Sensory Memory ($\mathbb{M_{S}}$) and Working Memory ($\mathbb{M_{W}}$)
$\bullet$ Long-term: Procedural Memory ($\mathbb{M_{P}}$) |  $\bullet$ Motor actions ($\mathbb{A}_{m}$): stay focused; move physical actuators
$\bullet$ Virtual action ($\mathbb{A}_{v}$): move virtual actuator
$\bullet$ Attentional actions ($\mathbb{A}_{a}$): direct attentional focus according to features top-down |  $\bullet$ Procedural Learning ($\mathbb{L_{P}}$): Q-Learning
$\bullet$ Motivation ($\mathbb{MO}$): Motivation modifies the curiosity drive’s activation. This driver incentives actions not tried in a particular state. Hence, encouraging new sensorimotor discoveries |  The agent receives +1 reward for each new data inserted in the procedural memory; +1 for keeping or directing an actuator (motor or virtual) to the winner of the attention; +1 when the agent identifies regions in which a certain desired feature is highlighted according to the top-down process; -10 for falling, motor actuators limits exceed, or if it does not have saliences in its attentional cycle for several iterations
TABLE III: Details of the experiments: substages, vision resolution, memories,
actions, cognitive functions and reward strategy.

####  V-E1 1st Substage: Use of Reflexes

The 1st Substage experiments dynamics is schematically presented in Figure 6.
This set of experiments investigates a computational process proposed to model
reflex reactions. In this set, there is no intentionality or motivation.

Figure 6: System dynamics in 1st Substage. $t_{1}$: The robot sensors sample
the environment; $t_{2}$: The attentional system generates the saliency map
($\mathcal{L}_{1}$); $t_{3}$: The working memory ($\mathbb{M_{W}}$) identifies
this new state of the world ($\mathbb{S}_{1}$) based on ($\mathcal{L}_{1}$).
The recall function ($\mathbb{R_{P}}$) is called to look for for any procedure
in $\mathbb{M_{P}}$ that could be applicable to the current state
$\mathbb{S}_{1}$; $t_{4}$: Such procedure is not found in $\mathbb{M_{P}}$;
$t_{5}$: The decision maker ($\mathbb{D}$) informs that the robot do not know
what to do in the current state; $t_{6}$: The decision maker ($\mathbb{D}$)
decides to start a new QTable associated to this state ($\mathbb{S}_{1}$) in
$\mathbb{M_{P}}$; $t_{7}$: The new QTable is created with random values,
enabling the cognitive robot to learn more about this state; $t_{8}$: The
recall function $\mathbb{R_{P}}$ returns the possible actions associated to
this state and their $\mathbb{Q}$ values; $t_{9}$: current state
$\mathbb{S}_{1}$ and possible actions are sent to the decision maker
($\mathcal{D}$); $t_{10}$: The decision maker ($\mathcal{D}$) chooses the next
action between a random action and the best action for the current state
($\mathcal{Q}_{max}$) according to an $\epsilon$-_greedy_ policy. In this
example, the action to move the robot’s head to the right ($\mathbb{A}_{2}$)
was selected; $t_{11}$: A new salience map ($\mathcal{L}_{2}$) is generated by
the attentional system; $t_{12}$: The working memory ($\mathbb{M_{W}}$)
identifies the new state ($\mathbb{S}_{2}$) based on ($\mathcal{L}_{2}$);
$t_{13}$: The decision maker ($\mathcal{D}$) evaluates the current state
($\mathbb{S}_{2}$) and the previous action performed ($\mathbb{A}_{2}$). Since
there is a synchronicity between the robot’s head-resulting position and the
external stimulus, a positive reinforcement $R_{\mathbb{S}_{1}\mathbb{A}_{2}}$
increases the $\mathbb{Q}$ value for the action $\mathbb{A}_{2}$ in the state
$\mathbb{S}_{1}$; $t_{14}$: The $\mathbb{Q}$ value is sent to the QTable;
$t_{15}$: The QTable is updated; $t_{16}$: A new recall $\mathbb{R_{L}}$ looks
for the possible actions for the current state $\mathbb{S}_{2}$; $t_{17}$: The
possible actions are found in $\mathbb{M_{P}}$; $t_{18}$: The decision maker
($\mathbb{D}$) is informed about the current state $\mathbb{S}_{2}$ and the
possible actions; $t_{19}$: The decision maker ($\mathbb{D}$) decides by the
action $\mathbb{A}_{4}$ (move the head bellow).

####  V-E2 2nd Substage: Primary Circular Reactions

In this set of experiments, we investigated whether the reflex reactions in
the agent, initiated in the 1st substage, can evolve into behaviors similar to
the primary circular reactions proposed by Piaget. As in the previous
experiment, only the _bottom-up_ stimuli were considered. Motivation. A
motivation function ($\mathbb{MO}$) is applied, related to the agent’s
curiosity about the effects of its actions, to encourage the agent to explore
schemes that have not yet been explored in the current episode. All previously
learned contents of $\mathbb{M_{P}}$ are preserved.

####  V-E3 3rd Substage: Secondary Circular Reactions

In this last set of experiments, we investigate the computational process by
which behaviors associated with the secondary circular reactions proposed by
Piaget can be observed. We verified whether the agent could intentionally
select an action that would allow him to reach a goal. The _top-down_
attention mechanism is used in this phase. All previously learned contents of
$\mathbb{M_{P}}$ are preserved.

##  VI Results and Discussion

We carried out three experiment sets to train and validate the integrated
architecture, corresponding to the first three substages of the sensorimotor
period in Piaget’s Theory [4]. At the end of each training episode, the reward
obtained and the number of actions performed were restarted and the robot
actuators returned to the starting position. The Pioneer P3DX robot was
randomly positioned in the scene. Figure 7 demonstrates the resulting reward
and the number of actions performed per episode for each learning experiment
performed. It can be noted from the top graphs that, as the agent reuses
knowledge from previous substages, both the reward and the number of actions
are greater for a more developed agent ($3rd>2nd>1st$). The bottom images
depict the training results when we do not reuse knowledge from prior stages.
We can note that, in these cases, either no learning occurs (the reward for
the 2nd substage experiment does not increase over episodes) or it results in
very low rewards when compared to the scenario where knowledge was reused (3rd
substage - reward peak around 200 versus 500).

Figure 7: Top: Resulting reward (left) per episode and number of actions
(right) for each learning experiment for all substages when incremental
learning is in course. Bottom: Resulting reward (left) per episode and number
of actions (right) for each learning experiment for all substages with no
incremental learning.

###  VI-A 1st Substage: Use of Reflexes.

The experiment demonstrates the attentional course of selection for perception
for the agent executing only the _bottom-up_ attentional course. The results
obtained for Procedural Learning (training) in this experiment are shown in
Figure 8. In the first episodes, the agent explores the limits of its
actuators due to the high exploration rate while promoting the refinement of
state-action pairs for the fovea selection virtual actuator. The agent
established its attentional focus on the Pioneer P3DX robot while it moved in
regions closer to the humanoid. With the distance from the Pioneer P3DX robot,
the agent directed its attention to other nearby objects and its own body. The
stimuli obtained while exploring the agent’s body reinforced the reflexes
used. The absence of a motivation system made the agent perform the reinforced
actions in greater quantity, even when the stimuli that promoted this
reinforcement were no longer present and in smaller quantities the actions
that did not participate in these interactions.

Figure 8: 1st Substage. Sensory data obtained in the 1st episode of Procedural
Learning. Left to Right: (a) Overview of the scene in _CoppeliaSim_ (t = 1s);
(b) Marta’s camera view (t = 1s); (c) Salience Map (t = 3s); (d) Winner of the
attentional cycle (t = 3s); (e) Agents and objects’ positions in scene.

Learning Validation - 1st Substage: For the validation of the agent in this
substage, the QTable resulting from the end of the last episode of Procedural
Learning was used. To evaluate the learned policy, 100 test episodes were
executed without updating the learning parameters for each experiment, with a
maximum of 500 actions.

####  VI-A1 Experiment A - 1st Substage - Object in fixed position and with
primary color

The humanoid Marta was positioned 80cm from the Pioneer P3DX robot. The
results obtained for this experiment are shown in Figure 9. The agent
initially directed its attention to the Pioneer P3DX robot, which remained
stationary during this experiment, as suggested by Berto (2020). [6]. However,
the action of excitatory and inhibitory cycles promoted by the CONAIM
attentional system directed the attentional focus to regions closer to the
humanoid. The performance of the reinforced reflexes during the exploration of
the agent’s body in Procedural Learning resulted in the alternation of the
agent’s actuators between the regions closest to the humanoid and the Pioneer
P3DX robot. As expected, the robot learned to respond to salient stimuli using
reflexes.

Figure 9: 1st Substage. Sensory data obtained in validation Experiment A. Left
to right: (a) Overview of the scene in _CoppeliaSim_ (t = 40s); (b) Marta’s
camera view (t = 40s); (c) Salience Map (t = 43s); (d) Winner of the
attentional cycle (t = 43s); (e) Agents and objects’ positions in scene.

####  VI-A2  Experiment B - 1st Substage - Object moving slowly and of primary
color

Marta was positioned 80cm in front of the Pioneer P3DX robot, which has a
Braitenberg algorithm and moves at a constant speed of 0.1m/s. The results of
this experiment are shown in Figure 10. The agent directed its attention to
the Pioneer P3DX robot while it remained in the regions closest to the agent
(frontal region). However, as the Pioneer P3DX moves to the lateral areas, the
performance of the reinforced reflexes during Procedural Learning again
resulted in directing attentional focus to the regions closer to the humanoid.
Thus, Marta could not track the moving object outside its visual field, as
expected for this developmental stage.

Figure 10: 1st Substage. Sensory data obtained in validation Experiment B.
Left to right: (a) Overview of the scene in the simulator _CoppeliaSim_ (t =
40s); (b) Marta’s camera view (t = 40s); (c) Salience Map (t = 43s); (d)
Winner of the attentional cycle (t = 43s); (e) Agents and objects’ positions
in scene.

###  VI-B 2nd Substage: Primary Circular Reactions.

In this experiment, Marta continues with only _bottom-up_ perception elements.
With the implementation of a motivation model, the agent starts to explore
possible actions that do not have defined schemes in the Procedural Memory
${M}_{p}$. The reflex reactions developed in the 1st substage can generate
primary circular reactions, stabilizing the learning of certain actions. The
results obtained for Procedural Learning (training) in this experiment are
shown in Figure 11. Using the QTable from the previous substage allows the
agent to perform better in the object-tracking task. The Attentional System of
the 2nd Substage, during Procedural Learning, allowed the agent to establish
its attentional focus on the Pioneer P3DX robot while it moved in regions
closer to the humanoid, as in the 1st substage. With the withdrawal of the
Pioneer P3DX robot, the agent returned to direct its attention to nearby
objects and its own body. However, with the action of the motivation system,
the agent was motivated to explore all possible actions for each new scheme
not found in the Procedural Memory. This behavior minimized the performance of
the reinforced actions developed in the 1st substage, allowing the agent to
acquire greater rewards and promoting the formation of primary circular
reactions.

Figure 11: 2nd Substage. Sensory data obtained in the 1st episode of
Procedural Learning. Left to right. (a) Overview of the scene in the simulator
_CoppeliaSim_ (t = 1s); (b) Marta’s camera view (t = 1s); (c) Salience Map (t
= 3s); (d) Winner of the attentional cycle (t = 3s); (e) Agents and objects’
positions in scene.

Learning Validation - 2nd Substage

####  VI-B1 Experiment A - 2nd Substage - Moving object with primary color

The QTable resulting from the end of the last episode of Procedural Learning
was used. Marta was positioned with the Pioneer P3DX robot out of its field of
view, as illustrated in the scene in Figure 12 (a). The Pioneer P3DX robot has
a Braitenberg algorithm and moves at a constant speed of 0.1m/s. The agent
directed its attention to the closest regions of its body when the Pioneer
P3DX robot left its field of vision. The use of primary circular reactions
promoted a performance with more actions performed compared to the previous
substage. The increased visual acuity and the refinement of actuator movement
in this substage also gave the agent greater control over its actuators.

Figure 12: 2nd Substage. Sensory data obtained in Experiment A. Left to right.
(a) Overview of the scene in the simulator _CoppeliaSim_ (t = 1s); (b) Marta’s
camera view (t = 1s); (c) Salience Map (t = 3s); (d) Winner of attentional
cycle (t = 3s); (e) Agents and objects’ positions in scene.

###  VI-C 3rd Substage: Secondary Circular Reactions.

In this substage, the humanoid Marta has a cognitive-attentional algorithm
that has all the elements shown in Figure 2, with elements of perception
_bottom-up_ and _top-down_ , and models of intentionality and motivation. The
agent can exploit the primary circular reactions developed in the previous
steps and develop secondary circular reactions. The results obtained for
Procedural Learning (training) in this experiment are shown in Figure 13. The
Attention System of the 3rd Substage, during Procedural Learning, allowed the
agent to establish its focus of attention on the P3DX robot. By using
attentional actions, the agent could follow the movement of the P3DX even when
it was in the most distant regions of the humanoid and out of its field of
vision.

Figure 13:  3rd Substage. Sensory data obtained in the 1st episode of
Procedural Learning. Left to right. (a) Overview of the scene in the simulator
_CoppeliaSim_ (t = 1s); (b) Marta’s camera view (t = 1s); (c) Salience Map (t
= 3s); (d) Winner of attentional cycle (t = 3s); (e) Agents and objects’
positions in scene.

Learning Validation - 3rd Substage

####  VI-C1 Experiment A - 3rd Substage - Moving object and primary color.

The QTable resulting from the last episode of Procedural Learning was used. We
positioned Marta with the Pioneer P3DX robot out of its field of view. The
Pioneer P3DX robot uses the Braitenberg algorithm and moves at a constant
speed of 0.1m/s. In this experiment, the agent maintained its focus on the
Pioneer P3DX robot even when it was far away because the agent was able to
track the moving robot. This is mostly caused by motivation and intention to
follow the moving object. The increased visual acuity in this substage and the
refinement of actuator movement gave the agent greater control over its
actuators than in previous substages. The secondary circular reactions
developed during this substage promoted a higher performance than in previous
substages, as we can see in Figure 7.

Figure 14: 3rd Substage. Sensory data obtained in validation experiment A.
Left to right. (a) Overview of the scene in the simulator _CoppeliaSim_ (t =
1s); (b) Marta’s camera view (t = 1s); (c) Salience Map (t = 3s); (d) Winner
of the attentional cycle (t = 3s); (e) Agents and objects’ positions in scene.

##  VII Conclusion

In this work, we proposed and implemented an incremental procedural learning
mechanism to create and reuse previously learned schemas inspired by Piaget’s
sensorimotor development substages. To this end, we employed a simulated
humanoid robot and static and moving objects. By building our cognitive agent,
we investigated which modules in a cognitive architecture are needed to
control a robot interacting with its environment while performing a set of
sensorimotor experiments with increasing difficulty. We discussed the
importance of motivation and attention to forming primary and secondary
circular reactions from reflexes. This approach allowed for employing a single
incremental mechanism that evolves over time. We showed that reusing previous
knowledge is mandatory for the success of incremental learning. The
experiments demonstrated the feasibility of using a cognitive-attentional
architecture based on CONAIM and implemented with CST. Furthermore, we
successfully implemented experiments corresponding to the first three
substages proposed by Berto (2020) [6] for tracking objects. With these
experiments, we could show which cognitive functions are required to achieve
specific levels of development through object-tracking experiments.

##  VIII Acknowledgments

This work was developed within the scope of PPI-Softex with support from MCTI
through the Technical Cooperation Term [01245.013778/2020-21].

## References

  * [1] J. Reggia, “The rise of machine consciousness: studying consciousness with computational models.” _Neural Networks_ , vol. 44, 2013. [Online]. Available: www.sciencedirect.com/science/article/abs/pii/S0893608013000968
  * [2] A. S. Simões, “Um modelo cognitivo baseado na atenção para consciência em robôs inteligentes,” _Universidade Estadual Paulista (Tese de Livre-Docência)_ , 2015.
  * [3] A. L. O. Paraense, K. Raizer, S. M. de Paula, R. R. Gudwin, and E. Rohmer, “The cognitive systems toolkit and the cst reference cognitive architecture,” _Biologically Inspired Cognitive Architectures_ , vol. 17, pp. 32–48, 2016. [Online]. Available: www.sciencedirect.com/science/article/pii/S2212683X1630038X
  * [4] J. Piaget, _The origins of intelligence in children._ , ser. The origins of intelligence in children., M. Cook, Ed. New York, NY, US: W Norton & Co, 1952.
  * [5] R. Sutton and A. Barto, _Reinforcement Learning: An Introduction._ MIT Press. Cambridge, Massachussetts., 2015.
  * [6] L. M. Berto, “Exploring cognitive functions in robotics,” _Institute of Computing of the University of Campinas (Master thesis)_ , 2020.
  * [7] I. Kotseruba and J. Tsotsos, “40 years of cognitive architectures: core cognitive abilities and practical applications.” _Artificial Intelligence Review_ , vol. 53, 2020. [Online]. Available: link.springer.com/article/10.1007/s10462-018-9646-y
  * [8] D. Vernon, C. von Hofsten, and L. Fadiga, “Desiderata for developmental cognitive architectures,” _Biologically Inspired Cognitive Architectures_ , vol. 18, pp. 116–127, 2016. [Online]. Available: https://www.researchgate.net/publication/309333693_Desiderata_for_developmental_cognitive_architectures
  * [9] J. Anderson and C. Lebiere, “The newell test for a theory of mind.” _Carnegie Mellon University_ , 1998. [Online]. Available: https://www.researchgate.net/publication/250037730_The_Newell_Test_for_a_Theory_of_Mind
  * [10] J. E. Laird, “Extending the soar cognitive architecture,” in _Proceedings of the First AGI Conference - Frontiers in Artificial Intelligence and Applications_. Amsterdam, Netherlands: Artificial General Intelligence by P. Wang, B. Goertzel, and S. Franklin, 2008, pp. 224––235. [Online]. Available: https://www.researchgate.net/publication/221328941_Extending_the_Soar_Cognitive_Architecture
  * [11] R. Thomson, S. Bennati, and C. Lebiere, “Extending the influence of contextual information in act-r using buffer decay,” in _Proceedings of the Annual Meeting of the Cognitive Science Society_ , 2014, p. 6. [Online]. Available: www.researchgate.net/publication/268520256_Extending_the_Influence_of_Contextual_Information_in_ACT-R_using_Buffer_Decay
  * [12] A. d. S. Simões, E. L. Colombini, and C. H. C. Ribeiro, “CONAIM: A Conscious Attention-Based Integrated Model for Human-Like Robots,” _IEEE Systems Journal_ , vol. 11, no. 3, pp. 1296–1307, 2017. [Online]. Available: ieeexplore.ieee.org/document/7383269
  * [13] E. L. Colombini, A. da Silva Simões, and C. H. C. Ribeiro, “An attentional model for autonomous mobile robots,” _IEEE Systems Journal_ , vol. 11, no. 3, pp. 1308–1319, 2017. [Online]. Available: ieeexplore.ieee.org/document/7384702
  * [14] E. L. Colombini, “An attentional model for intelligent robotics agents,” _Instituto Tecnológico de Aeronáutica, São José dos Campos (PhD Thesis)_ , 2014.
  * [15] R. R. Gudwin, K. Raizer, A. L. O. Paraense, S. M. de Paula, V. A. de Figueiredo, E. C. de Castro, E. Fróes, and W. S. P. Gibaut, “Cst - the cognitive systems toolkit,” 2019. [Online]. Available: https://github.com/CST-Group/cst
  * [16] R. D. Beer, “The Dynamics of Active Categorical Perception in an Evolved Model Agent,” _Adaptive Behavior_ , vol. 11, no. 4, pp. 209–243, 2003. [Online]. Available: journals.sagepub.com/doi/10.1177/1059712303114001
  * [17] A. Cangelosi and M. Schlesinger, _Developmental Robotics: From Babies to Robots_. The MIT Press, 2014. [Online]. Available: https://mitpress.mit.edu/books/developmental-robotics
  * [18] A. Stoytchev, “Some basic principles of developmental robotics,” _IEEE TRANSACTIONS ON AUTONOMOUS MENTAL DEVELOPMENT_ , vol. 1, 2009. [Online]. Available: https://ieeexplore.ieee.org/document/5200465?arnumber=5200465
  * [19] J. Piaget, _Seis estudos da Psicologia._ , M. A. M. D’Amorim and P. S. L. Silva, Eds. Forense Universitária, 1999\.
  * [20] ——, _Science of Education and the Psychology of the Child._ Penguin Books, 1971.
  * [21] D. Michalec, _Bayley Scales of Infant Development: Third Edition_. Boston, MA: Springer US, 2011, pp. 215–215. [Online]. Available: https://doi.org/10.1007/978-0-387-79061-9_295
  * [22] D. Boyd and H. Bee, _A Criança em Crescimento._ ARTMED, 2011.
  * [23] D. E. Papalia and R. D. Feldman, _Desenvolvimento Humano._ Mcgraw Hill - Artmed, 2013, vol. 12.
  * [24] F. Guerin, N. Kruger, and D. Kraft, “A Survey of the Ontogeny of Tool Use: From Sensorimotor Experience to Planning,” _IEEE Transactions on Autonomous Mental Development_ , vol. 5, no. 1, pp. 18–45, 2013. [Online]. Available: ieeexplore.ieee.org/document/6248675/
  * [25] R. Arrabales, “Consscale - a machine consciousness scale,” 2020. [Online]. Available: https://www.conscious-robots.com/consscale/index.html
  * [26] V. Braitenberg, _Vehicles: Experiments in Synthetic Psychology_. Cambridge MA: MIT Press, 1984.
  * [27] K. Nussenbaum and C. A. Hartley, “Reinforcement learning across development: What insights can we draw from a decade of research?” _Developmental Cognitive Neuroscience_ , vol. 40, p. 100733, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1878929319303202

|  Leonardo de Lellis Rossi graduated in Control and Automation Engineering
and received a Master’s degree in Electrical Engineering from the Institute of
Science and Technology of Sorocaba (ICTS) - Unesp. He is a Ph.D. candidate in
Electrical Engineering at the Faculty of Electrical and Computing Engineering
(FEEC/UNICAMP) and a member of the Cognitive Architectures area at H.IAAC (Hub
of Artificial Intelligence and Cognitive Architectures). His research
interests include Cognitive Robotics, Developmental Robotics, Machine
Learning, and Deep Learning.
---|---
|  Letícia Mara Berto has a B.S. degree in Computer Science at Federal
University of São Carlos (UFSCar), Brazil, an M.S. degree in Computer Science
at the Institute of Computing (IC), State University of Campinas (Unicamp),
Brazil. She is a Ph.D. candidate at the Institute of Computing (IC), State
University of Campinas (Unicamp), Brazil, and is visiting researcher at the
Italian Institute of Technology (IIT), Genova, Italy. Her research interests
include cognitive robotics, affective computing and human-robot interaction.
---|---
|  Eric Rohmer is a Professor at the Faculty of Electrical and Computer
Engineering, State University of Campinas (Unicamp), Brazil. He received the
M.S. in 2000 at the Ecole Supérieure d’Informatique et Applications de
Lorraine, Nancy, France, and the Ph.D. in Computer Science in 2005 at the
Advance Robotics Laboratory, Sendai, Japan. He acted as a researcher at the
Space Robotic Laboratory and at the Rescue Robotics Laboratory in Japan. His
research interests include planetary exploration, rescue robots, mobility on
rough terrain, reconfigurable mobile robots, assistive robotics and robot
control.
---|---
|  Paula Paro Costa is a Professor at the Faculty of Electrical and Computer
Engineering, State University of Campinas (Unicamp), Brazil. She received a
B.S. degree in Electrical Engineering in 2001 at the State University of
Campinas, Brazil, an M.S. in Mobile Communication in 2006 at the Polytechnique
of Torino, Italy, an M.S. (2009) and the Ph.D. (2015) in Computer Engineering
at State University Of Campinas, Brazil. Her research interests include
multimodal artificial intelligence, affective computing, and science
disclosure for young people.
---|---
|  Ricardo Ribeiro Gudwin is an Associate Professor at the Faculty of
Electrical and Computer Engineering, State University of Campinas - Brazil. He
was born in Campinas-SP, Brazil, in 1967. He received a B.S. degree in
Electrical Engineering in 1989, an M.S. degree in Electrical Engineering in
1992, and a Ph.D. in Electrical Engineering in 1996, all from the Faculty of
Electrical and Computer Engineering, State University of Campinas - Brazil.
His earlier research interests include fuzzy, neural, and evolutionary
systems. His current research interests include the study of cognitive
architectures, cognitive systems, intelligence and intelligent systems,
intelligent agents, semiotics, and computational semiotics.
---|---
|  Esther Luna Colombini is a Professor at the University of Campinas, Brazil,
where she coordinates the Laboratory of Robotics and Cognitive Systems
(LaRoCS). She holds an M.Sc. and a Ph.D. in computing engineering from the
Technological Institute of Aeronautics, Brazil. Her research interests include
machine learning and AI for autonomous robotics and healthcare, focusing on
Reinforcement Learning, Attentive Models and Cognitive Modeling. She is a
research member of the Advanced Institute for Artificial Intelligence (AI2),
the Brazilian Institute of Neuroscience and Neurotechnology (BRAINN), and the
Center for Artificial Intelligence (C4Ai). She acted as President of RoboCup
Brazil, co-founder, and chair of the Brazilian Robotics Olympiad and the
National Robotics Fair, two public initiatives - supported by the National
Council of Technological and Scientific Development - reaching yearly over
230,000 students aiming at attracting them to STEM careers. She coordinates
the multilateral BRICSmart Alliance supported by CNPq and the BRICS-STI
committee to employ ICT for Smart Resource Utilization to Combat Global
Pandemic Outbreaks and the Learning in Cognitive Architectures research in the
Hub for Artificial Intelligence and Cognitive Architectures (H.IAAC).
---|---
|  Alexandre da Silva Simões is an Associate Professor at the Control and
Automation Engineering Department (DECA) at Institute of Science and
Technology (ICT), Campus Sorocaba, São Paulo State University (Unesp), Brazil.
He received a B.S. degree in Electrical Engineering at Unesp, Campus Bauru, in
1998, an M.S. degree in Electrical Engineering (2000), and a Ph.D. in
Electrical Engineering (2006), both from the Polytechnic School of the São
Paulo University (USP), Brazil. His research interests include robotics,
machine learning, cognitive systems, and the interface between these research
fields, science disclosure, education, and arts.
---|---