# The $\operatorname{\mathcal{X}}$-series of a $p$-group and complements of
abelian subgroups

Stefanos Aivazidis  Department of Mathematics & Applied Mathematics,
University of Crete, Greece<EMAIL_ADDRESS>and  Maria Loukaki  Department
of Mathematics & Applied Mathematics, University of Crete, Greece
<EMAIL_ADDRESS>
###### Abstract.

Let $G$ be a $p$-group. We denote by $\operatorname{\mathcal{X}}_{i}(G)$ the
intersection of all subgroups of $G$ having index $p^{i}$, for
$i\leqslant\log_{p}(|G|)$. In this paper, the newly introduced series
$\\{\operatorname{\mathcal{X}}_{i}(G)\\}_{i}$ is investigated and a number of
results concerning its behaviour are proved. As an application of these
results, we show that if an abelian subgroup $A$ of $G$ intersects each one of
the subgroups $\operatorname{\mathcal{X}}_{i}(G)$ at
$\operatorname{\mathcal{X}}_{i}(A)$, then $A$ has a complement in $G$.
Conversely if an arbitrary subgroup $H$ of $G$ has a normal complement, then
$\operatorname{\mathcal{X}}_{i}(H)=\operatorname{\mathcal{X}}_{i}(G)\cap H$.

###### Key words and phrases:

$p$-groups, complements, Gaschütz’s theorem

###### 2010 Mathematics Subject Classification:

20D15, 20D25, 20E28

The first author is partially supported by the Hellenic Foundation for
Research and Innovation, Project HFRI-FM17-1733.

##  1\. Introduction

In this paper, we shall only be concerned with finite groups. We begin with
some general remarks on complementation. For a fuller discussion in textbook
format, the reader is invited to consult the recent monograph by Kirtland
[Kir17].

The (rather broad) question we ask is the following:

> Let $G$ be a group and let $N$ be a normal subgroup of $G$. Under which
> circumstances is $N$ guaranteed to have a complement in $G$?

Perhaps the most well-known and most general answer is the celebrated Schur-
Zassenhaus theorem which asserts that if $N$ is a Hall subgroup of $G$ (so
that $\gcd(|N|,|G:N|)=1$), then $N$ does indeed have a complement. In fact,
the theorem asserts more: all complements of $N$ in $G$ are $G$-conjugate. The
conjugacy part of the theorem relies on a solubility assumption: either $N$ or
$G/N$ must be soluble for it to admit a proof. This, in turn, can only be made
to hold unconditionally by an appeal to the deep Odd Order Theorem of Feit and
Thompson [FT63].

Another answer to the question above (relaxing the requirement in the abelian
case of the Schur-Zassenhaus theorem) is furnished by a theorem of Gaschütz
[Gas52]. For recent work discussing Gaschütz’s theorem, see [Sam22].

######  Theorem 1.1 (Gaschütz).

Let $N$ be an abelian normal subgroup of a group $G$. Let $N\leqslant
H\leqslant G$ such that $N$ has a complement in $H$ and $\gcd(|N|,|G:H|)=1$.
Then $N$ has a complement in $G$.

We see therefore that if $N$ is normal in $G$ and abelian then $N$ has a
complement in $G$ if and only if for each prime $p$ dividing $|N|$ the unique
Sylow $p$-subgroup of $N$ (which is normal in $G$) has a complement in some
(and thus in every) Sylow $p$-subgroup of $G$. In fact, the previous statement
is a Reduktionsatz in Gaschütz’s paper.

This global-to-local reduction that Gaschütz’s theorem achieves reaches an
obvious and natural limit if $G$ is itself a $p$-group. Thus we are led to
ask:

> Let $G$ be a $p$-group and $N$ be a (possibly normal) abelian subgroup of
> $G$. Is there a necessary and sufficient condition for $N$ to have a
> complement in $G$?

In fact, the previous question was the exact point of departure for the work
reported here.

Although we have not been able to formulate such a condition, we have found a
condition which is sufficient (and in some cases necessary). We delay
presenting the said result to discuss properties of the
$\operatorname{\mathcal{X}}$-series of a $p$-group first. This is done in
Section 2. Briefly, given a $p$-group $G$, we define
$\operatorname{\mathcal{X}}_{i}(G)$ to be the intersection of all subgroups of
$G$ having index $p^{i}$ in $G$ for $i\leqslant\log_{p}(|G|)$ and (for
technical reasons) extend its definition to all non-negative integers. We call
the associated series $\\{\operatorname{\mathcal{X}}_{i}(G)\\}_{i}$ the
$\operatorname{\mathcal{X}}$-series of $G$. This series has several nice
properties:

  * •

It is subgroup-monotone; cf. Proposition 2.4.

  * •

It respects direct products; cf. Theorem 2.8.

  * •

For every $i,j\geqslant 0$ we have
$\operatorname{\mathcal{X}}_{i}(\operatorname{\mathcal{X}}_{j}(G))\leqslant\operatorname{\mathcal{X}}_{i+j}(G)$.
Moreover, successive quotients of the $\operatorname{\mathcal{X}}$-series of
$G$ are elementary abelian $p$-groups; cf. Lemma 2.5.

  * •

For every $i\geqslant 0$ we have
$\gamma_{p^{i-1}+1}(G)\leqslant\operatorname{\mathcal{X}}_{i}(G)$; cf.
Proposition 2.11.

In Section 3 we resume our discussion by focussing first on abelian groups and
what is known in that setting as regards complementation. We then utilise the
tools assembled in Section 2 to prove our main result:

######  Theorem 1.2.

Let $A$ be an abelian subgroup of the $p$-group $G$ and suppose that

$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$

for all indices $i\geqslant 0$. Then $A$ has a complement in $G$.

The importance of this result and its relevance to other known theorems will
be further elucidated in due course. A sort of partial converse of Theorem 1.2
holds and its proof is given in Section 2:

######  Theorem 1.3.

Let $H$ be an arbitrary subgroup of the $p$-group $G$ and assume that $H$ has
a normal complement in $G$. Then

$H\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(H)$

for all indices $i\geqslant 0$.

###  1.1. Notation

We outline below some notational conventions that we will use throughout the
paper.

  1. (i)

$[n]$ denotes the set of the first $n$ positive integers $\\{1,2,\ldots,n\\}$.

  2. (ii)

Let $G$ be a $p$-group and let $i$ be a non-negative integer. We will write

$\mho_{i}(G)=\big{\langle}g^{p^{i}}:g\in G\big{\rangle}$

for the subgroup generated by the $p^{i}$-powers of elements of $G$. Clearly,
$\mho_{0}(G)=G$. The index $i=1$ is sometimes dropped, so that
$\mho_{1}(G)=\mho(G)$. These subgroups, for various indices $i$, are referred
to as the agemo subgroups of $G$. We remark that $G^{p^{i}}$ is sometimes used
instead of $\mho_{i}(G)$ in the literature.

  3. (iii)

We will have occasion to refer to the Frattini series of a group. For an
arbitrary group, the Frattini series is defined by $\Phi_{0}(G)=G$ and
$\Phi_{i}(G)=\Phi(\Phi_{i-1}(G))$, where $\Phi(G)$ is the Frattini subgroup of
$G$, i.e. the intersection of all maximal subgroups of $G$.

##  2\. Properties of the $\operatorname{\mathcal{X}}$-series

We begin in earnest by defining the $\operatorname{\mathcal{X}}$-series of a
$p$-group.

######  Definition 2.1.

Given a $p$-group $G$ and a positive integer $i\leqslant\log_{p}|G|$ we write

${\mathcal{P}}_{i}={\mathcal{P}}_{i}(G)\coloneqq\left\\{T\leqslant
G\,:\,|G:T|=p^{i}\right\\}$

for the collection of subgroups of $G$ of index $p^{i}$ in $G$. We also write

$\operatorname{\mathcal{X}}_{i}(G)\coloneqq\bigcap_{T\in{\mathcal{P}}_{i}(G)}T\,;$

that is, $\operatorname{\mathcal{X}}_{i}(G)$ is the intersection of all
subgroups of $G$ that have index $p^{i}$ in $G$ for all such $i$. We call the
associated series the $\operatorname{\mathcal{X}}$-series of $G$ and by
convention we write $\operatorname{\mathcal{X}}_{0}(G)=G$. Finally, if
$j>\log_{p}|G|$ then $\operatorname{\mathcal{X}}_{j}(G)=1$ by assumption.

Some first properties following the definition of the
$\operatorname{\mathcal{X}}$-series are collected in the following lemma.

######  Lemma 2.2.

If $G$ is a $p$-group then the subgroups $\operatorname{\mathcal{X}}_{i}(G)$
are characteristic subgroups of $G$ and

  1. (i)

$\operatorname{\mathcal{X}}_{1}(G)=\Phi(G)$ ;

  2. (ii)

$\operatorname{\mathcal{X}}_{k}(G)=\bigcap_{T\in{\mathcal{P}}_{k-j}(G)}\operatorname{\mathcal{X}}_{j}(T)$
for every $j\in[k]$ ;

  3. (iii)

$\operatorname{\mathcal{X}}_{i+1}(G)\leqslant\operatorname{\mathcal{X}}_{i}(G)$
;

  4. (iv)

$\Phi_{i}(G)\leqslant T$, for every $T\in{\mathcal{P}}_{i}(G)$ ;

  5. (v)

$\Phi_{i}(G)\leqslant\operatorname{\mathcal{X}}_{i}(G)$ for every $i$ ;

  6. (vi)

If $N\unlhd G$ and $N\leqslant\operatorname{\mathcal{X}}_{i}(G)$ for some
index $i$, then
$\operatorname{\mathcal{X}}_{i}(G/N)=\operatorname{\mathcal{X}}_{i}(G)N/N$.

Proof.  (i) Clearly the $\operatorname{\mathcal{X}}_{i}(G)$ are characteristic
subgroups of $G$, for all $i$, and

$\operatorname{\mathcal{X}}_{1}(G)=\bigcap_{T\in{\mathcal{P}}_{1}(G)}T=\Phi(G),$

as ${\mathcal{P}}_{1}(G)$ is exactly the set of maximal subgroups of $G$.

(ii) To see this observe that for every $j\in[k]$ we have

${\mathcal{P}}_{k}(G)=\bigcup_{T\in{\mathcal{P}}_{k-j}(G)}{\mathcal{P}}_{j}(T),$

due to the fact that for every subgroup $S$ of index $p^{k}$ in a $p$-group
$G$ there exists a subgroup $T\leqslant G$ of index $p^{k-j}$ that contains
$S$.

(iii) Now this part follows from (ii) as

$\operatorname{\mathcal{X}}_{i+1}(G)=\bigcap_{T\in{\mathcal{P}}_{i}(G)}\operatorname{\mathcal{X}}_{1}(T)\leqslant\bigcap_{T\in{\mathcal{P}}_{i}(G)}T=\operatorname{\mathcal{X}}_{i}(G)\,.$

(iv) We induce on $i$. For $i=1$ we clearly have $\Phi(G)\leqslant M$ for
every maximal subgroup $M$ of $G$. Assuming it holds for $i$, we will prove it
for $i+1$. Let $T\in{\mathcal{P}}_{i+1}(G)$ and pick
$K\in{\mathcal{P}}_{i}(G)$ with $T\leqslant K$. So $T$ is a maximal subgroup
of $K$ and $\Phi(K)\leqslant T$. Furthermore, the inductive hypothesis implies
that $\Phi_{i}(G)\leqslant K$ and therefore

$\Phi_{i+1}(G)=\Phi(\Phi_{i}(G))\leqslant\Phi(K)\leqslant T.$

(v) The proof of this part follows directly from (iv) and the definition of
$\operatorname{\mathcal{X}}_{i}(G)$.

(vi) Since $N\leqslant\operatorname{\mathcal{X}}_{i}(G)$, it follows from the
correspondence theorem that

${\mathcal{P}}_{i}(G/N)=\\{H/N:H\in{\mathcal{P}}_{i}(G)\\};$

this clearly proves the claim, and the proof of the lemma is complete.

$\blacksquare$

######  Remark 2.3.

Firstly, observe that equality really is possible in (iii), since in $Q_{8}$,
for example, we have
$\operatorname{\mathcal{X}}_{1}(Q_{8})=\operatorname{\mathcal{X}}_{2}(Q_{8})$.
Secondly, we comment on the behaviour of the
$\operatorname{\mathcal{X}}$-series under quotients. Assuming that $G$ is a
$p$-group, as usual, and that $N$ is an arbitrary normal subgroup of $G$, we
have
$\operatorname{\mathcal{X}}_{1}(G/N)=\operatorname{\mathcal{X}}_{1}(G)N\big{/}N$
since $\operatorname{\mathcal{X}}_{1}(G)=\Phi(G)=G^{\prime}\mho(G)$ and each
of $G^{\prime}$, $\mho(G)$ behaves well under quotients. However, things
change for larger-index terms of the $\operatorname{\mathcal{X}}$-series. It
is not true, for instance, that
$\operatorname{\mathcal{X}}_{2}(G/N)=\operatorname{\mathcal{X}}_{2}(G)N\big{/}N$
for arbitrary $G$, $N$. An example where this fails to be true is the group of
order $16$ with presentation

$\mathcal{G}=C_{4}\rtimes C_{4}=\left\langle
a,b\,\big{|}\,a^{4}=b^{4}=1,a^{b}=a^{-1}\right\rangle.$

The subgroup $N=\langle a^{2}b^{2}\rangle$ is central of order $2$ and affords
the quotient $G/N\cong Q_{8}$. To see this, observe that $G/N$ is non-abelian
(since, for example, $abN\neq baN$) and $a^{2}N=b^{2}N$ is the unique element
of order $2$ in $G/N$. Now $\operatorname{\mathcal{X}}_{2}(G)$ is trivial (see
Figure 1) and thus $\operatorname{\mathcal{X}}_{2}(G)N\big{/}N$ is trivial,
but $\operatorname{\mathcal{X}}_{2}(G/N)$ is not.

$C_{1}$$C_{2}$$C_{2}$$C_{2}=N$$C_{2}^{2}$$C_{4}$$C_{4}$$C_{4}$$C_{4}$$C_{2}{\times}C_{4}$$C_{2}{\times}C_{4}$$C_{2}{\times}C_{4}$$C_{4}{\rtimes}C_{4}$22
Figure 1. The poset of classes of subgroups of $\mathcal{G}$.

Next, we prove a useful property of the $\operatorname{\mathcal{X}}$-series.

######  Proposition 2.4.

Let $G$ be a $p$-group and let $M\leqslant G$. Then
$\operatorname{\mathcal{X}}_{i}(M)\leqslant\operatorname{\mathcal{X}}_{i}(G)$.

Proof.  Fix a subgroup $M\leqslant G$ and an index $i\geqslant 0$. To prove
the desired containment, it suffices to show that
$\operatorname{\mathcal{X}}_{i}(M)\leqslant K$ for every
$K\in{\mathcal{P}}_{i}(G)$. The equality $|MK|\big{/}|K|=|M|\big{/}|M\cap K|$
and the fact that $K$ has index $p^{i}$ in $G$ together imply that the index
of $K\cap M$ in $M$ is at most $p^{i}$. Thus there exists a subgroup $T$ of
$K\cap M$ whose index in $M$ is exactly $p^{i}$. Therefore

$\operatorname{\mathcal{X}}_{i}(M)\leqslant T\leqslant K\cap M\leqslant K.$

As this holds for every $K\in{\mathcal{P}}_{i}(G)$, the proposition follows.
$\blacksquare$

We will refer to this property of the $\operatorname{\mathcal{X}}$-series as
the subgroup-monotonicity of $\operatorname{\mathcal{X}}$. Moreover, the same
will be said for a subgroup series that satisfies the conclusion of
Proposition 2.4 (i.e. that it is subgroup-monotone).

######  Lemma 2.5.

Let $G$ be a $p$-group. Then

$\operatorname{\mathcal{X}}_{i}(\operatorname{\mathcal{X}}_{j}(G))\leqslant\operatorname{\mathcal{X}}_{i+j}(G).$

In addition,
$\operatorname{\mathcal{X}}_{i}(G)/\operatorname{\mathcal{X}}_{i+1}(G)$ is an
elementary abelian $p$-group for every non-negative integer $i$.

Proof.  In view of Lemma 2.2 we have
$\operatorname{\mathcal{X}}_{i+j}(G)=\bigcap_{T\in{\mathcal{P}}_{j}(G)}\operatorname{\mathcal{X}}_{i}(T)$.
Let

$K\coloneqq\bigcap_{T\in{\mathcal{P}}_{j}(G)}T=\operatorname{\mathcal{X}}_{j}(G).$

Then
$\operatorname{\mathcal{X}}_{i}(K)\leqslant\operatorname{\mathcal{X}}_{i}(T)$
for every $T\in{\mathcal{P}}_{j}(G)$ by Proposition 2.4. Hence

$\operatorname{\mathcal{X}}_{i+j}(G)=\bigcap_{T\in{\mathcal{P}}_{j}(G)}\operatorname{\mathcal{X}}_{i}(T)\,\geqslant\,\operatorname{\mathcal{X}}_{i}(K)=\operatorname{\mathcal{X}}_{i}(\operatorname{\mathcal{X}}_{j}(G)).$

Thus, the first part of the lemma holds, from which we deduce that
$\operatorname{\mathcal{X}}_{i}(G)/\operatorname{\mathcal{X}}_{i+1}(G)$ is an
elementary abelian group seeing as

$\Phi(\operatorname{\mathcal{X}}_{i}(G))=\operatorname{\mathcal{X}}_{1}(\operatorname{\mathcal{X}}_{i}(G))\leqslant\operatorname{\mathcal{X}}_{i+1}(G).$

This completes the induction and the proof of the lemma. $\blacksquare$

We now look at the situation where some normal subgroup $N$ of $G$ is
complemented in $G$. Theorem 1.3 is essentially the second part of the
following:

######  Theorem 2.6.

Let $G$ be a $p$-group and let $N$ be a normal subgroup of $G$. Suppose that
$N$ is complemented, by $H$ say, in $G$. Then for all $i\geqslant 0$ we have:

  1. (i)

$\operatorname{\mathcal{X}}_{i}(N)\operatorname{\mathcal{X}}_{i}(H)\leqslant\operatorname{\mathcal{X}}_{i}(G)\leqslant
N\operatorname{\mathcal{X}}_{i}(H)$;

  2. (ii)

$\operatorname{\mathcal{X}}_{i}(H)=\operatorname{\mathcal{X}}_{i}(G)\cap H$.

Proof.  (i) The leftmost inclusion is a consequence of the fact that the
product is well-defined (since $\operatorname{\mathcal{X}}_{i}(N)$ is
characteristic in $N$ and the latter is normal in $G$) and the subgroup-
monotonicity of the $\operatorname{\mathcal{X}}$-series.

As regards the right-hand-side inclusion, assume first that
$i\leqslant\log_{p}(|H|)$. Observe that for all $T,S\in{\mathcal{P}}_{i}(H)$
Dedekind’s lemma yields

$NS\cap NT=N(NS\cap T)$ (2.1)

and

$NS\cap T=NS\cap H\cap T=S(N\cap H)\cap T=S\cap T.$ (2.2)

By (2.1) and (2.2) we get $NS\cap NT=N(S\cap T)$ and repeatedly applying this
fact gives us

$\bigcap_{T\in{\mathcal{P}}_{i}(H)}NT=N\bigcap_{T\in{\mathcal{P}}_{i}(H)}T=N\operatorname{\mathcal{X}}_{i}(H).$

But if $T\in{\mathcal{P}}_{i}(H)$ then $NT\in{\mathcal{P}}_{i}(G)$, thus

$\operatorname{\mathcal{X}}_{i}(G)=\bigcap_{S\in{\mathcal{P}}_{i}(G)}S\leqslant\bigcap_{T\in{\mathcal{P}}_{i}(H)}NT=N\operatorname{\mathcal{X}}_{i}(H).$

In particular, for $i_{0}=\log_{p}(|H|)$ we get
$\operatorname{\mathcal{X}}_{i_{0}}(G)\leqslant
N\operatorname{\mathcal{X}}_{i_{0}}(H)=N$. Hence for $i\geqslant i_{0}$ we see
that
$\operatorname{\mathcal{X}}_{i}(G)\leqslant\operatorname{\mathcal{X}}_{i_{0}}(G)\leqslant
N$. Therefore the inclusion $\operatorname{\mathcal{X}}_{i}(G)\leqslant
N\operatorname{\mathcal{X}}_{i}(H)$ is valid for all non-negative integers
$i$, completing the proof.

(ii) We have that

$\operatorname{\mathcal{X}}_{i}(H)\leqslant
H\cap\operatorname{\mathcal{X}}_{i}(G)\leqslant H\cap
N\operatorname{\mathcal{X}}_{i}(H)=\operatorname{\mathcal{X}}_{i}(H)(N\cap
H)=\operatorname{\mathcal{X}}_{i}(H),$

where the first inclusion follows from the subgroup-monotonicity of the
$\operatorname{\mathcal{X}}$-series, the second inclusion from part (i) and
the penultimate equality by Dedekind’s lemma. This double inclusion forces
$\operatorname{\mathcal{X}}_{i}(H)=\operatorname{\mathcal{X}}_{i}(G)\cap H$,
as desired. $\blacksquare$

######  Corollary 2.7.

Let $G$ be a $p$-group and let $N$ be a normal subgroup of $G$ that is
complemented in $G$ by $H$, say. Then $N\cap\operatorname{\mathcal{X}}_{i}(G)$
has a complement in $\operatorname{\mathcal{X}}_{i}(G)$ for all $i\geqslant 0$
and, in fact, $\operatorname{\mathcal{X}}_{i}(H)$ is one such complement.

Proof.  Fix an index $i\geqslant 0$. We show that
$\operatorname{\mathcal{X}}_{i}(H)$ is a complement for
$N\cap\operatorname{\mathcal{X}}_{i}(G)$ in
$\operatorname{\mathcal{X}}_{i}(G)$ by observing the following:

$(N\cap\operatorname{\mathcal{X}}_{i}(G))\operatorname{\mathcal{X}}_{i}(H)=\operatorname{\mathcal{X}}_{i}(G)\cap
N\operatorname{\mathcal{X}}_{i}(H)=\operatorname{\mathcal{X}}_{i}(G).$

The first equality is yet another application of Dedekind’s lemma and the
second equality follows directly from Theorem 2.6 (i). The proof is complete.
$\blacksquare$

Theorem 2.6 will also aid us in the proof of the next result.

######  Theorem 2.8.

Given $p$-groups $G$ and $H$, we have $\operatorname{\mathcal{X}}_{i}(G\times
H)=\operatorname{\mathcal{X}}_{i}(G)\times\operatorname{\mathcal{X}}_{i}(H)$
for all non-negative integers $i$.

Proof.  Fix an index $i$. Proposition 2.4 clearly implies that
$\operatorname{\mathcal{X}}_{i}(G)\times
1=\operatorname{\mathcal{X}}_{i}(G\times
1)\leqslant\operatorname{\mathcal{X}}_{i}(G\times H)$ and similarly for $H$.
Hence

$\operatorname{\mathcal{X}}_{i}(G)\times\operatorname{\mathcal{X}}_{i}(H)\leqslant\operatorname{\mathcal{X}}_{i}(G\times
H).$

For the other direction, note that by Theorem 2.6 we have

$\operatorname{\mathcal{X}}_{i}(G\times H)\leqslant
G\operatorname{\mathcal{X}}_{i}(H)$ (2.3)

and

$\operatorname{\mathcal{X}}_{i}(G\times
H)\leqslant\operatorname{\mathcal{X}}_{i}(G)H.$ (2.4)

Combining (2.3) and (2.4) gives us

$\operatorname{\mathcal{X}}_{i}(G\times H)\leqslant
G\operatorname{\mathcal{X}}_{i}(H)\cap\operatorname{\mathcal{X}}_{i}(G)H=\operatorname{\mathcal{X}}_{i}(G)\left(G\operatorname{\mathcal{X}}_{i}(H)\cap
H\right)=\\\
=\operatorname{\mathcal{X}}_{i}(G)\left[\operatorname{\mathcal{X}}_{i}(H)(G\cap
H)\right]=\operatorname{\mathcal{X}}_{i}(G)\times\operatorname{\mathcal{X}}_{i}(H),$

where all equalities are applications of Dedekind’s lemma. Since both
inclusions are valid, the proof of the theorem is complete. $\blacksquare$

######  Proposition 2.9.

Assume that $G$ is an abelian $p$-group. Then
$\operatorname{\mathcal{X}}_{i}(G)=\Phi_{i}(G)=\mho_{i}(G)$ and
$\operatorname{\mathcal{X}}_{j}(\operatorname{\mathcal{X}}_{i}(G))=\operatorname{\mathcal{X}}_{i+j}(G)$,
for all integers $i,j\geqslant 1$.

Proof.  Assume first that $G$ is a cyclic group. Then
${\mathcal{P}}_{i}(G)=\mho_{i}(G)$ as $\mho_{i}(G)$ is the unique subgroup of
$G$ of index $p^{i}$ in $G$ and thus
$\operatorname{\mathcal{X}}_{i}(G)=\mho_{i}(G)=\Phi_{i}(G)$.

In the general case, let $G=C_{i}\times C_{2}\times\cdots\times C_{t}$, where
$C_{j}$ are cyclic subgroups of $G$. Then in view of Theorem 2.8 we get
$\operatorname{\mathcal{X}}_{i}(G)=\prod_{j=1}^{t}\operatorname{\mathcal{X}}_{i}(C_{j})$.
But for any cyclic subgroup $C$ we have
$\operatorname{\mathcal{X}}_{i}(C)=\mho_{i}(C)=\Phi_{i}(C)$. Hence

$\operatorname{\mathcal{X}}_{i}(G)=\prod_{j=1}^{t}\operatorname{\mathcal{X}}_{i}(C_{j})=\prod_{j=1}^{t}\mho_{i}(C_{j})=\prod_{j=1}^{t}\Phi_{i}(C_{j})=\Phi_{i}(G)=\mho_{i}(G),$

where the penultimate equality follows from the fact that the Frattini
subgroup respects direct products and an easy induction argument (see [Gas53,
Satz 6]).

The last part of the proposition follows directly from the corresponding
relations that the agemo subgroups of an abelian group111In fact, this
relation among the agemo subgroups is satisfied more generally by regular
$p$-groups; see Lemma 1.2.12 (ii) in [LGM02], for example. satisfy:

$\mho_{i}(\mho_{j}(G))=\mho_{i+j}(G).$

The proof of the proposition is now complete. $\blacksquare$

######  Proposition 2.10.

Let $C$ be a cyclic subgroup of $G$ of order $p^{n}$. Then the following are
equivalent:

  1. (i)

$C\cap\operatorname{\mathcal{X}}_{n}(G)=1$;

  2. (ii)

$C$ has a complement in $G$;

  3. (iii)

$C\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(C)$ for
every non-negative integer $i$.

Proof.  (i) $\to$ (ii) We assume that
$C\cap\operatorname{\mathcal{X}}_{n}(G)=1$ and wish to establish the existence
of a complement for $C$ in $G$. Indeed, if $C\cap T>1$ for all subgroups $T$
of index $p^{n}$ in $G$, then $C\cap T\geqslant\Omega_{1}(C)>1$, as
$\Omega_{1}(G)$ is the unique subgroup of order $p$ in $C$. Therefore
$C\cap\operatorname{\mathcal{X}}_{n}(G)\geqslant\Omega_{1}(C)>1$, and this is
clearly a contradiction. It follows that there exists at least one subgroup
$T$ of index $p^{n}$ in $G$ whose intersection with $C$ is trivial. That
subgroup $T$ serves as the desired complement.

(ii) $\to$ (iii) Let $T$ be, as in the previous paragraph, a complement for
$C$ in $G$. As $C$ is a cyclic group, we see that
$\operatorname{\mathcal{X}}_{i}(C)=\mho_{i}(C)$ is the unique subgroup of
order $p^{n-i}$ in $C$. Now we pick a subgroup series of $G$

$T=K_{n}<K_{n-1}<\ldots<K_{1}<G=K_{0}$

with $|K_{i}/K_{i+1}|=p$ and we claim that
$K_{i}=T\operatorname{\mathcal{X}}_{i}(C)$ and $C\cap
K_{i}=\operatorname{\mathcal{X}}_{i}(C)$. We clearly have $K_{i}=T(C\cap
K_{i})$, while

$|C\cap K_{i}|=\frac{|K_{i}|}{|T|}=\frac{|G|}{p^{i}}|T|=p^{n-i}.$

Since $\operatorname{\mathcal{X}}_{i}(C)$ is the unique subgroup of $C$ of
order $p^{n-i}$, we deduce that $C\cap
K_{i}=\operatorname{\mathcal{X}}_{i}(C)$, and the claim follows.

Now, $K_{i}$ has index $p^{i}$ in $G$ and therefore
$\operatorname{\mathcal{X}}_{i}(G)\leqslant K_{i}$. We conclude that

$C\cap\operatorname{\mathcal{X}}_{i}(G)\leqslant C\cap
K_{i}=\operatorname{\mathcal{X}}_{i}(C)\leqslant
C\cap\operatorname{\mathcal{X}}_{i}(G),$

and thus $\operatorname{\mathcal{X}}_{i}(G)\cap
C=\operatorname{\mathcal{X}}_{i}(C)$, as desired.

(iii) $\to$ (i) This follows easily from the fact that
$C\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(C)$ for
all $i\geqslant 0$ upon taking $i=n$ and observing that
$\operatorname{\mathcal{X}}_{n}(C)=1$. $\blacksquare$

Our next result outlines which members of the lower central series of $G$ are
guaranteed to be contained in some term of the
$\operatorname{\mathcal{X}}$-series of $G$. As we will see, our result is
essentially best possible.

######  Proposition 2.11.

If $G$ is a $p$-group and $i$ is a positive integer then

$\gamma_{p^{i-1}+1}(G)\leqslant\operatorname{\mathcal{X}}_{i}(G)$

and the index $p^{i-1}+1$ is optimal.

Proof.  Let $H$ be an arbitrary subgroup of $G$ of index $p^{i}$ in $G$. The
regular action of $G$ on the left cosets of $H$ in $G$ gives rise to a
homomorphism $\phi:G\to S_{p^{i}}$ whose kernel $N$ is the core of $H$ in $G$;
that is

$N=\bigcap_{g\in G}H^{g}\leqslant H.$

Let $L_{i}$ be a Sylow $p$-subgroup of $S_{p^{i}}$. Then $G/N$ is isomorphic
to a subgroup of $L_{i}$. Now $L_{i}$ is an iterated wreath product

$\underbrace{C_{p}\wr\ldots\wr C_{p}}_{\text{$i$ terms}}$

having order $p^{\frac{p^{i}-1}{p-1}}$ and class $c=p^{i-1}$ (cf. [Kal46]).
Hence $\gamma_{p^{i-1}+1}(L_{i})=1$, so $\gamma_{p^{i-1}+1}(G/N)=1$. But the
lower central series behaves well with respect to quotients and therefore

$1=\gamma_{p^{i-1}+1}(G/N)=\gamma_{p^{i-1}+1}(G)N/N.$

We conclude that $\gamma_{p^{i-1}+1}(G)\leqslant N\leqslant H$ and as $H$ was
arbitrary we see that

$\gamma_{p^{i-1}+1}(G)\leqslant\bigcap_{H\in{\mathcal{P}}_{i}(G)}H=\operatorname{\mathcal{X}}_{i}(G).$

To see that the index $p^{i-1}+1$ is optimal, observe that
$\gamma_{p^{i-1}+1}(L_{i})=1$ and the index $j=p^{i-1}+1$ is the least
positive integer such that $\gamma_{j}(L_{i})=1$ (since $L_{i}$ has class
$p^{i-1}$). In proof of the second claim of the proposition therefore, it will
be sufficient to establish that $\operatorname{\mathcal{X}}_{i}(L_{i})=1$ for
all positive integers $i$. We argue by induction on $i$, noting that the base
case $i=1$ is clearly valid since $L_{1}$ has prime order $p$ and thus its
Frattini subgroup is trivial. Assume validity of the claim for $i-1$. Note
that $L_{i}=L_{i-1}\wr C_{p}$ and that the base of $L_{i}$

$\underbrace{B=L_{i-1}\times\ldots\times L_{i-1}}_{\text{$p$ factors}}$

is a maximal subgroup of $L_{i}$.

On the other hand, we have by Lemma 2.2 (ii) (with $k=i$ and $j=i-1$)

$\operatorname{\mathcal{X}}_{i}(L_{i})=\bigcap_{M\in{\mathcal{P}}_{1}(L_{i})}\operatorname{\mathcal{X}}_{i-1}(M)\leqslant\operatorname{\mathcal{X}}_{i-1}(B).$

But

$\operatorname{\mathcal{X}}_{i-1}(B)=\operatorname{\mathcal{X}}_{i-1}(L_{i-1})\times\ldots\times\operatorname{\mathcal{X}}_{i-1}(L_{i-1})=1,$

where the first equality follows from Theorem 2.8 while the second equality is
valid by the inductive hypothesis. Thus,
$\operatorname{\mathcal{X}}_{i}(L_{i})=1$, completing the induction and the
proof. $\blacksquare$

######  Corollary 2.12.

Let $G$ be a $2$-group. Then
$[G,\Phi(G)]\leqslant\operatorname{\mathcal{X}}_{2}(G)$.

Proof.  We begin with the observation that

$[G,\Phi(G)]=[G,G^{\prime}\mho(G)]=[G,G^{\prime}][G,\mho(G)]=\gamma_{3}(G)[G,\mho(G)].$

We assert that both terms in the right-hand-side of the equality above are
contained in $\operatorname{\mathcal{X}}_{2}(G)$. That
$\gamma_{3}(G)\leqslant\operatorname{\mathcal{X}}_{2}(G)$ follows from
Proposition 2.11 with $p=i=2$. Thus, it will suffice to prove that
$[G,\mho(G)]\leqslant\operatorname{\mathcal{X}}_{2}(G)$. According to
Corollary 1.28 in [LGM02], we have

$[G,\mho(G)]\leqslant\mho([G,G])[G,G,G]=\mho(G^{\prime})\gamma_{3}(G)\leqslant\mho(G^{\prime})\operatorname{\mathcal{X}}_{2}(G)$

when $G$ is a $2$-group. In addition,

$\mho(G^{\prime})\leqslant\mho(\Phi(G))\leqslant\Phi(\Phi(G))=\Phi_{2}(G)\leqslant\operatorname{\mathcal{X}}_{2}(G),$

where the first containment is justified by the subgroup-monotonicity of the
agemo subgroup (cf. Lemma 1.2.7 in [LGM02]) and the last by Lemma 2.2 (v).
Hence the claim has been proved. $\blacksquare$

The content of Corollary 2.12 is no longer valid if $p>2$. Indeed, for any odd
prime $p$ the Sylow $p$-subgroup $C_{p}\wr C_{p}$ of $S_{p^{2}}$ satisfies
$\operatorname{\mathcal{X}}_{2}(C_{p}\wr C_{p})=1$ since it has a maximal
subgroup which is elementary abelian (in particular, its base), but its
Frattini subgroup is not central. To see why this last claim is true, assume
that $\Phi(G)\leqslant Z(G)$ and recall that by [Kal46] the class of $C_{p}\wr
C_{p}$ is $p$. Thus, if $\Phi(G)\leqslant Z(G)$ then $\Phi(G)=Z(G)$, since
$Z(G)$ has order $p$. On the other hand $G^{\prime}\leqslant\Phi(G)=Z(G)$,
forcing $\gamma_{3}(G)$ to be trivial, which is absurd (even for $p=3$).

##  3\. Complements and abelian subgroups

If $A$ is an abelian group then a condition which is both necessary and
sufficient for a subgroup $B$ of $A$ to have a complement in $A$ exists and is
based on Prüfer’s notion of a pure subgroup; see for example [Fuc15, Chap. 5].
We define this next.

######  Definition 3.1.

Let $A$ be an abelian group and let $B$ be a subgroup of $A$. Then $B$ is
called a pure subgroup of $A$ provided that if $a^{n}\in B$ for $a\in A$ and
$n\in\mathbb{N}$ then there exists $b\in B$ such that $a^{n}=b^{n}$.
Equivalently, $B$ is pure in $A$ if and only if $B^{n}=B\cap A^{n}$ for all
$n\in\mathbb{N}$.

For the convenience of the reader we present here an independent proof of the
following result. Note that no originality is claimed here. The result itself
should be well-known to abelian group theorists (and, in fact, holds more
generally than we have stated), but perhaps not so much to group theorists in
general.

######  Theorem 3.2.

Let $A$ be a abelian group and $B\leqslant A$. Then $B$ has a complement in
$A$ if and only if $B$ is a pure subgroup of $A$.

Proof.  Suppose first that $B$ is complemented in $A$ and let $C$ be a
complement. Then $A$ is the (internal) direct product $A=B\times C$. Thus, for
each $a\in A$ there exist unique $b\in B$ and $c\in C$ such that $a=bc$.
Therefore, if $a^{n}\in B$ for some $n\in\mathbb{N}$ then $b^{n}c^{n}\in B$
thus $c^{n}\in B\cap C=1$. So $a^{n}=b^{n}$ and thus $B$ is pure in $A$.

For the converse, suppose that $B$ is pure in $A$ and that $B\leqslant
C\leqslant A$, where $C/B$ is cyclic. Let $n=|C:B|$ and write $\langle
cB\rangle=C/B$. As $c^{n}\in B$, there exists $b\in B$ such that
$c^{n}=b^{n}$. Setting $g\coloneqq cb^{-1}$, we have $g^{n}=1$. Since $gB=cB$,
we see that $n=o(gB)$ in $C/B$. It follows that $C=B\times\langle g\rangle$
and thus $B$ is a direct factor of $C$.

Now for the general case, write $A/B$ as a direct product of cyclic groups
$C_{i}/B$. Let $X_{i}$ be a complement for $B$ in $C_{i}$ and note that
$|X_{i}|=|C_{i}:B|$. Write $D=\prod_{i}X_{i}$.

We have $BD\geqslant BX_{i}=C_{i}$ for all $i$, and thus
$BD=\prod_{i}C_{i}=A$. Also,

$|D|\leqslant\prod_{i}|X_{i}|=\prod_{i}|C_{i}:B|=|A/B|,$

and thus $A=B\times D$. Our proof is complete. $\blacksquare$

Now we focus our attention on $p$-groups. In view of Gaschütz’s theorem
mentioned in Section 1, this is natural. In this setting, the previous theorem
asserts the following:

Let $G$ be an abelian $p$-group and let $A$ be a subgroup of $G$. Then $A$ has
a complement in $G$ if and only if

$A^{n}=A\cap G^{n}$

for all $n\in\mathbb{N}$.

However, the only relevant exponents $n$ in this case are the powers of $p$,
thus $(\dagger)$ is essentially equivalent to

$\mho_{i}(A)=A\cap\mho_{i}(G)$

for all non-negative integers $i$. On the other hand, the $\mho_{i}$ subgroup
of an abelian $p$-group is its $\operatorname{\mathcal{X}}_{i}$ subgroup,
according to Proposition 2.9. Thus we have the following.

######  Corollary 3.3.

Let $G$ be a abelian $p$-group and let $A$ be a subgroup of $G$. Then $A$ has
a complement in $G$ if and only if

$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$

for all indices $i\geqslant 0$.

We are now ready to prove that the condition above, which is both necessary
and sufficient for complementation of a subgroup of an abelian $p$-group, is
sufficient in general.

Proof of Theorem 1.2. We begin with an easy observation; it will suffice to
establish the claim for all indices $i$ such that $p^{i}$ is at most the
exponent of the subgroup $A$. Moreover, notice that if $A$ is cyclic, then the
equivalence between (ii) and (iii) in Proposition 2.10 ensures that $A$ is
complemented in $G$.

Now we argue by induction on $|A|$, noting that we have addressed the base
case of the induction in the previous paragraph. We may also assume that $A$
is not cyclic. So let $C$ be a cyclic subgroup of maximal order in $A$ and
note that $|C|=\mathrm{exp}(A)=p^{r}$, say. By assumption,
$A\cap\operatorname{\mathcal{X}}_{r}(G)=\operatorname{\mathcal{X}}_{r}(A)=1$.
Then $C\cap\operatorname{\mathcal{X}}_{r}(G)\leqslant
A\cap\operatorname{\mathcal{X}}_{r}(G)=1$, thus
$C\cap\operatorname{\mathcal{X}}_{r}(G)=1$ and the equivalence between (i) and
(ii) in Proposition 2.10 suffices to guarantee that $C$ has a complement, say
$H$, in $G$.

By Dedekind’s lemma, $A=C\times B$, where $B\coloneqq A\cap H$. Observe that
since $B$ is complemented by $C$ in $A$ and taking into account Theorem 1.3,
we have
$B\cap\operatorname{\mathcal{X}}_{i}(A)=\operatorname{\mathcal{X}}_{i}(B)$ for
all $i\leqslant t$, where $p^{t}=\mathrm{exp}(B)$.

Since the $\operatorname{\mathcal{X}}$-series is subgroup-monotone, we have
for each $i\leqslant t$ the following:

$B\cap\operatorname{\mathcal{X}}_{i}(H)\leqslant
B\cap\operatorname{\mathcal{X}}_{i}(G)=B\cap(A\cap\operatorname{\mathcal{X}}_{i}(G))=B\cap\operatorname{\mathcal{X}}_{i}(A)=\operatorname{\mathcal{X}}_{i}(B).$

Also, again by subgroup-monotonicity we have
$\operatorname{\mathcal{X}}_{i}(B)\leqslant B$ and
$\operatorname{\mathcal{X}}_{i}(B)\leqslant\operatorname{\mathcal{X}}_{i}(H)$
for all appropriate indices $i$. Thus,
$\operatorname{\mathcal{X}}_{i}(B)\leqslant
B\cap\operatorname{\mathcal{X}}_{i}(H)$ and since the reverse containment also
holds we have
$\operatorname{\mathcal{X}}_{i}(B)=B\cap\operatorname{\mathcal{X}}_{i}(H)$.
The induction hypothesis applied to $B$ relative to the subgroup $H$ now
ensures that there exists a subgroup $D$ in $H$ which complements $B$. Then

$A\cap D=(A\cap H)\cap D=B\cap D=1.$

Moreover,

$|A|=|C||B|=|G:H||H:D|=|G:D|.$

We conclude that $D$ complements $A$ in $G$ thus our induction is complete.
$\blacksquare$

We follow up with a few remarks on the content and proof of Theorem 1.2.

######  Remark 3.4.

It should be noted that Theorem 1.2 works for a class of $p$-groups which is
strictly larger than the class of abelian $p$-groups. For $A$ to have a
complement in $G$ it is sufficient, as the proof of the theorem indicates,
that $A$ have the following property:

> for every $H\leqslant A$ there exists a normal (in $H$) cyclic subgroup
> $1<C\unlhd H$ that admits a complement in $H$,

For example, the property above is enjoyed by the dihedral $2$-groups
$D_{2^{n}}$ which are non-abelian for $n\geqslant 3$.

On the other hand, Theorem 1.2 is not true unconditionally; some assumption
about $A$ _must_ be made. Take $\mathcal{H}=C_{4}\times\mathcal{G}$, where
$\mathcal{G}$ is as in Remark 2.3. Then there exists a maximal subgroup $M$ of
$\mathcal{H}$ which satisfies the condition of Theorem 1.2, i.e.
$\Phi(M)=\Phi(\mathcal{H})$, yet $M$ has no complement in $\mathcal{H}$ since
$\Omega_{1}(\mathcal{H})=\Phi(\mathcal{H})$.

######  Remark 3.5.

On the surface, it looks as though a literal interpretation of purity might be
enough to guarantee the existence of a complement in Theorem 1.2. That is to
say, under the assumptions of the theorem, perhaps it is true that an abelian
subgroup $A$ of $G$ satisfying $A\cap\mho_{i}(G)=\mho_{i}(A)$ for all
$i\geqslant 0$ is necessarily complemented in $G$. However, this is not so;
purity _really must_ be stated in an equivalent form. To see why, consider an
odd prime $p$ and let $G$ be a non-abelian $p$-group of exponent $p$. Then it
is easy to see that the equality $A\cap\mho_{i}(G)=\mho_{i}(A)$ is valid for
every (abelian) subgroup of $G$, but clearly not every abelian subgroup can be
complemented; simply consider a subgroup of
$\operatorname{\mathcal{X}}_{1}(G)=\Phi(G)$ of order $p$.

######  Corollary 3.6.

Let $G$ be a $p$-group and let $A$ be a central subgroup of $G$. Then $A$ has
a complement in $G$ if and only if
$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$ for
all $i\geqslant 0$.

Proof.  Since $A$ is central in $G$, it is abelian. Thus Theorem 1.2 implies
that $A$ is complemented in $G$. Conversely, suppose that $A$ has a complement
in $G$, say $H$. Then $H$ is normalised by both $A$ and $H$, thus $G=A\times
H$ and $A$ is, in fact, normally complemented in $G$. Now Theorem 1.3 implies
that
$\operatorname{\mathcal{X}}_{i}(A)=A\cap\operatorname{\mathcal{X}}_{i}(G)$, as
desired. $\blacksquare$

In fact, looking carefully at the proof of Theorem 1.2 and extracting the
abstract properties of the $\operatorname{\mathcal{X}}$-series used in the
proof allows us to state a generalisation of Theorem 1.2. We need a definition
first.

######  Definition 3.7.

Given a $p$-group $G$ we call the series

$\operatorname{\mathcal{W}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(G)\cap\operatorname{\mathcal{B}}_{i}(G),$

a good series for $G$ if the following three conditions are satisfied:

  1. (i)

$\\{\operatorname{\mathcal{B}}_{i}(G)\\}$ is a descending normal series of
$G$;

  2. (ii)

$\operatorname{\mathcal{B}}_{i}(G)$ (and thus
$\operatorname{\mathcal{W}}_{i}(G)$ as well) is subgroup-monotone;

  3. (iii)

$\operatorname{\mathcal{B}}_{i}(G)$ (and thus
$\operatorname{\mathcal{W}}_{i}(G)$ as well) respects direct products;

  4. (iv)

If $C$ is a cyclic group, then
$\Omega_{1}(C)\leqslant\operatorname{\mathcal{B}}_{1}(C)$ (and thus
$\Omega_{1}(C)\leqslant\operatorname{\mathcal{W}}_{1}(C)$ as well).

Clearly, $\operatorname{\mathcal{X}}_{i}(G)$ itself is a good series for $G$.
Other examples of good series are the following:

  1. (1)

$\operatorname{\mathcal{W}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(G)\cap\Omega_{1}(G)$;

  2. (2)

$\operatorname{\mathcal{W}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(G)\cap\mho_{i-1}(G)$;

  3. (3)

$\operatorname{\mathcal{W}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(G)\cap\mho_{i-1}(G)\cap\Omega_{1}(G)$.

The generalisation asserted earlier now reads as follows.

######  Theorem 3.8.

Let $A$ be an abelian subgroup of the $p$-group $G$ and suppose that
$\operatorname{\mathcal{W}}_{i}$ is a good series for $G$ so that

$A\cap\operatorname{\mathcal{W}}_{i}(G)=\operatorname{\mathcal{W}}_{i}(A)$

for all integers $i\geqslant 1$. Then $A$ has a complement in $G$.

Since the proof is nearly identical to that of Theorem 1.2, we have elected to
omit it.

We remark that the following result, although resembling Corollary 2.7, is
ultimately independent, since no normality assumption is made.

######  Proposition 3.9.

Let $G$ be a $p$-group and $A$ an abelian subgroup of $G$ such that

$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$

for all non-negative integers $i$. Then
$A\cap\operatorname{\mathcal{X}}_{i}(G)$ has a complement in
$\operatorname{\mathcal{X}}_{i}(G)$ for all $i\geqslant 0$.

Proof.  By Theorem 1.2 we have already established the case $i=0$, as
$\operatorname{\mathcal{X}}_{0}(G)=G$ and $A$ has a complement in $G$.
According to the same theorem, the abelian group
$A\cap\operatorname{\mathcal{X}}_{i}(G)$ has a complement in
$\operatorname{\mathcal{X}}_{i}(G)$ provided we can show that

$(A\cap\operatorname{\mathcal{X}}_{i}(G))\cap\operatorname{\mathcal{X}}_{t}(\operatorname{\mathcal{X}}_{i}(G))=\operatorname{\mathcal{X}}_{t}(A\cap\operatorname{\mathcal{X}}_{i}(G))$
(3.0)

for every integer $t\geqslant 1$.

One inclusion is straightforward, as the $\operatorname{\mathcal{X}}$-series
is subgroup-monotone and therefore

$\operatorname{\mathcal{X}}_{t}(A\cap\operatorname{\mathcal{X}}_{i}(G))\leqslant(A\cap\operatorname{\mathcal{X}}_{i}(G))\cap\operatorname{\mathcal{X}}_{t}(\operatorname{\mathcal{X}}_{i}(G)).$

For the other inclusion, we first observe that

$\operatorname{\mathcal{X}}_{t}(A\cap\operatorname{\mathcal{X}}_{i}(G))=\operatorname{\mathcal{X}}_{t}(\operatorname{\mathcal{X}}_{i}(A))=\operatorname{\mathcal{X}}_{t+i}(A),$

where the last equality follows from Proposition 2.9. Hence

$(A\cap\operatorname{\mathcal{X}}_{i}(G))\cap\operatorname{\mathcal{X}}_{t}(\operatorname{\mathcal{X}}_{i}(G))=A\cap\operatorname{\mathcal{X}}_{t}(\operatorname{\mathcal{X}}_{i}(G))\leqslant
A\cap\operatorname{\mathcal{X}}_{t+i}(G)=\operatorname{\mathcal{X}}_{t+i}(A)=\operatorname{\mathcal{X}}_{t}(A\cap\operatorname{\mathcal{X}}_{i}(G)).$

So we have shown that equation (3.0) is valid for every $t\geqslant 1$, and
the proposition follows. $\blacksquare$

######  Corollary 3.10.

Let $G$ be a $p$-group and let $A$ be a normal abelian subgroup of $G$ with
the property

$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$

for all non-negative integers $i$. Then $A$ has a complement in $G$ and for
every complement $H$ the analogous property

$H\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(H)$

holds for all $i\geqslant 0$. As a consequence
$\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)\rtimes\operatorname{\mathcal{X}}_{i}(H)$
for all $i\geqslant 0$.

Proof.  Since
$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$ for
all $i\geqslant 0$, Theorem 1.2 implies that $A$ is complemented in $G$. If
$H$ is any such complement for $A$, then the fact that
$H\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(H)$
holds for all $i\geqslant 0$ is a consequence of Theorem 1.3. Moreover,
Corollary 2.7 implies that $A\cap\operatorname{\mathcal{X}}_{i}(G)$, which by
assumption is $\operatorname{\mathcal{X}}_{i}(A)$, is complemented in
$\operatorname{\mathcal{X}}_{i}(G)$ by $\operatorname{\mathcal{X}}_{i}(H)$ and
this holds, of course, for all $i\geqslant 0$. $\blacksquare$

We conclude our work by pointing out the limitations of Theorem 1.2.

First of all, by virtue of Proposition 2.10 it is the case that if $A$ is
cyclic then it is complemented in $G$ if and only if
$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$ for
all $i\geqslant 0$. Moreover, the condition works as “if and only if” in case
$A$ is central in $G$ owing to Corollary 3.6.

On the other hand, observe that an abelian subgroup $A$ of a $p$-group $G$ can
be complemented in $G$ without satisfying the condition
$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$ for
all $i\geqslant 0$. Concrete examples abound; one possibility is to take $A$
to be an elementary abelian group containing the Frattini subgroup. If $A$ is
complemented in $G$ (and this happens for many groups, e.g. $D_{8}$ with $A$
either of the two non-cyclic maximal subgroups), where $G$ is non-abelian,
then it cannot possibly be the case that
$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$ for
all $i\geqslant 0$. Things go awry as quickly as possible, i.e. for $i=1$.

In fact, an elementary abelian subgroup $A$ satisfies
$A\cap\operatorname{\mathcal{X}}_{i}(G)=\operatorname{\mathcal{X}}_{i}(A)$ for
all $i\geqslant 0$ if and only if $A\cap\Phi(G)=1$, thus Theorem 1.2 is only
useful when $A$ avoids the Frattini subgroup of $G$. However, in that case
Theorem 1.2 tells us nothing new since Satz 7 in Gaschütz’s paper [Gas53]
guarantees that $A$ has a complement in $G$ regardless of whether $G$ is a
$p$-group or not, as long as $G$ is finite.

What is perhaps not so obvious is the following.

######  Proposition 3.11.

Let $G$ be a $p$-group and let $A$ be a normal subgroup of $G$ such that
$A\cap\Phi(G)=1$. Then $A$ has a normal complement in $G$ and is thus a direct
factor of $G$.

Proof.  First of all, observe that we do not need to assume that $A$ is
abelian since it follows from

$\Phi(A)\leqslant A\cap\Phi(G)=1$

that $A$ must, in fact, be elementary abelian.

Now we argue by induction on $|A|$. The claim is evident if $A=1$, so the
induction begins and we assume that $A>1$. Since $A\cap\Phi(G)=1$ and $A>1$,
there exists a maximal subgroup $M$ of $G$ such that $A$ is not contained in
$M$. Write $D\coloneqq A\cap M$. In case $D=1$, we are done because $M$ is
normal in $G$. So suppose that $D>1$. Observe that since $AM>M$ and $M$ is
maximal in $G$, we have $G=AM$. Now both $M$ and $A$ normalise $D$, so $D$ is
normal in $G$.

The induction hypothesis applied to $D$ in $G$ ensures the existence of a
normal complement $K$. In other words, $G=D\times K$. By Dedekind’s lemma, we
have $A=D\times(A\cap K)$. The subgroup $A\cap K$ is normal in $K$ and proper
in $A$. Also, $\Phi(G)=\Phi(D)\times\Phi(K)=\Phi(K)$. This holds because $A$
is elementary abelian hence so is $D$.

Now

$(A\cap K)\cap\Phi(K)=A\cap\Phi(K)=A\cap\Phi(G)=1.$

Applying the induction hypothesis to $A\cap K$ in $K$, we get $K=(A\cap
K)\times H$ for some subgroup $H$. But now $H$ is normalised by $K$ and also
by $D$, because $D$ actually centralises it, and since $G=DK$ we see that $H$
is normal in $G$. It is easy to see that $A\cap H=1$, so $H$ is the required
normal complement and that completes our induction. $\blacksquare$

It is clear that the same proof would work just as well if we had assumed more
generally that $G$ is a nilpotent group instead of a $p$-group. We preferred
to state the proposition in a more restricted setting to remain in the geist
of the rest of our paper.

######  Acknowledgements.

That Proposition 3.11 is true was communicated to the first author, without
proof, by Martin Isaacs during the winter of 2020. Both authors are grateful
to him for this contribution and for various suggestions to improve the
exposition. Thanks also go to Benjamin Sambale for spotting typos in an early
draft of the paper and for suggesting a simplification of the proof of
Proposition 2.4.

## References

  * [FT63] W. Feit and J. G. Thompson, _Solvability of groups of odd order_ , Pacific J. Math. 13 (1963), 775–1029.
  * [Fuc15] L. Fuchs, _Abelian Groups_ , Springer Monogr. Math., Cham: Springer, 2015\.
  * [Gas52] W. Gaschütz, _Zur Erweiterungstheorie der endlichen Gruppen_ , J. Reine Angew. Math. 190 (1952), 93–107.
  * [Gas53] W. Gaschütz, _Über die $\Phi$-Untergruppe endlicher Gruppen_, Math. Z. 58 (1953), 160–170.
  * [Kal46] L. Kaloujnine, _Sur les $p$-groupes de sylow du groupe symétrique du degré $p^{m}$. (Suite centrale ascendante et descendante)_, C. R. Acad. Sci., Paris 223 (1946), 703–705 (French).
  * [Kir17] J. Kirtland, _Complementation of Normal Subgroups. In Finite Groups_ , Berlin: De Gruyter, 2017.
  * [LGM02] C. R. Leedham-Green and S. McKay, _The Structure of Groups of Prime Power Order_ , Lond. Math. Soc. Monogr., New Ser., vol. 27, Oxford: Oxford University Press, 2002.
  * [Sam22] B. Sambale, _On the converse of Gaschütz’ complement theorem_ , https://www.iazd.uni-hannover.de/fileadmin/iazd/sambale/pdfs/GaConverse.pdf, 2022\.


11institutetext: Department of Computer Science
University of Miami
1365 Memorial Drive, Coral Gables, FL, 33146 USA
<EMAIL_ADDRESS>
# Off-Policy General Value Functions to Represent Dynamic Role Assignments in
RoboCup 3D Soccer Simulation

Saminda Abeyruwan  Andreas Seekircher  Ubbo Visser

###### Abstract

Collecting and maintaining accurate world knowledge in a dynamic, complex,
adversarial, and stochastic environment such as the RoboCup 3D Soccer
Simulation is a challenging task. Knowledge should be learned in real-time
with time constraints. We use recently introduced Off-Policy Gradient Descent
algorithms within Reinforcement Learning that illustrate learnable knowledge
representations for dynamic role assignments. The results show that the agents
have learned competitive policies against the top teams from the RoboCup 2012
competitions for three vs three, five vs five, and seven vs seven agents. We
have explicitly used subsets of agents to identify the dynamics and the
semantics for which the agents learn to maximize their performance measures,
and to gather knowledge about different objectives, so that all agents
participate effectively and efficiently within the group.

ynamic Role Assignment Function, Reinforcement Learning, GQ($\lambda$),
Greedy-GQ($\lambda$), Off-PAC, Off-Policy Prediction and Control, and RoboCup
3D Soccer Simulation.

###### Keywords:

D

##  1 Introduction

The RoboCup 3D Soccer Simulation environment provides a dynamic, real-time,
complex, adversarial, and stochastic multi-agent environment for simulated
agents. The simulated agents formalize their goals in two layers:  1. the
physical layers, where controls related to walking, kicking etc. are
conducted; and  2. the decision layers, where high level actions are taken to
emerge behaviors.  In this paper, we investigate a mechanism suitable for
decision layers to use recently introduced Off-Policy Gradient Decent
Algorithms in Reinforcement Leaning (RL) that illustrate learnable knowledge
representations to learn about a dynamic role assignment function.

In order to learn about an effective dynamic role assignment function, the
agents need to consider the dynamics of agent-environment interactions. We
consider these interactions as the agent’s knowledge. If this knowledge is
represented in a formalized form (e.g., first-order predicate logic) an agent
could infer many aspects about its interactions consistent with that
knowledge. The knowledge representational forms show different degrees of
computational complexities and expressiveness [22]. The computational
requirements increase with the extension of expressiveness of the
representational forms. Therefore, we need to identify and commit to a
representational form, which is scalable for on-line learning while preserving
expressivity. A human soccer player knows a lot of information about the game
before (s)he enters onto the field and this prior knowledge influences the
outcome of the game to a great extent. In addition, human soccer players
dynamically change their knowledge during games in order to achieve maximum
rewards. Therefore, the knowledge of the human soccer player is to a certain
extent either predictive or goal-oriented. Can a robotic soccer player collect
and maintain predictive and goal-oriented knowledge? This is a challenging
problem for agents with time constraints and limited computational resources.

We learn the role assignment function using a framework that is developed
based on the concepts of Horde, the real-time learning methodology, to express
knowledge using General Value Functions (GVFs) [22]. Similar to Horde’s sub-
agents, the agents in a team are treated as independent RL sub-agents, but the
agents take actions based on their belief of the world model. The agents may
have different world models due to noisy perceptions and communication delays.
The GVFs are constituted within the RL framework. They are predictions or off-
policy controls that are answers to questions. For example, in order to make a
prediction a question must be asked of the form “If I move in this formation,
would I be in a position to score a goal?”, or “What set of actions do I need
to block the progress of the opponent agent with the number 3?”. The question
defines what to learn. Thus, the problem of prediction or control can be
addressed by learning value functions. An agent obtains its knowledge from
information communicated back and forth between the agents and the agent-
environment interaction experiences.

There are primarily two algorithms to learn about the GVFs, and these
algorithms are based on Off-Policy Gradient Temporal Difference (OP-GTD)
learning:  1. with action-value methods, a prediction question uses
GQ($\lambda$) algorithm [8], and a control or a goal-oriented question uses
Greedy-GQ($\lambda$) algorithm [9]. These algorithms learned about a
deterministic target policies and the control algorithm finds the greedy
action with respect to the action-value function; and  2. with policy-gradient
methods, a goal-oriented question can be answered using Off-Policy Actor-
Critic algorithm [24], with an extended state-value function, GTD($\lambda$)
[7], for GVFs. The policy gradient methods are favorable for problems having
stochastic optimal policies, adversarial environments, and problems with large
action spaces.  The OP-GTD algorithms possess a number of properties that are
desirable for on-line learning within the RoboCup 3D Soccer Simulation
environment:  1. off-policy updates;  2. linear function approximation;  3. no
restrictions on the features used;  4. temporal-difference learning;  5. on-
line and incremental;  6. linear in memory and per-time-step computation
costs; and  7. convergent to a local optimum or equilibrium point [23, 9].

In this paper, we present a methodology and an implementation to learn about a
dynamic role assignment function considering the dynamics of agent-environment
interactions based on GVFs. The agents ask questions and approximate value
functions answer to those questions. The agents independently learn about the
role assignment functions in the presence of an adversary team. Based on the
interactions, the agents may have to change their roles in order to continue
in the formation and to maximize rewards. There is a finite number of roles
that an agent can commit to, and the GVFs learn about the role assignment
function. We have conducted all our experiments in the RoboCup 3D Soccer
Simulation League Environment. It is based on the general purpose multi-agent
simulator SimSpark111http://svn.code.sf.net/p/simspark/svn/trunk/. The robot
agents in the simulation are modeled based on the Aldebaran
NAO222http://www.aldebaran-robotics.com/ robots. Each robot has 22 degrees of
freedom. The agents communicate with the server through message passing and
each agent is equipped with noise free joint perceptors and effectors. In
addition to this, each agent has a noisy restricted vision cone of $120^{o}$.
Every simulation cycle is limited to $20~{}ms$, where agents perceive noise
free angular measurements of each joint and the agents stimulate the necessary
joints by sending torque values to the simulation server. The vision
information from the server is available every third cycle ($60~{}ms$), which
provides spherical coordinates of the perceived objects. The agents also have
the option of communicating with each other every other simulation cycle
($40~{}ms$) by broadcasting a $20~{}bytes$ message. The simulation league
competitions are currently conducted with 11 robots on each side (22 total).

The remainder of the paper is organized as follows: In Section 2, we briefly
discuss knowledge representation forms and existing role assignment
formalisms. In Section 3, we introduce GVFs within the context of robotic
soccer. In Section 4, we formalize our mechanisms of dynamic role assignment
functions within GVFs. In Section 5, we identify the question and answer
functions to represent GVFs, and Section 6 presents the experiment results and
the discussion. Finally, Section 7 contains concluding remarks, and future
work.

##  2 Related Work

One goal of multi-agent systems research is the investigation of the prospects
of efficient cooperation among a set of agents in real-time environments. In
our research, we focus on the cooperation of a set of agents in a real-time
robotic soccer simulation environment, where the agents learn about an optimal
or a near-optimal role assignment function within a given formation using
GVFs. This subtask is particularly challenging compared to other simulation
leagues considering the limitations of the environment, i.e. the limited
locomotion capabilities, limited communication bandwidth, or crowd management
rules. The role assignment is a part of the hierarchical machine learning
paradigm [20, 19], where a formation defines the role space. Homogeneous
agents can change roles flexibly within a formation to maximize a given reward
function.

RL framework offerers a set of tools to design sophisticated and hard-to-
engineer behaviors in many different robotic domains (e.g., [4]). Within the
domain of robotic soccer, RL has been successfully applied in learning the
keep-away subtask in the RoboCup 2D [18] and 3D [16] Soccer Simulation
Leagues. Also, in other RoboCup leagues, such as the Middle Size League, RL
has been applied successfully to acquire competitive behaviors [2]. One of the
noticeable impact on RL is reported by the Brainstormers team, the RoboCup 2D
Simulation League team, on learning different subtasks [14]. A comprehensive
analysis of a general batch RL framework for learning challenging and complex
behaviors in robot soccer is reported in [15]. Despite convergence guarantees,
Q($\lambda$) [21] with linear function approximation has been used in role
assignment in robot soccer [5] and faster learning is observed with the
introduction of heuristically accelerated methods [3]. The dynamic role
allocation framework based on dynamic programming is described in [6] for
real-time soccer environments. The role assignment with this method is tightly
coupled with the agent’s low-level abilities and does not take the opponents
into consideration. On the other hand, the proposed framework uses the
knowledge of the opponent positions as well as other dynamics for the role
assignment function.

Sutton et al. [22] have introduced a real-time learning architecture, Horde,
for expressing knowledge using General Value Functions (GVFs). Our research is
built on Horde to ask a set of questions such that the agents assign optimal
or near-optimal roles within formations. In addition, following researches
describe methods and components to build strategic agents: [1] describes a
methodology to build a cognizant robot that possesses vast amount of situated,
reversible and expressive knowledge. [11] presents a methodology to “next” in
real time predicting thousands of features of the world state, and [10]
presents methods predict about temporally extended consequences of a robot’s
behaviors in general forms of knowledge. The GVFs are successfully used (e.g.,
[13, 25]) for switching and prediction tasks in assistive biomedical robots.

##  3 Learnable knowledge representation for Robotic Soccer

Recently, within the context of the RL framework [21], a knowledge
representation language has been introduced, that is expressive and learnable
from sensorimotor data. This representation is directly usable for robotic
soccer as agent-environment interactions are conducted through perceptors and
actuators. In this approach, knowledge is represented as a large number of
approximate value functions each with its  1. own policy;  2. pseudo-reward
function;  3. pseudo-termination function; and  4. pseudo-terminal-reward
function[22].  In continuous state spaces, approximate value functions are
learned using function approximation and using more efficient off-policy
learning algorithms. First, we briefly introduce some of the important
concepts related to the GVFs. The complete information about the GVFs are
available in [22, 8, 9, 7]. Second, we show its direct application to
simulated robotic soccer.

###  3.1 Interpretation

The interpretation of the approximate value function as a knowledge
representation language grounded on information from perceptors and actuators
is defined as:

###### Definition 1

The knowledge expressed as an approximate value function is true or accurate,
if its numerical values matches those of the mathematically defined value
function it is approximating.

Therefore, according to the Definition (1), a value function asks a question,
and an approximate value function is the answer to that question. Based on
prior interpretation, the standard RL framework extends to represent learnable
knowledge as follows. In the standard RL framework [21], let the agent and the
world interact in discrete time steps $t=1,2,3,\ldots$. The agent senses the
state at each time step $S_{t}\in\mathcal{S}$, and selects an action
$A_{t}\in\mathcal{A}$. One time step later the agent receives a scalar reward
$R_{t+1}\in\mathbb{R}$, and senses the state $S_{t+1}\in\mathcal{S}$. The
rewards are generated according to the reward function
$r:S_{t+1}\rightarrow\mathbb{R}$. The objective of the standard RL framework
is to learn the stochastic action-selection policy
$\pi:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$, that gives the probability
of selecting each action in each state,
$\pi(s,a)=\pi(s|a)=\mathcal{P}(A_{t}=a|S_{t}=s)$, such that the agent
maximizes rewards summed over the time steps. The standard RL framework
extends to include a terminal-reward-function,
$z:\mathcal{S}\rightarrow\mathbb{R}$, where $z(s)$ is the terminal reward
received when the termination occurs in state $s$. In the RL framework,
$\gamma\in[0,1)$ is used to discount delayed rewards. Another interpretation
of the discounting factor is a constant probability of $1-\gamma$ termination
of arrival to a state with zero terminal-reward. This factor is generalized to
a termination function $\gamma:\mathcal{S}\rightarrow[0,1]$, where
$1-\gamma(s)$ is the probability of termination at state $s$, and a terminal
reward $z(s)$ is generated.

###  3.2 Off-Policy Action-Value Methods for GVFs

The first method to learn about GVFs, from off-policy experiences, is to use
action-value functions. Let $G_{t}$ be the complete return from state $S_{t}$
at time $t$, then the sum of the rewards (transient plus terminal) until
termination at time $T$ is:

$G_{t}=\sum_{k=t+1}^{T}r(S_{k})+z(S_{T}).$

The action-value function is:

$Q^{\pi}(s,a)=\mathbb{E}(G_{t}|S_{t}=s,A_{t}=a,A_{t+1:T-1}\sim\pi,T\sim\gamma),$

where, $Q^{\pi}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. This is
the expected return for a trajectory started from state $s$, and action $a$,
and selecting actions according to the policy $\pi$, until termination occurs
with $\gamma$. We approximate the action-value function with
$\hat{Q}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$. Therefore, the
action-value function is a precise grounded question, while the approximate
action-value function offers the numerical answer. The complete algorithm for
Greedy-GQ($\lambda$) with linear function approximation for GVFs learning is
as shown in Algorithm (1).

1:Initialize $w_{0}$ to $0$, and $\theta_{0}$ arbitrary.

2:Choose proper (small) positive values for $\alpha_{\theta}$, $\alpha_{w}$,
and set values for $\gamma(.)\in(0,1]$, $\lambda(.)\in[0,1]$.

3:repeat

4: Initialize $e=0$.

5: Take $A_{t}$ from $S_{t}$ according to $\pi_{b}$, and arrive at $S_{t+1}$.

6: Observe sample, ($S_{t},A_{t},r(S_{t+1}),z(S_{t+1}),S_{t+1},$) at time step
$t$ (with their corresponding state-action feature vectors), where
$\hat{\phi}_{t+1}=\phi(S_{t+1},A_{t+1}^{*}),A_{t+1}^{*}=\operatornamewithlimits{argmax}_{b}{\bf\theta}_{t}^{\mathrm{T}}\phi(S_{t+1},b)$.

7: for each observed sample do

8: $\delta_{t}\leftarrow
r(S_{t+1})+(1-\gamma(S_{t+1}))z(S_{t+1})+\gamma(S_{t+1})\theta_{t}^{\mathrm{T}}\hat{\phi}_{t+1}-\theta_{t}^{\mathrm{T}}\phi_{t}$.

9: If $A_{t}=A_{t}^{*}$, then
$\rho_{t}\leftarrow\frac{1}{\pi_{b}(A_{t}^{*}|S_{t})}$; otherwise
$\rho_{t}\rightarrow 0$.

10: $e_{t}\leftarrow
I_{t}\phi_{t}+\gamma(S_{t})\lambda(S_{t})\rho_{t}e_{t-1}$.

11:
$\theta_{t+1}\leftarrow\theta_{t}+\alpha_{\theta}[\delta_{t}e_{t}-\gamma(S_{t+1})(1-\lambda(S_{t+1}))(w_{t}^{\mathrm{T}}e_{t})\hat{\phi}_{t+1}]$.

12: $w_{t+1}\leftarrow
w_{t}+\alpha_{w}[\delta_{t}e_{t}-(w_{t}^{\mathrm{T}}\phi_{t})\phi_{t})]$.

13: end for

14:until each episode.

Algorithm 1 Greedy-GQ($\lambda$) with linear function approximation for GVFs
learning [7].

The GVFs are defined over four functions: $\pi,\gamma,r,\mbox{and }z$. The
functions $r\mbox{ and }z$ act as pseudo-reward and pseudo-terminal-reward
functions respectively. Function $\gamma$ is also in pseudo form as well.
However, $\gamma$ function is more substantive than reward functions as the
termination interrupts the normal flow of state transitions. In pseudo
termination, the standard termination is omitted. In robotic soccer, the base
problem can be defined as the time until a goal is scored by either the home
or the opponent team. We can consider a pseudo-termination has occurred when
the striker is changed. The GVF with respect to a state-action function is
defined as:

$Q^{\pi,\gamma,r,z}(s,a)=\mathbb{E}(G_{t}|S_{t}=s,A_{t}=a,A_{t+1:T-1}\sim\pi,T\sim\gamma).$

The four functions, $\pi,\gamma,r,\mbox{and }z$, are the question functions to
GVFs, which in return defines the general value function’s semantics. The RL
agent learns an approximate action-value function, $\hat{Q}$, using the four
auxiliary functions $\pi,\gamma,r$ and $z$. We assume that the state space is
continuous and the action space is discrete. We approximate the action-value
function using a linear function approximator. We use a feature extractor
$\mathcal{\phi}:S_{t}\times A_{t}\rightarrow\\{0,1\\}^{N},N\in\mathbb{N}$,
built on tile coding [21] to generate feature vectors from state variables and
actions. This is a sparse vector with a constant number of “1” features,
hence, a constant norm. In addition, tile coding has the key advantage of
real-time learning and to implement computationally efficient algorithms to
learn approximate value functions. In linear function approximation, there
exists a weight vector, $\theta\in\mathbb{R}^{N},N\in\mathbb{N}$, to be
learned. Therefore, the approximate GVFs are defined as:

$\hat{Q}(s,a,\theta)=\theta^{\mathrm{T}}\phi(s,a),$

such that,
$\hat{Q}:\mathcal{S}\times\mathcal{A}\times\mathbb{R}^{N}\rightarrow\mathbb{R}$.
Weights are learned using the gradient-descent temporal-difference Algorithm
(1) [7]. The Algorithm learns stably and efficiently using linear function
approximation from off-policy experiences. Off-policy experiences are
generated from a behavior policy, $\pi_{b}$, that is different from the policy
being learned about named as target policy, $\pi$. Therefore, one could learn
multiple target policies from the same behavior policy.

###  3.3 Off-Policy Policy Gradient Methods for GVFs

The second method to learn about GVFs is using the off-policy policy gradient
methods with actor-critic architectures that use a state-value function
suitable for learning GVFs. It is defined as:

$V^{\pi,\gamma,r,z}(s)=\mathbb{E}(G_{t}|S_{t}=s,A_{t:T-1}\sim\pi,T\sim\gamma),$

where, $V^{\pi,\gamma,r,z}(s)$ is the true state-value function, and the
approximate GVF is defined as:

$\hat{V}(s,v)=v^{\mathrm{T}}\phi(s),$

where, the functions $\pi,\gamma,r,\mbox{and }z$ are defined as in the
subsection (3.2). Since our the target policy $\pi$ is discrete stochastic, we
use a Gibbs distribution of the form:

$\pi(a|s)=\frac{e^{u^{\mathrm{T}}\phi(s,a)}}{\sum_{b}e^{u^{\mathrm{T}}\phi(s,b)}},$

where, $\phi(s,a)$ are state-action features for state $s$, and action $a$,
which are in general unrelated to state features $\phi(s)$, that are used in
state-value function approximation.
$u\in\mathbb{R}^{N_{u}},N_{u}\in\mathbb{N}$, is a weight vector, which is
modified by the actor to learn about the stochastic target policy. The log-
gradient of the policy at state $s$, and action $a$, is:

$\frac{\nabla_{u}\pi(a|s)}{\pi(a|s)}=\phi(s,a)-\sum_{b}\pi(b|s)\phi(s,b).$

The complete algorithm for Off-PAC with linear function approximation for GVFs
learning is as shown in Algorithm (2).

1:Initialize $w_{0}$ to $0$, and $v_{0}$ and $u_{0}$ arbitrary.

2:Choose proper (small) positive values for $\alpha_{v}$, $\alpha_{w}$,
$\alpha_{u}$, and set values for $\gamma(.)\in(0,1]$, $\lambda(.)\in[0,1]$.

3:repeat

4: Initialize $e^{v}=0,\mbox{and }e^{u}=0$.

5: Take $A_{t}$ from $S_{t}$ according to $\pi_{b}$, and arrive at $S_{t+1}$.

6: Observe sample, ($S_{t},A_{t},r(S_{t+1}),z(S_{t+1}),S_{t+1}$) at time step
$t$ (with their corresponding state ($\phi_{t},\phi_{t+1}$) feature vectors,
where $\phi_{t}=\phi(S_{t})$).

7: for each observed sample do

8: $\delta_{t}\leftarrow
r(S_{t+1})+(1-\gamma(S_{t+1}))z(S_{t+1})+\gamma(S_{t+1})v_{t}^{\mathrm{T}}\phi_{t+1}-v_{t}^{\mathrm{T}}\phi_{t}$.

9: $\rho_{t}\leftarrow\frac{\pi(A_{t}|S_{t})}{\pi_{b}(A_{t}|S_{t})}$.

10: Update the critic (GTD($\lambda$) algorithm for GVFs).

11:
$e^{v}_{t}\leftarrow\rho_{t}(\phi_{t}+\gamma(S_{t})\lambda(S_{t})e^{v}_{t-1})$.

12: $v_{t+1}\leftarrow
v_{t}+\alpha_{v}[\delta_{t}e^{v}_{t}-\gamma(S_{t+1})(1-\lambda(S_{t+1}))({e^{v}_{t}}^{\mathrm{T}}w_{t})\phi_{t+1}]$.

13: $w_{t+1}\leftarrow
w_{t}+\alpha_{w}[\delta_{t}e_{t}-(w_{t}^{\mathrm{T}}\phi_{t})\phi_{t})]$.

14: Update the actor.

15:
$e^{u}_{t}\leftarrow\rho_{t}\left[\frac{\nabla_{u}\pi(A_{t}|S_{t})}{\pi(A_{t}|S_{t})}+\gamma(S_{t})\lambda(S_{t+1})e^{u}_{t-1}\right]$.

16: $u_{t+1}\leftarrow u_{t}+\alpha_{u}\delta_{t}e^{u}_{t}$.

17: end for

18:until each episode.

Algorithm 2 Off-PAC with linear function approximation for GVFs learning [7,
24].

We are interested in finding optimal policies for the dynamic role assignment,
and henceforth, we use Algorithms (1), and (2) for control purposes333We use
an C++ implementation of Algorithm (1) and (2) in all of our experiments. An
implementation is available in https://github.com/samindaa/RLLib. We use
linear function approximation for continuous state spaces, and discrete
actions are used within options. Lastly, to summarize, the definitions of the
question functions and the answer functions are given as:

###### Definition 2

The question functions are defined by:

  1. 1.

$\pi:S_{t}\times A_{t}\rightarrow[0,1]$ 35mm (target policy is greedy w.r.t.
learned value function);

  2. 2.

$\gamma:S_{t}\rightarrow[0,1]$ 35mm (termination function);

  3. 3.

$r:S_{t+1}\rightarrow\mathbb{R}$ 35mm (transient reward function); and

  4. 4.

$z:S_{t+1}\rightarrow\mathbb{R}$ 35mm (terminal reward function).

###### Definition 3

The answer functions are defined by:

  1. 1.

$\pi_{b}:S_{t}\times A_{t}\rightarrow[0,1]$ 35mm (behavior policy);

  2. 2.

$I_{t}:S_{t}\times A_{t}\rightarrow[0,1]$ 35mm (interest function);

  3. 3.

$\phi:S_{t}\times A_{t}\rightarrow\mathbb{R}^{N}$ 35mm (feature-vector
function); and

  4. 4.

$\lambda:S_{t}\rightarrow[0,1]$ 35mm (eligibility-trace decay-rate function).

##  4 Dynamic Role Assignment

A role is a specification of an internal or an external behavior of an agent.
In our soccer domain, roles select behaviors of agents based on different
reference criteria: the agent close to the ball becomes the striker. Given a
role space, $\mathcal{R}=\\{r_{1},\ldots,r_{n}\\}$, of size $n$, the
collaboration among $m\leq n$ agents, $\mathcal{A}=\\{a_{1},\dots,a_{m}\\}$,
is obtained through formations. The role space consists of active and reactive
roles. For example, the striker is an active role and the defender could be a
reactive role. Given a reactive role, there is a function, $R\mapsto T$, that
maps roles to target positions, $T$, on the field. These target positions are
calculated with respect to a reference pose (e.g., ball position) and other
auxiliary criteria such as crowd management rules. A role assignment function,
$R\mapsto A$, provides a mapping from role space to agent space, while
maximizing some reward function. The role assignment function can be static or
dynamic. Static role assignments often provide inferior performance in robot
soccer [6]. Therefore, we learn a dynamic role assignment function within the
RL framework using off-policy control.

Figure 1: Primary formation, [17]

###  4.1 Target Positions with the Primary Formation

Within our framework, an agent can choose one role among thirteen roles. These
roles are part of a primary formation, and an agent calculates the respective
target positions according to its belief of the absolute ball position and the
rules imposed by the 3D soccer simulation server. We have labeled the role
space in order to describe the behaviors associated with them. Figure (1)
shows the target positions for the role space before the kickoff state. The
agent closest to the ball takes the striker role (SK), which is the only
active role. Let us assume that the agent’s belief of the absolute ball
position is given by $(x_{b},y_{b})$. Forward left (FL) and forward right (FR)
target positions are offset by $(x_{b},y_{b})\pm(0,2)$. The extended forward
left (EX1L) and extended forward right ((EX1R)) target positions are offset by
$(x_{b},y_{b})\pm(0,4)$. The stopper (ST) position is given by
$(x_{b}-2.0,y_{b})$. The extended middle (EX1M) position is used as a blocking
position and it is calculated based on the closest opponent to the current
agent. The other target positions, wing left (WL), wing right (WR), wing
middle (WM), back left (BL), back right (BR), and back middle (BM) are
calculated with respect to the vector from the middle of the home goal to the
ball and offset by a factor which increases close to the home goal. When the
ball is within the reach of goal keeper, the (GK) role is changed to goal
keeper striker (GKSK) role. We slightly change the positions when the ball is
near the side lines, home goal, and opponent goal. These adjustments are made
in order to keep the target positions inside the field. We allow target
positions to be overlapping. The dynamic role assignment function may assign
the same role during the learning period. In order to avoid position conflicts
an offset is added; the feedback provides negative rewards for such
situations.

###  4.2 Roles to RL Action Mapping

The agent closest to the ball becomes the striker, and only one agent is
allowed to become the striker. The other agents except the goalie are allowed
to choose from twelve roles. We map the available roles to discrete actions of
the RL algorithm. In order to use Algorithm 1, an agent must formulate a
question function using a value function, and the answer function provides the
solution as an approximate value function. All the agents formulate the same
question: What is my role in this formation in order to maximize future
rewards? All agents learn independently according to the question, while
collaboratively aiding each other to maximize their future reward. We make the
assumption that the agents do not communicate their current role. Therefore,
at a specific step, multiple agents may commit to the same role. We discourage
this condition by modifying the question as What is my role in this formation
in order to maximize future rewards, while maintaining a completely different
role from all teammates in all time steps?

Figure 2: State variable representation and the primary function. Some field
lines are omitted due to clarity.

###  4.3 State Variables Representation

Figure 2 shows the schematic diagram of the state variable representation. All
points and vectors in Figure 2 are defined with respect to a global coordinate
system. $h$ is the middle point of the home goal, while $o$ is the middle
point of the opponent goal. $b$ is the ball position. $\parallel$.$\parallel$
represents the vector length, while $\angle pqr$ represents the angle among
three points $p,~{}q,\mbox{ and }r$ pivoted at $q$. $a_{i}$ represents the
self-localized point of the $i=1,\ldots,11$ teammate agent. $y_{i}$ is some
point in the direction of the robot orientation of teammate agents. $c_{j}$,
$j=1,\ldots,11$, represents the mid-point of the tracked opponent agent. $x$
represents a point on a vector parallel to unit vector $e_{x}$. Using these
labels, we define the state variables as:

$\displaystyle\\{\parallel\vec{v}_{hb}\parallel,\parallel\vec{v}_{bo}\parallel,\angle
hbo,\\{\parallel\vec{v}_{a_{i}b}\parallel,\angle y_{i}a_{i}b,\angle
a_{i}bx\\}_{i=n_{start}}^{n_{end}},\\{\parallel\vec{v}_{c_{j}b}\parallel,\angle
c_{j}bx,\\}_{j=1}^{m_{max}}\\}.$

$n_{start}$ is the teammate starting id and $n_{end}$ the ending id. $m_{max}$
is the number of opponents considered. Angles are normalized to
[$-\frac{\pi}{2},\frac{\pi}{2}$].

##  5 Question and Answer Functions

There are twelve actions available in each state. We have left out the striker
role from the action set. The agent nearest to the ball becomes the striker.
All agents communicate their belief to other agents. Based on their belief,
all agents calculate a cost function and assign the closest agent as the
striker. We have formulated a cost function based on relative distance to the
ball, angle of the agent, number of teammates and opponents within a region
near the ball, and whether the agents are active. In our formulation, there is
a natural termination condition; scoring goals. With respect to the striker
role assignment procedure, we define a pseudo-termination condition. When an
agent becomes a striker, a pseudo-termination occurs, and the striker agent
does not participate in the learning process unless it chooses another role.
We define the question and answer functions as follows:

###  5.1 GVF Definitions for State-Action Functions

Question functions:

  1. 1.

$\pi=$ greedy w.r.t. $\hat{Q}$,

  2. 2.

$\gamma(.)=0.8$,

  3. 3.

$r(.)=$ (a) the change of $x$ value of the absolute ball position;  (b) a
small negative reward of $0.01$ for each cycle;  (c) a negative reward of $5$
is given to all agents within a radius of 1.5 meters;

  4. 4.

$z(.)=$ (a) $+100$for scoring against opponent;  (b) $-100$for opponent
scoring; and

  5. 5.

$\mbox{time step}=2$ seconds.

Answer functions:

  1. 1.

$\pi_{b}=$ $\epsilon$-greedy w.r.t. target state-action function,

  2. 2.

$\epsilon=0.05$,

  3. 3.

$I_{t}(.)=1$,

  4. 4.

$\phi(.,.)=$ (a) we use tile coding to formulate the feature vector.
$n_{start}=2$ and $n_{end}=3,5,7$. $m_{max}=3,5,7$. Therefore, there are
$18,28,30$ state variables.  (b) state variable is independently tiled with 16
tilings with approximately each with $\frac{1}{16}$ generalization. Therefore,
there are $288+1,448+1,608+1$ active tiles (i.e., tiles with feature 1) hashed
to a binary vector dimension $10^{6}+1$. The bias feature is always active,
and

  5. 5.

$\lambda(.)=0.8$.

Parameters:
• 6mm 1. $\parallel{\bf{\theta}}\parallel=\parallel{\bf w}\parallel=10^{6}+1$;
2. $\parallel{\bf e}\parallel=2000$(efficient trace implementation); 6mm  3.
$\alpha_{\theta}=\frac{0.01}{289},\frac{0.01}{449},\frac{0.01}{609}$; and  4.
$\alpha_{w}=0.001\times\alpha_{\theta}$.

###  5.2 GVF for Gradient Descent Functions

Question functions:

  1. 1.

$\pi=$ Gibbs distribution,

  2. 2.

$\gamma(.)=0.9$,

  3. 3.

$r(.)=$ (a) the change of $x$ value of the absolute ball position;  (b) a
small negative reward of $0.01$ for each cycle;  (c) a negative reward of $5$
is given to all agents within a radius of 1.5 meters;

  4. 4.

$z(.)=$ (a) $+100$for scoring against opponent;  (b) $-100$for opponent
scoring; and

  5. 5.

$\mbox{time step}=2$ seconds.

Answer functions:

  1. 1.

$\pi_{b}=$ the learned Gibbs distribution is used with a small perturbation.
In order to provide exploration, with probability $0.01$, Gibbs distribution
is perturbed using some $\beta$ value. In our experiments, we use $\beta=0.5$.
Therefore, we use a behavior policy:
$\frac{e^{u^{\mathrm{T}}\phi(s,a)+\beta}}{\sum_{b}e^{u^{\mathrm{T}}\phi(s,b)+\beta}}$

  2. 2.

$\phi(.)=$ (a) the representations for the state-value function, we use tile
coding to formulate the feature vector. $n_{start}=2$ and $n_{end}=3,5,7$.
$m_{max}=3,5,7$. Therefore, there are $18,28,30$ state variables.  (b) state
variable is independently tiled with 16 tilings with approximately each with
$\frac{1}{16}$ generalization. Therefore, there are $288+1,448+1,608+1$ active
tiles (i.e., tiles with feature 1) hashed to a binary vector dimension
$10^{6}+1$. The bias feature is always set to active;

  3. 3.

$\phi(.,.)=$ (a) the representations for the Gibbs distribution, we use tile
coding to formulate the feature vector. $n_{start}=2$ and $n_{end}=3,5,7$.
$m_{max}=3,5,7$. Therefore, there are $18,28,30$ state variables.  (b) state
variable is independently tiled with 16 tilings with approximately each with
$\frac{1}{16}$ generalization. Therefore, there are $288+1,448+1,608+1$ active
tiles (i.e., tiles with feature 1) hashed to a binary vector dimension
$10^{6}+1$. The hashing has also considered the given action. The bias feature
is always set to active; and

  4. 4.

$\lambda_{\mbox{critic}}(.)=\lambda_{\mbox{actor}}(.)=0.3$.

Parameters:
• 6mm 1. $\parallel{\bf{u}}\parallel=10^{6}+1$;  2.
$\parallel{\bf{\theta}}\parallel=\parallel{\bf w}\parallel=10^{6}+1$; 6mm 3.
$\parallel{\bf e^{v}}\parallel=\parallel{\bf e^{u}}\parallel=2000$(efficient
trace implementation); 6mm 4.
$\alpha_{v}=\frac{0.01}{289},\frac{0.01}{449},\frac{0.01}{609}$;  5.
$\alpha_{w}=0.0001\times\alpha_{v}$; and  6.
$\alpha_{v}=\frac{0.001}{289},\frac{0.001}{449},\frac{0.001}{609}$.

##  6 Experiments

We conducted experiments against the teams Boldhearts and MagmaOffenburg, both
semi-finalists of the RoboCup 3D Soccer Simulation competition in Mexico
2012444The published binary of the team UTAustinVilla showed unexpected
behaviors in our tests and is therefore omitted.. We conducted knowledge
learning according to the configuration given in Section (5). Subsection (6.1)
describes the performance of the Algorithm (1), and Subsection (6.2) describes
the performance of the Algorithm (2) for the experiment setup.

###  6.1 GVFs with Greedy-GQ($\lambda$)

The first experiments were done using a team size of five with the RL agents
against Boldhearts. After 140 games our RL agent increased the chance to win
from 30% to 50%. This number does not increase more in the next games, but
after 260 games the number of lost games (initially 35%) is reduced to 15%. In
the further experiments we used the goal difference to compare the performance
of the RL agent. Figure (3) shows the average goal differences that the hand-
tuned role assignment and the RL agents archive in games against Boldhearts
and MagmaOffenburg using different team sizes. With only three agents per team
the RL agent only needs 40 games to learn a policy that outperforms the hand-
coded role selection (Figure (3(a))). Also with five agents per team, the
learning agent is able to increase the goal difference against both opponents
(Figure (3(b))). However, it does not reach the performance of the manually
tuned role selection. Nevertheless considering the amount of time spent for
fine-tuning the hand-coded role selection, these results are promising.
Furthermore, the outcome of the games depends a lot on the underlying skills
of the agents, such as walking or dribbling. These skills are noisy, thus the
results need to be averaged over many games (std. deviations in Figure (3) are
between 0.5 and 1.3).

((a)) Three vs three agents.

((b)) Five vs five agents.

((c)) Seven vs seven agents.

Figure 3: Goal difference in games with (a) three; (b) five; and (c) seven
agents per team using Greedy-GQ($\lambda$) algorithm.

The results in Figure (3(c)) show a bigger gap between RL and the hand-coded
agent. However, using seven agents the goal difference is generally decreased,
since the defense is easily improved by increasing the number of agents. Also
the hand-coded role selection results in a smaller goal difference.
Furthermore, considering seven agents in each team the state space is already
increased significantly. Only 200 games seem to be not sufficient to learn a
good policy. Sometimes the RL agents reach a positive goal difference, but it
stays below the hand-coded role selection. In Section 7, we discuss some of
the reasons for this inferior performances for the team size seven. Even
though the RL agent did not perform well considering only the goal difference,
it has learned a moderately satisfactory policy. After 180 games the amount of
games won is increased slightly from initially 10% to approximately 20%.

###  6.2 GVFs with Off-PAC

With Off-PAC, we used a similar environment to that of Subsection (6.1), but
with a different learning setup. Instead of learning individual policies for
teams separately, we learned a single policy for both teams. We ran the
opponent teams in a round robin fashion for 200 games and repeated complete
runs for multiple times. The first experiments were done using a team size of
three with RL agents against both teams. Figure (4(a)) shows the results of
bins of 20 games averaged between two trials. After 20 games, the RL agents
have learned a stable policy compared to the hand-tuned policy, but the
learned policy bounded above the hand-tuned role assignment function. The
second experiments were done using a team size of five with the RL agents
against opponent teams. Figure (4(b)) shows the results of bins of 20 games
averaged among three trials. After 100 games, our RL agent increased the
chance of winning to 50%. This number does not increase more in the next
games. As Figures (4(a)) and (4(b)) show, the three and five agents per team
are able to increase the goal difference against both opponents. However, it
does not reach the performance of the manually tuned role selection. Similar
to Subsection (6.1), the amount of time spent for fine-tuning the hand-coded
role selection, these results are promising, and the outcome of the experiment
heavily depends on the underlying skills of the agents.

((a)) Three vs three agents.

((b)) Five vs five agents.

((c)) Seven vs seven agents.

Figure 4: Goal difference in games with (a) three; (b) five; and (c) seven
agents per team using Off-PAC algorithm.

The final experiments were done using a team size of seven with the RL agents
against opponent teams. Figure (4(c)) shows the results of bins of 20 games
averaged among two trials. Similar to Subsection (6.1), with seven agents per
team, the results in Figure (4(c)) show a bigger gap between RL and the hand-
tuned agent. However, using seven agents the goal difference is generally
decreased, since the defense is easily improved by increasing the number of
agents. Also the hand-tuned role selection results in a smaller goal
difference. Figure 4(c) shows an increase in the trend of winning games. As
mentioned earlier, only 200 games seem to be not sufficient to learn a good
policy. Even though the RL agents reach a positive goal difference, but it
stays below the hand-tuned role selection method. Within the given setting,
the RL agents have learned a moderately satisfactory policy. Whether the
learned policy is satisfactory for other teams needs to be further
investigated.

The RoboCup 3D soccer simulation is inherently a dynamic, and a stochastic
environment. There is an infinitesimal chance that a given situation (state)
may occur for many games. Therefore, it is paramount important that the
learning algorithms extract as much information as possible from the training
examples. We use the algorithms in the on-line incremental setting, and once
the experience is consumed it is discarded. Since, we learned from off-policy
experiences, we can save the tuples,
$(S_{t},A_{t},S_{t+1},r(S_{t+1}),\rho_{t},z(S_{t+1}))$, and learn the policy
off-line. The Greedy-GQ($\lambda$) learns a deterministic greedy policy. This
may not be suitable for complex and dynamic environments such as the RoboCup
3D soccer simulation environment. The Off-PAC algorithm is designed for
stochastic environment. The experiment shows that this algorithm needs careful
tuning of learning rates and feature selection, as evident from Figure (4(a))
after 160 games.

##  7 Conclusions

We have designed and experimented RL agents that learn to assign roles in
order to maximize expected future rewards. All the agents in the team ask the
question “What is my role in this formation in order to maximize future
rewards, while maintaining a completely different role from all teammates in
all time steps?”. This is a goal-oriented question. We use Greedy-
GQ($\lambda$) and Off-PAC to learn experientially grounded knowledge encoded
in GVFs. Dynamic role assignment function is abstracted from all other low-
level components such as walking engine, obstacle avoidance, object tracking
etc. If the role assignment function selects a passive role and assigns a
target location, the lower-layers handle this request. If the lower-layers
fail to comply to this request, for example being reactive, this feedback is
not provided to the role assignment function. If this information needs to be
included; it should become a part of the state representation, and the reward
signal should be modified accordingly. The target positions for passive roles
are created w.r.t. the absolute ball location and the rules imposed by the 3D
soccer simulation league. When the ball moves relatively quickly, the target
locations change more quickly. We have given positive rewards only for the
forward ball movements. In order to reinforce more agents within an area close
to the ball, we need to provide appropriate rewards. These are part of reward
shaping [12]. Reward shaping should be handled carefully as the agents may
learn sub-optimal policies not contributing to the overall goal.

The experimental evidences show that agents are learning competitive role
assignment functions for defending and attacking. We have to emphasize that
the behavior policy is $\epsilon$-greedy with a relatively small exploration
or slightly perturbed around the target policy. It is not a uniformly
distributed policy as used in [22]. The main reason for this decision is that
when an adversary is present with the intention of maximizing its objectives,
practically the learning agent may have to run for a long period to observe
positive samples. Therefore, we have used the off-policy Greedy-GQ($\lambda$)
and Off-PAC algorithms for learning goal-oriented GVFs within on-policy
control setting. Our hypothesis is that with the improvements of the
functionalities of lower-layers, the role assignment function would find
better policies for the given question and answer functions. Our next step is
to let the RL agent learn policies against other RoboCup 3D soccer simulation
league teams. Beside the role assignment, we also contributed with testing
off-policy learning in high-dimensional state spaces in a competitive
adversarial environment. We have conducted experiments with three, five, and
seven agents per team. The full game consists of eleven agents. The next step
is to extend learning to consider all agents, and to include methods that
select informative state variables and features.

## References

  * [1] Degris, T., Modayily, J.: Scaling-up Knowledge for a Cognizant Robot. In Notes of the AAAI Spring Symposium on Designing Intelligent Robots: Reintegrating AI (2012)
  * [2] Gabel, T., Lange, S., Lauer, M., Riedmiller, M.: Bridging the Gap: Learning in the Robocup Simulation and Midsize League. In: Proceedings of the 7th Portuguese Conference on Automatic Control (Controlo) (2006)
  * [3] Gurzoni, Jr., J.A., Tonidandel, F., Bianchi, R.A.C.: Market-Based Dynamic Task Allocation using Heuristically Accelerated Reinforcement Learning. In: Proceedings of the 15th Portugese Conference on Progress in Artificial Intelligence. pp. 365–376. EPIA’11, Springer-Verlag, Berlin, Heidelberg (2011)
  * [4] Kober, J., Bagnell, J.A.D., Peters, J.: Reinforcement Learning in Robotics: A Survey. International Journal of Robotics Research (July 2013)
  * [5] Köse, H., Tatladede, U., Mericli, C., Kaplan, K., Akan, H.L.: Q-Learning Based Market-Driven Multi-Agent Collaboration in Robot Soccer. In: Proceedings of the Turkish Symposium on Artificial Intelligence and Neural Networks (TAINN). pp. 219–228 (2004)
  * [6] MacAlpine, P., Urieli, D., Barrett, S., Kalyanakrishnan, S., Barrera, F., Lopez-Mobilia, A., Ştiurcă, N., Vu, V., Stone, P.: UT Austin Villa 2011: A Champion Agent in the RoboCup 3D Soccer Simulation Competition. In: Proceedings of 11th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2012) (June 2012)
  * [7] Maei, H.R.: Gradient Temporal-Difference Learning Algorithms. PhD Thesis, University of Alberta. (2011), phD Thesis
  * [8] Maei, H.R., Sutton, R.S.: GQ($\lambda$): A General Gradient Algorithm for Temporal-Difference Prediction Learning with Eligibility Traces. Proceedings of the 3rd Conference on Artificial General Intelligence (AGI-10) pp. 1–6 (2010)
  * [9] Maei, H.R., Szepesvári, C., Bhatnagar, S., Sutton, R.S.: Toward Off-Policy Learning Control with Function Approximation. In: Proceedings of the 27th International Conference on Machine Learning (ICML 2010). pp. 719–726 (2010)
  * [10] Modayil, J., White, A., Pilarski, P.M., Sutton, R.S.: Acquiring a Broad Range of Empirical Knowledge in Real Time by Temporal-Difference Learning. In: Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC). pp. 1903–1910. IEEE (2012)
  * [11] Modayil, J., White, A., Sutton, R.S.: Multi-timescale Nexting in a Reinforcement Learning Robot. In: From Animals to Animats 12 - 12th International Conference on Simulation of Adaptive Behavior (SAB). pp. 299–309 (2012)
  * [12] Ng, A.Y., Harada, D., Russell, S.J.: Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping. In: Proceedings of the Sixteenth International Conference on Machine Learning (ICML). pp. 278–287. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1999)
  * [13] Pilarski, P., Dawson, M., Degris, T., Carey, J., Sutton, R.: Dynamic Switching and Real-Time Machine Learning for Improved Human Control of Assistive Biomedical Robots. In: 4th IEEE RAS EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob). pp. 296 –302 (June 2012)
  * [14] Riedmiller, M., Gabel, T.: On Experiences in a Complex and Competitive Gaming Domain: Reinforcement Learning Meets RoboCup. In: Third IEEE Symposium on Computational Intelligence and Games. pp. 17–23. IEEE (2007)
  * [15] Riedmiller, M., Gabel, T., Hafner, R., Lange, S.: Reinforcement Learning for Robot Soccer. Autonomous Robots 27, 55–73 (July 2009)
  * [16] Seekircher, A., Abeyruwan, S., Visser, U.: Accurate Ball Tracking with Extended Kalman Filters as a Prerequisite for a High-Level Behavior with Reinforcement Learning. In: The 6th Workshop on Humanoid Soccer Robots at Humanoid Conference, Bled (Slovenia) (2011)
  * [17] Stoecker, J., Visser, U.: Roboviz: Programmable Visualization for Simulated Soccer. In: Röfer, T., Mayer, N.M., Savage, J., Saranli, U. (eds.) RoboCup. pp. 282–293. Lecture Notes in Computer Science, Springer (2011)
  * [18] Stone, P., Sutton, R.S., Kuhlmann, G.: Reinforcement Learning for RoboCup-Soccer Keepaway. Adaptive Behavior 13(3), 165–188 (2005)
  * [19] Stone, P., Veloso, M.: Layered Learning. In: Proceedings of the Eleventh European Conference on Machine Learning. pp. 369–381. Springer Verlag (1999)
  * [20] Stone, P., Veloso, M.: Task Decomposition, Dynamic Role Assignment, and Low-Bandwidth Communication for Real-Time Strategic Teamwork. Artificial Intelligence 110(2), 241–273 (June 1999)
  * [21] Sutton, R.S., Barto, A.G.: Reinforcement Learning: An Introduction. MIT Press (1998)
  * [22] Sutton, R.S., Modayil, J., Delp, M., Degris, T., Pilarski, P.M., White, A., Precup, D.: Horde: A Scalable Real-Time Architecture for Learning Knowledge from Unsupervised Sensorimotor Interaction. In: The 10th International Conference on Autonomous Agents and Multiagent Systems. pp. 761–768. AAMAS ’11, International Foundation for Autonomous Agents and Multiagent Systems (2011)
  * [23] Sutton, R.S., Szepesvári, C., Maei, H.R.: A Convergent O(N) Algorithm for Off-Policy Temporal-Difference Learning with Linear Function Approximation. In: Advances in Neural Information Processing Systems (NIPS). pp. 1609–1616. MIT Press (2008)
  * [24] Thomas Degris, Martha White, R.S.S.: Off-Policy Actor-Critic. In: Proceedings of the Twenty-Ninth International Conference on Machine Learning (ICML) (2012)
  * [25] White, A., Modayil, J., Sutton, R.: Scaling Life-Long Off-Policy Learning. In: International Conference on Development and Learning and Epigenetic Robotics (ICDL), 2012 IEEE. pp. 1–6 (2012)