# PROJECTIVE SURJECTIVITY OF QUADRATIC STOCHASTIC OPERATORS ON $L_{1}$ AND ITS
APPLICATION

Farrukh Mukhamedov∗ Farrukh Mukhamedov
Department of Mathematical Sciences
College of Science, The United Arab Emirates University
P.O. Box, 15551, Al Ain
Abu Dhabi, UAE<EMAIL_ADDRESS><EMAIL_ADDRESS>,  Otabek Khakimov
Otabek Khakimov
Department of Algebra and Its Applications
Institute of Mathematics
P.O. Box, 100125, Tashkent
Uzbekistan<EMAIL_ADDRESS>and  Ahmad Fadillah Embong  Ahmad Fadillah Embong
Department of Mathematical Sciences
Faculty of Science, Universiti Teknologi Malaysia
81310 Johor Bahru, Malaysia<EMAIL_ADDRESS>
###### Abstract.

A nonlinear Markov chain is a discrete time stochastic process whose
transitions depend on both the current state and the current distribution of
the process. The nonlinear Markov chain over a infinite state space can be
identified by a continuous mapping (the so-called nonlinear Markov operator)
defined on a set of all probability distributions (which is a simplex). In the
present paper, we consider a continuous analogue of the mentioned mapping
acting on $L^{1}$-spaces. Main aim of the current paper is to investigate
projective surjectivity of quadratic stochastic operators (QSO) acting on the
set of all probability measures. To prove the main result, we study the
surjectivity of infinite dimensional nonlinear Markov operators and apply them
to the projective surjectivity of a QSO. Furthermore, the obtained result has
been applied for the existence of positive solution of some Hammerstein
integral equations.
Mathematics Subject Classification: 45G10, 47H30, 47H25, 37A30, 47H60
Key words: quadratic stochastic operator; projective surjection; nonlinear
equation;

∗ Corresponding author

##  1\. Introduction

Recently nonlinear Markov chains become an interesting subject in many areas
of applied mathematics. These chains are discrete time stochastic processes
whose transitions, which are defined by stochastic hypermatrix
$\mathcal{P}=(P_{i_{1},\dots,i_{m},k})_{i_{1},\dots,i_{m},k\in I}$, where
$I\subset{\mathbb{N}}$ depend on both the current state and the current
distribution of the process [14]. These processes were introduced by McKean
[19] and have been extensively studied in the context of the nonlinear
Chapman- Kolmogorov equation [11] as well as the nonlinear Fokker-Planck
equation [9]. On the other hand, we stress that such types of chains are
generated by tensors (hypermatrices), therefore, this topic is closely related
to the geometric and algebraic structures of tensors which have been
systematically studied and has wide applications in scientific and engineering
communities. One of the intrinsic features of tensors is the concept of tensor
eigenvalues and eigenvectors which turns out to be much more complex than that
of the square matrix case (see for example, [8, 16, 17, 31]).

Let us denote

$S^{I}=\left\\{\textbf{x}=(x_{i})_{i\in I}\in\mathbb{R}^{I}\ :\ x_{i}\geq
0,\quad\sum\limits_{i\in I}x_{i}=1\right\\}.$

By means of $\mathcal{P}$ one defines an operator $V:S^{I}\to S^{I}$ by

(1.1) $(V({\mathbf{x}}))_{k}=\sum\limits_{i_{1},i_{2},\dots,i_{m}\in
I}P_{i_{1}i_{2}\dots i_{m},k}x_{i_{1}}x_{i_{2}}\dots x_{i_{m}},\ \ k\in I.$

This operator is called $m$-ordered polynomial stochastic operator (in short
$m$-ordered PSO). We note that PSO has a direct connection with non-linear
Markov operators which are intensively studied by many scientists (see [14]
for recent review). Therefore, it is crucial to study several properties of
this operator [8, 17, 30]. One of the important ones is its surjectivity. It
turn out that when the set $I$ is finite, in [24] we have established that the
sturjectivity of $V$ is equivalent to its orthogonal preserving property.

We notice that if the order of the operator is 2 (i.e. $m=2$), then the
nonlinear Markov operator $V$ given by (1.1) is called a discrete quadratic
stochastic operator (DQSO) which has many applications in population genetics
[13]. Note that such kind of operators is traced back to Bernstein’s work [7]
where they appeared from the problems of population genetics. We refer the
reader to [11, 18, 27] as the exposition of the recent achievements and open
problems in the theory of the QSO can be further researched. In [22, 23] the
surjectivity of DQSO and its relation with orthogonal preserving property of
$V$ have been investigated.

On the other hand, there has been much interest in recent years in self-
organizing search methods in the q.s.o. field. Recently, in [10] it has been
considered QSO on the set of $\sigma$-additive measures on $[0,1]$ (see also
[28]). In particular, points of the unit interval $[0,1]$ serve to code
(continuum valued) traits attributed to each individual from a considered
population. The main aim of the present paper is to investigate projective
surjectivity of such kind of continuum analogous of QSO. First, in Section 3,
we revise the results of [22], and in section 4, we apply them to the
projective surjectivity of QSO acting on the set of probability measures. We
notice that very particular cases of QSO have been studied (see for example,
[3, 5, 6, 10]). In the last section 5, a short application of the main result
to the existence of positive solutions nonlinear Hammerstein integral
equations is carried out. Certain Hammerstein integral equations associated
with finite dimensional DQSO has been investigated in [12]. The obtained
results are even new and opens new insight to this topic.

##  2\. Discrete Quadratic Stochastic Operators

In this section we give basic notations and some known results from the theory
of discrete quadratic stochastic operators.

Let $E$ be a subset of $\mathbb{N}$ such that $|E|\geq 2$. Denote

$S^{E}=\left\\{\textbf{x}=(x_{i})_{i\in E}\in\mathbb{R}^{E}\ :\ x_{i}\geq 0,\
\forall i\in E\quad\mbox{and}\ \sum\limits_{i\in E}x_{i}=1\right\\}.$

We notice that there is only two case for cardinality of $E$. So, in special
cases we write $S$ or $S^{d-1}$ instead of $S^{E}$ when $E$ is infinite or
$|E|=d$ respectively. In what follows, by ${\mathbf{e}}_{i}$ we denote the
standard basis in $S^{E}$, i.e. ${\mathbf{e}}_{i}=(\delta_{ik})_{k\in E}$
($i\in E$), where $\delta_{ij}$ is the Kronecker delta.

Let $V$ be a mapping on $S^{E}$ defined by

(2.1) $\displaystyle(V({\mathbf{x}}))_{k}=\sum\limits_{i,j\in
E}P_{ij,k}x_{i}x_{j},\ \ \ \ \forall k\in E,$

here $P_{ij,k}$ are hereditary coefficients which satisfy

(2.2) $\displaystyle P_{ij,k}\geq 0,\quad
P_{ij,k}=P_{ji,k},\quad\sum\limits_{k\in E}P_{ij,k}=1,\quad\forall i,j,k\in E$

One can see that $V$ maps $S^{E}$ into itself and $V$ is called Discrete
Quadratic Stochastic Operator (DQSO) [20].

By support of ${\mathbf{x}}=(x_{i})_{i\in E}\in S^{E}$ we mean a set
$supp({\mathbf{x}})=\left\\{i\in E\ :\ x_{i}\neq 0\right\\}$. A sequence
$\\{A_{k}\\}_{k\geq 1}$ of sets is called cover of a set $B$ if
$\bigcup\limits_{k=1}^{\infty}A_{k}=B$ and $A_{i}\cap A_{j}=\emptyset$ for
$i,j\in{\mathbb{N}}$ ($i\neq j$). Recall that two vectors
${\mathbf{x}},{\mathbf{y}}\in S^{E}$ are called orthogonal (denoted by
${\mathbf{x}}\perp{\mathbf{y}}$) if $supp({\mathbf{x}})\cap
supp({\mathbf{y}})=\emptyset$. If ${\mathbf{x}},{\mathbf{y}}\in S^{E}$, then
one can see that ${\mathbf{x}}\perp{\mathbf{y}}$ if and only if
${\mathbf{x}}\circ{\mathbf{y}}=0$. Here,
${\mathbf{x}}\circ{\mathbf{y}}=\sum_{i\in E}x_{i}y_{i}$.

######  Definition 2.1.

A DQSO $V$ given by (2.1) is called orthogonality preserving DQSO (OP DQSO) if
for any ${\mathbf{x}},{\mathbf{y}}\in S$ with ${\mathbf{x}}\perp{\mathbf{y}}$
one has $V({\mathbf{x}})\perp V({\mathbf{y}})$.

Recall [20, 21] that a DQSO $V:S^{E}\rightarrow S^{E}$ is called Volterra if
one has

(2.3) $\displaystyle P_{ij,k}=0\textmd{ if }k\notin\\{i,j\\},\ \ \forall
i,j,k\in E.$

######  Remark 2.2.

In [21] it was given an alternative definition Volterra operator in terms of
extremal elements of $S^{E}$.

One can check [20, 21] that a DQSO $V$ is Volterra if and only if one has

$(V({\mathbf{x}}))_{k}=x_{k}\left(1+\sum\limits_{i\in E}a_{ki}x_{i}\right),\ \
\ \ \ \forall k\in E,$

where $a_{ki}=2P_{ik,k}-1$ for all $i,k\in E$. One can see that
$a_{ki}=-a_{ik}$. This representation leads us to the following definitions.

######  Definition 2.3.

A DQSO $V:S^{E}\rightarrow S^{E}$ is called $\pi$-Volterra if there is a
permutation $\pi$ of $E$ such that $V$ has the following form

$(V({\mathbf{x}}))_{k}=x_{\pi(k)}\left(1+\sum\limits_{i\in
E}a_{\pi(k)i}x_{i}\right),\ \ \ \ \ \forall k\in E,$

where $a_{\pi(k)i}=2P_{i\pi(k),k}-1$, $a_{\pi(k)i}=-a_{i\pi(k)}$ for any
$i,k\in E$.

In [1, 28] it has been proved the following result.

######  Theorem 2.4.

Let $|E|=d$ and $V$ be a DQSO on $S^{d-1}$. Then the following statements are
equivalent:

  * (i)

$V$ is orthogonality preserving;

  * (ii)

$V$ is surjective;

  * (iii)

$V$ is $\pi$-Volterra QSO.

We emphasize that there is a big difference between finite and infinite
dimensional settings. It is known [22] that in the infinite dimensional
setting, some implication of Theorem 2.4 fails.

######  Theorem 2.5.

[22] Let $V$ be infinite dimensional DQSO such that
$V({\mathbf{e}}_{i})={\mathbf{e}}_{\pi(i)}$ for some permutation $\pi$ of
${\mathbb{N}}$. Then the following statements are equivalent:

  * (i)

$V$ is surjective;

  * (ii)

$V$ is orthogonality preserving;

  * (iii)

$V$ is $\pi-$Volterra QSO.

######  Theorem 2.6.

[22] Let $V$ be a surjective and OP infinite dimensional DQSO. Then $V$ is a
$\pi$-Volterra for some permutation $\pi:\mathbb{N}\to\mathbb{N}$.

##  3\. Surjectivity of DQSO

In this section, we are going to provide a sufficient condition for the
surjectivity of infinite dimensional DQSOs.

Let $E$ be a subset of $\mathbb{N}$. We denote

${\bf B}_{1}^{+}=\left\\{\textbf{x}=(x_{i})_{i\in E}\in\mathbb{R}^{E}\ :\
x_{i}\geq 0,\ \ \ \forall i\in E\ \mbox{and }\sum_{j\in E}x_{j}\leq
1\right\\}.$

We can extend each DQSO $V$ from $S^{E}$ to ${\bf B}_{1}^{+}$ by the same
formula (2.1). Then the following crucial result is true.

######  Lemma 3.1.

Let $V$ be a DQSO on ${\bf B}_{1}^{+}$. Then one has

$V(S^{E})\subset S^{E},\ \ \ \ V({\bf B}_{1}^{+}\setminus S^{E})\subset{\bf
B}_{1}^{+}\setminus S^{E}.$

###### Proof.

For a given $r\geq 0$ we denote

$S_{r}^{E}=\left\\{\textbf{x}=(x_{i})_{i\in E}\in\mathbb{R}^{E}\ :\ x_{i}\geq
0,\ \ \ \forall i\in E\ \mbox{and }\sum_{j\in E}x_{j}=r\right\\}.$

We notice that $S_{1}^{E}=S^{E}$. Then it is obvious that

${\bf B}_{1}^{+}=\bigcup_{r\in[0,1]}S_{r}^{E}.$

Take any $r\in[0,1]$ and an arbitrary ${\mathbf{x}}\in S_{r}^{E}$. One can
check that $(V({\mathbf{x}}))_{k}\geq 0$ for all $k\geq 1$. Furthermore, using
(2.2) we get

$\sum\limits_{k=1}^{\infty}(V({\mathbf{x}}))_{k}=\sum\limits_{i,j\in
E}x_{i}x_{j}=r^{2}.$

From this we find $V({\mathbf{x}})\in S_{r^{2}}^{E}$. Hence, we infer that

$V(S^{E})\subset S^{E}\ \ \mbox{and}\ \ V({\bf B}_{1}^{+}\setminus
S^{E})\subset{\bf B}_{1}^{+}\setminus S^{E},$

which completes the proof. ∎

######  Theorem 3.2.

Let $V$ be a DQSO on ${\bf B}_{1}^{+}$. Then the following statements are
equivalent:

  * (i)

$V$ is surjective on ${\bf B}_{1}^{+}$;

  * (ii)

$V$ is surjective on $S^{E}$;

  * (iii)

$V$ is surjective on ${\bf B}_{1}^{+}\setminus S^{E}$.

###### Proof.

Thanks to Lemma 3.1 the implication $(i)\Rightarrow(ii)$ is obvious.

$(ii)\Rightarrow(iii)$. Assume that $V$ is surjective on $S^{E}$. Let
$r\in(0,1)$, then we define an operator $T_{r}:S_{r}^{E}\to S^{E}$ as follows
$T_{r}({\mathbf{x}})=r^{-1}{\mathbf{x}}$ for all ${\mathbf{x}}\in S_{r}^{E}$.
We notice that $T_{r}$ is a bijection. Then keeping in mind
$V(rS^{E})=r^{2}V(S^{E})$ and $r^{2}S^{E}=S_{r^{2}}^{E}$, one gets

$V(S_{r}^{E})=V(rT_{r}(S^{E}_{r}))=V(rS^{E})=S_{r^{2}}^{E}.$

From the arbitrariness of $r$ and $V({\bf 0})={\bf 0}$, we find

$V\bigg{(}\bigcup_{r\in[0,1)}S_{r}^{E}\bigg{)}=\bigcup_{r\in[0,1)}S_{r}^{E},$

which means $V({\bf B}_{1}^{+}\setminus S^{E})={\bf B}_{1}^{+}\setminus
S^{E}$.

$(iii)\Rightarrow(i)$. One can see that $V(S_{r}^{E})\subset S_{r^{2}}^{E}$
for any $r\in[0,1)$. Due to the surjectivity of $V$ on
$\bigcup_{r\in[0,1)}S_{r}^{E}$, we conclude

$V(S_{r}^{E})=S_{r^{2}}^{E},\ \ \ \ \forall r\in[0,1).$

Then, for any $r>0$, one has

$V(S^{E})=V(r^{-1}S_{r}^{E})=r^{-2}V(S_{r}^{E})=r^{-2}S_{r^{2}}^{E}=S^{E}.$

The last one together with $V({\bf B}_{1}^{+}\setminus S^{E})={\bf
B}_{1}^{+}\setminus S^{E}$ implies that $V({\bf B}_{1}^{+})={\bf B}_{1}^{+}$.
This completes the proof. ∎

######  Remark 3.3.

Thanks to Theorem 3.2, to establish the surjectivity of DQSO $V$ on ${\bf
B}_{1}^{+}$ it is enough to consider it only on $S^{E}$.

Let us recall the Cauchy Product which has the following form:

(3.1)
$\left(\sum\limits_{i=1}^{\infty}x_{i}\right)^{m}=\sum\limits_{i_{1},\dots,i_{m}\in{\mathbb{N}}}x_{i_{1}}\cdots
x_{i_{m}},\ \ \ \forall m\in\mathbb{N},$

where $\sum_{i=1}^{\infty}x_{i}<\infty$.

######  Theorem 3.4.

Let $V$ be a surjective DQSO on $S$. Then there exists a sequence
$\\{j_{k}\\}_{k\geq 1}\subset\mathbb{N}$ such that $P_{j_{k}j_{k},k}=1$ for
all $k\in{\mathbb{N}}$.

###### Proof.

Let us denote

$I_{k}=\left\\{j\in\mathbb{N}:\ P_{jj,k}=1\right\\}$

First of all, we show that surjectivity of $V$ implies $I_{k}\neq\emptyset$
for any $k\in\mathbb{N}$. Thanks to the surjectivity of $V$, for every
$k\in\mathbb{N}$ there is an ${\mathbf{x}}^{(k)}\in S$ such that

(3.2) $\displaystyle V({\mathbf{x}}^{(k)})={\mathbf{e}}_{k}.$

Now, we consider two cases.

Case 1: Let $\left|supp({\mathbf{x}}^{(k)})\right|=1$. Then one can find a
number $j_{k}\in{\mathbb{N}}$ such that $supp({\mathbf{x}}^{(k)})=\\{j_{k}\\}$
and

$(V({\mathbf{x}}^{(k)}))_{k}=P_{j_{k}j_{k},k}x_{j_{k}}^{2}=1,$

which yields that $P_{j_{k}j_{k},k}=1$. Hence we get $j_{k}\in I_{k}$.

Case 2: Let $|supp({\mathbf{x}}^{(k)})|>1$ and $A:=supp({\mathbf{x}}^{(k)})$.
From (3.2) we get

(3.3) $\displaystyle(V({\mathbf{x}}^{(k)}))_{k}=\sum\limits_{i,j\in
A}P_{ij,k}x_{i}x_{j}=1.$

Now suppose that there are $\bar{i},\bar{j}\in A$ such that
$P_{\bar{i}\bar{j},k}<1$. One has

$\displaystyle(V({\mathbf{x}}^{(k)}))_{k}$ $\displaystyle=$
$\displaystyle\sum\limits_{i,j\in A}P_{ij,k}x_{i}x_{j}$ $\displaystyle\leq$
$\displaystyle\sum\limits_{\\{i,j\\}\subset
A\setminus\\{\bar{i},\bar{j}\\}}x_{i}x_{j}+P_{\bar{i}\bar{j},k}x_{\bar{i}}x_{\bar{j}}$
$\displaystyle<$ $\displaystyle\sum\limits_{i,j\in A}x_{i}x_{j}.$

Using (3.1) the right side of the last one we get

$(V({\mathbf{x}}^{(k)}))_{k}<1,$

which contradicts to (3.2). So, we conclude that

$P_{ij,k}=1,\ \ \ \ \ \ \forall i,j\in{A}.$

In particular we have $P_{ii,k}=1$ for any $i\in A$. This means that $A\subset
I_{k}$. From this we immediately get $I_{k}\neq\emptyset$ for any $k\geq 1$.

Thus, we know that $I_{k}\neq\emptyset$ for every $k\geq 1$. Now, we can
define a sequence $\\{j_{k}\\}_{k\geq 1}$ by $j_{k}=\inf I_{k}$. Due to
construction of $I_{k}$, we get $P_{j_{k}j_{k},k}=1$ for all $k\in\mathbb{N}$.
This completes the proof. ∎

Next result gives a sufficient condition for the surjectivity of DQSO.

######  Theorem 3.5.

Let $V$ be a DQSO on $S$. Assume that there exists a sequence
$\\{j_{n}\\}_{n\geq 1}\subset\mathbb{N}$ such that

(3.4) $P_{j_{n}j_{m},k}=0,\ \ \ \ \ \forall k\notin\\{n,m\\}.$

Then $V$ is surjective.

###### Proof.

Let $I:=\\{j_{n}\\}_{n\geq 1}$ be a subset of $\mathbb{N}$ for which (3.4) is
satisfied. Now, we define a new cubic matrix
$\tilde{\mathcal{P}}=(\tilde{P}_{ij,k})_{i,j,k\geq 1}$ as follows

$\tilde{P}_{ij,k}=\left\\{\begin{array}[]{ll}P_{\alpha(i)\alpha(j),k},&k\in\\{i,j\\},\\\
0,&\mbox{otherwise},\end{array}\right.$

where $\alpha(k)=j_{k}$ for all $k\geq 1$. We consider a DQSO $\tilde{V}$ is
given by

$(\tilde{V}({\mathbf{x}}))_{k}=\sum_{i,j\geq 1}\tilde{P}_{ij,k}x_{i}x_{j},\ \
\ \forall{\mathbf{x}}\in S.$

Due to the construction of the cubic matrix $\tilde{\mathcal{P}}$ one
concludes that $\tilde{V}$ is a Volterra DQSO. Then, thanks to Theorem 2.5,
the operator $\tilde{V}$ is a surjection on $S$.

Let us denote $S_{I}=\\{{\mathbf{x}}\in S:supp({\mathbf{x}})\subset I\\}$.
Then it is obvious that operator $T:S\to S_{I}$ given by
$T({\mathbf{x}})_{k}=x_{\alpha(k)}$ is a bijection. Furthermore, we have
$\tilde{V}=V\circ T$. Keeping in mind that $\tilde{V}$ is surjective and $T$
is bijection, we infer that $V=\tilde{V}\circ T^{-1}$ is surjective, which
completes the proof. ∎

We stress that unfortunately, we are not able to prove that (3.4) is a
necessary for the surjectivity of $V$. However, all construced examples show
it is true. So, we may formulate the following conjecture.

######  Conjecture 3.6.

Let $V$ be a surjective DQSO on $S$. Then there exists a sequence
$\\{j_{n}\\}_{n\geq 1}\subset\mathbb{N}$ such that (3.4) holds.

##  4\. Quadratic Stochastic operators on $L^{1}$ and associated DQSO

In this section, we consider QSO on $L^{1}$ and construct associated DQSO. Let
$(X,\mathcal{F},\lambda)$ be a measurable space with a $\sigma$-finite measure
$\lambda$. By $L^{1}(X,\mathcal{F},\lambda)$ we define an usual $L^{1}$ space.
We notice that $L^{1}$ can be identified with the set of all measures (signed
ones) absolutely continuous w.r.t. $\lambda$. Namely, for every non negative
$f\in L^{1}$ we can define a measure $\mu_{f}$ by

$\mu_{f}(A)=\int_{A}fd\lambda,\ \ \ \ \ \forall A\in\Omega.$

Therefore, in what follows, we may identify measures with functions and visa
versa.

By $S(X)$ we denote the set of all probability measures on $X$ which are
absolutely continuous w.r.t. $\lambda$.

Recall that a collection of measurable sets $\mathcal{B}=\\{B_{k}\\}_{k\geq
1}$ is called partition of $X$ (w.r.t. $\lambda$) if it satisfies

  1. (1)

$X=\bigcup_{k\geq 1}B_{k}$;

  2. (2)

$B_{i}\cap B_{j}=\emptyset$ for all $i\neq j$;

  3. (3)

$0<\lambda(B_{k})<\infty$ for every $k\geq 1$.

We denote by $\mathcal{P}(X)$ the set of all partitions of $X$. Since
$\lambda$ is $\sigma$-finite, we infer that $\mathcal{P}(X)\neq\emptyset$.

Let us take $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}\in\mathcal{P}(X)$. For any
${\mathbf{x}}\in\ell^{1}$ we define a measure $\mu_{\mathbf{x}}^{\mathcal{B}}$
on $\mathcal{F}$ as follows:

(4.1)
$\mu_{\mathbf{x}}^{\mathcal{B}}(A)=\sum_{k=1}^{\infty}\frac{x_{k}}{\lambda(B_{k})}\lambda(A\cap
B_{k}),\ \ \ \ \ \ \forall A\in\mathcal{F}.$

We notice that $\mu_{\mathbf{x}}^{\mathcal{B}}$ is not probability measure
when $\|{\mathbf{x}}\|_{\ell^{1}}\neq 1$. A natural question arises: what kind
of ${\mathbf{x}}\in\ell^{1}$ is it true $\mu_{\mathbf{x}}^{\mathcal{B}}\in
S(X)$?

Recall that $S=\\{{\mathbf{x}}\in\ell^{1}:x_{i}\geq 0,\ \forall i\geq 1\
\mbox{and }\parallel{\mathbf{x}}\parallel_{\ell^{1}}=1\\}$. Then the following
result holds.

######  Lemma 4.1.

Let $\mu_{\mathbf{x}}^{\mathcal{B}}$ be a measure given by (4.1). Then
$\mu_{\mathbf{x}}^{\mathcal{B}}\in S(X)$ iff ${\mathbf{x}}\in S$.

###### Proof.

Let us assume that $\mu_{\mathbf{x}}^{\mathcal{B}}\in S(X)$. We have
$\mu_{\mathbf{x}}^{\mathcal{B}}(B_{k})=x_{k}$ for every $k\in\mathbb{N}$. It
yields that $0\leq x_{k}\leq 1$ for every $k\in\mathbb{N}$. On the other hand,
we obtain

$1=\mu_{\mathbf{x}}^{\mathcal{B}}(X)=\mu_{\mathbf{x}}^{\mathcal{B}}(\bigcup_{k\geq
1}B_{k})=\sum_{k\geq 1}x_{k},$

which implies that ${\mathbf{x}}\in S$.

Now we suppose that ${\mathbf{x}}\in S$. Then we get
$\mu_{\mathbf{x}}^{\mathcal{B}}(X)=\sum_{k\geq 1}x_{k}=1$. This means that
$\mu_{\mathbf{x}}^{\mathcal{B}}$ is a probability measure on $X$. Moreover, it
is obvious that measure given by (4.1) is absolutely continuous w.r.t.
$\lambda$. Hence, we infer that $\mu_{\mathbf{x}}\in S(X)$. ∎

######  Remark 4.2.

For a given partition $\mathcal{B}$ of $X$, thanks to Lemma 4.1 there exists a
one-to-one correspondence between $S$ and
$M(X,\mathcal{B}):=\\{\mu_{\mathbf{x}}^{\mathcal{B}}\in S(X):\
{\mathbf{x}}\in\ell^{1}\\}$. In other words, every $\mu\in M(X,\mathcal{B})$
is uniquely defined by the values $\mu(B_{k})$, $k\geq 1$.

######  Proposition 4.3.

Let $\mathcal{B}\in\mathcal{P}(X)$. Then $M(X,\mathcal{B})$ is a convex and
closed set w.r.t. strong convergence. Moreover, $M(X,\mathcal{B})$ is not
compact w.r.t. weak convergence.

###### Proof.

One can see that $T:S\to M(X,\mathcal{B})$ given by
$T{\mathbf{x}}=\mu_{{\mathbf{x}}}^{\mathcal{B}}$ is a bijection. Then any
sequence on $M(X,\mathcal{B})$ is defined by a sequence
$\\{{\mathbf{x}}^{(n)}\\}_{n\geq 1}\subset S$. It is obvious that if
${\mathbf{x}}^{(n)}\stackrel{{\scriptstyle\left\|\cdot\right\|_{\ell_{1}}}}{{\longrightarrow}}{\mathbf{x}}$
then
$\lim\limits_{n\to\infty}\mu_{{\mathbf{x}}^{(n)}}^{\mathcal{B}}(A)=\mu_{{\mathbf{x}}}^{\mathcal{B}}(A)$
for all $A\in\mathcal{F}$.

Let us pick a sequence $\\{\mu_{{\mathbf{x}}^{(n)}}^{\mathcal{B}}\\}_{n\geq
1}\subset M(X,\mathcal{B})$. Assume that
$\mu(A)=\lim\limits_{n\to\infty}\mu_{{\mathbf{x}}^{(n)}}^{\mathcal{B}}(A)$ for
every $A\in\mathcal{F}$. Then we have $\mu\in S(X)$. On the other hand, we
obtain

$\mu(B_{k})=\lim_{n\to\infty}x_{k}^{(n)},\ \ \ \ \ \forall k\geq 1.$

The last one together with $\mu(X)=1$ implies that ${\mathbf{x}}^{(n)}$
converges on $S$ w.r.t. $\ell^{1}$-norm. Hence, we conclude that $T$ is a
homeomorphism. Then due to closedness and convexity of $S$ we infer that
$M(X,\mathcal{B})$ has the same topological properties. We notice that $S$ is
not compact w.r.t. $\ell^{1}$-norm. Consequently, $M(X,\mathcal{B})$ is not a
compact w.r.t. weak convergence. ∎

######  Lemma 4.4.

Let $\tilde{S}(X)=\left\\{\mu_{f}\in S(X):f\ \mbox{is a simple function on}\
L^{1}\right\\}$. Then

(4.2) $\tilde{S}(X)=\bigcup_{\mathcal{B}\in\mathcal{P}(X)}M(X,\mathcal{B}).$

###### Proof.

It is clear that
$\bigcup\limits_{\mathcal{B}\in\mathcal{P}(X)}M(X,\mathcal{B})\subset\tilde{S}(X)$.
Indeed, for any $\mu_{\mathbf{x}}^{\mathcal{B}}$ we define a simple function

$f_{\mu_{\mathbf{x}}^{\mathcal{B}}}(u)=\frac{x_{k}}{\lambda(B_{k})},\ \ \ \ \
\forall u\in B_{k},\ \forall k\geq 1,$

which satisfies

$\mu_{\mathbf{x}}^{\mathcal{B}}(A)=\int_{A}f_{\mu_{\mathbf{x}}^{\mathcal{B}}}d\lambda,\
\ \ \ \ \forall A\in\mathcal{F}.$

Now, we take an arbitrary $\mu_{f}\in\tilde{S}(X)$. Then for any $i\geq 1$ we
have a measurable set $A_{i}=\left\\{u\in X:f(u)=y_{i}\right\\}$. One may
assume that $\lambda(A_{i})>0$ for every $i\geq 1$. We notice that if
$\lambda(A_{i})<\infty$ for each $i\in\mathbb{N}$ then
$\mathcal{A}=\\{A_{i}\\}_{i\geq 1}$ is a partition of $X$ and
$\mu_{f}=\mu_{\mathbf{x}}^{\mathcal{A}}$, where
${\mathbf{x}}=(\mu_{f}(A_{1}),\mu_{f}(A_{2}),\dots)\in S$.

If $\lambda(A_{j})=\infty$ for some $j\geq 1$ then one has $y_{j}=0$
(otherwise $f$ is not integrable). Hence, $\mu_{f}(A_{j})=0$. So, without loss
of generality we may assume that $y_{1}=0$, $\lambda(A_{1})=\infty$ and
$y_{i}>0$, $\lambda(A_{i})<\infty$ for any $i>1$. Pick any partition
$\\{B_{k}\\}_{k\geq 1}$ of $X$ and define a new partition
$\tilde{\mathcal{B}}$ of $X$ as follows:

$\tilde{B}_{k}=\left\\{\begin{array}[]{ll}A_{1}\cap
B_{\frac{k+1}{2}},&\mbox{if }k\ \mbox{is odd},\\\ A_{\frac{k+2}{2}},&\mbox{if
}k\ \mbox{is even}.\end{array}\right.$

Then $\mu_{f}=\mu_{\tilde{{\mathbf{y}}}}^{\tilde{\mathcal{B}}}$, where
coordinates of $\tilde{{\mathbf{y}}}\in S$ are given by

$\tilde{y}_{k}=\left\\{\begin{array}[]{ll}0,&\mbox{if }k\ \mbox{is odd},\\\
\mu_{f}(A_{\frac{k+2}{2}}),&\mbox{if }k\ \mbox{is even}.\end{array}\right.$

The arbitrariness of $\mu_{f}$ yields that
$\tilde{S}(X)\subset\bigcup_{\mathcal{B}\in\mathcal{P}(X)}M(X,\mathcal{B})$.
The last one together with
$\bigcup_{\mathcal{B}\in\mathcal{P}(X)}M(X,\mathcal{B})\subset\tilde{S}(X)$
implies (4.2). ∎

Due to the density argument, from Lemma 4.4 we immediately infer the following
result.

######  Proposition 4.5.

One has
$S(X)=\overline{\bigcup_{\mathcal{B}\in\mathcal{P}(X)}M(X,\mathcal{B})}$, here
the closure in sense of weak convergence.

###  4.1. Projective surjectivity of QSO

Now, we consider a measurable function $P:X\times X\times\mathcal{F}\to[0;1]$
which satisfies the following conditions:

(4.3) $P(u,v,A)=P(v,u,A),\ \ \ \ \ \forall u,v\in X,\ \ \forall
A\in\mathcal{F},$ (4.4) $P(u,v,\cdot)\in S(X),\ \ \ \ \ \forall u,v\in X.$

This function is called transition kernel, and defines a Quadratic Stochastic
Operator (in short QSO) by

(4.5) $(\mathcal{V}\mu)(A)=\int_{X}\int_{X}P(u,v,A)d\mu(u)d\mu(v),\ \ \ \
\forall\mu\in S(X),\ \forall A\in\mathcal{F}.$

Clearly, $\mathcal{V}$ maps $S(X)$ into itself.

######  Definition 4.6.

A QSO $\mathcal{V}$ given by (4.5) is called projective surjection if it is
surjective on $M(X,\mathcal{B})$ for some $\mathcal{B}\in\mathcal{P}(X)$.

Now we are going to find projective surjection QSOs. For a given QSO
$\mathcal{V}$ we associate DQSO (this DQSO depends on partition
$\\{B_{k}\\}_{k\geq 1}$) $V_{\mathcal{B}}:S\to S$ by

(4.6) $\left(V_{\mathcal{B}}({\mathbf{x}})\right)_{k}=\sum_{i,j\geq
1}P_{ij,k}^{\mathcal{B}}x_{i}x_{j},\ \ \ \ \ \ \ \forall k\geq 1,$

where

(4.7)
$P_{ij,k}^{\mathcal{B}}=\frac{1}{\lambda(B_{i})\lambda(B_{j})}\int_{B_{i}}\int_{B_{j}}P(u,v,B_{k})d\lambda(u)d\lambda(v),\
\ \ \ \ \ \ \forall k\geq 1.$

######  Lemma 4.7.

Let $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}\in\mathcal{P}(X)$. Then for every
${\mathbf{x}}\in S$ it holds

$(\mathcal{V}\mu_{{\mathbf{x}}}^{\mathcal{B}})(B_{k})=(V_{\mathcal{B}}({\mathbf{x}}))_{k},\
\ \ \ \ \forall k\geq 1.$

###### Proof.

Let $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}$ be a partition of $X$ and
${\mathbf{x}}\in S$. Then for any $k\geq 1$ we have

$\displaystyle(\mathcal{V}\mu_{\mathbf{x}}^{\mathcal{B}})(B_{k})$
$\displaystyle=$
$\displaystyle\int_{X}\int_{X}P(u,v,B_{k})d\mu_{\mathbf{x}}^{\mathcal{B}}(u)d\mu_{\mathbf{x}}^{\mathcal{B}}(v)$
$\displaystyle=$ $\displaystyle\sum_{i,j\geq
1}\frac{x_{i}x_{j}}{\lambda(B_{i})\lambda(B_{j})}\int_{B_{i}}\int_{B_{j}}P(u,v,B_{k})d\lambda(u)d\lambda(v)$
$\displaystyle=$ $\displaystyle\sum_{i,j\geq
1}P_{ij,k}^{\mathcal{B}}x_{i}x_{j}$ $\displaystyle=$
$\displaystyle(V_{\mathcal{B}}({\mathbf{x}}))_{k}.$

∎

######  Proposition 4.8.

Let $\mathcal{V}$ be a QSO given by (4.5) and $\mathcal{B}\in\mathcal{P}(X)$.
If $P(u,v,\cdot)\in M(X,\mathcal{B})$ for every $(u,v)\in X^{2}$ then
$\mathcal{V}(M(X,\mathcal{B}))\subset M(X,\mathcal{B})$.

###### Proof.

Let $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}$ be partition of $X$. Assume that
$P(u,v,\cdot)\in M(X,\mathcal{B})$ for every $(u,v)\in X^{2}$. Then for
arbitrary $(u,v)\in X^{2}$ we obtain

(4.8) $P(u,v,A_{k})=\frac{\lambda(A_{k})}{\lambda(B_{k})}P(u,v,B_{k}),\ \ \ \
\forall A_{k}\subset B_{k},\ \forall k\geq 1.$

For any ${\mathbf{x}}\in S$ we define ${\mathbf{y}}\in S$ as follows
$y_{k}=(V_{\mathcal{B}}({\mathbf{x}}))_{k}$, $k\geq 1$. The due to Lemma 4.7
we get

$(\mathcal{V}\mu_{{\mathbf{x}}}^{\mathcal{B}})(B_{k})=y_{k},\ \ \ \ \ \forall
k\geq 1.$

Let us establish that
$\mathcal{V}\mu_{\mathbf{x}}^{\mathcal{B}}=\mu_{\mathbf{y}}^{\mathcal{B}}$.
Take an arbitrary measurable $A\in\mathcal{F}$ and denote $A_{k}=A\cap B_{k}$
for every $k\geq 1$. Keeping in mind (4.8) one gets

$\displaystyle(\mathcal{V}\mu_{\mathbf{x}}^{\mathcal{B}})(A)$ $\displaystyle=$
$\displaystyle\int_{X}\int_{X}P(u,v,A)d\mu_{\mathbf{x}}^{\mathcal{B}}(u)d\mu_{\mathbf{x}}^{\mathcal{B}}(v)$
$\displaystyle=$ $\displaystyle\sum_{i,j\geq
1}\frac{x_{i}x_{j}}{\lambda(B_{i})\lambda(B_{j})}\sum_{k\geq
1}\int_{B_{i}}\int_{B_{j}}P(u,v,A_{k})d\lambda(u)d\lambda(v)$ $\displaystyle=$
$\displaystyle\sum_{i,j\geq
1}\frac{x_{i}x_{j}}{\lambda(B_{i})\lambda(B_{j})}\sum_{k\geq
1}\frac{\lambda(A_{k})}{\lambda(B_{k})}\int_{B_{i}}\int_{B_{j}}P(u,v,B_{k})d\lambda(u)d\lambda(v)$
$\displaystyle=$ $\displaystyle\sum_{k\geq
1}\frac{\lambda(A_{k})}{\lambda(B_{k})}y_{k}$ $\displaystyle=$
$\displaystyle\mu_{\mathbf{y}}^{\mathcal{B}}(A),$

which yields
$\mathcal{V}\mu_{\mathbf{x}}^{\mathcal{B}}=\mu_{\mathbf{y}}^{\mathcal{B}}$.
The arbitrariness of ${\mathbf{x}}\in S$ implies
$\mathcal{V}(M(X,\mathcal{B}))\subset M(X,\mathcal{B})$. The proof is
complete. ∎

Now we are going to find sufficiently conditions of projective surjectivity of
QSO given by (4.5).

######  Theorem 4.9.

Let $(X,\mathcal{F},\lambda)$ be a measurable space with $\sigma$-finite
measure $\lambda$ and $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}\in\mathcal{P}(X)$.
Assume that the transition kernel $P$ of QSO $\mathcal{V}$ satisfies the
followings conditions:

  * (i)

$P(u,v,\cdot)\in M(X,\mathcal{B}),\ \ \ \forall(u,v)\in X^{2}$;

  * (ii)

there exists a sequence $\\{j_{n}\\}_{n\geq 1}\subset\mathbb{N}$ such that

(4.9) $P(u,v,B_{k})=\frac{\lambda(B_{k}\cap
B_{n})}{2\lambda(B_{n})}+\frac{\lambda(B_{k}\cap B_{m})}{2\lambda(B_{m})},\ \
\ \ \forall(u,v)\in B_{j_{n}}\times B_{j_{m}},\ \forall k\in\mathbb{N}.$

Then $\mathcal{V}$ is projective surjection.

###### Proof.

Assume that all conditions of the theorem hold. From the condition (i),
according to Proposition 4.8 we have $\mathcal{V}:M(X,\mathcal{B})\to
M(X,\mathcal{B})$.

Now, let us show that $\mathcal{V}(M(X,\mathcal{B}))=M(X,\mathcal{B})$. For
any triple $(n,m,k)\in\mathbb{N}^{3}$ from (4.9) after simple calculations, we
get

$P_{j_{n}j_{m},k}^{\mathcal{B}}=P_{j_{m}j_{n},k}^{\mathcal{B}}=\left\\{\begin{array}[]{ll}1,&\mbox{if
}\ n=m=k;\\\\[2.84526pt] \frac{1}{2},&\mbox{if }\ n=k\neq m;\\\\[2.84526pt]
0,&\mbox{if }\ k\notin\\{n,m\\}.\end{array}\right.$

Then, due to Theorem 3.5, the corresponding DQSO $V_{\mathcal{B}}$ is a
surjection. Hence, for any ${\mathbf{y}}\in S$ we can find ${\mathbf{x}}\in S$
such that $V_{\mathcal{B}}({\mathbf{x}})={\mathbf{y}}$. Consequently, by Lemma
4.7 one has

$(\mathcal{V}\mu_{\mathbf{x}}^{\mathcal{B}})(B_{k})=\mu_{\mathbf{y}}^{\mathcal{B}}(B_{k}),\
\ \ \ \ \forall k\in\mathbb{N}.$

From the last one, keeping in mind
$\mathcal{V}\mu_{\mathbf{x}}^{\mathcal{B}}\in M(X,\mathcal{B})$ thanks to
Remark 4.2 we infer

$\mathcal{V}\mu_{\mathbf{x}}^{\mathcal{B}}=\mu_{\mathbf{y}}^{\mathcal{B}}.$

Finally, the arbitrariness of ${\mathbf{y}}\in S$ yields that
$\mathcal{V}(M(X,\mathcal{B}))=M(X,\mathcal{B})$. This completes the proof. ∎

######  Remark 4.10.

We notice that the conclusion of the last theorem will be true if (4.9) holds
almost everywhere in $B_{j_{n}}\times B_{j_{m}}$.

For any measurable set $A\subset X$ we define

$\mathcal{E}_{A}=\left\\{(x,y)\in(X\setminus A)^{2}:\ P(x,y,A)\neq
0\right\\}.$

######  Theorem 4.11.

Let $(X,\mathcal{F},\lambda)$ be a measurable space with $\sigma$-finite
measure $\lambda$ and $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}\in\mathcal{P}(X)$.
Then there is only one QSO $\mathcal{V}$ whose transition kernel satisfies the
followings:

  * (i)

$P(u,v,\cdot)\in M(X,\mathcal{B}),\ \ \ \forall(u,v)\in X^{2}$;

  * (ii)

$\lambda(\mathcal{E}_{B_{k}})=0$ for every $k\in\mathbb{N}$.

Moreover, $\mathcal{V}$ is projective surjection.

###### Proof.

From (i) we have $P(u,v,\cdot)=\mu_{{\mathbf{x}}(u,v)}^{\mathcal{B}}$ for any
$u,v\in X$. Without loss of generality we may replace the condition (ii) to
the following one

$\mu_{{\mathbf{x}}(u,v)}^{\mathcal{B}}(B_{k})=0,\ \ \ \ \
\forall(u,v)\in(X\setminus B_{k})^{2},\ \forall k\in\mathbb{N}.$

Then for any $(u,v)\in B_{n}\times B_{m}$ we obtain

$\mu_{{\mathbf{x}}(u,v)}^{\mathcal{B}}(B_{n})+\mu_{{\mathbf{x}}(u,v)}^{\mathcal{B}}(B_{m})=1.$

Keeping in mind ${\mathbf{x}}(u,v)={\mathbf{x}}(v,u)$, from the last one we
have ${\mathbf{x}}(u,v)=\frac{1}{2}{\bf e}_{n}+\frac{1}{2}{\bf e}_{m}$ for
every $(u,v)\in B_{n}\times B_{m}$. Hence, the transitional kernel has the
following form

(4.10) $P(u,v,A)=\frac{\lambda(A\cap
B_{n})}{2\lambda(B_{n})}+\frac{\lambda(A\cap B_{m})}{2\lambda(B_{m})},\ \ \ \
\forall(u,v)\in B_{n}\times B_{m},\ \forall A\in\mathcal{F}.$

We notice that (4.10) implies (4.9) for the sequence $\\{n\\}_{n\geq 1}$.
Then, due to Theorem 4.9 we conclude that $\mathcal{V}$ is projective
surjection. The proof is complete. ∎

######  Corollary 4.12.

Let $(X,\mathcal{F},\lambda)$ be a measurable space with $\sigma$-finite
measure $\lambda$ and $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}\in\mathcal{P}(X)$.
There is only one CQSO $\mathcal{V}$ whose transition kernel satisfies the
followings:

  * (i)

$P(u,v,\cdot)\in M(X,\mathcal{B}),\ \ \ \forall u,v\in X$;

  * (ii)

$\lambda(\mathcal{E}_{A})=0$ for every $A\in\mathcal{F}$.

Moreover, $\mathcal{V}$ is projective surjection.

##  5\. Application

In this section we are give a direct application of the projective
surjectivity of QSO to the existence of positive solutions of certain
nonlinear integral equations.

Let $(X,\mathcal{F},\lambda)$ be a measurable space with a $\sigma$-finite
measure $\lambda$. Let us consider the following nonlinear Hammerstein
integral equation:

(5.1) $\int_{X}\int_{X}K(u,v,t)x(u)x(v)d\lambda(u)d\lambda(v)=\varphi(t),$

where $K$ is some positive kernel and $\varphi\in L^{1}_{+}$ is a given
function.

We note that this type of equation appeared in several problems of
astrophysics, mechanics, and biology. Here in the equation,
$K:X^{3}\to{\mathbb{R}}$ and $\varphi:X\to{\mathbb{R}}$ are given functions,
and $x:\Omega\to{\mathbb{R}}$ is an unknown one. Generally speaking, in order
to solve the nonlinear Hammerstein integral equation (5.1) over some functions
space, one should impose on some constraints on $K(\cdot,\cdot,\cdot)$. There
are several works where the existence of solutions the above given equation
have been carried out by means of contraction methods (see [2, 4, 15, 29]). In
this section, we are going to another approach for the existence of positive
solutions of (5.1). In what follows, we consider the equation (5.1) over
$L^{1}$-spaces.

Multiplying (5.1) by a function $g$ from $L^{\infty}$ and integrating it, we
obtain

(5.2)
$\int_{X}\int_{X}\int_{X}g(t)K(u,v,t)x(u)x(v)d\lambda(u)d\lambda(v)d\lambda(t)=\int_{X}g(t)\varphi(t)d\lambda(t).$

We stress that the arbitrariness of $g$ implies that (5.2) and (5.1) are
equivalent.

Now, assume that there is a transition kernel $P$ such that

$\int_{X}\int_{X}\int_{X}g(t)K(u,v,t)x(u)x(v)d\lambda(u)d\lambda(v)d\lambda(t)=\int_{X}\int_{X}\int_{X}g(t)P(u,v,dt)x(u)x(v)d\lambda(u)d\lambda(v)$

for all $x\in L^{1}$ and $g\in L^{\infty}$.

Then (5.2) is reduced to

$\int_{X}\int_{X}\int_{X}g(t)P(u,v,dt)x(u)x(v)d\lambda(u)d\lambda(v)=\int_{X}g(t)\varphi(t)d\lambda(t).$

Now, taking $g=\chi_{A}$, $A\in\mathcal{F}$, we arrive at

$(\mathcal{V}\mu_{x})(A)=\mu_{\varphi}(A),$

where, as before, $\mu_{x}(A)=\int_{A}x(u)d\lambda(u)$. Assume that

$\int_{X}\varphi d\lambda=1.$

Hence, the integral equation (5.1) is reduced to the equation

(5.3) $\mathcal{V}\mu=\mu_{\varphi},$

where $\mu\in S(X)$.

Hence, the following result is true.

######  Theorem 5.1.

Let $(X,\mathcal{F},\lambda)$ be a measurable space with $\sigma$-finite
measure $\lambda$ and $\mathcal{B}=\\{B_{k}\\}_{k\geq 1}\in\mathcal{P}(X)$.
Assume that QSO $\mathcal{V}$ is a projective surjection on
$M(X,\mathcal{B})$. Then for any $\mu_{\varphi}\in M(X,\mathcal{B})$ the
equation (5.3) has a solution in $M(X,\mathcal{B})$.

## Acknowledgments

The present work is supported by the UAEU UPAR Grant No. G00003447. The first
named author (A.F.E.) acknowledges the Ministry of Higher Education (MOHE) and
Research Management Centre-UTM, Universiti Teknologi Malaysia (UTM) for the
financial support through the research grant (vote number 17J93).

## References

  * [1] Akin H., Mukhamedov F. Orthogonality preserving infinite dimensional quadratic stochastic operators, AIP Conf. Proc. 1676(2015), 020008.
  * [2] Atkinson K.E., A survey of numerical methods for solving nonlinear integral equations, J. Integral Equations 4(1992), 15-46.
  * [3] Badocha M., Bartoszek W., Quadratic stochastic operators on Banach lattices, Positivity, 22(2018), 477-492.
  * [4] Banas J., Martinon A., Monotonic solutions of a quadratic integral equation of Volterra type, Comput. Math. Appl. 47 (2004) 271–279.
  * [5] Bartoszek K., Domsta J., Pulka M., Weak Stability of Centred Quadratic Stochastic Operators, Bulletin Malays. Math. Sci. Soc. 42(2019), 1813-1830.
  * [6] Bartoszek K., Pulka M., Prevalence problem in the set of quadratic stochastic operators acting on $L^{1}$, Bulletin Malays. Math. Sci. Soc. 41(2018), 159–173.
  * [7] Bernstein S. N. The solution of a mathematical problem concerning the theory of heredity, _Ann. Math. Statistics_ 13(1924) 53–61.
  * [8] Cui L.B., Li W., Ng M.K., Birkhoff–von Neumann theorem for multistochastic tensors, SIAM Journal on Matrix Analysis and Applications 35 (2014), 956–973.
  * [9] Frank T.D.,  Nonlinear Fokker-Planck Equations Fundamentals and Applications, Springer-Verlag, Berlin, 2005.
  * [10] Ganikhodjaev N., Saburov M., Muhitdinov R. On Lebesgue nonlinear transformations, Bull. Korean Math. Soc. 54(2017), 607-618.
  * [11] Ganikhodzhaev R., Mukhamedov F., Rozikov U. Quadratic stochastic operators and processes: results and open problems, _Infin. Dimens. Anal. Quant. Prob. Relat. Top._ 14(2011) 279–335.
  * [12] Ganikhodzhaev R., Mukhamedov F., Saburov M. Elliptic quadratic operator equations, Acta Appl. Math. 159(2019), no. 1, 29–74.
  * [13] Jamilov U. U. Quadratic stochastic operators corresponding to graphs, Lobachevskii J. Math 34(2013), 148–151
  * [14] Kolokoltsov V. N., Nonlinear Markov Processes and Kinetic Equations, Cambridge Univ. Press, New York, 2010\.
  * [15] Krasnosel’skii M.A., Topological methods in the theory of nonlinear integral equations, Pergamon (1964).
  * [16] Li W., Ng M.K. , On the limiting probability distribution of a transition probability tensor,  Linear And Multilinear Algebra 62 (2014), 362–385.
  * [17] Li C.-K., Zhang S., Stationary probability vectors of higher-order Markov chains, Linear Algebra Appl. 473 (2015) 114–125.
  * [18] Lyubich Yu. I. Mathematical structures in population genetics, Berlin, Springer-Verlag, 1992.
  * [19] McKean H.P. A class of Markov processess associated with nonlinear parabolic equations, Proc. Natl. Acad. Sci. USA 56(1966), 1907–1911.
  * [20] Mukhamedov F. M. On infinite dimensional Volterra operators, Russian Math. Surveys 55(2000), 1161–1162
  * [21] Mukhamedov F., Akin H. and Temir S. On infinite dimensional quadratic Volterra operators, J. Math. Anal. Appl. 310(2005), 533–556.
  * [22] Mukhamedov F., Khakimov O.N., Embong A.F. On Surjective second order non-linear Markov operators and associated nonlinear integral equations, Positivity, 22(2018), 1445–1459.
  * [23] Mukhamedov F., Khakimov O., Embong A.F., Solvability of nonlinear integral equations and surjectivity of non-linear Markov operators, Math. Methods in Appl. Sci. 43 (2020), no. 15, 9102-9118.
  * [24] Mukhamedov F., Embong A.F. On non-linear Markov operators: surjectivity vs orthogonal preserving property, Lin. Multilin. Alg. 66(2018), 2183–2190.
  * [25] Mukhamedov F., Embong A.F. Infinite dimensional orthogonality preserving nonlinear Markov operators, Lin. Multilin. Alg. DOI: 10.1080/03081087.2019.1607241
  * [26] Mukhamedov F., Embong A.F., Rosli A. Orthogonal preserving and surjective cubic stochastic operators, Ann Funct. Anal. 8 (2017), 490– 501.
  * [27] Mukhamedov F. and Ganikhodjaev N. Quantum Quadratic Operators and Processes, Berlin, Springer, 2015
  * [28] Mukhamedov F. and Taha M. H., On Volterra and orthoganality preserving quadratic stochastic operators, Miskloc Math. Notes 17(2016) 457–470.
  * [29] Some B., Some recent numerical methods for solving nonlinear Hammerstein integral equations, Math and Computer Modelling, 18:9, 55–62 (1993).
  * [30] Qi L., Eigenvalues of a real supersymmetric tensor, J. Symbolic Comput., 40(2005), 1302–1324.
  * [31] Qi L., Luo Z.  Tensor Analysis: Spectral Theory and Special Tensors, SIAM, 2017.
  * [32] Raftery A., A model of high-order markov chains, J. Royal Statist. Soc. 47(1985), 528–539.


* [Kor93] N. Korevaar, Upper bounds for eigenvalues of conformal metrics, J. Differential Geom., 37, 1993, 1, 73–93
  * [Kra25] Krahn, E., Über eine von Rayleigh formulierte Minimaleigenschaft des Kreises, Math. Ann., 94, 1925, 1, 97–100
  * [LP19] P. Laurain, R. Petrides, Existence of min-max free boundary disks realizing the width of a manifold, Advances in Mathematics 352, 2019, 326–371.
  * [LY82] P. Li, S.T. Yau, A new conformal invariant and its applications to the Willmore conjecture and the first eigenvalue of compact surfaces, Invent.. Math. 69, 1982, 269–291.
  * [Nad96] N. Nadirashvili, Berger’s isoperimetric problem and minimal immersions of surfaces, Geom. Func. Anal. 6, 1996, 877-897.
  * [NS15] Nikolai Nadirashvili and Yannick Sire. Conformal spectrum and harmonic maps. Moscow J. of maths. Volume 15, 2014, 1, 123–140
  * [Pet14] R. Petrides, Maximization of the second conformal eigenvalue of spheres, Proc. Amer. Math. Soc., 142, 2014, 7 , 2385–2394
  * [Pet15] R. Petrides, On a rigidity result for the first conformal eigenvalue of the Laplacian, J. Spectr. Theory, 5, 2015, no.1, 227–234
  * [Pet14a] R. Petrides, Existence and regularity of maximal metrics for the first Laplace eigenvalue on surfaces, Geom. Funct. Anal. 24, 2014, 1336–1376.
  * [Pet18] R. Petrides, On the existence of metrics which maximize Laplace eigenvalues on surfaces, Int. Math. Res. Not., 14, 2018, 4261–4355.
  * [Pet19] R. Petrides, Maximizing Steklov eigenvalues on surfaces, J. Differential Geom. Volume 113, 2019, no.1, 95-188.
  * [Pet21] R. Petrides, Extremal metrices for combinations of Laplace eigenvalues and minimal surfaces into ellipsoids, submitted
  * [Pet22] R. Petrides, Shape optimization for combinations of Steklov eigenvalues on Riemannian surfaces, submitted
  * [Pet22a] R. Petrides, Maximizing one Laplace eigenvalue on n-dimensional manifolds, submitted
  * [PT22] R. Petrides, D. Tewodrose, Subdifferentials and critical points of eigenvalue functionals
  * [Riv08] T. Rivière, Conservation laws for conformally invariant variational problems, Inventiones Mathematicae, 168, 2007, 1–22.
  * [Sze54] G. Szegö, Inequalities for certain eigenvalues of a membrane of given area, J. Rational Mech. Anal., 3, 1954, 343–356
  * [Str08] M. Struwe, Variational Methods: Applications to Nonlinear Partial Differential Equations and Hamiltonian Systems, Springer Berlin, Heidelberg, Edition 4, 2008, XX, 302
  * [Wei56] H.F. Weinberger, An isoperimetric inequality for the $N$-dimensional free membrane problem, J. Rational Mech. Anal., 5,1956, 633–636
  * [YY80] P.C. Yang, S.T. Yau, Eigenvalues of the Laplacian of compact Riemannian surfaces and minimal submanifolds, Ann. Scuola Norm. Sup. Pisa Cl. Sci. (4) 7, 1980, no.1, 53–63.


# Multi-Environment Pretraining Enables Transfer to Action Limited Datasets

David Venuto $1,2$, Mengjiao Yang$3,4$, Pieter Abbeel$4$,
Doina Precup$1,2,5$, Igor Mordatch$3$, Ofir Nachum$3$
$1$McGill University, $2$Mila, $3$Google Brain, $4$University of California,
Berkeley, $5$DeepMind
<EMAIL_ADDRESS>Work done as a Student Researcher at Google Brain

###### Abstract

Using massive datasets to train large-scale models has emerged as a dominant
approach for broad generalization in natural language and vision applications.
In reinforcement learning, however, a key challenge is that available data of
sequential decision making is often not annotated with actions - for example,
videos of game-play are much more available than sequences of frames paired
with their logged game controls. We propose to circumvent this challenge by
combining large but sparsely-annotated datasets from a _target_ environment of
interest with fully-annotated datasets from various other _source_
environments. Our method, Action Limited PreTraining (ALPT), leverages the
generalization capabilities of inverse dynamics modelling (IDM) to label
missing action data in the target environment. We show that utilizing even one
additional environment dataset of labelled data during IDM pretraining gives
rise to substantial improvements in generating action labels for unannotated
sequences. We evaluate our method on benchmark game-playing environments and
show that we can significantly improve game performance and generalization
capability compared to other approaches, using annotated datasets equivalent
to only $12$ minutes of gameplay. Highlighting the power of IDM, we show that
these benefits remain even when target and source environments share no common
actions.

##  1 Introduction

The training of large-scale models on large and diverse data has become a
standard approach in natural language and computer vision applications (Devlin
et al., 2019; Brown et al., 2020; Mahajan et al., 2018; Zhai et al., 2021).
Recently, a number of works have shown that a similar approach can be applied
to tasks more often tackled by reinforcement learning (RL), such as robotics
and game-playing. For example, Reed et al. (2022) suggest combining large
datasets of expert behavior from a variety of RL domains in order to train a
single generalist agent, while Lee et al. (2022) demonstrate a similar result
but using non-expert (offline RL) data from a suite of Atari game-playing
environments and using a decision transformer (DT) sequence modeling objective
(Chen et al., 2021b).

Applying large-scale training necessarily relies on the ability to gather
sufficiently large and diverse datasets. For RL domains, this can be a
challenge, as the most easily available data – for example, videos of a human
playing a video game or a human completing a predefined task – often does not
contain _labelled_ actions, i.e., game controls or robot joint controls. We
call such datasets _action limited_ , because little or none of the dataset is
annotated with action information. Transferring the success of approaches like
DT to such tasks is therefore bottlenecked by the ability to acquire action
labels, which can be expensive and time-consuming (Zolna et al., 2020).

Some recent works have explored approaches to mitigate the issue of action
limited datasets. For example, Video PreTraining (VPT) (Baker et al., 2022)
proposes gathering a small amount ($2$k hours of video) of labeled data
manually which is used to train an inverse dynamics model (IDM) (Nguyen-Tuong
et al., 2008); the IDM is then used to provide action labels on a much larger
video-only dataset ($70$k hours). This method is shown to achieve human level
performance in Minecraft. It has also been demonstrated that some agents can
learn directly from videos without any action labels (Seo et al., 2022).

While VPT shows promising results, it still requires over $2$k hours of
manually-labelled data; thus, a similar amount of expensive labelling is
potentially necessary to extend VPT to other environments. In this paper, we
propose an orthogonal but related approach to VPT: leveraging a large set of
labeled data from various _source_ domains to learn an agent policy on a
limited action dataset of a _target_ evaluation environment. To tackle this
setting, we propose Action Limited Pretraining (ALPT), which relies on the
hypothesis that shared structures between environments can be exploited by
non-causal (i.e., bidirectional) transformer IDMs. This allows us to look at
both past and future frames to infer actions. In many experimental settings,
the dynamics are far simpler than multi-faceted human behavior in the same
setting. It has been suggested that IDMs are therefore more data efficient and
this has been empirically shown (Baker et al., 2022). ALPT thus uses the
multi-environment source datasets as pretraining for an IDM, which is then
finetuned on the action-limited data of the target environment in order to
provide labels for the unlabelled target data, which is then used for training
a DT agent.

Through various experiments and ablations, we demonstrate that leveraging the
generalization capabilities of IDMs is critical to the success of ALPT, as
opposed to, for example, pretraining the DT model alone on the multi-
environment datasets or training the IDM only on the target environment. On a
benchmark game-playing environment, we show that ALPT yields as much as $5$x
improvement in performance, with as little as $10k$ labelled samples required
(i.e., $0.01\%$ of the original labels), derived from only $12$ minutes of
labelled game play (Ye et al., 2021). We show that these benefits even hold
when the source and target environments use distinct action spaces; i.e., the
environments share similar states but _no_ common actions, further
demonstrating the power of IDM pretraining.

While ALPT is, algorithmically, a straightforward application of existing
offline RL approaches, our results provide a new perspective on large-scale
training for RL. Namely, our results suggest that the most efficient path to
large-scale RL methods may be via generalist inverse dynamics modelling paired
with specialized agent finetuning, instead of generalist agent training alone.

Figure 1: The dynamics model pretraining procedure of ALPT using the source
set of environments along with the limited action target environment dataset.

##  2 Related Work

In this section, we briefly review relevant works in multi-task RL, meta-
learning for RL, semi-supervised learning, and transfer learning.

Multi-Task RL. It is commonly assumed that similar tasks share similar
structure and properties (Caruana, 1997; Ruder, 2017; Zhang et al., 2014;
Radford et al., 2019a). Many multi-task RL works leverage this assumption by
learning a shared low-dimensional representation across all tasks
(Calandriello et al., 2014; Borsa et al., 2016; D’Eramo et al., 2020). These
methods have also been extended to tasks where the action space does not align
completely (Bräm et al., 2020). Other methods assume a universal dynamics
model when the reward structure is shared but dynamics are not (Zhang et al.,
2021a). Multi-task RL has generally relied on a task identifier (ID) to
provide contextual information, but recent methods have explored using
additional side information available in the task meta-data to establish a
richer context (Sodhani et al., 2021). ALPT can be seen as multi-task RL,
given that we train both the sequence model and IDM using multiple different
environments, but we do not explicitly model context information or have
access to task IDs.

Meta RL. Meta-learning is a set of approaches for _learning to learn_ which
leverages a set of meta-training tasks (Schmidhuber, 1987; Bengio et al.,
1991), from which an agent can learn either parts of the learning algorithms
(eg how to tune the learning rate) or the entire algorithm (Lacombe et al.,
2021; Kalousis, 2002). In this setting, meta-learning can be used to learn
policies (Duan et al., 2017; Finn et al., 2017) or dynamics models (Clavera et
al., 2019). A distribution of tasks is assumed to be available for sampling,
in order to provide additional contextual information to the policy. One such
method models contextual information as probabilistic context variables which
condition the policy (Rakelly et al., 2019). This method has been shown to
learn from only a handful of trajectories. Meta-training can be used to learn
policies offline, while using online interaction to correct for distribution
shift, without requiring any rewards in the online data (Pong et al., 2022).
These methods are commonly used to train on a source set of tasks, like ALPT,
but usually require task labels. Meta-training tasks need to be hand-selected,
and the results are highly dependent on the quality of that process.

Semi-supervised learning. _Semi-supervised_ learning uses both labelled and
unlabelled data to improve supervised learning performance (Zhu et al., 2009).
It is especially useful when a limited amount of labelles data is given and
additional labels are difficult to acquire, unlabelled data is plentiful.
Early methods of this type infer unknown labels using a classifier trained on
the labeled data (Zhu & Ghahramani, 2002). Other methods rely on additional
structural side information to regularize supervised objectives (Szummer &
Jaakkola, 2001), such as the time scale of a Markov random walk over a
representation of the data. Many methods, especially those using deep
learning, combine supervised and unsupervised learning objectives (Rasmus et
al., 2015). More recent methods use generative models and approximate Bayesian
inference to fill in missing labels (Kingma et al., 2014). The problem of
semi-supervised learning is especially relevant in RL, where large datasets of
experience containing action descriptions or rewards may hard to acquire, eg.
through manual annotation of videos or running robotic experiments. By using
an inverse dynamics model, ALPT applies semi-supervised learning to label
actions in a large dataset of experience frames, given only limited labeled
action data.

Transfer Learning and Zero-shot RL. Policies learned by RL in one domain can
have limited capability to generalize to new settings (Oh et al., 2016). The
most difficult problem is zero-shot RL, where the agent must generalize at
evaluation time to a new environment that was not seen in training, without
acquiring any new data. Transfer learning (Taylor & Stone, 2009) tackles a
subset of generalization problems where the agent can access interactions from
a related environment or task during training. This prior experience in other
environments is leveraged to improve learning in novel environments. Transfer
learning has been applied across both environments (Mordatch et al., 2016;
Tzeng et al., 2020) and tasks (Rusu et al., 2016; Parisotto et al., 2016). It
has also been examined in hard exploration games, using imitation learning
from human-generated video data (Aytar et al., 2018). ALPT can be seen as
tackling the transfer learning problem, with limited action data from the
target environment and providing pseudo-labels for actions. Notably, we
consider the under-explored scenario where the action space is not completely
shared between the training and test environments.

Offline RL. Offline RL describes a setting that during learning, the agent has
access to only a fixed dataset of experience. When function approximation
becomes necessary in environments with large, complex state spaces, online RL
algorithms are extremely sensitive to the dataset distribution. It has been
shown that even when the logged data is collected with a single behavior
policy, standard online algorithms fail (Fujimoto et al., 2019). This has been
hypothesized to be due erroneous generalization of the state action function.
One such class of methods applies random ensembles of Q-value function targets
(Agarwal et al., 2020a). Other works suggest regularizing the agent policy to
the behavior policy (Zhang et al., 2021b). In offline RL, errors arise when
needed to bootstrap the value function and algorithms must apply strong
regularizations on both learned policy and value function to achieve stable
performance (Wu et al., 2020; Kumar et al., 2019; Zhang et al., 2021b; Nachum
et al., 2019). We use decision transformers (DT) (Chen et al., 2021b) as the
backbone for ALPT, and these have been shown to be successful in learning
generalizable agents from logged data in a diverse set of environments (Lee et
al., 2022).

##  3 Background

In this section, we review the standard offline RL setting and the use of
decision transformers (DT) as a sequence modelling objective for offline RL.
We then define the setting of multi-environment offline RL with action-limited
data, which is our focus.

###  3.1 Offline Reinforcement Learning

We consider an agent acting within a Markov decision process (MDP) defined by
$\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}\rangle$, where
$\mathcal{S}$ is the set of states, $\mathcal{A}$ is the set of actions,
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\rightarrow\mathrm{Dist}(\mathcal{S})$
is the transition probability kernel and
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow[0,1]$ is the scalar
reward function.

In offline RL, the agent is given a dataset of episodes, i.e., sequences of
states, actions, and rewards collected by unknown policies interacting with
the environment:

$\langle\dots,\mathbf{s_{t}},a_{t},r_{t},\dots\rangle.$ (1)

The objective is typically to use this dataset in order to learn a conditional
action distribution, $P_{\theta}(a_{t}|\mathbf{s}_{\leq t},a_{<t},r_{<t})$,
that maximizes the expectation of the total return, $G_{t}=\sum_{k\geq
0}r_{t+k}$ when used to interact with the environment from which the training
episodes were generated.

###  3.2 Offline RL as Sequence Modeling

Decision transformer (DT) (Chen et al., 2021a) is an approach to offline RL
which formulates this problem as sequence modeling, and then uses transformer-
based architectures to solve it. For this purpose, the episodes in the offline
dataset are augmented with the returns associated with each step:

$\tau=\langle\dots,\mathbf{s_{t}},a_{t},r_{t},G_{t},\dots\rangle.$ (2)

This sequence is tokenized and passed to a causal transformer $P_{\theta}$,
which predicts both returns and actions using a cross-entropy loss. Thus, the
learning objective for $\theta$ is:

$J(\theta)=\mathbb{E}_{\tau}\left[\sum_{t}-\log
P_{\theta}(G_{t}|\mathbf{s}_{\leq t},a_{<t},r_{<t})-\log
P_{\theta}(a_{t}|\mathbf{s}_{\leq t},a_{<t},r_{<t},G_{t})\right].$ (3)

During inference, at each timestep $t$, after observing $\mathbf{s}_{t}$, DT
uses the predicted return distribution $P_{\theta}(G_{t}|\mathbf{s}_{\leq
t},a_{<t},r_{<t})$ to choose an optimistic estimate $\hat{G}_{t}$ of return,
before using $P_{\theta}(a_{t}|\mathbf{s}_{\leq t},a_{<t},r_{<t},\hat{G}_{t})$
to select an action $\hat{a}_{t}$ (see Lee et al. (2022) for details).

###  3.3 Multi-Environment and Action Limited Datasets

Our goal is to use pretraining on a set of environments where labelled data is
plentiful, in order to do well on a target environment where only limited
action-labelled data is available. Therefore, the offline RL setting we
consider includes multiple environments and action-limited datasets, as we
detail below.

We consider a set of $n$ _source_ environments, defined by a set of MDPs:
$E=\\{\mathcal{M}_{1},\dots,\mathcal{M}_{n}\\}$, and a single _target_
environment $\mathcal{M}_{\star}$. For each source environment
$\mathcal{M}_{d}$, we have an offline dataset of episodes generated from
$\mathcal{M}_{d}$, denoted by
$\mathcal{D}_{d}=\left\\{\tau:=\langle\dots,\mathbf{s}_{t},a_{t},r_{t},\dots\rangle\right\\}$,
fully labelled with actions. For the target environment, the agent has access
to a small labelled dataset from $\mathcal{M}_{\star}$, denoted as
$\mathcal{D}^{+}_{\star}=\left\\{\tau:=\langle\dots,\mathbf{s}_{t},a_{t},r_{t},\dots\rangle\right\\}$,
and a large dataset without action labels,
$\mathcal{D}^{-}_{\star}=\left\\{\tau:=\langle\dots,\mathbf{s}_{t},r_{t},\dots\rangle\right\\}$.

##  4 Action Limited Pretraining (ALPT)

We now describe our proposed approach to offline RL in multi-environment and
action limited settings. ALPT relies upon an inverse dynamics model (IDM)
which uses the combined labelled data in order to learn a representation that
generalizes well to the limited action data from the target environment. The
predicted labels of the IDM on the unlabelled portion of the target
environment dataset are then used for training a sequence model parameterized
as a decision transformer (DT). We elaborate on this procedure below and
summarize the full algorithm in Table 1.

###  4.1 Inverse Dynamics Modeling

Our inverse dynamics model (IDM) is a bidirectional transformer trained to
predict actions from an action-unlabelled sub-trajectory of an episode. The
training objective for learning an IDM $P_{\beta}$ is

$J(\beta)=\mathbb{E}_{\tau}\left[\sum_{t}\sum_{i=0}^{k-1}-\log
P_{\beta}(a_{t+i}|\mathbf{s}_{t},\dots,\mathbf{s}_{t+k})\right],$ (4)

where $k$ is the length of training sub-trajectories. In our experiments, we
use $k=5$ and parameterize $P_{\beta}$ using the GPT-2 transformer
architecture (Radford et al., 2019b), modified to be bidirectional by changing
the attention mask.

###  4.2 Multi-Environment Pretraining and Finetuning

ALPT is composed of a two-stage pretraining and finetuning process.

During pretraining, we use the combined labelled datasets for all source
environments combined with the labelled portion of the target environment
dataset:
$\left(\bigcup_{d=1}^{n}\mathcal{D}_{d}\right)\cup\mathcal{D}^{+}_{\star}$, to
train the IDM $P_{\beta}$. Concurrently, we also train the DT $P_{\theta}$ on
the combined labelled and unlabelled datasets for all source environments
combined with the target environment datasets, by using the IDM to provide
action labels on the unlabelled portion $\mathcal{D}^{-}_{\star}$. The DT
training dataset is therefore:
$\left(\bigcup_{d=1}^{n}\mathcal{D}_{d}\right)\cup\mathcal{D}^{+}_{\star}\cup\mathcal{D}^{-}_{\star}$.

During finetuning, we simultaneously train both the IDM and DT exclusively on
the target environment dataset. We train the IDM on the labelled portion
$\mathcal{D}^{+}_{\star}$. We train DT on the full action limited dataset
$\mathcal{D}^{+}_{\star}\cup\mathcal{D}^{-}_{\star}$ by using the IDM to
provide action labels on the unlabelled portion $\mathcal{D}^{-}_{\star}$.

Finally, during evaluation we use the trained DT agent to select actions in
the target environment $\mathcal{M}_{\star}$, following the same protocol
described in Section 3.2.

##  5 Experiments

Table 1: A summary of ALPT. Step | Procedure
---|---
Pretraining | Train IDM on all labelled data: $\left(\bigcup_{d=1}^{n}\mathcal{D}_{d}\right)\cup\mathcal{D}^{+}_{\star}$.
| Train DT on all data:
$\left(\bigcup_{d=1}^{n}\mathcal{D}_{d}\right)\cup\mathcal{D}^{+}_{\star}\cup\mathcal{D}^{-}_{\star}$,
with IDM
| providing action labels on $\mathcal{D}^{-}_{\star}$
Finetuning | Train IDM on labelled data in target environment dataset: $\mathcal{D}^{+}_{\star}$.
| Train DT on all data in target environment dataset:
$\mathcal{D}^{+}_{\star}\cup\mathcal{D}^{-}_{\star}$,
| with IDM providing action labels on $\mathcal{D}^{-}_{\star}$
Evaluation | Use trained DT agent to interact with target environment $\mathcal{M}_{\star}$.

We evaluate ALPT on a multi-game Atari setup similar to Lee et al. (2022). Our
findings are three-fold: (1) ALPT, when pretrained on multiple source games,
demonstrates significant benefits on the target game with limited action
labels; (2) ALPT maintains its significant benefits even when pretrained on
just a single source game with a disjoint action space, (3) we demonstrate
similar benefits on maze navigation tasks.

###  5.1 Experimental Procedure

#### Architecture and Training.

Our architecture and training protocol follow the multi-game Atari setting
outlined in Lee et al. (2022). Specifically, we use a transformer with 6
layers of 8 heads each and hidden size 512. The rest of the architecture and
training hyperparameters remain unchanged for experiments on Atari. For the
Maze navigation experiments, we modify the original hyperparameters to use a
batch size of $256$ and a weight decay of $5\times 10^{-5}$. During pre-
training, we train the DT and IDM for $1M$ frames.

#### Datasets

As in Lee et al. (2022), we use the standard offline RL Atari datasets from RL
Unplugged (Gulcehre et al., 2020). Each game’s dataset consists of 100M
environment steps of training a DQN agent (Agarwal et al., 2020b). For the
source games, we use this dataset in its entirety. For the target game, we
derive an action-limited dataset by keeping the action labels for randomly
sampled sequences consisting of a total 10k transitions (0.01% of the original
dataset, equivalent to 12 minutes of gameplay) and removing action labels in
the remainder of the dataset. For the maze navigation experiments, we generate
the data ourselves. The offline datasets for each maze configuration contain
$500$ trajectories with a length of $500$ steps or until the goal state is
reached. They are generated using an optimal policy for each maze, with an
$\epsilon$-greedy exploration rate of $0.5$ to increase data diversity.

###  5.2 Baseline Methods

We detail the methods that we compare ALPT to below.

Single-game variants. To evaluate the benefit of multi- versus single-
environment training, we assess the performance of training either DT alone or
DT and IDM simultaneously on the target game. When training DT alone (DT1), we
train it only on the $10$k subset of data that is labelled, while when
training DT and IDM simultaneously (DT1-IDM) we use the IDM to provide action
labels on the unlabelled portion of the data.

Multi-game DT variants. To assess the need for IDM versus training on DT
alone, we evaluate a multi-game baseline (DT5) composed of DT alone. For this
baseline, we pretrain DT on all labelled datasets combined from both the
source and target environments before finetuning the DT model on the $10$k
labelled portion of the target game.

Return prediction DT variants. As an alternative way for DT to leverage the
unlabelled portion of the target environment dataset, we evaluate a baseline
(DT5-RET) that uses the unlabelled portion for training its return prediction.
The model still undergoes a pretraining and finetuning stage, first
pretraining on all available data and then finetuning only on data from the
target game.

We give a summary of the baseline methods as well as ALPT in Table 2. We also
present results of an additional variant of ALPT in which only the IDM is
pretrained (rather than both the IDM and DT) in Appendix A.

Table 2: A summary of the baseline methods. Method | Training Games | IDM
---|---|---
DT-1 | $1$ | $\times$
DT-5 | $5$ | $\times$
DT-1-IDM | $1$ | ✓
DT-5-RET | $1$ | $\times$
ALPT-$X$ |  $X$ if specified, otherwise $5$ | ✓

###  5.3 How does ALPT perform compared to the baselines?

We focus our first set of multi-game pretraining experiments on 5 Atari games:
$\\{$_Asterix, Breakout, SpaceInvaders, Freeway, Seaquest_ $\\}$. This subset
of games is selected due to having a similar shared game structure and access
to high-quality and diverse pretraining data. We evaluate each choice of
target game in this setting, i.e., for each game we evaluate using it as the
target game while the remaining $4$ games comprise the source environments.

We compare our pretraining regime (ALPT) with the single-game variant and
standard DT baselines in Figure 2. We see that pretraining ALPT on the source
games results in substantial downstream performance improvements. We show that
there are relatively minimal performance improvements when pretraining on
datasets that do not include any non-target environments (DT$1$-IDM).
Utilizing ALPT results in improvements up to $\approx 500\%$ higher than the
single-game training regime. The performance difference is especially stark in
_Breakout_ and _Seaquest_. Additionally, we show that performance is not
recovered under pretraining only the sequence model (DT$5$) on the action rich
environments, indicating that most generalization benefits are occurring due
to the IDM pretraining. We also show that performance using DT1 and DT5-RET is
poor, highlighting the need for explicit re-labelling to achieve good
performance during sequence model finetuning.

We also encourage the reader to look to Appendix A for a comparison to a
variant of ALPT for which only the IDM is pretrained, while the DT is
initialized from scratch during finetuning. We find that this variant
maintains strong performance compared to DT1-IDM, suggesting that the main
benefit of pretraining is the IDM.

(a)

(b)

(c)

(d)

(e)

Figure 2: Game performance across the ALE environments for the baseline and
ALPT. The figure shows the evaluation game performance (Episodic Return) of
our DT policies during finetuning on the limited action target dataset. Higher
score is better. The shaded area represents the standard deviation over $3$
random seeds. The $x$-axis shows the number of finetuning steps. We evaluate
ALPT on 16 episodes of length 2500 each following (Lee et al., 2022).

###  5.4 Is dynamics modelling improved under more or less source datasets?

In this set of experiments, we expand the set of source and target games. As
in the previous experiments, each source game provides a fully-labelled
dataset of size 100M, while each target game has a dataset of size 100M with
only 10k action labels. We evaluate performance using 5 target games ({‘Pong’,
‘SpaceInvaders’, ‘StarGunner’, ‘MsPacman’, ‘Alien’}) and using either 36
source games (ALPT-36, trained on all 41 game datasets available in Agarwal et
al. (2020b) minus the 5 target games) and using 9 source games (ALPT-9,
trained on {‘Asterix’, ‘Breakout’, ‘Freeway’, ‘Seaquest’, ‘Atlantis’,
‘DemonAttack’, ‘Frostbite’, ‘Gopher’, ‘TimePilot’}). Note that for each of
these ALPT variants, we perform a _single_ pretraining phase and then
_multiple_ finetuning phases (one for each target game), thus showing that a
single pretrained model can be transferred to various target games.

We compare pretraining with ALPT to training DT1-IDM on each target game
alone. Results are presented in Figure 3, and show that target game
performance improves with more source games.

(a)

(b)

(c)

(d)

(e)

Figure 3: We evaluate performance of ALPT with a higher number of source
games. We show performance of ALPT trained on 36 Atari source games (ALPT-36)
and ALPT trained on 9 Atari source games (ALPT-9). We find that performance
generally improves with more source games.

###  5.5 Can ALPT help when source and target have disjoint action spaces?

The previous experiments have included at least one source game during
pretraining whose the action space overlaps with that of the target
environment. The next set of experiments aims to explore pretraining on source
environments where the action space ($\mathcal{A}_{d}$) is disjoint with the
target environment action space ($\mathcal{A}_{*}$), that is,
$\mathcal{A}_{d}\cap\mathcal{A}_{*}=\emptyset$. To do so, we use _Freeway_ as
our single source environment dataset. In _Freeway_ , the action space
consists of {Up, Down}. In contrast, a game such as _Breakout_ has an action
space consisting of {Left, Right}. Surprisingly, using _Freeway_ as a source
environment and _Breakout_ as a target environment still yields significant
benefits for ALPT. We present this result in Figure 4, as well as a variety of
other choices for the target environment, none of which share any actions with
the source environment _Freeway_.

We hypothesize that performance improvements are not unexpected due to the
already known broad generalization capabilities of transformer architectures.
It may also be the case that, despite the action spaces being disjoint, they
still exhibit similar structure. For example, a top-down action space and a
left-right action space are structurally opposite to each other in terms of
movement, differing only in orientation. This similar structure is potentially
learned and leveraged during finetuning.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Figure 4: We evaluate performance of ALPT when source and target games have
disjoint action spaces. In each of these plots we pretrain using a single
source game Freeway. Despite the disjoint action space, we still see benefits
of pretraining.

###  5.6 Do ALPT’s benefits persist in other domains ?

We now demonstrate ALPT’s benefits on an action-limited navigation task. This
corresponds to a scenario where we have densely annotated navigation maps for
a set of source regions but only a sparsely annotated navigation map for a
target region. We would like to evaluate whether ALPT, pretrained on source
regions with abundant action labels, can generalize to navigating in a target
region (of a different layout) with limited action labels.

Maze Navigation Environments. To answer the above question, we consider a
gridworld navigation task where an agent seeks to navigate to a goal location
in a $20\times 20$ 2D maze from a random starting location to a random goal
location using 4 discrete actions: {Up, Down, Left, Right}. The agent receives
a reward of $1$ at the goal state and $r=0$ otherwise. To collect the offline
training datasets, we follow Yang et al. (2022); Zhang et al. (2018) to
algorithmically generate maze layouts with random internal walls that form
blocked or tunneled obstacles as shown in Figure 5. We start with blocked
obstacles, and generate one _source_ maze from which we collect $500$
trajectories with full action labels. We then use a different random seed to
generate the _target_ maze, from which we collect $500$ trajectories with only
$250$ action labels ($0.5\%$ of the full action labels). We then train the IDM
of ALPT-Blocked on both the source and target datasets, labeling the missing
actions from the target game, and train DT all at the same time (no separate
finetuning stage). ALPT-Blocked and other baselines are evaluated in the
target maze only.

Results. Performance in the target maze environment with limited action labels
is presented in Figure 5 (b). ALPT-Blocked trained on both source and target
mazes allows us to solve the target task twice as fast compared to only
training on the target maze without access to another source maze. To further
illustrate the benefit of multi-environment training on more diverse data, we
introduce the tunnelled maze, and train ALPT-Blocked+Tunnelled on $500$
trajectories with full actions from a source blocked maze and a source
tunneled maze, respectively, as well as $250$ action samples of the target
blocked maze. Training on both tunnelled and blocked mazes enables greater
dataset diversity, which further improves generalization, leading to even
faster convergence on the target task. These preliminary results on navigation
suggest that multi-environment pretraining can benefit a broad set of tasks.

(a)

(b)

Figure 5: (a) An example diagram of the Blocked (above) and Tunneled (below)
mazes. The green cell is the goal state. (b) Evaluation performance while
training on the 20x20 Maze dataset. Higher score is better. The shaded area
represents the standard deviation over $3$ random seeds. The action limited
target dataset contains $250$ labelled actions in these experiments.

##  6 Conclusion

We explored the problem of learning agent policies from action limited
datasets. Inspired by the paradigm of large-scale, multi-environment training,
we proposed ALPT, which pretrains an inverse dynamics model (IDM) on multiple
environments to provide accurate action labels for a decision transformer (DT)
agent on an action limited target environment dataset. Our experiments and
ablations highlight the importance of pretraining the IDM, as opposed to
pretraining the DT agent alone. Our results support the importance of
generalist inverse dynamics models as an efficient way to implement large-
scale RL. As more labelled data becomes available for training offline RL
agents, ALPT provides an efficient way of bootstrapping performance on new
tasks with action limited data.

###  6.1 Limitations

The largest limitation of ALPT is the assumption that we would have plentiful
labelled data from related environments. One interesting fact we uncover is
that this data does not have to be based on the same action space as the
desired target environment, indicating the versatility of ALPT and its ability
to ingest diverse source environment training data. We also caution that we
have only evaluated ALPT so far on limited, self-contained video game tasks
and simple navigation environments. We hope that as more labelled data becomes
available in RL domains, ALPT will have wider applicability, allowing RL
agents to scale and bootstrap to new environments. It would also be useful to
investigate further how much labelled data from a limited set of source
environments is required to be able to handle a much larger set of unlabelled
datasets.

## References

  * Agarwal et al. (2020a) Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  An optimistic perspective on offline reinforcement learning.  In _Proceedings of the 37th International Conference on Machine Learning_ , ICML’20, 2020a.
  * Agarwal et al. (2020b) Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi.  An optimistic perspective on offline reinforcement learning.  In _International Conference on Machine Learning_ , pp. 104–114. PMLR, 2020b.
  * Aytar et al. (2018) Yusuf Aytar, Tobias Pfaff, David Budden, Tom Le Paine, Ziyu Wang, and Nando de Freitas.  Playing hard exploration games by watching youtube.  NIPS’18, 2018.
  * Baker et al. (2022) Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.  Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022.
  * Bengio et al. (1991) Y. Bengio, S. Bengio, and J. Cloutier.  Learning a synaptic learning rule.  In _IJCNN-91-Seattle International Joint Conference on Neural Networks_ , 1991.
  * Borsa et al. (2016) Diana Borsa, Thore Graepel, and John Shawe-Taylor.  Learning shared representations in multi-task reinforcement learning, 2016\.
  * Bräm et al. (2020) Timo Bräm, Gino Brunner, Oliver Richter, and Roger Wattenhofer.  Attentive multi-task deep reinforcement learning.  In _Machine Learning and Knowledge Discovery in Databases_ , 2020\.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , 2020.
  * Calandriello et al. (2014) Daniele Calandriello, Alessandro Lazaric, and Marcello Restelli.  Sparse multi-task reinforcement learning.  In _Advances in Neural Information Processing Systems_ , 2014.
  * Caruana (1997) Rich Caruana.  Multitask learning.  1997\.
  * Chen et al. (2021a) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling, 2021a.
  * Chen et al. (2021b) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  In _Advances in Neural Information Processing Systems_ , 2021b.
  * Clavera et al. (2019) Ignasi Clavera, Anusha Nagabandi, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn.  Learning to adapt in dynamic, real-world environments through meta-reinforcement learning.  In _International Conference on Learning Representations_ , 2019.
  * D’Eramo et al. (2020) Carlo D’Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters.  Sharing knowledge in multi-task deep reinforcement learning.  In _International Conference on Learning Representations_ , 2020.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  Association for Computational Linguistics, 2019.
  * Duan et al. (2017) Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, and Pieter Abbeel.  RL^2: Fast reinforcement learning via slow reinforcement learning, 2017.
  * Finn et al. (2017) Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , 2017.
  * Fujimoto et al. (2019) Scott Fujimoto, David Meger, and Doina Precup.  Off-policy deep reinforcement learning without exploration.  In _International Conference on Machine Learning_ , pp. 2052–2062, 2019.
  * Gulcehre et al. (2020) Caglar Gulcehre, Ziyu Wang, Alexander Novikov, Thomas Paine, Sergio Gómez, Konrad Zolna, Rishabh Agarwal, Josh S Merel, Daniel J Mankowitz, Cosmin Paduraru, et al.  Rl unplugged: A suite of benchmarks for offline reinforcement learning.  _Advances in Neural Information Processing Systems_ , 33:7248–7259, 2020.
  * Kalousis (2002) Alexandros Kalousis.  _Algorithm selection via meta-learning_.  PhD thesis, University of Geneva, 2002.
  * Kingma et al. (2014) Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling.  Semi-supervised learning with deep generative models.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems_ , 2014.
  * Kumar et al. (2019) Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine.  Stabilizing off-policy q-learning via bootstrapping error reduction.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Kumar et al. (2020) Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine.  Conservative q-learning for offline reinforcement learning.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , NIPS’20, 2020.
  * Lacombe et al. (2021) Thomas Lacombe, Yun Sing Koh, Gillian Dobbie, and Ocean Wu.  A meta-learning approach for automated hyperparameter tuning in evolving data streams.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , 2021.
  * Lee et al. (2022) Kuang-Huei Lee, Ofir Nachum, Mengjiao Yang, Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch.  Multi-game decision transformers, 2022.
  * Mahajan et al. (2018) Dhruv Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten.  Exploring the limits of weakly supervised pretraining.  In _Computer Vision - ECCV 2018 - 15th European Conference, Proceedings, Part II_ , 2018.
  * Mordatch et al. (2016) Igor Mordatch, Nikhil Mishra, Clemens Eppner, and Pieter Abbeel.  Combining model-based policy search with online model learning for control of physical humanoids.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , 2016.
  * Nachum et al. (2019) Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans.  Algaedice: Policy gradient from arbitrary experience, 2019.
  * Nguyen-Tuong et al. (2008) D. Nguyen-Tuong, J. Peters, M. Seeger, and B. Schölkopf.  Learning inverse dynamics: A comparison.  In _Advances in Computational Intelligence and Learning: Proceedings of the European Symposium on Artificial Neural Networks_ , 2008.
  * Oh et al. (2016) Junhyuk Oh, Valliappa Chockalingam, Satinder Singh, and Honglak Lee.  Control of memory, active perception, and action in minecraft.  In _Proceedings of the 33rd International Conference on International Conference on Machine Learning_ , 2016.
  * Parisotto et al. (2016) Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov.  Actor-mimic: Deep multitask and transfer reinforcement learning.  In _ICLR (Poster)_ , 2016.
  * Pong et al. (2022) Vitchyr H Pong, Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine.  Offline meta-reinforcement learning with online self-supervision.  In _Proceedings of the 39th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, 2022.
  * Radford et al. (2019a) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019a.
  * Radford et al. (2019b) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019b.
  * Rakelly et al. (2019) Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine, and Deirdre Quillen.  Efficient off-policy meta-reinforcement learning via probabilistic context variables.  In _Proceedings of the 36th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, 2019.
  * Rasmus et al. (2015) Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko.  Semi-supervised learning with ladder networks.  In _Proceedings of the 28th International Conference on Neural Information Processing Systems_ , 2015.
  * Reed et al. (2022) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas.  A generalist agent, 2022.
  * Ruder (2017) Sebastian Ruder.  An overview of multi-task learning in deep neural networks, 2017.
  * Rusu et al. (2016) Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell.  Policy distillation.  In _ICLR (Poster)_ , 2016.
  * Schmidhuber (1987) Jürgen Schmidhuber.  Evolutionary principles in self-referential learning. (on learning how to learn: The meta-meta-… hook.  Technical report, PhD Thesis, 1987.
  * Seo et al. (2022) Younggyo Seo, Kimin Lee, Stephen L James, and Pieter Abbeel.  Reinforcement learning with action-free pre-training from videos.  In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), _Proceedings of the 39th International Conference on Machine Learning_ , 2022.
  * Sodhani et al. (2021) Shagun Sodhani, Amy Zhang, and Joelle Pineau.  Multi-task reinforcement learning with context-based representations.  In _Proceedings of the 38th International Conference on Machine Learning_ , Proceedings of Machine Learning Research, 2021.
  * Szummer & Jaakkola (2001) Martin Szummer and Tommi Jaakkola.  Partially labeled classification with markov random walks.  In _Advances in Neural Information Processing Systems_ , 2001.
  * Taylor & Stone (2009) Matthew E. Taylor and Peter Stone.  Transfer learning for reinforcement learning domains: A survey.  _Journal of Machine Learning Research_ , 2009.
  * Tzeng et al. (2020) Eric Tzeng, Coline Devin, Judy Hoffman, Chelsea Finn, Pieter Abbeel, Sergey Levine, Kate Saenko, Trevor Darrell, Pieter Abbeel, Kostas Bekris, and Lauren Miller.  _Adapting Deep Visuomotor Representations with Weak Pairwise Constraints_.  2020\.
  * Wu et al. (2020) Yifan Wu, George Tucker, and Ofir Nachum.  Behavior regularized offline reinforcement learning, 2020.  URL https://openreview.net/forum?id=BJg9hTNKPH.
  * Yang et al. (2022) Mengjiao Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum.  Chain of thought imitation with procedure cloning.  _arXiv preprint arXiv:2205.10816_ , 2022.
  * Ye et al. (2021) Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao.  Mastering atari games with limited data.  In _Advances in Neural Information Processing Systems_ , 2021.
  * Zhai et al. (2021) Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers, 2021.
  * Zhang et al. (2021a) Amy Zhang, Shagun Sodhani, Khimya Khetarpal, and Joelle Pineau.  Learning robust state abstractions for hidden-parameter block {mdp}s.  In _International Conference on Learning Representations_ , 2021a.
  * Zhang et al. (2021b) Chi Zhang, Sanmukh Rao Kuppannagari, and Viktor Prasanna.  {BRAC}+: Going deeper with behavior regularized offline reinforcement learning, 2021b.  URL https://openreview.net/forum?id=bMCfFepJXM.
  * Zhang et al. (2018) Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio.  A study on overfitting in deep reinforcement learning.  _arXiv preprint arXiv:1804.06893_ , 2018.
  * Zhang et al. (2014) Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang.  Facial landmark detection by deep multi-task learning.  In _Computer Vision – ECCV 2014_ , 2014.
  * Zhu & Ghahramani (2002) Xiaojin Zhu and Zoubin Ghahramani.  Learning from labeled and unlabeled data with label propagation.  Technical report, 2002.
  * Zhu et al. (2009) Xiaojin Zhu, Andrew B. Goldberg, Ronald Brachman, and Thomas Dietterich.  _Introduction to Semi-Supervised Learning_.  2009\.
  * Zolna et al. (2020) Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed.  Offline learning from demonstrations and unlabeled experience, 2020.

##  Appendix A Experiments with no DT pretraining

In the following set of experiments, we pretrain only the IDM component of
ALPT and not the DT. We show the finetuning performance results for the Narrow
set of Atari games in Figure 6. Note that the axis here is up to $100$k steps
as opposed to 1M for the figures in the main text.

(a)

(b)

(c)

(d)

(e)

Figure 6: Evaluation game performance during finetuning of ALPT and DT$1$-IDM.
In these experiments we do not pretrain the DT. $100k$ steps are shown.

##  Appendix B Implementation Details

In Table 3 we give the implementation details of our IDM and DT transformer
architectures.

The IDM model is the same as the DT model, except that it is non-causal. This
is enforced by changing the attention mask to a matrix of all $1$ values in
the IDM.

Table 3: A summary of the transformer model parameters. Parameter | Value
---|---
Layers | $6$
Hidden Size | $512$
Heads | $8$
Batch Size | $256$
Weight Decay | $5\times 10^{-5}$
Learning Rate | $3\times 10^{-4}$
Gradient Clipping | $1.0$
$\beta_{1},\beta_{2}$ | $0.9,0.999$
Warm-up Steps | $4000$
Optimizer | LAMB

##  Appendix C Experiments with Conservative Q-Learning (CQL)

In this set of experiments, we examine the performance of Conservative
Q-Learning (CQL) (Kumar et al., 2020) trained on a dataset of $10,000$ frames,
as opposed to $500,000$ in the original work (Table 3 of CQL, $1\%$ dataset
size), from various Atari games utilized in our experiments. In Table 4 we
report the final evaluation performance on the game after training for $100$
iterations. All implementation details are consistent with the original
implementation in the cited work. We utilize the CQL$(\mathcal{H}$) method.

Table 4: The final evaluation game performance after training CQL for 100 iterations on a dataset of 10,000 labelled frames from each Atari game. Game Name | Final Performance
---|---
Asterix | $227.5$
Breakout | $12.3$
Freeway | $10.2$
Seaquest | $236.0$
SpaceInvaders | $250.9$