# Rethinking Teacher-Student Curriculum Learning through the Cooperative
Mechanics of Experience

Manfred Diaz<EMAIL_ADDRESS>
Department of Computer Science and Operations Research
University of Montreal
Mila, Québec AI Institute
Liam Paull<EMAIL_ADDRESS>
Department of Computer Science and Operations Research
University of Montreal
Mila, Québec AI Institute
Canada CIFAR AI Chair Andrea Tacchetti<EMAIL_ADDRESS>
Google DeepMind

###### Abstract

Teacher-Student Curriculum Learning (TSCL) is a curriculum learning framework
that draws inspiration from human cultural transmission and learning. It
involves a teacher algorithm shaping the learning process of a learner
algorithm by exposing it to controlled experiences. Despite its success,
understanding the conditions under which TSCL is effective remains
challenging. In this paper, we propose a data-centric perspective to analyze
the underlying mechanics of the teacher-student interactions in TSCL. We
leverage cooperative game theory to describe how the composition of the set of
experiences presented by the teacher to the learner, as well as their order,
influences the performance of the curriculum that are found by TSCL
approaches. To do so, we demonstrate that for every TSCL problem, there exists
an equivalent cooperative game, and several key components of the TSCL
framework can be reinterpreted using game-theoretic principles. Through
experiments covering supervised learning, reinforcement learning, and
classical games, we estimate the cooperative values of experiences and use
value-proportional curriculum mechanisms to construct curricula, even in cases
where TSCL struggles. The framework and experimental setup we present in this
work represent a novel foundation for a deeper exploration of TSCL, shedding
light on its underlying mechanisms and providing insights into its broader
applicability in machine learning.

##  1 Introduction

Controlling the sequence of tasks that a learning algorithm is exposed to
through curriculum has been shown to potentially enhance learning efficiency
(Elman, 1993; Krueger & Dayan, 2009; Bengio et al., 2009). One widely used
curriculum framework, known as Teacher-Student Curriculum Learning (TSCL)
(Graves et al., 2017; Matiisen et al., 2020), specifically gives a teacher
algorithm the ability to control this sequence. While it is commonly
understood that presenting tasks in increasing difficulty can improve
learning, the underlying dynamics and structure of teacher-student interaction
in this context are still relatively unexplored. Very few works have attempted
to understand when, and how TSCL works (Lee et al., 2021; Wu et al., 2020)
while most have focused on providing algorithmic improvements to the problem
(Portelas et al., 2019a; Turchetta et al., 2020; Liu et al., 2020; Feng et
al., 2021). In this paper, we propose a novel _data-centric_ perspective (Ng,
2021) to understand and analyze TSCL algorithms.

We begin by formalizing a general notion of units of experience to describe
the control objects of the teacher algorithm (that are consumed by the
learner). Subsequently, our approach draws inspiration from work on feature
attribution (Patel et al., 2021), data valuation (Ghorbani & Zou, 2019; Yan &
Procaccia, 2021) and explainability (Lundberg & Lee, 2017), and leverages
tools from cooperative game theory (Von Neumann & Morgenstern, 1944; Shapley,
1952) to analyze how the compositions of these units impact teacher-student
interactions. We show that, for every TSCL problem, there exists an equivalent
cooperative game where units of experience are players and teacher-student
interactions approximate a sequential coalition formation process (section 4).
As a result, the learning progression objective (Schmidhuber, 1991; Oudeyer et
al., 2007; Graves et al., 2017) and the teacher bandit policy (Gittins, 1979;
Matiisen et al., 2020), two essential components of TSCL, have alternative
interpretations as an approximation of player (unit) marginal contribution
(Weber, 1988) and a fair allocation mechanism, respectively (Sec. 4.2 & 4.3).
Furthermore, because the order matters in the case of curriculum learning
(Krueger & Dayan, 2009; Bengio et al., 2009), traditional cooperative game-
theoretic arguments produce unintuitive results (Nowak & Radzik, 1994). Thus,
we leverage generalized cooperative games and their solution concepts (Nowak &
Radzik, 1994; Sanchez & Bergantiños, 1997) to overcome these limitations and
formally extend these data-centric game-theoretic formulations to the
curriculum learning setting.

To demonstrate the predictive power and range of problems where this game-
theoretic and data-centric interpretation of TSCL applies, we build an
experimental setting that evaluates the prospect of cooperation among units of
experience in problems spanning supervised learning (SL), reinforcement
learning (RL), and classical games (section 5). These experiments simulate
ordered and unordered coalition formation processes and approximate the
cooperative games we developed to describe TSCL. For every problem, we
estimate units _a priori_ value (e.g., Shapley or Nowak & Radzik values) and
demonstrate that these _a priori_ values, although expensive to compute (Deng
& Papadimitriou, 1994), are useful proxies to find curricula. To this end, we
design unordered and ordered value-proportional curriculum mechanisms inspired
by value-proportional allocations (Bachrach et al., 2020). In most settings,
the unordered mechanism fails to find a reasonable curriculum, demonstrating
the unsuitability of traditional game-theoretic tools for the TSCL problem.
However, the ordered mechanism consistently finds an optimal or near-optimal
ordering (i.e., a curriculum) even when TSCL fails (Sec. 5.5). To understand
what impacts the ability of TSCL in those settings, we leverage another
cooperative game-theoretic tool, namely, _measures of interactions_ (Grabisch
& Roubens, 1999; Procaccia et al., 2014), and in particular the Value of a
Player to other Player (vPoP) (Hausken & Mohr, 2001), to quantify positive,
neutral, or negative pairwise interactions among units. We show that in
settings with considerable unit interference, as characterized by their
negative pairwise interactions, TSCL is unable to produce useful curricula.

##  2 Preliminaries

###  2.1 Cooperative Game Theory

Cooperative Games. Cooperative games model problems where players interact to
maximize collective gain (Roth, 1988). In a (traditional) cooperative game in
characteristic function form among a set of players U, denoted by
$\mathcal{G}=\left<\textbf{U},v\right>$, the characteristic function
$v:2^{\textbf{U}}\to\mathbb{R}$ associates to each coalition $\textbf{C}\in
2^{\textbf{U}}$, belonging to the powerset $2^{\textbf{U}}$, a real number
that represents the benefits produced by the players in C acting jointly. In a
cooperative game, a solution concept represents a mechanism that produces
allocation vectors $\phi\in\mathbb{R}^{|\textbf{U}|}$ (Shubik, 1981).
Particularly, Shapley’s value (Shapley, 1952) allocates to each player
$\textbf{u}\in\textbf{U}$ its average marginal contribution
$v(\textbf{C}+\textbf{u})-v(\textbf{C})$ to coalitions
$\textbf{C}\subseteq\textbf{U}$, where $\textbf{u}\in\textbf{U}-\textbf{C}$

$\phi(\textbf{u})=\sum_{\textbf{C}:\textbf{u}\not{\in}\textbf{C}}\frac{|\textbf{C}|!(|\textbf{U}|-|\textbf{C}|-1)!}{|\textbf{U}|!}\left[v(\textbf{C}+\textbf{u})-v(\textbf{C})\right]$
(1)

and uniquely satisfies the axioms of efficiency, null-player, symmetry, and
linearity which are generally considered to be properties of a fair allocation
mechanism (van den Brink & van der Laan, 1998).

Generalized Cooperative Games. When the order on which players join determines
coalitional worth, traditional cooperative games and their solution concepts
(e.g., Shapley’s value) may produce unintuitive allocations (Nowak & Radzik,
1994). In these games, the generalized characteristic function
$v:\mathcal{P}(2^{\textbf{U}})\to\mathbb{R}$ assigns to every ordered
coalition $\textbf{C}\in\mathcal{P}(2^{\textbf{U}})$ in the powerset of
permutations $\mathcal{P}(2^{\textbf{U}})$ its worth if members join in the
permutation order. Nowak & Radzik (1994) and Sanchez & Bergantiños (1997)
extended Shapley’s work and propose solution concepts for these generalized
cooperative games. We focus on the former due to its intuitive formulation

$\phi_{\textsc{NR}}(\textbf{u})=\frac{1}{|\textbf{U}|!}\sum_{\begin{subarray}{c}\textbf{C}\in\mathcal{P}(2^{\textbf{U}})\\\
\textbf{C}:\textbf{u}\not\in\textbf{C}\end{subarray}}\left[v(\textbf{C}:\textbf{u})-v(\textbf{C})\right]\leavevmode\nobreak\
$ (2)

that averages, for all ordered coalitions
$\textbf{C}\in\mathcal{P}(2^{\textbf{U}})$ where the unit
$\textbf{u}\in\textbf{U}$ is appended last, its marginal contribution to the
newly formed ordered coalition $\textbf{C}:\textbf{u}$.

Measures of Interactions. In a cooperative game, a measure of interaction
(Grabisch & Roubens, 1999; Procaccia et al., 2014) computes players’
influences on other players’ outcomes. In particular, we leverage the _value
of a player to another player_ (vPoP) (Hausken & Mohr, 2001). For the games
above, vPoP constructs a matrix whose entries
$\phi(\textbf{u}_{i},\textbf{u}_{j})\in\mathbb{R}$ measure the influence
player $\textbf{u}_{i}$ exerts over player $\textbf{u}_{j}$. It measures how
the Shapley value of a unit changes in the absence of another. More precisely,

$\phi(\textbf{u}_{i},\textbf{u}_{j})=\sum_{\begin{subarray}{c}\textbf{C}\subseteq\textbf{U}\\\
\textbf{u}_{i},\textbf{u}_{j}\in\textbf{C}\end{subarray}}\frac{(|\textbf{U}|-|\textbf{C}|)(|\textbf{C}|-1)!}{|\textbf{U}|!}\left[\phi(\textbf{u}_{j},\textbf{C})-\phi(\textbf{u}_{j},\textbf{C}-\textbf{u}_{i})\right]$
(3)

where $\phi(\textbf{u}_{j},\textbf{C})$ is the Shapley value of unit
$\textbf{u}_{j}$ (Eq. 1) in the cooperative game restricted to players in C.
This matrix marginal
$\phi(\textbf{u}_{i})=\sum_{j}\phi(\textbf{u}_{i},\textbf{u}_{j})$ corresponds
to each player’s Shapley value. We extend vPoP to games in generalized
characteristic function form by applying Eq. 3 mutatis mutandis using Nowak &
Radzik (1994) value to provide an ordered pairwise interaction metric
$\phi_{\textsc{NR}}(\textbf{u}_{i},\textbf{u}_{j})$.

###  2.2 Bandit Algorithms

Multi-armed bandit algorithms provide a solution to problems decision-making
under uncertainty (Gittins, 1979; Lattimore & Szepesvári, 2020) where, at each
interaction, a decision must be made about with arm $\textbf{u}\in\textbf{U}$
must be pulled. We are particularly interested on action-value based
algorithms that maintain empirical value estimates $q_{k}(\textbf{u})$
computed as

$q_{k}(\textbf{u})\approx\frac{1}{N^{\textbf{u}}_{k}}\sum_{i=1}^{k-1}r(\textbf{u}_{i})\mathbb{I}_{\textbf{u}_{i}=\textbf{u}}$
(4)

and that estimate the average reward received by the algorithm in the
iterations $N^{\textbf{u}}_{k}\leq k$ where the u-arm has been pulled. Bandit
algorithms, like the ones Graves et al. (2017) and Matiisen et al. (2020) use
in their work, transform the estimated average contributions into arms
interactions by deriving from estimated values a Boltzmann policy
$\tau_{k}\in\Delta(\textbf{U})$ such that the probability of interaction is
proportional to the value estimates:

$\tau_{k}(\textbf{u})\propto\mathcal{B}(q_{k}(\textbf{u}))=\frac{e^{\frac{q_{k}(\textbf{u})}{T}}}{\sum_{\textbf{u}^{\prime}}e^{\frac{q_{k}(\textbf{u}^{\prime})}{T}}}$
(5)

More sophisticated approaches (e.g., the Exp3 (Auer et al., 2003) used in our
experiments) account for other factors, like recency, bias, stochasticity, or
non-stationarity (Lattimore & Szepesvári, 2020).

##  3 Experience to Control

The TSCL framework commonly operates under the assumption that tasks presented
to a learning algorithm can influence its learning dynamics. Modern iterative
learning algorithms process tasks in discrete units. For instance, SL and RL
algorithms operate over instances and transitions, respectively. But also,
collections of these elementary units, such as batches or episodes, datasets
or environments, or more generally benchmarks or environment suites describe a
hierarchy of aggregations of experience. Henceforth, we utilize the term _unit
of experience_ for referring to any collection of discrete units that a
teacher algorithm can use to control the dynamics of the learner algorithm.

Algorithm 1 Generalized Teacher-Student Curriculum Learning.

1:procedure GenTSCL

2: inputs policy: $\pi_{0}$, algorithm: $\mathcal{L}$, units: U, metric:
$\mathcal{J}$

3: teacher: $\tau_{0}\in\Delta(\textbf{U})$, targets: $\bar{\textbf{U}}$,
budget: $K$

4: for $k=1\ldots K$ do

5: $\textbf{u}_{k}\sim\tau_{k}(\textbf{u})$

6: $\pi_{k}\sim\mathcal{L}(\pi_{k-1},\textbf{u}_{k})$

7:
$r_{k}\leftarrow\mathcal{J}(\pi_{k},\bar{\textbf{U}})-\mathcal{J}(\pi_{k-1},\bar{\textbf{U}})$

8: $\tau_{k+1}\leftarrow\textsc{UpdateRule}(\tau_{k},\textbf{u}_{k},r_{k})$

9: end for

10: output: $\pi_{K}$

11:end procedure

######  Example 3.1.

For an analysis, we may define a unit of experience as the set of instances of
class in a SL classification problem. For example, in the _MNIST_ dataset
(LeCun & Cortes, 2010), there may be ten units of experience, namely, classes
$\textsc{zero},\textsc{one},\textsc{two},\ldots,\textsc{nine}$.

The units of experience abstraction indistinctly applies to supervised or
reinforcement learning problems. On either paradigm, any iterative learning
algorithm is a controllable system whose control inputs are units of
experience.

######  Example 3.2.

There are four control inputs in mini-batch gradient descent (Goodfellow et
al., 2016): the mini-batch $\left\\{x_{1},\ldots,x_{B}\right\\}$, the loss
function $\ell$, the parameters $\theta$, and the learning rate $\eta$ such
that:

$\theta_{k+1}=\theta_{k}-\eta\nabla_{\\!\theta_{k}}\sum_{i=1}^{B}\ell(\theta_{k},x_{i})$

A TSCL-style algorithm, as presented in Alg. 1, solves a data-centric control
problem. The learner algorithm $\mathcal{L}(\pi_{k-1},\textbf{u}_{k})$ is a
_black-box_ system (line 6) controlled by a teacher algorithm through units
drawn with probability $\textbf{u}_{k}\sim\tau_{k}(\textbf{u})$. The learner
output, at each iteration $k$, is policy or model $\pi_{k}$ whose performance
is measured by a metric function $\mathcal{J}$ that quantifies model’s
performance on a set of evaluation units $\bar{\textbf{U}}$. The teacher’s
objective is to maximize the cumulative learning progression reward (line 7).
For the teacher’s UpdateRule, we focus on multi-armed bandit learning (see
Sec. 2.2). We adopt a data-centric perspective to perform a systematic
investigation of its components.

##  4 The Cooperative Mechanics of Experience

The ideal teacher-student interaction mechanics assume that the learner
monotonically increases its performance on the target task. We conjecture that
a prerequisite for this idealistic curriculum learning dynamics (Matiisen et
al., 2020) to occur within TSCL-style algorithms is that experience (or data)
presented to the learner should not interfere with each other. In other words,
units of experience should interact cooperatively. We explain how this
cooperative mechanics may emerge among units by examining the history of
teacher-student interactions, the reward function, and the bandit selection
policy from a cooperative game-theoretic perspective.

###  4.1 The Mechanics of Coalition Formation

We establish a cooperative game where each unit of experience
$\textbf{u}\in\textbf{U}$ is a player. Next, we interpret the history of
$k\leq K$ teacher-student interactions
$\textbf{H}_{k}=\\{\textbf{u}_{1},\ldots,\textbf{u}_{k}\\}$ through their
empirical frequencies $p_{k}(\textbf{u})\in\Delta_{\textbf{U}}$ which form
unit vectors that lie in the $|\textbf{U}|$-probability simplex
$\Delta(\textbf{U})$. The effective support (i.e., non-zero probabilities)
determine an unordered coalition (i.e., a set)
$\textbf{C}_{k}\subseteq\textbf{U}$ (see Faigle (2022), Chapter 8), formed by
the units presented to the learner up to interaction $k\leq K$. We study this
interpretation through a cooperative game in characteristic function form
(Sec. 2.1).

######  Example 4.1.

(Example 3.1 cont’d) In the class-as-unit equivalence on MNIST, an unordered
training coalition, e.g., the two-unit coalition
$\textbf{C}=\\{\textsc{zero},\textsc{nine}\\}$, describes teacher-student
interactions limited to instances from those classes.

Next, we note that the outcome of a coalition’s work is the policy or model
$\pi_{k}$. Thus, estimating the performance of the policy $\pi_{k}$ through
the metric function $\mathcal{J}$ is akin to approximating the characteristic
function $v(\textbf{C}_{k})$ (Alg. 1, line 7). Moreover, these approximations
are conditioned on an evaluation (or target) unit
$\bar{\textbf{u}}\in\bar{\textbf{U}}$. We model the target-task and multiple-
task settings (Graves et al., 2017) where units of experience should increase
learner performance on an evaluation unit (e.g., a task, or an environment) or
on multiple evaluation units (e.g., a set of tasks or environments).
Consequently, every notion of coalitional worth is conditional on the
evaluation units, thus generating a space of cooperative games.

######  Definition 4.1.

(TSCL Cooperative Games) Let U denote a set of units of experience
$\textbf{u}\in\textbf{U}$ and $\bar{\textbf{U}}$ a set evaluation units
$\bar{\textbf{u}}\in\bar{\textbf{U}}$. Every evaluation coalition
$\bar{\textbf{C}}\in 2^{\bar{\textbf{U}}}$ induces a parameterized
characteristic function $v_{\bar{\textbf{C}}}(\textbf{C}_{k})\in\mathbb{R}$
whose value measures the worth of a coalition $\textbf{C}_{k}$ when the
members of $\bar{\textbf{C}}$ are the evaluation units. Therefore, the TSCL-
family of algorithms operate over a parameterized space of cooperative games:

$\boldsymbol{\mathcal{G}}\left<\textbf{U},\cdot\right>=\left\\{\left<\textbf{U},v_{\bar{\textbf{C}}}\right>\leavevmode\nobreak\
|\leavevmode\nobreak\ \bar{\textbf{C}}\subseteq\bar{\textbf{U}}\right\\}$

comprising $2^{|\textbf{U}|}\times 2^{|\bar{\textbf{U}}|}$ possible games and
where the target-task (i.e., $\bar{\textbf{C}}=\bar{\textbf{u}}$) and the
multiple-tasks (i.e., $\bar{\textbf{C}}=\bar{\textbf{U}}$) settings are
special cases.

######  Example 4.2.

If a learner algorithm is presented with units from
$\textbf{C}=\\{\textsc{zero},\textsc{nine}\\}$ on MNIST, the following
condition is expected to hold:

$v_{\bar{\textbf{C}}=\\{\textsc{zero},\textsc{nine}\\}}(\textbf{C})>v_{\bar{\textbf{C}}=\\{\textsc{zero},\textsc{one}\\}}(\textbf{C})$

###  4.2 Marginal Contributions to Learning

The notions of coalitions and coalitional worth above induce a game-theoretic
interpretation of the learning progression reward. At any iteration $k\leq K$,
this reward $r(\textbf{u}_{k})\in\mathbb{R}$ (Alg. 1, line 7) measures the
improvement in policy performance after the teacher presents a unit
$\textbf{u}_{k}$ to the learner algorithm that produces a new policy
$\pi_{k}\sim\mathcal{L}(\pi_{k-1},\textbf{u}_{k})$. Thus, we can restate this
reward in terms of a game in characteristic function form:

$r(\textbf{u}_{k})=v(\textbf{C}_{k})-v(\textbf{C}_{k-1})=v(\textbf{C}_{k-1}+\textbf{u}_{k})-v(\textbf{C}_{k-1})$
(6)

and note its equivalence to computing the marginal contribution (see section 2
and Eq. 1) of aggregating the unit of experience $\textbf{u}_{k}=\textbf{u}$
to the existing coalition $\textbf{C}_{k-1}$.

###  4.3 A Fair Allocation Mechanism

A principle of fair attribution in cooperative games is that players get
assigned values proportional to their expected marginal contribution. We note
that under the learning progression objective, a bandit action-value estimate
$q_{k}(\textbf{u})$ (Eq. 4) approximates every unit’s (or arm’s) average
marginal contribution after $k$ interactions. Moreover, as discussed in Sec.
2.2, multi-arm bandit algorithms may transform action-values through a
Boltzmann projection (Eq. 5) that converts the value estimates into units’
probabilities of interaction with the learner (i.e., the (stochastic) policy
$\tau_{k}(\textbf{u})$). Consequently, the units that up to interaction $k\leq
K$ have produced larger increases on learner performance would be allocated
larger fractions of the remaining $K-k$ interactions.

Thus, a multi-armed bandit teacher implements a fair allocation mechanism that
computes units’ values by approximating their average marginal contributions
and converts these approximations into the currency-like utility of the TSCL
games, namely, interactions with the learner.

##  5 An Experiment on The Prospect of Cooperation

We introduce an experimental setting that, for any given set of units of
experience that may be given to the teacher algorithm, approximates the a
priori units’ values in a series of experiments that include problems in
supervised learning, reinforcement learning, and classical games. Through the
prospect prior, as we dub these experiments, we study how units’ interactions
impact TSCL’s prospects to find useful curricula.

###  5.1 A Simulation of Cooperation

We simulate two coalition formation processes where units of experience (e.g.,
classes, environments, or opponents) in each coalition fairly share a finite
interaction budget $K\in\mathbb{N}$. First, to simulate a traditional
cooperative game and approximate its characteristic function (section 2), we
draw at each interaction $k\leq K$ a unit $\textbf{u}_{k}\in\textbf{C}$ with
uniform probability
$\tau_{\textbf{C}}(\textbf{u}_{k})\propto\frac{1}{|\textbf{C}|}$, from a
coalition $|\textbf{C}|$, and present it to a learner with algorithm
$\mathcal{L}$. We repeat this procedure for every unordered coalition of units
$\textbf{C}\in 2^{\textbf{U}}$. The uniform distribution reflects our
ignorance on units’ values before simulating their effect on the learner.
Then, to study order in curriculum, we simulate a generalized cooperative game
and approximate its corresponding characteristic function, a unit
$\textbf{u}\in\textbf{C}$ is continually presented to the learner, for
$\lfloor{K}/{|\textbf{C}|}\rfloor$ interactions, in its permutation order on
an ordered coalition C. We repeat this procedure for every
$\textbf{C}\in\mathcal{P}(2^{\textbf{U}})$. Similarly to the unordered case,
we select an ordered equipartition of interactions to reflect ignorance about
a priori units’ values.

Coalitional Worth. In both simulations, and for every coalition, we obtain a
model $\pi_{K}$ at the end of the $K$ interactions. The resulting policy or
model is not biased with respect to any evaluation unit or coalition. Thus, we
leverage the established equivalence between policy or model performance and
(conditional) coalitional worth to estimate the value of every parameterized
characteristic function and form the space of traditional or generalized
cooperative games. By estimating every coalition worth, we have the complete
specification of a cooperative game.

######  Example 5.1.

(Example 4.1 cont’d) Assume a budget of $K=100$ interactions and a subset
(coalition) of classes from MNIST, for instance, units (classes) zero and
nine. In the simulation of a traditional game, for a coalition
$\textbf{C}=\\{\textsc{zero},\textsc{nine}\\}$, we uniformly draw instances
from each unit with probability $\tau(\textbf{u})=\frac{1}{2}$. For a
coalition $\textbf{C}=[\textsc{zero},\textsc{nine}]$ in a generalized game,
instances from unit zero are presented for the first $k=50$ iterations
followed by $k=50$ instances from nine.

###  5.2 Value Approximations

The central quantity for the cooperative solution concepts we introduced in
section 2 is a unit’s marginal contribution. We design the prospect prior
experiment such that adding a unit u to a coalition C has the effect of
reducing the learner algorithm’s interactions with the existing units in the
coalition by keeping the budget $K$ fixed, regardless of the coalition size.
In the traditional cooperative game simulation, the probability of drawing any
unit $\textbf{u}_{k}$ in C gets reduced from
$p_{\textbf{C}}(\textbf{u}_{k})=\frac{1}{|\textbf{C}|}$ to $p_{\textbf{C +
{u}}}(\textbf{u}_{k})=\frac{1}{|\textbf{C}|+1}$. Similarly, in the generalized
game simulation adding a unit $\textbf{u}_{k}$ to a coalition C reduces the
number of interactions of units in C from
$\lfloor\frac{K}{|\textbf{C}|}\rfloor$ to
$\lfloor\frac{K}{|\textbf{C}+1|}\rfloor$. Consequently, in either prospect
prior simulation, a unit u marginal contribution
$v(\textbf{C}+\textbf{u})-v(\textbf{C})$ measures the change in performance
produced by increasing interactions with u while reducing interactions of the
existing units in C.

######  Example 5.2.

(Example 5.1 cont’d) In a traditional game simulation on MNIST, a marginal
contribution such as
$v(\\{\textsc{zero},\textsc{nine}\\})-v(\\{\textsc{zero}\\})$ measures the
change in learner performance produced by exchanging approximately $K/2$
interactions with unit zero for interactions with nine. However, in the
generalized game simulation, the same expression measures the change in
performance produced by exchanging $K$ interactions with unit
$\\{\textsc{zero}\\}$ for $K/2$ with zero first (pre-training) followed by
$K/2$ with nine (fine-tuning).

Thus, the solution concepts for each simulated cooperative game, namely, the
Shapley value for traditional games (Eq. 1) and the _Nowak & Razik’s value_
for generalized games (Eq. 2) estimate a unit’s average marginal contribution
to learning, thus capturing its helpfulness or cooperativeness.

###  5.3 Supervised Classification

The first setting we examine is inspired by our running examples on MNIST.
There, we considered instances aggregated in classes as units of experience.
The main benefit of this toy example is that it is straightforward to produce
ground truth information about units’ interactions by training a model on the
complete dataset (e.g., for $200$ epochs), extracting the model’s confusion
matrix on validation, and identifying the top-k most confused pairs of
classes. We applied the same general procedure to CIFAR10 (Krizhevsky, 2009).
On MNIST, we selected the five classes
$\textsc{two},\textsc{three},\textsc{four},\textsc{five}$ and seven belonging
to top-three most confused pairs (see Appendix A, 8(a)), grouped their
instances into five units of experience, and conduct the prospect prior
simulations for a traditional cooperative game. We did similarly for CIFAR10
six classes with larger pairwise confusion errors on validation, namely, car,
cat, deer, dog, frog and horse (see Appendix A, 8(b)).

Units’ Values. For either Mnist or Cifar10, each unit’s Shapley value
estimated from the traditional cooperative game simulations correctly matches
the ground truth information. In the target-unit setting, where each unit of
experience is also used as evaluation unit, each unit of experience (or class)
matching the evaluation unit (or class) has the largest Shapley value, as
depicted in 2(a) (first five targets) for MNIST and 2(c) (first six targets)
for CIFAR10. For instance, on MNIST, unit $\textbf{u}=\textsc{two}$ has the
largest Shapley value $\phi(\textsc{two})=0.995$ when the evaluation unit is
$\bar{\textbf{u}}=\textsc{two}$. We observed a similar effect on Cifar10.
Then, for the all-units setting, every unit’s Shapley value is approximately
equal on both MNIST and CIFAR10 (2(a) and 2(c), all column), matching the
intuition that, conditional on an all units evaluation, every unit of
experience should be equally valuable. These results confirm the prospect
prior’s ability to correctly estimate units’ values.

Measures of Interactions. We also computed the vPoP measure (see Sec. 2.1, Eq.
3) to verify whether its decomposition of Shapley values into pairwise
interaction values correctly identifies the most confused pairs of classes.
For both MNIST and CIFAR10, their respective vPoP matrices, displayed in 2(b)
and 2(d), provide a good approximation to the ground-truth pairwise
interactions extracted from the confusion matrices. For instance, on MNIST the
units two and seven have the lowest interaction value
$\phi(\textsc{two},\textsc{seven})=\phi(\textsc{seven},\textsc{two})=-0.007$
which corresponds to largest entry $M(2,7)=20$ of the confusion matrix (see
Appendix A, 8(a)). Also, in Cifar10 the units dog and cat have the lowest
interaction value $\phi(\textsc{dog},\textsc{cat})=-0.0164$ coinciding with
the most confused classes $M(\textsc{dog},\textsc{cat})=66$ on validation (see
Appendix A, 8(b)). However, we note that, for instance, on MNIST confusion
matrix $M(2,7)\neq M(7,2)$ and, similarly, on CIFAR10’s
$M(\textsc{dog},\textsc{cat})\neq M(\textsc{cat},\textsc{dog})$. Nevertheless,
we interpret these values as good proxies for units _negative, positive, or
neutral_ pairwise interactions.

(a)

(b)

(c)

(d)

Figure 2: We validated the prospect prior using the class-as-a-unit analogy on
MNIST and Cifar10. In Figures (a) and (c), each column represents units’
Shapley values $\phi(\textbf{u})$ in each cooperative game parameterized by a
target-unit $\bar{\textbf{u}}$ and the target coalition of all units. In
Figures (b) and (d), we present the vPoP decomposition matrix (Eq. 3)
measuring the pairwise interaction values
$\phi(\textbf{u}_{i},\textbf{u}_{j})$ among units in the _all-units_ target.

###  5.4 Other Learning Paradigms

Our formulation is not limited to the toy supervised classification setting.
We also demonstrate its general applicability to problems in RL and classical
games. Details to reproduce these experiments are provided in Appendix A.

####  5.4.1 Reinforcement Learning

We investigate the Minigrid-Rooms (Chevalier-Boisvert et al., 2018) set of
three environments, namely, TwoRooms, FourRooms, and SixRooms for which it is
folk knowledge that an optimal curriculum exists. We apply the _prospect
prior_ simulation of a generalized game where we consider each environment a
_unit of experience_. As a learner algorithm, we used PPO (Schulman et al.,
2017) with an interaction budget of $K=500,000$ steps, and estimated, from the
outcome of these simulations, the Nowak & Radzik values (section 2, Eq. 2),
conditioned on the every environment, and on a uniform distribution over all,
as evaluation targets. In 4(a) we show that the _Nowak & Radzik_ values we
estimate match _folk knowledge_. First, there is no requirement for
environments other than TwoRooms as the only positive value
$\phi_{\bar{\textbf{u}}}(\textsc{TwoRooms})=0.423$ correctly measures. Then,
for FourRooms, the values of
$\phi_{\bar{\textbf{u}}}(\textsc{TwoRooms})=0.041$ and
$\phi_{\bar{\textbf{u}}}(\textsc{FourRooms})=0.107$ indicate that both
environments are required. And finally, environments values of
$\phi_{\bar{\textbf{u}}}(\textsc{FourRooms})=0.03$ and
$\phi_{\bar{\textbf{u}}}(\textsc{SixRooms})=0.03$ indicate that both are
needed for solving SixRooms.

####  5.4.2 Classical Games

We introduce an experimental setting, the Adversarial Sparse Iterated
Prisoner’s Dilemma (A-SIPD), that utilizes the Prisioner’s Dilemma (Flood,
1952; Axelrod & Hamilton, 1981) classical two-player game as a base but in a
more challenging sparse and iterated version where at the end of a finite
number of interactions (e.g., $200$ steps), a win-draw-loss reward is given to
the learner if it accrues more cumulative payoff than its opponent. Opponents
are drawn from a population of five well-known strategies: AlwaysCooperate,
AlwaysDefect, WinStayLoseSwitch, TitForTat (Axelrod, 1981) and a
ZeroDeterminant strategy (Hilbe et al., 2013). We apply the _prospect prior_
simulation of a generalized game where we consider each opponent a _unit of
experience_. As a learner algorithm, we used PPO (Schulman et al., 2017) with
a budget of $K=100,000$ interactions. We estimated the Nowak & Radzik values
(section 2, Eq. 2), conditioned on each opponent, and on a uniform mixture
over all, as evaluation targets. The results we present in 4(c) show that
playing uniquely against TitForTat is sufficient across all evaluation
targets, including the strongest opponents, AlwaysDefect and ZeroDeterminant.
This result contrasts with folk knowledge in population-based training (e.g,
playing against the population’s Nash strategy (Nash, 1950)). We defer to Sec.
5.6 a more in-depth discussion of this finding.

(a) Nowak & Radzik Values

(b) Learning Curves [all]

MiniGrid-Rooms

(c) Nowak & Radzik Values

(d) Learning Curves [all]

A-SIPD

Figure 4:  Nowak & Radzik values (a, c) conditional on each _single-unit_ and
the _multiple-units_ (all) evaluations, and the learning curves (b,d) for our
mechanisms and TSCL.

###  5.5 Prospect Values and Curriculum

Inspired by value-proportional allocations (Bachrach et al., 2020), we
developed a mechanism that turns units’ values into interactions with the
learner by projecting any value vector
$\phi(\textbf{u})\in\mathbb{R}^{|\textbf{U}|}$ onto vectors
$\tau(\textbf{u})\in\Delta_{\textbf{U}}$ in a $|\textbf{U}|$-simplex. While
Bachrach et al. (2020) use a linear projection unable to handle negative
marginal contributions, we We investigate the Boltzmann or softmax projection,
commonly used in multi-arm bandit algorithms (see Sec. 2.2, Eq. 5), and an
Euclidean projection (Blondel et al., 2014) that projects to zero any unit
with negative value.

When values $\phi(\textbf{u})$ are Shapley values, the projected vectors are
used as pre-computed teacher policies (i.e., probability distributions),
mimicking TSCL’s interactions but fixed a-priori knowledge of units’ value.
However, when values $\phi(\textbf{u})$ are _Nowak & Radzik_, vectors
$\tau\in\Delta_{\textbf{U}}$ are used as ordered compositional vectors
(Aitchison, 1982) that represent ordered fractions of $K$ interactions. Thus,
we construct a pre-computed teacher policy that, first orders units by their
_Nowak & Radzik values_, projects the ordered values onto
$\tau\in\Delta_{\textbf{U}}^{K}$, and presents the i-th ranked unit
$\textbf{u}_{i}\in\textbf{U}$ to the learner for the number of interactions
indicated by $\tau_{i}\in\mathbb{N}$. This mechanism preserves the ordered
values captured by Nowak & Radzik’s solution concept.

We compare these value-proportional mechanisms with the multi-arm bandit
approach to TSCL using Exp3 (Auer et al., 2003; Graves et al., 2017). Figure 4
shows, for the all-units evaluation, that for both Minigrid-Rooms and A-SIPD
the teacher policies obtained from the Euclidean projection of Nowak & Radzik
values (i.e, _nowak-all-simplex_) consistently produce learner-induced
policies outperforming those produced by TSCL (i.e, _tscl-all-exp3s_), and the
other mechanisms. These results highlight TSCL’s inability to produce
effective curriculum in the presence of units that have non-cooperative
interactions. Figure 6 presents the vPoP decomposition of Shapley and Nowak &
Radzik values conditioned on the all-units evaluation. In both settings, the
interactions measured from the Shapley-based decomposition produce lower
(negative) value than those obtained with Nowak & Radzik. As we show through
section 4 and section 5, we take the Shapley-based interaction values as fair
approximations of TSCL interactions, and thus they provide a data-centric
explanation to TSCL failures (see Appendix B). On the other hand, the Nowak &
Radzik-based values explain the nowak-all-simplex mechanism success, along
with the elimination, through the Euclidean projection, of negatively-valued
units. These pruning strategies are employed effectively by data valuation
techniques (Yan & Procaccia, 2021).

(a) Shapley-vPoP

(b) Nowak&Radzik-vPoP

(c) Shapley-vPoP

(d) Nowak&Radzik-vPoP

Figure 6: The vPoP decomposition of Shapley’s and Nowak & Radzik values,
conditioned on the all-units evaluation target, for the MiniGrid-Rooms (a, b)
and A-SIPD (c, d) problem settings.

###  5.6 Single and Population-based Curriculum

These results offer an alternative approach to what population-based training
approaches prescribe as curriculum (Lanctot et al., 2017; Balduzzi et al.,
2019; Garnelo et al., 2021; Liu et al., 2022). Generally, _meta-strategy
solvers_ for population-based training leverage tools from non-cooperative
game theory (Von Neumann & Morgenstern, 1944) to find, for instance, the mixed
_Nash equilibrium_ (Nash, 1950) of the empirical meta-game (Wellman, 2006)
played by the population of opponents. In this sense, we may also understand
TSCL as a cooperative meta-strategy solver that prioritizes among a fixed
population of (non-learning) opponents (units of experience) those that
improve the learning progression of a single learning player against one or
more opponents of the same population.

In the sparse and iterated version of Prisoner’s Dilemma that we introduced,
the _Defector_ strategy remains to be the (empirical) game Nash equilibrium.
However, the ordered prospect prior results in 4(c) show that when evaluated
on the population Nash $\bar{\textbf{u}}=\textit{Defector}$, the largest Nowak
& Radzik value corresponded to the _TitForTat_ strategy
$\phi_{\bar{\textbf{u}}}(\textit{TitForTat})=0.34$. Playing against the
_TitForTat_ strategy remains to be the optimal solution across all evaluation
targets, meaning that _TitForTat_ is the best proxy opponent to beat and reach
the Nash equilibrium strategy. We can interpret this result from two
perspectives. First, it could indicate that the sparse, iterated, and overtly
adversarial version of the game we constructed is a more complicated problem
than the original, and the Nash equilibrium, the _Defector_ strategy, is a
stronger opponent. However, these result may also indicate that a cooperative
approach to meta-strategy selection may improve performance in some scenarios.
We believe that this connection warrants further investigation.

##  6 Limitations

The simulations we computed in our experiments are computationally expensive.
It is well-established that cooperative solution concepts are NP-hard (Deng &
Papadimitriou, 1994; Elkind et al., 2009). However, better approximations are
possible (Yan & Procaccia, 2021; Mitchell et al., 2022) although we do not
explore them in this work. Consequently, we do not consider the prospect prior
experiments and the value-proportional curriculum mechanism as algorithmic
replacements of TSCL. The data-centric approach we present studies the limits
imposed on TSCL-style algorithms by the (non)cooperative mechanics among units
of experience. However, we acknowledge that the mechanics of units’
interactions also affect other aspects of TSCL. These aspects may include, for
instance, the teacher’s credit-assignment problem (Gittins, 1979) or neural
networks learning and forgetting dynamics (e.g., Lee et al. (2021)). We
control for these factors by keeping them constant in our experiments (see
Appendix A) but do not undertake their analysis here. Our work is a starting
point for more thorough explorations of TSCL and curriculum learning, their
underlying mechanisms and broader applicability in machine learning.

##  7 Related Work

##### Curriculum Learning.

Since the seminal works of Elman (1993); Krueger & Dayan (2009) and Bengio et
al. (2009), a large body of literature has been produced on curricula for
machine learning algorithms. Excellent surveys presented by Narvekar et al.
(2022); Wang et al. (2022), and Soviany et al. (2022) offer a comprehensive
state of recent advances in the field. Our work closely inspects the TSCL
framework concurrently introduced by Graves et al. (2017) and Matiisen et al.
(2020). While follow-up works have generally focused on either algorithmic
innovations (Weinshall et al., 2018; Portelas et al., 2020; Feng et al., 2021;
Liu et al., 2020; Portelas et al., 2019b; Racaniere et al., 2020; Florensa et
al., 2017; Campero et al., 2021; Du et al., 2022; Florensa et al., 2018) or
evaluation benchmarks (Chevalier-Boisvert et al., 2019; Romac et al., 2021),
the data-centric perspective (Ng, 2021; Karpathy & Abbeel, 2021) we proposed
here is less explored. The work by Wu et al. (2020) empirically verifies the
same fundamental questions on _why_ , _when_ , and _how_ curriculum learning
works. However, while our work is more specifically focused on TSCL, we
provide a game-theoretic grounding that could be further extended to analyze
Wu et al. (2020) setting. On the other hand, Lee et al. (2021) specifically
explore TSCL but with a focus on the deep neural network dynamics leading to
catastrophic forgetting McCloskey & Cohen (1989). The evaluation mechanisms
that we propose in the prospect prior simulation and the game-theoretic
characterization of experience interference we introduce here provide a more
general grounding for the problems studied there.

##### Machine Learning & Game Theory.

Our work draws much of its inspiration from the tradition of cross-pollination
between machine learning and game theory research (Cesa-Bianchi & Lugosi,
2006). In recent years, game theory research has fuelled work in optimization
(Daskalakis & Panageas, 2018; Jin et al., 2019), generative modelling
(Goodfellow et al., 2014; Farnia & Ozdaglar, 2020; Mohebbi Moghaddam et al.,
2023), or robustness (Madry et al., 2018; Huang et al., 2022a). Also, game-
theoretic arguments have been used to revitalize long-standing problems in
machine learning. For instance, Gemp et al. (2021) reframed the century-old
problem of Principal Component Analysis (PCA) (F.R.S., 1901; Hotelling, 1933)
as the Nash equilibrium (Nash, 1950) of a multiplayer game. Recently, Chang et
al. (2020) reformulated sequential decision-making problems (i.e., RL (Sutton
& Barto, 2018)) as local economic transactions among specialized self-
interested agents. Moreover, and closer to our data-centric approach, Lundberg
& Lee (2017) pioneered work on feature attribution and _explainable machine
learning models_ through cooperative game theory. In particular, they
approximate Shapley’s value (Shapley, 1952) (Eq. 1) to compute the influence
input features have on model predictions. In a way, our work extends Lundberg
& Lee (2017) input-feature-as-a-player analogy through the notion of units of
experience and introduces substantial elements to discuss how the cooperative
mechanics of experience affects TSCL at different levels of abstraction (i.e.,
from features to tasks) and varied learning paradigms.

Active Learning. As Graves et al. (2017) and Matiisen et al. (2020) noted in
their works, TSCL is an experience-prioritization mechanism holding a solid
connection to algorithms that actively select experience. The central problem
in active learning is to decide which experience to present to a learning
algorithm by querying an expert or oracle (Settles, 2010). Areas of research
that have also adopted this paradigm of _actively sampling experience_ include
_experience replay_ mechanisms in RL Schaul et al. (2016); Andrychowicz et al.
(2017); Fedus et al. (2020) where transitions stored in a _replay buffer_
(Lin, 1992) are sampled according to some prioritization mechanism. This
category of active methods fits perfectly under the TU-game among units of
experience we introduced in section 4.

##### Multitask Learning.

The cooperative game we designed also provides grounding to the problem of
determining which tasks should be learned together and what their optimal
order is, originally from the literature of multitask learning (Caruana,
1997). The discussion on experience interference in section 4, the _prospect
prior_ simulation and our computation of ordered and unordered Shapley values,
and the vPoP metric are some novel tools we introduce that could shed new
light on this problem. Recent work has explored the different avenues of how
task interference and ordering affect learning outcomes (Standley et al.,
2019; Shamsian et al., 2023; Zhang et al., 2021; Fifty et al., 2021; Lin et
al., 2019). Our work provides a novel grounding that relates easily to these
efforts.

##### Continual Learning.

Finally, another area of interest to our work is continual learning Parisi et
al. (2019); De Lange et al. (2022); Mundt et al. (2023). We note that the
ordered prospect prior simulates a continual learning setting. Units of
experience are presented to the learning algorithm in order, and their
importance (i.e., value in the cooperative game) is estimated using Nowak &
Radzik (1994) extension (Eq. 2) to Shapley’s value. This perspective offers a
principled approach to rank and order tasks by their importance, a critical
aspect of the continual learning setting. We believe that future extensions to
the ordered prospect prior experiment may be part of a toolbox to understand
continual learning benchmarks (Lomonaco & Maltoni, 2017; Lin et al., 2021;
Srinivasan et al., 2022), although we do not undertake that task here.

##  8 Conclusions & Future Work

We reexamined TSCL through the lens of cooperative game theory. By drawing
inspiration from work on data valuation, feature attribution and
explainability, we provide a novel data-centric perspective that re-frames
several of its components through alternative cooperative game-theoretic
interpretations. Our experiments confirmed the appropriateness of studying
TSCL-style under this framework and highlights the impact of units cooperative
mechanics on this problem. However, we only began to unveil the potential of
allocation mechanisms, solution concepts, and measures of interactions to
explain some fundamental aspects of TSCL and hope our work inspires an influx
of novel game-theoretic approaches to the problem. Future work would explore
more theoretically-grounded analysis of this problem through the connection
between convex games (Shapley, 1971) and super(sub)modularity in discrete
combinatorial optimization (Dughmi, 2009; Bach, 2011; Krause & Guestrin, 2011)
and the extension to continuous set of units through values of non-atomic
games (Aumann & Shapley, 1974).

## Acknowledgements

MD would like to thank Eugene Vinitsky for helpful feedback on the manuscript.
MD and LP were supported for NSERC Discovery Grant and the Canada CIFAR AI
Chair program. This research was enabled in part by compute resources provided
by Mila, Québec AI Institute.

## References

  * Aitchison (1982) J Aitchison.  The statistical analysis of compositional data.  _Journal of the Royal Statistical Society_ , 44(2):139–160, January 1982.
  * Andrychowicz et al. (2017) Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Openai Pieter Abbeel, and Wojciech Zaremba.  Hindsight experience replay.  In I Guyon, U Von Luxburg, S Bengio, H Wallach, R Fergus, S Vishwanathan, and R Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.
  * Auer et al. (2003) Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire.  The nonstochastic multiarmed bandit problem.  _SIAM J. Comput._ , 32(1):48–77, jan 2003.  ISSN 0097-5397.  doi: 10.1137/S0097539701398375.  URL https://doi.org/10.1137/S0097539701398375.
  * Aumann & Shapley (1974) R. J. Aumann and L. S. Shapley.  _Values of Non-Atomic Games_.  Princeton University Press, 1974.  URL http://www.jstor.org/stable/j.ctt13x149m.
  * Axelrod & Hamilton (1981) R Axelrod and W D Hamilton.  The evolution of cooperation.  _Science_ , 211(4489):1390–1396, March 1981.
  * Axelrod (1981) Robert Axelrod.  The emergence of cooperation among egoists.  _The American political science review_ , 75(2):306–318, 1981.
  * Bach (2011) Francis Bach.  Learning with submodular functions: A convex optimization perspective.  November 2011.
  * Bachrach et al. (2020) Yoram Bachrach, Richard Everett, Edward Hughes, Angeliki Lazaridou, Joel Z Leibo, Marc Lanctot, Michael Johanson, Wojciech M Czarnecki, and Thore Graepel.  Negotiating team formation using deep reinforcement learning.  _Artificial intelligence_ , 288(103356):103356, November 2020.
  * Balduzzi et al. (2019) David Balduzzi, Marta Garnelo, Yoram Bachrach, Wojciech Czarnecki, Julien Perolat, Max Jaderberg, and Thore Graepel.  Open-ended learning in symmetric zero-sum games.  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 434–443. PMLR, 2019.
  * Bengio et al. (2009) Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In _Proceedings of the 26th Annual International Conference on Machine Learning_ , ICML ’09, pp. 41–48, New York, NY, USA, 2009. Association for Computing Machinery.  ISBN 9781605585161.  URL https://doi.org/10.1145/1553374.1553380.
  * Besson (2018) Lilian Besson.  SMPyBandits: an Open-Source Research Framework for Single and Multi-Players Multi-Arms Bandits (MAB) Algorithms in Python.  Online at: github.com/SMPyBandits/SMPyBandits, 2018.  URL https://github.com/SMPyBandits/SMPyBandits/.  Code at https://github.com/SMPyBandits/SMPyBandits/, documentation at https://smpybandits.github.io/.
  * Blondel et al. (2014) Mathieu Blondel, Akinori Fujino, and Naonori Ueda.  Large-Scale multiclass support vector machine training via euclidean projection onto the simplex.  In _2014 22nd International Conference on Pattern Recognition_ , pp. 1289–1294, August 2014.
  * Campero et al. (2021) Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B. Tenenbaum, Tim Rocktäschel, and Edward Grefenstette.  Learning with {amig}o: Adversarially motivated intrinsic goals.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=ETBc_MIMgoX.
  * Caruana (1997) Rich Caruana.  Multitask learning.  _Machine learning_ , 28:41–75, 1997.
  * Cesa-Bianchi & Lugosi (2006) Nicolo Cesa-Bianchi and Gabor Lugosi.  _Prediction, Learning, and Games_.  Cambridge University Press, 2006.  doi: 10.1017/CBO9780511546921.
  * Chang et al. (2020) Michael Chang, Sid Kaushik, S Matthew Weinberg, Tom Griffiths, and Sergey Levine.  Decentralized reinforcement learning: Global Decision-Making via local economic transactions.  In Hal Daumé Iii and Aarti Singh (eds.), _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 1437–1447. PMLR, 2020.
  * Chevalier-Boisvert et al. (2018) Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal.  Minimalistic gridworld environment for gymnasium, 2018.  URL https://github.com/Farama-Foundation/Minigrid.
  * Chevalier-Boisvert et al. (2019) Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio.  BabyAI: First steps towards grounded language learning with a human in the loop.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=rJeXCo0cYX.
  * Daskalakis & Panageas (2018) Constantinos Daskalakis and Ioannis Panageas.  The limit points of (optimistic) gradient descent in min-max optimization.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc., 2018.  URL https://proceedings.neurips.cc/paper_files/paper/2018/file/139c3c1b7ca46a9d4fd6d163d98af635-Paper.pdf.
  * De Lange et al. (2022) Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Gregory Slabaugh, and Tinne Tuytelaars.  A continual learning survey: Defying forgetting in classification tasks.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(7):3366–3385, July 2022.
  * Deng & Papadimitriou (1994) Xiaotie Deng and Christos H. Papadimitriou.  On the complexity of cooperative solution concepts.  19:257–266, 1994.  ISSN 0364-765X.
  * Du et al. (2022) Yuqing Du, P Abbeel, and Aditya Grover.  It takes four to tango: Multiagent selfplay for automatic curriculum generation.  _International Conference on Learning Representations_ , 2022.
  * Dughmi (2009) Shaddin Dughmi.  Submodular functions: Extensions, distributions, and algorithms. a survey.  December 2009.
  * Elkind et al. (2009) Edith Elkind, Leslie Ann Goldberg, Paul W Goldberg, and Michael Wooldridge.  On the computational complexity of weighted voting games.  _Annals of mathematics and artificial intelligence_ , 56(2):109–131, June 2009.
  * Elman (1993) J L Elman.  Learning and development in neural networks: the importance of starting small.  _Cognition_ , 48(1):71–99, July 1993.
  * Faigle (2022) Ulrich Faigle.  _Mathematical Game Theory_.  2022\.
  * Farnia & Ozdaglar (2020) Farzan Farnia and Asuman Ozdaglar.  GANs may have no nash equilibria.  _arXiv e-prints_ , pp. arXiv:2002.09124, February 2020.
  * Fedus et al. (2020) William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney.  Revisiting fundamentals of experience replay.  In Hal Daumé III and Aarti Singh (eds.), _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pp. 3061–3071. PMLR, 13–18 Jul 2020.  URL https://proceedings.mlr.press/v119/fedus20a.html.
  * Feng et al. (2021) Dieqiao Feng, Carla P Gomes, and Bart Selman.  A novel automated curriculum strategy to solve hard sokoban planning instances.  October 2021.
  * Fifty et al. (2021) Christopher Fifty, E Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn.  Efficiently identifying task groupings for Multi-Task learning.  _Advances in neural information processing systems_ , 2021.
  * Flood (1952) Merrill M. Flood.  _Some Experimental Games_.  RAND Corporation, Santa Monica, CA, 1952.
  * Florensa et al. (2017) Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel.  Reverse curriculum generation for reinforcement learning.  In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , pp. 482–495. PMLR, 2017.
  * Florensa et al. (2018) Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel.  Automatic goal generation for reinforcement learning agents.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 1515–1528. PMLR, 2018.
  * F.R.S. (1901) Karl Pearson F.R.S.  Liii. on lines and planes of closest fit to systems of points in space.  _The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science_ , 2(11):559–572, 1901.  doi: 10.1080/14786440109462720.  URL https://doi.org/10.1080/14786440109462720.
  * Garnelo et al. (2021) Marta Garnelo, Wojciech Marian Czarnecki, Siqi Liu, Dhruva Tirumala, Junhyuk Oh, Gauthier Gidel, Hado van Hasselt, and David Balduzzi.  Pick your battles: Interaction graphs as population-level objectives for strategic diversity.  In _Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems_ , AAMAS ’21, pp. 1501–1503, Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems.  ISBN 9781450383073.
  * Gemp et al. (2021) Ian Gemp, Brian McWilliams, Claire Vernade, and Thore Graepel.  Eigengame: {PCA} as a nash equilibrium.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=NzTU59SYbNq.
  * Ghorbani & Zou (2019) Amirata Ghorbani and James Zou.  Data shapley: Equitable valuation of data for machine learning.  April 2019.
  * Gittins (1979) J C Gittins.  Bandit processes and dynamic allocation indices.  _Journal of the Royal Statistical Society_ , 41(2):148–164, January 1979.
  * Goodfellow et al. (2014) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.), _Advances in Neural Information Processing Systems_ , volume 27. Curran Associates, Inc., 2014.  URL https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep learning_.  MIT press, 2016.
  * Grabisch & Roubens (1999) Michel Grabisch and Marc Roubens.  An axiomatic approach to the concept of interaction among players in cooperative games.  _International Journal of Game Theory_ , 28(4):547–565, Nov 1999.  ISSN 0020-7276.
  * Graves et al. (2017) Alex Graves, Marc G Bellemare, Jacob Menick, Rémi Munos, and Koray Kavukcuoglu.  Automated curriculum learning for neural networks.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 1311–1320. PMLR, 2017.
  * Hausken & Mohr (2001) Kjell Hausken and Matthias Mohr.  The value of a player in n-person games.  _Social choice and welfare_ , 18(3):465–483, 2001.  ISSN 0176-1714.
  * Hilbe et al. (2013) Christian Hilbe, Martin A. Nowak, and Karl Sigmund.  Evolution of extortion in iterated prisoner’s dilemma games.  _Proceedings of the National Academy of Sciences of the United States of America_ , 110(17):6913–6918, Apr 2013.  ISSN 0027-8424.
  * Hotelling (1933) Harold Hotelling.  Analysis of a complex of statistical variables into principal components.  _Journal of Educational Psychology_ , 24:498–520, 1933.
  * Huang et al. (2022a) Peide Huang, Mengdi Xu, Fei Fang, and Ding Zhao.  Robust reinforcement learning as a stackelberg game via adaptively-regularized adversarial training.  February 2022a.
  * Huang et al. (2022b) Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo.  Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.  _Journal of Machine Learning Research_ , 23(274):1–18, 2022b.  URL http://jmlr.org/papers/v23/21-1342.html.
  * Jin et al. (2019) Chi Jin, Praneeth Netrapalli, and Michael I Jordan.  Minmax optimization: Stable limit points of gradient descent ascent are locally optimal.  _arXiv.org_ , 2019.
  * Karpathy & Abbeel (2021) Andrej Karpathy and Pieter Abbeel.  The robot brains podcast: Andrej karpathy on the visionary ai in tesla’s autonomous driving.  https://podcasts.apple.com/us/podcast/andrej-karpathy-on-visionary-ai-in-teslas-autonomous/id1559275284?i=1000513993723, 2021.
  * Kingma & Ba (2015) Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Knight et al. (2021) Vince Knight, Owen Campbell, Marc, T J Gaffney, Eric Shaw, Vsn Reddy Janga, Nikoleta Glynatsi, James Campbell, Karol M Langner, Sourav Singh, Julie Rymer, Thomas Campbell, Jason Young, MHakem, Geraint Palmer, Kristian Glass, Daniel Mancia, edouardArgenson, Martin Jones, kjurgielajtis, Yohsuke Murase, Sudarshan Parvatikar, Melanie Beck, Cameron Davidson-Pilon, Marios Zoulias, Adam Pohl, Paul Slavin, Timothy Standen, Aaron Kratz, and Areeb Ahmed.  Axelrod-Python/Axelrod: v4.12.0, October 2021.
  * Krause & Guestrin (2011) Andreas Krause and Carlos Guestrin.  Submodularity and its applications in optimized information gathering.  _ACM Trans. Intell. Syst. Technol._ , 2(4):1–20, July 2011.
  * Krizhevsky (2009) Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, April 2009.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Krueger & Dayan (2009) Kai A Krueger and Peter Dayan.  Flexible shaping: how learning in small steps helps.  _Cognition_ , 110(3):380–394, March 2009.
  * Lanctot et al. (2017) Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel.  A unified game-theoretic approach to multiagent reinforcement learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Lattimore & Szepesvári (2020) Tor Lattimore and Csaba Szepesvári.  _Bandit Algorithms_.  Cambridge University Press, July 2020.
  * LeCun & Cortes (2010) Yann LeCun and Corinna Cortes.  MNIST handwritten digit database.  2010\.  URL http://yann.lecun.com/exdb/mnist/.
  * Lee et al. (2021) Sebastian Lee, Sebastian Goldt, and Andrew M. Saxe.  Continual learning in the teacher-student setup: Impact of task similarity.  _International Conference on Machine Learning_ , 2021.  URL https://www.semanticscholar.org/paper/57db7f24f15150ef7ea0db1fed20e6ee752792ec.
  * Lin (1992) Long-Ji Lin.  Self-improving reactive agents based on reinforcement learning, planning and teaching.  _Machine learning_ , 8(3):293–321, May 1992.
  * Lin et al. (2019) Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and S Kwong.  Pareto Multi-Task learning.  _Advances in neural information processing systems_ , 2019.
  * Lin et al. (2021) Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan.  The clear benchmark: Continual learning on real-world imagery.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_ , 2021.
  * Liu et al. (2020) Rui Liu, Tianyi Wu, and Barzan Mozafari.  Adam with bandit sampling for deep learning.  October 2020.
  * Liu et al. (2022) Siqi Liu, Luke Marris, Daniel Hennes, Josh Merel, Nicolas Heess, and Thore Graepel.  NeuPL: Neural population learning.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=MIX3fJkl_1.
  * Lomonaco & Maltoni (2017) Vincenzo Lomonaco and Davide Maltoni.  Core50: a new dataset and benchmark for continuous object recognition.  In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), _Proceedings of the 1st Annual Conference on Robot Learning_ , volume 78 of _Proceedings of Machine Learning Research_ , pp. 17–26. PMLR, 13–15 Nov 2017.  URL https://proceedings.mlr.press/v78/lomonaco17a.html.
  * Lundberg & Lee (2017) Scott M. Lundberg and Su-In Lee.  A unified approach to interpreting model predictions.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, pp. 4768–4777, Red Hook, NY, USA, 2017. Curran Associates Inc.  ISBN 9781510860964.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=rJzIBfZAb.
  * Matiisen et al. (2020) Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman.  Teacher-Student curriculum learning.  _IEEE transactions on neural networks and learning systems_ , 31(9):3732–3740, September 2020.
  * McCloskey & Cohen (1989) Michael McCloskey and Neal J Cohen.  Catastrophic interference in connectionist networks: The sequential learning problem.  In Gordon H Bower (ed.), _Psychology of Learning and Motivation_ , volume 24, pp. 109–165. Academic Press, January 1989.
  * Mitchell et al. (2022) Rory Mitchell, Joshua Cooper, Eibe Frank, and Geoffrey Holmes.  Sampling permutations for shapley value estimation.  _Journal of machine learning research: JMLR_ , 23(43):1–46, 2022.
  * Mohebbi Moghaddam et al. (2023) Monireh Mohebbi Moghaddam, Bahar Boroomand, Mohammad Jalali, Arman Zareian, Alireza Daeijavad, Mohammad Hossein Manshaei, and Marwan Krunz.  Games of GANs: game-theoretical models for generative adversarial networks.  _Artificial Intelligence Review_ , February 2023.
  * Mundt et al. (2023) Martin Mundt, Yongwon Hong, Iuliia Pliushch, and Visvanathan Ramesh.  A wholistic view of continual learning with deep neural networks: Forgotten lessons and the bridge to active and open world learning.  _Neural networks: the official journal of the International Neural Network Society_ , 160:306–336, March 2023.
  * Narvekar et al. (2022) Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone.  Curriculum learning for reinforcement learning domains: a framework and survey.  _Journal of machine learning research: JMLR_ , 21(1):7382–7431, June 2022.
  * Nash (1950) J F Nash.  Equilibrium points in N-Person games.  _Proceedings of the National Academy of Sciences of the United States of America_ , 36(1):48–49, January 1950.
  * Ng (2021) Andrew Ng.  Mlops: From model-centric to data-centric ai.  https://www.youtube.com/watch?v=06-AZXmwHjo, 2021.
  * Nowak & Radzik (1994) Andrzej S Nowak and Tadeusz Radzik.  The shapley value for n-person games in generalized characteristic function form.  _Games and economic behavior_ , 6(1):150–161, January 1994.
  * Oudeyer et al. (2007) Pierre-Yves Oudeyer, Frédéric Kaplan, and Verena V Hafner.  Intrinsic motivation systems for autonomous mental development.  _IEEE transactions on evolutionary computation_ , 11(2):265–286, 2007.
  * Parisi et al. (2019) German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter.  Continual lifelong learning with neural networks: A review.  _Neural networks: the official journal of the International Neural Network Society_ , 113:54–71, May 2019.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In _Advances in Neural Information Processing Systems 32_ , pp. 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Patel et al. (2021) Roma Patel, Marta Garnelo, Ian Gemp, Chris Dyer, and Yoram Bachrach.  Game-theoretic vocabulary selection via the shapley value and banzhaf index.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pp. 2789–2798, Online, June 2021. Association for Computational Linguistics.
  * Portelas et al. (2019a) Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer.  Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments.  October 2019a.
  * Portelas et al. (2019b) Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Yves Oudeyer.  Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments.  October 2019b.
  * Portelas et al. (2020) Rémy Portelas, Clément Romac, Katja Hofmann, and Pierre-Yves Oudeyer.  Meta automatic curriculum learning.  November 2020.
  * Procaccia et al. (2014) Ariel Procaccia, Nisarg Shah, and Max Tucker.  On the structure of synergies in cooperative games.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 28(1), Jun 2014.  ISSN 2374-3468.  doi: 10.1609/aaai.v28i1.8812.  URL https://ojs.aaai.org/index.php/AAAI/article/view/8812.
  * Racaniere et al. (2020) Sebastien Racaniere, Andrew Lampinen, Adam Santoro, David Reichert, Vlad Firoiu, and Timothy Lillicrap.  Automated curriculum generation through setter-solver interactions.  In _International Conference on Learning Representations_ , 2020.
  * Romac et al. (2021) Clément Romac, Rémy Portelas, Katja Hofmann, and Pierre-Yves Oudeyer.  TeachMyAgent: a benchmark for automatic curriculum learning in deep RL.  March 2021.
  * Roth (1988) Alvin E Roth (ed.).  _The Shapley value: essays in honor of Lloyd S. Shapley_.  Cambridge University Press, October 1988.
  * Sanchez & Bergantiños (1997) Estela Sanchez and Gustavo Bergantiños.  On values for generalized characteristic functions.  _Operations-Research-Spektrum_ , 19(3):229–234, September 1997.
  * Schaul et al. (2016) Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.  Prioritized experience replay.  In _4th International Conference on Learning Representations_ , 2016.
  * Schmidhuber (1991) J. Schmidhuber.  _A possibility for implementing curiosity and boredom in model-building neural controllers_.  The MIT Press, 1991.  ISBN 9780262256674.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Settles (2010) Burr Settles.  Active learning literature survey.  _Machine learning_ , 15(2):201–221, 2010.
  * Shamsian et al. (2023) Aviv Shamsian, Aviv Navon, Neta Glazer, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya.  Auxiliary learning as an asymmetric bargaining game.  _ArXiv_ , 2023.
  * Shapley (1952) Lloyd S Shapley.  A value for N-Person games.  Technical report, RAND Corporation, 1952.
  * Shapley (1971) Lloyd S Shapley.  Cores of convex games.  _International Journal of Game Theory_ , 1(1):11–26, December 1971.
  * Shubik (1981) Martin Shubik.  Game theory models and methods in political economy.  In Kenneth J Arrow and Michael D Intriligator (eds.), _Handbook of Mathematical Economics_ , volume 1, pp. 285–330. Elsevier, 1981.
  * Soviany et al. (2022) Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe.  Curriculum learning: A survey.  _International journal of computer vision_ , 130(6):1526–1565, June 2022.
  * Srinivasan et al. (2022) Tejas Srinivasan, Ting-Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason.  Climb: A continual learning benchmark for vision-and-language tasks.  volume 35, pp. 29440–29453, 2022.
  * Standley et al. (2019) Trevor Standley, Amir R Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese.  Which tasks should be learned together in multi-task learning?  May 2019.
  * Sutton & Barto (2018) Richard S Sutton and Andrew G Barto.  _Reinforcement Learning: An Introduction_.  The MIT Press, 2nd edition, 2018.
  * Turchetta et al. (2020) Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal.  Safe reinforcement learning via curriculum induction.  _Advances in neural information processing systems_ , 33:12151–12162, 2020.
  * van den Brink & van der Laan (1998) René van den Brink and Gerard van der Laan.  Axiomatizations of the normalized banzhaf value and the shapley value.  _Social Choice and Welfare_ , 15(4):567–582, 1998.  ISSN 01761714, 1432217X.  URL http://www.jstor.org/stable/41106281.
  * Von Neumann & Morgenstern (1944) John Von Neumann and Oskar Morgenstern.  _Theory of games and economic behavior_.  Princeton University Press, Princeton, NJ, 1944.
  * Wang et al. (2022) Xin Wang, Yudong Chen, and Wenwu Zhu.  A survey on curriculum learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(9):4555–4576, September 2022.
  * Weber (1988) Robert J Weber.  Probabilistic values for games.  In A E Roth (ed.), _Essays on the Shapley Value and Its Applications_ , pp. 101–119. Cambridge University Press, 1988.
  * Weinshall et al. (2018) Daphna Weinshall, Gad Cohen, and Dan Amir.  Curriculum learning by transfer learning: Theory and experiments with deep networks.  Technical report, 2018.
  * Wellman (2006) Michael P. Wellman.  Methods for empirical game-theoretic analysis.  In _Proceedings of the 21st National Conference on Artificial Intelligence - Volume 2_ , AAAI’06, pp. 1552–1555. AAAI Press, 2006.  ISBN 9781577352815.
  * Wu et al. (2020) Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur.  When do curricula work?  _International Conference on Learning Representations_ , 2020.
  * Yadan (2019) Omry Yadan.  Hydra - a framework for elegantly configuring complex applications.  Github, 2019.  URL https://github.com/facebookresearch/hydra.
  * Yan & Procaccia (2021) Tom Yan and Ariel D Procaccia.  If you like shapley then you’ll love the core.  _Proceedings of the … AAAI Conference on Artificial Intelligence. AAAI Conference on Artificial Intelligence_ , 35(6):5751–5759, May 2021.
  * Zhang et al. (2021) Yipeng Zhang, Tyler L Hayes, and Christopher Kanan.  Disentangling transfer and interference in Multi-Domain learning.  _ArXiv_ , 2021.

##  Appendix A Prospect Prior Experiments Details

We provide for all problems, models or policies architectures, algorithm
hyperparameters, and other reproducibility details111Regardless, we plan to
release the complete source code of all our experiments.. All models and
architectures are implemented with Pytorch (Paszke et al., 2019), are
configured using Hydra (Yadan, 2019), and fit on a workstation equipped with a
$16$ GB NVIDIA RTX A4000 GPU, $32$ GB of RAM, and $32$ CPU cores.

###  A.1 Supervised Learning

MNIST. We trained a model on the Mnist (LeCun & Cortes, 2010) supervised
10-digits classification task. Specification of the model architecture and
hyper-parameters selection are provided in Table 1.

Hyperparameter | Value
---|---
optimizer |  Adam (Kingma & Ba, 2015)
learning-rate | $10^{-4}$
betas | $(0.9,0.999)$
eps | $10^{-8}$
batch-size | $4$
epochs | $200$
shuffle | Yes

Model
---
${\textsc{Conv2d}(32,3,1)}$
ReLU()
Conv2d (64, 3, 1)
ReLU()
MaxPool2d (2, 2)
DropOut(0.25)
Flatten()
Linear(9216, 128)
ReLU()
DropOut(0.5)
Linear(128, 10)

Table 1: Details on the learning algorithm hyperparameters (left) and model
architecture (right) used in the MNIST (LeCun & Cortes, 2010) experiments.
Model components and the optimizer are provided by Pytorch (Paszke et al.,
2019). These details remained constant throughout the rest of the experiments
with MNIST.

CIFAR10. The experiments on Cifar10 (Krizhevsky et al., 2009) follow the same
setting as those on MNIST. We similarly trained a model on the supervised
10-classes task. Specification of the model architecture and hyperparameter
selection are provided in Table 2.

Hyperparameter | Value
---|---
optimizer |  Adam (Kingma & Ba, 2015)
learning-rate | $10^{-4}$
betas | $(0.9,0.999)$
eps | $10^{-8}$
batch-size | $4$
epochs | $200$
shuffle | Yes

Model
---
Conv2d (3, 6, 5)
ReLU()
MaxPool2d (2, 2)
Conv2d (6, 16, 5)
ReLU()
MaxPool2d (2, 2)
Flatten()
Linear(400, 120)
ReLU()
Linear(120, 84)
ReLU()
Linear(84, 10)

Table 2: Details on the learning algorithm hyperparameters (left) and model
architecture (right) used in the Cifar10 (Krizhevsky et al., 2009)
experiments. Model components and the optimizer are provided by Pytorch
(Paszke et al., 2019). These details remained constant throughout the rest of
the experiments with Cifar10.

(a) MNIST

(b) CIFAR10

Figure 8: The class-as-a-unit analogy applied to MNIST (a) and Cifar10 (b)
served as our ground truth. For each problem, we derived the Shapley’s value
from the precomputed priors (left) [Eq. 1] on each cooperative game (section
5). Our results verify that units values on the _target-unit_ settings
approximately ordered the most confused pairs of classes. For instance, digits
2 & 7 in MNIST, or dog & cat in Cifar10. When the target is all classes, the
vPoP decomposition (right) also (Sec. 2.1) identifies interfering pairs.

###  A.2 Reinforcement Learning

Minigrid Rooms. We utilized a sequence of TwoRooms, FourRooms, and SixRooms
gridworlds provided on Minigrid (Chevalier-Boisvert et al., 2018) as units of
experience. As the learning algorithm, we trained for $500,000$ steps a PPO
Schulman et al. (2017) agent, whose implementation we derived from CleanRL
Huang et al. (2022b). Policy and actor-critic architecture, with shared
backbone, as well as other PPO hyperparameters details are presented in Table
3. For the TSCL experiments, we leveraged Exp3S (Auer et al., 2003)
implementation from Besson (2018) with default hyperparameters
$\alpha=10^{-5}$ and $\gamma=0.05$, as defined in Graves et al. (2017).

Hyperparameter | Value
---|---
optimizer |  Adam Kingma & Ba (2015)
learning-rate | $0.0025$
annealing | Yes
num-steps | $128$
total-timesteps | $500,000$
seeds | $5$
gamma | $0.99$
GAE-lambda | $0.95$
num-minibatches | $4$
update-epochs | $4$
advantage-normalization | Yes
clip-value-loss | Yes
clip-coeff | $0.2$
entropy-coeff | $0.01$
vf-coeff | $0.5$
max-grad-norm | $0.5$
target-kl | No

Actor | Critic
---|---
Conv2d(16, 2, 2)
ReLU()
MaxPool2d(2, 2)
Conv2d(16, 32, 2, 2)
ReLU()
Conv2d(16, 64, 2, 2)
ReLU()
$\textsc{Linear}(64,64)$ | $\textsc{Linear}(64,64)$
$\textsc{Tanh}()$ | $\textsc{Tanh}()$
$\textsc{Linear}(64,7)$ | $\textsc{Linear}(64,1)$

Table 3: Details on the PPO hyperparameters (left) and actor-critic
architecture (right) used in the MinigridRooms (Chevalier-Boisvert et al.,
2018) experiments. Policy and critic components, and the optimizer, are
provided by Pytorch (Paszke et al., 2019). Implementation and default
hyperparameters are derived from CleanRL (Huang et al., 2022b). These details
remained constant throughout the rest of the experiments with MinigridRooms.

###  A.3 Populations & Games.

Adversarial SIPD. In our more challenging sparse and iterated version of
Prisoner’s Dilemma, at the end of $200$ interactions, inspired by Axelrod’s
competition (Axelrod, 1981). A win-draw-loss reward $r=\\{-1,0,1\\}$ is given
to a learning player if it beats a fixed opponent. Opponents are drawn from a
population of five well-known strategies: always cooperate, always defect,
win-stay-lose-switch, tit-for-tat, and a zero-determinant strategy (Axelrod,
1981; Hilbe et al., 2013; Knight et al., 2021). We trained for $500$ episodes
(or $100,000$ steps) a PPO (Schulman et al., 2017) agent adapted from CleanRL
Huang et al. (2022b) default implementation. Policy and actor-critic
architecture, without shared backbone, as well as other PPO hyperparameters
details are presented in Table 4. For the TSCL experiments, we leveraged Exp3S
(Auer et al., 2003) implementation from Besson (2018) with default
hyperparameters $\alpha=10^{-5}$ and $\gamma=0.05$, as defined in Graves et
al. (2017).

Hyperparameter | Value
---|---
optimizer |  Adam (Kingma & Ba, 2015)
learning-rate | $0.0025$
annealing | Yes
num-steps | $128$
timesteps | $100,000$
seeds | $5$
gamma | $0.99$
GAE-lambda | $0.95$
minibatches | $4$
epochs | $4$
advantage-norm | Yes
clip-value-loss | Yes
clip-coeff | $0.2$
entropy-coeff | $0.01$
vf-coeff | $0.5$
max-grad-norm | $0.5$
target-kl | No

Actor | Critic
---|---
$\textsc{Linear}(2,64)$ | $\textsc{Linear}(2,64)$
$\textsc{OrthoInit}()$ | $\textsc{OrthoInit}()$
$\textsc{Tanh}()$ | $\textsc{Tanh}()$
$\textsc{Linear}(64,64)$ | $\textsc{Linear}(64,64)$
$\textsc{OrthoInit}()$ | $\textsc{OrthoInit}()$
$\textsc{Tanh}()$ | $\textsc{Tanh}()$
$\textsc{Linear}(64,2)$ | $\textsc{Linear}(64,1)$

Table 4: Details on the PPO hyperparameters (left) and actor-critic
architecture (right) used in the Adverarial-SIPD experiments. Policy and
critic components, and the optimizer, are provided by Pytorch (Paszke et al.,
2019). Implementation and default hyperparameters are derived from CleanRL
(Huang et al., 2022b). These details remained constant throughout the rest of
the experiments.

##  Appendix B Extended Experiments Results

###  B.1 Value-Proportional Curriculum

(a) MinigridRooms

(b) Adversarial-SIPD

Figure 10: We also investigated the _prior-proportional curriculum_ in the
_target-unit_ setting. For each target unit, we allocate to each training unit
interactions proportional to their pre-computed values for each target. For
the Adversarial-SIPD and MiniGrid-Rooms controlled their learning dynamics by
presenting the units according to unordered and ordered mechanisms in section
5. On each task, the _value-proportional curriculum_ derived from the
_prospect priot_ outperforms TSCL _(tscl-*-exp3s)_. We further investigate the
reason for TSCL failures on this scenario.

###  B.2 TSCL Failures

(a) Individual Runs

(b) vPoP Metric

Figure 12: To understand the failure modes of TSCL on Adversarial-SIPD, we
represented the individual runs (i.e., each of the five seeds) on every target
unit. TSCL _(tscl-*-exp3s)_ (top row) is extremely brittle, unstable, and
generally not robust to units interference. We surmise that these failures are
related to the exploration-exploitation dilemma. Exploratory steps presenting
a negatively-valued unit are hard to overcome (forgetting dynamics). This
issue requires further investigation, and we defer it to future work.

(a) MinigridRooms Figure 14: We found that TSCL presents a similar problem in
MinigridRooms. When actions (units) need to be almost deterministically drawn
for several steps, and other actions (units) have negative interference with
the target, TSCL is unable to find a stable and robust solution to the p
roblem.

  *[TSCL]: Teacher-Student Curriculum Learning
  *[SL]: supervised learning
  *[RL]: reinforcement learning
  *[vPoP]: Value of a Player to other Player
  *[A-SIPD]: Adversarial Sparse Iterated Prisoner’s Dilemma