# Scope Loss for Imbalanced Classification and RL Exploration

Hasham Burhani
RBC CM
&Xiao Qi Shi
RBC CM
&Jonathan Jaegerman
University of Toronto
&Daniel Balicki
RBC CM
<EMAIL_ADDRESS>equal<EMAIL_ADDRESS>equal
<EMAIL_ADDRESS>equal
<EMAIL_ADDRESS>equal contribution

###### Abstract

We demonstrate equivalence between the reinforcement learning problem and the
supervised classification problem. We consequently equate the exploration
exploitation trade-off in reinforcement learning to the dataset imbalance
problem in supervised classification, and find similarities in how they are
addressed. From our analysis of the aforementioned problems we derive a novel
loss function for reinforcement learning and supervised classification. Scope
Loss, our new loss function, adjusts gradients to prevent performance losses
from over-exploitation and dataset imbalances, without the need for any
tuning. We test Scope Loss against SOTA loss functions over a basket of
benchmark reinforcement learning tasks and a skewed classification dataset,
and show that Scope Loss outperforms other loss functions.

##  1 Introduction

Supervised learning has achieved state of the art performance in many domains
over the past decade. Notably, supervised learning has recently achieved near-
human performance in image classification He et al. [2016], Redmon et al.
[2016], Wu et al. [2019], Ramesh et al. [2022], Saharia et al. [2022] and
natural language processing Raffel et al. [2020], Devlin et al. [2018],
Radford et al. [2018]. Meanwhile reinforcement learning (RL) has also made
significant progress in gaming environments Silver et al. [2016], Vinyals et
al. [2019], Berner et al. [2019] and robotics Ziebart [2010]. Although the
problems of supervised classification and RL are dominated by similar deep
learning solutions, they are treated as mutually exclusive subjects in
academia and industry. We will first eliminate the barrier between them by
framing the supervised classification problem as an RL problem. In other
words, the set of all classification problems is a subset of all RL problems
with several defining properties. Therefore, all of the terms describing a
classification problem have one or more corresponding terms in an RL
formulation of the same problem. As a lemma, we will draw the equivalence
between the imbalanced dataset problem in supervised classification and the
exploration exploitation trade-off in RL.

The imbalanced dataset problem is very common in many practical machine
learning applications. The imbalanced dataset problem is commonly defined as
the presence of a non-uniform distribution of classes in a supervised learning
dataset. The dataset imbalance can cause performance losses as gradients from
dominant classes can drown out gradients from other classes. In RL, it is
important to balance exploitation with exploration because over-exploitation
can cause the agent to get caught in sub-optimal policies. If an agent only
exploits knowledge it already has, it will be unable to generate gradients
towards new states and actions that could lead to better rewards. In other
words, an exploitative agent generates an imbalanced dataset of the
environment to learn from.

Over the past decade, many efforts have been devoted to coming up with
solutions to both problems. Despite progress, the problems still persist.
Notably, better loss functions have been invented to address the imbalanced
dataset problem Li et al. [2021], Lin et al. [2017a] and the exploration
exploitation trade-off Mnih et al. [2016, 2013]. We propose a novel loss
function that inspires itself from solutions to both problems, and generalizes
to work for both supervised classification and RL.

##  2 Imbalanced Dataset Problem vs Exploitation Problem in RL

From here, we can note that the problem of dataset imbalance in supervised
classification is equivalent to a lack of exploration in RL. An exploitative
RL agent may generate an imbalanced dataset of the environment to learn from;
limiting how optimal its policy can become. In classification, with an
imbalanced dataset, the classification agent’s policy might become too
exploitative, repeatedly choosing actions that only work because of a slight
statistical advantage in the environment. We can avoid this behaviour in both
cases by promoting exploration, which would induce a robust understanding of
the environment, from which the agent can form a more intelligent policy. As
with any RL, exploration can be promoted with entropy loss functions and
exploration rewards.

With the following example, we demonstrate how an RL model can self-reinforce
to generate an imbalanced dataset. Consider a binary path maze as illustrated
in Figure 1. The agent starts in the $Init$ state and can decide to take the
path to the left ($Cat$) or right ($Dog$). If the agent were to focus on
exploration, there would be roughly equal number of Dog or Cat states in its
replay buffer, i.e. the agent produces a balanced dataset of the environment.
On the other hand if the agent were to focus on exploitation, knowing it can
receive more reward from the $Dog$ state, the agent would quickly converge to
mostly taking the right path. Thus, the replay buffer would be filled with
more $Dog$ states than $Cat$ states, i.e. the agent produces an imbalanced
dataset of the environment.

Despite equivalence, results from research regarding each problem are not
shared and both problems are solved quite differently. Notably, loss function
adaptations are completely inconsistent.

Figure 1: An illustration of how exploitation can generate imbalanced dataset.

##  3 Background

Imbalanced Dataset Classification: The problem of imbalanced datasets in
classification is most commonly addressed with hard negative mining Sung
[1996], Shrivastava et al. [2016]. With hard negative mining, difficult or
underrepresented training examples are resampled or weighted stronger
gradients. New loss functions have also been created as a robust method to
automatically adjust the weights of training sample gradients relative to how
easily the model classifies them. Modified Huber Loss Hastie et al. [2001],
Gokcesu and Gokcesu [2021] combines properties of both MSE Loss and MAE Loss
to down-weigh the gradients of outliers and well-classified examples based on
the magnitude of their error. Designed specifically for the imbalanced object
detection problem, Focal Loss Lin et al. [2017b] down-weighs gradients of
well-detected training examples based on the magnitude of the probability of
the detection.

RL Exploration: RL exploration can be promoted via careful design of intrinsic
reward functions Aubret et al. [2019]. Using an environment model, Gregor et
al. [2016] promotes exploration with intrinsic rewards assigned to policies
that can reach more states. DORA Choshen et al. [2018] learns a secondary MDP
in parallel; this MDP carries rewards that surrogate the knowledge value of
state-actions. NGU Badia et al. [2020] uses episodic memory to reward
trajectories that don’t revisit states. An entropy term in RL loss functions
has recently emerged as a robust method for promoting exploration with minimal
tuning Ziebart [2010].

RL for Classification and Regression: With a significant increase in the
performance of RL over the last decade, RL has recently been applied to the
mainstream problems of classification and regression. By repeatedly training
neural networks, framing model design and hyperparameter selection as RL
actions, and using validation accuracy as a reward function, RL has seen
success addressing the problems of neural architecture search Zoph and Le
[2016], hyperparameter optimization Jomaa et al. [2019] and transfer learning
Mahmud et al. [2022]. With a considerable loss in training efficiency, RL has
also been used to directly train neural networks for high generalization
performance in complex classification tasks Lin et al. [2019], Wiering et al.
[2011], Zhang et al. [2018], Zhao et al. [2017], Obando-Ceron et al. [2019].
In these examples, the supervised classification problem is framed as an RL
problem by treating an attempted classification as an action, and generating a
reward function from data labels.

###  3.1 Imbalanced Datasets in Supervised Classification

The imbalanced dataset problem is present in supervised classification
environments with dataset distribution imbalances. Although this definition is
widely used and captures most examples, it does fail under two scenarios.
First, a model will have no problem classifying a trivial dataset regardless
of whether the dataset distribution is skewed. Second, a non-skewed dataset
with some classes that are much harder to classify than others will cause a
similar performance loss. Consider, for example, a dataset with an equal
number of class A and class B labelled images, but the quality of the class A
images is much worse than the quality of the class B images. We propose to
define the imbalanced dataset problem generally as follows: given a model and
a dataset, the imbalanced dataset problem is present when classes are
disparately difficult to classify, resulting in the model performing sub-
optimally on some classes.

With a dataset imbalance, a classification agent can learn to perform
relatively well by exploiting the imbalance and predicting easier classes,
rather than learning a smarter optimal policy that maximizes predictions over
each class. In supervised classification, the common solution is to
artificially balance the dataset with hard negative mining, data augmentation
or adaptive loss functions that balance gradients.

####  3.1.1 Focal Loss

Cross Entropy Loss is the original loss function used for supervised multi-
class classification.

$\text{Cross Entropy Loss (Supervised)}=-log(p_{i})$ (1)

Where $p_{i}$ is the probability of predicting the target class. Focal loss is
an adaptive loss function that balances gradients by reducing the magnitude of
gradients from well-classified examples. Focal Loss applies a scaling term to
Cross Entropy Loss that decays to zero as the probability of the target class
increases to one.

$\text{Focal Loss (Supervised)}=-(1-p_{i})^{\gamma}log(p_{i})$ (2)

Where $\gamma\geq 0$ is a focusing parameter, which controls the curvature of
the scaling term as $p_{i}$ varies from zero to one.

###  3.2 Exploration Exploitation in RL

The exploration exploitation trade-off is a difficult problem in RL. RL agents
learn an optimal behavior from environments by experience; at the same time,
an agent’s behaviour determines what it will experience from the environment.
To maximize the quality of learning from an environment, the RL agent must
explore the environment and discover how it can receive the most reward. To
maximize returns from an environment, the RL agent must instead exploit what
it already knows to receive reward. In RL algorithms, we generally find an
agent that explores near the start of its training and exploits more as it
understands the environment. This agent, however, often explores too much and
never fully converges to an efficient exploitative policy, or the agent
exploits too much and settles at a sub-optimal policy. Vanilla RL algorithms
are mostly exploitative and have trouble performing well in environments with
sparse reward functions that require a lot of exploration.

####  3.2.1 Policy and Entropy Loss

Policy Loss is one of the original loss functions used for Actor Critic RL
algorithms.

$\text{Policy Loss (Actor Critic)}=-Alog(p_{i})$ (3)

Where $A$ is the advantage term in the Actor Critic setup, and $p_{i}$ is the
probability of selecting the chosen action. One common method of encouraging
exploration in Actor Critic algorithms is to introduce an entropy term into
the loss function. The (negative) entropy term is minimized when all action
probabilities are equal. Whereas the policy term maximizes exploitation of
known environmental advantages by increasing the probability of actions that
result in a high advantage, the entropy term promotes exploration by blindly
increasing the probability of all actions.

$\begin{split}\text{Policy and Entropy Loss (Actor
Critic)}&=-Alog(p_{i})+\alpha\sum_{j}p_{j}log(p_{j})\\\ &=-Alog(p_{i})+\alpha
p_{i}log(p_{i})+\alpha\sum_{j,j\neq i}p_{j}log(p_{j})\\\ &=-(A-\alpha
p_{i})log(p_{i})+\alpha\sum_{j,j\neq i}p_{j}log(p_{j})\end{split}$ (4)

Where $\alpha\geq 0$ adjusts the magnitude of the entropy term, which
increasingly encourages exploration as $\alpha$ increases. Policy and Entropy
Loss contains bias in its equation from the $\alpha\sum_{j,j\neq
i}p_{j}log(p_{j})$ term. Namely, if $\alpha>0$, the optimal action’s
probability will never converge to $1$, as the entropy term will not be
minimized.

###  3.3 Supervised Classification as RL

RL environments are usually modeled as a Markov Decision Process (MDP), and
most RL algorithms are designed around MDP concepts. The classification task
can easily be modeled as an MDP. The unlabelled data comprises of a Markov
state $s$ from which the agent will choose to classify the data as any of the
available classes. The act of classifying is equivalent to a Markov action $a$
selected from a discrete action space, and a reward $r$ is provided based on
the correctness of the classification action. Given training example labels,
the reward function can be known before having taken an action.

The classification environment can be framed as a one-step episodic MDP, in
which each new image is the start of a new episode, with a single action to be
immediately followed by a reward and a terminal state $S=(s_{0},s_{T})$,
$A=(a_{0})$, $G=R=(r_{0})$. The environment can also otherwise be framed as a
continuous MDP, with random state transitions, for which a non-zero discount
factor is redundant $\gamma=0$, $S=(s_{0},s_{1},...)$, $A=(a_{0},a{1},...)$,
$G=R=(r_{0},r{1},...)$. The classification environment can therefore be
generalized as the subset of MDPs with the following three key properties:

  1. 1.

A discrete action space.

  2. 2.

A known reward function.

  3. 3.

Episodic and all states transition to the terminal state OR continuous and all
state transitions are random.

With this MDP model, the classification environment can be approached as an RL
problem and solved using standard RL methods. Furthermore, by setting $A=1$
and $i=\text{index of the correct class}$, we can apply Policy and Entropy
Loss to classification.

$\text{Policy and Entropy Loss
(Supervised)}=-log(p_{i})+\alpha\sum_{j}p_{j}log(p_{j})$ (5)

In fact, if we also set $\alpha=0$ (i.e. remove the entropy term), the Policy
Loss will be exactly equivalent to Cross Entropy Loss 1. Existing supervised
classification training methods can be interpreted as sample efficiency and
training stability tricks that exploit classification’s unique defining
properties. Conversely, Focal Loss can be extended to Actor Critic RL
algorithms by injecting an advantage term $A$ into the scaling term, and an
additional hyperparameter $\alpha$ to scale $p_{i}$ w.r.t. $A$.

$\text{Focal Loss (Actor Critic)}=-(A-\alpha p_{i})^{\gamma}log(p_{i})$ (6)

##  4 Method

###  4.1 Scope Loss

We inspire ourselves from Focal Loss and Policy and Entropy Loss and present
Scope Loss. Like Focal Loss, Scope Loss prevents performance losses from class
imbalances by balancing class gradients relative to certainty. Equivalently,
like Policy and Entropy Loss, Scope Loss promotes exploration by avoiding high
probability policies.

$\text{Scope Loss (Actor Critic)}=-(A-\alpha p_{i})log(p_{i})$ (7)

Where $(A-\alpha p_{i})$ is the scope scaling factor, and $\alpha$ is the
scope hyperparameter that scales $p_{i}$ relative to the magnitude of $A$
(which varies from environment to environment). Where Policy and Entropy Loss
promotes exploration by blindly incorporating an arbitrary amount of
exploration into the agent, Scope Loss indirectly promotes exploration by
setting an upper limit on how quickly an agent can become exploitative. When
$A>0$ a greater certainty $p_{i}$ will decrease the gradient, thereby
preventing exploitation. Note than Scope Loss can be derived by removing the
bias term in 4, which allows the agent to theoretically reach the optimal
policy. We can also derive Scope Loss by setting $\gamma=1$ in 2 (removing the
additional hyperparameter). For supervised classification we can set $A=1$ and
$\alpha=1$.

$\text{Scope Loss (Supervised)}=-(1-p_{i})log(p_{i})$ (8)

Determining the value of $\alpha$ for RL is more tricky, because the advantage
term A distribution is not constant between environments or throughout
training. $\alpha$ should be small enough that the scope term is still
dominated by the advantage term. Instead of manually scheduling and tuning
$\alpha$ for each environment, we can instead fix $\alpha$ and employ z-score
batch normalization of advantages. With this approach we can find a single
optimal value for $\alpha$ for any environment and throughout all stages of
training.

$A:=\frac{A-\mu_{A}}{\sigma_{A}}$ (9)

###  4.2 Scope Loss for PPO

We defined Scope Loss for Actor Critic RL algorithms and supervised
classification. PPO RL algorithms have proven very effective in distributed
settings so would like to extend Scope Loss to PPO algorithms. We can derive a
PPO equivalent following a similar procedure. We start from vanilla Policy
Loss for PPO.

$\text{Policy Loss (PPO)}=-\min(\frac{p_{i}}{p^{k}_{i}}A^{k},\
g(\epsilon,A^{k})),\text{ where
}g(\epsilon,A)=\begin{cases}A(1+\epsilon)&\text{if }A\geq 0\\\
A(1-\epsilon)&\text{if }A<0\end{cases}$ (10)

Where $i$ indicates the index of the chosen action, $k$ indicates the index of
the worker model that is using its experience to update the main model, and
$\epsilon$ is the PPO clipping sensitivity hyperparameter that determines how
different the worker and main policies must be for the loss gradient to be
truncated. We can now add an entropy term for a PPO Policy and Entropy Loss.

$\begin{split}\text{Policy and Entropy Loss
(PPO)}&=-\min(\frac{p_{i}}{p^{k}_{i}}A^{k},\
g(\epsilon,A)^{k})+\alpha\sum_{j}p_{j}log(p_{j})\\\
&=-\min(\frac{p_{i}}{p^{k}_{i}}A^{k},\ g(\epsilon,A^{k}))+\alpha
p_{i}log(p_{i})+\alpha\sum_{j,j\neq i}p_{j}log(p_{j})\end{split}$ (11)

Here, due to the $\min$ term, we cannot factorize to reproduce the scope
factor. By letting $\alpha=0$ we can arrive at Scope Loss for PPO.

$\text{Scope Loss (PPO)}=-\min(\frac{p_{i}}{p^{k}_{i}}A^{k},\
g(\epsilon,A^{k}))+\alpha p_{i}log(p_{i})$ (12)

When $\frac{p_{i}}{p^{k}_{i}}A^{k}$ results as the minimum, we can reformulate
to isolate the PPO version of the scope scaling factor.

$\begin{split}&=-\frac{p_{i}}{p^{k}_{i}}A+\alpha p_{i}log(p_{i})\\\
&=-\frac{p_{i}}{p^{k}_{i}}(A-\alpha p_{i}^{k}log(p_{i}))\end{split}$ (13)

Where $(A-\alpha p_{i}^{k}log(p_{i}))$ is the scope scaling factor. In this
case, the scope scaling factor incorporates the cross entropy of the main
model policy w.r.t. the worker model policy. The scope scaling factor will
balance gradients such that:

  1. 1.

When main model certainty $p_{i}$ is large and $A>0$, the gradient is
decreased to prevent over-exploitation.

  2. 2.

When worker model certainty $p_{i}^{k}$ is smaller than main model certainty
$p_{i}$ and $A>0$, the gradient is decreased to limit the distance between the
models.

  3. 3.

When worker model certainty $p_{i}^{k}$ is greater than main model certainty
$p_{i}$ and $A<0$, the gradient is decreased to limit distance between the
models.

This gradient balancing incorporates ideas from both PPO Loss design and Scope
loss design.

We also extend Focal Loss to the PPO setting, for a fair comparison in our
experiments, by fixing $\gamma=2$ (one of the recommended values in the
original paper), expanding the square in 6, and replacing any resulting Actor
Critic policy loss terms 3 with PPO Policy Loss 10.

$\text{Focal Loss (PPO)}=-\min(\frac{p_{i}}{p^{k}_{i}}A^{k},\
g(\epsilon,A^{k}))A+2\alpha\min(\frac{p_{i}}{p^{k}_{i}}A^{k},\
g(\epsilon,A^{k}))p_{i}-\alpha^{2}p_{i}^{2}log(p_{i})$ (14)

##  5 Results

We first compare Scope Loss with Focal Loss for RL and supervised
classification, and use Policy Loss and Policy and Entropy Loss as baselines.
We demonstrate that Scope Loss generally performs best in both settings with
no tuning, whereas FL fails to learn a meaningful policy in RL. We ran all of
the experiments on a single V100 NVIDIA GPU.

###  5.1 RL

The selected environments include a basket of Atari benchmarks; where
Asterix-V0, Seaquest-v0, Venture-v0, Gravitar-v0 and MontezumaRevenge-v0 were
selected to test for sufficient exploration. Procgen StarPilot Cobbe et al.
[2019] is also included to test for generalization performance. We train the
Impala architecture Espeholt et al. [2018] using PPO with the hyperparameters
outlined in 3. We run each trial on the same environment seed using the PPO
loss functions introduced in the previous section: Policy Loss 10, Policy and
Entropy Loss 11, Scope Loss 12, and Focal Loss 14.

We also estimate advantages with the Generalized Advantage Estimation (GAE)
formulation Schulman et al. [2015] and apply advantage batch normalization 9.
We can see in Figure 2 that Focal Loss consistently under-performs, and fails
to generalize to RL. We can also see that Scope Loss generally outperforms the
other loss functions across a variety of environments, usually taking off the
fastest and finishing and generating one of the top two final policies. Scope
Loss, like Policy and Entropy Loss, finds better policies via exploration, but
Scope Loss also consistently finishes with a slight advantage. We deduce this
final advantage to be a result of the removal of the additional bias in the
Policy and Entropy Loss.

Figure 2: RL Loss Comparison Training Curves

###  5.2 Supervised Classification

We chose the Caltech-256 Griffin et al. [2022] dataset as our benchmark for
supervised classification. Caltech-256 contains 30607 images, of 257 classes
with strong class imbalances. Notably, the classes have a minimum count of 80
examples and a maximum of 827. We train a ResNet101 model from scratch using
an Adam optimizer with the hyperparameters outlined in 4. We run each trial on
the same dataset split using the supervised classification loss functions
introduced in the previous sections: Cross Entropy Loss (CEL) 1, Focal Loss
(FL) 2, Policy and Entropy Loss (PEL) 5, and Scope Loss (SL) 8. We train until
convergence and keep the model checkpoint with the highest validation
accuracy. We then evaluate each trial’s best model over a test set and compute
their precision over each class.

We can see in the table below that Scope Loss achieves the greatest mean
precision, with a 20% relative improvement compared to the other loss
functions. We can also see that Policy and Entropy Loss achieves the lowest
standard deviation of precision, with a 17% relative improvement compared to
the other loss functions. Where Policy and Entropy Loss attempts to maximize
all class precisions equally, by forcing the classification agent to consider
predicting unlikely classes; Scope Loss and Focal Loss attempt to maximize all
class precisions independently without being blinded by class imbalances, by
weighing gradients dynamically, and avoiding exploitation of the easiest
classes. This distinction enables Scope Loss to promote exploration without
risking losses in performance.

Precision Metric | PEL | SL | FL | CEL
---|---|---|---|---
Mean | $0.2303$ | $0.2814$ | $0.2320$ | $0.2383$
Standard Deviation | $0.2116$ | $0.2436$ | $0.2360$ | $0.2471$
Table 1: Supervised Classification Loss Comparison Precision Metrics

We repeat the experiment with a suite of image augmentation techniques and
manual SGD learning rate tuning and scheduling. We employ a cosine decay
scheduler with hyperparameters outlined in 5. We train for 100 of epochs, over
which the cosine scheduler decays the learning rate to 0. We run a separate
trial for the same loss functions and test the best checkpoint models on a
test set.

We can see the same patterns emerge in the table below. Scope Loss results in
the greatest mean precision and Policy and Entropy Loss results in the lowest
standard deviation of precision. In this experiment, however, the differences
in precision metrics between loss trials is much less significant. This
decrease in the effect of our exploratory loss functions is expected, as the
data augmentation effectively reduces the dataset imbalance. The augmentation
techniques remove any "obvious" features that the classification agent could
use to exploit otherwise easier classes.

Precision Metric | PEL | SL | FL | CEL
---|---|---|---|---
Mean | $0.5823$ | $0.5956$ | $0.5723$ | $0.5929$
Standard Deviation | $0.2055$ | $0.2195$ | $0.2227$ | $0.2196$
Table 2: Augmented Supervised Classification Loss Comparison Precision Metrics

##  6 Conclusion

We generalized the problem of supervised classification dataset imbalances to
the reinforcement learning (RL) exploration exploitation trade-off, and
introduced Scope Loss as a generalization of the loss functions used to solve
these two problems. Scope Loss indirectly promotes exploration by preventing
over-exploitation and can easily be applied to both supervised classification
and RL. Scope Loss inspires itself from Policy and Entropy Loss and Focal
Loss, but it only carries a single hyperparameter that requires no tuning, and
it removes the theoretical bias in Policy and Entropy Loss. We then
demonstrated that Scope Loss is a robust loss function that outperforms Focal
Loss, Cross Entropy and Policy Loss with and without Entropy in both RL and
supervised classification.

## References

  * Aubret et al. [2019] A. Aubret, L. Matignon, and S. Hassas.  A survey on intrinsic motivation in reinforcement learning.  _CoRR_ , abs/1908.06976, 2019.  URL http://arxiv.org/abs/1908.06976.
  * Badia et al. [2020] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt, and C. Blundell.  Never give up: Learning directed exploration strategies, 2020.  URL https://arxiv.org/abs/2002.06038.
  * Berner et al. [2019] C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dębiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, C. Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Choshen et al. [2018] L. Choshen, L. Fox, and Y. Loewenstein.  DORA the explorer: Directed outreaching reinforcement action-selection.  _CoRR_ , abs/1804.04012, 2018.  URL http://arxiv.org/abs/1804.04012.
  * Cobbe et al. [2019] K. Cobbe, C. Hesse, J. Hilton, and J. Schulman.  Leveraging procedural generation to benchmark reinforcement learning.  _CoRR_ , abs/1912.01588, 2019.  URL http://arxiv.org/abs/1912.01588.
  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Espeholt et al. [2018] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu.  IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures.  _CoRR_ , abs/1802.01561, 2018.  URL http://arxiv.org/abs/1802.01561.
  * Gokcesu and Gokcesu [2021] K. Gokcesu and H. Gokcesu.  Generalized huber loss for robust learning and its efficient minimization for a robust statistics, 2021.  URL https://arxiv.org/abs/2108.12627.
  * Gregor et al. [2016] K. Gregor, D. J. Rezende, and D. Wierstra.  Variational intrinsic control, 2016.  URL https://arxiv.org/abs/1611.07507.
  * Griffin et al. [2022] G. Griffin, A. Holub, and P. Perona.  Caltech 256, Apr 2022.
  * Hastie et al. [2001] T. Hastie, R. Tibshirani, and J. Friedman.  _The Elements of Statistical Learning: Data Mining, Inference, and Prediction (Second Edition)_.  Springer, Stanford, 2001.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Jomaa et al. [2019] H. S. Jomaa, J. Grabocka, and L. Schmidt-Thieme.  Hyp-rl : Hyperparameter optimization by reinforcement learning, 2019.  URL https://arxiv.org/abs/1906.11527.
  * Li et al. [2021] M. Li, X. Zhang, C. Thrampoulidis, J. Chen, and S. Oymak.  Autobalance: Optimized loss functions for imbalanced data.  _Advances in Neural Information Processing Systems_ , 34:3163–3177, 2021.
  * Lin et al. [2019] E. Lin, Q. Chen, and X. Qi.  Deep reinforcement learning for imbalanced classification.  _CoRR_ , abs/1901.01379, 2019.  URL http://arxiv.org/abs/1901.01379.
  * Lin et al. [2017a] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2980–2988, 2017a.
  * Lin et al. [2017b] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár.  Focal loss for dense object detection, 2017b.  URL https://arxiv.org/abs/1708.02002.
  * Mahmud et al. [2022] T. Mahmud, N. Frumkin, and D. Marculescu.  RL-tune: A deep reinforcement learning assisted layer-wise fine-tuning approach for transfer learning.  In _First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022_ , 2022.  URL https://openreview.net/forum?id=_trbCTrj9xs.
  * Mnih et al. [2013] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Mnih et al. [2016] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , pages 1928–1937. PMLR, 2016.
  * Obando-Ceron et al. [2019] J. S. Obando-Ceron, V. Romero-Cano, and W. M. Toro.  Exploiting the potential of deep reinforcement learning for classification tasks in high-dimensional and unstructured data.  _CoRR_ , abs/1912.09595, 2019.  URL http://arxiv.org/abs/1912.09595.
  * Radford et al. [2018] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * Raffel et al. [2020] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67, 2020.  URL http://jmlr.org/papers/v21/20-074.html.
  * Ramesh et al. [2022] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Redmon et al. [2016] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. Denton, S. K. S. Ghasemipour, B. K. Ayan, S. S. Mahdavi, R. G. Lopes, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _arXiv preprint arXiv:2205.11487_ , 2022.
  * Schulman et al. [2015] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel.  High-dimensional continuous control using generalized advantage estimation, 2015.  URL https://arxiv.org/abs/1506.02438.
  * Shrivastava et al. [2016] A. Shrivastava, A. Gupta, and R. B. Girshick.  Training region-based object detectors with online hard example mining.  _CoRR_ , abs/1604.03540, 2016.  URL http://arxiv.org/abs/1604.03540.
  * Silver et al. [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Sung [1996] K.-K. Sung.  Learning and example selection for ob ject and pattern detection.  _AITR_ , 1572, 1996.  URL https://dspace.mit.edu/bitstream/handle/1721.1/6774/AITR-1572.pdf.
  * Vinyals et al. [2019] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Wiering et al. [2011] M. A. Wiering, H. van Hasselt, A.-D. Pietersma, and L. Schomaker.  Reinforcement learning algorithms for solving classification problems.  _ADPRL_ , 2011.  URL https://www.ai.rug.nl/~mwiering/GROUP/ARTICLES/rl_classification.pdf.
  * Wu et al. [2019] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick.  Detectron2.  https://github.com/facebookresearch/detectron2, 2019.
  * Zhang et al. [2018] T. Zhang, M. Huang, and L. Zhao.  Learning structured representation for text classification via reinforcement learning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), Apr. 2018.  doi: 10.1609/aaai.v32i1.12047.  URL https://ojs.aaai.org/index.php/AAAI/article/view/12047.
  * Zhao et al. [2017] D. Zhao, Y. Chen, and L. Lv.  Deep reinforcement learning with visual attention for vehicle classification.  _IEEE Transactions on Cognitive and Developmental Systems_ , 9:356–367, 2017.
  * Ziebart [2010] B. D. Ziebart.  _Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy_.  Carnegie Mellon, Pittsburgh, 2010.
  * Zoph and Le [2016] B. Zoph and Q. V. Le.  Neural architecture search with reinforcement learning, 2016.  URL https://arxiv.org/abs/1611.01578.

##  7 Appendix

###  7.1 RL Hyperparameters

Environment | $lr$ | ${\alpha}_{value}$ | ${\alpha}_{entropy}$ | ${\alpha}_{scope}$ | ${\alpha}_{focal}$ | $\gamma$ | $\lambda$
---|---|---|---|---|---|---|---
Pong-v0 | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
Breakout-v0 | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
Asterix-v0 | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
Seaquest-v0 | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
Venture-v0 | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
Gravitar-v0 | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
MontezumaRevenge-v0 | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
Procgen StarPilot (Hard 500) | $10^{-4}$ | 0.5 | 0.01 | 0.01 | 2 | 0.999 | 0.95
Caltech 256 Classification | $10^{-3}$ | 0.5 | 0.05 | NA | NA | 0 | 0
Table 3: Hyperparameters used for all RL experiments.

###  7.2 Supervised Classification Hyperparameters

$lr$ | $\alpha_{scope}$ | $\alpha_{entropy}$ | $\alpha_{focal}$
---|---|---|---
$10^{-3}$ | 1 | $8*10^{-6}$ | 2
Table 4: Hyperparameters used in the first Supervised Classification experiment. $lr$ | $epochs$ | $\alpha_{scope}$ | $\alpha_{entropy}$ | $\alpha_{focal}$ | $color$
---|---|---|---|---|---
$10^{-2}$ | 100 | 1 | $8*10^{-6}$ | 2 | $\mathcal{N}(1,0.3)$
contrast | brightness | sharpness | crop | flip |
$\mathcal{N}(1,0.1)$ | $\mathcal{N}(1,0.1)$ | $\mathcal{N}(1,0.3)$ | $\mathcal{U}$ | $Bernoulli(0.5)$ |
Table 5: Hyperparameters used in the augmented Supervised Classification
experiment.