# Weakly Supervised Classification Using Group-Level Labels

Guruprasad Nayak<EMAIL_ADDRESS>University of MinnesotaMinneapolisMNUSA ,
Rahul Ghosh<EMAIL_ADDRESS>University of MinnesotaMinneapolisMNUSA ,
Xiaowei Jia<EMAIL_ADDRESS>University of PittsburghPittsburghPAUSA and
Vipin Kumar<EMAIL_ADDRESS>University of MinnesotaMinneapolisMNUSA

(2021)

###### Abstract.

In many applications, finding adequate labeled data to train predictive models
is a major challenge. In this work, we propose methods to use group-level
binary labels as weak supervision to train instance-level binary
classification models. Aggregate labels are common in several domains where
annotating on a group-level might be cheaper or might be the only way to
provide annotated data without infringing on privacy. We model group-level
labels as Class Conditional Noisy (CCN) labels for individual instances and
use the noisy labels to regularize predictions of the model trained on the
strongly-labeled instances. Our experiments on real-world application of land
cover mapping shows the utility of the proposed method in leveraging group-
level labels, both in the presence and absence of class imbalance.

††copyright: acmcopyright††journalyear: 2021††doi:
xx.xxxx/xxxxxxx.xxxxxxx††conference: DeMaL ’21: International Workshop on
Data-Efficient Machine Learning (DeMaL) at ACM Confierence on Knowledge
Discovery and Data Mining (KDD) ; August 14–18, 2021; Singapore††booktitle:
DeMaL ’21: International Workshop on Data-Efficient Machine Learning (DeMaL),
August 14–18, 2021, Singapore††price: xx.xx††isbn: xxx-x-xxxx-XXXX-X/xx/xx

##  1\. Introduction

Traditionally, classification models are trained using a set of training
samples for which features and their corresponding class labels are known.
However, in several practical applications, we are faced with two significant
challenges, the first of which is the challenge of inadequate training data.
The number of samples required to train an accurate classifier increases with
(a) complexity of the chosen classifier and (b) the heterogeneity of the
feature space. For example, consider the example of using satellite images to
map the extent of urban areas on the surface of the Earth. Urban areas (and
non-urban areas) are heterogeneous enough globally that they require complex
models like deep neural networks to predict them accurately. This creates the
need for large number of training samples which is hard to meet in this
application since manual annotation is the primary source of obtaining labeled
data. Similar challenges occur in several other application areas of
predictive modeling where the scale and variety of data instances is large.

The second key challenge that we address in this paper is that of class
imbalance. In many applications, it is common to have an imbalance in the
number of instances belonging to the two classes like in the case of spam
identification (Liu et al., 2003) or land cover monitoring problems like urban
areas (Potere et al., 2009) or burned forests (Mithal et al., 2017). Thus, the
learning algorithms have to optimize for the performance on the rare class
unlike most standard classifiers that optimize for measures like accuracy.

While each of these two challenges are significant by themselves, the problem
becomes much harder when they occur simultaneously, as is the case in many
practical applications such as land cover mapping. In the case of inadequate
training data, the number of instances from the rare class might be too few or
the class imbalance might be misrepresented owing to limited samples.

Figure 1. Example of group-level labels for the application of urban area
mapping. The group-labels are obtained from a manual annotator who marks a
general region as having buildings, which creates the group-level labels, as
opposed to marking actual boundaries around each building, which would
constitute the actual instance-level labels. Note that obtaining group-level
labels is less expensive than instance-level labels.

One way to handle the challenge of not having adequate training samples is to
use weak supervision. In a weakly-supervised learning (Zhou, 2017) scenario,
you have very few (if any) training samples that have exact labels
corresponding to the target variable. However, you have plenty of weakly-
labeled instances i.e you have an imperfect version of the target variable for
these instances. The idea is that, by modeling the imperfection in the weak
labels, we can mitigate the lack of (strongly-labeled) training data. In many
binary classification problems, weak supervision might be available in the
form of group-level labels. Aggregate labels are common in several domains
where annotating on a group-level might be cheaper or might be the only way to
provide annotated data without infringing on privacy. Figure 1 shows an
example of group-labels in the application of mapping urban areas using
satellite data. It is far less expensive to annotate that a general region has
houses and hence is urban than to individually mark a polygon around each
house. Another example of cheap group-level labels, in the context of land
cover mapping is when you are trying to classify instances at a finer
resolution (e.g, 10m *10m pixels), but you have weak labels available at
coarser resolution (e.g, 250m*250m pixels). This happens because most existing
work has been done on creating coarser resolution maps and hence many accurate
products exist at this resolution. Thus, each of the 625 10m*10m pixels within
a 250m*250m pixel become part of the same group, for which a single binary
label is available. A good example of when privacy concern leads to group-
level labeling is census data when information is aggregated at the level of
administrative boundaries. Group-level labels are common in several other
domains including drug activity prediction (Dietterich et al., 1997), object
recognition (Kuck and de Freitas, 2012), image retrieval (Yang and Lozano-
Perez, 2000) and text categorization (Andrews et al., 2003). Note that in our
case, the group-level labels are binary as well. However, distribution of
instance labels within a group and the size of the group are both variable.

Based on prior research, there can be two approaches to using group-level
binary labels for instance-level classification. The first is to treat it as
multiple instance learning problem (Cheplygina et al., 2015) where the goal is
to make group-level predictions. An aggregation function (eg. max or mean) is
assumed to generate group-level predictions from predictions on constituting
instances and the classifier is optimized for group-level performance metrics.
This approach is limiting since it does not optimize for instance-level
performance. The second approach is to propagate the group-level label to all
instances within the group and treat this as a noisy label for learning an
instance-level classifier. Most approaches for handling label noise rely on
the Class Conditional Noise (CCN) assumption. Class Conditional Noise (CCN) is
a well-studied model for learning with label noise (Menon et al., 2015; Mithal
et al., 2017; Raykar et al., 2010; Liu et al., 2003). Class conditional
independence is also a key assumption in several prior papers on using group-
level supervision with label proportions (Quadrianto et al., 2009; Patrini et
al., 2014). In the group-label setting, this assumption would mean that the
class-conditional distribution of the instances is independent of the groups.
However, this assumption may not completely hold in practice. For example, in
figure 1, there are certain kinds of non-urban areas like farms that are much
more likely to be adjacent to urban areas and hence to be a part of urban-
labeled groups than otherwise.

In this paper, we consider the scenario where we have very few strongly
labeled samples and a large number of weakly labeled instances for training.
The strongly-labeled instances are the regular training set used for learning
classification models, with features and labels for individual instances. The
weakly-labeled instances have instance-level features, but labels are only
available on a group-level. We propose to train a regular classifier using the
strongly-labeled instances and regularize it using its estimated performance
on the weakly-labeled instances through CCN modeling. While the CCN modeling
of group-labels for constituting instances may not exactly hold in practice
and using it by itself might be limiting, as explained previously, using it
together with strong supervision can help mitigate this limitation. The
strongly-labeled instances are free of the label noise and hence have no
bearing on the CCN modeling of the weakly-labeled instances, while the weakly-
labeled instances provide the benefit of being available in plenty which the
strong supervision lacks. Our approach leverages this complementarity between
strong and weak supervision to train robust and accurate models.

###  1.1. Contributions

In a nutshell, the contributions of this work are as follows:

  1. (1)

We formalize the commonly-occuring weak label learning scenario of binary
group-level labels in the context of instance level binary classification.

  2. (2)

We propose a principled approach to use group-level labeled instances (in
addition to instance-level labeled instances) to train classification model to
optimize for instance-level classifier performance. By modeling group-labels
as CCN labels for constituting instances, we leverage prior research to
optimize instance-level metrics.

  3. (3)

The proposed approach can also optimize for class imbalance, if it is known to
exist in the data. This is especially relevant in the context of inadequate
training data when the limited number of strongly-labeled instances may
misrepresent the class imbalance or might not have enough samples from the
rare class.

  4. (4)

Our method demonstrably leverages the group-level labels for landcover
classification tasks (urban areas, burned vegetation) which are problems of
great environmental and societal significance.

##  2\. Related Work

###  2.1. Learning with group-level supervision

Group-level supervision has been extensively studied before, most prominently
as the paradigm of multiple instance learning (MIL), first introduced in
(Dietterich et al., 1997). In a MIL problem setting, the goal is to learn a
classifier to predict the label for groups of instances. The classic MIL
assumption (Cheplygina et al., 2015) is that the binary label of the group is
the disjunction of the unknown binary labels of its constituting instances i.e
the presence of a positive label instance within a group makes the group label
positive. The classic MIL assumption has been extended to more general
threshold settings like in (Tao et al., 2004). Alternately, the MIL problem
can be solved using single instance classification algorithms by learning a
embedding function for groups like in (Chen et al., 2006). However, unlike the
classic MIL setting, our goal is to predict the instance-level labels, not the
group-label.

More recently, group-level supervision, in the form of label proportions, has
been studied for learning instance-level classifiers. In this setting, the
proportion of constituent instances belonging to the each class is known for
each training group. There are two kinds of approaches in this category -
probabilistic and non-probabilistic. The probabilistic models (Quadrianto et
al., 2009; Patrini et al., 2014) assume that the class-conditional
distribution of the data is independent of the bags. Using this assumption,
they estimate a sufficient statistic from the label proportions on the
training groups and use it to maximize the likelihood for instance-level
prediction. The non-probabilistic approach (Kuck and de Freitas, 2012; Felix
et al., 2013) is similar to the threshold-based MIL setting where the
underlying instance labels are generated to remain consistent with the label
proportions known for the corresponding groups. While we borrow the class
conditional independence modeling assumption from these label proportion
methods, our method does not assume the knowledge of proportion of labels on
training groups, which might be harder to come by in many applications such as
land cover mapping, where getting binary group labels might be easier.
Moreover, to the best of our knowledge, none of these methods do rare class
optimization on instance-level prediction.

###  2.2. Learning with noisy labels

Learning with noisy labels is a problem of great practical significance and
has been widely studied in literature (Frénay and Verleysen, 2013; Blum and
Mitchell, 1998). A large number of papers try to handle this problem by
assuming a class conditional noise (CCN) model, where the probability of
corrupting the label is the same for all instances within a given class. Under
the CCN assumption, some approaches like (Natarajan et al., 2013; Patrini et
al., 2017) try to estimate a surrogate loss which is corrected for the noise
in the labels. Other CCN based approaches try to estimate the noise rate
before optimizing the desired metric on the noisy data (Menon et al., 2015;
Mithal et al., 2017). This approach has also been used in crowdsourcing
settings (Raykar et al., 2010) where we have multiple sources of noisy labels,
each with its own noise rate. Recently, (Goldberger and Ben-Reuven, 2016)
proposed to augment deep neural networks with a noise adaptation layer that
learns the noise rates along with the neural network parameters in an EM-style
framework. The second category of approaches in learning with noisy labels
train the classifier, usually a neural network, to learn from clean
information initially before subjecting it to noisy labels. Some of these
methods (Li et al., 2017; Veit et al., 2017) require access to a clean data
set to pre-train the neural network while other more recent methods like (Han
et al., 2018) use the memorization effect (Arpit et al., 2017) of deep
learning models where these models are known to learn the clean information in
initial iterations before overfitting on the noisy labels. Again, most methods
in the noisy label category, do not optimize for rarity, except for (Mithal et
al., 2017). Also, to the best of our knowledge, these methods have not been
applied to the group-label weak supervision problem.

##  3\. Method

###  3.1. Problem setting

We consider a two-class classification problem where the goal is to learn a
model that predicts the binary label $y\in\\{0,1\\}$ for a test instance given
its features $\bm{x}$. During the training phase, we have two sets of
instances:

  * (a)

Strongly-labeled data set of $N_{s}$ instances
$\\{(\bm{x_{i}},y_{i})\\}_{i=1}^{N_{s}}$ having features $\bm{x_{i}}$ and
labels $y_{i}$ for each instance $i$.

  * (b)

Weakly-labeled data set of $N_{g}$ instances
$\\{(\bm{x_{i}},g_{i})\\}_{i=1}^{N_{g}}$ having features $\bm{x_{i}}$ and
labels $g_{i}\in\\{0,1\\}$ of the corresponding group for each instance $i$.

Note that the distribution of true instance labels within a group and the size
of the group are both variable. Also note that due to the relatively high cost
involved in labeling individual instances, it is more likely that
$N_{s}<<N_{g}$.

Figure 2. Graphical model diagram for Class Conditional Noisy (CCN) labels
$g$, as they relate to features $\bm{x}$ and true labels $y$. The model
assumes $g\perp\\!\\!\\!\perp\bm{x}|y$. The conditional probability $g|y$ is
encoded by parameters $\alpha$ and $\beta$, while parameter $w$ defines the
distribution $y|\bm{x}$.

###  3.2. Modeling group-labels as CCN labels

Class Conditional Noise (CCN) is a widely studied model for noisy labels
(Menon et al., 2015), (Natarajan et al., 2013), (Blum and Mitchell, 1998).
Figure 2 shows a graphical model for the CCN model. The key assumption of the
CCN model is $g\perp\\!\\!\\!\perp\bm{x}|y$, where $g,y,\bm{x}$ denote the
corrupted label, true label and the feature vector respectively. In other
words, given a particular class, the probability of the noisy label making an
error is the same for all instances within the class. For a two-class problem,
the noise rates can be encoded as,

$\begin{split}\alpha&=Pr(g=0|y=1)\\\ \beta&=Pr(g=1|y=0)\end{split}$

####  3.2.1. Assumptions for using group-level labels as weak labels for
instance-level classification

Group-level labels can be used to learn predictive models on an instance-level
under two assumptions.
(a) Instance-level pseudo-labels are CCN CCN assumption assumes that the
probability of flipping the class label on an instance is the same for all
instances in the class irrespective of their feature values. Since we are
propagating the group-label to all instances within the group, the 0-labeled
instances within a 1-label group are being flipped to 1 (and vice-versa). For
the CCN assumption to hold, these flipped instances should be no different in
the feature space than instances that don’t get flipped i.e 0-label instances
within 0-label bags. In other words, the 0-label instances within 1-label bags
and those within 0-label bags come from the same distribution (and vice-versa
for 1-label instances).
(b). Sum of noise rates is less than 1 (Blum and Mitchell, 1998; Natarajan et
al., 2013) This assumption states i.e $\alpha+\beta<1$, which can also be
written as $((1-\alpha)+(1-\beta))/2>0.5$. $1-\alpha$ is the probability that
the true label is 0 when the CCN label is 0 (negative class accuracy) and
$1-\beta$ is the probability that the true label is 1 when the CCN label is 1
(positive class accuracy). So, intuitively, this assumption is saying that the
probability of the CCN label being true is better than 0.5 i.e the probability
that a noisy label generated at random would be correct i.e. the CCN label
itself is a weak learner.

$\begin{split}&\alpha+\beta<1\\\ &\implies
Pr(y_{g}=1|y=0)+Pr(y_{g}=0|y=1)<1\\\
&\implies\frac{\sum_{y_{i}=1\&y_{g_{i}}=0}1}{\sum_{y_{i}=0}1}+\frac{\sum_{y_{i}=0\&y_{g_{i}}=1}1}{\sum_{y_{i}=1}1}<1\\\
\end{split}$

i.e the sum of the fraction of 1-label instances that fall in 0-bags and the
fraction of 0-label instances that fall in 1-bags should be less than 1.

###  3.3. Training objective

The key idea of our weakly-supervised classification algorithm is to
regularize the performance of a classifier trained on the strongly-labeled
training set with its performance on the weakly-labeled instances. For the
classification model, we consider a differentiable function $f(\bm{x})$ that
models $Pr(y=1|\bm{x})$ and a threshold $\gamma$ that converts it to a binary
prediction $\hat{y}$ i.e $\hat{y_{i}}=1$ if $f(\bm{x})>\gamma$ and $0$
otherwise. Because the step-function is not differentiable at the threshold,
we use a soft approximation to the step function, that thresholds the numeric
value $f(\bm{x})$ at $\gamma$ as shown below

(1) $\hat{y^{\gamma}}=\frac{1}{1+e^{-s*(f(\bm{x})-\gamma)}}$

where $s$ is a large constant that controls how quickly $\hat{y_{\gamma}}$
goes to 0/1 on either end of the threshold $\gamma$.
The training objective is a sum of the objective over the two kinds of
training instances - strong ($O_{s}$) and weak ($O_{g}$). The optimal
parameters $\hat{w}$ for $f$ are estimated as,

(2)
$\begin{split}\hat{w}&=\operatorname*{\arg\\!\max}_{w}\max_{\gamma}O(w;\gamma)\\\
&=\operatorname*{\arg\\!\max}_{w}\displaystyle\left[\max_{\gamma}O_{s}(w;\gamma)+\lambda
O_{g}(w;\gamma)\displaystyle\right]\end{split}$

where $\lambda$ is a weighting parameter fixed using cross-validation. We use
gradient descent to optimize the above objective function. To keep the
objective function differentiable, the maximum over different values of
$\gamma$ is computed using a softmax function.

####  3.3.1. $O_{s}$: Objective on strongly-labeled instances

The performance of model $f$ with parameters $w$ for a given threshold
$\gamma$ over the strongly-labeled samples $O_{s}(w;\gamma)$ is simply the
classification accuracy, which can be computed as,

(3)
$O_{s}(w;\gamma)=(1/N_{s})*\sum_{i=1}^{N_{s}}\hat{y^{\gamma}_{i}}y_{i}+(1-\hat{y^{\gamma}_{i}})(1-y_{i})$

####  3.3.2. $O_{g}$: Objective on weakly-labeled instances

Prior research has shown that it is possible to optimize certain performance
metrics using surrogate losses that require only CCN labels (not the true
labels). We use the surrogate versions of metrics proposed in these works to
define the objective on weakly-labeled instances. We consider 2 cases,
depending on the presence or absence of class imbalance.

  1. (1)

_Balanced case:_ In (Menon et al., 2015), the authors showed that the accuracy
of a classifier computed using CCN labels can be used as a surrogate for its
true accuracy. Thus, for the balanced class scenario, we define our objective
on weakly-labeled instances to be the surrogate accuracy i.e

(4)
$O_{g}(w;\gamma)=(1/N_{g})*\sum_{i=1}^{N_{g}}\hat{y^{\gamma}_{i}}g_{i}+(1-\hat{y^{\gamma}_{i}})(1-g_{i})$

  2. (2)

_Imbalanced case:_ Similarly, for rare class scenarios, where metrics like
precision and recall are more important, the authors in (Mithal et al., 2017)
showed that the geometric mean of precision and recall (G-measure) can be
optimized using a surrogate, computed using only the knowledge of CCN labels.
This surrogate for G-measure can be used as our objective on weakly-labeled
instances like so,

(5)
$\begin{split}O_{g}(\gamma)&=(Pr(g=1|\hat{y}=1)-\beta)^{2}{Pr(\hat{y}=1)}\\\
&=(\frac{\sum_{i=1}^{N_{g}}\hat{y^{\gamma}_{i}}g_{i}/N_{g}}{\sum_{i=1}^{N_{g}}\hat{y^{\gamma}_{i}}/N_{g}}-\beta)^{2}\sum_{i=1}^{N_{g}}\hat{y^{\gamma}_{i}}/N_{g}\\\
&=\frac{(\sum_{i=1}^{N_{g}}\hat{y^{\gamma}_{i}}g_{i}-\beta\sum_{i=1}^{N_{g}}\hat{y^{\gamma}_{i}})^{2}}{N_{g}\sum_{i=1}^{N_{g}}\hat{y^{\gamma}_{i}}}\end{split}$

where $\beta=Pr(g=1|y=0)$ is a property of the weak label that can be
estimated using the strongly labeled samples.

_Estimating_ $\beta$: To estimate $\beta=Pr(g=1|y=0)$, we follow the same
procedure as the authors in (Mithal et al., 2017). Ideally, one would want
sufficient number of instances with both $y$ and $g$ labels. However, since
this is not the case, we use the strongly-labeled samples to estimate
sufficient samples with $y=0$. We learn a classifier with just the strongly-
labeled samples. Even if this classifier is not very accurate, we expect the
instances with very low probability assigned from this classifier to be mostly
negative. Thus, we take the bottom 5% of weakly-labeled instances with lowest
values of estimated $Pr(y=0|x)$ using this classifier trained with only
strongly-labeled instances. The estimate for $\beta$ is the fraction of these
instances with $g=1$.

##  4\. Evaluation

We evaluate the methods proposed in this paper on 7 data sets from two real
world applications - urban area detection and burned area detection, that use
satellite-collected observations of different locations to automatically track
changes changes on the surface of the Earth. Group-level labels are derived
from auxiliary satellite products, that are easily available. Since many of
these satellite products are available at a coarser resolution in comparison
to the locations being classified, they act as group-level labels for the
classifier being trained. For the urban extent mapping application, we
consider group-level label derived from the Night time light satellite product
(DAAC, 2009). The average group size in this case is about 2000. Similarly,
for the burned area mapping problem, we use another satellite signal called
Active Fire (DAAC, 2009). The average group size in this case is about $9$.
More details about these data sets are provided in table 1. The satellite
signals used in the paper are freely available from (DAAC, 2009) and will also
be made available through the author’s personal website after publication,
along with the code for the algorithms proposed in this paper.

Table 1. Summary of data sets used in this paper. Data set | Phenomenon |  Strongly labeled instances |  Average group size |  Weakly labeled instances | $\alpha$ | $\beta$ |  skew = #neg/#pos
---|---|---|---|---|---|---|---
D1 | Fires, CA |  200 |  8.91 |  20535 | 0.338 | 0.064 |  1
D2 | Fires, MT |  200 |  8.94 |  20506 | 0.309 | 0.0057 |  1
D3 | Fires, GA |  100 |  8.82 |  582 | 0.888 | 0.04 |  1
D4 | Urbanization, MN |  20 |  2034.67 |  11190685 | 0.067 | 0.0039 |  1
D5 | Fires, CA |  200 |  8.91 |  17482 | 0.487 | 0.029 |  4.81
D6 | Fires, MT |  200 |  8.94 |  17428 | 0.435 | 0.028 |  4.80
D7 | Fires, GA |  100 |  8.82 |  977 | 0.898 | 0.051 |  2.82

###  4.1. Comparison with baselines

For our experiments, we compare the performance of our method (called _WeaSL_)
against the following baselines:

  * (a)

OnlyStrong Only use the strongly labeled data and no weak supervision to learn
the model parameters.

  * (b)

MIL (balanced) (Ray and Craven, 2005) MI/LR is a multi-instance learning
algroithm proposed in (Ray and Craven, 2005), where the goal is to predict
labels for groups of instances. The algorithm learns an instance level
classifier and uses a mean-aggregation on the prediction of instances within a
bag to predict the group-label. We learn MIL on groups of instances in the
weakly-labeled data set and use the learned instance-level classifier within
MIL to make predictions on the test set. We also experimented with max-
aggregation for generating the group-label from instance labels, however,
mean-aggregation seemed to work better for our data sets.

  * (c)

MIL (imbalanced) This is similar to MIL (balanced) (Ray and Craven, 2005),
however, instead of optimizing for accuracy on the group-level predictions,
this method optimizes for F-measure. Note that optimizing for F-measure on the
group-predictions is not equivalent to optimizing F-measure on the instance
level. In an extreme situation, the group-level labels might be balanced while
the instances themselves can be skewed, as in the group-purity experiment in
the supplement.

  * (d)

OnlyWeak (Menon et al., 2015; Mithal et al., 2017) Variant of WeaSL where we
ignore the strongly-labeled instances and only optimize the performance
(accuracy or G-measure) over weakly-labeled instances. Thus, instead of
considering $O_{s}+\lambda O_{w}$ in equation 2, our objective function is
simply $O_{w}$ in this case. This method is similar to the learning algorithms
used to learn with CCN labels in (Menon et al., 2015; Mithal et al., 2017).

  * (e)

Co-teaching (Han et al., 2018) This is another approach of handling noisy
labels that relies on the memorization effects of the classification models
instead of assuming a CCN model for label noise. This is also a method that
uses only weak supervision and no strong supervision.

Unless specified otherwise, the base classifier for all methods in this
section is a neural network with 2 hidden layers with 128 and 64 units
respectively.

####  4.1.1. Evaluation metrics:

We evaluate the performance on the balanced data sets using accuracy and on
the imbalanced data sets using F-measure.

####  4.1.2. Balanced data sets:

Table 2 reports the average accuracy over 5 iterations for the balanced data
sets in table 1. Since we have very few strongly labeled samples, the
_OnlyStrong_ baseline does quite poorly, unlike the other methods that
leverage weakly labeled data. Methods _MIL (balanced)_ and _OnlyWeak_ ignore
the strongly-labeled instances and use only weakly-labeled instances for
training. _MIL (balanced)_ is designed to predict group-labels and thus, does
not optimize for instance-level prediction performance. In comparison,
_OnlyWeak_ optimizes for instance-level accuracy through CCN modeling of the
group-label for individual instances. Thus, it mostly seems to do better than
MIL (balanced). _Co-teaching_ is another method that uses only weak
supervision through a different noise handling strategy. Thus, it performs
similar to _OnlyWeak_. However, since the noise modeling assumption may not
exactly hold in the data, these methods fall short of _WeaSL_ that benefits
from using the weakly-labeled data in conjunction with the strongly-labeled
data.

Table 2. Comparison with baselines: balanced data sets (Accuracy) Method | D1 | D2 | D3 | D4
---|---|---|---|---
OnlyStrong | $\underset{(0.013)}{0.741}$ | $\underset{(0.011)}{0.754}$ | $\underset{(0.022)}{0.688}$ | $\underset{(0.075)}{0.681}$
OnlyWeak | $\underset{(0.004)}{0.763}$ | $\underset{(0.002)}{0.769}$ | $\underset{(0.022)}{0.529}$ | $\underset{(0.009)}{0.791}$
WeaSL | $\underset{(0.005)}{\textbf{0.819}}$ | $\underset{(0.003)}{\textbf{0.805}}$ | $\underset{(0.008)}{\textbf{0.714}}$ | $\underset{(0.004)}{\textbf{0.847}}$
MIL (balanced) | $\underset{(0.035)}{0.632}$ | $\underset{(0.013)}{0.655}$ | $\underset{(0.001)}{0.554}$ | $\underset{(0.005)}{0.714}$
Co-teaching | $\underset{(0.002)}{0.757}$ | $\underset{(0.001)}{0.763}$ | $\underset{(0.031)}{0.511}$ | $\underset{(0.001)}{0.803}$

####  4.1.3. Imbalanced data sets:

Table 3 reports the average F-measure over 5 iterations for the imbalanced
data sets in table 1. As shown in table 1, note that unlike the balanced case
where the number of instances in both classes were the same, here, the
majority class has more than $90\%$ of the instances in some cases. Similar to
the balanced data sets, the performance of _OnlyStrong_ is relatively poor
because it only uses limited strongly-labeled training samples. _MIL
(balanced)_ and _MIL(imbalanced)_ optimize for group-level prediction
performance and hence they do not perform as well as our method that
explicitly optimizes for instance-level precision and recall. Note that _MIL
(imbalanced)_ still optimizes F-measure on the group-level and thus, performs
sub-optimally on instance-level predictions. Similar to _MIL_ , _OnlyWeak_ and
_Co-teaching_ consider only weak supervision for training. But instead of
optimizing group-level performance, _OnlyWeak_ models the group-labels as CCN
labels for instances and optimizes instance-level G-measure. Thus, its
performance is significantly better than _MIL_. However, it still falls short
of _WeaSL_ that not only optimizes for rarity but also uses strongly-labeled
instances that helps it to overcome the limitations of CCN modeling.

Table 3. Comparison with baselines: imbalanced data sets (F-measure) Method | D5 | D6 | D7
---|---|---|---
OnlyStrong | $\underset{(0.027)}{0.462}$ | $\underset{(0.0187)}{0.4918}$ | $\underset{(0.044)}{0.5947}$
OnlyWeak | $\underset{(0.008)}{0.461}$ | $\underset{(0.008)}{0.437}$ | $\underset{(0.025)}{0.429}$
WeaSL | $\underset{(0.002)}{\textbf{0.602}}$ | $\underset{(0.012)}{\textbf{0.629}}$ | $\underset{(0.003)}{\textbf{0.623}}$
MIL (balanced) | $\underset{(0.044)}{0.294}$ | $\underset{(0.043)}{0.334}$ | $\underset{(0.056)}{0.215}$
MIL (imbalanced) | $\underset{(0.016)}{0.414}$ | $\underset{(0.021)}{0.454}$ | $\underset{(0.048)}{0.355}$
Co-teaching | $\underset{(0.003)}{0.560}$ | $\underset{(0.004)}{0.572}$ | $\underset{(0.036)}{0.225}$

Table 4 reports the estimated $\beta$ along with its true value for the 3 dats
sets. As we can clearly see, the estimates are close enough to the actual
values.

Table 4. Estimates of noise rate $\beta=Pr(g=1|y=0)$ | D5 | D6 | D7
---|---|---|---
$\beta$ (actual) | 0.029 | 0.028 | 0.051
$\beta$ (estimated) | 0.027 | 0.035 | 0.041

###  4.2. Further analysis

In the introduction, we motivated the use of weak labels by raising the issues
of limited strong supervision, complementarity between strong and weak
supervision, the need to use more complex models to solve challenging problems
and the heterogeneity in the data population. In this section we will present
additional experiments to analyze each one of these factors.

####  4.2.1. Effect of more strongly-labeled training samples

_WeaSL_ tries to leverage the abundant weakly labeled data to improve the
performance of a model that just uses strongly labeled data, typically
available in small numbers. As the number of strongly-labeled instances
increases, the gain obtained by using the weakly-labeled instances reduces. We
demonstrate this empirically by increasing the number of strongly labeled
instances in data set D2 from $10$ up to $10000$. From the results in figure
3, we observe that the $accuracy$ values for all methods increase as the
number of strongly labeled samples are increased. Finally, given enough number
of samples, _OnlyStrong_ catches up to _WeaSL_ in terms of its accuracy value.

Figure 3. Gain with increasing number of strongly-labeled samples: The x-axis
shows the number of strongly labeled samples provided to the algorithms.

####  4.2.2. Complementarity between strong and weak supervision

As explained in the introduction, by making use of strong and weak supervision
together, _WeaSL_ attempts to overcome the limitations of both. Here, we try
to empirically demonstrate the complementarity between strong and weak
supervision and its benefit to _WeaSL_. Figure 4 shows a Venn diagram of the
errors made by models trained with strong supervision (_OnlyStrong_), weak
supervision (_OnlyWeak_) and both (_WeaSL_) on the data set D1. Firstly, we
can see that the total number of errors made by _WeaSL_ is less than
_OnlyStrong_ and _OnlyWeak_. Secondly, a significant number of errors made
individually by either _OnlyStrong_ or _OnlyWeak_ are not made by _WeaSL_. In
particular, $67\%$ of the errors made by _OnlyStrong_ are not made by _WeaSL_.
Similarly, $33\%$ of the errors made by _OnlyWeak_ are not made by _WeaSL_.
Thirdly and perhaps, most importantly, only a very small fraction of the
errors made by _WeaSL_ are new i.e not made by either _OnlyStrong_ or
_OnlyWeak_. Thus _WeaSL_ significantly reduces the errors relative to both
_OnlyStrong_ and _OnlyWeak_ , while not adding a significant number of new
errors.

Figure 4. Venn diagram of errors made on D1 by different models. _WeaSL_
leverages the complementarity between models trained with only strong and only
weak supervision separately.

####  4.2.3. Increasing model complexity with weak labels

The second key motivation for using weak labels is that for many applications,
simpler models do not suffice and we would like to use models with more
parameters like deep neural networks. However, the more complex the model, the
more samples needed to train it. Here again, weak labels can be used to
improve generalization performance of complex models that would otherwise have
poor generalization performance due to limited strongly-labeled samples. To
demonstrate this, we train a logistic regression model and a artificial neural
network (trained using dropout (Srivastava et al., 2014) regularization) with
two hidden layers on the data set D2. The ANN is a more complex model, so it
is expected to overfit on the training data if we use only strong labeled
samples. However, once we use weak labeled samples as well, it outperforms the
logistic regression model, as shown in figure 5.

Figure 5. Using weak labels enables training more complex models.

####  4.2.4. Mitigating heterogeneity with weak labels

In the presence of heterogeneity in the data population, one needs samples
from every homogeneous variety of instances in the data for the classifier to
have good generalization performance. However, the generalizability of a model
trained with only strong instances improves when weak supervision added to its
training. This can be inferred to some extent by the results in section 4.1.
To demonstrate this effect further, we test the performance of a model trained
on one data set as tested on a completely different data set, both relating to
fires. The results in table 5 show that the model trained on the source data
set does not do well on the target data set, when trained just with strong
supervision, however, adding weak supervision significantly improves the
performance, thus demonstrating the generalizability of the model to
heterogeneous regions.

Table 5. Mitigating heterogeneity with weak labels: Table reports accuracy as models trained on a source data set are applied on a different target data set. Source |  Target |  OnlyStrong |  OnlyWeak |  WeaSL
---|---|---|---|---
D2 |  D1 |  0.565 |  0.604 |  0.682
D1 |  D2 |  0.637 |  0.532 |  0.684

###  4.3. Sensitivity Experiments

In this section, we test the sensitivity of the proposed method to changes in
different aspects of the problem setting. We consider 3 different factors of
the problem setting and measure the performance of our methods (and some
baselines for comparison) as these factors are changed. The 3 factors that we
consider are - skew, purity of instance-labels relative to group labels.

####  4.3.1. Skew

One of the key aspect of _WeaSL_ is its ability to model imbalance in classes
and thus not be biased to perform well on the majority class and avoid false-
positives. In this experiment, we consider balanced data set D2 and sample the
instances in the positive class so that the skew in the new data set
progressively increases from 1:2 initially to 1:12 in the end. The strongly-
labeled data set is still balanced, while the weakly-labeled training set,
validation set and test set are skewed. Note that the groups are not broken
during this process, they are sampled so that the desired skew in instances is
maintained. Figure 6 shows the F-measure for each setting of class skew. The
x-axis shows the skew in the data defined as the ratio of negatives (label 0)
to positives (label 1). Notice how the benefit of using _WeaSL_ over other
methods increases as the skew in the data is increased.

Figure 6. Tolerance to skew: The x-axis shows the skew, defined as the ratio
of negatives (label 0) to positives (label 1) instances. Notice how the
relative benefit of using _Imbalanced WeaSL_ increases as the skew in the data
is increased.

####  4.3.2. Purity of groups

In this experiment, we change the purity of groups in the weakly-labeled data,
where the purity of a group is defined as the number of instances in the group
that have the same label as the group-label. For this experiment, we create a
synthetic data set with 2-dimensional features, where the positive instances
are sampled from 2 Gaussians with means (1,1), (3,3) respectively and standard
deviation 1 in each dimension while the negative instances are sampled from a
Gaussian distribution with mean (0,0) and standard deviation 1 in each
dimension. Figure 7(a) shows the scatter plot for the instances, where the
positive instances are colored red while the negative instances are colored
blue. For the weakly-labeled data set, groups of size $20$ are created, where
the negatively-labeled groups are $100\%$ pure i.e they are made of entirely
negative instances. The positive groups, on the other hand, have a certain
fraction $f$ of their 20 instances as positive, while $(1-f)$ fraction
instances in the positive groups are negative. The weakly-labeled data has
equal number of positive and negative groups i.e the group skew is 1. Note
that in this setting, the instance skew i.e the ratio of negative to positive
instances in the weakly-labeled set is $(2/f-1)$. Thus, as the purity of the
positive bags, encoded by the fraction $f$ decreases, the skew in the data set
increases. Therefore, if one were to just optimize group-level predictions,
one would be totally oblivious to the underlying skew in the instances. For
each value of group purity $f$, we learn the algorithms _Balanced MIL_ ,
_Imbalanced MIL_ , _Imbalanced WeaSL_ and _OnlyStrong_ and test them on a data
set with the same instance-level skew i.e $(2/f-1)$. Figure 7(b) shows the
relative performance of the different algorithms as the purity of the groups
is reduced. Note than the _Imbalanced WeaSL_ is much more tolerant to
decreases in purity of the groups than other methods, giving reasonable
performance even at low purity levels. Moreover, the benefit of using
_Imbalanced WeaSL_ relative to other methods increases with decreasing purity,
since the explicit modeling of noise and rarity becomes more relevant.

(a) Synthetic data distribution. Red denote positives, blue denote negatives

(b) Effect of varying purity of groups in weakly-labeled data

Figure 7. Tolerance to purity of groups in weakly-supervised set: 7(a) shows
the distribution from which instances in each class are sampled from. 7(b)
shows the performance of different methods as group purity $f$ is varied.
X-axis shows values of $f$ while the height of the bars show the performance
in F-score.

##  5\. Conclusion

The paradigm of weakly-supervised learning (Zhou, 2017) has shown great
promise in mitigating the problem of not having adequate labeled samples for
training predictive models. In this paper, we proposed a new problem setting
where binary labels, available for groups of instances, can be used as weak
labels for regularizing a model trained for an instance-level binary
classification problem. We modeled the group-labels as Class Conditional Noisy
(CCN) labels for instances and proposed a learning algorithm _WeaSL_ that
regularizes the performance over weakly-labeled instances by optimizing the
desired metric i.e accuracy for balanced class problems and G-measure for
imbalanced class problems. Our experiments over real-world applications of
land cover monitoring problems show the utility of the proposed approach and
its superiority over other baseline techniques to solve this problem. We also
demonstrate that _WeaSL_ is robust to changes in rarity of the classes, purity
of instances within groups and number of available strongly-labeled instances.
Moreover, we show that _WeaSL_ demonstrably leverages complementarity between
the two sources of supervision - strong and weak, enables the use of more
complex models and helps mitigate data heterogeneity, which are some of the
key factors that motivated the use of group-level weak labels. Further
research includes relaxing the CCN assumption for modeling group-level labels
and exploring other kinds of weak labels for binary classification problems.

## References

  * (1)
  * Andrews et al. (2003) Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. 2003\.  Support vector machines for multiple-instance learning. In _Advances in neural information processing systems_. 577–584.
  * Arpit et al. (2017) Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017\.  A closer look at memorization in deep networks. In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_. JMLR. org, 233–242.
  * Blum and Mitchell (1998) Avrim Blum and Tom Mitchell. 1998.  Combining labeled and unlabeled data with co-training. In _Proceedings of the eleventh annual conference on Computational learning theory_. Citeseer, 92–100.
  * Chen et al. (2006) Yixin Chen, Jinbo Bi, and James Ze Wang. 2006.  MILES: Multiple-instance learning via embedded instance selection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 28, 12 (2006), 1931–1947.
  * Cheplygina et al. (2015) Veronika Cheplygina, David MJ Tax, and Marco Loog. 2015\.  On classification with bags, groups and sets.  _Pattern recognition letters_ 59 (2015), 11–17.
  * DAAC (2009) LP DAAC. 2009.  Land Processes Distributed Active Archive Center.
  * Dietterich et al. (1997) Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. 1997.  Solving the multiple instance problem with axis-parallel rectangles.  _Artificial intelligence_ 89, 1-2 (1997), 31–71.
  * Felix et al. (2013) X Yu Felix, Dong Liu, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang. 2013.  $\propto$SVM for Learning with Label Proportions.  (2013).
  * Frénay and Verleysen (2013) Benoît Frénay and Michel Verleysen. 2013.  Classification in the presence of label noise: a survey.  _IEEE transactions on neural networks and learning systems_ 25, 5 (2013), 845–869.
  * Goldberger and Ben-Reuven (2016) Jacob Goldberger and Ehud Ben-Reuven. 2016.  Training deep neural-networks using a noise adaptation layer.  (2016).
  * Han et al. (2018) Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor Tsang, and Masashi Sugiyama. 2018\.  Co-teaching: Robust training of deep neural networks with extremely noisy labels. In _Advances in neural information processing systems_. 8527–8537.
  * Kuck and de Freitas (2012) Hendrik Kuck and Nando de Freitas. 2012.  Learning about individuals from group statistics.  _arXiv preprint arXiv:1207.1393_ (2012).
  * Li et al. (2017) Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. 2017\.  Learning from noisy labels with distillation. In _Proceedings of the IEEE International Conference on Computer Vision_. 1910–1918.
  * Liu et al. (2003) Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and S Yu Philip. 2003.  Building Text Classifiers Using Positive and Unlabeled Examples.. In _ICDM_ , Vol. 3. Citeseer, 179–188.
  * Menon et al. (2015) Aditya Menon, Brendan Van Rooyen, Cheng Soon Ong, and Bob Williamson. 2015.  Learning from corrupted binary labels via class-probability estimation. In _International Conference on Machine Learning_. 125–134.
  * Mithal et al. (2017) Varun Mithal, Guruprasad Nayak, Ankush Khandelwal, Vipin Kumar, Nikunj C Oza, and Ramakrishna Nemani. 2017\.  Rapt: Rare class prediction in absence of true labels.  _IEEE Transactions on Knowledge and Data Engineering_ 29, 11 (2017), 2484–2497.
  * Natarajan et al. (2013) Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. 2013.  Learning with noisy labels. In _Advances in neural information processing systems_. 1196–1204.
  * Patrini et al. (2014) Giorgio Patrini, Richard Nock, Paul Rivera, and Tiberio Caetano. 2014.  (Almost) no label no cry. In _Advances in Neural Information Processing Systems_. 190–198.
  * Patrini et al. (2017) Giorgio Patrini, Alessandro Rozza, Aditya Krishna Menon, Richard Nock, and Lizhen Qu. 2017\.  Making deep neural networks robust to label noise: A loss correction approach. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1944–1952.
  * Potere et al. (2009) David Potere, Annemarie Schneider, Shlomo Angel, and Daniel L Civco. 2009.  Mapping urban areas on a global scale: which of the eight maps now available is more accurate?  _International Journal of Remote Sensing_ 30, 24 (2009), 6531–6558.
  * Quadrianto et al. (2009) Novi Quadrianto, Alex J Smola, Tiberio S Caetano, and Quoc V Le. 2009.  Estimating labels from label proportions.  _Journal of Machine Learning Research_ 10, Oct (2009), 2349–2374.
  * Ray and Craven (2005) Soumya Ray and Mark Craven. 2005.  Supervised versus multiple instance learning: An empirical comparison. In _Proceedings of the 22nd international conference on Machine learning_. ACM, 697–704.
  * Raykar et al. (2010) Vikas C Raykar, Shipeng Yu, Linda H Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. 2010.  Learning from crowds.  _Journal of Machine Learning Research_ 11, Apr (2010), 1297–1322.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ 15, 1 (2014), 1929–1958.
  * Tao et al. (2004) Qingping Tao, Stephen Scott, NV Vinodchandran, and Thomas Takeo Osugi. 2004.  SVM-based generalized multiple-instance learning via approximate box counting. In _Proceedings of the twenty-first international conference on Machine learning_. ACM, 101\.
  * Veit et al. (2017) Andreas Veit, Neil Alldrin, Gal Chechik, Ivan Krasin, Abhinav Gupta, and Serge Belongie. 2017\.  Learning from noisy large-scale datasets with minimal supervision. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 839–847.
  * Yang and Lozano-Perez (2000) Cheng Yang and Tomas Lozano-Perez. 2000.  Image database retrieval with multiple-instance learning techniques. In _Proceedings of 16th International Conference on Data Engineering (Cat. No. 00CB37073)_. IEEE, 233–243.
  * Zhou (2017) Zhi-Hua Zhou. 2017\.  A brief introduction to weakly supervised learning.  _National Science Review_ 5, 1 (2017), 44–53.


4. B.4 what is the point of proposing a variety of cost-sensitive learning methods? Why not directly use the accuracy of each class to weight the loss?
          3. C Experiments on Stanford point cloud manifolds
          4. D Experimental Details
            1. D.1 Marginal Effect of Semantic Scale
            2. D.2 Quantification of Semantic Scale Imbalance
            3. D.3 Semantic Scale Imbalance on Long-Tailed Data
            4. D.4 Semantic Scale Imbalance for More Datasets
            5. D.5 Experimental Settings for Section 5.3 and More Experiments
              1. D.5.1 Experimental Settings for Section 5.3
              2. D.5.2 More Experiments
                1. D.6 Results on the fundus datasets OIA-ODIR and OIA-ODIR-B
                  1. D.6.1 Dataset Introduction
                  2. D.6.2 Backbone Network and Experimental Parameters
                  3. D.6.3 Results on OIA-ODIR
                  4. D.6.4 Results on OIA-ODIR-B
                    1. D.7 Remote sensing image scene classification
                      1. D.7.1 Dataset Introduction
                      2. D.7.2 Backbone Network and Experimental Parameters
                      3. D.7.3 Results on RSSCN7
                      4. D.7.4 Results on NWPU-RESISC45
                        1. E Pseudo Code for Sample Volume
                        2. F Dynamic Semantic-Scale-Balanced Learning
                          1. F.1 DSB-NSM, DSB-ST and DSB-Focal Loss
                          2. F.2 Dynamic Re-Weighting Training Framework
                        3. G Volume Formula for the Low-Dimensional Parallel Hexahedron in the High-Dimensional Space
                        4. H More analysis
                          1. H.1 The effectiveness of dynamic semantic-scale-balanced learning without considering inter-class interference
                          2. H.2 Comparison with other methods of measuring class-level difficulty
                          3. H.3 Performance of Dynamic Semantic-Scale-Balanced Learning on Three Subsets of ImageNet
                        5. I Apply Semantic Scale to Solve Other Problems
                          1. I.1 Select Well-Represented Data
                          2. I.2 Guide Data Collection
                        6. J Future Work
                          1. J.1 Model-Independent Measure of Data Difficulty
                          2. J.2 A Geometric Perspective on Data Classification
                          3. J.3 Introduce Semantic Scale Imbalance in Object Detection
                        7. K More explanation of Figure 2
                        8. L Can the semantic scale capture the hierarchical structure?

###### List of Figures

  1. 1 The features from "Bird" mapped by CNNs are concentrated on a low-dimensional manifold, and the three-color point sets represent the three sub-classes of "Bird". Among them, the orange point set represents "Swan", whose feature volume is obviously smaller than that of "Bird". The classification experiments on sample-balanced datasets show that the models are biased towards the classes with larger semantic scales, such as the decision surface shown by the green line. In this case, the re-weighting strategy based on the number of samples does NOT work, while our proposed re-weighting approach based on the semantic scale biases the decision surface toward the class with a larger feature volume (red line).
  2. 2 Left column: curves of semantic scales with increasing number of samples for the first ten classes from different datasets. Right column: for different sub-datasets, curves of the sum of semantic scales for all classes and top-1 accuracy curves of trained ResNet-18 and ResNet-34. All models are trained using the Adam optimizer [28] with an initial learning rate of 0.01 and then decayed by 0.98 at each epoch.
  3. 3 Correlation study of accuracy with the number of samples and semantic scale on MNIST, MNIST-LT (Appendix D.3), CIFAR-10-LT and CIFAR-100-LT datasets.
  4. 5 The performance of different models with different losses on different datasets for different values of $n$.
  5. 6 Increase three Stanford point cloud manifolds, and calculate their semantic scales.
  6. 7 Semantic scale and number of samples per class. Different angles of the radar plot represent different classes, and the number of samples in the largest class is normalized to 0.5 for ease of observation.
  7. 8 Long-tailed CIFAR100 and Cars196 with different imbalance factors. We use the exponential function ${n_{i}}=N\\!{\mu^{{i\mathord{/{\vphantom{i{({1-M})}}}\kern-1.2pt}{({1-M})}}}}$ to yield the number of training samples for each class, where $i$ is the class index (0-indexed), $N$ is the number of training samples in the largest class, $\mu$ is the imbalance factor, and $M$ is the total number of classes.
  8. 9 Eight fundus images in the OIA-ODIR dataset.
  9. 10 The number of training and test samples for each category and the degree of semantic scale imbalance in OIA-ODIR and OIA-ODIR-B.
  10. 11 The enhancement effect of our method for CE, BS, Focal, and LDAM on the OIA-ODIR.
  11. 12 Performance gains from our approach for multiple backbone networks on OIA-ODIR.
  12. 13 The seven scenarios are included in the RSSCN7 dataset.
  13. 14 Left column: confusion matrix of VGG-16, GoogLeNet, and ResNet-34 on the RSSCN7 dataset. Right column: confusion matrix of VGG-16-DSB, GoogLeNet-DSB, and ResNet-34-DSB on the RSSCN7 dataset.
  14. 15 Comparison of four backbone networks before and after combining with dynamic semantic scale-balanced learning on dataset NWPU-RESISC45.
  15. 16 The three-stage training framework. The features in the storage pool are continuously updated during training and semantic scales are calculated using all the latest features.
  16. 17 Accuracy comparison on ImageNet-LT.
  17. 18 Accuracy comparison with other methods of measuring class-level difficulty.
  18. 19 Comparison of CE and DSB-CE on ImageNet, where the backbone networks are VGG-16 and ResNet-34, respectively. It can be observed that dynamic semantic-scale-balanced learning significantly improves the tail class performance.
  19. 20 Schematic diagram of pizza sampling. Yellow samples indicate the selected samples and green samples indicate the discarded samples.
  20. 21 Changes in the geometry of data manifolds as they are transformed in a deep neural network. The classification process of the data includes untangling the manifolds from each other and separating the different manifolds.
  21. 22 Left column: curves of semantic scales with increasing number of samples for the first ten classes from different datasets. Right column: for different sub-datasets, curves of the sum of semantic scales for all classes and top-1 accuracy curves of trained ResNet-18 and ResNet-34. All models are trained using the Adam optimizer [28] with an initial learning rate of $0.01$ and then decayed by $0.98$ at each epoch.
  22. 23 Image datasets typically contain multiple semantic hierarchies.

###### List of Tables

  1. 1 Pearson correlation coefficients between the accuracy of classes and the semantic scales $S$ with different $\alpha$. $N$ denotes the number of samples, and $S^{\prime}$ represents the semantic scale without considering inter-class interference. $E_{n}$ denotes the number of effective samples.
  2. 2 Top-1 Acc(%) on ImageNet-LT and iNaturalist2018. We use ResNext-50 [70] on ImageNet-LT and ResNet-50 [17] on iNaturalist2018 as the network backbone for all methods. And we conduct model training with the SGD optimizer based on batch size 256 (for ImageNet-LT) / 512 (for iNaturalist), momentum 0.9, weight decay factor 0.0005, and learning rate 0.1 (linear LR decay).
  3. 3 Comparison on ImageNet and CIFAR-100. On ImageNet, we use random clipping, mixup [76], and cutmix [75] to augment the training data, and all models are optimized by Adam with batch size of 512, learning rate of 0.05, momentum of 0.9, and weight decay factor of 0.0005. On CIFAR-100, we set the batch size to 64 and augment the training data using random clipping, mixup, and cutmix. An Adam optimizer with learning rate of 0.1 (linear decay), momentum of 0.9, and weight decay factor of 0.005 is used to train all networks.
  4. 6 Evaluation on MSCOCO-GLT.
  5. 7 The sample-balanced sub-datasets with a total of $31$. Among them, $13$ sub-datasets are from CIFAR-10, $9$ sub-datasets are from CIFAR-100, and the rest are from Mini-ImageNet. The test set remains the original test set. $C$ denotes the total number of classes in the original dataset and $m$ is the number of samples per class in the sub-dataset.
  6. 8 The two long-tailed MNIST datasets resampled from MNIST.
  7. 10 Comparison on Mini-ImageNet and CIFAR-100.
  8. 11 Comparison of DSB-ST and SoftTriple in terms of memory consumption and training speed. The speed is measured by the average number of iterations per second. The additional video memory consumption due to our method is almost negligible.
  9. 12 Results of matching parent class for each child class, where the Ratio of semantic scales denotes the ratio of the semantic scales of the parent class after mixing to before mixing, Predicted parent class means the parent class we matched for the child class, and Real parent class denotes the real parent class corresponding to the child class.