# LLP-Bench: A Large Scale Tabular Benchmark for
Learning from Label Proportions

Anand Brahmbhatt*
Google Research India
<EMAIL_ADDRESS>Mohith Pokala*
Google Research India
<EMAIL_ADDRESS>Rishi Saket
Google Research India
<EMAIL_ADDRESS>Aravindan Raghuveer
Google Research India
<EMAIL_ADDRESS>
###### Abstract

In the task of Learning from Label Proportions (LLP), a model is trained on
groups (a.k.a bags) of instances and their corresponding label proportions to
predict labels for individual instances. LLP has been applied pre-dominantly
on two types of datasets - image and tabular. In image LLP, bags of fixed size
are created by randomly sampling instances from an underlying dataset. Bags
created via this methodology are called random bags. Experimentation on Image
LLP has been mostly on random bags on CIFAR-* and MNIST datasets. Despite
being a very crucial task in privacy sensitive applications, tabular LLP does
not yet have a open, large scale LLP benchmark. One of the unique properties
of tabular LLP is the ability to create feature bags where all the instances
in a bag have the same value for a given feature. It has been shown in prior
research that feature bags are very common in practical, real world
applications [7, 33].

In this paper, we address the lack of a open, large scale tabular benchmark.
First we propose LLP-Bench, a suite of 56 LLP datasets (52 feature bag and 4
random bag datasets) created from the Criteo CTR prediction dataset consisting
of 45 million instances. The 56 datasets represent diverse ways in which bags
can be constructed from underlying tabular data. To the best of our knowledge,
LLP-Bench is the first large scale tabular LLP benchmark with an extensive
diversity in constituent datasets. Second, we propose four metrics that
characterize and quantify the hardness of a LLP dataset. Using these four
metrics we present deep analysis of the 56 datasets in LLP-Bench. Finally we
present the performance of 9 SOTA and popular tabular LLP techniques on all
the 56 datasets. To the best of our knowledge, our study consisting of more
than 2500 experiments is the most extensive study of popular tabular LLP
techniques in literature.

††* – equal contribution

##  1 Introduction

In traditional supervised learning, _training_ data consists of feature-
vectors (instances) along with their labels. A model trained using such data
is then used during inference to predict the labels of _test_ instances. In
recent times, primarily due to privacy concerns and relative rarity of high
quality large-scale supervised data, the weakly supervised framework of
_learning from label proportions_ (LLP) has gained importance [34, 33, 30, 38]
In LLP, the training data is aggregated into bags. Each bag contains a bunch
of instances (and their feature vectors) and their corresponding aggregated
label count. The goal is to learn a classification model for predicting the
class-labels of individual instances [10, 25].

Study of LLP has recently gained importance due to developments in the privacy
landscape. In particular, restrictions on tracking of user events have led to
an LLP formulation of user-modeling in _online advertising_ [26]. Since only
the average label for a bag of users is revealed, the size of the bags is a
measure of the privacy afforded. Other applications include medical records
anonymization [38], IVF prediction [18], image classification [3, 27], mass
spectrometry [5], datasets with legal constraints [30, 38] and inadequate or
costly supervision [11, 5].

Such LLP techniques have primarily been evaluated and studied on image [21,
40, 37, 23, 14, 2] and tabular [33, 4, 29] datasets. On images, well known
datasets like CIFAR-10, CIFAR-100, MNIST are used – typically by randomly
partitioning the dataset into bags – to create medium-large scale LLP
datasets. On the other hand, tabular data consists of independent rows of
feature vectors with one more labels attached to each feature vector. Often,
previous works used tabular LLP datasets derived from small UCI [12] datasets
which fail to simulate the diversity and scale of applications involving such
data. Notably, tabular datasets are extremely common in real world
classification and regression tasks for online advertising [26], health care
research [30, 38] and scientific simulation studies [5]. Such applications
tend to use very large scale data either from online user interaction [24, 16]
or user studies [15]. Impact of privacy leaks due to inadvertent exposure of
sensitive data is much higher in large scale datasets. Therefore LLP on large
scale tabular datasets is a very critical application that is receiving
increasing attention from the research community. While image LLP has large
scale benchmark datasets derived from CIFAR-*, an equivalent benchmark does
not exist for tabular data.

Figure 1: Dataset D1 is formed by randomly choosing without replacement from
the instance dataset to form bags of size =3. Dataset D2 is feature bag formed
by creating bags such that within a bag instances have the same value for
features F1, F2. The fourth bag is removed because it has only one instance
i7. Dataset D3 is similarly formed by using feature F3 as the grouping key.
Notice the bags have become substantially larger than those in D1.

A unique property of tabular LLP as pointed out in recent literature [33, 7]
is the notion of feature bags. In feature bags, bags are constructed such that
all instances within the bag have the same value for given key(s) called the
grouping key(s). Such bags occur in critical real-world applications such as
user modelling in online advertising where the conversion labels are
aggregated over pre-selected categorical features [7, 4, 26]. Figure 1 shows
three datasets created from the same instance datasets. The first dataset is
created like random sampling much like the ones created using CIFAR-* [21, 40,
37]. The second and third datasets are features bags created by using F1, F2
and F3 as the grouping keys respectively. Note that feature bags can also be
made to have fixed size.

Motivated by the above observations and the richness of the problem, we
propose LLPBench a large scale, diverse benchmark for tabular LLP. We make
three contributions in this paper as listed below.

1. We propose LLP-Bench, a suite of 56 LLP datasets (52 feature bag and 4 random bag datasets) created from the popular Criteo CTR prediction dataset consisting of 45 million instances [9]. The 56 datasets represent diverse ways in which bags can be constructed from underlying tabular data. The datasets have between 13.5 million to 24.75 million bags. To the best of our knowledge, LLP-Bench is the first large scale tabular LLP benchmark with an extensive diversity in constituent datasets (Section 4).

2. We propose four metrics that help characterize and quantify the hardness of an LLP dataset. Using these four‘ metrics we present deep analysis of the 56 datasets in LLP-Bench (Sections 3, 5).

3. We present the performance of 9 SOTA and popular tabular LLP techniques on all the 56 datasets. To the best of our knowledge, our study consisting of more than 2500 experiments is the most extensive study of popular tabular LLP techniques in literature (Sections 6, 7).

Choice of base dataset. The Criteo CTR dataset is relevant to our work -
firstly it is a CTR dataset which corresponds to a natural application where
LLP aggregation can occur due to privacy [26]. Another reason is that it is a
rare publicly available large scale tabular dataset with several categorical
features, which allow for a rich collection of feature bag datasets. Our
techniques for feature-based grouping and LLP dataset analysis are more
generally applicable. However, since we focus on a tabular LLP Benchmark, we
restrict ourselves to Criteo CTR in which feature-vectors representing
impressions are given binary $\\{0,1\\}$-labels indicating a click.

The binary label setting is widely studied in supervised machine learning, and
in the LLP setting as well with many real-world applications: see for e.g.
references [18] and [11] in the paper for applications in IVF prediction and
high-energy physics. In particular, the important task of user-modeling on
online advertising platforms has recently seen privacy related restrictions
leading to an LLP formulation for it (see Section 1 of [4]), which is
typically a binary label problem.

##  2 Related Work

Several techniques for LLP have been studied over the years. The work of [10,
17] applied trained probabilistic models using Monte-Carlo methods. Subsequent
works [25, 30] extended supervised learning techniques such neural nets, SVM
and $k$-_nearest neighbors_ to LLP, others adapted clustering based approaches
[8, 36], while [39] proposed a novel $\propto$-SVM method for LLP. The work of
[29] estimated model parameters from label proportions for the exponential
generative model with certain assumptions on label distributions of bags.
Their method was further applied by [28] for more general models and relaxed
distributional assumptions. More recent works have investigated deep neural
network based LLP methods [3, 1, 20], techniques using bag combinations [34,
33], curated bags [6] and training on derived _surrogate_ labels for instances
[4]. Recently, [31, 32] initiated a theoretical study of LLP from the
computational learning perspective.

All of the previous works in LLP experimentally evaluate their methods on LLP
datasets consisting of bags randomly created from some real-world supervised
learning dataset. In these _pseudo-synthetic_ LLP datasets, instances are
randomly sampled/partitioned into the different bags, where in [28] and [33]
this process also clusters feature-vectors to generate more complicated bag
distributions. Almost all of the above works use limited scale data, typically
small to medium scale UCI datasets [39, 28, 34], image datasets [20], social
media data [1]. In general, there have been very few large scale tabular
datasets created and used for LLP. To the best of our knowledge, [33] and [6]
are the only works that explore a large dataset (Criteo) to test their
methodology. Since the primary contribution of these works is algorithmic,
they do not justify their choice of how bags were created nor do they explore
the many choices and tradeoffs of creating bags from a large scale instance
dataset. In our work we precisely address this gap - we not only create
benchmark with 56 diverse datasets, but we study in detail the tradeoffs
involved and analyse the performance of 9 important baselines. We study the
performance of the following 9 baselies on the LLP-Bench benchmark.

${\bf DLLP}$: This a the standard LLP baseline method use in previous works
[1], which optimizes a bag-level loss between the average label proportion and
the predicted average label proportion. We evaluate ${\sf DLLP\text{-}BCE}$
and ${\sf DLLP\text{-}MSE}$ which use the bag-level BCE and MSE losses
respectively using the minibatch training described above.
${\bf GenBags}$: This is the generalized bags method of [33], and since we
only have one collection of bags in a given experiment we use random Gaussian
combining weights to construct 120 different generalized bags per mini-batch
of 8 bags.
${\bf Easy\text{-}LLP}$: In this technique proposed by [4], mean-bag labels
along with the global average label are used to define a _surrogate_ label for
training instances over which the model is optimized using the BCE loss.
${\bf OT}$ methods: There are the optimal transport based techniques proposed
in [22, 13]. The first variant ${\bf OT\text{-}LLP}$ included is the non-
regularized optimal transport for disjoint bags which can be implemented via a
greedy approach. We also have ${\sf Hard\text{-}EROT\text{-}LLP}$ and ${\sf
Soft\text{-}EROT\text{-}LLP}$ – the hard and soft entropic regularized OT
methods of [22].
${\bf SIM\text{-}LLP}$: In this method proposed by [19], the bag-level DLLP
loss is augmented with a pairwise similarity based loss penalizing different
predictions of geometrically close feature-vectors. Since the similarity based
loss has number of terms which is square in the number of feature-vectors in a
minibatch, we sample a random set of $400$ feature-vectors from each minbatch
to apply this loss.
${\bf Mean\text{-}Map}$: This is the well-known technique of [29] for
linearized exponential generative models, consisting of two steps: computing
the quantity using the bag-label proportions followed by optimizing for the
model parameters over all feature-vectors. While the first step is a
straightforward calculation, the second step is implemented using a minibatch
optimization.

##  3 LLP Dataset Characteristics

In this section, we first establish some notation, define LLP terminology and
then propose four metrics to characterize and quantify the hardnesss of an LLP
dataset.

In our exploration of LLP we shall only consider binary i.e.,
$\\{0,1\\}$-valued instance labels.
Notation: $X:=\\{{\mathbf{x}}^{(i)}\in\mathbb{R}^{n}\\}_{i=1}^{m}$ is a
dataset of $m$ feature vectors in $n$-dimensional space with labels given by
$Y:=\\{y^{(i)}\in\\{0,1\\}\\}_{i=1}^{m}$. We denote by
$\hat{Y}:=\\{\hat{y}^{(i)}\in[0,1]\\}_{i=1}^{m}$ the corresponding model
predictions which are probabilities of the predicted label being $1$. A _bag_
$B\subseteq[m]$ consists of feature vectors $X_{B}:=\cup_{i\in
B}{\mathbf{x}}^{(i)}$ and with the corresponding label sum $y_{B}:=\sum_{i\in
B}y^{(i)}$. The label proportion of the bag is $y_{B}/|B|$.

######  Definition 3.1 (LLP Dataset)

A _learning from label proportions (LLP)_ dataset corresponding to a
collection of bags $\mathcal{B}:=\\{B_{j}\\}_{j=1}^{N}$ is given by
$\\{(X_{B},y_{B})\,\mid\,B\in\mathcal{B}\\}$. The _label bias_ of training
dataset is
$\mu(\mathcal{B},Y):=\left(\sum_{B\in\mathcal{B}}y_{B}\right)/\left(\sum_{B\in\mathcal{B}}|B|\right)$,
while the average label proportion is
$\hat{\mu}(\mathcal{B},Y):=\tfrac{1}{N}\left(\sum_{B\in\mathcal{B}}y_{B}/|B|\right)$.

The LLP datasets considered in the paper have disjoint bags.

In the following we define statistics comparing the separation among feature-
vectors within bags and their separation across bags. using a natural notion
of bag separation.

######  Definition 3.2 (Bag Separation)

For a distance $d$ on $\mathbb{R}^{n}$ and collection of bags
$\mathcal{B}=\\{B_{1},\dots,B_{M}\\}$ the corresponding separation function is
defined as ${\sf
BagSep}(B,B^{\prime},d):=\frac{1}{|B||B^{\prime}|}\sum_{{\mathbf{x}}\in
B}\sum_{{\mathbf{x}}^{\prime}\in
B^{\prime}}d({\mathbf{x}},{\mathbf{x}}^{\prime})$. We define the $M\times M$
matrix ${\sf BagSepMatrix}(\mathcal{B},d)$ whose $(i,j)$th element is given by
${\sf BagSep}(B_{i},B_{j},d)$.

We use ${\sf BagSep}$ to compute the average separation between pairs of bags
and the average separation within each bag. If the feature-vectors in bags are
clustered together and far away from those of other bags, we expect the former
to be significantly greater than the later.

######  Definition 3.3 (Inter-Bag Separation for a bag)

Given $\mathcal{B}$, and metric $d$ on $\mathbb{R}^{n}$, the average inter-bag
distance for a bag $B\in\mathcal{B}$ is defined as ${\sf
InterBagSep}(B,d):=\frac{1}{|\mathcal{B}|-1}\sum_{B^{\prime}\in\mathcal{B},B^{\prime}\neq
B}{\sf BagSep}(B,B^{\prime},d)$.

For computing the average statistic for the entire dataset we define the
following.

###### Definition 3.4

The mean intra-bag separation of $\mathcal{B}$ is defined as ${\sf
MeanIntraBagSep}(\mathcal{B},d):=\frac{1}{|\mathcal{B}|}\sum_{B\in\mathcal{B}}{\sf
BagSep}(B,B,d)$. The mean of average inter-bag separation is defined as ${\sf
MeanInterBagSep}(\mathcal{B},d):=\frac{1}{|\mathcal{B}|}\sum_{B\in\mathcal{B}}{\sf
InterBagSep}(B,d)$.

###  3.1 Hardness metrics for LLP datasets

We are now ready to present four metrics that characterize the hardness of a
LLP dataset: (i) standard deviation of label proportion (ii) inter vs intra
bag separation ratio (iii) mean bag size, and (iv) cumulative bag size
distribution.

${\bf LabelPropStdev}$: This is the standard deviation of the label
proportions of the collection of bags i.e.,
$\sqrt{\textnormal{Var}_{B\leftarrow\mathcal{B}}\left[y_{B}/|B|\right]}$. A
higher ${\sf LabelPropStdev}$ typically provides more model training
supervision. For e.g. consider $(X,Y)$ with two different bag collections
$\mathcal{B}_{1}$ and $\mathcal{B}_{2}$ with
$\alpha=\hat{\mu}(\mathcal{B}_{1},Y)=\hat{\mu}(\mathcal{B}_{1},Y)$ while
$0\approx\beta_{1}={\sf LabelPropStdev}(\mathcal{B}_{1},Y)\ll{\sf
LabelPropStdev}(\mathcal{B}_{2},Y)=:\beta_{2}$. Consider a model that predicts
$\alpha$ for every feature-vector. Since $\beta_{1}\approx 0$, the predicted
label proportions of this model are will be close to the true label
proportions for most bags in $\mathcal{B}_{1}$ unlike in $\mathcal{B}_{2}$
(since $\beta_{2}\gg\beta_{2}$). For many bag-level losses, this results in
such a model being much closer to optimal for $\mathcal{B}_{1}$ rather
$\mathcal{B}_{2}$. On the other hand, the model is only learning the average
label proportion and not discriminating among the instances, therefore not
desirable and is ruled out when ${\sf LabelPropStdev}$ is higher.

${\bf InterIntraRatio}$: This denotes ${\sf
MeanInterBagSep}(\mathcal{B},d)/{\sf MeanIntraBagSep}(\mathcal{B},d)$ when
$d=\ell_{2}^{2}$. A dataset with large ${\sf InterIntraRatio}$ has well
separated bags and therefore the label proportion supervision provided per bag
carries more information and hence easier to learn from compared to a dataset
with a smaller ${\sf InterIntraRatio}$.

${\bf MeanBagSize}$: Since we have only have a label proportion for each bag,
informally speaking, the larger the bag size the lower the amount of label
supervision for that bag. The third and a simple metric to characterize this
is ${\sf MeanBagSize}$ i.e., the mean size of all the bags in the dataset.
Therefore, a dataset with larger mean bag size is a much harder dataset to
learn from compared to one with a much smaller mean bag size.

${\bf CumuBagSizeDist}$: The bag sizes for any dataset are characterized by
their cumulative distribution function which plots the fraction of bags of
size at most $t$ for all $t\geq 1$. We compute the bag sizes at the $50$,
$70$, $85$ and $95$ percentile of cumulative distribution plot, for each
dataset. Short-tailed distributions have most bags of small size and a very
few large sized bags whereas Long-tailed distributions contain many bags of
large sizes. Bags of large sizes provide a very little label information for a
lot of feature level information. Therefore a LLP dataset with a long-tailed
distribution of bag sizes is a much harder dataset to learn from than a short-
tailed one.

##  4 LLP Dataset: Bag creation

The Criteo dataset ([9]) is a large scale well known binary classification
dataset for ad click prediction. We build the datasets in LLP-Bench using the
instances and labels in the Criteo CTR dataset.

The Criteo dataset has 13 numerical and 26 categorical features and a binary
label. Each of the approximately 45 million rows (instances) represents an
impression (online ad) and the label indicates a click. The semantics of all
the features are undisclosed and the values of all the categorical features
hashed into 32-bits for anonymization. Additionally, the dataset has missing
values. We use a preprocessed version of the dataset as done for the AutoInt
([35]) model, described and implemented in their provided code111
https://github.com/DeepGraphLearning/RecommenderSystems/tree/master/featureRec
. We choose AutoInt because that is among the best performing models on the
Criteo benchmark222 https://paperswithcode.com/sota/click-through-rate-
prediction-on-criteo. For convenience we label the numerical and categorical
features (in their order of occurrence) as
$\textnormal{N}1,\dots,\textnormal{N}13$ and
$\textnormal{C}1,\dots\textnormal{C}26$. The preprocessing applies
$\textnormal{int}(\log^{2}(x))$ transformation when $x>2$ on the numerical
feature values $x$, and we further additively scale so that their values are
non-negative integers. The categorical features are encoded as non-negative
integers.

We create two types of LLP Data sets. We create 4 Random Bag datasets by
randomly sampling without replacement (Similar to D1 in Figure 1) to create
bags of fixed sizes of 64, 128, 256, 512. Next we create 52 Feature Bag
datasets by grouping instances by subsets
$\mathcal{C}\subseteq\\{\textnormal{C}1,\dots,\textnormal{C}26\\}$ of the
categorical columns, where $\mathcal{C}\leq 2$. The feature grouping subset
used for a dataset is called its grouping key. For each setting of the values
of $\mathcal{C}$ we obtain a bag with instances with those values of
$\mathcal{C}$ (Similar to D2 and D3 in Figure 1). Each such key grouping
yields an LLP dataset333 Note that for model training purposes such bags may
be created from only the _train set_ portion of the entire dataset. Thus, we
obtain ${26\choose 2}+26=351$ LLP datasets, each referred to also as a dataset
on $\mathcal{C}$ ($|\mathcal{C}|\leq 2$). Note that for any dataset, the set
of bags partition the dataset and therefore each instance occurs in exactly
one bag. In Appendix A.13 we demonstrate the creation and performance metrics
of _fixed size feature bag_ datasets as well – in which for each of the 52
groupings and bag size $q\in\\{64,128,256,512\\}$, the train instances are
ordered according to the grouping features and consecutive $q$-sized sequences
are made into bags.

Next we describe two filtering strategies to remove datasets that are
ineffective in practice. The first strategy removes very small or very large
bags within a dataset. The second strategy drops a dataset entirely if the
first filtering method results in pruning a large portion of the underlying
dataset.

###  4.1 Bag Filtering

The feature bag creation step leads to bags of varying sizes. Very small bags
are not practical because they do not preserve enough privacy. For instance,
datasets on $\\{\textnormal{C}10,\textnormal{C}16\\}$ and
$\\{\textnormal{C}4,\textnormal{C}10\\}$ each contain more than $8\times
10^{6}$ bags. We introduce a hyper-parameter $low_{thresh}$ which represents
the size of the smallest bag that can be present in a LLP-Bench dataset.

Similarly, a very large bag is almost useless since the information lost via
aggregation is significant and hence the dataset cannot be used to build a
useful classifier. For instance, the initial dataset on $\textnormal{C}9$
creates only $3$ bags and the dataset on $\textnormal{C}20$ creates only $4$
bags. We introduce a hyper-parameter $high_{thresh}$. which represents the
size of the largest bag that can be present in a LLP-Bench dataset.

For our experiments we set $low_{thresh}$ to be 50 and $high_{thresh}$ to be
2500.

###  4.2 Dataset Filtering

If the $low_{thresh}$ and $high_{thresh}$ based filtering remove a significant
fraction of bags then most of the underlying instances will be lost. This will
cause the LLP dataset’s performance to be poor because we are not left with
enough signal to train on. We hence drop datasets that have less than
$instance_{thresh}\%$ of the original instance data size. We set
$instance_{thresh}$ to be $30\%$ in our experiments. After applying this
filter, we are left with $52$ datasets. All the datasets in single columns are
filtered out as the maximum percentage of instances any of these datasets
retains is $21.68\%$ ($\textnormal{C}4$). This analysis is presented in detail
in Appendix A.9. For notational convenience in the rest of this paper, we
shall call $(A,B)$ where $A,B\in\\{\textnormal{C}1,\dots,\textnormal{C}26\\}$
the LLP dataset formed via grouping by the subset $\mathcal{C}=\\{A,B\\}$.

##  5 Diversity of the Benchmark

Figure 3 depicts for each of the 52 datsets chosen in Sec. 4.2 the values of
three different bag-level metrics: (i) ${\sf MeanBagSize}$ \- the average size
of bags, (ii) ${\sf LabelPropStdev}$ \- the standard deviation of the bag
label proportions, and (ii) ${\sf InterIntraRatio}$ as given defined in Sec.
3.1. Apart from capturing the bag size and label proportion distribution, the
third metric also quantifies the geometric distribution of the feature-vectors
w.r.t the bags, in particular how clustered the feature-vectors in an average
bag are.

We see in Figure 2(a) that ${\sf MeanBagSize}$ values range from nearly 500 to
around 200. While most of them are in $[150,250]$, around one-fourths of the
values are above $300$, indicating significant diversity in the values of this
metric.

In Figure 2(b) we see a similar trend in ${\sf LabelPropStdev}$ – while most
of the values are in the range $[0.15,0.18]$, around 25% of the them are below
$0.14$. This unsurprising since larger bags would typically lead to more
concentrated label proportions, and thus ${\sf MeanBagSize}$ is loosely anti-
correlated to ${\sf LabelPropStdev}$ and our collection of datasets have
similar diversity of the latter’s values.

Figure 2(c) has the values for ${\sf InterIntraRatio}$ showing that they are
well spread across the range $[1.1,1.6]$. While most values are below $1.4$,
there are around 17% of them which are above $1.5$, indicating that the
distribution has a fat tail and metric values are diverse.

(a) ${\sf MeanBagSize}$

(b) ${\sf LabelPropStdev}$

(c) ${\sf InterIntraRatio}$

Figure 2: Datasets vs. bag-level metrics: $y$-axis has the metric, $x$-axis
has the datsets.

(d) ${\sf MeanBagSize}$

(e) ${\sf LabelPropStdev}$

(f) ${\sf InterIntraRatio}$

Figure 3: Datasets performance: AUC scores on the y-axis, $x$-axis has the
datasets ordered according to increasing metric.

In Appendix A.17 we include the 3-D scatter plot of the the three metrics
${\sf MeanBagSize}$, ${\sf LabelPropStdev}$ and ${\sf InterIntraRatio}$ as
well as three 2-D scatter plots for each of the three pairs to show that the
metrics have significant variability and limited dependency with respect to
each other. Appendix A.17 also incliudes magnified versions of Figures 3 and 3
for ease of readability.

In Appendix A.18 we compute the Cramer’s V between grouping-key pairs $(A,B)$
and the label for each feature-bag LLP-Bench Dataset. We observe that there is
a significant diversity in the values which are all bounded away from $1$,
indicating that the datasets are diverse in terms of the bag vs label
correlations.

##  6 Evaluation of baselines on LLPBench

_Training and test data setup._ For each feature-grouping based dataset
$(A,B)$ available after filtering in Section 4.2, we create the a 5 fold
train/test split as follows. We first create the dataset $(A,B)$ over the
entire Criteo dataset and filter out the bags as mentioned in Section 4.1.
Using the feature-vectors in the remaining bags and their original labels, we
recreate the instance-level dataset. This is then used to create a 5-fold
train/test split. For each train split we recreate the bag-level training data
via grouping by $(A,B)$. For the fixed-size random bag datasets, we first do a
5-fold train/test split of the entire data and partition the train splits into
bags of the given fixed size.

_Training Methodology._ All of our baselines (listed below) are trained on the
LLP datasets using minibatch training. We sample $8$ bags in each minibatch
and do a forward-pass of the model on all the feature-vectors in the minibatch
of bags. This allows us to obtain the predicted label proportions for each of
the bags as well as the instance-level predicted-labels. Using these along
with the true label proportions we compute the appropriate loss functions of
the different methods. The back-propagation step then updates the model
parameters.

We evaluate the following baseline methods on LLP-Bench whose methdology was
explained in Section 2. ${\sf DLLP}$ [1], ${\sf GenBags}$: [33], ${\sf
Easy\text{-}LLP}$[4], ${\sf OT}$ methods:[22, 13] (${\sf
Hard\text{-}EROT\text{-}LLP}$ and ${\sf Soft\text{-}EROT\text{-}LLP}$, the
hard and soft entropic regularized OT methods of [22]), ${\sf
SIM\text{-}LLP}$: [19] ${\sf Mean\text{-}Map}$: [29].

Using each of the above methods, a $2$-layer perceptron is trained using a
fixed learning rate of $1e\text{-}5$. It has a multi-hot encoding layer which
takes as input index encoded feature-vectors, followed by $128$ and $64$ node
hidden layers respectively with relu activation. The final output node is
sigmoid activated. We report the _area under the ROC curve_ (AUC) scores for
each of the above LLP methods. We use an ${\sf Adam}$ optimizer with learning
rate of $1e-5$. We also implement early-stopping with patience of $3$ epochs
which monitors accuracy on the test set. Additional details of the
implementation of the above methods are included in Appendix A.6. The AUC
scores for all the LLP-Bench datasets for all the baselines are included in
Appendix A.8.

###  6.1 Performance of baseslines on Feature Bags

Fig. 2(d) presents the trend of AUC scores for the previously described LLP
methods w.r.t the ${\sf MeanBagSize}$ metric where the x-axis has the datasets
ordered by increasing ${\sf MeanBagSize}$. Similarly in Figures 2(e) and 2(f)
the datasets are ordered on the x-axis by increasing ${\sf LabelPropStdev}$
and ${\sf InterIntraRatio}$ respectively.

First we observe that the ${\sf SIM\text{-}LLP}$, ${\sf DLLP\text{-}BCE}$,
${\sf DLLP\text{-}MSE}$, and ${\sf GenBags}$ methods are the best performing
on all of the datasets with AUC scores in the $72\%$-$78\%$ range. Within
them, ${\sf SIM\text{-}LLP}$ performs the best on 41, ${\sf DLLP\text{-}MSE}$
on 6 and ${\sf DLLP\text{-}BCE}$ on 5 datasets, indicating that the additional
similarity based loss helps on feature bag datasets. The AUC scores of ${\sf
GenBags}$ is consistently lower than ${\sf SIM\text{-}LLP}$ and the ${\sf
DLLP}$ methods, possibly due to the fact that our scenario does not have
multiple bag distributions and therefore no corresponding convex programming
solution to obtain the combining weights. This leads to undesirable
combinations of large bags with smaller ones with roughly equal weights
(ideally smaller bags should receive larger weights), leading to loss in the
bag-label supervision.

On the other hand, the AUC scores of ${\sf Mean\text{-}Map}$ are the lowest
for nearly all the datasets, remaining below $67\%$. The ${\sf
Easy\text{-}LLP}$ and the ${\sf OT}$ methods have scores typically in the
range of $65\%$-$70\%$. For reference, the same model trained on instance-
level data yields around $80\%$ AUC score (see Appendix A.7).

Since the datasets are created by feature-based aggregation they may not
satisfy the distributional and generative model assumptions of [29], possibly
explaining the lower scores of ${\sf Mean\text{-}Map}$. Similarly, ${\sf
Easy\text{-}LLP}$ is tailored towards random-bags, and therefore may have
lower scores on these datasets. The lower performance of the ${\sf OT}$
methods indicates that the pseudo-labels computed in their optimization step
could significantly differ from the true labels. On the other hand, optimizing
the bag-level losses as in ${\sf DLLP}$ based methods leads to more accurate
model training.

The trends w.r.t. the metrics are as expected. There is a moderately
decreasing trend of AUC scores with increasing ${\sf MeanBagSize}$ which is
unsurprising since larger bags provide lower label supervision. More
interestingly, there is a clear increasing trend with increasing ${\sf
LabelPropStdev}$ along with a moderate increasing trend with increasing ${\sf
InterIntraRatio}$. The latter two trends are also expected, given the
explanations in Sec. 3.

Bag size |
---|---
Method | 64 | 128 | 256 | 512
${\sf DLLP\text{-}BCE}$ | 77.54 | 76.96 | 76.24 | 75.22
${\sf DLLP\text{-}MSE}$ | 77.56 | 77.03 | 76.33 | 75.42
${\sf GenBags}$ | 77.08 | 76.5 | 75.75 | 75.22
${\sf Easy\text{-}LLP}$ | 75.69 | 74.18 | 72.32 | 70.13
${\sf OT\text{-}LLP}$ | 74.25 | 71.53 | 68.1 | 65.26
${\sf SIM\text{-}LLP}$ | 77.41 | 76.73 | 75.47 | 73.13
${\sf Soft\text{-}EROT\text{-}LLP}$ | 74.43 | 71.82 | 68.34 | 65.16
${\sf Hard\text{-}EROT\text{-}LLP}$ | 74.27 | 71.65 | 68.08 | 65.54
${\sf Mean\text{-}Map}$ | 63.17 | 63.06 | 62.83 | 62.26

Table 1: AUC scores on Random Bags

Method | Min AUC | Max AUC
---|---|---
${\sf DLLP\text{-}BCE}$ | 72.92 | 77.04
${\sf DLLP\text{-}MSE}$ | 72.96 | 77.03
${\sf GenBags}$ | 71.45 | 76.8
${\sf Easy\text{-}LLP}$ | 64.59 | 71.62
${\sf OT\text{-}LLP}$ | 67.57 | 72.38
${\sf SIM\text{-}LLP}$ | 72.8 | 77.57
${\sf Soft\text{-}EROT\text{-}LLP}$ | 67.8 | 72.5
${\sf Hard\text{-}EROT\text{-}LLP}$ | 64.75 | 72.49
${\sf Mean\text{-}Map}$ | 60.38 | 67.43

Table 2: Range of AUC scores on feature-bag datasets

###  6.2 Performance of Baselines on Random Bags

Table 2 reports the AUC scores obtained by running baselines on random bags of
sizes $64,128,256,512$. We observe that the performance of ${\sf
Easy\text{-}LLP}$ is better random bags as compared to its performance on
feature bags. As mentioned above, the performance guarantees for ${\sf
Easy\text{-}LLP}$ assume random bags therefore this improvement is expected.
We also notice the ${\sf SIM\text{-}LLP}$ no longer outperforms ${\sf
DLLP\text{-}BCE}$ and ${\sf DLLP\text{-}MSE}$ on random bags as it did on
feature bags. The instances in feature bags are closer than those in random
bags. As pairs of instances sampled in the same mini-batch are likely belong
the same bag, the weight factor in the similarity loss
($\exp(\|\mathbf{x}_{i}-\mathbf{x}_{j}\|_{2}^{2})$) is likely to be higher in
case of feature bags leading to greater supervision as compared to random
bags. Further, as the random bag size increases, in ${\sf SIM\text{-}LLP}$ due
to lower label supervision the magnitude of the bag-loss could decrease,
making the similarity loss more dominant which spuriously penalizes pairs of
instances with different labels, leading to an overall moderation in
performance.

While the performance of ${\sf Easy\text{-}LLP}$ reduces noticeably with
increase in bags size, the corresponding degradation of the ${\sf OT}$ methods
is particularly significant. Notably, these techniques derive instance level
surrogate or pseudo-labels as training labels for the model. These trends
suggest that such pseudo-labeling techniques are more severely affected by
increasing bag size.

##  7 Detailed Analysis

In this section, we present more detailed analysis of various baselines and
interesting datasets in the benchmark.

###  7.1 Best and worst performance of every baseline

Table 2 lists the range of AUC scores of each method on our benchmark. First,
a range of AUC scores of at least $4$ percentage points can be seen for all
the baselines. This shows that LLP-Bench has enough diversity in the
underlying datasets and that it can be used to find opportunities to improve
SOTA algorithms to further LLP research on Criteo CTR and other tabular
datasets.

Next, we observe that all baselines perform the best on the dataset
$(\textnormal{C}3,\textnormal{C}11)$ and perform the worst on the dataset
$(\textnormal{C}6,\textnormal{C}10)$. $(\textnormal{C}3,\textnormal{C}11)$ is
a very short-tailed dataset with high standard deviation in the label
proportion and it well-separated. This makes this dataset relatively easy to
learn on. On the other hand, $(\textnormal{C}6,\textnormal{C}10)$ is a long-
tailed dataset with very low standard deviation in the label proportion and it
Less-Separated. This dataset represents the worst-case scenario of the three
feature combination making it very hard to learn on.

###  7.2 Dataset Analysis

In this section, we select a few datasets that do not perform as expected in
the trend-lines of Figure 3. We analyse them further to explain the
performance metrics that we observe. In order to perform a subjective analysis
of these datasets, we classify them based on their metric values. We classify
them based on the following criteria:
Tail size: We perform 4-Means clustering where each dataset is represented by
a four tuple of bag size at x percentile where $x\in\\{50,70,85,95\\}$ (See
Sec 3.1). We name these clusters as very short-tailed, short-tailed, long-
tailed and very long-tailed in increasing order of mean bag size at 70
percentile of each cluster. The classification of all LLP-Bench datasets on
tail size is reported in Appendix A.10
Label Variation: We perform 4-Means clustering with each dataset represented
by ${\sf LabelPropStdev}$. We classify the datasets as low, medium, high and
very high ${\sf LabelPropStdev}$ dataset in increasing order of the mean ${\sf
LabelPropStdev}$ of the cluster. The classification of all LLP-Bench datasets
on label variation is reported in Appendix A.11
Bag Separation: We perform 4-Means clustering with each dataset represented by
${\sf InterIntraRatio}$. We classify the datasets as less-separated, medium-
separated, well-separated and far-separated in increasing order of mean ${\sf
InterIntraRatio}$ of each cluster. The classification of all LLP-Bench
datasets on ${\sf InterIntraRatio}$ is reported Appendix A.12

Using this classification, we present analysis of a few datasets below:

1. Datasets $(\textnormal{C}4,\textnormal{C}15)$ and $(\textnormal{C}4,\textnormal{C}10)$ are medium-separated but they perform better than other such datasets (Fig 2(f)). This is because they are short-tailed and very short-tailed respectively.
2. Datasets $(\textnormal{C}7,\textnormal{C}8)$ and $(\textnormal{C}1,\textnormal{C}7)$ are well-separated but they perform worse than other such datasets (Fig 2(f)) because they have very low ${\sf LabelPropStddev}$ and are long-tailed. Similarly, datasets $(\textnormal{C}6,\textnormal{C}7),(\textnormal{C}7,\textnormal{C}14)$ and $(\textnormal{C}7,\textnormal{C}20)$ are all well-separated and yet they perform poorly that other such datasets. They are all very long-tailed and have very low ${\sf LabelPropStdDev}$ (except $(\textnormal{C}7,\textnormal{C}14)$ which has a low ${\sf LabelPropStdDev}$).
3. It can also be observed from Fig 2(e) that the datasets $(\textnormal{C}7,\textnormal{C}20),(\textnormal{C}1,\textnormal{C}7)$ and $(\textnormal{C}7,\textnormal{C}8)$ actually perform better as compared to other very low ${\sf LabelPropStdDev}$ datasets because they are long-tailed and well-separated (except $(\textnormal{C}7,\textnormal{C}20)$ which is very long-tailed)
4. Dataset $(\textnormal{C}4,\textnormal{C}15)$ performs poorly as compared to other medium ${\sf LabelPropStddev}$ datasets even when it is short-tailed as it is less-separated (Fig 2(e))
5. Dataset $(\textnormal{C}7,\textnormal{C}26)$ is really interesting since it performs poorly as compared to other medium ${\sf LabelPropStddev}$, far-separated and short-tailed datasets.
6. Datasets $(\textnormal{C}2,\textnormal{C}11)$ and $(\textnormal{C}2,\textnormal{C}13)$ perform slightly better than other datasets with comparable ${\sf MeanBagSize}$ (Fig 2(d)). They are both long-tailed, with low ${\sf LabelPropStddev}$ and medium-separated and hence their higher performance cannot be explained based on our metrics.

##  8 Conclusion

We present the design of LLP-Bench: a diverse collection of tabular LLP
datasets from the Criteo dataset as a benchmark for evaluating LLP techniques.
In this process, our work analyzes bag collections given by grouping on at
most two categorical features, based on their distribution of bags as well as
label proportions. We show that LLP-Bench has significant diversity in the
nature of the datasets that are present in it. To the best of our knowledge,
LLP-Bench is the first large scale tabular benchmark with extensive diversity
in the underlying datasets. We presented a detailed analysis of 9 SOTA
baselines on LLP-Bench and explained their performance by correlating it with
the underlying dataset characteristics. Again, to the best of our knowledge no
other study has compared SOTA techniques to this level of detail that we have.
We believe our work addresses to a great extent the current lack of a large
scale tabular LLP benchmark. LLP-Bench along with the four dataset hardness
metrics can be used to systematically study and design new LLP techniques.

Limitations and Future Work. While the performance and outlier analysis
(Sections 6.1 and 7) use three metrics, a deeper explanation of outliers could
be possible using additional metrics. Future work could also incorporate more
LLP algorithms as well as additional model architectures. The grouping based
bag creation, and the data analysis techniques proposed in this work are
applicable to other tabular datasets. However, we use only the Criteo CTR
dataset to construct the LLP benchmark datasets, and future work including
more such tabular datasets would yield a greater diversity of LLP datasets.

## References

  * Ardehaly and Culotta [2017] Ehsan Mohammady Ardehaly and Aron Culotta.  Co-training for demographic classification using deep learning from label proportions.  In _ICDM_ , pages 1017–1024, 2017.
  * Bortsova et al. [2018a] Gerda Bortsova, Florian Dubost, Silas Ørting, Ioannis Katramados, Laurens Hogeweg, Laura Thomsen, Mathilde Wille, and Marleen de Bruijne.  Deep learning from label proportions for emphysema quantification.  In _Medical Image Computing and Computer Assisted Intervention–MICCAI 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II 11_ , pages 768–776. Springer, 2018a.
  * Bortsova et al. [2018b] Gerda Bortsova, Florian Dubost, Silas N. Ørting, Ioannis Katramados, Laurens Hogeweg, Laura H. Thomsen, Mathilde M. W. Wille, and Marleen de Bruijne.  Deep learning from label proportions for emphysema quantification.  In _MICCAI_ , volume 11071 of _Lecture Notes in Computer Science_ , pages 768–776. Springer, 2018b.  URL https://arxiv.org/abs/1807.08601.
  * Busa-Fekete et al. [2023] Róbert Istvan Busa-Fekete, Heejin Choi, Travis Dick, Claudio Gentile, and Andrés Muñoz Medina.  Easy learning from label proportions.  _CoRR_ , abs/2302.03115, 2023.  doi: 10.48550/arXiv.2302.03115.  URL https://doi.org/10.48550/arXiv.2302.03115.
  * Chen et al. [2004] Lei Chen, Zheng Huang, and Raghu Ramakrishnan.  Cost-based labeling of groups of mass spectra.  In _Proceedings of the 2004 ACM SIGMOD international conference on Management of data_ , pages 167–178, 2004.
  * Chen et al. [2023a] Lin Chen, Gang Fu, Amin Karbasi, and Vahab Mirrokni.  Learning from aggregated data: Curated bags versus random bags.  _CoRR_ , abs/2305.09557, 2023a.  doi: 10.48550/arXiv.2305.09557.  URL https://doi.org/10.48550/arXiv.2305.09557.
  * Chen et al. [2023b] Lin Chen, Thomas Fu, Amin Karbasi, and Vahab Mirrokni.  Learning from aggregated data: Curated bags versus random bags.  _arXiv preprint arXiv:2305.09557_ , 2023b.
  * Chen et al. [2009] Shuo Chen, Bin Liu, Mingjie Qian, and Changshui Zhang.  Kernel k-means based framework for aggregate outputs classification.  In Yücel Saygin, Jeffrey Xu Yu, Hillol Kargupta, Wei Wang, Sanjay Ranka, Philip S. Yu, and Xindong Wu, editors, _ICDM_ , pages 356–361, 2009.
  * Criteo [2014] Criteo.  Kaggle display advertising challenge dataset, 2014.  URL http://labs.criteo.com/2014/02/kaggle-display-advertising-challenge-dataset/.
  * de Freitas and Kück [2005] Nando de Freitas and Hendrik Kück.  Learning about individuals from group statistics.  In _UAI_ , pages 332–339, 2005.
  * Dery et al. [2017] L. M. Dery, B. Nachman, F. Rubbo, and A. Schwartzman.  Weakly supervised classification in high energy physics.  _Journal of High Energy Physics_ , 2017(5):1–11, 2017.
  * Dua and Graff [2017] Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.  URL http://archive.ics.uci.edu/ml.
  * Dulac-Arnold et al. [2019] Gabriel Dulac-Arnold, Neil Zeghidour, Marco Cuturi, Lucas Beyer, and Jean-Philippe Vert.  Deep multi-class learning from label proportions.  _CoRR_ , abs/1905.12909, 2019.  URL http://arxiv.org/abs/1905.12909.
  * Dulac-Arnold et al. [2019] Gabriel Dulac-Arnold, Neil Zeghidour, Marco Cuturi, Lucas Beyer, and Jean-Philippe Vert.  Deep multi-class learning from label proportions.  _arXiv preprint arXiv:1905.12909_ , 2019.
  * Florencio and Herley [2007] Dinei Florencio and Cormac Herley.  A large-scale study of web password habits.  In _Proceedings of the 16th international conference on World Wide Web_ , pages 657–666, 2007.
  * He et al. [2014] Xinran He, Junfeng Pan, Ou Jin, Tianbing Xu, Bo Liu, Tao Xu, Yanxin Shi, Antoine Atallah, Ralf Herbrich, Stuart Bowers, et al.  Practical lessons from predicting clicks on ads at facebook.  In _Proceedings of the eighth international workshop on data mining for online advertising_ , pages 1–9, 2014.
  * Hernández-González et al. [2013] Jerónimo Hernández-González, Iñaki Inza, and José Antonio Lozano.  Learning bayesian network classifiers from label proportions.  _Pattern Recognit._ , 46(12):3425–3440, 2013\.
  * Hernández-González et al. [2018] Jerónimo Hernández-González, Iñaki Inza, Lorena Crisol-Ortíz, María A. Guembe, María J Iñarra, and Jose A Lozano.  Fitting the data from embryo implantation prediction: Learning from label proportions.  _Statistical methods in medical research_ , 27(4):1056–1066, 2018.
  * Kotzias et al. [2015] Dimitrios Kotzias, Misha Denil, Nando de Freitas, and Padhraic Smyth.  From group to individual labels using deep features.  In _SIGKDD_ , pages 597–606, 2015.
  * Liu et al. [2019a] Jiabin Liu, Bo Wang, Zhiquan Qi, Yingjie Tian, and Yong Shi.  Learning from label proportions with generative adversarial networks.  In _NeurIPS_ , pages 7167–7177, 2019a.
  * Liu et al. [2019b] Jiabin Liu, Bo Wang, Zhiquan Qi, Yingjie Tian, and Yong Shi.  Learning from label proportions with generative adversarial networks.  _Advances in neural information processing systems_ , 32, 2019b.
  * Liu et al. [2021] Jiabin Liu, Bo Wang, Xin Shen, Zhiquan Qi, and Yingjie Tian.  Two-stage training for learning from label proportions.  In Zhi-Hua Zhou, editor, _Proc. IJCAI_ , pages 2737–2743, 2021\.
  * Liu et al. [2022] Jiabin Liu, Zhiquan Qi, Bo Wang, YingJie Tian, and Yong Shi.  Self-llp: Self-supervised learning from label proportions with self-ensemble.  _Pattern Recognition_ , 129:108767, 2022.
  * McMahan et al. [2013] H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al.  Ad click prediction: a view from the trenches.  In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 1222–1230, 2013.
  * Musicant et al. [2007] David R. Musicant, Janara M. Christensen, and Jamie F. Olson.  Supervised learning by training on aggregate outputs.  In _ICDM_ , pages 252–261. IEEE Computer Society, 2007.
  * O’Brien et al. [2022] Conor O’Brien, Arvind Thiagarajan, Sourav Das, Rafael Barreto, Chetan Verma, Tim Hsu, James Neufeld, and Jonathan J. Hunt.  Challenges and approaches to privacy preserving post-click conversion prediction.  _CoRR_ , abs/2201.12666, 2022.  URL https://arxiv.org/abs/2201.12666.
  * Ørting et al. [2016] Silas Nyboe Ørting, Jens Petersen, Mathilde Wille, Laura Thomsen, and Marleen de Bruijne.  Quantifying emphysema extent from weakly labeled ct scans of the lungs using label proportions learning.  In _The Sixth International Workshop on Pulmonary Image Analysis_ , pages 31–42, 2016.
  * Patrini et al. [2014] Giorgio Patrini, Richard Nock, Tibério S. Caetano, and Paul Rivera.  (almost) no label no cry.  In Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinberger, editors, _Advances in Neural Information Processing Systems_ , pages 190–198, 2014.
  * Quadrianto et al. [2009] Novi Quadrianto, Alexander J. Smola, Tibério S. Caetano, and Quoc V. Le.  Estimating labels from label proportions.  _J. Mach. Learn. Res._ , 10:2349–2374, 2009.
  * Rüping [2010] Stefan Rüping.  SVM classifier estimation from group probabilities.  In Johannes Fürnkranz and Thorsten Joachims, editors, _ICML_ , pages 911–918, 2010.
  * Saket [2021] Rishi Saket.  Learnability of linear thresholds from label proportions.  In _NeurIPS_ , pages 6555–6566, 2021.
  * Saket [2022] Rishi Saket.  Algorithms and hardness for learning linear thresholds from label proportions.  In _NeurIPS_ , 2022.
  * Saket et al. [2022] Rishi Saket, Aravindan Raghuveer, and Balaraman Ravindran.  On combining bags to better learn from label proportions.  In _AISTATS_ , volume 151 of _Proceedings of Machine Learning Research_ , pages 5913–5927. PMLR, 2022.  URL https://proceedings.mlr.press/v151/saket22a.html.
  * Scott and Zhang [2020] Clayton Scott and Jianxin Zhang.  Learning from label proportions: A mutual contamination framework.  In _NeurIPS_ , 2020.
  * Song et al. [2019] Weiping Song, Chence Shi, Zhiping Xiao, Zhijian Duan, Yewen Xu, Ming Zhang, and Jian Tang.  Autoint: Automatic feature interaction learning via self-attentive neural networks.  In _CIKM_ , 2019.
  * Stolpe and Morik [2011] Marco Stolpe and Katharina Morik.  Learning from label proportions by optimizing cluster model selection.  In Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba, and Michalis Vazirgiannis, editors, _ECML PKDD Proceedings, Part III_ , volume 6913, pages 349–364. Springer, 2011.
  * Tsai and Lin [2020] Kuen-Han Tsai and Hsuan-Tien Lin.  Learning from label proportions with consistency regularization.  In _Asian Conference on Machine Learning_ , pages 513–528. PMLR, 2020\.
  * Wojtusiak et al. [2011] J. Wojtusiak, K. Irvin, A. Birerdinc, and A. V. Baranova.  Using published medical results and non-homogenous data in rule learning.  In _Proc. International Conference on Machine Learning and Applications and Workshops_ , volume 2, pages 84–89. IEEE, 2011.
  * Yu et al. [2013] Felix X. Yu, Dong Liu, Sanjiv Kumar, Tony Jebara, and Shih-Fu Chang.  $\propto$SVM for learning with label proportions.  In _ICML_ , volume 28 of _JMLR Workshop and Conference Proceedings_ , pages 504–512, 2013.
  * Zhang et al. [2022] Jianxin Zhang, Yutong Wang, and Clay Scott.  Learning from label proportions by learning with label noise.  _Advances in Neural Information Processing Systems_ , 35:26933–26942, 2022.

##  Appendix A Proofs of Lemmas and Algorithms

While ${\sf BagSep}$ is not a metric since ${\sf BagSep}(B,B)$ is not
necessarily zero, the following lemma (proved in Appendix A.1) shows that it
does satisfy the other metric properties.

###### Lemma A.1

${\sf BagSep}$ satisfies non-negativity, symmetry and triangle inequality.

We have the following lemma proved in Appendix A.2.

###### Lemma A.2

For any bag $B$, (i) ${\sf InterBagSep}(B,d)/{\sf BagSep}(B,B,d)\geq 1/2$ when
$d$ is a metric, (ii) ${\sf InterBagSep}(B,d)/{\sf BagSep}(B,B,d)\geq 1/4$
when $d$ is the $\ell_{2}^{2}$ distance.

The following is a straightforward corollary of Lemma A.2.

###### Corollary A.3

(i) When $d$ is a metric: ${\sf MeanInterBagSep}(\mathcal{B},d)/{\sf
MeanIntraBagSep}(\mathcal{B},d)\geq 1/2.$ (ii) When $d$ is the $\ell_{2}^{2}$
distance: ${\sf MeanInterBagSep}(\mathcal{B},d)/{\sf
MeanIntraBagSep}(\mathcal{B},d)\geq 1/4.$

We expect this ratio to achieve values substantially less than $1$ in
adversarial cases. Appendix A.5 provides an example of such a case. For
convenience, for $\mathcal{B}$, we use ${\sf InterIntraRatio}$ to denote ${\sf
MeanInterBagSep}(\mathcal{B},d)/{\sf MeanIntraBagSep}(\mathcal{B},d)$ when
$d=\ell_{2}^{2}$.

###  A.1 Proof of Lemma A.1

  * Proof.

From Def. 3.2, the non-negativity and symmetry properties are obvious.

Triangle Inequality : let $B_{1},B_{2},B_{3}\in\mathcal{B}$, and we use the
following notation for convenience: $B_{1}=\\{x_{i}|i\in[n]\\}$,
$B_{2}=\\{y_{j}|j\in[m]\\}$, $B_{3}=\\{z_{k}|k\in[l]\\}$. As $d$ is a metric,
we know that for all $i\in[n],j\in[m]$ and $k\in[l]$, $d(x_{i},z_{k})\leq
d(x_{i},y_{j})+d(y_{j},z_{k})$. Hence,

$\displaystyle
d(x_{i},z_{k})\leq\frac{\sum_{j=1}^{j=m}d(x_{i},y_{j})}{m}+\frac{\sum_{j=1}^{j=m}d(y_{j},z_{k})}{m}$
$\displaystyle\Rightarrow$
$\displaystyle\frac{\sum_{i=1}^{i=n}d(x_{i},z_{k})}{n}\leq\frac{\sum_{i=1}^{i=n}\sum_{j=1}^{j=m}d(x_{i},y_{j})}{nm}+\frac{\sum_{j=1}^{j=m}d(y_{j},z_{k})}{m}$
$\displaystyle\Rightarrow$
$\displaystyle\frac{\sum_{k=1}^{k=l}\sum_{i=1}^{i=n}d(x_{i},z_{k})}{ln}\leq\frac{\sum_{i=1}^{i=n}\sum_{j=1}^{j=m}d(x_{i},y_{j})}{nm}+\frac{\sum_{k=1}^{k=l}\sum_{j=1}^{j=m}d(y_{j},z_{k})}{ml}$
$\displaystyle\Rightarrow$ $\displaystyle{\sf BagSep}(B_{1},B_{3},d)\leq{\sf
BagSep}(B_{1},B_{2},d)+{\sf BagSep}(B_{2},B_{3},d)$

$\square$

###  A.2 Proof of Lemma A.2

  * Proof.

Let $B\in\mathcal{B}$. Using triangle inequality and symmetry from Lemma A.1:

$\displaystyle\leavevmode\nobreak\ \forall B^{\prime}\in\mathcal{B},{\sf
BagSep}(B,B,d)\leq{\sf BagSep}(B,B^{\prime},d)+{\sf BagSep}(B^{\prime},B,d)$
$\displaystyle\Rightarrow$ $\displaystyle\leavevmode\nobreak\ \forall
B^{\prime}\in\mathcal{B},{\sf BagSep}(B,B,d)\leq 2{\sf
BagSep}(B^{\prime},B,d)$ $\displaystyle\Rightarrow$ $\displaystyle{\sf
BagSep}(B,B,d)\leq 2\frac{\sum_{B^{\prime}\in\mathcal{B},B^{\prime}\not=B}{\sf
BagSep}(B^{\prime},B,d)}{|\mathcal{B}|-1}$ $\displaystyle\Rightarrow$
$\displaystyle{\sf BagSep}(B,B,d)\leq 2{\sf InterBagSep}(B,d)$
$\displaystyle\Rightarrow$ $\displaystyle{\sf InterBagSep}(B,d)/{\sf
BagSep}(B,B,d)\geq 1/2$

$\square$

The squared euclidean distance is not a metric as it follows all properties
other than the triangle inequality. Hence, we show the following

###### Lemma A.4

For any $a,b\in R^{n}$,
$\frac{1}{2}||a+b||_{2}^{2}\leq||a||_{2}^{2}+||b||_{2}^{2}$

###### Theorem A.5

Given $X$, $Y$ and $\mathcal{B}$, for any $B_{1},B_{2},B_{3}\in\mathcal{B}$,

$\frac{1}{2}{\sf BagSep}(B_{1},B_{3},\ell_{2}^{2})\leq{\sf
BagSep}(B_{1},B_{2},\ell_{2}^{2})+{\sf BagSep}(B_{2},B_{3},\ell_{2}^{2})$

  * Proof.

Follows by replacing triangle inequality in Lemma A.1 with inequality in Lemma
A.4 $\square$

###### Corollary A.6

${\sf InterBagSep}(B,\ell_{2}^{2})/{\sf BagSep}(B,B,\ell_{2}^{2})\geq 1/4$

  * Proof.

Follows by replacing inequality in proof of Lemma A.2 with inequality in
Theorem A.5 $\square$

###  A.3 Proof of Corollary A.3

  * Proof.

Given $X$, $Y$ and $\mathcal{B}$, and metric $d$ in $R^{n}$. Starting with
inequality in Lemma A.2

$\displaystyle\leavevmode\nobreak\ \forall B\in\mathcal{B},{\sf
BagSep}(B,B,d)\leq 2{\sf InterBagSep}(B,d)$ $\displaystyle\Rightarrow$
$\displaystyle\sum_{B\in\mathcal{B}}{\sf BagSep}(B,B,d)\leq
2\sum_{B\in\mathcal{B}}{\sf InterBagSep}(B,d)$ $\displaystyle\Rightarrow$
$\displaystyle\frac{1}{|\mathcal{B}|}{\sf BagSep}(B,B,d)\leq
2\frac{1}{|\mathcal{B}|}{\sf InterBagSep}(B,d)$ $\displaystyle\Rightarrow$
$\displaystyle{\sf MeanInterBagSep}(\mathcal{B},d)/{\sf
MeanIntraBagSep}(\mathcal{B},d)\geq 1/2$

Starting with inequality for $\ell_{2}^{2}$-distance in Lemma A.2, we get
${\sf MeanInterBagSep}(\mathcal{B},\ell_{2}^{2})/{\sf
MeanIntraBagSep}(\mathcal{B},\ell_{2}^{2})\geq 1/2$ $\square$

###  A.4 Bag Distance Results using squared euclidean distance

We use the squared euclidean distance to compute the bag distances as it makes
the computation faster. Algorithm 1 is used to compute the Bag Separation for
any general metric $d$.

Data: Set of bags $\mathcal{B}$, metric $d$ on $R^{n}$

Result: ${\sf BagSepMatrix}(\mathcal{B},d)$

${\sf BagSepMatrix}\leftarrow[0]_{|\mathcal{B}|x|\mathcal{B}|}$

for  _$B_{1}\in\mathcal{B}$_ do

for  _$B_{2}\in\mathcal{B}$_ do

for  _$i\in B_{1}$_ do

for  _$j\in B_{2}$_ do

${\sf BagSepMatrix}[B_{1},B_{2}]\leftarrow{\sf
BagSepMatrix}[B_{1},B_{2}]+d(x^{(i)},x^{(j)})$

end for

end for

${\sf BagSepMatrix}[B_{1},B_{2}]\leftarrow{\sf
BagSepMatrix}[B_{1},B_{2}]/(|B_{1}||B_{2}|)$

end for

end for

Algorithm 1 Compute Bag Separation of a dataset

###### Theorem A.7

Assuming the Bags to be disjoint, the running time of Algorithm 1 is
$O(m^{2}n)$ where $m$ is the number of examples and n is the dimension of the
input space.

  * Proof.

Runtime =
$\sum_{B_{1}\in\mathcal{B}}\sum_{B_{2}\in\mathcal{B}}|B_{1}||B_{2}|n=m^{2}n$
$\square$

Now, this computation can be simplified due to the following. Let
$\|B\|:=\frac{1}{|B|}\sum_{x\in B}\|x\|_{2}^{2}$ and
$\mu(B):=\frac{1}{|B|}\sum_{x\in B}\,x$

###### Lemma A.8

For any $B,B^{\prime}\in\mathcal{B}$,

${\sf
BagSep}(B,B^{\prime},\ell_{2}^{2})=\|B\|+\|B^{\prime}\|-2\langle\mu(B),\mu(B^{\prime})\rangle$

  * Proof.

Let $B=\\{x_{i}|i\in[n]\\},B^{\prime}=\\{y_{j}|j\in[m]\\}$

$\displaystyle{\sf
BagSep}(B,B^{\prime},\ell_{2}^{2})=\frac{1}{mn}\sum_{i=1}^{i=n}\sum_{j=1}^{j=m}\|x_{i}-y_{j}\|_{2}^{2}$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{i=n}\|x_{i}\|_{2}^{2}+\frac{1}{m}\sum_{j=1}^{j=m}\|y_{j}\|_{2}^{2}-\frac{2}{mn}\sum_{i=1}^{i=n}\sum_{j=1}^{j=m}\langle
x_{i},y_{j}\rangle$
$\displaystyle=\frac{1}{n}\sum_{i=1}^{i=n}\|x_{i}\|_{2}^{2}+\frac{1}{m}\sum_{j=1}^{j=m}\|y_{j}\|_{2}^{2}-\frac{2}{mn}\langle\sum_{i=1}^{i=n}x_{i},\sum_{j=1}^{j=m}y_{j}\rangle$

$\square$

###### Lemma A.9

Given $\mathcal{B}$ and $B\in\mathcal{B}$,

$\displaystyle{\sf IntraBagSep}(B,\ell_{2})$ $\displaystyle=$ $\displaystyle
2[\|B\|-\|\mu(B)\|_{2}^{2}]$ $\displaystyle{\sf
MeanInterBagSep}(\mathcal{B},\ell_{2})$ $\displaystyle=$
$\displaystyle\frac{2}{|\mathcal{B}|}\sum_{B\in\mathcal{B}}\|B\|$
$\displaystyle+$
$\displaystyle\frac{2}{|\mathcal{B}|(|\mathcal{B}|-1)}\left[\|\sum_{B\in\mathcal{B}}\mu(B)\|_{2}^{2}-\sum_{B\in\mathcal{B}}\|\mu(B)\|_{2}^{2}\right]$

  * Proof.

First part is trivial from Lemma A.8. If $\mathcal{B}=\\{B_{i}|i\in[m]\\}$,

$\displaystyle{\sf
MeanInterBagSep}(\mathcal{B},\ell_{2})=\frac{1}{m(m-1)}\underset{i\not=j}{\sum_{i=1}^{m}\sum_{j=1}^{m}}{\sf
BagSep}(B_{i},B_{j},\ell_{2})$ $\displaystyle=$
$\displaystyle\frac{1}{m(m-1)}\underset{i\not=j}{\sum_{i=1}^{m}\sum_{j=1}^{m}}(\|B_{i}\|+\|B_{j}\|-\langle\mu(B_{i}),\mu(B_{j})\rangle)$
$\displaystyle=$ $\displaystyle\frac{2}{m}\sum_{i=1}^{m}\|B_{i}\|$
$\displaystyle-$
$\displaystyle\frac{2}{m(m-1)}\left[\sum_{i=1}^{m}\sum_{j=1}^{m}\langle\mu(B_{i}),\mu(B_{j})\rangle-\sum_{i=1}^{m}\|\mu(B_{i})\|_{2}^{2}\right]$
$\displaystyle=$
$\displaystyle\frac{2}{m}\sum_{i=1}^{m}\|B_{i}\|-\frac{2}{m(m-1)}\left[\|\sum_{i=1}^{m}\mu(B_{i})\|_{2}^{2}-\sum_{i=1}^{m}\|\mu(B_{i})\|_{2}^{2}\right]$

$\square$

Algorithm 2 is used to compute the Bag Separation for squared euclidean
distance.

Data: Set of bags $\mathcal{B}$

Result: ${\sf MeanIntraBagSep}(\mathcal{B},\ell_{2}^{2}),{\sf
MeanInterBagSep}(\mathcal{B},\ell_{2}^{2})$

${\sf MeanIntraBagSep}\leftarrow 0$

${\sf MeanInterBagSep}\leftarrow 0$

${\sf AvgSqNorm}\leftarrow[0]_{|\mathcal{B}|}$

${\sf BagMeans}\leftarrow[0]_{|\mathcal{B}|xn}$

${\sf SumofAvgSqNorm}\leftarrow 0$

${\sf SumofBagMeans}\leftarrow[0]_{1xn}$

${\sf SumofBagMeansNorms}\leftarrow 0$

for  _$B\in\mathcal{B}$_ do

for  _$i\in B$_ do

${\sf AvgSqNorm}(B)\leftarrow{\sf AvgSqNorm}(B)+\|x^{(i)}\|_{2}^{2}$

${\sf BagMeans}(B)\leftarrow{\sf BagMeans}(B)+x^{(i)}$

end for

${\sf AvgSqNorm}(B)\leftarrow{\sf AvgSqNorm}(B)/|B|$

${\sf BagMeans}(B)\leftarrow{\sf BagMeans}(B)/|B|$

end for

for  _$B\in\mathcal{B}$_ do

${\sf MeanIntraBagSep}\leftarrow{\sf MeanIntraBagSep}+2[{\sf
AvgSqNorm}(B)-\|{\sf BagMeans}(B)\|_{2}^{2}]$

${\sf SumofAvgSqNorm}\leftarrow{\sf SumofAvgSqNorm}+{\sf AvgSqNorm}(B)$

${\sf SumofBagMeans}\leftarrow{\sf SumofBagMeans}+{\sf BagMeans(B)}$

${\sf SumofBagMeansNorms}\leftarrow{\sf SumofBagMeansNorms}+\|{\sf
BagMeans}(B)\|_{2}^{2}$

end for

${\sf MeanIntraBagSep}\leftarrow{\sf MeanIntraBagSep}/|\mathcal{B}|$

${\sf MeanInterBagSep}\leftarrow\frac{2}{|\mathcal{B}|}{\sf
SumofAvgSqNorms}-\frac{2}{|\mathcal{B}|(|\mathcal{B}|-1)}[\|{\sf
SumofBagMeans}\|_{2}^{2}-{\sf SumofBagMeansNorms}]$

Algorithm 2 Compute Bag Separation with squared euclidean distance

###### Theorem A.10

Assuming the Bags to be disjoint, the running time of Algorithm 2 is
$O(mn+|\mathcal{B}|n+|\mathcal{B}|)$ where $m$ is the number of examples and n
is the dimension of the input space.

  * Proof.

Runtime =
$\sum_{B\in\mathcal{B}}|B|n+\sum_{B\in\mathcal{B}}(1+n)=mn+|\mathcal{B}|n+|\mathcal{B}|$
$\square$

###  A.5 Adversarial Example of Bags with Ratio of Mean Inter to Intra Bag
Separation as 1/2

Consider $X=\\{x^{(1)},x^{(2)},x^{(3)}\\}$ which lie on a straight line. The
distances are as follows:

  * •

$d(x^{(1)},x^{(2)})=d_{1}$

  * •

$d(x^{(2)},x^{(3)})=d_{2}$

  * •

$d(x^{(1)},x^{(3)})=d_{1}+d_{2}$

We have two bags $B_{1}=\\{x^{(1)},x^{(3)}\\}$ and $B_{2}=\\{x^{(2)}\\}$. The
Intra-bag separations for both of them are as follows:

  * •

${\sf
BagSep}(B_{1},B_{1},d)=\frac{1}{2^{2}}(d(x^{(1)},x^{(1)})+d(x^{(1)},x^{(3)})+d(x^{(3)},x^{(1)})+d(x^{(3)},x^{(3)}))=\frac{1}{2}(d_{1}+d_{2})$

  * •

${\sf BagSep}(B_{2},B_{2},d)=0$

Hence, ${\sf MeanIntraBagSep}(\mathcal{B},d)=\frac{1}{4}(d_{1}+d_{2})$. Now,
the bag separation between the bags is as follows:

  * •

${\sf BagSep}(B_{1},B_{2},d)=\frac{1}{1\times
2}(d(x^{(1)},x^{(2)})+d(x^{(3)},x^{(2)}))=\frac{1}{2}(d_{1}+d_{2})$

  * •

${\sf InterBagSep}(B_{1},d)=\frac{1}{2-1}({\sf
BagSep}(B_{1},B_{2},d))=\frac{1}{2}(d_{1}+d_{2})$

  * •

${\sf InterBagSep}(B_{2},d)=\frac{1}{2-1}({\sf
BagSep}(B_{2},B_{1},d))=\frac{1}{2}(d_{1}+d_{2})$

Hence, ${\sf MeanInterBagSep}(\mathcal{B},d)=\frac{1}{2}(d_{1}+d_{2})$.
Hence, ${\sf MeanInterBagSep}(\mathcal{B},d)/{\sf
MeanIntraBagSep}(\mathcal{B},d)=1/2$

###  A.6 Additional Details of Experimental Setup

We begin with additional details on the different baselines evaluated in our
experiments.

${\bf DLLP}$: For a bag $B$, the ${\sf DLLP\text{-}BCE}$ loss is given by
${\sf bce}(y_{B}/|B|,\hat{y}_{B}/|B|)$ where $y_{B}$ and $\hat{y}_{B}$ are the
given and predicted label sums of bag $B$ where ${\sf bce}$ is the binary
cross-entropy, and that of ${\sf DLLP\text{-}MSE}$ is
$(y_{B}-\hat{y}_{B})^{2}$. The minibatch loss is the sum of the per-bag losses
over the $8$ bags in the minibatch.
${\bf GenBags}$: We divide the $8$ bags in a minibatch into $2$ blocks of $4$
bags each. For each block $60$ iid values of $\mathbf{w}=(w_{1},\dots,w_{4})$
are sampled from $N(\mathbf{0},\bm{\Sigma})$ where $\bm{\Sigma}$ is the inner
product matrix of the directions of the corners of the tetrahedron centered at
origin. In particular, the diagonal entries of $\bm{\Sigma}$ are $1$ and all
the off-diagonal entries are $-1/3$. This is a solution to the SDP of [33] for
the case of the $4$ bags in a minibatch being iid random, and we utilize this
in our implementation. For each of the $60$ samples of $\mathbf{w}$ we create
a generalized bag with those weights. In total we have $120$ generalized bags
derived from a minibatch of $8$ bags.
${\bf Easy\text{-}LLP}$: We directly implement the soft-surrogate label loss
given in Defn. 3.4 of [4] which is instantiated using the BCE loss at the
instance-level.
${\bf OT}$ methods: There are pseudo-labeling techniques based on optimal
transport proposed in [22]. We first train our model to convergence using
${\bf DLLP\text{-}BCE}$. Then we begin pseudo-labeling by constructing an OT
problem described in Eqn 7 of [22]. We implement this without Entropic
Regularization which we call ${\bf OT\text{-}LLP}$. We also implement it with
Entropic Regularization. We call these ${\bf Hard\text{-}OT\text{-}LLP}$ and
${\bf Soft\text{-}OT\text{-}LLP}$ based on whether we use hard or soft pseudo-
labels.
${\bf SIM\text{-}LLP}$: In this method proposed by [19], the bag-level DLLP
loss is augmented with a pairwise similarity based loss penalizing different
predictions of geometrically close feature-vectors (Eqn 3 in [19]). Since the
similarity based loss has number of terms which is square in the number of
feature-vectors in a minibatch, we sample a random set of $400$ feature-
vectors from each minbatch to apply this loss.
${\bf Mean\text{-}Map}$: The optimization given in Algorithm 1 of [29] is
implemented in two steps. The quantities $\hat{\mu}_{XY}$ therein are first
computed and then the computation for $\hat{\theta}^{*}$ is implemented using
a minibatch optimization along the same lines as the above methods.

###  A.7 Instance-level Model Training Results

We perform instance-level training of our model on Criteo Dataset for
comparison. We perform a train-test spilt of 80:20 on the dataset. We then
train using instance level mini-batch gradient descent for the same number of
epochs, using the same optimizer, model, learning rate schedule and the
instance-level variant of the loss function. We obtain an AUC score of 80.1
with BCE loss and an AUC score of 79.94 with MSE loss.

###  A.8 Baseline Training Results

Table 5 reports the AUC scores of all the baselines on our bags. We take the
best test AUC score during each training configuration. The mean and standard
deviation over 5 splits has been reported.

###  A.9 Bag creation and filtering Statistics

The statistics of datasets created as described in Section 4 before and after
clipping for all $349$ datasets are in Table LABEL:tab:bag_stats. We report
the number of bags created, number of bags retained after clipping, percentage
of instances left after clipping and the mean and standard deviation of bag
size in each dataset. The datasets which are emboldened pass our filter and
are used for training.

###  A.10 ${\bf CumuBagSizeDist}$ for LLP-Bench datasets

Table 6 contains the threshold bags sizes such that $t\%$ of the bags have at
most that size, for $t=50,70,85,95$ for LLP-Bench datasets. The Tail size
cluster to which each dataset is assigned is also listed.

###  A.11 ${\bf LabelPropStdev}$ and Label Variation clusters for LLP-Bench
datasets

Table 7 contains the ${\sf LabelPropStddev}$ for all LLP-Bench datasets. The
Label Variation cluter to which each dataset is assigned is also listed.

###  A.12 ${\sf InterIntraRatio}$ and Bag Separation clusters for all datasets

Table 8 contains ${\sf MeanInterBagSep}$, ${\sf MeanIntraBagSep}$ and ${\sf
InterIntraRatio}$ for all datasets in LLP-Bench. It also contains the
information of the Bag Separation cluster to which each of these datasets are
assigned.

###  A.13 Training results on Fixed size Feature-bags Datasets

We also create and train our model on fixed size feature bags. To create these
datasets, we first perform a 5-fold split of the Criteo Dataset. Next, for
each group key $\mathcal{C}$ corresponding to an LLP-Bench dataset, we
construct a random ordering of the train set with the constraint that feature
vectors with same values of attributes in $\mathcal{C}$ lie in a contiguous
segment. We then assign contiguous segments of size $k$ to the same bag to
create fixed size feature bags for $k\in\\{64,128,256,512\\}$. We train our
${\sf DLLP},{\sf GenBags},{\sf Easy\text{-}LLP},{\sf OT\text{-}LLP},{\sf
SIM\text{-}LLP}$ and ${\sf Mean\text{-}Map}$ baselines on these fixed size
feature bags and report the mean and std of test AUC scores in Table
LABEL:tab:feature_rand_auc. For each group key $\mathcal{C}$, Table
LABEL:tab:feature_rand_auc contains $4$ contiguous rows, one for bag size
$\\{64,128,256,512\\}$ each in ascending order.

###  A.14 Training results on Random Bags

Table 2 is replicated along with the standard deviation of AUC scores across 5
splits in Table 10.

###  A.15 Feature Bag Datasets with Group Key Size $3$

We also calculate the number of additional datasets which would have been
created had we also considered group keys of size $3$. Using
$instance_{thresh}\%$ as $30\%$, we retain $1195$ of ${26\choose 3}$ datasets.
It would be intractable to handle so many datasets and we believe that the
current benchmark provides sufficient diversity.

###  A.16 Analysis of label proportions of large bags

Large bags with label proportions close to $0$ or $1$ contain a considerable
amount of information. In this section we calculate the percentage of
instances which lie in large bags (bags with size greater than
$high_{thresh}=2500$) with skewed label proportions. We calculate this for
each feature bag dataset. Since we throw these bags out of our dataset, if the
percentage of such instances is low then we do not lose much information. We
say that the label proportion of a bag is skewed if either it is less than
${\varepsilon}$ or greater than $1-{\varepsilon}$.
Table 3 reports these percentages for ${\varepsilon}=0.1$ and
${\varepsilon}=0.05$ respectively. It can be seen that the maximum percentages
over all datasets with ${\varepsilon}=0.1$ and ${\varepsilon}=0.05$ are
$6.66\%$ and $1.42\%$ respectively. This shows that there are relatively low
number of such datapoints and they can be dropped so that neural network
training is tractable.

###  A.17 Dataset diversity analysis

Figure 6(a) shows a scatter plot with the three metrics, ${\sf MeanBagSize}$,
${\sf LabelPropStddev}$ and ${\sf InterIntraRatio}$ on its axes. Each point
represents one of the datasets in our benchmark. Figures 6(b), 6(c) and 6(d)
shows the projections of Figure 6(a) on ${\sf MeanBagSize}$ vs ${\sf
LabelPropStddev}$ plane, ${\sf LabelPropStddev}$ vs ${\sf InterIntraRatio}$
plane and ${\sf InterIntraRatio}$ vs ${\sf MeanBagSize}$ plane respectively.
The diversity of LLP-Bench is more apparent from these scatter plots as
diversity afforded due to combination of these metrics can be visualized.

For better readability and ease of reference we replicate (magnified versions
of) Figures 3 and 3 as Figures 4 and 5 respectively.

###  A.18 Cramer’s V between grouping-key and label for LLP-Bench Datasets

It is important to see how each grouping-key pair $(A,B)$ of LLP-Bench
feature-bag datasets is correlated with the label. If the correlation is high,
then most bags of the dataset corresponding to $(A,B)$ will have label
proportions close to $0$ or $1$. On the other hand, bags have mixed labels if
the correlation is low. Since both, the labels and grouping-key $(A,B)$ are
categorical, we compute Cramer’s V between them as follows. Given two
categorical features $(X,Y)$ such that $X\in\\{1,\dots,r\\}$ and
$Y\in\\{1,\dots,c\\}$, let $N$ be the total number of data points. Let
$O_{i,j}$ be the total number of times $X$ takes the value $i$ and $Y$ takes
the value $j$ for $(i,j)\in\\{1,\dots,r\\}\times\\{1,\dots,c\\}$. We define
the expected occurence of this event assuming independence of $X$ and $Y$ as
$E_{i,j}=Np_{i}q_{j}$ where $p_{i}=\sum_{j=1}^{c}O_{i,j}/N$ and
$q_{j}=\sum_{i=1}^{r}O_{i,j}/N$. We define ${\sf Cramer^{\prime}sV}$ as
follows.

$\displaystyle{\sf Cramer^{\prime}s\,V}$
$\displaystyle:=\sqrt{(\chi^{2}/N)/\min(r-1,c-1)}$ (1)
$\displaystyle\text{where}\quad\chi^{2}=\sum_{i=1}^{r}\sum_{j=1}^{c}\frac{(O_{i,j}-E_{i,j})^{2}}{E_{i,j}}$

For our case, we use the pairs $(A,B)$ as $X$ and the label as $Y$. Thus, $r$
will be the number of bags in that dataset and $c=2$ since it is a binary
classification dataset. $O_{i,0}$ and $O_{i,1}$ will be the number of
instances in the $i^{th}$ bag labeled $0$ and $1$ respectively. $N$ will be
the total number of instances in the dataset. We report the $\chi^{2}$ values,
total number of instances and ${\sf Cramer^{\prime}s\,V}$ for all LLP-Bench
datasets in Table 4. We observe a minimum ${\sf Cramer^{\prime}s\,V}$ of
$0.22$ and a maximum ${\sf Cramer^{\prime}s\,V}$ of $0.39$. The ${\sf
Cramer^{\prime}s\,V}$ is concentrated towards the maximium. We see sufficient
diversity in LLP-Bench dataset with respect to ${\sf Cramer^{\prime}s\,V}$.

(a) ${\sf MeanBagSize}$

(b) ${\sf LabelPropStdev}$

(c) ${\sf InterIntraRatio}$

Figure 4: Datasets vs. bag-level metrics: $y$-axis has the metric, $x$-axis
has the datsets. Replication of Figure 3.

(a) ${\sf MeanBagSize}$

(b) ${\sf LabelPropStdev}$

(c) ${\sf InterIntraRatio}$

Figure 5: Datasets performance: AUC scores on the y-axis, $x$-axis has the datasets ordered according to increasing metric. Replication of Figure 3.  Table 3: Percentage of large bags with label proportion less than ${\varepsilon}$ or greater than $1-{\varepsilon}$. Col1 | Col2 |  ${\varepsilon}$ = 0.1 |  ${\varepsilon}$ = 0.05
---|---|---|---
C1 | C7 | 4.18 | 0.64
C1 | C10 | 0.56 | 0.09
C2 | C7 | 4.05 | 1.16
C2 | C10 | 3.19 | 0.13
C2 | C11 | 3.7 | 1.11
C2 | C13 | 4.15 | 1.06
C3 | C7 | 2.12 | 0.51
C3 | C10 | 1.88 | 0.34
C3 | C11 | 2.23 | 0.63
C3 | C13 | 2.3 | 0.57
C4 | C7 | 1.45 | 0.62
C4 | C10 | 2.35 | 0.3
C4 | C11 | 1.63 | 0.61
C4 | C13 | 2.0 | 0.61
C4 | C15 | 6.66 | 1.42
C6 | C7 | 5.77 | 0.85
C6 | C10 | 0.79 | 0.1
C7 | C8 | 4.83 | 0.74
C7 | C10 | 6.28 | 1.06
C7 | C12 | 1.96 | 0.5
C7 | C14 | 6.23 | 1.07
C7 | C15 | 1.51 | 0.72
C7 | C16 | 1.5 | 0.5
C7 | C18 | 2.54 | 0.92
C7 | C20 | 5.64 | 1.07
C7 | C21 | 1.78 | 0.51
C7 | C24 | 2.01 | 0.79
C7 | C26 | 3.49 | 0.72
C10 | C12 | 1.92 | 0.34
C10 | C14 | 1.31 | 0.44
C10 | C15 | 3.87 | 0.79
C10 | C16 | 1.96 | 0.33
C10 | C17 | 3.45 | 0.13
C10 | C18 | 3.64 | 0.53
C10 | C20 | 0.7 | 0.11
C10 | C21 | 1.91 | 0.33
C10 | C24 | 2.63 | 0.3
C10 | C26 | 2.22 | 0.26
C11 | C12 | 2.08 | 0.59
C11 | C15 | 1.82 | 0.7
C11 | C16 | 1.69 | 0.57
C11 | C18 | 2.49 | 0.92
C11 | C21 | 1.84 | 0.6
C11 | C24 | 2.16 | 0.76
C11 | C26 | 3.35 | 0.59
C12 | C13 | 2.21 | 0.57
C13 | C15 | 2.28 | 0.7
C13 | C16 | 1.99 | 0.61
C13 | C18 | 3.07 | 0.95
C13 | C21 | 2.11 | 0.57
C13 | C24 | 2.46 | 0.72
C13 | C26 | 3.28 | 0.51

(a) ${\sf MeanBagSize}$ vs ${\sf LabelPropStddev}$ vs ${\sf InterIntraRatio}$

(b) ${\sf MeanBagSize}$ vs ${\sf LabelPropStddev}$

(c) ${\sf LabelPropStddev}$ vs ${\sf InterIntraRatio}$

(d) ${\sf InterIntraRatio}$ vs ${\sf MeanBagSize}$

Figure 6: Scatter plots for ${\sf MeanBagSize}$ vs ${\sf LabelPropStddev}$ vs ${\sf InterIntraRatio}$ and it’s projections Table 4: Cramer’s V between (Col1, Col2) and the label for each LLP-Bench dataset. Col1 | Col2 | $\chi^{2}$ | No. of inst. | Cramer’s V
---|---|---|---|---
C1 | C7 | 1282596.95 | 17379031 | 0.27
C1 | C10 | 736694.22 | 14870760 | 0.22
C2 | C7 | 2347843.42 | 22344522 | 0.32
C2 | C10 | 1511078.88 | 18901970 | 0.28
C2 | C11 | 1786730.98 | 17917297 | 0.32
C2 | C13 | 1504395.05 | 15984434 | 0.31
C3 | C7 | 2503281.56 | 17363106 | 0.38
C3 | C10 | 1704870.67 | 14670297 | 0.34
C3 | C11 | 2366786.11 | 16400086 | 0.38
C3 | C13 | 2269071.36 | 15942850 | 0.38
C4 | C7 | 3055496.7 | 20086331 | 0.39
C4 | C10 | 2131444.78 | 16541093 | 0.36
C4 | C11 | 2942577.74 | 20363904 | 0.38
C4 | C13 | 2854323.34 | 20103419 | 0.38
C4 | C15 | 1997730.99 | 14239811 | 0.37
C6 | C7 | 1220650.09 | 16429515 | 0.27
C6 | C10 | 767574.25 | 14575634 | 0.23
C7 | C8 | 1105463.37 | 15295058 | 0.27
C7 | C10 | 1188510.98 | 18574195 | 0.25
C7 | C12 | 2570090.33 | 17988476 | 0.38
C7 | C14 | 1451330.24 | 16030155 | 0.3
C7 | C15 | 3602798.92 | 24654648 | 0.38
C7 | C16 | 2803986.14 | 19177296 | 0.38
C7 | C18 | 3141314.13 | 23522367 | 0.37
C7 | C20 | 1072930.65 | 14600225 | 0.27
C7 | C21 | 2703648.85 | 18533629 | 0.38
C7 | C24 | 2931923.95 | 21495871 | 0.37
C7 | C26 | 2119663.82 | 16298693 | 0.36
C10 | C12 | 1739977.52 | 15026104 | 0.34
C10 | C14 | 1014656.15 | 14720299 | 0.26
C10 | C15 | 2379871.25 | 20472608 | 0.34
C10 | C16 | 1892293.04 | 15642966 | 0.35
C10 | C17 | 812645.45 | 14093150 | 0.24
C10 | C18 | 2004699.68 | 19393230 | 0.32
C10 | C20 | 731843.15 | 13752247 | 0.23
C10 | C21 | 1818401.1 | 15274402 | 0.35
C10 | C24 | 1936886.48 | 17551841 | 0.33
C10 | C26 | 1499388.71 | 14385623 | 0.32
C11 | C12 | 2450005.65 | 17203486 | 0.38
C11 | C15 | 3401121.57 | 24473873 | 0.37
C11 | C16 | 2695404.25 | 19011310 | 0.38
C11 | C18 | 2826737.32 | 21780144 | 0.36
C11 | C21 | 2602326.93 | 18061655 | 0.38
C11 | C24 | 2732340.42 | 20765237 | 0.36
C11 | C26 | 1920038.27 | 14392865 | 0.37
C12 | C13 | 2353479.64 | 16757452 | 0.37
C13 | C15 | 3295774.49 | 24040639 | 0.37
C13 | C16 | 2595905.41 | 18627647 | 0.37
C13 | C18 | 2608262.34 | 20669765 | 0.36
C13 | C21 | 2496448.67 | 17624005 | 0.38
C13 | C24 | 2599533.6 | 20247173 | 0.36
C13 | C26 | 1883510.23 | 13936389 | 0.37
Table 5: Test AUC scores on training baselines on LLP-Bench. Col1 | Col2 | ${\sf DLLP\text{-}BCE}$ | ${\sf DLLP\text{-}MSE}$ | ${\sf GenBags}$ | ${\sf Easy\text{-}LLP}$ | ${\sf OT\text{-}LLP}$ | ${\sf SIM\text{-}LLP}$ | ${\sf Soft\text{-}EROT\text{-}LLP}$ | ${\sf Hard\text{-}EROT\text{-}LLP}$ | ${\sf Mean\text{-}Map}$
---|---|---|---|---|---|---|---|---|---|---
C1 | C7 |  74.91±0.06 |  74.97±0.04 |  73.34±0.08 |  66.16±0.38 |  69.11±0.07 |  74.4±0.08 |  69.26±0.08 |  69.14±0.06 |  61.71±0.01
C1 | C10 |  73.69±0.05 |  73.67±0.07 |  71.87±0.1 |  65.84±0.09 |  68.96±0.08 |  73.01±0.05 |  69.27±0.05 |  65.3±8.55 |  61.81±0.04
C2 | C7 |  75.61±0.03 |  75.61±0.04 |  75.06±0.03 |  68.06±0.19 |  68.01±0.09 |  75.81±0.03 |  68.11±0.07 |  67.96±0.09 |  64.39±0.03
C2 | C10 |  74.44±0.04 |  74.42±0.04 |  73.71±0.05 |  67.3±0.15 |  67.85±0.2 |  74.41±0.02 |  67.99±0.07 |  68.03±0.12 |  64.23±0.03
C2 | C11 |  75.55±0.04 |  75.57±0.03 |  75.15±0.05 |  69.66±0.14 |  67.75±0.16 |  75.65±0.03 |  68.1±0.12 |  68.1±0.1 |  64.71±0.02
C2 | C13 |  75.24±0.06 |  75.24±0.04 |  74.83±0.06 |  69.5±0.05 |  67.57±0.17 |  75.34±0.04 |  67.8±0.13 |  67.96±0.13 |  64.41±0.02
C3 | C7 |  76.87±0.04 |  76.91±0.04 |  76.59±0.04 |  69.95±0.34 |  69.52±0.11 |  77.33±0.03 |  69.63±0.12 |  69.57±0.05 |  66.66±0.03
C3 | C10 |  75.86±0.03 |  75.84±0.04 |  75.29±0.03 |  69.82±0.29 |  69.86±0.18 |  76.21±0.06 |  70.0±0.08 |  70.02±0.09 |  66.45±0.05
C3 | C11 |  77.04±0.05 |  77.03±0.07 |  76.8±0.03 |  71.28±0.44 |  70.04±0.1 |  77.57±0.05 |  70.21±0.08 |  70.2±0.09 |  67.14±0.02
C3 | C13 |  76.89±0.03 |  76.89±0.03 |  76.64±0.07 |  71.62±0.27 |  70.0±0.03 |  77.44±0.05 |  70.1±0.14 |  70.08±0.11 |  67.08±0.03
C4 | C7 |  76.93±0.04 |  76.94±0.04 |  76.66±0.04 |  69.68±0.3 |  69.6±0.05 |  77.53±0.02 |  69.74±0.06 |  69.7±0.11 |  67.29±0.04
C4 | C10 |  76.28±0.02 |  76.29±0.02 |  75.78±0.03 |  69.76±0.39 |  70.15±0.15 |  76.75±0.02 |  70.22±0.06 |  70.2±0.11 |  67.15±0.02
C4 | C11 |  76.7±0.03 |  76.71±0.05 |  76.58±0.03 |  70.76±0.28 |  69.75±0.04 |  77.36±0.03 |  69.85±0.11 |  69.83±0.02 |  67.43±0.03
C4 | C13 |  76.53±0.04 |  76.54±0.02 |  76.44±0.05 |  70.83±0.49 |  69.58±0.05 |  77.17±0.02 |  69.69±0.15 |  69.68±0.14 |  67.31±0.03
C4 | C15 |  75.5±0.04 |  75.49±0.06 |  74.87±0.05 |  69.16±0.25 |  72.38±0.11 |  76.28±0.03 |  72.5±0.06 |  72.49±0.09 |  66.35±0.04
C6 | C7 |  73.64±0.03 |  73.68±0.05 |  72.83±0.06 |  65.62±0.21 |  68.42±0.17 |  73.76±0.06 |  68.52±0.04 |  68.45±0.11 |  61.75±0.03
C6 | C10 |  72.92±0.07 |  72.96±0.05 |  71.68±0.06 |  66.7±0.18 |  68.43±0.07 |  72.8±0.05 |  68.7±0.1 |  68.45±0.1 |  62.2±0.03
C7 | C8 |  74.61±0.03 |  74.6±0.03 |  73.15±0.05 |  66.03±0.21 |  68.93±0.13 |  74.13±0.04 |  69.11±0.11 |  68.96±0.19 |  61.57±0.04
C7 | C10 |  73.53±0.06 |  73.52±0.04 |  71.8±0.12 |  64.59±0.23 |  68.33±0.15 |  73.31±0.05 |  68.58±0.07 |  68.44±0.17 |  60.38±0.01
C7 | C12 |  76.8±0.05 |  76.83±0.03 |  76.56±0.02 |  69.82±0.14 |  69.3±0.12 |  77.27±0.03 |  69.47±0.17 |  69.47±0.17 |  66.58±0.02
C7 | C14 |  74.68±0.03 |  74.73±0.07 |  73.89±0.06 |  67.38±0.15 |  68.38±0.11 |  74.68±0.05 |  68.55±0.19 |  68.45±0.11 |  62.78±0.03
C7 | C15 |  76.84±0.03 |  76.88±0.03 |  76.5±0.02 |  69.04±0.03 |  69.01±0.12 |  77.23±0.03 |  69.17±0.08 |  65.36±8.59 |  66.34±0.04
C7 | C16 |  76.84±0.02 |  76.85±0.02 |  76.58±0.05 |  69.73±0.27 |  69.48±0.08 |  77.42±0.04 |  69.63±0.14 |  69.54±0.1 |  66.84±0.02
C7 | C18 |  76.52±0.03 |  76.53±0.06 |  76.15±0.01 |  68.92±0.17 |  68.74±0.12 |  76.92±0.03 |  68.97±0.11 |  65.1±8.44 |  65.84±0.02
C7 | C20 |  74.49±0.09 |  74.5±0.07 |  73.99±0.07 |  65.81±0.29 |  68.81±0.14 |  74.46±0.05 |  69.03±0.19 |  68.77±0.11 |  61.56±0.04
C7 | C21 |  76.84±0.04 |  76.85±0.03 |  76.6±0.07 |  69.73±0.3 |  69.44±0.08 |  77.37±0.04 |  69.52±0.1 |  69.51±0.07 |  66.78±0.05
C7 | C24 |  76.3±0.04 |  76.32±0.05 |  76.08±0.05 |  68.95±0.34 |  69.18±0.14 |  76.86±0.05 |  69.25±0.14 |  69.25±0.1 |  66.53±0.04
C7 | C26 |  75.42±0.03 |  75.45±0.05 |  75.45±0.05 |  68.87±0.13 |  68.78±0.14 |  75.9±0.02 |  68.92±0.13 |  68.85±0.09 |  65.59±0.03
C10 | C12 |  75.89±0.04 |  75.88±0.06 |  75.29±0.07 |  69.85±0.32 |  69.79±0.25 |  76.24±0.04 |  69.94±0.07 |  69.98±0.1 |  66.47±0.03
C10 | C14 |  73.61±0.06 |  73.64±0.04 |  72.55±0.05 |  66.89±0.12 |  68.37±0.12 |  73.46±0.08 |  68.62±0.09 |  68.54±0.14 |  62.89±0.04
C10 | C15 |  75.73±0.05 |  75.73±0.05 |  75.2±0.06 |  68.63±0.12 |  69.2±0.2 |  75.9±0.05 |  69.34±0.12 |  69.17±0.1 |  65.88±0.03
C10 | C16 |  76.0±0.03 |  76.0±0.01 |  75.58±0.04 |  69.98±0.35 |  70.03±0.12 |  76.45±0.03 |  70.08±0.12 |  70.13±0.12 |  66.74±0.03
C10 | C17 |  73.25±0.05 |  73.23±0.06 |  71.45±0.1 |  64.78±0.18 |  68.43±0.11 |  72.85±0.09 |  68.6±0.17 |  64.75±8.25 |  60.89±0.04
C10 | C18 |  75.27±0.03 |  75.26±0.05 |  74.66±0.03 |  68.14±0.55 |  68.54±0.05 |  75.48±0.05 |  68.71±0.04 |  68.61±0.08 |  65.4±0.04
C10 | C20 |  73.43±0.06 |  73.44±0.06 |  72.59±0.08 |  65.74±0.24 |  68.67±0.2 |  73.23±0.09 |  68.99±0.1 |  68.88±0.19 |  62.56±0.06
C10 | C21 |  75.98±0.03 |  75.99±0.02 |  75.49±0.07 |  69.7±0.31 |  69.89±0.11 |  76.4±0.05 |  70.03±0.1 |  69.98±0.08 |  66.62±0.03
C10 | C24 |  75.35±0.03 |  75.36±0.04 |  74.88±0.06 |  69.77±0.09 |  69.22±0.09 |  75.82±0.03 |  69.29±0.11 |  69.25±0.09 |  66.33±0.02
C10 | C26 |  74.72±0.03 |  74.76±0.05 |  74.53±0.08 |  69.34±0.1 |  68.71±0.03 |  75.11±0.03 |  68.98±0.1 |  68.78±0.06 |  65.52±0.04
C11 | C12 |  76.91±0.05 |  76.92±0.03 |  76.76±0.08 |  71.07±0.51 |  69.81±0.04 |  77.45±0.03 |  70.0±0.09 |  69.94±0.17 |  67.03±0.02
C11 | C15 |  76.81±0.04 |  76.84±0.02 |  76.48±0.02 |  69.81±0.19 |  68.93±0.09 |  77.13±0.02 |  69.14±0.12 |  69.13±0.08 |  66.46±0.01
C11 | C16 |  76.79±0.03 |  76.82±0.02 |  76.62±0.03 |  70.78±0.57 |  69.65±0.12 |  77.44±0.04 |  69.85±0.1 |  69.83±0.08 |  67.13±0.03
C11 | C18 |  76.5±0.04 |  76.53±0.06 |  76.2±0.03 |  70.01±0.33 |  68.39±0.17 |  76.82±0.06 |  68.6±0.06 |  68.57±0.07 |  66.1±0.03
C11 | C21 |  76.94±0.02 |  76.92±0.02 |  76.74±0.04 |  71.21±0.13 |  69.94±0.1 |  77.53±0.03 |  69.94±0.05 |  69.98±0.06 |  67.19±0.01
C11 | C24 |  76.27±0.01 |  76.23±0.05 |  76.08±0.08 |  70.74±0.2 |  69.43±0.08 |  76.84±0.04 |  69.61±0.09 |  69.48±0.1 |  66.9±0.01
C11 | C26 |  75.89±0.04 |  75.85±0.04 |  75.86±0.12 |  70.17±0.16 |  69.29±0.1 |  76.42±0.05 |  69.4±0.12 |  69.39±0.1 |  66.24±0.05
C12 | C13 |  76.77±0.04 |  76.77±0.06 |  76.56±0.05 |  71.51±0.26 |  69.76±0.12 |  77.33±0.05 |  69.89±0.1 |  69.95±0.1 |  66.99±0.05
C13 | C15 |  76.76±0.03 |  76.78±0.02 |  76.45±0.03 |  70.11±0.18 |  68.87±0.13 |  77.08±0.03 |  69.05±0.1 |  69.02±0.17 |  66.43±0.01
C13 | C16 |  76.66±0.04 |  76.66±0.03 |  76.46±0.04 |  70.96±0.67 |  69.67±0.12 |  77.28±0.03 |  69.77±0.1 |  69.81±0.06 |  67.03±0.04
C13 | C18 |  76.29±0.04 |  76.3±0.05 |  76.02±0.04 |  70.29±0.24 |  68.18±0.22 |  76.62±0.03 |  68.32±0.15 |  68.36±0.14 |  66.02±0.03
C13 | C21 |  76.79±0.04 |  76.79±0.01 |  76.56±0.06 |  71.36±0.49 |  69.73±0.13 |  77.39±0.04 |  69.93±0.07 |  69.91±0.07 |  67.12±0.03
C13 | C24 |  76.08±0.02 |  76.08±0.07 |  75.97±0.06 |  70.73±0.42 |  69.19±0.11 |  76.67±0.03 |  69.38±0.03 |  69.37±0.14 |  66.84±0.03
C13 | C26 |  75.93±0.02 |  75.93±0.04 |  75.93±0.05 |  70.55±0.1 |  69.43±0.14 |  76.49±0.05 |  69.58±0.06 |  69.48±0.2 |  66.3±0.03
Table 6: Bag size at 50, 70, 85, 95 %ile with tail size clusters Col1 | Col2 | Bag size: 50 %ile | Bag size: 70 %ile | Bag size: 85 %ile | Bag size: 95 %ile | Tail size cluster
---|---|---|---|---|---|---
C1 | C7 | 140 | 271 | 569 | 1270 | Long-tailed
C1 | C10 | 122 | 224 | 454 | 1068 | Short-tailed
C2 | C7 | 136 | 253 | 503 | 1102 | Short-tailed
C2 | C10 | 119 | 210 | 420 | 957 | Short-tailed
C2 | C11 | 157 | 311 | 637 | 1329 | Long-tailed
C2 | C13 | 165 | 334 | 688 | 1398 | Long-tailed
C3 | C7 | 103 | 173 | 334 | 804 | Very Short-tailed
C3 | C10 | 100 | 164 | 312 | 753 | Very Short-tailed
C3 | C11 | 105 | 179 | 351 | 853 | Very Short-tailed
C3 | C13 | 106 | 180 | 357 | 864 | Very Short-tailed
C4 | C7 | 100 | 161 | 296 | 685 | Very Short-tailed
C4 | C10 | 97 | 157 | 290 | 669 | Very Short-tailed
C4 | C11 | 104 | 174 | 333 | 780 | Very Short-tailed
C4 | C13 | 106 | 177 | 341 | 800 | Very Short-tailed
C4 | C15 | 114 | 203 | 417 | 1030 | Short-tailed
C6 | C7 | 222 | 443 | 854 | 1600 | Very Long-tailed
C6 | C10 | 144 | 271 | 567 | 1246 | Long-tailed
C7 | C8 | 153 | 303 | 646 | 1383 | Long-tailed
C7 | C10 | 151 | 296 | 613 | 1316 | Long-tailed
C7 | C12 | 103 | 173 | 329 | 787 | Very Short-tailed
C7 | C14 | 205 | 417 | 833 | 1577 | Very Long-tailed
C7 | C15 | 103 | 168 | 306 | 696 | Very Short-tailed
C7 | C16 | 102 | 168 | 311 | 727 | Very Short-tailed
C7 | C18 | 110 | 188 | 360 | 830 | Very Short-tailed
C7 | C20 | 261 | 523 | 977 | 1703 | Very Long-tailed
C7 | C21 | 103 | 171 | 321 | 761 | Very Short-tailed
C7 | C24 | 104 | 172 | 322 | 748 | Very Short-tailed
C7 | C26 | 106 | 181 | 364 | 887 | Very Short-tailed
C10 | C12 | 99 | 163 | 309 | 744 | Very Short-tailed
C10 | C14 | 141 | 272 | 557 | 1233 | Long-tailed
C10 | C15 | 101 | 164 | 303 | 705 | Very Short-tailed
C10 | C16 | 98 | 158 | 296 | 697 | Very Short-tailed
C10 | C17 | 145 | 280 | 581 | 1281 | Long-tailed
C10 | C18 | 105 | 176 | 338 | 786 | Very Short-tailed
C10 | C20 | 151 | 295 | 614 | 1342 | Long-tailed
C10 | C21 | 99 | 161 | 302 | 721 | Very Short-tailed
C10 | C24 | 100 | 165 | 310 | 728 | Very Short-tailed
C10 | C26 | 104 | 176 | 341 | 833 | Very Short-tailed
C11 | C12 | 106 | 181 | 356 | 857 | Very Short-tailed
C11 | C15 | 110 | 188 | 355 | 810 | Very Short-tailed
C11 | C16 | 106 | 180 | 349 | 830 | Very Short-tailed
C11 | C18 | 119 | 211 | 417 | 951 | Short-tailed
C11 | C21 | 106 | 181 | 353 | 852 | Very Short-tailed
C11 | C24 | 109 | 187 | 364 | 859 | Very Short-tailed
C11 | C26 | 106 | 181 | 356 | 884 | Very Short-tailed
C12 | C13 | 107 | 183 | 362 | 871 | Very Short-tailed
C13 | C15 | 113 | 195 | 374 | 868 | Very Short-tailed
C13 | C16 | 107 | 182 | 359 | 854 | Very Short-tailed
C13 | C18 | 122 | 220 | 436 | 997 | Short-tailed
C13 | C21 | 107 | 183 | 361 | 869 | Very Short-tailed
C13 | C24 | 111 | 191 | 373 | 880 | Very Short-tailed
C13 | C26 | 107 | 183 | 357 | 890 | Very Short-tailed
Table 7: Label Variation cluster and LabelPropStdDev values for LLP-Bench Col1 | Col2 | ${\sf LabelPropStdev}$ | Label Variation Cluster
---|---|---|---
C1 | C7 | 0.12 | Very Low
C1 | C10 | 0.1 | Very Low
C2 | C7 | 0.14 | Low
C2 | C10 | 0.13 | Low
C2 | C11 | 0.14 | Low
C2 | C13 | 0.14 | Low
C3 | C7 | 0.18 | High
C3 | C10 | 0.16 | Medium
C3 | C11 | 0.18 | High
C3 | C13 | 0.18 | High
C4 | C7 | 0.18 | High
C4 | C10 | 0.17 | Medium
C4 | C11 | 0.17 | High
C4 | C13 | 0.17 | High
C4 | C15 | 0.17 | Medium
C6 | C7 | 0.12 | Very Low
C6 | C10 | 0.1 | Very Low
C7 | C8 | 0.12 | Very Low
C7 | C10 | 0.11 | Very Low
C7 | C12 | 0.17 | High
C7 | C14 | 0.14 | Low
C7 | C15 | 0.17 | Medium
C7 | C16 | 0.17 | High
C7 | C18 | 0.16 | Medium
C7 | C20 | 0.12 | Very Low
C7 | C21 | 0.17 | High
C7 | C24 | 0.17 | Medium
C7 | C26 | 0.17 | Medium
C10 | C12 | 0.16 | Medium
C10 | C14 | 0.12 | Very Low
C10 | C15 | 0.15 | Medium
C10 | C16 | 0.16 | Medium
C10 | C17 | 0.11 | Very Low
C10 | C18 | 0.15 | Low
C10 | C20 | 0.1 | Very Low
C10 | C21 | 0.16 | Medium
C10 | C24 | 0.15 | Medium
C10 | C26 | 0.15 | Medium
C11 | C12 | 0.18 | High
C11 | C15 | 0.17 | Medium
C11 | C16 | 0.17 | High
C11 | C18 | 0.16 | Medium
C11 | C21 | 0.17 | High
C11 | C24 | 0.17 | Medium
C11 | C26 | 0.17 | Medium
C12 | C13 | 0.17 | High
C13 | C15 | 0.16 | Medium
C13 | C16 | 0.17 | High
C13 | C18 | 0.16 | Medium
C13 | C21 | 0.17 | High
C13 | C24 | 0.17 | Medium
C13 | C26 | 0.17 | Medium
Table 8: BagSep statistics along with the Bag Separation Cluster for each dataset in LLP-Bench Col1 | Col2 | MeanInterBagSep | MeanIntraBagSep | InterIntraRatio | Bag Separation cluster
---|---|---|---|---|---
C1 | C7 | 0.81 | 0.6 | 1.34 | Well-separated
C1 | C10 | 0.83 | 0.73 | 1.14 | Less-separated
C2 | C7 | 0.78 | 0.55 | 1.44 | Well-separated
C2 | C10 | 0.81 | 0.67 | 1.21 | Less-separated
C2 | C11 | 0.76 | 0.6 | 1.26 | Medium-separated
C2 | C13 | 0.75 | 0.61 | 1.22 | Medium-separated
C3 | C7 | 0.84 | 0.54 | 1.56 | Far-separated
C3 | C10 | 0.83 | 0.67 | 1.25 | Medium-separated
C3 | C11 | 0.84 | 0.62 | 1.35 | Well-separated
C3 | C13 | 0.84 | 0.63 | 1.32 | Well-separated
C4 | C7 | 0.83 | 0.53 | 1.55 | Far-separated
C4 | C10 | 0.83 | 0.66 | 1.26 | Medium-separated
C4 | C11 | 0.82 | 0.62 | 1.34 | Well-separated
C4 | C13 | 0.82 | 0.63 | 1.31 | Well-separated
C4 | C15 | 0.82 | 0.68 | 1.2 | Less-separated
C6 | C7 | 0.78 | 0.58 | 1.36 | Well-separated
C6 | C10 | 0.81 | 0.7 | 1.17 | Less-separated
C7 | C8 | 0.81 | 0.6 | 1.34 | Well-separated
C7 | C10 | 0.84 | 0.6 | 1.41 | Well-separated
C7 | C12 | 0.83 | 0.54 | 1.56 | Far-separated
C7 | C14 | 0.79 | 0.57 | 1.38 | Well-separated
C7 | C15 | 0.81 | 0.53 | 1.54 | Far-separated
C7 | C16 | 0.83 | 0.53 | 1.55 | Far-separated
C7 | C18 | 0.8 | 0.53 | 1.51 | Far-separated
C7 | C20 | 0.8 | 0.58 | 1.38 | Well-separated
C7 | C21 | 0.84 | 0.54 | 1.56 | Far-separated
C7 | C24 | 0.83 | 0.54 | 1.53 | Far-separated
C7 | C26 | 0.84 | 0.55 | 1.54 | Far-separated
C10 | C12 | 0.83 | 0.66 | 1.25 | Medium-separated
C10 | C14 | 0.82 | 0.7 | 1.17 | Less-separated
C10 | C15 | 0.82 | 0.65 | 1.25 | Medium-separated
C10 | C16 | 0.83 | 0.66 | 1.25 | Medium-separated
C10 | C17 | 0.83 | 0.68 | 1.23 | Medium-separated
C10 | C18 | 0.81 | 0.66 | 1.23 | Medium-separated
C10 | C20 | 0.83 | 0.7 | 1.18 | Less-separated
C10 | C21 | 0.83 | 0.66 | 1.25 | Medium-separated
C10 | C24 | 0.83 | 0.67 | 1.24 | Medium-separated
C10 | C26 | 0.84 | 0.67 | 1.26 | Medium-separated
C11 | C12 | 0.83 | 0.62 | 1.35 | Well-separated
C11 | C15 | 0.8 | 0.6 | 1.33 | Well-separated
C11 | C16 | 0.83 | 0.62 | 1.34 | Well-separated
C11 | C18 | 0.79 | 0.6 | 1.32 | Well-separated
C11 | C21 | 0.83 | 0.62 | 1.35 | Well-separated
C11 | C24 | 0.82 | 0.62 | 1.31 | Well-separated
C11 | C26 | 0.83 | 0.63 | 1.31 | Well-separated
C12 | C13 | 0.83 | 0.63 | 1.32 | Well-separated
C13 | C15 | 0.79 | 0.61 | 1.3 | Medium-separated
C13 | C16 | 0.83 | 0.63 | 1.32 | Well-separated
C13 | C18 | 0.78 | 0.61 | 1.28 | Medium-separated
C13 | C21 | 0.83 | 0.63 | 1.32 | Well-separated
C13 | C24 | 0.82 | 0.64 | 1.28 | Medium-separated
C13 | C26 | 0.83 | 0.64 | 1.29 | Medium-separated
Table 9: Test AUC scores of training baselines on fixed size feature bags. Each grouping-key $({\sf Col1},{\sf Col2})$ corresponds to $4$ contiguous rows, one for bag size $\\{64,128,256,512\\}$ each in ascending order. Col1 | Col2 | ${\sf DLLP\text{-}BCE}$ | ${\sf DLLP\text{-}MSE}$ | ${\sf GenBags}$ | ${\sf Easy\text{-}LLP}$ | ${\sf OT\text{-}LLP}$ | ${\sf SIM\text{-}LLP}$ | ${\sf Mean\text{-}Map}$
---|---|---|---|---|---|---|---|---
C1 | C7 |  77.6±0.03 |  77.63±0.02 |  77.31±0.03 |  64.01±0.05 |  72.32±0.03 |  76.54±0.1 |  62.2±0.03
C1 | C7 |  76.99±0.02 |  77.07±0.03 |  76.7±0.02 |  64.52±0.07 |  69.98±0.06 |  75.2±0.15 |  61.88±0.03
C1 | C7 |  76.27±0.02 |  76.42±0.03 |  75.91±0.03 |  64.82±0.07 |  68.41±0.16 |  74.24±0.15 |  61.67±0.03
C1 | C7 |  75.51±0.02 |  75.76±0.02 |  75.09±0.06 |  65.77±0.2 |  67.88±0.08 |  73.51±0.1 |  61.49±0.02
C1 | C10 |  77.56±0.02 |  77.57±0.01 |  77.23±0.04 |  60.39±0.04 |  73.3±0.07 |  76.36±0.13 |  61.59±0.0
C1 | C10 |  76.9±0.03 |  76.96±0.01 |  76.59±0.02 |  61.02±0.13 |  70.83±0.1 |  74.56±0.15 |  61.07±0.01
C1 | C10 |  76.13±0.04 |  76.25±0.02 |  75.73±0.05 |  61.99±0.1 |  68.67±0.06 |  72.86±0.27 |  60.66±0.01
C1 | C10 |  75.16±0.03 |  75.38±0.04 |  74.65±0.04 |  63.47±0.41 |  67.8±0.18 |  71.6±0.15 |  60.32±0.01
C2 | C7 |  77.37±0.03 |  77.41±0.03 |  77.24±0.05 |  66.49±0.05 |  70.52±0.13 |  77.12±0.06 |  64.47±0.02
C2 | C7 |  76.77±0.01 |  76.82±0.03 |  76.51±0.03 |  66.59±0.03 |  68.67±0.06 |  76.48±0.03 |  64.31±0.02
C2 | C7 |  76.27±0.03 |  76.33±0.01 |  75.97±0.04 |  66.41±0.03 |  67.72±0.04 |  75.97±0.03 |  64.16±0.02
C2 | C7 |  75.94±0.02 |  75.99±0.02 |  75.6±0.04 |  67.04±0.15 |  67.74±0.07 |  75.51±0.05 |  63.95±0.03
C2 | C10 |  77.33±0.02 |  77.33±0.03 |  77.12±0.04 |  64.25±0.11 |  71.57±0.09 |  76.67±0.08 |  64.07±0.02
C2 | C10 |  76.67±0.02 |  76.67±0.02 |  76.36±0.01 |  64.32±0.1 |  69.18±0.1 |  75.73±0.16 |  63.91±0.03
C2 | C10 |  76.01±0.02 |  76.02±0.04 |  75.59±0.05 |  65.05±0.06 |  68.01±0.09 |  74.93±0.06 |  63.73±0.02
C2 | C10 |  75.49±0.03 |  75.5±0.03 |  75.01±0.04 |  66.66±0.16 |  68.17±0.13 |  74.3±0.05 |  63.54±0.02
C2 | C11 |  77.36±0.03 |  77.41±0.03 |  77.26±0.04 |  65.21±0.05 |  70.99±0.06 |  76.91±0.05 |  64.71±0.03
C2 | C11 |  76.69±0.03 |  76.78±0.03 |  76.47±0.03 |  65.56±0.07 |  68.83±0.13 |  76.23±0.03 |  64.58±0.02
C2 | C11 |  76.14±0.02 |  76.22±0.04 |  75.85±0.04 |  66.25±0.06 |  67.85±0.16 |  75.64±0.02 |  64.51±0.02
C2 | C11 |  75.78±0.03 |  75.84±0.02 |  75.39±0.03 |  67.32±0.24 |  67.79±0.08 |  75.32±0.07 |  64.34±0.03
C2 | C13 |  77.38±0.03 |  77.41±0.04 |  77.24±0.05 |  65.06±0.04 |  70.97±0.05 |  76.83±0.03 |  64.76±0.02
C2 | C13 |  76.7±0.03 |  76.77±0.05 |  76.5±0.03 |  65.44±0.08 |  68.76±0.11 |  76.17±0.04 |  64.67±0.02
C2 | C13 |  76.14±0.04 |  76.21±0.03 |  75.84±0.03 |  66.31±0.02 |  67.78±0.05 |  75.6±0.04 |  64.59±0.02
C2 | C13 |  75.75±0.04 |  75.8±0.03 |  75.36±0.06 |  67.45±0.09 |  67.7±0.08 |  75.26±0.05 |  64.45±0.02
C3 | C7 |  77.64±0.04 |  77.66±0.02 |  77.58±0.04 |  67.51±0.11 |  70.38±0.1 |  77.47±0.02 |  64.98±0.03
C3 | C7 |  77.16±0.01 |  77.2±0.03 |  76.88±0.02 |  67.41±0.01 |  68.75±0.04 |  76.93±0.02 |  64.65±0.02
C3 | C7 |  76.74±0.01 |  76.77±0.03 |  76.31±0.02 |  67.11±0.09 |  68.02±0.07 |  76.33±0.02 |  64.3±0.03
C3 | C7 |  76.33±0.02 |  76.39±0.03 |  75.84±0.03 |  67.83±0.09 |  68.02±0.06 |  75.7±0.03 |  63.89±0.02
C3 | C10 |  77.53±0.02 |  77.55±0.04 |  77.51±0.03 |  67.23±0.08 |  71.61±0.06 |  77.19±0.02 |  65.02±0.03
C3 | C10 |  76.98±0.02 |  76.97±0.04 |  76.69±0.04 |  67.7±0.08 |  69.57±0.1 |  76.29±0.06 |  64.7±0.02
C3 | C10 |  76.39±0.03 |  76.38±0.03 |  75.94±0.04 |  67.41±0.14 |  68.71±0.08 |  75.65±0.08 |  64.39±0.02
C3 | C10 |  75.86±0.03 |  75.82±0.04 |  75.26±0.04 |  68.1±0.21 |  68.63±0.14 |  74.85±0.15 |  64.04±0.03
C3 | C11 |  77.67±0.03 |  77.72±0.05 |  77.68±0.05 |  67.42±0.05 |  70.76±0.15 |  77.4±0.04 |  65.29±0.03
C3 | C11 |  77.17±0.05 |  77.18±0.04 |  76.91±0.04 |  68.15±0.06 |  69.16±0.09 |  76.91±0.06 |  65.05±0.03
C3 | C11 |  76.7±0.03 |  76.75±0.03 |  76.27±0.03 |  67.8±0.04 |  68.29±0.09 |  76.37±0.05 |  64.75±0.02
C3 | C11 |  76.34±0.03 |  76.41±0.02 |  75.82±0.02 |  68.23±0.44 |  68.53±0.09 |  75.75±0.07 |  64.38±0.02
C3 | C13 |  77.65±0.03 |  77.66±0.04 |  77.65±0.08 |  67.37±0.11 |  70.85±0.08 |  77.46±0.04 |  65.42±0.03
C3 | C13 |  77.18±0.04 |  77.22±0.05 |  76.94±0.02 |  68.16±0.14 |  69.08±0.08 |  76.87±0.07 |  65.18±0.02
C3 | C13 |  76.74±0.04 |  76.77±0.04 |  76.32±0.05 |  67.98±0.08 |  68.25±0.06 |  76.37±0.04 |  64.89±0.02
C3 | C13 |  76.32±0.02 |  76.41±0.05 |  75.84±0.06 |  68.48±0.18 |  68.38±0.07 |  75.74±0.05 |  64.57±0.02
C4 | C7 |  77.66±0.04 |  77.68±0.02 |  77.68±0.05 |  68.38±0.04 |  70.09±0.21 |  77.7±0.03 |  65.45±0.02
C4 | C7 |  77.14±0.04 |  77.18±0.05 |  76.93±0.06 |  68.38±0.08 |  68.74±0.1 |  77.24±0.02 |  65.13±0.03
C4 | C7 |  76.78±0.02 |  76.8±0.03 |  76.32±0.06 |  68.26±0.05 |  68.08±0.04 |  76.76±0.01 |  64.8±0.01
C4 | C7 |  76.49±0.04 |  76.52±0.05 |  75.96±0.05 |  68.73±0.45 |  68.16±0.08 |  76.2±0.06 |  64.41±0.02
C4 | C10 |  77.55±0.03 |  77.56±0.04 |  77.52±0.06 |  67.59±0.08 |  71.28±0.03 |  77.39±0.04 |  65.27±0.02
C4 | C10 |  76.97±0.02 |  76.96±0.03 |  76.71±0.03 |  67.78±0.07 |  69.51±0.06 |  76.7±0.08 |  64.97±0.02
C4 | C10 |  76.44±0.02 |  76.41±0.04 |  75.96±0.05 |  67.84±0.08 |  68.85±0.1 |  75.98±0.06 |  64.68±0.03
C4 | C10 |  75.98±0.04 |  75.96±0.06 |  75.29±0.08 |  68.59±0.13 |  68.78±0.15 |  75.25±0.06 |  64.34±0.02
C4 | C11 |  77.64±0.03 |  77.71±0.02 |  77.7±0.05 |  67.86±0.07 |  70.44±0.14 |  77.69±0.01 |  65.69±0.02
C4 | C11 |  77.13±0.04 |  77.15±0.04 |  76.94±0.05 |  68.13±0.13 |  69.0±0.09 |  77.25±0.02 |  65.43±0.03
C4 | C11 |  76.73±0.01 |  76.77±0.01 |  76.31±0.02 |  68.43±0.1 |  68.39±0.06 |  76.82±0.08 |  65.13±0.01
C4 | C11 |  76.49±0.01 |  76.5±0.02 |  75.9±0.06 |  68.96±0.4 |  68.6±0.11 |  76.3±0.03 |  64.79±0.02
C4 | C13 |  77.63±0.03 |  77.66±0.07 |  77.73±0.02 |  67.79±0.07 |  70.48±0.13 |  77.67±0.05 |  65.81±0.02
C4 | C13 |  77.11±0.04 |  77.15±0.04 |  76.92±0.05 |  68.24±0.07 |  68.96±0.03 |  77.24±0.05 |  65.55±0.02
C4 | C13 |  76.75±0.03 |  76.76±0.03 |  76.29±0.05 |  68.55±0.04 |  68.35±0.09 |  76.82±0.03 |  65.26±0.01
C4 | C13 |  76.48±0.03 |  76.49±0.02 |  75.91±0.03 |  69.19±0.34 |  68.52±0.09 |  76.24±0.06 |  64.93±0.02
C4 | C15 |  77.35±0.05 |  77.36±0.11 |  77.62±0.08 |  66.37±0.07 |  72.34±0.09 |  77.3±0.03 |  65.49±0.02
C4 | C15 |  76.31±0.07 |  76.35±0.09 |  76.29±0.19 |  66.24±0.07 |  70.84±0.07 |  76.57±0.07 |  65.28±0.02
C4 | C15 |  75.4±0.04 |  75.46±0.05 |  74.93±0.07 |  66.01±0.04 |  70.04±0.07 |  75.8±0.04 |  65.04±0.02
C4 | C15 |  74.85±0.04 |  74.8±0.06 |  74.1±0.1 |  67.04±0.15 |  69.77±0.11 |  75.04±0.02 |  64.81±0.02
C6 | C7 |  77.25±0.16 |  77.4±0.03 |  77.2±0.03 |  64.4±0.04 |  72.28±0.11 |  76.74±0.08 |  62.59±0.03
C6 | C7 |  76.45±0.09 |  76.43±0.06 |  76.17±0.09 |  64.88±0.06 |  70.06±0.09 |  75.61±0.11 |  62.31±0.03
C6 | C7 |  75.5±0.07 |  75.68±0.05 |  75.17±0.04 |  65.08±0.08 |  68.57±0.08 |  74.84±0.06 |  62.1±0.02
C6 | C7 |  74.82±0.05 |  74.91±0.06 |  74.44±0.05 |  65.93±0.42 |  67.96±0.06 |  74.24±0.09 |  61.92±0.02
C6 | C10 |  77.33±0.04 |  77.32±0.02 |  77.1±0.09 |  60.69±0.07 |  73.23±0.04 |  76.41±0.09 |  62.15±0.02
C6 | C10 |  76.6±0.04 |  76.67±0.07 |  76.3±0.03 |  61.48±0.06 |  70.8±0.11 |  74.84±0.13 |  61.74±0.01
C6 | C10 |  75.75±0.04 |  75.86±0.05 |  75.37±0.05 |  62.64±0.14 |  68.97±0.07 |  73.22±0.07 |  61.41±0.02
C6 | C10 |  74.88±0.04 |  75.06±0.04 |  74.46±0.05 |  64.37±0.69 |  67.8±0.15 |  72.34±0.11 |  61.12±0.01
C7 | C8 |  77.59±0.01 |  77.62±0.01 |  77.31±0.04 |  64.03±0.05 |  72.3±0.04 |  76.41±0.16 |  62.2±0.03
C7 | C8 |  77.0±0.02 |  77.07±0.03 |  76.7±0.03 |  64.45±0.07 |  70.0±0.05 |  75.17±0.17 |  61.91±0.04
C7 | C8 |  76.27±0.02 |  76.41±0.05 |  75.9±0.04 |  64.77±0.05 |  68.45±0.08 |  74.22±0.11 |  61.7±0.03
C7 | C8 |  75.53±0.03 |  75.73±0.05 |  75.07±0.05 |  65.84±0.13 |  67.96±0.09 |  73.55±0.04 |  61.53±0.03
C7 | C10 |  77.44±0.02 |  77.43±0.02 |  77.08±0.04 |  64.61±0.03 |  72.39±0.04 |  76.66±0.09 |  62.36±0.03
C7 | C10 |  76.7±0.02 |  76.75±0.02 |  76.34±0.03 |  64.77±0.09 |  70.31±0.15 |  75.58±0.08 |  62.07±0.03
C7 | C10 |  75.87±0.02 |  75.96±0.02 |  75.43±0.04 |  65.07±0.06 |  68.72±0.12 |  74.68±0.12 |  61.88±0.02
C7 | C10 |  75.03±0.04 |  75.17±0.04 |  74.54±0.03 |  66.17±0.24 |  68.16±0.09 |  74.12±0.04 |  61.72±0.03
C7 | C12 |  77.65±0.04 |  77.68±0.05 |  77.62±0.04 |  67.55±0.07 |  70.33±0.16 |  77.51±0.04 |  65.0±0.02
C7 | C12 |  77.16±0.02 |  77.19±0.03 |  76.89±0.02 |  67.42±0.08 |  68.68±0.19 |  77.03±0.04 |  64.68±0.03
C7 | C12 |  76.73±0.03 |  76.77±0.02 |  76.34±0.03 |  67.11±0.04 |  67.96±0.05 |  76.35±0.06 |  64.34±0.02
C7 | C12 |  76.37±0.02 |  76.42±0.04 |  75.93±0.05 |  67.76±0.24 |  68.09±0.11 |  75.74±0.05 |  63.92±0.02
C7 | C14 |  77.29±0.03 |  77.33±0.03 |  77.08±0.07 |  65.72±0.09 |  71.38±0.06 |  76.57±0.04 |  63.34±0.03
C7 | C14 |  76.57±0.03 |  76.61±0.03 |  76.26±0.05 |  66.01±0.06 |  69.13±0.08 |  75.69±0.07 |  63.13±0.01
C7 | C14 |  75.81±0.03 |  75.91±0.03 |  75.5±0.03 |  66.01±0.07 |  67.9±0.12 |  75.03±0.08 |  62.95±0.01
C7 | C14 |  75.22±0.06 |  75.28±0.06 |  74.87±0.06 |  66.8±0.32 |  67.38±0.17 |  74.56±0.09 |  62.8±0.01
C7 | C15 |  77.4±0.04 |  77.47±0.05 |  77.32±0.04 |  68.3±0.06 |  69.35±0.25 |  77.5±0.04 |  65.28±0.01
C7 | C15 |  76.91±0.03 |  76.95±0.04 |  76.61±0.06 |  68.06±0.07 |  68.34±0.1 |  77.0±0.03 |  65.02±0.02
C7 | C15 |  76.59±0.03 |  76.62±0.03 |  76.18±0.05 |  67.81±0.06 |  67.87±0.09 |  76.58±0.06 |  64.72±0.02
C7 | C15 |  76.4±0.01 |  76.41±0.01 |  75.99±0.03 |  68.66±0.19 |  68.11±0.12 |  76.16±0.04 |  64.34±0.02
C7 | C16 |  77.66±0.04 |  77.68±0.02 |  77.59±0.06 |  68.1±0.05 |  70.15±0.2 |  77.63±0.04 |  65.33±0.03
C7 | C16 |  77.16±0.03 |  77.19±0.04 |  76.91±0.03 |  67.92±0.07 |  68.74±0.14 |  77.14±0.03 |  65.01±0.02
C7 | C16 |  76.73±0.04 |  76.77±0.05 |  76.31±0.03 |  67.54±0.07 |  67.98±0.04 |  76.56±0.04 |  64.65±0.02
C7 | C16 |  76.39±0.02 |  76.43±0.01 |  75.88±0.05 |  68.26±0.2 |  68.09±0.09 |  75.97±0.08 |  64.25±0.01
C7 | C18 |  77.37±0.03 |  77.43±0.02 |  77.26±0.02 |  67.67±0.07 |  69.69±0.14 |  77.37±0.03 |  65.11±0.04
C7 | C18 |  76.85±0.02 |  76.89±0.03 |  76.56±0.04 |  67.6±0.06 |  68.38±0.11 |  76.89±0.05 |  64.87±0.03
C7 | C18 |  76.46±0.01 |  76.48±0.01 |  76.06±0.04 |  67.29±0.04 |  67.68±0.05 |  76.46±0.02 |  64.64±0.02
C7 | C18 |  76.22±0.01 |  76.23±0.02 |  75.74±0.01 |  68.13±0.09 |  67.76±0.12 |  76.09±0.03 |  64.31±0.03
C7 | C20 |  77.47±0.01 |  77.51±0.03 |  77.23±0.03 |  65.19±0.08 |  71.72±0.08 |  76.78±0.04 |  63.63±0.03
C7 | C20 |  76.86±0.02 |  76.91±0.03 |  76.6±0.02 |  65.43±0.03 |  69.32±0.07 |  76.01±0.08 |  63.44±0.03
C7 | C20 |  76.25±0.04 |  76.33±0.03 |  75.93±0.05 |  65.67±0.06 |  68.05±0.08 |  75.38±0.08 |  63.31±0.02
C7 | C20 |  75.7±0.04 |  75.82±0.02 |  75.37±0.06 |  66.17±0.35 |  67.66±0.18 |  74.93±0.12 |  63.19±0.02
C7 | C21 |  77.66±0.05 |  77.68±0.04 |  77.64±0.04 |  67.81±0.08 |  70.25±0.13 |  77.57±0.03 |  65.23±0.03
C7 | C21 |  77.12±0.02 |  77.19±0.03 |  76.9±0.05 |  67.7±0.04 |  68.77±0.11 |  77.01±0.01 |  64.93±0.02
C7 | C21 |  76.72±0.02 |  76.77±0.04 |  76.31±0.04 |  67.37±0.05 |  68.0±0.03 |  76.49±0.07 |  64.59±0.02
C7 | C21 |  76.38±0.03 |  76.41±0.04 |  75.89±0.04 |  68.4±0.15 |  68.09±0.08 |  75.86±0.04 |  64.16±0.02
C7 | C24 |  77.46±0.04 |  77.51±0.04 |  77.5±0.1 |  68.02±0.09 |  70.27±0.15 |  77.54±0.02 |  65.16±0.03
C7 | C24 |  76.93±0.02 |  76.99±0.03 |  76.69±0.02 |  68.03±0.1 |  68.72±0.09 |  77.12±0.04 |  64.88±0.03
C7 | C24 |  76.56±0.04 |  76.58±0.02 |  76.11±0.03 |  68.03±0.04 |  67.91±0.05 |  76.64±0.05 |  64.56±0.02
C7 | C24 |  76.29±0.04 |  76.31±0.03 |  75.73±0.06 |  68.45±0.35 |  67.96±0.08 |  76.18±0.07 |  64.22±0.02
C7 | C26 |  77.52±0.01 |  77.52±0.03 |  77.36±0.06 |  66.8±0.06 |  70.73±0.17 |  77.36±0.02 |  64.93±0.02
C7 | C26 |  76.98±0.02 |  77.01±0.02 |  76.68±0.03 |  66.89±0.05 |  69.02±0.04 |  76.8±0.02 |  64.64±0.02
C7 | C26 |  76.53±0.03 |  76.57±0.03 |  76.12±0.05 |  66.97±0.06 |  68.23±0.1 |  76.34±0.04 |  64.36±0.02
C7 | C26 |  76.15±0.01 |  76.2±0.01 |  75.66±0.02 |  67.49±0.13 |  68.38±0.12 |  75.86±0.05 |  64.05±0.02
C10 | C12 |  77.57±0.05 |  77.58±0.04 |  77.53±0.04 |  67.3±0.03 |  71.53±0.05 |  77.22±0.05 |  64.99±0.04
C10 | C12 |  76.96±0.04 |  76.97±0.03 |  76.69±0.05 |  67.7±0.09 |  69.53±0.05 |  76.44±0.07 |  64.67±0.02
C10 | C12 |  76.4±0.06 |  76.38±0.06 |  75.95±0.08 |  67.3±0.08 |  68.71±0.1 |  75.66±0.1 |  64.36±0.01
C10 | C12 |  75.89±0.03 |  75.85±0.02 |  75.25±0.03 |  67.81±0.16 |  68.66±0.08 |  74.95±0.02 |  64.0±0.02
C10 | C14 |  77.32±0.04 |  77.32±0.06 |  77.02±0.05 |  61.88±0.07 |  72.64±0.06 |  76.28±0.09 |  62.85±0.02
C10 | C14 |  76.58±0.03 |  76.61±0.03 |  76.23±0.06 |  62.93±0.11 |  70.2±0.1 |  74.91±0.08 |  62.58±0.02
C10 | C14 |  75.67±0.02 |  75.72±0.05 |  75.27±0.03 |  64.4±0.12 |  68.61±0.11 |  73.67±0.08 |  62.35±0.02
C10 | C14 |  74.79±0.05 |  74.86±0.02 |  74.33±0.04 |  65.61±0.27 |  68.22±0.17 |  73.02±0.05 |  62.13±0.02
C10 | C15 |  77.35±0.04 |  77.37±0.03 |  77.16±0.03 |  66.69±0.13 |  70.76±0.1 |  77.06±0.05 |  65.07±0.03
C10 | C15 |  76.81±0.02 |  76.77±0.02 |  76.44±0.05 |  66.68±0.11 |  69.06±0.09 |  76.44±0.04 |  64.87±0.02
C10 | C15 |  76.4±0.01 |  76.34±0.02 |  75.87±0.07 |  67.26±0.07 |  68.52±0.03 |  75.86±0.04 |  64.62±0.02
C10 | C15 |  76.08±0.02 |  76.07±0.06 |  75.51±0.07 |  68.49±0.26 |  68.72±0.1 |  75.45±0.05 |  64.34±0.02
C10 | C16 |  77.57±0.04 |  77.55±0.07 |  77.52±0.05 |  67.53±0.05 |  71.42±0.13 |  77.27±0.05 |  65.14±0.03
C10 | C16 |  77.0±0.05 |  77.0±0.03 |  76.74±0.05 |  67.71±0.06 |  69.48±0.08 |  76.54±0.08 |  64.83±0.03
C10 | C16 |  76.46±0.02 |  76.46±0.03 |  76.0±0.03 |  67.24±0.11 |  68.74±0.17 |  75.83±0.07 |  64.51±0.02
C10 | C16 |  75.99±0.04 |  75.93±0.04 |  75.34±0.04 |  67.9±0.14 |  68.81±0.09 |  75.13±0.04 |  64.14±0.02
C10 | C17 |  77.44±0.03 |  77.46±0.03 |  77.13±0.05 |  62.58±0.07 |  72.84±0.07 |  76.33±0.07 |  61.83±0.03
C10 | C17 |  76.76±0.02 |  76.82±0.02 |  76.44±0.03 |  62.22±0.03 |  70.55±0.06 |  74.96±0.12 |  61.47±0.03
C10 | C17 |  75.95±0.03 |  76.06±0.04 |  75.58±0.04 |  62.95±0.08 |  68.62±0.12 |  73.71±0.15 |  61.22±0.03
C10 | C17 |  75.1±0.03 |  75.22±0.04 |  74.68±0.07 |  64.71±0.38 |  67.83±0.16 |  72.85±0.16 |  61.04±0.03
C10 | C18 |  77.33±0.03 |  77.31±0.03 |  77.19±0.05 |  65.77±0.05 |  70.81±0.07 |  76.96±0.05 |  64.89±0.02
C10 | C18 |  76.71±0.03 |  76.67±0.03 |  76.37±0.05 |  66.01±0.13 |  68.96±0.09 |  76.25±0.06 |  64.7±0.02
C10 | C18 |  76.18±0.02 |  76.12±0.03 |  75.68±0.05 |  66.7±0.05 |  68.19±0.1 |  75.68±0.04 |  64.5±0.02
C10 | C18 |  75.82±0.05 |  75.78±0.03 |  75.19±0.04 |  68.07±0.23 |  68.28±0.12 |  75.32±0.06 |  64.26±0.03
C10 | C20 |  77.47±0.03 |  77.47±0.02 |  77.17±0.02 |  61.01±0.05 |  72.91±0.12 |  76.4±0.11 |  62.84±0.02
C10 | C20 |  76.79±0.02 |  76.82±0.01 |  76.47±0.03 |  61.87±0.08 |  70.54±0.13 |  74.97±0.06 |  62.62±0.01
C10 | C20 |  76.07±0.03 |  76.15±0.03 |  75.68±0.05 |  62.99±0.15 |  68.73±0.09 |  73.58±0.14 |  62.43±0.01
C10 | C20 |  75.31±0.04 |  75.43±0.04 |  74.84±0.03 |  64.01±0.52 |  67.97±0.14 |  72.77±0.14 |  62.24±0.01
C10 | C21 |  77.56±0.04 |  77.56±0.03 |  77.54±0.05 |  67.47±0.07 |  71.45±0.05 |  77.27±0.04 |  65.15±0.03
C10 | C21 |  76.99±0.02 |  76.98±0.02 |  76.71±0.04 |  67.81±0.09 |  69.49±0.15 |  76.49±0.04 |  64.84±0.02
C10 | C21 |  76.41±0.04 |  76.42±0.03 |  76.01±0.06 |  67.36±0.1 |  68.85±0.07 |  75.79±0.06 |  64.53±0.02
C10 | C21 |  75.97±0.04 |  75.94±0.04 |  75.38±0.07 |  68.03±0.23 |  68.87±0.15 |  75.02±0.06 |  64.18±0.02
C10 | C24 |  77.38±0.06 |  77.41±0.03 |  77.36±0.04 |  66.78±0.1 |  71.36±0.13 |  77.19±0.05 |  64.91±0.02
C10 | C24 |  76.78±0.04 |  76.79±0.03 |  76.5±0.04 |  67.16±0.05 |  69.33±0.06 |  76.46±0.09 |  64.62±0.02
C10 | C24 |  76.19±0.02 |  76.17±0.02 |  75.7±0.03 |  67.66±0.12 |  68.45±0.11 |  75.77±0.05 |  64.37±0.02
C10 | C24 |  75.68±0.05 |  75.67±0.06 |  75.02±0.05 |  68.6±0.29 |  68.44±0.07 |  75.07±0.05 |  64.09±0.02
C10 | C26 |  77.46±0.05 |  77.45±0.03 |  77.26±0.06 |  64.57±0.24 |  71.67±0.06 |  76.98±0.05 |  64.72±0.04
C10 | C26 |  76.88±0.02 |  76.87±0.03 |  76.55±0.04 |  66.21±0.17 |  69.68±0.1 |  76.2±0.08 |  64.45±0.01
C10 | C26 |  76.29±0.01 |  76.31±0.04 |  75.82±0.03 |  67.17±0.13 |  68.68±0.08 |  75.47±0.13 |  64.21±0.01
C10 | C26 |  75.79±0.03 |  75.8±0.03 |  75.17±0.07 |  67.91±0.49 |  68.72±0.14 |  74.96±0.04 |  63.97±0.02
C11 | C12 |  77.66±0.05 |  77.7±0.03 |  77.67±0.06 |  67.49±0.12 |  70.76±0.1 |  77.52±0.03 |  65.33±0.03
C11 | C12 |  77.17±0.01 |  77.2±0.04 |  76.95±0.03 |  68.1±0.08 |  68.99±0.11 |  76.95±0.03 |  65.06±0.02
C11 | C12 |  76.74±0.03 |  76.79±0.03 |  76.33±0.03 |  67.77±0.06 |  68.24±0.05 |  76.43±0.04 |  64.75±0.02
C11 | C12 |  76.39±0.02 |  76.44±0.01 |  75.87±0.02 |  68.31±0.31 |  68.45±0.13 |  75.82±0.03 |  64.39±0.01
C11 | C15 |  77.34±0.07 |  77.4±0.03 |  77.33±0.12 |  67.55±0.05 |  69.95±0.11 |  77.31±0.06 |  65.5±0.01
C11 | C15 |  76.8±0.01 |  76.84±0.03 |  76.49±0.03 |  67.66±0.06 |  68.55±0.15 |  76.79±0.03 |  65.33±0.02
C11 | C15 |  76.53±0.01 |  76.51±0.02 |  76.05±0.04 |  67.91±0.05 |  68.01±0.05 |  76.38±0.02 |  65.08±0.01
C11 | C15 |  76.3±0.02 |  76.32±0.02 |  75.8±0.03 |  68.68±0.36 |  68.34±0.08 |  76.03±0.01 |  64.76±0.02
C11 | C16 |  77.65±0.03 |  77.73±0.02 |  77.69±0.04 |  67.92±0.1 |  70.63±0.1 |  77.62±0.03 |  65.56±0.03
C11 | C16 |  77.16±0.03 |  77.2±0.03 |  76.97±0.03 |  68.29±0.04 |  69.06±0.06 |  77.12±0.04 |  65.33±0.01
C11 | C16 |  76.72±0.03 |  76.75±0.03 |  76.32±0.04 |  68.1±0.04 |  68.34±0.11 |  76.59±0.03 |  65.01±0.02
C11 | C16 |  76.4±0.04 |  76.44±0.03 |  75.86±0.03 |  68.83±0.3 |  68.47±0.12 |  76.1±0.03 |  64.64±0.01
C11 | C18 |  77.36±0.04 |  77.42±0.03 |  77.31±0.05 |  66.63±0.03 |  70.05±0.09 |  77.19±0.03 |  65.34±0.02
C11 | C18 |  76.75±0.02 |  76.81±0.03 |  76.52±0.03 |  67.01±0.05 |  68.62±0.05 |  76.69±0.03 |  65.18±0.02
C11 | C18 |  76.35±0.03 |  76.38±0.03 |  75.95±0.04 |  67.52±0.03 |  67.87±0.06 |  76.3±0.03 |  65.01±0.02
C11 | C18 |  76.11±0.02 |  76.16±0.02 |  75.65±0.03 |  68.39±0.22 |  68.07±0.09 |  75.97±0.04 |  64.76±0.03
C11 | C21 |  77.66±0.06 |  77.71±0.03 |  77.68±0.03 |  67.63±0.12 |  70.67±0.17 |  77.54±0.05 |  65.5±0.04
C11 | C21 |  77.15±0.04 |  77.18±0.03 |  76.91±0.04 |  68.2±0.04 |  69.04±0.08 |  77.06±0.04 |  65.25±0.02
C11 | C21 |  76.72±0.03 |  76.78±0.04 |  76.33±0.04 |  67.98±0.05 |  68.34±0.09 |  76.55±0.04 |  64.96±0.02
C11 | C21 |  76.37±0.03 |  76.45±0.03 |  75.86±0.06 |  68.8±0.05 |  68.53±0.12 |  76.02±0.05 |  64.6±0.02
C11 | C24 |  77.46±0.06 |  77.5±0.03 |  77.51±0.06 |  67.23±0.16 |  70.64±0.13 |  77.48±0.03 |  65.43±0.02
C11 | C24 |  76.89±0.02 |  76.94±0.03 |  76.68±0.06 |  67.55±0.07 |  68.98±0.07 |  77.04±0.04 |  65.17±0.01
C11 | C24 |  76.47±0.03 |  76.52±0.03 |  76.02±0.05 |  68.28±0.13 |  68.14±0.1 |  76.64±0.05 |  64.91±0.02
C11 | C24 |  76.23±0.04 |  76.24±0.04 |  75.65±0.02 |  69.08±0.44 |  68.18±0.06 |  76.22±0.06 |  64.58±0.03
C11 | C26 |  77.48±0.03 |  77.51±0.04 |  77.37±0.04 |  66.31±0.08 |  71.01±0.08 |  77.24±0.02 |  65.13±0.03
C11 | C26 |  76.92±0.02 |  76.95±0.02 |  76.64±0.02 |  66.89±0.04 |  69.1±0.1 |  76.7±0.05 |  64.88±0.02
C11 | C26 |  76.43±0.02 |  76.47±0.03 |  76.01±0.03 |  67.41±0.13 |  68.32±0.03 |  76.23±0.04 |  64.64±0.01
C11 | C26 |  76.07±0.04 |  76.07±0.03 |  75.52±0.03 |  67.98±0.13 |  68.54±0.04 |  75.83±0.04 |  64.35±0.01
C12 | C13 |  77.65±0.03 |  77.7±0.05 |  77.65±0.06 |  67.51±0.08 |  70.65±0.18 |  77.48±0.04 |  65.45±0.04
C12 | C13 |  77.17±0.04 |  77.2±0.03 |  76.96±0.03 |  68.17±0.06 |  68.97±0.08 |  76.93±0.07 |  65.2±0.02
C12 | C13 |  76.76±0.03 |  76.78±0.03 |  76.34±0.05 |  67.89±0.08 |  68.26±0.08 |  76.45±0.08 |  64.92±0.03
C12 | C13 |  76.37±0.03 |  76.44±0.01 |  75.88±0.05 |  68.31±0.43 |  68.29±0.12 |  75.85±0.08 |  64.57±0.02
C13 | C15 |  77.3±0.04 |  77.34±0.05 |  77.28±0.06 |  67.41±0.04 |  69.8±0.07 |  77.22±0.06 |  65.59±0.03
C13 | C15 |  76.77±0.02 |  76.78±0.03 |  76.48±0.03 |  67.6±0.07 |  68.45±0.08 |  76.72±0.06 |  65.42±0.03
C13 | C15 |  76.47±0.04 |  76.46±0.01 |  76.01±0.03 |  67.91±0.08 |  68.05±0.07 |  76.31±0.06 |  65.21±0.02
C13 | C15 |  76.3±0.03 |  76.29±0.05 |  75.82±0.06 |  68.83±0.08 |  68.38±0.05 |  76.03±0.05 |  64.92±0.02
C13 | C16 |  77.65±0.03 |  77.71±0.03 |  77.69±0.06 |  67.91±0.07 |  70.63±0.11 |  77.6±0.01 |  65.67±0.03
C13 | C16 |  77.16±0.02 |  77.18±0.02 |  76.98±0.05 |  68.35±0.07 |  69.04±0.08 |  77.11±0.08 |  65.44±0.03
C13 | C16 |  76.74±0.02 |  76.78±0.04 |  76.33±0.04 |  68.15±0.05 |  68.27±0.08 |  76.64±0.06 |  65.16±0.01
C13 | C16 |  76.41±0.04 |  76.46±0.04 |  75.85±0.05 |  68.68±0.42 |  68.45±0.08 |  76.13±0.06 |  64.81±0.03
C13 | C18 |  77.34±0.03 |  77.37±0.01 |  77.27±0.05 |  66.54±0.05 |  70.13±0.13 |  77.15±0.04 |  65.41±0.02
C13 | C18 |  76.77±0.04 |  76.78±0.02 |  76.49±0.04 |  67.04±0.09 |  68.59±0.05 |  76.66±0.08 |  65.28±0.02
C13 | C18 |  76.35±0.03 |  76.35±0.02 |  75.98±0.04 |  67.53±0.07 |  67.87±0.07 |  76.19±0.05 |  65.12±0.02
C13 | C18 |  76.11±0.02 |  76.14±0.04 |  75.61±0.05 |  68.33±0.2 |  68.06±0.15 |  75.95±0.03 |  64.9±0.02
C13 | C21 |  77.65±0.03 |  77.72±0.06 |  77.72±0.04 |  67.65±0.12 |  70.66±0.1 |  77.56±0.02 |  65.61±0.02
C13 | C21 |  77.16±0.05 |  77.18±0.03 |  76.94±0.05 |  68.28±0.06 |  69.02±0.09 |  77.07±0.06 |  65.35±0.02
C13 | C21 |  76.74±0.04 |  76.78±0.02 |  76.33±0.04 |  68.13±0.02 |  68.31±0.06 |  76.54±0.07 |  65.1±0.02
C13 | C21 |  76.4±0.05 |  76.45±0.03 |  75.86±0.04 |  68.54±0.45 |  68.47±0.07 |  75.97±0.07 |  64.75±0.02
C13 | C24 |  77.45±0.04 |  77.52±0.04 |  77.53±0.02 |  67.22±0.09 |  70.57±0.11 |  77.46±0.06 |  65.53±0.02
C13 | C24 |  76.88±0.02 |  76.91±0.03 |  76.7±0.05 |  67.59±0.06 |  68.9±0.08 |  77.02±0.04 |  65.3±0.01
C13 | C24 |  76.48±0.04 |  76.51±0.04 |  76.0±0.06 |  68.33±0.08 |  68.13±0.05 |  76.65±0.06 |  65.03±0.01
C13 | C24 |  76.23±0.04 |  76.24±0.04 |  75.58±0.04 |  69.0±0.43 |  68.21±0.13 |  76.18±0.04 |  64.73±0.01
C13 | C26 |  77.47±0.02 |  77.51±0.03 |  77.37±0.04 |  66.1±0.15 |  71.0±0.08 |  77.16±0.06 |  65.16±0.01
C13 | C26 |  76.91±0.05 |  76.94±0.04 |  76.63±0.05 |  66.74±0.16 |  69.15±0.06 |  76.55±0.04 |  64.95±0.01
C13 | C26 |  76.4±0.04 |  76.43±0.04 |  75.99±0.04 |  67.46±0.08 |  68.36±0.08 |  76.15±0.06 |  64.7±0.02
C13 | C26 |  76.03±0.03 |  76.02±0.04 |  75.41±0.03 |  68.35±0.24 |  68.41±0.06 |  75.76±0.05 |  64.43±0.01
Table 10: AUC scores of training baselines of Random Bags with error bars. Bag size | 64 | 128 | 256 | 512
---|---|---|---|---
${\sf DLLP\text{-}BCE}$ | 77.54±0.02 | 76.96±0.02 | 76.24±0.04 | 75.22±0.05
${\sf DLLP\text{-}MSE}$ | 77.56±0.02 | 77.03±0.02 | 76.33±0.03 | 75.42±0.05
${\sf GenBags}$ | 77.08±0.03 | 76.5±0.03 | 75.75±0.08 | 75.22±0.05
${\sf Easy\text{-}LLP}$ | 75.69±0.18 | 74.18±0.54 | 72.32±0.11 | 70.13±0.47
${\sf OT\text{-}LLP}$ | 74.25±0.1 | 71.53±0.09 | 68.1±0.07 | 65.26±0.48
${\sf SIM\text{-}LLP}$ | 77.41±0.03 | 76.73±0.08 | 75.47±0.16 | 73.13±0.27
${\sf Soft\text{-}EROT\text{-}LLP}$ | 74.43±0.07 | 71.82±0.09 | 68.34±0.16 | 65.16±0.51
${\sf Hard\text{-}EROT\text{-}LLP}$ | 74.23±0.06 | 71.65±0.1 | 68.08±0.15 | 65.54±0.4
${\sf Mean\text{-}Map}$ | 63.17±0.03 | 63.06±0.05 | 62.83±0.06 | 62.26±0.06
Table 11: Bag Level Statistics of all the Groupings (Emboldened : Used for Training) Col1 | Col2 | No. bags created | No. bags after clipping |  $\%$ Inst. after clipping | Mean bag size | Stddev of bag sizes
---|---|---|---|---|---|---
C1 | - | 1443 | 1261 | 0.7 | 256.16 | 386.61
C3 | - | 175781 | 39052 | 18.42 | 216.17 | 316.89
C4 | - | 128509 | 38802 | 21.68 | 256.11 | 359.9
C7 | - | 11930 | 7839 | 12.39 | 724.69 | 636.25
C8 | - | 629 | 531 | 0.34 | 295.24 | 446.88
C10 | - | 41224 | 20252 | 17.01 | 384.92 | 482.64
C11 | - | 5160 | 2519 | 4.57 | 832.04 | 683.76
C12 | - | 174835 | 39444 | 18.79 | 218.32 | 318.82
C13 | - | 3175 | 1221 | 2.98 | 1120.01 | 684.56
C15 | - | 11254 | 6514 | 7.28 | 512.67 | 569
C16 | - | 165206 | 40109 | 20.06 | 229.24 | 334.09
C18 | - | 4605 | 2623 | 3.32 | 580.78 | 607.85
C19 | - | 2017 | 1300 | 1.99 | 701.55 | 651.27
C21 | - | 172322 | 39781 | 19.28 | 222.14 | 322.1
C24 | - | 56456 | 21694 | 14.66 | 309.88 | 421.33
C26 | - | 43356 | 17702 | 11.57 | 299.58 | 404.4
C1 | C2 | 144029 | 11999 | 9.12 | 348.43 | 463.68
C1 | C3 | 1986996 | 47251 | 22.36 | 216.96 | 318.6
C1 | C4 | 1807068 | 58010 | 28.87 | 228.13 | 328.8
C1 | C5 | 4852 | 4274 | 2.27 | 243.02 | 370.43
C1 | C6 | 9950 | 2075 | 1.53 | 337.95 | 461.16
C1 | C7 | 724000 | 55937 | 37.91 | 310.69 | 417.54
C1 | C8 | 6577 | 5828 | 2.95 | 232.16 | 362.12
C1 | C9 | 3005 | 1501 | 0.86 | 261.47 | 391.39
C1 | C10 | 1034267 | 55528 | 32.44 | 267.81 | 374.19
C1 | C11 | 421601 | 38117 | 27.5 | 330.67 | 441.68
C1 | C12 | 1998608 | 48523 | 23.07 | 217.95 | 320.44
C1 | C13 | 334925 | 32966 | 24.27 | 337.48 | 448.48
C1 | C14 | 16449 | 2503 | 1.87 | 341.7 | 462.37
C1 | C15 | 614530 | 39530 | 26.63 | 308.8 | 418.5
C1 | C16 | 1992989 | 52671 | 25.52 | 222.14 | 324.99
C1 | C17 | 12199 | 2245 | 1.7 | 346.25 | 460.77
C1 | C18 | 365498 | 25570 | 18.07 | 324.03 | 435.59
C1 | C19 | 144901 | 10476 | 7.18 | 314.3 | 423.13
C1 | C20 | 5772 | 1732 | 1.26 | 332.59 | 460.3
C1 | C21 | 2003740 | 50168 | 23.93 | 218.63 | 320.6
C1 | C22 | 7269 | 1895 | 1.18 | 285.31 | 422.19
C1 | C23 | 13043 | 2353 | 1.76 | 343.42 | 465.82
C1 | C24 | 1062954 | 44816 | 24.81 | 253.74 | 359.8
C1 | C25 | 23208 | 3126 | 2.34 | 342.44 | 458.17
C1 | C26 | 816802 | 34315 | 18.37 | 245.35 | 349.16
C2 | C3 | 645467 | 40900 | 19.63 | 219.98 | 321.84
C2 | C4 | 588748 | 44692 | 24.03 | 246.51 | 351.32
C2 | C5 | 41463 | 5606 | 4.9 | 400.34 | 505.1
C2 | C6 | 4834 | 1907 | 2.85 | 684.3 | 653.76
C2 | C7 | 444591 | 78261 | 48.74 | 285.51 | 377.68
C2 | C8 | 73020 | 7884 | 6.42 | 373.27 | 480.22
C2 | C10 | 945374 | 75614 | 41.23 | 249.98 | 345.32
C2 | C11 | 221445 | 53308 | 39.09 | 336.11 | 431.53
C2 | C12 | 649357 | 41375 | 19.97 | 221.23 | 322.15
C2 | C13 | 162788 | 45206 | 34.87 | 353.59 | 446.07
C2 | C14 | 2102 | 850 | 1.53 | 824.67 | 693.24
C2 | C15 | 11390 | 6516 | 7.28 | 512.53 | 568.97
C2 | C16 | 664651 | 43198 | 21.66 | 229.8 | 334.93
C2 | C17 | 4855 | 2646 | 3.93 | 681.16 | 614.67
C2 | C18 | 4631 | 2624 | 3.32 | 580.59 | 607.81
C2 | C19 | 43411 | 10084 | 7.49 | 340.45 | 449.13
C2 | C21 | 658369 | 42128 | 20.57 | 223.84 | 325.78
C2 | C22 | 4337 | 1579 | 1.96 | 567.74 | 610.32
C2 | C23 | 5080 | 2523 | 3.49 | 633.74 | 611.09
C2 | C24 | 344266 | 31123 | 19.15 | 282.09 | 397.8
C2 | C25 | 5228 | 1275 | 1.32 | 475.44 | 564.53
C2 | C26 | 247348 | 24100 | 14.25 | 271.12 | 381.53
C3 | C4 | 247003 | 55817 | 26.21 | 215.23 | 310.73
C3 | C5 | 1114179 | 43934 | 20.88 | 217.81 | 318.81
C3 | C6 | 844001 | 43641 | 20.62 | 216.6 | 316.98
C3 | C7 | 7358757 | 80788 | 37.88 | 214.92 | 310.29
C3 | C8 | 1438370 | 45282 | 21.51 | 217.78 | 319.66
C3 | C9 | 309182 | 40293 | 19.2 | 218.42 | 320.38
C3 | C10 | 7699949 | 71467 | 32 | 205.27 | 300.03
C3 | C11 | 5826125 | 73331 | 35.78 | 223.64 | 324.41
C3 | C12 | 187130 | 39997 | 18.82 | 215.65 | 316.49
C3 | C13 | 5313650 | 70587 | 34.78 | 225.86 | 328.16
C3 | C14 | 413784 | 40592 | 19.43 | 219.46 | 322.47
C3 | C15 | 1076734 | 48427 | 26.22 | 248.23 | 363.96
C3 | C16 | 228945 | 48449 | 21.91 | 207.32 | 303.14
C3 | C17 | 1082770 | 45864 | 22.1 | 220.88 | 325.63
C3 | C18 | 726224 | 43469 | 22.2 | 234.16 | 346.86
C3 | C19 | 483684 | 41188 | 19.99 | 222.47 | 327.54
C3 | C20 | 346464 | 39600 | 18.8 | 217.58 | 316.56
C3 | C21 | 205311 | 42537 | 19.54 | 210.52 | 309.43
C3 | C22 | 561157 | 41869 | 19.95 | 218.46 | 323.34
C3 | C23 | 898234 | 44976 | 21.41 | 218.17 | 318.12
C3 | C24 | 229746 | 54447 | 27.12 | 228.34 | 329.83
C3 | C25 | 352479 | 40796 | 19.45 | 218.52 | 318.79
C3 | C26 | 406506 | 53753 | 26.8 | 228.57 | 329.43
C4 | C5 | 955570 | 51305 | 26.41 | 235.97 | 336.72
C4 | C6 | 649962 | 53352 | 27.1 | 232.83 | 332.09
C4 | C7 | 7408443 | 103144 | 43.82 | 194.74 | 275.28
C4 | C8 | 1264446 | 54197 | 27.55 | 233.03 | 334.98
C4 | C9 | 230748 | 42003 | 22.99 | 250.9 | 353.24
C4 | C10 | 8060081 | 86205 | 36.08 | 191.88 | 273.47
C4 | C11 | 5628943 | 96009 | 44.42 | 212.1 | 299.69
C4 | C12 | 245809 | 55816 | 26.26 | 215.67 | 311.09
C4 | C13 | 5060557 | 93015 | 43.86 | 216.13 | 306.73
C4 | C14 | 311943 | 47208 | 24.98 | 242.56 | 342.57
C4 | C15 | 969150 | 56468 | 31.06 | 252.18 | 360.88
C4 | C16 | 230280 | 49211 | 23.8 | 221.65 | 323.65
C4 | C17 | 825745 | 56219 | 28.55 | 232.81 | 334.87
C4 | C18 | 628441 | 47904 | 26.35 | 252.14 | 360.8
C4 | C19 | 682269 | 44217 | 23.23 | 240.81 | 343.11
C4 | C20 | 262150 | 43501 | 23.37 | 246.24 | 347.36
C4 | C21 | 242560 | 54642 | 25.65 | 215.17 | 309.81
C4 | C22 | 436550 | 46518 | 24.77 | 244.09 | 348.14
C4 | C23 | 686712 | 54106 | 27.73 | 234.96 | 333.37
C4 | C24 | 177482 | 45078 | 24.31 | 247.24 | 350.87
C4 | C25 | 259638 | 43224 | 23.8 | 252.44 | 356.34
C4 | C26 | 325526 | 49851 | 26.9 | 247.37 | 351.63
C5 | C6 | 2260 | 579 | 0.51 | 401.17 | 511.77
C5 | C7 | 290810 | 37354 | 29.42 | 360.98 | 463.2
C5 | C8 | 2870 | 2486 | 1.44 | 265.84 | 406.85
C5 | C10 | 480175 | 42678 | 27.35 | 293.75 | 398.83
C5 | C11 | 159530 | 22921 | 19.1 | 381.99 | 488.02
C5 | C12 | 1116548 | 44853 | 21.39 | 218.6 | 318.78
C5 | C13 | 122621 | 18931 | 16.14 | 390.76 | 494.88
C5 | C14 | 3934 | 780 | 0.68 | 399.88 | 504.78
C5 | C15 | 235816 | 25697 | 19.44 | 346.74 | 454.78
C5 | C16 | 1096716 | 47775 | 23.45 | 224.99 | 326.23
C5 | C17 | 2619 | 649 | 0.57 | 403.67 | 512.2
C5 | C18 | 128596 | 15061 | 12.02 | 366 | 472.77
C5 | C19 | 53803 | 6342 | 5.04 | 364.25 | 467.44
C5 | C21 | 1114287 | 46016 | 22.18 | 220.94 | 321.83
C5 | C23 | 2904 | 698 | 0.62 | 404.87 | 524.76
C5 | C24 | 517424 | 36088 | 21.14 | 268.51 | 375.4
C5 | C25 | 5983 | 1066 | 0.91 | 390.25 | 492.66
C5 | C26 | 401151 | 28221 | 16.01 | 260.12 | 365.26
C6 | C7 | 82996 | 38449 | 35.84 | 427.31 | 492.33
C6 | C8 | 4482 | 1006 | 0.82 | 371.25 | 477.59
C6 | C10 | 235940 | 46981 | 31.8 | 310.24 | 411.03
C6 | C11 | 38136 | 18249 | 21.92 | 550.51 | 575.83
C6 | C12 | 840420 | 44685 | 21.2 | 217.45 | 318.9
C6 | C13 | 25706 | 13586 | 17.92 | 604.75 | 595.34
C6 | C15 | 70477 | 21774 | 19.65 | 413.78 | 509.4
C6 | C16 | 808500 | 47935 | 23.52 | 224.89 | 324.94
C6 | C18 | 31408 | 10996 | 10.79 | 449.9 | 531.49
C6 | C19 | 14898 | 6625 | 6.02 | 416.53 | 489.81
C6 | C21 | 833932 | 46083 | 21.94 | 218.28 | 317.68
C6 | C24 | 301823 | 37787 | 22.33 | 270.9 | 377.4
C6 | C26 | 238487 | 30115 | 17.26 | 262.65 | 372.55
C7 | C8 | 436540 | 45042 | 33.37 | 339.57 | 446.24
C7 | C9 | 11932 | 7842 | 12.4 | 724.75 | 636.39
C7 | C10 | 238182 | 56575 | 40.52 | 328.31 | 428.64
C7 | C11 | 17182 | 10090 | 14.47 | 657.49 | 620.35
C7 | C12 | 7437839 | 84716 | 39.24 | 212.34 | 304.78
C7 | C13 | 12348 | 7840 | 12.39 | 724.6 | 636.25
C7 | C14 | 91450 | 38685 | 34.97 | 414.38 | 491.63
C7 | C15 | 2885460 | 123209 | 53.78 | 200.1 | 280.57
C7 | C16 | 7589473 | 94444 | 41.84 | 203.06 | 287.73
C7 | C17 | 40660 | 21577 | 24.25 | 515.12 | 550.29
C7 | C18 | 1770225 | 104806 | 51.31 | 224.44 | 314.1
C7 | C19 | 1417980 | 41919 | 29.7 | 324.83 | 446.6
C7 | C20 | 47371 | 30420 | 31.85 | 479.96 | 523.93
C7 | C21 | 7515688 | 88970 | 40.43 | 208.31 | 297.62
C7 | C22 | 70043 | 24036 | 23.42 | 446.7 | 517.49
C7 | C23 | 69312 | 24818 | 23.79 | 439.47 | 519.91
C7 | C24 | 5036272 | 103413 | 46.89 | 207.86 | 294.56
C7 | C25 | 166138 | 26949 | 25.82 | 439.26 | 523.34
C7 | C26 | 3804785 | 71088 | 35.56 | 229.28 | 335.78
C8 | C9 | 1331 | 666 | 0.44 | 301.59 | 445.21
C8 | C10 | 672470 | 48179 | 29.58 | 281.42 | 386.29
C8 | C11 | 246919 | 28835 | 22.65 | 360.15 | 468.25
C8 | C12 | 1443335 | 46342 | 22.08 | 218.43 | 319.62
C8 | C13 | 192815 | 24342 | 19.56 | 368.4 | 476.4
C8 | C14 | 7603 | 1321 | 1.05 | 364.99 | 472.07
C8 | C15 | 360202 | 31235 | 22.42 | 329.01 | 437.19
C8 | C16 | 1426019 | 49809 | 24.34 | 224 | 325.98
C8 | C17 | 5354 | 1163 | 0.95 | 374.7 | 489.44
C8 | C18 | 204516 | 19086 | 14.39 | 345.67 | 453.35
C8 | C19 | 83464 | 7972 | 5.88 | 338.4 | 444.15
C8 | C20 | 2516 | 798 | 0.63 | 360.99 | 467.7
C8 | C21 | 1442803 | 47665 | 22.87 | 219.92 | 320.34
C8 | C22 | 3359 | 901 | 0.65 | 330.91 | 475.51
C8 | C23 | 5847 | 1206 | 0.99 | 376.07 | 492.87
C8 | C24 | 709532 | 39791 | 22.66 | 261.1 | 366.7
C8 | C25 | 11146 | 1738 | 1.41 | 371.65 | 483.72
C8 | C26 | 548366 | 30787 | 17.03 | 253.52 | 358.22
C9 | C10 | 57586 | 23958 | 19.2 | 367.44 | 466.48
C9 | C11 | 6317 | 3221 | 5.71 | 812.21 | 679.5
C9 | C12 | 307826 | 40753 | 19.57 | 220.16 | 321.14
C9 | C13 | 4085 | 1765 | 3.98 | 1033.66 | 690.28
C9 | C15 | 22903 | 10022 | 10.06 | 459.94 | 537.52
C9 | C16 | 293126 | 42011 | 20.92 | 228.26 | 332.12
C9 | C18 | 9681 | 4547 | 5.06 | 509.69 | 566.6
C9 | C19 | 4237 | 2214 | 2.6 | 538.05 | 582.85
C9 | C21 | 304213 | 41192 | 20.05 | 223.11 | 322.92
C9 | C24 | 104354 | 25184 | 16.51 | 300.58 | 411.1
C9 | C26 | 79650 | 20247 | 12.76 | 288.85 | 394.88
C10 | C11 | 91642 | 33364 | 27.97 | 384.27 | 480.8
C10 | C12 | 7813021 | 73831 | 32.78 | 203.52 | 296.05
C10 | C13 | 70927 | 29424 | 24.97 | 389.02 | 485.56
C10 | C14 | 265775 | 47774 | 32.11 | 308.12 | 412
C10 | C15 | 3893598 | 102841 | 44.66 | 199.07 | 283.28
C10 | C16 | 8081277 | 79973 | 34.12 | 195.6 | 280.63
C10 | C17 | 212092 | 44463 | 30.74 | 316.96 | 419.23
C10 | C18 | 2524716 | 90181 | 42.31 | 215.05 | 305.56
C10 | C19 | 1736492 | 48288 | 28.12 | 266.92 | 378.75
C10 | C20 | 160915 | 41714 | 30 | 329.68 | 433.6
C10 | C21 | 7935189 | 76455 | 33.32 | 199.78 | 288.52
C10 | C22 | 169631 | 33712 | 24.05 | 327.08 | 430.99
C10 | C23 | 196907 | 38464 | 27.12 | 323.17 | 423.14
C10 | C24 | 5740783 | 87116 | 38.29 | 201.48 | 285.8
C10 | C25 | 283759 | 37913 | 26.43 | 319.56 | 425.56
C10 | C26 | 4449999 | 65658 | 31.38 | 219.1 | 319.86
C11 | C12 | 5873140 | 76545 | 37.53 | 224.75 | 324.16
C11 | C13 | 5460 | 2521 | 4.57 | 830.67 | 683.58
C11 | C14 | 40358 | 19527 | 21.42 | 502.94 | 556.55
C11 | C15 | 1847996 | 110386 | 53.39 | 221.71 | 306.51
C11 | C16 | 5919824 | 85995 | 41.47 | 221.08 | 315.6
C11 | C17 | 31753 | 16992 | 20.57 | 554.93 | 576.07
C11 | C18 | 1088147 | 87394 | 47.51 | 249.22 | 344.12
C11 | C19 | 947216 | 31892 | 20.29 | 291.63 | 426.76
C11 | C20 | 20474 | 12946 | 18.01 | 637.83 | 604.71
C11 | C21 | 5913565 | 80683 | 39.4 | 223.86 | 322.08
C11 | C22 | 33530 | 12068 | 13.13 | 498.66 | 559.33
C11 | C23 | 30567 | 13330 | 13.9 | 478.04 | 553.47
C11 | C24 | 3654197 | 91578 | 45.3 | 226.75 | 319.09
C11 | C25 | 91831 | 14952 | 14.43 | 442.27 | 545.98
C11 | C26 | 2775118 | 63466 | 31.4 | 226.78 | 329.92
C12 | C13 | 5350294 | 73654 | 36.56 | 227.52 | 329.06
C12 | C14 | 410926 | 41573 | 19.85 | 218.85 | 320.3
C12 | C15 | 1078982 | 49466 | 26.56 | 246.12 | 361.67
C12 | C16 | 227478 | 48044 | 21.84 | 208.34 | 304.65
C12 | C17 | 1078501 | 46908 | 22.61 | 220.91 | 324.81
C12 | C18 | 728462 | 44143 | 22.46 | 233.28 | 344.19
C12 | C19 | 489684 | 41677 | 20.36 | 223.94 | 329.07
C12 | C20 | 346040 | 40127 | 19.11 | 218.34 | 316.26
C12 | C21 | 201953 | 41802 | 19.51 | 213.92 | 313.62
C12 | C22 | 559523 | 42652 | 20.45 | 219.76 | 324.27
C12 | C23 | 894488 | 46019 | 21.95 | 218.66 | 318.19
C12 | C24 | 228628 | 54766 | 27.35 | 228.93 | 329.94
C12 | C25 | 351278 | 41326 | 19.84 | 220.07 | 319.52
C12 | C26 | 406749 | 54177 | 27.06 | 229 | 329.34
C13 | C14 | 26418 | 14778 | 17.58 | 545.39 | 576.63
C13 | C15 | 1541997 | 103804 | 52.44 | 231.6 | 320.76
C13 | C16 | 5369689 | 82839 | 40.64 | 224.87 | 321.63
C13 | C17 | 21711 | 13065 | 17.24 | 605.06 | 594.38
C13 | C18 | 891380 | 80044 | 45.09 | 258.23 | 354.54
C13 | C19 | 814645 | 29681 | 18.26 | 282.07 | 416.78
C13 | C20 | 12667 | 8879 | 14.33 | 740 | 625.94
C13 | C21 | 5379799 | 77621 | 38.45 | 227.05 | 327.39
C13 | C22 | 23085 | 8892 | 10.41 | 536.5 | 583.78
C13 | C23 | 20835 | 10425 | 11.47 | 504.4 | 567.35
C13 | C24 | 3231274 | 87527 | 44.17 | 231.32 | 326.29
C13 | C25 | 69825 | 11776 | 11.75 | 457.51 | 559.89