# Fusing Conditional Submodular GAN and Programmatic Weak Supervision

Kumar Shubham, Pranav Sastry, Prathosh AP

###### Abstract

Programmatic Weak Supervision (PWS) and generative models serve as crucial
tools that enable researchers to maximize the utility of existing datasets
without resorting to laborious data gathering and manual annotation processes.
PWS uses various weak supervision techniques to estimate the underlying class
labels of data, while generative models primarily concentrate on sampling from
the underlying distribution of the given dataset. Although these methods have
the potential to complement each other, they have mostly been studied
independently. Recently, WSGAN proposed a mechanism to fuse these two models.
Their approach utilizes the discrete latent factors of InfoGAN to train the
label model and leverages the class-dependent information of the label model
to generate images of specific classes. However, the disentangled latent
factors learned by InfoGAN might not necessarily be class-specific and could
potentially affect the label model’s accuracy. Moreover, prediction made by
the label model is often noisy in nature and can have a detrimental impact on
the quality of images generated by GAN. In our work, we address these
challenges by (i) implementing a noise-aware classifier using the pseudo
labels generated by the label model (ii) utilizing the noise-aware
classifier’s prediction to train the label model and generate class-
conditional images. Additionally, we also investigate the effect of training
the classifier with a subset of the dataset within a defined uncertainty
budget on pseudo labels. We accomplish this by formalizing the subset
selection problem as a submodular maximization objective with a knapsack
constraint on the entropy of pseudo labels. We conduct experiments on multiple
datasets and demonstrate the efficacy of our methods on several tasks vis-a-
vis the current state-of-the-art methods. Our implementation is available at
https://github.com/kyrs/subpws-gan

## Introduction

The success of many deep learning methods is often credited to the
availability of large amounts of labeled data (Deng et al. 2009; Cao et al.
2018; Roberts et al. 2021; Chen et al. 2020a). However, labeling necessitates
manual annotation, which is a time-consuming and labor-intensive process.
Distant supervision or weak supervision methods address this issue by using an
external knowledge base (Hoffmann et al. 2011), pre-trained models (Bach et
al. 2019), and heuristics approaches (Awasthi et al. 2020) to generate
inexpensive but potentially noisy class labels for downstream tasks. These
methods use an alternative source of information to assign labels to the
training data and thus avoid the need for extensive manual annotation.
However, the pseudo labels generated by such methods are often noisy in nature
and can have a detrimental impact on downstream performance.

Programmatic weak supervision (Zhang et al. 2022a; Dehghani et al. 2017; Lang
and Poon 2021; Ratner et al. 2016, 2020; Riedel, Yao, and McCallum 2010)
addresses this issue by combining the prediction of multiple such noisy
sources of labels often known as label functions (LFs) systematically. In PWS,
a label model is trained to combine the prediction of these label functions to
estimate the unobserved ground truth label for a given training sample. The
generated pseudo label is then used for downstream training. For a given set
of label functions, the quality of the pseudo label is often determined by the
performance of the label model. Various approaches have been suggested to
effectively train the label model (Zhang et al. 2022a) without knowing the
underlying ground truth class labels. While approaches like (Ratner et al.
2016, 2020, 2019; Fu et al. 2020) use only the prediction of label functions
to estimate the pseudo label, encoder-based label models also consider data
features to estimate pseudo labels (Cachay et al. 2021; Boecking et al. 2023).

Recently, Boecking et al. (2023) proposed WSGAN, a novel training strategy to
efficiently generate pseudo labels for a given image dataset. Their approach
combines InfoGAN with an encoder-based label model (Cachay et al. 2021). The
key idea is to train both InfoGAN (Chen et al. 2016) and label model
simultaneously and leverage the disentangled latent factors learned by InfoGAN
to align the label model’s predictions. This training process offers several
advantages. The synchronized training of the label model and InfoGAN allows
the efficient utilization of the disentangled latent factors of InfoGAN for
the training of the label model. Simultaneously, the pseudo labels generated
by the label model facilitate the generation of class-conditioned images,
which can benefit downstream tasks as an additional form of data augmentation.

Nevertheless, the implementation of such a training process presents several
challenges. The disentangled latent representation of InfoGAN is learned by
maximizing the mutual information between the discrete conditional input and
the generated images. However, the representations (latent factors) learned
from InfoGAN may not be class-specific. In such a scenario, aligning the
prediction of the label model with such latent factors can hurt the accuracy
of the label model. Furthermore, the predictions of the label models are often
noisy in nature, and using the pseudo labels directly for GAN training can
make the overall training process unstable, impacting the quality of images
generated by InfoGAN.

In this study, we address these challenges by training a noise-aware
classifier-guided conditional GAN (Odena, Olah, and Shlens 2017) for the
training of label model and synthetic image generation. In contrast to
conventional conditional GANs that rely on authentic ground truth labels
during training, our approach operates within a weak supervision framework,
where the pseudo labels generated by the label model are used to train the
classifier. To handle the noise associated with the pseudo labels, the
classifier is trained using a noise-aware symmetric cross entropy loss (Wang
et al. 2019) in an adaptive fashion (Morerio et al. 2020). Within the current
setup, as training proceeds, a new set of refined dataset and associated
pseudo labels generated by the label model is used to train the classifier.

Further, a recent line of research by Lang, Vijayaraghavan, and Sontag (2022)
shows that, in weak supervision, a subset of the most representative training
samples yields superior performance compared to using the entire dataset for
downstream training of a classifier. Motivated by this, we propose to utilize
a subset of highly representative and diverse examples that exhibit minimal
uncertainty with the pseudo labels. We accomplish this by incorporating a data
subset selection process for weak supervision by formulating it as a
submodular maximization problem (Bilmes 2022; Krause and Golovin 2014) with a
knapsack constraint defined over the overall entropy of the pseudo labels of
the selected samples. (Krause and Golovin 2014; Xiong, Mehta, and Singh 2019).
Our contributions are summarized below:

(1) We propose a novel technique to fuse a classifier-guided conditional GAN
with an encoder-based label model. Within the framework of programmatic weak
supervision (PWS), this helps in efficient training of the label model and
generation of class conditional images, (2) We present a novel approach for
subset selection to be used in tandem with PWS by identifying the most diverse
and representative samples via a submodular maximization technique, aiding in
the reduction of the uncertainty associated with the training dataset, and (3)
We investigate the impact on the overall performance of the label model as
well as on the quality of images using the proposed subset selection scheme
and compare our method with the current state-of-the-art.

## Related Work

### Programmatic Weak Supervision

Programmatic Weak Supervision (Ratner et al. 2016) is an efficient technique
that addresses the issue of lack of ground truth labels for a downstream task.
Within this framework, a subject matter expert uses different sources of noisy
labeling schemes to annotate the unlabeled dataset. These sources often known
as label functions (LFs) provides partial information about the true labels
and exhibit superior performance than a random model. Generally, these label
functions are often approximated using an external knowledge base (Hoffmann et
al. 2011), pre-trained models (Bach et al. 2019), heuristics approaches
(Awasthi et al. 2020) and other similar techniques (Zhang et al. 2022a). A
common practice in PWS is to define a threshold associated with each of these
label functions so that an LF can abstain from making an uncertain prediction
about any example. One of the main tasks in PWS is to train a label model that
combines the prediction from different LFs to estimate the underlying ground
truth labels. This is done by estimating the accuracy and dependency between
LFs and then using it to weight their prediction to estimate the pseudo
labels. Different methods have used different approximation techniques to make
informed predictions about the pseudo labels. Ratner et al. (2016) modeled the
label model using a factor graph, Dawid and Skene (1979) used expectation
maximization to estimate the pseudo labels, Ratner et al. (2019) used a Markov
network and matrix completion techniques to recover the associated parameters,
FlyingSquid (Fu et al. 2020) used ising model to predict pseudo labels. While
most of the methods make an estimate about the pseudo labels by only
considering the prediction made by the label functions, Cachay et al. (2021)
proposed to use a neural network, which implicitly captures these dependencies
and use data-dependent features to estimate the accuracy of the label
functions. WSGAN (Boecking et al. 2023) and WSVAE (Tonolini et al. 2023)
further built on top of the idea proposed by Cachay et al. (2021), and use
generative models such as InfoGAN and VAE to train the label model. While
WSGAN proposed the given fusion for image data, WSVAE primarily focused on
improving the label model’s performance. To improve the performance of the
downstream model, recent works (Zhang et al. 2022b; Yu, Ding, and Bach 2022)
have proposed to use noise-aware loss in PWS. However, these losses utilize
the weights of label model to improve the downstream prediction task and,
hence, are not suitable for the training of the label model.
Label models are generally data agnostic and only depend on the prediction
made by the label functions. However, the label function used to generate
noisy labels can be different for different types of data modalities. For
instance, Varma and Ré (2018); Fries et al. (2019) proposed to generate pseudo
labels using heuristic rules. Similarly, Chen et al. (2019); Hooper et al.
(2020) define rules on top of the prediction made by a surrogate model. Joulin
et al. (2016); Wang et al. (2017); Irvin et al. (2019); Boecking et al.
(2020); Eyuboglu et al. (2021) used the information from other modalities of
data like text to estimate the labels.

### Conditional GANs with noisy labels

Synthetic data generated using class-conditional generative models, such as
Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) is another
approach to efficiently utilize the available dataset. However, in a weak
supervision setting, such a model needs to be trained with noisy class labels.
In a noisy environment, several methods have been proposed to efficiently
train conditional GAN (Thekumparampil et al. 2018; Han et al. 2020; Kaneko,
Ushiku, and Harada 2019; Kaneko and Harada 2021). However, most of these
methods use a predefined set of noisy class labels to train the GAN. Recently,
Morerio et al. (2020) have shown that in a multi-domain setting, the
performance of a classifier can further be improved by utilizing a conditional
GAN (Mirza and Osindero 2014). They proposed a unique training paradigm where
the GAN and the classifier are jointly trained in an iterative fashion, in
which the classifier provides refined labels for the GAN, and the GAN provides
better samples to train the classifier. However, the training of such a model
is limited only to a multi-domain setting, where accurate class labels for at
least one domain are available.

### Subset selection for weak supervision

The performance of a classifier in a weakly supervised setting is influenced
by the data used for its training (Lang, Vijayaraghavan, and Sontag 2022;
Angelova, Abu-Mostafam, and Perona 2005; Maheshwari et al. 2020;
Mirzasoleiman, Cao, and Leskovec 2020; Karamanolakis et al. 2021). There
exists a trade-off between the quantity of data used and the accuracy of the
corresponding pseudo labels (Lang, Vijayaraghavan, and Sontag 2022).
Generally, it is customary to utilize the entire dataset for training
purposes. However, this approach may introduce unnecessary noise and
potentially hinder the overall performance of the classifier. A possible
solution is to formulate the given problem as a subset selection problem and
select the most representative samples with the least entropy associated with
the pseudo labels for the training of the classifier (Lang, Vijayaraghavan,
and Sontag 2022). However, these methods do not provide any guarantee for the
representativeness and diversity of the samples used for training and are not
computationally suited for an adaptive training setting, which requires
selecting multiple pseudo-labeled subsets.

Selecting an optimal subset of representative samples from the training data
is an NP-hard problem. However, if the function used to measure the utility of
a given set is shown to be submodular, then the set generated by maximizing
the given utility using a greedy based selection scheme with constraints like
cardinality (Nemhauser, Wolsey, and Fisher 1978) and knapsack (Xiong, Mehta,
and Singh 2019; Leskovec et al. 2007; Krause and Golovin 2014), is guaranteed
to be very close to the optimal set, thereby approximately solves the subset
selection problem.

Recent works (Bilmes 2022; Xiong, Mehta, and Singh 2019; Joseph et al. 2019;
Simsar et al. 2023; Wei, Iyer, and Bilmes 2015; Maheshwari et al. 2021; Sinha
et al. 2020), have used submodular functions to select a subset of the entire
data for efficient training of deep learning models.

## Proposed Method

### Problem setting and overview

Let there be $n$ training data samples
$\tilde{{\mathcal{D}}}=\\{x_{1},x_{2},\ldots x_{n}\\}$ drawn i.i.d from a
distribution ${\mathcal{P}}_{x}$. The objective of our formulation is two
folds: firstly, we want to infer the class labels for these samples i.e.,
$y\in\\{0\ldots{\mathcal{C}}\\}$ and secondly, we want to sample synthetic
images whose marginal is represented as ${\mathcal{P}}_{x}$. In our setting,
instead of observing true labels, we have access to the predictions of
${\mathcal{K}}$ label functions on given training data. Each label function
$\lambda_{k}(x_{i})$ (where $k\in\\{1\ldots{\mathcal{K}}\\}$) generates a
noisy prediction of the true class label for a sample $x_{i}$. These label
functions can either predict among a given set of possible class labels i.e,
$\lambda_{k}(x_{i})\in\\{0\ldots{\mathcal{C}}\\}$ or can abstain from making a
prediction because of low confidence (Ratner et al. 2016). Let us refer to the
data with a prediction from at least one label function as
$\tilde{{\mathcal{D}}_{t}}\in\tilde{{\mathcal{D}}}$ (non-abstained dataset).
For each data sample we have an associated prediction from ${\mathcal{K}}$
label functions represented as
$\Lambda(x_{i})=(\lambda_{1}(x_{i}),\lambda_{2}(x_{i})\ldots\lambda_{{\mathcal{K}}}(x_{i}))$.
A label model (${\mathcal{L}}$) utilizes the predictions made by these LFs to
predict a final pseudo label ($\hat{y_{i}}$) for a given input sample $x_{i}$.
To train our noise-aware classifier, we generate a subset of original training
data $\tilde{{\mathcal{D}}_{0}}\in\tilde{{\mathcal{D}}_{t}}$ and their
associated pseudo labels
($\hat{{\mathcal{Y}}}=\\{\hat{y_{1}}\ldots\hat{y}_{|\tilde{{\mathcal{D}}}_{0}|}$
}) and study the impact of training the noise-aware classifier with the given
subset on the overall performance of the conditional GAN and the label model.

Figure 1: Overview of the proposed fusion between conditional GAN and
Programmatic Weak Supervision. In the current setup, a subset of the non-
abstained samples, selected by submodular maximization under a knapsack
constraint, is used to train a noise-aware classifier. This classifier shares
its network weights with the discriminator and label model. The noise-aware
classifier is then employed to train a conditional GAN and refine the label
model. As the training progresses and the label model becomes more accurate, a
better subset is utilized to train the classifier.

### Noise-aware classifier training

As stated previously, we have used a classifier to facilitate the training of
the conditional GAN and the alignment of the label model. Given the absence of
ground truth labels, the classifier is trained using the pseudo labels
generated by the label model.

The cross-entropy loss is a common choice of the loss function used to train a
classifier. However, recent studies (Wang et al. 2019; Feng et al. 2021) have
shown that a classifier trained only with cross-entropy loss under a noisy
setting tends to exhibit prediction bias and is susceptible to overfitting. In
the context of weak supervision, such a classifier can not only impact the
performance of the conditional GAN but can also lead to further deterioration
of the quality of pseudo labels generated by the label model.

In our study, we address this issue by training our classifier using a
symmetric cross-entropy loss (${\bm{l}}_{sce}$) (Wang et al. 2019) between the
pseudo label ($\hat{y}\sim\hat{{\mathcal{Y}}}$) generated by the label model,
and the class conditional distribution modeled by the classifier
(${\mathcal{P}}(t|x)$). The classifier is trained on a subset of the training
dataset ($\tilde{{\mathcal{D}}}_{0}$) selected by our proposed submodular
maximization scheme. Symmetric cross-entropy loss is a weighted combination of
standard cross-entropy loss (${\bm{l}}_{ce}$) and reverse cross-entropy loss
(${\bm{l}}_{rce}$). While the cross-entropy loss provides a better convergence
to the classifier, reverse cross-entropy loss provides robustness to different
types of noise such as uniform and class conditional noise (Wang et al. 2019;
Ghosh, Kumar, and Sastry 2017), thereby ensuring that the classifier does not
overfit to noisy class labels. Mathematically, the symmetric cross-entropy
loss (${\bm{l}}_{sce}$) is described in Eq. 1.

$\displaystyle{\bm{l}}_{ce}(x,\hat{y})$
$\displaystyle=-\sum_{t=0}^{{\mathcal{C}}}\hat{y}_{t}\log({\mathcal{P}}(t|x))$
$\displaystyle{\bm{l}}_{rce}(x,\hat{y})$
$\displaystyle=-\sum_{t=0}^{{\mathcal{C}}}{\mathcal{P}}(t|x)\log(\hat{y}_{t})$
$\displaystyle{\bm{l}}_{sce}(x,\hat{y})$
$\displaystyle=\alpha{\bm{l}}_{ce}(x,\hat{y})+\beta{\bm{l}}_{rce}(x,\hat{y})$
$\displaystyle{\bm{l}}_{sce}^{\tilde{{\mathcal{D}}_{0}}}$
$\displaystyle=\frac{1}{|\tilde{{\mathcal{D}}_{0}}|}\sum_{x\sim\tilde{{\mathcal{D}}_{0}},\hat{y}\sim\hat{{\mathcal{Y}}}}{\bm{l}}_{sce}(x,\hat{y})$
(1)

The classifier is trained in an adaptive manner (Morerio et al. 2020), where a
new set of data ($\tilde{D}_{0}$) is introduced for training after a specific
number of epochs. During the training, the given examples and their associated
pseudo labels remain unchanged until the next iteration. This ensures that as
the accuracy and performance of the label model improve, a more refined
dataset and its corresponding pseudo labels are utilized for training.
Simultaneously, it prevents the model from diverging due to frequent
alterations in data samples and their associated pseudo labels.

### Conditional GAN

In our study, we have employed a classifier-guided generative adversarial
network (GAN) to generate class conditional images (Odena, Olah, and Shlens
2017). The generator (${\mathcal{G}}$) is fed a noise sample $z\sim
N(0,\textit{I})$ and a one hot representation of class label
$c\sim{\mathcal{P}}({\mathcal{C}})$ to sample an image ($x_{g}$ =
${\mathcal{G}}(z,c)$) from the parameterized distribution
(${\mathcal{P}}_{\theta}$). The discriminator ($\mathcal{D}$) tries to
distinguish between the images sampled from real data distribution
(${\mathcal{P}}_{x}$) and the images sampled from parameterized distribution
(${\mathcal{P}}_{\theta}$). For conditional image generation, the generator
further maximizes the likelihood of the image ($x_{g}$) belonging to class c
with respect to our noise-aware classifier. The objective function that is
optimized for Conditional GAN training is as follows:

$\displaystyle{\bm{l}}({\mathcal{G}},{\mathcal{D}})=$
$\displaystyle\underset{x\sim{\mathcal{P}}_{x}}{\mathbb{E}}\left[\log({\mathcal{D}}(x))\right]+\underset{x\sim{\mathcal{P}}_{\theta}}{\mathbb{E}}\left[\log(1-{\mathcal{D}}(x))\right]$
$\displaystyle\min_{{\mathcal{G}}}\max_{{\mathcal{D}}}{\bm{l}}_{gan}=$
$\displaystyle{\bm{l}}({\mathcal{G}},{\mathcal{D}})+\underset{x\sim{\mathcal{G}}(z,c),c\sim{\mathcal{P}}({\mathcal{C}})}{\mathbb{E}}\left[{\bm{l}}_{ce}(x,c)\right]$
(2)

### Construction of label models

In programmatic weak supervision, the label model accumulates the predictions
made by different label functions to make an informed final estimation of the
pseudo label associated with the given training sample. Generally, this is
done by defining an accuracy potential function ($\phi_{k}$) to capture the
association between the prediction made by a given label function and any
class label $y$ ($\phi_{k}(\lambda_{k},y)$). Similar to previous works
(Boecking et al. 2023; Ratner et al. 2020), we have defined the accuracy
potential as an indicator function,
$\phi_{k}(\lambda_{k},y)={\bm{1}}(\lambda_{k}=y)$ which captures the notion
whether a given label function ($\lambda_{k}$) has predicted class $y$ for a
given input sample. With each label function, there is an associated accuracy
parameter (${\mathcal{A}}_{k}:{\mathcal{R}}^{N}\rightarrow(0,1)$), which
determines the accuracy of its prediction. We have used sample-dependent
accuracy parameters, where the accuracy of every such association is different
for different input samples (Cachay et al. 2021; Boecking et al. 2023). The
label model (${\mathcal{L}}({\Lambda(x)})_{j}$), defined below, captures the
probability associated with class $j$ for any input sample $x$.

$\displaystyle{\mathcal{L}}({\Lambda(x)})_{j}=\frac{\exp(\sum_{i=0}^{{\mathcal{K}}}{\mathcal{A}}(x)_{i}\phi_{i}(\lambda_{i}(x),j))}{\sum_{{\bm{s}}\in
y}\exp(\sum_{i=0}^{{\mathcal{K}}}{\mathcal{A}}(x)_{i}\phi_{i}(\lambda_{i}(x),{\bm{s}}))}$
$\displaystyle\text{ where }j\in\\{0\ldots{\mathcal{C}}\\}$ (3)

The final pseudo label, associated with the input sample $x$, is defined as
the class with the maximum probability.

$\displaystyle\hat{y}=\operatorname*{arg\,max}_{j}{\mathcal{L}}({\Lambda(x)})_{j}$
(4)

To enhance the precision of the generated pseudo labels, the label model is
trained using the probabilities generated by the classifier for the samples
from the non-abstained dataset ($\tilde{{\mathcal{D}}}_{t}$). In this
approach, we employ a linear layer followed by a softmax
($\hat{{\mathcal{F}}}$) for the alignment of probabilities between the output
of the label model and the classifier (${\mathcal{P}}(t|x)$) (Boecking et al.
2023), giving rise to the following loss function.

$\displaystyle\centering{\bm{l}}_{align}(x)=-\sum_{t=0}^{{\mathcal{C}}}{\mathcal{P}}(t|x)\log(\hat{{\mathcal{F}}}({\mathcal{L}}(\Lambda(x))_{t})\@add@centering$
(5)

Since our classifier is trained iteratively by utilizing the pseudo labels
generated through the label model $\big{(}\hat{y}$ in Eq. 1$\big{)}$, we
ensure that the pseudo labels are not entirely random, especially during
initial epochs. To accomplish this, we employ a new loss ${\bm{l}}_{decay}$
that assigns equal weights to all the accuracy parameters and gradually decay
this loss as the label model becomes more accurate (Boecking et al. 2023).
Under this loss, the accuracy parameter for all the label functions is
approximated to a value of $0.5$, and the overall loss is decayed by a rate
$\mu$ as the training proceeds. Mathematically, it is equivalent to generating
an accuracy parameter of 0.5 for all the label functions, which is
approximated by a vector $0.5*\overrightarrow{{\bm{1}}}$ where
$\overrightarrow{{\bm{1}}}$ is a vector of ones with dimension equal to the
number of label functions, giving rise to the following loss function :

$\displaystyle\centering{\bm{l}}_{decay}(x)=\@add@centering$
$\displaystyle\mu||{\mathcal{A}}(x)-0.5*\overrightarrow{{\bm{1}}}||^{2}$ (6)

This process ensures that during the initial epochs, the label model
approximates a majority vote, and as training progresses, it becomes more
data-specific. Finally, the label model is trained using ${\bm{l}}_{lm}$, by
combining alignment and decay loss
${\bm{l}}_{lm}={\bm{l}}_{align}+{\bm{l}}_{decay}$.

### Subset selection for classifier training

To facilitate better training of our noise-aware classifier, we select a
subset of data ($\tilde{D}_{0}$) from the training examples that have received
at least one vote from the label functions ($\tilde{D}_{t}$). We formulate
this as a submodular optimization problem as in (Bilmes 2022; Mirzasoleiman et
al. 2013).

#### Preliminary on submodularity

###### Definition 1

A function ${\mathcal{F}}:2^{{\mathcal{V}}}\rightarrow R$ is said to be
submodular for a finite set ${\mathcal{V}}$, if for every $A\subseteq
B\subseteq{\mathcal{V}}$ and $\forall$ $v\in{\mathcal{V}}\setminus B$
${\mathcal{F}}\big{(}A\bigcup
v\big{)}-{\mathcal{F}}\big{(}A\big{)}\geq{\mathcal{F}}\big{(}B\bigcup
v\big{)}-{\mathcal{F}}\big{(}B\big{)}$

Further, a specific class of submodular functions is called monotone if
$\forall\big{(}A,B\subset{\mathcal{V}}\big{)}$ s.t. $A\subseteq B,$
${\mathcal{F}}\big{(}A\big{)}\leq{\mathcal{F}}\big{(}B\big{)}$.

A monotonous submodular function can be used to determine the utility of a
given set of data for a downstream task, and an efficient subset of the data
can be selected by maximizing the given function. For a cardinality
constraint, the process begins with an empty set (${\mathcal{S}}_{0}$), and
the selection proceeds iteratively in a greedy manner, selecting the data
point that maximizes the submodular function until the given budget on the
size is exhausted. Mathematically, if $v$ denotes the datapoint to be
selected, then
$v=\operatorname*{arg\,max}_{v\in{\mathcal{V}}\setminus{\mathcal{S}}_{n-1}}{\mathcal{F}}\big{(}v\bigcup{\mathcal{S}}_{n-1}\big{)}-{\mathcal{F}}\big{(}{\mathcal{S}}_{n-1}\big{)}$
where $\big{(}{\mathcal{S}}_{n}={\mathcal{S}}_{n-1}\bigcup{v}\big{)}$.
Although this heuristic method doesn’t guarantee the generation of an optimal
subset due to the NP-hard nature of the problem, it can still yield a solution
that is assured to be in proximity to the optimal set (Nemhauser, Wolsey, and
Fisher 1978) and hence approximately solves the submodular maximization
problem. Knapsack constraint (Krause and Golovin 2014) is another special case
of submodular maximization, where there are different costs associated with
adding every new element.

#### A submodular framework for subset selection

In our work, we have used a generalized graphcut-based monotonous submodular
function (Iyer et al. 2021; Bilmes 2022) that uses a greedy approach to select
a set of representative and diverse examples to train the classifier. Further,
the selection is done under a knapsack constraint where the addition of any
new data sample has a cost equal to the entropy of the pseudo label associated
with it, and the selection of samples is done under a budget on the overall
entropy of the subset. Eq. 7 illustrates the formulation of the generalized
graphcut algorithm. The function $({\mathcal{F}}\big{(}S\big{)})$ represents
the utility of the set $(S)$, and the variable $s_{i,j}$ represents a kernel
function that quantifies the similarity between the $i^{th}\text{ and }j^{th}$
samples in the dataset. We have used cosine similarity over the Inception
features (DeVries, Drozdzal, and Taylor 2020) of the images to define our
kernel function. The term
$\big{(}\sum_{i\in\tilde{{\mathcal{D}}}_{t}}\sum_{j\in{\mathcal{S}}}{\bm{s}}_{ij}\big{)}$
represents a cumulative score for the similarity between the samples in set
$(S)$ and the data present in the non-abstained dataset
($\tilde{{\mathcal{D}}}_{t}$), capturing the representativeness of the
selected samples in comparison to $\big{(}\tilde{{\mathcal{D}}}_{t}\big{)}$.
Additionally, the term
$\big{(}\sum_{l,m\in{\mathcal{S}}}{\bm{s}}_{l,m}\big{)}$ represents the
similarity between the selected samples in set $\big{(}S\big{)}$, minimizing
which, leads to the selection of diverse data points. The hyperparameter
$\gamma$ controls the tradeoff between representativeness and diversity.
Further, choosing $\gamma\geq 2$ ensures that the given function exhibits
monotonicity.

$\displaystyle{\mathcal{F}}\big{(}{\mathcal{S}}\big{)}=\gamma\sum_{i\in\tilde{{\mathcal{D}}}_{t}}\sum_{j\in{\mathcal{S}}}{\bm{s}}_{ij}-\sum_{l,m\in{\mathcal{S}}}{\bm{s}}_{l,m}$
(7)

We have defined a knapsack constraint on the overall entropy of the pseudo
labels generated by label model for the selected samples. Under the given
constraint, every data sample in $\tilde{{\mathcal{D}}_{t}}$ has an associated
selection cost equal to the entropy of the pseudo label, and the overall
entropy of the selected samples cannot exceed a given budget (${\mathcal{B}}$)
(Eq. 8). This enforces the selection of diverse and representative samples
while ensuring that uncertainty associated with the pseudo labels for the
selected samples is within a limit. The complete objective of our formulation
is stated as follows:

$\displaystyle\max_{{\mathcal{S}}\subset\tilde{{\mathcal{D}}}_{t}}{\mathcal{F}}\big{(}{\mathcal{S}}\big{)}$
$\displaystyle\text{s.t. Cost (${\mathcal{S}}$)}\leq{\mathcal{B}}$ (8)

The budget ${\mathcal{B}}$ on the overall entropy of the pseudo labels for the
selected samples is defined to be a fraction ($\eta\text{ where }0<\eta<1$) of
the total entropy of the pseudo labels ($Cost(\tilde{{\mathcal{D}}_{t}})$) for
the non-abstained data set. Eq. 9 illustrates the overall constraint on the
given submodular maximization problem, where ${\mathcal{H}}$ is the entropy
defined for a given sample ($x_{l}$), and ${\mathcal{L}}(\Lambda(x_{l}))$ is
the probability distribution of the label model associated with it.

$\displaystyle Cost({\mathcal{S}})$
$\displaystyle=\sum_{x_{l}\in{\mathcal{S}}}{\mathcal{H}}\big{(}x_{l};{\mathcal{L}}(\Lambda(x_{l}))\big{)}$
$\displaystyle Cost(\tilde{{\mathcal{D}}_{t}})$
$\displaystyle=\sum_{x_{l}\in\tilde{{\mathcal{D}}_{t}}}{\mathcal{H}}\big{(}x_{l};{\mathcal{L}}(\Lambda(x_{l}))\big{)}$
$\displaystyle{\mathcal{B}}$
$\displaystyle=\eta*Cost(\tilde{{\mathcal{D}}_{t}})$ (9)

For the submodular maximization problem under a knapsack constraint, a naive
greedy algorithm can perform arbitrarily bad as it is indifferent to the cost
associated with each sample. To account for the selection cost associated with
every sample, the normal greedy-based selection scheme can further be modified
to create a cost-effective greedy algorithm (CEG) (Eq. 10) to select the
optimal set, where $c(.)$ is the cost function defined for every input sample,
$v_{k}$ is the data point that is selected at $k^{th}$ iteration,
${\mathcal{F}}\big{(}{\mathcal{S}}_{k-1}\big{)}$ is the utility of the set
($S_{k-1}$) as per the the submodular function defined in Eq. 7 and
${\mathcal{S}}_{k}$ is the new set created by including the data point
${\bm{v}}_{k}$ in the set ${\mathcal{S}}_{k-1}$. In our formulation, the
function $c(.)$ is set to be equal to the entropy associated with the pseudo
label generated by label model
$\big{(}c(v)={\mathcal{H}}(v;{\mathcal{L}}(\Lambda(v)))\big{)}$. The cost-
effective greedy can be summarized as below :

$\displaystyle v_{k}=\operatorname*{arg\,max}_{v\in\tilde{{\mathcal{D}}_{t}}/\
{\mathcal{S}}_{k-1}}\frac{{\mathcal{F}}\big{(}{\mathcal{S}}_{k-1}\cup{v}\big{)}-{\mathcal{F}}\big{(}{\mathcal{S}}_{k-1}\big{)}}{c(v)}$
$\displaystyle\text{ where }{\mathcal{S}}_{k}={\mathcal{S}}_{k-1}\bigcup
v_{k}$ (10)

The given algorithm is run iteratively until the total cost of the selected
set is less than the predefined budget (${\mathcal{B}}$).

Even though cost-effective greedy considers the selection cost for each sample
while selecting a subset, it can still perform arbitrarily bad in some extreme
scenarios (Leskovec et al. 2007). Fortunately, for the given submodular
maximization objective, the cost-effective greedy algorithm can be adapted to
provide a constant factor of $\frac{1}{2}\big{(}1-\frac{1}{e}\big{)}$ to the
optimal solution (Prop. 1 in supplementary material) (Leskovec et al. 2007;
Xiong, Mehta, and Singh 2019), hence generating a subset close to the optimal
set. In fact, at least one among the final set (${\mathcal{S}}_{uniform}$)
selected by using uniform selection cost ($(c(v)=1)$) and the final set
(${\mathcal{S}}_{cost}$) selected using desired cost function
$\big{(}c(v)={\mathcal{H}}(v;{\mathcal{L}}(\Lambda(v)))\big{)}$ will have a
constant approximation bound with respect to the optimal solution, and the set
with the maximum utility score can be used for the training of the classifier
($\tilde{{\mathcal{D}}}_{0}=\operatorname*{arg\,max}_{{\mathcal{S}}}({\mathcal{F}}({\mathcal{S}}_{uniform}),{\mathcal{F}}({\mathcal{S}}_{Cost})\big{)}$.

## Experiments

We assess the effectiveness of our method by experimenting with it on
different datasets and label functions. Specifically, we use the label
functions provided by the authors of WSGAN (Boecking et al. 2023). The primary
experiments are done on five datasets, namely Animals with Attributes 2 (AWA2)
(Xian et al. 2018), DomainNet (Peng et al. 2019), CIFAR10 (Krizhevsky, Hinton
et al. 2009), MNIST (LeCun et al. 1998), FashionMNIST (Xiao, Rasul, and
Vollgraf 2017) and German Traffic Sign Benchmark (GTSRB) (Stallkamp et al.
2012). These experiments are conducted on the following four types of label
functions:

Dataset | MV | MeTaL | FS | Snorkel | DS | HLM | WSGAN | Ours (comp) | Ours (sub)
---|---|---|---|---|---|---|---|---|---
AWA2 | 0.627 | 0.622 | 0.621 | 0.617 | - | 0.651 | 0.670 | 0.701 | 0.698
CIFAR10-A | 0.831 | 0.804 | 0.800 | 0.804 | 0.850 | 0.842 | 0.872 | 0.890 | 0.890
CIFAR10-B | 0.719 | 0.708 | 0.708 | 0.709 | 0.677 | 0.726 | 0.729 | 0.740 | 0.740
DomainNet | 0.595 | 0.486 | 0.635 | 0.507 | 0.658 | 0.646 | 0.643 | 0.650 | 0.653
FashionMNIST | 0.735 | 0.730 | 0.734 | 0.725 | 0.717 | 0.735 | 0.744 | 0.754 | 0.754
GTSRB | 0.816 | 0.619 | 0.815 | 0.671 | 0.814 | - | 0.825 | 0.829 | 0.828
MNIST | 0.778 | 0.758 | 0.773 | 0.759 | 0.729 | 0.781 | 0.816 | 0.818 | 0.819

Table 1: Comparison between the average posterior accuracy of the label models
for samples with at least one vote from the label function. The best performer
across methods is denoted as bold.

Dataset | WSGAN | Ours (comp) | Ours (sub)
---|---|---|---
AWA2 | 36.00 | 25.92 | 25.23
CIFAR10-A | 22.71 | 18.00 | 17.50
CIFAR10-B | 24.41 | 18.72 | 18.72
DomainNet | 44.35 | 37.42 | 36.82
GTSRB | 73.96 | 62.55 | 59.09
FashionMNIST | 15.94 | 11.78 | 11.81
MNIST | 5.35 | 4.00 | 4.00

Table 2: Comparison of image quality (mean FID) score of the proposed method
with WSGAN. The best performer across all methods is denoted as bold.

Dataset | WG-S | WG-LF |  Ours(S)-L | Ours(s)-L | Ours(C)-CL | Ours(S)-CL
---|---|---|---|---|---|---
AWA2 | 0.99 | 0.99 | 2.58 | 1.59 | 1.59 | 1.79
CIFAR10-A | 1.38 | 0.46 | 0.50 | 0.76 | 1.74 | 2.16
CIFAR10-B | 0.66 | 0.36 | 0.42 | 0.82 | 1.22 | 1.56
DomainNet | 0.84 | 0.84 | 1.88 | 3.14 | 3.77 | 1.88
FashionMNIST | 0.36 | 2.72 | 2.74 | 2.92 | 0.74 | 0.64
GTSRB | 0.05 | 0.32 | 1.06 | 0.60 | 0.81 | 1.27
MNIST | 0.30 | 0.08 | 0.34 | 0.22 | 0.52 | 0.38

Table 3: Percentage increase in the test accuracy of ResNet-18 using synthetic
data augmentation. As a baseline, the WSGAN model with synthetic pseudo labels
(WG-S) and label function-based pseudo labels (WG-LF) is used. For the
proposed methods, we have compared the complete data setting (Ours(C)) and
subset selection setting (Ours(S)) using conditional latent vectors (L) and
classifier-based (CL) pseudo labels. The best performer for a given pseudo
label is highlighted in bold.

Domain transfer: Experiments related to DomainNet use a domain transfer
framework, wherein the label functions are trained using images from distinct
source domains, like paintings, art and are subsequently used to generate weak
labels for a new target domain like real-world images (Mazzetto et al. 2021).
Attribute heuristics: Experiments related to AWA2 use attribute classifiers
that are trained on a fixed set of animal images. The weak labels for the
unseen dataset are generated by training a decision tree on top of the
predictions made by these classifiers. Self-supervised learning: Experiments
related to CIFAR10-B, MNIST, GTSRB, and FashionMNIST use label functions
generated by finetuning a shallow MLP network over a small validation dataset.
A SimCLR (Chen et al. 2020b) model pre-trained on ImageNet was utilized to
generate features for this experiment. Synthetic: Experiments related to
CIFAR10-A are conducted using label functions generated by synthetically
corrupting the true class labels of data under given a propensity and accuracy
rate. Further details about the label functions and their associated accuracy
and coverage can be found in WSGAN (Boecking et al. 2023).

### Baselines and evaluation metrics

We analyze the performance of two versions of our model. Firstly, we conduct
experiments by considering all the samples where the label function has
provided at least one vote for the classifier’s training i.e.,
$\tilde{{\mathcal{D}}_{0}}=\tilde{{\mathcal{D}}_{t}}$. The associated results
related to these experiments are reported as Ours (comp). Secondly, we conduct
experiments where the dataset for classifier training is selected using the
proposed submodular scheme. The results for these experiments are reported as
Ours (sub). We used Apricot library (Schreiber, Bilmes, and Noble 2020) to
perform submodular optimization for subset selection using the lazy greedy
method. All the major experiments are conducted using a DCGAN (Radford, Metz,
and Chintala 2015) architecture. In the current design, the discriminator
network shares weights with the classifier and accuracy parameter-based model.
Further, we have provided a network ablation over styleGAN-ADA (Karras et al.
2020) architecture under a similar configuration for CIFAR10-B and high-
resolution images in the supplementary material, which also includes the
hyperparameters of subset selection and other implementation details.

We utilized the FID (Heusel et al. 2017) score to assess and compare the image
quality produced by our approach. Similar to WSGAN, we conducted five
independent iterations of our proposed technique and recorded the average FID
score. Additionally, as PWS operates in a transductive setting, we follow the
prior work and have reported the mean accuracy of the pseudo labels on the
training data to compare the performance of the label model with other
baselines. Metrics related to the F1 score, mean precision, and other related
scores are provided in the supplementary material. We have also evaluated the
performance of the proposed method on the downstream data augmentation task.

$0$$50$$100$$150$$200$$0$$0.2$$0.4$$0.6$$0.8$EpochARIGTSRB$0$$50$$100$$150$$200$$0$$0.2$$0.4$$0.6$$0.8$EpochAWA2$0$$50$$100$$150$$200$$0$$0.2$$0.4$$0.6$$0.8$EpochDomainNet$0$$20$$40$$60$$80$$100$$0$$0.2$$0.4$$0.6$$0.8$EpochMNIST

$0$$50$$100$$150$$200$$0$$0.2$$0.4$$0.6$$0.8$EpochARICIFAR10-A$0$$50$$100$$150$$200$$0$$0.2$$0.4$$0.6$$0.8$EpochCIFAR10-B$0$$50$$100$$150$$200$$0$$0.2$$0.4$$0.6$$0.8$EpochFashionMNISTOurs(comp)Ours(sub)WSGAN

Figure 2: Comparison of Adjusted Rank Index (ARI) of proposed methods and
WSGAN.

To show the improvement in the accuracy of the classifier, we have reported
the Adjusted Rank Index score (ARI) for the training epochs. We have conducted
a comparative analysis with different label models, including Majority Voting
(MV), MeTaL (Ratner et al. 2019), FlyingSquid (FS) (Fu et al. 2020), Snorkel
(Ratner et al. 2020), hyper-label model(HLM) (Wu et al. 2023) and DawidSkene
(DS) (Dawid and Skene 1979). To facilitate this comparison, we employed the
label model codebase from Wrench (Zhang et al. 2021) and utilized the official
codebase provided by WSGAN and hyper-label model. If any conflicts emerged
during the implementation process due to version disparities, we referred to
the numerical values reported in the paper. When information was unavailable,
we omitted to report those particular numbers.

### Results on the classifier performance

Figure-2 shows the Adjusted Rank Index between the prediction made by our
classifier and the underlying ground truth label. We have further compared it
with the prediction made by WSGAN. The results indicate that the noise-aware
classifier is more effective than WSGAN in learning the underlying ground
truth labels.

### Results on label model performance

Table-1 shows the comparison of the mean posterior accuracy of our proposed
method with other baselines. Compared to WSGAN, our proposed method gives a
3.0% improvement in AWA2 dataset and around 1.8% improvement in CIFAR10-A.
Among the subset selection and the non-abstained data, we found a marginal
difference in the performance of the label model.

### Results on image generation

The results in Table-2 highlight the average FID scores achieved by our
methods compared to those of WSGAN. On average, our method achieves 7.08 point
FID improvement compared to WSGAN. Additionally, we observe that employing a
subset selection scheme for the classifier further improves the quality of
generated images. This distinction is particularly notable for the GTSRB
dataset, with a 3.46-point FID enhancement.

### Results on data augmentation

To examine the improvement in test accuracy for the data augmentation task, we
trained a ResNet-18 (He et al. 2016) model using around 1000 synthetic images
generated by the GAN, in addition to the original training dataset. For pseudo
labels, we have considered two settings. Firstly, we use the conditional
discrete latent vector (L) used for image generation. Secondly, we use the
output of the classifier to generate the associated class label for the images
(CL). The results of the experiment is reported in Table-3. Comparatively, we
found that our proposed method generates better results than the baseline.
Among the subset selection scheme and the non-abstained data, we found the
performance of the subset selection scheme to be better for DomainNet,
FashionMNIST, CIFAR10-B, CIFAR10-A for conditional discrete latent vector (L),
and AWA2, CIFAR10-A, GTSRB, and CIFAR10-B for classifier based pseudo labels
(CL).

## Conclusions and Acknowledgments

In this work, we introduced a new framework to fuse the training of
conditional GAN and label model using a noise-aware weakly supervised
classifier. We have further investigated the impact of training the classifier
using a subset of representative and diverse samples with limited uncertainty
associated with the pseudo labels. Our empirical results indicate an
improvement in the quality of images and the accuracy of pseudo labels
compared to existing state-of-the-art techniques. We further demonstrate that
employing a subset selection strategy helps in improving the image quality of
GAN, and a small set of training data can be used to generate comparable
performance of label model. It is worth noting that the subset selection
approach has limitations tied to the size of the non-abstained sample set,
which can result in computational challenges for very large datasets. In
future work, we aim to extend our approach to bigger datasets.
This work was supported (in part for setting up the GPU compute) by the Indian
Institute of Science through a start-up grant. Prathosh is supported by the
Infosys Foundation Young Investigator Award.

## References

  * Angelova, Abu-Mostafam, and Perona (2005) Angelova, A.; Abu-Mostafam, Y.; and Perona, P. 2005.  Pruning training sets for learning of object categories.  In _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , volume 1, 494–501. IEEE.
  * Awasthi et al. (2020) Awasthi, A.; Ghosh, S.; Goyal, R.; and Sarawagi, S. 2020.  Learning from rules generalizing labeled exemplars.  _arXiv preprint arXiv:2004.06025_.
  * Bach et al. (2019) Bach, S. H.; Rodriguez, D.; Liu, Y.; Luo, C.; Shao, H.; Xia, C.; Sen, S.; Ratner, A.; Hancock, B.; Alborzi, H.; et al. 2019.  Snorkel drybell: A case study in deploying weak supervision at industrial scale.  In _Proceedings of the 2019 International Conference on Management of Data_ , 362–375.
  * Bilmes (2022) Bilmes, J. 2022.  Submodularity in machine learning and artificial intelligence.  _arXiv preprint arXiv:2202.00132_.
  * Boecking et al. (2020) Boecking, B.; Neiswanger, W.; Xing, E.; and Dubrawski, A. 2020.  Interactive weak supervision: Learning useful heuristics for data labeling.  _arXiv preprint arXiv:2012.06046_.
  * Boecking et al. (2023) Boecking, B.; Roberts, N.; Neiswanger, W.; Ermon, S.; Sala, F.; and Dubrawski, A. 2023.  Generative Modeling Helps Weak Supervision (and Vice Versa).  In _ICLR_.
  * Cachay et al. (2021) Cachay; Rühling, S.; Boecking, B.; and Dubrawski, A. 2021.  End-to-end weak supervision.  _NeurIPS_ , 34: 1845–1857.
  * Cao et al. (2018) Cao; Jiajiong, Y.; Li, Z.; and Zhang. 2018.  Celeb-500k: A large training dataset for face recognition.  In _2018 25th IEEE International Conference on Image Processing (ICIP)_ , 2406–2410. IEEE.
  * Chen et al. (2020a) Chen, H.; Xie, W.; Vedaldi, A.; and Zisserman, A. 2020a.  Vggsound: A large-scale audio-visual dataset.  In _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 721–725. IEEE.
  * Chen et al. (2020b) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020b.  A simple framework for contrastive learning of visual representations.  In _ICML_ , 1597–1607. PMLR.
  * Chen et al. (2019) Chen, V.; Wu, S.; Ratner, A. J.; Weng, J.; and Ré, C. 2019.  Slice-based learning: A programming model for residual learning in critical data slices.  _NeurIPS_ , 32.
  * Chen et al. (2016) Chen, X.; Duan, Y.; Houthooft, R.; Schulman, J.; Sutskever, I.; and Abbeel, P. 2016.  Infogan: Interpretable representation learning by information maximizing generative adversarial nets.  _NeurIPS_ , 29.
  * Choi et al. (2020) Choi, Y.; Uh, Y.; Yoo, J.; and Ha, J.-W. 2020.  Stargan v2: Diverse image synthesis for multiple domains.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 8188–8197.
  * Dawid and Skene (1979) Dawid, A. P.; and Skene, A. M. 1979.  Maximum likelihood estimation of observer error-rates using the EM algorithm.  _Journal of the Royal Statistical Society: Series C (Applied Statistics)_ , 28(1): 20–28.
  * Dehghani et al. (2017) Dehghani, M.; Zamani, H.; Severyn, A.; Kamps, J.; and Croft, W. B. 2017.  Neural ranking models with weak supervision.  In _Proceedings of the 40th international ACM SIGIR conference on research and development in information retrieval_ , 65–74.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. IEEE.
  * DeVries, Drozdzal, and Taylor (2020) DeVries, T.; Drozdzal, M.; and Taylor, G. W. 2020.  Instance selection for gans.  _NeurIPS_ , 33: 13285–13296.
  * Eyuboglu et al. (2021) Eyuboglu, S.; Angus, G.; Patel, B. N.; Pareek, A.; Davidzon, G.; Long, J.; Dunnmon, J.; and Lungren, M. P. 2021.  Multi-task weak supervision enables anatomically-resolved abnormality detection in whole-body FDG-PET/CT.  _Nature communications_ , 12(1): 1880.
  * Feng et al. (2021) Feng, L.; Shu, S.; Lin, Z.; Lv, F.; Li, L.; and An, B. 2021.  Can cross entropy loss be robust to label noise?  In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , 2206–2212.
  * Fries et al. (2019) Fries, J. A.; Varma, P.; Chen, V. S.; Xiao, K.; Tejeda, H.; Saha, P.; Dunnmon, J.; Chubb, H.; Maskatia, S.; Fiterau, M.; et al. 2019.  Weakly supervised classification of aortic valve malformations using unlabeled cardiac MRI sequences.  _Nature communications_ , 10(1): 3111.
  * Fu et al. (2020) Fu, D.; Chen, M.; Sala, F.; Hooper, S.; Fatahalian, K.; and Ré, C. 2020.  Fast and three-rious: Speeding up weak supervision with triplet methods.  In _ICML_ , 3280–3291. PMLR.
  * Ghosh, Kumar, and Sastry (2017) Ghosh, A.; Kumar, H.; and Sastry, P. S. 2017.  Robust loss functions under label noise for deep neural networks.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 31.
  * Goodfellow et al. (2014) Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014.  Generative adversarial nets.  _NeurIPS_ , 27.
  * Han et al. (2020) Han, L.; Gao, R.; Kim, M.; Tao, X.; Liu, B.; and Metaxas, D. 2020.  Robust conditional GAN from uncertainty-aware pairwise comparisons.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 10909–10916.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Heusel et al. (2017) Heusel, M.; Ramsauer, H.; Unterthiner, T.; Nessler, B.; and Hochreiter, S. 2017.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium.  In Guyon, I.; Luxburg, U. V.; Bengio, S.; Wallach, H.; Fergus, R.; Vishwanathan, S.; and Garnett, R., eds., _NeurIPS_ , volume 30. Curran Associates, Inc.
  * Hoffmann et al. (2011) Hoffmann, R.; Zhang, C.; Ling, X.; Zettlemoyer, L.; and Weld, D. S. 2011.  Knowledge-based weak supervision for information extraction of overlapping relations.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies_ , 541–550.
  * Hooper et al. (2020) Hooper, S.; Wornow, M.; Seah, Y. H.; Kellman, P.; Xue, H.; Sala, F.; Langlotz, C.; and Re, C. 2020.  Cut out the annotator, keep the cutout: better segmentation with weak supervision.  In _ICLR_.
  * Irvin et al. (2019) Irvin, J.; Rajpurkar, P.; Ko, M.; Yu, Y.; Ciurea-Ilcus, S.; Chute, C.; Marklund, H.; Haghgoo, B.; Ball, R.; Shpanskaya, K.; et al. 2019.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, 590–597.
  * Iyer et al. (2021) Iyer, R.; Khargoankar, N.; Bilmes, J.; and Asanani, H. 2021.  Submodular combinatorial information measures with applications in machine learning.  In _Algorithmic Learning Theory_ , 722–754. PMLR.
  * Joseph et al. (2019) Joseph, K.; Singh, K.; Balasubramanian, V. N.; et al. 2019.  Submodular batch selection for training deep neural networks.  _arXiv preprint arXiv:1906.08771_.
  * Joulin et al. (2016) Joulin, A.; Van Der Maaten, L.; Jabri, A.; and Vasilache, N. 2016.  Learning visual features from large weakly supervised data.  In _Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part VII 14_ , 67–84. Springer.
  * Kaneko and Harada (2021) Kaneko, T.; and Harada, T. 2021.  Blur, noise, and compression robust generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13579–13589.
  * Kaneko, Ushiku, and Harada (2019) Kaneko, T.; Ushiku, Y.; and Harada, T. 2019.  Label-noise robust generative adversarial networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2467–2476.
  * Karamanolakis et al. (2021) Karamanolakis, G.; Mukherjee, S.; Zheng, G.; and Awadallah, A. H. 2021.  Self-Training with Weak Supervision.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 845–863. Online: Association for Computational Linguistics.
  * Karras et al. (2020) Karras, T.; Aittala, M.; Hellsten, J.; Laine, S.; Lehtinen, J.; and Aila, T. 2020.  Training generative adversarial networks with limited data.  _NeurIPS_ , 33: 12104–12114.
  * Krause and Golovin (2014) Krause, A.; and Golovin, D. 2014.  Submodular function maximization.  _Tractability_ , 3(71-104): 3.
  * Krizhevsky, Hinton et al. (2009) Krizhevsky, A.; Hinton, G.; et al. 2009.  Learning multiple layers of features from tiny images.
  * Lang and Poon (2021) Lang, H.; and Poon, H. 2021.  Self-supervised self-supervision by combining deep learning and probabilistic logic.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 35, 4978–4986.
  * Lang, Vijayaraghavan, and Sontag (2022) Lang, H.; Vijayaraghavan, A.; and Sontag, D. 2022.  Training subset selection for weak supervision.  _NeurIPS_.
  * LeCun et al. (1998) LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ , 86(11): 2278–2324.
  * Leskovec et al. (2007) Leskovec, J.; Krause, A.; Guestrin, C.; Faloutsos, C.; VanBriesen, J.; and Glance, N. 2007.  Cost-effective outbreak detection in networks.  In _Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining_ , 420–429.
  * Maheshwari et al. (2020) Maheshwari, A.; Chatterjee, O.; Killamsetty, K.; Ramakrishnan, G.; and Iyer, R. 2020.  Semi-supervised data programming with subset selection.  _arXiv preprint arXiv:2008.09887_.
  * Maheshwari et al. (2021) Maheshwari, A.; Killamsetty, K.; Ramakrishnan, G.; Iyer, R.; Danilevsky, M.; and Popa, L. 2021.  Learning to robustly aggregate labeling functions for semi-supervised data programming.  _arXiv preprint arXiv:2109.11410_.
  * Mazzetto et al. (2021) Mazzetto, A.; Sam, D.; Park, A.; Upfal, E.; and Bach, S. 2021.  Semi-supervised aggregation of dependent weak supervision sources with performance guarantees.  In _International Conference on Artificial Intelligence and Statistics_ , 3196–3204. PMLR.
  * Mirza and Osindero (2014) Mirza, M.; and Osindero, S. 2014.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_.
  * Mirzasoleiman, Cao, and Leskovec (2020) Mirzasoleiman, B.; Cao, K.; and Leskovec, J. 2020.  Coresets for robust training of deep neural networks against noisy labels.  _NeurIPS_ , 33: 11465–11477.
  * Mirzasoleiman et al. (2013) Mirzasoleiman, B.; Karbasi, A.; Sarkar, R.; and Krause, A. 2013.  Distributed submodular maximization: Identifying representative elements in massive data.  _NeurIPS_ , 26.
  * Morerio et al. (2020) Morerio, P.; Volpi, R.; Ragonesi, R.; and Murino, V. 2020.  Generative pseudo-label refinement for unsupervised domain adaptation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 3130–3139.
  * Nemhauser, Wolsey, and Fisher (1978) Nemhauser, G. L.; Wolsey, L. A.; and Fisher, M. L. 1978.  An analysis of approximations for maximizing submodular set functions—I.  _Mathematical programming_ , 14: 265–294.
  * Odena, Olah, and Shlens (2017) Odena, A.; Olah, C.; and Shlens, J. 2017.  Conditional image synthesis with auxiliary classifier gans.  In _ICML_ , 2642–2651. PMLR.
  * Peng et al. (2019) Peng, X.; Bai, Q.; Xia, X.; Huang, Z.; Saenko, K.; and Wang, B. 2019.  Moment matching for multi-source domain adaptation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 1406–1415.
  * Radford, Metz, and Chintala (2015) Radford, A.; Metz, L.; and Chintala, S. 2015.  Unsupervised representation learning with deep convolutional generative adversarial networks.  _arXiv preprint arXiv:1511.06434_.
  * Ratner et al. (2020) Ratner, A.; Bach, S. H.; Ehrenberg, H.; Fries, J.; Wu, S.; and Ré, C. 2020.  Snorkel: Rapid training data creation with weak supervision.  _The VLDB Journal_ , 29(2-3): 709–730.
  * Ratner et al. (2019) Ratner, A.; Hancock, B.; Dunnmon, J.; Sala, F.; Pandey, S.; and Ré, C. 2019.  Training complex models with multi-task weak supervision.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, 4763–4771.
  * Ratner et al. (2016) Ratner, A. J.; De Sa, C. M.; Wu, S.; Selsam, D.; and Ré, C. 2016.  Data programming: Creating large training sets, quickly.  _NeurIPS_ , 29.
  * Riedel, Yao, and McCallum (2010) Riedel, S.; Yao, L.; and McCallum, A. 2010.  Modeling relations and their mentions without labeled text.  In _Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part III 21_ , 148–163. Springer.
  * Roberts et al. (2021) Roberts, M.; Ramapuram, J.; Ranjan, A.; Kumar, A.; Bautista, M. A.; Paczan, N.; Webb, R.; and Susskind, J. M. 2021.  Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding.  In _International Conference on Computer Vision (ICCV) 2021_.
  * Schreiber, Bilmes, and Noble (2020) Schreiber, J.; Bilmes, J.; and Noble, W. S. 2020.  apricot: Submodular selection for data summarization in Python.  _The Journal of Machine Learning Research_ , 21(1): 6474–6479.
  * Simsar et al. (2023) Simsar, E.; Kocasari, U.; Er, E. G.; and Yanardag, P. 2023.  Fantastic Style Channels and Where to Find Them: A Submodular Framework for Discovering Diverse Directions in GANs.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 4731–4740.
  * Sinha et al. (2020) Sinha, S.; Zhang, H.; Goyal, A.; Bengio, Y.; Larochelle, H.; and Odena, A. 2020.  Small-gan: Speeding up gan training using core-sets.  In _ICML_ , 9005–9015. PMLR.
  * Stallkamp et al. (2012) Stallkamp, J.; Schlipsing, M.; Salmen, J.; and Igel, C. 2012.  Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition.  _Neural networks_ , 32: 323–332.
  * Thekumparampil et al. (2018) Thekumparampil, K. K.; Khetan, A.; Lin, Z.; and Oh, S. 2018.  Robustness of conditional gans to noisy labels.  _NeurIPS_ , 31.
  * Tonolini et al. (2023) Tonolini, F.; Aletras, N.; Jiao, Y.; and Kazai, G. 2023.  Robust Weak Supervision with Variational Auto-Encoders.
  * Varma and Ré (2018) Varma, P.; and Ré, C. 2018.  Snuba: Automating weak supervision to label training data.  In _Proceedings of the VLDB Endowment. International Conference on Very Large Data Bases_ , volume 12, 223. NIH Public Access.
  * Wang et al. (2017) Wang, X.; Peng, Y.; Lu, L.; Lu, Z.; Bagheri, M.; and Summers, R. M. 2017.  Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2097–2106.
  * Wang et al. (2019) Wang, Y.; Ma, X.; Chen, Z.; Luo, Y.; Yi, J.; and Bailey, J. 2019.  Symmetric cross entropy for robust learning with noisy labels.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , 322–330.
  * Wei, Iyer, and Bilmes (2015) Wei, K.; Iyer, R.; and Bilmes, J. 2015.  Submodularity in data subset selection and active learning.  In _ICML_ , 1954–1963. PMLR.
  * Wu et al. (2023) Wu, R.; Chen, S.-E.; Zhang, J.; and Chu, X. 2023.  Learning Hyper Label Model for Programmatic Weak Supervision.  In _ICLR_.
  * Xian et al. (2018) Xian, Y.; Lampert, C. H.; Schiele, B.; and Akata, Z. 2018.  Zero-shot learning—a comprehensive evaluation of the good, the bad and the ugly.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(9): 2251–2265.
  * Xiao, Rasul, and Vollgraf (2017) Xiao, H.; Rasul, K.; and Vollgraf, R. 2017.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_.
  * Xiong, Mehta, and Singh (2019) Xiong, Y.; Mehta, R.; and Singh, V. 2019.  Resource constrained neural network architecture search: Will a submodularity assumption help?  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 1901–1910.
  * Yu et al. (2015) Yu, F.; Seff, A.; Zhang, Y.; Song, S.; Funkhouser, T.; and Xiao, J. 2015.  Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop.  _arXiv preprint arXiv:1506.03365_.
  * Yu, Ding, and Bach (2022) Yu, P.; Ding, T.; and Bach, S. H. 2022.  Learning from Multiple Noisy Partial Labelers.  In _Proceedings of The 25th International Conference on Artificial Intelligence and Statistics_ , 11072–11095. PMLR.
  * Zhang et al. (2022a) Zhang, J.; Hsieh, C.-Y.; Yu, Y.; Zhang, C.; and Ratner, A. 2022a.  A survey on programmatic weak supervision.  _arXiv preprint arXiv:2202.05433_.
  * Zhang et al. (2022b) Zhang, J.; Wang, H.; Hsieh, C.-Y.; and Ratner, A. J. 2022b.  Understanding Programmatic Weak Supervision via Source-aware Influence Function.  In _NeurIPS_ , volume 35, 2862–2875.
  * Zhang et al. (2021) Zhang, J.; Yu, Y.; Li, Y.; Wang, Y.; Yang, Y.; Yang, M.; and Ratner, A. 2021.  Wrench: A comprehensive benchmark for weak supervision.  _arXiv preprint arXiv:2109.11377_.

## Supplemantary Material

### Architecture

For performance comparison, we used the DCGAN architecture (Radford, Metz, and
Chintala 2015) for 32X32 images. Our classifier network shares all the
convolutional layers with the discriminator and uses a separate, fully
connected layer to predict the associated class for a given image. Further, we
have considered a 100-dimensional latent vector for image generation and
sample the class conditional vector from a uniform distribution.

For the accuracy parameters associated with the label model(${\mathcal{A}}$),
we have used the image features extracted from our shared convolutional
network. The accuracy parameters are modeled using an MLP network with the
same output dimension as the number of label functions used for the
experiment. The network consists of three layers of feed-forward network with
dimensions [256, 128, 64] and uses a ReLU activation for intermediate layers
and Sigmoid for the final prediction layer. The fully connected layer
${\mathcal{F}}$ used in training the classifier consists of a simple linear
layer with softmax output.

### Training

The complete model is trained for 200 epochs using five Adam optimizers, one
for each loss function that comprises adversarial loss for the generator,
adversarial loss for the discriminator, loss associated with the label model,
classifier loss, and log-likelihood maximization-based loss for the generator.
The learning rate associated with each of these optimizers are
$1\text{x}10^{-4}$, $4\text{x}10^{-4}$ , $8\text{x}10^{-5}$,
$1.8\text{x}10^{-4}$, $1\text{x}10^{-5}$ respectively. We have used a weight
of 0.7 for cross entropy and 0.3 for reverse cross-entropy loss for the soft
cross-entropy loss. In our subset selection-based experiment, we normalized
the entropy to ensure that the total entropy $(Cost(\tilde{D}_{t}))$ equals
the cardinality of the non-abstained samples. This normalization was necessary
to facilitate the use of the apricot library for knapsack-based subset
selection. We performed a grid search for other hyper-parameters associated
with subset selection to identify the best for all the datasets. Specifically,
for entropy budget ($\eta$), we considered a range of values such as [0.7,
0.8, 0.9], and for the trade-off between representativeness and diversity in
generalized graph cut ($\gamma$), we explored values such as [2, 2.5, 3, 3.5].
In ${\bm{l}}_{decay}$ loss for the hyper-parameter $\mu$ we have formulated it
as $\frac{{\mathcal{C}}}{e*\delta+1}$ (Boecking et al. 2023), where
${\mathcal{C}}$ is the total number class in the data, $e$ is the current
epoch of training and $\delta$ is set to be 1 for experiments across all
datasets. Further, the label model and classifier were trained using an
appropriate stop gradient operator on the classifier’s prediction and pseudo
labels, respectively. The pseudo labels for the training of the classifier are
updated after every epoch.

## Algorithm

In Algorithm - 1, we describe our proposed method for training the label model
and the conditional GAN.

Algorithm 1 Pseudo code of our proposed method.

Input training dataset $(\tilde{{\mathcal{D}}})$, Label functions ($\Lambda$),
label model (${\mathcal{L}}$), non-abstained dataset
$(\tilde{{\mathcal{D}}}_{t})$, hyper-parameter($p,\eta,\omega,{\mathcal{B}}$)

for i in [0…epoch] do

if i%p == 0 then

See (Eq. 10) for CEG Formulation.

${\mathcal{S}}_{cost}$ =
CEG($\tilde{{\mathcal{D}}_{t}},{\mathcal{B}},c=entropy\text{ }function(.)$) #
final set selected using entropy-based cost function (Eq. 9).

${\mathcal{S}}_{uniform}$ = CEG($\tilde{D_{t}},{\mathcal{B}},c=1$) # final set
selected using uniform cost.

$\tilde{{\mathcal{D}}}_{0}=\operatorname*{arg\,max}_{{\mathcal{S}}}({\mathcal{F}}({\mathcal{S}}_{uniform}),{\mathcal{F}}({\mathcal{S}}_{Cost}))))$
# select the set with maximum utility (Eq. 7).

$\hat{y}=\operatorname*{arg\,max}{\mathcal{L}}(\Lambda(x_{i});\theta_{{\mathcal{A}}}^{i})$
where $x_{i}\in\tilde{{\mathcal{D}}}_{0}$,
$\hat{{\mathcal{Y}}}=\\{\hat{y_{1}}\ldots\hat{y}_{\tilde{|{\mathcal{D}}_{0}|}}\\}$
# pseudo labels generated by label model (Eq. 4).

for b in training batches : do

$\theta_{{\mathcal{G}}}^{i+1}=\theta_{{\mathcal{G}}}^{i}-\omega_{0}\nabla({\bm{l}}_{gan}({\mathcal{G}},{\mathcal{D}},{\mathcal{C}};\theta_{{\mathcal{G}}}^{i},\theta_{{\mathcal{D}}}^{i},\theta_{{\mathcal{C}}}^{i}))$
# training the parameters of generator (Eq. 2).

$\theta_{{\mathcal{D}}}^{i+1}=\theta_{{\mathcal{D}}}^{i}+\omega_{1}\nabla({\bm{l}}_{gan}({\mathcal{G}},{\mathcal{D}},{\mathcal{C}};\theta_{{\mathcal{G}}}^{i},\theta_{{\mathcal{D}}}^{i},\theta_{{\mathcal{C}}}^{i}))$
# training the parameters of discriminator (Eq. 2).

$\theta_{{\mathcal{A}}}^{i+1}=\theta_{{\mathcal{A}}}^{i}-\omega_{2}\nabla({\bm{l}}_{lm}({\mathcal{A}};\theta_{{\mathcal{A}}}^{i}))$
# training the parameters of label model (Eq. 5, 6).

$\theta_{{\mathcal{C}}}^{i+1}=\theta_{{\mathcal{C}}}^{i}-\omega_{3}\nabla({\bm{l}}_{sce}(\tilde{{\mathcal{D}}}_{0},\hat{{\mathcal{Y}}};\theta_{\mathcal{C}}^{i}))$
# training the parameters of classifier(Eq. 1).

### Subset selection of data under the knapsack constraint

###### Proposition 1

(Leskovec et al. 2007; Xiong, Mehta, and Singh 2019) If
$\mathcal{{\mathcal{F}}}$ is a non decreasing submodular function with
${\mathcal{F}}\big{(}\phi\big{)}$ = 0 then CEG algorithm achieves a constant
ratio of $\frac{1}{2}\big{(}1-\frac{1}{e}\big{)}$ to the optimal solution
under the knapsack constraints.

$\displaystyle\max\big{(}{\mathcal{F}}({{\mathcal{S}}}_{uniform}),{\mathcal{F}}({{\mathcal{S}}}_{cost})\big{)}\geq\frac{1}{2}\big{(}1-\frac{1}{e}\big{)}\max_{Cost({\mathcal{S}})\leq{\mathcal{B}}}{\mathcal{F}}({\mathcal{S}})$

In our setting, ${\mathcal{F}}({{\mathcal{S}}}_{uniform}$) and
${\mathcal{F}}({{\mathcal{S}}}_{cost}$) are the final subsets selected using
the cost-effective greedy algorithm, and ${\mathcal{F}}$ is the generalized
graph cut algorithm.

## Network Ablation using StyleGAN-ADA

To show the compatibility of our proposed method with different architectures,
we trained styleGAN-ADA (Karras et al. 2020) on CIFAR10 images using self-
supervised based label functions (CIFAR10-B). The implementation uses the
official StyleGAN-ADA codebase with a setup similar to previous work (Boecking
et al. 2023). However, we have used a depth mapping of 7 and an embedding size
of 150 for all the experiments related to CIFAR10.

Further, we performed these experiments on a single GPU because of limited
computing resources. For CIFAR10-B, we run the model until the discriminator
has seen around 19M real images. For the setting where the complete set of
non-abstained samples ($\tilde{{\mathcal{D}}_{0}}=\tilde{{\mathcal{D}}_{t}}$)
was used for the training of the classifier, we achieved an FID of 4.96 and an
accuracy of 73.31% for label model. On the other hand, using the subset
selection method, we achieved an FID of 4.94 and an accuracy of 73.18%. Figure
- 3 shows the images generated by our model.

Figure 3: Images generated by our method using StyleGAN-ADA based architecture
on CIFAR10-B. Figure 4: Images generated by our method using StyleGAN-ADA
based architecture on LSUN.

## Results on High-Resolution Images

To demonstrate the effectiveness of our method on high-resolution images, we
experimented with LSUN (Yu et al. 2015) (256x256, 10 class, self-supervised
learning based LFs) and AFHQ (Choi et al. 2020) (256x256, 3 class, synthetic
LFs) datasets containing 100000 and 14629 samples, respectively. The LFs for
LSUN are randomly sampled from the original ssl-based LFs provided by Boecking
et al. (2023). The styleGAN-ADA-based (Karras et al. 2020) network was
employed for both of these datasets. Results in Table-4 demonstrate the
efficacy of our method on generation quality and label model performance on a
high-resolution dataset.

| WSGAN | Ours(comp) | Ours(sub)
---|---|---|---
Dataset | FID | Accuracy | FID | Accuracy | FID | Accuracy
LSUN | 14.47 | 0.765 | 17.42 | 0.773 | 14.24 | 0.777
AFHQ | 24.72 | 0.823 | 24.31 | 0.815 | 23.13 | 0.815

Table 4: FID & train posterior accuracy for high-resolution images

## Sensitivity of subset selection based hyper-parameters on the performance
of the model

To better understand the impact of different hyper-parameters on the
performance of the label model and the quality of images generated by the GAN.
We conducted a sensitivity test on the hyper-parameters, such as the budget
ratio on knapsack constraint($\eta$) and the tradeoff between
representativeness and diversity($\gamma$) of the selected samples in
generalized graph cut. For these experiments, we have used optimal values for
the remaining hyper-parameters.

### Sensitivity test on knapsack constraint ($\eta$)

In the given experiment, we fixed the value of $\gamma$ equal to 3.0 and
varied the hyper-parameter associated with the knapsack constraint ($\eta$)
between a range of 0.2 to 0.8 with 0.2 as the step size. Table - 5 shows the
result of the given experiment on the quality of images generated by the
conditional GAN. Similarly, Table - 6 shows the performance of the label model
on a non-abstained dataset.

For the FID score, we found that for datasets like GTSRB, MNIST, and
CIFAR10-B, the best performance was generated for a smaller subset of data.
However, for AWA2, CIFAR10-A, and FashionMNIST, a bigger data set ($\eta=0.8$)
generates the best results. For posterior accuracy, we found that for datasets
like AWA2, CIFAR10-A, CIFAR10-B, and FashionMNIST, the model’s performance
increases as the subset size increases. However, the performance of a smaller
subset of data is still comparable to the best results generated during the
experiment.

$\eta$ | AWA2 | CIFAR10-A | CIFAR10-B | DomainNet | FashionMNIST | GTSRB | MNIST
---|---|---|---|---|---|---|---
0.2 | 29.207 | 20.696 | 19.972 | 40.690 | 13.686 | 61.260 | 4.202
0.4 | 29.683 | 20.175 | 19.702 | 40.861 | 13.080 | 56.443 | 2.732
0.6 | 29.744 | 19.606 | 19.000 | 41.250 | 12.542 | 60.255 | 4.495
0.8 | 28.956 | 18.331 | 19.578 | 37.709 | 11.747 | 59.590 | 4.838

Table 5: Sensitivity test on the hyper-parameter $\eta$ over FID. This
experiment illustrates the impact of selecting subsets of different sizes on
the quality of images generated by GAN. The best score for a dataset is
highlighted in bold.

$\eta$ | AWA2 | CIFAR10-A | CIFAR10-B | DomainNet | FashionMNIST | GTSRB | MNIST
---|---|---|---|---|---|---|---
0.2 | 0.672 | 0.873 | 0.731 | 0.633 | 0.739 | 0.824 | 0.815
0.4 | 0.684 | 0.883 | 0.735 | 0.640 | 0.750 | 0.827 | 0.820
0.6 | 0.693 | 0.887 | 0.738 | 0.647 | 0.752 | 0.829 | 0.822
0.8 | 0.698 | 0.889 | 0.739 | 0.644 | 0.753 | 0.828 | 0.820

Table 6: Sensitivity test on the hyper-parameter $\eta$ over average posterior
accuracy. This experiment illustrates the impact of selecting subsets of
different sizes on the average posterior accuracy of the label models for
samples with at least one vote from the label function. The best score for a
specific dataset is highlighted in bold.

$\gamma$ | AWA2 | CIFAR10-A | CIFAR10-B | DomainNet | FashionMNIST | GTSRB | MNIST
---|---|---|---|---|---|---|---
2.0 | 25.439 | 20.220 | 18.499 | 37.368 | 10.533 | 55.710 | 4.323
2.5 | 26.061 | 19.913 | 18.159 | 36.561 | 11.293 | 61.622 | 6.013
3.0 | 26.016 | 17.502 | 18.938 | 36.008 | 11.293 | 56.494 | 3.494
3.5 | 23.163 | 18.933 | 18.478 | 36.456 | 11.971 | 57.574 | 6.264

Table 7: Sensitivity test on the hyper-parameter $\gamma$ over FID. This
experiment illustrates the impact of the trade-off between representativeness
and diversity of the selected subset on the quality of images generated by
GAN. The best score for a dataset is highlighted in bold.

$\gamma$ | AWA2 | CIFAR10-A | CIFAR10-B | DomainNet | FashionMNIST | GTSRB | MNIST
---|---|---|---|---|---|---|---
2.0 | 0.702 | 0.888 | 0.740 | 0.652 | 0.754 | 0.828 | 0.821
2.5 | 0.702 | 0.888 | 0.740 | 0.655 | 0.754 | 0.829 | 0.821
3.0 | 0.703 | 0.888 | 0.740 | 0.651 | 0.754 | 0.828 | 0.821
3.5 | 0.703 | 0.889 | 0.741 | 0.650 | 0.753 | 0.829 | 0.820

Table 8: Sensitivity test on the hyper-parameter $\gamma$ over average
posterior accuracy. This experiment illustrates the impact of selecting
representative and diverse subsets on the average posterior accuracy of the
label models for samples with at least one vote from the label function. The
best score for a dataset is highlighted in bold.

### Sensitivity test on generalized graph cut($\gamma$)

We have further conducted a sensitivity test on the hyper-parameter $\gamma$
to understand the impact of the tradeoff on representativeness and diversity
for selected samples on the performance of the model. For this experiment, the
entropy budget ratio ($\eta$) was fixed at 0.8, and the hyperparameter
$\gamma$ was varied between 2 and 3.5 at a step size of 0.5. Table-7 and
Table-8 illustrate the impact of the given hyper-parameter on the performance
of conditional GAN and label model.

We can clearly see the impact of the given hyperparameter on the FID score.
However, the label model’s performance remains relatively stable with a change
in these values.

## Comparison of baseline on different metrics

Table - 9, Table-10 and Table-11 show the F1-score, Recall, and Precision
achieved by our methods in comparison to other label models. The score clearly
illustrates that our methods generate better performance than other baselines.

Dataset | MV | MeTaL | FS | Snorkel | DS | WSGAN | Ours(comp) | Ours(sub)
---|---|---|---|---|---|---|---|---
AWA2 | 0.609 | 0.569 | 0.596 | 0.581 | - | 0.652 | 0.683 | 0.678
CIFAR10-A | 0.828 | 0.798 | 0.795 | 0.798 | 0.850 | 0.870 | 0.887 | 0.887
CIFAR10-B | 0.715 | 0.703 | 0.703 | 0.704 | 0.672 | 0.725 | 0.736 | 0.735
DomainNet | 0.581 | 0.446 | 0.622 | 0.436 | 0.655 | 0.632 | 0.642 | 0.641
FashionMNIST | 0.704 | 0.698 | 0.705 | 0.703 | 0.691 | 0.715 | 0.724 | 0.724
GTSRB | 0.802 | 0.800 | 0.628 | 0.799 | 0.616 | 0.811 | 0.814 | 0.814
MNIST | 0.759 | 0.738 | 0.755 | 0.744 | 0.716 | 0.798 | 0.800 | 0.800

Table 9: Comparison between the F1 scores of the label models for samples with
at least one vote from the label function. The best performer across methods
is highlighted in bold.

Dataset | MV | MeTaL | FS | Snorkel | DS | WSGAN | Ours(comp) | Ours(sub)
---|---|---|---|---|---|---|---|---
AWA2 | 0.627 | 0.619 | 0.621 | 0.626 | - | 0.670 | 0.701 | 0.696
CIFAR10-A | 0.831 | 0.804 | 0.800 | 0.804 | - | 0.872 | 0.890 | 0.889
CIFAR10-B | 0.719 | 0.708 | 0.708 | 0.709 | - | 0.729 | 0.740 | 0.739
DomainNet | 0.595 | 0.485 | 0.635 | 0.478 | - | 0.642 | 0.650 | 0.649
FashionMNIST | 0.735 | 0.730 | 0.734 | 0.734 | - | 0.744 | 0.754 | 0.754
GTSRB | - | - | - | - | - | 0.825 | 0.828 | 0.828
MNIST | 0.778 | 0.759 | 0.773 | 0.764 | - | 0.816 | 0.818 | 0.818

Table 10: Comparison between the Recall scores of the label models for samples
with at least one vote from the label function. The best performer across
methods is highlighted in bold.

Dataset | MV | MeTaL | FS | Snorkel | DS | WSGAN | Ours(comp) | Ours(sub)
---|---|---|---|---|---|---|---|---
AWA2 | 0.644 | 0.637 | 0.650 | 0.629 | - | 0.674 | 0.709 | 0.705
CIFAR10-A | 0.839 | 0.823 | 0.819 | 0.821 | 0.866 | 0.875 | 0.892 | 0.892
CIFAR10-B | 0.738 | 0.736 | 0.736 | 0.737 | 0.659 | 0.746 | 0.756 | 0.755
DomainNet | 0.667 | 0.700 | 0.676 | 0.697 | 0.702 | 0.682 | 0.694 | 0.692
FashionMNIST | 0.687 | 0.688 | 0.690 | 0.711 | 0.712 | 0.699 | 0.707 | 0.707
GTSRB | 0.718 | 0.761 | 0.714 | 0.772 | 0.731 | 0.809 | 0.812 | 0.811
MNIST | 0.753 | 0.744 | 0.749 | 0.742 | 0.772 | 0.787 | 0.789 | 0.789

Table 11: Comparison between the Precision scores of the label models for
samples with at least one vote from the label function. The best performer
across methods is highlighted in bold.

## Generated Images

This section presents the images generated through our approach with a DCGAN
architecture. For each dataset, five classes were randomly chosen, and for
each of the classes, ten randomly chosen images were shown. For AWA2
(Figure-9) and DomainNet(Figure-10) the complete training process uses around
7000 samples, which makes the generation of images relatively harder in
comparison to other datasets.

Figure 5: CIFAR10 images generated by DCGAN that utilizes our proposed method.
Figure 6: GTSRB images generated by DCGAN that utilizes our proposed method.
Figure 7: FashionMNIST images generated by DCGAN that utilizes our proposed
method. Figure 8: MNIST images generated by DCGAN that utilizes our proposed
method. Figure 9: AWA2 images generated by DCGAN that utilizes our proposed
method. It is to be noted that the generation of images is challenging due to
the small size of our dataset (around 7000 images). Figure 10: DomainNet
images generated by DCGAN that utilizes our proposed method. It is to be noted
that the generation of images is challenging due to the small size of our
dataset (around 7000 images).