# Split-PU: Hardness-aware Training Strategy for Positive-Unlabeled Learning

Chengming Xu  School of Data Science, Fudan University China
<EMAIL_ADDRESS>,  Chen Liu  Department of Mathematics Hong Kong
University of Science and Technology China<EMAIL_ADDRESS>,  Siqian
Yang  ,  Yabiao Wang  Tencent Youtu Lab China<EMAIL_ADDRESS><EMAIL_ADDRESS>,  Shijie Zhang  School of Data Science, Fudan
University China<EMAIL_ADDRESS>,  Lijie Jia  Shanghai Jiaotong
University China<EMAIL_ADDRESS>and  Yanwei Fu  School of Data Science,
Fudan University China<EMAIL_ADDRESS>
(2022)

###### Abstract.

Positive-Unlabeled (PU) learning aims to learn a model with rare positive
samples and abundant unlabeled samples. Compared with classical binary
classification, the task of PU learning is much more challenging due to the
existence of many incompletely-annotated data instances. Since only part of
the most confident positive samples are available and evidence is not enough
to categorize the rest samples, many of these unlabeled data may also be the
positive samples. Research on this topic is particularly useful and essential
to many real-world tasks which demand very expensive labelling cost. For
example, the recognition tasks in disease diagnosis, recommendation system and
satellite image recognition may only have few positive samples that can be
annotated by the experts. While this problem is receiving increasing
attention, most of the efforts have been dedicated to the design of
trustworthy risk estimators such as uPU (Du Plessis et al., 2014) and nnPU
(Kiryo et al., 2017) and direct knowledge distillation, e.g., Self-PU (Chen et
al., 2020a). These methods mainly omit the intrinsic hardness of some
unlabeled data, which can result in sub-optimal performance as a consequence
of fitting the easy noisy data and not sufficiently utilizing the hard data.
In this paper, we focus on improving the commonly-used nnPU (Kiryo et al.,
2017) with a novel training pipeline. We highlight the intrinsic difference of
hardness of samples in the dataset and the proper learning strategies for easy
and hard data. By considering this fact, we propose first splitting the
unlabeled dataset with an early-stop strategy. The samples that have
inconsistent predictions between the temporary and base model are considered
as hard samples. Then the model utilizes a noise-tolerant Jensen-Shannon
divergence loss for easy data; and a dual-source consistency regularization
for hard data which includes a cross-consistency between student and base
model for low-level features and self-consistency for high-level features and
predictions, respectively. Our method achieves much better results compared
with existing methods on CIFAR10 and two medical datasets of liver cancer
survival time prediction, and low blood pressure diagnosis of pregnant,
individually. The experimental results validates the efficacy of our proposed
method. Codes and models are available at
https://github.com/loadder/SplitPU_MM2022.

Positive-Unlabeled learning; Consistency Regularization

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Multimedia; October 10–14, 2022; Lisboa,
Portugal††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisboa, Portugal††price: 15.00††doi:
10.1145/3503161.3548045††isbn: 978-1-4503-9203-7/22/10††ccs: Computing
methodologies††ccs: Computing methodologies Artificial intelligence††ccs:
Computing methodologies Computer vision††ccs: Computing methodologies Computer
vision problems

##  1\. Introduction

The deep models of classical recognition task, especially binary
classification, has been supported by plenty of positive and negative samples,
which are comprehensively annotated in general. On the other hand, the
collection of training data could not always be perfect. In many Multimedia
applications, only part of the positive samples can be annotated, while the
other samples including both positive and negative ones can only be left
without labels. These cases may happen quite often. For example, the annotator
source is severely limited for tasks, such as military targets recognition
from satellite images (Fan et al., 2020), since expertise and confidential
knowledge is required. In medical image analysis such as liver cancer
diagnosis, the annotators can only provide confident judgement to few positive
samples, while the other images without obvious symptom could also be positive
but cannot be explicitly annotated. Essentially, the Positive-Unlabeled (PU)
learning is studied to handle these tasks. With a well-trained PU learning
model, many real-world applications of different modalities with only positive
and unlabeled data can thus be effectively solved, e.g. medical diagnosis,
recommendation system, satellite image recognition, etc.

Figure 1.  The binary classification versus PU learning. Left: The most
commonly-seen visual data with distinguished categories. The data can be
easily identified as an animal or not animal. Right: Data for low blood
pressure diagnosis. Without sufficient expertise annotators, only part of the
most confident positive samples can be recognized. Therefore the data in such
an area is often taken as PU data.

Due to the special structure of training set, PU learning is generally more
challenging than standard supervised binary classification. Directly applying
existing methods in both fully-supervised and semi-supervised learning would
lead to great performance degradation. The classical PU learning methods can
be traced back to confident negative sample selection(Liu et al., 2002, 2003)
in the early stages, and focus on designing unbiased risk estimator (Du
Plessis et al., 2014; Kiryo et al., 2017) recently. While some previous works
like Self-PU (Chen et al., 2020a) and PUUPL (Dorigatti et al., 2022) have
explored combining together the well-designed objective function and sample
selection for better performance, these methods neglect an important fact in
the PU data: Some samples are relatively much harder to learn by deep neural
networks than most data.

Such hardness is not similar to the definition of ’easiness’ depicted in Self-
PU which is measured by the confidence of predictions. It is the intrinsic
property of image data that we focus on, which could be resulted from many
different reasons. For example, for commonly-used image datasets, those images
with extremely small, distorted or camouflaged objects can be relatively
harder to learn than other normal one. Similarly, in medical image analysis,
some samples may have severe artifacts, affecting training and inference of
models. These artifacts may be caused by the errors of medical instruments,
which can hardly be avoided in all cases.

By considering this fact, we advocate that the unlabeled data should be
further divided into easy and hard set. The easy data dominates the unlabeled
set, being easy to fit but vulnerable to noise, while the hard data is on the
opposite. Specifically, inspired by Self-PU, we can annotate these unlabeled
samples with pseudo-labels generated by a base model, e.g. a nnPU model. Then
both sets contain wrongly-labeled (noisy) data and correctly-labeled (clean)
data. We highlight two important points which imply that these different types
of data serve as different roles in training. (1) Easy data is not totally
reliable. Especially, since the easy noisy samples can be fitted by the model
facilely, it is more likely for these samples than the hard noisy ones to hurt
the generalization ability of the learned PU models. Thus potentially, it may
be questionable to directly utilize all pseudo-labels from the easy set in a
vanilla objective functions like cross entropy losses. (2) Hard data is more
difficult to deal with and requires objective functions that exclude the
pseudo-labels. As the name of hard data suggests, the model can hardly learn
information from the pseudo-labels of hard data no matter whether they are
clean or not. Nevertheless, the hard clean samples are essentially valuable
for estimating the decision boundary. For example, as a common rule of the
thumb, the hard and ambiguous training instances may also be taken as support
vectors to compute the classifier margins in support vector machine.
Therefore, it is a problem to make use of the hard data in a more effective
way. Given these two points, the naïve selection-distillation pipeline as in
Self-PU, i.e. gradually selecting confident samples and training with their
pseudo-labels, may lead to sub-optimal performance, as all unlabeled samples
are treated with the same form of objective functions.

To this end, we propose a novel Split-PU method that employs a novel hardness-
aware training strategy for positive-unlabeled learning. Our Split-PU method
takes apart the easy and hard data from all PU data to deal with,
respectively. Specifically, the whole method contains four steps. (1) We first
train a base teacher model of nnPU (Kiryo et al., 2017) as objective function.
The base model is applied to generate the initial pseudo-labels for all
unlabeled data. (2) Then a temporary student model is trained via knowledge
distillation from the teacher model, with recording training accuracy of each
epoch. When the temporary model reaches the specified performance, we split
out those training samples as hard data that have the inconsistent predictions
between the temporary model and the base one, with the rest as easy data. (3)
Further, different learning strategies are employed to handle the easy and
hard data. For easy data, we find out that those easy noisy samples cannot be
detected as effectively as the hard noisy samples by using existing noise
detection methods. As an alternative, we thus for the first time introduce the
Jensen-Shannon divergence loss (Englesson and Azizpour, 2021) to optimize the
PU learning model for noisy-robust PU learning. For hard data, we develop a
dual-source consistency regularization which covers low-level and high-level
features together with predictions, thus fully utilizing the supervision from
both the base and the student model. We have to regularize such consistency
here: a cross-consistency between the low-level features of base model and
student model, which is aimed to guide the student model with well-trained
low-level features like edges and colors from base model; and a self-
consistency on the high-level features and predictions of student model
between different views of each image, which is used to improve the stability
of training for better visual representations. With the above training
strategy, we can better handle the unlabeled data with the pseudo-labels
predicted by the base model, leading to a student model with better
performance. (4) Finally, we replace the base model with the student model to
iterate the training algorithm for further improvement.

To validate the effectiveness of our proposed method, we conduct extensive
experiments. First, one of the most widely-used visual benchmarks CIFAR10 is
adopted, following the existing methods. Additionally, we adopt a MRI dataset
on revival time prediction for liver cancer and a myometrioum image dataset of
low blood pressure diagnosis for the pregnant to further testify the
capability of our model on real-life applications. The results on these three
datasets illustrate that our model can significantly improve the base model
trained with nnPU and surpass the state-of-the-art methods. Moreover, when
trained with fewer positive labeled data, our model still performs better than
competitors which use more labeled data, which supports the argument that our
model can solve the data scarcity problem to some extent.

In summary, our paper has the following contributions:

  * •

We analyze the essence of the PU data and find that those hard data could make
the existing pseudo-labeling methods less effective.

  * •

To take advantage of the analysis, we propose to use an early-stop splitting
strategy to excavate the easy and hard data from the unlabeled set.

  * •

We leverage the Jensen-Shannon divergence loss to learn the easy data, and
propose a novel dual-source consistency regularization to efficiently utilize
the hard unlabeled data.

  * •

With the help of our proposed techniques, we consistently improve the base
model among different datasets when using various amount of labeled samples.

In the following context, we first briefly introduce the related works in Sec.
2, which includes PU learning, learning with noisy labels and consistency in
deep learning. Then in Sec. 3 we give the problem formulation of PU learning
and our proposed method. Experiment results among multiple datasets are listed
in Sec. A.

##  2\. Related Works

Positive-unlabeled Learning Positive-unlabeled learning is a surging research
topic that aims to learn patterns on a training set with only few labeled
positive data and sufficient unlabeled data. The research on PU learning can
be dated back to (Liu et al., 2002, 2003; Yu et al., 2002) in which reliable
negative samples are selected and used to train a set of classifiers. The
recent effort on PU learning with deep deep learning can be generally
categorized into two buckets. One is about the design of objective function
that can unbiasedly estimate the risk. uPU (Du Plessis et al., 2014) shows
that a cost-sensitive classifier can be used by reform the risk in original
classification, with known class prior. nnPU (Kiryo et al., 2017) points out
the tendency of overfitting complex models when using uPU. As a solution they
impose a non-negative constraint on the objective function of uPU, which leads
to better generalization. Some other papers follow these two works to find
powerful objective functions in different settings such as imbalanced data (Su
et al., 2021), biased negative data (Hsieh et al., 2019), data selection bias
(Kato et al., 2018), etc. Some other works focus on sample selection from the
unlabeled samples based on the former methods. For example, (Xu et al., 2019)
analyze the reasonability of only selecting positive data, Self-PU (Chen et
al., 2020a) leverages self-paced learning to gradually update the base model
with newly-learned knowledge, PUUPL (Dorigatti et al., 2022) strengthens the
calibration of base model so that the pseudo-labels are all uncertainty-aware.
Note that the concept of hardness has also been mentioned in Self-PU. However,
they refer to the this concept as the reliability of pseudo-labels, which is
reflected by the confidence of model predictions. Our method split the data
into easy and hard set based on the intrinsic property of data, which is
different with the existing ideas. Moreover, we point out that directly using
cross entropy on pseudo-labeled data and PU loss on unlabeled data is
inappropriate. Instead, we propose to conquer the easy and hard unlabeled data
based on noise-tolerant learning and dual-source consistency regularization
respectively. Additionally, PU learning is different with semi-supervised
learning which also uses the unlabeled data as the regularizer. The unlabeled
data in PU learning plays a more important role of estimating the data
distribution since no labeled negative samples are available.

Learning with noisy labels Since the pseudo-labeling strategy in PU learning
inevitably generates wrongly-labeled samples, i.e. noisy samples, such methods
are correlated with the task of learning with noisy labels, which is aimed to
train a more robust model from the noisy dataset. Methods in learning with
noisy labels mainly belong to two groups: noise robust algorithm and noise
detection. Noise robust algorithms target designing network architectures
(Chen and Gupta, 2015; Goldberger and Ben-Reuven, 2016) or objective function
(Zhang and Sabuncu, 2018; Ghosh et al., 2017; Englesson and Azizpour, 2021)
that are robust against side effect of noisy samples. Noise detection methods
involve a pipeline for selecting and processing the noisy data from the
training set. Typical selection basis includes large error (Shen and Sanghavi,
2019), incoherent gradient (Chatterjee, 2020), inconsistency among different
networks or different views of the same input (Yu et al., 2019; Zhou et al.,
2020) and so on. As for the strategies for handling noisy samples, label-
correction modules are proposed in many works (Li et al., 2017; Tanaka et al.,
2018; Vahdat, 2017). We in this paper leverage the technique in this topic to
solve PU learning, which has never been studied in the previous works.
Moreover, while directly applying these advantaged noise detection methods may
also help dealing with the noisy pseudo-labels in PU learning, we will show in
the experiments that these methods could easily fail when facing with some of
the training data.

Consistency Regularization Recently consistency regularization has been widely
used in deep learning as extra supervision besides the annotations.
Specifically, contrastive learning methods (He et al., 2020; Chen et al.,
2020b; Caron et al., 2020; Chen and He, 2021a) take as objective function the
similarity among features of different views of the same image, which can be
seen as the feature level consistency. Such objectives are utilized to enhance
the instance discriminative ability for networks. Moreover, for semi-
supervised learning, many works (Sohn et al., 2020; Verma et al., 2019; Jeong
et al., 2019) adopt prediction level consistency to increase the stability of
training. Compared with these methods, our proposed dual-source consistency
regularization takes into account the various properties of features and
predictions, which leads to consistency with more capacity from multiple
sources.

##  3\. Methodology

Problem Formulation Suppose we have a data distribution $\mathcal{T}$ which
can be represented as a joint distribution of image distribution $\mathcal{I}$
and label distribution $\mathcal{Y}$, i.e.,
$\mathcal{T}=\mathcal{I}\times\mathcal{Y}$. In PU learning, the training set
$\mathcal{T}^{train}$ is composed of two parts including $\mathcal{T}^{p}$
which only contains few labeled positive samples and $\mathcal{T}^{u}$ which
contains abundant unlabeled samples. The labeled positive
$\mathcal{T}^{p}=\\{(I_{i}^{p},y_{i}^{p})\\}_{i=1}^{n_{p}}$, where
$I_{i}^{p}\sim\mathcal{I},y_{i}^{p}=1$. The unlabeled
$\mathcal{T}^{u}=\\{I_{i}^{u}\\}_{i=1}^{n_{u}}$, where
$I_{i}^{u}\sim\mathcal{I}$. Our goal is to learn a network $\phi$, which can
well generalize to the test set
$\mathcal{T}^{test}=\\{(I_{i},y_{i})\\}_{i=1}^{n_{t}}$.

Figure 2. Overview of our model. We first train a base teacher model using
nnPU as objective function. Then the unlabeled data is split into easy and
hard set via a temporary model which is trained with knowledge distillation
given the base model. Then we iterate a new training process applying
different learning strategies to the easy and hard data to train the student
model.

Overview. We propose a hardness-based splitting strategy to deal with the PU
learning, which is schematically shown in Fig. 2. Concretely, we first train a
base model $\phi_{base}$ with nnPU (Kiryo et al., 2017) (Sec. 3.1), from which
all of the following supervision is generated. After attaining this base
model, we split the unlabeled set $\mathcal{T}^{u}$ into easy set
$\mathcal{T}^{e}$ and hard set $\mathcal{T}^{h}$ by the training accuracy of a
temporary student network $\phi_{tmp}$ which is trained with naive knowledge
distillation from $\phi_{base}$ (Sec. 3.2). Then we re-train a network $\phi$
with different learning strategies on $\mathcal{T}^{easy}$ and
$\mathcal{T}^{hard}$. For the easy set $\mathcal{T}^{easy}$, we utilize a
noise-tolerant objective function to make use of the pseudo-labels from
$\phi_{base}$ (Sec. 3.3). For the hard set $\mathcal{T}^{hard}$, we adopt a
dual-source consistency regularization to take advantage of the helpful
knowledge from $\phi_{base}$ without using the pseudo-labels (Sec. 3.4). Such
a learning strategy can be repeated by replacing the teacher model
$\phi_{base}$ with the newly-learned $\phi$ to further improve the performance
not only for commonly-used visual datasets, but also for PU learning settings
in real scenario.

###  3.1. Preliminary: Base Model Training

We first briefly introduce the nnPU (Kiryo et al., 2017) as our model. The
nnPU is a non-negative version of an unbiased risk estimator as following:

(1) $\displaystyle L_{nnPU}$
$\displaystyle=\frac{\pi}{n_{p}}\sum_{i=1}^{n_{p}}L(\phi(I_{i}^{p}),1)+$ (2)
$\displaystyle
max\\{0,\frac{1}{n_{u}}\sum_{i=1}^{n_{u}}L(\phi(I_{i}^{u}),-1)-\frac{\pi}{n_{u}}\sum_{i=1}^{n_{u}}L(\phi(I_{i}^{p}),-1)\\}$

where $\pi$ is the class prior of positive samples, i.e.
$\pi=P(I_{i}|y_{i}=1)$, which is supposed to be known as (Kiryo et al., 2017).
By training with $L_{nnPU}$ on $\mathcal{T}^{train}$, we can get our base
model $\phi_{base}$ and correspondingly a pseudo-labeled extension of
$\mathcal{T}^{u}$:

(3) $\displaystyle\hat{\mathcal{T}}^{u}$
$\displaystyle=\\{(I_{i}^{u},\hat{y_{i}}^{u})\\}_{i=1}^{n_{u}}$ (4)
$\displaystyle\hat{y_{i}}^{u}$ $\displaystyle=\phi_{base}(I_{i}^{u})$

Note that $\hat{y_{i}}^{u}$ can be either soft-label or hard-label. In this
way, $\hat{\mathcal{T}}^{u}$ is composed of correctly-labeled data (i.e. clean
data), and wrongly-labeled data (i.e., noisy data). A straightforward way,
like in Self-PU (Chen et al., 2020a), to make use of $\hat{\mathcal{T}}^{u}$
is to train a supervised learning model based on these pseudo-labels and then
iteratively rectify the noisy or wrongly pseudo-labeled samples. However, as
we will show in the experiments (see Fig. 8), some training samples are
intrinsically hard to learn. This implies that the models can barely receive
knowledge from these samples no matter they are affiliated with correct
pseudo-labels or not. Thus such a learning strategy will lead to sub-optimal
performance. As a solution, we propose to use the following method to separate
these hard samples from $\mathcal{T}^{u}$ and then learn the easy and hard
data respectively.

###  3.2. Early-stop Splitting Strategy

Given the base model, we consider splitting $\mathcal{T}^{u}$ apart into easy
and hard data, and then conquer them respectively. Exactly, we learn a new
student model $\phi_{tmp}$ based on the pseudo-labels generated by
$\phi_{base}$ with soft-label cross entropy as objective function:

(5) $L_{tmp}=\sum_{i=1}^{n_{u}}-\hat{y}_{i}^{u}log(\phi_{tmp}(I_{i}^{u}))$

We set a small learning rate for this optimization process to track the
variety of $\phi_{tmp}$ in detail. After each epoch, we record the training
accuracy. Once the accuracy is over a threshold $\tau$, we stop the training
and split the unlabeled set $\hat{\mathcal{T}}^{u}$ based on the prediction
coherence between $\phi_{tmp}$ and $\phi_{base}$. The samples that are still
not learned by $\phi_{tmp}$ are categorized as hard samples, which compose the
hard set $\mathcal{T}^{hard}$. The rest are called easy set, denoted as
$\mathcal{T}^{easy}$. Since the noisy samples generally have different
patterns from clean samples as in existing works (Chatterjee, 2020; Arpit et
al., 2017), most of them would be learned more slowly than clean ones. As a
consequence, $\mathcal{T}^{hard}$ can have a much larger proportion of noisy
samples than $\mathcal{T}^{u}$, and the noise rate of $\mathcal{T}^{easy}$ is
much smaller. Meanwhile, the roles of these two subsets are totally different:
(1) The easy clean samples take up the most part of $\mathcal{T}^{u}$, from
which the model learns its patterns. (2) The easy noisy samples, despite with
a small sample size, can be easily fitted by the model and hurt the
generalization ability. (3) The hard samples, while can hardly be learned by
$\phi_{tmp}$ and have high noise rate, can regularize the model for better
performance. Especially, the hard clean samples can provide data knowledge
which is not contained in the easy set. Therefore, it is essential to utilize
a learning strategy robust to noise on $\mathcal{T}^{easy}$ and a new
supervision other than pseudo-labels on $\mathcal{T}^{hard}$. To this end, we
propose the following noise-tolerant learning on easy samples and dual-source
consistency regularization on hard samples.

###  3.3. Noise-tolerant Learning on Easy Samples

As mentioned above, the easy set $\mathcal{T}^{easy}$ has two noticeable
properties: (1) The noise rate is much smaller, thus most pseudo-labels are
reliable. (2) The negative effect of easy noisy samples on generalization
ability cannot be neglected. Directly learning the easy samples with plain
objective like cross entropy will make the model be trapped by the noisy
samples. A straightforward thought is if we can use noise detection methods to
further cull the noisy easy samples. However, due to the fact that the slowly-
learned hard noisy samples have already been taken out from
$\mathcal{T}^{easy}$, most methods based on the difference of fitting speed or
fitting performance between noisy and clean data could fail in our case, as we
will show in experiments. As an alternative solution, we adopt a noise-robust
objective function to process the easy data, which was proposed in (Englesson
and Azizpour, 2021). Concretely, for each easy sample
$I_{i}^{easy}\in\mathcal{T}^{easy}$ and its corresponding pseudo-label
$\hat{y}_{i}$, the following objective function is used:

(6) $L_{easy}=\frac{(\rho
D_{KL}(\phi(I_{i}^{easy})\|m)+(1-\rho)D_{KL}(\hat{y}_{i}\|m))}{-(1-\rho)log(1-\rho)}$

where $m=\rho\phi(I_{i}^{easy})+(1-\rho)\hat{y}_{i}$ which is an interpolation
between the pseudo-label and prediction from $\phi$, $D_{KL}$ denotes the
Kullback-Leibler divergence, $\rho$ is a hyper-parameter. Accordingly, Eq. 6
can take into account both robustness to noise and the convergence speed. As
we will show in our experiments, this term is more effective than plain
knowledge distillation using pseudo-labels.

###  3.4. Dual-source Consistency Regularization on Hard Samples

The hard set $\mathcal{T}^{hard}$ has a totally different characteristic
compared with $\mathcal{T}^{easy}$ and $\hat{\mathcal{T}}^{u}$.
$\mathcal{T}^{hard}$ contains much less samples than the other two sets, while
its proportion of noisy samples are much larger as we have explained above.
Besides, these samples can be hardly learned with the pseudo-labels, which is
reflected by the low training accuracy in the splitting phase. Therefore, it
is not proper to adopt the same training strategy as for the easy samples. To
this end, we explore to use these data via a dual-source consistency
regularization without using the pseudo-labels. Specifically, we use the
following objective function to learn the hard samples:

(7) $L_{hard}=L_{low}+L_{high}$

where $L_{low}$ is a low-level cross-consistency between $\phi_{base}$ and
$\phi$, and $L_{high}$ is a high-level self-consistency of $\phi$, which will
be introduced in the following.

Low-level cross-consistency. While the pseudo-label generated by $\phi_{base}$
is not usable due to the high noisy rate of $\mathcal{T}^{hard}$, the teacher
model can still provide some guidance of visual knowledge embedded in the low-
level features. According to the existing study (Lee et al., 2016), compared
with high-level features that is correlated with the semantic category of each
sample, these low-level features contain more basic and class-agnostic visual
patterns, such as edges and colors. Such patterns are still reusable no matter
if it leads to noisy pseudo-prediction in the $\mathcal{T}^{hard}$. Therefore,
we regularize $\phi$ with the consistency between low-level features of $\phi$
and $\phi_{base}$ on the same sample. Specifically, given a hard sample
$I_{i}^{hard}\in\mathcal{T}^{hard}$, we simultaneously extract the
corresponding features from $\phi$ and $\phi_{base}$ and compute their
similarity as the objective function:

(8) $L_{cross}=\|\phi^{1}(I_{i}^{hard})-\phi_{base}^{1}(I_{i}^{hard})\|_{2}$

where $\phi^{1}$ denotes the first convolutional layer of $\phi$, and the same
for $\phi_{base}^{1}$.

High-level self-consistency. The guidance from $\phi_{base}$ is sub-optimal
for the high-level features and the predictions. Therefore, inspired by recent
research on self-supervised learning (Chen and He, 2021b) and semi-supervised
learning (Sohn et al., 2020), we propose to utilize self-consistency on these
high-level information. Specifically, for a hard sample $I_{i}^{hard}$, we
first process it via both weak and strong augmentation pipelines
$A^{weak},A^{strong}$, which leads to two different views of this image
$\hat{I}_{i}^{hard}=A^{weak}(I_{i}^{hard}),\tilde{I}_{i}^{hard}=A^{strong}(I_{i}^{hard})$.
Then the high-level features are generated from the last convolutional layer
of $\phi$, together with the classification prediction from which the
consistency can be calculated:

(9) $\displaystyle\hat{X}_{i}$
$\displaystyle=\phi^{l}(\hat{I}_{i}^{hard}),\hat{y}_{i}=\phi(\hat{I}_{i}^{hard})$
(10) $\displaystyle\tilde{X}_{i}$
$\displaystyle=\phi^{l}(\tilde{I}_{i}^{hard}),\tilde{y}_{i}=\phi(\tilde{I}_{i}^{hard})$
(11) $\displaystyle L_{pred}$
$\displaystyle=D_{KL}(\hat{y}_{i}\|\tilde{y}_{i})$

After that a new predictor $f$ is introduced to transform the high-level
features into another embedding space. Based on that we can regularize $\phi$
with high-level feature consistency:

(12)
$L_{feat}=\frac{1}{2}D(\hat{X}_{i},f(\tilde{X}_{i}))+\frac{1}{2}D(\tilde{X}_{i},f(\hat{X}_{i}))$

where $D$ denotes the normalized negative cosine similarity. The high-level
self-consistency can then be summarized as the combination of $L_{pred}$ and
$L_{feat}$ with tunable weights $\alpha,\beta$:

(13) $L_{high}=\alpha L_{pred}+\beta L_{feat}$

This objective function can help the model with better instance discrimination
and to be more stable against disturbance on the samples. Compared with the
self-supervised consistency used in Self-PU (Chen et al., 2020a), our method
considers the intrinsically high noise-rate of hard data, thus making better
use of teacher model and leading to more effective regularization. Meanwhile,
our proposed consistency does not require training multiple networks at one
time so that we can reduce the computation cost.

Remark: why not use nnPU on $\mathcal{T}^{hard}$. One would ask if a simple
solution works for the hard samples, i.e., still taking them as unlabeled
samples and learning them by nnPU loss together with $\mathcal{T}^{p}$ as in
Eq. 1. We here provide an intuitive explanation about this problem. There are
two key components that makes $L_{nnPU}$ works: (1) The $\mathcal{T}^{u}$ can
well represent the data distribution. (2) We can have an accurate estimation
of class prior $\pi$. However, since the the sample size of
$\mathcal{T}^{hard}$ is much smaller than that of $\mathcal{T}^{u}$, there is
little chance that it can provide as much information to estimate the data
distribution as $\mathcal{T}^{u}$ can. On the other hand, if we think of
$\mathcal{T}^{p}$ and $\mathcal{T}^{hard}$ as another training set in PU
setting, then using $L_{nnPU}$ requires a new class prior $\pi^{\prime}$,
which cannot be easily estimated. Therefore it is inappropriate to use
$L_{nnPU}$ on $\mathcal{T}^{hard}$, which will be empirically verified in Sec.
A.

###  3.5. Iterative Training

With the above pipeline, we can get a $\phi$ that not only inherits the useful
knowledge from $\phi_{base}$, but also eliminate the side-effect of noisy
samples. Moreover, $\phi$ can be used as the teacher model again to train
another new student model for further improvement.

##  4\. Experiments

Figure 3. Image examples in the liver cancer dataset and myometrium dataset.

Dataset To validate the efficacy of our model, we first follow Self-PU (Chen
et al., 2020a) to conduct extensive experiments on CIFAR10 (Krizhevsky et al.,
2009). CIFAR10 is a basic visual dataset containing 50000 training images and
10000 testing images from 10 categories. We follow the setting in Self-PU to
take 4 categories of vehicles (i.e., ’airplane’, ’automobile’, ’ship’,
’truck’) to compose the positive class and the other animal classes (i.e.,
’bird’, ’cat’, ’deer’, ’dog’, ’frog’, ’horse’) for the negative class. In each
experiment, 500/1000/3000 positive samples are randomly chosen and the rest
are set as unlabeled samples. Moreover, as shown in Fig. 3, we further adopt
two new benchmarks of liver cancer data and myometrium data, which would be
publicly released. The liver cancer dataset contains MRI data of 259 patients
and their corresponding survival time. We split the dataset by setting samples
with survival time less than 90 as positive. This leads to totally 145
positive samples, from which we randomly choose 40 as labeled samples, and 114
negative samples. For test set, we have 65 negative samples and 35 positive
samples. The Myometrium data is used to diagnose the low blood pressure of the
pregnant, with 327 positive images and 233 negative images in total. We split
the dataset into 395 training images, 53 validation images and 112 testing
images, and then randomly sample 40 positive images as labeled set.

Implementation Details For CIFAR10 we use the same network architecture as in
Self-PU, i.e. a 13-layer CNN for CIFAR10. About Liver, we adopt a multi-branch
ResNet18, with the detail in supplementary. For Myometrium we finetune the
whole layers of an ImageNet-pretrained ResNet18.

For all datasets, we first train a base nnPU model for 50 epoches, using Adam
(Kingma and Ba, 2014) with learning rate of $10^{-4}$. Batch size is set to
512 for CIFAR10 and 64 for other datasets. After training the base model, we
fix the splitting of labeled and unlabeled samples and then further train
$\phi_{tmp}$ using SGD as optimizer with learning rate of $10^{-3}$ to avoid
converging to quickly. $\phi$ is trained for 100 epoches using Adam with
learning rate of $0.00005$. As for the hyper-parameters,
$\tau,\rho,\alpha,\beta$ are set as 92%, 0.7, 0.3, 0.1, respectively. We use
random cropping and flipping as weak augmentation. Color jittering is adopted
as strong augmentation for CIFAR10 and CutOut is adopted for the other two
medical datasets.

Metrics For all results, we report the mean accuracies and standard deviations
among 5 repetitions of each experiment.

Competitors We choose several former methods in PU learning as our competitors
including uPU (Du Plessis et al., 2014), nnPU (Kiryo et al., 2017), DAN (Liu
et al., 2019), Self-PU (Chen et al., 2020a), P3MIX-C (Li et al., 2021), PUUPL
(Dorigatti et al., 2022).

Model | CIFAR10
---|---
500 | 1000 | 3000
uPU (Du Plessis et al., 2014) | — | 88.00$\pm$0.62 | —
nnPU (Kiryo et al., 2017) | — | 88.60$\pm$0.40 | —
nnPU∗ | 87.22$\pm$0.46 | 89.02$\pm$0.39 | 90.53$\pm$0.36
Self-PU (Chen et al., 2020a) | — | 89.68$\pm$0.22 | 90.77$\pm$0.21
PUbN (Hsieh et al., 2019) | — | 89.30$\pm$0.57 | —
P3MIX-C (Li et al., 2021) | — | 87.90$\pm$0.50 | —
VPU (Chen et al., 2020c) | — | 85.06$\pm$0.55 | 87.50$\pm$1.05
DAN (Liu et al., 2019) | — | — | 89.70$\pm$0.40
PUUPL (Dorigatti et al., 2022) | — | 89.84$\pm$0.13 | 91.37$\pm$0.05
Ours | 89.18$\pm$0.12 | 90.51$\pm$0.10 | 92.51$\pm$0.10
Table 1.  Test accuracies and standard deviation on CIFAR10 with three positive size, * denotes results reproduced by us. Model | Myometrium | Liver
---|---|---
uPU (Du Plessis et al., 2014) | 60.18$\pm$2.33 | 40.40$\pm$5.50
nnPU (Kiryo et al., 2017) | 62.32$\pm$2.71 | 56.60$\pm$5.90
Self-PU (Chen et al., 2020a) | 59.11$\pm$1.16 | 55.60$\pm$8.17
Ours | 65.89$\pm$2.13 | 63.80$\pm$1.64
Supervised | 72.32 | 68.00
Table 2.  Test accuracies and standard deviation on Myometrium and Liver. All
results of competitors are reproduced by us.

###  4.1. Main Results

We compare our model with the competitors in Tab. 1 and Tab. 2. Note that for
nnPU on CIFAR10 we both report the results in the original paper and the one
reproduced by us.

CIFAR10 result. For CIFAR10, we train our model among three different positive
sample size. When $n_{p}=500$, the test accuracy of our model is 1.96% better
than that of nnPU, and comparable with nnPU model trained with 1000 labeled
positive samples. This means that support nnPU with our proposed method is
able to solve the problem of data scarcity for positive samples. When
$n_{p}=1000$, which is the most common setting, our proposed model attains
1.49% accuracy increasement against nnPU and performs 0.67% better than the
best counterpart PUUPL. When $n_{p}=3000$, the improvement on nnPU of our
model reaches 1.6%, leading to a much better result than all of the
competitors. Such an improvement is consistently significant among these three
settings, which validates the effectiveness of our proposed method and shows
the potential that our model can be used as a fixed suppplement for nnPU. As
for the standard deviation, our model is on par with the competitors, which
shows our method is stable enough to be safely utilized.

Myometrium and Liver results. We report results of different methods and one
oracle model trained with whole labeled training set together on the two
medical datasets in Tab. 2. For Myometrioum, our model is better than nnPU by
3.75% with comparable standard deviation. For Liver, our model improves nnPU
by 7.2% with remarkably more stability. It is noticeable that Self-PU is worse
than nnPU on these datasets. We suspect this is due to the poor performance of
nnPU leads to unreliable pseudo-labels, thus increasing the size of clean
noisy samples. In this way, since Self-PU cannot well handle such data, the
performance could be damaged.

###  4.2. Ablation Study

To further verify the efficacy of our contributions, we conduct several
ablation studies on CIFAR10, including each part of our model and the choice
of hyper-parameters.

Figure 4. (a) Size of noisy samples in easy and hard set with different $\tau$. (b) Proportion of noisy samples in easy and hard set with different $\tau$. (c) Test accuracy when trained with the easy and hard set that are generated with different $\tau$. (d) Number of clean and noisy samples that are selected from the clean set by SPR in each epoch. Early stop | Acc.
---|---
500 | 1000 | 3000
w/o | 88.57 | 89.86 | 91.73
w | 89.28 | 90.51 | 92.51
Table 3.  Test accuracies on CIFAR10 of models trained with and without early stop splitting. Loss | label type | Acc.
---|---|---
500 | 1000 | 3000
CE | hard | 87.51 | 89.02 | 91.55
soft | 88.15 | 89.48 | 92.06
DJS | hard | 88.61 | 89.58 | 92.11
soft | 89.28 | 90.51 | 92.51
Table 4.  Test accuracies on CIFAR10 of our model trained with soft cross entropy, hard cross entropy, soft Jensen-Shannon Divergence and hard Jensen-Shannon Divergence. Loss | Acc.
---|---
500 | 1000 | 3000
no | 88.52 | 89.71 | 91.46
nnPU | 87.91 | 89.43 | 90.66
self-consistency | 89.03 | 90.09 | 92.11
cross consistency | 89.17 | 90.42 | 92.27
dual-source consistency | 89.28 | 90.51 | 92.51
Table 5.  Test accuracies on CIFAR10 of our model trained with different
losses on hard data.

What can early-stop splitting do? First of all, we would like to show the
effectiveness of the early-stop splitting strategy. To do so, we conduct an
oracle experiment where all training set labels are available for indicating
which samples are wrongly labeled by $\phi_{base}$. Then we gradually increase
the accuracy threshold $\tau$ and visualize the sample size and proportion of
noisy samples that are categorized as easy and hard ones respectively. The
results are shown in Fig. 4. We can find that (1) As $\tau$ increases, less
samples can be filtered out from $\mathcal{T}^{u}$, but the noise rate of
$\mathcal{T}^{hard}$ gets larger. (2) When $\tau$ is relatively small as 70%,
the proportion of noisy sample in $\mathcal{T}^{hard}$ is twice as larger than
that of $\mathcal{T}^{u}$, which coincides with the common sense that noisy
samples are generally learned more slowly than clean samples. Also, such a
strategy can effectively reduce the noise rate of $\mathcal{T}^{easy}$. (3)
There is a balance between the threshold and the test accuracy. The benefit
from increasing $\tau$ saturates at about 92%, then the test accuracy
gradually decreases. This is reasonable since larger $\tau$ means smaller hard
set, which leads to a less representative hard set. (4) As shown in 3,
performance is worse when early-stop strategy is not used , i.e., $L_{easy}$
is applied to all unlabeled data, which indicates the efficacy of this
strategy.

Can noise detection methods fit in our case? As analyzed in Sec. 3.3, noise
detection methods may not be proper for further detecting noisy samples in
$\mathcal{T}^{easy}$. To verify this, we adopt two state-of-the-art methods in
noisy label learning, SPR (Wang et al., 2022) and CGH (Chatterjee, 2020). SPR
is based on penalized regression to build a regularization path, which is used
to detect the most confident noisy samples. Fig. 4(d) shows the result using
SPR. With about 9% noisy samples in $\mathcal{T}^{easy}$, SPR can only filter
out a set with noise rate of 10% in each epoch, which is even worse than
uniform sampling. The result of CGH which is provided in the supplementary is
also unsatisfactory, failing to detect enough noisy samples. Considering these
results, we speculate that the easy set is too challenging for these noise
detection methods to work.

Choices of objective functions for easy data. In Tab. 4 we compare the noise-
tolerant DJS loss $L_{easy}$ with cross entropy, using both hard and soft
labels. The results demonstrate that: (1) DJS loss is more effective than
cross entropy. When using soft label, DJS loss is better than cross entropy by
1.13%, 1.03% and 0.45% when using 500, 1000 and 3000 positive samples. (2)
Using soft label is better than hard label no matter which objective function
is utilized in training. We think this is due to the base model is calibrated
enough. Therefore transferring low-confidence soft labels to hard labels could
hurt the model training.

#Iter | Acc.
---|---
500 | 1000 | 3000
1 | 88.86 | 90.07 | 92.02
2 | 89.28 | 90.51 | 92.51
3 | 89.19 | 90.55 | 92.64
4 | 89.12 | 90.49 | 92.93
Table 6.  Test accuracies on CIFAR10 of our model trained with different
number of iterations.

Choices of objective functions for hard data. In Tab. 5 we compare several
choices to optimize hard data, including (1) not using hard data. (2) nnPU
loss on hard data and positive labeled data. (3) cross consistency as in Eq.
8. (4) self-consistency as in Eq. 13. (5) dual-source consistency as in Eq. 7.
The results reflect several important points: (1) Either forms of consistency
can improve the model. Specifically, the model using self-consistency is
better than the model without using hard data by 0.51%, 0.32% and 0.65% among
three settings, and the model using cross consistency is better by 0.65%,
0.71% and 0.81%. (2) Using $L_{nnPU}$ worsens the performance by 0.61%, 0.32%
and 0.80%, which is compatible with our analysis in Sec. 3.4. (3) Cross
consistency is more effective than self-consistency, which is reasonable
because cross consistency can guide the model to learn better low-level
feature. Based on such features, the model can then learn to build useful
high-level features easily.

Relationship between test accuracy and number of iterations. Since our method
utilizes an iterative training strategy, we investigate how the number of
iterations could affect the final result. Tab. 6 shows that while training for
two iterations is much better than single iteration, i.e. 0.44% improvement,
repeating the algorithm more times does not significantly boost the
performance when using 500 or 1000 positive samples. Under the setting of 3000
positive samples, training 4 iterations leads to the best result. The drawback
of using more iterations is the longer training time, which could make the
algorithm less practical and not comparable with the existing methods. Also,
since the variation caused by the randomness can stack during each iteration,
the final result with high iteration number would be more unstable.

Usage | Acc.
---|---
500 | 1000 | 3000
all data | 88.96 | 90.49 | 92.25
hard data | 89.28 | 90.51 | 92.51
Table 7.  Test accuracies on CIFAR10 of our model trained with dual-source
consistency applied to all unlabeled data or only hard data.

Usage of consistency regularization As a straightforward thought, our proposed
dual-source consistency regularization can also be applied to easy data. We
testify this idea with results in Tab. 7. By applying dual-source consistency
on all unlabeled data, the model gets accuracy of 88.96%, 90.49% and 92.25%,
receiving no marginal advancement. We think the reason is that since most
pseudo-labels of easy data is reliable and easy to fit, the supervision with
regard to semantic labels would be stronger than other forms of guidance.
Therefore, using DJS loss is enough for the easy data.

What kind of data is selected by early-stop splitting? We visualize some
positive and negative data from easy and hard set of three datasets in Fig. 8.
We can find that the easy data is discriminative enough, with obvious patterns
and symptoms. On the other side, the hard samples are generally much harder to
distinguish. For example the hard negative sample of CIFAR10 is actually an
aeroplane but also looks like a bird, and the hard positive sample in
Myometrium does not have the same symptoms like apparent blood vessels with
dark red color.

##  5\. Conclusion

In this paper we explore the task of PU learning. The problem of intrinsic
hardness of some unlabeled samples is highlighted which may affect the
existing methods’ performance. We analyze the different role of easy and hard
data in training and propose a novel training strategy as a solution to this
problem. Our model includes an early-stop splitting strategy, a noise-tolerant
learning strategy for easy data and a dual-source consistency regularization
for hard data. We achieve remarkable results on CIFAR10 and two medical
datasets, which shows the strength of our method as a promising choice for
solving PU learning both in scientific research and in real-life applications.

## References

  * (1)
  * Arpit et al. (2017) Devansh Arpit, Stanisław Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. 2017\.  A closer look at memorization in deep networks. In _International conference on machine learning_. PMLR, 233–242.
  * Caron et al. (2020) Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. 2020\.  Unsupervised learning of visual features by contrasting cluster assignments.  _Advances in Neural Information Processing Systems_ 33 (2020), 9912–9924.
  * Chatterjee (2020) Satrajit Chatterjee. 2020\.  Coherent gradients: An approach to understanding generalization in gradient descent-based optimization.  _arXiv preprint arXiv:2002.10657_ (2020).
  * Chen et al. (2020c) Hui Chen, Fangqing Liu, Yin Wang, Liyue Zhao, and Hao Wu. 2020c.  A Variational Approach for Learning from Positive and Unlabeled Data.  _Advances in Neural Information Processing Systems_ 33 (2020), 14844–14854.
  * Chen et al. (2020b) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020b.  A simple framework for contrastive learning of visual representations. In _International conference on machine learning_. PMLR, 1597–1607.
  * Chen et al. (2020a) Xuxi Chen, Wuyang Chen, Tianlong Chen, Ye Yuan, Chen Gong, Kewei Chen, and Zhangyang Wang. 2020a.  Self-pu: Self boosted and calibrated positive-unlabeled training. In _International Conference on Machine Learning_. PMLR, 1510–1519.
  * Chen and Gupta (2015) Xinlei Chen and Abhinav Gupta. 2015.  Webly supervised learning of convolutional networks. In _Proceedings of the IEEE international conference on computer vision_. 1431–1439.
  * Chen and He (2021a) Xinlei Chen and Kaiming He. 2021a.  Exploring simple siamese representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15750–15758.
  * Chen and He (2021b) Xinlei Chen and Kaiming He. 2021b.  Exploring simple siamese representation learning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 15750–15758.
  * Dorigatti et al. (2022) Emilio Dorigatti, Jann Goschenhofer, Benjamin Schubert, Mina Rezaei, and Bernd Bischl. 2022\.  Positive-Unlabeled Learning with Uncertainty-aware Pseudo-label Selection.  _arXiv preprint arXiv:2201.13192_ (2022).
  * Du Plessis et al. (2014) Marthinus C Du Plessis, Gang Niu, and Masashi Sugiyama. 2014\.  Analysis of learning from positive and unlabeled data.  _Advances in neural information processing systems_ 27 (2014).
  * Englesson and Azizpour (2021) Erik Englesson and Hossein Azizpour. 2021.  Generalized Jensen-Shannon Divergence Loss for Learning with Noisy Labels.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Fan et al. (2020) Deng-Ping Fan, Ge-Peng Ji, Guolei Sun, Ming-Ming Cheng, Jianbing Shen, and Ling Shao. 2020\.  Camouflaged object detection. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 2777–2787.
  * Ghosh et al. (2017) Aritra Ghosh, Himanshu Kumar, and PS Sastry. 2017.  Robust loss functions under label noise for deep neural networks. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 31.
  * Goldberger and Ben-Reuven (2016) Jacob Goldberger and Ehud Ben-Reuven. 2016.  Training deep neural-networks using a noise adaptation layer.  (2016).
  * He et al. (2020) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020.  Momentum contrast for unsupervised visual representation learning. In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_. 9729–9738.
  * Hsieh et al. (2019) Yu-Guan Hsieh, Gang Niu, and Masashi Sugiyama. 2019.  Classification from positive, unlabeled and biased negative data. In _International Conference on Machine Learning_. PMLR, 2820–2829.
  * Jeong et al. (2019) Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. 2019\.  Consistency-based semi-supervised learning for object detection.  _Advances in neural information processing systems_ 32 (2019).
  * Kato et al. (2018) Masahiro Kato, Takeshi Teshima, and Junya Honda. 2018\.  Learning from positive and unlabeled data with a selection bias. In _International conference on learning representations_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Kiryo et al. (2017) Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. 2017.  Positive-unlabeled learning with non-negative risk estimator.  _Advances in neural information processing systems_ 30 (2017).
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * Lee et al. (2016) Gayoung Lee, Yu-Wing Tai, and Junmo Kim. 2016.  Deep saliency with encoded low level distance map and high level features. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 660–668.
  * Li et al. (2021) Changchun Li, Ximing Li, Lei Feng, and Jihong Ouyang. 2021\.  Who Is Your Right Mixup Partner in Positive and Unlabeled Learning. In _International Conference on Learning Representations_.
  * Li et al. (2017) Yuncheng Li, Jianchao Yang, Yale Song, Liangliang Cao, Jiebo Luo, and Li-Jia Li. 2017\.  Learning from noisy labels with distillation. In _Proceedings of the IEEE International Conference on Computer Vision_. 1910–1918.
  * Liu et al. (2003) Bing Liu, Yang Dai, Xiaoli Li, Wee Sun Lee, and Philip S Yu. 2003.  Building text classifiers using positive and unlabeled examples. In _Third IEEE international conference on data mining_. IEEE, 179–186.
  * Liu et al. (2002) Bing Liu, Wee Sun Lee, Philip S Yu, and Xiaoli Li. 2002\.  Partially supervised classification of text documents. In _ICML_ , Vol. 2. Sydney, NSW, 387–394.
  * Liu et al. (2019) Fangqing Liu, Hui Chen, Liyue Zhao, and Hao Wu. 2019\.  Discriminative adversarial networks for positive-unlabeled learning.  _arXiv_ (2019), arXiv–1906.
  * Shen and Sanghavi (2019) Yanyao Shen and Sujay Sanghavi. 2019.  Learning with bad training data via iterative trimmed loss minimization. In _International Conference on Machine Learning_. PMLR, 5739–5748.
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in Neural Information Processing Systems_ 33 (2020), 596–608.
  * Su et al. (2021) Guangxin Su, Weitong Chen, and Miao Xu. 2021.  Positive-unlabeled learning from imbalanced data. In _Proceedings of the 30th International Joint Conference on Artificial Intelligence, Virtual Event_.
  * Tanaka et al. (2018) Daiki Tanaka, Daiki Ikami, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2018.  Joint optimization framework for learning with noisy labels. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 5552–5560.
  * Vahdat (2017) Arash Vahdat. 2017\.  Toward robustness against label noise in training deep discriminative neural networks.  _Advances in Neural Information Processing Systems_ 30 (2017).
  * Verma et al. (2019) Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz. 2019\.  Interpolation consistency training for semi-supervised learning.  _arXiv preprint arXiv:1903.03825_ (2019).
  * Wang et al. (2022) Yikai Wang, Xinwei Sun, and Yanwei Fu. 2022.  Scalable Penalized Regression for Noise Detection in Learning with Noisy Labels.  _arXiv preprint arXiv:2203.07788_ (2022).
  * Xu et al. (2019) Miao Xu, Bingcong Li, Gang Niu, Bo Han, and Masashi Sugiyama. 2019.  Revisiting sample selection approach to positive-unlabeled learning: Turning unlabeled data into positive rather than negative.  _arXiv preprint arXiv:1901.10155_ (2019).
  * Yu et al. (2002) Hwanjo Yu, Jiawei Han, and Kevin Chen-Chuan Chang. 2002\.  PEBL: positive example based learning for web page classification using SVM. In _Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining_. 239–248.
  * Yu et al. (2019) Xingrui Yu, Bo Han, Jiangchao Yao, Gang Niu, Ivor Tsang, and Masashi Sugiyama. 2019\.  How does disagreement help generalization against label corruption?. In _International Conference on Machine Learning_. PMLR, 7164–7173.
  * Zhang and Sabuncu (2018) Zhilu Zhang and Mert Sabuncu. 2018.  Generalized cross entropy loss for training deep neural networks with noisy labels.  _Advances in neural information processing systems_ 31 (2018).
  * Zhou et al. (2020) Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. 2020.  Robust curriculum learning: from clean label detection to noisy label self-correction. In _International Conference on Learning Representations_.

##  Appendix A Experimental Detail

###  A.1. Detailed Information for Liver dataset

Each sample in the Liver dataset denotes a patient with liver cancer. The data
consists of several modalities of MRI images from all patient, including T1WI,
T1CE1, T1IP and T2WI. We simply adopt T1CE1 as the input modality of our
experiment and leave the full usage of all modalities as future works. the
model reads all of the slices of each sample with Pydicom and resize them to
$256\times 256$ without additional normalization. The labels are transformed
from the survival time of each patient, which ranges from 0 to 120.

Figure 5. Detailed architecture of our network for Liver dataset. N, C, H, W
denotes number of slices, number of channels, image height and image width
respectively.

###  A.2. Detailed Architecture for Liver dataset

As we have mentioned in our main context, since each liver cancer sample is
composed of several slices, we adopt a special architecture to deal with such
data. Essentially, as shown in Fig. 5, the basic ResNet18 is used to extract
features for all slices. The network weights are shared among different
slices. Then for slices from one sample, a multi-head attention module is used
to aggregate these features. Then a pooling layer and a classifier are used,
as the same in original ResNet18.

##  Appendix B Additional Experiment Results

Figure 6. Training accuracy of clean and noisy data respectively in easy set.
Figure 7. Gradient visualization of both clean and noisy samples on CIFAR10.

###  B.1. Results on noise detection methods

To support our claim in the main context that noise detection methods may not
be proper for further detecting noisy samples in $\mathcal{T}^{easy}$, we
first plot the training accuracy of noisy and clean samples in the easy set.
As depicted in Fig. 6, while the training accuracy of clean samples is higher
than that of noisy ones for each epoch, the actual size of wrongly-predicted
clean samples is also much higher than noisy set. The proportion of wrongly-
predicted noisy samples among all samples that are wrongly predicted is much
smaller than that in the original unlabeled data as. This means these easy
noisy samples do not enjoy the slow-learning property to the same extend as
the hard noisy ones. Moreover, the result of CGH (Chatterjee, 2020) is shown
in Fig. 7, in which we visualize the gradient of the first weight in the first
convolutional layer of the whole network with regard to both easy noisy
samples and easy clean samples. We can find that both the scale and
distribution are similar between these two kinds of data, which means the
coherent gradient hypothesis also does not hold so well on the easy data.

$\rho$ | Acc.
---|---
500 | 1000 | 3000
0.3 | 88.26 | 89.47 | 91.75
0.4 | 88.68 | 89.51 | 92.03
0.5 | 88.55 | 90.02 | 92.06
0.6 | 88.92 | 90.24 | 92.03
0.7 | 89.28 | 90.51 | 92.51
0.8 | 89.18 | 90.31 | 92.13
0.9 | 89.16 | 90.23 | 92.17
Table 8.  Test accuracies on CIFAR10 of our model trained with different
$\rho$ in Eq. (6) in the main context.

###  B.2. Choices of $\rho$

We compare different $\rho$ in Eq. (6) in the main context, with the results
in Tab. 8. As mentioned in the original paper (Englesson and Azizpour, 2021),
smaller $\rho$ leads to faster convergence and less robustness, which is
roughly consistent with our results. When $\rho=0.3$, the model gets 88.26%,
89.47% and 91.75%, which is the worst among all $\rho$ and almost similar to
the results when using soft cross entropy. As $\rho$ gets larger, the
performance gets better and saturates at $\rho=0.7$.

Model | Acc.
---|---
500 | 1000 | 3000
uPU | 85.98$\pm$1.09 | 88.22$\pm$0.28 | 89.83$\pm$0.42
uPU+Split-PU | 87.44$\pm$0.18 | 90.57$\pm$0.26 | 91.51$\pm$0.13
Table 9.  Test accuracies on CIFAR10 of models trained with and without early
stop splitting.

###  B.3. Choices of $\alpha$ and $\beta$

We conduct the comparison on $\alpha$ and $\beta$ with the same setting as in
the main paper, i.e. 500/1000/3000 positive labeled samples on CIFAR10. As
shown in Tab. 10 and Tab. 11, the performance of our model among different
hyper-parameter settings is relatively robust. Meanwhile, the trend of
performance is generally consistent with few exceptions. For example,
$\alpha=0.5$ or $\beta=0.2$ lead to the highest accuracy when using 3000
positive labeled samples. We advocate that these two hyper-parameters control
the balance among different loss terms adopted by our method. In particular,
larger $\alpha$ and $\beta$ lead to the scale of $L_{pred}$ and $L_{feat}$, as
depicted in Eq. (11) and Eq. (12), larger than that of $L_{easy}$. The
strength of annotations is thus weakened in this way. Consequently, the model
becomes poorer at learning label information from the dataset, which leads to
worse performance. On the other hand, smaller $\alpha$ makes the effect of
prediction consistency negligible, hence resulting in lower accuracy. We will
add these results to our paper.

$\alpha$ | Acc.
---|---
500 | 1000 | 3000
0.1 | 87.68 | 89.87 | 90.84
0.2 | 88.05 | 90.08 | 92.05
0.3 | 89.18 | 90.51 | 92.51
0.4 | 88.61 | 90.03 | 92.27
0.5 | 88.52 | 90.08 | 92.55
0.6 | 88.48 | 89.93 | 92.41
0.7 | 88.60 | 90.09 | 92.13
0.8 | 88.68 | 89.56 | 91.17
0.9 | 88.30 | 89.30 | 90.53
Table 10.  Test accuracies on CIFAR10 of our model trained with different $\alpha$ in Eq. (13) in the main context. $\beta$ | Acc.
---|---
500 | 1000 | 3000
0.1 | 89.18 | 90.51 | 92.51
0.2 | 88.96 | 90.31 | 92.63
0.3 | 89.02 | 90.39 | 92.37
0.4 | 88.67 | 90.11 | 92.43
0.5 | 88.84 | 89.87 | 92.14
0.6 | 88.81 | 90.09 | 92.37
0.7 | 89.07 | 90.08 | 91.71
0.8 | 88.48 | 89.14 | 91.24
0.9 | 88.26 | 89.49 | 91.54
Table 11.  Test accuracies on CIFAR10 of our model trained with different
$\beta$ in Eq. (13) in the main context.

###  B.4. Application of Split-PU on other base models

To further verify the efficacy of our proposed method, we use another popular
objective function uPU (Du Plessis et al., 2014) as our base model, i.e.
replacing $L_{nnPU}$ with the following $L_{uPU}$ when training $\phi_{base}$:

(14) $\displaystyle L_{uPU}$
$\displaystyle=\frac{\pi}{n_{p}}\sum_{i=1}^{n_{p}}L(\phi(I_{i}^{p}),1)+$ (15)
$\displaystyle\frac{1}{n_{u}}\sum_{i=1}^{n_{u}}L(\phi(I_{i}^{u}),-1)-\frac{\pi}{n_{u}}\sum_{i=1}^{n_{u}}L(\phi(I_{i}^{p}),-1)$

The results on CIFAR10 are shown in Tab. 9, which indicates that the
improvement of our Split-PU on both nnPU and uPU is consistent. It is
noticeable that when using 1000 positive labeled samples, using uPU with
Split-PU is even better than using nnPU with Split-PU, receiving accuracy of
90.57% with comparable standard deviation.

Figure 8. Visualization of the easy and hard sets.


# A TPC-based tracking system for a future Belle II upgrade

Andreas Löschcke Centeno  University of Bonn, Institute of Physics, Nußallee
12, 53115 Bonn, Germany  Christian Wessel  University of Bonn, Institute of
Physics, Nußallee 12, 53115 Bonn, Germany  Peter M. Lewis email:
<EMAIL_ADDRESS>University of Bonn, Institute of Physics, Nußallee
12, 53115 Bonn, Germany  Oskar Hartbrich  University of Hawaii, Department of
Physics and Astronomy, 2505 Correa Rd., Honolulu, HI 96822, USA  Jochen
Kaminski  University of Bonn, Institute of Physics, Nußallee 12, 53115 Bonn,
Germany  Carlos Mariñas  University of Valencia - CSIC, Instituto de Fisica
Corpuscular (IFIC), Spain  Sven Vahsen  University of Hawaii, Department of
Physics and Astronomy, 2505 Correa Rd., Honolulu, HI 96822, USA

###### Abstract

In the next decade, intensity frontier experiments will require tracking
systems that are robust against high event and background rates while
maintaining excellent tracking performance. We develop a first conceptual
design of a tracking system for a hypothetical future experiment—here imagined
as a successor to Belle II—built around a time projection chamber (TPC) with
high resolution readout. This choice necessitates a significant expansion of
the silicon vertex detector as well as a new fast timing layer. We simulate
the performance of such a system in the Belle II simulation framework, probe
its major technical challenges, and demonstrate that such a system is suitable
for projected luminosities at the next generation of intensity-frontier
colliders.

Submitted to the Proceedings of the US Community Study

on the Future of Particle Physics (Snowmass 2021)

##  1 introduction

Intensity frontier experiments, particularly those at the $B$ Factories,
require high-precision tracking for fairly low-momentum tracks in the presence
of high event and beam-induced background rates. Such experiments typically
rely on drift chambers to measure helical segments of charged tracks with a
minimal material budget. Operational experience in the early stages of Belle
II, the current state-of-the-art $B$ Factory experiment, has shown that the
drift chamber technology may be reaching its limit due to high occupancy. To
address this issue, we have developed and simulated a first conceptual design
for a Time Projection Chamber (TPC)-based tracking system for a hypothetical
future $B$ factory experiment, which we outline here. Further technical
details of the TPC can be found in Ref. [1].

For convenience and concreteness, we suppose that SuperKEKB and Belle II have
been superseded by ultra-high luminosity upgrades, operating with the same
beam energies but with five times the maximum instantaneous luminosity
($5\times 6.5\times 10^{35}$\mathrm{c}\mathrm{m}^{-2}\,\mathrm{s}^{-1}$$), and
assume that the proposed tracking system is surrounded by existing Belle II
components. However, in principle the concept is equally suitable for a Belle
II upgrade or a future, unrelated intensity-frontier experiment.

##  2 A tracking concept

In the ultra-high luminosity scenario, we assume that the geometry is
constrained by the existing PID and electromagnetic calorimetry systems. With
this constraint, we have considered three competing proposals. The first is an
upgraded drift chamber. We consider this solution unsuitable given the current
challenges of operating Belle II’s drift chamber (CDC) at SuperKEKB well below
design luminosity. The second option is a full silicon tracker. We have
conducted preliminary simulations of such a system and found that it
significantly degrades the $p_{T}$ resolution of tracks due to increased
multiple scattering. This, coupled with the intrinsic cost and structural
difficulties of such a system, suggests that such a system is not suitable.
The third option is a TPC-based tracker, which should provide a significant
reduction in occupancy because it is a true three-dimensional detector, while
drift chambers are effectively two-dimensional. However, a TPC tracker has
some intrinsic limitations: it cannot provide a trigger signal as the CDC
does, and event pileup can be very high due to the long electron drift time.
We will see that these problems are easily overcome, and we base our
conceptual design on such a system.

Our basic conceptual design consists of three components, illustrated in Fig.
1. First, the current CDC is replaced by a TPC with a single drift volume and
readout on the backward endcap. Second, the current silicon vertex detectors
are replaced with a new detector, which we base off of the VTX upgrade
proposal. In order to maintain an annular cylinder geometry for the TPC, we
extend the VTX from a radius of $13.5\text{\,}\mathrm{c}\mathrm{m}$ to
$44\text{\,}\mathrm{c}\mathrm{m}$. Third, we insert a multilayer fast timing
detector [2], possibly silicon, at $r=25$ cm or $r=45$ cm in order to replace
the triggering role of the CDC and additionally provide particle
identification (PID) via time-of-flight (TOF) for low-$p_{T}$ tracks.

Figure 1: A schematic cross-section of the (top) Belle II and (bottom)
hypothetical high-luminosity upgrade tracking systems. The electron beam
travels along the bottom of the figures to the right and collides with the
left-going positron beam at the black semicircle.

###  2.1 Technical requirements

We consider the tracking system concept viable if it achieves similar
performance to the CDC in realistic conditions. We perform our performance
tests at either of two scenarios: at SuperKEKB full luminosity (the “$\times
1$” scenario) or at projected ultra-high luminosity (“$\times 5$”).

###  2.2 Technical challenges

We target our early studies at answering the most challenging questions raised
by the tracking TPC concept, including:

  1. 1.

Event pileup: Can tracks be reliably associated with unique events despite a
high degree of event overlap?

  2. 2.

Background pileup: Can beam-induced background hits be reliably identified and
rejected despite integrating over $60\text{\,}\mathrm{\SIUnitSymbolMicro m}$
of drift time?

  3. 3.

Ion density: Will the ion backflow rate be small enough that it does not
significantly degrade the tracking performance?

  4. 4.

Continuous readout: Can all of these be accomplished with a continuous readout
design (without gating), which is necessitated by the high physics event
rates?

  5. 5.

Diffusion: Can the TPC match the tracking performance of the CDC by using a
high number of space-time points to overcome the limitations of diffusion?

We consider demonstrations of positive answers to these questions to
constitute a proof-of-principle for the tracking TPC project. In order to
focus on these key challenges instead of more basic technical design
optimizations, we borrow heavily from work already done for the ILD TPC [3].
In the next section, we explain our assumptions and design.

###  2.3 Proof-of-principle TPC design

Our basic design consists of a single gas volume of length
$242\text{\,}\mathrm{c}\mathrm{m}$ with high-resolution readout tiling at the
backward endcap. We do not use any form of ion gating and we assume that an
external detector will provide the trigger, which will be used to isolate
discrete event “windows” in the continuous data.

We assume that the TPC uses atmospheric pressure T2K gas mixture [4]. We use
MAGBOLTZ [5] to determine the a drift field ($289$ V/cm) that minimizes the
drift time of electrons
($v_{D}=$7.89\text{\,}\mathrm{c}\mathrm{m}\mathrm{/}\mathrm{\SIUnitSymbolMicro}\mathrm{s}$$,
leading to a maximum drift time of roughly
$30\text{\,}\mathrm{\SIUnitSymbolMicro s}$ for a maximum drift length of
$242\text{\,}\mathrm{cm}$). From the same simulation we find that the
longitudinal and transverse diffusion coefficients are
$\sigma_{L}=$200\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}\mathrm{/}\mathrm{c}\mathrm{m}$^{0.5}$
and
$\sigma_{T}=$84\text{\,}\mathrm{\SIUnitSymbolMicro}\mathrm{m}\mathrm{/}\mathrm{c}\mathrm{m}$^{0.5}$.

For charge amplification and readout, we take inspiration from the GridPix
system proposed for use in LCTPC. GridPix consists of an array of
$55\times$55\text{\,}\mathrm{\SIUnitSymbolMicro m}$$ pixels with a MICROMEGAS
mesh mounted onto the surface. For our purposes, this technology presents a
number of advantages: first, the small pixels and direct mapping between
amplification cells and pixels constitute essentially a best-case resolution
scenario. Second, in theory such a sensor can be operated in binary readout
mode in which each individual hit represents exactly one electron and consists
only of the pixel ID and a relative threshold-crossing time. This can
dramatically reduce the data throughput, which we anticipate to be a
significant technical challenge at ultrahigh luminosities and with continuous
readout. Finally, it is very easy to implement such a detector in the
digitization simulation.

##  3 Simulation

We use the Belle II event and detector simulation to generate primary
ionization for physics events and beam-induced backgrounds. However, due to
the event-based nature of the simulation framework, these simulated events
must be merged in external code, in such a way that the effects of event
overlay, drift, and diffusion are encapsulated.

###  3.1 Event overlay

We generate a collection of events of each of the expected event types
(including Bhabha scattering and $ee\to\\{\Upsilon(4S)$, $u\bar{u},\
d\bar{d},\ s\bar{s},\ c\bar{c},\ \gamma\gamma,\ \mu\mu(\gamma),\tau\tau,\
eeee,$ and $ee\mu\mu$}) in proportion to their known cross sections. We then
simulate ionization in the TPC using Geant4 within the basf2 framework [6].

We then distribute these events randomly in time and add Gaussian smearing
independently in the longitudinal direction and in the transverse plane to
approximate diffusion depending on the total drift distance of each primary
charge. After drift and diffusion, we bin into 3D voxels (nominally $55\times
55\times$55\text{\,}{\mathrm{\SIUnitSymbolMicro m}}^{3}$$) to approximate the
amplification digitization in a GridPix-like detector operating in binary
readout mode.

Finally, we center the time window on a single $\Upsilon(4S)$ event and
discard hits from outside a $30\text{\,}\mathrm{\SIUnitSymbolMicro s}$ window
approximating the total drift time of a single event. An example of a single
overlay is shown in Fig. 2.

##  4 Studies

Our proof of concept consists of simulation studies targeting the primary
concerns of Sec. 2.2, using the simulation techniques described in Sec. 3.

###  4.1 Event pileup

Using the event overlay procedure described in Sec. 3.1 with 100 simulated
events, we find that the mean number of physics background tracks per
triggered event is approximately $9$. This is somewhat smaller than the
typical number of $\Upsilon(4S)$ tracks in an event (roughly $11$). The
background tracks are almost entirely from Bhabha scattering (Fig. 2). All
physics background tracks originating from collisions apart from the triggered
one should be straightforward to identify and remove: not only will they fail
to point to the interaction point, but they will have a transverse diffusion
width inconsistent with their relative $z$ position.

Even at extrapolated luminosities, we find that the number of tracks
overlapping with the triggered event is small, and that these are relatively
easily identified and removed. Consequently, we do not consider event pileup
to be a major technical barrier to the TPC tracking concept.

###  4.2 Background pileup

Beam-induced background hits are largely produced by microcurlers. These are
low-$p_{T}$ electrons produced when low-energy photons interact in the TPC
volume. These electrons are fully captured in the parallel electric and
magnetic fields, and have far longer path lengths in the TPC volume than
typical physics tracks. Consequently, a single microcurler typically deposits
far more ionization energy than a physics track. This, coupled with the huge
abundance of low-energy beam-induced photons, means that beam-induced
backgrounds dominate the ionization rate at both the $\times 1$ and $\times 5$
luminosity scenarios.

Figure 2 shows typical overlaid events with and without beam-induced
backgrounds ($\times 5$ scenario). Figure 3 shows the typical deposited charge
in triggered $\Upsilon(4S)$ events by source. In both figures it is clear that
beam-induced background hits dominate over physics sources. In fact, this
estimate might be quite optimistic: SuperKEKB uses continuous injection [7],
and associated backgrounds are not included in the beam simulation. Assuming
that the ultra-high luminosity upgraded accelerator similarly uses continuous
injection, we would expect the TPC to typically integrate over the backgrounds
produced by multiple individual injections. Consequently, the total beam-
induced background ionization rate may be significantly higher than seen in
our simulations.

However, even if we multiply the mean beam-induced background hit rate in the
$\times 5$ scenario by a safety factor of 5 to account for probable injection
backgrounds, we find a typical voxel occupancy of $2\times 10^{-4}$, assuming
relatively large $200\times 200\times$200\text{\,}{\mathrm{\SIUnitSymbolMicro
m}}^{3}$$ voxels. Furthermore, microcurlers are typically quite distinctive,
depositing small rings of high charge density on the readout plane, and
consequently a large number of them can be easily identified and rejected,
perhaps even before readout. We find that simply rejecting clusters of hits
that are isolated in $2\times$2\text{\,}{\mathrm{cm}}^{2}$$ cells can reject a
majority of microcurlers. This scenario mimics an easily implemented self-
trigger mask that requires hits near the edges of a chip to accept an external
readout trigger. Further front-end measures to prevent even reading out
microcurlers are possible.

Given the low total occupancy of beam-induced background hits and the relative
ease of identifying and rejecting them early, we conclude that they do not
constitute a major technical barrier to the TPC concept. However, this
certainly presents one of the more challenging and interesting considerations
for the design of the readout system.

Figure 2: Typical overlays without (top) and with (bottom) $\times 5$ beam-
induced background hits. Three TPC volumes are shown, with the central volume
between $z=0$ and $242$ cm centered on an $\Upsilon(4S)$ decay and
corresponding to the read-out event. An external trigger on the $\Upsilon(4S)$
event is assumed. Although modest overlaps from physics backgrounds (primarily
Bhabha scattering) exist, the total ionization rate is dominated by
microcurlers from beam-induced backgrounds. Figure 3: A histogram of the total
ionization charge (equivalent to detector hits) deposited in the TPC volume
for a number of externally triggered $\Upsilon(4S)$ events. The approximate
mean number of hits from $\Upsilon(4S)$ events is $200\times 10^{3}$, from
Bhabha events is $130\times 10^{3}$, from all other physics backgrounds is
$6\times 10^{3}$ and from beam-induced backgrounds ($\times 1$) is $2200\times
10^{3}$.

###  4.3 Ion density

A second potential challenge posed by the high ionization rates is ion
backflow (IBF). We initially assume that no gating is used, in order to
preserve continuous readout. Therefore, the mean IBF rate is determined by the
primary ionization rate, the ion drift velocity, the gain, and the fraction of
backflowing ions from the amplification. Using an educated guess of $1\%$
backflow at a gain of 2000, and using measurements of the mobilities of the
primary ions in the T2K gas [4], we find a typical ion charge density
comparable to that of other tracking TPCs. However, this number is certainly
optimistic: injection backgrounds are not simulated and we expect them to be
very high since continuous injection will presumably be used. Furthermore, the
TPC ionization rate is dominated by microcurlers, most of which come from
beam-induced low-energy photons. Therefore, the ion density depends strongly
on our SuperKEKB beam background simulation that may not be suitable for the
future upgrade scenario. Therefore, we cannot rule out the possibility that
high backflowing ion densities will substantially degrade the tracking
performance.

Considerable further work is necessary to simulate and measure the IBF and to
determine its effects on the tracking performance. The mean “time occupancy”
of $\Upsilon(4S)$ events in the $5\times$ scenario is roughly $15\%$, which
suggests that gating is not feasible, as was assumed.

###  4.4 Tracking performance

The key question concerning tracking performance is whether the increase in
the number of spatial hit points (up to the theoretical maximum; the number of
ionizations) can win out over diffusion when determining tracking parameters.
Here we focus primarily on the transverse momentum $p_{T}$. We use a set of
$1000$ simulated muons in discrete bins of $p_{T}$ and distributed uniformly
in $\theta$ over the acceptance of the TPC.

Figure 4 shows the simulated $p_{T}$ resolution for the current (CDC+VXD) vs.
proposed (TPC+VTX) tracking systems averaged over $\theta$. The vertical
offset is due to multiple scattering, which is largely determined by the
material budget of the vertex detectors, which for the VTX is highly
speculative. In principle, due to significant thinning of each layer, the
TPC+VTX should be able to achieve comparable or lower levels of multiple
scattering compared to the CDC+VXD.

The linear slope in $p_{T}$, due to position measurement resolution, is far
shallower for the TPC compared to the CDC. This is due to the large number of
hit points in the TPC, rendering its effective point resolution far superior
to the hit resolution in the CDC. We conclude that the TPC is capable of
matching or surpassing the tracking resolution of the CDC. However, the
tracking performance in the critical range $p_{T}<$1\text{\,}\mathrm{GeV}$$
depends more strongly on the amount of material in the VTX than it does on the
differences between the CDC and TPC.

Figure 4: A comparison of the average $p_{T}$ resolution for the CDC+VXD and
TPC+VTX tracking systems.

With transverse tracks ($\theta=$90\text{\,}\mathrm{\SIUnitSymbolDegree}$$,
with a drift length of $85$ cm), the CDC+VXD has better $p_{T}$ resolution
than the TPC+VTX, due to a combination of multiple scattering in the VTX and
diffusion in the TPC (Fig. 5). Additionally, at this angle the tracks have
their minimum path length in the TPC, leading to a minimum in the number of
hits. However, the TPC+VTX resolution degrades far less quickly for shallower
angles due to an increase in the number of hit points. This benefit compared
to CDC tracking is amplified in the backward region where diffusion in the TPC
is at its minimum.

Figure 5: A comparison of the $p_{T}$ resolution for CDC+VXD compared to
TPC+VTX vs. polar angle $\theta$. The gray histogram in the background
indicates the distribution of tracks in $\Upsilon(4S)$ decays. The dashed
lines marks the boundary between the TPC endcaps and barrel.

In order to determine the contributions to the effective point resolution, we
study how the $p_{T}$ resolution depends on diffusion and pixel pitch (Fig.
6). We find that diffusion dominates the effective resolution, and
consequently there is no benefit to decreasing the pitch far below the typical
diffusion width (roughly $1\text{\,}\mathrm{mm}$). Under the assumption that
it is desirable to minimize the channel count, we select the largest pixel
pitch that shows now significant degradation in $p_{T}$ compared to the ideal
case: $200\text{\,}\mathrm{\SIUnitSymbolMicro m}$.

Figure 6: A comparison of the TPC+VTX tracking resolution for various readout
scenarios. Ideal is for a TPC with no diffusion and perfect resolution,
idealDiff adds realistic diffusion, and the other three scenarios indicate
simulated pixel pitches.

###  4.5 Triggering

For the timing layer, we envision multiple layers of fast silicon sensors
between the beam pipe and the TPC. For concreteness, we assume that this
sensor uses the same technology as in the STOPGAP proposal [2], which uses
these sensor to increase the hermiticity of the Belle II PID system outside of
the tracking volume. As designed, this detector has a time resolution better
than $100\text{\,}\mathrm{ps}$, and we place it in the middle of the VTX (at
$r=$25\text{\,}\mathrm{cm}$$) or between the VTX and the TPC (at
$r=$45\text{\,}\mathrm{cm}$$). Triggers are generated based on coincidence
between layers. Based on simulation with a 3-layer detector at 25cm, a two-
track trigger will have a fake rate of less than 1Hz (Fig. 7).

Figure 7: Simulation of the trigger performance (in the $\times 1$ scenario)
of the fast timing detector in both 2- and 3-layer versions, and at positions
$r=25$ and $45$ cm.

A promising capability of such a timing detector is that it provides PID
information based on time-of-flight (TOF). This compensates for the loss of
$dE/dx$ PID information from the inner CDC volume abandoned to the VTX. Fig. 8
shows the $K/\pi$ discrimination power of the simulated timing detector acting
as a TOF PID detector. For the $r=25$ cm scenario, excellent separation down
to $50\text{\,}\mathrm{MeV}$ is possible, with improved performance relative
to the CDC.

From this preliminary work, we conclude that the combination of a TPC with a
multi-layer timing detector at $r=$25\text{\,}\mathrm{cm}$$ is capable of
matching or surpassing the triggering and low-$p_{T}$ PID capabilities of the
CDC.

Figure 8: Simulated $K/\pi$ separation power vs. $p_{T}$ for a mutilayer fast-
timing detector located at $r=25$ and $45\text{\,}\mathrm{cm}$. The dashed
vertical lines indicate the $p_{T}$ acceptance of each detector.

###  4.6 Additional studies

Further studies, detailed in Ref. [1], suggest that the $p_{T}$ resolution is
fairly insensitive to electron efficiency, that the maximum data throughput
would be comparable to the current VXD ($7$ GB/s), assuming front-end hit
filtering), and that the pixel dead-time has a negligible effect on electron
efficiency. In aggregate, these studies indicate no additional cause for
serious doubt about the prospects of TPC-based tracking at the next generation
of intensity-frontier experiments.

##  5 Conclusions

Based on these studies, we conclude that a gas TPC-based tracking system may
be viable for intensity frontier experiments like the hypothetical ultra-high
luminosity upgrade to Belle II. Our design relies heavily on the capabilities
of the GridPix sensor, particularly the association of a single pixel with a
single electron with low ion backflow and excellent 3D resolution. The primary
difficulty of such a system is the cost of tiling these sensors over the
$>$3\text{\,}{\mathrm{m}}^{2}$$ endplate. However, we find that binary readout
with relatively large pixels
($200\times$200\text{\,}\mathrm{\SIUnitSymbolMicro m}$$) is sufficient to meet
the performance objectives of the tracking system, which decreases the channel
count and data throughput of the system, perhaps offsetting some of the costs.
The timing layer proposal is an essential part of the TPC-based tracking
system, but is perhaps independently desirable as a source of low-$p_{T}$ PID
discrimination.

The sole major outstanding technical question is whether the backflowing ion
densitities will remain tolerable after factoring in real-world beam-induced
backgrounds, including those caused by injection. Considerable work is needed
to answer this question definitively, including the development of prototypes,
testbeam campaigns, and a mature design and simulation of the upgraded
accelerator and interaction area. For now, we can only conclude that our
preliminary simulations do not rule out the tracking TPC concept on the basis
of ion density.

## References

  * [1] A. Löschcke Centeno  “First Conceptual Design and Studies fo a Tracking Time Projection Chamber for the Belle II Experiment”, 2021  URL: https://docs.belle2.org/record/2631/files/BELLE2-MTHESIS-2021-073.pdf
  * [2] Oskar Hartbrich, 2022  arXiv:2203.04847 [hep-ex]
  * [3] J. Kaminski  “Large Area Coverage of a TPC Endcap with GridPix Detectors”  In _EPJ Web Conf._ 174, 2018  DOI: 10.1051/epjconf/201817402001
  * [4] A… Cortez  “Experimental ion mobility measurements for the LCTPC Collaboration”  In _NIM-A_ 936, 2019, pp. 451–452  DOI: https://doi.org/10.1016/j.nima.2018.11.049
  * [5] S F Biagi  “Monte Carlo simulation of electron drift and diffusion in counting gases under the influence of electric and magnetic fields”  In _NIM-A_ 421, 1999  DOI: 10.1016/S0168-9002(98)01233-9
  * [6] D.-Y. Kim  “The simulation library of the Belle II software system”  In _Journal of Physics: Conference Series_ 898, 2017  DOI: 10.1088/1742-6596/898/4/042043
  * [7] Y. Ohnishi  “Accelerator design at SuperKEKB”  In _Progress of Theoretical and Experimental Physics_ 2013.3, 2013  DOI: 10.1093/ptep/pts083