# Active label cleaning: Improving dataset quality
under resource constraints

Mélanie Bernhardt  Health Intelligence, Microsoft Research Cambridge,
Cambridge, CB1 2FB, UK  These authors contributed equally to this work. Daniel
C. Castro  Health Intelligence, Microsoft Research Cambridge, Cambridge, CB1
2FB, UK  These authors contributed equally to this work. Ryutaro Tanno  Health
Intelligence, Microsoft Research Cambridge, Cambridge, CB1 2FB, UK  Anton
Schwaighofer  Health Intelligence, Microsoft Research Cambridge, Cambridge,
CB1 2FB, UK  Kerem C. Tezcan  Health Intelligence, Microsoft Research
Cambridge, Cambridge, CB1 2FB, UK
Miguel Monteiro  Health Intelligence, Microsoft Research Cambridge, Cambridge,
CB1 2FB, UK  Shruthi Bannur  Health Intelligence, Microsoft Research
Cambridge, Cambridge, CB1 2FB, UK  Matthew Lungren  Department of Radiology,
Stanford University, Palo Alto, CA 94304, USA  Aditya Nori  Health
Intelligence, Microsoft Research Cambridge, Cambridge, CB1 2FB, UK  Ben
Glocker  Health Intelligence, Microsoft Research Cambridge, Cambridge, CB1
2FB, UK
Javier Alvarez-Valle  Health Intelligence, Microsoft Research Cambridge,
Cambridge, CB1 2FB, UK  Ozan Oktay  Health Intelligence, Microsoft Research
Cambridge, Cambridge, CB1 2FB, UK

###### Abstract

Imperfections in data annotation, known as label noise, are detrimental to the
training of machine learning models and have an often-overlooked confounding
effect on the assessment of model performance. Nevertheless, employing experts
to remove label noise by fully re-annotating large datasets is infeasible in
resource-constrained settings, such as healthcare. This work advocates for a
data-driven approach to prioritising samples for re-annotation—which we term
“active label cleaning”. We propose to rank instances according to estimated
label correctness and labelling difficulty of each sample, and introduce a
simulation framework to evaluate relabelling efficacy. Our experiments on
natural images and on a new medical imaging benchmark show that cleaning noisy
labels mitigates their negative impact on model training, evaluation, and
selection. Crucially, the proposed active label cleaning enables correcting
labels up to 4× more effectively than typical random selection in realistic
conditions, making better use of experts’ valuable time for improving dataset
quality.

## Introduction

The success of supervised machine learning primarily relies on the
availability of large datasets with high-quality annotations. However, in
practice, labelling processes are prone to errors, almost inevitably leading
to noisy datasets—as seen in ML benchmark datasets [1]. Labelling errors can
occur due to automated label extraction [2, 3], ambiguities in input and
output spaces [4], or human errors [5] (e.g. lack of expertise). At training
time, incorrect labels hamper the generalisation of predictive models, as
labelling errors may be memorised by the model resulting in undesired biases
[6, 7]. At test time, mislabelled data can have detrimental effects on the
validity of model evaluation, potentially leading to incorrect model selection
for deployment as the true performance may not be faithfully reflected on
noisy data. Label cleaning is therefore crucial to improve both model training
and evaluation.

Figure 1: Overview of the proposed active label cleaning. A dataset with noisy
labels is sorted to prioritise clearly mislabelled samples, maximising the
number of corrected samples given a fixed relabelling budget.

Relabelling a dataset involves a laborious manual reviewing process and in
many cases the identification of individual labelling errors can be
challenging. It is typically not feasible to review every sample in large
datasets. Consider for example the NIH ChestXray dataset [3], containing 112k
chest radiographs depicting various diseases. Diagnostic labels were extracted
from the radiology reports via an error-prone automated process [8]. Later, a
subset of images (4.5k) from this dataset were manually selected and their
labels were reviewed by expert radiologists in an effort driven by Google
Health [2]. Similarly, 30k randomly selected images from the same dataset were
relabelled for the RSNA Kaggle challenge [9]. Such relabelling initiatives are
extremely resource-intensive, particularly in the absence of a data-driven
prioritisation strategy to help focusing on the subset of the data that most
likely contains errors.

Due to the practical constraints on the total number of re-annotations,
samples often need to be prioritised to maximise the benefits of relabelling
efforts (see Fig. 1), as the difficulty of reviewing labelling errors can vary
across samples. Some cases are easy to assess and correct, others may be
inherently ambiguous even for expert annotators (Fig. 2). For such difficult
cases, several annotations (i.e. expert opinions) may be needed to form a
ground-truth consensus [10, 2], which comes with increasing relabelling
“cost”. Hence, there is a need for relabelling strategies that consider both
resource constraints and individual sample difficulty—especially in
healthcare, where availability of experts is limited and variability of
annotations is typically high due to the difficulty of the tasks [11].

While there are learning approaches designed specifically to handle label
noise during training, we consider these strategies to be insufficient for two
reasons. First and most importantly, in situations with noisy evaluation data,
there is no reliable method to determine whether robust learning is effective,
as the performance metrics are directly compromised by label noise. Second,
noise-robust learning attempts to train a maximally accurate model in the
presence of incorrect labels, often by disregarding samples that could
otherwise be highly informative, some of which may even be correctly labelled.
In contrast, label cleaning aims to improve the quality and usefulness of a
dataset to preserve as many cases as possible by fixing incorrect labels. This
is imperative in safety-critical domains such as healthcare, as model
robustness must be validated on clean labels.

Prioritising samples for labelling also underpins the paradigm of active
learning, whose goal is to select unlabelled samples that would be most
beneficial for training in order to improve the performance of a predictive
model on a downstream task. The key difference here for the proposed approach
is that our goal is not only to improve model performance but also to maximise
the quality of labels given limited resources, which makes it valuable for
both training and evaluation of predictive models. In more detail, we
demonstrate how active learning and noise-robust learning (NRL) can play
complementary roles in coping with label noise.

In this work, we begin by defining the active label cleaning setting in
precise terms, along with the proposed relabelling priority score. Using
datasets of natural images and of chest radiographs, we then demonstrate
experimentally the negative impacts of label noise on training and evaluating
predictive models, and how cleaning the noisy labels can mitigate those
effects. Third, we show via simulations that the proposed active label
cleaning framework can effectively prioritise samples to re-annotate under
resource constraints, with substantial savings over naive random selection.
Fourth, we analyse how robust-learning [12] and self-supervision [13]
techniques can further improve label cleaning performance. Lastly, we validate
our choice of scoring function, which accounts for sample difficulty and noise
level, comparing with an active learning baseline.

Figure 2: Image labelling can become difficult due to ambiguity in input space
[14]. Top row shows the spectrum of ambiguity for cat images sampled from
CIFAR10H dataset. The 2D plot illustrates different types of mislabelled
samples: clear noise and difficult cases. We expect the former to be adjacent
to semantically similar samples with a different label, and the latter to be
closer to the optimal decision boundary.

## Results

### Active label cleaning.

In this work, we introduce a sequential label cleaning procedure that
maximises the number of corrected samples under a total resource budget
$B\in\mathds{N}$:

$\max\;\underbrace{\frac{1}{N}\sum_{i=1}^{N}\mathbf{1}\mathopen{}\mathclose{{}\left[\hat{y}_{i}=y_{i}}\right]}_{\text{\clap{correctness
of majority
labels}}}\quad\text{s.t.}\;\underbrace{\sum_{i=1}^{N}\lVert\hat{\ell}_{i}\rVert_{1}\leq
B}_{\text{budget constraint}}\,,$ (1)

where we assume access to a dataset
$\mathcal{D}=\\{(\mathbf{x}_{i},\hat{\ell}_{i})\\}_{i=1}^{N}$, with the
$i^{\text{th}}$ image $\mathbf{x}_{i}$, label counts vector
${\hat{\ell}_{i}\in\mathds{N}^{C}}$ with $C$ classes, and corresponding
majority label
$\hat{y}_{i}=\operatorname*{arg\,max}_{c\in\\{1,\dots,C\\}}\hat{\ell}_{i}^{c}$.
All samples are assumed to initially contain at least one label
($\lVert\hat{\ell}_{i}\rVert_{1}=\sum_{c=1}^{C}\hat{\ell}_{i}^{c}\geq 1$), and
some instances are mislabelled, i.e. their initial (majority) label $\hat{y}$
deviates from the true class $y$. Unlike the traditional active learning
objective, we aim to achieve a clean set of labels that can later be used not
only for model training but also for benchmarking purposes. In that regard,
our setting differs from active learning by allowing for re-annotation of
already labelled instances as in ‘re-active’ learning [15]. This is an
important consideration as there is often a trade-off between label quality
and downstream model performance [16, 15]. Yet, conventional approaches [17,
18] may not be practically feasible with modern deep learning, as they often
entail repeated retraining of models to measure the impact of each sample.

The proposed framework (see Table 1) determines relabelling priority based on
predicted posteriors from a trained classification model, $\mathop{\mkern
0.0mup_{\theta}}(\hat{y}\nonscript\mkern 1.5mu|\nonscript\mkern
1.5mu\mathopen{}\allowbreak\mathbf{x})$, parametrised by $\theta$. Label
cleaning is performed over multiple iterations and at each iteration either a
single or a batch of samples are relabelled. Within an iteration, samples are
first ranked according to predicted label correctness and ambiguity (i.e.
annotation difficulty). Then, each prioritised sample is reviewed sequentially
by different annotators until a majority is formed among all the collected
labels $\hat{\ell}_{i}$. In the next iteration, the remaining samples are re-
prioritised and the process repeats until the relabelling budget ($B$) is
exhausted. Assuming that each annotation has a fixed cost, the more
annotations a sample requires until a majority is reached, the more expensive
its relabelling. Hence, to achieve the objective in Eq. 1, clearly mislabelled
samples need to be prioritised over difficult cases, and correctly labelled
samples should have the lowest relabelling priority.

To this end, we propose to rank available samples by the following scoring
function $\Phi$:

$\begin{split}\Phi(\mathbf{x},\hat{\ell};\theta)&=\underbrace{\mathit{CE}(\hat{\ell},\mathop{\mkern
0.0mup_{\theta}})}_{\text{\clap{noisiness
$\uparrow$}}}\;-\;\underbrace{H(\mathop{\mkern
0.0mup_{\theta}})}_{\text{\clap{ambiguity $\downarrow$}}}\,.\end{split}$ (2)

The first term, defined as the cross-entropy from the normalised label counts
to the predicted posteriors,

$\mathit{CE}(\hat{\ell},\mathop{\mkern 0.0mup_{\theta}})=-\mathop{\mkern
0.0mu\mathbb{E}_{\hat{\ell}/\lVert\hat{\ell}\rVert_{1}}}[\log\mathop{\mkern
0.0mup_{\theta}}(\hat{y}\nonscript\mkern 1.5mu|\nonscript\mkern
1.5mu\mathopen{}\allowbreak\mathbf{x})]\,,$ (3)

corresponds to the estimated noisiness (i.e. negative log-likelihood) of the
given labels. On the other hand, obtaining a majority label vote requires
different numbers of re-annotations for different images, depending on their
difficulty. This is quantified by the entropy term $H(\mathop{\mkern
0.0mup_{\theta}})$ defined over posteriors,

$H(\mathop{\mkern 0.0mup_{\theta}})=-\mathop{\mkern
0.0mu\mathbb{E}_{\mathop{\mkern 0.0mup_{\theta}}(\hat{y}\nonscript\mkern
1.5mu|\nonscript\mkern
1.5mu\mathopen{}\allowbreak\mathbf{x})}}[\log\mathop{\mkern
0.0mup_{\theta}}(\hat{y}\nonscript\mkern 1.5mu|\nonscript\mkern
1.5mu\mathopen{}\allowbreak\mathbf{x})]\,,$ (4)

which penalises ambiguous cases in the ranking. Similar objectives have been
used in the contexts of semi-supervised learning [19] and entropy
regularisation [20] to penalise over-confident posterior predictions from
models. In Methods section, we present different options for the predictive
model $\mathop{\mkern 0.0mup_{\theta}}$ to be used in computing these
quantities.

### Datasets used in the experiments.

To analyse label noise scenarios, we experiment with two imaging datasets,
namely CIFAR10H and NoisyCXR, containing multiple annotations per data point,
which helps us to model the true label distributions and associated labelling
cost. In CIFAR10H [14, 5], on average 51 manual annotations were collected for
each image in the test set of CIFAR10 [21], a collection of 10k natural images
grouped in 10 classes. The experiments on CIFAR10H are intended to understand
the challenges associated with different noise rates, noise models, and active
relabelling scenarios. We also set up a new benchmark for label cleaning on
medical images, NoisyCXR, comprising 26.6k chest radiographs and multiple
labels from clinical datasets indicating the presence of pneumonia-like
opacities [9, 3]. This serves as a more challenging imaging benchmark, where
modelling difficulties are coupled with the challenges associated with noisy
labels and sample prioritisation.

### Label noise undermines both model training and evaluation.

In particular, training with noisy labels is known to impair the performance
of predictive models [6, 7], as the latter may be forced to memorise wrong
labels instead of learning generalisable patterns. As an example, a vanilla
convolutional neural network (CNN) classifier trained on CIFAR10H with clean
labels achieves 73.6% accuracy on a clean test set of 50k images. However, if
the same model is trained on a version of this dataset with 30% label noise,
its test accuracy degrades to 64.1%, a substantial drop of 8.5%. Even a noise-
robust classifier, initialised with self-supervision (i.e. pre-trained with
images only), drops from 80.7% to 78.7% in accuracy when fine-tuned on the
same noisy training labels as opposed to clean labels.

We also highlight the adverse implications of validation label noise on model
evaluation. Its most evident impact is that true model performance can be
underestimated. For instance, consider CIFAR10H with 40% noise rate in both
training and validation labels (instance-dependent noise [22]; see Methods). A
self-supervised classifier, as above, is estimated to be only 46.5% accurate
on these noisy validation labels, compared to 65.8% on the true, clean labels.
Additionally, model rankings based on noisy validation metrics may be
unreliable, leading to misinformed model selection or design decisions. For
example, a second classifier, trained on the same training set via co-teaching
[12], achieves a lower 45.9% accuracy on the same noisy validation set, while
its real accuracy (unobservable in practice) on clean labels is 69.9%. In
other words, the worst performing model (here, self-supervised) would have
been chosen for deployment because of misleading validation metrics obtained
on noisy labels.

In summary, label noise can negatively affect not only model building, but
also validation. The latter is especially relevant in high-risk applications,
e.g. for the regulation of models in healthcare settings. Our results in later
sections demonstrate how active cleaning of noisy training and evaluation
labels can help mitigate such issues.

### Simulation of sequential relabelling.

A traditional way to evaluate a label cleaning algorithm is via its ability to
separate noisy from clean labels (e.g. detection rate) [17, 18]. However, such
an evaluation does not necessarily take into account the sequential nature of
sample selection, label acquisition, and model updates. Further, it typically
assumes all new labels are correct, neglecting the effect of sample ambiguity
on how many annotations are required. Both are crucial factors in measuring
the effectiveness of label cleaning efforts in terms of resource usage and
final outcome.

Table 1: Active label cleaning

Given: |  $Y=\\{\ell_{i}\\}_{i=1}^{N}$: True label distributions
---|---
Input: |  $\mathcal{D}=\\{(\mathbf{x}_{i},\hat{\ell}_{i})\\}_{i=1}^{N}$: Dataset with noisy labels
|  $B\in\mathds{N}$: Relabelling budget
|  $b\in\mathds{N}$: Update frequency

1:$\theta\leftarrow\textsc{TrainRobustModel}(\mathcal{D})$

2:$\mathcal{I}_{\mathrm{avail}}\leftarrow\\{1,\dots,N\\},\quad\mathcal{I}_{\mathrm{cleaned}}\leftarrow\emptyset$

3:count $\leftarrow$ 0

4:while count $<B$ do $\triangleright$ If budget remains

5:
$j\leftarrow\operatorname*{arg\,max}_{i\in\mathcal{I}_{\mathrm{avail}}}\Phi(\mathbf{x}_{i},\hat{\ell}_{i};\theta)$
$\triangleright$ Rank (Eq. 2)

6: repeat

7: $\hat{\ell}_{j}\leftarrow\hat{\ell}_{j}+\textsc{Sample}(\ell_{j})$
$\triangleright$ Acquire one-hot label

8: count $\leftarrow$ count + 1

9: until majority formed in $\hat{\ell}_{j}$

10:
$\mathcal{I}_{\mathrm{avail}}\leftarrow\mathcal{I}_{\mathrm{avail}}\setminus\\{j\\},\quad\mathcal{I}_{\mathrm{cleaned}}\leftarrow\mathcal{I}_{\mathrm{cleaned}}\cup\\{j\\}$

11:
$\mathcal{D}\leftarrow\\{(\mathbf{x}_{i},\hat{\ell}_{i}):i\in\mathcal{I}_{\mathrm{avail}}\cup\mathcal{I}_{\mathrm{cleaned}}\\}$

12: if count divisible by $b$ then

13: $\theta\leftarrow\textsc{Update}(\theta,\mathcal{D})$ $\triangleright$
Fine-tune model

14: end if

15:end while

16:return $\mathcal{D}$

To account for these factors, we propose a realistic simulation of the entire
relabelling process that leverages the true label distribution of each sample.
As shown in Table 1, the simulation proceeds sequentially. Once a sample has
been selected for relabelling, new labels are drawn from its true distribution
(Table 1, 7), which reflects the probabilities of real annotators assigning
each class to a given sample. This enables more faithful modelling of the
expected relabelling effort for each sample and of the likely label errors.
Through this simulation, the performance of sample selection algorithms can be
measured in terms of percentage of corrected labels in the dataset as a
function of the number of re-annotations.

The relabelling simulation is used as a testbed to evaluate three different
sample ranking algorithms: standard CNN (vanilla), co-teaching [12], and self-
supervised pre-training with fine-tuning [13]. All approaches employed the
same image encoder (ResNet-50), augmentations, and optimiser type (see
supplement for implementation details). At each iteration, new labels are
sampled from the true label distribution, and experiments are repeated with 5
independent random seeds. Performance of selection algorithms is compared to
two baselines: the oracle and random selector, which determine respectively
the upper and lower bounds. The random selector chooses the next sample to be
annotated uniformly at random as done in previous studies [9, 2], while the
oracle simulates the ideal ranking method by having access to true label
distribution. Knowledge of the true distribution enables the oracle to
prioritise least difficult noisy samples first, hence maximising the number of
corrected samples for a given resource constraint. The maximum relabelling
budget ($B$) in the simulation is set to the expected number of re-annotations
required by the oracle to clean all mislabelled samples.

(a)

(b) Figure 3: Results of the label cleaning simulation on training datasets. a
NoisyCXR ($\eta=12.7\%$); b CIFAR10H ($\eta=15\%$). For a given number of
collected labels ($x$-axis), a cost-efficient algorithm should maximise the
number of samples that are now correctly labelled ($y$-axis). The correctness
of acquired labels is measured in terms of accuracy. The area-under-the-curve
(AUC) is reported as a summary of cleaning efficiency of each selector across
different relabelling budgets. The upper and lower bounds are set by oracle
(blue) and random sampling (red) strategies. The pink curve (a) illustrates
the practical upper bound of cleaning performance when the selector model is
trained solely on clean labels.

Table 2: Classification accuracy (%) before and after label cleaning.

| Selector | Scoring | Classifier |  Before cleaning |  After cleaning
---|---|---|---|---|---
(1) | Vanilla | Eq. 2 | Vanilla |  64.1 |  68.4
(2) | Vanilla |  BALD[23] | Vanilla |  64.1 |  68.3
(3) | SSL | Eq. 2 | Vanilla |  64.1 |  70.9
(4) | SSL | Eq. 2 | SSL |  78.7 |  80.3
(5) | Co-teaching | Eq. 2 | Co-teaching |  66.5 |  68.8
(6) | – | – |  ELR[24] |  67.0 |  –
(7) | (Clean training) | Vanilla |  73.6 |  –
(8) | (Clean training) | SSL |  80.7 |  –

Models are evaluated on a clean test set ($N=50k$) before and after
relabelling $32.7\%$ of samples in the training set (CIFAR10H, $N=5k$,
$\eta=30\%$).

### Cleaning noisy training labels improves predictive accuracy.

All label cleaning models are first trained on their respective training
datasets (CIFAR10H and NoisyCXR), and subsequently used to rank samples within
the same set for relabelling. In the rest of the manuscript, the term
“selector” refers to the models used for label cleaning. Table 2 shows that
all training methods benefit from newly acquired labels regardless of their
underlying weight initialisation (e.g. self-supervised learning – SSL) and
noise-robust learning (NRL) strategy, as in ELR[24]. In that regard, label
cleaning serves as a complementary approach to NRL by recycling data samples
containing noisy labels. Additionally, the comparison between rows 3 and 6
show that acquiring a new set of labels can yield significantly better results
than state-of-the-art NRL models trained on noisy labels, which performs
closer to the upper-bound (row 7) where training is done with all true labels.
Class imbalance, inherent noise model, and difficulty of samples are expected
to introduce challenges to most NRL methods.

### Label cleaning can be done in a resource-effective manner.

In Table 2, we also observe that sample ranking methods (selectors) can
influence the outcome of label cleaning efforts in terms of predictive
accuracy (see rows 1–3). To further investigate the impact of selector types,
we carried out label cleaning simulations (see Fig. 3) on both CIFAR10H and
NoisyCXR datasets for initial noise rates of $\eta=15\%$ and $12.7\%$,
respectively. The methods are compared quantitatively regarding fraction of
corrected labels (i.e. reduction in noise rate) for varying number of re-
annotations. Additionally, the sample ranking is visually illustrated in Fig.
4 along with the predicted label correctness and ambiguity scores.

Figure 4: Ranking of CIFAR10H samples (15% initial noise rate) by the SSL-
Linear algorithm. The top row illustrates a representative subset of images
ranked at the top-10 percentile with the highest priority for relabelling.
Similarly, the second and third rows correspond to 25–50 and 50–75
percentiles, respectively. At the bottom, ambiguous examples that fall into
the bottom 10% of the list ($N=2241$) are shown. Each example is shown
together with its true label distribution to highlight the associated
labelling difficulty. This can be compared against the label noisiness (cross-
entropy; CE) and sample ambiguity (entropy; AMB) scores predicted by the
algorithm (see Eq. 2), shown above each image. As pointed out earlier,
adjudication of samples provided at the bottom does require a large number of
re-annotations to form a consensus. The authors in [14] explore the causes of
ambiguity observed in these samples.

It is apparent that, for a fixed improvement in number of corrected labels,
all proposed selectors require much fewer re-annotations than random. For
instance, the vanilla selector can reach 90% correct labels with 3$\times$
(NoisyCXR) or 2.5$\times$ (CIFAR10H) fewer re-annotations than random. More
specifically, we can see that noise-robust learning (here, co-teaching)
benefits selection algorithms in prioritising noisy labels. Additionally, SSL
yields further performance improvements by learning generic semantic
representations without requiring labels, which in principle reduces chances
of memorising noise. In that regard, it is a complementary feature to
traditional noise-robust learning approaches as long as there is enough data
for training. In particular, SSL pre-training yields improved noisy sample
detection on NoisyCXR dataset in comparison to initialisation from scratch or
pre-trained ImageNet weights. Figure 5 exemplifies clear noisy and difficult
cases for NoisyCXR.

The experiments are also repeated for larger noise rates (see Supplementary
Figure 2) to identify the limits beyond which sample selection algorithms
deteriorate towards random sampling. The results show similar performance as
long as diagonal dominance [25] holds on average in class confusion matrices
(see Fig. 6), which is verified by experimenting with uniform and class-
dependent noise models [26]. Moreover, graph-based selectors outperform the
ones with linear head on higher noise rates, experimentally suggesting that
transductive approaches may be more resilient against noise in these regimes.

### Sample scoring function impacts cleaning performance.

We study the influence of the self-entropy term ($H(\mathop{\mkern
0.0mup_{\theta}})$) used in Eq. 2 in an ablation study. We see that selectors
prioritising clear label noise cases yield higher numbers of corrected label
noise, and this effect is further pronounced by including the entropy term in
the scoring function (see Supplementary Figure 3).

Additionally, ranking samples for labelling is related to active learning,
whereby one annotates unlabelled instances in order to maximise model
performance metrics. Typical active-learning methods attempt to identify the
most informative examples [27], relying on e.g. core-sets [28] or high
predictive uncertainty [29]. A representative example is Bayesian active
learning by disagreement (BALD) [29, 23], which quantifies the mutual
information between each sample’s unknown label and the model parameters.
However, these criteria may simply prioritise under-represented parts of the
data space rather than samples with noisy labels.

To analyse the suitability for label cleaning of such an approach, we repeat
these experiments with the BALD score in replacement of the function in Eq. 2,
using the same relabelling budget. Rows 1–2 in Table 2 show that
classification accuracy gains on the test set are comparable between the two.
However, the label quality improvement is significantly inferior for BALD
(Supplementary Figure 2), as it favours samples with the highest
disagreement—which may not correspond to label noise. Thus, in addition to
improving model accuracy, active label cleaning offers the flexibility to
improve the quality of noisy datasets.

(a)

(b)

(c)

Figure 5: Chest X-ray images selected from NoisyCXR dataset, which do not
contain “pneumonia” label in the NIH dataset [3]. a Correctly identified noise
case with pneumonia-like opacities shown with bounding boxes. b Wrongly
flagged sample with a correct label; here the model confuses lung nodules with
pneumonia-like opacities. c A difficult case with subtle abnormality where
radiologists indicated medium-confidence in their diagnosis as shown by the
highlighted region (RSNA study [9]).

### Cleaning noisy evaluation labels mitigates confounding in model selection.

In noise-robust learning [12, 30, 31, 32], validation data is traditionally
assumed to be perfectly labelled and can be relied on for model benchmarking
and hyperparameter tuning. Furthermore, classification accuracy has been shown
to be robust against noise [25, 33] assuming that labelling errors and inputs
are conditionally independent, $P(\hat{y}\nonscript\mkern
1.5mu|\nonscript\mkern 1.5mu\mathopen{}\allowbreak
y,\mathbf{x})=P(\hat{y}\nonscript\mkern 1.5mu|\nonscript\mkern
1.5mu\mathopen{}\allowbreak y)$. While these assumptions may be valid in
certain applications, they often do not hold for real-world datasets, wherein
the labelling process clearly relies on examining the inputs. Thus, models can
learn biases from the training data that would correlate with labelling errors
in the evaluation set (see Fig. 6).

To illustrate the pitfalls of evaluating models on noisy labels, we have
conducted experiments on CIFAR10H using symmetric (SYM) and instance-dependent
noise (IDN) models [22] (see supplement for details). In these experiments,
diagonal dominance [25] is preserved for all confusion matrices, and the
statistical dependence between labelling errors, input, and true label are the
same in both training and validation sets. Furthermore, all the available
relabelling resources are used for cleaning the noisy labels in the evaluation
set instead of training data to obtain an unbiased estimate of model ranking
and performance.

ResNet-50 models [34] with different regularisation settings are trained with
noisy training labels ($\lvert\mathcal{D}_{\mathrm{train}}\rvert=10k$) and are
later used to clean labels on evaluation set
($\lvert\mathcal{D}_{\mathrm{eval}}\rvert=50k$). Labels for sets
$\mathcal{D}_{\mathrm{train}}$ and $\mathcal{D}_{\mathrm{eval}}$ are
determined for each experiment separately using SYM and IDN noise models with
40% noise rate on average. The experimental results (Table 3) show that
standard performance metrics and model ranking strongly depend on the
underlying conditions leading to labelling errors, which shows the need for
clean validation sets in practical applications for model selection.

Table 3: Classification accuracy under noisy evaluation labels.

| Model | True | Noisy (40%) | Cleaned (10%) | Cleaned (20%)
---|---|---|---|---|---
SYM | M1 | 73.32 | 45.19 (.21) | 54.37 (.13) | 62.53 (.09)
M2 | 80.16 |  48.93 (.23) |  58.45 (.10) | 67.42 (.10)
IDN | M1 | 69.91 | 45.93 (.10) | 49.97 (.06) |  55.87 (.06)
M2 | 65.76 |  46.50 (.13) | 49.90 (.12) | 55.28 (.10)

Co-teaching (M1) and SSL (M2) models are compared on a noisy CIFAR10H
validation set $\mathcal{D}_{\mathrm{eval}}$ over three runs using different
label initialisations. Both approaches use ResNet-50 with different weight
initialisation and regularisation. We compare classification accuracy on true,
noisy, and cleaned labels. M2 is deliberately less regularised (i.e. weight
decay) and is expected to perform worse on a more challenging IDN noise model.

To tackle these challenges, we applied the active relabelling procedure (SSL-
Linear) on this noisy validation set by using the same corresponding training
data ($\mathcal{D}_{\mathrm{train}}$). Through sample prioritisation and
relabelling of 10% of the entire set ($\mathcal{D}_{\mathrm{eval}}$), the bias
in model selection can be alleviated as shown in Table 3. At the end of the
label cleaning procedure, the noise rates are reduced to 31.32% (IDN) and
30.25% (SYM); in other words, 86.8% (IDN) and 97.5% (SYM) of the selected
images were initially mislabelled and then corrected. Here, IDN model is
observed to be more challenging for identifying noisy labels.

## Discussion

This work investigated the impact of label noise on model training and
evaluation procedures by assessing its impact in terms of (I) predictive
performance drop, (II) model evaluation results, and (III) model selection
choices. As potential ways to mitigate this problem can be resource-demanding
depending on the application area, we defined cost-effective relabelling
strategies to improve the quality of datasets with noisy class labels. These
solutions are benchmarked in a newly proposed simulation framework to quantify
potential resource savings and improvement in downstream use-cases.

In particular, we highlight the importance of cleaning labels in a noisy
evaluation set. We showed that neglecting this step may yield misleading
performance metrics and model rankings that do not generalise to the test
environment. This can, in turn, lead to overoptimistic design decisions with
negative consequences in high-stakes applications such as healthcare. One of
our main findings is that the patterns of label error in the data (i.e.,
structural assumptions about label errors shown in Fig. 6) can have as large
an impact on the efficacy of label cleaning and robust-learning methods as the
average noise rates, as evidenced by the results obtained on both training and
validation sets. We therefore recommend carefully considering the underlying
mechanisms of label noise when attempting to compare possible solutions.

The results also suggest that even robust-learning approaches may not fully
recover predictive performance under high noise rates. In such cases, SSL pre-
training is experimentally shown to be a reliable alternative, outperforming
noise-robust models trained from scratch, even more so with the increasing
availability of unlabelled datasets. Lastly, we show that acquiring new labels
can complement noise-robust learning by recycling data samples even if their
labels are noisy, and can also handle biased labels. Thus, the two domains can
be combined to obtain not only a better model, but also clean data labels for
downstream applications.

### Future work directions.

Although the present study focused on imaging, the proposed methodology is not
limited to this data modality, and empirical validation with other input types
is left for future work. It will also be valuable to explore, for example,
having the option to also annotate unlabelled samples, or actively choosing
the next annotator to label a selected instance. Such extensions to active
cleaning will significantly broaden its application scope, enabling more
reliable deployment of machine learning systems in resource-constrained
settings.

Similarly, multi-label fusion techniques [35, 36, 37, 38] can be used within
the proposed label cleaning procedure to restore true label distribution by
modelling labelling process and aggregating multiple noisy annotations. Such
approaches critically rely on the availability of multiple labels for each
sample—which can be realised towards the end of relabelling efforts.

(a)

(b)

(c)

Figure 6:  Understanding label noise patterns. a Different label noise models
used in robust learning. The statistical dependence between input image ($X$),
true label ($Y$), observed label ($\hat{Y}$), and error occurrence ($E$) is
shown with arrows (adapted from Frénay et al.[39]). b–c CIFAR10H class
confusion matrices (temperature $\tau=2$) for all samples (b) and difficult
samples only (c).

## Methods

### CIFAR10H dataset.

CIFAR10H [14, 5] is an image dataset comprising the test set of CIFAR10 (10k
images) and multiple labels collected from crowdsourcing. Each image on
average contains 51.1 labels distributed over 10 class categories. In our
study, the true label distributions were calculated by normalising the
histogram of these labels on a per image basis. In the experiments, we assume
that our initial dataset contains labelling errors. These partially noisy
labels were obtained by sampling the initial label by taking into account the
label distribution for each sample. The noise rate in experiments is
controlled by applying temperature scaling [40] ($\tau$) to each distribution,
which preserves the statistical dependence between input images and their
corresponding labels. The higher the temperature, the closer to a uniform
distribution the label distribution becomes, hence the higher the probability
of sampling a noisy label for a given sample.

Samples in this dataset are not all equally difficult to label or classify
[14]. For instance, for some images, all annotators agreed on the same label,
for some others collected labels were more diverse as the image was harder to
identify. Hence, we defined sample difficulty as a function of the normalised
Shannon entropy of its target distribution,
$\sum_{c=1}^{C}p_{c}\log_{C}p_{c}$. If the entropy associated to this
distribution was higher than a certain threshold (0.3 in our experiments), the
sample was classified as “difficult”. The entropy of the target distribution
can also be related to the expected number of relabellings required for this
sample (see problem definition in Results). In Fig. 6, we show the average
class confusion matrices over the CIFAR10H dataset for $\tau=2$. In Fig. 6,
the confusion matrix of only the difficult samples is shown to highlight the
classes that are confused the most in a standard labelling process.

Table 4: Correspondence of chest radiography labels from the RSNA challenge
[9], NIH [3], and NoisyCXR datasets.

|  RSNA labels [9]
---|---
| Pneumonia-like | No pneumonia-like
| opacity | opacity
NIH labels [3]: |  |
Pneumonia | 367 | 441
Consolidation/infiltration (not pneumonia) | 3,988 | 8,551
Other diseases only | 1,101 | 5,567
No finding | 556 | 6,113
NoisyCXR labels: |  |
Pneumonia-like opacity | 3,956 | 1,296
No pneumonia-like opacity | 2,056 | 19,376
Total | 6,012 | 20,672

The disagreements between two label sets are highlighted in bold font. Fields
in italic indicate an unclear agreement between both sets of labels. NoisyCXR
labels are collected from the original NIH labels where possible, and the
remainder are uniformly sampled (10%) from the “Consolidation/infiltration”
category to increase the noise rate further in the experiments.

### NoisyCXR: a medical benchmark dataset for label cleaning.

In the experiments, we used a subset of the medical image dataset released by
the NIH [3]. The original dataset contains 112k chest radiographs with labels
automatically extracted from medical records using a natural language
processing algorithm. In the original data release, labels are grouped into 14
classes (non-mutually exclusive) indicating the presence of various diseases
or no finding at all. However, there have been studies [8] showing errors in
these auto-extracted image labels. Later, the RSNA released the Pneumonia
Detection Challenge [9] aiming to detect lung opacities indicative of
pneumonia. The challenge dataset is comprised of 30k images randomly sampled
from the NIH ChestXray dataset mentioned above. To adjudicate the original
labels released by the NIH, a board of radiologists reviewed and re-assessed
each image on this dataset and delimited the presence of lung opacities
associated to pneumonia. Samples are hence classified into two categories:
“pneumonia-like lung opacity” and “no pneumonia-like lung opacity”. Here we
aim to utilise a realistically noisy version of this RSNA dataset encountered
in real-world application scenarios. To this end, the original NIH labels are
leveraged to construct a noisy version of the true labels released by the
RSNA. In more detail, we analysed the data in four categories (see rows in
Table 4) based on their original NIH labels and the class taxonomy [41] in
order to map this original label to a binary label indicating presence or
absence of pneumonia-like opacities:

  * •

Cases where the NIH label contains “Pneumonia”. All these samples were
assigned to the “pneumonia-like opacity” category in NoisyCXR.

  * •

Cases associated to opacities potentially linked to pneumonia: NIH label
contains “consolidation” or “infiltration” (but not “pneumonia”). For these
cases, it is unclear whether the original NIH label should be mapped to
“pneumonia-like opacity” or not [41, 42]. Hence, for these cases, in NoisyCXR,
we attributed the correct (final RSNA) label to 90% of the cases and flipped
the label of the remaining 10%.

  * •

Cases tagged as “no finding”, all assigned to the “no pneumonia-like opacity”
category.

  * •

Cases linked to other pathology (i.e., original label only contains pathology
unrelated to pneumonia e.g. pleural effusion, nodules, fluid [3]), all
assigned to the “no pneumonia-like opacity” category.

The obtained binary mapping hence gives us a set of noisy labels for the
“pneumonia-like opacity” classification task whereas the labels released as
part of the challenge are considered as the correct labels for the data
cleaning experiment. The final noisy dataset contains 26,684 images with a
noise rate of 12.6% (see Table 4). As in the case of CIFAR10H, the uncertainty
in the final labels permits to separate ambiguous from easy cases. In
particular, using the confidence associated to each pneumonia-like lung
opacity bounding box [43] we distinguished cases as follows:

  * •

If the final label was “Pneumonia-like opacity” but all bounding boxes were of
low confidence, we considered the case as difficult and define the label
distribution for relabel sampling to
$\mathbb{P}_{\mathrm{label}}(\text{Pneumonia opacity})=0.66$. If at least one
of the boxes had a medium or high confidence score, the case is considered
easy and we set $\mathbb{P}_{\mathrm{label}}(\text{Pneumonia opacity})=1$.

  * •

If the final label was “No Pneumonia-like opacity” but some readers delineated
opacities, the sample is considered ambiguous and it’s class label is set to
$\mathbb{P}_{\mathrm{label}}(\text{Pneumonia opacity})=0.33$, Otherwise, it is
considered easy and $\mathbb{P}_{\mathrm{label}}(\text{Pneumonia opacity})=0$.

### Learning noise-robust image representations.

Noise-robust learning (NRL) is essential for handling noise and
inconsistencies in labels which can severely impact the predictive performance
of models [6, 7]. Although, this type of machine learning is most commonly
employed to improve model’s predictive performance, in our setting, we use NRL
methods to obtain an unbiased sample scoring function in active label
cleaning. Interested readers should refer to the extended literature review
[44] for further information on this topic.

The sample selection algorithms presented in the next section heavily rely on
models trained with NRL methods; in particular, we are utilising methods from
two broad categories: model regularisation [45, 46, 47] and sample exclusion
approaches [12, 48, 49]. The former set of methods have been shown to be
effective against memorising noisy labels [6], by favouring simpler decision
boundaries. Sample exclusion approaches, on the other hand, aim to identify
noisy samples during training by tracking loss values of individual samples
[12, 49] or gradient vector distributions [24, 50]. Alternative meta-model
based approaches are not considered in this study as unbiased clean validation
set may not always be available in real-world scenarios.

### Sample selection algorithms.

Here we propose three selection algorithms for accurately estimating class
posteriors, $\mathop{\mkern 0.0mup_{\theta}}(\hat{y}\nonscript\mkern
1.5mu|\nonscript\mkern 1.5mu\mathopen{}\allowbreak\mathbf{x})$, to prioritise
mislabelled data points in the scoring function $\Phi$ (see Eq. 2). In detail,
we first train deep neural networks with noise robustness, then use them to
identify the corrupted labels. The methods differ in how the robustness is
introduced and how the networks are used afterwards. These methods were
proposed and studied to take into account different noise cleaning setups,
including the number of labelled and unlabelled samples, and prior knowledge
on the expected noise rate.

As a baseline approach, networks are trained with noisy labels and augmented
images by minimising a negative log-likelihood loss term, which is referred to
as vanilla. This approach is expected to be biased by the noisy labels and to
perform sub-optimally in prioritising samples for relabelling.

#### Supervised co-teaching training.

To cope with noisy labels, we first propose to use co-teaching [12] in a
supervised learning setting, which has shown promising performance in
classification accuracy and robustness. In the co-teaching scheme, two
identical networks $f_{\theta_{1}}$ and $f_{\theta_{2}}$ with different
initialisation are trained simultaneously, but the batch of images at each
training step for $f_{\theta_{1}}$ is selected by $f_{\theta_{2}}$ and vice
versa. The rationale is that images with high loss values during early
training are ones that are difficult to learn, because their labels disagree
with what the network has learned from the easy cases—indicating these labels
might be corrupted. Also, using two networks rather than one prevents a self-
confirmation loop as in [51]. Co-teaching thus provides a method for robust
training with noisy labels that can be employed to identify corrupted labels.
We ensemble the predictions of both trained networks as
$\frac{1}{2}\sum_{m}f_{\theta_{m}}(\mathbf{x}_{i})=\hat{\boldsymbol{\pi}}_{i}$,
where $\hat{\boldsymbol{\pi}}_{i}\in[0,1]^{C}$ and
$\sum_{c=1}^{C}\hat{\pi}_{i}^{c}=1$, and use the predicted distribution,
$\hat{\boldsymbol{\pi}}_{i}$, in the scoring function $\Phi$ to rank the
samples.

#### Self-supervision for robust learning.

Noise-robust learning can still be influenced by label noise and exclude
difficult samples, as they yield large loss values. Self-supervised learning
(SSL) is proposed as a second approach to address this and further improve the
sample ranking performance whenever an auxiliary task can be defined to pre-
train models. As the approach does not require target labels and only extracts
information from the images, it is an ideal way to learn domain-specific image
encoders that are unbiased by the label noise. It can also be used in
conjunction with the co-teaching algorithm in an end-to-end manner.

To this end we use BYOL [13] to learn an embedding for the images, i.e., a
non-linear transformation $\mathbf{x}_{i}\mapsto
g_{\theta}(\mathbf{x}_{i})=\mathbf{h}_{i}$, where
$\mathbf{h}_{i}\in\mathds{R}^{L}$ and $L$ is the size of the embedding space.
As SSL is agnostic to the labels, the embedding will be formed based only on
the image content, such that similar images will be close in the embedding
space and dissimilar images will be further apart. The learnt encoder can
later be used to build noise-robust models. One way to achieve this is by
fixing the weights of $g_{\theta}$ and train a linear classifier on top of the
learned embedding as $\mathbf{h}_{i}\mapsto
f_{\varphi}(\mathbf{h}_{i})=\hat{\boldsymbol{\pi}}_{i}$, which is referred to
as “SSL-Linear”. To obtain better calibrated posteriors, large logits are
penalised at training time with function
$\gamma(x)=\alpha\cdot\tanh(x/\alpha)$ [13] and optimised with smoothed labels
[52]. The linear classifier can only introduce a linear separation of the
images according to their given labels. Assuming images with similar contents
(i.e. nearby in embedding space) should have similar labels, the simple linear
decision boundary introduces robustness to mislabels.

#### Self-supervision with graph diffusion.

As an alternative to the linear classifier, we employ a graph classifier to
embody the idea that the label for each sample should be consistent with its
neighbourhood. We construct a graph based on the SSL embeddings
$\mathbf{h}_{i}$ (see above), as $g_{\theta}$ already optimises for their
cosine similarities during training. We thus define the graph’s affinity
matrix $\mathbf{W}$ as
$W_{ij}=\frac{1}{2}(1+\mathbf{h}_{i}^{\mkern-1.5mu\mathsf{T}}\mathbf{h}_{j}/\lVert\mathbf{h}_{i}\rVert\lVert\mathbf{h}_{j}\rVert)$
for $i\neq j$ and $W_{ii}=0$. We additionally set $W_{ij}=0$ for all but the
$K$ nearest neighbours $j$ of each node $i$. We then follow the regularised
label spreading approach [53], which minimises

$Q(\mathbf{F})=\frac{1}{2}\Big{(}\underbrace{\operatorname{tr}[\mathbf{F}^{\mkern-1.5mu\mathsf{T}}(\mathbf{I}-\mathbf{D}^{-\frac{1}{2}}\mathbf{W}\mathbf{D}^{-\frac{1}{2}})\mathbf{F}]}_{\mathclap{\smash[b]{\sum_{i,j}W_{ij}\big{\lVert}\mathbf{f}_{i}/D_{ii}^{1/2}-\mathbf{f}_{j}/D_{jj}^{1/2}\big{\rVert}^{2}}}}+\mu\lVert\mathbf{F}-\mathbf{Y}\rVert_{F}^{2}\Big{)},$
(5)

where $\mathbf{f}_{i}\in[0,1]^{C}$ are the soft labels of each node
($i^{\text{th}}$ row of $\mathbf{F}$), $\mathbf{D}$ is the degree matrix,
$\mathbf{Y}$ are the given labels of all nodes, $\mathbf{I}$ is the identity
matrix, and $\mu>0$ is a regularisation coefficient. The first term in Eq. 5
measures label consistency between neighbours, while the second is the
discrepancy between predicted and given labels. The optimal spread labels are
given in closed form as
$\mathbf{F}^{*}=\frac{\mu}{1+\mu}(\mathbf{I}-\frac{1}{1+\mu}\mathbf{D}^{-\frac{1}{2}}\mathbf{W}\mathbf{D}^{-\frac{1}{2}})^{-1}\mathbf{Y}$.

## Data availability

The CIFAR10H dataset is available at https://github.com/jcpeterson/cifar-10h.
The raw data for NoisyCXR can be downloaded from
https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/data, and the
labels used in our experiments are processed in the linked code repository.

## Code availability

All the code for our label cleaning benchmarks, robust learning models, and
experiments will be released in an open-source repository upon publication.

## References

  * [1] Northcutt, C. G., Athalye, A. & Lin, J.  Pervasive label errors in ML benchmark test sets, consequences, and benefits.  In _NeurIPS 2020 Workshop on Security and Data Curation Workshop_ (2020).
  * [2] Majkowska, A. _et al._ Chest radiograph interpretation with deep learning models: assessment with radiologist-adjudicated reference standards and population-adjusted evaluation.  _Radiology_ 294, 421–431 (2020).
  * [3] Wang, X. _et al._ ChestX-Ray8: Hospital-scale chest X-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 3462–3471 (2017).
  * [4] Beyer, L., Hénaff, O. J., Kolesnikov, A., Zhai, X. & van den Oord, A.  Are we done with ImageNet?  _arXiv preprint arXiv:2006.07159_ (2020).
  * [5] Peterson, J. C., Battleday, R. M., Griffiths, T. L. & Russakovsky, O.  Human uncertainty makes classification more robust.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 9617–9626 (2019).
  * [6] Arpit, D. _et al._ A closer look at memorization in deep networks.  In _Proceedings of the 34th International Conference on Machine Learning_ , vol. 70 of _PMLR_ , 233–242 (PMLR, 2017).  arXiv:1706.05394.
  * [7] Zhang, C., Bengio, S., Hardt, M., Recht, B. & Vinyals, O.  Understanding deep learning requires rethinking generalization.  In _International Conference on Learning Representations_ (2017).  URL https://openreview.net/forum?id=Sy8gdB9xx.
  * [8] Oakden-Rayner, L.  Exploring large-scale public medical image datasets.  _Academic Radiology_ 27, 106 – 112 (2020).  URL http://www.sciencedirect.com/science/article/pii/S107663321930488X.  Special Issue: Artificial Intelligence.
  * [9] RSNA pneumonia detection challenge.  https://www.kaggle.com/c/rsna-pneumonia-detection-challenge (2018).
  * [10] Krause, J. _et al._ Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy.  _Ophthalmology_ 125, 1264–1272 (2018).
  * [11] Gulshan, V. _et al._ Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs.  _JAMA_ 316, 2402–2410 (2016).
  * [12] Han, B. _et al._ Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _Advances in Neural Information Processing Systems_ , 8527–8537 (2018).
  * [13] Grill, J.-B. _et al._ Bootstrap your own latent - a new approach to self-supervised learning.  In _Advances in Neural Information Processing Systems_ , vol. 33, 21271–21284 (2020).
  * [14] Battleday, R. M., Peterson, J. C. & Griffiths, T. L.  Capturing human categorization of natural images by combining deep networks and cognitive models.  _Nature Communications_ 11, 1–14 (2020).
  * [15] Lin, C. H., Mausam & Weld, D. S.  Re-active learning: Active learning with relabeling.  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ , 1845–1852 (AAAI Press, 2016).
  * [16] Ipeirotis, P. G., Provost, F., Sheng, V. S. & Wang, J.  Repeated labeling using multiple noisy labelers.  _Data Mining and Knowledge Discovery_ 28, 402–441 (2014).
  * [17] Ekambaram, R. _et al._ Active cleaning of label noise.  _Pattern Recognition_ 51, 463–480 (2016).
  * [18] Northcutt, C. G., Jiang, L. & Chuang, I. L.  Confident learning: Estimating uncertainty in dataset labels.  _arXiv preprint arXiv:1911.00068_ (2019).
  * [19] Grandvalet, Y. & Bengio, Y.  Semi-supervised learning by entropy minimization.  In _Proceedings of the 17th International Conference on Neural Information Processing Systems_ , 529–536 (2004).
  * [20] Pereyra, G., Tucker, G., Chorowski, J., Kaiser, Ł. & Hinton, G.  Regularizing neural networks by penalizing confident output distributions.  _arXiv preprint arXiv:1701.06548_ (2017).
  * [21] Krizhevsky, A.  Learning multiple layers of features from tiny images.  Tech. Rep., University of Toronto (2009).
  * [22] Xia, X. _et al._ Part-dependent label noise: Towards instance-dependent label noise.  In _Advances in Neural Information Processing Systems_ , vol. 33 (2020).
  * [23] Gal, Y., Islam, R. & Ghahramani, Z.  Deep Bayesian active learning with image data.  In _International Conference on Machine Learning_ , 1183–1192 (PMLR, 2017).
  * [24] Liu, S., Niles-Weed, J., Razavian, N. & Fernandez-Granda, C.  Early-learning regularization prevents memorization of noisy labels.  _Advances in Neural Information Processing Systems_ 33 (2020).
  * [25] Chen, P., Ye, J., Chen, G., Zhao, J. & Heng, P.-A.  Robustness of accuracy metric and its inspirations in learning with noisy labels.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, 11451–11461 (2021).
  * [26] Patrini, G., Rozza, A., Krishna Menon, A., Nock, R. & Qu, L.  Making deep neural networks robust to label noise: A loss correction approach.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 1944–1952 (2017).
  * [27] Toneva, M. _et al._ An empirical study of example forgetting during deep neural network learning.  In _International Conference on Learning Representations_ (2018).
  * [28] Sener, O. & Savarese, S.  Active learning for convolutional neural networks: A core-set approach.  In _International Conference on Learning Representations_ (2018).
  * [29] Houlsby, N., Huszár, F., Ghahramani, Z. & Lengyel, M.  Bayesian active learning for classification and preference learning (2011).  URL https://arxiv.org/abs/1112.5745.
  * [30] Junnan Li, S. C. H. H., Richard Socher.  DivideMix: Learning with noisy labels as semi-supervised learning.  In _International Conference on Learning Representations_ (2020).
  * [31] Nguyen, D. T. _et al._ SELF: Learning to filter noisy labels with self-ensembling.  In _International Conference on Learning Representations_ (2020).
  * [32] Zhang, Z. & Sabuncu, M.  Generalized cross entropy loss for training deep neural networks with noisy labels.  In _Advances in Neural Information Processing Systems 31_ , 8778–8788 (2018).
  * [33] Lam, C. P. & Stork, D. G.  Evaluating classifiers by means of test data with noisy labels.  In _IJCAI_ , vol. 3, 513–518 (2003).
  * [34] He, K., Zhang, X., Ren, S. & Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 770–778 (2016).
  * [35] Khetan, A., Lipton, Z. C. & Anandkumar, A.  Learning from noisy singly-labeled data.  In _International Conference on Learning Representations_ (2018).  arXiv:1712.04577.
  * [36] Rodrigues, F., Pereira, F. & Ribeiro, B.  Learning from multiple annotators: distinguishing good from random labelers.  _Pattern Recognition Letters_ 34, 1428–1436 (2013).
  * [37] Tanno, R., Saeedi, A., Sankaranarayanan, S., Alexander, D. C. & Silberman, N.  Learning from noisy labels by regularized estimation of annotator confusion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 11244–11253 (2019).
  * [38] Warfield, S. K., Zou, K. H. & Wells, W. M.  Simultaneous truth and performance level estimation (STAPLE): An algorithm for the validation of image segmentation.  _IEEE Transactions on Medical Imaging_ 23, 903–921 (2004).
  * [39] Frénay, B. & Verleysen, M.  Classification in the presence of label noise: a survey.  _IEEE Transactions on Neural Networks and Learning Systems_ 25, 845–869 (2014).
  * [40] Guo, C., Pleiss, G., Sun, Y. & Weinberger, K. Q.  On calibration of modern neural networks.  In _International Conference on Machine Learning_ , 1321–1330 (2017).
  * [41] Irvin, J. _et al._ CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 590–597 (2019).
  * [42] Oakden-Rayner, L.  Quick thoughts on ChestXray14, performance claims, and clinical tasks (2017).  URL https://lukeoakdenrayner.wordpress.com/2017/11/18/quick-thoughts-on-chestxray14-performance-claims-and-clinical-tasks/.
  * [43] Stein, A.  Pneumonia boxes likelihood data – RSNA pneumonia detection challenge (2020).  URL https://www.kaggle.com/c/rsna-pneumonia-detection-challenge/discussion/150643.
  * [44] Song, H., Kim, M., Park, D. & Lee, J.-G.  Learning from noisy labels with deep neural networks: A survey.  _arXiv preprint arXiv:2007.08199_ (2020).
  * [45] Krogh, A. & Hertz, J. A.  A simple weight decay can improve generalization.  In _Advances in Neural Information Processing Systems 4_ , 950–957 (Morgan-Kaufmann, 1992).
  * [46] Lukasik, M., Bhojanapalli, S., Menon, A. & Kumar, S.  Does label smoothing mitigate label noise?  In _Proceedings of the 37th International Conference on Machine Learning_ , 6448–6458 (2020).
  * [47] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ 15, 1929–1958 (2014).
  * [48] Huang, J., Qu, L., Jia, R. & Zhao, B.  O2U-Net: A simple noisy label detection approach for deep neural networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 3326–3334 (2019).
  * [49] Yu, X. _et al._ How does disagreement help generalization against label corruption?  In _International Conference on Machine Learning_ (2019).
  * [50] Parascandolo, G., Neitz, A., Orvieto, A., Gresele, L. & Schölkopf, B.  Learning explanations that are hard to vary.  _arXiv preprint arXiv:2009.00329_ (2020).
  * [51] Jiang, L., Zhou, Z., Leung, T., Li, L.-J. & Fei-Fei, L.  MentorNet: Learning data-driven curriculum for very deep neural networks on corrupted labels.  In _International Conference on Machine Learning_ , 2309–2318 (2018).
  * [52] Müller, R., Kornblith, S. & Hinton, G. E.  When does label smoothing help?  In _Advances in Neural Information Processing Systems_ , vol. 32 (2019).
  * [53] Zhou, D., Bousquet, O., Lal, T. N., Weston, J. & Schölkopf, B.  Learning with local and global consistency.  In _Advances in Neural Information Processing Systems 16_ , 321–328 (2004).


[a]Deivid Ribeiro

# Prototype Schwarzschild-Couder Telescope for the Cherenkov Telescope Array:
Commissioning the Optical System

###### Abstract

A prototype Schwarzschild-Couder Telescope (pSCT) has been constructed at the
Fred Lawrence Whipple Observatory as a candidate for the medium-sized
telescopes of the Cherenkov Telescope Array Observatory (CTAO). CTAO is
currently entering early construction phase of the project and once completed
it will vastly improve very high energy gamma-ray detection component in
multi-wavelength and multi-messenger observations due to significantly
improved sensitivity, angular resolution and field of view comparing to the
current generation of the ground-based gamma-ray observatories H.E.S.S., MAGIC
and VERITAS. The pSCT uses a dual aspheric mirror design with a $9.7$ m
primary mirror and $5.4$ m secondary mirror, both of which are segmented. The
Schwarzschild-Couder (SC) optical system (OS) selected for the prototype
telescope achieves wide field of view of $8$ degrees and simultaneously
reduces the focal plane plate scale allowing an unprecedented compact ($0.78$m
diameter) implementation of the high-resolution camera ($6$mm/ $0.067$deg per
imaging pixel with $11,328$ pixels) based on the silicon photo-multipliers
(SiPMs). The OS of the telescope is designed to eliminate spherical and
comatic aberrations and minimize astigmatism to radically improve off-axis
imaging and consequently angular resolution across all the field of view with
respect to the conventional single-mirror telescopes. Fast and high imaging
resolution OS of the pSCT comes with the challenging submillimeter-precision
custom alignment system, which was successfully demonstrated with an on-axis
point spread function (PSF) of $2.9$ arcmin prior to the first-light detection
of the Crab Nebula in 2020. Ongoing commissioning activities aim to meet the
on-axis PSF design goal of $2.6$ arcmin, verify the off-axis performance of
the pSCT OS, and develop techniques to maintain alignment stability over
telescope structural deformations from pointing and temperature variations. In
this contribution, we report on the commissioning status, the optical
alignment procedures adopted for segmented OS, and alignment progress to
verify and validate design requirements.

##  1 Introduction

Recent advances in the observational capabilities of gravitational waves and
detectors of very-high-energy (VHE) neutrinos and gamma-rays have expanded the
field of multi-messenger astrophysics. To further explore the new sources
found by these detectors, which encompass high-energy, transient, point-like
or extended sources, observational synergies require improvements to the VHE
detectors for photons with energies greater than 100 GeV.

The Earth’s atmosphere interacts with VHE primary photons, creating cascades
of secondary particles that emit Cherenkov radiation broadly distributed
between $250$ nm and $800$ nm, peaking around $350-400$ nm. The arrays of
imaging atmospheric Cherenkov telescopes (IACTs) indirectly observe VHE
photons by stereoscopic imaging of the shower cascades and reconstructing
their energy and direction from the ground. Current-generation of small arrays
of IACTs use prime-focus OS based on the Davies-Cotton (DC) or parabolic
design, which consists of a single segmented mirror with large aperture (>10
m) and $f/\\#$ ratio in the range of $1.2-1.4$. As an alternative, the design
described in this proceeding, the fast Schwarzschild-Couder OS ($f/0.578$)
uses two segmented aplanatic mirrors to achieve the best possible imaging
resolution in the CTA core energy range (0.1 TeV - 10 TeV). These optical
improvements enable a high-resolution SiPM camera (see [1]), but require sub-
mm and sub-mrad alignment precision.

The initial alignment of the segmented primary (M1) and secondary (M2) mirrors
based on the mirror panel edge sensors (MPESs), which reduced relative
positional accuracy of the mirror segments from the telescope structure
manufacturing errors of $\sim 10$ mm to about $0.3$ mm is described in [2, 3].
These proceedings review the ongoing development of the optical alignment
methods utilizing de-focused images of bright stars to achieve relative
positional accuracy of M1 and M2 mirror segments relevant to the point spread
function (PSF) degrees of freedom (d.o.f.) of the order of $30$ microns or
better. A particular attention is given to the alignment of the M2 inner ring
of mirror segments, which is the key to achieving design requirement off-axis
performance. This report is structured as follows: section 2 reviews the
overall optical design and subsystems; section 3 reviews the optical alignment
procedures; section 4 reports the commissioning of an alignment method for
panels of the M2 inner ring; section 5 reports on the ability to maintain the
alignment with changes in elevation; and section 6 summarizes our work.

##  2 Overview of The pSCT Optical System

The SCT OS is designed to fully correct for spherical and comatic aberrations
in the full $8$ degrees field of view (FoV). Astigmatism dominates PSF at the
edge of the FoV and it is minimized by implementing a curved focal plane. The
choice of a two-mirror design with a de-magnifying secondary mirror reduces
the focal plane (FP) plate scale to provide compatibility with the SiPM
detectors [4]. The $9.7$m diameter primary mirror (M1) is segmented into $48$
mirror panels, split between two rings: an inner ring (P1) of $16$ panels and
an outer ring (P2) of $32$ panels. Similarly, the $5.4$m-diameter secondary
mirror (M2) is segmented into $24$ mirror panels, split between two rings: an
inner ring (S1) of $8$ panels and an outer ring (S2) of $16$ panels. The focal
length of the fast SC OS is $5.586$m. To achieve the point spread function
(PSF) of the OS in the FoV compatible with the SiPM pixel size ($6$mm) the
sub-mm and sub-mrad alignment is required [5].

To achieve the alignment goals of the pSCT, a central computer monitors and
controls the two major subsystems: the _global_ alignment system (GAS) that
measures the relative positions of the main optical elements such as M1, M2
and FP; and the _panel-to-panel_ alignment system (P2PAS) that measures the
relative misalignment between neighbouring panels using mirror panel edge
sensors (MPESs). The motion of each panel is controlled by the six actuators
assembled into a Stewart Platform [6]. The controller board (CB) and micro-
computer installed on each panel controls all six d.o.f. of each panel with
the precision of $3$ microns per step. The panel CBs and central computer
communicate with software built using the OPC-UA protocol [7]. Each CB can
receive commands in parallel, enabling fast, system-wide motion of all panels.

##  3 Overview of Optical Alignment Procedure

The optical PSF is most sensitive to the tip/tilt rotation of the M1 and M2
panels and also the translations of the M2 panels. These degrees of freedom
are constrained using de-focused images of stars. This section provides a
brief summary of the optical alignment procedure when the inner ring of S1
panels of M2 is assumed to be aligned (see next section). For more detailed
information on this alignment procedure and earlier results, see [2, 3].

The de-focused image of a star is created by the reflection from a pair of M1
and M2 panels in which each panel is purposely misaligned to form patterns of
rings of individual pairs so that the 80 images created by all possible pairs
in the OS are identifiable.

Images of a star produced by pairs of panels P1-S1, P2-S1, and P2-S2 are
arranged into three concentric ring configurations, which are shown in Figure
1: the aligned panels in the focused configuration are shown in Figure 1 (A),
while two different states of the de-focused configuration are shown in
Figures 1 (B) and (C). These de-focused configurations facilitate easy
identification of images produced by each combination of primary and secondary
panels and enables simultaneous and efficient alignment of each image into the
predetermined pattern, which must be produced by a star. In addition, the
characteristics of individual images (major and minor axes and elongation) are
used to guide the relative global positioning of M1, M2 and FP.

To relate the motion of the images on the FP to the adjustments of the
actuators of Stewart platforms, the response matrices for each panel are
simultaneously measured at the 3-ring configuration (Figure 1 (C)) by rotating
(tip and tilt) each mirror panel and observing the motion of the corresponding
focal-plane image.

Based on the response matrices, the displacement between predetermined pattern
of pairs of images of the star with respect to the focal point can be used to
guide the precise tip/tilt rotation of each panel in order to achieve the
focus of the OS. This procedure using measured response matrices constitutes
the zeroth-order alignment of the mirror panels, which is not yet ideal
because the response matrices are measured at the 3-ring configurations
instead of at the focused configuration where linear approximation may be
insufficient. To further improve alignment, the development of the 1st-order
alignment correction at the focal point is currently in progress.

Figure 1: A general overview of the alignment process using a de-focused star
projected on the focal plane. The diagram perspective shows a cross-section of
primary mirror panels P1 and P2; secondary mirror panels S1 and S2; and the
focal plane. The telescope is pointing toward a star along optical axis
(upward in diagram). A) The aligned panels project a single focus point of the
star. B) Primary panels P1 and P2 are rotated radially outward to de-focus the
star into two concentric rings, projecting the individual panels from each P1
and P2 sectors reflecting off the aligned secondary panels. C) Secondary
panels S2 are rotated radially outward to defocus the star even further,
separating the projection of P2 panels from their S1 and S2 counterparts.

##  4 Alignment of the S1 Panels

Utilizing the three-ring de-focused on-axis images of a star the P1, P2, and
S2 panels can be aligned precisely to the focus while the S1 panels remain
fixed. Any mis-alignment in S1 ring therefore will be corrected on-axis by the
accurate "mis-alignment" of the P1, P2 and S2 panels. This degeneracy in the
on-axis alignment is not realized for off-axis imaging and, therefore, the
precise optical alignment of S1 ring is the key to off-axis optical imaging.

The alignment of S1 panels, however, requires a more complicated procedure,
which was successfully developed and is described in this section. This
procedure, together with the three-ring configuration, makes for a nearly
complete set of methods of optical alignment of the pSCT using de-focused
images of stars.

Nominally, the light from an on-axis star reflected from a P1 panel will be
reflected from a single S1 panel before arriving at the focal plane. As
illustrated by Figure 2 (A), a tangential rotation (about the radial axis) of
a P1 panel will change the optical path so that the reflected light from this
panel will be reflected from two neighboring S1 panels. If the neighboring S1
panels are not properly aligned relative to one another, the light from the P1
panel will produce two optical images on the focal plane. The relative
brightness of the two split images can be adjusted by the degree of tangential
motion of the P1 panels. Therefore, this pair of S1 panels can be aligned by
applying rotation to one of the two S1 panels so that the pair of split images
is merged (as illustrated by Figure 2 (B)).

Figure 2: A general overview of the S1 alignment process. A) Assuming that S1
panels are misaligned, P1 panels are rotated tangentially to create a second
set of centroids on the focal plane. B) S1 panels are rotated tangentially to
remove the new centroids, therefore aligning the S1 panels. C) Focal plane
image of the projected centroids. Inset: 2 pairs of split images visible by
the misalignment of S1 panels, to be merged and corrected.

The pSCT optical support structure (OSS) incorporates twice as many M1 panels
as M2 panels. Therefore, the on-axis incident light that reaches each S1 panel
is first reflected from two P1 panels. These two P1 panels can be rotated in
opposite directions so that any misalignment between the S1 panel and both of
its neighboring S1 panels are detected. Following this pattern, all 16 P1
panels are rotated about their radial axes in alternating directions, creating
a maximum of 32 images if all S1-S1 edges are misaligned. Figure 2 (C) shows
an example of such an OSS configuration, with the zoom-in inset illustrating
the split image pairs due to S1-S1 misalignment.

Applying a tip or tilt motion of an S1 panel will cause the separation of two
pairs of split images to change (due to the chosen alternating direction of
the applied P1 rotation), and a response matrix can be measured. With the
response matrices of S1 panels as well as the separation between the split
images, S1 panels can be aligned with respect to each other. Note that not all
images have a split counterpart, because three S1-S1 edges are blocked by the
OSS support beams. For these three S1-S1 edges, pointing at a $1.2$ degrees
offset from a bright star, in combination with an appropriate degree of P1
rotation, has been verified to successfully avoid the beam obscuration. As of
June 2021, the tip/tilt of all S1 panels were aligned using the method
described above.

##  5 Elevation Dependence of On-Axis PSF

The structural deformations due to changing gravitational load of the pSCT OSS
at different elevation angles leads to deviations in panel alignment and
affects the PSF in non-negligible ways. In this section, we describe how we
directly measure the on-axis optical PSF as a function of elevation to
quantify the effect of the structural deformations.

The process of aligning the panels to create a focused star on the focal plane
assumes a single set of rotations of all panels from the same initial state.
In this initial state, the panels are positioned in the predetermined "ideal"
configuration for the de-focused star pattern. To align the mirror panels to
fully focused configuration at any elevation, the same set of tip/tilt
rotations should be applied if the panels are positioned in the predetermined
ideal de-focused star pattern in that elevation. A database was created to
save this starting state for all panel positions for each elevation.

The procedure to create the database consisted of three steps, designed to
maintain internal alignment between panels for each ring pattern and a
consistent central position. In the first step, the centroids for the P1 ring
were found on the focal plane and corrections were found for the ideal ring
configuration. After moving these panels to the ideal P1 ring configuration,
the focal plane image was again digitized to find both P1 and P2 ring
centroids. This process was repeated a third time to correct the S2 panel
positions (see inset in (B) of Figure 1). With all three rings corrected, the
state was saved for this elevation angle and the entire procedure was repeated
for another elevation angle.

All of the panels were then moved using the default set of rotations to
collapse the rings into the focal point (see inset in (A) of Figure 1). The
database of the de-focused star patterns per elevation was used to interpolate
the actuator lengths of all panels at any given elevation. Once interpolated,
this state was loaded to reproduce the de-focused star pattern, and then the
default rotations were applied to align the system.

The resulting on-axis PSF in the focused configuration was measured at each
elevation. The PSF is defined as $2\times\textrm{max}(\sigma_{x},\sigma_{y})$
from the fit of a 2-D Gaussian on the centroid image. This value is converted
to arcmin using the plate scale of the CCD pointing at the focal plane
(0.241/1.625 arcmin/px). An example of the focal point is shown in Figure 3,
showing Arcturus at 76° elevation with a PSF of $2.93$’.

Figure 4 shows the PSF in arcmin as a function of elevation achieved using
outlined above methodology. The overall behavior is a stable PSF averaging
3.1’ and getting better with increased elevation. To illustrate the necessity
of elevation corrections of PSF, we aligned the pSCT OS at 77° and performed
observations at 40° elevation which resulted in significantly degraded PSF of
$4.16$’. This larger PSF is highlighted in Figure 4 to illustrate how large
the PSF can get if active re-alignment is not attempted. This difference shows
the negative impact of failing to align with the proper elevation-dependent
state, revealing the effect of the structural deformation between extreme
elevation angles on the PSF.

In the process of measuring the elevation dependent alignment states, response
matrices mapping panel rotations to 2-D centroid translations were used to
position the panels in the focus position without non-linear correction, i.e.
using the zeroth-order response matrices describing linear approximation.
Iterating the motions of individual panels to minimize the offset from the
target focus positions constitute finding the first-order corrections, which
is part of an ongoing campaign. Applying the first-order corrections will
improve the net PSF on all corrected panels since each panel centroid will be
co-centered with every other panel, minimizing the spread of centroid offsets.
Initial results indicate that it is possible to achieve alignment of the P1
ring in which the PSF of the ring is entirely dominated by the PSF of
individual panels and the spreads of centroids is suppressed to a negligible
contribution.

The stability of the PSF based on pointing is an encouraging sign that full
alignment corrections may be applied to the telescope while operating. The
linear interpolation of actuator lengths between measured elevation states is
a successful step that can be applied for any elevation. The speed of panel
motion coupled with the relatively minor structural deformation dependent
corrections (<0.5 mm in actuators) indicate that alignment may be kept stable
throughout an observation, either actively at some frequency or in between
observation runs.

Figure 3: Image of Arcturus at 76° elevation. The PSF is 2.93’. The boxes
indicate the relative size of the imaging (8’) and trigger (4’) pixels of the
SiPM camera.  Figure 4: PSF measured per elevation. PSF is defined as
$2\times\textrm{max}(\sigma_{x},\sigma_{y})$. At 40°, the panel alignment
setting from 77° was loaded to measure the largest PSF deviation from the
optimal setting, shown in purple. Panel alignment settings derived without
corrections for all elevations.

##  6 Summary and Outlook

In these proceedings, a method was developed to improve the alignment of the
secondary mirror and the optical PSF was measured for a range of elevation
angles. The method to align S1 was proved successful. This is a crucial step
that ensures the correct tip/tilt alignment of all mirror panels of the pSCT
and off-axis pSCT performance. The optical PSF was stable for a wide range of
elevation angles, validating the creation of a database of panel
configurations that can be used for active alignment corrections based on
elevation.

Ray-tracing studies indicated an ideal on-axis optical PSF better than the
zeroth-order states measured per elevation angles here, which were stable at
3.1’. Future work on the optical system includes measurement of the first-
order corrections and PSF per panel. Single panel PSF will provide the lower
bound for the optical PSF alignment range, so each panel PSF will be studied.
Additionally, off-axis alignment will be measured and verified and the GAS
components will be commissioned.

## References

  * [1] L. P. Taylor, Design and performance of the prototype schwarzschild-couder telescope camera, in these proceedings, PoS (ICRC2021) 748, 2021.
  * [2] C. Adams and others, Verification of the optical system of the 9.7-m prototype Schwarzschild-Couder Telescope, in Optical System Alignment, Tolerancing, and Verification XIII (J. Sasián and R. N. Youngworth, eds.), vol. 11488, pp. 10 – 28, International Society for Optics and Photonics, SPIE, 2020.
  * [3] C. B. Adams and others, Alignment of the optical system of the 9.7-m prototype Schwarzchild-Coulder Telescope, in Ground-based and Airborne Telescopes VIII (H. K. Marshall, J. Spyromilio, and T. Usuda, eds.), p. 96, SPIE, Dec., 2020.
  * [4] V. Vassiliev, S. Fegan, and P. Brousseau, Astroparticle Physics 28 (Sept., 2007) 10–27.
  * [5] A. N. Otte and others, arXiv e-prints (Sept., 2015) arXiv:1509.02345.
  * [6] S. Sreenivasan, K. Waldron, and P. Nanua, Mechanism and Machine Theory 29 (1994) 855 – 864.
  * [7] W. Mahnke and S.-H. Leitner, ABB Review 3 (2009) 3.

## Full Authors List: CTA SCT Consortium

C. B. Adams1, G. Ambrosi2, M. Ambrosio3, C. Aramo3, P. I. Batista4, W.
Benbow5, B. Bertucci2,6, E. Bissaldi7,8, M. Bitossi9, A. Boiano3, C.
Bonavolontà3,10, R. Bose11, A. Brill1, J. H. Buckley11, R. A. Cameron12, R.
Canestrari13, M. Capasso14, M. Caprai2, C. E. Covault15, D. Depaoli16,17, L.
Di Venere7,8, M. Errando11, S. Fegan18, Q. Feng14, E. Fiandrini2,19, A.
Furniss20, A. Gent21, N. Giglietto7,8, F. Giordano7,8, E. Giro22, R.
Halliday23, O. Hervet24, T. B. Humensky25, S. Incardona26,27, M. Ionica2, W.
Jin28, D. Kieda29, F. Licciulli8, S. Loporchio7,8, G. Marsella26,27, V.
Masone3, K. Meagher21,30, T. Meures30, B. A. W. Mode30, S. A. I. Mognet31, R.
Mukherjee14, D. Nieto32,33, A. Okumura34, N. Otte21, F. R. Pantaleo7,8, R.
Paoletti35,9, G. Pareschi36, F. Di Pierro17, E. Pueschel4, D. Ribeiro1, L.
Riitano30, E. Roache5, J. Rousselle37, A. Rugliancich9, M. Santander28, R.
Shang38, L. Stiaccini35,9, L. P. Taylor30, L. Tosti2, G. Tovmassian39, G.
Tripodo26,27, V. Vagelli40,2, M. Valentino10,3, J. Vandenbroucke30, V. V.
Vassiliev38, D. A. Williams24, P. Yu38,

1Physics Department, Columbia University, New York, NY 10027, USA. 2INFN Sezione di Perugia, 06123 Perugia, Italy. 3INFN Sezione di Napoli, 80126 Napoli, Italy. 4Deutsches Elektronen-Synchrotron, Platanenallee 6, 15738 Zeuthen, Germany. 5Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA 02138, USA. 6Dipartimento di Fisica e Geologia dell’Università degli Studi di Perugia, 06123 Perugia, Italy. 7Dipartimento Interateneo di Fisica dell’Università e del Politecnico di Bari, 70126 Bari, Italy. 8INFN Sezione di Bari, 70125 Bari, Italy. 9INFN Sezione di Pisa, 56127 Pisa, Italy. 10CNR-ISASI, 80078 Pozzuoli, Italy. 11Department of Physics, Washington University, St. Louis, MO 63130, USA. 12Kavli Institute for Particle Astrophysics and Cosmology, SLAC National Accelerator Laboratory, Stanford University, Stanford, CA 94025, USA. 13INAF IASF Palermo, 90146 Palermo, Italy. 14Department of Physics and Astronomy, Barnard College, Columbia University, NY 10027, USA. 15Department of Physics, Case Western Reserve University, Cleveland, Ohio 44106, USA. 16Dipartimento di Fisica dell’Università degli Studi di Torino, 10125 Torino, Italy. 17INFN Sezione di Torino, 10125 Torino, Italy. 18LLR/Ecole Polytechnique, Route de Saclay, 91128 Palaiseau Cedex, France. 19Y. 20Department of Physics, California State University - East Bay, Hayward, CA 94542, USA. 21School of Physics & Center for Relativistic Astrophysics, Georgia Institute of Technology, Atlanta, GA 30332-0430, USA. 22INAF Osservatorio Astronomico di Padova, 35122 Padova, Italy. 23Dept. of Physics and Astronomy, Michigan State University, East Lansing, MI 48824, USA. 24Santa Cruz Institute for Particle Physics and Department of Physics, University of California, Santa Cruz, CA 95064, USA. 25Science Department, SUNY Maritime College, Throggs Neck, NY 10465. 26Dipartimento di Fisica e Chimica "E. Segrè", Università degli Studi di Palermo, via delle Scienze, 90128 Palermo, Italy. 27INFN Sezione di Catania, 95123 Catania, Italy. 28Department of Physics and Astronomy, University of Alabama, Tuscaloosa, AL 35487, USA. 29Department of Physics and Astronomy, University of Utah, Salt Lake City, UT 84112, USA. 30Department of Physics and Wisconsin IceCube Particle Astrophysics Center, University of Wisconsin, Madison, WI 53706, USA. 31Pennsylvania State University, University Park, PA 16802, USA. 32Institute of Particle and Cosmos Physics (IPARCOS), Universidad Complutense de Madrid, E-28040 Madrid, Spain . 33Department of EMFTEL, Universidad Complutense de Madrid, E-28040 Madrid, Spain. 34Institute for Space–Earth Environmental Research and Kobayashi–Maskawa Institute for the Origin of Particles and the Universe, Nagoya University, Nagoya 464-8601, Japan. 35Dipartimento di Scienze Fisiche, della Terra e dell’Ambiente, Università degli Studi di Siena, 53100 Siena, Italy. 36INAF - Osservatorio Astronomico di Brera, 20121 Milano/Merate, Italy. 37Subaru Telescope NAOJ, Hilo HI 96720, USA. 38Department of Physics and Astronomy, University of California, Los Angeles, CA 90095, USA. 39Instituto de Astronomía, Universidad Nacional Autónoma de México, Ciudad de México, Mexico. 40Agenzia Spaziale Italiana, 00133 Roma, Italy.