# Understanding top-down attention
using task-oriented ablation design

Freddie Bickford Smith
University of Oxford
<EMAIL_ADDRESS>&Brett D Roads
University College London
<EMAIL_ADDRESS>Xiaoliang Luo
University College London
<EMAIL_ADDRESS>&Bradley C Love
University College London
<EMAIL_ADDRESS>
###### Abstract

Top-down attention allows neural networks, both artificial and biological, to
focus on the information most relevant for a given task. This is known to
enhance performance in visual perception. But it remains unclear how attention
brings about its perceptual boost, especially when it comes to naturalistic
settings like recognising an object in an everyday scene. What aspects of a
visual task does attention help to deal with? We aim to answer this with a
computational experiment based on a general framework called task-oriented
ablation design. First we define a broad range of visual tasks and identify
six factors that underlie task variability. Then on each task we compare the
performance of two neural networks, one with top-down attention and one
without. These comparisons reveal the task-dependence of attention’s
perceptual boost, giving a clearer idea of the role attention plays. Whereas
many existing cognitive accounts link attention to stimulus-level variables,
such as visual clutter and object scale, we find greater explanatory power in
system-level variables that capture the interaction between the model, the
distribution of training data and the task format. This finding suggests a
shift in how attention is studied could be fruitful. We make publicly
available our code and results, along with statistics relevant to ImageNet-
based experiments beyond this one. Our contribution serves to support the
development of more human-like vision models and the design of more
informative machine-learning experiments.

##  1 Introduction

Ada is apparently nowhere to be seen in the crowded town square. Spotting her
among the hundreds of other people seems an impossible task. But a moment
later she sends a message telling you she is wearing her red coat. Within
seconds you process the few dashes of red in the crowd and you spot Ada. She
was hiding in plain sight.

At play here is the phenomenon of top-down attention. While the raw sensory
input, the pattern of light on the retina, does not change, the processing of
that input is dynamically modulated so that task-relevant visual features are
emphasised (Carrasco, 2011; Kruschke, 1992; Lindsay, 2020a). This is useful:
it boosts perceptual abilities. Intriguingly, it is more useful in some tasks
than in others. Suppose now you are trying to spot Ada but the town square is
not crowded at all. This time it is impossible _not_ to see her. Attention
cannot make much difference: there is little benefit to emphasising task-
relevant information. Empirical evidence agrees with this intuitive case
(Section 2).

The task-dependence of attention’s influence provides a window into issues at
the core of attention research. What aspects of a task does it help to deal
with? If it were absent, in what ways would a task become more challenging?
Cognitive science’s answers to these questions remain incomplete despite more
than a century of investigation (James, 1890; Helmholtz, 1896). While lab
experiments with simple visual stimuli have yielded valuable findings, it
remains an open question whether these findings hold in naturalistic settings
(Peelen & Kastner, 2014). This is a considerable gap in the theory of
attention, a gap that matters both for understanding human cognition and for
developing new machine-learning methods. Addressing this requires dealing with
the complicated, multivariate nature of real-world visual tasks. Seeking a
tractable means of going about this, we propose characterising _when_
attention helps as a way of explaining _how_ it helps. If attention produces a
perceptual boost in one task but not in another, the differences between the
tasks point to the mechanism at play.

Understanding how the influence of attention varies in naturalistic settings,
where sources of variation are complex and numerous, dictates a systematic and
large-scale experimental approach. Computational modelling with a
convolutional neural network (CNN) makes this possible by allowing us to scale
human-like perception to thousands of visual tasks: although not perfect, CNNs
are state-of-the art models of human vision (Kriegeskorte, 2015; Lindsay,
2020b; Serre, 2019; Yamins & DiCarlo, 2016). This approach also allows us to
examine phenomena internal to the model, such as the representational
similarity of stimuli, not just simple stimulus statistics. Our experiment
thus stands to help develop a richer cognitive account of visual perception.
At the same time, a careful analysis of attention’s influence in a CNN serves
to inform the development and deployment of machine-learning methods in
settings other than cognitive science.

We hypothesise that qualitative differences between visual tasks can be
captured in numerical quantities, and that these quantities will help explain
why attention is more useful in some tasks than in others. Each task we
consider consists of classifying images from a chosen task set, a pair of
classes from the ImageNet dataset (Russakovsky et al, 2015). Based on past
work as well as intuitions about how our model might behave, we identify six
quantitative dimensions along which there is substantial variation between
task sets. We then aim to establish a relationship between these task-set
properties and the extent to which attention produces a perceptual boost. Does
attention’s role in our model match findings from lab experiments using simple
stimuli, or is it altogether different when we consider more complex settings?
It is unclear at the outset which will be true. Our investigation reveals that
it is the latter: the influence of attention is strongly connected to _system-
level_ variables—those associated with the model, the distribution of training
data and the task format—rather than _stimulus-level_ variables that describe
visual appearance.

Figure 1:  Task-oriented ablation design (TOAD), a general experimental
framework for understanding the interaction between a machine-learning method
and the tasks it is applied to. The core idea is to make quantitative
comparisons between tasks, supporting more systematic exploration and more
granular analysis than in a typical ablation study, which involves comparing a
new method to a baseline method in terms of average performance on a standard
test dataset. TOAD results in a precise description of the relationship
between the nature of a task and the performance impact of a methodological
change. For each step, we show a simplified version of how we do that step in
our study of attention, with the highlighted element (in blue) corresponding
to a single task within the experiment. Other possible applications of TOAD
include understanding how the accuracy of a sentiment-analysis technique
varies with language style, how the error of a rainfall-forecast model varies
with the climate, and how the intersection-over-union metric of a medical-
imaging method varies between cancers.

In the process of testing our hypothesis, we define an experimental framework
applicable across a broad range of settings in machine-learning research.
Ablation studies provide crucial insight into the effects of changing an
algorithm or model. With task-oriented ablation design (TOAD), it is possible
to extract even more scientific understanding from each ablation study. The
core idea of TOAD is to make quantitative distinctions between tasks in order
to support more targeted experiments and more granular analysis than usual
(Figure 1). Despite its simplicity, TOAD is a powerful framework, allowing
detailed insights into the relationship between the nature of a task and the
effects of a methodological change. Our study of attention is a concrete
demonstration of this: TOAD makes it clear how to capitalise on the diversity
of ImageNet so as to understand the complex factors underlying the task-
dependence of attention’s perceptual boost.

As well as defining a broadly applicable experimental framework, we provide
data that is similarly transferable to other lines of inquiry. This data takes
the form of statistics describing how classes vary within ImageNet. For each
individual class, we provide three scores: clutter, difficulty and scale
(Figure 2). For each pair of classes, we provide three measures of similarity:
one based on CNN ouputs, one based on human judgements and one based on a
formal semantic hierarchy (Figure 3).

Figure 2:  Three class-level statistics that help quantify the diversity of
ImageNet. Clutter represents how much distracting information is present in a
typical image from the class. Images from high-clutter classes often include
multiple objects that have little relevance to the subject of the image; those
from low-clutter classes have few or no distracting cues. Difficulty is the
misclassification rate of an ImageNet-trained convolutional neural network
evaluated on images from the class. Images from high-difficulty classes are
often ambiguous. Scale is the relative size of the subject in a typical image
from the class. Images from high-scale classes are almost completely occupied
by the subject; those from low-scale classes have the subject appear in only a
small proportion of the pixels. For each statistic, we show the distribution
of values across ImageNet (coloured lines; lighter hue indicates higher score)
along with representative images from selected classes.

Our contribution is thus threefold. First, we systematically characterise the
perceptual boost of top-down attention in naturalistic visual tasks,
considering system-level variables not previously studied in this context. Our
findings contrast with expectations based on prior work, with implications for
how models of human visual perception should be analysed and developed.
Second, we define TOAD, a general framework for quantitatively understanding
the interaction between a methodological change and the tasks it is applied
to. This has the potential to see wide use in machine learning. Third, we
provide ready-to-use statistics describing the 1000 classes that comprise
ImageNet. These should help the research community better understand this
widely used dataset. Collectively, the three components of our contribution
stand to advance both cognitive science and machine learning.

##  2 Background and related work

Human visual perception is in a constant state of flux: the brain processes an
identical retinal stimulus differently whenever the task or context changes
(Carrasco, 2011). A key contributor to this flexibility is attention, which
emphasises the visual information most relevant in a given setting. Our work
concerns _top-down_ attention, where relevance depends on the task at hand and
the surrounding context; this contrasts with bottom-up attention, where
relevance is determined solely by the stimulus (Borji & Itti, 2013; Itti &
Koch, 2001). Top-down attention is known to enhance people’s performance on
visual tasks (Lindsay, 2020a). At the same time, it remains poorly understood
exactly how attention brings about its perceptual boost. What aspects of a
task does attention interact with? In what ways would a task become harder if
attention were not applied? Addressing these questions is crucial for
fundamental research in cognitive science but also for transferring ideas to
machine-learning methodology (Hassabis et al, 2017).

#### Task-dependence as a window into attention’s role

How can we better understand attention’s role in enhancing perceptual
abilities? We propose building on the finding that the influence of attention
varies between tasks. Attention’s effect on signal-detection sensitivity and
bias varies with task type (Downing, 1988); its modulation of visual area 1,
visual area 4 and the inferior temporal cortex varies with task difficulty
(Boudreau et al, 2006; Chen et al, 2008; Spitzer et al, 1988; Spitzer &
Richmond, 1991); its impact decreases with greater scene complexity (Rolls,
2008); its strength varies with the size of the stimulus relative to the
attentional field (Lindsay et al, 2020; Reynolds & Heeger, 2009); and its
modulation of the middle temporal visual area changes with the number of
stimuli in the receptive field (Lee & Maunsell, 2010). While this is
compelling evidence for the task-dependence of attention’s influence, it does
not necessarily tell the whole story. There are many different conceptions of
attention; the findings cited above are unlikely to all describe the same
mechanism. In addition, existing experimental data was gathered using a narrow
range of simplistic visual tasks, which to some degree reflects the
practicalities of lab studies in cognitive science (Barbosa et al, 2021).
There have been few studies of attention’s impact across a broad range of
naturalistic tasks (Peelen & Kastner, 2014).

Figure 3:  Three ways of measuring the similarity of a pair of ImageNet
classes. CNN similarity reflects how frequently the two classes are confused
for each other by an ImageNet-trained convolutional neural network. Human
similarity relates to how closely people associate the two classes, inferred
from comparisons they make between images in psychological tests. Semantic
similarity represents the conceptual proximity of the two classes, according
to the semantic hierarchy associated with ImageNet. To contrast the three
similarity measures, we show four query classes along with the most similar
classes according to each measure. For some queries, the three class rankings
are alike; for others, the rankings are notably different. The ‘lighter’ query
illustrates the source of differences: CNN similarity ranks classes based only
on visual appearance; semantic similarity ranks classes based only on abstract
concepts; human similarity involves a mixture of the two.

#### Studying attention in a convolutional neural network

Convolutional neural networks are state-of-the-art models of human visual
perception in terms of both explaining neural activity and predicting
behaviour (Lindsay, 2020b; Schrimpf et al, 2020). Computational modelling with
CNNs is thus a way of developing a cognitive account of naturalistic vision at
a level of complexity previously impossible due to the practical barriers
mentioned above. It also provides an exciting opportunity to exchange ideas
between cognitive science and machine learning (Luo et al, 2021). Top-down
attention can be modelled as a modulation of the bottom-up representation
computed by a standard CNN, analogous to the influence of the prefrontal
cortex on the visual cortex (Bar, 2003; Bichot et al, 2015; Lindsay, 2020a;
Paneri & Gregoriou, 2017; Rossi et al, 2007). This shapes the flow of
information to emphasise task-relevant features, supporting adaptation between
tasks.

Our model is intentionally straightforward and similar to those used in
related work. The foundation of the model is VGG16, a widely used CNN that
takes as input an image and produces as output a predicted label for the image
(Simonyan & Zisserman, 2015). Following Occam’s razor (Mackay, 2003), we use
one of the simplest possible attention mechanisms: attention modulates the
representation at the final convolutional layer, multiplying each feature map
by a separate weight. As well as being conceptually simple, this model closely
resembles the attention-augmented CNNs used by Bickford Smith et al (2020),
Lindsay & Miller (2018) and Luo et al (2021). This similarity supports
straightforward comparisons between past work and ours. It also allows us to
focus on drawing insights from the model instead of worrying about whether the
model is useful in the first place.

Three aspects of this model are valuable to note to understand how it fits
into the broader literature. First, the attention weights are a function
_only_ of the task at hand: if the task does not change, the weights do not
change, even if the input to the network does. This makes the mechanism
different from many in machine learning, including those proposed by Ba et al
(2015), Chen et al (2017), de Vries et al (2017), Denil et al (2012),
Dosovitskiy et al (2021), Larochelle & Hinton (2010), Mnih et al (2014), Perez
et al (2018), Ranzato (2014), Stollenga et al (2014) and Vaswani et al (2017),
as summarised in a recent review (Lindsay, 2020a). Second, attention here is
feature-based as opposed to spatial: when a representational feature is
modulated, it is modulated across the whole visual field. Attention is also
covert, meaning there is no notion of glancing between points in space. Third,
the model is feedforward, giving a setup comparable to short-duration object-
recognition episodes in studies of human perception (DiCarlo et al, 2012).

#### Using ablation studies to understand machine-learning methods

Having defined a model of visual processing with attention, the challenge is
to understand it. A simple yet powerful approach is to perform an ablation
study. The general formula for an ablation study is to evaluate the
performance of method A relative to method B on a task. In a typical setup,
method A incorporates a design change, method B is a relevant baseline (often
a version of method A where the design change is disabled) and evaluation
involves measuring average performance across a test dataset. The relative
performance of the methods reveals information about the effects of the design
change. This information is prized in machine learning: ablation studies are
the predominant way of assessing new techniques.

While valuable, the typical setup needlessly throws away useful information:
comparing _average_ performance amounts to treating the test dataset as a
black box. Task-oriented ablation design instead recognises a dataset as a
diverse mix of examples, subsets of which can be used to define tasks. After
identifying factors of variation between tasks, it becomes straightforward to
derive an experiment that reveals how the nature of a task relates to the
effects of a methodological change (Figure 1).

TOAD is at the core of our investigation. It guides us in capitalising on the
diversity of ImageNet: instead of using the dataset to define a single task,
we use it to generate a wide array of smaller tasks, across which we can
compare the perceptual boost of attention. It also streamlines execution:
having identified candidate tasks and quantified differences between them
(steps 1-2 in Figure 1), the rest of the process (steps 3-5) can be automated.

Since TOAD is simple and general by design, its possible applications extend
far beyond our experiment. In natural-language processing, it could help
determine how text style (eg, sentence length) affects the accuracy of a
sentiment-analysis technique. In meteorology, it could be used to understand
how the errors made by a rainfall-forecast model depend on the climate (eg,
regional average temperature). In computational medicine, it could reveal how
the predictions of an image-segmentation method vary in quality between forms
of cancer (eg, growth rate). Whereas a standard ablation study would give a
crude indication of whether a new method works better or worse, applying TOAD
leads to specific insights about what it is about a task that a method does or
does not deal with.

#### Measuring variation across ImageNet-based tasks

ImageNet is a strikingly diverse dataset: its 1000 classes constitute a rich
cross-section of the naturalistic visual world, and _within each class_ there
is a large degree of heterogeneity (Roads & Love, 2021; Russakovsky et al,
2015). This makes it an excellent source of data for an experiment based on
the TOAD framework. In our experiment, each task consists of classifying
images from a task set comprised of two ImageNet classes. To measure how tasks
differ, we identify six task-set properties (factors of variation) that might
be linked to the influence of attention in our model: clutter, difficulty,
scale, CNN similarity, human similarity and semantic similarity. Notably,
these factors are likely to be relevant to other ImageNet-based studies.

##  3 Method

How does the perceptual boost of top-down attention vary with the nature of a
visual task? To answer this, we use an instance of task-oriented ablation
design. The experiment consists of five parts.

  1. 1.

Task template: classifying images from a task set comprised of two ImageNet
classes.

  2. 2.

Task descriptors: six task-set properties based on class-level statistics.

  3. 3.

Exploration: quasi-random selection of tasks using a Sobol sequence.

  4. 4.

Evaluation: comparing the accuracy of two CNNs, one with top-down attention
and one without.

  5. 5.

Analysis: linear regression with the task-set properties as covariates and
accuracy as response.

Code to reproduce our experiment, along with associated data, is available at
github.com/fbickfordsmith/attention-toad (MIT license). See Appendix B for
details on the resources used in the experiment.

###  3.1 Tasks: image classification on pairs of ImageNet classes

Each task we consider consists of classifying images from a chosen _task set_
, a pair of ImageNet classes. There can be clear qualitative differences
between task sets: compare, say, {magpie, robin} to {magpie, ambulance}. For
precision and scalability, we aim to capture such differences with six
numerical quantities called _task-set properties_. Then, to ensure we consider
a broad range of tasks, we select a collection of task sets that vary widely
with respect to these properties. We cover the key ideas here and provide a
more detailed description in Appendix A.

#### Task-set properties

Clutter represents how much distracting information is present in a typical
image from a task set. To measure this, we use an object detector to count the
number of distracting cues in each image, building on work by Kuznetsova et al
(2020), Ren et al (2015), Russakovsky et al (2015) and Szegedy et al (2017).
Difficulty is the error rate of our baseline model (Section 3.2) on images
from a task set. Scale is the relative size of the subject in a typical image
from a task set. We estimate this using bounding-box annotations associated
with ImageNet. CNN similarity reflects how frequently the classes in a task
set are confused for each other by our baseline model. Human similarity
relates to how closely people associate the classes in a task set, inferred
from comparisons they make between images. For this measure, we make use of
psychological embeddings developed by Roads & Love (2021). Semantic similarity
represents the conceptual proximity of the classes in a task set according to
WordNet (Miller, 1995), the semantic hierarchy associated with ImageNet.
Computing the pairwise similarity of classes is made straightforward by
semantic embeddings created by Barz & Denzler (2019).

#### Selecting task sets

Having defined six task-set properties, we need to select a collection of task
sets that covers an interesting range of combinations of these properties. To
do this, we enumerate all possible task sets and compute their properties.
This gives an interval of possible values for each property. The Cartesian
product of these intervals is a volume in six-dimensional space.
Systematically exploring combinations of the task-set properties requires
filling this volume evenly with a collection of points, where each point
represents the properties of a task set. An effective way of doing this is to
generate points with a Sobol sequence, a low-discrepancy sequence sometimes
used for quasi-Monte Carlo integration (Morokoff & Caflisch, 1995). In
general, these points correspond to hypothetical, not actual, task sets. So we
treat them as targets and find actual task sets whose properties are closest
to the targets (Figure 5, Appendix A).

###  3.2 Model: attention-augmented convolutional neural network

VGG16, the foundation of our model, is a function, $f$, that takes an image,
$x\in\mathbb{R}^{224\times 224\times 3}$, as input and predicts the ImageNet
class, $c\in\\{1,2,3,\ldots,1000\\}$, the image belongs to:

$f(x)=p(c\mid x)$

This function can be decomposed into two parts. The convolutional layers,
$f_{c}$, transform $x$ to a latent representation, $z\in\mathbb{R}^{7\times
7\times 512}$. The densely connected layers, $f_{d}$, map from $z$ to $p(c\mid
x)$. That is,

$f_{c}(x)=z\qquad\qquad f_{d}(z)=p(c\mid x)$

In our model, top-down attention is a multiplicative modulation of
$f_{c}(x)=z$ by a collection of nonnegative attention weights,
$a\in\mathbb{R}_{\geq 0}^{512}$. The result is an augmented version of VGG16
given by

$\hat{f}(x,a)=p(c\mid x,a)=f_{d}(a\odot z)\qquad\qquad(a\odot
z)_{i,j,k}=a_{k}z_{i,j,k}$

#### Training

We treat the ImageNet-pretrained $f_{c}$ and $f_{d}$ as fixed functions: the
attention weights are the only trainable parameters. Training consists of
minimising the cross entropy between the model’s predictions and the dataset
labels. Each update to the attention weights is computed by the Adam optimiser
(Kingma & Ba, 2015) using gradients computed on a minibatch of 128 examples
(Adam settings: $\alpha=0.0003$, $\beta_{1}=0.9$, $\beta_{2}=0.999$,
$\hat{\varepsilon}=10^{-7}$). Images are preprocessed in the same way as for
VGG16. Training stops once the cross entropy starts to increase on the
validation dataset.

#### Baseline

To set up the baseline model for our experiment, we set the value of each
attention weight to 1 and then train the weights on examples from all 1000
ImageNet classes. The resulting weights are task-generic: they are optimised
for ImageNet as a whole, not for a particular task set. Using an initial value
of 1 for the weights means that at the beginning of training the network has
the same input-output mapping as a standard VGG16. The heterogeneity of
VGG16’s pretrained parameters means that homogeneity in the attention weights
does not cause training issues.

#### Top-down attention

To produce attention weights specialised for a task set, we initialise the
weights to those of the baseline model and then train them on a mixture of
examples from both inside and outside the task set. In order to balance the
performance costs and benefits of attention (Luo et al, 2021), we include
equal numbers of task-set and non-task-set examples, with a fresh random
sample of non-task-set examples for each epoch. For instance, suppose the two
classes that comprise a task set each have 1000 examples in the training
dataset. When training attention weights on this task set, updates are
computed using all 2000 examples from the task set as well as 2000 examples
from the 998 ImageNet classes not in the task set. The non-task-set examples
are drawn evenly from the classes: in this case, there are at least two
examples from each of the 998 classes.

#### Evaluation

Inferring the perceptual boost of attention on a task set is straightforward.
We retrieve the ImageNet validation images that belong to the classes in the
task set. Then, using those images, we assess the accuracy of the baseline
model as well as the accuracy of a model whose attention weights are optimised
for the task set. The difference between those two accuracies indicates the
impact of applying task-specific attention weights.

##  4 Results

We present a large-scale source of empirical evidence on the task-dependence
of attention’s influence in naturalistic vision. This is the product of
training attention weights on 2000 task sets, requiring thousands of hours of
GPU runtime.

[ head to column names, tabular=ttttl, table head=$i$ $\beta_{i}$
$\beta_{i}^{\prime}$ $\textrm{SE}(\beta_{i})$ Description
, table foot=, ]figures/accuracy_change.csv

Coefficients of the linear-regression model shown in the plots, for which
$R^{2}=0.88$. $\beta_{i}^{\prime}$ is the standardised version of $\beta_{i}$.
$\textrm{SE}(\beta_{i})$ is the standard error of $\beta_{i}$. For
$i\in\\{1,2,4,5\\}$, $p(\beta_{i})<0.005$.  (a) The perceptual boost produced
by top-down attention on 2000 visual tasks. Each task consists of classifying
images from a task set (pair of ImageNet classes). The perceptual boost of
attention is the accuracy change when switching from task-generic attention
weights (baseline) to task-specific attention weights in a convolutional
neural network. The strength of the perceptual boost varies with the
properties of the task set: clutter, difficulty, CNN similarity and human
similarity all have statistically significant effects. Each grey point
corresponds to a single task set. The blue lines illustrate a linear-
regression model fit by least-squares estimation.

#### Relating task-set properties to attention’s perceptual boost

For all task sets, applying top-down attention leads to an improvement in the
model’s image-classification accuracy. This perceptual boost is stronger when
clutter increases, difficulty increases, CNN similarity decreases, or human
similarity increases (Figure 4(a)). Scale and semantic similarity do not have
statistically significant effects.

#### Explaining the significance of difficulty

Difficulty is the task-set property most closely associated with the
perceptual boost of attention. Aiming to understand this, we compare the task-
generic attention weights of the baseline model to the task-specific attention
weights trained on each of the task sets in the experiment (Figure 4). This
yields a key insight into what underlies the difficulty of a task set and, in
turn, why attention’s perceptual boost is greater on more difficult task sets
(Section 5).

#### Exploring within-class variation in clutter and scale

Clutter and scale vary substantially between images belonging to the same
class (Figure 6, Appendix A). This observation supports an explanation for why
clutter and scale have relatively little effect on the perceptual boost of
attention (Section 5).

##  5 Discussion

It is striking that stimulus-level variables traditionally associated with
attention, such as visual clutter and object scale, have less explanatory
power than system-level variables that capture the interaction between the
model, the distribution of training data and the task format. This implies
tension between our results and expectations based on prior studies involving
simple stimuli. One interpretation of this finding is that understanding
attention’s influence in our model requires resisting simplistic, stimulus-
level descriptions of task differences. Consider difficulty, the task-set
property most strongly linked to the influence of attention. It is common to
frame difficulty as an inherent property of particular images. This is
certainly true to some extent: some images portray an object unambiguously
while others do not (compare ‘daisy’ to ‘sunscreen’ in Figure 2). But there is
more going on than just this. In our experiment, even a simple attention
mechanism yields substantially improved performance on the most difficult task
sets (Figure 4(a)).

Analysing the attention weights of our model reveals why attention is so
useful for difficult tasks. Training these weights (and the parameters of
neural networks more generally) involves trading off features in terms of
their overall contributions to performance (Hermann & Lampinen, 2020; Sutton
et al, 2006). A feature might be discriminative for some examples but
confounding for others; the weight placed on that feature is determined by the
feature’s overall effect on performance across the training data. The baseline
version of our model has task-generic attention weights that trade off
features such that the model’s performance is (locally) optimal when evaluated
across all ImageNet classes. Training the model on a particular task set,
comprising only two ImageNet classes, corresponds to a relaxation of this
feature tradeoff. The attention weights can now prioritise features that are
helpful for this task but were previously suppressed. As a result, the
distribution of attention weights changes; crucially, it changes more
drastically as difficulty increases (Figure 4). This reveals a deep insight
into what makes a task set difficult. The most difficult task sets are those
for which the optimal set of attention weights are most dissimilar from the
baseline set of attention weights; they are also the task sets for which
attention produces the greatest change in the model’s accuracy. In other
words, attention has greater impact for more difficult task sets because their
key discriminative features are more strongly suppressed when training on all
ImageNet classes. From this perspective, difficulty is clearly a system-level
phenomenon rather than just an inherent property of a particular image. It
encompasses the host of factors that determine the representations the model
learns to compute and how those representations are weighted.

A similar argument justifies the importance of CNN similarity relative to
human similarity and semantic similarity. Whereas the latter two measures are
derived from external data sources, CNN similarity incorporates the complex
interplay between the model and external factors. Taking an internal view
provides an intuitive case for the link between CNN similarity and the
perceptual boost of attention. For attention to help discern between the two
classes in a task set, it must weight the features of the model’s
representations such that the dissimilarity of the two classes is emphasised,
leading to separability at the model’s output. If the two classes in a task
set have low CNN similarity, the classes’ internal representations are
unalike, which makes it straightforward to find attention weights that
emphasise the dissimilarity. If the two classes have high similarity, the
opposite is true.

Why is clutter, the presence of distracting visual cues, not strongly linked
to the perceptual boost of attention in our model? Clutter is often cited in
descriptions of attention (Chikkerur et al, 2010; Navalpakkam & Itti, 2007;
Peelen & Kastner, 2014). The association is a natural one to make: an
empirically validated theory is that stimuli compete for limited computational
resources (Carrasco, 2011); viewing attention as a mechanism for allocating
those resources (Lindsay, 2020a), if follows that clutter and attention should
be linked. Yet our results do not support this. The variation of clutter
across the tasks we study barely affects the influence of attention. One
possible explanation is that ImageNet-trained CNNs process images in a less
object-centric way than humans do. CNNs recognise textures more than they do
shapes (Baker et al, 2018; Geirhos et al, 2019; Hermann et al, 2020; Kucker et
al, 2019). This means that the model we use is relatively insensitive to
clutter by default, which in turn means that attention does not have much
impact. This contrasts with attention in humans (Nuthmann & Henderson, 2010).
If future work aims to replicate in models the influence of attention as
measured in lab studies, it might be necessary to address the texture bias of
models.

Scale, the relative size of the subject in a typical image, also has little
effect on the perceptual boost of attention. It is tempting to argue that
scale is a spatial phenomenon and is thus unlikely to interact with the
feature-based attention mechanism in our model. But this argument is probably
too simplistic. Scale is closely linked to what visual features are present in
an image: even for a fixed image subject, changing the scale can result in
considerable differences in terms of visual features. A more convincing
explanation is that scale is an instance-level phenomenon that needs to be
dealt with using bottom-up signals. There is so much within-class variation in
scale (Figure 6, Appendix A) that it is hard to determine a set of attention
weights that is useful on average. In this light, it seems sensible that the
top-down form of attention in our model, in which attention weights are
purposefully not conditioned on image information, has little interaction with
scale. The same logic provides further grounds for explaining the clutter
result discussed above: the task-level attention weights do not deal with
clutter because it varies so much from one image to another.

Figure 4:  Comparing task-generic and task-specific attention weights. Left:
the distribution of task-specific weights is broader than the distribution of
task-generic weights, with standard deviations of 0.39 and 0.23 respectively.
Right: the shift from task-generic to task-specific weights is larger for more
difficult task sets, with a correspondingly larger perceptual boost of
attention. For each task set, we plot (a) the Euclidean distance between the
attention weights trained on the task set and the task-generic attention
weights of the baseline model against (b) the difficulty of the associated
task set. The colour of each point denotes the accuracy change produced by
switching from the task-generic weights to the task-specific weights.

##  6 Conclusion

What does top-down attention do to enhance performance in naturalistic visual
tasks? To understand _how_ attention helps, it is instructive to work out
_when_ it helps. Addressing this at a level of scale and complexity beyond
that in past studies, we investigate the perceptual boost of attention in a
CNN on thousands of tasks. Task-oriented ablation design, a broadly applicable
experimental framework, makes this possible. Our key finding is that
explaining the influence of attention in our model demands a system-level
perspective on what makes a task challenging and how attention helps to deal
with this. This raises the question of whether, in the context of real-world
visual tasks, it is actually helpful to observe the effect of a small number
of simple stimulus-level variables. Instead, perhaps we must take an inside
view, informed by variables that require access to the model and the data it
is trained on. There is a distinct contrast between this view and prior
approaches to studying attention. Are we satisfied by simple descriptions or
do we need to embrace system-level explanations?

## Acknowledgments and Disclosure of Funding

We thank Ed Grefenstette, Chris Summerfield and Will Tebbutt for valuable
advice. Freddie Bickford Smith is supported by the Engineering & Physical
Sciences Research Council and Kellogg College, Oxford. This work was supported
by NIH Grant 1P01HD080679, Wellcome Trust Investigator Award WT106931MA and
Royal Society Wolfson Fellowship 183029 to Bradley C Love.

## References

  * (1)
  * Abadi et al (2016) Abadi, Barham, Chen, Chen, Davis, Dean, Devin, Ghemawat, Irving, Isard, Kudlur, Levenberg, Monga, Moore, Murray, Steiner, Tucker, Vasudevan, Warden, Wicke, Yu, Zheng (2016). TensorFlow: a system for large-scale machine learning. USENIX Symposium on Operating Systems Design & Implementation.
  * Alexe et al (2012) Alexe, Deselaers, Ferrari (2012). Measuring the objectness of image windows. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * Ba et al (2015) Ba, Mnih, Kavukcuoglu (2015). Multiple object recognition with visual attention. International Conference on Learning Representations.
  * Baker et al (2018) Baker, Lu, Erlikhman, Kellman (2018). Deep convolutional networks do not classify based on global object shape. PLoS Computational Biology
  * Bar (2003) Bar (2003). A cortical mechanism for triggering top-down facilitation in visual object recognition. Journal of Cognitive Neuroscience.
  * Barbosa et al (2021) Barbosa, Stein, Summerfield, Soto-Faraco, Hyafil (2021). A practical guide for studying human behavior in the lab. PsyArXiv.
  * Barz & Denzler (2019) Barz, Denzler (2019). Hierarchy-based image embeddings for semantic image retrieval. IEEE Winter Conference on Applications of Computer Vision.
  * Bichot et al (2015) Bichot, Heard, DeGennaro, Desimone (2015). A source for feature-based attention in the prefrontal cortex. Neuron.
  * Bickford Smith et al (2020) Bickford Smith, Luo, Roads, Love (2020). The perceptual boost of visual attention is task-dependent in naturalistic settings. Bridging AI & Cognitive Science, International Conference on Learning Representations.
  * Borji & Itti (2013) Borji, Itti (2013). State-of-the-art in visual attention modelling. IEEE Transactions on Pattern Analysis and Machine Intelligence.
  * Boudreau et al (2006) Boudreau, Williford, Maunsell (2006). Effects of task difficulty and target likelihood in area V4 of macaque monkeys. Journal of Neurophysiology.
  * Carrasco (2011) Carrasco (2011). Visual attention: the past 25 years. Vision Research.
  * Chen et al (2008) Chen, Martinez-Conde, Macknik, Bereshpolova, Swadlow, Alonso (2008). Task difficulty modulates the activity of specific neuronal populations in primary visual cortex. Nature Neuroscience.
  * Chen et al (2017) Chen, Zhang, Xiao, Nie, Shao, Liu, Chua (2017). SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning. Conference on Computer Vision and Pattern Recognition.
  * Chikkerur et al (2010) Chikkerur, Serre, Tan, Poggio (2010). What and where: a Bayesian inference theory of attention. Vision Research.
  * da Costa-Luis (2019) da Costa-Luis (2019). tqdm: a fast, extensible progress meter for Python and CLI. Journal of Open Source Software.
  * de Vries et al (2017) de Vries, Strub, Mary, Larochelle, Pietquin, Courville (2017). Modulating early visual processing by language. Neural Information Processing Systems.
  * Denil et al (2012) Denil, Bazzani, Larochelle, de Freitas (2012). Learning where to attend with deep architectures for image tracking. Neural Computation.
  * DiCarlo et al (2012) DiCarlo, Zoccolan, Rust (2012). How does the brain solve visual object recognition? Neuron.
  * Dosovitskiy et al (2021) Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, Houlsby. An image is worth 16$\times$16 words: transformers for image recognition at scale. International Conference on Learning Representations.
  * Downing (1988) Downing (1988). Expectancy and visual-spatial attention: effects on perceptual quality. Journal of Experimental Psychology.
  * Geirhos et al (2019) Geirhos, Rubisch, Michaelis, Bethge, Wichmann, Brendel (2019). ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. International Conference on Learning Representations.
  * Gillies et al (2007) Gillies, Shapely contributors (2007). Shapely: manipulation and analysis of geometric objects.
  * Harris et al (2020) Harris, Millman, van der Walt, Gommers, Virtanen, Cournapeau, Wieser, Taylor, Berg, Smith, Kern, Picus, Hoyer, van Kerkwijk, Brett, Haldane, Fernandez del Rio, Wiebe, Peterson, Gerard-Marchant, Sheppard, Reddy, Weckesser, Abbasi, Gohlke, Oliphant (2020). Array programming with NumPy. Nature.
  * Hassabis et al (2017) Hassabis, Kumaran, Summerfield, Botvinick (2017). Neuroscience-inspired artificial intelligence. Neuron.
  * Helmholtz (1896) Helmholtz (1896). Handbuch der Physiologischen Optik.
  * Hermann & Lampinen (2020) Hermann, Lampinen (2020). What shapes feature representations? Exploring datasets, architectures, and training. Neural Information Processing Systems.
  * Hermann et al (2020) Hermann, Chen, Kornblith (2020). The origins and prevalence of texture bias in convolutional neural networks. Neural Information Processing Systems.
  * Hunter (2007) Hunter (2007). Matplotlib: a 2D graphics environment. Computing in Science & Engineering.
  * Itti & Koch (2001) Itti, Koch (2001). Computational modelling of visual attention. Nature Reviews Neuroscience.
  * James (1890) James (1890). The Principles of Psychology.
  * Kingma & Ba (2015) Kingma, Ba (2015). Adam: a method for stochastic optimization. International Conference on Learning Representations.
  * Klein et al (1970) Klein, Plomp, Pols (1970). Vowel spectra, vowel spaces, and vowel identification. Journal of the Acoustical Society of America.
  * Kluyver et al (2016) Kluyver, Ragan-Kelley, Perez, Granger, Bussonnier, Frederic, Kelley, Hamrick, Grout, Corlay, Ivanov, Avila, Abdalla, Willing, Jupyter development team (2016). Jupyter Notebooks—a publishing format for reproducible computational workflows. Positioning and Power in Academic Publishing: Players, Agents and Agendas.
  * Kriegeskorte (2015) Kriegeskorte (2015). Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual Review of Vision Science.
  * Kruschke (1992) Kruschke (1992) ALCOVE: an exemplar-based connectionist model of category learning. Psychological Review.
  * Kucker et al (2019) Kucker, Samuelson, Perry, Yoshida, Colunga, Lorenz, Smith (2019). Reproducibility and a unifying explanation: lessons from the shape bias. Infant Behavior and Development.
  * Kuznetsova et al (2020) Kuznetsova, Rom, Alldrin, Uijlings, Krasin, Pont-Tuset, Kamali, Popov, Malloci, Kolesnikov, Duerig, Ferrari (2020). The Open Images Dataset V4. International Journal of Computer Vision.
  * Larochelle & Hinton (2010) Larochelle, Hinton (2010). Learning to combine foveal glimpses with a third-order Boltzmann machine. Neural Information Processing Systems.
  * Lee & Maunsell (2010) Lee, Maunsell (2010). Attentional modulation of MT neurons with single or multiple stimuli in their receptive fields. Journal of Neuroscience.
  * Lindsay (2020a) Lindsay (2020a). Attention in psychology, neuroscience, and machine learning. Frontiers in Computational Neuroscience.
  * Lindsay (2020b) Lindsay (2020b). Convolutional neural networks as a model of the visual system: past, present, and future. Journal of Cognitive Neuroscience.
  * Lindsay & Miller (2018) Lindsay, Miller (2018). How biological attention mechanisms improve task performance in a large-scale visual system model. eLife.
  * Lindsay et al (2020) Lindsay, Rubin, Miller (2020). A unified circuit model of attention: neural and behavioral effects. bioRxiv.
  * Luo et al (2021) Luo, Roads, Love (2021). The costs and benefits of goal-directed attention in deep convolutional neural networks. Computational Brain & Behaviour.
  * Mackay (2003) MacKay (2003). Information Theory, Inference, and Learning Algorithms.
  * McKinney (2010) McKinney (2010). Data structures for statistical computing in Python. Python in Science Conference.
  * Miller (1995) Miller (1995). WordNet: a lexical database for English. Communications of the ACM.
  * Mnih et al (2014) Mnih, Hees, Graves, Kavukcuoglu (2014). Recurrent models of visual attention. Neural Information Processing Systems.
  * Morokoff & Caflisch (1995) Morokoff, Caflisch (1995). Quasi-Monte Carlo integration. Journal of Computational Physics.
  * Navalpakkam & Itti (2007) Navalpakkam, Itti (2007). Search goal tunes visual features optimally. Neuron.
  * Nuthmann & Henderson (2010) Nuthmann, Henderson (2010). Object-based attentional selection in scene viewing. Journal of Vision.
  * Paneri & Gregoriou (2017) Paneri, Gregoriou (2017). Top-down control of visual attention by the prefrontal cortex: functional specialization and long-range interactions. Frontiers in Neuroscience.
  * Pedregosa et al (2011) Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel, Prettenhofer, Weiss, Dubourg, VanderPlas, Passos, Cournapeau, Brucher, Perrot, Duchesnay (2011). Scikit-learn: machine learning in Python. Journal of Machine Learning Research.
  * Peelen & Kastner (2014) Peelen, Kastner (2014). Attention in the real world: toward understanding its neural basis. Trends in Cognitive Sciences.
  * Perez et al (2018) Perez, Strub, de Vries, Dumoulin, Courville (2018). FiLM: visual reasoning with a general conditioning layer. AAAI Conference on Artificial Intelligence.
  * Ranzato (2014) Ranzato (2014). On learning where to look. arXiv.
  * Ren et al (2015) Ren, He, Girshick, Sun (2015). Faster R-CNN: towards real-time object detection with region proposal networks. Neural Information Processing Systems.
  * Reynolds & Heeger (2009) Reynolds, Heeger (2009). The normalization model of attention. Neuron.
  * Roads & Love (2021) Roads, Love (2021). Enriching ImageNet with human similarity judgments and psychological embeddings. Conference on Computer Vision and Pattern Recognition.
  * Rolls (2008) Rolls (2008). Top-down control of visual perception: attention in natural vision. Perception.
  * Rossi et al (2007) Rossi, Bichot, Desimone, Ungerleider (2007). Top-down attentional deficits in macaques with lesions of lateral prefrontal cortex. Journal of Neuroscience.
  * Russakovsky et al (2015) Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, Berg, Fei-Fei (2015). ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision.
  * Schrimpf et al (2020) Schrimpf, Kubilius, Lee, Murty, Ajemian, DiCarlo (2020). Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron.
  * Seabold & Perktold (2010) Seabold, Perktold (2010). Statsmodels: econometric and statistical modeling with Python. Python in Science Conference.
  * Serre (2019) Serre (2019). Deep learning: the good, the bad, and the ugly. Annual Review of Vision Science.
  * Simonyan & Zisserman (2015) Simonyan, Zisserman (2015). Very deep convolutional networks for large-scale image recognition. International Conference on Learning Representations.
  * Spitzer et al (1988) Spitzer, Desimone, Moran (1988). Increased attention enhances both behavioral and neuronal performance. Science.
  * Spitzer & Richmond (1991) Spitzer, Richmond (1991). Task difficulty: ignoring, attending to, and discriminating a visual stimulus yield progressively more activity in inferior temporal neurons. Experimental Brain Research.
  * Stollenga et al (2014) Stollenga, Masci, Gomez, Schmidhuber (2014). Deep networks with internal selective attention through feedback connections. Neural Information Processing Systems.
  * Sutton et al (2006) Sutton, Sindelar, McCallum (2006). Reducing weight undertraining in structured discriminative learning. Human Language Technology Conference of the NAACL.
  * Szegedy et al (2017) Szegedy, Ioffe, Vanhoucke, Alemi (2017). Inception-v4, Inception-ResNet and the impact of residual connections on learning. AAAI Conference on Artificial Intelligence.
  * van Rossum & Drake (1995) van Rossum, Drake (1995). Python Reference Manual.
  * Vaswani et al (2017) Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin (2017). Attention is all you need. Neural Information Processing Systems.
  * Virtanen et al (2020) Virtanen, Gommers, Oliphant, Haberland, Reddy, Cournapeau, Burovski, Peterson, Weckesser, Bright, van der Walt, Brett, Wilson, Millman, Mayorov, Nelson, Jones, Kern, Larson, Carey, Polat, Feng, Moore, VanderPlas, Laxalde, Perktold, Cimrman, Henriksen, Quintero, Harris, Archibald, Ribeiro, Pedregosa, van Mulbregt, SciPy 1.0 contributors (2020). SciPy 1.0: fundamental algorithms for scientific computing in Python. Nature Methods.
  * Waskom (2021) Waskom (2021). seaborn: statistical data visualization. Journal of Open Source Software.
  * Yamins & DiCarlo (2016) Yamins, DiCarlo (2016). Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience.
  * Yang et al (2021) Yang, Yau, Fei-Fei, Deng, Russakovsky (2021). A study of face obfuscation in ImageNet. arXiv.

##  Appendix A Task-set properties

Let a task set, $\\{c_{1},c_{2}\\}\subset\\{1,2,3,\ldots,1000\\}$, be a subset
of two of the 1000 ImageNet classes, where each integer class label
corresponds to an object (basketball, lemon, zebra and so on). Due to the
diversity of ImageNet classes, any two task sets can differ substantially,
most obviously with respect to their visual appearance. We aim to describe
this variation numerically with six task-set properties: clutter, difficulty,
scale, CNN similarity, human similarity and semantic similarity.

#### Clutter

The amount of irrelevant visual information, or clutter, varies between
images. To measure this, we count the number of distracting cues in each
image. Automating this is made possible by an object-detection model that
predicts the locations and identities of objects present in an image,
assigning a confidence score to each object it detects. This model is a Faster
R-CNN (Ren et al, 2015) that uses an ImageNet-pretrained Inception ResNet v2
(Szegedy et al, 2017) as its feature extractor and is trained on the Open
Images v4 dataset (Kuznetsova et al, 2020).

Our approach is inspired by Russakovsky et al (2015), who also measured
clutter by counting the objects in an image. But, crucially, whereas they
decided to treat all objects equally by using a class-agnostic object detector
(Alexe et al, 2012), we make use of the knowledge that, depending on the
subject of an image, some objects are more distracting than others. This helps
us to satisfy three conditions that we identify as important when measuring
clutter.

  1. 1.

Each unique object is counted only once. If this were not the case, an image
showing a bunch of bananas would receive a higher clutter score than one that
shows a single banana, even if both images contain the same number of
distracting cues.

  2. 2.

Each object’s contribution to the clutter score is weighted by the model’s
confidence that the object is present in the image. Given ambiguous images and
an imperfect model, an object should only count insofar as it actually appears
to be in the image.

  3. 3.

The more consistently an object occurs in images of a given class, the more
relevant it is to that class and the less it contributes to the clutter score
for an image from that class. An ice cream should count as clutter in raincoat
images more than it does in beach images.

Based on these conditions, we define the clutter score of task set
$\\{c_{1},c_{2}\\}$ as the mean of $\textrm{clutter}(c_{1})$ and
$\textrm{clutter}(c_{2})$, with the class-level clutter score given by

$\textrm{clutter}(c)=\underset{{x\in\mathcal{X}_{c}^{t}}}{\textrm{median}}\left(\sum_{o\in\mathcal{O}_{x}}\textrm{irrelevance}(c,o)\cdot\textrm{confidence}(x,o)\right)$

where $\mathcal{X}_{c}^{t}$ is the subset of ImageNet training images with
label $c$ and $\mathcal{O}_{x}\subset\\{1,2,3,\dots,600\\}$ is the subset of
Open Images object labels predicted for image $x$. Conditions 1 and 2 are
satisfied by summing the object-recognition model’s maximal confidence for
each unique object, $o$, in each image, $x$. Condition 3 is satisfied by
including an irrelevance score for image label $c$ and object label $o$, based
on a cooccurrence matrix, $C$:

$\textrm{irrelevance}(c,o)=1-\hat{C}_{c,o}\qquad\qquad
C_{c,o}=\sum_{x\in\mathcal{X}_{c}^{t}}\sum_{o\in\mathcal{O}_{x}}\textrm{confidence}(x,o)$

where $\hat{C}$ is a row-normalised version of $C$. According to the
irrelevance score, the less frequently object $o$ occurs in images of class
$c$, the less relevant it is to class $c$.

#### Difficulty

Whereas images from some ImageNet classes can be classified with perfect
accuracy by modern CNNs, those from others are frequently misclassified
(Russakovsky et al, 2015). In other words, classes vary in difficulty. We
define the difficulty score of task set $\\{c_{1},c_{2}\\}$ as the mean of
$\textrm{difficulty}(c_{1})$ and $\textrm{difficulty}(c_{2})$, with the
difficulty score of a class being the error rate of our baseline model
(Section 3.2) on images from that class:

$\textrm{difficulty}(c)=\frac{1}{|\mathcal{X}_{c}^{v}|}\sum_{x\in\mathcal{X}_{c}^{v}}\mathbb{I}\left(\arg\max_{c^{\prime}}(p(c^{\prime}\mid
x,a_{0}))\neq c\right)$

where $\mathcal{X}_{c}^{v}$ is the subset of ImageNet validation images with
label $c$, $\mathbb{I}$ is an indicator function and $p(c\mid x,a_{0})$ is the
output of the baseline model.

Figure 5:  Exploring combinations of task-set properties. Enumerating all
499,500 possible task sets (pairs of ImageNet classes) and computing their
properties gives a set of candidate combinations of task-set properties
(grey). To explore a diverse collection of task sets, we use a quasi-random
sequence to identify spaced-out target locations, then for each target
location find the task set whose properties (blue) are closest to the target.
In the upper-right corner of each plot, we show the Spearman correlation
between the corresponding pair of task-set properties, computed using the grey
points in the plot.

#### Scale

Images vary with respect to scale, the size of the object they portray
relative to the size of the whole image. Scale can be estimated using the
ImageNet examples for which human annotators have drawn a bounding box around
the subject of an image (or multiple boxes if the subject appears more than
once). We define the scale score of task set $\\{c_{1},c_{2}\\}$ as the mean
of $\textrm{scale}(c_{1})$ and $\textrm{scale}(c_{2})$, where the class-level
scale score is the median proportion of an image covered by its subject:

$\textrm{scale}(c)=\underset{x\in\mathcal{X}_{c}^{b}}{\textrm{median}}\left(\frac{\textrm{area}(\mathcal{B}_{x})}{\textrm{area}(x)}\right)$

where $\mathcal{X}_{c}^{b}$ is the subset of ImageNet training images with
label $c$ and with a non-empty set of bounding-box annotations,
$\mathcal{B}_{x}$, and area is measured in pixels. In cases where there are
multiple bounding boxes in $\mathcal{B}_{x}$, we compute the area of their
union. This ensures scale is large for images where multiple small instances
of the subject together cover much of the image.

#### CNN similarity

Some pairs of ImageNet classes are more visually similar than others. One way
to capture this quantitatively is to look at the mistakes that a CNN makes
when classifying images. Using the ImageNet validation set, we compute the
confusion matrix, $D$, of our baseline model (Section 3.2), where $D_{i,j}$
denotes how often the predicted class is $j$ when the true class is $i$. We
convert this to a symmetric similarity matrix, $S$ (Klein et al, 1970):

$S_{i,j}=S_{j,i}=\frac{1}{2}\sum_{k}D_{i,k}+D_{j,k}-|D_{i,k}-D_{j,k}|$

The CNN similarity of task set $\\{c_{1},c_{2}\\}$ is simply
$S_{c_{1},c_{2}}$. The more often the classes in the task set are confused for
each other by the CNN, the greater their similarity.

#### Human similarity

When people make comparisons between objects, factors other than visual
appearance influence their perception of similarity. For instance, two objects
might be visually unalike yet still be perceived as similar because they are
contextually linked: consider cyclists and traffic lights. To capture a rich
notion of similarity that complements the CNN-based measure, we make use of
psychological embeddings developed by Roads & Love (2021). These embeddings
were inferred from the results of a large-scale experiment in which people
were shown a series of query images and for each query asked to identify the
most similar images among of a set of candidates. Having human similarity
judgements encoded in the form of embeddings allows us to compute a matrix,
$S^{h}$, that summarises how similarly people perceive each pair of ImageNet
classes. The human similarity of task set $\\{c_{1},c_{2}\\}$ is then
$S^{h}_{c_{1},c_{2}}$.

#### Semantic similarity

ImageNet classes correspond to distinct but related concepts. A pair of
classes can be semantically similar (speedboat and canoe), dissimilar
(speedboat and bagel) or somewhere in between (speedboat and minibus). This
information is naturally represented in WordNet (Miller, 1995), the semantic
hierarchy associated with ImageNet. The conceptual proximity of two classes
can be measured by looking at how far apart the classes are in the hierarchy.
To automate this, we make use of semantic embeddings created by Barz & Denzler
(2019). These embeddings, one for each ImageNet class, were designed such that
computing the dot product between a pair of class embeddings yields a
similarity score for those classes, corresponding to how close the two classes
are in the WordNet hierarchy. In other words, if $e_{1}$ and $e_{2}$ are the
embeddings for classes $c_{1}$ and $c_{2}$, the semantic similarity of task
set $\\{c_{1},c_{2}\\}$ is given by $e_{1}^{\top}e_{2}$.

Figure 6:  Distributions of image-level clutter and scale scores within
randomly sampled ImageNet classes. Clutter represents how much distracting
information is present in an image. Scale is the size of an image’s subject
relative to the size of the overall image. Across each class, there is
substantial variance in clutter and scale. Left: each horizontal axis spans
the $[0,10]$ interval. Right: each horizontal axis spans the $[0,1]$ interval.

##  Appendix B Resources used in this work

Our experiment is based on the ImageNet dataset. The presence of people in
ImageNet images raises issues around demographic representation, inappropriate
content and privacy. We strongly support work, such as recent research by Yang
et al (2021), that addresses these issues and other ethical concerns
associated with ImageNet.

The implementation of our experiment builds on a number of existing software
projects (Table B). To run the experiment, we use an internal server equipped
with Nvidia GeForce RTX 2080 Ti processors.

[ head to column names, tabular=llll, table head=Project Citation License URL
, table foot=, ]figures/software_citations.csv

Table 1:  Software projects used in this work.


Figure 16:  Left: Source detection efficiency estimated from ACIS observations
with exposure times $\sim\\!10$ (red), $61$ (green), and $122\,ks$ (blue) as a
function of broad band energy flux, for detections with off-axis angles
$\theta\leq 6^{\prime}$. Typical error bars are indicated for each curve.
Right: Same, except for detections with off-axis angles $\theta>6^{\prime}$.

###  4.3 Astrometric Accuracy

The absolute astrometric accuracies of CSC 2.0 and CSC 2.1 were evaluated by
cross-matching CSC master source positions with their optical stellar sources
from SDSS DR15 (Blanton et al., 2017) for CSC2.0 and Gaia DR3 (Gaia
Collaboration et al., 2023) for CSC 2.1, using a technique similar to Rots &
Budavári (2011). The measured angular separations between CSC master source
positions and matched optical stellar sources are presented in Figure 17. For
separations larger than $\sim\\!1\farcs 0$–$2\farcs 0$ the histogram fractions
become increasingly overestimated due to an increased percentage of poor
matches, typically due to invalid matches between X-ray and optical sources or
due to matches with off-axis CSC detections that have large and asymmetric
PSFs. Because of the difficulty in unambiguously excluding poor matches a
priori we choose not to remove them to avoid biasing the distribution. For
both catalog versions, the master source positions are determined by combining
stack detection positions as described in § 3.9.1. For CSC 2.0, the stack
detection positions are computed using the individual observation aspect
solutions after applying the relative astrometric corrections in § 3.4.
However, no absolute astrometric corrections are applied to the CSC 2.0 stack
detection positions. For CSC 2.1, the stack detection positions are corrected
to the Gaia-CRF3 astrometric reference frame as described in § 3.8. Correcting
the CSC astrometry to Gaia-CRF3 in CSC 2.1 reduces the median radial
astrometric error from $0\farcs 72$ to $0\farcs 30$.

Figure 17:  Normalized histograms of the astrometric angular separations
between reference catalogs (SDSS DR15 for CSC 2.0 and Gaia DR3 for CSC 2.1)
and cross-matched CSC master source positions for CSC 2.0 (red) and CSC 2.1
(blue), showing the overall improvement from matching release 2.1 astrometry
to the Gaia-CRF3 reference frame.

##  5 CONCLUSIONS

The Chandra Source Catalog is a virtual X-ray astrophysics facility that
provides to the astronomical community carefully-curated, high-quality, and
uniformly calibrated and analyzed properties for X-ray sources detected in
ACIS and HRC-I imaging observations obtained by the Chandra X-ray Observatory.
Besides including more recent observations obtained since CSC release 1 was
completed, the Release 2 Series improve upon the original catalog versions in
several ways. New background and source detection algorithms enable reliable
detection of compact sources down to roughly 5 net counts over much of the
field-of-view, yielding a significantly fainter catalog limiting sensitivity
that was possible for release 1. The new algorithms also allow the detection
of bright, extended X-ray emission regions that were deliberately excluded
from the earlier release. Starting with release 2.1, catalog positions are
corrected to the Gaia-CRF3 astrometric reference frame.

X-ray source and detection properties recorded in the catalog tables include
position and position error, spatial extent compared to the local PSF, multi-
band aperture photometry, hardness ratios, spectral fits using multiple
models, and intra- and inter-observation temporal variability measures,
typically with independent lower and upper confidence limits, extracted in 5
energy bands for ACIS observations and a single energy band for HRC data. The
catalog tables record the extracted properties for each source detected in the
stacked observations, as well as separately for each observation, so that any
temporal changes between observations are available for study. Best estimates
of properties derived from multiple detections of the same X-ray source on the
sky are provided at the master source level. The catalog includes 37 different
types of source, detection, and field-based FITS format science-ready X-ray
data products that can be used directly as a starting point for further
analysis.

Release 2.0 includes 317,167 unique X-ray sources detected in ACIS and HRC-I
imaging observations covering $559\,{\rm deg}^{2}$ of sky that were released
publicly prior to the end of 2014, while release 2.1 includes $\sim\\!407,806$
sources from observations released publicly prior to the end of 2021 with
$730\,{\rm deg}^{2}$ sky coverage.

The authors wish to thank the many former and current Chandra X-ray Center
staff who have contributed to the development and production operations
necessary to complete the Chandra Source Catalog Release 2 Series. In
particular, Stephen Doe, Dan Nguyen, and Roger Hain contributed extensively to
the Maximum Likelihood Estimator code. Christopher Allen implemented the
Bayesian X-ray aperture photometry algorithms. Extensive additional software
development support was provided by Jamie Budynkiewicz, Judy Chen, Danny Gibbs
II, Kenny Glotfelty, Diane Hall, Peter Harbo, Omar Laurino, Warren McLaughlin,
Charles Paxson, and Panagoula Zografou. Craig Anderson, Douglas Morgan, Amy
Mossman, Joy Nichols, and Beth Sundheim all supported catalog human quality
assurance reviews. Padmanabhan Ramadurai and the CXC Systems group provided
extensive installation and operational support for the catalog high-
performance cluster, and David W. Van Stone provided database administration
services for the catalog databases. The authors acknowledge the support and
guidance of the current and former Chandra X-ray Center directors, Patrick
Slane and Belinda Wilkes. The authors further wish to thank the anonymous
referee for a careful review of the manuscript, in addition to several
recommendations and suggestions that improved the quality and readability of
the paper. This paper makes use of data from the Sloan Digital Sky Survey
(SDSS). Funding for the Sloan Digital Sky Survey has been provided by the
Alfred P. Sloan Foundation, the Participating Institutions, the National
Aeronautics and Space Administration, the National Science Foundation, the
U.S. Department of Energy, the Japanese Monbukagakusho, and the Max Planck
Society. The SDSS Web site is http://www.sdss.org/. The SDSS is managed by the
Astrophysical Research Consortium for the Participating Institutions. The
Participating Institutions are The University of Chicago, Fermilab, the
Institute for Advanced Study, the Japan Participation Group, The Johns Hopkins
University, Los Alamos National Laboratory, the Max-Planck-Institute for
Astronomy, the Max-Planck-Institute for Astrophysics, New Mexico State
University, University of Pittsburgh, Princeton University, the United States
Naval Observatory, and the University of Washington. This publication makes
use of data products from the Wide-field Infrared Survey Explorer, which is a
joint project of the University of California, Los Angeles, and the Jet
Propulsion Laboratory/California Institute of Technology, and NEOWISE, which
is a project of the Jet Propulsion Laboratory/California Institute of
Technology. WISE and NEOWISE are funded by the National Aeronautics and Space
Administration. This work has made use of data from the European Space Agency
(ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia
Data Processing and Analysis Consortium (DPAC,
https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement. Support for development of
the Chandra Source Catalog is provided by the National Aeronautics and Space
Administration through the Chandra X-ray Center, which is operated by the
Smithsonian Astrophysical Observatory for and on behalf of the National
Aeronautics and Space Administration under contract NAS 8-03060.

## References

  * Akima (1970) Akima, H. 1970, JACM, 17, 589
  * Anders & Grevesse (1989) Anders, E., & Grevesse, N. 1989, Geochim. Cosmochim. Acta, 53, 197, doi: 10.1016/0016-7037(89)90286-X
  * Arnaud (1996) Arnaud, K. A. 1996, in Astronomical Society of the Pacific Conference Series, Vol. 101, Astronomical Data Analysis Software and Systems V, ed. G. H. Jacoby & J. Barnes, 17
  * Arnaud et al. (2004) Arnaud, K. A., George, I. M., & Tennant, A. F. 2004, The OGIP Spectral File Format, OGIP Memo OGIP/92-007, Tech. rep., NASA Goddard Space Fight Center, Greenbelt, MD, USA.  https://heasarc.gsfc.nasa.gov/docs/heasarc/ofwg/docs/spectra/ogip_92_007.pdf
  * Balucinska-Church & McCammon (1992) Balucinska-Church, M., & McCammon, D. 1992, ApJ, 400, 699, doi: 10.1086/172032
  * Bautz et al. (1998) Bautz, M. W., Pivovaroff, M., Baganoff, F., et al. 1998, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 3444, X-Ray Optics, Instruments, and Missions, ed. R. B. Hoover & A. B. Walker, 210–224, doi: 10.1117/12.331238
  * Blanton et al. (2017) Blanton, M. R., Bershady, M. A., Abolfathi, B., et al. 2017, AJ, 154, 28, doi: 10.3847/1538-3881/aa7567
  * Brassington et al. (2010) Brassington, N. J., Fabbiano, G., Blake, S., et al. 2010, ApJ, 725, 1805, doi: 10.1088/0004-637X/725/2/1805
  * Carrera et al. (2007) Carrera, F. J., Ebrero, J., Mateos, S., et al. 2007, A&A, 469, 27, doi: 10.1051/0004-6361:20066271
  * Cash (1979) Cash, W. 1979, ApJ, 228, 939, doi: 10.1086/156922
  * Cautun & van de Weygaert (2011) Cautun, M. C., & van de Weygaert, R. 2011, arXiv e-prints, arXiv:1105.0370, doi: 10.48550/arXiv.1105.0370
  * Cox & Reynolds (1987) Cox, D. P., & Reynolds, R. J. 1987, ARA&A, 25, 303, doi: 10.1146/annurev.aa.25.090187.001511
  * Cruddace et al. (1988) Cruddace, R. G., Hasinger, G. H., & Schmitt, J. H. 1988, in European Southern Observatory Conference and Workshop Proceedings, Vol. 28, European Southern Observatory Conference and Workshop Proceedings, 177–182
  * Cutri et al. (2021) Cutri, R. M., Wright, E. L., Conrow, T., et al. 2021, VizieR Online Data Catalog, II/328
  * Davis (2001) Davis, J. E. 2001, ApJ, 548, 1010, doi: 10.1086/319002
  * Davis (2007) —. 2007, Pile-up Fractions and Count Rates, Tech. rep., Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Davis_pileup.pdf
  * Davis et al. (2012) Davis, J. E., Bautz, M. W., Dewey, D., et al. 2012, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 8443, Space Telescopes and Instrumentation 2012: Ultraviolet to Gamma Ray, ed. T. Takahashi, S. S. Murray, & J.-W. A. den Herder, 84431A, doi: 10.1117/12.926937
  * de Hoon et al. (2004) de Hoon, M. J. L., Imoto, S., Nolan, J., & Miyano, S. 2004, Bioinformatics, 20, 1453, doi: 10.1093/bioinformatics/bth078
  * Dickey & Lockman (1990) Dickey, J. M., & Lockman, F. J. 1990, ARA&A, 28, 215, doi: 10.1146/annurev.aa.28.090190.001243
  * Ebeling & Wiedenmann (1993) Ebeling, H., & Wiedenmann, G. 1993, Phys. Rev. E, 47, 704, doi: 10.1103/PhysRevE.47.704
  * Egger & Aschenbach (1995) Egger, R. J., & Aschenbach, B. 1995, A&A, 294, L25, doi: 10.48550/arXiv.astro-ph/9412086
  * Evans et al. (2007) Evans, I. N., Graessle, D. E., He, X. H., & Glotfelty, K. J. 2007, in Astronomical Society of the Pacific Conference Series, Vol. 376, Astronomical Data Analysis Software and Systems XVI, ed. R. A. Shaw, F. Hill, & D. J. Bell, 335
  * Evans et al. (2006a) Evans, I. N., Cresitello-Dittmar, M., Doe, S., et al. 2006a, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 6270, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, ed. D. R. Silva & R. E. Doxsey, 62701U, doi: 10.1117/12.672641
  * Evans et al. (2010) Evans, I. N., Primini, F. A., Glotfelty, K. J., et al. 2010, ApJS, 189, 37, doi: 10.1088/0067-0049/189/1/37
  * Evans et al. (2006b) Evans, J. D., Cresitello-Dittmar, M., Doe, S., et al. 2006b, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 6270, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, ed. D. R. Silva & R. E. Doxsey, 62700N, doi: 10.1117/12.672660
  * Freeman et al. (2001) Freeman, P., Doe, S., & Siemiginowska, A. 2001, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4477, Astronomical Data Analysis, ed. J.-L. Starck & F. D. Murtagh, 76–87, doi: 10.1117/12.447161
  * Freeman et al. (2002) Freeman, P. E., Kashyap, V., Rosner, R., & Lamb, D. Q. 2002, ApJS, 138, 185, doi: 10.1086/324017
  * Fruscione et al. (2006) Fruscione, A., McDowell, J. C., Allen, G. E., et al. 2006, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 6270, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, 62701V, doi: 10.1117/12.671760
  * Gaia Collaboration et al. (2021) Gaia Collaboration, Brown, A. G. A., Vallenari, A., et al. 2021, A&A, 649, A1, doi: 10.1051/0004-6361/202039657
  * Gaia Collaboration et al. (2022) Gaia Collaboration, Klioner, S. A., Lindegren, L., et al. 2022, A&A, 667, A148, doi: 10.1051/0004-6361/202243483
  * Gaia Collaboration et al. (2023) Gaia Collaboration, Vallenari, A., Brown, A. G. A., et al. 2023, A&A, 674, A1, doi: 10.1051/0004-6361/202243940
  * Garmire et al. (2003) Garmire, G. P., Bautz, M. W., Ford, P. G., Nousek, J. A., & Ricker, George R., J. 2003, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4851, X-Ray and Gamma-Ray Telescopes and Instruments for Astronomy., ed. J. E. Truemper & H. D. Tananbaum, 28–44, doi: 10.1117/12.461599
  * Gelman et al. (2013) Gelman, A., Carlin, J. B., Stern, H. S., et al. 2013, Bayesian Data Analysis (3rd ed.) (Boca Raton, FL, USA: CRC Press), doi: 10.1201/b16018
  * George et al. (2007) George, I. M., Arnaud, K. A., Pence, W., Ruamsuwan, L., & Corcoran, M. F. 2007, The Calibration Requirements for Spectral Analysis (Definition of RMF and ARF file formats), OGIP Calibration Memo CAL/GEN/92-002, Tech. rep., NASA Goddard Space Fight Center, Greenbelt, MD, USA.  https://heasarc.gsfc.nasa.gov/docs/heasarc/caldb/docs/memos/cal_gen_92_002/cal_gen_92_002.pdf
  * Górski et al. (2005) Górski, K. M., Hivon, E., Banday, A. J., et al. 2005, ApJ, 622, 759, doi: 10.1086/427976
  * Graessle et al. (2006) Graessle, D. E., Evans, I. N., Glotfelty, K., et al. 2006, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 6270, Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, ed. D. R. Silva & R. E. Doxsey, 62701X, doi: 10.1117/12.672876
  * Gregory & Loredo (1992) Gregory, P. C., & Loredo, T. J. 1992, ApJ, 398, 146, doi: 10.1086/171844
  * Hoffman & Gelman (2014) Hoffman, M. D., & Gelman, A. 2014, JMLR, 15, 1593, doi: 10.5555/2627435.2638586
  * Houck (2000) Houck, J. C. 2000, Removing Streaks from ACIS-S4, Tech. rep., Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/ciao/download/doc/destreak.ps
  * Houck (2013) —. 2013, Constructing Background Images using Voronoi Tessellation, Tech. rep., Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Houck_mkvtbkg.pdf
  * Jerius et al. (1995) Jerius, D., Freeman, M., Gaetz, T., Hughes, J. P., & Podgorski, W. 1995, in Astronomical Society of the Pacific Conference Series, Vol. 77, Astronomical Data Analysis Software and Systems IV, ed. R. A. Shaw, H. E. Payne, & J. J. E. Hayes, 357
  * Jerius et al. (2004) Jerius, D. H., Cohen, L., Edgar, R. J., et al. 2004, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 5165, X-Ray and Gamma-Ray Instrumentation for Astronomy XIII, ed. K. A. Flanagan & O. H. W. Siegmund, 402–410, doi: 10.1117/12.509378
  * Joye & Mandel (2003) Joye, W. A., & Mandel, E. 2003, in Astronomical Society of the Pacific Conference Series, Vol. 295, Astronomical Data Analysis Software and Systems XII, ed. H. E. Payne, R. I. Jedrzejewski, & R. N. Hook, 489
  * Karzas & Latter (1961) Karzas, W. J., & Latter, R. 1961, ApJS, 6, 167, doi: 10.1086/190063
  * Kellogg et al. (1975) Kellogg, E., Baldwin, J. R., & Koch, D. 1975, ApJ, 199, 299, doi: 10.1086/153692
  * Kuiper (1960) Kuiper, N. H. 1960, in K. Ned. Akad. Wet. Proc. Ser. A, Vol. 63, 38–47
  * Li et al. (2004) Li, J., Kastner, J. H., Prigozhin, G. Y., et al. 2004, ApJ, 610, 1204, doi: 10.1086/421866
  * Luo et al. (2017) Luo, B., Brandt, W. N., Xue, Y. Q., et al. 2017, ApJS, 228, 2, doi: 10.3847/1538-4365/228/1/2
  * Martínez-Galarza (2022) Martínez-Galarza, J. R. 2022, Specification Document: Tying CSC Astrometry to the Gaia Reference Frame. Translation-only approach, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Martinez-Galarza_stack_astrometry_translation_spec_doc.pdf
  * Martínez-Galarza (2024) —. 2024, Specification Document: Revision to the MLE algorithm for CSC source fitting, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Martinez-Galarza_Revision_to_the_MLE_algorithm_for_CSC_source_fitting.pdf
  * Massey (1951) Massey, F. J. 1951, J. Am. Stat. Assoc., 46, 68, doi: 10.1080/01621459.1951.10500769
  * McCollough & Rots (2005) McCollough, M. L., & Rots, A. H. 2005, in Astronomical Society of the Pacific Conference Series, Vol. 347, Astronomical Data Analysis Software and Systems XIV, ed. P. Shopbell, M. Britton, & R. Ebert, 478
  * McCollough & Rots (2008) McCollough, M. L., & Rots, A. H. 2008, in Astronomical Society of the Pacific Conference Series, Vol. 394, Astronomical Data Analysis Software and Systems XVII, ed. R. W. Argyle, P. S. Bunclark, & J. R. Lewis, 559
  * McCollough et al. (2015) McCollough, M. L., Siemiginowska, A., & Burke, D. 2015, Chandra Source Catalog (CAT2) Spectra and Spectral Fitting Specifications, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/McCollough_CSC_2_Spectra_Fitting_Spec.pdf
  * McCollough et al. (2019) —. 2019, Default Spectral Fit Parameters for the Chandra Source Catalog 2.0, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/McCollough_Def_Spec_Parm_CSC2.pdf
  * McCollough et al. (2021) McCollough, M. L., Siemiginowska, A., Burke, D., & Graessle, D. 2021, Specifications for Including the APEC Model in CSC 2.1, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/McCollough_specfit_csc2.1_apec_spec.pdf
  * McDowell (2001) McDowell, J. C. 2001, Coordinate Systems for Analysis of On-Orbit Chandra Data, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/contrib/jcm/ncoords.ps
  * Murray et al. (2000) Murray, S. S., Austin, G. K., Chappell, J. H., et al. 2000, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4012, X-Ray Optics, Instruments, and Missions III, ed. J. E. Truemper & B. Aschenbach, 68–80, doi: 10.1117/12.391591
  * Nowak (2012) Nowak, M. A. 2012, Background Filtering for Release 2 of the Chandra Source Catalog, Tech. rep., Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Nowak_l3_bkg_test.pdf
  * Nowak (2016a) —. 2016a, Suggestions for False Source Likelihood Thresholds, Tech. rep., Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Nowak_csc2_thresholds_hrc.pdf
  * Nowak (2016b) —. 2016b, False Source Likelihood Thresholds, HRC, Tech. rep., Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Nowak_csc2_thresholds.pdf
  * Nowak (2016c) —. 2016c, Specifications for Chandra Source Catalog Release 2: Variability Flags and Hardness Ratios, Tech. rep., Kavli Institute for Astrophysics and Space Research, Massachusetts Institute of Technology, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Nowak_csc2_variability_and_color.pdf
  * Park et al. (2006) Park, T., Kashyap, V. L., Siemiginowska, A., et al. 2006, ApJ, 652, 610, doi: 10.1086/507406
  * Primini (2012) Primini, F. A. 2012, Limiting Sensitivity Specification, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Primini_limiting-sensitivity.pdf
  * Primini (2013) —. 2013, Aperture Photometry Specifications for CSC R2, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Primini_aperture_photometry_specs.pdf
  * Primini (2019) —. 2019, Specifications for Combining Data from Multiple ObsIDs in Aperture Photometry, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Primini_Combining_Data_from_Multiple_ObsIDs_in_Aperture_Photometry.pdf
  * Primini (2020) —. 2020, Implementing pymc3 in CSC 2.1 Aperture Photometry, Tech. rep., Center for Astrophysics | Harvard & Smithsonian, Cambridge, MA, USA.  https://cxc.cfa.harvard.edu/csc/memos/files/Primini_Implementing_pymc3_in_CSC2_1_Aperture_Photometry.pdf
  * Primini & Kashyap (2014) Primini, F. A., & Kashyap, V. L. 2014, ApJ, 796, 24, doi: 10.1088/0004-637X/796/1/24
  * Primini et al. (2011) Primini, F. A., Houck, J. C., Davis, J. E., et al. 2011, ApJS, 194, 37, doi: 10.1088/0067-0049/194/2/37
  * Rots & Budavári (2011) Rots, A. H., & Budavári, T. 2011, ApJS, 192, 8, doi: 10.1088/0067-0049/192/1/8
  * Salvatier et al. (2016) Salvatier, J., Wiecki, T., & Fonnesbeck, C. 2016, PeerJ Computer Science, 2, e55, doi: 10.7717/peerj-cs.55
  * Scargle et al. (2013) Scargle, J. D., Norris, J. P., Jackson, B., & Chiang, J. 2013, ApJ, 764, 167, doi: 10.1088/0004-637X/764/2/167
  * Schaap (2007) Schaap, W. E. 2007, PhD thesis, Kapteyn Astronomical Institute
  * Shewchuk (1996) Shewchuk, J. R. 1996, in Workshop on Applied Computational Geometry, Springer, 203–222
  * Siemiginowska et al. (2011) Siemiginowska, A., Kashyap, V., Refsdal, B., et al. 2011, in Astronomical Society of the Pacific Conference Series, Vol. 442, Astronomical Data Analysis Software and Systems XX, ed. I. N. Evans, A. Accomazzi, D. J. Mink, & A. H. Rots, 439
  * Smith et al. (2001) Smith, R. K., Brickhouse, N. S., Liedahl, D. A., & Raymond, J. C. 2001, ApJ, 556, L91, doi: 10.1086/322992
  * Storn & Price (1997) Storn, R., & Price, K. 1997, JGlobOpt, 11, 341, doi: 10.1023/A:1008202821328
  * Tanemura (2003) Tanemura, M. 2003, Forma, 221
  * Taylor (2005) Taylor, M. B. 2005, in Astronomical Society of the Pacific Conference Series, Vol. 347, Astronomical Data Analysis Software and Systems XIV, ed. P. Shopbell, M. Britton, & R. Ebert, 29
  * van Dyk et al. (2001) van Dyk, D. A., Connors, A., Kashyap, V. L., & Siemiginowska, A. 2001, ApJ, 548, 224, doi: 10.1086/318656
  * Watson et al. (2009) Watson, M. G., Schröder, A. C., Fyfe, D., et al. 2009, A&A, 493, 339, doi: 10.1051/0004-6361:200810534
  * Weisskopf et al. (2002) Weisskopf, M. C., Brinkman, B., Canizares, C., et al. 2002, PASP, 114, 1, doi: 10.1086/338108
  * Weisskopf et al. (2000) Weisskopf, M. C., Tananbaum, H. D., Van Speybroeck, L. P., & O’Dell, S. L. 2000, in Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, Vol. 4012, X-Ray Optics, Instruments, and Missions III, ed. J. E. Truemper & B. Aschenbach, 2–16, doi: 10.1117/12.391545