# Measuring Unintended Memorisation of Unique Private Features in Neural
Networks

John Hartley  School of Engineering, University of Edinburgh, West Mains Rd,
Edinburgh EH9 3FB, UK<EMAIL_ADDRESS>Sotirios A. Tsaftaris  School of
Engineering, University of Edinburgh, West Mains Rd, Edinburgh EH9 3FB, UK
The Alan Turing Institute, London, UK<EMAIL_ADDRESS>
###### Abstract

Neural networks pose a privacy risk to training data due to their propensity
to memorise and leak information. Focusing on image classification, we show
that neural networks also unintentionally memorise unique features even when
they occur only once in training data. An example of a unique feature is a
person’s name that is accidentally present on a training image. Assuming
access to the inputs and outputs of a trained model, the domain of the
training data, and knowledge of unique features, we develop a score estimating
the model’s sensitivity to a unique feature by comparing the KL divergences of
the model’s output distributions given modified out-of-distribution images.
Our results suggest that unique features are memorised by multi-layer
perceptrons and convolutional neural networks trained on benchmark datasets,
such as MNIST, Fashion-MNIST and CIFAR-10. We find that strategies to prevent
overfitting (e.g. early stopping, regularisation, batch normalisation) do not
prevent memorisation of unique features. These results imply that neural
networks pose a privacy risk to rarely occurring private information. These
risks can be more pronounced in healthcare applications if patient information
is present in the training data.

##  1 Introduction

Deep Neural Networks (DNNs) are a powerful tool for classifying images (LeCun
et al.,, 1989; Krizhevsky et al.,, 2012; He et al.,, 2016; Huang et al.,,
2017). It is now commonly known that DNNs memorise training labels when the
number of trainable parameters in the network is greater than the cardinality
of the training set (Zhang et al., 2021a, ). This is the case for real, noisy,
and random data, whether their labels have been shuffled or not (Arplt et
al.,, 2017).

Figure 1: What will happen if a training dataset of X-ray images mistakenly
contains a single image that shows the name of a patient? Will this unique
information be exploited (in other words memorised) by a neural network? As we
describe in the text, such a mistake is possible as data sanitisation is not
always perfect. Here we explore these questions by injecting fake visual
features in canary images from known benchmark datasets (an example canary
image is shown on the right). We devise a memorisation score that assesses
whether such unintended memorisation is possible. We explore whether
architecture, regularisation, and the canary image matter, respectively. The
short answer is Yes. [To alleviate any privacy concerns, this X-ray image is
in the public domain Wikipedia, (2022), and the patient name we use is
fictional and commonly used in the media to denote an unknown individual.]

Recent work by Feldman, (2020); Feldman and Zhang, (2020) has established
theoretically and empirically that DNNs can achieve close to optimal
generalisation error in image classification tasks data when examples,
predominantly rare and atypical from long-tailed data distributions, are
memorised. It has also been shown empirically for several benchmark datasets,
that memorised examples exert a large influence on the network predictions for
atypical but visually similar examples in the test set (Feldman and Zhang,,
2020).

The memorisation of examples poses a privacy risk since information relating
to the example is encoded directly in the weights of a neural network Golatkar
et al., (2020). For example, an adversary could construct a readout function
acting on the weights or network outputs to discover information about a given
example Shokri et al., (2017). Data leakage is particularly problematic when
datasets contain private information for which disclosure is limited. For
example, DNNs used in healthcare may encode information about patients in
their weights (Bar et al.,, 2015; Zech et al.,, 2018), for which disclosure is
limited in the EU by the General Data Protection Regulation (GDPR).

Herein, we focus on the unintended memorisation of specific image features as
opposed to examples or training labels. Although it has been shown that
training examples are memorised (Zhang et al., 2021a, ) it is not clear
whether an example is memorised in its entirety or whether specific features
of the image are memorised. This distinction is important since private
features in an image could be memorised, and leaked. For example, hospitals
frequently employ sanitisation processes to remove patient names when they
appear overlaid on X-ray films (see example in Figure 1). But not all
processes are foolproof, and occasionally an image with a patient’s name will
make it to the training dataset. A classifier trained on this data may
misdiagnose other patients with the same name if those names also have not
been removed. Another more likely risk is that this unintended presence may
lead to incomplete extraction of the correct discriminative features from the
image. Such a risk is similar to decision making based on spurious
correlations, except that only a single spurious feature is contained in the
dataset (Zech et al.,, 2018; Geirhos et al.,, 2020; Idrissi et al.,, 2021).

We propose that unique feature memorisation can be detected through data
leakage. Methods that exploit data leakage to uncover information about
training data are called privacy attacks. One example is the membership
inference attack (MIA) which finds whether an example is in the training set.
This is achieved by exploiting a model’s overconfidence on examples it has
seen (Shokri et al.,, 2017; Sablayrolles et al.,, 2018; Salem et al.,, 2018;
Liu and Tsaftaris,, 2020; Choquette-Choo et al.,, 2021). Here instead we focus
on the memorisation of unique features and not whole data. Adopting the
terminology from Jegorova et al., (2021) we will call this feature leakage.

Feature leakage occurs in large language models (LLM) due to unintended
memorisation (Carlini et al., 2019b, ; Carlini et al.,, 2021). Unique
sequences, such as credit card numbers or rarely occurring phrases, are likely
to be memorised by a model since they are random and cannot be learnt from
similar patterns in the rest of the training data.

We extend the notion of feature leakage in large language models to the
supervised image classification setting. We consider a training set that
contains a single private unique feature $\mathbf{z}_{\mathrm{u}}$ which is
present in a single image $\mathbf{x}$, an example of which we show in Figure
1. We are interested in detecting whether $\mathbf{z}_{\mathrm{u}}$ is
memorised by a neural network.

We aim to develop a black-box methodology to measure the memorisation of known
unique private features in a trained model, assuming access only to the inputs
and outputs of the model and to the unique private feature. We assume that we
do not have access to the training data, but we do know its domain 111For
example, the domain of an X-ray dataset would be X-rays..

Our main contributions are summarised as follows:

  * •

A new setting of assessing unique feature memorisation in images.

  * •

A score to assess the memorisation of unique features in neural networks where
only access to the inputs/outputs and the unique feature is allowed.

  * •

We find that neural networks memorise unique features injected into several
benchmark datasets MNIST, Fashion-MNIST and CIFAR-10. This holds even when a
single unique feature is present once in the dataset, and before overfitting
in an average sense occurs.

  * •

We find that the risk of such memorisation is not eliminated by adding
explicit/implicit regularisation. On the contrary, memorisation is more
likely.

  * •

We hypothesise that hard-to-learn data influence the memorisation of features.
We find that the degree of memorisation is weakly correlated with the self-
influence score (a measure of learning difficulty) of the training image on
which the unique feature is present.

##  2 Related Works

To the best our of knowledge, no previous study has investigated methods that
detect memorisation of unique features in image classification models.
However, existing research recognises the critical role that memorisation
plays in the overfitting and generalisation of neural networks and the
security implications of attacks using adversarial training examples. Below we
outline work on memorisation that has provided encouraging findings to help us
shape this work. We hypothesise that hard-to-learn examples can make
memorisation of features more likely so we cover also related work on finding
high influence examples.

###  2.1 Memorisation

The main objective of this work is to understand whether memorisation occurs
for unique features that occur once per dataset and are located on a single
training example. A secondary aim is to show that on average, the onset of
this behaviour occurs before overfitting happens.

Previous research has established that over-trained, over-parameterised neural
networks memorise training labels (Zhang et al., 2021a, ), with several
metrics for assessing the memorisation of training labels having been
previously introduced (Feldman and Zhang,, 2020; Jiang et al.,, 2021).

It was shown that DNNs first learn common patterns in training examples, after
which they memorise labels (Arplt et al.,, 2017; Kim et al., 2018b, ). More
recently, it has been shown that learning and memorisation occur
simultaneously (Liu et al., 2021b, ). These works also show that
regularisation does not eliminate memorisation. We take inspiration from these
works to explore whether the effects of regularisation are similar in unique
feature memorisation.

Recent work by Carlini et al., 2019b  has established that large language
models memorise unique phrases early in training. They show a black-box
inference method for measuring memorisation in trained models. This work has
inspired us to draw upon the use of canaries to measure the memorisation of
unique features in images. We define a canary as a training example that
contains an artificially injected unique feature. We follow their idea to also
investigate memorisation before overlearning occurs.

Property inference attacks attempt to learn some group property/feature of the
dataset. For example, what proportion of people in the training set wear
glasses? (Ateniese et al.,, 2015; Ganju et al.,, 2018). These attacks are
typically whitebox and proceed by inference on model weights. Feature
memorisation, as we investigate here, can be viewed as an extreme property
inference attack where a unique feature, a person who wears glasses, occurs
only once in the dataset. Such approaches, however, cannot address unique
feature memorisation since labelling the training weights requires knowledge
of whether the feature was memorised or not.

We also draw upon the idea of concept activation vectors (CAV), as introduced
by Kim et al., 2018a , to infer the sensitivity of the network to the feature
we are interested in vs. random features that we are not. However, instead of
image concepts such as stripes, we focus on unique features and use only the
outputs of the trained model as opposed to the internal activations of the
network. Our memorisation score also does not require the label of the unique
feature.

###  2.2 Finding high influence examples

Memorisation can occur when DNNs generalise from examples that are mislabelled
or belong to sub-populations/long-tails within classes to samples in the test
set (Feldman,, 2020; Feldman and Zhang,, 2020). Similarly, research into long-
tailed learning establishes that models have a worst-group accuracy as a
result of class imbalances or spurious correlations in datasets (Liu et al.,,
2020; Zhang et al., 2021b, ; Liu et al., 2021a, ).

We hypothesise that unique features injected on such examples are more likely
to be memorised. These examples can be found using influence functions, or
proxy functions to influence functions (Koh and Liang,, 2017; Katharopoulos
and Fleuret,, 2018; Carlini et al., 2019a, ; Ghorbani and Zou,, 2019; Toneva
et al.,, 2019; Feldman and Zhang,, 2020; Garima et al.,, 2020; Guo et al.,,
2020; Baldock et al.,, 2021; Harutyunyan et al.,, 2021; Jiang et al.,, 2021).
Influence functions are generally expensive to compute for deep learning
models since they rely on the difference in the expected model predictions of
models which are trained with and without the example under test. However,
approximations using sampling, model approximations or alternatives based on
learning dynamics are more feasible. We make extensive use of the
computationally efficient TracIn score to estimate the influence of an example
on its prediction by a trained model (Garima et al.,, 2020).

##  3 Methodology

###  3.1 Problem statement

DNNs are known to memorise training examples in image classification tasks. We
propose that DNNs also independently memorise unique features of image
examples even when they occur extremely rarely in the training data. This
behaviour is a privacy concern when those features are private, such as a name
or an address. To answer this question we discuss a metric that approximates
the memorisation of a unique feature that occurs once in a training set.

We discuss a score that can verify the privacy of a model trained on data that
may or may not contain private information. Our setting assumes the
possibility of a fallible preprocessor that while tasked to remove private
features fails on a single case.

Concretely, we limit our discussion to the following case: we measure whether
a given unique feature is memorised given black-box access to the model
(inputs and outputs), and without access to training examples or to the
training labels.

We define a unique feature as a unique set of pixels in a single image within
the training set. This feature could be a name, a postcode, or other
information which should not be disclosed. We do not know a priori whether the
feature is in the dataset or not since we cannot catch errors in the
preprocessor, and therefore we cannot construct a dataset without the unique
feature.222Thus, ruling out leave-one-out measures for memorisation. However,
we do assume we know the form that the unique feature takes since we are aware
of the process which generates it (see Section 3.4.1).

###  3.2 Notation

We define a neural network image classification model
$f(\mathbf{x};D_{\mathrm{t}})$, which maps an image $\mathbf{x}$ to a vector
$\mathbf{y}$ where each element represents the conditional probability of the
class label $y$ given the image $\mathbf{x}$,
$D_{\mathrm{t}}=\\{\mathbf{x}^{i},y^{i}\\}^{n}_{i=0}$ is the training data
where $\mathbf{x}_{p}\in\mathbb{R}^{n\times n}$, and $y$ is the ground truth
class label of $\mathbf{x}$. $D_{\mathrm{t}}$ may or may not contain a datum
$\mathbf{x}_{p}$ with a unique feature
$\mathbf{z}_{\mathrm{u}}\in\mathbb{R}^{m\times m}$ with $m<n$. We define an
additional random feature $\mathbf{z}_{\mathrm{r}}\sim U$, with the same
dimensionality as $\mathbf{z}_{\mathrm{u}}$. We define three datasets
$D_{\mathrm{c}}=\\{\mathbf{x_{c}}^{i}\\}^{n}_{i=0}$,
$D_{\mathrm{u}}=\\{\mathbf{x_{u}}^{i}\\}^{n}_{i=0}$,
$D_{\mathrm{r}}=\\{\mathbf{x_{r}}^{i}\\}^{n}_{i=0}$ out-of-distribution to
$D_{\mathrm{t}}$ which we use to perform inference on $f$ to learn about
$\mathbf{z}_{\mathrm{u}}$. We make use of the KL divergence between two
discrete probability distributions to measure the difference in the model’s
outputs, $D_{\mathrm{KL}}(P||Q)=\mathbb{E}_{\mathrm{x}\sim
P}\big{[}\log{\frac{P(x)}{Q(x)}}\big{]}$.

###  3.3 Method to measure unique feature memorisation

The central idea to approximate the memorisation of $\mathbf{z}_{\mathrm{u}}$
is to perform a set of inferences on $f$ using out-of-distribution image pairs
which are clean,333Clean means that the image does not contain
$\mathbf{z}_{\mathrm{u}}$ or $\mathbf{z}_{\mathrm{r}}$. or contain
$\mathbf{z}_{r}$ or $\mathbf{z}_{u}$.

We define three datasets for inference $D_{\mathrm{c}}$, $D_{\mathrm{u}}$,
$D_{\mathrm{r}}$. The datasets we use are out-of-distribution (OOD) to the
training dataset. I.e. they are from a different domain to $D_{t}$. We assume
that any dataset which is not the training set, $D_{\mathrm{t}}$, is OOD and
can be used for inference. This is because our method finds only the relative
distances between model outputs from image pair inputs and as such the
specific distribution which the model outputs is not important. Each dataset
is a replica of the base dataset $D_{c}$ with the exception that every image
in $D_{u}$ also contains $\mathbf{z}_{\mathrm{u}}$ and every image in $D_{r}$
contains $\mathbf{z}_{\mathrm{r}}$, where $\mathbf{z}_{\mathrm{r}}$ is drawn
randomly for every image. Figure 2 shows examples of
$\mathbf{x}_{\mathrm{c}}$, $\mathbf{x}_{\mathrm{u}}$,
$\mathbf{x}_{\mathrm{r}}$.

(a) $\mathbf{x}_{\mathrm{c}}$

(b) $\mathbf{x}_{\mathrm{u}}$

(c) $\mathbf{x}_{\mathrm{r}}$

Figure 2: Example image pairs from greyscale CIFAR-10 used for inference on
$f$ trained on MNIST or Fashion-MNIST.

We assume that an $f$ which has memorised $\mathbf{z}_{\mathrm{u}}$ will, on
average, be more sensitive to images that contain $\mathbf{z}_{\mathrm{u}}$
than $\mathbf{z}_{\mathrm{r}}$. We define memorisation as any learning which
takes place on $\mathbf{z}_{\mathrm{u}}$. Since $\mathbf{z}_{\mathrm{u}}$ is
unique and cannot be learnt from any other label structure in the training
data, any learning which does occur must be memorisation.

We assume that $f$ has been trained with a softmax activation on the final
layer under maximum-likelihood such that $f(\mathbf{x}_{\mathrm{c}})$,
$f(\mathbf{x}_{\mathrm{u}})$, $f(\mathbf{x}_{\mathrm{r}})$ are valid
probability distributions, and such that the KL divergences of any combination
are valid. Consider measuring the network outputs of $f$ for every image pair
in the OOD inference datasets $D_{\mathrm{c}}$, $D_{\mathrm{u}}$,
$D_{\mathrm{r}}$; we calculate the following divergences:

$d_{u}^{i}=D_{\mathrm{KL}}\big{(}f(\mathbf{x}_{\mathrm{c}}^{i})||f(\mathbf{x}_{\mathrm{u}}^{i})\big{)},$
(1)
$d_{r}^{i}=D_{\mathrm{KL}}\big{(}f(\mathbf{x}_{\mathrm{c}}^{i})||f(\mathbf{x}_{\mathrm{r}}^{i})\big{)},$
(2)

where $d_{u}^{i}$, and $d_{r}^{i}$ measure the distances in the predictions
from $\mathbf{x}_{\mathrm{c}}^{i}$ to $\mathbf{x}_{\mathrm{u}}^{i}$ and
$\mathbf{x}_{\mathrm{r}}^{i}$ where $i$ is a sample index in the datasets
$D_{u}$, $D_{c}$ and $D_{r}$.

When the divergences $d_{u}^{i}$, and $d_{r}^{i}$ are zero there is no
difference between the prediction on the clean image and the random or unique
feature image, whereas a network that is more sensitive to
$\mathbf{z}_{\mathrm{u}}$ than any given $\mathbf{z}_{\mathrm{r}}$ will have
$d_{u}^{i}>d_{r}^{i}$.

Since $\mathbf{z}_{\mathrm{r}}$ is random it is possible that
$\mathbf{z}_{\mathrm{r}}$ is already a feature learnt by the classifier, and
thus $d_{u}^{i}<d_{r}^{i}$. However, we assume that the subspace of such
features is far smaller than the randomness space of
$\mathbf{z}_{\mathrm{r}}$, and thus most examples that we draw will not be
features learnt by the classifier. Also for some $\mathbf{z}_{\mathrm{r}}$ the
network may be more or less sensitive due to the lack of robustness of CNNs to
out-of-distribution data (Shao et al.,, 2020). Therefore, to mitigate the
similarity of random features to those learnt by the classifier and robustness
issues, we measure the sensitivity of the network by taking the average over
every $\mathbf{z}_{\mathrm{r}}$ and $\mathbf{x}_{\mathrm{c}}$. We abuse the
notation and assume that each draw of $\mathbf{z}_{\mathrm{r}}$ results in an
$\mathbf{x}_{\mathrm{r}}$ with a new random unique feature.

$X_{\mathrm{u}}=\mathbb{E}_{\mathrm{\mathbf{x}_{c}}\sim
P}\Big{[}D_{\mathrm{KL}}\big{(}f(\mathbf{x}_{\mathrm{c}})||f(\mathbf{x}_{\mathrm{u}})\big{)}\Big{]},$
(3) $X_{\mathrm{r}}=\mathbb{E}_{\mathrm{\mathbf{z}_{r}}\sim
U}\Big{[}\mathbb{E}_{\mathrm{\mathbf{x}_{c}}\sim
P}\Big{[}D_{\mathrm{KL}}\big{(}f(\mathbf{x}_{\mathrm{c}})||f(\mathbf{x}_{\mathrm{r}})\big{)}\Big{]}\Big{]},$
(4)

where $P$ is the data distribution of the base OOD dataset. We assume that
drawing $\mathbf{x}_{\mathrm{c}}$ also draws the image pair
$\mathbf{x}_{\mathrm{u}}$ with the unique feature. We approximate equations 3
and 4 by sampling from a benchmark dataset that is not the training dataset,
and by $\mathbf{z}_{r}\sim U$ for every image in the OOD dataset. For
memorisation tests on MNIST and Fashion-MNIST we use a greyscale version of
CIFAR-10 for inference, and for tests on CIFAR-10 we use a 3-channel version
of the MNIST dataset.

The score $M$ We define our memorisation score for $\mathbf{z}_{\mathrm{u}}$
as $M=X_{u}-X_{r}$, where $M>0$ corresponds to memorisation of the unique
feature. To illustrate the statistical significance of the differences in the
means we use a one-tail t-test with the alternative hypothesis that
$X_{u}>X_{r}$.

###  3.4 Setting up canaries

####  3.4.1 Measuring the memorisation of canaries

We use the following test setup to approximate the memorisation of unique
features. An ideal test of the memorisation score is to measure the
memorisation of a feature on an image that we know contains a unique feature.
Since we do not know a priori which unique features have been accidentally
left in the training data by a fallible pre-processor, we adopt the strategy
of placing canaries into the training data artificially (Carlini et al.,
2019b, ). Once a training example is identified as a canary, the model is
trained on the augmented dataset. Next, the memorisation of the unique feature
is approximated using the memorisation score method.

The experiments in this work use as unique feature a tiny patch of the letter
‘A’. This patch is $5\times 5$ pixels, and it is inserted into the top-left
corner one pixel from the top-left corner of the canary image. Examples of the
unique feature, A, embedded in canary images are shown in Figure 3. We assume
that the service knows the typography of the letter A, its size and its
location in the image. For some applications, this is a reasonable assumption
since these properties of the feature are public. For example, consider a
personal ID verification classifier. Although the name and address of an ID
owner are private, the typography, size and location of the private
information are the same on every ID and therefore public.

####  3.4.2 Selecting high self-influence examples as canaries

We do not expect that the unique feature will be memorised for every canary
since not all training examples are memorised by neural networks (Krueger et
al.,, 2019). Therefore it is unknown a priori which canaries will have unique
features that will be memorised. This makes it challenging find to find true
positives to test our score against.

We do not want to search every example for the possibility of unique feature
memorisation for computational reasons. Additionally, we cannot add unique
features to every training example and train once since this will alter the
characteristics of the dataset. Therefore we should choose efficiently which
examples should have the unique feature. Here we develop a strategy to choose
examples based on a self-influence score for each example.

It has been shown that memorisation of training labels occurs when training
examples from sub-populations of a class are mislabelled, or are members of an
under-sampled class (Feldman and Zhang,, 2020). It is also expected that sub-
population or mislabelled examples are highly-influential. This is because
they have unique features which require memorisation to classify them
correctly (Harutyunyan et al.,, 2021).

We suppose that unique features are more likely to be memorised when they are
present on examples that are highly influential to predictions made on
themselves. Conversely, we expect that if $\mathbf{z}_{\mathrm{u}}$ is present
on a low-influence sample it is unlikely to be memorised since the sample can
be predicted by features in similar low-influence samples. Furthermore the
information bottleneck (IB) principle suggests that the network should learn
only relevant, and minimal, information (representations) between the input
and the output task. Thus, according to the IB principle, a network should
ignore unique, features which appear random, (Tishby and Zaslavsky,, 2015;
Achille and Soatto,, 2018), as such features will be considered nuisance.
Finally, it is possible that a unique feature may contribute to shortcut
learning (Geirhos et al.,, 2020) as we touch on in our discussion.

To determine high-influence examples we use TracIn (Garima et al.,, 2020).
TracIn measures the self-influence of an example by computing the training
loss on itself between successive iterations of stochastic gradient descent
(SGD) where the loss is computed for that example. Large successive
differences in loss will be associated with samples whose predictions can only
be improved upon by the network learning from that example or a small subset
of similar examples. Small changes will be associated with low-influence
examples since learning will be spread over many examples with correlated
features. TracIn is approximated for mini-batch (SGD) by measuring the sums of
the squared gradients of the loss evaluated on the test example over several
checkpoints (see equation 5). We use TracIn as opposed to other influence
functions as it is extremely computationally efficient. The function for
computing TracIn using checkpoints is given by

$\textrm{TracInCP}=\sum{\eta_{i}}||\nabla l(\mathbf{w}_{i},\mathbf{x})||^{2},$
(5)

where $\mathbf{w}_{i}$ and $\eta_{i}$ are the weights and learning rate of $f$
and the optimisation algorithm at checkpoint $i$. $\mathbf{x}$ is the self-
influence test image.

For each dataset/model combination in this work, we measure the self-influence
of all training examples by selecting 10 evenly spaced checkpoints which
account for a 95% reduction in the training loss (Garima et al.,, 2022). We
select the top-15 and bottom-15 examples as canaries. We create 30 models,
each of which is trained on the canary with the addition of a unique feature.

####  3.4.3 Early Stopping

It is well known that over-parameterised neural networks memorise random
training labels when they are trained indefinitely (Zhang et al., 2021a, ).
However, it has also been shown that LLMs memorise unique phrases even before
overfitting in an average sense occurs (Carlini et al., 2019b, ). We refer to
models trained without overfitting on average as well-trained. We suggest that
unique features are also memorised by image classification models even when
these models are well-trained. Specifically, we test this hypothesis by making
use of early stopping during model training with unique features. All models
are trained for 500 epochs with 10 patient epochs, and after training, we
select the model checkpoint which has the lowest validation loss. We train
models using the Adam optimiser and with a cross-entropy loss function (Kingma
and Ba,, 2015).

##  4 Results

Our primary aim is to show that MLPs and CNNs memorise unique features. In
Section 4.1 we start showing that the memorisation score is capable of
detecting lack of memorisation and actual memorisation in experiments with
MLPs on MNIST (Lecun et al.,, 1998) and Fashion-MNIST (Xiao et al.,, 2017). We
then proceed (Section 4.2) to show that unique features are also memorised in
MNIST, Fashion-MNIST and CIFAR-10 (Krizhevsky et al.,, 2009) for three common
CNN architectural styles. One might expect that explicit and implicit
regularisers which prevent overfitting should also reduce memorisation;
however in Section 4.3 we show results to the contrary. Finally, in Section
4.4 we discuss the correlation between unique feature memorisation and the
self-influence of training examples.

###  4.1 Memorisation of unique features in MLPs

(a) MNIST canary with unique feature A

(b) Fashion-MNIST canary with unique feature A

(c) CIFAR-10 canary with unique feature A

Figure 3: Canaries for testing unique feature memorisation. Each canary has an
A burnt into its top-left corner.

####  4.1.1 Baseline memorisation test on MNIST

We show in an experiment on MNIST that the $M$ score is less than zero for all
canaries when the unique features are omitted from the training data, and that
it can be greater than zero for canaries when unique features are memorised.

We use an MLP with two dense layers with 512, 256 outputs respectively, and
non-linear ReLU activation functions on each layer. We train with a learning
rate of $1\text{\times}{10}^{-4}$ and a batch size of 128.

We consider two setups: the first where the training dataset is clean, and the
second where we introduce a canary on a random image sampled from the training
data. We repeat each setup 15 times.

We find no cases where $M>0$ on the clean dataset. This expected result
suggests that our memorisation score has high specificity. When we introduce
features we find eight canaries (with injected unique features) with $M>0$.
(These results are statistically significant using the method in Section 3.3.)
We do not expect that a unique feature will be memorised for every canary;
hence, we do not include canaries (with injected features) that have $M\leq
0$.

####  4.1.2 MLP memorisation test results

Next, we evaluate unique feature memorisation in MNIST and Fashion-MNIST for
MLP models. The setup of the network is the same as before.

Table 1 shows the memorisation scores for canaries. Each row shows the result
for a new model trained on a dataset containing a single canary. First, we
show the scores for the top-5 most memorised canaries over the several models
we trained, and second, we show the average value of the memorisation score
over canaries having $M>0$ belonging to a particular dataset. Such averaging
isolates instance-based variation and allows us to investigate how the
memorisation of memorised unique features varies across approaches.

Top-5 results clearly show that memorisation occurs for unique features in the
canaries since $M>0$. We emphasise here that for each result memorisation
occurs when only a single unique feature has been included in the training
data, and that there are no unique features in the test data.

We will use the average memorisation score as a comparison between model
architectures where the choice of canaries varies. The variation occurs
because canaries are chosen by the self-influence score which depends on the
model architecture.

Table 1: Memorisation scores ($M$) of unique features in MLPs trained on MNIST and F-MNIST. All results in bold correspond to test statistics with p-values $<0.05$. The bottom two rows indicate values of $M$ which have been averaged over canaries with $M>0$. IMAGE ID | DATASET | $X_{u}$ | $X_{r}$ | $M$
---|---|---|---|---
27225 | MNIST | 0.021 | 0.017 | 0.0038
6885 | MNIST | 0.027 | 0.025 | 0.0021
27155 | MNIST | 0.011 | 0.0094 | 0.0017
11708 | MNIST | 0.01 | 0.0089 | 0.0012
8898 | MNIST | 0.0088 | 0.0081 | 0.0007
37251 | F-MNIST | 0.058 | 0.028 | 0.03
2731 | F-MNIST | 0.047 | 0.026 | 0.021
2181 | F-MNIST | 0.039 | 0.021 | 0.019
3694 | F-MNIST | 0.051 | 0.033 | 0.019
16002 | F-MNIST | 0.059 | 0.046 | 0.013
AVERAGE | MNIST | 0.013 | 0.012 | 0.0013
AVERAGE | F-MNIST | 0.033 | 0.024 | 0.0093

###  4.2 Memorisation of unique features in CNNs

We extend our investigation to CNN architectures on the same datasets. We test
for memorisation on three architectures: CNN-1 a simple two convolutional
layer CNN; CNN-2 a small VGG-type CNN (Simonyan and Zisserman,, 2015); and
DenseNet a DenseNet with 100 layers (Huang et al.,, 2017).444Only four
canaries were chosen in DenseNet due to computational constraints. See
Appendix A for details on the architectures and their training.

Table 2 shows memorisation scores for canaries. We show first the scores for
the top-5 most memorised canaries over the several models we trained, and
second the average value of the memorisation score for all memorised canaries
belonging to a particular dataset.

It can be seen from the top-5 data in Table 2 that memorisation occurs in all
network architectures for each of the datasets tested (since $M>0$).

We now compare with the results in Table 1 for the MLP architecture.

The average memorisation is greater in the CNN architecture than the MLP
architecture for MNIST ($0.012>0.0013$) and Fashion-MNIST ($0.028>0.0093$). In
fact, the average memorisation in the CNNs is over an order of magnitude
greater for MNIST. We also cautiously suggest that the average memorisation is
greater for CIFAR-10 in DenseNet than in CNN-2 ($0.38>0.018$) though the
sample size for the DenseNet results is small.

Feldman and Zhang, (2020) suggests that example memorisation is closely linked
to model accuracy. We suggest that the increases in memorisation seen in the
CNNs and the DenseNet can also be explained by the model’s accuracy. For
example, DenseNet has a validation accuracy of 92% and CNN-2 72%.

Table 2: Memorisation ($M$) scores of unique features in CNN-1, and CNN-2, a DenseNet trained on MNIST, Fashion-MNIST and CIFAR-10. Bold denotes statistical significance. ID | DATASET | MODEL | $M$
---|---|---|---
51508 | MNIST | CNN-1 | 0.073
14873 | MNIST | CNN-1 | 0.035
7080 | MNIST | CNN-1 | 0.021
43454 | MNIST | CNN-1 | 0.02
47034 | MNIST | CNN-1 | 0.02
59677 | F-MNIST | CNN-1 | 0.085
23711 | F-MNIST | CNN-1 | 0.068
15748 | F-MNIST | CNN-1 | 0.059
12168 | F-MNIST | CNN-1 | 0.058
10477 | F-MNIST | CNN-1 | 0.05
23308 | CIFAR-10 | CNN-2 | 0.058
9461 | CIFAR-10 | CNN-2 | 0.023
7371 | CIFAR-10 | CNN-2 | 0.018
35174 | CIFAR-10 | CNN-2 | 0.017
15726 | CIFAR-10 | CNN-2 | 0.013
32574 | CIFAR-10 | DenseNet | 0.72
772 | CIFAR-10 | DenseNet | 0.25
8022 | CIFAR-10 | DenseNet | 0.18
AVERAGE | CNN-1 | MNIST | 0.012
AVERAGE | CNN-1 | F-MNIST | 0.028
AVERAGE | CNN-2 | CIFAR-10 | 0.018
AVERAGE | DenseNet | CIFAR-10 | 0.38

###  4.3 Effects of explicit and implicit regularisation on memorisation

Typically regularisation is used to reduce a model’s ability to overfit
training data. It has been shown that explicit and implicit regularisation
strategies do not reduce the ability of a network to fit random training data
(Arplt et al.,, 2017). This suggests that they should also have a low
influence on the memorisation of unique features. This experiment explores
exactly this: i.e., whether these strategies eliminate the risk of
memorisation of unique features in networks that on average are not
overfitted. We use three regularisation strategies: dropout, data
augmentation, and batch normalisation, applied in various combinations. After
training, we measure the memorisation of each of the canaries.

Table 3 shows the average memorisation score overall canaries where $M>0$ for
MNIST, Fashion-MNIST and CIFAR-10 trained by two network architectures. Whilst
the top-5 scores have been omitted from the table they show that the
regularisation methods do not eliminate memorisation of unique features since
$M>0$ for every canary. The mean scores in this table can be compared with the
MLP and CNN data in Tables 1 and 2.

The results, taking MNIST for example, illustrate that almost all values are
higher than the average of 0.0013 obtained from Table 1 without any
regularisation. Canaries in CIFAR-10 have greater memorisation scores than in
the non-regularised models shown in Table 2. These results extend to the level
of features, findings that have been made previously for the memorisation of
whole training examples (Zhang et al., 2021a, ).

Data augmentation is a particularly interesting case. When CNN-2 is trained on
CIFAR-10 with data augmentation the average memorisation increases from 0.018
to 0.13. This is interesting because many datasets require data augmentation
to achieve the best accuracy. We offer below a potential explanation as to why
such an increase happens.

Data augmentation is employed to improve the learning of translational and/or
rotational in/equi-variances to image features. Perturbation of canaries
implicitly adds additional canaries to the dataset, and inadvertently increase
the spurious correlation between the unique feature and its label. This
enables easier learning of the unique feature since the collective
contribution to the training loss from the canaries is greater during
training.

Table 3: Average memorization ($M$) scores over canaries with $M>0$ for unique features for models with explicit and implicit regularisers, such Dropout, Data Augmentation (Augm.), and Batch Normalization (Batch Norm.). DATASET | MODEL | REGULARISATION | $M$
---|---|---|---
MNIST | MLP | Dropout | 0.02
MNIST | MLP | Augm. | 0.0033
MNIST | MLP | Dropout & Augm. | 0.0013
MNIST | MLP | Batch Norm. | 0.039
F-MNIST | MLP | Dropout | 0.025
F-MNIST | MLP | Augm. | 0.0065
F-MNIST | MLP | Dropout & Augm. | 0.029
F-MNIST | MLP | Batch Norm. | 0.00097
CIFAR-10 | CNN-2 | Dropout | 0.38
CIFAR-10 | CNN-2 | Augm. | 0.13
CIFAR-10 | CNN-2 | Dropout & Augm. | 0.44
CIFAR-10 | CNN-2 | Batch Norm. | 1.7

###  4.4 Unique feature memorisation for high/low self-influence examples

In the previous experiments, we selected canaries based on self-influence. To
examine the relationship between memorisation and self-influence further we
measure Pearson’s correlation coefficient.

Figure 4 shows the memorisation scores and self-influence scores for the
top-15 and bottom-15 canaries by self-influence score on MNIST for the MLP.
The correlation coefficient is 0.43, indicating a weak correlation between
memorisation score and self-influence. However, we did not find a strong
correlation for F-MNIST and CIFAR-10 using CNN-1. Instead, the results showed
that there was a high degree of unique feature memorisation in the low self-
influence examples. We comment further on these results in the discussion
section.

Figure 4: Memorisation scores and self-influence scores for canaries fitted by
MLP on MNIST.

###  4.5 Discussion

Our main finding is that commonly used neural networks architectures memorise
unique features when the unique feature occurs once in training data for
several scenarios.

We have shown that in the MNIST dataset a correlation exists between the self-
influence score (measured using TracIn) and the memorisation score. Contrary
to our expectations, we did not find a significant correlation in our
experiments for Fashion-MNIST and CIFAR-10. We suspect that one reason is the
influence of short-cut learning. For example, under the Principle of Least
Effort, the network may find it easier to fit the unique feature, than to
learn features for training examples from Fashion-MNIST or CIFAR-10 (Geirhos
et al.,, 2020). For MNIST the task of learning is significantly easier and the
unique feature is only memorised on very high influence examples. Another
possibility is that TracIn does not trace examples that are necessarily
memorable in datasets with more complex features, and hence the unique
features are not memorised either.

##  5 Conclusion

In this work, we present a memorisation score to measure the memorisation of
unique features in imaging datasets by neural network image classification
models. We focus on the case where the unique feature appears on a single
image in the training data, and where we have access only to the model’s
inputs and outputs, the unique feature, and no access to the training data
except for its domain.

We show that unique feature memorisation does occur in this setting, and is
not eliminated by typical explicit and implicit regularisation strategies,
dropout, data augmentation and batch normalisation. We derive these results
for benchmark datasets and a range of neural network architectures.

The results, even in standard benchmark datasets, suggest that neural networks
pose a privacy risk to unique sensitive information in imaging datasets even
if the information occurs once. The information does not have to be rare in
the wild. In the context of a healthcare application, the information could be
a patient name that was not removed by an image pre-processor.

##  6 Acknowledgements

This work is supported by iCAIRD, which is funded by Innovate UK on behalf of
UK Research and Innovation (UKRI) [project number 104690]. S.A. Tsaftaris
acknowledges also support by a Canon Medical / Royal Academy of Engineering
Research Chair under Grant RCSRF1819\8\25. This work was partially supported
by the Alan Turing Institute under EPSRC grant EP/N510129/1.

## References

  * Achille and Soatto, (2018) Achille, A. and Soatto, S. (2018).  Emergence of invariance and disentanglement in deep representations.  J. Mach. Learn. Res., 19(1):1947–1980.
  * Arplt et al., (2017) Arplt, D., Jastrzebskl, S., Bailas, N., Krueger, D., Bengio, E., Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A., Benglo, Y., and Lacoste-Julien, S. (2017).  A closer look at memorization in deep networks.  34th International Conference on Machine Learning, ICML 2017, 1:350–359.
  * Ateniese et al., (2015) Ateniese, G., Mancini, L. V., Spognardi, A., Villani, A., Vitali, D., and Felici, G. (2015).  Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers.  Int. J. Secur. Netw., 10(3):137–150.
  * Atienza, (2022) Atienza, R. (2022).  Advanced-Deep-Learning-with-Keras.
  * Baldock et al., (2021) Baldock, R. J. N., Maennel, H., and Neyshabur, B. (2021).  Deep Learning Through the Lens of Example Difficulty.
  * Bar et al., (2015) Bar, Y., Diamant, I., Wolf, L., Lieberman, S., Konen, E., and Greenspan, H. (2015).  Chest pathology detection using deep learning with non-medical training.  In 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI), pages 294–297.
  * (7) Carlini, N., Erlingsson, U., and Papernot, N. (2019a).  Prototypical Examples in Deep Learning: Metrics, Characteristics, and Utility.
  * (8) Carlini, N., Liu, C., Erlingsson, U., Kos, J., and Song, D. (2019b).  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In Proceedings of the 28th USENIX Conference on Security Symposium, SEC’19, page 267–284, USA.
  * Carlini et al., (2021) Carlini, N., Tramer, F., Wallace, E., Jagielski, M., Herbert-Voss, A., Lee, K., Roberts, A., Brown, T., Song, D., Erlingsson, U., et al. (2021).  Extracting training data from large language models.  In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650.
  * Choquette-Choo et al., (2021) Choquette-Choo, C. A., Tramer, F., Carlini, N., and Papernot, N. (2021).  Label-only membership inference attacks.  In International Conference on Machine Learning, pages 1964–1974. PMLR.
  * Feldman, (2020) Feldman, V. (2020).  Does learning require memorization? a short tale about a long tail.  In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954–959.
  * Feldman and Zhang, (2020) Feldman, V. and Zhang, C. (2020).  What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 2881–2891.
  * Ganju et al., (2018) Ganju, K., Wang, Q., Yang, W., Gunter, C. A., and Borisov, N. (2018).  Property Inference Attacks on Fully Connected Neural Networks Using Permutation Invariant Representations.  In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS ’18, pages 619–633.
  * Garima et al., (2020) Garima, Liu, F., Kale, S., and Sundararajan, M. (2020).  Estimating training data influence by tracing gradient descent.  Advances in Neural Information Processing Systems, 2020-Decem.
  * Garima et al., (2022) Garima, Liu, F., Kale, S., and Sundararajan, M. (2022).  TrackIn FAQ.
  * Geirhos et al., (2020) Geirhos, R., Jacobsen, J. H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. (2020).  Shortcut learning in deep neural networks.  Nature Machine Intelligence, 2(11):665–673.
  * Ghorbani and Zou, (2019) Ghorbani, A. and Zou, J. (2019).  Data shapley: Equitable valuation of data for machine learning.  36th International Conference on Machine Learning, ICML 2019, 2019-June:4053–4065.
  * Golatkar et al., (2020) Golatkar, A., Achille, A., and Soatto, S. (2020).  Eternal sunshine of the spotless net: Selective forgetting in deep networks.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 9301–9309.
  * Guo et al., (2020) Guo, H., Rajani, N. F., Hase, P., Bansal, M., and Xiong, C. (2020).  FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debugging.
  * Harutyunyan et al., (2021) Harutyunyan, H., Achille, A., Paolini, G., Majumder, O., Ravichandran, A., Bhotika, R., and Soatto, S. (2021).  Estimating informativeness of samples with smooth unique information.  In International Conference on Learning Representations.
  * He et al., (2016) He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-Decem:770–778.
  * Huang et al., (2017) Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q. (2017).  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708.
  * Huang et al., (2018) Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q. (2018).  Densely connected convolutional networks.
  * Idrissi et al., (2021) Idrissi, B. Y., Arjovsky, M., Pezeshki, M., and Lopez-Paz, D. (2021).  Simple data balancing achieves competitive worst-group-accuracy.  140:1–14.
  * Jegorova et al., (2021) Jegorova, M., Kaul, C., Mayor, C., O’Neil, A. Q., Weir, A., Murray-Smith, R., and Tsaftaris, S. A. (2021).  Survey: Leakage and privacy at inference time.
  * Jiang et al., (2021) Jiang, Z., Zhang, C., Talwar, K., and Mozer, M. C. (2021).  Characterizing structural regularities of labeled data in overparameterized models.  In Meila, M. and Zhang, T., editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 5034–5044. PMLR.
  * Katharopoulos and Fleuret, (2018) Katharopoulos, A. and Fleuret, F. (2018).  Not All Samples Are Created Equal: Deep Learning with Importance Sampling.  In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2525–2534. PMLR.
  * (28) Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. (2018a).  Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav).  In International conference on machine learning, pages 2668–2677. PMLR.
  * (29) Kim, Y., Kim, M., and Kim, G. (2018b).  Memorization precedes generation: Learning unsupervised GANs with memory networks.  In International Conference on Learning Representations.
  * Kingma and Ba, (2015) Kingma, D. P. and Ba, J. (2015).  Adam: A method for stochastic optimization.  In ICLR (Poster).
  * Koh and Liang, (2017) Koh, P. W. and Liang, P. (2017).  Understanding black-box predictions via influence functions.  In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1885–1894.
  * Krizhevsky et al., (2009) Krizhevsky, A., Hinton, G., et al. (2009).  Learning multiple layers of features from tiny images.
  * Krizhevsky et al., (2012) Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012).  Imagenet classification with deep convolutional neural networks.  In Pereira, F., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems, volume 25. Curran Associates, Inc.
  * Krueger et al., (2019) Krueger, D., Ballas, N., Jastrzebski, S., Arpit, D., Kanwal, M. S., Maharaj, T., Bengio, E., Fischer, A., and Courville, A. (2019).  Deep nets don’t learn via memorization.  5th International Conference on Learning Representations, ICLR 2017 - Workshop Track Proceedings, pages 1–4.
  * LeCun et al., (1989) LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., and Jackel, L. D. (1989).  Backpropagation applied to handwritten zip code recognition.  Neural Computation, 1(4):541–551.
  * Lecun et al., (1998) Lecun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998).  Gradient-based learning applied to document recognition.  Proceedings of the IEEE, 86(11):2278–2324.
  * (37) Liu, E. Z., Haghgoo, B., Chen, A. S., Raghunathan, A., Koh, P. W., Sagawa, S., Liang, P., and Finn, C. (2021a).  Just train twice: Improving group robustness without training group information.  In Proceedings of the 38th International Conference on Machine Learning, volume 139, pages 6781–6792. PMLR.
  * (38) Liu, F., Lin, T., and Jaggi, M. (2021b).  Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimation.  pages 1–14.
  * Liu et al., (2020) Liu, J., Sun, Y., Han, C., Dou, Z., and Li, W. (2020).  Deep representation learning on long-tailed data: A learnable embedding augmentation perspective.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  * Liu and Tsaftaris, (2020) Liu, X. and Tsaftaris, S. A. (2020).  Have you forgotten? A method to assess if machine learning models have forgotten data.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 95–105. Springer.
  * Sablayrolles et al., (2018) Sablayrolles, A., Douze, M., Schmid, C., and Jégou, H. (2018).  Deja Vu: an empirical evaluation of the memorization properties of ConvNets.  (2015):1–22.
  * Salem et al., (2018) Salem, A., Zhang, Y., Humbert, M., Berrang, P., Fritz, M., and Backes, M. (2018).  ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models.
  * Shao et al., (2020) Shao, L., Song, Y., and Ermon, S. (2020).  Understanding classifier mistakes with generative models.
  * Shokri et al., (2017) Shokri, R., Stronati, M., Song, C., and Shmatikov, V. (2017).  Membership inference attacks against machine learning models.  In 2017 IEEE Symposium on Security and Privacy (SP), pages 3–18. IEEE.
  * Simonyan and Zisserman, (2015) Simonyan, K. and Zisserman, A. (2015).  Very deep convolutional networks for large-scale image recognition.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  * Tishby and Zaslavsky, (2015) Tishby, N. and Zaslavsky, N. (2015).  Deep learning and the information bottleneck principle.  In 2015 IEEE Information Theory Workshop (ITW), pages 1–5.
  * Toneva et al., (2019) Toneva, M., Trischler, A., Sordoni, A., Bengio, Y., Des Combes, R. T., and Gordon, G. J. (2019).  An empirical study of example forgetting during deep neural network learning.  7th International Conference on Learning Representations, ICLR 2019, pages 1–19.
  * Wikipedia, (2022) Wikipedia (2022).  Chest radiograph — Wikipedia, the free encyclopedia.  https://en.wikipedia.org/wiki/Chest_radiograph#/media/File:Normal_posteroanterior_(PA)_chest_radiograph_(X-ray).jpg.  [Online; accessed 17-January-2022].
  * Xiao et al., (2017) Xiao, H., Rasul, K., and Vollgraf, R. (2017).  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.
  * Zech et al., (2018) Zech, J. R., Badgeley, M. A., Liu, M., Costa, A. B., Titano, J. J., and Oermann, E. K. (2018).  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study.  PLoS Medicine, 15(11):1–17.
  * (51) Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2021a).  Understanding deep learning (still) requires rethinking generalization.  Commun. ACM, 64(3):107–115.
  * (52) Zhang, Y., Kang, B., Hooi, B., Yan, S., and Feng, J. (2021b).  Deep long-tailed learning: A survey.

##  Appendix A Network Architectures

We evaluate our memorisation score using several common architectural styles
of neural networks. The first, CNN-1 is trained on MNIST and Fashion-MNIST
datasets. It is comprised of: Conv2D(32,3,3) $\rightarrow$ ReLU $\rightarrow$
Conv2D(64,3,3) $\rightarrow$ MaxPool2d(2,2) $\rightarrow$ ReLU $\rightarrow$
Dense(128) $\rightarrow$ ReLU $\rightarrow$ Dense(128) $\rightarrow$ ReLU
$\rightarrow$ Dense(#classes) $\rightarrow$ Softmax.

We train CNN-1 with a learning rate of $3\text{\times}{10}^{-4}$ and a batch
size of 128.

CNN-2 is small VGG-style network trained on CIFAR-10. It is comprised of:
Conv2D(32,3,3) $\rightarrow$ ReLU $\rightarrow$ Conv2D(32,3,3) $\rightarrow$
ReLU $\rightarrow$ MaxPool2d(2,2) $\rightarrow$ Conv2D(64,3,3) $\rightarrow$
ReLU $\rightarrow$ Conv2D(64,3,3) $\rightarrow$ ReLU $\rightarrow$
MaxPool2d(2,2) $\rightarrow$ Dense(1024) $\rightarrow$ ReLU $\rightarrow$
Dense(#classes).

We train CNN-2 with a learning rate of $3\text{\times}{10}^{-4}$ and a batch
size of 512.

DenseNet is DenseNet trained on CIFAR-10 Huang et al., (2018). The network has
100 layers, a growth factor of 12, and three dense blocks. We use an existing
implementation and train using the same parameters given in Atienza, (2022).

##  Appendix B Experimental codes

The code for the experiments is forthcoming
https://github.com/jasminium/unintended-memorisation.


# Detecting Cattle and Elk in the Wild from Space

Caleb Robinson<EMAIL_ADDRESS>Microsoft, AI for Good Research
LabUSA ,  Anthony Ortiz<EMAIL_ADDRESS>Microsoft, AI for Good
Research LabUSA ,  Lacey Hughey  Smithsonian Conservation Biology InstituteUSA
,  Jared A. Stabach  Smithsonian Conservation Biology InstituteUSA and  Juan
M. Lavista Ferres  Microsoft, AI for Good Research LabUSA

(2021)

###### Abstract.

Localizing and counting large ungulates – hoofed mammals like cows and elk –
in very high-resolution satellite imagery is an important task for supporting
ecological studies. Prior work has shown that this is feasible with deep
learning based methods and sub-meter multi-spectral satellite imagery. We
extend this line of work by proposing a baseline method, CowNet, that
simultaneously estimates the number of animals in an image (counts), as well
as predicts their location at a pixel level (localizes). We also propose an
methodology for evaluating such models on counting and localization tasks
across large scenes that takes the uncertainty of noisy labels and the
information needed by stakeholders in ecological monitoring tasks into
account. Finally, we benchmark our baseline method with state of the art
vision methods for counting objects in scenes. We specifically test the
temporal generalization of the resulting models over a large landscape in
Point Reyes Seashore, CA. We find that the LC-FCN model performs the best and
achieves an average precision between $0.56$ and $0.61$ and an average recall
between $0.78$ and $0.92$ over three held out test scenes.

deep neural networks, cattle, satellite imagery

††copyright: none††journalyear: 2021††conference: Fragile Earth ’21: Fragile
Earth: Accelerating Progress towards Equitable Sustainability; August 14–18,
2021; Virtual††ccs: Applied computing††ccs: Computing methodologies Supervised
learning††ccs: Computing methodologies Neural networks††ccs: Social and
professional topics Sustainability

##  1\. Introduction

Over the past decade, very high-resolution (VHR) remotely sensed imagery has
become increasingly available to conservation scientists for research
purposes. Processing these data manually, however, is a labor intensive task
that requires specialized skills and equipment. Further, the effort required
to process VHR imagery for landscape-scale ecological analyses is prohibitive.
For example, it typically takes manual annotators weeks or months to complete
labeling tasks required by biological assessments conducted at ecologically
meaningful spatial scales. However, the ability to accurately assess
environmental change across large, remote landscapes is essential to
conservation success and has many applications in scientific research, policy
making, and protected area management.

VHR imagery holds particular promise as a tool for conducting wildlife
population surveys across a range of species and environmental contexts (e.g.,
(LaRue et al., 2017; Wang et al., 2019)). Recent work, for example,
demonstrates that VHR data can be used to study fundamental aspects of
behavioral ecology, such as species interactions in ecologically sensitive
landscapes (Hughey et al., 2021) and emergent properties of collective motion
in animal groups (Hughey et al., 2018). The early promise of such studies
underscores the need for efficient, scalable techniques to rapidly extract
large amounts of information from VHR imagery.

Figure 1. Examples of VHR (0.3m/px) panchromatic imagery, point labels, and
CowNet model localization predictions for two areas in a held out test scene.

Automated approaches can bridge the gap between the amount of high-resolution
imagery available for performing ecological analyses and the amount of manual
labor required to interpret the imagery. For instance, deep learning
approaches for counting and localizing cattle in high-resolution imagery (at
spatial resolutions of $<0.5\text{ m}/\text{pixel}$) have recently been shown
to be possible (Laradji et al., 2020b). Other work using object detection
models has been shown to detect and count African elephants, with a key goal
of generalizing models across new landscapes where training data do not exist
(Duporge et al., 2020). Fully automated approaches also exist, with
researchers surveying seals on pack ice at a rate ten times faster than
experienced human observers (Gonçalves et al., 2020).

At least two challenges exist in using automated methods for supporting
ecological analyses with high-resolution remotely sensed imagery. These
include (1) designing or adapting methods to work in new settings, either
spatially or temporally, and (2) measuring the effectiveness of results with
reliable validation data. For example, Hughey et al. measure the spatial
overlap between cattle and reintroduced elk populations partially supported by
animal location data derived from high-resolution satellite imagery over
multiple points in time (Hughey et al., 2021). For an automated approach to
adequately support this ecological task, models must localize cattle and elk
in large amounts of high-resolution imagery with high recall, and generalize
across imagery taken at the same location for different points in time.

In this work, we aim to explore both of these challenges within the context of
the aforementioned example – counting and localizing cattle and elk in a time
series of VHR imagery over Point Reyes, CA in the USA. First, we propose a
baseline deep learning-based approach for counting and localizing cows and
elk. This approach treats counting and localizing animals as two separate
tasks, each of which share the same spatial representation. Second, we test
the performance of automated methods on this problem in several novel ways: we
explicitly account for the spatial noise in our labeled data to better
estimate model performance and we propose metrics that can be applied over
large scenes (as opposed to fixed patch based evaluations). We explicitly test
the temporal generalization of our baseline method and several other common
methods from computer vision literature.

##  2\. Related Work

Different approaches have been proposed for localizing and counting objects
using computer vision techniques. The problem of counting objects have been
approached in two different directions. The first approach is detection-based
counting, which requires object detection or segmentation before performing
the counting task (Girshick et al., 2014; Laradji et al., 2018, 2020a). The
second approach is density estimation-based and does not requires detection or
segmentation to predict for the count (Xie et al., 2018; Li et al., 2018).

The most related work to ours is by Laradji et al. (Laradji et al., 2020b)
which compares CSRNet (Li et al., 2018) and LC-FCN (Laradji et al., 2018)
modeling approaches on the task of counting and localizing the number of
cattle in high resolution satellite imagery using a dataset of 12,252 labelled
patches from Maxar satellites. The authors compare models with the mean
absolute percentage error (MAPE) and grid average mean absolute percentage
error (GAMPE) metrics on different groups of held out patches that are
partitioned by density. We significantly extend this evaluation methodology
considering the questions of: “How to use noisy point labels?”, “How to
evaluate model performance at a scene level?”, and “How to evaluate counting
and localization performance separately?”. This is important for using such
models in ecological applications as we discuss later.

##  3\. Problem formulation

We assume that we are given a time-series of $t$ VHR satellite image scenes
covering roughly the same spatial location,
$\left[\mathbf{X}^{1},\ldots,\mathbf{X}^{t}\right]$, as well as noisy point
labels for each scene that indicate the presence of a cow/elk. These point
labels can be represented as per scene binary masks,
$\left[\mathbf{Y}^{1},\ldots,\mathbf{Y}^{t}\right]$ where
$\mathbf{Y}^{i}_{xy}$ is $1$ if there is a cow/elk within $d$ meters from the
$y^{\text{th}}$ row and $x^{\text{th}}$ column of scene $i$, but is $0$
otherwise. The spatial dimensions of $\mathbf{Y}^{i}$ are the same as
$\mathbf{X}^{i}$. Here, the value $d$ represents the extent of the spatial
noise within the dataset and will often be unknown for datasets that have been
curated without machine learning applications in mind111For example, it is
unlikely that an ecologist will need pixel perfect point labels of the
locations of cow and elk over large scenes and will tolerate errors of several
meters when creating such datasets.. Importantly, we do not assume that the
time-series of scenes are precisely co-registered. For example, the scenes can
be imaged from large off-nadir angles and have errors in their
georegistration. We say that the scenes in the time series cover roughly the
same area, meaning the the union of their spatial extents, $E^{i}$, is not
empty, i.e. $\cup_{i=1}^{t}E^{i}\neq\emptyset$. Finally, we expect that these
scenes are much larger than traditional image inputs in computer vision
problems. For example, scenes cover many square kilometers at sub-meter
spatial resolution (i.e. each will have on the order of one hundred million
pixels).

We would like to train a model that can count and localize cows/elk given new
imagery of the same location, i.e. in the form
$f(\mathbf{X}^{j};\mathbf{\theta})=\mathbf{\hat{Y}}^{j}$ where
$\mathbf{\theta}$ are the parameters of the model and $j>t$. Here,
$\mathbf{\hat{Y}}^{j}$ should be greater than $0$ where a cow or elk exists in
$\mathbf{X}^{j}$ and $0$ otherwise. We fit the parameters $\theta$ following
empirical risk minimization, for some loss function
$J(f(\operatorname{\mathbf{X}};\theta),\operatorname{\mathbf{Y}})$:

$\min_{\theta}\frac{1}{t}\sum_{i=1}^{t}J(f(\operatorname{\mathbf{X}}^{t};\theta),\operatorname{\mathbf{Y}}^{t})$

In Section 4.1 we discuss different architectural choices for
$f(\cdot;\theta)$, as well as different loss functions for encoding the
problem. Finally, we discuss metrics for measuring model performance in
counting and localizing given a trained model and held out imagery/labels in
Section 4.2.

##  4\. Methods

###  4.1. Modeling approaches

Following the methodology from density based crowd counting methods (Lempitsky
and Zisserman, 2010; Boominathan et al., 2016) we convert the binary point
label masks, $\mathbf{Y}^{i}$, to either density masks, $\mathbf{Z}^{i}$, or
segmentation masks, $\mathbf{S}^{i}$, to use in training of density based and
localization based approaches respectively.

Specifically, to generate training labels for our density based approaches
(CSRNet, FCRN, FCRN-LCN, and CowNet – see below) we initialize a $7\times 7$
Gaussian kernel, $K$, and convolve it with the given point label to produce a
density mask, $\mathbf{Y}^{i}*K=\mathbf{Z}^{i}$. Here, we expect the sum of
the predicted density will equal the number of predicted animals over
different sized image regions. Similarly, for our segmentation approaches
(UNet and CowNet – see below) we apply a maximum filter of size $7\times 7$ to
$\mathbf{Y}^{i}$ and obtain a noisy mask of the objects, $\mathbf{S}^{i}$.

CowNet:

is our proposed baseline approach. It is a UNet architecture, with two task-
specific heads attached to the final spatial representation. The density head
consists of a 1x1 convolution layer that predicts the per pixel density, while
the segmentation head (also a 1x1 convolution layer) predicts the presence or
absence of an animal. We train this architecture with a three term loss
function designed to jointly optimize for: accurate segmentation, count
density estimation, and global count. We train the segmentation head with
labels $\mathbf{S}^{i}$, and the density head with labels $\mathbf{Z}^{i}$.

CSRNet (Li et al., 2018):

is a density estimation-based neural network architecture for object counting.
It uses a two VGG-like (Simonyan and Zisserman, 2014) convolutional neural
networks (CNN) as a feature extractor and density estimator, respectively.
Notably, the density estimator sub-network uses dilated kernel convolutions to
deliver larger receptive fields and to replace all pooling layers. This
approach is trained with a mean square error loss with labels
$\mathbf{Z}^{i}$.

LC-FCN (Laradji et al., 2018):

is a state of the art method for training segmentation based counting models
(with previous applications to counting penguins, fish, cows and other objects
(Laradji et al., 2020b, c, 2018)). Notably, the loss does not reason about the
size or shape of the objects and instead forces the model to predict a blob of
positive predictions for each object instance using point-level annotations.
The loss consist of four terms: an image-level and a point-level loss, a split
level loss to enforce the prediction of a single blob per object instance, and
a false positive loss. We use the same architecture as (Laradji et al., 2018),
a FCN-18 (Long et al., 2015) with a ResNet18 backbone. The LC-FCN models are
trained using $\mathbf{Y}^{i}$.

FCRN (Xie et al., 2018) / FCRN-LCN:

is an architecture for detecting and counting cells in microscopy by
performing density estimation. The FCRN-LCN variant includes local context
normalization (Ortiz et al., 2020) instead of batch normalization layers. The
network predicts object density at a pixel level and is trained with a mean
squared error loss using $\mathbf{Z}^{i}$ labels.

UNet (Ronneberger et al., 2015):

is a popular encoder-decoder network architecture for performing general
semantic segmentation tasks. UNets and various variants based on the UNet
architecture have previously achieved state of the art results in a wide
variety of application domains (Falk et al., 2019). This approach is trained
with a pixel-wise weighted cross entropy loss using $\mathbf{S}^{i}$ labels.

###  4.2. Measuring counting and localization performance with noisy point
labels

Considering that we are interested in counting and localizing cows and elk in
VHR satellite imagery scenes, we need to measure performance with respect to
the domain and purpose of the models – large VHR satellite image scenes, and
for ecological analyses. As

####  4.2.1. Counting

With existing counting metrics such as mean absolute error (MAE) or mean
absolute percentage error (MAPE), the difference between a predicted count,
$\hat{y}$, and the ground truth count, $y$, is calculated per image and
averaged over all images in a dataset. In our problem setting, we will have
few large “images” (i.e. scenes), therefore calculating MAE and MAPE will not
be meaningful. The grid average mean absolute error (GAME) metric was
introduced in (Guerrero-Gómez-Olmedo et al., 2015) and extended to MAPE in
(Laradji et al., 2020b) to measure how well counting algorithms also perform
at localizing objects. GAME computes the average of the sum of the absolute
errors calculated across non-overlapping windows from a dataset of $N$ images:

(1)
$\text{GAME}(L)=\frac{1}{N}\cdot\sum_{n=1}^{N}\left(\sum_{l=1}^{4^{L}}|\hat{y}_{n}^{l}-y_{n}^{l}|\right)$

where $L$ is a parameter that controls the number of non-overlapping windows
that each image is divided into, and $\hat{y}_{i}^{l}$ is the $l^{\text{th}}$
window from the $i^{\text{th}}$ image in the dataset. When $L=0$, GAME is
equivalent to MAE, while when $L>0$, GAME will measure how a model is able to
localize an object with increasing fidelity.

We adapt this idea to work within a single scene of VHR satellite imagery. We
define a different version of “gridded” MAE, whereby a scene is split into a
grid of $r\times r$ cells, $\mathcal{R}$, and the MAE is calculated over each
cell:

(2)
$\text{GMAE}(r)=\frac{1}{|\mathcal{R}|}\sum_{i\in\mathcal{R}}|\hat{y}_{i}-y_{i}|.$

As satellite imagery has a fixed spatial resolution, $r$ is a physical
distance. GMAE roughly measures how well a model is able to localize objects
within a distance $r$ of their correct location. For example, if GMAE is high
for small values of $r$, but lower for larger values of $r$, then the model is
able to correctly identify how many objects there are in a scene, but is not
able to correctly determine where each object is. This can be the case with
“glance” type object counting methods (Chattopadhyay et al., 2017) that are
trained to directly regress the number of objects in a patch of input imagery,
without necessarily localizing each object. To determine the per cell count
for density based models, we sum the predicted density within each cell, while
for segmentation based models, we sum the number of centroids of connected
components of predicted positive pixels that fall within a cell. We can
further normalize $\text{GMAE}(r)$ to GMAE per km2, assuming $r$ is in units
of meters, by multiplying by $\frac{1000000}{r^{2}}$222Multiplying by
$\frac{1}{\text{m}^{2}\text{ per
cell}}\cdot\frac{1000000\text{m}^{2}}{1\text{km}^{2}}$. This is necessary to
compare GMAE across different values of $r$ for a fixed area.

Using the same grid setup as GMAE, we also compute the coefficient of
determination (R2) between model predicted counts and ground truth counts over
all cells:

(3)
$\text{R}^{2}=1-\frac{\sum_{i\in\mathcal{R}}(y_{i}-\hat{y}_{i})^{2}}{\sum_{i\in\mathcal{R}}(y_{i}-\bar{y})^{2}}$

where $\bar{y}=\frac{1}{|\mathcal{R}|}\sum_{i\in\mathcal{R}}y_{i}$. Note that
this value can be less than 0 when a model is making predictions that are
worse than a hypothetical model which simply always returns the dataset mean.

####  4.2.2. Localizing

In (Laradji et al., 2018) the authors point out that GAME might not be an
accurate localization metric and compute classification metrics (specifically,
F-Score) by matching predicted blobs to the point-level annotations. We extend
this idea to our problem setting where point labels can be a distance $d$ away
from the objects that they are annotating, and the objects themselves can be
of a size less than $d^{2}$. Here, a reasonable model may (correctly) identify
a group of pixels, that does not include the ground truth point label, as a
positive class. Thus, we propose two matching algorithms for pairing predicted
blobs with labeled points that takes this label uncertainty into account: an
optimistic matching algorithm and a conservative matching algorithm.

In both algorithm we first group predicted foreground pixels in
$\mathbf{\hat{Y}}^{j}$ into connected components (i.e. “blobs” in other object
counting literature). In the optimistic algorithm we buffer each labeled point
by a “cutoff distance” of $d$ and record the intersections between the
buffered points and all connected components. The number of true positives is
the number of labeled points that intersect with any predicted connected
component, the number of false positives is the number of predicted connected
components that do not intersect with any buffered points, and the number of
false negatives is the number of buffered points that do not intersect with
any predicted connected component. This is an “optimistic” accounting of a
model’s performance because a predicted connected component can intersect with
more than one buffered label. In areas with dense labels (i.e. many cows/elk
in a small space) this may be a desirable property as individual objects can
be hard to separate. On the other hand, it will not penalize a model that
over-segments the imagery in non-desirable cases. For example, a model that
predicts a positive foreground class for an entire scene would not be
penalized. This metric may be appropriate for ecological applications as
finding cows/elk in large amounts of satellite imagery is similar to finding
needles in a haystack. If any predicted connected components are near to
labeled points then the model has correctly localized the objects of interest,
i.e. located the needles.

In the conservative algorithm we enforce the property that each predicted
connected component can only count towards a single true positive. Here, we
create a bipartite graph representation where each predicted connected
component is represented as a node in the set $\mathbf{U}$ and each label is
represented as a node in the set $\mathbf{V}$. We add edges between nodes
$(u,v)$ if they are within the $d$ cutoff distance from each other. The number
of true positives is now the size of the maximum cardinality matching on this
graph, the number of false positives is the number of unmatched nodes from
$\mathbf{U}$ and the number of false negatives is the number of unmatched
nodes from $\mathbf{V}$. The maximum cardinality matching ensures that each
predicted connected component is paired with a single labeled point. This
accounting of a model’s performance will correctly handle the degenerate case
where a model predicts a positive foreground class for the entire input. Here,
the “optimistic” algorithm will report perfect performance, while this
algorithm will report a single true positive, no false positives, and
$\text{``number of labels''}-1$ false negatives.

##  5\. Data

We use a dataset of 11 VHR panchromatic scenes from Maxar’s satellite
catalogue covering Point Reyes National Seashore, CA from 2013 to 2018. The
spatial resolution of the scenes varies from $0.3\text{ m}/\text{pixel}$ to
$0.5\text{ m}/\text{pixel}$, while off nadir position ranges from 11∘ to 34∘.
The time of day and day of the year that each scene was imaged also differ
between scenes, impacting lighting conditions and producing different shadow
conditions throughout each image. We re-sample applicable scenes to a
$0.3\text{ m}/\text{pixel}$ resolution with bilinear interpolation. Multiple
experts exhaustively annotated the positions of cattle and elk in each scenes,
resulting in 10,529 labeled points. See Figure 1 to see a visualization of the
panchromatic layer and labels from two areas of one of the test scenes.

| Localization Metrics | Counting Metrics
---|---|---
Method | Precision | Recall | GMAE | R2
LC-FCN | (0.563, 0.606) | (0.779, 0.923) | 0.134 | 0.766
CowNet | (0.465, 0.499) | (0.787, 0.890) | 0.241 | 0.557
FCRN-LCN | n/a | n/a | 0.190 | 0.212
FCRN | n/a | n/a | 0.268 | 0.038
CSRNet | n/a | n/a | 0.338 | 0.005
UNet | (0.387, 0.427) | (0.605, 0.692) | 0.304 | ¡ 0
Table 1. Model performance averaged over three held out test scenes. The
localization metrics are calculated with $d=4$ meters and shown as
(conservative estimate, optimistic estimate). The counting metrics are
calculated over an $r=100m$ grid covering the land area in each scene. The R2
(i.e. the coefficient of determination) values can be less than $0$ – this
indicates worse performance than a model which simply predicts the dataset
average ground truth count for each grid cell. Figure 2. Counting performance
of different density based models on one held out scene. We tile the study
area with different cell sizes and measure the GMAE/km2 and R2 between the
predicted count of cows per grid cell and ground truth number of cows per cell
for each tiling. The black horizontal line on the MAE/km2 plots shows the
ground truth density of cows ($\sim 20/\text{km}^{2}$) recorded for the scene.
Figure 3. We show the sensitivity of our method for computing precision and
recall to the cutoff distance used with a set of model predictions over a held
out scene. The blue line shows the conservative method that enforces a one-to-
one matching between connected components of predicted pixels and labeled
points, while the orange line shows the optimistic method that pairs any
nearby predicted connected components with labeled points. We observe a sharp
jump in estimated performance as the cutoff distance increases from 1 meter to
3 meters as the labeled points are not always precisely aligned to animals in
the imagery, with diminishing returns after a cutoff of 4 meters.

##  6\. Experiments and Results

To test the temporal generalization performance of the models, we select the
earliest seven scenes from our dataset, spanning from February 2013 to
December 2014, for training, a single scene from September 2016 for
validation, and three scenes, spanning from March 2015 to April 2017, for
testing. We train all models on the train split, select top performing models
based on validation loss performance, and evaluate on the test split.

Table 1 shows the average performance of all models over the test scenes.
Here, the counting metrics over each scene are calculated at a $r=100$ meter
resolution grid and the localization metrics are calculated with a cutoff
distance of $d=4$. We evaluate both segmentation and density based models as
described in Section 4.2.1. The LC-FCN model performs the best across all
metrics, for both counting and localization. We find that the LC-FCN, CowNet,
and FCRN-LCN models are consistently making reasonable predictions across all
scenes, while the performance of the other models is mixed. For example, the
U-Net model suffers from many false positives/low precision, which reduces its
counting performance metrics, and the CSRNet and FCRN models do not perform
well in terms of counting metrics (i.e. estimating density) at a $100$ meter
resolution. The LC-FCN and U-Net models have a large range between their
conservative and optimistic recall estimates as they tend to predict large
connected components of pixels for more dense groups of cattle which will only
count as a single true positive in the conservative estimate.

We choose a cutoff distance of $d=4$ to compute the localization metrics based
on the sensitivity analysis in Figure 3. Here, we compute precision and recall
of the CowNet model for different cutoff distances. We observe that there is a
large jump in performance between $d=1$ and $d=2$ compared to subsequent
values. This shows the effect of the label noise on the results (a significant
number of labels are $>1$ meter away from the objects that they are
annotating). As $d$ increases the estimated performance of the model increases
as more predicted connected components are matched with labeled points that
are farther away. We choose $d=4$ as the cutoff to report values in Table 1 as
there is another jump in the “optimistic” matching performance between $d=4$
and $d=5$, and diminishing returns after. We believe this is a reasonable
choice that reflects true model performance.

Figure 2 shows the counting metrics as a function of grid cell size ($r$) for
the density based models over one test scene. We observe that the performance
of all the counting models improves with increasing cell size. With larger
cell sizes, false positives and false negative errors can cancel out. At the
lowest resolution grid cell size (1024 meters) the CowNet model achieves an
0.82 R2. Notably, the R2 if the CSRNet model is only positive after the grid
cell size becomes larger than the training input size (i.e. when grid cells
are larger than 256 pixels). CSRNet estimates coarse density over its inputs,
therefore cannot localize at high resolutions.

##  7\. Conclusion

In this work we benchmark different localization and counting based deep
learning approaches for detecting cows and elk in very high-resolution
satellite imagery. We specifically measure the temporal generalization of the
models to test how they can perform in ecological monitoring settings when
trained in a particular area. We propose evaluation methods that are tailored
for this task, and find that an LC-FCN based models perform the best.

It is important for future work in this direction to a.) test the spatial
generalization of similar models; b.) compare the model estimated counts with
ground truth counts or aerial survey results in order to determine how such
models can fit into larger cattle monitoring efforts; c.) develop techniques
for better training and evaluating models in cases where there are dense
groups of animals; and d.) develop post-processing methods appropriate for
cleaning predictions made over VHR satellite imagery.

## References

  * (1)
  * Boominathan et al. (2016) Lokesh Boominathan, Srinivas SS Kruthiventi, and R Venkatesh Babu. 2016.  Crowdnet: A deep convolutional network for dense crowd counting. In _Proceedings of the 24th ACM international conference on Multimedia_. 640–644.
  * Chattopadhyay et al. (2017) Prithvijit Chattopadhyay, Ramakrishna Vedantam, Ramprasaath R Selvaraju, Dhruv Batra, and Devi Parikh. 2017.  Counting everyday objects in everyday scenes. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 1135–1144.
  * Duporge et al. (2020) Isla Duporge, Olga Isupova, Steven Reece, David W Macdonald, and Tiejun Wang. 2020\.  Using very-high-resolution satellite imagery and deep learning to detect and count African elephants in heterogeneous landscapes.  _Remote Sensing in Ecology and Conservation_ (2020).
  * Falk et al. (2019) Thorsten Falk, Dominic Mai, Robert Bensch, Özgün Çiçek, Ahmed Abdulkadir, Yassine Marrakchi, Anton Böhm, Jan Deubner, Zoe Jäckel, Katharina Seiwald, et al. 2019\.  U-Net: deep learning for cell counting, detection, and morphometry.  _Nature methods_ 16, 1 (2019), 67–70.
  * Girshick et al. (2014) Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. 2014.  Rich feature hierarchies for accurate object detection and semantic segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 580–587.
  * Gonçalves et al. (2020) Bento Collares Gonçalves, Bradley Spitzbart, and Heather J Lynch. 2020.  SealNet: A fully-automated pack-ice seal detection pipeline for sub-meter satellite imagery.  _Remote Sensing of Environment_ 239 (2020), 111617.
  * Guerrero-Gómez-Olmedo et al. (2015) Ricardo Guerrero-Gómez-Olmedo, Beatriz Torre-Jiménez, Roberto López-Sastre, Saturnino Maldonado-Bascón, and Daniel Onoro-Rubio. 2015.  Extremely overlapping vehicle counting. In _Iberian Conference on Pattern Recognition and Image Analysis_. Springer, 423–431.
  * Hughey et al. (2018) Lacey F Hughey, Andrew M Hein, Ariana Strandburg-Peshkin, and Frants H Jensen. 2018.  Challenges and solutions for studying collective animal behaviour in the wild.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ 373, 1746 (2018), 20170005.
  * Hughey et al. (2021) Lacey F Hughey, Kevin T Shoemaker, Kelley M Stewart, Douglas J McCauley, and J Hall Cushman. 2021\.  Effects of human-altered landscapes on a reintroduced ungulate: Patterns of habitat selection at the rangeland-wildland interface.  _Biological Conservation_ 257 (2021), 109086.
  * Laradji et al. (2020b) Issam Laradji, Pau Rodriguez, Freddie Kalaitzis, David Vazquez, Ross Young, Ed Davey, and Alexandre Lacoste. 2020b.  Counting Cows: Tracking Illegal Cattle Ranching From High-Resolution Satellite Imagery.  _arXiv preprint arXiv:2011.07369_ (2020).
  * Laradji et al. (2020c) Issam Laradji, Alzayat Saleh, Pau Rodriguez, Derek Nowrouzezahrai, Mostafa Rahimi Azghadi, and David Vazquez. 2020c.  Affinity lcfcn: Learning to segment fish with weak supervision.  _arXiv preprint arXiv:2011.03149_ (2020).
  * Laradji et al. (2020a) Issam H Laradji, Rafael Pardinas, Pau Rodriguez, and David Vazquez. 2020a.  Looc: Localize overlapping objects with count supervision. In _2020 IEEE International Conference on Image Processing (ICIP)_. IEEE, 2316–2320.
  * Laradji et al. (2018) Issam H Laradji, Negar Rostamzadeh, Pedro O Pinheiro, David Vazquez, and Mark Schmidt. 2018\.  Where are the blobs: Counting by localization with point supervision. In _Proceedings of the European Conference on Computer Vision (ECCV)_. 547–562.
  * LaRue et al. (2017) Michelle A LaRue, Seth Stapleton, and Morgan Anderson. 2017\.  Feasibility of using high-resolution satellite imagery to assess vertebrate wildlife populations.  _Conservation biology_ 31, 1 (2017), 213–220.
  * Lempitsky and Zisserman (2010) Victor Lempitsky and Andrew Zisserman. 2010.  Learning to count objects in images.  _Advances in neural information processing systems_ 23 (2010), 1324–1332.
  * Li et al. (2018) Yuhong Li, Xiaofan Zhang, and Deming Chen. 2018.  Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 1091–1100.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015\.  Fully convolutional networks for semantic segmentation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 3431–3440.
  * Ortiz et al. (2020) Anthony Ortiz, Caleb Robinson, Dan Morris, Olac Fuentes, Christopher Kiekintveld, Md Mahmudulla Hassan, and Nebojsa Jojic. 2020\.  Local context normalization: Revisiting local normalization. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11276–11285.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-net: Convolutional networks for biomedical image segmentation. In _International Conference on Medical image computing and computer-assisted intervention_. Springer, 234–241.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ (2014).
  * Wang et al. (2019) Dongliang Wang, Quanqin Shao, and Huanyin Yue. 2019.  Surveying wild animals from satellites, manned aircraft and unmanned aerial systems (UASs): A review.  _Remote Sensing_ 11, 11 (2019), 1308.
  * Xie et al. (2018) Weidi Xie, J Alison Noble, and Andrew Zisserman. 2018\.  Microscopy cell counting and detection with fully convolutional regression networks.  _Computer methods in biomechanics and biomedical engineering: Imaging & Visualization_ 6, 3 (2018), 283–292.