##  1 Framework and main results

In this section we formally state our main results and the relevant
definitions. We parameterize machine learning models by real-valued vectors
$\theta\in\Theta$, where the classification space $\Theta$ is a closed, convex
subset of $\mathbb{R}^{d}$. In round $t$, the institution chooses a classifier
$\theta_{t}$. The adversary responds with a distribution $d_{t}$ over
instances $Z=(X,Y)$ of feature-label pairs. Let $\Delta(\mc{Z})$ denote the
space of distributions. The adversary is restricted to pick its distribution
according to a deterministic transition map:

$\Tr:\Theta\times\Delta(\mc{Z})\rightarrow\Delta(\mc{Z}),$

so that $\Tr(d_{t-1},\theta_{t})=d_{t}$. We assume that an initial
distribution $d_{0}$ is publicly known. From this online game view, the work
of Perdomo et al. assumes a myopic adversary that _only_ sees the
institution’s most recent play. We remove that condition.

Performative prediction with state [1] initial distribution
$d_{0}\in\Delta(\mc{Z})$ Publicly known t = 1,2,… Institution plays
$\theta_{t}\in\Theta$. Adversary computes $d_{t}=\Tr(d_{t-1};\theta_{t})$.
$\Tr(;)$ function unknown to institution Institution observes $d_{t}$, suffers
loss $\E_{Z\sim d_{t}}[\ell(Z;\theta_{t})]$.

###  1.1 Repeated risk minimization and stable points

Perdomo et al. propose the following strategy for the institution: at each
round, play the classifier that minimizes loss on the previous distribution.
We investigate the same strategy in our stateful framework.

######  Definition 1 (Repeated risk minimization (RRM)).

Denote by $G(d)$ the updated classifier111For the scenarios we consider, the
set $\mathrm{argmin}_{\theta^{\prime}}$ will be non-empty. When the set has
more than one element, we choose a value for $G(d)$ from the set arbitrarily.
:

$G(d):=\underset{\theta^{\prime}}{\mathrm{argmin}}\>\underset{Z\sim
d}{\E}\ell(Z;\theta^{\prime}).$

Following the notation of Algorithm 1, at round $t$, the institution updates
its classifier to $\theta_{t}=G(d_{t-1})$.

We consider two sufficient conditions for the convergence of RRM in objective
value: approaching a _fixed point distribution_ and approaching a classifier
that is optimal for this distribution.

######  Definition 2 (Fixed point distribution).

A distribution $d_{\theta}$ is a fixed point for $\theta$ if
$\Tr(d_{\theta},\theta)=d_{\theta}$.

######  Definition 3 (Stable points).

A distribution-classifier pair $(d_{\mathrm{PS}},\theta_{\mathrm{PS}})$ is a
performatively stable point if the following hold:

  1. 1.

$\Tr(d_{\mathrm{PS}},\theta_{\mathrm{PS}})=d_{\mathrm{PS}}$, i.e. $d_{\PS}$ is
a fixed point distribution for $\theta_{\PS}$.

  2. 2.

$\theta_{\mathrm{PS}}=G(d_{\mathrm{PS}})$, i.e. $\theta_{\mathrm{PS}}$
minimizes the loss on $d_{\mathrm{PS}}$.

Once the game approaches the distribution $d_{\PS}$, the institution can
repeatedly play $\theta_{\PS}$ with no need for retraining, while incurring
the lowest possible loss on the distribution $d_{\PS}$. It is not obvious
however that such stable points exist for every setting. Nevertheless, we
shows sufficient conditions on the loss and transition function for RRM to
converge to a stable point.

######  Definition 4 ($\eps$-joint sensitivity).

The transition map $\Tr(;)$ is $\eps$-jointly sensitive if, for all
$\theta,\theta^{\prime}\in\Theta$ and $d,d^{\prime}\in\Delta(\mc{Z})$,

$\mc{W}_{1}(\Tr(d;\theta),\Tr(d^{\prime};\theta^{\prime}))\leq\eps\lVert\theta-\theta^{\prime}\rVert_{2}+\eps\mc{W}_{1}(d,d^{\prime}),$

where $\mc{W}_{1}$ denotes the Wasserstein-1 distance between distributions.

######  Definition 5 (Strong convexity).

A loss function $\ell(z;\theta)$ is $\gamma$-strongly convex if, for all
$\theta,\theta^{\prime}\in\Theta$ and $z\in\mc{Z}$,

$\ell(z;\theta)\geq\ell(z;\theta^{\prime})+\nabla_{\theta}\ell(z;\theta^{\prime})^{\top}(\theta-\theta^{\prime})+\frac{\gamma}{2}\lVert\theta-\theta^{\prime}\rVert_{2}^{2}.$

######  Definition 6 (Smoothness).

A loss function $\ell(z;\theta)$ is $\beta$-jointly smooth if the gradient
with respect to $\theta$ is $\beta$-Lipschitz in $\theta$ and $z$:

$\lVert\nabla_{\theta}\ell(z;\theta)-\nabla_{\theta}\ell(z;\theta^{\prime})\rVert_{2}\leq\beta\lVert\theta-\theta^{\prime}\rVert_{2},\quad\lVert\nabla_{\theta}\ell(z;\theta)-\nabla_{\theta}\ell(z^{\prime};\theta)\rVert_{2}\leq\beta\lVert
z-z^{\prime}\rVert_{2},$

for all $\theta,\theta^{\prime}\in\Theta$ and $z,z^{\prime}\in\mc{Z}$.

Finally, our theorems are clearer with additional notation wrapping the
institution’s and adversary’s actions into one step.

######  Definition 7 (RRM map).

Define the RRM map
$f\colon\Delta(\mathcal{Z})\times\Theta\rightarrow\Delta(\mathcal{Z})\times\Theta$
as:

$f(d,\theta)=(\Tr(d,\theta),G(\Tr(d,\theta))).$

In our game, $f(d_{t-1},\theta_{t})=(d_{t},G(d_{t}))=(d_{t},\theta_{t+1})$.

We endow the space $\Delta(\mathcal{Z})\times\Theta$ with the product metric
$\mathrm{dist}$, so that:

$\mathrm{dist}((d,\theta),(d^{\prime},\theta^{\prime}))=\mc{W}_{1}(d,d^{\prime})+\lVert\theta-\theta^{\prime}\rVert_{2}.$

The next example shows that, without the right interplay of the above
parameters, there are settings for which RRM may not converge.

[RRM may not converge] _Suppose that the loss function is the squared loss
$\ell(z;\theta)=(y-\theta)^{2}$ for $\theta\in[1,\infty)$. The loss function
is $\beta$-jointly smooth and $\gamma$-strongly convex, with $\beta=\gamma=2$.
Consider the transition map $\Tr(d;\theta)=1+\eps d+\eps\theta$, which
operates on point mass distributions $d\in[1,\infty)$ of the outcome $Y$.
Clearly, the transition function $\Tr$ is $\eps$-jointly sensitive. Finally,
let some $d_{0}\in[1,\infty)$ be the starting distribution of the game._

_When the institution uses RRM starting from $d_{0}$, we get that:_ θ_t+1 &=
G(d_t) = θargmin Z ∼d_t ℓ(Z; θ) = d_t,
d_t+1 = (d_t; θ_t+1) = 1 + d_t + θ_t+1. _Consequently,
$\theta_{t+2}=d_{t+1}=1+\eps d_{t}+\eps\theta_{t+1}=1+2\eps\theta_{t+1}.$_

_The distance between two successive $\theta$s is
$|\theta_{t+2}-\theta_{t+1}|=|(1+2\eps\theta_{t+1})-(1+2\eps\theta_{t}|)=2\eps|\theta_{t+1}-\theta_{t}|$._

_If we only require $\eps<\frac{\gamma}{\beta}=1$, then whenever
$\eps>\frac{1}{2}$, the sequence of $\theta$s produced by RRM fails to
converge. For $\frac{\eps}{1-\eps}<\frac{\beta}{\gamma}$, i.e.
$\eps<\frac{1}{2}$, the sequence converges._

Our main result is sufficient conditions for the convergence of RRM in the
stateful framework.

######  Theorem 8.

Suppose the transition map $\Tr(;)$ is $\eps$-jointly sensitive with $\eps<1$,
the loss function $\ell(z;\theta)$ is $\beta$-jointly smooth and
$\gamma$-strongly convex. Then for the RRM map $f$ it holds that:

  1. [(a)]

  2. 1.

$\mathrm{dist}(f(d,\theta),f(d^{\prime},\theta^{\prime}))\leq\frac{\eps}{1-\eps}\frac{\beta}{\gamma}\lVert\theta-\theta^{\prime}\rVert_{2}+\frac{\eps}{1-\eps}\frac{\beta}{\gamma}\mc{W}_{1}(d,d^{\prime})$.

  3. 2.

In particular, if $\frac{\eps}{1-\eps}<\frac{\gamma}{\beta}$, then $f$ has a
unique fixed point which is a stable point with respect to $\Tr(;)$. RRM will
converge to this stable point at a linear rate.

Our conditions for convergence of RRM are similar to, but stricter, than those
of the setting of Perdomo et al. In particular, their results only require
that $\eps<\frac{\gamma}{\beta}$. rrm_may_not_converge shows that our stricter
requirement is necessary.

###  1.2 Performative optimality

iterative_risk_min guarantees that RRM converges to an equilibrium, but this
stable point might not be optimal in a more global sense. In fact, it is not
obvious how to define optimal strategies within our framework. Since the
sequence of distributions played by the adversary depends on the initial
distribution $d_{0}$, the best possible strategy for the learner might depend
on $d_{0}$. This is further complicated by the fact that repeatedly playing
the same fixed classifier might result in a distribution shift. Therefore, we
restrict our attention to scenarios where the transition map $\Tr(;)$ is
$\eps$-jointly sensitive with $\eps<1$. In that case, repeated application of
the same classifier $\theta$ is guaranteed to converge to a fixed point
distribution (fixed_point).

######  Claim 9.

If the transition map $\Tr(;)$ is $\eps$-jointly sensitive with $\eps<1$, then
for each $\theta\in\Theta$, there exists a unique fixed point distribution
$d_{\theta}$. For any starting distribution $d_{0}$, iterated application of
the same classifier $\theta$, will result in a sequence of distributions that
converges to $d_{\theta}$ at a linear rate.

banach follows immediately from Banach’s fixed point theorem.

Our definition of the optimal strategy considers the “long-run” loss of a
fixed classifier. Assume the institution plays the same fixed classifier
$\theta$ for all rounds of the game. We measure the long-run loss of $\theta$
as the loss on its corresponding fixed point distribution $d_{\theta}$. The
optimal $\theta$ is the one which minimizes its long-run loss.

######  Definition 10 (Performative optimality).

The long-run loss of a classifier $\theta$ is the loss $\underset{Z\sim
d_{\theta}}{\E}\ell(Z;\theta)$, where $d_{\theta}$ denotes the unique fixed
point distribution for the classifier $\theta$. A classifier
$\theta_{\mathrm{PO}}$ is performatively optimal if achieves the minimum long-
run loss amongst all classifiers in $\Theta$.

If an institution had prior knowledge of the transition map, a reasonable
strategy would be to play the fixed classifier $\theta_{\PO}$ for all rounds
of classification. We note that $\theta_{\mathrm{PO}}$ is undefined when no
classifier has a fixed point distribution. However, it is guaranteed to exist
when $\Tr(;)$ is $\eps$-jointly sensitive with $\eps<1$.

Our definitions of stable and optimal points generalize those of Perdomo et
al. for the stateless framework. As pointed out in Perdomo et al., for a given
setting, the optimal classifier does not necessarily coincide with a stable
point. Our next result shows that RRM approximately approaches optimal points.

######  Theorem 11.

Suppose that the loss $\ell(z;\theta)$ is $L_{z}$-Lipschitz, $\gamma$-strongly
convex, and that the transition map is $\eps$-jointly sensitive with $\eps<1$.
Then for every stable point $\theta_{\mathrm{PS}}$ and optimal point
$\theta_{\mathrm{PO}}$:

$\lVert\theta_{\mathrm{PO}}-\theta_{\mathrm{PS}}\rVert_{2}\leq\frac{2L_{z}\eps}{\gamma(1-\eps)}.$


# Memorization Through the Lens of Curvature of Loss Function Around Samples

Isha Garg , Deepak Ravikumar, Kaushik Roy
Department of Electrical and Computer Engineering,
Purdue University
West Lafayette, IN 47906, USA
<EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Deep neural networks are over-parameterized and easily overfit the datasets
they train on. In the extreme case, it has been shown that these networks can
memorize a training set with fully randomized labels. We propose using the
curvature of loss function around each training sample, averaged over training
epochs, as a measure of memorization of the sample. We use this metric to
study the generalization versus memorization properties of different samples
in popular image datasets and show that it captures memorization statistics
well, both qualitatively and quantitatively. We first show that the high
curvature samples visually correspond to long-tailed, mislabeled, or
conflicting samples, those that are most likely to be memorized. This analysis
helps us find, to the best of our knowledge, a novel failure mode on the
CIFAR100 and ImageNet datasets: that of duplicated images with differing
labels. Quantitatively, we corroborate the validity of our scores via two
methods. First, we validate our scores against an independent and
comprehensively calculated baseline, by showing high cosine similarity with
the memorization scores released by Feldman & Zhang (2020). Second, we inject
corrupted samples which are memorized by the network, and show that these are
learned with high curvature. To this end, we synthetically mislabel a random
subset of the dataset. We overfit a network to it and show that sorting by
curvature yields high AUROC values for identifying the corrupted samples. An
added advantage of our method is that it is scalable, as it requires training
only a single network as opposed to the thousands trained by the baseline,
while capturing the aforementioned failure mode that the baseline fails to
identify.

##  1 Introduction

Deep learning has been hugely successful in many fields. With increasing
availability of data and computing capacity, networks are getting larger,
growing to billions of parameters. This overparametrization often results in
the problem of overfitting. An extreme form of overfitting was demonstrated by
Zhang et al. (2017), who showed that networks can memorize a training set with
fully randomized labels. Further, networks make overconfident predictions,
even when the predictions are incorrect (Guo et al., 2017), memorizing both
mislabeled and long-tailed outliers alike. This can prove to be harmful in
real-world settings, such as with data poisoning attacks(Biggio et al., 2012;
Chen et al., 2017) and privacy leakage in the form of membership inference
attacks (Shokri et al., 2017). There has been considerable research effort
towards countering overfitting, with different forms of regularization (Krogh
& Hertz, 1991), dropout (Srivastava et al., 2014), early stopping (Yao et al.,
2007) and data augmentation (DeVries & Taylor, 2017; Zhang et al., 2018). In
this paper, we exploit overfitting to propose a new metric for measuring
memorization of a data point, that of the curvature of the network loss around
a data point. We overfit intentionally and utilize this to study the samples
the network is memorizing.

We focus our analysis on examples with excessive loss curvature around them
compared to other training data samples. We find that such samples with high
curvature are rare instances, i.e. drawn from the tail of a long-tailed
distribution (Feldman, 2020; Feldman & Zhang, 2020), have conflicting
properties to their labels or are associated with other labels, such as
multiple objects, and most significantly, mislabeled samples. Figure 1 shows
cherry-picked examples from the 100 highest curvature samples from different
vision datasets.

Figure 1: Cherry picked examples from the 100 highest curvature training
examples on MNIST, FMNIST, CIFAR100 and ImageNet.

Consider the images shown for CIFAR100, categories such as ‘girl’ and ‘baby’
and ‘folding_chair’ and ‘rocking_chair’ from ImageNet. The images are the
same, and they semantically belong to both categories, but would be confusing
for single label datasets. Similarly in CIFAR100, images for the classes of
‘plain’, ‘mountain’, and ‘cloud’ have a large overlap. Such similar or
overlapping samples can be challenging to catch while creating datasets, and
can then get allotted to separate annotators that label them differently.
ImageNet labels are further complicated due to multi-object images as is the
case with ‘red_wine’ and ‘corkscrew’ where both the objects are visible in the
image, hence creating conflicting but semantically correct labels. Further,
when looking at high curvature ImageNet images we also find other failure
modes, such as a non-prototypical image for ‘sunglasses’, mislabelled examples
of ‘cassette’ when clearly it is a ‘cassette_player’, and conflicting
conflicting labels for ‘chime’ and ‘gong’.

The quality of datasets impacts not only the accuracy of the network but also
encodes hidden assumptions into the datasets, raising issues of bias and
unfairness. Underrepresented samples of a class in a dataset can have a
significant impact on the performance of the network in real-life settings
(Mehrabi et al., 2021). For datasets not vetted by the public, or those that
are weakly labeled or have noisy annotations, curvature analysis can help to
audit datasets and improve their quality. As an example, while studying
curvature properties of CIFAR100 and ImageNet vision datasets we find that
nearly half of the top 100 high curvature images correspond to duplicated
images with different labels in their training sets (Figure 3 and Appendix
Figure 10). While the duplication of images in CIFAR100 has been noted before
by Recht et al. (2019) and Barz & Denzler (2020), differently labeled
duplicated images are, to the best of our knowledge, a novel observation.

In order to validate our measure of curvature quantitatively, we show that our
scores achieve a high cosine similarity with memorization scores released by
Feldman & Zhang (2020) for CIFAR100 and ImageNet datasets. The baseline scores
were calculated from extensive experimentation, and required training
thousands of models. In contrast, we can calculate curvature while training
only one network, while identifying failure modes that the baseline fails to
capture. As a second, independent check, we synthetically mislabel a small
proportion ($1-10\%$) of the dataset by introducing uniform noise in the
labels and overfit a network to this noisy dataset. We then use our method to
calculate the AUROC score for identifying the mislabeled examples. We
consistently get high AUROC numbers (between $0.84$ and $1$) for the datasets
studied, highlighting that the mislabeled samples were learned with high
curvature.

##  2 Related Work

Deep neural networks are highly overparametrized, thus suffer from
overfitting. Overfitting leads networks to memorize training data to perfect
accuracy, while not improving generalization capability. There has been work
showing that overfitting in neural networks might be benign, and might not
harm testing accuracy (Bartlett et al., 2020). However, overfitting can create
other problems such as increased susceptibility to membership inference
attacks (Shokri et al., 2017; Carlini et al., 2019b), and compromised
adversarial robustness (Rice et al., 2020). The extremity of memorization was
first noted in neural networks by Zhang et al. (2017). Since then, there has
been work in understanding memorization better (Arpit et al., 2017; Feldman &
Zhang, 2020; Feldman, 2020; Jiang et al., 2021; Stephenson et al., 2021).
Feldman & Zhang (2020) suggest calculating memorization score by removing a
set of examples from the training set and observing the change in their
prediction upon training with and without the samples. Related to
memorization, research on the easiness or hardness of examples by Johnson &
Guestrin (2018) and Katharopoulos & Fleuret (2018) utilize the gradient of the
sample as the metric of its importance. AUM (Pleiss et al., 2020) considers
the difference between the target logit and the next best logit as a measure
of importance, and Maini et al. (2022) captures the sample importance in the
time it takes to be forgotten when the network is trained with the sample
removed. Carlini et al. (2019a) study the prototypicality and memorization of
samples based on 5 different correlated metrics.

Methods to counter overfitting include weight decay penalty (Krogh & Hertz,
1991), dropout (Srivastava et al., 2014) and augmentation (Shorten &
Khoshgoftaar, 2019). However, the success of these techniques depends on
having a clean and reliable training dataset. Confident Learning (Northcutt et
al., 2021a) uses the principles of pruning, ranking and counting with out of
prediction probabilities to find mislabeled data. The authors extend it
(Northcutt et al., 2021b) to show errors in many common image datasets. Recht
et al. (2019) build new test sets for CIFAR and ImageNet to check if there has
been unintentional overfitting to the released validation sets. Barz & Denzler
(2020) find many duplicated samples in the CIFAR dataset and create a tool to
flag possible duplicates. We looked at the worst-case examples from these
methods and found that they did not outrightly catch duplicated examples with
different labels, whereas our method does. The curvature of the loss function
with respect to the parameters is well studied in its connection to the
generalization capability of the solution (Keskar et al., 2016; Dinh et al.,
2017; Ghorbani et al., 2019). However, far fewer works focus on the properties
of curvature of loss with respect to data. Prior work in this area has focused
on adversarial robustness (Moosavi-Dezfooli et al., 2019; Fawzi et al., 2018),
and on coresets (Garg & Roy, 2023). However, in this paper, we propose a new
application of input loss curvature as a metric for memorization.

##  3 Methodology

In this section, we outline how we measure curvature and the computational
cost involved. A similar method is often used to calculate the curvature of
loss with respect to the network parameters to determine solution stability
(Dinh et al., 2017; Ghorbani et al., 2019; Keskar et al., 2016). However, we
use this method to calculate the curvature of the loss with respect to the
input points. It is conceptually similar to the method used for curvature
regularization in Moosavi-Dezfooli et al. (2019) and Garg & Roy (2023) with
differences in the choice of random variables, normalization, and
hyperparameters. Details regarding the hyperparameters are provided in
Appendix B.

###  3.1 Measuring Curvature

Let $X\in\mathbb{R}^{D}$ be the input to a neural network. Let
$y\in\\{1,2,..,C\\}$ be the assigned label for this point, corresponding to
the index of the true class among C classes. Let
$\hat{y}=f(X,W)\in\mathbb{R}^{C}$ be the output (pre-softmax logits) of the
network with weights W. Let $L(X)=CrossEntropy(\hat{y},y)\in\mathbb{R}$ be the
loss of the network on this data point. We are interested in the Hessian of
the loss with respect to $X$, $H(X)\in\mathbb{R}^{d\times d}$, where each
element of the matrix is defined by

$[H(X)]_{i,j}=\left[\frac{\partial^{2}L(X)}{\partial x_{i}\partial
x_{j}}\right];i,j=1,2...,D$ (1)

Henceforth, we refer to $H(X)$ as $H$ from now, implicitly understanding that
it is calculated with respect to datapoint $X$ at a given weight $W$. The
local curvature is determined by the eigenvalues of $H$ (Dinh et al., 2017;
Ghorbani et al., 2019; Keskar et al., 2016). The sum of the eigenvalues is
also the trace of the $H$, and can be calculated using Hutchinson’s trace
estimator (Hutchinson, 1990).

$Tr(H)=\mathbb{E}_{v}\left[v^{T}Hv\right]$ (2)

where $v\in\mathbb{R}^{D}$ belongs to a Rademacher distribution, i.e
$v_{i}=\\{+1,-1\\}$ with equal probability. However, we are more interested in
themagnitude of the curvature rather than the definiteness, and hence we look
at the trace of the square of the hessian, which computes to the sum of the
square of the eigenvalues. Since the Hessian is symmetric, we have:

$\displaystyle Tr(H^{2})$
$\displaystyle=\mathbb{E}_{v}\left[v^{T}H^{2}v\right]$
$\displaystyle=\mathbb{E}_{v}\left[(Hv)^{T}(Hv)\right]$
$\displaystyle=\mathbb{E}_{v}\lVert Hv\rVert_{2}^{2}$
$\displaystyle=\frac{1}{n}\sum_{i=0}^{n}\rVert Hv_{i}\rVert_{2}^{2}$ (3)

where n is the number of Rademacher variables to average over. Similar to
Moosavi-Dezfooli et al. (2019) and Garg & Roy (2023), we use finite step
approximation to calculate this efficiently.

$\displaystyle Hv$ $\displaystyle=\frac{1}{h}\left[\frac{\partial
L(x+hv)}{\partial x}-\frac{\partial L(x)}{\partial x}\right]$ (4)
$\displaystyle Hv$
$\displaystyle\propto\frac{\partial\left(L(x+hv)-L(x)\right)}{\partial x}$ (5)

We drop constants as we are only interested in the relative curvatures of
datapoints. For our final curvature estimate, we average curvature over all
training epochs, T, to give reliable results. Putting this together, we have
the curvature estimator of a datapoint $X$ at any epoch as:

$\displaystyle
Curv(X)=\frac{1}{nT}\sum_{t=1}^{T}\sum_{i=0}^{n}\left\lVert\frac{\partial(L(x+hv)-L(x))}{\partial
x}\right\rVert_{2}^{2}$ (6)

###  3.2 Computational Cost

The cost of performing the calculation in Equation 6 is $O(n)$ forward and
backward passes, where $n$ is the number of Randemacher variables $v$ to
average. The backward passes can be parallelized as they are performed at a
static $W$. In case of limited compute capacity, curvature could be estimated
every few epochs. Note that we do not explicitly compute the Hessian or any
second-order gradient despite relying on a second-order metric.

##  4 Experiments and Discussion

In this section, we present qualitative and quantitative results on MNIST
(Deng, 2012), FashionMNIST (Xiao et al., 2017), CIFAR10/100 (Krizhevsky et
al., 2009), and ImageNet (Russakovsky et al., 2015) datasets to support our
claim that curvature can be used to measure memorization. The histograms of
curvatures are shown in Figure 4, and we can see that they reflect the long-
tailed nature of natural images. We use ResNet18 (He et al., 2016) for all our
experiments. Curvature for all datasets except ImageNet is calculated every
epoch and averaged. For ImageNet we average curvature calculated every 4
epochs for computational ease. Details of the setup are given in Appendix 4.
This section first shows qualitative results by visualizing high curvature
samples, and showing that they are long-tailed or mislabeled images in section
4.1. We then validate our score quantitatively in section 4.2 by comparing
against a baseline and in section 4.3 by synthetically mislabeling data and
showing that it gets learned with high curvature. Lastly, in section 4.4, we
study how our measure of curvature evolves during training for further
insight, and so that practitioners with limited compute can better choose
limited epochs for curvature calculation.

figureHistogram of curvature scores

tableCosine Similarity (CS) between curvature and FZ scores with and without
weight decay (WD). Top-K CS is the CS of the top 5,000 (and 50,000) FZ score
samples of CIFAR100 (and ImageNet).  Dataset WD CS Top-K CS CIFAR100 1e-04
0.82 0.90 0.0 0.73 0.82 ImageNet 1e-04 0.72 0.87 0.0 0.66 0.82

###  4.1 Visualizing Low and High Curvature Samples

In Figure 2, we visualize the 10 lowest and highest curvature training samples
for each class of MNIST and FMNIST respectively. These are examples sorted by
curvature scores averaged over all training epochs of a modified ResNet18
architecture, trained without weight decay and allowed to be fully memorized
(100% training accuracy). We see that low curvature samples appear to be
easily recognizable as belonging to their class. Whereas, high curvature
samples, are made of both long-tailed (harder samples or rare occurrences) and
mislabeled samples. The mislabeled samples in MNIST are underlined in red, and
we note that the remaining high curvature samples are also ambiguous. In
particular, we see high overlap between classes ‘4’ and ‘9’, and ‘1’ and ‘7’.

(a)

(b)

Figure 2: Low (left) and high (right) curvature samples of MNIST and
FashionMNIST training data

The classes for FashionMNIST are 0:T-shirt/top, 1:Trouser, 2:Pullover,
3:Dress, 4:Coat, 5:Sandal, 6:Shirt, 7:Sneaker, 8:Bag, 9:Ankle boot. We see
that examples seem to have significant overlap for classes ‘T-shirt’,
‘Pullover’ and ‘Coat’, and among ‘Ankle boots’ and ‘Sneakers’. We also see
long-tailed trends in the row of ‘handbags’, with some uncommon handbags being
represented in the high curvature case. Conflicting samples are apparent in
the row for pants, as there are full body pictures of models, wearing both
pants and shirts. This analysis reveals a bias in the FashionMNIST dataset:
darker clothes with lower contrast tend to show up as high curvature samples,
whereas in the real world, this may not be the case. The corresponding
pictures for the training sets of CIFAR10, CIFAR100 and ImageNet are shown in
Appendix D. We check the validation data as well by training and overfitting
on the validation sets, and the results are shown in Appendix E.

###  4.2 Comparing Against a Baseline: FZ scores

Here, we are interested in an independent measure of memorization that does
not utilize training dynamics. The most suitable metric for comparison comes
from Feldman & Zhang (2020), who remove datapoints from the dataset one at a
time, train a new network on the altered dataset, and measure the change in
prediction on the datapoint as the sample memorization score. These scores are
likely to be independent of spurious correlations to curvature that other
methods such as confidence of prediction might have, and hence serve as a good
baseline. The authors released scores for the CIFAR100 and ImageNet dataset on
their site111https://pluskid.github.io/influence-memorization/, and we refer
to these as FZ scores. We calculate the cosine similarity of our curvature
with the FZ scores in Table 4. We achieve a cosine similarity of 0.82 and 0.72
for CIFAR100 and ImageNet respectively when we use weight decay, and 0.90 and
0.87 on the most memorized samples (see Table 4). These scores are a high
match since the vector we are using is of dimension 50,000 in the case of
CIFAR100, and $1,268,356$ in the case of ImageNet (number of training samples
in the dataset). We emphasize that we achieve this match with only 1 training
run, whereas FZ scores required training thousands of models.

Counter intuitively, we note that the cosine similarity drops by $\sim 0.1$
when not using weight decay. To understand why, we visualize the examples with
the highest curvature without weight decay in Figure 3 for CIFAR 100 and
ImageNet. We note that 36 out of the 60 highest curvature samples for
CIFAR100, and 45 out of top 100 for ImageNet are duplicated pairs with
conflicting labels. These are marked with a red dot. In contrast, FZ scores do
not catch duplicated samples. Similarly, weight-regularized, highest curvature
samples only catch a few of the duplicated samples, as regularization would
deter the network from memorizing these samples. The 100 most memorized
samples from FZ scores and the highest curvature samples for training with
weight decay are shown in Appendix F. These duplicated samples are indeed
memorized during training but possibly missed by FZ scores due to the fact
that they do not train until complete memorization due to the computational
expense of training thousands of models. Additionally, FZ scores are
calculated by removing a proportion of the dataset at a time and training on
the remaining samples. Duplicates can compromise the reliability of these
scores. We note that there are many other duplicate samples that our method
does not identify (Barz & Denzler, 2020), since these duplicate samples have
the same label and do not pose a boundary conflict. They are also not likely
to be memorized.

To summarize, one of the reasons we do not get near perfect match with FZ
scores (and why we get a higher match with weight decay) is because we catch a
novel failure mode that the FZ scores fail to, despite being $\sim 3\times$
more computationally expensive. For the sake of completeness, we recommend
that practitioners try both settings (with and without weight decay) to try
and catch the different kinds of boundary conflicts that may be revealed via
curvature analysis. We do a more detailed analysis in section 4.4, where we
show the cosine matches per epoch.

Figure 3: 50 highest curvature samples from the training set of CIFAR100 and
ImageNet, index, label (and curvature rank for ImageNet) shown above each
image. We highlight duplicated samples with differing labels with a red dot.
Extended versions in Appendix, Figures 8 and 10.

###  4.3 Synthetic Label Corruption

To provide further evidence of curvature as a memorization metric, we devise
an experiment to measure how well our method captures synthetically mislabeled
examples, since they are most likely to be memorized. We randomly introduce
noise into a proportion of the labels, uniformly changing its class label to a
different class label. The same proportion of labels in each class are
corrupted. We then train on these samples and measure the curvature scores.
First, we randomly corrupt the label of 60 images per class of MNIST
(corrupting 1% of the dataset), and train until full memorization, i.e. 100%
training accuracy, indicating that the mislabeled samples were memorized. We
plot the histogram of the cumulative curvature scores and mark the scores of
the corrupted ones with a black $\times$ in Figure 4.4. The training dataset
consists of 60,000 samples and we can see that most of the samples have very
low curvature. However, we see that the curvature scores of the corrupted
examples are among the highest, confirming the link between curvature and
memorization.

To do a more exhaustive study, we sort samples by their curvature estimate and
report the AUROC (Area under the ROC curve) scores for separating the
corrupted samples from the clean ones in Table 1, for corruptions ranging
between 1-10% of 3 widely used vision datasets. The best results are
highlighted in bold red. Since the uncorrupted class has far higher number of
samples than the corrupted class for this binary classification case, ROC
curves are more reliable than the precision-recall curves. As shown later
while studying curvature dynamics in section 4.4, the scores averaged over
training are stable when using weight decay, and hence we use a weight decay
of $10^{-4}$ for all models for these experiments.

Table 1: AUROC for identifying corrupted samples with synthetic label noise for different corruption percentages of datasets. The best results are shown in red. Inconf. refers to inconfidence score, and CL refers to Confident Learning (Northcutt et al., 2021a). | MNIST | CIFAR10 | CIFAR100
---|---|---|---
Corr. | Inconf. | CL | Curv. | Inconf. | CL | Curv. | Inconf. | CL | Curv.
1% | 99.4% | 99.7% | 100.0% | 84.2% | 92.4% | 97.4% | 85.0% | 83.1% | 90.9%
2% | 99.0% | 99.3% | 100.0% | 82.5% | 93.1% | 96.6% | 84.0% | 84.2% | 89.6%
4% | 98.4% | 99.1% | 99.9% | 81.8% | 93.4% | 95.5% | 83.5% | 85.3% | 88.3%
6% | 97.7% | 98.9% | 99.9% | 81.6% | 93.4% | 94.4% | 83.6% | 86.3% | 86.8%
8% | 97.1% | 98.9% | 99.9% | 81.5% | 93.5% | 94.1% | 83.6% | 84.3% | 85.7%
10% | 96.1% | 98.9% | 99.9% | 81.5% | 93.4% | 92.9% | 83.4% | 84.6% | 84.3%

To compare against more baselines, we also provide AUROC results using
inconfidence (Carlini et al., 2019a) and Confident Learning (Northcutt et al.,
2021a). Inconfidence ($=1-$ confidence) was explored as one of the 5 metrics
by Carlini et al. (2019a) to identify long tail and outlier examples.
Confident Learning Confident Learning (CL) (Northcutt et al., 2021a) involves
4 sets of cross-validation experiments and identifies examples as error-prone
after a certain threshold based on counting and ranking principles. The
experiments for CL were performed using the code provided by the authors. From
Table 1, we conclude that our method is more informative than just confidence
values on all datasets and outperforms cleanlab on most corruptions and
datasets. We note that the AUROC score of our method on all datasets except
CIFAR100 is above 92.9%, and above 84.3% on CIFAR100. We wish to emphasize
here that we do not claim that curvature is the best way of finding mislabeled
examples. Our primary motivation is to show that curvature is a good measure
of sample memorization, and exploiting it can find samples with failure modes
that other methods might miss. We recommend that curvature analysis should be
used in conjunction with other checks, for a holistic view of dataset
integrity.

###  4.4 Curvature Dynamics During Training

Until now, we have studied curvature score as the average curvature of each
training epoch. In this section, we study how the curvature of the training
set develops as training progresses, to motivate why averaging curvature
scores over training is needed rather than calculating it at the end of
training. We study per-epoch curvature now, different from the definition of
curvature score in Equation 6. In Figure 4.4, we plot the training and
validation losses and the per-epoch curvature score averaged for the training
set of CIFAR100 on ResNet18. We can see that the network starts to overfit
very early, around epoch 20. We see an odd trend in the per-epoch curvature:
The phase before overfitting is marked by large curvature growth until
overfitting takes over and the curvature starts to decrease. The same trend is
also observed on ImageNet (see Figure 5b).

To understand why this happens, we created a toy example in 2D for
visualization. Training and setup details can be found in Appendix A. The top
left of Figure 4 shows the training and test data, which consist of 2 classes
marked in red and blue sampled from a spiral dataset222Inspired by the
TensorFlow playground https://playground.tensorflow.org. The testing data is
abundant (100 points for each of the 2 classes) and noise-free, representing
the true distribution. However, the training data consists of only 15 points
for each of the 2 classes.

figureHistogram of MNIST trainset curvature. Curvature of 60 corrupted samples
marked

figureLoss curves and average trainset curvature estimates on CIFAR100

Further, 30% of the training data has noise added to it to encourage the
network to memorize these samples. The top right of Figure 4 shows the loss
curves for training and testing data (cross-entropy loss). At the bottom, we
show snapshots of the output space (logits) at the top and the input space at
the bottom. For the output space, we show the $x$ and $y$ axes, corresponding
to each class. The decision boundary in the output space is the line $y=x$,
shown running diagonally. The input space at the bottom visualizes the
probability heatmap of each point in the grid belonging to the red class.

Figure 4: Visualization of curvature of decision boundary around data points.
Train and test datasets are shown on the top left, and loss curves on the top
right. The bottom figures show the logit space and the decision boundary with
input points in the input space at different epochs of training.

From the final learned boundaries in the input space, we note that the regions
without conflicting data points have largely linear decision boundaries, with
complicated boundaries manifesting where data points of both classes are
closely intertwined. From the loss and curvature plots on the top right, we
can see the same increase and decrease of curvature as before. To understand
why, we look at the input and the logit space. From the logit space
visualization, we see that the focus of early epochs is to get all data points
on the correct side of the boundary. Up to this point, the curvature
increases. At this point, even though all the datapoints are on the correct
side, the cross entropy loss can never go to zero, and is further minimized by
maximally separating the correct class logits from the others. This margin
maximization is evident in the last three snapshots: the logits get further
and further apart, while little change is seen in the input space. This is
where the curvature measure starts to drop. As the logits get further apart
from the boundary, the softmax layer included in the cross entropy loss makes
very confident predictions. Thus, small perturbations have very little impact
as the softmax output is already saturated, and hence we see little change in
the gradients, required to measure curvature.

We emphasize that this trend does not affect the results discussed in sections
4.1. 4.2 and 4.3, as we do not compute our curvature score per-epoch, but as
the average. Averaging the scores across epochs allows us to distill all the
epoch-specific variations into one score and allows us to be oblivious to the
exact statistics of learning, including when overfitting began and when the
curvature scores started to change trend. However, understanding this trend
may help practitioners gain insight into important epochs to average over, in
the case of limited compute. Additionally, we are interested only in the
relative ranking of the samples. As can be seen from Figure 5a and Figure 5b,
the curvature of the 5,000 (50,000) most memorized samples as per FZ scores
always remains higher, in each epoch for CIFAR100 (ImageNet)

Another interesting observation is that the curvature of validation samples
keeps increasing throughout training. From Figures 5a and 5b, we can see that
the prolonged overfitting that causes the training set curvature to decrease
due to margin maximization caused by cross entropy loss does not affect the
validation samples. This is expected, since we do not minimize loss on the
validation or test samples. This affords us some insight into two unrelated
mechanisms. First, the sensitivity of test samples to perturbations keeps
increasing with training epochs, and this supports the fact that early
stopping helps lower adversarial vulnerability (Rice et al., 2020). Second,
longer training make the model more susceptible to membership inference
attacks, wherein attackers try to identify the samples used to train the
network (Shokri et al., 2017). In a white box scenario, measuring the
curvature of a sample would help an attacker identify if the sample was used
to train the network.

(a) Average trainset, testset and memorized trainset curvatures on CIFAR100.

(b) Average trainset, testset and memorized trainset curvatures on ImageNet.

Figure 5: Curvature dynamics for CIFAR100 and ImageNet. Note for ImageNet we
evaluate every 4 epochs for compute efficiency.

To round up these results, we show the cosine similarity match between per-
epoch curvature scores and FZ scores, along with the cosine similarity between
the cumulative curvature scores and FZ scores in the Appendix (Section D.4 and
Figures 11b, 11a and 12). We see that right before overfitting, the cosine
match between per-epoch curvature and FZ score is the same as the cosine match
between cumulative curvature and FZ score. The match starts to drop for per-
epoch case and for the case without weight decay, as cross entropy
maximization occurs at the later epochs, and far more aggressively without
weight decay regularization. Hence, averaging over all the weights gives us
the most stable and reliable results.

##  5 Conclusion

Overparametrized networks are known to overfit to the training dataset, and
often achieve 100% training accuracy by memorizing training datapoints. Zhang
et al. (2017) demonstrate an extreme version of this, wherein they show that
the networks can fully memorize a training dataset with all labels randomized
to perfect accuracy. This raises concerns about the networks consuming
erroneous data, which can have an adverse impact when the decisions made by
neural networks are used in real-life scenarios. In this paper, we propose
curvature of the loss function around the datapoint, measured as the trace of
the square of the Hessian, as a metric of memorization. We overfit to the
training datasets, and measure the curvature of the loss around each sample.
We validate curvature as a good measure of memorization in three ways. First,
we visualize the highest curvature samples, and note that they are made of
mislabeled, long-tailed, multiple class, or otherwise conflicting samples that
are not clearly representative of their labels. Second, We also show that
curvature estimates have a very high cosine similarity match with FZ scores,
which are calculated extensively by training thousands of models per dataset.
Instead, our method only requires training one network. Using our method, we
catch a failure mode on CIFAR100 and ImageNet that is to the best of our
knowledge, unobserved until now; that of duplicated images with different
labels. Third, we show that using curvature to identify mislabeled samples in
the case of synthetically mislabeled training sets achieves high AUROC scores.
These three experiments help us establigh curvature as a reliable and scalable
method for measuring memroization of a sample by a network. This can be
utilized to check the integrity of different datasets, to identify areas that
might be undersampled, or badly annotated. Finally, we study how curvature
develops during training, helping us gain insight into the dynamics of
overfitting.

## Acknowledgement

The authors wish to thank Dr. Andy Buchanan for the insightful discussions and
suggestions. This work was supported in part by the Center for Brain Inspired
Computing (C-BRIC), one of the six centers in JUMP, a Semiconductor Research
Corporation (SRC) program sponsored by DARPA, by the Semiconductor Research
Corporation, the National Science Foundation, Intel Corporation, the DoD
Vannevar Bush Fellowship, and by the U.S. Army Research Laboratory.

## References

  * Arpit et al. (2017) Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien.  A closer look at memorization in deep networks.  In Doina Precup and Yee Whye Teh (eds.), _Proceedings of the 34th International Conference on Machine Learning_ , volume 70 of _Proceedings of Machine Learning Research_ , pp. 233–242. PMLR, 06–11 Aug 2017.  URL https://proceedings.mlr.press/v70/arpit17a.html.
  * Bartlett et al. (2020) Peter L. Bartlett, Philip M. Long, Gá bor Lugosi, and Alexander Tsigler.  Benign overfitting in linear regression.  _Proceedings of the National Academy of Sciences_ , 117(48):30063–30070, apr 2020.  doi: 10.1073/pnas.1907378117.  URL https://doi.org/10.1073%2Fpnas.1907378117.
  * Barz & Denzler (2020) Björn Barz and Joachim Denzler.  Do we train on test data? purging cifar of near-duplicates.  _Journal of Imaging_ , 6(6):41, 2020.
  * Biggio et al. (2012) Battista Biggio, Blaine Nelson, and Pavel Laskov.  Poisoning attacks against support vector machines.  In _Proceedings of the 29th International Coference on International Conference on Machine Learning_ , ICML’12, pp. 1467–1474, Madison, WI, USA, 2012. Omnipress.  ISBN 9781450312851.
  * Carlini et al. (2019a) Nicholas Carlini, Ulfar Erlingsson, and Nicolas Papernot.  Distribution density, tails, and outliers in machine learning: Metrics and applications.  _arXiv preprint arXiv:1910.13427_ , 2019a.
  * Carlini et al. (2019b) Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song.  The secret sharer: Evaluating and testing unintended memorization in neural networks.  In _USENIX Security Symposium_ , volume 267, 2019b.
  * Chen et al. (2017) Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song.  Targeted backdoor attacks on deep learning systems using data poisoning.  _arXiv preprint arXiv:1712.05526_ , 2017.
  * Deng (2012) Li Deng.  The mnist database of handwritten digit images for machine learning research.  _IEEE Signal Processing Magazine_ , 29(6):141–142, 2012.
  * DeVries & Taylor (2017) Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_ , 2017.
  * Dinh et al. (2017) Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio.  Sharp minima can generalize for deep nets.  In _International Conference on Machine Learning_ , pp. 1019–1028. PMLR, 2017.
  * Fawzi et al. (2018) Alhussein Fawzi, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, and Stefano Soatto.  Empirical study of the topology and geometry of deep networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 3762–3770, 2018.
  * Feldman (2020) Vitaly Feldman.  Does learning require memorization? a short tale about a long tail.  In _Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing_ , pp. 954–959, 2020.
  * Feldman & Zhang (2020) Vitaly Feldman and Chiyuan Zhang.  What neural networks memorize and why: Discovering the long tail via influence estimation.  _Advances in Neural Information Processing Systems_ , 33:2881–2891, 2020.
  * Garg & Roy (2023) Isha Garg and Kaushik Roy.  Samples with low loss curvature improve data efficiency.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 20290–20300, 2023.
  * Ghorbani et al. (2019) Behrooz Ghorbani, Shankar Krishnan, and Ying Xiao.  An investigation into neural net optimization via hessian eigenvalue density.  In _International Conference on Machine Learning_ , pp. 2232–2241. PMLR, 2019.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In _International conference on machine learning_ , pp. 1321–1330. PMLR, 2017.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hutchinson (1990) Michael F Hutchinson.  A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines.  _Communications in Statistics-Simulation and Computation_ , 19(2):433–450, 1990.
  * Jiang et al. (2021) Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer.  Characterizing structural regularities of labeled data in overparameterized models.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 5034–5044. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/jiang21k.html.
  * Johnson & Guestrin (2018) Tyler B Johnson and Carlos Guestrin.  Training deep models faster with robust, approximate importance sampling.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Katharopoulos & Fleuret (2018) Angelos Katharopoulos and François Fleuret.  Not all samples are created equal: Deep learning with importance sampling.  In _International conference on machine learning_ , pp. 2525–2534. PMLR, 2018.
  * Keskar et al. (2016) Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang.  On large-batch training for deep learning: Generalization gap and sharp minima.  _arXiv preprint arXiv:1609.04836_ , 2016.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images, 2009.
  * Krogh & Hertz (1991) Anders Krogh and John Hertz.  A simple weight decay can improve generalization.  In J. Moody, S. Hanson, and R.P. Lippmann (eds.), _Advances in Neural Information Processing Systems_ , volume 4. Morgan-Kaufmann, 1991.  URL https://proceedings.neurips.cc/paper/1991/file/8eefcfdf5990e441f0fb6f3fad709e21-Paper.pdf.
  * Maini et al. (2022) Pratyush Maini, Saurabh Garg, Zachary C Lipton, and J Zico Kolter.  Characterizing datapoints via second-split forgetting.  _arXiv preprint arXiv:2210.15031_ , 2022.
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan.  A survey on bias and fairness in machine learning.  _ACM Computing Surveys (CSUR)_ , 54(6):1–35, 2021\.
  * Moosavi-Dezfooli et al. (2019) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard.  Robustness via curvature regularization, and vice versa.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 9078–9086, 2019.
  * Northcutt et al. (2021a) Curtis Northcutt, Lu Jiang, and Isaac Chuang.  Confident learning: Estimating uncertainty in dataset labels.  _Journal of Artificial Intelligence Research_ , 70:1373–1411, 2021a.
  * Northcutt et al. (2021b) Curtis G Northcutt, Anish Athalye, and Jonas Mueller.  Pervasive label errors in test sets destabilize machine learning benchmarks.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_ , 2021b.  URL https://openreview.net/forum?id=XccDXrDNLek.
  * Pleiss et al. (2020) Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger.  Identifying mislabeled data using the area under the margin ranking.  _Advances in Neural Information Processing Systems_ , 33:17044–17056, 2020.
  * Recht et al. (2019) Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do ImageNet classifiers generalize to ImageNet?  In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), _Proceedings of the 36th International Conference on Machine Learning_ , volume 97 of _Proceedings of Machine Learning Research_ , pp. 5389–5400. PMLR, 09–15 Jun 2019.  URL https://proceedings.mlr.press/v97/recht19a.html.
  * Rice et al. (2020) Leslie Rice, Eric Wong, and Zico Kolter.  Overfitting in adversarially robust deep learning.  In _International Conference on Machine Learning_ , pp. 8093–8104. PMLR, 2020.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Shokri et al. (2017) Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov.  Membership inference attacks against machine learning models.  In _2017 IEEE symposium on security and privacy (SP)_ , pp. 3–18. IEEE, 2017.
  * Shorten & Khoshgoftaar (2019) Connor Shorten and Taghi M Khoshgoftaar.  A survey on image data augmentation for deep learning.  _Journal of big data_ , 6(1):1–48, 2019.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(56):1929–1958, 2014.  URL http://jmlr.org/papers/v15/srivastava14a.html.
  * Stephenson et al. (2021) Cory Stephenson, Suchismita Padhy, Abhinav Ganesh, Yue Hui, Hanlin Tang, and SueYeon Chung.  On the geometry of generalization and memorization in deep neural networks, 2021.  URL https://arxiv.org/abs/2105.14602.
  * Xiao et al. (2017) Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * Yao et al. (2007) Yuan Yao, Lorenzo Rosasco, and Andrea Caponnetto.  On early stopping in gradient descent learning.  _Constructive Approximation_ , 26(2):289–315, 2007.
  * Zhang et al. (2017) Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=Sy8gdB9xx.
  * Zhang et al. (2018) Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _International Conference on Learning Representations_ , 2018.  URL https://openreview.net/forum?id=r1Ddp1-Rb.

##  Appendix A 2D Toy Example Visualization

We use a network of 7 fully connected layers of size
[2,100,100,500,200,100,1000]. Each layer is followed by a batchnorm layer and
ReLU layer. The network is trained for 150 epochs with an SGD optimizer and a
learning rate of 0.1, momentum of 0.9 and a weight decay of 5e-4. There is
only one minibatch per epoch. The training dataset consists of 15 points for
each of the 2 classes, with a noise ratio of 0.3 introduced to 30% of the
data. The test dataset consists of 100 points for each class with no noise. In
Figure 4, at the bottom we show the input space, specifically, we visualize
the probability heatmap of each point in the grid belonging to the red class.
And we also add the visualization of the output space. Early epochs show large
changes in the curvature around points in the input space. We can see from the
output space that the focus of early epochs is to get all datapoints on the
right side of the boundary. Once the datapoints are on the right side, the
cross entropy loss still aims to maximize the log confidence of the correct
class, by maximally separating the correct class logit from the others. In
these epochs, while the logits get more separated, little change is seen in
the input space. This leads to decreasing curvature after overfitting.

##  Appendix B Hyperparameters

Our estimator introduces two hyperparameters, $h$ and $n$. We tune h in the
range of $\left[10^{-2},10^{-4}\right]$ by evaluating the cosine similarity
between our scores and FZ scores. We note that input $X$ is in the range of
$\left[0,1\right]$ (before mean-standard normalization) and all elements of
$v\in\\{+1,-1\\}$. This means that the $L_{2}$ distance of the noise added to
the input remains in the range of $||hv||_{2}=h\sqrt{D}$, or equivalently, we
add perturb each pixel by $\pm h$. Empirically, we found best results at
$h=0.001$ and $n=10$. A larger $n$ gives very slightly better results, but
increases compute cost.

##  Appendix C Network Architecture and Training details for Curvature
Estimates

We use modified versions of ResNet18 for all experiments with appropriately
modified input sizes and channels. For MNIST and FashionMNIST, we use the
downscaled ResNet 333https://github.com/bearpaw/pytorch-
classification/blob/master/models/cifar/resnet.py with the average pooling
layer downsized from 8 to 7 due to the reduced input resolution of MNIST and
FashionMNIST. For CIFAR datasets we use the full-size
ResNet18444https://github.com/kuangliu/pytorch-
cifar/blob/master/models/resnet.py. WE use PyTorch provided ResNet18 for
ImageNet models. We use no augmentation for MNIST and FashionMNIST, and random
horizontal flips and crops for CIFAR and ImageNet datasets. We train for 300
epochs on CIFAR datasets, with a learning rate of 0.1, scaled by 0.1 on the
$150^{th}$ and $250^{th}$ epoch. For MNIST and FashionMNIST, we train for 200
epochs, with a learning rate of 0.1 scaled by 0.1 on the $80^{th}$ and
$160^{th}$ epoch. For ImageNet we train for 200 epochs with a learning rate of
0.1, scaled by 0.1 on the $120^{th}$ and $160^{th}$ epoch. Where weight decay
is used, its value is set to $10^{-4}$.

##  Appendix D Training Samples with Low and High Curvature

###  D.1 CIFAR10

Figure 6: Low and High curvature samples from CIFAR10. Classes from top to
bottom: [airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and
trucks]

The low curvature samples are shown on the left, with the high curvature
samples on the right. The classes in order are: [airplanes, cars, birds, cats,
deer, dogs, frogs, horses, ships, and trucks].

###  D.2 CIFAR100

The low curvature samples are shown on the left, with the high curvature
samples on the right. We show samples only from the first 10 classes, with the
following labels: [apple, aquarium fish, baby, bear, beaver, bed, bee, beetle,
bicycle, bottle].

Figure 7: Low and High curvature samples from CIFAR100. Classes from top to
bottom: [apple, aquarium fish, baby, bear, beaver, bed, bee, beetle, bicycle,
bottle]. Figure 8: 60 highest curvature samples from the training set of
CIFAR100, identified by training on ResNet18. The index and label of the
training sample are mentioned above the picture. We highlight samples that are
duplicated with differing labels with a red dot.

###  D.3 ImageNet

For ImageNet we show samples only from the first 10 classes, with the
following labels: [tench, goldfish, great white shark, tiger shark,
hammerhead, electric ray, stingray, cock, hen, ostrich]. Note that low
curvature samples on ‘tench’ on ImageNet reveal a spurious correlation learnt
by the network between tench and people holding the ‘tench’ fish.

(a) Low curvature samples from ImageNet

(b) High curvature samples from ImageNet

Figure 9: Low and High curvature samples from ImageNet. Classes from top to
bottom: [tench, goldfish, great white shark, tiger shark, hammerhead, electric
ray, stingray, cock, hen, ostrich]. Figure 10: Top curvature samples from the
training set of ImageNet along with duplicates with corresponding curvature
ranks. Scores identified by training on ResNet18. We highlight samples that
are duplicated with differing labels with a red dot.

###  D.4 Epoch-wise Cosine Similarity

We plot the epoch-wise and cumulative cosine similarity between the FZ score
and curvature scores for CIFRAR100 (Figure 11b) and ImageNet (Figure 11a).
Further, we also plot the epoch-wise and cumulative cosine similarity between
top 50K FZ sample memorization scores and the corresponding curvature scores
for ImageNet (Figure 12). We get very high similarity $\sim 0.9$

(a) Cumulative and epoch-wise curvature of the training set of ImageNet with
and without weight decay.

(b) Cumulative and epoch-wise curvature of the training set of CIFAR100, with
and without weight decay

Figure 11: Cosine similarity between curvature and FZ Memorization score for
ResNet18 on ImageNet (left) and CIFAR100 (right). ImageNet results are plotted
every 4 epochs for efficiency. Figure 12: Cumulative and epoch-wise curvature
of the top 50K FZ samples of ImageNet with and without weight decay.

##  Appendix E Validation Samples with High Curvature

In this section, we train the same network as described in section 4 on the
validation sets of all 4 datasets and show the highest curvature samples for
these. Results are of curvature averaged over all training epochs, and the
network is trained without weight decay.

(a) MNIST

(b) FashionMNIST

Figure 13: High curvature samples from validation sets

(a) CIFAR10

(b) CIFAR100

Figure 14: High curvature samples from validation sets

##  Appendix F CIFAR100 most memorized samples

Here we show the hundred most memorized examples as identified by FZ scores,
and with curvature when training with weight decay $=1e-4$.

(a) Most Memorized according to FZ scores

(b) Highest Curvature with Weight Decay on

Figure 15: High curvature samples from training sets of CIFAR100