# Use of a Generalized Energy Mover’s Distance in the Search for Rare
Phenomena at Colliders

M. Crispim Romão<EMAIL_ADDRESS>LIP, Av. Professor Gama Pinto 2, 1649-003
Lisboa, Portugal  N. F. Castro<EMAIL_ADDRESS>LIP, Av.
Professor Gama Pinto 2, 1649-003 Lisboa, Portugal  Departamento de Física,
Escola de Ciências, Universidade do Minho, 4710-057 Braga, Portugal  J. G.
Milhano<EMAIL_ADDRESS>LIP, Av. Professor Gama Pinto 2, 1649-003 Lisboa,
Portugal  Instituto Superior Técnico, Universidade de Lisboa, Av. Rovisco Pais
1, 1609-001, Lisboa, Portugal  R. Pedro<EMAIL_ADDRESS>LIP, Av. Professor Gama
Pinto 2, 1649-003 Lisboa, Portugal  T. Vale<EMAIL_ADDRESS>LIP, Av.
Professor Gama Pinto 2, 1649-003 Lisboa, Portugal  Departamento de Física,
Escola de Ciências, Universidade do Minho, 4710-057 Braga, Portugal

###### Abstract

In this paper we expand on the previously proposed concept of Energy Mover’s
Distance. The resulting observables are shown to provide a way of identifying
rare processes in proton-proton collider experiments. It is shown that
different processes are grouped together differently, and that this can
contribute to the improvement of experimental analyses. The $t\bar{t}Z$
production at the Large Hadron Collider is used as a benchmark to illustrate
the applicability of the method. Furthermore, we study the use of these
observables as new features which can be used in the training of Deep Neural
Networks.

##  I Energy Mover’s Distance as a Tool for Measurements at Colliders

The concept of a metric for the space of collider events based on the Energy
Mover’s Distance (EMD) has been recently proposed in EventDistance ;
EnergyDistance . Here, we apply an EMD for full event properties to the
classification of physical processes. This application can be particularly
important for the measurement of rare Standard Model (SM) processes, which
typically have cross-sections several orders of magnitude below the
backgrounds affecting their measurement. In such cases, a good discrimination
between signal and background is critical and thus any new variables
contributing to a correct classification of events are of utmost importance,
particularly so for precision measurements. Similarly, extracting information
from the event kinematics to search for new physics phenomena is also a key
aspect when analyzing data produced at colliders Mullin:2019mmh .

In order to expand the concept of EMD to the full reconstructed events, we
propose the generalization of the EMD definition, introducing a new factor
encoding information on the identity of the reconstructed physics objects
present in each event, and the generalised distance $d(I,J)$ between the
events $I$ and $J$ becomes:

$\displaystyle d(I,J)={}$ $\displaystyle{\rm{min}}(\sum_{i,j}\Delta
R_{ij}\times|p_{T,i}-p_{T,j}|\times ID(i,j))$ (1)
$\displaystyle+|E_{I}-E_{J}|\ ,$

where $i$ and $j$ are the final state objects of the events $I$ and $J$,
respectively. The five leading small-$R$ jets and large-$R$ jets, the two
leading electrons and muons and the missing transverse energy ($MET$) are the
final state objects considered. When an event has fewer objects, the non-
existing ones enter in the algorithm as null four-vectors, providing a proxy
to the object multiplicity in the event. Jets are built from calorimeter
energy clusters grouped using the jet finder algorithm anti-$k_{t}$ antikt  as
implemented in the FastJet package fastjet , with radius parameter $R$=0.4 and
$R$=1.0 for small- and large-$R$ jets, respectively.

In Eq. 1, $p_{T}$ is the momentum of the final state objects in the transverse
plane 111The transverse plane is defined with respect to the proton colliding
beams., and $\Delta
R_{ij}=\sqrt{\Delta\phi{{}_{ij}^{2}}+\Delta\eta{{}_{ij}^{2}}}$ is the radial
distance between the object $i$ in the event $I$ and object $j$ in the event
$J$. $\phi$ is the azimuthal angle defined in the transverse plane and $\eta$
is the pseudo-rapidity. Before computing $d(I,J)$ from the simulated Monte
Carlo samples, the events are first boosted to their centre-of-mass frame and
then rotated to align the hardest object vertically in the $(\eta,\phi)$
plane. Since physical laws are Lorentz invariant, this procedure simply
removes spurious information.

The first term in Eq. 1 defines an overall distance between events weighted by
the $p_{T}$ difference of their objects. The factor $ID(i,j)$ is introduced to
encode information on the identity of the final state objects but implies that
Eq. 1 can not be, in general, interpreted as a distance in the geometric term.
For simplicity we still call it a distance throughout the paper. $ID(i,j)$
consists of a variable scale factor that penalises the distance between two
objects if they are of different type, where small-$R$ jets, large-$R$ jets,
electrons, muons and $MET$ are considered different types of objects:

$ID(i,j)=\begin{cases}1&\text{if }ID(i)=ID(j)\\\ ID_{scale}&\text{if
}ID(i)\neq ID(j)\text{ }(ID_{scale}\geq 1)\ .\end{cases}$ (2)

Computing the minimal distance implies minimizing the first term of the Eq. 1.
We address this by using the Earth Mover’s Distance algorithm implemented in
the Python Optimal Transport ot library optimalTransport . Conceptually, the
algorithm computes the minimal cost to transform one event into another.

The second term of the equation takes into account the total energy $E$
difference between the events $I$ and $J$. We study four variations of the
distance between events, resulting from the combination of two options: adding
the energy term or not and employing or not the $ID(i,j)$ scaler, i.e. setting
$ID_{scale}=\\{1,2\\}$:

  * •

$d(I,J)$: $|E_{I}-E_{J}|$ not considered and $ID_{scale}=1$

  * •

$d(I,J){{}^{ID}}$: $|E_{I}-E_{J}|$ not considered and $ID_{scale}=2$

  * •

$d(I,J){{}_{\Delta E}}$: $|E_{I}-E_{J}|$ considered and $ID_{scale}=1$

  * •

$d(I,J){{}_{\Delta E}^{ID}}$: $|E_{I}-E_{J}|$ considered and $ID_{scale}=2$ .

Despite which of the aforementioned options is at hand, distances between
events of similar topology or kinematics will be short while events yielding
different final states will be more separate. This suggests that such an
approach could help to differentiate between physical processes, providing an
additional tool in tasks that demand high discrimination. Its impact could be
specially relevant in studies of rare signals, often the case of searches for
new physics, where the discriminative performance plays a crucial role. We
highlight the adaptable nature of the constructed observables – distances can
be defined regardless of the event topology, the data filters employed or
channel to be analysed – and are therefore suitable for generic and model-
independent searches for new physics and for anomaly detection.

The time performance of the workflow is key to establish its practicability in
a real experimental environment where billions of events need to be processed.
In order to extract discriminative information about the events in a sample
with $N$ events, we would, in principle, need to compute the distances between
all the pairwise combinations of events in the sample, i.e. $N!/(2(N-2)!)$,
which is not feasible even when resourcing to parallel computing and attaining
an average processing time of around 1 ms/distance with the Python
multiprocessing module.

To overcome this drawback we define event references per process sample, that
can later be used as the sample representatives to assess how far/near a given
event is from the represented process. For each sample we compute the
distances between all its events and then use a clustering technique to
capture the structures existent in the data such as different kinematic
regimes. We employ the k-Medoids clustering algorithm with the pyclustering
Python library pyclustering  and identify the medoid of each cluster, i.e.,
the central event according to Eq. 1. The medoid approach was used in
EventDistance  to visualize sub-categories of jets. Here we expand this idea
and use the medoids as the event references per process.

Figure 1: Event distances for a sample of 3 k $t\bar{t}Z$ simulated events
(up) for all pairwise combination of events in the sample, (middle) for
pairwise combination of events belonging to the same cluster and (bottom)
between the cluster events and its medoid.

The number of clusters per sample is optimized using the Silhouette technique
implemented in pyclustering pyclustering . Two clusters were found to be
optimal. Fig. 1 shows the distribution of the event distances for a sample of
simulated $t\bar{t}Z$ events for all pairwise combination of events in the
sample, for pairwise combination of events belonging to the cluster, and
between the cluster events and its medoid. The distributions follow a Landau
curve, typical of many observables in collider experiments. Events within the
first cluster are closer to each other as indicates the lower average and
standard deviation. The second cluster is composed of events more far apart
than in the first cluster but less scattered with respect to the original
distribution, as seen from the lower standard deviation. The distances between
the events and the cluster’s medoids are even shorter as expected from the
k-Medoids clustering.

##  II Physics case and Data Simulation

We use simulated samples of proton-proton collision events generated with
MADGRAPH5_MCATNLO 2.6.5 madgraph  at leading order with a centre-of-mass
energy of 13 TeV. The parton showering and hadronisation was performed with
Pythia 8.2 pythia , using the CMS underlying event tune CUETP8M1 CUETP8M1  and
the NNPDF 2.3 NNPDF2.3  parton distribution functions. The detector simulation
employs the Delphes 3 delphes  multipurpose detector simulator with the
default configuration, corresponding to the parameters of the CMS detector.

The $t\bar{t}Z$ process is used as benchmark, corresponding to a typical
measurement of a rare process at the Large Hadron Collider (LHC). Both the
ATLAS and CMS Collaborations have considered trilepton final states for the
measurement of the $t\bar{t}Z$ cross-sections Aaboud:2019njj ; CMS:2019too
and, therefore, we focus on such topologies. For this we select events with a
final state composed of at least three leptons (_i.e._ electrons or muons)
compatible with the $Z\rightarrow\ell\ell$ decay and a leptonic top decay. Our
main source of background is composed of $t\bar{t}X$ ($X={W,Z,H}$), $tX$
($X={WZ,Zj}$) and dibosons ($WZ$ and $ZZ$). In addition, fake leptons arising
from the misidentification of jets makes $t\bar{t}$+jets and $Z$+jets an
additional non-negligible source of background.

In order to increase the efficiency of the tri-leptonic selection and obtain a
good statistical representation of the different processes, the individual
samples are generated with a dileptonic decay filter. Particle decays are
implemented with MadSpin madspin1 ; madspin2 , a simulator of narrow
resonances decay that preserves spin and correctly implements its angular
correlation scheme in the decay products.

Around 22 M events were simulated in order to achieve a statistical
uncertainty which would be adequate to the analysis of 150 fb-1 of data
produced by the LHC:

  * •

100 k for the $t\bar{t}Z$, $t\bar{t}W$ and $tX$ ($X={WZ,Zj}$) processes;

  * •

500 k for $t\bar{t}H$ and for each diboson ($WZ$ and $ZZ$) sample;

  * •

8 M for the $t\bar{t}$+jets process;

  * •

12 M for $Z$+jets events.

Each process was normalized to the expected yield for the considered benchmark
luminosity of 150 fb-1, assuming the SM cross-sections computed at leading
order with MADGRAPH5.

##  III EMD as High-Level Features

In order to study the use of EMD as high-level features, we compute the
distances between the events of all generated processes and the two medoids
representing each process sample for each four distance options considered -
$d(I,J)$, $d(I,J){{}^{ID}}$, $d(I,J){{}_{\Delta E}}$ and $d(I,J){{}_{\Delta
E}^{ID}}$ \- defined previously.

Figure 2: Distribution of the Event Distances $d(I,J)$ to a (up) $t\bar{t}Z$
medoid and to a (bottom) $WZ$ medoid for each process sample. All
distributions are normalized to the unit area.

Fig. 2 shows two example distributions of the event distances to a $t\bar{t}Z$
medoid and a $WZ$ medoid. Both the average and the median distance to the
$t\bar{t}Z$ and $WZ$ medoids are lower for the $t\bar{t}Z$ and $WZ$ samples,
respectively, as expected. Moreover, $ZZ$ and $Z$+jets events are in average
close to the $WZ$ medoid, and the $t\bar{t}X$ and $tX$ processes exhibit a
short distance from the $t\bar{t}Z$ medoid. This observation provides evidence
that the constructed set of distance observables has the ability of
discriminating between event topologies. This conclusion is valid across all
distributions of the distance observables and even if definite conclusions
would require detailed detector simulation used by the LHC Collaborations
Abdullin:2011zz ; Aad:2010ah , the presented results look promising.

To further investigate the potential of the proposed generalization of EMD to
distinguish physical processes, we determine distances of events with respect
to each sample medoids and use it as a disciminant against the medoid event
process. Corresponding Receiver Operating Characteristic (ROC) curves are
shown in Fig. 3 for one example medoid per process. Distances computed with
respect to the $t\bar{t}Z$, $t\bar{t}X$, $t\bar{t}$+jets and $tX$ medoids
allow to discriminate the diboson and $Z$+jets processes. Conversely,
distances to the diboson and $Z$+jets medoids are sensitive to processes
containing top quarks. It is interesting to note that the constructed
observable does not allow to distinguish $Z$+jets from diboson events. With
the hardest jets originating from gluon splitting and the jet system recoiling
against a dileptonic $Z$, the $Z+$jets events constitute indeed irreducible
background against the diboson signals.

Figure 3: Illustrative receiver operating characteristic (ROC) curves,
corresponding to the event distances to a (a) $t\bar{t}Z$ medoid, (b)
$t\bar{t}H$ medoid, (c) $WZ$ medoid, (d) $tZj$ medoid and (e) $t\bar{t}$+jets
medoid for each process sample. For each case, the ROC evaluates the task of
distinguishing the process represented by the medoid reference from the
remaining processes and the corresponding Area Under the Curve (AUC) is shown.

(a) $d(I,J)$ to a $t\bar{t}Z$ medoid

(b) $d(I,J){{}_{\Delta E}^{ID}}$ to a $t\bar{t}H$ medoid

(c) $d(I,J){{}_{\Delta E}}$ to a $t\bar{t}W$ medoid

(d) $d(I,J){{}_{\Delta E}}$ to a $t\bar{t}$+jets medoid

(e) $d(I,J){{}_{\Delta E}}$ to a $tZj$ medoid

(f) $d(I,J)$ to a $tWZ$ medoid

(g) $d(I,J)$ to a $WZ$ medoid

(h) $d(I,J){{}^{ID}}$ to a $ZZ$ medoid

(i) $d(I,J)$ to a $Z$+jets medoid

In order to further explore how this technique can be used in the context of
High Energy Physics measurements we selected a set of high-level reconstructed
event variables, from which we will derive a baseline to access its
discriminant power, as well as to assess how different distances impact the
corresponding separation performance. Following a typical choice of
information set used in dedicated analysis at the LHC, the selected
reconstructed variables used as features are:

  * •

($p_{T}$, $\eta$, $\phi$) of the two leptons with the highest $p_{T}$;

  * •

($p_{T}$, $\eta$, $\phi$, $m$) of the two small-$R$ jets with highest $p_{T}$;

  * •

$(b_{1},b_{2})$, being two binary variables indicating if the jets were tagged
as originated by a $b$-quark;

  * •

($p_{T}$, $\eta$, $\phi$, $m$, $\tau_{1}$, …, $\tau_{5}$) of the large-$R$ jet
with the highest $p_{T}$;

  * •

small-$R$ jet, electron, muon and large-$R$ multiplicities;

  * •

scalar sum of all the reconstructed objects $p_{T}$, $H_{T}$;

  * •

missing transverse energy and corresponding $\phi$.

With both the event distances and the selected high-level features, we
performed an exploratory analysis by embedding the events into a two
dimensional space using UMAP mcinnes2018umap , as implemented by
mcinnes2018umap-software . The embeddings for the selected features, for all
the event distances, and for the combination of all event distances with the
selected features can be seen in Fig. 4. In this picture we notice how, in a
completely unsupervised manner, the embedding of the events through the
selected features seems to be able to isolate clusters of events from
different samples. The fact that the diboson events appear to be quite
separated from those with a $t$-quark suggests that these events are the
easiest to classify against the other classes, followed by $t\bar{t}Z$ events,
which occupy mostly a single cluster. We also notice that fakes seem to mostly
spread throughout all the clusters, highlighting the difficulty of isolating
them. In the middle figure we show the resulting embedding if we use all the
event distances defined above. Here again, we confirm the conclusion drawn in
the previous section: these distances convey a notion of continuity from
diboson events to $tX$ events. In the third figure we used all the event
distances in addition to the selected features. In this case we notice that we
can identify the same clusters as those appearing in the first picture, but
that the event distances brought in the notion of continuity between events,
continuously connecting some of the clusters.

Figure 4:  UMAP projections of three combination of features. Units of final
embedding space are arbitrary.

(a) Selected Reconstructed Variables

(b) All Event Distances

(c) All Event Distances and Selected Reconstructed Variables

##  IV Deep Learning Application

Since the event distances, either alone or combined with other high-level
features, present a good discriminating power between physics processes, we
went a step forward and studied how such discrimination compares with the one
obtained through advanced machine learning techniques, namely Dense Neural
Network (DNN). For this, we implemented DNN discriminants to perform the
multiclassification task across the different sample classes (diboson, fakes,
$tX$, $t\bar{t}X$ and $t\bar{t}Z$), corresponding to the physics process
defined above.

We use TensorFlow 2.0 tensorflow2015-whitepaper  through its internal Keras
API and followed the same sequential general architecture: input layer with
width matching the number of input features, and a Softmax layer with five
units as the output layer. The hyperparameters were fixed using HyperBand
li2017hyperband  as implemented by Keras-Tuner kerastuner  for each set of
features. A 1:1:1 train-validation-test split was performed for the whole
process and the final results presented here were derived from the test set.

In Fig. 5 we show the confusion matrices for the three combination of features
of Fig. 4, for two operating points. The first operating point (up) is defined
by only accepting predictions, where the most likely prediction is greater
than 0.2. This excludes the cases where the DNN cannot differentiate between
any class and predicts 0.2 for all five classes. The second operating point
(down) is set to 0.9, which will only retain very confident predictions. In
these confusion matrices we notice that for low operation points, the
inclusion of event distances to the high-level features has little performance
impact. For a high operating point, we see that the event distances seem to
help retain a fair discriminative power of fakes. These operating points are
only meant to illustrate the potential of the proposed method since for each
specific analysis they would need to be optimized. A more realistic
experimental analysis would also need to take into account the effect of
systematic sources of uncertainty in such optimization.

Figure 5:  Normalised Confusion Matrices for all DNN, depending on the trained
features.

(a) Selected Reconstructed Variables
Operating point of 0.2

(b) All Event Distances
Operating point of 0.2

(c) All Event Distances and
Selected Reconstructed Variables
Operating point of 0.2

(d) Selected Reconstructed Variables
Operating point of 0.9

(e) All Event Distances
Operating point of 0.9

(f) All Event Distances and
Selected Reconstructed Variables
Operating point of 0.9

Next, in Fig. 6, we present the values of the areas under the ROCs for the
multiclass discrimination using the different feature combinations on top, and
how these compare to the baseline of using the selected reconstructed
variables when training a DNN below. We see that each distance has
discriminative power which depends slightly on the class we are trying to
isolate. For example, the task of identifying fakes seems to benefit from the
inclusion of the distances that include $\Delta E$ contribution to the
distance, and even more from taking all distances into account. Identifying
the remainder of the classes seem to benefit little or not at all from the
inclusion of different event distances as features.

Finally, in Fig. 7, we show how the ROC curves for the task of discriminating
fakes from the remainder of the classes. In this figure we see how different
event distances provide different discriminant power for this specific case.
We also notice that the combination of all event distances without the
selected features has better performance than each distance separately.
Finally, we observe that the ROC curve for the combination of all event
distances with the selected features is the outermost curve for the large
portion of the operating points.

Figure 6: Areas under the ROC curves for all signals for all feature
combinations (left) and how they differ from the baseline of selected
reconstructed features (right).

Figure 7: ROC Curves for the Fake Identification Under Different Combination
of Features

##  V Conclusions

In this paper, the Energy Mover’s Distance concept was used to create a new
set of observables that could be used in the measurement of rare processes at
proton-proton colliders, using $t\bar{t}Z$ as a study case. We have shown that
such new observables, which build on the previously proposed concept of EMD,
perform well in the task of grouping together different processes based on
their topologies, showing a fair discrimination power by themselves. Namely,
it can be seen that the distances between $t\bar{t}Z$ and $t\bar{t}X$ are
smaller than $ZZ$ and $WZ$. This indicates that the EMD based observables can
be useful in the analysis of collider data, providing an interesting way to
explore data and classify it in generic classes, which can be matched with
significant accuracy to different physics processes.

Additionally, the use of these observables in the training of a DNN was
tested. Even if the overall performance of the DNN is not, in general,
significantly increased, such observables are interesting on themselves since
they provide event level information which is beneficial for the
classification of processes with fake leptons in some scenarios. Furthermore,
such event-level observables might be affected differently by systematic
uncertainties - a study beyond the scope of the current paper which deserves
further investigation.

##  VI Acknowledgments

We thank Jesse Thaler and Guilherme Guedes for the very useful discussions. We
also acknowledge the support from FCT Portugal, Lisboa2020, Compete2020,
Portugal2020 and FEDER under the projects PTDC/FIS-PAR/29147/2017 and
CERN/FIS-PAR/0024/2019 and through the grant PD/BD/135435/2017. The
computational part of this work was supported by INCD (funded by FCT and FEDER
under the project 01/SAICT/2016 nr. 022153) and by the Minho Advanced
Computing Center (MACC). The Titan Xp GPU card used for the training of the
Deep Neural Networks developed for this project was kindly donated by the
NVIDIA Corporation.

## References

  * (1) P. T. Komiske, E. M. Metodiev, and J. Thaler.  Metric Space of Collider Events.  Phys. Rev. Lett., 123(4):041801, 2019.
  * (2) P. T. Komiske, E. M. Metodiev, and J. Thaler.  The Hidden Geometry of Particle Collisions.  arXiv preprint arXiv:2004.04159, 2020.
  * (3) A. Mullin, H. Pacey, M. Parker, M. White, and S. Williams.  Does SUSY have friends? A new approach for LHC event analysis.  arXiv preprint 1912.10625, 2019.
  * (4) M. Cacciari, G. P. Salam, and G. Soyez.  The anti-$k_{t}$ jet clustering algorithm.  JHEP, 04:063, 2008.
  * (5) M. Cacciari, G. P. Salam, and G. Soyez.  FastJet User Manual.  Eur. Phys. J., C72:1896, 2012.
  * (6) The transverse plane is defined with respect to the proton colliding beams.
  * (7) R. Flamary and N. Courty.  Pot python optimal transport library, 2017.
  * (8) A. Novikov.  PyClustering: Data mining library.  Journal of Open Source Software, 4(36):1230, apr 2019.
  * (9) J. Alwall et al.  The automated computation of tree-level and next-to-leading order differential cross sections, and their matching to parton shower simulations.  JHEP, 07:079, 2014.
  * (10) T. Sjöstrand et al.  An Introduction to PYTHIA 8.2.  Comput. Phys. Commun., 191:159–177, 2015.
  * (11) V. Khachatryan et al.  Event generator tunes obtained from underlying event and multiparton scattering measurements.  Eur. Phys. J., C76(3):155, 2016.
  * (12) R. D. Ball et al.  Parton distributions with LHC data.  Nucl. Phys., B867:244–289, 2013.
  * (13) J. de Favereau et al.  DELPHES 3, A modular framework for fast simulation of a generic collider experiment.  JHEP, 02:057, 2014.
  * (14) ATLAS Collaboration.  Measurement of the $t\bar{t}Z$ and $t\bar{t}W$ cross sections in proton-proton collisions at $\sqrt{s}=13$ TeV with the ATLAS detector.  Phys. Rev., D99(7):072009, 2019.
  * (15) CMS Collaboration.  Measurement of top quark pair production in association with a Z boson in proton-proton collisions at $\sqrt{s}=$ 13 TeV.  JHEP, 03:056, 2020.
  * (16) P. Artoisenet, R. Frederix, O. Mattelaer, and R. Rietkerk.  Automatic spin-entangled decays of heavy resonances in Monte Carlo simulations.  JHEP, 03:015, 2013.
  * (17) S. Frixione, E. Laenen, P. Motylinski, and B. R. Webber.  Angular correlations of lepton pairs from vector boson and top quark decays in Monte Carlo simulations.  JHEP, 04:081, 2007.
  * (18) S. Abdullin, P. Azzi, F. Beaudette, P. Janot, and A. Perrotta.  The fast simulation of the CMS detector at LHC.  J. Phys. Conf. Ser., 331:032049, 2011.
  * (19) ATLAS Collaboration.  The ATLAS Simulation Infrastructure.  Eur. Phys. J., C70:823–874, 2010.
  * (20) L. McInnes, J. Healy, and J. Melville.  Umap: Uniform manifold approximation and projection for dimension reduction.  arXiv preprint arXiv:1802.03426, 2018.
  * (21) L. McInnes, J. Healy, N. Saul, and L. Grossberger.  Umap: Uniform manifold approximation and projection.  The Journal of Open Source Software, 3(29):861, 2018.
  * (22) M. Abadi et al.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.
  * (23) L. Li, K. Jamieson, G. DeSalvo, A. Rostamizadeh, and A. Talwalkar.  Hyperband: A novel bandit-based approach to hyperparameter optimization.  The Journal of Machine Learning Research, 18(1):6765–6816, 2017\.
  * (24) Keras Team.  Keras tuner: A hyperparameter tuner for keras.


meaningful way to group data points qualitatively. Our findings suggest that
we can unify the understanding and improve uncertainty quantification and
recalibration methods with atypicality, however, practitioners should be
careful about incorporating atypicality, as poor atypicality estimates can
lead to worse performance.

###  H.2 Subgroup Fairness Experiments

While the literature on algorithms to satisfy group fairness is rich [31, 26],
here we wanted to give a case study with skin-lesion classification. Our goal
was to provide further evidence that atypicality awareness could improve
fairness algorithms. In this domain, there is more verification to do to
better characterize how and when atypicality helps, thus, to better understand
which subgroups could benefit more from atypicality-aware algorithms and
whether these findings apply generally in the literature.

###  H.3 Theoretical Analysis

Following the earlier work [7, 62], we analyzed the calibration behavior of
well-specified logistic regression. However, our empirical findings suggest
that the phenomena are much more broadly applicable. We suggest that future
work can analyze the behavior in more general settings to better understand
the dynamics.

Figure 14: Distribution of Input Atypicality. Here, we give the distribution
of Atypicality for ResNet18 on ImageNet, using GMM and KNN methods. Figure 15:
Label Atypicality and Confidence. Here, the x-axis reflects the label
atypicality quantile, and the y-axis indicates confidence. The coloring for
the figure on the left indicates the accuracy within a bin, and the figure on
the right has the difference between confidence and accuracy within a bin.
Similar to Figure 6, we observe that atypical examples have lower accuracy,
and predictions are more overconfident. Figure 16: Input Atypicality and
Confidence for ResNet152. We provide the input atypicality for ResNet152, in
the same structure as Figure 15.