# Finding the semantic similarity in single-particle diffraction images
using self-supervised contrastive projection learning

Julian Zimmermann<EMAIL_ADDRESS>ETH Zürich, 8092 Zürich,
Switzerland  Fabien Beguet  ETH Zürich, 8092 Zürich, Switzerland  Daniel
Guthruf  ETH Zürich, 8092 Zürich, Switzerland  Bruno Langbehn  Technische
Universität Berlin, 10623 Berlin, Germany  Daniela Rupp  ETH Zürich, 8092
Zürich, Switzerland  Max-Born-Institut, 12489 Berlin, Germany

###### Abstract

Single-shot diffraction imaging of isolated nanosized particles has seen
remarkable success in recent years, yielding in-situ measurements with ultra-
high spatial and temporal resolution. The progress of high-repetition-rate
sources for intense X-ray pulses has further enabled recording datasets
containing millions of diffraction images, which are needed for structure
determination of specimens with greater structural variety and for dynamic
experiments. The size of the datasets, however, represents a monumental
problem for their analysis. Here, we present an automatized approach for
finding semantic similarities in coherent diffraction images without relying
on human expert labeling. By introducing the concept of projection learning,
we extend self-supervised contrastive learning to the context of coherent
diffraction imaging. As a result, we achieve a semantic dimensionality
reduction producing meaningful embeddings that align with the physical
intuition of an experienced human researcher. The method yields a substantial
improvement compared to previous approaches, paving the way toward real-time
and large-scale analysis of coherent diffraction experiments at X-ray free-
electron lasers.

††preprint: v1.0

##  I Introduction

A guiding principle in fundamental condensed matter research is that for
understanding function, we have to study structure [1]. Techniques based on
lensless diffractive imaging, like X-ray crystallography and coherent
diffraction imaging (CDI), are powerful and widely used tools to discover
structures up to atomic resolutions [2]. In the recent past, single-particle
coherent diffraction imaging using intense coherent X-ray pulses from free-
electron lasers (SP-CDI) [3, 2] has revolutionized the field of structural
characterization [4, 5, 6, 7, 8, 9, 10, 11, 12]. SP-CDI is a technique with
which in-situ measurements of isolated and non- fixated nano-scaled targets
can be acquired. Depending on the experimental scheme, each recorded
diffraction image is a complete and self-contained experiment that needs
individual analysis [3]. However, due to the advent of high repetition-rate
sources like the European XFEL [13] and LCLS-II [14], millions of images are
typically recorded during one experimental campaign [15]. Manual analysis of
such amounts of data represents an enormous problem. It may leave researchers
unable to analyze significant amounts of their data comparatively as they have
to resort to large-scale averaging, which might _wash out_ or _conceal_
important information, or manually select subsets of the dataset. In this
work, we present a novel embedding technique for diffraction images called
contrastive projection learning (CPLR) based on contrastive learning (CLR)
[16, 17]. CPLR produces a dimensionality-reduced embedding space with which
semantic comparisons between diffraction images become possible and, thus,
enables human-level comparative analysis on big-data scale datasets.

At the very core of every comparative analysis is an assumption establishing a
_similarity measure_ between samples. However, current approaches for
establishing such a measure for diffraction images cannot compete with the
perception of a trained researcher. This perceived similarity, or _semantic
similarity_ [18], is contextually aware [19], whereas computational methods
for diffraction image data currently lack such awareness.

Available strategies for comparative analysis are based on either supervised
classifier schemes [20, 21] or unsupervised sorting methods [22, 23, 24, 25,
26]. However, all these methods come with significant trade-offs: Supervised
algorithms align with human perception and produce high-accuracy results [21,
27] but are in real-world scenarios unavailable as they require time- and
labor-expensive manually fabricated expert labels. Unsupervised routines work
in such cases but do not reach comparable accuracy levels and introduce
additional restrictions or requirements. For example, traditional cluster
techniques produce a lot of unwanted predictions [22], threshold-based
approaches act primarily as hit-finder [23, 28], autocorrelation-based methods
only extract particle size information and are computationally costly [24],
auxiliary approaches rely on rarely available additional experimental data
[24], and Fourier-inversion based techniques [24, 26, 15] are only applicable
to SP-CDI data from the _small-angle-scattering_ regime, where reconstruction
by Fourier inversion is possible [10, 29].

Our CPLR method can potentially improve all strategies mentioned above; It can
serve as an improved similarity measure for unsupervised methods, as in the
context of regular self-supervised learning [30, 31, 32], and can act as a
powerful pretraining for subsequent supervised or distillation-based training
[17]. Furthermore, CPLR directly establishes a way to find _semantically
similar_ diffraction images in a fully self-supervised fashion. In self-
supervised contrastive learning, a supervised task is constructed by
artificially creating label information via domain-specific augmentation
strategies [33, 34, 16]. In this work, we design a novel augmentation approach
for diffraction image data where a deep neural network contrasts images from
different coordinate projections.

The quality of the CPLR embedding space is evaluated using a publicly
available diffraction image dataset [35] from an SP-CDI experiment on
superfluid helium nanodroplets, for which semantically sensitive expert labels
are available [12, 21]. Using the broadly established linear evaluation
protocol [36, 16, 17], we show that our method outperforms non-contrastive
methods by a large margin while improving the contrastive-learning baseline by
$610\text{\,}\mathrm{\char 37\relax}$.

Figure 1:  Illustrating semantic similarity. a, b and c are diffraction images
taken from a publicly available dataset [35] from an SP-CDI experiment on
superfluid helium nanodroplets [12, 21]. a and b are semantically similar
while a and c are not. d shows four widely used similarity measures (see
[37]), that disagree with the human perception, while only our method
(labelled CPLR) agrees with it. Color-coding is that the pair of images that
is _more-similar_ is green, while the one that is _less-similar_ is orange.

Figure 1 provides a concrete example of three diffraction images from [12], a,
b, and c. Two images (b and c) are to be compared to a. A human immediately
identifies the elongated streak-like feature in a as the dominant
characteristic and can identify b as being more similar to a than c is to a.
And, indeed, this is correct from a physical perspective. The nanoparticles’
structures that produce a and b are more similar to each other than those that
produce a and c [12]. However, from an algorithmic perspective, this is not a
trivial problem. Figure 1 d shows for b and c five similarity measures: Four
widely used measures for image similarity (see [37]) and our method (labeled
CPLR). Color-coding is that the image that is _more similar_ to a is green,
while the _less similar_ image is orange. The only measure that agrees with
the human perception, and the physics of the problem, is our CPLR method.

Our approach is not limited to data from SP-CDI experiments. Theoretically,
CPLR can be applied to diffraction data from all experimental techniques
operating in Polar coordinates, including X-ray crystallography and
traditional CDI approaches. Ultimately, CPLR provides a path for analyzing the
impending amounts of diffraction data where the human perceived similarity is
maintained even among millions of diffraction images.

##  II Results and discussion

Contrastive learning is about augmentation, not architecture. In contrastive
learning (CLR), we artificially create label information for supervised
learning by designing augmentation pipelines that consider domain [38] and
task [39] knowledge [16]. Therefore, CLR is an instance of self-supervised
learning [40]. The fundamental assumption in self-supervised learning is that
the input data contain more task-specific information than sparse categorical
ground truth data in supervised learning [41]. Consequently, a careful
augmentation design should provide better results on downstream tasks than a
supervised learning scenario [42, 41]. While improvements in _Accuracy_ in
supervised learning are usually related to architecture modifications,
regularization, or loss function, CLR is about domain-specific augmentation
strategies above anything else [16, 41]. Formally, CLR is a technique to
create an embedding space from arbitrary input modalities, enabling
comparative analysis. CLR dates back to work done in the nineties [42] but
only recently has seen a renaissance, yielding State-of-the-art results in
visual- [36, 16, 17, 43], audio- [44, 45, 46], video- [47, 48, 49], and text-
representation [50, 51] learning.

Contrastive baseline and contrastive projection learning. In this study, we
use the experimental design presented in [17], called _SimCLRv2_ , as a
baseline to compare our results. A large encoder and a smaller transformation
neural network produce the representations in two stages. First, the encoder
acts as a feature extractor; then, the transformation network learns an
optimized representation that minimizes the CLR loss, termed normalized
temperature-scaled cross-entropy loss (NT-Xent) [52]. Conceptually, a
duplicate is produced for each input image where both images are heavily
augmented. All duplicates form so-called _positive pairs_ with their
originals, while all images with all other images but their duplicates form
_negative pairs_. Then, the network learns to discriminate between positive
and negative pairs during training.

Figure 2:  Showing details on the CPLR architecture, the concept, and the
classes within the dataset. i) Schematic of the used architecture. In this
example, a positive pair, consisting of one diffraction image in its natural
Polar and its Cartesian form, are passed on to the feature-extractor network,
a modified ResNet50-D [53]. The adjacent grey box then shows the
transformation network, a three layer MLP, where the first layer - in green -
forms the embedding space within the here used SimCLRv2 [17] framework. ii)
The conceptual idea in more detail. The goal during training is to maximize
similarity (minimize distance in embedding space) for positive pairs and
minimize similarity (maximize distance in embedding space) for negative pairs.
iii) The seven possible, non-exclusive classes of the here used multiclass and
multilabel dataset. The top row shows the diffraction image in Polar
coordinates and the bottom row the Cartesian projection. In addition, the
absolute and relative class counts are given alongside the class names.
Several different features can appear in a single image which then belongs to
several classes simultaneously. For example, a _Streak_ feature is also
present in the example for the _Layered_ class.

In contrastive projection learning (CPLR), we produce the positive pairs not
from the same image, as in [16, 17], but project the diffraction images, which
are naturally recorded in Polar coordinates, to Cartesian coordinates. Figure
2 i) and ii) provide a schematic overview of the network design and the
conceptual idea. Using coordinate projections as an augmentation strategy
implicitly penalizes that trivial Polar symmetries are learned and explicitly
enforces that learned representations are invariant under rotational and
translational changes. A simple example can be constructed with the help of
figure 2 iii). There, example images for every class in the dataset are shown.
While figure 2 iii) is fully explained in the following subsection, here, we
concentrate on the Polar and Cartesian projection of the _Elliptical_ class -
the first column. The Polar form shows the characteristic _Airy_ rings typical
for single-laser-shot and single-particle imaging data [12]. Usually, learning
rotational invariance in arbitrary image data is achieved via random
rotational transformations during the augmentation stage of training a
network. However, such a transformation would yield no, or very little, change
with such diffraction images due to the high degree of rotational symmetry.
Therefore, we encourage the network to decouple the learned representation
from rotational symmetries by correlating the Polar form with the Cartesian
form. In addition, after the coordinate transformation, we leverage a
stochastic augmentation pipeline [16, 17]. More details can be found in the
_Methods_ section _Augmentation strategy_.

The dataset. Helium nanodroplets were imaged at XUV photon energies in a
single-shot single-particle experiment [12] at the FERMI free-electron laser
[54]. The scattering images were recorded using a non-linear MCP-type detector
[55] in a so-called wide-angle setting where each diffraction image contains
3D-structural information and cannot be reconstructed via Fourier inversion
[10, 56, 12]. The publicly available and hand-curated dataset [35] contains
$7264$ diffraction images with semantically sensitive labels and was
previously used in a supervised classification task [21]. We discarded $6000$
diffraction images as they either exhibited strictly round or no Airy patterns
at all. The round Airy patterns are by far the most common class, and we
removed them to create a more balanced dataset since they can be reliably
sorted using radial slices and a classical peak-finder [57].

The provided expert labels can be used for multiclass and multilabel analysis,
meaning every diffraction image has binary label information for multiple
classes that are often mutually non-exclusive. Figure 2 iii) shows this
dataset’s seven possible classes and their absolute and relative occurrence.
For every class, one example is given in Polar and Cartesian form. To
illustrate the multilabel property: The characteristic streak-like feature
that defines the _Streak_ class can also be found, for example, in the image
for the _Layered_ class. For this reason, the given percentages do not add up
to $100\text{\,}\mathrm{\char 37\relax}$, as multiple images belong to
multiple classes and most classes are heavily under-represented. This
pronounced multilabel imbalance is typical for diffraction image datasets [12,
9].

Training and evaluation. As in [17], we use a $$2$x$-wide [58], selective-
kernel [59] ResNet50-D [53] network with squeeze-excitation blocks [60] as
feature extractor, and a three-layer multi-layer- perceptron (MLP) as
transformation network. We train for $1000$ epochs, with a batch size of
$628$, using the LARS optimizer [61] with a cosine schedule [62], ten warmup
epochs [63], and optimizing the NT-Xent loss [52]. Training takes half an hour
on four NVIDIA $3090$ GPUs. The code, pretrained models, and training results
are openly available at [64].

The quality of the learned embedding space is evaluated using the so-called
linear evaluation protocol [36, 16, 17], which is carried out as follows:
After self-supervised training, we freeze the feature extractor network and
use the first layer of the transformation network (this is indicated by the
green layer in 2 i)) as embedding space. Then, for every ground truth class in
the dataset, we train a linear classifier on top of the learned
representations and calculate the _Precision_ and _Recall_ score. Both metrics
are obtained for every class via $5$-fold stratified cross-validation to
account for statistical fluctuations from sampling and the dataset’s class
imbalance. It has turned out to be important to use _Precision_ and _Recall_
as most classes are very rare, with three out of seven classes appearing in
under $7\text{\,}\mathrm{\char 37\relax}$ of all images. In these cases,
_Accuracy_ would produce very high scores for classifiers predicting every
image as not being part of any class. Moreover, we calculate an additional
metric called _Overlap_ in order to compare our method with metrics operating
on raw images, such as the structural similarity (SSIM) index [65], the
complex wavelet SSIM (CWSSIM) index [66], and the keypoint-based scale-
invariant feature transform (SIFT) distance [67]. _Overlap_ is the global
average of the normalized dot-product between the ground truth of every image
and its $13$ closest images according to the pairwise-calculated distances.
$13$ was chosen as it corresponds to $1\text{\,}\mathrm{\char 37\relax}$ of
all images in the dataset, which sets the _Overlap_ score to be a local-
neighborhood evaluation. Consequently, an _Overlap_ score of $0.5$ corresponds
to: _On average, the $13$ most similar images shared $50\text{\,}\mathrm{\char
37\relax}$ of the original image’s labels_.

We compare our method to the SimCLRv2 baseline and other approaches that have
been used in the past with diffraction images in the following sections. More
details on training and evaluation can be found in the _Methods_ section
_Training and linear evaluation strategy_.

The embedding space is linearly separable into semantic features.

Table 1:  Macroaverage [68] results on the helium nanodroplets dataset. _Contrastive-based_ are the results for the CLR baseline and the CPLR method, where the arrows indicate with which projections the contrastive task was constructed. _Continuous latent variables_ or _variational Bayesian_ methods list techniques that have been used with diffraction data in the past. _Random baseline_ gives the result for an artificial embedding space built from uniform noise, this is the lowest possible score. _Direct measures_ are applied directly on the images, and cannot be evaluated using the linear evaluation protocol. All methods except for the _Direct measures_ are trained / ran for five times where each time the evaluation scores where obtained via $5$-fold stratified cross validation. The standard deviation of all methods is equal or below $0.01$. The best result for each score is given in bold letters.  | Linear evaluation | Local similarity
---|---|---
Method | Precision | Recall | Measure | Overlap
Contrastive-based
CLR
Polar$\,\Leftrightarrow\,$Polar | $0.49$ | $0.50$ | Cosine | $0.49$
Cartesian$\,\Leftrightarrow\,$Cartesian | $0.49$ | $0.48$ | Cosine | $0.43$
CPLR
Cartesian$\,\Leftrightarrow\,$Polar | $0.51$ | $0.54$ | Cosine | $0.52$
Polar$\,\Leftrightarrow\,$Cartesian | $0.52$ | $0.55$ | Cosine | $0.52$
Continuous latent variables methods
Factor Analysis | $0.28$ | $0.35$ | Euclidean | $0.28$
PCA | $0.29$ | $0.35$ | Euclidean | $0.40$
Correlation | $0.43$
Kernel PCA | $0.28$ | $0.35$ | Euclidean | $0.43$
Correlation | $0.40$
UMAP [69] | $0.30$ | $0.31$ | Euclidean | $0.41$
Variational Bayesian methods
VAE [70, 71] | $0.34$ | $0.35$ | Wasserstein $W_{2}$ | $0.42$
Random baseline
Uniform Noise | $0.23$ | $0.27$ | Euclidean | $0.28$
Direct measures
SSIM [65] | N/A | N/A | Custom | $0.37$
CWSSIM [66] | N/A | N/A | Custom | $0.36$
SIFT [67] | N/A | N/A | Euclidean | $0.32$

The evaluation scores after training are provided in table 1. The second and
third columns show the _Precision_ and _Recall_ score of the linear evaluation
protocol, and the last two columns show the used metric for calculating the
pairwise distances to calculate the _Overlap_ score, which is given in the
last column. _Contrastive-based_ shows the results for the CLR baseline and
our CPLR method. The arrows indicate the coordinate projections used, where
the first term is used for inference and the second is used for constructing
the contrastive task. Consequently, _Polar $\,\Leftrightarrow\,$Polar_ and
_Cartesian $\,\Leftrightarrow\,$Cartesian_ are cases of the standard CLR
framework with either purely unmodified (_Polar_) or purely projected
(_Cartesian_) diffraction images. _Cartesian $\,\Leftrightarrow\,$Polar_ and
_Polar $\,\Leftrightarrow\,$Cartesian_ are cases of our CPLR method, where the
difference between the two is that we changed the input for inference at
evaluation time to either the _Cartesian_ or the _Polar_ form. _Continuous
latent variables_ and _variational Bayesian_ methods are techniques used
previously in the context of diffraction images. More details are given in the
_Methods_ section _Non-contrastive-based methods_. _Random baseline_ gives the
result for an artificial embedding space built from uniform noise; this is the
lowest possible score. This baseline is equivalent to a _Random guesser_ with
no learned information about the dataset. _Direct measures_ are methods
applied directly to the images, which cannot be evaluated using the linear
evaluation protocol.

All methods except for the _Direct measures_ are trained / run five times
using five different integer _random_state_ keys where the evaluation scores
were each time obtained via $5$-fold stratified cross-validation. The macro
average [68] over all classes for all train and cross-validation runs is given
in the table. The standard deviation of all methods is equal to or below
$0.01$.

Relative to the CLR baseline, the CPLR method yields significant improvements
on all metrics improving _Precision_ by $6\text{\,}\mathrm{\char 37\relax}$
and _Recall_ by $10\text{\,}\mathrm{\char 37\relax}$. Relative to the best
non-CLR methods (VAE for _Precision_ and all but UMAP for _Recall_), the CPLR
method improves _Precision_ by $35\text{\,}\mathrm{\char 37\relax}$ and
_Recall_ by $36\text{\,}\mathrm{\char 37\relax}$. In addition, the _Overlap_
score is relatively improved by about $6\text{\,}\mathrm{\char 37\relax}$
compared to the CLR baseline and $17\text{\,}\mathrm{\char 37\relax}$ compared
to the two best PCA-based approaches. CPLR is the only method that achieves
_Precision_ and _Overlap_ scores above $0.50$.

CPLR is more robust with fewer samples. The general idea of using the linear
evaluation protocol for evaluation is to look for linearly separable regions
in the embedding space. Therefore, this method only applies to _one-hot_ [72]
ground truth data, meaning _multiclass but single label_. However, the helium
nanodroplets dataset has multiclass and multilabel [73] ground truth data,
where each image has multiple associated labels. Moreover, as typical in
datasets on helium nanodroplets, the dataset is heavily unbalanced, where
simpler shapes, like _Elliptical_ , dominate other classes [9].

Table 2:  Results for every class in the helium nanodroplets dataset. Compared to table 1, we only show the four best-performing methods, namely our CPLR (_Polar $\,\Leftrightarrow\,$Cartesian_) method along with the CLR (_Polar $\,\Leftrightarrow\,$Polar_) baseline, and the VAE, and Kernel PCA (Using the euclidean metric) approaches. All methods have been trained / run for five times where each time the evaluation scores where obtained via $5$-fold stratified cross validation. The standard deviation of all methods and for all classes is equal or below $0.01$. The best result for each score is highlighted in bold letters.  |  |  | CPLR | CLR | VAE | Kernel PCA
---|---|---|---|---|---|---
|  |  | Linear eval. | Similarity | Linear eval. | Similarity | Linear eval. | Similarity | Linear eval. | Similarity
Class | $\mathbf{n_{abs}}$ | $\mathbf{n_{rel}}$ | Precision | Recall | Overlap | Precision | Recall | Overlap | Precision | Recall | Overlap | Precision | Recall | Overlap
Elliptical | 793 | $62.9\text{\,}\mathrm{\char 37\relax}$ | 0.79 | 0.77 | 0.65 | 0.78 | 0.77 | 0.65 | 0.71 | 0.71 | 0.66 | 0.68 | 0.64 | 0.61
Streak | 228 | $18.1\text{\,}\mathrm{\char 37\relax}$ | 0.91 | 0.90 | 0.66 | 0.89 | 0.86 | 0.67 | 0.83 | 0.72 | 0.61 | 0.46 | 0.47 | 0.57
Bent | 367 | $29.1\text{\,}\mathrm{\char 37\relax}$ | 0.51 | 0.53 | 0.48 | 0.51 | 0.51 | 0.49 | 0.40 | 0.50 | 0.44 | 0.35 | 0.41 | 0.41
Asymmetric | 358 | $28.4\text{\,}\mathrm{\char 37\relax}$ | 0.42 | 0.38 | 0.55 | 0.38 | 0.40 | 0.55 | 0.33 | 0.33 | 0.56 | 0.29 | 0.37 | 0.52
Newton Rings | 78 | $6.2\text{\,}\mathrm{\char 37\relax}$ | 0.32 | 0.34 | 0.51 | 0.24 | 0.25 | 0.48 | 0.06 | 0.04 | 0.48 | 0.09 | 0.21 | 0.44
Double Rings | 38 | $3.0\text{\,}\mathrm{\char 37\relax}$ | 0.34 | 0.40 | 0.45 | 0.34 | 0.40 | 0.43 | 0.05 | 0.04 | 0.43 | 0.06 | 0.22 | 0.41
Layered | 27 | $2.1\text{\,}\mathrm{\char 37\relax}$ | 0.37 | 0.37 | 0.33 | 0.31 | 0.32 | 0.30 | 0.02 | 0.09 | 0.26 | 0.08 | 0.25 | 0.29

It is, therefore, instructive to look at the individual averages for every
class, which are given in table 2. The CPLR method performs significantly
better than the CLR baseline and non-contrastive methods in linear evaluation.
The most significant improvement is with rarely occurring classes that appear
only in $\leq$7\text{\,}\mathrm{\char 37\relax}$$ of all images. VAE and PCA-
based techniques fail entirely to place these diffraction images in a linearly
separable region of the embedding space, resulting in poor _Precision_ and
_Recall_ scores. However, the CLR baseline also yields limited success in the
case of radial symmetry-breaking features like the _Newton Rings_ and
_Layered_ class. There, the diffraction images contain features that either
break radial symmetry (_Layered_) or introduce a second radially symmetric
feature (_Newton Rings_), cf. figure 2 iii), which in combination with a low
class-count brings the CLR method to its limits. The symmetry-breaking
projection of the CPLR method helps in those cases and yields better results
when fewer images are available.

Figure 3:  Qualitative results on the helium nanodroplets dataset. In i) to
iii) the column _Input image_ shows three randomly chosen images - from a pre-
defined class combination. Next to the input images are the four closest
diffraction images according to calculated pairwise distances for the CPLR,
CLR, and VAE method using the metric given in table 1. Every image is
augmented in the top left corner by the class names given in the ground truth
data and - for all images but the input image - the _Overlap_ score with its
corresponding input image.

A qualitative comparison of the CPLR, CLR, and VAE results is given in figure
3 i) to iii). In each plot, the column _Input image_ shows the same three
diffraction images, randomly chosen out of the three classes _Streak_ ,
_Layered_ , and _Double Rings_. Next to each input image are the four
diffraction images belonging to the four closest embeddings in the embedding
space. Additionally, every image is augmented in the top left corner by the
class names given in the ground truth data and, for all images but the input
image, by the _Overlap_ score with its corresponding input image.

The images for the _Streak_ / _Bent_ class combination in the first row show
strong _Overlap_ scores for all three frameworks ($1.00$ for CPLR, $0.92$ for
CLR, and $0.79$ for VAE). However, only the contrastive-based methods placed
those embeddings of images next to each other where the characteristic
_streak_ feature is orientated and elongated differently than in the input
image. We consider this a strength highlighting that both contrastive methods
focus more on the semantics within a diffraction image than the pixel-wise
similarity. This can also be seen in the _Streak_ / _Bent_ / _Layered_ class
combination in the second row of figure 3, where the direction of the
characteristic _Streak_ and _Bent_ features vary substantially in size and
orientation for both CLR-based methods but are identically aligned within the
nearest neighbors of the VAE framework.

As already discussed above, our CPLR method outperforms the baseline CLR
methods, especially in scenarios with low sample counts and symmetry-breaking
features. This behavior is best seen in the second row for the _Streak_ /
_Bent_ / _Layered_ class combination. Only $27$ images in the dataset have a
_Layered_ label assigned to them, and the CLR method fails to learn this
characteristic having an average _Overlap_ score of $0.25$ for this example
and only placing one additional image with a _Layered_ label near the input
image in the embedding space. On the other hand, the CPLR method performs
significantly better, with an average _Overlap_ score of $0.67$ and placing
two images with a _Layered_ label, two with a _Streak_ , and one with a _Bent_
label next to the input image.

This observation also holds for the third example with the _Elliptical_ /
_Double Rings_ example, where the CPLR method reaches an average _Overlap_
score of $0.58$, compared to $0.54$ and $0.38$ for the CLR and the VAE method,
respectively.

These qualitative observations, along with the quantitative results presented
above, show conclusively that the CPLR method is introducing significant
improvements compared to previous methods for finding the semantic similarity
in diffraction images and the baseline CLR method.

##  III Summary and outlook

We have introduced a novel method for finding the semantic similarities in
diffraction images without relying on expert labeling. Based on contrastive
learning (CLR), we introduced contrastive projection learning (CPLR), where
the contrastive learning task is constructed from coordinate- projections of
an input diffraction image and not from the same image as in CLR. This
relatively easy alternation of the learning scenario substantially improves
the quality of the learned embedding space on all metrics and scores. CPLR,
therefore, provides a much-needed pathway for the upcoming big-data challenges
within the coherent diffraction imaging (CDI) community since similarity
calculations are at the core of almost every segmentation, classification, and
clustering algorithm. Consequently, CPLR can be implemented as a stand-in-
replacement for other similarity metrics in all so-far published
classification and clustering approaches for diffraction images, potentially
improving a wide range of long-established working routines in research
groups.

In addition, our method can, theoretically, also be applied to all data that
inherit Polar symmetry, such as in X-ray crystallography.

Our results have the potential to enable multiple future possibilities. For
example, currently, $3$D reconstruction via CDI methods can either be done in
the small-angle regime, where reconstruction by Fourier inversion is possible
[10, 29], using the _Expand-Maximize-Compress_ (EMC) algorithm [74], or in the
wide-angle regime, via a recursive forward-fitting _Multi-Slice-Fourier-
Transform_ (MSFT) approach [10, 56, 29]. In both cases, the similarity between
diffraction images needs to be calculated. As of today, the EMC method can be
applied to datasets on the order of millions of images [26, 15]. However, the
similarity calculation is currently done using the cross-correlation between
radial intensity profile lines of the diffraction images at different angles
[26, 15], which is computationally costly, and, as the authors in [26] pointed
out, may not be sufficient for more complex patterns. As with the MSFT method,
similarity calculations are currently done using either the MSE or even manual
estimation by researchers [10, 12, 29, 75].

Ultimately, CPLR can provide a path to apply the EMC algorithm on more complex
datasets, get better results on simpler datasets, and replace the MSE metric
in MSFT-based approaches.

Furthermore, it enables quick and reliable statistical reasoning on the
variability and occurrence of features within diffraction image datasets, as
was done in [12], for example.

Finally, recent research on contrastive methods in computer vision [76, 43]
promises accuracies comparable to supervised methods or surpassing them and
can be easily implemented into our framework.

Therefore, this manuscript stands as a stepping stone for adapting self-
supervised learning to the domain of diffraction imaging.

All code for the discussed experiments, pretrained models, and extracted
embedding spaces are available at our ETH Gitlab repository [64].

##  IV Methods

Augmentation strategy. A well-defined augmentation strategy is critical in
contrastive learning [16]. As pointed out by [16, 17], the essential parts of
constructing this strategy are random cropping and random color distortion
transformations. The latter is targeted towards histogram and color-channel
correlation-based overfitting of the network. Since diffraction data is
monochrome, we replace the channel-independent RGB distortion with a single-
channel jitter distortion. Furthermore, as in [16, 17], we use a probabilistic
augmentation strategy that includes _Flip_ , _Rotation_ , _Crop & Resize_,
_Jitter_ , _Fill_ , and _Translation_ transformations on all input patches.
However, our _Crop & Resize_ routine is not changing the aspect ratio, as is
usually done in other contrastive learning augmentation pipelines. Changing
the aspect ratio would break the correlation between the _Polar_ and
_Cartesian_ projections. Every transformation has a fixed probability of
$50\text{\,}\mathrm{\char 37\relax}$ for being applied at every invocation. We
implemented the entire pipeline using TensorFlow augmentation layers placed on
the GPU itself. Code is available in the official repository [64].

Training and linear evaluation strategy. The NT-Xent loss that is minimized
during training is given by:

$\displaystyle\mathbf{l}_{i,j}=-\log\frac{\exp\left(\text{sim}\left(\mathbf{z}_{i},\mathbf{z}_{j}\right)/\tau\right)}{\sum^{2N}_{k=1}\mathbf{1}_{[k\neq{i}]}\exp\left(\text{sim}\left(\mathbf{z}_{i},\mathbf{z}_{k}\right)/\tau\right)},$

where
$\text{sim}\left(\mathbf{u},\mathbf{v}\right)=\mathbf{u}^{T}\mathbf{v}/\left(||\mathbf{u}||\,||\mathbf{v}||\right)$
denotes the cosine similarity between two vectors $\mathbf{u}$ and
$\mathbf{v}$, $\mathbf{1}_{[k\neq{i}]}\in$ {$0,1$} is an indicator function
evaluating to $1$ if, and only if, $k\neq{i}$, and $\tau$ denotes a
temperature parameter. We performed extensive hyper-parameter optimization to
obtain the best possible values for the temperature parameter $\tau$, which
are $0.200$ for Polar$\,\Leftrightarrow\,$Polar, $0.200$ for
Cartesian$\,\Leftrightarrow\,$Cartesian, $0.075$ for
Cartesian$\,\Leftrightarrow\,$Polar, $0.100$ for
Polar$\,\Leftrightarrow\,$Cartesian. The results of this hyper-search, as well
as scripts to re-run it, can be found in the official repository [64].

The linear classifier we used for linear evaluation was a single-layer
perceptron with an _inverse-scaling_ learning rate schedule and a l$2$ penalty
of $0.0001$. We used the implementation provided by the _sklearn_ Python
package. Code is available in the official repository [64].

Non-contrastive-based methods. Listed in table 1 are the _Factor-Analysis_
(FA), the _Principal-Component-Analysis_ (PCA), the Kernel-PCA, the _Uniform
Manifold Approximation & Projection_ (UMAP) [69], and the _Variational
Autoencoder_ (VAE) [70, 71] methods. All of these have been used with various
forms of spectrographic image data. FA- and PCA-based methods are parameter-
free dimensionality reduction techniques that are regularly used within all
scientific disciplines; while FA considers the dataset’s variance, PCA
considers the covariance of the data. FA and PCA-based methods have been used
with powder diffraction data [77, 78] and X-ray diffraction phase analysis
[79] and as a dimensionality reduction for subsequent classification [80] and
clustering [22].

A VAE is a generative _variational Bayesian_ model where the input information
is encoded to a low dimensional representation via an encoder function and
then recreated by a decoder function. The loss function, called the _Evidence
lower bound_ , is a lower bound on the marginal likelihood [70]. VAEs have
been used with diffraction images in various tasks, such as anomaly-detection
[81], dimensionality reduction [82], phase reconstruction [83, 84], and
modeling the continuous $3$D shape transition in heterogeneous samples [26].
We train the VAE as described in [71], using the code from [85].

UMAP is a dimensionality reduction technique based on manifold learning and
topological data analysis and has been used with other spectrographic image
data, such as Ronchigrams [86] and Audio spectrograms [87, 88]. We use UMAP
with the default parameters and a fixed integer _random_state_ for
reproducibility.

The size of the low-dimensional representation for all mentioned methods was
set to $1024$, identical to the dimensionality of the CLR-based representation
space.

## References

  * [1] The phrase was coined by Nobel laureate Francis Crick in his book _What Mad Pursuit: A personal view of scientific discovery_ (Pumyang, 1991).
  * [2] Miao, J., Ishikawa, T., Robinson, I. K. & Murnane, M. M.  Beyond crystallography: diffractive imaging using coherent x-ray light sources.  _Science_ 348, 530–535 (2015).
  * [3] Chapman, H. N. & Nugent, K. A.  Coherent lensless x-ray imaging.  _Nat. Photonics_ 4, 833–839 (2010).
  * [4] Seibert, M. M. _et al._ Single mimivirus particles intercepted and imaged with an x-ray laser.  _Nature_ 470, 78–81 (2011).
  * [5] Bostedt, C. _et al._ Clusters in intense FLASH pulses: ultrafast ionization dynamics and electron emission studied with spectroscopic and scattering techniques.  _J. Phys. B At. Mol. Opt. Phys._ 43, 194011 (2010).
  * [6] Loh, N. D. _et al._ Fractal morphology, imaging and mass spectrometry of single aerosol particles in flight.  _Nature_ 486, 513–517 (2012).
  * [7] Xu, R. _et al._ Single-shot three-dimensional structure determination of nanocrystals with femtosecond x-ray free-electron laser pulses.  _Nat. Commun._ 5, 4061 (2014).
  * [8] Gorkhover, T. _et al._ Nanoplasma dynamics of single large xenon clusters irradiated with superintense x-ray pulses from the linac coherent light source free-electron laser.  _Phys. Rev. Lett._ 108, 245005 (2012).
  * [9] Gomez, L. F. _et al._ Helium superfluidity. shapes and vorticities of superfluid helium nanodroplets.  _Science_ 345, 906–909 (2014).
  * [10] Barke, I. _et al._ The 3d-architecture of individual free silver nanoparticles captured by x-ray scattering.  _Nat. Commun._ 6, 6187 (2015).
  * [11] Ekeberg, T. _et al._ Three-dimensional reconstruction of the giant mimivirus particle with an x-ray free-electron laser.  _Phys. Rev. Lett._ 114, 098102 (2015).
  * [12] Langbehn, B. _et al._ Three-Dimensional shapes of spinning helium nanodroplets.  _Phys. Rev. Lett._ 121, 255301 (2018).
  * [13] Tschentscher, T. _et al._ Photon beam transport and scientific instruments at the european XFEL.  _NATO Adv. Sci. Inst. Ser. E Appl. Sci._ 7, 592 (2017).
  * [14] Stohr, J.  Linac coherent light source II (LCLS-II) conceptual design report.  Tech. Rep. SLAC-R–978, SLAC National Accelerator Laboratory (United States). Funding organisation: US Department of Energy (United States) (2011).
  * [15] Ayyer, K. _et al._ 3D diffractive imaging of nanoparticle ensembles using an x-ray laser.  _Optica_ 8, 15 (2021).
  * [16] Chen, X., Fan, H., Girshick, R. & He, K.  Improved baselines with momentum contrastive learning.  _arXiv_ (2020).  eprint 2003.04297.
  * [17] Chen, T., Kornblith, S., Swersky, K., Norouzi, M. & Hinton, G.  Big self-supervised models are strong semi-supervised learners.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , no. Article 1865 in NIPS’20, 22243–22255 (Curran Associates Inc., Red Hook, NY, USA, 2020).
  * [18] We adopt this term from the domain of natural language processing, were it is used to differentiate between the _semantic_ \- contextual - and the _lexicographical_ \- word for word - similarity [89, 90]. The lexicographical measures are in our case pixel-wise or keypoint-based approaches.
  * [19] Compare the term _interference_ in [91]. In Human-Perceptual-Similarity, interference is the distortion that a judged truth of a property - like the length of a line - can exhibit. Meaning, the perceived length of the line is determined not only by its length, but also by its surrounding.
  * [20] Bobkov, S. A. _et al._ Sorting algorithms for single-particle imaging experiments at x-ray free-electron lasers.  _J. Synchrotron Radiat._ 22, 1345–1352 (2015).
  * [21] Zimmermann, J. _et al._ Deep neural networks for classifying complex features in diffraction images.  _Phys Rev E_ 99, 063309 (2019).
  * [22] Yoon, C. H. _et al._ Unsupervised classification of single-particle x-ray diffraction snapshots by spectral clustering.  _Opt. Express_ 19, 16542–16549 (2011).
  * [23] Park, H. J. _et al._ Toward unsupervised single-shot diffractive imaging of heterogeneous particles using x-ray free-electron lasers.  _Opt. Express_ 21, 28729–28742 (2013).
  * [24] Andreasson, J. _et al._ Automated identification and classification of single particle serial femtosecond x-ray diffraction data.  _Opt. Express_ 22, 2497–2510 (2014).
  * [25] Rose, M. _et al._ Single-particle imaging without symmetry constraints at an x-ray free-electron laser.  _IUCrJ_ 5, 727–736 (2018).
  * [26] Zhuang, Y. _et al._ Unsupervised learning approaches to characterizing heterogeneous samples using x-ray single-particle imaging.  _IUCrJ_ 9, 204–214 (2022).
  * [27] Ribeiro, M. T., Singh, S. & Guestrin, C.  “why should I trust you?”: Explaining the predictions of any classifier.  _arXiv_ (2016).  eprint 1602.04938.
  * [28] Barty, A. _et al._ Cheetah: software for high-throughput reduction and analysis of serial femtosecond x-ray diffraction data.  _J. Appl. Crystallogr._ 47, 1118–1131 (2014).
  * [29] Colombo, A. _et al._ The scatman: an approximate method for fast wide-angle scattering simulations.  _J. Appl. Crystallogr._ 55 (2022).
  * [30] Zhuang, C., Zhai, A. L. & Yamins, D.  Local aggregation for unsupervised learning of visual embeddings.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , vol. 2019-Octob, 6002–6012 (Institute of Electrical and Electronics Engineers Inc., 2019).
  * [31] Caron, M. _et al._ Unsupervised learning of visual features by contrasting cluster assignments.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , no. Article 831 in NIPS’20, 9912–9924 (Curran Associates Inc., Red Hook, NY, USA, 2020).
  * [32] Van Gansbeke, W., Vandenhende, S., Georgoulis, S., Proesmans, M. & Van Gool, L.  SCAN: Learning to classify images without labels.  In _Computer Vision – ECCV 2020_ , 268–285 (Springer International Publishing, 2020).
  * [33] Robinson, J. D. _et al._ Can contrastive learning avoid shortcut solutions?  Conference on Neural Information Processing Systems (2021).
  * [34] Chen, X., Hsieh, C.-J. & Gong, B.  When vision transformers outperform ResNets without pre-training or strong data augmentations.  In _International Conference on Learning Representations_ (2022).
  * [35] The dataset can be downloaded at: https://www.cxidb.org/id-94.html.
  * [36] van den Oord, A., Li, Y. & Vinyals, O.  Representation learning with contrastive predictive coding.  _arXiv_ (2018).  eprint 1807.03748.
  * [37] The structural similarity index (SSIM) [65], the correlation of the principal components, the mean-squared-error (MSE), and the keypoint-based scale-invariant feature transform (SIFT) distance, calculated using the ratio test as in [67].
  * [38] Computer vision with diffraction images is the domain in our case.
  * [39] A similarity analysis is the task in our case.
  * [40] For example, in computer vision, a neural network can extract more information from an image if it tries to predict masked parts of that image first before it tries to predict the image’s ground truth label. For a review on self-supervised learning, see [41].
  * [41] Liu, X. _et al._ Self-supervised learning: Generative or contrastive.  _IEEE Trans. Knowl. Data Eng._ 1–1 (2021).
  * [42] Becker, S. & Hinton, G. E.  Self-organizing neural network that discovers surfaces in random-dot stereograms.  _Nature_ 355, 161–163 (1992).
  * [43] Tomasev, N. _et al._ Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?  _arXiv_ (2022).  eprint 2201.05119.
  * [44] Al-Tahan, H. & Mohsenzadeh, Y.  CLAR: Contrastive learning of auditory representations.  _arXiv_ (2020).  eprint 2010.09542.
  * [45] Wang, L. & van den Oord, A.  Multi-Format contrastive learning of audio representations.  _arXiv_ (2021).  eprint 2103.06508.
  * [46] Saeed, A., Grangier, D. & Zeghidour, N.  Contrastive learning of General-Purpose audio representations.  In _ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 3875–3879 (2021).
  * [47] Liu, Y., Wang, K., Liu, L., Lan, H. & Lin, L.  TCGL: Temporal contrastive graph for Self-Supervised video representation learning.  _IEEE Trans. Image Process._ 31, 1978–1993 (2022).
  * [48] Dave, I., Gupta, R., Rizve, M. N. & Shah, M.  TCLR: Temporal contrastive learning for video representation.  _Comput. Vis. Image Underst._ 219, 103406 (2022).
  * [49] Pan, T., Song, Y., Yang, T., Jiang, W. & Liu, W.  VideoMoCo: Contrastive video representation learning with temporally adversarial examples.  _arXiv_ 11200–11209 (2021).  eprint 2103.05905.
  * [50] Gao, T., Yao, X. & Chen, D.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 6894–6910 (Association for Computational Linguistics, Stroudsburg, PA, USA, 2021).
  * [51] Rethmeier, N. & Augenstein, I.  A primer on contrastive pretraining in language processing: Methods, lessons learned and perspectives.  _arXiv_ (2021).  eprint 2102.12982.
  * [52] Sohn, K.  Improved deep metric learning with multi-class n-pair loss objective.  vol. 29, 1857–1865 (Curran Associates, Inc., 2016).
  * [53] He, T. _et al._ Bag of tricks for image classification with convolutional neural networks.  _arXiv_ (2018).  eprint 1812.01187.
  * [54] Allaria, E. _et al._ Highly coherent and stable pulses from the FERMI seeded free-electron laser in the extreme ultraviolet.  _Nat. Photonics_ 6, 699–704 (2012).
  * [55] Bostedt, C. _et al._ Ultrafast x-ray scattering of xenon nanoparticles: imaging transient states of matter.  _Phys. Rev. Lett._ 108, 093401 (2012).
  * [56] Rupp, D. _et al._ Coherent diffractive imaging of single helium nanodroplets with a high harmonic generation source.  _Nat. Commun._ 8, 493 (2017).
  * [57] See the section _Size distribution of helium nanodroplets_ in the supplemental material of [12].
  * [58] Zagoruyko, S. & Komodakis, N.  Wide residual networks.  In Richard C. Wilson, E. R. H. & Smith, W. A. P. (eds.) _Procedings of the British Machine Vision Conference 2016_ , 87, 87.1–87.12 (British Machine Vision Association, 2016).
  * [59] Li, X., Wang, W., Hu, X. & Yang, J.  Selective kernel networks.  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , vol. 2019-June, 510–519 (2019).
  * [60] Hu, J., Shen, L., Albanie, S., Sun, G. & Wu, E.  Squeeze-and-Excitation networks.  _arXiv_ 2011–2023 (2017).  eprint 1709.01507.
  * [61] You, Y., Gitman, I. & Ginsburg, B.  Large batch training of convolutional networks.  _arXiv_ (2017).  eprint 1708.03888.
  * [62] Loshchilov, I. & Hutter, F.  SGDR: Stochastic gradient descent with warm restarts.  International Conference on Learning Representations (2017).
  * [63] Goyal, P. _et al._ Accurate, large minibatch SGD: Training ImageNet in 1 hour.  _arXiv_ (2017).  eprint 1706.02677.
  * [64] https://gitlab.ethz.ch/nux/machine-learning/contrastive_projection_learning.
  * [65] Wang, Z., Bovik, A. C., Sheikh, H. R. & Simoncelli, E. P.  Image quality assessment: from error visibility to structural similarity.  _IEEE Trans. Image Process._ 13, 600–612 (2004).
  * [66] Wang, Z. & Simoncelli, E. P.  Translation insensitive image similarity in complex wavelet domain.  In _Proceedings. (ICASSP ’05). IEEE International Conference on Acoustics, Speech, and Signal Processing, 2005._ , vol. 2, ii/573–ii/576 Vol. 2 (2005).
  * [67] Lowe, D. G.  Object recognition from local scale-invariant features.  In _Proceedings of the Seventh IEEE International Conference on Computer Vision_ , vol. 2, 1150–1157 vol.2 (IEEE, 1999).
  * [68] The macroaverage computes the average independently for each class and then takes the global average. Thereby, treating all classes equally. A microaverage aggregates each contribution of all classes first and averages that. For a discussion on macro- and microaveraging see section 13.6 in [92].
  * [69] McInnes, L., Healy, J. & Melville, J.  UMAP: Uniform manifold approximation and projection for dimension reduction.  _arXiv_ (2018).  eprint 1802.03426.
  * [70] Kingma, D. P. & Welling, M.  Auto-Encoding variational bayes.  _arXiv_ (2013).  eprint 1312.6114v10.
  * [71] Burgess, C. P. _et al._ Understanding disentangling in $\beta$-VAE.  _arXiv_ (2018).  eprint 1804.03599.
  * [72] A multinomial distribution where the number of trials ($n$) is equal to the number of possible labels ($n_{\text{label}}$) and with probabilities $p_{n}=\nicefrac{{1}}{{n_{\text{label}}}}$.
  * [73] A categorical distribution with number of categories $k=1$, and with probabilities $p_{n}=\nicefrac{{1}}{{n_{\text{label}}}}$.
  * [74] Loh, N.-T. D. & Elser, V.  Reconstruction algorithm for single-particle diffraction imaging experiments.  _Phys. Rev. E Stat. Nonlin. Soft Matter Phys._ 80, 026705 (2009).
  * [75] Colombo, A. _et al._ Three-Dimensional coherent diffractive imaging of isolated faceted nanostructures.  _arXiv_ (2022).  eprint 2208.04044.
  * [76] Grill, J.-B. _et al._ Bootstrap your own latent a new approach to self-supervised learning.  In _Proceedings of the 34th International Conference on Neural Information Processing Systems_ , no. Article 1786 in NIPS’20, 21271–21284 (Curran Associates Inc., Red Hook, NY, USA, 2020).
  * [77] Westphal, T., Bier, T. A., Takahashi, K. & Wahab, M.  Using exploratory factor analysis to examine consecutive in-situ x-ray diffraction measurements.  _Powder Diffr._ 30, 340–348 (2015).
  * [78] Chernyshov, D., Dovgaliuk, I., Dyadkin, V. & van Beek, W.  Principal component analysis (PCA) for powder diffraction data: Towards unblinded applications.  _Crystals_ 10, 581 (2020).
  * [79] Camara, A. H.  The importance of factor analysis in quantitative and qualitative x-ray diffraction phase analysis.  _KOM – Corrosion and Material Protection Journal_ 58, 52–58 (2014).
  * [80] Matos, C. R. S., Xavier, M. J., Barreto, L. S., Costa, N. B., Jr & Gimenez, I. F.  Principal component analysis of x-ray diffraction patterns to yield morphological classification of brucite particles.  _Anal. Chem._ 79, 2091–2095 (2007).
  * [81] Banko, L., Maffettone, P. M., Naujoks, D., Olds, D. & Ludwig, A.  Deep learning for visualization and novelty detection in large x-ray diffraction datasets.  _npj Computational Materials_ 7, 1–6 (2021).
  * [82] Ruiz Vargas, J. C. _et al._ Shedding light on variational autoencoders.  In _2018 XLIV Latin American Computer Conference (CLEI)_ , 294–298 (Institute of Electrical and Electronics Engineers Inc., 2018).
  * [83] Yao, Y. _et al._ AutoPhaseNN: Unsupervised physics-aware deep learning of 3D nanoscale bragg coherent diffraction imaging.  _arXiv_ (2021).  eprint 2109.14053.
  * [84] Cherukara, M. J., Nashed, Y. S. G. & Harder, R. J.  Real-time coherent diffraction inversion using deep generative networks.  _Sci. Rep._ 8, 16520 (2018).
  * [85] https://gitlab.ethz.ch/nux/machine-learning/disentangling-vae.
  * [86] Li, X. _et al._ Manifold learning of four-dimensional scanning transmission electron microscopy.  _npj Computational Materials_ 5, 1–8 (2019).
  * [87] Sainburg, T., Thielk, M. & Gentner, T. Q.  Finding, visualizing, and quantifying latent structure across diverse animal vocal repertoires.  _PLoS Comput. Biol._ 16, e1008228 (2020).
  * [88] Thomas, M. _et al._ A practical guide for generating unsupervised, spectrogram-based latent space representations of animal vocalizations.  _J. Anim. Ecol._ (2022).
  * [89] Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I. & Specia, L.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , 1–14 (Association for Computational Linguistics, Vancouver, Canada, 2017).
  * [90] Chandrasekaran, D. & Mago, V.  Evolution of semantic Similarity—A survey.  _ACM Comput. Surv._ 54, 1–37 (2021).
  * [91] Santini, S. & Jain, R.  Similarity measures.  _IEEE Trans. Pattern Anal. Mach. Intell._ 21, 871–883 (1999).
  * [92] Manning, C. D., Raghavan, P. & Schütze, H.  _Introduction to Information Retrieval_ (Cambridge University Press, 2008).
  * [93] NUX-Noether is a single multi-GPU node with two $16$ Core AMD EPYC 7302 CPUs, 256 GB DDR4 Ram, and four NVIDIA RTX 3090 24GB GPUs.

###  IV.1 Acknowledgments

Excellent support has been provided by the ISG team of DPHYS at ETH Zürich.
Funding is acknowledged from the SNF via Grant No. $200021$E_$193642$ and the
NCCR MUST. Further funding was provided by the Leibniz Society via Grant No.
SAW/$2017$/MBI$4$, and from the DFG via Grant No. MO $719$/$14$-$2$.

###  IV.2 Author contributions

Earlier versions of the contrastive-based diffraction imaging approach were
tested by F.B. and D.G. under supervision of J.Z. and D.R.. Training and
evaluation of all models was written by J.Z. and performed on the _NUX-
Noether_ GPU high-performance-computer at ETH Zürich [93]. B.L. and D.R.
contributed to discussing the results. The manuscript was written by J.Z. with
input from all authors.

###  IV.3 Competing interests

The authors declare no competing financial interests.