additions

[2] [2]

## Abstract

In the biomedical environment, experiments assessing dynamic processes are
primarily performed by a human acquisition supervisor. Contemporary
implementations of such experiments frequently aim to acquire a maximum number
of relevant events from sometimes several hundred parallel, non-synchronous
processes. Since in some high-throughput experiments, only one or a few
instances of a given process can be observed simultaneously, a strategy for
planning and executing an efficient acquisition paradigm is essential. To
address this problem, we present two new methods in this paper. The first
method, Encoded Dynamic Process (EDP), is Artificial Intelligence (AI)-based
and represents dynamic processes so as to allow prediction of pseudo-time
values from single still images. Second, with Experiment Automation Pipeline
for Dynamic Processes (EAPDP), we present a Machine Learning Operations
(MLOps)-based pipeline that uses the extracted knowledge from EDP to
efficiently schedule acquisition in biomedical experiments for dynamic
processes in practice. In a first experiment, we show that the pre-trained
State-Of-The-Art (SOTA) object segmentation method Contour Proposal Networks
(CPN) works reliably as a module of EAPDP to extract the relevant object for
EDP from the acquired three-dimensional image stack.

##  1 Introduction

For the imaging-based assessment of dynamic processes in biomedical settings,
objects of interest must be identified and relevant events that characterize
the dynamic process must be recorded during their time of occurrence.
Commonly, a human operator controls the imaging instrument to examine a
biomedical sample using a microscope and relevant objects are found in the
sample by inspection. Alternatively, the operator estimates for each object of
interest the time at which an event of interest is expected to occur based on
previous experience and triggers the recording of the event at that time.
Nevertheless, many contemporary experiments provide several hundred relevant
objects that can, in principle, be imaged in parallel. Events of interest,
however, are non-synchronous and the estimation of future event times requires
extensive human effort, is prone to error, and not necessarily time-efficient.
These obstacles can result in unnecessarily large amounts of irrelevant data,
unnecessary experimental repeats, or experimental biases inflicted by
additional light exposure of the sample [18]. To address these obstacles, we
present two new methods for the automated, real-time planning and execution of
such experiments.

####  EDP.

The traditional method of capturing all data or relying on human experience to
predict future events is outdated and inefficient. Instead of relying on human
experience, an accurate and comprehensive model of the dynamic process should
be created. This model should be capable of uniquely identifying a relevant
object state through a process known as fingerprinting, similar to how humans
do it. In biomedicine, it is crucial that this fingerprint remains consistent
despite contextual changes such as noise, brightness changes, or affine
transformations. By modeling the relationship between the fingerprints, the
dynamic process can be represented orderly. This representation suits as an
approximation of relative progress within the dynamic process. This relative
progress can also be interpreted as a relative time, known as pseudo-time. By
adopting this method, we can achieve efficiency in predicting future events
and a deeper understanding of the dynamic process.

####  EAPDP

To unlock the full potential of the EDP, a well-designed pipeline is an
absolute must. Such a pipeline should be able to recognize specific states in
the real world, identify relevant objects, and then calculate a pseudo-time
for those states using the EDP. Armed with this knowledge, the pipeline can
automatically plan and execute a new state capture for any significant event
that occurs. Due to the uncertain nature of the EDP predictions, the pipeline
must be able to respond to unsuccessful recordings and learn from them. This
is where MLOps [1] comes in. By retraining an existing production model in
accordance with the live context, MLOps ensures that the pipeline always uses
a current and accurate model, resulting in a potentially better outcome in
real-world experiments.

##  2 Related Work

#### Object Extraction.

Basically, there are different possibilities in Computer Vision (CV) like
object detection and object segmentation, to identify individual objects in an
image [10] and thus extract them. In the biomedical context, many methods
mostly focus on segmentation [42] with SOTA methods like StarDist [38] and CPN
[46].

#### Pseudo-time predictions.

A first approach for pseudo-time predictions with classical, non Deep Learning
(DL) methods was presented in [14]. For extracting relevant objects from the
acquired image, thresholding is used as a classical CV segmentation method.
Then the object’s fingerprint is generated linearly with a Principal Component
Analysis (PCA) [19, 20]. However, biological processes are usually not linear
[5]. Therefore, recently, non-linear encodings of the dynamic processes using
DL methods have become popular [17, 9, 21, 32]. For example, [21] and [9]
present DL approaches to encode cell cycles and derive predefined cell phases.
However, this classification-based approach does not allow for deriving
continuous relations like the pseudo-time to each other directly. This
continuous relation was modeled with DeepCycle in [32]. The training of
DeepCycle is performed supervised. For this purpose, virtual labels are
calculated based on the fluorescence intensity in specifically labeled
channels of the cells. These classes can then be used as anchor points during
training to determine a (relative) cell state as a pseudotime. It is important
to note that the assumption that a correlation of fluorescence intensity to
cell phase can be used is not always true in the biological context. A DL
method that follows a comparable pseudo-time approach to the given
constraints, in this paper, was presented in [17]. In [17], an AutoEncoder
(AE) approach for pseudo-time approximation is used as a Self-Supervised
Learning (SSL) approach. A DL model is used as the Variational AutoEncoder
(VAE) [23] encoding, from whose Hierarchical Agglomerative Clustering (HAC)
and Minimum Spanning Tree (MST) code the pseudo-time is then determined.
However, this approach also has a few limitations. First, a recording
necessarily contains exactly one relevant object in one acquired image.
Second, the entire dataset was acquired under comparable acquisition
conditions, which also only contain identical positioned and oriented objects
and are not able to learn affine transformations [3] between objects. Both
constraints are generally not satisfied for microscopic images, such as in
[27, 36]. Furthermore, this pseudo-time method was not designed as an End-to-
end (E2E) model, which deprives the DL model of the ability to internally bind
affine transformed objects.

#### AutoEncoder.

Autoencoders are SSL methods to learn a representation from a given
suitability, such as an image [49, 47]. For example, the autoencoder can be
represented by a Conventional AutoEncoder (CAE) [49] and/or a VAE [23].
Especially recently, Masked AutoEncoders (MAEs) [15] have become more popular
than CAE because of their ability for a better visual representation learning
[49], either using a transformer-based approach [15] or the Convolutional
Neural Network (CNN)-based approach [47]. However, since in [47] the higher
efficiency of ConvNeXt V2 is shown, this model is chosen for this work. In
addition to the pure learning of a visual representation in the form of a
fingerprint, relations between the fingerprints can also be learned, e.g. with
VAE [17].

####  SSL

To train the DL model in a supervised manner, labeled data are generally rare
in the biomedical domain [13]. There are DataBases (DBs) like BioMedImage.io
[30] or several challenges with own datasets [26, 2, 28]. However, especially
for biological datasets with sometimes hundreds of relevant objects in an
image, the datasets are often limited to the 2D case. Furthermore, in the
context of this work, labeling the relevant events with pseudo-time stamps is
only approximate, demanding, error-prone and time-consuming. Therefore,
unsupervised or SSL methods are often used in the biomedical context [43].
Thereby, Active Learning (AL) [41] is used to selectively integrate expert
knowledge into the learning process. Since AL aims to keep the number of
interactions to a minimum [7, 33], data-efficient learning is preferable. For
example, existing datasets from a related context can be identified and
leveraged to train more robust models in transfer learning [11, 29, 48]. In
order to be able to use possibly directly existing pre-trained models from
similar contexts, a new concept was developed in [48].

##  3 Methodology

###  3.1 EDP

For the modeling of the dynamic process, a new concept is introduced with EDP.
The new concept of EDP is based on an AE and is visualized in Figure 1.

Figure 1: Visualization of the EDP model. A 3D object is transformed by an
encoder into a fingerprint (like MAE) and into a relation representation
between the fingerprints (like VAE). The example object is a recording of the
DNA channel of a nucleus from an internal zebrafish embryo dataset. A scale
bar of $2\mu m$ is indicated at the bottom of the input/output image.

The basic idea of the new concept is to separate the generation of the
fingerprint from the learning of the relation as a representation between all
states. The fingerprint generation is done using a MAE as an evolution of CAE.
Specifically, the SOTA MAE-based method ConvNeXt V2 is chosen. During the
learning process, a maximum recovery of the encoded image is aimed in
accordance with SSL. According to the challenge posed by biological objects, a
context-independent representation is required. For this purpose, the images
can be modified during the learning process through Data Augmentation [16]
techniques like Rotations, reflections, contrast adjustments or noise
additions.
In addition to the fingerprint, the relation must also be learned as the
actual modeling of the dynamic process. For this purpose, a VAE-like modeling
is used by learning the uncertainty $v$ in addition to the circle angle
$\alpha$. The assumption is that objects succeeded each other in the dynamic
process with the relative distance $t$ corresponding to this relative distance
and differ in the same ratio in the circle representation. Such an exemplary
circle representation is shown in Figure 2 using a cell division process of
the zebrafish embryo. The state of the cell after cell division is visualized
at 00 o’clock and up to the state of the cell just before cell division at 11
o’clock. This corresponds to a relative distance of $\sim$0.92 (normalized
between [0,1)). This temporal difference must also be valid in reality for the
temporal distance according to the model statement.

Figure 2: Example of a 2D feature space representation for an encoded dynamic
process. Each point represents an encoded image. The circle serves as an
estimation for the positioning of points in its vicinity. It’s worth noting
that due to the presence of uncertainty, the points may not always be
precisely on the circle, but rather in its proximity. For the seven red dots,
example images of cell nuclei from zebrafish embryos at various stages of cell
division are shown. A scale bar of $2\leavevmode\nobreak\ \mu m$ is indicated
at the bottom of each example image. The images are from an internal dataset.

###  3.2 EAPDP

The new EDP module is integrated as a module into the new MLOps-based pipeline
EAPDP. The pipeline concept is visualized in Figure 3 and contains nine other
modules besides the EDP module. Each of these ten modules is briefly described
below. The explanation of the modules and their relationships to each other is
based on the pipeline visualization of Figure 3.

Figure 3: MLOps pipeline with the new EDP module. AI-based MLOps modules are
marked with a green background, non AI-based ones with a blue background.
Additionally, all modules marked with a red border must be newly developed or
only partially adapted from existing methods.

#### Microscope setup.

In the EAPDP, the microscope is used as an actuator to the real-world
environment represented by a biomedical sample. For this purpose, all
microscope components relevant to image acquisition and the microscope
accessories, such as lasers in the case of a laser scanning microscope, must
be controllable via appropriate interfaces of the specific microscope setup.
In addition, the microscope must be able to react on given commands like image
acquisition or requested meta-information like the objective position in a
standardized way.

#### Image Pre-processing.

To optimize the analysis of dynamic processes in the biomedical environment,
the raw images acquired through experiments must be pre-processed according to
the microscope setup and the context of the targeted event. This may involve
methods such as cropping, contrast adjustment, or denoising. Various
libraries, such as Albumentations [4], offer pre-processing methods that can
be used to improve the quality of the images and optimize their analysis by
other modules in the machine learning operation pipeline.

#### Object Extraction.

During an acquisition, the relevant object and the surrounding context are
captured. In order to better analyze the object, it is necessary to separate
it from the surrounding context. The extraction from the whole image is done
via pre-trained segmentation methods. To find a suitable method, we compare
the SOTA cell segmentation algorithms StarDist and CPN using a microscopic
dataset in a first experiment. Importantly, the actual pseudo-time
determination cannot be performed if both methods’ segmentation is
insufficient. Therefore, this submodule is of particular importance. Because
(well) labeled data are generally scarce in the biological context, this work
evaluates generalization performance during inference with already pre-trained
models on new, unknown images. Since there are only pre-trained weights for 2D
segmentation for both methods, the dataset was split into 2D images along the
z-axis.

####  EDP.

The EDP module gets the extracted object and should pass the pseudo-time to
the experiment planner. In order to do this, the module is equipped by the
Experiment planner before with the appropriate experiment setup. With the
setup, the EDP model can then query according to its existing knowledge like
pre-trained models in the context of data-efficient learning. If no weights
are available, training can also be done with/without AL as specified by the
expert. After successful training, the model is passed to the DB with the
appropriate required metadata for possible further use. Then, when the
inference with the original extracted relevant object has been determined, the
results are passed to the expert planner accordingly. The recorded inference
image is also sent to the Data-efficient Learning module and stored in its DB.

#### Experiment planner.

The Experiment planner is the central module of the experiment automatization.
As input it gets the pseudo-times for the recorded objects. Based on the
experiment context, including interesting events, the Experiment planner can
plan future experiments with utmost precision. Once the plan is set, the
Experiment planner gives the microscope the command to ensure that the image
captures the object’s state at the right time, leaving no room for errors.
Additionally, it can query the state of the microscope to ensure that there
was no hardware drift, such as when moving to the object position. All the
information about the experiment’s state is then passed on to the User
Interface (UI), ensuring that all aspects of the experiment are under control.

####  UI.

The UI is the interface between the expert and the MLOps pipeline. On the one
hand, simple interactions can be provided, such as displaying meta-information
or adapting the experimental context, e.g. the cell classes that occur. On the
other hand, much more complex interactions such as result justifications of DL
models can be represented through Explainable Artificial Intelligence (XAI)
[34] or expert knowledge can be brought into the pipeline within the context
of AL. With XAI, the expert should be able to understand better the processes
in the DL models used and why decisions were made, e.g. for event detection.
This helps the expert to eliminate potential errors like unfavorable
experiment settings at an early stage. Such XAI methods can be realized using
a library like PyTorch Captum [25]. For AL, only if the expert can capture the
actual state in the best possible way, the expert can transfer his domain
knowledge to the method in the best possible way and support the method. For
example, points, boxes, or entire segmented regions can be passed to the
method as hints. For this purpose, a custom segmentation module can be
developed based on exiting AL labeling platforms like ObiWanMicrobi [40] or
Karlsruhe Image Data Annotation (KaIDA) [37].

#### Expert (Domain) Knowledge.

The domain knowledge contributed by the expert to the MLOps pipeline can take
several forms. For example, the context of the experiment with a specific cell
class can improve a more efficient event detection module. Furthermore,
knowledge can be injected, e.g. by labeling in the context of AL. For this,
the expert must ensure the quality of the injected domain knowledge with
maximum correctness. Incorrect information can affect the learning processes
in the network.

#### Data-efficient Learning.

To minimize AL interactions with the human expert, as much existing knowledge
as possible is reused. To this end, building on [48], a new AE-based
fingerprinting approach for datasets and Machine Learning (ML) models is being
implemented to reuse as much knowledge as possible. For this, from a DB,
context-given requirements can query existing knowledge. If no data is
available, it can be created synthetically e.g. with biophysical simulations
like Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) [44].

#### Microscope control.

In order for the planned experiment to be automated and performed in real-
time, a corresponding software library is needed to control the microscope.
Since the first release in 2010, $\mu$Manager has been used for this purpose
as one of the SOTA open-source solutions [6, 22, 31, 45]. Therefore, this is
also used in this work.

###  3.3 Exemplary Use Cases

Example use cases for the presented EAPDP with the EDP are presented using
record extracts in Figure 4 below. A first use case is shown in Figure 4(a)
and represents the temporal sorting of RiboNucleic Acid (RNA) Polymerase II
(Pol II) clusters that occur in the nuclei of pluripotent zebrafish embryos. A
method for this use case has already been presented in [14]. A comparison of
the pipeline based on classical ML methods with our DL-based EDP method allows
a direct statement about limitations or improvements of our approach. The
second use-case in Figure 4(b) is the recording of cell divisions in
pluripotent zebrafish embryos, where the time of reaching a new division stage
and thus the regions of an event of interest need to be extrapolated. A final
biological application from the field of microbiology is presented in Figure
4(c). In this example, one interesting event could be the state at which $n$
microbes reach the recording region. For this purpose, a modeling of the cell
division process with EDP can be used to plan the experiment accordingly and
automatically record the event of interest at a time $t$. The modeling of the
cell division process with EDP can be used for this purpose.

(a) [27]

(b) Internal dataset

(c) [39]

Figure 4: Three exemplary images from biological datasets for dynamic
processes. Figure 4(a) shows cell nuclei of zebrafish embryos with marked Pol
II Ser5P clusters. Figure 4(b) shows the DNA of zebrafish embryos nuclei. In
these first two images, a scale bar of $20\mu m$ is shown in the lower left.
The last Figure 4(c) shows a microbial cell division state.

In addition to these biological use cases, other use cases are also possible,
e.g. in medicine. For example, by modeling a tumor accordingly, a prediction
can be made about the relative stage. Consequently, a therapy concept such as
surgery or medication can be tailored to the patient.

##  4 Experiments

The comparison of segmentation algorithms was performed on Helmholtz AI
COmpute REssources (HAICORE) resources equipped with Intel Xeon Platinum 8368
Central Processing Units (CPUs) and an Nvidia A100-40 Graphics Processing Unit
(GPU) [24]. The operating system utilized was Red Hat Enterprise Linux (RHEL)
version 8.6.

###  4.1 Dataset

The internal microscope dataset from Figure 4(b) is used to compare the
segmentation algorithms. This dataset was chosen over the other two example
datasets from Figures 4(a) and 4(c) because of the challenging, frayed
structure of the nuclei as the relevant image object. This is because the
fibrillar structure of the nuclei sometimes deviates strongly from their
typical ellipsoidal shape as in Figure 4(a) due to individually advanced
cytokinesis. This poses a challenge because contiguous pixel regions are not
trivially identifiable and correct boundary segmentation is a challenge. With
microbeSEG [35], a working SOTA solution for microbes like in Figure 4(c) also
already exists.

For the dataset in Figure 4(b), zebrafish embryo DNA was imaged. DNA was
stained with 1:10000 5’-TMR Hoechst in TDE or glycerol. Confocal z-sections
were obtained using a commercial instant SIM microscope (iSIM, VisiTech). A
Nikon 100x oil immersion objective (NA 1.49, SR HO Apo TIRF 100xAC Oil) and a
Hamamatsu ORCA-Quest camera were used for image acquisition. In accordance
with a common problem in biology, no labels exist for this dataset. According
to the desired 2D segmentation, the 3D images are split into 2D images along
the z-axis.

###  4.2 Object extraction

These 2D images were then segmented using each of the two methods. In the
following, the results are evaluated qualitatively because of the non-existent
labels. Therefore, the results are shown in Figure 5. The comparison of the
original image in Figure 5(a) with the StarDist prediction in Figure 5(b),
shows that StarDist cannot well segment semantically related objects as the
nucleus in the upper area. For the method designs with center prediction,
StarDist focused primarily on segmenting ellipsoidal objects from [12] and was
trained only on these. The cell detection was designed to be more flexible and
additionally trained on a more heterogeneous set of non-elliptical cells such
as MCF7 from the dataset [8]. This leads to better generalization and results
in qualitatively evaluated good initial segmentation performance on this most
challenging of our datasets from Figure 4.

(a) Original

(b) StarDist [38]

(c) CPN [46]

Figure 5: Comparison of segmentation predictions for the two SOTA methods
StarDist [38] and CPN [46]. Figure 5(a) represents the original image,
duplicated from Figure 4(b). In Figure 5(b) and Figure 5(c), the predictions
of pre-trained StarDist or CPN are then shown. The predictions are highlighted
differently for better visual differentiation depending on the method used.

Thus, we could show that CPN is a good pre-trained SOTA approach for
extracting the relevant objects from the 2D decomposition. The 2D
segmentations can be reassembled back to 3D segmentations in post-processing,
e.g. using Nearest Neighbor. Based on this, the further submodules of EDP can
be developed in future work and the presented MLOps pipeline can be built upon
it.

##  5 Conclusion and Further Work

In this work, we motivated that due to the large number of parallel non-
synchronous dynamic processes, a novel concept for automated planning and
execution of two novel DL-based approaches is essential. First, the EDP was
introduced to model dynamic processes and derivate a pseudo-time for a given
object state. The pseudo-time prediction can then be used with the EAPDP for
real-time experiment automation. We explained the EDP realized within the
MLOps pipeline by an AE and trained using SSL with AL. At the same time, the
key advantage of higher execution speed and lower human cost while minimizing
user interactions with data-efficient learning was highlighted. Finally, as a
first Proof of Concept (PoC), we showed the necessary pre-processing step for
the EDP to extract the relevant objects based on good inference results of
CPN.

However, the lack of pre-trained weights for 3D segmentation was a drawback of
the segmentation experiments. However, since the fragmented objects are
partially reconnected along the z-axis, this could simplify the problem and
improve accuracy. This will be done as soon as appropriate weights are
available. In addition, a suitable affine-invariant 3D AE needs to be
developed for use within the EDP method. In this context, further research is
needed to investigate whether the ConvNeXt V2 is suitable for 3D segmentation,
also from an efficiency perspective. Of course, the modules of the MLOps
pipeline must be implemented accordingly.

## Acknowledgments

This project is funded by the Helmholtz Association under the Program Natural,
Artificial and Cognitive Information Processing (NACIP) and the Helmholtz
Association’s Initiative & Networking Fund through Helmholtz AI. All
experiments were performed on the HAICORE. We sincerely thank all of them for
supporting our research.

The authors have accepted responsibility for the entire content of this
manuscript and approved its submission. We describe the individual
contributions of N. Friederich (NF), A. Yamachui Sitcheu (AYS), O. Neumann
(ON), S. Eroglu-Kayıkçı (SEK), R. Prizak (RP), L. Hilbert (LH), R. Mikut (RM):
Conceptualization: NF, LH, RM; Methodology: NF, LH, RM; Software: NF;
Investigation: NF; Resources: SEK, RP, LH; Writing – Original Draft: NF;
Writing – Review & Editing: NF, AYS, ON, SEK, RP, LH, RM; Supervision: LH, RM;
Project administration: RM; Funding Acquisition: LH, RM

## References

  * [1] S. Alla, S. K. Adari, S. Alla, and S. K. Adari.  What is MLOps?  Beginning MLOps with MLFlow: Deploy Models in AWS SageMaker, Google Cloud, and Microsoft Azure, pages 79–124, 2021.
  * [2] M. Aubreville, C. Bertram, K. Breininger, S. Jabari, N. Stathonikos, and M. Veta.  MItosis DOmain Generalization Challenge 2022.  https://doi.org/10.5281/zenodo.6362337, 03 2022.
  * [3] E. Begelfor and M. Werman.  Affine Invariance Revisited.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 2087–2094, 2006.
  * [4] A. V. Buslaev, A. Parinov, E. Khvedchenya, V. I. Iglovikov, and A. A. Kalinin.  Albumentations: fast and flexible image augmentations.  CoRR, abs/1809.06839, 2018.
  * [5] J. Carballido-Landeira and B. Escribano.  Nonlinear dynamics in biological systems.  Springer, 2016.
  * [6] C. Conrad, A. Wünsche, T. H. Tan, J. Bulkescher, F. Sieckmann, F. Verissimo, A. Edelstein, T. Walter, U. Liebel, R. Pepperkok, et al.  Micropilot: automation of fluorescence microscopy–based imaging for systems biology.  Nature Methods, 8(3):246–249, 2011.
  * [7] A. M. Das, G. Bhatt, M. M. Bhalerao, V. R. Gao, R. Yang, and J. Bilmes.  Continual Active Learning.  https://openreview.net/forum?id=GC5MsCxrU-, 2023.
  * [8] C. Edlund, T. R. Jackson, N. Khalid, N. Bevan, T. Dale, A. Dengel, S. Ahmed, J. Trygg, and R. Sjögren.  LIVECell—A large-scale dataset for label-free live cell segmentation.  Nature Methods, 18(9):1038–1045, 2021.
  * [9] P. Eulenberg, N. Köhler, T. Blasi, A. Filby, A. E. Carpenter, P. Rees, F. J. Theis, and F. A. Wolf.  Reconstructing cell cycle and disease progression using deep learning.  Nature Communications, 8(1):463, 2017.
  * [10] X. Feng, Y. Jiang, X. Yang, M. Du, and X. Li.  Computer vision algorithms and hardware implementations: A survey.  Integration, 69:309–320, 2019.
  * [11] P. Godau and L. Maier-Hein.  Task Fingerprinting for Meta Learning inBiomedical Image Analysis.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2021: 24th International Conference, Strasbourg, France, September 27–October 1, 2021, Proceedings, Part IV 24, pages 436–446. Springer, 2021.
  * [12] A. Goodman, A. Carpenter, E. Park, jlefman nvidia, Josette-BoozAllen, Kyle, Maggie, Nilofer, P. Sedivec, and W. Cukierski.  2018 Data Science Bowl.  https://kaggle.com/competitions/data-science-bowl-2018, 2018.
  * [13] H. A. Gündüz, M. Binder, X.-Y. To, R. Mreches, B. Bischl, A. C. McHardy, P. C. Münch, and M. Rezaei.  A self-supervised deep learning method for data-efficient training in genomics.  Communications Biology, 6(1):928, 2023.
  * [14] H. Hajiabadi, I. Mamontova, R. Prizak, A. Pancholi, A. Koziolek, and L. Hilbert.  Deep-learning microscopy image reconstruction with quality control reveals second-scale rearrangements in RNA polymerase II clusters.  PNAS nexus, 1(3):pgac065, 2022.
  * [15] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick.  Masked autoencoders are scalable vision learners.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022.
  * [16] A. Hernandez-Garca and P. König.  Data augmentation instead of explicit regularization.  arXiv preprint arXiv:1806.03852, 2018.
  * [17] J. Hong, S. K. Kang, I. Alberts, J. Lu, R. Sznitman, J. S. Lee, A. Rominger, H. Choi, K. Shi, and A. D. N. Initiative.  Image-level trajectory inference of tau pathology using variational autoencoder for Flortaucipir PET.  European Journal of Nuclear Medicine and Molecular Imaging, 49(9):3061–3072, 2022.
  * [18] J. Icha, M. Weber, J. C. Waters, and C. Norden.  Phototoxicity in live fluorescence microscopy, and how to avoid it.  BioEssays, 39(8):1700003, 2017.
  * [19] J. E. Jackson.  A user’s guide to principal components.  John Wiley & Sons, 2005.
  * [20] I. T. Jolliffe.  Principal component analysis for special types of data.  Springer, 2002.
  * [21] A. Jose, R. Roy, D. Moreno-Andrés, and J. Stegmaier.  Automatic Detection of Cell-cycle Stages using Recurrent Neural Networks.  bioRxiv, pages 2023–02, 2023.
  * [22] R. A. Kellogg, R. Gómez-Sjöberg, A. A. Leyrat, and S. Tay.  High-throughput microfluidic single-cell analysis pipeline for studies of signaling dynamics.  Nature Protocols, 9(7):1713–1726, 2014.
  * [23] D. P. Kingma and M. Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [24] KIT.  HAICORE - Hardware Overview.  https://www.nhr.kit.edu/userdocs/haicore/hardware/, Sep 2022.
  * [25] N. Kokhlikyan, V. Miglani, M. Martin, E. Wang, B. Alsallakh, J. Reynolds, A. Melnikov, N. Kliushkina, C. Araya, S. Yan, and O. Reblitz-Richardson.  Captum: A unified and generic model interpretability library for PyTorch.  CoRR, abs/2009.07896, 2020.
  * [26] K. Lee, H. Byun, and H. Shim.  Cell Segmentation in Multi-modality High-Resolution Microscopy Images with Cellpose.  In Proceedings of The Cell Segmentation Challenge in Multi-modality High-Resolution Microscopy Images, volume 212 of Proceedings of Machine Learning Research, pages 1–11. PMLR, 28 Nov–09 Dec 2023.
  * [27] I. Mamontova, A. Pancholi, R. Prizak, and L. Hilbert.  Microscopy data: interaction of the gene klf2b with RNA polymerase II clusters during early zebrafish embryo development.  https://doi.org/10.5281/zenodo.5268833, 08 2021.
  * [28] M. Maška, V. Ulman, P. Delgado-Rodriguez, E. Gómez-de Mariscal, T. Nečasová, F. A. Guerrero Peña, T. I. Ren, E. M. Meyerowitz, T. Scherr, K. Löffler, et al.  The Cell Tracking Challenge: 10 years of objective benchmarking.  Nature Methods, pages 1–11, 2023.
  * [29] M. Molina-Moreno, M. P. Schilling, M. Reischl, and R. Mikut.  Automated Style-Aware Selection of Annotated Pre-Training Databases in Biomedical Imaging.  In 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI), pages 1–5, 2023.
  * [30] W. Ouyang, F. Beuttenmueller, E. Gómez-de Mariscal, C. Pape, T. Burke, C. Garcia-López-de Haro, C. Russell, L. Moya-Sans, C. de-la Torre-Gutiérrez, D. Schmidt, et al.  BioImage Model Zoo: A Community-Driven Resource for Accessible Deep Learning in BioImage Analysis.  bioRxiv, pages 2022–06, 2022.
  * [31] H. Pinkard, N. Stuurman, K. Corbin, R. Vale, and M. F. Krummel.  Micro-Magellan: open-source, sample-adaptive, acquisition software for optical microscopy.  Nature Methods, 13(10):807–809, 2016.
  * [32] L. Rappez, A. Rakhlin, A. Rigopoulos, S. Nikolenko, and T. Alexandrov.  DeepCycle reconstructs a cyclic cell cycle trajectory from unsegmented cell images using convolutional neural networks.  Molecular Systems Biology, 16(10):e9474, 2020.
  * [33] P. Ren, Y. Xiao, X. Chang, P.-Y. Huang, Z. Li, B. B. Gupta, X. Chen, and X. Wang.  A Survey of Deep Active Learning.  ACM Computing Surveys (CSUR), 54(9):1–40, 2021.
  * [34] W. Saeed and C. Omlin.  Explainable AI (XAI): A systematic meta-survey of current challenges and future opportunities.  Knowledge-Based Systems, 263:110273, 2023.
  * [35] T. Scherr, J. Seiffarth, B. Wollenhaupt, O. Neumann, M. P. Schilling, D. Kohlheyer, H. Scharr, K. Nöh, and R. Mikut.  microbeSEG: A deep learning software tool with OMERO data management for efficient and accurate cell segmentation.  PLOS ONE, 17(11):1–14, 11 2022.
  * [36] T. Scherr, J. Seiffarth, B. Wollenhaupt, O. Neumann, M. P. Schilling, D. Kohlheyer, H. Scharr, K. Nöh, and R. Mikut.  microbeSEG dataset.  https://doi.org/10.5281/zenodo.6497715, 04 2022.
  * [37] M. P. Schilling, S. Schmelzer, L. Klinger, and M. Reischl.  KaIDA: a modular tool for assisting image annotation in deep learning.  Journal of Integrative Bioinformatics, 19(4):20220018, 2022.
  * [38] U. Schmidt, M. Weigert, C. Broaddus, and G. Myers.  Cell Detection with Star-Convex Polygons.  In Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II, pages 265–273, 2018.
  * [39] J. Seiffarth, L. Blöbaum, K. Löffler, T. Scherr, A. Grünberger, H. Scharr, R. Mikut, and K. Nöh.  Data for - Tracking one in a million: Performance of automated tracking on a large-scale microbial data set.  https://doi.org/10.5281/zenodo.7260137, 10 2022.
  * [40] J. Seiffarth, T. Scherr, B. Wollenhaupt, O. Neumann, H. Scharr, D. Kohlheyer, R. Mikut, and K. Nöh.  ObiWan-Microbi: OMERO-based integrated workflow for annotating microbes in the cloud.  bioRxiv, pages 2022–08, 2022.
  * [41] B. Settles.  Active Learning Literature Survey.  University of Wisconsin-Madison Department of Computer Sciences, 2009.
  * [42] P. Shrestha, N. Kuang, and J. Yu.  Efficient end-to-end learning for cell segmentation with machine generated weak annotations.  Communications Biology, 6(1):232, 2023.
  * [43] R. Sun, J. Wu, Y. Miao, L. Ouyang, and L. Qu.  Progressive 3D biomedical image registration network based on deep self-calibration.  Frontiers in Neuroinformatics, 16:932879, 2022.
  * [44] A. P. Thompson, H. M. Aktulga, R. Berger, D. S. Bolintineanu, W. M. Brown, P. S. Crozier, P. J. in ’t Veld, A. Kohlmeyer, S. G. Moore, T. D. Nguyen, R. Shan, M. J. Stevens, J. Tranchida, C. Trott, and S. J. Plimpton.  LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales.  Comp. Phys. Comm., 271:108171, 2022.
  * [45] S. Tosi, A. Lladó, L. Bardia, E. Rebollo, A. Godo, P. Stockinger, and J. Colombelli.  AutoScanJ: A Suite of ImageJ Scripts for Intelligent Microscopy.  Frontiers in Bioinformatics, 1:627626, 2021.
  * [46] E. Upschulte, S. Harmeling, K. Amunts, and T. Dickscheid.  Contour proposal networks for biomedical instance segmentation.  Medical Image Analysis, 77:102371, 2022.
  * [47] S. Woo, S. Debnath, R. Hu, X. Chen, Z. Liu, I. S. Kweon, and S. Xie.  ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133–16142, 2023.
  * [48] A. Yamachui Sitcheu, N. Friederich, S. Baeuerle, O. Neumann, M. Reischl, and R. Mikut.  MLOps for Scarce Image Data: A Use Case in Microscopic Image Analysis.  In Proceedings 33. Workshop Computational Intelligence, volume 1, 11 2023.  Submitted manuscript.
  * [49] C. Zhang, C. Zhang, J. Song, J. S. K. Yi, K. Zhang, and I. S. Kweon.  A survey on masked autoencoder for self-supervised learning in vision and beyond.  arXiv e-prints, pages arXiv–2208, 2022.


# Generative adversarial networks for data-scarce spectral applications

J. J. García-Esteban  Departamento de Física Teórica de la Materia Condensada
and Condensed Matter Physics Center (IFIMAC), Universidad Autónoma de Madrid,
E-28049 Madrid, Spain  J. C. Cuevas  Departamento de Física Teórica de la
Materia Condensada and Condensed Matter Physics Center (IFIMAC), Universidad
Autónoma de Madrid, E-28049 Madrid, Spain  J. Bravo-Abad  Departamento de
Física Teórica de la Materia Condensada and Condensed Matter Physics Center
(IFIMAC), Universidad Autónoma de Madrid, E-28049 Madrid, Spain

###### Abstract

Generative adversarial networks (GANs) are one of the most robust and
versatile techniques in the field of generative artificial intelligence. In
this work, we report on an application of GANs in the domain of synthetic
spectral data generation, offering a solution to the scarcity of data found in
various scientific contexts. We demonstrate the proposed approach by applying
it to an illustrative problem within the realm of near-field radiative heat
transfer involving a multilayered hyperbolic metamaterial. We find that a
successful generation of spectral data requires two modifications to
conventional GANs:  _(i)_ the introduction of Wasserstein GANs (WGANs) to
avoid mode collapse, and, _(ii)_ the conditioning of WGANs to obtain accurate
labels for the generated data. We show that a simple feed-forward neural
network (FFNN), when augmented with data generated by a CWGAN, enhances
significantly its performance under conditions of limited data availability,
demonstrating the intrinsic value of CWGAN data augmentation beyond simply
providing larger datasets. In addition, we show that CWGANs can act as a
surrogate model with improved performance in the low-data regime with respect
to simple FFNNs. Overall, this work highlights the potential of generative
machine learning algorithms in scientific applications beyond image generation
and optimization.

##  1 Introduction

Machine learning, a rapidly expanding area within computer science, focuses on
advancing the foundations and technology that enables machines to learn from
data [1, 2, 3, 4]. Deep learning, on the other hand, is a subset of machine
learning techniques that uses artificial neural networks (ANNs) to model and
solve complex data-driven problems, such as image and speech recognition [5,
6], natural language processing [7], and autonomous driving [8], among many
others. With the current explosion of data and the rapid development of
improved hardware and algorithms, machine learning and deep learning are
becoming crucial tools in many industries, including healthcare, finance, and
manufacturing.

Motivated by this success, machine learning and deep learning techniques are
attracting increasing attention from a variety of scientific disciplines
beyond computer science, revolutionizing traditional approaches to the
modeling and analysis of data-driven scientific problems. In physics, these
techniques are employed to tackle complex problems [9], including the
representation of quantum many-body wave functions [10], the discovery and
identification of phase transitions in condensed-matter systems [11, 12, 13],
the solution of statistical problems [14], the development of novel quantum
information technologies [15], the modeling of gravitational waves [16], and
the design of nanophotonic devices with novel or improved functionalities [17,
18, 19, 20]. Machine learning algorithms have been effectively used for the
accelerated discovery and design of new materials and molecules in the fields
of materials science and chemistry [21, 22], being instrumental in molecular
dynamics simulations [23], in predicting chemical reactions [24], and in
modeling the quantum mechanical energies of molecules [25]. In the realm of
biology, the applications of machine learning and deep learning are also vast,
including breakthroughs in gene expression prediction tasks, prediction of
micro-RNAs targets, and novel single-cell methods [26]. Overall, despite
significant challenges, such as the interpretability and transferability,
machine learning and deep learning are playing an increasingly pivotal role in
the advance of a broad variety of scientific research methodologies.

In this context, generative adversarial networks (GANs), a powerful subset of
generative machine learning, have emerged as a versatile tool for creating new
data instances that closely resemble a given training set [27]. This
innovative paradigm involves a two-player adversarial setup where a generative
ANN strives to produce data instances that are indistinguishable from the
training set, while a discriminative network attempts to distinguish between
the instances generated by the generative network and the real data [28]. The
competing nature of these two networks drives the generative network to
generate increasingly realistic data, pushing the boundaries of what is
achievable with generative models. This technology has been instrumental
across a broad range of scientific disciplines, including physics, chemistry,
and biology. In physics, GANs have been used for simulating complex systems
and predicting outcomes of experiments, with examples in high energy physics
[29], condensed matter physics [30, 31, 32], nanophotonics [33, 34], and
cosmology [35]. In the field of chemistry, GANs have been harnessed to
generate novel chemical structures and predict their properties [36], thereby
accelerating the process of drug discovery and materials design [37]. In
biology, GANs have been employed in a variety of tasks, including protein
engineering [38] and generate biological imaging data [39]. This myriad of
applications demonstrate the significant potential of GANs in transforming
scientific research by providing a powerful tool for hypothesis generation,
experimental design, and data augmentation where empirical data is scarce or
expensive to obtain. Despite their significant success in image generation,
the use of GANs has been mostly limited to this area. It would be highly
desirable to see their application more widely spread in the generation of
scientific numerical data.

In this work, we introduce a novel application of GANs for synthetic spectral
data generation. This offers a solution to the data scarcity found in
scientific contexts where collecting a significant amount of spectral signals
is critical for a subsequent application of data-driven approaches. Such a
scenario is common across a wide range of fields, including physics,
chemistry, astronomy, biology, medicine, and geology. Here, we particularly
focus on an illustrative problem in the research area of near-field radiative
heat transfer, involving a multilayered hyperbolic metamaterial. We explore
the use of a Conditional Wasserstein GAN (CWGAN) for data augmentation and
investigate its impact on the predictive capabilities of a feed-forward neural
network (FFNN). We find that the successful production of spectral data
requires two main changes to standard GANs. Firstly, the implementation of
Wasserstein GANs (WGANs) is necessary to counteract mode collapse, and
secondly, these WGANs need to be conditioned to yield accurate labels for the
generated data. We demonstrate that a simple FFNN, when augmented with data
produced by a CWGAN, notably improves its performance under conditions of data
scarcity. This underscores the intrinsic value of CWGAN data augmentation, not
just as a means to expand datasets. Furthermore, we illustrate that CWGANs
have the ability to serve as efficient surrogate model in low-data regimes.
Overall, this research work contributes to highlighting generative AI
algorithms’ potential in applications extending beyond the conventional realm
of image generation. We also anticipate that our findings will contribute to
advancing the understanding and application of generative AI algorithms in
data-limited scientific contexts.

This work is organized as follows. In Section 2, we review the fundamentals of
the primary generative adversarial frameworks underlying this work. Section 3
discusses the basic principles of the particular physical problem we utilize
to exemplify our approach. In Section 4, we present and discuss the results
obtained from implementing the generative adversarial method detailed in
Section 2 to the specific example problem outlined in Section 3. Finally, in
Section 5 we sum up the conclusions of this work.

##  2 Generative adversarial route to synthetic spectral data generation

Figure 1: Schematic representation of a Conditional Generative Adversarial
Network (CGAN) employed to produce synthetic spectral data. The process begins
with the combination of geometrical data (_W_), and a random vector (_z_),
both of which are fed into the generator. The generator’s role is to create a
synthetic, or ’fake’, spectrum from these inputs. The generated spectrum is
then introduced to the discriminator, along with the real spectra and the
geometrical data. The discriminator’s task is to discern between the fake and
real spectra. The conclusions drawn by the discriminator are subsequently used
to guide the training of both the generator and discriminator networks,
enhancing their ability to generate realistic spectra and to distinguish
between real and synthetic data, respectively.

In this Section, we review the basics of the three main generative adversarial
frameworks that form the basis of this work, namely, GANs, Wasserstein GANs
(WGANs) and Conditional WGANs (CWGANs). Here, we assume that the origin of the
real spectra used in these approaches is completely general, coming from any
from physical, chemical or biological process (in the following Sections we
discuss the application to a particular problem within the realm of near-field
radiative heat transfer). Figure 1 shows a schematic representation of the
underlying architecture of a general GAN algorithm. It comprises two main
parts: a generator network and a discriminator network (represented as orange
and blue rectangles, respectively, in Fig. 1). The generator and the
discriminator are two interconnected networks in the GAN system. The generator
generally begins with random noise ($\bf{z}$ in Fig. 1) and uses it to create
new data samples (new spectra in our application). On the other hand, the
discriminator takes these spectra and calculates the likelihood that each one
originates from the actual training data set. The two networks have competing
objectives. The generator’s goal is to create spectra that perfectly mirrors
the distribution of the training data. In doing so, it aims at generating
spectral data so convincingly authentic that the discriminator cannot tell it
apart from the real training data. Meanwhile, the discriminator aims at
distinguishing between the actual training data and the data fabricated by the
generator. Both networks are trained together in a competition until a Nash-
type equilibrium is reached and the training process ends [27]. At that stage,
the generator is producing spectra that the discriminator can no longer
reliably classify as ‘real’ or ‘fake’, signaling the end of the training
process.

Mathematically, the GAN configuration can be formulated as a minimization
problem of the generator and discriminator loss functions, suitably written in
terms of differentiable functions representing the discriminator and the
generator network models, $D({\bf x};\bm{\theta}_{d})$ and $G({\bf
z};\bm{\theta}_{g})$, respectively ($\bm{\theta}_{d}$ and $\bm{\theta}_{g}$
are the parameters of the corresponding ANNs —in what follows, for the sake of
clarity, we do not include this dependence in the equations). We start by
focusing on the loss function of the discriminator, $L_{D}(G,D)$, which can be
written as [27]

$\centering L_{D}(G,D)=-\mathbb{E}_{{\bf x}\sim p_{\mathrm{data}}({\bf
x})}[\log(D({\bf x}))]-\mathbb{E}_{{\bf z}\sim p_{z}({\bf
z})}[\log(1-D(G(\bf{z})))],\@add@centering$

where $\mathbb{E}$ stands for expectation over either the training samples,
$\bf{x}$, or the input noise variables, $\bf{z}$ (characterized respectively
by a probability distribution $p_{\mathrm{data}}({\bf x})$ and a prior
distribution $p_{z}({\bf z})$). From Eq. (2), and considering that $D$ outputs
a single scalar representing a probability, we can see that the training of
the discriminator is trying to minimize the likelihood of mistaking a real
sample for a fake one or a fake sample for a real one (first and second terms,
respectively, of the right-hand side of Eq. (2)).

However, there are significant performance issues associated to this original
choice of the loss function $L_{D}(G,D)$, including the difficulty to reach
the above-mentioned Nash equilibrium state (each network is updated
independently, and given the competitive nature of the generator and the
discriminator, in general, there is no clear point to stop the training) or
the so-called _mode collapse_ (arising when a network fails to generalize
accurately to all regions of the training data distribution). To address these
issues, a different strategy to train GANs was introduced, the so-called
_Wasserstein Generative Adversarial Networks_ (WGANs) [40]. One of the main
modifications of WGANs with respect to the original GAN architecture is the
presence of a critic model, $C({\bf x})$, instead of the discriminator model.
Importantly, $C({\bf x})$ outputs a score instead of a probability, which, in
turn, allows us to define a new loss function for the critic, $L_{C}(G,C)$,
given by

$\centering L_{C}(G,C)=-\mathbb{E}_{x\sim p_{\mathrm{data}}({\bf x})}[C({\bf
x})]+\mathbb{E}_{z\sim p_{z}({\bf z})}[C(G({\bf z}))].\@add@centering$ (1)

This loss function summarizes well some of the main advantages of the WGANs.
WGANs help address the training challenges of GANs by using the Wasserstein
distance (or Earth Mover’s distance) as the loss function instead of the
Jensen-Shannon divergence used in original GANs [40]. In addition, it provides
a meaningful loss metric, i.e., the value of the critic in WGANs provides a
meaningful measure of the distance between the real and generated data
distributions. This is unlike the original GANs where the discriminator’s
output does not correlate well with the quality of the generated samples.

A pivotal aspect of WGANs is the need for the Critic to operate within the set
of the so-called _1-Lipschitz_ functions, a critical component of the model
[40]. Lipschitz functions are mathematical functions possessing a property
where there exists a real-valued constant such that, for every pair of points,
the absolute difference in function values can be bounded by this constant
times the absolute difference of input values. When this constant is 1, the
functions are known as 1-Lipschitz. The 1-Lipschitz constraint is crucial as
it bounds how much a function’s output can change with small variations in
input, ensuring the function does not change too abruptly. In WGANs, this is
vital to ensure that the Critic provides meaningful and stable gradients for
the generator to learn from, facilitating a more reliable learning process.
Originally, weight clipping was proposed to enforce this 1-Lipschitz
condition. However, this sometimes resulted in convergence failure [40]. To
counter these issues, a more robust technique known as the _gradient penalty_
was introduced. This method involves adding a loss term to maintain the L2
norm (a measure of the vector length of parameters or weights) of the Critic
close to a value of 1 [41]. This approach assists in keeping the Critic’s
function within the 1-Lipschitz constraint, enhancing the stability and
performance of WGANs. Incorporating these improvements, the full loss for the
critic in a WGAN now reads

$\displaystyle L_{C}(G,C)$ $\displaystyle=$ $\displaystyle-\mathbb{E}_{{\bf
x}\sim p_{\mathrm{data}}({\bf x})}[C({\bf x})]+\mathbb{E}_{{\bf z}\sim
p_{z}({\bf z})}[C(G({\bf z}))]+$ (2)
$\displaystyle\lambda\>\mathbb{E}_{{\bf\hat{x}}\sim
p_{\hat{x}}}\left[(||\nabla_{\hat{x}}C({\bf\hat{x}})||_{2}-1)^{2}\right],$

where $||\cdot||_{2}$ is the L2 norm and $\lambda$ is a weight parameter for
the gradient penalty (throughout this work we assume $\lambda=10$, as pointed
out in Ref. [41]), and $\hat{x}=x+\alpha(G(z)-x)$ is an interpolated point
between a real sample and generated sample on which to calculate the gradient,
with $\alpha\in[0,1]$. As a final remark, proper training of the WGAN requires
the critic to be trained ahead of the generator, so that for each training
step of the generator the critic is updated $n_{\rm train}$ times (following
[40], we chose $n_{\rm train}=5$ in all our models).

As for the generator loss function, $L_{G}(G,C)$, an important aspect to
realize is that for synthetic spectral data generation our focus is on a
regression problem. This implies that we need a greater control over the
output than that obtained just by acquiring a random but realistic sample. It
is therefore crucial to ensure that the generated data accurately corresponds
to the correct system parameters that yield that response. To achieve this, we
need to _condition_ the WGAN, leading to the creation of a Conditional WGAN
(CWGAN) [42]. In this work, we will implement that by adding an extra loss
term quantifying the Mean Absolute Error (MAE) between the conditioned
generated example and the training example corresponding to the _ground truth_
of the condition. Accounting for this conditioning, $L_{G}(G,C)$ can be
expressed as

$L_{G}(G,C)=\mathbb{E}_{{\bf z}\sim p_{z}({\bf z}),\>{\bf x}\sim p_{data}({\bf
x})}\left[\left|{\bf x}-G({\bf z}|{\bf W})\right|\right]-\mathbb{E}_{{\bf
z}\sim p_{z}({\bf z})}[C(G({\bf z}))],$ (3)

where ${\bf x}$ is the training example corresponding to the system parameters
${\bf W}$, and $G({\bf z}|{\bf W})$ is a generated example conditioned on the
same parameters ${\bf W}$. The first term in the r.h.s. of Eq. (3) corresponds
to the above discussed conditioning procedure, while the second term is
associated to the coupling of the generator and the critic.

##  3 Illustrative problem: Near-field radiative heat transfer spectra in
multilayer hyperbolic metamaterials

In this Section we provide an overview of the fundamentals of the specific
problem we use to illustrate the proposed approach. We have chosen a physical
problem in the context of near-field radiative heat transfer involving
multilayer hyperbolic metamaterials [43]. Despite the specific character of
this class of systems, the chosen problem can be considered both
representative of the types of problems that our approach can address
effectively and complex enough to showcase the versatility of our method.

One of the major advances in recent years in the field of thermal radiation
has been the experimental confirmation that the limit set by Stefan-
Boltzmann’s law for the radiative heat transfer between two bodies can be
largely overcome by bringing them sufficiently close [44]. This phenomenon is
possible because in the near-field regime, i.e., when the separation between
two bodies is smaller than the thermal wavelength $\lambda_{\rm Th}$ ($\sim$10
$\mu$m at room temperature), radiative heat can also be transferred via
evanescent waves (or photon tunneling). This new contribution is not taken
into account in Stefan-Boltzmann’s law and dominates the near-field radiative
heat transfer (NFRHT) for sufficiently small gaps or separations [45, 46, 47].
Among the different strategies that have been recently proposed to further
enhance NFRHT, one of the most popular ideas is based on the use of multiple
surface modes that can naturally appear in multilayer structures. In this
regard, a lot of attention has been devoted to multilayer systems where
dielectric and metallic layers are alternated to give rise to the so-called
_hyperbolic metamaterials_ [48, 49, 50, 51, 52, 53, 54, 55, 56, 57]. The
hybridization of surface modes appearing in different metal-dielectric
interfaces have indeed been shown to lead to a great enhancement of the NFRHT,
as compared to the case of two infinite parallel plates [55].

Figure 2: (a) Schematic representation of the physical system under study. It
features a multilayered hyperbolic metamaterial comprised of two identical
structures, each alternating between metal (depicted in grey) and dielectric
(blue) layers that extend infinitely. These two systems are separated by a
distance $d_{0}=10$ nm. Every layer has a thickness $d_{i}$, and both systems
are backed by a metallic substrate. (b) Transmission of evanescent waves as a
function of the frequency ($\omega$) and the parallel wavevector ($k$),
considering the case where there are 8 layers per system and both
$d_{i}=d_{0}=10$ nm. The transmission pattern exhibits a series of lines
nearing unity, resulting from the hybridization of surface plasmon polaritons
(SPPs) within the metallic layers. (c) Sample spectra of heat transfer
coefficients, $h_{\omega}$, for several representative combinations of layer
thicknesses, which are specified in Table 1.

Following Ref. [55], we consider here the radiative heat transfer between two
identical multilayer structures separated by a gap $d_{0}$, as shown in Fig.
2(a). Each body contains $N$ total layers alternating between a metallic layer
with a permittivity $\epsilon_{\mathrm{m}}$ and a lossless dielectric layer of
permittivity $\epsilon_{\mathrm{d}}$. The thickness of the layer $i$ is
denoted by $d_{i}$ and it can take any value within a given range (to be
specified below). While the dielectric layers will be set to vacuum
($\epsilon_{\mathrm{d}}=1$), the metallic layers will be described by a
permittivity given by a Drude model:
$\epsilon_{\mathrm{m}}(\omega)=\epsilon_{\infty}-\omega^{2}_{p}/[\omega(\omega+i\gamma)]$,
where $\epsilon_{\infty}$ is the permittivity at infinite frequency,
$\omega_{p}$ is the plasma frequency, and $\gamma$ de damping rate. From now
on, we set $\epsilon_{\infty}=1$, $\omega_{p}=2.5\times 10^{14}$ rad/s, and
$\gamma=1\times 10^{12}$ rad/s (these parameters provide a surface plasmon
frequency similar to the surface phonon-polariton frequency of the interface
between SiC and vacuum).

We describe the radiative heat transfer within the framework of fluctuational
electrodynamics [58, 59], particularly focusing on the near-field regime. In
this regime, the radiative heat transfer is dominated by TM- or $p$-polarized
evanescent waves and the heat transfer coefficient (HTC) between the two
bodies, i.e., the linear radiative thermal conductance per unit of area, is
given by [60]

$h=\frac{\partial}{\partial
T}\int^{\infty}_{0}\frac{d\omega}{2\pi}\>\Theta(\omega,T)\int^{\infty}_{\omega/c}\frac{dk}{2\pi}\,k\>\tau_{p}(\omega,k),$
(4)

where $T$ is temperature, $\Theta(\omega,T)=\hbar\omega/(e^{\hbar\omega/k_{\rm
B}T}-1)$ is the mean thermal energy of a mode of frequency $\omega$, $k$ is
the magnitude of the wave vector parallel to the surface planes, and
$\tau_{p}(\omega,k)$ is the transmission (between 0 and 1) of the
$p$-polarized evanescent modes given by

$\tau_{p}(\omega,k)=\frac{4\left[\mbox{Im}\left\\{r_{p}(\omega,k)\right\\}\right]^{2}e^{-2q_{0}d_{0}}}{|1-r_{p}(\omega,k)^{2}e^{-2q_{0}d_{0}}|^{2}}.$
(5)

Here, $r_{p}(\omega,k)$ is the Fresnel reflection coefficient of the
$p$-polarized evanescent waves from the vacuum to one of the bodies and
$q_{0}=\sqrt{k^{2}-\omega^{2}/c^{2}}$ ($\omega/c<k$) is the wave vector
component normal to the layers in vacuum. The Fresnel coefficient needs to be
computed numerically and we have done it by using the scattering matrix method
described in Ref. [61]. In our numerical calculations of the HTC we also took
into account the contribution of $s$-polarized modes, but it turns out to be
negligible for the gap sizes explored in this work.

Let us briefly recall that, as explained in Ref. [55], the interest in the
NFRHT in these multilayer structures resides in the fact that the heat
exchange in this regime is dominated by surfaces modes that can be shaped by
playing with the layer thicknesses. In the case of two parallel plates made of
a Drude metal, the NFRHT is dominated by the two cavity surface modes
resulting from the hybridization of the surface plasmon polaritons (SPPs) of
the two metal-vacuum interfaces [55]. These two cavity modes give rise to two
near-unity lines in the transmission function $\tau_{p}(\omega,k)$. Upon
introducing more internal layers with appropriate thicknesses, one can have
NFRHT contributions from surface states at multiple surfaces, as we illustrate
in Fig. 2(b) for the case of $N=8$ layers (4 metallic and 4 dielectric
layers), separated by a vacuum gap of $d_{0}=10$ nm at $T=300$ K. As shown in
Ref. [55], the contribution of these additional surface states originating
from internal layers can lead to a great enhancement of the NFRHT as compared
to the bulk system (two parallel plates) in a wide range of gap values.

Table 1: Thicknesses combinations of the representative samples of the
spectral heat transfer coefficient, $h_{\omega}$, shown in Fig. 2(c).

  * | Layer thicknesses (nm)
---|---
Label | $d_{1}$ | $d_{2}$ | $d_{3}$ | $d_{4}$ | $d_{5}$ | $d_{6}$ | $d_{7}$ | $d_{8}$
$S_{1}$ | 5.0 | 20.0 | 20.0 | 5.0 | 20.0 | 5.0 | 5.0 | 20.0
$S_{2}$ | 5.0 | 12.5 | 12.5 | 5.0 | 12.5 | 5.0 | 5.0 | 12.5
$S_{3}$ | 5.0 | 12.5 | 5.0 | 5.0 | 5.0 | 20.0 | 12.5 | 5.0
$S_{4}$ | 5.0 | 5.0 | 12.5 | 5.0 | 5.0 | 5.0 | 20.0 | 12.5
$S_{5}$ | 5.0 | 5.0 | 5.0 | 20.0 | 12.5 | 12.5 | 5.0 | 20.0
$S_{6}$ | 5.0 | 5.0 | 5.0 | 12.5 | 5.0 | 20.0 | 5.0 | 12.5
$S_{7}$ | 20.0 | 5.0 | 20.0 | 12.5 | 20.0 | 5.0 | 12.5 | 20.0
$S_{8}$ | 12.5 | 12.5 | 5.0 | 12.5 | 5.0 | 5.0 | 12.5 | 5.0

Of special interest for this work is the spectral HTC, $h_{\omega}$, defined
as the HTC per unit of frequency: $h=\int^{\infty}_{0}h_{\omega}\>d\omega$. To
create the initial dataset of real spectra (which later on will be augmented
by our generative adversarial approach), we apply the above-described
theoretical framework to compute a total of 6,561 $h_{\omega}$ spectra. The
thicknesses $d_{i}$ of each layer were varied between 5 and 20 nm, and every
spectrum contains 200 frequency points in the range $\omega\in[0.3,3]\times
10^{14}$ rad/s. As discussed in the next Section, that dataset of spectra will
be split in different proportions to become training and test sets. Figure
2(c) shows several representative samples of $h_{\omega}$ spectra,
corresponding to the following thicknesses combinations listed in Table 1.

The spectra displayed in Fig. 2(c) show the broad variety of spectral features
that can be obtained from the studied system (from double broad peaks with
very narrow resonances in between, to single narrow peaks, or to two resonant
peaks separated by a gap). This set of spectra essentially serves as a
comprehensive blueprint for the whole adversarial approach, guiding it on the
characteristics and features that should be exhibited in the synthetic data.
Hence, this allows the proposed approach to capture a wider range of
underlying patterns and relationships, which in turn allows it to generate a
more realistic and diverse array of synthetic spectral data.

##  4 Results and discussion

We proceed in this Section to report on the results obtained when applying the
generative adversarial approach summarized in Section 2 to the specific
illustrative problem described in Section 3. We begin by providing a strong
quantitative justification of the necessity of using a trained CWGAN for this
problem, instead of a plain CGAN (Conditional GAN —the details of the specific
architecture used in each case are provided below). Figures 3(a) and 3(b)
show, respectively, the ability to reproduce the training set of a trained
CGAN and a trained CWGAN projecting the data on two dimensions via a Principal
Component Analysis (PCA) [62], which retains most of the training data
structure due to its $>90\%$ reproduction rate ($RR$).

The PCA calculations shown in Fig. 3 were done as follows. First, we performed
a singular value decomposition (SVD) of the covariance matrix $\Sigma$:
$\left[{\bf U},{\bf S},{\bf V}\right]=S\,V\,D(\Sigma)$ with
$\Sigma=\frac{1}{m}X^{T}X$ [63]. Here, U and V are unitary matrices, S is the
singular value matrix, m is the total number of data examples and X is the
data matrix, containing in each column one data example x. To obtain the
reduced 2-dimensional (2D) representation of the data, we calculated a reduced
matrix $\mathrm{U}_{\mathrm{reduced}}$ retaining the first 2 columns of the U
matrix obtained from the SVD, and use it as the projection matrix,
$\bf\hat{x}=\mathrm{U}^{T}_{\mathrm{reduced}}X$, where $\bf\hat{x}$ is the
2-dimensional representation of the data. The reproduction rate ($RR$) of the
whole 2D-PCA analysis is then defined as the ratio of the first 2 singular
values over all the $N$ singular values obtained:

$\mathrm{RR}=\frac{\sum^{2}_{i=1}S_{ii}}{\sum^{N}_{i=1}S_{ii}}$ (6)

Figure 3: Reproduction of the training set using (a) a CGAN, showing in grey
the training set and in green the CGAN reproduction of the set; and (b) the
CWGAN, using the same color scheme. The results were obtained by applying PCA
to the corresponding spectra, and plotting the first two components, $z_{1}$
and $z_{2}$ (with a $>90\%$ reproduction rate).

The Conditional GAN and the Conditional WGAN share the same architectural
design, the specifics of which are detailed further on. Both networks
underwent identical training conditions, with the complete spectral dataset
partitioned into 80% for training and 20% for validation. The key
distinguishing factor lies in their respective loss functions ($L_{D}(G,D)$
for CGAN and $L_{C}(G,C)$ for CWGAN). The CGAN employs Eq. (2) along with a
generator loss which is the sum of the first term in the r.h.s. of Eq. (3)
(the conditional term) and the second term in the r.h.s. of Eq. (2) with
opposite sign. Meanwhile, the CWGAN utilizes the loss defined in Eq. (3) for
the generator and Eq. (2) for the critic. Notably, as depicted in Fig. 3(a),
the CGAN manifests evident signs of mode collapse, rendering it unable to
reproduce most examples beyond the principal cluster structures in the PCA. In
contrast, the CWGAN shows the capability of replicating most of the
complexities within the training set. These results can be considered as a
novel illustration, in the context of synthetic generation of spectral data,
of the key role played by Wasserstein’s loss function to create more robust
generative adversarial approaches.

Figure 4: Schematics of the architecture used for the Conditional Wasserstein
Generative Adversarial Network (CWGAN) employed in this work. Both the
generator and critic sections are shown in the left and right parts,
respectively, of the figure. As shown, the generator accepts geometrical
parameters as input and produces a heat transfer coefficient (HTC) spectrum as
output. It is constructed of four fully connected layers (FCs) (green blocks),
dropout layers placed after each FC layer (orange blocks) and a final linear
activation layer (red block). The critic section simultaneously accepts HTC
spectra produced by the generator, as well as _true_ spectra, and the
geometrical parameters as inputs, providing a scalar score as output. The
critic model consists of a unique FC layer for each parallel input branch
followed by a concatenation layer that combines information from both input
branches into a single vector. Afterwards, two more FC layers and a final
linear activation layer produce the score. In both the generator and critic
sections, normalization of the input data is applied as a preprocessing step
(grey blocks).

Figure 4 summarizes the specific CWGAN architecture we have found as most
efficient for the studied problem. The generator (left side of Fig. 4) is
composed of 4 hidden fully connected layers of an increasing number of neurons
in each layer (we consider 50, 100, 150, and 200 neurons for the four layers
—represented as green blocks, labeled as FC1–FC4, in Fig. 4). The generator
model takes the condition as input (the 8 values for the thicknesses of the
layers) and returns a generated $h_{\omega}$ spectrum (sampled by 200
frequencies). Consistently with the results in Refs. [64, 65], we found that
we did not need to feed a random data distribution z into the generator for
operation: 20$\%$ dropout layers between all layers of the generator (i.e.,
connections between two consecutive neurons are dropped with a 20% chance)
provide enough variability for this task (the dropout operations are
represented as orange blocks in Fig. 4). On the other hand, the critic (right
side of Fig. 4) takes both a sample spectrum and the condition (either
generated or from the training distribution) in parallel processing lines of a
single hidden layer (with 150 and 50 neurons, respectively —FC5 and FC6 in
Fig. 4). Then, it concatenates and processes the information through two
additional hidden layers (FC7 and FC8 in Fig. 4, with 100 and 50 neurons,
respectively) to output a single number, the score. All hidden layers feature
a scaled exponential linear unit (selu) activation function. In all models
discussed in this work, as a pre-processing step, we will also calculate the
logarithm of the spectra, subtract the mean of both the input parameters and
the spectra, and divide both the system parameters and the spectra by the
standard deviation (this normalization operation is represented by grey blocks
in Fig. 4). Finally in both the generator and the critic a final linear
activation layer is added to ensure the output has the correct size (red
blocks in Fig. 4).

We focus now on analyzing the evolution with the training steps of the loss
functions of the discriminator and the generator ($L_{C}(G,C)$ and
$L_{G}(G,C)$, respectively), as obtained by training the architecture
displayed in Fig. 4 with the 6,561 $h_{\omega}$ spectra described in Section
3. Monitoring the loss functions underlying our model during the training
process is critical to understanding, diagnosing, and improving the whole
generative adversarial framework studied in this work. The obtained numerical
results are summarized in Fig. 5 (panel (a) corresponds to $L_{C}(G,C)$, while
panel (b) displays the results for $L_{G}(G,C)$). As seen in Fig. 5(a),
$L_{C}(G,C)$ initially displays a transient behavior based on a fast drop in
value followed by a sudden increase and an oscillation (around $10^{3}$
training steps), until it reaches at stationary value at approximately
$10^{4}$ training steps. To get additional insight into the numerical origin
of this evolution, the inset of Fig. 5(a) shows the dependence with the
training steps of the three different terms forming $L_{C}(G,C)$, namely,
$\mathbb{E}_{{\bf x}\sim p_{\mathrm{data}}({\bf x})}[C({\bf x})]$,
$\mathbb{E}_{{\bf z}\sim p_{z}({\bf z})}[C(G({\bf z}))]$, and the gradient
penalty, $\lambda\>\mathbb{E}_{{\bf\hat{x}}\sim
p_{\hat{x}}}\left[(||\nabla_{\hat{x}}C({\bf\hat{x}})||_{2}-1)^{2}\right]$ (see
Eq. 2). As observed, $L_{C}(G,C)$ is dominated by the expected values of the
score value produced by the critic model for both the true training samples,
(${\bf x}$), and the samples fabricated by the generator, $G({\bf z})$ (black
and red lines, respectively, in inset of of Fig. 5(a) —the green line
corresponds to the gradient penalty term). A remarkable aspect of the overall
evolution of both expected value contributions to $L_{C}(G,C)$ is the fact
that, despite their complicated transient behavior, they lock their difference
in values after around $10^{4}$ training steps. This in turn allows
$L_{C}(G,C)$ to reach a stationary value (notice the difference in sign
between both terms in Eq. 2), which marks the completion of the learning
process by the critic.

Figure 5: Evolution of the critic and generator loss functions during the
training process ($L_{C}(G,C)$ and $L_{G}(G,C)$, panels (a) and (b),
respectively). Inset of panel (a) shows dependence on the training steps of
three contributions of $L_{C}(G,C)$ (see Eq. 2 in the main text). Inset of
panel (b) corresponds to evolution with the training step of the mean absolute
error (MAE) associated to the synthetic spectra produced by the generator.
Notice that in our framework the critic evaluates the loss 5 times per
generator loss evaluation.

Figure 5(b) shows the results for the evolution of the generator loss
function, $L_{G}(G,C)$, during the training process. The inset of Fig. 5(b)
displays the corresponding evolution of the MAE contribution to $L_{G}(G,C)$
(i.e., the evolution of the first term in the r.h.s. of Eq. 3). As deduced by
comparing the values reached by the MAE with those of $L_{G}(G,C)$, the
generator loss function is dominated by the expected value term,
$\mathbb{E}_{{\bf z}\sim p_{z}({\bf z})}[C(G({\bf z}))]$ (second term in the
r.h.s. of Eq. 3). As described above, for large values of the training step
($\gtrsim 10^{4}$), this latter term necessarily follows its counterpart term
in the critic loss function (so, as also pointed out above, their difference
is maintained and a stationary value of $L_{C}(G,C)$ is reached). Therefore,
since we do not expect $L_{G}(G,C)$ to reach a stationary value, MAE becomes
the key magnitude to monitor the quality of the learning process of the
generator (note that that this is also in accordance with the overall goal of
the proposed application: creating artificial spectral samples
indistinguishable from the real ones). Indeed, as shown in inset of Fig. 5(b),
the MAE computed for the studied problem converges to values $<0.2$ for the
larger training step values considered in our calculations.

Next, we proceed to quantify our model’s proficiency in reproducing the
spectral heat transfer coefficient (HTC) of the studied physical system, based
on the model architecture illustrated in Fig. 4. For this analysis, our focus
is on two distinctive metrics, concentrating on two different aspects of the
spectra. The first, the per-point relative mean error ($L_{\mathrm{point}}$),
gauges the model’s capability to accurately represent each point in the
spectrum through a relative error analysis. The second, the integral relative
mean error ($L_{\mathrm{integ}}$), is predominantly influenced by the model’s
accuracy in reproducing the primary spectral characteristics, notably the
resonances. Their respective definitions are as follows:

$\displaystyle
L_{\mathrm{point}}=\frac{1}{N\>m}\sum_{i=1}^{N}\sum_{j=1}^{m}\frac{y^{i}_{j}-\hat{y}_{j}^{i}}{y^{i}_{j}},$
(7) $\displaystyle
L_{\mathrm{integ}}=\frac{1}{N}\sum_{i=1}^{N}\frac{\int(y^{i}-\hat{y}^{i})\>d\omega}{\int
y^{i}\>d\omega},$ (8)

where N is the number of spectra, m the number of points in each spectrum,
$y^{i}$ is a data example and $\hat{y}^{i}$ is the corresponding generated
example by the model. Note that we undo all pre-processing operations to
perform these calculations, and that all integrals are performed over
frequency points.

We start our analysis by establishing a baseline outcome for comparison using
a simple FFNN, comprised of five hidden layers, each hosting 200 neurons,
characterized by a selu activation function, and a final linear activation
layer. An analogous network has been previously shown to effectively model
this identical physical system given ample data [43] —consequently, we
anticipate that this particular network will serve as a suitable benchmark.
Another important aspect is the apparent similarities between the generator
and the FFNN. We aim at demonstrating that this specific simple design, given
a sufficiently large dataset, can accurately replicate the spectral
characteristics inherent to the type of systems under study, without resorting
to any data augmentation technique. To accomplish this, we split the original
dataset, which consists of 6,561 $h_{\omega}$ spectra, allocating 80% for
training set and the remaining 20% for the validation set. Our findings reveal
that after 50,000 training iterations (epochs), and using the Adam
optimization algorithm with a steady learning rate of $3\times 10^{-4}$, this
simple neural network is proficient in replicating the system’s spectra,
reaching values of $L_{\mathrm{point}}$ and $L_{\mathrm{integ}}$ for the
validation set of $3.61\%$ and $1.45\%$, respectively (note that these results
are comparable to those found in Ref. [43]).

Then we proceed to assess the CWGAN capability as numerical engine for
spectral data augmentation. To do that, we follow a two-step approach. First,
once the CWGAN has been trained, we use it to generate a number of additional
spectra to include on the training set. Second, we retrain the above-described
FFNN with this new data set to create an _augmented_ FFNN. In this work, we
generate a total of 10,000 new spectra for this data augmentation process,
which are added to the training set. This particular amount of additional
spectra was chosen after observing that our results are converged when adding
to the original dataset a number of extra spectra in the range 5,000 – 10,000.
Note that, once the generator has been trained, this augmentation of the
original dataset is computationally inexpensive.

Following this approach, first, we found that when using the
training/validation split considered until now (80%-20%), the FFNN and the
augmented FFNN have similar performance, both in terms of per-point relative
mean accuracy and integral relative mean error (respective error values of
approximately $3.6\%$ and $1.5\%$ are obtained for both models). This suggests
that, as anticipated, using a sufficiently large dataset minimizes the impact
of augmenting the original dataset, making any data augmentation barely
noticeable. However, that conclusion changes when a different data scenario is
considered. To modify the amount of data available to the models, we
increasingly reduced the size of the training set by transferring part of it
to the validation set, and retrained both the simple and augmented FFNNs from
scratch (different values of the split training/validation were considered in
the range 80%-20% – 1%-99% —note that the data augmentation is done separately
for each split ratio).

Figure 6: (a) Evolution of the integral relative mean error for the validation
set as a function of the validation set fraction of the total data. (b) Same
as in (a), but now using the per-point relative mean error metric. In both (a)
and (b), blue and black dots (lines) show the results for a simple FFNN and a
CWGAN-augmented FFNN, respectively, whereas red dots (lines) correspond to the
results obtained by using the CWGAN as surrogate model.

Figure 6 summarizes the main results of this analysis using the two error
metrics considered in this work (panels (a) and (b) correspond to
$L_{\mathrm{integ}}$ and $L_{\mathrm{point}}$, respectively —blue dots (lines)
display the results for the simple FFNN, whereas black dots (lines) correspond
to the CWGAN-augmented FFNN). As seen, both in terms of $L_{\mathrm{integ}}$
and $L_{\mathrm{point}}$, the simple FFNN and the augmented FFNN have similar
performance when the validation set fraction is smaller than approximately 70%
(we can therefore ascribe that interval to a sufficiently large training set
scenario). However, in the case of $L_{\mathrm{integ}}$(Fig. 6(a)), further
increasing of the validation set fraction (i.e., further reduction of the
training set size) leads to the augmented FFNN to perform increasingly better
with training data reduction than the simple FFNN . As observed, values of
$L_{\mathrm{integ}}=13.2\%$ are achieved using a simple FFNN in the limit case
of a validation set fraction of 99%, whereas an augmented FFNN reduces that
error close to a than a half ($L_{\mathrm{integ}}=6.8\%$). The performance
improvement is not so dramatic in the case of the per-point relative mean
error (Fig. 6(b)), but we still observe slight improvements in the low-data
scenario (for the extreme case of a validation set fraction of 99%,
$L_{\mathrm{point}}=18.2\%$ is obtained for the simple FFNN, while the
augmented FFNN leads to $L_{\mathrm{point}}=16.8\%$).

Overall, the numerical results discussed above suggest that CWGAN data
augmentation has intrinsic value beyond simply providing larger datasets. In
particular, the reported results show that CWGAN data augmentation holds value
in creating synthetic data more adaptable and resilient to scenarios with
limited data. To the best of our knowledge, this finding is reported here for
the first time. We believe it could contribute to the development of more
efficient models that require smaller datasets, through the synthetic
generation capabilities offered by generative adversarial frameworks.

Finally, to conclude the present analysis, we compare the performance as
surrogate model to generate $h_{\omega}$ spectra of the trained CWGAN with
that of the simple and augmented FFNNs. This application arises from the
conditioning modification we introduced in this work to conventional GANs (see
Eq. 3 and the corresponding discussion in Section 2). This conditioning allows
us to accurately keep track of the geometrical parameters associated to each
spectra created by the generator of our CWGAN model, so once the generator is
trained, it can be decoupled from the whole architecture and used as a
standalone surrogate model. The obtained numerical results are also displayed
in Fig. 6 (red points and red lines corresponds to the results of a CWGAN used
as surrogate model). As observed, for the integral relative mean error (Fig.
6(a)) the CWGAN displays worse performance than both the simple and the
augmented FFNNs when the validation set fraction is below 70% approximately
(i.e., the data regime that we previously labeled as of sufficiently large
training set scenario for both the simple and the augmented FFNNs). However,
in the low-data regime (values of the validation set fraction larger than
$70\%$) we observe that the CWGAN surrogate gradually improves the results of
a simple FFNN, until fully reproducing the improvement reached by an augmented
FFNN in the extreme case of 99% validation set fraction. Regarding the per-
point relative mean error (Fig. 6(b)), the CWGAN surrogate displays a
significantly worse performance in comparison to both the simple and the
augmented FFNNs for the validation set fractions below 70%, but from that
point on, it starts converging to the results of an augmented FFNN in the low-
data regime. Ultimately, this comparison between the surrogate role of all
considered models reinforces our previous conclusion that the CWGAN shows
higher resilience to the reduction of training data than the simple FFNN,
becoming also in this context an efficient architecture in low-data scenarios.

##  5 Conclusions

In this work we have studied the application of Generative Adversarial
Networks (GANs) for synthetic spectral data generation, providing a solution
to the data scarcity challenge pervasive in scientific domains where acquiring
substantial spectral signals is of paramount importance. Our main focus has
been an illustrative problem in the domain of near-field radiative heat
transfer involving a multilayered hyperbolic metamaterial. We have analyzed
the use of a Conditional Wasserstein GAN (CWGAN) for data augmentation and
studied its influence on the predictive capacities of a feed-forward neural
network (FFNN). Our results reveal that generating spectral data effectively
entails two main modifications to traditional GANs: firstly, incorporating
Wasserstein GANs (WGANs) to prevent mode collapse, and secondly, conditioning
these WGANs to secure accurate labels for the generated data. It is
demonstrated that a basic FFNN, augmented with data yielded by a CWGAN,
substantially improves its efficiency in scenarios where data is scarce,
showing the inherent importance of CWGAN data augmentation beyond the simple
expansion of datasets. Moreover, we present that CWGANs can function as a
superior surrogate model when data is limited. Overall, this work aims at
contributing to the research area of generative AI algorithms’ applicability
beyond the conventional field of image generation. We believe that our
findings contribute to the understanding and applicability of generative AI
algorithms in data-constrained contexts and could stimulate further research
work in the application of generative AI in a variety of scientific scenarios.

## Acknowledgements

J.J.G.E. was supported by the Spanish Ministry of Science and Innovation
through a FPU grant (FPU19/05281). J.C.C. acknowledges funding from the
Spanish Ministry of Science and Innovation (PID2020-114880GB-I00). J.B.A.
acknowledges financial support from Ministerio de Ciencia, Innovación y
Universidades (RTI2018-098452-B-I00).

## References

## References

  * [1] Mitchell T 1997 Machine Learning (McGraw-Hill)
  * [2] LeCun Y, Bengio Y and Hinton G 2015 Nature 521 436–444
  * [3] Goodfellow I, Bengio Y and Courville A 2016 Deep Learning (MIT Press)
  * [4] Aggarwal C 2018 Neural Networks and Deep Learning (Cham: Springer)
  * [5] Krizhevsky A, Sutskever I and Hinton G 2012 Advances in Neural Information Processing Systems 25 1097–1105
  * [6] Hinton G, Deng L, Yu D, Dahl G, Mohamed A R, Jaitly N, Senior A, Vanhoucke V, Nguyen P, Sainath T and Kingsbury B 2012 IEEE Signal Process. Mag. 29 82–97
  * [7] Cho K, van Merrienboer B, Gulcehre C, Bahdanau D, Bougares F, Schwenk H and Bengio Y 2014 arXiv (Preprint 1406.1078)
  * [8] Shalev-Shwartz S, Shammah S and Shashua A 2016 arXiv (Preprint 1610.03295)
  * [9] Carleo G, Cirac I, Cranmer K, Daudet L, Schuld M, Tishby N, Vogt-Maranto L and Zdeborová L 2019 Rev. Mod. Phys. 91(4) 045002
  * [10] Carleo G and Troyer M 2017 Science 355 602–606
  * [11] Wang L 2016 Phys. Rev. B 94(19) 195105
  * [12] Carrasquilla J and Melko R G 2017 Nature Physics 13 431–434
  * [13] van Nieuwenburg E P L, Liu Y H and Huber S D 2017 Nature Physics 13 435–439
  * [14] Wu D, Wang L and Zhang P 2019 Phys. Rev. Lett. 122(8) 080602
  * [15] Dunjko V and Briegel H J 2018 Reports on Progress in Physics 81 074001
  * [16] Green S R, Simpson C and Gair J 2020 Phys. Rev. D 102(10) 104057
  * [17] Peurifoy J, Shen Y, Jing L, Yang Y, Cano-Renteria F, DeLacy B G, Joannopoulos J D, Tegmark M and Soljačić M 2018 Science Advances 4 eaar4206
  * [18] So S, Badloe T, Noh J, Bravo-Abad J and Rho J 2020 Nanophotonics 9 1041–1057
  * [19] Jiang J, Chen M and Fan J A 2021 Nature Reviews Materials 6 679–700
  * [20] Frising M, Bravo-Abad J and Prins F 2023 Machine Learning: Science and Technology 4 02LT02
  * [21] Butler K T, Davies D W, Cartwright H, Isayev O and Walsh A 2018 Nature 559 547–555
  * [22] Gómez-Bombarelli R, Wei J N, Duvenaud D, Hernández-Lobato J, Sánchez-Lengeling B, Sheberla D, Aguilera-Iparraguirre J, Hirzel T D, Adams R P and Aspuru-Guzik A 2018 ACS Central Science 4 268–276
  * [23] Zhang L, Han J, Wang H, Car R and E W 2018 Phys. Rev. Lett. 120(14) 143001
  * [24] Coley C W, Barzilay R, Jaakkola T S, Green W H and Jensen K F 2017 ACS Central Science 3 434–443
  * [25] Rupp M, Tkatchenko A, Müller K R and von Lilienfeld O A 2012 Phys. Rev. Lett. 108(5) 058301
  * [26] Ching T, Himmelstein D S, Beaulieu-Jones B K, Kalinin A A, Do B T, Way G P, Ferrero E, Agapow P M, Zietz M, Hoffman M M, Xie W, Rosen G L, Lengerich B J, Israeli J, Lanchantin J, Woloszynek S, Carpenter A E, Shrikumar A, Xu J, Cofer E M, Lavender C A, Turaga S C, Alexandari A M, Lu Z, Harris D J, DeCaprio D, Qi Y, Kundaje A, Peng Y, Wiley L K, Segler M H S, Boca S M, Swamidass S J, Huang A, Gitter A and Greene C S 2018 Journal of The Royal Society Interface 15 20170387
  * [27] Goodfellow I, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, Courville A and Bengio Y 2014 NIPS 2672–2680
  * [28] Salimans T, Goodfellow I, Zaremba W, Cheung V, Radford A and Chen X 2016 NIPS’16: Proceedings of the 30th International Conference on Neural Information Processing Systems 29 2234–2242
  * [29] Paganini M, de Oliveira L and Nachman B 2018 Phys. Rev. D 97(1) 014021
  * [30] Carrasquilla J, Torlai G, Melko R G and Aolita L 2019 Nature Machine Intelligence 1 155–161
  * [31] Koch R and Lado J 2022 Phys. Rev. Res. 4(3) 033223
  * [32] Carracedo-Cosme J and Pérez R 2022 arXiv (Preprint 205.00447)
  * [33] So S and Rho J 2019 Nanophotonics 8 1255–1261
  * [34] Christensen T, Loh C, Picek S, Jakobović D, Jing L, Fisher S, Ceperic V, Joannopoulos J D and Soljačić M 2020 Nanophotonics 9 4183–4192
  * [35] Rodríguez A C, Kacprzak T, Lucchi A, Amara A, Sgier R, Fluri J, Hofmann T and Réfrégier A 2018 Computational Astrophysics and Cosmology 5 4
  * [36] Anstine D M and Isayev O 2023 Journal of the American Chemical Society 145 8736–8750
  * [37] Sanchez-Lengeling B and Aspuru-Guzik A 2018 Science 361 360–365
  * [38] Repecka D, Jauniskis V, Karpus L, Rembeza E, Rokaitis I, Zrimec J, Poviloniene S, Laurynenas A, Viknander S, Abuajwa W, Savolainen O, Meskys R, Engqvist M K M and Zelezniak A 2021 Nature Machine Intelligence 3 324–333
  * [39] Goldsborough P, Pawlowski N, Caicedo J C, Singh S and Carpenter A E 2017 bioRxiv (Preprint 227645)
  * [40] Arjovsky M, Chintala S and Bottou L 2017 arXiv (Preprint 1701.07875)
  * [41] Gulrajani I, Ahmed F, Arjovsky M, Dumoulin V and Courville A 2017 arXiv (Preprint 1704.00028)
  * [42] Mirza M and Osindero S 2014 arXiv (Preprint 1411.1784)
  * [43] García-Esteban J J, Bravo-Abad J and Cuevas J C 2021 Phys. Rev. Appl. 16 064006
  * [44] Polder D and Van Hove M 1971 Phys. Rev. B 4 3303–3314
  * [45] Song B, Fiorino A, Meyhofer E and Reddy P 2015 AIP Adv. 5 053503
  * [46] Cuevas J C and García-Vidal F J 2018 ACS Photonics 5 3896–3915
  * [47] Biehs S A, Messina R, Venkataram P S, Rodriguez A W, Cuevas J C and Ben-Abdallah P 2021 Rev. Mod. Phys. 93 025009
  * [48] Guo Y, Cortes C L, Molesky S and Jacob Z 2012 Appl. Phys. Lett. 101 131106
  * [49] Biehs S A, Tschikin M and Ben-Abdallah P 2012 Phys. Rev. Lett. 109 104301
  * [50] Guo Y and Jacob Z 2013 Opt. Express 21 15014–15019
  * [51] Biehs S A, Tschikin M, Messina R and Ben-Abdallah P 2013 Appl. Phys. Lett. 102 131106
  * [52] Bright T J, Liu X L and Zhang Z M 2014 Opt. Express 22 A1112–A1127
  * [53] Miller O D, Johnson S G and Rodriguez A W 2014 Phys. Rev. Lett. 112 157402
  * [54] Biehs S A and Ben-Abdallah P 2017 Zeitschrift für Naturforschung A 72 115–127
  * [55] Iizuka H and Fan S 2018 Phys. Rev. Lett. 120 063901
  * [56] Song J, Cheng Q, Lu L, Li B, Zhou K, Zhang B, Luo Z and Zhou X 2020 Phys. Rev. Appl. 13 024054
  * [57] Moncada-Villa E and Cuevas J C 2021 Phys. Rev. B 103 075432
  * [58] Rytov S 1953 Theory of Electric Fluctuations and Thermal Radiation (Bedford: Air Force Cambrige Research Center)
  * [59] Rytov S, Kravtstov Y and Tatarskii V 1989 Principles of Statistical Radiophysics vol 3 (Berlin: Springer-Verlag)
  * [60] Basu S, Zhang Z M and Fu C J 2009 Int. J. Energy Res. 33 1203–1232
  * [61] Caballero B, García-Martín A and Cuevas J C 2012 Phys. Rev. B 85 245103
  * [62] Jolliffe I 2002 Principal Component Analysis (Springer)
  * [63] James G, Witten D, Hastie T and Tibshirani R 2013 An Introduction to Statistical Learning (Springer)
  * [64] Isola P, Zhu J Y, Zhou T and Efros A A 2016 arXiv (Preprint 1611.07004)
  * [65] Ahmed S, Sánchez Muñoz C, Nori F and Kockum A F 2021 Phys. Rev. Lett. 127 140502