# Electron Dynamics at High-Energy Densities in Nickel from Non-linear
Resonant X-ray Absorption Spectra

Robin Y. Engel  Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607
Hamburg, Germany  Department of Physics, Universität Hamburg, Luruper Chaussee
149, 22761 Hamburg, Germany  Oliver Alexander  Imperial College London,
Department of Physics, Exhibition Rd, London SW7 2BX, United Kingdom  Kaan
Atak  Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg,
Germany  Uwe Bovensiepen  Faculty of Physics and Center for Nanointegration
Duisburg-Essen (CENIDE), University of Duisburg-Essen, Lotharstr. 1, 47057
Duisburg, Germany  Institute for Solid State Physics, University of Tokyo,
Kashiwa, Chiba, 277-8581, Japan  Jens Buck  Deutsches Elektronen-Synchrotron
DESY, Notkestr. 85, 22607 Hamburg, Germany  Christian-Albrechts-Universität zu
Kiel, Institut für Experimentelle und Angewandte Physik, Leibnizstraße 11-19,
24118 Kiel, Germany  Robert Carley  European XFEL, Holzkoppel 4, 22869
Schenefeld, Germany  Michele Cascella  MAX IV Laboratory, Lund University, PO
Box 118, SE-221 00 Lund, Sweden  Valentin Chardonnet  Sorbonne Université,
CNRS, Laboratoire de Chimie Physique-Matière et Rayonnement, 4 Pl. Jussieu
Barre 43-44, 75005 Paris, France  Gheorghe Sorin Chiuzbaian  Sorbonne
Université, CNRS, Laboratoire de Chimie Physique-Matière et Rayonnement, 4 Pl.
Jussieu Barre 43-44, 75005 Paris, France  Christian David  Paul Scherrer
Institut, Forschungsstrasse 111, 5232 Villigen, Switzerland  Florian Döring
Paul Scherrer Institut, Forschungsstrasse 111, 5232 Villigen, Switzerland
Andrea Eschenlohr  Faculty of Physics and Center for Nanointegration Duisburg-
Essen (CENIDE), University of Duisburg-Essen, Lotharstr. 1, 47057 Duisburg,
Germany  Natalia Gerasimova  European XFEL, Holzkoppel 4, 22869 Schenefeld,
Germany  Frank de Groot  Utrecht University, Debye Institute for Nanomaterials
Science, Inorganic Chemistry and Catalysis, Princetonplein 1, Universiteitsweg
99, 3584 CC Utrecht, Netherlands  Loïc Le Guyader  European XFEL, Holzkoppel
4, 22869 Schenefeld, Germany  Oliver S. Humphries  European XFEL, Holzkoppel
4, 22869 Schenefeld, Germany  Manuel Izquierdo  European XFEL, Holzkoppel 4,
22869 Schenefeld, Germany  Emmanuelle Jal  Sorbonne Université, CNRS,
Laboratoire de Chimie Physique-Matière et Rayonnement, 4 Pl. Jussieu Barre
43-44, 75005 Paris, France  Adam Kubec  Paul Scherrer Institut,
Forschungsstrasse 111, 5232 Villigen, Switzerland  Tim Laarmann  Deutsches
Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany  The Hamburg
Centre for Ultrafast Imaging CUI, Luruper Chaussee 149, 22761 Hamburg, Germany
Charles-Henri Lambert  Department of Materials, ETH Zurich, 8093 Zurich,
Switzerland  Jan Lüning  Helmholtz-Zentrum Berlin für Materialien und Energie
GmbH, Hahn-Meitner-Platz 1, 14109 Berlin, Germany  Jonathan P. Marangos
Imperial College London, Department of Physics, Exhibition Rd, London SW7 2BX,
United Kingdom  Laurent Mercadier  European XFEL, Holzkoppel 4, 22869
Schenefeld, Germany  Giuseppe Mercurio  European XFEL, Holzkoppel 4, 22869
Schenefeld, Germany  Piter S. Miedema  Deutsches Elektronen-Synchrotron DESY,
Notkestr. 85, 22607 Hamburg, Germany  Katharina Ollefs  Faculty of Physics and
Center for Nanointegration Duisburg-Essen (CENIDE), University of Duisburg-
Essen, Lotharstr. 1, 47057 Duisburg, Germany  Bastian Pfau  Max Born Institute
for Nonlinear Optics and Short Pulse Spectroscopy, Max-Born-Str. 2A, 12489
Berlin, Germany  Benedikt Rösner  Paul Scherrer Institut, Forschungsstrasse
111, 5232 Villigen, Switzerland  Kai Rossnagel  Deutsches Elektronen-
Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany  Christian-Albrechts-
Universität zu Kiel, Institut für Experimentelle und Angewandte Physik,
Leibnizstraße 11-19, 24118 Kiel, Germany  Nico Rothenbach  Faculty of Physics
and Center for Nanointegration Duisburg-Essen (CENIDE), University of
Duisburg-Essen, Lotharstr. 1, 47057 Duisburg, Germany  Andreas Scherz
European XFEL, Holzkoppel 4, 22869 Schenefeld, Germany  Justine Schlappa
European XFEL, Holzkoppel 4, 22869 Schenefeld, Germany  Markus Scholz
Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany
European XFEL, Holzkoppel 4, 22869 Schenefeld, Germany  Jan O. Schunck
Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607 Hamburg, Germany
Department of Physics, Universität Hamburg, Luruper Chaussee 149, 22761
Hamburg, Germany  Kiana Setoodehnia  European XFEL, Holzkoppel 4, 22869
Schenefeld, Germany  Christian Stamm  Department of Materials, ETH Zurich,
8093 Zurich, Switzerland  Institute for Electric Power Systems, University of
Applied Sciences and Arts Northwestern Switzerland, 5210 Windisch, Switzerland
Simone Techert  Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607
Hamburg, Germany  Institute for X-ray Physics, Goettingen University,
Friedrich Hund Platz 1, 37077 Goettingen, Germany  Sam M. Vinko  Department of
Physics, Clarendon Laboratory, University of Oxford, Parks Road, Oxford OX1
3PU, United Kingdom  Central Laser Facility, STFC Rutherford Appleton
Laboratory, Didcot OX11 0QX, United Kingdom  Heiko Wende  Faculty of Physics
and Center for Nanointegration Duisburg-Essen (CENIDE), University of
Duisburg-Essen, Lotharstr. 1, 47057 Duisburg, Germany  Alexander A.
Yaroslavtsev  Uppsala University, Department of Physics and Astronomy,
Regementsvägen 1 Uppsala, Sweden  Zhong Yin  International Center for
Synchrotron Radiation Innovation Smart, Tohoku University, 2-1-1 Katahira,
Aoba-ku, Sendai, Miyagi 980-8577, Japan  ETH Zürich, Laboratorium für
Physikalische Chemie, Vladimir-Prelog-Weg 1-5, 8093 Zürich, Switzerland
Martin Beye  Deutsches Elektronen-Synchrotron DESY, Notkestr. 85, 22607
Hamburg, Germany  Department of Physics, Universität Hamburg, Luruper Chaussee
149, 22761 Hamburg, Germany<EMAIL_ADDRESS>
###### Abstract

The pulse intensity from X-ray free-electron lasers (FELs) can create extreme
excitation densities in solids, entering the regime of non-linear X-ray-matter
interactions. We show $L_{3}$-edge absorption spectra of metallic nickel thin
films with fluences entering a regime where several X-ray photons are incident
per absorption cross-section. Main features of the observed non-linear
spectral changes are described with a predictive rate model for electron
population dynamics during the pulse, utilizing a fixed density of states and
tabulated ground-state properties.

The modern understanding of complex solid materials relies on appropriate
approximations to the unabridged quantum mechanical description of the full,
correlated many-body problem. To assess the predictive power of theoretical
models and the selected approximations, detailed experimental studies far away
from known territory are especially insightful. Absorbing the high power
densities available from an X-ray free-electron laser (FEL) in a solid metal
generates a very unusual state of warm dense matter far from equilibrium:
Individual electronic excitations reach up to hundreds of eV and excitation
levels average out to many eV per atom [1, 2, 3, 4, 5, 6, 7, 8, 9]. As the
absorption of an intense X-ray pulse depends on the changes it drives in the
electronic system [10, 11, 12, 13, 14, 15, 16, 15], a single-pulse non-linear
absorption measurement can be used to investigate its evolution on the
timescale of the pulse duration.

We present fluence-dependent X-ray absorption spectra recorded with
monochromatic X-rays on metallic nickel thin films around the nickel
2$p_{3/2}$ ($L_{3}$) edge, revealing a changing valence electron system around
the Fermi level as a consequence of the high excitation densities from
fluences up to 60 J/cm2 (corresponding to 2$\times 10^{15}$ W/cm2).

The electronic processes that ensue after the absorption of photons at core
levels trigger a complex dynamical process that is challenging to treat in
purely ab-initio simulations [17, 18, 19, 20, 21]. Here, we take an
alternative approach and develop a simple rate equation model that provides an
intuitive understanding of the relevant processes [22]. The resulting picture
of the evolution of electronic populations within a fixed ground-state density
of states successfully describes the largest part of the non-linear changes in
the spectra. This corroborates the dominant impact of electron redistribution
from the strong non-equilibrium state towards a thermalized electronic system.
Some of the observed changes, especially in the close vicinity of the
resonance, deviate from the predictions of the rate model and call for more
evolved theories. Here, our work provides a benchmark to identify observations
of advanced physical processes and effects. While this letter discusses the
experiment and resulting insights, we lay out the framework of the model in
detail in a separate publication [22].

Additionally, our straightforward picture of intense core-resonant X-ray pulse
interaction with the valence system of a 3$d$ metal lays a solid knowledge-
based foundation for the planning and interpretation of non-linear X-ray
spectroscopy experiments at FELs; in particular, the relevance of electronic
scattering processes observed here is expected to affect methods relying on
stimulated emission from core excitations and X-ray or X-ray/optical wave-
mixing [23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38].

Figure 1:  (a&b) Sketch of absorption at different fluences. The unoccupied
states determine the XAS spectrum as they are probed by core-resonant photons.
(a) In the low-fluence case (blue unoccupied states and resulting spectrum),
the electronic system mostly remains in the ground state. (b) In the high-
fluence case (yellow unoccupied states and spectrum), later parts of the X-ray
pulse probe a hot electronic system and experience spectral bleaching at the
probed photon energy.
Setup for non-linear XAS (c) The split-beam-normalization scheme uses a
special zone plate [39], which generates two adjacent beam foci for
transmission through the sample and a reference membrane before the beams
impinge on the detector.

X-ray absorption spectra of the nickel 2$p_{3/2}$ ($L_{3}$) edge were recorded
at the Spectroscopy and Coherent Scattering Instrument (SCS) of the European
XFEL [40].

The XAS spectra were measured by continuously scanning the SASE3 monochromator
[41] (synchronized with the undulator gap) back-and-forth many times in the
range 846-856 eV. The photon bandwidth was about 420 meV and the FEL pulse
duration on the sample was about 30 fs FWHM. The overall beam intensity was
controlled using a gas attenuator filled with nitrogen and monitored using an
X-ray gas-monitor (XGM) downstream of the monochromator [42, 43].

For X-ray absorption measurements at FELs based on Self-Amplified Spontaneous
Emission (SASE), beam-splitting schemes can deliver optimal normalization of
SASE-fluctuations [44, 45, 46]. Here, a focusing and beam-splitting zone plate
also creates the required tight focusing to achieve extreme fluences. Figure 1
shows the schematic experimental layout.

The zone plate combines an off-axis Fresnel structure for focusing and a line
grating for beam-splitting in a single optical element [39]. It thus produces
two $\mu$m-sized, identical foci in the sample plane, 1.9 mm apart,
originating from the first-order diffraction of the zone plate, as well as the
positive and negative first orders of the line grating. The sample has a
square support of 25 mm size, containing Si3N4 membrane windows (orange in
Figure 1) of 0.5 mm size and 200 nm thickness with a distance of 2 mm between
adjacent windows. Every second pair of rows (blue in Figure 1) was
additionally coated with a 20 $\mathrm{nm}$ sample layer of polycrystalline
metallic Ni by sputter deposition, on top of a 2 $\mathrm{nm}$ bonding layer
of Ta; a 2 $\mathrm{nm}$ Pt capping layer prevents oxidation during sample-
handling.

The sample frame was positioned such that one zone plate focus impinged on a
nickel-coated membrane, while the other hit a bare silicon-nitride membrane.
Thus, the difference in transmission of both beams can be attributed solely to
the nickel film.

The detector was a fast readout-speed charge-coupled device (FastCCD) with
high dynamic range, enabling 10 Hz read-out and increasing the fluence range
available to the experiment [47, 48, 49]. Due to the unstable detector
temperature, significant retroactive calibration of the detector was necessary
(see supplement). To prevent detector saturation, an additional aluminum
filter of about 13 $\mathrm{\mu m}$ thickness was used between sample and
detector for measurements with the unattenuated beam.

During these high-intensity measurements, sample and reference films were
locally damaged by intense individual FEL shots. Thus, the FEL was operated in
single-shot mode at 10 Hz repetition rate, and the sample was scanned through
the beam continuously at 0.5 $\mathrm{mm\cdot s^{-1}}$, resulting in 10 shots
per membrane window.

The shot craters in the reference membranes were later analyzed with scanning
electron microscopy (SEM) to determine the effective focal size at specific
photon energies. The resulting spot sizes were used to calibrate ray-tracing
calculations which delivered the photon-energy-dependent spot size, ranging
from 0.4 $\mathrm{\mu m^{2}}$ to about 3 $\mathrm{\mu m^{2}}$ (see supplement
for details on the spot size determination).

Figure 2: Fluence-dependent Ni $L_{3}$-edge spectra, measured (top) and
simulated (bottom). The fluence of events contributing to each spectrum is
given in the legend in terms of mean and standard deviation. Dashed simulated
spectra do not have a corresponding measurement. The regions of interest from
which absorbance changes shown in panels b), d), and e) of Figure 3 were
quantified are shaded and labeled (I), (II), and (III), respectively. The
error bars are shown for the measured spectra and represent the 95% confidence
intervals for each bin of 102 meV width; solid lines of the measured spectra
are smoothed using a Savitzky-Golay filter using windows of 21 bins and 4th-
order polynomials. The experimental spectra are vertically offset by 100 mOD.

Figure 2 shows the spectra for the nickel $L_{3}$-edge next to simulated
spectra for increasing X-ray fluence over more than 3 orders or magnitude,
from 0.03 to 60 J/cm2. Each measured point represents an average of several
FEL shots, sorted by X-ray fluence and photon energy. The varying statistical
uncertainty is a result of the pulse intensity fluctuations of monochromatized
SASE radiation [50] in combination with photon energy-dependent spot sizes
(see supplement for details on the shot sorting).

We observe four main fluence-dependent effects, which we quantify and compare
to the simulated results in Figure 3: a) a red-shift of the absorption peak of
up to 0.9 $\pm$0.1 eV in the rising flank; b) an increase of the pre-edge
absorbance, as the rising edge of the absorption peak shifts and broadens; c)
a reduced peak absorbance and d), e) a reduced post-edge absorbance. The
integration regions from which the effects b), d) and e) are derived, are
highlighted in Figure 2 as (I), (II) and (III), respectively. The shift of the
absorption edge is quantified by the photon energy at which the absorbance
reaches half of the peak value; its uncertainty is propagated from the
statistical uncertainty of the absorption peak measurement.

Before we analyze these observations in detail, let us quickly paraphrase our
modeling approach [22]: In contrast to earlier rate models [51, 12], we
describe the evolution of the electronic system with an energy-resolved
population of the valence band. Tracking the full non-thermal population
history proved crucial to describe the non-linear absorption changes near and
around the Fermi level. As coupling between electrons and phonons in metals is
typically not yet important on the timescale of 30 fs [52, 53, 54] and we do
not account for collective electron correlation effects, we test the
approximation that the Density of States (DoS) remains unchanged within the
pulse duration.

Transition rates between electronic states are determined by scaling ground
state rates with the populations of initial and final states. The relevant
process rates are compiled into differentials of electronic populations and
photon density in space and time and implemented in a finite-element
simulation to derive the electron population history and ultimately the X-ray
transmission of a three-dimensional sample.

The model implements the processes of resonant absorption from the 2$p_{3/2}$
core level and non-resonant absorption from other (mostly valence) electrons.
Stimulated emission is described as a time-inverted resonant absorption
process. Electronic thermalization is modeled with a bulk timescale
$\tau_{\mathrm{th}}$ (essentially quantifying electron-electron scattering)
that moves the non-thermal valence electron distribution towards a target
Fermi-Dirac distribution that corresponds to the momentary internal energy and
population of the valence band. Finally, scattering cascades initiated by fast
Auger-electrons and photo-electrons from non-resonant absorption are
parameterized by another scattering time $\tau_{\mathrm{scatt}}$.

With this simple description of the underlying processes, we provide a
microscopic picture of the electronic system and its interaction with resonant
X-rays as a complementary approach to more complex calculations [20, 21].

Solely considering the population dynamics of the electronic system, the
simulation already achieves good agreement with the experimental data across
more than three orders of magnitude in fluence. This is particularly
remarkable since nearly all input parameters are experimental parameters or
well-known ground-state properties of the material, such as density,
electronic configuration, and ground-state spectrum. Only the valence
thermalization time $\tau_{\mathrm{th}}$ and electron scattering time
$\tau_{scatt}$ were varied to achieve the best match to the experimental
results. The found value $\tau_{\mathrm{th}}=6$ fs compares well to recent
estimates for excitations on this energy scale [33, 38, 55, 56].

The time constant $\tau_{\mathrm{{scatt}}}=1.5$ fs characterizes the secondary
electron scattering cascade which transfers energy (and population) from fast
electrons to (unoccupied) valence states. The constant summarizes many
individual electron scattering events and compares to the tabulated time
between individual collisions in ground-state nickel of roughly 100
attoseconds [57].

Figure 3: Comparison of spectral effects between simulation (blue lines) and
experiment (orange lines with error bars). The shift of absorption edge in
panel a) represents the photon energy at which the half-maximum of the
absorption peak is reached. The absorbance changes in panels b), d) and e) are
integrated from the gray shaded regions in Figure 2, while panel c) shows the
global maximum of the spectrum. Figure 4:  Evolution of electronic populations
(simulation) in a single voxel at the sample surface for a pulse of 858.3 eV,
with a pulse energy of 30 $\mathrm{J/cm^{2}}$. Panel a) shows the total DoS
used as an input for the simulation. Panel b) shows the energy-resolved
(relative to the Fermi energy) occupation of the valence band over time. The
population (in electrons/atom/eV) is the product of the DoS and the
occupation. The thermalized valence occupation lags a few femtoseconds behind
the current chemical potential $\mu$; the temperature $T$ of the valence
system rises rapidly, ultimately reaching up to 25 eV. The bleaching of
valence states (highlighted with a blue dotted ellipse) is visible as a high
non-thermal population at the resonant photon energy around 7 eV above the
Fermi level. Panel c) shows the number of core holes and free electrons over
time, as well as the number of electrons in the valence system below and above
the Fermi energy.

Figure 4 shows an example of a simulated valence band population history,
specifically from the uppermost 4 Å thick layer of the sample, excited with a
Gaussian pulse profile centered around $t=0$ with 30 fs FWHM duration and 30
J/cm2 fluence. While panel a) shows the calculated DoS as used by the
simulation and published in [58, 59], the colormap in b) shows the occupation
of these states over time. Panel c) shows the number of electrons per atom in
the valence band below and above the Fermi level (blue solid and dashed
curves, respectively) as well as the average number of core holes and the
number of free electrons over time. Even though the direct interaction with
the photons creates core holes via resonant absorption and free electrons via
non-resonant absorption, the excitation energy of both processes is so quickly
transferred to the valence electrons that only the valence electron
distribution ever deviates strongly from the ground state. By the end of the
pulse in this example, more than half of the $3d$ valence electrons are
excited to valence states above the Fermi level, while the highest
instantaneous number of core holes was only about one per 100 atoms, as shown
in Figure 4 c). Due to the small-bandwidth excitation, the core- and resonant
valence states operate like a two-level system. Since the number of resonant
valence states is small in comparison to the number of core electrons, the
resonant absorption process saturates due to occupied valence states long
before the core level is depleted. A heated Fermi-Dirac distribution further
contributes to the occupation of states above the Fermi level.

Since in our experiment, the same monochromatic pulse excites and probes the
sample, the situation is different for energies below the edge: absorption
only rises after non-resonant absorption has led to sufficient electronic
heating until the tail of the hot hole distribution reaches the probed energy.
Only then, additional resonant absorption begins to occur and accelerates
further electronic heating and in turn additional pre-edge absorption. Since
this process occurs exponentially faster near the absorption edge, it
contributes significantly to the observed spectral red-shift (see Figure 3 a)
and b)).

Another cause of the observed edge shift is the shift of the chemical
potential $\mu$, which strongly depends on the exact shape of the DoS and is
shown in Figure 4 b) as a green line. Initially, $\mu$ increases with absorbed
fluence, as thermally excited electrons from the $3d$ states must spread out
in energy to the lower DoS above the Fermi level. With rising electronic
temperature, the high DoS of the $3d$ states becomes less relevant and the
chemical potential drops again as expected in regular metals. A similar
evolution of the chemical potential and electronic temperature was predicted
for optically excited nickel by previous experiments and calculations [60, 61,
62, 4].

A significant deviation between model and experiment can be observed at the
resonance peak itself, where the simulated electron dynamics lead us to expect
a much stronger saturation effect than observed experimentally (Figure 3 c)).
This underestimation may be related to a fluence-dependent decrease of the
excited state lifetime and consequent energetic broadening of the resonant
core-valence transition, which is not considered in our model. While it is
unsurprising to find additional resonant effects in the resonance peak itself,
the lack of any significant saturation around 852 eV (Figure 3 d)) is even
more surprising. Both disagreements point to additional physical effects and
call for more sophisticated models.

We speculatively propose mechanisms which could contribute to these
disagreements: The transition matrix elements could get modified at higher
excitation densities, especially around the resonance, while we model the
absorption only based on the ground-state spectrum. An energy dependence of
the electron-electron scattering cross-section could allow for particularly
fast scattering of electrons with certain energies, counteracting the
saturation. Furthermore, a collective, correlated response of the electronic
system could modify the DoS or the transitions even on the fast time scale of
the FEL pulse duration [63]. Despite these remaining discrepancies, the main
aspects of the spectral changes are covered in our very simple population
dynamics model.

We want to point out that substantially smaller spectral red-shifts were
observed before in nickel after excitation with optical lasers, albeit at
three orders of magnitude lower excitation fluence. These required
qualitatively different interpretations [64, 65, 56, 63], where the
explanation for time-dependent changes included a variable DoS, calculated
using (Time-Dependent) Density Functional Theory (TD)DFT; this dependency is
overshadowed in our high-fluence study by the effects of electron population
dynamics.

To summarize, we interpret the fluence-dependent near-edge X-ray absorption
spectra of the nickel 2$p_{3/2}$ core level at X-ray fluences of up to 60
$\mathrm{J/cm^{2}}$. We propose a rate-equation model, describing the various
excitation and decay processes that connect core- and valence electronic
states using differential equations based on scaling of known ground-state
properties with the evolving electron populations. For the measured spectra of
metallic nickel, the model successfully predicts the increase of absorption
before and its decrease beyond the resonance, as well as the observed shift of
the absorption peak over more than three orders of magnitude in fluence.

It therefore allows us to identify the most important processes responsible
for spectral changes: Heating of valence electrons due to secondary electron
cascades from Auger electrons, as well as electrons emitted from the valence
band due to non-resonant absorption, appeared particularly relevant.
Furthermore, saturation appears dominated to by the heated valence states
rather than the core holes.

This study provides the fingerprints of how strong X-ray fluences may alter
the electronic system and thus the spectra in studies, where the X-ray pulses
were originally assumed to be non-disturbing. It becomes clear that a complete
modeling of high-fluence spectra needs to build upon dominant population
dynamics and requires special treatment around resonances. This provides an
excellent benchmark for sophisticated theories. Our results also apply to the
resonant regime which is particularly interesting for pioneering non-linear
X-ray studies.

###### Acknowledgements.

We acknowledge European XFEL in Schenefeld, Germany, for provision of X-ray
free-electron laser beamtime at the SCS instrument and would like to thank the
staff for their assistance. Funding by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) - Project-ID 278162697 - SFB 1242 and the
Helmholtz Association (grant VH-NG-1105) is gratefully acknowledged. Access to
Synchrotron SOLEIL through proposal ID 20160880 for characterization of static
properties of the Ni films is acknowledged. Parts of this work were funded by
the Swiss National Science Foundation (Grants No. PZ00P2-179944)

## Author Contributions

M.B., C.D., F.D., L.L.G., J.L., J.P.M., B.R. and S.T. conceptualized and
planned the experiment; M.C., C.D., F.D., N.G., L.L.G., M.I., E.J., A.K.,
C.-H.L., L.M., B.P., B.R., A.S., K.S., C.S., H.W. and A.Y. prepared the
measurement apparatus and samples; O.A., K.A., M.B., J.B., R.C., M.C., V.C.,
G.S.C., C.D., F.D., R.Y.E., A.E., N.G., L.L.G., O.S.H., M.I., L.M., G.M.,
P.S.M., B.R., N.R., A.S., J.S., S.T. A.Y. and Z.Y. performed the experiment;
O.A., R.Y.E, L.L.G., O.S.H., B.R. and N.R. analyzed and visualized the
results; M.B. and R.Y.E. wrote the manuscript; M.B., U.B., C.D., R.Y.E., A.E.,
L.L.G., O.S.H., M.I., E.J., T.L., P.S.M., K.O., K.R., M.S., J.O.S., S.M.V.,
H.W. and Z.Y. reviewed and edited the manuscript; M.B. and J.P.M. supervised
or administered the project.

## References

  * Zastrau  _et al._ [2008] U. Zastrau, C. Fortmann, R. R. Fäustlin, L. F. Cao, T. Döppner, S. Düsterer, S. H. Glenzer, G. Gregori, T. Laarmann, H. J. Lee, A. Przystawik, P. Radcliffe, H. Reinholz, G. Röpke, R. Thiele, J. Tiggesbäumker, N. X. Truong, S. Toleikis, I. Uschmann, A. Wierling, T. Tschentscher, E. Förster, and R. Redmer, Bremsstrahlung and line spectroscopy of warm dense aluminum plasma heated by xuv free-electron-laser radiation, Phys. Rev. E 78, 066406 (2008).
  * Vinko  _et al._ [2012] S. M. Vinko, O. Ciricosta, B. I. Cho, K. Engelhorn, H.-K. Chung, C. R. D. Brown, T. Burian, J. Chalupský, R. W. Falcone, C. Graves, V. Hájková, A. Higginbotham, L. Juha, J. Krzywinski, H. J. Lee, M. Messerschmidt, C. D. Murphy, Y. Ping, A. Scherz, W. Schlotter, S. Toleikis, J. J. Turner, L. Vysin, T. Wang, B. Wu, U. Zastrau, D. Zhu, R. W. Lee, P. A. Heimann, B. Nagler, and J. S. Wark, Creation and diagnosis of a solid-density plasma with an X-ray free-electron laser, Nature 482, 59 (2012).
  * Cho  _et al._ [2012] B. I. Cho, K. Engelhorn, S. M. Vinko, H.-K. Chung, O. Ciricosta, D. S. Rackstraw, R. W. Falcone, C. R. D. Brown, T. Burian, J. Chalupský, C. Graves, V. Hájková, A. Higginbotham, L. Juha, J. Krzywinski, H. J. Lee, M. Messersmidt, C. Murphy, Y. Ping, N. Rohringer, A. Scherz, W. Schlotter, S. Toleikis, J. J. Turner, L. Vysin, T. Wang, B. Wu, U. Zastrau, D. Zhu, R. W. Lee, B. Nagler, J. S. Wark, and P. A. Heimann, Resonant K$\alpha$ Spectroscopy of Solid-Density Aluminum Plasmas, Physical Review Letters 109, 245003 (2012).
  * Humphries  _et al._ [2020] O. S. Humphries, R. S. Marjoribanks, Q. Y. van den Berg, E. C. Galtier, M. F. Kasim, H. J. Lee, A. J. F. Miscampbell, B. Nagler, R. Royle, J. S. Wark, and S. M. Vinko, Probing the electronic structure of warm dense nickel via resonant inelastic x-ray scattering, Phys. Rev. Lett. 125, 195001 (2020).
  * García Saiz  _et al._ [2008] E. García Saiz, G. Gregori, D. O. Gericke, J. Vorberger, B. Barbrel, R. J. Clarke, R. R. Freeman, S. H. Glenzer, F. Y. Khattak, M. Koenig, O. L. Landen, D. Neely, P. Neumayer, M. M. Notley, A. Pelka, D. Price, M. Roth, M. Schollmeier, C. Spindloe, R. L. Weber, L. Van Woerkom, K. Wünsch, and D. Riley, Probing warm dense lithium by inelastic X-ray scattering, Nature Physics 4, 940 (2008).
  * Vinko [2015] S. M. Vinko, X-ray free-electron laser studies of dense plasmas, Journal of Plasma Physics 81, 365810501 (2015).
  * Bailey  _et al._ [2015] J. E. Bailey, T. Nagayama, G. P. Loisel, G. A. Rochau, C. Blancard, J. Colgan, P. Cosse, G. Faussurier, C. J. Fontes, F. Gilleron, I. Golovkin, S. B. Hansen, C. A. Iglesias, D. P. Kilcrease, J. J. MacFarlane, R. C. Mancini, S. N. Nahar, C. Orban, J. C. Pain, A. K. Pradhan, M. Sherrill, and B. G. Wilson, A higher-than-predicted measurement of iron opacity at solar interior temperatures, Nature 517, 56 (2015).
  * Hollebon  _et al._ [2019] P. Hollebon, O. Ciricosta, M. P. Desjarlais, C. Cacho, C. Spindloe, E. Springate, I. C. E. Turcu, J. S. Wark, and S. M. Vinko, Ab initio simulations and measurements of the free-free opacity in aluminum, Physical Review E 100, 043207 (2019), arXiv:1806.02726 .
  * Preston  _et al._ [2017] T. R. Preston, S. M. Vinko, O. Ciricosta, P. Hollebon, H.-K. Chung, G. L. Dakovski, J. Krzywinski, M. Minitti, T. Burian, J. Chalupský, V. Hájková, L. Juha, V. Vozda, U. Zastrau, R. W. Lee, and J. S. Wark, Measurements of the K-Shell Opacity of a Solid-Density Magnesium Plasma Heated by an X-Ray Free-Electron Laser, Physical Review Letters 119, 085001 (2017).
  * Nagler  _et al._ [2009] B. Nagler, U. Zastrau, R. R. Fäustlin, S. M. Vinko, T. Whitcher, A. J. Nelson, R. Sobierajski, J. Krzywinski, J. Chalupsky, E. Abreu, S. Bajt, T. Bornath, T. Burian, H. Chapman, J. Cihelka, T. Döppner, S. Düsterer, T. Dzelzainis, M. Fajardo, E. Förster, C. Fortmann, E. Galtier, S. H. Glenzer, S. Göde, G. Gregori, V. Hajkova, P. Heimann, L. Juha, M. Jurek, F. Y. Khattak, A. R. Khorsand, D. Klinger, M. Kozlova, T. Laarmann, H. J. Lee, R. W. Lee, K.-H. Meiwes-Broer, P. Mercere, W. J. Murphy, A. Przystawik, R. Redmer, H. Reinholz, D. Riley, G. Röpke, F. Rosmej, K. Saksl, R. Schott, R. Thiele, J. Tiggesbäumker, S. Toleikis, T. Tschentscher, I. Uschmann, H. J. Vollmer, and J. S. Wark, Turning solid aluminium transparent by intense soft x-ray photoionization, Nature Physics 5, 693 (2009).
  * Recoules and Mazevet [2009] V. Recoules and S. Mazevet, Temperature and density dependence of XANES spectra in warm dense aluminum plasmas, Physical Review B 80, 064110 (2009).
  * Di Cicco  _et al._ [2014] A. Di Cicco, K. Hatada, E. Giangrisostomi, R. Gunnella, F. Bencivenga, E. Principi, C. Masciovecchio, and A. Filipponi, Interplay of electron heating and saturable absorption in ultrafast extreme ultraviolet transmission of condensed matter, Phys. Rev. B 90, 220303(R) (2014).
  * Rackstraw  _et al._ [2015] D. S. Rackstraw, O. Ciricosta, S. M. Vinko, B. Barbrel, T. Burian, J. Chalupský, B. I. Cho, H.-K. Chung, G. L. Dakovski, K. Engelhorn, V. Hájková, P. Heimann, M. Holmes, L. Juha, J. Krzywinski, R. W. Lee, S. Toleikis, J. J. Turner, U. Zastrau, and J. S. Wark, Saturable absorption of an x-ray free-electron-laser heated solid-density aluminum plasma, Phys. Rev. Lett. 114, 015003 (2015).
  * Principi  _et al._ [2016] E. Principi, E. Giangrisostomi, R. Cucini, F. Bencivenga, A. Battistoni, A. Gessini, R. Mincigrucci, M. Saito, S. Di Fonzo, F. D’Amico, A. Di Cicco, R. Gunnella, A. Filipponi, A. Giglia, S. Nannarone, and C. Masciovecchio, Free electron laser-driven ultrafast rearrangement of the electronic structure in Ti, Structural Dynamics 3, 023604 (2016).
  * Chen  _et al._ [2018] Z. Chen, D. J. Higley, M. Beye, M. Hantschmann, V. Mehta, O. Hellwig, A. Mitra, S. Bonetti, M. Bucher, S. Carron, T. Chase, E. Jal, R. Kukreja, T. Liu, A. H. Reid, G. L. Dakovski, A. Föhlisch, W. F. Schlotter, H. A. Dürr, and J. Stöhr, Ultrafast Self-Induced X-Ray Transparency and Loss of Magnetic Diffraction, Physical Review Letters 121, 137403 (2018).
  * Yoneda  _et al._ [2014] H. Yoneda, Y. Inubushi, M. Yabashi, T. Katayama, T. Ishikawa, H. Ohashi, H. Yumoto, K. Yamauchi, H. Mimura, and H. Kitamura, Saturable absorption of intense hard x-rays in iron, Nature communications 5, 5080 (2014).
  * Chen and Lopata [2020] M. Chen and K. Lopata, First-Principles Simulations of X-ray Transient Absorption for Probing Attosecond Electron Dynamics, Journal of Chemical Theory and Computation 16, 4470 (2020).
  * Mo  _et al._ [2020] C. Mo, Z.-G. Fu, P. Zhang, W. Kang, W. Zhang, and X. T. He, First-principles method for x-ray Thomson scattering including both elastic and inelastic features in warm dense matter, Physical Review B 102, 195127 (2020).
  * Williams and Fajardo [2020] G. O. Williams and M. Fajardo, Collisional ionization and recombination in degenerate plasmas beyond the free-electron-gas approximation, Phys. Rev. E 102, 063204 (2020).
  * Medvedev  _et al._ [2018] N. Medvedev, V. Tkachenko, V. Lipp, Z. Li, and B. Ziaja, Various damage mechanisms in carbon and silicon materials under femtosecond x-ray irradiation, 4open 1, 3 (2018).
  * Lipp  _et al._ [2022] V. Lipp, V. Tkachenko, M. Stransky, B. Aradi, T. Frauenheim, and B. Ziaja, Density functional tight binding approach utilized to study x-ray-induced transitions in solid materials, Scientific reports 12, 1551 (2022).
  * Engel  _et al._ [2022] R. Y. Engel, M. Scholz, J. O. Schunck, and M. Beye, A rate-model for nonlinear x-ray near edge absorption spectra, Jointly submitted to Phys. Rev. B (2022).
  * Mukamel [2005] S. Mukamel, Multiple core-hole coherence in x-ray four-wave-mixing spectroscopies, Phys. Rev. B 72, 235110 (2005).
  * Glover  _et al._ [2012] T. E. Glover, D. M. Fritz, M. Cammarata, T. K. Allison, S. Coh, J. M. Feldkamp, H. Lemke, D. Zhu, Y. Feng, R. N. Coffee, M. Fuchs, S. Ghimire, J. Chen, S. Shwartz, D. A. Reis, S. E. Harris, and J. B. Hastings, X-ray and optical wave mixing, Nature 488, 603 (2012).
  * Beye  _et al._ [2013] M. Beye, S. Schreck, F. Sorgenfrei, C. Trabant, N. Pontius, C. Schüßler-Langeheine, W. Wurth, and A. Föhlisch, Stimulated x-ray emission for materials science, Nature 501, 191 (2013).
  * Weninger  _et al._ [2013] C. Weninger, M. Purvis, D. Ryan, R. A. London, J. D. Bozek, C. Bostedt, A. Graf, G. Brown, J. J. Rocca, and N. Rohringer, Stimulated electronic x-ray raman scattering, Phys. Rev. Lett. 111, 233902 (2013).
  * Shwartz  _et al._ [2014] S. Shwartz, M. Fuchs, J. B. Hastings, Y. Inubushi, T. Ishikawa, T. Katayama, D. A. Reis, T. Sato, K. Tono, M. Yabashi, S. Yudovich, and S. E. Harris, X-ray second harmonic generation, Physical Review Letters 112, 163901 (2014).
  * Tamasaku  _et al._ [2014] K. Tamasaku, E. Shigemasa, Y. Inubushi, T. Katayama, K. Sawada, H. Yumoto, H. Ohashi, H. Mimura, M. Yabashi, K. Yamauchi, and T. Ishikawa, X-ray two-photon absorption competing against single and sequential multiphoton processes, Nature Photonics 8, 313 (2014).
  * Bencivenga  _et al._ [2015] F. Bencivenga, R. Cucini, F. Capotondi, A. Battistoni, R. Mincigrucci, E. Giangrisostomi, A. Gessini, M. Manfredda, I. P. Nikolov, E. Pedersoli, E. Principi, C. Svetina, P. Parisse, F. Casolari, M. B. Danailov, M. Kiskinova, and C. Masciovecchio, Four-wave mixing experiments with extreme ultraviolet transient gratings, Nature 520, 205 (2015).
  * Schreck  _et al._ [2015] S. Schreck, M. Beye, and A. Föhlisch, Implications of stimulated resonant x-ray scattering for spectroscopy, imaging, and diffraction in the regime from soft to hard x-rays, Journal of Modern Optics 62, S34 (2015).
  * Lam  _et al._ [2018] R. K. Lam, S. L. Raj, T. A. Pascal, C. D. Pemmaraju, L. Foglia, A. Simoncig, N. Fabris, P. Miotti, C. J. Hull, A. M. Rizzuto, J. W. Smith, R. Mincigrucci, C. Masciovecchio, A. Gessini, E. Allaria, G. De Ninno, B. Diviacco, E. Roussel, S. Spampinati, G. Penco, S. Di Mitri, M. Trovò, M. Danailov, S. T. Christensen, D. Sokaras, T. C. Weng, M. Coreno, L. Poletto, W. S. Drisdell, D. Prendergast, L. Giannessi, E. Principi, D. Nordlund, R. J. Saykally, and C. P. Schwartz, Soft x-ray second harmonic generation as an interfacial probe, Physical Review Letters 120, 023901 (2018).
  * Tamasaku  _et al._ [2018] K. Tamasaku, E. Shigemasa, Y. Inubushi, I. Inoue, T. Osaka, T. Katayama, M. Yabashi, A. Koide, T. Yokoyama, and T. Ishikawa, Nonlinear spectroscopy with x-ray two-photon absorption in metallic copper, Physical Review Letters 121, 083901 (2018).
  * Higley  _et al._ [2019] D. J. Higley, A. H. Reid, Z. Chen, A. A. L. Loïc Le Guyader and, T. Liu, T. Chase, G. L. Dakovski, A. Mitra, E. Yuan, H. A. Dürr, W. F. Schlotter, and J. Stöhr, Femtosecond x-ray induced changes of the electronic and magnetic response of solids from electron redistribution, Nature communications 10, 5289 (2019).
  * Rottke  _et al._ [2022] H. Rottke, R. Y. Engel, D. Schick, J. O. Schunck, P. S. Miedema, M. C. Borchert, M. Kuhlmann, N. Ekanayake, S. Dziarzhytski, G. Brenner, U. Eichmann, C. von Korff Schmising, M. Beye, and S. Eisebitt, Probing electron and hole colocalization by resonant four-wave mixing spectroscopy in the extreme ultraviolet, Science Advances 8, eabn5127 (2022).
  * Rouxel  _et al._ [2021] J. R. Rouxel, D. Fainozzi, R. Mankowsky, B. Rösner, G. Seniutinas, R. Mincigrucci, S. Catalini, L. Foglia, R. Cucini, F. Döring, A. Kubec, F. Koch, F. Bencivenga, A. A. Haddad, A. Gessini, A. A. Maznev, C. Cirelli, S. Gerber, B. Pedrini, G. F. Mancini, E. Razzoli, M. Burian, H. Ueda, G. Pamfilidis, E. Ferrari, Y. Deng, A. Mozzanica, P. J. M. Johnson, D. Ozerov, M. G. Izzo, C. Bottari, C. Arrell, E. J. Divall, S. Zerdane, M. Sander, G. Knopp, P. Beaud, H. T. Lemke, C. J. Milne, C. David, R. Torre, M. Chergui, K. A. Nelson, C. Masciovecchio, U. Staub, L. Patthey, and C. Svetina, Hard X-ray transient grating spectroscopy on bismuth germanate, Nature Photonics 15, 499 (2021).
  * Bencivenga  _et al._ [2021] F. Bencivenga, R. Mincigrucci, F. Capotondi, A. Calvi, R. Cucini, L. Foglia, E. Pedersoli, E. Principi, A. Simoncig, P. Cinquegrana, M. B. Danailov, G. De Ninno, S. Di Mitri, G. Gaio, A. Gessini, L. Giannessi, N. Mahne, M. Manfredda, I. P. Nikolov, G. M. Penco, L. Raimondi, P. R. Ribic, C. Svetina, M. Trovo, M. Zangrando, and C. Masciovecchio, An approach for realizing four-wave-mixing experiments stimulated by two-color extreme ultraviolet pulses, in  _International Conference on X-Ray Lasers 2020_, July 2021, edited by D. Bleiner (SPIE, 2021) p. 28.
  * Wirok-Stoletow  _et al._ [2022] S. Wirok-Stoletow, R. Jin, D. Kolbasova, S.-K. Son, A. Aquila, and R. Santra, Nonsequential two-photon absorption in solid Ge irradiated by an intense x-ray free-electron-laser pulse, Physical Review A 106, 023118 (2022).
  * Higley  _et al._ [2022] D. J. Higley, Z. Chen, M. Beye, M. Hantschmann, A. H. Reid, V. Mehta, O. Hellwig, G. L. Dakovski, A. Mitra, R. Y. Engel, T. Maxwell, Y. Ding, S. Bonetti, M. Bucher, S. Carron, T. Chase, E. Jal, R. Kukreja, T. Liu, A. Föhlisch, H. A. Dürr, W. F. Schlotter, and J. Stöhr, Stimulated resonant inelastic x-ray scattering in a solid, Communications Physics 5, 1 (2022).
  * Döring  _et al._ [2020] F. Döring, B. Rösner, M. Langer, A. Kubec, A. Kleibert, J. Raabe, C. A. F. Vaz, M. Lebugle, and C. David, Multifocus off-axis zone plates for x-ray free-electron laser experiments, Optica 7, 1007 (2020).
  * Tschentscher  _et al._ [2017] T. Tschentscher, C. Bressler, J. Grünert, A. Madsen, A. Mancuso, M. Meyer, A. Scherz, H. Sinn, and U. Zastrau, Photon Beam Transport and Scientific Instruments at the European XFEL, Applied Sciences 7, 592 (2017).
  * Gerasimova  _et al._ [2022] N. Gerasimova, D. La Civita, L. Samoylova, M. Vannoni, R. Villanueva, D. Hickin, R. Carley, R. Gort, B. E. Van Kuiken, P. Miedema, L. Le Guyarder, L. Mercadier, G. Mercurio, J. Schlappa, M. Teichman, A. Yaroslavtsev, H. Sinn, and A. Scherz, The soft X-ray monochromator at the SASE3 beamline of the European XFEL: from design to operation, Journal of Synchrotron Radiation 29, 1299 (2022).
  * Grünert  _et al._ [2019] J. Grünert, M. P. Carbonell, F. Dietrich, T. Falk, W. Freund, A. Koch, N. Kujala, J. Laksman, J. Liu, T. Maltezopoulos, K. Tiedtke, U. F. Jastrow, A. Sorokin, E. Syresin, A. Grebentsov, and O. Brovko, X-ray photon diagnostics at the European XFEL, Journal of Synchrotron Radiation 26, 1422 (2019).
  * Maltezopoulos  _et al._ [2019] T. Maltezopoulos, F. Dietrich, W. Freund, U. F. Jastrow, A. Koch, J. Laksman, J. Liu, M. Planas, A. A. Sorokin, K. Tiedtke, and J. Grünert, Operation of X-ray gas monitors at the European XFEL, Journal of Synchrotron Radiation 26, 1045 (2019).
  * Engel  _et al._ [2021] R. Y. Engel, M. Ekimova, P. S. Miedema, C. Kleine, J. Ludwig, M. Ochmann, B. Grimm-Lebsanft, R. Ma, M. Teubner, S. Dziarzhytski, G. Brenner, M. K. Czwalinna, B. Rösner, T. K. Kim, C. David, S. Herres-Pawlis, M. Rübhausen, E. T. J. Nibbering, N. Huse, and M. Beye, Shot noise limited soft x-ray absorption spectroscopy in solution at a SASE-FEL using a transmission grating beam splitter, Structural Dynamics 8, 014303 (2021).
  * Schlotter  _et al._ [2020] W. F. Schlotter, M. Beye, S. Zohar, G. Coslovich, G. L. Dakovski, M. F. Lin, Y. Liu, A. Reid, S. Stubbs, P. Walter, K. Nakahara, P. Hart, P. S. Miedema, L. Le Guyader, K. Hofhuis, P. T. P. Le, J. E. T. Elshof, H. Hilgenkamp, G. Koster, X. H. Verbeek, S. Smit, M. S. Golden, H. A. Durr, and A. Sakdinawat, Balanced detection in femtosecond x-ray absorption spectroscopy to reach the ultimate sensitivity limit (2020), arXiv:2006.13968 .
  * Guyader  _et al._ [2022] L. L. Guyader, A. Eschenlohr, M. Beye, W. Schlotter, F. Döring, C. Carinan, D. Hickin, N. Agarwal, C. Boeglin, U. Bovensiepen, J. Buck, R. Carley, A. Castoldi, A. D’Elia, J.-T. Delitz, W. Ehsan, R. Engel, F. Erdinger, H. Fangohr, P. Fischer, C. Fiorini, A. Föhlisch, L. Gelisio, M. Gensch, N. Gerasimova, R. Gort, K. Hansen, S. Hauf, M. Izquierdo, E. Jal, E. Kamil, S. Karabekyan, T. Kluyver, T. Laarmann, T. Lojewski, D. Lomidze, S. Maffessanti, T. Mamyrbayev, A. Marcelli, L. Mercadier, G. Mercurio, P. S. Miedema, K. Ollefs, K. Rossnagel, B. Rösner, N. Rothenbach, A. Samartsev, J. Schlappa, K. Setoodehnia, G. S. Chiuzbaian, L. Spieker, C. Stamm, F. Stellato, S. Techert, M. Teichmann, M. Turcato, B. Van Kuiken, H. Wende, A. Yaroslavtsev, J. Zhu, S. Molodtsov, C. David, M. Porro, and A. Scherz, Photon shot-noise limited transient absorption soft x-ray spectroscopy at the european xfel (2022), arXiv:2211.0426 .
  * Denes  _et al._ [2009] P. Denes, D. Doering, H. A. Padmore, J.-P. Walder, and J. Weizeorick, A fast, direct x-ray detection charge-coupled device, Review of Scientific Instruments 80, 083302 (2009).
  * Januschek  _et al._ [2016] F. Januschek, I. Klačkova, N. Andresen, P. Denes, S. Hauf, J. Joseph, M. Kuster, and C. Tindall, Performance of the LBNL FastCCD for the European XFEL, in  _2016 IEEE Nuclear Science Symposium, Medical Imaging Conference and Room-Temperature Semiconductor Detector Workshop (NSS/MIC/RTSD)_ (2016) pp. 1–3.
  * Klačková  _et al._ [2019] I. Klačková, G. Blaj, P. Denes, A. Dragone, S. Göde, S. Hauf, F. Januschek, J. Joseph, and M. Kuster, Characterization of the ePix100a and the FastCCD semiconductor detectors for the european XFEL, Journal of Instrumentation 14 (01), C01008.
  * Saldin  _et al._ [2006] E. Saldin, E. Schneidmiller, and M. Yurkov, Statistical properties of the radiation from VUV FEL at DESY operating at 30 nm wavelength in the femtosecond regime, Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment 562, 472 (2006).
  * Hatada and Di Cicco [2014] K. Hatada and A. Di Cicco, Modeling saturable absorption for ultra short x-ray pulses, Journal of Electron Spectroscopy and Related Phenomena 196, 177 (2014).
  * Anisimov  _et al._ [1973] S. Anisimov, B. L. Kapeliovich, and T. L. Perel’man, Electron emission from metal surfaces exposed to ultrashort laser pulses, Sov. Phys. JETP 39, 375 (1973).
  * Chen  _et al._ [2006] J. Chen, D. Tzou, and J. Beraun, A semiclassical two-temperature model for ultrafast laser heating, International Journal of Heat and Mass Transfer 49, 307 (2006).
  * Hartley  _et al._ [2019] N. Hartley, J. Grenzer, W. Lu, L. Huang, Y. Inubushi, N. Kamimura, K. Katagiri, R. Kodama, A. Kon, V. Lipp, M. Makita, T. Matsuoka, N. Medvedev, S. Nakajima, N. Ozaki, T. Pikuz, A. Rode, K. Rohatsch, D. Sagae, A. Schuster, K. Tono, J. Vorberger, T. Yabuuchi, and D. Kraus, Ultrafast anisotropic disordering in graphite driven by intense hard X-ray pulses, High Energy Density Physics 32, 63 (2019).
  * Mueller and Rethfeld [2013] B. Y. Mueller and B. Rethfeld, Relaxation dynamics in laser-excited metals under nonequilibrium conditions, Phys. Rev. B 87, 035139 (2013).
  * Chang  _et al._ [2021] H.-T. Chang, A. Guggenmos, S. K. Cushing, Y. Cui, N. U. Din, S. R. Acharya, I. J. Porter, U. Kleineberg, V. Turkowski, T. S. Rahman, D. M. Neumark, and S. R. Leone, Electron thermalization and relaxation in laser-heated nickel by few-femtosecond core-level transient absorption spectroscopy, Phys. Rev. B 103, 064305 (2021).
  * Powell and Jablonski [2000] C. Powell and A. Jablonski, NIST electron inelastic-mean-free-path database 71, version 1.1 (2000).
  * Jain  _et al._ [2013] A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter, D. Skinner, G. Ceder, and K. a. Persson, The Materials Project: A materials genome approach to accelerating materials innovation, APL Materials 1, 011002 (2013).
  * Persson [2016] K. Persson, Materials data on Ni (sg:225) by materials project (2016).
  * Bévillon  _et al._ [2014] E. Bévillon, J. P. Colombier, V. Recoules, and R. Stoian, Free-electron properties of metals under ultrafast laser-induced electron-phonon nonequilibrium: A first-principles study, Phys. Rev. B 89, 115117 (2014).
  * Lin  _et al._ [2008] Z. Lin, L. V. Zhigilei, and V. Celli, Electron-phonon coupling and electron heat capacity of metals under conditions of strong electron-phonon nonequilibrium, Phys. Rev. B 77, 075133 (2008).
  * Lin and Zhigilei [2007] Z. Lin and L. V. Zhigilei, Temperature dependences of the electron–phonon coupling, electron heat capacity and thermal conductivity in Ni under femtosecond laser irradiation, Applied Surface Science 253, 6295 (2007).
  * Lojewski  _et al._ [2022] T. Lojewski, M. F. Elhanoty, L. L. Guyader, O. Grånäs, N. Agarwal, C. Boeglin, R. Carley, A. Castoldi, C. David, C. Deiter, F. Döring, R. Y. Engel, F. Erdinger, H. Fangohr, C. Fiorini, P. Fischer, N. Gerasimova, R. Gort, F. de Groot, K. Hansen, S. Hauf, D. Hickin, M. Izquierdo, B. E. Van Kuiken, Y. Kvashnin, C.-H. Lambert, D. Lomidze, S. Maffessanti, L. Mercadier, G. Mercurio, P. S. Miedema, K. Ollefs, M. Pace, M. Porro, J. Rezvani, B. Rösner, N. Rothenbach, A. Samartsev, A. Scherz, J. Schlappa, C. Stamm, M. Teichmann, P. Thunstrom, M. Turcato, A. Yaroslavtsev, J. Zhu, M. Beye, H. Wende, U. Bovensiepen, O. Eriksson, and A. Eschenlohr, The interplay of local electron correlations and ultrafast spin dynamics in fcc Ni (2022), arXiv:2210.13162 .
  * Stamm  _et al._ [2007] C. Stamm, T. Kachel, N. Pontius, R. Mitzner, T. Quast, K. Holldack, S. Khan, C. Lupulescu, E. F. Aziz, M. Wietstruk, H. A. Dürr, and W. Eberhardt, Femtosecond modification of electron localization and transfer of angular momentum in nickel, Nature materials 6, 740 (2007).
  * Dürr  _et al._ [2008] H. A. Dürr, C. Stamm, T. Kachel, N. Pontius, R. Mitzner, T. Quast, K. Holldack, S. Khan, C. Lupulescu, E. F. Aziz, M. Wietstruk, and W. Eberhardt, Ultrafast electron and spin dynamics in nickel probed with femtosecond x-ray pulses, IEEE Transactions on Magnetics 44, 1957 (2008).


11institutetext: Indian Institute of Technology Madras22institutetext:
Healthcare Technology Innovation Centre33institutetext: Sri Rama Chandra
Institute of Higher Education and Research

# MIST GAN: Modality Imputation using Style Transfer for MRI

Jaya Chandra Raju 1122 Kompella Subha Gayatri 1122 Keerthi Ram 22 Rajeswaran
Rangasami 33 Rajoo Ramachandran 33 Mohanasankar Sivaprakasam 1122

###### Abstract

MRI entails a great amount of cost, time and effort for generation of all the
modalities that are recommended for efficient diagnosis and treatment
planning. Recent advancements in deep learning research show that generative
models have achieved substantial improvement in the aspects of style transfer
and image synthesis. In this work, we formulate generating the missing MR
modality from existing MR modalities as an imputation problem using style
transfer. With a multiple-to-one mapping, we model a network that accommodates
domain specific styles in generating the target image. We analyse the style
diversity both within and across MR modalities. Our model is tested on the
BraTS’18 dataset and the results obtained are observed to be on par with the
state-of-the-art in terms of visual metrics, SSIM and PSNR. After being
evaluated by two expert radiologists, we show that our model is efficient,
extendable, and suitable for clinical applications.

###### Keywords:

MRI Image imputation Deep learning Image synthesis

##  1 Introduction

MRI is a versatile imaging technique that can image in multiple sequences each
highlighting specific tissue type based on the TR (Repetition time) and TE
(Time to Echo) parameters. The most commonly generated sequences are T1 (with
optional gadolinium enhancement), T2 and FLAIR (Fluid-Attenuated Inversion
Recovery), which are primarily used for diagnostic purposes in a clinical
setting. It is stated in[5] that in brain MRI, MR images with T1, T2, or FLAIR
contrast, are all required for accurate diagnosis and segmentation of cancer
margin. At the same time, acquiring all the sequences from a patient is also
an extremely tedious task, especially given the long scanning time and the
added cost. Even if all the sequences are acquired, there could be faulty
scans with distorted contrast which often are a result of motion artifacts or
noise. T1c requires injecting a dye which may induce side effects and is not
recommended for pregnant women or patients with kidney or heart issues.
Considering these clinical limitations and feasibility, it is therefore
necessary to explore the potential of cross modality synthesis for efficient
data infilling with respect to MRI.

Generating a new MRI modality from the existing modalities can be formulated
as an image-to-image translation problem, whose goal is to learn a mapping
between two different visual domains[10]. Here, domain implies a set of images
that can be grouped as a visually distinctive category and each image has a
unique appearance which is the style. For example, in MR images, each sequence
can be considered as a domain and different variants in each sequence can be
considered as different styles. In recent years, Generative Adversarial
Networks (GANs)[6] have been extensively studied for image-to-image
translation applications and showed promising outcomes[2][22][12]. Lately,
cross domain image translation using GANs have been considered for MRI. Due to
limited scalability, i.e. requiring N(N-1) generators for N-domain image
transfer, many networks have been restricted for one-to-one synthesis[18][4].
The next formulation of significance is multiple input-to-multiple target
synthesis of which models such as MM-GAN[19], AutoSyncoder[11], latent
representation[1],have performed efficiently, but these networks remain either
bulky or in need for a target based training. However, in medical image
synthesis, multiple-to-one mapping models are more relevant as learning
proprietary information of individual modalities can be more
synergistic[14][15]. They affirm that a more accurate target image is
generated when information from all the input modalities is used for training.

Though the aforementioned recent works successfully addressed the scalability
issue for multi-domain MRI synthesis/imputation, none of the models consider
the style diversity within the domain. Various MRI imaging centres follow
various imaging protocols giving multiple possible combinations of sequence
parameters each of which can be considered as a different style. Learning
these styles and being able to generate a target image of the required style
would greatly improve the comparability between two scans. For example, in
multiple sclerosis (MS) patients, treatment planning and decisions are majorly
based on longitudinal comparisons of MRI studies. Also, existing lesion
quantification tools require completely identical modalities scanned at
multiple time points[15]. It is therefore important to extract and reflect
this style information. For this, a very recent framework, StarGAN v2[3]
introduced the concept of style code to represent diverse styles of a specific
domain and it is learnt using two additional modules, style encoder and
mapping network. Another recent work ReMIC[20], used style based
representation disentanglement for image completion and segmentation in MRI.

Contribution: 1) We propose MIST GAN for MRI, which is a unified domain
scalable architecture that learns the multiple-to-one cross-modality mapping
with just a single generator and discriminator. 2) We establish that our model
is capable of learning and reflecting the diverse styles present within a
domain. We also prove that, contrary to the popular assumption, an MRI
modality is not just a single style, but a spectrum of styles.

Figure 1: MIST GAN overview. Left: Generates a target image from the given
three input images which is of the style specified by SE. Right: The cycle to
reconstruct the input images from the generated image with the help of Sep and
reusable CE, SE and Dec.

##  2 Proposed Framework

Let $D=\\{p,q,r,..\\}$ be the set of domains and $x_{d}$ be the image drawn
from domain $d\in D$. Given the target domain, $t$ that is randomly sampled
from $D$, the set of input domains will be $I=D-\\{t\\}$. When set of input
images, $X_{d}=\\{x_{d}|d\in I\\}$ and a target image $x_{t}$ is given, the
goal of the network is to learn the mappings from multiple input modalities to
the target modality while also learning varied styles within the domain. The
following modules are included in our network to realize the same:

Generator (G):

Style encoder (SE): extracts the fine style features present in a given
reference image. $SE$ takes the image, $x_{d}$ and the domain information,
$d$, as input to generate the domain specific style code $s_{d}=SE(x_{d},d)$.

Mapping Network (M): is a fully connected network that learns to map random
Gaussian noise ($z$), to latent style code distribution, $s_{d}=M(z,d)$. In
applications where the target image is not given for the style encoding, a
random target domain specific style code can be generated by using this
mapping network which is guided by the discriminator($Dsc$) during the
training.

Content Encoder (CE): is expected to take in an image, $X$, and generate its
content, $c=CE(x)$, in latent representation which is modality invariant.
Instance normalization (IN) is used for the down-sampling blocks.

Decoder (Dec): Given the content($c$) generated by $CE$ our decoder tries to
generate the target image, $\hat{x}_{t}=Dec(c,s_{t})$, reflecting the provided
target style code $s_{t}$. Here, we use Adaptive Instance
Normalization(AdaIN)[9] for the up-sampling blocks and add the skip
connections through a high-pass filter.

Combiner and Separator (Comb, Sep): The combiner generates a combined content,
$cc=Comb(concatenate(c_{p},c_{q},c_{r}))$ by integrating the content generated
by $CE$ for different domain images, $c_{p},c_{q},c_{r}$ where $p,q,r\in I$.
cc stores the useful key features extracted from all the input modalities. The
target image $\hat{x}_{t}$ is generated by passing combined content, $cc$,
through the $Dec$. Once we have generated the target image, $\hat{x}_{t}$, it
is encoded again using the $CE$ in order to get the new combined content,
$\hat{cc}$. Now, the separator takes in this content ($\hat{cc}$) and the
modality information ($d$) of each input and generates domain specific
content, $c_{d}=Sep(\hat{cc},d)$. This is given to the $Dec$ that generates
back the original input images (Fig. 1).

Discriminator (Dsc) used in this network is a multi-task discriminator with
multiple output branches where each branch learns a binary classification
determining whether an image is a real image, ($x_{d}$), or a decoder
outputted fake image, ($\hat{x}_{t}$). It takes in an image($x_{d}$) and its
domain information($d$), and predicts the
probability($p_{r/f}$)$=Dsc(x_{d}/\hat{x}_{t};d/t)$ of how real the given
image is.

###  2.0.1 Training Objectives:

#### Cyclic loss:

To ensure that the network properly preserves the domain invariant
characteristics of the given inputs, we enforce the Dec to reconstruct the
input images from the target image by incorporating a cyclic loss
($CL=CSL+CCL$) which consists of cyclic style loss($CSL$) and cyclic content
loss($CCL$). $CSL$ is the loss defined between the style information encoded
from the generated fake image and the style information encoded from the
target image sampled. To synthesize the input images from the imputed target
image, $\hat{x}_{t}$, $CE$, $Sep$ are used to get the corresponding modality
specific content and $SE$, $Dec$ are reused to generate the respective images,
$\hat{x}_{p}$, $\hat{x}_{q}$ and $\hat{x}_{r}$. Now, $CCL$ is defined as the
summation of domain-wise $L1$ loss between translated images and source
images.

$CSL=||SE(x_{t})-SE(\hat{x}_{t})||_{1}$ (1)
$CCL=\sum_{d=p,q,r}||\hat{x}_{d}-x_{d}||_{1}$ (2)

#### Adversarial loss:

The adversarial loss is defined to impose the network to produce images that
are indistinguishable from the true distribution. The adversarial loss
components are different for the generator part ($G_{adv}$) and the $Dsc$
($Dsc_{adv}$) as shown below:

$G_{adv}=1-Dsc(\hat{x}_{t},t)$ (3)
$Dsc_{adv}=(1-Dsc(x_{t},t))+(0-Dsc(\hat{x}_{t},t))$ (4)

#### Style Diversification Loss (SDL):

To enable the network to produce diverse styled images, we regularize it with
the diversity sensitive loss[3]. Two reference images are randomly sampled
from the target domain ($x_{t1};x_{t2}$) to get $s_{t1};s_{t2}$ estimated by
$SE$. We generate images using these two styles and maximize the difference
between them which forces the network to learn meaningful style features. The
total objective can be summarized for generator and discriminator as follows:

$\boldsymbol{\min_{G}}(CL+G_{adv}-SDL)\quad\textrm{and}\quad\boldsymbol{\min_{Dsc}}(Dsc_{adv})$
(5)

Figure 2: Qualitative results achieved by the proposed model on the HGG
BraTS’18 dataset. A sample of the domain-wise generated images along with
their respective ground truths are shown. The blue squares represent the
missing/target modality. Table 1: Quantitative performance of the model on
BraTS’18 HGG and LGG cohort.

| Modality | Metrics | Stargan-v2 | MM-GAN | Proposed
---|---|---|---|---|---
LGG | T1 | SSIM | 0.90143 $\pm$ 0.01685 | 0.9288 $\pm$ 0.0137 | 0.90193 $\pm$ 0.02157
PSNR | 22.96876 $\pm$ 3.28355 | 25.9841 $\pm$ 2.1926 | 24.68469 $\pm$ 3.87952
T1c | SSIM | 0.89411 $\pm$ 0.01774 | 0.9086 $\pm$ 0.0253 | 0.89597 $\pm$ 0.02294
PSNR | 24.61375 $\pm$ 2.66390 | 22.4980 $\pm$ 1.1684 | 25.07036 $\pm$ 3.43035
T2 | SSIM | 0.88846 $\pm$ 0.02120 | 0.9030 $\pm$ 0.0303 | 0.88411 $\pm$ 0.02094
PSNR | 24.60764 $\pm$ 2.88763 | 25.6005 $\pm$ 2.7909 | 24.761391 $\pm$ 2.57905
F | SSIM | 0.86251 $\pm$ 0.02914 | 0.8692 $\pm$ 0.0224 | 0.87498 $\pm$ 0.02930
PSNR | 22.67723 $\pm$ 2.65003 | 23.0852 $\pm$ 1.5142 | 23.72480 $\pm$ 2.86680
HGG | T1 | SSIM | 0.88832 $\pm$ 0.01787 | 0.9228 $\pm$ 0.0190 | 0.88226 $\pm$ 0.02094
PSNR | 23.99222 $\pm$ 2.58524 | 24.173 $\pm$ 3.2754 | 25.62511 $\pm$ 1.94889
T1c | SSIM | 0.87734 $\pm$ 0.02066 | 0.9239 $\pm$ 0.0375 | 0.87303 $\pm$ 0.02411
PSNR | 24.09959 $\pm$ 3.23823 | 24.372 $\pm$ 2.2792 | 26.42274 $\pm$ 1.80974
T2 | SSIM | 0.86777 $\pm$ 0.01812 | 0.9349 $\pm$ 0.0262 | 0.87160 $\pm$ 0.02162
PSNR | 23.45378 $\pm$ 2.26071 | 28.678 $\pm$ 2.3290 | 24.88888 $\pm$ 1.33794
F | SSIM | 0.84534 $\pm$ 0.02543 | 0.9150 $\pm$ 0.0275 | 0.85406 $\pm$ 0.02861
PSNR | 21.95010 $\pm$ 2.74631 | 26.397 $\pm$ 1.9733 | 24.79925 $\pm$ 2.10912

Figure 3: (a):Validation scores assigned by expert radiologists for all the
modalities when blinded to the ground truth label of the image. (b):t-SNE plot
showing disengagement of style codes across four domains and a spectrum of
styles within each domain. Figure 4: (a) Representation of style forms: the
second row shows the Flair generated images when its corresponding first row
image is given as a reference image to the style encoder. (b) Style
interpolation: T1 to F transition, achieved by linearly interpolating the
styles between mean style of T1 and mean style of Flair in steps of 0.1.

##  3 Results and Discussion

Dataset and Implementation: The performance of the proposed architecture is
evaluated on BraTS’18 dataset[16][17] that consists of 75 LGG and 210 HGG
patient records of T1, T1c, T2 and Flair modalities all of which are axially
acquired, co-registered and skull stripped. The dataset is split for training,
validation and testing in the ratio 3:1:1. All the slices are padded to a
uniform size of 256 x 256 and the intensity values are linearly normalized
between (0,1). For the training, the batch size is set to 2 and the model is
trained for 200k iterations. The training duration is approximately two days
on a single NVIDIA GEFORCE GTX 1080 Ti GPU, occupying about 10GB. Adam[13]
optimizer is adopted for training and the learning rate is set to 1e-4 for all
modules except mapping network for which it is set as 1e-6. All weights are
initialized using [7].

Qualitative and quantitative analysis: We generate a target modality that not
only learns relevant information from multiple input modalities, but also
reflects the user specified style. Any 3 of 4 MRI modalities, irrespective of
their ordering are taken as inputs and the missing modality is imputed using
style transfer. We ensure domain scalability and limit the network complexity
by reusing the core blocks ($CE$, $SE$, $Dec$). The model is studied for both
HGG and LGG datasets of BraTS’18 and quantitatively evaluated using PSNR (Peak
Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) metrics. The
results for the same are shown in Table 1. Based on these metrics it is
observed that our synthesized T1c is closer to the real T1c for both HGG and
LGG, thereby opening the possibility of generating a contrast enhanced image
without actually injecting the contrast agent, an application which is of high
clinical relevance.

We compare the achieved results with the results of a baseline model, MM-GAN,
which incorporates a target based training with supervised loss. Despite being
blind to the target image and solely guided by the discriminator, our model
performed on par, better for few modalities (LGG - T1c, F; HGG - T1, T1c) in
terms of PSNR (Table 1). We have also implemented StarGAN v2, a style based
model for MRI and recorded the metrics for comparison. We trained the model as
is, but while testing, instead of just giving one image, we reused the encoder
of the generator for multiple inputs and averaged the content in the latent
space and passed it to the decoder to get the generated image. It can be seen
from Table 1 that the proposed model clearly outperformed StarGAN v2. A sample
of our modality-wise generated output is shown in Fig. 2 and it can be
observed that, generated T1 and T1c are enhanced and more sharp than their
respective ground truths, a proper T2 is generated even though one of its
source images is faulty and generated Flair is quite close to its original
image.

Style Analysis: In order to study the efficiency of our model in learning,
extracting and reflecting varied styles, we attempted generating different
style forms of a single image in a given domain. We fed the network with any
arbitrary style image and made it generate the same target image in the
specified random style. Fig. 4(a) shows the style forms for Flair domain: top
row shows the given reference style images and the bottom row shows the
corresponding generated target images. Clear variations can be noticed in the
generated images indicating that any modality is not just a single style but a
spectrum of styles and our model succeeded in capturing them. The same is
evident from the t-SNE(t-distributed stochastic neighbor embedding) plot Fig.
3(b) of the latent style codes, in which it is also seen that style
information is properly disentangled between the four domains. With the
acquired domain specific mean style value, we observed the transition of a
generated image between two domains, while gradually changing the style
(simple weighted average/linear interpolation of latent style), thereby
enabling operator, the flexibility to choose and change between the styles.
Fig. 4(b) shows one such smooth transition achieved between T1 and Flair.
Indeed an unseen intermediate modality which is neither completely T1 or Flair
has surfaced in this interpolation whose clinical utility could be a
prospective research. However, some droplet artifacts are noticed when
interpolated with intermediate styles which is probably due to a harsh style
diversification loss. For quantitative analysis, mean FID (Frechét inception
distance)[8] and mean LPIPS (Learned perceptual image patch similarity)[21]
are extracted for all modalities. The mean FID (between the generated images
and the true images) and LPIPS (across the generated images) scores for T1,
T1c, T2 and Flair are [49.37, 70.52, 49.66, 74.25] and [0.0159, 0.0261,
0.0092, 0.0277] respectively when assessed for BraTS’18 dataset.

Visual Evaluations by Radiologists: For qualitative assessment of the achi-
-eved results, we consulted two expert radiologists with 10+ years of
professional experience. They each evaluated the output image on a scale of 5
(1, being very poor and 5, being excellent). In each trial, they were provided
with four images, the left three were the source images and on the right, a
corresponding target image was shown, which is either a real image or a
generated one. Radiologists are blinded to the nature (real or fake) of the
image so as to avoid any bias and to collect a more objective feedback. Each
of them are presented with 400 such random trials for all the four modalities.
The rating was given based on the image quality, capturing of small sub-
structures and efficiency in mapping the source images to the target image.
The ratings of the two radiologists are averaged and the boxplot of the
results are shown in Fig. 3(a). From the plot it is noticed that for T1, T1c
and T2, the mean rating of synthesized images lags the mean rating of real
images by about 0.5. But for the Flair modality, the generated images were
given a better rating when compared to the real images. Both the experts
acknowledged that the synthetic Flair images are as good as the original
flair. The radiologists’ opinion was also taken on the performed style
analysis of the output images and they both affirmed that the model captured
multiple styles within each domain and that such work would help in
longitudinal MR comparability studies of patients with progressive
neurological conditions.

##  4 Conclusion

Motivated by the need to create an efficient and novel method for MRI modality
synthesis, we have derived a style transfer formulation for imputation and
realized an efficient GAN based network that generates an enhanced version of
the missing MR modality from given three input modalities. We show that our
network is not only scalable to any number of input modalities, but also
capable of picking up style variations within each modality. Evaluation on the
BraTS’18 multimodal brain MRI dataset suggests that the method is promising
and opens new avenues for further research.

## References

  * [1] Chartsias, A., Joyce, T., Giuffrida, M.V., Tsaftaris, S.A.: Multimodal MR Synthesis via Modality-Invariant Latent Representation. IEEE Transactions on Medical Imaging 37(3), 803–814 (Mar 2018)
  * [2] Choi, Y., Choi, M., Kim, M., Ha, J.W., Kim, S., Choo, J.: StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation. In: CVPR 2018. pp. 8789–8797 (2018)
  * [3] Choi, Y., Uh, Y., Yoo, J., Ha, J.W.: Stargan v2: Diverse image synthesis for multiple domains. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2020)
  * [4] Dar, S.U., e.a.: Image synthesis in multi-contrast mri with conditional generative adversarial networks. In: IEEE transactions on medical imaging (2019)
  * [5] Drevelegas, A., Papanikolaou, N.: Imaging modalities in brain tumors. In Imaging of Brain Tumors with Histological Correlations. Springer (2011)
  * [6] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks (2014)
  * [7] He, K., Zhang, X., Ren, S., Sun., J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: ICCV (2015)
  * [8] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium (2018)
  * [9] Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization (2017)
  * [10] Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks (2018)
  * [11] K., R.J.M.B.R., M., S.: Autosyncoder: An adversarial autoencoder framework for multimodal mri synthesis. In: F., I.D., P., J., T., W., (eds), Y.J. (eds.) Lecture Notes in Computer Science, vol 12450. Springer, Cham. (2020), https://doi.org/10.1007/978-3-030-61598-7_10
  * [12] Kim, T., Cha, M., Kim, H., Lee, J.K., Kim, J.: Learning to discover cross-domain relations with generative adversarial networks (2017)
  * [13] Kingma, D.P., Ba., J.: Adam: A method for stochastic optimization. In: ICLR (2015)
  * [14] Lee, D., Moon, W.J., Ye, J.C.: Assessing the importance of magnetic resonance contrasts using collaborative generative adversarial networks. Nature Machine Intelligence 2(1), 34–42 (Jan 2020)
  * [15] Li, H., Paetzold, J.C., Sekuboyina, A., Kofler, F., Zhang, J., Kirschke, J.S., Wiestler, B., Menze, B.: DiamondGAN: Unified Multi-modal Generative Adversarial Networks for MRI Sequences Synthesis. In: MICCAI 2019 (2019)
  * [16] Menze, Bjoern, et al.: The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS). IEEE Transactions on Medical Imaging p. 33 (2014)
  * [17] MenzeBjoernHetal: The multimodal brain tumor image segmentation benchmark (brats). ”IEEE transactions on medical imaging vol. 34,10(2015): 1993-2024 (2015). https://doi.org/10.1109/TMI.2014.2377694
  * [18] Q., Y., Li, N.Z., et al., Z.: Mri cross-modality image-to-image translation. In: Sci Rep 10, 3753 (2020), https://doi.org/10.1038/s41598-020-60520-6
  * [19] Sharma, A., Hamarneh, G.: Missing MRI Pulse Sequence Synthesis Using Multi-Modal Generative Adversarial Network. IEEE Transactions on Medical Imaging 39(4), 1170–1183 (Apr 2020)
  * [20] Shen, L., Zhu, W., Wang, X., Xing, L., Pauly, J.M., Turkbey, B., Harmon, S.A., Sanford, T.H., Mehralivand, S., Choyke, P.L., Wood, B.J., Xu, D.: Multi-domain image completion for random missing input data. IEEE Transactions on Medical Imaging 40(4), 1113–1122 (2021). https://doi.org/10.1109/TMI.2020.3046444
  * [21] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR (2018)
  * [22] Zhu, J.Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks (2020)