[1,3]Ronald B. Liu

[2]Pierre O. Bagnaninchi

[3]Xavier Casadevall i Solvas

[1]Yunjie Yang

1]SMART Group, Institute for Imaging, Data and Communications, School of
Engineering, University of Edinburgh, UK

2]Institute Regeneration and Repair, College of Medicine and Veterinary
Medicine, University of Edinburgh, UK

3]Biomimetics Group, Department of Biosystems, Catholic University of Leuven,
Belgium

# Generative deep learning-enabled ultra-large field-of-view lens-free imaging

<EMAIL_ADDRESS>Zhe Liu<EMAIL_ADDRESS>Max G.A. Wolf
<EMAIL_ADDRESS>Krishna P. Purohit<EMAIL_ADDRESS>Gregor Fritz
<EMAIL_ADDRESS>Yi Feng<EMAIL_ADDRESS>Carsten G. Hansen
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>[  [  [

###### Abstract

Advancements in high-throughput biomedical applications necessitate real-time,
large field-of-view (FOV) imaging capabilities. Conventional lens-free imaging
(LFI) systems, while addressing the limitations of physical lenses, have been
constrained by dynamic, hard-to-model optical fields, resulting in a limited
one-shot FOV of approximately $20\text{\,}\mathrm{m}\mathrm{m}^{2}$. This
restriction has been a major bottleneck in applications like live-cell imaging
and automation of microfluidic systems for biomedical research. Here, we
present a deep-learning (DL)–based imaging framework - GenLFI - leveraging
generative artificial intelligence (AI) for holographic image reconstruction.
We demonstrate that GenLFI can achieve a real-time FOV over
$550\text{\,}\mathrm{m}\mathrm{m}^{2}$, surpassing the current LFI system by
more than 20-fold, and even larger than the world’s largest confocal
microscope by 1.76 times. The resolution is at the sub-pixel level of
$5.52\text{\,}\mathrm{\SIUnitSymbolMicro m}$, without the need for a shifting
light source. The unsupervised learning-based reconstruction does not require
optical field modeling, making imaging dynamic 3D samples (e.g., droplet-based
microfluidics and 3D cell models) in complex optical fields possible. This
GenLFI framework unlocks the potential of LFI systems, offering a robust tool
to tackle new frontiers in high-throughput biomedical applications such as
drug discovery.

## Main

The field-of-view (FOV) of an optical system denotes its maximum observable
area, which, together with resolution, is crucial in live-cell imaging for
monitoring cell and tissue dynamics for biomedical applications such as drug
screening and disease research. Most of the current biomedicine imaging
modalities rely on light microscopy, wherein light is either transmitted or
reflected by the sample through lenses. However, due to the lens’
magnification by the numerical aperture (NA) and the sensor size, FOV in lens-
based optical systems is inherently limited [1]. Efforts to expand the FOV
have historically focused on advancements in lens design and reconstruction
algorithms [2, 3, 4, 5]. For lens-based microscopes, despite significant
enhancements in contrast [6], specificity [7, 8], optical sectioning
capabilities [9, 10], and advances in super-resolution algorithms [11, 12, 13,
14, 15], FOV remains constrained by magnification and lens aberrations, with
commercial systems offering a maximum FOV of $312.5\,mm^{2}$ at minimal
resolution[16]. Consequently, existing biomedical imaging platforms, like
microscopes and plate readers, require scanning for a larger FOV by regularly
adjusting lens positions. This intrinsic limitation impedes real-time imaging
on large tissue sections (i.e. metastatic cancer processes across
microvessels), whole model organisms (e.g., zebrafish), and advanced cell
models (e.g., spheroids/organoids and organs on chip).

To overcome these limitations, lens-free imaging (LFI) strategies have been
developed. LFI, a form of computational imaging (CI), captures diffraction
patterns (i.e., holograms) directly with the image sensor and reconstructs
focused images using computational algorithms, bypassing the need for
conventional lenses. Like other CI systems, current LFI reconstruction
algorithms depend on a physical forward model for a resolvable optical field.
In 2D LFI systems [17, 18, 19, 20, 21, 22, 23], the distance between the
sample and the light source plays a vital role in inverse computation for the
diffraction field. This parameter limits the imaging of the sample to a thin
focal plane. In parallel, 3D tomographic LFI systems [24, 25, 26], which rely
on multi-position measurements, offer a degree of depth of field in the range
of micrometers. Nevertheless, these algorithms inherently demand a high-
quality and resolvable hologram, which, in real-world scenarios, can only come
from a clean, non-overlapping, and stable diffraction optical field modeling.
This necessitates precise control over the samples’ size and density to align
with precise propagation parameters, such as depth of field and illumination
angle [27]. Although deep-learning (DL) has improved holographic
reconstruction (re-focusing) algorithms enabling their generalization to
diverse sample types [17, 28, 29], such idealized measurement conditions limit
the applicability of current LFI systems to imaging of static, specific and
well-controlled samples. Moreover, the sensitivity of their optical
instruments to environmental factors such as vibration, temperature, and
contamination further limits their practical use in dynamic, real-world
scenarios [24, 20, 30]. Finally, to achieve a FOV and resolution similar to
confocal microscopes, current LFI systems [27, 30] usually require a shifting
light source to scan a large region-of-interest (ROI). Because of all these
limitations, state-of-the-art LFI systems currently offer little
characterization advantages over traditional optical imaging techniques.

Existing biomedical imaging systems all encounter a significant constraint:
limited throughput through one-shot FOV. However, a modest FDA-approved drug
library can contain around 2000 samples and 2 million or more pairwise
combinations [31], where the time delay to image different samples heavily
decreases the ability to interrogate dynamic events. Notably, the repetitive
nature of these samples’ high-content data presents an opportunity. In the
context of high-throughput imaging for drug screening, previous studies [32,
33, 34, 35, 36] have demonstrated the reliability of deep neural networks,
such as generative adversarial networks (GAN). By leveraging these recurrent
patterns as robust data guidance, it is possible to enhance the reconstruction
of sample morphology more effectively, thus mitigating FOV limitations and
enhancing the efficiency of high-throughput imaging.

Here, we propose a generative deep learning-enabled LFI framework, namely
GenLFI, employing DL-based computer vision techniques to address the
limitations of state-of-the-art LFI systems. GenLFI employs a universal
optical setup and significantly simplifies the imaging process by eliminating
the need for data pre-processing, annotation, and optical field modeling that
traditional LFI systems depend on. Our framework also incorporates an
unsupervised learning model, LensGAN, capable of autonomously learning image
features from unpaired and unaligned holograms and bright-field light
microscopy to generate high-resolution image reconstructions. This is achieved
through a comprehensive analysis of diffraction and projection principles,
establishing physical correlations between holograms and reconstructed bright-
field images.

Through a series of imaging experiments, we demonstrate how the GenLFI
framework advances LFI systems in three key aspects: (1) Our LensGAN’s robust
learning capability enables the reconstruction of high-quality images from
low-resolution holograms, achieving an ultra-large FOV in a single shot. This
FOV is solely constrained by the sensor size, with our validation on a full-
frame CMOS sensor demonstrating a FOV exceeding $550\,mm^{2}$, while still
preserving sub-sensor-pixel resolution. (2) Unlike state-of-the-art LFI
methods, GenLFI requires no optical field control or modeling, allowing it to
image complex 3D samples (i.e., microfluidics, 3D cell models). (3) LensGAN
boasts a faster inference speed than current hologram reconstruction
algorithms [17, 37, 38]. These advancements collectively extend the boundaries
of LFI technology in high-throughput biomedical imaging tasks. The expanded,
streamlined, and enhanced observational capacities of GenLFI can create new
possibilities for broad dynamic imaging applications in complex cell-based
microfluidic systems, phenotypic screening, and AI-enabled digital pathology.

## Results

### GenLFI framework

Figure 1: Overview of GenLFI framework. a, the left illustrates our ultra-
large FOV LFI setup, and the right showcases examples of 3D samples. The
middle depicts the complex optical field: as a coherent light source
illuminates the sample, scattered waves interfere with the undistributed
incident wave, forming a complex hologram above the CMOS sensor through the
interference pattern and shaded projection. b, the workflow of GenLFI.

Incorporating physics-based priors is essential in LFI. Previous LFI
reconstruction approaches rely on a forward imaging model, denoted by $H$,
that relates the measured holograms $i$ to the reconstruction $\hat{o}$, by
incorporating the physical principles of the LFI system [39]. The LFI hologram
reconstruction task can thus be formulated as [17]:

$\hat{o}=arg\min_{o}L(H(o),i)+R(o),$ (1)

where $i\in\mathbb{H}^{IK^{2}}$ denotes the measured holograms after
vectorization; $I$ represents the number of measured raw holograms, each with
a dimension of $K\times K$; $o$ and $\hat{o}\in\mathbb{R}^{K^{2}}$ are the
real-world ground truth and predicted target image with a dimension of
$K\times K$ after vectorization, respectively, which in this study are bright-
field images; $L(\cdot)$ and $R(\cdot)$ are the loss function and
regularization term, respectively.

However, crafting an accurate physical forward model $H$ within a complex
optical field, shown in Fig.1a, presents significant challenges due to several
factors. The 3D sample, with vertical dimensions measured in micrometers ($\mu
m$), can produce overlapped scattered waves from various axial distances and
angles, particularly when interacting with mediums of different refractive
indexes. In such scenarios, the optical field generates a complex hologram at
the sensor, incorporating diffraction waves from different height planes and
shadow projections. Furthermore, the optical field is subject to alterations
caused by the movement of dynamic objects, which introduces additional
distortions. While certain methods can enhance the robustness of light field
modeling to perturbations in complex optical fields [36, 21], these factors
collectively contribute to challenges in accurately describing, modeling, and
reconstructing the optical field using current computational algorithms [22,
29, 18, 17, 19, 28, 24, 40] (further discussion see Supplementary Information
2).

To address these challenges, we introduce GenLFI, a generative deep learning-
enabled LFI framework. It comprises a universal optical setup (see Methods),
real-world data guidance, and a hologram reconstruction algorithm, i.e.,
LensGAN. As shown in Fig.1b, The workflow of GenLFI is as follows: (1) The
unpaired dataset consists of the data acquired from our LFI setup (i.e.,
measured holograms) and the data from a bright-field microscope (i.e., real-
world bright-field images), which are rescaled to serve as the input for
LensGAN. (2) During training, LensGAN generalizes both the measured and
reconstructed holograms ($i$ and $\hat{i}$) and the bright-field images ($o$
and $\hat{o}$) into two domains, $X$ and $Y$, respectively. (3) The trained
LensGAN model is then deployed to rapidly reconstruct hologram patches, which
are finally synthesized into original-sized holograms.

LensGAN adopts an unsupervised learning approach, bypassing the reliance on
the forward model $H$ to decode complex optical fields. Instead, it leverages
the power of generative models for robust image transformations within complex
light fields. Simultaneously, LensGAN captures the inherent physical
correlations between holograms and bright-field images. Our approach
introduces two key enhancements to establish these physical prior correlations
and optimize the model. First, we designed a discriminator network ($D$) that
integrates one-layer attention between the downsampling convolutional layers
to improve the processing of fine details in amplitude (e.g., illumination
details). Second, we devise and incorporate a phase-contrast module to mimic
the mechanism of the phase-contrast microscope (PCM) and dark-field microscope
(DFM) due to the similar transparent nature of the objects (details in
Methods). This module enforces physical consistency between the phase details
(e.g., structures) of the hologram and the bright-field image. These
improvements eliminate the need for optical field modeling required by
existing LFI methods and manual annotation in typical supervised learning
frameworks.

This mechanism liberates the optical setup from the constraints of optical
field modeling, offering flexibility in setup design and robustness against
environmental disturbance. We constructed a universal LFI setup adhering to
the unit-magnification principle [41] (see Fig.1a). To capture an ultra-large
FOV hologram in a single shot, we considerably extended the distance beyond
that of conventional setups (details in meth-optset). We leverage this setup
to demonstrate the flexibility, accuracy, efficiency, and robustness of the
LensGAN algorithm in the following sections.

### FOV, Resolution and Multi-focus imaging

FOV: In LFI, the achievable FOV captured in a single shot (one-shot FOV) is
limited by the numerical aperture (NA) of the light source and the distance
$f$ between the light source and the detector, e.g., the CMOS sensor in
Fig.1a. While previous research[21, 28, 36] has demonstrated some flexibility
in adjusting this distance within a millimeter ($mm$) range, achieving the
best lateral resolution requires a specific optimized distance. This optimized
distance [42] can be calculated as:

$f_{opt}=\frac{1}{\frac{1}{d}-\frac{\lambda}{Np^{2}}}$ (2)

where $\lambda$ is the parameter wavelength; $N$ is the number of pixels of
the image sensor; $p$ denotes the pixel size, and $d$ is the distance between
the object and sensor. The optimized distance is essential for generating
high-resolution holograms, which is the cornerstone of existing LFI
holographic reconstruction algorithms. Therefore, with a given NA, the FOV of
existing LFI systems is constrained by the distance $f\approx f_{opt}$.
LensGAN overcomes this limitation by reconstructing images directly from low-
resolution holograms, eliminating the need for precise diffraction pattern
decoding. This enables the GenLFI framework to surpass the $f_{opt}$
constraint, offering flexibility on the distance $f$. The FOV limitation is
then primarily governed by the sensor size. In our demonstration, the GenLFI
framework achieves a FOV of $553\,mm^{2}$ (see Fig.2a with the setup in
Methods). Further improvements in FOV are possible through an upgraded optical
configuration, such as using a higher NA light source, and a larger and
higher-resolution image sensor.

Figure 2: FOV and resolution characterization. a shows different imaging
systems’ FOV on our microfluidic chip design. 4x: 4x objective of bright-field
microscope (details see Optical setup); state-of-the-art LFI [43]: the
existing LFI system’s one-shot FOV based on holographic in-line
reconstruction. AX 1x: Nikon AX [16], the world’s largest commercial confocal
microscope. The scanning 1x objective can cross a $25\,mm$ diagonal FOV in
multiple shots. b, Imaging of the USAF 1951 resolution test chart. The input
hologram was paired with the ROI of the smallest region of the test chart in
the bright field. The green box indicates the reconstructed resolvable region;
the curves of the cross-sectional profile at the bottom row demonstrate the
distinguishable line pairs.

Resolution: The image resolution attainable by GenLFI is affected by two
physical parameters. First, the Nyquist-Shannon sampling theorem dictates that
the lateral resolution of holograms is inherently constrained by the pixel
size of the CMOS sensor, which is $5.94\,\mu m$ in our setup. This limitation
suggests that details finer than the sensor’s pixel pitch cannot be adequately
captured in the hologram [30]. Second, the reconstructed image quality is
significantly influenced by the dimension of the input image of the model (see
Methods). In particular, LensGAN’s training is memory-intensive on GPU due to
the need to load six networks (see Methods), limiting the input image
dimensions due to hardware constraints. We imaged a 1951 USAF resolution test
chart and conducted a cross-sectional profile analysis following the standard
routine [44, 45, 46]. We selected an area within the measured hologram where
the smallest line bar is not resolvable (see Fig.2b). After reconstruction, we
found that the smallest resolvable line pair was element 4 of group 6
(Fig.2b), corresponding to the best observing lateral resolution of $5.52\,\mu
m$, which falls below the sensor’s pixel size of $5.94\,\mu m$. Furthermore,
the evaluation metrics, i.e., the Fréchet inception distance (FID) and
structural similarity index measure (SSIM), are 22.6855 and 0.8203,
respectively, when comparing the reconstruction with the ground truth obtained
by a bright-field microscope. Notably, using a higher-resolution sensor is a
straightforward approach to improve lateral resolution.

We hypothesize that the diversity and quantity of samples in the training
dataset also influence the final resolvable resolution. In the 1951 USAF
resolution test, the observed object is only one test chart (see Methods),
which will likely cause the over-fitting problem (see the failure case of
singular sample in Fig.S7 in Supplementary Information). To address this, we
conducted another experiment with $10\,\mu m$ diameter beads, increasing data
diversity and quantity. The reconstructions clearly recover and resolve the
bead shapes from different regions (see Fig.3a). Unlike traditional LFI
confined to a thin focal plane, our results demonstrate a higher depth of
field. This allows us to distinguish objects at various depths within the
water flow (see Methods). We visualize the cross-sectional profiles of three
beads at different depths in Fig.3a, along with their depth information
encoded by pixel intensity. This capability for depth recovery paves the way
for potential 3D high-throughput imaging using GenLFI.

Figure 3: Depth of field tests. a, $10\,\mu m$ diameter beads imaging. The
beads, which were black and opaque, were reconstructed by LensGAN using their
features from small dirt and transparent droplets. We select 3 beads at
different depths and visualize them by cross-sectional profiles. In this
particular case, the 3D depth map can be visualized based on pixel intensities
from the reconstruction. b, Results of handwritten digits slides stack. Left
shows the 4x objective focuses on “1” and is gradually out of focus from “2”
to “4”. Right is the reconstructed result; all numbers are focused.

Multi-focus: Unlike conventional light microscopes requiring manual focus
adjustments, GenLFI excels at reconstructing in-focus images from holograms
containing objects at different depths. This unique capability is attributed
to its virtual depth of field, offering significant flexibility for multi-
focal imaging. To showcase this advantage, we conducted experiments with two
stacks of 4 glass slides ($1\,mm$ thick), each marked with a written digit
(font-weight $0.3\,mm$). In traditional bright-field setups, focusing is
limited to a single depth plane, resulting in only one clear digit at a time
(see Fig.3b). With proper data guidance (see Methods), LensGAN can refocus and
resolve all digits across different slides in a single reconstruction,
achieving a superior depth of focus of 3 mm (Fig.3b). This indicates our
system’s versatility and underscores its potential in scenarios where
simultaneous multi-depth clarity is crucial.

Inference time: Once deployed, the trained model relies only on a single
network (i.e., the generator $G$ built by a 9-layer ResNet [47]) for
reconstruction, making it the fastest among existing hologram reconstruction
algorithms. Previous studies [17, 30] have suggested that DL can expedite the
reconstruction process through an end-to-end approach. Unlike many DL-based
reconstruction algorithms that rely on the multi-height phase recovery (MHPR)
approach, LensGAN distinguishes itself by eliminating the necessity for
multiple ($I$ number) hologram inputs for reconstruction. Notably, efficient
model compression techniques for GANs [48] could further reduce the
computational cost by $46\times$, but we maintain the original model size to
avoid resolution degradation. We evaluated LensGAN’s performance on our edge
device (see MethodsMethods) using hologram images with $512\times 512$ pixels
($px$). LensGAN was compared against the self-supervised model GedankenNet
[17], the supervised model U-Net [37], and the iterative algorithm MHPR [38],
as outlined in Table 1. LensGAN achieved an equivalent inference time of
$0.0031\,{s\cdot mm}^{-2}$ for a $1\,mm^{2}$ area of our microfluidic chip,
showing its exceptional speed and efficiency in hologram reconstruction.

Table 1: Holographic image inference time (for $I$ input hologram(s),
$512\times 512\,px$)

Method |  LensGAN ($I$=1) |  GedankenNet ($I$=2) |  U-Net ($I$=2) |  MHPR ($I$=2)
---|---|---|---|---
Inference time | 0.0015$s$ | 0.05$s$ | 0.02$s$ | 0.98$s$
#Parameters | 33.8M | 39.4M | 31.0M | -

00footnotetext: M: million. Evaluations were performed on a single PC with an
i9-13900K CPU and an RTX 3090 GPU.

### Large FOV imaging of microfluidics

Figure 4: Reconstruction of a microfluidic chip ($30\times 5\,mm$). a, ROI in
the hologram, i.e., the functional region containing dynamic droplets. b,
Synthesized reconstruction result of the ROI (also see Supplementary video 1).
We also analyzed the cross-sectional profiles at the center of 3 patches,
labeled $i$, $o$, and $\hat{o}$, as denoted by the white dash line at $i$.

Microfluidic platforms have transformed biomedical research by enabling high-
throughput experiments with biological samples down to the scale of single
cells or molecules [49]. Despite their increasing adoption, a critical
challenge remains. Existing high-resolution imaging techniques cannot
simultaneously monitor multiple ROIs on a single chip. For instance, imaging
the formation of droplets in droplet-based microfluidics often impedes
simultaneous observation of downstream operations, such as droplet sorting or
merging, occurring in a different ROI [49].

Our GenLFI approach overcomes this limitation. We demonstrate its capabilities
in a microfluidic experiment, generating and dynamically imaging droplets
across various chip sections concurrently. Data collection for the training
dataset of this microfluidic experiment (see Methods) took approximately 20
minutes. Unlike most LFI systems, this procedure is automated, requiring no
manual annotation. We acquired two-domain training data using our LFI system
and a conventional bright-field microscope, labeled as $i$ and $o$,
respectively (see Fig.1b). These two-domain images, i.e., $i$ and $o$,
constituted the ‘GenLFI-Microfluidics’ dataset for LensGAN training and
testing. Reconstruction results (see Fig.4) demonstrate that the trained
LensGAN can generate high-resolution ($1792\times 768\,px$) bright-field
images ($\hat{o}$) across the entire ROI (i.e., spanning a full microfluidic
chip).

The results show that our approach not only preserves the global structure
from the original hologram to the generated image but also ensures consistent
transverse x-coordinate positioning of features, as evident in the re-focusing
process shown in Fig.4b. Furthermore, the synthesized large FOV image (Fig.4b)
reveals that although the trained LensGAN model infers each patch
independently (see Methods), the microfluidic channels are aligned
consistently across patches. This suggests the robustness of our approach in
handling large-scale reconstruction tasks. This capability can facilitate
monitoring and visual-based control for intricate microfluidic systems.

### Large FOV imaging of spheroids

Figure 5: Reconstruction of cell spheroids. a, MeT-5A NF2KO spheroid images
acquired using different imaging systems: bright-field microscopy with a 10x
objective lens (left), digital holographic microscopy (DHM) with a 10x
objective lens (center), and our GenLFI platform (right). b, Full-FOV
reconstruction by LensGAN on an unseen well, post-processed by global panorama
stitching algorithm (see Supplementary Information 6) and local image-
stitching [50]. The structures are readily resolvable in the penetrated light
parts at the edges of spheroids.

Cell spheroids, as straightforward yet extensively utilized multi-cellular 3D
models, emerge from the natural inclination of most adherent cells to cluster
together [51]. The imaging of spheroids poses significant challenges due to
their considerable thickness, which can reach up to hundreds of microns, and
their high density. As noted previously, conventional holographic imaging
systems, including LFI systems, involve complex optical fields, resulting in
contaminated holograms with overlapping scattered patterns and shadows (see
Fig.1a). When utilizing digital holographic microscopy (DHM) with a 10x
objective lens and a $635\,nm$ wavelength laser light source, only limited
structural and intensity information can be retrieved (see Fig.5a).

With the addition of a light contrast-enhancing polarization filter, a bright-
field microscope equipped with a 10x objective lens can discern a partial
sectional structure at the periphery by adjusting the focus of the objective
lens (Fig.5a). A comparable light penetration capability can also be achieved
using our LFI platform by employing a red laser light source with a wavelength
of $635\,nm$. Therefore, we integrated inverted bright-field microscopy to
acquire data guidance images $o$, facilitating holographic reconstruction of
similar quality but across an ultra-large FOV.

Fig.5b shows that our approach can reconstruct the spheroid edge structure and
shapes from the measured holograms in a large FOV of an engineered cancer
model. Notably, spheroids exhibit lateral structures that might replicate some
of the more complex cellular environments, thereby mimicking more relevant
scenarios while using high throughput platforms. Leveraging the advantages of
the large FOV, our approach proves effective for monitoring the
multiparametric spheroid growth patterns, which will likely allow for insights
into biomechanical features, cell-cell interactions, and readily quantifiable
multiparametric features of spheroid outgrowth assays in a scalable and cost-
effective manner.

## Discussion

GenLFI, a generative deep learning framework for LFI, eliminates the need for
traditional forward imaging models by introducing data-driven guidance through
unsupervised learning. This framework enables imaging in dynamic, noisy
optical fields, previously not achievable with LFI. GenLFI achieves an over 20
times larger FOV than current LFI systems, enabling real-time monitoring for
in vivo (e.g., zebrafish, see Fig.S1 in Supplementary Information) or in vitro
(e.g., cell-in-droplets, 3D cell models) studies. Images are reconstructed
using a DL network, significantly faster than traditional iterative techniques
[22, 23, 52, 53, 38], while achieving sub-pixel resolution without requiring
adjustments to the light source. These advancements demonstrate the potential
of leveraging generative AI to overcome optical limitations.

The dynamic nature of our imaging scenarios makes it impractical to obtain
paired ground truth and raw holograms for training. Therefore, GenLFI
leverages unsupervised learning algorithms, inspired by those designed for
image-to-image (I2I) transformation tasks in computer vision. This approach
prioritizes hardware-agnostic and reproducible workflows with simple setups.
While the current implementation utilizes a telescope cold camera with a
resolution of $5.9\,\mu m/px$, GenFLI is compatible with higher-resolution
sensors and multi-angle illumination for potential 3D LFI capabilities.

Our investigation into the physical correlation between holograms and bright-
field images achieved leading results based on FID and SSIM metrics. However,
challenges remain with such I2I-based approaches, primarily concerning data
quality, quantity, and diversity. These factors directly impact reconstruction
results, as demonstrated by artifacts observed in reconstructions from limited
datasets (see the results from single-sample, paired datasets Fig.2b and
Fig.S7 in Supplementary Information). Additionally, training GANs can be
challenging due to potential issues like model collapse and non-convergence.
While exploring diffusion models [54, 55, 56, 57] as an alternative shows
promise, their limitations in handling unpaired and unaligned image
translation tasks require further investigation. Furthermore, existing
evaluation metrics like FID and KID primarily assess image-level distributions
and lack the ability to evaluate the alignments between objects across
different image patches. Developing new metrics accurately reflecting
alignment between different images captured from the same objects is crucial
for future advancements.

The combination of GenLFI and large vision models (LVM) [58, 59] holds promise
for achieving a more generalized reconstruction capability for various samples
and light field conditions. In addition, leveraging processors with advanced
and specified architectures [60, 61, 62] could enable real-time image
reconstruction with these large models.

Despite limitations due to optical conditions, GenLFI’s high-throughput
capability opens doors for various biomedical applications. GenLFI offers a
large FOV, enabling rapid analysis of large biological samples with minimal
equipment. This capability is particularly valuable in resource-limited
settings lacking traditional histopathology services. By incorporating digital
staining through generative deep learning, GenLFI can bring advanced imaging
techniques to such settings. Furthermore, GenLFI’s cost-effective and compact
setup facilitates routine monitoring of cell culture and spheroid growth
directly within incubators, reducing phototoxicity while enabling continuous
monitoring of dynamic cellular processes across a large FOV. The testing of
drug libraries based on dynamic processes, utilizing single-cell/cell models
or droplet-based microfluidic systems, can now be obtained simultaneously from
microplate-like platforms, paving the way for developing next-generation,
autonomous, high-throughput drug screening systems.

## Methods

### Optical setup

GenLFI utilizes a versatile optical setup for capturing holograms. The optical
setup comprises a light source, a CMOS sensor, and a test bench to stabilize
their positions. We employed two types of light sources, including a blue LED
light source for enhanced resolution (used for microfluidics, test chart,
beads) and a red laser light source for deeper penetration depth (used for
spheroids). For LED illumination, the light source is Thorlabs M420L3
(wavelength $420\,nm$) blue LED, with power supply (15V 1A) and T-Cube LED
Driver (Thorlabs LEDD1B). For laser illumination, we used an optical fibre-
coupled red laser source (Thorlabs S1FC635, wavelength $635\,nm$). The
mechanical setup consists of a cage plate (Thorlabs CP33/M), a mounting
bracket (ThorLabs 973/579-7227), a $25\,mm$ linear stage (Thorlabs XR25C) with
an end-mounted micrometer, a post holder (Thorlabs LPH200), a post (Thorlabs
P350/M), and an adjustable platform (Thorlabs C1519). The telescope camera
(ZWO ASI 2400MC Pro) features a full-frame Sony IMX410 CMOS sensor, $36\times
24\,mm$ in size, with a $5.94\,\mu m/px$ resolution. Mounted on an
experimental bracket (FAMA MV test stand), the camera aligns with the light
source and the imaging object along a single axis. The distance from the light
source to the object’s bottom plane is $430\,mm$, and from the object’s bottom
plane to the CMOS sensor is $12.5\,mm$.

### LensGAN architecture

LensGAN, a core component of GenLFI, reconstructs high-resolution images from
captured holograms. LensGAN generally considers the measured and reconstructed
holograms ($i$ and $\hat{i}$) and the measured and generated bright-field
images ($o$ and $\hat{o}$) into two domains, $X$ and $Y$, respectively. It
leverages a GAN framework with a cycle-consistent training strategy to learn
the complex mapping between holograms (domain $X$) and corresponding bright-
field images (domain $Y$). The architecture of LensGAN comprises 6 networks:
generators of reconstruction and reverse mappings $G:X\rightarrow Y$ and
$F:Y\rightarrow X$, selectors $\beta_{X}$ and $\beta_{Y}$, and discriminators
$D_{X}$ and $D_{Y}$. During training, all 6 networks function following the
training loop as shown in Fig.1a. In the testing phase, only the trained
generator $G$ is used for reconstruction. The outputs of LensGAN are
reconstructed bright-field images $\hat{o}=G(i)$ and generated holograms
$\hat{i}=F(o)$.

Generators $G$ and $F$ handle the fundamental image transformation task, while
the remaining 4 networks contribute to the training process. $G$ and $F$, each
composed of 9 residual blocks (ResNet [47]), feature an image encoder for
extracting image characteristics and a decoder for generating images. The
selector network $\beta$ [63] is responsible for pairing images based on
feature alignments and is constructed with an adaptive average pooling layer,
4 convolutional networks, and one fully connected network (FCN) with a Softmax
layer. The discriminator $D$ is designed to differentiate between images,
emphasizing structural information and intricate details within the complex
amplitude. Recent advancements in Transformers [64] have demonstrated the
equivalence of multi-layer and one-layer attention mechanisms [65]. By
leveraging this finding, we designed $D$ with a Markovian architecture,
merging a one-layer attention module with 5 downsampling convolutional layers
to augment amplitude details while maintaining minimal computational overhead
(see Supplementary Information 4, 5).

Combining these networks, the training loop adheres to the cycle structure
[66], guided by several loss functions, depicted in Fig.1b and Fig.S2 in
Supplementary Information. To augment phase structural feature information
during the training, we devised a phase contrast module positioned to
constrain the physical correlation between $X$ and $Y$ domains. The observed
samples in LFI are generally light-permeable, resulting in duotone holograms
captured under a single-wavelength light source. In such scenarios, phase
contrast [6] and dark-field microscopes are commonly employed, where a phase
ring and other lenses are combined to create a phase contrast. Inspired by
this technique, our approach is to develop a module that learns the phases of
the two domains contrastively, thereby mimicking a similar mechanism and
enhancing the physical correlations between the object’s hologram and bright-
field images. Technically, we create the phase-contrast loss function
$\mathcal{L}_{\mathrm{PC}}$ by a multi-scale structural similarity index
measure (MS-SSIM) [67] for the amplitude spectra:

$\min_{G}\mathcal{L}_{\mathrm{PC}}(X,Y)=\mathbb{E}_{i\sim
P_{X}}\left[1-\operatorname{MS-
SSIM}(|\mathcal{FT}(i)|,|\mathcal{FT}(\hat{o})|)\right],$ (3)

where $|\mathcal{FT}(\cdot)|$ denotes the magnitudes of the individual
frequency components obtained from the Fourier transform ($\mathcal{FT}$) of
an image, and $P_{X}$ is the probability distribution for $i$.
$\operatorname{MS-SSIM}$ is defined as:

$\operatorname{MS-
SSIM}(x,y)=\left[l_{M}(x,y)\right]^{\gamma_{M}}\cdot\prod_{j=1}^{M}\left[c_{j}(x,y)\right]^{\delta_{j}}\left[s_{j}(x,y)\right]^{\epsilon_{j}},$
(4)

with the terms

$\displaystyle
l(x,y)=\frac{2\mu_{x}\mu_{y}+C_{1}}{\mu_{x}^{2}+\mu_{y}^{2}+C_{1}},$ (5)
$\displaystyle
c(x,y)=\frac{2\sigma_{x}\sigma_{y}+C_{2}}{\sigma_{x}^{2}+\sigma_{y}^{2}+C_{2}},$
$\displaystyle s(x,y)=\frac{\sigma_{xy}+C_{3}}{\sigma_{x}\sigma_{y}+C_{3}},$

where $x$ and $y$ and two input image patches, $M=5$ denotes the highest scale
number set for the system to iteratively apply a low-pass filter between $x$
and $y$. $\mu$ and $\sigma^{2}$ stand for mean value and variance,
respectively, which are implemented by 2D convolutions. $l(x,y)$, $c(x,y)$,
and $s(x,y)$ represent luminance, contrast, and structure comparison measures
between $x$ and $y$, with control weights $\delta_{1}=\epsilon_{1}=0.0448$,
$\delta_{2}=\epsilon_{2}=0.2856$, $\delta_{3}=\epsilon_{3}=0.3001$,
$\delta_{4}=\epsilon_{4}=0.2363$, and
$\gamma_{5}=\delta_{5}=\epsilon_{5}=0.1333$. LensGAN enforces all inputs as
8-bit images with a pixel value range of 0 to 255; the constants are set as
$C_{1}=(0.01\times 255)^{2}$, $C_{2}=(0.03\times 255)^{2}$, and
$C_{3}=C_{2}/2$.

Note that in (3), we added $\mathcal{L}_{\mathrm{PC}}$ only for the mapping
$G:X\rightarrow Y$, the transformation process of holographic reconstruction,
with a control weight $\tau_{\mathrm{PC}}$. Similarly, only for $G$, selector
$\beta_{X}$ is regularized by the effective sample size loss [63] for
$\beta_{X}$:

$\min_{\beta_{X}}\mathcal{L}_{\mathrm{ESS}}(X)=\|\beta_{X}\|_{2},$ (6)

where $\|\cdot\|_{2}$ denotes the L2 norm.

For the whole cycle (reconstruction mapping $G:X\rightarrow Y$ and reverse
mapping $F:Y\rightarrow X$), there is a GAN loss function
$\mathcal{L}_{\mathrm{GAN}}$ to constrain $G$, $F$ and $D$, with two
consistency loss functions $\mathcal{L}_{\mathrm{cyc}}$ and
$\mathcal{L}_{\mathrm{idt}}$ to further regularize $G$ and $F$ [68, 66, 63].
They are reweighted least squares adversarial loss

$\displaystyle\min_{G,\beta_{X}}\max_{D_{Y}}$
$\displaystyle\mathcal{L}_{\mathrm{GAN}}(X,Y)$ (7) $\displaystyle=$
$\displaystyle\mathbb{E}_{i\sim
P_{X}}\beta_{X}(i)\left[\left(1-D_{Y}(\hat{o}\right)^{2}\right]+$
$\displaystyle\mathbb{E}_{o\sim P_{Y}}\beta_{Y}(o)\left[D_{Y}(o)^{2}\right],$

reweighted cycle consistency loss

$\displaystyle\min_{G,F}\mathcal{L}_{\mathrm{cyc}}(X,Y)=\mathbb{E}_{i\sim
P_{X}}\beta_{X}(i)\|i-F(\hat{o})\|_{1}$ (8) $\displaystyle+\mathbb{E}_{o\sim
P_{Y}}\beta_{Y}(o)\|o-G(\hat{i})\|_{1},$

and reweighted identity loss

$\displaystyle\min_{G,F}\mathcal{L}_{\mathrm{idt}}(X,Y)=\mathbb{E}_{i\sim
P_{X}}\beta_{X}(i)\|i-F(i)\|_{1}$ (9) $\displaystyle+\mathbb{E}_{o\sim
P_{Y}}\beta_{Y}(o)\|o-G(o)\|_{1},$

where $P_{X}$ and $P_{Y}$ are the probability distributions for $i$ and $o$,
respectively, and $\|\cdot\|_{1}$ denotes the L1 norm.

The LensGAN’s full objective of loss functions is combined with:

$\begin{gathered}\mathcal{L}_{main}=\min_{G,F}\max_{D_{X},D_{Y}}\mathcal{L}_{\mathrm{GAN}}(X,Y)+\mathcal{L}_{\mathrm{GAN}}(Y,X)\\\
\quad+\tau_{\mathrm{cyc}}\mathcal{L}_{\mathrm{cyc}}(X,Y)+\tau_{\mathrm{PC}}\mathcal{L}_{\mathrm{PC}}(X,Y)+\tau_{\mathrm{idt}}\mathcal{L}_{\mathrm{idt}}(X,Y),\\\
\mathcal{L}_{\beta_{X}}=\min_{\beta_{X}}\mathcal{L}_{\mathrm{GAN}}(X,Y)+\tau_{\mathrm{ESS}}\mathcal{L}_{\mathrm{ESS}}(Y),\\\
\mathcal{L}_{\beta_{Y}}=\min_{\beta_{Y}}\mathcal{L}_{\mathrm{GAN}}(Y,X)+\tau_{\mathrm{ESS}}\mathcal{L}_{\mathrm{ESS}}(Y),\end{gathered}$
(10)

where $\mathcal{L}_{main}$ is used to train networks $G$, $F$, and $D$,
$\mathcal{L}_{\beta_{X}}$ and $\mathcal{L}_{\beta_{Y}}$ are used to train
$\beta_{X}$ and $\beta_{Y}$, respectively; the weights $\tau_{\mathrm{cyc}}$,
$\tau_{\mathrm{PC}}$, $\tau_{\mathrm{idt}}$, and $\tau_{\mathrm{ESS}}$ control
the relative importance of different losses and can be adjusted as
hyperparameters.

### Training, testing and post-processing

For all experiments, we set hyperparameters $\tau_{\mathrm{cyc}}=10$,
$\tau_{\mathrm{PC}}=8$, and $\tau_{\mathrm{ESS}}=\tau_{\mathrm{idt}}=1$. We
use Adam solver [69] to initiate training of the networks from scratch,
maintaining a consistent learning rate of 0.0001 for the initial 50 epochs.
Subsequently, we linearly decrease the learning rate to zero over the
following 50 epochs. Each epoch processed 10,000 images (we set the input
dimension as $256\times 256\,px$) with a batch size 20. The training was on a
single Nvidia RTX A100 GPU; for dataset ‘GenLFI-Microfluidics’, it took 29
hours. All tests were conducted on a workstation (CPU Intel i9-13900K, GPU
Nvidia RTX3090). We utilized FID for quantification and comparative
evaluation. Following fast reconstruction, we performed post-processing for
large sample results using image stitching algorithms. As illustrated in
Fig.5b, we employed local object stitching [50] and global panorama stitching
(see Supplementary Information 6) separately.

### Microfluidics imaging

We fabricated a microfluidic chip with a flow-focusing geometry and conducted
water-in-oil droplet generation experiments to observe these dynamic samples.
The microfluidic channel width and height are $200\,\mu m$ and $35\,\mu m$,
respectively. We used piezoelectric disc pumps (TTP Ventus HP series) to
control the microfluidic flows and droplet production rates (ranging from 5 to
15 $Hz$). We conducted the imaging experiments under a bright-field microscope
(Cerna modular microscopy with Thorlabs CS135MU camera) and our LFI platform,
respectively, and acquired image data $i$ and $o$ for network training.

Using the GenLFI setup, we recorded 500 hologram frames ($i$) in total (.fit
format, resolution $6072\times 4042\,px$, acquired via the ASI Studio
software) focusing on the ROI of the microfluidic chip with dynamic droplets
(see Fig.4). Then, under the same experimental conditions, we repeated the
experiment to acquire real-world bright field images ($o$) as the data
guidance using a bright-field microscope. Due to the small FOV of the
microscope, we scanned the same ROI and recorded a 5-minute video containing
7000 frames of bright-field images.

We cropped the hologram images to patches with a size of $166\times 147\,px$.
Similarly, bright-field images were cropped to patches with $1024\times
1080\,px$. The final dataset ‘GenLFI-Microfluidics’ for this task comprises
121,886 training and 100 test cropped images from LFI hologram ROIs, and
57,193 training and 100 test cropped images from bright-field images.

### Cell spheroids imaging

Human pleural mesothelial cells MeT-5A NF2KO were cultured as 3D spheroids by
seeding approximately 30,000 cells into each well of a 6-well ultra-low
attachment culture plate (#3471; Corning, Thermo Scientific); details see
[70]. The dataset ‘GenLFI-Spheroids’ construction and network training for
this task follows a similar routine of Microfluidics imaging. We used a red
laser (see Optical setup) as the light source. The bright-field microscope for
recording is Leica DMIRE2 with a 10x objective lens, coupling with a
polarization filter to enhance the light contrast for spheroid structure
observation. The acquired raw data contains 132 frames of holograms and 26,581
frames of bright-field microscopic images. We constructed a dataset comprising
17,418 training and 100 test images (captured from an unseen well) for LFI
holograms (after Bicubic interpolation super-resolution, $808\times 808\,px$),
and 23,738 training and 100 test images for bright-field images (original
size, $1280\times 1024\,px$).

### USAF 1951 test chart imaging

We employed a USAF 1951 test chart (Thorlabs R3L3S1P) to evaluate the
resolution limits of our LFI system. Given that the thickness of the test
bench’s glass slide, i.e., $1.5\,mm$, affects the light diffraction, we placed
the printed side close to the microscope’s lens to minimize this effect when
capturing images through standard bright-field microscopy. Then, we placed the
sample in the same orientation to acquire holograms with GenLFI. For
microscopy images, we selected a square ROI with the smallest pair of lines.
For holograms, we cropped out the same ROI to a $220\times 220\,px$ image (see
Fig.2b). We carefully aligned these two domain images on the same scale. To
augment our dataset for this task, we introduced Gaussian noise to generate
10,000 variant images for each domain, of which 99 images with 1 original
noise-free image as test images, and the other images as training images.

### Beads imaging

We used $10\,\mu m$ diameter beads solution (Invitrogen CML latex, $4\%$ w/v,
H2392891) diluted with water on a microscope slide for imaging. The water flow
on the slide is approximately $0.1\,mm$ thick; therefore, the $10\,\mu m$
beads float or sink across the flow. We recorded 23 frames of hologram images
($6072\times 4042\,px$) under the LFI platform, and 1430 frames ($4096\times
2160\,px$) images under the bright-field microscope with a 4x objective. For
both recorded holograms and bright-field images, we applied Gaussian filtering
to remove shadows, then cropped and applied data augmentation with vertical
and horizontal flips to scale up the data size. The dataset, ‘GenLFI-Beads’,
consists of 96 test and 33,120 training images of hologram ($506\times
505\,px$); 96 test and 32,576 training images of bright-field images
($512\times 540\,px$).

### Multi-focus imaging

In evaluating multi-focus imaging capabilities, we stacked four glass slides,
each $1\,mm$ thick, upon which handwritten numbers (1,2,3,4) had been
inscribed. Another slack is marked with 5,6,7,8 at different heights as the
unseen object for the test. These numbers serve as markers at different
distances from the CMOS sensor’s imaging plane. When examined under bright-
field microscopy, focusing on any one number caused the others to appear
progressively more out of focus. We captured focused images of each number
separately to establish clear ground truth data for training the LensGAN. The
last dataset, ‘GenLFI-Multifocus’, consists of 43 test and 100 training images
of hologram ($1072\times(669\text{ or }1338)\,px$); and 27 test and 395
training images of bright-field images ($4096\times 2160\,px$). During
training, hologram images were guided solely by these focused images.

## References

  *   * [1] Lin, P.D.: Geometrical optical model. In: Lin, P.D. (ed.) Advanced Geometrical Optics. Progress in Optical Science and Photonics, pp. 71–114. Springer. https://doi.org/10.1007/978-981-10-2299-9_3 . https://doi.org/10.1007/978-981-10-2299-9_3 Accessed 2023-12-19
  * [2] Coskun, A.F., Su, T.-W., Ozcan, A.: Wide field-of-view lens-free fluorescent imaging on a chip 10(7), 824 https://doi.org/10.1039/b926561a . Publisher: NIH Public Access. Accessed 2023-10-31
  * [3] Peng, Y., Sun, Q., Dun, X., Wetzstein, G., Heidrich, W., Heide, F.: Learned large field-of-view imaging with thin-plate optics 38(6), 1–14 https://doi.org/10.1145/3355089.3356526 . Accessed 2023-10-02
  * [4] Scherrer, J.R., Lynch, G.F., Zhang, J.J., Fee, M.S.: An optical design enabling lightweight and large field-of-view head-mounted microscopes 20(4), 546–549 https://doi.org/10.1038/s41592-023-01806-1 . Number: 4 Publisher: Nature Publishing Group. Accessed 2023-10-06
  * [5] Ji, J., Xie, H., Yang, L.: Learned large field-of-view imager with a simple spherical optical module, 128918 https://doi.org/10.1016/j.optcom.2022.128918 . Accessed 2022-09-03
  * [6] Zernike, F.: Phase contrast, a new method for the microscopic observation of transparent objects 9(7), 686–698 https://doi.org/10.1016/S0031-8914(42)80035-X . Accessed 2023-12-19
  * [7] Menzies, S.W., Ingvar, C., McCarthy, W.H.: A sensitivity and specificity analysis of the surface microscopy features of invasive melanoma 6(1), 55 https://doi.org/10.1097/00008390-199602000-00008 . Accessed 2023-12-19
  * [8] Sensitivity and Specificity of Epiluminescence Microscopy: Evaluation on a Sample of 2731 Excised Cutaneous Pigmented Lesions | British Journal of Dermatology | Oxford Academic. https://academic.oup.com/bjd/article/142/5/893/6689918?login=true Accessed 2023-12-19
  * [9] Agard, D.A.: Optical sectioning microscopy: Cellular architecture in three dimensions 13(1), 191–219 https://doi.org/10.1146/annurev.bb.13.060184.001203 . _eprint: https://doi.org/10.1146/annurev.bb.13.060184.001203. Accessed 2023-12-19
  * [10] Conchello, J.-A., Lichtman, J.W.: Optical sectioning microscopy 2(12), 920–931 https://doi.org/10.1038/nmeth815 . Accessed 2023-12-19
  * [11] Nehme, E., Weiss, L.E., Michaeli, T., Shechtman, Y.: Deep-STORM: super-resolution single-molecule microscopy by deep learning 5(4), 458 https://doi.org/10.1364/OPTICA.5.000458 1801.09631 [physics]. Accessed 2023-12-22
  * [12] Choong, H.Y., Kumar, S., Van Gool, L.: Quantum Annealing for Single Image Super-Resolution. arXiv. http://arxiv.org/abs/2304.08924 Accessed 2023-04-24
  * [13] Wang, H., Zhang, Y., Qin, C., Gool, L.V., Fu, Y.: Global aligned structured sparsity learning for efficient image super-resolution, 1–16 https://doi.org/10.1109/TPAMI.2023.3268675 . Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence
  * [14] Ouyang, W., Aristov, A., Lelek, M., Hao, X., Zimmer, C.: Deep learning massively accelerates super-resolution localization microscopy 36(5), 460–468 https://doi.org/10.1038/nbt.4106 . Number: 5 Publisher: Nature Publishing Group. Accessed 2023-12-22
  * [15] Qiao, C., Li, D., Guo, Y., Liu, C., Jiang, T., Dai, Q., Li, D.: Evaluation and development of deep neural networks for image super-resolution in optical microscopy 18(2), 194–202 https://doi.org/10.1038/s41592-020-01048-5
  * [16] AX / AX R With NSPARC. Nikon Instruments Inc. https://www.microscope.healthcare.nikon.com/products/confocal-microscopes/ax Accessed 2024-01-23
  * [17] Huang, L., Chen, H., Liu, T., Ozcan, A.: Self-supervised learning of hologram reconstruction using physics consistency 5(8), 895–907 https://doi.org/10.1038/s42256-023-00704-7 . Number: 8 Publisher: Nature Publishing Group. Accessed 2023-12-19
  * [18] Wang, H., Lyu, M., Situ, G.: eHoloNet: a learning-based end-to-end approach for in-line digital holographic reconstruction 26(18), 22603–22614 https://doi.org/10.1364/OE.26.022603 . Publisher: Optica Publishing Group. Accessed 2022-08-14
  * [19] Jayapala, M., Stahl, R., Vanmeerbeeck, G., Lambrechts, A.: Lens-free holographic imaging of microscopic objects using photonic structures and CMOS imagers, 4
  * [20] Zhang, J., Sun, J., Chen, Q., Zuo, C.: Resolution analysis in a lens-free on-chip digital holographic microscope 6, 697–710 https://doi.org/%****␣main.bbl␣Line␣325␣****10.1109/TCI.2020.2964247 . Accessed 2023-12-19
  * [21] Ozcan, A., Rivenson, Y., Luo, Y., Huang, L.: Single-Shot Autofocusing of Microscopy Images Using Deep Learning. https://www.freepatentsonline.com/20230085827.html Accessed 2023-04-13
  * [22] Barkley, S., Dimiduk, T.G., Fung, J., Kaz, D.M., Manoharan, V.N., McGorty, R., Perry, R.W., Wang, A.: Holographic Microscopy with Python and HoloPy. arXiv. https://doi.org/10.48550/arXiv.1806.00058 . http://arxiv.org/abs/1806.00058 Accessed 2023-12-19
  * [23] Castañeda, R., Trujillo, C., Doblas, A.: pyDHM: A python library for applications in digital holographic microscopy 17(10), 0275818 https://doi.org/10.1371/journal.pone.0275818 . Publisher: Public Library of Science. Accessed 2022-11-29
  * [24] Luo, Z., Yurt, A., Stahl, R., Carlon, M.S., Ramalho, A.S., Vermeulen, F., Lambrechts, A., Braeken, D., Lagae, L.: Fast compressive lens-free tomography for 3d biological cell culture imaging 28(18), 26935 https://doi.org/10.1364/OE.393492 . Accessed 2022-06-22
  * [25] Chang, C., Dai, B., Zhu, D., Li, J., Xia, J., Zhang, D., Hou, L., Zhuang, S.: From picture to 3d holography: End-to-end learning of real-time 3d photorealistic hologram generation from 2d image input https://doi.org/10.1364/OL.478976 . Accessed 2023-01-10
  * [26] Isikman, S.O., Bishara, W., Mudanyali, O., Sencan, I., Su, T.-W., Tseng, D., Yaglidere, O., Sikora, U., Ozcan, A.: Lensfree on-chip microscopy and tomography for bio-medical applications 18(3), 1059–1072 https://doi.org/10.1109/JSTQE.2011.2161460 . Accessed 2023-12-19
  * [27] Greenbaum, A., Luo, W., Su, T.-W., Göröcs, Z., Xue, L., Isikman, S.O., Coskun, A.F., Mudanyali, O., Ozcan, A.: Imaging without lenses: Achievements and remaining challenges of wide-field on-chip microscopy 9(9), 889–895 https://doi.org/10.1038/nmeth.2114 . Accessed 2024-01-19
  * [28] Chen, H., Huang, L., Liu, T., Ozcan, A.: eFIN: Enhanced Fourier Imager Network for generalizable autofocusing and pixel super-resolution in holographic imaging. arXiv. http://arxiv.org/abs/2301.03162 Accessed 2023-01-12
  * [29] Chen, H., Huang, L., Liu, T., Ozcan, A.: Deep learning-based hologram reconstruction with superior external generalization (conference presentation). In: Quantitative Phase Imaging IX, vol. PC12389, p. 123890. SPIE. https://doi.org/10.1117/12.2648180
  * Luo [2021] Luo, Z.: Advanced lens-free imaging techniques for complex cell structures and organoids. Phd thesis, KU Leuven (January 2021). LIRIAS3344173, Available at https://kuleuven.limo.libis.be/
  * [31] Kulesa, A., Kehe, J., Hurtado, J.E., Tawde, P., Blainey, P.C.: Combinatorial drug discovery in nanoliter droplets 115(26), 6685–6690 https://doi.org/10.1073/pnas.1802233115 . Accessed 2022-09-21
  * [32] Zhang, H., Fang, C., Xie, X., Yang, Y., Mei, W., Jin, D., Fei, P.: High-throughput, high-resolution deep learning microscopy based on registration-free generative adversarial network 10(3), 1044–1063 https://doi.org/%****␣main.bbl␣Line␣525␣****10.1364/BOE.10.001044 30891329. Accessed 2024-02-05
  * [33] Hussain, S., Anees, A., Das, A., Nguyen, B.P., Marzuki, M., Lin, S., Wright, G., Singhal, A.: High-content image generation for drug discovery using generative adversarial networks 132, 353–363 https://doi.org/10.1016/j.neunet.2020.09.007 . Accessed 2024-02-05
  * [34] Huang, K., Matsumura, H., Zhao, Y., Herbig, M., Yuan, D., Mineharu, Y., Harmon, J., Findinier, J., Yamagishi, M., Ohnuki, S., Nitta, N., Grossman, A.R., Ohya, Y., Mikami, H., Isozaki, A., Goda, K.: Deep imaging flow cytometry 22(5), 876–889 https://doi.org/10.1039/D1LC01043C . Accessed 2022-06-15
  * [35] Bouchard, C., Wiesner, T., Deschênes, A., Bilodeau, A., Turcotte, B., Gagné, C., Lavoie-Cardinal, F.: Resolution enhancement with a task-assisted GAN to guide optical nanoscopy image analysis and acquisition 5(8), 830–844 https://doi.org/10.1038/s42256-023-00689-3 . Accessed 2024-02-05
  * [36] Lee, C., Song, G., Kim, H., Ye, J.C., Jang, M.: Deep learning based on parameterized physical forward model for adaptive holographic imaging with unpaired data 5(1), 35–45 https://doi.org/10.1038/s42256-022-00584-3 . Accessed 2024-02-12
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Navab, N., Hornegger, J., Wells, W.M., Frangi, A.F. (eds.) Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pp. 234–241. Springer, Cham (2015)
  * [38] Rivenson, Y., Zhang, Y., Günaydın, H., Teng, D., Ozcan, A.: Phase recovery and holographic image reconstruction using deep learning in neural networks 7(2), 17141–17141 https://doi.org/10.1038/lsa.2017.141 . Accessed 2024-01-22
  * Barbastathis et al. [2019] Barbastathis, G., Ozcan, A., Situ, G.: On the use of deep learning for computational imaging. Optica 6(8), 921–943 (2019) https://doi.org/10.1364/OPTICA.6.000921
  * [40] Zhang, X., Sheng, Z., Shen, H.-L.: FocusNet: Classifying better by focusing on confusing classes 129, 108709 https://doi.org/10.1016/j.patcog.2022.108709 . Accessed 2023-02-01
  * [41] Ozcan, A., Demirci, U.: Ultra wide-field lens-free monitoring of cells on-chip 8(1), 98–106 https://doi.org/10.1039/b713695a
  * [42] Serabyn, E., Liewer, K., Wallace, J.K.: Resolution optimization of an off-axis lensless digital holographic microscope 57(1), 172–180 https://doi.org/10.1364/AO.57.00A172 . Accessed 2024-02-05
  * [43] Lens-Free Imaging: Compact, Scalable, High-Res | Imec. https://www.imec-int.com/en/expertise/health-technologies/lens-free-imaging Accessed 2024-01-30
  * [44] Luo, W., Zhang, Y., Feizi, A., Göröcs, Z., Ozcan, A.: Pixel super-resolution using wavelength scanning 5(4), 16060–16060 https://doi.org/10.1038/lsa.2016.60 . Accessed 2024-02-02
  * [45] Luo, Z., Yurt, A., Stahl, R., Lambrechts, A., Reumers, V., Braeken, D., Lagae, L.: Pixel super-resolution for lens-free holographic microscopy using deep learning neural networks 27(10), 13581–13595 https://doi.org/10.1364/OE.27.013581 . Accessed 2024-02-02
  * [46] Kim, J., Song, S., Kim, H., Kim, B., Park, M., Oh, S.J., Kim, D., Cense, B., Huh, Y.-m., Lee, J.Y., Joo, C.: Ptychographic lens-less birefringence microscopy using a mask-modulated polarization image sensor 13(1), 19263 https://doi.org/10.1038/s41598-023-46496-z . Accessed 2024-02-02
  * [47] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778. IEEE. https://doi.org/10.1109/CVPR.2016.90 . http://ieeexplore.ieee.org/document/7780459/ Accessed 2023-12-19
  * Ren et al. [2021] Ren, Y., Wu, J., Xiao, X., Yang, J.: Online multi-granularity distillation for gan compression. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6773–6783. IEEE Computer Society, Los Alamitos, CA, USA (2021). https://doi.org/10.1109/ICCV48922.2021.00672 . https://doi.ieeecomputersociety.org/10.1109/ICCV48922.2021.00672
  * [49] Nan, L., Zhang, H., Weitz, D.A., Shum, H.C.: Development and future of droplet microfluidics https://doi.org/10.1039/D3LC00729D . Accessed 2024-01-23
  * [50] Weber, L., Schenk, D.: Automatische zusammenführung zertrennter konstruktionspläne von wasserbauwerken https://doi.org/10.1002/bate.202200010 . Accessed 2024-02-05
  * [51] 3D Spheroids | 3D Spheroid Cell Culture Models | Corning. https://www.corning.com/worldwide/en/products/life-sciences/applications/cell-culture/3D-cell-culture/spheroid-models.html Accessed 2024-02-14
  * [52] Berdeu, A., Flasseur, O., Méès, L., Denis, L., Momey, F., Olivier, T., Grosjean, N., Fournier, C.: Reconstruction of in-line holograms: combining model-based and regularized inversion 27(10), 14951–14968 https://doi.org/10.1364/OE.27.014951 . Publisher: Optica Publishing Group. Accessed 2023-12-19
  * [53] Denis, L., Lorenz, D., Thiébaut, E., Fournier, C., Trede, D.: Inline hologram reconstruction with sparsity constraints 34(22), 3475–3477 https://doi.org/10.1364/OL.34.003475 . Publisher: Optica Publishing Group. Accessed 2023-12-19
  * [54] Ho, J., Jain, A., Abbeel, P.: Denoising Diffusion Probabilistic Models. arXiv. http://arxiv.org/abs/2006.11239 Accessed 2022-10-06
  * [55] Zhao, M., Bao, F., Li, C., Zhu, J.: EGSDE: Unpaired Image-to-Image Translation via Energy-Guided Stochastic Differential Equations. arXiv. https://doi.org/10.48550/arXiv.2207.06635 . http://arxiv.org/abs/2207.06635 Accessed 2022-11-11
  * [56] Sasaki, H., Willcocks, C.G., Breckon, T.P.: UNIT-DDPM: UNpaired Image Translation with Denoising Diffusion Probabilistic Models. arXiv. http://arxiv.org/abs/2104.05358 Accessed 2023-02-10
  * [57] Liu, G.-H., Vahdat, A., Huang, D.-A., Theodorou, E.A., Nie, W., Anandkumar, A.: I2SB: Image-to-Image Schrödinger Bridge. arXiv. https://doi.org/10.48550/arXiv.2302.05872 . http://arxiv.org/abs/2302.05872 Accessed 2023-04-11
  * [58] Cui, Y., Song, Y., Sun, C., Howard, A., Belongie, S.: Large Scale Fine-Grained Categorization and Domain-Specific Transfer Learning. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4109–4118. IEEE. https://doi.org/10.1109/CVPR.2018.00432 . https://ieeexplore.ieee.org/document/8578530/ Accessed 2024-02-08
  * [59] Bai, Y., Geng, X., Mangalam, K., Bar, A., Yuille, A., Darrell, T., Malik, J., Efros, A.A.: Sequential Modeling Enables Scalable Learning for Large Vision Models. https://doi.org/10.48550/arXiv.2312.00785 . http://arxiv.org/abs/2312.00785 Accessed 2024-02-08
  * [60] Jia, Z., Tillman, B., Maggioni, M., Scarpazza, D.P.: Dissecting the Graphcore IPU Architecture Via Microbenchmarking. http://arxiv.org/abs/1912.03413 Accessed 2024-02-11
  * [61] Arcelin, B.: Comparison of Graphcore IPUs and Nvidia GPUsfor Cosmology Applications. http://arxiv.org/abs/2106.02465 Accessed 2024-02-11
  * [62] Peng, H., Ding, C., Geng, T., Choudhury, S., Barker, K., Li, A.: Evaluating Emerging AI/ML Accelerators: IPU, RDU, And NVIDIA/AMD GPUs. http://arxiv.org/abs/2311.04417 Accessed 2024-02-11
  * [63] Xie, S., Gong, M., Xu, Y., Zhang, K.: Unaligned image-to-image translation by learning to reweight. In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 14154–14164. https://doi.org/10.1109/ICCV48922.2021.01391 . ISSN: 2380-7504
  * [64] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need. https://doi.org/10.48550/arXiv.1706.03762 . http://arxiv.org/abs/1706.03762 Accessed 2023-04-06
  * [65] Wu, Q., Zhao, W., Yang, C., Zhang, H., Nie, F., Jiang, H., Bian, Y., Yan, J.: SGFormer: Simplifying and Empowering Transformers for Large-Graph Representations. https://doi.org/10.48550/arXiv.2306.10759 . http://arxiv.org/abs/2306.10759 Accessed 2024-02-29
  * [66] Zhu, J.-Y., Park, T., Isola, P., Efros, A.A.: Unpaired image-to-image translation using cycle-consistent adversarial networks. In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251. IEEE. https://doi.org/10.1109/ICCV.2017.244 . http://ieeexplore.ieee.org/document/8237506/ Accessed 2022-06-15
  * [67] Wang, Z., Simoncelli, E.P., Bovik, A.C.: Multiscale structural similarity for image quality assessment. In: The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003, vol. 2, pp. 1398–14022. https://doi.org/10.1109/ACSSC.2003.1292216 . https://ieeexplore.ieee.org/document/1292216 Accessed 2023-10-30
  * [68] Mao, X., Li, Q., Xie, H., Lau, R.Y.K., Wang, Z., Smolley, S.P.: Least Squares Generative Adversarial Networks. arXiv. https://doi.org/10.48550/arXiv.1611.04076 . http://arxiv.org/abs/1611.04076 Accessed 2023-12-19
  * [69] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization. arXiv. https://doi.org/10.48550/arXiv.1412.6980 . http://arxiv.org/abs/1412.6980 Accessed 2023-12-19
  * [70] Cunningham, R., Jia, S., Purohit, K., Salem, O., Hui, N.S., Lin, Y., Carragher, N.O., Hansen, C.G.: YAP/TAZ activation predicts clinical outcomes in mesothelioma and is conserved in in vitro model of driver mutations 13(2), 1190 https://doi.org/10.1002/ctm2.1190 . Accessed 2024-02-08
  * [71] Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-Image Translation with Conditional Adversarial Networks. arXiv. version: 3. http://arxiv.org/abs/1611.07004 Accessed 2023-11-07
  * [72] Zhang, H., Goodfellow, I., Metaxas, D., Odena, A.: Self-Attention Generative Adversarial Networks. arXiv. http://arxiv.org/abs/1805.08318 Accessed 2023-11-09
  * [73] Yang, S., Jiang, L., Liu, Z., Loy, C.C.: GP-UNIT: Generative Prior for Versatile Unsupervised Image-to-Image Translation. arXiv. version: 1. http://arxiv.org/abs/2306.04636 Accessed 2023-07-01
  * [74] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. In: Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2017/hash/8a1d694707eb0fefe65871369074926d-Abstract.html Accessed 2024-01-14
  * Fang et al. [2019] Fang, F., Wang, T., Fang, Y., Zhang, G.: Fast color blending for seamless image stitching. IEEE Geoscience and Remote Sensing Letters 16(7), 1115–1119 (2019)

## Acknowledgement

R.L. was supported by the Global Joint PhD Studentship between KU Leuven and
the University of Edinburgh. Y.Y. was supported by the Chancellor’s Fellowship
and Bayes Innovation Fellowship. C.H.: The early stages of this work were
supported by the JHMRF and Worldwide Cancer Research (19-0238). K.P. is
supported by an MRC Precision Medicine DTP Studentship.

## Authors contributions

R.L. contributed to all facets of this study, encompassing conceptualization,
algorithm design, experimentation, analysis, interpretation, figure creation,
and manuscript writing. Y.Y. and X.C. shared supervisory responsibilities for
this project. P.B. supervised the optics experiments and offered valuable
domain expertise in optics. Z.L. contributed to the panorama stitching
algorithm. Additionally, P.B., M.W., K.P., G.F., Y.F., and C.H. contributed by
providing samples and expertise for experimentation.

## Competing interests

The authors declare no competing interests.

## Data and code availability

Data and codes for the implementation are available at https://github.com/RL-
arch/LensGAN.

Supplementary Information

##  1 Zebrafish imaging

We applied GenLFI to in vivo samples of zebrafish. Zebrafish were cultured to
3 days old, anesthetized by MS222, and placed under petri dishes to be
observed. Due to the structure of zebrafish, the measured hologram on our
sensor (resolution $5.94\,\mu m/px$) can provide very limited information
inside the fish (see Fig.S1). Yet, our system can still reconstruct the
zebrafish’s basic structural information from raw holograms in a large FOV.

Figure S1:  a, Details of imaging for zebrafish; b, Synthetized full FOV
images.

##  2 Further discussion on complex optical field

Here we further discuss the optical field with large axial size samples and
mathematically prove the inadequacy of conventional model-based 3D lens-free
holographic microscopy (LFHM) tomographic reconstruction approaches. To this
end, we analyze the current 3D LFHM algorithm based on scalar diffraction
theory for 3D objects.

As shown in Fig.1 in the main text, the solution to the light propagation
problem is described as:

$u(\boldsymbol{r})=u^{inc}(\boldsymbol{r})+u^{sct}(\boldsymbol{r})$ (11)

where $u^{inc}(\boldsymbol{r})$ represents the incident light and
$u^{sct}(\boldsymbol{r})$ represents the scattered light. The Lippmann-
Schwinger equation (LSE) depicts the relationship between the 3D scattering
potential of the sample and the resulting scalar field at region $V$:

$u(\boldsymbol{r})=u^{inc}(\boldsymbol{r})+\int_{V}G(\boldsymbol{r}-\boldsymbol{r}^{\prime})f(\boldsymbol{r}^{\prime})y(\boldsymbol{r}^{\prime})dr^{\prime}$
(12)

where $G(\boldsymbol{r})=\frac{1}{4\pi R}\exp(jk_{b}R)$, $k_{b}=k_{0}n_{b}$ is
the wavenumber of the light in the background medium, and $R$ denotes the
length of the Euclidean vector $\boldsymbol{r}$. Discretized and denoted by
vectors, the resulting scalar field $\boldsymbol{\hat{u}}$:

$\widehat{\boldsymbol{u}}=\widehat{\boldsymbol{u}}^{inc}+\widehat{\boldsymbol{G}}\boldsymbol{\operatorname{diag}(f)}\boldsymbol{u}^{inc}=\mathcal{L}(\boldsymbol{f})$
(13)

where $\mathcal{L}(\cdot)$ is the light propagator,
$\boldsymbol{\operatorname{diag}(f)}\in\mathbb{C}^{N\times N}$ denotes a
diagonal matrix whose main diagonal entries consist of elements from the
vector $\boldsymbol{f}$. With multiple light sources, the final forward model
of the 3D LFHM can be presented by the intensity of the optical field
$\widehat{\boldsymbol{H}}$ under $p^{th}$ illumination:

$\widehat{\boldsymbol{H}}=\left|\widehat{\boldsymbol{u}}^{inc}+\widehat{\boldsymbol{G}}\boldsymbol{\operatorname{diag}(f)}\boldsymbol{u}^{inc}\right|^{2}=\left|\mathcal{L}_{p}(\boldsymbol{f})\right|^{2}$
(14)

The reconstruction algorithm can be formulated as the optimization problem:

$\boldsymbol{f}^{*}=\arg\min_{f\in\mathbb{C}^{N}}\\{\Sigma_{p=1}^{P}d(\left|\mathcal{L}_{p}(\boldsymbol{f})\right|^{2},\widehat{\boldsymbol{H}}_{p})\\},$
(15)

where $\mathcal{L}_{p}(\cdot)$ denotes the light propagator,
$\boldsymbol{f}^{*}$ denotes the amplitude of the light,
$\widehat{\boldsymbol{H}}_{p}$ represents the measured light intensity under
the $p^{th}$ illumination.

In the microfluidics experiment, the mediums consist of air, PDMS, water, oil,
and glass. In this condition, different components of the electromagnetic
field may make the vector diffraction theory of $u(\boldsymbol{r})$ in (11)
ineffective. Furthermore, the model-based algorithm’s approximation of the
hologram in the 2D space is affected by the thickness of the sample. For
example, the Born approximation for (13) is valid only if
$\boldsymbol{u}^{sct}\ll\boldsymbol{u}^{inc}$. This corresponds to
$l(n_{s}-n_{b})\ll\lambda$ , where $l$ is the thickness, $(n_{s}-n_{b})$
describes the medium’s refractive index variation, and $\lambda$ is the
wavelength of the incident light. Consider a PDMS chip that has water inside,
with a thickness of $5000\,\mu m$; there are $l(n_{s}-n_{b})$ =
$5000\times(1.40-1.33)=350\,\mu m$, while $\lambda=0.42\,\mu m$. It is obvious
that the model is not applicable to the condition of Born approximation for
large samples with strong scattering.

To provide a more accurate description of the physical process, some non-
linear models have been proposed [30]. These iterative methods involve either
multiple iterations by variations or dividing the sample into a series of thin
layers. However, these methods are impractical to apply in real experiments
and are restricted to sample-specific tools.

In summary, real-world scattered fields are complex and difficult to model.
The existing forward model-based LFI systems are limited to small and thin
samples.

##  3 Full architecture of LensGAN

Figure S2: The full architecture of LensGAN.

##  4 Discriminator design

Figure S3:  Architecture of the 5-convolutional-layer discriminator. f:
filter, k: kernel size, s: stride, p: padding. The self-attention layer
calculates the self-attention scores from 3 vectors: query, key, and value.
Then it uses the scores to create an attended feature map that is added back
to the input.

The discriminator network is based on $70\times 70$ PatchGANs [71], which aims
to classify whether $70\times 70$ overlapping image patches are real or fake.
The input images under our hyperparameter setting are $256\times 256$, then
downsampled by convolutional layers. The output is a one-channel prediction
map to discriminate images. As shown in Fig.S3, the convolutional layers only
process information in local neighborhoods. The sequential Markovian modeling
of long-range dependencies in images may cause a limited receptive field.
Therefore, a self-attention layer is added to solve this problem and make the
training more robust.

We implement this self-attention layer with the feature map used in SAGAN
[72]. The input image from the previous convolutional layer contains features
$x\in\mathbb{R}^{C\times N}$ that are transformed into two feature spaces
$\mathnormal{f}$ and $\mathnormal{g}$. Here, $C$ represents the number of
channels, and $N$ represents the number of feature locations. The attention
function is calculated by the following equation where $f(x)=W_{f}x$ ,
$g(x)=W_{g}x$ :

$\beta_{j,i}=\frac{\exp(s_{ij})}{\sum_{i=1}^{N}\exp(s_{ij})},\text{where
}s_{ij}=f(x_{i}^{T})g(x_{j})$ (16)

$\beta_{j,i}$ indicates the range of attention where the model computes
mapping of the $i^{th}$ location of the $j^{th}$ feature regions. The output
of the attention layer is
$a=(a_{1},a_{2},...,a_{j},...,a_{N})\in\mathbb{R}^{C\times N}$ , where

$a_{j}=v(\sum_{i=1}^{N}\beta_{j,i}h(x_{i})),h(x_{i}=W_{h}x_{i}),v(x_{i}=W_{v}x_{i})$
(17)

$W_{g}\in\mathbb{R}^{\overline{C}\times C}$ ,
$W_{f}\in\mathbb{R}^{\overline{C}\times C}$ ,
$W_{h}\in\mathbb{R}^{\overline{C}\times C}$ , and
$W_{v}\in\mathbb{R}^{C\times\overline{C}}$ are three learned weight metrics
that are implemented as $1\times 1$ convolutions. Practically, for memory
efficiency, the channel number of $\overline{C}$ is reduced to be $C/k$ ,
where $k$ is set as 8. By multiplying $a_{i}$ a learnable scale parameter
$\gamma$ and adding back the input feature map $x_{i}$, the final scaled
output is:

$y_{i}=\gamma o_{i}+x_{i}{.}$ (18)

##  5 Ablation study

Figure S4: Visual comparison of results by different I2I algorithms. We
selected 3 cases from the ’Hologram2Microscopy’ dataset and evaluated
different algorithms with FID: classical iterative holographic reconstruction
algorithm [22], our algorithm LensGAN, CycleGAN [66], our backbone IrwGAN
[63], IrwGAN enhanced with our self-attention discriminator, and SOTA unpaired
I2I algorithm GP-UNIT [73].

We evaluated LensGAN on an unseen test set (images from full ROI frames) with
Fréchet Inception Distance (FID) [74], as depicted in Fig.S4. FID is the
current standard for assessing the quality of generative models, where a lower
FID indicates a better correlation between the distributions of generated
images and ground truth. In the ablation study, we replaced the discriminator
in IrwGAN [63] with our own (enhanced with self-attention blocks, details in
Discriminator design), denoted as ‘IrwGAN + Self-attention.’ We also evaluated
the CycleGAN [66] and a SOTA unpaired I2I model GP-UNIT [73] on our dataset.

Fig.S4 shows the results that compared to the IrwGAN [63] backbone, the new
discriminator improves the FID score by 3.27, and together with the
$\mathcal{L}_{\mathrm{PC}}$, the LensGAN improves the FID score by 10.41. The
results in Fig.S4 demonstrate that our model achieved the lowest score and
successfully addressed the main distortions and artifacts.
$\mathcal{L}_{\mathrm{PC}}$ constrains the generated results effectively,
preventing the line artifacts in IrwGAN (see cases 1 & 3), and our
discriminator $D$ improves the reconstructed droplets’ illumination details
(see case 2).

##  6 Post-processing: Global panorama image stitching

After processing the image blocks separately, the results will be stitched
together to create the final panorama. This operation is divided into two
stages. The first stage involves intensity correction and image assembly. The
second stage aims to remove slowly varied artifacts.

For the first stage, considering two image blocks to be stitched, the color
difference usually occurs between them, causing a noticeable seam. We define
one image as the reference image $I^{r}$ and the other image as the target
image $I^{t}$. Pixel points at the seam line are denoted by
$p_{i},i=1,2,...,n$. Then the color difference $\Delta C(p_{i})$ between
$I^{r}$ and $I^{t}$ at the seam line is defined as:

$\Delta C(p_{i})=V^{r}(p_{i})-V^{t}(p_{i}),~{}~{}i=1,2,...,n$ (19)

where $V^{r}(p_{i})$ denotes the average pixel intensity at $p_{i}$ in the
$I^{r}$ and $V^{t}(p_{i})$ represents the average pixel intensity at $p_{i}$
in the $I_{t}$. $V^{r}(p_{i})$ is calculated by averaging the pixel intensity
at $p_{i}$ with its nearest three neighbors at the same row in the $I^{r}$.
And the same approach is adopted to estimate $V^{t}(p_{i})$. For each pixel
$p^{t}$ in the $I^{t}$, the intensity change $\Delta V$ is defined as [75]:

$\Delta V(p^{t})=\frac{1}{\sum_{i=1}^{n}w_{i}}\sum_{i=1}^{n}w_{i}\Delta
C(p_{i}),$ (20)

where the weight $w_{i}$ is expressed as:

$w_{i}=\mathrm{exp}\left(-\left(\frac{||V^{t}(p_{i})-I^{t}(p^{t})||^{2}}{\sigma_{1}^{2}}+\frac{||L(p_{i})-L(p^{t})||^{2}}{\sigma_{2}^{2}}\right)\right).$
(21)

Then, the corrected intensity at $p^{t}$ is formulated as:

$\overline{I^{t}(p^{t})}=I^{t}(p^{t})+\Delta V(p^{t}).$ (22)

There exist slowly varied artifacts in the generated panorama at the first
stage. In the second stage, we treat this type of artifact as the low-
frequency component of the image. Therefore, we can use the Gaussian filter to
estimate these artifacts. Suppose the initial panorama is denoted by $P$, the
artifact image is estimated by:

$A=P\ast\kappa,$ (23)

where $\ast$ represents the convolution, and $\kappa$ is the Gaussian kernel.
Therefore, the final panorama $\overline{P}$ is expressed as:

$\overline{P}=P-A.$ (24)

Figure S5: Stitching result for microfluidics. Up: fast reconstruction. Down:
after panorama stitching.  Figure S6: Stitching result for spheroids. Top
left: input raw hologram; Bottom left: fast reconstruction from LensGAN; Top
right: detected shadow; Bottom right: after global panorama stitching and
shadow removing.

##  7 Microfluidic chip fabrication

We fabricated a microfluidic chip with a flow-focusing geometry for droplet
production on PDMS bonded to a glass slide, and we conducted droplet
generation experiments to observe these dynamic samples across a microfluidic
system encompassing the droplet production area, a serpentine channel for
droplet incubation and an expansion area for enabling droplet contact and
merging. To fabricate the mold from which the microfluidic chip is made,
photolithography is used. A $35\,\mu m$ thick layer of photoresist (SU-8 2015)
is spin-coated, pre-baked, exposed, post-baked, and developed.

After the silicone wafer mold containing the design is created, a PDMS base
and curing agent (Sylgard 184) are combined in a 10:1 ratio in a plastic cup
and stirred for several minutes using a spatula. The plastic cup is placed in
a vacuum desiccator to extract trapped air in the PDMS/curing agent mix. After
approximately 20 minutes the mix was poured over the earlier created master,
after which it was placed in the vacuum desiccator until no bubbles were
visible anymore within the PDMS/curing agent mix. After this, the mix was
placed in an oven at $65^{\circ}$C for 3 hours to cure the PDMS.

After the PDMS was cured, the PDMS was removed from the master, and separate
devices were cut from this removed PDMS slab. Inlet and outlet holes were
punched using a $1.2\,mm$ biopsy punch. Using scotch tape, dust was removed
from both the glass slide and the PDMS devices, after which both were exposed
to oxygen plasma in a plasma cleaner (Harrick PDC-002). The PDMS-coated glass
slides and PDMS devices were then bonded by carefully connecting them.

Finally, the PDMS chip is silanized using a 5 w% silane and HFE-7500 mixture.

##  8 Spheroid genotypes

The genotypes included are the wild type (WT) as well as two MeT5A clones with
knockout (KO) of the NF2 gene – NF2 KO #1 and NF2 KO #2 (where #1 and #2
clones were generated using CRISPR constructs targeting different regions the
gene). The NF2 gene has been found to have loss of function mutations in
several cancers, including mesothelioma, and here these NF2 KO cell lines were
generated as a cellular model system for mesothelioma.

##  9 Failure case

Figure S7: Failure case of singular sample: reconstruction for an organ-on-
chip. We used a single raw hologram and BF microscopy image for training. Due
to the low data scale, although the reconstruction achieved a high SSIM score,
some information are lost in the reconstruction.


# A Generic Knowledge Based Medical Diagnosis Expert System

Xin Huang<EMAIL_ADDRESS>University of Maryland, Baltimore CountyUSA ,
Xuejiao Tang<EMAIL_ADDRESS>Leibniz University of
HannoverGermany ,  Wenbin Zhang<EMAIL_ADDRESS>University of Maryland,
Baltimore CountyUSA ,  Shichao Pei<EMAIL_ADDRESS>King Abdullah
University of Science and TechnologySaudi Arabia ,  Ji Zhang
<EMAIL_ADDRESS>University of Southern QueenslandAustralia ,  Mingli Zhang
<EMAIL_ADDRESS>Mcgill UniversityCanada ,  Zhen Liu
<EMAIL_ADDRESS>Guangdong University of Foreign StudiesChina ,  Ruijun
Chen<EMAIL_ADDRESS>Polytechnic University of MilanItaly and
Yiyi Huang<EMAIL_ADDRESS>University of ArizonaUSA

(2021)

###### Abstract.

In this paper, we design and implement a generic medical knowledge based
system (MKBS) for identifying diseases from several symptoms. In this system,
some important aspects like knowledge bases system, knowledge representation,
inference engine have been addressed. The system asks users different
questions and inference engines will use the certainty factor to prune out low
possible solutions. The proposed disease diagnosis system also uses a
graphical user interface (GUI) to facilitate users to interact with the expert
system. Our expert system is generic and flexible, which can be integrated
with any rule bases system in disease diagnosis.

††journalyear: 2021††copyright: acmcopyright††conference: The 23rd
International Conference on Information Integration and Web Intelligence;
November 29-December 1, 2021; Linz, Austria††booktitle: The 23rd International
Conference on Information Integration and Web Intelligence (iiWAS2021),
November 29-December 1, 2021, Linz, Austria††price: 15.00††doi:
10.1145/3487664.3487728††isbn: 978-1-4503-9556-4/21/11††ccs: Computer systems
organization Medical Diagnosis††ccs: Computer systems organization Expert
System ††ccs: Computer systems organization Artificial Intelligence

##  1\. Introduction

AI systems are now widely used in economics, medicine, engineering and social
medias, and are also built into many software applications and computer
strategy games such as computer chess and computer video games. Expert system
can process large amounts of known information and apply reasoning
capabilities to provide conclusions. An expert system is a system that employs
human knowledge captured in an automated system to solve problems that
typically require human expertise.

In this paper we propose the design and development of a medical knowledge
based system (MKBS) for disease diagnosis from symptoms. It provides rich
features for searching properties like symptoms, treatments, hierarchical
clusters of particular diseases. The system supports a knowledge construction
module and an inference engine module. The knowledge construction was built on
a concept of rules, which was represented in a tree structure, and properties
of a particular disease were stored as a semantic net. The inference engine
uses the interactive backward chaining technique to infer a diagnostic result
and. This medical expert system has been developed using JIProlog interpreter
and Java - which provides an interactive user interface by asking questions
and a visualization module that can be used by inference engine to produce
visual decision making tree. The latter one can be used by explanation
subsystem for visualizing queries. Furthermore, this system has a knowledge
base editor module to dynamically add, edit, and drop rules and their
premises.

Our Diagnosis Expert System’s knowledge base has 23 diseases from different
categories, as shown in Table 1. The user is asked to answer with certainty
factor, Yes or No if a certain symptom appears or not. Based on the user’s
answers and the reasoning of the inference engine, the name of the disease is
posted on the screen along with its certainty factor.

Table 1. Diseases diagnosed in our system. Cancer | Anxiety Disorders | Balding and hair loss | Thyroid disorders | Heart attack
---|---|---|---|---
Diabetes | Asthma | Erectile disorders | Migraine | Heart disease
Allergies | Prostate conditions | Lupus | Skin disorders | Chest Pain
Cohn’s Disease | Abdominal Pain | Thyroid disorders | Eye disorders | Respiratory problem
Common Cold | Anxiety | Impotence | Headache | Abdominal Pain
Diarrhea | Tonsil | Speech problem |  |

The contributions of the proposed medical diagnosis expert system are:

  * •

Build a medical expert software system for identifying diseases by several
symptoms and describing methods of treatment to be carried out.

  * •

Use MKBS system to diagnose a major disease by providing symptoms or get
properties information about medical diseases.

  * •

Provide a generic GUI visualization tool for any rule based expert system
build on core exshell. Users can use this visualization tool to understand how
the system making decisions about a disease.

  * •

Provide a knowledge based GUI editor to dynamically add, update or delete
rules in current working memory.

##  2\. Related Work

Expert Systems are computer programs that are derived from Artificial
Intelligence (AI). AI’s scientific goal is to understand intelligence and gain
insights by building computer programs that exhibit intelligent behavior
(Luger and Stubblefield, 2009). It contains concepts and methods of symbolic
inference, or reasoning by a computer, and the mechanism of using the
knowledge to make inferences inside the machine.

The development of traditional expert systems was aided by the development of
the symbolic processing languages Lisp and Prolog (Merritt, 2012). To avoid
reinventing the wheel, expert system shells were created with specialized
features for building large expert systems.

Recent advances in the field of artificial intelligence have led to the
emergence of medical expert systems (Salem, 2012) (Richens et al., 2020).
Successful expert systems MYCIN was an early expert system designed to
identify bacteria causing severe infections, such as bacteremia and
meningitis, and to recommend antibiotics treatment (Watcharachai and Walita,
2009). Developing a reliable medical diagnosis system is a complex task.
Recently several fuzzy logic based expert systems have been developed for
diagnosis of specific diseases for coronary artery disease (Muhammad and
Algehyne, 2021) and chronic kidney disease (Singla et al., 2020).

In this paper, we aim to develop a generic knowledge based medical expert
system. This medical expert system has the scope of representing some major
diseases’ symptoms and properties. One of the main focuses of this medical
system is to develop graphical user interface to visualize decision making and
provide a knowledge-based GUI editor.

Figure 1. High level design of knowledge based disease diagnosis expert
system.

##  3\. An Expert Framework for Disease Diagnosis

An expert system is composed of two major parts, a knowledge base and an
inference engine. The inference engine reasons about the knowledge base like a
human to run an expert system. A dialog interface is used to communicate
between users and the system, which has ability to conduct a conversation with
users via ”conversational” interface.

Our medical expert system is built with typical components of an expert
system. To introduce a visualization module in exshell, this system uses Java,
JIProlog - a java implementation of prolog (JIProlog, 2015) and visualization
tool Yfiles (Yfiles, 2015). Figure1 shows the high level design of our
knowledge based disease diagnosis expert system. This system has all of the
necessary components that an expert system must have. The major components
are:

  * •

Knowledge base - A declarative representation of the expertise, can either be
the IF-THEN rules based or knowledge base representation such as semantic net
and frame based representation.

  * •

Inference engine - It is the core of the system, which derives recommendations
from the knowledge base and problem-specific data in working memory.

  * •

Working memory - It is used to store user inputs and for further matching
queries, which help to not ask duplicate questions.

  * •

User interface - It is the interface that controls the interaction between the
user and the core expert system. This component has been built using JIProlog
and Java.

###  3.1. Knowledge structure

The knowledge construction was built on a concept of rules that were performed
in a hierarchical tree structure. For this part, rule based knowledge with a
hierarchical tree structure is used to represent the knowledge about different
diseases and its symptom.

This system has its own internal rule format with header information and all
premises are connected by AND or OR conditions. This format has a section for
representing a certainty factor of rules. This certainty factor will be used
by the inference engine to make decisions about a possible solution. The rule
can be nested, which means a rule can have nested sub-goal rules and different
facts.

###  3.2. Inference Engine

An inference engine is used to derive answers from the knowledge base. A
finite state machine in the inference engine can be described with a cycle of
three action states as match rules, select rules, and execute rules. Rules are
designed to be a tree like hierarchical structure to represent the knowledge
about different diseases and their symptoms. With the first state of match
rules, the inference engine searches all of the rules that are satisfied by
the current contents of the data target. In the second state, the inference
engine applies some selection strategy to determine which rules will actually
be executed. Finally, the inference engine executes or fires the selected
rules, with the instantiating data items as parameters.

Using this format of rules, the inference engine of this system can reason
knowledge about diseases by several symptoms. The main desired behaviors of
inference engine are:

  * •

combine certainty factors as indicated previously.

  * •

maintain working memory that is updated as new evidence is acquired.

  * •

locate all information specific to an attribute when it is asked for, and
store that information in working memory.

Backward chaining approach

The system works backward from the ultimate goal until all the sub-goals in
working memory are known to be true, indicating that the hypothesis has been
verified. This process is known as “backward chaining”.

Steps in backward chaining: System has hypothetical solution(s) (e.g. “The
patient has type I diagnosis(X)”), and tries to prove it.

  * •

Find rules that match the goal.

  * •

If antecedents match the facts, stop.

  * •

If not, make the antecedents the new sub-goals, and repeat.

Certainty Factors: The most common scheme for modeling uncertainty is to
assign a certainty factor to each piece of information in the system. The
inference engine automatically updates and maintains the certainty factors as
the inference proceeds. The certainty factors ($CF$) are integers from 0 (for
definitely false) to 1 (for definitely true). The rule format also allows for
the addition of certainty factors.

When the premise of a rule is uncertain due to uncertain facts, and the
conclusion is uncertain due to the specification in the rule, the following
formula is used to compute the adjusted certainty factor of the conclusion:

(1) $CF=RuleCF*PremiseCF$

###  3.3. Working memory

One can treat working memory as the brain processes used for temporary storage
and manipulation of information. It operates over a short time window which is
usually a few seconds, and it facilitates users to focus attentions, resist
distractions, and guide the decision-making.

Working memory simply contains the known facts about attribute-value pairs.
Known facts, represented as $Known(Goal,CF)$, are stored in the Prolog
database for further use. If a current fact and its certainty fact have been
already stored in working memory, the system would not ask users about it
further and it will calculate certainty fact using the stored data.

Figure 2. End to end implementation of our disease diagnosis system.

###  3.4. User interface

One of the important contribution in our expert system is introducing a user
interface to facilitate the visualization of decision making process.
Traditionally, most of the prolog program’s output are texts, which are
difficult for users to understand the solution provided by expert system. To
provide a user friendly interface for the expert system, we develop a
graphical user interface that can be used in any rule based system that is
built on top of the core exshell system. This user interface is a generic
interface which can be easily plugged into any rule based system built on top
of core exshell.

##  4\. Framework Implementation

The implementation of our disease diagnosis systems is shown in Fig. 2. One of
the main goals of this system is to introduce a user friendly user interface
for such a rule based expert system. With the combination of Java and prolog,
we develop a complete end to end GUI based expert system for disease
diagnosis. Three main technologies used are:

  * •

Java: Java is a widely used high level programming language.

  * •

JIProlog: Java Internet Prolog is a cross-platform pure Java 100% Prolog
interpreter that integrates Prolog and Java languages in a very fascinating
way. JIProlog allows calling Prolog predicates from Java without dealing with
native code (JNI). JIProlog can consume any prolog file that has ISO standard
prolog syntax.

  * •

Yfiles: YFiles for Java is an extensive Java class library that provides
algorithms and components enabling the analysis, visualization, and the
automatic layout of graphs, diagrams, and networks.

JIProlog is a Java version of Prolog implementation and it can load and
consume valid prolog files. JIProlog server supports socket communication.
From the java end, a socket will be open on a particular port and wait for the
client to connect. JIProlog server will be launched by loading a prolog file
which will contain the core exshell prolog code and knowledge representation
of data. JIProlog server is configured to communicate via socket and
send/receive user query/output to and from Java application. When a JIProlog
server finds a solution, an event is triggered and the solution is parsed from
the Java end to visualize different results that have been provided by
exshell. Yfiles has been used to visualize the tree diagram.

Figure 3. Graphical visualization of diagnosis with certainty factor.

##  5\. Case Studies

Fig. 3 shows an example of the diagnosis of Tuberculosis using our medical
diagnosis expert system. Our system asks several question about symptoms of
diseases. The inference engine reasons about the knowledge base and generates
decision making tree. The diagnosis result with certainty factor is visualized
on the graphical user interface.

Fig. 4 demonstrates an example of searching particular properties of lung
cancer using our expert system. Our system has a simple semantic
representation of diseases properties. In this system, several diseases
functionalities such as disease details, treatments, hierarchical diseases
information can be store in a semantic net. For example, lung cancer is a
specific type of cancers, while mesothelioma and primary lung cancer are
specific types of lung cancers, respectively. The treatment of the lung cancer
can be surgery, radio therapy, chemotherapy, hormonal therapy, etc. By using
knowledge representation for properties of diseases, our expert system can
serve user’s query to return a particular property of diseases and generate
the knowledge tree visualization on the graphical user interface.

Figure 4. Searching particular properties of lung cancer using our expert
system.

##  6\. Conclusion

In this paper, we design and implement a medical knowledge based expert system
for disease diagnosis from symptoms. The system provides features for
searching properties like symptoms, treatments, hierarchical clusters of
particular diseases. The proposed disease diagnosis system uses a graphical
user interface to facilitate users to interact with an expert system more
easily. This system is generic, knowledge based, and can be integrated with
any rule based expert system for diseases diagnosis. For future works, we plan
to increase the capacity of our expert system by implementing more inference
rules for semantic based knowledge representation and conduct real user study
with patients and medical professionals.

## Acknowledgement

This work was supported by a NVIDIA GPU Grant.

## References

  * (1)
  * JIProlog (2015) JIProlog. 2015.  _Java Internet Prolog is a cross-platform pure Java 100% Prolog interpreter_.  http://www.jiprolog.com/
  * Luger and Stubblefield (2009) George F Luger and William A Stubblefield. 2009.  _AI algorithms, data structures, and idioms in Prolog, Lisp, and Java_.  Pearson Addison-Wesley.
  * Merritt (2012) Dennis Merritt. 2012\.  _Building expert systems in Prolog_.  Springer Science; Business Media.
  * Muhammad and Algehyne (2021) LJ Muhammad and Ebrahem A Algehyne. 2021.  Fuzzy based expert system for diagnosis of coronary artery disease in nigeria.  _Health and Technology_ 11, 2 (2021), 319–329.
  * Richens et al. (2020) Jonathan G Richens, Ciarán M Lee, and Saurabh Johri. 2020\.  Improving the accuracy of medical diagnosis with causal machine learning.  _Nature communications_ 11, 1 (2020), 1–9.
  * Salem (2012) Abdel-Badeeh Mohamed Salem. 2012\.  Intelligent Technologies and Methodologies for Medical Knowledge Engineering.  _Global Telemedicine and eHealth Updates: Knowledge Resources_ 5 (2012).
  * Singla et al. (2020) Jimmy Singla, Balwinder Kaur, Deepak Prashar, Sudan Jha, Gyanendra Prasad Joshi, Kyungyun Park, Usman Tariq, and Changho Seo. 2020\.  A Novel Fuzzy Logic-Based Medical Expert System for Diagnosis of Chronic Kidney Disease.  _Mobile Information Systems_ 2020 (2020).
  * Watcharachai and Walita (2009) Wiriyasuttiwong Watcharachai and Narkbuakaew Walita. 2009.  Medical Knowledge-Based System for Diagnosis from Symptoms and Signs.  _international journal of applied biomedical engineering_ 2, 1 (2009).
  * Yfiles (2015) Yfiles. 2015.  _Java API for graphics model diagram generations_.  https://www.yworks.com/products/yfiles-for-java-2.x