# Local Conditional Neural Fields for Versatile and Generalizable Large-Scale
Reconstructions in Computational Imaging

Hao Wang  Department of Electrical and Computer Engineering, Boston
University, Boston, MA 02215, USA.  Jiabei Zhu  Department of Electrical and
Computer Engineering, Boston University, Boston, MA 02215, USA.  Yunzhe Li
Department of Electrical and Computer Engineering, Boston University, Boston,
MA 02215, USA.  Qianwan Yang  Department of Electrical and Computer
Engineering, Boston University, Boston, MA 02215, USA.  Lei Tian  Department
of Electrical and Computer Engineering, Boston University, Boston, MA 02215,
USA.  Department of Biomedical Engineering, Boston University, Boston, MA
02215, USA.  Correspondence<EMAIL_ADDRESS>
###### Abstract

Deep learning has transformed computational imaging, but traditional pixel-
based representations limit their ability to capture continuous, multiscale
details of objects. Here we introduce a novel Local Conditional Neural Fields
(LCNF) framework, leveraging a continuous implicit neural representation to
address this limitation. LCNF enables flexible object representation and
facilitates the reconstruction of multiscale information. We demonstrate the
capabilities of LCNF in solving the highly ill-posed inverse problem in
Fourier ptychographic microscopy (FPM) with multiplexed measurements,
achieving robust, scalable, and generalizable large-scale phase retrieval.
Unlike traditional neural fields frameworks, LCNF incorporates a local
conditional representation that promotes model generalization, learning
multiscale information, and efficient processing of large-scale imaging data.
By combining an encoder and a decoder conditioned on a learned latent vector,
LCNF achieves versatile continuous-domain super-resolution image
reconstruction. We demonstrate accurate reconstruction of wide field-of-view,
high-resolution phase images using only a few multiplexed measurements. LCNF
robustly captures the continuous object priors and eliminates various phase
artifacts, even when it is trained on imperfect datasets. The framework
exhibits strong generalization, reconstructing diverse objects even with
limited training data. Furthermore, LCNF can be trained on a physics simulator
using natural images and successfully applied to experimental measurements on
biological samples. Our results highlight the potential of LCNF for solving
large-scale inverse problems in computational imaging, with broad
applicability in various deep-learning-based techniques.

##  1 Introduction

Deep learning has revolutionized the field of computational imaging [1, 2],
providing powerful solutions to enhance performance and address various
challenges in areas such as phase retrieval [3, 4, 5, 6, 7], digital
holography [8, 9], diffraction tomography [10, 11, 12], ghost imaging [13],
super-resolution imaging [14, 15, 16], lightfield imaging [17, 18, 19],
lensless imaging [20, 21], and imaging through scattering media [22, 23, 24].
Computational imaging treats the image formation process as a two-step
procedure: the object information is first physically encoded in the
measurement through the imaging optics, and then the information is
computationally reconstructed by solving an inverse problem. The effectiveness
of deep learning in computational imaging lies in their ability to capture the
underlying imaging model and exploit object priors, enabling robust solutions
to ill-posed inverse problems [1]. However, the most widely used
reconstruction methods in computational imaging rely on discrete pixels to
represent the objects. For instance, a Convolutional Neural Network (CNN) for
computational imaging is typically trained on a fixed pixel or voxel grid [1].
This representation is inherently limited by the resolution and size of the
grid and does not capture the _continuous_ nature and _multiscale_ details of
the physical objects. Furthermore, the pixel grid representation poses
challenges in scaling to process and store large-scale multi-dimensional
computational imaging data [12].

To overcome these limitations, we propose a novel deep learning framework
called Local Conditional Neural Fields (LCNF) to solve the imaging inverse
problem using a _continuous-domain_ implicit neural representation that is
both compact and highly generalizable. By utilizing a continuous
representation of objects, the LCNF framework offers a more natural and
flexible representation that can capture fine-grained details and reconstruct
object features of varying scales. We showcase the unique capabilities of LCNF
to solve the highly ill-posed inverse problem in Fourier ptychographic
microscopy (FPM) with multiplexed measurements [25, 26], demonstrating robust,
scalable, and generalizable large-scale phase retrieval.

The Neural Fields (NF) framework [27] has recently gained significant interest
in computer vision for its ability to represent and render continuous 3D
scenes [28]. Unlike traditional CNN structures, NF uses a coordinate-based
representation, where spatial coordinates (e.g. $(x,y)$) are mapped to
physical values (e.g. $[0,1]$) using a multi-layer perceptron (MLP). This
unique characteristic of NF allows for the encoding of objects in a
_continuous_ representation, decoupled from a discrete grid. It enables on-
demand synthesis of any part of the object by simply querying relevant
coordinates across arbitrary dimensions and resolutions. Several NF-based deep
learning techniques have been introduced in computational imaging for solving
inverse problems using continuous object representations [29, 30, 31, 32, 33,
34, 12, 35, 36]. However, these methods are limited by the high computational
cost and limited generalization ability. They either require retraining a new
NF network for each object reconstruction [29, 30, 31, 32, 33, 34, 12, 36] or
suffer from the limited representation power of the latent space learned only
on the global scale [35], restricting their ability to generalize to diverse
objects.

Our proposed LCNF framework overcomes these limitations by leveraging a _local
conditional_ NF representation. The _conditional_ representation embeds
measurement-specific information into the latent space that promotes model
generalization. Additionally, the _local_ representation allows for the
incorporation of multiscale information and enables efficient processing of
large-scale imaging data. Together, LCNF enables highly scalable and
generalizable deep learning-based image reconstructions.

Figure 1: Conceptual illustration of our LCNF framework for FPM
reconstruction. (a) LCNF employs a CNN encoder to learn measurement-specific
information and encode them into a latent-space representation. The MLP
decoder reconstructs the phase values at specific locations with an increased
spatial resolution by synthesizing local conditional information from the
corresponding latent vectors. (b) FPM experimental setup and illumination
patterns for acquiring multiplexed BF and DF measurements. (c) Example low-
resolution BF measurement, high-resolution phase reconstruction from the
model-based FPM algorithm, and our LCNF method. The LCNF learns a
generalizable continuous-domain representation and can render phase maps on an
arbitrary pixel grid (illustration with $6\times,21\times,49.8\times$
upsampling compared with the raw measurement image).

A conceptual illustration of our proposed LCNF framework for FPM
reconstruction is shown in Fig. 1(a). Building on the concept of conditional
NF from prior work [27], our framework utilizes a CNN-based _encoder_ to learn
measurement-specific information from a set of 2D multiplexed FPM measurements
and encode them into a compact latent-space representation. In FPM, the phase
information of an object point is spread across multiple pixels on the
measured images due to light diffraction. The CNN-based encoder effectively
extracts this information by utilizing its extended receptive field,
condensing them into latent vectors. Next, an MLP _decoder_ is employed to
reconstruct the phase values of the object at specific locations based on the
corresponding latent information. Unlike the traditional NF framework [28, 29,
30, 31, 33, 34, 12] that perform a one-to-one mapping between a single
coordinate to the corresponding object value, our decoder is conditioned on a
learned latent vector that incorporates information across a region of the
input images. This conditioning enables adaptation to different objects since
each set of measurements is projected onto a distinct latent space
representation by the CNN-based encoder. A crucial aspect of FPM
reconstruction is achieving “super-resolution” reconstruction, surpassing the
diffraction limit of the input measurements. To achieve this goal, our
framework extracts “super-resolved” latent information beyond the “discrete”
pixel grid in the measurement by incorporating the Local Implicit Image
Function (LIIF) method [37] into the decoding process. By combining these
components, our LCNF framework achieves versatile deep-learning-based
continuous-domain super-resolution image reconstruction based on low-
resolution measurements that is applicable to arbitrary objects with varying
spatial scales and resolutions.

In this study, we present the capabilities of our proposed LCNF framework for
large-scale phase reconstruction based on multiplexed FPM measurements. FPM is
a well-established computational imaging technique that combines synthetic
aperture and phase retrieval principles to achieve high-resolution
reconstructions of amplitude and phase images over a wide field-of-view (FOV)
using low-resolution intensity images [38]. Here, we showcase the
effectiveness of our LCNF framework in accurately reconstructing continuous-
domain high-resolution phase images over a large FOV using only five
multiplexed measurements. Notably, our approach eliminates the need for
complex Generative Adversarial Network (GAN) training, as required in previous
state-of-the-art approaches [6, 7].

Our results highlight the ability of LCNF to capture the continuous and smooth
priors of the object, enabling robust reconstruction of high-resolution phase
images. First, using experimental datasets captured on Hela cells fixed in
ethanol or formalin, we show that the LCNF network can accurately reconstruct
complex cellular and subcellular structures. In addition, we highlight the
robustness of the LCNF framework when subjected to imperfect training
datasets, benefiting from the implicit continuous priors embedded in our
framework. Specifically, LCNF effectively eliminates common artifacts
encountered in traditional model-based FPM algorithms, such as residual phase
unwrapping errors, noise, and background artifacts, without the need for
additional parametric or learned constraints.

Furthermore, we showcase the strong generalization capabilities of our LCNF
framework. Firstly, we demonstrate that LCNF can consistently reconstruct
high-resolution phase images even when trained with very limited training data
or under different experimental conditions. Remarkably, we achieve high-
quality reconstructions even when the network is trained on a _single_ imaging
data pair. This superior generalization capability is attributed to our NF-
based training strategy, which utilizes pixels as training pairs and
effectively expands the training data from a single paired image to a diverse
set of pixels. Moreover, we demonstrate that LCNF can be trained using purely
simulated datasets composed of natural images. We show that the simulation-
trained LCNF network generalizes well when applied to experimental biological
measurements, consistently reconstructing detailed subcellular structures with
minimal artifacts. Finally, we establish that all LCNF networks, regardless of
the training strategy, reliably reconstruct high-resolution phase images
across a wide FOV.

In summary, we introduce the LCNF framework as a versatile and generalizable
approach for solving highly ill-posed large-scale imaging inverse problems in
computational imaging. By leveraging a continuous implicit neural
representation, LCNF effectively captures continuous multiscale object
information from low-resolution measurements. It provides robust super-
resolution reconstruction capabilities, bypassing the limitations of
traditional model-based and CNN-based methods that rely on discrete
representations. The framework’s ability to generalize with very limited
training data and its capacity to leverage simulated data further enhance its
potential for advancing deep learning-based computational imaging techniques,
making it highly attractive for challenging application scenarios where
collecting experimental training data is both time-consuming and costly.

##  2 Results

###  2.1 The LCNF framework

Our LCNF framework for phase reconstruction from multiplexed FPM measurements
is illustrated in Fig. 1(a). The encoder $E_{\theta_{e}}$ takes six low-
resolution images as input and projects the learned low-dimensional
information into a latent space. The input images consist of two brightfield
(BF) and three darkfield (DF) intensity measurements captured with the
illumination patterns shown in Fig. 1(b), along with a low-resolution linear
estimate of the object’s phase computed from the two BF measurements using the
differential phase contrast (DPC) method [25]. To handle the distinct
distributions of BF, DF, and DPC images, three separate encoders
$\\{e_{1},e_{2},e_{3}\\}$ are employed to effectively extract the underlying
latent information. Each encoder utilizes convolutional layers and residual
blocks [39] to extract spatial features. The lateral dimensions of the spatial
features match those of the input image, allowing for direct coordinate-
dependent latent information retrieval during decoding. The spatial features
learned from the three encoders are concatenated to form the final latent
space representation $\Phi\in\mathbb{R}^{H\times W\times D}$, where $H$ and
$W$ represent the lateral dimensions and $D$ represents the total number of
concatenated feature maps in the latent space.

To enable high-resolution phase reconstruction independent of a fixed grid, a
five-layer MLP denoted as $f_{\theta}$ is employed as the decoder. For local
conditioning, a specific latent vector $\phi\in\mathbb{R}^{1\times 1\times D}$
is concatenated with the corresponding spatial coordinate $\mathbf{c}$ before
being inputted to $f_{\theta}$. This conditioning mechanism ensures that the
learned mapping by the MLP is dependent on the input measurement, allowing for
generalizability across different objects. The output of the decoder is a
scalar representing the predicted phase value at the location $\mathbf{c}$.
The LCNF network is trained end-to-end in a supervised manner by minimizing
the loss function $L$:

$\underset{\theta_{e},\theta}{\operatorname{min}}L(f_{\theta}(\mathbf{c},\phi),\psi(\mathbf{c})),$
(1)

where $\theta_{e}$ and $\theta$ represent the network parameters of the
encoder and decoder respectively, $\phi=E_{\theta_{e}}(m,\mathbf{c})$ is the
latent vector encoded from the input $m$ for the queried coordinate
$\mathbf{c}$, and $\psi(\mathbf{c})$ is the high-resolution ground-truth phase
value at the position $\mathbf{c}$. The ground-truth phase images are
reconstructed using separate standard FPM measurements and a previously
developed model-based reconstruction algorithm [26, 7].

A key aspect of FPM is the reconstruction of super-resolved images beyond the
low-resolution input. To facilitate the learning of high-resolution
information beyond the low-resolution $H\times W$ grid, the LCNF network is
also trained on “off-the-grid” high-resolution data queried from a denser grid
$H^{\prime}\times W^{\prime}$. However, the corresponding “off-the-grid”
latent vector is not readily available from the encoded latent space. In
practice, the nearest latent vector (based on the Euclidean distance) is used
for the decoder. Additionally, to inform the decoder about the relative
position of the queried “off-the-grid” location with respect to the nearest
latent vector location, the implementation of Eq. (1) utilizes their relative
coordinate $\Delta\mathbf{c}$ instead of the absolute coordinate $\mathbf{c}$,
following the approach introduced in the LIIF method [37]. Furthermore, to
utilize the information provided by the neighboring latent vectors and improve
the continuity of the reconstruction, enhancement techniques including feature
unfolding, local ensemble, and cell decoding [37] are applied.

After training, the LCNF network allows for querying arbitrary points on the
object by providing the corresponding low-dimensional measurements and the
queried coordinates as the network input. This eliminates the requirement for
a fixed input grid in traditional model-based and deep neural network
architectures. The high-resolution phase reconstruction can be visualized on
any desired grid. This feature is demonstrated in the results depicted in Fig.
1(c), where reconstructions are queried at three distinct pixel densities,
showcasing smooth transitions across these diverse spatial scales without any
artifacts.

More details about the FPM setup, measurements, and model-based
reconstructions are provided in Sections 4.1 and 4.2. Additionally, further
details about the LCNF framework, including data acquisition and
preprocessing, network structure, reconstruction enhancement techniques, and
network training and inference are provided in Sections 4.3, and Figs. S1, S2
and S3.

###  2.2 LCNF reconstruction trained with experimental dataset

We first evaluate the performance of our LCNF network using Hela cells fixed
with ethanol or formalin as imaging samples. The network is trained separately
for these two cell types (see Section 4.3.4), and the reconstruction results
using the network trained on the same cell type are shown in Figs. 1(c) and 2.

In Fig. 1(c), we present the raw low-resolution BF intensity image, the model-
based FPM reconstruction, and our LCNF-based reconstruction of ethanol-fixed
Hela cells. Furthermore, we display two small subareas (area 1 and area 2).
From the figure, it is evident that our network successfully reconstructs
high-resolution phase images from low-resolution intensity images, accurately
recovering intricate subcellular structures.

Figure 2: Reconstruction results using the LCNF trained with our experimental
dataset. An example of BF low-resolution intensity image, the model-based FPM
reconstruction, and our LCNF network-based reconstruction for (a) Hela cells
fixed with ethanol, and (b) Hela cells fixed with formalin. Subareas (1)-(6)
highlight specific regions of interest, demonstrating our network’s capacity
to accurately reconstruct subcellular high-resolution features without any
artifacts.

To evaluate the continuous object representation capability of our LCNF
network, we conduct queries on arbitrary coordinates within subarea 1, as
shown at the bottom of Fig. 1(c). We perform queries at densities of
$6\times,21\times,49.8\times$ compared to the input low-resolution intensity
image. Our network successfully reconstructs the phase at these density grids.
For comparison, we also include the model-based FPM reconstruction of the same
area. Due to predefined grids, the FPM reconstruction exhibits discrete grid
artifacts in the enlarged image. Furthermore, it may suffer from phase
unwrapping artifacts (see Section 2.3 for more details). For an additional
comparison, we present high-resolution phase reconstructions with a grid
density of $105.9\times$ in Fig. S4. In contrast, our network provides a
continuous object reconstruction without any discrete or other phase
artifacts.

Figure 2(a) and (b) showcase additional reconstruction results for Hela cells
fixed with ethanol and formalin, respectively. As shown in the figures, we
successfully reconstruct high-resolution phase images from the low-resolution
intensity images, accurately capturing detailed cellular and subcellular
structures without any artifacts, as highlighted in the zoom-in regions
(1)-(6).

###  2.3 Robustness to phase artifacts

Next, we highlight the robustness of our LCNF network to various phase
artifacts that arise from practical FPM experiments, including noise, phase
unwrapping errors, and artifacts resulting from an imperfect imaging model.

Figure 3: LCNF’s robustness to phase artifacts. The LCNF reconstruction
completely eliminates (a) the discontinuous phase-unwrapping errors (indicated
by red arrows) and background ripping artifacts, as well as (b) random phase
noise.

As shown in Fig. 1(c) and Fig. 3(a), the model-based FPM reconstruction may
exhibit discontinuous artifacts due to imperfect phase unwrapping when dealing
with samples that have a phase range exceeding $2\pi$. Moreover, Fig. 3(a)
illustrates the presence of rippling artifacts in the background region of the
model-based FPM reconstruction, possibly resulting from the imperfect FPM
imaging model used for the model-based reconstruction [40]. Additionally, Fig.
3(b) demonstrates that the model-based FPM reconstruction can be susceptible
to random phase noise.

In contrast, our LCNF network effectively eliminates these artifacts and
achieves accurate, smooth, and continuous reconstructions, _even though it has
been trained using imperfect ground-truth images from experiments that
inevitably contain these artifacts._ We quantitatively evaluate the artifact-
suppression capability of our LCNF network using the method in [41]. Our
analysis shows that our LCNF network can reduce the background artifacts by
several folds compared with the model-based FPM reconstruction, as illustrated
in Fig. S5.

The robustness can be attributed to the implicit continuous priors embedded in
our LCNF network structure. The LCNF framework employs a two-step process to
achieve continuous representations. Firstly, it encodes the input images into
a continuous latent space representation, effectively filtering out noisy
information. Secondly, it decodes the queried point by conditioning it on the
selected latent vector. This process leverages the continuity priors embedded
in the MLP decoder, enabling it to learn a continuous neural representation of
the object.

Overall, our network demonstrates robust reconstruction capabilities even when
trained with imperfect datasets, benefiting from the continuity of the learned
latent space and the continuous representation imposed by the MLP decoder.

###  2.4 Generalizability of experimental data trained LCNF network

Figure 4: Generalization ability of experimental data trained LCNF network.
Reconstructions of (a) Hela cells fixed in ethanol and (b) Hela cells fixed in
formalin. The MSE of each reconstruction is noted at the bottom of the
respective image. Networks used for reconstruction:
Network${}_{\text{Hela-E(18)}}$ (trained with 18 pairs of ethanol-fixed Hela
cells), Network${}_{\text{Hela-F(16)}}$ (trained with 16 pairs of formalin-
fixed Hela cells), Network${}_{\text{Hela-E(1)}}$ (trained with a single pair
of ethanol-fixed Hela cells), and Network${}_{\text{Hela-F(1)}}$ (trained with
a single pair of formalin-fixed Hela cells).

One notable advantage of our LCNF framework is its superior generalization
capability, overcoming the limitations of traditional NF frameworks [12]. To
thoroughly evaluate its generalization performance, we conduct training using
three distinct types of experimentally collected datasets, as outlined in
Section 4.3.4. These training scenarios included: (1) utilizing one type of
experimental data to train the network and evaluating it on the same type of
dataset, (2) training the network with a single pair of data and testing it on
the same type of data, and (3) evaluating the aforementioned networks’
performance on other types of cells. The LCNF networks consistently
demonstrate successful reconstruction of high-resolution phase images across
all three training scenarios, as depicted in Fig. 4.

We quantitatively assess the performance of our LCNF-based reconstructions for
both image patches from the FOV regions matching the training conditions and
outside the training region. This assessment is informative because spatially
varying aberrations are known to degrade FPM reconstructions [42]. By
evaluating the reconstruction quality outside the training FOV, we gain
insights into the network’s robustness against realistic spatially varying
aberrations in our experiment. For the evaluation, we employ the mean square
error (MSE) as the objective metric. The results, presented in Fig. 4,
illustrate the robust performance of our LCNF network in all three scenarios,
with the corresponding MSE values provided at the bottom of the figure. We
also compute the peak signal-to-noise ratio (PSNR), structural similarity
index measure (SSIM), and frequency measurement metric (FM). The FM quantifies
the recovery of frequency components [43], where higher FM values represent
the recovery of more frequency components. The quantitative metrics for the
results in Fig. 4 are presented in Table S1, and the metrics for an additional
100 image patches outside the training FOV region are provided in Table S3.

As shown in Fig. 4, and Tables S1 and S3, the MSE generally increases, while
the PSNR, SSIM, and FM decrease when training the LCNF network with a very
limited dataset or a different type of data compared to the network trained
with the same cell type and the full experimental dataset. This indicates that
the network’s generalization performance generally degrades when it is trained
on a smaller training dataset or the distribution of the testing data is
shifted from that of the training data, which is expected. However, the
changes in the metric values are small and hardly noticeable in the
visualizations in Fig. 4, even for the network trained with _a single_ paired
training dataset. This highlights the robust generalization performance of our
LCNF network.

In addition, when the network is trained with ethanol-fixed Hela cells
(Network${}_{\text{Hela-E(18)}}$) and applied to formalin-fixed Hela cells,
the SSIM and FM are slightly higher than those of the network train with
formalin-fixed Hela cells (Network${}_{\text{Hela-F(16)}}$), as shown in Table
S3. We attribute this “unusual” result to the fact that ethanol-fixed Hela
cells contain more structural details and provide a broader spectrum compare
to formalin-fixed Hela cells (see Fig. 5(b)). Therefore, the network trained
with ethanol-fixed Hela cells may reconstruct more frequency components and
thus yield better results.

We attribute this generalization capability to our novel training strategy,
which utilizes _pixels_ as the training pairs (as shown in Eq. (1)). By
adopting this approach, we effectively expand the training data from a single
paired image ($250\times 250$ pixels for the input and $1500\times 1500$
pixels for the high-resolution reconstruction) to a diverse set of pixels.
This enables the network to learn from a larger and more varied dataset,
contributing to its remarkable generalization capabilities. As a result, our
LCNF network demonstrates the ability to reconstruct high-resolution phase
images even when trained with very limited training data. This not only
reduces the necessity for a large number of training samples but also
expedites the overall experimental process, making it highly suitable for
challenging experimental scenarios where collecting experimental training data
is both time-consuming and costly.

###  2.5 LCNF network generalizes from simulation to experiment

Figure 5: Generalization ability of simulation-trained LCNF network. (a)
Example simulation on a natural image. I: Simulated multiplexed BF intensity
image. II: High-resolution ground-truth image. III: Reconstruction by LCNF.
(i-iii) The spectrum of (i) simulated BF intensity image, (ii) ground truth,
and (iii) LCNF reconstruction. Blue, red, and brown circles indicate the
bandwidth of the objective lens (0.1 NA), the expected bandwidth of the BF
measurement (0.2 NA), and the theoretically achievable reconstruction
bandwidth (0.51 NA), respectively. (b) Reconstruction of experimental data
from Hela cells fixed in ethanol and formalin using (I, IV) model-based FPM
algorithm, (II, V) LCNF network trained with the full experimental dataset
with matching cell type (II: Network${}_{\text{Hela-E(18)}}$, V:
Network${}_{\text{Hela-F(16)}}$), and (III, VI) LCNF network trained with
simulated dataset (Network${}_{\text{Simulate}}$). (i-vi) Spectrum of
reconstructions I-VI labeled by i-vi in the right column, with different
circles representing different NAs as in (a).

We further demonstrate the robust generalization capability of our LCNF
network by employing a simulator-trained network for the reconstruction of the
experimental Hela cells dataset, as depicted in Fig. 5. When solving the
inverse problem using deep learning-based methods, acquiring paired datasets
for network training can be challenging. Traditional NF-based approaches
incorporate the imaging forward model within the network, allowing for self-
supervised training without the need for paired datasets. However, as
mentioned earlier, these methods often lack the ability to generalize across
different objects and necessitate separate training for each new object.

An alternative approach is to use the imaging forward model to generate
simulated paired datasets for network training [44, 10, 24, 19, 20]. However,
in the context of FPM reconstruction, the application of simulator-trained
networks has been obstructed by the use of the GAN structure that learns
highly specific but less generalizable object priors [7]. Here, we demonstrate
straightforward deployment of simulation-based training of our LCNF network
and achieve high-resolution, wide-FOV phase reconstructions on the
experimental biological dataset (see training details in Section 4.3.1).

We first evaluate the network’s performance on simulated data, as illustrated
in Fig. 5(a). The results confirm the successful reconstruction of high-
resolution phase images from low-resolution intensity images. The MSE, PSNR,
SSIM, and FM metrics are presented in Table S2. When evaluating the network
using the simulated dataset, we did not apply any preprocessing techniques
described in Section 4.3.1 to the ground-truth high-resolution natural images,
except for linearly matching pixel values with the phase range of [0, 9].
Consequently, we can assess the network’s performance more accurately without
the influence of data preprocessing effects. Notably, our network demonstrates
high effectiveness in recovering a significant portion of the spatial
frequency components, as evidenced in the visualization of the reconstructed
spectrum in the right column of Fig. 5(a) and Table S2.

Subsequently, we employ our simulator-trained network for the reconstruction
of experimentally captured Hela cells datasets, as depicted in Fig. 5(b). The
network successfully reconstructs Hela cells with detailed subcellular
structures and recovers the rich spatial frequency components.

The quantitative metrics, including MSE, PSNR, SSIM and FM, are provided in
Table S2. The results show that the simulator-trained network performs
slightly worse than the experimental data-trained network in terms of MSE,
PSNR, and SSIM. This is expected because our training data consists only of
natural images, which have significantly different image features compared to
the cells in the experiment. However, the FM metrics of our simulation-trained
network are consistently higher than all other methods. This observation
suggests that training the network on PSD-corrected natural images may promote
more effective learning for high-frequency content. To further evaluate the
performance of the simulation-trained network, we conduct additional
reconstructions on 100 experimental image patches beyond the training FOV. The
quantitative metrics are presented in Table S3.

These results clearly demonstrate the generalization capability of our
simulation-trained LCNF network in achieving high-resolution phase
reconstructions in the experiment.

###  2.6 Robust wide-FOV high-resolution phase reconstruction

Figure 6: Wide-FOV high-resolution phase reconstruction by the LCNF network.
(a) BF intensity image captured over a 2160-pixel diameter (3.51 mm) FOV.
Wide-FOV reconstruction by training the LCNF network with (b) an experimental
dataset and (c) a simulated dataset. (d-g) Selected subareas extracted from
the central to the edge of the FOV, identified as i-iv and enclosed within
different colored boxes. (d) BF intensity image. (e) model-based FPM
reconstruction. (f) LCNF reconstruction trained with the experimental dataset.
The experimental dataset used for training the LCNF network is obtained from
the central region, indicated by the dashed black square. (g) LCNF
reconstruction trained with the simulated dataset.

Finally, we employ our LCNF network for wide-FOV high-resolution phase
reconstructions, as shown in Fig. 6. The network is trained solely using the
central $250\times 250$-pixel region, indicated by the dashed black square in
Fig. 6(a). Subsequently, we perform phase image reconstruction across a much
larger FOV that encompasses a circular region with a 2160-pixel diameter in
the raw measurements (3.51 mm). The resulting wide-FOV reconstructions
(12960-pixel in diameter and 6$\times$ denser pixel grid compared to the
input), obtained using Network${}_{\text{Hela-E(18)}}$ and
Network${}_{\text{Simulate}}$, are presented in Fig. 6(b) and (c),
respectively. Additionally, Fig. 6(d)-(g) displays selected subareas extracted
from the central to the edge of the full-FOV image. The corresponding images
include the BF intensity image, model-based FPM reconstruction, LCNF
Network${}_{\text{Hela-E(18)}}$ reconstruction trained with the experimental
dataset (LCNF(exp)), and LCNF Network${}_{\text{Simulate}}$ reconstruction
trained with the simulated dataset (LCNF(sim)). Overall, the LCNF networks
achieve high-quality reconstructions, with subcellular features clearly
recovered and minimal artifacts. However, at the very edge of the image, as
observed in Fig. 6(d)-(g)iv, some distortions are present. This behavior is
expected due to the spatially varying aberrations present in our microscope
setup, which become more pronounced at the edge of the FOV [42]. The
experimental dataset used for training the network is obtained from the
central FOV, where aberrations are minimal. The simulation assumes a perfect
imaging system without any aberrations. Consequently, the network was not
exposed to these aberrations during training. Addressing this limitation will
require incorporating a spatially variant imaging model, which we plan to
consider in our future work.

Furthermore, we present additional wide-FOV reconstructions in Figs. S6, S7,
and S8 for Hela cells fixed in ethanol or formalin. These reconstructions were
obtained using the LCNF networks trained with all the strategies detailed in
Section 2.4 and 2.5, based on experimental or simulated datasets. The results
further underscore the reliability of our LCNF framework in achieving wide-FOV
high-resolution phase reconstructions, regardless of the training strategy
employed. Notably, our framework demonstrates excellent performance even when
trained with very limited data, including a single paired image at the extreme
case, or when utilizing simulated training data.

##  3 Discussion and Conclusion

In this study, we have introduced LCNF, a versatile and generalizable deep
learning framework for solving large-scale imaging inverse problems. Unlike
traditional CNN frameworks, LCNF leverages a continuous implicit neural
representation to enable flexible reconstruction of multiscale information. It
introduces a novel local conditioning approach, enhancing its generalization
capability compared to existing NF frameworks.

By applying LCNF to solve the multiplexed FPM reconstruction problem, we
demonstrate its effectiveness in achieving continuous-domain super-resolution
reconstruction from low-resolution measurements, applicable to objects of
varying spatial scales and resolutions. In addition, LCNF exhibits robustness
against noisy training data. The LCNF reconstructions are free from common
artifacts, such as residual phase unwrapping errors, noise and background
ripples, that contaminate the training data obtained by traditional FPM
reconstructions.

Furthermore, LCNF demonstrates remarkable generalization capabilities across
different object types and experimental conditions. Notably, LCNF can be
effectively trained even with limited datasets, including a single paired
image dataset, considerably simplifying the experimental training data
collection process. Additionally, we show that LCNF can be entirely on
simulated data and generalize well to experimental data without requiring
network retraining or transfer learning. This further highlights the
robustness and adaptability of our LCNF approach.

LCNF’s efficient processing of multiscale information makes it highly suitable
for large-FOV high-resolution image reconstruction applications. We showcase
LCNF’s ability to robustly perform large-scale super-resolution phase
reconstructions using multiplexed FPM measurements, regardless of the training
strategy employed.

In summary, we present LCNF as a robust and scalable deep-learning-based
continuous-domain reconstruction framework. Its ability to handle large FOV
and high-resolution imaging reconstruction, combined with its strong
generalization capabilities, makes it suitable for a wide range of
computational imaging techniques.

##  4 Methods

###  4.1 FPM experimental setup

Our LCNF network was developed based on the experimental data obtained from
our previous study in [7]. To briefly describe the FPM setup and the data
acquisition method, the illumination multiplexing scheme combined patterns
used in DPC [25] and randomly multiplexed FPM [26, 45] to efficiently encode
high-resolution phase information across a wide FOV. Specifically, we used
five LED illumination patterns (central wavelength $630\mathrm{nm}$),
including two BF semi-circle patterns and three $120^{\circ}$-arc patterns, as
illustrated in Fig. 1(b). To capture the standard FPM dataset, sequential
illumination with 185 LEDs was employed. In both illumination schemes, the
maximum illumination NA was 0.41. The samples used for testing were unstained
Hela cells fixed with ethanol or formalin. Intensity images were collected
using a $4\times$, 0.1 NA objective lens (Nikon CFI Plan Achromat) and an
sCMOS camera (PCO: pco.edge 5.5) with $2560\times 2160$ pixels and a pixel
size of $6.5\upmu\mathrm{m}$.

###  4.2 Model-based reconstructions

####  4.2.1 DPC-based phase imaging

Here, we briefly explain the principle of DPC phase imaging; additional
details can be found in [25]. DPC is a technique used to recover quantitative
phase information from intensity images acquired with asymmetric illumination
patterns. It offers an improved lateral resolution of $2\times$ compared to
the native objective NA.

Under the weak object assumption:
$o(\mathbf{c})=e^{-\mu(\mathbf{c})+i\psi(\mathbf{c})}\approx
1-\mu(\mathbf{c})+i\psi(\mathbf{c})$, where $\mu(\mathbf{c})$ represents
absorption and $\psi(\mathbf{c})$ represents phase, a BF intensity measurement
$I_{s}(\mathbf{c})$ can be approximated to have a linear relationship with the
sample [25]:

$I_{S}(\mathbf{u})=B\delta(\mathbf{u})+H_{abs}(\mathbf{u})M(\mathbf{u})+H_{ph}\Psi(\mathbf{u}),$
(2)

where $I_{S}(\mathbf{u}),M(\mathbf{u}),\Psi(\mathbf{u})$ denote the spectrum
of $I_{s}(\mathbf{c}),\mu(\mathbf{c})$ and $\psi(\mathbf{c})$, respectively,
and $\mathbf{u}=(u_{x},u_{y})$ represents the spatial frequency. $B$ is a
constant representing the background signal, and $\delta(\mathbf{u})$ is the
Dirac delta function. $H_{abs},H_{ph}$ are the transfer functions for
amplitude and phase, respectively [25].

By subtracting the background term and normalizing the acquired BF intensity
image, the DPC reconstruction can be formulated as:

$\underset{M,\Psi}{\operatorname{min}}\sum_{j=1}^{N_{\mathrm{bf}}}\|I_{S-,j}(\mathbf{u})-H_{abs,j}(\mathbf{u})M(\mathbf{u})-H_{ph,j}\Psi(\mathbf{u})\|_{2}^{2}+\tau_{1}R_{1}(M(\mathbf{u}))+\tau_{2}R_{2}(\Psi(\mathbf{u})),$
(3)

where $I_{S-}(\mathbf{u})$ represents the spectrum of the background-
subtracted intensity image, $j$ is the index of DPC measurements,
$N_{\mathrm{bf}}=2$ denotes the number of captured BF images, and
$\|\cdot\|_{2}$ represents the $L_{2}$ norm. $\tau_{1},\tau_{2}$ are the
regularization parameters, and $R_{1}$ and $R_{2}$ represent the
regularization terms that incorporate prior information about the sample.
Here, we utilized the $L_{2}$ regularization to solve inverse problem [25].

It should be noted that DPC reconstruction relies on the weak object
approximation, which means it only provides accurate results when the phase
change of the sample is below 0.64 radians [46]. However, in our experiment,
the Hela cell samples are fixed in ethanol or formalin, causing phase changes
exceeding $2\pi$. This violates the weak object approximation and leads to an
underestimation of the object’s phase. Despite its limitations, the DPC
estimation serves as a useful low-resolution initial guess for the object’s
phase [45, 41]. Therefore, we input this estimation into our network.

####  4.2.2 FPM forward model

The forward model of FPM describes the intensity image obtained from a single
LED illumination. After appropriate normalization to account for the system
magnification, it can be expressed as:

$I_{i}(\mathrm{\mathbf{c}})=|\mathcal{F}^{-1}_{[O(\mathrm{\mathbf{u}}-\mathrm{\mathbf{u}}_{i})P(\mathbf{u})]}(\mathbf{c})|^{2},$
(4)

where $I_{i}(\mathbf{c})$ represents the captured low-resolution intensity
image for the $i^{\mathrm{th}}$ LED. $|\cdot|$ takes the amplitude of the
complex field, and $\mathbf{c}=(x,y)$ denotes the lateral coordinates.
$\mathcal{F}^{-1}$ represents the inverse Fourier transform, and
$O(\mathbf{u})$ is the spectrum of the object $o(\mathbf{c})$. Each LED
illumination is modeled as a plane wave with spatial frequency
$\mathbf{u}_{i}=(u_{xi},u_{yi})=({\sin\theta_{xi}}/{\lambda},{\sin\theta_{yi}}/{\lambda})$,
where $(\theta_{xi},\theta_{yi})$ defines the illumination angle of the
$i^{\mathrm{th}}$ LED and $\lambda$ denotes the central wavelength. The pupil
function of the microscope, denoted by $P(\mathbf{u})$, is a circular low-pass
filter with a diameter of $2\mathrm{NA}/\lambda$, set by the objective lens
NA.

In the case of multiplexed illumination, the sample is illuminated by
different sets of LEDs based on different illumination patterns, as depicted
in Fig. 1(b). The captured intensity image can be modeled as the sum of
multiple intensity images obtained from individual LEDs [26]:

$I_{S}(\mathrm{\mathbf{c}})=\sum_{i\in S}I_{i}(\mathbf{c}),$ (5)

where the symbol $\in$ indicates that $i$ is an element of the illumination
set $S$.

####  4.2.3 Model-based FPM reconstruction

FPM is a recently developed computational imaging technique that enables
increasing the imaging system’s space-bandwidth product (SBP) by synthesizing
multiple low-resolution images into a high-resolution image across a wide FOV
[38]. The FPM reconstruction involves solving a non-convex optimization
problem that jointly estimates the object $O(\mathbf{u})$ and the pupil
function $P(\mathbf{u})$ by solving a minimization problem:

$\underset{O(\mathbf{k}),P(\mathbf{k}),\\{b_{i}\\}}{\operatorname{min}}\sum_{i=1}^{N_{\mathrm{led}}}\big{\|}\sqrt{I_{i}(\mathbf{c})}-|\mathcal{F}^{-1}_{[O(\mathrm{\mathbf{u}}-\mathrm{\mathbf{u}_{i}})P(\mathbf{u})]}(\mathbf{c})|-b_{i}\big{\|}^{2}_{2},$
(6)

where $b_{i}$ is the background offset for the $i^{\mathrm{th}}$ image, and
$N_{\mathrm{led}}$ is the total number of LEDs used in the sequential FPM
measurement. The reconstruction is performed by an iterative algorithm by
following [26].

###  4.3 The LCNF framework

####  4.3.1 Data acquisition and preparation

In our study, we investigated different strategies for training our LCNF
network using both experimental and simulated datasets.

The experimental data was obtained from [7] and was taken on Hela cells fixed
in ethanol or formalin. We collected 22 groups of low-resolution measurements
($2560\times 2160$ pixels) on ethanol-fixed Hela cells and 20 groups of
measurements on formalin-fixed Hela cells using multiplexed illumination. The
LED patterns used for illumination are described in Section 4.1, which
includes two BF and three DF patterns.

To prepare the input for training the LCNF network, we performed the following
steps. Firstly, we extracted the central $250\times 250$ pixels from the raw
low-resolution intensity images. Then, we applied dynamic range correction by
clipping the minimum $0.1\%$ and maximum $0.1\%$ pixel values for each
measurement, following the approach described in [7]. This correction helped
suppress shot noise and hot pixels. Next, we used the DPC reconstruction
algorithm, as explained in Section 4.2.1), to generate a linear estimation of
the phase based on the two BF intensity measurements. Additionally, we
normalized the LED intensities by dividing the intensity images by their mean
value. We also applied a morphological open operator to estimate and subtract
the slow-changing background, following the method described in [7]. This
process effectively eliminated the unwanted background components and improved
the accuracy of the subsequent learning process. Finally, we concatenated the
preprocessed low-resolution intensity images with the DPC image.

To obtain ground-truth high-resolution phase images for the experimental data,
we applied the following procedure. Firstly, for each standard FPM
measurement, we sequentially illuminated 185 LEDs and captured the
corresponding low-resolution intensity images. Then, we employed the model-
based FPM reconstruction algorithm, detailed in Section 4.2.3, to reconstruct
the phase of the central $250\times 250$-pixel region and produce a high-
resolution phase image of $1500\times 1500$ pixels. Next, we applied a phase
unwrapping algorithm [47] to unwrap the reconstructed high-resolution phase
image. Furthermore, we addressed the slow-varying background component present
in the reconstructed high-resolution phase image by utilizing a morphological
open operator with a kernel size of 50. This step removed the slowly changing
background, enhancing the clarity and quality of the phase image. To normalize
the range of values in the high-resolution phase image, we clipped the phase
range within [0, 12] for the Hela cells fixed in both ethanol and formalin.
Subsequently, we divided the phase images by this clipping threshold,
resulting in a normalized range of values within [0, 1]. Finally, we paired
the preprocessed low-resolution input images with the normalized high-
resolution reconstructions, which served as the training data for our neural
network.

For the simulated dataset, we utilized a portion of the high-resolution DIV2K
dataset [48] from the NTRE 2017 challenge [49] as our ground-truth phase
images. The dataset consisted of 900 cropped natural images, each with a size
of $600\times 600$ pixels. Since the natural images have different histogram
and spectral distributions compared to the biological cell images (see Fig.
S3), we performed a preprocessing procedure on these images. The preprocessing
involved removing the slowly varying background using an open operator with a
kernel size of 20. Then, we applied a maximum value threshold of 0.6 to crop
the image values and normalized the cropped images to the range $[0,1]$ by
dividing them by this threshold. To ensure consistency between the simulated
dataset and the experimental Hela cells fixed in ethanol (here, we only
utilize the data for the Hela cells fixed in ethanol since it contains more
frequency content), we took steps to match the power spectrum density (PSD) of
the simulated dataset with the experimental data. This involved multiplying
the spectrum of each simulated data with a correction map, whose value at a
specific frequency is determined by the ratio between the square root of the
PSDs of the experimental and the simulated dataset. We then normalized the
spectrum-corrected image by dividing it by its maximum value. The resulting
normalized and spectrum-corrected high-resolution images were used as ground
truth for our network. The effect of the preprocessing procedure for the
natural images can be observed in Fig. S3.

To generate the un-normalized object phase, we multiplied the normalized and
spectrum-corrected high-resolution images by a factor of 9 and then subtracted
2.5, resulting in a phase range of $[-2.5,6.5]$. This range corresponds
closely to the predominant distribution observed in the histogram of the
experimental ethanol-fixed Hela cell dataset and also balances the effect from
the large phase values observed in the experimental dataset (for additional
information, refer to Figure S3). To simulate the low-resolution intensity
images, we used Eq. (5) as the forward model and downsampled the simulated
intensity images to $100\times 100$ pixels. Throughout the simulation process,
we make the assumption that our simulated system does not exhibit aberration.
As a result, the pupil function $P(\mathbf{u})$ in Eq. (5) is considered an
ideal circular low-pass filter, with a value of 1 within the circular region
and 0 outside of it. We applied the same preprocessing steps used for the
experimental dataset to obtain the preprocessed simulated low-resolution
intensity images, which served as the input for the network. Finally, we
paired the preprocessed high-resolution images with their corresponding low-
resolution intensity images to create the training pairs for the network
training.

####  4.3.2 LCNF network structure

Our LCNF network consists of a CNN-based encoder and an MLP-based decoder. A
detailed visual illustration of the network can be found in Fig. S1.

Encoder: We use three CNN-based encoders, denoted as $\\{e_{1},e_{2},e_{3}\\}$
to independently encode three different types of input: BF, DF, and DPC
images. Each encoder follows a deep residual network structure similar to
[50]. The encoders take specific image types as input and initially extract
spatial features using a convolutional layer. The number of input channels for
the first convolutional layer varies according to the number of input images:
2 for two BF images, 3 for three DF images, and 1 for the DPC image. The
output channels for the first convolutional layer are fixed at 128 for all
encoders.

After the initial convolutional layer, we employ 32 residual blocks to further
extract spatial feature maps. Each residual block consists of two
convolutional layers 128 input and output channels, a ReLU activation layer,
and a multiplication layer with a factor of 1. Skip connections are
incorporated in the residual blocks, where feature maps are added together.
The spatial features extracted by the residual blocks are then passed through
an output convolutional layer with 128 input and output channels. Finally,
these features are added with the features maps provided by the initial
convolutional layer with a long skip connection. All convolutional layers use
$3\times 3$ convolutional kernels.

Once the feature maps are extracted from the input, they are concatenated to
form the encoded latent space representation $\Phi\in\mathbb{R}^{H\times
W\times D}$ of the image, where $H$ and $W$ represent the pixel resolution
along the $x$ and $y$ axes, respectively, while D represents the number of
concatenated channels. The $H$ and $W$ dimensions remain the same as the input
low-resolution measurements, as we do not include pooling or upsampling layers
in our encoder networks. The network structure of the encoders is visually
depicted in Fig. S1: Encoder.

Decoder: To represent a high-resolution image in a continuous representation,
we employ the LIIF approach [37], which represents an object in the encoded
latent space and utilizes an MLP as a decoding function to decode the object
from the latent space back to the object domain. In our case, we use a
standard 5-layer MLP as the decoder, denoted as $f_{\theta}$. Each layer of
the MLP has 256 neurons, and ReLU activation is applied to the first four
layers, while the last layer is unactivated. The input dimension of the MLP is
3460, which is obtained by $3$ (number of encoders) $\times$ 128 (feature maps
learned by each encoder) $\times$ 9 (feature unfolding) + 2 (dimension of the
coordinates) + 2 (cell decoding), where feature unfolding and cell decoding
are reconstruction enhancement techniques explained in Section 4.3.3. The
output dimension of the MLP is 1, representing the predicted phase value at
the queried location. The structure of the decoder is illustrated in Fig. S1:
Decoder.

The decoding function can be expressed as:

$\hat{\psi}(\mathbf{c})=f_{\theta}(\mathbf{c},\phi),$ (7)

where $\hat{\psi}(\mathbf{c})$ is the decoded physical quantity, such as the
phase value in our case, at the queried position $\mathbf{c}$. The variable
$\mathbf{c}$ represents the 2D coordinates in the continuous image domain,
assumed to range in $[-h,h]$ and $[-w,w]$ for the height and the width,
respectively. $\phi\in\mathbb{R}^{1\times 1\times D}$ is the selected latent
vector from the latent space representation $\Phi$, which is related to the
queried position. The decoding function $f_{\theta}(\mathbf{c},\phi)$ can be
seen as a mapping function
$f_{\theta}(\cdot,\phi):\mathbf{c}\rightarrow\psi(\mathbf{c})$ that maps a
coordinate to the phase value at the position $\mathbf{c}$, with the latent
vector $\phi$ as conditional parameters.

It should be noted that the latent space is a low-dimensional space with a
dimension $H\times W\times D$, where we assign 2D coordinates to each latent
vector with the pre-defined sparse grids, as depicted by the gray circles in
Fig. S1 Encoder. However, for a continuous representation, we may need to
query arbitrary coordinates that are not on the predefined grids, as shown by
the green circle in Fig. S1 Encoder. Consequently, we cannot obtain the exact
latent vector for the queried position $\mathbf{c}$ since the density of the
grid in the latent space is much lower than that of the high-resolution grid
($H^{\prime}\times W^{\prime}$) for the same FOV. To bypass this issue, we
adopt the LIIF approach [37], which assumes that the latent space is
continuous; in addition, each latent vector can represent a local part of the
continuous image and is responsible for predicting the signals at the set of
coordinates that are closest to itself. Accordingly, we reformulate Eq. (7)
as:

$\hat{\psi}(\mathbf{c})=f_{\theta}(\Delta\mathbf{c},\phi),$ (8)

where $\phi$ is the selected latent vector for coordinate $\mathbf{c}$,
determined by the nearest latent vector based on the Euclidean distance. Here,
$\Delta\mathbf{c}=\mathbf{c}-v$ and $v$ represent the actual coordinate of the
selected latent vector $\phi$, respectively. Taking Fig. S1 Encoder as an
example, the bottom-left gray circle represents the selected latent vector,
and $v$ denotes the coordinate of this chosen latent vector.

In summary, our network utilizes CNN-based encoders to encode the measurements
into a low-dimensional latent space representation, where coordinates are
assigned to latent vectors using predefined sparse grids. We can then query
the phase value at arbitrary coordinates and use the MLP decoder to decode the
physical quantity based on the selected latent vector. The latent space
representation, generated by the encoders, adapts to different objects,
allowing our decoding function $f_{\theta}(\cdot,\phi)$ to demonstrate robust
generalization capabilities compared to traditional NF methods.

####  4.3.3 Reconstruction enhancement techniques

To enhance the information extraction from the latent space and improve the
continuity of the reconstruction, we utilize feature unfolding, local
ensemble, and cell decoding techniques as described in the LIIF method [37].

Feature unfolding: To capture additional information beyond a single latent
vector $\phi$, we employ feature unfolding, which extends $\phi$ to
$\hat{\phi}$. Specifically, $\hat{\phi}$ is obtained by concatenating the
$3\times 3$ neighboring latent vectors of $\phi$, as illustrated in Fig.
S2(a), and is defined as:

$\hat{\phi}_{p,q}=\mathrm{Concat}(\\{\phi_{p+l,q+n}\\})_{l,n\in\\{-1,0,1\\}},$
(9)

where Concat represents the concatenation of a set of latent vectors. The
indices $p$ and $q$ correspond to the selected latent code $\phi$ that matches
the queried coordinate $\mathbf{c}$ in the latent space. If the queried
position is at the image’s edge, the latent space $\Phi$ is padded with zero-
vectors.

Cell decoding: We incorporate cell decoding, which takes into account the
pixel size information in the decoding function $f_{\theta}$, as illustrated
in Fig. S2(b). The updated decoding function is expressed as:

$\hat{\psi}(\mathbf{c})=f_{\theta,\mathrm{cell}}([\Delta\mathbf{c},c_{h},c_{w}],\hat{\phi}),$
(10)

where the $[c_{h},c_{w}]$ specifies the height and width of the query pixel
with the desired pixel size in the reconstruction. The notation
$[\Delta\mathbf{c},c_{h},c_{w}]$ denotes the concatenation of the coordinate
and the pixel size. Thus,
$f_{\theta,\mathrm{cell}}([\Delta\mathbf{c},c_{h},c_{w}],\hat{\phi})$
signifies that the decoding function reconstructs the value with the relative
coordinate $\Delta\mathbf{c}$ and the pixel size $(c_{h},c_{w})$, conditioned
on the “unfolded” latent vector $\hat{\phi}$ at the coordinate $\mathbf{c}$.

Local ensemble A concern with Eq. (10) is the discontinuous prediction when
the queried coordinate crosses the middle area between two neighboring latent
vectors, resulting in a switch between latent codes (i.e. the selection of the
nearest latent vector changes). For example, it occurs when the queried
coordinate $\mathbf{c}$ (green dot) crosses the dashed line depicted in Fig.
S2(c). Around such coordinates, predictions for two infinitesimally close
coordinates are generated based on different latent vectors. Due to
imperfections in the learned encoder $E_{\theta_{e}}$ and decoding function
$f_{\theta}$, these borders may exhibit discontinuous patterns. To address
this issue, we employ the local ensemble technique, extending Eq. (10) to:

$\hat{\psi}(\mathbf{c})=\sum_{t\in\\{00,01,10,11\\}}\frac{S_{t}}{S}\cdot
f_{\theta,\mathrm{cell}}([\Delta\mathbf{c}_{t},c_{h},c_{w}],\hat{\phi}_{t}),$
(11)

where $\hat{\phi}_{t}~{}(t\in\\{00,01,10,11\\})$ represents the four nearest
latent vectors (top-left, top-right, bottom-left, bottom-right) based on the
queried coordinate, $\Delta\mathbf{c}_{t}$ denotes the relative coordinate
between the queried coordinate and the selected latent vector, and $S_{t}$
indicates the area of the rectangle between the queried coordinate and the
coordinate of latent vector diagonal to the selected latent vector, as shown
in Fig. S2(c). The weights $S_{t}$ are normalized by $S=\sum_{t}S_{t}$.
Moreover, the latent space representation $\Phi$ is mirror-padded outside the
edge, allowing the above formula to work for coordinates near the image
borders.

####  4.3.4 Network training

Implementation details: To train our network, we follow the procedure outlined
in Section 4.3.1 to prepare the training data, which consists of paired input
images and the corresponding ground-truth phase images. During each training
step, we randomly crop a smaller patch of size $48\times 48$ pixels from the
input images. Recall that the size of the raw input measurements differs for
the experimental and simulated datasets, with dimensions of $250\times 250$
pixels and $100\times 100$ pixels respectively.

We encode the input using three encoders, as described in detail in Section
4.3.2, resulting in a latent space representation $\Phi$ with dimensions of
$H=48,W=48,D=384$.

Subsequently, we assign 2D coordinates to each latent vector
$\phi\in\mathbb{R}^{1\times 1\times 384}$, which is defined on a sparse grid
with the same grid density of $48\times 48$ as the input. The height and width
range of the latent space is set as $[-H,H]$ and $[-W,W]$ respectively,
resulting in a distance of 2 between neighboring latent vectors.

The high-resolution ground-truth phase images, with an original pixel
resolution of $1500\times 1500$ pixels for the experimental dataset and
$600\times 600$ pixels for the simulated dataset, are correspondingly cropped
into $288\times 288$-pixel patches to match the same FOV as the input images.
This scaling indicates that our ground-truth high-resolution phase image has a
pixel resolution $6\times$ higher than that of the input in both the $x$ and
$y$ directions.

Similar to the assignment of 2D coordinates in the latent space, we assign 2D
coordinates to the cropped high-resolution image within the height and width
range of $[-H,H]$ and $[-W,W]$ respectively. The grid density is increased to
$288\times 288$, and the distance between adjacent pixels is reduced to 1/3.
This coordinate assignment ensures positional consistency across the
measurement domain, latent space, and reconstruction domain, assuming that the
information within a 2D image is inherently positionally dependent and the
information at a given position is preserved across different domains.

Next, we randomly select 2304 pixels from the high-resolution image patch as
the ground-truth phase values by randomly picking the coordinates defined in
the high-resolution grid. These coordinates are also used to select the
corresponding latent vectors $\phi$ from $\Phi$, as described in Section.
4.3.2. The selected latent vectors and relative coordinates are concatenated
and input into the MLP. We further employ the reconstruction enhancement
techniques detailed in Section 4.3.3.

The output of the MLP is the predicted phase value at the queried position
$\mathbf{c}$, and we train our network by comparing this prediction with the
ground truth using the $L_{1}$ norm, as shown in Eq. (1).

It is important to note that during the training stage, we define grids for
the high-resolution ground-truth image and query the high-resolution image at
these predefined coordinates. However, after training, we no longer need to
query points at predefined grids and can freely query phase values at any
coordinates since our MLP can effectively represent the object in a continuous
manner.

We utilize the PyTorch framework for training our network. The Adam optimizer
is employed, with an initial learning rate of $1\times 10^{-4}$. To adaptively
adjust the learning rate during training, we use the ReduceLROnPlateau method
in PyTorch. This method reduces the learning rate by a factor of 0.2 when the
loss function fails to improve. During training, a batch size of 5 is used.

Training with different datasets: To comprehensively evaluate the
generalization capability of our LCNF framework, in total, we explored three
different training strategies and trained five different networks using
different datasets, as detailed below.

Training with the full experimental dataset.

In this case, we trained two networks using two different experimentally
captured Hela cell datasets, denoted as Network${}_{\text{Hela-E(18)}}$ and
Network${}_{\text{Hela-F(16)}}$.

For the first Hela(E) dataset, we gathered 22 groups of images for Hela cells
fixed in ethanol. Network${}_{\text{Hela-E(18)}}$ was trained using 18 paired
datasets, validated using 2 paired datasets, and tested using 2 paired
datasets.

For the second Hela(F) dataset, we captured 20 groups of Hela cells fixed in
formalin. Network${}_{\text{Hela-F(16)}}$ was trained using 16 paired
datasets, validated using 2 paired datasets, and tested using 2 paired
datasets.

Training with a single pair of experimental dataset.

To further evaluate the generalization capability of our network, we conducted
training of two networks using only a single training image pair from two
different cell types, denoted as Network${}_{\text{Hela-E(1)}}$ and
Network${}_{\text{Hela-F(1)}}$. The remaining images were designated as the
test set. This approach allows us to assess the network’s performance when
trained on extremely limited data, providing insights into its generalization
ability and the capability to reduce the complexity of acquiring experimental
training data.

Training with the simulated natural images dataset.

In addition to the experimental dataset, we also trained another network using
only simulated datasets on natural images, denoted as
Network${}_{\text{Simulate}}$. The data preparation is described in Section
4.3.1. For this purpose, we utilized a total of 800 paired images for
training, with 50 paired images for validation and another 50 paired images
for testing. This simulated dataset allows us to assess the performance of our
network in the absence of an experimental training dataset, providing insights
into its ability to generalize from simulation to experiment.

For all three training scenarios, the network typically converged at around
500 epochs. The training duration varied depending on the dataset. Training
the network with a single pair of the experimental dataset took approximately
3 hours while training with the full experimental dataset and the simulated
dataset took approximately 24 hours to converge using an NVIDIA Tesla P100 GPU
on the Boston University Shared Computing Cluster.

####  4.3.5 Network inference

Upon completion of network training, we can reconstruct high-resolution phase
images using a continuous local conditional neural field representation. To
perform network inference, we provide the preprocessed measurements of the
desired FOV as input and configure the pixel resolution for the resulting
reconstructed image. During the inference process, the network assigns
coordinates to each pixel, as described in Section 4.3.4, and predicts the
corresponding phase value for each queried position.

In contrast to previous NF frameworks [12] that require a consistent number of
input coordinates, resulting in a smaller FOV when aiming for a higher pixel
resolution image, our approach maintains the same FOV while increasing the
number of queried coordinates. We achieve this by employing varying grid
densities for reconstruction. Notably, the prediction process for a
$1500\times 1500$-pixel high-resolution image takes approximately 25 seconds,
resulting in an average rate of approximately $1\times 10^{-5}$ seconds per
pixel on a computer with an NVIDIA Quadro RTX8000 GPU.

For the reconstruction of the wide-FOV phase image, we performed inference
with a $6\times$ denser grid sampling compared to the raw measurement. We
first divided our measurement into a series of small patches with $250\times
250$ pixels each. Next, we reconstructed each patch individually, resulting in
high-resolution phase images with dimensions of $1500\times 1500$ pixels. To
create the final wide-FOV reconstruction image, we employed the alpha blending
algorithm [7] to stitch together the individual reconstructions, forming a
high-resolution phase image with a diameter of 12960 pixels. It is worth
noting that our LCNF network is inherently capable of directly inferring the
entire FOV image without requiring any stitching process. However, due to the
limitation of GPU memory (48 GB) on our computer, we utilized this patch-wise
inference method since direct inference of the entire FOV would exceed the
available memory.

## Acknowledgements

The authors acknowledge Boston University Shared Computing Cluster for proving
the computational resources. The work is funded by National Science Foundation
(1846784).

## Data availability

The neural network and the data set used in this work are available at
https://github.com/bu-cisl/LCNF.

## Conflict of interest

The authors declare no competing interests.

## References

  * [1] G. Barbastathis, A. Ozcan, and G. Situ, “On the use of deep learning for computational imaging,” Optica, vol. 6, no. 8, pp. 921–943, 2019.
  * [2] G. Volpe, C. Wählby, L. Tian, M. Hecht, A. Yakimovich, K. Monakhova, L. Waller, I. F. Sbalzarini, C. A. Metzler, M. Xie, K. Zhang, I. C. D. Lenton, H. Rubinsztein-Dunlop, D. Brunner, B. Bai, A. Ozcan, D. Midtvedt, H. Wang, N. Sladoje, J. Lindblad, J. T. Smith, M. Ochoa, M. Barroso, X. Intes, T. Qiu, L.-Y. Yu, S. You, Y. Liu, M. A. Ziatdinov, S. V. Kalinin, A. Sheridan, U. Manor, E. Nehme, O. Goldenberg, Y. Shechtman, H. K. Moberg, C. Langhammer, B. Špačková, S. Helgadottir, B. Midtvedt, A. Argun, T. Thalheim, F. Cichos, S. Bo, L. Hubatsch, J. Pineda, C. Manzo, H. Bachimanchi, E. Selander, A. Homs-Corbera, M. Fränzl, K. de Haan, Y. Rivenson, Z. Korczak, C. B. Adiels, M. Mijalkov, D. Veréb, Y.-W. Chang, J. B. Pereira, D. Matuszewski, G. Kylberg, I.-M. Sintorn, J. C. Caicedo, B. A. Cimini, M. A. L. Bell, B. M. Saraiva, G. Jacquemet, R. Henriques, W. Ouyang, T. Le, E. G. de Mariscal, D. Sage, A. Muñoz-Barrutia, E. J. Lindqvist, and J. Bergman, “Roadmap on deep learning for microscopy,” arXiv preprint arXiv:2303.03793, 2023.
  * [3] A. Sinha, J. Lee, S. Li, and G. Barbastathis, “Lensless computational imaging through deep learning,” Optica, vol. 4, no. 9, pp. 1117–1125, 2017.
  * [4] F. Wang, Y. Bian, H. Wang, M. Lyu, G. Pedrini, W. Osten, G. Barbastathis, and G. Situ, “Phase imaging with an untrained neural network,” Light: Science & Applications, vol. 9, no. 1, pp. 1–7, 2020.
  * [5] E. Bostan, R. Heckel, M. Chen, M. Kellman, and L. Waller, “Deep phase decoder: self-calibrating phase microscopy with an untrained deep neural network,” Optica, vol. 7, no. 6, pp. 559–562, 2020.
  * [6] T. Nguyen, Y. Xue, Y. Li, L. Tian, and G. Nehmetallah, “Deep learning approach for fourier ptychography microscopy,” Optics express, vol. 26, no. 20, pp. 26470–26484, 2018.
  * [7] Y. Xue, S. Cheng, Y. Li, and L. Tian, “Reliable deep-learning-based phase imaging with uncertainty quantification,” Optica, vol. 6, no. 5, pp. 618–629, 2019.
  * [8] Y. Rivenson, Y. Zhang, H. Günaydın, D. Teng, and A. Ozcan, “Phase recovery and holographic image reconstruction using deep learning in neural networks,” Light: Science & Applications, vol. 7, no. 2, pp. 17141–17141, 2018.
  * [9] Y. Rivenson, Y. Wu, and A. Ozcan, “Deep learning in holography and coherent imaging,” Light: Science & Applications, vol. 8, no. 1, p. 85, 2019.
  * [10] A. Matlock, J. Zhu, and L. Tian, “Multiple-scattering simulator-trained neural network for intensity diffraction tomography,” Optics Express, vol. 31, no. 3, pp. 4094–4107, 2023.
  * [11] A. Saba, C. Gigli, A. B. Ayoub, and D. Psaltis, “Physics-informed neural networks for diffraction tomography,” Advanced Photonics, vol. 4, no. 6, p. 066001, 2022.
  * [12] R. Liu, Y. Sun, J. Zhu, L. Tian, and U. S. Kamilov, “Recovery of continuous 3d refractive index maps from discrete intensity-only measurements using neural fields,” Nature Machine Intelligence, vol. 4, no. 9, pp. 781–791, 2022\.
  * [13] F. Wang, C. Wang, M. Chen, W. Gong, Y. Zhang, S. Han, and G. Situ, “Far-field super-resolution ghost imaging with a deep neural network constraint,” Light: Science & Applications, vol. 11, no. 1, p. 1, 2022.
  * [14] Y. Rivenson, Z. Göröcs, H. Günaydin, Y. Zhang, H. Wang, and A. Ozcan, “Deep learning microscopy,” Optica, vol. 4, no. 11, pp. 1437–1443, 2017.
  * [15] E. Nehme, L. E. Weiss, T. Michaeli, and Y. Shechtman, “Deep-storm: super-resolution single-molecule microscopy by deep learning,” Optica, vol. 5, pp. 458–464, Apr 2018.
  * [16] H. Wang, Y. Rivenson, Y. Jin, Z. Wei, R. Gao, H. Günaydın, L. A. Bentolila, C. Kural, and A. Ozcan, “Deep learning enables cross-modality super-resolution in fluorescence microscopy,” Nature Methods, vol. 16, no. 1, pp. 103–110, 2019.
  * [17] Z. Wang, L. Zhu, H. Zhang, G. Li, C. Yi, Y. Li, Y. Yang, Y. Ding, M. Zhen, S. Gao, T. Hsiai, and P. Fei, “Real-time volumetric reconstruction of biological dynamics with light-field microscopy and deep learning,” Nature Methods, vol. 18, no. 5, pp. 551–556, 2021.
  * [18] N. Wagner, F. Beuttenmueller, N. Norlin, J. Gierten, J. C. Boffi, J. Wittbrodt, M. Weigert, L. Hufnagel, R. Prevedel, and A. Kreshuk, “Deep learning-enhanced light-field imaging with continuous validation,” Nature Methods, vol. 18, no. 5, pp. 557–563, 2021.
  * [19] Y. Xue, Q. Yang, G. Hu, K. Guo, and L. Tian, “Deep-learning-augmented computational miniature mesoscope,” Optica, vol. 9, no. 9, pp. 1009–1021, 2022.
  * [20] K. Yanny, K. Monakhova, R. W. Shuai, and L. Waller, “Deep learning for fast spatially varying deconvolution,” Optica, vol. 9, no. 1, pp. 96–99, 2022\.
  * [21] S. S. Khan, V. Sundar, V. Boominathan, A. Veeraraghavan, and K. Mitra, “Flatnet: Towards photorealistic scene reconstruction from lensless measurements,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 4, pp. 1934–1948, 2020.
  * [22] Y. Li, Y. Xue, and L. Tian, “Deep speckle correlation: a deep learning approach toward scalable imaging through scattering media,” Optica, vol. 5, no. 10, pp. 1181–1190, 2018.
  * [23] M. Lyu, H. Wang, G. Li, S. Zheng, and G. Situ, “Learning-based lensless imaging through optically thick scattering media,” Advanced Photonics, vol. 1, no. 3, p. 036002, 2019.
  * [24] W. Tahir, H. Wang, and L. Tian, “Adaptive 3d descattering with a dynamic synthesis network,” Light: Science & Applications, vol. 11, no. 1, p. 42, 2022.
  * [25] L. Tian and L. Waller, “Quantitative differential phase contrast imaging in an led array microscope,” Opt. Express, vol. 23, pp. 11394–11403, May 2015\.
  * [26] L. Tian, X. Li, K. Ramchandran, and L. Waller, “Multiplexed coded illumination for fourier ptychography with an led array microscope,” Biomedical optics express, vol. 5, no. 7, pp. 2376–2389, 2014.
  * [27] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar, “Neural fields in visual computing and beyond,” in Computer Graphics Forum, vol. 41, pp. 641–676, Wiley Online Library, 2022.
  * [28] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” Communications of the ACM, vol. 65, no. 1, pp. 99–106, 2021\.
  * [29] Y. Sun, J. Liu, M. Xie, B. Wohlberg, and U. S. Kamilov, “Coil: Coordinate-based internal learning for tomographic imaging,” IEEE Transactions on Computational Imaging, vol. 7, pp. 1400–1412, 2021.
  * [30] A. W. Reed, H. Kim, R. Anirudh, K. A. Mohan, K. Champley, J. Kang, and S. Jayasuriya, “Dynamic ct reconstruction from limited views with implicit neural representations and parametric motion fields,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2258–2268, 2021\.
  * [31] L. Lozenski, M. A. Anastasio, and U. Villa, “A memory-efficient self-supervised dynamic image reconstruction method using neural fields,” IEEE Transactions on Computational Imaging, vol. 8, pp. 879–892, 2022.
  * [32] L. Shen, J. Pauly, and L. Xing, “Nerp: implicit neural representation learning with prior embedding for sparsely sampled image reconstruction,” IEEE Transactions on Neural Networks and Learning Systems, 2022.
  * [33] H. Zhu, Z. Liu, Y. Zhou, Z. Ma, and X. Cao, “Dnf: diffractive neural field for lensless microscopic imaging,” Opt. Express, vol. 30, pp. 18168–18178, May 2022.
  * [34] W. Ren, S. Shen, L. Li, S. Gao, Y. Wang, L. Gu, S. Li, X. Zhu, J. Jiang, and J. Yu, “High-resolution tomographic reconstruction of optical absorbance through scattering media using neural fields,” arXiv preprint arXiv:2304.01682, 2023.
  * [35] E. D. Zhong, T. Bepler, B. Berger, and J. H. Davis, “Cryodrgn: reconstruction of heterogeneous cryo-em structures using neural networks,” Nature methods, vol. 18, no. 2, pp. 176–185, 2021.
  * [36] B. Y. Feng, H. Guo, M. Xie, V. Boominathan, M. K. Sharma, A. Veeraraghavan, and C. A. Metzler, “Neuws: Neural wavefront shaping for guidestar-free imaging through static and dynamic scattering media,” Science Advances, vol. 9, no. 26, p. eadg4671, 2023.
  * [37] Y. Chen, S. Liu, and X. Wang, “Learning continuous image representation with local implicit image function,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8628–8638, 2021.
  * [38] G. Zheng, R. Horstmeyer, and C. Yang, “Wide-field, high-resolution fourier ptychographic microscopy,” Nature photonics, vol. 7, no. 9, pp. 739–745, 2013.
  * [39] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [40] L.-H. Yeh, J. Dong, J. Zhong, L. Tian, M. Chen, G. Tang, M. Soltanolkotabi, and L. Waller, “Experimental robustness of fourier ptychography phase retrieval algorithms,” Optics express, vol. 23, no. 26, pp. 33214–33240, 2015.
  * [41] H. Wang, J. Zhu, J. Sung, G. Hu, J. Greene, Y. Li, S. Park, W. Kim, M. Lee, Y. Yang, and L. Tian, “Fourier ptychographic topography,” Opt. Express, vol. 31, pp. 11007–11018, Mar 2023.
  * [42] X. Ou, G. Zheng, and C. Yang, “Embedded pupil function recovery for fourier ptychographic microscopy,” Optics express, vol. 22, no. 5, pp. 4960–4972, 2014.
  * [43] K. De and V. Masilamani, “Image sharpness measure for blurred images in frequency domain,” Procedia Engineering, vol. 64, pp. 149–158, 2013.
  * [44] F. Wang, H. Wang, H. Wang, G. Li, and G. Situ, “Learning from simulation: An end-to-end deep-learning approach for computational ghost imaging,” Opt. Express, vol. 27, pp. 25560–25572, Sep 2019.
  * [45] L. Tian, Z. Liu, L.-H. Yeh, M. Chen, J. Zhong, and L. Waller, “Computational illumination for high-speed in vitro fourier ptychographic microscopy,” Optica, vol. 2, no. 10, pp. 904–911, 2015.
  * [46] M. Chen, Z. F. Phillips, and L. Waller, “Quantitative differential phase contrast (dpc) microscopy with computational aberration correction,” Optics express, vol. 26, no. 25, pp. 32888–32899, 2018.
  * [47] D. C. Ghiglia and L. A. Romero, “Robust two-dimensional weighted and unweighted phase unwrapping that uses fast transforms and iterative methods,” JOSA A, vol. 11, no. 1, pp. 107–117, 1994.
  * [48] E. Agustsson and R. Timofte, “Ntire 2017 challenge on single image super-resolution: Dataset and study,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 126–135, 2017.
  * [49] R. Timofte, E. Agustsson, L. Van Gool, M.-H. Yang, and L. Zhang, “Ntire 2017 challenge on single image super-resolution: Methods and results,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 114–125, 2017.
  * [50] B. Lim, S. Son, H. Kim, S. Nah, and K. Mu Lee, “Enhanced deep residual networks for single image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp. 136–144, 2017.

## Supplementary information for:
Local Conditional Neural Fields for Versatile and Generalizable Large-Scale
Reconstructions in Computational Imaging

Hao Wang1, Jiabei Zhu1, Yunzhe Li1,†, Qianwan Yang1, Lei Tian1,2,∗

[1] Department of Electrical and Computer Engineering, Boston University,
Boston, MA 02215, USA.

[2] Department of Biomedical Engineering, Boston University, Boston, MA 02215,
USA.

[$\dagger$]Current address: Department of Electrical Engineering & Computer
Sciences, University of California, Berkeley, CA 94720, USA.
* Correspondence<EMAIL_ADDRESS>
Figure S1: Network structure. The LCNF network structure in our framework
consists of a CNN-based encoder and an MLP-based decoder. Encoder: The Encoder
module includes three CNNs ($e_{1}$,$e_{2}$,$e_{3}$). Each encoder consists of
an input convolutional layer, 32 residual blocks, and an output convolutional
layer. The input channel of the input convolutional layer depends on the
number of input images, denoted as “input” in the figure. For two brightfield
intensity images, it has 2 channels. For three darkfield intensity images, it
has 3 channels. And for the differential phase contrast image, it has 1
channel. The output channel for all three encoders is set to 128. Each
residual block involves two convolutional layers (input channel: 128, output
channel: 128), followed by a ReLU activation layer and a multiplication layer
with a multiplication factor of 1. Each residual block utilizes a skip
connection that adds the feature maps. The spatial features extracted by the
residual blocks are then passed through the output convolutional layer, which
has 128 input channels and 128 output channels. Finally, features extracted by
the output convolutional layer are added with the feature maps provided by the
input convolutional layer by a skip connection. All convolutional layers use
$3\times 3$ convolutional kernels. The feature maps generated by the three
encoders are concatenated together to form the encoded latent space
representation. Decoder: To generate a high-resolution image at the queried
position based on the encoded latent vectors, we employ a 5-layer MLP with
ReLU activation and hidden dimensions of 256 as the decoding function
$f_{\theta}$. The latent vector $\phi$ associated with the relative
coordinates $\Delta\mathbf{c}$ of the queried position $\mathbf{c}$ (green
dot) is constructed by concatenating the latent vectors generated by the three
encoders: $\phi_{1}$, $\phi_{2}$, and $\phi_{3}$. The input dimension of the
MLP is 3460, calculated by $3$ (number of encoders) $\times$ 128 (feature maps
learned by each encoder) $\times$ 9 (feature unfolding) + 2 (dimension of the
coordinates) + 2 (cell decoding). The output dimension of the MLP is 1,
representing the predicted phase value at the queried position. During
training, we compared the MLP prediction $f_{\theta}(\Delta\mathbf{c},\phi)$
with the ground truth $\psi(\mathbf{c})$ to train our network.  Figure S2:
Reconstruction enhancement techniques. (a) Feature unfolding. The $3\times 3$
neighborhood latent vectors (blue dots) surrounding the selected latent vector
(green dot) are concatenated together to provide enriched information. (b)
Cell decoding. The pixel size $(c_{h},c_{w})$ are concatenated with the
coordinates to improve the reconstruction. (c) Local ensemble. The local
ensemble is used to enhance the continuity of the reconstruction. We utilize
the nearest four latent vectors (blue dots,
$\\{\phi_{00},\phi_{01},\phi_{10},\phi_{11}\\}$) to make predictions. These
four predictions are then merged by voting with normalized confidences, which
are proportional to the area of the rectangle formed between the query point
and its selected latent vector’s diagonal counterpart.  Figure S3:
Preprocessing of natural images. To align the statistical characteristics of
natural images with the experimental cell images, a preprocessing step is
performed as described in Section 4.3.1. The figure presents the effects of
this preprocessing step. (a) Model-based reconstruction of ethanol-fixed Hela
cells. First column: an example of preprocessed high-resolution model-based
FPM reconstruction. Second column: the histogram of the phase values derived
from 22 preprocessed high-resolution Hela cell images. Third column: the PSD
of the 22 preprocessed high-resolution reconstructions. The figure
demonstrates that the phase values are predominantly distributed within the
range of [0, 7]. However, a small fraction, approximately $0.5\%$ of pixels
lie outside this range. To capture valuable information and eliminate noisy
pixels, the phase of the preprocessed experimental dataset is set within the
range of [0, 12]. (b) Raw natural images in the DIV2K dataset. First column:
an example high-resolution natural image in DIV2K dataset. Second column: the
histogram of the pixel values obtained from 900 high-resolution natural
images. Third column: the PSD of the 900 high-resolution natural images. (c)
Preprocessed natural images. First column: an example background-removed and
spectrum-corrected natural image. Second column: the histogram of the pixel
values derived from 900 preprocessed natural images. We experimentally
linearly match the pixel value to the phase range as [0, 9] since it aligns
with the predominant distribution of the experimental dataset and also covers
possible phase values larger than 7. Third column: the PSD of the 900
preprocessed images. The profiles of the PSD images along the diagonal
(indicated by the dashed black lines) are shown in the top right corner of
each image. Figure S4: The LCNF learns continuous-domain representation and
can reconstruct phase maps on an arbitrary pixel grid (illustration with
$6×\times,21\times,49.8\times,73.5\times$, and $105.9\times$ upsampling
compared with the raw measurement image). (a) Model-based FPM reconstruction
with zoom-in areas. Reconstruction artifacts are noted by red arrows. (b) LCNF
continuous reconstruction of high-resolution phase image.  Figure S5:
Quantitative evaluation of LCNF artifact removal. The red boxes indicate the
selected blank areas in the model-based FPM reconstruction and LCNF
reconstruction of cells fixed in (a) ethanol and (b) formalin. The histograms
of phase values in these blank areas are displayed in the rightmost column.
The standard deviation ($\sigma$) of phase values in the selected subareas,
indicated at the bottom of the reconstructions, is also calculated. The figure
demonstrates that our LCNF effectively eliminates fluctuating artifacts in the
background. Figure S6: Wide-FOV high-resolution reconstructions of Hela cells
fixed in ethanol using LCNF networks trained with different datasets. Figure
S7: Wide-FOV high-resolution LCNF reconstruction of Hela cells fixed in
formalin. (a) BF intensity image captured over a 2160-pixel diameter (3.51 mm)
FOV. Wide-FOV LCNF reconstruction by (b) Network${}_{\text{Hela-F(16)}}$ and
(c) Network${}_{\text{Simulate}}$. (d-g) Selected subareas extracted from the
central to the edge of the FOV, identified as i-iv and enclosed within
different colored boxes. (d) BF intensity image. (e) model-based FPM
reconstruction. (f) Network${}_{\text{Hela-F(16)}}$ reconstruction trained
with the experimental dataset. The experimental dataset used for training the
LCNF network is obtained from the central region, indicated by the dashed
black square. (g) Network${}_{\text{Simulate}}$ reconstruction trained with
the simulated dataset. LCNF can successfully reconstruct full-FOV high-
resolution images and eliminate the artifacts introduced by the model-based
reconstruction, as shown in iv. Figure S8: Wide-FOV high-resolution
reconstructions of Hela cells fixed in formalin using LCNF networks trained
with different datasets.

Table S1: Quantitative metrics for experimental dataset trained LCNF networks. In this context, “Hela (E)” and “Hela (F)” refers to fixed Hela cells preserved in ethanol (E) and formalin (F), respectively. Here, we consider the model-based FPM reconstruction as the ground truth for assessing the performance of the networks.  Dataset |  Method |  MSE |  PSNR(dB) |  SSIM |  FM
---|---|---|---|---|---
Hela (E) |  Network${}_{\text{Hela-E(18)}}$ |  $0.0935$ |  $28.3800$ |  $0.7560$ |  $0.0592$
|  Network${}_{\text{Hela-E(1)}}$ |  $0.1023$ |  $27.9902$ |  $0.7611$ |  $0.0584$
|  Network${}_{\text{Hela-F(16)}}$ |  $0.1139$ |  $27.5250$ |  $0.7586$ |  $0.0552$
|  Network${}_{\text{Hela-F(1)}}$ |  $0.1250$ |  $27.1220$ |  $0.7517$ |  $0.0486$
|  Model-based FPM |  - |  - |  - |  $0.0788$
Hela (F) |  Network${}_{\text{Hela-F(16)}}$ |  $0.1136$ |  $30.6720$ |  $0.8093$ |  $0.0430$
|  Network${}_{\text{Hela-F(1)}}$ |  $0.1527$ |  $29.7469$ |  $0.7970$ |  $0.0396$
|  Network${}_{\text{Hela-E(18)}}$ |  $0.1544$ |  $29.2155$ |  $0.8007$ |  $0.0434$
|  Network${}_{\text{Hela-E(1)}}$ |  $0.2650$ |  $26.8688$ |  $0.7528$ |  $0.0467$
|  Model-based FPM |  - |  - |  - |  $0.0510$
Table S2: Quantitative metrics for simulated dataset trained LCNF. In this context, “Natural image”, “Hela (E)”, and “Hela (F)” refers to the DIV2K natural image dataset, fixed Hela cells preserved in ethanol (E) and formalin (F), respectively. For the simulated natural image dataset, we have the ground truth high-resolution images from the DIV2K dataset, and for the experiment dataset, we use model-based FPM reconstruction as the ground truth. Dataset |  Method |  MSE |  PSNR(dB) |  SSIM |  FM
---|---|---|---|---|---
Natural image |  Network${}_{\text{Simulate}}$ |  $0.5821$ |  $21.4350$ |  $0.8533$ |  $0.0058$
|  Ground truth |  - |  - |  - |  $0.0059$
Hela (E) |  Network${}_{\text{Hela-E(18)}}$ |  $0.1317$ |  $28.1207$ |  $0.7798$ |  $0.0534$
|  Network${}_{\text{Simulate}}$ |  $0.3261$ |  $24.1839$ |  $0.6459$ |  $0.0680$
|  Model-based FPM |  - |  - |  - |  $0.0673$
Hela (F) |  Network${}_{\text{Hela-F(16)}}$ |  $0.0766$ |  $30.0294$ |  $0.8228$ |  $0.0383$
|  Network${}_{\text{Simulate}}$ |  $0.2503$ |  $24.8894$ |  $0.7399$ |  $0.0418$
|  Model-based FPM |  - |  - |  - |  $0.0391$

Table S3: Quantitative metrics for networks trained with varied datasets (mean $\pm$ standard deviation). In this context, “Hela (E)” and “Hela (F)” refers to fixed Hela cells preserved in ethanol (E) and formalin (F), respectively. The following metrics were computed by comparing the 100 high-resolution phase images ($1500\times 1500$ pixels) predicted by the networks trained with different datasets and the model-based FPM reconstructions. The reconstruction patches were extracted from FOVs beyond those in the training dataset. Dataset |  Method |  MSE |  PSNR(dB) |  SSIM |  FM
---|---|---|---|---|---
Hela (E) |  Network${}_{\text{Hela-E(18)}}$ |  $0.1753\pm 0.0606$ |  $30.0157\pm 1.7363$ |  $0.7999\pm 0.0243$ |  $0.0609\pm 0.0107$
|  Network${}_{\text{Hela-E(1)}}$ |  $0.2233\pm 0.0818$ |  $29.0015\pm 1.3947$ |  $0.7275\pm 0.0344$ |  $0.0573\pm 0.0099$
|  Network${}_{\text{Hela-F(16)}}$ |  $0.2356\pm 0.0848$ |  $28.7574\pm 1.8154$ |  $0.7715\pm 0.0231$ |  $0.0531\pm 0.0095$
|  Network${}_{\text{Hela-F(1)}}$ |  $0.2591\pm 0.0872$ |  $28.3186\pm 1.6702$ |  $0.7516\pm 0.0290$ |  $0.0463\pm 0.0079$
|  Network${}_{\text{Simulate}}$ |  $0.3704\pm 0.1290$ |  $26.7218\pm 1.6527$ |  $0.6435\pm 0.0310$ |  $0.0772\pm 0.0157$
|  Model-based |  - |  - |  - |  $0.0689\pm 0.0141$
Hela (F) |  Network${}_{\text{Hela-F(16)}}$ |  $0.1978\pm 0.0599$ |  $30.9115\pm 2.0947$ |  $0.7817\pm 0.0484$ |  $0.0445\pm 0.0061$
|  Network${}_{\text{Hela-F(1)}}$ |  $0.2322\pm 0.0685$ |  $30.1063\pm 1.6490$ |  $0.7738\pm 0.0499$ |  $0.0419\pm 0.0059$
|  Network${}_{\text{Hela-E(18)}}$ |  $0.2074\pm 0.0467$ |  $30.5551\pm 1.2916$ |  $0.7934\pm 0.0469$ |  $0.0448\pm 0.0065$
|  Network${}_{\text{Hela-E(1)}}$ |  $0.3075\pm 0.0917$ |  $28.9166\pm 1.1433$ |  $0.7252\pm 0.0557$ |  $0.0483\pm 0.0066$
|  Network${}_{\text{Simulate}}$ |  $0.3358\pm 0.0713$ |  $28.4408\pm 1.4819$ |  $0.6527\pm 0.0598$ |  $0.0503\pm 0.0092$
|  Model-based |  - |  - |  - |  $0.0479\pm 0.0080$