# Report on Methods and Applications for Crafting 3D Humans

Lei Liu, Ke Zhao, Bournemouth University

###### Abstract

This paper presents an in-depth exploration of 3D human model and avatar
generation technology, propelled by the rapid advancements in large-scale
models and artificial intelligence. The paper reviews the comprehensive
process of 3D human model generation, from scanning to rendering, and
highlights the pivotal role these models play in entertainment, VR, AR,
healthcare, and education. We underscore the significance of diffusion models
in generating high-fidelity images and videos. It emphasizes the indispensable
nature of 3D human models in enhancing user experiences and functionalities
across various fields. Furthermore, this paper anticipates the potential of
integrating large-scale models with deep learning to revolutionize 3D content
generation, offering insights into the future prospects of the technology. It
concludes by emphasizing the importance of continuous innovation in the field,
suggesting that ongoing advancements will significantly expand the
capabilities and applications of 3D human models and avatars.

###### Index Terms:

3D human, avatar, AIGC

##  I Introduction

The rapid advancement of large-scale models [1] and artificial intelligence
has significantly transformed the landscape of digital content creation. Among
the various innovations, 3D content generation technology stands out due to
its extensive applications across numerous domains. This paper focuses on the
generation of 3D human models and avatars, exploring their technical
foundations, applications, challenges, and future prospects. Diffusion models
have shown strong capabilities in high-fidelity images and video generation
[2, 3, 4, 5, 6, 7, 8]. Applying pre-trained diffusion models to 3D content
generation can significantly reduce the demand for computational resources and
the reliance on 3D datasets. 3D generation [9, 10, 11, 12, 13, 14, 15, 16, 17,
18] involves creating three-dimensional digital representations of objects,
beings, or environments through sophisticated algorithms and tools. This
technology has gained prominence in entertainment, virtual reality (VR),
augmented reality (AR), healthcare, and education, offering new dimensions of
interactivity and realism. Specifically, 3D human models and avatars play
pivotal roles in enhancing user experience and functionality in these fields.

The process of generating 3D human models [19, 19, 20, 21, 12] typically
includes scanning, modeling, and rendering. Scanning technologies capture the
geometric and textural details of a human subject using 3D scanners or multi-
view camera systems. The captured data is then processed through modeling
techniques such as mesh modeling and voxel modeling to create a high-fidelity
3D representation. Finally, rendering techniques, including ray tracing and
global illumination, convert these models into visually realistic images.
These technologies collectively enable the creation of lifelike digital humans
that can be animated and manipulated in various digital environments.

In entertainment and media, 3D human models are indispensable. Films and video
games leverage these models to bring characters to life, enhancing
storytelling and visual impact. For instance, blockbuster movies like ”Avatar”
and ”The Avengers” extensively use 3D modeling to create immersive visual
effects and realistic characters. Similarly, in video games, these models
contribute to creating engaging and interactive experiences, allowing players
to immerse themselves in virtual worlds.

Virtual reality (VR) and augmented reality (AR) applications also benefit
greatly from 3D human models. In VR, users can interact with highly detailed
and realistic human avatars, making the virtual experience more immersive and
engaging. AR applications overlay 3D human models onto the real world,
providing enhanced interactivity for educational, training, and entertainment
purposes. The realism and interactivity provided by 3D human models are
crucial for the effectiveness and appeal of VR and AR experiences.

Avatars, or digital representations of users, are another significant
application of 3D content generation technology. Avatars are extensively used
in social media, online communication, and virtual environments. Creating
avatars involves face recognition and modeling, skeletal animation, and
expression capture. These avatars can be customized to reflect the user’s
appearance, personality, and preferences, providing a personalized and
engaging digital presence. Social media platforms like Meta’s Horizon Worlds
and Snapchat’s Bitmoji allow users to create and interact through avatars,
enhancing social interactions and personal expression. In online education and
training, avatars represent teachers and students, facilitating interactive
and immersive learning experiences. Virtual meetings and conferences also
employ avatars to create more engaging and lifelike interactions, improving
communication and collaboration among remote participants.

The rapid evolution of artificial intelligence and machine learning has
ushered in a transformative era for 3D modeling, with text-to-image prior-
guided text-to-3D modeling at the forefront of this technological revolution.
This innovative approach leverages the power of natural language processing to
intelligently generate 3D models that correspond to textual descriptions,
significantly expanding the horizons and possibilities of 3D content creation.
Within this domain, a variety of more nuanced and specialized applications
have emerged, including personalized avatar creation.

The application of text-to-avatar allows users to craft unique 3D avatars
through simple textual input, reflecting not only their physical
characteristics but also their personality and emotional states. Across social
media platforms, gaming, and virtual reality environments, this technology
offers a novel mode of self-expression, enhancing user interaction and
immersive experiences.

This paper delves into text-guided 3D model generation models that are
predicated on text-to-image priors, examining their underlying principles,
technical frameworks, and practical effectiveness across various application
scenarios. Through these investigations, we aim to provide new perspectives
and insights into the field of 3D modeling, propelling continuous
technological innovation and advancement.

Looking ahead, the integration of large-scale models and deep learning into 3D
content generation holds great promise. AI can enhance the accuracy and
efficiency of 3D modeling processes, enabling the creation of more realistic
and complex models with less manual intervention. Deep learning techniques can
improve facial recognition, expression capture, and animation, making avatars
more lifelike and expressive.

##  II 3D Technologies

###  II-A 3D Representation

3D representation is vital in computer graphics, enabling the digital
visualization and manipulation of three-dimensional objects and scenes.
Different methods of 3D representation offer varying degrees of detail,
efficiency, and application suitability. Here, we delve into several key types
of 3D representations, including Explicit Representations, Point Clouds, and
Voxels.

#### Explicit Representations

Explicit representations define 3D objects with precise mathematical
descriptions. This category includes mesh-based methods, which represent
surfaces using vertices, edges, and faces. Meshes are widely used due to their
flexibility and efficiency in rendering.

#### Meshes, Point Clouds and Voxels

A mesh is composed of polygons, typically triangles, which are connected by
their edges and vertices to form the surface of a 3D object. Meshes are
favored in applications requiring detailed and smooth surfaces, such as video
games, animations, and simulations. They allow for efficient rendering [22,
23] and easy manipulation but can become complex when representing highly
detailed surfaces.

Point clouds [24, 25, 25] represent 3D shapes as collections of discrete
points in space. Each point has specific (x, y, z) coordinates and often
includes attributes like color or normal vectors. Point clouds are useful in
scanning and reconstruction applications, as they capture surface geometry
directly from real-world scans. They are employed in fields like autonomous
driving, where LiDAR sensors generate point cloud data to represent the
environment. However, point clouds can be sparse and may require significant
processing to convert into other forms of representation, like meshes or voxel
grids, for further use. The lack of explicit connectivity between points can
complicate operations like surface reconstruction and rendering.

Voxels [26, 27, 28], or volumetric pixels, are the 3D equivalent of pixels in
a 2D image. They divide the 3D space into a grid of small cubes, each storing
information about the material present at that location. Voxel grids are
suitable for applications requiring uniform space representation. They handle
complex topologies and internal structures of objects well. However, their
memory consumption is high, as high-resolution voxel grids can become very
large. This makes them less efficient for detailed surface representations
compared to meshes. Advancements in sparse voxel representations and efficient
storage techniques are mitigating these issues.

###  II-B NeRF and 3DGS

Neural Radiance Fields (NeRF) [29, 30, 31, 32] synthesize novel views of
complex 3D scenes from sparse sets of 2D images. NeRF represents a scene using
a continuous 5D function mapping spatial coordinates and viewing directions to
color and density values. This enables photorealistic image creation from new
viewpoints by interpolating between input images. NeRF employs a fully-
connected neural network to optimize a volumetric scene function, accumulating
colors and densities to form a 2D image. The process involves marching rays
through the scene, sampling 3D points, and predicting color and density values
using the neural network, allowing high-fidelity reconstructions with
intricate details and realistic lighting. Classic NeRF has limitations, such
as long training times and the need to train a new model for each scene.
Extensions like InstantNeRF [33], which uses a multiresolution hash table to
accelerate training, and PixelNeRF [34], which leverages convolutional neural
networks for view synthesis from a single image, address these challenges.

3D Gaussian Splatting [35, 36, 9, 37, 38] is a sophisticated volumetric
rendering technique used for creating and manipulating intricate visual
effects within a three-dimensional space. This method involves ”splatting”
Gaussian distributions throughout the volume data to simulate the impact of
light sources, material properties, and geometric shapes on a scene. Each
Gaussian distribution, or ”splat,” represents a minuscule volumetric element
of the scene, encapsulating attributes such as color, density, and
transparency. By blending these splats, 3D Gaussian Splatting can produce
highly realistic imagery, making it a valuable tool for applications in
computer graphics, visual effects, and scientific visualization. Potential
applications range from virtual property tours and urban planning to creating
photorealistic avatars for telepresence in VR environments.

###  II-C Diffusion Models

Diffusion models are generative models based on the idea of reversing a
diffusion process, transforming data from a simple initial distribution to a
complex target distribution. The diffusion process involves gradually adding
noise to data, while the reverse process denoises it to generate new samples.

#### Forward Process (Diffusion)

The forward process can be described as a Markov chain where noise is added to
the data at each time step. This is represented by the following equation:

$q(x_{t}|x_{t-1})=\mathcal{N}(x_{t};\sqrt{1-\beta_{t}}x_{t-1},\beta_{t}\mathbf{I})$

Here: \- $x_{t}$ is the data at time step $t$. \- $\beta_{t}$ is a variance
schedule controlling the amount of noise added. \- $\mathcal{N}$ denotes a
Gaussian distribution.

The forward process starts from the original data $x_{0}$ and progressively
adds noise:

$q(x_{t}|x_{0})=\mathcal{N}(x_{t};\sqrt{\bar{\alpha}_{t}}x_{0},(1-\bar{\alpha}_{t})\mathbf{I})$

where: \- $\bar{\alpha}_{t}=\prod_{s=1}^{t}(1-\beta_{s})$.

#### Reverse Process (Denoising)

The reverse process learns to denoise the data, moving from the noisy data
$x_{t}$ back to the original data $x_{0}$. The reverse process is
parameterized by a neural network $p_{\theta}$:

$p_{\theta}(x_{t-1}|x_{t})=\mathcal{N}(x_{t-1};\mu_{\theta}(x_{t},t),\sigma^{2}_{t}\mathbf{I})$

Here, $\mu_{\theta}(x_{t},t)$ is the mean predicted by the neural network, and
$\sigma^{2}_{t}$ is typically fixed.

#### Training Objective

The training objective is to minimize the difference between the true reverse
process and the model’s predictions. This can be formulated as:

$L_{\text{simple}}=\mathbb{E}_{t,x_{0},\epsilon}\left[\left\|\epsilon-\epsilon_{\theta}(x_{t},t)\right\|^{2}\right]$

where: \- $\epsilon$ is the noise added in the forward process. \-
$\epsilon_{\theta}$ is the neural network’s prediction of the noise.

#### Sampling

To generate new samples, the model starts from random noise $x_{T}$ and
iteratively applies the reverse process:

$x_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(x_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\epsilon_{\theta}(x_{t},t)\right)+\sigma_{t}z$

where $z\sim\mathcal{N}(0,\mathbf{I})$ is Gaussian noise.

In summary, diffusion models leverage a forward process that adds noise to the
data and a learned reverse process that removes noise, enabling the generation
of complex data distributions from simple initial noise.

Figure 1: DreamAvatar [39] can generate high-quality geometry and texture for
any type of human avatar. Figure 2: DreamAvatar can handle and control the 3D
generation with any pose. Figure 3: Overview of DreamAvatar. Figure 4: We show
the pipeline of Dreamface [19]. Figure 5: Geometry generation pipeline of
DreamFace. Figure 6: DreamFace generates facial assets of celebrities that
capture their personalized characteristics and achieve a high degree of
resemblance. Figure 7: Method Overview of AvatarCraft [20]. Figure 8:
Qualitative Result of AvatarCraft.

##  III Avatar

Text-to-3D [40, 10, 41, 11, 42, 43, 44, 45, 46, 47, 48] is an emerging
interdisciplinary field that focuses on converting textual descriptions into
three-dimensional models. This technology has a wide range of applications,
including virtual reality, gaming, and design.

In recent years, the creation of 3D graphical human models has drawn
considerable attention due to its extensive applications in areas such as
movie production, video gaming, AR/VR and human-computer interactions, and the
creation of 3D avatars through natural language could save resources and holds
great research prospects. 3D human generation is a complex and evolving field
in computer graphics and artificial intelligence, aiming to create detailed,
animatable human models with realistic movements and non-rigid components such
as hair and clothing. The work in this area can be broadly divided into two
categories: generating the full 3D human body and generating the 3D avatar
head. Below, we discuss significant contributions and methodologies in these
domains.

The generation of 3D human avatars involves creating diverse virtual humans
with different identities, shapes, and poses. This task is particularly
challenging due to the variety in clothed body shapes and their complex
articulations. Additionally, 3D avatars should be animatable, making the task
even more difficult as it requires creating non-rigid components.

Early work in 3D human mesh generation [49, 50, 51] utilized human parametric
models. These models represent human body shapes using a small set of
parameters that deform a template mesh. While these models facilitate easy
synthesis of human shapes by adjusting a few parameters, they often fail to
capture finer details due to the fixed mesh topology and limited resolution.

To address these limitations, more recent methods have employed implicit non-
rigid representations. For instance, SNARF [52] generate detailed 3D human
avatars by predicting skinning fields and surface normals. gDNA [53] extends
this approach by predicting a skinning field and normal field using an MLP,
trained on raw posed 3D scans. AvatarGen [54] leverages an SDF based on a SMPL
prior to provide disentangled control over geometry and appearance, while
GNARF [55] and utilize generative and compositional NeRFs for high-resolution
rendering and efficient training. Another approach focuses on creating
animatable 3D avatars from 3D scans. Traditional methods often used linear
blend skinning, which is simple but cannot produce pose-dependent variations,
leading to artifacts. To overcome this, techniques like dual quaternion blend
skinning [56] and multi-weight enveloping [57] have been developed.

Recently, implicit representations have become popular for generating
animatable 3D avatars from scans due to their resolution-independence and
smoothness. NASA [58] uses an occupancy field to model humans as deformable
components. Several method [52, 59] explore occupancy and SDF-based
representations for animatable human characters.

Further advancements include deformable and animatable NeRFs, such as those
proposed by previous works [60, 61, 62], which synthesize humans from novel
poses and views.

Generating 3D avatar heads requires fine-grained control over facial
expressions and realistic modeling of complex components like hair.
Traditional methods like the 3D Morphable Models [63] simplify face modeling
by fitting values in a linear subspace. Variants include multilinear models
[64], and fully articulated head models [65]. Deep generative methods [66] and
recent work has also focused on editing avatar faces. For example, IDE-3D [67]
allows interactive editing of shape and texture.

The generation of 3D human bodies and heads involves a variety of techniques
ranging from parametric models and implicit representations to advanced neural
network-based methods. These approaches strive to create detailed, realistic,
and animatable 3D human models, addressing challenges like capturing fine
details, managing non-rigid components, and ensuring realistic motion
generation.

##  IV Text-to-3D avatar generation

###  IV-A DreamAvatar

DreamAvatar [39] is a cutting-edge 3D human avatar generation framework. As
shown in Fig, 1, 2 and 3, this framework employs a trainable Neural Radiance
Field (NeRF) model to predict density and color for 3D points, complemented by
pretrained text-to-image diffusion models that offer 2D self-supervision. The
key innovations of DreamAvatar are threefold:

#### Utilization of the SMPL Model:

By leveraging the Skinned Multi-Person Linear (SMPL) model, DreamAvatar
provides robust shape and pose guidance, facilitating the generation process
to produce 3D models that adhere to human body structures.

#### Dual-Observation Space Design:

The framework introduces a novel design that incorporates two observation
spaces—a canonical space and a posed space—related by a learnable deformation
field, aiding in the generation of more comprehensive textures and geometries
that are faithful to the target pose.

#### Joint Optimization of Loss Functions:

DreamAvatar optimizes not only from the perspective of the full body but also
from a zoomed-in 3D head viewpoint, addressing the multi-face ”Janus” problem
and enhancing facial details in the generated avatars.

DreamAvatar details the implementation of the method, including the use of
threestudio for NeRF and Variational Score Distillation (VSD), as well as
specific versions of Stable Diffusion and ControlNet. Extensive evaluations on
various text prompts have demonstrated DreamAvatar’s effectiveness,
outperforming existing text-guided 3D generation methods by producing 3D human
avatars with high-quality geometry and consistent textures. Additionally, the
paper presents user studies and further analyses that substantiate
DreamAvatar’s leading position in 3D avatar generation. It also discusses the
limitations of the current implementation, such as the lack of consideration
for animation and potential biases inherited from the pretrained diffusion
model. Finally, the paper contemplates the societal impact of this technology,
highlighting both the positive contributions to metaverse development and the
risks associated with the malicious use of such technology.

###  IV-B DreamFace

DreamFace [19], a pioneering framework introduced, represents a significant
stride in the generation of 3D facial assets. As shown in Fig. 4, 5 and 6,
this technology stands out for its ability to transform textual prompts into
lifelike and customizable 3D faces with intricate details and animations. The
methodology underpinning DreamFace is a testament to its innovative approach,
combining a coarse-to-fine geometry generation strategy with a dual-path
mechanism that employs both a generic Latent Diffusion Model (LDM) and a
specialized texture LDM. This dual-path mechanism is pivotal as it ensures the
generation of facial textures that are not only diverse but also adhere to the
specific requirements of the UV space.

The framework’s two-stage optimization process is another keystone of its
success, allowing for efficient and fine-grained synthesis. This process is
carried out in both the latent and image spaces, which is crucial for mapping
the compact latent space to physically-based textures. The result is a
seamless integration of high-quality textures that are essential for photo-
realistic rendering.

Moreover, DreamFace’s animation capabilities are enhanced through the support
of blendshapes and the introduction of a cross-identity hypernetwork. This
empowers the generation of personalized and nuanced facial expressions,
enabling the creation of dynamic and engaging characters. The framework’s
compatibility with existing computer graphics pipelines and its ability to
animate using in-the-wild video footage significantly democratize the creation
of 3D facial assets, making it accessible to both professionals and novices
alike.

DreamFace’s methodological sophistication lies in its multi-stage generation
process that progressively refines the facial geometry, meticulously applies
physically-based textures, and enriches the asset with animatable features.
This comprehensive approach positions DreamFace at the forefront of 3D facial
asset generation technology, offering a versatile solution for a myriad of
applications across the digital content creation spectrum.

###  IV-C AvatarCraft

As shown in Fig. 7 and 8, AvatarCraft [20] is an innovative approach to
transforming text descriptions into high-quality 3D human avatars with
controllable poses and shapes. This method leverages diffusion models to guide
the learning of geometry and texture for a neural avatar based on a single
text prompt.

AvatarCraft begins with a basic neural human avatar template. Given a text
prompt, It uses diffusion models to update the template with geometry and
texture that align with the text description. To address the challenge of
distorted geometry and textures that can arise from direct application of
diffusion models, the team designed a novel coarse-to-fine multi-bounding-box
training strategy. This strategy improves global style consistency while
preserving fine details.

A key aspect of AvatarCraft is shape regularization, introduced to stabilize
the optimization process by penalizing the accumulated ray opacity of the
avatar. This approach helps to prevent the generation of degenerate solutions
such as empty volumes or flat geometry.

For animation capabilities, AvatarCraft employs a deformation technique that
uses an explicit warping field. This field maps the target human mesh to a
template human mesh, with both meshes represented using parametric human
models. This method simplifies the animation and reshaping of the generated
avatar by controlling pose and shape parameters, without the need for
additional training.

The AvatarCraft framework also introduces a coarse-to-fine generation strategy
that captures style details at different scales. This strategy is critical for
generating avatars with important visual features correctly aligned, as it
ensures that finer texture details are generated as the resolution increases.

In terms of implementation, the team combined the neural implicit field
architecture of NeuS with the fast training approach of Instant-NGP. This
combination improves reconstruction speed while maintaining quality. The
avatar creation process involves training on multi-view renderings of a bare
SMPL mesh, using the Score Distillation Score (SDS) loss from DreamFusion for
guidance.

Extensive experiments using various text descriptions demonstrate that
AvatarCraft is effective and robust in creating human avatars and rendering
novel views, poses, and shapes. The project’s webpage provides a more
immersive view into the 3D results generated by the method.

##  V Applications

The development of advanced techniques for 3D human and avatar generation has
opened up a myriad of applications across various industries. These
applications leverage the ability to create detailed, animatable, and
realistic virtual humans for purposes ranging from entertainment to
healthcare. Below, we explore several key areas where 3D human and avatar
technologies are making a significant impact.

###  V-A Entertainment and Media

One of the most prominent applications of 3D human and avatar generation is in
the entertainment and media industry. Virtual humans are used extensively in
video games, movies, and virtual reality (VR) experiences. In video games, 3D
avatars provide players with customizable characters that enhance immersion
and personal engagement. Games like ”The Sims” and ”Cyberpunk 2077” showcase
highly detailed and customizable avatars, contributing to a richer player
experience.

In the film industry, 3D avatars allow for the creation of realistic digital
doubles of actors. These digital doubles can perform stunts, de-age actors, or
even resurrect deceased performers, as seen with characters like Princess Leia
in ”Rogue One: A Star Wars Story.” Moreover, fully animated movies like
”Avatar” and ”Toy Story” rely heavily on 3D character generation to bring
fantastical worlds and characters to life.

Virtual reality and augmented reality (AR) also benefit from 3D avatars. In
VR, users can interact with fully immersive environments where avatars
represent themselves and others, enhancing social presence and interaction. In
AR, applications like Snapchat filters and virtual try-on for fashion use 3D
avatars to overlay virtual objects onto the real world.

###  V-B Social Media and Communication

Social media platforms and communication tools are increasingly incorporating
3D avatars to enhance user interaction. Platforms like Facebook and Snapchat
allow users to create personalized avatars, which can be used in posts,
stories, and as virtual representations in online meetings. These avatars add
a layer of personalization and expression, enabling users to convey emotions
and personalities more effectively than through text or 2D images alone.

In communication, 3D avatars are used in virtual meetings and telepresence
systems. Companies like Spatial and Zoom are developing technologies that
allow users to join meetings as 3D avatars, creating a more engaging and
interactive experience compared to traditional video calls. These avatars can
mimic the user’s gestures and facial expressions, making remote communication
feel more natural and connected.

###  V-C E-commerce and Retail

In the e-commerce and retail sectors, 3D avatars are revolutionizing the way
consumers shop online. Virtual try-on solutions allow customers to see how
clothes, accessories, and even cosmetics look on a 3D model of themselves
before making a purchase. This technology improves the shopping experience by
helping consumers make more informed decisions and reducing the rate of
returns due to sizing and fit issues.

For example, companies like Warby Parker and Sephora use augmented reality to
let users try on glasses and makeup virtually. Similarly, fashion retailers
are developing virtual fitting rooms where customers can upload a 3D scan of
their body to try on clothing. This not only enhances the shopping experience
but also allows for more personalized recommendations and a deeper connection
between brands and consumers.

###  V-D Healthcare and Rehabilitation

The healthcare industry is leveraging 3D human and avatar technologies for
various applications, including medical training, patient care, and
rehabilitation. In medical training, realistic 3D avatars can simulate complex
surgical procedures, allowing students and professionals to practice in a
controlled, risk-free environment. These simulations can include detailed
anatomy and realistic physiological responses, providing a valuable learning
tool.

For patient care, 3D avatars are used in telemedicine to facilitate remote
consultations. Doctors can use avatars to explain medical conditions and
treatments in a more visual and understandable way. In rehabilitation, virtual
avatars are used in therapeutic exercises, helping patients recover from
injuries or surgeries. For instance, patients can perform physical therapy
exercises guided by a virtual avatar that demonstrates the correct movements
and provides feedback.

###  V-E Fitness and Wellness

3D avatars are also finding applications in the fitness and wellness industry.
Virtual fitness trainers use avatars to guide users through workout routines,
providing a more engaging and personalized experience. These trainers can
adapt workouts based on the user’s performance and goals, offering real-time
feedback and motivation.

Moreover, wellness apps are incorporating avatars to help users visualize
their progress in areas like weight loss or muscle gain. By creating a 3D
model that evolves with the user’s physical changes, these apps provide a
tangible representation of progress, which can be highly motivating.

###  V-F Education and Training

In the field of education and training, 3D avatars are used to create
immersive learning environments. Virtual classrooms and training simulations
can incorporate avatars to represent students and instructors, facilitating
interactive and collaborative learning experiences. For example, language
learning apps can use avatars to create conversational scenarios where
learners practice speaking with virtual characters.

Corporate training programs also use 3D avatars to simulate real-world
scenarios, such as customer service interactions or emergency response drills.
These simulations provide employees with hands-on experience in a safe and
controlled setting, improving their skills and confidence.

###  V-G Fashion and Personalization

The fashion industry is leveraging 3D avatar technology to streamline design
processes and enhance personalization. Designers use 3D avatars to create
virtual prototypes of clothing, allowing for rapid iteration and visualization
of designs before physical production. This not only speeds up the design
process but also reduces waste by minimizing the need for physical samples.

For consumers, 3D avatars enable a new level of personalization. Custom
clothing brands can use avatars to create perfectly fitted garments based on a
3D scan of the customer’s body. This technology ensures a better fit and more
satisfaction with the final product.

###  V-H Cultural Heritage and Preservation

In the field of cultural heritage, 3D avatars are used to recreate historical
figures and scenes. Museums and educational institutions use these avatars to
bring history to life, providing visitors with interactive and engaging
experiences. Virtual tours can feature avatars of historical figures who guide
visitors through exhibits, sharing stories and insights.

Moreover, 3D avatars are used in the preservation of cultural heritage sites.
By creating detailed 3D models of sites and artifacts, researchers can study
and preserve these cultural treasures for future generations. These models can
be used for virtual reconstructions, allowing people to explore historical
sites in their original form.

The applications of 3D human and avatar generation are vast and diverse,
impacting numerous industries and aspects of daily life. From enhancing
entertainment and social media experiences to revolutionizing e-commerce,
healthcare, and education, the ability to create realistic and animatable 3D
humans offers significant benefits. As technology continues to advance, we can
expect even more innovative uses of 3D avatars, further integrating them into
our digital and physical worlds.

##  VI Conclusion

In conclusion, the integration of AI with 3D human model and avatar generation
has revolutionized digital content creation, offering higher quality and
broader access. The paper emphasizes the technology of personalized avatar
creation and the significant applications across industries. While the
technology promises significant benefits, it also presents challenges that
need to be managed responsibly. As these advancements continue, they are set
to reshape our digital experiences and interactions.

## References

  * [1] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2022, pp. 10 684–10 695.
  * [2] P. Li, Q. Huang, Y. Ding, and Z. Li, “Layerdiffusion: Layered controlled image editing with diffusion models,” in _SIGGRAPH Asia 2023 Technical Communications_ , 2023, pp. 1–4.
  * [3] A. Hertz, R. Mokady, J. Tenenbaum, K. Aberman, Y. Pritch, and D. Cohen-Or, “Prompt-to-prompt image editing with cross attention control,” _arXiv preprint arXiv:2208.01626_ , 2022.
  * [4] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman, “Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 22 500–22 510.
  * [5] J. Ho, W. Chan, C. Saharia, J. Whang, R. Gao, A. Gritsenko, D. P. Kingma, B. Poole, M. Norouzi, D. J. Fleet _et al._ , “Imagen video: High definition video generation with diffusion models,” _arXiv preprint arXiv:2210.02303_ , 2022.
  * [6] T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 392–18 402.
  * [7] G. Couairon, J. Verbeek, H. Schwenk, and M. Cord, “Diffedit: Diffusion-based semantic image editing with mask guidance,” _arXiv preprint arXiv:2210.11427_ , 2022.
  * [8] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever, “Zero-shot text-to-image generation,” in _International conference on machine learning_. Pmlr, 2021, pp. 8821–8831.
  * [9] J. Tang, J. Ren, H. Zhou, Z. Liu, and G. Zeng, “Dreamgaussian: Generative gaussian splatting for efficient 3d content creation,” _arXiv preprint arXiv:2309.16653_ , 2023.
  * [10] P. Li, C. Tang, Q. Huang, and Z. Li, “Art3d: 3d gaussian splatting for text-guided artistic scenes generation,” _arXiv:2405.10508_ , 2024.
  * [11] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis, S. Fidler, M.-Y. Liu, and T.-Y. Lin, “Magic3d: High-resolution text-to-3d content creation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 300–309.
  * [12] Z. Xiong, D. Kang, D. Jin, W. Chen, L. Bao, S. Cui, and X. Han, “Get3dhuman: Lifting stylegan-human into a 3d generative model using pixel-aligned reconstruction priors,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2023, pp. 9287–9297.
  * [13] R. Shi, H. Chen, Z. Zhang, M. Liu, C. Xu, X. Wei, L. Chen, C. Zeng, and H. Su, “Zero123++: a single image to consistent multi-view diffusion base model,” _arXiv preprint arXiv:2310.15110_ , 2023.
  * [14] A. Raj, S. Kaza, B. Poole, M. Niemeyer, N. Ruiz, B. Mildenhall, S. Zada, K. Aberman, M. Rubinstein, J. Barron _et al._ , “Dreambooth3d: Subject-driven text-to-3d generation,” _arXiv preprint arXiv:2303.13508_ , 2023.
  * [15] J. Sun, B. Zhang, R. Shao, L. Wang, W. Liu, Z. Xie, and Y. Liu, “Dreamcraft3d: Hierarchical 3d generation with bootstrapped diffusion prior,” _arXiv preprint arXiv:2310.16818_ , 2023.
  * [16] P. Li, B. Li, and Z. Li, “Sketch-to-Architecture: Generative AI-aided Architectural Design,” in _Pacific Graphics Short Papers and Posters_. The Eurographics Association, 2023.
  * [17] P. Li and Z. Li, “Efficient temporal denoising for improved depth map applications,” in _Proc. Int. Conf. Learn. Representations, Tiny papers_ , 2023.
  * [18] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, “Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 44, no. 3, pp. 1623–1637, 2020.
  * [19] L. Zhang, Q. Qiu, H. Lin, Q. Zhang, C. Shi, W. Yang, Y. Shi, S. Yang, L. Xu, and J. Yu, “DreamFace: Progressive generation of animatable 3d faces under text guidance,” _ACM Trans. Graph._ , 2023.
  * [20] R. Jiang, C. Wang, J. Zhang, M. Chai, M. He, D. Chen, and J. Liao, “Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control,” _arXiv preprint arXiv:2303.17606_ , 2023.
  * [21] C.-Y. Weng, B. Curless, P. P. Srinivasan, J. T. Barron, and I. Kemelmacher-Shlizerman, “Humannerf: Free-viewpoint rendering of moving people from monocular video,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern Recognition_ , 2022, pp. 16 210–16 220.
  * [22] M. Botsch, L. Kobbelt, M. Pauly, P. Alliez, and B. Lévy, _Polygon mesh processing_. CRC press, 2010.
  * [23] L. A. Shirman and C. H. Sequin, “Local surface interpolation with bézier patches,” _Computer Aided Geometric Design_ , vol. 4, no. 4, pp. 279–295, 1987.
  * [24] K.-A. Aliev, A. Sevastopolsky, M. Kolos, D. Ulyanov, and V. Lempitsky, “Neural point-based graphics,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16_. Springer, 2020, pp. 696–712.
  * [25] G. Kopanas, J. Philip, T. Leimkühler, and G. Drettakis, “Point-based neural rendering with per-view optimization,” in _Computer Graphics Forum_ , vol. 40, no. 4. Wiley Online Library, 2021, pp. 29–43.
  * [26] L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt, “Neural sparse voxel fields,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 15 651–15 663, 2020.
  * [27] C. Sun, M. Sun, and H.-T. Chen, “Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5459–5469.
  * [28] D. Maturana and S. Scherer, “Voxnet: A 3d convolutional neural network for real-time object recognition,” in _2015 IEEE/RSJ international conference on intelligent robots and systems (IROS)_. IEEE, 2015, pp. 922–928.
  * [29] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in _European conference on computer vision_. Springer, 2020, pp. 405–421.
  * [30] M. Niemeyer and A. Geiger, “GIRAFFE: Representing scenes as compositional generative neural feature fields,” in _CVPR_ , 2021.
  * [31] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, “Mip-nerf 360: Unbounded anti-aliased neural radiance fields,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 5470–5479.
  * [32] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su, “MVSNeRF: Fast generalizable radiance field reconstruction from multi-view stereo,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 124–14 133.
  * [33] T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” _ACM Transactions on Graphics (ToG)_ , vol. 41, no. 4, pp. 1–15, 2022.
  * [34] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, “pixelnerf: Neural radiance fields from one or few images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4578–4587.
  * [35] R. Abdal, W. Yifan, Z. Shi, Y. Xu, R. Po, Z. Kuang, Q. Chen, D.-Y. Yeung, and G. Wetzstein, “Gaussian shell maps for efficient 3d human generation,” _arXiv preprint arXiv:2311.17857_ , 2023.
  * [36] Z. Chen, F. Wang, and H. Liu, “Text-to-3d using gaussian splatting,” _arXiv preprint arXiv:2309.16585_ , 2023.
  * [37] X. Liu, X. Zhan, J. Tang, Y. Shan, G. Zeng, D. Lin, X. Liu, and Z. Liu, “Humangaussian: Text-driven 3d human generation with gaussian splatting,” _arXiv preprint arXiv:2311.17061_ , 2023.
  * [38] T. Yi, J. Fang, G. Wu, L. Xie, X. Zhang, W. Liu, Q. Tian, and X. Wang, “Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors,” _arXiv preprint arXiv:2310.08529_ , 2023.
  * [39] Y. Cao, Y.-P. Cao, K. Han, Y. Shan, and K.-Y. K. Wong, “Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models,” _arXiv preprint arXiv:2304.00916_ , 2023.
  * [40] L. G. Foo, H. Rahmani, and J. Liu, “Ai-generated content (aigc) for various data modalities: A survey,” _arXiv preprint arXiv:2308.14177_ , vol. 2, 2023.
  * [41] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall, “DreamFusion: Text-to-3d using 2d diffusion,” in _Int. Conf. Learn. Represent._ , 2023.
  * [42] P. Li and B. Li, “Generating daylight-driven architectural design via diffusion models,” _arXiv preprint arXiv:2404.13353_ , 2024.
  * [43] E. R. Chan, K. Nagano, M. A. Chan, A. W. Bergman, J. J. Park, A. Levy, M. Aittala, S. De Mello, T. Karras, and G. Wetzstein, “Generative novel view synthesis with 3d-aware diffusion models,” _arXiv preprint arXiv:2304.02602_ , 2023.
  * [44] J. Tang, T. Wang, B. Zhang, T. Zhang, R. Yi, L. Ma, and D. Chen, “Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior,” _arXiv preprint arXiv:2303.14184_ , 2023.
  * [45] T. Shen, J. Gao, K. Yin, M.-Y. Liu, and S. Fidler, “Deep marching tetrahedra: a hybrid representation for high-resolution 3d shape synthesis,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 6087–6101, 2021.
  * [46] Z. Wang, C. Lu, Y. Wang, F. Bao, C. Li, H. Su, and J. Zhu, “Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation,” _arXiv preprint arXiv:2305.16213_ , 2023.
  * [47] W. Li, R. Chen, X. Chen, and P. Tan, “Sweetdreamer: Aligning geometric priors in 2d diffusion for consistent text-to-3d,” _arXiv preprint arXiv:2310.02596_ , 2023.
  * [48] G. Metzer, E. Richardson, O. Patashnik, R. Giryes, and D. Cohen-Or, “Latent-nerf for shape-guided generation of 3d shapes and textures,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 12 663–12 673.
  * [49] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun, J. Rodgers, and J. Davis, “Scape: shape completion and animation of people,” in _ACM SIGGRAPH 2005 Papers_ , 2005, pp. 408–416.
  * [50] H. Joo, T. Simon, and Y. Sheikh, “Total capture: A 3d deformation model for tracking faces, hands, and bodies,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8320–8329.
  * [51] A. A. Osman, T. Bolkart, and M. J. Black, “Star: Sparse trained articulated human body regressor,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16_. Springer, 2020, pp. 598–613.
  * [52] X. Chen, Y. Zheng, M. J. Black, O. Hilliges, and A. Geiger, “Snarf: Differentiable forward skinning for animating non-rigid neural implicit shapes,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 594–11 604.
  * [53] X. Chen, T. Jiang, J. Song, J. Yang, M. J. Black, A. Geiger, and O. Hilliges, “gdna: Towards generative detailed neural avatars,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 20 427–20 437.
  * [54] J. Zhang, Z. Jiang, D. Yang, H. Xu, Y. Shi, G. Song, Z. Xu, X. Wang, and J. Feng, “Avatargen: a 3d generative model for animatable human avatars,” in _European Conference on Computer Vision_. Springer, 2022, pp. 668–685.
  * [55] A. Bergman, P. Kellnhofer, W. Yifan, E. Chan, D. Lindell, and G. Wetzstein, “Generative neural articulated radiance fields,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 19 900–19 916, 2022.
  * [56] L. Kavan, S. Collins, J. Žára, and C. O’Sullivan, “Geometric skinning with approximate dual quaternion blending,” _ACM Transactions on Graphics (TOG)_ , vol. 27, no. 4, pp. 1–23, 2008.
  * [57] X. C. Wang and C. Phillips, “Multi-weight enveloping: least-squares approximation techniques for skin animation,” in _Proceedings of the 2002 ACM SIGGRAPH/Eurographics symposium on Computer animation_ , 2002, pp. 129–138.
  * [58] B. Deng, J. P. Lewis, T. Jeruzalski, G. Pons-Moll, G. Hinton, M. Norouzi, and A. Tagliasacchi, “Nasa neural articulated shape approximation,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16_. Springer, 2020, pp. 612–628.
  * [59] G. Tiwari, N. Sarafianos, T. Tung, and G. Pons-Moll, “Neural-gif: Neural generalized implicit functions for animating people in clothing,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 11 708–11 718.
  * [60] S. Peng, J. Dong, Q. Wang, S. Zhang, Q. Shuai, X. Zhou, and H. Bao, “Animatable neural radiance fields for modeling dynamic human bodies,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 314–14 323.
  * [61] L. Liu, M. Habermann, V. Rudnev, K. Sarkar, J. Gu, and C. Theobalt, “Neural actor: Neural free-view synthesis of human actors with pose control,” _ACM transactions on graphics (TOG)_ , vol. 40, no. 6, pp. 1–16, 2021.
  * [62] Z. Zheng, H. Huang, T. Yu, H. Zhang, Y. Guo, and Y. Liu, “Structured local radiance fields for human avatar modeling,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 15 893–15 903.
  * [63] V. Blanz and T. Vetter, “A morphable model for the synthesis of 3d faces,” in _Seminal Graphics Papers: Pushing the Boundaries, Volume 2_ , 2023, pp. 157–164.
  * [64] D. Vlasic, M. Brand, H. Pfister, and J. Popovic, “Face transfer with multilinear models,” in _ACM SIGGRAPH 2006 Courses_ , 2006, pp. 24–es.
  * [65] P. Paysan, R. Knothe, B. Amberg, S. Romdhani, and T. Vetter, “A 3d face model for pose and illumination invariant face recognition,” in _2009 sixth IEEE international conference on advanced video and signal based surveillance_. Ieee, 2009, pp. 296–301.
  * [66] L. Tran and X. Liu, “On learning 3d face morphable model from in-the-wild images,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 43, no. 1, pp. 157–171, 2019.
  * [67] J. Sun, X. Wang, Y. Shi, L. Wang, J. Wang, and Y. Liu, “IDE-3D: Interactive disentangled editing for high-resolution 3d-aware portrait synthesis,” _ACM Trans. Graph._ , 2022.


# Ultra-sensitive separation estimation of optical sources

Clémentine Rouvière,1∗ David Barral,1 Antonin Grateau,1 Ilya Karuseichyk,1
Giacomo Sorelli,2 Mattia Walschaers,1 Nicolas Treps1
1Laboratoire Kastler Brossel, Sorbonne Université, CNRS, ENS-Université PSL,
Collège de France, 4 place Jussieu, Paris, F-75252, France
2Fraunhofer IOSB, Ettlingen, Fraunhofer Institute of Optronics, System
Technologies and Image Exploitation, Gutleuthausstr. 1, 76275 Ettlingen,
Germany
∗E-mail<EMAIL_ADDRESS>
(August 30, 2024)

###### Abstract

Historically, the resolution of optical imaging systems was dictated by
diffraction, and the Rayleigh criterion was long considered an unsurpassable
limit. In superresolution microscopy, this limit is overcome by manipulating
the emission properties of the object. However, in passive imaging, when
sources are uncontrolled, reaching sub-Rayleigh resolution remains a
challenge. Here, we implement a quantum-metrolgy-inspired approach for
estimating the separation between two incoherent sources, achieving a
sensitivity five orders of magnitude beyond the Rayleigh limit. Using a
spatial mode demultiplexer, we examine scenes with bright and faint sources,
through intensity measurements in the Hermite-Gauss basis. Analysing
sensitivity and accuracy over an extensive range of separations, we
demonstrate the remarkable effectiveness of demultiplexing for sub-Rayleigh
separation estimation. These results effectively render the Rayleigh limit
obsolete for passive imaging.

††preprint: APS/123-QED

## Introduction

The sensitivity and resolution of optical imaging systems play a crucial role
in numerous fields ranging from microscopy to astronomy [1, 2, 3, 4, 5]. The
core challenge, often used as a performance benchmark, is how precisely the
separation between two incoherent point sources can be resolved. This key
problem dates back to the late 19th century: the technological advances in
microscopy and astronomy enabled scientists to observe objects at higher
magnifications and with unprecedented levels of detail, but limited by
diffraction. Understanding the fundamental principles that govern the behavior
of light was essential to improve the resolution and accuracy of instruments.
Thus, some of the leading optical physicists of the time turned their
attention to this problem: Abbe [6], Rayleigh [7] and later Sparrow [8]
proposed criteria based on visual benchmarks and diffraction properties of
light. We know today that diffraction alone does not set a fundamental limit,
but combined with detectors characteristics and noise sources defines
practical boundaries [9]. Super-resolution techniques that circumvent the
diffraction limit have emerged over the last decades [10, 11, 12, 13].
However, these domain-specific techniques are hitherto limited to certain
types of microscopy. They require either intricate control over the light
source [10, 12] or manipulations of the illuminated sample [11, 13]. Thus,
these techniques are incompatible with passive imaging, where one does not
control the properties of the light incoming from the scene to be imaged.
Passive imaging with spatially resolved intensity measurement, a strategy
known as direct imaging (DI) that makes use of high-performance cameras,
provides only a limited improvement and prevents substantial advancement
beyond the limit imposed by the Rayleigh criterion [14]. However, recently,
Tsang et al. [15] approached the historic problem of estimating the separation
between two incoherent point sources adopting the framework of quantum
metrology. They demonstrated that the use of spatial-mode demultiplexing
(SPADE) combined with intensity measurements is optimal, in the sense that it
saturates the ultimate limit imposed by the laws of physics –the quantum
Cramér-Rao bound [16, 17, 18]. SPADE provides a scaling advantage for the
minimal resolvable distance compared with DI in an ideal scenario. This
advantage is preserved in the presence of experimental noise even if the
scaling is degraded [19, 20, 21, 22]. The advantages provided by this
metrology-inspired approach have been extended to optical imaging [23, 24] and
other related problems such as discrimination tasks [25, 26] and
multiparameter estimation [27, 28, 29], also including more general photon
statistics [30, 31].
Early experiments used interferometric schemes to implement a simplified
version of the demultiplexing approach [32, 33, 34, 35, 36, 37, 38, 39],
emulating the incoherence of the sources and restricting the estimation to
short separations by accessing only two modes. Recently, multi-plane light
conversion [40] has emerged as a promising technique for estimating
separation, enabling a multimodal approach with the potential to reach the
ultimate sensitivity at any separation. Two recent experiments explored this
approach in different regimes: Boucher et al. [41] with equal-brightness
sources and Santamaria et al. [42] with a strong brightness imbalance. They
demonstrated that this technique is potentially efficient, but did not achieve
an ultra-sensitive separation estimation.
Here, we implement separation estimation of two incoherent equally-bright
sources using spatial-mode demultiplexing over five spatial modes, combined
with intensity measurements (see Figure 1). For bright sources we directly
measure a sensitivity up to five orders of magnitude beyond the Rayleigh
criterion (in practice $20\text{\,}\mathrm{nm}$ sensitivity with
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}$ accuracy for a
$1\text{\,}\mathrm{mm}$ beam size). For faint sources, we show performances
unreachable with even ideal direct imaging (infinite resolution camera, no
noise, equivalent losses) and demonstrate
$20\text{\,}\mathrm{\SIUnitSymbolMicro m}$ precision for a
$1\text{\,}\mathrm{mm}$ beam size and approximately 200 measured photons in
the selected mode. Our experiment is the first practical demonstration of
passive imaging going significantly beyond the Rayleigh limit, using a simple
setup adaptable to standard passive imaging systems and with high-speed
performance.

Figure 1: Spatial-mode demultiplexing for separation estimation. SPADE
consists in decomposing the incident light over the Hermite-Gaussian mode
basis. Measuring the intensity corresponding to each mode was demonstrated to
be an optimal measurement for transverse separation estimation. It gives a
significant advantage with a gain of some orders of magnitude, compared to
direct imaging, on the sensitivity of the estimation of close incoherent
sources imaged through a diffraction-limited optical system. The numbers
presented in this Figure are typical results obtained with our experimental
setup, where $d$ is the transverse separation of two beams in the image plane
and $w_{0}$ is the waist of the beams in the same plane.

## Experimental setup

The experimental setup is detailed in Figure 2. The spatial-mode
demultiplexing system is a multi-plane light converter (MPLC, Proteus-C from
Cailabs). It decomposes an input light beam on the Hermite-Gaussian (HG) mode
basis, each mode being subsequently coupled to a single-mode fiber. It allows
for intensity measurements on several HG modes simultaneously. We use five
MPLC outputs (out of ten) corresponding to the modes HG00, HG01, HG10, HG02
and HG20. At the detection stage we use either photodiodes or single-photon
avalanche-photodiodes depending on the input light flux.

Figure 2: Experimental setup. The two incoherent sources are generated from
one continuous-wave fibered laser. The light is split into two paths that are
modulated independently with two phase modulators (PM). The beams are then
coupled into free space with collimators (CL) fixed either on a translation
stage (TS) or on a motorized translation stage (MTS), which are used to set
the separation between the two beams. The waist at the output of the
collimators is $1.135\text{\,}\mathrm{mm}$. The multiplexed beam is finally
coupled into the multi-plane light converter (MPLC), which intrinsic waist is
around $320\text{\,}\mathrm{\SIUnitSymbolMicro m}$ and the optical powers
corresponding to HG modes (HG00, HG01, HG10, HG02, HG20) are measured with
photodetectors (PD00, PD01…). The reference separation is estimated by
determining the position of each beam on a quadrant detector (QD), when the
other source is turned off. An external photodiode (PD) is used for
normalization in the high-flux regime.

Two incoherent optical sources are generated as follows: the light from a
single fibered-telecom CW laser is split into two paths and goes through
independent electro-optical modulators that apply random phases with high
frequencies (see Methods). The two incoherent guided modes are free-space
coupled by collimators and combined on a beam splitter, thus mimicking the
images of two point sources separated by a set separation. They are mounted on
independent translation stages so that the transverse separation between the
two beams is adjustable. The two beams are Gaussian with similar sizes given
by a waist of $w_{0}\approx$1.135\text{\,}\mathrm{mm}$$. The combined beam is
imaged at the input of the MPLC, and mode-matched to its waist
$w_{1}\approx$320\text{\,}\mathrm{\SIUnitSymbolMicro m}$$. A photodiode
monitors the power stability and half-wave plates combined with a polarizing
beam splitter allow to balance the brightness of the two sources.

The measurement device is calibrated using only one source, with a position
reference that is a quadrant detector in our case. Note that for single source
position estimation the quadrant detector allows for close-to-Cramér-Rao-
bound-limited estimation, and is thus a trustable reference [43]. The single
beam is aligned and centered on the MPLC using the five HG modes intensities.
Usage of all these modes, delivering information on both the centroid and
mode-matching [44], is critical for the robustness and repeatability of the
procedure. To proceed with calibration, the beam is translated in discrete
steps whose position is determined with the quadrant detector (see Methods)
and the HG01-mode output intensity is measured. This mode carries all the
information necessary to estimate small transverse displacements ($\lesssim
w_{0}$) [45], which is the regime we consider in this article. Thanks to this
precise procedure, this calibration can be done once and for all, and used for
every estimation. It becomes a specification of the apparatus that does not
need to be checked on a daily basis.

This calibration curve is used to perform parameter estimation. In our case,
we make the hypothesis that the scene is composed of two identical incoherent
sources and that the centroid is known. This information is used to compute a
”two-source” calibration curve from the symmetrisation of the apparatus
calibration curve (see Methods). It allows to infer the separation between two
sources from the knowledge of the optical power in mode HG01.

Then both optical sources are turned on, the beams are aligned and centered on
the MPLC in an identical manner using the five HG output intensities and the
quadrant detector. We perform a measurement as follows. First, the two beams
are displaced symmetrically by a certain distance, keeping the centroid
unchanged (the symmetry is guaranteed by the quadrant photodiode, but in this
configuration, this photodiode is unable to deliver any information on the
separation – see Methods). Then the optical power at the output of the HG01
fiber is measured over a specific integration time. Finally, the separation is
estimated using the ”two-source” calibration curve. For each optical setting,
this measurement is repeated 200 times in order to evaluate the statistical
error of the measurement (the estimated separation is then the average of the
200 estimations, and the error on this value is given by the statistical
standard deviation).

We estimated several separations in two intensity regimes ($3500$ and
$10^{13}$ detected photons per integration time).

## Low-flux regime

We first present our results for the separation estimation between two faint
sources, where the total incident power on the MPLC is around
$50\text{\,}\mathrm{fW}$ during an integration time of
$100\text{\,}\mathrm{ms}$ (resulting in 3500 detected photons). In Figure 3,
we plot the estimated separation as a function of the reference separation
$d_{\mathrm{ref}}$, for separations going from
$400\text{\,}\mathrm{\SIUnitSymbolMicro m}$ up to
$860\text{\,}\mathrm{\SIUnitSymbolMicro m}$. We see a perfect linear trend and
agreement between the measurement performed with the MPLC and the reference
separation (obtained by measuring independently the position of each source
with the quadrant detector (see Methods).

Figure 3: Low-flux measurements. The separation estimation is realized with
faint sources (3500 photons detected during $100\text{\,}\mathrm{ms}$), using
the mean value of the measured intensity corresponding to HG01 and the
calibration curve. The estimated separations are plotted as a function of the
reference separations determined with the quadrant detector. Both axes are
presented with absolute values and values relative to the size of the beam.
Error bars due to statistical uncertainty on the reference separation and the
estimation, determined with 200 measurements (each during one integration time
of $100\text{\,}\mathrm{ms}$), are displayed as well as the unbiased
estimation line (black line). The quantum Cramér-Rao bound – QCRB – (light
blue) and the Cramér-Rao bound for ideal direct imaging (red) are also plotted
as shaded areas for comparison. In the inset, we plot the sensitivity of the
SPADE measurement as a function of the separation, along with the quantum
Cramér-Rao bound (dashed blue line) and the Cramér-Rao bound for perfect
direct imaging (red line), calculated for the same number of detected photons.

In order to benchmark the performance of the estimation we compute the
statistical standard deviation, as explained in the experimental setup
section, and represent it as error bars on the experimental points in Figure
3. We also plot this measurement sensitivity as a function of the separation
in the inset of this figure. The measured sensitivity is very close to the
quantum Cramér-Rao bound ($19\text{\,}\mathrm{\SIUnitSymbolMicro m}$ when
calculated for the same number of measured photons), the discrepancy arising
from the level of dark counts of the detectors. As a matter of comparison, we
also compute the classical Cramér-Rao bound for separation estimation with
ideal direct imaging, considering infinitely small pixels, no noise, and the
same detector quantum efficiency as for our experiment (see Methods).
Remarkably, our scheme outperforms this idealised DI setting for small
separations ($<$500\text{\,}\mathrm{\SIUnitSymbolMicro m}$$). Furthermore, DI
requires the measurement of all the photons (3500 in our case), increasing the
influence of experimental noise, while modal decomposition allows to route the
photons that carry information into a specific output of the MPLC
(corresponding to the HG01 mode) and perform the detection on only 200 photons
with a single detector to deliver an efficient estimator.

## High-flux regime

We consider now bright optical sources, which correspond to an incident power
on the MPLC of around $650\text{\,}\mathrm{\SIUnitSymbolMicro W}$ for an
integration time of $5\text{\,}\mathrm{ms}$ (or $10^{13}$ detected photons).
Due to the scaling of the sensitivity versus the number of detected photons,
we expect a much better sensitivity in this regime.

Figure 4: High-flux measurements. The separation estimation is realized with
bright sources (${10^{13}}$ photons detected during $5\text{\,}\mathrm{ms}$),
using the mean value of the measured intensity corresponding to HG01 and the
calibration curve. (a) The estimated separations are plotted as a function of
the reference separations determined with the quadrant detector. Both axes are
presented with absolute values and values relative to the size of the beam.
Error bars due to statistical uncertainty on the reference separation and the
estimation, determined with 200 measurements (each during one integration time
of $5\text{\,}\mathrm{ms}$), are displayed as well as the unbiased estimation
line (black line). (b) The quantum Cramér-Rao bound (dashed blue line),
calculated for the same number of detected photons, and the sensitivity for
the SPADE measurement taking into account the detection noise (black line) are
also plotted for comparison. In the inset, we plot the estimated separations
versus the reference separations when performing a differential measurement. A
separation of around $50\text{\,}\mathrm{\SIUnitSymbolMicro m}$ is fixed, and
one source was displaced by several steps, each of approximately
$200\text{\,}\mathrm{nm}$. For each point, 200 measurements (each during one
integration time of $5\text{\,}\mathrm{ms}$) were realized to determine the
statistical errors. The experimental points follow a linear tendency (green
line).

Similarly to the low-flux regime, we plot the estimated separation as a
function of the reference separation in Figure 4(a), for separations ranging
from $20\text{\,}\mathrm{\SIUnitSymbolMicro m}$ to
$160\text{\,}\mathrm{\SIUnitSymbolMicro m}$. Once again, we note a perfect
linear agreement between the measured separations and the reference ones. The
reported error bars are computed from statistical error estimation. We can see
on the inset of Figure 4(a) that while these error bars are barely visible,
the estimated separation deviates on average by approximately
$1\text{\,}\mathrm{\SIUnitSymbolMicro m}$ from the reference separation. This
deviation from unbiased estimation limits the accuracy of our system in the
high-flux regime. This limitation can be traced back to the small differences
between the two sources, because the two-source separation estimator is
constructed from the single-source calibration under the hypothesis that both
sources are identical. In practice the images of the sources have slightly
different spatial shapes, which originates from how we generate them. This
deviation amounts for the $1\text{\,}\mathrm{\SIUnitSymbolMicro m}$ limit (see
Supplementary Materials). This is consistent with such an ultra-sensitive
apparatus and is not due the detection system itself. In realistic scenarios,
like microscopy or astronomy setups, this accuracy value will depend on the
optical imaging device and the type of sources, and can eventually be largely
improved.

To elucidate the full potential of our ultra-sensitive apparatus, we now focus
on the statistical errors. To do so, we perform differential measurements:
from a scene with a given separation between the two sources, for instance
$50\text{\,}\mathrm{\SIUnitSymbolMicro m}$, only one source is displaced by a
series of very small steps, of approximately $200\text{\,}\mathrm{nm}$ each.
At each step, both separation and sensitivity estimation are performed. The
results are displayed in the inset of Figure 4(b), where we plotted the
experimental points along with a linear fit as a guide for the eye. We observe
in this case statistical errors of about $20\text{\,}\mathrm{nm}$, and a
linear trend consistent with the error bars. The slope of the linear fit is
not equal to one, due to the limited accuracy, however, this demonstrates that
our apparatus displays the unprecedented ability to distinguish between two
scenes with a difference in separation of the order of
$20\text{\,}\mathrm{nm}$. Note that some slight deviations from the estimated
separation and the linear fit can be observed, this is to be expected in such
an ultra-sensitive measurement where the actual scene is dependent on any
mechanical or electronic noise (see Methods).

Finally, we plot in Figure 4(b) the sensitivity of source separation
estimation versus the value of the separation and compare it to theoretical
calculations. We demonstrate sensitivities ranging from
$97\text{\,}\mathrm{nm}$ for our shortest separation of
$20\text{\,}\mathrm{\SIUnitSymbolMicro m}$ to as low as
$20\text{\,}\mathrm{nm}$ for larger separations. This corresponds to five
orders of magnitude beyond the beam size. This feature is unique to our
system, whose practicality is ensured by the single source independent
calibration (made possible by the information from multimode MPLC outputs). We
compare these values to the quantum Cramér-Rao bound, which is
$0.4\text{\,}\mathrm{nm}$ for ${10^{13}}$ detected photons. The difference is
quantitatively reproduced by our theoretical model (see Methods) taking into
account the electronic noise of the detection apparatus, demonstrating that
this is the limiting factor in our experiment. Note that this sensitivity
corresponds to $\approx$5\text{\,}\mathrm{nm}$$ at the input plane of the MPLC
due to magnification $1/4$ of the mode-matching telescope.

## Conclusion

We realized a proof-of-principle experiment that demonstrates the
practicability of modal decomposition for sub-diffraction separation
estimation. We achieved a groundbreaking sensitivity in the estimation of the
separation between two incoherent point sources for high and low brightness
–beating the diffraction limit by five orders of magnitude and outperforming
ideal direct imaging, respectively–, thus setting a new standard in optical
resolution. By leveraging the multimode nature of our experiment, we
accomplished a robust calibration that led to this unprecedented level of
sensitivity. Our system is simple to implement and can be adapted to advanced
imaging systems, is fast and requires few detectors thus adaptable to any
input light flux. Notably, reducing noise sources –particularly at the
detection stage– could enhance the separation-estimation sensitivity even
further, eventually reaching the ultimate quantum limit.

Moreover, our singular scheme allows to explore more complex scenes. The
source-phase modulation opens a new avenue for the study of tunable coherence
situations [39]. In addition, an immediate extension of this work is the
exploitation of the MPLC higher-order modes to estimate larger separations
[46], but also for multi-parameter estimations [47, 27], leading to a more
complete scheme of passive imaging [23]. Finally, given the versatility of our
approach and setup, we believe this framework can be further developed for
applications in microscopy and astronomy within the near future.

## Acknowledgments

We thank Manuel Gessner, Claude Fabre, and Valentina Parigi for fruitful
discussions. We thank Eleni Diamanti for providing two single-photon avalanche
photodiodes. This work was partially funded by French ANR under COSMIC project
(ANR-19-ASTR0020-01). This work was supported by the European Union’s Horizon
2020 research and innovation programme under the QuantERA programme through
the project ApresSF. This work was carried out during the tenure of an ERCIM
‘Alain Bensoussan’ Fellowship Programme.

## References

  * Labeyrie  _et al._ [2006] A. Labeyrie, S. G. Lipson, and P. Nisenson,  _An Introduction to Optical Stellar Interferometry_ (Cambridge University Press, 2006) google-Books-ID: l3cSxcRmnu8C.
  * Henning  _et al._ [2014] T. K. Henning, C. P. Dullemond, R. S. Klessen, and H. Beuther,  _Protostars and Planets VI_ (University of Arizona Press, Tucson, 2014).
  * Dekker and Bos [1997] A. J. d. Dekker and A. v. d. Bos, Resolution: a survey, JOSA A 14, 547 (1997), publisher: Optica Publishing Group.
  * Nan  _et al._ [2013] X. Nan, E. A. Collisson, S. Lewis, J. Huang, T. M. Tamgüney, J. T. Liphardt, F. McCormick, J. W. Gray, and S. Chu, Single-molecule superresolution imaging allows quantitative analysis of RAF multimer formation and signaling, Proceedings of the National Academy of Sciences 110, 18519 (2013), publisher: Proceedings of the National Academy of Sciences.
  * Baddeley and Bewersdorf [2018] D. Baddeley and J. Bewersdorf, Biological Insight from Super-Resolution Microscopy: What We Can Learn from Localization-Based Images, Annual Review of Biochemistry 87, 965 (2018).
  * Abbe [1873] E. Abbe, Beiträge zur Theorie des Mikroskops und der mikroskopischen Wahrnehmung, Archiv für Mikroskopische Anatomie 9, 413 (1873).
  * Rayleigh [1879] Rayleigh, XXXI. Investigations in optics, with special reference to the spectroscope, The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science 8, 261 (1879).
  * Sparrow [1916] C. M. Sparrow, On spectroscopic resolving power, Astrophys. J. 44, 76 (1916).
  * Treps  _et al._ [2003] N. Treps, N. Grosse, W. P. Bowen, C. Fabre, H.-A. Bachor, and P. K. Lam, A Quantum Laser Pointer, Science 301, 940 (2003), publisher: American Association for the Advancement of Science.
  * Hell and Wichmann [1994] S. W. Hell and J. Wichmann, Breaking the diffraction resolution limit by stimulated emission: stimulated-emission-depletion fluorescence microscopy, Optics Letters 19, 780 (1994), publisher: Optica Publishing Group.
  * Betzig  _et al._ [2006] E. Betzig, G. H. Patterson, R. Sougrat, O. W. Lindwasser, S. Olenych, J. S. Bonifacino, M. W. Davidson, J. Lippincott-Schwartz, and H. F. Hess, Imaging Intracellular Fluorescent Proteins at Nanometer Resolution, Science 313, 1642 (2006), publisher: American Association for the Advancement of Science.
  * Hell [2007] S. W. Hell, Far-Field Optical Nanoscopy, Science 316, 1153 (2007), publisher: American Association for the Advancement of Science.
  * Dickson  _et al._ [1997] R. M. Dickson, A. B. Cubitt, R. Y. Tsien, and W. E. Moerner, On/off blinking and switching behaviour of single molecules of green fluorescent protein, Nature 388, 355 (1997), number: 6640 Publisher: Nature Publishing Group.
  * Shahram and Milanfar [2006] M. Shahram and P. Milanfar, Statistical and Information-Theoretic Analysis of Resolution in Imaging, IEEE Transactions on Information Theory 52, 3411 (2006).
  * Tsang  _et al._ [2016] M. Tsang, R. Nair, and X.-M. Lu, Quantum Theory of Superresolution for Two Incoherent Optical Point Sources, Physical Review X 6, 031033 (2016).
  * Helstrom [1973] C. Helstrom, Resolution of point sources of light as analyzed by quantum detection theory, IEEE Transactions on Information Theory 19, 389 (1973).
  * Giovannetti  _et al._ [2011] V. Giovannetti, S. Lloyd, and L. Maccone, Advances in quantum metrology, Nature Photonics 5, 222 (2011).
  * Barbieri [2022] M. Barbieri, Optical Quantum Metrology, PRX Quantum 3, 010202 (2022).
  * Gessner  _et al._ [2020] M. Gessner, C. Fabre, and N. Treps, Superresolution Limits from Measurement Crosstalk, Physical Review Letters 125, 100501 (2020).
  * Len  _et al._ [2020] Y. L. Len, C. Datta, M. Parniak, and K. Banaszek, Resolution limits of spatial mode demultiplexing with noisy detection, International Journal of Quantum Information 18, 1941015 (2020).
  * Lupo [2020] C. Lupo, Subwavelength quantum imaging with noisy detectors, Physical Review A 101, 022323 (2020).
  * Sorelli  _et al._ [2021a] G. Sorelli, M. Gessner, M. Walschaers, and N. Treps, Optimal Observables and Estimators for Practical Superresolution Imaging, Physical Review Letters 127, 123604 (2021a).
  * Pushkina  _et al._ [2021] A. Pushkina, G. Maltese, J. Costa-Filho, P. Patel, and A. Lvovsky, Superresolution Linear Optical Imaging in the Far Field, Physical Review Letters 127, 253602 (2021).
  * Bearne  _et al._ [2021] K. K. M. Bearne, Y. Zhou, B. Braverman, J. Yang, S. A. Wadood, A. N. Jordan, A. N. Vamivakas, Z. Shi, and R. W. Boyd, Confocal super-resolution microscopy based on a spatial mode sorter, Optics Express 29, 11784 (2021).
  * Lu  _et al._ [2018] X.-M. Lu, H. Krovi, R. Nair, S. Guha, and J. H. Shapiro, Quantum-optimal detection of one-versus-two incoherent optical sources with arbitrary separation, npj Quantum Information 4, 1 (2018).
  * Grace and Guha [2022] M. R. Grace and S. Guha, Identifying Objects at the Quantum Limit for Superresolution Imaging, Physical Review Letters 129, 180502 (2022).
  * Řehaček  _et al._ [2017a] J. Řehaček, Z. Hradil, B. Stoklasa, M. Paúr, J. Grover, A. Krzic, and L. L. Sánchez-Soto, Multiparameter quantum metrology of incoherent point sources: Towards realistic superresolution, Physical Review A 96, 062107 (2017a).
  * Napoli  _et al._ [2019] C. Napoli, S. Piano, R. Leach, G. Adesso, and T. Tufarelli, Towards Superresolution Surface Metrology: Quantum Estimation of Angular and Axial Separations, Physical Review Letters 122, 140505 (2019).
  * Tsang [2017] M. Tsang, Subdiffraction incoherent optical imaging via spatial-mode demultiplexing, New Journal of Physics 19, 023054 (2017).
  * Nair and Tsang [2016] R. Nair and M. Tsang, Far-Field Superresolution of Thermal Electromagnetic Sources at the Quantum Limit, Physical Review Letters 117, 190801 (2016).
  * Lupo and Pirandola [2016] C. Lupo and S. Pirandola, Ultimate Precision Bound of Quantum and Subwavelength Imaging, Physical Review Letters 117, 190802 (2016).
  * Yang  _et al._ [2016] F. Yang, A. Tashchilina, E. S. Moiseev, C. Simon, and A. I. Lvovsky, Far-field linear optical superresolution via heterodyne detection in a higher-order local oscillator mode, Optica 3, 1148 (2016).
  * Paúr  _et al._ [2016] M. Paúr, B. Stoklasa, Z. Hradil, L. L. Sánchez-Soto, and J. Rehacek, Achieving the ultimate optical resolution, Optica 3, 1144 (2016).
  * Tang  _et al._ [2016] Z. S. Tang, K. Durak, and A. Ling, Fault-tolerant and finite-error localization for point emitters within the diffraction limit, Optics Express 24, 22004 (2016).
  * Tham  _et al._ [2017] W.-K. Tham, H. Ferretti, and A. M. Steinberg, Beating Rayleigh’s Curse by Imaging Using Phase Information, Physical Review Letters 118, 070801 (2017).
  * Zhou  _et al._ [2019] Y. Zhou, J. Yang, J. D. Hassett, S. M. H. Rafsanjani, M. Mirhosseini, A. N. Vamivakas, A. N. Jordan, Z. Shi, and R. W. Boyd, Quantum-limited estimation of the axial separation of two incoherent point sources, Optica 6, 534 (2019).
  * Zanforlin  _et al._ [2022] U. Zanforlin, C. Lupo, P. W. R. Connolly, P. Kok, G. S. Buller, and Z. Huang, Optical quantum super-resolution imaging and hypothesis testing, Nature Communications 13, 5373 (2022).
  * Parniak  _et al._ [2018] M. Parniak, S. Borówka, K. Boroszko, W. Wasilewski, K. Banaszek, and R. Demkowicz-Dobrzański, Beating the Rayleigh Limit Using Two-Photon Interference, Physical Review Letters 121, 250503 (2018).
  * Wadood  _et al._ [2021] S. A. Wadood, K. Liang, Y. Zhou, J. Yang, M. A. Alonso, X.-F. Qian, T. Malhotra, S. M. H. Rafsanjani, A. N. Jordan, R. W. Boyd, and A. N. Vamivakas, Experimental demonstration of superresolution of partially coherent light sources using parity sorting, Optics Express 29, 22034 (2021).
  * Morizur  _et al._ [2010] J.-F. Morizur, L. Nicholls, P. Jian, S. Armstrong, N. Treps, B. Hage, M. Hsu, W. Bowen, J. Janousek, and H.-A. Bachor, Programmable unitary spatial mode manipulation, Journal of the Optical Society of America A 27, 2524 (2010).
  * Boucher  _et al._ [2020] P. Boucher, G. Labroille, C. Fabre, and N. Treps, Spatial optical mode demultiplexing as a practical tool for optimal transverse distance estimation, Optica 7, 1621 (2020).
  * Santamaria  _et al._ [2023] L. Santamaria, D. Pallotti, M. S. de Cumis, D. Dequal, and C. Lupo, Spatial-mode-demultiplexing for enhanced intensity and distance measurement, arXiv:2206.05246 (2023).
  * Fabre  _et al._ [2000] C. Fabre, J. B. Fouet, and A. Maıtre, Quantum limits in the measurement of very small displacements in optical images, Optics Letters 25, 3 (2000).
  * Chille  _et al._ [2015] V. Chille, P. Banzer, A. Aiello, G. Leuchs, C. Marquardt, N. Treps, and C. Fabre, Quantum uncertainty in the beam width of spatial optical modes, Optics Express 23, 32777 (2015).
  * Delaubert  _et al._ [2006] V. Delaubert, N. Treps, M. Lassen, C. C. Harb, C. Fabre, P. K. Lam, and H.-A. Bachor, TEM 10 homodyne detection as an optimal small-displacement and tilt-measurement scheme, Physical Review A 74, 053823 (2006).
  * Sorelli  _et al._ [2021b] G. Sorelli, M. Gessner, M. Walschaers, and N. Treps, Moment-based superresolution: Formalism and applications, Physical Review A 104, 033515 (2021b).
  * Řehaček  _et al._ [2017b] J. Řehaček, M. Paúr, B. Stoklasa, Z. Hradil, and L. L. Sánchez-Soto, Optimal measurements for resolution beyond the Rayleigh limit, Optics Letters 42, 231 (2017b).

##  Appendix A Methods

Generation of the two sources The two incoherent beams are generated from a
continuous-wave fibered laser at $1550\text{\,}\mathrm{nm}$ (Thorlabs
SFL1550P, coherence length $6\text{\,}\mathrm{km}$) that is split into two
paths. Each path is then phase modulated independently using electro-optical
phase modulators (iXblue MPX-LN-0.1). These modulators are driven with
independent FPGA hardware boards (Red Pitaya STEMlab 125-14) generating
voltages uniformly distributed between -1V and 1V, which are then amplified to
reach the voltage amplitude needed for covering [$-\pi$,$\pi$], at
$4\text{\,}\mathrm{MHz}$. The incoherence of the sources is assessed by the
vanishing of the interference fringes (see Supplementary Materials). This
configuration allows to have access to a tunable coherence between the two
sources if needed for future experiments.

Detection The spatial-mode demultiplexer (Cailabs Proteus-C) has ten outputs
but we only use five of them (corresponding to HG00, HG01, HG10, HG02 and
HG20) as we restrict ourselves to the estimation of short separations, i.e.
shorter than the beam radius $w_{0}$. The MPLC presents -3.9 dB (59%) of
losses. At the fiber outputs, photodetection is realized either with
switchable-gain photodiodes (Thorlabs PDA50B2, 85% responsivity) –using a gain
of 60 dB (bandwidth $630\text{\,}\mathrm{H}\mathrm{z}$) for the HG01 output–
or with single-photon avalanche-photodiodes (IDQuantique ID230, 25% quantum
efficiency).
For the high-flux measurements (around $100\text{\,}\mathrm{\SIUnitSymbolMicro
W}$ entering the MPLC), the integration time used for one measurement is
$5\text{\,}\mathrm{ms}$. This detection timescale ensures that we average
enough to measure incoherent beams. We normalize the optical powers at the
outputs of the MPLC with about 10% of the total incident light, measured with
an external photodiode (Thorlabs PDA20CS-EC). Data are acquired with two
oscilloscopes (Tektronix 4 series MSO44) at a sample rate of
$25\text{\,}\mathrm{kHz}$ for a total measurement time of
$10\text{\,}\mathrm{s}$. Due to the high sensitivity of high-flux
measurements, we are extremely sensitive to external noises such as mechanical
noise, air flow and fluctuations in detector offsets. In order to dissociate
from these noise sources, we chose to determine the statistical errors over
only $1\text{\,}\mathrm{s}$, which corresponds to
$\mathrm{2}\mathrm{0}\mathrm{0}$ consecutive points (see Supplementary
Materials).
For the low-flux regime (incident optical power around
$50\text{\,}\mathrm{fW}$), the integration time used for one measurement is
$100\text{\,}\mathrm{ms}$ to collect enough data for statistical analysis. In
this case, as we have only two single-photon detectors, we normalize the
optical power of the HG01 mode with the one measured at the output of the HG00
mode. The total measurement time with faint sources is
$20\text{\,}\mathrm{s}$, resulting in 200 points.

Reference separation We estimate the displacement of each source using a
quadrant detector (Thorlabs PDQ30C) signal. This detector is composed of two
photodiodes and the difference between their output currents gives access to
the displacement of the beam. Remarkably, this means that the quadrant
detector is blind in the presence of two symmetric sources and it cannot be
used for estimating a separation. During the calibration, we use it to
determine the displacement of the source with respect to the centroid of the
MPLC –on which the quadrant detector is aligned. Then, during the measurement,
we measure the displacement of each source while the other is turned off and
we then extract the reference separation $d_{\mathrm{ref}}$ and the associated
statistical error bars ranging from $20\text{\,}\mathrm{nm}$ to
$50\text{\,}\mathrm{nm}$.

Alignment procedure To ensure high accuracy and sensitivity, it is critical to
align the two beams identically. We use bright beams to this end. First, each
beam is roughly centered on the MPLC by maximizing the signal in the HG00
mode. Then, both beams are mode-matched on the MPLC: the focal lengths and the
positions of the lenses are chosen to minimize the signals in the HG02 and
HG20 modes. Finally, we minimize the HG01 and HG10 signals with a pair of
steering mirrors –per beam– in order to improve sensitivity on the position
and tilt of the beams at the input of the MPLC. The quadrant detector is used
to ensure that they are aligned on the same center point.

Calibration and estimation After the two beams are aligned, we use only one to
perform the calibration of the setup ensuring that calibration and source-
separation measurement are independent. The beam is displaced with respect to
the MPLC center with a motorized translation stage. We cover a position range
from $-100\text{\,}\mathrm{\SIUnitSymbolMicro m}$ to
$100\text{\,}\mathrm{\SIUnitSymbolMicro m}$ around the MPLC center, in steps
of $6\text{\,}\mathrm{\SIUnitSymbolMicro m}$. The optical power corresponding
to the HG01 mode is measured during $10\text{\,}\mathrm{s}$ –with a photodiode
or a single-photon avalanche photodiode depending on the flux– for every
position. Its mean value is normalized with the mean value of the optical
power measured by the external photodiode and in the HG00 mode for bright and
faint sources, respectively. The calibration curve for one source $f(x)$ is
obtained from a polynomial fit to the experimental points.
For each set separation, we measure the normalizing optical power $p_{1,2}$
and the quadrant-detector signal $x_{1,2}$ corresponding to each source. We
can then determine the reference separation value ${d_{\mathrm{ref}}=\mid
x_{1}-x_{2}\mid}$ and the two-source calibration curve for the separation
${g(d=2x)=\frac{p_{1}}{p_{1}+p_{2}}f(x)+\frac{p_{2}}{p_{1}+p_{2}}f(-x)}$. This
allows to take into account relative power changes due to polarization
fluctuations in the independent fiber paths.
Finally, to extract the estimated separation and its associated Finally, to
estimate the separation, we average measured optical power in the HG01 mode
during the integration time ($5\text{\,}\mathrm{ms}$ for the high flux or
$100\text{\,}\mathrm{ms}$ for the low flux) obtaining $p^{i}_{01}$. We
normalize the result with total power $p^{i}_{\mathrm{tot}}$, measured either
from the HG00 mode for low flux or from the external photodiode for high flux.
We repeat the procedure 200 times, the estimation $d$ and its sensitivity
$\Delta d$ is then obtained by determining the mean value and the standard
deviation over all the $d_{i}$, extracted by inverting the two-source
calibration curve for each bin $i$. We assess the accuracy of our estimation
by comparing it to the reference separation $d_{\mathrm{ref}}$ obtained with
the quadrature detector.

Theoretical sensitivity We estimate the sensitivity of ideal direct imaging
using the Fisher information. Considering continuous noiseless detection of
the intensity distribution the Fisher information per unit time takes the form

$F=\frac{1}{E_{\text{ph}}}\int\frac{1}{I(\vec{r})}\left(\frac{\partial
I(\vec{r})}{\partial d}\right)^{2}d\vec{r},$ (1)

where $E_{\text{ph}}$ is the energy of the photon. The intensity distribution
$I(\vec{r})$ for two incoherent sources equals
$I(x,y)=P_{1}\leavevmode\nobreak\ u^{2}(x+d/2,y)+P_{2}\leavevmode\nobreak\
u^{2}(x-d/2,y)$, where $P_{1,2}$ are the powers of the sources and $u(x,y)$ is
a Gaussian mode profile. Then the smallest possible variance of an estimator
of the separation between the sources is given by the Cramér-Rao bound
$\Delta^{2}d\geq 1/(F\leavevmode\nobreak\ t_{\text{int}})$, where
$t_{\text{int}}$ is an integration time. This bound is shown in Figure 3 in
red.

The quantum Cramér-Rao bound (Figure 3 and Figure 4(b) in blue) is given by
the inequality [15] $\Delta^{2}d\geq w_{0}^{2}/N$, where
$N=(P_{1}+P_{2})\leavevmode\nobreak\ t_{\text{int}}/E_{\text{ph}}$ is the
total number of detected photons.

To estimate the sensitivity of mode demultiplexing (Figure 4(b), black line)
we use the error propagation method. Since we build our estimator based on the
measured intensity $I_{01}$ of the mode HG01, its variance equals

$\Delta^{2}d=\frac{\Delta^{2}I_{01}}{\left(\partial I_{01}/\partial
d\right)^{2}}.$ (2)

The intensity noise $\Delta^{2}I_{01}$ consists of the shot noise and the
detection noise. The latter is dominant in the high-flux regime when intensity
is detected with photodiodes. The detection noise is estimated from the offset
measurements (without incident light). The intensity of HG01 mode is given by
[22]

$I_{01}=(P_{1}+P_{2})\frac{d^{2}}{4w_{0}^{2}}e^{-\frac{d^{2}}{4w_{0}^{2}}}.$
(3)

In practice measurement modes are slightly different from Hermite-Gaussian,
which is often referred to as cross-talk between modes. However, analysis
shows that the effect of the cross-talk in our case is negligible compared to
the detection noise.

##  Appendix B Calibration

One-source calibration. One beam is displaced, using a motorized translation
stage, from $-100\text{\,}\mathrm{\SIUnitSymbolMicro m}$ to
$100\text{\,}\mathrm{\SIUnitSymbolMicro m}$, by steps of
$6\text{\,}\mathrm{\SIUnitSymbolMicro m}$, with respect to the MPLC center.
The optical power corresponding to the HG01 mode is measured for every
position, for $10\text{\,}\mathrm{s}$, and its mean value is normalized with
the mean value of the optical power measured by the external photodiode or in
the HG00 for bright and faint sources respectively. The experimental points
are then fit with a polynomial function which is the calibration curve for one
source $f(x)$.

Figure 5: Calibration curve for one displaced beam. The light blue line is a
fit of the experimental points with a 6th-order polynomial function. Figure 6:
Calibration curve for the separation between two beams. This curve is obtained
by symmetrizing the light polynomial function from the one-source calibration
curve.

Two-source calibration curve. For each separation, we measure the normalizing
optical power $p_{1,2}$, as well as the quadrant detector signal $x_{1,2}$,
for each source. We can then determine the reference separation value
${d_{\mathrm{ref}}=\mid x_{1}-x_{2}\mid}$ and the symmetrized calibration
curve for the separation between two sources
${g(d=2x)=\frac{p_{1}}{p_{1}+p_{2}}f(x)+\frac{p_{2}}{p_{1}+p_{2}}f(-x)}$. This
allows to take into account relative power changes due to polarization
fluctuations in the independent fiber paths.

Separation estimation. Finally, to estimate the separation, we average
measured optical power in the HG01 mode during the integration time
($5\text{\,}\mathrm{ms}$ for the high flux or $100\text{\,}\mathrm{ms}$ for
the low flux) obtaining $p^{i}_{01}$. We normalize the result with total power
$p^{i}_{\mathrm{tot}}$, measured either from the HG00 mode for low flux or
from the external photodiode for high flux. To estimate the separation $d$ we
solve the equation

$\frac{p^{i}_{01}}{p^{i}_{\mathrm{tot}}}=g(d_{i}).$ (4)

We repeat the procedure 200 times, the estimation $d$ and its sensitivity
$\Delta d$ is then obtained by determining the mean value and the standard
deviation over all the $d_{i}$, and this can be compared to the reference
separation $d_{\mathrm{ref}}$.

##  Appendix C Indistinguishability of the two sources

The central assumption of the separation estimation we implemented is that the
two beams are identical. To test this assumption, we performed a calibration
for each source. Figure 7 presents typical calibration curves for the HG01
mode of each beam.

Figure 7: Calibration curves of the two sources for the mode HG01.

We observe that the two curves are slightly shifted horizontally and
vertically from each other. This originates from different tilts and positions
of the beams at the input of the MPLC, as well as from different mode shapes,
since, in our setup, the two modes are prepared independently. These
disparities are the main source of systematic errors. Although this has a
small influence on the estimation for symmetric situations, it results in more
striking effects when performing differential measurements. Indeed, since one
beam remains fixed, this asymmetry in position, combined with the differences
in the calibration curves, leads to variations in the slope of the linear
tendency (inset plot in Figure 4(b)), depending on the range of separations
that are considered. Note, that the trend always remains linear and
measurement results stay incredibly sensitive to small displacements of any
source.

##  Appendix D Estimation in the high-flux regime: temporal signals

As explained in the Methods section, the total measurement time for the high-
flux measurements is $10\text{\,}\mathrm{s}$. Figure 8 presents a typical
temporal signal acquired from the photodiode of the HG01 output, here for a
transverse separation between the two beams of
$120\text{\,}\mathrm{\SIUnitSymbolMicro m}$.

Figure 8: Temporal signal at the output of the HG01 mode, acquired with a
photodiode (Thorlabs PDA50B2, gain 60dB, integration time
$5\text{\,}\mathrm{ms}$) for a transverse separation of around
$120\text{\,}\mathrm{\SIUnitSymbolMicro m}$ between the two beams.

Fluctuations arise from offsets variations, air flows and imperfect
incoherence between the two sources, resulting in an increase in the standard
deviation of the signal. We chose to reduce the total measurement time to
$1\text{\,}\mathrm{s}$, with an integration time of $5\text{\,}\mathrm{ms}$,
to reduce the contributions of these low-frequency noise sources while keeping
a sufficient number of points for the statistical analysis.

##  Appendix E Incoherent sources

We generate two incoherent sources from a single laser using two independent
phase modulators (see Methods). The vanishing of the interference patterns
assesses the incoherence of the beams. Figure 9 displays images of the two
beams after mixing on a beam splitter, without the modulation and with the
random modulation, uniformly distributed between $-\pi$ and $\pi$ and with a 4
MHz bandwidth. The images are acquired with a CCD camera (Gentec
Beamage-3.0-IR, bandwidth $8\text{\,}\mathrm{Hz}$).

Figure 9: Images of the two beams after mixing on a beam splitter with a CCD
camera, without modulation (left) and with a random phase-modulation uniformly
distributed between $-\pi$ and $\pi$ (right).

Figure 10 shows temporal signals acquired with a photodiode (Thorlabs PDA50B2,
gain 20dB, integration time $5\text{\,}\mathrm{ms}$) after the beam splitter,
without (light blue) and with (dark blue) modulation.

Figure 10: Temporal signals of the two beams after mixing on a beam splitter
without (light blue) and with (dark blue) modulation, acquired with a
photodiode (Thorlabs PDA50B2, gain 20dB, integration time
$5\text{\,}\mathrm{ms}$).