# Foundation Models for the Digital Twin Creation of Cyber-Physical Systems
††thanks: All authors contributed equally.

Shaukat Ali
Simula Research Laboratory and
Oslo Metropolitan University
Oslo, Norway
<EMAIL_ADDRESS>
Paolo Arcaini
National Institute of Informatics
Tokyo, Japan
<EMAIL_ADDRESS>
Aitor Arrieta
Mondragon University
Mondragon, Spain
<EMAIL_ADDRESS>

###### Abstract

Foundation models are trained on a large amount of data to learn generic
patterns. Consequently, these models can be used and fine-tuned for various
purposes. Naturally, studying such models’ use in the context of digital twins
for cyber-physical systems (CPSs) is a relevant area of investigation. To this
end, we provide perspectives on various aspects within the context of
developing digital twins for CPSs, where foundation models can be used to
increase the efficiency of creating digital twins, improve the effectiveness
of the capabilities they provide, and used as specialized fine-tuned
foundation models acting as digital twins themselves. We also discuss
challenges in using foundation models in a more generic context. We use the
case of an autonomous driving system as a representative CPS to give examples.
Finally, we provide discussions and open research directions that we believe
are valuable for the digital twin community.

_K_ eywords Cyber-Physical Systems $\cdot$ Digital Twins $\cdot$ Cyber-
Physical Systems $\cdot$ Large-Language Models

##  1 Introduction

An increasing number of foundation models are being released, trained on vast
amounts of data [1]. Some models are trained for a specific modality (e.g.,
text, audio, video), whereas others are trained on multi-modal data (e.g., a
mix of text, audio, and video). Given that these models have learned generic
patterns, there is an increasing interest to tailor these models for specific
application contexts [2], such as code [3] or model [4] generation.

Digital twins (i.e., virtual replica [5]) for Cyber-Physical Systems (CPSs)
have been built with various methods, such as with the application of model-
based systems engineering approaches [6] and machine learning techniques [7].
In both cases, significant manual effort is required to create digital twins.
With the emergence of powerful foundation models, it is natural to investigate
whether such models can reduce the manual effort and the reliance on human
expertise required to create digital twins. Such an investigation is the main
objective of this paper.

Building on our previous conceptual work [8, 9] focusing on digital twins for
CPSs, we provide different perspectives on using foundation models for
creating digital twin models and the capabilities they provide. In particular,
we discuss foundation models used from two perspectives. First, we discuss
various possibilities for using foundation models to generate digital twin
models (e.g., in SysML and Open Modelica) and capabilities of digital twins
(e.g., in simulation models or machine learning models). Second, we discuss
the direct use of fine-tuned foundation models as the digital twin as a whole,
which does not require any explicit digital twin model. In addition, we use
autonomous driving as the CPS domain to illustrate possible solutions with
fine-tuned foundation models. From each perspective, we provide detailed
challenges and research opportunities that serve as a starting point to build
a research agenda for using foundation models for CPSs, which can be extended
to build digital twins for other systems (e.g., biological systems).

The rest of the paper is organized as follows: We start with relevant
background in Section 2. Next, in Section 3, we provide the overall context of
the use of foundation models in our case, followed by a discussion of the two
perspectives in Section 4 and Section 5. Finally, we provide discussions in
Section 6 and conclude the paper in Section 7.

##  2 Background

In this section, we provide a brief background on foundation models in Section
2.1, followed by introducing the example case of developing digital twins for
Autonomous Driving Systems (ADSs) in Section 2.2.

###  2.1 Foundation Models

Foundation models have recently grabbed the attention of scientists,
industries of all scales, public sectors, and even non-technical people such
as artists [10]. Such a model is a powerful machine learning model trained on
a large amount of data, which has learned generic patterns. Consequently, for
specific applications, foundation models need to be adapted (e.g., with fine-
tuning , retrieval augmented generation (RAG) or prompt engineering) to
improve their performance for the applications. Since these models are general
models, their potential applications are even unknown by their developers. As
a result, several opportunities exist to investigate the potential
applications of these foundation models across different domains. Several
applications exist across different domains, such as code generation in
software engineering, various types of content generation, summarization, and
other computer vision applications [1, 10, 11].

Different classes of foundation models exist, e.g., depending on the data on
which these are trained. For example, large-language models (LLMs) are trained
on vast amounts of textual data. Their examples include GPT-4, BERT, and
Llama. Similarly, large vision models are another type of foundation models
for computer vision, including CLIP [12] and LandingLens [13]. In addition,
multimodal foundation models are getting more popular, which are trained on
multiple data modalities such as text, images, and video [14]. Examples
include Fuyu-8B [15] and ALIGN [16].

###  2.2 Example Case: Autonomous Driving Systems (ADSs)

ADSs are key parts of self-driving cars responsible for understanding
environmental conditions, processing these conditions, and controlling the
cars. To achieve such functionalities, ADSs rely on various sensors (e.g.,
LiDAR) to obtain environmental conditions, such as road and weather. Once such
data about environmental conditions are obtained, ADSs process them and
accordingly devise a driving plan, followed by controlling the vehicle to
implement the plan. Due to safety considerations, lots of research is ongoing
on making ADSs dependable, e.g., with automated testing [17]. Digital twins
are also being investigated for ADSs for various purposes, e.g., running
advanced simulations and testing [18]. During the development of ADSs and its
corresponding digital twins, foundation models could potentially help address
several challenges. Examples include generating realistic environmental
conditions for simulation (e.g., weather conditions), extracting realistic
test cases from documented accident reports, and assisting developers in
building digital twin models. Addressing such challenges could lead to various
benefits, such as developing realistic ADS digital twins, decreasing the
development cost of digital twins and ADSs, and improving their dependability.

##  3 Overview: Application Contexts of Foundation Models for CPS Digital
Twins

In our previous conceptual works [8, 9], we defined the digital twin of a CPS
(i.e., physical twin) as consisting of two main components, i.e., the digital
twin model and the digital twin capability, as shown in Figure 1.

Figure 1: Digital Twins for Cyber-Physical Systems

In this case, the model refers to a virtual replica or representation of the
CPS, which simulates the CPS behavior. Such a model can be built with various
approaches, such as Open Modelica [19], SysML [20], or other simulation
platforms, depending on the underlying design and development of the
corresponding physical twin [21]. The digital twin capability refers to
various functionalities of the digital twin, such as timely prediction of CPS
anomalies before they actually happen [22, 23, 24]. A digital twin receives
data from its corresponding physical twin, i.e., a CPS in our context. With
the data, the digital twin can update its digital twin model and perform its
intended functionalities facilitated by the digital twin model.

Based on the conceptual model described above, we identified two main cases in
which foundation models can be used in the context of digital twins for CPSs
to reduce their current purely manual development efforts.

##  4 Case 1: Foundation Models to Generate Digital Twins

We will discuss the first case, in which foundation models are used to
generate digital twins (digital twin model, digital twin capability, or both)
of a CPS. First, we will discuss state of the art related to this case, where
AI techniques, in general, have been used to generate some aspect of digital
twins of CPSs in Section 4.1. Next, in Section 4.2, we will discuss a generic
solution to generate digital twins of CPSs, followed by discussing challenges
and opportunities related to generating digital twins with foundation models
in Section 4.3. Finally, we present some examples using the case study of ADSs
in Section 4.4.

###  4.1 State of the Art

Since, in this case, we generate a digital twin model and a digital twin
capability, we will discuss the state of the art related to these two aspects
in the following.

Regarding generating a digital twin model, we used foundation models to
generate such models, e.g., for stimulating a CPS. Depending on the CPS
domain, such models could be developed using different formalisms, such as
SysML and Open Modelica. To this end, manually creating digital twins with
SysML models has been explored in the literature [25, 26]. Similarly, Open
Modelica has also been used to create digital twins of CPSs [19]. In addition,
other model-driven approaches have been used to create digital twins of CPSs
in various notations [6]. However, a study [27] on using generative AI for
modeling tasks revealed that the quality of current LLMs is quite poor for
such modeling tasks. This suggests that generating digital twin models with
foundation models is still an early area of research.

Concerning digital twin capability, various AI methods have been used to build
various capabilities such as anomaly detection [22, 28], waiting time
prediction for elevators’ performance [29], and parameter prediction in
manufacturing [30]. In our context, using foundation models to generate AI-
based digital twin capability, including model structure, parameters, weights,
etc., is a research area that needs further investigation. In addition,
digital twin capabilities have also been built with traditional methods such
as model-based simulation methods [31]. To this end, generative AI [27]
techniques need improvement and tailoring for this purpose, consequently
requiring further investigation.

In summary, the current applications of generative AI techniques for
developing models for digital twins are immature due to a lack of methods and
tools to build specialized foundation models or fine-tune them to create
digital twin models based on existing popular modeling languages for digital
twins. Moreover, to generate AI-based capabilities, there is also a lack of
methods and automated tools to create various model structures and their
properties (e.g., weights).

###  4.2 Solution Proposal

In this case (see Figure 2), foundation models can be used to generate 1) the
entire digital twin, i.e., the model and capability, 2) only the digital twin
model, and 3) only the digital twin capability.

Figure 2: Foundation models to generate a digital twin: (1) both the digital
twin model and digital twin capability, (2) only digital twin model, (3) only
digital twin capability

Generally speaking, generic foundation models must be adapted to generate
digital twin models and capabilities, e.g., fine-tuning parameters or prompt-
tuning with the Tuner component shown in Figure 2. In addition, a Tuner could
be implemented with advanced machine learning techniques such as transfer
learning. Moreover, to generate a more realistic digital twin, tailoring the
foundation model will involve fine-tuning with CPS domain knowledge and the
specifications of the specific CPS for which the digital twin is being
generated. During the operation, data received from the real operation of CPS
can be used further to update both the digital twin of the CPS and the
foundation models with Tuner. Below, we discuss the solutions concerning model
generation and capability generation with foundation models.

####  4.2.1 Digital Twin Model

When generating digital twin models with foundation models, we generate
simulation models in various formalisms to assist in creating digital twin
models.

Different degrees of automation are possible for this purpose, which can be
supported by foundation models.

First, foundation models can be used as a recommendation system, meaning that
these models can help suggest different model elements when a user manually
creates a simulation model. Various foundation models can be used depending on
the type of CPS digital twin. For instance, if a simulation model is created
in Open Modelica or SysML, LLMs are suitable, similar to their use in code
generation. Depending on the formalism used, specific LLMs must be trained as
a recommendation system as part of the modeling or simulation software. Other
examples include creating a digital twin of the environment of a CPS, which
could be a simple model, a digital shadow, or a combination of both. For
example, in the case of a model, the environment could be represented as a
video. To this end, video-based foundation models can recommend various new
elements inside the digital twin. In the case of digital shadow, an initial
model could be built with model-based approaches. Next, the model can
continuously reflect the current state of the environment based on the data
about the environment collected through the sensors.

Second, foundation models could be used as digital assistants in the form of
chatbots to help create digital twin models. Compared to the recommendation
system, this is a more interactive use of the foundation model, where a user
interacts with the chatbot to create simulation models. The level of
automation can vary. For instance, in simpler use, a user can ask for various
suggestions, and the chatbot responds to them. In a more complex use, a user
can even ask a chatbot to create model elements in a simulation model, the
chatbot creates them, and the user confirms them.

Third, a more advanced use of foundation models would be to generate
simulation models by providing the foundation models with a CPS specification,
CPS models built during the design and development of CPS, and domain
knowledge of CPS. Specifically, after an initial version of simulation models
is generated classically, those can be further improved with interaction with
foundation models.

####  4.2.2 Digital Twin Capability

Regarding building digital twin capabilities, we deal with generating a
digital twin capability with foundation models.

To this end, foundation models could help in various ways, depending on how
the capability is built. For instance, if the capability is built in a
specific programming language, then quite a few specialized LLMs could be
used. An interested reader is referred to the following reference for more
details [32]. If the digital twin capability is built with simulation models
similar to those described in the previous section, then a similar discussion
applies here.

Increasingly, digital twin capability is also being built with machine
learning models such as neural network models (e.g., [33, 28]). Foundation
models could be used to create machine learning models to help create such
machine learning models. LLMs could be used to create model structures, given
that such models potentially have learned from many publicly available models.
On the other hand, there is also the possibility that machine learning could
be iteratively designed using LLMs as chatbots with whom a user interacts.

###  4.3 Challenges and Opportunities

Several challenges exist for the cases discussed in the previous section that
provide research opportunities. Regarding generating digital twin models,
there is not much work on generating models or assistance in general with
various types of models that could be used for simulation. This is mainly
because fewer open-source models (e.g., Simulink or Open Modelica) are
available to train foundation models, e.g., compared to open-source code;
thereby, more LLMs exist for generating the code. This would mean that
research opportunities exist to adapt foundation models at three levels.
First, we must adapt foundation models for each formalism, such as Open
Modelica, Simulink, and SysML. Second, such adapted foundation models must be
adapted to certain domains based on domain knowledge, for instance, maritime
and oil & gas. At this level, tuning foundation models with domain-specific
ontologies could help. Third, even after adapting such foundation models for
each formalism and domain, such models further need to be fine-tuned for each
specific CPS (e.g., with system specification or even existing CPS models that
were created during the design of the CPS) for which the digital twin is being
built. This is mainly because each CPS is built for a specific purpose, even
within the same domain. Such tailored foundation models can support the
various usages discussed in Section 4.2, i.e., model element recommenders,
modeling chatbots, and digital twin generators.

In addition, we see some research opportunities for automated translation of
digital twin models from one formalism to another with foundation models. This
would help in cases where fewer models are available for training; thereby,
models from various formalisms could be translated into one unified formalism
and used to train foundation models for generating digital twins.

Regarding digital twin capability generation, once again, various
opportunities exist. If the capability is built with simulation models, then
the discussion related to digital twin models in this section is still
applicable. Moreover, research opportunities exist to develop specialized
chatbots that can assist a user in building machine-learning models-based
digital twin capability.

###  4.4 Example Case of ADS

Foundation models can be used from various perspectives in autonomous driving
systems digital twins.

First, foundation models can automatically generate simulation models for the
environment of the ADSs. We argue that foundation models have learned lots of
public knowledge, e.g., about driving behaviors, accidents, weather
conditions, human behaviors, and rules and regulations that can enable them to
generate digital twins of the environment of the ADSs. Thus, such
environmental digital twins could be integrated, for instance, with a
particular autonomous driving algorithm and vehicle for the Software in the
Loop (SiL) simulation. Nonetheless, this requires integration work with a
particular ADS simulator being used. For example, environment digital twins
can be used as time series data that can be input as part of an existing ADS
simulator to generate a digital twin. For instance, LLMs could be relevant to
this end. On the other hand, digital twins could be generated as videos that
can be integrated into ADS simulators to mimic the environment. For such
digital twins, foundation models that generate videos are relevant.

Second, foundation models can also support various digital twins’ capabilities
for ADSs built with classical methods. One such key functionality is to assess
the realism of scenarios generated by digital twins, for example, to support
the testing of ADSs as one of its key functionalities. To this end, a recent
work [34] evaluates the capability of commonly used LLMs, i.e., GPT, Llama,
and Mistral, to assess the realism of scenarios generated by, e.g.,
reinforcement learning in the context of testing ADSs. Thus, such kind of
realism assessment must be an integral part of digital twins of ADSs, either
those digital twins generated by foundation models or only such assessment is
implemented as part of digital twin implemented in other ways (e.g., with
manual modeling and simulation approaches). Concerning this existing work
[34], it only assessed the realism of textual scenarios. Therefore, several
opportunities exist to determine the realism of scenarios generated as videos,
images, or audio as part of digital twins of ADSs.

##  5 Case 2: Fine-tuned Foundation Model as Digital Twin

This is the advanced case, where a foundation model would serve as the
backbone for the digital twin model only, the digital twin capability only, or
the whole digital twin (see Figure 3).

Figure 3: Foundation model as a whole digital twin, as a digital twin model,
or as digital twin capability

Such foundation model, i.e., the digital twin of the physical twin, can be for
the digital twin model, digital twin capability, or the whole digital twin. To
this end, foundation models must be fine-tuned with CPS specifications and
other domain knowledge. Such fine-tuning can be done, for example, with prompt
engineering or with advanced machine learning methods, e.g., transfer learning
and few-shot learning. In addition, with more operational data coming from the
physical twin, the digital twin (i.e., the foundation model) needs to be fine-
tuned continuously.

###  5.1 State of the Art

Since Case 2 aims to construct a fine-tuned foundation model to play the role
of the digital twin, in this section, we discuss the state of the art of
developing fine-tuned foundation models for achieving relevant digital twin
capabilities.

A comprehensive survey of LLMs and their abilities is presented in [35]. The
survey reports that the representative abilities of LLMs include _language
generation_ such as code synthesis, _knowledge utilization_ such as knowledge
completion, _complex reasoning_ such as symbolic reasoning, _tool
manipulation_ such as search engine and interaction with external environment.
The directly relevant one is the interaction with the external environment,
where LLMs can obtain feedback from the external environment and perform
actions according to instructions in natural language, for instance, to
manipulate agents. For example, VirtualHome, presented in [36], is a virtual
environment for simulating household tasks such as making coffee and cleaning,
in which agents can execute natural language actions generated by LLMs.

LLMs can also enable the creation of believable agents, which is useful for
realizing certain digital twins that simulate human behaviors and their
interactions with CPSs. For example, Part et al. [37] presented an agent
architecture for generating believable behavior using LLMs. Moreover, Xi et
al. [38] presented a comprehensive survey on LLM-based agents and explored
their applications in single-agent scenarios, multi-agent scenarios, and
human-agent cooperation scenarios. Particularly, Yang et al. [39] proposed an
LLM-based digital twin to simulate human activities in the shopping mall to
facilitate efficient training of a reinforcement learning algorithm to
optimize user-preference-based temperature control. Moreover, Wang et al. [40]
proposed TWIN-GPT, which is fine-tuned on a pre-trained LLM on clinical trial
datasets, and aims to generate personalized digital twins for different
patients.

LLMs can also be integrated with external knowledge bases or structured data
sources to enhance their understanding and reasoning capabilities. Using
external knowledge, an LLM-based digital twin can access domain-specific
information by interacting with the physical twin. In the literature, some
approaches have already been proposed to integrate LLMs with external
information. For example, Jin et al. [41] proposed to integrate health reports
and medical knowledge into a large model to ask relevant questions for the
prediction of diseases.

In summary, there is a lack of methods and tools to develop virtual
environments and agents with LLMs for constructing digital twins. Moreover,
there is a lack of methods to exploit the LLM’s interaction with the external
environment and generate accurate actions, which could potentially enable the
use of foundation models for developing digital twins that interact with their
physical twin counterparts.

###  5.2 Solution Proposal

Foundation model being the digital twin model only. In this scenario, a fine-
tuned foundation model serves as the digital twin model while the digital twin
capability is implemented with other ways. The fine-tuned foundation model is
mainly about capturing the domain knowledge and understanding CPS behavior,
while components implementing the digital twin capabilities handle the real-
time interactions with the CPS. The capability components leverage the
information provided by the fine-tuned foundation model (i.e., the digital
twin model) to perform tasks such as monitoring, predicting, optimizing and
controlling of the CPS. In this scenario, we can leverage the strengths of the
foundation model for understanding and modeling complex systems. In addition,
for digital twins that have high requirements on real-time interactions with
the CPSs, decoupling the digital twin model from real-time processing provides
the feasibility of allowing the fine-tuned foundation model perform
computations offline or on a schedule, and process data in batches. Doing so
allows for more efficient resource utilization, lower computation cost, and
being more scalable.

Foundation model being the digital twin capability only.  In this scenario, a
fine-tuned foundation model serves as the digital twin capability component,
while the digital twin model is constructed as prior-knowledge based models
(e.g., SysML models). This scenario is useful in application contexts where
digital twin models are mandatorily required to provide a transparent view of
the digital twin due to regulations, for instance. Moreover, with traditional
digital twin models, fine-grained simulations and analyses can be performed,
which empower engineers with more enhanced insights for risk assessment and
management. Moreover, it might be possible that analysis and design models
that have been already developed during the process of certain CPS development
can be fully or partially reused as the digital twin models, which
significantly reduces the cost of developing traditional digital twin models.

The traditional digital twin model can also be used to derive data
requirements and their dependencies which are useful information for guiding
the selection of training data and preprocessing pipelines for the foundation-
based digital twin models. In addition, traditional digital twin models may be
used to generate prompts for LLM-enabled digital twin capabilities. For
instance, if the digital twin model captures CPS anomalies, indicating places
where the foundation model-based digital twin capabilities may need to be
adjusted, then such anomalies can initiate fine-tuning processes for the
foundation model. One can also benefit from the behavior simulation capability
of the digital twin model by utilizing simulation outcomes as the indicator of
generating prompts that reflect unexpected behavior, potential performance
issues, etc.

Foundation model being the whole digital twin. In this scenario, the
foundation model essentially is the whole digital twin and interacts with the
CPS to provide real-time monitoring of the CPS, all kinds of analyses, and
even the controlling of the CPS to various extents. One key advantage of this
architecture is the fact that it significantly simplifies the development of
the explicit digital twin model, which requires manual effort from domain
experts who master at least one modeling notation and modeling tools such as
OpenModelica and SysML, as we discussed in Case 1, where the digital twin
model is an explicit representation of the physical twin. In addition,
replacing the explicit digital twin model with the foundation model eliminates
the large effort required to maintain the explicit digital twin model. This is
because we envision that, with advanced AI technologies, the foundation model
will self-maintain when receiving real-time operational data from the CPS.

To fine-tune a foundation model-based digital twin, we must first choose a
pre-trained foundation model. For instance, in autonomous driving, we can
leverage large models to generate continuous video driving scenarios for
testing ADS algorithms, detecting anomalies, identifying performance
degradation, diagnosing issues via runtime monitoring, etc. Such a foundation
model can already benefit from common-sense knowledge from pre-trained LLMs
about the world, e.g., detecting traffic rule violations and anticipating
potential safety threats. Which large models to choose as the base largely
depends on the characteristics and requirements of the digital twin, the CPS
and the end users’ needs. For example, if the CPS involves processing a large
amount of textual data, such as operating and maintenance logs, then LLMs
might be suitable. Vision models might be more appropriate when involving
videos and images. Also, vision-language models (VLMs) can process textual and
visual information to enable multi-modal data processing. The field is rapidly
booming, hence we envision that more and more large models will become
available in the near future.

After choosing a suitable foundation model and collecting available CPS data,
documented domain knowledge, the digital twin, and CPS specification if it
exists, the model can be fine-tuned to perform tasks. The process includes
selecting model architecture, configuring the foundation model, selecting the
fine-tuning strategy, training the fine-tuned model with the preprocessed CPS
data, evaluating the model performance, and fine-tuning hyperparameters to
obtain the best performance. Considering that the foundation model needs to be
fine-tuned with real-time data, a tuner must be developed to streamline these
fine-tuning tasks continuously and automatically.

###  5.3 Challenges and Opportunities

Though we see the clear benefits of fine-tuning a foundation model to act as a
digital twin for a CPS of interest, several challenges must be overcome.

First, selecting a suitable pretrained large model as the foundation model for
developing the digital twin of a CPS is critical. This is, to a large extent,
because it significantly impacts the performance and effectiveness of the
digital twin. During the selection, it is important to systematically and
comprehensively consider at least the following aspects: 1) the problem
domain, including the CPS domain and its characteristics, types of data
involved (e.g., sensor data, textual documentations), the complexity of tasks
to be performed, and constraints on the deployment of the digital twin, etc.;
2) the availability and quality of data for fine-tuning, which is especially
critical in the context of CPSs as we often do not have access to a large
amount of distribution-wise balanced, high-quality data; and 3) clear
understanding of the digital twin capabilities to be provided, such as anomaly
detection for high-speed trains, safety threat prediction for autonomous
driving, such that it makes it easier to decide which kinds of foundation
models to choose. Currently, there are no guidelines or best practices for
selecting suitable foundation models for developing digital twins for CPSs.
Therefore, developing such guidelines is an important direction for future
research.

Second, considering that the digital twin interacts with the physical twin at
runtime, the efficiency of the digital twin is crucial. Especially in the case
of the digital twin making decisions or providing crucial feedback to support
the CPS operation in real-time, the digital twin has to efficiently process
data, perform computations, and make decisions or generate responses.
Therefore, pursuing a high-efficiency model operating as the digital twin is
important, and advanced technologies will dramatically reshape how such a
model is constructed and deployed in the future.

Third, considering the scenario that the whole digital twin is represented as
a fine-tuned foundation model, we, therefore, do not have explicit
representations of the digital twin. Hence, it is challenging to gain a
comprehensive understanding of the digital twin’s structure, behavior, and
interactions with the CPS. Machine learning interpretability techniques (e.g.,
feature importance) might be needed to provide insights into the digital
twin’s behavior and enhance its transparency. However, such techniques are
limited compared to the cases of explicit representation of the digital twin
model. In addition, sufficiently verifying and validating the digital twin
itself also becomes challenging. Alternative approaches, such as robustness
testing and sensitivity analyses [42], might help assess the digital twin’s
performance and reliability without explicit representation of the digital
twin model. For safety-critical CPS applications, building fine-tuned
foundation models without having explicit representations might not be a
viable solution, as the trustworthiness of the digital twins is paramount,
especially when they directly interact with the safety-critical CPSs. All in
all, we see challenges but also opportunities in this field. For example,
established practices in the field of verifying and validating machine
learning models can be adapted to this context.

Fourth, when comparing digital twins with explicit representations to those
based on fine-tuned foundation models, there are also challenges in fidelity
that need to be considered. The former often captures detailed relationships,
dependencies, and behaviors; however, fine-tuned foundation models typically
abstract away details and hence reduce explicit modeling required complexity,
therefore leading to a loss of fidelity in certain cases. How much fidelity is
sacrificed largely depends on the CPS domain and expected digital twin
capabilities. For instance, comparing an elevator system and autonomous
driving, the loss of fidelity due to the lack of explicit representations
should be larger for autonomous driving as it is much more complex, and hence,
the abstraction gap is larger. Moreover, if an expected digital twin
capability is trivial, we expect low demand for the digital twin fidelity. For
instance, in the case of monitoring room temperature and humidity levels in a
smart home, the expected digital twin capability is to provide alerts and
notifications based on predefined thresholds. Therefore, the fidelity
requirements for the digital twin are low. Consequently, the abstraction gap
is small, and the impact of fidelity loss is minimal.

###  5.4 Example Case of ADS

In the literature, there are proposals for utilizing LLMs for autonomous
driving. For example, Cui et al. [43] proposed Talk2Drive, an LLM-based
framework to process verbal commands from humans and make decisions
autonomously based on contextual information. Talk2Drive is equipped with a
speech recognition module for translating verbal inputs from humans to textual
instructions, which are then sent to LLMs for reasoning and, afterwards,
generating control commands, which are then automatically executed with the
control module. Talk2Drive can be considered a personalized digital twin for
autonomous driving, aiming to reduce the human driver takeover rate of
autonomous vehicles. We also foresee that LLMs can be utilized to generate
digital twins for monitoring the safety of autonomous driving and, when
needed, triggering safety mechanisms (e.g., providing early warnings or
recommending risk mitigation strategies) and assessing the severity of safety
hazards, assisting in emergencies to ensure safe operation of ADSs. Moreover,
foundation model-based digital twins can continuously learn and refine their
capabilities over time.

##  6 Discussions

###  6.1 Uncertainty of Foundation Models

Though, as we discussed in the previous sections, foundation models offer
opportunities to develop digital twins of various forms, their applications,
especially in safety-critical and mission-critical domains, raise significant
concerns about their inherent uncertainty. For example, Huang et al. [44]
empirically explored different quantification methods of epistemic
uncertainties of LLMs, aiming to understand whether and how well uncertainty
estimation can help characterize LLMs’ capabilities in performing different
tasks. Tanneru et al. [45] proposed novel methods for quantifying uncertainty
in natural language explanations of LLMs. With such efforts on understanding
and even quantifying uncertainties of large models, we foresee that there is
still a long way to go to have reliable and trustworthy foundation models to
apply. This is an active and ongoing research area on its own. For CPS domains
that are highly concerned with safety, security, etc., cautions should be made
to ensure the deployment of foundation models is safe and reliable, such as
conducting thorough risk assessments and adhering to rigorous validation and
verification processes.

###  6.2 Cost and Effectiveness of Using Foundation Models

For any software engineering solutions, we need to thoroughly consider their
cost-effectiveness. Applying foundation models does not come with no cost.
First, developing foundation models can involve high costs of computational
resources, data collection and preprocessing, model training, fine-tuning,
etc. Such costs can vary across different CPS domains. Second, deploying
certain foundation models requires specialized hardware resources (e.g., GPU,
FPGA), especially considering that fine-tuned foundation models (directly
acting as digital twins – Case 2) need to communicate with CPSs, which demands
high throughput and low latency of communication and computation. In addition,
fine-tuned foundation models must be maintained regularly, contributing to
maintenance costs. However, as discussed in previous sections, there are
advantages in applying foundation models, such as reducing the manual effort
required to build traditional digital twin models. Therefore, in our opinion,
it is necessary to compare the cost-effectiveness of using foundation models
against traditional solutions via conducting empirical studies, such that
using foundation models is justified.

###  6.3 Lightweight Foundation Models for Digital Twins

In addition to tailoring large foundation models for use as digital twins, we
also foresee that there will be a need for lightweight specialized foundation
models in some contexts. This will, in particular, be important in cases where
digital twins are to be executed on dedicated hardware with low computational
power. Moreover, we foresee their applications in critical applications where
large foundation models cannot be accessed over the cloud. In addition, these
models will be suitable for digital twins where inference time is important
since the inference time of lightweight models is supposed to be shorter.
Consequently, these foundation models are more suitable for digital twins with
real-time applications. Finally, since these models can be deployed on
dedicated resources, there will be less need to send data, e.g., to the cloud,
thereby more security and privacy-preservation.

###  6.4 Challenges in Generating Digital Twins with Foundation Models

Using foundation models for the digital twin of CPSs comes with challenges,
which generally hold for such models. For example, how to deal with
hallucinations when generating digital twins or using them as digital twins.
To this end, studying and dealing with hallucinations in foundation models
(i.e., content generated by the foundation model that is not based on factual
information) is an active research field with several existing strategies, and
more strategies are appearing quickly [46, 47]. Investigation of existing
hallucination handling strategies is an interesting research direction to
follow.

Another challenge is how to assess the fidelity of digital twins. There are
two aspects. First, in terms of using foundation models to generate digital
twins, the fidelity of digital twins can be accessed similarly as typically
done (e.g., see [9]). Second, when foundation models are used as digital
twins, it opens up a new research direction for assessing their fidelity,
since this will require defining new metrics and methods.

Finally, foundation models pose many challenges related to ethical and legal
aspects. Using digital twins to generate foundation models naturally leads to
inheriting those challenges. For example, using foundation models to generate
digital twins may generate copyrighted models. Moreover, these models are
digital twins if such models have been trained on private data in certain
cases. Furthermore, another challenge is whether digital twins will not make
discriminatory and biased decisions is another challenge [2].

##  7 Conclusion

Due to their generic nature, foundation models offer many potential
applications. To this end, software engineering researchers have increasingly
been interested in discovering such applications. Along this line, we present
perspectives on how foundation models can be used in creating digital twins of
cyber-physical systems (CPSs). In particular, we present two perspectives,
i.e., using foundation models to generate digital twins and using foundations
as digital twins. We present state-of-the-art conceptual solutions,
challenges, and opportunities for each perspective. We also use the case of
digital twins of autonomous driving systems to present some ideas. Finally, we
present discussion and open research questions.

## Acknowledgments

S. Ali is supported by the Co-tester project (No. 314544) funded by the
Research Council of Norway and the RoboSAPIENS project financed by the
European Commission’s Horizon Europe programme under Grant 101133807. P.
Arcaini is supported by MIRAI Engineerable AI Project (No. JPMJMI20B8), JST.
Aitor Arrieta is part of the Software and Systems Engineering research group
of Mondragon Unibertsitatea (IT1519-22), supported by the Department of
Education, Universities and Research of the Basque Country.

## References

  * [1] Adam Kolides, Alyna Nawaz, Anshu Rathor, Denzel Beeman, Muzammil Hashmi, Sana Fatima, David Berdik, Mahmoud Al-Ayyoub, and Yaser Jararweh.  Artificial intelligence foundation and pre-trained models: Fundamentals, applications, opportunities, and social impacts.  Simulation Modelling Practice and Theory, 126:102754, 2023.
  * [2] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ B. Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen Creel, Jared Quincy Davis, Dorottya Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah D. Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark S. Krass, Ranjay Krishna, Rohith Kuditipudi, and et al.  On the opportunities and risks of foundation models.  CoRR, abs/2108.07258, 2021.
  * [3] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang.  Is your code generated by ChatGPT really correct? rigorous evaluation of large language models for code generation.  In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2024. Curran Associates Inc.
  * [4] Alessio Ferrari, Sallam Abualhaija, and Chetan Arora.  Model generation from requirements with LLMs: an exploratory study.  CoRR, abs/2404.06371, 2024.
  * [5] Mohsen Attaran and Bilge Gokhan Celik.  Digital twin: Benefits, use cases, challenges, and opportunities.  Decision Analytics Journal, 6:100165, 2023.
  * [6] John Fitzgerald, Peter Gorm Larsen, Tiziana Margaria, Jim Woodcock, and Cláudio Gomes.  Engineering of digital twins for cyber-physical systems.  In Tiziana Margaria and Bernhard Steffen, editors, Leveraging Applications of Formal Methods, Verification and Validation. Practice, pages 3–8, Cham, 2022. Springer Nature Switzerland.
  * [7] Stefan Mihai, Mahnoor Yaqoob, Dang V. Hung, William Davis, Praveer Towakel, Mohsin Raza, Mehmet Karamanoglu, Balbir Barn, Dattaprasad Shetve, Raja V. Prasad, Hrishikesh Venkataraman, Ramona Trestian, and Huan X. Nguyen.  Digital twins: A survey on enabling technologies, challenges, trends and future prospects.  IEEE Communications Surveys & Tutorials, 24(4):2255–2291, 2022.
  * [8] Tao Yue, Shaukat Ali, Paolo Arcaini, and Fuyuki Ishikawa.  Towards requirements engineering for digital twins of cyber-physical systems.  In Tiziana Margaria and Bernhard Steffen, editors, Leveraging Applications of Formal Methods, Verification and Validation. Practice, pages 9–21, Cham, 2022. Springer Nature Switzerland.
  * [9] Tao Yue, Paolo Arcaini, and Shaukat Ali.  Understanding digital twins for cyber-physical systems: A conceptual model.  In Tiziana Margaria and Bernhard Steffen, editors, Leveraging Applications of Formal Methods, Verification and Validation: Tools and Trends, pages 54–71, Cham, 2021. Springer International Publishing.
  * [10] Johannes Schneider, Christian Meske, and Pauline Kuss.  Foundation models.  Business & Information Systems Engineering, 66(2):221–231, Apr 2024.
  * [11] Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar.  Foundation models for generalist medical artificial intelligence.  Nature, 616(7956):259–265, Apr 2023.
  * [12] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021.
  * [13] Landing AI Website.  https://landing.ai/platform, 2024.  [Online; accessed May 6, 2024].
  * [14] Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, and Jianfeng Gao.  Multimodal foundation models: From specialists to general-purpose assistants.  Found. Trends Comput. Graph. Vis., 16(1-2):1–214, 2024.
  * [15] Fuyu-8B.  https://huggingface.co/adept/fuyu-8b, 2024.  [Online; accessed May 6, 2024].
  * [16] ALIGN Website.  https://huggingface.co/docs/transformers/en/model_doc/align, 2024.  [Online; accessed May 6, 2024].
  * [17] Shuncheng Tang, Zhenya Zhang, Yi Zhang, Jixiang Zhou, Yan Guo, Shuang Liu, Shengjian Guo, Yan-Fu Li, Lei Ma, Yinxing Xue, and Yang Liu.  A survey on automated driving system testing: Landscapes and trends.  ACM Trans. Softw. Eng. Methodol., 32(5), jul 2023.
  * [18] Shutong Deng, Liang Ling, Caizhi Zhang, Congbo Li, Tao Zeng, Kaiqing Zhang, and Gang Guo.  A systematic review on the current research of digital twin in automotive application.  Internet of Things and Cyber-Physical Systems, 3:180–191, 2023.
  * [19] Qiwei Chen, Tiexin Wang, Chengjie Lu, Tao Yue, and Shaukat Ali.  Enhancing the realism of autonomous driving simulation with real-time co-simulation.  In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, MODELS ’22, pages 659–667, New York, NY, USA, 2022. Association for Computing Machinery.
  * [20] Eldad Palachi, Chaim Cohen, and Sakairi Takashi.  Simulation of cyber physical models using SysML and numerical solvers.  In 2013 IEEE International Systems Conference (SysCon), pages 671–675, April 2013.
  * [21] John Fitzgerald, Peter Gorm Larsen, and Ken Pierce.  Multi-modelling and Co-simulation in the Engineering of Cyber-Physical Systems: Towards the Digital Twin, pages 40–55.  Springer, Cham, 2019.
  * [22] Qinghua Xu, Shaukat Ali, and Tao Yue.  Digital twin-based anomaly detection with curriculum learning in cyber-physical systems.  ACM Trans. Softw. Eng. Methodol., 32(5), jul 2023.
  * [23] Matthias Eckhart and Andreas Ekelhart.  Digital Twins for Cyber-Physical Systems Security: State of the Art and Outlook, pages 383–412.  Springer International Publishing, Cham, 2019.
  * [24] Qinghua Xu, Shaukat Ali, and Tao Yue.  Digital twin-based anomaly detection in cyber-physical systems.  In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), pages 205–216, 2021.
  * [25] Heinrich Wagner, Lukas Portenlänger, and Claudio Zuccaro.  Using SysML models as digital twins for early validation of modular systems and systems of systems.  In 2023 18th Annual System of Systems Engineering Conference (SoSe), pages 1–7, 2023.
  * [26] F. Wilking, C. Sauer, B. Schleich, and S. Wartzack.  SysML 4 digital twins – utilization of system models for the design and operation of digital twins.  Proceedings of the Design Society, 2:1815–1824, 2022.
  * [27] Javier Cámara, Javier Troya, Lola Burgueño, and Antonio Vallecillo.  On the assessment of generative AI in modeling tasks: an experience report with ChatGPT and UML.  Softw. Syst. Model., 22(3):781–793, may 2023.
  * [28] Qinghua Xu, Shaukat Ali, Tao Yue, Zaimovic Nedim, and Inderjeet Singh.  KDDT: Knowledge distillation-empowered digital twin for anomaly detection.  In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, pages 1867–1878, New York, NY, USA, 2023. Association for Computing Machinery.
  * [29] Qinghua Xu, Shaukat Ali, Tao Yue, and Maite Arratibel.  Uncertainty-aware transfer learning to evolve digital twins for industrial elevators.  In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2022, pages 1257–1268, New York, NY, USA, 2022. Association for Computing Machinery.
  * [30] Matteo Perno, Lars Hvam, and Anders Haug.  A machine learning digital twin approach for critical process parameter prediction in a catalyst manufacturing line.  Computers in Industry, 151:103987, 2023.
  * [31] Michel Fett, Fabian Wilking, Stefan Goetz, Eckhard Kirchner, and Sandro Wartzack.  A literature review on the development and creation of digital twins, cyber-physical systems, and product-service systems.  Sensors, 23(24), 2023.
  * [32] Zibin Zheng, Kaiwen Ning, Yanlin Wang, Jingwen Zhang, Dewu Zheng, Mingxi Ye, and Jiachi Chen.  A survey of large language models for code: Evolution, benchmarking, and future trends.  CoRR, abs/2311.10372, 2023.
  * [33] Hassan Sartaj, Shaukat Ali, Tao Yue, and Kjetil Moberg.  HITA: an architecture for system-level testing of healthcare IoT applications.  CoRR, abs/2309.04223, 2023.
  * [34] Jiahui Wu, Chengjie Lu, Aitor Arrieta, Tao Yue, and Shaukat Ali.  Reality bites: Assessing the realism of driving scenarios with large language models.  In Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering, FORGE ’24, pages 40–51, New York, NY, USA, 2024. Association for Computing Machinery.
  * [35] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models.  CoRR, abs/2303.18223, 2023.
  * [36] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.  Virtualhome: Simulating household activities via programs.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8494–8502, 2018.
  * [37] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein.  Generative agents: Interactive simulacra of human behavior.  In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22, 2023.
  * [38] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, and Tao Gui.  The rise and potential of large language model based agents: A survey.  CoRR, abs/2309.07864, 2023.
  * [39] Hanqing Yang, Marie Siew, and Carlee Joe-Wong.  An LLM-based digital twin for optimizing human-in-the loop systems.  CoRR, abs/2403.16809, 2024.
  * [40] Yue Wang, Yingzhou Lu, Yinlong Xu, Zihan Ma, Hongxia Xu, Bang Du, Honghao Gao, and Jian Wu.  TWIN-GPT: digital twins for clinical trials via large language model.  CoRR, abs/2404.01273, 2024.
  * [41] Mingyu Jin, Qinkai Yu, Dong Shu, Chong Zhang, Suiyuan Zhu, Mengnan Du, Yanda Meng, and Yongfeng Zhang.  Health-LLM: Personalized retrieval-augmented disease prediction system.  CoRR, abs/2402.00746, 2024.
  * [42] Liping Han, Shaukat Ali, Tao Yue, Aitor Arrieta, and Maite Arratibel.  Uncertainty-aware robustness assessment of industrial elevator systems.  ACM Trans. Softw. Eng. Methodol., 32(4), may 2023.
  * [43] Can Cui, Zichong Yang, Yupeng Zhou, Yunsheng Ma, Juanwu Lu, and Ziran Wang.  Large language models for autonomous driving: Real-world experiments.  CoRR, abs/2312.09397, 2023.
  * [44] Yuheng Huang, Jiayang Song, Zhijie Wang, Huaming Chen, and Lei Ma.  Look before you leap: An exploratory study of uncertainty measurement for large language models.  CoRR, abs/2307.10236, 2023.
  * [45] Sree Harsha Tanneru, Chirag Agarwal, and Himabindu Lakkaraju.  Quantifying uncertainty in natural language explanations of large language models.  In International Conference on Artificial Intelligence and Statistics, pages 1072–1080. PMLR, 2024.
  * [46] Vipula Rawte, Amit P. Sheth, and Amitava Das.  A survey of hallucination in large foundation models.  CoRR, abs/2309.05922, 2023.
  * [47] Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  CoRR, abs/2311.05232, 2023.


# Spectrally resolved NOON state interference

Rui-Bo Jin1,2, Ryosuke Shimizu3, Takafumi Ono2,4,5, Mikio Fujiwara2, Guang-Wei
Deng6 Qiang Zhou6<EMAIL_ADDRESS>Masahide Sasaki2<EMAIL_ADDRESS>Masahiro Takeoka2,7<EMAIL_ADDRESS>1Hubei Key Laboratory of Optical
Information and Pattern Recognition, Wuhan Institute of Technology, Wuhan
430205, China  2National Institute of Information and Communications
Technology, 4-2-1 Nukui-Kitamachi, Koganei, Tokyo 184-8795, Japan  3University
of Electro-Communications, 1-5-1 Chofugaoka, Chofu, Tokyo 182-8585, Japan
4Faculty of Engineering and Design, Kagawa University, 2217-20, Hayashi-cho,
Takamatsu, Kagawa Japan  5JST, PRESTO, 4-1-8 Honcho, Kawaguchi, Saitama,
332-0012, Japan  6Institute of Fundamental and Frontier Sciences and School of
Optoelectronic Science and Engineering,
University of Electronic Science and Technology of China, Chengdu 610054,
China  7Keio University, 3-14-1 Hiyoshi, Kohoku, Yokohama, Kanagawa 223-8522,
Japan

###### Abstract

NOON state interference (NOON-SI) is a powerful tool to improve the phase
sensing precision, and plays an important role in quantum measurement. In most
of the previous NOON-SI experiments, the measurements were performed in time
domain where the spectral information of the involved photons was integrated
and lost during the measurement. In this work, we experimentally measured the
joint spectral intensities (JSIs) at different positions of the interference
patterns in both time and frequency domains. It was observed that the JSIs
were phase-dependent and show odd (even)-number patterns at $0$ ($\pi$) phase
shift; while no interference appeared in time domain measurement, the
interference pattern clearly appeared in frequency domain. To our best
knowledge, the latter is the first observation of the spectrally resolved NOON
state interference, which provides alternative information that cannot be
extracted from the time-domain measurement. To explore its potential
applications, we considered the interferometric sensing with our setup. From
the Fisher information-based analysis, we show that the spectrally resolved
NOON-SI has a better performance at non-zero-delay position than its non-
spectrally resolved counterpart. The spectrally resolved NOON-SI scheme may be
useful for quantum metrology applications such as quantum phase sensing,
quantum spectroscopy, and remote synchronization.

_Introduction_ Multi-photon entanglement and multi-photon interference are
useful nonclassical resources in quantum information applications. In
particular, the so-called NOON state interference (NOON-SI) is a powerful tool
to improve the phase sensing precision. NOON state is an entangled state with
$N$ photons occupying either one of two optical modes (e.g., polarization
modes or path modes):
$\frac{1}{{\sqrt{2}}}(\left|{N0}\right\rangle+\left|{0N}\right\rangle)$ Boto
_et al._ (2000); Edamatsu  _et al._ (2002); Giovannetti  _et al._ (2002).
NOON-SI is a powerful tool to improve the phase sensing precision to
Heisenberg limit of $\Delta\phi=1/N$ , much higher than the shot noise limit
of $\Delta\phi=1/\sqrt{N}$, which is precision limit of the classical
interferometer, e.g., the Mach-Zehnder interferometer (MZI) Giovannetti  _et
al._ (2004). NOON-SI has been widely used in super-resolving phase
measurements Mitchell  _et al._ (2004); Walther  _et al._ (2004); Nagata  _et
al._ (2007); Afek  _et al._ (2010); Slussarenko  _et al._ (2017); Zhou  _et
al._ (2017), quantum lithography Boto  _et al._ (2000), quantum microscopy Ono
_et al._ (2013); Israel  _et al._ (2014), quantum spectroscopy Jin and Shimizu
(2018) and error correction Bergmann and van Loock (2016).

Most of the previous NOON-SI experiments were measured in time domain Boto
_et al._ (2000); Edamatsu  _et al._ (2002); Giovannetti  _et al._ (2002,
2004); Mitchell  _et al._ (2004); Walther  _et al._ (2004); Nagata  _et al._
(2007); Afek  _et al._ (2010); Slussarenko  _et al._ (2017); Zhou  _et al._
(2017); Ono  _et al._ (2013); Israel  _et al._ (2014); Jin and Shimizu (2018);
Bergmann and van Loock (2016), i.e., the interference patterns are obtained by
recording the coincidence counts as a function of the temporal delay. However,
the conjugate parameter of time, i.e., the spectral information of the
involved photons was integrated and lost during the time-domain measurement.
The knowledge of the spectral correlations of the interfering photons can
reveal some important information about the interference visibility Mosley
(2007); Gerrits  _et al._ (2015). Therefore, it is very valuable to explore
such spectral information. But in the past the spectral correlation content
could not be observed, partially due to the long acquisition times required
when measuring spectral correlations using two tunable bandpass filters Jin
_et al._ (2013). Recently, a new technique for measuring spectral correlations
in a Hong-Ou-Mandel (HOM) interference Hong  _et al._ (1987) was presented by
Gerrits, et al Gerrits  _et al._ (2015); Jin  _et al._ (2015, 2016). This
technique has the merits of short measurement time and high spectral
resolution, and therefore allows measuring and analyzing the spectral
correlations from a HOM interference Gerrits  _et al._ (2015); Jin  _et al._
(2015, 2016). This technique also make it possible to observe the joint
spectral information during the NOON-SI.

In this work, we apply the state-of-art spectral correlation measurement
technique in the NOON-SI, and measure the joint spectral intensities (JSIs) at
different positions of the interference. We observe the interference pattern
in the frequency domain clearly, while no interference appears in the time-
domain measurement. This implies that some new spectral information was indeed
observed, although it was integrated and lost during the time-domain
measurement.

Figure 1: (a): The experimental setup for NOON-SI using polarization-mode.
LPF=long pass filter, HWP=half wave plate, M=mirror, QWP=quarter wave plate,
PZT=piezo-electric linear actuator, PBS=polarization beam splitter,
SMFC=single-mode fiber coupler, DCFM=dispersion compensation fiber module,
TIA=time interval analyzer. The inset depicts a standard configuration of the
NOON-SI using path-mode. BS=beam splitter. (b): The NOON-SI pattern measured
in time domain by scanning a stepping motor with a step length of 4 $\mu m$.
Figure 2:  (a0, b0, c0): The experimental NOON-SI pattern measured in time
domain by scanning a PZT with a step length several nm, when fix the stepping
motor at the delay time of 0 ps, 5 ps and 10 ps, respectively. The visibility
is 97.09 $\pm$ 0.03%, 3.53$\pm$ 0.50% and 0% (a1-a5, b1-b5, c1-c5): The
measured JSIs at different phase delay from 0 to 2$\pi$ in (a0, b0, c0).

_Experiment and results_ The experimental setup for the NOON-SI is shown in
Fig. 1(a). Picosecond laser pulses (pulse repetition frequency = 76 MHz,
wavelength = 792 nm, temporal duration $\sim$ 2 ps) from a mode-locked
Titanium sapphire laser pump a 30-mm-long PPKTP crystal with a poling period
of 46.1 $\mu$m for type-II group-velocity-matched SPDC König and Wong (2004);
Jin  _et al._ (2013). The down converted photons, i.e., the signal and idler
photons, have orthogonal polarizations and degenerate wavelengths at 1584 nm.
To temporally overlap the signal and idler, the downconverted biphotons pass
through a 15-mm-long KTP crystal. Then, the biphotons are mixed on a half wave
plate (HWP1, at 22.5 ∘), which functions similarly as a beamsplitter in the
NOON-SI using path-mode (BS1 in the inset of Fig. 1(a)). After the biphotons
are separated by a polarization beam splitter (PBS1), they are sent to a time
delay system, which is composed of two quarter wave plates (QWP1 and QWP2, at
45 ∘) and two mirrors. One of the mirrors (M1) is set on a PZT to achieve a
scanning step in the order of nm, while the other one (M2) is on a stepping
motor to realize a scanning step of $\mu$m. Then, the biphotons are mixed
again on the second half wave plate (HWP2, at 22.5 ∘), which functions as the
BS2 in the inset. Finally, the biphotons are separated by PBS2 and collected
into two channels of single-mode fibers (Ch1 and Ch2), which is connected to a
dispersion spectrometer Avenhaus  _et al._ (2009); Gerrits  _et al._ (2011,
2015); Jin  _et al._ (2015, 2016), composed of a dispersion compensation fiber
module (DCFM), two superconducting nanowire single-photon detectors (SNSPDs)
Miki  _et al._ (2013) and a time interval analyzer (TIA). The DCFM has a
dispersion of 125.0 ps/km/nm (full dispersion of 941 ps/nm) and an insertion
loss of 5.2 dB. The SNSPDs have detection efficiencies around 70% with dark
counts less than 1 kHz. Assuming a system jitter of 100 ps, the resolution of
our fiber spectrometer is estimated as $0.11$ nm. To keep the phase stability
in the interference, we set the time delay system in an enclosed paper box to
avoid the air flow, such that the phase stability can be maintained for more
than 5 minutes.

The PPKTP crystal is pumped with a power of 50 mW. With the insertion of DCFM,
we obtain an average singles count rate of about 200 kcps, and a two-fold
coincidence count rate of about 7 kcps. First, we measure the NOON-SI in time
domain, i.e., scan the time delay using a stepping motor and record the
coincidence counts. Fig. 1(b) shows the measured interference pattern with a
full-width-at-half-maximum (FWHM) of 4.2 ps (1.3 mm). Then, we characterize
the NOON-SI in spectral domain, i.e., the JSI of the biphotons in Ch1 and Ch2
was measured at different delay time. We focus on three delay time: 0 ps, 5 ps
and 10 ps, by moving the stepping motor as the coarse adjustment and scan the
PZT as the fine adjustment.

Figure 2(a0) shows the interference pattern at 0 ps. By driving the PZT with a
voltage from 50 V to 53 V, the resultant relative phase delay is about 6$\pi$.
The interference visibility in Fig. 2(a0) is 97.09 $\pm$ 0.03% and the
oscillation period is 792 nm (0.462 fs), half of the biphoton’s wavelength.
Then, we measure the JSI using the dispersion spectrometer. Figure 2(a1-a5)
shows the measured JSIs at the relative phase from 0 to 2$\pi$ with a step of
$\pi$/2. The horizontal (vertical) axis is the arrival time of the photons in
Ch1 (Ch2). The arrival time is ranging from -1.5 ns to 1.5 ns, and the
corresponding wavelength is from 1582 nm to 1586 nm. Each figure was
accumulated for 60 seconds. The JSI at 0 ($2\pi$) is the brightest and at
$\pi$ is the darkest. The JSI in Fig. 2(a1) is equal to the JSI of the
biphoton source.

Figure 2(b0) shows the measured coincidence counts as a function of time delay
at 5 ps. The visibility is 3.53 $\pm$ 0.50%, which can also be confirmed in
Fig. 1(b). Figure 2(b1-b5) shows the measured JSIs at the phase of 0,
$\frac{\pi}{2}$, $\pi$, $\frac{3\pi}{2}$, and $2\pi$ in Fig. 2(b0). It can be
observed clearly that the distribution of the JSIs is splitting along the
diagonal axis. The JSI is phase sensitive, i.e, the distribution in the JSI is
different for different phases. At $\frac{\pi}{2}$, the up-right section is
brighter, while at $\frac{3\pi}{2}$, the down-left section is brighter.

Figure 2(c0) shows the temporal interference pattern at the time delay of 10
ps. The visibility is almost zero in Fig. 2(c0), as also confirmed in Fig.
1(b). Figure 2(c1-c5) shows the measured JSIs at the phase of 0,
$\frac{\pi}{2}$, $\pi$, $\frac{3\pi}{2}$, and $2\pi$ in Fig. 2(c0). It can be
noticed that the JSI is separated into odd-number sections at 0 ($2\pi$), and
even-number sections at $\pi$. It can be concluded that when the time-domain
interference visibility is almost zero, the interference patterns in the
spectral domain are still very clear.

_Theoretical analysis_ The experimental results in Fig. 2 is well explained in
theory. Assume the signal and idler photons from the SPDC has a joint spectral
amplitude (JSA) of $f(\omega_{s},\omega_{i})$, where $\omega$ is the angular
frequency; the subscripts $s$ and $i$ denote the signal and idler. For
simplicity, we further assume the JSA has the exchanging symmetry of
$f(\omega_{s},\omega_{i})=f(\omega_{i},\omega_{s})$. After a long calculation
using multi-spectral-mode theory Jin and Shimizu (2018); Ou (2007), the two-
fold coincidence probability $P$ as a time delay $\tau$ in a NOON-SI can be
described by

$P(\tau)=\frac{1}{2}\int_{0}^{\infty}\int_{0}^{\infty}d\omega_{s}d\omega_{i}\left|f(\omega_{s},\omega_{i})\right|^{2}[1+\cos(\omega_{s}+\omega_{i})\tau].$
(1)

The JSI $I(\omega_{s},\omega_{i},\tau)$ during the NOON-SI can be written as

$I(\omega_{s},\omega_{i},\tau)=\frac{1}{2}\left|f(\omega_{s},\omega_{i})\right|^{2}[1+\cos(\omega_{s}+\omega_{i})\tau].$
(2)

Using the parameters of the pump laser and the PPKTP crystal, we can calculate
the JSA. With the JSA, the time-domain interference patterns in Fig. 1(b) and
Fig. 2(a0, b0, c0) can be simulated using Eq.(1). The frequency-domain
interference patterns in Fig. 2(a1-a5, b1-b5, c1-c5) can be simulated by using
Eq.(2).

Figure 3 shows the simulated results. In Fig. 3(a0, b0, c0), the visibility
are 100%, 3.8% and 0.006%, which correspond well with the experimental results
in Fig. 2(a0, b0, c0). Figure 3(a1-a5, b1-b5, c1-c5) shows the simulated JSIs
at different phase delay from 0 to 2$\pi$ in (a0, b0, c0). The measured JSIs
are consistent with the simulated ones. The experimentally measured JSIs are
“fatter” than the simulated JSIs, since the resolution of the fiber spectral
meter is limited.

Figure 3:  (a0, b0, c0): The theoretically simulated NOON-SI pattern in time
domain at the delay time of 0 ps, 5 ps and 10 ps, respectively. The
visibilities are 100%, 3.8% and 0.006%. (a1-a5, b1-b5, c1-c5): The simulated
JSIs at different phase delay from 0 to 2$\pi$ in (a0, b0, c0).

_Discussion_ We compare the traditional non-spectrally resolved (NSR) NOON-SI
and the spectrally resolved NOON-SI in our scheme by analyzing the Fisher
information, which has been widely used to evaluate the phase sensing ability
of the NOON-SI Xiang  _et al._ (2013). The Fisher information for the non-
spectrally resolved NOON-SI ($F_{NSR}$) can be calculated as:

$F_{NSR}(\tau)=\frac{{\left[{P^{\prime}_{NSR}(\tau)}\right]^{2}}}{{P_{NSR}(\tau)[1-P_{NSR}(\tau)]}},$
(3)

where $P_{NSR}(\tau)=P(\tau)$ and $P^{\prime}_{NSR}$ is the differentiation of
$P_{NSR}$ with respect to the time $\tau$. Fisher information for the
spectrally resolved NOON-SI ($F_{SR}$) is

$F_{SR}(\tau)=\int_{0}^{\infty}\int_{0}^{\infty}|f(\omega_{s},\omega_{i})|^{2}\times(\omega_{s}{\rm{+}}\omega_{i})^{2}d\omega_{s}d\omega_{i}.$
(4)

With these two equations and the theoretical $f(\omega_{s},\omega_{i})$, we
simulated $F_{NSR}$ and $F_{SR}$ . It is noteworthy that $F_{SR}$ equals to
the maximal value of $F_{NSR}$ and does not degrade in delay. This feature
implies that the spectrally resolved techniques enable the acquisition of
higher Fisher information in wider range of delay time. This advantage is very
useful for time or phase estimation in wide range.

It is interesting to compare this spectrally resolved NOON-SI with the result
in the spectrally resolved HOM interference in Refs. Gerrits  _et al._ (2015);
Jin  _et al._ (2015, 2016). In the NOON-SI, the JSI is splitting along the
anti-diagonal direction, while the JSI is splitting along the diagonal
direction in the HOM interference. This difference is result from the facts
that the NOON-SI is a sum-frequency interference, which is described in
Eq.(1), while the HOMI is a difference-frequency interference, as explained in
Refs. Gerrits  _et al._ (2015); Jin  _et al._ (2015, 2016); Jin and Shimizu
(2018); Giovannetti  _et al._ (2002). It can be observed that the distribution
of the JSI is changing at different phase delay. This phase dependent feature
is totally different from the case of HOM interference Gerrits  _et al._
(2015); Jin  _et al._ (2015, 2016), which is phase independent. The spectrally
resolved HOM interference has been applied for the situation in which the
phase insensitivity is required, e.g., the generation and distribution of
frequency-entangled qudits Jin  _et al._ (2016), efficient quantum-optical
coherence tomography Yepiz-Graciano  _et al._ (2020).

We noticed the recent report on the spectrally-resolved quantum white-light
interferometry (WLI) for optical chromatic dispersion measurement, which has a
better precision than the classical WLI and can be used for absolute parameter
determination Kaiser  _et al._ (2018). In Ref. Kaiser  _et al._ (2018) only
one spectrometer was utilized, i.e., one-dimensional spectrally-resolved. In
contrast, our measurement are two-dimensional spectrally resolved, where two
spectrometer are utilized to measure the spectral correlation between the
signal and idler photons. This two dimensional spectral resolving technique
may also has the potential for higher precision parameter estimation in wider
range in the future.

It is important to discuss the feasibility of the future applications. The
interference visibility is almost 0 in time domain at 10 ps in Fig. 2(c0),
however, the visibility in spectral domain in Fig. 2(c1-c5) is still very
high. This implies that the spectral measurement can exploit some new
information that has never been measured in the time domain. This extra
spectral information might be an important supplementary to the time-domain
information, and might be used for promising applications, such as remote
synchronization Quan  _et al._ (2016), quantum spectroscopy Dinani  _et al._
(2016); Whittaker  _et al._ (2017, 2017) and quantum metrology Dowling (2008);
Taylor and Bowen (2016) in the future.

_Conclusion_ We have experimentally and theoretically demonstrated spectrally
resolved NOON-SI with a spontaneous parametric down conversion source at 1584
nm wavelength. It was confirmed that, while there is no interference in time
domain, the interference visibility in spectral domain is still very high. The
spectrally resolved NOON-SI has a higher Fisher information than the non-
spectrally resolved NOON-SI at the position far from the interference center,
therefore has a better performance for time or phase estimation in a wider
range. This spectrally resolved NOON-SI scheme may be applied for remote
synchronization, absolute parameter estimation, quantum spectroscopy, and
quantum metrology.

_Acknowledgements_ The authors thank Thomas Gerrits and Kentaro Wakui for
helpful discussions. This work is partially supported by the ImPACT Program of
Council for Science, Technology and Innovation (Cabinet Office, Government of
Japan), by the National Key R$\&$D Program of China (Grant No.
2018YFA0307400), the National Natural Science Foundations of China (Grant Nos.
91836102, 11704290, 12074299, 61775025, 12074058), and by a fund from the
Educational Department of Hubei Province, China (Grant No. D20161504).

## References

  * Boto  _et al._ (2000) A. N. Boto, P. Kok, D. S. Abrams, S. L. Braunstein, C. P. Williams, and J. P. Dowling, Phys. Rev. Lett. 85, 2733 (2000).
  * Edamatsu  _et al._ (2002) K. Edamatsu, R. Shimizu, and T. Itoh, Phys. Rev. Lett. 89, 213601 (2002).
  * Giovannetti  _et al._ (2002) V. Giovannetti, L. Maccone, J. H. Shapiro, and F. N. C. Wong, Phys. Rev. Lett. 88, 183602 (2002).
  * Giovannetti  _et al._ (2004) V. Giovannetti, S. Lloyd, and L. Maccone, Science 306, 1330 (2004).
  * Mitchell  _et al._ (2004) M. W. Mitchell, J. S. Lundeen, and A. M. Steinberg, Nature 429, 161 (2004).
  * Walther  _et al._ (2004) P. Walther, J.-W. Pan, M. Aspelmeyer, R. Ursin, S. Gasparoni, and A. Zeilinger, Nature 429, 158 (2004).
  * Nagata  _et al._ (2007) T. Nagata, R. Okamoto, J. L. O’Brien, K. Sasaki, and S. Takeuchi, Science 316, 726 (2007).
  * Afek  _et al._ (2010) I. Afek, O. Ambar, and Y. Silberberg, Science 328, 879 (2010).
  * Slussarenko  _et al._ (2017) S. Slussarenko, M. M. Weston, H. M. Chrzanowski, L. K. Shalm, V. B. Verma, S. W. Nam, and G. J. Pryde, Nat. Photon. 11, 700 (2017).
  * Zhou  _et al._ (2017) Z.-Y. Zhou, S.-L. Liu, S.-K. Liu, Y.-H. Li, D.-S. Ding, G.-C. Guo, and B.-S. Shi, Phys. Rev. Appl. 7, 064025 (2017).
  * Ono  _et al._ (2013) T. Ono, R. Okamoto, and S. Takeuchi, Nat. Commun. 4, 2426 (2013).
  * Israel  _et al._ (2014) Y. Israel, S. Rosen, and Y. Silberberg, Phys. Rev. Lett. 112, 103604 (2014).
  * Jin and Shimizu (2018) R.-B. Jin and R. Shimizu, Optica 5, 93 (2018).
  * Bergmann and van Loock (2016) M. Bergmann and P. van Loock, Phys. Rev. A 94, 012311 (2016).
  * Mosley (2007) P. J. Mosley,  _Generation of Heralded Single Photons in Pure Quantum States_ , Ph.D. thesis, University of Oxford (2007).
  * Gerrits  _et al._ (2015) T. Gerrits, F. Marsili, V. B. Verma, L. K. Shalm, M. Shaw, R. P. Mirin, and S. W. Nam, Phys. Rev. A 91, 013830 (2015).
  * Jin  _et al._ (2013) R.-B. Jin, R. Shimizu, K. Wakui, H. Benichi, and M. Sasaki, Opt. Express 21, 10659 (2013).
  * Hong  _et al._ (1987) C. K. Hong, Z. Y. Ou, and L. Mandel, Phys. Rev. Lett. 59, 2044 (1987).
  * Jin  _et al._ (2015) R.-B. Jin, T. Gerrits, M. Fujiwara, R. Wakabayashi, T. Yamashita, S. Miki, H. Terai, R. Shimizu, M. Takeoka, and M. Sasaki, Opt. Express 23, 28836 (2015).
  * Jin  _et al._ (2016) R.-B. Jin, R. Shimizu, M. Fujiwara, M. Takeoka, R. Wakabayashi, T. Yamashita, S. Miki, H. Terai, T. Gerrits, and M. Sasaki, Quantum Sci. Technol. 1, 015004 (2016).
  * König and Wong (2004) F. König and F. N. C. Wong, Appl. Phys. Lett. 84, 1644 (2004).
  * Avenhaus  _et al._ (2009) M. Avenhaus, A. Eckstein, P. J. Mosley, and C. Silberhorn, Opt. Lett. 34, 2873 (2009).
  * Gerrits  _et al._ (2011) T. Gerrits, M. J. Stevens, B. Baek, B. Calkins, A. Lita, S. Glancy, E. Knill, S. W. Nam, R. P. Mirin, R. H. Hadfield, R. S. Bennink, W. P. Grice, S. Dorenbos, T. Zijlstra, T. Klapwijk, and V. Zwiller, Opt. Express 19, 24434 (2011).
  * Miki  _et al._ (2013) S. Miki, T. Yamashita, H. Terai, and Z. Wang, Opt. Express 21, 10208 (2013).
  * Ou (2007) Z.-Y. J. Ou,  _Multi-photon quantum interference_ (Springer, 2007).
  * Xiang  _et al._ (2013) G. Y. Xiang, H. F. Hofmann, and G. J. Pryde, Sci. Rep. 3, 2684 (2013).
  * Yepiz-Graciano  _et al._ (2020) P. Yepiz-Graciano, A. M. A. Martinez, D. Lopez-Mago, H. Cruz-Ramirez, and A. B. U’Ren, Photon. Res. 8, 1023 (2020).
  * Kaiser  _et al._ (2018) F. Kaiser, P. Vergyris, D. Aktas, C. Babin, L. Labont , and S. Tanzilli, Light Sci. Appl. 7, 17163 (2018).
  * Quan  _et al._ (2016) R. Quan, Y. Zhai, M. Wang, F. Hou, S. Wang, X. Xiang, T. Liu, S. Zhang, and R. Dong, Sci. Rep. 6, 30453 (2016).
  * Dinani  _et al._ (2016) H. T. Dinani, M. K. Gupta, J. P. Dowling, and D. W. Berry, Phys. Rev. A 93, 063804 (2016).
  * Whittaker  _et al._ (2017) R. Whittaker, C. Erven, A. Neville, M. Berry, J. L. O’Brien, H. Cable, and J. C. F. Matthews, New J. Phys. 19, 023013 (2017).
  * Dowling (2008) J. P. Dowling, Contemp. Phys. 49, 125 (2008).
  * Taylor and Bowen (2016) M. A. Taylor and W. P. Bowen, Phys. Rep. 615, 1 (2016).