# Modeling Resilience of Collaborative AI Systems

Diaeddin Rimawi<EMAIL_ADDRESS>0000-0003-3791-399X Free University of
Bozen-Bolzanopiazza Università, 1BolzanoItaly39100 ,  Antonio Liotta
<EMAIL_ADDRESS>0000-0002-2773-4421 Free University of Bozen-
Bolzanopiazza Università, 1BolzanoItaly39100 ,  Marco Todescato
<EMAIL_ADDRESS>0000-0003-1449-5692 Fraunhofer
ItaliaBolzanoItaly39100 and  Barbara Russo<EMAIL_ADDRESS>0000-0003-3737-9264 Free University of Bozen-Bolzanopiazza Università,
1BolzanoItaly39100

(2024)

###### Abstract.

A Collaborative Artificial Intelligence System (CAIS) performs actions in
collaboration with the human to achieve a common goal. CAISs can use a trained
AI model to control human-system interaction, or they can use human
interaction to dynamically learn from humans in an online fashion. In online
learning with human feedback, the AI model evolves by monitoring human
interaction through the system sensors in the learning state, and actuates the
autonomous components of the CAIS based on the learning in the operational
state. Therefore, any disruptive event affecting these sensors may affect the
AI model’s ability to make accurate decisions and degrade the CAIS
performance. Consequently, it is of paramount importance for CAIS managers to
be able to automatically track the system performance to understand the
resilience of the CAIS upon such disruptive events. In this paper, we provide
a new framework to model CAIS performance when the system experiences a
disruptive event. With our framework, we introduce a model of performance
evolution of CAIS. The model is equipped with a set of measures that aim to
support CAIS managers in the decision process to achieve the required
resilience of the system. We tested our framework on a real-world case study
of a robot collaborating online with the human, when the system is
experiencing a disruptive event. The case study shows that our framework can
be adopted in CAIS and integrated into the online execution of the CAIS
activities.

Collaborative AI Systems, Resilience, Online Learning, Decision Making

††conference: 3rd International Conference on AI Engineering – Software
Engineering for AI; April 2024; Lisbon, Portugal††journalyear: 2024

##  1\. Introduction

A Cyber-physical system (CPS) has heterogeneous hardware-software components
that collaborate to deliver real-time services. The complexity of CPSs is
different from one domain to the other. A Collaborative Artificial
Intelligence System (CAIS) is a CPS that performs collaborative actions with
humans in a shared environment to achieve a common goal,
(bonfanti_gresilience_2023, 9, 8). Such systems need to be resilient and
recover fast from any disruptive events that degrade their performance and
eventually prevent them from providing their services within the due schedule,
(henry_generic_2012, 3, 9).

The AI component in CAIS represents the core decision-making instrument that
guides system-human interactions. This component can be trained either i)
using historical data (offline training), or ii) using data from run-time
(online training), (rimawi_green_2022, 7, 9, 8). When the AI component is
trained online, the CAIS resilience is of paramount importance, (kadgien_cais-
dma_2024, 8). A disruptive event may indeed affect the ability of the AI
component to restore its prediction accuracy in an acceptable time and, in
turn, leading to the degradation of overall the system’s performance and extra
interactions with the human to recover from the event
(camilli_microservices_2022, 1). Thus, CAISs managers need to ensure their
system’s resilience by monitoring its performance. Monitoring performance
allows the managers to understand if their system was able to detect
performance degradation and recover back to an acceptable performance state,
(henry_generic_2012, 3, 2).

The major goal of this paper is to provide a new framework that models CAIS’s
performance while learning online and facing a disruptive event. Our model
helps in showing the performance evolution of CAIS, the performance
degradation that may occur after a disruptive event, and rules and measures to
assess CAIS’s resilience upon disruptions. Our framework abstracts the states
of CAIS into a learning state and an operational state. CAIS enters the
learning state when first the AI model accuracy is not high enough to make a
trusted decision and human interaction is required. Second, when the human
intervenes to correct a false positive decision. The CAIS is in the
operational state when it completes its service autonomously. Our framework
tracks CAIS’s states during run-time and finds the ratio between the number of
the operational state over the number of the learning state during a specific
time frame, and provide a set of measures to evaluate the performance across
the states. We applied our framework by carrying out an experiment with a
real-world CAIS case study. Our CAIS is a collaborative robot that learns to
classify objects based on their colors in an online learning process. During
the experiment, we disrupted the learning process by turning off the
supporting lights of the RGB Camera sensor capturing the objects to be
classified. Our framework has shown the incremental evolution of the system
performance from a steady state, the performance degradation after the
disruptive event occurring within a disruptive state, and the recovery during
a recovered state. The model of resilience we have obtained also shows the
system entering into a second disruptive state caused by removing the cause of
the disruptive event. After such state again the system enters into recovered
state.

Our major contribution in this work can then be summarized as follows:

  1. (1)

We design a novel framework that models the evolution of CAIS performance. Our
framework defines the rules and the measurements to automatically track CAIS
performance, and detect performance degradation and eventual anomalies.

  2. (2)

We define specific measurements and rules to describe the system performance
at each state of its evolution. Our rules assess CAIS’s resilience upon
disruptive events and set the comparison baseline for future research on
resilient CAISs.

  3. (3)

We automated our framework with a real-world CAIS demonstrator, a
collaborative robotic arm with an AI component that online learns from human
gestures. We then performed an experiment in a laboratory setting, and we
obtained a model of a performance evolution over different states of the
system.

The rest of this paper is structured as follows. In Sec. 2 we overview the key
concepts concerning this work and the proposed methodology. In Sec. 3 we
introduce the research questions, our CAIS demonstrator, our experiment, and
the takeaways. In Sec. 4 we discuss threats to validity. In Sec. 5 we discuss
the related work. Finally, in Sec 6 we state our conclusion and the future
work.

##  2\. Methodology

In this paper, we aim to model the resilience111Resilience is a non-functional
property that enables a system to recover its performance after an event has
degraded it, (henry_generic_2012, 3). of a CAIS while it learns online from a
human upon disruptive events. Hence, our method starts by understanding the
online learning process, the different variables, and the states a CAIS passes
through to achieve resilience. With this knowledge, our method defines a
framework and its measures to support CAIS managers in understanding the
resilience of their system. The result is a model as in Fig. 3 equipped with
measurements as illustrated in the following.

We describe the online learning process by means of the CAIS of our case
study. As such, in the following, we refer to CAIS tasks as specific AI
classification tasks, but the online process can be similarly defined for
other AI tasks. Our robot learns object classification by color. Then, based
on the learning, the AI component autonomously recommends with a specific
probability an action to perform the classification. Fig. 1 shows the online
learning process in a CAIS, where the data is collected by the CAIS sensors,
and then preprocessed to extract the learning features. With the learning
features, the AI model estimates the probabilities of the classification
classes and computes their maximum (a.k.a. the confidence level of
prediction), $\epsilon\in[0,1]$ so that the class with the highest probability
is chosen by the robotic arm. To avoid cases in which more than one class can
be chosen as the probabilities are similar, the desired confidence level $K$
is set (e.g., $K=0.4$ for three classes). The value of $\epsilon$ is then
compared with $K$. If $\epsilon$ is less than $K$, the human is prompted to
perform the task, otherwise, the CAIS autonomously does it. Moreover, the
human has the possibility to intervene the robot misclassifies an object
(false positives) switching the robot to the “learning from human” mode.

Figure 1. Online Learning Process in CAIS.

###  2.1. Performance Evolution

AI models evolve with learning data. In our context, an AI component starts
without any knowledge about the action to perform, which means that $\epsilon$
is zero. As CAIS keeps running, the accuracy of the AI model increases with
$\epsilon$ in a positive relationship. With higher values of $\epsilon$, the
number of autonomous actions in a time frame increases, and, correspondingly,
the number of human actions decreases. The ratio between these two numbers
characterizes the interaction between the autonomous components in a CAIS and
the human. The goal of CAIS is to keep this ratio at its maximum. When CAIS
experiences a disruptive event, the data and the online learning process might
be affected. Therefore, with a disruptive event, the value of $\epsilon$ and
the model accuracy may decrease. When the $\epsilon$ value decreases below
$K$, the ratio between the number of autonomous actions and the number of
human actions decreases, indicating a degradation of the CAIS performance.
Hence, for CAIS’s AI model to be able to recover back its ability to perform
its tasks autonomously, it requires further training by switching to the
learning mode. The performance curve shows the performance degradation caused
by the disruptive event, and recovering back to an acceptable performance
level. This evolution shows evidence about CAIS resilience upon the current
disruptive event, as described in the following.

###  2.2. Modeling Performance

To model the CAIS performance during learning online from the human, we plot
the ratio between the number of times CAIS operates autonomously and the
number of times the human operates in a specific time frame. Fig. 2 shows the
ratio over the states of the system. Our model initiates a first-in first-out
queue with a time frame size of zeros. When a new object arrives, CAIS’s AI
model estimates the classification probability ($\epsilon$), then it enqueues
a zero if $\epsilon<K$ (learning state), and one wait to complete the
operating state to enqueue one (to allow the human to intervene in case of
false positives). After enqueueing a new value, our model dequeues the value
on top of the queue to keep only a time frame size elements. Then we define
our measurement, the Autonomous Classification Ratio (ACR) as the ratio of the
autonomous actions in a time frame. We compute the ACR by finding the queue
sum over the time frame size. For each new object, we plot the ACR value,
which results in the final performance model.

Figure 2. Framework execution states.

Fig. 3, shows the expected resilience model of CAIS performance evolution when
encountering a disruptive event. CAIS learns the classification until it
enters the $1^{st}$ Steady State. The start of this state is indicated by a
set of autonomous classifications for a whole time frame ($ACR=1$). In the
$1^{st}$ Steady State, we define the ACR Threshold, which is the minimum ACR
value. This value represents the level of performance when the AI component is
learning autonomously. A value of $ACR=0$ indicates the end of the $1^{st}$
Steady State and the start of the $1^{st}$ Disruptive State after a disruptive
event has occurred. During the $1^{st}$ Disruptive State and due to the
policies of the online learning process, CAIS aim to learn again to restore
the performance to an acceptable level ($ACR\geq ACR\,\,Threshold$). When the
level is reached, the CAIS enters into the Recovered State. The recovery at
this state, represents the system resilience during disruption. After
recovery, the cause of the disruptive event can be removed and the CAIS enters
with the same modalities as above in a Final State, which again includes a
$2^{nd}$ Disruptive State and a $2^{nd}$ Steady State. In the Final State, the
CAIS may behave differently than in the previous disruptive and steady states,
as it may maintain some historical memory of the disruptive event. It is
important to note that this cycle of states are repeatable per each disruptive
event that degrades the system performance. Table 1 summarizes each of the
performance states of CAIS to represent resilience.

Figure 3. CAIS Resilience Model. Table 1. Performance States Definitions. State Name |  Definition
---|---
$1^{st}\,\,Steady\,\,State$ |  After CAIS learns enough to start making decisions autonomously with a higher rate than requiring the human.
$ACR\,\,Threshold$ |  The minimum ACR value during the $1^{st}\,\,Steady\,\,State$.
$1^{st}\,\,Disruptive\,\,State$ |  Starts after a disruptive event has occurred, leading to performance degradation.
$Recovered\,\,State$ |  When a performance degradation is restored back to an acceptable value $ACR>ACR\,\,Threshold$.
$Final\,\,State$ |  Starts after the removal or fix of the disruptive event source.
$2^{nd}\,\,Disruptive\,\,State$ |  The performance degradation happens due to the addiction to the disruptive event.
$2^{nd}\,\,Steady\,\,State$ |  Recover back CAIS performance from degradation to an acceptable performance level.

We have defined a few measures (Table 2) and rules to support CAIS managers in
using our framework and assessing the CAIS resilience, as described in the
following. The State Length is an experimental rule that defines the minimum
number of iterations (each new object initiate a new iteration) of a state. We
set the state length as equal to the number of iterations in the $1^{st}$
Steady State. Then, in a disruptive state, we examine a period of close to or
equal to this number of iterations, starting from the last ACR point under the
ACR Threshold, to understand whether CAIS has recovered or not. Secondly, the
Points Under the Threshold (PUT) and the Points Above the Threshold (PAT).
Each of these measures indicates how well CAIS resilience is. The goal of
CAIS’s managers is to maximize the PAT ratio over the PUT. Finally, from a
business perspective, a goal of CAIS managers is to minimize the human
efforts. Thus, we define the Human Interaction Ratio (HI Average) during the
disruptive state to reduce human efforts. We defined the HI Average as the
average of the human interactions to classify an object during the disruptive
state.

Table 2. Resilience Rules and Measurements. Criteria |  Definition
---|---
State Length |  The length of the $1^{st}$ Steady State, which defines the minimum number of iterations to identify the recovered state.
PUT |  The number of ACR points under the ACR Threshold in the disruptive state.
PAT |  The number of ACR points above the ACR Threshold in the disruptive state.
PUT to PAT Ratio |  The ratio between the PUT to the PAT.
HI Average |  The average of human interactions’ in the disruptive state.

##  3\. Case Study

In this paper, we introduce our real-world CAIS case study. The CAIS
demonstrator, shown in Fig. 4, is a robotic arm (3) responsible for
classifying objects based on their color. The robot learns the object box (4)
by tracking the human movement (6) and mapping it with its color histogram.
The robot has two vision sensors, one is placed above the conveyor belt (2),
and another (5) tracks the human skeleton. The two vision sensors communicate
with computer vision software (1).

To understand if CAIS is resilience upon disruptive events by modeling CAIS
performance, we aim to answer the following research questions:

  * •

RQ1. How does CAIS’s performance in an online learning process evolve? To
answer this question, we will show a rendered plot of CAIS performance
learning from the human, to explain the performance evolution in each state.

  * •

RQ2. What are the rules and measures that indicate if CAIS is resilient upon
disruptive events or not? To answer this question, we will examine the set of
measurements and rules to tell if the resulting performance represents a
resilient behavior or not. Additionally, these measurements can set the
baseline for future research that aims to create more resilient CAISs.

To address these questions, we perform an experiment where we automate our
framework to track the performance evolution of our CAIS in classifying
objects of three colors (Red, Green, and Blue). The robot learns the three
classes (Box1, Box2, and Box3) from tracking the human hand movements and
gestures. We enforce a disruptive event to cause a performance degradation to
the classification learning process by turning off the supporting lights of
the RGB Camera (Fig. 4 (2)). The experiment then runs in iterations, where
each iteration represents a new object entered to be classified. To avoid the
learner being trained for one color more than the other, the objects will
enter the conveyor belt sorted red, green then blue, with the same quantity.

Figure 4. Collaborative Robot Learning from Demonstrations.

###  3.1. Experiment Execution

The experiment is executed in iterations and for each iteration, we collect
the AI prediction probability ($\epsilon$), and the iteration state
(learning/operating state). To choose the value of $K$, we consider the three
classes we have ($1/3\simeq 0.33$). Thus, we chose a value that is too low
($K=0.33$), which led to closer values of classification probabilities, and
thus, we had two classes with almost the same classification probability. On
the other hand, we chose a high-value ($K=0.50\%$), which required a longer
time of training to reach a steady state. The value that best fits our needs
is ($K=0.40$), which helps avoid the two problems.

During the execution, CAIS performance is expected to transit through
different states in which the AI component online learns its task. Fig. 5
shows the performance evolution of our CAIS demonstrator (collaborative robot)
as a function of time frame, where each time frame size is equal to five
overlapping iterations (or five objects). The figure shows the ACR values over
208 iterations, which shows all the different performance evolution states. We
triggered a disruptive event by switching off the lights over the conveyor
belt supporting the RGB Camera. This disrupts the robot’s vision and the color
histogram extracted from the object image. In this experiment, our robot was
able to recover both when we switched off and back on the light. The
performance evolution model shows that the system will manage to recover back
to a recovered state during the disruptive state. It will also be able to go
back to a steady state after removing the cause of the disruptive event.

Figure 5. Modeling of CAIS Performance Behavior.

###  3.2. Takeaways

In this experiment, we equipped our real-world demonstrator with our
framework. The results of the performance evolution model help address our
research questions as follows:

Answer to RQ1. The resilient behavior of CAIS is its ability to overcome
unforeseen disruptive events and recover its performance from a degradation
state to an acceptable state. Fig. 5 shows the performance evolution of our
CAIS demonstrator (collaborative robot). The performance evolution shows our
robot starting with $ACR=0$, and then starting to learn from the human
gestures until entering the $1^{st}$ Steady State. In the $1^{st}$ Steady
State, the performance formulates a pattern of going up and down, which is
expected due to slight changes of the environment (for example the object
position on the conveyor belt). The ACR Threshold is calculated to (0.40),
which is the minimum value in the $1^{st}$ Steady State. Then the disruptive
event was enforced, and the performance degraded to $ACR=0$ indicating the
beginning of the $1^{st}$ Disruptive State. During the $1^{st}$ Disruptive
State, our robot starts learning again until it manages to recover its
performance and enter the Recovered State. To complete the cycle, we removed
the source of the disruptive event (Fix Event - turn back the light on), where
we trigger a $2^{nd}$ Disruptive State in the Final State. The robot has
replicated the same behavior and learned to recover back into a $2^{nd}$
Steady State. Our experiment has shown the same performance evolution as the
expected performance evolution in Fig. 3.

Answer to RQ2. The answer to this question shows the feasibility of our rules
and measures defined in Table 2. First, the $State\,\,Length$, in our
experiment, the length of the $1^{st}$ Steady State was $34$ iterations, thus
when the $1^{st}$ Disruptive State started, we had to wait for at least $34$
iterations after the last point under the threshold. This results in a
Recovered State of length $37$, and the $2^{nd}$ Steady State, $40+$ since it
is the last state. The experiment shows a total number of $PUT=23$, and the
$PAT=59$, which makes the ratios equal $0.28$ PUT to $0.72$ PAT, the higher
ratio of PAT the better. Finally, for the $HIAverage$ we need to find the
number of times CAIS requires a human classification, in other words, the
number of times entering the learning state. In our experiment, the number of
human interactions was equal to $44$ and the number of iterations is $82$,
thus $HIAverage=44/82=0.54$. These measures summarize the ability of our CAIS
to be resilience to the disruptive event of turning off the lights.

##  4\. Threats to Validity

The major threat to the validity of our study is the internal, conclusion, and
external threats. Internal Validity. is related to the factors affecting the
research findings. In our experiment, we had to set the size of the time frame
to compute ACR. We use a window size that is not too small and not too long,
which is five. However, we plan to run a stress experiment with various sizes
of time frames to check the effect on the final results. Conclusion Validity.
concerns the appropriateness of the measurements analysis and inferences drawn
from the data. Our resilience rules and measurements are constructed from our
observation of our experimental protocol. However, to have a meaningful
representation, we consider them to be comparison rules for future research
studies to enhance resilience in CAIS. External Validity. is related to the
degree of support for the generalization of the theoretical results. The
nature of our research is mainly exploratory, and generalization beyond our
case study is needed for consolidating our claims. Thus, we plan to run more
experiments with another real-world demo we have in-house and with simulators.

##  5\. Related Work

We have reviewed existing literature according to i) resilience and ii) online
learning in the context of CAIS. In the following, we overview them.

Resilience. The state of the art has proposed several models to address
resilience, like (januario_distributed_2019, 4) using a multi-agent
model,(liu_distributionally_2022, 5) using a tri-optimization model, and
(zarandi_detection_2020, 11) using a deep learning model, to mention a few.
All proposed models aim to detect performance degradation or the disruptive
event that caused it and return the system to an acceptable performance state.
Disruptive events vary from one system to the other, for example, the
disruptive event can be a security vulnerability of the system
(zarandi_detection_2020, 11, 5), defect in the system cyber or physical parts
(januario_distributed_2019, 4), from an oracle component of the system such as
humans (bonfanti_gresilience_2023, 9), or it can come from the system
surrounding environment, (henry_generic_2012, 3).

Online learning. CAIS collaboration with humans is a process of transferring
knowledge from the human to the autonomous components of CAIS.
(wang_facilitating_2018, 10) has defined three stages for human-system
learning start teaching the robot through voice instructions, the robot then
learns, and finally, the human and robot collaborate. However, updating the
learning will require us to manually turn to the learning stage, which is not
the case in our dynamic continuous online learning. A recent survey of
learning strategy for robot-human collaboration by (mukherjee_survey_2022, 6)
has discussed several input modes, such as gaze, gesture, voice commands, and
facial emotions. Specifically, our system uses human gestures for object
classification, which requires scene understanding, (mukherjee_survey_2022,
6). The survey illustrates that the majority of the work is about the safety
and physical safety of the human-system collaboration.

##  6\. Conclusion and Future Work

In conclusion, this work proposes a novel framework to model the performance
evolution of CAIS, by tracking the different states of the online learning
process with the human. Additionally, it defines the measures and rules to
assess CAIS resilience over time. Finally, it automatizes the framework and
its measures by designing and executing an experiment with a real-world
collaborative robot. The results show the framework’s ability to render the
performance evolution of our CAIS passing through the different performance
states (steady, disrupted, and recovered).

In future work, we plan to execute additional experiments to generalize our
results to other real-world and simulated case studies. We are also designing
comparative experiments to automatically support CAIS in decision-making, to
enhance CAIS resilience based on the defined measures of this study. Moreover,
we plan to reconsider further human attributes, for example human energy.

###### Acknowledgements.

This work is supported by the doctoral fellowship of the National Operational
Program (PON) Research and Innovation 2014-2020 (CCI 2014IT16M2OP005), ESF
REACT-EU resources.

## References

  * (1) Matteo Camilli et al.  “Microservices Integrated Performance and Reliability Testing”  In _IEEE/ACM International Conference on Automation of Software Test, AST@ICSE 2022, Pittsburgh, PA, USA, May 21-22, 2022_ ACM/IEEE, 2022, pp. 29–39  DOI: 10.1145/3524481.3527233
  * (2) Silvia Colabianchi et al.  “Discussing resilience in the context of cyber physical systems”  In _Comput. Ind. Eng._ 160, 2021, pp. 107534  DOI: 10.1016/j.cie.2021.107534
  * (3) Devanandham Henry and Jose Emmanuel Ramirez-Marquez  “Generic metrics and quantitative approaches for system resilience as a function of time” Publisher: Elsevier  In _Reliability Engineering & System Safety_ 99, 2012, pp. 114–122
  * (4) Fábio Januário, Alberto Cardoso and Paulo Gil  “A Distributed Multi-Agent Framework for Resilience Enhancement in Cyber-Physical Systems”  In _IEEE Access_ 7, 2019, pp. 31342–31357  DOI: 10.1109/ACCESS.2019.2903629
  * (5) Zhaoxi Liu and Lingfeng Wang  “A Distributionally Robust Scheme for Critical Component Identification to Bolster Cyber-Physical Resilience of Power Systems” Conference Name: IEEE Transactions on Smart Grid  In _IEEE Transactions on Smart Grid_ , 2022, pp. 1–1  DOI: 10.1109/TSG.2022.3147421
  * (6) Debasmita Mukherjee, Kashish Gupta, Li Hsin Chang and Homayoun Najjaran  “A survey of robot learning strategies for human-robot collaboration in industrial settings” Publisher: Elsevier  In _Robotics and Computer-Integrated Manufacturing_ 73, 2022, pp. 102231  URL: https://www.sciencedirect.com/science/article/pii/S0736584521001137?casa_token=yszVkvN-l0IAAAAA:JPbxX5XoHfObJxfkKujclcBV3sJ490FLr_SOytZP2GB8QaU8JT-3w3h1FZkEMJeVFWc_LFwCXA
  * (7) Diaeddin Rimawi  “Green Resilience of Cyber-Physical Systems”  In _2022 IEEE International Symposium on Software Reliability Engineering Workshops (ISSREW)_ IEEE, 2022, pp. 105–109
  * (8) Diaeddin Rimawi, Antonio Liotta, Marco Todescato and Barbara Russo  “CAIS-DMA: A Decision-Making Assistant for Collaborative AI Systems”  In _Product-Focused Software Process Improvement_ 14483 Cham: Springer Nature Switzerland, 2024, pp. 183–199
  * (9) Diaeddin Rimawi, Antonio Liotta, Marco Todescato and Barbara Russo  “GResilience: Trading Off Between the Greenness and the Resilience of Collaborative AI Systems”  In _Testing Software and Systems_ 14131 Cham: Springer Nature Switzerland, 2023, pp. 266–273
  * (10) Weitian Wang et al.  “Facilitating human–robot collaborative tasks by teaching-learning-collaboration from human demonstrations” Publisher: IEEE  In _IEEE Transactions on Automation Science and Engineering_ 16.2, 2018, pp. 640–653  URL: https://ieeexplore.ieee.org/abstract/document/8403899/?casa_token=Lii8CTZluxcAAAAA:sXHJSxcT1hjADVradqjQf1PjvXWlo4YNjpYZc2l88H_XyXIXL0-7MH1qo6HIe14FR7tlFy7teA
  * (11) Zohre Nasiri Zarandi and Iman Sharifi  “Detection and Identification of Cyber-Attacks in Cyber-Physical Systems Based on Machine Learning Methods”  In _2020 11th International Conference on Information and Knowledge Technology (IKT)_ , 2020, pp. 107–112  DOI: 10.1109/IKT51791.2020.9345627


††thanks: These authors contributed equally to this work.††thanks: These
authors contributed equally to this work.

# In vacuum metasurface for optical microtrap array

Donghao Li  State Key Laboratory of Quantum Optics and Quantum Optics Devices,
Institute of Opto-electronics, Shanxi University, Taiyuan, Shanxi 030006,
China  Collaborative Innovation Center of Extreme Optics, Shanxi University,
Taiyuan, Shanxi 030006, China  Qiming Liao  School of Optics and Photonics,
Beijing Engineering Research Center of Mixed Reality and Advanced Display,
Beijing Institute of Technology, Beijing 100081, China  Beining Xu  State Key
Laboratory of Quantum Optics and Quantum Optics Devices, Institute of Opto-
electronics, Shanxi University, Taiyuan, Shanxi 030006, China  Yaoting Zhou
State Key Laboratory of Quantum Optics and Quantum Optics Devices, Institute
of Opto-electronics, Shanxi University, Taiyuan, Shanxi 030006, China  Keyu
Qin  State Key Laboratory of Quantum Optics and Quantum Optics Devices,
Institute of Opto-electronics, Shanxi University, Taiyuan, Shanxi 030006,
China  Zhongxiao Xu<EMAIL_ADDRESS>State Key Laboratory of Quantum
Optics and Quantum Optics Devices, Institute of Opto-electronics, Shanxi
University, Taiyuan, Shanxi 030006, China  Collaborative Innovation Center of
Extreme Optics, Shanxi University, Taiyuan, Shanxi 030006, China  Heng Shen
<EMAIL_ADDRESS>State Key Laboratory of Quantum Optics and Quantum Optics
Devices, Institute of Opto-electronics, Shanxi University, Taiyuan, Shanxi
030006, China  Collaborative Innovation Center of Extreme Optics, Shanxi
University, Taiyuan, Shanxi 030006, China  Lingling Huang
<EMAIL_ADDRESS>School of Optics and Photonics, Beijing Engineering
Research Center of Mixed Reality and Advanced Display, Beijing Institute of
Technology, Beijing 100081, China

###### Abstract

Optical tweezer arrays of laser-cooled and individual controlled particles
have revolutionized the atomic, molecular and optical physics, and they afford
exquisite capabilities for applications in quantum simulation of many-body
physics, quantum computation and quantum sensing. Underlying this development
is the technical maturity of generating scalable optical beams, enabled by
active components and high numerical aperture objective. However, such a
complex combination of bulk optics outside the vacuum chamber is very
sensitive to any vibration and drift. Here we demonstrate the generation of
3$\times$3 static tweezer array with a single chip-scale multifunctional
metasurface element in vacuum, replacing the meter-long free space optics.
Fluorescence counts on the camera validates the successfully trapping of the
atomic ensemble array. Further, we discuss the strategy to achieve low
scattering and crosstalk, where a metasurface design featuring dual-wavelength
independent control is included. Our results, together with other recent
development in integrated photonics for cold atoms, could pave the way for
compact and portable quantum sensors and simulators in platforms of neutral
atom arrays.

Assembling the individually controlled quantum objects lies in the heart of
quantum physics. Recent decade has witnessed great developments in the
experimental control over arrays of individually neutral atoms Endres et al.
(2016); Barredo et al. (2016); Kim et al. (2016); Browaeys and Lahaye (2020);
Sheng et al. (2022), emerging as a promising platform for quantum computation
Bluvstein et al. (2022); Evered et al. (2023); Ma et al. (2023); Graham et al.
(2022); Bluvstein et al. (2024), simulation of many-body physics Bernien et
al. (2017); Semeghini et al. (2021); Ebadi et al. (2022); Scholl et al.
(2021); Keesling et al. (2019); Bluvstein et al. (2021); Chen et al. (2023)
and quantum sensing. In this configuration, single neutral atoms or atomic
ensembles are initially trapped and assembled in arrays of tightly focused
laser beams, so-called static optical tweezers, while fast moving and single
addressing are enabled by dynamical optical tweezers. To generate such static
microtrap arrays, one typically leverages active components, such as spatial-
light-modulator (SLM) Barredo et al. (2016); Ebadi et al. (2021)and digital-
micromirror devices (DMD)Wang et al. (2020), or microlens arraysTanaka (2017);
Schäffner et al. (2020) in combination with conventional high-numerical-
aperture (NA) objectives. A meter-long optical path defined by these
elaborated elements is however susceptible to platform vibrations and drift,
degrading the beam-pointing stability and system performance.

In practice, integrated photonics has recently benefited atomic, molecular and
optical (AMO) experiments through miniaturizing the beam delivery components
and improving the reliability. For instance, silicon nitride
($\text{Si}_{3}\text{N}_{4}$) waveguides and grating couplers were exploited
to deliver the magneto-optical trap (MOT) beams in cold atoms Isichenko et al.
(2023) and all wavelengths of light necessary to control the ion qubit Mehta
et al. (2020); Niffenegger et al. (2020). Yet multifunctional and scalable
optical components is highly demanded, especially in applications outside the
laboratory such as mobile quantum sensors.

Metasurfaces, two-dimensional structures composed of optically thin arrays of
subwavelength dielectric or metallic scatters, offer the opportunity to fully
control light He et al. (2024a); Xu et al. (2024); Yin et al. (2024); He et
al. (2024b); Shao et al. (2024); Su et al. (2024); Huang et al. (2024).
Pioneering works have demonstrated their capabilities of addressing the
specific tasks aided with bulk free-space optic components, such as high-
efficiency MOT beam delivery Zhu et al. (2020) and high-NA optical objective
Hsu et al. (2022). However, multifunctional photonic elements are feasible by
manipulating the spatial patterns of nanostructures, which enables the beam
steering, local control of optical polarization and enhancement of the
emission and detection.

Here, we report the demonstration of optical metasurfaces enabled laser dipole
trapping of 87Rb atomic arrays. To our best knowledge, it is the first time to
generate the static optical tweezer array using a single chip-scale photonic
element in vacuum, which replaces the meter-long free-space optics including
active component (e.g. SLM and AOD) and high-NA objectives. In the experiment,
a collimated laser beam with the waist about 1 mm illuminates the metasurface
chip in vacuum, producing a 3$\times$3 optical tweezer array to trap atomic
ensembles. Fluorescence counts on the charge-coupled-device (CCD) camera
indicates our successful trapping. We further discuss the strategy to achieve
low scattering and crosstalk, where a metasurface design featuring wavelength-
selective control is presented.

Figure 1: Metasurface design and characterization. (a) The phase profile of
the metasurface generated using the optimized GS algorithm. (b) Transmission
coefficients and phase obtained by sweeping the diameter of a nanopillar
within a unit cell. (c) Top-view and side-view SEM images. (d) The schematic
diagram of the focal spot array focusing under 852 nm incident light and the
experimentally captured image of the focal plane. (e) The comparison of the
energy distribution of the middle row of focal spots between the experimental
results and the simulation results.

## Metalens design and characterization

To create a 3 $\times$ 3 focal spots array with a designed focal length of 1
mm and a spacing between each spot of 5 $\mu$m, we employ a modified
Gerchberg-Saxton (GS) algorithm to generate the desired phase-only hologram.
As a classical and efficient holography algorithm, the GS algorithm has the
advantages of high adaptability and reconstruction accuracy. To enable the
flexible design of the focal plane position, we propose an optimization scheme
that introduces angular spectrum diffraction as the propagation kernel in the
GS algorithm, transforming the design space from the Fourier plane to the
Fresnel domain. By replacing the fast Fourier transform with angular spectrum
diffraction and specifying the diffraction distance as the focal length, we
obtained the phase profile as shown in Fig. 1a that can achieve a high-quality
multiple focal spots array. Due to the introduction of diffraction theory into
calculation, this method can produce the focal spots array with more uniform
energy distribution and higher signal-to-noise ratio compared with the
traditional design method, which only simply superimposing multiple lens
phases. See Supplementary Materials.

Then we design and realize a polarization-independent metasurface using
amorphous silicon nanopillars patterned on a fused silica substrate which can
operate at $\lambda$ = 852 nm, the working wavelength for trapping rubidium
atoms. We calculate the transmission coefficient and phase responses of meta-
atoms with different diameters at 852 nm using the rigorous coupled-wave
analysis (RCWA) method. To cover the phase shifts from 0 to $2\pi$ with high
efficiency, the period and height of the nanopillars are chosen as 500 and 600
nm respectively and the diameter ranges from 80 nm to 180 nm. As shown in
Fig.1b, the selected structures have transmission coefficients exceeding 0.9
and can cover the phase range from 0 to $2\pi$. By engineering the
nanopillars, the desired phase profile can be converted into a diverse
diameter distribution across the metasurface. The fabricated metasurface is
composed of 2002 $\times$ 2002 nanopillars using electron beam lithography and
reactive ion etching, and the corresponding scanning electron microscopy
images with side and top views are shown in Fig. 1c.

Figure 2: Metasurface trapping setup. (a) Overview of trapping and imaging. A
laser beam at $\lambda$=840 nm passes through the metasurface and then a 3
$\times$ 3 tweezer array is created. The atomic fluorescence at $\lambda$=780
nm is collected by a objective lens and imaged on a EMCCD camera. We use the
forward collection method that several filters are used before the camera (not
shown here). (b) Photograph of the metasurface and its mounting structure. The
metasurface is first glued to a glass plate, which is fixed on the flange of
glass cell. (c) Schematics of the laser illumination. A collimated beam first
passes through an adjustable lens, resulting in a focus before the metasurface
sample. This diverging beam illuminates the metasurface and consequently
produces the trap tweezers. By moving the adjustable lens, we can shift the
focus position to ensure the overlap with 3D MOT.

Subsequently, we test the focusing performance of the fabricated sample under
incident light with a wavelength of 852 nm. At a distance of 1.01 mm from the
metasurface, we obtain the clearest image of a 3 $\times$ 3 focal spots array,
as shown in Fig. 1d, with the average distance between the focal spots
characterized as 5.2 $\mu$m and an average full width at half maximum (FWHM)
of 3.03 $\mu$m. For comparison, we simulate the energy distribution of the
focal points in the middle row by using angular spectrum diffraction method.
It can be observed in Fig. 1e that the experimental results exhibit a more
uniform energy distribution, which may be due to phase deviations from the
design values caused by fabrication errors, resulting in partial dispersion of
the energy that should have been entirely focused at the focal spots. This
also leads to a larger FWHM in the experimental results and a 0.2 $\mu$m
difference in the spacing between focal spots compared to the design value of
5 $\mu$m. We also characterized the energy distribution of the focal spots in
other rows and columns, as well as the energy distribution in the x-z plane.
More details can be found in See Supplementary Materials. The experimental
results demonstrate that the metasurface can generate a uniform focal spots
array, providing feasibility for subsequent optical tweezer experiments.

## Optical metasurface-enabled trapping of atom array

A diagram of the atom trapping through optical metalens is illustrated in Fig.
2a. The metasurface sample is mounted inside a glass vacuum cell (24
mm$\times$28 mm$\times$100 mm) with an antireflection coating on the outside
(Fig. 2b), and a high vacuum pressure of $1\times 10^{-6}$ Pa is maintained
with an ion pump. The experiment starts with the three-dimensional MOT, where
a cloud of rubidium atoms are cooled in the cross section of three retro-
reflected MOT laser beams. Such 1 mm-diameter MOT beams with a total power of
500 $\mu$W are red detuned for the 87Rb D2 $F$=2 to $F^{\prime}$=3 transition
by 19 MHz, and are spatially overlapped with repump beams at a power of 50
$\mu$W resonant with D1 $F$=1 to $F^{\prime}$=1 transition. The axial magnetic
field gradient is 16 G/cm through the trap center. This MOT loading lasts for
200 ms, and the absorption imaging time-of-flight (TOF) measurement give the
size of $1.2\times 10^{6}$ atoms and temperature of 160 $\mu$K.

Subsequently, a optical tweezer array is switched on, overlapping with the 3D
MOT. To produce this tweezer array, a laser beam is delivered from a
continuous wave Ti:S single frequency laser at $\lambda$ = 840 nm, and
spatially filtered by a polarization-maintaining single-mode fiber terminated
with zoom fiber collimator (Thorlabs ZC618APC-B). The normal incidence of this
beam at the metasureface sample induces a 3 $\times$ 3 beam array through the
optical diffraction. Notice that the optical diffraction on the sample is
polarization insensitive. And the trapped atom array is imaged in the
electron-multiplying charge-coupled device (EMCCD, Andor iXon Ultra) by
collecting the fluorescence via an objective (Mitutoyo G Plan Apo 50$\times$),
as shown in Fig. 2c. Since we collect the forward scattering fluorescence at
780 nm, three narrow-band filters (Lier - opto, LO78010254ZD) are used in
front of the camera to block the transmitted trap beam at 840 nm.

Figure 3: Imaging the atomic array in optical microtraps. (a) Fluorescence
counts distribution.The blue bars represent the background counts while the
orange bars represent the atomic fluorescence counts. (b) Fluorescence image
of the atomic array in the metasurface enabled optical tweezers.

## Imaging the atomic array

In practice, the designed focus plane is not employed when accounting for the
MOT configuration, and the associated reason is as follows. No wavelength-
selection in the design of metasurface sample causes the same diffraction of
MOT beams at 780 nm as that of trap beam at 840 nm, and all MOT beams thus
have to avoid the incidence on this optical element. We consider all the
factors, such as 3D MOT geometry, background scattering from the sample holder
and the probe beam delivery for trapped atom imaging, and finally decide to
shift the focus plane from 1 mm to 2.5 mm with respect to the metasurface by
using a lens (Fig. 2c). Experimental results show the average distance of
focal spots is about 1 $\mu$m, and a $1/e^{2}$ waist each spot is 5.5 $\mu$m.
More details can be found in Supplementary Materials.

In Fig. 3a, we plot a histogram of the fluorescence counts registered on the
EMCCD camera (400 ms) versus the occurrence from the red square region marked
in the insert (a single microtrap in the tweezer array, 25 pixels), where 500
loading interactions are averaged. The lower-count histogram peak in blue
represents the background signal with no atom, while the higher-count peak in
orange corresponds to the signal with atoms. However, the collision blockade
effect is prevented Schlosser et al. (2001, 2002) by the larger beam size (5.5
$\mu$m) induced by the focus plane shift with respect to the original design
(1 $\mu$m). Consequently, more than one atoms are loaded in each of the
optical tweezer. An averaged fluorescence image of 3$\times$3 atomic array is
exhibited in 3b. The inhomogeneity is attributed to the inhomogeneous
intensity distribution of trap beam array. And this is a consequence of the
focus plane shift as mentioned above. More details can be found in
Supplementary Materials.

## Discussion

In summary, we report the first demonstration of static optical tweezer array
with a single chip-scale metasurface sample in vacuum. A 3$\times$3 array of
atomic ensembles is confined in such optical microtraps, which is verified
through the fluorescence counts on the CCD camera. Our work sets an example of
integrated photonics facilitate cold atom system. In combination with
integrated optic 3D MOT Isichenko et al. (2023); Zhu et al. (2020), it offers
the possibility for portable quantum sensors with neutral atom array platform
in unprecedented ways Schäffner et al. (2024); Eckner et al. (2023); Shaw et
al. (2024).

Figure 4: New generation of dual-wavelength mtasurface design. (a) Blueprint
of the new experimental setup. The normal incidence of a trap beam at 1560 nm
on the metasurface sample forms a 3 $\times$ 3 tweezer array. While the
backward fluorescence is collected by the metasurface and imaged on the EMCCD.
(b) Focus spots formed by a collimated 1560 nm input beam. (c) Focus spot
formed by a collimated 780 nm input beam. For 1560 nm trap beam, metasurface
will generate a 3 $\times$ 3 tweezer array. For 780 nm atomic fluorescence,
metasurface acts as a high NA collective lens.

Despite the successful trapping, two key challenges for getting the best
performance exist. On one hand, the designed metasurface does not perform
distinct functions for the incident light and atomic excitation fluorescence
wavelengths. Therefore, a transmission-based system is required for
fluorescence collection, which undoubtedly increases the experimental
difficulty in background scattering. To address this issue, an improved scheme
is proposed, a polarization-independent dual-wavelength version. Consider the
dual-wavelength high power laser from Precilasers (780 nm and 1560 nm)
available in the lab, this scheme utilizes the strong optical field
manipulation capabilities of metasurface to achieve different functions at the
incident light at 1560 nm and the fluorescence light at 780 nm, respectively.
When 1560 nm light is incident on the metasurface, as shown in Fig. 4b, it
generates a 3$\times$3 focal spots array similar to the existing sample. For
the fluorescence at 780 nm, it only performs the focusing function as shown in
Fig. 4c. This allows for the backward collection of the excited fluorescence
(Fig. 4a), thereby simplifying the overall experimental complexity and
improving the collection efficiency.

To achieve polarization-independent dual-wavelength independent control in the
new design, various types of meta-atom structures will be combined, including
cross-shaped pillars, cylindrical pillars, and ring-shaped pillars, to meet
the design requirements. The phase response of these scanned structures needs
to simultaneously cover a phase space of 0-2$\pi$ at both 1560 nm and 780 nm
wavelengths. Subsequently, these structures will be encoded into the pre-
designed dual-wavelength phase profiles to realize a multi-focal array at 1560
nm and a focusing function at 780 nm.

On the other hand, shifting the focus plane is required as a result of the 3D
MOT configuration in the presented work, leading to the enlarged tweezer beams
that trap more than one atoms. In the updated design aforementioned, focal
length is 4 mm and the beam waist is 1 $\mu$m, where the collision blockade
condition is promised and the effect of stray light from the substrate of
metasurface sample is reduced. Further improvement in the loading rate can be
achieved through the polarization-gradient cooling (PGC), cooling the atomic
temperature to 40 $\mu$K (See supplementary materials for preliminary
results). This approach can directly applied to the generation of a large
scale atom array.

This work is supported by National Key R&D Program of China under Grant No.
2020YFA0309400, NNSFC under Grant No. 12222409 and 12174081, 11974228, and the
Key Research and Development Program of Shanxi Province (Grants No.
202101150101025). H. S. acknowledges financial support from the Royal Society
Newton International Fellowship Alumni follow-on funding (AL201024) of the
United Kingdom.

## References

  * Endres et al. (2016) M. Endres, H. Bernien, A. Keesling, H. Levine, E. R. Anschuetz, A. Krajenbrink, C. Senko, V. Vuletic, M. Greiner, and M. D. Lukin, Science 354, 1024 (2016).
  * Barredo et al. (2016) D. Barredo, S. de Léséleuc, V. Lienhard, T. Lahaye, and A. Browaeys, Science 354, 1021 (2016), publisher: American Association for the Advancement of Science.
  * Kim et al. (2016) H. Kim, W. Lee, H.-g. Lee, H. Jo, Y. Song, and J. Ahn, Nat. Commun. 7, 13317 (2016), ISSN 2041-1723.
  * Browaeys and Lahaye (2020) A. Browaeys and T. Lahaye, Nat. Phys. 16, 132 (2020), ISSN 1745-2473, 1745-2481.
  * Sheng et al. (2022) C. Sheng, J. Hou, X. He, K. Wang, R. Guo, J. Zhuang, B. Mamat, P. Xu, M. Liu, J. Wang, et al., Phys. Rev. Lett. 128, 083202 (2022), URL https://link.aps.org/doi/10.1103/PhysRevLett.128.083202.
  * Bluvstein et al. (2022) D. Bluvstein, H. Levine, G. Semeghini, T. T. Wang, S. Ebadi, M. Kalinowski, A. Keesling, N. Maskara, H. Pichler, M. Greiner, et al., Nature 604, 451 (2022), ISSN 1476-4687.
  * Evered et al. (2023) S. J. Evered, D. Bluvstein, M. Kalinowski, S. Ebadi, T. Manovitz, H. Zhou, S. H. Li, A. A. Geim, T. T. Wang, N. Maskara, et al., Nature 622, 268 (2023), ISSN 0028-0836, 1476-4687.
  * Ma et al. (2023) S. Ma, G. Liu, P. Peng, B. Zhang, S. Jandura, J. Claes, A. P. Burgers, G. Pupillo, S. Puri, and J. D. Thompson, Nature 622, 279 (2023), ISSN 0028-0836, 1476-4687.
  * Graham et al. (2022) T. M. Graham, Y. Song, J. Scott, C. Poole, L. Phuttitarn, K. Jooya, P. Eichler, X. Jiang, A. Marra, B. Grinkemeyer, et al., Nature 604, 457 (2022), ISSN 1476-4687.
  * Bluvstein et al. (2024) D. Bluvstein, S. J. Evered, A. A. Geim, S. H. Li, H. Zhou, T. Manovitz, S. Ebadi, M. Cain, M. Kalinowski, D. Hangleiter, et al., Nature 626, 58 (2024), ISSN 0028-0836, 1476-4687.
  * Bernien et al. (2017) H. Bernien, S. Schwartz, A. Keesling, H. Levine, A. Omran, H. Pichler, S. Choi, A. S. Zibrov, M. Endres, M. Greiner, et al., Nature 551, 579 (2017), ISSN 0028-0836, 1476-4687.
  * Semeghini et al. (2021) G. Semeghini, H. Levine, A. Keesling, S. Ebadi, T. T. Wang, D. Bluvstein, R. Verresen, H. Pichler, M. Kalinowski, R. Samajdar, et al., Science 374, 1242 (2021).
  * Ebadi et al. (2022) S. Ebadi, A. Keesling, M. Cain, T. T. Wang, H. Levine, D. Bluvstein, G. Semeghini, A. Omran, J.-G. Liu, R. Samajdar, et al., Science 376, 1209 (2022).
  * Scholl et al. (2021) P. Scholl, M. Schuler, H. J. Williams, A. A. Eberharter, D. Barredo, K.-N. Schymik, V. Lienhard, L.-P. Henry, T. C. Lang, T. Lahaye, et al., Nature 595, 233 (2021), ISSN 0028-0836, 1476-4687.
  * Keesling et al. (2019) A. Keesling, A. Omran, H. Levine, H. Bernien, H. Pichler, S. Choi, R. Samajdar, S. Schwartz, P. Silvi, S. Sachdev, et al., Nature 568, 207 (2019), ISSN 1476-4687.
  * Bluvstein et al. (2021) D. Bluvstein, A. Omran, H. Levine, A. Keesling, G. Semeghini, S. Ebadi, T. T. Wang, A. A. Michailidis, N. Maskara, W. W. Ho, et al., Science 371, 1355 (2021), publisher: American Association for the Advancement of Science.
  * Chen et al. (2023) C. Chen, G. Bornet, M. Bintz, G. Emperauger, L. Leclerc, V. S. Liu, P. Scholl, D. Barredo, J. Hauschild, S. Chatterjee, et al., Nature 616, 691 (2023), ISSN 1476-4687.
  * Ebadi et al. (2021) S. Ebadi, T. T. Wang, H. Levine, A. Keesling, G. Semeghini, A. Omran, D. Bluvstein, R. Samajdar, H. Pichler, and W. W. Ho, Nature 595, 227 (2021).
  * Wang et al. (2020) Y. Wang, S. Shevate, T. M. Wintermantel, M. Morgado, G. Lochead, and S. Whitlock, npj Quantum Information 6, 54 (2020).
  * Tanaka (2017) Y. Tanaka, Micro & Nano Letters 12, 689 (2017), ISSN 1750-0443, 1750-0443.
  * Schäffner et al. (2020) D. Schäffner, T. Preuschoff, S. Ristok, L. Brozio, M. Schlosser, H. Giessen, and G. Birkl, Optics Express 28, 8640 (2020).
  * Isichenko et al. (2023) A. Isichenko, N. Chauhan, D. Bose, J. Wang, P. D. Kunz, and D. J. Blumenthal, Nat Commun 14, 3080 (2023), ISSN 2041-1723.
  * Mehta et al. (2020) K. K. Mehta, C. Zhang, M. Malinowski, T.-L. Nguyen, M. Stadler, and J. P. Home, Nature 586, 533 (2020), ISSN 0028-0836, 1476-4687.
  * Niffenegger et al. (2020) R. J. Niffenegger, J. Stuart, C. Sorace-Agaskar, D. Kharas, S. Bramhavar, C. D. Bruzewicz, W. Loh, R. T. Maxson, R. McConnell, D. Reens, et al., Nature 586, 538 (2020), ISSN 0028-0836, 1476-4687.
  * He et al. (2024a) C. He, D. Zhao, F. Fan, H. Zhou, X. Li, Y. Li, J. Li, F. Dong, Y.-X. Miao, and Y. Wang, Opto-Electron Adv 7, 230005 (2024a).
  * Xu et al. (2024) C. Xu, R. Zhao, X. Zhang, S. Zhang, X. Li, G. Geng, J. Li, X. Li, Y. Wang, and L. Huang, eLight 4, 9 (2024), ISSN 2097-1710, 2662-8643.
  * Yin et al. (2024) Y. Yin, Q. Jiang, H. Wang, J. Liu, Y. Xie, Q. Wang, Y. Wang, and L. Huang, Adv. Mater. 36, 2312303 (2024), ISSN 0935-9648, 1521-4095.
  * He et al. (2024b) H. He, Y. Zhang, Y. Shao, Y. Zhang, G. Geng, J. Li, X. Li, Y. Wang, L. Bian, J. Zhang, et al., Adv. Mater. 36, 2313357 (2024b), ISSN 0935-9648, 1521-4095.
  * Shao et al. (2024) Y. Shao, Z. Su, H. He, X. Jing, Y. Liu, G. Geng, J. Li, Y. Wang, and L. Huang, Optics Express 32, 23268 (2024).
  * Su et al. (2024) Z. Su, Y. Yang, B. Xiong, R. Zhao, Y. Wang, and L. Huang, Adv. Opt. Mater. 12, 2303195 (2024), ISSN 2195-1071, 2195-1071.
  * Huang et al. (2024) R. Huang, F. Zhou, X. Li, P. Xu, Y. Wang, and M. Zhan, Opt. Express 32, 21293 (2024).
  * Zhu et al. (2020) L. Zhu, X. Liu, B. Sain, M. Wang, C. Schlickriede, Y. Tang, J. Deng, K. Li, J. Yang, M. Holynski, et al., Sci. Adv. 6, eabb6667 (2020), ISSN 2375-2548.
  * Hsu et al. (2022) T.-W. Hsu, W. Zhu, T. Thiele, M. O. Brown, S. B. Papp, A. Agrawal, and C. A. Regal, PRX Quantum 3, 030316 (2022), ISSN 2691-3399.
  * Schlosser et al. (2001) N. Schlosser, G. Reymond, I. Protsenko, and P. Grangier, Nature 411, 1024 (2001), ISSN 0028-0836, 1476-4687.
  * Schlosser et al. (2002) N. Schlosser, G. Reymond, and P. Grangier, Phys. Rev. Lett. 89, 023005 (2002), ISSN 0031-9007, 1079-7114.
  * Schäffner et al. (2024) D. Schäffner, T. Schreiber, F. Lenz, M. Schlosser, and G. Birkl, PRX Quantum 5, 010311 (2024).
  * Eckner et al. (2023) W. J. Eckner, N. D. Oppong, A. Cao, A. W. Young, W. R. Milner, J. M. Robinson, J. Ye, and A. M. Kaufman, Nature 621, 734 (2023).
  * Shaw et al. (2024) A. L. Shaw, R. Finkelstein, R.-S. Tsai, P. Scholl, T. H. Yoon, J. Choi, and M. Endres, Nat. Phys. 20, 195 (2024), ISSN 1745-2473.