# CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Painting

Peter Schaldenbrand1, Gaurav Parmar1, Jun-Yan Zhu1, James McCann1, and Jean
Oh1 1The Robotics Institute, Carnegie Mellon University {pschalde, jmccann,
gparmar, junyanz<EMAIL_ADDRESS>
###### Abstract

Prior robot painting and drawing work, such as FRIDA, has focused on
decreasing the sim-to-real gap and expanding input modalities for users, but
the interaction with these systems generally exists only in the input stages.
To support interactive, human-robot collaborative painting, we introduce the
Collaborative FRIDA (CoFRIDA) robot painting framework, which can co-paint by
modifying and engaging with content already painted by a human collaborator.
To improve text-image alignment–FRIDA’s major weakness–our system uses pre-
trained text-to-image models; however, pre-trained models in the context of
real-world co-painting do not perform well because they (1) do not understand
the constraints and abilities of the robot and (2) cannot perform co-painting
without making unrealistic edits to the canvas and overwriting content. We
propose a self-supervised fine-tuning procedure that can tackle both issues,
allowing the use of pre-trained state-of-the-art text-image alignment models
with robots to enable co-painting in the physical world. Our open-source
approach, CoFRIDA, creates paintings and drawings that match the input text
prompt more clearly than FRIDA, both from a blank canvas and one with human
created work. More generally, our fine-tuning procedure successfully encodes
the robot’s constraints and abilities into a foundation model, showcasing
promising results as an effective method for reducing sim-to-real gaps.
https://pschaldenbrand.github.io/cofrida/

##  I INTRODUCTION

While recent breakthroughs in text-to-image synthesis technologies have
ignited a boom in digital content generation, using them to produce art with
robots is still in its infancy due to a significant gap between simulated and
real-world environments. FRIDA [1] is a robotic framework that can take user
inputs, such as language descriptions or input images, to paint on a physical
canvas using a paintbrush and acrylic paint. While FRIDA aims at giving users
control over content generation, the users are allowed to add their input only
with an initial input prompt or image, after which they are excluded from the
creative process. While it is still debatable whether such an autonomous
creation is desired by humans practicing art [2], there is strong evidence of
the potential value of a co-creative agent [3, 4, 5, 6, 7, 8, 9] specifically
in the domain of art therapy [10, 11, 12, 13]. The benefits can be further
increased when paired with a physical embodiment of such an agent and drawing
in the real world [14, 15]. To invite users into the creative process and
bring the benefits of both co-creation and robotic embodiment, we build on
FRIDA to propose a Collaborative Framework and Robotics Initiative for
Developing Arts (CoFRIDA), as illustrated in Fig. 1.

Figure 1:  Co-Painting with CoFRIDA. We showcase how CoFRIDA collaboratively
paints with artists. The process begins with the artist sketching a table.
Building on that foundation, CoFRIDA adds to the canvas, guided by the
artist’s initial prompt: “A bulky robot arm on a table.” The artist then
iterates on the painting with additional strokes to add detail to the robot
arm, and provides a new text prompt, “A robot arm with a hand.” CoFRIDA
responds by completing the painting to match this new description.

One of the biggest challenges in human-robot co-creation is enabling the robot
to create new content that engages with the existing content that the human
drew, hereby referred to as co-painting. While there exist related image
editing problems, such as in-painting, co-painting is a new class of problems
with unique challenges as it is undesirable in co-painting to make radical
changes to the image that would overwrite the human’s previous work. In in-
painting, the area for editing an image is coarsely specified by the user and
the model is expected to drastically change the content within that local
region. By contrast, with co-painting, the edit is expected to preserve and
engage with the full canvas rather than re-imagining a local region. Whereas
in-painting is a localized edit by definition, co-painting is a continuous,
iterative completion, e.g., adding detail to an existing human-drawn rough
sketch.

Besides the challenges of co-painting, robotic image creation is difficult due
to real-world constraints, such as existing canvas state, limited abilities of
the robot, tools and materials available to the robot, and stochasticity in
robot performance. These robotic constraints vastly limit the content that is
capable of being created, as illustrated in the left side of Fig. 8. With a
large paintbrush, fine-details are not achievable, and with a single marker,
multi-color images are not possible. Multiple works address these constraints
to decrease the Sim2Real gap, but only paint from image inputs [16, 17, 18,
19]. Even fewer existing works use cameras to enable co-creation of images
[14].

FRIDA uses the data from a real robot to be able to simulate high-fidelity
brush strokes using the idea known as Real2Sim2Real. To paint from a language
input, FRIDA uses CLIP [20] to align language and image which tends to
generate noisy output. To improve the quality of paintings for CoFRIDA, we use
powerful image generators pre-trained using gigantic text-image paired data,
e.g., StableDiffusion [21] or Instruct-Pix2Pix [22]. Because such pre-trained
image generators do not know the capabilities of the robot, there is both a
large difference in pixel value and semantic meaning between the image
generator output and FRIDA’s simulated plan. The former difference is a
traditional Sim2Real gap, whereas the latter is a concept we introduce as the
Semantic Sim2Real Gap.

To reduce the Semantic Sim2Real Gap, we propose a self-supervised fine-tuning
for CoFRIDA. CoFRIDA adapts a pre-trained image generator to both generate
content within the abilities of the robot and perform co-painting to enable
human-robot collaborative drawing from language guidance, e.g., in this paper,
we use Instruct-Pix2Pix [22] as our base text-image model. To adapt a pre-
trained model for co-painting and encode robotic constraints, first we create
the self-supervised fine-tuning dataset by using FRIDA to generate full
drawings or paintings of images from a text-image dataset. Strokes from the
full paintings are removed selectively to form partial paintings. We fine-tune
Instruct-Pix2Pix by retraining it with a low learning rate to predict the full
painting from the partial painting and text prompt.

Figure 2:  Co-Painting. We introduce Co-Painting as a task in which a robot
must add content to a painting that engages with the current content without
destroying the existing work. We demonstrate that existing models (Instruct-
Pix2Pix, bottom row) often cannot successfully add content without making
unreasonably large edits to the canvas, overwriting any prior work, while
CoFRIDA (top row) adds content that harmonizes with the existing work.

CoFRIDA can successfully use an existing canvas state to generate future
actions towards a language goal without completely overwriting the existing
work as shown in Fig. 2. Based on a survey on Amazon Mechanical Turk (MTurk)
of 24 participants, CoFRIDA’s completed drawings from partial sketches were
found to be substantially more similar to the language goal when compared to
those by the baselines.

Our main contributions are summarized as: 1) we introduce co-painting, a new
class of image editing that is required for human-robot collaborative
creation; 2) we propose Collaborative FRIDA (CoFRIDA) to support human-robot
co-painting to produce real-world arts, e.g., paintings and drawings on
canvas; 3) we propose a generalizable method for reducing the Sim2Real gap
using self-supervised fine-tuning, enabling generic pre-trained models to be
used with physical robots; 4) CoFRIDA is open-
source111https://github.com/cmubig/Frida and available on XArm, Franka Emika,
and Rethink Sawyer robot platforms.

Figure 3:  Method Overview. Offline, we fine-tune a pre-trained Instruct-
Pix2Pix model on our self-supervised data. Online, the user can either draw or
give the robot a text description. The Co-Painting Module takes as input the
current canvas and text description to generate a pixel prediction of how the
robot should finish the painting using the fine-tuned Instruct-Pix2Pix model.
FRIDA predicts actions for the robot to create this pixel image and produces a
simulation. This process is repeated until the user is satisfied.

##  II RELATED WORK

###  II-A Computer-Based Image Co-Creation

Computer-based image co-creation generally involves turn taking between a
human and a computer in applying brush stroke primitives towards one of a
discrete set of goals, as in sketch-rnn [23] and Drawing Apprentice [4], or
even towards natural language goals [6, 5]. Computer-based studies have shown
creativity augmentation benefits of co-creation [6, 5, 4] since computer
agents can add serendipity and reformulate user’s original intentions leading
to unexpected by enjoyable outputs [7]. However, Computer-based painting
models do not transfer well out-of-the-box into the real world due to the
Sim2Real gap [16, 1, 24].

###  II-B Robotic Image Co-Creation

There exists many real-world methods for robot painting and drawing [25, 19,
17, 26], however, few systems have incorporated perception into their systems
to enable co-painting. Cobbie [14] is a co-drawing system that boosted
ideation for novice drawers, however, it is limited to drawing on blank areas
of the paper rather than engaging with the user drawn content. [13] created a
robot arm that can draw from speech inputs that are limited to simple objects
found in the Quick, Draw! dataset [27]. The FRIDA [1] system is the only
system directly capable of making physical paintings that engage with existing
content conditioned on natural language goals. While FRIDA plans based on
current canvas state, it uses CLIP and gradient descent for planning which
produces paintings that are very noisy and only loosely resemble the input
text.

##  III METHOD

Our approach, CoFRIDA shown in Fig. 3, is made up of three primary components:
(1) The Co-Painting Module, which produces images illustrating how the robot
should add content to an existing canvas given a text description, (2) FRIDA
[1], a robotic painting system for planning actions from given images, and (3)
a self-supervised method for creating training data using FRIDA to fine-tune
pre-trained models in the Co-Painting Module.

###  III-A Self-Supervised Data Creation

While there exist some supervised data of human-created co-paintings [28, 29],
they are only on the order of tens of examples and were not made using the
same materials available to our robot. To support co-painting tasks, we
propose a self-supervised method for generating training data to train a Co-
Painting Module. We simulate paintings of images from the art subset of the
LAION image-text dataset [30] using FRIDA with image-guidance loss (difference
of CLIP embeddings of images). To create partial paintings, strokes are
removed selectively to support a variety of co-painting tasks: remove all
strokes, a random subset of strokes, strokes corresponding to a salient region
(defined with CLIP as in [31]) of the image, and strokes from a semantic
region (using Segment Anything [32]). Illustrative examples are shown in Fig.
4.

Some source images cannot be accurately represented with the robot’s
abilities. We filter out such images by removing instances that have a
CLIPScore between the simulated full paintings and the text less than 0.5.

We use this self-curated data to fine-tune a base text-to-image generation
model to be able to 1) continue to create content on an existing canvas and 2)
generate images that the target robot is capable of painting.

Figure 4:  Self-Supervised Dataset Creation. We describe the process of
generating the self-supervised training data pairs for fine-tuning the Co-
Painting Module. We start with the input images from the LAION-art dataset and
convert them into simulated sketch outputs with the FRIDA simulator. Next, we
create partial sketches in four different ways: removing random strokes,
removing the salient region, removing a semantic region, and removing all
strokes.  Figure 5:  Qualitative Comparison. We show a comparison between
three methods of performing text-based canvas updates: FRIDA, CoFRIDA without
fine-tuning, and CoFRIDA with fine-tuning (ours). FRIDA uses a CLIP based
optimization and generates outputs that are noisy. CoFRIDA without fine-
tuning, is not aware of the constraints of the robot and generates an output
that is difficult for the robot to execute and often does not satisfy the text
prompt specified by the user. In contrast, CoFRIDA outputs an updated canvas
that reflects the user prompt without being noisy.

###  III-B Co-Painting Module

The goal of the Co-Painting Module (Fig.3) is to generate an image of how the
robot should complete the painting given a photograph of the current canvas
and a user given text description. The Co-Painting Module uses Instruct-
Pix2Pix [22] as a pre-trained model as it enables conditioning the output on
an input canvas. The pre-trained Instruct-Pix2Pix, however, has two
shortcomings to be used for co-painting: (1) the generated images do not
reflect actual robotic constraints, and (2) the existing canvas can sometimes
be overwritten completely as shown in Fig. 2. To overcome these limitations,
we fine-tune Instruct-Pix2Pix using the dataset of partial and full drawings
with their captions described in Sec. III-A.

Fine-tuning is performed using FRIDA’s simulated canvases because (1) it would
be infeasible to generate a large-scale dataset with the physical robot, and
(2) the Co-Painting Module output is eventually used with the FRIDA
simulation.

###  III-C FRIDA

In this work, we use the FRIDA [1] robotic painting system for both planning
actions from given images and creating self-supervision data. FRIDA uses
Real2Sim2Real methodology to enable it to paint with a variety of media such
as acrylic paint or markers. Because FRIDA plans using a perceptual loss, it
is capable of making, for example, drawings using markers from color
photographs.

FRIDA plans actions to guide the robot to make the current canvas look like
the output of the Co-Painting Module. Using FRIDA’s internal simulation, we
visualize a prediction of what the actions will look like, and the robot can
execute the actions to update the real canvas, Fig. 3.

##  IV EXPERIMENTS

###  IV-A Baselines

We compare CoFRIDA to the original FRIDA’s [1] CLIP-guided text-to-painting
method. We investigate the effects of our fine-tuning procedure on Instruct-
Pix2Pix in the Co-Painting Module by comparing our method (CoFRIDA) with pre-
trained Instruct-Pix2pix (CoFRIDA w/o fine-tuning).

###  IV-B Different Painting Settings

The FRIDA painting system can paint with various brushes and can have
different color constraints. We test CoFRIDA using three different painting
settings (1) acrylic painting using one brush and 12 colors which can differ
from painting to painting, (2) acrylic painting with a fixed 4-color palette,
and (3) a black Sharpie marker. Examples of these three settings are shown in
Fig. 4, 9, and 8 . The robot can only be used in one of these settings at a
time. However, users can paint using any media of choice, leading to mixed
media paintings in Fig. 7.

Figure 6:  User Preference Study. Results from two MTurk Surveys. Presented with a text description, participants chose which of two drawings (CoFRIDA versus either FRIDA or CoFRIDA without fine-tuning) was more similar to the text, neither, or both. See Fig. 5 for examples. TABLE I: CLIPScores and BLIPScores computed on robot simulated drawings (See Fig. 5). Sim-to-real gap measurements, $\Delta_{pix}$ and $\Delta_{sem}$, measure the difference between the Co-Painting Module output and the simulated drawing of that image. | CLIPScore $\uparrow$ | BLIPScore $\uparrow$ |  $\Delta_{pix}$ $\downarrow$ | $\Delta_{sem}\downarrow$
---|---|---|---|---
FRIDA | 0.741 | 0.192 | — | —
| CoFRIDA
---
w/o fine-tuning
0.595 | 0.162 | 0.195 | 0.241
CoFRIDA | 0.624 | 0.178 | 0.052 | 0.035

###  IV-C Evaluation

Text-Image Alignment \- Two automatic methods of comparing image and text are
CLIPScore [33] and BLIPScore [34], which measure the similarity between images
and text with a pre-trained image-text encoders. Because FRIDA directly
optimizes the CLIPScore to create images from text, this method is unfairly
advantaged when using CLIPScore. We use MTurk to achieve large-scale fair
evaluation of text-image alignment.

Semantic Sim2Real Gap \- It is important that between the output of the Co-
Painting Module ($I_{cpm}$) and the FRIDA simulation ($I_{sim}$) there is
little loss in semantic meaning. A naive approach at measuring this loss is
the mean-squared-error between the images’ pixels (Eq. 1). However, this is
sensitive to low-level variation in details such as color or tone differences
which are tolerable as long as the high-level content in the images is the
same. To measure the high-level difference, we propose to use the cosine
distance between CLIP image embeddings, $\Delta_{sem}$, Eq. 2, referred to as
the Semantic Sim2Real Gap.

$\displaystyle\vspace{-7pt}\Delta_{pix}$
$\displaystyle=||I_{cpm}-I_{sim}||^{2}_{2}$ (1) $\displaystyle\Delta_{sem}$
$\displaystyle=\cos(CLIP(I_{cpm}),CLIP(I_{sim}))$ (2)

A proper Sim2Real gap measurement would compare the output of the Co-Painting
Module to the real drawing, however, it is infeasible to generate a robust
number of real-world samples. Because the Sim2Real gap between the FRIDA
simulation and the real drawing is the same across all tested methods, we can
fairly use the FRIDA simulations in lieu of the real drawings for comparing
the Sim2Real gaps of Co-Painting Module variations.

##  V RESULTS

###  V-A Co-Painting

To test the ability of CoFRIDA to work with an existing canvas state, we focus
on Sharpie marker drawings where no erasing is possible, forcing the model to
have to adapt to and use the existing markings on the page. To create the
partial drawing, we generate an image with Stable Diffusion using prompts from
the PartiPrompts [35] dataset, then simulate the drawing with just 35 strokes
as depicted in Fig. 5. We generated 40 images from different prompts per
method. CLIPScore [33], BLIPScore, and Sim2Real gap measures are reported in
Table I. Since FRIDA maximizes CLIPScore, it was expected and confirmed to
have the highest CLIPScore. BLIP is also expected to correlate with CLIP,
leading FRIDA to have an artificially high BLIPScore.

To properly assess the image-text similarity of the drawings from partial
sketches, we conducted an MTurk survey summarized in Fig 6. 24 unique
participants were shown a language description then two images (one from ours
and the other one of the two baselines, in random order). Participants were
instructed to choose which image fits the given caption better, or to select
neither or both. Each image pair was evaluated by 4 unique participants
leading to 160 comparisons per baseline. While many participants found neither
image fit the text description (an indicator of the challenging nature of co-
painting), CoFRIDA was generally indicated as having clearer content over
FRIDA and CoFRIDA without our fine-tuning.

In terms of the proposed Semantic Sim2Real Gap, CoFRIDA outperforms the
baselines indicating that our fine-tuning guided Instruct-Pix2Pix to produce
images that were less likely to change meaning when painted by FRIDA.

Figure 7:  Mixed-Media Paintings. CoFRIDA can use markers and paintbrushes to
co-paint with a human. Despite being fine-tuned with a single medium, CoFRIDA
can still perform co-painting when a user uses different media such as
watercolors. Figure 8:  Learning Robotic Constraints. We compare images
generated by a pre-trained Stable Diffusion model (left) to those generated by
our proposed CoFRIDA module (right) with the prompt “A dog and a cat sitting
next to each other on the beach” in three different painting settings
(Sec.IV-B). The top row shows the images generated by each of the models and
the bottom row shows the corresponding FRIDA simulation.

###  V-B Multiple Turns

A co-painting system must be capable of accommodating multiple iterations of
human-robot interaction in which the robot adds content but does not
completely overwrite the human’s prior work. We simulate this by having the
robot create sequences of modifications to a simulated painting with different
text prompts in Fig. 2. The baseline methods tend to either avoid making
changes or make huge changes to the canvas, whereas CoFRIDA makes updates that
are more reasonable for the robot to achieve and integrate naturally with
prior work.

Figure 9:  Comparing CoFRIDA’s fine-tuned pre-trained image generator versus
FRIDA’s CLIP-guided method for generating paintings from the text “A sad, frog
ballerina doing an arabesque” in three painting settings.

###  V-C Text Conditioned Paintings

FRIDA’s text-to-painting method relies on feedback through CLIP which results
in noisy, unclear imagery. We compare CoFRIDA which uses a pre-trained
generative model to FRIDA in Fig. 9. CoFRIDA’s paintings are far more clear
and capture the caption better than FRIDA in various painting settings.

###  V-D Real Paintings

We used FRIDA’s simulation to make large scale data creation and evaluation
feasible. Fig. 7 displays multiple real-world examples of CoFRIDA’s drawings
and paintings. CoFRIDA is able to successfully use content on canvases that is
out of distribution from its fine-tuning training data as with the watercolor
and marker examples in Fig. 7.

##  VI DISCUSSION

Limitations and Ethical Considerations CoFRIDA stands out as a successful
collaborative painting system, but is limited to discrete turn-taking
interactions. While our self-supervised training data creation method (Fig. 4)
was informed by real co-painting data, a more end-to-end approach where the
system learns how to form the partial paintings could result in even better
results.

In light of recent discoveries of harmful content in the LAION dataset, we
have inspected the subset of data used to fine-tune the models used in this
paper and have not found harmful content. We have changed the code to use the
COCO dataset [36] and have not seen a degradation in the quality of results.
CoFRIDA is subject to the biases of Stable Diffusion [21] and its training
data [37], and so we recommend the usage of CoFRIDA with caution and solely
for research purposes.

Learning Robotic Abilities Our self-supervised fine-tuning procedure guided
the pre-trained model to generate images that, at a pixel-level, appeared
similar to what FRIDA can paint, but is it learning the actual robot
constraints or just a low-level style transfer? We computed the Sim2Real gap
measurements between the LAION images and their FRIDA simulations (as seen in
Fig. 4) along with the CLIPScore of the simulation and text prompt. We found
that $\Delta_{pix}$ had a small, insignificant Pearson correlation ($-0.08$,
$0.08$ $p$-value) with the CLIPScore of the painting whereas $\Delta_{sem}$
had a significant, negative correlation ($-0.48$, $2.4e-31$ $p$-value).
Because CoFRIDA greatly decreases the $\Delta{sem}$, this indicates that
CoFRIDA’s fine-tuning technique is not solely changing the low-level
appearance (akin to style-transfer) over the output of its base model. It
appears that CoFRIDA is learning the robot’s abilities, as seen in Fig. 8
where CoFRIDA’s Co-Painting Module produces images with (1) very prominent and
clear content, when the robot’s brush is large (2) select and limited colors,
when the robot paints with fixed palettes or markers, and (3) sparse, concise
drawings when the number of strokes is limited.

##  VII CONCLUSIONS

An end-to-end approach, like FRIDA, that optimizes the brush strokes towards
the text goal tends to produce noisy looking paintings that only loosely
resemble the text because it operates in a low-level space without a global
context. Additionally, it is hard to incorporate interactivity beyond an
initial input. We present Collaborative FRIDA (CoFRIDA), a hierarchical
approach for interactive human-robot co-painting where semantic planning via
pre-trained models happens in a high-level, pixel space before being
transferred to a low-level brush stroke planner. Pre-trained models do not
immediately provide the requirements for co-painting, as they do not know the
capabilities of the robot. Whereas the Real2Sim2Real methodology improves low-
level action-space planning in FRIDA, the proposed self-supervised fine-tuning
procedure provides a method for adapting powerful pre-trained models for high-
level robotic planning. CoFRIDA uses this hierarchical approach for reducing
the Sim2Real gap, achieving enhanced performance over the baselines.

##  VIII Acknowledgments

This work was partly supported by NSF IIS-2112633, the Packard Fellowship, and
the Technology Innovation Program (20018295, Meta-human: a virtual cooperation
platform for a specialized industrial services) funded By the Ministry of
Trade, Industry & Energy (MOTIE, Korea).

## References

  * [1] P. Schaldenbrand, J. McCann, and J. Oh, “Frida: A collaborative robot painter with a differentiable, real2sim2real planning environment,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 11 712–11 718.
  * [2] H. H. Jiang, L. Brown, J. Cheng, M. Khan, A. Gupta, D. Workman, A. Hanna, J. Flowers, and T. Gebru, “Ai art and its impact on artists,” in _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ , ser. AIES ’23. New York, NY, USA: Association for Computing Machinery, 2023, p. 363–374. [Online]. Available: https://doi.org/10.1145/3600211.3604681
  * [3] C. Bateman, “Creating for creatives: A humanistic approach to designing ai tools targeted at professional animators,” Ph.D. dissertation, Harvard University, 2021.
  * [4] N. Davis, C.-P. Hsiao, K. Yashraj Singh, L. Li, and B. Magerko, “Empirically studying participatory sense-making in abstract drawing with a co-creative cognitive agent,” in _Proceedings of the 21st International Conference on Intelligent User Interfaces_ , 2016, pp. 196–207.
  * [5] F. Ibarrola, T. Lawton, and K. Grace, “A collaborative, interactive and context-aware drawing agent for co-creative design,” _IEEE Transactions on Visualization and Computer Graphics_ , 2023.
  * [6] T. Lawton, F. J. Ibarrola, D. Ventura, and K. Grace, “Drawing with reframer: Emergence and control in co-creative ai,” in _Proceedings of the 28th International Conference on Intelligent User Interfaces_ , 2023, pp. 264–277.
  * [7] T. Lawton, K. Grace, and F. J. Ibarrola, “When is a tool a tool? user perceptions of system agency in human–ai co-creative drawing,” in _Proceedings of the 2023 ACM Designing Interactive Systems Conference_ , 2023, pp. 1978–1996.
  * [8] C. Jansen and E. Sklar, “Exploring co-creative drawing workflows,” _Frontiers in Robotics and AI_ , vol. 8, p. 577770, 2021.
  * [9] S. Lee and W. Ju, “Adversarial robots as creative collaborators,” _arXiv preprint arXiv:2402.03691_ , 2024.
  * [10] M. D. Cooney and M. L. R. Menezes, “Design for an art therapy robot: An explorative review of the theoretical foundations for engaging in emotional and creative painting with a robot,” _Multimodal Technologies and Interaction_ , vol. 2, no. 3, p. 52, 2018.
  * [11] M. Cooney and P. Berck, “Designing a robot which paints with a human: visual metaphors to convey contingency and artistry,” in _ICRA-X Robots Art Program at IEEE International Conference on Robotics and Automation (ICRA), Montreal QC, Canada_ , 2019, p. 2.
  * [12] M. Cooney, “Robot art, in the eye of the beholder?: Personalized metaphors facilitate communication of emotions and creativity,” _Frontiers in Robotics and AI_ , vol. 8, p. 668986, 2021.
  * [13] S. Z. Shaik, V. Srinivasan, Y. Peng, M. Lee, and N. Davis, “Co-creative robotic arm for differently-abled kids: Speech, sketch inputs and external feedbacks for multiple drawings,” in _Proceedings of the Future Technologies Conference (FTC) 2020, Volume 3_. Springer, 2021, pp. 998–1007.
  * [14] Y. Lin, J. Guo, Y. Chen, C. Yao, and F. Ying, “It is your turn: Collaborative ideation with a co-creative robot through sketch,” in _Proceedings of the 2020 CHI conference on human factors in computing systems_ , 2020, pp. 1–14.
  * [15] D. Herath, J. McFarlane, E. A. Jochum, J. B. Grant, and P. Tresset, “Arts+ health: New approaches to arts and robots in health care,” in _Companion of the 2020 ACM/IEEE International Conference on Human-Robot Interaction_ , 2020, pp. 1–7.
  * [16] P. Schaldenbrand and J. Oh, “Content masked loss: Human-like brush stroke planning in a reinforcement learning painting agent,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 35, no. 1, 2021, pp. 505–512.
  * [17] T. Lindemeier, “e-david: Non-photorealistic rendering using a robot and visual feedback,” Ph.D. dissertation, University of Konstanz, 2018.
  * [18] S. Wang, J. Chen, X. Deng, S. Hutchinson, and F. Dellaert, “Robot calligraphy using pseudospectral optimal control in conjunction with a novel dynamic brush model,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 6696–6703.
  * [19] Rob Carter and Nick Carter, “Dark factory portraits,” http://www.robandnick.com/dark-factory-portraits, 2017.
  * [20] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” _CoRR_ , vol. abs/2103.00020, 2021. [Online]. Available: https://arxiv.org/abs/2103.00020
  * [21] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High-resolution image synthesis with latent diffusion models,” 2021.
  * [22] T. Brooks, A. Holynski, and A. A. Efros, “Instructpix2pix: Learning to follow image editing instructions,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 18 392–18 402.
  * [23] D. Ha and D. Eck, “A neural representation of sketch drawings,” _arXiv preprint arXiv:1704.03477_ , 2017.
  * [24] A. Bidgoli, M. L. De Guevara, C. Hsiung, J. Oh, and E. Kang, “Artistic style in robotic painting; a machine learning approach to learning brushstroke from human artists,” in _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_. IEEE, 2020, pp. 412–418.
  * [25] G. Lee, M. Kim, M. Lee, and B.-T. Zhang, “From scratch to sketch: Deep decoupled hierarchical reinforcement learning for robotic sketching agent,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 5553–5559.
  * [26] M. C. Sola and V. Guljajeva, “Dream painter: Exploring creative possibilities of ai-aided speech-to-image synthesis in the interactive art context,” _Proceedings of the ACM on Computer Graphics and Interactive Techniques_ , vol. 5, no. 4, pp. 1–11, 2022.
  * [27] J. Jongejan, H. Rowley, T. Kawashima, J. Kim, and N. Fox-Gieg, “The quick, draw!-ai experiment,” _Mount View, CA, accessed Feb_ , vol. 17, no. 2018, p. 4, 2016.
  * [28] D. Parikh and C. L. Zitnick, “Exploring crowd co-creation scenarios for sketches,” _arXiv preprint arXiv:2005.07328_ , 2020.
  * [29] e. a. Juliet Shen, “Co-drawings,” 2016. [Online]. Available: https://www.codrawseattle.com/
  * [30] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, _et al._ , “Laion-5b: An open large-scale dataset for training next generation image-text models,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 25 278–25 294, 2022.
  * [31] Y. Vinker, E. Pajouheshgar, J. Y. Bo, R. C. Bachmann, A. H. Bermano, D. Cohen-Or, A. Zamir, and A. Shamir, “Clipasso: Semantically-aware object sketching,” _arXiv preprint arXiv:2202.05822_ , 2022.
  * [32] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, _et al._ , “Segment anything,” _arXiv preprint arXiv:2304.02643_ , 2023.
  * [33] J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y. Choi, “Clipscore: A reference-free evaluation metric for image captioning,” _arXiv preprint arXiv:2104.08718_ , 2021.
  * [34] J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 12 888–12 900.
  * [35] J. Yu, Y. Xu, J. Y. Koh, T. Luong, G. Baid, Z. Wang, V. Vasudevan, A. Ku, Y. Yang, B. K. Ayan, _et al._ , “Scaling autoregressive models for content-rich text-to-image generation,” _arXiv preprint arXiv:2206.10789_ , vol. 2, no. 3, p. 5, 2022.
  * [36] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_. Springer, 2014, pp. 740–755.
  * [37] A. Birhane, V. U. Prabhu, and E. Kahembwe, “Multimodal datasets: misogyny, pornography, and malignant stereotypes,” _arXiv preprint arXiv:2110.01963_ , 2021.


# Schrödinger’s cat state of optical parallel universes

Yu-Qing Cui1 Tian-Ming Zhao1<EMAIL_ADDRESS>Rong-Xin Miao2
<EMAIL_ADDRESS>Jin-Dong Wang3 Huanyang Chen4 1Guangdong Provincial
Key Laboratory of Nanophotonic Functional Materials and Devices, South China
Normal University, Guangzhou, 510631, China
2School of Physics and Astronomy, Sun Yat-Sen University, Zhuhai, 519082,
China
3Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum
Materials, South China Normal University, Guangzhou, 510631, China
4Department of Physics, Xiamen University, Xiamen 361005, China

###### Abstract

Parallel worlds are imaginative ideas in quantum mechanics and cosmology. The
superpositions of parallel worlds are novel states of quantum gravity and have
no classical correspondences generally. In this letter, we investigate the
superposition or the Schrödinger’s cat state of optical parallel worlds, which
could be realized in the laboratory and may shed some light on the detection
of parallel universes in the real world. We propose two realizable
experimental schemes that enable exploring the mysterious ‘parallel universes’
by a Mach-Zehnder interferometer. The first one is based on an atomic ensemble
in a superposition state, which is a fat Schrödinger’s cat state. The second
one is to prepare a photon in a superposition of different paths, where each
path lies in an optical parallel universe.

The parallel worlds are intriguing and imaginative ideas in quantum physics,
which are also popular science fiction topics. There are several different
kinds of parallel worlds. The famous one is the many world interpretation of
quantum mechanics Everett , which conjectures all the outcomes that can occur
actually do happen, but only one outcome can happen in each universe. Roughly
speaking, during the evolution of the Schrödinger equation, the world splits
into many different branches, and our universe is just one of them
Tegmark:2007wh . The second kind of parallel world is the multiverse of
eternal inflation Eternalinflation , where the space is broken up into
infinite causally disconnected bubble universes. In an eternally inflating
universe, anything that can happen will happen Guth . Interestingly, the many
worlds of quantum mechanics and the many worlds of the multiverse are
conjectured to be equivalent Nomura:2011dt ; Bousso:2011up . For other types
of parallel universes, please refer to Tegmark:2003sr .

Although novel, parallel worlds are difficult to be observed in experiments.
See Feeney:2010dd ; Feeney:2010jj  for some observational tests of
multiverses. Thus, it is interesting to study the optical analogy of parallel
worlds, which is much easier to be realized in experiments and may shed some
light on the detection of parallel universes in the real world. Thanks to the
rapid developments of metamaterials and transformation optics Leonhardt ;
Pendry , the optical mimicking of novel spacetime such as black holes
Chen:2009mk , worm holes Greenleaf:2007bd , de Sitter universes Li:2009pm ;
Li:2009zy , multiverses Smolyaninov:2010mc  and other geometries
Leonhardt:2006ai ; Zhang2009 ; Chen:2020dxy ; Zhao:2011nq  are possible in
laboratory. However, since metamaterials are classical objects, these works
can only mimic the classical spacetime. In this paper, we explore the quantum
effects: the superposition or the Schrödinger’s cat state of optical parallel
worlds.

Figure 1: Parallel worlds of ants. Due to the quantum fluctuations of
spacetime, the ant travels in infinite parallel worlds before it arrives at B.
Here dS (sphere), Min (plane) and AdS (saddle) denote the maximum symmetrical
spaces with positive, zero and negative curvatures.

Consider two quantum states of gravity which correspond to two kinds of
universes. Without loss of generality, we take Minkowski spacetime (Min)
$|\Psi_{\text{Min}}\rangle$ and de Sitter universe (dS)
$|\Psi_{\text{dS}}\rangle$ as examples. Since the expansion of our universe is
accelerating currently SupernovaSearchTeam:1998fmf ;
SupernovaCosmologyProject:1998vns , dS is a good approximation of our universe
at cosmic scale. On the other hand, Min is a good approximation of our
universe at small scale. According to the standard quantum mechanics, the
superposition of the two states

$\displaystyle|\Psi\rangle=\frac{1}{\sqrt{2}}\Big{(}|\Psi_{\text{Min}}\rangle+|\Psi_{\text{dS}}\rangle\Big{)},$
(1)

is also a quantum state of gravity. However, unlike
$|\Psi_{\text{Min}}\rangle$ and $|\Psi_{\text{dS}}\rangle$, the state (1) has
no classical correspondence. Obviously, one cannot live in two different
classical universes at the same time. Actually (1) is an analogy of
Schrödinger’s cat, where the poor cat is replaced by the spacetime. Another
way to understand this is as follows. The classical spacetime is a solution to
Einstein equations, which are highly non-linear equations. The linear
combinations of two solutions cannot be a solution to non-linear equations. As
a result, (1) cannot be classical spacetime. Instead, it is a non-trivial
state of quantum gravity. The main purpose of this letter is to study this
novel quantum state of parallel worlds in optical systems.

Note that (1) is the superposition of parallel worlds before a measurement is
performed. According to quantum mechanics, although it is difficult to
observe, such a superposition state does exist in a real world if gravity is
quantum Marletto ; Foo:2020jmi . Suppose that an ant crawls from point A to
point B and no observation is made in this process. Due to the quantum
fluctuations of spacetime, the ant travels in infinite parallel worlds before
it arrives at B. See Fig.1 for an example, where parallel worlds are labelled
by dS, Min and anti de Sitter space (AdS). At the end point B, the ant retains
the ‘memory’ of parallel worlds in the phases, which can be extracted by
interference experiments in principle. In this letter, the ant is replaced by
a photon and the many worlds of spacetime are mimicked by optical parallel
universes.

Mimicking spacetime — To start, let us give a quick review on how to mimic
curved spacetime in an optical system. Without loss of generality, we take dS
as an example. dS describes a universe with a constant rate of accelerated
expansion. In static coordinates, the metric is given by

$\displaystyle
ds^{2}=-f(\bar{r})c^{2}dt^{2}+\frac{d\bar{r}^{2}}{f(\bar{r})}+\bar{r}^{2}d\Omega^{2},$
(2)

where $f(\bar{r})=(1-H^{2}\bar{r}^{2})$, $H$ is the Hubble parameter, $c$ is
the velocity of light, $d\Omega^{2}=d\theta^{2}+\sin^{2}\theta d\phi^{2}$ is
the line element of unite sphere. According to Leonhardt:2006ai ; Plebanski ,
the curved spacetime with the metric
$ds^{2}=-g_{00}c^{2}dt^{2}+g_{ij}dx^{i}dx^{j}$ can be mimicked by a dielectric
with the refractive index

$\displaystyle n^{ij}=\frac{\sqrt{-g}}{\sqrt{\gamma}}\frac{g^{ij}}{g_{00}},$
(3)

where $g$ is the determinant of metric, $g^{ij}$ is the inverse metric,
$\gamma$ labels the determinant of the physical metric in laboratory. For
examples, $\gamma=1$ for Cartesian coordinates and
$\gamma=r^{4}\sin^{2}\theta$ for spherical coordinates. A photon cannot
distinguish if it lives in curved spacetime or a dielectric, since by design,
Maxwell’s equations are the same in them.

From (3), we notice that the refractive index of (2) is anisotropic, which
makes the experiment difficult. To get a isotropic refractive index, we
perform the coordinate transformation $\bar{r}=\frac{r}{1+(H^{2}r^{2}/4)}$ so
that $dr/r=d\bar{r}/(\bar{r}\sqrt{f(\bar{r})})$ and the metric (2) becomes

$\displaystyle
ds^{2}=-f(\bar{r})c^{2}dt^{2}+\frac{\bar{r}^{2}}{r^{2}}\Big{(}dr^{2}+r^{2}d\Omega^{2}\Big{)},$
(4)

where the spatial metric in the braces is the standard isotropic metric in
spherical coordinates. As a result, the refractive index (3) becomes isotropic

$\displaystyle n_{\text{dS}}(r)=\frac{1}{1-\frac{1}{4}H^{2}r^{2}}\approx
1+\frac{1}{4}H^{2}r^{2}+O(H^{4}).$ (5)

For the convenient of readers, in Table.1 we list the refractive indexes for
various static spacetime Chen:2020dxy ; note1  and for an expanding universe
with Robertson-Walker metric

$\displaystyle ds^{2}=-c^{2}dt^{2}+a^{2}(t)(dr^{2}+r^{2}d\Omega^{2}),$ (6)

where $a(t)$ is the scale factor. When $a(t)=\exp(Ht)$, (6) becomes the dS
metric in time-dependent coordinates.

Table 1: Refractive index of various spacetime Spacetime | Metric | Refractive index
---|---|---
Min | $f=1$ | $n=1$
Black hole (BH) | $f=1-\frac{2M}{\bar{r}}$ | $n=\frac{(M+2r)^{3}}{4r^{2}(2r-M)}$
dS | $f=1-H^{2}\bar{r}^{2}$ | $n=\frac{1}{1-\frac{1}{4}H^{2}r^{2}}$
dS-BH | $f=1-H^{2}\bar{r}^{2}-\frac{2M}{\bar{r}}$ | $n\approx 1+\frac{H^{2}r^{2}}{4}+\frac{2M}{r}$
AdS | $f=1+H^{2}\bar{r}^{2}$ | $n=\frac{1}{1+\frac{1}{4}H^{2}r^{2}}$
AdS-BH | $f=1+H^{2}\bar{r}^{2}-\frac{2M}{\bar{r}}$ | $n\approx 1-\frac{H^{2}r^{2}}{4}+\frac{2M}{r}$
Expanding universe | RW metric (6) | $n=a(t)$

Scheme I — Now we are ready to discuss the mimicking of parallel universes. We
have two experimental proposals. The first one is a perfect optical
realization of the Schrödinger’s cat state of parallel universes (1), which is
based on an atomic ensemble in a superposition state. See Fig.2 for schematic.
The external light can change the refractive index of an atomic ensemble.
Suppose that the refractive index is sensitive to the polarization of external
light, which can be realized by adding a polarizer in front of the atomic
ensemble. Prepare the external light in a superposition of coherent states
$|\alpha_{x}\rangle$ and $|\alpha_{y}\rangle$, where $x,y$ denote the
polarization directions. Let the atomic ensemble to be sensitive to only the
y-polarization so that $|\alpha_{y}\rangle$ changes the refractive index to be
$n_{2}$, while $|\alpha_{x}\rangle$ keeps invariant the refractive index
$n_{1}\approx 1$. Then the external light
$|\alpha\rangle=\frac{1}{\sqrt{2}}(|\alpha_{x}\rangle+|\alpha_{y}\rangle)$
makes the atomic ensemble in a superposition state

$\displaystyle|n\rangle=\frac{1}{\sqrt{2}}\Big{(}|n_{1}\rangle+|n_{2}\rangle\Big{)},$
(7)

which is an optical realization of the novel state of quantum gravity (1).
Here $n_{1}$ and $n_{2}$ denote the refractive indexes of the two parallel
worlds.

Some comments are in order. 1. The key point is to prepare an atomic ensemble
in the superposition state (7). The above scheme is just a suggestion.
Experimentalists may have more feasible method to produce it. For example,
design a light switch in a superposition of on and off
$\frac{1}{\sqrt{2}}(|\text{on}\rangle+|\text{off}\rangle)$. Then an external
light passing though the light switch can induce the state (7). 2. $n_{1}$ and
$n_{2}$ need not to be the refractive indexes of Min and dS. Note that Min and
dS are just examples used in (1) to illustrate the main ideas of parallel
worlds. They can be replaced by other spacetime listed in Table.1. 3. (7) is
actually a fat Schrödinger’s cat state of the atomic ensemble. The fat
Schrödinger’s cat state has been created for 430 atoms in experiments
factcatatom . There are also interesting proposals for the enlargement of the
optical Schrödinger’s cat states factcatoptic . 4. We don’t have to worry
about the decoherence. Recall that the parallel worlds of this letter are only
effective for photons. For the atomic ensemble of size $1cm$, the
characteristic time for a photon passing through it is about $t\approx 30ps$,
which is extremely short. As a result, one only needs to keep the coherence
time of atomic ensemble to be longer than $30ps$, which is possible in
laboratory. 5. As shown in Fig.2, consider a photon passing through an atomic
ensemble in the superposition state (7). The probability to observe the photon
at the end point B is $P=(1+\cos\Delta\phi)/2$, where $\Delta\phi$ denotes the
phase difference caused by the different refractive indexes of the optical
parallel worlds.

Figure 2: A photon passes through a superposition state of (optical) parallel
worlds
$|\Psi\rangle=\Big{(}|\text{World}_{1}\rangle+|\text{World}_{2}\rangle\Big{)}/\sqrt{2}$.

Scheme II — Let us go on to discuss the second scheme, which is a partial
simulation of the novel state of parallel worlds (1). See Fig.3. Let a photon
pass through two classical optical parallel worlds with equal probability
amplitude. In other words, the photon is in a superposition state of the light
path

$\displaystyle|\Psi^{\prime}\rangle=\frac{1}{\sqrt{2}}\Big{(}|\Psi^{\prime}_{\text{pass
Min}}\rangle+|\Psi^{\prime}_{\text{pass dS}}\rangle\Big{)}.$ (8)

From the viewpoint of photons, the optical parallel worlds are in the state
(1) effectively. In fact, a photon cannot distinguish scheme I and scheme II,
that is because the probability to observe the photon at the end point B is
exactly the same for scheme I and scheme II. Note that, in scheme II, we only
need to prepare a superposition state of a photon instead of an atomic
ensemble, thus it can be achieved more easily in experiments.

Figure 3: A photon in a superposition state of paths
$|\Psi^{\prime}\rangle=\Big{(}|\text{Path}_{1}\rangle+|\text{Path}_{2}\rangle\Big{)}/\sqrt{2}$
passes through two classical (optical) parallel worlds.

One can select arbitrary two kinds of spacetime of Table.1 to constitute the
parallel worlds. The simplest choice is a Min together with a dS as in (8),
which is close to our real world: an expanding universe with small quantum
fluctuations and small Hubble constant. To perform the experiments, one can
make further simplifications. Instead of making the difficult ball devices, we
propose to make narrow cylinders and replace $n(r)$ of Table.1 by $n(z)$,
where z denotes the axial direction. The photon propagating radially outward
the isotropic universe can be mimicked well by the photon propagating along
z-axial in the cylindrical optical device. That is because the s-wave is
dominated as long as the optical length is not too large. If we focus on the
s-wave, the narrow cylinder is a good approximation.

Experimental design — Now let us present the exact experimental design. The
experimental device is shown in Fig.5, where we lay a cylindrical vapor cell
(parallel universe) along one of the light paths. For scheme I, the atomic
ensemble in the vapor cell is in a superposition state of refractive indexes
$n_{1}$ and $n_{2}$ (7), while for scheme II, the refractive index of atomic
ensemble is $n_{2}$. We set

$\displaystyle n_{1}=1,\ \ n_{2}\approx 1+\frac{1}{4}H^{2}z^{2},$ (9)

which correspond to Min and dS respectively. Here $0\leq z\leq L$, $HL<2$, $L$
is the length of vapor cell and the Hubble parameter $H$ can be controlled by
the intensity of external lights. The phase difference between the two light
paths of scheme II is given by

$\displaystyle\Delta\phi=\frac{\pi}{6\lambda_{0}}H^{2}L^{3},$ (10)

where $\lambda_{0}$ is the wavelength of photons in vacuum. Take $L=1cm$ and
$\lambda_{0}=780nm$ as an example. By changing $HL$ from $0.01$ to $0.1$, the
refractive index $n_{2}(L)$ varies from $1.000025$ to $1.0025$ and
$\Delta\phi$ changes about $21\pi$. The probabilities of obtaining a photon at
the output ports $\pm$ are given by note2

$\displaystyle
P_{I\pm}=\frac{1}{4}\left(2\pm\sqrt{2+2\cos(\Delta\phi)}\right),$ (11)

for scheme I, and

$\displaystyle P_{II\pm}=\frac{1}{2}\Big{(}1\pm\cos(\Delta\phi)\Big{)},$ (12)

for scheme II.

Atomic ensemble — Below we take the atomic ensemble as an example to
illustrate the optical simulation of parallel worlds. Compared with solid
metamaterials, it has the advantages of adjustable optical parameters and low
absorption loss. In the past, the atomic ensemble is used to study the quantum
coherence in three-level systems, such as electromagnetically induced
transparency Fleischhauer:2005 , laser without inversion Harris:1989 , and
refractive index enhancement without absorption Zibrov:1996 . These coherence
studies have many applications, such as quantum memory Choi:2008 , precision
magnetometry Scully:1992 , and high-frequency laser Scully:1994 . This
research gives a full study on the elastic properties of atomic system
parameters and realizes controllable refractive index of optical parallel
worlds by the non-absorption index enhancement.

Figure 4: The energy level diagram. $E_{p}$ is the probe beam. $E_{c1}$ and
$E_{c2}$ are two control beams. Two-photon couple the ground state $|g>$ to
excited Raman states $|1>$ and $|2>$.

Non-absorption index enhancement is realized by constructing the coherent
superposition of two two-level systems. According to the theoretical and
experimental verification, it can be realized by far detuning a four-level
system Zibrov:1996 ; Yavuz:2005 ; Proite:2008 . The energy level structure is
shown in Fig.4. The controlling beams $E_{c1}$ and $E_{c2}$ construct two
$\Lambda$-type frequency-level systems with the probe beam $E_{p}$. And the
frequency detuning are $\delta\omega_{1}$ and $\delta\omega_{2}$ respectively.
When $I_{c1}=I_{c2}$ and $\delta\omega_{1}=\delta\omega_{2}$ are satisfied,
there is perfect destructive interference in the imaginary part of the
susceptibility. And the real part of the susceptibility increased
approximately linearly with intensities of the two controlling beams. Thus, by
steering the intensity of the control beams, the refractive index of the
medium can be greatly changed, and the characteristics of very low absorption
can be realized at the same time. We will use this property to simulate the
spacetime geometry.

Combined with the above content, we design the following experimental scheme.
A Rubidium vapor cell was used as an experimental medium. During the
experiment, two controlling beams are incident along a direction with an angle
of $\theta$ with respect to the probe light. We use the space change
attenuator to control the light intensity of the two controlling beams along
the probe light direction in the atomic ensemble, so that they satisfy the
following equation,

$\displaystyle I_{c1}(z)=I_{c2}(z)=\frac{C}{4cos(\theta)}H^{2}z^{2},$ (13)

where $C$ is a constant depending on the property of the atomic ensemble
Yavuz:2005 . In this way, the refractive index experienced by the probe light
during its transmission along the atomic cell satisfies $n_{2}$ in (9). This
corresponds to the scheme II. To realize scheme I, we need to prepare the
controlling beams in a superposition state so that the atomic ensemble is in
the cat state (7).

Figure 5: Schematic of experimental setup. The experimental is performed in a
magnetically shielded Rb vapor cell. $E_{p}$ is the probe beam. $E_{c1}$ and
$E_{c2}$ are two control beams. The inset figure shows the gradient of light
intensity along the axis due to the gradient attenuator (ND) across the
control beams.

As shown in Fig.5, we put the whole device in a Mach-Zehnder interferometer
(MZI). The probe light passes through the first beam splitter, one of which
passes through an optical path with an atomic vapor cell, and the other passes
through a path without a medium. We use the MZI to reform the information of
the phase change. At the output ports $PD_{\pm}$, one can observe
$n_{0}P_{\pm}$ photons, where $n_{0}$ is number of probe photons and $P_{\pm}$
are the probabilities given by (11,12). By controlling the periodic
oscillation of piezoelectric ceramics, we can obtain periodic oscillation
signals on the detector.

Conclusion — In this letter, we study the Schrödinger’s cat state of optical
parallel universes. We design two realizable experimental schemes, which
enable to explore the mysterious ‘parallel universes’ by a MZI. The main ideas
of this letter can be generalized to Hong-Ou-Mandel interference Hong:1987zz ,
which can reveal the quantum entanglement of optical parallel worlds, and to
acoustic system such as Bose-Einstein condensation Anderson:1995gf ;
Garay:1999sk . Besides, our work can also help to study the quantum effect of
‘gravitational waves’ LIGOScientific:2016aoc  in optical universes.

## Acknowledgements

We thank Y. Wang and L. Luo for valuable comments. Zhao is supported by NSFC
grant (No.11904422). Miao thank the support by NSFC grant (No.11905297).

## References

  * (1) Everett. H, Reviews of Modern Physics 29, No.3, 1957, pp. 454-462. Reprinted in The Many-Worlds Interpretation of Quantum Mechanics, ed. B. S. DeWitt and N. Graham (Princeton: Princeton University Press), pp. 141-149, 1973.
  * (2) M. Tegmark, Nature 448, 23 (2007) [arXiv:0707.2593 [quant-ph]].
  * (3) A. H. Guth and E. J. Weinberg, Nucl. Phys. B 212, 321 (1983); A. Vilenkin, Phys. Rev. D 27, 2848 (1983); A. D. Linde, Phys. Lett. B 175, 395 (1986); Mod. Phys. Lett. A 1, 81 (1986).
  * (4) A. H. Guth, Phys. Rept. 333, 555 (2000) [arXiv:astro-ph/0002156].
  * (5) Y. Nomura, JHEP 11, 063 (2011).
  * (6) R. Bousso and L. Susskind, Phys. Rev. D 85, 045007 (2012).
  * (7) M. Tegmark, [arXiv:astro-ph/0302131 [astro-ph]].
  * (8) S. M. Feeney, M. C. Johnson, D. J. Mortlock and H. V. Peiris, Phys. Rev. D 84, 043507 (2011).
  * (9) S. M. Feeney, M. C. Johnson, D. J. Mortlock and H. V. Peiris, Phys. Rev. Lett. 107, 071301 (2011).
  * (10) U. Leonhardt, Science 312, 1777-1780 (2006).
  * (11) J. B. Pendry, D. Schurig, and D. R. Smith, Science 312, 1780-1782 (2006).
  * (12) H. Chen, R. X. Miao and M. Li, Opt. Express 18, 15183-15188 (2010).
  * (13) A. Greenleaf, Y. Kurylev, M. Lassas and G. Uhlmann, Phys. Rev. Lett. 99, 183901 (2007).
  * (14) M. Li, R. X. Miao and Y. Pang, Phys. Lett. B 689, 55-59 (2010).
  * (15) M. Li, R. X. Miao and Y. Pang, Opt. Express 18, 9026-9033 (2010).
  * (16) I. I. Smolyaninov, J. Opt. 13, 024004 (2011).
  * (17) U. Leonhardt and T. G. Philbin, New J. Phys. 8, 247 (2006).
  * (18) D. A. Genov, S. Zhang, and X. Zhang, Nature Phys. 5, 687-692 (2009).
  * (19) H. Chen, S. Tao, J. Bělín, J. Courtial and R. X. Miao, Phys. Rev. A 102, no.2, 023528 (2020).
  * (20) T. M. Zhao and R. X. Miao, Opt. Lett. 36, 4467-4469 (2011).
  * (21) A. G. Riess et al. [Supernova Search Team], Astron. J. 116, 1009-1038 (1998).
  * (22) S. Perlmutter et al. [Supernova Cosmology Project], Astrophys. J. 517, 565-586 (1999).
  * (23) Marletto, C., Vedral, V. Nature 547, 156–158 (2017).
  * (24) J. Foo, R. B. Mann and M. Zych, Class. Quant. Grav. 38, no.11, 115010 (2021).
  * (25) J. Plebanski, Phys. Rev. 118, 1396 (1960).
  * (26) Note that one can perform a further coordinate transformation $t\to t/c_{1}$, then the refractive indexes of table.1 become $n\to c_{1}n$, where $c_{1}$ is a constant. For simplicity, we set $c_{1}=1$ in this letter.
  * (27) Gerlich, S., Eibenberger, S., Tomandl, M. et al. Nat Commun 2, 263 (2011).
  * (28) Sychev, D., Ulanov, A., Pushkina, A. et al. Nature Photon 11, 379–382 (2017).
  * (29) Since the atomic ensemble is in a superposition state for scheme I, there are three effective light paths in the experimental device of Fig.5 (two effective light paths inside the vapor cell). As a result, the probability of obtaining a photon at the output port is (11) instead of $P=(1\pm\cos\Delta\phi)/2$.
  * (30) M. Fleischhauer, A. Imamoglu, and J. P. Marangos, Rev. Mod. Phys. 77, 633-673 (2005).
  * (31) S. E. Harris, Phys. Rev. Lett. 62, 1033-1036 (1989).
  * (32) A. S. Zibrov, M. D. Lukin, L. Hollberg, et. al. Phys. Rev. Lett. 76, 3935-3938 (1996).
  * (33) K. S. Choi, H. Deng, J. Laurat and H. J. Kimble Nature 452, 67-71 (2008).
  * (34) M. O. Scully and M. Fleischhauer, Phys. Rev. Lett. 69, 1360-1363 (1992).
  * (35) M. O. Scully and M. Fleischhauer, Science 263, 337-338 (1994).
  * (36) D. D. Yavuz, Phys. Rev. Lett. 95, 223601 (2005).
  * (37) N. A. Proite, B. E. Unks, J. T. Green and D. D. Yavuz, Phys. Rev. Lett. 101, 147401 (2008).
  * (38) C. K. Hong, Z. Y. Ou and L. Mandel, Phys. Rev. Lett. 59, 2044-2046 (1987).
  * (39) M. H. Anderson, J. R. Ensher, M. R. Matthews, C. E. Wieman and E. A. Cornell, Science 269, 198-201 (1995).
  * (40) L. J. Garay, J. R. Anglin, J. I. Cirac and P. Zoller, Phys. Rev. Lett. 85, 4643-4647 (2000).
  * (41) B. P. Abbott et al. [LIGO Scientific and Virgo], Phys. Rev. Lett. 116, no.6, 061102 (2016).