# Behavioral Cloning via Search in Embedded Demonstration Dataset

Federico Malato111Equal contribution.
School of Computing
University of Eastern Finland
Joensuu, Finland
<EMAIL_ADDRESS>Florian Leopold∗
Bielefeld University
Bielefeld, Germany
<EMAIL_ADDRESS>Ville Hautamäki
School of Computing
University of Eastern Finland
Joensuu, Finland
<EMAIL_ADDRESS>Andrew Melnik
Bielefeld University
Bielefeld, Germany
<EMAIL_ADDRESS>
###### Abstract

Behavioural cloning uses a dataset of demonstrations to learn a behavioural
policy. To overcome various learning and policy adaptation problems, we
propose to use latent space to index a demonstration dataset, instantly access
similar relevant experiences, and copy behavior from these situations. Actions
from a selected similar situation can be performed by the agent until
representations of the agent’s current situation and the selected experience
diverge in the latent space. Thus, we formulate our control problem as a
search problem over a dataset of experts’ demonstrations. We test our approach
on BASALT MineRL-dataset in the latent representation of a Video PreTraining
model. We compare our model to state-of-the-art Minecraft agents. Our approach
can effectively recover meaningful demonstrations and show human-like behavior
of an agent in the Minecraft environment in a wide variety of scenarios.
Experimental results reveal that performance of our search-based approach is
comparable to trained models, while allowing zero-shot task adaptation by
changing the demonstration examples.

##  1 Introduction

Imitation Learning (IL) [1] and Reinforcement Learning (RL) [2, 3, 4, 5] are
commonly used methods to train agents to perform tasks in simulated and real
environments, including multi-skill agents. Despite the extensive research in
these fields, there are persistent challenges that pose difficulties for IL
and RL methods. Firstly, these methods typically require significant
computational resources for training. Secondly, when faced with new
experiences, retraining or fine-tuning of trained models becomes necessary.
Thirdly, the lack of one-shot or few-shot adaptability is a bottleneck, which
is current focus of numerous studies on large language [6] and vision models
[7, 8]. Exploring alternative approaches to IL and RL methods for control
problems can potentially address these challenges and yield advantages in
specific application domains.

A multi-modal latent space can be used for measuring similarity and multi-
modal search. Some common ways to achieve multi-modal mapping is contrastive
learning [9] and supervised learning, where in the later case representation
at the last layers or a few layers before the output can serve as a multi-
modal latent space [10]. For control tasks, such multi-modal latent space can
be used to connect visual and action experiences [11] based on a demonstration
dataset. Thus, database search in this latent space can provide actions for
control. Such an approach can lead to improvements with respect to the three
general challenges for IL and RL approaches mentioned above.

##### Motivation

The present study is motivated by the MineRL BASALT 2022 challenge [12, 13].
In the challenge, an agent must solve the four tasks: find a cave, build an
animal pen, build a village house, and make a waterfall [12, 13]. For these
tasks, no reward function is provided. Agents were judged by human contractors
on two criteria: success in the task and human-likeliness of behaviour.
Although, the objective is not mathematically well-defined, for a human, the
success in completing such tasks is easy to evaluate and select the best one
out of two performance examples. To solve the tasks, participants had a
dataset of expert demonstrations at their disposal. Each demonstration
trajectory of image-action pairs contains a single episode of one of the tasks
solved by a human expert. Simplest solution to this then is behavioral cloning
(BC), where policy is trained using supervised learning.

BC is limited by the expert demonstration trajectories used in the training,
and due to the training scheme used, sampled action per state is optimal in
terms of expectation. But in practice agents are facing different situations
throughout the episode, where one might be avoiding an obstacle and other to
search for suitable location. These situations each then clearly require
different solutions.

To this end, we introduce search-based behavioural cloning (S-BC), an approach
to BC that addresses that issue. S-BC generates a latent space of experts’
trajectories that encapsulate both present and past information, and
reformulates the control problem as a search problem in such a space. By
allowing to search for new closest situation from the set of expert
trajectories, our method in a effect adapts to the evaluation episode.

Figure 1: A scheme of the VPT model used in this study. An image input is
encoded with an IMPALA CNN and passed through four transformer heads. Then,
two MLP heads predict a keyboard and a mouse action respectively. Figure 2: An
example of the search mechanism. At each timestep, we keep track of the
distance between current and reference embedding. Whenever the distance
overcomes a threshold, a divergence-based search (red line) selects a new
reference embedding; if the agent follows a threshold for too long, a time-
based search (blue line) is triggered.

##  2 Related work

BC has been successfully applied to practical control problems ranging from
autonomous driving [14, 15, 16] to playing video games [17, 18, 19]. Despite
being tremendously popular due to its simplicity, BC suffers from a range of
problems such as distributional shift and causal confusion [20]. These
limitations are addressed using inverse reinforcement learning [21] and
generative adversarial imitation learning [22], which on the other hand tend
to be computationally heavy for complex problems and hard to train.

Recently proposed Video Pre-Training (VPT) model [10] (see Figure 1) is a
foundation model for BC trained on 70k+ hours of video content scraped from
the internet [10]. VPT is built on an IMPALA [23] convolutional neural network
(CNN) backbone. The CNN maps an image input to a 1024-dimensional feature
vector. VPT generates a batch of 129 vectors and forward them to four
transformer blocks. Each block is linked to a memory block containing the last
128 frames. At the end of the transformer pipeline, only the last frame is
retained and forwarded to two heads based on Multilayer Perceptrons (MLPs).
One head predicts a keyboard action and second head predicts a computer mouse
action. We provide an analysis of the architecture in Appendix A.

##  3 Methods

Our aim is to solve a complex problem in Minecraft environment, where no
explicit reward is available. The only data source that we can utilize is a
set of recorded expert trajectories solving that given task. The core idea
behind our approach is to reformulate the control problem as a search problem
on the experts’ demonstrations.

We use a pre-trained VPT model [10] to encode situations in latent space. The
pre-trained version of the model used in this study is available at the
official GitHub repository [24]. The repository features three foundation
models, namely 1x, 2x and 3x. The backbone of the three models is the same,
and they differ only for the weights width [24].

###  3.1 Search-based BC

S-BC solves control problems by retrieving relevant past experiences from
experts demonstrations. In this context, we define a situation as a set of
consecutive observation-action pairs
$\\{(o_{t},a_{t}),\dots,(o_{t+n},a_{t+n})\\}$, $n>0$.

We illustrate our approach in Figure 3. We use VPT to extract all the features
embeddings from an arbitrary subset $\mathcal{S}$ of the demonstrations
dataset $\mathcal{D}$, $\mathcal{S}\subseteq\mathcal{D}$. The set of these
embeddings constitutes the N-dimensional latent space that S-BC searches.
Moreover, the expert’s optimality assumption [25] from IL methods ensures that
each situation has been addressed and solved optimally.

During testing, we pass the current situation through VPT. Then, S-BC searches
for the most similar embedding point in the space. Here, we compute similarity
among any two situations as their L1 distance. We copy the actions of the
selected reference situation. At each time-step, we shift the current and
reference situations in time and recompute their distance. When situations
diverge over time, the approach performs a new divergence-triggered search
(red lines in Figure 2). If the reference embedding is followed form more than
$n$ time-steps, the approach performs a new time-triggered search (blue lines
in Figure 2).

Our S-BC process is faster than fine-tuning a VPT-based BC agent, training
GAIL or RL agent (see Figure 4). At each time-step of the evaluation (Figure
3B), the current observation coming from the MineRL environment is encoded
through VPT. Then, S-BC compute the L1 distance with the current reference
situation. If the two have diverged, a new search in latent space is performed
and a new reference situation is selected. Otherwise, we retain the same
reference situation. In both cases, S-BC copies the actions coming from the
current reference embedding.

Figure 3: Our approach. (A) Latent space generation: trajectories are
extracted from the demonstration dataset. Frames are encoded through a
provided VPT model, and paired with the corresponding actions. (B) Evaluation
loop: at each timestep, the new observation is forwarded to the same VPT
model. Then, L1 distance across current and reference embeddings is computed
and the most similar situation is found. S-BC acts in the environment
following the actions of the selected reference situation. Figure 4: Time
needed to train each agent on 100 trajectories on the FindCave task. In the
case of BC models, the training procedure consists of fine-tuning a pre-
trained VPT model. For the S-BC model, training means encoding a subset of
trajectories through the reference version of VPT. All models have been
trained on a single Tesla T4 GPU.

##  4 Experiments

The full FindCave dataset from the MineRL BASALT competition [12] consists of
5466 experts’ trajectories demonstrating the task, or around 150GB of data.
For each frame of an episode, only the unprocessed RGB image and the
corresponding action are available. That is, an episode is a set of image-
action pairs. No reward signal is available. Similarly, no measure of
performance is provided. In our study we use only a small subset of the
available data. Specifically, we consider only the first $100$ trajectories.

Additionally, we personally gather expert trajectories for the MineDojo task.
For all our tasks, we modified the framework to accept mouse and keyboard
inputs and recorded them in the same format as the FindCave dataset. For each
task, we record $100$ trajectories. Implementation details about MineDojo
modifications and data gathering procedures are reported in Appendix B1.

###  4.1 MineDojo tasks

We compare S-BC against state-of-the-art models in the Minecraft (VPT-based
$1x$, $2x$, and $3x$ models) and IRL domains (GAIL). We fine-tune each VPT-
based model for two full epochs on our gathered data, and fine-tune a
different model for each task. Moreover, we train GAIL using VPT-encoded
trajectories, to speed up training by reduce the size of the observation
space. We train GAIL for 6h for each task. We evaluate the models in two ways.
First, we use the MineDojo [26] framework to provide clear numerical results.

In the suite, authors distinguish ground-truth and creative tasks. On one
hand, ground-truth tasks feature a well-defined terminal condition, such as
"kill a specific unit" or "gather $x$ quantity of material", and are separates
in broad categories such as combat, build, harvest, and so on. On the other
hand, creative tasks do not feature a terminal condition. Although, they can
be evaluated indirectly through MineCLIP [26] by mapping a text prompt and a
video frame in a shared latent space and then computing cosine distance to
assess similarity between them. Such similarity score is computed for each
timestep, as in standard RL.

For our experiments, we select four ground-truth tasks, namely, combat spider
and combat zombie from the combat category and harvest log, harvest wheat seed
from the harvest category, and two creative tasks, "find ocean" and "dig a
hole".

###  4.2 FindCave

As second set of experiments, we compare the agents directly on the FindCave
task from BASALT [13], where no reward information is known or obtainable. To
support the evaluation process, we train a simple binary classifier to detect
the presence of a cave. Our classifier is composed by four convolutional
layers and two fully-connected layers. The network has been trained with
supervised learning on a dataset of cave frames, extracted from the original
FindCave data, and achieved a validation accuracy of $97.89\%$ over it.
Further details are discussed in Appendix B2. To ensure fairness over the
amount of accessible relevant knowledge to each model, we use the same subset
of $100$ trajectories from the original BASALT dataset on them.

We apply minimal changes to the task rules with respect to the official
competition, to ease the evaluation procedure while keeping the performance
evaluation intact. First, we disable the terminal action ’ESC’ in all the
agents. We support this decision by highlighting that terminal actions
constitute a minimal fraction of the amount of training data, and a BC agent
would likely ignore it, or use it incorrectly. Therefore, we removed the
action to evaluate the Second, the BASALT competition considers an episode
successful whenever an agent performs the terminal action while being inside a
cave. Instead, we consider an episode successful whenever an agent spends more
than five seconds in a cave. The maximum time limit for each evaluation
episode is set to three minutes, similarly to the official competition rules.

As in the evaluation process of the BASALT competition, we test our agent on
twenty seed values. Since the official values used in BASALT evaluation are
not publicly available, we have selected each seed manually to ensure the
presence of caves. For each run, we play 10 evaluation episodes.

##### Latent space visualization

We perform a study over the latent space used from our agent. We plot t-SNE
representation of a full latent space and evaluate position of the cave
frames, similarity between close points. Finally, we track the movement of an
agent using such latent space, to study its dynamics.

##### Evaluation procedure

Our evaluation procedure is as follows. First, we let an agent play the
evaluation episodes and record each one of them. Then, we pass each frame of
the video to the classifier, and label them as either cave or non-cave frames.
Whenever a cave frame is detected, we start counting the number of consecutive
cave frames. Following the MineRL specification [27], we assign each frame a
duration of $0.05$s. This lead to set the threshold of five seconds to $100$
consecutive cave frames. Whenever such threshold is exceeded, we consider an
episode successful.

To account for unique situations, we run additional evaluations lowering the
time threshold. For each spotted situation, we take a look at the current
frame and evaluate it manually. Unique situations include, but are not limited
to, agent reaching a cave and being killed before the five second mark, agent
digging a cave itself (not allowed from BASALT competition rules), or agent
walking in exceptionally dark areas that might be mistaken as caves.

##  5 Results

| BC 1x | BC 2x | BC 3x | GAIL | S-BC (Ours)
---|---|---|---|---|---
Find Ocean | $20.25\pm 1.10$ | $20.19\pm 1.04$ | $19.82\pm 1.05$ | $20.30\pm 0.74$ | $\textbf{20.47}\pm 1.11$
Dig Hole | $20.80\pm 0.88$ | $20.94\pm 0.85$ | $20.95\pm 0.87$ | $20.20\pm 0.24$ | $\textbf{20.97}\pm 0.92$
Table 1: Average MineCLIP score and its standard deviation per model on the
evaluated creative tasks. MineCLIP scores can be used to train agents in the
MineDojo suite in an RL fashion. Figure 5: Results for four MineDojo ground-
truth tasks. (A) Model success rate over 100 evaluation episodes. Reference
baseline model BC 1x is highlighted with a horizontal blue line in each plot.
(B) Average completion time for each task. Tasks that have never been
completed are associated with a blank space. Tasks completed only one time
show a single line corresponding to the completion time of that single
success.

###  5.1 MineDojo tasks

We evaluate our models over six diverse tasks of the MineDojo suite. Figure 5
shows the models success rate and completion time for each task. Additionally,
we report in Table 1 the average score obtained by each model in each creative
task.

Notably, S-BC performs similarly or better than its competitors. Figure 5A
shows that our method is the only one able to complete instances of the combat
tasks. In harvest tasks, our model either performs similarly or lose a mere 5%
in performance with respect to much bigger, learning-based models. It is also
interesting to notice that VPT-based foundation model were originally trained
on the ObtainDiamond track [28], which first step consists in collecting logs.
We highlight that GAIL was not able to complete any of the ground-truth tasks.
We point out that we have trained GAIL for a time that is comparable to the
fine-tuning time of the VPT-based models ( 1h). Therefore, due to the limited
amount of time used for training, it might under perform on some tasks.

Figure 5B shows the average time of completion for successful episodes. We
highlight that S-BC requires either the same on less time to complete, on
average. Our agent is on average the faster in completing harvest one log, and
performs similarly in harvest one wheat seed. In the combat spider case, S-BC
was the only method able to complete the task consistently, using on average
two thirds of the time at its disposal.

Table 1 describes the results of the creative tasks. According to MineCLIP,
all the models performed similarly. It is interesting to notice that in
creative tasks, GAIL seems to achieve comparable performance to other methods.

Figure 6: Average success rate for our tested model on the FindCave task over
$3$ runs. Each agent has been evaluated over $20$ seeds, playing 10 episodes
on each seed. Baseline model is highlithed with a vertical blue line.

###  5.2 FindCave

We report success rate for the models on the FindCave task in Figure 6. S-BC
has obtained the best performance, being able to complete the task 40% of the
times. GAIL was never able to find a cave, while VPT-based BC models were
quite successful, reporting success rates of $15\%$, $25\%$ and $30\%$ for the
1x, 2x, and 3x.

Figure 7: t-SNE diagrams of S-BC latent space with $10$ trajectories. (A)
Separation between exploration (blue) and cave (red) frames. Some examples of
cave frames are linked to the corresponding point (symbols) in the latent
space. (B) t-SNE representation of the same space with points separated
according to their belonging trajectory. Cave frames for each trajectory are
marked with a matching color star symbol. Zoom: a close-up look at some
clustered cave frames coming from different trajectories.

###  5.3 Latent space visualization

We have analyzed the VPT latent space used by our models and its
transformation over the number of encoded trajectories. To visualized it, we
have used t-SNE [29] plotting. Figures 7A and 8 show two examples of latent
space used by S-BC, with $10$ and $25$ encoded trajectories respectively. We
have classified each point as either a cave (red) or exploration frame (blue).
We used our cave classifier to determine all cave frames. In our study, we
have analyzed visual hints of structure of those spaces. Additionally, we have
extracted some meaningful pieces of information such as similarity across
frames and the search path of an episode.

####  5.3.1 Space structure & cave frames

Both $10$ and $25$ trajectories latent space plots (Figure 7A and 8,
respectively) show that the embeddings in the VPT latent space can be
separated to some degree, but are quite dense. Perhaps not surprisingly, an
increase in the number of encoded trajectories results in a less structured
space. This is supported by the visual appearance of a Minecraft frame: the
game features several biomes, that range from desert to snowy mountains. In
the FindCave task, an agent is bounded to spawn in a plain biome. Such biome
is characterized by the presence of fields, forests, ponds and, sometimes,
lakes or sea. Therefore, a number of visually different frames are possible.
But, whenever switching from two different environments (e.g. from lake to
forest), we expect some "transition frames" that contain a mixture of elements
of both. In a visualization method such as t-SNE, we expect such frames to be
equally distant from proper examples of any of the two environments. That is,
we expect them to be in the middle of two more dense regions of space. So, as
the number of trajectories increase, we might expect the number of transition
frames to increase, thus making well-defined clusters more rare.

The position of cave frames within the space constitute another notable result
of this visualization. The vast majority of cave frames are grouped in half of
the space. Within this region, some clusters are easily spotted, but in
general cave frames tend to be sparse.

####  5.3.2 Visual similarity across points

In our study, we have visualized several frames of the latent space, to
understand the relation between their visual resemblance and spatial
coordinates. We show some examples in Figure 7A: frames linked with a green
symbol are cave frames belonging to the same trajectory; frames corresponding
to an orange symbol are cave frames coming from different trajectories;
finally, frames marked with a yellow symbol are exploration frames belonging
to the same cluster.

By comparing green and orange examples, it is clear that each cave cluster
contains frames from a different trajectory. Additionally, frames linked to a
yellow symbol show how each cluster is formed by visually similar examples.
Such morphology of the space suggests that, depending on the current
situation, an S-BC agent will select a close point whenever the visual
transition across situations is smooth. On the contrary, whenever the
observation drastically changes, the agent will look for a distant new
situation to follow.

To account for the points distribution of each trajectory, we have plotted a
second version of the $10$ trajectories latent space (Figure 7B). In this
representation, each point is colored according to its belonging trajectory.
We have marked the cave frames of a specific trajectory with a star symbol of
the same color.

Notably, each cluster contains points of a variety of colors. This supports
the fact that visual resemblance plays a fundamental role in deciding a point
position within the space: similar frames (e.g. fields, forests, and so on)
coming from different trajectories are squeezed together. Therefore, one might
conclude that VPT embeddings are heavily biased towards visual similarity.
While we believe that such metric is indeed important, we have identified
clues that suggest other important factors in defining them. In particular, we
believe that past information is another key factor in this regard. To support
this claim, consider the yellow examples: while the square and triangle seems
to be the most visually similar pair, their distance is greater than e.g.
triangle and rhombus. Such observation imposes to consider other factors in
the characterization of a VPT embedding.

####  5.3.3 Example of one episode using S-BC

Figure 8: Visualization of the VPT embeddings latent space with 25
trajectories encoded. A yellow point marks a trajectory used from the agent
during the corresponding episode.

We have visualized the situations that an S-BC agent trained on 25
trajectories has followed in a successful evaluation episode. The result is
shown in Figure 8. Each yellow dot corresponds to a new search performed by
our agent at some point in the episode. Notably, $4$ out of $6$ searches were
performed in a cave frames cluster. In the episode, the agent enters the cave
after around $15$s. Then, it wanders there and gets killed after $30$s. The
fact that our agent has performed a total of $6$ searches suggests that, on
average, it has followed each selected trajectory for $5$s (around $100$
frames).

###  5.4 Perceptual evaluation

Team | FindCave | MakeWaterfall | AnimalPen | House | Average
---|---|---|---|---|---
GoUp | $0.31$ | 1.21 | 0.28 | 1.11 | 0.73
Our approach | 0.56 | -0.10 | 0.02 | 0.04 | 0.13
voggite | 0.21 | 0.43 | -0.20 | -0.18 | 0.06
JustATry | -0.31 | -0.02 | -0.15 | -0.14 | -0.15
TheRealMiners | 0.07 | -0.03 | -0.28 | -0.38 | -0.16
Human2 | 2.52 | 2.42 | 2.46 | 2.34 | 2.43
Human1 | 1.94 | 1.94 | 2.52 | 2.28 | 2.17
BC-Baseline | -0.43 | -0.23 | -0.19 | -0.42 | -0.32
Random | -1.80 | -1.29 | -1.14 | -1.16 | -1.35
Table 2: Top-5 ranking of the NeurIPS BASALT 2022 competition [13]. Below, the
TrueSkill [30] scores for two human expert players, a BC baseline and a random
agent are reported for comparison.

To determine its rank among the participants, the organizers of the
competition have used the TrueSkill [30] ranking system, which is widely used
in the Microsoft online gaming ecosystem. Given a set of competitors, the
system uses Bayesian inference to compute an ELO-like score, according to the
match history of each competitor.

In the context of the BASALT competition, the system considers each trained
agent as a participant. To create a match history for each of them, any two
participants are drawn and their evaluation videos are judged by a human
contractor. For each pair of videos, the contractor indicates which agent was
able to solve the task (or which one was closer), and how human-like an agent
appeared to be. Those two ratings are then aggregated to the match history of
each competitor. When a suitable amount of outcomes are available for an
agent, the TrueSkill for that agent is computed.

Our proposed S-BC agent was overall ranked second place in the challenge. The
results of our agent are described in 2. First place was awarded to team GoUp.
Their method uses detection methods and leverages human knowledge of the task
through scripting [13]. Notably, all the other learning-based methods achieved
lower performance than S-BC. Additionally, our method was awarded with $2$ out
of $5$ research innovation prizes.

####  5.4.1 Behavior of agents in videos

In our study we have collected and analyzed a wide number of videos that
demonstrate the general behavior of each agent. From those, we were able to
delineate some differences in the behavior of BC and S-BC agents:

##### BC models

As a premise, it is important to note that the reference version of VPT
provided by the organizers of the competition was trained on data from the
MineRL ObtainDiamond track [28] and then fine-tuned on the BASALT [12, 13]
dataset. Judging from the videos, our additional fine-tuning towards the
FindCave task shows some improvements of the agents. Overall, though, the
general behavior remains the primary objective of the ObtainDiamond
competition: it is not rare that a BC agent chops trees as first step. We
believe that the limited amount of episodes used for fine-tuning has had a
clear negative impact on the performance. We expect an agent fine-tuned on the
full FindCave dataset to be more aligned with the expected FindCave behavior,
at the cost of a drastic increase in training time.

##### S-BC

We found our agent to be quite robust in performance, and be human-like at the
same time. S-BC shows significantly more aligned behavior with the proposed
task than any BC agent. In general, our agent is able to recover meaningful
trajectories when the cave is nearby. Despite this, our agent is far from
being perfect: in some instances, S-BC has shown dichotomic behavior regarding
caves. At first, the agent would face a cave and quickly turn around. In the
following seconds, a new search correctly guiding the agent towards the same
cave was performed. We believe that this behavior comes from the possible
presence of enemies in caves, which would influence the retrieval of a better
suited reference situation.

##  6 Conclusions

We introduced S-BC, a BC-inspired search-based approach that uses the past
experience of experts to solve similar control problems. Our approach shows
better or similar performance, requiring less training time, allowing a few-
shot learning, and multi-skill performance when skill demonstration is
provided. We applied our method to the MineRL BASALT NeurIPS 2022 Challenge,
where our agent reached overall 2nd place. The agent had to demonstrate human-
like behavior while completing the tasks. Numerical evaluations over a suite
of MineDojo tasks shows that our agent reaches similar or better performance
than state-of-the-art BC and IRL methods, while requiring no re-training or
fine-tuning.

## References

  * [1] S. Schaal, “Learning from demonstration,” in Advances in Neural Information Processing Systems (M. Mozer, M. Jordan, and T. Petsche, eds.), vol. 9, MIT Press, 1996.
  * [2] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction.  The MIT Press, second ed., 2018.
  * [3] M. Schilling and A. Melnik, “An approach to hierarchical deep reinforcement learning for a decentralized walking control architecture,” in Biologically Inspired Cognitive Architectures 2018: Proceedings of the Ninth Annual Meeting of the BICA Society, pp. 272–282, Springer, 2019.
  * [4] N. Bach, A. Melnik, M. Schilling, T. Korthals, and H. Ritter, “Learn to move through a combination of policy gradient algorithms: Ddpg, d4pg, and td3,” in International Conference on Machine Learning, Optimization, and Data Science, pp. 631–644, Springer, 2020.
  * [5] M. Schilling, A. Melnik, F. W. Ohl, H. J. Ritter, and B. Hammer, “Decentralized control and local information for robust and adaptive decentralized deep reinforcement learning,” Neural Networks, vol. 144, pp. 699–725, 2021.
  * [6] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al., “Training language models to follow instructions with human feedback,” Advances in Neural Information Processing Systems, vol. 35, pp. 27730–27744, 2022.
  * [7] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023.
  * [8] K. Rana, A. Melnik, and N. Sünderhauf, “Contrastive language, action, and state pre-training for robot learning,” arXiv preprint arXiv:2304.10782, 2023.
  * [9] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning, pp. 8748–8763, PMLR, 2021.
  * [10] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, and J. Clune, “Video pretraining (vpt): Learning to act by watching unlabeled online videos,” Advances in Neural Information Processing Systems, vol. 35, pp. 24639–24654, 2022.
  * [11] S. Beohar, F. Heinrich, R. Kala, H. Ritter, and A. Melnik, “Solving learn-to-race autonomous racing challenge by planning in latent space,” arXiv preprint arXiv:2207.01275, 2022.
  * [12] R. Shah, C. Wild, S. H. Wang, N. Alex, B. Houghton, W. H. Guss, S. P. Mohanty, A. Kanervisto, S. Milani, N. Topin, P. Abbeel, S. Russell, and A. D. Dragan, “The minerl BASALT competition on learning from human feedback,” CoRR, vol. abs/2107.01969, 2021.
  * [13] S. Milani, A. Kanervisto, K. Ramanauskas, S. Schulhoff, , B. Houghton, S. Mohanty, B. Galbraith, K. Chen, Y. Song, T. Zhou, B. Yu, H. Liu, K. Guan, Y. Hu, T. Lv, F. Malato, F. Leopold, A. Raut, V. Hautamäki, A. Melnik, S. Ishida, J. F. Henriques, R. Klassert, W. Laurito, E. Novoseller, V. G. Goecks, N. Waytowich, D. Watkins, J. Miller, and R. Shah, “A retrospective of the minerl basalt 2022 competition]Towards Solving Fuzzy Tasks with Human Feedback: A Retrospective of the MineRL BASALT 2022 Competition,” arXiv, 2023.
  * [14] S. K. Saksena, B. Navaneethkrishnan, S. Hegde, P. Raja, and R. M. Vishwanath, “Towards behavioural cloning for autonomous driving,” 2019.
  * [15] T. V. Samak, C. V. Samak, and S. Kandhasamy, “Robust behavioral cloning for autonomous vehicles using end-to-end imitation learning,” SAE International Journal of Connected and Automated Vehicles, vol. 4, 2021.
  * [16] S. Beohar and A. Melnik, “Planning with rl and episodic-memory behavioral priors,” arXiv preprint arXiv:2207.01845, 2022.
  * [17] A. Kanervisto, J. Karttunen, and V. Hautamäki, “Playing minecraft with behavioural cloning,” CoRR, vol. abs/2005.03374, 2020.
  * [18] A. Kanervisto, J. Pussinen, and V. Hautamaki, “Benchmarking End-to-End Behavioural Cloning on Video Games,” in IEEE Conference on Computatonal Intelligence and Games, CIG, vol. 2020-August, 2020.
  * [19] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver, “Grandmaster level in starcraft ii using multi-agent reinforcement learning,” Nature, vol. 575, 2019.
  * [20] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” vol. 32, 2019.
  * [21] A. Y. Ng and S. J. Russell, “Algorithms for inverse reinforcement learning,” in Proceedings of the Seventeenth International Conference on Machine Learning, ICML ’00, (San Francisco, CA, USA), p. 663–670, Morgan Kaufmann Publishers Inc., 2000.
  * [22] J. Ho and S. Ermon, “Generative adversarial imitation learning,” 2016.
  * [23] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, S. Legg, and K. Kavukcuoglu, “IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures,” CoRR, vol. abs/1802.01561, 2018.
  * [24] “Video-pre-training.” https://github.com/openai/Video-Pre-Training/tree/main/lib.  Accessed: 2023-05-14.
  * [25] S. Russell, Human Compatible.  Penguin, 2019.
  * [26] L. Fan, G. Wang, Y. Jiang, A. Mandlekar, Y. Yang, H. Zhu, A. Tang, D.-A. Huang, Y. Zhu, and A. Anandkumar, “Minedojo: Building open-ended embodied agents with internet-scale knowledge,” in Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  * [27] W. H. Guss, B. Houghton, N. Topin, P. Wang, C. Codel, M. Veloso, and R. Salakhutdinov, “MinerL: A large-scale dataset of minecraft demonstrations,” in IJCAI International Joint Conference on Artificial Intelligence, vol. 2019-August, 2019.
  * [28] A. Kanervisto, S. Milani, K. Ramanauskas, N. Topin, Z. Lin, J. Li, J. Shi, D. Ye, Q. Fu, W. Yang, W. Hong, Z. Huang, H. Chen, G. Zeng, Y. Lin, V. Micheli, E. Alonso, F. Fleuret, A. Nikulin, Y. Belousov, O. Svidchenko, and A. Shpilman, “Minerl diamond 2021 competition: Overview, results, and lessons learned,” 2022.
  * [29] L. V. D. Maaten and G. Hinton, “Visualizing data using t-sne,” Journal of Machine Learning Research, vol. 9, 2008.
  * [30] T. Minka, R. Cleven, and Y. Zaykov, “Trueskill 2: An improved bayesian skill rating system,” Tech. Rep. MSR-TR-2018-8, Microsoft, March 2018.

##  Appendix A Appendix - Architectures details

###  A.1 Details on VPT architecture

OpenAI’s Video Pre-Training models(6) were trained using behavioral cloning on
a large video dataset, resulting in the foundation models that were able to
solve simple tasks in the MineRL environment without further fine-tuning like
RL. The authors also provide fine-tuned model variants, which are more
specialized to certain tasks. For similarity search of the embedded situations
and experiments, we used a pre-trained foundation-
model-1x.weights222https://openaipublic.blob.core.windows.net/minecraft-
rl/models/foundation-model-1x.weights VPT model.

The VPT models take a single RGB frame, resized to $128\times 128$, as the
input, which is passed through an IMPALA CNN (25) to generate
$1024$-dimensional image embeddings. These embeddings are then passed to four
residual, recurrent transformer blocks. These transformer blocks contain a
memory state, which consists of [key, value] pairs of the past $128$ steps
(see Figure 9), but only use the current input as the query. After these
blocks, a $1024$-dimensional intermediate representation is put out before the
MLP heads, which are used to predict keyboard and computer mouse actions.
These intermediate representation latents were used for the embedding of
situations. OpenAI’s VPT code implementation was made publicly available on
GitHub (18).

Figure 9: A scheme of the VPT model used in this study. An image input is
encoded with an IMPALA CNN (25) and passed through four transformer blocks.
Then, two MLP heads predict a keyboard and a mouse action respectively.

###  A.2 Details on MineCLIP

Figure 10: A high level scheme of the MineCLIP architecture. On one branch, a
text prompt is encoded with a variant of GPT on a 512d embedding. On the other
branch, 16 video frames are encoded through ViT/16 in an embedding of the same
size. Finally, a reward is computed using cosine similarity among these two
embeddings.

We use a pretrained MineCLIP (20) model for the evaluations, provided by the
original authors. Its architecture, as seen in Figure 10, consists of three
main components: an image encoder, a temporal aggregator and a text encoder.
The output scores are computed via a cosine similarity measurement between the
encoded observations and the encoded text prompt.

  * •

The frame-wise image encoder utilizes the ViT-B/16 architecture (26) and
embeds input images of size $256\times 160$ into a $512$-dimensional latent
space.

  * •

The temporal aggregator originally comes in two variants, with one being based
on average pooling, and the other utilizing self-attention. We used the
attention variant, as it had better results in the original paper (20). It is
a 2-layer, self-attention transformer with an embedding size of $512$ and $8$
attention heads. The authors also added two additional residual CLIP Adapter
(27) layers after the aggregator.

  * •

The text encoder utilizes a $12$-layer GPT with $8$ attention heads and an
embedding size of $512$. The corresponding text tokenizer has a maximum length
of $77$ tokens and uses a vocabulary of size $49152$.

MineCLIP is trained to assign a score to each frame based on the cosine
similarity between a text and an image embeddings, consistently with the
classic RL paradigm. Therefore, using the average episode score is a coherent
choice with assessing the quality of an agent.

##### Evaluation procedure

During the evaluation, we resize the observed frames from the original VPT
resolution (6) of $640\times 360$ to the appropriate MineCLIP size of
$256\times 160$, and use a sliding window of $16$ consecutive frames as the
input, as was proposed in the original paper (20). To compute the final
MineCLIP scores per task, we used the text prompts specified by MineDojo:
"Explore the world to find ocean." for the find ocean task and "Dig a hole in
the ground." for the dig a hole tasks.

##  Appendix B Appendix - Implementation details

###  B.1 MineDojo

For our experiments, we performed modifications on MineDojo. To be able to use
the OpenAI VPT models, which were created for MineRL 1.0 while MineDojo is
based on MineRL 0.4.4, the action space of the environments had to be adapted.

This includes removing event-level control, such as equip or craft actions,
and adding hotbar selection actions and fine-grained mouse control. These
modifications also allowed us to create a human interface to collect expert
trajectories for the corresponding tasks.

Additionally, we enabled the "fast reset" option for all environments, which
significantly speeds up their reset times by teleporting the agent to a random
place far away on a reset instead of generating an entirely new world each
time. However, by specifying a large number as the range parameter for this
option, diversity is still ensured.

More specifically, we used the following parameters for the different MineDojo
environments:

  * •

High range for fast resets of $5000$ blocks, $2000$ blocks for the combat
zombie and combat spider tasks.

  * •

Timelimit of $1200$ steps for all environments, i.e., $1$ minute real-time at
$20$ steps per second.

  * •

RGB observation space with size $640\times 360$.

Figure 11: Some examples of cave classification. A cave is spotted when an
agent stands in front of it (third row, first image) and in generally dark
frames (second row, second image; third row, third image).

###  B.2 Cave classifier

##### Dataset creation

The dataset for the cave classifier has been created from the original
FindCave dataset. Such dataset features 5466 demonstrations from experts
solving the task. For each of them, we use a combination of heuristics and
manual relabeling to ensure data coherence. First, it is worth noting that a
cave will appear only in the ending part of an episode. We estimate that a
cave frame will appear around the last 2 seconds of a successful episode (that
is, within the last 40 frames of an episode). Therefore, we empirically select
the $30^{th}$-to-last frame from each trajectory and extract it. This ensured
us to have around 5000 cave frames, but also some spurious frames coming from
unsuccessful episodes.

We removed a number of spurious frames by computing the average pixels value
in the 320 × 320 centered region of each image. We motivate this choice by
noting that a real cave frame would probably feature dark pixels at least in
its central region. Still, some specific situations (e.g. a particularly dark
forest) would resist to this method. We removed those frame by manually going
through the data.

To account for the non-cave class examples, we randomly extracted one frame
from the first half of each episode. As the FindCave dataset is composed by
trajectories from human experts, the presence of a cave in a frame is
ultimately decided by human judgement. Therefore, there is a chance of having
cave frames among the examples of the non-cave class. To address this problem,
similarly to the opposite class, we compute the average pixel values on the
central region of the image, and remove the false negatives manually.

Finally, we added the spurious frames of each class to the correct one. We
removed only the ones that were showing exceptional situations e.g. open
inventory, or ending screen of an episode. In the end, we counted 5456 cave
and 5455 non-cave examples. Each example has been resized to match the input
dimension of 320 × 320 by central cropping.

##### Cave classifier architecture

The architecture of our cave classifier is made of four convolutional layers,
each with 64 channels, 3 × 3 kernels, and ReLU activation. The input size is a
320 × 320 RGB image. Then, we flatten the embedding and forward it to two
fully-connected layers of 1024 units each, with ReLU activations. At the end
of the network, we apply softmax activation layer to get a probability
distribution over the two output classes.

##### Training procedure

We trained our cave classifier using a 90/10 splitting for training and
validation data. We have trained the network for 10 epochs, until convergence.
We tested our classifier over a subset of the validation data and obtained an
accuracy of 97.89%. Figure 11 shows some classification examples.

##  Appendix C Appendix - Evaluation details

Evaluating a model intended to work "like a human" is hard, as to this day no
comprehensive metrics for assessing human-likeliness is known. As an attempt
to encapsulate this complex set of behaviors in a number, BASALT organizers
use the TrueSkill score, as explained in Section 5.4. According to this
metric, our agent has indeed shown human-likeliness to some extent.

Nonetheless, assessing the human-likeliness of an agent does not answer
questions about its capabilities and limitations. In this regard, TrueSkill is
indeed limited: first, the score aims to establish a global ranking among $n$
agents competing against each other. Second, in the scope of BASALT such
ranking is based on a number of pairwise human evaluations, which is time
consuming to obtain. To gather an in-depth view of our agent capabilities, we
require a reliable and automatically computed metric.

Therefore, success rate represents the only objective way of evaluating the
agents of this study. Differently from the official BASALT competition, in the
FindCave context we define an episode to be successful whenever the agent
enters a cave and stays there for more than five seconds. We base our decision
on two main facts:

  * •

By definition of the task, a successful agent whose scope is to reach a cave,
will stay there indefinitely once it enters one (that is, if it works as
intended);

  * •

Any agent that uses some past reference, will need to demonstrate that the
action of entering the cave was willingful.

We argue that, in addressing success rate, this method is as robust as a
terminal action: on one hand, the first point accounts for the fact that any
agent fulfilling its scope has, at least to some degree, worked correctly; on
the other hand, to avoid validating agents that accidentally enter a cave, we
need to introduce a time threshold that clarifies an agent’s intent.

Additionally, we highlight that examples featuring such terminal action would
represent at most one action per episode. That is, terminal actions are a
minimal fraction of the total number of actions in the dataset. Large models,
such as VPT, fine-tuned on a relatively tiny amount of data (such as 10-100
trajectories) might fail to learn it. On the contrary, S-BC could make use of
a single instance of terminal action, given the right circumstances.
Therefore, keeping a terminal action would make the comparison between agents
unfair. In introducing the five seconds threshold, our agent is the only one
affected by our evaluation criterion.

To address the success rate problem, we make use of the cave classifier
described in Appendix B2. In our evaluation, such classifier was not used as
the only way of determining whether an episode was successful. Being aware of
the potential catastrophic implications of a false positive (or false
negative) in this context, we have manually evaluated each episode. To this
end, the cave classifier proved to be a nice tool in easing the retrieval of
potentially interesting segments of videos.

##  Appendix D Appendix - Ablation study

S-BC heavily relies on the number of trajectories encoded in its search space:
each point in the latent space corresponds to a potential solution to the
current situation. Therefore, being able to control the shape and the density
of such space is of the utmost importance. We tested the capabilities and
limitations of our model over the number of encoded trajectories. In this
experiment, we let each agent play $3$ runs of $30$ episodes for each task of
the MineDojo suite and the FindCave task from BASALT. We assess each agent
following the evaluation procedure detailed in Section 4.

Figure 12: Results of the ablation study on selected MineDojo tasks. (A) Combat spider: an agent must fight and successfully kill a spider before the time limit; (B) Combat zombie: an agent must overcome a zombie unit; (C) Harvest one log: an agent must successfully gather one unit of wood from a nearby tree; (D) Harvest one wheat seed: an agent must harvest one unit of wheat seed from the ground. Figure 13: Average time of completion for ground truth tasks from the MineDojo suite (A) "Combat spider", (B) "Combat zombie", (C) "Harvest one log" and (D) "Harvest one wheat seed". # | Task | BC 1x | BC 2x | BC 3x | GAIL | S-BC (Ours)
---|---|---|---|---|---|---
$10$ | Find Ocean | $20.46\pm 1.16$ | $20.28\pm 1.09$ | $19.74\pm 1.00$ | $\textbf{20.60}\pm 0.89$ | $20.31\pm 1.12$
| Dig Hole | $21.15\pm 0.91$ | $20.96\pm 0.84$ | $21.22\pm 0.87$ | $\textbf{21.58}\pm 1.10$ | $20.81\pm 0.90$
$25$ | Find Ocean | $20.31\pm 1.07$ | $20.26\pm 1.05$ | $20.12\pm 1.12$ | $\textbf{20.41}\pm 0.90$ | $\textbf{20.41}\pm 1.19$
| Dig Hole | $20.84\pm 0.83$ | $\textbf{20.95}\pm 0.92$ | $20.91\pm 0.79$ | $20.09\pm 1.02$ | $20.83\pm 0.94$
$50$ | Find Ocean | $\textbf{20.54}\pm 1.21$ | $20.24\pm 0.98$ | $20.05\pm 1.10$ | $20.51\pm 1.02$ | $20.33\pm 1.20$
| Dig Hole | $20.73\pm 0.84$ | $20.83\pm 0.82$ | $21.01\pm 0.85$ | $\textbf{21.44}\pm 0.80$ | $20.96\pm 0.86$
$100$ | Find Ocean | $20.25\pm 1.10$ | $20.19\pm 1.04$ | $19.82\pm 1.05$ | $20.30\pm 0.74$ | $\textbf{20.47}\pm 1.11$
| Dig Hole | $20.80\pm 0.88$ | $20.94\pm 0.85$ | $20.95\pm 0.87$ | $20.20\pm 0.24$ | $\textbf{20.97}\pm 0.92$
Table D: Results of the ablation study over available expert trajectories for
the evaluated creative tasks. Average MineCLIP score and its standard
deviation shown per model and available expert trajectories. MineCLIP scores
can be used to train agents in the MineDojo suite in an RL fashion.

##### MineDojo tasks

We trained each model described in main text using $10$, $25$, $50$ and $100$
trajectories. Results of the study are reported in Figure 12, Figure 13 and
Table D.

In Figure 12 we show that S-BC reaches better or similar performance to the
VPT-based models and GAIL in all the tasks, under almost all conditions. This
finding support our claim of zero-shot capabilities through mimicking of
experts’ trajectories, with no requirement for training.

Notably, Figure 12A and B show that our agent is the only one capable of
consistently solving combat-related tasks. In particular, we notice that S-BC
achieves very high success rate ($50-60\%$) on Combat zombie when using $10$,
$25$ or $50$ trajectories, while performance degrade on the $100$
trajectories. We hypothesize that such behavior might occur when the density
of information in latent space is too high with respect to the complexity of
the task. Nonetheless, we plan to conduct further studies on this aspect.

S-BC struggles in harvesting log (Figure 12C) when relying on a low number of
trajectories, while all the BC models reach around $30\%$ of success rate on
the task. It is interesting to notice that harvesting wood naturally matches
the aim of a VPT-based model. Therefore, we expected strong performances from
these models on this task. Surprisingly, S-BC compensates a significant
portion of this gap when using $100$ encoded trajectories. Given the dynamics
of the task, we hypothesize that having more trajectories improves our agent’s
navigation skill to a point where it become competitive with VPT-based BC
models.

Navigation-related problems are partially solved when harvesting a wheat seed
(Figure 12D), as these item are quite common on the ground of certain biomes,
and require therefore much weaker navigation skills. In this task, all the
agents perform similarly and seem to be independent from the number of
trajectories used.

It is interesting to notice how GAIL is completely unable to perform any of
the tasks. We hypothesize that the specific nature of the tasks and the
limited amount of data used was not representative enough to successfully
train it, despite being the method that required the highest amount of time to
be trained.

Evaluation on the creative tasks Find ocean and Dig a hole (Table D) revealed
that all the models achieve roughly the same performance. The loose
specification of the goal for those tasks poses a hard problem in describing
the absolute performance of an agent. Despite this, we highlight the highest
score for each situation. S-BC achieves the best performance in Find ocean
with 25 trajectories, and in both Find ocean and Dig a hole with 100 encoded
trajectories. Notably, GAIL yields similar performance and receives the
highest rewards in four occasions, namely Find ocean and Dig a hole with 10
trajectories, Find ocean with 25 trajectories and Dig a hole with 50
trajectories. Perhaps surprisingly, BC 3x appears to be consistently worse
than any other agent on the Find ocean task.

Figure 14: Ablation study over the number of trajectories in the FindCave
scenario. We encoded $10$, $25$, $50$ and $100$ trajectories for S-BC, and
used the same subset of trajectories to fine-tune several VPT foundation
models. Point in red refers to a different batch of 50 trajectories used for
S-BC.

##### FindCave task

Figure 14 shows how the number of encoded trajectories affect our agent
performance on the FindCave task. We have trained four versions of our model,
using a subset of $10$, $25$, $50$ and $100$ trajectories from the original
FindCave dataset. For comparison, we have fine-tuned a number of VPT-based
foundation models using the same subset of trajectories. In all cases, our
agent achieves baseline or better performance. More in detail, our agent
dramatically improves performance with as low as $10$ and $25$ encoded
trajectories, and reaches top performance with $100$ encoded trajectories,
where it reaches $90\%$ of success rate.

During our study, we noticed that the in the case of $50$ trajectories our
agent’s performance were degraded. We hypothesize that, since S-BC is heavily
reliant on the quality of latents, such drop would correspond to a low quality
batch of trajectories. Therefore, we re-trained our agent with a different
batch or trajectories and stated an increase in performance of $20\%$. While
this change did not result in similar performance to the other cases, it
supports the fact that our agent’s performance may significantly vary
according to the quality of data used to generate the latent space.


# High photon number entangled states and coherent state superposition from
the extreme-ultraviolet to the far infrared

Philipp Stammer<EMAIL_ADDRESS>ICFO – Institut de Ciencies
Fotoniques, The Barcelona Institute of Science and Technology, 08860
Castelldefels (Barcelona), Spain  Max Born Institute for Nonlinear Optics and
Short Pulse Spectroscopy, Max Born Strasse 2a, D-12489 Berlin, Germany  Javier
Rivera-Dean  ICFO – Institut de Ciencies Fotoniques, The Barcelona Institute
of Science and Technology, 08860 Castelldefels (Barcelona), Spain  Theocharis
Lamprou  Foundation for Research and Technology-Hellas, Institute of
Electronic Structure & Laser, GR-70013 Heraklion (Crete), Greece  Department
of Physics, University of Crete, P.O. Box 2208, GR-71003 Heraklion (Crete),
Greece  Emilio Pisanty  Max Born Institute for Nonlinear Optics and Short
Pulse Spectroscopy, Max Born Strasse 2a, D-12489 Berlin, Germany  Marcelo F.
Ciappina  Physics Program, Guangdong Technion–Israel Institute of Technology,
Shantou, Guangdong 515063, China  Technion – Israel Institute of Technology,
Haifa, 32000, Israel  Paraskevas Tzallas  Foundation for Research and
Technology-Hellas, Institute of Electronic Structure & Laser, GR-70013
Heraklion (Crete), Greece  ELI-ALPS, ELI-Hu Non-Profit Ltd., Dugonics tr 13,
H-6720 Szeged, Hungary  Maciej Lewenstein  ICFO – Institut de Ciencies
Fotoniques, The Barcelona Institute of Science and Technology, 08860
Castelldefels (Barcelona), Spain  ICREA, Pg. Lluís Companys 23, 08010
Barcelona, Spain

###### Abstract

We present a theoretical demonstration on the generation of entangled coherent
states and of coherent state superpositions, with photon numbers and
frequencies orders of magnitude higher than those provided by the current
technology. This is achieved by utilizing a quantum mechanical multimode
description of the single- and two-color intense laser field driven process of
high harmonic generation in atoms. It is found that all field modes involved
in the high harmonic generation process are entangled, and upon performing a
quantum operation, leads to the generation of high photon number optical cat
states spanning from the far infrared to the extreme-ultraviolet spectral
region. This provides direct insights into the quantum mechanical properties
of the optical field in intense laser matter interaction. Finally, these
states can be considered as a new resource for fundamental tests of quantum
theory, quantum information processing or sensing with non-classical states of
light.

The superposition of classically distinguishable states is of fundamental
interest since the development of quantum theory, and was brought to an
extreme by Schrödinger in his famous Gedankenexperiment [1]. In quantum optics
this notion can be retrieved by superpositions of coherent states [2, 3].
Beside their fundamental interest for testing quantum mechanics [4, 5], the
generation of these Schrödinger cat states, and of entangled coherent states
[6, 7], is also of direct technological importance. These states are a
powerful tool in quantum information processing [8, 9, 10, 11], quantum
computation [12, 13], quantum metrology [14], or can be used to visualize the
classical-to-quantum transition [15]. To generate superpositions of coherent
states, atom-light interaction in cavities [16, 17], or conditioning
approaches at the output of a beam-splitter [18, 19, 20] can be employed. Such
conditioning experiments are of general interest in quantum information theory
due to their ability for generating entangled optical states [21], to describe
quantum operations [22] or conditional quantum measurements [23, 24]. But, the
size of the generated superpositions of coherent states is limited to the
range of a few photons, corresponding to moderately small coherent state
amplitudes [3, 25, 26], restricting their applicability in quantum information
processing. However, due to the relevance of such non-classical states of
light in quantum technologies [8, 13, 27], it is of particular interest to
generate a superposition, and entanglement, of coherent states with high
photon numbers. In the present manuscript we show how both can be achieved by
means of a conditioning procedure, performed on a so far unrelated photonic
platform, namely intense laser-matter interaction. Laser sources can easily
reach intensities up to $10^{14}\operatorname{W}/\operatorname{cm}^{2}$, and
the field induced material response can be highly non-linear [28]. Since these
laser fields naturally involve very high photon numbers with corresponding
coherent state amplitudes in the range of $\absolutevalue{\alpha}=10^{6}$, it
will be of great advantage to use them for the generation of the sought high
photon number non-classical field states [29, 30]. Until recently the intense
laser-matter interaction was mainly described by a semi-classical theory, in
which the laser field was considered classically such that the properties of
the quantum state of the field were not envisioned. A commonly used intense
laser driven process is the generation of high-order harmonics, in which the
coherent properties of the driving laser are transferred to an electronic
wavepacket, and later returned to the field modes by the emission of coherent
radiation at frequencies of integer multiple of the driving laser field [31].
However, the recent advances in the quantum optical description of high
harmonic generation (HHG) [32, 33, 34, 35, 30, 36] allows to conceive new
experiments, in which non-classical properties of the field can be observed
with the prospective use for modern quantum technologies. In particular, it
was shown that a conditioning procedure on HHG can lead to non-classical
optical Schrödinger cat states in the infrared spectral range [30]. To extend
the approach to different spectral regions, and to unravel the entanglement
between all field modes participating in the HHG process, we have developed a
complete quantum mechanical multimode approach. This is used for the
description of the interaction of atoms with single- and two-color intense
laser fields. We show that all field modes involved in the HHG process are
naturally entangled, and upon performing a quantum operation leads to the
generation of high photon number coherent state superposition spanning from
the extreme-ultraviolet (XUV) to the far-infrared (IR). We provide the
conditions for the generation of XUV cat states, and for the generation of
entangled coherent states between two frequency modes in the IR regime with
very high photon numbers.

For the description we consider an uncorrelated state prior to the laser-
matter interaction $\ket{g}\otimes\ket{\phi}$, in which the atom is prepared
in its ground state $\ket{g}$ and the field is described by
$\ket{\phi}=\ket{\alpha}\otimes\ket{\\{0_{q}\\}}$, where
$\ket{\\{0_{q}\\}}=\bigotimes_{q}\ket{0_{q}}$. The intense driving laser in
the fundamental mode is in a coherent state $\ket{\alpha}$, and the harmonic
modes $q\in\\{2,\,...\,,N\\}$ are in the vacuum, where the generated harmonics
extend to a cutoff $N$. If the interaction is conditioned on the atomic ground
state (leading to HHG) [31, 30], and neglecting the correlations of the atomic
dipole moment [37], the effective interaction is described by a multimode
displacement operator [30, 38], $D(\chi)=\prod_{q=1}^{N}D(\chi_{q})$, where
$\chi_{q}=-i\kappa\sqrt{q}\expectationvalue{d}(q\omega)$, with coupling
constant $\kappa$, and the Fourier transform of the time-dependent dipole
moment expectation value
$\expectationvalue{d}(q\omega)=\int_{-\infty}^{\infty}dt\expectationvalue{d}(t)e^{iq\omega
t}$. Accordingly, the state of the field after the interaction is shifted
$\ket{\phi^{\prime}}=D(\chi)\ket{\phi}=\ket{\alpha+\chi_{1}}\otimes_{q=2}^{N}\ket{\chi_{q}}$.
The shift of the coherent state amplitude of the driving laser
$\chi_{1}=\delta\alpha$ accounts for the depletion of the fundamental mode due
to HHG, which are displaced by $\chi_{q}$. However, since the depletion of the
fundamental mode and the shift of the harmonic modes are correlated, the
actual mode which is excited due to the interaction with the atomic medium is
given by a wavepacket mode consisting of all field modes participating in the
process. The excitation of this wavepacket mode can be described by the
creation operator $B^{\dagger}$, with the corresponding number states
$\ket{\tilde{n}}$ satisfying
$B^{\dagger}B\ket{\tilde{n}}=\tilde{n}\ket{\tilde{n}}$. It is this wavepacket
mode which is excited during the HHG process [30]. In order to take into
account the correlation between the shift of the fundamental and harmonic
modes, we represent the total state $\ket{\phi^{\prime}}$ in terms of the
wavepacket mode $\ket{\tilde{n}}$. Considering only those cases where an
excitation of the wavepacket mode is present, but without discriminating
between the number of excitation, we project on $\sum_{\tilde{n}\neq
0}\outerproduct{\tilde{n}}{\tilde{n}}$, and obtain

$\displaystyle\ket{\psi}=\left[\mathds{1}-\outerproduct{\tilde{0}}{\tilde{0}}\right]\ket{\alpha+\delta\alpha}\otimes_{q=2}^{N}\ket{\chi_{q}}.$
(1)

Recalling that the vacuum state of this wavepacket mode is given by the
initial state before the interaction, i.e. $\ket{\tilde{0}}\equiv
D(\alpha)\ket{0}\otimes\ket{\\{0_{q}\\}}$, the total state of the field after
the HHG process is given by (up to normalization)

$\displaystyle\ket{\psi}=$
$\displaystyle\ket{\alpha+\delta\alpha}\otimes_{q=2}^{N}\ket{\chi_{q}}$ (2)
$\displaystyle-\bra{\alpha}\ket{\alpha+\delta\alpha}\ket{\alpha}\otimes_{q=2}^{N}\bra{0_{q}}\ket{\chi_{q}}\ket{0_{q}}.$

It shows that in the process of HHG all field modes, including the fundamental
and all harmonic modes, are naturally entangled. Thus, whenever harmonics are
generated the state of the total optical field is entangled. Note that the
field modes are entangled in such a way that measuring one mode can leave the
entanglement of the other modes intact, which suggest the ability of using HHG
for generating high dimensional optical cluster states [39] which are used for
measurement based quantum computation [40, 41].

Figure 1: Schematic illustration of the conditioning measurement performed in
HHG to generate coherent state superposition. The intense driving laser in the
fundamental mode described by the coherent state $\ket{\alpha}$ is interacting
with a HHG medium, and the generated optical field is in the wavepacket mode
corresponding to $\ket{\tilde{n}}$. This gives rise to an entangled state
between all field modes $\ket{\psi}$. Performing a conditioning measurement on
the harmonics modes, by projecting on
$\ket{\\{\chi_{q}\\}}=\bigotimes_{q}\ket{\chi_{q}}$, the fundamental mode is
found in the coherent state superposition $\ket{\Psi}$.

This entangled state, which is heralded by the generation of harmonic
radiation, shall now be used to generate non-classical coherent state
superposition from the far-IR to XUV spectral region. This is achieved by
using the scheme developed in [30] for the generation of optical cat states in
the IR regime. This scheme is illustrated in Fig. 1, and relies on a post-
selection procedure by performing a measurement on the harmonic modes without
photon number resolving detectors [30, 34]. Part of this measurement
constitutes a conditioning on HHG by separating it from other processes (like
ionization), i.e. taking only into account the wavepacket excitations via
$\ket{\tilde{n}}$. Thus, the fundamental mode conditioned on the harmonic
signal is given by projecting on the harmonic coherent states
$\ket{\Psi}=\bra{\\{\chi_{q}\\}}\ket{\psi}$ of amplitude $\chi_{q}$. The
fundamental mode, up to normalization, is then found to be in a superposition
of coherent states

$\displaystyle\ket{\Psi}=\ket{\alpha+\delta\alpha}-\bra{\alpha}\ket{\alpha+\delta\alpha}e^{-\Omega}\ket{\alpha},$
(3)

where $\Omega=\sum_{q>1}\absolutevalue{\chi_{q}}^{2}$. This state coincides
with the state recently reported and measured in [30], but with the proper
prefactor for the second term which takes into account all modes appearing in
the experiment. To understand the influence of the decoherence factor $\Omega$
we only consider energy conserving events during HHG, i.e.
$\absolutevalue{\delta\alpha}^{2}=\sum_{q=3}^{N}q\absolutevalue{\chi_{q}}^{2}$.
Assuming that the shift of the harmonics are equal, and using that in single-
color HHG only odd harmonics are generated, we find that
$\Omega=2\absolutevalue{\delta\alpha}^{2}(N-1)/(N^{2}+2N-3)$. We thus observe
that the influence of the harmonics scales as $\mathcal{O}(1/N)$ with the
harmonic cutoff. Due to the extension to large harmonic orders, it makes the
scheme intrinsically robust against this kind of decoherence (see
Supplementary Material (SM) for more details).

However, due to the complete multimode description of the HHG process
developed in this work, we can further generalize this scheme to generate non-
classical optical states in extreme wavelength regimes. In fact the process of
HHG allows to generate entanglement between different frequency modes of the
optical field ranging from the far-IR to the XUV regime. Depending on the
particular modes measured on (2), for instance measuring all modes except
$\tilde{q}\in\\{q_{i},\,q_{j}\\}$, we obtain the entangled state

$\displaystyle\ket{\Psi_{ij}}=\bigotimes_{\tilde{q}}\ket{\chi_{\tilde{q}}}-e^{-\Omega_{ij}}\bigotimes_{\tilde{q}}\bra{0_{\tilde{q}}}\ket{\chi_{\tilde{q}}}\ket{0_{\tilde{q}}},$
(4)

where $\Omega_{ij}=\sum_{q\neq\tilde{q}}\absolutevalue{\chi_{\tilde{q}}}^{2}$.
Note that for the driving laser mode we have $\ket{0_{1}}\equiv\ket{\alpha}$
and $\ket{\chi_{1}}\equiv\ket{\alpha+\delta\alpha}$. This scheme therefore
leads to entangled states between IR-IR, IR-XUV and XUV-XUV modes by choosing
$q_{i}$ and $q_{j}$ appropriately. For instance, by interchanging the role of
the fundamental with the harmonics, i.e. measuring the fundamental mode and
projecting (2) on the coherent state $\ket{\alpha+\delta\alpha}$, we obtain
the entangled state of all harmonic modes

$\displaystyle\ket{\Psi_{\Omega}}=\bigotimes_{q}\ket{\chi_{q}}-e^{-\absolutevalue{\delta\alpha}^{2}}\bigotimes_{q}e^{-\frac{1}{2}\absolutevalue{\chi_{q}}^{2}}\ket{0_{q}}.$
(5)

The fact that the remaining harmonic modes are still entangled, illustrates
the peculiar feature of the entangled state in (2) as an optical cluster state
with possible application in quantum information processing. If we further
measure the harmonic modes $q^{\prime}\neq q$, the state of the $q$-th
harmonic is given by

$\displaystyle\ket{\Psi_{q}}=\ket{\chi_{q}}-e^{-\gamma}\ket{0_{q}},$ (6)

where
$\gamma=\absolutevalue{\delta\alpha}^{2}+\Omega^{\prime}+\frac{1}{2}\absolutevalue{\chi_{q}}^{2}$
with $\Omega^{\prime}=\sum_{q^{\prime}\neq
q}\absolutevalue{\chi_{q^{\prime}}}^{2}$. The state $\ket{\Psi_{q}}$
represents a superposition of a coherent state with the vacuum in the XUV
regime. To characterize this state we compute the corresponding Wigner
function [42, 43]

$\displaystyle W_{q}(\beta)=$
$\displaystyle\frac{2N_{q}^{2}}{\pi}\left[e^{-2\absolutevalue{\beta-\chi_{q}}^{2}}+e^{-(\Omega+\Omega^{\prime})}e^{-2\absolutevalue{\delta\alpha}^{2}}e^{-2\absolutevalue{\beta}^{2}}\right.$
$\displaystyle\left.-e^{-\Omega}e^{-\absolutevalue{\delta\alpha}^{2}}e^{-2\absolutevalue{\beta}^{2}}\left(e^{2\beta\chi_{q}^{*}}+e^{2\beta^{*}\chi_{q}}\right)\right],$
(7)

with the normalization $N_{q}$ of (6). In Fig. 2 (a) we show the Wigner
function (High photon number entangled states and coherent state superposition
from the extreme-ultraviolet to the far infrared) for an XUV field of
wavelength $\lambda_{XUV}=72.7$ nm for the $11$-th harmonic of a driving laser
with frequency $\lambda_{IR}=800$ nm. For comparison, Fig. 2 (b) shows the
Wigner function of the IR field corresponding to (3). The non-classical
features of the XUV and IR coherent state superposition are clearly visible,
as both deviate from the Gaussian distribution of a coherent state and depict
negative values. Since the corresponding mean photon number for the Wigner
functions of Fig. 2 (a) is less than one, it makes this scheme an interesting
source for generating single XUV photons.

However, to obtain a genuine high photon number coherent state superposition
in the XUV regime, a second and independent HHG process can be added to the
proposed scheme. In the second HHG process harmonics are generated within the
same frequency mode with amplitude ${\chi}_{q}^{\prime}$. Formally, by
coherently adding the harmonic mode from both schemes, i.e. the low photon
number coherent state superposition (6) with the high photon number coherent
state $\ket{\chi_{q}^{\prime}}$, gives rise to the coherent state
superposition

$\displaystyle\ket{\Psi_{q}^{\prime}}=D(\chi_{q}^{\prime})\ket{\Psi_{q}}=\ket{\chi_{q}^{\prime}+\chi_{q}}-e^{i\phi^{\prime}}e^{-\gamma}\ket{\chi_{q}^{\prime}},$
(8)

where $\phi^{\prime}=\operatorname{Im}(\chi_{q}^{\prime}\chi_{q}^{*})$.
Regardless of the small average photon number in (6) with $\chi_{q}\ll 1$, the
spatiotemporal overlap with the large amplitude coherent state
$\ket{\chi_{q}^{\prime}}$ leads to the effective displacement operation in (8)
[44]. Thus, the coherent state superposition in the IR (3) and XUV (8)
describe large amplitude optical cat states. Taking into account the typical
photon numbers of the IR driving field and the conversion efficiency of the
HHG process [45], the IR and XUV cat state can be produced with photon numbers
in the range of $\expectationvalue{n_{IR}}\sim 10^{13}$ and
$\expectationvalue{n_{XUV}}\sim 10^{7}$ photons per pulse, respectively. Here
we want to emphasize that HHG is performed in vacuum such that the optical
field is subjected to negligible environmental decoherence effects during
propagation. In addition to the technological importance of generating
coherent state superpositions in the XUV and IR spectral range, they depict a
notable feature which has a direct consequence to fundamental test of quantum
theory. The opposite shift in the imaginary part of their Wigner function is a
result of the correlation in the shift of the coherent state amplitudes of the
field modes in the process of HHG. Such quantum correlations between field
modes can be used, via homodyne quadrature measurements, towards violating
Bell type inequalities [4, 5].

Figure 2: Wigner function of the coherent state superposition (a) of the
$q$-th harmonic Eq. (6) and (b) of the fundamental mode corresponding to Eq.
(3). The calculation has been performed using $\delta\alpha=-0.2$, such that
$\chi_{q}\approx 0.03$ for an harmonic cutoff $N=11$. The opposite shift in
imaginary part reflects the correlation between the field modes.

However, to generate genuine high photon number entangled coherent states in
the order of $\absolutevalue{\alpha}=10^{6}$ we generalize the HHG process by
using a two-color driving field. Such high harmonic generation experiments are
often performed for a $\omega-2\omega$ laser frequency configuration with
frequencies in the visible to far-infrared spectral region, with parallel or
orthogonal polarizations between the two driving lasers [46, 47, 48]. In this
case, the initial state of the two mode driving field is given by
$\ket{\alpha_{1}}\otimes\ket{\alpha_{2}}$, such that the total field after the
interaction with the HHG medium is given by
$\ket{\alpha_{1}+\delta\alpha_{1}}\otimes\ket{\alpha_{2}+\delta\alpha_{2}}\otimes\ket{\\{\bar{\chi}_{q}\\}}$,
where $\delta\alpha_{1}$ and $\delta\alpha_{2}$ are the depletion of the two
driving field modes, respectively. Following the procedure introduced above,
and after taking into account the correlations via the corresponding
wavepacket mode, the obtained state reads

$\displaystyle\ket{\Psi}=$
$\displaystyle\ket{\alpha_{1}+\delta\alpha_{1}}\otimes\ket{\alpha_{2}+\delta\alpha_{2}}\bigotimes_{q>2}\ket{\bar{\chi}_{q}}$
(9)
$\displaystyle-e^{-i\left(\varphi_{1}+\varphi_{2}\right)}e^{-\frac{1}{2}\Delta}\ket{\alpha_{1}}\otimes\ket{\alpha_{2}}\bigotimes_{q>2}e^{-\frac{1}{2}\absolutevalue{\bar{\chi}_{q}}^{2}}\ket{0_{q}},$

where $\varphi_{i}=\operatorname{Im}(\alpha_{i}\delta\alpha_{i}^{*})$ and
$\Delta=\absolutevalue{\delta\alpha_{1}}^{2}+\absolutevalue{\delta\alpha_{2}}^{2}$.
By conditioning on the harmonic signal, i.e. projecting on
$\ket{\\{\bar{\chi}_{q}\\}}$, we obtain

$\displaystyle\ket{ECS}=$
$\displaystyle\ket{\alpha_{1}+\delta\alpha_{1}}\otimes\ket{\alpha_{2}+\delta\alpha_{2}}$
(10)
$\displaystyle-e^{-i\left(\varphi_{1}+\varphi_{2}\right)}e^{-\frac{1}{2}\Delta}e^{-\bar{\Omega}}\ket{\alpha_{1}}\otimes\ket{\alpha_{2}},$

with $\bar{\Omega}=\sum_{q>2}\absolutevalue{\bar{\chi}_{q}}^{2}$. This scheme
can be utilized in the spectral range from the visible to the far-infrared
regime, which is within the telecom optical fiber wavelength regime useful for
long distance entanglement distribution for quantum information processing due
to minimized attenuation. Due to the high degree of coherent control in the
two-color HHG processes, the relative field amplitudes and phase of the
amplitude entangled state between the two physical frequency modes can be
tailored in a controllable way. For instance by independently varying the
driving field amplitudes $\alpha_{i}$ or the relative depletion
$\delta\alpha_{i}$ via the respective field polarization, e.g. linear or
circular orthogonal polarized fields.

In order to compare the single- and two-color HHG setup we shall quantify the
degree of entanglement between the field modes in each scheme. We will make
use of the degree of purity of the reduced density matrix of a subsystem.
Since the reduced density matrix of an entangled state is not pure
$\rho^{2}\neq\rho$, we use the linear entropy $S_{lin}=1-\Tr(\rho^{2})$ as a
quantitative measure of the involved entanglement between coherent states [49,
50]. Since $\Tr(\rho^{2})\leq 1$, where the equality only holds for pure
states, a non-vanishing linear entropy serves as a witness of entanglement in
the total system. For the single-color HHG experiment (2) we particularly
focus on two cases. First, on the entanglement between the fundamental driving
field with all harmonic modes, and second, on the entanglement of $n$ harmonic
modes with all remaining modes (including the fundamental). We thus compute
the reduced density matrices of the fundamental mode, and for the
$q\in\\{2,\,...,\,n+1\\}$ harmonics via
$\rho_{q=1}=\Tr_{q>1}(\outerproduct{\psi}{\psi})$, and
$\rho_{nq}=\Tr_{q^{\prime}\neq q}(\outerproduct{\psi}{\psi})$, respectively.
The corresponding linear entropy measures, for the fundamental mode
$S_{lin}^{1}$ (black, solid) and for $n$ harmonics $S_{lin}^{nq}$ (black
dashed and dotted), are shown in Fig. 3 as a function of
$\absolutevalue{\delta\alpha}$ (SM). The entanglement witness depends on the
depletion of the fundamental mode $\delta\alpha$, which is correlated to the
harmonic amplitude $\chi_{q}$. We observe that the entanglement between the
fundamental mode with the harmonics (solid) is larger than the entanglement
between $n$ harmonic modes with all other field modes (dashed, dotted), and
that the entanglement measure monotonically decreases for an increasing
depletion of the fundamental mode. For large $\delta\alpha$, all entanglement
decays since the amplitude of the second term in (2) vanishes due to the
decreasing overlap between the two coherent states. For the different
partitions of the $n$ harmonic modes we observe that for larger $n$ the
entanglement with the remaining modes is increased and almost negligible for
$n=1$. For the two-color HHG process in which an entangled pair of coherent
states of large amplitude can be generated (10), we quantify the involved
entanglement by tracing over the second $2\omega$ driving field mode in (10).
The corresponding linear entropy $S_{lin}^{\prime}$ is shown in Fig. 3 (red)
for different ratios of the depletion of the two modes
$r=\absolutevalue{\delta\alpha_{2}}^{2}/\absolutevalue{\delta\alpha_{1}}^{2}$
(SM). We observe that for small depletion the involved entanglement between
the two driving field modes in the two-color HHG experiment is larger than in
any single-color experiment, and is the largest for equal depletion (red,
solid). For a larger depletion $\absolutevalue{\delta\alpha}$ the entanglement
decays slower when the $2\omega$ field has smaller depletion than the $\omega$
mode, e.g. $r=0.5$ (red, dotted). In the opposite case, with $r=2.0$, the
entanglement is smallest (red, dashed).

Figure 3: Linear entropy measures $S_{lin}^{1}$ (black solid) and
$S_{lin}^{nq}$ for two different partitions of the entangled state (2) with
$n=1$(black dashed) and $n=10$ (black dotted) for increasing depletion of the
fundamental mode $\absolutevalue{\delta\alpha}$. The linear entropy measure
for the two-color high harmonic generation experiment $S_{lin}^{\prime}$ (red)
with different ratios of the depletion of the two driving fields
$r=\absolutevalue{\delta\alpha_{2}}^{2}/\absolutevalue{\delta\alpha_{1}}^{2}$.
In all cases we have used the harmonic cutoff at $N=11$.

In conclusion, we developed a quantum mechanical multimode approach for the
description of HHG driven by single- and two-color intense laser fields. We
showed that all field modes involved in the HHG process are naturally
entangled once harmonics are generated. Performing quantum operations on
particular field modes leads to the generation of high photon number coherent
state superpositions spanning from the XUV to the far IR spectral region. We
provided the conditions for the generation of XUV-IR correlated coherent state
superposition, and the generation of entangled states in the visible-IR
spectral region with controllable quantum features. The entangled states
generated by using HHG are deterministically generated whenever harmonic
radiation is emitted, and the coherent state superposition is heralded when
the conditioning measured is performed. Considering that similar HHG mechanism
underlie the majority of the intense-laser matter interactions [51, 35, 52],
we anticipate that the findings will set the stage for conceiving novel
experiments for the generation of a whole family of high photon number non-
classical entangled field states, challenging the quantum state
characterization schemes [53], advancing fundamental studies of quantum theory
and provide a new platform for optical quantum technologies [27]. Finally, we
note that the dynamics of the HHG process is intrinsically in the attosecond
time regime, which further stress the potential impact of the present work on
quantum information technologies towards a previously inaccessible time scale,
and can further be used for optical signaling and spectroscopy with non-
classical light states [54].

###### Acknowledgements.

ICFO group acknowledges support from ERC AdG NOQIA, from Agencia Estatal de
Investigaciín (the R$\&$D project CEX2019-000910-S, funded by MCIN/
AEI/10.13039/501100011033, Plan National FIDEUA PID2019-106901GB-I00, FPI,
QUANTERA MAQS PCI2019-111828-2, Proyectos de I+D+I “Retos Colaboración”
RTC2019-007196-7) from Fundació Cellex, Fundació Mir-Puig, and from
Generalitat de Catalunya through the CERCA program, AGAUR Grant No. 2017 SGR
134, QuantumCAT U16-011424, co-funded by ERDF Operational Program of Catalonia
2014-2020), EU Horizon 2020 FET-OPEN OPTOLogic (Grant No 899794), and the
National Science Centre, Poland (Symfonia Grant No. 2016/20/W/ST4/00314),
Marie Skłodowska-Curie grant STREDCH No 101029393, “La Caixa” Junior Leaders
fellowships (ID100010434), and EU Horizon 2020 under Marie Skłodowska-Curie
grant agreement No 847648 (LCF/BQ/PI19/11690013, LCF/BQ/PI20/11760031,
LCF/BQ/PR20/11770012).). FORTH group acknowledges LASERLABEUROPE
(H2020-EU.1.4.1.2 Grant ID 654148), FORTH Synergy Grant AgiIDA (Grand No.
00133), the EU’s H2020 framework programme for research and innovation under
the NFFA-Europe-Pilot project (Grant No. 101007417). J.R-D. acknowledges
support from the Secretaria d’Universitats i Recerca del Departament d’Empresa
i Coneixement de la Generalitat de Catalunya, as well as the European Social
Fund (L’FSE inverteix en el teu futur)–FEDER.

## References

  * [1] Erwin Schrödinger.  Die gegenwärtige situation in der quantenmechanik.  Naturwissenschaften, 23(49):823–828, 1935.
  * [2] CC Gerry and PL Knight.  Quantum superpositions and Schrödinger cat states in quantum optics.  American Journal of Physics, 65(10):964–974, 1997.
  * [3] Alexei Ourjoumtsev, Hyunseok Jeong, Rosa Tualle-Brouri, and Philippe Grangier.  Generation of optical ‘Schrödinger cats’ from photon number states.  Nature, 448(7155):784–786, 2007.
  * [4] Jérôme Wenger, Mohammad Hafezi, Frédéric Grosshans, Rosa Tualle-Brouri, and Philippe Grangier.  Maximal violation of bell inequalities using continuous-variable measurements.  Phys. Rev. A, 67(1):012105, 2003.
  * [5] Raul García-Patrón, Jaromır Fiurášek, Nicolas J Cerf, Jérôme Wenger, Rosa Tualle-Brouri, and Ph Grangier.  Proposal for a loophole-free bell test using homodyne detection.  Physical review letters, 93(13):130409, 2004.
  * [6] Barry C Sanders.  Entangled coherent states.  Phys. Rev. A, 45(9):6811, 1992.
  * [7] Xiaoguang Wang and Barry C Sanders.  Multipartite entangled coherent states.  Phys. Rev. A, 65(1):012303, 2001.
  * [8] Alexei Gilchrist, Kae Nemoto, William J Munro, Timothy C Ralph, Scott Glancy, Samuel L Braunstein, and Gerard J Milburn.  Schrödinger cats and their power for quantum information processing.  Journal of Optics B: Quantum and Semiclassical Optics, 6(8):S828, 2004.
  * [9] Alexei Ourjoumtsev, Rosa Tualle-Brouri, Julien Laurat, and Philippe Grangier.  Generating optical schrödinger kittens for quantum information processing.  Science, 312(5770):83–86, 2006.
  * [10] Brian Vlastakis, Gerhard Kirchmair, Zaki Leghtas, Simon E Nigg, Luigi Frunzio, Steven M Girvin, Mazyar Mirrahimi, Michel H Devoret, and Robert J Schoelkopf.  Deterministically encoding quantum information using 100-photon schrödinger cat states.  Science, 342(6158):607–610, 2013.
  * [11] Paul Jouguet, Sébastien Kunz-Jacques, Anthony Leverrier, Philippe Grangier, and Eleni Diamanti.  Experimental demonstration of long-distance continuous-variable quantum key distribution.  Nat. Phot., 7(5):378–381, 2013.
  * [12] Seth Lloyd and Samuel L Braunstein.  Quantum computation over continuous variables.  In Quantum information with continuous variables, pages 9–17. Springer, 1999.
  * [13] Timothy C Ralph, Alexei Gilchrist, Gerard J Milburn, William J Munro, and Scott Glancy.  Quantum computation with optical coherent states.  Phys. Rev. A, 68(4):042319, 2003.
  * [14] Jaewoo Joo, William J Munro, and Timothy P Spiller.  Quantum metrology with entangled coherent states.  Phys. Rev. Lett., 107(8):083601, 2011.
  * [15] Alessandro Zavatta, Silvia Viciani, and Marco Bellini.  Quantum-to-classical transition with single-photon-added coherent states of light.  Science, 306(5696):660–662, 2004.
  * [16] M Brune, Serge Haroche, JM Raimond, Luis Davidovich, and N Zagury.  Manipulation of photons in a cavity by dispersive atom-field coupling: Quantum-nondemolition measurements and generation of “schrödinger cat”states.  Phys. Rev. A, 45(7):5193, 1992.
  * [17] Bastian Hacker, Stephan Welte, Severin Daiss, Armin Shaukat, Stephan Ritter, Lin Li, and Gerhard Rempe.  Deterministic creation of entangled atom–light schrödinger-cat states.  Nat. Phot., 13(2):110–115, 2019.
  * [18] M Dakna, T Anhut, T Opatrnỳ, L Knöll, and D-G Welsch.  Generating schrödinger-cat-like states by means of conditional measurements on a beam splitter.  Phys. Rev. A, 55(4):3184, 1997.
  * [19] AP Lund, H Jeong, TC Ralph, and MS Kim.  Conditional production of superpositions of coherent states with inefficient photon detection.  Phys. Rev. A, 70(2):020101, 2004.
  * [20] Masahiro Takeoka and Masahide Sasaki.  Conditional generation of an arbitrary superposition of coherent states.  Phys. Rev. A, 75(6):064302, 2007.
  * [21] Jaromír Fiurášek.  Conditional generation of n-photon entangled states of light.  Phys. Rev. A, 65(5):053818, 2002.
  * [22] Michael A Nielsen and Isaac Chuang.  Quantum computation and quantum information, 2002.
  * [23] Yakir Aharonov, Peter G Bergmann, and Joel L Lebowitz.  Time symmetry in the quantum process of measurement.  Physical Review, 134(6B):B1410, 1964.
  * [24] Philipp Stammer.  State distinguishability under weak measurement and postselection: A unified system and device perspective.  Phys. Rev. A, 102(6):062206, 2020.
  * [25] Hiroki Takahashi, Kentaro Wakui, Shigenari Suzuki, Masahiro Takeoka, Kazuhiro Hayasaka, Akira Furusawa, and Masahide Sasaki.  Generation of large-amplitude coherent-state superposition via ancilla-assisted photon subtraction.  Phys. Rev. Lett., 101(23):233605, 2008.
  * [26] Evgeny V Mikheev, Alexander S Pugin, Dmitry A Kuts, Sergey A Podoshvedov, and Nguyen Ba An.  Efficient production of large-size optical schrödinger cat states.  Sci. Rep., 9(1):1–15, 2019.
  * [27] IA Walmsley.  Quantum optics: Science and technology in a new light.  Science, 348(6234):525–530, 2015.
  * [28] Thomas Brabec and Ferenc Krausz.  Intense few-cycle laser fields: Frontiers of nonlinear optics.  Rev. Mod. Phys., 72(2):545, 2000.
  * [29] Th Lamprou, I Liontos, NC Papadakis, and P Tzallas.  A perspective on high photon flux nonclassical light and applications in nonlinear optics.  High Power Laser Science and Engineering, 8, 2020.
  * [30] M Lewenstein, MF Ciappina, E Pisanty, J Rivera-Dean, P Stammer, Th Lamprou, and P Tzallas.  Generation of optical schrödinger cat states in intense laser–matter interactions.  Nature Physics, 17(10):1104–1108, 2021.
  * [31] Maciej Lewenstein, Ph Balcou, M Yu Ivanov, Anne L’huillier, and Paul B Corkum.  Theory of high-harmonic generation by low-frequency laser fields.  Phys. Rev. A, 49(3):2117, 1994.
  * [32] IK Kominis, G Kolliopoulos, D Charalambidis, and P Tzallas.  Quantum-optical nature of the recollision process in high-order-harmonic generation.  Phys. Rev. A, 89(6):063827, 2014.
  * [33] IA Gonoskov, N Tsatrafyllis, IK Kominis, and P Tzallas.  Quantum optical signatures in strong-field laser physics: Infrared photon counting in high-order-harmonic generation.  Sci. Rep., 6(1):1–9, 2016.
  * [34] N Tsatrafyllis, IK Kominis, IA Gonoskov, and P Tzallas.  High-order harmonics measured by the photon statistics of the infrared driving-field exiting the atomic medium.  Nat. Comm., 8(1):1–6, 2017.
  * [35] N Tsatrafyllis, S Kühn, M Dumergue, P Foldi, S Kahaly, E Cormier, IA Gonoskov, B Kiss, K Varju, S Varro, et al.  Quantum optical signatures in a strong laser pulse after interaction with semiconductors.  Phys. Rev. Lett., 122(19):193602, 2019.
  * [36] Alexey Gorlach, Ofer Neufeld, Nicholas Rivera, Oren Cohen, and Ido Kaminer.  The quantum-optical nature of high harmonic generation.  Nat. Comm., 11(1):1–11, 2020.
  * [37] Bala Sundaram and Peter W Milonni.  High-order harmonic generation: simplified model and relevance of single-atom theories to experiment.  Phys. Rev. A, 41(11):6571, 1990.
  * [38] Javier Rivera-Dean, Theocharis Lamprou, Emilio Pisanty, Philipp Stammer, Andrés F Ordóñez, Marcelo F Ciappina, Maciej Lewenstein, and Paraskevas Tzallas.  Quantum optics of strongly laser–driven atoms and generation of high photon number optical cat states.  arXiv preprint arXiv:2110.01032, 2021.
  * [39] Robert Raussendorf and Hans J Briegel.  A one-way quantum computer.  Phys. Rev. Lett., 86(22):5188, 2001.
  * [40] Daniel E Browne and Terry Rudolph.  Resource-efficient linear optical quantum computation.  Phys. Rev. Lett., 95(1):010501, 2005.
  * [41] Philip Walther, Kevin J Resch, Terry Rudolph, Emmanuel Schenck, Harald Weinfurter, Vlatko Vedral, Markus Aspelmeyer, and Anton Zeilinger.  Experimental one-way quantum computing.  Nature, 434(7030):169–176, 2005.
  * [42] Antoine Royer.  Measurement of quantum states and the wigner function.  Foundations of physics, 19(1):3–32, 1989.
  * [43] Javier Rivera-Dean, Philipp Stammer, Emilio Pisanty, Th Lamprou, Paraskevas Tzallas, Maciej Lewenstein, and Marcelo F Ciappina.  New schemes for creating large optical schrödinger cat states using strong laser fields.  Journal of Computational Electronics, 20(6):2111–2123, 2021.
  * [44] Matteo GA Paris.  Displacement operator by beam splitter.  Physics Letters A, 217(2-3):78–80, 1996.
  * [45] Stefanos Chatziathanasiou, Subhendu Kahaly, Emmanouil Skantzakis, Giuseppe Sansone, Rodrigo Lopez-Martens, Stefan Haessler, Katalin Varju, George D Tsakiris, Dimitris Charalambidis, and Paraskevas Tzallas.  Generation of attosecond light pulses from gas and solid state media.  In Photonics, volume 4, page 26. Multidisciplinary Digital Publishing Institute, 2017.
  * [46] I Jong Kim, Chul Min Kim, Hyung Taek Kim, Gae Hwang Lee, Yong Soo Lee, Ju Yun Park, David Jaeyun Cho, and Chang Hee Nam.  Highly efficient high-harmonic generation in an orthogonally polarized two-color laser field.  Phys. Rev. Lett., 94(24):243901, 2005.
  * [47] Johan Mauritsson, Per Johnsson, E Gustafsson, Anne L’Huillier, KJ Schafer, and MB Gaarde.  Attosecond pulse trains generated using two color laser fields.  Phys. Rev. Lett., 97(1):013001, 2006.
  * [48] Avner Fleischer, Ofer Kfir, Tzvi Diskin, Pavel Sidorenko, and Oren Cohen.  Spin angular momentum and tunable polarization in high-harmonic generation.  Nat. Phot., 8(7):543–549, 2014.
  * [49] GS Agarwal and Asoka Biswas.  Quantitative measures of entanglement in pair-coherent states.  Journal of Optics B: Quantum and Semiclassical Optics, 7(11):350, 2005.
  * [50] Kamal Berrada, Sayed Abdel-Khalek, Hichem Eleuch, and Yassine Hassouni.  Beam splitting and entanglement generation: excited coherent states.  Quantum information processing, 12(1):69–82, 2013.
  * [51] Edyta N Osika, Alexis Chacón, Lisa Ortmann, Noslen Suárez, Jose Antonio Pérez-Hernández, Bartłomiej Szafran, Marcelo F Ciappina, Fernando Sols, Alexandra S Landsman, and Maciej Lewenstein.  Wannier-bloch approach to localization in high-harmonics generation in solids.  Physical Review X, 7(2):021017, 2017.
  * [52] Theocharis Lamprou, Rodrigo Lopez-Martens, Stefan Haessler, Ioannis Liontos, Subhendu Kahaly, Javier Rivera-Dean, Philipp Stammer, Emilio Pisanty, Marcelo F Ciappina, Maciej Lewenstein, et al.  Quantum-optical spectrometry in relativistic laser–plasma interactions using the high-harmonic generation process: A proposal.  In Photonics, volume 8, page 192. Multidisciplinary Digital Publishing Institute, 2021.
  * [53] Silvio Fuchs, Johann J Abel, Jan Nathanael, Julius Reinhard, Felix Wiesner, Martin Wünsche, Slawomir Skruszewicz, Christian Rödel, Detlef Born, Heidemarie Schmidt, et al.  Photon counting of extreme ultraviolet high harmonics using a superconducting nanowire single-photon detector.  Applied Physics B, 128(2):1–8, 2022.
  * [54] Konstantin E Dorfman, Frank Schlawin, and Shaul Mukamel.  Nonlinear optical signals and spectroscopy with quantum light.  Rev. Mod. Phys., 88(4):045008, 2016.