# Framework for Learning and Control in the Classical and Quantum Domains

Seyed Shakib Vedaie  These authors contributed equally.
<EMAIL_ADDRESS>Institute for Quantum Science and Technology,
University of Calgary, Calgary, Alberta, Canada T2N 1N4  Archismita Dalal
These authors contributed equally<EMAIL_ADDRESS>Institute
for Quantum Science and Technology, University of Calgary, Calgary, Alberta,
Canada T2N 1N4  Eduardo J. Páez<EMAIL_ADDRESS>Institute for
Quantum Science and Technology, University of Calgary, Calgary, Alberta,
Canada T2N 1N4  Barry C. Sanders<EMAIL_ADDRESS>Institute for Quantum
Science and Technology, University of Calgary, Calgary, Alberta, Canada T2N
1N4

###### Abstract

Control and learning are key to technological advancement, both in the
classical and quantum domains, yet their interrelationship is insufficiently
clear in the literature, especially between classical and quantum definitions
of control and learning. We construct a framework that formally relates
learning and control, both classical and quantum, to each other, with this
formalism showing how learning can aid control. Furthermore, our framework
helps to identify interesting unsolved problems in the nexus of classical and
quantum control and learning and help in choosing tools to solve problems. As
a use case, we cast the well-studied problem of adaptive quantum-enhanced
interferometric-phase estimation as a supervised learning problem for devising
feasible control policies. Our unification of these fields relies on
diagrammatically representing the state of knowledge, which elegantly
summarizes existing knowledge and exposes knowledge gaps.

##  I Introduction

Closed-loop control aims to regulate a system of devices to produce a desired
output autonomously [1, 2], and machine learning (ML) is about improving a
procedure autonomously through experience [3]. ML can be subdivided into
reinforcement learning (RL) and semi-supervised learning (SSL), with
supervised learning (SL) and unsupervised learning (UL) the extreme cases [4].
Both control and learning are vital to advancing technology and are
interrelated: learning can assist with enhancing control (such as the field of
‘machine-learning control’) [5], and control tools could be used for improving
learning [6]. Adding to the richness of this field, both control and learning
can be cast in a classical (i.e., non-quantum) context [1, 7] and in a quantum
context [8, 9, 10, 11]. These fields are rapidly developing with concomitant
problems of inconsistent terminology, disparities between classical and
quantum versions, and currently less-than-clear relations between these four
topics: classical and quantum control and learning, i.e., four areas of
classical control ($\mathscr{C}$C), quantum control ($\mathscr{Q}$C),
classical learning ($\mathscr{C}$L) and quantum learning ($\mathscr{Q}$L). To
be succinct, we refer to these topics as ‘learning for control’, and our aim
is to construct a unified framework connecting these topics, which we call our
learning-for-control framework (LfC). To this end, we summarise state-of-the-
art in connecting these topics together and show the value of our LfC
framework by how it reveals knowledge gaps and by showing how applying SL to
quantum-enhanced metrology [12, 13] reveals a procedure for performing this
task autonomously.

We begin by summarizing state of the art, and we represent this state of the
art by constructing a knowledge graph [14] whose vertices are current topic
areas, and both directed and undirected edges show connections between topic
areas. Furthermore, knowledge graph edges are labelled by the actual
references. This knowledge graph is particularly useful to convey not just
state of the art but also to convey knowledge gaps.

Next, we unify $\mathscr{C}$C and $\mathscr{Q}$C. Although both of these
topics are focused on control, specifically closed-loop control, the
literature for $\mathscr{Q}$C differs, even from basic construction, from the
literature for $\mathscr{C}$C. $\mathscr{C}$C is founded on concepts such as a
controller, the controller’s policy for controlling a plant, a reference used
to achieve a particular output state, and feedback [1]. $\mathscr{Q}$C
typically focuses on techniques to optimise quantum systems by controlling
coefficients in a Hamiltonian or open-system evolution [8, 9]. We formulate a
new version of $\mathscr{Q}$C that essentially quantises $\mathscr{C}$C, which
is distinct from our earlier approach of defining $\mathscr{Q}$C independent
of $\mathscr{C}$C [15].

Building on this $\mathscr{C}$C-$\mathscr{Q}$C unification, we explore
unifying $\mathscr{C}$L and $\mathscr{Q}$L in the same way. Our work sets the
stage for this unification by revisiting foundational aspects of
$\mathscr{C}$L but considers whether the basic objects therein can be treated
mathematically. Specifically, we articulate desiderata for $\mathscr{Q}$L
based on extending $\mathscr{C}$L concepts and based on unification
commensurate with our $\mathscr{C}$C-$\mathscr{Q}$C analysis.

Our next step is to extend the idea of learning for control [16] to the
quantum domain. This extension involves allowing for quantum channels
connecting the controller, plant and learner. Furthermore, the no-cloning
principle of quantum mechanics [17] forbids the same feedback to be sent,
accurately and deterministically, to both learner and controller, which we
account for in our framework.

We base our approach on Fu’s framework for learning control systems [16]. We
explicitly separate teacher, user, learner and controller in our framework, as
distinct from Fu’s approach, which equates controller and learner and does not
include ‘user’ in the picture (hence, not complete). In our LfC framework, the
teacher implements the process of learning for control for RL and SSL, which
differs from Fu’s approach, where the teacher is only present for the SL case.

We demonstrate the utility of our LfC framework in casting adaptive quantum-
enhanced interferometric-phase estimation (A$\mathscr{Q}$P) [12, 13] as an SL
problem. The benefit of introducing SL into this framework is that new
circumstances—such as modified working conditions—can be accommodated by using
the SL model to predict new possibilities rather than having to optimise for
each case, which is intrinsically harder. A$\mathscr{Q}$P is a form of
quantum-enhanced metrology, useful for enhancing quantum clocks [18] and
interferometric position-shift measurements [19, 20, 21] inter alia, but with
feedback incorporated, yielding the advantage that only single-particle
measurements are required rather than joint measurements if adaptive feedback
methods are not used.

Beyond the practical application of using our framework for methodically
applying learning to control, our work is interesting on a philosophical
level. Does control theory make sense for a quantum controller, for example, a
controller with a quantum computer and with quantum information coming to or
leaving the controller? Could the controller prepare the plant in a
superposition of states or be entangled with the plant? Could the controller
have a superposition of policies? We do not have the answers to such
questions, but one value of our work is that such questions arise very clearly
from our framework.

Our paper continues in §II with a review of the key concepts in ML, control
theory, unification of classical and quantum mechanics, and learning for
A$\mathscr{Q}$P. We then elaborate on how we construct our LfC framework in
§III. Based on our literature survey and the key concepts established in §III,
we present our knowledge graph in §IV. Next, in §V, we describe the
application of our framework to A$\mathscr{Q}$P. Finally, we discuss our
results in §VI and conclude in §VII with a summary of our work and an outlook.
As we employ many abbreviations, we summarise these abbreviations and their
full expressions in Table 1 for the convenience of the reader.

Abbreviation | Description
---|---
ML | Machine learning
$\mathscr{C}$C | Classical control
$\mathscr{Q}$C | Quantum control
$\mathscr{C}$L | Classical learning
$\mathscr{Q}$L | Quantum learning
SL | Supervised learning
UL | Unsupervised learning
RL | Reinforcement learning
SSL | Semi-supervised learning
QML | Quantum machine learning
PAC | Probably approximately correct
POVM | Positive operator-valued measure
LfC | Learning-for-control
A$\mathscr{Q}$P | Adaptive quantum-enhanced
| interferometric-phase estimation
Table 1: Glossary

##  II Background

In this section, we give a review of the concepts of $\mathscr{C}$C and ML,
along with their quantum counterparts. We further provide a summary of the
concepts of learning for control, both in classical and quantum domains. The
essential background on the $\mathscr{Q}$C technique of A$\mathscr{Q}$EM is
also provided.

###  II.1 Machine learning

ML is proving to be highly valuable due to its capacity for making predictions
based on experience. We begin by discussing the framework for ML. Then we
briefly explain classical ML, which includes the widely used modes of SL and
UL as well as RL. Finally, we discuss relevant definitions and notions of
quantum ML (QML).

####  II.1.1 Framework for machine learning

Now we discuss the essentials of ML. First, we define ML and then
differentiate learning from optimisation. Finally, we explain the
classifications of an ML task.

We begin by establishing the concept of ML and discussing its extension to the
quantum case. We adopt and formalize Mitchell’s definition of ML.

######  Quotation 1 (Mitchell [3]).

A computer program is said to learn from experience $\mathscr{E}$ with respect
to some class of tasks $\mathscr{T}$ and performance measure $\mathscr{P}$, if
its performance at tasks in $\mathscr{T}$, as measured by $\mathscr{P}$,
improves with experience $\mathscr{E}$.

Russell and Norvig formalize the concept of a computer program in the context
of ML by introducing the notion of an agent.

######  Quotation 2 (Russell & Norvig [4]).

An agent is just something that acts …. Of course, all computer programs do
something, but computer agents are expected to do more: operate autonomously,
perceive their environment, persist over a prolonged time period, adapt to
change, and create and pursue goals.

Quotations 1 and 2 clearly highlight the essential components of an ML
implementation, namely learning agent $\mathscr{A}$, $\mathscr{E}$,
$\mathscr{T}$ and $\mathscr{P}$, which we regard as essential constructs for
QML as well.

A model is a mathematical representation of the data, specifically, a formula
or algorithm that labels the data in a probably approximately correct (PAC)
way. The goal of ML is to search for a model that optimises the performance of
a learning agent [3, 22]. Various notions of representation exist, so here we
give our definition, pertinent throughout this paper.

######  Definition 1.

A representation of a function $f(x)$ is $\tilde{f}_{d}(x)$, with $d$
specifying the size (e.g., number of bits) and $\|f-\tilde{f}_{d}\|$ the
distance between the function and its representation with respect to the norm
on function space.

######  Example 1.

A truncated Taylor expansion [23] is one example of a representation.

######  Example 2.

Another example of a representation of a function is a cumulant expansion [24]
with the sequence of cumulants denoted

$\bm{\varpi}:=\left(\kappa_{1},\kappa_{2},\kappa_{3},\ldots\right),$ (1)

for $\kappa_{1}$ the mean, $\kappa_{2}$ the variance and $\kappa_{3}$ the
skewness.

ML is classified based on the nature of $\mathscr{E}$. For example, we can
consider what we call the structure of $\mathscr{E}$—labelled or unlabelled
features as one pair of cases or state-and-reward as another case— as a way of
classifying; alternatively, we can consider all $\\{\mathscr{E}\\}$ being
provided up-front or is simultaneous with the ML process. ML can be
categorised into three major paradigms based on the structure of
$\mathscr{E}$, namely SL for $\mathscr{E}$ being labelled features, UL for
unlabelled features and RL for $\mathscr{E}$ being state-and-reward [25]. If
all $\\{\mathscr{E}\\}$ is provided up-front, ML is offline; ML is online if
$\mathscr{E}$ involves input data from the system in real time while learning.

Figure 1: Machine learning classification. a) An example of the supervised
learning paradigm showing a decision boundary that classifies input data. b)
An example of the unsupervised learning paradigm that represents the
clustering of input data. c) A typical workflow in the reinforcement learning
paradigm that demonstrates how learning proceeds via interactions between an
agent and its environment.

Every ML algorithm comprises three components—representation, evaluation and
optimisation [26]. The representation component concerns the hypothesis space
of the learner [3, 22]. The evaluation component concerns the objective
function for learning and the learning performance. Finally, the optimisation
component concerns the methods to search for a model in the hypothesis space
that maximizes the learning performance. Thus, optimization adds value to all
ML problems by enhancing ML, and optimization is not learning in itself.

####  II.1.2 Machine learning paradigms

Now we elaborate on three major paradigms of ML and discuss a typical ML
workflow. We begin by summarizing the ML pipeline, which entails four steps.
These four steps are preprocessing raw data into a training data set, followed
by the second step of training, followed by the validation step, and finally,
the testing step. We then discuss SL and UL briefly yet sufficiently to
establish their mutual differences and the comparative difference with RL,
which we elaborate on at the end.

We describe an ML pipeline as an iterative workflow with four steps typically,
namely

$\text{pre-
processing}\rightarrow\text{calibrating}\rightarrow\text{training}\rightarrow\text{testing}.$
(2)

The workflow begins with the data pre-processing step, where raw data is
manipulated to construct a data set $\mathscr{D}$ that is suitable for ML. The
pre-processing step involves several sub-tasks such as data cleansing, feature
extraction and feature selection [27]. The pre-processed data set is then
divided into the disjoint union of a calibrating and training set
$\mathscr{D}_{\text{model}}$ of size $\mathscr{M}$ and a testing set
$\mathscr{D}_{\text{test}}$ of size $\mathscr{M}^{\prime}$ as

$\mathscr{D}=\mathscr{D}_{\text{model}}\sqcup\mathscr{D}_{\text{test}},$ (3)

such that

$\mathscr{M}+\mathscr{M}^{\prime}=\mathscr{N}=\left|\mathscr{D}\right|$ (4)

In the calibration step, the hyperparameters of the model are tuned in an
iterative way [28]. For each tuple of hyperparameters, a model is trained on a
randomly-sampled subset $\mathscr{D}_{\text{train}}$ of
$\mathscr{D}_{\text{model}}$, and the model’s performance is evaluated on the
remaining data set. A mean performance, corresponding to each tuple of
hyperparameters, is then calculated by repeating these two sub-steps for
different $\mathscr{D}_{\text{train}}$. After repeating this process of
calculating mean performance for all possible tuples of hyperparameters, the
calibration step returns the tuple corresponding to the best model
performance. In the training step, the model data set
$\mathscr{D}_{\text{model}}$, along with the hyperparameters returned from the
calibration step, are used to construct an ML model. Finally, the model, with
selected parameters and hyperparameters, is assessed on
$\mathscr{D}_{\text{test}}$. The model passes or fails at the test step; if
the model passes, this model is then used on real data.

In the SL paradigm [4], using Mitchell’s terminology [3], $\mathscr{E}$ is
provided to $\mathscr{A}$ as a labelled data set $\mathscr{D}$ of size
$\mathscr{N}$. For each labelled data being a tuple of an $F$-dimensional
feature vector $\bm{x}_{i}$ and its corresponding label $y_{i}$, we can
formally state the data set, according to a convenient formalism [29], as

$\mathscr{D}=\\{(\bm{x}_{i},y_{i})\mid
i\in[\mathscr{N}]:=\\{0,\ldots,\mathscr{N}-1\\}\\}\subset\mathds{R}^{F}\times\mathds{R}.$
(5)

Theoretically, we allow real-valued entries, but computationally, we
approximate by floating-point numbers up to machine precision. $\mathscr{A}$
then devises a labelling map

$f:\mathds{R}^{F}\to\mathds{R}:\bm{x}\mapsto\tilde{y},$ (6)

where $\bm{x}$ is an unseen feature vector and $\tilde{y}$ is its
corresponding predicted label. Labelling is not guaranteed to be correct every
time but it rather is probabilistically approximately correct [30, 31].
Denoting the set of all labels as $Y$, the fitness of $f$ is quantified by a
‘loss function’

$L:Y\times Y\mapsto\mathds{R}^{+},$ (7)

that measures the difference between $y$ and $\tilde{y}$. Common examples of
loss functions include the absolute and the squared losses [32]. SL tasks are
further sub-classified into two types: classification and regression [22]. A
classification task is defined for discrete labels, whereas regression is
pertinent for the case of continuous labels.

In the case of UL, the agent has access to unlabelled input data
$\\{\bm{x}_{i}\\}$ but not to labels $\\{y_{i}\\}$. The task of a UL agent is
to recognise hidden patterns in the data set and to use these patterns to
cluster these data together into subsets [22]. In practical applications, SL
and UL can be combined to form SSL [22]. SSL is especially useful for tasks
involving training data with few input-label pairs and thus a large number of
input-only data.

RL is quite different from SL and UL: instead of having labelled or unlabelled
input data, respectively, RL involves an agent, states of the environment,
actions and rewards. These notions are subtle and inconsistently defined so we
continue our practice of relying on verbatim quotes from respected sources to
define these entities. The environment plays a key role for RL, with the
environment described in the following quote.

######  Quotation 3 (Russell & Norvig [4]).

The environment could be everything—the entire universe! In practice it is
just that part of the universe whose state we care about when designing this
agent—the part that affects what the agent perceives and that is affected by
the agent’s actions.

In RL, the agent performs actions on the environment that are intended to
yield the best outcome, and the agent is rewarded accordingly for good
actions.

The agent acquires information from the environment and acts on this
information to change the environment. The nature of the agent is explained in
the following quotation.

######  Quotation 4 (Russell & Norvig [4]).

An agent is anything that can be viewed as perceiving its environment through
sensors and acting upon that environment through actuators.

To be clear on how actuators and sensors are defined, we provide the following
quotations.

######  Quotation 5 (Dorf & Bishop [1]).

An actuator is a device employed by the control system to alter or adjust the
environment.

######  Quotation 6 (Dorf & Bishop [1]).

A sensor is a device that provides a measurement of a desired external signal.

The agent is not just reacting to the environment to modify its state but
rather learns how to improve the environment as Sutton and Barto explain.

######  Quotation 7 (Sutton & Barto [25]).

Reinforcement learning is learning what to do—how to map situations to
actions—so as to maximise a numerical reward signal.

Russell and Norvig elaborate on the agent’s nature.

######  Quotation 8 (Russell & Norvig [4]).

[A]gents are expected to … operate autonomously, perceive their environment,
persist over a prolonged time period, adapt to change, and create and pursue
goals. A rational agent is one that acts so as to achieve the best …expected
outcome.

Sutton and Barto articulate the “prolonged time period” in Quotation 8 as the
agent “maximizing not the immediate reward, but cumulative reward in the long
run” [25]. The “reward signal” in Quotation 7 and the “best outcome” in
Quotation 8 rely on an agent whom we regard as hidden.

This reward, or “reward signal” [25], which formalises “the purpose or goal of
the agent” [25] and is passed “from the environment to the agent” [25], has an
origin that transcends the agent and the environment as two entities. The
reward is vitally important to “formaliz[ing] the idea of a goal” [25] and
seems to demand that a third party come into play. Sutton and Barto say:

######  Quotation 9 (Sutton & Barto [25]).

The reward signal is your way of communicating to the agent what you want
achieved.

What does “your way of communicating to the agent” mean? This statement
implies subjectivity by another agent, namely you. The reward thus represents
the “creator” of the problem who imbues subjective values on how actions
should be rewarded.

######  Quotation 10 (Sutton & Barto [25]).

Our focus is on reinforcement learning methods that learn while interacting
with the environment, which evolutionary methods do not do. …Evolutionary
methods ignore much of the useful structure of the reinforcement learning
problem: they do not use the fact that the policy they are searching for is a
function from states to actions; they do not notice which states an individual
passes through during its lifetime, or which actions it selects. …Although
evolution and learning share many features and naturally collaborate, we do
not consider evolutionary methods by themselves to be especially well suited
to reinforcement learning problems …

####  II.1.3 Online versus offline learning

ML paradigms can be employed in both online and offline settings. First, we
define the concepts of online and offline learning. Then, we treat the concept
of RL in both online and offline settings separately as RL inherently assumes
online interaction with an environment.

We begin by explaining offline ML, which is applied for the case that all
$\mathscr{E}$ is available before ML starts. Offline ML then searches for a
feasible model by learning the parameters of the model by using the entire
training data set for $\mathscr{E}$, which is fully available at the outset.

In contrast to offline ML, online ML is employed while the training set of
$\mathscr{E}$ becomes available; ideally, online ML acts immediately as each
element of $\mathscr{E}$ is received rather than storing elements [33]. For
example, in online SL, $\mathscr{A}$ has to predict the label of the next
instance based on the labels of the previous instances that $\mathscr{A}$ has
already seen. Online learning is helpful if training over the entire data set
is computationally infeasible or the algorithm must adapt to new patterns in
the data dynamically. The following quote conveys the distinction between
offline and online learning.

######  Quotation 11 (Ben-David, Kushilevitz & Mansour [33]).

The difference between the models is that, while in the on-line model only the
set of possible elements is known, in the off-line model the sequence of
elements (i.e., the identity of the elements as well as the order in which
they are to be presented) is known to the learner in advance.

Now we consider online vs. offline RL. Per the construction of RL, which
involves $\mathscr{A}$ interacting with the environment discussed in 3, RL is
implicitly online. However, real-time collection of experience via interaction
with an environment could be expensive or dangerous [34, 35]. To avoid such
problems, offline RL is an important topic. Offline RL accommodates agents who
utilize data from previously collected experience, which could have been
obtained from simulation or for a safe setting. In contrast to offline SL and
UL, which are well posed and widely used, reconciling RL, which is implicitly
online, with offline methods is a work in progress, as emphasised in the
following quote.

######  Quotation 12 (Levine, Kumar, Tucker & Fu [34]).

[O]ffline RL is, at its core, a counter-factual inference problem: given data
that resulted from a given set of decisions, infer the consequence of a
different set of decisions. Such problems are known to be exceptionally
challenging in machine learning, because they require us to step outside of
the commonly used i.i.d. framework, which assumes that test-time queries
involve the same distribution as the one that produced the training data.

####  II.1.4 Quantum machine learning

We review definitions of QML as stated explicitly in the literature. Then we
explain the concept of QML as gleaned from these somewhat disparate
definitions. Finally, we provide some examples of QML.

A few comprehensive review articles [11, 36, 37] and textbooks [38, 39]
provide various subjective definitions of QML. Unlike classical ML, a compact
and unanimous definition for QML in terms of the learning components
$\mathscr{A}$, $\mathscr{E}$, $\mathscr{P}$ and $\mathscr{T}$ (in Quotation 1)
is lacking. As essential background, we provide key quotations from some of
the most notable literature in this field. In an early review paper, a
functional definition of QML, without hinting at the prospect of “quantum
advantage” [40, 41, 42, 37], follows [43].

######  Quotation 13 (Schuld, Sinayskiy & Petruccione [43]).

In quantum machine learning, quantum algorithms are developed to solve typical
problems of machine learning using the efficiency of quantum computing. This
is usually done by adapting classical algorithms or their expensive
subroutines to run on a potential quantum computer.

More recently, QML definitions embed the concept of quantum advantage. QML is
defined in a well-cited survey paper as follows.

######  Quotation 14 (Biamonte et al. [11]).

The field of quantum machine learning explores how to devise and implement
quantum software that could enable machine learning that is faster than that
of classical computers.

In a recent perspective article on this field, Wiebe defines QML from a data-
science and computer-science perspective as follows.

######  Quotation 15 (Wiebe [44]).

Quantum ML involves using a quantum device to solve a machine learning task
with greater speed or accuracy than its classical analogue would allow.

Thus, we see that QML is defined in three ways: by its utility as in Quotation
1, by the aspiration of QML as in Quotation 14 and by the QML device as in
Quotation 15.

Notable examples of QML are varied. One example is a quantum support-vector
machine, where a quantum matrix-inversion algorithm solves the system of
equations that appear in a least-squares formulation of the support-vector
machine [45]. Another example of QML is the analogue-quantum kitchen sink,
which quantises the classical random kitchen sinks algorithm. The classical
random kitchen sink algorithm estimates a kernel by using randomised features
and can be enhanced by employing adiabatic quantum evolution to yield
requisite randomness [46]. A third example is quantum-enhanced RL, for which
quantum speedups in the agent’s decision-making process and learning times
have been claimed [47, 48].

###  II.2 Control System

In this subsection, we summarise studies of control. We begin with a summary
of $\mathscr{C}$C. Then we discuss the topic of $\mathscr{Q}$C. Finally, we
summarise the status of applying ML to control.

####  II.2.1 Classical control system

Now we present the essentials of $\mathscr{C}$C. First, we discuss the task of
a control system. Then we present a schematic for $\mathscr{C}$C that
represents the essential elements and their connections. Finally, we discuss
the nature of a control policy and how this policy is devised.

A control task involves steering specific controllable degrees of freedom of a
physical system such that its dynamics yields the desired observations within
a required tolerance. This task is achieved by a control system, which can be
defined in one of the following two ways, as examples.

######  Quotation 16 (Dorf & Bishop [1]).

A control system is an interconnection of components forming a system
configuration that will provide a desired system response.

An alternative definition is the following.

######  Quotation 17 (Rosolia et al. [2]).

A control system is a device in which sensed quantities are used to generate
an autonomous behaviour through computation and actuation.

Control systems are classified into closed-loop and open-loop control
according to whether control depends on feedback or not, respectively.

######  Quotation 18 (Dorf & Bishop [1]).

A closed-loop system uses a measurement of the output signal and a comparison
with the desired output to generate an error signal that is used by the
controller to adjust the actuator.

We represent a generic closed-loop control system as a block diagram in Fig.
2. Control systems consist of two main parts: the controlled object and the
controller. Our block diagram explains the essential components of closed-loop
control.

Figure 2: A block diagram of a closed-loop $\mathscr{C}$C system, where the
arrows between two blocks represent the flow of classical information. A plant
P, comprising the physical system, actuators and sensors, is controlled by a
controller C who sends a control signal u to P. The filled circle beside P
represents a “switching” operation that either fed back P’s output into the
control system as a signal y or passes it on as the ultimate output signal z,
which typically contains the termination signal and observable representing
the final state of P. The switching operation in $\mathscr{C}$C is implemented
as a fan-out. An evaluator E estimates an error signal d between y and the
reference signal r using an evaluation function $\varepsilon_{\text{r}}$. C
computes u based on a control policy $\varrho_{\text{r}}$, which is designed
based on r, and d. In a setting where y=$\emptyset$, one obtains an open-loop
control system.

######  Remark 1.

The sets of all actuator data, all possible plant output and all error signals
are

$U:=\\{u\\},\,Y:=\\{y\\},\,D:=\\{d\\},$ (8)

with each element $u$ being distinct from all other elements in $U$ and,
similarly, each $y$ being distinct from all other elements in $Y$. Moreover,
the sets $U$, $Y$ and $D$ have constant cardinalities $|U|$, $|Y|$ and $|D|$,
respectively, over successive feedback loops.

######  Remark 2.

In the classical case, each channel is labelled by a letter such as u (for
actuator data), written in upright (e.g., Roman) font, with its value in some
instance given by (slanted, or italic, font) $u\in U$, with (capital letter)
$U$ being the set of all possible (control) values. The probability vector for
the state of u is

$\bm{p}^{\text{u}}:=\left(p^{\text{u}}_{u}\right)$ (9)

with the right-hand side being the sequence of probability-vector components,
which adds to $1$.

######  Remark 3.

The quantum state of a channel, such as u, is given by a density operator
$\rho^{\text{u}}$ [17], which is a positive semi-definite trace-class operator
on Hilbert space, and a classical probability distribution is recovered by
diagonalising $\rho^{\text{u}}$ [49].

A policy is a set of instructions that determine the control parameters, and
hence the effectiveness of the control scheme. In closed-loop feedback, the
controller executes a policy

$\varrho:\mathbb{Z}\times D\to U:(r,d)\mapsto u.$ (10)

with $r$ representing the value of the reference channel r. Contrariwise, in
an open-loop control scheme, where y is either not available or is
unnecessary, C steers P to obtain z without ever receiving any feedback from
P.

Now we explain two standard methods to construct $\varrho$, namely
conventional model-based control (MBC) methods and alternative data-driven
control (DDC) methods.

######  Quotation 19 (Hou & Wang [50]).

Data-driven control includes all control theories and methods in which the
controller is designed by directly using on-line or off-line I/O data of the
controlled system or knowledge from the data processing but not any explicit
information from mathematical model of the controlled process, and whose
stability, convergence, and robustness can be guaranteed by rigorous
mathematical analysis under certain reasonable assumptions.

MBC proceeds by modelling the plant either using first principles or
identification from data [50] and assumes a trusted mathematical model
governing plant dynamics with bounded modelling uncertainty. The plant’s model
is represented by the map

$\varsigma:U\to Y:u\mapsto y$ (11)

with resources (e.g., fresh water, battery energy, oxygen) implicitly consumed
through this mapping as elements of $R$ are not typically stated in standard
references. Then $\varrho$ is devised based on $\varsigma$ with the belief
that the model is a trustworthy approximation of the true system. If
$\varsigma$ (11) is either unknown or not readily solved analytically, then
DDC methods can help. For DDC, $\varrho$ is exclusively devised based on
input-output data taken from P and thus lacks systematic design-and-analysis
tools.

####  II.2.2 Quantum control system

Now we discuss $\mathscr{Q}$C. First, we review disparate definitions of
$\mathscr{Q}$C as stated explicitly in the literature. Then, we explain the
concept of $\mathscr{Q}$C extracted from these somewhat disparate definitions.

In contrast to $\mathscr{C}$C, a concise definition of $\mathscr{Q}$C is
lacking. Therefore, we provide two quotations from notable literature on
$\mathscr{Q}$C.

######  Quotation 20 (Walmsley & Rabitz [51]).

Quantum control refers to active intervention in a system’s dynamics to
maximise the probability, based on a given metric, that the system evolves
toward a desired target state.

######  Quotation 21 (Lloyd [52]).

In the conventional picture of quantum feedback control, sensors perform
measurements on the system, a classical controller processes the results of
the measurements, and actuators supply semiclassical potentials to alter the
behaviour of the quantum system.

Although the two quotations appear to be different, these quotations are
compatible in that the probability in Quotation 20 is inferred from sampling
by sensors in Quotation 21 and the active intervention in Quotation 20 is
elaborated as classical control and altering semi-classical potentials in
Quotation 21. Neither of these quotations is sufficiently clear on what
components of a $\mathscr{Q}$C system are classical or quantum.

Now we explain the techniques for constructing $\varrho$ in $\mathscr{Q}$C,
which have traditionally taken an MBC approach. The standard practice in
$\mathscr{Q}$C is to construct a first-principle model $\varsigma$ that
describes the evolution of P over time, with $u$ manipulating one or more
coefficients of the Hamiltonian describing P [53, 54, 10]. Then $\varrho$ is
devised based on $\varsigma$ via gradient-based greedy algorithms. A popular
greedy optimization algorithm is GRadient-Ascent Pulse Engineering (GRAPE),
which was first proposed to obtain $\varrho$ for control tasks in NMR
spectroscopy [55]. Although the fitness landscape for $\varrho$ in
$\mathscr{Q}$C is often compatible with greedy algorithms, sometimes greedy
algorithms yield poor results, especially for constrained large-dimensional
quantum systems [56]. Therefore, MBC methods employing global-optimization
algorithms, such as evolutionary algorithms [57], have been developed to
obtain $\varrho$ for $\mathscr{Q}$C problems such as quantum-enhanced adaptive
phase estimation [12, 13] and quantum gate design [58, 59, 56]. For the cases
where $\varsigma$ is either unknown or not readily solved analytically [54],
ideas from DDC $\mathscr{C}$C have been used in $\mathscr{Q}$C with success
[60, 61, 62, 63, 64, 65]. However, the lack of a formalized structure prevents
the rapid progress of DDC $\mathscr{Q}$C techniques.

####  II.2.3 Learning for control

We now proceed to discuss literature applying ML techniques for both types of
control, i.e., classical and quantum. We begin by summarizing the concept of
machine-learning control (MLC). Then we summarize the state of the art for
classical ML applied to both $\mathscr{C}$C and $\mathscr{Q}$C. Finally, we
summarize the literature on QML for $\mathscr{C}$C.

MLC is the concept of using ML algorithms to learn an effective $\varrho$ for
a control system. MLC is motivated by control problems, where identifying and
modelling a plant is challenging or infeasible due to unobservability and
highly non-linear effects. MLC typically involves a “learning controller” who
employs learning techniques to execute a control task [16, 5]. Learning
enables a controller, who is neither omniscient nor possesses a feasible
alternative, to execute the task successfully by assisting in designing
feasible control policies. ML methods are highly flexible and adaptable but
lack systematic design-and-analysis tools for studying their stability and
robustness in the context of control [50].

######  Remark 4.

To cast a control problem into a learning problem, we identify the components
of a control task as the components of learning, namely, $\mathscr{A}$,
$\mathscr{T}$, $\mathscr{P}$ and $\mathscr{E}$. $\mathscr{A}$ devises
$\varrho$ for C such that the control task is achieved. In this scenario
$\mathscr{E}$ comprises plant feedback and control signal. The user chooses
$\mathscr{P}$, which is calculated based on r.

Classical ML has been widely applied for controlling classical operations. We
highlight three approaches to incorporating ML into $\mathscr{C}$C. (i) Using
ML to construct $\varsigma^{-1}$ without prior knowledge of $\varsigma$ (11)
[5, 50]. (ii) Using ML for system identification, i.e., learning $\varsigma$
based on training data. The obtained $\varsigma$ is then used to design
$\varrho$ (10) following MBC methods [66, 67]. (iii) Using ML for safe
control, where a learning agent uses data to safely improve performance by
learning the uncertain dynamics [68]. Fu’s seminal work [16] introduces a
framework for utilizing $\mathscr{C}$L for $\mathscr{C}$C, where an agent
called teacher (T) assess and directs learning of a classical C towards better
performance as shown in Fig. 3.

Figure 3: A block diagram representing the existing learning-for-control
framework [16]. Here C is a learning controller that devises $\varrho$ using
the information y from P such that the control goal r is satisfied. The
teacher T evaluates the performance of C and directs the learning performed by
C such the system’s overall performance is gradually improved. T may or may
not be involved in the learning loop depending upon whether the learning is
done following an SL or UL scenario [16].

Classical ML has been applied to $\mathscr{Q}$C as well. Classical ML and
heuristic optimization techniques are actively used for designing high-
performance quantum gates for fault-tolerant quantum computing [69].
Additionally, neural networks are used in some other $\mathscr{Q}$C tasks
including adaptive quantum tomography and dynamic decoupling [62]. Meta-
heuristic algorithms are also shown to be advantageous in designing
$\mathscr{Q}$C policies that are robust to random noise and decoherence [70].

We now consider the application of QML for control. QML for control has been
studied in the context of quantum RL [71]. RL is a valuable tool for solving
direct adaptive optimal control problems [72]. Therefore, developing quantum
variations of RL algorithms is an important topic. Recent works on the concept
of quantum RL, however, are limited to solving simple $\mathscr{C}$C problems
such as the problem of maze traversal being solved on a quantum annealer [73,
74, 75, 76].

###  II.3 Unifying classical and quantum mechanics

In this subsection, we explain our approach for combining classical and
quantum descriptions of any physical system, and this framework is important
to our overall aim of unifying classical and quantum control and learning. We
begin with a brief review of C∗-algebra, which provides a consistent framework
for observables independent of whether the system is classical or quantum.
Then we discuss the correspondence principle and, finally, describe the
representation of system states for classical and quantum mechanics.

We adopt an operational viewpoint, which is about states being descriptions of
how a system is prepared, and measurements being about describing detection
[77, 78]. Formally, this description can be made rigorous by treating states
as linear functionals on the appropriate vector space for the given physics
(classical or quantum with particular symmetries) and measurement as a
positive operator-valued measure (POVM) [79]. A duality exists between states
and measurements, and the basic mathematical description is achieved by
employing observables within a C∗ algebra $\mathcal{A}$ [80]. Each observable
in $\mathcal{A}$ is self-adjoint and is a vector-space homomorphism (“linear
operator”). The vector space for classical physics is probability space and,
for quantum mechanics, is Hilbert space or its extension to generalised
functions for infinite-dimensional space [81, 82]; here we maintain the
simpler language of referring to Hilbert space whether or not the space is
finite- or infinite-dimensional, with the technicalities being covered by
literature on the Gel’fand triple [83, 84, 85]. Evolution is described by
mappings known as channels.

We now explain classical and quantum channels based on Wilde’s two
descriptions [17], which we present here as definitions. The following
definition rephrases slightly Wilde’s description of a classical channel.

######  Definition 2 (Wilde [17]).

A classical channel is a conditional probability distribution from input
random variable $X$ to output random variable $Y$.

The next definition is verbatim from Wilde’s book [17].

######  Definition 3 (Wilde [17]).

A quantum channel is a linear, completely positive, trace-preserving map.

As our objective is to describe learning and control agnostically, i.e.,
without regard to whether the object is quantum or classical, we need to
combine Defs. 2 and 3 into a single agnostic definition of a channel:

######  Definition 4.

A channel is a linear, completely positive, trace-preserving map acting on an
inner-product space.

Classical-quantum correspondence is a method either for quantizing a classical
system or dequantise a quantum system and can be achieved in various ways [86,
87]. One way to achieve classical-quantum correspondence is via geometric
correspondence. Geometry in the classical case arises from the Poisson bracket
with states being represented on phase space, which is a Poisson manifold
[88]. Quantum states, on the other hand, can be represented on a Kähler
manifold with commutators imbuing a geometry that is analogous to the
classical case [89, 90, 91]. This geometric correspondence can be challenging
to establish in a mathematically rigorous way, but we do not require such
sophisticated methods here [87]. In practice, as observed by Dirac [92],
Poisson brackets of observables are simply replaced by commutators with
$\text{i}:=\sqrt{-1}$ introduced as a scalar coefficient. With these notions
in mind, we can understand that combining classical and quantum definitions of
channel into 4 is consistent with Wilde’s claim below.

######  Quotation 22 (Wilde [17] §4.6.4).

[C]lassical channels are special cases of quantum channels.

Thus, if this inner-product space is Hilbert space, then the channel is
quantum in concordance with Def. 3. If, on the other hand, the inner-product
space is restricted to being probability space (as a subspace of Hilbert space
[93] (ch 20)), then the channel is classical, matching Def. 2.

Subsequent to the channel the system is subject to measurement, which we now
explain mathematically. Rigorously, measurement corresponds to a POVM which we
now describe classically and quantumly.

######  Definition 5 (van Fraassen [94]).

[M]easurement is an operation that locates an item (already classified as in
the domain of a given theory) in a logical space (provided by the theory to
represent a range of possible states or characteristics of such items).

In the quantum case, every ideal measurement corresponds to an observable,
which is a self-adjoint operator in the C∗-algebra and, for finite-dimensional
Hilbert space, can be represented by a Hermitian matrix. More generally, for
blurry measurement, the POVM assigns some probability to each observable.
Classically, the observables act on a probability space, and one way to obtain
the classical POVM is to diagonalize the matrices representing observables
[49].

We represent states of classical systems as elements of $L^{1}(\mathbb{R})$,
which are square-integrable real-valued functions over real numbers and their
limits, such as Dirac $\delta$. Uncertainties are associated with the spread
of these distributions, such as standard deviation [95], entropic uncertainty
[96] or Cramér-Rao lower bound [97]. In quantum mechanics, uncertainty arises
from Born’s rule, for which the distribution of measurement outcomes arises by
associating this distribution with the squared modulus of the wavefunction,
itself being a representation of the state in $L^{2}(\mathbb{R})$ (complex-
valued normalizable functions over real numbers) [98].

###  II.4 Learning for adaptive quantum-enhanced interferometric-phase
estimation

In this subsection, we summarize work to date on ML methods applied to
A$\mathscr{Q}$P. First, we explain the topic of A$\mathscr{Q}$P. Then we
discuss A$\mathscr{Q}$P in the context of $\mathscr{Q}$C with particular
attention to policies, i.e., control procedures, and, finally, we elaborate on
learning methods for such policies.

####  II.4.1 Adaptive quantum-enhanced interferometric-phase estimation

We now introduce the concept of A$\mathscr{Q}$P and its mathematical
description. First, we focus on the interferometric transformation and the
nature of the input state. We then use this mathematical description to
quantify the imprecision in terms of the Holevo variance.

Figure 4: A block diagram representing the A$\mathscr{Q}$P scheme as a closed-
loop control (Fig. 2). The scheme utilizes a passive linear lossless two-
channel interferometer, such Mach-Zehnder, which has a pair of input and
output ports, one element of each pair labelled 0 and the other 1. The
interferometer arms are also labelled 0 and 1. Within the interferometer, arm
0 undergoes a known controllable phase shift of $\Phi$, and arm 1 undergoes an
unknown phase shift of $\varphi$. The quantum P incorporates the
interferometer and its accompanying sensors and actuators and is represented
by a purple block. The classical C is represented by an orange block, and the
double-lined orange arrows represent the flow of classical information. C is
in charge of sending control signal $u$ to update $\Phi$. The update in $\Phi$
is performed based on a pre-established $\varrho$ and P’s feedback $y$, which
consists of the photon measurement information bit $b$ and the current value
of $\Phi$. y goes to z as a fan-out, and after all $N$ photons have been
injected into the interferometer and measurements are completed, the final bit
string for all measurements maps to $\tilde{\varphi}$.

The aim of A$\mathscr{Q}$P is an empirical unbiased estimate $\tilde{\varphi}$
of the relative unknown phase $\varphi-\Phi$ of a channel
$\mathcal{I}(\varphi;\Phi)$ describing an interferometer, with one tunable
parameter $\Phi$ and one unknown parameter $\varphi$, typically each being the
phase shift of one arm of a two-arm interferometer. The imprecision of the
estimate $\tilde{\varphi}$, namely $\Delta\tilde{\varphi}_{N}$, is a function
of the quantum resource, in this case the number $N$ of particles injected
sequentially into the interferometer. Without quantum resources, the standard
quantum limit (SQL) is [99, 100, 101, 102]

$\Delta\tilde{\varphi}_{N}\sim\nicefrac{{1}}{{\sqrt{N}}}.$ (12)

The ideal A$\mathscr{Q}$P scheme involves a passive linear lossless two-
channel interferometer, such as Michelson, Sagnac or, without loss of
generality, Mach-Zehnder [103, 104]. The interferometer has a pair of input,
and a pair of output, ports, which one element of each pair labelled 0 and the
other 1, as depicted in Fig. 4. Within the interferometer, the arms are
labelled 0 and 1 as well. The labelling of output ports conforms to our
requirement that a particle injected into the 0 input port emerges from the 0
output port for $\varphi=0$. Within the interferometer arm 0 undergoes a known
controllable phase shift of $\Phi$ and arm 1 undergoes an unknown phase shift
of $\varphi$.

Maximal quantum enhancement depends on choosing an appropriate input state
[105, 106, 107], which we assume to be pure (ideal case), and is a sequence of
photons with the choice of input port a degree of freedom. Extracting the
maximum quantum advantage depends on choosing an appropriate multi-photon
input state, denoted by $\ket{\Psi}_{N}$. This state is a superposition of
basis states like $\ket{01\cdots 11}$, denoting the first photon in port 0,
the second in port 1, the second last in port 1 and the last in port 1, as an
example. Although we do not treat loss here, we restrict our consideration to
permutationally-symmetric entangled states (changing the ordering of arrival
times does not change the multi-photon state) as such states are resilient to
photon loss [108].

We now elaborate on the procedure for obtaining $\tilde{\varphi}$. The
probability distribution for the unknown phase $\varphi$ is denoted
$p_{N}(\varphi;\bm{\varpi})$, with $\bm{\varpi}$ labelling a family of
distributions, determined by the input state (with $\bm{\varpi}$ serving as a
label for families of input states). Although the measurement could be general
[109], we restrict measurements to be in the computational basis, i.e. single-
photon detector at each output port. The information about single-photon
detector clicks is in $x_{m}\in\\{0,1\\}$ (0 for detecting at output port 0
and 1 for detecting at output port 1), with this bit then sent to C. Then C
employs policy $\varrho$, which we restrict to a binary decision tree [110],
to decide from bit $x_{m}$ what the next value of $\Phi_{m}$ should be. After
all $N$ photons have been injected into the interferometer and measurements
are completed, the final bit string $\bm{x}_{N}$ for all measurements maps to
$\tilde{\varphi}$ according to a formula that is sensitive to the choice of
$\varrho$.

We now discuss the scaling of $\Delta\tilde{\varphi}_{N}$ (12) for
A$\mathscr{Q}$P. $\Delta\tilde{\varphi}$ is the standard deviation of
$\tilde{\varphi}$ if $\Delta\tilde{\varphi}\in\mathbb{R}$, but some other
natural quantity serving as the variance is needed given that
$\tilde{\varphi}$ is a periodic variable; see Fig. 5. First, we introduce the
Fourier transform

$\check{p}_{N}(\nu;\bm{\varpi}):=\oint\text{d}\tilde{\varphi}\,p_{N}\left(\tilde{\varphi};\bm{\varpi}\right)\text{e}^{\text{i}\nu\tilde{\varphi}},$
(13)

with $\check{}$ our notation for Fourier transform, and $\nu$ is the discrete
conjugate variable to phase $\varphi$. By setting $\nu\leftarrow 1$, this
Fourier transform (13) yields sharpness

$S_{N}(\bm{\varpi})=\left|\check{p}_{N}(1;\bm{\varpi})\right|^{2},$ (14)

from which Holevo variance [111, 112], which is an appropriate variance for a
periodic variable, emerges:

$\Delta\tilde{\varphi}_{N}^{2}=:V_{N}:=S_{N}^{-2}-1.$ (15)

Estimating $S_{N}$ is achieved by injecting $\ket{\Psi}_{N}$, photon-by-
photon, many times for sampling purposes. The quantum limit to imprecision
scaling with respect to $N$ is the so-called Heisenberg limit (HL) [113]

$\Delta\tilde{\varphi}_{N}\sim\nicefrac{{1}}{{N}},$ (16)

but this limit is typically not reached in specific operational schemes [114,
115].

Figure 5: An example of a distribution of the estimate $\tilde{\varphi}$, with
$\bm{\varpi}$ labelling the family of distributions, and $\varphi$ indicating
the value to be estimated. When dealing with distributions over a compact
domain, such as a circle, there can be ambiguity in defining covariance. The
natural way to address this issue is to embed the circle in a complex plane or
represent it by using a set of real numbers. The concept of ‘sharpness’ is
introduced to quantify the spread of distribution along the boundary of the
circle. The centre of mass, despite the mass being restricted to the boundary,
is located inside the circle. The more spread out a distribution is, the
further towards the origin the centre of mass goes. For example, for a uniform
distribution around the circle, the centre of mass would be at the centre,
which would be the most uncertain distribution. The complex number for
sharpness represents the coordinate of the centre of mass, with the Fourier
transform (13) yielding the moment-generating function for the distribution.
Therefore (14) tells us how far away the centre of mass is from the centre
without considering the direction.

####  II.4.2 Devising feasible control policies for A$\mathscr{Q}$P

First, we describe relevant elements of $\mathscr{Q}$C. Then we explain how
$\Phi$ is updated according to $\varrho$. Finally, we introduce the term
‘policy orbit’, which comprises policies for each $\ket{\Psi}_{N}$ with
$N\in\\{4,\dots,N_{\text{max}}\\}$. Finally, we explain the feasibility of
policy orbits.

We recast A$\mathscr{Q}$P as $\mathscr{Q}$C, which is expressed in Quotation
20. The $\mathscr{Q}$C problem in A$\mathscr{Q}$P falls under the category of
measurement-based feedback control, which leads us to describe the components
in A$\mathscr{Q}$P as elements of $\mathscr{Q}$C. P incorporates the
interferometer and its accompanying sensors and actuators. C is in charge of
sending control signal $u$ to update $\Phi$. The update in $\Phi$ is performed
based on a pre-established $\varrho$ and P’s feedback $y$, which consists of
the photon measurement information bit $b$ and the current value of $\Phi$.

######  Task 1 (A$\mathscr{Q}$P control task).

Given an interferometer with an unknown phase shift $\varphi$ in one arm and a
tunable phase shift $\Phi$ in the other arm, described by
$\mathcal{I}(\varphi;\Phi)$, a quantum state of a sequence of photons
$\ket{\psi}_{N}$ entering the interferometer, fast measurement of which port
the photon leaves and feedback loop to control $\Phi$, devise a $\rho$ for C
that beats the SQL (12) for estimating $\tilde{\varphi}$.

We now discuss how C adaptively adjusts $\Phi$ based on $y$ and $\varrho$.
Neglecting loss, the $m$th photon comes out from either of the output ports
with a probability that depends on $\Phi_{m}-\Phi_{m-1}$. We label this
outcome by $b_{m}\in\\{0,1\\}$, where ‘0’ refers to the photon exiting the
first port and ‘1’ to the photon exiting the second port. Given the policy
vector

$\varrho_{N,\bm{\varpi}}:=\bm{\Delta}_{N,\bm{\varpi}}=(\Delta_{1,\bm{\varpi}},\Delta_{2,\bm{\varpi}},\dots,\Delta_{N,\bm{\varpi}})\in\mathbb{T}^{N},$
(17)

for

$\mathbb{T}^{N}:=\mathrel{\scalebox{1.2}{{\hbox{\varprodb}}}}^{N}\mathbb{S},\,\mathbb{S}:=[0,2\pi)$
(18)

being the $N$-torus and the circle (coordinate being the angle from $0$ to
$2\pi$ (radians), respectively. The value of $\Phi_{m}$ is updated
sequentially as

$\Phi_{m}=\Phi_{m-1}-(-1)^{b_{m}}\Delta_{m},$ (19)

by starting with $\Phi_{m}=0$, for every round of measurement $m$. Once all
photons in the input $\ket{\Psi}_{N}$ are exhausted by the $M$th measurement,
allowing for the loss of photons such that $1\leq M\leq N$, the estimate of
$\varphi$ is given by $\tilde{\varphi}\equiv\Phi_{M}$. As each measurement
outcome $b_{m}$ is a binary value, the value of $\tilde{\varphi}$ obtained by
this scheme is also discrete.

We now define a policy orbit and its use in assessing scaling. For a maximum
number of photons $N_{\text{max}}$, the ‘policy orbit’ is

$\mathbb{\Delta}_{{N_{\text{max}}},\bm{\varpi}}:=\left\\{\bm{\Delta}_{N,\bm{\varpi}}\right\\}^{N_{\text{max}}}_{N=4}$
(20)

with the subscript

$(N_{\text{max}},\bm{\varpi})$ (21)

describing both the maximum allowed $N$ and the input state as a function of
$N$; see Fig. 6. A feasible policy orbit
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}}$ satisfies
$\Delta\tilde{\varphi}_{N}$ having approximate power-law scaling with respect
to $N$, with the scaling surpassing $\nicefrac{{1}}{{\sqrt{N}}}$. Numerically,
the feasibility condition can be tested from a log-log plot of $V_{N}$ vs $N$
[70]. Specifically, this plot should fit a straight line

$\log V_{N}=-2\wp\log N+\text{const},\;\wp>\nicefrac{{1}}{{2}},$ (22)

with a goodness-of-fit $\overline{R^{2}}$ exceeding some acceptable value,
e.g. 0.999 [116].

Figure 6: A pictorial representation of a policy orbit (20), with
$N_{\text{max}}=7$, for the A$\mathscr{Q}$P control task. Each policy vector
$\bm{\Delta}_{N,\bm{\varpi}}$ is represented by the coordinates on the
corresponding stack of $N$ circles, where the red dots mark the values for
each component $\Delta_{i,\bm{\varpi}}$ (17). Physically, each red dot in one
column represents incremental changes in the estimation of $\varphi$ with a
fixed number of photons (19).

####  II.4.3 Learning for A$\mathscr{Q}$P

Now we critically review existing literature on the topic of artificial
intelligence for A$\mathscr{Q}$P. We begin by summarizing how meta-heuristic
optimization and ML have been employed to solve the A$\mathscr{Q}$P task. Then
we present our criticism of literature casting A$\mathscr{Q}$P as ML based on
Mitchell’s definition of learning in Quotation 1 and Sutton’s comment on
evolutionary methods for solving RL problems in Quotation 10.

A common tool for solving the A$\mathscr{Q}$P control task, namely, Task 1, is
to employ meta-heuristic optimization [117]. As examples, particle swarm
optimization (PSO) [118, 12], differential evolution (DE) [13, 70, 119],
genetic algorithms [120] and Bayesian optimisation [107, 121] have proven to
be useful for Task 1. PSO was leveraged to obtain
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}}$ numerically for
Task 1 [118, 12]. This estimation using PSO was employed successfully in an
optics experiment [122]. Using DE for AQP was shown to have a run-time
advantage over PSO [13].

In principle, ML can be employed to solve Task 1 as well. However, claims to
use ML for Task 1 [118, 12, 120, 123, 13, 122, 124, 125, 126, 127, 128, 129,
130, 131] do not comply with Mitchell’s definition of ML stated in Quotation
1. Our criticism is restricted to applying ML to Task 1, not to more general
instances of using ML for phase estimation [132, 133, 134, 135, 136, 137, 138,
139]. The reason for our criticism is that each article claiming to employ ML
for Task 1 actually cast the problem in the context of optimisation.

Now we consider specifically RL for Task 1 [118, 12, 140, 119, 128, 129]. The
criteria for claiming RL is more stringent because both Quotation 1 and
Quotation 10 have to be considered. Specifically, evolutionary algorithms,
such as genetic algorithms, genetic programming and simulated annealing, do
not qualify as RL because these algorithms do not incorporate learning while
interacting with the environment during the lifetime of the RL agent.

##  III Framework for learning and control

Now that we have completed our survey of key background notions and state of
the art, we proceed to set out how we approach building a framework for
learning and control, whether the system is appropriately described
classically or quantumly. To begin with, we explain our approach for combining
notions of learning and control without assuming underlying classical or
quantum constructs. In other words, we describe how we construct a framework
for control whose language does not presuppose either classical or quantum
rules. We then explain how we bring learning into this framework for control.

###  III.1 Unifying $\mathscr{C}$C and $\mathscr{Q}$C

Now we explain our approach to unifying classical and quantum control by
formulating control in a way that is independent of whether the underlying
physics is classical or quantum. First, we introduce a definition of control
that is agnostic (literally, ‘not known’) with respect to a classical vs. a
quantum framework: we can incorporate solely classical or solely quantum
components (elements of the control system) or a hybrid version with both
classical and quantum components. Second, we explain how we construct a
mathematical underpinning for our agnostic definition of control. Finally, we
discuss the elements of our control system, including controller C, plant P
and communication channels.

In §II.2, we gave two authoritative definitions of a control system, as
depicted in Fig. 2. However, each of these definitions is vexing.

  * •

Quotation 16 defines a control system by how it is made and whether the task
is completed but lacks a precise explanation of what must be done well.

  * •

Quotation 17 is unclear regarding what “sensed quantities” are and why
autonomous behaviour is needed (why can’t a sentient being, like a traffic
constable, be incorporated into a control system?).

Instead, we define the control system in two ways. One way is the top-down
definition.

######  Definition 6 (Top-down definition of control system).

A control system steers system variables so that pertinent observables reach
specific targets.

Here, the control system is a black-box (the inside is unimportant, hence not
seen), but input, output and performance measure are clear. The alternative
bottom-up definition establishes the control system in terms of its key
constituents.

Before providing the bottom-up definition, we define essential components of
the control system agnostically. The controller is described by the policy
$\varrho$ (20), and the plant is described by a physical process (11), but
these two descriptions assume classical input and classical output so are not
yet agnostic. To be fully agnostic, we consider generalizations of policy and
plant dynamics that accept both classical and quantum input and yield both
classical and quantum output. Therefore, in concordance with Remark 1 we
redefine error signals $D$, actuator actions $U$ and plant-based sensor
outputs $Y$ as being spaces and do not make suppositions about them being
Hilbert spaces or probability spaces.

######  Definition 7 (Evaluator).

An evaluator E is a channel described by $r$-dependent policy
$\varepsilon_{r}:Y\to D$.

######  Definition 8 (Controller).

A controller C is a channel described by $r$-dependent policy
$\varrho_{r}:D\to U$.

This definition captures the essence of Eq. (10) but does not presuppose that
$D$, $U$ and $Y$ are classical, although we treat the reference
$r\in\mathbb{Z}$ and $\varrho$ as classical.

######  Remark 5.

Our definition of the controller as an agent does not include sensors or
actuators explicitly, in contrast to typical definitions; for example,
Quotation 4 demands that C “can be viewed as [using] sensors and … actuators”,
but our definition of the full control system in Def. 6 makes equivalent
having sensors and actuators at P or at C.

######  Remark 6.

We assume that C is able to read and act on all input data and that all
outputs from C are faithfully sent without loss to P.

We separate the agent executing the policy from sensors and actuators, which
we regard as more conveniently defined as part of the plant. The reason for
defining C this way becomes evident as we describe our whole system-of-system
approach to control and learning, but the short version is that explicit
communication lines between C, P, and eventually the teacher are described
better using Definition 8, which accepts sensor input and yields outputs sent
to actuators.

The controller controls a plant, which we now define.

######  Definition 9 (Controlled plant).

A controlled plant P performs a given task $\varsigma$, which maps resources
$R$ and controlled actuator input $U$ to output $Y$.

######  Remark 7.

This definition for P does not require P to be successful at the given task;
the C’s job is to steer the plant to the successful execution of the task.

We now discuss the control switch S. S is a channel that guides y to z if
$y=r$ and to y otherwise; i.e.,

$\textbf{if}~{}y=r~{}\textbf{then}~{}\text{y}\to\text{z}~{}\textbf{else}~{}\text{y}\to\text{y},$
(23)

with $\to$ referring to ‘guided’. In the classical case, the statement $y=r$
is randomly assigned true or false with the proportion of true vs. false
determined by a threshold value. In the quantum case, $y=r$ is assigned true
or false following the quantum fingerprinting technique with a one-sided error
[141]. In the following, we first describe a classical S. Then we agnostisise
(Our term for modifying the description to be equally applicable to quantum
and classical cases) the definition of S. Finally, we show how the quantum S
yields classical S in the classical limit; see Fig. 7.

Figure 7: A diagram showing the relationships between classical
($\mathscr{C}$), quantum ($\mathscr{Q}$) and agnostic ($\mathscr{A}$)
descriptions of a physical system. To define any component (e.g. switch) of
our framework, we begin from its $\mathscr{C}$ description and “agnosticise”
to construct an equivalent $\mathscr{A}$ description. By “quantising” this
$\mathscr{A}$ description, we then derive the $\mathscr{Q}$ definition, which
can be related back to the original $\mathscr{C}$ description via
“dequantising”.

######  Remark 8.

We follow quantum-pseudocode conventions, where quantum data and quantum
logical operations are distinguished by an underline [142]. For example,

$\underline{\textbf{if}}~{}\underline{c}~{}\underline{\textbf{then}}~{}\text{cnot}(\underline{b},\underline{a}),$
(24)

implements a Toffoli gate [142], where $\underline{a}$, $\underline{b}$ and
$\underline{c}$ are quantum registers and
$\textsc{cnot}(\underline{b},\underline{a})$ is the controlled-not operation,
controlled by the first argument. We note that the distinction between the
classical and quantum syntactic annotation (meaning the distinction between
whether the logic or data are classical or quantum) is primarily semantic and
the quantum pseudocode without annotation makes sense operationally as
emphasised in the following two quotes. However, in Def. 11 of the quantum S
we annotate data and logic by underlining for clarity.

######  Quotation 23 (Knill [142]).

In principle, one can write quantum pseudocode without using annotation. Note
that only registers declared as bit sequences can be used for quantum
operations. From an operational point of view it suffices to describe what
happens to a register which is currently in superposition when subjected to a
classical (non-reversible) operation.

######  Quotation 24 (Knill [142]).

The conventions used here require that a register symbol is always considered
either classical or quantum. Semantically, which is in effect depends on the
most recent operation applied to it. If it has been declared as quantum, or a
proper quantum operation has been applied, then no further classical
operations can be used until it is measured. The syntactic annotation helps
keep track of the semantics of a register in any given section of code.

We cast S in the language of information theory, specifically as a logically
reversible gate. Logical reversibility connects well the classical and quantum
descriptions with reversible logic recasting irreversible Boolean functions as
logically reversible permutations whose smallest logical element has three
inputs and three outputs [143].

We regard S as having three inputs, various conditional information-processing
steps, and three outputs; see Fig. 8(a). The three inputs are the control bit
c whose value is $c\in\\{0,1\\}$, the plant-based sensor output dit y and the
reference signal dit r. The three conditional information-processing steps are
(i) swap y and r (i.e., swap(y,r) iff $c=1$ (if $c$ then swap(y,r)), (ii)
discard r (which might have changed value), and (iii) if $c$ then y $\to$ z
else y $\to$ y. The three outputs are c (unchanged from input), y and z (the
control system output).

Classically, the first step of S can be achieved by a Fredkin gate [144] with
input c replaced by the random Bernoulli variable $G$, which is a random
process labelled by threshold value $q\in(0,1)$ and expressed as

$\begin{cases}0&\text{if unif[0,1]}\geq q,\\\ 1&\text{otherwise},\end{cases}$
(25)

with unif referring to a uniform distribution. The threshold value $q$ is the
upper bound for the distance between $y$ and $r$ such that those values are
deemed to be approximately equal; i.e., if $\operatorname{dis}(y,r)<q$ then
swap(y,r) Conceptually, $q$ should be an overlap such as the Bhattacharyya
coefficient between two probability distributions [145]. The Fredkin gate
follows and then r is discarded. The final step of S is realized by the
railway switch RS, if $G$ then y $\to$ z else y $\to$ y.

Figure 8: A block diagram showing the internal components of a control switch
S, which we have previously represented as a filled circle in Fig. 2. We
regard S as having three inputs, various conditional information-processing
steps, and three outputs. (a) A classical S has three classical inputs,
namely, a control bit c whose value is $c\in\\{0,1\\}$, plant-based sensor
output dit y and reference dit r, and three outputs. The operation G converts
c into a random Bernoulli variable. If this random variable is 1, r and y are
swapped, followed by the trashing of r. Finally, the railway switch (RS)
directs y to the output or the feedback based on the value of the random
variable. The three outputs are c (unchanged from input), y and z (the control
system output). (b) Quantum S is similar to classical S except that we convert
data and logic to quantum data and quantum logic, and the controlled-SWAP
operation is realized by the quantised Fredkin gate with conjugation with
Hadamard gates on the control qubit. Similar to the classical S, a quantum S
has three inputs c, r, and y, which are qubit, qudit and qudit, respectively.
The final step of quantum S is achieved by measuring the control qubit and
feeding the one-bit measurement outcome to RS, which directs the qudit y
accordingly.

Before agnosticising S, we generalise c to accept either classical or quantum
input and to yield classical or quantum output, if the input is quantum or
classical or the input is quantum, respectively. Therefore, following Remark
1, we regard the set of all control signals $C:=\\{c\\}$ as forming a Hilbert
space without explicitly considering whether or not the Hilbert space is
restricted to a probability space.

######  Definition 10 (Agnostic S).

A control switch S is a channel that executes Procedure 1 ($|$ means
‘either’).

1:

2:bit|qubit control $\triangleright$ control $c$

3:dit|qudit sensor $\triangleright$ plant-based sensor output $y$

4:dit|qudit external $\triangleright$ reference $r$

5:

6:bit control

7:dit|qudit sensor

8:dit|qudit external $\triangleright$ control system output $z$

9:procedure agnosticS(control,sensor,external)

10: if control then swap(sensor,external)

11: discard external $\triangleright$ partial trace quantumly

12: control $\leftarrow$ control

13: if control then external $\leftarrow$ sensor $\triangleright$ This is
control-dependent guiding

14: return control,sensor,external

15:end procedure

Procedure 1 Classical$|$quantum control switch

We now cast S as a quantum channel by quantising the agnostic version in Def.
10 and Procedure 1; see Fig. 8(b). Quantum S is similar to classical S
discussed above except that we convert data and logic to quantum data and
quantum logic, and the controlled-SWAP operation is realized by the quantised
Fredkin gate [146, 147] with conjugation with Hadamard gates on the control
qubit [141]. The final step of quantum S is achieved by measuring the control
qubit and feeding the one-bit measurement outcome to RS. RS then executes if
$c$ then y $\to$ z else y $\to$ y.

Formally, we quantise S in Def. 10 by modifying Procedure 1 through the use of
quantum syntactic annotation for quantum data and logical operations.

######  Definition 11 (Quantum S).

A quantum S is a quantum channel that executes Procedure 2.

1:

2:qubit control $\triangleright$ control $c$

3:qudit sensor $\triangleright$ plant-based sensor output $y$

4:qudit external $\triangleright$ reference $r$

5:

6:bit control

7:qudit sensor

8:qudit external $\triangleright$ control system output $z$

9:procedure quantumS(control,sensor,external)

10: if control then swap(sensor,external)

11: discard external $\triangleright$ partial trace

12: control $\leftarrow$ control $\triangleright$ Measure control qubit

13: if control then external $\leftarrow$ sensor$\triangleright$ This is
control-dependent guiding

14: return control,sensor,external

15:end procedure

Procedure 2 Quantum control switch

######  Remark 9.

A consequence of quantum S is that, after each control-loop cycle, y could
become entangled with r, which causes decoherence in y.

######  Remark 10.

We now explain how we dequantise quantum S. In quantum S, the Hadamard
conjugation of c transforms it from the z-basis to the x-basis and back and
then it is measured. The conversion to the x-basis yields a two-by-two pure
density matrix. Classically this conversion is not allowed because there is
only one basis in the classical setting. Therefore, to recover classical S we
diagonalise the pure density matrix as it would make it an equal mixture of
zeros and ones. This procedure is simulated by $G$ (25) in Fig. 8(a). Whereas
there are two Hadamard gates for conjugation in the quantum case, this effect
is simulated by $G$ in the classical S and the second Hadamard maps to line.
The measurement is not necessary either as c is classical.

Next, we define bottom-up control, as a contrast to top-down control in Def.
6. Our bottom-up definition is related to Quotation 16, wherein a control
system is described as interconnected components required to build a standard
control system.

######  Definition 12 (Bottom-up definition of control system).

A control system comprises a controller, a controlled plant, communication
lines between the controller and the plant, input of reference information for
establishing targets and output being a task-relevant description of the
plant’s final state.

######  Remark 11.

Two examples of task-relevant descriptions include a simple beep that conveys
the plant’s task is complete or, alternatively, sensor readings that convey
temperature and consumed energy plus a beep that says the task is completed.

######  Remark 12.

Both Defs. 6 and 12 are agnostic with respect to whether the controller,
communication channels and plant are each quantum or classical.

######  Definition 13 (Quantum control).

$\mathscr{Q}$C is $\mathscr{C}$C with at least any one of the following being
quantum: E, C, P or S.

######  Remark 13.

Our Def. 13 reduces to Quotations 20 and 21 subject to the restriction that
P’s dynamics are strictly quantum.

Now that we have agnostic definitions of the control system in Defs. 6 and 12,
we describe mathematically the control system and its components without being
explicit regarding whether particular components behave classically or
quantumly; see Fig. 9. To this end, we treat E, C, P and S each as a classical
or quantum channel, and these channels are themselves connected to each other
by trivial channels, which also can be classical or quantum. Together, the
controller and the plant characterizes the control system channel.

Figure 9: A block diagram representing our agnostic closed-loop control
system, where the single-line arrows represent the flow of both classical and
quantum information, and the double-line arrows represent the flow of only
classical information. We modify the $\mathscr{C}$C loop of Fig. 2 to add an
agnostic S and make E, C and P act as either quantum or classical.

The input to the control system is zeros (i.e., $y\equiv 0$, with $0$ denoting
a string of zeros), and the output is labelled z. The termination condition
for the control system is baked into the policy (i.e., is written into the
code for the policy). We augment the channels between C, P and S with an extra
bit that carries a termination signal from C to S. Upon receiving the
termination signal, S switches its output from y to z. C uses an internal
clock, whose time can be indicated by integers, to determine whether the
termination condition is met.

###  III.2 Unifying $\mathscr{C}$L and $\mathscr{Q}$L

We now explain our approach to unifying classical and quantum ML. First, we
extend the classical definition of ML in Quotation 1 to the quantum case.
Next, following the definitions of classical and quantum ML we formulate
learning in a way that is independent of whether the $\mathscr{A}$ or
$\mathscr{E}$ are classical or quantum.

We now extend Mitchell’s description of ML in Quotation 1 to the quantum case.
We define QML by whether $\mathscr{A}$ and $\mathscr{E}$ are quantum. Building
on the effective definition of a learning agent $\mathscr{A}$ in Quotation 4,
a quantum learning agent is defined by whether any components of the agent are
quantum. Quantising $\mathscr{E}$ involves a superposition of inputs, e.g,
superposition of features $\ket{\text{feature}}$ for UL, superposition of
labelled features $\ket{\text{feature}}\ket{\text{label}}$ for SL and
superposition of action-observation [71] product states
$\ket{\text{action}}\otimes\ket{\text{observation}}$ for RL. As a classical
agent does not benefit from the richness of quantum $\mathscr{E}$, we only
consider a quantum agent in the context of quantum $\mathscr{E}$. Now,
following the definition of classical ML in Quotation 1, we define QML as
following

######  Definition 14 (Quantum ML).

A quantum agent $\mathscr{A}$ is said to learn from quantum experience
$\mathscr{E}$, with respect to some class of tasks $\mathscr{T}$ and classical
performance measure $\mathscr{P}$, if its performance at tasks in
$\mathscr{T}$, as measured by $\mathscr{P}$, improves with quantum experience
$\mathscr{E}$.

We now introduce a definition of ML that is agnostic with respect to a
classical vs. a quantum framework. Following Defs. 1 (classical learning) and
14 (quantum learning), plus Def. 4 (agent), we now define agnostic ML.

######  Definition 15 (Agnostic ML).

An agent $\mathscr{A}$ is said to learn from experience $\mathscr{E}$, with
respect to some class of tasks $\mathscr{T}$ and classical performance measure
$\mathscr{P}$, if its performance at tasks in $\mathscr{T}$, as measured by
$\mathscr{P}$, improves with experience $\mathscr{E}$.

###  III.3 Introducing learner and teacher/user

Having established the unified control and learning schemes, we now introduce
the two remaining components of our LfC framework, namely the learner and
teacher/user. To this end, we discuss why we treat the learner as an agnostic
agent, whereas we treat the teacher/user as a classical agent. Additionally,
we explain the connection between learner and teacher/user from the
perspective of an ML pipeline.

In our framework, we introduce a learner or learning agent (L) whose purpose
is to devise policies for C to execute. We define the combination of L and C
as a “learning controller”. As compared to Fu’s definition of a learning
controller, our learning controller is not one agent but comprises two
separate agents. This two-agent model simplifies the ideas of quantum and
classical learning controllers. Depending on whether L and C are classical or
classical four possibilities arise, namely $\mathscr{C}$L–$\mathscr{C}$C,
$\mathscr{C}$L–$\mathscr{Q}$C, $\mathscr{Q}$L–$\mathscr{C}$C and
$\mathscr{Q}$L–$\mathscr{Q}$C. Except for $\mathscr{C}$L–$\mathscr{C}$C, we
denote the rest as quantum-learning controllers as they include at least one
quantum component. Nevertheless, our LfC framework needs to be independent of
the underlying physics of C and L, which leads us to describe L agnostically.
We represent the learner L, which executes the learning algorithm, in a purple
box to account for the fact that it can be either classical or quantum.

Figure 10: A block diagram representing our LfC framework. Double-lined arrows
represent the flow of classical information and single-lined arrows represent
the flow of both classical and quantum information. The teacher M and user U
are classical agents and hence all connections between them and the rest of
the blocks are represented by double lines. We define the combination of L and
C as a “learning controller”. In the training phase, M directs the learning
agent L to devise the control policy $\varrho_{r}$ based on z from P. M
evaluates the performance of C and directs the learning process such that the
control system’s overall performance is gradually improved. Additionally, M
also sets parameters $\zeta$ of P and provides r to S and policy
$\varepsilon_{r}$ to E. In the test and validation phases, U apply the learned
policy $\varrho_{r}$ to execute the control task.

Thus far, our framework requires the policy to appear out of thin air, whereas
a complete picture would involve a teacher, whom we denote by M for the Latin
term magister. M trains L to develop a policy that becomes useful for the user
U, who replaces M when the training and validation are complete. M is an agent
who implements the process of learning for control. Our description of M is
different from Fu’s notion that M is only there to supervise or train. Our
notion of M/U is motivated by considering the physical realization of the ML
pipeline (2).

###  III.4 Learning for control pipeline

In this subsection, we present our agnostic learning for control scheme in
Fig. 10, where we draw all the components, namely C, P, S, E, L, M/U, and
their interconnections, and explain how the scheme works. We begin by
describing the data preprocessing, followed by model calibration for our ML
for control scheme. Then we explain the training and testing of these models.
Finally, we explain the subtleties related to online vs. offline learning for
control.

We now describe the pre-processing step of the ML pipeline for control. In the
pre-processing step, the data set

$\mathscr{D}=\mathscr{D}_{\text{model}}\sqcup\mathscr{D}_{\text{test}}$ (26)

is formed by applying pre-processing operations, such as data cleansing,
feature extraction and feature selection, on the raw data set [27]. The raw
data set is either generated using a simulation of the control system or
created based on data collected after executing the control system without the
learning loop for some limited settings. The pre-processing step usually
involves domain expertise and subjectivity [148, 149], which we are not
analyzing here.

The next step of the ML pipeline is calibrating, which we now describe in
detail. In this step, a tuple of feasible hyperparameters of the model is
obtained by searching over the hyperparameter space. During this step, we
treat M and not U. The calibrating step involves two sub-steps, namely,
training and testing. In the training step, M obtains a tuple of
hyperparameters, a randomly-sampled subset

$\mathscr{D}_{\text{train}}\subset\mathscr{D}_{\text{model}}$ (27)

and the subset

$\mathscr{D}^{\prime}_{\text{test}}=\mathscr{D}_{\text{model}}-\mathscr{D}_{\text{train}}$
(28)

for training L. Next, M provides L with $\mathscr{D}_{\text{train}}$ and the
hyperparameters and instructs L to obtain a $\varrho$ that maximizes the
control system’s performance on $\mathscr{D}_{\text{train}}$. Then, in the
testing step, M provides L with $\mathscr{D}^{\prime}_{\text{test}}$ and
instructs L to evaluate the performance of obtained $\varrho$ on
$\mathscr{D}^{\prime}_{\text{test}}$. M and L collaborate to evaluate this
performance for each tuple of hyperparameters by repeating these two sub-steps
for different $\mathcal{D}_{\text{train}}$. After repeating this process of
evaluating performance for all possible tuples of hyperparameters, the
calibration step returns the tuple corresponding to the maximum $\varrho$
performance.

In the training step, M provides L with $\mathscr{D}_{\text{model}}$, along
with the hyperparameters returned from the calibration step. M, then instructs
L to obtain $\varrho$ on this subset. Finally, in the testing step, M
instructs L to provide C with the $\varrho$ obtained from the training step. M
proceeds with providing C with $\mathscr{D}_{\text{test}}$, which is unseen in
the calibrating and training steps, and instructs C to commence the control
loop. L remains inactive for the remainder of the time. The control system
executing $\varrho$ either passes or fails at the test step; if the control
system passes, it is then used in the real world, wherein M is replaced by U
[149].

Here we discuss the subtleties of training and testing a learning controller
in both online and offline settings. Online ML methods are employed when
$\mathscr{E}$ become available in sequential order and at each step of the
control process. In online learning, there is no distinction between the
training and testing stages. Therefore, the roles of M and U become identical.
In this setting, M/U instructs both L and C to interact with P to generate
$\mathscr{E}$ online. L then obtains a $\varrho$ that improves the control
system’s performance with increasing $\mathscr{E}$. On the other hand, offline
ML methods obtain a $\varrho$ for a given data set $\mathscr{D}$. Whereas, in
the online learning setting, $\varrho$ updates as more $\mathscr{E}$ comes in;
in contrast, offline learning $\varrho$ is fixed once the training and testing
steps are finished. In offline learning, C and P are inactive in the training
and testing steps. M and L collaborate through the training and testing steps
until L obtains a $\varrho$ that satisfies the control system’s performance
requirement. If no $\varrho$ satisfies the performance requirement, then the
control system fails.

##  IV Graph representation of state-of-the-art

In this section, we provide a graphical interpretation of the research done in
the field connecting learning and control. We explain how we identify peer-
reviewed literature on any pair of the four topics $\mathscr{Q}$L and
$\mathscr{C}$L or $\mathscr{Q}$C and $\mathscr{C}$C. We employ a square
$\square$ to represent the state of the art with vertices representing each of
the four topics and edges representing overlaps between pairs of topics. Then
we discuss how edges are labelled; these edge labels represent state of the
art.

###  IV.1 Aggregating candidates

We begin with explaining how we aggregate the candidates for deciding their
memberships as vertices or edges of our knowledge graph. Our collection of
candidates is achieved by first searching the literature for relevant peer-
reviewed articles. Then we sort these candidates into classes corresponding to
different vertex and edge types.

We aggregate relevant literature by employing Google Scholar using
$\mathscr{C}$C, $\mathscr{Q}$C, $\mathscr{C}$L and $\mathscr{Q}$L keywords and
their pairwise combinations as search prompts. The search prompts we obtain
either corresponds to the four vertices of the $\square$ or the edges
connecting these vertices. Furthermore, we include references from review
articles [150, 41, 151, 152, 153, 137, 154, 139, 155, 71, 156, 157, 158, 159,
160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 16, 51, 170, 131, 171, 172,
36, 173, 174, 50, 43, 10, 175, 37, 11, 66, 176, 177] to our list of
literature. We then filter out non-relevant articles based on the abstracts
and criteria explained in the following subsections Section IV.2 and Section
IV.3.

We then sort the literature into four bins corresponding to the vertices of
the $\square$. One work can be sorted into more than one bin. Next, we assign
each article to an edge or hyperedge based on its membership to vertices. If
the article is a member of two vertices, we assign it to either a directed or
an undirected edge, which represents using one topic to address another or the
unification of topics. If the article is a member of three or more vertices,
we assign it to a hyperedge. However, in our search, no candidate was a member
of more than two vertices. Therefore a graph is sufficient to represent our
knowledge graph.

###  IV.2 Vertices

We now explain the vertices of the $\square$ and their membership criteria.
Each vertex corresponds to one of the four topics: $\mathscr{C}$C,
$\mathscr{Q}$C, $\mathscr{C}$L and $\mathscr{Q}$L. We orient the $\square$
such that the two vertices at the top represent learning, and the two at the
bottom represent control. The left side represents the classical regime, and
the right side represents the quantum regime. We decide membership based on
the definitions we have provided for each of these four topics, which in some
cases are quotations from the literature and in other cases, they form
definitions that we have constructed.

We now proceed to explain our criteria for deciding whether a given article is
accepted as being a member of either or both of the two control vertices. We
use the two authoritative definitions of $\mathscr{C}$C in Quotations 16 and
17 to decide memberships for the $\mathscr{C}$C vertex. A candidate article is
accepted as a member of $\mathscr{C}$C if its implicit or explicit definition
of $\mathscr{C}$C matches any of the above two definitions. In the quantum
setting, we use our own definition of $\mathscr{Q}$C in Def. 13 to decide
membership for the $\mathscr{Q}$C vertex.

######  Remark 14.

Of course, this procedure for deciding membership is somewhat subjective, so
we give an example of one case of rejecting membership to clarify how this
procedure works. An example of rejecting membership is given by Lloyd [52] for
the reason that neither implicit nor explicit separation of E, S and M/U are
given. The control loop evolves into a coherent superposition of C and P
without a clear distinction between them.

We follow Quotation 1 and our extension to that definition, Def. 14, as the
basis for defining learning in both classical and quantum domains.
Specifically, we use Quotation 1 to decide the membership of the
$\mathscr{C}$L vertex and Def. 14 to decide the membership of the
$\mathscr{Q}$L vertex. Following our definition of $\mathscr{Q}$L, we deem an
article as being a member of $\mathscr{Q}$L by whether $\mathscr{A}$ and
$\mathscr{E}$ are quantum.

###  IV.3 Edges

We now discuss the types of edges used to connect vertices in the $\square$.
An edge represents articles that incorporate both topics represented by the
two vertices connected by the edge. A directed edge represents articles that
use one topic to address another, and an undirected edge represents literature
uniting the two topics. The absence of an edge represents a lack of literature
relating these two topics. We also describe a hyperedge for the cases where an
article covers three or more topics. Each edge is labelled by the list of its
members, i.e., references pertaining to that overlap.

Instances of articles in the literature are decided to be members of a vertex
or an edge. A directed edge membership is decided based on whether the
literature only shows how to use knowledge in the vertex at the tail end of
the edge for the topic represented by the vertex at the head of the edge. We
refer to the knowledge represented by a directed edge as the knowledge going
one way, i.e., from the ‘tail’ topic to the ‘head’ topic. An edge is directed
only if all the literature is only one-way.

We now proceed to explain our criteria for deciding undirected-edge
memberships. An undirected edge membership is decided based on whether the
literature is both a member of two vertices and the two directed edges that
connect them. The undirected edge thus shows that the aggregate knowledge in
the literature goes both ways, _not_ that all the literature goes both ways.
Undirected edges signify some advance towards unifying the two topics. We use
our top-down and bottom-up agnostic definitions of control, Defs. 6 and 12,
respectively, to decide membership for articles that signify some advance
towards unifying $\mathscr{C}$C and $\mathscr{Q}$C, and our agnostic
definition of ML, Def. 15, to decide membership for articles that signify some
advance towards unifying learning in classical and quantum domains.

Edge memberships are useful if the literature strictly connects only two
topics at a time. If an article were to be a member of three or more vertices,
we would use a hypergraph with hyperedges connecting two or more vertices
together [178]. As we have not found such an article, a graph is sufficient to
represent our knowledge graph. Our article unites all four topics, so would
require a hyperedge if it were to be included in the knowledge representation,
but we do not include our article in this set.

###  IV.4 Knowledge graph

We now represent key literature on unifying classical and quantum control and
learning as a knowledge graph [14]. We employ a square $\square$ to represent
our knowledge graph with vertices representing each of the $\mathscr{C}$C,
$\mathscr{Q}$C, $\mathscr{C}$L and $\mathscr{Q}$L topics and both directed and
undirected edges representing connections between topic areas. Our knowledge
graph is particularly useful to convey not just state of the art but also
knowledge gaps. We begin with describing the $\square$. Then we explain each
of the connections in our knowledge graph and identify the knowledge gaps in
the literature.

Figure 11: Knowledge graph representing key literature concerning the
connection between each of the $\mathscr{C}$C, $\mathscr{Q}$C, $\mathscr{C}$L
and $\mathscr{Q}$L topics. Vertices represent each of the four topics, and
both directed and undirected edges represent connections between topic areas.
The edges are labelled by the list of corresponding member articles. If an
article is a member of two vertices, we assign it to either a directed or an
undirected edge, which represents using one topic to address another or the
unification of topics. Edge memberships are useful if the literature strictly
connects only two topics at a time. If an article were to be a member of three
or more vertices, we would use a hypergraph with hyperedges connecting two or
more vertices together. As we have not found such an article, a graph is
sufficient to represent our knowledge graph. An example of a hyperedge is
represented by the shaded grey area that connects $\mathscr{C}$C,
$\mathscr{Q}$C and $\mathscr{Q}$L. Our knowledge graph is particularly useful
to convey not just state of the art but also knowledge gaps, which are
represented by the missing edges.

Our knowledge graph, shown in Fig. 11, has four vertices labelled
$\mathscr{C}$L for classical learning, $\mathscr{Q}$L for quantum learning,
$\mathscr{C}$C for classical control, and $\mathscr{Q}$C for quantum control.
We do not discuss the status of knowledge for each of these four topics;
rather we are interested only in works establishing connections between these
four topics. Our graph differs from usual graphs by having three kinds of
edges allowed between vertices simultaneously.

Edges represent connections between pairs of the four topics, with each topic
represented by a vertex. We employ composite edges corresponding to directed
and undirected edges. Each edge in the knowledge graph is either
unidirectional if one topic builds on the other or undirected if each topic
builds on the other. The essence of the unidirectional
$\mathscr{C}$L-$\mathscr{Q}$L edges concerns quantum algorithms that enhance
classical ML [11, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190,
191, 192, 193, 194, 47, 45, 46, 48, 195, 196, 197, 198, 199, 200, 29] for the
rightward direction and quantum-inspired algorithms for classical ML [201,
202] in the case of the leftward-pointing edge. The undirected
$\mathscr{C}$L-$\mathscr{Q}$L edge represents literature reporting an advance
towards unifying the two topics [203, 204, 205, 65]. The unidirectional
$\mathscr{C}$C-$\mathscr{Q}$C edge represents literature concerning
generalisation of $\mathscr{C}$C schemes to $\mathscr{Q}$C [10, 8, 9] and the
undirected $\mathscr{C}$C-$\mathscr{Q}$C edge represents literature working
toward a unifying framework for $\mathscr{C}$C and $\mathscr{Q}$C [15, 206].
The downward unidirectional edge from $\mathscr{C}$L to $\mathscr{C}$C
represents literature extending $\mathscr{C}C$ to allow for the controller to
learn [66, 50, 5, 16], and the upward unidirectional edge concerns literature
employing mathematical tools developed for $\mathscr{C}$C to solve problems in
$\mathscr{C}$L, such as optimal parameter tuning for training neural networks
[6, 207] and optimal control techniques for RL [208, 209, 210, 150, 25]. The
undirected $\mathscr{C}$L-$\mathscr{C}$C edge represents literature reporting
an advance towards unifying the two topics [72, 211, 67, 68]. The sole
$\mathscr{C}$L-$\mathscr{Q}$C edge is unidirectional, and represents articles
employing ML for solving problems in $\mathscr{Q}$C such as quantum-gate
design [54, 69, 212, 213, 214, 215, 129, 134, 136, 130, 122, 128, 63, 62, 216,
64, 61]. The sole $\mathscr{Q}$L-$\mathscr{C}$C edge is unidirectional and
represents articles employing QML for solving problems in $\mathscr{Q}$C. QML
for control has been studied in the context of quantum RL and is limited to
solving simple $\mathscr{C}$C problems [217, 74, 73, 75, 76]. The downward
unidirectional edge from $\mathscr{Q}$L to $\mathscr{Q}$C represents
literature employing mathematical tools developed for $\mathscr{Q}$L to solve
problems in $\mathscr{Q}$C [218], and the upward unidirectional edge concerns
literature employing mathematical tools developed for $\mathscr{Q}$C to solve
problems in $\mathscr{Q}$L, such as diagnosing barren plateus [219] and tuning
variational quantum algorithms [220, 221, 222].

Now we discuss the missing edges, evident in Fig. 11, which represent gaps in
the literature. The first notable gap is represented by the lack of a directed
$\mathscr{Q}$C-$\mathscr{C}$C edge. An example of filling this gap would be an
article employing mathematical tools developed for $\mathscr{Q}$C to enhance
$\mathscr{C}$C. The gaps represented by the lack of directed
$\mathscr{C}$C-$\mathscr{Q}$L and $\mathscr{Q}$C-$\mathscr{C}$L edges are
surprising because literature exists that employs mathematical tools developed
for $\mathscr{C}$C and $\mathscr{Q}$C to solve problems in $\mathscr{C}$L and
$\mathscr{Q}$L respectively. Examples of filling these gaps would be articles
employing $\mathscr{C}$C to enhance $\mathscr{Q}$L or presenting quantum-
enhanced control leading to superior RL. The undirected
$\mathscr{Q}$C-$\mathscr{Q}$L gap is interesting because literature exists
that reports an advance towards unifying $\mathscr{C}$C with deep learning and
RL [72, 211, 67, 68] and the extension to the quantum case follows naturally.

##  V A$\mathscr{Q}$P as an SL problem

In this section, we apply our unified LfC framework to describe
A$\mathscr{Q}$P as an SL problem. First, we motivate and present our idea of
elevating the original control task in A$\mathscr{Q}$P to a generalized
control problem, which is to obtain
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}}$ for an unknown
$p(\varphi;\bm{\varpi})$. Then we discuss the mapping of this control problem
to the corresponding learning task, i.e., learning
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}}$. Finally, we
elaborate on our choice for the appropriate learning algorithm.

###  V.1 Mapping A$\mathscr{Q}$P to learning

Here we elaborate on an example of an application of our LfC framework.
Specifically, we analyse the A$\mathscr{Q}$P learning problem. First, we
discuss how we elevate the original A$\mathscr{Q}$P control task to a
generalised control problem. Then we elaborate on our approach for mapping the
generalized A$\mathscr{Q}$P control problem to an ML problem. Finally, we
explain our approach for constructing a computationally feasible $\mathscr{D}$
for our formulation of the A$\mathscr{Q}$P ML problem.

We generalise the A$\mathscr{Q}$P control problem, as described in Section
II.4, and construct a new control task. For given $\bm{\varpi}$, the re-
defined task is to devise
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}}$, defined just
below Eq. (20), for any unknown $p(\varphi;\bm{\varpi})$. Previously, devising
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}}$ is achieved by
obtaining $\bm{\Delta}_{N,\bm{\varpi}}$ (17) for each $N$ through constrained
optimisation. The constrained optimisation problem, whose fitness landscape is
highly non-convex, is solved using heuristic global-optimization algorithms
such as PSO [118, 12] and DE [13, 70, 119] in congruence with the scaling
condition (22). Each of these heuristic global-optimisation algorithms are
computationally expensive, thus severely limiting achievable $N_{\text{max}}$.
To address this drawback of the existing optimization schemes, we propose an
ML algorithm to solve the generalized control task in A$\mathscr{Q}$P.

We now elaborate on our procedure to map the generalised control task of
A$\mathscr{Q}$P as an ML problem. To cast a control problem into a learning
problem, we identify the components of a control task as the components of
learning, namely, $\mathscr{A}$, $\mathscr{T}$, $\mathscr{P}$ and
$\mathscr{E}$, with the agents being L, C and M described in Fig. 10. For our
A$\mathscr{Q}$P learning problem, $\mathscr{T}$ is the control task described
in Section II.4.1. $\mathscr{A}$ comprises L and C, where L devises $\varrho$
for C such that $\mathscr{T}$ is achieved. M chooses $\mathscr{P}$ in
congruence with Eq. 22, which is described based on r [1]. Each datum in
$\mathscr{E}$ comprises $p(\varphi;\bm{\varpi})$, $N_{\text{max}}$ and
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}}$, for which we
seek an efficient representation expressed in the following definition based
on our notion of representation in Def. 1.

######  Definition 16.

A representation $\tilde{f}_{d}$ efficiently represents a function $f$ if the
amount of information (e.g., bits) increases no more than
polylog$(\nicefrac{{1}}{{\epsilon}})$ for $\epsilon$, the distance between the
function and the representation.

######  Remark 15.

We represent $p(\varphi;\bm{\varpi})$ by a truncated cumulant expansion (1)

$\bm{\varpi}_{\zeta}:=\left(\kappa_{1},\kappa_{2},\kappa_{3},\ldots,\kappa_{\zeta}\right)\in\mathds{R}^{\zeta},$
(29)

which is convenient if the representation is efficient. Each value of
$p(\varphi;\bm{\varpi}_{\zeta})$ is then computed on-the-fly as the value is
required, turning a large space requirement into a slightly longer
computation.

Now we elaborate on how to construct $\mathscr{D}$ (3) for the A$\mathscr{Q}$P
learning problem. $\mathscr{D}$ is constructed in the pre-processing step of
the ML pipeline (2). In the pre-processing step, M first constructs a set of
parameters
$\bm{\alpha}_{\zeta}:=\left\\{\bm{\varpi}_{1,\zeta},\ldots,\bm{\varpi}_{\mathscr{N},\zeta}\right\\}$,
with $\mathscr{N}$ explained in Eq. (4), and provides P with one element of
$\bm{\alpha}_{\zeta}$. Then M collaborates with C to devise
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{i,\zeta}}$ by
obtaining $\bm{\Delta}_{N,\bm{\varpi}_{i,\zeta}}$ for each $N$ through
optimisation. After repeating the process of devising
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{i,\zeta}}$ for all
elements of $\bm{\alpha}_{\zeta}$, $\mathscr{D}$ comprises the data

$\left(\left(N_{\text{max}},\bm{\varpi}_{i,\zeta}\right),\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{i,\zeta}}\right)\,\forall
i\in[\mathscr{N}]$ (30)

with $\left(N_{\text{max}},\bm{\varpi}_{i,\zeta}\right)$ described in Eq.
(21). Finally, the pre-processing step returns $\mathscr{D}$.

###  V.2 Formalizing the SL problem

We now formally cast the A$\mathscr{Q}$P control problem as an SL problem.
First, we describe how we classify the A$\mathscr{Q}$P learning problem as one
of the SL, UL or RL paradigms of ML, depicted in Fig. 1, based on the nature
of $\mathscr{E}$. Then we introduce the formal learning problem corresponding
to the A$\mathscr{Q}$P control problem. Finally, we describe the ML workflow
for A$\mathscr{Q}$P, which comprises the calibrating, training and testing
steps.

We now classify the A$\mathscr{Q}$P learning problem based on the nature of
$\mathscr{E}$. For our A$\mathscr{Q}$P learning problem, $\mathscr{E}$ can be
viewed as comprising the pair
$\left(N_{\text{max}},\bm{\varpi}_{\zeta}\right)$ in Eq. (21) and the
corresponding feasible choice
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{\zeta}}$ for the
set $\mathbb{\Delta}_{N_{\text{max}},\bm{\varpi}_{\zeta}}$ in Eq. (20).
Therefore, the data structure in $\mathscr{E}$ naturally fits the SL paradigm
of ML, where $\mathscr{A}$ devises a labelling map (6)

$f:\mathbb{Z}\times\mathds{R}^{\zeta}\to\mathrel{\scalebox{1.2}{{\hbox{\varprodb}}}}_{j=4}^{N_{\text{max}}}\mathbb{T}^{j}:\left(N_{\text{max}},\bm{\varpi}_{\zeta}\right)\mapsto\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{\zeta}}.$
(31)

The pair $\left(N_{\text{max}},\bm{\varpi}_{\zeta}\right)$ is typically known
as a feature vector in SL, and each
$\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{\zeta}}$
corresponds to a label.

We now formalise the A$\mathscr{Q}$P SL problem. For a SL problem, we are
given $\mathscr{D}$ (5), where each datum is a tuple
$(\bm{x}_{i},\bm{y}_{i})$, with

$\bm{x}_{i}:=\left(N_{\text{max}},\bm{\varpi}_{i,\zeta}\right),\,\bm{y}_{i}:=\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{i,\zeta}}.$
(32)

The A$\mathscr{Q}$P SL problem then involves devising a labelling map $f$ (31)
such that, for an unseen feature vector $\bm{x}_{i}$ (32), the estimated label
is

$\tilde{\bm{y}}_{i}=\tilde{\mathbb{\Delta}}_{N_{\text{max}},\bm{\varpi}_{i,\zeta}}\approx\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{i,\zeta}}\in\mathrel{\scalebox{1.2}{{\hbox{\varprodb}}}}_{j=4}^{N_{\text{max}}}\mathbb{T}^{j}.$
(33)

We can use a ‘distance’

$L_{N_{\text{max}},\bm{\varpi}_{\zeta}}:=\text{dist}(\tilde{\mathbb{\Delta}}_{N_{\text{max}},\bm{\varpi}_{\zeta}},\mathbb{\Delta}^{\text{feas}}_{N_{\text{max}},\bm{\varpi}_{\zeta}}),$
(34)

as the loss function to be used during training. One example of the distance
would be the square root of the sum of squares of the distances between the
label coordinates on each hypertorus (17).

We now describe the ML workflow for our A$\mathscr{Q}$P SL problem, which
comprises the calibrating, training and testing steps. In the calibrating
step, M and L collaborate to obtain a tuple of hyperparameters that correspond
to an $f$ (31) which minimises $L_{N_{\text{max}},\bm{\varpi}_{\zeta}}$ (34)
on a set of randomly-sampled $\mathscr{D}^{\prime}_{\text{test}}$ (28)
according to Section III.4. In the training step, M provides L with
$\mathscr{D}_{\text{model}}$ (26), along with the hyperparameters returned
from the calibration step. M, then instructs L to obtain $f$ on this subset.
Finally, in the testing step, M instructs L to provide C with $\varrho=f$
obtained from the training step. M proceeds with providing C with
$\mathscr{D}_{\text{test}}$ (26), which is unseen in the calibrating and
training steps, and instructs C to commence the control loop. L remains
inactive for the remainder of the time. The control system executing $f$
either passes or fails at the test step; if the control system passes, it is
then used in the real world, wherein M is replaced by U.

##  VI Discussion

In this section, we discuss our results. We begin by discussing our agnostic
definitions of control and learning in classical and quantum settings and then
interpret our LfC framework. Next, we analyse the knowledge graph, which we
have constructed based on existing literature in the fields of control and
learning in both classical and quantum domains. Of particular interest is the
identification of gaps, which we found surprising as our approach identifies
these gaps and makes it clear that they need further study. Finally, we
explain the relevance of casting A$\mathscr{Q}$P as an SL task.

We have carefully crafted definitions of control and learning that are
agnostic in the sense that they hold regardless of whether the underlying
physics is classical or quantum. One key challenge that arises in unifying
classical and quantum definitions of control is that the usual notion of a
switch in classical control theory no longer applies. We define a switch which
is appropriate for an agnostic approach, Def. 1, in contrast to specifically
classical or quantum switches. Our solution is to cast the classical switch as
a logically reversible gate, thus naturally extending the switching operation
to the quantum domain. Interestingly, by following a quantum-pseudocode
convention for the quantum switch, we can easily “switch” between quantum and
classical descriptions. On the other hand, our agnostic definition of learning
builds on an established classical ML framework, thus avoiding the
discrepancies in the existing definitions of QML.

Our main result is our LfC framework which unifies learning and control in
both classical and quantum domains. Our framework includes extending the
classical control loop to the quantum case, dealing properly with the
switches, controller, what channels are classical, and what are quantum,
teacher and the user. Our framework differs from existing literature in two
aspects. Firstly, in contrast to literature not including a ‘teacher’ and a
‘user’, our framework includes a classical teacher, which trains a learner
(classical or quantum) for the control task, and a classical user who replaces
the teacher when the training and validation are complete. Secondly, our work
differs from Fu’s seminal work [16] in regards to the separation between
controller and learner and the role of teacher. By treating the learner and
controller separately, our framework becomes valid in the quantum domain as we
can just make the controller quantum and keep the learner compatible with the
classical setting of the real world. These new features make our framework
self-sufficient to be applied to any classical or quantum, or hybrid system.

We present the existing literature on control, learning and their connections,
in both classical and quantum domains, in the form of a square graph. An
intriguing fact about this knowledge graph is that it represents only a subset
of existing literature in the relevant fields. This is because of our strict
filtering criteria, which excludes misinterpreted works. For example, if a
paper claims to use ML for a control task but actually uses only optimization
under the hood, we exclude that paper. Thus, in addition to gaps, our
knowledge graph also exposes limitations and misinterpretations in the
existing literature.

Our unique way of representing literature is particularly interesting because
it helps us identify the knowledge gaps, which provides fodder for future
researchers. It is quite surprising to observe that although classical
learning is used for both classical and quantum control, the application of
quantum learning to quantum control has yet to be fully explored. Another
interesting observation is that classical and quantum learning has benefitted
from developments in classical and quantum control, respectively, but the
interrelations between classical control and quantum learning and quantum
control and classical learning are not explored yet. This knowledge graph also
conveys that the unification of classical and quantum control (and learning)
is not thoroughly explored, which we have addressed in this paper.

Guided by the knowledge of our LfC framework, we address the challenging task
of casting A$\mathscr{Q}$P as an SL problem. The A$\mathscr{Q}$P control task
is computationally expensive. Nevertheless, analysing this task in the light
of our framework allows us to employ ML to solve it, i.e. to potentially
achieve a scaling better than SQL for an unseen unknown-phase probability
distribution. In particular, we first elevate the control in A$\mathscr{Q}$P
(Task 1) to a generalised control task amenable to recasting as an SL problem.
By bringing learning into the picture, we could potentially reduce the
computational cost for calculating control policies that beat SQL.

##  VII Conclusions

The fields of quantum control and machine learning (ML) are rapidly
progressing, albeit mostly independent of each other. Although classical
control is a widely-popular and established field, quantum control is still in
its developing phase. On the other hand, quantum ML, concerning both quantum-
for-learning and learning-for-quantum ideas, is also a very popular research
topic. Despite separate research in these fields, not much attention was paid
to unifying the terminologies in quantum control (learning) and classical
control (learning); this lack of research might hinder quantum control and
learning from fully exploiting established techniques from their classical
counterparts. In this paper, we present unified, i.e. agnostic of whether
classical or quantum, definitions for control and learning and critically
review existing literature in the light of our agnostic definitions. Moreover,
we formulate a learning-for-control framework and explain how supervised
learning (SL) can be used to estimate the unknown phase in a quantum-enhanced
interferometric setup.

We review the relevant literature on ML, control systems, unification of
classical and quantum mechanics and adaptive quantum- enhanced
interferometric-phase estimation (A$\mathscr{Q}$P). In particular, we explain
key concepts by quoting from authoritative references and, in some cases,
formalizing them with mathematical relations. Moreover, we indicate popular
discrepancies in topics, including the definition of quantum ML, evolutionary
methods for reinforcement learning (RL) and the relation between optimization
and ML. We then discuss the equivalence between classical and quantum
descriptions of a physical system based on operational mechanics and geometric
correspondence. Lastly, we recap the quantum control task in A$\mathscr{Q}$P
and the optimization techniques, which were sometimes misrepresented as ML
techniques in literature, used to solve this control task.

The main result of our work is a learning-for-control framework, which holds
irrespective of whether learning and control are described by classical or
quantum mechanics. To do this, we first unify classical and quantum control by
quantizing the components of a typical closed-loop classical control system,
namely the evaluator, controller, plant, switch and their communication
channels. Using a similar approach, we then unify classical and quantum
learning. Finally, we constructed our “agnostic” framework by elevating an
established learning-for-control proposal to account for a classical/quantum
description of all components, including the learner, but separating the
teacher as a classical agent.

Based on our agnostic learning-for-control framework, we have two more
results. Firstly, we use our unified (quantum and classical) definitions of
control and learning to present the existing literature in these fields as a
knowledge graph. This square graph represents a subset of the existing
literature, which has been carefully filtered according to our clarified
definitions. Secondly, we cast the quantum control problem in A$\mathscr{Q}$P
as an SL problem and explain the calibration, training and testing steps using
our constructed framework. Although this control problem was framed as an RL
problem in the earlier works from our research group, the direct policy search
approach doesn’t comply with the conventional RL paradigm as put forward by
Sutton. Nevertheless, our new way of casting the A$\mathscr{Q}$P problem as an
SL problem has potential applications in enhancing quantum clocks [18] and
interferometric position-shift measurements [19, 20, 21].

Our work leads to many interesting questions and research directions. Although
the field of quantum ML is currently very popular, its applications to
classical and quantum control have not been investigated properly. Another
interesting yet unexplored topic is quantum control for classical and quantum
learning. We are particularly excited to study how control strategies on
current quantum hardwares can enhance the performance of quantum generative ML
to the point of achieving quantum advantage on these noisy devices. Based on
our proposal, one can implement an SL agent for solving the quantum control
problem of A$\mathscr{Q}$P to potentially achieve a scaling better than the
standard quantum limit.

Beyond the above-mentioned practical applicability, our learning-for-control
framework makes the reader ponder more philosophical questions about control,
learning and their interconnection. Can M/U be quantum? How does the
measurement paradox affect the control and learning feedback loops? Does
control theory make sense for a quantum controller? Although we do not have
the answers to such questions, one value of our work is that such previously
unexplored questions arise very clearly from our framework.

##  VIII Acknowledgements

We acknowledge the traditional owners of the land on which this work was
undertaken at the University of Calgary: the Treaty 7 First Nations
(www.treaty7.org). SSV would like to thank MITACS. AD would like to thank
MITACS and the Canadian Queen Elizabeth II Diamond Jubilee Scholarships
program. BCS appreciates financial support from the Natural Sciences and
Engineering Research Council of Canada. AD, EJP and BCS acknowledge the
support of the Major Innovation Fund, Government of Alberta, Canada. We thank
Carlo Maria Scandolo and Howard M. Wiseman for useful discussions on quantum
feedback.

SSV and AD contributed equally to this work.

## References

  * Dorf and Bishop [2008] R. Dorf and R. Bishop,  _Modern Control Systems_ , 11th ed. (Pearson, Boston, 2008).
  * Rosolia  _et al._ [2018] U. Rosolia, X. Zhang, and F. Borrelli, Data-driven predictive control for autonomous systems, Annu. Rev. Control Robot. Auton. Syst. 1, 259 (2018).
  * Mitchell [1997] T. M. Mitchell,  _Machine Learning_ , 1st ed. (McGraw-Hill, New York, 1997).
  * Russell and Norvig [2020] S. Russell and P. Norvig,  _Artificial Intelligence: A Modern Approach_ , 4th ed., Pearson Series in Artifical Intelligence (Pearson, Hoboken, 2020).
  * Duriez  _et al._ [2017] T. Duriez, S. L. Brunton, and B. R. Noack,  _Machine Learning Control-Taming Nonlinear Dynamics and Turbulence_, 1st ed., Fluid Mechanics and Its Applications (Springer, Cham, 2017).
  * E  _et al._ [2018] W. E, J. Han, and Q. Li, A mean-field optimal control formulation of deep learning, Res. Math. Sci. 6, 10 (2018).
  * Mehta  _et al._ [2019] P. Mehta, M. Bukov, C.-H. Wang, A. G. Day, C. Richardson, C. K. Fisher, and D. J. Schwab, A high-bias, low-variance introduction to machine learning for physicists, Phys. Rep. 810, 1 (2019).
  * Wiseman and Milburn [2009] H. M. Wiseman and G. J. Milburn,  _Quantum Measurement and Control_ (Cambridge University Press, Cambridge, 2009).
  * Jacobs [2014] K. Jacobs,  _Quantum Measurement Theory and its Applications_ (Cambridge University Press, Cambridge, 2014).
  * Zhang  _et al._ [2017] J. Zhang, Y.-x. Liu, R.-B. Wu, K. Jacobs, and F. Nori, Quantum feedback: theory, experiments, and applications, Phys. Rep. 679, 1 (2017).
  * Biamonte  _et al._ [2017] J. Biamonte, P. Wittek, N. Pancotti, P. Rebentrost, N. Wiebe, and S. Lloyd, Quantum machine learning, Nature 549, 195 (2017).
  * Hentschel and Sanders [2011a] A. Hentschel and B. C. Sanders, Efficient algorithm for optimizing adaptive quantum metrology processes, Phys. Rev. Lett. 107, 233601 (2011a).
  * Lovett  _et al._ [2013] N. B. Lovett, C. Crosnier, M. Perarnau-Llobet, and B. C. Sanders, Differential evolution for many-particle adaptive quantum metrology, Phys. Rev. Lett. 110, 220501 (2013).
  * Hogan  _et al._ [2021] A. Hogan, E. Blomqvist, M. Cochez, C. D’amato, G. D. Melo, C. Gutierrez, S. Kirrane, J. E. L. Gayo, R. Navigli, S. Neumaier, A.-C. N. Ngomo, A. Polleres, S. M. Rashid, A. Rula, L. Schmelzeisen, J. Sequeda, S. Staab, and A. Zimmermann, Knowledge graphs, ACM Comput. Surv. 54, 1 (2021).
  * Vedaie  _et al._ [2018] S. S. Vedaie, P. Palittapongarnpim, and B. C. Sanders, Reinforcement learning for quantum metrology via quantum control, in  _2018 IEEE Photonics Society Summer Topical Meeting Series (SUM)_ (IEEE, 2018) pp. 163–164.
  * Fu [1970] K.-S. Fu, Learning control systems–review and outlook, IEEE Trans. Automat. Contr. 15, 210 (1970).
  * Wilde [2017] M. M. Wilde,  _Quantum Information Theory_, 2nd ed. (Cambridge University Press, Cambridge, 2017).
  * Borregaard and Sørensen [2013] J. Borregaard and A. S. Sørensen, Near-Heisenberg-limited atomic clocks in the presence of decoherence, Phys. Rev. Lett. 111, 090801 (2013).
  * Hollenhorst [1979] J. N. Hollenhorst, Quantum limits on resonant-mass gravitational-radiation detectors, Phys. Rev. D 19, 1669 (1979).
  * Caves  _et al._ [1980] C. M. Caves, K. S. Thorne, R. W. P. Drever, V. D. Sandberg, and M. Zimmermann, On the measurement of a weak classical force coupled to a quantum-mechanical oscillator. I. Issues of principle, Rev. Mod. Phys. 52, 341 (1980).
  * Caves [1981] C. M. Caves, Quantum-mechanical noise in an interferometer, Phys. Rev. D 23, 1693 (1981).
  * Mohri  _et al._ [2018] M. Mohri, A. Rostamizadeh, and A. Talwalkar,  _Foundations of Machine Learning_ , 2nd ed., Adaptive Computation And Machine Learning series (MIT press, Cambridge, 2018).
  * Rudin [1976] W. Rudin,  _Principles of Mathematical Analysis_ , 3rd ed., International Series in Pure and Applied Mathematics (McGraw-Hill, New York, 1976).
  * van Kampen [2007] N. van Kampen,  _Stochastic Processes in Physics and Chemistry_, 3rd ed., North-Holland Personal Library (Elsevier, Amsterdam, 2007).
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto,  _Reinforcement Learning: An Introduction_ , 2nd ed., Adaptive Computation and Machine Learning series (MIT Press, Cambridge, 2018).
  * Domingos [2012] P. Domingos, A few useful things to know about machine learning, Commun. ACM 55, 78–87 (2012).
  * Al-jabery  _et al._ [2020] K. K. Al-jabery, T. Obafemi-Ajayi, G. R. Olbricht, and D. C. Wunsch II, 2 - Data Preprocessing, in  _Computational Learning Approaches to Data Analytics in Biomedical Applications_, edited by K. K. Al-jabery, T. Obafemi-Ajayi, G. R. Olbricht, and D. C. Wunsch II (Academic Press, London, 2020) 1st ed., pp. 7–27.
  * Burman [1989] P. Burman, A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods, Biometrika 76, 503 (1989).
  * Dalal  _et al._ [2021] A. Dalal, M. Bagherimehrab, and B. C. Sanders, Quantum-assisted support vector regression for detecting facial landmarks (2021), arXiv:2111.09304 .
  * Kearns and Vazirani [1994] M. J. Kearns and U. Vazirani,  _An Introduction to Computational Learning Theory_ (MIT Press, Cambridge, 1994).
  * Valiant [1984] L. G. Valiant, A theory of the learnable, Commun. ACM 27, 1134–1142 (1984).
  * Klebanov  _et al._ [2009] L. B. Klebanov, S. T. Rachev, and F. J. Fabozzi,  _Robust and Non-Robust Models in Statistics_ (Nova Science Publishers, Hauppauge, 2009).
  * Ben-David  _et al._ [1997] S. Ben-David, E. Kushilevitz, and Y. Mansour, Online learning versus offline learning, Mach. Learn. 29, 45 (1997).
  * Levine  _et al._ [2020] S. Levine, A. Kumar, G. Tucker, and J. Fu, Offline reinforcement learning: tutorial, review, and perspectives on open problems (2020), arXiv:2005.01643 .
  * Prudencio  _et al._ [2023] R. F. Prudencio, M. R. O. A. Maximo, and E. L. Colombini, A survey on offline reinforcement learning: taxonomy, review, and open problems, IEEE Trans. Neural Netw. Learn. Syst. , 1 (2023).
  * Dunjko and Briegel [2018] V. Dunjko and H. J. Briegel, Machine learning & artificial intelligence in the quantum domain: a review of recent progress, Rep. Prog. Phys. 81, 074001 (2018).
  * Ciliberto  _et al._ [2018] C. Ciliberto, M. Herbster, A. D. Ialongo, M. Pontil, A. Rocchetto, S. Severini, and L. Wossnig, Quantum machine learning: a classical perspective, Proc. R. Soc. A: Math. Phys. Eng. Sci. 474, 20170551 (2018).
  * Schuld and Petruccione [2021] M. Schuld and F. Petruccione,  _Machine Learning with Quantum Computers_, 2nd ed., Quantum Science and Technology (Springer, Cham, 2021).
  * Wittek [2014] P. Wittek,  _Quantum Machine Learning: What Quantum Computing Means to Data Mining_ (Academic Press, New York, 2014).
  * Preskill [2012] J. Preskill, Quantum computing and the entanglement frontier (2012), arXiv:1203.5813 .
  * Preskill [2018] J. Preskill, Quantum computing in the NISQ era and beyond, Quantum 2, 79 (2018).
  * Harrow and Montanaro [2017] A. W. Harrow and A. Montanaro, Quantum computational supremacy, Nature 549, 203 (2017).
  * Schuld  _et al._ [2015] M. Schuld, I. Sinayskiy, and F. Petruccione, An introduction to quantum machine learning, Contemp. Phys. 56, 172 (2015).
  * Wiebe [2020] N. Wiebe, Key questions for the quantum machine learner to ask themselves, New J. Phys. 22, 091001 (2020).
  * Rebentrost  _et al._ [2014] P. Rebentrost, M. Mohseni, and S. Lloyd, Quantum support vector machine for big data classification, Phys. Rev. Lett. 113, 130503 (2014).
  * Noori  _et al._ [2020] M. Noori, S. S. Vedaie, I. Singh, D. Crawford, J. S. Oberoi, B. C. Sanders, and E. Zahedinejad, Analog-quantum feature mapping for machine-learning applications, Phys. Rev. Appl. 14, 034034 (2020).
  * Paparo  _et al._ [2014] G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Quantum speedup for active learning agents, Phys. Rev. X 4, 031002 (2014).
  * Saggio  _et al._ [2021] V. Saggio, B. E. Asenbeck, A. Hamann, T. Strömberg, P. Schiansky, V. Dunjko, N. Friis, N. C. Harris, M. Hochberg, D. Englund, S. Wölk, H. J. Briegel, and P. Walther, Experimental quantum speed-up in reinforcement learning agents, Nature 591, 229 (2021).
  * Zurek [1981] W. H. Zurek, Pointer basis of quantum apparatus: Into what mixture does the wave packet collapse?, Phys. Rev. D 24, 1516 (1981).
  * Hou and Wang [2013] Z.-S. Hou and Z. Wang, From model-based control to data-driven control: Survey, classification and perspective, Inf. Sci. 235, 3 (2013).
  * Walmsley and Rabitz [2003] I. Walmsley and H. Rabitz, Quantum physics under control, Phys. Today 56, 43 (2003).
  * Lloyd [2000] S. Lloyd, Coherent quantum feedback, Phys. Rev. A 62, 022108 (2000).
  * Borzì  _et al._ [2017] A. Borzì, G. Ciaramella, and M. Sprengel, Chapter 2: Quantum Mechanics and the Schrödinger Equation, in  _Formulation and Numerical Solution of Quantum Control Problems_, Computational Science & Engineering (Society for Industrial and Applied Mathematics, Philadelphia, 2017) pp. 7–71.
  * Brif  _et al._ [2010] C. Brif, R. Chakrabarti, and H. Rabitz, Control of quantum phenomena: past, present and future, New J. Phys. 12, 075008 (2010).
  * Khaneja  _et al._ [2005] N. Khaneja, T. Reiss, C. Kehlet, T. Schulte-Herbrüggen, and S. J. Glaser, Optimal control of coupled spin dynamics: design of NMR pulse sequences by gradient ascent algorithms, J. Magn. Reson. 172, 296 (2005).
  * Zahedinejad  _et al._ [2014] E. Zahedinejad, S. Schirmer, and B. C. Sanders, Evolutionary algorithms for hard quantum control, Phys. Rev. A 90, 032310 (2014).