given state. Thus, we will bootstrap the estimation at time-step
$T_{\text{seg}}$ by using an approximation for the value function 555A simple
$n$-step bootstrapped estimation of the advantage takes the form
$A(s_{t},a_{t})=r_{t}+r_{t+1}+\dots+r_{T-1}+V_{\phi}(s_{T})$. . The estimator
that we use is a truncated version of the generalized advantage estimator
[97]:

$A(s_{t},a_{t})=\delta_{t}+(\gamma\lambda)\delta_{t+1}+\cdots+(\gamma\lambda)^{T-t+1}\delta_{T-1},$
(27)

where $\delta_{t}=r_{t}+\gamma V_{\phi}(s_{t+1})-V_{\phi}(s_{t})$; $\gamma$ is
the discount factor; $\lambda$ is the weight averaging factor, and $V_{\phi}$
is the critic neural network approximating the value function (see Fig. 19,
Table 1).

At every iteration, before computing the objective, the advantages are
standardized on the mini-batch level to have zero mean and unit variance:

$\hat{A}(s_{t},a_{t})=\frac{A(s_{t},a_{t})-\mu_{A}}{\sigma_{A}}.$ (28)

This modification makes use of a constant baseline for all (state, action)
pairs in the batch and effectively re-scales the learning rate by a factor of
$1/\sigma_{A}$.

Note that the entropy regularization term is also computed over the mini-
batch. This is done because after every update we change the policy weights
and hence the entropy.

Finally, at the end of every epoch we check the KL divergence between the
newest and the original policy and stop the learning phase if a given
threshold ($KL_{\text{lim}}$) is reached. The $KL$ divergence can be
calculated as follows:

$\displaystyle D_{\text{KL}}(\pi_{\text{old}}||\pi)$
$\displaystyle=\underset{a\sim\pi_{\text{old}}}{\mathbb{E}}\bigg{[}\log\frac{\pi_{\text{old}}(a|s)}{\pi(a|s)}\bigg{]}$
(29)
$\displaystyle=\underset{a\sim\pi_{\text{old}}}{\mathbb{E}}\big{[}\log\pi_{\text{old}}(a|s)-\log\pi(a|s)\big{]}.$

This is needed, because simply clipping the objective might not be enough to
capture the divergence between the current policy $\pi_{\theta}$ and the old
policy $\pi_{\theta_{\text{old}}}$. Having this additional check also allows
us to terminate the learning phase if the new policy starts to deviate too
much. For more details see Algorithm 2.

input : obs \- environment observations

input : acts \- agent actions

input : returns \- obtained returns

input : K \- number of training epochs

input : KL${}_{\text{lim}}$ \- $D_{\text{KL}}$ threshold for stopping

$adv\leftarrow\textsc{GAE}(returns)$ ;

$\triangleright$ Eq. (27)

$p\\_old\leftarrow\textsc{Policy}(obs)$ ;

$\triangleright$ calc. $\pi$ before updates

$i\leftarrow 1$;

for  _$i\leq K$_ do

/* sample mini-batches from the experiences */

/* $\footnotesize o\text{ :~{}a mini-batch of observations}$ */

/* $\footnotesize a\text{ :~{}corresp.~{}selected actions}$ */

/* $\footnotesize ad\text{ :~{}corresp.~{}calculated advantages}$ */

/* $\footnotesize p_{o}\text{:~{}corresp.~{}probs using }\pi\text{ before
update}$ */

for  _$o,a,ad,p\\_o\in\textsc{Batch}(obs,acts,adv,p\\_old)$_ do

$p\leftarrow\textsc{Policy}(o)$;

$ad\leftarrow\displaystyle\frac{ad-ad.\textsc{mean}()}{ad.\textsc{std}()}$ ;

$\triangleright$ Eq. (28)

$\rho\leftarrow\displaystyle\frac{p\\_{old}}{p}$;

$loss\leftarrow\textsc{min}\displaystyle\big{(}\rho\times
ad,\textsc{clip}(\rho,1-\epsilon,1+\epsilon)\times ad\big{)}$;

$loss\leftarrow loss.\textsc{mean}()$ ;

$\triangleright$ Eq. (24)

$loss\leftarrow loss-\log\textsc{Policy}(a|o)$ ;

$\triangleright$ Eq. (26)

/* backward pass to update policy network */

end for

$i\leftarrow i+1$;

$p\leftarrow\textsc{Policy}(obs)$;

$KL\leftarrow\log p\\_old-\log p$ ;

$\triangleright$ Eq. (29)

if  _ $KL>KL_{\mathrm{lim}}$_ then

Break;

end if

end for

Algorithm 2 Training with PPO

####  B.2.4 Value function optimization

In addition to optimizing the policy we also need to optimize the weights of
the value network (value net). The value network is trained by minimizing the
mean squared error between its predictions and the computed returns from the
trajectories, which are given by

$V_{\text{target}}=A(s_{t},a_{t})+V_{\phi}(s_{t}).$

The value network (critic) is updated simultaneously with the policy network
(actor) using the same mini-batching strategy, i.e., we train in $K$ epochs
with mini-batches of size $B$. If the early stopping criterion is met then we
stop updating the value network as well.

Just like the clipped objective for the policy network, we also clip the value
loss before updating the parameters:

$\displaystyle V_{\text{clip}}$
$\displaystyle=\text{clip}(V_{\phi},V_{\phi_{\text{old}}}-\epsilon_{V},V_{\phi_{\text{old}}}+\epsilon_{V})$
$\displaystyle L^{V}$
$\displaystyle=\max[(V_{\phi}-V_{\text{target}})^{2},(V_{\text{clip}}-V_{\text{target}})^{2}],$

where $\epsilon_{V}$ is a hyper-parameter for clipping the value objective.
The policy network and the value network use different learning rate
parameters. The corresponding objectives are clipped using different clipping
values. All hyper-parameters used for training the models are given in Table
1.

###  B.3 Policy and value network architectures

In what follows we will describe the architectures of the neural networks used
to approximate the policy and value functions of our agents.

Figure 19: Policy and Value networks architecture. The policy network (a)
accepts as inputs the symmetrized two-qubit reduced density matrices of a
quantum state and outputs a probability distribution over the actions. The
policy is a permutation-equivariant network consisting of a linear layer, a
stack of transformer encoder blocks, and a final linear layer with softmax
non-linearity. The value network (b) is a simple three-layer fully-connected
network that accepts the same input flattend and returns a single number – the
value for that state. Both networks use ‘ReLU’ non-linearities. The Encoder
block (c) has a self-attention layer followed by a position-wise fully-
connected network. The block accepts a sequence of vectors $\mathrm{x_{1}\dots
x_{n}}$ as input and outputs the encoding vectors $\mathrm{r_{1}\dots r_{n}}$.
We make use of a multi-headed self-attention layer (d) allowing each element
to ”pay attention” to multiple elements. The attention layer accepts three
different inputs, namely keys ($\mathbf{K}$), queries ($\mathbf{Q}$), and
values ($\mathbf{V}$). For self-attention we have
$\mathbf{K}=\mathbf{Q}=\mathbf{V}=\mathbf{X}$ (see Eq. (B.4)).

The policy network is designed to be permutation-equivariant (see App. B.4).
It is a stack of $N_{\text{layer}}$ identical transformer encoder blocks
([71]) that are applied one after another (see Fig. 19a). Each encoder block
has a multi-headed self-attention layer followed by a position-wise fully-
connected network (multilayer perception). After each layer, a residual
connection is applied followed by a normalization layer (Fig. 19c). We do not
use dropout in the policy network. In each of the encoder blocks the self-
attention layer has queries, keys, and values of dimensionality
$D_{\text{qkv}}$ split into $N_{\text{heads}}$ attention heads (Fig. 19d). The
fully connected network has one hidden layer with dimensionality
$D_{\text{mlp}}$ and ‘ReLU’ non-linearity. Both the input and output
dimensions of the fully connected network are $D_{\text{qkv}}$.

Note that the inputs and the outputs of each of the encoder blocks have the
same dimensionality, i.e., the transformer encoders operate in an embedding
space with a fixed dimension (in this case $D_{\text{qkv}}$).

The policy network receives as input the symmetrized two-qubit density
matrices (see App. B.1.4). Before being forwarded through the encoder blocks,
the reduced density matrices are forwarded through a position-wise linear
layer with no non-linearities; they are embedded into
$D_{\text{qkv}}$-dimensional space.

The output of the policy network should be a vector containing the probability
of taking each of the actions from the action set (see App. B.1.5),
representing the probability of choosing that action. Thus, the outputs of the
encoder blocks are forwarded through another position-wise linear layer and
are embedded in one-dimensional space. After that, we apply a softmax non-
linearity to get the probability scores.

The value network is a simple three-layer fully connected network that accepts
the same input as the policy network but flattened (Fig. 19b). The network
uses ‘ReLU’ non-linearities and each of the hidden layers has size
$h_{\text{hid}}$. The output is a single number – the value for the given
quantum state.

It is worth mentioning here that we do not require that the value network
exhibits the permutation-equivariance/invariance property. Note that the value
network is only used for training the agent when computing the advantages [Eq.
(27)]. During inference, solely the design of the policy is important so that
the agent remains insensitive to qubit permutations. In our experiments, we
did not find any improvement in the policy training when using a permutation-
insensitive architecture for the value network. On the downside, however,
training took longer due to the increased computational complexity.

###  B.4 Permutation-equivariant deep learning architecture

In this section, we briefly discuss the inherent permutation equivariance
property of the self-attention layer. The equations governing the self-
attention layer (see Fig. 19c) are:

$\displaystyle\mathbf{Z}$ $\displaystyle=\textsc{SelfAttention}(\mathbf{X})$
$\displaystyle=\text{Softmax}\bigg{(}\frac{\mathbf{XW_{Q}W_{K}^{T}X^{T}}}{\sqrt{d_{k}}}\bigg{)}\mathbf{XW_{V}},$
(30)

where $\mathbf{W_{Q}}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}$,
$\mathbf{W_{K}}\in\mathbb{R}^{d_{\text{model}}\times d_{k}}$,
$\mathbf{W_{V}}\in\mathbb{R}^{d_{\text{model}}\times d_{\text{v}}}$ are the
query, key, and value weight matrices respectively,
$\mathbf{X}\in\mathbb{R}^{N\times d_{\text{model}}}$ is the sequence of inputs
to the layer, and $\mathbf{Z}\in\mathbb{R}^{N\times d_{\text{model}}}$ is the
sequence of output embeddings 666In our case, we have
$d_{v}=d_{k}=d_{q}=d_{\text{model}}=D_{\text{qkv}}$. However, the expressions
here hold in general, i.e., for any $d_{k}$, $d_{v}$..

Let us consider how an arbitrary embedding $z_{i}$ is computed. First, the
attention probability matrix ($\mathbf{A}\in\mathbb{R}^{N\times N}$) is
computed, by matrix-vector multiplying the query embeddings with the key
embeddings and applying the Softmax function on the result:

$\displaystyle\mathbf{A}$
$\displaystyle=\text{Softmax}\bigg{(}\frac{\mathbf{XW_{Q}W_{K}^{T}X^{T}}}{\sqrt{d_{k}}}\bigg{)}$
(31) $\displaystyle=\text{Softmax}\bigg{(}\mathbf{XWX^{T}}\bigg{)}.$ (32)

The row-vector $a_{i}$ of $\mathbf{A}$ contains the attention probabilities
with which element $x_{i}$ attends to all the elements in the sequence:

$a_{i}=\text{Softmax}\bigg{(}x_{i}\mathbf{WX^{T}}\bigg{)}.$ (33)

In particular, the entry $a_{ij}$ gives us the attention probability with
which element $x_{i}$ attends to element $x_{j}$:

$a_{ij}=\frac{\text{exp}(x_{i}Wx_{j}^{T})}{\sum_{k}\text{exp}(x_{i}Wx_{k}^{T})}.$
(34)

The embedding $z_{i}$ of $x_{i}$ is then computed as:

$\displaystyle z_{i}$ $\displaystyle=a_{i}\mathbf{XW_{V}}$ (35)
$\displaystyle=\sum_{j}a_{ij}x_{j}\mathbf{W_{V}}$
$\displaystyle=\sum_{j}\bigg{(}\frac{\text{exp}(x_{i}Wx_{j}^{T})}{\sum_{k}\text{exp}(x_{i}Wx_{k}^{T})}x_{j}\mathbf{W_{V}}\bigg{)}.$

Let us now assume that the order of the vectors
$\\{x_{0},x_{2},\dots,x_{N-1}\\}$ is permuted according to $i\rightarrow
i^{\prime}$; i.e., $i^{\prime}$ is the new position corresponding to the
vector $x_{i}$ from the original sequence. Consulting Eq. (35) we can see that
the embedding $z_{i^{\prime}}=z_{i}$, implying that permuting the input
sequence results in an equivariant permutation of the output sequence.

Alternatively, one can denote the permutation by a matrix $\mathbf{P}$, and
describe its action on the input data by $\mathbf{X}\to\mathbf{PX}$.
Substituting this into Eq. (B.4), we find:

$\displaystyle\textsc{SelfAttention}(\mathbf{PX})=$
$\displaystyle=\text{Softmax}\bigg{(}\frac{\mathbf{PXW_{Q}W_{K}^{T}X^{T}P^{T}}}{\sqrt{d_{k}}}\bigg{)}\mathbf{PXW_{V}}$
$\displaystyle=\mathbf{P}\;\text{Softmax}\bigg{(}\frac{\mathbf{XW_{Q}W_{K}^{T}X^{T}}}{\sqrt{d_{k}}}\bigg{)}\mathbf{P^{T}}\mathbf{PXW_{V}}$
$\displaystyle=\mathbf{P}\;\text{Softmax}\bigg{(}\frac{\mathbf{XW_{Q}W_{K}^{T}X^{T}}}{\sqrt{d_{k}}}\bigg{)}\mathbf{XW_{V}}$
$\displaystyle=\mathbf{PZ},$

where we used that
$\text{Softmax}(\mathbf{PXP^{T}}){=}\mathbf{P}\text{Softmax}(\mathbf{X})\mathbf{P^{T}}$,
and $\mathbf{PP^{T}}=\mathbf{1}$ since permutations are unitary maps. Hence,
the output of the self-attention layer is correctly transformed when permuting
its input.

###  B.5 Attention Heads

Section B.4 defined the single-head self-attention. The transformer
architecture uses multi-head self-attention and each head has its own set of
learnable parameters $\mathbf{W_{Q}},\mathbf{W_{K}},\mathbf{W_{V}}$. Multi-
head attention is a way to linearly project the keys, queries, and values
$N_{\text{heads}}$ time with different
$\mathbf{W_{Q}},\mathbf{W_{K}},\mathbf{W_{V}}$ matrices. The projections
become ($D_{\text{qkv}}$/$N_{\text{heads}}$)-dimensional vectors (instead of
$D_{\text{qkv}}$-dimensional) which are then concatenated into a single
$D_{\text{qkv}}$-dimensional output embedding: hence, each head computes only
part of the output embedding for each input token (Fig. 19(d)). Since heads do
not share $\mathbf{W_{Q}},\mathbf{W_{K}},\mathbf{W_{V}}$ it follows that each
head has an attention distribution $\mathbf{A}$ specific to it.

Figure 20:  Attention Heads. Attention scores for the state
$\mathrm{|0_{1}\rangle|Bell_{23}\rangle|0_{4}\rangle}$ (a), and for
$\mathrm{|R_{12}\rangle|R_{34}\rangle}$ (b), see colorbar. Row vectors $a_{i}$
give the attention scores with which element $x^{(i,j)}$ attends to all
elements in the sequence. All elements in the row vector $a_{i}$ add up to
unity, because $a_{i}$ is normalized to be a probability distribution, cf. Eq.
(33).

The attention distribution $\mathbf{A}$ is an $\mathcal{|A|\times|A|}$
dimensional matrix, and can be plotted as a heatmap. In fact, this is one of
the aspiring aspects of the self-attention mechanism: as an architecture that
has its roots in machine translation, the attention scores can reveal which
words in the source language are being attended to when generating each word
in the target language. By visualizing attention distributions, one can
identify which source words contribute the most to generating specific target
words. Larger attention scores indicate stronger connections between tokens
(embeddings), implying that those input tokens are more influential in
generating the output tokens. Although the inputs to our multi-head attention
layers are linear embeddings of vectors derived from reduced density matrices
(see Eq. (B.1.4) and Fig. 19(a)) instead of language words, they can
nevertheless shed some light on the decision-making process of the agent.
Since our results show that the agents are capable of recognizing local
entanglement structure (see Fig. 8), one can suppose that the attention
distributions of the policy networks may show some information that correlates
with the entanglement structure.

We show the attention distributions from our 4-qubit agent for two exemplary
initial states in Fig. 20. The policy network of this agent has 2 encoder
layers, each with 2 heads [Table 1]. Each $x^{(i,j)}$ is an embedding of an
observation $o^{(i,j)}$ which is a symmetrized reduced density matrix (see
Sec. B.1.4). We use “token” and “embedding” interchangeably in the text. Note
that each token is specified with two super-indices (i,j), which indicate that
the RDMs are computed over qubits $i$ and $j$; each element $a_{kl}$ in the
matrix $\mathcal{A}$ is the attention score between two tokens.

Fig. 20(a) shows the attention distribution for state
$\mathrm{|0_{1}\rangle|Bell_{23}\rangle|0_{4}\rangle}$ with only 2 entangled
qubits. For this state, connections can be drawn between the attention
distribution and the entanglement structure. Each row $a_{i}$ in Layer 1, Head
1 has nonzero attention only on embeddings $x^{(i,j)}$ such that one of $i$ or
$j$ is from the Bell subsystem (qubits $(2,3)$), and the other – from the
$(1,4)$ qubit subsystem. Head 2 in Layer 1 attends primarily to the Bell
subsystem, except $x^{(1,4)}$ which attends to itself. In Layer 2, Head 1 we
see that $x^{(2,3)}$ attends only to $x^{(1,4)}$ and $x^{(2,3)}$. As
$x^{(1,4)}$ and $x^{(2,3)}$ are embeddings of $o^{(1,4)}$ and $o^{(2,3)}$,
they represent the $\mathrm{|0_{1}\rangle}\mathrm{|0_{4}\rangle}$ and
$|\mathrm{Bell}_{23}\rangle$ subsystems, respectively. Recall also that the
policy network maps each observation $o^{(i,j)}$ to the probability for taking
action $(i,j)$ (see Secs. B.1.5 and B.1.4). A connection can then be drawn
between the selected action and the entanglement structure of the state: the
action with the highest probability, in this case, is (2,3) and its
corresponding embedding $x^{(2,3)}$ attends only to the embeddings that
represent subsystems $\mathrm{|0_{1}\rangle}\mathrm{|0_{4}\rangle}$ and
$|\mathrm{Bell}_{23}\rangle$.

Figure 20(b) shows the attention distribution for a state with two pairs of
entangled subsystems – this time the Haar-random state
$\mathrm{|R_{12}\rangle|R_{34}\rangle}$. In Layer 1, Head 1 every $x^{(i,j)}$
attends to $x^{(1,2)}$ which represents $\mathrm{|R_{12}\rangle}$. In Head 2
every (row) token attends to only one (column) token; except for $x^{(2,3)}$
which attends to itself, all other tokens $x^{(i,j)}$ attend to a token that
includes either qubit $i$ or $j$ in its RDM. In Layer 2, Head 1 most of the
tokens attend to $x^{(2,4)}$. Contrary to the
$\mathrm{|0_{1}\rangle|Bell_{23}\rangle|0_{4}\rangle}$ example, here it is
harder to draw a connection between the action the agent selects and the
attention distribution.

###  B.6 Hyper-Parameters

In the following Table 1 we provide all of the hyper-parameters needed to
reproduce the results from the experiments. The hyper-parameters are split
into three categories: i) hyper-parameters concerning the environment; ii)
hyper-parameters concerning the agent policy architecture; iii) hyper-
parameters concerning the PPO training procedure.

Table 1: Hyper-parameters used for training the RL agents on quantum systems of different sizes. The hyper-parameters are split into three categories: environment, policy, and training hyper-parameters. Description |  Notation |  $L=4$ |  $L=5$ |  $L=6$
---|---|---|---|---
Environment |  |  |  |
Number of states in the vectorized environment (Sec. B.1.1) |  $B$ |  64 |  128 |  512
Maximum number of episode steps (Sec. B.1.2) |  $T_{\text{trunc}}$ |  8 |  40 |  90
Disentangling threshold for $S_{\text{avg}}$ (Sec. B.1.2) |  $\epsilon$ |  1e-3 |  1e-3 |  1e-3
Minimum support for generating initial states (Sec. B.1.3) |  $p$ |  2 |  2 |  3
Policy (Sec. B.3) |  |  |  |
Number of transformer encoders |  $N_{\text{layers}}$ |  2 |  4 |  4
Number of attention heads per layer |  $N_{\text{heads}}$ |  2 |  4 |  4
Embedding dimension for the self-attention layer |  $D_{\text{qkv}}$ |  128 |  128 |  256
Fully-connected encoder layer dimension |  $D_{\text{mlp}}$ |  256 |  512 |  1024
Value network hidden layers size |  $h_{\text{hid}}$ |  128 |  256 |  256
Training |  |  |  |
Number of steps per trajectory segment (Sec. B.2.3) |  $T_{\text{seg}}$ |  16 |  64 |  90
Number of training iterations |  $N_{\text{iters}}$ |  4000 |  10000 |  12000
Policy network learning rate |  $lr_{\pi}$ |  2e-4 |  2e-4 |  2e-4
Clip value for PPO updates of the policy (Sec. B.2.1) |  $\epsilon_{\pi}$ |  0.2 |  0.2 |  0.2
Value network learning rate |  $lr_{V}$ |  3e-4 |  3e-4 |  3e-4
Clip value for updates of the value network (Sec. B.2.4) |  $\epsilon_{V}$ |  10. |  10. |  10.
$D_{\text{KL}}$ threshold for early stopping (Eq. (29)) |  $KL_{\text{lim}}$ |  0.01 |  0.01 |  0.01
Discount factor for advantage estimation (Eq. (27) |  $\lambda_{\text{GAE}}$ |  0.95 |  0.95 |  0.95
Gradient clipping by norm for both networks [99] |  $g_{\text{clip}}$ |  1.0 |  1.0 |  1.0
Temperature for entropy regularization (Eq. (25) |  $\beta^{-1}$ |  0.1 |  0.1 |  0.1
Number of epochs for PPO updates (Sec. B.2.3) |  $K$ |  3 |  3 |  3
Batch size for PPO updates (Sec. B.2.3) |  $B_{\text{PPO}}$ |  128 |  128 |  512
Discount factor for future rewards (Eq. (2)) |  $\gamma$ |  1.0 |  1.0 |  1.0

##  Appendix C Beam search for circuit optimization

###  C.1 Tree search algorithms

As discussed in Sec. II, the disentangling problem can be reduced to two sub-
problems: (1) identifying the sequence of pairs of qubits to apply the two-
qubit gates on, and (2) finding the corresponding optimal two-qubit unitary
gates. A partial analytical solution to (2) is provided in Sec. A.1. In this
section, we propose an alternative solution to (1) using a tree search
algorithm which is very suitable for noise-free environments.

Given the combinatorial nature of the problem, and more precisely, the fact
that different sequences of actions need to be explored before a solution is
found, one may argue that other classes of algorithms, such as tree-search
algorithms, could also be suitable for this task.

Figure 21: An example execution of the beam search algorithm. Subfigure (a)
shows three iterations of the algorithm. At each iteration the current leaf
nodes are compared and sorted using a heuristic function $h$. Only the best
$k=2$ nodes are expanded and the rest (shown in gray) are pruned. Expanding a
node, shown in (b), consists of applying every possible action from the action
space $|\mathcal{A}|=4$ to this node - the new states
$\ket{\psi_{0}},\dots,\ket{\psi_{3}}$ become the child nodes of the expanded
node.

Indeed, in the case of a noise-free environment, a tree-search procedure may
be used to produce the disentangling circuit offline. In this case, a tree-
search agent takes as input the initial quantum state $|\psi\rangle$ in its
entirety (i.e., no partial observations), and produces the actions that need
to be applied for the state to be disentangled. We call the resulting sequence
of actions a plan. Note that in case the environment is non-deterministic
(e.g., due to noise), using the plan is not guaranteed to lead to a solution.
(Obviously, deviating from the planned trajectory due to some non-determinism
would set the agent off-course and, thus, render the plan useless.)

The number of possible $T$-step sequences of actions is $|\mathcal{A}|^{T}$,
where $|\mathcal{A}|=L(L-1)/2$, and $L$ is the number of qubits in the quantum
state. As discussed in Sec. III, the number of actions needed to disentangle a
quantum system grows exponentially with the system size, i.e. $T\sim\exp(L)$.
Thus, for the size $|\mathcal{S}|$ of the search space of our tree-search
algorithm, we have $|\mathcal{S}|\sim\exp(\exp(L))$ – a super-exponential
growth.

Therefore, in large state spaces global search algorithms are very inefficient
to run because they need to explore the entire space to find the optimal
solution. Local search algorithms, on the other hand, are usually very
efficient and manage to find near-optimal solutions in large state spaces. In
what follows we describe how the beam search algorithm works and how it is
used to produce disentangling circuits.

A search algorithm [100] works by examining the various paths/branches that
are formed from the initial state, trying to find a path that reaches the goal
state. Starting with the initial state at the root of a search tree, the
algorithm sequentially expands state nodes from the tree until it reaches the
goal state. The process of expanding a state node consists of applying all of
the actions from the action space separately to the current state and creating
child state nodes for each of the produced states. This implies that state
nodes that are ready for expansion are leaf nodes, while inner nodes are
already expanded.

A global search algorithm proceeds by expanding every node state in the search
tree. Local search algorithms, and beam search in particular, expand only a
subset of the nodes, and the rest are pruned. At every iteration of the beam
search algorithm, a set of $k$ leaf nodes are selected and expanded, while the
rest of the leaf nodes are marked non-expandable, and are irreversibly
discarded from the tree-search. The value $k$ is called the beam size of the
algorithm. An example execution of the beam search algorithm can be seen in
Fig. 21.

###  C.2 Heuristic functions for evaluating tree nodes

The performance of the beam search algorithm depends on the ranking algorithm
that is used to pick the $k$-best leaf nodes for expansion. In what follows we
describe the ranking heuristic $h$ that we use for producing disentangling
circuits.

The heuristic $h$ is a function that takes as input a pure $L$-qubit quantum
state and returns a real number:

$h:\mathbb{C}^{2^{L}}\rightarrow\mathbb{R}.$

Note that, besides assuming a noise-free environment, we also assume that at
every state node we have access to the full quantum state at this step (the
second assumption needs to be relaxed for the algorithm to be applicable in an
experiment).

Our first choice of a heuristic function is the average entanglement of the
entire state calculated as given in Eq. (1):

$h(|\psi\rangle)=S_{\text{avg}}=\frac{1}{L}\sum_{j=1}^{L}S_{\text{ent}}[\rho^{(j)}],$
(36)

where $\rho^{(j)}$ is the reduced density matrix of qubit $j$ for the pure
state $|\psi\rangle$.

Figure 22: Comparison between the standard beam search algorithm (red) and the
modified ‘qubit-by-qubit’ version (blue) [see text]. (a) Average number of
gates $M$ needed to disentangle a quantum state. (b) Average number of nodes
that the algorithms visits before disentangling a quantum state. The modified
‘qubit-by-qubit’ version requires fewer number of visited nodes while also
producing shorter disentangling protocols. The modification provides an
improvement of around $10\%$ for both metrics. The results are averaged over
100 test simulations with different random initial states. Both algorithms are
evaluated on the same set of states. We render a state disentangled when
$S_{\text{avg}}<\epsilon$. The value of $\epsilon$ for $L=5,6$ is as per Table
1; for $L=8,10$ we set $\epsilon=0.01$.

However, a very simple improvement can be made by noting that once a qubit is
disentangled from the rest, then actions that do not involve this qubit can no
longer influence the single-qubit entanglement entropy
$S_{\text{ent}}[\rho^{(j)}]$ of that qubit. Thus, we focus on disentangling
the rest of the system, effectively reducing the system size. Instead of
aiming to disentangle the entire state as a whole, we run the beam search
algorithm with the goal to disentangle a single fixed qubit, say qubit $j$.
The heuristic function that we use for this run of the algorithm is then:

$h(|\psi\rangle)=S_{\text{ent}}[\rho^{(j)}].$

Once qubit $j$ is disentangled, we arrive at the new state
$|\psi_{1}\rangle=|0_{j}\rangle|\varphi_{1,\dots,j-1,j+1,\dots,L}\rangle$. We
then repeat the procedure but for the state $|\psi_{1}\rangle$ with the goal
to disentangle another qubit, say qubit $k$. The heuristic function that we
use for this second run is:

$h(|\varphi_{1,\dots,j-1,j+1,\dots,L}\rangle)=S_{\text{ent}}[\rho^{(k)}],$

where $\rho^{(k)}$ is the reduced density matrix of qubit $k$ for the state
$|\varphi_{1,\dots,j-1,j+1,\dots,L}\rangle$.

We keep repeating this procedure iteratively until the entire system is
disentangled qubit by qubit.

This strategy provides an improvement to the algorithm of around $10\%$ both
in terms of number of actions and in terms of speed of execution. A comparison
between the standard beam search algorithm using the heuristic function from
Eq. (36) and this modified ‘qubit-by-qubit’ version can be seen in Fig. 22.
Not only does using the modified heuristic function produce shorter sequences,
but the algorithm also runs faster.

Even though the beam search algorithm manages to find near-optimal solutions
to the problem, it is obvious that the time needed to run the algorithm
continues to grow exponentially with the number of qubits. Moreover, the
search algorithm has no learning component, i.e., it has to be re-run for
every different initial state separately. Finally, we mention in passing that
the solution produced by the beam search algorithm need not be physically
optimal: shorter sequences may exist that bring the system into a product
state using fewer gates. For all of these reasons, we focus on models trained
on data that can learn the correct sequence of actions without having to
perform this expensive look-ahead search. In the future, it will be
interesting to combine search algorithms with deep learning [39] to solve the
state disentangling problem.

## References

  * Nielsen and Chuang [2010] M. A. Nielsen and I. L. Chuang,  _Quantum computation and quantum information_ (Cambridge university press, 2010).
  * Raimond  _et al._ [2001] J. M. Raimond, M. Brune, and S. Haroche, Rev. Mod. Phys. 73, 565 (2001).
  * Laflorencie [2016] N. Laflorencie, Physics Reports 646, 1 (2016).
  * Kraus and Cirac [2001] B. Kraus and J. I. Cirac, Phys. Rev. A 63, 062309 (2001).
  * Watts  _et al._ [2015] P. Watts, J. c. v. Vala, M. M. Müller, T. Calarco, K. B. Whaley, D. M. Reich, M. H. Goerz, and C. P. Koch, Phys. Rev. A 91, 062306 (2015).
  * Goerz  _et al._ [2015] M. H. Goerz, G. Gualdi, D. M. Reich, C. P. Koch, F. Motzoi, K. B. Whaley, J. c. v. Vala, M. M. Müller, S. Montangero, and T. Calarco, Phys. Rev. A 91, 062307 (2015).
  * Eldredge  _et al._ [2017] Z. Eldredge, Z.-X. Gong, J. T. Young, A. H. Moosavian, M. Foss-Feig, and A. V. Gorshkov, Phys. Rev. Lett. 119, 170503 (2017).
  * Omran  _et al._ [2019] A. Omran, H. Levine, A. Keesling, G. Semeghini, T. T. Wang, S. Ebadi, H. Bernien, A. S. Zibrov, H. Pichler, S. Choi, _et al._ , Science 365, 570 (2019).
  * Madjarov  _et al._ [2020] I. S. Madjarov, J. P. Covey, A. L. Shaw, J. Choi, A. Kale, A. Cooper, H. Pichler, V. Schkolnik, J. R. Williams, and M. Endres, Nature Physics 16, 857 (2020).
  * Holland  _et al._ [2023] C. M. Holland, Y. Lu, and L. W. Cheuk, Science 382, 1143 (2023).
  * Iqbal  _et al._ [2023] M. Iqbal, N. Tantivasadakarn, T. M. Gatterman, J. A. Gerber, K. Gilmore, D. Gresh, A. Hankin, N. Hewitt, C. V. Horst, M. Matheny, _et al._ , arXiv preprint arXiv:2302.01917 (2023).
  * Bluvstein  _et al._ [2024] D. Bluvstein, S. J. Evered, A. A. Geim, S. H. Li, H. Zhou, T. Manovitz, S. Ebadi, M. Cain, M. Kalinowski, D. Hangleiter, _et al._ , Nature 626, 58 (2024).
  * Terno [1999] D. R. Terno, Phys. Rev. A 59, 3320 (1999).
  * Mor [1999] T. Mor, Phys. Rev. Lett. 83, 1451 (1999).
  * Zhou and Guo [2000] Z.-W. Zhou and G.-C. Guo, Phys. Rev. A 61, 032108 (2000).
  * Bandyopadhyay  _et al._ [1999] S. Bandyopadhyay, G. Kar, and A. Roy, Physics Letters A 258, 205 (1999).
  * Ghosh  _et al._ [2000] S. Ghosh, S. Bandyopadhyay, A. Roy, D. Sarkar, and G. Kar, Phys. Rev. A 61, 052301 (2000).
  * Dehaghani  _et al._ [2024] N. B. Dehaghani, A. P. Aguiar, and R. Wisniewski, arXiv preprint arXiv:2403.16321 (2024).
  * Hauschild  _et al._ [2018] J. Hauschild, E. Leviatan, J. H. Bardarson, E. Altman, M. P. Zaletel, and F. Pollmann, Phys. Rev. B 98, 235163 (2018).
  * Ljubotina  _et al._ [2022] M. Ljubotina, B. Roos, D. A. Abanin, and M. Serbyn, PRX Quantum 3, 030343 (2022).
  * Lu  _et al._ [2023] Y. Lu, P. Shi, X.-H. Wang, J. Hu, and S.-J. Ran, arXiv preprint arXiv:2307.11609 (2023).
  * Shende  _et al._ [2006] V. Shende, S. Bullock, and I. Markov, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 25, 1000–1010 (2006).
  * Chamon  _et al._ [2014] C. Chamon, A. Hamma, and E. R. Mucciolo, Phys. Rev. Lett. 112, 240501 (2014).
  * Shaffer  _et al._ [2014] D. Shaffer, C. Chamon, A. Hamma, and E. R. Mucciolo, Journal of Statistical Mechanics: Theory and Experiment 2014, P12007 (2014).
  * Yang  _et al._ [2017] Z.-C. Yang, A. Hamma, S. M. Giampaolo, E. R. Mucciolo, and C. Chamon, Phys. Rev. B 96, 020408 (2017).
  * Araujo  _et al._ [2024] I. F. Araujo, C. Blank, I. C. S. Araújo, and A. J. da Silva, IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems 43, 161 (2024).
  * Mottonen  _et al._ [2005] M. Mottonen, J. J. Vartiainen, V. Bergholm, and M. M. Salomaa, Quantum Information & Computation 5, 467–473 (2005).
  * Sutton and Barto [2018] R. S. Sutton and A. G. Barto,  _Reinforcement learning: An introduction_ (MIT Press, 2018).
  * Mehta  _et al._ [2019] P. Mehta, M. Bukov, C.-H. Wang, A. G. Day, C. Richardson, C. K. Fisher, and D. J. Schwab, Physics Reports 810, 1 (2019), A high-bias, low-variance introduction to Machine Learning for physicists.
  * Carleo  _et al._ [2019] G. Carleo, I. Cirac, K. Cranmer, L. Daudet, M. Schuld, N. Tishby, L. Vogt-Maranto, and L. Zdeborová, Rev. Mod. Phys. 91, 045002 (2019).
  * Carrasquilla [2020] J. Carrasquilla, Advances in Physics: X 5, 1797528 (2020).
  * Krenn  _et al._ [2023] M. Krenn, J. Landgraf, T. Foesel, and F. Marquardt, Phys. Rev. A 107, 010101 (2023).
  * Giordano and Martin-Delgado [2022] S. Giordano and M. A. Martin-Delgado, Phys. Rev. Res. 4, 043056 (2022).
  * An  _et al._ [2022] Z. An, C. Cao, C.-Q. Xu, and D. L. Zhou, Learning quantum phases via single-qubit disentanglement (2022), arXiv:2107.03542 [quant-ph] .
  * Metz and Bukov [2023] F. Metz and M. Bukov, Nature Machine Intelligence 5, 780–791 (2023).
  * Fösel  _et al._ [2018] T. Fösel, P. Tighineanu, T. Weiss, and F. Marquardt, Phys. Rev. X 8, 031084 (2018).
  * Bukov  _et al._ [2018] M. Bukov, A. G. R. Day, D. Sels, P. Weinberg, A. Polkovnikov, and P. Mehta, Phys. Rev. X 8, 031086 (2018).
  * Bukov [2018] M. Bukov, Phys. Rev. B 98, 224305 (2018).
  * Dalgaard  _et al._ [2020] M. Dalgaard, F. Motzoi, J. J. Sørensen, and J. Sherson, npj Quantum Information 6, 6 (2020).
  * Haug  _et al._ [2020] T. Haug, W.-K. Mok, J.-B. You, W. Zhang, C. E. Png, and L.-C. Kwek, Machine Learning: Science and Technology 2, 01LT02 (2020).
  * Mackeprang  _et al._ [2020] J. Mackeprang, D. B. R. Dasari, and J. Wrachtrup, Quantum Machine Intelligence 2, 5 (2020).
  * Yao  _et al._ [2020] J. Yao, M. Bukov, and L. Lin, in  _Proceedings of The First Mathematical and Scientific Machine Learning Conference_, Proceedings of Machine Learning Research, Vol. 107, edited by J. Lu and R. Ward (PMLR, 2020) pp. 605–634.
  * Yao  _et al._ [2022a] J. Yao, P. Kottering, H. Gundlach, L. Lin, and M. Bukov, in  _Proceedings of the 2nd Mathematical and Scientific Machine Learning Conference_, Proceedings of Machine Learning Research, Vol. 145, edited by J. Bruna, J. Hesthaven, and L. Zdeborova (PMLR, 2022) pp. 1044–1081.
  * Guo  _et al._ [2021] S.-F. Guo, F. Chen, Q. Liu, M. Xue, J.-J. Chen, J.-H. Cao, T.-W. Mao, M. K. Tey, and L. You, Phys. Rev. Lett. 126, 060401 (2021).
  * He  _et al._ [2021a] R.-H. He, R. Wang, S.-S. Nie, J. Wu, J.-H. Zhang, and Z.-M. Wang, EPJ Quantum Technology 8, 29 (2021a).
  * Cao  _et al._ [2022] C. Cao, Z. An, S.-Y. Hou, D. L. Zhou, and B. Zeng, Communications Physics 5, 57 (2022).
  * Porotti  _et al._ [2023] R. Porotti, V. Peano, and F. Marquardt, PRX Quantum 4, 030305 (2023).
  * Porotti  _et al._ [2022] R. Porotti, A. Essig, B. Huard, and F. Marquardt, Quantum 6, 747 (2022).
  * Sivak  _et al._ [2022] V. V. Sivak, A. Eickbusch, H. Liu, B. Royer, I. Tsioutsios, and M. H. Devoret, Phys. Rev. X 12, 011059 (2022).
  * Reuer  _et al._ [2023] K. Reuer, J. Landgraf, T. Fösel, J. O’Sullivan, L. Beltrán, A. Akin, G. J. Norris, A. Remm, M. Kerschbaum, J.-C. Besse, F. Marquardt, A. Wallraff, and C. Eichler, Nature Communications 14, 7138 (2023).
  * Yao  _et al._ [2022b] J. Yao, H. Li, M. Bukov, L. Lin, and L. Ying, in  _Proceedings of Mathematical and Scientific Machine Learning_, Proceedings of Machine Learning Research, Vol. 190, edited by B. Dong, Q. Li, L. Wang, and Z.-Q. J. Xu (PMLR, 2022) pp. 49–64.
  * Zhang  _et al._ [2019] X.-M. Zhang, Z. Wei, R. Asad, X.-C. Yang, and X. Wang, npj Quantum Information 5, 85 (2019).
  * Qiu  _et al._ [2022] Y. Qiu, M. Zhuang, J. Huang, and C. Lee, New Journal of Physics 24, 083011 (2022).
  * Wauters  _et al._ [2020] M. M. Wauters, E. Panizon, G. B. Mbeng, and G. E. Santoro, Phys. Rev. Res. 2, 033446 (2020).
  * Kölle  _et al._ [2024] M. Kölle, T. Schubert, P. Altmann, M. Zorn, J. Stein, and C. Linnhoff-Popien, A reinforcement learning environment for directed quantum circuit synthesis (2024), arXiv:2401.07054 [quant-ph] .
  * Zhang  _et al._ [2020] Y.-H. Zhang, P.-L. Zheng, Y. Zhang, and D.-L. Deng, Phys. Rev. Lett. 125, 170501 (2020).
  * Bolens and Heyl [2021] A. Bolens and M. Heyl, Phys. Rev. Lett. 127, 110502 (2021).
  * Moro  _et al._ [2021] L. Moro, M. G. A. Paris, M. Restelli, and E. Prati, Communications Physics 4, 178 (2021).
  * He  _et al._ [2021b] Z. He, L. Li, S. Zheng, Y. Li, and H. Situ, New Journal of Physics 23, 033002 (2021b).
  * Fösel  _et al._ [2021] T. Fösel, M. Y. Niu, F. Marquardt, and L. Li, arXiv preprint arXiv:2103.07585 (2021).
  * Li  _et al._ [2023] Y. Li, W. Liu, M. Li, and Y. Li, Quantum Information Processing 22 (2023).
  * Quetschlich  _et al._ [2023] N. Quetschlich, L. Burgholzer, and R. Wille, in  _2023 60th ACM/IEEE Design Automation Conference (DAC)_ (2023) pp. 1–6.
  * Pozzi  _et al._ [2022] M. G. Pozzi, S. J. Herbert, A. Sengupta, and R. D. Mullins, ACM Transactions on Quantum Computing 3, 10.1145/3520434 (2022).
  * Herrera-Martí [2022] D. A. Herrera-Martí, Quantum 6, 797 (2022).
  * Chen  _et al._ [2022] Q. Chen, Y. Du, Q. Zhao, Y. Jiao, X. Lu, and X. Wu, Efficient and practical quantum compiler towards multi-qubit systems with deep reinforcement learning (2022), arXiv:2204.06904 [quant-ph] .
  * Nguyen  _et al._ [2023] H. N. Nguyen, F. Motzoi, M. Metcalf, K. B. Whaley, M. Bukov, and M. Schmitt, arXiv preprint arXiv:2311.03684 (2023).
  * Ivanova-Rohling  _et al._ [2024] V. N. Ivanova-Rohling, N. Rohling, and G. Burkard, arXiv preprint arXiv:2402.10559 (2024).
  * Kremer  _et al._ [2024] D. Kremer, V. Villar, H. Paik, I. Duran, I. Faro, and J. Cruz-Benito, arXiv preprint arXiv:2405.13196 (2024).
  * Note [1] Note that the most general product (i.e., zero-entanglement) pure state can be mapped to $|\psi_{\ast}\rangle$ by applying at most $L$ single-qubit rotations. Since these extra operations do not alter the entanglement structure of the state, we will consider them part of $U$.
  * Note [2] We experimented also with ordered pairs but found no improvement in the learning behavior of the agent; however, the computational cost increased considerably.
  * Vaswani  _et al._ [2023] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all you need (2023), arXiv:1706.03762 [cs.CL] .
  * Li  _et al._ [2024] R. Li, J. Du, Z. Qin, S. Zhang, C. Du, Y. Zhou, and Z. Xiao, arXiv preprint arXiv:2401.03400 (2024).
  * Schulman  _et al._ [2017] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, arXiv preprint arXiv:1707.06347 (2017).
  * Wang  _et al._ [2018] J. Wang, S. Paesani, Y. Ding, R. Santagati, P. Skrzypczyk, A. Salavrakos, J. Tura, R. Augusiak, L. Mančinska, D. Bacco, _et al._ , Science 360, 285 (2018).
  * Rietsch  _et al._ [2024] S. Rietsch, A. Y. Dubey, C. Ufrecht, M. Periyasamy, A. Plinge, C. Mutschler, and D. D. Scherer, arXiv preprint arXiv:2404.14865 (2024).
  * Alagic  _et al._ [2020] G. Alagic, C. Majenz, and A. Russell, in  _Advances in Cryptology–EUROCRYPT 2020_ (Springer International Publishing, 2020) pp. 759–787.
  * Koh  _et al._ [2023] J. M. Koh, S.-N. Sun, M. Motta, and A. J. Minnich, Nature Physics 19, 1314 (2023).
  * Gillman  _et al._ [2022] E. Gillman, D. C. Rose, and J. P. Garrahan, arXiv preprint arXiv:2209.14089 (2022).
  * Vidal [2007] G. Vidal, Phys. Rev. Lett. 99, 220405 (2007).
  * Morral-Yepes  _et al._ [2024] R. Morral-Yepes, A. Smith, S. Sondhi, and F. Pollmann, PRX Quantum 5, 010309 (2024).
  * [81] The source code can be found at https://github.com/mgbukov/RL_disentangle/ .
  * Note [3] In general, for two-qubit mixed states the von Neumann entropy of the single-particle density matrix of a given qubit is not a measure of entanglement between the two qubits. However, recalling that the entire $L$-qubit system is in a pure state $|\psi\rangle$, the single-qubit von-Neumann entropy gives the entanglement between the single qubit and all $L-1$ remaining qubits.
  * Jevtic  _et al._ [2012] S. Jevtic, D. Jennings, and T. Rudolph, Phys. Rev. A 85, 052121 (2012).
  * Zhang  _et al._ [2003] J. Zhang, J. Vala, S. Sastry, and K. B. Whaley, Phys. Rev. A 67, 042313 (2003).
  * Shende  _et al._ [2004] V. V. Shende, I. L. Markov, and S. S. Bullock, Phys. Rev. A 69, 062321 (2004).
  * Vatan and Williams [2004] F. Vatan and C. Williams, Phys. Rev. A 69, 032315 (2004).
  * Vidal and Dawson [2004] G. Vidal and C. M. Dawson, Phys. Rev. A 69, 010301 (2004).
  * Note [4] An arbitrary real-valued two-qubit rotation gate $U^{(1,2)}{\in}\text{SO}(4)$ contains at most 2 CNOT gates, since it can be parametrized as  $\displaystyle U^{(1,2)}$ $\displaystyle=$ $\displaystyle Y^{(1)}(\alpha)Y^{(2)}(a)\tmspace+{.2777em}\text{CNOT}^{(1,2)}\times$ $\displaystyle\times Y^{(1)}(\beta)Y^{(2)}(b)\tmspace+{.2777em}\text{CNOT}^{(1,2)}Y^{(1)}(\gamma)Y^{(2)}(c),$ at Euler angles $\alpha,\beta,\gamma,a,b,c\in[0,2\pi)$, where $Y^{(j)}(\alpha)=\exp(-i\alpha S^{y}_{j})$ is a single-qubit rotation about the $y$-axis; this identity follows from the Lie algebra decomposition $\mathfrak{so}(4){=}\mathfrak{so}(3){\oplus}\mathfrak{so}(3)$, together with the defining properties of Euler angles. Real-valued gates with a negative determinant, like the SWAP gate, cannot be decomposed in this way.
  * Zhang  _et al._ [2022] Z. Zhang, H. Zhao, S. Wu, T. Wu, X. Qiao, Z. Gao, R. Agarwal, S. Longhi, N. M. Litchinitser, L. Ge, _et al._ , Nature 612, 246 (2022).
  * Smolin  _et al._ [2012] J. A. Smolin, J. M. Gambetta, and G. Smith, Phys. Rev. Lett. 108, 070502 (2012).
  * Akshay Agrawal and Boyd [2018] S. D. Akshay Agrawal, Robin Verschueren and S. Boyd, Journal of Control and Decision 5, 42 (2018).
  * Diamond and Boyd [2016] S. Diamond and S. Boyd, Journal of Machine Learning Research 17, 1 (2016).
  * Qiskit contributors [2023] Qiskit contributors, Qiskit: An open-source framework for quantum computing (2023).
  * [94] IonQ, https://ionq.com/, 2023.
  * [95] IBM Quantum, https://quantum.ibm.com/, 2023.
  * Note [5] A simple $n$-step bootstrapped estimation of the advantage takes the form $A(s_{t},a_{t})=r_{t}+r_{t+1}+\dots+r_{T-1}+V_{\phi}(s_{T})$.
  * Schulman  _et al._ [2018] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel, High-dimensional continuous control using generalized advantage estimation (2018), arXiv:1506.02438 [cs.LG] .
  * Note [6] In our case, we have $d_{v}=d_{k}=d_{q}=d_{\text{model}}=D_{\text{qkv}}$. However, the expressions here hold in general, i.e., for any $d_{k}$, $d_{v}$.
  * Pascanu  _et al._ [2012] R. Pascanu, T. Mikolov, and Y. Bengio, CoRR abs/1211.5063 (2012), 1211.5063 .
  * Russell and Norvig [2016] S. J. Russell and P. Norvig,  _Artificial intelligence: a modern approach_ (Malaysia; Pearson Education Limited, 2016) ISBN: 978-0-13-604259-4.