# Learning quantum phases via single-qubit disentanglement

Zheng An  Institute of Physics, Beijing National Laboratory for Condensed
Matter Physics,
Chinese Academy of Sciences, Beijing 100190, China  School of Physical
Sciences, University of Chinese Academy of Sciences, Beijing 100049, China
Chenfeng Cao  Department of Physics, The Hong Kong University of Science and
Technology, Clear Water Bay, Kowloon, Hong Kong, China  Cheng-Qian Xu
Institute of Physics, Beijing National Laboratory for Condensed Matter
Physics,
Chinese Academy of Sciences, Beijing 100190, China  School of Physical
Sciences, University of Chinese Academy of Sciences, Beijing 100049, China  D.
L. Zhou<EMAIL_ADDRESS>Institute of Physics, Beijing National Laboratory
for Condensed Matter Physics,
Chinese Academy of Sciences, Beijing 100190, China  School of Physical
Sciences, University of Chinese Academy of Sciences, Beijing 100049, China
Collaborative Innovation Center of Quantum Matter, Beijing 100190, China
Songshan Lake Materials Laboratory, Dongguan, Guangdong 523808, China

(August 27, 2024; August 27, 2024)

###### Abstract

Identifying phases of matter is a complicated process, especially in quantum
theory, where the complexity of the ground state appears to rise exponentially
with system size. The entanglement of quantum many-body systems exhibits
different complex structures in different phases. It is a natural way to
distinguish different quantum phases by detecting different entanglement
structures. As a method that works directly on the detection of the
entanglement structure, disentanglement can provide factual information on the
entanglement structure. In this work, we follow a radically different approach
to identifying quantum phases: we utilize disentanglement to determine the
entanglement structures under different phases to distinguish the phases of
Ising spin chain systems. Here, we propose a reinforcement learning (RL)
approach to finding phase transitions based on the performance of the RL-
designed disentangling circuit. Further, our method uses limited local
operations and one qubit measurement, making the quantum circuit scalable and
easily extended to large-sized systems. We demonstrate the success of this
method on the quantum phase transition in the transverse field Ising model
(TFIM) and the XXZ model. Moreover, we find that the RL agent learns the
Kramers-Wannier duality on entanglement structures under the TFIM. This study
sheds light on characterizing quantum phases with the entanglement structures
of quantum many-body systems.

Entanglement is an important quantum mechanical phenomenon where the many-body
quantum states cannot be described separately [1]. Motivated by the
development of quantum resource theory and quantum computation, entanglement
has become a fundamental resource in quantum technologies such as quantum
teleportation, quantum cryptography, and quantum metrology [2, 3, 4, 5, 6].
The manipulation of entanglement, especially the destruction of entanglement
via general operations, is still under study [7, 8].

A phase diagram depicts qualitative changes in many-body systems as a function
of physical system parameters. The objective of a phase classifier is to
determine the phase diagram by analyzing some measurement results of
observables. For example, in the Landau theory of phase transitions, only the
average measurement value of one local variable, the so-called local order
parameter, is sufficient to determine the phase diagram. In fact, a local
order parameter’s discontinuity or one of its derivatives indicates a phase
transition. Further, quantum entanglement has played a significant role in
characterizing many-body quantum systems [9, 10]. In particular, quantum
entanglement has been an essential tool in characterizing quantum phase
transition and fingerprinting topological orders [11, 12, 13, 14]. It is
believed that quantum entanglement plays a crucial role in understanding
quantum phase transition: different structures in quantum entanglement
characterizes different quantum phases.

Among various studies of quantum entanglement, we found disentanglement has a
strong correlation with entanglement structure. Disentanglement is a process
in which an initial entangled state of a composite system is transformed into
a separable state without affecting the reduced density matrices of the
subsystems. However, a universal disentangling machine does not exist [15, 16,
17]. Nevertheless, reducing the constraint that the reduced density matrices
of subsystems are perfectly unaffected, local operations can realize
approximate disentanglement [18, 19, 20]. The central idea of disentanglement
is to decouple the correlations among the subsystems. Thus, it is natural to
investigate whether a local unitary transformation realized by a quantum
circuit can destroy the local entanglement within a quantum state. Further, we
want to investigate the relationship between the entanglement structure and
the circuit structure, especially for the states in the same phase.

One of the main objects in the variational quantum algorithm framework is the
construction of the unitary transformation. The variational quantum circuit is
a powerful tool to approximate a known or unknown unitary evolution, and
relevant hybrid quantum-classical algorithms have been demonstrated
experimentally on noisy intermediate-scale quantum (NISQ) devices [21, 22, 23,
24]. Generally, we can encode any optimization problem, e.g., MaxCut, to a
specific Hamiltonian with the ground state being the solution [25], then
optimize a parameterized quantum circuit to find the ground state and ground
energy. Variational quantum algorithms have been designed for a wide range of
problems, such as quantum chemistry [26, 27, 28], quantum data compression
[29, 30, 31], quantum fidelity estimation [32, 33], state diagonalization [34,
35], Hamiltonian diagonalization [36].

The variational quantum algorithm usually rely on a fixed circuit structure,
which may not be suited for the disentanglement tsak. The main challenge of
designing a desired disentangling circuit is to discover suitable sequences of
unitary gates; the size of the search space and the lack of simple strategies
in many situations make this a challenging problem. Reinforcement learning
(RL) is an area of machine learning concerned with finding optimal strategies
for complex decision-making problems [37]. RL has recently been employed in
broad applications, such as video games [38, 39], board games [40, 41, 42],
and neural architecture search [43]. Primarily, it has been widely applied to
a wide array of physics problems, such as quantum control [44, 45, 46],
quantum error correction [47], and quantum circuit optimization [48, 49, 50].
Also, quantum machine learning, especially quantum reinforcement learning, may
produce quantum advantages with machine learning [51, 52, 53]. Those successes
naturally raise the question of how much quantum circuit design might benefit
from reinforcement learning.

In this letter, we discuss a novel method of classifying phase transitions,
i.e., designing the disentangle ansatz of the ground states to identify
different phases of Ising spin chain systems (Fig 1). To this end, we apply
deep reinforcement learning to variational quantum circuit design. For more
practical purposes, the goal of the circuit is to disentangle one qubit from
the rest of the systems and the operation space is limited to nearest or next-
nearest qubits. To explore the optimal strategy of the problem in this
scenario, we use the Deep Q-network (DQN) [38] to study this problem.

Figure 1: Classifying quantum phases via single-qubit disentanglement. The
local entanglement of the ground state of a quantum many body system may
contain many complex structures, and some of the main structures may be
closely related to the quantum phase (Left). The task of the RL agent is to
eliminate the local entanglement of the ground state under a given quantum
phase through local operation and local observation. Therefore, we can infer
the main entanglement structure in the local entanglement and further classify
the quantum phase (Right).

##  I Results

Variational quantum disentangler assisted by deep Q-learning. The
disentangling circuit design is a challenging problem, consisting of designing
a special unitary transformation as a finite sequence of unitary operators
$A_{t}$ is chosen from a universal set of gates $\mathcal{G}$. In this work,
we ask the agent to design a disentanglement unitary operation $U$, which can
erase a target local entanglement entropy. Therefore, the goal of the agent is
to find a unitary gate $U$, resulting from the composition of the gates in the
sequence, that sufficiently erases the target entanglement entropy.

We use the entanglement entropy $S$ to measure how much entanglement in the
ground state of quantum systems. When a bipartite quantum system $AB$ is in a
pure state, the entanglement entropy is calculated from the reduced density
matrix $\rho_{A}$ or $\rho_{B}$ according to the formula

$S\equiv-\operatorname{tr}\left(\rho_{A}\log_{2}\rho_{A}\right)=-\operatorname{tr}\left(\rho_{B}\log_{2}\rho_{B}\right).$
(1)

When either subsystem $A$ or $B$ is a spin-$\frac{1}{2}$ system, $S$ varies
from $S=0$ (product state) to $S=1$ (maximally entangled state). For the
ground state of the two studied models, we regard one site as subsystem $A$
and the rest as subsystem $B$, which means the measurement is performed on
only one qubit.

(a)

(b)

Figure 2: The deep reinforcement learning (DRL) architecture. (a) The DRL
environment can be described as a quantum circuit modeled by the optimized
variational quantum circuit $U^{opt}_{t}$ with initial ground state
$|g\rangle$ at each episode. At each time step $t$, the agent receives the
current state of $U_{t}$ and based on that information, it chooses from the
gate set $\mathcal{G}$ the next gate $A_{t}$ to apply on the quantum circuit.
Therefore, the environment returns the real-valued reward $R_{t}$ to the
agent, which is a function of the entanglement entropy $S$. The policy $\pi$
of the agent is encoded in a deep neural network. (b) State representation for
RL agent. At each time-step, the agent receives as input a vector made by the
architecture of the quantum circuit $U_{t}$. An array represents the circuit,
and each element in this array represents a quantum gate in the given gate
set.

In our framework, the environment consists of a quantum circuit that starts as
the identity at the beginning of each episode. It is built incrementally at
each step by the agent, choosing a gate from $\mathcal{G}$ according to the
policy $\pi$ encoded in the deep neural network, as shown in Fig. 2(a).

In particular, we consider a gate set $\mathcal{G}$ consisting of single-qubit
rotation (along with x, y, or z-direction), Hadamard, and controlled-NOT
(CNOT) gates. Together, they form a universal gate set. Whereas continuous
variables parameterize rotation gates, respectively, the CNOT and Hadmard are
fixed gates. All of the parameters of single-qubit rotation gates are
initialized to $\pi$. Notice that the operation space is limited, which
performs on the nearest qubits or next-nearest qubits of the target qubit.

The reward of the agent received in each step is

$R_{t}=\begin{cases}0,&t\in\\{0,1,\dots,N-1\\}\\\ \mathcal{R},&t=N\end{cases}$
(2)

where $\mathcal{R}$ is the modified reward function for our learning task. The
agent will get a reward not immediately, but at time $N$. In particular, we
define $\mathcal{R}$ as

$\mathcal{R}=\begin{cases}0,&\text{ if }S_{RL}\geq S_{0}\\\
\frac{S_{0}-S_{RL}}{S_{0}},&\text{otherwise}\end{cases}$ (3)

where $S_{RL}$ is the final result entanglement entropy given by our RL-
assistant quantum algorithm, and $S_{0}$ is the ground-state entanglement
entropy given by the physical system.

In this work, we exploit Deep Q-learning algorithms to train the agents,
depending on the reward function. The details of the algorithm are described
in supplementary materials. For the classical optimizer, we employ the BFGS
algorithm implemented in SciPy [54]. This helps us determine the parameters on
each gate from $U_{t}$ to $U^{opt}_{t}$.

In the following, we propose a training scheme to identify phases and the
critical point. This scheme is the main result of this work. We suppose that
two states depend on a parameter that is $a$ and $b$, and we assume that there
exists a critical point $a<c<b$ such that the states can be classified into
two groups. However, we do not know the value of $c$. Then, we train our RL
agent with the two states to get the disentangling circuits until convergence
with circuit layer $p$. Next, we evaluate the performance of the disentangling
circuits on the parameter range$(a,b)$ and refer to its total performance
($S_{a}$ and $S_{b}$), concerning the proposed critical point $c$, as
$S_{a}(c)$ and $S_{b}(c)$. We will show that the performance of those two RL-
ansatz disentangling circuits with deep layers is opposite, with the cross
point at the correct critical point $c$, i.e., $S_{a}(c)=S_{b}(c)$.

The performance of the circuit can be understood as follows. We assume that
the entanglement has two different structures in the regimes below $c$ (phase
$\mathcal{A}$) and above $c$ (phase $\mathcal{B}$) and that the circuits can
distinguish and erase them. We refer to these different structures as
features. When we set the states on the parameter range$(a,c^{\prime})$ where
$c^{\prime}<c$, the circuit distinguishes the feature of phase $\mathcal{A}$
and thus correctly erases those entanglements. When in range$(c^{\prime},c)$,
in the phase $\mathcal{A}$, the circuit sees states with different features,
but the majority feature is the same feature from $a$ to $c^{\prime}$. The
$S_{a}$ will increase, which means some entanglements cannot be eliminated.
When in range$(c,b)$, the majority feature has changed, and the $S_{a}$ will
rapidly rise until saturated. The situation of $S_{b}$ is precisely the
opposite of $S_{a}$.

If phases determine the entanglement structures, the number of features of the
two phases will be the same when they are at the critical point $c$, i.e.,
$S_{a}(c)=S_{b}(c)$. On the contrary, if $S_{a}(c)\neq S_{b}(c)$, then the
intersection must not be the critical point $c$ because the phase from the
cross point to the critical point will be another phase.

Transverse field Ising model. We first apply our method to one-dimensional
transverse-field Ising model. The Hamiltonian for the Ising model on a 1D
lattice with N sites in a transverse field is given by

$H=-\sum_{j=0}^{N-1}\left(\lambda\sigma_{j}^{x}\sigma_{j+1}^{x}+\sigma_{j}^{z}\right),$
(4)

where $\sigma_{j}^{a}$ is the $a$ component of Pauli matrix ($a=x,y$ or $z$)
at site $j$, and $\lambda$ is the inverse strength of the external field. We
assume the periodic boundary conditions, i.e., the $N$th site is identified
with the $0$th site. As is well known, there is a quantum phase transition at
the critical point $\lambda=1$ for the transverse Ising model, which means the
entanglement properties of the ground state changes dramatically when crossing
this critical point.

When $\lambda$ approaches zero, the TFIM ground state becomes a product of
spins pointing in the positive $z$ direction,

$|\psi_{\text{TFIM}}\rangle_{\lambda\rightarrow 0}=|0\rangle^{\otimes N},$ (5)

where $|0\rangle$ and $|1\rangle$ denote the single spin state with spin up
and with spin down along $z$-direction respectively.

In the $\lambda\rightarrow\infty$ limit, the ground state becomes the GHZ
state [55]:

$|\psi_{\text{TFIM}}\rangle_{\lambda\rightarrow\infty}=\frac{1}{\sqrt{2}}(|+\rangle^{\otimes
N}+|-\rangle^{\otimes N}),$ (6)

where $|+\rangle$ and $|-\rangle$ denote the single spin state with spin up
and with spin down along $x$-direction respectively.

Under the above two limits, the entanglement entropy of the ground state
behaves as

$\displaystyle S_{\lambda\rightarrow 0}$ $\displaystyle=0,$ (7) $\displaystyle
S_{\lambda\rightarrow\infty}$ $\displaystyle=1.$ (8)

At the critical point, $\lambda=1$, there is a phase transition in the ground
state structure in the thermodynamic limit [56]. The correlation function
$\langle\sigma_{i}^{\alpha}\sigma_{j}^{\beta}\rangle-\langle\sigma_{i}^{\alpha}\rangle\langle\sigma_{j}^{\beta}\rangle$
decays polynomially as a function of separation at this point, while for all
other values of $\lambda$, this decay is exponential.

Figure 3:  Entanglement entropy versus $\lambda$ for 8-site TFIM. $P$ is the
number of circuit layers and $|\psi_{\text{TFIM}}\rangle$ denotes the ground
state of the model. The solid line denotes the disentangling circuit training
from $\lambda=0.5$; the dashed line denotes the disentangling circuit training
from $\lambda=1.5$. The dash-dotted line denotes the critical point at
$\lambda=1$. We sample 11 points of TFIM ground states with
$\lambda=0.5,0.6,0.7,0.8,0.9,1,1.1,1.2,1.3,1.4,1.5$.

In Fig. 3, we show the results of the optimal variational algorithm to
disentangle the ground state with different values of $\lambda$. We take two
ground-states from two phases of the system to get the RL-ansatz quantum
circuit. The operation space is set to the nearest qubits, i.e., three qubits.

The performance of the designed algorithm is proportional to the circuit layer
$p$. As the RL-ansatz disentangling circuit with one layer ($p=1$), the
transformation space is not large enough to cut the correlation with the two
subsystems, so different training samples have the same results. However,
unlike the raw local entanglement entropy case, the disentangle circuit
entropy shows three regions with different values of $\lambda$. In both cases,
the entanglement entropy rises quickly from a small number from point
$\lambda=0.5$ to point $\lambda=0.9$ and then slowly rises to the maximum
value around the critical point $\lambda=1$. The situation after the critical
point $\lambda=1$ is precisely the opposite of before. So we can divide the
space of $\lambda$ into “fast-slow-fast” three regions according to the speed
of entropy change. We find that the most significant entanglement is present
in the parameter region close to the critical point. The slow region
corresponds with the quantum critical region [57]. This indicates that the
local quantum entanglement has a deep relation with the phase transition
region.

However, with the layer number larger than one, the results show different
structures of circuits with different phases of $\lambda$. As Fig. 3 shows,
the designed circuit structures with two layers have two optimal circuits in
different phases. Also, in this case, the characteristic is obtained, and we
detect the transition at $\lambda=1$. We confirm that the results with $p=2$
are indeed convergence (see Supplementary materials, Fig. S3).

Understanding Agent Actions. To confirm what agents have learned from
disentanglement under TFIM, we first determine the optimal local
disentanglement in theory. We proved that the minimum entropy $\rho_{P}$
decreased by local unitary operation is determined by the initial state
$\rho_{RP}$ of the region $RP$, on which the unitary operation acts. More
specifically, we have the following theorem.

###### Theorem 1

Suppose the eigen-decomposition of the state $\rho_{RP}$ is
$\rho_{RP}=\sum_{i=1}^{d_{P}d_{R}}p_{i}\ket{\psi_{i}}\bra{\psi_{i}}$, where
$d_{P}$ ($d_{R}$) is the dimension of subsystem $P$ ($R$). Without loss of
generality, we assume these eigenvalues are in decreasing order, i.e.,
$p_{i}\geq p_{i+1}$. By performing a local unitary operation $U$ on the $RP$,
the minimal entanglement entropy of the subsystem $P$ is

$\min_{U}S(\rho_{P}(U))=-\sum_{m}q_{m}{\rm\log_{2}}q_{m},$ (9)

where $\rho_{P}(U)={\rm Tr}_{R}\left[U\rho_{R}PU^{\dagger}\right]$,
$S(\sigma)=-{\rm Tr}\left[\sigma{\rm\log_{2}}\sigma\right]$ is the Von-Neumann
entropy of state $\sigma$, and $q_{m}=\sum_{j=1}^{d_{R}}p_{(m-1)d_{R}+j}$.

###### Proof.

The proof is given in the given in Supplementary materials, Section S-I. ∎

Although Therorem 1 gives the optimal local disentanglement theoretically, it
does not imply that we can distinguish quantum phases by directly applying the
theorem. In fact, the relation between disentanglement and quantum phase is
subtle. Because the optimal solution will destroy all entanglement structures
equally, the minimal entanglement entropy in our problem is so small in both
phases that it can not be used to distinguish the phases. However, the RL
agent often find a local minimum but not a global one, which leads it to
destroy the dominant entangled structure under a specific phase. Therefore,
even if RL cannot give an optimal strategy in many cases, it may fits the
problem.

Interestingly, we found the RL agent finds the optimal solution to disentangle
the local entanglement of the nearest two-body in the phase of $\lambda>1$
(Fig 4(a)), which is exactly the solution of Theorem 1 when $RP$ is restricted
to two nearest spins. At first glance, the RL-ansatz disentangling circuit in
the phase of $\lambda<1$ may be a suboptimal disentanglement of a complex
three-body local entanglement. However, we realize that there is a well known
Kramers-Wannier duality in TFIM [58, 59]. For the model, the self-dual point
is the phase transition point $\lambda=1$. When we perform the duality
transformation which defines new Pauli operators $\mu_{i}^{z}$ and
$\mu_{i}^{x}$ in a dual lattice

$\displaystyle\mu_{j}^{x}=\prod_{i\leq j}\sigma_{i}^{z}$ (10)
$\displaystyle\mu_{j}^{z}=\sigma_{j+1}^{x}\sigma_{j}^{x},$ (11)

and the original Hamiltonian can be written as

$H=-\sum_{j=0}^{N-1}\left(\mu_{j}^{x}\mu_{j+1}^{x}+\lambda\mu_{j}^{z}\right).$
(12)

We found the optimal disentanglement of the nearest neighbor two-body with
duality just coincides with the RL-ansatz in the phase of $\lambda<1$ (Fig
4(b)).

(a)

(b)

Figure 4: Entanglement entropy versus $\lambda$ with the RL-ansatz
disentangling circuit and the nearest two-body optimal disentanglement for
TFIM.

These results suggest that the agent learns that the duality of the
Hamiltonian, which is an important feature in classifying quantum phases. This
further shows that the RL agent has found the main local entanglement
structure between different phases.

XXZ model. The second model we apply our method to is the XXZ model. The
Hamiltonian of the antiferromagnetic XXZ model on a 1D lattice with N sites is

$H=\sum_{j=0}^{N-1}\left(\sigma_{j}^{x}\sigma_{j+1}^{x}+\sigma_{j}^{y}\sigma_{j+1}^{y}+\Delta\sigma_{j}^{z}\sigma_{j+1}^{z}\right),$
(13)

where $\Delta$ is a dimensionless parameter characterizing the anisotropy of
the model. The XXZ spin chain has been widely studied [60, 61, 62, 63, 64].
For the XXZ model, there exist two different phases at the ground state where
$\Delta>0$, i.e., metallic phase, $0<\Delta\leq 1$, and antiferromagnetic
phase, $\Delta>1$, which is resulted from that the former is gap-less while
the latter is gapful. The critical point of quantum phase transition locates
at the isotropic point $\Delta=1$ which is the Kosterlitz-Thouless quantum
phase transitions (KT-QPTs).

Figure 5:  Entanglement entropy versus $\Delta$ for 8-site XXZ model. $P$ is
the number of circuit layers, and $|\psi_{\text{XXZ}}\rangle$ denotes the
ground state of the model. The solid line denotes the disentangling circuit
training from $\Delta=0.5$; the dashed line denotes the disentangling circuit
training from of $\Delta=1.5$. The dash-dotted line denotes the critical point
at $\Delta=1$. We sample 11 points of XXZ model ground states with
$\Delta=0.5,0.6,0.7,0.8,0.9,1,1.1,1.2,1.3,1.4,1.5$.

First, we perform our learning scheme, using the same condition for the TFIM
case. However, we found the results do not convergent. So we set the operation
space to next-nearest qubits, i.e., five qubits.

In Fig. 5, we show the results of the optimal circuits to disentangle the
ground state with different values of $\Delta$. Similar to the TFIM model
mentioned above, the performance of the designed circuits is proportional to
the circuit depth $p$. As the RL-ansatz disentangling circuit with one layer
($p=1$), this case has a universal circuit structure for different values of
$\Delta$, which is similar to the TFIM result.

Nevertheless, unlike the TFIM case, the performance is different in the two-
layer case. From Fig. 5, we observe interesting differences for disentangling
circuit results at the crossing points. The intersection is far from the
critical point because the number of layers of the quantum circuit is just
between too shallow and enough layers. So we can see that there are two
optimal solutions, but the intersection of the two is not the critical point.
The above situation indicated that the ground-state entanglement structure of
the XXZ model might be more complex than the TFIM case.

When $p=3$, the optimal disentangling circuit is convergent (see Supplementary
materials, Fig. S4), and the result is consistent with what we discussed in
the former section. Also, we detect the transition at $\Delta=1$.

The robustness of the RL-ansatz disentangling circuit. To confirm that the
training scheme indeed extracts non-trivial structures from the input state,
we have checked the performance from the training scheme when the quantum
circuit is trained on different initial values and states. To show the
robustness of the RL-ansatz quantum circuit, we examined our method with two
example cases: (i) the initial parameters of rotation gate extracted from the
uniform distribution with mean value $\\{0,\pi/2,\pi\\}$ and (ii) the
disentangling circuit training from different ground states of parameters
($\lambda$ or $\Delta$).

(a)

(b)

(c)

Figure 6: Entanglement entropy versus $\lambda$ with the RL-ansatz
disentangling circuit for TFIM with $N=8$ of (i) under different initial
values. (A): uniform distribution with mean value $0$ (B): uniform
distribution with mean value $\pi/2$ (C): uniform distribution with mean value
$\pi$. The solid line denotes the disentangling circuit training from
$\lambda=0.5$; the dashed line denotes the disentangling circuit training from
$\lambda=1.5$.

(a)

(b)

(c)

Figure 7: Entanglement entropy versus $\Delta$ with the RL-ansatz
disentangling circuit for XXZ model with $N=8$ of (i) under different initial
values. (A): uniform distribution with mean value $0$ (B): uniform
distribution with mean value $\pi/2$ (C): uniform distribution with mean value
$\pi$. The solid line denotes the disentangling circuit training from
$\Delta=0.5$; the dashed line denotes the disentangling circuit training from
$\Delta=1.5$.

(a)

(b)

Figure 8: Entanglement entropy versus $\lambda$ with the RL-ansatz
disentangling circuit for TFIM with $N=8$ of (ii) under different training
states. (A): ground state with $\lambda=0.6,1.3$ (B): ground state with
$\lambda=0.7,1.4$. The solid line denotes the disentangling circuit training
from $\lambda=0.6,0.7$; the dashed line denotes the disentangling circuit
training from $\lambda=1.3,1.4$.

(a)

(b)

Figure 9: Entanglement entropy versus $\Delta$ with the RL-ansatz
disentangling circuit for XXZ model with $N=8$ of (ii) under different
training states. (A): ground state with $\Delta=0.6,1.3$ (B): ground state
with $\Delta=0.7,1.4$. The solid line denotes the disentangling circuit
training from $\Delta=0.6,0.7$; the dashed line denotes the disentangling
circuit training from $\Delta=1.3,1.4$.

As Fig. 6 and Fig. 8 show, the algorithm’s performance with the TFIM is robust
under different conditions. The same conclusion can be drawn in Fig. 7 and
Fig. 9 with the XXZ model. In other words, we may also look for the phase
transition at which the training is most independent of chosen conditions.

Scalability of the RL-ansatz disentangling circuit. In previous discussions,
there was no apparent relationship between the entanglement structure and the
size of the system. Further, our method only operates on the local space of
the target qubit, which is easy to expand the scale of the system. So we want
to study the effect of disentangling circuit training in the small system on
an extensive system.

In Fig. 10 and Fig. 11, we show the results of the disentangling circuit
trained in 8-qubit systems on N-qubit problems with $N>10$. For the RL-
designed disentangling circuit, the circuit is obtained by a training process
on a problem with qubit number $N=8$ then applied to problems having a larger
number of qubits ($N=10,12,14$). The results show that our solution is not
affected by the size of the system. This further implies that the entanglement
structure is not affected by the size but is more closely related to the
quantum phase.

(a)

(b)

(c)

Figure 10: Performance of RL-designed disentangling circuits applied to TFIM
with different numbers of qubits. In this plot, the RL-designed circuits is
obtained by training on the problem with qubit number $N=8$ then applied to
problems with a large number of qubits, (A):$N=10$ (B):$N=12$ (C):$N=14$.

(a)

(b)

(c)

Figure 11: Performance of RL-designed disentangling circuits applied to XXZ
model with different numbers of qubits. In this plot, the RL-designed circuits
is obtained by training on the problem with qubit number $N=8$ then applied to
problems with a large number of qubits, (A):$N=10$ (B):$N=12$ (C):$N=14$.

##  II Conclusions and Outlook

In this work, we show how to identify quantum phase transition via a single-
qubit disentanglement. The RL-ansatz on disentangling circuits found by the
method is capable of perfectly distinguishing quantum phases in the TFIM and
the XXZ model. Further, for the TFIM we theoretically explain the results
learned by the RL agent: a $2$-body optimal disentanglement when $\lambda>1$
and its Kramers-Wannier duality when $\lambda<1$, which allows the
characterization of the quantum phase transition in the TFIM. According to the
similarity in the numerical results learned between the TFIM and the XXZ
model, we conjecture that some similar theoretical explanation exists for the
XXZ model. Our results show a new perspective to study the relation between
entanglement structures and quantum phase transition.

We have proposed an RL-based framework to design a particular variational
quantum circuit. This framework fits our problem very well. It can quickly
find the main entanglement structure under the quantum phase and eliminate it.
Using the advantage of RL to identify the central object quickly, we can
expand the scope of RL in quantum information and study more complex systems
without the need for optimal global solutions. Also, the current approach is
flexible enough to be applied to different problems, such as quantum circuit
optimization, ground state preparation, and novel quantum algorithm design.

In summary, we associate the local entanglement structure of a quantum phase
with an optimal single-qubit disentangling circuit. We show that the
disentangling circuit can be used to characterize different phases in two
typical models. We expect that our approach may be extended to characterize
more exotic quantum phases, e.g., a phase with topological order and a spin
liquid. It provides a universal way to quantify entanglement structures of
quantum phases.

Acknowledgments.– This work is supported by the National Key Research and
Development Program of China (Grant No. 2021YFA0718302 and No.
2016YFA0300603), NSF of China (Grants No. 11775300 and No. 12075310), and the
Strategic Priority Research Program of Chinese Academy of Sciences (Grant No.
XDB28000000).

## References

  * Einstein  _et al._ [1935] A. Einstein, B. Podolsky, and N. Rosen, Phys. Rev. 47, 777 (1935).
  * Bouwmeester  _et al._ [1997] D. Bouwmeester, J.-W. Pan, K. Mattle, M. Eibl, H. Weinfurter, and A. Zeilinger, Nature 390, 575 (1997).
  * Furusawa  _et al._ [1998] A. Furusawa, J. L. Sørensen, S. L. Braunstein, C. A. Fuchs, H. J. Kimble, and E. S. Polzik, Science 282, 706 (1998).
  * Karlsson and Bourennane [1998] A. Karlsson and M. Bourennane, Phys. Rev. A 58, 4394 (1998).
  * Gisin  _et al._ [2002] N. Gisin, G. Ribordy, W. Tittel, and H. Zbinden, Rev. Mod. Phys. 74, 145 (2002).
  * Jennewein  _et al._ [2000] T. Jennewein, C. Simon, G. Weihs, H. Weinfurter, and A. Zeilinger, Phys. Rev. Lett. 84, 4729 (2000).
  * Bennett  _et al._ [1996] C. H. Bennett, D. P. DiVincenzo, J. A. Smolin, and W. K. Wootters, Phys. Rev. A 54, 3824 (1996).
  * Vedral and Plenio [1998] V. Vedral and M. B. Plenio, Phys. Rev. A 57, 1619 (1998).
  * Laflorencie [2016] N. Laflorencie, Physics Reports 646, 1 (2016), quantum entanglement in condensed matter systems.
  * Amico  _et al._ [2008] L. Amico, R. Fazio, A. Osterloh, and V. Vedral, Rev. Mod. Phys. 80, 517 (2008).
  * Chen  _et al._ [2010] X. Chen, Z.-C. Gu, and X.-G. Wen, Phys. Rev. B 82, 155138 (2010).
  * Osborne and Nielsen [2002] T. J. Osborne and M. A. Nielsen, Phys. Rev. A 66, 032110 (2002).
  * Vidal  _et al._ [2003] G. Vidal, J. I. Latorre, E. Rico, and A. Kitaev, Phys. Rev. Lett. 90, 227902 (2003).
  * Osterloh  _et al._ [2002] A. Osterloh, L. Amico, G. Falci, and R. Fazio, Nature 416, 608 (2002).
  * Terno [1999] D. R. Terno, Phys. Rev. A 59, 3320 (1999).
  * Mor and Terno [1999] T. Mor and D. R. Terno, Phys. Rev. A 60, 4341 (1999).
  * Mor [1999] T. Mor, Phys. Rev. Lett. 83, 1451 (1999).
  * Ghosh  _et al._ [2000] S. Ghosh, S. Bandyopadhyay, A. Roy, D. Sarkar, and G. Kar, Phys. Rev. A 61, 052301 (2000).
  * Zhou and Guo [2000] Z.-W. Zhou and G.-C. Guo, Phys. Rev. A 61, 032108 (2000).
  * Yu  _et al._ [2004] Y.-f. Yu, J. Feng, X.-q. Zhou, and M.-s. Zhan, International Journal of Theoretical Physics 43, 21 (2004).
  * Peruzzo  _et al._ [2014] A. Peruzzo, J. McClean, P. Shadbolt, M.-H. Yung, X.-Q. Zhou, P. J. Love, A. Aspuru-Guzik, and J. L. O’Brien, Nature Communications 5, 4213 (2014).
  * Preskill [2018] J. Preskill, Quantum 2, 79 (2018).
  * Cerezo  _et al._ [2020a] M. Cerezo, A. Arrasmith, R. Babbush, S. C. Benjamin, S. Endo, K. Fujii, J. R. McClean, K. Mitarai, X. Yuan, L. Cincio, _et al._ , arXiv preprint arXiv:2012.09265 (2020a).
  * Bharti  _et al._ [2021] K. Bharti, A. Cervera-Lierta, T. H. Kyaw, T. Haug, S. Alperin-Lea, A. Anand, M. Degroote, H. Heimonen, J. S. Kottmann, T. Menke, _et al._ , arXiv preprint arXiv:2101.08448 (2021).
  * Farhi  _et al._ [2014] E. Farhi, J. Goldstone, and S. Gutmann, arXiv preprint arXiv:1411.4028 (2014).
  * McClean  _et al._ [2014] J. R. McClean, R. Babbush, P. J. Love, and A. Aspuru-Guzik, J. Phys. Chem. Lett. 5, 4368 (2014).
  * Kandala  _et al._ [2017] A. Kandala, A. Mezzacapo, K. Temme, M. Takita, M. Brink, J. M. Chow, and J. M. Gambetta, Nature 549, 242 (2017).
  * Barkoutsos  _et al._ [2018] P. K. Barkoutsos, J. F. Gonthier, I. Sokolov, N. Moll, G. Salis, A. Fuhrer, M. Ganzhorn, D. J. Egger, M. Troyer, A. Mezzacapo, S. Filipp, and I. Tavernelli, Phys. Rev. A 98, 022322 (2018).
  * Romero  _et al._ [2017] J. Romero, J. P. Olson, and A. Aspuru-Guzik, Quantum Science and Technology 2, 045001 (2017).
  * Bondarenko and Feldmann [2020] D. Bondarenko and P. Feldmann, Physical review letters 124, 130502 (2020).
  * Cao and Wang [2021] C. Cao and X. Wang, Physical Review Applied 15, 054012 (2021).
  * Cerezo  _et al._ [2020b] M. Cerezo, A. Poremba, L. Cincio, and P. J. Coles, Quantum 4, 248 (2020b).
  * Chen  _et al._ [2020] R. Chen, Z. Song, X. Zhao, and X. Wang, arXiv preprint arXiv:2012.05768 (2020).
  * LaRose  _et al._ [2019] R. LaRose, A. Tikku, É. O’Neel-Judy, L. Cincio, and P. J. Coles, npj Quantum Information 5, 1 (2019).
  * Cerezo  _et al._ [2020c] M. Cerezo, K. Sharma, A. Arrasmith, and P. J. Coles, arXiv preprint arXiv:2004.01372 (2020c).
  * Commeau  _et al._ [2020] B. Commeau, M. Cerezo, Z. Holmes, L. Cincio, P. J. Coles, and A. Sornborger, arXiv preprint arXiv:2009.02559 (2020).
  * Sutton and Barto [1998] R. S. Sutton and A. G. Barto,  _Reinforcement Learning: An Introduction_ (MIT Press, 1998).
  * Mnih  _et al._ [2015] V. Mnih, K. Kavukcuoglu, D. Silver, _et al._ , Nature 518, 529 (2015).
  * Vinyals  _et al._ [2019] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, J. Oh, D. Horgan, M. Kroiss, I. Danihelka, A. Huang, L. Sifre, T. Cai, J. P. Agapiou, M. Jaderberg, A. S. Vezhnevets, R. Leblond, T. Pohlen, V. Dalibard, D. Budden, Y. Sulsky, J. Molloy, T. L. Paine, C. Gulcehre, Z. Wang, T. Pfaff, Y. Wu, R. Ring, D. Yogatama, D. Wünsch, K. McKinney, O. Smith, T. Schaul, T. Lillicrap, K. Kavukcuoglu, D. Hassabis, C. Apps, and D. Silver, Nature 575, 350 (2019).
  * Silver  _et al._ [2016] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, Nature 529, 484 (2016).
  * Silver  _et al._ [2017] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, _et al._ , Nature 550, 354 (2017).
  * Silver  _et al._ [2018] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis, Science 362, 1140 (2018).
  * Zoph and Le [2017] B. Zoph and Q. V. Le, in  _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_ (OpenReview.net, 2017).
  * Bukov  _et al._ [2018] M. Bukov, A. G. R. Day, D. Sels, P. Weinberg, A. Polkovnikov, and P. Mehta, Phys. Rev. X 8, 031086 (2018).
  * An and Zhou [2019] Z. An and D. L. Zhou, EPL 126, 60002 (2019).
  * An  _et al._ [2021] Z. An, H.-J. Song, Q.-K. He, and D. L. Zhou, Phys. Rev. A 103, 012404 (2021).
  * Fösel  _et al._ [2018] T. Fösel, P. Tighineanu, T. Weiss, and F. Marquardt, Phys. Rev. X 8, 031084 (2018).
  * Fösel  _et al._ [2021] T. Fösel, M. Y. Niu, F. Marquardt, and L. Li, Quantum circuit optimization with deep reinforcement learning (2021), arXiv:2103.07585 [quant-ph] .
  * Ostaszewski  _et al._ [2021] M. Ostaszewski, L. M. Trenkwalder, W. Masarczyk, E. Scerri, and V. Dunjko, arXiv preprint arXiv:2103.16089 (2021).
  * Cao  _et al._ [2021] C. Cao, Z. An, S.-Y. Hou, D. Zhou, and B. Zeng, arXiv preprint arXiv:2105.08696 (2021).
  * Albarrán-Arriagada  _et al._ [2018] F. Albarrán-Arriagada, J. C. Retamal, E. Solano, and L. Lamata, Phys. Rev. A 98, 042315 (2018).
  * Paparo  _et al._ [2014] G. D. Paparo, V. Dunjko, A. Makmal, M. A. Martin-Delgado, and H. J. Briegel, Phys. Rev. X 4, 031002 (2014).
  * Lamata [2017] L. Lamata, Scientific Reports 7, 1609 (2017).
  * Virtanen  _et al._ [2020] P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, S. J. van der Walt, M. Brett, J. Wilson, K. J. Millman, N. Mayorov, A. R. J. Nelson, E. Jones, R. Kern, E. Larson, C. J. Carey, . Polat, Y. Feng, E. W. Moore, J. VanderPlas, D. Laxalde, J. Perktold, R. Cimrman, I. Henriksen, E. A. Quintero, C. R. Harris, A. M. Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, A. Vijaykumar, A. P. Bardelli, A. Rothberg, A. Hilboll, A. Kloeckner, A. Scopatz, A. Lee, A. Rokem, C. N. Woods, C. Fulton, C. Masson, C. Häggström, C. Fitzgerald, D. A. Nicholson, D. R. Hagen, D. V. Pasechnik, E. Olivetti, E. Martin, E. Wieser, F. Silva, F. Lenders, F. Wilhelm, G. Young, G. A. Price, G.-L. Ingold, G. E. Allen, G. R. Lee, H. Audren, I. Probst, J. P. Dietrich, J. Silterra, J. T. Webber, J. Slavič, J. Nothman, J. Buchner, J. Kulick, J. L. Schönberger, J. V. de Miranda Cardoso, J. Reimer, J. Harrington, J. L. C. Rodríguez, J. Nunez-Iglesias, J. Kuczynski, K. Tritz, M. Thoma, M. Newville, M. Kümmerer, M. Bolingbroke, M. Tartre, M. Pak, N. J. Smith, N. Nowaczyk, N. Shebanov, O. Pavlyk, P. A. Brodtkorb, P. Lee, R. T. McGibbon, R. Feldbauer, S. Lewis, S. Tygier, S. Sievert, S. Vigna, S. Peterson, S. More, T. Pudlik, T. Oshima, T. J. Pingel, T. P. Robitaille, T. Spura, T. R. Jones, T. Cera, T. Leslie, T. Zito, T. Krauss, U. Upadhyay, Y. O. Halchenko, Y. Vázquez-Baeza, and S. . Contributors, Nature Methods 17, 261 (2020).
  * Zeng and Wen [2015] B. Zeng and X.-G. Wen, Phys. Rev. B 91, 125121 (2015).
  * Sachdev [2011] S. Sachdev,  _Quantum Phase Transitions_, 2nd ed. (Cambridge University Press, 2011).
  * Sachdev [1999] S. Sachdev,  _Quantum Phase Transitions_ (Cambridge University Press, Cambridge, 1999).
  * Kramers and Wannier [1941] H. A. Kramers and G. H. Wannier, Phys. Rev. 60, 252 (1941).
  * Fradkin and Susskind [1978] E. Fradkin and L. Susskind, Phys. Rev. D 17, 2637 (1978).
  * Yang and Yang [1966a] C. N. Yang and C. P. Yang, Phys. Rev. 150, 321 (1966a).
  * Yang and Yang [1966b] C. N. Yang and C. P. Yang, Phys. Rev. 150, 327 (1966b).
  * Yang and Yang [1966c] C. N. Yang and C. P. Yang, Phys. Rev. 151, 258 (1966c).
  * Barouch  _et al._ [1970] E. Barouch, B. M. McCoy, and M. Dresden, Phys. Rev. A 2, 1075 (1970).
  * Barouch and McCoy [1971] E. Barouch and B. M. McCoy, Phys. Rev. A 3, 786 (1971).
  * Bahtia [1997] R. Bahtia,  _Matrix analysis_ (Springer, 1997).
  * Watkins [1989] C. Watkins,  _Learning From Delayed Rewards_ , Ph.D. thesis, Cambridge University Psychology Department (1989).
  * Schaul  _et al._ [2015] T. Schaul, J. Quan, I. Antonoglou, and D. Silver,  (2015), arXiv:1511.05952 .

Supplemental Materials: Learning quantum phases via single-qubit
disentanglement

##  Appendix S-I Optimal local disentanglement

Suppose state $\ket{\Phi}$ is a pure state of system $RPQ$. Then the reduced
density matrices of subsystem $RP$ is defined as

$\rho_{RP}={\rm Tr}_{Q}\left[\ket{\Phi}\bra{\Phi}\right],$ (S1)

When we perform a local unitary operation $U$ acting on subsystem $RP$, the
reduced state of subsystem $P$ becomes

$\rho_{P}(U)={\rm Tr}_{R}\left[U\rho_{RP}U^{\dagger}\right]$ (S2)

By choosing a proper unitary transformation we aim to minimize the entropy

$\min_{U}S(\rho_{P}(U))$ (S3)

where $S(\rho)=-{\rm Tr}\left[\rho{\rm\log_{2}}\rho\right]$, and $S(\rho_{P})$
characterizes the entanglement between subsystem $P$ and subsystem $RQ$.

Let the dimension of the Hilbert space of subsystem $P$ be $d_{P}$, and let
the dimension of the Hilbert space of subsystem $R$ be $d_{R}$. We start with
the eigen-decomposition of the state $\rho_{RP}$

$\rho_{RP}=\sum_{i=1}^{d_{P}d_{R}}p_{i}\ket{\psi_{i}}\bra{\psi_{i}},$ (S4)

where $p_{i}$ is the $i$-th eigenvalue, and $\ket{\psi_{i}}$ is the
corresponding eigenstate. Without loss of generality, we assume these
eigenvalues are in decreasing order, i.e., $p_{i}\geq p_{i+1}$. Then, for
$1\leq m\leq d_{P}$ let

$q_{m}=\sum_{j=1}^{d_{R}}p_{(m-1)d_{R}+j}.$ (S5)

Now we are ready to give the following theorem on optimal local
disentanglement.

Theorem: The minimal entanglement entropy

$\min_{U}S(\rho_{P}(U))=H(\\{q_{m}\\})\equiv-\sum_{m}q_{m}\log_{2}q_{m}.$ (S6)

Proof: Let us take a basis of $\mathcal{H}_{P}$ as $\\{|m\rangle,1\leq m\leq
d_{P}\\}$, and a basis of $\mathcal{H}_{R}$ as
$\\{|\alpha\rangle,1\leq\alpha\leq d_{R}\\}$. Then we construct a unitary
transformation

$V|\psi_{(m-1)d_{R}+\alpha}\rangle=|m,\alpha\rangle$ (S7)

Then

$\displaystyle\rho_{P}(V)$
$\displaystyle=\sum_{m,\alpha}p_{(m-1)d_{R}+\alpha}\Tr_{R}(|m,\alpha\rangle\langle
m,\alpha|)$ $\displaystyle=\sum_{m}q_{m}|m\rangle\langle m|$ (S8)

which implies that

$S(\rho_{P}(V))=H(\\{q_{m}\\})$ (S9)

For any local unitary $U$, the eigen-decomposition of $\rho_{P}(U)$ is

$\rho_{P}(U)=\sum_{m}q_{m}^{(U)}|m^{(U)}\rangle\langle m^{(U)}|$ (S10)

Then

$\displaystyle\rho_{P}(U)$ $\displaystyle=\rho_{P}(WV)$
$\displaystyle=\sum_{m,\alpha}p_{(m-1)d_{R}+\alpha}\Tr_{R}(W|m,\alpha\rangle\langle
m,\alpha|W^{\dagger})$
$\displaystyle=\sum_{n,\beta}\sum_{m,\alpha}p_{(m-1)d_{R}+\alpha}\langle
n^{(U)},\beta|W|m,\alpha\rangle\langle
m,\alpha|W^{\dagger}|n^{(U)},\beta\rangle|n^{(U)}\rangle\langle n^{(U)}|$
$\displaystyle=\sum_{n,\beta}p^{(U)}_{n,\beta}|n^{(U)}\rangle\langle n^{(U)}|$
(S11)

where $W=UV^{-1}$ and

$\displaystyle
p^{(U)}_{n,\beta}=\sum_{m,\alpha}B_{n,\beta;m,\alpha}p_{(m-1)d_{R}+\alpha}$
(S12)

with

$B_{n,\beta;m,\alpha}=\langle n^{(U)},\beta|W|m,\alpha\rangle\langle
m,\alpha|W^{\dagger}|n^{(U)},\beta\rangle$ (S13)

Combining Eq. S10 with Eq. S-I, we arrives at

$q_{m}^{(U)}=\sum_{\beta}p^{(U)}_{m,\beta}$ (S14)

Notice that $B_{n,\beta;m,\alpha}\geq 0$ and

$\sum_{n,\beta}B_{n,\beta;m,\alpha}=\sum_{m,\alpha}B_{n,\beta;m,\alpha}=1$
(S15)

Namely, $B$ is a doubly stochastic matrix. According to Theorem II.1.9 in
Bhatia’s book [65], we obtain

$p^{(U)}\prec p$ (S16)

Now we denote $p^{(U)}_{\downarrow}$ as $p^{(U)}$ in the decreasing order.
Similarly, we introduce

$q^{(U)}_{\downarrow,m}=\sum_{\alpha}p^{(U)}_{\downarrow,(m-1)d_{R}+\alpha}$
(S17)

Then according to the definition of majorization,

$\displaystyle q^{(U)}_{\downarrow}$ $\displaystyle\prec q$ (S18)
$\displaystyle q^{(U)}$ $\displaystyle\prec q^{(U)}_{\downarrow}$ (S19)

which imples that $q^{(U)}\prec q$. Thus according to Theorem II.3.1 in
Bhatia’s book [65],

$S(\rho_{P}(V))\leq S(\rho_{P}(U)).$ (S20)

This completes our proof.

In our context, $|\Phi\rangle$ is the ground state of the considered
Hamiltonian $H$, $P$ is the target qubit that we aim to disentangle from the
ground state, and $RP$ is the set of qubits that the unitary transformations
$U$ for disentangling $P$ act on.

##  Appendix S-II Reinforcement learning: Q-learning method

The reinforcement learning process is a finite Markov decision process [37].
As shown in Fig. S1, this Markov process is described as a state-action-reward
sequence, a state $S_{t}$ at time $t$ is transmitted into a new state
$S_{t+1}$ together with giving a scalar reward $R_{t+1}$ at time $t+1$ by the
action $A_{t}$ with the transmission probability
$p(S_{t+1};R_{t+1}|S_{t};A_{t})$. For a finite Markov decision process, the
sets of the states, the actions and the rewards are finite. The total
discounted return at time $t$

$G_{t}=\sum_{k=0}^{\infty}\gamma^{k}R_{t+k+1},$ (S21)

where $\gamma$ is the discount rate and $0\leq\gamma\leq 1$.

Figure S1: A schematic diagram of Markov decision process.

The goal of an RL algorithm is to maximize the total discounted return for
each state and action selected by the policy $\pi$, which is specified by a
conditional probability of selecting an action $A$ for each state $S$, denoted
as $\pi(A|S)$.

In the Q-learning algorithm [66], the objective is to maximize the value of
state-action function, the conditional discount return

$Q_{\pi}(s,a)=E_{\pi}(G_{t}|S_{t}=s,A_{t}=a).$ (S22)

If we have a policy $\pi$, then we calculate the value of state-action
function $Q_{\pi}(S,A)$. For each state $s$, we take an action maximizing the
value of state action $Q_{\pi}(s,A)$, which forms our new policy
$\pi^{\prime}$. Then we calculate the value of state-action function
$Q_{\pi^{\prime}}(S,A)$. Repeating the above procedure until the new policy
equals the updated one, which is the optimal policy $\pi^{*}$ we are looking
for.

The update of Q-learning algorithm is defined as

$Q(S_{t},A_{t})\leftarrow Q(S_{t},A_{t})+\Delta Q$ (S23)

with

$\Delta Q=\alpha[R_{t+1}+\gamma\max_{a}Q(S_{t+1},a)-Q(S_{t},A_{t})],$ (S24)

where $\alpha$ is the step size parameter.

Figure S2: A diagram of deep Q-network architecture.

As shown in Fig. S2, a deep Q-network is composed by a deep neural network,
trained with a variant of Q-learning, whose input is state vector and whose
output is a value function estimating future rewards. One significant
improvement is that it designed a memory buffer to store the agent’s
experiences and used them later. At each learning step, we fed the agent with
a minibatch of experiences $\\{S_{t},A_{t},R_{t+1},S_{t+1}\\}$ with a modified
prioritized experience replay (PER) method [67]. The state $S_{t}$ is fed into
the neural network to calculate the state-action value
${Q}(S_{t},A_{t};\theta)$. At the same time, a target Q-network is to
calculate $\max\limits_{a^{\prime}}{Q}(S_{t+1},a^{\prime};\theta^{-})$ in Eq.
(S25). At the end of each step of training, the evaluation network is updated
through the back-propagation by minimizing the loss. Based on Eq. (S23), the
loss is the mean square error (MSE) of the difference between the evaluation
${Q}(S_{t},A_{t};\theta)$ and the target
$\max\limits_{a^{\prime}}{Q}(S_{t+1},a^{\prime};\theta^{-})$

$\rm{loss}=\rm{MSE}((R_{t+1}+\gamma\max\limits_{a^{\prime}}{{Q}}(S_{t+1},a^{\prime};\theta^{-}))-{Q}(S_{t},A_{t};\theta)).$
(S25)

During the learning episodes, the agent updates the parameters of the target
network $\theta^{-}\rightarrow\theta$ to make better decisions.

##  Appendix S-III The convergency of RL-ansatz disentangling circuit

As Fig. S3 and Fig. S3 shown, the performance of the RL-ansatz disentangling
circuit is convergence with the increase of circuit layers.

(a)

(b)

Figure S3: Entanglement entropy versus $\lambda$ for 8-site TFIM. $P$ is the
number of circuit layers. (a): circuit layers of $p=2$ (b): circuit layers of
$p=3$. The solid line denotes the disentangling circuit training from
$\lambda=0.5$; the dashed line denotes the disentangling circuit training from
$\lambda=1.5$.

(a)

(b)

Figure S4: Entanglement entropy versus $\Delta$ for 8-site XXZ model. $P$ is
the number of circuit layers (a): circuit layers of $p=3$ (b): circuit layers
of $p=4$. The solid line denotes the disentangling circuit training from
$\Delta=0.5$; the dashed line denotes the disentangling circuit training from
$\Delta=1.5$.

##  Appendix S-IV Hyper-parameters in RL algorithm

Our RL agent makes use of a deep neural network to approximate the Q values
for the possible actions of each state. The network consists of 6 layers of
the neural network. All layers have ReLU activation functions except the
output layer which has linear activation. The hyper-parameters of the network
are summarized in Table LABEL:para.

Table S1: Training Hyper-Parameters Hyper-parameter | Values
---|---
Neurons in neural network | $\\{512,512,512,512,512,512\\}$
Minibatch size | 120
Replay memory size | 100000
Learning rate | $0.0001$
Update period | 100
Reward decay $\gamma$ | 0.99