# Structural properties of local integrals of motion across the many-body
localization transition via a fast and efficient method for their construction

Safa Adami1 Mohsen Amini1<EMAIL_ADDRESS>Morteza Soltani1 1 Department
of Physics, Faculty of Physics, University of Isfahan, Isfahan 81746-73441,
Iran

###### Abstract

Many-body localization (MBL) is a novel prototype of ergodicity breaking due
to the emergence of local integrals of motion (LIOMs) in a disordered
interacting quantum system. To better understand the role played by the
existence of such macroscopically LIOMs, we explore and study some of their
structural properties across the MBL transition. We first, consider a one-
dimensional XXZ spin chain in a disordered magnetic field and introduce and
implement a non-perturbative, fast, and accurate method of constructing LIOMs.
In contrast to already existing methods, our scheme allows obtaining LIOMs not
only in the deep MBL phase but rather, near the transition point too. Then, we
take the matrix representation of LIOM operators as an adjacency matrix of a
directed graph whose elements describe the connectivity of ordered eigenbasis
in the Hilbert-space. Our cluster size analysis for this graph shows that the
MBL transition coincides with a percolation transition in the Hilbert-space.
By performing finite-size scaling, we compare the critical disorder and
correlation exponent $\nu$ both in the presence and absence of interaction.
Finally, we also discuss how the distribution of diagonal elements of LIOM
operators in a typical cluster signals the transition.

###### pacs:

##  I Introduction

Currently, there is a great scientific interest to gain deeper insight into
the localization phenomena in many-body quantum systems. By now, it has been
found that, in one-dimension (1D), an isolated interacting system of fermions
that is subject to quenched disorder can undergo a phase transition from a
thermal regime where transport is diffusive or subdiffusive Mirlin2005 ; BAA ;
Agarwal15 ; Anto2016 ; Schulz ; Panda  to the MBL phase where the transport
coefficients are exponentially small in the system size Oganesyan ; Znidaric ;
Bardarson ; Gopalakrishnan  and memory of the initial state is retained to
arbitrarily long times.

It is thought that the MBL phase of such systems can be described in terms of
emergent LIOMs which form a complete set of quasilocal conserved quantities
Serbyn2013 ; Huse2014 ; ROS . In the absence of interaction, such a system
exhibits Anderson localization Anderson  for an arbitrarily small amount of
disorder. The corresponding single-particle wavefunctions are exponentially
localized in real space over a characteristic length scale which is called
localization length. In this case, a complete set of LIOMs can be identified
by the occupancies of these single-particle orbitals ROS . Upon turning the
interaction on, multiparticle resonances start to proliferate and, hence, a
stronger disorder is needed to keep the system localized Oganesyan . However,
if the disorder strength is sufficiently larger than the interaction strength,
the system remains in the MBL phase and LIOMs can be understood as weakly
dressed single-particle orbitals ROS ; Imbrie .

In general, the number of ways in which a set of LIOMs can be arranged is very
large, and, therefore, the calculation of all LIOMs is a complicated task
practically. It was first pointed out that a complete set of LIOMs for a
finite-size system can be obtained via labeling the eigenstates of the system
by their corresponding LIOM-eigenvalues uniquely Serbyn2013 ; Huse2014 . Then,
it was suggested to construct LIOMs (which do not form a complete basis) by
computing an infinite-time average of initially local operators Chandran ;
Geraedts . In this regard, various approaches like using Monte-Carlo
stochastic method Stephen , exact diagonalization techniques Rademaker ;
Abanin2016 ; Goihl ; Peng  , and tensor networks Chandran2015 ; Pekker ;
Pollmann ; Pal2017  have been developed.

Although the above-mentioned construction algorithms for LIOMs have made great
progress, developing and implementing a simple method that allows constructing
a complete set of LIOMs with the following properties simultaneously is of
major interest. A method that (i) is non-perturbative and provides quasi-local
LIOMs that commute strictly with the Hamiltonian, (ii) not essentially
requires strong disorder intensity, and (iii) costs much less computationally
but yet has enough accuracy. In particular, the second property makes it
possible to move away from the deep MBL phase toward the transition point and
study some aspects of the phase transition using LIOMs. Thus, the main
objective of the following paper is two-folded. Firstly, we present an
efficient scheme for computing a complete set of LIOMs in a non-perturbative
manner. And, secondly, capture certain aspects of the MBL transition by
considering the resulting LIOM operators as an adjacency matrix of a graph
that represents the connectivity of the eigenbasis in the Hilbert-space and
undergoes a percolation-like transition.

In this work, we describe and develop a fast method to construct a complete
set of LIOMs explicitly for the random-field XXZ spin chain that can be used
to study some statistical properties of LIOMs near the MBL to ergodic
transition. We perform our algorithm via arranging an optimized set of the
eigenstates of the system in a quasi-local unitary operator which maps the
physical spin operators onto effective spins operators. Such an ordered set of
the eigenbasis can be obtained by assigning an integer index number to each
eigenstate which determines its order in our desired set. We recognize this
index number by locating the original basis vector of the Hilbert-space on
which that eigenstate has the largest absolute amplitude among all the
eigenstates of the system. Then, in the next step, we consider the resulting
LIOM operator as an adjacency matrix of a network whose elements indicate the
connectivity of the eigenbasis in the Hilbert-space. We illustrate that this
network undergoes a percolation-like transition on crossing the transition
from MBL into the ergodic phase. The percolation transition can be understood
within the Hilbert-space cluster size analysis of the fragmentation of the
network associated with LIOMs. Such a classical percolation analogy for the
MBL transition was previously observed either by considering the Hamiltonian
as a tight-binding model in Fock-space Chalker ; Chalker2  or by retaining
only resonant contributions and mapping the quantum problem to rate equations
Mierzejewski . However, in this paper, we underline the importance of such a
transition in a network associated with LIOMs which is a key concept in MBL
transition. We further provide an analysis of how local observables on the
clusters of this network can quantitatively capture the ergodic to MBL
transition.

The rest of the paper is organized as follows. In Sec. II, we describe our
spin-1/2 model employed, and introduce our algorithm to construct LIOM
operators. Sec. III contains numerical results obtained by the implementation
of our algorithm. We first represent the results concerning the locality of
obtained LIOM operators. We then use the LIOM operators and show that the
ergodic to MBL transition coincides with a percolation transition in a graph
of eigenvectors in the Hilbert-space whose structure is described by the
matrix representation of LIOMs. To illustrate how the transition takes place
we perform cluster size analysis and apply finite-size scaling to compare the
percolation threshold and correlation exponent $\nu$ in the presence and
absence of interaction. We further discuss how the distribution of local
magnetization of clusters may signals the transition and finally, concluding
remarks are given in Sec. IV

##  II Model and approach

###  II.1 Model Hamiltonian

We consider a standard model of MBL which is a spin-$1/2$ chain of length $L$
in a random magnetic field in the $z$-direction and can be written as:

$H=\sum_{i=1}^{L-1}J\left({\sigma}^{+}_{i}{\sigma}^{-}_{i+1}+{\sigma}^{-}_{i}{\sigma}^{+}_{i+1}+\frac{1}{2}\Delta{\sigma}^{z}_{i}{\sigma}^{z}_{i}\right)+h_{i}{\sigma}^{z}_{i}$
(1)

where $\sigma^{\pm}_{i}=\sigma^{x}_{i}\pm i\sigma^{y}_{i}$ are the raising and
lowering spin-$1/2$ operators and $\sigma_{i}^{x,y,z}$ denote the Pauli
operators acting on spin $i$. Here, we use open boundary conditions and fix
the exchange interaction coupling at $J=1$. The values $h_{i}$ are, also,
drawn independently from a random uniform distribution $[-W,W]$ and the
parameter $\Delta$ determines the anisotropy of the model. This model is known
to undergo a phase transition at a critical disorder strength $W=W_{c}=3.5\pm
0.5$ from an ergodic phase to an MBL phase which depends on energy density
Pal-Huse ; Alet2015 . In the current study, we focus on the MBL side of the
transition, $W>W_{c}$, in which the existence of LIOMs prevents
thermalization. Using the Jordan‐Wigner transformation JW , this model can be
mapped to a model of spinless fermions and we are interested in two different
cases when $\Delta=0$ and $\Delta=1$ which corresponds to the non-interacting
Anderson model and an interacting and disordered fermionic model respectively.

###  II.2 Approach

To begin with, let us review the basic idea behind the LIOM scheme. We first
consider a non-interacting system in which $\Delta=0$. Upon diagonalization of
the Hamiltonian in Eq.( 1), one obtains a set of energy eigenvalues that
uniquely identifies the system’s eigenstates. In this system, which is
equivalent to a single-particle Anderson model, eigenstates are exponentially
localized around some localization center and their occupation numbers are
mutually commuting, conserved quantities, and hence, can form a complete set
of LIOMs. These are the number operators,

$n_{\alpha}=\sum_{ij}\psi^{*}_{\alpha}(i)\psi_{\alpha}(j)c^{\dagger}_{i}c_{j},$
(2)

in terms of which the Hamiltonian can be rewritten as

$H=\sum_{\alpha}2\varepsilon_{\alpha}n_{\alpha}-\sum_{\alpha}\varepsilon_{\alpha},$
(3)

where the last term on the right hand side is the vacuum constant energy
shift. It is now straightforward to define the corresponding LIOM operators in
terms of the original spin operators via the Jordan-Wigner string operator as

$\tau_{\alpha}^{z}=2\sum_{ij}\psi^{*}_{\alpha}(i)\psi_{\alpha}(j)\sigma^{+}_{i}\left(\prod_{k={\rm
min}(i,j)}^{{\rm max}(i,j)}\sigma^{z}_{k}\right)\sigma^{-}_{j}-1,$ (4)

which allows to write the Hamiltonian as

$H=\sum_{\alpha}\varepsilon_{\alpha}\tau_{\alpha}^{z}.$ (5)

Given the locality of the $\tau_{\alpha}$, one could as well associate an
index $i$ of the lattice to each index $\alpha$, for example considering the
maximum of $|\psi_{\alpha}(i)|^{2}$.

In the presence of interactions, however, the basic idea behind the LIOMs
scheme is to find a unitary transformation $U$ that defines a similar complete
set of independent pseudospin-$1/2$ operators

$\tau_{i}^{z}=U\sigma_{i}^{z}U^{\dagger}.$ (6)

With the above considerations, the following properties are fulfilled by the
$\tau_{i}^{z}$ operatorsROS :

(i) $\tau_{i}^{z}$’s are quasi local operators, in the sense that:

$||[\tau_{i}^{z},\sigma^{a}_{j}]||<ce^{-|i-j|/\xi},$ (7)

for $a=+,-,z$ and some $\xi,c$.

(ii) $\tau_{i}^{z}$ are exactly conserved: $[H,\tau_{i}^{z}]=0$.

(iii) $\tau_{i}^{z}$ have eigenvalues $\pm 1$ ($(\tau^{z})^{2}=1$) and each
subspace has exactly dimension $2^{L-1}$.

By the locality of the terms of $H$ and of $\tau^{z}_{i}$, it follows that
$J_{i_{1},...,i_{a}}$ are local, in the sense that they decay exponentially as
a function of any couple of indices. Therefore, the Hamiltonian $H$, in the
MBL phase, takes the form

$H=\sum\limits_{i}{{\varepsilon_{i}}{{\tau}_{i}^{z}}+\sum\limits_{ij}{{J_{ij}}}}{{\tau}_{i}^{z}}{\tau_{j}^{z}}+\sum\limits_{ijk}{{J_{ijk}}}{{\tau}_{i}^{z}}{{\tau}_{j}^{z}}{{\tau}_{k}^{z}}+...,$
(8)

where the number of different sums is $L$. The unitary $U$ is a composition of
local unitary transformations as described by Refs Imbrie2 ; Anto . With the
same unitary transformation one can also define $\tau^{\pm}_{i}$, which
complete the Pauli algebra.

It is obvious, that each arbitrary arrangement of the eigenvectors of
Hamiltonian $H$ in the unitary matrix $U$ of Eq. (6) results in a new set of
$\tau_{i}$ operators which satisfy the above-mentioned properties (ii) and
(iii) by default. However, we are interested in finding a complete set of
$\tau_{i}$ operators that fulfills also the quasi-locality requirement which
is defined in Eq. (7). Therefore, our goal is to identify a specific
arrangement of the eigenstates in $U$ that best fulfill properties (i)-(iii)
altogether.

For all choices of $W$ and $\Delta$, total magnetization is a conserved
quantity which implies the conservation of the $z$-component of the total
spin, $[S_{t}^{z},H]=0$ with $S_{t}^{z}=\sum_{i=1}^{L}S_{i}^{z}$. Therefore,
it defines a good quantum number and we can consider different magnetization
sectors separately. Throughout the paper we use the standard notation
$|n\rangle\equiv|S_{1}^{z},S_{2}^{z},...,S_{L}^{z}\rangle$ with
$S_{i}^{z}=\uparrow,\downarrow$ for the basis states in real space. In this
notation, $n$ in $|n\rangle$ is a decimal integer can be obtained from the
$L$-bit binary representation $|n_{1}n_{2}...n_{L}\rangle$ as
$n=\sum_{i=1}^{L}n_{i}2^{i-1}$ where $n_{i}=0,1$ stands for
$S_{i}^{z}=\downarrow,\uparrow$ respectively. Using these basis states, we
consider an initial set of the basis vectors $\\{|n\rangle\\}$ in such a way
that the Hamiltonian $H$ is block diagonalized and each block corresponds to a
subspace with a fixed magnetization. Here, we are interested to introduce an
efficiant way of ordering the LIOM basis states, and hence, we will use the
same labeling scheme in which each basis state can be shown by
$\widetilde{|n\rangle}\equiv|\tau_{1}^{z},\tau_{2}^{z},...,\tau_{L}^{z}\rangle$
with effective pseudo spin $\tau_{i}^{z}=\tilde{\downarrow},\tilde{\uparrow}$.
Again, it is convenient to obtain the corresponding integer $\tilde{n}$ for
each LIOM basis vector from its binary representation as before. In what
follows, we introduce an optimal ordered set of basis states that makes the
unitary operator

$U=\sum_{n}\widetilde{|n\rangle}\langle n|$ (9)

and can be used in Eq. (6) to form a complete set of LIOMs with our desired
properties (i)-(iii).

We begin to construct our own approach by considering the non-interacting
case, ($\Delta=0$). In this case, as we already mentioned, LIOMs can be
characterized by conserved occupations of single-particle eigenstates and
hence, only the first term on the right-hand side remains in Eq. (8). We start
with the reference state $|0\rangle$ with all spins down as the only possible
state in its magnetization sector which is also an eigenstate of the system.
Therefore, it has a similar representation on both original and LIOM basis,
i.e.
$\underbrace{|\tilde{\downarrow}\tilde{\downarrow}...\tilde{\downarrow}\rangle}_{L}=\underbrace{|\downarrow\downarrow...\downarrow\rangle}_{L}$.

By flipping one spin in $|0\rangle$, we get a new state with $S_{t}^{z}=L/2-1$
and since we have $L$ places for this spin, we have $L$ states in this sector.
These states, which are supposed to be ordered according to their binary code,
form the original basis stats spanning the single-particle block of the
Hamiltonian $H$. After diagonalizing Hamiltonian $H$, we obtain a set of
eigenstates $|\psi_{m}\rangle$ which needs to be ordered. According to the
quasi-locality criterion of Eq. (7), we expect an ordered set in which each
pseudospin operator $\tau_{i}^{z}$ is mostly localized around a physical spin
operator in real space. Therefore, we can order the obtained eigenstates by
determining their maximum overlap with the original basis states. For
instance, the first eigenstate is the one that has maximum overlap with the
first original basis state. Thus, we need to find the maximum available
overlap among the set
$\\{|\langle\uparrow\underbrace{\downarrow\downarrow...\downarrow}_{L-1}|\psi_{m}\rangle|^{2},\;m=1,...,L\\}$.
If the $m_{0}$-th eigenstate is the one with maximum overlap with the first
original basis state, it is the first basis state in psudo spin space which
means that
$|\tilde{\uparrow}\underbrace{\tilde{\downarrow}\tilde{\downarrow}...\tilde{\downarrow}}_{L-1}\rangle=|\psi_{m_{0}}\rangle$.
By the same token, it is possible to determine the $j$th-eigenstate of this
sector by defining the following sequence of eigenstate overlaps,

$\\{\alpha_{m}^{j}=|\langle\downarrow...\underbrace{\uparrow}_{\text{$j$th}}\downarrow|\psi_{m}\rangle|^{2}\\},\;\;m=1,...,L\;,$
(10)

and finding the eigenstate which maximizes the above overlap and label it with
$|\tilde{\downarrow}...\underbrace{\tilde{\uparrow}}_{\text{$j$th}}\tilde{\downarrow}\rangle$.

We now proceed to the next sector which has $L(L-1)/2$ basis states with two
flipped spins which can be represented by the following notation,

$|j_{1},j_{2}\rangle=S_{j_{1}}^{+}S_{j_{2}}^{+}|0\rangle=|\downarrow...\underbrace{\uparrow}_{\text{$j_{2}$th}}...\underbrace{\uparrow}_{\text{$j_{1}$th}}...\downarrow\rangle.$
(11)

The indexes $j_{1}$ and $j_{2}$ immediately determine the associated integer
number $n$ of this basis vector accordingly. Therefore, if we are looking for
the $\tilde{n}$-th eigenstate in our optimized set, we should find the one
with maximum overlap with its corresponding basis vector. That is finding the
maximum value among the following set of overlaps

$\\{|\langle j_{1},j_{2}|\psi_{m}\rangle|^{2},\;\;m=1,...,L(L-1)/2\\},$ (12)

and label it as $\tilde{n}$. The same analysis can be performed in higher-
excitation sectors to arrange the final ordered set of the eigenstates
properly.

Besides its ease of use and implementation, the main advantage of the above-
mentioned algorithm is that it can be generalized even for the case of an
interacting system ($\Delta\neq 0$). At the same time, it provides an
opportunity to consider the whole Hamiltonian in the full Hilbert-space of the
system, simultaneously. Therefore, In the remainder of this subsection, we
elaborate on the implementation of the algorithm which allows ordering a
generic set of energy eigenstates of the system in such a way that the
resulting LIOMs satisfy our desired properties (i)-(iii).

Suppose that we have an initial set of the basis vectors $|n\rangle$ in which
our Hamiltonian matrix is block-diagonal. We can diagonalize this Hamiltonian
and obtain the set of energy eigenbasis which is an arbitrary (but fixed)
arrangement of energy eigenbasis of the system. Our aim is to rearrange them
by assigning a decimal integer that determines their index in our final
optimum set. Doing so, we use the fact that each eigenstate of the system is a
$2^{L}$ component vector which can be expanded based on the Hilbert-space
original basis vectors $|n\rangle$ as
$|\psi_{i}\rangle=\sum_{n^{\prime}=1}^{2^{L}}A_{n^{\prime}}^{i}|n^{\prime}\rangle$.
We can label a given eigenbasis $|\psi_{i}\rangle$ by integer number $n$ if
this eigenstate has the largest absolute amplitude on $|n\rangle$ among all
the eigenstates of the system. This means that one needs only to find the
index $n$ in such a way that $|A_{n}^{i}|^{2}$ is the largest value of the set
$\\{|A_{n}^{i}|^{2},i=1,...,2^{L}\\}$. In the other word, if we consider the
matrix $U$ which initially contains the eigenbasis of the Hamiltonian in its
columns with an arbitrary arrangement, to find the $n$-th eigenstate in our
desired order, we need just to look at the $n$-th row of $U$ and determine
which column has the maximum absolute value in this row. This procedure is
represented graphically in Fig. 1. The repeated execution of this procedure
results in our optimal arrangement of the eigenstates which can optimally
satisfy our desired conditions. Since the procedure outlined above does not
essentially require the strong disorder limit, and on the other hand, is very
simple and fast we can use it to obtain our optimum and complete set of LIOMs
rapidly and study some structrual properties of the system across the phase
transition.

Figure 1:  Graphical representation of the procedure outlined in section II.2
to arrange the eigenstates of the system in the unitary matrix $U$ which is
obtained after diagonalization. The left side shows the matrix of eigenvectors
obtained via the exact diagonalization procedure in which usually the
eigenstates are ordered according to their corresponding eigenvalues. The
right side is the matrix of eigenstates after rearranging them using our
algorithm.

##  III Results

To examine our method we have carried out numerical calculations based on the
exact diagonalization technique. In what follows, we consider a spin chain
with $L$ spins and open boundary condition. In order to gain a deeper
understanding of the role of interaction in the MBL case, we consider our
model in both interacting ($\Delta=1$) and non-interacting ($\Delta=0$)
regimes, and depending on the system size $L$, $10^{5}$ to $5\times 10^{3}$
disorder realizations are employed to obtain the statistics.

###  III.1 Effective characterization of LIOM locality

In this section, we demonstrate the quasi-locality of the resulting LIOM
operators obtained by our algorithm. To this end, we use the two-point
correlator between a LIOM operator $\tau_{j}^{z}$ and physical spin
$\sigma_{k}$ which is expected Chandran ; Peng ; Geraedts  to decay
exponentially with distance $|k-j|$ as:

$\langle\tau_{j}^{z}\sigma_{k}^{z}\rangle={\rm\,Tr\,}(\tau_{j}^{z}\sigma_{k}^{z})\sim\exp{(-|k-j|/\zeta)},$
(13)

when $j$ and $k$ are far apart in the MBL regime. In Eq. (13) $\zeta$ defines
a lengthscale over which the corresponding LIOM operators are localized. This
lengthscale is related to the spatial correlation length of the eigenstate
amplitudes on the Fock-space Ivan ; Logan  and expects to diverge at the
critical point.

Fig. 2 (a) shows the behavior of the logarithm of
$\langle\tau_{j}^{z}\sigma_{k}^{z}\rangle$ versus $|j-k|$ both for $\Delta=0$
and $\Delta=1$. It is obvious that in the deep localized regime $(W\geq 5)$,
the LIOM operators $\tau_{j}^{z}$ are strongly localized, and the
$\langle\tau_{j}^{z}\sigma_{k}^{z}\rangle$ profile is mostly localized near
the origin $j$ with a fast decaying function to the neighborhood. This is in
contrast to the delocalized regime $(W\leq 2.5)$ in which such a fast decaying
part is obviously absent. Furthermore, there is a clear size dependency
specially near the origin which is the characteristic feature of ergodic
regime. In order to make a comparison with non-interacting system $(\Delta=0)$
we have shown the behavior of the $\langle\tau_{j}^{z}\sigma_{k}^{z}\rangle$
profile in Fig. 2 (b). In this case ($\Delta=0$), even for very small disorder
strength $W=0.5$ the faster decaying behavior as well as weaker size
dependency can be observed in comparison to the MBL counterpart ($\Delta=1$).

It is also worth mentioning that the characteristic lengthscale $\zeta$ can be
extracted from the linear part of the
$\log(\langle\tau_{j}^{z}\sigma_{k}^{z}\rangle)$ versus $|j-k|$ for the
largest system size to observe its divergence near the transition point. The
inset shows the power-law divergence of the $\zeta$ as a function of
$(W-W_{c})$ on approaching the transition point ($W_{c}=3.0$ and $W_{c}=0.0$
in the presence and absence of interaction respectively) in the localized
regime ($W>W_{c}$).

Although the above comparison between the locality of LIOMs in both
interacting and non-interacting systems is a piece of qualitative evidence for
their difference in the sense of critical disorder needed for localization
transition, we will elaborate on this more quantitatively in the coming
sections and discuss the critical disorder $W_{c}$ on which the transition
takes place in detail.

Figure 2: Decay of two-point correlator
($\log(\langle\tau_{j}^{z}\sigma_{k}^{z}\rangle)$) for the LIOM operator
localized near the chain center is averaged over many disorder realizations
versus $|j-k|$ for different system sizes $L=8-16$ and disorder intensities in
both (a) interacting ($\Delta=1$) and (b) non-interacting ($\Delta=0$)
regimes. Insets show the divergence of lengthscale $\zeta$ as a function of
$(W-W_{c})$ in the presence and absence of interaction respectively.

###  III.2 Percolation transition in connected clusters associated with LIOMs
in the Hilber-space

In this section, we introduce a classical percolation problem associated with
clusters of LIOMs in the Hilber-space. Indeed, LIOMs are dressed versions of
spin operators as given by Eq. (6) and can be viewed as a matrix in the
Hilbert-space eigenbasis. Our idea is to interpret this matrix as an adjacency
matrix representation of a finite directed graph. Under this interpretation,
the nodes are the eigenbasis of the Hilbert-space and the entries in a matrix
associated with each LIOM indicate whether two nodes are adjacent or not.
Obviously, to make our percolation picture we need to consider a connectivity
threshold (CT) for deciding below which edge between a pair of basis states on
the Hilbert-space graph will be removed which we describe how to estimate in
the following.

Figure 3: The Left side shows the matrix representation of a typical LIOM
operator obtained using Eq. (6) and the ordered set of eigenvectors in unitary
matrix $U$. The right side is the corresponding adjacent graph of the same
LIOM operator in the Hilbert-space eigenbasis for a spin chain with length
$L=10$ spins for two different disorder intensities $W=2$ (upper panel) and
$W=6$ (lower panel) in the MBL regime ($\Delta=1$).

One detail should be described before discussing the estimation procedure. The
point is to restrict our numerical calculations to the largest subspace with
zero total spin, because the size of the Hilbert-space grows exponentially and
total magnetization is a conserved quantity in our system. This subspace
contains $N_{H}=\left(\begin{array}[]{c}L\\\ \frac{L}{2}\end{array}\right)$
states (nodes). Thus, one naturally expects to have only a single connected
cluster with size (number of nodes) $N_{H}$ for a very weak disorder
intensity, namely $0<W\ll 1$. This criterion will give us an upper bound for
the CT parameter. In our computation below we take the maximum possible value
for CT according to its upper bound. Fig. 3 shows a typical LIOM operator
(represented as a matrix) and its adjacent Hilbert-space graph which is
obtained with $CT=0.05$ for a spin chain with $L=10$ spins for two random
realizations of disorder with disorder strengths $W=2,6$ in the MBL regime
($\Delta=1$).

####  III.2.1 Cluster size analysis

In the theory of lattice percolation, the emergence of a spanning cluster at
the percolation threshold which connects two opposite boundaries on the
lattice is a measure of percolation transition Bunde . It is obvious that such
a definition doesn’t make really sense for our considered network here.
Therefore, in this subsection, we characterize the percolation transition by
analyzing the size of the largest and second-largest connected cluster SS2
associated with LIOMs in the Hilbert-space. Starting from the low disorder
limit, $W\ll 1$, we expect to have only a single connected cluster with size
$S_{1}=N_{H}$ which contains all the nodes of Hilbert-space. Due to the
localization of eigenstates, one expects to observe a decrease in the largest
cluster size by increasing the disorder intensity which means that it only
contains a finite fraction of the Hilbert-space nodes. In the percolation
language, this is equivalent to the formation of smaller size clusters in the
network.

Figure 4: Behavior of the average largest and second-largest cluster sizes
versus disorder intensity $W$ for a spin chain with $L=16$ spins in both (a)
the non-interacting $(\Delta=0)$ and (b) interacting $(\Delta=1)$ regimes. It
is obvious one needs a stronger disorder to reach the non-percolating regime
in the presence of interaction.

We start to illustrate our percolation scenario by calculating the fraction of
the largest cluster, defined as the relative size of the largest connected
cluster with respect to the Hilbert-space dimension, $S_{1}/N_{H}$. Fig. 4
shows how the mean largest cluster size which is averaged over different
realizations of disorder decreases as a function of disorder intensity $W$ for
a spin chain of length $L=16$. Additionally, we can also compute the average
size of the second-largest cluster $S_{2}$ to confirm the transition
threshold. We observe that the normalized size of the second-largest cluster,
$S_{2}/S_{2}^{max}$, also peaks near the percolation transition. We have
plotted the same quantities for both non-interacting and interacting regimes
in Fig. 4 (a) and (b) respectively in order to make a comparison possible. It
is obvious that the presence of interaction shifts the percolation threshold
$W_{c}$ toward the stronger intensity of disorder, however, we leave the
discussion of more accurate determination of such a percolation threshold for
the sections that follow.

Figure 5: The resulting data collapse of $S_{typ}/N_{H}$ onto a scaling
function of $(W-W_{c})L^{1/\nu}$which is obtained for spin chains with
different lengths $(L=8-16)$ both in the (a )absence $\Delta=0$ and (b)
presence $\Delta=1$ of interaction in the localized phase. The critical
parameters $W_{c}=3.0,0.0$ and $\nu=2.0,2.5$ is obtained for the case of MBL
and Anderson transition respectively. Insets show the corresponding raw data.

####  III.2.2 Universal feature and finite-size scaling analysis near the
transition point

The more precise determination of the percolation threshold for the adjacent
graph of the LIOMs can be obtained using scaling analysis. Following the
arguments of Ref. Chalker , we first focus our attention on the scaling of the
mean cluster size, and since the largest cluster is not essentially a typical
one we take the cluster $C$ that contains the eigenstates $|\psi_{0}\rangle$
with the closest energy to the mean value of the energy spectrum and compute
the geometric average of its size over different realizations of disorder as,

$S_{\text{typ}}=\exp{(\frac{1}{N_{r}}\sum_{r}\ln(s_{r}))}.$ (14)

Here, $s_{r}$ is the number of nodes (eigenstates) in the cluster $C$ for a
given realization $r$ of disorder and $N_{r}$ is the number of disorder
realizations. According to the finite size scaling Bunde  the scaling of the
normalized cluster sizes near the transition point can be stated as Chalker

$S_{\text{typ}}/N_{H}\sim f((W-W_{c})L^{\frac{1}{\nu}}),$ (15)

in which the exponent $\nu$ is called the correlation length exponent.
Therefore, by performing data collapse analysis it is possible to obtain the
percolation threshold $W_{c}$ and critical exponent $\nu$. The results of such
data collapse, yielding the critical exponent $\nu(\Delta=1)=2.0$ and
$\nu(\Delta=0)=2.5$ in the presence and absence of interaction is shown in
Fig. 5 (a) ,(b) respectively. We should emphasize that in the data collapse
procedure in the non-interacting regime $\Delta=0$ we constrained the critical
point $W_{c}=0.0$ as a transition point of the corresponding XX modelXX . We
note that the resulting exponent $\nu$ satisfies the Harris-CCFS bound
$(\nu\geq\frac{2}{d})$ HC1 ; HC2  confirmed for the ergodic to MBL transition
recently Chalker ; Chandran2 ; Anto22 .

In addition, the percolation threshold is quite different as $W_{c}=3.0$ and
$W_{c}=0.0$ for the case of the interacting and non-interacting systems
respectively. It is worth mentioning, that beyond the error of our analysis,
the percolation threshold for the case of the interacting system coincides
with the ergodic to MBL transition point $W_{c}\approx 3.5$ Luitz  which shows
that our system experiences a percolation transition in its corresponding
LIOMs across the MBL transition.

Before ending this subsection let us shed more light on the effect of changing
the estimated CT parameter in our clusterization mechanism. As we already
discussed, we set the value of this parameter by the largest value of CT for
which the size of the largest cluster is exactly equal to the Hilbert-space
dimension $N_{H}$. It is certainly possible to consider smaller values for CT
parameter. However, our investigations show that the changing of CT parameter
may change a bit the percolation threshold $W_{c}$, but, the correlation
length exponent $\nu$ will not change.

###  III.3 Distribution of local observables on the clusters

The last quantity which we are interested in is the distribution of the
eigenstate expectation values of local observables which are shown to vary
significantly across the MBL transition Anto06 ; Luitz06 . In doing so, let us
consider the cluster $C$ which contains the eigenstate $|\psi_{0}\rangle$ with
size $s$, as we discussed before, and define the following expectation value
for the cluster Chalker ,

$m_{l}=\frac{1}{s}\sum^{\prime}_{\tilde{n}}\langle\tilde{n}|\tau^{z}_{i}|\tilde{n}\rangle,$
(16)

where $\sum^{\prime}$ denotes the restrictions imposed by considering only the
eigenstates in $C$ in summation. This is indeed the average magnetization of
the cluster and can be obtained by averaging the diagonal entries of the
corresponding LIOM operator which belongs to the cluster $C$ and we are
interested in the distribution of this quantity over different realizations of
disorder. Fig. 6 shows the distribution of this quantity for the largest
system size $L=16$ spins in both MBL and Anderson regimes. In the case of
interacting regime $\Delta=1$ for weak disorder intensity $W=0.5$ we have a
single peak around zero. This is because in this regime the cluster $C$
contains all the eigenstates of the system which means that we are computing
the trace of $\tau_{i}^{z}$ operator which we expect to be zero. However, by
increasing the disorder intensity some of the nodes (eigenstates) will be
excluded from this summation, and therefore we are computing the partial trace
of the local operator $\tau_{i}^{z}$. In the deep MBL phase, we expect to have
clusters with smaller sizes which for the limiting case of very strong
disorder one may observe clusters with only a single node with $m_{l}=\pm 1$.
The plots in Fig. 6 show that the existence of such single-node clusters is
much more probable even for very small disorder intensities in the case of the
non-interacting regime ($\Delta=0$).

Figure 6: Distribution function $P(m_{l})$ of the average magnetization of the
cluster $m_{l}$ which is the average expectation value of LIOM operator
$\tau^{z}_{i}$ over the nodes of this cluster both for $\Delta=0$ and
$\Delta=1$.

##  IV CONCLUDING REMARKS

We provided a fast and accurate method to efficiently obtain the LIOMs for a
disordered system that undergoes MBL transition. We showed that an optimized
set of eigenvectors obtained by locating their maximum overlap with Hilbert-
space basis can be used to obtain the desired set of LIOM operators. We showed
that the resulting LIOMs experience a percolation transition in their graph
representation in the Hilbert-space by increasing disorder intensity. We also
compared the critical disorder and critical exponent describing percolation
transition for both interacting (MBL) and non-interacting (Anderson) regimes.
Our analysis showed that there is a concrete connection, between the ergodic
to MBL transition and the structural properties of LIOMs in their graph
representation on the Hilbert-space.

###### Acknowledgements.

We are extremely grateful to A. Scardicchio for multiple illuminated
discussions, insights, and guidance to improve our work. We acknowledge the
CINECA award under the ISCRA initiative, for the availability of high
performance computing resources and support. MA acknowledges the support of
the Abdus Salam (ICTP) associateship program.

## References

  * (1) K. Agarwal, S. Gopalakrishnan, M. Knap, M. Müller, and E. Demler, Anomalous Diffusion and Griffiths Effects Near the Many-Body Localization Transition, Phys. Rev. Lett. 114, 160401 (2015).
  * (2) I. V. Gornyi, A. D. Mirlin, and D. G. Polyakov, Interacting Electrons in Disordered Wires: Anderson Localization and Low-$T$ Transport, Phys. Rev. Lett. 95, 206603 (2005).
  * (3) D. M. Basko, I. L. Aleiner, and B. L. Altshuler, Metal–insulator transition in a weakly interacting many-electron system with localized single-particle states, Ann. Phys. (NY) 321, 1126 (2006).
  * (4) M. Znidaric, A. Scardicchio, and V. K. Varma, Diffusive and Subdiffusive Spin Transport in the Ergodic Phase of a Many-Body Localizable System, Phys. Rev. Lett. 117, 040601 (2016).
  * (5) R. K. Panda, A. Scardicchio, M. Schulz, S. R. Taylor, M. Znidaric, Can we study the many-body localisation transition?, Europhysics Letters 128 (6), 67003 (2020).
  * (6) M. Schulz, S. R. Taylor, C. A. Hooley, A .Scardicchio, Energy transport in a disordered spin chain with broken $U(1)$ symmetry: Diffusion, subdiffusion, and many-body localization,Phys. Rev. B 98, 180201 (2018).
  * (7) V. Oganesyan and D. A. Huse, Localization of interact- ing fermions at high temperature, Phys. Rev. B 75, 155111 (2007).
  * (8) M. Znidaric, T. Prosen, and P. Prelovšek, Many-body localization in the Heisenberg XXZ magnet in a random field, Phys. Rev. B 77, 064426 (2008).
  * (9) J. H. Bardarson, F. Pollmann, and J. E. Moore, Unbounded Growth of Entanglement in Models of Many-Body Localization, Phys. Rev. Lett. 109, 017202 (2012).
  * (10) S. Gopalakrishnan, M. Müller, V. Khemani, M. Knap, E. Demler, and D. A. Huse, Low-frequency conductivity in many-body localized systems ,Phys. Rev. B 92, 104202 (2015).
  * (11) A. De Luca, A. Scardicchio, Ergodicity breaking in a model showing many-body localization, Europhysics Letters, 101 (3), 37003 (2013).
  * (12) V. Ros, M. Müller, and A. Scardicchio, Integrals of motion in the many-body localized phase, Nucl. Phys. B 891, 420 (2015).
  * (13) M. Serbyn, Z. Papić, and D. A. Abanin, Local Conservation Laws and the Structure of the Many-Body Localized States, Phys. Rev. Lett. 111, 127201 (2013).
  * (14) D. A. Huse, R. Nandkishore, and V. Oganesyan, Phenomenology of fully many-body-localized systems, Phys. Rev. B, 90, 174202 (2014).
  * (15) P. W. Anderson, Absence of Diffusion in Certain Random Lattices, Phys. Rev. 109, 1492 (1958).
  * (16) J. Z. Imbrie, Diagonalization and Many-Body Localization for a Disordered Quantum Spin Chain, Phys. Rev. Lett. 117, 027201 (2016).
  * (17) A. Chandran, I. H. Kim, G. Vidal, and D. A. Abanin, Diagonalization and Many-Body Localization for a Disordered Quantum Spin Chain, Phys. Rev. B 91, 085425 (2015).
  * (18) S. D. Geraedts, R. N. Bhatt, and R. Nandkishore, Emergent local integrals of motion without a complete set of localized eigenstates, Phys. Rev. B 95, 064204 (2017).
  * (19) S. Inglis, and Lode Pollet, Accessing Many-Body Localized States through the Generalized Gibbs Ensemble, Phys. Rev. Lett. 117, 120402 (2016).
  * (20) L. Rademaker and M. Ortuño, Explicit Local Integrals of Motion for the Many-Body Localized State, Phys. Rev. Lett. 116, 010404 (2016).
  * (21) T. E. O’Brien, D.A. Abanin, G. Vidal, and Z. Papic, Explicit construction of local conserved operators in disordered many-body systems, Phys. Rev. B 94, 144208 (2016).
  * (22) M. Goihl, M. Gluza, C. Krumnow, and J. Eisert, Construction of exact constants of motion and effective models for many-body localized systems, Phys. Rev. B 97, 134202 (2018).
  * (23) P. Peng, Z. Li, H. Yan, K. X. Wei, and P. Cappellaro, Comparing many-body localization lengths via nonperturbative construction of local integrals of motion, Phys. Rev. B 100, 214203 (2019).
  * (24) A. Chandran, J. Carrasquilla, I. H. Kim, D. A. Abanin, and G. Vidal, Spectral tensor networks for many-body localization, Phys. Rev. B 92, 024201 (2015).
  * (25) David Pekker and Bryan K. Clark, Encoding the structure of many-body localization with matrix product operators, Phys. Rev. B 95, 035116 (2017).
  * (26) F. Pollmann, V. Khemani, J. I. Cirac, and S. L. Sondhi, Efficient variational diagonalization of fully many-body localized Hamiltonians, Phys. Rev. B 94, 041116 (2016).
  * (27) T. B. Wahl, A. Pal, and S. H. Simon, Efficient Representation of Fully Many-Body Localized Systems Using Tensor Networks, Phys. Rev. X 7, 021018 (2017).
  * (28) S. Roy, J. T. Chalker, and D. E. Logan, Percolation in Fock space as a proxy for many-body localization, Phys. Rev. B 99, 104206 (2019).
  * (29) S. Roy, D. E. Logan, and J. T. Chalker, Exact solution of a percolation analog for the many-body localization transition, Phys. Rev. B 99, 220201(R) (2019).
  * (30) P. Prelovsek, M. Mierzejewski, J. Krsnik, and O. S. Barisic, Many-body localization as a percolation phenomenon, Phys. Rev. B 103, 045139 (2021).
  * (31) A. Pal and D.A. Huse, Many-body localization phase transition, Phys. Rev. B 82, 174411 (2010).
  * (32) D. J. Luitz, N. Laflorencie, and F. Alet, Many-body localization edge in the random-field Heisenberg chain, Phys. Rev. B 91, 081103 (2015).
  * (33) P. Jordan and E. Wigner, Uber das paulische aquivalenzverbot, Z. Phys. 47, 631 (1928).
  * (34) J. Z. Imbrie, On Many-Body Localization for Quantum Spin Chains, Journal of Statistical Physics, 163(5), 998 (2016).
  * (35) J. Z. Imbrie, V. Ros, and A. Scardicchio, Local integrals of motion in many-body localized systems, Annalen der Physik, 529(7),1600278 (2017).
  * (36) G. De Tomasi, I. M. Khaymovich, F. Pollmann, and S. Warzel, Rare thermal bubbles at the many-body localization transition from the fock space point of view, Phys. Rev. B 104, 024202 (2021).
  * (37) S. Roy, and D.E. Logan,Fock-space anatomy of eigenstates across the many-body localization transition, Phys. Rev. B 104, 174201 (2021).
  * (38) A.Bunde and S. Havlin, Fractals and disordered systems (Springer, 1996).
  * (39) A. Margolina, H. J. Herrmann, and D. Stauffer, Size of largest and second largest cluster in random percolation, Physics Letters A 93, 73 (1982).
  * (40) T. Devakul, V. Khemani, F. Pollmann, D. A. Huse, and S. L. Sondhi, Obtaining highly excited eigenstates of the localized xx chain via dmrg-x, Phil. Trans. R. Soc. A 375 20160431 (2017).
  * (41) A. B. Harris, Effect of random defects on the critical behaviour of Ising models, J. Phys. C: Solid State Phys. 7, 1671 (1974).
  * (42) J. T. Chayes, L. Chayes, D. S. Fisher, and T. Spencer, Finite-Size Scaling and Correlation Lengths for Disordered Systems, Phys. Rev. Lett. 57, 2999 (1986).
  * (43) A. Chandran, C. R. Laumann, and V. Oganesyan, Finite size scaling bounds on many-body localized phase transitions, arXiv: 1509.04285 (2015).
  * (44) P. Sierant, M. Lewenstein, A. Scardicchio, and J. Zakrzewski, Stability of many-body localization in Kicked Ising model, arXiv: 2203.15697 (2022).
  * (45) D.J. Luitz, N. Laflorencie, and F. Alet, Many-body localization edge in the random-field Heisenberg chain, Phys. Rev. B 91, 081103 (2015).
  * (46) C. L. Baldwin, C. R. Laumann, A. Pal, and A. Scardicchio, The many-body localized phase of the quantum random energy model, Phys. Rev. B 93, 024202 (2016).
  * (47) D. J. Luitz, Long tail distributions near the many-body localization transition, Phys. Rev. B 93, 134201 (2016).


First give the current screenshot and previous things we did a short reflection, then RETURN ME THE CODE OR SPECIAL CODE I ASKED FOR. NEVER EVER RETURN ME ANYTHING ELSE.

§.§ Accessibility Tree Filtering

Since the original tree is large (usually over 1 million tokens in XML format), we
filter the accessibility tree nodes by their tags, visibilities, availabilities,
etc. The concrete rules are illustrated in
the following Table <ref>.

Criteria for keeping a11y tree nodes on Ubuntu and Windows platforms

Condition     Ubuntu     Windows

Condition     Ubuntu     Windows

Node Tags     document$*$, $*$item, $*$button, $*$heading, $*$label, $*$scrollbar, $*$searchbox, $*$textbox, $*$link, $*$tabelement, $*$textfield, $*$textarea, $*$menu,
alert, canvas, check-box, combo-box, entry, icon, image, paragraph, scroll-bar, section, slider, static, table-cell, terminal, text, netuiribbontab, start, trayclockwclass, traydummysearchcontrol, uiimage, uiproperty, uiribboncommandbar     Same as Ubuntu

Showing     True     Not Applicable

Visible     True     True

Enabled or Editable or Expandable or Checkable     True     True

Has Name or Text or Image     True     True

Position      $>= 0$        $>= 0$

Size      $> 0$        $> 0$

§.§ Set-of-Mark Implementation Details

Our methodology involves an initial transformation of the original observational image acquired from our environment into the standardized SoM format and be putted into VLMs together with a table of the marks with metadata information such as tags and names.
This format consists of bounding boxes that bound the sub-images of interest, each associated with a corresponding integer mark.
Notably, our approach incorporates the utilization of the accessibility tree to identify the bounding boxes associated with all clickable buttons within the current image observation, instead of using segmentation models like the original SoM paper.
Some examples of screenshots after applying SoM are shown in Figure <ref>.
We can observe the emergence of some low-quality, unmodelable tasks, and even misleading bounding boxes, depending on the level of support from the software ecosystem.
This could be another reason for the poor performance of SoM. Future improvements could be made in this regard.

Showcase of example screenshots marked by SoM across different app GUI.

§.§ Full Results of Baseline Methods

Here we show the break-down results of baseline methods from different LLMs and VLMs for follow-up reference.

We have also compiled the distribution of steps taken by the GPT-4V model under our four settings: Accessibility Tree (A11y Tree), Screenshot, Screenshot combined with Accessibility Tree (Screenshot+A11y Tree), and Set-of-Mark.
This data (as shown in Fig. <ref>) provides potential clues for future work.
Overall, there are observable differences in how many steps the agent chooses to execute and when it decides to terminate under different settings.
More detailed control and analysis of these behaviors can be explored in subsequent research efforts.

Distribution of steps taken by the GPT-4V based agents across four different settings.

Detailed success rates of baseline LLM and VLM agents on , divided by apps (domains): OS, LibreOffice Calc, LibreOffice Impress, LibreOffice Writer, Chrome, VLC Player, Thunderbird, VS Code, GIMP and Workflow which is comprehensive with multiple apps, for gaining insights from interfaces and operation logics.

2*Inputs     2*Model     9cSuccess Rate (↑)

OS     Calc     Impress     Writer     VLC     TB     Chrome     VSC     GIMP     Workflow

A11y     Mixtral-8x7B      12.50     0.00     0.39     4.34      10.22    6.67     2.17     8.69     3.85     0.10

GPT-3.5       4.17     2.13     6.77     4.35     6.53     0.00     2.17     0.00     0.00     1.62

Gemini-Pro       4.17     0.00     2.13     4.35     12.41     0.00     2.17     0.00     7.69     0.63

GPT-4       20.83     0.00     6.77     4.35     23.53     26.67     26.09     30.43     23.08     2.97

Screen     CogAgent        4.17     0.00     0.00     4.34     6.53    0.00     2.17     0.00     0.00     0.00

Gemini-ProV      8.33     0.00     6.77     4.35     12.41     0.00     6.52     8.70     23.08     2.08

GPT-4V        12.50     0.00     2.52     4.35     18.34     0.00     6.52     0.00     7.69     6.04

Claude-3-Opus      4.17     0.00     2.52     4.34     6.53    0.00     2.17     0.00     3.84     2.61

Screen     CogAgent      4.17     2.17     0.00     4.35     6.53    0.00     2.17     0.00     0.00     0.10

+ A11y     Gemini-ProV      4.17     2.13     6.77     4.35     18.30     0.00     4.35     0.00     0.00     1.52

GPT-4V      16.67     0.00     6.77     21.73     24.18     33.33     21.74     21.74     15.38     4.59

Claude-3-Opus      12.5     2.13     4.65     4.34     18.30    0.00     2.17     8.69     7.69     0.99

SoM     CogAgent      4.17     2.17     0.00     4.34     6.53     0.00     2.17     0.00     0.00     0.00

Gemini-ProV      4.17     0.00     0.39     4.34     6.53     0.00     0.00     0.00     0.00     0.63

GPT-4V      8.33     8.51     6.38     13.04     10.66     33.33     23.91     17.39     11.54     6.57

Claude-3-Opus      12.5     2.13     0.39     8.70     6.53    13.33     17.39     0.00     11.54     4.49

2cHuman Performance      75.00     61.70     80.85     73.91     70.59     46.67     78.26     73.91     73.08     73.27

§ EXAMPLES OF QUALITATIVE ANALYSIS

Here we give the specific code execution process for the examples in the qualitative analysis in <ref> and add more error cause analysis, trying to get insights for further improvement.

§.§ Success and Failure Cases

[title=Success Task: I downloaded an episode of Friends to practice listening, but I don't know how to remove the subtitles. Please help me remove the subtitles from the video and export it as "subtitles.srt" and store it in the same directory as the video.]

# Action 1
import pyautogui
import time
# Move the mouse to the top bar of the media player window to access the menu
pyautogui.moveTo(800, 10)
# Assuming the menu is at the top of the screen

# Action 2
# Move the mouse to the Activities corner to open the GNOME Activities Overview
pyautogui.moveTo(10, 10)
# Assuming the Activities corner is at the top-left
# Type `terminal' to search for the Terminal application
# Press Enter to open the Terminal application

# Action 3
# Focus the Terminal window and type the command to extract subtitles
pyautogui.click(640, 360)
# Clicking in the center of the screen to focus the Terminal
pyautogui.write(`ffmpeg -i video.mp4 -map 0:s:0 subtitles.srt')

# Action 4
# Focus the Terminal window and type the command
to create a new video without subtitles
pyautogui.click(1100, 350)
# Clicking inside the Terminal window to focus it
pyautogui.write(`ffmpeg -i video.mp4 -c copy -sn no_subtitles_video.mp4')

[title=Failure Task: Help me center align the heading in LibreOffice.]

# Action 1
import pyautogui
import time
# Move the mouse to the LibreOffice Writer document icon and double-click
pyautogui.moveTo(1100, 720, duration=1)

# Action 2
# Move the mouse to the beginning of the heading text
and click and drag to select the heading
pyautogui.moveTo(200, 230, duration=1)
pyautogui.moveTo(800, 230, duration=1)
# Move the mouse to the center align button in the toolbar and click
pyautogui.moveTo(400, 150, duration=1)

# Action 3
# Assuming the text is still selected,
move the mouse to the center align button in the toolbar and click
pyautogui.moveTo(400, 150, duration=1)

# Action 4
# Move the mouse to the center align button in the toolbar and click
pyautogui.moveTo(340, 80, duration=1)

The figure shows examples of usage errors in three types of software. The first row is for GIMP, the second row is for LibreOffice Calc, and the third row is for Google Chrome. Detailed code execution processes are provided below.

[title=GIMP Task: Could you tone down the brightness of my photo?]

# Action 1
import pyautogui
import time
# Coordinates for the image icon on the desktop
# Adding half of the width and height to get the center
image_icon_x, image_icon_y = 1793 + 60, 920 + 17
# Double-click on the image icon
pyautogui.doubleClick(image_icon_x, image_icon_y)

# Action 2
# Coordinates for the `Colors' menu
# Adding half of the width and height to get the center
colors_menu_x, colors_menu_y = 344 + 28, 64 + 12
# Click on the `Colors' menu
pyautogui.click(colors_menu_x, colors_menu_y)

# Action 3
# Coordinates for the `Brightness-Contrast...' option in the `Colors' dropdown
# Assuming each menu item is approximately 28 pixels apart
brightness_contrast_option_x, brightness_contrast_option_y
= 344 + 28, 64 + 12 + 28 * 6
# Click on the `Brightness-Contrast...' option
pyautogui.click(brightness_contrast_option_x, brightness_contrast_option_y)

# Action 4
# Coordinates for the `Cancel' button in the `Shadows-Highlights' dialog
# Adding half the width and height to get the center
cancel_button_x, cancel_button_y = 375 + 42, 625 + 16
# Click the `Cancel' button
pyautogui.click(cancel_button_x, cancel_button_y)
# Coordinates for the `Brightness-Contrast...' option in the `Colors' dropdown
# Adjusted index for Brightness-Contrast option
brightness_contrast_option_x, brightness_contrast_option_y
= 344 + 28, 64 + 12 + 28 * 5
pyautogui.click(brightness_contrast_option_x, brightness_contrast_option_y)

# Action 5
# Coordinates for the `Cancel' button in the `Exposure' dialog
# Adding half the width and height to get the center
cancel_button_x, cancel_button_y
= 375 + 42, 391 + 16
pyautogui.click(cancel_button_x, cancel_button_y)
# Coordinates for the `Brightness-Contrast...' option in the `Colors' dropdown
# Adjusted index for Brightness-Contrast option
brightness_contrast_option_x, brightness_contrast_option_y
= 344 + 28, 64 + 12 + 28 $*$ 6
pyautogui.click(brightness_contrast_option_x, brightness_contrast_option_y)

# Action 6
# Coordinates for the `Cancel' button in the `Color Balance' dialog
# Adding half the width and height to get the center
cancel_button_x, cancel_button_y
= 375 + 42, 588 + 16
pyautogui.click(cancel_button_x, cancel_button_y)
# Since the `Brightness-Contrast...' option was not correctly opened before,
we will use an estimated position based on the dropdown menu.
# Assuming `Brightness-Contrast...' is the second option in the `Colors' menu.
brightness_contrast_option_x, brightness_contrast_option_y
= colors_menu_x, colors_menu_y + 28 * 2
pyautogui.click(brightness_contrast_option_x, brightness_contrast_option_y)

# Action 7
# Coordinates for the `Cancel' button in the `Color Temperature' dialog
# Adding half the width and height to get the center
cancel_button_x, cancel_button_y = 375 + 42, 403 + 16
pyautogui.click(cancel_button_x, cancel_button_y)
# Since the `Brightness-Contrast...' option was not correctly opened before,
we will use an estimated position based on the dropdown menu.
# Assuming `Brightness-Contrast...' is the second option in the `Colors' menu.
brightness_contrast_option_x, brightness_contrast_option_y
= colors_menu_x, colors_menu_y + 28 * 2
# Click on the `Brightness-Contrast...' option
pyautogui.click(brightness_contrast_option_x, brightness_contrast_option_y)

# ... More Wrong Duplicate Actions; Can not find the right coordination.

# Failed

[title=Libreoffice calc Task: I now want to count the meeting cities of the three machine learning conferences in the past ten years from 2013 to 2019(including 2013 and 2019). I have listed the names and years of the conferences in excel. Please fill in the vacant locations.]

# Action 1
# Move to cell C2 which is the first empty cell under `City' for ICLR 2013
pyautogui.click(319, 222)
# Coordinates of cell C2
# Type the city name for ICLR 2013
# Press Enter to move to the next cell

# Action 2
# Move to cell C3 which is the city cell for ICML 2013
pyautogui.click(319, 268)
# Coordinates of cell C3
# Type the city name for ICML 2013
# Press Enter to move to the next cell

# Action 3
# Move to cell C4 which is the city cell for NeurIPS 2013
pyautogui.click(319, 289)
# Coordinates of cell C4
# Type the city name for NeurIPS 2013
pyautogui.write(`Lake Tahoe')
# Press Enter to move to the next cell

... More copy-and-paste operations

# Action 15
# Move to cell C16 which is the city cell for NeurIPS 2017
pyautogui.click(319, 541)
# Coordinates of cell C16
# Type the city name for NeurIPS 2017
pyautogui.write(`Long Beach')
# Press Enter to move to the next cell

[title=Chrome Task: Can you make Bing the main search thingy when I look stuff up on the internet?]

# Action 1
pyautogui.click(x=1280, y=50)
# Approximate coordinates for the `More' button
pyautogui.move(0, 150)
# Moving the cursor down to `Settings' in the drop-down menu
# Clicking on `Settings'

# Action 2
# Click on the `Finish update' button to
approximate the location of `More' button
pyautogui.move(-50, 0)
# Move left from the `Finish update' button to approximate the `More' button
# Click on the `More' button
pyautogui.move(0, 150)
# Move down to the `Settings' option in the drop-down menu
# Click on `Settings'

# Action 3
# Click on the `Customise Chrome' button which should be the `More' button
# Wait for the dropdown menu to appear
# The `Settings' option is usually located towards the bottom of
the dropdown menu, so I'll move the cursor down to reach it
pyautogui.move(0, 200)
# Move down to the `Settings' option in the drop-down menu
# Click on `Settings'

# Action 4
# Click on the `Close' button to close the `Customise Chrome' menu
# Click on the `New Tab' button to open a new tab
# Wait for the new tab to open
# Type the direct URL to the search engine settings page

# Action 5
# Simulate pressing the `Tab' key multiple times to navigate through the options
for _ in range(10):
# Assuming it might take around 10 tabs to reach the search engine list
# Wait for the focus to move to the next element
# Simulate pressing the `Enter' key to select an option,
which would be done once we reach the Bing option

# Action 6
# Click on the `Cancel' button to exit the edit mode for the '360' search engine

§.§ Common Errors by GPT-4V Agents

In the following two figures ( Figure <ref> and Figure <ref>), we take the web page tasks as examples to show the two most common types of errors in GPT4-V:
mouse click inaccuracies and inadequate handling of environmental noise.

The error shown in the figure is due to mouse click inaccuracies. The agent was supposed to click on the product category images of the Nike official store and follow the instructions to search for women's jerseys priced over 60 dollars. However, due to a coordinate calculation error, it mistakenly clicked on the favorite button, creating a significant discrepancy between the current state and the target state. Under these circumstances, the agent is unable to backtrack to the previous state and start over.

The error shown in the figure is due to inadequate handling of environmental noise. When the agent encounters web pages with cookies or pop-up ads, its unfamiliarity with the structure of such pages leads to attempts to interact directly with the web page without bypassing the cookies or pop-ups. Since the pop-ups or cookie requests are not closed, all actions are ineffective.

§.§ Discrepancies in Task Difficulty between Agent and Human

Supplementary examples of tasks performed by humans and agents.

[title=Human > Agent Task: erase all the highlighted marks in this document]

# Action 1
import pyautogui
import time
# Move to the `Character Highlighting Color' button and click
pyautogui.moveTo(1020, 122)

# Action 2
# Assuming the `No Color' option is the first option in the dropdown
# Move the mouse slightly below the `Character Highlighting Color' button
to click on `No Color'
pyautogui.moveTo(1020, 180)

# Action 3
# Move to the `Save' button and click
pyautogui.moveTo(184, 108)

# Failed.

[title=Human > Agent Task: On page 2, add a note same to the title into the slide. Make the font of title bold.]

# Action 1
# Click on the vertical scrollbar to scroll down and reveal more slides

# Action 2
# Click on the second slide thumbnail to select it

# Action 3
# Click on the text box to add text
pyautogui.typewrite(`Team Members', interval=0.1)
# Type the title
pyautogui.hotkey(`ctrl', `a')
# Select all text
pyautogui.hotkey(`ctrl', `b')
# Make the text bold

# Action 4
# Click on the Bold button to apply bold formatting

[title=Agent > Human Task: use GIMP to cut out the 2s to 4s part of a video]

# Action 1
import pyautogui
import time
# Move the mouse to the assumed terminal icon position
at the bottom of the dock and click to open it
# Assuming the terminal icon is at the bottom of the dock
pyautogui.moveTo(70, 1350, duration=1)
# Type the command to use ffmpeg to trim the video
ffmpeg_command = `ffmpeg -ss 00:00:02 -to 00:00:04 -i /home/user/fullvideo.mp4
-c copy /home/user/trimmedvideo.mp4'
pyautogui.write(ffmpeg_command, interval=0.05)

# Action 2
# Close the search dialog
# Open the terminal using a keyboard shortcut (Ctrl+Alt+T is a common shortcut
for opening a terminal in many Linux distributions)
pyautogui.hotkey('ctrl', 'alt', 't')

# Action 3
# Move the mouse to the terminal and click to focus on the terminal window
pyautogui.moveTo(260 + 722/2, 103 + 434/2, duration=1)
# Type the command to use ffmpeg to trim the video
pyautogui.write(ffmpeg_command, interval=0.05)

# Done, but not follow the demanding in instruction.

[title=Agent > Human Task: I want to understand the resource usage of my Ubuntu system under normal workloads. Please use the `sar` command in the `sysstat` toolkit to monitor system activity, evaluate the status once every second for 30 seconds, output the results to "System Resources Report.txt" under Desktop.]
# Action 1
import pyautogui
import time
# Press the keyboard shortcut to open the terminal
pyautogui.hotkey(`ctrl', `alt', `t')
# Wait a moment for the terminal to open

# Action 2
# Click inside the terminal to focus on it
pyautogui.click(600, 300)
# Assuming the terminal is in the upper left area of the screen
n# Type the command to monitor system activity
pyautogui.typewrite(`sar 1 30 >  /Desktop/System Resources Report.txt')

# Action 3
# Click on the `System Resources Report.txt' file to select it
pyautogui.click(1850, 940)
# Coordinates taken from the label position and size
# Right-click to open the context menu
pyautogui.click(1850, 940, button=`right')
# Move to the `Open With Other Application' option in the context menu
pyautogui.move(0, 70)
# Assuming `Open With Other Application' is 70 pixels
down from the right-click position
# Click on `Open With Other Application'
# Type `gedit' to select the text editor
# Press `Enter' to open the file with gedit

# Action 4
# Click inside the terminal to ensure it's in focus
pyautogui.click(500, 200)
# Coordinates are estimated to be within the terminal window
# Type the command to open the file with gedit
pyautogui.typewrite(`gedit  /Desktop/System Resources Report.txt')

[title=Agent > Human Task: Hey, my LibreOffice Writer seems to have frozen and I can't get it to close normally. Can you help me force quit the application from the command line? I'm on Ubuntu and I don't want to restart my computer or lose any other work I have open.]

# Action 1
import pyautogui
import time
# Open a terminal window
pyautogui.hotkey(`ctrl', `alt', `t')
# Type the command to kill all instances of LibreOffice Writer
pyautogui.typewrite(`killall soffice.bin')

# Action 2
# Open a terminal window
pyautogui.hotkey(`ctrl', `alt', `t')
# Type the command to kill all instances of LibreOffice Writer
pyautogui.typewrite(`killall soffice.bin')

# Action 3
# Ensure the terminal window is focused
pyautogui.click(500, 200)
# Assuming the terminal is at this position
# Type the pkill command to forcefully kill the LibreOffice Writer application
pyautogui.typewrite(`pkill -9 soffice.bin')

§.§ Comparative Analysis: Claude-3 vs. GPT-4V

Screenshots of error example analysis of Claude-3 Opus.

[title=Claude3 Failed Task1: I remember there is a file named "secret.docx" on this computer, but I can't remember where it is. Please find the path where this file is stored and copy it to the clipboard.]

# Action 1
# click the Files icon to open the file manager

# Action 2
# click the search icon to open the search bar

# Action 3
# type the file name to search for
# press enter to start the search

# Action 4
# double click the secret.docx file to select it
pyautogui.hotkey(`ctrl', 'c')
# press Ctrl+C to copy the selected file path to clipboard

[title=Claude3 Failed Task2: copy the data in Cell B6 in this Libreoffice Calc file and search it in the Chrome browser.]

# Action 1
import pyautogui
import time
# Scroll down to view more search results if needed
# The answer appears to be in the snippet under the first search result
# `TV 31. u76f4u64adu4e2d.' which translates to `TV 31. Live.'
# So channel 31 in Hong Kong is RTHK TV 31
print(`Found the answer. Channel 31 in Hong Kong is RTHK TV 31.')
# The original task of copying cell B36 from the spreadsheet
and searching it in the browser is complete
print(`Task complete.')

# Action 2
# Activate the Chrome browser window
# Click on the address/search bar
# Paste the copied value `Dewitt'
pyautogui.hotkey(`ctrl', `v')
# Press Enter to perform the search
print(`Searched for `Dewitt' in the Chrome browser.')

# Action 3

# Action 4

[title=Claude3 Failed Task3: Please help me change all the places in this document that say `text' to `test'.]

# Action 1
pyautogui.hotkey(`ctrl', `h')

# Action 2

# Action 3
pyautogui.hotkey(`ctrl', `s')

§.§ Analysis on SoM Effectiveness

[title=Task w/o SoM: turning off the code checking function of the python plugin in vscode]

# Action 1
import pyautogui
import time
# Move the mouse to the gear icon (Manage) and click it
pyautogui.moveTo(472, 1034, duration=0.5)

# Action 2
# Move the mouse to the S̈ettingsm̈enu item and click it
# A rough central point of the `Settings' menu item
pyautogui.moveTo(497 + 385/2, 832 + 25/2, duration=0.5)

# Action 3
# Move the mouse to the search settings bar and click it
# A rough central point of the search settings bar
pyautogui.moveTo(783 + 587/2, 390 + 19/2, duration=0.5)
# Type the query to search for Python-related settings
pyautogui.write('python', interval=0.05)

# Action 4
# Move the mouse to the `Edit in settings.json' button and click it
# A rough central point of the button
pyautogui.moveTo(990 + 116/2, 624 + 15/2, duration=0.5)

# Action 5
# Move the cursor to the correct position
pyautogui.moveTo(889 + 1/2, 432 + 1/2, duration=0.5)
# Type the configuration to disable Python missing imports error reporting
pyautogui.write(`reportMissingImports: `none”, interval=0.05)
# Save the changes
pyautogui.hotkey(`ctrl', `s')

# Done

[title=Task w/ SoM:   turning off the code checking function of the python plugin in vscode]

# Action 1
import pyautogui
import time
pyautogui.hotkey(`ctrl', `shift', `p')

# Action 2
pyautogui.write(`Preferences: Open Settings (UI)', interval=0.05)

# Action 3
# Click on the search settings input field
pyautogui.click(x=600, y=175)
# These coordinates may need to be adjusted for your screen resolution
# Type `Python' into the search field
pyautogui.write(`Python', interval=0.05)

# Action 4
# Click on the search settings bar and type in the query again
pyautogui.click(x=800, y=150)
# Click on the search bar, the coordinates may need adjustment
pyautogui.typewrite(`Python missing imports', interval=0.25)

# ... Can not find the right click area

# Failed

§.§ Analysis on SoM Effectiveness

The SoM, effective for web tasks focusing only on labeled bounding boxes, did not excel in our diverse tasks that required more expertise and flexibility. It shortens the action space of the agent and thus hinders the agent's exploration and adaptability.
For example in Figure <ref>, in the task of "turning off the code checking function of the python plugin." in VS Code, agents without SoM succeeded by editing the settings.json, whereas SoM-equipped agents struggled with finding and deselecting the checkbox in the settings.
The latter has a longer action path and fails due to incorrect mouse clicks.
Furthermore, SoM's labeling of elements in professional software also has such errors, that agents without SoM can use a11y tree to calculate and adjust coordinates, while the elements' blocks under SoM can be hard to change during the task.

A task showcase where the SoM-equipped agent tried to find interactive settings, while the screen-a11ytree-equipped agents directly modified the value in the setting.json file.