# Learning effective Hamiltonians for adaptive time-evolution quantum
algorithms

Hongzheng Zhao<EMAIL_ADDRESS>School of Physics, Peking University, 100871
Beijing, China.  Max Planck Institute for the Physics of Complex Systems,
Nöthnitzer Str. 38, 01187 Dresden, Germany.  Ao Chen  Theoretical Physics III,
Center for Electronic Correlations and Magnetism, Institute of Physics,
University of Augsburg, 86135 Augsburg, Germany.  Shu-Wei Liu  Max Planck
Institute for the Physics of Complex Systems, Nöthnitzer Str. 38, 01187
Dresden, Germany.  Marin Bukov  Max Planck Institute for the Physics of
Complex Systems, Nöthnitzer Str. 38, 01187 Dresden, Germany.  Markus Heyl  Max
Planck Institute for the Physics of Complex Systems, Nöthnitzer Str. 38, 01187
Dresden, Germany.  Theoretical Physics III, Center for Electronic Correlations
and Magnetism, Institute of Physics, University of Augsburg, 86135 Augsburg,
Germany.  Roderich Moessner  Max Planck Institute for the Physics of Complex
Systems, Nöthnitzer Str. 38, 01187 Dresden, Germany.

###### Abstract

Digital quantum simulation of many-body dynamics relies on Trotterization to
decompose the target time evolution into elementary quantum gates operating at
a fixed equidistant time discretisation. Recent advances have outlined
protocols enabling more efficient adaptive Trotter protocols, which have been
shown to exhibit a controlled error in the dynamics of local observables and
correlation functions. However, it has remained open to which extent the
errors on the actual generator of the dynamics, i.e., the target many-body
Hamiltonian, remain controlled. Here, we propose to use quantum Hamiltonian
learning to numerically obtain the effective Hamiltonian and apply it on the
recently introduced ADA-Trotter algorithm as a concrete demonstration. Our key
observation is that deviations from the target generator remain bounded on all
simulation times. This result suggests that the ADA-Trotter not only generates
reliable digital quantum simulation of local dynamics, but also controllably
approximates the global quantum state of the target system. Our proposal is
sufficiently general and readily applicable to other adaptive time-evolution
algorithms.

Introduction.— Recent advances in Noisy Intermediate Scale Quantum (NISQ)
devices have charted new pathways in the study of time-evolving quantum
systems. Of particular interest is the digital quantum simulation (DQS) of
many-body dynamics far away from equilibrium – a central short-term
application of quantum computation and simulation [1, 2]. Pioneering
demonstrations of DQS have been achieved on various platforms, e.g.,
superconducting qubits [3, 4, 5, 6, 7, 8], trapped ions [9, 10, 11, 12], NV
centers [13, 14] and Rydberg circuits [15, 16].

DQS of real-time dynamics generated by a static target Hamiltonian
$H_{\ast}{=}\sum_{j}^{N}H_{j}$ over a short time $\tau$ normally requires a
decomposition of the time-evolution operator, e.g.,
$U(\tau){=}\prod_{j=1}^{N}\exp(-i\tau H_{j})$ – a protocol known as
Trotterization [17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31,
32, 33, 34, 35, 36, 37, 38]. However, due to noncommutativity,
$[H_{i},H_{j}]{\neq}0$ ($i{\neq}j$), this procedure inevitably generates so-
called Trotter errors. Since the repeated application of $U(\tau)$ to simulate
dynamics over a long time encodes temporal periodicity into the propagator
(akin to Floquet systems), one can construct an effective Hamiltonian
$H_{\mathrm{eff}}$ perturbatively in $\tau$, such that $U(\tau){=}\exp(-i\tau
H_{\mathrm{eff}})$ [39, 40, 41]. $H_{\mathrm{eff}}$ provides crucial insights
to analyze Trotter errors, leading to prominent findings, e.g., rigorous
bounds on Trotter errors [41, 42, 43, 44], Trotter thresholds from
localization to chaos [45, 46, 47, 48], and error mitigation strategies for
efficient implementation of DQS in practice [49, 50].

Very recently there has been considerable progress in developing adaptive
Trotter algorithms [51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61]. By measuring
certain quantifiers of an input quantum state (such as local observables of
interest or conservation laws), one can adaptively update Trotter protocols
and distribute the quantum computation resources, e.g., quantum gates, more
efficiently. Hence, adaptive algorithms promise to improve the performance of
DQS in the NISQ era substantially. Various strategies have been proposed, such
as adaptively modifying the specific form of the Trotter decomposition [56],
the step size $\tau$ [57, 59], or the variational manifold for the time-
evolved quantum state [52].

In this work, we raise the question of how to construct effective Hamiltonians
for adaptive Trotter algorithms. Addressing this issue is pivotal to enhancing
the stability, efficiency, and applicability of various adaptive algorithms,
thereby augmenting their utility on current NISQ platforms. However,
identifying an effective Hamiltonian for adaptive algorithms faces key
challenges. (i) Adaptive updates explicitly depend on input quantum states and
one protocol optimized w.r.t. certain states may completely fail for others.
This compromises the very existence of an effective Hamiltonian. (ii) Even if
the latter exists, the well-established theoretical framework to
perturbatively identify Trotter errors based on, e.g., Floquet-Magnus
expansion which is independent of quantum states, does not apply.

Here, we aim to address these problems by harnessing the idea of Quantum
Hamiltonian Learning (QHL) [62]: it was originally proposed to learn the
Hamiltonian of a black-box quantum device; here we adapt it as a theoretical
scheme to deepen our understanding of various adaptive algorithms. The core
idea is based on quantum state reconstruction (QSR), namely, we numerically
learn a suitable Hamiltonian $H_{\text{eff}}^{[t]}$ such that the quantum
states obtained by running the adaptive algorithm can be reconstructed from
the time evolution generated by $H_{\text{eff}}^{[t]}$. Note,
$H_{\text{eff}}^{[t]}$ is static but the adaptive updates make it
parametrically dependent on time $t$: once $t$ is fixed,
$H_{\text{eff}}^{[t]}$ is also fixed throughout the entire time evolution over
the interval $[0,t]$.

Albeit intuitive, QSR is generally a resource-demanding computational task in
practice, if there is no prior knowledge of the quantum system. This is
particularly costly in a many-body setup where the optimization parameter
space grows exponentially with the system size. More efficient methods have
been developed for static and driven systems which are not adaptive [63, 64,
65, 66, 67, 68, 69, 70, 71, 72, 73, 74]. For instance, by measuring energy or
local observables for an ensemble of initial states at various times, the
Hamiltonian can be learned efficiently [65, 67, 68]. Unfortunately, these
methods are not applicable here since the adaptive protocol explicitly depends
on both time and the input quantum states.

The main finding of this work is to show that for interacting systems
involving geometrically local and few-body operators, QHL is indeed a natural
tool to identify effective Hamiltonians. For a concrete demonstration, we
focus on ADA-Trotter algorithm with adaptive step sizes [57]. However, the
following QHL routine is expected to be sufficiently general and readily
applicable to other adaptive time-evolution algorithms. Leveraging the fact
that the adaptive time evolution is dominated by a target Hamiltonian together
with the intuition that most relevant Trotter errors should be sufficiently
local, we first propose a Hamiltonian ansatz to parametrize
$H_{\text{eff}}^{[t]}$. It involves terms that are local in space and obey the
symmetry constraint imposed by the Trotter decomposition. We optimize it for
state reconstruction via a Gradient Descent (GD) algorithm [75, 76].
Crucially, we find that its efficiency can be remarkably improved by recycling
the optimized outcome as the starting point for the optimization onwards. This
GD results in high-quality QSR for the entire time evolution, cf. Fig. 1.

QHL provides valuable insights for Trotter errors in adaptive algorithms that
are not accessible otherwise. As a key result, we show that although
$H_{\mathrm{eff}}^{[t]}$ parametrically depends on time, it always stays close
to the target Hamiltonian, cf. Fig. 2. Hence, ADA-Trotter not only generates
reliable DQS of local dynamics, but it also closely mimics global properties
of the target many-body system. Further, by analysing the statistical behavior
of errors terms, we show that leading Trotter errors derived from the Floquet-
Magnus expansion still contribute notably, despite step sizes now being
adaptive in time.

Methods.— We first briefly review the adaptive algorithm and elaborate on the
QHL method and the associated optimization process. ADA-Trotter was first
introduced in Ref. [57] for DQS of quantum many-body dynamics generated by a
target Hamiltonian $H_{\ast}.$ For concreteness, we consider a target non-
integrable quantum Ising model, $H_{\ast}{=}H_{z}{+}H_{x}$, for a chain of
length $L$:

$H_{z}{=}J_{z}\sum\nolimits_{j}Z_{j}Z_{j+1}{+}h_{z}\sum\nolimits_{j}Z_{j},\ \
H_{x}{=}h_{x}\sum\nolimits_{j}X_{j},$ (1)

with Pauli operators $Z_{j},X_{j},Y_{j}$ and periodic boundary conditions. A
second order Trotter protocol,
$U(\tau_{m})=e^{-i\tau_{m}H_{x}/2}e^{-i\tau_{m}H_{z}}e^{-i\tau_{m}H_{x}/2},$
is used here but generalization to other Trotter decompositions is
straightforward. An effective Hamiltonian for a single step can be defined via
the relation $U(\tau_{m}){:=}\exp(-i\tau_{m}H_{\mathrm{eff}})$. For a small
step $\tau_{m}$, the Baker–Campbell–Hausdorff (BCH) yields

$\displaystyle H_{\text{eff}}\approx
H_{\ast}{+}\frac{\tau_{m}^{2}}{24}\left([[H_{z},H_{x}],H_{x}]{+}2[[H_{z},H_{x}],H_{z}]\right),$
(2)

with error $\mathcal{O}(\tau_{m}^{4})$. The leading order matches the target
Hamiltonian. ADA-Trotter automatically chooses the largest step sizes
$\tau_{m}$ to evolve the system, as long as expectation values of energy
density and variance are preserved within preset tolerances
$d_{\mathcal{E}},d_{\delta\mathcal{E}^{2}}$, which are the control parameters
of the algorithm and set the simulation error. More precisely, for a given
state $|\psi(t_{m})\rangle$, by a feedback process we search for the largest
possible time step $\tau_{m}$, such that, in the time-evolved state
$|\psi(t_{m}{+}\tau_{m})\rangle{=}U(\tau_{m})|\psi(t_{m})\rangle$, the energy
density
$\mathcal{E}_{m+1}{=}L^{-1}\langle\psi(t_{m}{+}\tau_{m})|H_{\ast}|\psi(t_{m}{+}\tau_{m})\rangle$
and its fluctuations density
$\delta\mathcal{E}^{2}_{m+1}{=}L^{-1}\langle\psi(t_{m}{+}\tau_{m})|H_{\ast}^{2}|\psi(t_{m}{+}\tau_{m})\rangle-L\mathcal{E}^{2}_{m+1}$
both remain bounded: $|\mathcal{E}_{m+1}{-}\mathcal{E}|{<}d_{\mathcal{E}}$,
$|\delta\mathcal{E}^{2}_{m+1}-\delta\mathcal{E}^{2}|{<}d_{\delta\mathcal{E}^{2}},$
where $\mathcal{E},\delta\mathcal{E}^{2}$ are energy and variance density
calculated with respect to the initial state $\left|\psi(0)\right\rangle$
[57], which evolves to $\left|\psi_{\text{ADA}}(t)\right\rangle$ at time $t$
after a few steps.

The perturbative expansion in Eq. 2 only works for each small time duration
$\tau_{m}$. For two consecutive steps with different $\tau_{m}$ and
$\tau_{m^{\prime}}$, Eq. 2 does not hold for the new Hamiltonian
$H^{\prime}_{\text{eff}}$, defined via
$\exp[-i(\tau_{m}{+}\tau_{m^{\prime}})H^{\prime}_{\mathrm{eff}}]=U(\tau_{m})U(\tau_{m^{\prime}})$
for duration $\tau_{m}{+}\tau_{m^{\prime}}$. Our central goal is to identify
one appropriate effective Hamiltonian $H_{\text{eff}}^{[t]}$, such that
$\left|\psi_{\text{ADA}}(t)\right\rangle$ can be reconstructed by the
effective time evolution,
$\left|\psi_{\text{L}}(t)\right\rangle{=}\exp(-itH_{\text{eff}}^{[t]})\left|{\psi}(0)\right\rangle$
from the same initial state. This method avoids the resource demanding quantum
process tomography, hence permitting more efficient QHL for large system
sizes.

We now assume that the effective Hamiltonian has the following form,
$H_{\text{eff}}^{[t]}{=}\sum_{\alpha}C^{[t]}_{\alpha}\mathcal{O}_{\alpha},$
where $\alpha$ denotes a Pauli string, and $\mathcal{O}_{\alpha}$ represents
the corresponding tensor product of Pauli matrices. For instance, the
coefficient for the term $\sum_{i}Z_{i}Z_{i+1}$ is $C_{ZZ}$ and from now on we
will drop the parametric dependence on time for simplicity. Therefore, QHL is
recast in identifying the most relevant terms in such an ansatz. However, the
number of all possible terms (even after fully exploiting the translation and
inversion symmetry of the system) on neighbouring $R$ sites scales
exponentially in the spatial support $R$. Hence, determining these
coefficients even for systems of small size can still be resource-demanding.
Although Eq. (2) does not apply straightforwardly in defining
$H_{\text{eff}}$, it actually indicates that Trotter errors are generated
locally in space, via the form of nested commutators of $H_{x}$ and $H_{z}$.
Hence, a substantial simplification for QHL can be made by truncating the
ansatz to all possible terms in a finite and small support $R$. In the
following, we choose $R=5$, corresponding to a $N=207$ operator basis, which
is sufficiently large to capture the most relevant local Trotter errors while
sufficiently small to efficiently implement the GD algorithm to be introduced
below.

For perfect construction of $H_{\text{eff}}$, the state infidelity
$(1{-}|\langle\psi_{\text{ADA}}(t)\left|\psi_{\text{L}}(t)\right\rangle|^{2})$
vanishes. Therefore, it provides a natural loss function to quantify the
learning error and optimize over the possible parameter space by a GD
algorithm. Importantly, since the step sizes are generally smaller than the
local energy scale of the target system, we expect that $C_{\alpha}$ does not
change abruptly between two neighboring steps. Therefore, by using the learned
coefficients obtained at one previous time step as the starting optimization
point for the next time step, one can significantly improve the GD efficiency.
For the first step, we set the starting optimization point as
$(C_{X},C_{Z},C_{ZZ}){=}(h_{x},h_{z},J_{zz})$ – the same as in the target
Hamiltonian – and all other coefficients are zero. More details about our GD
implementation are given in SM.

Generally the solution for $H_{\text{eff}}$ identified via QSR is not unique,
simply because terms in the Hamiltonian which have negligible projection onto
the initial states barely cause any dynamical effects. During the optimization
process, they do not make much of a difference to the loss function, hence GD
in effect concentrates on terms that actually dominate Trotter errors.

Figure 1: (a) Comparison between the magnetization of the quantum state
obtained by ADA-Trotter and Hamiltonian learning. Deviations between the two
in local magnetization are barely visible, implying a high-quality state
reconstruction by the effective Hamiltonian. $N=207$ for the complete basis.
Learning error in the three-site reduced density matrix (b) and the overall
quantum state infidelity (c). Increasing the basis improves the learning
outcome. A quantum Ising model is used with parameters
$L=16,J_{z}=-1,h_{x}=-1.7,h_{z}=0.5$ and total Trotter steps $M=50$ for
numerical simulation. The tolerances in ADA-Trotter are
$(d_{\mathcal{E}},d_{\delta\mathcal{E}^{2}})=(0.02,0.01)$.

Results.— We now numerically implement the ADA-Trotter algorithm and confirm
that GD can reach a sufficiently small learning error. Then we will discuss
the behavior of $C_{\alpha}$ and its implication for ADA-Trotter.

In Fig. 1(a) we plot the dynamics of the local magnetization
$M_{z}{=}\sum_{j}Z_{j}$ (black line) computed for the exact target system,
starting from the initial state
$\exp(-i\theta_{y}\sum_{j}Y_{j})|\downarrow\dots\downarrow\rangle$ with
$\theta_{y}{=}\pi/3$ 111More numerical results with different initial states
are shown in the SM.. $M_{z}$ oscillates coherently at early times and the
oscillation amplitude gradually decays as is typical for many-body systems due
to quantum thermalization. Due to finite size effects, even at long times
these oscillations persist but their amplitude is expected to vanish for
$L{\to}\infty$. As shown by the blue dots, ADA-Trotter closely mimics the
exact dynamics, and deviations only become noticeable at long times
($t{\geq}8$). In fact, these deviations can be systematically suppressed by
using smaller tolerances of the algorithm.

Figure 2: Learned coefficients in the effective Hamiltonian. (a) Time trace of
the learned coefficients with the target highlighted as a black star.
Trajectories cluster in a finite region and their deviations from the target
remain bounded. ADA-Trotter thus induces an effective Hamiltonian that closely
mimics the target Hamiltonian. (b) These coefficients change in time and
always remain close to target values (grey). We use the same Hamiltonian
parameters as in Fig. 1 and tolerance values
$(d_{\mathcal{E}},d_{\delta\mathcal{E}^{2}})$ are shown in the plot. Basis
size $N=106$ in (a) and 207 in (b).

We now quantify the accuracy of QSR by analyzing the dynamics of $M_{z}$,
reduced density matrix, and quantum state infidelity. In Fig. 1(a), we plot
the reconstructed local dynamics as red crosses, and its deviation from ADA-
Trotter (blue dots) is barely visible with a naked eye. The two-norm of the
difference between two reduced density matrix on three consecutive sites
($\rho_{\mathrm{ADA}}$ for ADA-Trotter and $\rho_{\mathrm{L}}$ for the
reconstructed state) is depicted in Fig. 1(b). With the largest learning basis
(dark green, $N{=}207$), their deviations increase in time but only at a very
slow pace. Similar behavior also occurs in the state infidelity as shown in
Fig. 1(c): even at long times $t\sim 10$ where quantum thermalization
effectively already occurs, the state infidelity still remains below 0.02. All
of these evidences ensure that the learning error is sufficiently low and
hence the construction of $H_{\text{eff}}$ is reliable.

As GD is time-consuming for a large operator basis set, we illustrate the
possibility of truncating the basis size $N$ for the improvement of the
optimization efficiency, while still maintaining a high learning quality. One
possible measure to quantify the importance of a Pauli operator in
contributing to Trotter errors is the deviation ($\Delta C_{\alpha}$) between
the learned coefficients and their target values, averaged over both time and
different tolerances when running ADA-Trotter. In SM, we sort out the operator
basis and obtain an importance hierarchy. We truncate the operator basis set
accordingly by only involving the largest $N$ terms and again perform GD. As
shown in Fig. 1(b) and (c), $N{=}16$ already leads to a decent learning
quality, e.g., around $t{=}10$ the loss function can be optimized down to
0.05. Increasing $N$ generally improves the optimization outcome and requires
longer optimization time, cf. SM for additional discussion.

Figure 3: Violin plots for the learned coefficients after the first 10 steps
for two different initial states. Leading order terms in the BCH expansion
(highlighted in red) dominate the Trotter errors for adaptive algorithms.
$\gamma$ stands for the increasing rate of the state infidelity, which remains
low for these two cases suggesting high-quality QSR, cf. SM. We use the same
Hamiltonian parameters as in Fig. 1,
$(d_{\mathcal{E}},d_{\delta\mathcal{E}^{2}})=(0.5,0.01),N=106$ for (a) and
$(d_{\mathcal{E}},d_{\delta\mathcal{E}^{2}})=(0.02,0.01),N=207$ for (b).
Figure 4: Dependence of a time-averaged error in ADA-Trotter algorithm on the
tolerances in energy variance $d_{\delta\mathcal{E}^{2}}$ (a) and energy
$d_{\mathcal{E}}$ (b). This error saturates in the shaded region with small
$d_{\delta\mathcal{E}^{2}}$, but it can be further suppressed by a smaller
$d_{\mathcal{E}}$. The red arrow points to the data obtained by the same set
of tolerance values. We fix $d_{\mathcal{E}}{=}0.02$ and
$d_{\delta\mathcal{E}^{2}}{=}0.01$ in (a) and (b), respectively. Hamiltonian
parameters are the same as in Fig. 1 and $M{=}100$, $N{=}106$.

With this QHL scheme we can explicitly construct $H_{\text{eff}}$, providing
valuable insights for Trotter errors in ADA-Trotter. In Fig. 2(a), we plot the
coefficient trajectories of the three most important terms
($C_{X},C_{Z},C_{ZZ}$) which are non-vanishing in $H_{\ast}$, for three
different tolerance values. Initially, at $t=0$, no optimization is needed and
we set $(C_{X},C_{Z},C_{ZZ})=(h_{x},h_{z},J_{zz})$ (black star) as the origin
of these trajectories. At short times the state
$\left|\psi_{\text{ADA}}(t)\right\rangle$ is close to the initial state, so GD
quickly converges and identifies the suitable parameters for QSR. The
trajectory starts deviating from the target after a short transient period.
However, crucially, at later times and for all tolerance values, trajectories
cluster in a finite region and their deviations from $H_{\ast}$ remain bounded
throughout the entire time evolution. In Fig. 2 (b), we also plot the time
trace of $C_{ZZ}$ which always stays close to its target value (grey line).
For terms that do not appear in $H_{\ast}$, e.g.,
$\sum_{i}Z_{i}Y_{i+1}Z_{i+2}$, its coefficient indeed oscillates around zero.
Therefore, we confirm that $H_{\text{eff}}$ for ADA-Trotter closely mimics the
target Hamiltonian.

Interestingly, although the ADA-Trotter violates the temporal periodicity,
terms appearing in the leading order BCH expansion, including
$X,ZXZ,XZ,YY,Z,ZZ$ (cf. SM), still dominate Trotter errors. To see this, in
Fig. 3, we depict a collection of violin plots for different learned
coefficients $C_{\alpha}$ where the Pauli strings corresponding to BCH
expansion are highlighted in red. At the middle of each violin plot the time-
averaged value ($\overline{\Delta C_{\alpha}}$) is shown, and the cap
corresponds to their extreme values. The overall shape of the violin plot is a
kernal density plot to illustrate the probability distribution of
$C_{\alpha}$. For clarity we only plot 20 terms with large $\overline{\Delta
C_{\alpha}}$. Although certain errors can exhibit large extrem values, e.g.,
$\Delta C_{Y}$ in Fig. 3(a), statistically those six terms notably contributes
to Trotter errors.

Trotter errors strongly depend on the tolerances preset in the adaptive
algorithm. For instance, as shown in Fig. 1(d), trajectory clusters drift
further away for larger tolerances. This occurs simply because larger
tolerances generally encourage ADA-Trotter to choose larger step sizes and
hence cause more simulation errors. With QHL one can now quantify such a
dependence by using $\overline{\Delta C_{\alpha}}$. In Fig. 4, we consider
three different terms and show this quantity for various energy variance
tolerances in panel (a) and energy tolerances in panel (b). We note that
deviations in both $C_{ZZ}$ (orange) and $C_{X}$ (grey) are sensitive to the
variation of the tolerance values, and a tighter tolerance generally
suppresses the error. In contrast, terms of a larger support can have
relatively smaller errors, for instance, $C_{XXZXX}$ is barely noticeable and
always remains close to zero regardless of the tolerances. In the shaded
region in Fig. 4(a), $\overline{\Delta C_{\alpha}}$ shows little dependence on
the tolerance in energy variance, indicating that there errors are mostly
induced by deviations from the energy conservation. Indeed in Fig. 4(b) when
we fix tolerance in energy variance, errors can be more effectively controlled
by tightening the energy tolerance $d_{\mathcal{E}}$.

Discussion.— Here, we use QHL to identify a suitable $H_{\text{eff}}$ for
adaptive time-evolution algorithms and apply it to ADA-Trotter as a concrete
demonstration. Although ADA-Trotter only approximately conserves the lowest
two moments of $H_{\ast}$, the precise reconstruction of $H_{\mathrm{eff}}$
suggests that the generator of the entire time evolution remains close to
$H_{\ast}$. Their deviation is also controllable upon reducing tolerances.
Therefore, ADA-Trotter generates reliable and controlled many-body states
close to the target. For other adaptive algorithms, this property is not
guaranteed and errors in $H_{\text{eff}}$ may diverge in time.

The efficiency in QHL relies on the a priori intuition of target systems and
Trotter errors. The local structure with symmetries in $H_{\ast}$, together
with the simple Trotter decomposition, greatly simplifies the parametrization
of the ansatz. For more general cases, where systems involve non-local terms
or circuit decomposition adapts variationally, a natural choice to construct
operator basis may not exist. If so, one may consider Hamiltonian structure
learning [71] or machine learning-assisted algorithms [78, 79, 80, 69] to make
QHL efficient.

It is worth emphasising that Trotter error analysis should explicitly
incorporate information of input quantum states and microscopic details of the
adaptive protocol. This is particularly challenging for quantum many-body
systems where normally only upper bounds on errors derived for worst-case
scenarios or asymptotic-in-time local errors can be obtained [42, 57]. Our
method thus can systematically benchmark different adaptive algorithms from
the effective Hamiltonian perspective on an equal footing. Indeed, not being
restricted to adaptive algorithms, it also applies to other time-dependent
protocols like aperiodically driven systems where effective Hamiltonians may
also be difficult to obtain [81, 82].

Finally, let us point out a connection between our approach and certain
concepts in the field of shortcuts to adiabaticity [83]. Notice that the
states visited by ADA-Trotter can be thought of as a trajectory in Hilbert
space; in this sense, QHL can be viewed as the problem of identifying a parent
Hamiltonian which generates a dynamics that traces the same trajectory. For
many-body states allowing few-parameter parameterizations, parent Hamiltonians
can be reconstructed efficiently using tensor networks [84] under certain
conditions [85], and the variational principle for adiabatic gauge potentials
in counterdiabatic driving [86, 84]. It is an interesting open problem how to
combine these techniques with QHL for trajectories of less structured states.

Acknowledgments.— We thank Peter Zoller for initiating this work and
stimulating discussions. We also thank Tobias Olsacher and Christian Kokail
for many useful discussions in the beginning of this project. This work is
supported by “The Fundamental Research Funds for the Central Universities,
Peking University”, and by ”High-performance Computing Platform of Peking
University” and by the Deutsche Forschungsgemeinschaft under the cluster of
excellence ct.qmat (EXC 2147, project-id 390858490). This project has received
funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (grant agreement No. 853443).
This work was supported by the German Research Foundation DFG via project
499180199 (FOR 5522). Funded by the European Union (ERC, QuSimCtrl,
101113633). Views and opinions expressed are however those of the authors only
and do not necessarily reflect those of the European Union or the European
Research Council Executive Agency. Neither the European Union nor the granting
authority can be held responsible for them.

## References

  * Georgescu  _et al._ [2014] I. M. Georgescu, S. Ashhab, and F. Nori, Quantum simulation, Reviews of Modern Physics 86, 153 (2014).
  * Daley  _et al._ [2022] A. J. Daley, I. Bloch, C. Kokail, S. Flannigan, N. Pearson, M. Troyer, and P. Zoller, Practical quantum advantage in quantum simulation, Nature 607, 667 (2022).
  * Smith  _et al._ [2019] A. Smith, M. Kim, F. Pollmann, and J. Knolle, Simulating quantum many-body dynamics on a current digital quantum computer, npj Quantum Information 5, 1 (2019).
  * Han  _et al._ [2021] J. Han, W. Cai, L. Hu, X. Mu, Y. Ma, Y. Xu, W. Wang, H. Wang, Y. Song, C.-L. Zou, _et al._ , Experimental simulation of open quantum system dynamics via trotterization, Physical Review Letters 127, 020504 (2021).
  * Guo  _et al._ [2021] Q. Guo, C. Cheng, Z.-H. Sun, Z. Song, H. Li, Z. Wang, W. Ren, H. Dong, D. Zheng, Y.-R. Zhang, _et al._ , Observation of energy-resolved many-body localization, Nature Physics 17, 234 (2021).
  * Mi  _et al._ [2022] X. Mi, M. Ippoliti, C. Quintana, A. Greene, Z. Chen, J. Gross, F. Arute, K. Arya, J. Atalaya, R. Babbush, _et al._ , Time-crystalline eigenstate order on a quantum processor, Nature 601, 531 (2022).
  * Frey and Rachel [2022] P. Frey and S. Rachel, Realization of a discrete time crystal on 57 qubits of a quantum computer, Science advances 8, eabm7652 (2022).
  * Keenan  _et al._ [2022] N. Keenan, N. Robertson, T. Murphy, S. Zhuk, and J. Goold, Evidence of kardar-parisi-zhang scaling on a digital quantum simulator, arXiv preprint arXiv:2208.12243 (2022).
  * Lanyon  _et al._ [2011] B. P. Lanyon, C. Hempel, D. Nigg, M. Müller, R. Gerritsma, F. Zähringer, P. Schindler, J. T. Barreiro, M. Rambach, G. Kirchmair, _et al._ , Universal digital quantum simulation with trapped ions, Science 334, 57 (2011).
  * Martinez  _et al._ [2016] E. A. Martinez, C. A. Muschik, P. Schindler, D. Nigg, A. Erhard, M. Heyl, P. Hauke, M. Dalmonte, T. Monz, P. Zoller, _et al._ , Real-time dynamics of lattice gauge theories with a few-qubit quantum computer, Nature 534, 516 (2016).
  * Zhang  _et al._ [2017] J. Zhang, P. W. Hess, A. Kyprianidis, P. Becker, A. Lee, J. Smith, G. Pagano, I.-D. Potirniche, A. C. Potter, A. Vishwanath, _et al._ , Observation of a discrete time crystal, Nature 543, 217 (2017).
  * Monroe  _et al._ [2021] C. Monroe, W. C. Campbell, L.-M. Duan, Z.-X. Gong, A. V. Gorshkov, P. Hess, R. Islam, K. Kim, N. M. Linke, G. Pagano, _et al._ , Programmable quantum simulations of spin systems with trapped ions, Reviews of Modern Physics 93, 025001 (2021).
  * Randall  _et al._ [2021] J. Randall, C. Bradley, F. Van Der Gronden, A. Galicia, M. Abobeih, M. Markham, D. Twitchen, F. Machado, N. Yao, and T. Taminiau, Many-body–localized discrete time crystal with a programmable spin-based quantum simulator, Science 374, 1474 (2021).
  * Beatrez  _et al._ [2023] W. Beatrez, C. Fleckenstein, A. Pillai, E. de Leon Sanchez, A. Akkiraju, J. Diaz Alcala, S. Conti, P. Reshetikhin, E. Druga, M. Bukov, _et al._ , Critical prethermal discrete time crystal created by two-frequency driving, Nature Physics 19, 407 (2023).
  * Bluvstein  _et al._ [2022] D. Bluvstein, H. Levine, G. Semeghini, T. T. Wang, S. Ebadi, M. Kalinowski, A. Keesling, N. Maskara, H. Pichler, M. Greiner, _et al._ , A quantum processor based on coherent transport of entangled atom arrays, Nature 604, 451 (2022).
  * Bluvstein  _et al._ [2024] D. Bluvstein, S. J. Evered, A. A. Geim, S. H. Li, H. Zhou, T. Manovitz, S. Ebadi, M. Cain, M. Kalinowski, D. Hangleiter, _et al._ , Logical quantum processor based on reconfigurable atom arrays, Nature 626, 58 (2024).
  * Suzuki [1991] M. Suzuki, General theory of fractal path integrals with applications to many-body theories and statistical physics, Journal of Mathematical Physics 32, 400 (1991).
  * Berry  _et al._ [2007] D. W. Berry, G. Ahokas, R. Cleve, and B. C. Sanders, Efficient quantum algorithms for simulating sparse hamiltonians, Communications in Mathematical Physics 270, 359 (2007).
  * Poulin  _et al._ [2014] D. Poulin, M. B. Hastings, D. Wecker, N. Wiebe, A. C. Doherty, and M. Troyer, The trotter step size required for accurate quantum simulation of quantum chemistry, arXiv preprint arXiv:1406.4920 (2014).
  * Babbush  _et al._ [2016] R. Babbush, D. W. Berry, I. D. Kivlichan, A. Y. Wei, P. J. Love, and A. Aspuru-Guzik, Exponentially more precise quantum simulation of fermions in second quantization, New Journal of Physics 18, 033032 (2016).
  * Vanicat  _et al._ [2018] M. Vanicat, L. Zadnik, and T. Prosen, Integrable trotterization: Local conservation laws and boundary driving, Physical review letters 121, 030606 (2018).
  * Tranter  _et al._ [2019] A. Tranter, P. J. Love, F. Mintert, N. Wiebe, and P. V. Coveney, Ordering of trotterization: Impact on errors in quantum simulation of electronic structure, Entropy 21, 1218 (2019).
  * Grimsley  _et al._ [2019] H. R. Grimsley, S. E. Economou, E. Barnes, and N. J. Mayhall, An adaptive variational algorithm for exact molecular simulations on a quantum computer, Nature communications 10, 3007 (2019).
  * Cirstoiu  _et al._ [2020] C. Cirstoiu, Z. Holmes, J. Iosue, L. Cincio, P. J. Coles, and A. Sornborger, Variational fast forwarding for quantum simulation beyond the coherence time, npj Quantum Information 6, 1 (2020).
  * Barthel and Zhang [2020] T. Barthel and Y. Zhang, Optimized lie–trotter–suzuki decompositions for two and three non-commuting terms, Annals of Physics 418, 168165 (2020).
  * Bolens and Heyl [2021] A. Bolens and M. Heyl, Reinforcement learning for digital quantum simulation, Physical Review Letters 127, 110502 (2021).
  * Lin  _et al._ [2021] S.-H. Lin, R. Dilip, A. G. Green, A. Smith, and F. Pollmann, Real-and imaginary-time evolution with compressed quantum circuits, PRX Quantum 2, 010342 (2021).
  * Richter and Pal [2021] J. Richter and A. Pal, Simulating hydrodynamics on noisy intermediate-scale quantum devices with random circuits, Physical Review Letters 126, 230501 (2021).
  * Mansuroglu  _et al._ [2021] R. Mansuroglu, T. Eckstein, L. Nützel, S. A. Wilkinson, and M. J. Hartmann, Variational hamiltonian simulation for translational invariant systems via classical pre-processing, arXiv e-prints , arXiv (2021).
  * Keever and Lubasch [2022] C. M. Keever and M. Lubasch, Classically optimized hamiltonian simulation, arXiv preprint arXiv:2205.11427 (2022).
  * Tepaske  _et al._ [2022] M. S. Tepaske, D. Hahn, and D. J. Luitz, Optimal compression of quantum many-body time evolution operators into brickwall circuits, arXiv preprint arXiv:2205.03445 (2022).
  * Chen  _et al._ [2022] Y.-A. Chen, A. M. Childs, M. Hafezi, Z. Jiang, H. Kim, and Y. Xu, Efficient product formulas for commutators and applications to quantum simulation, Physical Review Research 4, 013191 (2022).
  * Berthusen  _et al._ [2022] N. F. Berthusen, T. V. Trevisan, T. Iadecola, and P. P. Orth, Quantum dynamics simulations beyond the coherence time on noisy intermediate-scale quantum hardware by variational trotter compression, Physical Review Research 4, 023097 (2022).
  * Mansuroglu  _et al._ [2023a] R. Mansuroglu, F. Fischer, and M. J. Hartmann, Problem specific classical optimization of hamiltonian simulation, arXiv preprint arXiv:2306.07208 (2023a).
  * Granet and Dreyer [2023] E. Granet and H. Dreyer, Continuous hamiltonian dynamics on noisy digital quantum computers without trotter error, arXiv preprint arXiv:2308.03694 (2023).
  * Mansuroglu  _et al._ [2023b] R. Mansuroglu, T. Eckstein, L. Nützel, S. A. Wilkinson, and M. J. Hartmann, Variational hamiltonian simulation for translational invariant systems via classical pre-processing, Quantum Science and Technology 8, 025006 (2023b).
  * Tepaske and Luitz [2023] M. S. Tepaske and D. J. Luitz, Compressed quantum error mitigation, Physical Review B 107, L201114 (2023).
  * Trenkwalder  _et al._ [2023] L. M. Trenkwalder, E. Scerri, T. E. O’Brien, and V. Dunjko, Compilation of product-formula hamiltonian simulation via reinforcement learning, arXiv preprint arXiv:2311.04285 (2023).
  * Goldman and Dalibard [2014] N. Goldman and J. Dalibard, Periodically driven quantum systems: effective hamiltonians and engineered gauge fields, Physical review X 4, 031027 (2014).
  * Eckardt and Anisimovas [2015] A. Eckardt and E. Anisimovas, High-frequency approximation for periodically driven quantum systems from a floquet-space perspective, New journal of physics 17, 093039 (2015).
  * Kuwahara  _et al._ [2016] T. Kuwahara, T. Mori, and K. Saito, Floquet–magnus theory and generic transient dynamics in periodically driven many-body quantum systems, Annals of Physics 367, 96 (2016).
  * Childs  _et al._ [2021] A. M. Childs, Y. Su, M. C. Tran, N. Wiebe, and S. Zhu, Theory of trotter error with commutator scaling, Physical Review X 11, 011020 (2021).
  * Ikeda  _et al._ [2023] T. N. Ikeda, A. Abrar, I. L. Chuang, and S. Sugiura, Minimum trotterization formulas for a time-dependent hamiltonian, Quantum 7, 1168 (2023).
  * Gong  _et al._ [2023] W. Gong, S. Zhou, and T. Li, A theory of digital quantum simulations in the low-energy subspace, arXiv preprint arXiv:2312.08867 (2023).
  * Heyl  _et al._ [2019] M. Heyl, P. Hauke, and P. Zoller, Quantum localization bounds trotter errors in digital quantum simulation, Science advances 5, eaau8342 (2019).
  * Sieberer  _et al._ [2019] L. M. Sieberer, T. Olsacher, A. Elben, M. Heyl, P. Hauke, F. Haake, and P. Zoller, Digital quantum simulation, trotter errors, and quantum chaos of the kicked top, npj Quantum Information 5, 78 (2019).
  * Vernier  _et al._ [2023] E. Vernier, B. Bertini, G. Giudici, and L. Piroli, Integrable digital quantum simulation: Generalized gibbs ensembles and trotter transitions, Physical Review Letters 130, 260401 (2023).
  * Suchsland  _et al._ [2023] P. Suchsland, R. Moessner, and P. W. Claeys, Krylov complexity and trotter transitions in unitary circuit dynamics, arXiv preprint arXiv:2308.03851 (2023).
  * Vovrosh  _et al._ [2021] J. Vovrosh, K. E. Khosla, S. Greenaway, C. Self, M. S. Kim, and J. Knolle, Simple mitigation of global depolarizing errors in quantum simulations, Physical Review E 104, 035309 (2021).
  * Yang  _et al._ [2023] Y. Yang, A. Christianen, S. Coll-Vinent, V. Smelyanskiy, M. C. Bañuls, T. E. O’Brien, D. S. Wild, and J. I. Cirac, Simulating prethermalization using near-term quantum computers, PRX Quantum 4, 030320 (2023).
  * Wiebe  _et al._ [2011] N. Wiebe, D. W. Berry, P. Høyer, and B. C. Sanders, Simulating quantum dynamics on a quantum computer, Journal of Physics A: Mathematical and Theoretical 44, 445308 (2011).
  * Yao  _et al._ [2021] Y.-X. Yao, N. Gomes, F. Zhang, C.-Z. Wang, K.-M. Ho, T. Iadecola, and P. P. Orth, Adaptive variational quantum dynamics simulations, PRX Quantum 2, 030307 (2021).
  * Xu  _et al._ [2021] X. Xu, J. Sun, S. Endo, Y. Li, S. C. Benjamin, and X. Yuan, Variational algorithms for linear algebra, Science Bulletin 66, 2181 (2021).
  * Miessen  _et al._ [2021] A. Miessen, P. J. Ollitrault, and I. Tavernelli, Quantum algorithms for quantum dynamics: A performance study on the spin-boson model, Physical Review Research 3, 043212 (2021).
  * Irmejs  _et al._ [2023] R. Irmejs, M.-C. Bañuls, and J. I. Cirac, Quantum simulation of z 2 lattice gauge theory with minimal resources, Physical Review D 108, 074503 (2023).
  * Zhang  _et al._ [2023] Z.-J. Zhang, J. Sun, X. Yuan, and M.-H. Yung, Low-depth hamiltonian simulation by an adaptive product formula, Physical Review Letters 130, 040601 (2023).
  * Zhao  _et al._ [2023a] H. Zhao, M. Bukov, M. Heyl, and R. Moessner, Making trotterization adaptive and energy-self-correcting for nisq devices and beyond, PRX Quantum 4, 030319 (2023a).
  * Linteau  _et al._ [2023] D. Linteau, S. Barison, N. Lindner, and G. Carleo, Adaptive projected variational quantum dynamics, arXiv preprint arXiv:2307.03229 (2023).
  * Zhao  _et al._ [2023b] H. Zhao, M. Bukov, M. Heyl, and R. Moessner, Adaptive trotterization for time-dependent hamiltonian quantum dynamics using instantaneous conservation laws, arXiv preprint arXiv:2307.10327 (2023b).
  * Ikeda and Fujii [2023] T. N. Ikeda and K. Fujii, Trotter24: A precision-guaranteed adaptive stepsize trotterization for hamiltonian simulations (2023), arXiv:2307.05406 [quant-ph] .
  * Zhao  _et al._ [2024] Q. Zhao, Y. Zhou, and A. M. Childs, Entanglement accelerates quantum simulation (2024), arXiv:2406.02379 [quant-ph] .
  * Gebhart  _et al._ [2023] V. Gebhart, R. Santagati, A. A. Gentile, E. M. Gauger, D. Craig, N. Ares, L. Banchi, F. Marquardt, L. Pezzè, and C. Bonato, Learning quantum systems, Nature Reviews Physics 5, 141 (2023).
  * Holzäpfel  _et al._ [2015] M. Holzäpfel, T. Baumgratz, M. Cramer, and M. B. Plenio, Scalable reconstruction of unitary processes and hamiltonians, Physical Review A 91, 042129 (2015).
  * Wang  _et al._ [2017] J. Wang, S. Paesani, R. Santagati, S. Knauer, A. A. Gentile, N. Wiebe, M. Petruzzella, J. L. O’brien, J. G. Rarity, A. Laing, _et al._ , Experimental quantum hamiltonian learning, Nature Physics 13, 551 (2017).
  * Bairey  _et al._ [2019] E. Bairey, I. Arad, and N. H. Lindner, Learning a local hamiltonian from local measurements, Physical review letters 122, 020504 (2019).
  * Hou  _et al._ [2019] P.-Y. Hou, L. He, F. Wang, X.-Z. Huang, W.-G. Zhang, X.-L. Ouyang, X. Wang, W.-Q. Lian, X.-Y. Chang, and L.-M. Duan, Experimental hamiltonian learning of an 11-qubit solid-state quantum spin register, Chinese Physics Letters 36, 100303 (2019).
  * Li  _et al._ [2020] Z. Li, L. Zou, and T. H. Hsieh, Hamiltonian tomography via quantum quench, Physical review letters 124, 160502 (2020).
  * Pastori  _et al._ [2022] L. Pastori, T. Olsacher, C. Kokail, and P. Zoller, Characterization and verification of trotterized digital quantum simulation via hamiltonian and liouvillian learning, arXiv preprint arXiv:2203.15846 (2022).
  * Nandy  _et al._ [2023] S. Nandy, M. Schmitt, M. Bukov, and Z. Lenarčič, Reconstructing effective hamiltonians from nonequilibrium (pre-) thermal steady states, arXiv preprint arXiv:2308.08608 (2023).
  * Haah  _et al._ [2024] J. Haah, R. Kothari, and E. Tang, Learning quantum hamiltonians from high-temperature gibbs states and real-time evolutions, Nature Physics , 1 (2024).
  * Bakshi  _et al._ [2024] A. Bakshi, A. Liu, A. Moitra, and E. Tang, Structure learning of hamiltonians from real-time evolution (2024), arXiv:2405.00082 [quant-ph] .
  * An  _et al._ [2024] Z. An, J. Wu, M. Yang, D. Zhou, and B. Zeng, Unified quantum state tomography and hamiltonian learning: A language-translation-like approach for quantum systems, Physical Review Applied 21, 014037 (2024).
  * Mirani and Hayden [2024] A. Mirani and P. Hayden, Learning interacting fermionic hamiltonians at the heisenberg limit, arXiv preprint arXiv:2403.00069 (2024).
  * Olsacher  _et al._ [2024] T. Olsacher, T. Kraft, C. Kokail, B. Kraus, and P. Zoller, Hamiltonian and liouvillian learning in weakly-dissipative quantum many-body systems (2024), arXiv:2405.06768 [quant-ph] .
  * Kingma and Ba [2017] D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2017).
  * Mehta  _et al._ [2019] P. Mehta, M. Bukov, C.-H. Wang, A. G. Day, C. Richardson, C. K. Fisher, and D. J. Schwab, A high-bias, low-variance introduction to machine learning for physicists, Physics reports 810, 1 (2019).
  * Note [1] More numerical results with different initial states are shown in the SM.
  * Granade  _et al._ [2012] C. E. Granade, C. Ferrie, N. Wiebe, and D. G. Cory, Robust online hamiltonian learning, New Journal of Physics 14, 103013 (2012).
  * Gentile  _et al._ [2021] A. A. Gentile, B. Flynn, S. Knauer, N. Wiebe, S. Paesani, C. E. Granade, J. G. Rarity, R. Santagati, and A. Laing, Learning models of quantum systems from experiments, Nature Physics 17, 837 (2021).
  * Flynn  _et al._ [2022] B. Flynn, A. A. Gentile, N. Wiebe, R. Santagati, and A. Laing, Quantum model learning agent: characterisation of quantum systems through machine learning, New Journal of Physics 24, 053034 (2022).
  * Else  _et al._ [2020] D. V. Else, W. W. Ho, and P. T. Dumitrescu, Long-lived interacting phases of matter protected by multiple time-translation symmetries in quasiperiodically driven systems, Physical Review X 10, 021032 (2020).
  * Mori  _et al._ [2021] T. Mori, H. Zhao, F. Mintert, J. Knolle, and R. Moessner, Rigorous bounds on the heating rate in thue-morse quasiperiodically and randomly driven quantum many-body systems, Physical Review Letters 127, 050602 (2021).
  * Chen  _et al._ [2010] X. Chen, A. Ruschhaupt, S. Schmidt, A. del Campo, D. Guéry-Odelin, and J. G. Muga, Fast optimal frictionless atom cooling in harmonic traps: Shortcut to adiabaticity, Physical review letters 104, 063002 (2010).
  * Ljubotina  _et al._ [2022] M. Ljubotina, B. Roos, D. A. Abanin, and M. Serbyn, Optimal steering of matrix product states and quantum many-body scars, PRX Quantum 3, 030343 (2022).
  * Fernández-González  _et al._ [2015] C. Fernández-González, N. Schuch, M. M. Wolf, J. I. Cirac, and D. Perez-Garcia, Frustration free gapless hamiltonians for matrix product states, Communications in Mathematical Physics 333, 299 (2015).
  * Kolodrubetz  _et al._ [2017] M. Kolodrubetz, D. Sels, P. Mehta, and A. Polkovnikov, Geometry and non-adiabatic response in quantum and classical systems, Physics Reports 697, 1 (2017).
  * Casas and Murua [2009] F. Casas and A. Murua, An efficient algorithm for computing the baker–campbell–hausdorff series and some of its applications, Journal of Mathematical Physics 50 (2009).

Supplementary Material
Learning effective Hamiltonians for adaptive time-evolution quantum algorithms

###### Contents

  1. SM 1 Technical details for QHL
    1. SM 1.1 Construction of basis
    2. SM 1.2 Details for the Gradient Descent algorithm
  2. SM 2 Reduction of the operator basis
    1. SM 2.1 Performance of truncated basis—basis size $N$
    2. SM 2.2 Performance of truncated basis—different tolerance values for ADA-Trotter
  3. SM 3 Relation to Floquet-Magnus expansion

##  SM 1 Technical details for QHL

###  SM 1.1 Construction of basis

In this section we explain how to choose an operator basis for the effective
Hamiltonian ansatz. In the main text, we consider the target Hamiltonian Eq.
1, which has both the spatial translation symmetry and mirror symmetry. Since
the operators $\mathcal{O}_{\alpha}$ in the ansatz Hamiltonian should
originate from the nested commutators of $H_{x}$ and $H_{z}$, they also obey
these symmetries. On a given site $j$, there are three possible Pauli
operators $X_{j},Y_{j},Z_{j}$ and to obey the symmetry constraint one can only
allow operators that uniformly sum over all sites, e.g.,
$\mathcal{O}_{X}=\sum_{j}X_{j}$ with the corresponding coefficient $C_{X}$.
For two-sites and multi-sites operators, two possible cases can occur: First,
each individual operator are already mirror symmetric w.r.t the center of the
support, e.g. $X_{j}X_{j+1}$ and $X_{j}Z_{j+1}X_{j+2}$, we use
$\mathcal{O}_{XX}=\sum_{j}X_{j}X_{j+1}$ and
$\mathcal{O}_{XZX}=\sum_{j}X_{j}Z_{j+1}X_{j+2}$; Second, each term is not
individually mirror symmetric, e.g. $X_{j}Y_{j+1}$, we use
$\mathcal{O}_{XY}=\sum_{j}X_{j}Y_{j+1}+Y_{j}X_{j+1}$ such that after summing
over all sites the operator satisfies the symmetry constraint.

###  SM 1.2 Details for the Gradient Descent algorithm

We define the loss function as

$l=1-|\langle\psi_{\text{ADA}}(t)\left|\psi_{\text{L}}(t)\right\rangle|^{2}$
(S.1)

the infidelity between the two states, such that a small loss function
indicates a high-quality of QSR. The parameters to be optimized are the
coefficients $C_{\alpha}$ in
$H_{\text{eff}}^{[t]}{=}\sum_{\alpha}C^{[t]}_{\alpha}\mathcal{O}_{\alpha},$
and they enter the loss function via
$\left|\psi_{\text{L}}(t)\right\rangle{=}\exp(-itH_{\text{eff}}^{[t]})\left|{\psi}(0)\right\rangle$.
Building with the Optax library for Google JAX, we implement an adaptive
moment estimation (Adam)[75] in the optimization algorithm. The key difference
between Adam and standard stochastic Gradient Descen (GD) is that Adam
computes individual learning rates (i.e. the step size $\alpha$ in each epoch
of the GD) adaptively for each parameter based on the estimated first and
second moments of the gradients, while in standard stochastic GD a single
learning rating is maintained for all parameter updates. We find this feature
of Adam particularly suitable for our optimization task because the loss
function landscape in the coefficients space can be very complicated, and the
appropriate learning rate in each epoch may vary notably throughout the
optimization process. In our numerical simulation, the initial learning rate
is set to be $\alpha=10^{-5}$. The concept of momentum in parameter
optimization is incorporated by two hyperparameters $\beta_{1}$ and
$\beta_{2}$ which are exponential decay rates for the first and second moment
estimates while taking the exponential moving average. Specifically,
$\beta_{1}$ controls how much previous gradients take part in the new update,
and $\beta_{2}$ is used in scaling the learning rate for each parameter;
parameters with a high second moment (i.e. parameters whose gradients are
large and varying) receive smaller updates and vice versa. Because such
average is involved, another parameter $\epsilon$ is put in place to prevent
any division-by-zero error. We use
$\beta_{1}=0.9,\beta_{2}=0.99,\epsilon=10^{-8}$, default values as in JAX, but
these hyperparameters typically require very little tuning. In order to ensure
efficient training, we also implement a cut-off threshold
($l_{\mathrm{min}}=10^{-4}$) such that the optimization process is terminated
once the the loss function drops below this threshold. The maximal number of
training epochs we set is 5000. Indeed, at early times, for instance, within
five Trotter steps, $l_{\mathrm{min}}$ can be achieved quickly so we can
terminate the optimization with a few number of epochs. For longer times, as
shown in Fig. 1, we cannot achieve this low loss function cut-off and 5000
epochs are fully exploited.

At a given time, the optimization process starts from one initial guess of the
coefficients. For the first Trotter step, we use
$(C_{X},C_{Z},C_{ZZ}){=}(h_{x},h_{z},J_{zz})$ – the same as in the target
Hamiltonian – and all other coefficients start from zero. Later, we pass the
optimized parameters as the initial guess for the next training, which notably
improves the optimization efficiency.

A possible numerical issue in GD is the accidental degeneracy encountered when
evaluating the gradients of loss. In the computation of gradients, one
intermediate quantity is
$\partial\left|\psi_{\text{L}}\right\rangle/\,\partial C_{\alpha}$, which can
be viewed as the changing rate of the outcome state
$\left|\psi_{\text{L}}\right\rangle$ under a perturbation of the Hamiltonian
$H_{\text{eff}}$. When the Hamiltonian has two identical eigenvalues up to
machine precision, this becomes degenerate perturbation and a naive gradient
will fail. Whenever this problem happens, a small change is added to the
coefficients $C_{\alpha}$ to lift the system away from numerical degeneracy.

##  SM 2 Reduction of the operator basis

GD in a large parameter space can be time-consuming and it is possible to
truncate the operator basis to speed up the optimization procedure. Of course,
one can use the complete operator basis corresponding to a small support $R$.
Here we illustrate another method to truncate the basis while still keeping
some operators with a large support. The central idea is to perform QHL for
many realizations of ADA-Trotter, and then we try to sort all operator basis
according to their averaged coefficients deviations.

Figure S1: Tolerance-time-averaged deviations in the effective Hamiltonian for
each individual term. For clarity we only plot the largest 15 terms. Few terms
are notably larger than others, e.g. $X$ or $XZX$ and most of the averaged
values are very small and comparable to each other. Error bars correspond to
their standard deviations among the data obtained for different tolerance
values. The following Hamiltonian parameters are used for ADA-Trotter
$J_{z}=-1,h_{x}=-1.7,h_{z}=0.5,M=100$, and multiple sets of tolerances are
used $d_{\mathcal{E}}=\\{0.02,0.01,0.005,0.002,0.001,0.0005,0.0001\\}$ and
$d_{\delta\mathcal{E}^{2}}=\\{0.02,0.03\\}$; $d_{\mathcal{E}}=\\{0.01,0.02\\}$
and $d_{\delta\mathcal{E}^{2}}=\\{0.05,0.04,0.03,0.02,0.01,0.005\\}$.

To do so, we still use the same initial state as in the main text
$\exp(-i\theta_{y}\sum_{j}Y_{j})|\downarrow{\dots}\downarrow\rangle$ with
$\theta_{y}=\pi/3$ and implement ADA-Trotter as well as QHL for $L=10$. Then
we compute the absolute value of $C^{[t]}_{\alpha}$ subtracted by its target
value and average it over time, as an indicator for the time-averaged errors
in the effective Hamiltonian. We further average this time-averaged error over
different realizations of ADA-Trotter given various tolerances, and sort
tolerance-time-averaged errors as shown in Fig. S1. The error bar denotes the
standard deviation among the data for different tolerances. Note, for clarity
we only show the largest 15 terms out of the full operator basis with $207$
terms for $R=5$. Apart from very few terms that are notably larger than
others, e.g. $X$ or $XZX$, it is generally very difficult to distinguish error
terms that are dominating. Most of the averaged values are very small and
comparable to each other. We truncate the operator basis by only including the
largest $N$ terms and in the following we discuss the performance of GD with
the truncated basis.

###  SM 2.1 Performance of truncated basis—basis size $N$

Figure S2: (a) Grey: Loss function after optimization of GD at each time. Its
growth can be well fitted with a straight line (black) of slope $\gamma$.
Note, the fit is only performed after the first five Trotter steps to avoid
the impact of the early-time state reconstruction that is almost perfect. (b)
Performance of the GD with a truncated operator basis. A larger $N$ enlarges
the optimization parameter space, hence generally improves QHL and reducing
the slope. We use $L=16,J_{z}=-1,h_{x}=-1.7,h_{z}=0.5,$
$(d_{\mathcal{E}},d_{\delta\mathcal{E}^{2}})$ are noted in the legend.

As already discussed in the main text, for $N=16$ GD can already achieve a
sufficiently good quantum state reconstruction. Here we supply further
analysis based on the growing rate $\gamma$ of the loss function after
optimization at each time. As shown in Fig. S2 (a), at very early times, since
the time-evolved state $\left|\psi_{\text{ADA}}(t)\right\rangle$ remains close
to the initial state, GD can always reconstruct the state up to the highest
accuracy (loss value $\sim 10^{-4}$) that is preset in GD algorithm. Then the
state infidelity (grey dots) approximately grows linearly in time, and its
slope $\gamma$ can be numerically fitted. Fig. S2 (b) depicts the dependence
of $\gamma$ versus different $N$ in a log scale. For a small system size
$L=10$ and a given tolerance value (grey and orange data), linearly increasing
the size of the operator basis seems to exponentially improve the learning
quality. A larger system size $L=16$ (blue) notably increases the growth rate
in contrast to $L=10$ (grey). This occurs simply because the total Hilbert
space enlarges exponentially in system size $L$, and hence it becomes more
difficult to reconstruct a quantum state by GD. We also use $\gamma$ to
quantify the learning quality for results shown in Fig. 3 in the main text.

###  SM 2.2 Performance of truncated basis—different tolerance values for ADA-
Trotter

Now we confirm that for all tolerances considered in this work, learning basis
involving $N=106$ operators is sufficient to achieve high-quality QHL. As
shown in Fig. S3 panels (b) and (c), for different energy and energy variance
tolerances the fitted slope $\gamma$ is always sufficiently small ($\lesssim
0.006$). Also, in general the learning quality improves for smaller
tolerances. It naturally happens since the ADA-Trotter algorithm tends to use
smaller step sizes throughout the entire time evolution, hence the Trotter
errors can remain sufficiently local which are earlier to learn.

Figure S3: Growth rate $\gamma$ of the optimized state infidelity after GD,
for different tolerance values. $\gamma$ always remains small and becomes even
smaller for extremely tight tolerances. Although not shown in the plot, we
have also verified that the standard deviation of the fitted slope is
generally smaller than or comparable to the size of the data points. We use
$L=16,J_{z}=-1,h_{x}=-1.7,h_{z}=0.5,$ $N=106$, $M=100$ for numerical
simulation and fix $d_{\delta\mathcal{E}^{2}}=0.01$ in panel (a) and
$d_{\mathcal{E}}=0.02$ in (b).

##  SM 3 Relation to Floquet-Magnus expansion

The Floquet-Magnus expansion has been often used to derive effective
Hamiltonians for Trotterized systems or periodically driven systems. However,
since ADA-Trotter adapts the Trotter step, temporal periodicity is violated
and the Floquet-Magnus expansion does not apply. However, we note that the
effective Hamiltonian obtained by QHL is not completely chaotic. Rather, it
exhibits certain interesting connections with the Floquet-Magnus expansion.
Apart from the statistical behavior of Trotter errors as shown by the violin
plot in Fig. 3, here we directly plot the trajectory of the learned
coefficients and compare it with the corresponding Floquet results.

To do so, we first derive the effective Hamiltonian $H_{\mathrm{eff}}(\tau)$
corresponding to the second order Trotter decomposition $U(\delta
t_{i})=e^{-i\tau H_{x}/2}e^{-i\tau H_{z}}e^{-i\tau H_{x}/2}$ with a fixed step
size $\tau$. By employing the expansion for the symmetric BCH expansion
$e^{C}=e^{A/2}e^{B}e^{A/2}$, where
$C=A+B-\frac{1}{24}[[B,A],A]-\frac{1}{12}[[B,A],B]$ [87], we obtain

$\displaystyle H_{\text{eff}}$
$\displaystyle=H_{x}+H_{z}+\frac{\tau^{2}}{24}[[H_{z},H_{x}],H_{x}]+\frac{\tau^{2}}{12}[[H_{z},H_{x}],H_{z}]+\mathcal{O}(\tau^{4})$
(S.2)
$\displaystyle=C_{X}\sum_{j}X_{j}+C_{Z}\sum_{j}Z_{j}+C_{ZZ}\sum_{j}Z_{j}Z_{j+1}$
$\displaystyle+C_{YY}\sum_{j}Y_{j}Y_{j+1}+C_{ZX}\sum_{j}(X_{j}Z_{j+1}+Z_{j}X_{j+1})+C_{ZXZ}\sum_{j}Z_{j-1}X_{j}Z_{j+1}+\mathcal{O}(\tau^{4}),$

with the coefficients

$\displaystyle
C_{X}=h_{x}-\frac{2\tau^{2}}{3}J_{z}^{2}h_{x}-\frac{\tau^{2}}{3}h_{z}^{2}h_{x},\
\ C_{Z}=h_{z}+\frac{\tau^{2}}{6}h_{x}^{2}h_{z},\ \
C_{ZZ}=J_{z}+\frac{\tau^{2}}{3}J_{z}h_{x}^{2},$ (S.3) $\displaystyle
C_{YY}=\frac{\tau^{2}}{3}J_{z}h_{x}^{2},\ \
C_{ZX}=-\frac{2\tau^{2}}{3}J_{z}h_{x}h_{z},\ \
C_{ZXZ}=-\frac{2\tau^{2}}{3}J_{z}^{2}h_{x}.$

Higher order terms are negligible for the small step size considered here.

Figure S4: Comparison between trajectories of learned coefficients for ADA-
Trotter (blue), fix-step Trotter (red) and the target (black). The cluster of
$C_{\alpha}$ seems to be close to the fix-step Trotter prediction. We use
$L=16,J_{z}=-1,h_{x}=-1.7,h_{z}=0.5,$ $N=106$,$M=100$ for numerical
simulation. Tolerances used for ADA-Trotter are shown in the plot.
$\tau=0.198$ for (a) and 0.288 for (b).

Similar to Fig. 1 in the main text, here in Fig. S4 we also show the time
trace of the learned coefficients for ADA-Trotter (blue). We calculate the
average step size and the corresponding coefficients (red) for fix-step
Trotter via Eq. S.3 for two different sets of tolerances in Fig. S4 panels (a)
and (b). Rather than being completely chaotic, these blue trajectories form a
cluster in this three dimensional space and are close to the red star. It may
imply that different error terms in the effective Hamiltonian
$H_{\text{eff}}^{[t]}$ are correlated with each other, in a similar fashion as
in fix-step Trotter algorithm. We leave detailed analysis for the shape of the
cluster and their relation with fix-step Trotter error to future works.


11institutetext: Kansas State University, 2164 Engineering Hall, Manhattan, KS
43017-6221, USA
11email: {okerinde, schoggatt, divyalakki, ncbrubaker,
bhsu, lshamir<EMAIL_ADDRESS>
# Self-Supervised Approach to Addressing Zero-Shot Learning Problem

Ademola Okerinde  Sam Hoggatt  Divya Vani Lakkireddy  Nolan Brubaker  William
Hsu  Lior Shamir  Brian Spiesman

###### Abstract

In recent years, self-supervised learning has had significant success in
applications involving computer vision and natural language processing. The
type of pretext task is important to this boost in performance. One common
pretext task is the measure of similarity and dissimilarity between pairs of
images. In this scenario, the two images that make up the negative pair are
visibly different to humans. However, in entomology, species are nearly
indistinguishable and thus hard to differentiate. In this study, we explored
the performance of a Siamese neural network using contrastive loss by learning
to push apart embeddings of bumblebee species pair that are dissimilar, and
pull together similar embeddings. Our experimental results show a 61% F1-score
on zero-shot instances, a performance showing 11% improvement on samples of
classes that share intersections with the training set.

###### Keywords:

Self-supervised learning, Zero-shot learning, Siamese neural network, Pretext
task, Contrastive loss

##  1 Introduction

Zero-shot learning (ZSL) is a problem in machine learning, where during test
time, a learner observes samples from classes not observed during training
when the learner needs to predict sample class [1].

Deep learning (DL) requires large amounts of carefully labeled data, which is
very often difficult to acquire and expensive to annotate. Even with large
amounts of data, supervised learning still has blind spots in learning useful
and rich representations. Moreover, the supervision signal can bias the
network in unexpected ways. Self-supervised learning (SSL) is an effective way
to extract training signals from massive amounts of unlabeled data and to
learn good representation to facilitate downstream tasks where it is expensive
to collect task-specific labels. This representation is a feature vector that
contains semantic information about the input image. SSL is a special type of
representation learning that enables learning good data representation from
unlabeled dataset. SSL is necessary because data labeling is often expensive
and thus high-quality labeled datasets are limited. Also, a good learned
representation enables easier transfer of useful information/signals to a
variety of downstream tasks. In our research, the downstream tasks consists of
zero-shot and few-shot transfer to new tasks.

The use of contrastive loss as the cost function is critical to the success of
SSL. This objective function pulls together embeddings of augmentations of the
same bumblebee species pair and pushes embeddings of views of different
species pairs far apart. This is arguably a challenging task, even to humans,
because many species of bumblebees are quite similar, even to experts.

Pretext tasks shown in Figure 2 are an important component of any SSL training
pipeline. Our goal was to have a model learn on downstream tasks like ZSL
although models trained on pretext tasks were transferred for specific tasks.
On some occasions, pre-training data could be the same as the task-specific
data. In our research, we explored two options/approaches. Our pre-training
data could be either bumblebees images or a CIFAR-10 dataset. In recent years,
researchers have become more creative at determining different forms of
pretext tasks. In [2], the task was to have the model predict the degree of
rotation in an image. After a patch, [3] predicted which of eight neighboring
locations had another patch. This forces the model to learn the innate
relationships among patches of the input image. In our research, we formulated
our pretext task as a measure of similarity in the same species and
dissimilarity between different species of bumblebees.

Existing image-text retrieval models are often fine-tuned on pretext tasks in
which the data distribution shift is minimal, but the domain of entomology has
terms that are rare in most pre-trained tasks.

The Siamese neural network (SNN) is an architecture that comprises two
identical neural networks with shared weights; their parameters are trained to
determine the similarity between two data samples using a distance metric on
the embeddings. Our research used SNN to determine if two bumblebee images
belong to the same specie.

##  2 Related Work

Recently, SSL has gained a lot of momentum as a new paradigm of representation
learning because supervised learning often requires a large number of labels,
which is time consuming or not available in certain domains. As a result,
researchers have become more creative: using part of the data itself as a form
of supervision, a seemingly frivolous exercise that has produced remarkable
outcomes.

Self-supervision is often achieved either through self-prediction or
contrastive learning (CL). In [4], video colorization as a self-supervised
learning method provides a rich representation that can be used for video
segmentation and unlabeled visual region tracking without extra fine-tuning.
Without using supervised labels for training, the zero-shot CLIP [5]
classifier performs quite well on challenging image-to-text classification
tasks. [6] learned a joint visual-language representation using CL and
training on one billion noisy image alt-text pairs. To the best of our
knowledge, such an enormous dataset is not yet available in entomology, which
makes the learning task harder.

CL has helped make many SSL applications successful. Ideally, CL keeps
positive pairs close to each other and negative pairs far apart in the feature
space. According to [7, 8], CL currently performs at state-of-the-art level in
SSL. [9] adopted a memory bank to store the instance class representation
vector. Momentum Contrast (MoCo) [10] trained a visual representation encoder
by matching an encoded query q to a dictionary of encoded keys with
contrastive loss. SimCLR [7] used the normalized temperature-scaled cross-
entropy loss as contrastive loss. BYOL [8] relied only on positive pairs, not
negative pairs, to learn feature representation. SimSiam [7, 10] removed the
momentum encoder and provided a simpler design based on BYOL.

In the first paper in which SNN was published, [11], the author depicted the
significance of the network for the signature verification task. The goal was
to identify signature authenticity.

ZSL is a model’s ability to detect classes never seen during training. Even
for humans, this is difficult. These classes are sometimes referred to as Out
Of Distribution (OOD) samples. Earlier work in ZSL use attributes in a two-
step approach to infer unknown classes. In vision research, more recent,
advanced models learn mappings from image feature space to semantic space.
[12] proposed a TF-VAEGAN to address generalized zero-shot recognition. In
that research, both visual features and corresponding semantics are fed into
the architecture. In contrast, we based our approach on only visual signals.

##  3 Proposed Approach

To determine how well our model used a pretext task in a downstream task of
novelty detection, we trained our model not on an output class of bumblebees
but rather the similarities and dissimilarities between differrent species
using contrastive loss as shown in Figure 5. This task forces the model to be
aware of visual signals needed in the downstream task. Figure 1 shows our
convolutional SNN, which uses feature extractors pretrained on ImageNet.

The SNN has three convolutional layers with 16, 32, and 64 filters. We use 3*3
kernel size with the same padding. The final fully connected layer has 128
neurons on which the energy function is computed to estimate the distance
vector. Euclidean distance is used as the energy function. Rectified linear
activation function (ReLU) is applied after each conv layer followed by
maxpooling2D. To avoid overfitting, we applied dropout with probability of 20%
after the feature extraction stage. Before passing the image pairs to the
feature extractors, we augmented the images by randomly flipping and rotating.
We use contrastive loss as the objective function to be optimized. The loss
equation is defined in Equation 1

$\begin{split}L(Y,B1,B2,W)=(1-Y)\cdot\frac{1}{2}D_{w}^{2}+Y\frac{1}{2}(max(0,m-D_{w}))^{2}\\\
\\\ D^{2}_{w}=(f(B1)-f(B2))^{2}\end{split}$ (1)

$B1,B2$ are bumblebee specie image pair. $D_{w}^{2}$ is the euclidean distance
which represent the energy function. $Y$ can either be 1 or 0 depending or
whether the pair is similar or not. And, $m$ represent margin, a threshold for
which the embedding is satisfactory separable. $W$ are the network parameters
to be learnt during training.

We experimented with MobileNetV2 and ResNet-101 as feature extractors before
measuring the energy function between each embedding of the input image pair.

##  4 Methodology and Experimental Design

###  4.1 Dataset

The dataset consisted of 45 species of bumblebee totalling 104770 samples as
shown in Table 8. Figure 3 shows the percentage of the different species in an
histogram. Bombus impatiens dominates with 9.6% of the entire dataset.

Table 1: Bumblebee (B.) Species. (Zero and few shots are in bold) Species | #samples | Species | #samples
---|---|---|---
B. appositus | 888 | B. affinis | 1855
B. crotchii | 457 | B. bifarius | 5237
B. bohemicus | 21 | B. bimaculatus | 7595
B. caliginosus | 147 | B. borealis | 2256
B. franklini | 9 | B. citrinus | 1449
B. cockerelli | 3 | B. centralis | 1755
B. cryptarum | 419 | B. fernaldae flavidus | 1047
B. suckleyi | 14 | B. flavifrons | 3866
B. fraternus | 424 | B. fervidus californicus | 5020
B. frigidus | 179 | B. griseocollis | 8787
B. variabilis | 12 | B. impatiens | 10008
B. jonellus | 5 | B. insularis | 1241
B. kirbiellus | 69 | B. melanopygus | 5892
B. morrisoni | 276 | B. mixtus | 3359
B. natvigi | 3 | B. nevadensis | 1982
B. occidentalis | 943 | B. pensylvanicus sonorus | 6522
B. polaris | 11 | B. perplexus | 2503
B. sandersoni | 163 | B. rufocinctus | 4719
B. sitkensis | 647 | B. huntii | 3335
B. sylvicola | 366 | B. ternarius | 6467
B. vandykei | 625 | B. terricola | 2145
|  | B. vagans | 3546
|  | B. vosnesenskii | 6284
|  | B. auricomus | 2219
|  | Total = 104770 |
Table 2: ResNet-101 confusion matrix at 0.5 threshold on testset-ALL (45 species) | Predicted similar | Predicted dissimilar
---|---|---
Actually similar | 61 | 29
Actually dissimilar | 43 | 47
Table 3: ResNet-101 confusion matrix at 0.5 threshold on testset (24 species) | Predicted similar | Predicted dissimilar
---|---|---
Actually similar | 3725 | 1267
Actually dissimilar | 1514 | 3478
Table 4: ResNet-101 Confusion matrix at 0.5 threshold on 21 unknown test set species - zero-shot learning | Predicted similar | Predicted dissimilar
---|---|---
Actually similar | 24 | 18
Actually dissimilar | 15 | 27
Table 5: F1-scores of zero-shot specie pair | B. natvigi | B. variabilis | B. appositus | B. fraternus | B. frigidus
---|---|---|---|---|---
B. cockerelli | 0.5 | 0.88 | 0.28 | 1.00 | 1.00
B. polaris | 0.5 | 0.43 | 0.50 | 0.72 | 0.46
B. occidentalis | 0.11 | 0.48 | 0.52 | 0.60 | 0.63
B. cryptarum | 0.62 | 0.49 | 0.51 | 0.74 | 0.58
B. caliginosus | 0.50 | 0.39 | 0.52 | 0.60 | 0.59
Table 6: ResNet-101 confusion matrix at 0.5 threshold on validation | Predicted similar | Predicted dissimilar
---|---|---
Actually similar | 3197 | 1123
Actually dissimilar | 1243 | 3077
Table 7: MobileNetV2 confusion matrix at 0.5 threshold on validation | Predicted similar | Predicted dissimilar
---|---|---
Actually similar | 2897 | 1423
Actually dissimilar | 1608 | 2712
Table 8: Number of samples in the dataset. Dataset | images | species-represented
---|---|---
Train + Validation | 79280 | 24
Test | 25490 | 45

###  4.2 Data Preprocessing

We preprocessed the dataset by moving all the species with fewer than a
thousand samples to the test set, which means that 21 species are not
represented in the training set. We then randomly selected 20% of samples of
each remaining species to include in the test data. All 45 species are
represented in the test data.

Figure 1: Siamese Neural Network Figure 2: Pretext Tasks Figure 3: Pie chart
of the various species Figure 4: Example images of various bee species

Figure 5: Positive pair and Negative pair

###  4.3 Experimental results

We trained a convolutional SNN for a maximum of 100 epochs, with patience
level of 7, on 80% of the training set. We then evaluated the trained model on
the 20% validation set. The results are shown in Table 9.

##  5 Results and Discussion

Table 9 shows the average accuracy, recall, precision, and F1 score at
different thresholds/cut-offs. Of all the thresholds, the results reveal that
0.5 is a good decision boundary for most species. Similarity scores below the
threshold are classified as similar and vice versa. We also inspected the
performance of contrastive learning within each species of the bumblebees to
discover hard and easy data pairs. On the test data, the best performing SNN
model, which uses ResNet-101 as base feature extractor, achieves 73%
precision, 72% recall, 73% F1-score, and 72% accuracy.

The results shown in Table 10 show that our SSL network can detect zero shot
instances of the bumblebees although the SNN performs much better on species
that share intersection with the training dataset. Table 5 shows that our
network can still differentiate between pairs of zero-shot instances. The
network could differentiate the B. cockerelli and B. fraternus pair as well as
the B. cryptarum and B. fraternus pair with a high confidence. This is useful
in deciding in what species the novel instance falls. With this, an
entomologist can quickly narrow down a novel species class to either in-
distribution or OOD samples.

###  5.1 Conventional Zero-shot learning

This involves learning a classifier that can detect only unknown test samples.
Our network achieved 61% F1-score on 5681 unknown species of bumblebees. As
explained in [13], this task is made more difficult because of a high class
imbalance with some species having very few samples, as can be seen in Table
1.

###  5.2 Generalized Zero-shot learning

The goal of generalized zero-shot learning is to train a classifier that can
detect test instances from seen or unseen classes. This is more useful, and
thus more challenging. With the entire 45 species in the test set, our network
achieved a 60% F1-score on various pairing of the species as shown in Table
10.

##  6 Conclusion

In this work, we explored using a self-supervised learning approach to detect
novel bumblebee instances in a zero-shot setting. We showed that our model
performs well in learning the embeddings of similar and dissimilar bumblebee
species pairs. Our experimental results showed that our model can generalize
to zero shot instances that were not part of the training. Future work will
focus on improving the performance on the ZSL task by learning a multi-modal
network that maps images to text-attribute descriptions. We can also explore
cosine similarity as the energy and triplet loss as the loss function.

Table 9: Classification Accuracy, F1, Precision, Recall when using different methods on validation set. Models | Threshold | Precision | Recall | F1 | Accuracy
---|---|---|---|---|---
SNN + ResNet-101[14] | 0.5 | 0.73 | 0.72 | 0.73 | 0.72
SNN + MobileNetV2[15] | 0.1 | 0.66 | 0.51 | 0.35 | 0.50
0.2 | 0.62 | 0.55 | 0.47 | 0.55
0.3 | 0.63 | 0.60 | 0.58 | 0.60
0.4 | 0.63 | 0.63 | 0.64 | 0.63
0.5 | 0.65 | 0.65 | 0.65 | 0.65
0.6 | 0.65 | 0.65 | 0.64 | 0.65
0.7 | 0.67 | 0.66 | 0.65 | 0.66
0.8 | 0.69 | 0.65 | 0.63 | 0.65
0.9 | 0.69 | 0.62 | 0.58 | 0.62
Table 10: Classification Accuracy, F1, Precision, Recall on test set using ResNet-101 as feature extractor. | Threshold | Precision | Recall | F1 | Accuracy
---|---|---|---|---|---
21 species (Zero-Shot) | 0.5 | 0.61 | 0.61 | 0.61 | 0.61
45 species (ALL) | 0.5 | 0.61 | 0.60 | 0.60 | 0.60
24 species | 0.5 | 0.73 | 0.72 | 0.72 | 0.72

## References

  * [1] Wikipedia contributors: Zero-shot learning — Wikipedia, the free encyclopedia (2021), https://en.wikipedia.org/w/index.php?title=Zero-shot_learning&oldid=1018108538, [Online; accessed 26-April-2021]
  * [2] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv:1803.07728 (2018)
  * [3] Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE international conference on computer vision. pp. 1422–1430 (2015)
  * [4] Vondrick, C., Shrivastava, A., Fathi, A., Guadarrama, S., Murphy, K.: Tracking emerges by colorizing videos. In: Proceedings of the European conference on computer vision (ECCV). pp. 391–408 (2018)
  * [5] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020 (2021)
  * [6] Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q.V., Sung, Y., Li, Z., Duerig, T.: Scaling up visual and vision-language representation learning with noisy text supervision. arXiv preprint arXiv:2102.05918 (2021)
  * [7] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International conference on machine learning. pp. 1597–1607. PMLR (2020)
  * [8] Grill, J.B., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.A., Guo, Z.D., Azar, M.G., et al.: Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 (2020)
  * [9] Wu, Z., Xiong, Y., Yu, S.X., Lin, D.: Unsupervised feature learning via non-parametric instance discrimination. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 3733–3742 (2018)
  * [10] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9729–9738 (2020)
  * [11] Bromley, J., Bentz, J.W., Bottou, L., Guyon, I., LeCun, Y., Moore, C., Säckinger, E., Shah, R.: Signature verification using a “siamese” time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence 7(04), 669–688 (1993)
  * [12] Narayan, S., Gupta, A., Khan, F.S., Snoek, C.G., Shao, L.: Latent embedding feedback and discriminative features for zero-shot classification. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXII 16. pp. 479–495. Springer (2020)
  * [13] Okerinde, A., Hsu, W., Theis, T., Nafi, N., Shamir, L.: egan: Unsupervised approach to class imbalance using transfer learning. In: International Conference on Computer Analysis of Images and Patterns. pp. 322–331. Springer (2021)
  * [14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)
  * [15] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4510–4520 (2018)