# Flat bands in bilayer graphene induced by proximity with polar $h$-BN
superlattices

Marta Brzezińska  Oleg V. Yazyev  Institute of Physics, Ecole Polytechnique
Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland

###### Abstract

Motivated by the observation of polarization superlattices in twisted
multilayers of hexagonal boron nitride ($h$-BN), we address the possibility of
using these heterostructures for tailoring the properties of multilayer
graphene by means of the electrostatic proximity effect. By using the
combination of first-principles and large-scale tight-binding model
calculations coupled via the Wannier function approach, we demonstrate the
possibility of creating a sequence of well-separated flat-band manifolds in
AB-stacked bilayer graphene at experimentally relevant superlattice
periodicities above $\sim$30 nm. Our calculations show that the details of
band structures depend on the local inversion symmetry breaking and the
vertical electrical polarization, which are directly related to the atomic
arrangement. The results advance the atomistic characterization of graphene-
based systems in a superlattice potential beyond the continuum model.

Introduction —  Van der Waals (vdW) heterostructures based on two-dimensional
(2D) materials manifest dramatically different electronic, optical, and
structural properties as opposed to their parent layered materials Novoselov
_et al._ (2016); Liu  _et al._ (2016); Duong  _et al._ (2017). More recently,
moiré superlattices, with twisted bilayer graphene (TBG) being the first and
the most prominent example, were shown to exhibit a wide range of exotic
phenomena including unconventional superconductivity Cao  _et al._ (2018a);
Isobe  _et al._ (2018); Wu  _et al._ (2018); Po  _et al._ (2018); Yankowitz
_et al._ (2019); Lian  _et al._ (2019); Saito  _et al._ (2020); Oh  _et al._
(2021); Löthman  _et al._ (2022) and the Chern insulator phase Cao  _et al._
(2018b); Nuckolls  _et al._ (2020); Serlin  _et al._ (2020); Ledwith  _et al._
(2020); Repellin and Senthil (2020); Stepanov  _et al._ (2021); Wu  _et al._
(2021); Pierce  _et al._ (2021); Saito  _et al._ (2021). Hexagonal boron
nitride ($h$-BN) is usually employed as an encapsulating component in devices
based on graphene heterostructures thanks to the closely matching crystal
structure and large band gap of this material Dean  _et al._ (2010); Xue  _et
al._ (2011); Dean  _et al._ (2012); Wang  _et al._ (2017). Bulk $h$-BN is a
layered material characterized by the AA’ stacking configuration of individual
honeycomb lattice layers: boron (nitrogen) atoms in one layer oppose nitrogen
(boron) atoms in the two adjacent layers, resulting in a crystal with
inversion symmetry Constantinescu  _et al._ (2013). However, polar $h$-BN
heterostructures realizing stacking sequences with no inversion symmetry can
be engineered. Very recently, it was shown that interfacial ferroelectricity
in such artificial Bernal-stacked (AB configuration) $h$-BN bilayers can be
controlled by external electric field and the lateral shift of one of the
layers Woods  _et al._ (2021); Yasuda  _et al._ (2021); Stern  _et al._
(2021). This unconventional sliding mechanism for the manipulation of the out-
of-plane electrical polarization has been identified in heterostructures based
on other vdW materials such as transition metal dichalcogenides Fei  _et al._
(2018); Wan  _et al._ (2022); Wang  _et al._ (2022) and persists at room
temperature He  _et al._ , holding promise for atomically thin transistors Wu
and Li (2021); Deb  _et al._ (2022); Koprivica and Sela (2022); Wang  _et al._
(2023).

These polar heterostructures typically exhibit a superlattice of triangular
domains with alternating vertical polarization of magnitude $P_{z}$ due to a
small residual twist accompanied by lattice relaxation. Such a staggered
polarization results a superlattice potential that can significantly affect
the electronic structure of a system. For instance, it has been demonstrated
that $h$-BN bilayers can be used to induce ferroelectricity in BLG, which has
no electric order in its initial state Zheng  _et al._ (2020). While several
general frameworks have recently been proposed to recognize the origin and
quantify 2D ferroelectricity (using group theory Ji  _et al._ (2023) or local
registry index Cao  _et al._ (2022)), a comprehensive understanding of the
connection between atomic environment, polarization, and electronic properties
in these complex heterostructures is still missing.

In this work, we show that polarization superlattices based on $h$-BN can be
used to induce flat bands in AB-stacked bilayer graphene (BLG) with no twist
whatsoever via the electrostatic proximity effect. More specifically, we
perform a systematic study of all possible non-centrosymmetric BLG/$h$-BN
heterostructures, in which the graphene component is either deposited on polar
$h$-BN bilayers (denoted 2$h$-BN/BLG) or sandwiched between them
(2$h$-BN/BLG/2$h$-BN) [cf. Fig. 1(a)]. By combining first-principles
calculations with the Wannier functions formalism, we address the gap opening
mechanisms in the BLG spectrum. To substantiate our results, we construct a
tight-binding (TB) model incorporating alternating electrostatic potential
distribution constructed from the on-site terms obtained via the
wannierization procedure. The TB calculations performed on large
heterostructures show that for experimentally relevant superlattice
periodicities $L>30$ nm, multiple of flat-band manifolds separate from the
remote bands. Our results go beyond the band structure engineering using a
superlattice potential in continuum models Park  _et al._ (2008); Barbier  _et
al._ (2010); Dubey  _et al._ (2013); Forsythe  _et al._ (2018); kun Shi  _et
al._ (2019); Huber  _et al._ (2020); Li  _et al._ (2021); Yang  _et al._
(2022); Garcia-Ruiz  _et al._ (2021, 2023); Park  _et al._ , including BLG
Killi  _et al._ (2011); Wu  _et al._ (2012); Ghorashi  _et al._ (2023); Zhu
_et al._ (2022), and complement recent findings on the band structure
modifications by $h$-BN environment Long  _et al._ (2023) or by encapsulation
with other polar materials Fumega and Lado (2023).

Electronic properties of 2$h$-BN/BLG and 2$h$-BN/BLG/2$h$-BN heterostructures
—  We start with the analysis of the electronic properties of few-layered
heterostructures, shown in Fig. 1(a), within the density functional theory
(DFT) framework. We neglect the moiré potential arising from a small lattice
mismatch between $h$-BN and graphene layers by setting in-plane lattice
constant $a=2.46$ Å of both components and the interlayer distances to
$d=3.35$ Å. By fixing the bottom $h$-BN bilayer to be in either AB or AC
stacking order, there are 8 non-equivalent configurations of 2$h$-BN/BLG. For
2$h$-BN/BLG/2$h$-BN with 6 layers in total, the number of possible
configurations is larger. Assuming the same order of B and N atoms in each
$h$-BN layer, it is possible to construct 32 configurations. Only 16 of them
are truly independent; for example, the ABCABC configuration has a symmetric
partner, ACBACB, related by a combination of the mirror symmetry along the $z$
axis, $\mathcal{M}_{z}$, and an in-plane shift by a bond length $\tau_{||}$.
Conversely, if the positions of B and N atoms are exchanged in one of the
bilayers (which we denote by prime), we can identify 12 stackings connected to
their symmmetric partners by the combination of
$\mathcal{M}_{z}\mathcal{M}_{x}\tau_{||}$ (ABCAC’B’ $\longleftrightarrow$
ACBCA’B’). The remaining 8 combinations such as ACACA’C’ restore the inversion
symmetry, hence their vertical polarization $P_{z}$ vanishes. The emerging
mirror symmetries in encapsulated heterostructures constrain the value of
$P_{z}$ of symmetry-related stackings to be the same up to the sign Wu and Li
(2021). In 2$h$-BN/BLG heterostructures, there are no spatial symmetries that
restrict $P_{z}$.

Firstly, let us recall the electronic properties of pristine AB-stacked $h$-BN
and graphene bilayers. According to our DFT calculations, free-standing AB-
stacked 2$h$-BN has an indirect band gap $\Delta E=4.78$ eV and a polarization
per area $P_{z}/A=-1.026$ pC/m, with $A=5.2\cdot 10^{-4}$ Å2 being the area of
the primitive unit cell 111We point out that a discrepancy between our values
for $P_{z}/A$ and Refs. Yasuda  _et al._ , 2021; Wu and Li, 2021; He  _et al._
,  is due to a smaller interlayer distance used therein. To corroborate our
results, in the Supporting Information document we present the calculations of
equilibrium interlayer distances obtained at different levels of DFT. The
value of $P_{z}$ strongly depends on the interlayer distance $d$ and can be
further enhanced by decreasing $d$, for example, by applying out-of-plane
compressive strain. On the other hand, BLG is inversion symmetric and gapless
with a touching of parabolic bands at the $K$ point. When inversion symemetry
is broken–either by adding extra layers comprising a non-centrosymmetric
composite system or by electric field–the BLG develops a gap McCann (2006);
Oostinga  _et al._ (2008); Zhang  _et al._ (2009). The gap opening mechanism
in BLG due to sublattice asymmetry has been widely discussed in the literature
Zhang  _et al._ (2011); Tang  _et al._ (2017), but we want to highlight the
effect of structural details on the resulting gap. In Fig. 1(b), we show the
band structure close to point $K$ of two representative configurations, ABCABC
and ACBA. We found that the band gap $\Delta E$ in graphene/$h$-BN multilayers
can vary from $\lesssim$1 meV to 62 meV, whereas $P_{z}/A$ ranges from
$\sim$0.002 to 4.3 pC/m (see Supporting Information for the tabulated results
for all studied configurations). For 2$h$-BN/BLG/2$h$-BN configurations, in
which one or more layers have vertically aligned atoms of the same type
(below, denoted as primed) $\Delta E$ is on average smaller than in non-primed
stacks due to a “weaker” violation of inversion symmetry. Overall, sandwiching
BLG with polar $h$-BN bilayers leads to a larger gap opening as compared to
one-sided systems. Interestingly, we do not observe any straightforward
relation between $\Delta E$ and $P_{z}$.

In polar $h$-BN multilayers, one expects that the total polarization of
stacked polar bilayers (with the polarization pointing in the same direction)
would scale linearly with the number of interfaces Stern  _et al._ (2021);
Yasuda  _et al._ (2021). A similar behaviour is observed in graphene/$h$-BN
heterostructures. We highlight that if non-polar AA’ $h$-BN bilayers are used
instead, the entire system can still exhibit a gap and finite polarization. We
therefore conclude the local atomic arrangement at the interfaces is at least
an important mechanism responsible for gap opening.

Figure 1: Non-centrosymmetric graphene/$h$-BN heterostructures and their
electronic properties. (a) Schematic representation of two representative
stacking configurations, ACBA and ABCABC, where BLG is deposited on or
encapsulated between polar $h$-BN bilayers. (b) Dispersion relation around the
K point for pristine BLG, ACBA 2$h$-BN/BLG, and ABCABC 2$h$-BN/BLG/2$h$-BN. In
the presence of $h$-BN, BLG develops a gap, the value of which depends on the
details of the structure. (c) Projected band structures for the ABCABC
configuration. The bands at the Fermi level are composed exclusively of the
$p_{z}$ orbitals of C atoms.

From Wannier functions to effective TB models —  As shown in Fig. 1(c), the
higher conductance bands are composed of the $p_{z}$ orbitals of B atoms, the
lower valence bands have clear contribution of the $p_{z}$ orbitals of N
atoms, while four bands around the Fermi level originate from the $p_{z}$
orbitals of C atoms. This clear separation of the $p_{z}$ orbitals of carbon
atoms allows for the construction of the projector Wannier functions basis
sets Scaramucci  _et al._ (2015); Pasquier and Yazyev (2019). The diagonal
matrix elements of the resulting ab initio Hamiltonian correspond to the on-
site energies of the $p_{z}$ orbitals. Build-in electric field arising from
$h$-BN bilayers is then reflected as an asymmetry of the on-site terms. To
quantify this sublattice symmetry breaking, we introduce the notion of
sublattice imbalance $\Delta
V_{\mathrm{sublattice,U/L}}=V_{\mathrm{A,U/L}}-V_{\mathrm{B,U/L}}$, that is, a
potential difference between A and B sublattices in the same (upper or lower)
layer, and the layer imbalance $\Delta
V_{\mathrm{layer}}=(V_{\mathrm{A,L}}+V_{\mathrm{B,L}}-V_{\mathrm{A,U}}-V_{\mathrm{B,U}})/2$
characterizing a potential difference between two layers. Larger values of
$\Delta V_{\mathrm{layer}}$ lead to a stronger modification of the BLG
spectrum. Once the on-site energies from wannierization are obtained, we use
them for the construction of an electrostatic potential distribution function
$V(\mathbf{r})$, which is then added to a simple nearest-neighbour TB model of
AB-stacked BLG

$H=\begin{pmatrix}V(\mathbf{r})&t_{||}&0&0\\\
t_{||}&V(\mathbf{r})&t_{\perp}&0\\\ 0&t_{\perp}&V(\mathbf{r})&t_{||}\\\
0&0&t_{||}&V(\mathbf{r})\end{pmatrix}.$ (1)

The in-plane hopping $t_{||}=-2.8$ eV and interlayer hopping $t_{\perp}=-0.3$
eV of the Wannier Hamiltonian are consistent with previous reports McCann and
Koshino (2013). To accurately mimic the experimental setup, we impose on
$V(\mathbf{r})$ i) a three-fold rotational symmetry $C_{3v}$ along the $z$
axis, and ii) constant value in the triangular domains except the smooth
domain boundaries of 4 nm thickness Woods  _et al._ (2021).

Figure 2: Electronic properties of the BLG superlattices in a polar
environment. (a) Electrostatic potential distribution $V(\mathbf{r})$ showing
the triangular domains. (b) Corresponding potential cross sections for lower
($V_{L}$) and upper ($V_{U}$) layers along the line connecting the centers of
triangular domains. Grey shading indicates the domain boundary width. (c) Band
structure evolution along $\mathrm{\Gamma}$-$M$-$K$-$\mathrm{\Gamma}$ path at
different superlattice periodicities $L$ for the on-site energies
corresponding to the ABCABC stacking. At $L=50$ nm, we observe two subsets of
flat bands ($E\sim-8$ and $E\sim-18$ meV) decoupled from the remote bands.
Upon increasing $L$, a sequence of flat-band manifolds develops at the Fermi
level. (d) Variation of the energy gaps above and below the first flat band
manifold ($\Delta E_{0}$ and $\Delta E_{-1}$, respectively), together with its
bandwidth $W$, as a function of $L$. From $L=30-50$ nm, the midgap $\Delta
E_{0}$ decreases by $\sim$20 meV, whereas $W$ and $\Delta E_{-1}$ do not
change by more than 3 meV.

In Figs. 2(a,b), we show a distribution of $V(\mathbf{r})$ constructed using
the on-site terms for the ABCABC configuration and the potential cross
sections along the line connecting the centers of the domains, respectively.

Flat-band manifolds —  Once parameters dependent on the local structure and
composition are clarified, we will address the effect of the superlattice
period $L$ as the main tuning knob of the discussed heterostructures. From now
on, we will focus on the ABCABC stacking configuration characterized by the
largest energy gap and $\Delta V_{\mathrm{layer}}=32$ meV. The results of the
wannierization for other heterostructures are presented in the Supporting
Information. In Fig. 2(c), we show the band structure evolution as a function
of the superlattice period $L$ along the high-symmetry path in the mini-
Brillouin zone (mBZ). The spectral symmetry with respect to $E=0$ is broken
due to the asymmetry between sublattices in the two layers. From $L=20$ nm
(not shown here), a set of two energy bands starts to become disconnected from
the rest of the occupied bands. At approximately $L=30$ nm, the system
develops a gap of 43 meV and these two bands (around $E\sim-18$ meV) further
shift in energy. This subspace is stable in a broad range of values of $L$ as
shown in Fig. 2(d). As $L$ increases, more bands shift towards the center of
the spectrum, forming a series of separated flat-band manifolds composed of
even number of bands. To characterize these bands, we compute the flatness
ratio $F$, defined as $F=\Delta/W$, where $\Delta$ is the minimal energy
separation from other bands and $W$ is the bandwidth. At $L=50$ nm, there is a
subset of two bands around $E=-8$ meV with $W\sim 8$ meV (and $F=0.5$) and
four bands stack around $E=-20$ meV with $W\sim 5$ meV ($F=0.3$). We emphasize
that the flat bands should appear for all unprimed configuration with finite
$\Delta V_{\mathrm{layer}}$, but at larger $L$. For primed configurations,
though, the situation is more complex–a parabolic dispersion at the $K$ point
is still present at $L=30$ nm, but then it evolves into a Dirac point close to
the charge neutrality (see Supporting Information).

A natural question to ask is whether the observed flat bands are topological,
similarly to minimally twisted BLG Vaezi  _et al._ (2013); Zhang  _et al._
(2013); Huang  _et al._ (2018); Liu  _et al._ (2020) or other bipartite
lattice systems Ma  _et al._ (2020); Regnault  _et al._ (2022). For now, we
focus on two flat-band subspaces appearing at $L=50$ nm (shaded in Fig. 2(c)).
The states have weights on both layers, either in the corners or at the
centers of domains (consult Fig. 1 in Supporting Information). The two-fold
degeneracy is a consequence of the fact that the supercell Hamiltonian (1)
features the $C_{3v}$ symmetry. As a consequence of band degeneracy at the
$\Gamma$ point, the Wilson loop phase $\theta$ is pinned at zero, resulting in
a zero winding. In order to investigate whether the bands can be gapped by a
perturbation, we construct an effective Hamiltonian $H_{\mathrm{eff}}$ in the
upper flat bands subspace. We start with an experimentally accessible way to
lift the degeneracy by an external bias $V_{0}$, which can be realized by a
term proportional to the $\sigma_{z}$ Pauli matrix. We found that the states
remain degenerate for any globally applied realistic electric field: a
perturbation $V_{0}$ of the order of $10$ eV results in sub-meV band
splitting. If we allow for local perturbations (that is, adding the $V_{0}$ to
the small subset of atomic sites), we can achieve a splitting of few meV for
$V_{0}$ being of the order of few eV. We highlight that the exact value of
$V_{0}$ and the number of sites to which the perturbation must be applied is
rather fine-tuned. The general conclusion is that $V_{0}$ does not change the
topology of the bands of interest.

Conclusions —  In summary, we have discussed an experimentally accessible way
to alter the electronic properties of bilayer graphene, and vdW materials in
general, by electrostatic proximity effect. We showed that the electronic
structure of bilayer graphene can be altered by the electrostatic proximity to
polar $h$-BN, and the relevant vertical polarization $P_{z}$ is controlled by
the stacking of individual layers. We further identified stacking
configurations that result in largest gap $\Delta E$ and polarization $P_{z}$.
We effectively captured the electrostatic potential pattern arising from polar
$h$-BN bilayers by performing wannierization onto the $p_{z}$ orbitals of
carbon atoms and used the on-site energies to build TB Hamiltonians for
modeling superlattices of relevant periodicities. By analyzing the TB model
for the selected ABCABC stacking of the encapsulated heterostructure, we found
a series of flat-band manifolds close to the charge neutrality for a broad
range of superlattice periods $L$. The obtained flat bands manifolds have
bandwidths comparable to the bands in twisted bilayer graphene at the magic
angle ($5-10$ meV Cao  _et al._ (2018a, b)). Moreover, the emergence of flat
bands has been observed in TBG that is partially aligned to the underlying
$h$-BN substrate Wong  _et al._ . While the flat bands subspaces are
topologically trivial in a single-particle picture, we expect that
interactions can open a gap in the many-body spectrum without mixing with
remote bands. Similarly to bilayer graphene or transition metal
dichalcogenides, strong electron-electron interactions are expected to lift
spin and valley degeneracies, therefore inducing topologically distinct phases
Zhang  _et al._ (2013). This is particularly important in the light of
experimental realization of the quantum anomalous Hall state with Chern number
$C=\pm 1$ at 3/4-filling in the magic-angle twisted bilayer graphene aligned
with encapsulating $h$-BN Sharpe  _et al._ (2019); Serlin  _et al._ (2020). We
leave the analysis of correlations in $h$-BN/BLG polar heterostructures for
future work. A lack of the one-to-one correspondence between the polarization
and on-site potentials suggests the importance of electric field screening
effects Tepliakov  _et al._ (2021). Therefore, it is of great interest to
generalize the considerations to more complex graphene/$h$-BN multilayer
heterostructures, including the coupling to the in-plane polar textures
Bennett  _et al._ (2023, ).

Acknowledgements —  We thank Daniele Passerone for fruitful discussions. This
work was partially supported by the Swiss National Science Foundation (grant
No. 172543). The computations were performed at the Swiss National
Supercomputing Centre (CSCS) under Projects No. s1008 and s1146.

Supporting information —  Details of the computational methodology, discussion
on the interlayer distances, methods for computing polarization, details of
the Wannier function analysis, discussion of the localization of flat-band
states, band structures of primed configurations, tabulated results for all
studied configurations.

## References

  * Novoselov  _et al._ (2016) K. S. Novoselov, A. Mishchenko, A. Carvalho, and A. H. C. Neto, Science 353, aac9439 (2016).
  * Liu  _et al._ (2016) Y. Liu, N. O. Weiss, X. Duan, H.-C. Cheng, Y. Huang, and X. Duan, Nature Reviews Materials 1, 16042 (2016).
  * Duong  _et al._ (2017) D. L. Duong, S. J. Yun, and Y. H. Lee, ACS Nano 11, 11803 (2017).
  * Cao  _et al._ (2018a) Y. Cao, V. Fatemi, S. Fang, K. Watanabe, T. Taniguchi, E. Kaxiras, and P. Jarillo-Herrero, Nature 556, 43 (2018a).
  * Isobe  _et al._ (2018) H. Isobe, N. F. Q. Yuan, and L. Fu, Phys. Rev. X 8, 041041 (2018).
  * Wu  _et al._ (2018) F. Wu, A. H. MacDonald, and I. Martin, Phys. Rev. Lett. 121, 257001 (2018).
  * Po  _et al._ (2018) H. C. Po, L. Zou, A. Vishwanath, and T. Senthil, Phys. Rev. X 8, 031089 (2018).
  * Yankowitz  _et al._ (2019) M. Yankowitz, S. Chen, H. Polshyn, Y. Zhang, K. Watanabe, T. Taniguchi, D. Graf, A. F. Young, and C. R. Dean, Science 363, 1059 (2019).
  * Lian  _et al._ (2019) B. Lian, Z. Wang, and B. A. Bernevig, Phys. Rev. Lett. 122, 257002 (2019).
  * Saito  _et al._ (2020) Y. Saito, J. Ge, K. Watanabe, T. Taniguchi, and A. F. Young, Nature Physics 16, 926 (2020).
  * Oh  _et al._ (2021) M. Oh, K. P. Nuckolls, D. Wong, R. L. Lee, X. Liu, K. Watanabe, T. Taniguchi, and A. Yazdani, Nature 600, 240 (2021).
  * Löthman  _et al._ (2022) T. Löthman, J. Schmidt, F. Parhizgar, and A. M. Black-Schaffer, Communications Physics 5, 92 (2022).
  * Cao  _et al._ (2018b) Y. Cao, V. Fatemi, A. Demir, S. Fang, S. L. Tomarken, J. Y. Luo, J. D. Sanchez-Yamagishi, K. Watanabe, T. Taniguchi, E. Kaxiras, R. C. Ashoori, and P. Jarillo-Herrero, Nature 556, 80 (2018b).
  * Nuckolls  _et al._ (2020) K. P. Nuckolls, M. Oh, D. Wong, B. Lian, K. Watanabe, T. Taniguchi, B. A. Bernevig, and A. Yazdani, Nature 588, 610 (2020).
  * Serlin  _et al._ (2020) M. Serlin, C. L. Tschirhart, H. Polshyn, Y. Zhang, J. Zhu, K. Watanabe, T. Taniguchi, L. Balents, and A. F. Young, Science 367, 900 (2020).
  * Ledwith  _et al._ (2020) P. J. Ledwith, G. Tarnopolsky, E. Khalaf, and A. Vishwanath, Phys. Rev. Research 2, 023237 (2020).
  * Repellin and Senthil (2020) C. Repellin and T. Senthil, Phys. Rev. Res. 2, 023238 (2020).
  * Stepanov  _et al._ (2021) P. Stepanov, M. Xie, T. Taniguchi, K. Watanabe, X. Lu, A. H. MacDonald, B. A. Bernevig, and D. K. Efetov, Phys. Rev. Lett. 127, 197701 (2021).
  * Wu  _et al._ (2021) S. Wu, Z. Zhang, K. Watanabe, T. Taniguchi, and E. Y. Andrei, Nature Materials 20, 488 (2021).
  * Pierce  _et al._ (2021) A. T. Pierce, Y. Xie, J. M. Park, E. Khalaf, S. H. Lee, Y. Cao, D. E. Parker, P. R. Forrester, S. Chen, K. Watanabe, T. Taniguchi, A. Vishwanath, P. Jarillo-Herrero, and A. Yacoby, Nature Physics 17, 1210 (2021).
  * Saito  _et al._ (2021) Y. Saito, J. Ge, L. Rademaker, K. Watanabe, T. Taniguchi, D. A. Abanin, and A. F. Young, Nature Physics 17, 478 (2021).
  * Dean  _et al._ (2010) C. R. Dean, A. F. Young, I. Meric, C. Lee, L. Wang, S. Sorgenfrei, K. Watanabe, T. Taniguchi, P. Kim, K. L. Shepard, and J. Hone, Nature Nanotechnology 5, 722 (2010).
  * Xue  _et al._ (2011) J. Xue, J. Sanchez-Yamagishi, D. Bulmash, P. Jacquod, A. Deshpande, K. Watanabe, T. Taniguchi, P. Jarillo-Herrero, and B. J. LeRoy, Nature Materials 10, 282 (2011).
  * Dean  _et al._ (2012) C. Dean, A. Young, L. Wang, I. Meric, G.-H. Lee, K. Watanabe, T. Taniguchi, K. Shepard, P. Kim, and J. Hone, Solid State Communications 152, 1275 (2012).
  * Wang  _et al._ (2017) J. Wang, F. Ma, and M. Sun, RSC Adv. 7, 16801 (2017).
  * Constantinescu  _et al._ (2013) G. Constantinescu, A. Kuc, and T. Heine, Phys. Rev. Lett. 111, 036104 (2013).
  * Woods  _et al._ (2021) C. R. Woods, P. Ares, H. Nevison-Andrews, M. J. Holwill, R. Fabregas, F. Guinea, A. K. Geim, K. S. Novoselov, N. R. Walet, and L. Fumagalli, Nature Communications 12, 347 (2021).
  * Yasuda  _et al._ (2021) K. Yasuda, X. Wang, K. Watanabe, T. Taniguchi, and P. Jarillo-Herrero, Science 372, 1458 (2021).
  * Stern  _et al._ (2021) M. V. Stern, Y. Waschitz, W. Cao, I. Nevo, K. Watanabe, T. Taniguchi, E. Sela, M. Urbakh, O. Hod, and M. B. Shalom, Science 372, 1462 (2021).
  * Fei  _et al._ (2018) Z. Fei, W. Zhao, T. A. Palomaki, B. Sun, M. K. Miller, Z. Zhao, J. Yan, X. Xu, and D. H. Cobden, Nature 560, 336 (2018).
  * Wan  _et al._ (2022) Y. Wan, T. Hu, X. Mao, J. Fu, K. Yuan, Y. Song, X. Gan, X. Xu, M. Xue, X. Cheng, C. Huang, J. Yang, L. Dai, H. Zeng, and E. Kan, Phys. Rev. Lett. 128, 067601 (2022).
  * Wang  _et al._ (2022) X. Wang, K. Yasuda, Y. Zhang, S. Liu, K. Watanabe, T. Taniguchi, J. Hone, L. Fu, and P. Jarillo-Herrero, Nature Nanotechnology 17, 367 (2022).
  * (33) R. He, B. Zhang, H. Wang, L. Li, T. Ping, G. Bauer, and Z. Zhong, arXiv:2212.14203 .
  * Wu and Li (2021) M. Wu and J. Li, Proceedings of the National Academy of Sciences 118, e2115703118 (2021).
  * Deb  _et al._ (2022) S. Deb, W. Cao, N. Raab, K. Watanabe, T. Taniguchi, M. Goldstein, L. Kronik, M. Urbakh, O. Hod, and M. Ben Shalom, Nature 612, 465 (2022).
  * Koprivica and Sela (2022) D. Koprivica and E. Sela, Phys. Rev. B 106, 144110 (2022).
  * Wang  _et al._ (2023) C. Wang, L. You, D. Cobden, and J. Wang, Nature Materials 22, 542 (2023).
  * Zheng  _et al._ (2020) Z. Zheng, Q. Ma, Z. Bi, S. de la Barrera, M.-H. Liu, N. Mao, Y. Zhang, N. Kiper, K. Watanabe, T. Taniguchi, J. Kong, W. A. Tisdale, R. Ashoori, N. Gedik, L. Fu, S.-Y. Xu, and P. Jarillo-Herrero, Nature 588, 71 (2020).
  * Ji  _et al._ (2023) J. Ji, G. Yu, C. Xu, and H. J. Xiang, Phys. Rev. Lett. 130, 146801 (2023).
  * Cao  _et al._ (2022) W. Cao, O. Hod, and M. Urbakh, ACS Applied Materials & Interfaces 14, 57492 (2022).
  * Park  _et al._ (2008) C.-H. Park, L. Yang, Y.-W. Son, M. L. Cohen, and S. G. Louie, Phys. Rev. Lett. 101, 126804 (2008).
  * Barbier  _et al._ (2010) M. Barbier, P. Vasilopoulos, and F. M. Peeters, Phys. Rev. B 81, 075438 (2010).
  * Dubey  _et al._ (2013) S. Dubey, V. Singh, A. K. Bhat, P. Parikh, S. Grover, R. Sensarma, V. Tripathi, K. Sengupta, and M. M. Deshmukh, Nano Letters 13, 3990 (2013).
  * Forsythe  _et al._ (2018) C. Forsythe, X. Zhou, K. Watanabe, T. Taniguchi, A. Pasupathy, P. Moon, M. Koshino, P. Kim, and C. R. Dean, Nature Nanotechnology 13, 566 (2018).
  * kun Shi  _et al._ (2019) L. kun Shi, J. Ma, and J. C. W. Song, 2D Materials 7, 015028 (2019).
  * Huber  _et al._ (2020) R. Huber, M.-H. Liu, S.-C. Chen, M. Drienovsky, A. Sandner, K. Watanabe, T. Taniguchi, K. Richter, D. Weiss, and J. Eroms, Nano Letters 20, 8046 (2020).
  * Li  _et al._ (2021) Y. Li, S. Dietrich, C. Forsythe, T. Taniguchi, K. Watanabe, P. Moon, and C. R. Dean, Nature Nanotechnology 16, 525 (2021).
  * Yang  _et al._ (2022) X.-C. Yang, H. Yu, and W. Yao, Phys. Rev. Lett. 128, 217402 (2022).
  * Garcia-Ruiz  _et al._ (2021) A. Garcia-Ruiz, H.-Y. Deng, V. V. Enaldiev, and V. I. Fal’ko, Phys. Rev. B 104, 085402 (2021).
  * Garcia-Ruiz  _et al._ (2023) A. Garcia-Ruiz, V. Enaldiev, A. McEllistrim, and V. I. Fal’ko, Nano Letters (2023).
  * (51) Y. Park, Y. Kim, B. L. Chittari, and J. Jung, arXiv:2304.12874 .
  * Killi  _et al._ (2011) M. Killi, S. Wu, and A. Paramekanti, Phys. Rev. Lett. 107, 086801 (2011).
  * Wu  _et al._ (2012) S. Wu, M. Killi, and A. Paramekanti, Phys. Rev. B 85, 195404 (2012).
  * Ghorashi  _et al._ (2023) S. A. A. Ghorashi, A. Dunbrack, A. Abouelkomsan, J. Sun, X. Du, and J. Cano, Phys. Rev. Lett. 130, 196201 (2023).
  * Zhu  _et al._ (2022) Z. Zhu, S. Carr, Q. Ma, and E. Kaxiras, Phys. Rev. B 106, 205134 (2022).
  * Long  _et al._ (2023) M. Long, Z. Zhan, P. A. Pantaleón, J. A. Silva-Guillén, F. Guinea, and S. Yuan, Phys. Rev. B 107, 115140 (2023).
  * Fumega and Lado (2023) A. O. Fumega and J. L. Lado, Nanoscale 15, 2181 (2023).
  * Note (1) We point out that a discrepancy between our values for $P_{z}/A$ and Refs. Yasuda2021, Wu2021a, He2022 is due to a smaller interlayer distance used therein. To corroborate our results, in the Supporting Information document we present the calculations of equilibrium interlayer distances obtained at different levels of DFT.
  * McCann (2006) E. McCann, Phys. Rev. B 74, 161403 (2006).
  * Oostinga  _et al._ (2008) J. B. Oostinga, H. B. Heersche, X. Liu, A. F. Morpurgo, and L. M. K. Vandersypen, Nature Materials 7, 151 (2008).
  * Zhang  _et al._ (2009) Y. Zhang, T.-T. Tang, C. Girit, Z. Hao, M. C. Martin, A. Zettl, M. F. Crommie, Y. R. Shen, and F. Wang, Nature 459, 820 (2009).
  * Zhang  _et al._ (2011) W. Zhang, C.-T. Lin, K.-K. Liu, T. Tite, C.-Y. Su, C.-H. Chang, Y.-H. Lee, C.-W. Chu, K.-H. Wei, J.-L. Kuo, and L.-J. Li, ACS Nano 5, 7517 (2011).
  * Tang  _et al._ (2017) S. Tang, W. Wu, X. Xie, X. Li, and J. Gu, RSC Adv. 7, 9862 (2017).
  * Scaramucci  _et al._ (2015) A. Scaramucci, J. Ammann, N. A. Spaldin, and C. Ederer, Journal of Physics: Condensed Matter 27, 175503 (2015).
  * Pasquier and Yazyev (2019) D. Pasquier and O. V. Yazyev, 2D Mater. 6, 025015 (2019).
  * McCann and Koshino (2013) E. McCann and M. Koshino, Reports on Progress in Physics 76, 056503 (2013).
  * Vaezi  _et al._ (2013) A. Vaezi, Y. Liang, D. H. Ngai, L. Yang, and E.-A. Kim, Phys. Rev. X 3, 021018 (2013).
  * Zhang  _et al._ (2013) F. Zhang, A. H. MacDonald, and E. J. Mele, Proceedings of the National Academy of Sciences 110, 10546 (2013).
  * Huang  _et al._ (2018) S. Huang, K. Kim, D. K. Efimkin, T. Lovorn, T. Taniguchi, K. Watanabe, A. H. MacDonald, E. Tutuc, and B. J. LeRoy, Phys. Rev. Lett. 121, 037702 (2018).
  * Liu  _et al._ (2020) Y.-W. Liu, Y. Su, X.-F. Zhou, L.-J. Yin, C. Yan, S.-Y. Li, W. Yan, S. Han, Z.-Q. Fu, Y. Zhang, Q. Yang, Y.-N. Ren, and L. He, Phys. Rev. Lett. 125, 236102 (2020).
  * Ma  _et al._ (2020) D.-S. Ma, Y. Xu, C. S. Chiu, N. Regnault, A. A. Houck, Z. Song, and B. A. Bernevig, Phys. Rev. Lett. 125, 266403 (2020).
  * Regnault  _et al._ (2022) N. Regnault, Y. Xu, M.-R. Li, D.-S. Ma, M. Jovanovic, A. Yazdani, S. S. P. Parkin, C. Felser, L. M. Schoop, N. P. Ong, R. J. Cava, L. Elcoro, Z.-D. Song, and B. A. Bernevig, Nature 603, 824 (2022).
  * (73) D. Wong, K. P. Nuckolls, M. Oh, R. L. Lee, K. Watanabe, T. Taniguchi, and A. Yazdani, arXiv:2303.08246 .
  * Sharpe  _et al._ (2019) A. L. Sharpe, E. J. Fox, A. W. Barnard, J. Finney, K. Watanabe, T. Taniguchi, M. A. Kastner, and D. Goldhaber-Gordon, Science 365, 605 (2019).
  * Tepliakov  _et al._ (2021) N. V. Tepliakov, Q. Wu, and O. V. Yazyev, Nano Lett. 21, 4636 (2021).
  * Bennett  _et al._ (2023) D. Bennett, G. Chaudhary, R.-J. Slager, E. Bousquet, and P. Ghosez, Nature Communications 14, 1629 (2023).
  * (77) D. Bennett, W. J. Jankowski, G. Chaudhary, E. Kaxiras, and R.-J. Slager, arXiv:2305.01404 .


# Untangling Knots: Leveraging LLM for Error Resolution in Computational
Notebooks

Konstantin Grotov<EMAIL_ADDRESS>JetBrains ResearchGermany ,
Sergey Titov<EMAIL_ADDRESS>JetBrains ResearchCyprus ,  Yaroslav
Zharov<EMAIL_ADDRESS>JetBrains ResearchGermany and  Timofey
Bryksin<EMAIL_ADDRESS>JetBrains ResearchCyprus

(2018)

###### Abstract.

Computational notebooks became indispensable tools for research-related
development, offering unprecedented interactivity and flexibility in the
development process. However, these benefits come at the cost of
reproducibility and an increased potential for bugs. There are many tools for
bug fixing; however, they are generally targeted at the classical linear code.
With the rise of code-fluent Large Language Models, a new stream of smart bug-
fixing tools has emerged. However, the applicability of those tools is still
problematic for non-linear computational notebooks. In this paper, we propose
a potential solution for resolving errors in computational notebooks via an
iterative LLM-based agent. We discuss the questions raised by this approach
and share a novel dataset of computational notebooks containing bugs to
facilitate the research of the proposed approach.

Computational notebooks, Agents, Planning

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††conference:
Make sure to enter the correct conference title from your rights confirmation
email; June 03–05, 2018; Woodstock, NY††booktitle: Woodstock ’18: ACM
Symposium on Neural Gaze Detection, June 03–05, 2018, Woodstock, NY††isbn:
978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Planning for
deterministic actions

##  1\. Introduction

Computational notebooks have become a popular medium for development during
the last decade, especially for data analysis, machine learning (Perkel,
2018), and creating educational (Barba et al., 2019), or scientific content
(Perkel, 2021). One of the main features of computational notebooks is the
possibility of a non-linear development process enabled by creating and
executing cells with source code in any order. While this feature allows one
to efficiently go through hypotheses and experiment a lot (Rule et al., 2018),
it causes high code entanglement (Ramasamy et al., 2023; Rule et al., 2018)
and, thus, a higher amount of errors in code. As a result, notebooks are
struggling with low reproducibility rates (Pimentel et al., 2021, 2019; Wang
et al., 2021), when notebooks hardly show the same results after re-running or
even can not be executed without errors. And on the other hand, the notebooks’
code often contains many stylistic errors (Grotov et al., 2022).

In the previous works, several approaches were suggested to solve these
issues. To mitigate the code entanglement problem, an algorithm was proposed
for the preventive linearization (Macke et al., 2020). While existing linters
improve code quality, they are not adapted for notebooks and usually produce
false positive errors. Hence, in the recent studies, additional notebook-
specific linters and guidelines (Samuel et al., 2020) and tools to match them
were presented. Also, special versioning systems (Deo et al., 2022) were
proposed to solve the reproducibility problem. The emergence of coding Large
Language Models (OpenAI, 2023; Roziere et al., 2023; Li et al., 2023; Jiang et
al., 2024) (LLMs) opens new prospects in this direction since these models can
dynamically analyze the context, and address errors in a more nuanced way,
compared to static analysis tools.

LLMs demonstrated decent bug-fixing abilities in common coding settings —
repositories with regular scripts code in separate files (Jin et al., 2023).
However, there is not as much work on applying these techniques to the format
of notebooks (McNutt et al., 2023). We suggest that solving the bug-fixing
problem in the context of notebooks will benefit both fields — LLMs and
computational notebooks. For the former, notebooks are an example of an
interactive environment like works by Fan et al. (2022); Goss et al. (2023) to
test their planning and execution abilities. In turn, LLMs could solve the
most crucial problems for notebooks: code quality and reproducibility.

In this work, we take the first steps toward such a solution. Namely, we

  1. (1)

collect and share with the community a dataset (Grotov et al., 2023) of Python
projects that contain computational notebooks with at least one thrown
exception;

  2. (2)

outline the agent-based (Shavit et al., 2023; Durante et al., 2024) approach
for LLM usage in the notebooks and discuss the research questions we deem
important for the practical applicability of this approach, as well as the way
to estimate the quality of the solution.

##  2\. Dataset of Notebook Errors

Since the rise of the popularity of LLMs, they have already been used as
error-solving tools (Xia et al., 2023; Zhang et al., 2023). However, they were
not applied to computational notebooks. To start filling that gap, we are
sharing the dataset of computational notebooks containing errors (Grotov et
al., 2023). The dataset contains 10,000 notebooks collected by taking a subset
from a bigger unpublished corpus. The full corpus consists of all repositories
with 10 or more stars on GitHub and all repositories containing Kotlin files.
Before further filtering, we excluded forks and no longer actively maintained
(archived) repositories to keep the variety of code in the dataset; we also
removed the repositories with the last commit date before 2020 to have as many
of the modern errors as possible and since popular datasets from Grotov et al.
(2022) and Pimentel et al. (2019) provide a good subset of older notebooks. To
gather the notebooks, we took the repositories with Python or Jupyter marked
as primary languages, collected all notebooks containing errors, and randomly
took the first 10,000 notebooks out of 12,609 from 4,636 repositories. We
classified a notebook as containing errors if any of the cells’ outputs
resulted in a Python exception. The detection of these errors was facilitated
by using the output_type property found within the JSON source of the notebook
cells. The GitHub snapshot was accurate as of February 21, 2024. All the
repositories in the dataset have a permissive license for future research
(MIT, Apache-2.0, BSD-3-Clause) (Golubev et al., 2020).

Figure 1. (a) Distribution of top-8 most common errors in GitHub notebooks.
(b) The ratio of internal and external errors for every of the top-8 error
types.

To that end, we analyzed the distribution of the most common error types in
the collected dataset. We selected all cells that contain errors in the
output. Then, for every error, we determined the location of the code that
raised the exception; we denote an error as internal if this location is
within the notebook and as external otherwise. We determined every error’s
type using the default Python exception notation. Even though the dataset
presents various error types (354 different types), most of the errors belong
to a few classes. As illustrated in Figure 1 (a), the distribution of diverse
error types aligns with previous studies on notebook reproducibility (Pimentel
et al., 2019). Notably, most errors (78% of all errors found in the dataset)
fall into eight categories. It is worth mentioning that the rest of the
categories are sparse, and 282 of them contain less than 5 errors each.
However, some of these are pretty narrow and understandable, such as
SyntaxError or ModuleNotFoundError, while the top four error types are pretty
general and could appear in various circumstances. These top-4 errors can be
linked to diverse low-level issues commonly encountered in the non-linear
development process (Pimentel et al., 2021) used in notebooks, as well as the
loss of focus associated with it. For instance, a NameError could occur when
the execution order becomes non-linear, and cells in the notebook are used
before they are initialized. Furthermore, TypeError and ValueError errors
might arise due to incorrect modifications of objects when the execution
reverts back to a linear order. For instance, these errors could arise from
the use of temporary variables that have the same name and are reassigned to
different parts of the notebook. Also, our visual analysis of randomly chosen
errors shows that many errors result from incorrect usage of packages and
methods or their incompatibility with the installed environment. It’s also
important to mention that, as shown in Figure 1 (b), most of the mentioned
errors are external (caused by the incorrect behavior outside of the notebook,
for example, in an imported package), so the gathering additional context is
valuable for solving these errors. To facilitate further research, we have
made the dataset publicly available and shared the analysis code on Hugging
Face (Grotov et al., 2023).

##  3\. Future Work

In this section, we briefly outline how to solve such a problem using the
approach of a developing field of AI agents. Following this, we pose the
research questions that we consider vital for the success of this approach.

###  3.1. Error Solving using LLM Agents

The field of LLM-based AI agents gained traction recently. Such agents have
shown abilities to engage with software engineering tasks (Tufano et al.,
2024; Si et al., 2024), interact with web environments (Drouin et al., 2024;
Zhou et al., 2023), or operate an embodied agent (Wang et al., 2023a). We
suggest employing such agents for iterative error resolution in computational
notebooks. The core concept is to allow the agent to code and execute cells,
utilizing the notebook’s natural interactivity. The agent, therefore, will be
able to expand and refine its context before solving the error.

Expanding the context by utilizing methods akin to ones that are used for
regular code (Wang et al., 2023b), such as incorporating notebook feedback on
the proposed resolution or drawing from data sources and the notebook itself,
can indeed be insightful. However, the interactive nature of notebooks might
offer additional value.

The AI agent we envision gains control only after an error occurs and the user
seeks a solution. This approach allows the agent to investigate the
environment independently and will not destructively interfere with the
developers’ workflow. As mentioned, this boundary-free exploration is
achievable by creating and executing temporary cells and processing the
feedback. This provides an avenue for yielding supplemental contextual
information which cannot be explicitly present.

###  3.2. Research Questions

We pose several research questions for ourselves and the community to solve.
We think solving these questions will streamline the further development of
LLM-based AI agents for the computational notebooks’ bugfixing.

_RQ1: How to develop a secure playground?_ The security aspect is vital as the
environment should provide interaction and error-solving from the agent side,
as well as reversibility to the initial user’s codebase. We are considering
various ways to implement this, from duplicating a notebook and running it in
a sandbox to railguard the agent’s actions.

_RQ2: Which metrics should be used to specifically evaluate an agent’s
performance in a computational notebook environment?_ While we can certainly
apply existing metrics to make comparisons among agents within a notebook,
these measures may not always provide a meaningful view of performance given
the context of a notebook environment. More specifically, even if a cell shows
no errors, it is possible that the outcome may not align with the developer’s
original goal due to contextual differences.

_RQ3: What tools can (or should) an agent utilize in a computational
notebook?_ The expansion of the toolset increases the length of the context
window occupied by the tool description. Since agents are, naturally,
iterative, the context window can quickly be overflown by the dialogue. This
leads to a desire to find a minimal yet complete set of tools. While we
mentioned several possible tools for extending the context behind the
exception message, we hypothesize that it is beneficial to study the impact of
different tools on the agent performance.

_RQ4: Is it possible to accurately solve errors in computational notebooks
using Open-Source LLMs?_ Open models have already shown decent results for
correcting errors in code (Xia et al., 2023), but their performance on
computational notebooks has not yet been studied. Transitioning to open models
could allow one to solve errors without the need for code sharing with third
parties. Moreover, it will allow fine-tuning according to the relevant
context.

_RQ5: Is the interaction between agents useful for error solving in
computational notebooks?_ One of the questions open to discussion is whether
AI agents perform better as standalone error solvers or when interacting with
other agents. Due to the potential for productive communication among agents
possessing diverse specific features, we posit that these agents may yield
more precise solutions (Iqbal and Sha, 2019). Additionally, interaction with
the user may be helpful (Guo et al., 2021).

##  4\. Conclusion

In this paper, we considered the task of resolving errors in computational
notebooks. We discussed the possibility of solving such issues with the
interactive LLM-based agents. To facilitate further research, we stated
questions that we consider essential for future developments. Alongside this,
we collected, published, and analyzed a dataset of notebooks containing
errors. We hypothesize that solving errors in computational notebooks with
agents can be beneficial for research both on AI agents and human-notebook
interaction.

## References

  * (1)
  * Barba et al. (2019) Lorena A Barba, Lecia J Barker, Douglas S Blank, Jed Brown, Allen B Downey, Timothy George, Lindsey J Heagy, Kyle T Mandli, Jason K Moore, David Lippert, et al. 2019\.  Teaching and learning with Jupyter.  _Recuperado: https://jupyter4edu. github. io/jupyter-edu-book_ (2019), 1–77.
  * Deo et al. (2022) Nachiket Deo, Boris Glavic, and Oliver Kennedy. 2022.  Runtime provenance refinement for notebooks. In _Proceedings of the 14th International Workshop on the Theory and Practice of Provenance_. 1–4.
  * Drouin et al. (2024) Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H. Laradji, Manuel Del Verme, Tom Marty, Léo Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, Nicolas Chapados, and Alexandre Lacoste. 2024.  WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?  arXiv:2403.07718 [cs.LG]
  * Durante et al. (2024) Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. 2024\.  Agent ai: Surveying the horizons of multimodal interaction.  _arXiv preprint arXiv:2401.03568_ (2024).
  * Fan et al. (2022) Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar. 2022.  MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge. In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.  https://openreview.net/forum?id=rc8o_j8I8PX
  * Golubev et al. (2020) Yaroslav Golubev, Maria Eliseeva, Nikita Povarov, and Timofey Bryksin. 2020.  A study of potential code borrowing and license violations in java projects on github. In _Proceedings of the 17th International Conference on Mining Software Repositories_. 54–64.
  * Goss et al. (2023) Stephen A. Goss, Robert J. Steininger, Dhruv Narayanan, Daniel V. Olivença, Yutong Sun, Peng Qiu, Jim Amato, Eberhard O. Voit, Walter E. Voit, and Eric J. Kildebeck. 2023.  Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence Agents.  arXiv:2301.11891 [cs.AI]
  * Grotov et al. (2022) Konstantin Grotov, Sergey Titov, Vladimir Sotnikov, Yaroslav Golubev, and Timofey Bryksin. 2022.  A large-scale comparison of Python code in Jupyter notebooks and scripts. In _Proceedings of the 19th International Conference on Mining Software Repositories_. 353–364.
  * Grotov et al. (2023) Konstantin Grotov, Sergey Titov, Yaroslav Zharov, and Timofey Bryksin. 2023.  Dataset of Errors in Jupyter Notebooks.  https://huggingface.co/datasets/JetBrains-Research/jupyter-errors-dataset.
  * Guo et al. (2021) Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021.  Abg-CoQA: Clarifying Ambiguity in Conversational Question Answering. In _3rd Conference on Automated Knowledge Base Construction_.  https://openreview.net/forum?id=SlDZ1o8FsJU
  * Iqbal and Sha (2019) Shariq Iqbal and Fei Sha. 2019.  Actor-attention-critic for multi-agent reinforcement learning. In _International conference on machine learning_. PMLR, 2961–2970.
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024\.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_ (2024).
  * Jin et al. (2023) Matthew Jin, Syed Shahriar, Michele Tufano, Xin Shi, Shuai Lu, Neel Sundaresan, and Alexey Svyatkovskiy. 2023.  InferFix: End-to-End Program Repair with LLMs. In _Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (, San Francisco, CA, USA,) _(ESEC/FSE 2023)_. Association for Computing Machinery, New York, NY, USA, 1646–1656.  https://doi.org/10.1145/3611643.3613892
  * Li et al. (2023) Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023.  Textbooks are all you need ii: phi-1.5 technical report.  _arXiv preprint arXiv:2309.05463_ (2023).
  * Macke et al. (2020) Stephen Macke, Hongpu Gong, Doris Jung-Lin Lee, Andrew Head, Doris Xin, and Aditya Parameswaran. 2020.  Fine-grained lineage for safer notebook interactions.  _arXiv preprint arXiv:2012.06981_ (2020).
  * McNutt et al. (2023) Andrew M McNutt, Chenglong Wang, Robert A Deline, and Steven M Drucker. 2023.  On the design of ai-powered code assistants for notebooks. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_. 1–16.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  _ArXiv_ abs/2303.08774 (2023).  https://arxiv.org/abs/2303.08774
  * Perkel (2018) Jeffrey M Perkel. 2018.  Why Jupyter is data scientists’ computational notebook of choice.  _Nature_ 563, 7732 (2018), 145–147.
  * Perkel (2021) Jeffrey M Perkel. 2021.  Ten computer codes that transformed science.  _Nature_ 589, 7842 (2021), 344–349.
  * Pimentel et al. (2019) João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2019.  A large-scale study about quality and reproducibility of jupyter notebooks. In _2019 IEEE/ACM 16th international conference on mining software repositories (MSR)_. IEEE, 507–517.
  * Pimentel et al. (2021) João Felipe Pimentel, Leonardo Murta, Vanessa Braganholo, and Juliana Freire. 2021.  Understanding and improving the quality and reproducibility of Jupyter notebooks.  _Empirical Software Engineering_ 26, 4 (2021), 65.
  * Ramasamy et al. (2023) Dhivyabharathi Ramasamy, Cristina Sarasua, Alberto Bacchelli, and Abraham Bernstein. 2023.  Workflow analysis of data science code in public GitHub repositories.  _Empirical Software Engineering_ 28, 1 (2023), 7.
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023\.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ (2023).
  * Rule et al. (2018) Adam Rule, Aurélien Tabard, and James D Hollan. 2018.  Exploration and explanation in computational notebooks. In _Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems_. 1–12.
  * Samuel et al. (2020) Sheeba Samuel, Frank Löffler, and Birgitta König-Ries. 2020.  Machine learning pipelines: provenance, reproducibility and FAIR data principles. In _International Provenance and Annotation Workshop_. Springer, 226–230.
  * Shavit et al. (2023) Yonadav Shavit, Sandhini Agarwal, Miles Brundage, Steven Adler, Cullen O’Keefe, Rosie Campbell, Teddy Lee, Pamela Mishkin, Tyna Eloundou, Alan Hickey, et al. 2023\.  _Practices for Governing Agentic AI Systems_.  Technical Report. OpenAI Technical Report. https://cdn. openai. com/papers/practices-for ….
  * Si et al. (2024) Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. 2024.  Design2Code: How Far Are We From Automating Front-End Engineering?  arXiv:2403.03163 [cs.CL]
  * Tufano et al. (2024) Michele Tufano, Anisha Agarwal, Jinu Jang, Roshanak Zilouchian Moghaddam, and Neel Sundaresan. 2024.  AutoDev: Automated AI-Driven Development.  arXiv:2403.08299 [cs.SE]
  * Wang et al. (2023b) Hanbin Wang, Zhenghao Liu, Shuo Wang, Ganqu Cui, Ning Ding, Zhiyuan Liu, and Ge Yu. 2023b.  INTERVENOR: Prompt the Coding Ability of Large Language Models with the Interactive Chain of Repairing.  _arXiv preprint arXiv:2311.09868_ (2023).
  * Wang et al. (2021) Jiawei Wang, Li Li, and Andreas Zeller. 2021.  Restoring execution environments of Jupyter notebooks. In _2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)_. IEEE, 1622–1633.
  * Wang et al. (2023a) Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023a.  Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents.  arXiv:2302.01560 [cs.AI]
  * Xia et al. (2023) Chunqiu Steven Xia, Yuxiang Wei, and Lingming Zhang. 2023.  Automated program repair in the era of large pre-trained language models. In _Proceedings of the 45th International Conference on Software Engineering (ICSE 2023). Association for Computing Machinery_.
  * Zhang et al. (2023) Kechi Zhang, Zhuo Li, Jia Li, Ge Li, and Zhi Jin. 2023.  Self-Edit: Fault-Aware Code Editor for Code Generation.  _arXiv preprint arXiv:2305.04087_ (2023).
  * Zhou et al. (2023) Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. 2023.  WebArena: A Realistic Web Environment for Building Autonomous Agents.  arXiv:2307.13854 [cs.AI]