# Achieving adjustable elasticity with non-affine to affine transition

Xiangying Shen1,2,3†, Chenchao Fang1,3†, Zhipeng Jin1,3, Hua Tong4,5, Shixiang
Tang1, Hongchuan Shen1, Ning Xu6, Jack Hau Yung Lo1∗, Xinliang Xu2,7∗, Lei
Xu1,3∗ 1 Department of Physics, The Chinese University of Hong Kong, Hong
Kong, China
2 The Beijing Computational Science Research Center, Beijing, China
3 Shenzhen Research Institute, The Chinese University of Hong Kong, Shenzhen,
China
4 School of Physics and Astronomy, Shanghai Jiao Tong University, Shanghai
200240, China
5 Department of Physics, University of Science and Technology of China, Hefei
230026, China
6 Hefei National Laboratory for Physical Sciences at the Microscale, CAS Key
Laboratory of Soft Matter Chemistry, and Department of Physics, University of
Science and Technology of China, Hefei 230026, China
7 Department of Physics, Beijing Normal University, Beijing 100875, China

###### Abstract

For various engineering and industrial applications, it is desirable to
realize mechanical systems with broadly adjustable elasticity in response to
external environment. Here in this work, we discover a topology-correlated
transition between affine and non-affine regimes in elasticity in both 2D and
3D packing-derived networks. Based on this transition, we numerically design
and experimentally realize multifunctional systems with adjustable elasticity.
Within one system, we achieve solid-like affine response, liquid-like non-
affine response, and a continuous tunability in between. Moreover, the system
also exhibits a broadly tunable Poisson’s ratio from positive to negative
values, which is of practical interest for energy absorption and fracture
resistant materials. Our study reveals a fundamental connection between
elasticity and network topology, and demonstrates its practical potential in
designing mechanical systems and metamaterials.

Metamaterials kadic2013  are artificial materials with unusual properties
difficult to find in nature. These properties originate mainly from their
structures, instead of their compositions. In particular, the mechanical
metamaterials berger2017 ; rock2017 ; reid2018 ; nicolaou2012 ; coulais2017 ;
kane2014 ; zheng2014 ; coulais2018 ; goodrich2015 ; florijn2014  are specially
designed structures with unusual elastic responses, such as negative Poisson’s
ratio rock2017 ; reid2018 , negative compressibility nicolaou2012 , non-
reciprocity coulais2017 , topological boundary modes kane2014 , and ultrahigh
stiffness zheng2014 . Adjustable mechanical metamaterials whose elasticities
can be self-adjusted coulais2018 ; goodrich2015 ; florijn2014  are of enormous
interest for industrial and engineering applications. Such materials have been
achieved mostly by regular combination of identical units. However, disordered
jamming structures, which are physically well defined and allow rigorous
mathematical analysis, were recently discovered as another promising approach
rock2017 ; reid2018 ; goodrich2015 . In this study, we realize multifunctional
elastic systems with disordered jamming structures.

Jamming is a special state in material packing that occurs in a wide range of
occasions, such as the jam of particles, emulsions and foams majmudar2007 ;
dauchot2005 ; keys2007 ; alexander1998 ; silbert2002 , traffic jam, tumor
progression tumor  and even in embryonic morphogenesis embryo . At the jamming
transition, rigidity arises and the system changes from fluid-like to solid-
like liu1998 ; olsson2007 ; head2009 . However, a marginally-jammed solid near
jamming transition differs significantly from typical solid: for a typical
solid, different moduli such as the shear modulus ($G$) and the bulk modulus
($K$) are closely related with a roughly constant ratio; while for a
marginally-jammed solid the shear modulus approaches zero but the bulk modulus
remains finite, making their ratio to diverge somfai2005 ; makse1999 ;
makse2000 . This $G\ll K$ behavior is similar to a liquid, which is easy to
shear but difficult to compress. Extensive studies majmudar2007 ; dauchot2005
; keys2007 ; alexander1998 ; silbert2002 ; liu1998 ; olsson2007 ; head2009 ;
zhang2009 ; weitz2004 ; tkachenko1999 ; somfai2005 ; ellenbroek2009 ;
wyart2005epl ; wyart2005 ; makse1999 ; makse2000 ; ellenbroek2006 ; donev2005
; silbert2005 ; zhao2011 ; mao2010 ; xu2010 ; liu2010 ; hecke2010  demonstrate
that the unique elastic properties near jamming come from the non-affine
response, which indicates to what extent the internal strain is independent
from the external load. The non-affine properties also enable amazing designs
of disordered metamaterials based on jamming rock2017 ; reid2018 ;
goodrich2015 . However, an important issue remains open: can such a non-affine
system turn into affine, and realize both affine and non-affine elasticities
within one single system?

We first illustrate this issue in 2D. The network is derived from a 2D packing
system with $N$ particles, as shown in Fig.1(a): to prevent crystallization,
the particles have two sizes with the size ratio of 1:1.4 and the number ratio
of 1:1. The total number of contacts between particles is $N_{C}$ and the
contact interaction is harmonic with identical spring constant $k$. Based on
this packing system, we construct a network with $N$ nodes corresponding to
$N$ particle, and $N_{C}$ bonds (i.e., springs) corresponding to $N_{C}$
contacts (see SI-I). As illustrated in the upper panel of Fig.1(a), we can
vary the contact number by numerically inflating all particles. As a result,
more particles previously not in contact can touch each other, which
corresponds to adding more bonds to the network, as illustrated in the lower
panel. During the particle inflation process, we keep all particle or node
positions fixed and identical to the jamming point configuration, which makes
the experimental operations simple and practical as demonstrated later.

Such a network satisfies the modified Maxwell isostatic theorem Calladine1978
:

$dN-N_{C}=f(d)+N_{0}-N_{S}$ (1)

where $d$ is the dimension, $f(d)=d(d+1)/2$, $N_{0}$ is the number of floppy
modes, and $N_{S}$ is the number of self-stress-states wyart2008 ;
lubensky2015 . The coordination number, $z=2N_{C}/N$, gives the average number
of bonds per node. At the jamming point $N_{0}=N_{S}$ and we obtain the
isostatic condition: $z_{C}=2N_{C}/N=2d-2f(d)/N\approx 2d$ for large enough
$N$.

Above the jamming point (i.e., $z>z_{C}$) the system changes from fluid-like
to solid-like. However, a marginally-jammed solid differs significantly from a
typical solid. The underling reason is the affine versus non-affine elastic
response. The affine response comes from a linear correlation between the
internal displacement field and the external strain at the boundary, which
typically appears in ordered or homogeneous structures. In disordered systems,
however, their non-regular structures can disturb and destroy this linear
correlation, and produce non-affine response. In general, ordered or
homogeneous systems are typically affine while disordered or inhomogeneous
systems are non-affine. However, in this study we realize both affine and non-
affine elasticities within one disordered jamming system.

To realize the affine elasticity in jamming system, we minimize the non-
affinity or the characteristic length $l^{*}$, which is the typical size of a
patch cut out from the system to create a floppy mode wyart2008 ;
ellenbroek2009 ; wyart2005 ; wyart2005epl  (see SI-IV). According to Eq.(1),
$l^{*}$ reaches minimum when the system reaches complete triangulation, as
shown in Fig.1(b) (see Methods). Moreover, the 2D topological invariant, Euler
characteristic, further gives $z=2N_{C}/N=6-12/N\approx 6$ at this complete
triangulation (see Methods). Therefore, we identify a critical coordination
number, $z_{aff}=6$, at which the entire system reaches complete triangulation
and the affine property reaches maximum.

For direct visualization, we plot the internal displacement fields under
external strain for $z<6$ and $z_{aff}=6$ in Fig.1(c). For $z<6$ the internal
fields look rather random, while at $z_{aff}=6$ a linear correlation appears.
Despite the underlying disordered structure, the system at $z_{aff}=6$
exhibits displacement fields almost identical to regular systems, confirming
its affine nature (see SI-II for comparison with high-packing-density
systems).

Above $z_{aff}=6$, cross or intersecting bonds start to emerge and the system
is not a planar graph anymore trudeau1993 ; nakamura2008 ; gimenez2009 : the
topology changes and the Euler characteristic breaks down. The system also
goes beyond the typical packing scenario, as packed particles can only touch
nearby neighbors and never penetrate through them to form cross bonds.
Therefore, this transition corresponds to a fundamental change in both
topology and packing scenario. To reach the cross-bond regime, we first
connect all nearest neighbors in the system to reach complete triangulation,
and then connect the next nearest neighbors with cross bonds.

We further show that the minimum non-affinity achieved at $z_{aff}=6$ actually
approaches zero and the transition is affine. To quantify non-affinity, we
define a dimensionless energy difference between the ideal affine displacement
field and the actual displacement field: $\Delta E=(E_{aff}-E)/E_{aff}$. Here
$E_{aff}$ is the elastic energy stored in the affine displacement field
mandatorily imposed according to a certain external strain, and $E$ is the
energy stored in the real displacement field under the same external strain.
For any non-affine system, the mandatorily-imposed affine displacement field
is not a force equilibrium state and requires more energy than the real
displacement field. Thus $\Delta E$ is non-negative and only approaches zero
as the system approaches affine elasticity. In Fig.1(d) left panel we plot
$\Delta E$ versus $\Delta z=z-z_{C}$ ($z_{C}=4$ for 2D) in our system. For
both shear and compression, there is a transition at $\Delta z=2$ or
$z_{aff}=6$: to its left $\Delta E$ drops rapidly towards zero, and to its
right $\Delta E$ is locked around a small value about $0.1$. This transition
thus enables either a significant adjustment or a lock-in capability.
Moreover, $\Delta E$ approaches zero at $z_{aff}=6$, indicating a non-affine
to affine transition. We show its configuration in Fig.1(d) right panel.

For comparison, we also calculate $\Delta E$ in a completely random system in
Fig.1(e). Once again we observe a kink at $\Delta z=2$, which separates two
distinct regimes and reveals a general feature that the topology change at
$\Delta z=2$ produces two elastic regimes. However, Fig.1(e) also differs
significantly from Fig.1(d): at $\Delta z=2$, $\Delta E$ is far above zero and
thus the kink is not an affine transition. In fact, this random system remains
highly non-affine throughout the entire $\Delta z$ range, which could be a
general feature for common disordered systems.

Therefore, the special combination of particle positions at the jamming point
and bond connections at $\Delta z=2$ leads to a non-affine to affine
transition. To verify it, we perturb our system in Fig.1(f): as each particle
is randomly displaced from the jamming configuration by larger and larger
distance $d$, the non-affinity $\Delta E$ at $\Delta z=2$ rises rapidly from
zero. This unambiguously shows that the affine transition correlates closely
to the jamming configuration, and perturbations in general destroy this affine
transition. The underlying reason is probably the high local uniformity of the
jamming configuration, which is similar to ordered lattice structures but in
sharp contrast to the random system (see right panels of Fig.1d-e). Thus the
jamming configuration’s locally-uniform node positions coupled with
triangulated bond connections produce affine elasticity, and realize this
ordered structure’s property in a disordered system (see Fig.SI-3 in SI-III).

Apparently, besides the jamming transition at $z_{C}=4$, there is another non-
affine to affine transition at $z_{aff}=6$. Our simulations on bulk and shear
moduli $K$ and $G$ can directly verify this transition, as shown in Fig.2(a):
two renormalized moduli, $K/K_{max}$ and $G/G_{max}$, are plotted as a
function of $\Delta z$. Clearly near $\Delta z=0$, $G$ approaches zero while
$K$ remains finite, making their ratio, $K/G$, to diverge (i.e., highly non-
affine). However, for $\Delta z>2$ or $z>6$, the two curves merge together,
indicating a constant ratio of $K/G=K_{max}/G_{max}$ that is the affine-like
behavior. The inset further shows that only in large enough systems, $N>500$,
$z_{aff}$ stabilizes at 6. Fig.2(b) plots the actual ratio, $K/G$, versus
$\Delta z$ for an $N=1024$ system: for $\Delta z<2$, $K/G\sim\Delta z^{-1/2}$,
which is different from the result of $K/G\sim\Delta z^{-1}$ reported in
previous studies hecke2010 ; ellenbroek2009epl  due to our different bond-
addition protocol; while for $\Delta z\geq 2$, $K/G\equiv 2$ and the system is
affine-like.

We derive a universal expression for an arbitrary modulus, $M$, in both affine
and highly non-affine situations (see SI-V):

$M\sim kz\int^{\pi}_{0}\cos^{2}\alpha\cdot P(\alpha)d\alpha$ (2)

with $\alpha$ the angle between an arbitrary bond and its two nodes’ relative
displacement due to external load (see Fig.2(c)), and $P(\alpha)$ the
probability distribution of $\alpha$ in the entire system ellenbroek2006 . We
theoretically calculate $K$ and $G$ with Eq.(2) and compare with numerical
simulations in Fig.2(d) and (e), and observe a good agreement. Therefore,
$P(\alpha)$ in Eq.(2) fundamentally explains affine and non-affine behaviors.
In the non-affine regime, $P(\alpha)$ varies differently in $K$ and $G$
ellenbroek2006 ; ellenbroek2009epl , making $K/G$ varies with $z$. In the
affine regime, however, $P(\alpha)$ is fixed and thus $K/G$ remains unchanged.

Combining affine and non-affine behaviors, we design networks with both types
of tunability: our design is well above the jamming point $z_{C}=4$ and closer
to $z_{aff}=6$, typically at $z>5$ to ensure enough locally-triangulated
regions. One specific design is illustrated in the left panel of Fig.3(a) with
$z=5.28$. Clearly this network contains two distinct local structures: the
affine regions with complete triangulation and the non-affine regions close to
the jamming point structure. These two types of regions exhibit distinct local
properties (see Fig.SI-6 in SI), which form the basis of realizing the two-way
tunability.

The non-affine regime enables us to adjust the ratio, $K/G$, across a broad
range. Because some bonds make more contributions to $K$ and some are more
important to $G$ goodrich2015 , deleting or adding such bonds can
significantly change one modulus while keep the other one stable. We first
reduce $K$ significantly by deleting the bonds important to $K$, as shown by
the first two panels of Fig.3(a). The effect is shown in Fig.3(b) main panel:
as the most important bonds are continuously deleted, $K$ drops dramatically
while $G$ is relatively stable. It even reaches the unusual situation of
$K<G$, as shown in the upper inset. Correspondingly, their ratio $K/G$
decreases significantly as shown in the lower inset. Because $K/G$ typically
correlates to the Poisson’s ratio, $\nu$, this also tunes $\nu$ from positive
to negative, as shown in Fig.3(c) and Movie-1. Therefore, our network realizes
non-affine elasticity and achieves a broad tunability in both $K/G$ and $\nu$.

We then demonstrate the affine-like behavior with a constant $K/G$ in the same
network, based on the middle configuration of Fig.3(a). Clearly this
configuration contains many triangulated regions with $z=6$ locally. According
to our previous result, when the global structure reaches complete
triangulation, the system becomes an affine solid and $K/G\equiv 2$ is fixed.
We now show that the elasticity is also correlated to local structures
wyart2008  (See SI-VI). When cross bonds are added to the locally-triangulated
regions, as indicated by the red bonds in Fig.3(a) right panel, both $K$ and
$G$ increase while their ratio, $K/G$, remains a constant, as shown in
Fig.3(d). Correspondingly, the Poisson’s ratio $\nu$ remains unchanged, as
shown in Fig.3(e) and Movie-2. More interestingly, now $K/G$ is not just fixed
at 2 but can be locked at an arbitrary value, such as 0.5 in this example,
which provides an extra dimension of tunability (See SI-VII).

Similar to Fig.3(a), we can also reduce $G$ while keep $K$ relatively stable
by deleting the bonds important to $G$ only, as shown in Fig.3(f). This
procedure decreases $G$ by two orders of magnitude while keeps $K$ relatively
stable, as shown in Fig.3(g). Due to increasing anisotropy, this operation
does not change $\nu$ significantly (see Fig.3(h) and SI-VIII). Based on the
configuration in Fig.3(f) middle panel, we again add cross bonds in the right
panel and confirm the affine function, as shown in Fig.3(i) and (j).

Combining affine and non-affine features, our network realizes three
functions: (1) keeping $K$ stable and significantly changing $G$, (2) keeping
$G$ stable and significantly changing $K$, and (3) keeping $K/G$ stable and
simultaneously changing $K$ and $G$. All functions also apply to other pairs
of moduli and thus achieve a broadly-adjustable elasticity (see SI-VII).

Next we realize this powerful system experimentally. The key issue is to
eliminate the bending energy and realize pure harmonic interaction
theoretically assumed. We use identical springs inside acrylic tubes as bonds
and attach multiple bonds onto one smooth rod that behaves as a node. All
bonds can rotate freely around each node with negligible friction, which
effectively eliminates the bending energy because all bonds prefer to rotate
rather than bend. To avoid conflicts between cross bonds in the same plane, we
design a multi-layer system and place bonds at different heights, as shown in
Fig.4(a)-(c). For simplicity, we design a small system with $N=50$ nodes, and
all bonds can be reversibly added or deleted (see Movie-3).

Because the bulk modulus $K$ cannot be measured easily, we measure the Young’s
modulus $E$ and the shear modulus $G$ (see Methods). We first realize the non-
affine property by significantly reducing $E$ while changing $G$ more gently.
As shown in Fig.4(d), the colored bonds are continuously deleted with the ones
more important to $E$ first, which reduces $E$ significantly while $G$ changes
more gently (see Fig.4(e)). Correspondingly, $E/G$ and $\nu$ change
significantly in Fig.4(f). Similarly, we can reduce $G$ significantly while
keep $E$ relatively stable by deleting another set of bonds, as shown in
Fig.4(g)-(i). These non-affine behaviors agree well with the simulations in
Fig.3.

We then realize the affine property in the same system. We first tune $E$ and
$G$ to a desirable $E/G$, as shown by the black bonds in Fig.4(j). At this
configuration, we then demonstrate the affine function by adding the cross
bonds in triangulated regions, as shown by the colored bonds. This varies both
$E$ and $G$ simultaneously (see Fig.4(k)), while keeps $E/G$ and $\nu$ at a
constant (see Fig.4(l)). Thus we realize an independent tuning on $E$, $G$,
$E/G$, and $\nu$, and achieve affine and non-affine elasticity experimentally.

Furthermore, all our 2D results can be extended to 3D. Similar to complete
triangulation in 2D, when the network is fully tetrahedralized in 3D, a non-
affine to affine transition appears (see Methods). Our simulation confirms
this transition at $z_{aff}=12.8$ in Fig.5(b) ($z_{aff}\neq 12$ due to
polydisperse particle size).

Above $z_{aff}$, cross bonds start to appear. Note that in 3D they go across a
face while in 2D they go across another bond, as shown in Fig.5(a). Similar to
2D, the 3D cross bonds also break a fundamental topology, the Euler-Poincaré
characteristic nakamura2008 , which is the 3D generalization of the Euler’s
characteristic in 2D (see SI-X). Therefore, in both 2D and 3D packing-derived
networks, a fundamental link between the non-affine to affine transition and
the change in topology exists.

Analogous to 2D, adding cross bonds inside locally-tetrahedralized regions can
again lock the non-affinity. As shown in Fig.5(c), the non-affinity $\Delta E$
drops rapidly before $z_{aff}$ and stabilizes around a small value after
$z_{aff}$, confirming the non-affine and affine regimes. Moreover, we can tune
$\Delta E$, $K/G$ or $\nu$ to a desirable value, and then lock it by adding
cross bonds, as shown by the triangle symbols and the inset. Therefore, all
the affine and non-affine functions in 2D are realized in 3D.

For practical applications, realizing actual 3D metamaterials is essential.
Using 3D printing, we achieve such metamaterials in Fig.5 (d)-(f): the
$5\times 5\times 5$ node positions of these three networks are identical and
from the same packing configuration, differing only in their bond number or
$z$. From (d) to (e), $z$ increases from 7.696 to 9.312 by adding non-cross
bonds, which tunes $K/G$ and $\nu$. From (e) to (f), however, $z$ increases
from 9.312 to 10.432 by adding cross bonds, which locks $K/G$ and $\nu$ while
at the same time strengthening the system by increasing both $K$ and $G$.

To verify their performance, we apply identical external strain to these three
networks, and compare their internal strain fields which essentially determine
the elastic responses such as non-affinity and Poisson’s ratio. As illustrated
in Fig.5(g)-(i), row 1 and 2 exhibit significant difference in internal
strain, demonstrating distinct elastic responses due to our tuning operation;
however, row 2 and 3 show almost identical internal strain, verifying the same
elastic responses due to our locking operation (also see Movie-4 and Movie-5).
The forces generated by this strain are also measured as 7.86 N, 10.47 N, and
12.24 N respectively, confirming the expected increase of system rigidity with
respect to $z$. Note that our 3D-printed bonds do exhibit bending force, which
does not match the theoretical assumption perfectly. However, Fig.5(g)-(i)
still show a nice experimental outcome, demonstrating the robustness of our
design, which paves the way for realizing actual metamaterials beyond ideal
spring networks. We further note that all bonds can be 3D-printed as
detachable to realize all functions in a single system (see Movie-6). In
addition, the size range of the bonds and the overall system may vary multiple
orders to satisfy various practical requirements. The plasticity and failure
behaviors can also withstand a broad range of external stress due to its
heterogeneous structure, and avoid the sudden failure of ordered lattice
systems, which are composed mostly by identical bonds that may fail
simultaneously.

To conclude, we discover a non-affine to affine transition at the point of
topology change. Based on this fundamental transition, we realize networks
broadly tunable in $K$, $G$, $K/G$ and $\nu$, and achieve both affine and non-
affine elasticities within one system. Moreover, such systems might be self-
assembled by packing granular materials and control connections either
chemically li2021  or electrically with electromagnetic devices, which may
greatly improve their practical applications. Our study reveals a fundamental
connection between elasticity and topology, and provides a practical design
principle for multifunctional mechanical metamaterials.

Data availability. All the raw data of the figures presented in this
manuscript can be obtained by accessing to the Open Science Framework (DOI
10.17605/OSF.IO/7EQ5Z) or directly visiting https://osf.io/7eq5z/ rawdata .

## References

  * (1) M. Kadic, T. Bückmann, R. Schittny, and M. Wegener, “Metamaterials beyond electromagnetism”, Reports on Progress in Physics 76, 126501 (2013).
  * (2) J. B. Berger, H. N. G. Wadley, R. M. McMeeking, “Mechanical metamaterials at the theoretical limit of isotropic elastic stiffness”, Nature 543, 533 (2017).
  * (3) J. W. Rocks, N. Pashine, I. Bischofberger, C. P. Goodrich, A. J. Liu and S. R. Nagel, “Designing allostery-inspired response in mechanical networks”, Proceedings of the National Academy of Sciences of the United States of America (PNAS) 114, 2520 (2017).
  * (4) D. R. Reid, N. Pashine, J. M. Wozniak, H. M. Jaeger, A. J. Liu, S. R. Nagel and J. J. de Pablo, “Auxetic metamaterials from disordered networks”, Proceedings of the National Academy of Sciences of the United States of America (PNAS) 115, 1384 (2018).
  * (5) Z. G. Nicolaou, A. E. Motter, “Mechanical metamaterials with negative compressibility transitions”, Nature materials 11, 608 (2012).
  * (6) C. Coulais, D. Sounas, A. Alù, “Static non-reciprocity in mechanical metamaterials”, Nature 542, 461 (2017).
  * (7) C. L. Kane, T. C. Lubensky, “Topological boundary modes in isostatic lattices”, Nature Physics 10, 39 (2014).
  * (8) X. Zheng, et. al., “Ultralight, ultrastiff mechanical metamaterials”, Science 344, 1373 (2014).
  * (9) C. Coulais, A. Sabbadini, F. Vink, M. van Hecke, “Multi-step self-guided pathways for shape-changing metamaterials”, Nature 561, 512 (2018).
  * (10) C. P. Goodrich, A. J. Liu and S. R. Nagel, “The Principle of Independent Bond-Level Response: Tuning by Pruning to Exploit Disorder for Global Behavior”, Physical Review Letters, 114, 225501 (2015).
  * (11) B. Florijn, C. Coulais, M. van Hecke, “Programmable mechanical metamaterials”, Physcial Review Letters 113, 175503 (2014).
  * (12) T. S. Majmudar, M. Sperl, S. Luding and R. P. Behringer, “Jamming transition in granular systems”, Physical Review Letters 98, 058001 (2007).
  * (13) O. Dauchot, G. Marty and G. Biroli, “Dynamical heterogeneity close to the jamming transition in a sheared granular material”, Physical Review Letters 95, 265701 (2005).
  * (14) A. S. Keys, A. R. Abate, S. C. Glotzer and D. J. Durian, “Measurement of growing dynamical length scales and prediction of the jamming transition in a granular material”, Nature Physics 4, 260 (2007).
  * (15) S. Alexander, “Amorphous solids: their structure, lattice dynamics and elasticity”, Physics Reports 296, 65 (1998).
  * (16) L. E. Silbert, D. Ertas, G. S. Grest, T. C. Halsey and D. Levine, “Geometry of frictionless and frictional sphere packings”, Physical Review E 65, 031304 (2002).
  * (17) Oswald, L., Grosser, S., Smith, D. M. & Kas, J. A., “Jamming transitions in cancer”, J. Phys. D Appl. Phys. 50, 483001 (2017).
  * (18) Mongera, A. et al. “A fluid-to-solid jamming transition underlies vertebrate body axis elongation”, Nature 561, 401 (2018).
  * (19) A. J. Liu and S. R. Nagel, “Nonlinear dynamics: Jamming is not just cool any more”, Nature 396, 21 (1998).
  * (20) P. Olsson and S. Teitel, “Critical scaling of shear viscosity at the jamming transition”, Physical Review Letters 99, 178001 (2007).
  * (21) D. A. Head, “Critical scaling and aging in cooling systems near the jamming transition”, Physical Review Letters 102, 138001 (2009).
  * (22) E. Somfai, J. N. Roux, J. H. Snoeijer, M. Van Hecke and W. Van Saarloos, “Elastic wave propagation in confined granular systems”, Physical Review E 72, 1301 (2005).
  * (23) H. A. Makse, N. Gland, D. L. Johnson, and L. M. Schwartz, “Why effective medium theory fails in granular materials”, Physical Review Letters 83, 5070 (1999).
  * (24) H. A. Makse, D. L. Johnson, and L. M. Schwartz, “Packing of compressible granular materials”, Physical Review Letters 84, 4160 (2000).
  * (25) Z. X. Zhang, N. Xu, D. T. N. Chen, P. Yunker, A. M. Alsayed, K. B. Aptowicz, P. Habdas, A. J. Liu and S. R. Nagel, “Thermal vestige of the zero-temperature jamming transition”, Nature 459, 230 (2009).
  * (26) D. Weitz, “Packing in the spheres”, Science 303, 968 (2004).
  * (27) A. V. Tkachenko and T. A. Witten, “Stress propagation through frictionless granular material”, Physical Review E 60, 687 (1999).
  * (28) W. G. Ellenbroek, M. van Hecke, and W. van Saarloos, “Jammed frictionless disks: Connecting local and global response”, Physical Review E 80, 061307 (2009).
  * (29) M. Wyart, S. R. Nagel and T. A. Witten, “Geometric origin of excess low-frequency vibrational modes in weakly connected amorphous solids”, EPL 72 486 (2005).
  * (30) M. Wyart, L. E. Silbert, S. R. Nagel, and T. A. Witten, “Effects of compression on the vibrational modes of marginally jammed solids”, Physical Review E 72, 051306 (2005).
  * (31) W. G. Ellenbroek, E. Somfai, M van Hecke and W. Van Saarloos, “Critical scaling in linear response of frictionless granular packings near jamming”, Physical Review Letters 97, 258001 (2006).
  * (32) A. Donev, S. Torquato and F. H. Stillinger, “Pair correlation function characteristics of nearly jammed disordered and ordered hard-sphere packings”, Physical Review E 71, 011105 (2005).
  * (33) L. E. Silbert, A. J. Liu and S. R. Nagel, “Vibrations and Diverging Length Scales Near the Unjamming Transition”, Physical Review Letters 95, 098301 (2005).
  * (34) C. Zhao, K. Tian, N. Xu, “New jamming scenario: from marginal jamming to deep jamming”, Physical review letters 106, 125503 (2011).
  * (35) X. Mao, N. Xu, T. C. Lubensky, “Soft modes and elasticity of nearly isostatic lattices: Randomness and dissipation”, Physical review letters 104, 085504 (2010).
  * (36) N. Xu, V. Vitelli, A. J. Liu, S. R. Nagel, “Anharmonic and quasi-localized vibrations in jammed solids—modes for mechanical failure”, EPL 90, 56001 (2010).
  * (37) A. J. Liu and S. R. Nagel, “The Jamming Transition and the Marginally Jammed Solid”, Annual Review of Condensed Matter Physics 1, 347 (2010).
  * (38) M. van Hecke, “Jamming of soft particles: geometry, mechanics, scaling and isostaticity”, Journal of Physics: Condensed Matter 22, 033101 (2010).
  * (39) C. R. Calladine, “Buckminster Fuller tensegrity structures and Clerk Maxwell rules for the construction of stiff frames”, International Journal of Solids Structures 14, 161 (1978).
  * (40) M. Wyart, H. Liang, A. Kabla, and L. Mahadevan, “Elasticity of floppy and stiff random networks”, Physical Review Letters 101, 215501 (2008).
  * (41) T. C. Lubensky, C. L. Kane, X. Mao, A, Souslov, and K. Sun, “Phonons and elasticity in critically corrdinated lattices”, Reports on Progress in Physics 78, 073901 (2015).
  * (42) R. J. Trudeau, “Introduction to graph theory”, New York: Dover Pub (1993).
  * (43) N. Nakamura, “Geometry, topology and physics”, Bristol: Institue of Physics Publishing (2008).
  * (44) O. Giménez, M. Noy, “Asymptotic enumeration and limit laws of planar graphs”, Journal of the American Mathematical Society 22, 309 (2009).
  * (45) W. G. Ellenbroek, Z. Zeravcic, W. Van Saarloos and M. Van Hecke, “Non-affine response: Jammed packings vs. spring networks”, EPL 87, 34004 (2009).
  * (46) S. Li, B. Deng, A. Grinthal, A. Schneider-Yamamura, J. Kang, R. S. Martens, C. T. Zhang, J. Li, S. Yu, K. Bertoldi, J. Aizenberg, “Liquid-induced topological transformations of cellular microstructures”, Nature 592, 386 (2021).
  * (47) X. Shen, C. Fang, Z. Jin, H. Tong, S. Tang, H. Shen, N. Xu, J. H. Y. Lo, X. Xu, L. Xu “Achieving adjustable elasticity with non-affine to affine transition”, Open Science Framework, DOI 10.17605/OSF.IO/7EQ5Z (or directly visiting https://osf.io/7eq5z/) (2021).

Methods
Deriving the transition in 2D. The non-affinity will be minimized when $l^{*}$
reaches minimum, i.e., no floppy mode can be created even in an arbitrary
smallest patch. A smallest patch is formed by 3 nearby and non-collinear nodes
and we apply the theorem of Eq.(1) onto this 3-node patch: no floppy mode
means $N_{0}=dN-N_{C}-f(d)+N_{S}=0$. Plugging in $d=2,N=3,f(d)=3,N_{S}=0$ (no
self-stress-state can exist in 3 non-collinear nodes), we get $N_{C}=3$. This
means 3 bonds are required for this 3-node patch, i.e., a triangle is needed
for zero floppy mode, as shown in Fig.1(b) left panel. If we eliminate one
bond as shown in the right panel, the 3-node patch can now freely rotate
without costing any energy, which creates one floppy mode. Therefore, to
minimize $l^{*}$ and non-affinity, 3 bonds are required for every 3-node patch
and the system reaches the complete triangulation.

A completely triangulated configuration contains no intersecting or cross
bonds and belongs to the planar graph, which obeys the topological invariant -
the Euler characteristic: $N_{f}-N_{C}+N=2$ trudeau1993 ; nakamura2008 , with
$N_{f}$ the number of faces enclosed by edges or bonds. For a triangulated
large-$N$ system whose boundary influence can be neglected, each face is
surrounded by 3 bonds, and each bond is shared by 2 faces, and thus
$3N_{f}=2N_{C}$. Plugging $N_{f}=2N_{C}/3$ into the Euler characteristic, we
get: $N_{C}=3N-6$ and thus $z=2N_{C}/N=6-12/N\approx 6$. Thus at $z_{aff}=6$
the system reaches the non-affine to affine transition.

Deriving the transition in 3D. The smallest patch in 3D is formed by 4 non-
coplanar nodes and we apply the Maxwell isostatic theorem, $N_{0}=dN-
N_{C}-f(d)=0$, to it. Plugging in $d=3$, $N=4$, $N_{S}=0$, and $f(d)=6$, we
get $N_{C}=6$. Thus the smallest patch without floppy mode is a tetrahedron
with four nodes and six bonds. Thus when the network is fully tetrahedralized,
a non-affine to affine transition will appear.

Measuring $E$ and $G$ in the spring system. We first measure the stress by
fixing one boundary of the system and moving the opposite boundary with $5\%$
strain, in either parallel ($G$) or perpendicular ($E$) direction. We then
divide the stress with the strain and obtain the modulus. Due to boundary
effect, these measurements still have about $10\%$ deviations from the
rigorous definition (see SI-IX) but are good enough experimental
approximations.

Data availability All the raw data of the figures presented in this manuscript
can be obtained by accessing to the Open Science Framework (DOI
10.17605/OSF.IO/7EQ5Z) or directly visiting https://osf.io/7eq5z/.

Code availability All custom computer code or algorithm used to generate
results that are reported in the paper are available upon request.

Acknowledgements The experiments are performed in the Chinese University of
Hong Kong, and we acknowledge the computational support from the Beijing
Computational Science Research Center. L. X. acknowledges the financial
support from NSFC-12074325, Guangdong Basic and Applied Basic Research Fund
2019A1515011171, GRF-14306518, CRF-C6016-20G, CRF-C1018-17G, CUHK United
College Lee Hysan Foundation Research Grant and Endowment Fund Research Grant,
CUHK direct grant 4053354, X. X. acknowledges the financial support from NSFC
11974038 and U1930402, X. S. acknowledges the financial support from Guangdong
Basic and Applied Basic Research Foundation 2019A1515110211, and Project
funded by China Postdoctoral Science Foundation 2020M672824.

Author contributions X. S. and C. F. contribute equally to this research, L.
X. conceived the research, X. S., C. F., J. H. Y. L., X. X. and L. X. designed
the research, X. S. performed most theoretical and numerical analysis, C. F.
performed most experiment, Z. J., S. T., H. S., H. T., and N. X. helped in the
experiment or the simulation, X. S., C. F. and L. X. prepared the manuscript,
L. X. and X. X. supervised the research.

Competing interests The authors declare no competing financial interests.

Correspondence and requests for materials should be addressed to L. X.
(xuleixu@cuhk.edu.hk), X. X. (xinliang@csrc.ac.cn), and J. H. Y. L.
(hylo@cuhk.edu.hk).

Figure 1: Non-affine to affine transition in packing derived networks. a,
converting a particle packing configuration into a spring network. Particles
correspond to nodes and contacts correspond to bonds. We can vary the packing
density by inflating particles (upper panels), which increases the bonds of
the network (lower panels). b, for a smallest patch with 3 nodes cut out of
the system, 3 bonds are required to eliminate the floppy mode. If one bond is
missing, one floppy mode by free rotation will be created. c, displacement
field before ($z=4.04$) and at ($z=6$) the non-affine to affine transition
under shear and compression stress. The field is random before the transition
and becomes linear at the transition. d, the non-affinity $\Delta E$
approaches zero at $\Delta z=z-z_{C}=2$, indicating an almost ideal affine
transition in our packing derived network, whose configuration is shown in the
right panel. e, $\Delta E$ never approaches zero in a random network,
indicating its non-affine nature. Similar to d, a universal kink appears at
$\Delta z=2$ which separates two distinct elastic regimes. The right panel
shows the random network configuration. f, as every node position is perturbed
by larger and larger displacement $d$, the non-affinity $\Delta E$ at the kink
$\Delta z=2$ rises sharply, confirming that the affine transition uniquely
occurs at the jamming point configuration, $d=0$. The right panel shows the
configuration at $d=0.4$ ($d$ has the unit of small particle’s diameter that
is close to an average bond length).

Figure 2: Understanding the variation of moduli at single particle level. a,
the non-affine to affine transition at $\Delta z=2$ in an $N=1024$ system.
When $\Delta z<2$, $G/G_{max}$ and $K/K_{max}$ vary differently; however they
collapse onto one curve for $\Delta z>2$. The inset shows that the transition
stabilizes at $\Delta z=2$ for $N>500$ systems. b, $K/G\sim\Delta z^{-1/2}$
before the transition while $K/G=2$ after the transition. c, the schematics of
the angle $\alpha$ between an arbitrary bond,
$\vec{r}_{ij}=\vec{r}_{j}-\vec{r}_{i}$, and its two nodes’ relative
displacement under external stress, $\vec{\delta r}_{ij}=\vec{\delta
r}_{j}-\vec{\delta r}_{i}$. d, comparison between theory and simulation for
renormalized shear modulus, $G/G_{max}$. e, comparison between theory and
simulation for renormalized bulk modulus, $K/K_{max}$. In both d and e, the
agreement is reasonable with the largest deviation around $10\%$ to $15\%$.

Figure 3: The numerical design of a network with both affine and non-affine
tunability. a, panel 1: the original network with $z=5.28$, panel 2: achieving
non-affine tunability by removing the bonds important to $K$, panel 3:
realizing affine tunability by adding the red cross bonds. b, with the non-
affine operation of bond removal, $K$ reduces significantly while $G$ changes
more gently (note that $z$ axis decreases with this operation). Interestingly,
$K/G$ can decrease even below unity to reach the unusual situation of $K<G$
(see the two insets). c, the Poisson’s ratio $\nu$ is also broadly tunable
from positive to negative by this operation. d, with the affine operation of
cross bond addition, both $K$ and $G$ increase but $K/G$ remains roughly a
constant (note that $z$ axis increases with this operation). e, the Poisson’s
ratio $\nu$ also stays constant under this affine operation. f, operations
similar to a except that the bonds important to $G$ are now removed and then
the cross bonds are added. g, when the bonds important to $G$ are removed, $G$
reduces by two orders while $K$ remains relatively stable. Correspondingly,
$K/G$ increases significantly. h, $\nu$ does not change much with bond
removal. i and j, the affine operation of cross bond addition changes $K$ and
$G$ simultaneously, while $K/G$ and $\nu$ remain unchanged as expected.

Figure 4: Experimental realization of affine and non-affine tunability. a,
schematics of our experimental system. The bonds are identical springs
confined in acrylic tubes. The nodes are long rods for the bonds to hold on.
The system is designed as multi-layered to avoid conflicts between crossing
bonds, and vertically symmetric to eliminate net torques. b, top view image of
the actual system with all nodes ($N=50$) labeled in red. $z=5.42$ in this
system. c, side view image of the actual system. d, realizing the non-affine
property by removing the colored bonds which are critical to $E$ in the order
from 0 to 10, with more important ones removed earlier. e, the removal of
bonds decreases $E$ significantly but much less in $G$. Note that both $E$ and
$G$ are measured over two perpendicular directions and then averaged. f, the
bond removal achieves significant adjustment on $E/G$ and $\nu$. g, similar
operation of bond removal for the ones critical to $G$ only (shown in the
order from 0 to 10). h, $G$ reduces significantly while $E$ decreases gently.
i, the bond removal increases $E/G$ significantly but $\nu$ does not change
much. j, realizing the affine property by adding the colored cross bonds in
the order from 0 to 10. k, with bond addition both $E$ and $G$ increase at the
same rate. l, $E/G$ and $\nu$ stay unchanged, as expected for an affine
system.

Figure 5: Extending the 2D results into 3D. a, schematics showing the cross
bond (denoted in red) in 2D and 3D respectively. In 2D it goes across a bond
and in 3D it goes across a face. b, $G/G_{max}$ and $K/K_{max}$ merge together
above $z_{aff}=12.8$ in a $32\times 32\times 32$ nodes packing derived
network, confirming the affine transition in 3D. Inset shows their ratio
$K/G$. c, the non-affinity $\Delta E$ under both shear and compression strain
decreases with $z$ and stabilizes in the affine regime of $z\geq 12.8$
(circles and squares). The triangles show the locking of non-affinity by
adding cross bonds. Inset shows the tuning and locking of Poisson’s ratio
$\nu$. d-f, side view of the 3D printed $5\times 5\times 5$ nodes networks at
$z=7.696$, $z=9.312$, and $z=10.432$ respectively. They have identical node
positions but different bond numbers or z. From the top to the middle panel,
non-cross bonds are added which tunes the affinity $K/G$ and the Poisson’s
ratio $\nu$. From the middle to the bottom panel, cross bonds are added which
locks $K/G$ and $\nu$. g-i compare the internal strain fields of the three
networks under an identical strain: g and h exhibit different internal strain
due to the tuning operation, while h and i exhibit similar internal strain due
to the locking operation.


# Learn from Yesterday: A Semi-Supervised Continual Learning Method for
Supervision-Limited Text-to-SQL Task Streams

Yongrui Chen1, Xinnan Guo1, Tongtong Wu1,2, Guilin Qi1, Yang Li3, Yang Dong4

###### Abstract

Conventional text-to-SQL studies are limited to a single task with a fixed-
size training and test set. When confronted with a stream of tasks common in
real-world applications, existing methods struggle with the problems of
insufficient supervised data and high retraining costs. The former tends to
cause overfitting on unseen databases for the new task, while the latter makes
a full review of instances from past tasks impractical for the model,
resulting in forgetting of learned SQL structures and database schemas. To
address the problems, this paper proposes integrating semi-supervised learning
(SSL) and continual learning (CL) in a stream of text-to-SQL tasks and offers
two promising solutions in turn. The first solution Vanilla is to perform
self-training, augmenting the supervised training data with predicted pseudo-
labeled instances of the current task, while replacing the full volume
retraining with episodic memory replay to balance the training efficiency with
the performance of previous tasks. The improved solution SFNet takes advantage
of the intrinsic connection between CL and SSL. It uses in-memory past
information to help current SSL, while adding high-quality pseudo instances in
memory to improve future replay. The experiments on two datasets shows that
SFNet outperforms the widely-used SSL-only and CL-only baselines on multiple
metrics.

## Introduction

Relational databases (RDs) store a vast amount of today’s information and
provide the foundation for applications such as customer relationship
management (Anshari et al. 2019), financial markets (Lewis et al. 2017), and
medical records (Wang, Shi, and Reddy 2020). Text-to-SQL technology trains a
parser to translate natural language problems into machine-readable SQL
programs, providing an ideal way for non-technical users to easily interact
with their data stored in RDs. Current research on text-to-SQL has covered
single-table (Zhong, Xiong, and Socher 2017), multi-table (Yu et al. 2018),
and conversation (Yu et al. 2019a) scenarios, with a common assumption that
the size of training and testing data does not change over time.
Unfortunately, in real-world applications, new databases are always emerging
to adapt to changing circumstances (e.g., new diseases, adjusted financial
policies), thus continuously generating new tasks for the parser. Although
machine learning-based text-to-SQL methods have achieved state-of-the-art
(SOTA) performance, they suffer from the following two challenges in the face
of rapidly growing tasks.

Figure 1: Mutual promotion of SSL and CL. Sup denotes labeled data, Unsup
denotes unlabeled data, green cylindrical denotes the database against the
task.

1) Limited supervised data. For a new text-to-SQL task against unseen
databases, it is often impractical to annotate sufficient SQL labels for
training in a short period of time, resulting in parsers that are prone to
overfitting. 2) Costly full volume retraining. Considering a new task, an
intuitive idea is to train the model from scratch on all seen tasks.
Unfortunately, the computational cost of such retraining is unaffordable due
to the increasing size of pre-trained models, even with limited training data
(Wu et al. 2022). Assuming that there are $K$ tasks and the average training
time of one task is $T_{\text{avg}}$, the total time of full volume retraining
on these $K$ tasks is $T=K(K+1)T_{\text{avg}}/2$. When $K$ is large, the
squared relationship makes $T$ intolerable.

Existing works provide ideas to address these two challenges separately.
First, semi-supervised learning (SSL) (Guo et al. 2022) can compensate for the
lack of supervised data by mining the potential value of NLQs without SQL
labels. Second, continuous learning (CL) (Li, Qu, and Haffari 2021) provides
an alternative cost-effective training paradigm without using all instances of
previous tasks. However, both of them focuses on one of the challenges but
ignores the impact of the other. For text-to-SQL task streams, it remains a
pressing issue to address both challenges simultaneously.

In this paper, we propose to integrate SSL and CL to solve the supervision-
limited text-to-SQL task stream. We first give a Vanilla solution in which the
parser applies self-training to predict pseudo-labeled instances to improve
generalization to the current task (SSL) while replaying small portions of
past instances stored in memory to alleviate forgetting of previous tasks
(CL). Despite its simplicity, Vanilla has experimentally proven to be
sufficient to outperform most SSL-only and CL-only methods in text-to-SQL task
streams. Thereafter, we further hypothesize that SSL and CL can boost each
other. On the one hand, some instances in previous tasks can supply valuable
information to SSL for predicting pseudo-labels of unlabeled instances. As
shown in Figure 1(a), although instances $A$ and $B$ are associated with
different databases, Perpetrator and Race_track, respectively, the target SQL
of $A$ is similar to that of $B$. The parser might learn from $A$ on how to
predict the pseudo-label of $B$. On the other hand, high-quality pseudo-
labeled instances can also enrich the memory of past tasks. In the three tasks
shown in Figure 1(b), the pseudo instances can supplement the supervised-only
memories in terms of both RD schemas and SQL keywords. Motivated by this, we
propose the soft fusion network (SFNet), which applies a teacher-student
framework to separately cope with SSL and CL processes. Specifically, Teacher
is committed to the optimum on each single task via self-training, while
Student learns the pseudo labels predicted by Teacher on all seen tasks via
replaying to achieve the optimum on the entire task stream. To utilize the
mutual promotion of CL and SSL, SFNet performs dual sampling: when training
Teacher, past instances relevant to the current task are used to prompt the
SSL process; when training Student, both labeled and unlabeled instances of
previous tasks are sampled to guarantee complete memory for replay.
Comprehensive experiments on two text-to-SQL benchmarks show that our SFNet
further improves on the Vanilla solution, achieving SOTA results on multiple
metrics. In summary, the contributions of this paper include:

  * •

We propose a Vanilla solution that combine SSL and CL to solve the problem of
supervision-limited text-to-SQL task stream. To the best of our knowledge,
this is the first time that the two technologies have been integrated in text-
to-SQL.

  * •

Based on the Vanilla solution, we propose an improved solution SFNet that
applies a teacher-student framework to isolate the different optimization
goals of SSL and CL, and employs a dual sampling strategy to exploit their
mutual facilitation.

  * •

We covert two mainstream text-to-SQL datasets into the form of task streams
and conduct comprehensive experiments. Our methods outperform the existing CL-
only and SSL-only competitors and achieves SOTA performance on multiple
metrics.

## Preliminaries

### A Base Text-to-SQL Parser

Given an NLQ $q$ and a schema $\mathcal{S}=(\mathcal{C},\mathcal{T})$ for an
RD, conventional text-to-SQL aims to generate a SQL program $y$ by

$y=\mathcal{F}_{\theta}(q,\mathcal{S}),$ (1)

where $\mathcal{S}$ consists of columns
$\mathcal{C}=\\{c_{1},...,c_{|\mathcal{C}|}\\}$, tables
$\mathcal{T}=\\{t_{1},...t_{|\mathcal{T}|}\\}$, and $\mathcal{F}_{\theta}$
denotes a parser with parameters $\theta$. Most SOTA text-to-SQL parsers for
the single task (Guo et al. 2019; Wang et al. 2020a; Cao et al. 2021)
represent the desired $y$ as an abstract syntax tree (AST) $\mathcal{Z}_{y}$
via a context-free grammar and adopt a sequence-to-sequence architecture
(Luong, Pham, and Manning 2015) to synthesize $\mathcal{Z}_{y}$. To make a
fair comparison between different learning algorithms, we built a robust
parser $\mathcal{F}_{\theta}$ along their lines.

#### Encoder

An strong table pre-trained model Grappa (Yu et al. 2021) is employed to
encode $q$ and $\mathcal{S}$ into a sequence of contextual word
representations, $\mathbf{q}$ and $\mathbf{S}$. It was pre-trained on an
866.5k table-text corpus, being injected with structural properties common to
the semantic parsing of tables.

#### Decoder

A long short-term memory (LSTM) network is used as a decoder to synthesize
$\mathcal{Z}_{y}$ by generating a sequence of actions
$Z(y)=\\{z_{1},z_{2}...z_{|Z(y)|}\\}$. These actions determine the SQL
skeleton and the used schemas according to a top-down grammar, SemQL (Guo et
al. 2019). In particular, the probability of $\mathcal{Z}_{y}$ is estimated by

$P(\mathcal{Z}_{y}|\mathbf{q},\mathbf{S},\theta)={\prod_{j=1}^{|Z(y)|}P(z_{j}|\mathbf{q},\mathbf{S},z_{<j},\theta)},$
(2)

where $z_{j}$ is the $j$-th action. $P(z_{j}|q,\mathcal{S},z_{<j})$ denotes
the predicted probability of $z_{j}$, determined by the normalized inner
product of the hidden state ${\mathbf{h}}_{j}$ of the LSTM and the embeddings
of candidate actions.

### Problem Formulation

In our scenarios, $\mathcal{F}_{\theta}$ is trained continually by a sequence
of $K$ distinct tasks
$\\{\mathcal{D}^{1},\mathcal{D}^{2},...,\mathcal{D}^{K}\\}$. Each task
$\mathcal{D}^{i}$ consists of a labeled training set
$\mathcal{A}^{i}=\\{a_{1}^{i}...,a_{|\mathcal{A}^{i}|}^{i}\\}$, a validation
set $\mathcal{D}^{i}_{\text{valid}}$, and a test set
$\mathcal{D}^{i}_{\text{test}}$, where each training instance
$a_{k}^{i}=(q_{k}^{i},\mathcal{S}_{k}^{i},\tilde{y}_{k}^{i})$ has a gold SQL
program $\tilde{y}_{k}^{i}$ as the label. Considering the limited supervision,
we let $|\mathcal{A}^{i}|$ far smaller than the size of the general text-to-
SQL datasets. In addition, we also assume that there is an unlabeled set
$\mathcal{U}^{i}=\\{u_{1}^{i}...,u_{|\mathcal{U}^{i}|}^{i}\\}$ for
$\mathcal{D}^{i}$, where $u_{k}^{i}=(q_{k}^{i},\mathcal{S}_{k}^{i})$ does not
have gold SQL labels. Different tasks depend on different RDs, i.e., for
$\forall\mathcal{D}^{i},\forall\mathcal{D}^{j},$ if $i\neq j$, then
$S(\mathcal{D}^{i})\cap S(\mathcal{D}^{j})=\emptyset$, where
$S(\mathcal{D}^{i})$ denotes the set of corresponding RDs. Our final goal is
to make $\mathcal{F}_{\theta}$ achieve a good SQL generation accuracy on each
$\mathcal{D}^{i}_{\text{test}}$ after the training of all $K$ tasks.

## A Vanilla Solution

We start with a Vanilla solution to overcome the challenges of training
$\mathcal{F}_{\theta}$ on
$\\{\mathcal{D}^{1},\mathcal{D}^{2},...,\mathcal{D}^{K}\\}$, consisting of in-
task self-training and cross-task episodic memory replay. Figure 2 shows its
entire architecture.

### In-task Self-training

As a classical SSL method, self-training (ST) uses the model’s predictions to
product pseudo-labeled data to augment the limited supervision (the first
challenge). Inspired by this, during the learning of task $\mathcal{D}^{i}$,
we adopt a two-stage strategy to improve the generalization capability of
$\mathcal{F}_{\theta}$.

Figure 2: Architecture of our proposed Vanilla solution.

#### Warm Start

In the first stage, $\mathcal{F}_{\theta}^{i}$ is trained on labeled set
$\mathcal{A}^{i}$ for multiple epochs to fully understand $\mathcal{D}^{i}$,
thus guaranteeing the quality of the first batch of pseudo labels predicted by
$\mathcal{F}_{\theta}^{i}$. Specifically, for each labeled instance
$a_{k}^{i}=(q_{k}^{i},\mathcal{S}_{k}^{i},\tilde{y}_{k}^{i})\in\mathcal{A}^{i}$,
the loss is calculated by a log-likelihood,

$\mathcal{L}(a_{k}^{i};\theta^{i})=-{\sum_{j=1}^{|Z|}\log
P(\tilde{z}_{j}|\mathbf{q}_{k}^{i},\mathbf{S}_{k}^{i},z_{<j}^{i},\theta^{i})}.$
(3)

#### Self-updating

In the second stage, $\mathcal{F}_{\theta}^{i}$ is trained on
$\mathcal{A}^{i}\cup\mathcal{U}^{i}$. At each epoch, it first predicts a SQL
program $\hat{y}_{k}$ for each unlabeled instance
$u^{i}_{k}\in\mathcal{U}^{i}$ to a pseudo-labeled instance
$p^{i}_{k}=(q_{k}^{i},\mathcal{S}_{k}^{i},\hat{y}_{k}^{i})$. Random $k$
instances $p^{i}_{k}$ are selected to compose the pseudo-label set
$\mathcal{P}^{i}$. Subsequently, $\mathcal{F}_{\theta}$ is updated by
optimizing

$\mathcal{L}_{\text{ST}}(\mathcal{A}^{i}\cup\mathcal{P}^{i};\theta^{i})=\sum_{a^{i}_{k}\in\mathcal{A}^{i}}\mathcal{L}(a^{i}_{k};\theta^{i})+\sum_{p^{i}_{k}\in\mathcal{P}^{i}}\mu_{k}\mathcal{L}(p^{i}_{k};\theta^{i}),$
(4)

where $\mu_{k}=P(\mathcal{Z}_{y}|\mathbf{q},\mathbf{S},\theta)$ is the
confidence score to evaluate the contribution of each $p^{i}_{k}$ to the loss.

### Cross-task Episodic Memory Replay

Considering the second challenge, $\mathcal{F}_{\theta}$ cannot be retrained
with the full set of previous instances when a new task is encountered. This
also brings up a new problem of catastrophic forgetting (CF), i.e.,
$\mathcal{F}_{\theta}$ forgets the past tasks after learning the new ones.
Fortunately, episodic memory replay (EMR) can balance the efficiency and CF by
allowing the model to review a portion of the experienced instances. In
addition, compared with other gradient-based CL methods (Kirkpatrick et al.
2016; Lopez-Paz and Ranzato 2017), EMR is well suited for the text-to-SQL task
with complex labels $Z(y)$ because of its simple process. Therefore, we add
the following process along with ST.

#### Memory Construction

As a preparation, we construct a fixed-size memory
$\mathcal{M}^{i}=\\{m_{1}^{i},m_{2}^{i},...,m_{|\mathcal{M}^{i}|}^{i}\\}$
associated with each $\mathcal{D}^{i}$ to store a small number of replay
instances, where $m_{k}^{i}=(q_{k}^{i},\mathcal{S}_{k}^{i},\tilde{y}_{k}^{i})$
is sampled by from $\mathcal{A}^{i}$. This setup is practical for text-to-SQL
parsers because they have difficulty recalling information from the past
except for reusing the instances stored in memory.

#### Replay Loss

Concretely, whenever $\mathcal{F}_{\theta}^{i}$ performs self-training,
$\mathcal{L}_{\text{EMR}}$, the loss of all replayed instances stored in
$\mathcal{M}^{1}$ to $\mathcal{M}^{i-1}$, is added to
$\mathcal{L}_{\text{ST}}$,

$\mathcal{L}_{\text{EMR}}(\mathcal{M}^{j};\theta^{i})=\sum_{j=1}^{i-1}\sum_{m_{k}^{j}\in\mathcal{M}^{j}}{\mathcal{L}(m_{k}^{j};\theta^{i})}.$
(5)

Following the common practice, at the end of the training of
$\mathcal{D}^{i}$, we randomly select a $M$ labeled instances from
$\mathcal{A}^{i}$ and store them in $\mathcal{M}^{i}$ for replay in future
tasks.

## Soft Fusion Network

Figure 3 shows the architecture of our proposed SFNet, which is an improved
version of Vanilla. Since SSL is dedicated to the optimization of a single
(current) task, while CL is more concerned with the overall performance of all
tasks, SFNet applies a Teacher-Student Framework (TS) to perform them
separately. It composes of two base parsers, Teacher
$\mathcal{F}_{\text{tea}}$ for SSL and Student $\mathcal{F}_{\text{stu}}$ for
CL. During the task $D^{i}$, both $\mathcal{F}_{\text{tea}}^{i}$ and
$\mathcal{F}_{\text{stu}}^{i}$ are initialized from
$\mathcal{F}_{\text{stu}}^{i-1}$ but with separate parameter updates during
each task. To drive the mutual promotion of SSL and CL (Figure 1), SFNet uses
dual sampling that contains two different strategies to augment the training
data of $\mathcal{F}^{i}_{\text{tea}}$ and $\mathcal{F}^{i}_{\text{stu}}$,
respectively.

Figure 3: Architecture of our proposed SFNet.

### Teacher-Student Framework

####  Teacher Parser

The sole goal of $\mathcal{F}^{i}_{\text{tea}}$ is to offer correct pseudo
labels of the current task $\mathcal{D}^{i}$ via the in-task ST. Therefore, it
only focuses on $\mathcal{D}^{i}$ and is allowed to forget past tasks that is
not associated with $\mathcal{D}^{i}$. In contrast, previous instances
relevant to $\mathcal{D}^{i}$ can be highlighted in order to deepen
$\mathcal{F}^{i}_{\text{tea}}$’s understanding of $\mathcal{D}^{i}$. To
achieve this goal, during the ST, we refer to EMR to replay the appropriate
instances having potential prompts. In particular, when observing each task
$\mathcal{D}^{i}$, $N$ labeled instances relevant to $\mathcal{D}^{i}$ are
drawn from $\bigcup_{k=1}^{i-1}\mathcal{A}^{k}\cup\mathcal{P}^{k}$ to compose
the memory $\mathcal{M}_{\text{tea}}^{i}=\\{m_{1}^{i}...,m_{N}^{i}\\}$. We
call this step prompt sampling and detail it in the subsequent section.
Thereafter, $\mathcal{F}^{i}_{\text{tea}}$ follows Vanilla to perform in-task
ST except that it replaces $\mathcal{L}_{\text{ST}}$ in self-updating with the
following loss,

$\mathcal{L}_{\text{tea}}(\theta^{i}_{\text{tea}})=\mathcal{L}_{ST}(\mathcal{A}^{i}\cup\mathcal{P}^{i};\theta^{i}_{\text{tea}})+\sum_{m^{i}_{k}\in\mathcal{M}^{i}_{\text{tea}}}\mathcal{L}(m^{i}_{k};\theta_{\text{tea}}^{i}).$
(6)

When the training converges, $\mathcal{F}^{i}_{\text{tea}}$ is considered to
be approaching the optimum of $\mathcal{D}^{i}$. Note that the cost here is
that $\mathcal{F}^{i}_{\text{tea}}$ may forget some key information of
$\mathcal{D}^{1}...,\mathcal{D}^{i-1}$.

####  Student Parser

Assuming that trained $\mathcal{F}^{i}_{\text{tea}}$ is an expert proficient
in $\mathcal{D}^{i}$, then the task stream could theoretically provide $K$
experts for $\mathcal{D}^{1}...,\mathcal{D}^{K}$. Intuitively, if a parser can
inherit the capabilities of all these experts, then it will be overall optimal
of the entire task stream. Our $\mathcal{F}^{i}_{\text{stu}}$ aims to be such
a parser that for each task $\mathcal{D}^{j}$ ($1\leq j\leq i$) learns from
the pseudo-labeled instances $\mathcal{P}^{j}$ generated by the trained
$\mathcal{F}^{j}_{\text{tea}}$ in addition to the original $\mathcal{A}^{j}$.
Considering the training efficiency, our practice is that using cross-task EMR
during the learning of each $\mathcal{D}^{i}$. Concretely, the loss of
$\mathcal{F}^{i}_{\text{stu}}$ contains 1) the task loss on $\mathcal{D}^{i}$
and 2) the replay loss on $\mathcal{D}^{1}...,\mathcal{D}^{i-1}$,

$\displaystyle\mathcal{L}_{\text{stu}}(\theta^{i}_{\text{stu}})=\sum_{x^{i}_{k}\in{\mathcal{A}^{i}\cup\mathcal{P}^{i}}}\mathcal{L}(x^{i}_{k};\theta_{\text{stu}}^{i})+\mathcal{L}_{\text{EMR}}(\mathcal{M}^{j}_{\text{stu}};\theta_{\text{stu}}^{i}),$
(7)

where the replayed instances
$\mathcal{M}^{j}_{\text{stu}}=\\{m_{1}^{j}...,m_{M}^{j}\\}$ ($1\leq j\leq i$)
are sampled from $\mathcal{A}^{j}\cup\mathcal{P}^{j}$ via review sampling
(detailed in the next section). The sampled instances are diversified in both
the SQL skeleton and the RD schema so that $\mathcal{F}^{i}_{\text{stu}}$ can
recall complete information about past tasks.

### Dual Sampling

#### Prompt Sampling

x

Algorithm 1 details the process of prompt sampling (PS). First,
$\textsc{TopK}(\mathcal{R},\Omega,\tilde{N})$ denotes that selecting
$\tilde{N}$ instances $x_{j}$ from
$\mathcal{R}=\bigcup_{k=1}^{i-1}\mathcal{A}^{k}\cup\mathcal{P}^{k}$ to form a
temporary memory $\tilde{\mathcal{M}}_{\text{tea}}$ according to the relevance
score $\Omega=\\{\omega(x_{1}),...,\omega(x_{|\mathcal{R}|})\\}$,

$\omega(x_{j})=\max_{x_{l}\in\mathcal{U}^{i}}d_{\text{sch}}(x_{l},x_{j}),$ (8)

where
$d_{\text{sch}}(x_{l},x_{j})=\sqrt{\sum_{\psi\in\Psi}(\mathbf{v}_{\psi}^{l}-\mathbf{v}_{\psi}^{j})^{2}}$
denotes the distance of $x_{l}$ and $x_{j}$ on the schemas $\mathcal{S}^{k}$
and $\mathcal{S}^{j}$. Here $\Psi$ is the vocabulary of schema tokens and
$\mathbf{v}\in\mathbb{R}^{|\Psi|}$ is the hash vector of $\mathcal{S}$.
Specifically, if the token $\psi$ exists in $y$, then $\mathbf{v}_{\psi}$ is
1, otherwise it is 0. Thereafter, $\tilde{\mathcal{M}}_{\text{tea}}$ is
partitioned into $n$ clusters with structure distance $d_{\text{stru}}$ to
meet the diversity of SQL skeletons. $d_{\text{stru}}$ is formally similar to
$d_{\text{sch}}$, but replacing $\Psi$ with the SQL keywords vocabulary $\Phi$
(including GROUP BY, LIMIT, etc.). The reason for using hashing techniques for
$d_{\text{stru}}$ and $d_{\text{sch}}$ is to simplify the process. More
complex distance metrics are left for future work. Finally, the central
instance of each cluster is selected to compose $\mathcal{M}_{\text{tea}}$.
Note that although our PS requires re-traversing all past instances for each
new task, its time is only a fraction of the total time of SFNet in our
experiments.

#### Review Sampling

Review sampling (RS) is an advanced version of the random sampling in cross-
task EMR of Vanilla. The most significant difference between them is that RS
samples pseudo-labeled instances $\mathcal{P}^{i}$ in addition to
$\mathcal{A}^{i}$, where the labels of each $p\in\mathcal{P}^{i}$ is predicted
by $\mathcal{F}^{i}_{\text{tea}}$ via the in-task ST. In this way, the
resulting memory is augmented, allowing $\mathcal{F}^{i}_{\text{stu}}$ to
recall the task $\mathcal{D}^{i}$ more fully. Naturally, we would like sampled
instances to be representative of $\mathcal{D}^{i}$ in terms of both SQL
skeletons and RD schemas. Thus, we define a combined distance
$d(x_{1},x_{2})=d_{\text{stru}}*d_{\text{sch}}$ and use it to partition
$\mathcal{A}^{i}\cup\mathcal{P}^{i}$ into $M$ clusters. Here we use the
product in order to balance the weights of $d_{\text{stru}}$ and
$d_{\text{sch}}$. Consistent with PS, the result
$\mathcal{M}^{j}_{\text{stu}}$ consists of all central instances of the
cluster, which can be considered as the representative of $\mathcal{D}^{i}$.

Algorithm 1 Prompt Sampling

0: Past labeled instances set
$\mathcal{R}=\bigcup_{k=1}^{i-1}\mathcal{A}^{k}\cup\mathcal{P}^{k}$, current
unlabeled set $\mathcal{U}^{i}$, score
$\mathbf{\Omega}=\\{\omega(x_{1}),...,\omega(x_{|\mathcal{R}|})\\}$

1:
$\tilde{\mathcal{M}}_{\text{tea}}\leftarrow\textsc{TopK}(\mathcal{R},\Omega,\tilde{N})$,
$\mathcal{M}_{\text{tea}}\leftarrow\emptyset$

2: Partition $\tilde{\mathcal{M}}_{\text{tea}}$ into $N$ clusters, denoted as
$C$, with the SQL structure distance metric $d_{\text{stru}}$

3: for $c_{j}\in C$ do

4:
$\mathcal{M}_{\text{tea}}\leftarrow\mathcal{M}_{\text{tea}}\cup\\{(q^{*},\mathcal{S}^{*},y^{*})\\}$,
where $(q^{*},\mathcal{S}^{*},y^{*})$ is closest to the center of cluster
$c_{j}$

5: end for

6: return $\mathcal{M}_{\text{tea}}$

$1$$2$$3$$4$$5$$6$$7$$8$$9$$10$$200$$400$$600$$800$$1{,}000$Seconds

$1$$2$$3$$4$$5$$6$$7$$8$$9$$10$$0$$500$$1{,}000$Seconds$|\mathcal{A}^{i}|$$|\mathcal{U}^{i}|$$|\mathcal{D}^{i}_{\text{test}}|$ZS
number

Figure 4: Statistics of each task in WikiSQL and Spider.

## Experiments

### Experimental Setup

#### Datasets

To evaluate our proposed methods, we construct two task streams using the
following two text-to-SQL datasets:
WikiSQL111https://github.com/salesforce/WikiSQL (Zhong, Xiong, and Socher
2017) contains more than 20k tables collected from Wikipedia and 80,654 NLQ-
SQL instances. Each instance corresponds to only a single table and the
structure of the target SQL is relatively simple. Spider222https://yale-
lily.github.io//spider (Yu et al. 2018) contains 8,659 training instances
across 146 RDs in total, and covers a wide range of domains, including
flights, geography, movies, and more. Unlike WikiSQL, it instance corresponds
to an RD containing multiple tables, and the target SQL may have a complex
syntax.

Table 1: Experimental results for comparison with baselines.

Method | Spider | WikiSQL
---|---|---
$\text{ACC}_{\text{a}}$ | $\text{ACC}_{\text{w}}$ | BWT | FWT | $\text{ACC}_{\text{a}}$ | $\text{ACC}_{\text{w}}$ | BWT | FWT
Fine-tune | 47.5 | 45.5 | -11.3 | 38.4 | 69.8 | 69.2 | -1.5 | 63.0
Self-training (Goldwasser et al. 2011) | 48.3 | 46.6 | -10.9 | 40.4 | 70.4 | 69.9 | -3.3 | 63.5
SETNet (Wang et al. 2020b) | 47.8 | 46.1 | -14.6 | 41.6 | 70.7 | 70.2 | -2.1 | 61.8
MST-SQL (Guo et al. 2022) | 49.6 | 47.3 | -6.6 | 40.7 | 70.7 | 70.1 | -1.7 | 61.7
EWC (Kirkpatrick et al. 2016) | 48.3 | 47.2 | -7.9 | 38.4 | 70.0 | 69.6 | -2.0 | 61.4
HAT (Serrà et al. 2018) | 49.4 | 47.7 | -8.4 | 39.3 | 70.0 | 69.6 | -1.4 | 61.8
EMR (Wang et al. 2019) | 50.1 | 49.1 | -3.2 | 40.3 | 71.1 | 70.7 | -2.2 | 63.1
EMAR (Han et al. 2020) | 50.3 | 49.6 | -4.6 | 40.4 | 70.8 | 70.5 | -1.5 | 62.7
APPER (Mi et al. 2020) | 50.7 | 49 | -7.7 | 40.0 | 70.2 | 69.9 | -3.0 | 62.7
Total-Recall (Li, Qu, and Haffari 2021) | 53.4 | 51.6 | -5.1 | 40.3 | 71.5 | 71.1 | -2.1 | 62.7
Vanilla | 53.9 | 52.9 | -4.0 | 40.6 | 72.2 | 71.9 | -2.0 | 64.0
SFNet | 56.0 | 53.6 | -1.0 | 45.9 | 73.6 | 73.3 | -2.3 | 65.6
Oracle (all tasks w/o Unsup.) | 62.9 | 63.4 | 5.2 | 48.7 | 73.1 | 72.7 | 2.6 | 64.2

Following the problem formulation before, we divided each dataset into 10
tasks based on the domain of RD, and set the size of $\mathcal{A}^{i}$ for
most tasks to less than 500. To provide a stable initialization for all
compared baselines, we choose the task with the largest $\mathcal{A}^{i}$ as
the first task. Refer to the practice in SSL (Guo et al. 2022), we let most
$\mathcal{U}^{i}$ be larger than $\mathcal{A}^{i}$. In addition, (Yu et al.
2018) has pointed out that a good text-to-SQL method must have the ability to
resolve zero-shot RDs. Thus, we guarantee that $\mathcal{D}^{i}_{\text{test}}$
has RDs or tables that are not seen in $\mathcal{A}^{i}$. Detailed statistics
of our split datasets are illustrated in Figure 4.

#### Evaluation Metrics

Following previous works (Wang et al. 2019; Li, Qu, and Haffari 2021), we
adopt four metrics to evaluate the performance of the methods: 1)
$\text{ACC}_{\text{a}}=\frac{1}{K}\sum_{i=1}^{K}acc_{i,K}$; 2)
$\text{ACC}_{\text{w}}=acc_{\mathcal{D}_{\text{test}}^{(1:K)}}$; 3) $k<i$:
$\text{BWT}=\frac{1}{K-1}\sum_{i=1}^{K-1}acc_{i,K}-acc_{i,i}$; 4)
$\text{FWT}=\frac{1}{K-1}\sum_{i=2}^{K}acc_{i,i-1}-\bar{b}_{i}$, where
$acc_{i,j}$ denotes the test accuracy on $\mathcal{D}^{i}_{\text{test}}$ after
the training of $\mathcal{D}^{j}$ and $\bar{b}_{i}$ denotes the test accuracy
on $\mathcal{D}^{i}_{\text{test}}$ at random initialization. The first two
metrics mainly measure the comprehensive performance of parsers. The BWT
measures forgetting of past tasks, while the FWT measures zero-shot
performance on new tasks.

#### Implementation Details

Our method ran on one Tesla A100 Super GPUs. We use pre-trained Grappa-Large
as the encoder and K-medoids as the clustering algorithm by default. The
hyper-parameters were set as follow: (1) The maximum sizes of memory
$\mathcal{M}_{\text{tea}}^{i}$ and $\mathcal{M}_{\text{stu}}^{i}$ were set to
30% of $|\mathcal{D}^{i}_{\text{train}}|$. (2) The learning rate is set to
$2\times 10^{-5}$ for Grappa and $4\times 10^{-4}$ for other modules. All our
datasets and codes are publicly available333https://github.com/Bahuia/SSCL-
Text2SQL.

#### Baselines

Fine-tune is an naïve baseline that uses only labeled data to fine-tune the
model for the new task based on the previous model. Self-training (Goldwasser
et al. 2011), SETNet (Wang et al. 2020b), and MST-SQL (Guo et al. 2022)
composes the SSL-only baselines that also fine-tune the model based on the
previous model but utilize both unlabeled and labeled data, where SETNet and
MST-SQL apply mean-teacher and meta-learning to improve the generalization
capability, respectively. The CL-only baselines consists of EWC (Kirkpatrick
et al. 2016), HAT (Serrà et al. 2018), EMR (Wang et al. 2019), EMAR (Han et
al. 2020), APPER (Mi et al. 2020), and Total-Recall (Li, Qu, and Haffari
2021). EWC uses regularization to constrain the learning of the current task.
HAT applies a task-specific mask to guide each task. EMAR improves EMR with
the constructed prototypes to avoid overfitting to memory. ARPER adds EWC
regularization to the EMR loss and designs a priority-based sampling method.
Total-Recall performs memory replay with a sampling method that balances the
distribution of parsed actions and trains the semantic parser with a two-stage
update strategy. Here we did not compare with GEM (Lopez-Paz and Ranzato 2017)
because its desired GPU memory is too large to run on our device. Finally, we
set an approximate upper boundary Oracle that uses the full volume retraining,
i.e., for each task $\mathcal{D}^{i}$, the parser is trained by the combined
data of all $\mathcal{A}^{j}$ ($1\leq j\leq i$).

### Overall Results

The experimental results are shown in Table 1. The gap between Fine-tune and
Oracle on WikiSQL are not as large as on Spider because the multiple tables
and complex SQL syntax makes Spider more challenging. Despite the simplicity
of the Vanilla process, it outperforms all the baselines in terms of
$\text{ACC}_{\text{a}}$ and $\text{ACC}_{\text{w}}$. More excitingly, our
proposed SFNet further improves its $\text{ACC}_{\text{a}}$ by 1.4% (WikiSQL)
and 2.1% (Spider), and achieves SOTA performance in almost all metrics on two
datasets. Although SFNet does not perform best on WikiSQL in terms of BWT, it
still shows the competitiveness with other methods and brings significant
overall improvements on strong Total Recall by 2.1% in
$\text{ACC}_{\text{a}}$.

12345678910$45$$50$$55$$60$$65$$\text{ACC}_{\text{a}}$

12345678910$40$$45$$50$$55$$60$$65$$\text{ACC}_{\text{w}}$

$2$$3$$4$$5$$6$$7$$8$$9$$10$$-25$$-20$$-15$$-10$$-5$$0$BWT

123456789$30$$35$$40$$45$$50$FWTFine-tuneSelf-trainingSETNetMST-
SQLEWCHATEMREMARAPPERTotal-RecallVanillaSFNet

Figure 5: $\text{ACC}_{\text{a}}$, $\text{ACC}_{\text{w}}$, BWT, FWT till the
seen tasks on Spider after learning on each task sequentially.

The SSL-only methods improve the overall performance of Fine-tune using the
information provided by unsupervised data, while they perform poorly in
fighting against the CF (-10.9% and -14.6%). MST-SQL achieves better results,
probably because of its meta-learning that captures common features across
tasks. The replay-based EMR, EMAR, and Total Recall perform better than other
CL-only methods on BWT, which proves that past instances may be more important
to overcome the CF of the text-to-SQL parser. Although Total Recall benefits
from its fine-grained semantic parsing sampling algorithm to achieve a best
results of the baselines, its results are still limited by ignoring
unsupervised information. Unlike them, our SFNet deeply integrates SSL and CL
and leads them to reinforce each other, thus achieving overall excellent
results.

Table 2: Ablation study results of SFNet

Setting | Spider | WikiSQL
---|---|---
A | W | B | F | A | W | B | F
SFNet | 56.0 | 54.1 | -1.0 | 45.9 | 73.6 | 73.3 | -2.3 | 65.6
w/o $\mathcal{F}^{i}_{\text{tea}}$ | 54.5 | 52.5 | -4.0 | 42.9 | 72.0 | 71.6 | -1.3 | 63.7
w/o $\mathcal{F}^{i}_{\text{stu}}$ | 50.4 | 48.2 | -7.2 | 45.1 | 71.6 | 72.7 | -2.0 | 63.5
Combine $\mathcal{F}^{i}_{\text{tea}}$ & $\mathcal{F}^{i}_{\text{stu}}$ | 54.8 | 53.0 | -3.0 | 43.6 | 71.7 | 71.3 | -2.0 | 62.9
w/o PS | 53.7 | 52.9 | -1.1 | 41.8 | 70.4 | 70.0 | -4.6 | 62.6
RS only using $\mathcal{A}^{i}$ | 54.2 | 51.4 | -0.9 | 43.1 | 71.4 | 71.0 | -2.8 | 62.7

### Detailed Results and Analysis

#### Results till the Seen Tasks

Figure 5 shows the results of four metrics till the seen tasks on Spider after
learning one each task. We can see that our proposed SFNet (blue) is always
more stable than the other baselines in all metrics and this stability becomes
more pronounced as the number of tasks grows. In terms of BWT and FWT, the
improvement brought by SFNet is more significant, which proves the
effectiveness of our fusion manner to SSL and CL, not only alleviating
forgetfulness of past tasks, but also improving the generalization capability
for zero-shot RDs. Notice that the performance of almost all methods in BWT
improves slightly as the number of tasks increases. This may be related to the
combined generalization of text-to-SQL tasks, i.e., the parser might resolve
the few instances that were previously incorrectly predicted with the SQL
fragments learned in the new task. Building on this foundation, our SFNet
further exacerbates this trend using PS.

#### Ablation Test

Table 3: $\text{ACC}_{\text{a}}$ (A), $\text{ACC}_{\text{w}}$ (W), BWT (B),
and FWT (F) of SFNet with different sampling strategies.

Methods | Spider | WikiSQL
---|---|---
A | W | B | F | A | W | B | F
Random | 53.2 | 52.7 | -2.2 | 41.3 | 72.2 | 71.8 | -3.0 | 62.9
Schema Sim. | 54.3 | 53.2 | -1.0 | 45.0 | 72.8 | 72.4 | -2.0 | 63.9
Random | 52.3 | 51.4 | -3.3 | 41.5 | 71.5 | 71.1 | -1.8 | 64.6
FSS | 52.6 | 51.9 | -2.0 | 42.0 | 72.3 | 72.1 | -2.0 | 64.7
Prior | 52.6 | 51.1 | -4.0 | 42.0 | 71.9 | 71.3 | -1.5 | 64.1
Balance | 52.0 | 49.6 | -2.4 | 41.1 | 71.9 | 71.4 | -2.8 | 64.3
LFS | 52.4 | 51.2 | -6.3 | 40.4 | 72.1 | 71.6 | -2.2 | 63.5
DLFS | 52.8 | 51.9 | -1.5 | 40.5 | 72.2 | 71.9 | -2.7 | 64.7
Schema Clus. | 54.9 | 53.3 | -1.1 | 43.8 | 72.3 | 71.8 | -2.1 | 64.1
Dual Sampling | 55.7 | 54.1 | -1.0 | 45.9 | 73.6 | 73.3 | -2.3 | 65.6

To explore the contributions of each component of our proposed SFNet, we
compared the performance of the following settings:

  * •

w/o $\mathcal{F}^{i}_{\text{tea}}$ We remove $\mathcal{F}^{i}_{\text{tea}}$
and in-task ST to verify the contribution of SSL;

  * •

w/o $\mathcal{F}^{i}_{\text{stu}}$ We remove $\mathcal{F}_{\text{stu}}$ and
cross-task EMR, and only use $\mathcal{F}_{\text{tea}}$ to predict, to
evaluate the contribution of CL;

  * •

Combine $\mathcal{F}^{i}_{\text{tea}}$ & $\mathcal{F}^{i}_{\text{stu}}$ We
merged $\mathcal{F}_{\text{tea}}$ and $\mathcal{F}_{\text{stu}}$ into a single
parser to assess the necessity of the TS framework.

  * •

w/o PS We remove the prompt sampling to evaluate the improvements brought to
SSL by past information.

  * •

RS only using $\mathcal{A}^{i}$; We review the sampling using only the labeled
set $\mathcal{A}^{i}$ to assess the improvement of CL by pseudo-supervision
information.

Table 2 shows the $\text{ACC}_{\text{a}}$ (A), $\text{ACC}_{\text{w}}$ (W),
BWT (B), and FWT (F) of different settings. Our SFNet equipped with all
modules performs best in terms of $\text{ACC}_{\text{a}}$ and FWT. Its non-
negligible improvements in overall performance allow us to overlook the minor
shortcomings in BWT. By removing $\mathcal{F}^{i}_{\text{tea}}$,
$\text{ACC}_{\text{a}}$ decreases by 1.6% on WikiSQL and 1.5% for Spider,
which proves that the value of the unsupervised information. Discarding
$\mathcal{F}^{i}_{\text{stu}}$ leads to a decrease in terms of
$\text{ACC}_{\text{a}}$ (-2.0% & -5.4%) because of the forgetting of previous
tasks. The performance drop (-1.9% & -1.2%) brought by abandoning the TS
framework demonstrates the necessity of handling SSL and CL separately. On
both datasets, removing PS results in an absolute decrease in all metrics.
This reveals that the relevant information from past tasks can prompt the SSL
process of current task. In particular, the drop in FWT on Spider (-4.1%) is
more significant, probably due to the fact that Spider can provide information
not only on the RD schema but also on the SQL structure. The degradation in
$\text{ACC}_{\text{a}}$ aspect when using only $\mathcal{A}^{i}$ and not
$\mathcal{P}^{i}$ in the RS process reflects that the SSL results can also
contribute to the CL process.

WikiSQLSpider$-500$$0$$500$$1{,}000$$1{,}500$$2{,}000$SecondsFine-tuneSelf-
trainingSETNetMST-SQLEWCHATEMREMARAPPERTotall-RecallVanillaSFNetOraclePS
Figure 6: Average training time of each method on one task.

#### Impact of Sampling Methods

To further evaluate our proposed PS and RS, we replace them with some other
existing sampling strategies. For prompt sampling, we replace it Random and
Schema Sim., where the former randomly samples training instances and the
latter only selects the top-$k$ instances by schema similarity $\omega(x)$
without using $d_{\text{stru}}$ to clustering. For review sampling, we replace
it with the sampling strategies widely-used in CL, including FSS (Wang et al.
2019; Aljundi et al. 2019), LFS (Li, Qu, and Haffari 2021), Prior (Mi et al.
2020), and Balance (Jian, Yi, and Zhang 2022), and our defined Schema Clus.,
which performs clustering only uses $d_{\text{sch}}$. Here, Schema Sim. and
Schema Clus. are used to evaluate the contribution of the diverse SQL
structures to the performance. From Table 3 we can see that our dual sampling
outperforms all other baselines on both datasets in terms of
$\text{ACC}_{\text{a}}$ and $\text{ACC}_{\text{w}}$. More importantly, the
improvements over Schema Sim. and Schema Clus. in terms of FWT reveals the
fact that for text-to-SQL, diverse SQL structures in the training data are
useful for the parser to generate unseen SQL programs.

#### Training Time Analysis

The average training times of different methods on each task are depicted in
Figure 6. EMR and EMAR are slower than other CL-only methods like HAT and EWC
because they need to replay the past instances during the training. Similarly,
SSL-only methods predicts pseudo-labels on unsupervised data to augment the
training data, and thus also require longer training time. In addition, the
training process of MST-SQL is exceptionally time-consuming because it
requires the construction of multiple sets of meta-learning tasks with
possible data duplication. Notably, our SFNet contains the training process
for both models and utilizes both unsupervised and replayed instances, while
it takes only half of the time used by Oracle that does not uses unsupervised
data. Moreover, although our PS accesses the full amount of past data each
time, the fast hashing strategy makes it take only about 1/6 of the total time
SFNet uses.

## Related Work

Text-to-SQL Research on text-to-SQL can be roughly divided into three
directions. The first one is the single-table task represented by WikiSQL
(Zhong, Xiong, and Socher 2017), whose target SQL programs contain only simple
syntaxes. The SOTA methods (Hwang et al. 2019; Chen et al. 2021a; Xu et al.
2022) of WikiSQL typically treat the problem as several subtasks and leverage
multi-task learning to solve them. The second direction is a cross-domain
multi-table scenario represented by Spider (Yu et al. 2018). Its target
programs cover a variety of complicated SQL syntaxes including GROUP BY and
nested queries, to better meet the needs of real-world applications. In this
scenario, most SOTA parsers (Guo et al. 2019; Wang et al. 2020a; Cao et al.
2021) apply top-down grammar-based decoding, which is consistent with our
work. The last one is a conversational task, such as SParC (Yu et al. 2019b)
and CoSQL (Yu et al. 2019a), forcing the parser to learn to consider the
context information when generating SQL in a multi-turn dialogue (Zhang et al.
2019; Zheng et al. 2022).

SSL in Semantic Parsing Multiple classical methods have been applied to
address the challenge of lack of annotation for semantic parsing, such as SVM
(Kate and Mooney 2007), self-training (Goldwasser et al. 2011), dual learning
(Chen et al. 2021b), auto-encoder (Yin et al. 2018), and mean-teacher (Wang et
al. 2020b). We finally adopted self-training in the proposed solutions for
simplicity. Unlike previous methods, we propose relevance sampling to utilize
the continual scenario to boost the performance of SSL.

CL in Semantic Parsing There is relatively little work that applies CL to
semantic parsing. (Lialin et al. 2020) and (Li, Qu, and Haffari 2021) apply
EWC (Kirkpatrick et al. 2016) and EMR (Wang et al. 2019), respectively, to
handle the task stream of traditional semantic parsing benchmarks. Different
from them, our method focuses on text-to-SQL with larger application scenarios
and further argument the memories in the continual process using unsupervised
data.

Semi-supervised Continual Learning Recent SSCL methods (Brahma, Verma, and Rai
2021; Luo et al. 2022; Smith et al. 2021; Wang et al. 2021) have focused on
image classification tasks. They are either based on backbone models widely
used in Computer Vision (e.g., generative adversarial networks) or on gradient
prediction which is computationally expensive and therefore not suitable for
Natural Language Processing tasks with large-scale pre-trained models. In
contrast, our research is dedicated to obtaining a simple and efficient method
for text-to-SQL within the allowed time-space overhead.

## Conclusion

In this paper, we presented two methods that integrates semi-supervised
learning (SSL) and continual learning (CL) to address the problem of
supervision-limited text-to-SQL task stream. The first approach combines self-
training and episodic memory replay to enhance supervision while balancing
training efficiency and performance of the overall task stream. The improved
method SFNet drives the intrinsic connection between CL and SSL by using in-
memory past information to help current SSL, while adding high-quality pseudo
instances in memory to improve future replay. The experiments on two
benchmarks shows that indicate that our method provide a promising way for
supervision-limited text-to-SQL task stream. In future work, we will try to
introduce prompt learning into the continual process to augment supervision
and alleviate forgetting.

## References

  * Aljundi et al. (2019) Aljundi, R.; Lin, M.; Goujaud, B.; and Bengio, Y. 2019.  Gradient based sample selection for online continual learning.  In Wallach, H. M.; Larochelle, H.; Beygelzimer, A.; d’Alché-Buc, F.; Fox, E. B.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , 11816–11825.
  * Anshari et al. (2019) Anshari, M.; Almunawar, M. N.; Lim, S. A.; and Al-Mudimigh, A. 2019.  Customer relationship management and big data enabled: Personalization & customization of services.  _Applied Computing and Informatics_ , 15(2): 94–101.
  * Brahma, Verma, and Rai (2021) Brahma, D.; Verma, V. K.; and Rai, P. 2021.  Hypernetworks for Continual Semi-Supervised Learning.  _CoRR_ , abs/2110.01856.
  * Cao et al. (2021) Cao, R.; Chen, L.; Chen, Z.; Zhao, Y.; Zhu, S.; and Yu, K. 2021.  LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations.  In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , 2541–2555. Association for Computational Linguistics.
  * Chen et al. (2021a) Chen, Y.; Guo, X.; Wang, C.; Qiu, J.; Qi, G.; Wang, M.; and Li, H. 2021a.  Leveraging Table Content for Zero-shot Text-to-SQL with Meta-Learning.  In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_ , 3992–4000. AAAI Press.
  * Chen et al. (2021b) Chen, Z.; Chen, L.; Li, H.; Cao, R.; Ma, D.; Wu, M.; and Yu, K. 2021b.  Decoupled Dialogue Modeling and Semantic Parsing for Multi-Turn Text-to-SQL.  In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , 3063–3074. Association for Computational Linguistics.
  * Goldwasser et al. (2011) Goldwasser, D.; Reichart, R.; Clarke, J.; and Roth, D. 2011.  Confidence Driven Unsupervised Semantic Parsing.  In Lin, D.; Matsumoto, Y.; and Mihalcea, R., eds., _The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Proceedings of the Conference, 19-24 June, 2011, Portland, Oregon, USA_ , 1486–1495. The Association for Computer Linguistics.
  * Guo et al. (2019) Guo, J.; Zhan, Z.; Gao, Y.; Xiao, Y.; Lou, J.; Liu, T.; and Zhang, D. 2019.  Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation.  In Korhonen, A.; Traum, D. R.; and Màrquez, L., eds., _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , 4524–4535. Association for Computational Linguistics.
  * Guo et al. (2022) Guo, X.; Chen, Y.; Qi, G.; Wu, T.; and Xu, H. 2022.  Improving Few-Shot Text-to-SQL with Meta Self-Training via Column Specificity.  In Raedt, L. D., ed., _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022_ , 4150–4156. ijcai.org.
  * Han et al. (2020) Han, X.; Dai, Y.; Gao, T.; Lin, Y.; Liu, Z.; Li, P.; Sun, M.; and Zhou, J. 2020\.  Continual Relation Learning via Episodic Memory Activation and Reconsolidation.  In Jurafsky, D.; Chai, J.; Schluter, N.; and Tetreault, J. R., eds., _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , 6429–6440. Association for Computational Linguistics.
  * Hwang et al. (2019) Hwang, W.; Yim, J.; Park, S.; and Seo, M. 2019.  A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization.  _CoRR_ , abs/1902.01069.
  * Jian, Yi, and Zhang (2022) Jian, Y.; Yi, J.; and Zhang, L. 2022.  Adaptive Feature Generation for Online Continual Learning from Imbalanced Data.  In Gama, J.; Li, T.; Yu, Y.; Chen, E.; Zheng, Y.; and Teng, F., eds., _Advances in Knowledge Discovery and Data Mining - 26th Pacific-Asia Conference, PAKDD 2022, Chengdu, China, May 16-19, 2022, Proceedings, Part I_ , volume 13280 of _Lecture Notes in Computer Science_ , 276–289. Springer.
  * Kate and Mooney (2007) Kate, R. J.; and Mooney, R. J. 2007.  Semi-Supervised Learning for Semantic Parsing using Support Vector Machines.  In Sidner, C. L.; Schultz, T.; Stone, M.; and Zhai, C., eds., _Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, April 22-27, 2007, Rochester, New York, USA_ , 81–84. The Association for Computational Linguistics.
  * Kirkpatrick et al. (2016) Kirkpatrick, J.; Pascanu, R.; Rabinowitz, N. C.; Veness, J.; Desjardins, G.; Rusu, A. A.; Milan, K.; Quan, J.; Ramalho, T.; Grabska-Barwinska, A.; Hassabis, D.; Clopath, C.; Kumaran, D.; and Hadsell, R. 2016.  Overcoming catastrophic forgetting in neural networks.  _CoRR_ , abs/1612.00796.
  * Lewis et al. (2017) Lewis, R.; McPartland, J.; Ranjan, R.; et al. 2017.  Blockchain and financial market innovation.  _Economic Perspectives_ , 41(7): 1–17.
  * Li, Qu, and Haffari (2021) Li, Z.; Qu, L.; and Haffari, G. 2021.  Total Recall: a Customized Continual Learning Method for Neural Semantic Parsers.  In Moens, M.; Huang, X.; Specia, L.; and Yih, S. W., eds., _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , 3816–3831. Association for Computational Linguistics.
  * Lialin et al. (2020) Lialin, V.; Goel, R.; Simanovsky, A.; Rumshisky, A.; and Shah, R. 2020.  Update frequently, update fast: Retraining semantic parsing systems in a fraction of time.  _arXiv preprint arXiv:2010.07865_.
  * Lopez-Paz and Ranzato (2017) Lopez-Paz, D.; and Ranzato, M. 2017.  Gradient Episodic Memory for Continual Learning.  In Guyon, I.; von Luxburg, U.; Bengio, S.; Wallach, H. M.; Fergus, R.; Vishwanathan, S. V. N.; and Garnett, R., eds., _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 6467–6476.
  * Luo et al. (2022) Luo, Y.; Wong, Y.; Kankanhalli, M. S.; and Zhao, Q. 2022.  Learning to Predict Gradients for Semi-Supervised Continual Learning.  _CoRR_ , abs/2201.09196.
  * Luong, Pham, and Manning (2015) Luong, M.-T.; Pham, H.; and Manning, C. D. 2015.  Effective Approaches to Attention-based Neural Machine Translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , 1412–1421.
  * Mi et al. (2020) Mi, F.; Chen, L.; Zhao, M.; Huang, M.; and Faltings, B. 2020.  Continual Learning for Natural Language Generation in Task-oriented Dialog Systems.  In Cohn, T.; He, Y.; and Liu, Y., eds., _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , 3461–3474. Association for Computational Linguistics.
  * Serrà et al. (2018) Serrà, J.; Suris, D.; Miron, M.; and Karatzoglou, A. 2018.  Overcoming Catastrophic Forgetting with Hard Attention to the Task.  In Dy, J. G.; and Krause, A., eds., _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , 4555–4564. PMLR.
  * Smith et al. (2021) Smith, J.; Balloch, J.; Hsu, Y.; and Kira, Z. 2021.  Memory-Efficient Semi-Supervised Continual Learning: The World is its Own Replay Buffer.  In _International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021_ , 1–8. IEEE.
  * Wang et al. (2020a) Wang, B.; Shin, R.; Liu, X.; Polozov, O.; and Richardson, M. 2020a.  RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers.  In Jurafsky, D.; Chai, J.; Schluter, N.; and Tetreault, J. R., eds., _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , 7567–7578. Association for Computational Linguistics.
  * Wang et al. (2019) Wang, H.; Xiong, W.; Yu, M.; Guo, X.; Chang, S.; and Wang, W. Y. 2019.  Sentence Embedding Alignment for Lifelong Relation Extraction.  In Burstein, J.; Doran, C.; and Solorio, T., eds., _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , 796–806. Association for Computational Linguistics.
  * Wang et al. (2021) Wang, L.; Yang, K.; Li, C.; Hong, L.; Li, Z.; and Zhu, J. 2021.  ORDisCo: Effective and Efficient Usage of Incremental Unlabeled Data for Semi-Supervised Continual Learning.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021_ , 5383–5392. Computer Vision Foundation / IEEE.
  * Wang, Shi, and Reddy (2020) Wang, P.; Shi, T.; and Reddy, C. K. 2020.  Text-to-SQL generation for question answering on electronic medical records.  In _Proceedings of The Web Conference 2020_ , 350–361.
  * Wang et al. (2020b) Wang, X.; Sun, H.; Qi, Q.; and Wang, J. 2020b.  SETNet: A Novel Semi-Supervised Approach for Semantic Parsing.  In Giacomo, G. D.; Catalá, A.; Dilkina, B.; Milano, M.; Barro, S.; Bugarín, A.; and Lang, J., eds., _ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020)_ , volume 325 of _Frontiers in Artificial Intelligence and Applications_ , 2236–2243. IOS Press.
  * Wu et al. (2022) Wu, T.; Caccia, M.; Li, Z.; Li, Y.-F.; Qi, G.; and Haffari, G. 2022.  Pretrained Language Model in Continual Learning: A Comparative Study.  In _International Conference on Learning Representations_.
  * Xu et al. (2022) Xu, K.; Wang, Y.; Wang, Y.; Wang, Z.; Wen, Z.; and Dong, Y. 2022.  SeaD: End-to-end Text-to-SQL Generation with Schema-aware Denoising.  In Carpuat, M.; de Marneffe, M.; and Ruíz, I. V. M., eds., _Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022_ , 1845–1853. Association for Computational Linguistics.
  * Yin et al. (2018) Yin, P.; Zhou, C.; He, J.; and Neubig, G. 2018.  StructVAE: Tree-structured Latent Variable Models for Semi-supervised Semantic Parsing.  In Gurevych, I.; and Miyao, Y., eds., _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , 754–765. Association for Computational Linguistics.
  * Yu et al. (2021) Yu, T.; Wu, C.; Lin, X. V.; Wang, B.; Tan, Y. C.; Yang, X.; Radev, D. R.; Socher, R.; and Xiong, C. 2021.  GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Yu et al. (2019a) Yu, T.; Zhang, R.; Er, H.; Li, S.; Xue, E.; Pang, B.; Lin, X. V.; Tan, Y. C.; Shi, T.; Li, Z.; Jiang, Y.; Yasunaga, M.; Shim, S.; Chen, T.; Fabbri, A. R.; Li, Z.; Chen, L.; Zhang, Y.; Dixit, S.; Zhang, V.; Xiong, C.; Socher, R.; Lasecki, W. S.; and Radev, D. R. 2019a.  CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases.  In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , 1962–1979. Association for Computational Linguistics.
  * Yu et al. (2018) Yu, T.; Zhang, R.; Yang, K.; Yasunaga, M.; Wang, D.; Li, Z.; Ma, J.; Li, I.; Yao, Q.; Roman, S.; Zhang, Z.; and Radev, D. R. 2018.  Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.  In Riloff, E.; Chiang, D.; Hockenmaier, J.; and Tsujii, J., eds., _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , 3911–3921. Association for Computational Linguistics.
  * Yu et al. (2019b) Yu, T.; Zhang, R.; Yasunaga, M.; Tan, Y. C.; Lin, X. V.; Li, S.; Er, H.; Li, I.; Pang, B.; Chen, T.; Ji, E.; Dixit, S.; Proctor, D.; Shim, S.; Kraft, J.; Zhang, V.; Xiong, C.; Socher, R.; and Radev, D. R. 2019b.  SParC: Cross-Domain Semantic Parsing in Context.  In Korhonen, A.; Traum, D. R.; and Màrquez, L., eds., _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , 4511–4523. Association for Computational Linguistics.
  * Zhang et al. (2019) Zhang, R.; Yu, T.; Er, H.; Shim, S.; Xue, E.; Lin, X. V.; Shi, T.; Xiong, C.; Socher, R.; and Radev, D. R. 2019.  Editing-Based SQL Query Generation for Cross-Domain Context-Dependent Questions.  In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , 5337–5348. Association for Computational Linguistics.
  * Zheng et al. (2022) Zheng, Y.; Wang, H.; Dong, B.; Wang, X.; and Li, C. 2022.  HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing.  In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., _Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022_ , 2997–3007. Association for Computational Linguistics.
  * Zhong, Xiong, and Socher (2017) Zhong, V.; Xiong, C.; and Socher, R. 2017.  Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning.  _CoRR_ , abs/1709.00103.