#  $\delta$-PVDF Based Flexible Nanogenerator

Varun Gupta  Quantum Materials and Devices Unit, Institute of Nano Science and
Technology, Knowledge City, Sector 81, Mohali 140306, India  Anand Babu
Quantum Materials and Devices Unit, Institute of Nano Science and Technology,
Knowledge City, Sector 81, Mohali 140306, India  Sujoy Kumar Ghosh  Department
of Physics, Jadavpur University, Kolkata 700032, India  Present address: NEST,
Istituto Nanoscienze-CNR and Scuola Normale Superiore, Piazza San Silvestro
12, I-56127 Pisa, Italy  Zinnia Mallick  Quantum Materials and Devices Unit,
Institute of Nano Science and Technology, Knowledge City, Sector 81, Mohali
140306, India  Hari Krishna Mishra  Quantum Materials and Devices Unit,
Institute of Nano Science and Technology, Knowledge City, Sector 81, Mohali
140306, India  Dipankar Mandal<EMAIL_ADDRESS>Quantum Materials and
Devices Unit, Institute of Nano Science and Technology, Knowledge City, Sector
81, Mohali 140306, India

###### Abstract

Delta ($\delta$) phase comprising polyvinylidene fluoride (PVDF) nanoparticles
are fabricated through electrospray technique by applying 0.1 MV/m electric
field at ambient temperature and pressure, which is 103 times lower than the
typical value, required for $\delta$-phase transformation. The X-ray
diffraction (XRD) and selected area electron diffraction (SAED) patterns are
clearly indicating the $\delta$-phase formation. The piezo- and ferro-
electric response of the $\delta$-PVDF nanoparticles has been demonstrated
through scanning probe microscopic technique based on piezoresponse force
microscopy (PFM). The vertical piezoelectric response, indicated by d33
coefficient, is found $\sim$-11 pm/V. Kink propagation model is adopted to
justify the $\delta$-phase conversion in electrospray system. The electrical
response from $\delta$-PVDF nanoparticle comprised nanogenerator under the
external impacts and acoustic signal indicates that molecular ferroelectric
dipoles responsible for piezoelectric responses, are poled in-situ during
nanoparticle formation, thus further electrical poling is not necessary.

The piezoelectric polymers have been extensively explored due to their
potential applications as transducers, actuators, sensors and energy
harvesters. Won  _et al._ (2015); Reece  _et al._ (2003); Singh  _et al._
(2014); Ghosh and Mandal (2017); Martins, Lopes, and Lanceros-Mendez (2014);
Ghosh and Mandal (2016); Wang, Wang, and Yang (2016); Ranjan  _et al._ (2007);
Mandal, Yoon, and Kim (2011); Müller  _et al._ (2008) PVDF is one of such
piezoelectric semi-crystalline polymer that has been widely investigated and
considered to have at least five distinct crystalline phases. Lovinger (1983)
These phases have different chain conformations designated as all trans (TTTT)
planar zigzag $\beta$-phase (form I), T3GT3G´ $\gamma$-phase (form III),
$\epsilon$-phase (form V), TGTG´ (trans-gauche-trans-gauche) $\alpha$-phase
(form II) and $\delta$-phase (form IV). Banik  _et al._ (1979); Furukawa
(1989) Despite of the same chain conformations of $\alpha$ and $\delta$-phase,
the $\alpha$-phase is non-polar and paraelectric in nature due to its centro-
symmetric Broadhurst  _et al._ (1978); Lovinger (1981a); Li  _et al._ (2013a);
Hasegawa  _et al._ (1972); Hasegawa, Kobayashi, and Tadokoro (1972) (P21/c)
unit cell. The marcomolecular chain of PVDF (chemical formula, -(-CH2-CF2-)n-)
consists of repeated units of -CH2-CF2\- as a monomer. The electronegative F
and electropositive H atoms are the basis of molecular dipoles, (i.e. -CH2\-
and -CF2\- dipoles) perpendicular to molecular c-axis. The dipole moments
inside the unit cell of $\alpha$-phase are aligned in anti-parallel direction,
resulting a non-polar state of PVDF. Lovinger (1983) In contrast,
$\delta$-phase exhibiting a non centro-symmetric (P21cn) unit cell Bachmann
_et al._ (1980) that gives rise to piezoelectric, pyroelectric and
ferroelectric properties. In case of $\delta$-phase, every second chain along
c-axis is rotated about 180 degrees, as a result the dipole moments in unit
cell of $\delta$-phase are aligned parallel to each other and perpendicular to
molecular axis, giving rise to a polar state of PVDF. Dvey-Aharon, Taylor, and
Hopfinger (1980) Also, the X-ray diffraction (XRD) studies have shown that the
$\delta$-phase has the similar interplanar d-spacing as of $\alpha$-phase but
with the significant changes in their relative intensities for the specific
set of h, k, l values. In a $\delta$-phase unit cell, with an n glide
perpendicular to c-axis, hk0 reflections are absent for h + k = 2n + 1.
Bachmann  _et al._ (1980); Naegele, Yoon, and Broadhurst (1978) Theoretically,
the most accepted mechanism to achieve $\delta$-phase is, to rotate the each
of second polymer chain about its c-axis of $\alpha$-phase unit cell.Naegele,
Yoon, and Broadhurst (1978) Experimentally, in 1978, the pioneer group, Davis
et al. had shown that $\delta$-phase could be achieved from $\alpha$-phase by
applying a strong electrical field of $\sim$170 MV/m. Davis  _et al._ (1978)
In subsequent years, many other groups Naegele, Yoon, and Broadhurst (1978);
Lovinger (1981b); Costa and Giacometti (1993); Anwar and Asadi (2019); Davies
and Singh (1979) has also demonstrated the formation of $\delta$-phase by
applying higher electric field, with the coercive field of more than 100 MV/m.
In addition, there have been also reported a very few different techniques
such as electroforming Li  _et al._ (2013a); Davis  _et al._ (1978); Li  _et
al._ (2013b), solid state processing Martín  _et al._ (2017) and quenching to
achieve the $\delta$-phase Gan, Abd Majid, and Furukawa (2016); Servet and
Rault (1979); Qian  _et al._ (2018); García-Zaldívar  _et al._ (2017) in PVDF
(Discussion S1 and Fig. S1, supplementary material). Apart from thick and thin
films of PVDF different types of structures such as rods, tubes, particles,
flakes have also gained the interest due to their technological importance. Fu
_et al._ (2020)

In this work, we are reporting the formation of piezoelectric $\delta$-phase
comprising PVDF (henceforth, abbreviated as $\delta$-PVDF) nanoparticles, at
the lowest possible electric field reported till date (Table SI, supplementary
material). The application of lower electric field gives an advantage over the
dielectric fatigue and electrical breakdown of the polymer samples in
comparison to higher electrical poling field. In this process, we have used
the electrospray technique Wilm and Mann (1994) to achieve the $\delta$-PVDF
nanoparticles at room temperature (Discussion S2 and Fig. S2, supplementary
material).

Figure 1: (a) SAED pattern of $\delta$-phase in PVDF nanoparticles. Inset
shows the TEM imaging of PVDF nanoparticles (scale bar 200 nm). (b) X-ray
diffraction pattern of PVDF nanoparticles in different phases. Experimental
observation of $\delta$-phase at 30ºC (marked as $\delta$) and its conversion
to $\alpha$-phase at 170ºC after temperature treatment (marked as $\alpha$).
(c) Morphology of $\delta$-PVDF nanoparticles as uniform distribution in SEM
(scale bar 5 $\mu$m).

The SAED pattern (Fig. 1(a)) obtained in transmission electron microscope
(TEM) (Discussion S3, supplementary material) confirms the diffraction pattern
of $\delta$-PVDF nanoparticles (inset of Fig. 1(a)) with similar interplanar
spacing (d) values as obtained in XRD. The d-spacing of 0.441 nm corresponding
to peak for (110) shows the brightest ring in electron diffraction. The (110)
plane is consistent with the highest intensity peak in XRD. The reflections of
plane (020) has not observed in the SAED, since the (110) and (020) are very
closely originated and cannot be distinguished in bright rings as diffraction
angles are close for these two planes. The next diffraction ring is observed
for plane (111). Since (021) and (111) planes are closely spaced and have
lesser intense rings due to structure factors dependency of these reflections,
so only (111) was observed. The diffraction rings for plane (041), (211) and
(310) was also observed in SEAD which belongs to $\delta$-phase PVDF Bachmann
_et al._ (1980) with much smaller d-spacing, for higher 2$\theta$ values above
40º. The XRD pattern of PVDF nanoparticles at room temperature (RT) in Fig.
1(b) (upper panel, marked as $\delta$), resembles precisely with the
$\delta$-phase represented XRD patterns, having similar d-spacing Bachmann
_et al._ (1980) and interplanar reflections of (020), (110), (021) and (111)
at 2$\theta$ peak positions of 18.3º, 19.9º, 26.7º and 28.1º respectively, as
earlier reported by pioneer groups Naegele, Yoon, and Broadhurst (1978); Davis
_et al._ (1978) with lattice parameters of a = 4.96 Å, b = 9.64 Å, c = 4.62 Å.
It confirms the presence of $\delta$-phase in these PVDF nanoparticles
fabricated through the electrospray system. Further, we have demonstrated the
reverse phase transformation of PVDF from $\delta$-phase to $\alpha$-phase in
nanoparticles. In this case, higher temperature treatment (170ºC) was given to
the $\delta$-PVDF nanoparticles. The XRD pattern of the PVDF nanoparticles
after heat treatment at 170ºC (Fig. 1(b) lower panel, marked as $\alpha$)
shows the $\alpha$-phase PVDF characteristic diffraction peaks, appeared at
(2$\theta$) 17.6º, 18.3º, 19.9º, 25.6º, 26.7º and 28.1º attribute to lattice
planes of (100), (020), (110), (120), (021) and (111), respectively. Hasegawa,
Kobayashi, and Tadokoro (1972); Ince-Gunduz  _et al._ (2010) This phase change
from $\delta$ to $\alpha$-phase has occurred due to the temperature treatment
above the melt crystallization of PVDF, that destroys dipolar alignments of
$\delta$-phase (ferroelectric) and converts it to non-polar $\alpha$-phase
(paraelectric). The comparative study of both phases (Fig. 1(b)) shows that
the only significant change has observed as decrease in the peak intensities
of reflections (100) and (120) at 2$\theta$ position 17.6º and 25.6º in
$\delta$-phase, compared to the $\alpha$-phase PVDF, with the same d-spacing
and lattice parameters. So, it can be concluded that there is no disordering,
contraction or change in lattice shape and size. Then the only possible
changes in both the phases could be due to symmetry change of the unit cell
lattice. Gan, Abd Majid, and Furukawa (2016); Naegele, Yoon, and Broadhurst
(1978) This conclusion is consistent with basic unit cell structure of
$\alpha$ and $\delta$-phase. That reconfirms the $\delta$-phase formation in
electrospray system. The crystallite size of the $\delta$-phase and
$\alpha$-phase crystalline lamella was calculated to be $\sim$6 nm and
$\sim$13 nm, respectively (Discussion S4, supplementary material). Also, the
degree of crystallinity ($\chi_{c}$) of $\delta$-phase and $\alpha$-phase,
from XRD pattern, was found to be 47% and 52%, respectively (Discussion S4,
supplementary material). The fourier transform infrared spectroscopy (FTIR)
study of the nanoparticles (Fig. S3 (marked as $\delta$), supplementary
material) shows the IR absorption bands of $\delta$-PVDF as reported by K.
Tashiro et al.Tashiro  _et al._ (2021). The vibrational bands at 1182 cm-1 and
1209 cm-1 are one of the conclusive way to identify the $\delta$-phase through
FTIR. The IR exposure causes the symmetric ($\nu_{s}$) and anti-symmetric
($\nu_{as}$) stretching of -CF2\- dipoles at 1182 cm-1 and 1209 cm-1,
respectively. The transition dipoles of $\nu_{s}$(CF2) at 1182 cm-1 are in the
same direction in $\alpha$ and $\delta$-phase unit cell and hence results no
change in intensity at 1182 cm-1 and it remains constant. While the
transitional dipole moments of $\nu_{as}$(CF2) at 1209 cm-1, are in the
opposite direction in unit cell of $\delta$-phase as compared to unit cell
$\alpha$-phase. Thus, as a result 1209 cm-1 absorption band does not appear in
IR spectra of $\delta$-phase. Tashiro  _et al._ (2021) This observation is
consistent with FTIR spectra of PVDF nanoparticles processed via electrospray.
Additionally, the FTIR spectra (Fig. S3 (marked as $\alpha$), supplementary
material) also confirms the phase change, occurring from initial
$\delta$-phase to the $\alpha$-phase due to the heat treatment at 170ºC. The
scanning electron microscopy (SEM) of the $\delta$-PVDF nanoparticles is
illustrated in Fig. 1(c), shows the formation of nanoparticles, ranges between
$\sim$ 60 - 250 nm. The 2D and 3D AFM topography image (Fig. S4(a) and Fig.
S4(b), supplementary material) displays the particles are spherical in shape
with typical diameter around 90 nm.

Figure 2:  (a) Phase (b) Amplitude response of $\delta$-PVDF nanoparticles in
PFM, after applying a DC bias of $\pm$ 35 V. Temperature dependent study of
$\delta$-PVDF nanoparticles as (c) Heat flow in DSC at 1ºC/ min. followed by
(i) first heating (ii) second heating cycle (inset shows the estimated
crystallinity from DSC and XRD). (d) Dielectric constant
($\epsilon^{{}^{\prime}}$) and loss tangent (tan $\delta$) of $\delta$-PVDF
for variable temperature at 1 kHz frequency.

To study the piezoelectric and ferroelectric properties of $\delta$-PVDF
nanoparticles, piezoresponse force microscopy (PFM) was performed. An AC
modulating voltage was applied between the conductive probe and ITO coated
substrate to induce the mechanical oscillation normal to the surface in
contact mode. The lock-in drive frequency of 12 kHz with drive amplitude of 8
V was employed to get the proper amplitude (A) and phase response ($\phi$) of
$\delta$-PVDF nanoparticles. The DC bias voltage of +35 V to -35 V was applied
to obtain the corresponding phase and amplitude response of $\delta$-PVDF
nanoparticles. The hysteresis plot (phase response) in Fig. 2(a) shows that
the dipoles of the $\delta$-PVDF nanoparticles are reorienting itself under
the application of external bias voltage. As the positive bias voltage is
applied, the dipoles inside the crystalline lamella of $\delta$-PVDF
nanoparticles are being aligned in the opposite direction of applied electric
field as negative phase response. The maximum reorientation takes place until
the bias voltage reaches to +35 V, with maximum dipole switching with phase of
-130º and therefore saturation takes place. Now, as the bias voltage starts
decreasing and it crosses the coercive voltage (Vc), the dipoles start
reorienting itself and keeps reorienting till it saturates to other end with
bias voltage of -35 V at 180º phase difference from the earlier phase. The
coercive voltage (Vc) was found to be $\pm$ 2.5 V. This final hysteresis plot
verifies the ferroelectric nature of these $\delta$-PVDF nanoparticles. The
butterfly loop (amplitude response) shown in Fig. 2(b), is the evidence of
piezoelectric response. As the bias voltage crosses the coercive voltage, the
crystal lattice expansion takes place and it attains the maximum deformation
of 0.27 nm at +35 V bias. As the applied DC voltage decreases, then it starts
contracting itself until it reaches the minimal amplitude position, and start
further expanding after increase in the magnitude of negative bias voltage
above Vc and reaches to almost similar maximum deformation amplitude, before
it start decreasing with decrease in the magnitude of negative bias voltage.
In this case, nanoparticle attains minimal position lower than the previous
amplitude. The butterfly loop confirms the piezoelectric response of
ferroelectric $\delta$-PVDF nanoparticles. The magnitude of d33 value was
calculated from the slope of the butterfly loop. Al-Jishi and Taylor (1985)
The obtained d33 value for $\delta$-PVDF nanoparticles is $\sim$-11 pm/V. The
negative sign in this piezoelectric coefficient is associated with the
direction of applied electric field with respect to the direction of
polarization arising in the material. Burkard and Pfister (1974)

The differential scanning calorimetry (DSC) was performed at the ramping rate
of 1ºC /min under N2 environment (Fig. 2(c)). The first heating cycle (Fig.
2(c-i)) from 120ºC - 200ºC, indicates the melting of different phases of PVDF.
In this cycle, two peaks arise due to presence of two different phases of PVDF
García-Zaldívar  _et al._ (2017), which could be predicated as $\delta$ and
$\alpha$-phase melting at 165ºC and 170ºC, respectively. In the cooling cycle,
the sample crystallization takes place. Finally, the 2nd heating cycle ((Fig.
2(c-ii)) indicates only a single melting peak at 169ºC, that belongs to a
single phase of PVDF. It can be interpreted as presence of $\alpha$-phase
only, since this observation is consistent with the XRD (Fig. 1(b)) and FTIR
(Fig. S3, supplementary material) results obtained from $\delta$-phase to
$\alpha$-phase transformation when temperature is employed. The degree of
crystallinity ($\chi_{c}$) was calculated (Discussion S5, supplementary
material) from DSC thermogram and it is consistent with estimated
crystallanity from XRD (Fig. 2(c) inset). The subtle difference in
crystallinity value arises as compared to XRD, due to the melting enthalpy
provided to polymer chain during DSC process, which results the reorientation
of molecular chain and changes the overall crystalline region in polymer
chain. The dielectric constant and loss tangent measurements of $\delta$-PVDF
nanoparticles were performed at 1 kHz frequency with a temperature range of 80
to 155ºC has shown in Fig. 2(d). At room temperature (RT) and 1 kHz, the
dielectric constant ($\epsilon^{{}^{\prime}}$) for $\delta$-PVDF nanoparticle
was measured $\sim$7.5, with the loss tangent (tan$\delta$) $\sim$0.25. For an
increasing temperature range, the dielectric constant and loss tangent
decreases with increase in frequency (Fig. S5, supplementary material). On the
other hand, at a constant frequency (1 kHz), the dielectric constant and loss
tangent increases gradually as the temperature increases. At lower
temperature, these dipole molecules do not possess the flexibility to reorient
themselves in the polymer chain. However, as the temperature increases, the
possibility of dipoles reorientation results the increases in dielectric
constant. In this scenario, the dielectric response is governed by the
anomalous behavior of the glass transition of the polymer, but in the further
increase in temperature results into the intense thermal vibrations that leads
to suppress the degree of reorientation, resulting the reduction in dielectric
constant of the polymer. El-Sayed, Abdel-Baset, and Hassen (2014) This
particular behavior in this temperature region could be explained as
ferroelectric to paraelectric like phase transition in $\delta$-PVDF.

The piezoelectric $\delta$-PVDF nanoparticles were obtained at significantly
lower electric field (0.1 MV/m) in comparison to earlier reported work. The
reason for the formation of $\delta$-phase nanoparticles, is majorly governed
by Taylor cone Vaseashta (2007) formation in electrospray process, in the
presence of applied external electric field.Ryan, Smith, and Stark (2012)
However, the reason of low electric field driven $\delta$-phase formation
through 180º chain rotation, can be explained by kink-propagation model. Dvey-
Aharon  _et al._ (1980); Dvey-Aharon, Taylor, and Hopfinger (1980) It also
demonstrates that at lower electric fields, this phase transformation is
significantly dependent on the temperature and the kink propagation velocity.
To explain the kink rotation mechanism of TGTG´ chain unit by 180º, the motion
of the chain has been defined by the following model.
The rotational kinetic energy of the molecular chain is defined as

$T=\dfrac{1}{2}\ I\ \sum_{i}\biggl{(}\frac{\partial{\theta_{i}}}{\partial
t}\biggr{)}^{2}$ (1)

where, I is moment of inertia of monomer unit about center of mass axis of the
chain; $\theta_{i}$ is the rotation angle of $i^{th}$ monomer unit from the
a-axis of the crystal, for a given time t.
The potential energy of the molecular chain is defined as

$U=\sum_{i}\ [\
A_{1}(1-\cos{\theta_{i}})+A_{2}(1-\cos{2\theta_{i}})+\frac{1}{2}\kappa({\theta_{i}}-{\theta_{i+1}})^{2}\
]$ (2)

where, $A_{1}$ and $A_{2}$ are constants associated with dipole moment of
TGTG´ unit and $\kappa$ is torsional constant. The first two terms (in Eq. 2)
denote the combined inter-chain interaction potential with external electric
field and last term appears as potential energy due to the torsional rigidity
of the molecular chain.
The Hamiltonian for this modelled system can be written as

$H=\dfrac{1}{2}\ I\ \sum_{i}\biggl{(}\frac{\partial{\theta_{i}}}{\partial
t}\biggr{)}^{2}+\sum_{i}\ [\
A_{1}(1-\cos{\theta_{i}})+A_{2}(1-\cos{2\theta_{i}})\\\
+\frac{1}{2}\kappa({\theta_{i}}-{\theta_{i+1}})^{2}\ ]$ (3)

Taking the Brownian motion of adjacent chains and anharmonic phonon forces
into consideration for energy transfer, the modelled system can be
approximated to Langevin equation.Dvey-Aharon, Taylor, and Hopfinger (1980)
The approximation to minimize the thermal contribution force term into the
continuum limit, results a generalized double Sine - Gordon equation:

$I\ \frac{\partial^{2}{\theta}}{\partial t^{2}}=-A_{1}\ \sin{\theta}-2A_{2}\
\sin{2\theta}+\kappa c^{2}\ \frac{\partial^{2}{\theta}}{\partial
x^{2}}-\lambda I\ \frac{\partial{\theta}}{\partial t}$ (4)

where, $\lambda$ is the damping constant and c is the periodicity along chain
axis. The possible solution to the above equation can be written as

$\theta(x,t)=2\ \arctan\ \exp\biggl{(}\frac{2(x+vt)}{d}\biggr{)}$ (5)

where, d is the width of the kink. The above equation represents the motion of
the soliton wave of electric polarization. For $\theta$ = 0 to $\theta$ =
$\pi$ at any time t, $\theta$(x,t) gives the probable propagation direction of
kink that could travel from unpoled state to the poled state under the
influence of applied electric field with kink velocity v. Eventually,
resulting as a chain segment rotation by 180 degrees. i.e. the possible
favorable pathway to achieve the $\delta$-phase in PVDF chain by rotating its
alternate second chain around c-axis.

Figure 3:  Kink propagation model for $\delta$-phase conversion in the
electrospray system demonstrated by double Sine-Gordon solution. The ordinate
shows the x/c, i.e. the distance (x) traveled by kink along the c-axis of unit
cell and abscissa shows the kink rotation angle at any time given t, in
presence of electric field generated through electrospray system (Fig. S2).
The beginning of the kink (right red dot) shows initial orientation of the
polymer chain at taylor cone and at the final stage of the kink (left red dot)
shows the 180º rotated PVDF polymer chain.

Now, following the continuum approximation, the propagation speed of kink
along c-axis is given by

$v=v_{o}[1+({E_{o}}/{E})^{2}]^{-{{1}/{2}}}$ (6)

where, $v_{o}$ is the max kink velocity at larger electric field $E_{o}$.Dvey-
Aharon, Taylor, and Hopfinger (1980) As discussed by DveyAharon et al. that at
larger electric fields, the kink velocity doesn’t depend on temperature. But
at lower electric fields, the temperature plays a significant role to achieve
the sufficient kink velocity for rotation. In our case, $E_{o}$ has been
approximated to the limit to achieve a stable Taylor cone formation Kim  _et
al._ (2007), i.e. $\sim$0.1 MV/m, for $\delta$-phase in electrospray system.
We have predicted an approximated pathway, for $\delta$-phase conversion in
electrospray process (Fig. 3) by simulating the kink propagation model for a
lamella size of $\sim$12 nm.Martín  _et al._ (2017) Also, it is notable that,
in Taylor cone formation, the polymer is initially present in gaseous state,
associated with molecular chains having higher entropy and free energy.
Eventually, leading the kink propagation at very much faster rate as compare
to thin films, in presence of electric field of same strength. So, this might
be further reason for even lesser field requirement to achieve the
$\delta$-phase in electrospray process. This theoretical model significantly
predicts the phase conversion into $\delta$-phase at lesser electric field
that observed experimentally.

Furthermore, $\delta$-PVDF nanoparticle based nanogenerator (NG) was prepared
(Discussion S6, supplementary material), to show the application as an
excellent piezoelectric energy harvester. The mechanical energy harvesting
mechanism by the $\delta$-PVDF nanoparticles made device was explored by the
finite element method (FEM) based theoretical simulation. It has observed that
the NG generates higher stress (Fig. 4(a)) and thus higher piezo-potential
(Fig. S6, supplementary material), in comparison that of planar PVDF film
based device, due to stress confinement effect under application of 100 kPa
pressure. The energy harvesting performance of NG was experimentally recorded
in terms of open circuit output voltage (Voc) and short-circuit current (Isc)
by repetitive finger imparting motion with the applied force amplitude of
$\sim$4.5 N. The peak to peak response generated (Fig. 4(b)) as Voc $\sim$2.8
V and Isc $\sim$1.4 $\mu$A . The acoustic response of the NG was measured
(with 40 W RMS) at $\sim$80 dB of sound pressure level (SPL) with tunable
sound frequencies (40-140 Hz) (Fig. S7, supplementary material) indicating
acoustic sensitivity of $\sim$2.75 V/Pa. This fact indicates that
$\delta$-PVDF nanoparticles are in-situ poled during the electrospray process,
thus further electrical poling step is not required like typical piezoelectric
based sensor, actuators and energy harvesters.

Figure 4:  FEM based theoretical simulation of piezoelectric response in NG
made with $\delta$-PVDF nanoparticles. (a) Stress distribution of (i)
nanoparticle based $\delta$-PVDF NG under 100 kPa pressure. (ii) stress
confinement between the interfacial area of nanoparticles and (iii) stress-
distribution of the planar $\delta$-Phase NG showing no variation of stress
throughout the film. Piezoelectric response of fabricated NG as (b) Open
circuit output voltage (Voc) and short-circuit current (Isc).

In summary, we conclude that $\delta$-phase comprising PVDF nanoparticles were
successfully fabricated through electrospray process under 0.1 MV/m of
electric field. The ferro- and piezo- electric response of the $\delta$-PVDF
nanoparticle is evident from saturated phase response and from butterfly
response in PFM, respectively. The piezoelectric coefficient (d33) was
obtained $\sim$-11 pm/V. The phase transformation study indicates that the
$\delta$-phase has relatively lower melting temperature of 165ºC than the
$\alpha$-phase PVDF. Ferroelectric to paraelectric phase transition is also
observed via DSC study. The $\delta$-phase transformation in electrospray
process can be predicated by kink propagation model and can justify the low
electric field requirement. The fabricated nanogenerator has shown a very
prompt response with output voltage of $\sim$2.8 V and $\sim$1.4 $\mu$A
current and can be useful in prospective applications for self-powered
wearable electronics and sensors.

See supplementary material for experiment procedure, materials and methods.

###### Acknowledgements.

This work was financially supported by a grant from DST (DST/TMD/MES/2017/91),
Government of India. V. G. is thankful to DST for awarding INSPIRE fellowship
(IF180159).

## References

  * Won  _et al._ (2015) S. S. Won, M. Sheldon, N. Mostovych, J. Kwak, B.-S. Chang, C. W. Ahn, A. I. Kingon, I. W. Kim, and S.-H. Kim, Appl. Phys. Lett. 107, 202901 (2015).
  * Reece  _et al._ (2003) T. J. Reece, S. Ducharme, A. Sorokin, and M. Poulsen, Appl. Phys. Lett. 82, 142 (2003).
  * Singh  _et al._ (2014) P. Singh, H. Borkar, B. Singh, V. Singh, and A. Kumar, AIP Adv. 4, 087117 (2014).
  * Ghosh and Mandal (2017) S. K. Ghosh and D. Mandal, Appl. Phys. Lett. 110, 123701 (2017).
  * Martins, Lopes, and Lanceros-Mendez (2014) P. Martins, A. Lopes, and S. Lanceros-Mendez, Prog. Polym. Sci. 39, 683 (2014).
  * Ghosh and Mandal (2016) S. K. Ghosh and D. Mandal, Appl. Phys. Lett. 109, 103701 (2016).
  * Wang, Wang, and Yang (2016) S. Wang, Z. L. Wang, and Y. Yang, Adv. Mater. 28, 2881 (2016).
  * Ranjan  _et al._ (2007) V. Ranjan, L. Yu, M. B. Nardelli, and J. Bernholc, Phys. Rev. Lett. 99, 047801 (2007).
  * Mandal, Yoon, and Kim (2011) D. Mandal, S. Yoon, and K. J. Kim, Macromol. Rapid Commun. 32, 831 (2011).
  * Müller  _et al._ (2008) K. Müller, D. Mandal, K. Henkel, I. Paloumpa, and D. Schmeisser, Appl. Phys. Lett. 93, 112901 (2008).
  * Lovinger (1983) A. J. Lovinger, Science 220, 1115 (1983).
  * Banik  _et al._ (1979) N. Banik, F. Boyle, T. Sluckin, P. Taylor, S. Tripathy, and A. Hopfinger, Phys. Rev. Lett. 43, 456 (1979).
  * Furukawa (1989) T. Furukawa, Phase Transit. 18, 143 (1989).
  * Broadhurst  _et al._ (1978) M. Broadhurst, G. Davis, J. McKinney, and R. Collins, J. Appl. Phys. 49, 4992 (1978).
  * Lovinger (1981a) A. J. Lovinger, Macromolecules 14, 322 (1981a).
  * Li  _et al._ (2013a) M. Li, H. J. Wondergem, M.-J. Spijkman, K. Asadi, I. Katsouras, P. W. Blom, and D. M. De Leeuw, Nat. Mater. 12, 433 (2013a).
  * Hasegawa  _et al._ (1972) R. Hasegawa, Y. Takahashi, Y. Chatani, and H. Tadokoro, Polym. J. 3, 600 (1972).
  * Hasegawa, Kobayashi, and Tadokoro (1972) R. Hasegawa, M. Kobayashi, and H. Tadokoro, Polym. J. 3, 591 (1972).
  * Bachmann  _et al._ (1980) M. Bachmann, W. Gordon, S. Weinhold, and J. Lando, J. Appl. Phys. 51, 5095 (1980).
  * Dvey-Aharon, Taylor, and Hopfinger (1980) H. Dvey-Aharon, P. Taylor, and A. Hopfinger, J. Appl. Phys. 51, 5184 (1980).
  * Naegele, Yoon, and Broadhurst (1978) D. Naegele, D. Yoon, and M. Broadhurst, Macromolecules 11, 1297 (1978).
  * Davis  _et al._ (1978) G. Davis, J. McKinney, M. Broadhurst, and S. Roth, J. Appl. Phys. 49, 4998 (1978).
  * Lovinger (1981b) A. J. Lovinger, Macromolecules 14, 225 (1981b).
  * Costa and Giacometti (1993) M. M. Costa and J. A. Giacometti, Appl. Phys. Lett. 62, 1091 (1993).
  * Anwar and Asadi (2019) S. Anwar and K. Asadi, ACS Macro Lett. 8, 525 (2019).
  * Davies and Singh (1979) G. Davies and H. Singh, Polymer 20, 772 (1979).
  * Li  _et al._ (2013b) M. Li, I. Katsouras, K. Asadi, P. W. Blom, and D. M. De Leeuw, Appl. Phys. Lett. 103, 072903 (2013b).
  * Martín  _et al._ (2017) J. Martín, D. Zhao, T. Lenz, I. Katsouras, D. M. de Leeuw, and N. Stingelin, Mater. Horiz 4, 408 (2017).
  * Gan, Abd Majid, and Furukawa (2016) W. Gan, W. Abd Majid, and T. Furukawa, Polymer 82, 156 (2016).
  * Servet and Rault (1979) B. Servet and J. Rault, J. Phys. 40, 1145 (1979).
  * Qian  _et al._ (2018) J. Qian, S. Jiang, Q. Wang, S. Zheng, S. Guo, C. Yi, J. Wang, X. Wang, K. Tsukagoshi, Y. Shi, and Y. Li, Sci. Rep. 8, 532 (2018).
  * García-Zaldívar  _et al._ (2017) O. García-Zaldívar, T. Escamilla-Díaz, M. Ramírez-Cardona, M. Hernández-Landaverde, R. Ramírez-Bon, J. Yañez-Limón, and F. Calderón-Piñar, Sci. Rep. 7, 1 (2017).
  * Fu  _et al._ (2020) C. Fu, H. Zhu, N. Hoshino, T. Akutagawa, and M. Mitsuishi, Langmuir 36, 14083 (2020).
  * Wilm and Mann (1994) M. S. Wilm and M. Mann, Int. J. Mass Spectrom. Ion Processes 136, 167 (1994).
  * Ince-Gunduz  _et al._ (2010) B. S. Ince-Gunduz, R. Alpern, D. Amare, J. Crawford, B. Dolan, S. Jones, R. Kobylarz, M. Reveley, and P. Cebe, Polymer 51, 1485 (2010).
  * Tashiro  _et al._ (2021) K. Tashiro, H. Yamamoto, S. Kummara, T. Takahama, K. Aoyama, H. Sekiguchi, and H. Iwamoto, Macromolecules 54, 2334 (2021).
  * Al-Jishi and Taylor (1985) R. Al-Jishi and P. Taylor, J. Appl. Phys. 57, 902 (1985).
  * Burkard and Pfister (1974) H. Burkard and G. Pfister, J. Appl. Phys. 45, 3360 (1974).
  * El-Sayed, Abdel-Baset, and Hassen (2014) S. El-Sayed, T. Abdel-Baset, and A. Hassen, AIP Adv. 4, 037114 (2014).
  * Vaseashta (2007) A. Vaseashta, Appl. Phys. Lett. 90, 093115 (2007).
  * Ryan, Smith, and Stark (2012) C. Ryan, K. Smith, and J. Stark, J. Appl. Phys. 112, 114510 (2012).
  * Dvey-Aharon  _et al._ (1980) H. Dvey-Aharon, T. J. Sluckin, P. L. Taylor, and A. J. Hopfinger, Phys. Rev. B 21, 3700 (1980).
  * Dvey-Aharon, Taylor, and Hopfinger (1980) H. Dvey-Aharon, P. L. Taylor, and A. J. Hopfinger, J. Appl. Phys. 51, 5184 (1980).
  * Kim  _et al._ (2007) H. Kim, D. Luo, D. Link, D. A. Weitz, M. Marquez, and Z. Cheng, Appl. Phys. Lett. 91, 133106 (2007).


#  QTSumm: A New Benchmark for Query-Focused Table Summarization

Yilun Zhao1 Zhenting Qi2 Linyong Nan1 Boyu Mi2
Yixin Liu1 Weijin Zou1 Simeng Han1 Xiangru Tang1 Yumo Xu3
Arman Cohan1,4 Dragomir Radev1
1Yale University 2Zhejiang University 3Amazon Research 4Allen Institute for AI
<EMAIL_ADDRESS>
###### Abstract

People primarily consult tables to conduct data analysis or answer specific
questions. Text generation systems that can provide accurate table summaries
tailored to users’ information needs can facilitate more efficient access to
relevant data insights. However, existing table-to-text generation studies
primarily focus on converting tabular data into coherent statements, rather
than addressing information-seeking purposes. In this paper, we define a new
query-focused table summarization task, where text generation models have to
perform human-like reasoning and analysis over the given table to generate a
tailored summary, and we introduce a new benchmark named QTSumm for this task.
QTSumm consists of 5,625 human-annotated query-summary pairs over 2,437 tables
on diverse topics. Moreover, we investigate state-of-the-art models (i.e.,
text generation, table-to-text generation, and large language models) on the
QTSumm dataset. Experimental results and manual analysis reveal that our
benchmark presents significant challenges in table-to-text generation for
future research.

Figure 1: An example of QTSumm. Given the numerous data points in the table,
different users may be interested in various aspects for their own
information-seeking or decision-making purposes. The system needs to perform
human-like reasoning and analysis over relevant table regions to generate a
tailored table summary.

Dataset | Table Source | # Tables / Statements |  | # Words /
---
Statement
Explicit Control |  | Rich in
---
Reasoning?
ToTTo Parikh et al. (2020) | Wikipedia | 83,141 / 83,141 | 17.4 | Table region | ✗
LogicNLG Chen et al. (2020a) | Wikipedia | 7,392 / 36,960 | 14.2 | Table regions | ✓
HiTab Cheng et al. (2022) | Statistics web | 3,597 / 10,672 | 16.4 | Table regions & reasoning operator | ✓
FeTaQA Nan et al. (2022b) | Wikipedia | 10,330 / 10,330 | 18.9 | Queries rewritten from ToTTo | ✗
RotoWire Lebret et al. (2016) | NBA games | 4,953 / 4,953 | 337.1 | ✗ | ✗
SciGen Moosavi et al. (2021) | Sci-Paper | 1,338 / 1,338 | 116.0 | ✗ | ✓
NumericNLG Suadaa et al. (2021) | Sci-Paper | 1,355 / 1,355 | 94.2 | ✗ | ✓
QTSumm | Wikipedia | 2,437 / 5,625 | 67.7 | Queries from a real-world scenario | ✓

Table 1: Comparison of existing table-to-text generation datasets.

##  1 Introduction

In the era of data-driven decision-making, tabular data plays a crucial role
in facilitating data analysis, serving as a concise and structured
representation of information Kukich (1983); Pasupat and Liang (2015); Chen et
al. (2020c); Zhu et al. (2021); Zhao et al. (2022a). People often consult
tables to extract valuable insights and make informed decisions. For example,
sales managers typically explore large tables with specific business questions
to gain insights about clients and processes. Sports coaches will analyze
performance tables containing various statistics to develop effective game
strategies and make critical team adjustments. However, effectively accessing
and comprehending the information contained within a large and complex table
can be time-consuming for users Hurst (2000); Pasupat and Liang (2015); Pujara
et al. (2021). Text generation systems that can accurately summarize the
provided table, tailored to users’ information needs, have the potential to
greatly enhance data analysis and expedite the process of obtaining data
insights.

Existing work and datasets on table-to-text generation Parikh et al. (2020);
Chen et al. (2020a); Cheng et al. (2022); Lebret et al. (2016); Moosavi et al.
(2021); Suadaa et al. (2021) have mainly focused on converting tabular data
into coherent statements, aiming to present the structured data in a human-
readable format. However, these approaches have overlooked the fundamental
goal of addressing users’ _information-seeking_ purposes. Table-to-text
generation systems should adopt a more flexible and interactive approach that
allows people to obtain a user-customized summary tailored to their
information needs Dang (2006); Xu and Lapata (2020); Zhong et al. (2021); Xu
and Lapata (2022), as illustrated in Figure 1. While table question answering
(QA) Pasupat and Liang (2015); Iyyer et al. (2017); Zhong et al. (2018); Chen
et al. (2020c); Nan et al. (2022b) has made significant progress in answering
fact-based questions, the primary focus of their approaches is on extracting
relevant facts or entities from the table and composing short-form answers.
Nevertheless, in real-world scenario, users often have more complex and
diverse information needs that extend beyond simple fact retrieval. They
expect models to perform _human-like reasoning_ and provide explanations or
justifications that accompany the extracted insights.

With comprehensive consideration of the real-world information needs of users
when consulting tabular data, we propose a new task, query-focused table
summarization. In this task, the model is required to generate a user-
customized summary given the table and user query. To enable research in this
area, we construct a human-annotated table-to-text generation dataset named
QTSumm 111The data and code will be released at
https://github.com/yilunzhao/QTSumm upon publication., that contains 5,625
query-summary pairs over 2,437 Wikipedia tables covering diverse topics. Table
QTSumm: A New Benchmark for Query-Focused Table Summarization compares QTSumm
with previous table-to-text generation work. To the best of our knowledge,
QTSumm is the first table-to-text generation dataset that tackle tasks of
generating user-customized table summary based on the real-world scenarios.

Furthermore, we provide a comprehensive evaluation of current state-of-the-art
models, including text generation Lewis et al. (2020); Raffel et al. (2020);
Chung et al. (2022), table-to-text generation models Liu et al. (2022b); Zhao
et al. (2022b); Liu et al. (2022a); Xie et al. (2022), and large language
models Brown et al. (2020); Wei et al. (2022). Our results and analysis from
different perspectives reveal that the existing models struggle in solving
this new task, highlighting the challenges the models face when performing
human-like reasoning and analysis to generate summary tailored to users’
information needs. We are releasing our dataset and baselines to support
additional research in query-focused table summarization.

The main contribution of this work is three-fold:

  * •

We propose a new task, query-focused table summarization. The task requires
the model to generate customized summary tailored to users’ information needs,
driving future research on table-to-text generation from a user-centric
perspective.

  * •

We construct a new large-scale dataset, QTSumm, with 5,626 query-summary pairs
collected under real-world scenarios. We apply strict quality control to
ensure high quality of dataset.

  * •

We conduct a systematic study of state-of-the-art models on QTSumm, and
demonstrate that they are still far behind expert performance, strongly
motivating future research.

##  2 Related Work

#### Table-to-Text Generation

Existing work and datasets on table-to-text generation typically pose the
problem as either a single-sentence generation task Chen et al. (2020a);
Parikh et al. (2020); Cheng et al. (2022); Liu et al. (2022a); Nan et al.
(2022a); Zhao et al. (2023), or a generic summarization task Lebret et al.
(2016); Moosavi et al. (2021); Suadaa et al. (2021). In the single-sentence
generation task, the focus is on generating fluent and faithful descriptions
using provided table regions as a control for text generation. For example,
the ToTTo Parikh et al. (2020) and HiTab Cheng et al. (2022) datasets require
models to convert highlighted table regions into a single-sentence
description. The LogicNLG dataset Chen et al. (2020a) involves generating
statements that can be logically entailed by the facts in the provided table
regions. Nevertheless, using table regions for controlling text generation
does not correspond with real-world scenarios where people refer to tabular
data for information-seeking purposes. The generic table summarization tasks
aim to create concise and informative summaries based on the content of a
given table. RotoWire Lebret et al. (2016) requires models to produce a
coherent summary given a game performance table. SciGen Moosavi et al. (2021)
and NumericNLG Suadaa et al. (2021) datasets require models to perform
arithmetic reasoning and generate descriptions for tables extracted from
scientific papers. Compared to these domain-specific table summarization
datasets, the tables in QTSumm covers diverse topics. Furthermore, considering
the numerous data points in the table, various users may be interested in
different aspects, making it challenging to create a generic summary that
encompasses all the salient information within the table. Therefore, we
propose and investigate a new query-focused task setting.

#### Table QA

Table QA is a task that involves answering user questions based on the
information in a source table. Existing Table QA datasets, such as
WikiTableQuestions Pasupat and Liang (2015), SQA Iyyer et al. (2017), and
WikiSQL Zhong et al. (2018), primarily focus on _short-form_ answers, which
usually consist of a single cell value or aggregation of multiple cell values.
However, in real-world scenario, users often have more complex and diverse
information needs that requires to engage a deeper understanding and reasoning
over the given table. FeTaQA Nan et al. (2022b) is a recently introduced Table
QA dataset that collects queries by rewriting ToTTo’s Parikh et al. (2020)
statements into questions and uses the same statements as the answers. In
comparison with FeTaQA, the queries in QTSumm were annotated under real-world
scenarios, making them more natural and better reflecting users’ actual
information needs. Furthermore, FeTaQA’s queries typically focus on surface-
level facts, whereas QTSumm requires models to perform human-like reasoning
and analysis when generating summaries.

#### Query-Focused Summarization

Initially formulated as a document summarization task, QFS aims to generate
summaries from documents that are tailored to specific user queries Dang
(2006). Despite its potential real-world applications, QFS remains a
challenging task due to the lack of large-scale training data. Existing works
have attempted to address this issue by leveraging distant NLP resources,
including question answering Xu and Lapata (2020) and paraphrase
identification Su et al. (2020), and generic summarization Xu and Lapata
(2022). Recently, Zhong et al. (2021) adopted QFS for meeting summarization
and proposed a human-annotated benchmark over meeting transcripts. Similar to
text, effectively accessing and comprehending the information contained within
a large and complex table can be time-consuming for users, while QFS remains
unexplored in table-to-text generation. In this work, we extend QFS to this
new modality for more effective information-seeking and decision-making
purposes.

##  3 QTSumm Dataset

In this section, we introduce the QTSumm task and its data collection process.

###  3.1 Problem Formulation

We formally define the proposed query-focused table summarization task as
follows. The input is a user query $\bm{Q}$, and a table $\bm{T}$. The table
$\bm{T}=W\bm{\cup}\\{T_{i,j}|i\leq R_{T},j\leq C_{T}\\}$ has $R_{T}$ rows and
$C_{T}$ columns, with $W$ being the table title and $T_{i,j}$ being the
textual content in the $(i,j)$-th cell. The task objective of QTSumm is to
generate a paragraph-long textual summary $\bm{Y}=(y_{1},y_{2},\dots,y_{n})$
given the user query $\bm{Q}$ and source table $\bm{T}$:

$\bm{Y}=\mathrm{argmax}\prod_{i=1}^{n}P(y_{i}|y_{<i},\bm{Q},\bm{T};\theta),$
(1)

where $\theta$ denotes the parameters of a neural text generation model, and
$y_{i}$ denotes the $i$-th tokens in the generated summary.

###  3.2 Data Collection Principles

At a high level, the goal of the data collection process is to obtain high-
quality user queries and their corresponding paragraph-long summaries grounded
on the tabular data. We enumerate our key desiderata for designing a benchmark
capable of through evaluation of table-to-text summarization capabilities of
models. To achieve this, we first design three principles for annotating a
good query-summary pair:

  * •

Attributablity & Faithfulness: The query should be answerable using only
information from the source table. The summary should be grounded on the
source table, and not contain any unfaithful or nonsensical text.

  * •

Comprehensiveness: The paragraph-long summary should provide enough details of
the source table to respond to the user query so as to satisfy the user’s
information need.

  * •

Fluency: Both the user query and its corresponding table summary should be
coherent and fluent, without any grammar errors.

###  3.3 Annotation Pipeline

To ensure that QTSumm annotation fulfills the aforementioned principles, we
carefully design an annotation pipeline consisting of four steps, with details
discussed as follows:

#### Source Table Collection

QTSumm uses tables from LogicNLG Chen et al. (2020a) and ToTTo Parikh et al.
(2020) datasets as source tables, as these tables are crwaled from Wikipedia
and covers diverse domains and topics. We filter out tables that are 1) too
large or too small, 2) with only string-type columns, or 3) with hierarchical
structures (e.g., containing more than one table header). Then we randomly
sample 2,000 candidate tables from LogicNLG and ToTTo, respectively, for the
query-summary annotation.

#### User Query Annotation

Given a table, the annotators are required to read its content, and determine
whether the table is informative and intelligible to common web users. Then
they were asked to come up with two or three queries, assuming they are users
seeking certain information from the table. We require each query to be
answerable using information only from the table. Moreover, as this work
focuses on paragraph-long summaries as query responses, we avoid queries that
can be answered in a short sentence (e.g., “Which country held the 2022 FIFA
World Cup?”).

#### Query-Focused Summary Annotation

Given a table and user query, we ask another annotator to use only information
from the source table to write a paragraph-long summary that satisfies the
user’s information need. We encourage annotators to produce sophisticated
summaries that 1) contain as much information from the table as possible, and
2) involve more types of reasoning over multiple relevant table regions. To
further encourage high quality annotations, similar to TabFact Chen et al.
(2020b), we adopt the "two channel collection" design, in which the annotators
would be paid 60% more if their summaries are manually verified to exhibit
adequate complexity. Finally, the annotators are required to record the row
IDs of relevant table regions that are covered in the written summary, to
allow us quantifying how grounded the summaries are to the table in the future
research.

#### Multi-Round Validation

We conduct a multi-round validation protocol to ensure that the annotated data
fulfills the aforementioned annotation principles. We first assign query
annotators to validate each summary against their corresponding queries, and
fix the mistakes if there are any. Then we check 1) whether a query-summary
pair contain adequate information and complex aggregation by examining the
length of the summary, and 2) whether the information in summary is essential
in responding to the user query. We manually revise pairs that do not meet the
above standard.

###  3.4 Annotation Quality Control

Along with the multi-round validation, we also carefully design several
strategies to ensure the high quality of QTSumm annotation.

#### Expert Annotators

To help improve the annotation process, five experts with professional
experience in the text summarization tasks are invited to conduct the
_internal annotation_. They are asked to provide feedback regarding the task
instructions and the user experience of the annotation interface, based on
which we iteratively modify the annotation guideline and interface design. In
the stage of _external annotation_ , we enroll 17 graduate students majoring
in STEM fields (10 females, and 7 males). We do not use the crowd-source
annotation platform such as Mechanical Turk as our preliminary study indicates
that annotators on MTurk fail to annotate high-quality query-summary data.
Before starting the official annotation process, each annotator is given a
two-hour training session to learn the annotation requirements and interface.

#### Annotation De-biasing

We observed several kinds of annotation bias during our internal annotation,
and we proposed countermeasures as follows for annotation de-biasing:

Source Table Diversity: During internal annotation, we found that many tables
in LogicNLG have similar content. For example, there are around 200 tables
describing the results of football games, with identical table headers. To
ensure the diversity of source tables, we keep only one table for each unique
table header.

Query Diversity: When annotating queries, annotators may prefer simpler ones,
resulting in low query diversity. Therefore, we frequently monitor the
diversity of queries for each annotator. Annotators are also encouraged to
craft queries that are either creative or require complex reasoning in
summarization, resulting in a doubled payment to compensate them for the extra
time.

Supporting Fact Position: We found that annotators prefer to raise queries
regarding the first few rows of each table. To deal with such bias regarding
supporting fact positions, we randomly highlight certain rows for each table
in the annotation interface. We require the annotators to write queries whose
summaries should cover at least two rows of the highlighted regions.

###  3.5 Dataset Analysis

Property | Value
---|---
Unique Tables | 2,437
Query-Summary Pairs | 5,625
Rows per Table (Median/Avg) | 9 / 10.1
Columns per Table (Median/Avg) | 6 / 6.3
Table Title Length (Median/Avg) | 5 / 4.8
Query Length (Median/Avg) | 19 / 18.6
Relevant Rows (Median/Avg) | 4 / 3.9
Summary Length (Median/Avg) | 71 / 67.7
Training Set Size (Table/Summary) | 1,713 / 3,937 (70%)
Development Set Size (Table/Summary) | 362 / 843 (15%)
Test Set Size (Table/Summary) | 362 / 845 (15%)

Table 2: Basic statistics of QTSumm dataset.

Figure 2: Domain distribution of QTSumm tables.

Annotation Quality | %S $\geq$ 4 | Agree |  | Kappa
---
/ 95% CI
Table Informativeness | 84.9 | 0.81 | 0.77 / [0.72, 0.82]
Query Meaningfulness | 93.2 | 0.89 | 0.84 / [0.79, 0.89]
Query Complexity | 91.4 | 0.87 | 0.81 / [0.75, 0.87]
Query Fluency | 97.2 | 0.94 | 0.92 / [0.90, 0.94]
Relevant Rows Correctness | 89.7 | 0.85 | 0.83 / [0.79, 0.88]
Summary Comprehensiveness | 97.5 | 0.97 | 0.93 / [0.90, 0.96]
Summary Faithfulness | 91.6 | 0.90 | 0.88 / [0.84, 0.92]
Summary Fluency | 96.1 | 0.93 | 0.89 / [0.86, 0.92]

Table 3: Human evaluation over 200 samples of QTSumm. Three internal
evaluators were asked to rate the samples on a scale of 1 to 5. We report 1)
percent of samples that have an average score $\geq$ 4 to indicate the
annotation quality of QTSumm; and 2) percent of agreement and Randolph’s Kappa
with 95% CI Randolph (2005) to show the inter-annotator agreement.

Table 3.5 describes the basic statistics of QTSumm, and Table QTSumm: A New
Benchmark for Query-Focused Table Summarization shows a comprehensive
comparison of related table-to-text generation datasets. QTSumm is the first
dataset for research in query-focused summarization over tabular data. In
contrast to ToTTo, LogicNLG, and HiTab, which use specific table regions as
explicit control and focus on single-sentence generation, QTSumm requires
models to generate paragraph-long summaries via interacting with natural
language queries, thus is challenging and better reflect the real-world use
cases. Compared with FeTaQA, the queries in QTSumm were gathered from real-
world scenarios. Additionally, to better reflect users’ information needs,
QTSumm demands models to perform more complex and diverse types of reasoning
and aggregation over a larger relevant table region than FeTaQA. As
illustrated in Figure 2, QTSumm is also the first table summarization dataset
with an _open-domain_ table source, containing a large number of tables on
diverse topics.

#### Annotator Agreement

To evaluate the annotation quality of QTSumm, we also reported the human
evaluation scores and inter-evaluator agreements over 200 query-summary pairs
that were randomly sampled from the development set. As shown in Table 3.5,
QTSumm has a high annotation quality and inter-annotator agreement.

##  4 QTSumm Evaluation

As discussed in Section 3.2, a good summary should be fluent and without any
unfaithful or nonsensical text. Moreover, it should provide enough details of
the source table to satisfy the user information need. With this motivation,
we proposed both automated and human-evaluation metrics to measure the
_fluency_ , _faithfulness_ , and _comprehensiveness_ aspects of our task.

###  4.1 Automated Evaluation

BLEU Papineni et al. (2002) computes the geometric average of the precision
over output text’s n-grams. We used SacreBLEU Post (2018) that produces
comparable and reproducible BLEU scores.

ROUGE Lin and Hovy (2003) measures the word overlap between the candidate and
reference summaries. We reported F1 score for ROUGE-L (longest common
subsequences).

BERTScore Zhang et al. (2020) computes the similarity between the reference
and generated summary using contextual word embeddings.

LitePyramids Zhang and Bansal (2021) uses a semantic role labeling model to
decompose the reference summary into several more fine-grained semantic
triplets, each of which contains a single fact from the reference. It then
reports an entailment score between these decomposed semantic triplets and the
generated summary.

###  4.2 Human Evaluation

To gain a more comprehensive understanding of the system’s performance, we
also designed a human evaluation system. Specifically, the summaries from
different models were evaluated by experts from three criteria (i.e.,
_comprehensiveness_ , _faithfulness_ , and _fluency_) that have been discussed
in Section 3.2. Each summary was scored from 1 (worst) to 5 (best) for each
criteria, with the final score averaged across different expert evaluators.

##  5 Models

We evaluate following three types of models.

###  5.1 Text Generation Models

In this approach, we modeled QTSumm as an end-to-end, sequence-to-sequence
task, and applied following three popular text generation models.

BART Lewis et al. (2020) is a pre-trained denoising autoencoder with
transformer-based architecture and shows effectiveness in NLG tasks.

T5 Raffel et al. (2020) demonstrates effectiveness in NLG tasks by treating
all natural language problems as text-to-text tasks during pre-training stage.

Flan-T5 Chung et al. (2022) enhances T5 by scaling instruction fine-tuning and
demonstrates better human-like reasoning abilities than the T5 model.

###  5.2 Table-to-Text Generation Models

TAPEX Liu et al. (2022b) continues pre-training the BART model by using a
large-scale corpus of synthetic SQL query execution data. It shows better
table understanding and reasoning abilities.

ReasTAP Zhao et al. (2022b) enhances the table understanding and reasoning
abilities of BART by pre-training on a synthetic Table QA corpus.

PLOG Liu et al. (2022a) continues pre-training text generation models on a
table-to-logic-form generation task (i.e., T5 model), improving the
faithfulness of table-to-text generation.

UnifiedSKG Xie et al. (2022) unifies and multi-tasks structured knowledge
grounding with text-to-text language models (i.e., T5), achieving high
performance on various table-to-text tasks.

###  5.3 Large Language Models

GPT-3 Brown et al. (2020); Wei et al. (2022) is a powerful large language
model which is capable of generating human-like text and performing a wide
range of NLP tasks in a zero- or few-shot setting.

Type | Model | Backbone | ScareBLEU | ROUGE-L | BERTScore | LitePyramid
---|---|---|---|---|---|---
Text Generation | BART | – | 19.9 | 42.7 | 74.9 | 54.5
T5 | – | 20.3 | 42.9 | 75.2 | 55.6
Flan-T5 | T5 | 23.1 | 45.8 | 79.5 | 60.5
1-7 Table-to-Text | TAPEX | BART | 22.3 | 44.8 | 78.0 | 57.3
ReasTAP | BART | 23.2 | 45.0 | 78.7 | 58.1
UnifiedSKG | T5 | 23.4 | 46.2 | 79.3 | 60.2
PLOG | T5 | 23.7 | 46.1 | 79.9 | 60.9
1-7 LLM | GPT-3 0-shot | – | 16.7 | 34.2 | 62.4 | 46.2
GPT-3 2-shot | – | 19.5 | 41.3 | 73.2 | 56.5

Table 4: Automated evaluation results of baseline systems on the QTSumm test
set.

Model | Fluency | Comprehensiveness | Faithfulness
---|---|---|---
PLOG | 4.71 | 3.62 | 3.44
GPT-3 2-shot | 4.84 | 3.54 | 3.50
Flan-T5 | 4.80 | 3.75 | 3.56

Table 5: Human evaluation results (Likert Scale Scoring) of three best-
performance baselines. Three experts were enrolled to evaluate 50 predictions
for each model, which were randomly sampled from QTSumm test set.

##  6 Experiments

###  6.1 Implementation Details

#### Input Data Serialization

The input contains a user query, and corresponding table data. For text
generation and large language models (Section 5.1 & 5.3), we followed recent
works on table-to-text generation Liu et al. (2022b); Xie et al. (2022); Chen
(2023) to flatten the table data as T=[HEADER]:h,
[ROW]1:$r_{1}$,...,[ROW]n:$r_{n}$, where h is table header, $r_{i}$ is the
i-th table row, [HEADER] and [ROW] are special tokens indicating the region of
table headers and rows respectively. We also separated headers or cells in
different columns using a vertical bar |. In this way, the flattened table
input can be fed directly into text generation models. For table-to-text
generation models (Section 5.2), we followed their original data processing
methods to input the query and table data.

#### Experiment Setting

The fine-tuning experiments were conducted on an 8 NVIDIA RTX 3090 24GB
cluster. We selected the large version for all baseline models. For each fine-
tuning experiment, we ran 30 epochs with a batch size of 128. The best fine-
tuning checkpoints were selected according to the validation loss. For LLM
experiments, we randomly sampled 200 examples from the QTSumm test set. We
used text-davinci-003 via the public OpenAI APIs222https://openai.com/api/.
For its parameter setting, we used a temperature of 0.7, maximum output length
of 512 tokens, without any frequency or presence penalty. Figure 3 in Appendix
shows a prompt prefix example for text-davinci-003 in zero-shot setting.

###  6.2 Main Results

We evaluate all baseline systems using automated evaluations. Additionally, we
conduct a small-scale human evaluation involving a subset of models to further
verify the results of automated evaluation. Table 5.3 and Table 5 display the
results of automated and human evaluations, respectively.

###  6.3 Automated Evaluation Results

We draw the following two conclusions based on the automated evaluation
results.

#### Importance of table structure understanding

Table-to-text generation models achieve better performance than their
corresponding text generation backbones. For example, PLOG model that
continues pretraining T5 with table-to-logic-form corpus achieves a 5.3%
improvement on the LitePyramid metric. This improvement demonstrates the
importance of considering table structure understanding for the QTSumm task.

#### Importance of human-like reasoning

Among text generation models, Flan-T5, which enhances T5 by scaling
instruction fine-tuning, achieves the best performance. This finding
demonstrates the importance of human-like reasoning and anslysis for the
QTSumm task.

###  6.4 Human Evaluation Results

We observe that Flan-T5 performs better in terms of comprehensiveness and
faithfulness. Despite receiving low scores in automated evaluation, GPT-3
achieves results comparable to state-of-the-art fine-tuned models in human
evaluation. This motivates us to explore how to develop automated evaluation
metrics for the QTSumm dataset that align better with human judgments. We also
plan to further investigate the faithful table-to-text generation capability
of large language models in future work.

Source Task | ScareBLEU | ROUGE-L | BERTScore | LitePyramid
---|---|---|---|---
– | 20.3 | 42.9 | 75.2 | 54.6
RotoWire | 21.6 (+1.3) | 43.9 (+1.0) | 79.1 (+3.9) | 57.4 (+2.8)
HiTab | 22.4 (+2.1) | 44.2 (+1.3) | 79.8 (+4.6) | 57.9 (+3.3)

Table 6: Automated evaluation results of T5 on QTSumm development set with
multi-task fine-tuning.

##  7 Analysis

###  7.1 Multi-Task Results

The generative model can perform various downstream table-to-text tasks with
the same architecture. Therefore, we also investigated whether the generative
models can benefit from multi-task learning. We chose RotoWire Lebret et al.
(2016) and HiTab Cheng et al. (2022) as the source tasks, whose tables do not
overlap with QTSumm. The generative model (i.e, T5) was first fine-tuned on
the source task and then fine-tuned and evaluated on QTSumm. As shown in Table
6, multi-task fine-tuning can improve model performance on QTSumm. Moreover,
fine-tuning on HiTab brings the T5 model better improvement. This is because
HiTab requires text generation models to perform various kinds of reasoning
operations over tables covering diverse topics. And such reasoning
capabilities are also beneficial for QTSumm task.

###  7.2 Case Study

For a deeper understanding of the query-focused table summarization task on
QTSumm, we conducted a case study on the development set to illustrate
existing challenges. Table 7 shows user query, reference summary, and
predictions from the best-performance fine-tuned model (i.e. PLOG). We
identify the following common mistakes that current text generation models are
likely to make.

#### Hallucination

Consistent with many investigations on natural language generation tasks Ji et
al. (2022); Tang et al. (2022), we also observed that systems tend to
hallucinate in their generations. Specifically, we found a strong presence of
extrinsic hallucination, i.e., mentioning facts that are not shown in the
table.

#### Factual Incorrectness

We also found our best system suffers from intrinsic hallucination, that is,
presenting information that contradicts the source table. We suspect this can
be largely attributed to the failures in inaccurate reasoning.

#### User Intent Misunderstanding

Some system outputs include information that is irrelevant to the user
question. The third example of Table 7 presents a system generated summary
that contains extra information not requested by the user.

#### Repetition

We further noticed that some system generated summaries contain repetitive
information. The last example in Table 7 shows that the system mentions the
same information multiple times throughout the whole summary, which is not
desirable as summaries should be informative.

##  8 Conclusion

We introduce the task of query-focused table summarization, together with
QTSumm, the first dataset for the proposed task consisting of complex
questions paired with high quality summaries in large-scale. We elaborate on
many challenges that QTSumm poses to the existing summarization models, and
demonstrate with automatic and human evaluations of baseline systems that we
propose as benchmarks for our dataset. Our analysis suggests that existing
models struggle with producing factually correct and highly relevant summaries
given the user question, and more research efforts in this direction are
needed.

## Acknowledgements

We would like to dedicate this paper to the memory of Dr. Dragomir Radev. Dr.
Radev’s leadership, guidance, and expertise were instrumental in shaping the
direction and quality of this project. His loss is deeply felt by all of us
involved. We extend our heartfelt gratitude to Dr. Radev for his passion and
dedication to the whole NLP community.

## Ethical Consideration

The source tables in QTSumm were collected from LogicNLG Chen et al. (2020a)
and ToTTo Parikh et al. (2020) datasets, which are publicly available under
the MIT license333https://opensource.org/licenses/MIT and CC BY-SA 3.0 license
444https://creativecommons.org/licenses/by-sa/3.0/, respectively. They both
permit us to compose, modify, publish, and distribute additional annotations
upon the original dataset.

For the external annotation of QTSumm, we hired 17 graduate students majoring
in STEM majors. We regard 1) creating three queries for one table, and
validating the corresponding summaries annotated by others, and 2) composing a
query-focused summary response as a unit task. And we paid around $1.5 for
each unit task. For creative annotation rewards, we paid additional $0.5 for a
query, and $1.5 for a summary. Averagely, an annotator can finish 7 unit tasks
per hour after training and practicing. And the hourly rates are in the range
of $9 and $13 based on the different working speed (above the local average
wage of similar jobs). We recommended that annotators complete a maximum of 30
unit tasks per day in order to reduce pressure and maintain a comfortable
pace. In total, the approximate working hours to annotate QTSumm dataset was
1,400 hours. The whole annotation work lasted about 40 days.

## References

  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chen (2023) Wenhu Chen. 2023.  Large language models are few(1)-shot table reasoners.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 1120–1130, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Chen et al. (2020a) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a.  Logical natural language generation from open-domain tables.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7929–7942, Online. Association for Computational Linguistics.
  * Chen et al. (2020b) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b.  Tabfact : A large-scale dataset for table-based fact verification.  In _International Conference on Learning Representations (ICLR)_ , Addis Ababa, Ethiopia.
  * Chen et al. (2020c) Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c.  HybridQA: A dataset of multi-hop question answering over tabular and textual data.  pages 1026–1036.
  * Cheng et al. (2022) Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. 2022.  HiTab: A hierarchical table dataset for question answering and natural language generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1094–1110, Dublin, Ireland. Association for Computational Linguistics.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _ArXiv_ , abs/2210.11416.
  * Dang (2006) Hoa Trang Dang. 2006.  DUC 2005: Evaluation of question-focused summarization systems.  In _Proceedings of the Workshop on Task-Focused Summarization and Question Answering_ , pages 48–55, Sydney, Australia. Association for Computational Linguistics.
  * Hurst (2000) Matthew F. Hurst. 2000.  The interpretation of tables in texts.
  * Iyyer et al. (2017) Mohit Iyyer, Wen-tau Yih, and Ming-Wei Chang. 2017.  Search-based neural structured learning for sequential question answering.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1821–1831, Vancouver, Canada. Association for Computational Linguistics.
  * Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022.  Survey of hallucination in natural language generation.  _ACM Comput. Surv._ Just Accepted.
  * Kukich (1983) Karen Kukich. 1983.  Design of a knowledge-based report generator.  In _21st Annual Meeting of the Association for Computational Linguistics_ , pages 145–150, Cambridge, Massachusetts, USA. Association for Computational Linguistics.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213, Austin, Texas. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin and Hovy (2003) Chin-Yew Lin and Eduard Hovy. 2003.  Automatic evaluation of summaries using n-gram co-occurrence statistics.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_ , pages 150–157.
  * Liu et al. (2022a) Ao Liu, Haoyu Dong, Naoaki Okazaki, Shi Han, and Dongmei Zhang. 2022a.  PLOG: Table-to-logic pretraining for logical table-to-text generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5531–5546, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Liu et al. (2022b) Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and Jian-Guang Lou. 2022b.  TAPEX: Table pre-training via learning a neural SQL executor.  In _International Conference on Learning Representations_.
  * Moosavi et al. (2021) Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021\.  Scigen: a dataset for reasoning-aware text generation from scientific tables.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Nan et al. (2022a) Linyong Nan, Lorenzo Jaime Flores, Yilun Zhao, Yixin Liu, Luke Benson, Weijin Zou, and Dragomir Radev. 2022a.  R2D2: Robust data-to-text with replacement detection.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6903–6917, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Nan et al. (2022b) Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Hailey Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, Dragomir Radev, and Dragomir Radev. 2022b.  FeTaQA: Free-form table question answering.  _Transactions of the Association for Computational Linguistics_ , 10:35–49.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1173–1186, Online. Association for Computational Linguistics.
  * Pasupat and Liang (2015) Panupong Pasupat and Percy Liang. 2015.  Compositional semantic parsing on semi-structured tables.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1470–1480, Beijing, China. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Belgium, Brussels. Association for Computational Linguistics.
  * Pujara et al. (2021) Jay Pujara, Pedro Szekely, Huan Sun, and Muhao Chen. 2021.  From tables to knowledge: Recent advances in table understanding.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, KDD ’21, page 4060–4061, New York, NY, USA. Association for Computing Machinery.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Randolph (2005) Justus J Randolph. 2005.  Free-marginal multirater kappa (multirater k [free]): An alternative to fleiss’ fixed-marginal multirater kappa. _Advances in Data Analysis and Classification_ , 4.
  * Su et al. (2020) Dan Su, Yan Xu, Tiezheng Yu, Farhad Bin Siddique, Elham Barezi, and Pascale Fung. 2020.  CAiRE-COVID: A question answering and query-focused multi-document summarization system for COVID-19 scholarly information management.  In _Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020_ , Online. Association for Computational Linguistics.
  * Suadaa et al. (2021) Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. 2021.  Towards table-to-text generation with numerical reasoning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1451–1465, Online. Association for Computational Linguistics.
  * Tang et al. (2022) Xiangru Tang, Arjun Nair, Borui Wang, Bingyao Wang, Jai Desai, Aaron Wade, Haoran Li, Asli Celikyilmaz, Yashar Mehdad, and Dragomir Radev. 2022.  CONFIT: Toward faithful dialogue summarization with linguistically-informed contrastive fine-tuning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5657–5668, Seattle, United States. Association for Computational Linguistics.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.  UnifiedSKG: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  pages 602–631.
  * Xu and Lapata (2020) Yumo Xu and Mirella Lapata. 2020.  Coarse-to-fine query focused multi-document summarization.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 3632–3645, Online. Association for Computational Linguistics.
  * Xu and Lapata (2022) Yumo Xu and Mirella Lapata. 2022.  Document summarization with latent queries.  _Transactions of the Association for Computational Linguistics_ , 10:623–638.
  * Zhang and Bansal (2021) Shiyue Zhang and Mohit Bansal. 2021.  Finding a balanced degree of automation for summary evaluation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6617–6632, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao et al. (2022a) Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022a.  MultiHiertt: Numerical reasoning over multi hierarchical tabular and textual data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6588–6600, Dublin, Ireland. Association for Computational Linguistics.
  * Zhao et al. (2022b) Yilun Zhao, Linyong Nan, Zhenting Qi, Rui Zhang, and Dragomir Radev. 2022b.  ReasTAP: Injecting table reasoning skills during pre-training via synthetic reasoning examples.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9006–9018, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhao et al. (2023) Yilun Zhao, Zhenting Qi, Linyong Nan, Lorenzo Jaime Flores, and Dragomir Radev. 2023\.  LoFT: Enhancing faithfulness and diversity for table-to-text generation via logic form control.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 554–561, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021.  QMSum: A new benchmark for query-based multi-domain meeting summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5905–5921, Online. Association for Computational Linguistics.
  * Zhong et al. (2018) Victor Zhong, Caiming Xiong, and Richard Socher. 2018.  Seq2SQL: Generating structured queries from natural language using reinforcement learning.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021.  TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3277–3287, Online. Association for Computational Linguistics.

##  Appendix A Appendix

Figure 3: An example of text-davinci-003 zero-shot prompt prefix for the
QTSumm dataset.

Query |  System output |  Reference |  Error analysis
---|---|---|---
Q: Which driver is more competitive, Tracy or Haberfeld, and why is that? |  Paul Tracy is more competitive than Mario Haberfeld, as evidenced in their results from the 2004 Centrix Financial Grand Prix of Denver. In the race, Tracy finished in second place, while Haberfeld placed in eighth. Additionally, Tracy belongs to Forsythe Racing, which has had a more successful track record compared to Haberfeld’s Walker Racing. Additionally, Tracy was able to maintain his position in the race for the full 90 laps, whereas Haberfeld was only able to make it to 86 laps. Based on these results, it is clear that Tracy was more competitive compared to Mario Haberfeld. |  Paul Tracy is more competitive. Tracy finished the race in second place with a time of +7.446 seconds, whereas Haberfeld finished in eighth place with a time of +24.141 seconds. This demonstrates that Tracy was able to navigate the track much faster than Haberfeld and place higher in the race. Additionally, Tracy earned 28 points while Haberfeld earned only 15 points, which confirms that Tracy is stronger competitor. In conclusion, Tracy is the more competitive driver. |  Error Type:
Hallucination

_Analysis_ :
There is no information showing that Forsythe Racing has a more successful
track record compared to Walker Racing, and the model hallucinates about
Haberfeld making only 86 laps, which in fact should be 90.
Q: According to the voting result, which representative election in North Carolina districts was the most competitive, and why? |  The most competitive representative election in North Carolina districts was the one for the 8th district, in which incumbent Larry Kissell (D) lost re-election to Richard Hudson (R). This was the only race which was decided by less than 10 percentage points, with Hudson winning 54.1% of the vote to Kissell’s 45.9%. The close margin of the race demonstrates that both candidates were competitive, and it was an intense contest until the very end. |  The race in the North Carolina 7th district was the most competitive, as the Democratic incumbent Mike McIntyre won by a slim margin, with only a 0.2% difference between him and his Republican challenger David Rouzer. Furthermore, this election was the only one among all North Carolina districts in 2012 that resulted in a margin of victory within less than 1 percent. |  Error Type:
Factually
Incorrectness

_Analysis_ :
The most competitive representative election should be the North Carolina 7th
district rather than 8th district, in which the difference between two rivals
was just 0.2%.
Q: Analyse the countries in East Asia with Human Development Index (HDI) higher than 0.8. |  The countries in East Asia with Human Development Index (HDI) higher than 0.8 are Hong Kong, Japan, Macau, Mongolia, and South Korea. Hong Kong has the highest HDI at 0.898, followed by Japan at 0.901. Macau has the lowest HDI at 0.653, and North Korea has the highest at 0.897. All of these countries have populations greater than 1.8 million, with the highest population density being in Japan at 337 people per km square. |  The countries in East Asia with an HDI higher than 0.8 are Hong Kong (PRC), Japan, and South Korea. Hong Kong has an HDI of 0.898 and a population density per square kilometer of 6390. Japan has an HDI of 0.901 and a population density per square kilometer of 337. South Korea has an HDI of 0.897 and a population density per square kilometer of 500. All three countries have a comparatively high population density which is likely to be a factor in their high HDI. |  Error Type:
User Intent
Misunderstanding

_Analysis_ :
The query does not ask for country with lowest HDI, or any country with HDI
lower than 0.8. Besides misunderstanding the user intent, the system output
also displays many other errors such as inconsistency (two countries have the
highest HDI), hallucination (confusing South Korea with North Korea),
factually incorrectness (Japan has highest HDI).
Q: What are the buildings that have been the tallest in Portland, Oregon between 1927 and 1965? Summarize the basic information of these buildings. |  The buildings that have been the tallest in Portland, Oregon between 1927 and 1965 are the Public Service Building, the Hilton Portland Hotel, and the Harrison West Condominium Tower. The Public Service Building is located at 920 Sw Sixth Avenue and has been the tallest building in Portland, Oregon. It stands at a height of 220 feet (67 meters) and has 16 floors. The Hilton Portland Hotel is located at 921 Sw Sixth Avenue and has been the tallest building in Portland, Oregon. |  Between 1927 and 1965, the tallest buildings in Portland, Oregon were the Public Service Building, the Hilton Portland Hotel, and the Harrison West Condominium Tower. The Public Service Building was built in 1927, was 220 feet / 67 meters tall, had 16 floors, and held the title of tallest building until 1962. The Hilton Portland Hotel was built in 1963, was 241 feet / 73 meters tall, had 22 floors, and held the title of tallest building until 1965. The Harrison West Condominium Tower was built in 1965, was 256 feet / 78 meters tall, had 25 floors, and held the title of tallest building until 1969. |  Error Type:
Repetition

_Analysis_ :
The information of these buildings being the tallest in Portland, Oregon has
been mentioned repetitively throughout the system output, while the system
fail to also distinguish them (until which year each of them was the tallest
respectively).

Table 7:  Case study for common errors made by PLOG. The colored text
highlights problematic parts of output.