# Rapid Chemical Enrichment by Intermittent Star Formation in GN-z11

Chiaki Kobayashi Centre for Astrophysics Research, Department of Physics,
Astronomy and Mathematics, University of Hertfordshire, Hatfield, AL10 9AB, UK
<EMAIL_ADDRESS>Andrea Ferrara Scuola Normale Superiore, Piazza dei
Cavalieri 7, 50126 Pisa, Italy

(Received August 29, 2023; Revised December 22, 2023; Accepted January 12,
2024)

###### Abstract

We interpret the peculiar super-solar nitrogen abundance recently reported by
the James Webb Space Telescope observations for GN-z11 ($z=10.6$) using our
state-of-the-art chemical evolution models. The observed CNO ratios can be
successfully reproduced – independently of the adopted initial mass function,
nucleosynthesis yields, and presence of supermassive ($>$1000$M_{\odot}$)
stars – if the galaxy has undergone an intermittent star formation history
with a quiescent phase lasting $\sim$100 Myr, separating two strong
starbursts. Immediately after the second burst, Wolf–Rayet stars (up to
$120M_{\odot}$) become the dominant enrichment source, also temporarily ($<$1
Myr) enhancing particular elements (N, F, Na, and Al) and isotopes (13C and
18O). Alternative explanations involving (i) single burst models, also
including very massive stars and/or pair-instability supernovae, or (ii) pre-
enrichment scenarios fail to match the data. Feedback-regulated, intermittent
star formation might be common in early systems. Elemental abundances can be
used to test this hypothesis and to get new insights on nuclear and stellar
astrophysics.

##  1 Introduction

The James Webb Space Telescope (JWST) is expected to find the first galaxies –
those that host or have hosted metal-free (known as Population III) stars.
Surprisingly, though, one of the most distant galaxies detected, GN-z11 at
redshift $z=10.6$, showed strong metal lines (Bunker et al., 2023). Even more
puzzlingly, GN-z11 shows an unusually high ($>\\!4\times$ solar) N/O ratio.
Super-massive stars have been suggested (e.g., Charbonnel et al., 2023;
Senchyna et al., 2023; Nagele & Umeda, 2023) as the N source in GN-z11. Is
such an unusual stellar population truly required, or would it be possible to
reproduce the observations more simply with a varying star formation history?

At the end of the ‘dark ages’ of the Universe, the cosmic dawn was heralded by
the birth of the first stars and galaxies. The first cosmic star formation is
driven by inefficient cooling from hydrogen molecules. Thus, on general
grounds, the first stars were expected to be massive, with masses
$M_{\star}\approx 100M_{\odot}$ (e.g., Abel et al., 2002; Bromm & Larson,
2004). However, the initial stellar mass depends on complex physical
processes, such as gas fragmentation, ionization, accretion, and feedback from
newborn stars. Once these processes are included in modern numerical
simulations, it seems possible to form lower-mass stars (Greif et al., 2011;
Hirano et al., 2014; Rossi et al., 2021), and even binaries (Stacy & Bromm
2013; see also Hartwig et al. 2023 for observational signatures).

The properties of the first stars, i.e., mass, rotation, multiplicity, and
magnetic fields, are important for the reionization and chemical enrichment of
intergalactic medium, seeding of super-massive black holes (BHs), and
gravitational wave emission. However, a direct detection of Pop III stars is
still lacking, i.e., no zero-metal star or galaxy has yet been found. Instead,
the nature of the first stars has been studied using the second generation of
stars born out of gas enriched by Pop III stars.

There is a consensus that second-generation stars can be found among extremely
metal-poor (EMP, [Fe/H] $<\\!-3$) stars in the Milky Way (Beers & Christlieb,
2005) and in dwarf spheroidals (Skúladóttir et al., 2021). From the analysis
of the elemental abundances of EMP stars, it has been deduced that the first
enrichment sources were likely to be $M_{\star}\approx 10$–$40M_{\odot}$
stars, which exploded as ‘faint’ supernovae (e.g., Umeda & Nomoto, 2003;
Ishigaki et al., 2018). Similar results are obtained also for quasar
absorption line systems, such as metal-poor damped Lyman-$\alpha$ systems
(DLAs), where accurate (barring uncertainties on dust depletion) elemental
abundances are measured (Kobayashi et al., 2011; Saccardi et al., 2023).

Theoretically, stellar rotation becomes more important at low metallicities,
because weaker stellar winds result in a smaller angular momentum loss than at
solar metallicity. As a result, if massive stars are fast rotators, rotational
mixing brings CNO cycle products into the convective He-burning layers, and
the stellar envelope containing light elements such as C, N, and F may be
ejected in stellar winds of Wolf–Rayet (WR) stars (Meynet & Maeder, 2002;
Limongi & Chieffi, 2018); without them, the N abundance is a dex lower (see
Fig. 9 of Kobayashi & Taylor 2023). WR stars can also explain the detection of
highly enhanced HF (hydrogen fluoride) in NGP–190387 at $z=4.42$, a dusty
star-forming galaxy discovered by the Atacama Large Millimeter/submillimeter
Array (ALMA; Franco et al., 2021).

In addition to NGP–190387, and rather unexpectedly, many high-redshift
galaxies contain large ($\simeq 10^{7}M_{\odot}$) amounts of dust (e.g.,
Ferrara et al., 2022; Inami et al., 2022). Although supernovae are usually
considered the main sources at high-$z$ ($z\sim 7$; Todini & Ferrara, 2001;
Dayal et al., 2022; Witstok et al., 2023), dust can be produced also by WR
stars (e.g., Lau et al., 2022).

The earliest JWST observations have revealed an unexpected abundance of super-
early ($z>10$), massive ($M_{*}\,\approx 10^{9}M_{\odot}$) galaxies at the
bright end ($M_{\rm UV}\approx-21$) of the ultraviolet luminosity function.
These galaxies tend to have very blue spectral slopes ($\beta<-2.4$). Hence,
the dust produced by massive stars associated with the observed stellar
population must have been efficiently evacuated (or destroyed) along with most
of the gas by powerful galactic outflows driven by the radiation pressure
produced by their compact ($\simeq 100$ pc), young (20–30 Myr) stellar
component (Ferrara et al., 2023; Ziparo et al., 2023; Fiore et al., 2023).
Such feedback temporarily quenches star formation (Looser et al., 2023; Gelli
et al., 2023) until the gas content of the galaxy is restored by efficient
cosmological gas accretion, entailing a duty cycle of $\simeq$50–100 Myr.

In this Letter, we aim to construct a scenario connecting these observations
with galactic chemical evolution (GCE) models. As different chemical elements
are produced by stars with different masses on different timescales, their
abundance ratios can uniquely constrain the star formation and enrichment
histories of galaxies.

| $t_{1}$ | $t_{2}$ | $\tau_{\rm i,1}=\tau_{\rm i,2}$ | $\tau_{\rm s,1}$ | $\tau_{\rm s,2}$ | $f_{\rm g,0}$ | $t_{\rm age}^{\rm max}$ | $\left<t_{\rm age}\right>_{\rm M}$ | $\psi$ |  $\log$ O/H $+12$ |  $\log$ N/O
---|---|---|---|---|---|---|---|---|---|---|---
| [Gyr] | [Gyr] | [Gyr] | [Gyr] | [Gyr] |  | [Myr] | [Myr] | [Gyr-1] |  |
Single starburst | - | - | 0.001 | 0.0002 | - | 0 | 4.2 | 3 | 22 | 7.787 | $-2.198$
Dual starburst (fiducial) | 0.1 | 0.2 | 0.001 | 0.2 | 0.0002 | 0 | 204 | 26 | 20 | 7.850 | 0.246
Dual starburst | 0.05 | 0.1 | 0.001 | 0.2 | 0.0002 | 0 | 103 | 9 | 28 | 7.754 | $-0.033$
Dual starburst | 0.1 | 0.1 | 0.001 | 0.2 | 0.0002 | 0 | 103 | 11 | 38 | 7.754 | 0.047
Dual starburst | 0.1 | 0.2 | 0.01 | 0.2 | 0.0002 | 0 | 204 | 34 | 41 | 7.769 | 0.208
Dual starburst | 0.1 | 0.2 | 0.001 | 0.2 | 0.001 | 0 | 205 | 27 | 33 | 7.864 | $-0.081$
Pre-enrichment | - | - | 0.001 | 0.0002 | - | 0.3 | 2.8 | 2 | 58 | 7.768 | $-0.374$
GN-z11 |  |  |  |  |  |  | $19_{-5}^{+10}$ | $10_{-2}^{+3}$ | $19_{-12}^{+23}$ | $7.82$ | $>\\!-0.25$
Table 1: Input parameters describing the adopted star formation histories
shown in Figs. 2 and 4. Here, $t_{1}$ and $t_{2}$ are the beginning (end) of
the first burst in dual burst models. The formation epoch $t_{\rm age}^{\rm
max}$ is chosen to match the observed O/H. The last four columns are model
output values at the observed epoch ($z=10.6$): mass-weighted age of stars
$\left<t_{\rm age}\right>_{\rm M}$, star formation rate $\psi$, mass-weighted
O abundance and N/O ratio. The observed values are taken from Bunker et al.
(2023).

##  2 Models

###  2.1 Galactic chemical evolution models

Figure 1:  Star formation history adopted by three different GCE models for
GN-z11: single starburst (green short-dashed line); dual starburst (red
solid); and single burst with pre-enrichment (blue long-dashed). The vertical
dotted line denotes the observed epoch of GN-z11.

Figure 2:  Evolution of CNO abundance ratios (by number), compared to the
observational data for GN-z11 (Cameron et al., 2023, gray areas/bars). Left
panels: as a function of time, for the same models in Fig. 1 including WR
stars and standard IMF. Middle panels: similar to the left panels but as a
function of gas oxygen abundance. Right panels: same as the middle panels but
only for our fiducial model, i.e., dual starburst with standard IMF, color-
coded with the time (in Gyr) elapsed since the first star formation episode.
The dotted lines indicate the solar ratios.

We use the GCE code from Kobayashi et al. (2000), implementing the so-called
one-zone model, which integrates the following equation:

$\displaystyle\frac{d(Z_{i}f_{\rm g})}{dt}$ $\displaystyle=$ $\displaystyle
E_{\rm SW}+E_{\rm SNcc}+E_{\rm SNIa}$ (1) $\displaystyle-$ $\displaystyle
Z_{i}\psi+Z_{i,{\rm inflow}}R_{\rm inflow}-Z_{i}R_{\rm outflow},$

where the mass fraction $Z_{i}$ of each element $i$ in gas-phase ($f_{\rm g}$
denotes the gas fraction111Ratio of gas mass to the total mass provided by
gas, stars in the galaxy, and the gas “reservoir”, i.e.
$\int_{0}^{\infty}R_{\rm inflow}\,dt+f_{\rm g,0}=1$, where $f_{\rm g,0}$ is
the initial gas fraction.) increases via element production ($E_{\rm
SW},E_{\rm SNcc}$, and $E_{\rm SNIa}$); the other terms are star formation
($\psi=f_{\rm g}/\tau_{\rm s}$), gas inflow $R_{\rm inflow}=\exp(-t/\tau_{\rm
i})/\tau_{\rm i}$, and outflow ($R_{\rm outflow}$) rates, respectively. The
model assumes instantaneous mixing of the elements but not instantaneous
recycling. A complete description can be found in Kobayashi et al. (2000) and
Kobayashi & Taylor (2023, hereafter KT23). The adopted parameters are
summarized in Table 1.

The code includes the latest nucleosynthesis yields of asymptotic giant branch
(AGB) stars, super-AGB stars, and core-collapse supernovae (including
hypernovae and failed supernovae) from Kobayashi et al. (2020a, hereafter
K20), as well as WR stars described in § 4.1. Type Ia supernovae are also
included ($E_{\rm SNIa}$) using the model in Kobayashi & Nomoto (2009) and
yields from Kobayashi et al. (2020b). However, their contribution becomes
important only at [Fe/H] $>\kern-11.99998pt\lower
4.73611pt\hbox{$\sim$}~{}-1$.

The compositions of the infalling gas ($Z_{i,{\rm inflow}}$) is set to be
primordial (see § 2.2.1 of K20). All models presented in this Letter have no
stars at the start. Our fiducial model assumes initial gas fraction $f_{{\rm
g},0}=0$, but we also present a pre-enriched model with $f_{{\rm g},0}=0.3$
and initial composition ($Z_{i,0}$) deduced from another GCE model.

The initial mass function (IMF), taken from Kroupa (2008), is a broken power
law in three mass ranges. Later in §4, we vary the massive-end slope $x$, and
lower and upper mass limits [$m_{\ell},m_{\rm u}$] for the Population I/II and
III stars, separately. These two IMFs are switched at the threshold (absolute)
metallicity $Z_{\rm th}=0.0001$.

It is important to note that our adopted nucleosynthesis yields reproduce the
observed elemental abundances in the Milky Way with the standard IMF:
$x=1.3,m_{\ell}=0.01M_{\odot}$, and $m_{\rm u}=120M_{\odot}$. This is
fundamentally different from the recent modeling works by Isobe et al. (2023);
Marques-Chaves et al. (2024); Bekki & Tsujimoto (2023).

##  3 Results

###  3.1 Single burst models

We first assume a single starburst starting 4 Myr before the observed epoch;
this is because the estimated age of GN-z11 stellar populations is $\sim$10
Myr (Bunker et al., 2023). The corresponding star formation rate per unit mass
is shown as a green short-dashed line in Figure 1.

The left panels of Figure 2 show the elemental abundance ratios of the
interstellar medium (ISM) predicted by the GCE models as a function of time,
while the middle panels $\log$ O/H $+12$, a commonly used proxy for
metallicity222The solar oxygen abundance adopted in K20 is 8.76.. At low
metallicities, WR stars produce very high (C,N)/O ratios, which quickly
decrease due to the large O production from supernovae. The N/O ratio
increases again at high metallicities because of ‘primary’ production of N
from AGB stars, as well as ‘secondary’ production of N (from initially
existing CNO) in massive stars. For the same reason, the initial C/N ratio
increase is followed by a rapid drop at high metallicities.

In general, a shorter $\tau_{\rm s}$ shifts the CNO tracks to the right in the
middle panels. However, the adopted $\tau_{\rm i}$ and $\tau_{\rm s}$ values
(Table 1) are very short, compared with those for present-day massive
galaxies. Even shorter $\tau_{\rm s}$ values do not affect the solution any
further. The CNO tracks are insensitive to $\tau_{\rm i}$. In conclusion,
although N/O can be high at very low or very high metallicities, with a single
starburst it is not possible to reproduce the observed ratio of GN-z11 at the
observed oxygen abundance (7.82; Cameron et al., 2023).

###  3.2 Dual burst models

We have seen that WR stars can potentially produce high (C,N)/O ratios, but
only at metallicities much lower than observed. However, there is a way to
overcome the problem, and this consists of assuming a dual starburst model. We
now assume that star formation starts333Results are insensitive to the
formation epoch, and thus pre-enrichment from Pop III stars formed at
$z>\kern-11.99998pt\lower 4.73611pt\hbox{$\sim$}~{}20$ is included in this
subsection. at $z\sim 16.7$ and continues for 100 Myr (Fig. 1); the infall and
star formation timescales are set to $\tau_{\rm i}=0.001$ and $\tau_{\rm
s,1}=0.2$ Gyr, respectively444The allowed range is $\tau_{\rm s,1}=0.1$–1 Gyr,
so that the ISM is sufficiently enriched by the initial star formation.
Results are insensitive to $\tau_{\rm i}$..

The results of this fiducial model (red curve) are shown in the left and
middle panels of Figure 2. After the first burst the metallicity already
reaches $\log$ O/H $+12=8.5$, and the evolutionary tracks up to this point are
similar to those for the single burst case.

Then, star formation is assumed to be completely quenched for 100 Myr555A
marginally consistent, lower-quality fit can nevertheless be obtained also
without a quiescent interval between the two bursts., possibly due to feedback
associated with the onset of an outflow. At $t=0.2$ Gyr, the secondary infall
($\tau_{\rm i}=0.001$ Gyr) of primordial gas occurs, which initially causes
dilution reducing the metallicity. As a more extreme, second starburst
($\tau_{\rm s,2}=0.2$ Myr)666The condition to reach the observed N/O is
$\tau_{\rm s,2}\leq 1$ Myr. is triggered, WR stars quickly enhance (C,N)/O
ratios until supernovae produce a large amount of O. The O abundance peaks at
$t=0.207$ Gyr, which corresponds to the lifetime of $\sim 30M_{\odot}$ stars,
and gradually decreases until $t=0.23$ Gyr. This, along with secondary N
production from metal-rich SNe and primary N production from AGB stars
(lifetime $\sim$40–150 Myr), leads to a final N/O increase.

This N/O evolution predicted by the dual burst fiducial model crosses the
observed range only once, at $t=0.204$ Gyr, i.e., 4 Myr after the onset of the
second burst (the right panels of Figure 2). The time spent in the data box is
very short ($\sim$0.6 Myr). Hence, we conclude that N-enriched objects similar
to GN-z11 might be rare. On the other hand, lower ratios than in GN-z11 (e.g.,
$\log$ N/O $=-0.4$ at $z\sim 6$ in Isobe et al. 2023) can be reproduced in our
scenario with longer $\tau_{\rm s,2}$. In this case, the evolution is slower
and these objects become more common.

At the observed epoch, the star formation rate in the fiducial model is 20
Gyr-1. This is in excellent agreement with the observed value
$\sim$19$M_{\odot}$ yr-1, provided the total stellar mass is $M_{\star}\sim
10^{9}M_{\odot}$. The amount of stars formed during the first burst is only
16% of the final stellar mass. The mass-weighted age is 26 Myr, which is
comparable to the estimated value of $\sim$10 Myr. With shorter first bursts
and/or shorter quiescence intervals, the mass-weighted age could be made as
short as $\sim$10 Myr.

###  3.3 Pre-enrichment?

An alternative way to change the gas metallicity before the first starburst is
by pre-enrichment from external galaxies. This model is shown in Figure 2, the
left and middle panels (blue long-dashed curve), where it is also compared
with the single and dual burst cases.

The pre-enriched model assumes the same $\tau_{\rm i}$ and $\tau_{\rm s}$ as
in the single burst one (Table 1), but with a gas chemical composition,
$Z_{i,0}$, taken from another (independent) GCE model with $\tau_{\rm i,0}=1$
and $\tau_{\rm s,0}=0.3$ Gyr. We set the initial gas fraction $f_{\rm
g,0}=0.3$. These $f_{\rm g,0}$ and $Z_{i,0}$ values are very similar to those
obtained for the dual starburst model during the interval after the first
burst, i.e., at $t=0.1$–0.2 Gyr. However, in this case, the initial stellar
fraction is zero, which is the key difference from the dual burst model (red
curve).

The pre-enrichment model starts at $\log$ O/H $+12=8.3$. Due to the dilution
with the infalling pristine gas, the metallicity (horizontally) decreases
keeping the same CNO ratios. The (C,N)/O ratios quickly (vertically) increase
due to enrichment from WR stars, and they return to their initial composition
values. Hence, the triangle track does not cross the observed range. The above
results are insensitive to the parameters of $\tau_{\rm i}$ and $\tau_{\rm
s}$. By changing the initial gas fraction $f_{\rm g,0}$ and/or the initial
composition $Z_{i,0}$, we cannot find any tracks that match the observed
abundance range. We conclude that this simple pre-enrichment prescription
cannot solve the problem at hand.

##  4 IMF dependence

Figure 3:  Evolution of CNO abundance ratios (by number) vs. time (left) and
gas oxygen abundance (right) for single starburst models with different IMFs.
The standard model includes WR stars assuming Kroupa IMF with massive-end
slope of $x=1.3$ (red solid lines). The additional curves assume instead a
top-heavy IMF for Pop III stars with a slope $x=0$ in the mass range
$30$–$120M_{\odot}$ (green short-dashed), $100$–$280M_{\odot}$ to add PISNe
(blue long-dashed), and $100$–$1000M_{\odot}$ to add VMSs (magenta dotted).
Gray areas/bars are the observational data for GN-z11 (Cameron et al., 2023).

###  4.1 Fate of massive stars

While the ultimate fate of stars depends on their initial mass777With the term
initial mass we refer to single stars, as stars can lose their envelope also
due to binary interactions., the final mass and nucleosynthesis yields are
determined by stellar mass-loss (e.g., Vink et al., 2011) as well as
convection treatment and nuclear reaction rates. We include nucleosynthesis
yields covering a full mass range up to $1000M_{\odot}$ as follows.

10–50$M_{\odot}$: These stars become core-collapse supernovae. Although
Limongi & Chieffi (2018) provided yields including explosive nucleosynthesis,
their yields do not match observations in the Milky Way, probably because of
the lack of hypernovae and mixing fallback. Therefore, we combine their
stellar wind yields (see below) with the explosive nucleosynthesis yields of
CO cores from K20, which result in an excellent agreement with the
observations of almost all elements888At $Z=0$, some stars in the range
13–40$M_{\odot}$ may explode as ‘faint’ supernovae/hypernovae. These stars
produce only a tiny amount of iron because of the relatively large remnant BH
($\sim$5$M_{\odot}$). Their contribution is negligible and is not included
here.. Roughly half of stars with 20–50$M_{\odot}$ are assumed to explode as
hypernovae leaving a BH, probably due to rotation and/or binary interaction.
The exact fraction of hypernovae can be determined from chemodynamical
simulations of a Milky Way-type galaxy; Kobayashi & Nakasato (2011) find
$\epsilon_{\rm HN}=(0.5,0.5,0.4,0.01,0.01)$ for $Z=(0,0.001,0.004,0.02,0.05)$
. All stars with 13–20$M_{\odot}$, and the rest of stars with
20–30$M_{\odot}$, explode as normal core-collapse supernovae with $10^{51}$
erg of explosion energy, leaving a neutron star. The rest of stars with
30–50$M_{\odot}$ are assumed to be ‘failed’ supernovae, which form a
$\sim$10$M_{\odot}$ BH. This is based on the unsuccessful explosion
simulations of such massive stars (Janka, 2012; Burrows & Vartanyan, 2021) and
the lack of massive progenitors expected at supernova locations in HST data
(Smartt, 2009), but the threshold was determined from GCE models (Fig. 4 of
K20).

60–140$M_{\odot}$: Stars above $\sim$90$M_{\odot}$ become pulsating pair-
instability supernovae. Their evolution and nucleosynthesis are uncertain, but
they will leave a $\sim$100$M_{\odot}$ BH. Although this phase is not
calculated, we use the stellar wind yields from Limongi & Chieffi (2018) for
13–120$M_{\odot}$ with four different metallicities and three rotational
velocities. With rotation, lower-mass stars become WR stars. We apply rotating
models to a metallicity-dependent fraction $\epsilon_{\rm HN}(Z)$ (given
above) of the stars; among these, 3% (97%) rotate at 300 (150) km s-1. The
remaining $1-\epsilon_{\rm HN}(Z)$ fraction of stars are assumed to be non-
rotating.

160–280$M_{\odot}$: If the masses of the first stars are
$\sim$160–280$M_{\odot}$, they explode as pair-instability supernovae (PISNe;
Barkat et al., 1967; Heger & Woosley, 2002; Nomoto et al., 2013; Takahashi et
al., 2018) leaving no remnant. PISNe have a very distinct nucleosynthetic
pattern. Although considerable effort has been made to detect such a
characteristic pattern, no observational signature for the existence of PISNe
has yet been convincingly found, neither in EMP stars (Cayrel et al., 2004;
Aguado et al., 2023) nor in DLAs (Kobayashi et al., 2011; Saccardi et al.,
2023). We apply a mix of non-magnetic, rotating and non-rotating models from
Takahashi et al. (2018) at $Z=0$ only, using $\epsilon_{\rm HN}(Z)$ described
above.

300–1000$M_{\odot}$: Very-massive stars (VMSs; $>$100$M_{\odot}$) have been
solidly identified in the Tarantula Nebula of the Large Magellanic Cloud
(Schneider et al., 2018). Various stellar evolution models (e.g., Szécsi et
al., 2022), and some nucleosynthesis yields (Yusof et al., 2013; Martinet et
al., 2022; Volpato et al., 2023) exist, although the results significantly
depend on the input physics. We take the nucleosynthesis yields of the ‘max’
mass-loss models from Volpato et al. (2023) for 300–1000$M_{\odot}$ and
100–1000$M_{\odot}$ at metallicity $Z=0$ and $0.0002$, respectively. We also
take the stellar wind yields of the V11 models from Higgins et al. (2023) for
100–500$M_{\odot}$ at $Z=0.014$, assuming very strong winds from Vink et al.
(2011). N production is seen as a result of the CNO cycle during core
H-burning. These yields do not include explosive nucleosynthesis (i.e.,
stellar winds only).

We do not include super-massive stars (SMSs; $>$1000$M_{\odot}$) in our GCE
models, as their properties and yields are uncertain. SMSs are originally
defined as those that collapse on the general relativistic instability before
igniting H-burning (Fuller et al., 1986). Incomplete core H-burning of metal-
enhanced stars with $>10^{4}M_{\odot}$ has been proposed as an explanation for
the abundance anomaly (Denissenkov & Hartwick, 2014), and the so-called O–Na
anti-correlation (Kraft et al., 1997) often seen in globular clusters of the
Milky Way. These stars have been invoked also to account for the high N/O
ratio measured in GN-z11 (Nagele & Umeda, 2023), although lower-mass stars
survive until He-burning and enhance C and O. However, the structure and
evolution depend not only on their initial masses and metallicities, but also
their formation path (Woods et al., 2019). In a very narrow mass range, zero-
metallicity stars may explode as general relativistic supernovae (Chen et al.,
2014), which will have a significant effect and would be inconsistent with the
observation of GN-z11.

###  4.2 CNO evolution

The IMF dependence on the CNO tracks are shown in Figure 3 as a function of
time (left) and ISM oxygen abundance (right). In order to maximize the impact
of VMSs, a top-heavy IMF is assumed for Pop III stars with a flat mass
distribution ($x=0$). At low metallicities, compared to the standard IMF (red
solid lines), a top-heavy999We adopt $m_{\ell}=30M_{\odot}$ for Pop III to
minimize O production from core-collapse supernovae. Pop III IMF up to $m_{\rm
u}=120M_{\odot}$ (green short-dashed) can produce up to $\sim 1$ dex
systematic increase of N/O. If the Pop III IMF is extended up to $m_{\rm
u}=280M_{\odot}$ (blue long-dashed), PISNe produce a large amount of Fe and
also O, and the initial (C,N)/O ratios become very low. Finally, if the Pop
III IMF is extended up to $m_{\rm u}=1000M_{\odot}$ (magenta dotted), the
initial (C,N)/O ratios can become high again due to the VMS contribution
(§4.1).

In the last two models with $m_{\rm u}>120M_{\odot}$, while PISNe are assumed
to occur only from Pop III stars, VMSs produce some metals via winds also at
higher metallicities. This causes a $\sim 2$ dex increase of N at intermediate
metallicities ($\log$ O/H $+12\sim 8$) which is however insufficient to match
the observed range of GN-z11. Larger $m_{\rm u}$ values do not increase the
N/O ratio further because of the Pop I/II IMF slope of $x=1.3$. The VMS
contribution is washed out once AGB stars start producing N at high
metallicities. Also note that VMSs decrease C (as it is transformed into N),
and these two models do not reproduce the observed C/O ratio either.

##  5 Conclusions

We have presented, for the first time, a chemical evolution model that
naturally explains the “anomalous” elemental abundance ratios of GN-z11.
Without changing the IMF or the nucleosynthesis yields, the observed data can
be reproduced if this galaxy has experienced a star formation history
featuring a quiescent phase, lasting $\sim$100 Myr, separating two strong
starbursts. Importantly, the observed ratios cannot be explained by single
burst models with VMSs and/or PISNe. Essentially, this is because VMSs do not
increase N/O sufficiently, while PISNe rather decrease N/O ratios.

In our successful models, prior to the observed epoch ($z=10.6$), the galaxy
has been chemically enriched. This pre-enrichment is likely caused by
internal, rather than external, sources. For a brief period after the second
burst, WR stars (up to $120M_{\odot}$) become the dominant enrichment source,
which explains the high (C,N)/O ratios at the observed metallicity of GN-z11.
WR stars also enhance particular elements (N, F, Na, and Al; see Appendix) and
isotopes (13C and 18O); the prediction of high fluorine abundance can be
tested with ALMA (e.g., Franco et al., 2021).

Our results strongly suggest that super-early galaxies undergo a feedback-
regulated, stochastic (Pallottini & Ferrara, 2023), or even intermittent (Cole
et al., 2023), star formation history. In spite of this, detecting the
chemical anomalies (e.g. high N/O) produced by multiple bursts might not be
easy as the chemical evolution proceeds quite rapidly, thus erasing this
signature on short time scales ($<\kern-11.99998pt\lower
4.73611pt\hbox{$\sim$}~{}1$ Myr). This scenario is also consistent with the
presence of strong radiation-driven outflows (Carniani et al., 2023), which
are necessary to clear the dust produced by the observed stars in super-early
galaxies like GN-z11 (Ferrara et al., 2023; Ziparo et al., 2023; Fiore et al.,
2023). At lower redshifts ($z\sim 7$), dust instead may accumulate inside
galaxies, which might correspond to dusty galaxies observed by ALMA (e.g.,
Dayal et al., 2022).

Our findings highlight the potential of chemical evolution models and data on
elemental abundances (and isotopic ratios), not only to investigate the
physics and evolution of the most distant galaxies but also to provide new
constraints on nuclear and stellar astrophysics.

We acknowledge the stimulating atmosphere of the IFPU Focus Week (May 2023),
where this work has been initially worked out. We thank M. Limongi, K.
Takahashi, and G. Volpato for providing the nucleosynthesis data. CK
acknowledges funding from the UK Science and Technology Facility Council
through grant ST/R000905/1, ST/V000632/1. The work was also funded by a
Leverhulme Trust Research Project Grant on “Birth of Elements”. Support from
ERC Advanced Grant INTERSTELLAR H2020/740120 (AF) is kindly acknowledged.

## Data Availability

The data underlying this article will be shared on reasonable request to the
corresponding author.

Our conclusions are based on the following parameter study. The left panels of
Figure 4 show a variety of dual burst models that can reproduce the
observations, compared to the fiducial model (red solid line); the differences
are small depending on the details of the dual burst models. (i) If the
duration of the first burst and/or interval between the two bursts are short
($t_{1}=0.05,t_{2}=0.1$ Gyr; green short-dashed), similar CNO tracks are
obtained, with a slightly lower peak N/O ratio. (ii) If no interval
($t_{1}=t_{2}=0.1$ Gyr; blue long-dashed) is assumed, the peak N/O ratio is
only marginally consistent with data, since the metallicity decrease due to
dilution after the secondary gas inflow is weaker. (iii) Similar CNO tracks
are also obtained with longer inflow timescales ($\tau_{\rm i}=0.01$ Gyr),
which also result in a higher star formation rate at the observed epoch. (iv)
Finally, with a weaker second burst ($\tau_{\rm s,2}=0.001$ Gyr), the N/O
ratio becomes lower, and the condition to reach the observed N/O is $\tau_{\rm
s,2}\leq 1$ Myr.

Our GCE models can predict elemental abundances and isotopic ratios, self-
consistently. The right panels of Figure 4 shows the [X/Fe]–[Fe/H] relations
of all stable elements from C to Zn, for our fiducial, dual burst model (red
solid line) and a single burst model (green dashed). For GN-z11, it would be
difficult to measure the abundances of elements other than CNO. Nevertheless,
these predictions can be compared with peculiar stars in the Milky Way to
study its formation process.

Figure 4:  Left: same as the middle panel of Fig. 2 but for a variety of dual
burst models assuming a shorter interval (green short-dashed line), no
interval (blue long-dashed), longer $\tau_{\rm i}$ (cyan dot-dashed), and
shorter $\tau_{\rm s,2}$ (magenta dotted), compared to the fiducial model (red
solid). The corresponding star formation histories are shown on the top
panels. Right: the [X/Fe]–[Fe/H] relations for the dual (red solid line) and
single (green dashed) burst models in Fig. 2.

## References

  * Abel et al. (2002) Abel, T., Bryan, G. L., & Norman, M. L. 2002, Science, 295, 93, doi: 10.1126/science.295.5552.93
  * Aguado et al. (2023) Aguado, D. S., Salvadori, S., Skúladóttir, Á., et al. 2023, MNRAS, 520, 866, doi: 10.1093/mnras/stad164
  * Barkat et al. (1967) Barkat, Z., Rakavy, G., & Sack, N. 1967, Phys. Rev. Lett., 18, 379, doi: 10.1103/PhysRevLett.18.379
  * Beers & Christlieb (2005) Beers, T. C., & Christlieb, N. 2005, ARA&A, 43, 531, doi: 10.1146/annurev.astro.42.053102.134057
  * Bekki & Tsujimoto (2023) Bekki, K., & Tsujimoto, T. 2023, MNRAS, 526, L26, doi: 10.1093/mnrasl/slad108
  * Bromm & Larson (2004) Bromm, V., & Larson, R. B. 2004, ARA&A, 42, 79, doi: 10.1146/annurev.astro.42.053102.134034
  * Bunker et al. (2023) Bunker, A. J., Saxena, A., Cameron, A. J., et al. 2023, A&A, 677, A88, doi: 10.1051/0004-6361/202346159
  * Burrows & Vartanyan (2021) Burrows, A., & Vartanyan, D. 2021, Nature, 589, 29, doi: 10.1038/s41586-020-03059-w
  * Cameron et al. (2023) Cameron, A. J., Katz, H., Rey, M. P., & Saxena, A. 2023, MNRAS, 523, 3516, doi: 10.1093/mnras/stad1579
  * Carniani et al. (2023) Carniani, S., Venturi, G., Parlanti, E., et al. 2023, arXiv e-prints, arXiv:2306.11801, doi: 10.48550/arXiv.2306.11801
  * Cayrel et al. (2004) Cayrel, R., Depagne, E., Spite, M., et al. 2004, A&A, 416, 1117, doi: 10.1051/0004-6361:20034074
  * Charbonnel et al. (2023) Charbonnel, C., Schaerer, D., Prantzos, N., et al. 2023, A&A, 673, L7, doi: 10.1051/0004-6361/202346410
  * Chen et al. (2014) Chen, K.-J., Heger, A., Woosley, S., et al. 2014, ApJ, 790, 162, doi: 10.1088/0004-637X/790/2/162
  * Cole et al. (2023) Cole, J. W., Papovich, C., Finkelstein, S. L., et al. 2023, arXiv e-prints, arXiv:2312.10152, doi: 10.48550/arXiv.2312.10152
  * Dayal et al. (2022) Dayal, P., Ferrara, A., Sommovigo, L., et al. 2022, MNRAS, 512, 989, doi: 10.1093/mnras/stac537
  * Denissenkov & Hartwick (2014) Denissenkov, P. A., & Hartwick, F. D. A. 2014, MNRAS, 437, L21, doi: 10.1093/mnrasl/slt133
  * Ferrara et al. (2023) Ferrara, A., Pallottini, A., & Dayal, P. 2023, MNRAS, 522, 3986, doi: 10.1093/mnras/stad1095
  * Ferrara et al. (2022) Ferrara, A., Sommovigo, L., Dayal, P., et al. 2022, MNRAS, 512, 58, doi: 10.1093/mnras/stac460
  * Fiore et al. (2023) Fiore, F., Ferrara, A., Bischetti, M., Feruglio, C., & Travascio, A. 2023, ApJ, 943, L27, doi: 10.3847/2041-8213/acb5f2
  * Franco et al. (2021) Franco, M., Coppin, K. E. K., Geach, J. E., et al. 2021, Nature Astronomy, 5, 1240, doi: 10.1038/s41550-021-01515-9
  * Fuller et al. (1986) Fuller, G. M., Woosley, S. E., & Weaver, T. A. 1986, ApJ, 307, 675, doi: 10.1086/164452
  * Gelli et al. (2023) Gelli, V., Salvadori, S., Ferrara, A., Pallottini, A., & Carniani, S. 2023, Quiescent Low-mass Galaxies Observed by JWST in the Epoch of Reionization, doi: 10.3847/2041-8213/acee80
  * Greif et al. (2011) Greif, T. H., Springel, V., White, S. D. M., et al. 2011, ApJ, 737, 75, doi: 10.1088/0004-637X/737/2/75
  * Hartwig et al. (2023) Hartwig, T., Ishigaki, M. N., Kobayashi, C., Tominaga, N., & Nomoto, K. 2023, ApJ, 946, 20, doi: 10.3847/1538-4357/acbcc6
  * Heger & Woosley (2002) Heger, A., & Woosley, S. E. 2002, ApJ, 567, 532, doi: 10.1086/338487
  * Higgins et al. (2023) Higgins, E. R., Vink, J. S., Hirschi, R., Laird, A. M., & Sabhahit, G. N. 2023, MNRAS, 526, 534, doi: 10.1093/mnras/stad2537
  * Hirano et al. (2014) Hirano, S., Hosokawa, T., Yoshida, N., et al. 2014, ApJ, 781, 60, doi: 10.1088/0004-637X/781/2/60
  * Inami et al. (2022) Inami, H., Algera, H., Schouws, S., et al. 2022, MNRAS, doi: 10.1093/mnras/stac1779
  * Ishigaki et al. (2018) Ishigaki, M. N., Tominaga, N., Kobayashi, C., & Nomoto, K. 2018, ApJ, 857, 46, doi: 10.3847/1538-4357/aab3de
  * Isobe et al. (2023) Isobe, Y., Ouchi, M., Tominaga, N., et al. 2023, ApJ, 959, 100, doi: 10.3847/1538-4357/ad09be
  * Janka (2012) Janka, H.-T. 2012, Annual Review of Nuclear and Particle Science, 62, 407, doi: 10.1146/annurev-nucl-102711-094901
  * Kobayashi et al. (2020a) Kobayashi, C., Karakas, A. I., & Lugaro, M. 2020a, ApJ, 900, 179, doi: 10.3847/1538-4357/abae65
  * Kobayashi et al. (2020b) Kobayashi, C., Leung, S.-C., & Nomoto, K. 2020b, ApJ, 895, 138, doi: 10.3847/1538-4357/ab8e44
  * Kobayashi & Nakasato (2011) Kobayashi, C., & Nakasato, N. 2011, ApJ, 729, 16, doi: 10.1088/0004-637X/729/1/16
  * Kobayashi & Nomoto (2009) Kobayashi, C., & Nomoto, K. 2009, ApJ, 707, 1466, doi: 10.1088/0004-637X/707/2/1466
  * Kobayashi & Taylor (2023) Kobayashi, C., & Taylor, P. 2023, arXiv e-prints, arXiv:2302.07255, doi: 10.48550/arXiv.2302.07255
  * Kobayashi et al. (2011) Kobayashi, C., Tominaga, N., & Nomoto, K. 2011, ApJ, 730, L14, doi: 10.1088/2041-8205/730/2/L14
  * Kobayashi et al. (2000) Kobayashi, C., Tsujimoto, T., & Nomoto, K. 2000, ApJ, 539, 26, doi: 10.1086/309195
  * Kraft et al. (1997) Kraft, R. P., Sneden, C., Smith, G. H., et al. 1997, AJ, 113, 279, doi: 10.1086/118251
  * Kroupa (2008) Kroupa, P. 2008, in Astronomical Society of the Pacific Conference Series, Vol. 390, Pathways Through an Eclectic Universe, ed. J. H. Knapen, T. J. Mahoney, & A. Vazdekis, 3.  https://arxiv.org/abs/0708.1164
  * Lau et al. (2022) Lau, R. M., Hankins, M. J., Han, Y., et al. 2022, Nature Astronomy, 6, 1308, doi: 10.1038/s41550-022-01812-x
  * Limongi & Chieffi (2018) Limongi, M., & Chieffi, A. 2018, ApJS, 237, 13, doi: 10.3847/1538-4365/aacb24
  * Looser et al. (2023) Looser, T. J., D’Eugenio, F., Maiolino, R., et al. 2023, arXiv e-prints, arXiv:2302.14155.  https://arxiv.org/abs/2302.14155
  * Marques-Chaves et al. (2024) Marques-Chaves, R., Schaerer, D., Kuruvanthodi, A., et al. 2024, A&A, 681, A30, doi: 10.1051/0004-6361/202347411
  * Martinet et al. (2022) Martinet, S., Meynet, G., Nandal, D., et al. 2022, A&A, 664, A181, doi: 10.1051/0004-6361/202243474
  * Meynet & Maeder (2002) Meynet, G., & Maeder, A. 2002, A&A, 390, 561, doi: 10.1051/0004-6361:20020755
  * Nagele & Umeda (2023) Nagele, C., & Umeda, H. 2023, ApJ, 949, L16, doi: 10.3847/2041-8213/acd550
  * Nomoto et al. (2013) Nomoto, K., Kobayashi, C., & Tominaga, N. 2013, ARA&A, 51, 457, doi: 10.1146/annurev-astro-082812-140956
  * Pallottini & Ferrara (2023) Pallottini, A., & Ferrara, A. 2023, A&A, 677, L4, doi: 10.1051/0004-6361/202347384
  * Rossi et al. (2021) Rossi, M., Salvadori, S., & Skúladóttir, Á. 2021, MNRAS, 503, 6026, doi: 10.1093/mnras/stab821
  * Saccardi et al. (2023) Saccardi, A., Salvadori, S., D’Odorico, V., et al. 2023, ApJ, 948, 35, doi: 10.3847/1538-4357/acc39f
  * Schneider et al. (2018) Schneider, F. R. N., Sana, H., Evans, C. J., et al. 2018, Science, 359, 69, doi: 10.1126/science.aan0106
  * Senchyna et al. (2023) Senchyna, P., Plat, A., Stark, D. P., & Rudie, G. C. 2023, arXiv e-prints, arXiv:2303.04179, doi: 10.48550/arXiv.2303.04179
  * Skúladóttir et al. (2021) Skúladóttir, Á., Salvadori, S., Amarsi, A. M., et al. 2021, ApJ, 915, L30, doi: 10.3847/2041-8213/ac0dc2
  * Smartt (2009) Smartt, S. J. 2009, ARA&A, 47, 63, doi: 10.1146/annurev-astro-082708-101737
  * Stacy & Bromm (2013) Stacy, A., & Bromm, V. 2013, MNRAS, 433, 1094, doi: 10.1093/mnras/stt789
  * Szécsi et al. (2022) Szécsi, D., Agrawal, P., Wünsch, R., & Langer, N. 2022, A&A, 658, A125, doi: 10.1051/0004-6361/202141536
  * Takahashi et al. (2018) Takahashi, K., Yoshida, T., & Umeda, H. 2018, ApJ, 857, 111, doi: 10.3847/1538-4357/aab95f
  * Todini & Ferrara (2001) Todini, P., & Ferrara, A. 2001, MNRAS, 325, 726, doi: 10.1046/j.1365-8711.2001.04486.x
  * Umeda & Nomoto (2003) Umeda, H., & Nomoto, K. 2003, Nature, 422, 871, doi: 10.1038/nature01571
  * Vink et al. (2011) Vink, J. S., Muijres, L. E., Anthonisse, B., et al. 2011, A&A, 531, A132, doi: 10.1051/0004-6361/201116614
  * Volpato et al. (2023) Volpato, G., Marigo, P., Costa, G., et al. 2023, ApJ, 944, 40, doi: 10.3847/1538-4357/acac91
  * Witstok et al. (2023) Witstok, J., Shivaei, I., Smit, R., et al. 2023, arXiv e-prints, arXiv:2302.05468, doi: 10.48550/arXiv.2302.05468
  * Woods et al. (2019) Woods, T. E., Agarwal, B., Bromm, V., et al. 2019, PASA, 36, e027, doi: 10.1017/pasa.2019.14
  * Yusof et al. (2013) Yusof, N., Hirschi, R., Meynet, G., et al. 2013, MNRAS, 433, 1114, doi: 10.1093/mnras/stt794
  * Ziparo et al. (2023) Ziparo, F., Ferrara, A., Sommovigo, L., & Kohandel, M. 2023, MNRAS, 520, 2445, doi: 10.1093/mnras/stad125


# ImPaKT: A Dataset for Open-Schema Knowledge Base Construction

Luke Vilnis Zach Fisher Bhargav Kanagal Patrick Murray Sumit Sanghai
Google Research
<EMAIL_ADDRESS>
###### Abstract

Large language models have ushered in a golden age of semantic parsing. The
seq2seq paradigm allows for open-schema and abstractive attribute and relation
extraction given only small amounts of finetuning data. Language model
pretraining has simultaneously enabled great strides in natural language
inference, reasoning about entailment and implication in free text. These
advances motivate us to construct ImPaKT, a dataset for open-schema
information extraction, consisting of around 2500 text snippets from the C4
corpus, in the shopping domain (product buying guides), professionally
annotated with extracted attributes, types, attribute summaries (attribute
schema discovery from idiosyncratic text), many-to-one relations between
compound and atomic attributes, and implication relations. We release this
data in hope that it will be useful in fine tuning semantic parsers for
information extraction and knowledge base construction across a variety of
domains. We evaluate the power of this approach by fine-tuning the open source
UL2 language model on a subset of the dataset, extracting a set of implication
relations from a corpus of product buying guides, and conducting human
evaluations of the resulting predictions.

##  1 Introduction

Figure 1: Parsing a semi-structured implication relation from web text. We
annotate attributes, and their summarized forms (in gray), attribute types (in
bold), and implication relationships.

Knowledge bases (KBs) have long been important components of machine learning
systems for natural language understanding and information synthesis, from
commonsense knowledge bases like Cyc Lenat (1995) and ConceptNet Speer et al.
(2017), to general KBs like FreeBase Google (2022), DBpedia Auer et al.
(2007), Wikidata Vrandečić and Krötzsch (2014), YAGO Suchanek et al. (2007),
or domain-specific KBs like the medical UMLS KB Bodenreider (2004).

Knowledge of structured relations, entities, and attributes is important for
providing reliable grounding for downstream tasks like question answering Gu
et al. (2022), summarization Narayan et al. (2021) and faceted search
Tunkelang (2009) and recommendations Sun and Zhang (2018). In commercial
settings, such as the shopping domain Yang et al. (2022), it is useful for
understanding user preferences and use cases for different products, and how
features interact.

As machine learning is applied more deeply and to more domains, manually
curated knowledge bases become less adequate for the task, and automated
knowledge base construction (AKBC) becomes a more important problem than ever.
Additionally, the fixed schemas of attribute types and relations in many
traditional knowledge bases prove brittle and inflexible in their application
to new domains. Approaches to open-schema KBC such as OpenIE (Banko et al.,
2007), NELL (Mitchell and Fredkin, 2014), and Universal Schema (Riedel et al.,
2013) are more important than ever in domains such as common sense reasoning,
medical, shopping, materials science, and others with an ever-evolving set of
attributes, types, and relationships. Finding the mentions, attributes,
relations for these open schemas can be difficult though and relies on human
labeling, syntactic rules, and heuristics like Hearst patterns Hearst (1992).
For example, OpenIE and Universal Schema both rely on pre-trained dependency
parsers to extract syntactic substructures as candidates for relations and
types.

Recently, neural networks, and specifically Large Language Models (LLMs) such
as GPT-3 Brown et al. (2020) and T5 Raffel et al. (2020); Xue et al. (2020)
have ushered in a golden age of semantic parsing. These models are capable of
extracting semistructured and structured data from freeform text with
remarkably little supervision, and the seq2seq paradigm Sutskever et al.
(2014) enables us to construct structured annotations through simple text-
based templating Lu et al. (2022).

As open information extraction enters the LLM age, we are inspired to release
the ImPaKT (Implication Parsing and Knowledge Extraction) dataset, a dataset
for training semantic parsers for information extraction and open-schema
semistructured AKBC. Our dataset consists of thousands of annotations of fine-
grained attributes, types, and implication relationships on C4 Raffel et al.
(2020) sentences from the shopping domain, but we believe it will be both
useful training data and a useful annotation template for constructing KBs in
many domains. In addition to being a commercially valuable domain to
understand in its own right, the shopping domain possess qualities such as
fine grained, ever evolving attribute type schemas, and common-sense style
relations, that make it ideal testbed for many annotation patterns of
interest.

While many existing datasets for attribute extraction exist, ours is uniquely
tailored for the strengths of LLMs. Attribute extraction datasets like MAVE
Yang et al. (2022) and MAE IV et al. (2017) have a large number of typed
shopping-related attributes, but they correspond strictly to text spans, and
lack the mapping of raw or loosely paraphrased text to the human-created
summarized forms that we provide. This allows models trained on our data to
have higher recall and even to do schema discovery in new domains. Further,
they lack the many-to-one associations between compound and atomic attributes
that allow us to parse complex relations. Commonsense datasets like ConceptNet
may contain some relations reminiscent of our implications, but they lack the
rest of our context and provenance, and are noisy.

We choose to annotate implication relationships because they possess many
qualities of commonsense reasoning, a long standing goal of natural language
understanding, as well as being valuable to understand in the shopping domain
for applications such as understanding consumer use cases e.g. understanding
that a cell phone with _large buttons_ would also be a cell phone that is
_good for seniors_ to use. Parsing implication relationships shares many
similarities with the natural language inference (NLI) (MacCartney, 2009;
Bowman et al., 2015) and recognizing textual entailment (RTE) (Baroni et al.,
2012; Dagan et al., 2013), wherein entailment (implicative) relations between
sentences or concepts are classified, a problem of longstanding interest to
the NLU community. Implication relationships themselves are also useful for
performing multi-hop reasoning and discovering causal chains between
attributes, an active area of research in KG completion Das et al. (2016,
2017).

We demonstrate the utility of our dataset in training information extraction
models by finetuning a UL2 language model on a mixture of seq2seq tasks
including attribute discovery, annotation, and implication extraction, and
conduct a human study of the resulting knowledge base. We release our dataset
publicly in hopes that researchers will use it to train their own parsers,
construct their own fine grained knowledge bases, and advance the study of
automated semistructured knowledge base construction.

##  2 Related Work

Machine-learned open-schema information extraction has a long history,
pioneered by works such as OpenIE (Banko et al., 2007), NELL (Mitchell and
Fredkin, 2014), and Universal Schema (Riedel et al., 2013).

Our work in open-schema and fine grained semistructured semantic parsing, with
an emphasis on common nouns, shares a lot of similarities with the literature
on commonsense and commonsense knowledge base construction, a fact that has
been noted by others studying information extraction in the shopping domain
(Yu et al., 2022). Early systems like Cyc (Lenat, 1995) rely on a manually
curated set of relations between common nouns, while later work such as NELL
(Mitchell and Fredkin, 2014) and ConceptNet has included more automatic
extractions (Speer et al., 2017). More recently, neural networks have been
applied to commonsense KB completion (Li et al., 2016) and now with the rise
of large language models, there is considerable research interest in the kind
of commonsense knowledge implicitly learned in the weights of these models (Li
et al., 2021). To this end, recent work has attempted to explicitly construct
a knowledge base using the latent knowledge in the language model (Hao et al.,
2022).

In contrast to this work, which has mostly focused on KB completion and
implicit knowledge, our work brings large language models to bear on the
problem of explicit information extraction. In many domains, we wish to parse
textually well-supported statements from a corpus, and leave inferences for a
separate part of the KB pipeline. Recent work in the shopping domain has
extracted large sets of attributes and types from Amazon data (Yang et al.,
2022). We extend this approach with a more fine-grained annotation scheme
including attribute summaries, compound and atomic attributes, and implication
relationships, which we believe is suited for even more domains and
applications within the shopping domain, such as recommendation and
summarization (Narayan et al., 2021; Brahman et al., 2022).

The implication relationships we annotate share similarities with the
literature on natural language inference (NLI) (MacCartney, 2009; Bowman et
al., 2015) and recognizing textual entailment (RTE) (Baroni et al., 2012;
Dagan et al., 2013). In contrast to this work, which aims to find when
sentences or concepts are logically entailed from one another (performing a
sort of inference over implicit knowledge), we focus on parsing textually
grounded statements where the text itself asserts an implication relationship.

Neural networks have long been employed for structured prediction and
information extraction tasks such as semantic and syntactic parsing, often
using custom architectures such as graph neural networks Battaglia et al.
(2018) or pointer networks Vinyals et al. (2015). Like much other recent work,
our proposed semantic parsing model follows the seq2seq framework (Sutskever
et al., 2014) in encoding input and output structures as simple textual
sequences encoded with a sentencepiece model, like recent work in sequence
tagging with transformers (Raman et al., 2022), event extraction Hsu et al.
(2022), and autoregressive entity retrieval (Cao et al., 2020). Our use of a
single LLM to generate multiple structured annotations is in the vein of the
Universal Information Extraction model of Lu et al. (2022).

##  3 Structure of the ImPaKT Dataset

| Annotation Aspects
---|---
| Snippets | Coarse Attribute | Implication | Atomic Attribute | Atomic Implication
Count | 2489 | 5655 | 3719 | 6117 | 3587
Table 1: Count of annotation aspects in dataset. Each aspect of annotation has
potentially multiple annotations associated with it. Snippets are labeled with
GPC categories and classification decisions (1628 positively containing an
implication, 859 negatives either for reason of being malformed text, in the
wrong category, or containing no implications). Coarse attributes are further
labeled with summaries. Atomic attributes are also labeled with summaries and
types.

⬇

"snippet":

"Uses durable 7075 aluminum stakes that won’t succumb to conditions",

"provenance": {

"url": "https://www.gearhungry.com/best-solo-tents-for-camping/",

"timestamp": "2019-04-19T21:11:49Z",

"span_start": 1531,

"span_end": 1597

},

"category":

"Sporting Goods > Outdoor Recreation > Camping & Hiking > Tents",

"classification": "Yes",

"attributes": [

{

"attribute": "durable 7075 aluminum stakes",

"summary": "durable 7075 aluminum stakes"

},

{

"attribute": "won’t succumb to conditions",

"summary": "impervious to conditions"

}

],

"atomic_attributes": {

"durable 7075 aluminum stakes": [

{

"attribute": "durable",

"summary": "durable",

"attribute_type": "stake durability"

},

{

"attribute": "aluminum",

"summary": "aluminum",

"attribute_type": "stake material"

},

{

"attribute": "7075 aluminum",

"summary": "7075 aluminum",

"attribute_type": "stake aluminum alloy"

}

],

"won’t succumb to conditions": [

{

"attribute": "won’t succumb to conditions",

"summary": "durable",

"attribute_type": "stake durability"

}

]

},

"implications": [

{

"antecedent": "durable 7075 aluminum stakes",

"consequent": "won’t succumb to conditions"

}

]

Figure 2: Example annotation. In this case, several atomic attributes
correspond to one compound antecedent in an implication.

In creating this dataset, we had to reconcile the annotation of fine-grained
structure with the intrinsic ambiguity of natural language and human semantic
judgments, as well as making it possible for human annotators to provide
reasonable labels. To that end, we split up some of the annotations into
several levels of granularity. The attributes come in two varieties, general
(possibly _compound_) attributes and _atomic_ attributes. All attributes are
represented by a string name and a possibly shorter and more normalized name
called a _summary_. The atomic attributes further come with a type annotation.

Since our dataset is constructed in the consumer product domain, our examples
will also come from that domain. These annotation principles should carry over
to many domains that can benefit from semantic parsing of semistructured
knowledge.

###  3.1 Attributes

In this work we refer to an _attribute_ , or _coarse_ attribute when context
requires disambiguation, as a string describing any aspect of a product in a
particular category, relevant to the use of the product or the product
category, and not necessarily based on any normalized schema. For example,
_good for large sandwiches_ could be a perfectly good attribute describing a
toaster oven or panini press. Clearly, the full set of attributes for any
given product is potentially extremely large and sometimes dependent on the
opinions of the users of that product, so in this work we do not attempt to
construct a knowledge base of every possible attribute. Instead we annotate
specific textual _snippets_ with particular choices of attributes, without
duplicating concepts, and when in doubt how to phrase the particular attribute
string we try to make it as close as possible to the phrasing in the snippet.

Often times, the surface text of attributes is presented in a way that is not
suitable for a more structured annotation. For this reason, we additionally
annotate each attribute with a _summary_. For example, for a snippet like _a
quality PC chair must offer proper neck and lumbar support, allowing the user
to adopt an ergonomic posture_ , we might lightly rephrase one attribute as
_allows user to adopt an ergonomic posture_ , but we can further summarize
this in context as simply _ergonomic_.

###  3.2 Implications

In the context of our dataset, we define an _implication_ between two
attributes $A$ and $B$ in the same category as a judgment that the existence
of attribute $A$ describing a product also leads us to believe that attribute
$B$ describes the product, _grounded in a specific textual statement or
statements_. That is, the text itself states the implication exists, we do not
infer the implication. For example, the snippet _the chair has a memory foam
cushion to promote good posture_ expresses an implication.

Either the left or right hand side of an implication might have several
different components in conjunction, disjunction, or even some other fuzzy
qualifiers. This is why we do not require our notion of attributes to
correspond to one single crisp concept, or else we would be unable to annotate
many implications of interest, such as _these white Reebok tennis shoes are
comfortable on a hot day_ , which has individual aspects of color, brand, and
type all bundled up into one side of the implication in an ambiguous manner.

###  3.3 Atomic Attributes and Attribute Types

While attributes are simply short descriptors of a product, in order to
provide more finegrained semantic annotations, we consider the idea of an
_atomic attribute_.

An atomic attribute is an attribute that further expresses only one concept of
one type. Our coarse attributes may correspond to multiple atomic attributes,
a mapping for which is provided in our dataset. Like coarse attributes, each
atomic attribute comes with a summary, however unlike for coarse attributes,
the summary might only convey the full information of the attribute in
combination with the attributes _type_ , which is also provided.

For example, in the golf-related snippet _wedges tend to have the largest loft
angles in a club set_ , the attributes _wedges_ and _largest loft angles_
could be normalized into one choice of atomic attributes as _golf club type =
wedge_ and _loft angle size = largest_ (where we write _type = summary_). This
gives us an annotation methodology to provide powerful semistructured open
schemas that stand a good chance of mapping multiple source strings to the
same attributes.

###  3.4 Atomic Implications

An atomic implication is an implication where both sides, antecedent and
consequent, are atomic attributes. While the experiments in this paper are
conducted only with respect to the atomic implications, we are excited to use
this dataset to learn structures for more complex implications between
compound attributes in future work (such as those found in Figure 2), and to
see what the community does with this data as well.

Statistics for each type of annotation in the dataset are found in Table 1,
and a full annotated example is found in Figure 2.

##  4 Data Collection

Figure 3: Annotation workflow for raters.

The data were collected over a period of about six months from a team of 5-10
skilled professional annotators working on a variety of projects. Due to the
inherent difficulty and ambiguity in providing these kind of semantic
annotations, as well as finding candidate sentences to annotate, we conducted
multiple rounds of annotation and correction cycles. What we describe here is
an abridged version which captures the essential process, an overview of which
is given in Figure 3.

###  4.1 Candidate Generation

Finding candidate sentences that contain implications between shopping
attributes, with limited annotation resources, suffers from a cold-start
problem. We chose to focus our efforts on _buying guides_ , web pages that
compare and contrast products in various categories and offering
recommendations on the best ones to purchase. We use a nonpublic corpus of
candidate buying guide URLs, each automatically annotated with a product
category from the publicly available Google Product Category taxonomy, and
then further filter down this set of URLs with a regular expression 111We ue
the following regex:  in order to create a simple buying guide corpus.

We split the pages into short sentences/snippets with a simple regex, took a
sample of 500 sentences from these webpages, along with the category
information, and asked raters to label them as containing an implication
between attributes in that category (details given later). Using these 500
labels, we finetuned a pre-trained T5-like language model as a classifier to
generate the rest of the candidates, sampling from the corpus a 70:30 ratio of
sentences classified as containing an implication to sentences classified as
not. During further rounds of data collection we continued to retrain the
classifier in order to improve candidate generation.

###  4.2 Coarse Annotation of Implications

Given the set of category-annotated candidate snippets, we first asked
annotators to annotate coarse, potentially compound, attributes and summaries
of those attributes, as well as any implications between them. We asked the
annotators to try their best to make the attributes atomic, except when making
them compound was necessary to properly describe either side of an
implication. We iterated on this task for a few rounds providing feedback, but
keeping the annotation structure and documents quite similar, only adding
examples of better and worse annotations. Relevant annotator instructions are
provided in Appendix A.

###  4.3 Fine-grained Annotation of Atomic Attributes

We followed up by dividing those potentially compound attributes into atomic
attributes, with summaries and attribute type annotations, and describing
which of the previously annotated attributes each of the atomic attributes
might be a sub-attribute of. Relevant instructions are provided in Appendix B.

##  5 Experiments

Figure 4: seq2seq encoding for attribute discovery and implication parsing.
Discovered attributes are encoded as ``tag == summary >> facet| and
implications are encoded as ``attribute -> attribute|. Multiple annotations
are separated with ``##|.

We conduct a holistic evaluation of our annotation dataset by training a
parser and constructing a small knowledge base. Evaluating so called _cold-
start_ knowledge bases with open schemas of entities and/or relations is well
known to be quite difficult. To this end, we employ human raters to evaluate
the precision of a small open KB of attribute implications created using a
model trained on our data.

First, we restrict ourselves to set of annotated atomic attributes, and the
set of _atomic implications_ , implications whose left and righthand sides
correspond to only a single atomic attribute. We use these data to create
training examples for several different tasks centered around predicting
various aspects of attributes, summaries, types, and implications. The two
relevant tasks are:

  * •

Attribute discovery: given a text snippet and a product category, produce the
full set of atomic attributes contained in that snippet relevant to the
product category, along with a summary and a type for each attribute.

  * •

Implication parsing: given a text snippet and a product category, produce the
set of implications between atomic attributes contained in that snippet
relevant to the product category, along with a summary and a type for each.

For a base model, we choose the publicly available 20B parameter UL2 language
model (Tay et al., 2022) implemented in the T5X and SeqIO frameworks (Roberts
et al., 2022).

We use a seq2seq encoding of our tasks using the mT5 vocabulary, which we
demonstrate in Figure 4. For multiple attributes or implications, we choose
the order according to a fuzzy order of appearance in the input string given
by a partial string overlap function. We use simple unconstrained greedy
decoding to perform inference, and parse the outputs using a regular
expression. Malformed spans of the output will simply fail to parse with the
regex, and result in no annotation being produced for that span, but we find
this does not happen in practice.

We finetune the model on our task mixture until the IID eval set converged,
and then run inference with the attribute discovery and implication parsing
tasks on a subsample of 1000000 sentences from the corpus. In order to
construct our demonstration knowledge base, we aggregate these parses in two
ways. For our parsed attributes, we discard the raw attributes and consider
the attribute summary along with the type to be the canonical attribute: two
attributes are the same if they have the same summary and type. For parsed
implications, we consider two implications to be the same if both their left-
and righthand sides match.

For our first experiment, we aggregate the parsed implications over the entire
corpus and consider the count of times the implication appears as its
confidence score, and take the top 100 most confident implications. We call
these _parsed implications_.

Figure 5: Counts of rater evaluations for common implication candidates
generated directly by the parser, or by PMI between parsed attributes.
Candidates generated by the parser have stronger directionality and precision.
(rating of ``A->B| is $1$, rating of ``B->A| is otherwise).

Category | Antecedent | Consequent
---|---|---
Type | Attribute Summary | Type | Attribute Summary
Pillows | material | latex | naturality | natural
Bed Sheets | material | flannel | warmth | warm
Vitamins & Supplements | nutrient | calcium | benefit | bone health
Shoes | upper material | mesh | durability | durable
Bicycles | material | aluminum | features | rust-resistant

Table 2: Example of aggregated implications extracted from the corpus by the
parser, judged by raters to have strong directionality (rating of ``A->B| is
$1$, rating of ``B->A| is otherwise).

For our second experiment, we aggregate counts of attributes and pairs of
attributes, and calculate the pointwise mutual information between all pairs
of attributes. This relationship does not discriminate directionally, but
still captures many interesting implications. We call these _PMI
implications_.

To evaluate the precision of these knowledge bases without any notion of a
ground truth schema, we employ an additional set of raters. The raters are
asked to judge the quality of the parsed implications $A\to B$ as well as the
reverse implication $B\to A$ using a Likert-style scale:

Instructions — Implication precision Two properties, A and B, can describe a
product in category C. What is the most likely relationship out of the
following? 1. Given A, B must necessarily be true. 2. Given A, B is likely to
be true. 3. A and B are usually not related. 4. Given A, B is not likely to be
true. 5. I don’t have enough information or background.

We provide the 100 examples from the parser and 100 from PMI to 9 raters, and
duplicate each example 3 times to get multiple ratings for each. Because this
is a difficult evaluation task, we selected only 4 of the 9 raters to include
in our final evaluation, based on their understanding of some control
questions for which we judged the ground truth as particularly clear. This
left us with 100 annotations for the PMI implications, and 151 for the parsed
implications.

We present the results of these experiments in Figure 5. Blue bars indicate
the judgments of raters of implications in one direction (in the case of
parsing, this is direction indicated by the model), and orange bars indicate
the judgments in the opposite direction of the model. In the case of PMI
examples we can see that the implications learned are generally judged to be
bidrectional, as would be expected. In the case of the parsed examples we see
significantly more directionality for the implications labeled with option 1,
that is, the judgment that if $A$ is true, $B$ must be true. For implications
labeled with option 2, we see more of a lack of directionality, possibly
because the judgment _Given A, B is likely to be true_ can be quite common in
both directions for a variety of attributes of consumer products (e.g.
attributes like _lightweight_ , _portable_ , _breathable_ , _comfortable_ ,
etc), and we are evaluating specifically the most commonly found implications
in the corpus.

We provide examples in Table 2 of some of the implications found by the parser
which have the most directionality according to the raters. While both the
parser and PMI method appear able to learn many associations, valuable
information for our open-schema knowledge base, the parser is higher precision
and can further find reasonable directional implications between product
attributes.

##  6 Conclusion and Future Work

In this work, we introduce a new dataset for training models to perform open-
schema knowledge base construction in the shopping domain. We annotate
thousands of free text types, attributes, and implication relationships. We
demonstrate that high-precision implication relationships from text can be
annotated using a UL2 language model finetuned on tasks from our corpus, both
directly parsing the implications, and looking at PMI between pairs of parsed
attributes.

We release this data to the community in hope that researchers will use it to
study not just shopping attribute and relation annotations, but also other
domains that benefit from fine-grained free text, semistructured data, and
open-schemas, such as commonsense knowledge. More broadly, we advocate for
practitioners to take advantage of the tremendous power of large language
models for semantic parsing, and to construct ambitious fine grained knowledge
bases in a variety of domains.

## References

  * Auer et al. (2007) Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007.  Dbpedia: A nucleus for a web of open data.  In _Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference_ , ISWC’07/ASWC’07, page 722–735, Berlin, Heidelberg. Springer-Verlag.
  * Banko et al. (2007) Michele Banko, Michael J. Cafarella, Stephen Soderland, Matt Broadhead, and Oren Etzioni. 2007.  Open information extraction from the web.  In _Proceedings of the 20th International Joint Conference on Artifical Intelligence_ , IJCAI’07, page 2670–2676, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Baroni et al. (2012) Marco Baroni, Raffaella Bernardi, Ngoc-Quynh Do, and Chung-chieh Shan. 2012.  Entailment above the word level in distributional semantics.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 23–32.
  * Battaglia et al. (2018) Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George E. Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew M. Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. 2018.  Relational inductive biases, deep learning, and graph networks.  _CoRR_ , abs/1806.01261.
  * Bodenreider (2004) Olivier Bodenreider. 2004.  The unified medical language system (umls): integrating biomedical terminology.  _Nucleic acids research_ , 32(suppl_1):D267–D270.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.  _arXiv preprint arXiv:1508.05326_.
  * Brahman et al. (2022) Faeze Brahman, Baolin Peng, Michel Galley, Sudha Rao, Bill Dolan, Snigdha Chaturvedi, and Jianfeng Gao. 2022.  Grounded keys-to-text generation: Towards factual open-ended generation.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Cao et al. (2020) Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2020.  Autoregressive entity retrieval.  _CoRR_ , abs/2010.00904.
  * Dagan et al. (2013) Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013.  Recognizing textual entailment: Models and applications.  _Synthesis Lectures on Human Language Technologies_ , 6(4):1–220.
  * Das et al. (2017) Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, Luke Vilnis, Ishan Durugkar, Akshay Krishnamurthy, Alex Smola, and Andrew McCallum. 2017.  Go for a walk and arrive at the answer: Reasoning over paths in knowledge bases using reinforcement learning.  _arXiv preprint arXiv:1711.05851_.
  * Das et al. (2016) Rajarshi Das, Arvind Neelakantan, David Belanger, and Andrew McCallum. 2016.  Chains of reasoning over entities, relations, and text using recurrent neural networks.  _arXiv preprint arXiv:1607.01426_.
  * Google (2022) Google. 2022.  Freebase data dumps.  https://developers.google.com/freebase/data.
  * Gu et al. (2022) Yu Gu, Vardaan Pahuja, Gong Cheng, and Yu Su. 2022.  Knowledge base question answering: A semantic parsing perspective.
  * Hao et al. (2022) Shibo Hao, Bowen Tan, Kaiwen Tang, Hengzhe Zhang, Eric P Xing, and Zhiting Hu. 2022\.  Bertnet: Harvesting knowledge graphs from pretrained language models.  _arXiv preprint arXiv:2206.14268_.
  * Hearst (1992) Marti A. Hearst. 1992.  Automatic acquisition of hyponyms from large text corpora.  In _COLING 1992 Volume 2: The 14th International Conference on Computational Linguistics_.
  * Hsu et al. (2022) I-Hung Hsu, Kuan-Hao Huang, Elizabeth Boschee, Scott Miller, Prem Natarajan, Kai-Wei Chang, and Nanyun Peng. 2022.  Degree: A data-efficient generation-based event extraction model.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1890–1908.
  * IV et al. (2017) Robert L. Logan IV, Samuel Humeau, and Sameer Singh. 2017.  Multimodal attribute extraction.  _CoRR_ , abs/1711.11118.
  * Lenat (1995) Douglas B Lenat. 1995.  Cyc: A large-scale investment in knowledge infrastructure.  _Communications of the ACM_ , 38(11):33–38.
  * Li et al. (2016) Xiang Li, Aynaz Taheri, Lifu Tu, and Kevin Gimpel. 2016.  Commonsense knowledge base completion.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1445–1455.
  * Li et al. (2021) Xiang Lorraine Li, Adhi Kuncoro, Cyprien de Masson d’Autume, Phil Blunsom, and Aida Nematzadeh. 2021.  A systematic investigation of commonsense understanding in large language models.  _arXiv preprint arXiv:2111.00607_.
  * Lu et al. (2022) Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022.  Unified structure generation for universal information extraction.  _arXiv preprint arXiv:2203.12277_.
  * MacCartney (2009) Bill MacCartney. 2009.  _Natural language inference_.  Stanford University.
  * Mitchell and Fredkin (2014) Tom Mitchell and E. Fredkin. 2014.  Never-ending language learning.  In _2014 IEEE International Conference on Big Data (Big Data)_ , pages 1–1.
  * Narayan et al. (2021) Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simoes, Vitaly Nikolaev, and Ryan McDonald. 2021.  Planning with learned entity prompts for abstractive summarization.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Raman et al. (2022) Karthik Raman, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, and Krishna Srinivasan. 2022.  Transforming sequence tagging into a seq2seq task.
  * Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013.  Relation extraction with matrix factorization and universal schemas.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 74–84, Atlanta, Georgia. Association for Computational Linguistics.
  * Roberts et al. (2022) Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, Curtis Hawthorne, Aitor Lewkowycz, Alex Salcianu, Marc van Zee, Jacob Austin, Sebastian Goodman, Livio Baldini Soares, Haitang Hu, Sasha Tsvyashchenko, Aakanksha Chowdhery, Jasmijn Bastings, Jannis Bulian, Xavier Garcia, Jianmo Ni, Andrew Chen, Kathleen Kenealy, Jonathan H. Clark, Stephan Lee, Dan Garrette, James Lee-Thorp, Colin Raffel, Noam Shazeer, Marvin Ritter, Maarten Bosma, Alexandre Passos, Jeremy Maitin-Shepard, Noah Fiedel, Mark Omernick, Brennan Saeta, Ryan Sepassi, Alexander Spiridonov, Joshua Newlan, and Andrea Gesmundo. 2022.  Scaling up models and data with t5x and seqio.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_.
  * Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007.  Yago: A core of semantic knowledge.  In _Proceedings of the 16th International Conference on World Wide Web_ , WWW ’07, page 697–706, New York, NY, USA. Association for Computing Machinery.
  * Sun and Zhang (2018) Yueming Sun and Yi Zhang. 2018.  Conversational recommender system.  In _The 41st international acm sigir conference on research & development in information retrieval_, pages 235–244.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022.  Unifying language learning paradigms.  _arXiv preprint arXiv:2205.05131_.
  * Tunkelang (2009) Daniel Tunkelang. 2009.  Faceted search.  _Synthesis lectures on information concepts, retrieval, and services_ , 1(1):1–80.
  * Vinyals et al. (2015) Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015.  Pointer networks.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_.
  * Yang et al. (2022) Li Yang, Qifan Wang, Zac Yu, Anand Kulkarni, Sumit Sanghai, Bin Shu, Jon Elsas, and Bhargav Kanagal. 2022.  Mave: A product dataset for multi-source attribute value extraction.  In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ , pages 1256–1265.
  * Yu et al. (2022) Changlong Yu, Weiqi Wang, Xin Liu, Jiaxin Bai, Yangqiu Song, Zheng Li, Yifan Gao, Tianyu Cao, and Bing Yin. 2022.  Folkscope: Intention knowledge graph construction for discovering e-commerce commonsense.

Appendix

##  Appendix A Implication parsing annotator instructions

We provided raters with the following instructions for annotating
implications:

> We use the nomenclature of attributes and types. A attribute is a specific
> value that describes a product, for example “red” or “affordable”. A type is
> a key (i.e. a name) that describes a collection of related attributes. For
> example, the attribute “red” belongs to the type of “color,” and the
> attribute “affordable” belongs to the type of “price.” In this task, we will
> ask you to identify the attributes in a snippet of text relevant to a
> specific provided product category that contain an _Implies_ relationship
> between them, as evidenced by the snippet. We wish to determine if a snippet
> is telling us that one attribute implies another, from information contained
> in that snippet. For example, “This beer is crisp and refreshing” does not
> tell us that there is any implication between the attributes “crisp” and
> “refreshing”, nor does it imply that all beers are crisp and refreshing, so
> we would mark this as “No implication”. However, “This computer has a fast
> GPU which is great for gaming” does tell us that the attribute “fast GPU”
> implies the attribute “great for gaming”. It should already be clear that
> there is a degree of fuzzy logical ambiguity here, since maybe not all fast
> GPUs are great for gaming, but in the context of the sentence this
> implication holds. Extracting attributes: The first step is to identify the
> relevant implication-participating attributes mentioned in the provided
> snippet. In some cases there is ambiguity as to what the attribute is
> exactly, for example “the jacket is waterproof” could have the attribute “is
> waterproof” or just simply “waterproof”. Either choice is okay, but as a
> rule of thumb, the shortest text that captures the full meaning of the
> attribute is preferable. For example, “the refrigerator has cheap
> maintenance” might have the attribute “cheap maintenance,” as shortening
> this further to just “cheap” or “maintenance” loses the meaning of the
> attribute. Although a product category is provided, we would like to extract
> all attributes relevant to implications, regardless of category. For
> example, in the category of “Shoes”, given the snippet “These neutral color
> shoes look great with my red cocktail dress or other formal attire”, we
> could extract “neutral color” and “look great with red cocktail dress or
> other formal attire” as attributes related to the shoes that have an implied
> relationship. But we would also want to extract “cocktail dress” and “formal
> attire” because of the implied relationship between these two attributes
> even though the dress itself is not relevant to the category “Shoes”. The
> categories are only provided to give context in understanding the snippet,
> not to limit the scope of possible attributes and implications. When
> generating attributes, they can be copied directly (from the provided
> snippet) when the copy makes sense on its own. However, in some cases, it
> may not make sense to directly copy; in this case, please add in the
> relevant text to make the attribute make sense on its own. For example, when
> generating numeric attributes, do your best to include units, even if they
> are not directly found in the text; in the phrase “The two possible screen
> sizes are 11 and 13 inches”, there should be a attribute for “11 inches” and
> one for “13 inches.” Paraphrasing attributes: Sometimes you will need to
> paraphrase rather than copy the text directly.  • Example 1: “I’m not going
> to write about the great tones and playability of this guitar, because its
> plays and sounds like any other $200 guitar on the market.” In this case,
> you would not want the attribute to be “great tones” since the user is
> essentially saying that the tones are average and so is the playability. So
> go ahead and paraphrase the attributes as “Average tones” and “Average
> playability” and mark an X in the Paraphrase column. • Example 2: Be concise
> when paraphrasing, but aim to keep the specificity of the original text.
> Text passage: That said, there are currently two major disadvantages we’ve
> experienced with some of the Mirrorless cameras we’ve encountered so far.
> The first is the inability of many models to match the autofocus speed of
> DSLRs when conducting continuous or predictive AF Rather than “slower
> autofocus”, the attribute should be more specific and therefore include more
> information: “slower autofocus speed than DSLRs when conducting continuous
> or predictive AF” is the preferred attribute. Do not alter the text snippets
> in the text column. However, if you’re extracting/paraphrasing a attribute
> with a misspelling, please spell the word(s) correctly in the attribute
> column. Marking implications: In this task, we are interested in finding
> implication relations between attributes. Given a snippet, we want to
> establish whether the text of the snippet itself (along with some basic
> “commonsense” about the product category and any other context that an
> average person could be expected to know) gives reason to believe one of
> these implications holds between the attributes. For two attributes A and B
> from snippet S, the existence of the relation Implies(A, B) means that given
> the information in S, we can infer that the attribute A implies the
> attribute B, perhaps not in a strict logical sense, but in context. For
> example, the sentence “This computer has a fast GPU which is great for
> gaming” would give us the relevant (implication-participating) attributes
> “fast GPU” and “great for gaming”, along with the relation Implies(fast GPU,
> great for gaming). This might not be a strict logical implication (what if
> the fast GPU overheats and makes it hard to play games?), but in the context
> of the sentence, this implication holds. “Is-”: A subset of implications
> come about strictly because of a straightforward is-a / sub-tag relationship
> between the attributes. For example, the snippet “Wool is very good for
> casual wear as it doesn’t smell and merino wool is very soft and
> comfortable” could give us the relevant attributes “wool”, “good for casual
> wear”, “doesn’t smell”, “soft” and “comfortable”, along with the relations
> Implies(wool, good for casual wear), Implies(wool, doesn’t smell),
> Implies(merino wool, soft), Implies(merino wool, comfortable), and
> Implies(merino wool, wool). This latter implication, Implies(merino wool,
> wool), can be inferred from the sentence, and you should feel free to mark
> it, but if it becomes distracting to figure out too many of these implied
> taxonomic sort of relationships, do not worry too much about missing them.
> Implied Negations: Some snippets will imply a negation of one of the
> attributes, for example, in a snippet like “The priciest jeans are made with
> high-quality cotton, while bargain brands will often use a synthetic mix”,
> we can infer that “high-quality cotton” implies not “synthetic mix” in the
> context of clothing and jeans. We can denote this as Implies(high-quality
> cotton, NOT synthetic mix) with the “NOT” capitalized. The capitalization
> denotes that we added this “NOT” as an inference step, as opposed to seeing
> the “not” directly in the text, but if you are confused between whether the
> “NOT” should be capitalized, it is not very important and it is ok to make
> it lower case. Implicit ANDs vs implicit ORs: in the example above, we saw
> that a group of attributes were implicitly related by an OR in the context
> of the implication. This is not always the case though, sometimes a whole
> group of things are required to be present at once in order for the
> implication relationship to hold. For example, given the snippet “For summer
> season, a muslin cotton swaddling blanket is always a perfect choice”, we
> can infer that a “swaddling blanket” is not always a perfect choice for
> summer, it must be a “muslin cotton swaddling blanket”. So we could write
> the implication Implies(muslin cotton swaddling blanket, perfect for summer
> season). It would also be acceptable to write Implies(muslin AND cotton AND
> swaddling blanket, perfect for summer season), but breaking out the “logical
> operator” like this is not nearly as important in the case of ANDs as it is
> in the case of ORs, since in the latter we are using OR as simply a way to
> concisely write down a whole group of relations and make annotation easier.
> Bad examples and “no implications”: There are two ways for a given snippet
> to not have any useful implications to extract. In one case, the snippet
> might be a meaningful sentence but simply not give evidence of any
> implication relationship. In that case, we mark the example as “No
> implication”. Some snippets, on the other hand, are just “bad examples”,
> meaning that they are malformed in some way such as not being meaningful
> sentences. A common way for that to happen is with product titles — our data
> is extracted automatically and some of the extracted snippets are simply
> titles of products, like “2019 New Trend Fashion Summer Solid Color Shirt
> Men Stand Collar Cotton Linen Half Sleeve Shirt Slim Type Male Casual Linen
> Tops”. This should simply be marked as “Bad example”. In early iterations of
> this task, there might be a fair amount of bad examples!

In addition to these instructions, we provided the raters with demonstrations
of the task. We include a few below as they were presented to the annotators:

> Example 1: • Category is “Socks” • Snippet is “This combination makes the
> fabric breathable, and durable with a comfortable grip” • Annotations: No
> implication Example 2: • Category is “Shirts & Tops” • Snippet is “2019 New
> Trend Fashion Summer Solid Color Shirt Men Stand Collar Cotton Linen Half
> Sleeve Shirt Slim Type Male Casual Linen Tops” • Annotations: Bad example
> Example 3: • Category is “Quilts & Duvets” • Snippet is “Egyptian cotton is
> soft and strong at the same time, making it durable.” • Annotations:
> Implies(Egyptian cotton, durable)
> Implies(Egyptian cotton, soft)
> Implies(Egyptian cotton, strong)
> Implies(soft and strong, durable) Example 4: • Category is “Tea & Infusions”
> • Snippet is “Matcha green tea is more potent than regular green tea” •
> Annotations: Implies(Matcha green tea, more potent than regular green tea)
> Example 5: • Category is “Handbags & Purses” • Snippet is “It is made of
> polyester and nylon, making it durable and lightweight” • Annotations:
> Implies(polyester, durable)
> Implies(nylon, durable)
> Implies(polyester, lightweight)
> Implies(nylon, lightweight) Note: Even though the sentence says “and” here,
> we know that in the context of clothes, everything in the fabric has to have
> these properties. If unsure, the next best thing is to just say:
> Implies(polyester and nylon, durable)
> Implies(polyester and nylon, lightweight)

##  Appendix B Attribute splitting and typing annotator instructions

We provided raters with the following instructions for splitting attributes:

> We use the nomenclature of attributes and types. A attribute is a specific
> value that describes a product, for example “red” or “affordable”. A type is
> a key (i.e. a name) that describes a collection of related attributes. For
> example, the attribute “red” belongs to the type of “color,” and the
> attribute “affordable” belongs to the type of “price.”  • The goal of this
> task is to get an exhaustive list of all the atomic attributes (attributes
> describing one simple aspect) relevant to the category contained in the
> snippet, along with type information for each attribute, and a short
> “canonical” summary of the attribute in the context of that type.
> “Canonical” here means that the summary you write in different snippets
> should be the same when describing a attribute with roughly the same
> meaning. • In this task, we will ask you to identify all of the attributes
> in a snippet of text relevant to a specific provided product category. If
> these attributes are associated with one of the provided previously
> annotated attributes, this should be specified. • It might often be the case
> that a previously annotated attribute is not atomic, i.e. it describes a
> combination of more than one simple aspect of a product. In that case we
> want to annotate all those atomic attributes while also indicating that they
> correspond to a single compound previous attribute. • In other cases,
> previously annotated attributes might include unnecessary extra text that
> becomes clear once types are annotated. In this case we also want to
> indicate that the new crisp, typed attribute corresponds to the previously
> annotated attribute.

We provided raters with the following instructions for typing attributes:

> A type is a named collection of attributes that describe a product.
> Examples:  • Attribute: Red > Type: Color • Attribute: Glass > Type:
> Material Naming a type might require world knowledge or additional searching
> the web. Some methods for type determination are listed below. If there are
> multiple applicable types, say a fine grained and a coarse grained one, list
> both comma separated.  • Read the snippet. Sometimes, the type can be found
> directly in the text. For example, to determine the type for the attribute
> “ivory” the snippet might just contain “ivory is a very traditional wedding
> dress color”. • Research the attribute on Google.com. For example, searching
> for [what is an ivory wedding dress] gives an answer that specifies that it
> is a color, not a material. • Some attributes might be very product-specific
> like “prevents skin from feeling sticky” in “this pillow prevents skin from
> feeling sticky.” In those cases, it’s ok to use a more vague type like
> “Benefits”. Vague types like “benefits” should be avoided when possible
> though. For example if a watch is listed as “waterproof”, we want a type
> like “Water-Resistance” and not simply “Benefits”, since “Water-Resistance”
> is something that people are often concerned about with watches.

We also provided raters with several demonstrations of the task, an example of
which is provided in Table 3. Because these annotations are so structured, we
provided them in a spreadsheet form.

Category |  Snippet | Previous Tag |  Previous Summary | New Facet | New Tag |  New Summary | Associated Prev. Tag
---|---|---|---|---|---|---|---
Food Mixers & Blenders |  Overall, the performance of Orient mixer grinder is good and best to use for small homes | Orient mixer grinder |  Orient mixer grinder | Performance | good performance |  good |
for small homes |  small homes | Type | mixer grinder |  mixer grinder | Orient mixer grinder
|  | Brand | Orient |  Orient | Orient mixer grinder
|  | Use Case | for small homes |  small homes | for small homes
Table 3: Example annotation for splitting and typing task, as raters would see
it.