# Differential imprints of distinct ENSO flavors in global extreme
precipitation patterns

###### Abstract

The specific impacts of El Niño’s two flavors, East Pacific (EP) and Central
Pacific (CP) El Niño, have been studied intensively in recent years, mostly by
applying linear statistical or composite analyses. These techniques, however,
focus on average spatio-temporal patterns of climate variability and do not
allow for a specific assessment of related extreme impacts. Here, we use event
coincidence analysis to study the differential imprints of EP and CP types of
both, El Niño and La Niña on global extreme precipitation patterns. We provide
a detailed statistical assessment of seasonal precipitation extremes related
to the two flavors of both phases of the El Niño Southern Oscillation (ENSO)
in terms of their likelihood and spatial extent. Our analysis recaptures
previously reported interrelations and uncovers further regional extremes
arising along with different ENSO phases that have not been reported so far.

Geophysical Research Letters

Potsdam Institute for Climate Impact Research — Telegrafenberg A 31, 14473
Potsdam, Germany, EU Department of Physics, Humboldt University — Newtonstraße
15, 12489 Berlin, Germany, EU Institute of Earth and Environmental Science,
University of Potsdam, Karl-Liebknecht-Straße 24-25, 14476 Potsdam-Golm,
Germany Stockholm Resilience Centre, Stockholm University — Kräftriket 2B, 114
19 Stockholm, Sweden, EU Marc<EMAIL_ADDRESS>
Global assessment of ENSO related impacts on seasonal precipitation extremes

East Pacific El Niños and La Niñas trigger spatially coherent extreme
precipitation responses

Central Pacific events result in less spatially coherent precipitation
extremes

##  1 Introduction

The El Niño Southern Oscillation (ENSO) with its positive (El Niño) and
negative phase (La Niña) is known to trigger climatic responses in various
parts of the Earth, an effect that is often referred to as teleconnectivity
(Trenberth, 1997; Neelin et al., 2003). Specifically, it has been observed in
recent years that El Niño further exhibits two distinct types, usually
referred to as the East Pacific or canonical El Niño and the Central Pacific
El Niño or El Niño Modoki, respectively (Ashok et al., 2007; Kao and Yu,
2009). A similar proposition regarding the existence of two distinct flavors
was made recently for La Niña as well (Kug and Ham, 2011), but is a subject of
ongoing studies (Kao and Yu, 2009; Ren and Jin, 2011). For El Niño, it has
been shown that its two types may cause different climatic responses in
certain regions, such as reduced rainfall over eastern Australia during EP El
Niños (Chiew et al., 1998) contrasted by an increase in precipitation over the
same area during CP El Niños (Taschetto and England, 2009).

Most previous studies on El Niño’s teleconnective impacts have either applied
linear statistical tools, such as correlation analysis (Diaz et al., 2001), or
investigated corresponding composites of the climatic observables under study
(Hoell et al., 2014). Both approaches share the limitation that they focus on
linear or average interdependencies between ENSO and possible response
variables, which do not necessarily hold for extreme values. However, with
global climate change projected to increase the strength and frequency of
both, extreme climatic events (Jones et al., 2007; Karl and Trenberth, 2003;
Easterling et al., 2000) as well as extreme ENSO periods (Cai et al., 2014) it
has become a pressing issue to specifically assess possible linkages between
these two aspects (Allan and Soden, 2008). Therefore, we devote the present
work to quantify and spatially resolve signatures of extreme climatic events
that are likely to coincide with an ENSO phase of a certain type. In
particular, we focus here on heavy precipitation and dry periods, i.e., hydro-
meterological extreme events, as ENSO has been shown to largely affect
rainfall patterns at both global and regional scales (Dai and Wigley, 2000;
Ropelewski and Halpert, 1987).

The main goal of this study is to quantify the likelihood of simultaneous or
time-delayed occurrences of localized precipitation extremes along with an
ENSO phase of a given type. For this purpose, we employ event coincidence
analysis (ECA) (Donges et al., 2011, 2016) for estimating the probability of
co-occurrences between such events. This framework has already been
successfully applied to quantifying the likelihood of climatic extreme events
triggering certain ecological or social responses, such as extreme annual
(Rammig et al., 2015) and daily (Siegmund et al., 2016a) tree growth or
flowering dates (Siegmund et al., 2016b) as well as the outbreak of epidemics
(Donges et al., 2016) or violent conflicts (Schleussner et al., 2016).
Acknowledging the already observed differential impacts of EP and CP El Niños,
we aim here to further characterize specific extreme impacts of the two
flavors of both, El Niño and La Niña.

Several schemes to distinguish EP from CP events have been proposed in the
recent past. One prominent example is the ENSO Modoki Index that is computed
as the weighted average sea surface temperature in the equatorial Pacific
(Ashok et al., 2007). Other studies used empirical orthogonal functions (Graf
and Zanchettin, 2012; Kao and Yu, 2009) or combinations of the Nino3 and Nino4
indices (Kim et al., 2011; Hu et al., 2011) to provide the desired
discriminations. In contrast to statistics related solely to the dynamics
within the equatorial Pacific, Wiedermann et al. (2016) recently introduced an
index to discriminate different ENSO flavors based on the assessment of global
teleconnections in surface air temperature. This index shows a more distinct
and sharp discrimination as compared to previously proposed measures based on
average temperature observations (Ashok et al., 2007). This particular
property is important for our present analysis as ECA relies on the assessment
of event sequences, which are easily computed for a spike-like signal in
contrast to a smoothly varying one.

We use the index proposed by Wiedermann et al. (2016) together with the
classical Oceanic Nino Index (ONI) (provided by the Climate Prediction Center
of the National Oceanic and Atmospheric Administration, NOAA-CPC) to construct
four event series representing EP and CP El Niño and La Niña phases. We then
utilize ECA to quantify simultaneous occurrences of either of the four ENSO
types with spatially resolved seasonal precipitation extremes (wet and dry) in
boreal fall and winter of the same year as the start of a certain ENSO
episode, and spring of the following year. The data and methods applied in
this work are presented in Sec. 2. Section 3 shows the results of our
analysis. We start by evaluating the effects of the canonical El Niño on
extreme precipitation and demonstrate the validity of our approach by
comparing the obtained spatial patterns with recent studies. We then focus on
CP El Niños and highlight differences with their EP counterparts. Ultimately,
we study La Niña periods and show that most extreme rainfall responses are
attributed to the EP type of ENSO’s negative phase. We thus conclude that in
the light of recent discussions on the existence of two types of La Niña (Kug
and Ham, 2011) it is meaningful to distinguish one type that significantly
affects global precipitation signals and one that shows hardly any spatially
coherent impact. Finally, Sec. 4 provides the conclusions of this work as well
as suggestions for future research.

##  2 Data & Methods

###  2.1 GPCC rainfall data

We utilize gridded monthly precipitation data provided by the Global
Precipitation Climatology Centre (GPCC) at a spatial resolution of
$2.5^{\circ}\times 2.5^{\circ}$ (Schneider et al., 2015). Since reliable
estimations of El Niño and La Niña periods according to the ONI as well as a
discrimination into their respective EP and CP flavors are only available for
the second half of the 20th century so far, we restrict our analysis to the
period from 1951 to 2015\. Note that the density of stations from which the
data is derived varies between 0 and more than 100 per grid cell and for
different years (Lorenz and Kunstmann, 2012), which generally results in a
lower accuracy and reliability of the data for those areas with only few
stations (Rudolf et al., 1994). We therefore consider only grid cells where
the average number of stations between 1951 and 2010 is larger than 1,
yielding a total number of $N=2248$ cells. From the monthly time series, we
derive separate records for three seasons $s$ by aggregating the precipitation
amounts of the corresponding three-month periods September to November (SON),
December to February (DJF) and March to May (MAM). This results in three time
series $P_{i,s}(t)$ per grid cell $i$ with $M=62$ annual values each.

###  2.2 Network transitivity index of ENSO flavor

Figure 1: (A) Network transitivity index indicating EP El Niños or La Niñas if
its value exceeds the baseline (dashed line), and CP phases otherwise. Red and
blue shaded areas indicate periods of El Niño and La Niña according to the
ONI. (B) Corresponding series of EP (solid) and CP (dashed) El Niño (red) and
La Niña (blue) events, respectively.

Previous approaches to distinguish different flavors of El Niño (Hendon et
al., 2009) like empirical orthogonal function (EOF) analysis (Graf and
Zanchettin, 2012) or the ENSO Modoki Index (EMI) (Ashok et al., 2007) often
yielded ambiguous or mutually inconsistent results (Wiedermann et al., 2016).
Radebach et al. (2013) were the first to observe that characteristics related
to the global correlation structure of surface air temperatures expressed as a
complex network can provide sharp discriminators between EP and CP El Niños.
Inspired by this approach, Wiedermann et al. (2016) recently developed an
index based on climate network transitivity $\mathcal{T}$, a measure to
quantify the degree of localization of ENSO’s global teleconnections,
displaying different spatial characteristics for EP and CP flavors of El Niño
and La Niña.

The transitivity index is computed upon daily global surface air temperature
anomalies (SATA) from the NCEP/NCAR reanalysis (Kistler et al., 2001) used to
construct a sequence of 365-days long running-window cross-correlation
matrices $\textbf{C}_{n}=(C_{n,ij})$ between all pairs of time series in the
data set. The most relevant information captured by $C_{n}$ is contained in
the $0.5\%$ of strongest absolute correlations for each window $n$ and
expressed using thresholds $T_{n}$ such that

$W_{n,ij}=|C_{n,ij}|\cdot\Theta(|C_{n,ij}|-T_{n}),$ (1)

with $\Theta(\cdot)$ denoting the Heaviside function. One major characteristic
of the resulting matrix $W_{n}$ is the transitivity index

$\mathcal{T}_{n}=\frac{\sum_{i,j,k}w_{i}W_{n,ij}w_{j}W_{n,jk}w_{k}W_{n,ki}}{\sum_{i,j,k}w_{i}W_{n,ij}w_{j}W_{n,jk}w_{k}}\in[0,1]$
(2)

with

$\quad w_{i}=\cos(\lambda_{i}).$ (3)

Here, $\lambda_{i}$ denotes the latitudinal position of each time series’
corresponding grid point on the Earth’s surface (Heitzig et al., 2012; Tsonis
et al., 2006). A comprehensive description of the above framework is given in
Wiedermann et al. (2016).

Figure 1A shows the temporal evolution of the transitivity index. In contrast
to the EMI, it provides a sharp and distinct discrimination between the two El
Niño flavors (and also two corresponding types of La Niña phases), such that a
peak in the index indicates an EP type while its absence points to a CP event.
A corresponding baseline $\underline{\mathcal{T}}$ (dashed horizontal line in
Fig. 1A) above which values of $\mathcal{T}$ are considered a peak is given as
well. It is derived as the transitivity computed over the same 30-year periods
that are used to determine the base periods for the computation of sea surface
temperature anomalies in the definition of the ONI.

###  2.3 Data preprocessing

We define years with seasons $s$ (DJF, SON, or MAM, see Sec. 2.1) exhibiting
extraordinary high or low precipitation amounts from the corresponding time
series $P_{s,i}(t)$ for each grid cell $i$, individually. Specifically, we
consider values above (below) the 80th (20th) percentile $p_{s,i}^{+}$
($p_{s,i}^{-}$) in each of the time series $P_{s,i}(t)$ as extraordinary high
(low) seasonal precipitation sums. We choose these particular thresholds to
ensure the presence of a sufficient number of particularly dry and wet seasons
that is comparable with the number of different ENSO phases in the considered
time period. It has been checked that the results obtained in the following do
not change qualitatively if more restrictive thresholds are applied (not
shown).

According to these considerations, we obtain six binary time series
$P^{\pm}_{s,i}(t)$ per grid cell,

$P^{\pm}_{s,i}(t)=\Theta(\pm P_{s,i}(t)\mp p^{\pm}_{s,i}),$ (4)

where $P^{\pm}_{s,i}(t)=1$ indicates the presence of a seasonal precipitation
extreme at grid cell $i$ during season $s$ in year $t$. By following the above
procedure, no further de-seasonalisation of the precipitation data is
necessary, since the grid cell-specific seasonality of precipitation is
already taken into account. Furthermore, the events are defined for each
location independent from the others and therefore spatial heterogeneity is
taken into account as well.

In addition to the precipitation data set, we create similar binary indicator
time series for the different ENSO phases (Fig. 1B), e.g. an EP El Niño series
$X_{EPEN}(t)$ with $X_{EPEN}=1$ if $t$ marks the onset year of an El Niño
period (as defined by the ONI) and $\mathcal{T}_{n}$ exceeds
$\underline{\mathcal{T}}$ for at least one month of that considered period.
All remaining El Niño periods are classified as CP events, yielding a
corresponding event series $X_{CPEN}(t)$ (solid and dashed red lines in Fig.
1B). The same procedure is applied to La Niña periods, yielding event series
$X_{EPLN}(t)$ and $X_{CPLN}(t)$, respectively (blue lines in Fig. 1B).

###  2.4 Event Coincidence Analysis

We apply event coincidence analysis (ECA), a statistical tool to quantify
simultaneities between events in two series (Donges et al., 2011; Rammig et
al., 2015; Donges et al., 2016). It computes for each grid cell the fraction
of EP (CP) El Niño (La Niña) phases that coincide with extreme precipitation
sums in SON or DJF of the same year or MAM of the following year, revealing if
the timing of precipitation extremes is non-randomly related to the presence
of a given type of ENSO phase. Specifically, the event coincidence rate
$ECR^{\pm}_{s,i,\bullet}$ for one pair of ENSO and precipitation event series
is given by

$ECR^{\pm}_{s,i,\bullet}=\frac{\sum_{t}X_{\bullet}(t)P\pm_{s,i}(t-\tau)}{\sum_{t}X_{\bullet}(t)}.$
(5)

Here, $X_{\bullet}(t)$ represents one of the four time series indicating EP
and CP types of El Niño and La Niña. The offset $\tau$ reads $\tau=0$ for SON
and DJF and $\tau=1$ for MAM.

To assess the statistical significance of the obtained event coincidence
rates, we assume both involved event sequences to be distributed randomly,
independently and uniformly (Donges et al., 2011; Rammig et al., 2015;
Siegmund et al., 2016b, a). A corresponding $p$-value is estimated
analytically from the (binomial) probability distribution of event coincidence
rates that would occur by chance only. We consider an empirical event
coincidence rate as statistically significant if its $p$-value is smaller than
a confidence level of $\alpha=0.05$ (for mathematical details, see Donges et
al. (2016)).

##  3 Results & Discussion

###  3.1 Seasonal precipitation extremes and EP El Niño

Figure 2: Statistically significant event coincidence rates between years of
EP (left column) and CP (right column) El Niños and very dry or very wet
seasons (SON, DJF, MAM). Red (blue) squares represent grid cells with
significant event coincidence rates between EP or CP El Niño years and
seasonal precipitation sums below (above) the 20th (80th) percentile of all
years from 1951 to 2015.

We first investigate event coincidences rates between EP El Niños and the
timing of seasonal precipitation extremes. Figures 2A,C,E show locations with
significant coincidence rates $ECR^{\pm}_{\bullet,i,ENEP}$ between EP El Niños
and extremely dry (red squares) and wet periods (blue squares) in SON, DJF and
MAM, respectively.

During SON of EP El Niño years, we find an elevated probability of extremely
dry conditions over Indonesia, the Philippines and the southwestern Pacific
islands as well as over northern South America and the northern Amazon Basin
(Fig. 2A). For the same season, we also observe an increased likelihood of
very wet conditions over the central Pacific islands and the west coast of
North America. (Fig. 2A). Similarly wet conditions are found over Ecuador and
the South American east coast in SON and DJF (Fig. 2A,B). All these
observations agree well with previous studies (e.g. Diaz et al., 2001).
Coinciding with EP El Niños, we also observe more frequent wet extremes over
the Mediterranean region (Fig. 2A) in SON, which was partially also reported
earlier by Shaman and Tziperman (2010). Furthermore, the wet conditions over
East Africa during the same season (Fig. 2A) have previously been described in
parts by Camberlin et al. (2001). Finally, we observe very dry conditions in
southwestern Africa during DJF (Fig. 2C) as also reported by Hoell et al.
(2014).

Generally, our ECA results are in good agreement with previously reported
interrelations between El Niño and global precipitation, which have mostly
been identified using linear correlation analysis. Thus, we conclude that (i)
the application of ECA to assess ENSO-related extreme impacts provides valid
results, and that (ii) extreme responses to the canonical El Niño display
roughly similar spatial patterns as the average statistical interdependency
between ENSO-related indices and climatic observables. However, we note that
most previous studies have not discriminated between the two El Niño flavors.
Thus, the agreement between our results for EP El Niños and the recent
literature suggests that the previously observed linear effects might just be
dominated by the (on average stronger) EP events.

###  3.2 Seasonal precipitation extremes and CP El Niño

Next, we focus on the specific effects of the less intensively studied CP El
Niño and estimate event coincidence rates between these phases and seasons
with extreme precipitation sums (Fig. 2B,D,E). We first discuss those regions
that display significant coincidence rates for EP El Niños (see Sec. 3.1) but
an absence or altered likelihood of precipitation extremes during CP periods.
Notably, the dry events over Central America and the Amazon that frequently
occur together with EP El Niños become more likely during DJF (Fig. 2D) but
show insignificant coincidence rates in SON (Fig,. 2B). The latter also holds
for the wet events along the western coast of Central and North America that
have been observed for EP El Niños. In the same manner, the wet SON patterns
over Southern China, the Mediterranean, and South-East Africa coinciding with
EP El Niños disappear during CP events (Fig,. 2B). We further note, that the
large-scale dry events over Indonesia observed along with EP events during SON
become less spatially coherent for CP El Niños (Fig. 2B). For MAM, the wet and
dry patterns over Europe become insignificant (Fig. 2F).

While the above observations indicate decreased or weakened impacts of CP El
Niños in comparison with the EP type, we also observe the emergence of new
patterns that are not present during EP El Niños but emerge only along with CP
phases. Specifically, dry events become more likely along Australia’s east
coast during SON (Fig. 2B), and significant coincidence rates with wet events
arise over the south of Chile pointing towards increased rainfall during CP El
Niños as compared to their canonical counterparts. In DJF months coinciding
with CP events, we observe the emergence of new wet patterns over Central Asia
and China as well as a dry pattern over the north of Chile (Fig. 2D).
Ultimately, for MAM we observe a pronounced dry pattern over Southeast Africa
(Fig. 2F). Generally, we note a decrease in the spatial coherence of seasonal
precipitation extremes coinciding with CP El Niños in comparison with EP
phases.

Figure 3: Same as Fig. 2 for La Niña periods.

###  3.3 Seasonal precipitation extremes and EP/CP La Niña

We finally perform the same analysis for La Niña periods. For the EP, i.e.
canonical, La Niña type (Fig. 3A,C,E) we again find various patterns that were
reported in earlier studies. Specifically, during SON coinciding with EP La
Niña phases (Fig. 3A) we find the expected wet conditions over Australia and
Indonesia (Arblaster et al., 2002) and exceptionally dry conditions in parts
of southern Europe (Pozo-Vázquez et al., 2005) and the south of Brazil
(Ropelewski and Halpert, 1996). We further observe significant coincidence
rates with dry events in the middle East and with extreme rainfall over
central Europe. For DJF, our analysis confirms findings by Nicholson and
Selato (2000) of wet conditions over South Africa and dry events over West
Africa (Fig. 3C). We further observe a prominent extreme precipitation dipole
with dry conditions over Mexico and increased rainfall in the southwestern
part of Canada. For MAM seasons associated with EP La Niñas, we observe a
tendency towards extreme rainfall over the Amazon (Rogers, 1988) and parts of
Northern Australia (Arblaster et al., 2002) (Fig. 3).

In contrast to the coherent patterns observed for EP La Niña phases, we find
only a few spatially organized structures during CP events (Fig. 3B, E, F).
Most prominently, we recover previously reported wet conditions over parts of
Australia in DJF (Arblaster et al., 2002; Cai and Cowan, 2009) (Fig. 3B).
Additionally, we uncover strongly reduced rainfall over Florida in DJF, and
over Scandinavia and the south of Russia in MAM. Thus, similar as for El Niño,
we again observe that CP La Niñas coincide with less spatially coherent
extreme precipitation responses.

Following upon the presented findings, we pick up the recently raised question
whether it is actually meaningful to distinguish La Niña into two flavors in
the same fashion as El Niño (Kug and Ham, 2011). While Kao and Yu (2009) and
Ashok and Yamagata (2009) advocated for such a distinction, Kug et al. (2009)
and Ren and Jin (2011) argued that based on correlation analyses between La
Niña related SST patterns no distinct discrimination into two types is
evident. Contributing to this discussion, Wiedermann et al. (2016)
demonstrated that according to the transitivity index characterizing ENSO
teleconnectivity it is indeed meaningful to provide a discrimination of La
Niño periods analogous to El Niño. The results of our current work demonstrate
that seasonal precipitation extremes accompanied by EP La Niñas are generally
more likely to arise in a spatially coherent way (Fig. 3) than such observed
along with CP phases. The same property also holds for El Niño periods (Sec.
3.2), which highlights again a symmetry not only within the spatial patterns
of El Niño and La Niña themselves, but also with respect to their induced
extreme precipitation responses. Thus, from an impact-oriented point of view,
our work provides further evidence in favor of a distinction between two types
of La Niña indicated by the presence or absence of induced seasonal
precipitation extremes. In other words, from the viewpoint of extreme events
(and thus possibly in contrast to observations based on linear statistical
analysis) it seems reasonable to discriminate La Niña into two types in the
same way as for El Niño periods.

##  4 Conclusion & Outlook

We have carried out a detailed analysis of ENSO imprints in global patterns of
seasonal precipitation extremes. Specifically, we discriminated both, El Niño
and La Niña into two distinct types (East Pacific and Central Pacific) by
utilizing the network transitivity index $\mathcal{T}$ introduced by
Wiedermann et al. (2016). From $\mathcal{T}$, four distinct event series were
defined such that an event in each series corresponds to the presence of the
respective type of ENSO event. The globally gridded GPCC rainfall data set was
aggregated to grid point-wise seasonal precipitation sums, and corresponding
event series were obtained by applying the 80th (20th) percentile to define
extremely wet (dry) periods.

Figure 4: Schematic summary of the results of this work. Shaded areas indicate
geographic locations for which extreme (high or low) seasonal precipitation
sums exhibit significant coincidence rates with EP (A,C) or EP (B,D) El Niño
(A,B) or La Niña phases (C,D). For all indicated areas related to EP (CP)
phases, seasonal precipitation extremes occur with a probability of 30-50%
(50-70%), respectively.

We then used event coincidence analysis (Siegmund et al., 2016a; Donges et
al., 2016) to identify grid points with significant coincidence rates between
different types of ENSO phases and seasonal precipitation extremes. Our
analysis confirmed that previously observed interrelationships based on linear
correlation or composite analysis also apply to the associated extremes. In
addition, we identified further impacts of ENSO episodes, especially of the
Central Pacific types of both, El Niño and La Niña, which have to our current
knowledge not been described so far. This implies that in these cases,
classical analysis methods may not be sufficienct to capture such
interrelations. Moreover, it shows that even though a general linear
relationship between ENSO and precipitation might be weak or absent for a
given region, dry periods or heavy rainfall can still be possible effects of
specific types of ENSO phases. At the same time we observe that most
previously observed interrelations only hold for events corresponding to East
Pacific ENSO flavors. This implies that with a proper discrimination of event
types, threats like possible droughts or flooding can be better anticipated,
since for most regions of the Earth, CP types of ENSO events have no or a less
spatially organized impact on seasonal precipitation extremes.

Our analysis contributed to the ongoing discourse whether or not it is
(statistically and climatologically) meaningful to discriminate La Niña into
two flavors in the same way as done for El Niño. The obtained results suggest
that such congruence between ENSO’s positive and negative phase is meaningful
in both ways as (i) the transitivity index clearly distinguishes between two
types of events for El Niño as well as La Niña and (ii) event coincidence
analysis shows that the EP La Niña displays more spatially coherent seasonal
precipitation extremes, while fewer impacts are found for its Central Pacific
counterpart.

In conclusion, our analysis provided a detailed and global overview on the
large-scale imprints of different ENSO phases and flavors in extreme seasonal
precipitation patterns. All findings are ultimately summarized in Fig. 4,
which displays schematically all regions for which the four different types of
ENSO phases show significant coincidence rates with seasonal precipitation
extremes together with these rates.

Future work shall apply the concepts used in this work to also study ENSO
related extreme impacts on further climatic observables, such as temperature.
Also, a more thorough intercomparison between results obtained from linear and
event statistics could prove useful in assessing which regions are most
affected in terms of extreme climatic responses to the presence of either type
and phase of ENSO. Ultimately, it could be useful to apply the presented
framework also to future climate projections in order to assess possible
changes in spatial extent and frequency of ENSO related extreme events.

###### Acknowledgements.

MW, JFS and RVD acknowledge funding by the German Federal Ministry for
Education and Research (BMBF) via the Young Investigators Group CoSy-CC2
(grant no. 01LN1306A). JFD thanks the Stordalen Foundation and the Leibniz
Society (project DOMINOES) for financial support. JK and RVD acknowledge the
IRTG 1740 funded by DFG and FAPESP. JFS has been supported by the
Evangelisches Studienwerk Villigst e.V. All computations have been performed
using the R package CoinCalc (Siegmund et al., 2017).

## References

  * Allan and Soden (2008) Allan, R. P., and B. J. Soden (2008), Atmospheric Warming and the Amplification of Precipitation Extremes, Science, 321(5895), 1481–1484, 10.1126/science.1160787.
  * Arblaster et al. (2002) Arblaster, J., G. Meehl, and A. Moore (2002), Interdecadal modulation of Australian rainfall, Clim. Dynam., 18(6), 519–531, 10.1007/s00382-001-0191-y.
  * Ashok and Yamagata (2009) Ashok, K., and T. Yamagata (2009), Climate change: The El Niño with a difference, Nature, 461(7263), 481–484, 10.1038/461481a.
  * Ashok et al. (2007) Ashok, K., S. K. Behera, S. A. Rao, H. Weng, and T. Yamagata (2007), El Niño Modoki and its possible teleconnection, J. Geophys. Res., 112(C11), C11,007, 10.1029/2006JC003798.
  * Cai and Cowan (2009) Cai, W., and T. Cowan (2009), La Niña Modoki impacts Australia autumn rainfall variability, Geophys. Res. Lett., 36(12), L12,805, 10.1029/2009GL037885.
  * Cai et al. (2014) Cai, W., S. Borlace, M. Lengaigne, P. van Rensch, M. Collins, G. Vecchi, A. Timmermann, A. Santoso, M. J. McPhaden, L. Wu, M. H. England, G. Wang, E. Guilyardi, and F.-F. Jin (2014), Increasing frequency of extreme El Niño events due to greenhouse warming, Nature Clim. Change, 4(2), 111–116, 10.1038/nclimate2100.
  * Camberlin et al. (2001) Camberlin, P., S. Janicot, and I. Poccard (2001), Seasonality and atmospheric dynamics of the teleconnection between African rainfall and tropical sea-surface temperature: Atlantic vs. ENSO, Int. J. Climatol., 21(8), 973–1005, 10.1002/joc.673, wOS:000169831700004.
  * Chiew et al. (1998) Chiew, F. H. S., T. C. Piechota, J. A. Dracup, and T. A. McMahon (1998), El Niño/Southern Oscillation and Australian rainfall, streamflow and drought: Links and potential for forecasting, J. Hydrol., 204(1–4), 138–149, 10.1016/S0022-1694(97)00121-2.
  * Dai and Wigley (2000) Dai, A., and T. M. L. Wigley (2000), Global patterns of ENSO-induced precipitation, Geophys. Res. Lett., 27(9), 1283–1286, 10.1029/1999GL011140.
  * Diaz et al. (2001) Diaz, H. F., M. P. Hoerling, and J. K. Eischeid (2001), ENSO variability, teleconnections and climate change, Int. J. Climatol., 21(15), 1845–1862, 10.1002/joc.631.
  * Donges et al. (2011) Donges, J. F., R. V. Donner, M. H. Trauth, N. Marwan, H.-J. Schellnhuber, and J. Kurths (2011), Nonlinear detection of paleoclimate-variability transitions possibly related to human evolution, P. Natl. Acad. Sci., 108(51), 20,422–20,427, 10.1073/pnas.1117052108.
  * Donges et al. (2016) Donges, J. F., C.-F. Schleussner, J. F. Siegmund, and R. V. Donner (2016), Event coincidence analysis for quantifying statistical interrelationships between event time series: on the role of flood events as triggers of epidemic outbreaks, Eur. Phys. J. S. T., 225(3), 471–487, 10.1140/epjst/e2015-50233-y.
  * Easterling et al. (2000) Easterling, D. R., J. Evans, P. Y. Groisman, T. Karl, et al. (2000), Observed variability and trends in extreme climate events: a brief review, Bull. Amer. Meteor. Soc., 81(3), 417.
  * Graf and Zanchettin (2012) Graf, H.-F., and D. Zanchettin (2012), Central Pacific El Niño, the subtropical bridge, and Eurasian climate, J. Geophys. Res., 117(D1), 10.1029/2011JD016493.
  * Heitzig et al. (2012) Heitzig, J., J. F. Donges, Y. Zou, N. Marwan, and J. Kurths (2012), Node-weighted measures for complex networks with spatially embedded, sampled, or differently sized nodes, Eur. Phys. J. B, 85(1), 1–22, 10.1140/epjb/e2011-20678-7.
  * Hendon et al. (2009) Hendon, H. H., E. Lim, G. Wang, O. Alves, and D. Hudson (2009), Prospects for predicting two flavors of El Niño, Geophys. Res. Lett., 36(19), L19,713, 10.1029/2009GL040100.
  * Hoell et al. (2014) Hoell, A., C. Funk, T. Magadzire, J. Zinke, and G. Husak (2014), El Niño-Southern Oscillation diversity and Southern Africa teleconnections during Austral Summer, Clim. Dynam., 45(5-6), 1583–1599, 10.1007/s00382-014-2414-z.
  * Hu et al. (2011) Hu, Z.-Z., A. Kumar, B. Jha, W. Wang, B. Huang, and B. Huang (2011), An analysis of warm pool and cold tongue El Niños: air-sea coupling processes, global influences, and recent trends, Clim. Dynam., 38(9-10), 2017–2035, 10.1007/s00382-011-1224-9.
  * Jones et al. (2007) Jones, P., K. Trenberth, P. Ambenje, R. Bojariu, D. Easterling, T. Klein, D. Parker, J. Renwick, M. Rusticucci, B. Soden, et al. (2007), Observations: surface and atmospheric climate change, Climate change 2007: the physical science basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change, pp. 235–336.
  * Kao and Yu (2009) Kao, H.-Y., and J.-Y. Yu (2009), Contrasting Eastern-Pacific and Central-Pacific Types of ENSO, J. Climate, 22(3), 615–632, 10.1175/2008JCLI2309.1.
  * Karl and Trenberth (2003) Karl, T. R., and K. E. Trenberth (2003), Modern global climate change, Science, 302(5651), 1719–1723.
  * Kim et al. (2011) Kim, W., S.-W. Yeh, J.-H. Kim, J.-S. Kug, and M. Kwon (2011), The unique 2009-2010 El Niño event: A fast phase transition of warm pool El Niño to La Niña, Geophys. Res. Lett., 38(15), L15,809, 10.1029/2011GL048521.
  * Kistler et al. (2001) Kistler, R., W. Collins, S. Saha, G. White, J. Woollen, E. Kalnay, M. Chelliah, W. Ebisuzaki, M. Kanamitsu, V. Kousky, H. van den Dool, R. Jenne, and M. Fiorino (2001), The NCEP-NCAR 50-Year Reanalysis: Monthly Means CD-ROM and Documentation, Bull. Amer. Meteor. Soc., 82(2), 247–267, 10.1175/1520-0477(2001)082¡0247:TNNYRM¿2.3.CO;2.
  * Kug and Ham (2011) Kug, J.-S., and Y.-G. Ham (2011), Are there two types of La Nina?, Geophys. Res. Lett., 38(16), L16,704, 10.1029/2011GL048237.
  * Kug et al. (2009) Kug, J.-S., F.-F. Jin, and S.-I. An (2009), Two Types of El Niño Events: Cold Tongue El Niño and Warm Pool El Niño, J. Climate, 22(6), 1499–1515, 10.1175/2008JCLI2624.1.
  * Lorenz and Kunstmann (2012) Lorenz, C., and H. Kunstmann (2012), The hydrological cycle in three state-of-the-art reanalyses: Intercomparison and performance analysis, J. Hydrometeorol., 13(5), 1397–1420, 10.1175/JHM-D-11-088.1.
  * Neelin et al. (2003) Neelin, J. D., C. Chou, and H. Su (2003), Tropical drought regions in global warming and El Niño teleconnections, Geophys. Res. Lett., 30(24), 2275, 10.1029/2003GL018625.
  * Nicholson and Selato (2000) Nicholson, S., and J. Selato (2000), The influence of La Niña on African rainfall, Int. J. Climatol., 20(14), 1761–1776, 10.1002/1097-0088(20001130)20:14¡1761::AID-JOC580¿3.0.CO;2-W.
  * Pozo-Vázquez et al. (2005) Pozo-Vázquez, D., S. R. Gámiz-Fortis, J. Tovar-Pescador, M. J. Esteban-Parra, and Y. Castro-Díez (2005), El Niño–Southern Oscillation events and associated European winter precipitation anomalies, Int. J. Climatol., 25(1), 17–31, 10.1002/joc.1097.
  * Radebach et al. (2013) Radebach, A., R. V. Donner, J. Runge, J. F. Donges, and J. Kurths (2013), Disentangling different types of El Niño episodes by evolving climate network analysis, Phys. Rev. E, 88(5), 052,807, 10.1103/PhysRevE.88.052807.
  * Rammig et al. (2015) Rammig, A., M. Wiedermann, J. F. Donges, F. Babst, W. von Bloh, D. Frank, K. Thonicke, and M. D. Mahecha (2015), Coincidences of climate extremes and anomalous vegetation responses: comparing tree ring patterns to simulated productivity, Biogeosciences, 12(2), 373–385, 10.5194/bg-12-373-2015.
  * Ren and Jin (2011) Ren, H.-L., and F.-F. Jin (2011), Niño indices for two types of ENSO, Geophys. Res. Lett., 38(4), L04,704, 10.1029/2010GL046031.
  * Rogers (1988) Rogers, J. C. (1988), Precipitation Variability over the Caribbean and Tropical Americas Associated with the Southern Oscillation, J. Climate, 1(2), 172–182, 10.1175/1520-0442(1988)001¡0172:PVOTCA¿2.0.CO;2.
  * Ropelewski and Halpert (1987) Ropelewski, C. F., and M. S. Halpert (1987), Global and Regional Scale Precipitation Patterns Associated with the El Niño/Southern Oscillation, Mon. Wea. Rev., 115(8), 1606–1626, 10.1175/1520-0493(1987)115¡1606:GARSPP¿2.0.CO;2.
  * Ropelewski and Halpert (1996) Ropelewski, C. F., and M. S. Halpert (1996), Quantifying Southern Oscillation-Precipitation Relationships, J. Climate, 9(5), 1043–1059, 10.1175/1520-0442(1996)009¡1043:QSOPR¿2.0.CO;2.
  * Rudolf et al. (1994) Rudolf, B., H. Hauschild, W. Rueth, and U. Schneider (1994), Terrestrial Precipitation Analysis: Operational Method and Required Density of Point Measurements, in Global Precipitations and Climate Change, edited by M. Desbois and F. Désalmand, no. 26 in NATO ASI Series, pp. 173–186, Springer Berlin Heidelberg, dOI: 10.1007/978-3-642-79268-7_10.
  * Schleussner et al. (2016) Schleussner, C.-F., J. F. Donges, R. V. Donner, and H. J. Schellnhuber (2016), Armed-conflict risks enhanced by climate-related disasters in ethnically fractionalized countries, P. Natl. Acad. Sci., 113(33), 9216–9221, 10.1073/pnas.1601611113.
  * Schneider et al. (2015) Schneider, U., A. Becker, P. Finger, A. Meyer-Christoffer, B. Rudolf, and M. Ziese (2015), GPCC Full Data Reanalysis Version 7.0 (at 0.5∘, 1.0∘, 2.5∘): Monthly Land-Surface Precipitation from Rain-Gauges built on GTS-based and Historic Data.
  * Shaman and Tziperman (2010) Shaman, J., and E. Tziperman (2010), An Atmospheric Teleconnection Linking ENSO and Southwestern European Precipitation, J. Climate, 24(1), 124–139, 10.1175/2010JCLI3590.1.
  * Siegmund et al. (2016a) Siegmund, J. F., T. G. M. Sanders, I. Heinrich, E. van der Maaten, S. Simard, G. Helle, and R. V. Donner (2016a), Meteorological Drivers of Extremes in Daily Stem Radius Variations of Beech, Oak, and Pine in Northeastern Germany: An Event Coincidence Analysis, Front. Plant Sci., 7(733), 10.3389/fpls.2016.00733.
  * Siegmund et al. (2016b) Siegmund, J. F., M. Wiedermann, J. Donges, and R. V. Donner (2016b), Impact of temperature and precipitation extremes on the flowering dates of four German wildlife shrub species, Biogeosciences, 13, 5541–5555, 10.5194/bg-13-5541-2016.
  * Siegmund et al. (2017) Siegmund, J. F., N. Siegmund, and R. V. Donner (2017), CoinCalc - A new R package for quantifying simultaneities of event series, Comput. and Geosci., 98, 64–72, 10.1016/j.cageo.2016.10.004.
  * Taschetto and England (2009) Taschetto, A. S., and M. H. England (2009), El Niño Modoki Impacts on Australian Rainfall, J. Climate, 22(11), 3167–3174, 10.1175/2008JCLI2589.1.
  * Trenberth (1997) Trenberth, K. E. (1997), The Definition of El Niño, Bull. Amer. Meteor. Soc., 78(12), 2771–2777, 10.1175/1520-0477(1997)078¡2771:TDOENO¿2.0.CO;2.
  * Tsonis et al. (2006) Tsonis, A. A., K. L. Swanson, and P. J. Roebber (2006), What Do Networks Have to Do with Climate?, Bull. Amer. Meteor. Soc., 87(5), 585–595, 10.1175/BAMS-87-5-585.
  * Wiedermann et al. (2016) Wiedermann, M., A. Radebach, J. F. Donges, J. Kurths, and R. V. Donner (2016), A climate network-based index to discriminate different types of El Niño and La Niña, Geophys. Res. Lett., 43(13), 2016GL069,119, 10.1002/2016GL069119.


# NewsQs: Multi-Source Question Generation for the Inquiring Mind

Alyssa Hwang2 Kalpit Dixit1 Miguel Ballesteros1
Yassine Benajiba1 Vittorio Castelli1 Markus Dreyer3
Mohit Bansal1 Kathleen McKeown1
1AWS AI Labs 2University of Pennsylvania 3Alexa
<EMAIL_ADDRESS>
{kddixit, ballemig, benajiy, vittorca,
mddreyer, mobansal<EMAIL_ADDRESS>
Work conducted during an internship at Amazon. Corresponding Author

###### Abstract

We present NewsQs (news-cues), a dataset that provides question-answer pairs
for multiple news documents. To create NewsQs, we augment a traditional multi-
document summarization dataset with questions automatically generated by a
T5-Large model fine-tuned on FAQ-style news articles from the News On the Web
corpus. We show that fine-tuning a model with control codes produces questions
that are judged acceptable more often than the same model without them as
measured through human evaluation. We use a QNLI model with high correlation
with human annotations to filter our data. We release our final dataset of
high-quality questions, answers, and document clusters as a resource for
future work in query-based multi-document summarization.111To be released upon
publication.

##  1 Introduction

Providing the ability to answer questions about events and people in the news
would help compensate for information overload in the modern digital age.
Curating datasets that teach machines to answer such questions, however, is
challenging because the datasets require three components: clusters of
documents each conveying different aspects of an event or person, questions
covering the documents, and answers to the questions. Existing datasets
related to this area draw source material from social media, Wikipedia, or
stories, leaving a gap for news-based resources.

Figure 1: Resources used to create NewsQs. Multi-News contains document
clusters and summaries. QA pairs from NOW are used for fine-tuning.

We generate our dataset, NewsQs (news-cues), by augmenting an existing dataset
called Multi-News Fabbri et al. (2019) with automatically generated questions.
Multi-News is a multi-document summarization (MDS) dataset that provides
clusters of documents and human-written summaries. Augmenting Multi-News with
questions would enable NewsQs to serve as a resource for query-based multi-
document summarization (qMDS), which is the task of generating paragraph-long
answers to open-ended questions by drawing information from multiple sources.
Pivoting our dataset generation task into a question generation (QG) task also
simplifies the process because we no longer need to collect all three
necessary components. We already have the documents and answers; we just need
to generate the questions to connect them.

In our work, we experiment with fine-tuning a large language model on FAQ-
style news articles from the News On the Web (NOW) corpus Davies (2022) to
generate questions for Multi-News. Our experiments show that adding control
codes while fine-tuning helps the model generate topical questions related to
the entire summary. Since we do not have reference questions for automatic
evaluation, we use a QNLI model with high correlation with human annotations
to discard low-quality examples. Our contributions include:

  * •

A dataset, NewsQs, of 21,000 high-quality question-answer pairs for multiple
documents.

  * •

A method for using two existing datasets, each containing two of the three
necessary components, to create NewsQs.

  * •

A human evaluation task designed for long text that demonstrates fine-tuning
with control codes produces better quality questions.

##  2 Related Work

Datasets for query-based multi-document summarization (qMDS) need (1) multiple
source documents, (2) questions, and (3) long-form answers. Only a few
datasets fulfill all three requirements: ELI5 from KILT Petroni et al. (2021),
ASQA Stelmakh et al. (2022), DuReader He et al. (2018), AQuaMuSe (Kulkarni et
al., 2020), and SQuALITY Wang et al. (2022). Concerningly, these datasets
contain source documents that were automatically gathered by a variety of
heuristics, so the original ELI5 dataset provides support documents that
contain the full answer only 65% of the time (Fan et al., 2019). Providing
source documents that do not fully contain the answer teaches models to
hallucinate Krishna et al. (2021).

Other related resources for qMDS are missing at least one component. Multi-
document summarization datasets contain multiple source documents and
summaries but not questions DUC (2007); Gholipour Ghalandari et al. (2020).
Multi-document question answering datasets contain short answers of up to a
few words rather than longer, more detailed responses. Through question
generation, we are able to leverage these existing resources to construct a
dataset with all three components that is substantially larger (Duan et al.,
2017; Dugan et al., 2022; Chakrabarty et al., 2022).

##  3 Methods

Our goal is to produce a dataset of questions and answers that are relevant to
topical document clusters. Our approach leverages two existing datasets, each
providing only two of the three components. We experiment with three methods
for a range of models (see Appendix A for prompts).

###  3.1 Data

A large dataset containing two of the three necessary components already
exists: Multi-News (Fabbri et al., 2019). This dataset contains approximately
56,000 clusters of related news articles and human-written summaries. Zero-
shot generation of questions on Multi-News does not produce suitable questions
(see Section 4), so we experiment with fine-tuning models on a small training
set of similar data from the News On the Web (NOW) corpus (Davies, 2022).

We sample 1,200 question-answer pairs from FAQ-style articles—a popular style
of news article that presents information as a series of Frequently Asked
Questions—and split them into 80/20 train/validation sets. We manually exclude
QA pairs that are not self-contained, such as a question like “What about it?”
We also exclude QA pairs with answers that are fewer than 30 words or longer
than 350 words to keep our fine-tuning dataset similar to our final inference
dataset; Multi-News summaries have an average of 266 words. An example
question-answer pair from NOW is shown in Appendix Table 7.

###  3.2 Fine-Tuning (FT) Experiments

Based on preliminary zero-shot experiments, we fine-tune T5-Large to generate
questions for NewsQs (see Appendix Table 8).

#### Vanilla

We fine-tune T5-Large by including the paragraph-long NOW answer followed by
the reference question and training for 100, 200, and 500 epochs with learning
ranges of $1.00\times 10^{-4}$, $3.00\times 10^{-4}$, and $1.00\times
10^{-3}$, for a total of 9 experiments. The best settings in our parameter
search were 500 epochs with a learning rate of $3.00\times 10^{-4}$, which we
consider our baseline.

#### Control Codes

We experiment with prepending prompts with control codes (i.e., [dogs; cats;
pets]) before fine-tuning with the baseline settings to encourage T5-Large to
write a question about the full paragraph (see Appendix A.1). We use salient
keywords or entities as control codes because main ideas of news articles tend
to come from these types of words. After investigating different numbers of
output from Yet Another Keyword Extractor (Campos et al., 2020, 2018a, 2018b)
and an off-the-shelf entity detection model, we use the top three outputs from
the entity detection model as the control codes.

##  4 Results of Fine-Tuning Experiments

We show performance on the NOW training set using ROUGE-L Lin (2004) and
BERTScore Zhang et al. (2019) F1-scores (see Table 1). The improvement from
zero-shot T5-Large to fine-tuning with T5-Large is clearly evident; ROUGE-L
F1-score increases by 0.2 absolute and BERTScore by 0.42. Automatic metrics do
not show an improvement when control codes are added. A spot check of
questions by the authors, however, suggested that generated questions are more
appropriate for the full paragraph answer (see examples in Appendix Table 8),
so we continued with human evaluation.

| ROUGE-L | BERTScore
---|---|---
Vanilla ZS | 0.12 | -0.05
Vanilla FT | 0.38 | 0.47
Control Codes FT | 0.39 | 0.47
Table 1: Fine-tuning results. Automatic metrics show improvement after fine-tuning but not after adding control codes, which is more apparent in human evaluation. | Train | Val. | Test
---|---|---|---
Number of Examples | 17K | 2.1K | 1.6K
Avg. Len. of Question | 11.0 | 11.0 | 11.1
Avg. Len. of Answer | 288 | 288 | 287
# Entities Per Answer | 8.46 | 8.55 | 8.55
% Overlap of Entities | 43.2 | 43.4 | 44.1
Avg. QNLI Score | 0.936 | 0.939 | 0.941
Avg. QNLI Score (BF) | 0.387 | 0.387 | 0.396
Table 2: NewsQs dataset statistics. QNLI scores substantially improved before (BF) and after filtering. The lengths of questions and answers are balanced across splits. The percentage overlap of entities between questions and answers indicates that generated questions cover a fair amount of the main ideas in the answers. | Train | Val. | Test
---|---|---|---
who | 6.77 | 7.15 | 6.32
what | 48.2 | 46.7 | 49.6
when | 1.50 | 1.34 | 1.06
where | 2.51 | 2.54 | 2.70
why | 8.70 | 9.32 | 7.82
how | 22.8 | 22.8 | 23.1
Table 3: Percentage of questions in each data split that start with the
specified question words.

##  5 Human Evaluation

| Sentence Relevance ($\kappa$) | Question Acceptability ($\kappa$) | Ranking (%)
---|---|---|---
| All | Accept. Only | Multiclass | Binary | Mean | > | < | =
Vanilla | 0.32 | 0.32 | 0.20 | 0.27 | 2.32 | 45.7 | 27.2 | 27.1
Control Codes | 0.24 | 0.56 | 0.26 | 0.34 | 2.53
Table 4: Results of human evaluation for both fine-tuned models. We report
Cohen’s Kappa (left) for all and acceptable questions. We also compute scores
for each question by ranking the question labels. We report the mean for each
model and the percentage of times the control codes model had a higher, lower,
or equal score (right).

Since we ran inference on the Multi-News dataset to create new questions, we
do not have reference questions for automatic metrics. We need human input to
evaluate the final output of our model.

###  5.1 Design

Evaluating our data requires human annotators to read questions and paragraph-
long answers, which can be very cognitively demanding. We designed a human
evaluation task that breaks down long text into sentence-level annotations to
make this process easier. The task is composed of two subtasks: Sentence
Relevance and Question Acceptability.

For each question in NewsQs, we show the answer one sentence at a time and ask
annotators to judge if each sentence is relevant to the question. When they
have seen the entire paragraph, annotators are asked to rate the question as
one of the following: acceptable, narrow topic, broad topic, irrelevant topic,
wrong type, incoherent, incomplete, not a question, or other. Along with
decreasing the amount of new information annotators have to review at a time,
our task collects human judgment at the sentence and paragraph levels.

###  5.2 Results and Discussion

At least one annotator judged 81.6% of control codes questions as acceptable,
demonstrating the high quality of our questions. We report Cohen’s Kappa
scores across tasks and experiments in the left half of Table 4. Two
annotators evaluated data for each experiment, for a total of four annotators
in our study. When inspecting agreement for each input paragraph, we realized
that inter-annotator agreement on Sentence Relevance was much lower for
questions marked as not acceptable because relevance to a low-quality question
is not well defined. When including only questions marked as acceptable by at
least one annotator, inter-annotator agreement increases substantially: from a
Cohen’s Kappa of 0.32 (fair) to 0.56 (moderate) for Sentence Relevance on the
control codes FT model.

We estimated the level of quality of a question by ranking the possible
question judgments. A judgment of acceptable was assigned rank 3, broad or
narrow topic was assigned rank 2, and the remaining were assigned rank 1. We
then computed an average score for all 188 questions. The average vanilla
model ranking was 2.32 and the average control codes model ranking was 2.53,
where the rankings ranged from 1 to 3 and higher is better. These results are
significant with p-value 0.0002, according to the Bootstrap Test Dror et al.
(2018). For a deeper analysis, we compared all of the control codes and
vanilla scores. As shown on the right side of Table 4, the control codes model
scores higher 45% of the time and equal 27.1%. These two methods for analyzing
human evaluation results show that the control codes FT model outperforms the
vanilla FT model on acceptability.

By also asking annotators to rate relevance one sentence at a time, we were
able to collect finer-grained data on how relevant a question is to an input
paragraph. We computed the Sentence Relevance Score (SRS) for each question by
counting the number of times both annotators judged a sentence as relevant to
the question and normalizing by the number of sentences. SRS ranges from 0 to
1, where higher is better. The average SRS of the vanilla and control codes
models was 0.86 and 0.96, respectively. These results demonstrate that
questions generated with control codes are more likely to require a larger
portion of the sentences in the summary for a suitable response.

##  6 Automatic Evaluation

Since we generated new questions to augment Multi-News, we do not have
reference questions for automatic metrics like ROUGE or BERTScore. We instead
use a QNLI model to measure if the question is answerable given the paragraph.
We find that QNLI correlates highly with human judgments of question
acceptability ($\rho=0.41$) and passage relevance ($\rho=0.23$). We use QNLI
to filter our dataset from 54K to 21K high-quality examples (17,377
train/2,167 validation/1,597 test).

NewsQs Examples
---
Question What’s the most earth-like planet in the solar system?
Answer if you’re sick of earth and have a spacecraft capable of traveling
hundreds of light-years, astronomers have spotted the most promising
destination yet. kepler-438b is a newly-confirmed potential “earth twin”
detected by the kepler space telescope, the bbc reports, one of eight
confirmed new exoplanets. [+174 words]
Documents [2,949 words from 3 articles]
Question Do ants really just turn left?
Answer roughly nine in 10 humans are right-handed, an example of “brain
lateralization” that’s pretty common among vertebrates—and now apparently
invertebrates. researchers in the uk are finding that even ants—which are
invertebrates, meaning they have exoskeletons—carry an innate directional
bias, in their case almost always turning left when exploring new territory,
reports science daily. [+94 words]
Documents [486 words from 2 articles]
Table 5: Examples from our NewsQs dataset. Each example in NewsQs contains a
machine-generated question and an answer and cluster of source documents from
Multi-News.

##  7 Discussion and Conclusion

In our work, we presented a method for fine-tuning T5-Large with control codes
to generate questions that can be answered by an input paragraph. We used this
method to create NewsQs and showed that it produces acceptable questions 80.3%
of the time, which were further filtered with a QNLI model. Designing a good
human evaluation task for NewsQs was challenging because the natural instinct
to display the question with the input paragraph would have been very
cognitively demanding. In discussions with the team that evaluated NewsQs, the
annotators asked us to “remember the human in human evaluation,” so we
adjusted by displaying paragraphs one sentence at a time. We believe that more
work in human-centered annotation tasks will improve evaluation… and
remembering the human will improve NLP as a whole, for annotators,
researchers, and users alike.

## Limitations

Our work augments an existing dataset, Multi-News, which means that it depends
on its quality and inherits its limitations. We chose the Multi-News dataset
because it is publicly available, vetted and published by a reputable venue,
and curated through careful human effort. Humans, however, can still introduce
error even in the most carefully designed settings. Despite using a qualified
team of linguistics experts for human evaluation, inter-annotator agreement is
low, suggesting that the complexity of NLP tasks makes high agreement
difficult to achieve.

We cannot release our fine-tuned models because our fine-tuning dataset, the
News On the Web (NOW) corpus, is not open to the public. The NOW corpus,
however, generously offers free access to researchers affiliated with an
educational institution who apply for it. We use a very small subset of the
articles contained in the NOW corpus, which were scraped from publicly
available websites. Researchers who are interested in reproducing our work or
generating similar questions on a different inference set can use our methods,
which were designed for modest computational resources—our fine-tuning method
takes about two days per T5-Large model on an AWS p3.16x. This also comes with
an important trade-off: having smaller resources means using smaller models,
and smaller models are known to perform worse than newer, much larger models.
We believe that our work helps combat this trade-off, allowing more people to
use NLP technology.

## Ethical Considerations

We used email to recruit a qualified team of linguistics experts to evaluate
our dataset, ensuring that payment for their services was fair given their
country of residence and amount of work the task required. The experts are
based in the United States and diverse in gender, ethnicity, country of
origin, and languages spoken. We provided in-depth instructions on how their
annotations would be used and hosted several meetings throughout the process
to introduce our task, answer questions, and improve our evaluation design.

Our work presents a dataset for explaining multiple news articles, which vary
widely in content. We manually inspected our fine-tuning dataset from the NOW
corpus for quality and ensured that information about the author of each
article was not included during training. We add machine-generated questions
to the Multi-News dataset, which has already been vetted and published,
without modifying it. To the best of our knowledge through periodic empirical
analysis and final human evaluation, the questions we generate by fine-tuning
on NOW and running inference on Multi-News have low risk of causing harm. We
will release the questions judged acceptable by both annotators as a gold test
set upon publication for future work.

## References

  * Campos et al. (2020) Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Jorge, Célia Nunes, and Adam Jatowt. 2020.  YAKE! Keyword extraction from single documents using multiple local features.  _Information Sciences_ , 509:257–289.  Publisher: Elsevier.
  * Campos et al. (2018a) Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Célia Nunes, and Adam Jatowt. 2018a.  Yake! collection-independent automatic keyword extractor.  In _European Conference on Information Retrieval_ , pages 806–810. Springer.
  * Campos et al. (2018b) Ricardo Campos, Vítor Mangaravite, Arian Pasquali, Alípio Mário Jorge, Célia Nunes, and Adam Jatowt. 2018b.  YAKE! Collection-Independent Automatic Keyword Extractor.  In Gabriella Pasi, Benjamin Piwowarski, Leif Azzopardi, and Allan Hanbury, editors, _Advances in Information Retrieval_ , volume 10772, pages 806–810. Springer International Publishing, Cham.  Series Title: Lecture Notes in Computer Science.
  * Chakrabarty et al. (2022) Tuhin Chakrabarty, Justin Lewis, and Smaranda Muresan. 2022.  CONSISTENT: Open-Ended Question Generation From News Articles.
  * Davies (2022) Mark Davies. 2022.  Corpus of News on the Web (NOW).
  * Dror et al. (2018) Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018.  The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1383–1392. Association for Computational Linguistics.  Event-place: Melbourne, Australia.
  * Duan et al. (2017) Nan Duan, Duyu Tang, Peng Chen, and Ming Zhou. 2017.  Question Generation for Question Answering.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 866–874, Copenhagen, Denmark. Association for Computational Linguistics.
  * DUC (2007) DUC. 2007.  Document Understanding Conferences - Past Data.
  * Dugan et al. (2022) Liam Dugan, Eleni Miltsakaki, Shriyash Upadhyay, Etan Ginsberg, Hannah Gonzalez, DaHyeon Choi, Chuning Yuan, and Chris Callison-Burch. 2022.  A feasibility study of answer-agnostic question generation for education.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1919–1926, Dublin, Ireland. Association for Computational Linguistics.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  ELI5: Long Form Question Answering.  _arXiv:1907.09190 [cs]_.  ArXiv: 1907.09190.
  * Gholipour Ghalandari et al. (2020) Demian Gholipour Ghalandari, Chris Hokamp, Nghia The Pham, John Glover, and Georgiana Ifrim. 2020.  A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1302–1308, Online. Association for Computational Linguistics.
  * He et al. (2018) Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. 2018\.  DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applications.  ArXiv:1711.05073 [cs].
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to progress in long-form question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4940–4957, Online. Association for Computational Linguistics.
  * Kulkarni et al. (2020) Sayali Kulkarni, Sheide Chammas, Wan Zhu, Fei Sha, and Eugene Ie. 2020.  AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarization.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Petroni et al. (2021) Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021\.  KILT: a benchmark for knowledge intensive language tasks.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2523–2544, Online. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language Models are Unsupervised Multitask Learners.  page 24.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _arXiv:1910.10683 [cs, stat]_.  ArXiv: 1910.10683.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask Prompted Training Enables Zero-Shot Task Generalization.  _arXiv:2110.08207 [cs]_.  ArXiv: 2110.08207.
  * Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022.  ASQA: Factoid Questions Meet Long-Form Answers.  _arXiv:2204.06092 [cs]_.  ArXiv: 2204.06092.
  * Wang et al. (2022) Alex Wang, Richard Yuanzhe Pang, Angelica Chen, Jason Phang, and Samuel R. Bowman. 2022.  SQuALITY: Building a Long-Document Summarization Dataset the Hard Way.  _arXiv:2205.11465 [cs]_.  ArXiv: 2205.11465.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019\.  BERTScore: Evaluating Text Generation with BERT.

##  Appendix A Model-Specific Prompts

For T5 models, we start the prompt with a short natural-language command
(“generate question”) followed by the answer paragraph and the pad token
(Raffel et al., 2020). Prompts for T0 models are similar, except the short
natural-language command is replaced by a sentence-long natural-language
instruction (“Give me a question about this answer”) (Sanh et al., 2022).
Prompts for GPT-2 incorporate “[answer]” and “[question]” special tokens to
mark the beginnings of the input answer paragraph and the output question
(Radford et al., 2019). BART is prompted with the answer paragraph surrounded
by beginning-of-sentence and end-of-sentence special tokens (Lewis et al.,
2019).

###  A.1 T5-Large Control Code Prompt

Following from Section 3.2. A full prompt with control codes for T5-Large
would look like:

> generate question: [dogs; cats; pets] Dogs and cats are popular pets…<pad>

##  Appendix B Full Fine-Tuning Results

We report zero-shot and fine-tuned T5-Large performance across six ngram-,
semantic similarity-, and model-based automatic metrics in Table 6.

##  Appendix C Examples

We show examples from the fine-tuning dataset (News On the Web), inference
dataset (Multi-News), and human evaluation of our dataset (NewsQs) in Tables
7, 8, and 9.

| ROUGE-1 | ROUGE-2 | ROUGE-L | BERTScore | BARTScore
---|---|---|---|---|---
| P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Vanilla ZS | 0.24 | 0.11 | 0.12 | 0.09 | 0.03 | 0.04 | 0.25 | 0.10 | 0.12 | -0.10 | 0.03 | -0.05 | -4.9
Vanilla FT | 0.41 | 0.40 | 0.39 | 0.23 | 0.22 | 0.22 | 0.40 | 0.39 | 0.38 | 0.51 | 0.43 | 0.47 | -3.6
Control Codes FT | 0.43 | 0.41 | 0.40 | 0.25 | 0.24 | 0.24 | 0.41 | 0.39 | 0.39 | 0.51 | 0.43 | 0.47 | -3.6
Table 6: Full fine-tuning results for T5-Large: precision, recall, and F1-scores for ROUGE-1, ROUGE-2, ROUGE-L, and BERTScore; BARTScore. News On the Web Answer |  Questions
---|---
Several human-induced factors are contributing to a spread in water conditions that lead to harmful algal blooms around the world. The first is land management. Agricultural runoff, such as fertilizer, and urban runoff, such as sewage, are depositing extra nutrients into bodies of water, stimulating eutrophication. The construction of dams has also slowed water movement in some areas, creating stagnant bodies that foster cyanobacteria. But climate change is also leading to hotter temperatures and warmer bodies of water that stimulate the growth of cyanobacteria . Droughts also lead to water and @ @ @ @ @ @ @ @ @ @ we continue to see extreme warmth, if we continue to see droughts,” Caron said, “we will probably see more of these toxins produced and the potential for more animals and humans to get in the way of those toxins.” |  Reference
Why are blue-green algae spreading?
Fine-Tuned
Vanilla What causes algal blooms?’
Control Codes What causes these algal blooms?’
Zero-Shot
T5-Small sewage, are depositing extra nutrients into bodies of water,
stimulating eutrophication’
BART-Large Several human-induced factors are contributing to a spread in water
conditions that lead to harmful algal blooms around the world. The first is
land management. Agricultural runoff, such as fertilizer, and urban runoff,
which are both increasing the amount of nutrients in the water, are depositing
extra nutrients into bodies of water, stimulating eutrophication. The
construction of dams has also slowed water movement in some areas, creating
stagnant bodies that foster cyanobacteria. But climate change is also leading
to hotter temperatures and warmer bodies of food that stimulate the growth of
cyanob bacteria. Droughts also lead to water pollution. “If we continue to see
extreme heat and @ @ @ & @ @ # @ @ we continue- see extreme warmth, if we
continue, see droughts, ” Caron said, “ we will probably see more of these
toxins produced and the potential for more animals and humans to get in the
way of those toxins. ” ’
T5-Base warmer bodies of water that stimulate the growth of cyanobacteria’
T5-Large harmful algal blooms’
GPT-2 58 8 8’
T5-3B Agricultural runoff, such as fertilizer, and urban runoff, such as
sewage’
T0-3B What is the reason for the spread of harmful algal blooms?’
T0pp What is the main cause of cyanobacteria?’
Table 7: Examples from the fine-tuning dataset (News On the Web). Multi-News Summary |  Generated Questions
---|---
jpmorgan chase ceo jamie dimon plans to apologize before members of congress tomorrow for a trading loss that has cost the bank more than $ 2 billion. he also will say that the bank has taken steps to make sure it does not happen again. “we have let a lot of people down, and we are sorry for it,” dimon says in testimony prepared for his appearance before the senate banking committee. “while we can never say we won’t make mistakes—in fact, we know we will—we do believe this to be an isolated event,” dimon says. the wall street journal has the full text, in which dimon also castigates traders for badly misunderstanding the risks involved and “generally ineffective” managers for failing to rein them in, notes the journal. |  Fine-Tuned
Vanilla What will jpmorgan chase ceo jamie dimon say?’
Control Codes What will the ceo say?’
Zero-Shot
T5-Small jpmorgan chase ceo jamie dimon’
BART-Large jpmorgan chase ceo to apologize for trading lossJPMorgan Chase ceo
jamie dimon plans to apologize before members of congress tomorrow for a
trading loss that has cost the bank more than $ 2 billion. he also will say
that the bank has taken steps to make sure it does not happen again. “ we have
let a lot of people down, and we are sorry for it, ” dimon says in testimony
prepared for his appearance before the senate banking committee. “ while we
can never say we wont́ make mistakes — in fact, we know we will — we do
believe this to be an isolated event, ”dimon says. the wall street journal has
the full text, in which dimon also castigates traders for badly
misunderstanding the risks involved and “ generally ineffective ” managers for
failing to rein them in, notes the journal.’
T5-Base trading loss’
T5-Large senate banking committee’
GPT-2 18 18 18’
T5-3B jpmorgan chase ceo jamie dimon plans to apologize before members of
congress tomorrow for a trading loss that has cost the bank more than $ 2
billion’
T0-3B dimon to apologize before congress for trading loss’
T0pp jpmorgan chief to apologize to congress for trading loss’
Table 8: Examples from the Multi-News dataset. Multi-News Summary some 33
years after it was launched in 1977, voyager 1 has reached the outer edge of
the solar system and is on course to become the first man-made device to sail
into the vast stretches of space that lie beyond. astronomers have confirmed
that the spacecraft has reached a region called the heliopause, where the
solar winds that have blown past voyager for the last 10 billion miles slow to
a stop, discover reports. in another few years, the spacecraft will emerge
from the shell of gases that surrounds the solar system and enter interstellar
space. “when voyager was launched, the space age itself was only 20 years old,
so there was no basis to know that spacecraft could last so long,” a project
scientists tells the bbc. “we had no idea how far we would have to travel to
get outside the solar system. we now know that in roughly five years, we
should be outside for the first time.”
NewsQs Question When will voyager 1 reach the outer edge of the solar system?
Annotations acceptable, acceptable
---
Multi-News Summary the annual report from the sustainable development
solutions network on the world’s happiest and least happy countries is out,
and if you live in the united states, sorry: the us doesn’t make it into the
top 10( it’s ranked no. 13). the survey ranks 157 countries using factors
including gdp, years of healthy life expectancy, freedom from business and
government corruption, and "having someone to count on in times of trouble."
the happiest: denmark switzerland iceland norway finland the least happy( in
order from most to least happy): benin afghanistan togo syria burundi the sdsn
notes that the editors of the list are encouraging a focus on "happiness
inequality," saying that they have found such inequality has increased, and
that people are happier in societies where there is more happiness equality.
click for the top and bottom 10 in each category from reuters.
NewsQs Question the report on happiness inequality has been released. which
countries have improved their ranking?
Annotations narrow topic, acceptable
Multi-News Summary more than a week after america went to the polls, seven
house races are still too close to call and two statewide elections are
undecided. the results look certain to make the historic republican win in the
house even bigger, the los angeles times notes, with democratic candidates
defending all seven districts and leading in just two — california’s 11th and
kentucky’s 6th. the differential is fewer than 700 votes in most races; a 10th
race in north carolina’s 2nd has been called for the republican, but a recount
is expected. alaska is preparing to examine write-in ballots to decide its
senate race; in minnesota, the last unresolved governor’s race in the nation
shows no sign of being decided anytime soon, the ap reports. republican tom
emmer trails democrat mark dayton by 8,750 votes — well within the margin for
a recount — and the fight is expected to continue into december. gov. tim
pawlenty has held transition talks with both men and may have to extend his
term if the race isn’t resolved by jan. 3.
NewsQs Question what’s the difference between a republican and a democratic
candidate?
Annotations broad topic, broad topic
Table 9: Examples of NewsQs human evaluation.