# Real-Time Machine Learning Enabled Low-Cost Magnetometer System
††thanks: This work was supported by NSF EPSCoR Award OIA-1920965.

Talha Siddique and Md. Shaad Mahmud  Department of Electrical and Computer
Engineering
University of New Hampshire
Durham, NH, USA
<EMAIL_ADDRESS>
###### Abstract

Geomagnetically Induced Currents (GICs) are one of the most hazardous effects
of space weather. The rate of change in ground horizontal magnetic component
$dB_{H}/dt$ is used as a proxy measure for GIC. In order to monitor and
predict $dB_{H}/dt$, ground-based fluxgate magnetometers are used. However,
baseline correction is crucial before such magnetometer data can be utilized.
In this paper, a low-cost Machine Learning (ML) enabled magnetometer system
has been implemented to perform real-time baseline correction of magnetometer
data. The predicted geomagnetic components are then used to derive a forecast
for $dB_{H}/dt$. Two different ML models were deployed, and their real-time
and offline prediction accuracy were examined. The localized peaks of the
predicted $dB_{H}/dt$ are further validated using binary event analysis.

###### Index Terms:

Magnetometer Sensor, Real-Time Machine Learning, GIC, Magnetic Components,
Baseline Correction

##  I Introduction

Geomagnetically Induced Currents (GICs) are current induced on the Earth’s
surface conductors due to Geomagnetic Disturbances (GMDs) or Geostorms [1]
[2]. GMDs occur due to the interaction of charged particles from the Sun with
the Earth’s Magnetosphere [3]. The flow of GIC into power transmission lines
can lead to the malfunction of electrical power devices, and transformers [4].
Therefore, to get an adequate opportunity to mitigate the risks posed by
future Geostorms, there is a need to analyze and predict GIC in near real-
time. [5] [6].

GICs are difficult to estimate, and the rate of change in the local ground
horizontal magnetic component ($dB_{H}/dt$) is used as a proxy value [7]. The
disturbances in the geomagnetic field are monitored through the use of ground-
based magnetometers. However, such devices’ ground magnetic component readings
tend to suffer from background noise and interference [7]. For such data to be
utilized for GIC prediction, initiatives have been undertaken where
magnetometer data are curated after baseline correction. For example, SuperMAG
is an association of organizations that operates more than 300 ground-based
magnetometers. Their objective is to perform baseline removal from the
collected data and disperse them for the wider scientific community [8]. Also,
there is no scientific consensus regarding a common technique for baseline
correction [8]. The task mentioned above usually includes performing complex
mathematical operations that are offline in nature. Because of this, the
dispersion of the processed data is time-consuming. The literature consists of
studies where machine learning (ML) architectures like Convolutional Neural
Network (CNN) have been implemented for baseline removal [9]. However, these
techniques do not process the data in real-time. With the availability of raw
data, there is also a drive within the space science community to develop
Artificial Intelligence (AI)-enabled hardware [10]. In recent years,
advancements have been made in the Internet of Things (IoT) and Single Board
Computer (SBC) technology, where the data is partly processed on edge devices.
However, such technology is still in its infancy [11]. To address these gaps,
a low-cost ML-enabled magnetometer system has been developed that can predict
baseline-corrected magnetic field component values in real-time. Section-II in
this paper covers the methodology, which describes the hardware setup, data
acquisition and processing, and an overview of the implemented model. Section-
III provides an analysis of the obtained results. Finally, section-IV
concludes the paper and discusses the future possibilities.

##  II Methodology

###  II-A Hardware and Experimental Setup

Figure 1: PNI RM3100 magnetometer sensor shown next to a US cent coin for size
comparison. Figure 2: Schematic diagram of hardware setup, with key hardware
components labelled

For this study, an iBUG development board was connected to an RM3100 magnetic
sensor and an SD card module to store the raw and predicted magnetic component
data. These devices were connected to the development board using the Serial
Peripheral Interface (SPI) protocol. The board was powered via its USB port by
an external portable power pack. The power pack had a capacity of 10000 mAh
and an output of 5V. The iBUG board is an ML enabled IoT sensing platform,
which has been used in the past literature for real-time environmental
monitoring [11]. It has a RAK11300 module with a 133MHz processor and a dual-
core Raspberry RP2040 Microcontroller Unit (MCU). RM3100 is the propriety
magneto-inductive geomagnetic sensor developed by Positioning Navigation
Intelligence (PNI) Corporation [12]. Magneto-inductive based sensors have been
deemed a promising technology to study space and ground-based geomagnetic
activity [13]. Figure-1 depicts a picture of the RM3100 sensor. In addition,
the sensor was chosen due to its low cost and compatibility with development
boards like iBUG. The entire hardware setup costs around $90, whereas the
average cost of a professional fluxgate magnetometer system is about $1100
[14]. The schematic diagram of the experimental setup is illustrated in
Figure-2.

###  II-B Data Acquisition and Processing

This paper used the baseline-corrected $X^{\prime}$ and $Y^{\prime}$ component
data from a fluxgate magnetometer as the model’s target variables. The raw
data from the RM3100 sensor, $X$ and $Y$, were used as input variables.
University of New Hampshire (UNH) Space Weather Underground (SWUG) is a
collaboration between the UNH and Northern New England high schools to
assemble and deploy ”3-Axis Simple Aurora Monitor (SAM-III)” fluxgate
magnetometers, hereinafter referred to as SWUG magnetometer. This initiative
aims to collect geomagnetic data to study the impact of GMDs, and GICs [14].
The data collected by SWUG on a given day is processed for missing data and
released on its website 24 hours later, along with its unprocessed counterpart
[15]. The SWUG deployed magnetometer nearest to our location was in Exeter,
NH. The experimental setup described in the previous sub-section was placed in
Exeter, NH but not in the same geo-location as the SWUG magnetometer.

The input data was collected using the RM3100 sensor at 2 seconds interval for
16 hours. The correlation between the $X$ and $Y$ component data of both the
SWUG magnetometer and the RM3100 sensors was examined. This was done to check
the calibration difference between the SWUG magnetometer and the RM3100
sensor. The $X$-components have a correlation of 0.53, whereas the
$Y$-component has a correlation of 0.79. Because of this difference in
calibration, the raw data from the RM3100 were used as the input variable
instead of the data from the SWUG magnetometer. The data for the output
variables $X^{\prime}$ and $Y^{\prime}$ were derived from the SWUG
magnetometer component data by performing a baseline-correction. As mentioned
in Section-I, there is no consensus within the scientific community regarding
a common methodology for baseline correction. Therefore, this paper used
Asymmetric Least Square (ALS) for baseline fitting. The ALS algorithm aims to
match the baseline with the measured data while simultaneously penalizing the
roughness of the baseline [16]. A general objective function of ALS is
exhibited in Equation-1, where $a_{i}$ represents the data; $z_{i}$ is the
estimated baseline; $w_{i}$ is the weight; $\lambda$ is the penalty factor;
and $\Delta^{d}$ represents the finite difference operator of order $d$. A
python library called ”pybaselines” was used to carry out the baseline-
correction on the $X$ and $Y$ components from the SWUG magnetometer data.

$\sum\limits_{i=1}^{N}w_{i}(a_{i}-z_{i})^{2}+\lambda\sum\limits_{i=1}^{N-d}(\Delta^{d}z_{i})^{2}$
(1)

The input and output variable datasets were void of missing data points. The
dataset was split for training and testing, where 80% of the data was used for
offline training, and the remainder was used for offline testing. The
validation set was created by taking 20% of the data from the training
dataset.

###  II-C Model and Deployment Overview

Two different, off-the-shelf Deep Learning (DL) architectures were
implemented- a feed-forward Artificial Neural Network (ANN) and a
Convolutional Neural Network (CNN). Both the Neural Networks (NN) has three
hidden layers and use the Adam optimizer to minimize Root Mean Square Error
(RMSE) as the loss function. Both the models are trained separately offline,
using the training and validation set. The models are multivariable and
multivariate, as they take $X$ and $Y$ components from RM3100 as input and
predict the baseline-corrected $\hat{X}$ and $\hat{Y}$ as output. The
implemented magnetometer system then derives the forecasted rate of change in
the local ground magnetic horizontal component (${d\hat{B}_{H}/dt}$). The
derivation of ${d\hat{B}_{H}/dt}$ is exhibited in Equation-2, where $dt$ is
over a time unit of 1 second [7]. Tensorflow Lite (TF-Lite) version of the
models was generated, and the TF-Lite models were deployed at different
instances to test their real-time prediction capability. The experimental
setup was placed in the same geo-location as before to test the real-time
prediction capability. For each deployed model, 2 hours of real-time predicted
data were collected.

$\frac{d\hat{B}_{H}}{dt}=\sqrt{(\frac{d\hat{X}}{dt})^{2}+(\frac{d\hat{Y}}{dt})^{2}}$
(2)

##  III Results and Discussion

The trained model was tested offline using the test dataset. Table-I
summarizes the accuracy of the real-time and offline prediction of the two
components for each model. Normalized RMSE was used as the error metric. It
can be observed that in the case of real-time prediction, the CNN model
outperforms the ANN model. However, the two models have the same accuracy for
offline prediction. Also, the offline model performs better relative to its
real-time counterpart. The offline model performs better because the training
and test set was collected on the same day. Given the lack of variation in the
dataset, the trained model did not achieve the required level of
generalization to provide a comparable real-time prediction carried out on a
different day.

TABLE I: Real-time and offline model accuracy MODEL | REAL-TIME RMSE | OFFLINE RMSE
---|---|---
ANN ($X$) | 0.60 | 0.29
ANN ($Y$) | 0.61 | 0.30
CNN ($X$) | 0.67 | 0.28
CNN ($Y$) | 0.68 | 0.28
Figure 3: Magnetic component ($X$ and $Y$) versus time

Figure-3 illustrates the offline baseline-corrected SWUG magnetic component
data ($X^{\prime}$ and $Y^{\prime}$ ) and their real-time predicted
counterparts ($\hat{X}$ and $\hat{Y}$ ) over time. For a given time frame, the
$\hat{X}$ and $\hat{Y}$ predicted by the ANN and CNN have a greater degree of
fluctuation than $X^{\prime}$ and $Y^{\prime}$. The high RMSE values for the
real-time prediction stem from these fluctuations. Despite the high RMSE, both
the ANN and CNN predictions were able to capture the trend of the baseline-
corrected data collected from the SWUG magnetometer.

The past literature on space science recommends using the Heidke Skill Score
(HSS) to measure a model’s performance [17]. The HSS measures the fractional
improvement of the forecast over the standard forecast. It is also a means of
validating how well the predicted data captures the time-series trend of the
ground truth. HSS is determined through a binary event analysis, where four
thresholds (0.3 nT/s, 0.7 nT/s, 1.1 nT/s, 1.5 nT/s) are used as metrics for
validating predictions of $dB_{H}/dt$ [7] [17]. The actual and forecasted time
series $dB_{H}/dt$ values are divided into non-overlapping, 1200 second
windows, and for each window the local maximum $dB_{H}/dt$ is determined. For
a given threshold, the actual and forecasted local maximum $dB_{H}/dt$ are
examined to determine whether they cross the threshold or not. From the
aforementioned analysis, the number of true positives (H), true negatives (N),
false positives (F), and false negatives (M) can be determined. The HSS for a
particular threshold is then determined using Equation-3 [18].

$HSS=\frac{2(HN-MF)}{(H+M)(M+N)+(H+F)(F+N)}$ (3)

The HSS for the implemented models is shown in Table-II for each of the four
threshold values. HSS ranges from $-\infty$ to 1, where a negative value
indicates that the model forecasts better by chance; 0 means the model has no
skill; 1 represents perfect skill. As shown in Table-II, both the models have
positive skill scores, with the ANN model performing marginally better than
CNN. Given that the models could capture the general trend with positive skill
scores, this implemented system lays the foundation for real-time peak
detection of forecasted $dB_{H}/dt$, which is crucial for predicting GMDs.

TABLE II: Heidke Skill Score(HSS) for $dB_{H}/dt$ forecasted by each model THRESHOLD (nT/s) | HSS (ANN $dB_{H}/dt$) | HSS (CNN $dB_{H}/dt$)
---|---|---
0.3 | 0.47 | 0.41
0.7 | 0.38 | 0.32
1.1 | 0.28 | 0.23
1.5 | 0.18 | 0.10

##  IV Conclusion and Future Work

This paper implements a low-cost ML-enabled magnetometer system that can
predict real-time baseline correction of $X$ and $Y$ geomagnetic components.
The system used an RM3100 magneto-inductive sensor connected to an iBUG
development board. Two different ML architectures were deployed, and their
real-time and offline prediction accuracy were examined against the baseline
removed data from a fluxgate magnetometer. Although the offline model
outperforms the real-time model, the real-time model captured the general
trend of the ground truth data from the fluxgate magnetometer. This, in turn,
opens up the possibility of peak detection in real-time using ML, which is a
crucial process for GMD detection. Going forward, distributed real-time ML
architectures like federated learning will also be explored.

## Acknowledgment

We thank members of the the “Machine-learning Algorithms for Geomagnetically
Induced Currents In Alaska and New Hampshire” (MAGICIAN) team. This work was
supported by NSF EPSCoR Award OIA-1920965.

## References

  * [1] E. Camporeale, J. R. Johnson, and S. Wing, Eds., _Machine learning techniques for space weather_. Amsterdam, Netherlands ; Cambridge, MA: Elsevier, 2018, oCLC: on1042806041.
  * [2] R. Pirjola, “Geomagnetically induced currents during magnetic storms,” _IEEE Transactions on Plasma Science_ , vol. 28, no. 6, pp. 1867–1873, Dec. 2000. [Online]. Available: http://ieeexplore.ieee.org/document/902215/
  * [3] G. S. Lakhina and B. T. Tsurutani, “Geomagnetic storms: historical perspective to modern view,” _Geoscience Letters_ , vol. 3, no. 1, p. 5, Dec. 2016\. [Online]. Available: https://geoscienceletters.springeropen.com/articles/10.1186/s40562-016-0037-4
  * [4] V. N. Rajput, D. H. Boteler, N. Rana, M. Saiyed, S. Anjana, and M. Shah, “Insight into impact of geomagnetically induced currents on power systems: Overview, challenges and mitigation,” _Electric Power Systems Research_ , vol. 192, p. 106927, Mar. 2021. [Online]. Available: https://linkinghub.elsevier.com/retrieve/pii/S0378779620307252
  * [5] S. K. Morley, D. T. Welling, and J. R. Woodroffe, “Perturbed Input Ensemble Modeling With the Space Weather Modeling Framework,” _Space Weather_ , vol. 16, no. 9, pp. 1330–1347, Sep. 2018. [Online]. Available: https://onlinelibrary.wiley.com/doi/10.1029/2018SW002000
  * [6] H. Lundstedt, “Progress in space weather predictions and applications,” _Advances in Space Research_ , vol. 36, no. 12, pp. 2516–2523, 2005, space Weather. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0273117705000104
  * [7] A. M. Keesee, V. Pinto, M. Coughlan, C. Lennox, M. S. Mahmud, and H. K. Connor, “Comparison of Deep Learning Techniques to Model Connections Between Solar Wind and Ground Magnetic Perturbations,” _Frontiers in Astronomy and Space Sciences_ , vol. 7, no. October, pp. 1–8, 2020.
  * [8] J. W. Gjerloev, “The SuperMAG data processing technique,” _Journal of Geophysical Research (Space Physics)_ , vol. 117, no. A9, p. A09213, Sep. 2012\.
  * [9] M. N. Schmidt, T. S. Alstrøm, M. Svendstorp, and J. Larsen, “Peak detection and baseline correction using a convolutional neural network,” in _ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2019, pp. 2757–2761.
  * [10] M. Maskey, “Nasa smd ai workshop report,” 2021.
  * [11] M. F. Yousuf, T. Siddique, and M. S. Mahmud, “iBUG: AI Enabled IoT Sensing Platform for Real-time Environmental Monitoring.” IEEE, 2022.
  * [12] A. Leuzinger and A. Taylor, “Magneto-inductive technology overview,” 2010.
  * [13] L. H. Regoli, M. B. Moldwin, M. Pellioni, B. Bronner, K. Hite, A. Sheinker, and B. M. Ponder, “Investigation of a low-cost magneto-inductive magnetometer for space science applications,” Magnetometers, preprint, Dec. 2017. [Online]. Available: https://gi.copernicus.org/preprints/gi-2017-53/gi-2017-53.pdf
  * [14] C. W. Smith, J. A. Blackwell, L. Ercoline, S. Goelzer, A. Johnson, H. Kucharek, M. Lessard, R. Levergood, W. D. Reeve, M. R. Routhier, N. Schwadron, A. S. Watson, and C. Young, “The Space Weather Underground: Design and Early Results from A Student-Built Array of Ground-Based Fluxgate Magnetometers in Northern New England,” in _AGU Fall Meeting Abstracts_ , vol. 2020, Dec. 2020, pp. SM011–12.
  * [15] U. E. Gssc, “Space weather underground.” [Online]. Available: https://swug.unh.edu/
  * [16] J. Peng, S. Peng, A. Jiang, J. Wei, C. Li, and J. Tan, “Asymmetric least squares for multiple spectra baseline correction,” _Analytica chimica acta_ , vol. 683, no. 1, pp. 63–68, 2010.
  * [17] A. Pulkkinen, L. Rastätter, M. Kuznetsova, H. Singer, C. Balch, D. Weimer, G. Toth, A. Ridley, T. Gombosi, M. Wiltberger, J. Raeder, and R. Weigel, “Community-wide validation of geospace model ground magnetic field perturbation predictions to support model transition to operations: GEOSPACE MODEL TRANSITION,” _Space Weather_ , vol. 11, no. 6, pp. 369–385, Jun. 2013. [Online]. Available: http://doi.wiley.com/10.1002/swe.20056
  * [18] V. A. Pinto, A. M. Keesee, M. Coughlan, R. Mukundan, J. W. Johnson, C. M. Ngwira, and H. K. Connor, “Revisiting the Ground Magnetic Field Perturbations Challenge: A Machine Learning Perspective,” _Frontiers in Astronomy and Space Sciences_ , vol. 9, p. 869740, May 2022\. [Online]. Available: https://www.frontiersin.org/articles/10.3389/fspas.2022.869740/full


#  µplan: Summarizing using a Content Plan as Cross-Lingual Bridge

Fantine Huot Joshua Maynez Chris Alberti
Reinald Kim Amplayo Priyanka Agrawal Constanza Fierro
Shashi Narayan Mirella Lapata

Google DeepMind
<EMAIL_ADDRESS>
###### Abstract

Cross-lingual summarization consists of generating a summary in one language
given an input document in a different language, allowing for the
dissemination of relevant content across speakers of other languages. However,
this task remains challenging, mainly because of the need for cross-lingual
datasets and the compounded difficulty of summarizing _and_ translating. This
work presents µplan, an approach to cross-lingual summarization that uses an
intermediate planning step as a cross-lingual bridge. We formulate the plan as
a sequence of entities that captures the conceptualization of the summary,
i.e. identifying the salient content and expressing in which order to present
the information, separate from the surface form. Using a multilingual
knowledge base, we align the entities to their canonical designation across
languages. µplan models first learn to generate the plan and then continue
generating the summary conditioned on the plan and the input. We evaluate our
methodology on the XWikis dataset on cross-lingual pairs across four languages
and demonstrate that this planning objective achieves state-of-the-art
performance in terms of ROUGE and faithfulness scores. Moreover, this planning
approach improves the zero-shot transfer to new cross-lingual language pairs
compared to non-planning baselines.

##  1 Introduction

Given a document or multiple documents in a source language (e.g., English),
cross-lingual summarization consists of producing a summary in a different
target language (e.g., Czech or German) Wang et al. (2022). It enables the
rapid dissemination of relevant content across speakers of other languages.
For instance, providing summaries of English news articles to Czech or German
speakers; or producing summary descriptions of products, services, and
knowledge only available in other languages.

Figure 1: Example source document in English, intermediate content plan in
English, and target summaries in Czech, German, and French.

Recent years have seen tremendous progress in abstractive summarization Rush
et al. (2015); Zhang et al. (2020) thanks to advances in neural network models
and the availability of large-scale datasets Sandhaus (2008); Hermann et al.
(2015); Grusky et al. (2018). While initial efforts have focused on English,
more recently, with the advent of cross-lingual representations Ruder et al.
(2019) and large pre-trained models Devlin et al. (2019); Liu et al. (2020),
research on multilingual summarization (i.e., building monolingual
summarization systems for different languages) has also gained momentum Chi et
al. (2020); Scialom et al. (2020); Aharoni et al. (2022).

Cross-lingual summarization faces the compounded challenge of having to tackle
difficulties relating to both monolingual summarization (e.g., long inputs and
outputs, faithfulness to the input documents Maynez et al. (2020)) _and_
challenges seen in machine translation (e.g., data imbalance, alignment across
languages Koehn and Knowles (2017)). Recent work has shown that introducing an
intermediate content planning step is helpful for summarization in English,
resulting in higher quality summaries, especially in terms of faithfulness
Puduppully et al. (2019a); Moryossef et al. (2019); Narayan et al. (2021,
2022); Huot et al. (2023). In this work, we present µplan, a cross-lingual
summarization method that uses such a content plan (i.e., what to say and in
what order) as a cross-lingual bridge (Figure 1).

Many hallucinations in summarization are tied to errors in the entities Zhao
et al. (2020); Maynez et al. (2020). Therefore, we build upon the work
described in Narayan et al. (2021) and express the content plans as ordered
sequences of salient entities. These entity chains serve as an intermediate
representation before the abstractive summary generation in the target
language. In particular, the planning step provides a way of sharing task-
specific knowledge across languages. Indeed, it formalizes some of the
summarization aspects of the task, i.e., identifying the salient content in
the source documents and expressing in which order to organize this
information, and standardizes it across different source and target language
pairs. The source document and target summary being in different languages
raises the question of which language to use for the content plan. Herein, we
use a multilingual knowledge base to align the entities across languages,
enabling us to canonically transpose the plan to different languages without
the use of machine translation.

We use a Transformer-based encoder-decoder Vaswani et al. (2017) that first
encodes the input document in the source language and then decodes to generate
$(i)$ an intermediate plan representation; and $(ii)$ the summary in target
language conditioned on the plan and the input. We evaluate our method on the
XWikis dataset Perez-Beltrachini and Lapata (2021), a cross-lingual
abstractive summarization dataset derived from
Wikipedia111https://www.wikipedia.org/ articles aligned across four different
languages (English, Czech, French, and German). We augment the training data
for fine-tuning by annotating each target summary with its corresponding
content plan. At inference time, the model must generate both the content plan
followed by the summary.

We investigate two distinct cross-lingual tasks: $(i)$ from English to other
languages ($\textsc{en}~{}\rightarrow~{}\textsc{all}$) and $(ii)$ from other
languages to English ($\textsc{all}~{}\rightarrow~{}\textsc{en}$). We
demonstrate that models fine-tuned with our planning objective outperform
regular generated summaries both in terms of ROUGE and faithfulness on the
XWikis dataset across all language pairs for both the
$\textsc{en}\rightarrow\textsc{all}$ and $\textsc{all}\rightarrow\textsc{en}$
tasks.

In addition, given the scarcity of cross-lingual datasets, we investigate the
zero-shot cross-lingual transfer to new language pairs. To do this we hold out
cross-lingual data from a specific language pair and include monolingual
summarization data instead. We demonstrate that the models with planning
perform better in this setting than non-planning models.

Our main contributions are as follows:

#### Planning as bridge between languages

We introduce a training objective for cross-lingual abstractive summarization
that does content planning through entity chains. It can be used as a fine-
tuning objective without modelling changes. This planning objective
standardizes the summarization task across different source and target
language pairs. It enables better quality summaries and better zero-shot
cross-lingual transfer to new language pairs than non-planning baselines.

#### Canonical transposition between languages

By leveraging a multilingual knowledge base to annotate the training data with
plans, we transpose all entity names to their canonical designation in all
languages, avoiding errors induced by mistranslation altogether. This
annotation strategy enables the mapping of entities that do not have an
equivalent name in the target language to fully-localized paraphrases.

##  2 Related Work

#### Cross-lingual Summarization

One of the challenges in cross-lingual summarization is the need for cross-
lingual datasets. Indeed, while creating large-scale multilingual
summarization datasets has proven feasible Straka et al. (2018); Scialom et
al. (2020), naturally occurring documents in a source language paired with
summaries in different target languages are rare. For this reason, existing
cross-lingual approaches create large-scale synthetic data using back-
translation Zhu et al. (2019); Cao et al. (2020) or translate the input
documents Ouyang et al. (2019).

Cross-lingual datasets include WikiLingua Ladhak et al. (2020), a dataset
derived from multilingual how-to guides, but how-to guides are relatively
short and their summaries are limited to brief instructional sentences. Cross-
Sum Bhattacharjee et al. (2021) contains over a million article and summary
samples, aligned from the multilingual XL-Sum Hasan et al. (2021) dataset, but
its summaries are limited to one or two sentences. Herein, we use the XWikis
dataset Perez-Beltrachini and Lapata (2021), a cross-lingual dataset derived
from Wikipedia, since their input documents and target summaries are
considerably longer.

#### Content Plans for Summarization

Content planning has proven helpful for conditional generation tasks by
breaking down the generation into smaller steps. Konstas and Lapata (2013)
proposed content plans based on grammar rules, and recent work has extended
content planning to neural models (Puduppully et al., 2019b; Moryossef et al.,
2019; Puduppully and Lapata, 2021). Lately, content plans have been proposed
for summarization and long-form question answering (Narayan et al., 2021,
2022). Our work builds upon the content planning approach proposed by Narayan
et al. (2021) which showed that models could generate more faithful and
entity-specific summaries when using an intermediate planning step as an
ordered sequence of salient entities. Herein, we explore whether such a
content plan can serve as a cross-lingual bridge and enable task transfer
between languages.

#### Zero-shot Cross-lingual Transfer

A substantial portion of the work in zero-shot cross-lingual transfer has
focused on classification tasks (Hu et al., 2020), such as XNLI (Artetxe and
Schwenk, 2019), part-of-speech tagging, dependency parsing, named entity
recognition (Ansell et al., 2021), and question answering (Conneau et al.,
2020). Whitehouse et al. (2022) performed data augmentation with code-switched
entities for a number of entity-centric downstream tasks. Some recent work has
investigated generative tasks in the zero-shot setting as well. Johnson et al.
(2017) prepended a special token to the input text to indicate the target
language of the translation and showed that this could be used for zero-shot
translation between languages pairs unseen during training. Chen et al. (2021)
proposed an approach for zero-shot cross-lingual machine translation where
they used parallel data in only one language pair and leveraged a multilingual
encoder to support inference in other languages. Vu et al. (2022) studied how
to fine-tune language models on only one language to perform zero-shot cross-
lingual summarization in other languages. They used prompt tuning (Lester et
al., 2021) and added unlabeled data in other languages to prevent the model
from catastrophically forgetting how to generate text in other languages.

##  3 Plans as a Cross-Lingual Bridge

###  3.1 Problem Formulation

We formalize the cross-lingual abstractive summarization task as follows:
Given an input document $d$ in a source language src, generate a summary $s$
in target language tgt. We model this as $p(s|d)$.

For the content planning objective, our goal is to teach the model to first
generate a content plan $c$ for the summary as $p(c|d)$, before generating the
summary as $p(s|c,d)$. Following Narayan et al. (2021), instead of modeling
$p(c|d)$ and $p(s|c,d)$ separately, we train the model to generate the
concatenated plan and summary sequence $c;s$. As a consequence, the model
first generates the content plan $c$ and then continues to generates the
summary $s$ conditioned on both $c$ and $d$. In the following section, we
describe how we annotate the data with content plans for this planning
objective.

###  3.2 Content Plans

Figure 2: Example summary with its salient entities highlighted in yellow.
Corresponding canonical entities in English are shown below, most times
matching the surface form in the summary (in red), other times canonical
versions have the same root (in green) but could differ greatly when entities
need disambiguation (in blue). Finally, the aligned German content plan is
shown at the bottom.

We formulate the content plan as an ordered sequence of canonical entities
extracted from the target summary. Figure 2 illustrates the content plan
annotation process. We annotate each summary with a content plan by extracting
the salient entities from the summary. In our case, we consider salient all
the entities that are important to mention when summarizing. We then link each
entity to its entry in a multilingual knowledge base.

We use the inter-language information from the knowledge base to standardize
the content plans across languages. We construct the content plan using the
canonical designation of each entity in both the source and target language,
removing morphology and selecting the most common designation out of multiple
aliases. The knowledge base also provides disambiguation when it is needed.
This annotation strategy avoids errors induced by translation. We use entities
names in the content plans, instead of indices, in order to leverage the
natural language knowledge that the language model inherently learned during
pretraining. Examples of the multilingual mappings in our annotated content
plans include:

    Lady Diana [EN] - Diana, Princess of Wales [DE]
    I, Robot [EN] - Les Robots [FR]
    dpartement [FR] - Department of France [EN]

In particular, we see that this strategy enables the mapping of entities that
do not have an equivalent name in the target language to fully-localized
names. This way the model learns to generate a content plan of localized
entities in the target language rather than translated entity plans.

##  4 Task Definition

###  4.1 Cross-Lingual Tasks

In what follows, let $\mathcal{L}$ be the set of all languages. We denote the
cross-lingual data as $\mathcal{D}_{\textsc{src}\rightarrow\textsc{tgt}}$,
i.e., $\mathcal{D}_{\textsc{en}\rightarrow\textsc{cs}}$ for Czech summaries
aligned with English inputs. Analogously, we denote the monolingual data as
$\mathcal{D}_{\textsc{lang}}$, i.e., $\mathcal{D}_{\textsc{cs}}$ for Czech
summaries with Czech inputs.

Herein, we investigate two specific cross-lingual tasks: $(i)$ from English to
other languages and $(ii)$ from other languages to English, which we denote as
$\textsc{en}\rightarrow\textsc{all}$ and $\textsc{all}\rightarrow\textsc{en}$,
respectively. The $\textsc{en}\rightarrow\textsc{all}$ task is particularly
interesting because it would make a large amount of English information
available to speakers of other languages. Therefore, the main focus of our
work is on the $\textsc{en}\rightarrow\textsc{all}$ task. However, it is a
challenging task because it involves a cross-lingual summarization model that
can generate fluent text in many languages.

We define the data mixture for the $\textsc{en}\rightarrow\textsc{all}$ task
as:

$\mathcal{D}_{\textsc{en}\rightarrow\textsc{all}}=\mathcal{D}_{\textsc{en}}\cup\bigcup_{\textsc{tgt}\in\mathcal{L}-\\{\textsc{en}\\}}\mathcal{D}_{\textsc{en}\rightarrow\textsc{tgt}},$

and for the $\textsc{all}\rightarrow\textsc{en}$, task as:

$\mathcal{D}_{\textsc{all}\rightarrow\textsc{en}}=\mathcal{D}_{\textsc{en}}\cup\bigcup_{\textsc{src}\in\mathcal{L}-\\{\textsc{en}\\}}\mathcal{D}_{\textsc{src}\rightarrow\textsc{en}}.$

Note that both tasks include the monolingual en data.

###  4.2 Zero-Shot Cross-Lingual Tasks

Given the scarcity of cross-lingual datasets, we investigate whether µplan can
help with the zero-shot cross-lingual transfer to new language pairs. For each
target language tgt, we perform the zero-shot transfer experiments on the
$\textsc{en}\rightarrow\textsc{all}$ task by holding out the
$\textsc{en}\rightarrow\textsc{tgt}$ cross-lingual data during fine-tuning. We
then evaluate performance on the $\textsc{en}\rightarrow\textsc{tgt}$ test
data. To ensure that the model maps the language token to the correct language
and to prevent catastrophic forgetting of the tgt language during fine-tuning
Vu et al. (2022), we include the tgt monolingual summarization data to the
fine-tuning data mixture, under the assumption that the monolingual data is
easier to come by than cross-lingual data. We denote this zero-shot cross-
lingual transfer task as $\textsc{en}\rightarrow\textsc{tgt}_{\textsc{zs}}$
and define it as follows:

$\mathcal{D}_{\textsc{en}\rightarrow\textsc{tgt}_{\textsc{zs}}}=\mathcal{D}_{\textsc{en}}\cup\mathcal{D}_{\textsc{tgt}}\cup\bigcup_{\textsc{l}\in\mathcal{L}-\\{\textsc{en,tgt}\\}}\mathcal{D}_{\textsc{en}\rightarrow\textsc{l}}.$

##  5 Experimental Setup

###  5.1 Dataset

The XWikis dataset Perez-Beltrachini and Lapata (2021) was created from
Wikipedia articles under the assumption that the body and lead paragraph
constitute a document-summary pair. Cross-lingual document-summary instances
were derived by combining lead paragraphs and articles’ bodies from language-
aligned Wikipedia titles. Although XWikis covers only four languages, English
(en), Czech (cs), German (de), and French (fr), the dataset creation procedure
is general and applicable to any languages represented in Wikipedia.

| Train | Validation | Test
---|---|---|---
en | 624,178 | 8,194 | 7,000
$\textsc{en}\rightarrow\textsc{cs}$ | 134,996 | 250† | 6,855†
$\textsc{en}\rightarrow\textsc{de}$ | 409,012 | 250† | 9,750†
$\textsc{en}\rightarrow\textsc{fr}$ | 451,964 | 250† | 9,727†
$\textsc{cs}\rightarrow\textsc{en}$ | 48,519 | 2,549 | 6,999
$\textsc{de}\rightarrow\textsc{en}$ | 344,438 | 18,160 | 6,999
$\textsc{fr}\rightarrow\textsc{en}$ | 283,182 | 14,899 | 6,992
Table 1: Number of data samples in the XWikis dataset for the language pairs
and data splits considered in this work. The original
$\textsc{en}\rightarrow\textsc{all}$ cross-lingual language pairs do not have
a test split, therefore we separate the validation split into two, taking the
first 250 examples for validation and the rest for testing. These edited
splits are marked by †.

Table 1 shows the number of data samples for each language pair. All the
cross-lingual language pairs in the $\textsc{all}\rightarrow\textsc{en}$
setting have separate training, validation and test splits, but in the
$\textsc{en}\rightarrow\textsc{all}$ setting there are only training and
validation splits. Therefore, for all the $\textsc{en}\rightarrow\textsc{all}$
cross-lingual language pairs, we separate the validation split into two,
taking the first 250 examples for validation and the rest for testing.

The XWikis dataset provides the input documents as a list of section titles
and paragraphs that constitute the body of the Wikipedia article to summarize.
We format the input documents by concatenating the titles and paragraphs,
marking each title with an end-of-title token eot and each paragraph with an
end-of-paragraph token eop. We prepend the source language code and target
language code to the input document for each cross-lingual document and
summary pair.

Since the XWikis dataset is derived from Wikipedia, we annotate the plans by
extracting the entities from the summaries using the embedded hyperlinks that
point to Wikidata222https://www.wikidata.org/ knowledge base entries. Not all
the links correspond to entities, and we exclude those pointing to phonetic
pronunciations. The XWikis dataset was generated from a Wikipedia data dump
from 2016 and we used one from 2023 for extracting the hyperlinks from the
summaries. Therefore, for articles that went through significant changes
between 2016 and 2023, the pages were not aligned and we did not annotate
these examples with content plans. Depending on the language pair, this
problem affects between 2.57% ($\textsc{de}\rightarrow\textsc{en}$) and 5.21%
(en) of the training data. We create a _filtered_ version of the training data
that excludes these examples with missing content plans.

###  5.2 Modeling Approaches

We demonstrate µplan on both the $\textsc{en}\rightarrow\textsc{all}$ and
$\textsc{all}\rightarrow\textsc{en}$ tasks and compare it with a number of
different modeling approaches.

#### Machine Translation

A common approach to cross-lingual summarization is to use a pipeline approach
that relies on machine translation. Machine translation can be used in two
ways: $(i)$ first translate the original document into the target language and
then summarize the translated document or $(ii)$ first summarize the original
document and then translate the summary Ouyang et al. (2019); Wan et al.
(2010); Ladhak et al. (2020). We denote the former approach as translate-train
and the latter as translate-test. We perform the machine translation with
Google Translate.

Previous work Kramchaninova and Defauw (2022); Vu et al. (2022) has
highlighted various limitations with these approaches such as $(i)$ their
dependence on the quality of available machine translation systems in a given
language and in turn the availability of high-quality parallel data, $(ii)$ a
potential misalignment of the data after translation, and $(iii)$
translationese artifacts Clark et al. (2020).

#### End-to-end Cross-lingual Summarization

This approach consists of directly fine-tuning on the cross-lingual data. It
matches the one described in Perez-Beltrachini and Lapata (2021). We denote
this approach as baseline. This approach avoids the potential error
propagation problem of machine translation pipeline systems.

#### µplan

We experiment with different plan formulations to establish which type of plan
performs well as a cross-lingual bridge. The language of the source document
being different from the language of the target summary raises the question of
which language to use for the content plans. We explore variations of the
language of the plans: $(i)$ source language, $(ii)$ target language; and
plans containing entities in both languages: $(iii)$ source & target language.
Due to the nature of auto-regressive generation, the mixed language plans are
constructed by concatenating the aligned entity names in source and target
language before combining them into a sequence, instead of concatenating the
sequence in the source language followed by the same sequence in the target
language. This concatenation strategy avoids long distance relations in the
language modeling. Table 2 presents examples of different language plans. By
default, we refer to µplan as using the mixed language plans $(iii)$.

Plan Type |  Predicted Plan |  Gold Plan
---|---|---
src[en] |  Dutch | fortification | Banda Neira | Maluku Islands | Netherlands | Dutch East Indies |  Banda Neira | Banda Islands | Maluku Islands | Indonesia | Maluku | nutmeg
tgt[de] |  Estland | Folk Metal | Band | Tallinn | Markus Lõhmus |  Estland | Folk Metal | Euphemismus | Wolf
src[en]_tgt[fr] |  county seat & siège de comté | Crawford County & comté de Crawford | Arkansas & Arkansas | United States of America & États-Unis |  Arkansas & Arkansas | United States of America & États-Unis
Table 2: Examples of generated and gold content plans for different plan
languages.

Additionally, we experiment with the internal constitution of the plans: we
provide the length of the gold plan during training [length], and shuffle
entities to investigate the importance of the sequence order [shuffle].

#### Model Training

The models are based on the mT5 model Xue et al. (2021). We fine-tune the XL
3B-parameter model with a maximum input and output sequence lengths of 2,048
and 256 tokens, respectively. The models are finetuned with a learning rate of
0.002, a batch size of 128, up to 80,000 steps, evaluating every 1000 steps.
We select the best checkpoints by measuring ROUGE-L on 250 examples of the
validation split for each language pair and take the best unweighted average
across all language pairs. We limit the evaluation data size for faster
computation.

###  5.3 Evaluation

Automatic evaluation is required in three different dimensions of quality:
summary relevance, summary faithfulness, and content plan relevance.
Evaluating these dimensions in the cross-lingual setup has certain challenges,
which we describe below.

For summary relevance, we use ROUGE Lin (2004) to compare system-generated
summaries with gold-standard ones. Since the availability of word tokenizers
differs for non-English languages, we follow the methodology dscribed in
Aharoni et al. (2022) and use a SentencePiece tokenizer Kudo and Richardson
(2018) trained on mC4 Xue et al. (2021), which can tokenize texts written in
101 languages, including all the languages in our experiments. In Section 6,
we denote ROUGE-1, ROUGE-2, and ROUGE-L as R1, R2, and RL.

In terms of summary faithfulness, following Honovich et al. (2022), we employ
an entailment classifier that predicts whether the input document supports the
output summary. In line with previous work Narayan et al. (2022); Schuster et
al. (2022), we split the summary into sentences for a more fine-grained
evaluation. We predict the entailment of each sentence and average the
entailment scores. The classifier is an mT5-XXL model Xue et al. (2021)
trained on XNLI Conneau et al. (2018), a multilingual NLI dataset. There are
currently no cross-lingual datasets for NLI, however our preliminary analysis
reported in Appendix A shows that an XNLI-trained mT5 model works well in
predicting cross-lingual entailment. It has the added benefit of avoiding
potential error propagation from introducing a machine translation step in the
evaluation process.

Finally, for evaluating plan relevance, we evaluate the generated content
plans against the gold-standard ones. We compute F1 scores on the entities in
the predicted summaries against the corresponding reference entities. In
Section 6, we denote as F1 this F1 score computed on the plan entities.

##  6 Results

| $\textsc{en}\rightarrow\textsc{all}$ | $\textsc{all}\rightarrow\textsc{en}$
---|---|---
| R1 / R2 / RL | XNLI | R1 / R2 / RL | XNLI
translate-train | 44.83 / 28.56 / 37.42 | 42.32 | – | –
translate-test | 39.89 / 20.74 / 30.41 | 43.94 | – | –
baseline | 44.54 / 28.57 / 37.40 | 42.75 | 43.54 / 23.44 / 33.79 | 37.58
_filtered_ | 44.87 / 28.63 / 37.56 | 41.77 | 43.51 / 23.55 / 33.92 | 37.43
µplan | 45.75 / 29.40 / 38.30 | 47.39 | 43.56 / 24.19 / 34.36 | 43.00
Table 3: Overview of the cross-lingual summarization results for both the $\textsc{en}\rightarrow\textsc{all}$ and $\textsc{all}\rightarrow\textsc{en}$ tasks. Best results are boldfaced. | $\textsc{en}\rightarrow\textsc{all}$ | $\textsc{all}\rightarrow\textsc{en}$
---|---|---
| en | $\textsc{en}\rightarrow\textsc{cs}$ | $\textsc{en}\rightarrow\textsc{de}$ | $\textsc{en}\rightarrow\textsc{fr}$ | en | $\textsc{cs}\rightarrow\textsc{en}$ | $\textsc{de}\rightarrow\textsc{en}$ | $\textsc{fr}\rightarrow\textsc{en}$
trans.-train | 37.42 | 32.81 | 38.28 | 41.19 | – | – | – | –
trans.-test | 35.54 | 26.26 | 28.47 | 31.59 | – | – | – | –
baseline | 37.57 | 32.74 | 38.58 | 41.36 | 35.47 | 33.30 | 33.70 | 33.22
µplan | 39.53 | 33.18 | 38.94 | 41.57 | 36.09 | 32.82 | 34.32 | 34.20
$\textsc{xwikis}^{\dagger}$ | – | – | – | – | 32.53 | 33.72 | 32.95 | 31.84
Table 4: ROUGE-L cross-lingual summarization results per individual language
pair for both the $\textsc{en}\rightarrow\textsc{all}$ and
$\textsc{all}\rightarrow\textsc{en}$ tasks. The bottom row, marked by †,
reports the results on the XWikis task by Perez-Beltrachini and Lapata (2021).
Best results are boldfaced.

#### Value of µplan planning

Table 3 presents an overview of the cross-lingual summarization results for
both the $\textsc{en}\rightarrow\textsc{all}$ and
$\textsc{all}\rightarrow\textsc{en}$ tasks. The reported scores are averaged
over the individual language pairs. We see that µplan consistently outperforms
both the translation-based approaches and the non-planning baseline in terms
of ROUGE and faithfulness scores on both $\textsc{en}\rightarrow\textsc{all}$
and $\textsc{all}\rightarrow\textsc{en}$ tasks. This result shows that an
intermediate content planning step is valuable for cross-lingual
summarization.

There is little difference overall between the _filtered_ and non-filtered
training data, which is expected since the difference in the number of samples
is relatively small (of the order of 5%). Therefore, in what follows, we
simplify our notation and only report metrics obtained on the filtered data,
since µplan models only use the filtered data.

Table 4 shows the detail of the results per individual language pairs. We see
that the previous results hold when evaluated per language pair. We compare
our results with those reported by Perez-Beltrachini and Lapata (2021) and see
that µplan establishes new state-of-the-art ROUGE results on the XWikis
benchmark. They trained their models on individual language pairs and the
models they used have a smaller number of parameters than the mT5 models we
use in our study, which explains some of the gains we observe.

| RL | XNLI | F1
---|---|---|---
µplan | 38.30 | 47.39 | 0.27
µplan${}_{\textsc{src}}$ | 38.14 | 47.72 | 0.27
µplan${}_{\textsc{tgt}}$ | 37.97 | 47.37 | 0.26
µplan${}_{\textsc{length}}$ | 37.09 | 45.71 | 0.25
µplan${}_{\textsc{shuffle}}$ | 38.01 | 46.25 | 0.17
µplanoracle | 48.28 | – | 1.00
µplan${}_{\textsc{src}}^{oracle}$ | 47.96 | – | 1.00
µplan${}_{\textsc{tgt}}^{oracle}$ | 48.13 | – | 1.00
Table 5: Comparison of different µplan plan formulations on the
$\textsc{en}\rightarrow\textsc{all}$ task. Best results are boldfaced.

#### Best plans include entities in both the source and target language

We compare different types of plan formulations on the
$\textsc{en}\rightarrow~{}\textsc{all}$ task and report our results in Table
5. We find that mixed language plans that contain entities in both the source
and target language, which is the default µplan setting, deliver better
results than the plans with entities in only one language (marked here as src
and tgt) . Table 2 shows some plans generated by µplan under these different
settings and compares them to the gold ones. For a given summary, we find both
overlapping and diverse entities among the plans.

On average, we find that the predicted plans have around 20% shorter plans in
comparison to the entity plans extracted from gold summaries. Their F1 scores
are around 0.27, but we do not expect perfect overlap between generated and
gold plans. Indeed, since the gold summaries in XWikis are derived from the
lead paragraphs of Wikipedia articles, some of the entities in the gold plans
might not appear in the source document.

Providing information about the length of the gold plan during training,
reported as length, does not affect the results very much and actually yields
slightly lower metrics than the default µplan setup. The shuffle metrics, for
which the entity order is shuffled, are also slightly lower than the default
setup. This result indicates that the order of the entities matters for
planning the summary generation. The plan F1 scores are lower, which is
expected since F1 is sensitive to the order of the entities.

#### Oracle plan summaries

For comparison, we report results when models have access to oracle content
plans, which we denote as _oracle_. At inference time, the encoder first
encodes the source document, while the decoder gets the gold plan as a forced
prompt before generating the summary. These oracle experiments provide an
upper bound of how these models would perform in a best case scenario. In
Table 5, we see that the oracle metrics are higher by a wide margin, of around
10 ROUGE-L points, from the best predicted results. This behavior is expected
and shows that models can correctly generate summaries in target language from
plans in the target language but also from aligned English plans. Moreover, it
shows that µplan’s mixed language plans provide additional information that
models can leverage effectively.

|  | RL | XNLI
---|---|---|---
baseline | ${\textsc{en}\rightarrow~{}\textsc{cs}_{\textsc{zs}}}$ | 15.10 | 34.95
${\textsc{en}\rightarrow~{}\textsc{de}_{\textsc{zs}}}$ | 17.50 | 48.80
${\textsc{en}\rightarrow~{}\textsc{fr}_{\textsc{zs}}}$ | 18.54 | 45.51
µplan | ${\textsc{en}\rightarrow~{}\textsc{cs}_{\textsc{zs}}}$ | 18.64 | 39.04
${\textsc{en}\rightarrow~{}\textsc{de}_{\textsc{zs}}}$ | 19.18 | 45.51
${\textsc{en}\rightarrow~{}\textsc{fr}_{\textsc{zs}}}$ | 23.61 | 45.96
Table 6: Zero-shot cross-lingual transfer experiments. Best results are
boldfaced.

####  µplan enables zero-shot transfer to new language pairs

Table 6 shows the results of the zero-shot cross-lingual transfer experiments.
We see that µplan results in higher ROUGE-L when evaluated on an unseen
language pair. This result demonstrate that an intermediate planning step can
enable the transfer of task knowledge to new language pairs.

##  7 Discussion

µplan can be applied without requiring any modelling changes, making it a
versatile approach to cross-lingual summarization. While the XWikis dataset
covers only four languages, the methodology for generating the dataset can be
extended to any language pairs available in Wikipedia Perez-Beltrachini and
Lapata (2021). In this work, we use the embedded hyperlinks in Wikipedia
articles to extract the salient entities and align them on the Wikidata
knowledge base. With recent entity annotation systems such as REFINED Ayoola
et al. (2022), this same operation can be applied on out-of-domain data,
including the multilingual alignment of the entity names.

The content plans are expressed in natural language and thus human-
interpretable, unlike intermediate representations in latent variables. As a
consequence, this approach lends itself to plan editing approaches, such as
selective filtering of the entities at inference time or human-in-the-loop
plan editing as described by Narayan et al. (2021, 2022); Huot et al. (2023).
Using forced prompting methods as described in the oracle experiments, would
also allow for providing localized entity names at inference time from a
knowledge base.

The planning step separates the salient content selection from the source
document from the surface form generation in the target language, unlocking
better multitasking capabilities across language pairs. It enables better use
of a small amount of cross-lingual data, allowing for better transfer to new
cross-lingual language pairs. We cannot realistically expect to have large-
scale cross-lingual data for fine-tuning our language models on all possible
language pairs. µplan is one way of performing the cross-lingual task
transfer.

##  8 Conclusion

µplan is an approach to cross-lingual summarization that uses an intermediate
planning step as a cross-lingual bridge. Since hallucinations and
mistranslations in cross-lingual summarization are often tied to incorrect
entities, we formulate the content plan as a sequence of entities. This entity
content plan captures the conceptualization of the summary, i.e. identifying
the salient content and expressing in which order to present the information,
separate from the surface form generation in the target language. We evaluate
our methodology on the XWikis dataset and demonstrate that this planning
objective achieves state-of-the-art performance both in the $(i)$ English to
all languages and $(ii)$ all languages to English cross-lingual settings.
Moreover, we demonstrate that µplan enables the zero-shot cross-lingual
transfer to new language pairs.

## References

  * Aharoni et al. (2022) Roee Aharoni, Shashi Narayan, Joshua Maynez, Jonathan Herzig, Elizabeth Clark, and Mirella Lapata. 2022.  mface: Multilingual summarization with factual consistency evaluation.  _arXiv preprint arXiv:2212.10622_.
  * Ansell et al. (2021) Alan Ansell, Edoardo Maria Ponti, Jonas Pfeiffer, Sebastian Ruder, Goran Glavaš, Ivan Vulić, and Anna Korhonen. 2021.  MAD-G: Multilingual adapter generation for efficient cross-lingual transfer.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4762–4781, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Artetxe and Schwenk (2019) Mikel Artetxe and Holger Schwenk. 2019.  Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond.  _Transactions of the Association for Computational Linguistics_ , 7:597–610.
  * Ayoola et al. (2022) Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022.  ReFinED: An efficient zero-shot-capable approach to end-to-end entity linking.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Industry Track_ , pages 209–220, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
  * Bhattacharjee et al. (2021) Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ahmad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2021.  Crosssum: Beyond english-centric cross-lingual abstractive text summarization for 1500+ language pairs.  _arXiv preprint arXiv:2112.08804_.
  * Cao et al. (2020) Yue Cao, Hui Liu, and Xiaojun Wan. 2020.  Jointly learning to align and summarize for neural cross-lingual summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6220–6231, Online. Association for Computational Linguistics.
  * Chen et al. (2021) Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, and Furu Wei. 2021.  Zero-shot cross-lingual transfer of neural machine translation with multilingual pretrained encoders.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 15–26, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chi et al. (2020) Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, and Heyan Huang. 2020\.  Cross-lingual natural language generation via pre-training.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence_ , pages 7570–7577. AAAI Press.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Grusky et al. (2018) Max Grusky, Mor Naaman, and Yoav Artzi. 2018.  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 708–719, New Orleans, Louisiana. Association for Computational Linguistics.
  * Hasan et al. (2021) Tahmid Hasan, Abhik Bhattacharjee, Md Saiful Islam, Kazi Samin, Yuan-Fang Li, Yong-Bin Kang, M Sohel Rahman, and Rifat Shahriyar. 2021.  Xl-sum: Large-scale multilingual abstractive summarization for 44 languages.  _arXiv preprint arXiv:2106.13822_.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28_ , pages 1693–1701. Curran Associates, Inc.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  TRUE: Re-evaluating factual consistency evaluation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3905–3920, Seattle, United States. Association for Computational Linguistics.
  * Hu et al. (2020) Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020.  XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 4411–4421. PMLR.
  * Huot et al. (2023) Fantine Huot, Joshua Maynez, Shashi Narayan, Reinald Kim Amplayo, Kuzman Ganchev, Annie Priyadarshini Louis, Anders Sandholm, Dipanjan Das, and Mirella Lapata. 2023.  Text-blueprint: An interactive platform for plan-based conditional generation.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 105–116, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017.  Google’s multilingual neural machine translation system: Enabling zero-shot translation.  _Transactions of the Association for Computational Linguistics_ , 5:339–351.
  * Koehn and Knowles (2017) Philipp Koehn and Rebecca Knowles. 2017.  Six challenges for neural machine translation.  In _Proceedings of the First Workshop on Neural Machine Translation_ , pages 28–39, Vancouver. Association for Computational Linguistics.
  * Konstas and Lapata (2013) Ioannis Konstas and Mirella Lapata. 2013.  Inducing document plans for concept-to-text generation.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1503–1514, Seattle, Washington, USA. Association for Computational Linguistics.
  * Kramchaninova and Defauw (2022) Alina Kramchaninova and Arne Defauw. 2022.  Synthetic data generation for multilingual domain-adaptable question answering systems.  In _Proceedings of the 23rd Annual Conference of the European Association for Machine Translation_ , pages 151–160, Ghent, Belgium. European Association for Machine Translation.
  * Kudo and Richardson (2018) Taku Kudo and John Richardson. 2018.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 66–71, Brussels, Belgium. Association for Computational Linguistics.
  * Ladhak et al. (2020) Faisal Ladhak, Esin Durmus, Claire Cardie, and Kathleen McKeown. 2020.  WikiLingua: A new benchmark dataset for cross-lingual abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4034–4048, Online. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2020) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020.  Multilingual denoising pre-training for neural machine translation.  _Transactions of the Association for Computational Linguistics_ , 8:726–742.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Moryossef et al. (2019) Amit Moryossef, Yoav Goldberg, and Ido Dagan. 2019.  Step-by-step: Separating planning from realization in neural data-to-text generation.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2267–2277, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Narayan et al. (2022) Shashi Narayan, Joshua Maynez, Reinald Kim Amplayo, Kuzman Ganchev, Annie Louis, Fantine Huot, Dipanjan Das, and Mirella Lapata. 2022.  Conditional generation with a question-answering blueprint.  _arXiv preprint arXiv:2207.00397_.
  * Narayan et al. (2021) Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, and Ryan McDonald. 2021.  Planning with learned entity prompts for abstractive summarization.  _Transactions of the Association for Computational Linguistics_ , 9:1475–1492.
  * Ouyang et al. (2019) Jessica Ouyang, Boya Song, and Kathy McKeown. 2019.  A robust abstractive system for cross-lingual summarization.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2025–2031, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Perez-Beltrachini and Lapata (2021) Laura Perez-Beltrachini and Mirella Lapata. 2021.  Models and datasets for cross-lingual summarisation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9408–9423, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Puduppully et al. (2019a) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019a.  Data-to-text generation with content selection and planning.  In _Proceedings of the 33rd AAAI Conference on Artificial Intelligence_ , Honolulu, Hawaii.
  * Puduppully et al. (2019b) Ratish Puduppully, Li Dong, and Mirella Lapata. 2019b.  Data-to-text generation with content selection and planning.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 6908–6915.
  * Puduppully and Lapata (2021) Ratish Puduppully and Mirella Lapata. 2021.  Data-to-text generation with macro planning.  _Transactions of the Association for Computational Linguistics_ , 9:510–527.
  * Ruder et al. (2019) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2019.  A survey of cross-lingual word embedding models.  _Journal of Artificial Intelligence Research_ , 65(1):569–630.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Lisbon, Portugal. Association for Computational Linguistics.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The New York Times Annotated Corpus.  _Linguistic Data Consortium, Philadelphia_ , 6(12).
  * Schuster et al. (2022) Tal Schuster, Sihao Chen, Senaka Buthpitiya, Alex Fabrikant, and Donald Metzler. 2022.  Stretching sentence-pair NLI models to reason over long documents and clusters.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 394–412, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Scialom et al. (2020) Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2020.  MLSUM: The multilingual summarization corpus.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8051–8067, Online. Association for Computational Linguistics.
  * Straka et al. (2018) Milan Straka, Nikita Mediankin, Tom Kocmi, Zdeněk Žabokrtský, Vojtěch Hudeček, and Jan Hajič. 2018.  SumeCzech: Large Czech news-based summarization dataset.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , NIPS’17, page 6000–6010, Red Hook, NY, USA. Curran Associates Inc.
  * Vu et al. (2022) Tu Vu, Aditya Barua, Brian Lester, Daniel Cer, Mohit Iyyer, and Noah Constant. 2022\.  Overcoming catastrophic forgetting in zero-shot cross-lingual generation.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9279–9300, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wan et al. (2010) Xiaojun Wan, Huiying Li, and Jianguo Xiao. 2010.  Cross-language document summarization based on machine translation quality prediction.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_ , pages 917–926, Uppsala, Sweden. Association for Computational Linguistics.
  * Wang et al. (2022) Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022.  A survey on cross-lingual summarization.  _Transactions of the Association for Computational Linguistics_ , 10:1304–1323.
  * Whitehouse et al. (2022) Chenxi Whitehouse, Fenia Christopoulou, and Ignacio Iacobacci. 2022.  EntityCS: Improving zero-shot cross-lingual transfer with entity-centric code switching.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6698–6714, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 483–498, Online. Association for Computational Linguistics.
  * Zhang et al. (2020) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020.  PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 11328–11339. PMLR.
  * Zhao et al. (2020) Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020.  Reducing quantity hallucinations in abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2237–2249, Online. Association for Computational Linguistics.
  * Zhu et al. (2019) Junnan Zhu, Qian Wang, Yining Wang, Yu Zhou, Jiajun Zhang, Shaonan Wang, and Chengqing Zong. 2019.  NCLS: Neural cross-lingual summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3054–3064, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Cross-lingual NLI

Table 7 compares different ways of computing NLI. It is computed on the
summaries generated by the baseline model on the
$\textsc{en}\rightarrow\textsc{all}$ and
$\textsc{all}~{}\rightarrow~{}\textsc{en}$ tasks. The first setting, denoted
as ANLI, is the English setting, for which we translate the non-English
document ($\textsc{all}~{}\rightarrow~{}\textsc{en}$) or summary
($\textsc{en}\rightarrow\textsc{all}$) to English and apply an NLI model
trained on an English corpus. The second one is the multilingual NLI setting,
which we denote as XNLI-m. For the cross-lingual language pairs, we translate
the English document or summary such that both document and summary are in the
same language (which is either the source or target language, depending on
whether it is the $\textsc{en}\rightarrow\textsc{all}$ or
$\textsc{all}~{}\rightarrow~{}\textsc{en}$ task). We then apply a multilingual
NLI model. The last setting is the cross-lingual setting, which we denote as
XNLI-x. In this setting, we do not use translation, and directly apply the
multilingual NLI model to the cross-lingual data.

|  | ANLI | XNLI-m | XNLI-x
---|---|---|---|---
$\textsc{en}\rightarrow\textsc{all}$ | en | 54.04 | – | 53.63
$\textsc{en}\rightarrow\textsc{cs}$ | 32.09 | 31.15 | 35.88
$\textsc{en}\rightarrow\textsc{de}$ | 38.47 | 39.89 | 40.15
$\textsc{en}\rightarrow\textsc{fr}$ | 43.09 | 35.74 | 41.32
$\textsc{all}\rightarrow\textsc{en}$ | en | 57.91 | – | 53.05
$\textsc{cs}\rightarrow\textsc{en}$ | 34.73 | 32.95 | 29.74
$\textsc{de}\rightarrow\textsc{en}$ | 40.28 | 38.64 | 35.12
$\textsc{fr}\rightarrow\textsc{en}$ | 37.28 | 35.71 | 32.40
Table 7: Entailment metrics on English, multilingual, and cross-lingual
settings.