#  GAE-ISumm: Unsupervised Graph-Based Summarization of Indian Languages

Lakshmi Sireesha Vakada1, Anudeep Ch1, Mounika Marreddy1, Subba Reddy Oota2,
Radhika Mamidi1 1 IIIT Hyderabad, 2 Inria Bordeaux, France
{lakshmi.sireesha<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>and
<EMAIL_ADDRESS>
###### Abstract

Document summarization aims to create a precise and coherent summary of a text
document. Many deep learning summarization models are developed mainly for
English, often requiring a large training corpus and efficient pre-trained
language models and tools. However, English summarization models for low-
resource Indian languages are often limited by rich morphological variation,
syntax, and semantic differences. In this paper, we propose GAE-ISumm, an
unsupervised Indic summarization model that extracts summaries from text
documents. In particular, our proposed model, GAE-ISumm uses Graph Autoencoder
(GAE) to learn text representations and a document summary jointly. We also
provide a manually-annotated Telugu summarization dataset TELSUM, to
experiment with our model GAE-ISumm. Further, we experiment with the most
publicly available Indian language summarization datasets to investigate the
effectiveness of GAE-ISumm on other Indian languages. Our experiments of GAE-
ISumm in seven languages make the following observations: (i) it is
competitive or better than state-of-the-art results on all datasets, (ii) it
reports benchmark results on TELSUM, and (iii) the inclusion of positional and
cluster information in the proposed model improved the performance of
summaries. We open-source our dataset and code
111https://github.com/scsmuhio/Summarization.

###### Index Terms:

component, formatting, style, styling, insert

##  I Introduction

Document summarization aims to minimize the content in a text document and
preserve the salient information. There are usually two categories of
summarization techniques: Extractive [1] and Abstractive [2]. Extractive
summarization extracts salient text from the document. Whereas Abstractive
summarization concisely paraphrases the information contained in the document.
Understanding text’s contextual and semantic representation is one of the main
challenges for effective summaries. Traditional methods extract summaries
based on hand-crafted features, including Term Frequency [3], Sentence
Position, and Length [4], and largely depend on the availability of NLP tools.

Significant progress has been made in single document extractive summarization
by using recent popular deep learning models such as RNNs [1], CNNs [5],
attention-based models [6], and sequence to sequence models [7]. Also, most of
these models follow encoder-decoder-based approaches to generate summaries in
a supervised or unsupervised setting [8]. However, these models lack in
capturing the global context and the long-distance sentence relationships
present in a document. Modeling long-range inter-sentence relationships with
transformer-based models are still challenging [9] and require massive
computation and memory. By capturing the long-term dependencies and treating
the document as a graph[10]. , graph-based approaches assist in overcoming
these limitations.

Recently, Graph based models have been used to capture the cross-sentence
relationships for summarization [11, 12]. They have also proven their
dominance in other fields such as classification [13, 14], and semantic role
labelling [15]. Graph-based models are capable of drawing syntactic
information, exploiting long-range multi-word relations, and have been
deployed on document-word relationships [16, 17].

Figure 1:  Outline of GAE-ISumm. Our model GAE-ISumm involves two phases:
Document Encoding - a) Document Graph Construction and b) Obtaining Graph-
based Representations; Sentence Embedding and Summary Generation involves
seven steps starting from 1) Sentence Graph Construction to 7) Loss
calculation. Figure 2:  GAE model Figure 3:  Working of GRU

To make our model more effective and computation friendly, we get to the idea
of a recently proposed method, Graph Autoencoder (GAE) [18]. GAE here captures
the hidden semantic information between documents and sentences by using the
idea of an autoencoder (AE) [19] to graph-structured data. Few recent papers
have also obtained benchmark results on text classification using GAEs [20] .
However, the application of GAE to document summarization is an unexplored
area. Also, adopting an AE or GAE trained on English corpora for Indian
languages have significant limitations: (i) lack of such large-scale human-
annotated datasets, (ii) rich morphological variation, and (iii) syntactic and
semantic differences. Moreover, due to the dearth of various qualitative tools
and scarcity of annotated data, summarization models are not well studied for
low-resource Indian languages.

Unlike English, few preparatory works have studied text summarization for
Indian languages. In these works, the authors use existing methods such as
keyword extraction [21], hand-crafted features [22], and TextRank algorithm
[23] to extract the summaries. In another recent work, [24] proposes a
heuristic model based on the frequency score of named entities and the
vocabulary of the document to produce summaries. Unfortunately, all these
works were limited to existing baseline models that rank sentences based on
term frequency or similarity heuristics. Also, none of the works focused on
learning sentence or document representations, which is crucial for an
effective summarization model. In this scenario, unsupervised approaches are
alluring as they do not require any labeled data for training.

Inspired by the GAE [18], SummPip [25], and Salience Score Estimation [26], we
propose GAE-ISumm: an unsupervised extractive summarization model that jointly
optimizes the loss between document representations, sentence representations
and generated summaries to showcase better performance. More importantly, our
proposed model simultaneously learns the sentence representations and summary
of the document, while earlier methods extract summaries after learning the
sentence representations. Fig. 1 illustrates our proposed method, GAE-ISumm.

Our main contributions are as follows:

  * •

We propose GAE-ISumm, an unsupervised model that learns a summary from the
input document. Also, to the best of our knowledge, we are the first to apply
GAE for the summarization task.

  * •

We formulate the problem as a graph network to learn sentence and document
representations using GAE and perform text summarization jointly with our
proposed method.

  * •

To the best of our knowledge, we are the first to investigate the
effectiveness of graph-based embeddings for different Indian languages in an
unsupervised setting.

  * •

We further introduce TELSUM, a manually annotated Telugu summarization dataset
of 501 document-summary pairs.

  * •

We experimented our proposed model, GAE-ISumm, on different datasets with
monolingual and bilingual settings across seven Indian languages.

This paper aims to bridge the gap by creating models and resources for the
summarization task of Indian languages. The proposed method, GAE-ISumm, can be
extended to other resource languages that are closer to Indian languages
culturally and linguistically by translating this resource without losing the
rich morphological variations.

##  II Datasets

Most of the summarization datasets of Indian languages are web scraped and
consider the headline or the first lines of the text as summary [27]. Table I
reports the statistics of different summarization datasets available for
Indian languages. Based on the compression ratio, we notice that the summary
of the article is either under-representing or over-representing the document.
Unlike existing datasets, we introduce a dataset TELSUM for Telugu that is
manually annotated (human-written summaries) to create a gold-standard
summarization dataset. Here we provide a brief overview of our TELSUM dataset
and other existing Indic datasets, which we employ in our experiments.

### TELSUM: New Telugu Summarization Dataset

For TELSUM, we scraped a total of 4020 documents (news articles) from a Telugu
news website samyam222https://telugu.samayam.com/. After crawling the
documents, we cleaned and preprocessed 333https://tinyurl.com/2p8pf4kr the
data by removing the unwanted URLs, hash-tags, hyperlinks, English text, and
redundant documents (¡5 sentences). Post-processing, we obtained a total of
3098 documents, of which 2597 documents are used for training the GAE-ISumm in
an unsupervised setting, and the remaining documents (501) form the test
dataset, henceforth referred to as
TELSUM444https://github.com/scsmuhio/Summarization. Our TELSUM dataset
consists of 501 document-summary pairs. These summaries were manually written
by two professional annotators who are native Telugu speakers. Three highly
proficient Telugu native speakers verified the written summaries regarding
readability, relevance, and creativity. Further, we use these annotated
summaries to evaluate our model GAE-ISumm. The privacy details, fair
compensation for annotators, and ethical concerns are discussed in Section
VII. Further details regarding the annotation guidelines are provided in the
appendix.

Table I:  Statistics of TELSUM and different available summarization datasets
in Indian languages. Here, the Compression ratio is the average length of the
summary to the average document length.

Dataset |  Lang |  #Docs |  Avg len of doc |  Compression ratio
---|---|---|---|---
TELSUM |  Telugu (te) |  501 |  17.43 |  0.170
NCTB |  Bengali (bn) |  200 |  08.06 |  0.518
BNLPC |  Bengali (bn) |  139 |  12.78 |  0.129
Marathi |  Marathi (mr) |  100 |  14.80 |  0.513
Hindi Short |  Hindi (hi) |  66,367 |  17.91 |  0.056
|  Telugu (te) |  11,308 |  31.14 |  0.032
|  Bengali (bn) |  8,226 |  41.53 |  0.024
|  Tamil (ta) |  17,846 |  33.59 |  0.030
XL-sum |  Gujarati (gu) |  9,665 |  49.68 |  0.020
|  Punjabi (pa) |  8,678 |  41.02 |  0.020
|  Marathi (mr) |  11,164 |  55.13 |  0.018
|  Hindi (hi) |  51,715 |  29.79 |  0.036

#### Other Datasets

XL-sum: XL-sum is a comprehensive and diverse dataset [27] extracted from the
BBC website. It covers 44 languages, of which we consider seven Indian
languages and English for our experiments.

NCTB and BNLPC: NCTB and BNLPC [28] are Bengali abstractive and extractive
summarization datasets extracted from Bangladesh textbooks and Bengali daily
newspapers, respectively.

Marathi, and Hindi Short Summaries: Marathi dataset is extractive and
collected from a news website 555https://tinyurl.com/mtz473dr. Hindi short
summarization dataset consists of 330k articles scraped from a news Hindi
website 666https://tinyurl.com/4pd86b55 which considers the headline as the
summary of the article.

##  III Methodology

In this section, we explain (i) the overview of GAE, (ii) Graph construction,
and (iii) a detailed description of the proposed method GAE-ISumm

###  III-A Graph Autoencoder (GAE)

Graph Autoencoder (GAE) [18] takes input, an undirected weighted graph
$\mathcal{G}$ := ($\mathcal{V}$, A, X), where $\mathcal{V}$ is a set of N
nodes $(v_{1},v_{2},..v_{N})$, A $\in\mathbb{R}^{NXN}$ is a symmetric
adjacency matrix representing node relationships and X $\in\mathbb{R}^{NXD}$
is the node feature matrix. GAE obtains an encoding Z $\in\mathbb{R}^{NXP}$, a
reduced dimension space of X. The GAE model tries to reconstruct an Adjacency
matrix A${}^{{}^{\prime}}$ close to the empirical graph (A) while Z captures
the essential components of $\mathcal{G}$. We stack an inner product decoder
to reconstruct the graph as follows:

$\displaystyle A^{{}^{\prime}}=g(AH^{(1)}\Theta_{1})$ (1) $\displaystyle
Z=H^{(1)}=f(AX\Theta_{0})$ (2)

where g is an activation function, and $\Theta_{0}$, $\Theta_{1}$ are the
weights learned from the graph reconstruction. Fig. 2 briefly describes the
workflow of the GAE model.

###  III-B Graph Construction

The GAE-ISumm incorporates graph construction at two levels - one at the
sentence level (each node representing a sentence in the sentence graph) and
another at the document level (each node representing a document in the
document graph), as shown in Fig. 1. We use cosine similarity to identify the
relationship between the nodes, which helps us build the graph’s adjacency
matrix. The goal of graph construction is to capture the global context either
within the document (sentence level graph) (or) across all the documents
(document level graph).

###  III-C GAE-ISumm

The overall pipeline of our proposed model GAE-ISumm is described in Fig. 1.
Our proposed model involves training in two phases: i) document encoding and
ii) sentence encoding and summary generation, where each phase is trained
separately. The following subsections explain all the key components of GAE-
ISumm.

####  III-C1 Document Level Graph Construction and Encoding:

To build a document-level graph across all the documents, first, we obtain
sentence representations for each document as further discussed in section 2)
Sentence Encoding In order to get the document representation, we map each
node or document $D_{j}$ to a fixed-length vector ($X_{doc}$) by averaging all
the sentence representations in $D_{j}$. Finally, the document-level graph is
fed into GAEdoc to obtain the graph-based latent document representations
($Z_{doc}$). The GAEdoc model is trained independently by minimizing the
reconstruction loss of document level graph (${\text{Recon\,loss}}_{doc}$).
These obtained latent document representations ($Z_{doc}$) are further used in
the GAE-ISumm model while calculating loss.

####  III-C2 Sentence Encoding and Summary generation

Here, we try to explain the remaining components of our model in detail. They
are organized in the following manner: (i) sentence level graph construction,
(ii) sentence encoding, (iii) clustering and cluster embeddings, (iv) sentence
scoring and selection, and (v) loss calculation. All these components are
processed at a single document level.

#### 1) Sentence Level Graph Construction

We build a sentence-level graph for each document D :=
$(S_{1},S_{2},..,S_{N})$ where each sentence $S_{i}$ is considered as a node.
To obtain a sentence-level graph, we use existing Indic pre-trained language
models [29, 30, 31] and various multilingual pre-trained models [32, 33, 34]
to get a fixed-length vector representation for each sentence. The pre-trained
language models help us detect the high-level contextual features capturing
precise semantic and syntactic relationships. We also investigate monolingual
distributed word embeddings and pre-trained language models available for
Telugu language [31]. After mapping each node to a fixed-length vector, the
sentence-level graph is fed into ${GAE}_{sent}$ to obtain latent graph-based
representations ($Z_{sent}$) of each sentence in the document.

#### 2) Sentence Encoding

We experiment with different pre-trained language models to obtain text
representations. The details of the language models are as follows:

Distributed Word Embeddings: We use Word2vec-Te, Glove-Te, and FastText-Te
word embedding models, trained on a large Telugu dataset of 8 million
sentences [31]. We average the word embeddings in the sentence to obtain the
sentence representation.

Pre-trained Telugu Transformer Language Models: We use the monolingual pre-
trained Transformer models such as BERT-Te, Albert-Te, and Roberta-Te
available for Telugu [31].

Multilingual Embeddings Recently, the NLP community has contributed large-
scale multilingual models for Indian languages performing exceptionally well
compared to other representations. We experiment with few such multilingual
models: IndicBERT [29], mBERT [32], XLM-R [33], mT5 [34] to obtain sentence
representations.

#### 3) Clustering and Cluster Embeddings

A document usually consists of multiple events (or) a series of events; we
believe that clustering on the document helps to segregate better and
understand the document. To accomplish this, we generate cluster
representations to incorporate the cluster information in the final summary.
We applied spectral clustering [35] on the latent sentence representations
$Z_{sent}$ obtained from ${GAE}_{sent}$, where the spectral clustering method
partitions a document of N sentences into M clusters $(c_{1},c_{2},.,c_{M})$.

For each cluster $c_{i}$ with $|c_{i}|$ sentences, the GRU [36] mechanism
outputs a cluster embedding $C_{i}$ on top of sentence embeddings in cluster
$c_{i}$ (please refer Fig. 3). Here, the sentences are passed into GRU
according to their relative position in the document. Finally, we extract the
last hidden state $h_{|c_{i}|}$ of GRU to obtain cluster embedding $C_{i}$, as
shown in Equation (4). This cluster embedding has a semantic overview of the
entire cluster, which helps to capture significant text.

$\displaystyle h_{t}=\text{GRU}(h_{t-1},{S_{t}}^{i})$ (3) $\displaystyle
C_{i}=h_{|c_{i}|}$ (4)

where ${S_{t}}^{i}$ represents sentence at $t^{th}$ time unit in cluster
$c_{i}$.

####  4) Sentence Scoring and Selection

For each sentence $S_{i}$ of cluster $c_{j}$ in the document D, we estimate
the sentence score using two criteria, (i) Sentence relevance score
$({score}_{rel})$ and (ii) Sentence position score $({score}_{pos})$. To
estimate ${score}_{rel}$, we first calculate weighted relevance scores using
the $f(S_{i},D)$ in Equation (5) similar to attention mechanism [37]. Later,
the scores are normalized via Softmax to obtain ${score}_{rel}$ as shown in
Equation (6).

$\displaystyle f(S_{i},D)=\omega^{T}\tanh(W_{1}*Z_{sent}^{S_{i}}+W_{2}*C_{j})$
(5)
$\displaystyle{score}_{rel}(S_{i},D)=\frac{f(S_{i},D)}{{\Sigma}_{{S_{p}}\in{C_{j}}}f(S_{p},D)}$
(6)

where $Z_{sent}^{S_{i}}$ denotes the graph-based latent sentence
representation of $S_{i}$, $C_{j}$ represents its cluster embedding, and
{$\omega$, $W1$, $W2$} are trainable parameters. Inspired from [38],
${score}_{pos}$ (refer Equation (7)) is calculated based on the relative
position $P(S_{i})\in$ [1,2,..,N] of sentence $S_{i}$ in the Document D :=
$(S_{1},S_{2},..,S_{N})$. Sentences at the start of the document are given
high priority than the rest as they provide more relevant information about
the entire document [39]. The final sentence score is calculated as shown in
Equation (8).

${score}_{pos}(S_{i},D)=\max\left(0.5,exp\frac{-P(S_{i})}{\sqrt[3]{N}}\right)$
(7)
$\text{score}(S_{i},D)=\alpha*{\text{score}}_{rel}(S_{i},D)+\beta*{\text{score}}_{pos}(S_{i},D)$
(8)

The variables in Equation (8): $\alpha$, $\beta$ $\in$ [0,1] with
$\alpha$+$\beta$=1, assign relative weights to ${score}_{rel}$ and
${score}_{pos}$, respectively. In every iteration, we sort the sentences in
descending order of their sentence scores, and the top K sentences are
considered our predicted summary ($\hat{S}$) of the document. We obtain our
final candidate summary whenever our model reaches the local minima solution.

#### 5) Loss calculation

We estimate the contrastive loss [40] between the graph-based latent document
representations ($Z_{doc}^{D}$) and the average of all sentence
representations in the candidate summary. The final loss $\mathcal{L}$ is
calculated as follows:

$\displaystyle\mathcal{L}$ $\displaystyle=\text{Reconstruction
loss}({GAE}_{sent})+$ (9) $\displaystyle\text{Contrastive
loss}(\hat{S},Z_{doc}^{D})$

##  IV Experiments and Evaluation

This section describes GAE-ISumm training setup, hyper-parameter tuning, and
evaluation metrics.

###  IV-A Model Training Setup & Hyperparameters

Here, we describe the training details of document encoding (${GAE}_{doc}$),
sentence encoding (${GAE}_{sent}$), and summary generation over all documents.
In our GAE-ISumm, we train each phase (${GAE}_{doc}$, ${GAE}_{sent}$ and
summary generation) separately. The final training was performed by minimizing
the joint loss (Reconstruction loss \+ Contrastive loss) over all the
documents, as mentioned in Equation (9).

The model trainable parameters in ${GAE}_{doc}$, ${GAE}_{sent}$, GRU, and
sentence scoring {$\omega$, $W1$, $W2$} are described below. To perform
document summarization using GAE-ISumm, we set the first convolution layer’s
embedding size as 128 for distributed word embeddings and 256 for remaining
representations. The input feature vectors are extracted from different
language models with an input size of 300 for word embeddings and 768 for
other representations. Since IndicBERT [29] shows superior performance over
other multilingual models, in this paper, we use IndicBERT for input feature
extraction for all other experiments.

We use Adam optimizer with an initial learning rate of 0.001 to train
$GAE_{doc}$. We use the scikit-learn package of Spectral Clustering to perform
sentence graph clusterization. After experimenting with various values, the
number of clusters is considered close to the average number of sentences in
annotated summaries. The joint loss function is optimized with an Adam
optimizer and a learning rate of 0.0005. We experimented with a range of
values to determine the choice of $\alpha$ and $\beta$. The model was
effective when $\alpha$=0.6 and $\beta$=0.4. To extract the summary, we chose
the value of K (number of sentences in the predicted summary) dependent on the
number of clusters. We train each phase of the model to a maximum of 40
epochs. The experiments were performed on a single V100 16GB RAM GPU machine.

###  IV-B Evaluation Metrics

We use ROUGE [41] F1-metric (ROUGE-1 (R-1), ROUGE-2 (R-2), ROUGE-L (R-L) ) to
evaluate our model. ROUGE-N score refers to N-grams overlap between candidate
and gold summary (human reference summary). While ROUGE-L refers to the
longest matching sub-sequence of candidate and gold summary.

Table II: Comparison of ROUGE score results of GAE-ISumm with other methods on
TELSUM dataset.

Model | R-1 | R-2 | R-L
---|---|---|---
TextRank | 36.77 | 22.14 | 34.30
LexRank | 38.89 | 26.65 | 36.45
SumBasic | 34.97 | 20.24 | 33.65
KL Greedy | 33.85 | 18.71 | 31.77
GAE-ISumm | Word2Vec | 42.81 | 30.13 | 41.53
Glove | 42.81 | 30.13 | 41.53
FastText | 42.49 | 33.42 | 41.49
BERT-Te | 45.12 | 34.81 | 43.83
ALBERT-Te | 44.45 | 31.78 | 43.20
Robert-Te | 45.84 | 37.19 | 44.69
XLM-R | 45.24 | 38.49 | 44.14
Indicbert | 46.29 | 36.6 | 44.9
mbert | 44.12 | 34.72 | 41.95
mT5 | 45.47 | 35.08 | 43.83
mbart | 44.86 | 32.9 | 42.27

##  V Results and Analysis

The effectiveness of our proposed model is evaluated by comparing with several
existing baselines such as Textrank [42], LexRank [4], SumBasic [43], and KL
Greedy [44]. In particular, we analyze our GAE-ISumm method on TELSUM as well
as on other existing Indian language datasets. We also describe the
performance of each component of our proposed method in the ablation study.

###  V-A Performance on TELSUM Dataset

Table II reports the ROUGE scores of the baseline methods and our proposed
method GAE-ISumm with various pre-trained language model features as input. We
make the following observations from Table II: (i) All the baseline models
follow a basic heuristic based on similarity, which showcases lower ROGUE
scores than GAE-ISumm. GAE-ISumm demonstrates its efficacy by surpassing all
baselines using any Telugu or multilingual pre-trained model. We contend that
graph-based representations facilitate the capture of both global (document-
level) and local (sentence-level) context information. (ii) The LexRank model
reports a higher ROUGE score among all the baseline models. (iii) All the pre-
trained transformer-based models performed similarly, while IndicBERT and
XLM-R report the highest ROUGE scores for R-1, R-L, and R-2, respectively.
Fig. 4 shows human-annotated summary and the summary predicted by GAE-ISumm
for an example article from TELSUM dataset. From Fig. 4, we observe that GAE-
ISumm extracts all the essential named entities and highly coincide with the
manual summaries in terms of coverage and consistency.

Figure 4: Example of Human annotated summary and GAE-ISumm predicted summary
from TELSUM Figure 5: Analysis of clusters: the clusters formed by GAE-ISumm
for an example article from TELSUM. Table III:  Ablation studies of GAE-ISumm
on TELSUM dataset with Indicbert embeddings.

Ablation studies on GAE-ISumm | R-1 | R-2 | R-L
---|---|---|---
Without ${GAE}_{sent}$ and ${GAE}_{doc}$ | 42.94 | 33.04 | 39.19
With ${GAE}_{sent}$ and without ${GAE}_{doc}$ | 43.36 | 32.45 | 41.58
With ${GAE}_{doc}$ and without ${GAE}_{sent}$ | 43.49 | 33.09 | 40.99
Without clustering | 44.61 | 31.27 | 41.09
Without ${score}_{pos}$ | 43.14 | 32.97 | 39.95

Table IV: GAE-ISumm on XL-sum: ROUGE score results on seven different Indian
languages. These results are compared with mBART (MB) and IndicBART (IB)
results from [45].

Metric | R-1 | R-2 | R-L
---|---|---|---
Language$\downarrow$ / Method$\rightarrow$ | MB | IB | GAE-Summ | MB | IB | GAE-Summ | MB | IB | GAE-Summ
bn | 26.81 | 25.27 | 22.10 | 10.57 | 9.55 | 9.20 | 22.45 | 21.51 | 20.10
gu | 21.49 | 21.66 | 22.20 | 7.48 | 7.43 | 8.40 | 19.08 | 19.39 | 19.43
hi | 39.72 | 38.25 | 30.10 | 17.46 | 16.51 | 13.83 | 32.46 | 31.48 | 26.99
mr | 21.46 | 22.26 | 22.75 | 9.53 | 9.94 | 9.95 | 19.26 | 20.08 | 20.14
pa | 28.15 | 30.28 | 23.50 | 10.30 | 11.88 | 8.7 | 22.75 | 24.38 | 17.98
te | 16.16 | 16.39 | 16.92 | 4.95 | 5.40 | 6.87 | 14.36 | 14.71 | 15.25
ta | 22.47 | 21.79 | 22.60 | 10.22 | 9.75 | 10.1 | 20.33 | 19.67 | 20.90

Table V: ROUGE score results on Other Summarization Datasets using GAE-ISumm.
Here ‘-’ indicates that the dataset has no state-of-art results.

Metric | R-1 | R-2 | R-L | state-of-art method
---|---|---|---|---
Dataset$\downarrow$ / Method$\rightarrow$ | GAE-ISumm | State-of-art | GAE-ISumm | State-of-art | GAE-ISumm | State-of-art |
BNLPC [28] | 71.8 | 61.6 | 66.8 | 56.5 | 71.2 | 61.1 |  BenSumm [28]
NCTB [28] | 12.33 | 12.17 | 3.19 | 1.92 | 10.99 | 11.35 |  BenSumm [28]
Marathi 5 | 80.5 | 64.8 | 75.5 | 59.1 | 80.0 | 66.1 | Similarity based text rank method
Hindi Short 6 | 20.2 | - | 12.0 | - | 16.1 | - | -

Table VI:  Cross-lingual experiments of GAE-ISumm: Bilingual setting of “hi”
and “en” with the other Indian languages from XL-sum. Here ‘-’ represents no
cross-lingual experiment for that permutation of languages.

Metric | R-1 | R-2 | R-L
---|---|---|---
Lang-2$\downarrow$ / Lang-1$\rightarrow$ | hi | en | hi | en | hi | en
bn | 23.26 | 21.49 | 9.6 | 9.21 | 21.45 | 19.5
gu | 22.49 | 19.95 | 7.48 | 7.03 | 19.16 | 16.14
hi | - | 29.56 | - | 12.78 | - | 24.07
mr | 21.89 | 19.13 | 8.78 | 8.26 | 19.72 | 18.93
pa | 23.76 | 24.1 | 8.68 | 8.89 | 18.34 | 17.41
te | 17.81 | 16.55 | 7.89 | 7.83 | 15.43 | 14.85
ta | 22.36 | 21.24 | 9.79 | 9.1 | 20.1 | 19.54

### Analysis of Clusters

Fig. 5 represents the clusters formed for an example article from TELSUM
dataset.(English version of these clusters is present in appendix Fig. 7). The
example article is about a movie that was yet to release. Of the 3 clusters
formed, the first cluster talks about the expected release date of the movie
and the main crew involved. The second cluster talks about the male lead and
the characterization of the male lead. The third cluster talks about the
female-lead cast and her previous career details. We can see that
clusterization helps us in grouping relevant information, getting the full
coverage of the article, and removing redundant information present in the
article.

### Ablation Studies

Here, we analyze the importance of the individual components of GAE-ISumm and
compare them with the benchmark result obtained in Table III. To investigate
the performance of each component, we conducted several ablation experiments:
(i) without graph components, (ii) without clustering and (iii) removal of
sentence position scores. In the second aspect, the model is trained without
including cluster information i.e., we estimate $score_{rel}$ without any
cluster information in Equation (5). The results reflect that, without
clustering, the model fails to capture the complete significant information
from the document. In the third aspect, we remove the sentence position score
from the final sentence score estimation in Equation (8). We observe that
removing positional information yields a relative drop of 6.8% in R-1, proving
the importance of sentence position while generating summaries.

Why Graph-based representations? From Table III, we can observe that either
removal of any of the graph components ($GAE_{sent}$ or $GAE_{doc}$ or both)
resulted in a significant drop in ROUGE scores in all the cases. Graph-based
models tries to capture more relevant information with the help of neighboring
nodes. Also, graph autoencoders further help us to achieve this in an
effective and computation-friendly manner.

Importance of contrastive loss Contrastive loss [40] effectively operates on a
pair of embeddings from similar vector space, specifying whether the two
samples are “similar” or “dissimilar”. Hence, we minimize the contrastive loss
between the graph-based document representation and the average of predicted
top K sentences.

###  V-B Performance on XL-sum (Seven Indian Languages)

We compare the performance of our model GAE-ISumm on the multilingual XL-sum
[27] dataset extracted from BBC news. To evaluate the model performance, we
consider seven Indian language datasets (“te, ta, gu, pa, bn, mr, hi”) from
XL-sum and compare GAE-ISumm performance with the current state-of-art
multilingual approach proposed by [45]. IndicBERT [29] is used as the feature
extraction model to obtain the text representations, and we consider the top
one (or) two scored sentences (according to the dataset average summary
length) predicted by our model as summary. Table IV illustrates the results
obtained on the XL-sum dataset. From Table IV, we observe that GAE-ISumm
outperformed the state-of-the-art models for four Indian languages (“gu, mr,
te, and ta”). In particular, our proposed model performed well on the
Dravidian languages (“te, ta”). Although R-2 scores of GAE-ISumm showcase
close performance to mBART and IndicBART for “bn, hi, and pa”, however, GAE-
ISumm report lower scores in the case of R-1 and R-L. The lower performance
can be mainly because the document size of XL-sum datasets ranges from 6 to
110 sentences. However, their gold summaries extracted are confined to one or
two sentences (under-representation of the document). Overall, we compare our
unsupervised model with their supervised multilingual setting and achieve
competitive or better results.

###  V-C Does Cross-Lingual Models Improve Summarization Performance?

Generally, multilingual pre-trained models are evaluated by their capacity for
knowledge transfer across languages. To investigate the cross-lingual language
transfer, we also experiment with our GAE-ISumm in bilingual settings on the
XL-sum dataset. Bilingual summarization can be done either by training the
model on a single language (high-resource language) alone and testing on the
second language (low-resource language) or by training on both languages and
testing on the second language. This allows models to benefit from the high-
resource languages. We examine our GAE-ISumm in a cross-lingual environment to
further evaluate the use of the proposed dataset TELSUM and the current
multilingual dataset (XL-sum). Here, we performed our cross-lingual
experiments by considering “Hindi (hi)” and “English (en)” from XL-sum dataset
as the high-resource languages and “bn, gu, mr, pa, ta, and te” to be the low-
resource languages. We trained on both the high and low resource languages and
tested on the low-resource languages. We also experimented by considering “hi”
as the low-resource language and “en” as the high-resource language. From
Table VI, we observe that training in a bilingual setting has improved ROUGE
scores. The bilingual setting of “hi” with “te, bn, gu, pa” has improved ROUGE
scores compared to the monolingual setting. Also, the bilingual setting of
“hi” with other languages performed better than “en” due to the morphological
and structural similarity of Indian languages. We report the cross-lingual
transfer results for TELSUM dataset in Appendix (refer Table III).

###  V-D Performance on Other Datasets

We employ our GAE-ISumm model on other existing summarization datasets of
different Indian languages, as reported in Table V. Observations from Table V
that GAE-ISumm yield superior performance on all the datasets with high ROUGE
scores (R-1, R-2, and R-L). Therefore, we argue that our GAE-ISumm model can
be generalizable to any resource-poor language dataset.

##  VI Conclusion and Future Work

This study addresses the robust representational capabilities of graph-based
techniques, RNNs, and neural networks by proposing a unique unsupervised
summarization model. Using cluster representations from GRU and GAE, we create
a model that learns text representations and a document summary. The
positions, relevance, and semantic value of the sentences are also considered
while ranking them in our approach. Our experiments showcase that GAE-ISumm
outperforms all baseline models and provide benchmark results on the TELSUM.
With the help of existing summarization datasets, we examine the effectiveness
of our model for other Indian languages in both monolingual and bilingual
settings.

In the future, we plan to introduce an abstractive model taking summarization
a step forward in low-resource Indian languages. We also aim to expand our
model to other non-Indian language datasets by adapting to diverse languages’
syntactical and morphological features.

##  VII Ethical Statement

We reused the publicly available datasets XL-sum
(https://github.com/csebuetnlp/xl-sum, BNLPC, NCTB [28], Marathi summarization
dataset https://github.com/pratikratadiya/marathi-news-document-dataset,
Hindi-short summarization dataset (
https://www.kaggle.com/datasets/disisbig/hindi-text-short-summarization-
corpus) to compare our state-of-art models. We read the terms and conditions
that these authors/publishers provided for the usage of these datasets and we
see no harm.

Fair Compensation: We provided the data to _Elancer IT Solutions Private
Limited_ 777http://elancerits.com/ company for getting the annotated summary.
In order to perform the annotation process, _Elancer IT Solutions Private
Limited_ chose five native speakers of Telugu with excellent fluency. The
company itself properly remunerates all the annotators.

Privacy Concerns: We have gone through the privacy policy of samyam website
888https://telugu.samayam.com/privacy-policy/privacypolicy/64302688.cms.
However, the copyrights of the articles remain with the publisher/authors
ONLY. We do not foresee any harmful uses of using the data from the website.

## References

  * [1] R. Nallapati, F. Zhai, and B. Zhou, “Summarunner: A recurrent neural network based sequence model for extractive summarization of documents,” in _AAAI_ , 2017.
  * [2] R. Paulus, C. Xiong, and R. Socher, “A deep reinforced model for abstractive summarization,” _arXiv preprint arXiv:1705.04304_ , 2017.
  * [3] H. P. Luhn, “The automatic creation of literature abstracts,” _IBM Journal of research and development_ , vol. 2, no. 2, pp. 159–165, 1958.
  * [4] G. Erkan and D. R. Radev, “Lexrank: Graph-based lexical centrality as salience in text summarization,” _JAIR_ , vol. 22, pp. 457–479, 2004.
  * [5] Y. Zhang, J. E. Meng, and M. Pratama, “Extractive document summarization based on convolutional neural networks,” in _IECON_. IEEE, 2016, pp. 918–922.
  * [6] P. Ren, Z. Chen, Z. Ren, F. Wei, J. Ma, and M. de Rijke, “Leveraging contextual sentence relations for extractive summarization using a neural attention model,” in _SIGIR_ , 2017, pp. 95–104.
  * [7] R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang _et al._ , “Abstractive text summarization using sequence-to-sequence rnns and beyond,” _arXiv preprint arXiv:1602.06023_ , 2016.
  * [8] S. Xu, X. Zhang, Y. Wu, F. Wei, and M. Zhou, “Unsupervised extractive summarization by pre-training hierarchical transformers,” in _EMNLP_ , 2020, pp. 1784–1795.
  * [9] J. Xu, Z. Gan, Y. Cheng, and J. Liu, “Discourse-aware neural extractive text summarization,” _arXiv preprint arXiv:1910.14142_ , 2019.
  * [10] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv preprint arXiv:1609.02907_ , 2016.
  * [11] P. Cui, L. Hu, and Y. Liu, “Enhancing extractive text summarization with topic-aware graph neural networks,” _arXiv preprint arXiv:2010.06253_ , 2020\.
  * [12] J. Xu, Z. Gan, Y. Cheng, and J. Liu, “Discourse-aware neural extractive text summarization,” in _ACL_ , 2020, pp. 5021–5031.
  * [13] L. Yao, C. Mao, and Y. Luo, “Graph convolutional networks for text classification,” in _AAAI_ , vol. 33, no. 01, 2019, pp. 7370–7377.
  * [14] M. Marreddy, S. R. Oota, L. S. Vakada, V. C. Chinni, and R. Mamidi, “Multi-task text classification using graph convolutional networks for large-scale low resource language,” in _IJCNN_. IEEE, 2022.
  * [15] D. Marcheggiani and I. Titov, “Encoding sentences with graph convolutional networks for semantic role labeling,” in _EMNLP_ , 2017, pp. 1506–1515.
  * [16] Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” in _EMNLP-IJCNLP_ , 2019, pp. 3730–3740.
  * [17] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines to read and comprehend,” _NeurIPS_ , vol. 28, pp. 1693–1701, 2015.
  * [18] T. N. Kipf and M. Welling, “Variational graph auto-encoders,” _arXiv preprint arXiv:1611.07308_ , 2016.
  * [19] J. Schmidhuber, “Deep learning in neural networks: An overview,” _Neural networks_ , vol. 61, pp. 85–117, 2015.
  * [20] Q. Xie, J. Huang, P. Du, M. Peng, and J.-Y. Nie, “Inductive topic variational graph auto-encoder for text classification,” in _NAACL: Human Language Technologies_ , 2021, pp. 4218–4227.
  * [21] R. Naidu, S. K. Bharti, K. S. Babu, and R. K. Mohapatra, “Text summarization with automatic keyword extraction in telugu e-newspapers,” in _Smart computing and informatics_. Springer, 2018, pp. 555–564.
  * [22] A. N. Gulati and S. Sawarkar, “A novel technique for multidocument hindi text summarization,” in _2017 International Conference on Nascent Technologies in Engineering (ICNTE)_. IEEE, 2017, pp. 1–6.
  * [23] K. U. Manjari, “Extractive summarization of telugu documents using textrank algorithm,” in _I-SMAC_. IEEE, 2020, pp. 678–683.
  * [24] K. K. Mamidala _et al._ , “A heuristic approach for telugu text summarization with improved sentence ranking,” _TURCOMAT_ , vol. 12, no. 3, pp. 4238–4243, 2021.
  * [25] J. Zhao, M. Liu, L. Gao, Y. Jin, L. Du, H. Zhao, H. Zhang, and G. Haffari, “Summpip: Unsupervised multi-document summarization with sentence graph compression,” in _SIGIR_ , 2020, pp. 1949–1952.
  * [26] M. Yasunaga, R. Zhang, K. Meelu, A. Pareek, K. Srinivasan, and D. Radev, “Graph-based neural multi-document summarization,” _arXiv preprint arXiv:1706.06681_ , 2017.
  * [27] T. Hasan, A. Bhattacharjee, M. S. Islam, K. Mubasshir, Y.-F. Li, Y.-B. Kang, M. S. Rahman, and R. Shahriyar, “Xl-sum: Large-scale multilingual abstractive summarization for 44 languages,” in _ACL-IJCNLP_ , 2021, pp. 4693–4703.
  * [28] R. R. Chowdhury, M. T. Nayeem, T. T. Mim, M. S. R. Chowdhury, and T. Jannat, “Unsupervised abstractive summarization of bengali text documents,” in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 2021, pp. 2612–2619.
  * [29] D. Kakwani, A. Kunchukuttan, S. Golla, N. Gokul, A. Bhattacharyya, M. M. Khapra, and P. Kumar, “Indicnlpsuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for indian languages,” in _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2020, pp. 4948–4961.
  * [30] M. Marreddy, S. R. Oota, L. S. Vakada, V. C. Chinni, and R. Mamidi, “Clickbait detection in telugu: Overcoming nlp challenges in resource-poor languages using benchmarked techniques,” in _IJCNN_. IEEE, 2021, pp. 1–8.
  * [31] ——, “Am i a resource-poor language? data sets, embeddings, models and analysis for four different nlp tasks in telugu language,” _Transactions on Asian and Low-Resource Language Information Processing_ , 2022.
  * [32] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [33] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” _arXiv preprint arXiv:1911.02116_ , 2019.
  * [34] L. Xue, N. Constant, A. Roberts, M. Kale, R. Al-Rfou, A. Siddhant, A. Barua, and C. Raffel, “mt5: A massively multilingual pre-trained text-to-text transformer,” _arXiv preprint arXiv:2010.11934_ , 2020.
  * [35] A. Y. Ng, M. I. Jordan, and Y. Weiss, “On spectral clustering: Analysis and an algorithm,” in _NeurIPS_ , 2002, pp. 849–856.
  * [36] K. Cho, B. van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder–decoder for statistical machine translation,” in _EMNLP_ , 2014, pp. 1724–1734.
  * [37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NeurIPS_ , 2017, pp. 5998–6008.
  * [38] A. Joshi, E. Fidalgo, E. Alegre, and L. Fernández-Robles, “Summcoder: An unsupervised framework for extractive text summarization based on deep auto-encoders,” _Expert Systems with Applications_ , vol. 129, pp. 200–215, 2019.
  * [39] C.-Y. Lin and E. Hovy, “Identifying topics by position,” in _Fifth Conference on Applied Natural Language Processing_ , 1997, pp. 283–290.
  * [40] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _CVPR’06_ , vol. 2. IEEE, 2006, pp. 1735–1742.
  * [41] C.-Y. Lin and E. Hovy, “Manual and automatic evaluation of summaries,” in _ACL-02 Workshop on Automatic Summarization_ , 2002, pp. 45–51.
  * [42] R. Mihalcea and P. Tarau, “Textrank: Bringing order into text,” in _EMNLP_ , 2004, pp. 404–411.
  * [43] A. Nenkova and L. Vanderwende, “The impact of frequency on summarization,” _Microsoft Research, Redmond, Washington, Tech. Rep. MSR-TR-2005_ , vol. 101, 2005.
  * [44] A. Haghighi and L. Vanderwende, “Exploring content models for multi-document summarization,” in _NAACL_ , 2009, pp. 362–370.
  * [45] R. Dabre, H. Shrotriya, A. Kunchukuttan, R. Puduppully, M. M. Khapra, and P. Kumar, “Indicbart: A pre-trained model for natural language generation of indic languages,” _arXiv preprint arXiv:2109.02903_ , 2021.

Table VII:  Crosslingual experiments on TELSUM dataset.

Crosslingual language | R-1 | R-2 | R-L
---|---|---|---
hi | 46.8 | 37.9 | 44.2
en | 47.8 | 37.91 | 44.8

Figure 6: Example of Actual and Predicted summary on XL-sum. Figure 7:
Clusters formed by a sample article from the TELSUM dataset(English version).


* Szegedy et al. (2013) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  _arXiv:1312.6199_ , 2013.
  * Villani (2003) C. Villani.  _Topics in Optimal Transportation Theory_.  American Mathematical Society, 2003.
  * Wiesemann et al. (2014) Wolfram Wiesemann, Daniel Kuhn, and Melvyn Sim.  Distributionally robust convex optimization.  _Operations Research_ , 62(6):1358–1376, 2014\.
  * Yang and Loog (2018) Yazhou Yang and Marco Loog.  A benchmark and comparison of active learning for logistic regression.  _Pattern Recognition_ , 83:401–415, 2018.
  * Yuan et al. (2019) Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li.  Adversarial examples: Attacks and defenses for deep learning.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2019\.