# Label-Free Topic-Focused Summarization Using Query Augmentation

Wenchuan Mu  Information Systems Technology and Design
Singapore University of Technology and Design
Singapore
<EMAIL_ADDRESS>Kwan Hui Lim  Information Systems Technology and
Design
Singapore University of Technology and Design
Singapore
<EMAIL_ADDRESS>
###### Abstract

In today’s data and information-rich world, summarization techniques are
essential in harnessing vast text to extract key information and enhance
decision-making and efficiency. In particular, topic-focused summarization is
important due to its ability to tailor content to specific aspects of an
extended text. However, this usually requires extensive labelled datasets and
considerable computational power. This study introduces a novel method,
Augmented-Query Summarization (AQS), for topic-focused summarization without
the need for extensive labelled datasets, leveraging query augmentation and
hierarchical clustering. This approach facilitates the transferability of
machine learning models to the task of summarization, circumventing the need
for topic-specific training. Through real-world tests, our method demonstrates
the ability to generate relevant and accurate summaries, showing its potential
as a cost-effective solution in data-rich environments. This innovation paves
the way for broader application and accessibility in the field of topic-
focused summarization technology, offering a scalable, efficient method for
personalized content extraction.

###### Index Terms:

Topic-focused summarization, Query augmentation, Transfer learning.

##  I Introduction

In today’s world, where we are constantly bombarded with vast amounts of text,
the ability to efficiently summarize information has become crucial [1].
Summarization, the process of condensing extensive texts into shorter,
digestible formats, is important for both individuals and organizations. It
enables quicker understanding and better management of information. However,
this process is not without its challenges. One major difficulty is creating
summaries that are useful for different purposes and to different people, as
everyone has unique needs and interests [2].

Addressing these issues, topic-focused summarization offers a new approach. It
focuses on generating summaries based on specific topics, making the
information more relevant to the reader’s interests [3]. This task involves
analyzing the content to identify key themes and then highlighting these in
the summary. Yet, implementing topic-focused summarization is challenging. It
typically requires a lot of organized data to start with and demands high
computational power to process this information effectively [4].

###  I-A Recent Developments and Limitations

Recent advances in automated summarization have primarily focused on
abstractive and extractive techniques [5]. Abstractive summarization models
[1], such as those employing neural language models, attempt to generate a
concise and coherent summary by understanding and paraphrasing the original
text. These models, often based on complex architectures like transformers and
attention mechanisms, have shown remarkable capability in mimicking human-like
summary generation [5, 6]. However, they are typically dependent on extensive
supervised training, requiring large labelled datasets that are not always
feasible or available for specific topics or domains [7]. Furthermore, the
unconstrained nature of abstractive summarization often leads to challenges in
controlling the focus of the summaries, particularly in aligning with specific
user queries or topics [2].

Start Raw input / Context Topical query … I have attached images for
reference. Leaking pipes/ceiling in the master bedroom.? need help to resolve
asap; unsatisfactory workmanship - this has gone on for multiple times. I have
invested time and no one is here to ensure quality. I. … What is the problem?
Paraphrase generation Paraphrased topical query \- What is the problem? \-
What is complained? Question answering Answers \- Leaking pipes/ceiling in the
master bedroom. - unsatisfactory workmanship Hierarchical clustering Relevant
answers (only relevant answers) Abstractive summarization Topic focused
summary Leading pipes/ceiling in master bedroom and unsatisfactory workmanship
End

Figure 1: The Augmented-Query Summarization (AQS) pipeline consists of four
pretrained key components: a paraphrasing model, a question-answering model,
hierarchical clustering, and an abstractive summarization model. AQS takes two
text inputs: the query related to the topic and the typically longer context.
It generates a single, topic-focused summary as its output. AQS is an
adaptation approach, as all the key components can be derived from generic
tasks, such as generic abstractive summarization.

Extractive summarization, on the other hand, involves selecting relevant
sentences or segments directly from the source text [8]. This approach, while
simpler and more interpretable, often struggles with generating summaries that
are as coherent and fluent as those produced by abstractive methods [9, 10].
Additionally, both abstractive and extractive methods face challenges in
topic-focused summarization. They either require substantial customization and
retraining for each new topic or fail to adequately prioritize information
relevant to specific user queries [3]. This limitation becomes particularly
evident in dynamic and diverse real-world applications where the ability to
adapt to varying topics and queries rapidly is essential [4]. Therefore, there
is a pressing need for a summarization approach that is both flexible and
efficient in handling diverse topics without extensive retraining or reliance
on large labelled datasets [7].

###  I-B Our Contributions

To address the challenges of topic-focused summarization in the absence of
labelled data, our proposed method, Augmented-Query Summarization (AQS),
employs a novel approach that leverages the strengths of existing natural
language processing techniques [11]. AQS integrates four key components:
paraphrasing for query augmentation, standard question answering, hierarchical
clustering, and generic abstractive summarization. This combination of
techniques allows us to effectively generate topic-focused summaries by
processing and analyzing the context and content of source texts through a
sequence of finely tuned steps. Each component of AQS plays a crucial role in
ensuring that the final summary is not only contextually relevant but also
concise and coherent, addressing the specific topic of interest without
requiring extensive labelled datasets for training.

The practicality and efficiency of AQS are further bolstered by its design,
which accommodates the integration of a range of pre-trained models [11, 12].
This flexibility not only makes our method adaptable to various domains and
topics, but also significantly reduces the time and resources typically
required for model training and fine-tuning. Our approach is especially
advantageous in situations where rapid deployment and adaptability are
essential. Furthermore, by harnessing the capabilities of pre-existing models,
AQS stands at the forefront of summarization technology, offering a robust,
scalable solution that promises significant improvements in personalized
content extraction and information synthesis in data-rich environments. A
summary of our key contributions are:

1) Investigating how query and context variations affect the transferability
of question-answering models, as well as the impact of input changes on
generic abstractive summarization models. 2) Introducing an algorithm that
does not require training on topic-focused summarization or extensive labelled
datasets. 3) Illustrating the effectiveness and efficiency of our method on
real-world data through qualitative and quantitative analysis and experiments.

##  II Related Work

Abstractive summarization models strive to produce brief, precise, and easily
understandable text that captures the most essential information from a
document. In recent years, notable progress has been achieved in the field of
generic abstractive summarization [9, 13, 10]. This success can be credited to
the development of advanced neural architectures and the accessibility of
extensive datasets [1, 14, 15]. The BART architecture [11], for example,
employs transformers and implements sequence-to-sequence functionality,
eliminating the need for a separate encoder-decoder structure.

Topic-focused summarization is a more challenging task that involves
generating a summary specifically tailored to a given query and its relevant
document(s), as illustrated in Example 1. [8] involved treating the extractive
text of an extractive-abstractive model as a latent variable. [16] introduced
an encode-attend-decode system incorporating query and diversity-based
attention mechanisms. This approach aimed to produce a summary that is more
relevant to the given query. [4] trains a relevance prediction model directly
on data using the original, non-masked query. [17] ranks the sentences in
context (specific body of text from which answers to the augmented queries are
to be extracted and summarized) according to embedding similarity to create
weak-supervising labels before end-to-end training. [18] propose a model that
incorporates the explicit answer relevance of the source documents given the
query via a question-answering model, to generate coherently and answer-
related summaries. [19] and [20] also employed QA models to rank answer
evidence at the sentence or paragraph level. However, these methods might not
be able to fully harness the potential of QA models in actively retrieving
highly relevant text segments from the context.

Our work leverages the power of pretrained QA models, which excel at attending
to queries, with the capabilities of pretrained abstractive summarization
models, which generate coherent sentences. In addition, our method is able to
combine with any neuron architecture without the training needed.

######  Example 1.

Context $\rhd$ After 30 years of being ‘stuck’ on the spot, the world’s
biggest iceberg is on the move. Called A23a, the block of ice is around 1,540
sq miles in area – more than twice the size of Greater London …111From
Dailymail article “The world’s biggest iceberg is on the move”
https://www.dailymail.co.uk/sciencetech/article-12786619/.

Generic summary $\rhd$ With an area of 1,540 sq miles, iceberg A23a is the
current world record holder The largest iceberg in the world was A76 before it
fragmented into three pieces

Topical query $\rhd$What did Dr Fleming say?

Topic-focused summary $\rhd$ A structure, likely an ice shelf, stationary
since 1986, began moving due to size reduction over time, with consensus
suggesting natural progression rather than temperature changes as the cause.

##  III Proposed Method: Augmented-Query Summarization (AQS)

We present Augmented-Query Summarization (AQS), which integrates four
elements: paraphrase generation, question answering, hierarchical clustering,
and generic abstractive summarization. The workflow of this pipelined method
can be viewed in Figure 1.

In this section, we introduce how the proposed pipelined method is constructed
and explain how all elements collectively offer a potential solution for
topic-focused summarization without training on dedicated data.

####  III-1 Paraphrase generation

Modifications need to be made to the traditional QA pipeline. In other words,
the answers should encompass more than a single segment, allowing for the
consideration of information from different parts of the context to provide
comprehensive responses. To this end, we resort to tweaking the
question/query. The intuition is that when the query is subtly changed while
retaining the main focus, we may be able to capture varied content segments.
This is because each segment related to the topic can be a valid QA model
output, but typically only one is produced. By introducing a degree of
uncertainty or variability to the input, we anticipate a broader range of
outputs. To this end, our options include varying the query, context, or both.
In this study, we primarily delve into modifying the query.

We propose to augment the query through paraphrase generation. By generating
multiple queries with the same meaning, we naturally generate multiple
potential answers, which may originate from different locations within the
context. Additionally, by perturbing the query with semantic constraints, we
hypothesize that some of the query paraphrases will effectively activate the
transferability of the QA model. The effect of inputting semantically similar
but syntactically different queries into the QA model can be captured in
Example 2 below.

######  Example 2.

Context $\rhd$ The Amazon rainforest includes territory belonging to nine
nations. The majority of the forest is contained within Brazil, with 60% of
the rainforest, followed by Peru with 13%, Colombia with 10%, and minor
amounts in Venezuela, Ecuador, Bolivia, Guyana, …

Query 1 $\rhd$ Which country is the 3rd largest in the forest?

Response $\rhd$ Peru (incorrect)

Query 2 $\rhd$ Which country has the third largest land in the forest?

Response $\rhd$ Colombia (correct)

We note that a slight syntax change in the query leads to a significant
improvement in extracting the correct answer. Although the original and
paraphrased queries convey the same meaning, they differ in their phrasing.
The improved performance can be attributed to the QA model’s sensitivity to
query syntax and structure. This phenomenon has also been discussed in
literature about the robustness of models [21].

The paraphrased query aligns with the patterns and structures learned by the
model during training, enabling more accurate identification of relevant
information in the context and producing a more precise answer. Therefore, by
using paraphrase generation to augment the original query, we can enhance the
chances of extracting a satisfactory answer from the context. This part of the
method is shown in lines 2-12 in Algorithm 1. Now that we have augmented the
topic-related query, the next step is to extract answers using these queries.

####  III-2 Question answering

To address this challenge, we propose to leverage question-answering (QA)
models. The intuition of using QA for topic-focused summarization is that we
can use the topic to form a question and then extract the topic-related
content from the given context. In an adaptation setup, QA is suitable for
extracting topic-focused content from a context for the following reasons.
First, QA and topic-focused summarization share a common input format,
consisting of two texts: one for the query/question and the other for the
context. Secondly, they have a comparable output format, both being text
strings.

However, there are also challenges in using QA for topic-focused
summarization, as there is a key distinction in their output constraints. In
QA, the output is typically a substring extracted from the context, whereas
topic-focused summarization does not impose such limitations. In other words,
the (single-answer) QA task is a task of extracting a single segment from the
context, whereas topic-focused summarization may expect content from many
discrete places in the context. We suggest a parallel question-answering
method that processes augmented queries alongside a singular context. In this
approach, every query is matched with the context to generate a response.
Theoretically, if a standard QA model has a probability $p$ of producing
successful answers, the efficacy of the collective response improves as the
quantity of augmented queries rises, converging towards $p$. Moreover, if $p$
exceeds 1/2, it is highly probable to obtain a successful answer, as depicted
in Figure 2(b).

(a) Examples of paraphrased queries. These queries are paraphrased from an
original query, “What is complained?”

(b) Distribution of the effective queries: Single queries lead to binary
outcomes (‘correct’/‘incorrect’). Multiple queries produce probabilistic
outcomes. Horizontal axis: the proportion of queries with correct answers
(effective queries). Vertical axis: the number of documents input (%). Line
curves: kernel density estimation.

(c) Text redundancy, a measure of irrelevant content in answers or summarised
answers. Without clustering, answer redundancy is 0.78, leading to a summary
redundancy of 0.43. With clustering, answer redundancy decreases to 0.13,
resulting in a summary redundancy of 0.11.

Figure 2: Illustration of the functionality of each part in the proposed
method. Paraphrasing generates multiple queries from a single one. Using
multiple queries may stabilise the QA model performance. On some inputs, a
single query fails to yield a correct answer, while using multiple queries may
yield, say 70%, correct answers. When the majority part of answers are
correct, then the answer can be considered as correct. Note that a correct
answer from QA might also contain irrelevant content, _e.g._ , function words.
Clustering significantly reduces redundancy in QA. This is likely because some
incorrect answers are pretty long, and thus account for a large proportion of
content. Therefore, removing these incorrect answers before summarization is
needed. Besides, summarization helps reform a fluent sentence from the
clustering-selected answers. Intuitively, the text redundancy in the summaries
is even lower.

####  III-3 Hierarchical clustering

The challenge that comes along with using an augmented set of queries is
effectively managing the increased complexity and diversity of the query
space. When we introduce a larger set of augmented queries, it becomes crucial
to ensure that the QA system can efficiently process and evaluate each query
in a timely manner. The system should be able to handle the expanded query
space without sacrificing performance or introducing unnecessary computational
overhead. We need to determine the optimal balance between query diversity and
redundancy. While generating a diverse set of augmented queries can enhance
the chances of capturing different perspectives and nuances, it also increases
the likelihood of introducing redundant or overlapping queries. Managing this
redundancy becomes important to avoid excessive computational costs and
potential confusion in the answer selection process.

Moreover, there is a need to assess the quality and relevance of the augmented
queries. Not all generated paraphrases or variations may be equally useful or
effective in extracting the desired information from the context. Careful
evaluation and filtering techniques are necessary to identify the most
promising queries that are likely to yield satisfactory answers.

Addressing these challenges requires the development of efficient query
management strategies, intelligent query selection techniques, and robust
evaluation mechanisms. By effectively addressing these challenges, we can
harness the benefits of using an augmented set of queries to improve the
overall performance and effectiveness of QA systems. Thus, we propose
clustering answers into relevant and irrelevant ones, eliminating off-topic or
poorly formatted answers, to improve coherence and focus in the summarized
output.

Hierarchical clustering is employed in this approach, where a symmetric
similarity score is calculated between each pair of answers. The two most
similar answers are then combined into a longer answer. This process is
repeated until the longest answer contains at least half of the original
individual answers. Subsequently, the longest answer is used as input for the
abstractive summarization model. Refer to lines 17-20 in Algorithm 1 for a
detailed description.

Algorithm 1 Topic-Focused Summarization
$(\mathbf{x},\mathbf{z},n\in\mathbb{N}_{+}),q\in[0,1)$

1:$\mathbf{x}$: Topic inquiry, $\mathbf{z}$: Input text, $n$: Beam size, $q$:
patience factor

2:$\mathcal{H}_{cur}\leftarrow\\{(\epsilon,0.0)\\}$ $\triangleright$
Initialize with empty generation prefix and zero score

3:repeat

4: $\mathcal{H}_{next}\leftarrow\emptyset$

5: for $(\mathbf{y},p)\in\mathcal{H}_{cur}$ do

6: if $y_{|\mathbf{y}|}=\mathtt{</s>}$ then

7: $\mathcal{H}_{next}\leftarrow\mathcal{H}_{next}\cup\\{(\mathbf{y},p)\\}$
$\triangleright$ Hypotheses ending with $\mathtt{</s>}$ are not expanded

8: else

9:
$\mathcal{H}_{next}\leftarrow\mathcal{H}_{next}\cup\bigcup_{w\in\mathcal{T}}(\mathbf{y}\cdot
w,p+\log P(w|\mathbf{x},\mathbf{y}))$ $\triangleright$ Add all possible
continuations

10: end if

11: end for

12:
$\mathcal{H}_{cur}\leftarrow\\{(\mathbf{y},p)\in\mathcal{H}_{next}:|\\{(\mathbf{y}^{\prime},p^{\prime})\in\mathcal{H}_{next}:p^{\prime}>p\\}|<n\\}$
$\triangleright$ Select $n$-best

13:until $\forall(\mathbf{y},p)\in\mathcal{H}_{cur}.~{}y=\mathtt{</s>}$

14:$\mathcal{H}_{seg}\leftarrow\emptyset$

15:for $(\mathbf{y},p)\in\mathcal{H}_{cur}$ do

16:
$\mathcal{H}_{seg}\leftarrow\mathcal{H}_{seg}\cup\set{Q(\mathbf{y}\cdot\mathbf{z})}$
$\triangleright$ Each answer is a group

17:end for

18:repeat

19:
$(\mathbf{u},\mathbf{v})\leftarrow\operatorname*{arg\,min}_{i,j\in\mathcal{H}_{seg}}d(i,j)$

20:
$\mathcal{H}_{seg}\leftarrow\mathcal{H}_{seg}\cup\set{\mathbf{u}\cdot\mathbf{v}}\setminus\set{\mathbf{u},\mathbf{v}}$
$\triangleright$ Merge the two most similar groups

21:until
$\max_{\mathbf{u}\in\mathcal{H}_{seg}}\left\lvert\mathbf{u}\right\rvert>q\sum_{\mathbf{u}\in\mathcal{H}_{seg}}\left\lvert\mathbf{u}\right\rvert$
return
$S(\operatorname*{arg\,max}_{\mathbf{u}\in\mathcal{H}_{seg}}\left\lvert\mathbf{u}\right\rvert)$
$\triangleright$ Summarize the largest group

####  III-4 Generic abstractive summarization

Now that we have obtained effective answers, the next step is to write a
fluent overall answer. We hypothesize that a generic text-to-text model can
cope with this problem. Generic abstractive summarization condenses text by
generating concise summaries that capture main ideas and context. Moreover,
there are abundant off-the-shelf models. The following example illustrates the
impact of introducing irrelevant information into a text on the performance of
abstractive summarization models. Such perturbations can lead to a decrease in
the quality of the generated summaries.

######  Example 3.

Input $\rhd$ Brief of Incident, Subject Vehicle Illegal passing the gantry
place1Case 01, Gantry at place1Case 01 Vehicle Illegal.

Generated Summary $\rhd$ Case 01, Gantry at place1. Vehicle Illegal passing
the gantry. (improved)

Slightly Changed Input $\rhd$ Thank you subject vehicle ID 4e9aM0$3N5Thank you
Incident, Subject Vehicle Illegal passing the gantry placeCase 01, Gantry at
place1Case 01 Vehicle Illegal, Thank you.

Generated Summary $\rhd$ Vehicle ID 4e9aM0$3N5Thank you Incident, (hardly
readable)

However, current abstractive summarizers, especially neural models, can be
sensitive to input syntax, resulting in potential errors or inconsistencies
(Example 3). It is unwise to directly input answers from QA tasks into
abstractive summarizers for two reasons. Firstly, answers with bad syntax or
characters extracted from the original context can cause out-of-distribution
issues. Secondly, redundancy in answers can propagate to the summaries if
concatenated directly. Therefore, we need to use hierarchical clustering to
select appropriate queries among all we obtain from paraphrase generation,
which leads us to the topic-focused summary.

##  IV Experiment and Result

TABLE I: Estate Complaints/Feedback (ECF) Dataset Demo with five fields: Subject, Description, Category, Sub Category, and Record Key. It includes 13 categories and 64 subcategories. The Subject field often lacks or is poorly written, necessitating topic-focused summarization. Unique Case Record Key |  Subject |  Reporting Category |  Reporting Sub Category |  Description
---|---|---|---|---
1***81 |  FW: Vehicle Illegal Bypassing the Gantry at *place*-Case 01 |  Illegal Parking |  Illegal Parking - Public *org* Car Parks/Service Roads |  *date* *time* PM To … Subject Vehicle Illegal Bypassing the Gantry at *place* 01 á Dear *name* á Brief of Incident á On the … á á Please see the document as attached á á Thank You á Best Regards … *org* Pte Ltd *org*
TABLE II: ROUGE-F1 scores are presented for the QMSum and Debatepedia
datasets. The previous work can be classified into three categories: 1) pre-
trained transformers on a generic summarization dataset, 2) fine-tuning of
these transformers on a query-focused summarization dataset, and 3) models
specifically designed for supervised training in query-focused summarization.

Approach (Train set) | Validate on QMSum | Validate on Debatepedia
---|---|---
|  R-1 |  R-2 |  R-L |  R-1 |  R-2 |  R-L
BART (CNN/Dailymail) |  31.87 |  9.08 |  27.50 |  37.03 |  27.63 |  36.77
T5 (CNN/Dailymail) |  32.45 |  9.80 |  28.48 |  39.72 |  29.33 |  37.37
CAiRE-COVID |  30.98 |  4.70 |  25.80 |  13.32 |  2.47 |  12.18
BART (QMSum) |  32.42 |  9.62 |  28.37 |  42.11 |  38.04 |  41.42
T5 (QMSum) |  34.24 |  9.30 |  28.91 |  40.96 |  37.61 |  38.71
SEGENC (QMSum) |  37.80 |  13.43 |  33.38 |  41.50 |  38.02 |  39.83
QFS-BART (QMSum) |  36.07 |  10.89 |  34.65 |  38.94 |  36.68 |  39.63
BART (Debatepedia) |  31.93 |  8.33 |  27.27 |  57.96 |  44.09 |  57.40
T5 (Debatepedia) |  31.38 |  8.39 |  27.85 |  53.45 |  40.39 |  52.42
SEGENC (Debatepedia) |  33.77 |  9.51 |  31.86 |  54.41 |  38.49 |  51.18
QFS-BART (Debatepedia) |  33.30 |  8.94 |  30.76 |  59.02 |  44.59 |  57.44
AQS (Label-free) |  33.84 |  9.79 |  32.38 |  46.07 |  37.98 |  44.43
The best adaptation method is indicated in bold, second and third italics, and
shaded cells represent different splits of the same dataset, not demonstrating
adaptation behaviour.

TABLE III: ROUGE-F1 scores and Bert-Score are reported for the real-world ECF dataset. Moreover, we present the sentiment consistency (CNS) and human satisfactory score, as described in Section IV-A. All the methods listed are applied in adaptation mode, _i.e._ , with no training or fine-tuning on the tested distribution. Approach |  R-1 |  R-2 |  R-L |  Bert-Score |  Sentiment CNS. |  Human Eval
---|---|---|---|---|---|---
BART (CNN/Dailymail) |  4.23 |  2.02 |  3.84 |  82.17 |  0.211 |  1.95
T5 (CNN/Dailymail) |  3.51 |  1.47 |  2.33 |  84.62 |  0.219 |  0.63
CAiRE-COVID |  11.97 |  2.23 |  11.78 |  86.56 |  0.672 |  4.03
BART (QMSum) |  10.91 |  3.45 |  8.74 |  82.48 |  0.607 |  2.48
T5 (QMSum) |  12.78 |  3.92 |  9.58 |  84.93 |  0.699 |  3.87
SEGENC (QMSum) |  15.87 |  4.25 |  12.64 |  88.56 |  0.493 |  4.39
QFS-BART (QMSum) |  13.03 |  4.29 |  12.61 |  87.88 |  0.515 |  2.01
BART (Debatepedia) |  2.71 |  0.63 |  1.17 |  86.47 |  0.528 |  0.42
T5 (Debatepedia) |  4.35 |  1.04 |  2.98 |  81.22 |  0.716 |  3.57
SEGENC (Debatepedia) |  16.23 |  5.92 |  11.09 |  86.79 |  0.745 |  4.15
QFS-BART (Debatepedia) |  10.56 |  4.78 |  9.26 |  82.97 |  0.453 |  1.33
AQS (Label-free) |  21.94 |  6.37 |  15.68 |  88.75 |  0.807 |  4.82
The best adaptation method is indicated in bold, the runner-up in italics
TABLE IV: Ablation study on the components of the proposed AQS. In rows 2 to 4, we made the following modifications: a) replacing the pretrained transformer from BART to T5, b) removing the clustering step, and c) eliminating the query paraphrasing step. This table shows what are the factors influential/non-influential to our method. Approach |  R-1 |  R-2 |  R-L |  Bert-Score |  Sentiment CNS. |  Human Eval
---|---|---|---|---|---|---
AQS |  21.94 |  6.37 |  15.68 |  88.75 |  0.807 |  4.82
AQS - BART + T5 |  19.40 |  6.38 |  13.29 |  88.60 |  0.833 |  4.40
AQS - Clus. |  6.47 |  1.22 |  2.39 |  79.45 |  0.203 |  1.57
AQS - Para. |  1.51 |  0.17 |  1.23 |  81.43 |  0.208 |  0.47
The best adaptation method is indicated in bold, the runner-up in italics

###  IV-A Setup

##### AQS Implementation

We implement AQS with 4 established models: Pegasus [22] for paraphrase
generation, BART [11] for QA, RoBERTa [12] for text embedding, and BART for
abstractive summarization. These models are respectively pretrained on the QQP
dataset 222https://quoradata.quora.com/First-Quora-Dataset-Release-Question-
Pairs, the SQuAD [23], and the CNN/Dailymail dataset [24]. While specific
models are utilized in our experiments, our framework’s design allows for the
substitution of comparable models, such as T5 [25], to affirm its versatility.
It is critical to underscore that all models are trained on generic datasets
recognized for each task’s benchmarking, such as SQuAD for QA and
CNN/Dailymail for summarization, without fine-tuning on the same or similar
datasets for the task at hand. This ensures the validity of adaptation in the
proposed method. The paragraph used is shown in Figure 2(a) and they are
generated using the off-the-shelf model.

##### Validation Datasets

We assess AQS on two benchmark datasets: Debatepedia [16] and QMSum [26].
Debatepedia comprises 13,719 query-context-summary triplets, specifically
designed for debate summarization tasks. QMSum offers a collection of 1,808
triplets from 232 English-language meetings across varied fields, including
product design, academia, and politics. AQS is further evaluated on the real-
world Estate Complaints/Feedback (ECF) dataset to avoid adaptive overfitting,
which contains feedback records for an estate department from 2016 to 2021.
The structure of the ECF dataset samples is outlined in Table I. The
evaluation utilizes the most recent data split from 2021, with 161,101
samples. For ECF Feedback, we use two fixed queries: “What is complained?” to
identify the main issues, and “What is the emotion?” to determine sentiment
tones.

##### Metrics

ROUGE [27] is the standard for summarization model assessment. This work
utilizes (F1) ROUGE-1, 2, and L for automatic model evaluation on QMSum and
Debatepedia. For the ECF dataset, we measure ROUGE and Bert-Score [28] with
reference to the Reporting Sub Category. A sentiment analysis
model333Sentiment analysis pipeline from Hugging Face
https://huggingface.co/blog/sentiment-analysis-python compares original and
summary sentiment, using Pearson’s correlation. Two postHuman Annotators also
conduct manual evaluations on a random 50-sample subset from ECF’s 2021 data.

##### Baselines

We benchmark our AQS against 1) Pretrained transformers like T5 [25] which are
conventionally adapted for topic-focused summarization at inference; 2)
Transformers fine-tuned on combined queries and contexts, suitable for
supervised or adaptation modes; 3) Other label-free models, such as CAiRE-
COVID, originally design for shortlisting COVID-19 information [20]; 4)
Advanced end-to-end methods such as QFS-BART [29] or SEGENC [4]. This method
can also be employed in either supervised or adaptation mode. For instance,
SEGENC can be trained on QMSum and then validated on Debatepedia or ECF
dataset.

###  IV-B Result

##### Effectiveness

Our primary goal is to assess how well a method performs on a new dataset it
has not seen during training. This ‘different dataset’ is distinct from merely
splitting the training and validation data. As shown in Table II, AQS
outperforms other methods on both the QMSum and Debatepedia datasets, in terms
of summary effectiveness. The summary effectiveness is measured by the
closeness [27] between the generated summary and the reference in the test
dataset. In contrast, when generic summarization models are used for query-
focused summarization, their effectiveness is notably limited. Similarly,
dedicated query-focused summarization methods find it challenging to adapt to
shifts in data distribution between training and targeted test data. In short,
AQS produce well-formed summaries compared to baseline methods on unseen
topic-focused summarization tasks.

##### Efficiency

AQS, featuring four transformer models with approximately $380\pm 100$ M
parameters each, emphasizes efficiency. It can execute sequentially (or
parallel), sidestepping processing bottlenecks, and achieves rapid
performance, processing 183 samples per minute on a single NVIDIA Tesla T4
GPU. In contrast, QFS-BART requires double the GPU memory due to larger model
dimensions, while SEGENC, over ten times larger than AQS, demands a high-end
A100 GPU. AQS represents a significant step forward in optimizing the balance
between model complexity and operational efficiency.

##### Use Case

Since the ECF dataset lacks reference summaries, the Reporting Sub Category
serves as a reference. While exact matches are not always necessary between
sub-category names and predicted summaries, assessing semantic similarity can
gauge model performance. When a summary closely aligns with its sub-category
name, it indicates better performance. Similarly, sentiment preservation
evaluation matches the predicted summary with the original sentiment. As shown
in Table III, AQS generates highly favourable summaries for Descriptions,
benefiting from its use of well-pretrained transformers with strong
transferability from generic task training.

##### Ablation

We explore how AQS effectively combines pre-trained transformers for topic-
focused summarization. The ablation study presented in Table IV examines the
performance when gradually removing specific components of AQS. The results
reveal that a significant drop in performance occurs if either the clustering
or paraphrasing part is excluded. Without clustering, the presence of
redundant text from the answers can dilute the relevant information, leading
to off-topic summaries. Similarly, without paraphrase generation, relying
solely on a QA model results in a single, often short answer, rendering the
presence or absence of downstream models inconsequential. Figure 2 visually
demonstrates the impact of AQS’s paraphrase module and clustering module on
the relevance of information at each stage of text processing.

##  V Conclusion

This study investigates how pretrained neural models can be adapted to create
summaries focused on specific topics, especially when there are no labelled
examples of such summaries available. Our approach involves using paraphase-
based query augmentation, question answering, and hierarchical clustering to
shift from generic summary generation to topic-specific summarization. This
method does not need data specifically labelled for topic-focused
summarization. Instead, it uses four distinct elements that can be developed
separately and thus benefit from a broader range of data, such as that used
for generic abstractive summarization. The effectiveness of this method is
validated on an existing test set for topic-focused summarization. Besides,
this strategy presents a viable option for real-world applications without
demanding substantial computational resources.

Acknowledgment. This research/project is supported by the National Research
Foundation, Singapore, and Ministry of National Development, Singapore under
its Cities of Tomorrow R&D Programme (CoT Award COT-V2-2020-1). Any opinions,
findings and conclusions or recommendations expressed in this material are
those of the author(s) and do not reflect the views of National Research
Foundation, Singapore and Ministry of National Development, Singapore. K. H.
Lim is supported by a MOE AcRF Tier 2 (MOE-T2EP20123-0015).

## References

  * [1] A. M. Rush, S. Chopra, and J. Weston, “A neural attention model for abstractive sentence summarization,” in _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. Lisbon, Portugal: Association for Computational Linguistics, Sep. 2015, pp. 379–389.
  * [2] S. A. Bahrainian, S. Feucht, and C. Eickhoff, “NEWTS: A corpus for news topic-focused summarization,” in _Findings of the Association for Computational Linguistics: ACL 2022_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 493–503.
  * [3] J. M. Conroy, J. D. Schlesinger, and D. P. O’Leary, “Topic-focused multi-document summarization using an approximate oracle score,” in _Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions_. Sydney, Australia: Association for Computational Linguistics, Jul. 2006, pp. 152–159.
  * [4] J. Vig, A. Fabbri, W. Kryscinski, C.-S. Wu, and W. Liu, “Exploring neural models for query-focused summarization,” in _Findings of the Association for Computational Linguistics: NAACL 2022_. Seattle, United States: Association for Computational Linguistics, Jul. 2022, pp. 1455–1468.
  * [5] E. Chu and P. Liu, “MeanSum: A neural model for unsupervised multi-document abstractive summarization,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 1223–1232.
  * [6] P. Li, W. Lam, L. Bing, W. Guo, and H. Li, “Cascaded attention based unsupervised information distillation for compressive summarization,” in _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Copenhagen, Denmark: Association for Computational Linguistics, Sep. 2017, pp. 2081–2090.
  * [7] H. Yu, “Survey of query-based text summarization,” 2022.
  * [8] Z. Mao, C. H. Wu, A. Ni, Y. Zhang, R. Zhang, T. Yu, B. Deb, C. Zhu, A. Awadallah, and D. Radev, “DYLE: Dynamic latent extraction for abstractive long-input summarization,” in _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Dublin, Ireland: Association for Computational Linguistics, May 2022, pp. 1687–1698.
  * [9] A. See, P. J. Liu, and C. D. Manning, “Get to the point: Summarization with pointer-generator networks,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1073–1083.
  * [10] L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, and H. Hon, “Unified language model pre-training for natural language understanding and generation,” in _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada_ , H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. B. Fox, and R. Garnett, Eds., 2019, pp. 13 042–13 054.
  * [11] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, and L. Zettlemoyer, “BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,” in _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_. Online: Association for Computational Linguistics, Jul. 2020, pp. 7871–7880.
  * [12] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019.
  * [13] S. Gehrmann, Y. Deng, and A. Rush, “Bottom-up abstractive summarization,” in _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_. Brussels, Belgium: Association for Computational Linguistics, Oct.-Nov. 2018, pp. 4098–4109.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Advances in Neural Information Processing Systems_ , I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds., vol. 30. Curran Associates, Inc., 2017.
  * [15] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Minneapolis, Minnesota: Association for Computational Linguistics, Jun. 2019, pp. 4171–4186.
  * [16] P. Nema, M. M. Khapra, A. Laha, and B. Ravindran, “Diversity driven attention model for query-based abstractive summarization,” in _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Vancouver, Canada: Association for Computational Linguistics, Jul. 2017, pp. 1063–1072.
  * [17] Y. Xu and M. Lapata, “Generating query focused summaries from query-free resources,” in _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Online: Association for Computational Linguistics, Aug. 2021, pp. 6096–6109.
  * [18] Z. Su, Z. Tang, X. Guan, L. Wu, M. Zhang, and J. Li, “Improving temporal generalization of pre-trained language models with lexical semantic change,” in _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, Dec. 2022, pp. 6380–6393.
  * [19] Y. Xu and M. Lapata, “Coarse-to-fine query focused multi-document summarization,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 3632–3645.
  * [20] D. Su, Y. Xu, T. Yu, F. B. Siddique, E. Barezi, and P. Fung, “CAiRE-COVID: A question answering and query-focused multi-document summarization system for COVID-19 scholarly information management,” in _Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020_. Online: Association for Computational Linguistics, Dec. 2020.
  * [21] S. Garg and G. Ramakrishnan, “BAE: BERT-based adversarial examples for text classification,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Online: Association for Computational Linguistics, Nov. 2020, pp. 6174–6181.
  * [22] J. Zhang, Y. Zhao, M. Saleh, and P. J. Liu, “PEGASUS: pre-training with extracted gap-sentences for abstractive summarization,” in _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , ser. Proceedings of Machine Learning Research, vol. 119. PMLR, 2020, pp. 11 328–11 339.
  * [23] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang, “SQuAD: 100,000+ questions for machine comprehension of text,” in _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Austin, Texas: Association for Computational Linguistics, Nov. 2016, pp. 2383–2392.
  * [24] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, and P. Blunsom, “Teaching machines to read and comprehend,” in _Advances in Neural Information Processing Systems_ , C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, Eds., vol. 28. Curran Associates, Inc., 2015.
  * [25] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” _CoRR_ , vol. abs/1910.10683, 2019.
  * [26] M. Zhong, D. Yin, T. Yu, A. Zaidi, M. Mutuma, R. Jha, A. H. Awadallah, A. Celikyilmaz, Y. Liu, X. Qiu, and D. Radev, “QMSum: A new benchmark for query-based multi-domain meeting summarization,” in _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Online: Association for Computational Linguistics, Jun. 2021, pp. 5905–5921.
  * [27] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in _Text Summarization Branches Out_. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81.
  * [28] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with BERT,” _CoRR_ , vol. abs/1904.09675, 2019.
  * [29] D. Su, T. Yu, and P. Fung, “Improve query focused abstractive summarization by incorporating answer relevance,” in _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_. Online: Association for Computational Linguistics, Aug. 2021, pp. 3124–3131.


# Consolidating Commonsense Knowledge

Filip Ilievski<EMAIL_ADDRESS>
Pedro Szekely<EMAIL_ADDRESS>
Information Sciences Institute, University of Southern California, Marina del
Rey, CA, USA Jingwei Cheng<EMAIL_ADDRESS>
Fu Zhang<EMAIL_ADDRESS>
School of Computer Science and Engineering, Northeastern University, Shenyang,
China Ehsan Qasemi<EMAIL_ADDRESS>
Information Sciences Institute, University of Southern California, Marina del
Rey, CA, USA

###### Abstract

Commonsense reasoning is an important aspect of building robust AI systems and
is receiving significant attention in the natural language understanding,
computer vision, and knowledge graphs communities. At present, a number of
valuable commonsense knowledge sources exist, with different foci, strengths,
and weaknesses. In this paper, we list representative sources and their
properties. Based on this survey, we propose principles and a representation
model in order to consolidate them into a Common Sense Knowledge Graph (CSKG).
We apply this approach to consolidate seven separate sources into a first
integrated CSKG. We present statistics of CSKG, present initial investigations
of its utility on four QA datasets, and list learned lessons.

##  1 Introduction

Capturing, representing, and leveraging commonsense knowledge has been a
paramount for AI since its early days, cf. McCarthy (1960). In the light of
the modern large (commonsense) knowledge graphs and various neural
advancements, the DARPA Machine Common Sense program Gunning (2018) represents
a new effort to understand commonsense knowledge through question-answering
evaluation benchmarks. An example of such question from the SWAG dataset
Zellers et al. (2018) describes a woman that takes a sit at the piano:

                       Q: On stage, a woman takes a seat at the piano. She:
                          1. sits on a bench as her sister plays with the doll.
                          2. smiles with someone as the music plays.
                          3. is in the crowd, watching the dancers.
                       -> 4. nervously sets her fingers on the keys.

Realizing that the logical next step is her “nervously setting her fingers on
the keys” is out of reach for typical information retrieval strategies, as
there is no lexical overlap between the situation and the correct answer.
Although language models Devlin et al. (2018); Liu et al. (2019) capture
linguistic patterns that allow them to perform well on many questions, they
have no mechanism to fill gaps of knowledge in communication.111Recent work
provides evidence that language models, while probably useful and often
impressive, are non-robust when applied to semantic tasks, lacking mechanisms
to understand plausibility or keep track of evolving states of events and
entities Marcus (2020). They struggle with higher number of inference steps
Richardson and Sabharwal (2019), role-based event prediction Ettinger (2020),
as well as numeric, emotional, and spatial inference Bhagavatula et al.
(2019). On the other hand, systems like KagNet Lin et al. (2019) and HyKAS Ma
et al. (2019) have managed to enhance language models by combining them with
background knowledge from ConceptNet Speer et al. (2017). Filling such gaps
requires a more complex, situational reasoning, for which the language models
need to be enriched with suitable background knowledge, as in Lin et al.
(2019).

Intuitively, graphs of (commonsense) knowledge contain such background
knowledge that humans possess and apply, but machines cannot access or distill
directly in communication. A number of such knowledge sources exist today,
which presents a unique opportunity for reasoning in downstream tasks.
Taxonomies, like WordNet Miller (1995), organize conceptual knowledge into a
hierarchy of classes. An independent ontology, coupled with rich instance-
level knowledge, is provided by Wikidata Vrandečić and Krötzsch (2014), a
structured version of Wikipedia. FrameNet Baker et al. (1998), on the other
hand, defines an orthogonal structure of frames and roles; each of which can
be filled with a WordNet/Wikidata class or instance. Sources like ConceptNet
Speer et al. (2017) or WebChild Tandon et al. (2017), provide more ‘episodic’
commonsense knowledge, whereas ATOMIC Sap et al. (2019a) captures pre- and
post-situations for an event. Finally, image description datasets, like Visual
Genome Krishna et al. (2017), have visual commonsense knowledge.

Considering the above example, ConceptNet’s triples state that pianos have
keys and are used to perform music, which supports the correct option and
discourages answer 2. WordNet states specifically, though in natural language,
that pianos are played by pressing keys. According to an image description in
Visual Genome, a person could play piano while sitting and having their hands
on the keyboard. In natural language, ATOMIC indicates that before a person
plays piano, they need to sit at it, be on stage, and reach for the keys.
ATOMIC also lists strong feelings associated with playing piano. FrameNet’s
frame of a performance contains two separate roles for the performer and the
audience, meaning that these two are distinct entities, which can be seen as
evidence against answer 3. While these sources clearly provide complementary
knowledge that can help commonsense reasoning, their representation formats,
principles and foci are different, making integration difficult.

In this paper, we propose an approach for integrating these (and more sources)
into a single Common Sense Knowledge Graph (CSKG). We start by surveying
existing sources of commonsense knowledge to understand their particularities
(section 2). We summarize key challenges and related efforts on consolidating
commonsense knowledge in section 3. Based on the survey and the listed
challenges, we devise five principles and a representation model for a
consolidated CSKG (section 4). In section 5 we apply our approach to build the
first version of CSKG, by combining seven complementary, yet disjoint,
sources. Here, we also compare the evidence provided by CSKG compared to
ConceptNet on four commonsense QA datasets. In section 6 we reflect on CSKG
and discuss its role in future research. We conclude the paper in section 7.

Table 1: Survey of existing sources of commonsense knowledge. |  describes |  creation |  size |  mappings |  examples
---|---|---|---|---|---
Concept Net |  everyday objects, actions, states, relations (multilingual) |  crowd-sourcing |  36 relations, 8M nodes, 21M edges |  WordNet, DBpedia, OpenCyc, Wiktionary |  /c/en/piano /c/en/piano/n /c/en/piano/n/wn /r/relatedTo
Web Child |  everyday objects, actions, states, relations |  curated automatic extraction |  4 relation groups, 2M nodes, 18M edges |  WordNet |  hasTaste fasterThan
ATOMIC |  event pre/post-conditions |  crowd-sourcing |  9 relations, 300k nodes, 877k edges |  ConceptNet, Cyc |  wanted-to impressed
Wikidata |  instances, concepts, relations |  crowd-sourcing |  1.2k relations, 75M objects, 900M edges |  various |  wd:Q1234 wdt:P31
CEO |  event pre/ post-conditions |  manual |  121 properties, 223 events |  FrameNet, SUMO |  ceo:Damaging hasPostSituation
WordNet |  words, concepts, relations |  manual |  10 relations, 155k words, 176k synsets |  |  dog.n.01 hypernymy
Roget |  words, relations |  manual |  2 relations, 72k words, 1.4M edges |  |  truncate antonym
VerbNet |  verbs, relations |  manual |  273 top classes 23 roles, 5.3k senses |  FrameNet, WordNet |  perform-v performance-26.7-1
FrameNet |  frames, roles, relations |  manual |  1.9k edges, 1.2k frames, 12k roles, 13k lexical units |  |  Activity Change_of_leadership New_leader
Visual Genome |  image objects, relations, attributes |  crowd-sourcing |  42k relations, 3.8M nodes, 2.3M edges, 2.8M attributes |  WordNet |  fire hydrant white dog
ImageNet |  image objects |  crowd-sourcing |  14M images, 22k synsets |  WordNet |  dog.n.01
Flickr 30k |  image objects |  crowd-sourcing |  30k images, 750 objects |  |  her backyard red bags

##  2 Sources of Common Sense Knowledge

We survey existing commonsense knowledge sources: ConceptNet Speer et al.
(2017), WebChild Tandon et al. (2017), ATOMIC Sap et al. (2019a), Wikidata
Vrandečić and Krötzsch (2014), CEO Segers et al. (2018), WordNet Miller
(1995), Roget Kipfer (2005), VerbNet Schuler (2005), FrameNet Baker et al.
(1998), Visual Genome Krishna et al. (2017), ImageNet Deng et al. (2009), and
Flickr30k Plummer et al. (2016).222Labels that refer to the same image object
in Flickr30k were clustered by van Miltenburg (2016). Table 1 summarizes their
content, creation method, size, external mappings, and example resources.

Primarily, we observe that the commonsense knowledge is spread over a number
of sources with different focus: commonsense knowledge graphs (e.g.,
ConceptNet), general-domain knowledge graphs (e.g., Wikidata), lexical
resources (e.g., WordNet, FrameNet), taxonomies (e.g., Wikidata, WordNet), and
visual datasets (e.g., Visual Genome). Therefore, these sources together cover
a rich spectrum of knowledge, ranging from everyday knowledge, through event-
centric knowledge and taxonomies, to visual knowledge. While the taxonomies
have been created manually by experts, most of the commonsense and visual
sources have been created by crowdsourcing or curated automatic
extraction.333Commonsense subsets of existing knowledge sources are sometimes
also included, e.g., ConceptNet reuses knowledge from Wiktionary and DBpedia.
Similarly, commonsense and general knowledge graphs tend to be relatively
large, with millions of nodes and edges; whereas the taxonomies and the
lexical sources are notably smaller. Despite the diverse nature of these
sources , we observe that many contain mappings to WordNet, as well as a
number of other sources. These mappings might not be complete, e.g., only a
small portion of ATOMIC can be mapped to ConceptNet and less than 1% to Cyc.
Nevertheless, these high-quality mappings provide an opening for consolidation
of commonsense knowledge, a goal we pursue in this paper.

##  3 Problem Statement

Combining such commonsense knowledge sources in a single graph faces three key
challenges.

Firstly, the sources follow different knowledge modeling approaches. One such
difference concerns the relation set: there are very few relations in
ConceptNet and WordNet, but (tens of) thousands of them in Wikidata and Visual
Genome. Consolidating these sources then inherently requires a decision on how
to model the relations. The granularity of knowledge is another factor of
variance. While regular RDF triples fit some sources (e.g., ConceptNet),
representing entire frames (e.g., in FrameNet), event conditions (e.g., in
ATOMIC), or compositional image data (e.g., Visual Genome) might benefit from
a more open format. An ideal representation would support the entire spectrum
of granularity.

Secondly, as a number of these sources have been created to support natural
language applications, they often contain imprecise descriptions. Natural
language phrases are often the main node types in the provided knowledge
sources, which provides the benefit of easier access for natural language
algorithms, but it introduces ambiguity which might be undesired from a formal
semantics perspective. An ideal representation would consolidate various
phrasings that share a concept or a referent, while still allowing easy and
efficient access to these concepts based on their natural language labels or
aliases.

Thirdly, although these sources contain links to existing ones, we observe
sparse overlap. As these external links are typically to WordNet, and vary in
terms of their version (3.0 or 3.1) or target (lemma or synset), the sources
are still disjoint and establishing (identity) connections is difficult.
Bridging these gaps, through optimally leveraging existing links, or extending
them with additional ones automatically, is a modeling and integration
challenge.

Previous efforts that combine commonsense resources exist. A unidirectional
manual mapping from VerbNet classes to WordNet and FrameNet is provided by the
Unified Verb Index Trumbo (2006). The Predicate Matrix De Lacalle et al.
(2016) has a full automatic mapping between lexical resources, including
FrameNet, WordNet, and VerbNet. PreMOn Corcoglioniti et al. (2016) formalizes
these in RDF. Miller and Gurevych (2014); McCrae (2018) produce partial
mappings between WordNet and Wikipedia/DBpedia. Recent systems integrate parts
of these sources in an ad-hoc manner to reason on a downstream task, e.g.,
Zareian et al. (2020) combine edges from Visual Genome, WordNet, and
ConceptNet in a neural network that produces a scene graph from an image.

##  4 Approach

To address the aforementioned challenges, we devise principles and a
respective representation format that are driven by: simplicity, modularity,
and utility. It should be simple to integrate the graph represented in this
format and its arbitrary subsets in reasoning systems, compute (graph and
word) embeddings, and run off-the-shelf link prediction tools.

###  4.1 Principles

We propose that the construction of a unified CSKG should follow five
principles:

P1. Embrace heterogeneity of nodes While building CSKG, one should preserve
the natural node diversity inherent to the variety of sources considered. This
entails blurring the distinction between objects (such as those in Visual
Genome or Wikidata), classes (such as those in WordNet or ConceptNet), words
(in Roget), actions (in ATOMIC or ConceptNet), frames (in FrameNet), and
states (as in ATOMIC). It also allows formal nodes, describing unique objects,
to co-exist with fuzzy nodes describing ambiguous lexical expressions.

P2. Reuse edge types across resources To support reasoning algorithms, edge
types should be kept to minimum and reused across resources wherever possible.
For instance, the ConceptNet edge type /r/RelatedTo could be reused to relate
a Visual Genome object (e.g., ‘piano’) to its attributes (e.g., ‘black’ or
‘room’). Note that we do not propose to impoverish the semantics of existing
relations.

P3. Leverage external links The separate graphs are mostly disjoint according
to their formal knowledge. However, high-quality links may exist or may be
easily inferred, in order to connect these graphs and enable path finding. For
instance, while ConceptNet and Visual Genome do not have direct connections,
they both make reference to WordNet synsets. Investing an effort in aligning
these WordNet synsets would produce a number of very valuable connections
between the two knowledge sources.

P4. Generate high-quality probabilistic links Experimenting with inclusion of
additional probabilistic links would be beneficial, as it would combat
sparsity and help path finding algorithms reason over CSKG. These could be
inferred with off-the-shelf link prediction algorithms, or with specialized
algorithms (see section 5 for an example).

P5. Enable access to labels The text typically associated with KG entities,
like labels or aliases, provides application-friendly and human-readable
access to the CSKG. It can also help us unify descriptions of the same/similar
concept across sources. We need to ensure that the graph format supports easy
and efficient natural language queries over this text.

###  4.2 Representation

We model CSKG as a property graph, consisting of nodes and edges information
in a tabular format. We opted for this representation rather than the
traditional RDF/OWL2 because it allows us to fulfill our goals (of simplicity
and utility) and follow our principles more directly, without compromising on
the format. For instance, natural language access (principle P5) to RDF/OWL2
nodes requires graph traversal over its rdfs:label relations. Including both
reliable and probabilistic nodes would require a mechanism to easily indicate
edge weights, which in RDF/OWL2 entails inclusion of blank nodes, and a number
of additional edges. Moreover, the simplicity of our tabular format allows us
to use standard off-the-shelf functionalities and mature tooling, like the
pandas444https://pandas.pydata.org/ and graph-tool555https://graph-
tool.skewed.de/ libraries in Python. We can also compute embeddings with Lerer
et al. (2019) or plug CSKG in a reasoner, e.g., Wang et al. (2018), with
minimal adaptation, as these expect tabular inputs.

Node representation Each node is described by six columns: id (reused from the
source data when possible), label (its primary label), aliases (additional
labels), pos (part-of-speech tag, if applicable), datasource (original source,
one of: ‘cn’, ‘vg’, ‘wn‘, ‘rg’, ‘wd‘, ‘fn’, ‘at’, or ‘mowgli’ for custom
nodes), and other (a dictionary with provenance information, e.g.,
‘image_id’). Due to its generality, this node representation is suitable for
any kind of node: object, state, class, or action, thus satisfying our
principle P1. To accommodate P5, we ensure easy access to node labels and
aliases by assigning them dedicated columns.

Edge representation Edges are described by six columns: subject (the subject
ID of the edge), predicate (edge label), object (its object ID), datasource
(its original source: ‘cn’, ‘vg’, ‘wn’, ‘rg’, ‘wd’, ‘fn’, ‘at’, or ‘mowgli’
for custom relations), weight (as given by the source), and other (provenance
information, e.g., source sentence). Following the approach of ConceptNet and
our principle P2, we minimize the set of edges. We introduce a dedicated
relation, mw:SameAs, to indicate identity. When identity between two nodes is
readily available in an external source, e.g., as a mapping, we create a link
with a weight of 1.0 (principle P3). Per P4, we also predict implicit edges
with weights between 0 and 1.

Figure 1: Snippet from the Common Sense Knowledge Graph (CSKG). The yellow
nodes are created by merging nodes from individual resources, e.g.,
vg:key+/c/en/key combines one lexical expression from Visual Genome and one
from ConceptNet.

##  5 The Common Sense Knowledge Graph

In its current form (Figure 1), CSKG integrates seven sources: a commonsense
knowledge graph ConceptNet, a visual commonsense source Visual Genome, a
procedural source ATOMIC, a general-domain source Wikidata, and three lexical
sources, WordNet, Roget, and FrameNet. Here, we present our design decisions
per source, their integration, and statistics on the resulting CSKG graph. We
also present initial investigations on utilizing CSKG for commonsense question
answering.

###  5.1 Consolidation

####  5.1.1 Individual sources

ConceptNet We keep its original data and representation, and we increase its
consistency and connectivity in two ways. Firstly, we compute closure over the
seven symmetric relations defined in ConceptNet as the symmetry was not
consistently reflected in the data. Secondly, we observe that different forms
of a concept are not formally linked. We introduce two mutually inverse
relations, mw:POSForm and mw:IsPOSFormOf, to link a lemma to its part-of-
speech version. Also, we define a mw:PartOfSpeech class, and we add edges of
type mw:OMWordnetOffset between a lemma node and its WordNet v3.1 offset form.

Visual Genome does not have its data formatted in a Semantic Web compliant
format. We follow the ConceptNet approach and represent the label of each
object, relation, and attribute as a node in the graph (e.g., vg:dog). We
introduce two relations vg:Subject and vg:Object to indicate the subject and
the object of a relation node. We also establish direct, symmetric links
between a subject and an object node by reusing the relation /r/RelatedTo from
ConceptNet. This relation is also leveraged to represent relations between
nodes and their properties. To contextualize an object/relation in terms of
its image, we create nodes for image objects (prefix vg:I*) and utilize the
relation vg:InImage. For each object, relation, or attribute, we use the
relation mw:PWordnetSynset to connect it to its WordNet v3.0 synset provided
in the data dump.

WordNet We include hypernymy from WordNet v3.0, via the rdfs:subClassOf
relation.

Roget We include all synonyms and antonyms between words in Roget, by reusing
the Concept relations /r/Synonym and /r/Antonym.

ATOMIC We include the entire knowledge graph, preserving the original nodes
and relations. To enhance lexical matching over the labels in CSKG, we
normalize the labels of the events and their attributes: converting them to
lowercase, removing references to ‘Person*’, and excluding ‘none’ values.

Wikidata We include the Wikidata taxonomy through the rdfs:subClassOf
relation.

FrameNet Four node types from the FrameNet ontology are imported into CSKG :
Frames, Frame Elements (FEs), Lexical units (LUs), and Semantic Types (STs).
We reuse 5 categories of FrameNet edges: Frame-Frame (13 edge types), Frame-FE
(1 edge type), Frame-LU (1 edge type), FE-ST (1 edge type), and ST-ST (3 edge
types).

####  5.1.2 Mappings

WordNet-WordNet The WordNet v3.1 offsets in ConceptNet and the WordNet v3.0
synsets from Visual Genome are aligned by leveraging ILI: the WordNet
InterLingual Index.666https://github.com/globalwordnet/ili The generated
117,097 mappings are expressed through our identity relation, mw:SameAs.

WordNet-Wikidata We compute probabilistic links between WordNet synsets and
Wikidata taxonomy nodes. Our approach consists of three components: a
Candidate Retrieving Module (CRM), a Similarity Calculating Module (SCM), and
a Mapping Module (MM). CRM retrieves candidate nodes from a customized
ElasticSearch index of Wikidata. Concretely, it matches a synset word in any
text field (including labels, aliases, and descriptions), and it ranks the
candidates with a version of the default TF-IDF-based algorithm which adapts
the score proportionally to the number of incoming links.777Documentation:
shorturl.at/jpBY3 The top-ranked $n=50$ candidates are retained. Then, SCM
computes sentence embeddings of the descriptions of the WordNet synset and
each of the Wikidata candidates by using a pre-trained XLNet model Yang et al.
(2019). The similarity between a synset and a Wikidata node is computed as a
cosine similarity between their corresponding embeddings. MM creates a
mw:SameAs edge between a WordNet synset and the Wikidata candidate with
highest similarity. This similarity is represented as weight of the mapping
edge. The accuracy of each mapping has been validated by one student. In
total, 17 students took part in this validation. Out of all edges produced by
the algorithm (112,012), the manual validation marked 57,145 as correct. We
keep these in CSKG and discard the rest.

FrameNet-ConceptNet We connect the FrameNet nodes to ConceptNet in two ways.
Its lexical units are mapped to corresponding ConceptNet nodes through the
Predicate Matrix (cf. section 3), producing $3,016$ mw:SameAs edges.888During
this step, we manually fixed approximately $250$ nodes which contained
spelling and tokenization errors in the Predicate Matrix. Then, we use
$\approx 200k$ hand-annotated sentences from the FrameNet corpus, each
annotated with its target frame, a set of FEs, and the words associated with
each FE. We consider the set of words to be an instance of that specific FE.
We ground these sets of words to ConceptNet with the rule-based method of Lin
et al. (2019), thus adding 45,659 mw:HasInstance edges.

Roget-ConceptNet We establish 60,307 mw:SameAs relations between word-
representing nodes in Roget and in ConceptNet by a simple lexical match of
their labels.

Visual Genome-ConceptNet We establish 32,283 mw:SameAs relations between
lexical nodes in Visual Genome and in ConceptNet by exact matching over their
labels.

ATOMIC-ConceptNet We establish 14,272 mw:SameAs relations between lexical
nodes in ATOMIC and in ConceptNet by an exact match of their labels.

####  5.1.3 Refinement

We consolidate the seven sources and six mappings as follows. Firstly, we
deduplicate each edge table by aggregating over its first three columns
(subject, predicate, object) and combining the values for the remaining
columns. Similarly, we deduplicate each node table by aggregating over its id
column and combining the values for the other columns. Secondly, we
concatenate all 13 edge tables into one, same for the 6 node tables, to
produce a raw version of CSKG. Thirdly, we merge identical nodes, by combining
nodes that are connected with a mw:SameAs link. This operation is reflected in
both the nodes and the edges table. The deduplicated version of the result is
our consolidated CSKG.

Table 2: CSKG statistics. Abbreviations: CN=ConceptNet, VG=Visual Genome, WN=WordNet, RG=Roget, WD=Wikidata, FN=FrameNet, AT=ATOMIC. | CN | VG | WN | RG | WD | FN | AT | CSKG
---|---|---|---|---|---|---|---|---
# nodes | 1,787,276 | 316,660 | 87,942 | 71,804 | 2,388,479 | 36,582 | 288,943 | 4,738,502
# edges | 7,211,322 | 4,833,879 | 89,089 | 1,403,461 | 2,926,639 | 79,060 | 704,315 | 17,210,065
max degree | 587,358 | 127,580 | 404 | 1,549 | 964,400 | 2,438 | 9,856 | 964,400
mean degree | 8.07 | 30.53 | 2.03 | 39.09 | 2.45 | 4.32 | 4.87 | 7.26
std degree | 0.34 | 0.83 | 0.02 | 0.34 | 0.58 | 0.16 | 0.06 | 0.32
Figure 2: Degree distribution (log-log plots).

###  5.2 Statistics

Basic statistics are shown in Table 2. In total, our mappings produce 284,121
mw:SameAs links. A small portion (less than $1\%$) of the nodes and edges were
duplicates. After refinement, i.e., removal of the duplicates and merging of
the identical nodes, CSKG consists of 4.7 million nodes and 17.2 million
edges. In terms of edges, its largest subgraph is ConceptNet (7.2 million),
whereas Visual Genome comes second with 4.8 million edges. Wikidata
contributes with the largest number of nodes, closely followed by ConceptNet.
The three most common relations in CSKG are: /r/RelatedTo (4.6 million),
vg:InImage (3 million), and rdfs:subClassOf (3 million).

Degree distribution The mean degree of CSKG, after merging identical nodes,
grows from 7.00 to 7.26. Its standard deviation is 0.32, similar to
ConceptNet. The best connected subgraphs are Roget and Visual Genome, with
mean degrees of 39.09 and 30.53, respectively. The least connected graphs are
the hierarchies of WordNet and Wikidata. The degrees of WordNet and ATOMIC are
fairly uniform across their nodes, whereas Visual Genome and Wikidata have
higher variation. The large difference between the variations of WordNet and
Wikidata can be explained by the fact that WordNet nodes typically have a
single parent, whereas the Wikidata ontology is more flat and has many arcs
from its leaves to high-level nodes. The maximum degree in CSKG is nearly a
million, which is due to Wikidata.

Figure 3: PageRank distribution.

Figure 2 presents the input, output, and total degree distributions. The
highest out-degree is 10 times lower than the highest in-degree. Although most
nodes have a low degree and the frequency decreases for the higher degrees,
nearly $10^{5}$ nodes still have a total degree of 10, and many others have a
much higher one. This indicates that CSKG is a well-connected graph.

Centrality We compute PageRank and HITS metrics of CSKG. Its PageRank
distribution (Figure 3) indicates that while most nodes have a low PageRank
value, $\approx 100$ have PageRank of over 0.001, The top PageRank nodes are:
wn:polypeptide.n.01+/c/en/polypeptide/n/wn/substance+wd:Q8054 (polypeptide as
a noun, merged from three sources), wd:Q7187 (gene), and wd:Q20747295
(protein-coding gene). According to HITS, the hubs in CSKG are: wd:Q20747295
(protein-coding gene), wd:Q7187 (gene), and wd:Q427087 (non-coding RNA). The
top authorities are also Wikidata nodes. The dominance of Wikidata in the
centrality metrics is due to its taxonomy, where subclass relations are often
directed at high-level nodes. It is unclear, however, whether such bio-
informatics nodes hold value for commonsense reasoning. Future work should
investigate which subsets of Wikidata contain commonsense knowledge.

###  5.3 CSKG on Downstream Tasks

Table 3: Number of triples retrieved with ConceptNet (CN), CSKG’s ConceptNet subset (CSKG-CN), and CSKG on different datasets. #Q=number of questions. | _train_ | _dev_
---|---|---
| #Q | CN | CSKG-CN | CSKG | #Q | CN | CSKG-CN | CSKG
CSQA | 9,741 | 78,729 | 106,619 | 153,442 | 1,221 | 9,758 | 13,132 | 19,036
SIQA | 33,410 | 126,596 | 189,859 | 330,200 | 1,954 | 7,850 | 11,654 | 19,953
PIQA | 16,113 | 18,549 | 28,996 | 70,131 | 1,838 | 2,170 | 3,401 | 8,071
aNLI | 169,654 | 257,163 | 389,640 | 771,318 | 1,532 | 5,603 | 8,477 | 16,456

In a preliminary investigation, we measure the relevance of CSKG for
commonsense question answering tasks, by comparing the number of retrieved
triples that connect keywords in the question and in the answers. For this
purpose, we adapt the lexical grounding in HyKAS Ma et al. (2019) to retrieve
triples from CSKG instead of its default knowledge source, ConceptNet. We
expect that CSKG can provide much more evidence than ConceptNet, both in terms
of number of triples and their diversity. We experiment with four commonsense
datasets: CommonSense QA (CSQA) Talmor et al. (2018), Social IQA (SIQA) Sap et
al. (2019b), Physical IQA (PIQA) Bisk et al. (2019), and abductive NLI (aNLI)
Bhagavatula et al. (2019). As shown in Table 3, CSKG significantly increases
the number of evidence triples that connect terms in questions with terms in
answers, in comparison to ConceptNet. We note that the increase on all
datasets is roughly three-fold, the expected exception being CSQA, which was
inferred from ConceptNet.

We inspect a sample of questions to gain insight into whether the additional
triples are relevant and could benefit reasoning. For instance, let us
consider the CSQA question “Bob the lizard lives in a warm place with lots of
water. Where does he probably live?”, whose correct answer is “tropical
rainforest”. In addition to the ConceptNet triple /c/en/lizard
/c/en/AtLocation /c/en/tropical_rainforest, CSKG provides two additional
triples, stating that tropical is an instance of place and that water is
related to tropical.999fn:fe:place mw:HasInstance
rg:tropical+/c/en/tropical+vg:tropical
101010rg:water+/c/en/water+at:water+vg:water /r/RelatedTo
rg:tropical+/c/en/tropical+vg:tropical The first additional edge stems from
our mappings from FrameNet to ConceptNet, whereas the second comes from Visual
Genome. Interestingly, the above example comes from CSQA, which has been
inferred from ConceptNet, showing that most commonsense knowledge is still
largely missing in existing resources. We note that, while CSKG increases the
coverage with respect to available commonsense knowledge, it is also
incomplete: in the above example, useful information such as warm temperatures
being typical for tropical rainforests is still missing.

##  6 Discussion

The graph metrics in the previous section indicate that CSKG is well-
connected, thus showing the impact of our mappings across sources and the
merge of identical nodes. Furthermore, the novel evidence brought by CSKG on
downstream QA tasks (section 5.3) is a signal that can be exploited by
reasoning systems to enhance their performance and robustness. What are the
next steps for CSKG? We discuss three ongoing pursuits.

Downstream tasks Injecting knowledge from ConceptNet has improved the
performance of existing question-answering systems like KagNet Lin et al.
(2019), GapQA Khot et al. (2019), and HyKAS Ma et al. (2019).111111For an
overview of commonsense knowledge sources, reasoners, and benchmarks, see
Storks et al. (2019). At the same time, the error analyses and discussions of
these systems reveals that missing knowledge is directly responsible for a
portion of their errors. We expect that the richer and more diverse knowledge
captured by CSKG, as quantified in section 5, would benefit commonsense QA
systems. To validate this expectation, we developed a modular evaluation
framework, and we computed various graph and word embeddings of CSKG and it
subsets. At present, we are running experiments with adaptations of HyKAS and
KagNet on six such datasets.

New resources and mappings We intend to continue integrating the resources
listed in section 2: WebChild, VerbNet, and CEO. We envision their integration
following our approach to be fairly straightforward. Moreover, as machine
common sense knowledge is an active research area (section 1), we expect
additional knowledge sources to be released and integrated in CSKG in the near
future. Similarly, we intend to create further links between the resources to
maximize the connectivity of the ingredients within CSKG.

Semantic enrichment The semantics of CSKG could be improved by refining its
relations. For instance, its most common ConceptNet relation, /r/RelatedTo
abstracts over various specific predicates, and its full set of relations is
unknown and potentially large. In Figure 1, it expresses containment (piano
related to piano key), and inheritance (piano key related to key), while
elsewhere (farmer related to man), it obfuscates occupation. Clustering its
knowledge into more specific predicates would improve the semantics of CSKG.

##  7 Conclusions and Future Work

The traditional goal of capturing, representing, and leveraging commonsense
knowledge has recently gained traction, thanks to initiatives like DARPA’s
Machine Common Sense Gunning (2018). In this paper, we reviewed representative
commonsense knowledge sources. While they contain complementary knowledge that
would be beneficial as a whole for downstream tasks, such usage is prevented
by their different approaches, foci, strenghts, and weaknesses. Optimizing for
simplicity, modularity, and utility, we proposed a property graph that
describes many nodes with a few edge types, maximizes the high-quality links
across subgraphs, and enables natural language access. We applied this
approach to consolidate a commonsense knowledge graph (CSKG) from seven very
diverse sources: a text-based commonsense knowledge graph ConceptNet, a
general-purpose taxonomy Wikidata, an image description dataset Visual Genome,
a procedural knowledge source ATOMIC, and three lexical sources: WordNet,
Roget, and FrameNet. It describes 4.7 million nodes with 17.2 million
statements. Our analysis showed that CSKG is a well-connected graph and more
than ‘a simple sum of its parts’. On four commonsense QA datasets, it
consistently increased the recall of relevant triples by 2-4 times compared to
ConceptNet. At present, we are investigating whether this additional signal
brought by CSKG helps reasoning on these tasks, and we are integrating further
resources reviewed in this paper. CSKG will be released under CC BY-SA 4.0,
the most permissive license allowed by its components.

## References

  * Baker et al. (1998) Collin F Baker, Charles J Fillmore, and John B Lowe.  The berkeley framenet project.  In _Proceedings of the 17th international conference on Computational linguistics-Volume 1_ , pages 86–90. Association for Computational Linguistics, 1998.
  * Bhagavatula et al. (2019) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi.  Abductive commonsense reasoning.  _arXiv preprint arXiv:1908.05739_ , 2019.
  * Bisk et al. (2019) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  _arXiv preprint arXiv:1911.11641_ , 2019.
  * Corcoglioniti et al. (2016) Francesco Corcoglioniti, Marco Rospocher, Alessio Palmero Aprosio, and Sara Tonelli.  Premon: a lemon extension for exposing predicate models as linked data.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 877–884, 2016.
  * De Lacalle et al. (2016) Maddalen Lopez De Lacalle, Egoitz Laparra, Itziar Aldabe, and German Rigau.  Predicate matrix: automatically extending the semantic interoperability between predicate resources.  _Language Resources and Evaluation_ , 50(2):263–289, 2016.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Ettinger (2020) Allyson Ettinger.  What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48, 2020.
  * Gunning (2018) David Gunning.  Machine common sense concept paper.  _arXiv preprint arXiv:1810.07528_ , 2018.
  * Khot et al. (2019) Tushar Khot, Ashish Sabharwal, and Peter Clark.  What’s missing: A knowledge gap guided approach for multi-hop question answering.  _arXiv preprint arXiv:1909.09253_ , 2019.
  * Kipfer (2005) BA Kipfer.  Roget’s 21st century thesaurus in dictionary form (éd. 3). new york: The philip lief group, 2005.
  * Krishna et al. (2017) Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  _International Journal of Computer Vision_ , 123(1):32–73, 2017.
  * Lerer et al. (2019) Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich.  Pytorch-biggraph: A large-scale graph embedding system.  _arXiv preprint arXiv:1903.12287_ , 2019.
  * Lin et al. (2019) Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren.  Kagnet: Knowledge-aware graph networks for commonsense reasoning.  _arXiv preprint arXiv:1909.02151_ , 2019.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ , 2019.
  * Ma et al. (2019) Kaixin Ma, Jonathan Francis, Quanyang Lu, Eric Nyberg, and Alessandro Oltramari.  Towards generalizable neuro-symbolic systems for commonsense question answering.  _arXiv preprint arXiv:1910.14087_ , 2019.
  * Marcus (2020) Gary Marcus.  The next decade in ai: four steps towards robust artificial intelligence.  _arXiv preprint arXiv:2002.06177_ , 2020.
  * McCarthy (1960) John McCarthy.  _Programs with common sense_.  RLE and MIT computation center, 1960.
  * McCrae (2018) John P McCrae.  Mapping wordnet instances to wikipedia.  In _Proceedings of the 9th Global WordNet Conference (GWC 2018)_ , pages 62–69, 2018.
  * Miller (1995) George A Miller.  Wordnet: a lexical database for english.  _Communications of the ACM_ , 38(11):39–41, 1995\.
  * Miller and Gurevych (2014) Tristan Miller and Iryna Gurevych.  Wordnet―wikipedia―wiktionary: Construction of a three-way alignment.  In _LREC_ , pages 2094–2100, 2014.
  * Plummer et al. (2016) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  _International Journal of Computer Vision_ , pages 1–20, 2016.
  * Richardson and Sabharwal (2019) Kyle Richardson and Ashish Sabharwal.  What does my qa model know? devising controlled probes using expert knowledge.  _arXiv preprint arXiv:1912.13337_ , 2019.
  * Sap et al. (2019a) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 3027–3035, 2019a.
  * Sap et al. (2019b) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi.  Socialiqa: Commonsense reasoning about social interactions.  _arXiv preprint arXiv:1904.09728_ , 2019b.
  * Schuler (2005) Karin Kipper Schuler.  Verbnet: A broad-coverage, comprehensive verb lexicon.  2005\.
  * Segers et al. (2018) Roxane Segers, Tommaso Caselli, and Piek Vossen.  The circumstantial event ontology (ceo) and ecb+/ceo: An ontology and corpus for implicit causal relations between events.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , 2018.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-First AAAI Conference on Artificial Intelligence_ , 2017\.
  * Storks et al. (2019) Shane Storks, Qiaozi Gao, and Joyce Y Chai.  Commonsense reasoning for natural language understanding: A survey of benchmarks, resources, and approaches.  _arXiv preprint arXiv:1904.01172_ , 2019.
  * Talmor et al. (2018) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  _arXiv preprint arXiv:1811.00937_ , 2018.
  * Tandon et al. (2017) Niket Tandon, Gerard De Melo, and Gerhard Weikum.  Webchild 2.0: Fine-grained commonsense knowledge distillation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 115–120, 2017.
  * Trumbo (2006) Derek Trumbo.  _Increasing the usability of research lexica_.  PhD thesis, University of Colorado at Boulder, 2006.
  * van Miltenburg (2016) Emiel van Miltenburg.  Stereotyping and bias in the flickr30k dataset.  In Jens Edlund, Dirk Heylen, and Patrizia Paggio, editors, _Proceedings of Multimodal Corpora: Computer vision and language processing (MMC 2016)_ , pages 1–4, 2016.  URL http://www.lrec-conf.org/proceedings/lrec2016/workshops/LREC2016Workshop-MCC-2016-proceedings.pdf.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85, 2014\.
  * Wang et al. (2018) Liang Wang, Meng Sun, Wei Zhao, Kewei Shen, and Jingming Liu.  Yuanfudao at semeval-2018 task 11: Three-way attention and relational knowledge for commonsense machine comprehension.  _arXiv preprint arXiv:1803.00191_ , 2018.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in neural information processing systems_ , pages 5754–5764, 2019.
  * Zareian et al. (2020) Alireza Zareian, Svebor Karaman, and Shih-Fu Chang.  Bridging knowledge graphs to generate scene graphs.  _arXiv preprint arXiv:2001.02314_ , 2020.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  _arXiv preprint arXiv:1808.05326_ , 2018.