<EMAIL_ADDRESS>
OGCOSMO: An auxiliary tool for the study of the Universe within hierarchical
scenario of structure formation

Eduardo S<EMAIL_ADDRESS>Oswaldo D. Miranda
<EMAIL_ADDRESS>
National Institute for Space Research, São José dos Campos, SP, Brazil

Received on September , 2010 / accepted on *****, 2010

###### Abstract

In this work is presented the software OGCOSMO. This program was written using
high level design methodology (HLDM), that is based on the use of very high
level (VHL) programing language as main, and the use of the intermediate level
(IL) language only for the critical processing time. The languages used are
PYTHON (VHL) and FORTRAN (IL). The core of OGCOSMO is a package called
OGC_lib. This package contains a group of modules for the study of
cosmological and astrophysical processes, such as: comoving distance, relation
between redshift and time, cosmic star formation rate, number density of dark
matter haloes and mass function of supermassive black holes (SMBHs). The
software is under development and some new features will be implemented for
the research of stochastic background of gravitational waves (GWs) generated
by: stellar collapse to form black holes, binary systems of SMBHs. Even more,
we show that the use of HLDM with PYTHON and FORTRAN is a powerful tool for
producing astrophysical softwares.

Keywords:Computational Physics, Cosmology, Gravitational Waves, Black Hole,
Structure Formation.

1\. INTRODUCTION

Around 300 thousand years after the Big Bang, matter and radiation decoupled
and the photons could freely travel through the space. Henceforth, today we
can obtain information about this period from the Cosmic Microwave Background
Radiation (CMBR). Moreover, about 840 million years after the Big Bang, the
Universe was reionized by the first stars. However, we have no direct
information from the period between 300 thousand years and 840 million years.
This period is so-called dark age (DA). On the other hand, the theory of
general relativity predicts the existence of gravitational waves (GWs) as
perturbations of space-time which propagate at the speed of light. The
detection of GWs will open a new astronomical window for observing the
Universe. In particular, they will allow us a deeper understand about the DA.

In this context, we are interested in to describe, by analytical and semi-
analytical models, some astrophysical and cosmological processes as, for
example, the cosmic star formation rate, and its connection with the
stochastic background of GWs [4], this kind of study will shed light on the
knowledge of the process that took place at the end of the DA.

As such, this modeling framework will provide us with an important auxiliary
tool for the reconstruction of the cosmic history. Thus, we decided to
organize the computational development as a software, it called OGCOSMO, whose
main characteristics are discussed in this work.

In section 2 it will be showed a short description of the astrophysical and
cosmological model adopted. In section 3 it will be presented the main
characteristics of the software and the type of programing used. In section 4
we present some results. Finally, in section 5 are the final considerations.

2\. The Model

In this work we are considering the general theory of relativity with cold
dark matter and cosmological constant ($\Lambda$CDM model) for background
cosmology.

Here is assumed the hierarchical scenario of structure formation, having as
base the Press-Schechter (PS) like formalism [6]. The basic idea behind this
scenario is that the formation of objects like galaxies and galaxy clusters
occur in the following way: First, when the mean density of dark matter
perturbation, within a given volume, is larger than a threshold level,
$\delta_{c}$, the matter leaves the linear regime and collapses to form small-
halo objects. These halos become gravitational wells that attract the baryonic
matter, that is the ordinary matter that form stars and planets. Durant all
this process, greater halos are formed by fusion of the minors and more
baryons fall into structures. The star formation starts and black holes grow
up by accretion of matter. The complete details, about the all considerations
and the main results obtained with this model, can be seen in [4, 5].

3\. The Software

The OGCOSMO software was written using object-oriented programing paradigm,
that permits to construct codes that are really reusable and clean. The main
programming language used was PYTHON [7], that is a very-high level language
programing, and FORTRAN only for the critical time parts of the code. This
form of writing codes is called of high level design [3, 2]. The external
modules used were basically: Tkinter [8] for construction of the Graphical
Using Interface (GUI), scipy and numpy for numerical methods [9].

The core of the OGCOSMO is the OGC_lib, that is a package containing the
following principal modules, up to now:

  * •

OGC_cosmo: This module contains the class “Cosmo()” that is composed by the
cosmological background methods, and callbacks, that are, comoving distance
and comoving volume, relation between time and redshift ($z$), matter density
evolution (dark and baryonic matters), grow function of matter perturbation,
variance of matter linear density field, linear extrapolation of the critical
density for collapse of structures in a given $z$ , comoving volume of dark
matter halo.

  * •

OGC_PS: This module contains the class “PressSchechter(Cosmo)”, that inherit
the methods of Cosmo class. This class contains the base of Press-Schechter
like formalism, that is, the functions used for study of structure formation
as, mass function and numerical density of dark haloes, fraction and accretion
rate of baryonic matter within structures and the cosmic star formation rate -
CSFR.

  * •

OGC_SMBH: In this module is the “SuperBH(PressSchechter)” class, that is a
class under development and it will be used for the study of the evolution of
supermassive black holes through of its mass function and by the coalescence
of these objects [5].

4\. Results

4.1 The OGCOSMO

In the figure 1 we show the GUI of OGCOSMO. In A are the spaces for entry
parameters such as the cosmological and those associated with the star
formation rate. In B are the buttons that start the calculus of the CSFR,
supermassive black holes mass function and some buttons for creation of
specific plots as the CSFR (see figure 2).

In the figure 2 is represented the CSFR behavior, whose result can be seen by
pressing the button “Grafico CSFR”. The Salpeter exponent of initial mass
function is $x=1.35$, the characteristic scale for star formation is
$\tau=2.5\times 10^{9}$ years and the exponent of the star formation rate
$n=1$. For more details see [4].

Figure 1: GUI of OGCOSMO. In A are showed the entry parameters. In B are
showed the buttons for running the models and generate some graphics.

Figure 2: Plot of the cosmic star formation rate behavior. In this case was
considered $\Omega_{m}=0.24$, $\Omega_{b}=0.04$, $\Omega_{\Lambda}=0.76$ and
$h=0.73$, for the cosmological parameters, $x=1.35$ and $\tau=2.5\times
10^{9}$ years.

4.2 OGC_lib Usage

In Python a package is basically a hierarchical file directory structure which
defines a environment that consists of modules. This directory contains a file
named __ini__.py that identifies the directory as a package [1]. In the
context of this work, this means that if a user does not want to use the GUI
of OGCOSMO, he (or she) can use only determined methods by importing its
specific class from a module within OGC_lib. For example, an user want to call
the method age from the class Cosmo of the module OGC_cosmo:

$[1]>>>$ from OGC_lib.OGC_cosmo import Cosmo

$[2]>>>$ MyUniverse = Cosmo(0.04,0.24,0.73,0.76,6.0,20.0,’./trabalho’)

$[3]>>>$ Age = MyUniverse.age(5)

$[4]>>>$ print‘ Age = %3.9e’ %Age

$[5]$ Age = 1.189273236e+09

In line $[1]$ was imported, from OGC_cosmo module within OGC_lib package, the
class Cosmo. In line $[2]$ was created a new object (MyUniverse) and in line
$[3]$ was used the method age. The example above show that for
$\Omega_{m}=0.24$, $\Omega_{b}=0.04$, $\Omega_{\Lambda}=0.76$ and $h=0.73$,
the age of the Universe at $z=5$ is $1.2\times 10^{9}$ years. The last
argument of the class Cosmo (’./trabalho’) is the directory where the data
will be saved.

This result shows that OGC_lib can be used as a framework for fast an easy
development of astrophysical and cosmological applications for the study of
the cosmic history, within hierarchical scenario for structure formation.

5\. Final Considerations

In this work are presented both the initial stage of the OGCOSMO software and
the OGC_lib package, that have been developed to be an auxiliary tool for
reconstruction of the cosmic history. Moreover, it is possible for an user to
have access for a specific method of the OGC_lib. This is an important feature
to show that this package can be used as a framework for construction of
others softwares. An another aspect is that the program can be used as
didactic tools in lectures of theory of general relativity and cosmology.

The next steps will be to write classes to obtain the mass function of SMBHs;
research of stochastic background of gravitational waves (GW) generated by:
collapse of stars to black holes, binary systems of SMBH; calculate the
signal/noise rate for GW detectors (LISA, LIGO, Decigo, BBO).

Even more, it was showed that the use of PYTHON with FORTRAN for high-level
design program is a powerful tool for development of astrophysical softwares.

ACKNOWLEDGMENTS: E. S. Pereira would like to thank the Brazilian Agency CAPES
for support. O. D. Miranda would like to thank the Brazilian Agency CNPq for
partial support (grant 300713/2009-6)

## References

  * [1] GUPTA, R, 2002. Making Use of Python. Wiley Publishing.
  * [2] HINSEN, K, 2007. Parallel scripting wiht Python. Computing in science & engineering. Nov./Dec. 82-89.
  * [3] HINSEN, K, LANGTANGEN, HP, SKAVHAUG, O, ØDEGÅRD, Å, 2006. Using BSP and Python to simplify parallel programming. Future Generation Computer Systems. 22: 123-157.
  * [4] PEREIRA, ES, MIRANDA, OD, 2010. Stochastic background of gravitational waves generated by pre-galactic black holes. Mon. Not. R. Astron. Soc. 401: 1924-1932.
  * [5] PEREIRA, ES, MIRANDA, OD, 2010. Massive Black Hole Binary Systems in Hierarchical Scenario of Structure Formation. International Journal of Modern Physics D. 19: 1271-1274
  * [6] PRESS, WH, SCHECHTER, P, 1974. Formation of galaxies and clusters of galaxies by self-similar gravitational condesation. Apj, 193, 425-438.
  * [7] http://www.python.org/ $<$accessed in 13 august 2010$>$
  * [8] http://docs.python.org/library/tkinter.html $<$accessed in 13 august 2010$>$
  * [9] http://www.scipy.org/ $<$accessed in 13 august 2010$>$
  * [10] http://dirac.cnrs-orleans.fr/plone/software/scientificpython/ $<$accessed in 13 august 2010$>$


# Implicit Premise Generation with Discourse-aware Commonsense Knowledge
Models

Tuhin Chakrabarty 1, Aadit Trivedi2, and Smaranda Muresan1,3
1Department of Computer Science, Columbia University
2College of Computing, Georgia Tech
3Data Science Institute, Columbia University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Enthymemes are defined as arguments where a premise or conclusion is left
implicit. We tackle the task of _generating the implicit premise in an
enthymeme_ , which requires not only an understanding of the stated conclusion
and premise, but also additional inferences that could depend on commonsense
knowledge. The largest available dataset for enthymemes Habernal et al. (2018)
consists of 1.7k samples, which is not large enough to train a neural text
generation model. To address this issue, we take advantage of a similar task
and dataset: Abductive reasoning in narrative text Bhagavatula et al. (2020).
However, we show that simply using a state-of-the-art seq2seq model fine-tuned
on this data might not generate meaningful implicit premises associated with
the given enthymemes. We demonstrate that encoding discourse-aware commonsense
during fine-tuning improves the quality of the generated implicit premises and
outperforms all other baselines both in automatic and human evaluations on
three different datasets.

##  1 Introduction

In argumentation theory, an enthymeme is defined as an incomplete argument
found in discourse, where some components are explicit, but other propositions
are left implicit and need to be _filled in_ as premises or conclusions to
fully understand what the argument is Walton and Reed (2005). In many
instances the missing proposition is a premise. The well-cited example of the
Silver Blade case from one of Sherlock Holmes’ stories Walton and Reed (2005)
presents such as an incomplete argument

> A dog was kept in the stable, and yet, though someone had been in and
> fetched out a horse, he had not barked enough to rouse the two lads in the
> loft. Obviously, the midnight visitor was someone whom the dog knew well.

The missing premise in this case is the generalization “Dogs generally bark
when a person enters an area unless the dog knows the person well."

While there has been work on identification (i.e., classification) and
reconstruction of implicit premises in enthymemes Rajendran et al. (2016);
Habernal et al. (2018); Reisert et al. (2015); Boltužić and Šnajder (2016);
Razuvayevskaya and Teufel (2017), to our knowledge, _automatically generating
an implicit premise from a given enthymeme_ is a new task. There are two main
challenges that need to be addressed: 1) lack of large scale data of
incomplete arguments together with annotated missing premises needed to train
a sequence-to-sequence model (the largest such set contains 1.7K instances
Habernal et al. (2018)); and 2) the inherent need to model commonsense or word
knowledge.

We propose an approach for generating an implicit premise given a incomplete
argument that aims to address these two challenges. Our contributions are
three fold.

Reason | Vaccinations save lives
---|---
Claim |  | Vaccination should be mandatory
---
for all children
ZeroShot | Vaccines save lives, they save money
| Fine-tuned on
---
ART
| Vaccinations are the best way to
---
protect children.
| Fine-tuned on
---
ART +PARA-C
| Vaccinations are the best way to
---
prevent childhood diseases.
Table 1: Implicit Premise Generation by BART Lewis et al. (2020) in three
different setting for an input enthymeme from dataset by Habernal et al.
(2018)

_A new task of generating an implicit premise given an incomplete argument
(enthymeme)._ Given an enthymeme consisting of a stated conclusion and a
stated premise, generate the implicit/missing premise. As the backbone
sequence-to-sequence architecture we use BART Lewis et al. (2020).

_Leverage abductive reasoning as an auxiliary task_. To address the first
challenge, we rely on an observation from argumentation theory that incomplete
arguments in naturally occurring discourse, more often than not, require
abductive reasoning (plausible explanations) rather than the more strict form
of reasoning based on deductive logic Walton and Reed (2005); Sabre (1990).
The Silver Blaze case is such an example. We leverage the Abductive Reasoning
in Narrative Text (ART) dataset introduced by Bhagavatula et al. (2020) to
fine-tune a BART model. ART consists of pairs of observations together with
the plausible explanation to be generated (Section 3).

_Encoding discourse-aware common sense knowledge._ To address the second
challenge, we rely on PARA-COMET Gabriel et al. (2021), a discourse-aware
knowledge model that incorporates paragraph-level information to generate
coherent commonsense inferences from narratives. We encode the outputs of
PARA-COMET during fine-tuning BART on our auxillary dataset (ART) (Section 4).
We show on three different datasets (Section 3) that this knowledge-enhanced
model performs best both in automatic and human-based evaluations (Section 5).

Table 1 shows an example of an enthymeme consisting of a stated premise and
conclusion and the generated implicit premise by a BART model (zero-shot), by
a BART model fine-tuned on ART dataset, and a BART model fine-tuned on ART
augmented with discourse-aware commonsense knowledge derived from PARA-COMET.
We make the code available at
https://github.com/tuhinjubcse/EnthymemesEMNLP2021.

##  2 Related Work

Prior work on enthymeme reconstruction has focused primarily on the
identification (i.e., classification) of implicit premises in enthymemes
Rajendran et al. (2016); Habernal et al. (2018); Reisert et al. (2015);
Boltužić and Šnajder (2016); Razuvayevskaya and Teufel (2017). Boltužić and
Šnajder (2016) study how to identify enthymemes in online discussions, while
Habernal et al. (2018) present the task of identifying the correct warrant
given two candidates warrants in order to reconstruct an enthymeme. Rajendran
et al. (2016) introduce an approach to classify the stance of a statement as
implicit or explicit, as a first step towards the long term goal of enthymeme
reconstruction. Unlike these works which propose discriminative approaches to
identify an enthymeme or the (correct) implicit premises, we focus on
generative models that aim to _generate an implicit premise given an
enthymeme_ , using abductive reasoning and discourse-aware commonsense
knowledge.

Alshomary et al. (2020) introduce a closely related task of generating an
argument’s conclusion from its premises. Specifically, they focus on the
subtask of inferring the conclusion’s target from the premises. They develop
two complementary target inference approaches: one ranks premise targets and
selects the top-ranked target as the conclusion target, the other finds a new
conclusion target in a learned embedding space using a triplet neural network.
Unlike this paper, our work focuses on a new task of generating an implicit
premise given an enthymeme that consists of a stated conclusion and a stated
premise.

##  3 Datasets

### Training dataset.

Based on the theoretical connection between enthymemes and abductive
reasoning, we use the Abductive Reasoning in narrative Text (ART) data
developed for the abductive NLG task Bhagavatula et al. (2020) to train our
models. The task is framed as: given two observations (O1 and O2) from a
narrative, generate the most plausible explanation (hypothesis) (Table 2). The
observations O1, O2 in ART are drawn from the ROCStories Mostafazadeh et al.
(2016) dataset, a large collection of short, manually curated five sentence
stories. The beginning and ending of each story maps to the first (O1) and
second (O2) observations in ART, respectively. Bhagavatula et al. (2020)
presented O1 and O2 as narrative context to crowdworkers and prompted them to
generate plausible and implausible Hypotheses (H) to explain the observations.
To avoid annotation artifacts, Bhagavatula et al. (2020) applied an
adversarial filtering step to retain one challenging pair of plausible and
implausible hypotheses that are hard to distinguish between. The ART training
set consists of 50481 instances, while the validation and test set consist of
7252 and 14313 instances, respectively. As can be seen in Table 2 the
observations O1 and O2 could be "mapped" to the stated Premise and the stated
Claim in an enthymeme, while the hypothesis H is mapped to the _implicit
premise_ we try to generate.

O1 | Alex had his heart set on an ivy league college
---|---
O2 |  | Alex ended up achieving his dream
---
of getting into the school.
H | Alex applied to Harvard
Table 2: Instances from the ART dataset.

### Test datasets.

We test our models on three different datasets of incomplete arguments
(enthymeme) annotated with human-generated implicit/missing premises. First,
we use the Argument Reasoning Comprehension Task dataset released by Habernal
et al. (2018) (D1), which contains 1654 {claim, premise, warrant(implicit
premise)} triples. Second, we used the dataset introduced by Boltužić and
Šnajder (2016), which contains 494 enthymemes from an online debate forum with
human annotated implicit premises (D2). Third, we use the dataset introduced
by Becker et al. (2020) (D3), which contains implicit premises annotated for
each arguments from the MicroText Corpus Peldszus and Stede . For D3, we focus
only arguments that are in a _support_ relation since this corresponds to our
task. Moreover, we choose the cases where there is only one implicit premise,
rather than a chain of linked premises. This results in a total of 112
enthymemes for D3. For all datasets, we apply automatic filtering to keep only
full-formed sentences as claim and premises (e.g., remove cases where the
stated premise/claim consists of a noun-phrase, a partial clauses, or many
sentences).

| Encoder
---
Input
| Amy was looking through her
---
mother’s old scrapbooks. [SEP] Amy
realized her mother had dated her
history professor.
| Encoder
---
Input +
PARA-COMET
| Amy was looking through her
---
mother’s old scrapbooks. [SEP] to
find something [SEP] Amy realized
her mother had dated her history
professor.
| Decoder
---
Ouput
| Amy was looking through her
---
mother’s old scrapbooks. And since
Amy found pictures of her history
professor and mother together. Amy
realized her mother had dated her
history professor.
Table 3:  Encoder input in two settings: fine-tuning on ART and fine-tuning on
ART \+ PARA-COMET (the green text between [SEP]). For decoder’s output every
hypothesis is prepended by And since in bolded blue.

##  4 Method

For our generation model, we use BART Lewis et al. (2020), a pre-trained
conditional language model that combines bidirectional and auto-regressive
transformers. It is implemented as a sequence-to-sequence model with a
bidirectional encoder over corrupted text and a left-to-right auto-regressive
decoder.

### Fine-tuning BART on ART.

To fine-tune BART on the ART dataset (Section 3), we concatenate O1 and O2
with a special delimiter [SEP] as input to BART encoder as shown in Table 3
Row 1. For decoding, we focus on reconstructing the entire argument given an
enthymeme. To encourage fluency and coherence in our generated argument, we
prepend the plausible hypothesis (implicit premise) with a discourse marker
And since (Table 3 Row 3) during fine-tuning.

### Fine-tuning BART on PARA-COMET enhanced ART.

Adapted knowledge models such as COMET Bosselut et al. (2019) have been shown
to generate implicit commonsense inferences along several dimensions
(depending on what knowledge graphs they were pre-trained on). PARA-COMET
Gabriel et al. (2021), is an extension of COMET pre-trained on ATOMIC Sap et
al. (2019) that is able to generate discourse-aware common sense knowledge.
ATOMIC is a knowledge graph that contains 9 relations related to social
commonsense knowledge, including dynamic aspects of events such as causes and
effects, if-then conditional statements, and mental states. Given a text with
T sentences ${S_{1},S_{2}...S_{T}}$, PARA-COMET generates a set of commonsense
inferences for the 9 inferential relations from ATOMIC for each sentence
$S_{i}$, which are consistent with the entire narrative. Following PARA-
COMET’s input format, we create a discourse of two sentences containing
[O1,O2] from ART. We then feed this as an input to the trained PARA-COMET
model and obtain 9 commonsense relations for both O1 and O2. Given the causal
nature of the implicit premises for this work we use only the relation
xIntent. Given an event (e.g., “X compliments Y"), xIntent states the likely
intents of person X (e.g., “X wants to be nice"). We only consider xIntent
returned for O1 (Premise on our task). We experimented with other relations as
well as xIntent for both O1 and O2 but the results were not better. After
obtaining discourse-aware commonsense, we concatenate {O1, commonsense, O2} in
a sequential order as shown in Table 3 Row 2 and pass it to BART’s encoder for
fine-tuning. For decoding, we use the same process as before (Table 3 Row 3).

### Inference-time decoding.

For generation on our task and test sets, we concatenate the {Premise, Claim}
or {Premise, commonsense, Claim} in a given enthymeme in the same way as shown
in Table 3 and pass as an input to the encoder of fine-tuned BART. The fine-
tuned BART model then generates the entire argument along with the implicit
premise auto-regressively. We use beam search with a beam width of 5 for
generation. Post decoding, we split the argument into 3 individual sentences
and treat the middle sentence starting with And since as the implicit premise
after removing the artificially added discourse marker.

For zero-shot setting, we use the pre-trained BART (bart-large) model. We use
the format {_Premise. And since [MASK]. Claim_} and let the language model
generate an implicit premise.

##  5 Evaluation and Results

We evaluate three setups: 1) directly use pre-trained BART (Zero-shot); 2)
fine-tune BART on ART; 3) fine-tune BART on ART+PARA-COMET.

### Automatic Evaluation Setup.

We use BLEU Papineni et al. (2002), one of the most widely used automatic
metrics for generation tasks to compute BLEU-1 and BLEU-2 scores between the
system output and the human written gold implicit premise. We also report
F1-Score of BERTScore, a metric for evaluating text generation using
contextualized embeddings.

### Human evaluation setup.

We select 50 enthymemes from each test set (total of 150 enthymemes) and the
output of our fine-tune BART models (with or without PARA-COMET). We hired
crowdworkers on the Amazon Mechanical Turk platform. Given an enthymemes they
were asked if the generated implicit premises were plausible or not
(agreement: 0.56 based on Krippendorff’s $\alpha$). Each enthymeme was judged
for plausibility by 3 distinct Turkers (50 crowdworkers overall). As it was a
binary judgement, we took majority voting which means if 2/3 of the annotators
thought it was plausible we marked it as plausible. Plausibility judgement
considers whether the generated premise was grammatical, relevant to the
argument, coherent with our commonsense and completes the argument.

### Results.

While pre-trained language models often contain structured commonsense Davison
et al. (2019); Zhou et al. (2020) Table 4 shows that pre-trained BART cannot
generate plausible implicit premises. Fine-tuning on the ART dataset improves
the results significantly. Finally, the model that encodes discourse-aware
commonsense outperform all baselines on all test datasets (D1, D2 and D3).
Human evaluation further demonstrates that encoding commonsense knowledge
leads to better implicit premise generation (Table 5).

### Analysis.

We notice that adding commonsense beams from PARA-COMET makes the generated
implicit premise more plausible. For instance, for the stated claim and
premise from D3 in Table 6, we see that PARA-COMET adds a beam to feel better.
Similarly it adds a beam to learn more for the stated claim and premise from
D1 for both examples shown in Table 6. We posit that adding these in
combination with the stated claim and premise, leads our model to infer more
plausible implicit premises compared to the ones generated by BART fine-tuned
on ART. Finally, given that D3 has been annotated with argument schemes Musi
et al. (2018), we can explore their role in enthymeme reconstruction. We
notice that most of the generated plausible implicit premises belong to
enthymemes annotated with Practical Evaluation argument scheme, where “the
premise is an evaluation about something being ‘good’ or ‘bad’, while the
claim expresses a recommendation/advice about stopping/continuing an action"
(Table 6 ).

Data | System | BLEU1 | BLEU2 | BS
---|---|---|---|---
D1 | ZeroShot | 6.02 | 2.17 | 42.88
ART | 9.16 | 3.11 | 48.35
| +PARA-COMET
---
10.56 | 3.90 | 50.22
D2 | ZeroShot | 28.24 | 15.13 | 46.96
ART | 37.77 | 18.76 | 60.63
| +PARA-COMET
---
44.12 | 24.14 | 67.75
D3 | ZeroShot | 12.58 | 6.25 | 44.64
ART | 14.89 | 6.34 | 51.78
| +PARA-COMET
---
15.56 | 7.50 | 53.38
Table 4:  Automatic evaluation of implicit premise generation by BART in 3 settings based on BLEU1, BLEU2 and BertScore(BS). Difference is significant, $(\alpha<0.005)$ via Wilcoxon signed-rank test. Data | System | Plausibility
---|---|---
D1 | ART | 50%
| +PARA-COMET
---
58%
D2 | ART | 48%
| +PARA-COMET
---
66%
D3 | ART | 38%
| +PARA-COMET
---
56%
Table 5: Human evaluation results our finetuned BART models in two settings.  |  |
---|---|---
D1 |  St Premise |  Deaf students need more specialized education
St Claim |  States need special schools for the deaf
Gold |  Their parents can’t always enroll them in a deaf private school
Zero-shot |  We can’t afford it, we shouldn’t
ART |  The deaf students are not getting enough education.
+PARA-COMET |  Deaf students are not being served well in the schools
D1 |  St Premise |  Understanding other culture is more important now than ever before.
St Claim |  Colleges need humanities programs
Gold |  More people now fail to understand other cultures
Zero-shot |  It’s the humanities, we need them
ART |  The humanities are the most important subjects in college.
+PARA-COMET |  There is a lot of misinformation out there about other cultures
D2 |  St Premise |  Bush new spending in 8 years? $5.07 TRILLION Obama total New Spending (projected out for the next 8 years)? $1.44 TRILLION. And of that total, only $430 billion is non-recession related.
St Claim |  Fixed the economy
Gold |  Obama spends less money than Bush.
Zero-shot |  We are talking about the economy
ART |  The Obama administration has spent $1 trillion.
+PARA-COMET |  The Obama’s spending is much less than Bush’s.
D3 |  St Premise |  The morning-after pill has a number of side effects.
St Claim |  The morning-after pill should only be prescribed after counselling by a physician or pharmacist.,
Gold |  Physicians and pharmacists inform about side effects.
Zero-shot |  Morning-after pills are not FDA approved, they should be avoided .
ART |  The morning- after pill can cause depression.
+PARA-COMET |  The side effects can be very serious.
Table 6: Enthymeme generation for a given stated Premise and Claim by BART in
3 settings: zero-shot; fine-tuned on ART; and fine-tuned on ART \+ PARA-COMET.
Text bolded in green displays how generations are more plausible due to
incorporation of discourse aware commonsense.

##  6 Conclusions

We propose an end-to-end approach for a new task of _automatically generating
an implicit premise given an enthymeme_. We show how leveraging abductive
reasoning as an auxiliary task improves over zero-shot performance of a state-
of-the-art generative language model. Finally, we build a knowledge-enhanced
model by encoding discourse-aware commonsense that outperforms all existing
baselines in terms of automatic metrics as well as plausibility judgements
from crowdworkers. Future work includes exploring other sources for
commonsense knowledge, experimenting with improved decoding techniques, as
well as studying the role of argument schemes in enthymemes reconstruction.

##  7 Ethical Considerations

Although we use language models trained on data collected from the Web, which
have been shown to have issues with bias and abusive language Sheng et al.
(2019); Wallace et al. (2019), the inductive bias of our models should limit
inadvertent negative impacts. Unlike model variants such as GPT, BART is a
conditional language model, which provides more control of the generated
output. Finally, we finetune our model on the ART dataset, which is built on
five sentence short stories which is devoid of harmful and toxic text
especially targeted at marginalized communities.

While dual-use concerns are certainly possible here, we think that open-
sourcing this technology will help to facilitate understanding of arguments
with more balanced and better reasoning. The technology should be used
responsibly, particularly making sure the generation is controllable by
providing the stated premise, claim and any commonsense knowledge pertaining
to the enthymeme in textual form. Finally, we pay the Turkers $15/hour,
complying with minimum wage standards in US.

## References

  * Alshomary et al. (2020) Milad Alshomary, Shahbaz Syed, Martin Potthast, and Henning Wachsmuth. 2020.  Target inference in argument conclusion generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4334–4345, Online. Association for Computational Linguistics.
  * Becker et al. (2020) Maria Becker, Katharina Korfhage, and Anette Frank. 2020.  Implicit knowledge in argumentative texts: An annotated corpus.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 2316–2324, Marseille, France. European Language Resources Association.
  * Bhagavatula et al. (2020) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020.  Abductive commonsense reasoning.  In _International Conference on Learning Representations_.
  * Boltužić and Šnajder (2016) Filip Boltužić and Jan Šnajder. 2016.  Fill the gap! analyzing implicit premises between claims from online debates.  In _Proceedings of the Third Workshop on Argument Mining (ArgMining2016)_ , Berlin, Germany. Association for Computational Linguistics.
  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019.  COMET: Commonsense transformers for automatic knowledge graph construction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4762–4779, Florence, Italy. Association for Computational Linguistics.
  * Davison et al. (2019) Joe Davison, Joshua Feldman, and Alexander Rush. 2019.  Commonsense knowledge mining from pretrained models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 1173–1178, Hong Kong, China. Association for Computational Linguistics.
  * Gabriel et al. (2021) Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, and Yejin Choi. 2021.  Paragraph-level commonsense transformers with recurrent memory.  In _AAAI_.
  * Habernal et al. (2018) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018.  The argument reasoning comprehension task: Identification and reconstruction of implicit warrants.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1930–1940, New Orleans, Louisiana. Association for Computational Linguistics.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Mostafazadeh et al. (2016) Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 839–849.
  * Musi et al. (2018) Elena Musi, Manfred Stede, Leonard Kriese, Smaranda Muresan, and Andrea Rocci. 2018\.  A multi-layer annotated corpus of argumentative text: From argument schemes to discourse relations.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting on association for computational linguistics_ , pages 311–318. Association for Computational Linguistics.
  * (13) Andreas Peldszus and Manfred Stede.  An annotated corpus of argumentative microtexts.
  * Rajendran et al. (2016) Pavithra Rajendran, Danushka Bollegala, and Simon Parsons. 2016.  Contextual stance classification of opinions: A step towards enthymeme reconstruction in online reviews.  In _Proceedings of the Third Workshop on Argument Mining (ArgMining2016)_ , pages 31–39, Berlin, Germany. Association for Computational Linguistics.
  * Razuvayevskaya and Teufel (2017) Olesya Razuvayevskaya and Simone Teufel. 2017.  Finding enthymemes in real-world texts: A feasibility study.  _Argument Comput._ , 8:113–129.
  * Reisert et al. (2015) Paul Reisert, Naoya Inoue, Naoaki Okazaki, and Kentaro Inui. 2015.  A computational approach for generating toulmin model argumentation.  In _Proceedings of the 2nd Workshop on Argumentation Mining_ , pages 45–55, Denver, CO. Association for Computational Linguistics.
  * Sabre (1990) Ru Michael Sabre. 1990.  Peirce’s abductive argument and the enthymeme.  volume 26, pages 363–372. Indiana University Press.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A. Smith, and Yejin Choi. 2019.  Atomic: An atlas of machine commonsense for if-then reasoning.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 33(01):3027–3035.
  * Sheng et al. (2019) Emily Sheng, Kai-Wei Chang, Premkumar Natarajan, and Nanyun Peng. 2019.  The woman worked as a babysitter: On biases in language generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3407–3412, Hong Kong, China. Association for Computational Linguistics.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Walton and Reed (2005) D. Walton and C.A. Reed. 2005.  Argumentation schemes and enthymemes.  _Synthese_ , 145:339–370.
  * Zhou et al. (2020) Xuhui Zhou, Y. Zhang, Leyang Cui, and Dandan Huang. 2020.  Evaluating commonsense in pre-trained language models.  _ArXiv_ , abs/1911.11931.