# Leveraging Affirmative Interpretations from Negation
Improves Natural Language Understanding

Md Mosharaf Hossain8,U Work was done prior to joining Amazon. Eduardo Blanco7
8Department of Computer Science and Engineering, University of North Texas
UAmazon
7Department of Computer Science, University of Arizona
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Negation poses a challenge in many natural language understanding tasks.
Inspired by the fact that understanding a negated statement often requires
humans to infer affirmative interpretations, in this paper we show that doing
so benefits models for three natural language understanding tasks. We present
an automated procedure to collect pairs of sentences with negation and their
affirmative interpretations, resulting in over 150,000 pairs. Experimental
results show that leveraging these pairs helps (a) T5 generate affirmative
interpretations from negations in a previous benchmark, and (b) a RoBERTa-
based classifier solve the task of natural language inference. We also
leverage our pairs to build a plug-and-play neural generator that given a
negated statement generates an affirmative interpretation. Then, we
incorporate the pretrained generator into a RoBERTa-based classifier for
sentiment analysis and show that doing so improves the results. Crucially, our
proposal does not require any manual effort.

##  1 Introduction

Natural Language Understanding is a crucial component to build intelligent
systems that interact with humans seamlessly. While recent papers sometimes
report so-called superhuman performance, simple adversarial attacks including
adding negation and other input modifications remain a challenge despite they
are obvious to humans Naik et al. (2018); Wallace et al. (2019). Further, many
researchers have found that state-of-the-art systems struggle with texts
containing negation. For example, Kassner and Schütze (2020) show that
pretrained language models such as BERT Devlin et al. (2019) do not
differentiate between negated and non-negated cloze questions (e.g., _Birds
cannot [MASK]_ vs. _Birds can [MASK]_). Other studies show that transformers
perform much worse in many other natural language understanding tasks when
there is a negation in the input sentence Ribeiro et al. (2020); Ettinger
(2020); Hossain et al. (2020b); Hosseini et al. (2021); Hossain et al.
(2022a); Truong et al. (2022).

English-Norwegian (en-no) parallel sentences:
---
(en) _There is no more than one Truth._
(no) _Og det finnes kun en Sannhet._
Backtranslation: _And there is only one truth._
English-Spanish (en-es) parallel sentences:
(en) _The term gained traction only after 1999._
(es) _El término no se popularizó hasta después del 1999._
Backtranslation: _The term was not popular until 1999._
Figure 1: Parallel sentences from bitext corpora (English-Norwegian and
English-Spanish) and backtranslations into English. Either the original
English sentence or the backtranslation contains a negation, and the other one
is an affirmative interpretation. In this paper, we show that leveraging
sentences with negation and their affirmative interpretations is beneficial
for several natural language understanding tasks including natural language
inference and sentiment analysis.

In this paper, we address this challenge building upon the following
observation: negation often carries affirmative meanings Horn (1989); Hasson
and Glucksberg (2006). For example, people intuitively understand that _John
read part of the book_ from _John didn’t read the whole book_. Our fundamental
idea is to leverage a large collection of sentences containing negation and
their affirmative interpretations. We define an affirmative interpretation as
a semantically equivalent sentence that does not contain negation. We explore
this idea by automatically collecting pairs of sentences with negation and
their affirmative interpretations from parallel corpora and backtranslating.
Figure 1 exemplifies the idea with English-Norwegian and English-Spanish
parallel sentences. Note that (a) either the original English sentence or the
backtranslation have a negation (the one that does not is the affirmative
interpretation) and (b) the meaning of both is equivalent.

Armed with the large collection of sentences containing negation and their
affirmative interpretations, we show that leveraging them yields improvements
in three natural language understanding tasks. First, we address the problem
of generating affirmatively interpretations in the AFIN benchmark Hossain et
al. (2022b), a collection of sentences with negation and their manually
curated affirmative interpretations. Second, we address natural language
inference using three common benchmarks: RTE Dagan et al. (2006); Bar-Haim et
al. (2006); Giampiccolo et al. (2007); Bentivogli et al. (2009), SNLI Bowman
et al. (2015), and MNLI Williams et al. (2018). Third, we address sentiment
analysis using SST-2 Socher et al. (2013). The main contributions of this
paper are:111Code and data available at
https://github.com/mosharafhossain/large-afin-and-nlu.

  1. 1.

A large collection (153,273) of pairs of sentences containing negation and
their affirmative interpretations. We present an automated procedure to get
these pairs and an analysis of the negation types (single tokens,
morphological, lexicalized, etc.).

  2. 2.

Experimental results with the T5 transformer Raffel et al. (2020) showing that
blending our pairs during the fine-tuning process is beneficial to generate
affirmative interpretations from the negations in AFIN.

  3. 3.

Experimental results showing that a RoBERTa-based classifier Liu et al. (2019)
to solve the task of natural language inference benefits from training with
new premise-hypothesis derived from our pairs (two entailments per pair).

  4. 4.

Experimental results showing that a RoBERTa-based classifier for sentiment
analysis benefits from a novel component that automatically generates
affirmative interpretations from the input sentence.

The key resource enabling the experimental results is our large collection of
pairs of sentences containing negation and their affirmative interpretations.
As we shall see, the experiments under (2) and (3) are a somewhat
straightforward applications of these pairs. The affirmative interpretation
generator we use to improve sentiment analysis, however, has the potential to
improve many natural language understanding tasks.

##  2 Related Work

Solving natural language understanding tasks when the input text contains
negation is challenging. Researchers have approached negation processing
mainly by identifying the scope Vincze et al. (2008); Morante and Daelemans
(2012a) and focus Blanco and Moldovan (2011). Scope refers to the part of the
meaning that is negated and focus refers to the part of the scope that is most
prominently negated Huddleston and Pullum (2002). There are many works
targeting scope detection Fancellu et al. (2016, 2017); Li and Lu (2018);
Jumelet and Hupkes (2018); Chen (2019); Zhao and Bethard (2020) and focus
detection Zou et al. (2014, 2015); Shen et al. (2019); Hossain et al. (2020a).
While scope and focus pinpoint what is and what is not negated, they do not
reveal affirmative interpretations as defined in this paper. Additionally,
there is limited empirical evidence showing that scope or focus is beneficial
to solve a natural language understanding task. Jiménez-Zafra et al. (2021)
show that scope improves sentiment analysis, but they do not experiment with
modern networks that may not benefit from explicit scope information.

Outside of scope and focus, Jiang et al. (2021) work with commonsense
implications involving negations (e.g., “If X doesn’t wear a mask” then “X is
seen as carefree”). Closer to our work, Hosseini et al. (2021) pretrain BERT
with an unlikelihood loss calculated with automatically obtained negated
statements. Their negated statements do not preserve meaning. The authors show
that their method, BERTNOT, outperforms BERT with LAMA Kassner and Schütze
(2020) and the same natural language inference corpora we work with. The work
proposed here outperforms theirs (Section 4.2) and does not require any manual
effort.

We are not the first to work with affirmative interpretations from negated
statements. For example, Sarabi et al. (2019) create a small corpus of verbal
negations retrieved from Simple Wikipedia and their affirmative
interpretations (total: 5,900). Simple Wikipedia is a version of Wikipedia
that uses shorter sentences and simpler language. Hossain et al. (2022b)
propose a question-answer driven approach to create AFIN, a collection of
3,001 sentences with negation and their affirmative interpretations. Both of
these previous efforts employ humans to collect affirmative interpretations
and neither one conducts extrinsic evaluations. Unlike them, we automatically
collect pairs of sentences with negation and their affirmative
interpretations. Additionally, extrinsic evaluations show that despite our
collection procedure is noisy, leveraging our pairs is beneficial to solve
three natural language understanding tasks.

##  3 Collecting Sentences with Negation and Their Affirmative Interpretations

This section outlines our approach to create a large collection of sentences
containing negation and their affirmative interpretations. First, we present
the sources of parallel corpora we work with. Second, we describe our
multilingual negation cue detector to identify negation cues in the parallel
sentences. Third, we describe the backtranslation step and a few checks to
improve quality. Lastly, we present an analysis of the resulting sentences
with negation and their affirmative interpretations.

###  3.1 Selecting Parallel Corpora

We select parallel sentences in English and either Norwegian or Spanish for
two reasons: (a) large parallel corpora are available in these language pairs
and (b) negation cue annotations are available in monolingual corpora for the
three languages. The latter is a requirement to build a multilingual cue
detector (Section 3.2). We extract the parallel sentences from three parallel
corpora available in the OPUS portal Tiedemann (2012)): WikiMatrix Schwenk et
al. (2021a), CCMatrix Schwenk et al. (2021b); Fan et al. (2021), and UNPC
Ziemski et al. (2016). Table 1 (Column 3) shows the number of parallel
sentences we collect from each of the corpora and language pair (total: 14.6
million).

| Source | #parl. sents. | #pairs | %pairs
---|---|---|---|---
en-no | WikiMatrix | 530,000 | 10,274 | 1.94
CCMatrix | 8,000,000 | 73,394 | 0.92
en-es | UNPC | 2,800,000 | 28,028 | 1.00
WikiMatrix | 3,290,000 | 41,577 | 1.26
| All | 14,620,000 | 153,273 | 1.05
Table 1:  Number of parallel sentences in the English-Norwegian and English-
Spanish parallel corpora we work with, and pairs of sentences with negation
and affirmative interpretations we automatically generate via backtranslation.
The yield (%pairs) is low, but as we shall see these pairs are useful to solve
natural language understanding tasks when negation is present without hurting
results when negation is not present.

###  3.2 Identifying Negation Cues in Multiple Languages

In order to detect negation in the parallel sentences, we develop a
multilingual negation cue detector that works with English, Norwegian, and
Spanish texts. To this end, we fine-tune a multilingual BERT
(mBERT)222https://github.com/google-research/bert/blob/master/multilingual.md
Devlin et al. (2019) with negation cue annotations in the three languages we
work with: English Morante and Daelemans (2012b), Norwegian Mæhlum et al.
(2021), and Spanish Jiménez-Zafra et al. (2018). We fine-tune jointly for all
three languages by combining the original training splits into a multilingual
training split. We terminate the training process after the F1 score in the
(combined) development split does not increase for 5 epochs; the final model
is the one which yields the highest F1 score during the training process.
Additional details regarding training procedure and hyperparameters are
provided in Appendix A. Our multilingual detector is not perfect but obtains
competitive results (F1 scores): English: 91.96 (test split), Norwegian: 93.40
(test split), and Spanish: 84.41 (dev split, as gold annotations for the test
split are not publicly available). The system detects various negation cue
types including single tokens (no, never, etc.), affixal, and lexicalized
negations (Section 3.4).

We use our multilingual cue detector to detect negation in the 14.6 million of
parallel sentences. In the English-Norwegian parallel sentences (8.5M),
negation is present in both sentences (WikiMatrix: 7.3%, CCMatrix: 14.2%),
either sentence (WikiMatrix: 5.2%, CCMatrix: 5.2%), or neither sentence
(WikiMatrix: 87.5%, CCMatrix: 80.6%). Similarly, in English-Spanish parallel
sentences, negation is present in both sentences (UNPC: 10.7%, WikiMatrix:
5.7%), either sentence (UNPC: 4.6%, WikiMatrix: 4.4%), or neither sentence
(UNPC: 84.7%, WikiMatrix: 89.9%). Since we are interested in sentences
containing negation and their affirmative interpretations, we only keep the
sentences in which either the source or target sentence contains negation.

Negation Type | Examples
---|---
Single tokens (49.6%) |  They are still not integrated into the German community.
_Cues: not, n’t, no, never, without, nothing, nowhere, nobody, none, etc._ | They have yet to integrate into German society.
I have no doubt that we will reach our goal.
| We shall surely get there!
|  This process allows for higher precision that could never be achieved by
hand.
| This process allows more precision than anyone performed manually.
Affixal (30.15%) |  The north wing was left largely untouched and forms the present house.
_Cues: un-, in-, -less, etc._ | Only the North wing remained quite intact, and constitutes the current house.
|  We fall in love, and any attempt at logic is useless.
| We fall in love and any attempt at logic is futile.
Lexicalized (8.76%) |  A further problem was the lack of skilled labour.
_Cues: prevent, lack, etc._ | Another problem was the issue of obtaining sufficiently qualified personnel.
Multitoken (2.58%) |  After some time, the drainage of water no longer occurs.
_Cues: no longer, not at all, etc._ | After a certain time, the drainage of water ends.
Multiple negations (8.95%) |  The declaration before the courts is not valid if the child is not 14 years old.
| Any statement in a court is invalid if the child is below 14 years of age.
Table 2: Examples of sentences with negation and their affirmative
interpretations automatically obtained from bitext corpora via
backtranslation. We present examples for several negation types; common
_single-tokens_ that are not lexicalized negations are the most frequent.
These sentences with negations and their affirmative interpretations come from
our collection (Section 3) and include errors. For example, the affirmative
interpretation in the second to last example includes _ends_ , a lexicalized
negation, because our cue detector did not identify it.

###  3.3 Generating Affirmative Interpretations

After identifying negation cues in the parallel sentences, we backtranslate
into English the sentence in the target language (either Norwegian or Spanish;
they may or may not contain a negation). In particular, we utilize Google
Cloud Translation API.333Google Translate API -
https://cloud.google.com/translate Before backtranslating, we exclude
sentences in the target language if they are longer than 40 tokens, as longer
sentences tend to result in lower translation quality Fonteyne et al. (2020).

Backtranslating into English from either Norwegian or Spanish may introduce or
remove a negation cue. We discard such backtranslations since our goal is to
obtain pairs of sentences containing negation and its affirmative
interpretation (i.e., a semantically equivalent sentence that does not contain
negation). The last two columns in Table 1 present how many pairs we obtain
(total: 153,273). While the yield is small (1.05%), we note that the process
is automated and could be expanded to use additional parallel corpora.

###  3.4 Quality and Analysis

The process to collect pairs of sentences with negation and their affirmative
interpretations is noisy. First, the negation cue detector is not perfect thus
there are pairs in which the affirmative interpretation contains a negation.
Additionally, backtranslating introduces errors thus the affirmative
interpretations are not always semantically equivalent to the sentences
containing negation. Our goal is not to create a gold standard but to collect
a large collection that we can leverage to improve models for natural language
understanding tasks (Section 4).

Despite 100% correctness is not the goal, we conducted a manual validation
with a random sample of 100 pairs. We discovered that 78% are correct, where
correct means that the affirmative interpretation satisfies the definition
(i.e., no negation and semantically equivalent to the sentence with negation).
We found two main reasons for incorrect pairs. First, the negation cue
detector sometimes fails to detect cues, resulting in affirmative
interpretations that contain negation (e.g., _The execution had been
unlawful_: _This act would have been illegal_; the prefix _il-_ is not
identified as a negation cue). Second, the backtranslation sometimes results
in the original sentence without the negation cue and thus opposite meanings.
(e.g., English-Norwegian parallel sentences: _How can you not enjoy this
trip!_, _Hvordan kan du nyte denne turen!_ ; backtranslation: _How can you
enjoy this trip!_).

#### Analyzing the Negations and Affirmative Interpretations

The negation cue detector identifies several types of negation cues. As a
result, our collection of pairs of sentences with negation and their
affirmative interpretations includes several negation types (Table 2). Note
that this table presents real examples from our collection including erroneous
ones (e.g., some affirmative interpretations contain negation). The most
frequent negation type (49.6%) are common single-token negation cues such as
_not_ , _n’t_ and _never_. Affixal negations are surprisingly common (30.15%)
and include both prefixes (e.g., _un touched_) and suffixes (_use less_).
Lexicalized negations usually take the form of a noun (e.g., lack, dismissal)
or verb (e.g., prevent, avoid) and account for almost 9%. Finally, a few
negations (2.58%) are multitoken (e.g., no longer, not at all), and several
negations (almost 9%) appear in sentences with at least one more negation.

The corresponding affirmative interpretation is never just the original
sentence with negation after removing the negation cue—doing so results in a
sentence that is not semantically equivalent. The required modification are
sometimes relatively simple and mainly require swapping a verb or adjective.
For example, _are still not integrated_ becomes _have yet to integrate_ ,
_largely untouched_ becomes _quite intact_ , and _is use less_ becomes _is
futile_. Yet the affirmative interpretation often is a more thorough rewrite
of the original sentence:

  * •

_I have no doubt that we will reach our goal._ becomes _We shall surely get
there!_ ;

  * •

_higher precision that could never be achieved by hand_ becomes _more
precision than anyone performed manually; and_

  * •

_lack of skilled labour_ becomes _issue of obtaining sufficiently qualified
personnel_.

##  4 Experiments with Natural Language Understanding Tasks

We leverage our collection of sentences with negation and their affirmative
interpretations to enhance models for three natural language understanding
tasks. First, we leverage them in a blending training setup to generate
affirmative interpretations from the negations in a previous benchmark
(Section 4.1). Second, we leverage them to create new premise-hypothesis pairs
and build more robust models for natural language inference (Section 4.2).
Third, we use them to train a plug-and-play neural component to generate
affirmative interpretations from negation. Then, we incorporate the generator
into the task of sentiment analysis (Section 4.3). We use existing corpora for
all tasks as described below; for natural language inference and sentiment
analysis we use the versions released by the GLUE benchmark Wang et al.
(2018).

Our experimental results show that leveraging the large collection of
negations and their affirmative interpretations improves results across all
tasks using previously proposed benchmarks. Specifically, we obtain either
slightly better or comparable results when negation is not present in the
input, and always better results when negation is present.

###  4.1 Generating Affirmative Interpretations from Negation

There are a couple corpora with sentences containing negation and their
manually curated affirmative interpretations (Section 2). In our first
experiment, we explore whether leveraging our collection is beneficial to
generate affirmative interpretations from the negations in AFIN Hossain et al.
(2022b), a manually curated corpus that is publicly available. AFIN contains
3,001 sentences with negation and their affirmative interpretations. Unlike
our collection (Section 3), AFIN only considers verbal negations (i.e., the
negation cues always modify a verb). Here are some examples:

  * •

_It was not formed by a natural process._
_It was formed by an artificial process._

  * •

_An extinct volcano is one that has not erupted in recent history._
_An extinct volcano erupted in the past._

The AFIN authors experiment with the T5 Raffel et al. (2020) transformer to
automatically generate affirmative interpretations. While the task remains a
challenge and our results are much worse than the human upper bound, we show
that incorporating our collection of sentences containing negation and their
affirmative interpretations during the training process results in a more
robust generator.

#### Blending Our Collection of Negations and Affirmative Interpretations

We adopt a blending technique by Shnarch et al. (2018) in order to maximize
the chances that the training process benefits from our collection of
negations and affirmative interpretations. Since our collection is much larger
than the training split in AFIN (153k vs. 2.1k), simply adding our collection
to the training split and fine-tuning T5 as usual would result in a model that
underperforms with AFIN. There are three phases in the training process. In
the first phase, we fine-tune T5 with the combination of our collection and
the training split in AFIN for $m$ epochs. In the second phase, we continue to
fine-tune T5 blending our collection and the training split in AFIN for $n$
epochs. The blending factor ($[0..1]$) determines the number of instances from
our collection that we incorporate in each epoch. This number decreases after
each epoch. In the third phase, we fine-tune using the training split in AFIN
for $k$ epochs. We refer the reader to Appendix B for additional details on
the training process and hyperparameters.

| BLEU | chrf++ | METEOR
---|---|---|---
T5 transformer | 26.5 | 50.5 | 43.5
\+ blending Ours | 28.6 | 52.5 | 45.8
Table 3: Automatic evaluation of the T5 transformer on the task of generating affirmative interpretations as defined in the AFIN benchmark Hossain et al. (2022b). Blending our pairs in the process of fine-tuning T5 yields improvements with all metrics.  | Validation Scores
---|---
| 4 | 3 | 2 | 1 | 0
Upper Bound | 86.2 | 11.6 | 2.0 | 0.2 | n/a
T5 transformer | 32.0 | 15.3 | 12.0 | 3.3 | 37.3
\+ blending Ours | 37.0 | 14.0 | 13.0 | 10.0 | 26.0
Table 4:  Manual evaluation of the T5 transformer on the task of generating
affirmative interpretations as defined in the AFIN benchmark. The upper bound
comes from the manual validation by the creators of AFIN. Blending our pairs
in the fine-tuning process generates more correct interpretations (higher
validation scores are better).

#### Results and Discussion

Table 3 presents the evaluation with the test split in AFIN using automatic
metrics: BLEU-2 Papineni et al. (2002), chrf++ Popović (2017), and METEOR
Banerjee and Lavie (2005). We obtained these scores comparing the gold
affirmative interpretations in AFIN and the predicted ones by T5. Despite our
collection of negations and affirmative interpretation is noisy, out-of-
domain, and considers more negation types, leveraging it is beneficial.
Indeed, we observe improvements across the three metrics (BLEU-2: 28.6 vs.
26.5, chrf++: 52.5 vs. 50.5, and METEOR: 45.8 vs. 43.5).

|  | RTE |  | SNLI |  | MNLI
---|---|---|---|---|---|---
|  | dev | neg. P-H |  | dev | neg. P-H |  | dev | neg. P-H
w/o negation fine-tuning |  |  |  |  |  |  |  |  |
BERT Devlin et al. (2019) |  | 66.10 | 57.60 |  | 89.90 | 44.40 |  | 83.20 | 63.90
XLNet Yang et al. (2019) |  | 69.90 | 60.90 |  | 90.60 | 51.50 |  | 86.70 | 66.30
RoBERTa Liu et al. (2019) |  | 75.80 | 62.50 |  | 91.60 | 51.90 |  | 87.90 | 66.70
w/ negation fine-tuning |  |  |  |  |  |  |  |  |
BERTNOT Hosseini et al. (2021) |  | 69.68 | 74.47 |  | 89.00 | 45.96 |  | 84.31 | 60.89
RoBERTa blending Ours |  | 77.62 | 78.13 |  | 91.35 | 54.87 |  | 87.00 | 67.89
Table 5:  Results (accuracy) using several transformers and (a) the
development splits in RTE, SNLI, and MNLI, and (b) the _new_ premise-
hypothesis containing negation (neg. P-H) from Hossain et al. (2020b). RoBERTa
blending new premise-hypothesis derived from our sentences with negation and
their affirmative interpretations substantially outperforms the three
transformers without any negation fine-tuning and BERTNOT with the new
premise-hypothesis that contain negation while obtaining comparable results
with the original development splits.

#### Manual Validation

Automatic metrics for generation tasks are useful but have well-known
limitations Mathur et al. (2020); Zhang et al. (2020). Following the AFIN
authors, we also conduct a manual evaluation. Specifically, we validate a
sample of 100 automatically generated affirmative interpretations. The
validation consists in assigning a score indicating how confident they are in
the correctness of an affirmative interpretation given the sentence containing
negation (4: extremely confident, 3: very confident, 2: moderately confident,
1: slightly confident). We also include 0 to indicate that the affirmative
interpretations is wrong. We show examples of each score in Appendix B.

Table 4 shows the manual evaluation. Blending our collection of negations and
affirmative interpretations yields better results. While still far from the
upper bound, blending increases the confidence scores. Most notably, the
percentage of incorrect affirmative interpretations decreases from 37.3% to
26.0% ($\Delta=-30.3\%$).

###  4.2 Natural Language Inference

Our collection of pairs of sentences with negation and their affirmative
interpretations can be seen as semantically equivalent sentences in which only
one statement contains negation. By definition, there are two entailment
relationships between semantically equivalent sentences—using either sentence
as premise and the other one as hypothesis. We thus create two premise-
hypothesis sets from each pair in our collection and label them as
_entailment_ to create a large collection of entailments involving negation.
For example, we generate the following entailments from the pair (_The
universal nature of these rights and freedoms does not admit doubts_, _The
universal nature of these rights and freedoms is beyond question_):

  * •

Premise: _The universal nature of these rights and freedoms does not admit
doubts._
Hypothesis: _The universal nature of these rights and freedoms is beyond
question._

  * •

Premise: _The universal nature of these rights and freedoms is beyond
question._
Hypothesis: _The universal nature of these rights and freedoms does not admit
doubts._

This process results in 306,546 new premise-hypothesis annotated _entailment_
(2 per pair in our collection) without any manual effort.

We experiment with (a) three transformer-based classifiers without any fine-
tuning designed to improve results when there is a negation in the premise or
hypothesis, (b) BERTNOT Hosseini et al. (2021), a BERT transformer pretrained
with a modified loss calculated in part with automatically obtained negated
statements, and (c) a RoBERTa-based classifier blending our 306k new
_entailment_ premise-hypothesis using the strategy presented in Section 4.1.
We refer the reader to Appendix C for additional details about the models,
training process, and hyperparameters. Regarding corpora, we work with RTE
Dagan et al. (2006); Bar-Haim et al. (2006); Giampiccolo et al. (2007);
Bentivogli et al. (2009), SNLI Bowman et al. (2015), and MNLI Williams et al.
(2018). Additionally, we work with the 4,500 new premise-hypothesis pairs by
Hossain et al. (2020b), who derive them from RTE, SNLI and MNLI by adding a
negation to a premise, hypothesis or both.

#### Results and Discussion

Table 5 presents the results. We train all models with the corresponding
training split, except _RoBERTa blending Ours_ , which also blends our 306k
new premise-hypothesis pairs during the training process. We present results
with the corresponding development split (gold labels for the test split are
not available for all them) and the premise-hypothesis including negation. We
find that blending our 306k premise-hypothesis is beneficial despite these
pairs (a) only include entailments and (b) inherit the errors present in our
collection of sentences with negation and their affirmative interpretations.
With the original development splits in RTE, SNLI, and MNLI, we either obtain
slightly better results (RTE, +1.82) or comparable (SNLI: -0.25, MNLI: -0.90).
The improvements are consistent, however, with the pairs that include negation
(neg. P-H). RTE and SNLI benefit the most (78.13 vs. 62.50, 54.87 vs. 51.90).
We hypothesize that MNLI benefits the least (67.89 vs. 66.70) because premises
in MNLI are often multiple sentences and our new premise-hypothesis are always
single sentences.

Figure 2:  Standard architecture for a transformer-based classifier (top) and
modification to include our affirmative interpretation generator (bottom). If
a sentence contains a negation, we provide RoBERTa with the original sentence
and the affirmative interpretation. The generator is pretrained with our large
collection of sentences with negation and their affirmative interpretations.

###  4.3 Sentiment Analysis

We close our experiments exploring the task of sentiment analysis (i.e.,
classifying sentences according to their sentiment: positive or negative). Our
motivation is that state-of-the-art systems for sentiment analysis face
challenges with negation Ribeiro et al. (2020). Unlike generating affirmative
interpretations (Section 4.1) and natural language inference (Section 4.2),
however, it is unclear how to leverage our large collection of sentences with
negation and their affirmative interpretations to alleviate the issue.

We propose a task-agnostic solution: complement input sentences containing
negation with their automatically generated affirmative interpretations.
Figure 2 illustrates this proposal in the architecture at the bottom. Rather
than feeding all sentences to a transformer-based classifier (top), we first
check whether input sentences contain a negation with our cue detector
(Section 3). If they do not, we feed them to the classifier as usual. If they
do, we (a) automatically generate its affirmative interpretation with the
generator described in Section 4.1 and (b) feed to the transformer-based
classifier both the original sentence with negation and the affirmative
interpretation separated by the [SEP] token. Appendix D provides additional
details about the training process.

We experiment here with RoBERTa as it produces very competitive results. Note
that our strategy to complement negated inputs with their affirmative
interpretations could be used with any classifier for any task as long as it
takes a text as its input. Regarding corpora, we use SST-2 Socher et al.
(2013) as released by GLUE Wang et al. (2018). It consists of 70,042 movie
reviews and sentiment annotations for each sentence.

Here are a few examples of automatically generated affirmative interpretations
from sentences containing negation in SST-2:

  * •

_It is not a bad film._
Affirmative interpretation: _It is a good movie._

  * •

_She may not be real, but the laughs are._
Affirmative interpretation: _She is fictional._

  * •

_He feels like a spectator and not a participant_
Affirmative interpretation: _He feels like a spectator rather than
participant._

  * •

_The movie has no idea of it is serious._
Affirmative interpretation: _The movie has a lack of idea that it is serious._

  * •

_A thriller without a lot of thrills._
Aff. interpretation: _A thriller with little thrills._

Note that they are by no means perfect, but they mostly preserve meaning while
not using negation. For example, the second affirmative interpretation only
covers the meaning of part of the original sentence with negation (i.e., _She
may not be real_), and the second to last includes a negation (_lack_).

| affirmative intpn. generator?
---|---
|  No |  Yes
dev. w/o neg. |  94.0 |  94.7 (+0.7%)
dev. w/ neg. |  93.0 |  94.8 (+1.9%)
important negs. |  86.0 |  89.8 (+4.4%)
unimportant negs. |  95.0 |  95.8 (+0.8%)
Table 6: Results (macro F1) with RoBERTa using the SST-2 development split. We
provide results with instances that contain and do not contain negation as
well as important and unimportant negations. Our affirmative interpretation
generator yields improvements across the board, especially with instances
containing important negations (i.e., when removing the negation changes the
sentiment polarity (positive or negative).

#### Results and Discussion

Table 6 presents the results (macro F1). We provide results with the sentences
in the development split depending on whether they contain a negation (gold
labels for the test split are not publicly available). Additionally, we use
the grouping of negations by Hossain et al. (2022a): important or unimportant.
A negation is unimportant if removing it does not change the sentiment (e.g.,
both _I got a headache watching this meaning less downer_ and _I got a
headache watching this downer_ are negative.

Incorporating our affirmative interpretation generator is always beneficial.
This includes instances containing negation (94.8 vs. 93.0) and, surprisingly,
instances that do not contain negation (94.7 vs. 94.0). We hypothesize that
this is the case because our negation cue detector is not perfect thus
instances are sometimes fed through the incorrect branch after the _Negation?_
fork (Figure 2). As one would expect, the generator makes the classifier more
robust with important negations (89.8 vs. 86), but we also observe
improvements with unimportant negations (95.8 vs. 95.0).

##  5 Conclusions

Negation poses a challenge for natural language understanding. Understanding
negation requires humans to infer affirmative meanings Horn (1989) (e.g., _The
lot has not been vacant_ conveys _The lot has been occupied_). Inspired by
this insight, we collect a large collection (153k) of pairs of sentences
containing negation and their affirmative interpretations. We define the
latter as a semantically equivalent sentence that does not contain negation.
Our collection process relies on parallel corpora and backtranslation and is
automated.

We show that leveraging our collection is beneficial to solve three natural
language understanding tasks: (a) generating affirmative interpretations, (b)
natural language inference, and (c) sentiment analysis. All our experiments
use out-of-domain, manually curated corpora. Crucially, our proposal yields
better results when negation is present in the input while slightly improving
or obtaining comparable results when it is not. Additionally, our proposal
does not require any manual annotations.

## Limitations

In order to create a large collection of sentences with negation and their
affirmative interpretations, we use publicly available parallel sentences. We
note that in two of the three sources (i.e., WikiMatrix Schwenk et al. (2021a)
and CCMatrix Schwenk et al. (2021b); Fan et al. (2021)), the authors use auto-
alignment methodologies to collect the parallel sentences. This step may
introduce errors in the original sources. Next, to detect negation cues in the
huge collections of parallel sentences (14.6 millions), we develop a
multilingual cue detection system that is certainly not 100% perfect. While
the cue detector performs well on the negation corpora it is trained with
(Section 3.2), some incorrect predictions can be expected in the parallel
corpora we use. Furthermore, the translation API introduces additional noise
backtranslating from Norwegian or Spanish into English (Section 3.3).
Regarding models and experiments, we leverage RoBERTa and T5 (Section 4) as
systems based on them perform well on natural language understanding
tasks.444https://super.gluebenchmark.com/leaderboard However, we acknowledge
that other transformers such as XLNet Yang et al. (2019) and DeBERTa He et al.
(2021) may yield better results.

## Acknowledgements

This material is based upon work supported by the National Science Foundation
under Grant No. 1845757. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the authors and do not
necessarily reflect the views of the NSF. Computational resources were
provided by the UNT office of High-Performance Computing. Further, we
leveraged computational resources from the Chameleon platform Keahey et al.
(2020). We also thank the reviewers for insightful comments.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  Meteor: An automatic metric for mt evaluation with improved correlation with human judgments.  In _Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization_ , pages 65–72.
  * Bar-Haim et al. (2006) Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second pascal recognising textual entailment challenge.  In _Proceedings of the second PASCAL challenges workshop on recognising textual entailment_ , volume 6, pages 6–4. Venice.
  * Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.  The fifth pascal recognizing textual entailment challenge.
  * Blanco and Moldovan (2011) Eduardo Blanco and Dan Moldovan. 2011.  Semantic representation of negation using focus detection.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_ , pages 581–589, Portland, Oregon, USA. Association for Computational Linguistics.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Chen (2019) Long Chen. 2019.  Attention-based deep learning system for negation and assertion detection in clinical notes.  _International Journal of Artificial Intelligence and Applications (IJAIA)_ , 10(1).
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The pascal recognising textual entailment challenge.  In _Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment_ , MLCW’05, pages 177–190, Berlin, Heidelberg. Springer-Verlag.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ettinger (2020) Allyson Ettinger. 2020.  What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Fan et al. (2021) Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021.  Beyond english-centric multilingual machine translation.  _Journal of Machine Learning Research_ , 22(107):1–48.
  * Fancellu et al. (2016) Federico Fancellu, Adam Lopez, and Bonnie Webber. 2016.  Neural networks for negation scope detection.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 495–504, Berlin, Germany. Association for Computational Linguistics.
  * Fancellu et al. (2017) Federico Fancellu, Adam Lopez, Bonnie Webber, and Hangfeng He. 2017.  Detecting negation scope is easy, except when it isn’t.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 58–63, Valencia, Spain. Association for Computational Linguistics.
  * Fonteyne et al. (2020) Margot Fonteyne, Arda Tezcan, and Lieve Macken. 2020.  Literary machine translation under the magnifying glass: Assessing the quality of an NMT-translated detective novel on document level.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 3790–3798, Marseille, France. European Language Resources Association.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third PASCAL recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing_ , pages 1–9, Prague. Association for Computational Linguistics.
  * Hasson and Glucksberg (2006) Uri Hasson and Sam Glucksberg. 2006.  Does understanding negation entail affirmation?: An examination of negated metaphors.  _Journal of Pragmatics_ , 38(7):1015–1032.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Horn (1989) Laurence Horn. 1989.  _A Natural History of Negation_.  University of Chicago Press.
  * Hossain et al. (2022a) Md Mosharaf Hossain, Dhivya Chinnappa, and Eduardo Blanco. 2022a.  An analysis of negation in natural language understanding corpora.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 716–723, Dublin, Ireland. Association for Computational Linguistics.
  * Hossain et al. (2020a) Md Mosharaf Hossain, Kathleen Hamilton, Alexis Palmer, and Eduardo Blanco. 2020a.  Predicting the focus of negation: Model and error analysis.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8389–8401, Online. Association for Computational Linguistics.
  * Hossain et al. (2022b) Md Mosharaf Hossain, Luke Holman, Anusha Kakileti, Tiffany Iris Kao, Nathan Raul Brito, Aaron Abraham Mathews, and Eduardo Blanco. 2022b.  A question-answer driven approach to reveal affirmative interpretations from verbal negations.  _arXiv preprint arXiv:2205.11467_.
  * Hossain et al. (2020b) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020b.  An analysis of natural language inference benchmarks through the lens of negation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118, Online. Association for Computational Linguistics.
  * Hosseini et al. (2021) Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021.  Understanding by understanding not: Modeling negation in language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1301–1312, Online. Association for Computational Linguistics.
  * Huddleston and Pullum (2002) Rodney D. Huddleston and Geoffrey K. Pullum. 2002.  _The Cambridge Grammar of the English Language_.  Cambridge University Press.
  * Jiang et al. (2021) Liwei Jiang, Antoine Bosselut, Chandra Bhagavatula, and Yejin Choi. 2021.  “I’m not mad”: Commonsense implications of negation and contradiction.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4380–4397, Online. Association for Computational Linguistics.
  * Jiménez-Zafra et al. (2018) Salud María Jiménez-Zafra, Mariona Taulé, M Teresa Martín-Valdivia, L Alfonso Urena-López, and M Antónia Martí. 2018.  Sfu reviewsp-neg: a spanish corpus annotated with negation for sentiment analysis. a typology of negation patterns.  _Language Resources and Evaluation_ , 52(2):533–569.
  * Jiménez-Zafra et al. (2021) Salud María Jiménez-Zafra, Noa P. Cruz-Díaz, Maite Taboada, and María Teresa Martín-Valdivia. 2021.  Negation detection for sentiment analysis: A case study in spanish.  _Natural Language Engineering_ , 27(2):225–248.
  * Jumelet and Hupkes (2018) Jaap Jumelet and Dieuwke Hupkes. 2018.  Do language models understand anything? on the ability of LSTMs to understand negative polarity items.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 222–231, Brussels, Belgium. Association for Computational Linguistics.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Keahey et al. (2020) Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi S. Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. 2020.  Lessons learned from the chameleon testbed.  In _Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC ’20)_. USENIX Association.
  * Li and Lu (2018) Hao Li and Wei Lu. 2018.  Learning with structured representations for negation scope extraction.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 533–539, Melbourne, Australia. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mæhlum et al. (2021) Petter Mæhlum, Jeremy Barnes, Robin Kurtz, Lilja Øvrelid, and Erik Velldal. 2021.  Negation in Norwegian: an annotated dataset.  In _Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa)_ , pages 299–308, Reykjavik, Iceland (Online). Linköping University Electronic Press, Sweden.
  * Mathur et al. (2020) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020.  Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4984–4997, Online. Association for Computational Linguistics.
  * Morante and Daelemans (2012a) Roser Morante and Walter Daelemans. 2012a.  ConanDoyle-neg: Annotation of negation cues and their scope in Conan Doyle stories.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012)_. European Language Resources Association (ELRA).
  * Morante and Daelemans (2012b) Roser Morante and Walter Daelemans. 2012b.  ConanDoyle-neg: Annotation of negation cues and their scope in conan doyle stories.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 1563–1568, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Naik et al. (2018) Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018.  Stress test evaluation for natural language inference.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th annual meeting of the Association for Computational Linguistics_ , pages 311–318.
  * Popović (2017) Maja Popović. 2017.  chrf++: words helping character n-grams.  In _Proceedings of the second conference on machine translation_ , pages 612–618.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Sarabi et al. (2019) Zahra Sarabi, Erin Killian, Eduardo Blanco, and Alexis Palmer. 2019.  A corpus of negations and their underlying positive interpretations.  In _Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019)_ , pages 158–167, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Schwenk et al. (2021a) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021a.  WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1351–1361, Online. Association for Computational Linguistics.
  * Schwenk et al. (2021b) Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021b.  CCMatrix: Mining billions of high-quality parallel sentences on the web.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6490–6500, Online. Association for Computational Linguistics.
  * Shen et al. (2019) Longxiang Shen, Bowei Zou, Yu Hong, Guodong Zhou, Qiaoming Zhu, and AiTi Aw. 2019\.  Negative focus detection via contextual attention mechanism.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2251–2261, Hong Kong, China. Association for Computational Linguistics.
  * Shnarch et al. (2018) Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. 2018.  Will it blend? blending weak and strong labeled data in a neural network for argumentation mining.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 599–605, Melbourne, Australia. Association for Computational Linguistics.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1631–1642, Seattle, Washington, USA. Association for Computational Linguistics.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Truong et al. (2022) Hung Thinh Truong, Yulia Otmakhova, Timothy Baldwin, Trevor Cohn, Karin Verspoor, and Jey Han Lau. 2022.  Not another negation benchmark: The nan-nli test suite for sub-clausal negation.  _arXiv preprint arXiv:2210.03256_.
  * Vincze et al. (2008) Veronika Vincze, György Szarvas, Richárd Farkas, György Móra, and János Csirik. 2008.  The BioScope corpus: biomedical texts annotated for uncertainty, negation and their scopes.  _BMC bioinformatics_ , 9(11):S9.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhao and Bethard (2020) Yiyun Zhao and Steven Bethard. 2020.  How does BERT’s attention change when you fine-tune? an analysis methodology and a case study in negation scope.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4729–4747, Online. Association for Computational Linguistics.
  * Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.  The United Nations parallel corpus v1.0.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 3530–3534, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Zou et al. (2014) Bowei Zou, Guodong Zhou, and Qiaoming Zhu. 2014.  Negation focus identification with contextual discourse information.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 522–530, Baltimore, Maryland. Association for Computational Linguistics.
  * Zou et al. (2015) Bowei Zou, Guodong Zhou, and Qiaoming Zhu. 2015.  Unsupervised negation focus identification with word-topic graph model.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1632–1636, Lisbon, Portugal. Association for Computational Linguistics.

##  Appendix A Additional Details on Identifying Negation Cues in Multiple
Languages

Referring to Section 3.2 of the paper, we employ an off-the-shelf multilingual
BERT-Base model (cased version) pretrained on 104
languages.555https://github.com/google-
research/bert/blob/master/multilingual.md We concatenate the contextualized
representations from the last and third-to-last layers. Then, we pass the
concatenation to a fully connected layer. Finally, we leverage a conditional
random field (CRF) layer that yields the output sequence identifying negation
cues. Since a negation cue can consist of multiple tokens (e.g., _by no
means_), we use the BIO (B: Beginning, I: Inside, and O: Outside) tagging
scheme. The system takes 1.5 hours on average to train on a single core NVIDIA
Tesla V100 (32GB). Table 7 lists the tuned hyperparameters for the cue
detector. We avail the code for all our experiments at
https://github.com/mosharafhossain/large-afin-and-nlu.

Hyperparameter |
---|---
Maximum Epochs | 25
Batch Size | 10
Patience | 5
Maximum sentence length | 150
Optimizer | AdamW
Learning rate (mBERT) | 1e-5
Learning rate (FC and CRF) | 1e-3
Weight decay (mBERT) | 1e-5
Weight decay (FC) | 1e-3
Dropout (mBERT) | 0.5
Gradient clipping | 5.0
Warmup epochs | 5
Accumulate step | 1
Table 7:  Hyperparameters for finetuning the multilingual cue detector (Section 3.2 in the paper). FC refers to Fully Connected layer.  Hyperparameter |
---|---
Maximum Epochs | 20
Batch Size | 8
Patience | 5
Input sentence length (max.) | 80
Target sentence length (max.) | 50
Optimizer | Adafactor
Learning rate | 1e-5
Weight decay | 1e-6
Gradient clipping | 5.0
Warmup epochs | 3
Accumulate step | 1
top_k | 50
top_p | 0.95
repetition_penalty | 2.5
Table 8:  Hyperparameters for finetuning our affirmative interpretation
generator (Section 4.1 in the paper).

|  Examples
---|---
Extremely confident  (Score: 4) |  Move them to low places so that they do not fall.
Affirmative Interpretation: They fall when they are in higher places.
|  The ones that were not rewarded were not marked with fields.
|  Affirmative Interpretation: The ones that were rewarded were marked with
fields.
Very confident  (Score: 3) |  The most recent successful bids for the Olympic and Paralympic Games were in cities that had never hosted them before.
Affirmative Interpretation: Other cities had hosted them once.
|  No other studies could find a link between the vaccine and autism.
|  Affirmative Interpretation: A study found a link between the vaccine and
autism.
Moderately confident  (Score: 2) |  In 1984, because the games were in New York, and because of the boycott, from when we boycotted in 1980, not a lot of European countries came over.
Affirmative Interpretation: Lots of European countries came over in 1980.
|  It occurs when the body does not receive enough iron.
|  Affirmative Interpretation: The body receives too little iron.
Slightly confident  (Score: 1) |  I understand third party candidates have no success.
Affirmative Interpretation: Third party candidates have minimal success.
|  I don’t expect that the lack of British participation will stop any action.
|  Affirmative Interpretation: I expect that the lack of British participation
will slow down any action.
Wrong affirmative interpretation  (Score: 0) |  I have throughout my career not supported needle exchanges as anti-drug policies.
I have supported needle exchanges as anti-drug policies.
|  Unlike other organelles, the ribosome is not surrounded by a membrane.
|  The ribosome is surrounded by a membrane.

Table 9:  Examples of sentences containing negation from AFIN and their
affirmative interpretations automatically generated. The generator uses T5
trained with our large collection of sentences with negation and their
affirmative interpretations (Section 4.1). We show examples of the manual
validation; scores range from 0 to 4.

##  Appendix B Additional Details on Generating Affirmative Interpretations
from Negation

We utilize the Huggingface implementation Wolf et al. (2020) of T5, a
conditional generation model (Section 4.1). In each run, the system requires
approximately 7.2 hours to train on a single core NVIDIA Tesla V100 (32GB).
Table 8 shows the hyperparameters for this experiment. Regarding Section 4.1
in the paper (Manual Validation), we present examples of each score in Table
A.

Hyperparameter | RTE | SNLI | MNLI
---|---|---|---
Maximum epochs | 10 | 8 | 8
Warmup epochs | 4 | 2 | 3
Batch size | 24 | 16 | 10
Patience | 5 | 5 | 5
Maximum sentence length | 100 | 80 | 100
Optimizer | AdamW | AdamW | AdamW
Learning rate | 1e-5 | 1e-5 | 1e-5
Weight decay | 0.0 | 5e-6 | 5e-6
Gradient clipping | 5.0 | 5.0 | 5.0
Dropout | 0.2 | 0.3 | 0.3
Table 10:  Hyperparameters for finetuning RoBERTa with blending our pairs and
the NLI benchmarks (Section 4.2 in the paper).

##  Appendix C Additional Details on Solving Natural Language Inference

We evaluate the systems on the development splits of the NLI benchmarks we
work with (Section 4.2) as test split labels are not publicly available. So,
we randomly select 15% examples of the original training split in order to
tune the hyperparameters and to select the best model during the training
process for each benchmark. We note that we evaluate on the development split
with matched genres for MNLI. In each run, the system (blending with ours)
requires approximately 2.1 hours to train for RTE, 7.8 hours for SNLI, and 9.6
hours for MNLI on a single core NVIDIA Tesla V100 (32GB). Table 10 presents
the hyperparameters we use in this experiment.

Hyperparameter |
---|---
Maximum Epochs | 5
Batch Size | 16
Patience | 3
Maximum sentence length | 80
Optimizer | AdamW
Learning rate | 1e-5
Weight decay | 0.0
Dropout | 0.5
Gradient clipping | 5.0
Warmup epochs | 5
Accumulate step | 1
Table 11:  Hyperparameters for finetuning our SST-2 system presented in
Section 4.3 in the paper.

##  Appendix D Additional Details on Solving Sentiment Analysis

To experiment with SST-2, we randomly select 5% examples of the original
training split for tuning the hyperparameters as well as for selecting the
best model during the training process since test labels are not publicly
available in SST-2 (part of GLUE Wang et al. (2018)). On average, the system
takes half an hour to train on a single core NVIDIA Tesla V100 (32GB). We
share the tuned hyperparameters in Table 11.