#  InferES : A Natural Language Inference Corpus for Spanish
Featuring Negation-Based Contrastive and Adversarial Examples

Venelin Kovatchev
School of Information
The University of Texas at Austin
<EMAIL_ADDRESS>
Mariona Taulé
Centre de Llenguatge i Computació
Institut de Recerca en Sistemes Complexos
Universitat de Barcelona
<EMAIL_ADDRESS>

###### Abstract

In this paper, we present InferES \- an original corpus for Natural Language
Inference (NLI) in European Spanish. We propose, implement, and analyze a
variety of corpus-creating strategies utilizing expert linguists and crowd
workers. The objectives behind InferES are to provide high-quality data, and,
at the same time to facilitate the systematic evaluation of automated systems.
Specifically, we focus on measuring and improving the performance of machine
learning systems on negation-based adversarial examples and their ability to
generalize across out-of-distribution topics.

We train two transformer models on InferES (8,055 gold examples) in a variety
of scenarios. Our best model obtains 72.8% accuracy, leaving a lot of room for
improvement. The “hypothesis-only” baseline performs only 2%-5% higher than
majority, indicating much fewer annotation artifacts than prior work. We find
that models trained on InferES generalize very well across topics (both in-
and out-of-distribution) and perform moderately well on negation-based
adversarial examples.

##  1 Introduction

In the task of Natural Language Inference (NLI), an automated system has to
determine the meaning relation that holds between two texts. The model has to
make a three-way choice between entailment: a hypothesis (h) is true given a
premise (p) (e.g. 1.); contradiction: a hypothesis (h) is false given a
premise (p) (e.g. 2.); or neutral: the truth value of the hypothesis (h)
cannot be determined solely based on the premise (p) (e.g.: 3.).

  * 1.

p) John goes to work every day with a car.
h) John has a job.

  * 2.

p) John goes to work every day with a car.
h) John takes the bus to go to work.

  * 3.

p) John goes to work every day with a car.
h) John has a Porsche.

NLI (formerly known as Recognizing Textual Entailment (RTE)) is one of the
core tasks in the popular benchmarks for Natural Language Understanding GLUE
Wang et al. (2018) and Super GLUE Wang et al. (2019). Hundreds of machine
learning systems compete on these benchmarks, improving the state of NLU.

One key limitation of NLI research is that most of the existing corpora are
only for English. Limited research has been done on multilingual and non-
English corpora Peñas et al. (2006); Conneau et al. (2018); Amirkhani et al.
(2020); Ham et al. (2020); Hu et al. (2020); Mahendra et al. (2021).

Another well-known issue with NLI is the quality of the existing datasets and
the limitations of the models trained on them. On most NLI corpora, state-of-
the-art transformer based models can obtain quantitative results (Accuracy and
F1) that equal or exceed human performance. Despite this high performance,
researchers have identified numerous limitations and potential problems.
Poliak et al. (2018) found that annotation artifacts in the datasets enable
the models to predict the label by only looking at the hypothesis. NLI models
are often prone to adversarial attacks Williams et al. (2018) and may fail on
instances that require specific linguistic capabilities Hossain et al. (2020);
Saha et al. (2020).

In this paper we address both of these shortcomings in NLI research. We
present InferES \- to the best of our knowledge, the first original NLI corpus
for Spanish, not adapted from another language or task. We study prior work
for strategies that can reduce annotation artifacts and increase the
linguistic variety of the corpus, resulting in a dataset that is more
challenging for automated systems to solve. We also design the corpus in a way
that facilitates systematic evaluation of automated systems on: 1) negation-
based adversarial examples; 2) out-of-distribution examples.

We propose, implement, and analyze three different strategies for the
generation and annotation of text pairs. In the generation strategy, expert
linguists write original hypotheses given a premise. In the rewrite strategy,
expert linguists create contrastive and adversarial examples by rewriting and
re-annotating “generated” pairs. In the annotation strategy, we first generate
text pairs in a semi-automated manner and then use crowd annotators to
determine the meaning relation. The final InferES corpus contains 8,055 gold
standard premise-hypothesis pairs. The core part of the corpus is expert-
generated and we make an additional effort to ensure the quality of the data
and the linguistic diversity of the examples.

We provide two baseline for InferES by finetuning multilingual BERT and BETO
(Spanish BERT) transformer models. On the full dataset, BETO obtains 72.8%
accuracy, indicating that the classification task is non-trivial. Both mBERT
and BETO perform poorly in the “hypothesis-only” condition, indicating fewer
annotation artifacts in the corpus compared to prior work. Both systems
generalize well across the different topics in InferES both “in-distribution”
and “out-of-distribution”. We notice a substantial drop in performance when
evaluating negation-based adversarial examples, however the systems still
outperform majority and “hypothesis-only”.

InferES expands the scope of the NLI research in Spanish, provides new set of
naturally occurring contrastive and adversarial examples, and facilitates the
study of negation and coreference in the context of NLI. As part of the corpus
creation, we also present and analyze three unique strategies for creating
examples. All our data and baseline models are being released to the
community111At https://github.com/venelink/inferes InferES is also added as a
HuggingFace dataset.

The rest of this article is organized as follows. Section 2 discusses the
related work. Section 3 formulates our objectives and introduces the different
corpus-creation strategies. Section 4 describes the final corpus and basic
statistical data regarding it. Section 5 presents the machine learning
experimental setup and results. Section 6 is devoted to a discussion of the
results and their implications. Finally, Section 7 concludes the article.

##  2 Related Work

The task of Recognizing Textual Entailment (RTE) was proposed in Dagan et al.
(2006) as a binary classification (“entailment” / “non-entailment”). The RTE
competition ran for seven editions Bar Haim et al. (2006); Giampiccolo et al.
(2007, 2008); Bentivogli et al. (2009, 2010, 2011). RTE was later reformulated
as a three-way decision and ultimately renamed Natural Language Inference in
the SNLI Bowman et al. (2015) and the MNLI Williams et al. (2018) corpora.
Both the RTE and the NLI tasks form part of the Natural Language Understanding
benchmarks GLUE Wang et al. (2018) and Super-GLUE Wang et al. (2019). The NLU
benchmarks attracted a lot of attention from the community and by 2020 the
state-of-the-art systems reported human level performance. Parrish et al.
(2021) proposed a “linguist-in-the-loop” corpus creation to improve the
quality of the data.

The “super-human” performance of NLI systems has been questioned by a number
of researchers. Poliak et al. (2018) found that annotation artifacts in the
datasets enable the models to predict the label by only looking at the
hypothesis. McCoy et al. (2019) and Gururangan et al. (2018) demonstrate that
state-of-the-art NLI systems often rely on heuristics and annotation
artifacts.

Systematic approaches to evaluation propose different sets of stress-tests for
NLI and NLU systems Kovatchev et al. (2018a); Naik et al. (2018); Wallace et
al. (2019); Kovatchev et al. (2019); Ribeiro et al. (2020); Kovatchev et al.
(2020). The attacks can be inspired by linguistic phenomena or empirical use
cases. Systematic evaluations show that NLI and other NLU systems often
underperform on complex linguistic phenomena such as conjunction Saha et al.
(2020), negation Hossain et al. (2020), and coreference Kovatchev et al.
(2022). Researchers also experimented with creating contrastive examples that
differ only slightly from training examples, but have a different label
Glockner et al. (2018); Kaushik et al. (2020); Gardner et al. (2020).
Adversarially created datasets such as Adversarial NLI Nie et al. (2020) and
Dynabench NLI Kiela et al. (2021) demonstrate that there is a lot of room for
improvement regarding NLI datasets and models.

Most of the available resources for NLI research are in English. Conneau et
al. (2018) present XNLI, a multilingual dataset created by translating English
NLI examples into other languages. The interest in multilingual NLI has
resulted in the creation of some novel non-English resources such as the
Korean NLI corpus Ham et al. (2020), Chinese NLI corpus Hu et al. (2020),
Persian NLI corpus Amirkhani et al. (2020), Indonesian NLI corpus Mahendra et
al. (2021), and indigenous languages of the Americas NLI corpus Ebrahimi et
al. (2022). For Spanish, the only available resources are the Spanish portion
of XNLI and the SPARTE corpus for RTE Peñas et al. (2006) which was adapted
from Question Answering data.

##  3 Objectives and Corpus Creation

When creating InferES we experimented with different strategies for obtaining
gold examples. To the best of our knowledge, this is the first time various
annotation strategies are combined and compared in a single NLI corpus. We
adopt three different approaches, used in prior work: our generation strategy
is similar to the original RTE and NLI corpus creation; our rewrite strategy
is inspired from work in generating adversarial and contrastive examples; our
annotation strategy scales well with data and allows us to compare expert- and
crowd-crated datasets. Our aim was to provide interesting and diverse examples
that cover a large range of use cases and linguistic phenomena. We hope that
InferES can be used not only to train automated systems, but also to better
understand the nature of inference. We formulated three main objectives:

  * O1

To create a native NLI dataset for the Spanish language. The existing
resources are either an adaptation from a different task or a translation from
English.

  * O2

To promote better data quality and corpus creation practices. We aim to create
a more challenging dataset and simultaneously reduce the number of annotation
artifacts.

  * O3

To facilitate the research on negation and coreference in the context of NLI.
More specifically, we focus on contrastive and adversarial examples.

###  3.1 Premise Extraction

In the first step of the process, we extracted a set of candidate premises. We
decided to use a single sentence premise, similar to SNLI and MNLI datasets.
We defined two requirements for our premise sentences: 1) that they cover a
range of different topics; and 2) that they be complex enough to entail or
contradict multiple possible hypotheses.

#### Choice of topics

As a source for premises we used the Spanish version of Wikipedia from October
2019. We chose six topics, covering five different domains: history, art,
sports, technology, and politics. We also selected the topics in pairs
hypothesizing that this selection might facilitate the creation of contrastive
examples, specifically in the context of coreference.

  * •

famous historical figures:
Pablo Picasso (ES: Pablo Picasso)
Christopher Columbus (ES: Cristobal Colón)

  * •

types of “games”:
Olympic games (ES: Juegos Olímpicos)
Video games (ES: videojuegos)

  * •

types of multinational “unions”:
The European Union (ES: Unión Europeo)
The Union of Soviet Socialist Republics
(ES: Unión Sovética)

#### Extraction process

We extracted the main Wikipedia article for each topic and preprocessed it
(sentence segmentation and tokenization) using Spacy Honnibal and Montani
(2017). We split the text by paragraphs and discarded paragraphs that
contained only one sentence or more than five sentences. Then, from each
paragraph, we selected a single sentence, prioritizing sentences containing
negation222To check for negation, we used a simple keyword based search, using
a list of the most common negative particles, adverbs, and verbs in Spanish.
The list is available at https://github.com/venelink/inferes where possible,
otherwise selecting a sentence at random. We ensured that each selected
sentence had a length between 15 and 45 tokens.

#### Post-processing

At the end of the extraction process, we had 471 candidate-premise sentences
as follows: 82 for Picasso, 60 for Columbus, 68 for the Olympic games, 73 for
video games, 107 for the EU, and 81 for the USSR. For each sentence, we also
kept the corresponding paragraph to enable experimental setup where we provide
an additional context to the machine learning models at train and test time.
We also used the “context paragraphs” when generating “neutral” pairs. One of
the authors manually inspected all 471 candidate-premise sentences. They
manually resolved problems with sentence segmentation, removed URLs and
internal wikipedia document references, and explicitly resolved any
coreferential and anaphorical ambiguities (i.e.: replaced pronouns and
coreferential entities with an unambiguous referent).

###  3.2 Expert “Generation” Strategy

#### Task formulation

We formulated two separate generation tasks: the generation of entailment
pairs and generation of contradiction pairs. We defined the tasks as follows:

Entailment: Given a premise, write two different sentences that are true.

Contradiction: Given a premise, write two different sentences that are false.
Our guidelines enforced a strict definition of contradiction and required our
generators to write sentences that explicitly contradict the premise, rather
than implicitly rely on event and actor coreference333For a discussion of the
definition of contradiction in the context of NLI, we refer the reader to Gold
et al. (2019).. We asked the generators to provide multiple examples,
requiring the use of different strategies. We further instructed the corpus
generators to: 1) generate hypotheses that have a low lexical overlap with the
premise; 2) generate one affirmative and one negated sentence for each
relation; 3) where possible, replace named entities with pronouns or other
instances of coreference. Our instructions aim to encourage generators to come
up with difficult and diverse examples. We also ensure a high frequency of
entailment pairs containing negation and of affirmative contradiction pairs.
For a reference, the readers can see an example of generated entailment and
contradiction hypotheses in 4.

  * 4.

(premise) En la década de 1980 el soporte habitual para el software era el
cartucho en las videoconsolas, y el disco magnético o la cinta de casete en
los ordenadores.444EN: “In the 1980s, the usual medium for software was the
cartridge in video consoles, and the magnetic disk or cassette tape in
computers.”

(entailment) Es poco probable que en los 1980s las videoconsolas y los
ordenadores utilizaran el mismo soporte.555EN: “It is unlikely that in the
1980s video game consoles and computers used the same medium.”

(contracition) Aunque el cartucho se había utilizado en el pasado, en los 80
ya se consideraba desfasado.666EN: “Although the cartridge had been used in
the past, by the 1980s it was already considered outdated.”

#### Sentence “generators”

Four graduate students of linguistics were trained for this task by the
authors of the paper. They received detailed instructions, examples, and a
two-hour interactive training session prior to the start of the corpus
creation. The students met with the authors of the paper on a weekly basis to
discuss challenging or interesting examples. To further increase the diversity
of the corpus, we recruited 24 undergraduate students for a two-hour
hypothesis generation session preceded by a one-hour interactive training
session. All generators were native speakers of European Spanish.

#### “Generated” portion of the corpus

We distributed the premises extracted in Section 3.1 between the four graduate
students balancing the number of entailment and contradiction pairs per topic
and per premise. For any given premise, a single expert would generate only
one of the relations, never both. Some premises were used more than once. Our
selection strategy aimed to create maximum diversity in the data and reduce
the potential bias from a relatively small number of data generators. The four
graduate students created 2,284 pairs from the original 471 premises. The 24
undergraduate students generated further 872 pairs. The final corpus from the
generation strategy contains 3,156 pairs, split equally between entailment and
contradiction. We describe the process of obtaining “neutral” text pairs in
Section 3.5.

Figure 1: Pipeline for the annotation corpus creation strategy.

###  3.3 Expert “Rewrite” Strategy

#### Task formulation

The rewrite strategy is based on the pairs from the generation strategy. We
defined the task as follows:

Given an existing premise–hypothesis pair, modify both the premise and the
hypothesis so that:

1\. the resulting sentence has a substantial difference in meaning from
original

2\. where possible, change the negation status of a sentence. That is, an
affirmative sentence would become negated, while a negated sentence would
become affirmative

3\. where possible, replace some words in the original sentences with synonyms
and/or coreferential entities

We further instructed the “rewriters” not to resort only to simple negation.
An example of the rewrite strategy can be seen in 5. and 6.: when rewriting
the premise, our expert replaced “descartaba” (EN:“ruled out”) with “aceptaba”
(EN: “accepted”); when rewriting the hypothesis, they changed “inaceptable”
(EN: “unacceptable”) for “viable” (EN: “feasible”). The resulting adversarial
examples include lexical and morphological negation and are more complex than
the “simple negation” benchmark of Hossain et al. (2020).

  * 5.

(Pr) La reina llamó entonces a Colón, diciéndole que no descartaba totalmente
su plan.777EN: The queen then called Columbus, telling him that she did not
fully rule out his plan.

(Hyp) La reina le hizo saber a Colón que su plan no era del todo
inaceptable.888EN: The queen let Columbus know that his plan was not entirely
unacceptable.

  * 6.

(Pr_rw) La reina llamó entonces a Colón, diciéndole que no aceptaba totalmente
su plan.999EN: The queen then called Columbus, telling him that she did not
fully accept his plan.

(Hyp_rw) La reina le hizo saber a Colón que su plan no era del todo
viable.101010EN: The queen let Columbus know that his plan was not entirely
feasible.

#### “Rewritten” portion of the corpus

The rewrite process was carried out by two graduate students of linguistics.
After rewriting both the premise and the hypothesis, we create three new
combinations involving an original or rewritten hypothesis. In the provided
example, those are the pairs 5.(Pr)–6.(Hyp_rw), 6.(Pr_rw)–5.(Hyp) and
5.(Hyp)–6.(Hyp_rw).111111We do not to use 5.(Pr)–6(Pr_rw) due to sentence
length. Our “rewriters” then annotated the relations between the new pairs (in
the example, the relations are “contradiction”, “neutral”, and “entailment”
respectively). As a souorce, we selected 20 entailment and 20 contradiction
per topic, a total of 240 “generated” pairs. We ensured equal distribution of
the original “generators” and created 720 new adversarial “rewrite” pairs.

###  3.4 Crowd “Annotation” Strategy

#### Task formulation

For the crowd annotation strategy we adopted the three-step approach proposed
by Gold et al. (2019). The authors first semi-automatically generated a large
pool of premise-hypothesis pairs. Then, they applied stratified sub-sampling.
Finally, they recruited crowd workers to annotate the meaning relations
between the texts. Figure 1 illustrates the annotation strategy. We choose
this approach since it’s compatible with our generation and rewrite
strategies. We were interested in comparing and combining expert- and crowd-
created corpora, which, to the best of our knowledge has not been done before
for NLI.

#### Creating a sentence pool

The first step of the process was identical to the generation strategy. We
chose 20 of the original premises, five from Picasso, Columbus, Olympic games,
and video games. We chose premises that contain multiple predicates and would
allow for creativity in generating entailment and contradiction pairs. In
Figure 1, these premises are called “source sentences”. 121212EN: “The PC or
personal computer is also a video game platform, but since its function is not
only to play video games, it is not considered a video game console.”

We recruited 26 undergraduate students of linguistics and provided them with
one-hour interactive training for the task of generating entailment and
contradiction pairs. Each student generated 20 entailment and 20 contradiction
pairs from the same 20 “source sentences”. In Figure 1, these hypotheses are
called “true/false sentences”. 131313EN: (true) “The PC can be used for non-
gaming.”141414EN: (false) “The computer only runs video games and is therefore
a game console.”  At this step, the students generated a “sentence pool” of of
1,040 “true/false sentences”.

#### Pair generation

In the second step we combined the sentences from the “sentence pool” in pairs
using three different selection strategies. The “true-true” strategy combines
two “true” sentences derived from the same premise. The “false-false” strategy
combines two “false” sentences from the same premise. The “true-false”
strategy combines one true and one false sentence derived from the same
premise. Unlike Gold et al. (2019), we do not include a random pairing and do
not downsample “false-false” and “true-false” strategies. We randomly selected
2,000 of the pairs for annotation, ensuring equal distribution of strategies.

#### Pair annotation

In the third step, we asked crowd workers to annotate the textual relation
between pairs. We used the WARP-Text Kovatchev et al. (2018b) annotation
interface for the annotation. Following Gold et al. (2019), we created two
separate binary annotation tasks - one for entailment and one for
contradiction. For entailment, we included each pair twice, changing the order
of P and H to reflect the directional nature of the relation. If a sentence
was annotated as not-entailment and not-contradiction, we marked it as
“neutral”.

We use three annotators for each example. Following prior work Marelli et al.
(2014); Gold et al. (2019), we calculated the agreement as the average % of
annotators that voted for the majority label. We obtained 86.9% agreement for
the “entailment” relation and 85.6% agreement for the “contradiction”
relation. We also calculated the Fleiss’ kappa score, obtaining a “moderate
agreement” of 55.151515The lower kappa is likely due to the label imbalance in
the corpus: only about 22% of the pairs contain entailment and 20% contain
contradiction. Our agreement and label distribution of labels are consistent
with the results reported by Gold et al. (2019) for 10 annotators. Since 56%
of the pairs were labeled “neutral”, we kept all “entailment” and
“contradiction” pairs and randomly downsampled the “neutral” to obtain a
balance between the classes. The annotate portion contains 1,290 pairs.

###  3.5 Generating Neutral Pairs

Using our generation strategy, we created “entailment” and “contradiction”
pairs. Using our rewrite strategy, we created pairs with all three relations.
However the “neutral” class was underrepresented compared to the other two. To
create a balanced dataset for training automated systems, we needed a separate
strategy to introduce more pairs with the “neutral” label. In this subsection,
we describe four different rule-based strategies that we used to generate
“neutral” pairs in an automated manner.

#### Shuffling existing P and H (same topic)

We matched each premise to two random hypotheses, generated for different
premises on the same topic.

#### Matching existing P with random contexts (same topic)

In Section 3.1, we kept a “context” paragraph for each premise that we
extracted. We matched each premise to two random “contexts” on the same topic.
We then randomly selected a sentence from each of those contexts.

#### Matching existing H with random contexts (same topic)

Similar to the previous strategy, we randomly matched each hypothesis to a
sentence from a “context” paragraph on the same topic.

#### Shuffling existing P and H (different topics)

Typically, the premise and the hypothesis have at least some degree of
semantic similarity. We argue that an automated NLI solution should be able to
label unrelated pairs. We created a small fixed number of unrelated pairs by
matching texts and hypotheses from different topics.

#### Validating neutral pairs

We selected 240 pairs, 60 from each of the four strategies, to manually
validate the quality of the “neutral” pairs. One of our “sentence generators”
performed a two-stage annotation on each pair. At the first stage they
annotated “whether the premise and hypothesis are semantically releated”. At
the second stage they annotated “whether the meaning relation is neutral,
despite a potential semantic relatedness”. 55% of the “neutral” pairs had some
semantic relation (e.g., shared topic or named entities), and 26% had a strong
relation. 237 out of the 240 examples (98.75%) were annotated as “neutral”.
Two pairs were found to be “entailment” and one - “contradiction”.

#### The “neutral” portion of the corpus

Through generation and downsampling, we obtain a total of 1,291 “neutral”
sentences for InferES . We use 298 of them to re-balance the rewrite portion
of the corpus and the remaining 1,893 to complete the generation portion of
the corpus.

##  4 InferES

We combined the examples from all corpus creation strategies to create InferES
\- a corpus of NLI for Spanish containing 8,055 text pairs. Table 1 shows the
distribution of pairs and labels based on the creation strategy. Note that for
generation and rewrite strategies, the “neutral” examples were at least in
part generated automatically to ensure label balance. For “annotate” the
“neutral” pairs are naturally occurring. More than half of the corpus, 5,029
text pairs, was created using the generate strategy. This is the core part of
the corpus, in which we have incorporated multiple strategies for ensuring the
quality and the linguistic diversity of the examples. 1,716 pairs were created
using the rewrite strategy and 1,290 pairs were generated using the annotate
strategy. All six topics are represented roughly equally. In the generate and
rewrite portions, we aimed to ensure that each original premise has the same
number of hypotheses, distributed equally across relations.

Strategy | Pairs | Ent | Cnt | Neu
---|---|---|---|---
Full | 8,055 | 2,399 | 2,687 | 2,969
Generate | 5,029* | 1,574 | 1,582 | 1,893*
Rewrite | 1,716* | 398 | 712 | 606*
Annotate | 1,290 | 427 | 393 | 470
Table 1: Distribution of labels InferES by strategy

We measured the vocabulary size and the lexical overlap between the premise
and hypothesis. The full InferES has a vocabulary size of 12,877 unique types.
On average, 22.6% of the tokens from the premise also appear in the
hypothesis. 33.4% of the tokens from the hypothesis also appear in the
premise. The two numbers differ since we count the number of non-unique
tokens, including repetition, and we normalize them using a different
denominator (length of premise/hypothesis). The overlap is comparable with
prior work for English (20% and 38% for MNLI and 18% and 34% for linguist-in-
the-loop NLI).

##  5 Machine Learning Experiments

To demonstrate the utility of InferES , we carried out a set of machine
learning experiments. The design of InferES allows us to test NLI models under
a variety of conditions: standard train/test split, hypothesis-only condition,
performance on negation-based adversarial examples, and performance by topic
in- and out-of-distribution.

#### Machine learning models

We used two transformer based models, pretrained for Spanish: the multilingual
version of BERT Devlin et al. (2019) and the Spanish version of BERT, BETO
Cañete et al. (2020). We used the version of the models available on
HuggingFace Wolf et al. (2020) as of May 2022 and finetuned them on InferES .
After experimenting with different hyperparameter settings, we empirically
found the best performance using a PolinomialDecay learning rate scheduler and
training the model for five epochs. We kept the rest of the hyperparameters at
their default values and used ADAM optimizer161616The code for the experiments
and all hyperparameters are available at https://github.com/venelink/inferes.
All reported results are the average of five different random initializations.

Condition | MB | mBERT | BETO
---|---|---|---
Full Dataset | 36.8 | 69.6 | 72.8
Hyp. Only | 36.8 | 38.8 | 42.3
Adv. Negation | 41.5 | 52 | 51.2
Table 2: Performance of multilingual-BERT and Spanish BERT (BETO) on InferES
across different conditions. MB: “majority baseline”. ‘Full”: standard
train/test split.“Hyp. Only”: hypothesis-only. “Adv. Negation”: negation-based
adversarial examples.

#### Full corpus performance

In a standard “full corpus” condition, we used 80% of InferES for training,
10% for validation, and 10% for testing. We use the examples generated by all
three strategies for both training and testing. As shown in Table 2, BETO
obtained 72.8% accuracy and outperformed mBERT (69.6%). While both models
reached a fair performance on the test set, the results were much lower than
the “super-human” performance that state-of-the-art transformer models obtain
on popular benchmarks for English. For a reference, the official performance
for the Spanish portion of XNLI is 82% for BETO-cased, and 78.5% for
mBERT171717See https://github.com/dccuchile/beto.

#### Hypothesis-only performance

In the “hypothesis-only” condition, the models do not have access to the
“premise” during training or testing. NLI explores the meaning relation
between the two texts - the premise and the hypothesis. If a model is exposed
only to one of the texts, its performance should not exceed that of a random
guess, roughly equal to predicting the most common class.

Prior work has shown that existing datasets for English contain a large number
of “annotation artifacts” and models are able to obtain much higher
performance than chance. Poliak et al. (2018) report 55% accuracy for the
“hypothesis-only” condition on the MNLI corpus using non-transformer models,
an increase of 20% over the majority baseline. Parrish et al. (2021) show that
their “linguist-in-the-loop” approach is less biased in the hypothesis-only
condition, but they also report accuracy over 50% on the full dataset using a
ROBERTA transformer.

In the “hypothesis-only” condition on InferES , BETO obtained 42.3%, and mBERT
– 38.8%, which is respectively 5.5% and 2% higher than the majority baseline.
The relatively small improvement over the majority baseline indicates that the
hypothesis-only artifacts in InferES are substantially fewer than in previous
work.

#### Performance on negation-based adversarial examples

For this experiment, we trained the models on the generation and annotation
portion of the corpus and evaluated them on rewrite. The setup is similar to
the one from Hossain et al. (2020). The performance of mBERT and BETO drops
significantly when facing adversarial examples (See Table 2). The models
obtain 52% and 51.1% accuracy, about 10% higher than the majority baseline.
Hossain et al. (2020) report that on two of the three datasets they use, BERT
performs worse than the majority baseline. Our experimental setup is arguably
more difficult, since InferES contains multiple negation strategies rather
than just negating the main verb. The rewrite portion of the corpus can
provide further insight into the use of negation in NLI and we hope it would
facilitate further research and improvement in the area.

#### In- and Out-of-distribution generalization by topic

We also carried out a set of experiments to determine the ability of models to
generalize across the six different topics. We evaluated the models in two
different conditions.

Top | mBERT | BETO
---|---|---
| IND | OOD | ID | OOD
All | 69.6 | 72.8
$1$ | $73.1\pm 4$ | $67.9\pm 2$ | $78.5\pm 4$ | $73.8\pm.6$
$2$ | $68.1\pm 4$ | $69.9\pm 1$ | $74.6\pm 3$ | $72.0\pm.7$
$3$ | $70.8\pm 3$ | $70.9\pm 1$ | $74.2\pm 3$ | $73.3\pm.7$
$4$ | $71.6\pm 2$ | $68.9\pm.7$ | $69.4\pm 5$ | $69.7\pm.7$
$5$ | $77.1\pm 3$ | $76.1\pm.5$ | $78.3\pm 4$ | $77.2\pm.9$
$6$ | $65.3\pm 4$ | $69.8\pm.9$ | $68.6\pm 5$ | $69.6\pm.9$
Table 3: In-distribution (ID) and Out-of-distribution (OOD) performance of
mBERT and BETO on different topics within InferES . ID: model trained on data
covering all 6 topics. OOD: model trained on 5 topics and evaluated on the
unseen 6th. 1 (Picasso), 2 (Columbus), 3 (Olympics), 4 (Videogames), 5 (EU), 6
(USSR).

The in-distribution (ID) condition is an extension of the “full corpus”
condition. We split the full corpus containing all six topics in an 80/10/10
ratio. Then, when evaluating the performance of the models, we split the test
set in six sub-sets, based on their topic and we measured the model accuracy
on each sub-set. To ensure that the variation of the model performance is not
due to a sampling bias, we re-trained each model five times, using a different
80/10/10 random split each time. In Table 3, we report the average accuracy
across the five different splits. Both models obtained the highest ID
performance on the topics of “Picasso” and “The European Union” and the lowest
ID performance on “The USSR”.

For the out-of-distribution (OOD) condition, we designed a transfer learning
experiment, in which we trained mBERT and BETO on five of the topics and
evaluate on the sixth. The results presented in Table 3 demonstrate that the
models are able to generalize well across the topics, even in a transfer
learning setup. For both models, the OOD performance on most topics drops
between 1% and 5% compared to ID. The performance on “Olympics” for mBERT and
on “videogames” for BETO was almost identical between conditions. For “The
USSR”, both models obtained higher performance for OOD. We inspected the
matter further and noticed that due to the corpus size, the ID random split is
not very stable (the ID test set only contains between 100 and 120 instances
of each topic) and the average can be affected by outliers. The OOD results
are much more stable due to the test set having over 1,000 examples per topic.
This can be seen in the difference in standard deviation: between 2% and 5%
for ID, and below 1% for OOD. Our experiments demonstrate that the models are
able to generalize well even to topics they have never seen during training, a
promising finding for the overall generalizability of NLI models.

##  6 Discussion

In Section 3, we formulated three main objectives behind InferES . In this
section, we want to revisit those objectives and briefly discuss the
importance of our work for the NLP and NLI communities.

Our first objective was “To create a native NLI dataset for the Spanish
language”. To the best of our knowledge, InferES is the first native NLI
dataset for Spanish which is not adapted or translated. We described the
creation process and validated that it can be used to train different machine
learning models. We have successfully contributed a new resource to the
Spanish NLP community and we hope that InferES can facilitate the further
creation of tools and resources for that language.

Our second objective was “To promote better data quality and corpus creation
practices.”. We proposed, implemented, analyzed, and compared several
different strategies for creating text pairs. The resulting dataset proves
non-trivial to state-of-the-art NLP models with an overall accuracy in the low
70s. This leaves a lot of room for improvement and future research. The
results using a “hypothesis-only” baseline indicate that InferES contains
fewer annotation artifacts than prior work. At the same time, models trained
on the dataset are able to generalize well across different topics, even in
our “out-of-distribution“ condition. Overall, we can conclude that InferES is
of high quality and achieves the objective of promoting better data by design.

Our third objective was “To facilitate the research on negation and
coreference in the context of NLI.”. Our rewrite strategy was focused on
creating naturally occurring contrastive and adversarial examples based on
negation and coreference. We followed prior work on evaluating systems’
performance and demonstrated that those examples are non-trivial to solve.
However, the two models are still able to outperform the majority baseline by
over 10%. These findings indicate that the problem is not unsolvable and the
models are learning something about complex negation from the data. InferES
can facilitate the resaerch of negation in Spanish both in the context of NLI
and in isolation. We leave quantifying the importance of coreference in the
rewrite section for future work.

Overall, we have achieved all our objectives: 1) we created a novel dataset
for Spanish; 2) we used different generation and annotation strategies to
obtain a challenging corpus with fewerr annotation artifacts; 3) we created a
set of high-quality contrastive and adversarial examples based on negation and
coreference. We believe that InferES is an important contribution to Spanish
NLP, and also to researchers interested in NLI, negation, and coreference. We
hope that this dataset can be used to train and evaluate more accurate
automated systems, but also to better understand the nature of those
linguistic phenomena.

##  7 Conclusions

We presented InferES \- a new corpus of Natural Language Inference for
Spanish. To the best of our knowledge, this is the first original Spanish NLI
corpus that is not a translation or an adaptation of an existing dataset. We
explored several different strategies for corpus creation and put the emphasis
on creating diverse and non-trivial examples, that are also linguistically
interesting. More specifically, we created contrastive and adversarial
examples involving complex negation and coreference.

We provided two baseline transformer-based systems finetuned on the dataset.
We demonstrated that InferES is challenging and contains fewer anotation
artifacts than prior work. We also evaluated the performance of automated
systems on adversarial examples and the ability of the models to generalize
across topics in- and out-of-distribution. The results validated the quality
and the difficulty of the corpus. InferES leaves a room for analysis and
improvement.

Our work opens several directions for future work: studying and improving the
performance of NLI models for Spanish; expanding the research on negation in
Spanish, and specifically the complex and lexical negation; evaluating the
importance of coreference in the context of NLI. We believe that InferES will
be useful both to Spanish researchers and to the general NLP community.

## Acknowledgements

We want to thank M. Antónia Martí and Montse Nofre for their suggestions and
support for this work. We are grateful to Patricia Grau Francitorra, Eugenia
Verjovodkina, Víctor Bargiela Zotes, and Xavier Bonet Casals for the
annotations and the discussions about the annotation process. We also want to
thank the anonymous reviewers for their feedback and suggestions.

This work was partially funded by the project “FairTransNLP-Language:
Analysing Toxicity and Stereotypes in Language for Unbiased, Fair and
Transparent Systems (PID2021-124361OB-C33)” funded by Ministerio de Ciencia e
Innovación (Spain) and co-funded by the European Regional Development Fund
(FEDER). Most of this work was carried out during the first author’s APIF
grant at the University of Barcelona.

## References

  * Amirkhani et al. (2020) Hossein Amirkhani, Mohammad AzariJafari, Zohreh Pourjafari, Soroush Faridan-Jahromi, Zeinab Kouhkan, and Azadeh Amirak. 2020.  Farstail: A persian natural language inference dataset.
  * Bar Haim et al. (2006) Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006.  The second pascal recognising textual entailment challenge.
  * Bentivogli et al. (2009) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009.  The Fifth PASCAL Recognizing Textual Entailment Challenge.  In _TAC_.
  * Bentivogli et al. (2010) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2010.  The Sixth PASCAL Recognizing Textual Entailment Challenge.  In _TAC_.
  * Bentivogli et al. (2011) Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2011.  The Seventh PASCAL Recognizing Textual Entailment Challenge.  In _TAC_.
  * Bowman et al. (2015) Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642.
  * Cañete et al. (2020) José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, and Jorge Pérez. 2020.  Spanish pre-trained bert model and evaluation data.  In _PML4DC at ICLR 2020_.
  * Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  XNLI: Evaluating cross-lingual sentence representations.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2475–2485, Brussels, Belgium. Association for Computational Linguistics.
  * Dagan et al. (2006) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006.  The PASCAL recognising textual entailment challenge.  In _Machine learning challenges. evaluating predictive uncertainty, visual object classification, and recognising tectual entailment_ , pages 177–190. Springer.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ebrahimi et al. (2022) Abteen Ebrahimi, Manuel Mager, Arturo Oncevay, Vishrav Chaudhary, Luis Chiruzzo, Angela Fan, John Ortega, Ricardo Ramos, Annette Rios, Ivan Vladimir Meza Ruiz, Gustavo Giménez-Lugo, Elisabeth Mager, Graham Neubig, Alexis Palmer, Rolando Coto-Solano, Thang Vu, and Katharina Kann. 2022.  AmericasNLI: Evaluating zero-shot natural language understanding of pretrained multilingual models in truly low-resource languages.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6279–6299, Dublin, Ireland. Association for Computational Linguistics.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating models’ local decision boundaries via contrast sets.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1307–1323, Online. Association for Computational Linguistics.
  * Giampiccolo et al. (2008) Danilo Giampiccolo, Hoa Trang Dang, Bernardo Magnini, Ido Dagan, Elena Cabrio, and Bill Dolan. 2008.  The Fourth PASCAL Recognizing Textual Entailment Challenge.  In _TAC_.
  * Giampiccolo et al. (2007) Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007.  The third pascal recognizing textual entailment challenge.  In _Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing_ , pages 1–9. Association for Computational Linguistics.
  * Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018.  Breaking NLI systems with sentences that require simple lexical inferences.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 650–655, Melbourne, Australia. Association for Computational Linguistics.
  * Gold et al. (2019) Darina Gold, Venelin Kovatchev, and Torsten Zesch. 2019.  Annotating and analyzing the interactions between meaning relations.  In _Proceedings of the 13th Linguistic Annotation Workshop_ , pages 26–36, Florence, Italy. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ham et al. (2020) Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020.  KorNLI and KorSTS: New benchmark datasets for Korean natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 422–430, Online. Association for Computational Linguistics.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Hossain et al. (2020) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020.  An analysis of natural language inference benchmarks through the lens of negation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118, Online. Association for Computational Linguistics.
  * Hu et al. (2020) Hai Hu, Kyle Richardson, Liang Xu, Lu Li, Sandra Kübler, and Lawrence Moss. 2020\.  OCNLI: Original Chinese Natural Language Inference.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3512–3526, Online. Association for Computational Linguistics.
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020.  Learning the difference that makes a difference with counterfactually-augmented data.  In _International Conference on Learning Representations_.
  * Kiela et al. (2021) Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021.  Dynabench: Rethinking benchmarking in NLP.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4110–4124, Online. Association for Computational Linguistics.
  * Kovatchev et al. (2022) Venelin Kovatchev, Trina Chatterjee, Venkata S Govindarajan, Jifan Chen, Eunsol Choi, Gabriella Chronis, Anubrata Das, Katrin Erk, Matthew Lease, Junyi Jessy Li, Yating Wu, and Kyle Mahowald. 2022.  longhorns at DADC 2022: How many linguists does it take to fool a question answering model? a systematic approach to adversarial attacks. In _Proceedings of the First Workshop on Dynamic Adversarial Data Collection_ , pages 41–52, Seattle, WA. Association for Computational Linguistics.
  * Kovatchev et al. (2020) Venelin Kovatchev, Darina Gold, M. Antonia Marti, Maria Salamo, and Torsten Zesch. 2020.  Decomposing and comparing meaning relations: Paraphrasing, textual entailment, contradiction, and specificity.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 5782–5791, Marseille, France. European Language Resources Association.
  * Kovatchev et al. (2018a) Venelin Kovatchev, M. Antònia Martí, and Maria Salamó. 2018a.  ETPC - a paraphrase identification corpus annotated with extended paraphrase typology and negation.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Kovatchev et al. (2018b) Venelin Kovatchev, M. Antònia Martí, and Maria Salamó. 2018b.  WARP-text: a web-based tool for annotating relationships between pairs of texts.  In _Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations_ , pages 132–136, Santa Fe, New Mexico. Association for Computational Linguistics.
  * Kovatchev et al. (2019) Venelin Kovatchev, M. Antonia Marti, Maria Salamo, and Javier Beltran. 2019.  A qualitative evaluation framework for paraphrase identification.  In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)_ , pages 568–577, Varna, Bulgaria. INCOMA Ltd.
  * Mahendra et al. (2021) Rahmad Mahendra, Alham Fikri Aji, Samuel Louvan, Fahrurrozi Rahman, and Clara Vania. 2021.  IndoNLI: A natural language inference dataset for Indonesian.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10511–10527, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Naik et al. (2018) Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018.  Stress test evaluation for natural language inference.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2340–2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Nie et al. (2020) Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020.  Adversarial NLI: A new benchmark for natural language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4885–4901, Online. Association for Computational Linguistics.
  * Parrish et al. (2021) Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alexia Warstadt, Karmanya Aggarwal, Emily Allaway, Tal Linzen, and Samuel R. Bowman. 2021\.  Does putting a linguist in the loop improve NLU data collection? In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4886–4901, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Peñas et al. (2006) Anselmo Peñas, Álvaro Rodrigo, and Felisa Verdejo. 2006.  Sparte, a test suite for recognising textual entailment in spanish.  In _Computational Linguistics and Intelligent Text Processing_ , pages 275–286, Berlin, Heidelberg. Springer Berlin Heidelberg.
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.  Hypothesis only baselines in natural language inference.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Saha et al. (2020) Swarnadeep Saha, Yixin Nie, and Mohit Bansal. 2020.  ConjNLI: Natural language inference over conjunctive sentences.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8240–8252, Online. Association for Computational Linguistics.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2153–2162, Hong Kong, China. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.


# Noise Reduction in Underexposed Radiographic and Tomographic Multi-Channel
Images with Self-Supervised Deep Learning

Yaroslav Zharov1,5,*,$\dagger$ Evelina Ametova1,2,$\dagger$ Tilo Baumbach1,3
Genoveva Burca6,4,2  Vincent Heuveline5  1Laboratory for Applications of
Synchrotron Radiation (LAS), Karlsruhe Institute of Technology, Germany
2Department of Mathematics, The University of Manchester, United Kingdom
3Institute for Photon Science and Synchrotron Radiation (IPS), Karlsruhe
Institute of Technology, Germany
4ISIS Pulsed Neutron and Muon Source, STFC, UKRI, Rutherford Appleton
Laboratory, UK
5Engineering Mathematics and Computing Lab (EMCL), Interdisciplinary Center
for Scientific Computing (IWR), Heidelberg University, Germany,
6Diamond Light Source, Harwell Campus, Didcot, Oxfordshire, OX11 0QX, UK
$\dagger$Contributed equally<EMAIL_ADDRESS>
###### Abstract

Noise reduction is an important issue for all radiographic or tomographic
imaging techniques. It becomes particularly critical in applications where
additional constraints force a strong reduction of the signal-to-noise ratio
per image. These constraints may result from limitations on the maximum
allowable dose and the associated restriction on radiation flux or exposure
time. Often, however, a reduction of the SNR per image is accepted in order to
be able to take more images of the same object, depending on further
parameters and aiming to gain additional information. In these cases, the
total available signal capacity is distributed over a larger number of input
channels. These can be, for example, time channels in case of cine-radiography
or cine-tomography - or energy channels in the case of spectroscopic, i.e.,
energy dispersive imaging. Conventional image denoising methods work on an
image-per-image basis and rely on certain assumptions about image properties.
Consequently, they perform well when the assumptions are met and fail
otherwise. At the same time, tremendous progress in machine learning
demonstrated that data-driven methods are much more flexible in accommodating
various image characteristics. In this paper we demonstrate a method to
improve the quality of multi-channel (time or energy-resolved) datasets. The
method relies on the recent _Noise2Noise_ (N2N) self-supervised denoising
approach that learns to predict a noise-free signal without access to noise-
free data. N2N in turn requires drawing pairs of samples from a data
distribution sharing identical signal while being exposed to different samples
of random noise. As multi-channel images naturally share structural
information and provide multiple noisy observations, the method is applicable
with minor relaxations of the original constraints to accommodate data-
specific features. We demonstrate several representative case studies, namely
spectroscopic (k-edge) X-ray tomography, in-vivo X-ray radiography, and
energy-dispersive (Bragg edge) neutron tomography. In all cases, the N2N
method shows drastic improvement and outperforms conventional denoising
methods. Hence, the method can be reliably used to shorten exposure times,
thus dose, or improve the throughput of real-world imaging studies.

††journal: oe††articletype: Research Article

##  1 Introduction

Many imaging modalities rely on the penetration ability of radiation to
visualize the interior of an object. Depending on the employed physical
interaction, among others, absorption, scattering or phase-shift can be used
to obtain contrast inside the object of interest. Either _radiography_ (single
view) or _tomography_ (multiple views with subsequent volumetric image
reconstruction) are commonly acquired. As particle emission is a stochastic
process and a detector has finite detection efficiency, longer exposure time
improves signal quality. However, there are many scenarios where sufficient
exposure can not be acquired. A well-known example is _in vivo_ imaging. Here,
the radiation dose ultimately limits the amount of information acquired
[Moosmann2013]. Another example is spectroscopic imaging with a polychromatic
beam where the acquired particle counts are shared across multiple energy bins
leading to a significant increase of exposure times, hence, limiting the
experiment throughput [warr2021enhanced]. Both imaging modes can be
generalized as _multi-channel_ in the sense that multiple images share similar
structural information.

Given the aforementioned physical constraints, we need to rely on image
processing techniques to improve image quality and extract valuable data. A
group of the methods used to improve image quality affected by noise is called
_denoising_. Similar to other domains, methods based on _Machine Learning_
(ML) have revolutionized denoising. In this paper, we demonstrate an ML
approach to improve the quality of underexposed images in challenging
applications such as spectroscopic (k-edge) X-ray tomography, in-vivo X-ray
radiography, and energy-dispersive (Bragg edge) neutron tomography. The method
is based on the recent Noise2Noise (N2N) self-supervised denoising approach
[Lehtinen2018]. The N2N main assumption enabling the N2N method is formulated
as follows. Consider two images $I_{1}$ and $I_{2}$ that share the same
structural information $S$ but are affected by the independent, identically
distributed (iid) instances of noise $\sigma_{1}$ and $\sigma_{2}$. If the
model is trained to predict $I_{1}$, given $I_{2}$ as input, the best
prediction possible is $S$, because the $\sigma_{2}$ is conditionally
independent of $\sigma_{1}$, given $S$.

This paper is organized as follows. First, we outline the related work to put
our work in context. Then, the N2N method details are followed by three
rigorous case studies. Finally, a discussion of our findings in the context of
multi-channel imaging is provided.

##  2 Related Work

Artifacts are inherent to digitally acquired images, as an acquisition
function is subject to many uncertainties and in general, is not accurately
known. The acquisition function includes optical distortion, lens and detector
array heterogeneity, and inherent noise driven by the stochastic nature of
photons. As optical distortion is a misplacement of information, a distortion
map can be estimated and applied to compensate for it. For the lens and
detector matrix heterogeneity, flat and dark field corrections are used. Flat
and dark fields refer to the measurement of detector response with and without
source illumination, respectively. Finally, denoising is used to compensate
for the inherent stochastic noise. Hence, the denoising problem is to restore
signal $S$ from a noisy observation $I$ [Gonzalez2008]:

$I=S+\sigma(S)$ (1)

where $\sigma(S)$ is the inherent noise of the imaging device. The noise
depends on photon stochasticity, electronics and setup distortion, and also
the transformation applied to correct the image. Which makes closed-form
distribution estimation problematic.

The existing image denoising approaches can be roughly categorized into 2
large groups: classical image processing and ML approaches. Typically,
classical image processing approaches work on a per-image basis and
incorporate expert beliefs about the noise nature. ML approaches in contrast
employ the idea of fitting a data-driven model with none to minimal expert
knowledge about the nature of the data.

###  2.1 Classical Image Processing

The basic spatial filtering methods are mean, median, or Gaussian kernel
filters [Gonzalez2008]. For each pixel, these filters select a new value,
based on the weighted values of the neighboring pixels. These filters are
fast, robust, well-understood, and work fairly well in many situations. The
main drawback of these classical filters is their tendency to not only remove
the bright noise outliers but also blur the sharp edges.

More advanced spatial filtering approaches, _e.g._ , non-local means (NLM),
use more information from the whole image [Buades2005]. Instead of taking an
average of the direct neighbourhood of the pixel, NLM takes an average of the
large region, weighted by the similarity between the “donor” and the
“recipient” pixel. The process of revisiting multiple locations in the image,
comparing their surroundings, and computing the average can take minutes for
one image. In return, this method is capable of producing sharper denoised
images [Fan2019].

Alternatively, denoising can be formulated as an optimization problem and
regularization can be used to incorporate some prior knowledge about the image
properties [gu2019brief]. These methods are very powerful but require deep
mathematical knowledge and handcrafted regularizers in many cases, making
their application for experimental data challenging. One of the most
successful regularizers is Total Variation (TV) (and its derivatives) which
encourages piece-wise constant image regions separated with sharp boundaries
[rodriguez2013total].

To summarize, classical methods require fine parameter tuning by an expert to
balance smoothing and denoising, hence there is a significant risk of
information loss if applied incorrectly. A more comprehensive overview of the
classical denoising methods can be found elsewhere [Fan2019].

###  2.2 Machine Learning Approaches

The evolution of the classical methods may be seen as a series of steps taken
to increase the amount of information used to correct a single pixel value. In
this respect, ML-based approaches appear as a natural further step: a _model_
, trained to correct the noise, implicitly incorporates knowledge about the
whole dataset.

Early ML-based image denoising approaches worked in a supervised manner,
_i.e._ a model was trained on a set of noisy images to predict a noise-free
image (_target_). Recently, authors of the N2N method demonstrated that there
is no need for a noise-free target: if one uses a pair of noisy images
(affected by iid instances of the noise) as an input and as a target for the
training, the model will predict the noise-free image [Lehtinen2018]. The
underlying intuition is that independent instances of noise are uncorrelated
and cannot be predicted, hence the model is forced to extract features. Even
though N2N does not explicitly require a set of noise-free images, the authors
of [Lehtinen2018] synthetically formed noisy pairs by adding noise to noise-
free images.

There have been several attempts to extend the N2N method for denoising
problems where pairs of images are not naturally available. Noise2Self
[Batson2019] and Noise2Void [Krull2019] generate the required pair of images
by taking random pixels in the noisy image and disturbing them with yet
another noise distribution. In this way, multiple training pairs can be
constructed from a single noisy image. Noise2Stack [Papkov2021] was designed
for three-dimensional tomographic data and is based on an assumption that
tomographic data is typically smooth. Therefore, slice-to-slice changes are
assumed to be significantly smaller than the slice-wise variability caused by
noise, hence, neighbouring slices can be used for training.

Alternatively, constrained autoencoders can be used to denoise images
[Vincent2008]. During the training, autoencoders use the same image as an
input and as a target and attempt to compress (encode) the input image into
its lower-dimensional representation. The denoising properties of this
approach rely on the assumption that the noise, due to its stochastic nature,
is harder to encode, than the signal. To additionally limit the capacity of
the model to store information about the noise, it can be restricted by
limiting the computational capacity of the model, lowering the dimensionality
of the learned representation, or introducing synthetic noise into it
[Vincent2008]. However, the autoencoders are inefficient if the noise is
spatially correlated and can be easily memorized by the model. The
Hierarchical DivNoising (HDN) method addresses this issue by training a
variational autoencoder with a noise model imposed over output [Prakash2022].
The authors proposed a way to find particular components of the model that
encode information about the noise so that they can remove those components.
Even though the proposed methods provide a valuable alternative to the N2N
approach, the authors highlight that the N2N approach is a hard-to-beat
baseline [Prakash2022].

##  3 Model Training

As previously discussed, the N2N method assumes that a pair of images contains
the same signal and iid noise. Our adaptation of the method to multi-channel
data takes its inspiration from the denoising of Synthetic Aperture Radar
(SAR) images [Dalsasso2022]. In SAR imaging, both phase and amplitude of
received microwaves are measured in each pixel; commonly the phase information
is ignored. However, the authors demonstrated that the amplitude and the phase
contain complimentary information, hence, can be used as a basis for N2N
denoising. We hypothesize that in multi-channel imaging, adjacent time frames
or energy levels, indeed share sufficiently similar signals, and have noise
samples close to being iid. Therefore, we generate the required pair images
based on this hypothesis. To help the model catch complex spatial structures
of the signal, we also feed it with several adjacent energies or time frames
as input, whenever it is possible.

Following [Krull2019, Batson2019, Lehtinen2018], we use the fully-
convolutional neural networks as a model architecture. We employ U-Net with
ResNet-50 as the backbone and rely on the Adam optimizer with a $3e-4$
learning rate, without scheduling. We augment each pair of image pairs with
random crops, shifts, scale, rotations, distortions, and different types of
blur. We acknowledge, that there is room for quality improvement via larger
models, modern architectures, better optimization procedures, or more
aggressive augmentations. The sensitivity study of training parameters is a
topic of future investigation.

##  4 Results

###  4.1 Simulated spectral X-ray tomography

There is a growing interest in energy-dispersive X-ray tomography, especially
for biomedical imaging [warr2021enhanced]. For a long time, polychromatic
emission of laboratory X-ray sources has been considered a drawback in lab-
based imaging. In conventional absorption mode, each detector pixel integrates
all photons irrespective of their energy. As attenuation is a function of
photon energy, conventional (linear) tomographic reconstruction might exhibit
so-called beam-hardening artifacts [davis2008modelling]. However, combined
with an energy-dispersive X-ray detector, such an acquisition mode allows to
segment materials of similar electron density inseparable otherwise. As the
X-ray spectrum shows sharp discontinuities at energies equal to the binding
energies of the core-electron states, so-called _k-edges_ , the spectrum in
each reconstructed voxel can be used to identify the corresponding material.
State-of-the-art energy-dispersive X-ray detectors have energy resolution
around 1 keV [egan20153d], allowing distinguishing even neighbouring chemical
elements. However, high spectral resolution comes at a cost of long exposures
as acquired counts are shared across multiple bins. Therefore, a state-of-the-
art reliable denoising approach might help to improve experiment throughput.
In this study, to ensure strictly controlled conditions, we simulated the
tomographic acquisition.

####  4.1.1 Data

We generated a volumetric phantom by combining several three-dimensional point
clouds: two Swiss rolls, two moon crescents, and an s-curve. All point clouds
were generated by the Scikit-Learn library [Pedregosa2012]; for 2D point
clouds, the last axis was added by randomly sampling from the Uniform
distribution. To convert the point clouds to a raster volume, we selected a
priority order (which value takes place, if there are several objects in one
voxel) and size (in voxels) of each point in the point cloud. The spatial size
of the phantom was set to $512\times 512\times 512$. Structurally, all slices
of the dataset are the same. However, surface texture varied because of the
random nature of the point clouds. A single slice is shown in Figure 3(a)
(left).

We assigned the simulated objects with energy-dependent mass attenuation
coefficient (MAC) of Europium($\prescript{}{63}{\mathbf{Eu}}$, k-edge = 48.5
keV), Gadolinium ($\prescript{}{64}{\mathbf{Ga}}$, k-edge = 50.2 keV),
Ytterbium ($\prescript{}{70}{\mathbf{Yb}}$, k-edge = 61.3 keV), Lutetium
($\prescript{}{71}{\mathbf{Lu}}$, k-edge = 63.3 keV), and Uranium
($\prescript{}{92}{\mathbf{U}}$, k-edge = 115.6 keV). The background was
assigned with MAC of air. This particular choice of materials was inspired by
the study of the separability of k-edge nanoparticles presented in
[getzin2018increased]. Two pairs of materials have neighbouring atomic
numbers, hence very close k-edges, and are barely distinguishable in a noisy
image; Uranium was added to have a k-edge in the noisiest part of the
spectrum, to check the ability of the method to locate the k-edge in extreme
noise conditions.

We used MATLAB package PhotonAttenuation to generate the energy-dependent MAC
of the selected materials [tuszynski2006]. A spectrum profile of a
Boone/Fewell source with the tube potential 150 kV (no kV Ripple and filters)
was generated using the MATLAB package spektr 3.0 [punnoose2016spektr]. The
obtained source spectrum was normalized and scaled to have a maximum value of
$175\cdot 1e3$ photons $/\mathrm{mm}^{2}$ to imitate short exposure
acquisition. MAC of selected materials and the source spectrum are shown in
Figure 1

We generated 135 energy bins between 15 and 150 keV with a 1 keV step. For
each bin, we simulated 120 equally-spaced parallel-beam CT projections over
180°. The spectral characteristics of the material and the source are shown in
Figure 1. We used the conventional FBP algorithm to reconstruct tomographic
data (implemented in in [jorgensen2021core]). Examples of reconstructed slices
for 40 keV (high flux) and 140 keV (low flux) are shown in Figure 3(b)
(right). As expected, at 140 keV the reconstructed slice is uninterpretable.

Figure 1:  Energy dependent mass attenuation coefficients (MAC) of the
simulated materials, and simulated source profile.

####  4.1.2 Training and Processing

The model $f_{\theta}$ was trained by optimizing

$\mathbb{E}_{i,j}\|f_{\theta}(x_{i,j-1},x_{i,j+1})-x_{i,j}\|_{1}\xrightarrow[\theta]{}\min,$
(2)

where $x_{i,j}$ is a projection acquired at angle $i$ and in energy bin $j$.
We randomly split the whole set of projection angles into train and validation
sets with a splitting ratio of $80/20$. Note, that the energy level that is
required to be predicted, should never be fed into the model, to avoid the
trivial solution. This forms a gap of one energy level in the inputs.

During the inference, we feed the model with the adjacent energy bins without
the gap used in training, to avoid blur in the spectral domain:

$\tilde{x}_{i,j-0.5}=f_{\theta}(x_{i,j-1},x_{i,j}).$ (3)

However, since the model predicts the energy level which is averaged between
two input levels, it will inevitably predict an energy level between two
adjacent ones used as input. It is important yet easy to compensate for this.

As before, the denoised tomographic datasets were reconstructed with the
conventional FBP algorithm. Here, each energy bin was reconstructed separately
resulting in 135 volumes. To obtain the spatial distribution of individual
materials in the sample, we performed material decomposition as described in
[ametova2021crystalline]. The employed decomposition relies on the assumption
that each voxel is a unit volume and each material occupies a volume fraction
in this unit volume (the fraction can be 0). Under this assumption, a voxel-
wise sum of all material maps is equal to 1 in each voxel.

####  4.1.3 Results

Figures 3(a) and 3(b), show two-dimensional slices for selected (individual)
energy bins. N2N demonstrates drastic improvement in reconstruction quality
and noise suppression. For 40 keV (high source flux, Figure 3(a)) the
reconstructed slice appears to be clean from noise or artifacts; the slice is
sharp and all structures clearly visible. Even though no signal seems to be
invisible in the 140 keV slice (Figure 3(b)) prior to denoising, N2N is able
to partially recover structures in the slice.

Individual spectra reconstructed for one voxel in each of the five materials
are plotted in Figure 3, right, alongside the theoretical MAC. The voxel
locations inside the materials were chosen arbitrarily. Denoising yields sharp
and accurately positioned k-edges, hence, supports further material
decomposition. Even the Uranium k-edge is visible in the denoised spectrum.

(a) Noisy and denoised slices at 40 keV (near peak source flux)

(b) Noisy and denoised slices at 140 keV (low source flux)

(c) Spectral profiles

Figure 3: Qualitative examination of the denoising with the N2N method. We
present both spectral and spatial domains.

To quantitatively evaluate denoising results, we perform material
decomposition. As the sum of volume fractions corresponding to each material,
obtained through material decomposition, is bound to 1 in each voxel, we can
treat the estimated volume fractions as probabilities. Hence, material
decomposition can be considered a classification problem and related quality
assessment metrics can be applied to quantitatively assess the results. The
comparison results are shown in Figures 4(a) and 4(b). In the top row we show
binarized material decomposition error (black corresponds to erroneous
material prediction). The confusion matrices between the predicted and true
materials for each pixel are presented in the bottom row (perfect
classification results in the identity confusion matrix). A high level of
noise in the simulated data causes misclassifications between close materials
(e.g., Lutetium and Ytterbium). Also, as it is visible on the top row, these
errors are distributed evenly throughout the sample. Hypothetically, this can
be compensated by enforcing an assumption of material homogeneity. However,
this assumption might cause severe errors close to material interfaces. The
errors in the denoised volume are mostly concentrated around the borders (see
the top row), and mainly correspond to misclassification for air due to slight
blur (see the bottom row). Overall, the confusion matrix for the denoised
dataset is considerably closer to the identity matrix.

We also present the Area Under Precision-Recall Curve (AUPRC), measured for
each material 1. AUPRC for ideal classification is 1. AUPRC results
additionally highlight improvement after denoising: N2N provides a boost of
more than $10\%$ of mean AUPRC for the downstream material decomposition. To
assess quality loss caused by reconstruction itself (without any effect of
denoising), we generated another set of projections with very high flux (all
other parameters the same). Material decomposition for this volume shows a
mean AUPRC of $0.999$, with the lowest _precision_ of $0.996$ for the air. We
conclude that the reconstruction losses are negligible in this experiment.

(a) Low exposure

(b) Low exposure with denoising

Figure 4: Quantitative comparison of the different imaging conditions via material decomposition. Top row: binarized decomposition error. Bottom row: confusion matrix for different materials; each row is normalized to the total count of ground truth pixels of each class. |  | Air | Eu | Gd | Yb | Lu | U | mean
---|---|---|---|---|---|---|---|---
AUPRC | noisy | 0.999 | 0.917 | 0.873 | 0.787 | 0.645 | 0.998 | 0.870
denoised | 0.999 | 0.998 | 0.998 | 0.996 | 0.995 | 0.999 | 0.998
Table 1: Comparison of the AUPRC for different materials. We compare the
material decomposition before and after denoising.

###  4.2 Neutron imaging

Neutron imaging provides a complementary contrast to conventional X-ray
imaging. Neutrons interact only with atomic nuclei, hence, a neutron beam
passing through an object can capture information about the material
structure. The neutron transmission intensity spectrum of a polychromatic
thermalized neutron beam contains sudden and sharp increase at wavelengths
equal twice the interplanar distance between scattering planes as a result of
the crystalline properties of the sample material [fundamentals1993nuclear].
Energy dispersive images can be acquired by combining a pulsed neutron
spallation source and a suitable time-sensitive detector by using the time-of-
flight (ToF) method. As more energetic particles have higher velocities, they
reach the detector faster. Measuring the time of arrival of a neutron at the
detector and knowing its flight path, its velocity and therefore its
wavelength (energy) can be determined. High energy resolution requires long
flight distances and many time bins in the detector. Hence, only a few pulses
per second can be measured, and acquired counts are shared between multiple
bins [Santisteban2001]. More details on this acquisition mode can be found
elsewhere, for both the measurement setup [kockelmann2007energy] and
applications [santisteban2002engineering, strobl2009advances]. Neutron
generation is extremely expensive and demand for neutron beamtime exceeds the
supply [Bentley2020]. Therefore, there is a high interest in efficient image
denoising techniques to reduce exposure time and subsequently increase
experiment throughput.

####  4.2.1 Data

In this study, we employ the dataset [jorgensen2019neutron] acquired at the
Imaging and Materials Science & Engineering (IMAT) beamline operating at the
ISIS spallation neutron source (Rutherford Appleton Laboratory, U.K.)
[burca2013modelling, kockelmann2018time]. More details on acquisition
parameters and preprocessing can be found elsewhere [ametova2021crystalline];
here we only briefly recall details relevant to this study.

A sample contains 6 aluminium tubes: five filled with metallic powder (copper
(Cu), aluminium (Al), zinc (Zn), iron (Fe) and nickel (Ni)), and one empty.
The neutron detector has $512\times 512$ pixels, 0.055 mm pixel size. A set of
spectral projections were acquired at 120 equally-spaced angular positions
over 180° rotation with 15 min exposure. Additionally, 8 flat field images (4
before and 4 after the acquisition) were acquired with the same exposure.

A typical problem of spectral measurements is that noise statistics vary quite
drastically across the spectrum. The beam spectrum at the IMAT beamline has a
crude bell shape with a peak around 3 Å [burca2013modelling]. Additionally,
the time-sensitive detector suffers from dead time meaning counts loss, hence,
additional signal distortions [tremsin2012high]. To alleviate the count loss
problem, the time (wavelength) domain is split in several independent
measurement intervals (4 in this case) and a special correction technique is
applied to the measured data [tremsin2012high]. Each interval has an
individual bin width; for this study the following bin width was used:
$0.7184\cdot 10^{-3}$ Å, $1.4368\cdot 10^{-3}$ Å, $\cdot 10^{-3}$ Å and
$2.8737\cdot 10^{-3}$ Å. To benchmark N2N, we generated three additional
datasets by rebinning the dataset in the original resolution (2840 energy bins
split into 4 measurement intervals with (1141, 814, 424, 464) bins in each).
The rebinning was performed individually in each interval by summing every (4,
2, 2, 1), (8, 4, 4, 2) and (16, 8, 8, 4) bins, resulting in datasets with
1366, 681 and 339 wavelength bins, respectively.

In Figure 5 we show average counts ($\mu$) for a single flat field image for
every dataset. Additionally, we show 2.5% and 97.5% quantiles (95% coverage
interval) of acquired counts to indicate noise levels across the spectrum.
Vertical dotted lines separate independent intervals. Note, that noise
dispersion increases drastically with the increase of flux (the effect of
counts loss becomes more apparent).

Figure 5: Noise and mean values in every individual wavelength channel for
different binning parameters.

####  4.2.2 Training and Analysis Details

In this experiment, we compare denoising performed on projections (N2N(P)) and
on reconstructed slices (N2N(S)). For both, we trained a model $f_{\theta}$ by
optimizing essentially the same loss as in the previous case study

$\mathbb{E}_{i,j}\|f_{\theta}(x_{i,j-1},x_{i,j+1})-x_{i,j}\|_{1}\xrightarrow[\theta]{}\min,$
(4)

where $x_{i,j}$ represents either a projection for an angle $i$ and an energy
channel $j$, or a slice number $i$ and an energy channel $j$. We used $i$ to
randomly split the dataset into the train and validation subsets in the
$80/20$ ratio.

A combination of the N2N denoising approach and the conventional FBP
reconstruction was compared with the advanced iterative reconstruction routine
proposed in [ametova2021crystalline]. The latter relies on expert expectations
on how the reconstructed image should look like. As the reconstructed
cylinders are expected to appear as solids, _i.e._ homogeneous regions, the
authors assumed a piece-wise constant signal in the spatial domain. This prior
knowledge is enforced through TV regularization [rudin1992nonlinear,
sidky2006accurate]. The signal in the spectral domain is expected to be piece-
wise smooth based on theoretical predictions for the materials employed in
this study [boin2012nxs]. In this case, regularization is achieved through
Total Generalized Variation (TGV) prior [bredies2010total]. Hence, we refer to
the iterative reconstruction method as TV-TGV. As before, the reconstruction
was implemented in CIL [Papoutsellis2021]. Code to reproduce results is
available from [evelina_ametova_2021_4884710].

####  4.2.3 Results Discussion

We begin with a visual comparison of different denoising approaches in the
spectral domain (Figure 6). We perform the comparison for 339 channels image
as the same binning was used for the case study in [ametova2021crystalline].
The theoretical predictions provide the ground truth for the comparison. As in
the previous case study, the conventional FBP reconstruction results are
uninterpretable. N2N performance is comparable to TV-TGV reconstruction. TV-
TGV provides smoother spectra at a cost of resolution loss. In contrast, N2N
results appear sharper but noisier for low-attenuative materials. Hence, we
conclude that there is a certain threshold noise level N2N can handle
efficiently.

Figure 7(a) shows a comparison of white beam (sum of all energies) and single
channel reconstruction for TV-TGV, N2N(S), and N2N(P), through direct
comparisons of reconstructed slices in the transverse plane. While for Fe and
Ni, both N2N(P) and N2N(S) perform comparably, for Cu and Al their performance
differs. The attenuation of Al is drastically lower than other materials,
which could lead to inconsistent predictions of the model for the projections
when another material occludes the Al cylinder. This problem is not relevant
for N2N(S). The Cu powder has a larger mean particle size than other powders
(the mean particle size is comparable to the voxel size), hence, stronger
spatial structures are visible in the cross-section. The structure changes
randomly through sample height. Therefore the N2N(S) model has less
information about the structure and might fail to recover it correctly.

As a reference revealing structures, we use an FBP slice averaged across all
energy levels, sacrificing spectral information for spatial. We also report
the structural similarity index (SSIM) between the single-energy slices and
the reference slice [Wang2004]. Both N2N approaches provide a sharper, more
detailed image than TV-TGV. Interestingly, while N2N(S) provides a visually
better, sharper image, this image has lower SSIM, compared to the N2N(P). We
hypothesize, that this is caused by the unintentional reduction of the streak
artifacts (highlighted in the top left callout in the N2N(P) slice). Streak
artifacts are very common in tomographic imaging and are caused by
insufficient angular sampling [kak2001principles].

We next explore denoising quality in the spatial (Figure 7(c)) and the
spectral domain (Figure 7(b)) given the increase of noise levels in the input
data. We control noise levels by changing binning: the smaller is the binning
step–the lower is SNR. We use the white beam slice reconstructed with FBP and
the theoretical predictions for SSIM calculations in the spatial and spectral
domains, respectively. While iterative reconstruction provides the best
results for the spectral domain, it provides the worst result for the spatial
domain. Excellent TV-TGV performance heavily capitalizes on the fact that the
cylinders are homogeneous inside. In terms of SSIM, N2N(P) outperforms N2N(S)
because N2N(S) additionally minimizes streak artifacts due to the angular
undersampling, hence, the discrepancy between the reference image and the
denoised one grows.

Another important observation is that N2N can be computed for the higher
number of channels. The training time of the model stays almost the same,
around 20 hours on average for the full volume, calculated on a $4\times
A5000$ machine. After the training, the model is capable of inferencing one
projection/slice at the rate of 20-30 energy channels per second. While TV-TGV
reconstruction for one slice with 339 channels takes several hours to complete
and reconstruction time increases with the increase in the number of channels
or the number of slices.

Figure 6: Comparison of denoising techniques in the spectral domain. Spectral
signal in a single reconstructed voxel is plotted. The experimental signal is
overlaid with the theoretical prediction (black solid line). Left: signal
before denoising; right: TV-TGV, N2N(S), and N2N(P) results are plotted. All
results are presented for the datatset with 339 energy bins.

(a) Comparison of denoising methods. Transverse slices showing white beam (sum
of all energies) and single channel at the energy level of 1.63
Åreconstruction for TV-TGV, N2N(S), and N2N(P).

(b) SSIM between predicted and experimental spectra.

(c) Averaged SSIM between denoised slices and white beam slice reconstructed
with FBP.

Figure 7: Qualitative (left) and quantitative (right) comparisons of the
denoising methods.

###  4.3 Cineradiography with the Bragg Magnifier

Cineradiography is an indispensable _in-vivo_ imaging technique relying on
X-rays to study processes over time. Unlike tomography where the total dose
can be controlled through both the number of projections and exposure, here
the total exposure time is defined by the duration of the process under
investigation. Hence, flux has to be reduced in order to reduce the dose.

For this case study, we employed the propagations-based phase contrast imaging
(PB-PCI) image formation mechanism. This mechanism is commonly used for
studies of weakly absorbing objects such as biological specimens and thin
samples [Fitzgerald2000]. The idea is to use phase shifts in passing rays to
enhance the otherwise low-contrast images. The object information can be
reconstructed from the detected interference pattern by additional algorithmic
treatment (so-called _phase retrieval_ [Lohse2020]). Depending on the actual
measurement setup and material under study, the phase retrieval problem can be
solved with different methods. In this specific case study, we used
convolution with a dedicated low-pass filter in the spatial domain (typically
referred to as Paganin method [Paganin2002]). Paganin low-pass filter heavily
affects noise distribution. On one hand, it significantly reduces high-
frequency noise (hence, increases PSNR). On the other hand, low-frequency
noise becomes more prominent causing so-called “cloudy” artifacts
[paganin2004quantitative]. For a single image, the effect of the low-frequency
noise might not be substantial. However, for a fluoroscopy image sequence,
this effect results in strong flickering affecting as the location of these
“clouds” changes randomly from frame to frame, affecting image
interpretability by domain specialists.

####  4.3.1 Data

In this case study, we used a batch of _in-vivo_ fluoroscopic series imaging a
parasitic wasp emerging from its host egg. The full dataset contained $138$
videos, imaged with $15$ fps ($0.066$ s exposure time per frame) with lengths
between $81$ and $7142$ frames per image series. The total number of frames is
$263,875$.

Additionally, we found a series of 100 frames, where the wasp was completely
still. We then calculated an average frame and used it as a reference image
with low noise. This averaged image was used for qualitative results
calculations. The average Peak Signal-to-Noise Ratio (PSNR) value before phase
retrieval is $25.2$ with a standard deviation of $0.02$. After the Paganin
phase retrieval, PSNR increases to $35.9$ with a standard deviation of $1.2$.

Because of the high potential impact of the data on life sciences, we can not
disclose any images from this case study. We will update the paper as soon as
a concurrent paper with experimental details is published.

####  4.3.2 Training and Analysis Details

Because of the movement of the sample, we can not use more than one frame as
the model input. We train the model $f_{\theta}$ by optimizing the loss

$\mathbb{E}_{i,j}\|f_{\theta}(x_{i,j-1})-x_{i,j}\|_{1}\xrightarrow[\theta]{}\min,$
(5)

where $x_{i,j}$ stands for the frame number $j$ from the image sequence number
$i$. We split all frames randomly in train and validation sets in $80/20$
ratio according to the index $i$. Additionally, we noticed that in some cases
temporal resolution was not high enough to smoothly capture fast movements
resulting in significant changes in sample position between adjacent frames.
We introduced additional filtering to alleviate potential blur due to the
large differences between neighbouring frames. During the training, we discard
the image pairs that have SSIM lower than a hand-optimized threshold.

####  4.3.3 Results

We applied the N2N denoising before and after phase retrieval. Table 2
summarizes PSNR and SSIM for both cases (the average of 100 frames without
movement was used as a reference for metrics calculation). Applying denoising
before the phase retrieval procedure provides significant improvement in PSNR
and SSIM. The benefits are preserved even after phase retrieval. Also, to
check the practical significance of the results we have questioned domain
experts to comment on the quality improvement. _After the denoising, the noise
appears less flickering, which makes it easier to concentrate on the sample
movements. While denoising makes the image smoother, there is no drastic blur,
and even relatively small details (_e.g._ , legs or antennae) are preserved._

| measured before Paganin | measured after Paganin
---|---|---
| PSNR | SSIM | PSNR | SSIM
no denoising | $25.2\pm 4e-3$ | $0.41\pm 1e-4$ | $36.0\pm 0.2$ | $0.97\pm 2e-3$
denoising before Paganin | $33.1\pm 13e-3$ | $0.49\pm 3e-4$ | $37.3\pm 0.3$ | $0.98\pm 1e-3$
denoising after Paganin | - | - | $36.0\pm 0.3$ | $0.97\pm 1e-3$
Table 2:  Comparison of the denoising done before and after Paganin phase
retrieval. We used the average of 100 frames where the wasp was still as a
noise-free image for these calculations. We report mean value and 95%
confidence interval.

##  5 Discussion

In general, systematic artifacts present in all channels (energy bins or time
frames) will not be compensated, hence relevant corrections (such as centre-
of-rotation compensation in CT) are essential. On the other hand, we noticed
that N2N applied to reconstructed slices significantly reduced the appearance
of the ring (Figure 3(b)) and undersampling artifacts (Figure 7(a)). The
magnitude and the limits of this secondary effect are in the scope of future
research.

We observed superficial “cloudy” artefacts after denoising present in
homogeneous image regions (such as the background). These artifacts do not
have any significant effect on CT data as they have much lower contrast than
actual image features. However, in the BM time series, they affect the overall
image perception as their location changes randomly from frame to frame
introducing the flickering effect. This flickering effect does not affect
image interpretability as it has very low contrast compared to image features.
Hence it is more of a cosmetic effect as the human eye is still sensitive to
it. A way around this is to generate a few images for each frame by adding
some Gaussian noise (_i.e._ to increase noise level) and take a median value
of the resulting denoised images. However, increasing the number of used
denoised images or the variance of the noise leads to blurring. We set these
parameters, guided by the expert judgement on the resulting image.

N2N assumes that the image pairs have equal signal values and independent
noise drawn from the same distribution. Strictly speaking, both assumptions
might be violated in spectral and time-resolved imaging where values in each
individual channel are either energy or time-dependent, and noise distribution
might be partially correlated (see data discussion in 4.2.1 for details). The
application of the method to such data is based on the assumption that the
variability of noise between the twin images is larger than the variability of
the signal.

In spectral CT, N2N can be applied to both projection images and to slices
after reconstruction. Any corrections in the projection domain are challenging
as they might cause or exaggerate existing inconsistency between projections
(consistent sinogram has strong restrictions expressed as Helgason–Ludwig
consistency condition [helgason1965radon]). However, our empirical studies did
not show any noticeable artifacts due to this inconsistency.

##  6 Conclusion

In this paper, we explored the applicability of the N2N method to denoising of
the time or energy-resolved images. N2N is a distribution-agnostic method, it
does not explicitly assume any properties of the signal or noise. The only
requirement is the ability to sample pairs of images that share the signal but
have iid noise. We demonstrated that this requirement, while not exactly met
for the multichannel data, can be relaxed to successfully apply the method.
The presented case studies showed that this method offers a robust and
efficient alternative to conventional denoising methods and regularized
iterative reconstruction methods. The method does not require fine-tuning of
parameters or handcrafted regularization term for any new dataset. Therefore,
its application can be heavily automated. Finally, the N2N method relies on a
rather intuitive assumption, hence it can be easily explained to non-experts
in the ML domain and smoothly introduced into the measurement practice.

Funding We gratefully acknowledge beamtime RB1820541 at the IMAT Beamline of
the ISIS Neutron and Muon Source, Harwell, UK. E A was funded by the Federal
Ministry of Education and Research (BMBF) and the Baden-Württemberg Ministry
of Science as part of the Excellence Strategy of the German Federal and State
Governments.

Acknowledgments This work made use of computational support by CoSeC, the
Computational Science Centre for Research Communities, through the
Collaborative Computational Project in Tomographic Imaging (CCPi).

Data Availability Statement The neutron tomography dataset is openly available
at the following URL/DOI: (10.5286/ISIS.E.RB1820541). The code to reproduce
the phantom and the dataset will be made publicly available upon the
publication of the paper.