#  R2D2: Robust Data-to-Text with Replacement Detection

Linyong Nan Lorenzo Jaime Yu Flores Yilun Zhao Yixin Liu
Luke Benson Weijin Zou Dragomir Radev
Yale University
{linyong.nan, lj.flores<EMAIL_ADDRESS>
###### Abstract

Unfaithful text generation is a common problem for text generation systems. In
the case of Data-to-Text (D2T) systems, the factuality of the generated text
is particularly crucial for any real-world applications. We introduce R2D2, a
training framework that addresses unfaithful Data-to-Text generation by
training a system both as a generator and a faithfulness discriminator with
additional replacement detection and unlikelihood learning tasks. To
facilitate such training, we propose two methods for sampling unfaithful
sentences. We argue that the poor entity retrieval capability of D2T systems
is one of the primary sources of unfaithfulness, so in addition to the
existing metrics, we further propose NER-based metrics to evaluate the
fidelity of D2T generations. Our experimental results show that R2D2 systems
could effectively mitigate the unfaithful text generation, and they achieve
new state-of-the-art results on FeTaQA, LogicNLG, and ToTTo, all with
significant improvements.

R2D2: Robust Data-to-Text with Replacement Detection

Linyong Nan Lorenzo Jaime Yu Flores Yilun Zhao Yixin Liu Luke Benson Weijin
Zou Dragomir Radev Yale University {linyong.nan, lj.flores,
<EMAIL_ADDRESS>

##  1 Introduction

Data-to-Text generation is the task of generating a text sequence that
describes some salient information of a knowledge source. Unlike Text-to-Text
generation whose input source is a text sequence containing knowledge that is
not extracted and represented in the canonical structured format, we assume
that the input of a Data-to-Text system is represented in some structured
format, e.g., RDF Gardent et al. (2017), relational or entity tables Lebret et
al. (2016); Wiseman et al. (2017). The Data-to-Text task can be divided into
two distinct processes as in many other text generation tasks Reiter and Dale
(2000); Gatt and Krahmer (2018). The first process involves selecting salient
information from the structured knowledge either based on natural language
query or other indication of saliency, and the second process comprises
organizing and planning of the previous selections to allow realization of the
surface text. Although this task has been studied comprehensively in many
works, from task design, modeling techniques, to application in different
domains Gardent et al. (2017); Lebret et al. (2016); Wiseman et al. (2017);
Novikova et al. (2017); Parikh et al. (2020); Nan et al. (2022), existing
Data-to-Text (D2T) systems exhibit a shortcoming that cannot be neglected.
They fail to reliably generate sentences that are faithful given the salient
content of the input table Chen et al. (2020a, b, 2021a); Uehara et al.
(2020); Ji et al. (2022). This limitation prevents the application of D2T
systems in real world scenarios. We therefore need to investigate possible
remedies.

We introduce a framework that prevents unfaithful Data-to-Text generation by
training a Data-to-Text system both as a generator as well as a faithfulness
discriminator. For faithfulness discrimination, we adopt the replaced token
detection objective, which was first proposed in ELECTRA Clark et al. (2020).
It was applied to the pre-training stage of the large-scale language models
for more sample-efficient training of contextualized representations of
sentences. ELECTRA is tasked to discriminate between original natural
sentences and token-replaced sentences by locating the positions of
replacement. The replaced tokens are sampled from a proposal distribution
using a generator such as a Masked Language Model to fill some masked tokens.

In our work, we perturbed the entailed reference sentences with two different
methods, a knowledge-based one and a model-based one, to obtain unfaithful
sentences whose surface forms are close to those of original sentences
(therefore having similar sequence likelihoods), but contradict to the input
table. Then we investigated ways of incorporating the discrimination task into
the existing maximum likelihood learning. Specifically, we explored the
settings of learning the sentence-level detection and generation in tandem,
and the token-level detection and generation in tandem. In addition, we also
experiment with incorporating the unlikelihood training objective Welleck et
al. (2019) on these unfaithful sentences to test its utility.

We conduct experiments on three Data-to-Text datasets to test the general
applicability of our approach: FeTaQA Nan et al. (2022), LogicNLG Chen et al.
(2020a), and ToTTo Parikh et al. (2020). Each dataset presents distinct
challenges while faithful generation is a common problem. We find that adding
the faithfulness discrimination task effectively mitigates the unfaithful
Data-to-Text generation, supported by our results on multiple datasets, on all
of which we are able to achieve new state-of-the-art results with evident
improvements. We compare and analyze the performance of our system and
existing state-of-the-art systems. To ensure the competence of comparison, we
also evaluate various metrics for their aptness of faithfulness evaluation. We
released our model and code at https://github.com/Yale-LILY/r2d2.

##  2 Method

###  2.1 Preliminaries

The de facto Data-to-Text task requires conditional language modeling of the
sequence pair $X=(x_{1},\ldots,x_{M}),Y=(y_{1},\ldots,y_{N})$ using a neural
model parameterized with $\theta$:
$p_{\theta}(y_{1},\ldots,y_{N}|x_{1},\ldots,x_{M})$, where
$(y_{1},\ldots,y_{N})$ is a natural language sentence that faithfully
describes the salient part of the input data which is linearized, along with
other contexts such as query or metadata, into the sequence
$X=(x_{1},\ldots,x_{M})$.

We want to sample unfaithful sentences that are in the vicinity of surface
forms of reference sentences, therefore also likely to be generated when only
learning with maximum likelihood loss. We aim to examine the effectiveness of
our proposed discrimination objectives in guiding the D2T model to attain
separable representations for these superficially similar but factually
critical sentences, and more importantly, we investigate how generation can
benefit from these additional objectives for robustness. We call our method
Robust Data-to-Text with Replacement Detection (R2D2), because we assign the
Data-to-Text model both a generation task and a discrimination task
(replacement detection). This process is illustrated in Figure 1.

Figure 1: Illustrated R2D2 training

Many existing works that study the unfaithful text generation problem in
summarization and translation have investigated the source of inconsistencies
between the system output and the input Cao et al. (2017); Maynez et al.
(2020); Goyal and Durrett (2020, 2021); Chen et al. (2021a). The main source
of unfaithfulness is that the outputs contain facts that cannot be entailed
(necessary consequence) from any explicitly stated facts or inferences derived
from the input table. Since facts can be represented using subjects,
predicates, and objects, these contradictions originate from wrong predictions
of entities (subject or object), predicates, or wrong arrangements. This
motivates our proposal of different methods of obtaining unfaithful sentences
in Section 2.2. Then we describe two learning objectives that we proposed to
add to the Data-to-Text modeling: 1) replacement detection objective in
Section 2.3; and 2) unlikelihood objective in Section 2.4. In Section 2.5, we
formulate our R2D2 fine-tuning that leverages these two objectives in addition
to the standard negative log likelihood loss for robust training of a Data-to-
Text model. We also propose NER-based evaluation metrics to complement the
existing evaluations of faithfulness in Section 2.6.

Figure 2: Faithfulness-based replacement method

###  2.2 Faithfulness-based Replacement

We aim to obtain sentences that are not entailed from the input table by
replacing tokens in the original sentences. As we discussed in the previous
paragraph, one of the major sources of unfaithfulness is the erroneous
prediction of entities or predicates. It is challenging to reliably extract
the predicates in canonical form, compare those in the system’s generations to
those in the input table, and find replacements for them. However, it is
feasible to extract and compare entities from the system generations and the
input table. Therefore we replace a span of tokens that constitute an entity
in the original sentence with another candidate entity comprising one or more
tokens. We adopted a RoBERTa-large-based named entity
recognizer111https://huggingface.co/Jean-Baptiste/roberta-large-ner-english to
extract all the entities in a sentence. To sample one unfaithful sentence, we
select one of these entities, with those located near the end of the sentence
to have a higher probability of being selected. This way, for the token-level
detection task, the model has more chances of obtaining a reasonable amount of
context for performing discrimination. We propose two different methods of
determining the candidate replacement, which are illustrated in Figure 2 and
described next.

#### Knowledge-based Method

Entities of similar types are usually the suspects of the wrong predictions.
Identifying such entities from the input table is nontrivial in other text
generation tasks whose input table is unstructured text. In D2T, the source
arranges similar entities together. For example, when the input is a table,
similar entities can be accessed in the same column; when the input is
semantic triple set, we can obtain similar entities by searching those
applicable to certain predicates. Our knowledge-based replacement method
exploits the structure of the input table to retrieve similar entities,
replacing the original entity with one in the sentence will lead to
contradiction. This process is illustrated in Figure 2. Note that for the
datasets we experimented on, we assume the entity to be replaced is the only
choice to make the sentence entailed, and replacement of any other entities
shown in the input table will violate the faithfulness of sentences. This
assumption does not necessarily hold in some examples, on which we will
elaborate in Section 3.6.

#### Model-based Method

Another way to obtain replacement candidates is by sampling from a proposal
distribution, in a way similar to ELECTRA. We sample replacements from the
baseline Data-to-Text model by teacher forcing partial sentence up to the
entity that needs to be replaced. Next we collect the D2T model predictions of
the continuation with nucleus sampling Holtzman et al. (2020), followed by
extracting altered entities shown in the predicted continuations to determine
the replacement candidates. With this approach, we are able to expose and
further train the D2T model with errors of its own predictions.

$\displaystyle\mathcal{L}_{\text{RD}_{\text{sent}}}(X^{(i)},Y)=-\left[l\cdot\log
p(l|Y,X^{(i)})+(1-l)\cdot\log\left(1-p(l|Y,X^{(i)})\right)\right]$

(1)

$\displaystyle\mathcal{L}_{\text{RD}_{\text{token}}}(X^{(i)},Y)=-\left[\sum_{t=1}^{|Y|}l_{t}\cdot\log
p(l_{t}|y_{\leq t},X^{(i)})+(1-l_{t})\cdot\log\left(1-p(l_{t}|y_{\leq
t},X^{(i)})\right)\right]$

(2)

$\displaystyle\mathcal{L}_{\text{UL}}(X^{(i)},Y_{\text{False}}^{(i,j)})=-\left[\sum_{t=\\{t|y_{t}\in\mathcal{C}^{(i,j)}\\}}^{|\mathcal{C}^{(i,j)}|}y_{t}\cdot\log(1-\hat{y_{t}})+(1-y_{t})\cdot\log(\hat{y_{t}})+\sum_{t=\\{t|y_{t}\notin\mathcal{C}^{(i,j)}\\}}^{|Y_{\text{False}}^{(i,j)}\setminus\mathcal{C}^{(i,j)}|}y_{t}\cdot\log(\hat{y_{t}})+(1-y_{t})\cdot\log(1-\hat{y_{t}})\right]$

(3)

$\displaystyle\mathcal{L}_{\text{NLL}}(X^{(i)},Y_{\text{True}}^{(i)})=-\sum_{t=1}^{|Y_{\text{True}}^{(i)}|}y_{t}\cdot\log(\hat{y_{t}})+(1-y_{t})\cdot\log(1-\hat{y_{t}})$

(4)

$\displaystyle\mathcal{L}_{\text{R2D2}}=\sum_{i=1}^{|\mathcal{D}|}\frac{1}{|N^{(i)}|+1}\left[\lambda\left(\mathcal{L}_{\text{NLL}}(X^{(i)},Y_{\text{True}}^{(i)})+\sum_{j=1}^{|N^{(i)}|}\mathcal{L}_{\text{UL}}(X^{(i)},Y_{\text{False}}^{(i,j)})\right)+(1-\lambda)\left(\mathcal{L}_{\text{RD}_{\\{\text{sent},\text{token}\\}}}(X^{(i)},Y_{\text{True}}^{(i)})+\sum_{j=1}^{|N^{(i)}|}\mathcal{L}_{\text{RD}_{\\{\text{sent},\text{token}\\}}}(X^{(i)},Y_{\text{False}}^{(i,j)})\right)\right]$

(5)

###  2.3 Replacement Detection in Generation

For each entailed sentence $Y_{\text{True}}^{(i)}$ that we sample from the
original dataset $\mathcal{D}$ where $i=1,\ldots,|\mathcal{D}|$, we generate
$N^{(i)}$ contradictory sentences which we denote as
$Y_{\text{False}}^{(i,j)}$ for $j=1,\ldots,|N^{(i)}|$. The number of
contradictory sentences we can generate given an entailed sentence depends on
the number of entities found in the original sentence, the input, and the
replacement method applied.

Figure 3: R2D2 sentence-level architecture

As shown in Figure 3, we add a sentence-level replacement detection task to
the existing Sequence-to-Sequence framework by eliciting the decoder to
generate a probability of the teacher-forced sentence being entailed or
contradictory at the end of the generation, similar to the sequence
classification usage of BART Lewis et al. (2020), except that in BART, the
same sequence that needs to be classified is fed into both the encoder and
decoder. The loss for sentence-level replacement detection is defined by
Equation (1).

Figure 4: R2D2 token-level architecture

A more challenging task is to perform a fine-grained, token-level
discrimination, as shown in Figure 4. Instead of predicting a discrimination
probability at the end of generation, we task the decoder to perform
discrimination at every step of token generation. Specifically, we use the
per-step last hidden output of the decoder, which encodes the source contexts
and teacher-forced partial generation contexts, to compute the discrimination
probability with a linear and sigmoid layer. The token-level replacement
detection loss is defined by Equation (2).

###  2.4 Replacement Unlikelihood Training

Unlikelihood training is first proposed in Welleck et al. (2019) to address
the repetition problem of the neural text generation. This training objective
aims to decrease the decoder’s probability of generating tokens that are
already seen in the teacher-forced generation contexts. The applicability of
this objective to the Data-to-Text task is also argued in Uehara et al.
(2020).

Instead of using the generated tokens to construct the negative candidate set
defined for each step, we define the sentence-level negative candidate span
$\mathcal{C}^{(i,j)}$ for each contradictory sentence
$Y_{\text{False}}^{(i,j)}$. The span contains, for each time step, one
replaced token that should have a low probability of being generated. We
calculate the sequence-level unlikelihood loss for this replaced token span
and apply regular likelihood loss for other original tokens, which we denote
as $Y_{\text{False}}^{(i,j)}\setminus\mathcal{C}^{(i,j)}$. We denote the per
step prediction as $\hat{y_{t}}=p(y_{t}|y_{<t},X^{(i)})$. The unlikelihood
loss for the entire sentence is specified in Equation (3).

###  2.5 R2D2 Fine-tuning

We propose the final R2D2 fine-tuning loss objective as in equation (5). It
combines a generation task loss component and a discrimination task loss
component. Each of them is calculated from one entailed instance and $N^{(i)}$
contradictory instances. The generation task component consists of a regular
negative log likelihood loss for the entailed instance, as described in
Equation (4), and an unlikelihood loss for the contradictory instances. The
discrimination task component contains either sentence-level or token-level
replacement detection loss for both entailed and contradictory instances. We
use $\lambda$ to balances the importance between the two loss components.

###  2.6 NER-based Evaluation Metrics

To better understand how the D2T system’s retrieval capability correlates with
faithfulness, we propose information-extraction based metrics that compare the
named entities contained in the generated sentences to those contained in the
reference sentences or input data. We believe these metrics help us better
distinguish between the unfaithfulness caused by wrong retrieval of entities
and that caused by wrong prediction of predicates/relations. Specifically, we
propose the following five indicators:

  * •

Reference coverage (RC): percentage of entities in the reference that are also
shown in the prediction.

  * •

Ref-hit & Input-hit (RI): percentage of entities shown in the prediction that
are shown in both the reference and input table.

  * •

Ref-hit & Input-miss (RM): percentage of predicted entities that are shown in
the reference but not the input table. This case is rare since it indicates
the existence of entities that are not input-grounded in the reference.

  * •

Ref-miss & Input-hit (MI): percentage of predicted entities that are shown in
the table, but not in the reference. This case identifies wrong or unnecessary
retrieval of entities from the input (when the indication of saliency is
evident).

  * •

Ref-miss & Input-miss (MM): percentage of predicated entities that are neither
shown in the table nor the reference. This case identifies the prediction of
entities likely by hallucination.

Figure 5: NER-based metrics vs sacreBLEU Figure 6: NER-based metrics vs NLI-
Acc

Figure 5 and Figure 6 show the correlation between NER-based metrics and
sacreBLEU, NLI-Acc, respectively. As expected, the reference coverage rate
positively correlates with both metrics. While both reference hit and input
hit are important, the rate of predicted entities not shown in reference
negatively correlates with sacreBLEU (MI and MM) and NLI-Acc (MI). The trend
is less clear for RM and MM since these are rare cases, which can also be
shown in Table 3.3.

Since many existing automatic metrics for text generation tasks are not
proposed with an aim of reflecting the faithfulness of the sentences, an
examination of all metrics reported in our work is crucial for interpreting
the results. As we are able to reliably generate an unfaithful version of most
of the reference texts, we contaminate the references in the FeTaQA test split
in a controlled manner: we generate five variants of texts with different
percentage of the references being replaced with their unfaithful parallel
($0\%$ version contains only the references and $100\%$ version contains only
the unfaithful sentences). We evaluate the variants that are contaminated to
different degrees using the evaluation metrics we reported, in order to
investigate how reliable they are in reflecting the faithfulness of any system
generated sentences.

% Unfaithful Sentences | Fact-based | String-based | NER-based
---|---|---|---
NLI-Acc | sacreBLEU | Rouge-1/2/L | PARENT | TER | METEOR | BERTScore | RC | RI | RM | MI | MM
0% (ref.) | 86.17 | 100.0 | 100/100/100 | 89.87 | 0.0 | 99.99 | 100.0 | 99.95 | 94.15 | 5.80 | 0.0 | 0.05
25% | 73.29 | 96.1 | 97.62/96.31/97.62 | 87.23 | 2.57 | 98.08 | 99.43 | 96.37 | 90.82 | 5.67 | 2.98 | 0.53
50% | 61.21 | 92.2 | 95.28/92.63/95.29 | 84.63 | 5.18 | 96.24 | 98.87 | 93.01 | 87.53 | 5.55 | 5.86 | 1.06
75% | 48.63 | 88.3 | 92.87/88.87/92.87 | 81.91 | 7.84 | 94.36 | 98.29 | 89.32 | 84.33 | 5.31 | 8.81 | 1.54
100% | 36.20 | 84.4 | 90.47/85.13/90.48 | 79.16 | 10.36 | 92.48 | 97.72 | 85.70 | 80.93 | 5.24 | 11.72 | 2.11

Table 1: Reliability test result of evaluation metrics for detecting
unfaithfulness

As shown in Table 1, most of the metrics are able to reflect the degree of
unfaithfulness contained in the prediction texts, though our test only
contains the type of unfaithfulness that originates from erroneous selection
of entities. A more rigorous study would test other types of unfaithfulness,
such as wrong prediction of relations or arrangement of entities.
Nevertheless, we observe that some metrics, especially NLI-Acc, are more
sensitive to the type of unfaithfulness that we tested, while an unfaithful
sentence can still obtain a very high BERTScore Zhang et al. (2020).

##  3 Experiments

We first introduce the datasets that we experiment with in Section 3.1, and
the metrics we adopted for evaluation in Section 3.2. Then we report the
baseline models we are comparing with, and the implementation and training
details in Section 3.3. In Section 3.4 and 3.5, we report both the automatic
evaluation results and the human evaluation results, followed by an analysis
of them in Section 3.6.

###  3.1 Datasets

FeTaQA Nan et al. (2022) is a free-form table question answering dataset. It
introduces a task that requires retrieving the correct contents from the table
based on the question, integrating and inferring from the retrieved facts, and
generating a free-form answer. A sentence that contains erroneous selections
of facts, even if they appear in the input table, are still considered as
unfaithful, because the sentence is inconsistent (incorrect) with the input
question.

LogicNLG Chen et al. (2020a) is a table-to-text dataset that tasks the the
models to generate logically entailed sentences. There is no indication of
what is considered as salient given a table, as long as the generated sentence
is entailed given all the facts contained in the table. Since there are
numerous entailed facts that are different from the references in the surface-
form, they propose input-based metrics that compare the facts in the generated
sentence and those in the input. Although we report the experiment results of
reference-based metrics for this dataset, it is worth noting that unlike other
datasets mentioned above, faithfulness to the input is more important for
LogicNLG then faithfulness to the references.

ToTTo Parikh et al. (2020) is a table-to-text dataset that contains
annotations of salient content of tables (highlighted table cells). The task
does not require any content selection (when only highlighted cells constitute
the input), but only text planning and surface realization of the inputs,
which are expected to be described with full coverage, and the median number
of highlighted cells is 3.

###  3.2 Evaluation Metrics

We report results of a variety of automatic evaluation metrics used in the
past studies to provide a comprehensive comparison of existing methods and our
proposed method. We include fact-verification based metrics, NLI-Acc Chen et
al. (2020a, b), which specifically aims to evaluate the faithfulness of the
sentences. We also report string-based metrics that evaluate the $n$-gram or
other types of string match between predictions and references, such as
sacreBLEU Post (2018), ROUGE-{1, 2, L} Lin (2004), TER Snover et al. (2006),
METEOR Banerjee and Lavie (2005), PARENT Dhingra et al. (2019) (which also
leverages the input data). In addition, we report evaluation results of the
NER-based metrics.

###  3.3 Experiment Settings

#### Baselines

The state-of-the-art system for the Data-to-Text task is obtained by fine-
tuning a pre-trained sequence-to-sequence model, such as T5 Raffel et al.
(2020), BART Lewis et al. (2020) or Longformer-Encoder-Decoder (LED) Beltagy
et al. (2020). We report evaluations of existing state-of-the-art models for
all datasets. We also fine-tuned T5 on FeTaQA, LogicNLG and ToTTo by
ourselves, so that the learning objective is the key control variable in our
comparison.

#### Implementation

We use T5-base as the pre-trained checkpoint from which we fine-tune either
regularly (Reg-FT) or using our proposed method (R2D2-FT). For R2D2-FT, we
initialize our model from a checkpoint that has been fine-tuned regularly for
15 epochs, and train the additional linear layer for sentence or token
replacement detection from random initialization. We find this fine-tuning
warmup help improve the performance in general. We use the Adafactor optimizer
Shazeer and Stern (2018) with a learning rate of 5e-5. We use the batch size
of 8 for FeTaQA and 32 for the others.

####  R2D2 Configuration

To assess the necessity of the discrimination loss and unlikelihood loss, we
experimented fine-tuning T5 only with the discrimination loss, or only with
the unlikelihood loss, or both (all in addition to the NLL loss). For
discrimination loss, we also experiment with adding sentence-level or token-
level discrimination to investigate the effect of discrimination granularity
in assisting faithful text generation. For all the training variants above, we
also compare two methods of obtaining the contradictory sentences, knowledge-
based and model-based methods. Since the number of contradictory sentences
obtained (which we denote as $N^{(i)}$ in Section 2.5) varies depending on the
method used (as shown in Table 2), we also experiment with using different
numbers of contradictory sentences in the R2D2 fine-tuning: $N^{(i)}=1$
(xsmall), $3$ (small), $5$ (medium), $10$ (large) or max (full). Since the
maximum size of the perturbations obtained by model-based method is small, we
only compared xsmall and full for the model-based method setting.

Dataset |  | Original
---
Train-split
Size
Knowledge-based | Model-based
FeTaQA | 7,325 | 238,891 ($\times$32.6) | 25,313 ($\times$3.5)
LogicNLG | 21,873 | 413,247 ($\times$18.9) | 91,183 ($\times$4.2)
ToTTo | 120,761 | 1,165,067 ($\times$9.6) | 450,770 ($\times$3.7)

Table 2: Number of obtainable contradictory sentences for the train-split of
all datasets.

Systems | Fact-based | String-based
---|---|---
NLI-Acc | sacreBLEU | Rouge-1/2/L | PARENT | TER | METEOR
Xie et al. (2022) | - | 29.9 | 61.77/39.44/51.93 | - | - | 48.53
Reg-FT | 74.79 | 30.5 | 63.47/41.77/54.04 | 44.24 | 66.37 | 55.2
R2D2-FT | 77.93 | 31.5 | 63.50/41.71/54.05 | 45.32 | 68.55 | 56.27

Table 3: FeTaQA-test automatic evaluation result

Systems | Fact-based | String-based
---|---|---
NLI-Acc | SP-Acc | BLEU-1/2/3 | sacreBLEU | Rouge-1/2/L | PARENT | TER | METEOR
Chen et al. (2021b) | 76.9 | 43.9 | 49.5/28.6/15.3 | - | - | - | - | -
Reg-FT | 84.11 | 45.97 | 51.63/32.24/18.75 | 18.2 | 42.74/20.89/36.77 | 32.36 | 86.38 | 36.55
R2D2-FT | 85.57 | 50.80 | 51.76/32.42/18.65 | 18.5 | 42.63/20.73/36.84 | 31.38 | 80.97 | 35.73

Table 4: LogicNLG-test automatic evaluation result

Systems | Split | Fact-based | String-based
---|---|---|---
NLI-Acc | sacreBLEU | Rouge-1/2/L | PARENT | TER | METEOR
| All | - | 47.7 | - | 57.1 | - | -
| Overlap | - | - | - | - | - | -
Kale and Rastogi (2020) | Nonoverlap | - | 39.6 | - | 52.6 | - | -
| All | 90.29 | 48.7 | 75.92/55.99/67.40 | 58.43 | 48.53 | 71.01
| Overlap | 91.17 | 56.6 | 79.95/62.39/72.78 | 62.88 | 40.85 | 75.56
Reg-FT | Nonoverlap | 89.42 | 41.0 | 72.04/49.80/62.20 | 54.14 | 55.69 | 66.54
| All | 91.27 | 49.2 | 75.54/55.59/67.06 | 59.05 | 50.40 | 71.99
| Overlap | 91.70 | 56.7 | 79.42/61.68/72.14 | 62.89 | 42.72 | 76.47
R2D2-FT | Nonoverlap | 90.86 | 41.9 | 71.79/49.69/62.15 | 55.35 | 57.57 | 67.67

Table 5: ToTTo-dev automatic evaluation result

Dataset | Systems | Split | RC | RI | RM | MI | MM
---|---|---|---|---|---|---|---
FeTaQA | Reg | A | 72.30 | 69.88 | 1.48 | 24.90 | 3.73
R2D2 | A | 73.06 | 70.92 | 1.62 | 23.82 | 3.64
LogicNLG | Reg | A | 37.29 | 26.07 | 0.97 | 61.60 | 11.36
R2D2 | A | 37.93 | 26.62 | 0.44 | 67.12 | 5.82
ToTTo | Reg | A | 82.47 | 74.95 | 3.14 | 17.66 | 4.24
O | 84.87 | 77.95 | 3.40 | 15.10 | 3.54
N | 80.14 | 72.04 | 2.89 | 20.14 | 4.92
R2D2 | A | 83.24 | 74.06 | 3.26 | 18.31 | 4.37
O | 85.45 | 76.90 | 3.63 | 15.70 | 3.77
N | 81.08 | 71.30 | 2.89 | 20.85 | 4.95

Table 6: NER-based automatic evaluation result. A, O, N stands for All,
Overlap and Nonoverlap.

###  3.4 Automatic Evaluation

We report the performance of the previous state-of-the-art system, T5 fine-
tuned only with negative log likelihood (NLL) loss by ourselves, and the best
T5 fine-tuned with R2D2 loss for FeTaQA (Table 3.3), LogicNLG (Table 3.3) and
ToTTo (Table 3.3), based on metrics used in the existing literature. In Table
3.3, we also report their performances using the NER-based metrics that we
proposed. We also report the full experiment results of FeTaQA that contain
evaluations of different R2D2 configurations in Table A of the Appendix.

We obverse that across all the datasets, most of the systems fine-tuned with
different R2D2 configurations are able to perform better than system that is
fine-tuned only with NLL loss. As expected, the improvements are more evident
in the fact-verification-based metrics that evaluate the faithfulness of the
sentences. We find that the best R2D2 configuration requires both the token-
level discrimination and the unlikelihood learning objectives, and that
contradictory sentences obtained by knowledge-based perturbation are more
beneficial than those obtained by model-based perturbation for fine-tuning
when all other configuration constants are controlled. We also find that
system performance does not necessarily improve as we increase the number of
unfaithful sentences used for fine-tuning, and that the best configuration for
$N^{(i)}$ seems to be three to five in most cases.

We examine the improvement of faithfulness using the NER-based metrics, and
find that the coverage of the entities appeared in the reference (RC) improves
across all datasets. For FeTaQA, we notice that our system is able to retrieve
input-grounded entities more accurately (shown by increased RI and decreased
MI scores). For LogicNLG which has no right or wrong retrieval of input-
grounded entities as long as the description of them is faithful, we obverse
an evident decline of MM and increments of both RI and MI (with more evident
gain in MI), indicating that our system is able to reduce hallucinations of
irrelevant entities and instead retrieving input-grounded ones.

###  3.5 Human Evaluation

Since the automatic evaluations are not always reliable in determining the
faithful aspect of a sentence, which can be seen in our metrics reliability
test shown in Table 1: around 14% faithful sentences are deemed to be
unfaithful by the NLI-Acc metric, and more importantly, it fails to identify
around 36% of the unfaithful sentences. We conduct the human evaluation based
on two criteria: a sentence is (1) faithful if all facts contained are
entailed by the input, and when a question is present in the input, the
sentence only contains necessary facts; (2) adequate with respect to reference
if the sentence contains same or more facts than the reference. We asked three
human evaluators to evaluate 200 samples of each dataset (100 samples in each
of the overlap/nonoverlap split for ToTTo), and each sample is provided with
all the inputs, the reference, and two system generated sentences. We report
the percentage of faithful and adequate sentences generated by the baseline
system and our system on all datasets in Table 3.5, and the results validate
R2D2’s effectiveness in faithful text generation.

Dataset | Systems | Split | Faithfulness (%) |  | Coverage
---
w.r.t Ref (%)
FeTaQA | Reg-FT | A | 61.33 | 54.83
R2D2-FT | A | 68.67 | 61.83
LogicNLG | Reg-FT | A | 40.67 | 69
R2D2-FT | A | 41.17 | 66.50
ToTTo | Reg-FT | A | 81 | 81.17
O | 83.66 | 82.66
N | 78.32 | 79.66
R2D2-FT | A | 83.16 | 84.67
O | 84.34 | 88.66
N | 82 | 80.66

Table 7: Human evaluation result. A, O, N stands for All, Overlap and
Nonoverlap.

###  3.6 Further Analysis

Comparing the effectiveness of R2D2 on different datasets and with different
evaluations, we found that FeTaQA and ToTTo benefit more than LogicNLG. We
speculate this is because many sentences of LogicNLG describe some entailed
facts of entities of a single table column (usually involving comparisons),
which usually contain single and less restricted predicate that could be
applied to many homogeneous entities, and this would invalidate our
perturbation methods. We provide one such example in Figure 7 of the Appendix.
We also notice that for ToTTo, the improvement is less evident than that for
FeTaQA. Besides less room for improvement, we observe no evident change in the
entities retrieved by both systems compared with those in the reference or
input, while human evaluation indicates there are still around 17% unfaithful
sentences. We speculate the source of unfaithfulness of these sentences are
due to wrong predictions of relations/predicates, which are not captured and
included into the R2D2 fine-tuning by our current perturbation method. To
avoid invalidation of perturbation (as in the case of LogicNLG) and also to
capture erroneous relation predictions, a better perturbation method has to
operate on fact triples instead of entities, but this requires a reliable and
domain-independent fact extraction system, which we will explore in future.

##  4 Related Work

###  4.1 Unfaithfulness in Text Generation

In the context of text generation, hallucination refers to the phenomenon of
neural models “generating unfaithful or nonsensical text” (Ji et al., 2022).
For instance, the generated text can contradict information in the source, or
have nothing to do with the source, and hence cannot be verified. One reason
for such hallucination is poor data collection; for example, information in
the target sentence may not actually be present in the table, which can occur
when the tables and sentences do not actually align. Another source of such
hallucination is training, such as the exposure bias, or the task expects more
output diversity. Metrics based on information extraction, question answering,
and natural language inference, have been proposed to measure such
hallucination, which we employ to evaluate the performance and faithfulness of
R2D2.

###  4.2 Contrastive Learning

Contrastive learning (Hadsell et al., 2006) tasks the model with maximizing
the representation similarity between neighboring examples while minimizing
the similarity between distant examples. Contrastive learning has recently
been used in various NLP tasks, including sentence embedding (Gao et al.,
2021), machine translation (Yang et al., 2019; Pan et al., 2021), text
summarization (Cao and Wang, 2021; Liu and Liu, 2021; Xu et al., 2021; Sun and
Li, 2021; Wang et al., 2021; Liu et al., 2022), data-to-text generation
(Uehara et al., 2020) and other tasks (Cho et al., 2021; Lee et al., 2021; Su
et al., 2022, 2021). Unlike Uehara et al. (2020), in which unfaithful
sentences are obtained by replacing a set of keywords (such as replacing low
to high, gain to drop) that only apply to the finance domain, we propose
domain-independent methods for sampling unfaithful sentences either by
exploiting the structure of input knowledge or utilizing the D2T model’s own
mistakes.

###  4.3 Unlikelihood Training

To address the degeneration problems of models trained only with Maximum
Likelihood Estimation, many works have proposed alternative approaches Tu et
al. (2016); Li et al. (2020); Holtzman et al. (2020); Lin et al. (2021). Among
them, unlikelihood training was introduced as a means of decreasing the
probability that the model generates certain tokens Welleck et al. (2019). In
a D2T context, we adopt unlikelihood training to decrease the probability that
the model generates tokens which are not entailed by the given contexts.

###  4.4 Evaluation Metrics

Ideally, a data-to-text model should be evaluated based on its ability to
generate logical sentences based on the provided reference data. Current
methods however, typically only compare the model output summary to the gold
standard label. This includes n-gram based (e.g. BLEU, ROUGE, and METEOR) or
edit distance based metrics (e.g. TER) Sai et al. (2022), or embedding-based
similarity metrics (e.g. BERTScore) Zhang et al. (2019). Another set of
metrics compare the information present in the output and the label. This is
done by extracting subject, object, and their relations in the output and
label, and comparing both sets of elements Wiseman et al. (2017). We evaluate
our model using multiple metrics to understand different aspects of its
performance.

###  4.5 Natural Language Inference

Natural language inference (NLI) refers to the task of classifying whether a
hypothesis entails, contradicts, or is unrelated to a premise Bowman et al.
(2015). Benchmarks for this task include the Stanford NLI dataset (SNLI)
Bowman et al. (2015), the Question-Answering NLI dataset (QNLI) Wang et al.
(2018), and MultiNLI Williams et al. (2017). In the context of D2T, NLI can be
used to evaluate whether a model’s generated text can be inferred from the
input table Chen et al. (2020a). An ideal model would generate sentences which
an NLI model would classify as being entailed by the input table. In line with
the work of TabFact Chen et al. (2020b), LogicNLG Chen et al. (2020a), and
SnowBall Shu et al. (2021), R2D2 incorporates this idea into data-to-text
training by using NLI as a learning objective during the training procedure.

##  5 Conclusion

In this work, we introduced R2D2, a training framework that effectively
mitigates the unfaithful text generation problem for the D2T task. Training
with the regular maximum likelihood loss can lead to generation of sentences
that are similar to the references but are unfaithful to the input. We
therefore propose to add a discrimination task and an unlikelihood training to
encourage the model to generate separable representations of these critical
sentences. We proposed two methods of sampling these critical sentences that
are unfaithful: the knowledge-based method exploits the structure of the input
knowledge, and the model-based method samples the D2T model’s own mistakes. We
examined existing metrics for evaluating faithfulness of generated sentences,
and proposed NER-based metrics that assess the entity retrieval capability of
the Data-to-Text systems, as we argued the incompetence of which is one of the
leading causes of unfaithfulness. We experimented on multiple Data-to-Text
datasets of different task constructs, and achieved noticeable improvements
over the state-of-the-art.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6633–6649, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Cao et al. (2017) Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017.  Faithful to the original: Fact aware neural abstractive summarization.
  * Chen et al. (2021a) Sihao Chen, Fan Zhang, Kazoo Sone, and Dan Roth. 2021a.  Improving faithfulness in abstractive summarization with contrast candidate generation and selection.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5935–5941, Online. Association for Computational Linguistics.
  * Chen et al. (2020a) Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a.  Logical natural language generation from open-domain tables.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7929–7942, Online. Association for Computational Linguistics.
  * Chen et al. (2020b) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b.  Tabfact : A large-scale dataset for table-based fact verification.  In _International Conference on Learning Representations (ICLR)_ , Addis Ababa, Ethiopia.
  * Chen et al. (2021b) Wenqing Chen, Jidong Tian, Yitian Li, Hao He, and Yaohui Jin. 2021b.  De-confounded variational encoder-decoder for logical table-to-text generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5532–5542, Online. Association for Computational Linguistics.
  * Cho et al. (2021) Woon Sang Cho, Yizhe Zhang, Sudha Rao, Asli Celikyilmaz, Chenyan Xiong, Jianfeng Gao, Mengdi Wang, and Bill Dolan. 2021.  Contrastive multi-document question generation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 12–30, Online. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  ELECTRA: Pre-training text encoders as discriminators rather than generators.  In _ICLR_.
  * Dhingra et al. (2019) Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019.  Handling divergent reference texts when evaluating table-to-text generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4884–4895, Florence, Italy. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gardent et al. (2017) Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017.  The WebNLG challenge: Generating text from RDF data.  In _Proceedings of the 10th International Conference on Natural Language Generation_ , pages 124–133, Santiago de Compostela, Spain. Association for Computational Linguistics.
  * Gatt and Krahmer (2018) Albert Gatt and Emiel Krahmer. 2018.  Survey of the state of the art in natural language generation: Core tasks, applications and evaluation.  _J. Artif. Int. Res._ , 61(1):65–170.
  * Goyal and Durrett (2020) Tanya Goyal and Greg Durrett. 2020.  Evaluating factuality in generation with dependency-level entailment.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3592–3603, Online. Association for Computational Linguistics.
  * Goyal and Durrett (2021) Tanya Goyal and Greg Durrett. 2021.  Annotating and modeling fine-grained factuality in summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1449–1462, Online. Association for Computational Linguistics.
  * Hadsell et al. (2006) Raia Hadsell, Sumit Chopra, and Yann LeCun. 2006.  Dimensionality reduction by learning an invariant mapping.  In _Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 2_ , CVPR ’06, page 1735–1742, USA. IEEE Computer Society.
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The curious case of neural text degeneration.  In _International Conference on Learning Representations_.
  * Ji et al. (2022) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2022.  Survey of hallucination in natural language generation.
  * Kale and Rastogi (2020) Mihir Kale and Abhinav Rastogi. 2020.  Text-to-text pre-training for data-to-text tasks.  In _Proceedings of the 13th International Conference on Natural Language Generation_ , pages 97–102, Dublin, Ireland. Association for Computational Linguistics.
  * Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016.  Neural text generation from structured data with application to the biography domain.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1203–1213, Austin, Texas. Association for Computational Linguistics.
  * Lee et al. (2021) Seanie Lee, Dong Bok Lee, and Sung Ju Hwang. 2021.  Contrastive learning with adversarial perturbations for conditional text generation.  In _International Conference on Learning Representations_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2020) Margaret Li, Stephen Roller, Ilia Kulikov, Sean Welleck, Y-Lan Boureau, Kyunghyun Cho, and Jason Weston. 2020.  Don’t say that! making inconsistent dialogue unlikely with unlikelihood training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4715–4728, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2021) Xiang Lin, Simeng Han, and Shafiq Joty. 2021.  Straight to the gradient: Learning to use novel tokens for neural text generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 6642–6653. PMLR.
  * Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021.  SimCLS: A simple framework for contrastive learning of abstractive summarization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 1065–1072, Online. Association for Computational Linguistics.
  * Liu et al. (2022) Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022.  BRIO: Bringing order to abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2890–2903, Dublin, Ireland. Association for Computational Linguistics.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On faithfulness and factuality in abstractive summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1906–1919, Online. Association for Computational Linguistics.
  * Nan et al. (2022) Linyong Nan, Chiachun Hsieh, Ziming Mao, Xi Victoria Lin, Neha Verma, Rui Zhang, Wojciech Kryściński, Nick Schoelkopf, Riley Kong, Xiangru Tang, Mutethia Mutuma, Ben Rosand, Isabel Trindade, Renusree Bandaru, Jacob Cunningham, Caiming Xiong, and Dragomir Radev. 2022.  FeTaQA: Free-form Table Question Answering.  _Transactions of the Association for Computational Linguistics_ , 10:35–49.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, and Verena Rieser. 2017.  The E2E dataset: New challenges for end-to-end generation.  In _Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 201–206, Saarbrücken, Germany. Association for Computational Linguistics.
  * Pan et al. (2021) Xiao Pan, Mingxuan Wang, Liwei Wu, and Lei Li. 2021.  Contrastive learning for many-to-many multilingual neural machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 244–258, Online. Association for Computational Linguistics.
  * Parikh et al. (2020) Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020.  ToTTo: A controlled table-to-text generation dataset.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1173–1186, Online. Association for Computational Linguistics.
  * Post (2018) Matt Post. 2018.  A call for clarity in reporting BLEU scores.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 186–191, Brussels, Belgium. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Reiter and Dale (2000) Ehud Reiter and Robert Dale. 2000.  _Building Natural Language Generation Systems_.  Studies in Natural Language Processing. Cambridge University Press.
  * Sai et al. (2022) Ananya B. Sai, Akash Kumar Mohankumar, and Mitesh M. Khapra. 2022.  A survey of evaluation metrics used for nlg systems.  _ACM Comput. Surv._ , 55(2).
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4596–4604. PMLR.
  * Shu et al. (2021) Chang Shu, Yusen Zhang, Xiangyu Dong, Peng Shi, Tao Yu, and Rui Zhang. 2021.  Logic-consistency text generation from semantic parses.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4414–4426, Online. Association for Computational Linguistics.
  * Snover et al. (2006) Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006\.  A study of translation edit rate with targeted human annotation.  In _Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers_ , pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas.
  * Su et al. (2022) Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. 2022\.  A contrastive framework for neural text generation.  _ArXiv_ , abs/2202.06417.
  * Su et al. (2021) Yixuan Su, Fangyu Liu, Zaiqiao Meng, Lei Shu, Ehsan Shareghi, and Nigel Collier. 2021.  Tacl: Improving bert pre-training with token-aware contrastive learning.  _ArXiv_ , abs/2111.04198.
  * Sun and Li (2021) Shichao Sun and Wenjie Li. 2021.  Alleviating exposure bias via contrastive learning for abstractive text summarization.  _CoRR_ , abs/2108.11846.
  * Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016.  Modeling coverage for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 76–85, Berlin, Germany. Association for Computational Linguistics.
  * Uehara et al. (2020) Yui Uehara, Tatsuya Ishigaki, Kasumi Aoki, Hiroshi Noji, Keiichi Goshima, Ichiro Kobayashi, Hiroya Takamura, and Yusuke Miyao. 2020.  Learning with contrastive examples for data-to-text generation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 2352–2362, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2021) Danqing Wang, Jiaze Chen, Hao Zhou, Xipeng Qiu, and Lei Li. 2021.  Contrastive aligned joint learning for multilingual summarization.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2739–2750, Online. Association for Computational Linguistics.
  * Welleck et al. (2019) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2019.  Neural text generation with unlikelihood training.
  * Williams et al. (2017) Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2017.  A broad-coverage challenge corpus for sentence understanding through inference.
  * Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2253–2263, Copenhagen, Denmark. Association for Computational Linguistics.
  * Xie et al. (2022) Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022.  Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.  _arXiv preprint arXiv:2201.05966_.
  * Xu et al. (2021) Shusheng Xu, Xingxing Zhang, Yi Wu, and Furu Wei. 2021.  Sequence level contrastive learning for text summarization.  _CoRR_ , abs/2109.03481.
  * Yang et al. (2019) Zonghan Yang, Yong Cheng, Yang Liu, and Maosong Sun. 2019.  Reducing word omission errors in neural machine translation: A contrastive learning approach.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6191–6196, Florence, Italy. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020\.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.

##  Appendix A Appendix

Discrimination Loss |  Discrimination Granularity |  Unlikelihood Loss |  Perturbation Method |  Perturbation Size | Fact-based | String-based
---|---|---|---|---|---|---
NLI-Acc | sacreBLEU | Rouge-1/2/L | PARENT | TER | METEOR
✓ | sent-level | ✗ |  Knowledge based | xsmall | 75.79 | 29.7 | 63.26/41.08/53.54 | 43.21 | 66.65 | 54.69
small | 74.59 | 29.7 | 63.53/41.54/54.06 | 43.43 | 65.75 | 54.67
medium | 77.13 | 30.2 | 63.47/41.43/53.68 | 43.99 | 66.96 | 55.09
large | 74.64 | 29.2 | 62.57/40.74/53.22 | 42.74 | 66.93 | 53.76
full | 73.99 | 29.4 | 63.26/41.24153.83 | 43.45 | 66.58 | 54.32
✓ | sent-level | ✗ |  Model based | xsmall | 76.83 | 30.5 | 63.46/41.57/53.84 | 44.49 | 66.47 | 55.33
full | 74.69 | 29.2 | 63.08/41.03/53.57 | 43.01 | 66.76 | 54.39
✓ | sent-level | ✓ |  Knowledge based | xsmall | 76.14 | 30.4 | 63.36/41.03/53.64 | 43.61 | 67.89 | 55.61
small | 76.44 | 31.4 | 63.56/41.33/53.93 | 44.68 | 68.95 | 56.34
medium | 76.59 | 30.7 | 63.11/40.75/53.58 | 43.83 | 69.73 | 55.86
large | 78.33 | 31.2 | 63.41/41.21/53.87 | 44.87 | 69.37 | 56.57
full | 78.18 | 30.9 | 62.95/40.95/53.26 | 44.45 | 71.78 | 55.81
✓ | sent-level | ✓ |  Model based | xsmall | 77.58 | 30.7 | 63.30/41.14/53.34 | 44.18 | 68.48 | 55.47
full | 75.39 | 29.6 | 62.43/40.33/52.95 | 42.45 | 69.75 | 54.26
✓ | token-level | ✗ |  Knowledge based | xsmall | 74.64 | 30.0 | 63.01/40.38/53.16 | 43.15 | 68.68 | 55.46
small | 75.74 | 30.8 | 63.98/41.92/54.39 | 44.35 | 66.81 | 55.75
medium | 75.54 | 30.5 | 63.41/41.42/53.93 | 44.36 | 67.47 | 55.60
large | 74.14 | 29.9 | 63.41/41.16/53.68 | 43.66 | 67.61 | 55.31
full | 74.94 | 29.2 | 63.23/40.73/53.64 | 42.62 | 67.89 | 54.33
✓ | token-level | ✗ |  Model based | xsmall | 75.39 | 30.3 | 63.27/40.96/53.66 | 43.39 | 67.40 | 55.16
full | 74.44 | 30.4 | 63.34/41.24/53.92 | 44.11 | 67.30 | 55.58
✓ | token-level | ✓ |  Knowledge based | xsmall | 75.79 | 30.7 | 63.52/41.02/53.75 | 43.83 | 68.09 | 56.19
small | 77.73 | 31.2 | 63.30/41.01/53.54 | 44.55 | 69.52 | 56.26
medium | 77.93 | 31.5 | 63.50/41.71/54.05 | 45.32 | 68.55 | 56.27
large | 76.78 | 30.9 | 62.67/40.76/53.41 | 43.93 | 71.47 | 55.59
full | 78.28 | 30.9 | 62.50/40.43/53.07 | 44.29 | 71.24 | 55.69
✓ | token-level | ✓ |  Model based | xsmall | 75.28 | 29.8 | 62.82/40.33/53.10 | 42.72 | 68.92 | 54.48
full | 75.19 | 29.2 | 61.88/39.49/52.11 | 41.68 | 72.00 | 54.05
✗ | - | ✓ |  Knowledge based | xsmall | 75.39 | 30.0 | 63.00/40.41/53.49 | 42.85 | 67.95 | 54.93
small | 77.93 | 31.1 | 63.74/41.68/54.26 | 44.59 | 68.43 | 56.11
medium | 77.58 | 30.8 | 63.34/41.08/53.70 | 43.87 | 68.56 | 55.77
large | 76.29 | 31.5 | 63.11/41.20/53.82 | 45.11 | 69.93 | 56.30
full | 77.68 | 31.0 | 63.17/40.96/53.53 | 44.20 | 70.97 | 56.24
✗ | - | ✓ |  Model based | xsmall | 74.34 | 30.2 | 63.06/40.56/53.41 | 43.62 | 68.65 | 55.22
full | 77.33 | 29.9 | 62.27/40.16/52.50 | 43.00 | 73.97 | 54.95

Table 8: Full R2D2 experiment results for FeTaQA.

Figure 7: Sentences for which our current perturbation methods do not apply.


# ¡Qué maravilla! Multimodal Sarcasm Detection in Spanish: a Dataset and a
Baseline

Khalid Alnajjar
Department of Digital Humanities
University of Helsinki
<EMAIL_ADDRESS>
Mika Hämäläinen
Department of Digital Humanities
University of Helsinki
<EMAIL_ADDRESS>

###### Abstract

We construct the first ever multimodal sarcasm dataset for Spanish. The
audiovisual dataset consists of sarcasm annotated text that is aligned with
video and audio. The dataset represents two varieties of Spanish, a Latin
American variety and a Peninsular Spanish variety, which ensures a wider
dialectal coverage for this global language. We present several models for
sarcasm detection that will serve as baselines in the future research. Our
results show that results with text only (89%) are worse than when combining
text with audio (91.9%). Finally, the best results are obtained when combining
all the modalities: text, audio and video (93.1%).

##  1 Introduction

Figurative language is one of the most difficult forms of natural language to
model computationally and there have been several studies in the past focusing
on its subcategories such as metaphor interpretation Xiao et al. (2016);
Hämäläinen and Alnajjar (2019a), humor generation Hämäläinen and Alnajjar
(2019b) and analyzing idioms Flor and Klebanov (2018). Sarcasm is one of the
extreme forms of figurative language, where the meaning of an utterance has
little to do with the surface meaning (see Kreuz and Glucksberg 1989).

Understanding sarcasm is difficult even for us humans as it requires certain
mental capacities such as a theory of mind (see Zhu and Wang 2020) and it is
very dependent on the context and speaker who is being sarcastic. There are
also very different view to sarcasm in the literature, for example, according
to Kumon-Nakamura et al. (1995) sarcasm requires an allusion to a failed
expectation and pragmatic insincerity (see Grice 1975) to be present in the
same time. However, Utsumi (1996) highlights that these two preconditions are
not enough, as sarcasm needs an ironic context to take place.

Haverkate (1990) argues that, in the context of sarcasm, the meaning
difference can either be the complete opposite of the semantic meaning of a
sentence or somewhat different as seen in the lexical opposition of the words
and the intended meaning. The fact that there are several different
theoretical ways of understanding sarcasm, highlights the complexity of the
phenomenon.

In this paper, we present an audio aligned dataset for sarcasm detection in
Spanish. The dataset containing text and video timestamps has been released
openly on Zenodo111Open access version of the data (contains text only)
https://zenodo.org/record/4701383. An access to the dataset with the video
clips222Access by request version of the data (videos and text)
https://zenodo.org/record/4707913 can be granted upon request for academic use
only. In addition, we will present a baseline model for this dataset to
conduct multimodal sarcasm detection in Spanish.

Speaker | Utterance | Translation | Sarcasm
---|---|---|---
Archer | No, Lana, para nada | No, Lana, not at all | true
Stan |  | Lo siento chicos, mi papá dice que está muy
---
ocupado con los Broncos, no tiene tiempo
| I am sorry guys, my dad says he is very
---
busy with the Broncos, he doesn’t have time
false
Lana | Decías algo acerca de un plan | You said something about a plan | true
Table 1: Example sentences from the dataset.

##  2 Related work

In this section, we will present some of the recent related work on sarcasm
detection. There has been some work also on sarcasm generation Chakrabarty et
al. (2020) and interpretation Peled and Reichart (2017), but they are rather
different as tasks and we will not discuss them in detail.

Badlani et al. (2019) show an approach for sarcasm detection in online
reivews. They train a CNN (convolutional neural network) based model on
separate feature embeddings for sarcasm, humor, sentiment and hate speech.
Similarly, Babanejad et al. (2020) also detect sarcasm in text. They combine
an LSTM (long short-term memory) model with BERT. Dubey et al. (2019) also
work on text only by detecting sarcastic numbers in tweets. They experiment
with rules, SVMs (support vector machines) and CNNs.

Cai et al. (2019) use an LSTM model to detect sarcasm in tweets. Their
approach is multimodal in the sense that it takes text and images into
account, but it does not deal with audio and video like our approach.

Castro et al. (2019) present a multimodal sarcasm dataset in English. The
dataset consists of annotated videos from TV sitcoms such as Friends and the
Big Bang Theory, apart from being in English instead of Spanish, the main
difference is that our dataset consists of animated cartoons instead of TV
shows played by real people. Another big difference is in the data collection
as they opted for querying sarcastic video clips, where as the data we work
with represents full episodes. Chauhan et al. (2020) use this data and present
a multimodal sarcasm detection framework based on a Bi-GRU model.

Many of the related work has been focusing on text only. Research on
multimodal approaches has been carried out only for English data, not unlike
the textual approaches.

##  3 Dataset

We base our work on the sarcasm annotated dataset from the MA thesis of the
second author of this paper Hämäläinen (2016)333Available on
https://www.kaggle.com/mikahama/the-best-sarcasm-annotated-dataset-in-spanish.
This dataset is based on two episodes of South Park with voice-overs in Latin-
American Spanish and two episodes of Archer with voice-overs in Spanish of
Spain. The dataset has the speaker, their utterance and sarcasm annotations
for each utterance in all of the episodes. However, the released data has been
released shuffled for copyright reasons and it contains text only. Unlike the
recent multimodal dataset for English Castro et al. (2019), this data is
expert annotated according to several different theories on sarcasm.

Annotation based on theories is important in order to establish a concrete
meaning for sarcasm and to avoid the same mistakes as Castro et al. (2019)
had. In their paper, they report that the most sarcastic character in The Big
Bang Theory is Sheldon, however this cannot be true as one of the main
characteristics of Sheldon is that he does not understand sarcasm. Therefore,
their annotations ignore the fundamentally important characteristic of
sarcasm, which is speaker intent, and rather they consider sarcasm purely
based on subjective intuition.

In order to produce a multimodal dataset out of the existing one, we locate
the corresponding videos for the annotations and manually align them with the
video. We use our own in-house tool JustAnnotate for this
task444https://mikakalevi.com/downloads/JustAnnotate.exe. This was a time
consuming task, but as a result we ended up with a high-quality dataset with
audio, video and text aligned. While aligning the dataset, we found several
errors in the original transcriptions that we fixed. We did not alter the
sarcasm annotations. In addition to the alignment, we introduced scene
annotations. An episode of a TV show consists of many different scenes, and
sarcasm is typically highly contextual, we indicate in the data which
utterances belong to the same scene to better capture the context of each
utterance.

Table 1 shows an example of the dataset. The English translation is provided
for convenience, but it is not included in the dataset itself. Each line is
aligned with audio and video. As we can see from these examples, sarcasm in
the dataset is very contextually dependent as the sarcastic sentences
presented in the table might equally well be sincere remarks if uttered by
someone else or in a different context.

Figure 1: Archer uttering a sarcastic sentence that goes against the common
sense

Figure 1 shows an example of a scene in the corpus. In this particular scene,
Archer asks sarcastically ¿Dónde se compra la leche materna? (Where does one
buy breast milk?). This is an example of sarcasm in the corpus where sarcasm
violates common sense. Depending on the speaker, the utterance might be
sarcastic or the speaker might lack knowledge on the topic.

Figure 2: Cartman uttering a sarcastic sentence that can be resolved only by
visual cues.

In Figure 2 Cartman comments on the neckpiece of Stan by saying Esas corbatas
están de moda, tiene suerte de tenerla (Those neckpieces are fashionable, you
are lucky to have one). This is an example of a very different type of sarcasm
that cannot be detected just by having common knowledge about the world. In
order to understand the sarcastic intent, a system would need to have an
access to the video as well to detect the unfashionable neckpiece and the
disappointed facial expression of Stan.

##  4 Method

In this section, we present our method for detecting sarcasm in the multimodal
dataset. We experiment with text only, text and audio and all modalities. All
models are trained by using the same random train (80%) and test (20%) splits.
For the neural model, 10% of the training split is used for validation.

###  4.1 Text only

For the text only model, we experiment with two models. In the first one, we
use an off the shelf OpenNMT Klein et al. (2017) model. We train the model
using a bi-directional long short-term memory (LSTM) based model Hochreiter
and Schmidhuber (1997) with the default settings except for the encoder where
we use a BRNN (bi-directional recurrent neural network) Schuster and Paliwal
(1997) instead of the default RNN (recurrent neural network). We use the
default of two layers for both the encoder and the decoder and the default
attention model, which is the general global attention presented by Luong et
al. (2015). The model is trained for the default 100,000 steps.

The second model is a Support Vector Machine (SVM) Schölkopf et al. (2000),
due to its efficiency when dealing with a high dimensional space and ability
to train a model with small data. We use the SVM implementation provided in
Scikit-learn Pedregosa et al. (2011). Following the work of Castro et al.
(2019), we use an RBF kernel and a scaled gamma. The regularization parameter
$C$ is set for 1000. This setup is followed in all of our SVM models.

Regarding the textual features of the SVM, we make use of GloVe Pennington et
al. (2014) embeddings555https://github.com/dccuchile/spanish-word-embeddings
trained on the Spanish Billion Words Corpus Cardellino (2019) and ELMo Peters
et al. (2018) embeddings provided by Che et al. (2018). Each textual instance
is tokenized using TokTok666https://github.com/jonsafari/tok-tok, and then a
sentence-level vector is constructed by computing the centroid (i.e., average
vector) of all tokens, for each word embeddings type. In the case of ELMo, the
vector of each token is the average of the last three layers of the neural
network. The input to the SVM model is the concatenation of the two types of
sentence embeddings.

###  4.2 Text and audio

This model is an SVM based model that extends the textual SVM model with audio
features. We do not extend the OpenNMT model with audio features as the
library does not provide us with audio and video inputters.

For all the audio, we set their sample size into 22 kHz to convert the data
into a manageable and consistent size. Thereafter, we extract different audio
features using librosa McFee et al. (2020). These features include short-time
Fourier transform Nawab and Quatieri (1987), mel-frequency cepstral
coefficients Stevens et al. (1937), chroma, Tonnetz Harte et al. (2006), zero-
crossing rate, spectral centroid and bandwidth, and pitches. In total, 13
features777We used the following methods from librosa: stft, mfcc,
chroma_stft, spectral_centroid, spectral_bandwidth, spectral_rolloff,
zero_crossing_rate, piptrack, onset_strength, melspectrogram,
spectral_contrast, tonnetz and harmonic were extracted. By combining all these
features, we get the audio vector.

###  4.3 All modalities

For videos, instead of trying to represent an entire video as a vector like
some of the existing approaches Hu et al. (2016) to video processing, we
extract 3 frames for each video corresponding to an utterance. We extract the
frames by dividing the frames of a video clip into three evenly sized chunks
and taking the first frame of each chunk. The key motivation behind this is
that we are working with animation, where most of the frames are static and
changes in between frames are not big. Therefore representing the entire video
clip is not important and it would only increase the complexity of the system.

We extract visual features from each of the three frames extracted using a
pre-trained ResNet-152 model He et al. (2016). Features are taken from the
last layer in the network, and the overall video vector is the sequence of the
three feature embeddings, in the same order. All the vectors described above
(i.e., textual, audio and visual vectors) are passed as input to the all-
modalities SVM model.

##  5 Results

In this section, we report the accuracy of predictions by the neural model and
the three SVM models that are based on 1) text only, 2) text and audio, and 3)
text, audio and video. The results can be seen in Table 2.

Input | Accuracy
---|---
Neural Model
Text | 87.5%
SVM
Text | 89.0%
Text + Audio | 91.9%
Text + Audio + Video | 93.1%
Table 2: Accuracies of the predictions by all models for the sarcasm detection
task.

As we can see in the results, having more modalities in the training improved
the results. The audio features were able to capture more features important
for sarcasm than pure text. Having all the three modalities at the same time
gave the best results, with a 4.1% gain in the accuracy from the text-based
model. The neural model reached to the lowest accuracy, most likely due to the
fact that it was not trained with pretrained embeddings, a source of
information that was available to the SVM models.

###  5.1 Error analysis

When we look at the predictions by the model best model (text + audio +
video), we can see that the sarcasm detection is not at all an easy task.

An interesting example of a correctly predicted sarcastic utterance is Lucen
bien muchachos. ¡A patear culos! (You look great, guys. Let’s kick some ass!).
This is an example of a visually interpretable sarcasm where the kids the
sentence was uttered to looked all ridiculous. This would seem, at first, to
highlight that the model has learned something important based on the visual
features. However, we can see that this is not at all the case as the model
predicts incorrectly the following sarcastic utterance: Sí Stan, es lo que
quiere la gente. No te preocupes, luces genial. (Yes Stan, that is what the
people want. Don’t worry, you look great.) The context is similar to the one
where the model predicted the sarcasm correctly, which means that the visual
features are not representative enough for the model to correctly annotate
detect this sarcastic utterance.

Interestingly, the model predicted Sí amigo, es una réplica de la corbata del
Rey Enrique V (Yes friend, it is a replica of the neckpiece of the King Henry
V) as sarcastic while in fact the uttrance was not sarcastic. This utterance
refers to the same neckpiece as seen in Figure 2. The neckpiece appeared
frequently in sarcastic contexts, so the model overgeneralized that anything
said about the neckpiece must be sarcastic.

##  6 Conclusions

We have presented the first multimodal dataset for detecting sarcasm in
Spanish. The dataset has been released on Zenodo. Our initial results serve as
a baseline for any future work on sarcasm detection on this dataset.

Based on the results, it is clear that multimodality aids in detecting sarcasm
as more contextual information is exposed to the model. Despite the
improvements when considering multiple modalities, sarcasm detection is a very
difficult task to model as it demands a global understanding of the world and
the specific context the sarcastic utterance is in, as discussed in our error
analysis. Even though the overall accuracy is high, it is clear the model
makes errors that indicate that it has learned the data, but not the
phenomenon.

## References

  * Babanejad et al. (2020) Nastaran Babanejad, Heidar Davoudi, Aijun An, and Manos Papagelis. 2020.  Affective and contextual embedding for sarcasm detection.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 225–243, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Badlani et al. (2019) Rohan Badlani, Nishit Asnani, and Manan Rai. 2019.  An ensemble of humour, sarcasm, and hate speechfor sentiment classification in online reviews.  In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_ , pages 337–345, Hong Kong, China. Association for Computational Linguistics.
  * Cai et al. (2019) Yitao Cai, Huiyu Cai, and Xiaojun Wan. 2019.  Multi-modal sarcasm detection in Twitter with hierarchical fusion model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2506–2515, Florence, Italy. Association for Computational Linguistics.
  * Cardellino (2019) Cristian Cardellino. 2019.  Spanish Billion Words Corpus and Embeddings.
  * Castro et al. (2019) Santiago Castro, Devamanyu Hazarika, Verónica Pérez-Rosas, Roger Zimmermann, Rada Mihalcea, and Soujanya Poria. 2019.  Towards multimodal sarcasm detection (an _obviously_ perfect paper).  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4619–4629.
  * Chakrabarty et al. (2020) Tuhin Chakrabarty, Debanjan Ghosh, Smaranda Muresan, and Nanyun Peng. 2020.  R^3: Reverse, retrieve, and rank for sarcasm generation with commonsense knowledge.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7976–7986, Online. Association for Computational Linguistics.
  * Chauhan et al. (2020) Dushyant Singh Chauhan, Dhanush S R, Asif Ekbal, and Pushpak Bhattacharyya. 2020\.  Sentiment and emotion help sarcasm? a multi-task learning framework for multi-modal sarcasm, sentiment and emotion analysis.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4351–4360, Online. Association for Computational Linguistics.
  * Che et al. (2018) Wanxiang Che, Yijia Liu, Yuxuan Wang, Bo Zheng, and Ting Liu. 2018.  Towards better UD parsing: Deep contextualized word embeddings, ensemble, and treebank concatenation.  In _Proceedings of the CoNLL 2018 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies_ , pages 55–64, Brussels, Belgium. Association for Computational Linguistics.
  * Dubey et al. (2019) Abhijeet Dubey, Lakshya Kumar, Arpan Somani, Aditya Joshi, and Pushpak Bhattacharyya. 2019.  “when numbers matter!”: Detecting sarcasm in numerical portions of text.  In _Proceedings of the Tenth Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 72–80, Minneapolis, USA. Association for Computational Linguistics.
  * Flor and Klebanov (2018) Michael Flor and Beata Beigman Klebanov. 2018.  Catching idiomatic expressions in efl essays.  In _Proceedings of the Workshop on Figurative Language Processing_ , pages 34–44.
  * Grice (1975) H Paul Grice. 1975.  Logic and conversation.  _1975_ , pages 41–58.
  * Hämäläinen (2016) Mika Hämäläinen. 2016.  Reconocimiento automático del sarcasmo - ¡Esto va a funcionar bien! Master’s thesis, University of Helsinki, Finland.  URN:NBN:fi:hulib-201606011945.
  * Hämäläinen and Alnajjar (2019a) Mika Hämäläinen and Khalid Alnajjar. 2019a.  Let’s FACE it. Finnish poetry generation with aesthetics and framing.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 290–300, Tokyo, Japan. Association for Computational Linguistics.
  * Hämäläinen and Alnajjar (2019b) Mika Hämäläinen and Khalid Alnajjar. 2019b.  Modelling the socialization of creative agents in a master-apprentice setting: The case of movie title puns.  In _Proceedings of the 10th International Conference on Computational Creativity_. Association for Computational Creativity.
  * Harte et al. (2006) Christopher Harte, Mark Sandler, and Martin Gasser. 2006.  Detecting harmonic change in musical audio.  In _Proceedings of the 1st ACM workshop on Audio and music computing multimedia_ , pages 21–26.
  * Haverkate (1990) Henk Haverkate. 1990.  A speech act analysis of irony.  _Journal of pragmatics_ , 14(1):77–109.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun. 2016.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hu et al. (2016) Sheng-Hung Hu, Yikang Li, and Baoxin Li. 2016.  Video2vec: Learning semantic spatio-temporal embeddings for video representation.  In _2016 23rd International Conference on Pattern Recognition (ICPR)_ , pages 811–816. IEEE.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017\.  OpenNMT: Open-Source Toolkit for Neural Machine Translation.  In _Proc. ACL_.
  * Kreuz and Glucksberg (1989) Roger J Kreuz and Sam Glucksberg. 1989.  How to be sarcastic: The echoic reminder theory of verbal irony.  _Journal of experimental psychology: General_ , 118(4):374.
  * Kumon-Nakamura et al. (1995) Sachi Kumon-Nakamura, Sam Glucksberg, and Mary Brown. 1995.  How about another piece of pie: The allusional pretense theory of discourse irony.  _Journal of Experimental Psychology: General_ , 124(1):3.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015.  Effective approaches to attention-based neural machine translation.  _arXiv preprint arXiv:1508.04025_.
  * McFee et al. (2020) Brian McFee, Vincent Lostanlen, Alexandros Metsai, Matt McVicar, Stefan Balke, Carl Thomé, Colin Raffel, Frank Zalkow, Ayoub Malek, Dana, Kyungyun Lee, Oriol Nieto, Jack Mason, Dan Ellis, Eric Battenberg, Scott Seyfarth, Ryuichi Yamamoto, Keunwoo Choi, viktorandreevichmorozov, Josh Moore, Rachel Bittner, Shunsuke Hidaka, Ziyao Wei, nullmightybofo, Darío Hereñú, Fabian-Robert Stöter, Pius Friesch, Adam Weiss, Matt Vollrath, and Taewoon Kim. 2020.  librosa/librosa: 0.8.0.
  * Nawab and Quatieri (1987) S. Hamid Nawab and Thomas F. Quatieri. 1987.  _Short-Time Fourier Transform_ , page 289–337. Prentice-Hall, Inc., USA.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Peled and Reichart (2017) Lotem Peled and Roi Reichart. 2017.  Sarcasm SIGN: Interpreting sarcasm with sentiment based monolingual machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1690–1700, Vancouver, Canada. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proc. of NAACL_.
  * Schölkopf et al. (2000) Bernhard Schölkopf, Alex J. Smola, Robert C. Williamson, and Peter L. Bartlett. 2000.  New support vector algorithms.  _Neural Comput._ , 12(5):1207–1245.
  * Schuster and Paliwal (1997) Mike Schuster and Kuldip K Paliwal. 1997.  Bidirectional recurrent neural networks.  _IEEE transactions on Signal Processing_ , 45(11):2673–2681.
  * Stevens et al. (1937) Stanley Smith Stevens, John Volkmann, and Edwin Broomell Newman. 1937.  A scale for the measurement of the psychological magnitude pitch.  _The journal of the acoustical society of america_ , 8(3):185–190.
  * Utsumi (1996) Akira Utsumi. 1996.  A unified theory of irony and its computational formalization.  In _COLING 1996 Volume 2: The 16th International Conference on Computational Linguistics_.
  * Xiao et al. (2016) Ping Xiao, Khalid Alnajjar, Mark Granroth-Wilding, Kat Agres, Hannu Toivonen, et al. 2016.  Meta4meaning: Automatic metaphor interpretation using corpus-derived word associations.  In _Proceedings of the Seventh International Conference on Computational Creativity_. Sony CSL Paris.
  * Zhu and Wang (2020) Ning Zhu and Zhenlin Wang. 2020.  The paradox of sarcasm: Theory of mind and sarcasm use in adults.  _Personality and Individual Differences_ , 163:110035.