#  Donkii: Can Annotation Error Detection Methods Find Errors in Instruction-
Tuning Datasets?

Leon Weber-Genzel Robert Litschko Ekaterina Artemova
Barbara Plank
MaiNLP, Center for Information and Language Processing, LMU Munich, Germany
Munich Center for Machine Learning (MCML), Munich, Germany
Department of Computer Science, IT University of Copenhagen, Denmark
{leonweber, robert.litschko, ekaterina.artemova<EMAIL_ADDRESS>
###### Abstract

Instruction-tuning has become an integral part of training pipelines for Large
Language Models (LLMs) and has been shown to yield strong performance gains.
In an orthogonal line of research, Annotation Error Detection (AED) has
emerged as a tool for detecting quality issues of gold-standard labels. But so
far, the application of AED methods is limited to discriminative settings. It
is an open question how well AED methods generalize to generative settings
which are becoming widespread via generative LLMs. In this work, we present a
first and new benchmark for AED on instruction-tuning data: Donkii. It
encompasses three instruction-tuning datasets enriched with annotations by
experts and semi-automatic methods. We find that all three datasets contain
clear-cut errors that sometimes directly propagate into instruction-tuned
LLMs. We propose four AED baselines for the generative setting and evaluate
them comprehensively on the newly introduced dataset. Our results demonstrate
that choosing the right AED method and model size is indeed crucial, thereby
deriving practical recommendations. To gain insights, we provide a first case-
study to examine how the quality of the instruction-tuning datasets influences
downstream performance.

##  1 Introduction

Recent success in instruction tuning (InstT) has shown to enable Large
Language Models (LLMs) to generalise to a wide range of tasks in the zero-shot
setting (Wei et al., 2022; Sanh et al., 2022; Ouyang et al., 2022). InstT
achieves this by training an LLM on instruction-output pairs, where the
instruction describes the task and the output contains the expected solution
to the task. After fine-tuning on the InstT dataset and an optional
reinforcement learning phase (Ouyang et al., 2022), LLMs are able to
generalize to instructions unseen during fine-tuning. In an orthogonal line of
inquiry, researchers have studied Annotation Error Detection (AED) that allows
to detect erroneous annotations in labelled datasets. These low quality
instances are then corrected or removed in a semi-automatic process (Vlachos,
2006; Klie et al., 2022; Weber and Plank, 2023). In this work, we combine both
strands of research and ask whether AED methods can help to detect errors in
InstT datasets and thus help to improve model quality by improving data
quality.

Figure 1: The three Donkii datasets help to design AED methods that can clean
InstT datasets.

Applying AED methods to InstT datasets poses a series of challenges. (1)
Systematically developing and comparing AED methods requires datasets with
annotations indicating which instances contain annotation errors. Such
datatsets are not yet available for InstT. (2) To the best of our knowledge,
so far researchers have applied AED methods only in the discriminative setting
(Klie et al., 2022) and it is not immediately clear how to adapt existing
methods to generative problems. (3) It is not obvious what even constitutes an
error in InstT or what the exact impact of data quality on model quality is.

In this work, we address these three challenges; see also Figure 1 for an
illustration of our contributions.111Data and code will be made available
after publication. (1) We present Donkii, the first instruction tuning dataset
to enable evaluation of AED methods. Donkii contains annotations of errors on
top of three existing InstT datasets derived from manual error annotation
efforts. Further, we introduce a hierarchy of error types for InstT datasets;
see Figure 2. (2) We derive four AED baselines for generative problems derived
from recent work on training dynamics for AED (Swayamdipta et al., 2020;
Pleiss et al., 2020). (3) We use Donkii to evaluate the proposed AED baselines
and study the effects of model size, different types of errors, and different
types of InstT data. The results surprisingly show that there exists a clear
best performing AED method for InstT data among the evaluated four. In a
second experiment, we perform a case study on the impact of InstT data quality
on downstream performance. Here, we find that for our setup the errors have
only a small quantitative effect but observe qualitatively that they can
directly translate into unwanted model behaviour.

##  2 Background

###  2.1 Instruction Tuning

Instruction tuning (InstT) is an emerging paradigm that leverages natural
language instructions to fine-tune language models and thus improve zero-shot
performance on unseen tasks (Sanh et al., 2022; Ouyang et al., 2022; Wei et
al., 2022; Wang et al., 2022b, inter alia). In InstT, an LLM is fine-tuned to
produce a desired output given an instruction text. In some datasets, the
instruction is further divided into a definition or prompt component that
defines the task and an optional input component (Wang et al., 2022b; Taori et
al., 2023). In this work, we distinguish three types of InstT datasets based
on their provenance: meta-datasets, human-authored datasets and LLM-authored
datasets.

The first InstT datasets were meta-datasets that converted existing NLP
datasets into InstT data with human-authored prompt templates (Khashabi et
al., 2020; Ye et al., 2021; Mishra et al., 2022; Sanh et al., 2022; Wei et
al., 2022). Researchers typically construct them by writing one or more prompt
templates for an existing NLP dataset. This template is subsequently used to
transform every instance of the existing dataset into an InstT instance. Here,
we call the combination of existing dataset and prompt template a task. For
Human-authored InstT datasets on the other hand, the dataset creators ask
human annotators to author InstT instances (Ouyang et al., 2022; Dat, 2023) or
mine InstT instances from existing human-authored resources such as forums and
wikis (Zhou et al., 2023). LLM-authored datasets instead are generated by
LLMs. Dataset creators typically achieve this by prompting the LLM with a few
examples of how InstT instances look like and instructing the model to come up
with new ones (Wang et al., 2022a; Honovich et al., 2022; Taori et al., 2023)
or by providing elaborate rules of what properties InstT instances should
possess (Bai et al., 2022; Sun et al., 2023).

Finally, dataset creators have proposed mixtures of these approaches, e.g. by
manually correcting LLM-authored instances (Ruebsamen and Contributors, 2023)
or by combining instances generated by different approaches (Zhou et al.,
2023). Some of these works highlight the importance of high quality data (Zhou
et al., 2023; Ruebsamen and Contributors, 2023), but to the best of our
knowledge, our study is the first to evaluate AED techniques systematically.

###  2.2 Annotation Error Detection

AED for Natural Language Processing (NLP) datasets has a long tradition which
Klie et al. (2022) recently surveyed comprehensively. Existing AED methods can
be divided into six different categories (Klie et al., 2022): variation-based
methods exploit the observation that instances with similar surface forms tend
to have the same label (Dickinson and Meurers, 2003; Larson et al., 2020).
Model-based methods use a cross-validation scheme to produce predictions for
the whole dataset and then operate on these predictions (Amiri et al., 2018;
Yaghoub-Zadeh-Fard et al., 2019). Training-dynamics-based approaches calculate
statistics over quantities collected during training (Swayamdipta et al.,
2020; Pleiss et al., 2020; Siddiqui et al., 2022). Vector-space-proximity-
based methods assume that instances that are close in a suitable vector space
should have the same label (Larson et al., 2019; Grivas et al., 2020).
Ensembling-based methods use statistics of the predictions of ensemble members
to find errors (Alt et al., 2020; Varshney et al., 2022) and rule-based
approaches rely on manually defined rules to spot erroneous instances (Kvĕtoň
and Oliva, 2002). In this work, we focus on training dynamics because they
performed well in a recent evaluation (Klie et al., 2022) and have a low
compute overhead. We leave the evaluation of other types of methods for future
work.

An orthogonal classification of AED methods is into flaggers and scorers (Klie
et al., 2022). Flaggers model AED as a binary classification task (error vs
non-error) and scorers assign an error score to each instance which reflects
how likely the instance is to contain an annotation error. In this work, we
focus on scorers, because the ranking induced by them allows more fine-
granular decisions on which instances to inspect (Weber and Plank, 2023) and
they can be converted to flaggers by choosing a suitable threshold
(Swayamdipta et al., 2020).

##  3 Proposed AED baselines for text generation datasets

We introduce four AED baselines for text generation datasets. For this, we
adapt methods based on training-dynamics that were previously used for AED in
classification problems (Swayamdipta et al., 2020; Pleiss et al., 2020). All
four methods assign an error score to an instance, where a higher score
ideally reflects a higher probability of an erroneous annotations. All scores
use the probabilities $p_{e,l}$ that the model assigned to the token $l$ of
the output sequence of the instance at epoch $e$ during training. We propose
the following measures: (1) Perplexity, which is the epoch-averaged perplexity
of the instance based on $p_{e,l}$:

$\text{PPL}=\frac{1}{E}\sum_{e=1}^{E}\text{ppl}_{e},$ (1)

where $E$ is the number of epochs and $\text{ppl}_{e}$ the perplexity at epoch
$e$. (2) The (negative) average probability, determined by averaging
$p_{e,l}$:

$P_{\mu}=-\frac{1}{E}\sum_{e=1}^{E}\frac{1}{L}\sum_{l=1}^{L}p_{e,l},$ (2)

where $L$ is the number of tokens in the output sequence.

(3) The (negative) minimum probability, derived from the minimum of $p_{e,l}$:

$P_{\text{min}}=-\frac{1}{E}\sum_{e=1}^{E}\min_{l=1}^{L}p_{e,l}.$ (3)

(4) The (negative) Area-under-the-Margin score (AUM) (Pleiss et al., 2020),
which we adapt to the generative setting by computing it for each token in the
output sequence and averaging the resulting scores:

$\text{AUM}=\frac{1}{E}\sum_{e=1}^{E}\frac{1}{L}\sum_{l=1}^{L}\max_{y_{l}^{\prime}\neq
y_{l}}p_{e}(y_{l}^{\prime}|x_{l})-p_{e,l},$ (4)

where $y_{l}$ is the annotated label for token $l$ and
$\max_{y_{l}^{\prime}\neq y_{l}}p_{e}(y_{l}^{\prime}|x_{l})$ is the maximum
probability assigned by the model at epoch $e$ for token $l$ excluding the
assigned label. Additionally, we consider a variant of each score that uses
only the final epoch; see §5.2 for the results of this ablation.

##  4 Datasets

We describe three different instruction-tuning datasets which we have enriched
with annotations of erroneous instances: P3-Donkii, SNI-Donkii, and ADC-
Donkii.222See Appendix A for our data statement. Each of these builds on an
existing InstT dataset and on manual inspection of the errors in the
respective dataset: P3-Donkii is derived from the meta-dataset Public Pool of
Prompts (Sanh et al., 2022), SNI-Donkii from the meta-dataset Super-Natural
Instructions (Wang et al., 2022b), and ADC-Donkii from the LLM-authored
dataset Alpaca (Taori et al., 2023) and its partially corrected version
AlpacaDatasetCleaned (Ruebsamen and Contributors, 2023). We enrich each of
these datasets with labels of which instances contain errors by using varying
mixtures of expert annotations and programmatic analysis of the source data.
For each dataset, we construct three different sets of instances:
$\mathcal{X}^{*}$ for which we assume that it does not contain errors,
$\mathcal{X}_{\text{err}}$, for which we assume that it contains errors, and
$\mathcal{X}_{\text{unk}}$, for which we do not know whether it contains
errors. We require $\mathcal{X}_{\text{unk}}$ for an accurate evaluation
because we do not exhaustively annotate the datasets with errors due to their
sheer size and resource availability; see §5.1 for details.

Source data $|\mathcal{X}_{\text{unk}}|$ $|\mathcal{X}^{*}|$
$|\mathcal{X}_{\text{err}}|$ $|\mathcal{T}|$ $|\mathcal{T}_{\text{err}}|$
$\bar{L}_{\text{inp}}$ $\bar{L}_{\text{out}}$ Err Prov P3 Sanh et al. (2022)
399,472 12,237 12,237 417 20 118 9 Syn. Meta SNI Wang et al. (2022b) 101,783
1,088 585 1,613 17 165 6 Nat. Meta ADC Taori et al. (2023) (Ruebsamen and
Contributors, 2023) 48,425 173 146 - - 15 44 Nat LLM

Table 1: Statistics for the three Donkii datasets. $|\mathcal{T}|$ denotes the
total number of tasks, and $|\mathcal{T}_{\text{err}}|$ the number of tasks
with at least one instance with an error. Note, that ADC does not provide a
grouping of instances into tasks.
$\bar{L}_{\text{inp}}$/$\bar{L}_{\text{out}}$ denotes the average input/output
length in white-space-delimited tokens. ‘Err’ is the type of error (synthetic
or naturally ocurring) and ‘Prov’ the provenance (meta-dataset vs LLM-
authored).

Error Source Instruction Input Output (shortened) Requires web browsing Alpaca
Summarize the New York Times article about the rise of AI.
www.nytimes.com/2020/06/16/ opinion/ai-growth.html The New York Times article
titled “The Rise of Artificial Intelligence” looks at the growing impact of AI
on the world today. <…> Bad math Alpaca Calculate the mean of the numbers 3,
4, 6, 10, 11 The mean of the numbers 3, 4, 6, 10, 11 is 6.4. This is
calculated by adding the numbers together and then dividing by the number of
values (5). (3 + 4 + 6 + 10 + 11 = 34; 34 / 5 = 6.4). Unclear instruction
Alpaca Sort the following 10 fruits into three categories. Bananas, Apples,
Pineapples, Plums, Cherries, Peaches, Grapes, Watermelons, Mangoes, Oranges
Category 1: Bananas, Apples, Pineapples, Plums, Cherries;

Table 2: Examples of two error categories.

###  4.1 P3-Donkii

Public Pool of Prompts (P3) (Sanh et al., 2022) is a meta-dataset for InstT
which was created by asking researchers and open-source contributors to
transform existing datasets by writing prompts with the InstT-templating
engine promptsource (Bach et al., 2022). We construct the P3-Donkii dataset by
introducing different types of synthetic errors into P3 data. We use this
synthetic setup for P3 so that we have full control over the the number and
types of errors in the dataset. To find realistic error classes, we first
detect existing errors in the P3 data. We use PPL to assign error scores to
tasks in P3, employing both mean and median aggregation (see §5.2 for details
on this).333In principle, this semi-automatic process of finding error
categories with PPL could bias our evaluation results. However, our purely
manual analysis of SNI reveals similar error categories and thus, we are
confident that the introduced bias is minimal. Then we manually inspect the
top 20 highest scoring tasks by looking at their top-scoring instances. In our
manual inspection, we found the following types of problems:

Empty output: The output is an empty string where it should not be.

Incorrect output: The output contains severe orthographic or factual errors.

Missing context information: The prompt is truncated during preprocessing.
This can make crucial information unavailable, e.g. missing context in
extractive QA.

We then correct the errors that we detected in the 20 tasks in P3. We rebuild
the empty-output data from scratch using promptsource and verify that the
output strings are non-empty. We remove tasks that contain a high number of
low-quality outputs. We discard instances that do not fit the set maximum
length, so that this cannot lead to missing information.

Finally, we reintroduce the detected errors in a controlled fashion by
modelling them synthetically. For each type of error, we randomly sample five
tasks and perturb their instances:

Empty output: We replace the output in all instances for the task with an
empty string.

Low quality output: For each instance of the task, with a probability of
$0.5$, we replace the gold-standard output with output generated by prompting
Llama-7b (Touvron et al., 2023).

Missing information: For each instance of the task, with a probability of
$0.5$, we truncate the gold-standard prompt to half of its original length.

Flipped output: For each instance, with a probability of $0.5$, we replace the
output with the output of another instance. This is a widely used perturbation
used in AED research (Klie et al., 2022), which we adapted to InstT datasets.

We collect all perturbed instances in $\mathcal{X}_{\text{error}}$ and those
of the same original unperturbed task in $\mathcal{X}^{*}$, the instances of
unperturbed tasks constitute $\mathcal{X}_{\text{unk}}$.

###  4.2 SNI-Donkii

We construct SNI-Donkii by mining errors that arose during creation of the
Super-natural instructions (SNI) (Wang et al., 2022b) dataset and were
corrected for the current version of SNI.
SNI444https://github.com/allenai/natural-instructions is a meta-dataset for
InstT that a large number of researchers created by transforming existing
datasets into InstT tasks. It contains in total 1,616 tasks spanning a large
variety of NLP tasks including for instance Question Answering, Text
Classification, Sentiment Analysis, Textual Entailment, and Summarization. The
project implemented quality control through peer-review conducted via
GitHub555https://github.com/allenai/natural-instructions/commits/master and a
crowd-sourced evaluation.

We create SNI-Donkii by comparing the version of each task before the final
round of peer review (the first version uploaded to GitHub) and after peer
review (the version on GitHub at the time of writing). From the 1,613 tasks
that we could download without error, we collect all 455 tasks
$t\in\mathcal{T}$ for which the output of at least one instance changed. For
17 of these tasks, all expert annotators (two co-authors and one NLP MSc
student) agreed666We opt for a roundtable discussion instead of aggregating
via majority vote, because we found that annotating errors in InstT datasets
is a hard task. Even though we exclusively relied on expert annotators, they
sometimes missed crucial detail about instances and revised their annotations
during the discussion. A downside of this discusison-based setup is that we
cannot reliably estimate inter-annotator agreement. that at least 90% of the
changed instances contain an error. See Table 5 for an example of the
annotation task and Table 2 for examples of found errors. For the annotation
guidelines see Appendix B.

From this annotation, we construct SNI-Donkii as follows: For all non-
erroneous tasks, we add 64 instances from the newest version – or less if the
task contains fewer than 64 instances – of it to $\mathcal{X}_{\text{unk}}$.
For the erroneous tasks, we add 64 changed instances from the oldest version
to $\mathcal{X}_{\text{err}}$ and 64 from the newest version to
$\mathcal{X}^{*}$.777We chose $64$ instances because Wang et al. (2022b) find
that performance plateaus with more instances per task. When we had fewer than
64 updated instances, we put them all into $\mathcal{X}_{\text{err}}$. In this
case, we filled up $\mathcal{X}_{\text{unk}}$ with extra instances from the
oldest version to keep the number of instances about the same for each task.
Table 1 gives the statistics of the resulting dataset.

Figure 2: The Donkii taxonomy of six error categories, four of which are
further divided into more specific subcategories.

###  4.3 ADC-Donkii

Alpaca (Taori et al., 2023) is an LLM-authored dataset, which the authors
constructed by following the self-instruct recipe proposed by Wang et al.
(2022a). The creators of Alpaca repeatedly prompted text-
davinci-003888https://platform.openai.com/docs/models with in-context examples
of InstT instances sampled from a pool of human and LLM-authored instances and
asked the LLM to provide a new instance. This yielded a dataset of 52,000
different InstT instances. In a separate effort called AlpacaDataCleaned (ADC)
(Ruebsamen and Contributors, 2023), members of the open-source community
corrected errors in the Alpaca data using a mixture of manual and rule-based
annotation.999https://github.com/gururise/AlpacaDataCleaned/issues/31 To
construct ADC-Donkii, we collect 300 instances from Alpaca that do not occur
in ADC and pair each of them with the instance with the closest BM25 score
from ADC. Using these pairs, three authors of this study manually annotate
whether one of the two instances is clearly preferable because the other
contains at least one error. The annotation guidelines can be found in
Appendix C. As for SNI, we do not disclose which of the instances comes from
Alpaca and which from ADC to avoid introducing unnecessary bias. If all three
annotators agree after a roundtable discussion, that one of the two instances
is preferable, we add it to $\mathcal{X}^{*}$ and the other to
$\mathcal{X}_{\text{err}}$. We add all other instances from Alpaca and ADC to
$\mathcal{X}_{\text{unk}}$. Table 1 shows the statistics for the resulting
dataset.

###  4.4 Error categories

During annotation, we identified six main categories of errors, each with
several subcategories. The proposed hierarchy is displayed in Figure 2; a
sample from Donkii errors is presented in Table 2. More examples for each
category can be found in the Appendix D. The error categories are the
following:

Incorrect output: Issues are observed in the outputs. These may include
providing inaccurate or wrong outputs, such as providing a three-letter
abbreviation when a two-letter abbreviation was requested. Other issues in
this category include displaying no output at all, flipping the label to the
opposite in binary tasks in SNI-Donkii, providing output that is in the wrong
output space, such as responding with a/b/c/d in a multiple-choice question
when the options are listed with 1/2/3/4. Additionally, the output may be an
off-instruction response that is related to the instruction but does not
follow it, such as responding with a code sample that can compute an average
instead of outputting the mean of the given numbers as requested. Finally, the
output may include ungrammatical text.

Factual knowledge and mathematics: This category accounts for outputs that may
be time-specific, contain factual errors, or contain incorrect arithmetics.

Noise: Instances in which the instruction, input, or output contains some form
of noise. This noise may range from NoInput stubs to duplicating the
instruction in the output.

Underspecified input: Instances in which the instruction and input do not
provide sufficient information to complete the task. For instance, a task may
request finding the mean value of a set of numbers without providing the
actual numbers. Additionally, this category includes cases where the
instruction is unclear and cannot be completed correctly due to a lack of
specification. For example, a task may require classifying data points into
multiple categories without explicitly describing the semantics of the
categories. We argue that these instances represent errors, because the LLM
should rather ask the user for the necessary input than to assume (i.e.
‘hallucinate’) input.

Model-modality mismatch: Instances that require additional modalities are
placed into this category since we assume that current instruction datasets
are used for text-only LLMs. Unsupported modalities may include tables, images
(as in Table 5 in the appendix, and the use of additional tools to browse and
retrieve information from the internet.

Formatting: Instances with corrupted formatting, such as the absence of white
spaces and the use of punctuation instead of white spaces.

In general, the identified errors closely resemble the known errors made by
models for open-ended text generation Dou et al. (2022); Ge et al. (2022),
with less emphasis on issues related to syntax and word choice.

##  5 How well does AED do in Instruction Tuning data?

In this section, we provide baselines results for detecting errors in InstT
datasets. For this, we evaluate the baselines introduced in §3 using the
Donkii datasets proposed in §4.

small base large xl rand PPL $P_{\mu}$ $P_{\textit{min}}$ AUM PPL $P_{\mu}$
$P_{\textit{min}}$ AUM PPL $P_{\mu}$ $P_{\textit{min}}$ AUM PPL $P_{\mu}$
$P_{\textit{min}}$ AUM P3 $50.0$ $73.6_{\textit{4.4}}$
$\bm{84.3_{\textit{1.0}}}$ $51.3_{\textit{0.8}}$ $52.7_{\textit{0.7}}$
$76.1_{\textit{0.6}}$ $81.7_{\textit{0.3}}$ $51.2_{\textit{0.0}}$
$53.6_{\textit{0.1}}$ $70.7_{\textit{3.5}}$ $77.4_{\textit{0.4}}$
$49.4_{\textit{0.6}}$ $52.2_{\textit{0.2}}$ $61.3_{\textit{0.0}}$
$66.0_{\textit{0.0}}$ $58.6_{\textit{0.0}}$ $46.0_{\textit{0.0}}$ SNI $34.9$
$30.7_{\textit{0.2}}$ $30.3_{\textit{0.2}}$ $27.9_{\textit{0.1}}$
$28.7_{\textit{0.1}}$ $31.8_{\textit{0.4}}$ $42.4_{\textit{0.9}}$
$30.2_{\textit{0.3}}$ $29.8_{\textit{0.1}}$ $33.4_{\textit{1.0}}$
$\bm{48.1_{\textit{1.7}}}$ $34.5_{\textit{0.9}}$ $34.1_{\textit{0.6}}$
$33.0_{\textit{0.7}}$ $38.9_{\textit{0.2}}$ $32.2_{\textit{0.2}}$
$31.6_{\textit{0.4}}$ ADC $45.1$ $54.5_{\textit{0.3}}$
$\bm{55.4_{\textit{0.3}}}$ $50.3_{\textit{0.5}}$ $47.9_{\textit{0.7}}$
$53.7_{\textit{0.21}}$ $51.9_{\textit{0.5}}$ $50.8_{\textit{0.7}}$
$51.4_{\textit{0.7}}$ $52.2_{\textit{0.4}}$ $48.9_{\textit{0.2}}$
$47.5_{\textit{0.5}}$ $47.2_{\textit{1.2}}$ $53.1_{\textit{0.0}}$
$53.1_{\textit{0.0}}$ $51.3_{\textit{0.0}}$ $47.3_{\textit{0.0}}$ AVG $43.3$
$52.9$ $56.7$ $43.2$ $43.1$ $53.9$ $\bm{58.7}$ $44.1$ $44.9$ $52.1$ $58.1$
$43.8$ $44.5$ $49.1$ $52.7$ $47.4$ $41.6$

Table 3: Results of four different AED methods applied to the Donkii datasets.
All scores are Average Precision in percent. The larger number is the mean
across three seeds and the smaller number the standard deviation. The best
result per dataset is in bold. Rand is the random baseline.

###  5.1 Evaluation protocol

We follow the evaluation protocol for scoring-based AED methods for
discriminative setups of Klie et al. (2022) and Chong et al. (2022) – albeit
with one modification. We follow the protocol by treating the problem as a
ranking task in which an AED model assigns an error score to each instance
$x_{i}$. However, different from Klie et al. (2022) and Chong et al. (2022),
we have three sets of instances instead of two: $\mathcal{X}^{*}$, which we
expect to contain few to no errors, $\mathcal{X}_{\text{err}}$ which we expect
to contain many errors and $\mathcal{X}_{\text{unk}}$ for which we do not know
the proportion of errors. We judge the quality of the ranking by how well it
discriminates between $\mathcal{X}^{*}$ and $\mathcal{X}_{\text{err}}$ and
ignore $\mathcal{X}_{\text{unk}}$ during evaluation. Note that while we use
only $\mathcal{X}^{*}$ and $\mathcal{X}_{\text{err}}$ for evaluation, we train
on
$x_{i}\in\mathcal{X}^{*}\cup\mathcal{X}_{\text{err}}\cup\mathcal{X}_{\text{unk}}$.
We use average precision (AP), i.e. the area under the precision-recall curve,
implemented with scikit-learn (Pedregosa et al., 2011) to score the rankings
and use
$\frac{|\mathcal{X}_{\text{err}}|}{|\mathcal{X}^{*}|+|\mathcal{X}_{\text{err}}|}$
as an estimator for the random baseline (Bestgen, 2015).

We conduct all experiments with four differently-sized models from the T5
family101010https://huggingface.co/google/t5-base-lm-adapt (Raffel et al.,
2020) in the version that Lester et al. (2021) continually fine-tuned as
language models. We chose T5 because it worked well in previous InstT works
(Sanh et al., 2022; Wei et al., 2022; Wang et al., 2022b). See Appendix E for
the hyperparameters. We repeat all experiments across three different seeds
and report mean and standard deviation of the results.

###  5.2 Results

The results can be found in Table 3. On average, $P_{\mu}$ (average
probability) performs the best across all model sizes, with PPL taking the
second place. AUM is tied with $P_{\text{min}}$ for third place, each
outperforming the other for two of the four model sizes. This ranking is
relatively stable for each individual dataset and the best configuration
always uses $P_{\mu}$. We conclude from this that $P_{\mu}$ clearly emerges as
the best performing baseline for AED in our generative setup. The improvement
over the random baseline is relatively large with over 34 percentage points
(pp) for P3-Donkii but more modest for SNI-Donkii and ADC-Donkii with 13.2 pp
and 10.3 pp respectively. This is likely due to the fact that synthetically
introduced errors are generally easier to detect than naturally occurring ones
(Klie et al., 2022).

Regarding model size, small performs best for P3 and ADC, while large is the
best for SNI. On average, base and large achieve the best performance, while
small also performs surprisingly well. Thus, on a new InstT dataset, we
recommend starting with a base-sized model for efficiency reasons.

P3 out ($9777$) inp ($2460$) - - - rand $50.0$ $50.0$ - - - $P_{\mu}$
$89.4_{\textit{0.9}}$ $68.0_{\textit{0.1}}$ - - - ADC out ($13$) inp ($13$)
noi ($77$) fac ($14$) mul (29) rand $37.0$ $48.0$ $48.4$ $29.8$ $50.9$
$P_{\mu}$ $62.6_{\textit{0.8}}$ $72.2_{\textit{0.2}}$ $49.8_{\textit{0.4}}$
$55.7_{\textit{0.8}}$ $61.5_{\textit{0.5}}$ SNI out form ($64$) noi ($2$) -
mul ($3$) rand $38.2$ $50.0$ $3.0$ - $2.3$ $P_{\mu}$ $51.7_{\textit{1.7}}$
$51.9_{\textit{0.9}}$ $30.6_{\textit{8.6}}$ $14.9_{\textit{3.9}}$

Table 4: Results per error category. All scores are AP (higher is better) in
percent of $P_{\mu}$ using the best performing model size for the dataset. The
category names are abbreviated: out: incorrect output, inp: underspecified
input, noi: noise, fac: factual error, mul: multi-modality, form: formatting.
The number in brackets gives the number of instances per category.

We analyse the performance of the different scorers per annotated error
category. For each dataset, we use $P_{\mu}$ with the respective best-
performing model size. The results can be found in Table 4. Interestingly, the
results differ strongly across error categories and dataset. We find it
encouraging that $P_{\mu}$ outperforms the random baseline for all but two
categories, which are noisy instances in ADC-Donkii and formatting errors in
SNI-Donkii. Surprisingly, other configurations are able to beat the random
baseline for these error types with the respectively best scorers being better
than random by $18.1$/$13.7$ pp for noise/formatting.

On instance vs task-level and epoch aggregation Our annotation of SNI and P3
showed that errors in meta-datasets frequently affect a large portion of all
instances for a given task.111111This observation motivated our annotation
efforts for P3 and SNI. We wondered whether we could exploit this
characteristic by aggregating error scores across all instances for a given
task and thus perform AED on tasks instead of instances. For this, we
performed additional experiments using SNI-Donkii, where we computed two
scores for each task by taking the mean and median over all instances for the
given task. We then follow the same ranking-based evaluation protocol as for
individual instances. Here, we observe a slightly different ordering of
methods with PPL yielding the highest score. On average, the aggregation with
median yielded higher scores than with mean. The absolute AP is much higher
than for detecting errors in individual instances with $69.3\%$ (vs $48.1\%$),
suggesting that task-aggregation might be useful to detect systematic errors
in meta-datasets.

Additionally, we investigate the effect of aggregating scores over all epochs.
For this, we ablate the epoch-aggregation by directly using the final logits
to compute the AED scores. We compute the difference between the best
performing size-score combination with and without epoch-aggregation for each
dataset. We find that the scores drop by 1.3/3.9/1.2 percentage points AP for
P3/SNI/ADC respectively without aggregation over epoch. This provides further
support for the observation that averaging AED scores over epochs generally
improves performance (Swayamdipta et al., 2020; Pleiss et al., 2020; Weber and
Plank, 2023).

##  6 What is the impact of errors on downstream performance?

We investigate this question by observing how errors in InstT datasets impact
downstream performance in a case study. For this, we use the training and
evaluation setup of Tk-Instruct (Wang et al., 2022b), which is the main model
trained on SNI using the code provided by the
authors.121212https://github.com/yizhongw/Tk-Instruct To investigate the
impact of errors, we contrast two models: $\text{Tk-Instruct}_{\text{err}}$
and $\text{Tk-Instruct}^{*}$. Both models build on the three billion parameter
version of T5. For $\text{Tk-Instruct}_{\text{err}}$, we use all 17 tasks
which we found to be erroneous in SNI-Donkii. To these, we add a sample of
additional 100 tasks from the original Tk-Instruct training data. This results
in a training data set of $6,985$ instances across 117 tasks and an error rate
of roughly 8%. For $\text{Tk-Instruct}^{*}$ we replace all erroneous instances
with corrected instances from the same task. We adjust the training pipeline
to our restricted compute budget, see Appendix E for details. We use the same
held-out task mixture for evaluation as Wang et al. (2022b) but remove all
tasks that are in our training data. To evaluate the impact of errors on
instruction tuning, we follow Wang et al. (2022b) and use RougeL for
evaluation. Surprisingly, we find that the difference between both models is
small. $\text{Tk-Instruct}^{*}$ achieves an overall RougeL score of $35.9\%$,
whereas $\text{Tk-Instruct}_{\text{err}}$ obtains $35.7\%$. Moreover,
$\text{Tk-Instruct}_{\text{err}}$ even generates the correct answer for
instances where it observed wrong answers during training. Both observations
suggest that instruction-tuned models might be robust to small numbers of
errors in their training data. However, when we prompt the published version
of T0131313https://huggingface.co/bigscience/T0_3B, the model trained on P3,
with a prompt template for which it during training erroneously always
observed empty strings as output141414”Question 1: […]? Question 2: […]? Do
these questions convey the same meaning? Yes or no?”, we find that it will
always respond with an empty string. This motivates further research into when
and how errors in InstT datasets propagate into models.

##  7 Conclusion

This work presents a first study on annotation error detection for generation
tasks, in particular, instruction-tuning data. Despite the popularity of InsT,
no evaluation datasets for AED with marked errors exists. Therefore, we
present Donkii, a suite of three existing InstT datasets enriched with novel
error annotations which we derive from manual annotation efforts. We propose
four different AED methods for generative models and systematically evaluate
them on the Donkii datasets. We find that with $P_{\mu}$ there is a clear best
performing method and that the choice of model size is crucial for optimal AED
performance. Finally, we investigate how the presence of errors in InstT
datasets affects model quality. We observe only a small quantitative
deterioration due to errors but find qualitatively that errors can directly
translate to unwanted model behaviour. For future work, we plan to apply AED
methods to more structured generative meta-datasets like Huguet Cabot and
Navigli (2021) or Fries et al. (2022).

## Limitations

#### Identified errors in InsT datasets.

We acknowledge that the error categories we have identified are not
exhaustive. This is because the current errors were annotated based on a
manual exploration of middle-sized samples. Furthermore, we recognise that our
error category does not cover issues related to toxicity, hallucinations, and
safety, because we believe that these issues so important that they require
specialized treatment in more focused work (Gehman et al., 2020; Sap et al.,
2022; Raunak et al., 2021; Dziri et al., 2022; Greshake et al., 2023, inter
alia).

#### Small sample size for individual categories

We invested significant manual effort in annotation, but strongly favoured
precision over quantity, with three expert annotators first labeling each
sample individually and then discussing the results. Consequently, the numbers
of found errors per category are moderate to small (see Table 4). We believe
that to make sure that all findings regarding error categories are robust, a
future even larger annotation effort would have to be undertaken.

## Ethics & Broader Impact

Instruction-tuned LLMs have recently seen wide adoption by non-expert users
(OpenAI, 2023). We believe that this makes fine-grained control over the model
outputs and thus, by extension, over the contents of the InstT dataset an
ethical imperative. One facet of this are errors in the data and thus we
believe that using AED methods to analyse InstT datasets can potentially have
positive impact on LLM users. However, the demographics of all annotators are
somewhat uniform and still, in some cases, there was substantial disagreement
on what constitutes an error. Thus, we think that a broader discussion
including more stakeholders is required to get a diverse perspective on what
is the desired behaviour of LLMs and thus what constitutes an error in InstT
datasets.

## Acknowledgments

We thank Shijia Zhou for helping with the annotation of SNI-Donkii. Many
thanks to the members of MaiNLP and NLPNorth for their comments on the paper.
This research is in parts supported by European Research Council (ERC) grant
agreement No. 101043235.

## References

  * Dat (2023) 2023\.  Databricks/databricks-dolly-15k $\cdot$ Datasets at Hugging Face.  https://huggingface.co/datasets/databricks/databricks-dolly-15k.
  * Alt et al. (2020) Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020.  TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1558–1569, Online. Association for Computational Linguistics.
  * Amiri et al. (2018) Hadi Amiri, Timothy Miller, and Guergana Savova. 2018.  Spotting Spurious Data with Neural Networks.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2006–2016, New Orleans, Louisiana. Association for Computational Linguistics.
  * Bach et al. (2022) Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022.  PromptSource: An integrated development environment and repository for natural language prompts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 93–104, Dublin, Ireland. Association for Computational Linguistics.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosiute, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemí Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. 2022.  Constitutional AI: harmlessness from AI feedback.  _CoRR_ , abs/2212.08073.
  * Bender and Friedman (2018) Emily M. Bender and Batya Friedman. 2018.  Data statements for natural language processing: Toward mitigating system bias and enabling better science.  _Transactions of the Association for Computational Linguistics_ , 6:587–604.
  * Bestgen (2015) Yves Bestgen. 2015.  Exact expected average precision of the random baseline for system evaluation.  _Prague Bull. Math. Linguistics_ , 103:131–138.
  * Chong et al. (2022) Derek Chong, Jenny Hong, and Christopher Manning. 2022.  Detecting label errors by using pre-trained language models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9074–9091, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Dickinson and Meurers (2003) Markus Dickinson and W. Detmar Meurers. 2003.  Detecting Errors in Part-of-Speech Annotation.  In _10th Conference of the European Chapter of the Association for Computational Linguistics_ , Budapest, Hungary. Association for Computational Linguistics.
  * Dou et al. (2022) Yao Dou, Maxwell Forbes, Rik Koncel-Kedziorski, Noah A. Smith, and Yejin Choi. 2022\.  Is GPT-3 text indistinguishable from human text? scarecrow: A framework for scrutinizing machine text.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 7250–7274, Dublin, Ireland. Association for Computational Linguistics.
  * Dziri et al. (2022) Nouha Dziri, Sivan Milton, Mo Yu, Osmar Zaiane, and Siva Reddy. 2022.  On the origin of hallucinations in conversational models: Is it the datasets or the models? In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5271–5285, Seattle, United States. Association for Computational Linguistics.
  * Fries et al. (2022) Jason Alan Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Myungsun Kang, Ruisi Su, Wojciech Kusa, Samuel Cahyawijaya, Fabio Barth, Simon Ott, Matthias Samwald, Stephen Bach, Stella Biderman, Mario Sänger, Bo Wang, Alison Callahan, Daniel León Periñán, Théo Gigant, Patrick Haller, Jenny Chim, Jose David Posada, John Michael Giorgi, Karthik Rangasai Sivaraman, Marc Pàmies, Marianna Nezhurina, Robert Martin, Michael Cullan, Moritz Freidank, Nathan Dahlberg, Shubhanshu Mishra, Shamik Bose, Nicholas Michio Broad, Yanis Labrak, Shlok S. Deshmukh, Sid Kiblawi, Ayush Singh, Minh Chien Vu, Trishala Neeraj, Jonas Golde, Albert Villanova del Moral, and Benjamin Beilharz. 2022.  BigBio: A Framework for Data-Centric Biomedical Natural Language Processing.  In _Thirty-Sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , New Orleans, Louisiana, USA.
  * Ge et al. (2022) Huibin Ge, Xiaohu Zhao, Chuang Liu, Yulong Zeng, Qun Liu, and Deyi Xiong. 2022.  TGEA 2.0: A Large-Scale Diagnostically Annotated Dataset with Benchmark Tasks for Text Generation of Pretrained Language Models.  _Advances in Neural Information Processing Systems_ , 35:31612–31626.
  * Gehman et al. (2020) Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020\.  RealToxicityPrompts: Evaluating neural toxic degeneration in language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3356–3369, Online. Association for Computational Linguistics.
  * Greshake et al. (2023) Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023.  Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection.
  * Grivas et al. (2020) Andreas Grivas, Beatrice Alex, Claire Grover, Richard Tobin, and William Whiteley. 2020.  Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports.  In _Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis_ , pages 24–37, Online. Association for Computational Linguistics.
  * Honovich et al. (2022) Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022.  Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor.
  * Huguet Cabot and Navigli (2021) Pere-Lluís Huguet Cabot and Roberto Navigli. 2021.  REBEL: Relation extraction by end-to-end language generation.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2370–2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UNIFIEDQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907, Online. Association for Computational Linguistics.
  * Klie et al. (2022) Jan-Christoph Klie, Bonnie Webber, and Iryna Gurevych. 2022.  Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future.  _Computational Linguistics_ , pages 1–42.
  * Kvĕtoň and Oliva (2002) Pavel Kvĕtoň and Karel Oliva. 2002.  (Semi-)Automatic Detection of Errors in PoS-Tagged Corpora.  In _COLING 2002: The 19th International Conference on Computational Linguistics_.
  * Larson et al. (2020) Stefan Larson, Adrian Cheung, Anish Mahendran, Kevin Leach, and Jonathan K. Kummerfeld. 2020.  Inconsistencies in Crowdsourced Slot-Filling Annotations: A Typology and Identification Methods.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5035–5046, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Larson et al. (2019) Stefan Larson, Anish Mahendran, Andrew Lee, Jonathan K. Kummerfeld, Parker Hill, Michael A. Laurenzano, Johann Hauswald, Lingjia Tang, and Jason Mars. 2019\.  Outlier Detection for Improved Data Quality and Diversity in Dialog Systems.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 517–527, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3045–3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  GPT-4 technical report.  _CoRR_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _NeurIPS_.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pleiss et al. (2020) Geoff Pleiss, Tianyi Zhang, Ethan Elenberg, and Kilian Q Weinberger. 2020.  Identifying Mislabeled Data using the Area Under the Margin Ranking.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 17044–17056. Curran Associates, Inc.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Raunak et al. (2021) Vikas Raunak, Arul Menezes, and Marcin Junczys-Dowmunt. 2021.  The curious case of hallucinations in neural machine translation.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1172–1183, Online. Association for Computational Linguistics.
  * Ruebsamen and Contributors (2023) Gene Ruebsamen and Contributors. 2023.  Alpacadatacleaned.  https://github.com/gururise/AlpacaDataCleaned.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
  * Sap et al. (2022) Maarten Sap, Swabha Swayamdipta, Laura Vianna, Xuhui Zhou, Yejin Choi, and Noah A. Smith. 2022.  Annotators with attitudes: How annotator beliefs and identities bias toxic language detection.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5884–5906, Seattle, United States. Association for Computational Linguistics.
  * Siddiqui et al. (2022) Shoaib Ahmed Siddiqui, Nitarshan Rajkumar, Tegan Maharaj, David Krueger, and Sara Hooker. 2022.  Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics.
  * Sun et al. (2023) Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David D. Cox, Yiming Yang, and Chuang Gan. 2023.  Principle-driven self-alignment of language models from scratch with minimal human supervision.  _CoRR_ , abs/2305.03047.
  * Swayamdipta et al. (2020) Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. 2020.  Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9275–9293, Online. Association for Computational Linguistics.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _CoRR_ , abs/2302.13971.
  * Varshney et al. (2022) Neeraj Varshney, Swaroop Mishra, and Chitta Baral. 2022.  ILDAE: Instance-Level Difficulty Analysis of Evaluation Data.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3412–3425, Dublin, Ireland. Association for Computational Linguistics.
  * Vlachos (2006) Andreas Vlachos. 2006.  Active Annotation.  In _Proceedings of the Workshop on Adaptive Text Extraction and Mining (ATEM 2006)_.
  * Wang et al. (2022a) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a.  Self-Instruct: Aligning Language Model with Self Generated Instructions.
  * Wang et al. (2022b) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Weber and Plank (2023) Leon Weber and Barbara Plank. 2023.  ActiveAED: A Human in the Loop Improves Annotation Error Detection.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned Language Models are Zero-Shot Learners.  In _International Conference on Learning Representations_.
  * Yaghoub-Zadeh-Fard et al. (2019) Mohammad-Ali Yaghoub-Zadeh-Fard, Boualem Benatallah, Moshe Chai Barukh, and Shayan Zamanirad. 2019.  A Study of Incorrect Paraphrases in Crowdsourced User Utterances.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 295–306, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ye et al. (2021) Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021.  CrossFit: A few-shot learning challenge for cross-task generalization in NLP.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7163–7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhou et al. (2023) Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023.  LIMA: Less Is More for Alignment.

##  Appendix A Donkii Data Statement

Following Bender and Friedman (2018), the following outlines the data
statement for Donkii:

  * •

A. CURATION RATIONALE Enrichment of existing instruction-tuning datasets with
annotations for erroneous instances

  * •

B. LANGUAGE VARIETY English with the exact variant(s) unkown because of the
large number of different sources of data

  * •

C. SPEAKER DEMOGRAPHIC Unknown because of the large number of different data
sources

  * •

D. ANNOTATOR DEMOGRAPHIC Three post-doctoral researchers and one Master’s
student (age: 25-40), gender: male and female. Native language: Russian,
German. Socioeconomic status: higher-education student and university
researchers.

  * •

E. SPEECH SITUATION Unknown because of the large number of different data
sources

  * •

F. TEXT CHARACTERISTICS Unknown because of the large number of different data
sources

  * •

PROVENANCE APPENDIX

    * –

Alpaca (Taori et al., 2023), CC By NC 4.0, https://github.com/tatsu-
lab/stanford_alpaca

    * –

AlpacaDataCleaned (Ruebsamen and Contributors, 2023), Apache 2.0,
https://github.com/gururise/AlpacaDataCleaned

    * –

Public Pool of Prompts (Sanh et al., 2022), Apache 2.0,
https://huggingface.co/datasets/bigscience/P3

    * –

Super-Natural Instructions (Wang et al., 2022b), Apache 2.0,
https://github.com/allenai/natural-instructions

##  Appendix B SNI Annotation Guidelines

For this annotation effort, we assume a pair-wise annotation setting. You are
shown two tasks and the instances that differ between them. You should judge
whether one of the two tasks contains fewer errors than the other one. Each
task has the following fields:

  * •

Definition: The instruction to the language model. E.g. ‘solve the following
equation for x’

  * •

Instances: Each with the following fields:

  * •

Input: The input complementing the instruction. E.g. ‘equation: x + 2 = 5’.
Output: The gold-standard output expected from the model

There are four possible labels:

  1. 1.

A is better than B

  2. 2.

B is better than A

  3. 3.

A and B are the same

  4. 4.

I don’t know

Additionally, there is a field for short free-form comments where you can (but
don’t have to) note a reason for your annotation.

We assume that the dataset is used to train a current text-only vanilla LLM
like GPT3 or Llama. That is, it does not have access to tools and cannot
process multi-modal input.

The following rules apply for differences between tasks A and B. We assume
that the difference mentioned in the rule is the only difference between both
(ceteris paribus). If more than one rule applies, we leave the choice to the
best judgment of the annotator. The goal is to make only relative judgments
for the given pair without considering the “absolute quality” of the
instances. Even when both contain very little or many errors, if B clearly
contains more significant errors than A, this should be annotated as “A is
better than B”.

Rules:

  * •

If B contains more errors than A, but those are only few and thus don’t affect
the majority of the instances differing between A and B, then A and B are
equal. As a guideline: If more not more than 90% contain the error, then they
probably should be equal.

  * •

Be lenient in your annotations. If you are unsure whether something is an
error, then better go for A and B are equal.

  * •

Factual correctness: If the output of A can be interpreted as factually
correct, but the one in B cannot, then A is better than B. Example:

    * –

Instruction: Tell me the title of the most popular song released in 2020 so
far.

    * –

Output A: The most popular song released in 2020 so far is "Blinding Lights"
by The Weeknd.

    * –

Output B: The most popular song released in 2020 so far is "The Box" by Roddy
Rich.

    * –

Explanation: A is better than B, because, while the answer to A is ambiguous
(there are multiple measures of popularity), “The Box” was released in 2019
and thus is clearly wrong.

  * •

Noise: If B contains noise (e.g. technical artifacts) but A does not, then A
is better than B. Example:

    * –

Instruction: Suggest the best strategy for a five-second TV commercial.

    * –

Input A:

    * –

Input B: “NoInput”

    * –

Explanation: A is better than B, because “NoInput” is clearly a technical
artifact (even despite A being empty - i.e. no output better than noise).

  * •

Only output: Judge A and B based on the output field not instruction or input.
Justification: It is not clear whether low-quality input with high-quality
output improves or diminishes instruction tuning performance. Example:

    * –

Instruction A: Convert the following number in hexadecimal format.

    * –

Input A: 18

    * –

Instruction B: Convert the number 18 to hexadecimal

    * –

Input B:

    * –

Explanation B: A and B are equal, even though one could prefer A over B
because input and instruction are cleanly separated.

  * •

Unclear instruction: If it is impossible to guess user intent based on the
instruction in B, but it is possible to guess it in A, then A is better than
B. Example:

    * –

Instruction A: Find the average value of the following list of numbers

    * –

Instruction B: Process the following data and output the results

    * –

Input: List: [3, 7, 2, 5]

    * –

Explanation: A is better than B because for B it is not clear at all how the
model should process the data.

  * •

Tool usage: If B requires tool usage (e.g. access to a search engine) but A
doesn’t, then A is better than B. Justification: We assume that the dataset is
used to instruction-tune a vanilla LM without access to tools. Example:

    * –

Instruction A: Provide a brief overview about the following topic.

    * –

Input A: Volcanology

    * –

Instruction B: Take a Wikipedia article and rewrite it in your own words.

    * –

Input B: https://en.wikipedia.org/wiki/Volcanology

    * –

Explanation: A is better than B because B requires access to a web browser.

  * •

Multi-modal input: If B contains multi-modal input (e.g. an image file) but A
doesn’t, then A is better than B. Justification: We assume that the dataset is
used to instruction-tune a vanilla text-only LM. Example:

    * –

Instruction: Critique the given painting.

    * –

Input A: The painting is an abstract composition of vibrant yellow, blue, and
pink hues that appear in an haphazard, yet balanced form and serve as an
evocation of life, joy, and emotion.

    * –

Input B: [Painting attached]

    * –

Explanation: A is better than B because B contains multimodal input.

  * •

Temporal knowledge: If B contains temporal knowledge but A doesn’t, then A is
better than B. Justification: We want the instruction-tuned model to handle
temporal knowledge gracefully. Example:

    * –

Instruction A: What is the name of the 46th president of the United States?

    * –

Instruction B: What is the name of the current president of the United States?

    * –

Explanation: A is better than B because the answer to B will change over time
while the answer to A is static.

  * •

Formatting: If A and B differ only in formatting, then A and B are equal
Example:

    * –

Output A: \- Astonished \- Amazed \- Shocked \- Stunned \- Speechless \-
Bewildered"

    * –

Output B: Astonished, amazed, shocked, stunned, speechless, bewildered.
Explanation: A is equal to B because the output only differs in formatting

##  Appendix C ADC Annotation Guidelines

For this annotation effort, we assume a pair-wise annotation setting. You are
shown two instances and have to judge which of both would you preferably
include in an instruction-tuning dataset. Each instance has two to three
fields:

  * •

Instruction: The instruction to the language model. E.g. ‘solve the following
equation for x’

  * •

Input (optional): The input complementing the instruction. E.g. ‘equation: x +
2 = 5’. Instructions can be self-contained, thus Input is optional.

  * •

Output: The gold-standard output expected from the model

There are four possible labels:

  1. 1.

A is better than B

  2. 2.

B is better than A

  3. 3.

A and B are the same

  4. 4.

I don’t know

Additionally, there is a field for short free-form comments where you can (but
don’t have to) note a reason for your annotation.

We assume that the dataset is used to train a current text-only vanilla LLM
like GPT3 or Llama. That is, it does not have access to tools and cannot
process multi-modal input.

The following rules apply for differences between instances A and B. We assume
that the difference mentioned in the rule is the only difference between both
(ceteris paribus). If more than one rule applies, we leave the choice to the
best judgment of the annotator. The goal is to make only relative judgments
for the given pair without considering the “absolute quality” of the
instances. Even when both are very high or low quality, if B is clearly worse
than A, this should be annotated as “A is better than B”.

Rules:

  * •

Factual correctness: If the output of A can be interpreted as factually
correct, but the one in B cannot, then A is better than B. Example:

    * –

Instruction: Tell me the title of the most popular song released in 2020 so
far.

    * –

Output A: The most popular song released in 2020 so far is "Blinding Lights"
by The Weeknd.

    * –

Output B: The most popular song released in 2020 so far is "The Box" by Roddy
Rich.

    * –

Explanation: A is better than B, because, while the answer to A is ambiguous
(there are multiple measures of popularity), “The Box” was released in 2019
and thus is clearly wrong.

  * •

Noise: If B contains noise (e.g. technical artifacts) but A does not, then A
is better than B. Example:

    * –

Instruction: Suggest the best strategy for a five-second TV commercial.

    * –

Input A:

    * –

Input B: “NoInput”

    * –

Explanation: A is better than B, because “NoInput” is clearly a technical
artifact (even despite A being empty - i.e. no output better than noise).

  * •

Only output: Judge A and B based on the output field not instruction or input.
Justification: It is not clear whether low-quality input with high-quality
output improves or diminishes instruction tuning performance. Example:

    * –

Instruction A: Convert the following number in hexadecimal format.

    * –

Input A: 18

    * –

Instruction B: Convert the number 18 to hexadecimal

    * –

Input B:

    * –

Explanation B: A and B are equal, even though one could prefer A over B
because input and instruction are cleanly separated.

  * •

Unclear instruction: If it is impossible to guess user intent based on the
instruction in B, but it is possible to guess it in A, then A is better than
B. Example:

    * –

Instruction A: Find the average value of the following list of numbers

    * –

Instruction B: Process the following data and output the results

    * –

Input: List: [3, 7, 2, 5]

    * –

Explanation: A is better than B because for B it is not clear at all how the
model should process the data.

  * •

Tool usage: If B requires tool usage (e.g. access to a search engine) but A
doesn’t, then A is better than B. Justification: We assume that the dataset is
used to instruction-tune a vanilla LM without access to tools. Example:

    * –

Instruction A: Provide a brief overview about the following topic.

    * –

Input A: Volcanology

    * –

Instruction B: Take a Wikipedia article and rewrite it in your own words.

    * –

Input B: https://en.wikipedia.org/wiki/Volcanology

    * –

Explanation: A is better than B because B requires access to a web browser.

  * •

Multi-modal input: If B contains multi-modal input (e.g. an image file) but A
doesn’t, then A is better than B. Justification: We assume that the dataset is
used to instruction-tune a vanilla text-only LM. Example:

    * –

Instruction: Critique the given painting.

    * –

Input A: The painting is an abstract composition of vibrant yellow, blue, and
pink hues that appear in an haphazard, yet balanced form and serve as an
evocation of life, joy, and emotion.

    * –

Input B: [Painting attached]

    * –

Explanation: A is better than B because B contains multimodal input.

  * •

Temporal knowledge: If B contains temporal knowledge but A doesn’t, then A is
better than B. Justification: We want the instruction-tuned model to handle
temporal knowledge gracefully. Example:

    * –

Instruction A: What is the name of the 46th president of the United States?

    * –

Instruction B: What is the name of the current president of the United States?

    * –

Explanation: A is better than B because the answer to B will change over time
while the answer to A is static.

  * •

Formatting: If A and B differ only in formatting, then A and B are equal
Example:

    * –

Output A: \- Astonished \- Amazed \- Shocked \- Stunned \- Speechless \-
Bewildered"

    * –

Output B: Astonished, amazed, shocked, stunned, speechless, bewildered.
Explanation: A is equal to B because the output only differs in formatting

  * •

Global properties: If A and B differ only with respect to the full dataset,
e.g. because A increases grammatical diversity in the input but B doesn’t,
then A and B are equal Example:

    * –

Instruction A: The average of 10 numbers is 85. If the numbers 70 and 76 are
removed from the set of numbers, what is the average of the remaining numbers?

    * –

Instruction B: Find the average of 85, 44 and 102.

    * –

Explanation: A and B are equal, even though more complex problems as A are
much less frequent in the whole dataset than problems of the type B.

  * •

Subjectivity: If the annotator feels that their preference for one instance is
strongly subjective, then A and B are equal.

  * •

Leniency: Be lenient in your annotations. If you are unsure whether something
is an error, then better go for A and B are equal. If both outputs can be
interpreted as correct, then A and B are equal

Instruction 1: Name two deserts in the Sahara. Input 1: Output 1: The two
deserts in the Sahara are the Great Western Erg and the Great Eastern Erg.
Instruction 2: Recognize the following bird’s species. Input 2: <Image of a
bird> Output 2: Western Great Egret (Ardea alba maxima). Label: 1 is better
than 2 Error category: Image in input

Table 5: The pairwise annotation task example contains an error in Instruction
2, as it includes an imaginary image in the input.

##  Appendix D Examples of errors

Error Source Instruction Input Output (shortened) Incorrect output Wrong
output SNI Given a sentence and an entity, the task is to select the authors
sentiment towards the enity. Sentiments can be Positive, Neutral and Negative.
<…> What is the sentiment of the following document towards the entity Hayley
Smith ? Hayley Smith was diagnosed with chronic depression in her early
twenties: “I’d been bottling up quite a bit through most of my teens. Then it
hit me a like a brick wall ” she said. Neutral Empty output Alpaca Create a
flow chart to explain the working of a web browser. Labels flipped SNI In this
task you will be given a claim and a perspective. You should determine whether
that perspective supports or undermines the claim. If the perspective could
possibly convince someone with different view, it is supporting, otherwise it
is undermining. claim: Corporations are a threat to Democracy and society.
perspective: pers Support Output space SNI You will see a sentence and a given
pronoun. Answer with the word in the sentence to which the pronoun is
referring. text: as andrea in the crop duster passed over susan, she could see
the landing gear.; pronoun: she susan Off- instruction Alpaca Given three
numbers (x, y, z), find the greatest common factor. 2, 4, 2008 The greatest
common factor of 2, 4, and 8 is 2. Grammar error SNI In this task, you are
given a question and an answer, you would be asked to create the sentence
based on the Question-Answer provided. It should be contained within the
Question-Answer provided. Question: What is the most common type of cancer in
adult males? Answer: prostate Prostate is the most common type of cancer in
adult males. Factual Errors Temporal context Alpaca What is the average annual
salary for a customer service developer? The average annual salary for a
customer service developer is usually around $74,508. Bad math Alpaca
Calculate the average of the following three numbers: 10, 12, 15. The average
of 10, 12, and 15 is 12.67. Encyclopedic Alpaca What is the average air
pressure of a human’s lungs? The average air pressure in a human’s lungs is
between four and seven centimeters of water (4–7 cm H2O). Noise Noise Alpaca
Write an essay about the benefits of public transportation. No input required.
Public transportation is an invaluable resource for communities and cities
around the world. <…> Underspecified Input Missing context information Alpaca
Write a tweet summarizing this article about an animal rescue in Thailand. The
article Amazing news! An animal rescue in Thailand has successfully freed 63
bears from animal trafficking. The rescued bears will now receive the proper
care and attention they need. Unclear instruction Alpaca Sort the following 10
fruits into three categories. Bananas, Apples, Pineapples, Plums, Cherries,
Peaches, Grapes, Watermelons, Mangoes, Oranges Category 1: Bananas, Apples,
Pineapples, Plums, Cherries; Category 2: Peaches, Grapes, Watermelons;
Category 3: Mangoes, Oranges Model-modality mismatch Requires web browsing
Alpaca Provide a summarization of the article. https://en.wikipedia.org/wiki/
Artificial_intelligence Artificial Intelligence, commonly known as AI, refers
to the simulation of human intelligence processes by machines, especially
computer systems. <…> Table in input Alpaca Dataset of 500 people with the
variable “income”. The mean income is $50,000. Image in input Alpaca How would
you use this photo in a marketing campaign? <image included in email> The
photo can be used to create a compelling marketing campaign that draws
attention to the product or service. <…> Image in output Alpaca Make a word
cloud on the given topic. Artificial Intelligence <Word Cloud Output>
Formatting Formatting SNI In this task, you will be given a short story. One
sentence from the story is chosen. Consider the events that happen before that
sentence, or are likely to have happened before it. Does any of them directly
cause it, or simply make it possible? You should write your answer in the form
" A >causes/enables> B". Try to use phrases and sentences from the story to
compose your answer when possible. story: I went down to the tidepool to watch
the tide roll out. I sat on the dock and waited, while listening to my mp3
player. Once the tide was out, I saw Something shiny in the muddy bottoms. I
went down and found that it was a gold ring! Today was my lucky day! selected
sentence: I went down to the tidepool to watch the tide roll out. I decide
togotothe tidepool >Causes/Enables> I gotothe tidepool

Table 6: Examples of errors.

##  Appendix E Hyperparameters

###  E.1 AED Experiments

We experiment with four sizes, namely small (60 million parameters), base (220
million), large (770 million), and 3B (3 billion). We train each of the models
for 10 epochs as a seq2seq LM using a batch size of 60 and a learning rate of
$1e-3$. Note, that we train separate models for each of the three datasets and
leave the exploration of possible synergies across datasets for future work.
We set the maximum source length to 512/768/768 for P3-Donkii/SNI-Donkii/APC-
Donkii and the output length to 256.

###  E.2 Experiment on effect of errors on downstream performance

We train and evaluate the model in a strict zero-shot setting without
providing few-shot examples to reduce the input length of the instances.
Second, we use only at most 64 instances per task, because Wang et al. (2022b)
find that increasing this number does not improve performance. We train the
model for 30 epochs with a batch size of 1024.


# Ensemble based discriminative models for Visual Dialog Challenge 2018

Shubham Agarwal
Heriot Watt University
Edinburgh, UK
<EMAIL_ADDRESS>Adeptmind Scholar Raghav Goyal
University of British Columbia
Vancouver, Canada
<EMAIL_ADDRESS>work done at TwentyBN

###### Abstract

This manuscript describes our approach for the Visual Dialog Challenge 2018.
We use an ensemble of three discriminative models with different encoders and
decoders for our final submission. Our best performing model on ‘test-std’
split achieves the NDCG score of $55.46$ and the MRR value of $63.77$,
securing third position in the challenge.

##  1 Introduction

Visual dialog [2] is an interesting new task combining the research efforts
from Computer Vision, Natural Language Processing and Information Retrieval.
While [6] presents some tips and tricks for VQA 2.0 Challenge, we follow their
guidelines for the Visual Dialog challenge 2018. Our models use attention
similar to [1] to get object level image representations from Faster R-CNN
model [5]. We experiment with different encoder mechanisms to get
representations of conversational history.

##  2 Models

Common to all the models, we initialize our embedding matrix with pre-trained
Glove word vectors of 300 dimensions using 6B tokens
111https://nlp.stanford.edu/projects/glove/. Out of $11319$ tokens present in
the dataset, we found $188$ tokens missing from the pre-trained Glove
embeddings, so we manually map these tokens to words conveying semantically
similar meaning, e.g. we map over ten variations of the word “yes” -
misspelled or not picked up by tokenizer - “*yes”, “yesa”, “yess”, “ytes”,
“yes-”, “yes3”, “yyes”, “yees”, etc.

For image features, we extract Faster R-CNN features with ResNet-101 backbone
trained on Visual genome [3] dataset, similar to [1]. We use an adaptive
number of object proposals per-image ranging from $10$ to $100$ generated
using a fixed confidence threshold and each object is then associated with
$2048$-dimensional mean-pooled features using ROI pooling. We use
discriminative decoding throughout our models.

We first describe our models individually and then the ensembling technique
that we employ. In the following, MN denotes Memory Networks to encode
conversational history, RCNN signify R-CNN for object level representations of
an image, Wt represents additional linear layer in the decoder, and LF a late
fusion mechanism as defined in [2].

### LF-RCNN

Late fusion encoder [2] with concatenated history. We use two-layered LSTMs
with $512$ hidden units for embedding questions and history. The object-level
features are weighed using only question embeddings. The word embeddings from
Glove vectors are frozen and are not fine-tuned. Figure 3 gives an overview of
the architecture.

### MN-RCNN

Memory network encoder [2] with bi-directional GRUs and word embeddings fine-
tuned. Object-level features are weighed by question and caption embedding.
The rest of the scheme is same as above. (Figure 3)

### MN-RCNN-Wt

Same as above but with an additional linear layer applied to the dot product
of candidate answer and encoder output, and gated using tanh function. Compare
Figure 3 with Figure 3

### Ensembling

We ensembled final layer’s log-softmax output - which is a distribution over
candidate answers for each round (Figure 4). We use the three models described
above and take the mean of the results (we also tried taking maximum of the
results but found mean to perform better). We also tried ensembling a subset
of the above three models, but found the combination of all three to
outperform the rest.

---

Figure 1: LF-RCNN model as defined in Section 2. Number indicates the output dimension of each layer. Figure 2: MN-RCNN model Figure 3: MN-RCNN-Wt model Figure 4: Ensemble using mean of log probabilities from the three individual models. Model | NDCG (x 100) | MRR (x 100) | R@1 | R@5 | R@10 | Mean
---|---|---|---|---|---|---
LF-RCNN | 51.69 | 61.03 | 47.03 | 77.83 | 87.55 | 4.70
MN-RCNN | 53.59 | 61.25 | 46.78 | 79.43 | 87.93 | 4.63
MN-RCNN-Wt | 53.20 | 61.50 | 47.10 | 78.7 | 88.38 | 4.54
Ensemble (all three) | 55.46 | 63.77 | 49.8 | 81.22 | 90.03 | 4.11
Table 1: Results for the challenge on test-std. We took an ensemble of best
performing models for the final submission.

Model | MRR | R@1 | R@5 | R@10 | Mean
---|---|---|---|---|---
Baseline | 57.57 | 42.98 | 74.64 | 84.91 | 5.48
MN | 59.24 | 44.64 | 76.48 | 86.41 | 5.14
MN-Wt | 59.54 | 44.98 | 77.10 | 86.38 | 5.03
[0.5pt/2pt] LF-RCNN | 61.94 | 48.08 | 79.04 | 88.23 | 4.61
MN-RCNN | 62.99 | 49.07 | 80.13 | 88.74 | 4.45
MN-RCNN-Wt | 63.11 | 49.29 | 80.10 | 89.09 | 4.43

Table 2: Results on validation set. We show the impact of using additional
gated linear layer in decoder. Compare MN with MN-Wt.

##  3 Experiments and Results

We used Pytorch222https://pytorch.org/ [4] for implementation.333We are
thankful to the organizers for providing a starter code in Pytorch on which we
build our model. https://github.com/batra-mlp-lab/visdial-challenge-starter-
pytorch In our experiments, we find that fine-tuning initialized Glove
embeddings performed better than frozen embeddings. Object level
representations play a critical role to generate a correct response from the
model. Eventually, we use an ensemble of all the models described above for
our final submission. Table 2 summarizes our results on validation set while
Table 1 on Test-Standard split.

##  4 Conclusion

We experimented with discriminative models for our submission. Object level
image representations gave a huge uplift in the evaluation metrics. Bi-
directional GRUs constantly performed better than uni-directional LSTMs with
Memory Networks outperforming Late fusion encoders for encoding conversational
history. We even found that fine-tuning Glove embeddings performed better than
their counterparts. Our final submission is an ensemble of three
discriminative models and achieve the NDCG of 55.46 on test-std.

## References

  * [1] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang.  Bottom-up and top-down attention for image captioning and visual question answering.  In CVPR, 2018.
  * [2] A. Das, S. Kottur, K. Gupta, A. Singh, D. Yadav, J. M. Moura, D. Parikh, and D. Batra.  Visual Dialog.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
  * [3] R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, et al.  Visual genome: Connecting language and vision using crowdsourced dense image annotations.  IJCV, 2017.
  * [4] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer.  Automatic differentiation in pytorch.  In NIPS-W, 2017.
  * [5] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [6] D. Teney, P. Anderson, X. He, and A. van den Hengel.  Tips and tricks for visual question answering: Learnings from the 2017 challenge.  In CVPR, 2018.