(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext:
%TODO␣FINAL:␣Replace␣with␣your␣institution␣list.https://huggingface.co/q-future/co-
instruct

# Towards Open-ended Visual Quality Comparison

Haoning Wu Equal contribution. ${}^{\text{1}}$Nanyang Technological
University. ${}^{\text{2}}$City University of Hong Kong.
${}^{\text{3}}$Shanghai Jiao Tong University. ${}^{\text{4}}$Sensetime
Research.11 Hanwei Zhu⋆ 22 Zicheng Zhang⋆ 33 Erli Zhang
Chaofeng Chen 1111 Liang Liao 11 Chunyi Li 33 Annan Wang 11 Wenxiu Sun
Qiong Yan 4444 Xiaohong Liu 33 Guangtao Zhai 33 Shiqi Wang 22 Weisi Lin 11

###### Abstract

Comparative settings (_e.g_. pairwise choice, listwise ranking) have been
adopted by a wide range of subjective studies for image quality assessment
(IQA), as it inherently standardizes the evaluation criteria across different
observers and offer more clear-cut responses. In this work, we extend the edge
of emerging large multi-modality models (LMMs) to further advance visual
quality comparison into open-ended settings, that 1) can respond to open-range
questions on quality comparison; 2) can provide detailed reasonings beyond
direct answers. To this end, we propose the Co-Instruct. To train this first-
of-its-kind open-source open-ended visual quality comparer, we collect the Co-
Instruct-562K dataset, from two sources: (a) LLM-merged single image quality
description, (b) GPT-4V “teacher” responses on unlabeled data. Furthermore, to
better evaluate this setting, we propose the MICBench, the first benchmark on
multi-image comparison for LMMs. We demonstrate that Co-Instruct not only
achieves in average 30% higher accuracy than state-of-the-art open-source
LMMs, but also outperforms GPT-4V (its teacher), on both existing related
benchmarks and the proposed MICBench.

###### Keywords:

Large Multi-modality Models (LMM) Visual Quality Assessment Visual Quality
Comparison Visual Question Answering

Figure 1: The proposed Co-Instruct, first-of-its-kind open-source LMM with
capability on open-question & detailed-reasoning visual quality comparison. It
outperforms existing LMMs on the proposed MICBench as well as existing quality
evaluation benchmarks. Figure 2: The motivation of open-ended visual quality
comparison: comparative settings can effectively avoid the ambiguity on
absolute evaluations for single images, and provide more clear-cut judgements
to serve as downstream guidances [53, 65].

##  1 Introduction

Image quality assessment (IQA) has been an important domain in visual
computing, as it provides effective recommendation [57] on high-quality visual
contents and valuable guidance [53, 61, 65] for potential improvements. Most
recently, several pioneer studies [50, 49, 51] have explored large multi-
modality models (LMMs, _e.g_. GPT-4V) [34, 9, 25, 56, 4, 5], on expanding IQA
from giving a scalar score (e.g. 3.457) to the open-ended scenarios, that
allows evaluations in response to open-range questions, and provide detailed
reasonings beyond an overall score.

While these studies sufficiently emulate human ability on IQA, they also
suffer from the same drawback as human: ambiguity on absolute evaluations. For
instance, as shown in Fig. 2 (a), different human observers hold different
standards on the exposure level on single images, and henceforth provide
diversified absolute evaluations. Nevertheless, while asked to compare the
exposure level of the images, all observers agree with the rank
(1)>(2)>(3)>(4) (Fig. 2(b)); all observers also agree that (1) has the highest
exposure, though not all choose the option high while evaluating it
independently. Given this observation, comparison has been a traditional human
study setting for quality evaluation [43] and adopted by a wide range of
existing subjective studies [35, 62, 10, 53, 52]. Furthermore, to avoid the
ambiguity, the comparative settings are also predomintantly adopted [65, 53]
while applying IQA algorithms for improvement guidance.

Figure 3: The construction methodology of Co-Instruct-562K, a combination of
(a) Merge2Compare (LLM comparison from human-labeled single image quality
descriptions) and (b) Teach2Compare (GPT-4V comparison on multiple unlabeled
images).

While comparison has widely-recognized significance for IQA, existing related
datasets [35] and methods [63, 28] are generally based on overall quality
comparison and have not extended to the open-ended scenarios; on the other
hand, open-source LMMs [50, 25, 4] are usually only fine-tuned with single
image instruction tuning datasets [26, 13, 1] and proved to lack enough
capability even on two image comparison settings [67, 69]. While these gaps
have clearly indicated the need of a specific instruction tuning dataset for
visual quality comparison, it is too expensive to collect such a dataset from
human. To avoid costly human labors, we propose an alternative strategy to
collect the training dataset, named Collaborative Instruction Tuning from Weak
Supervisors (Co-Instruct). Specifically, we adopt two non-perfect supervisors:
1) Merge2Compare (Fig. 3(a)). Originated from single image quality
descriptions on 19K images as labeled by human in the Q-Pathway [50] dataset,
we randomly match them into 100K groups (2-4 images per group) with removing
the most similar descriptions with an text embedding model [44]. Then, we
prompt a single-modal large language model (LLM) [16] to compare multiple
human descriptions in a group, and “merge” them into 100K pseudo comparisons.
2) Teach2Compare (Fig. 3(b)). Observing that GPT-4V has especially high
accuracy on pairwise settings [69, 67] among existing LMMs, following existing
practices [2, 17], we leverage GPT-4V responses to further expand our dataset.
We collect 9K unlabeled images and randomly match into 30K image groups (also
2-4 images per group), and obtain GPT-4V responses on both caption-like
general comparisons and question-answer pairs for comparisons. By integrating
Q-Instruct-200K [50] (on single images), Merge2Compare, and Teach2Compare, we
construct the Co-Instruct-562K, the first instruction tuning dataset for open-
ended visual quality comparison.

To correctly refer to each specific image during conversation, we define a
specific image-text interleaved format [5] to handle multi-image cases, as
follows:
User: The first image: <img${}_{\texttt{0}}$> The second image:
<img${}_{\texttt{1}}$> ... <query>
Assistant: <response>

Moreover, as we need to feed multiple images together to the LLM decoder,
adopting the most popular LLaVA [26, 39, 25] structure that linearly projects
visual embeddings will exceed the context window of the language models [42]
and cause errors. Henceforth, we adopt an alternative visual abstractor
structure [56] to first reduce visual token length (from 1,025 to 65 tokens
per image), and then concatenate them with text embeddings to pass to language
decoders. By learning from the Co-Instruct-562K dataset and the specially-
designed input structure, we present the Co-Instruct, with up to 86%
improvements than its baseline [56], and 61% better than the state-of-the-art
open-source LMM. More importantly, despite using GPT-4V as one of its
teachers, it still surpasses the GPT-4V teacher in a variety of multi-choice
question (MCQ) benchmarks, and matches GPT-4V ability in scenarios requiring
detailed language reasonings.

After training the model Co-Instruct, our another concern is the lack of
abundant evaluation settings on multi-image comparison: while Q-Bench [49, 67]
series have covered multiple formats on single images and image pairs, there
is no existing evaluation scenario for quality comparison beyond two images.
To complement this evaluation setting, we construct the MICBench.
Specifically, the MICBench contains 2,000 multi-choice questions (MCQ)
comparing the quality or related attributes among a group of three or four
images (each half), in which over half of the questions are Which questions
(e.g. which image has highest clarity?). Aiming to extract an image with a
specific quality-related appearance from a group, Which questions are the most
important questions related to image comparison. Despite Which questions, the
MICBench also includes Yes-or-No questions and other question types
(What/How/How-Many, etc) to provide a holistic benchmark setting for multi-
image quality comparison.

In summary, we conduct a systematical study towards open-ended visual quality
comparison. Our contributions can be summarized as three-fold:

  1. 1.

We construct the first instruction-tuning dataset for visual quality
comparison, the Co-Instruct-562K. With data collected from two “weak
supervisors”, Merge2Compare (LLM-merged comparisons) and Teach2Compare (GPT-4V
pseudo-labeled comparisons), our public dataset significantly expands the
capabilities of open-source LMMs on visual comparative settings.

  2. 2.

We propose the most capable model for open-ended visual comparison, the Co-
Instruct. With image-text interleaved input format and fine-tuned with the Co-
Instruct-562K dataset, it significantly outperforms existing methods (and even
GPT-4V) in multiple open-ended visual comparison tasks. With open weights, it
allows for broader application than proprietary models.

  3. 3.

We construct the benchmark, the MICBench, as the first benchmark to evaluate
LMMs for quality comparison on multiple images (more than two). It covers
2,000 diverse-type open-range MCQs related to visual quality comparison among
three or four images. The MICBench contributes to more holistic evaluation
studies on the visual quality comparison problem.

##  2 Related Works

###  2.1 Visual Quality Comparison

Visual quality comparison (especially paired comparison) is a widely used
subjective quality assessment methodology, serving as the most reliable way to
collect human opinions [31]. However, when the number of images increases, the
experiments become infeasible because of the exponential growth of pairwise
comparisons [14]. While many active sampling methods have been proposed to
reduce the number of pairs [54, 21, 33], they are computationally expensive
and unsuitable for large-scale experiments. Despite subjective studies,
learning to rank quality is widely proven as effective by many objective
approaches [27, 30, 8, 63, 46, 48, 64]. Nonetheless, they typically only
predict a scalar score or a binary judgement for overall comparison, limiting
their ability to provide meaningful feedbacks into specific types of
distortions or preferences.

###  2.2 LMMs for Visual Quality Evaluation

Several recent studies have explored LMMs for visual quality evaluation. The
Q-Bench [49] proposes a holistic benchmark for LMMs on low-level perception
(quality-related MCQs), description (quality-related captioning) and
assessment (predicting scores). Following this path, Q-Instruct [50] advances
the ability of LMMs with a large-scale human-annotated dataset, and Q-Align
[51] designs a text-guided instruction tuning for score predictions and
outperforms non-LMM approaches. However, these explorations are based on
single images and have not covered comparative settings. While most recent
benchmarks [67, 69] suggest that open-source LMMs trained with single image
datasets cannot perform well on comparative settings, to bridge this gap, we
collect the first instruction tuning dataset to teach LMMs to compare visual
quality, the Co-Instruct-562K, and our model significantly improves the
ability of open-source LMMs on comparative settings, moving a step forward on
the basis of existing explorations.

Table 1: Statistics of our Co-Instruct-562K dataset. ‘#’ denotes “the number
of”.

Subsets |  Q-Instruct-200K [50] | Merge2Compare |  Teach2Compare-general |  Teach2Compare-Q&A | All
---|---|---|---|---|---
Instruction Type | Detailed Reasoning, Question Answering | Detail Reasoning | Detail Reasoning | Question Answering | –
# Total Images | 19K (shared, both using Q-Pathway images) | 9K (using shared images) | 28K
# Total Data Items | 202K | 100K | 30K | 230K | 562K
# Single Images | 202K | 0 | 0 | 0 | 202K
# Image Pairs | 0 | 70K | 18K | 134K | 222K
# Groups of Three | 0 | 20K | 6K | 51K | 77K
# Groups of Four | 0 | 10K | 6K | 45K | 61K

##  3 Data Construction

In this section, we elaborate on the construction process of the Co-
Instruct-562K dataset. Though human annotation is the most direct approach to
collect data, as is widely acknowledged [21, 54], acquiring sufficient
comparative data on a large set of images demands a markedly increased volume
of human annotations [10, 35] in contrast to gathering opinions on the same
set of individual images. To avoid this unbearable cost, we propose an
alternative data construction strategy without additional human annotation, by
following three key principles:

  1. 1.

Convert: Utilize reliable information from existing datasets.

  2. 2.

Learn-from-Model: Leverage the verified capabilities of models.

  3. 3.

Co-Instruct: Collecting diverse subsets that complement each other.

Under the principles, we collect two different subsets for instruction tuning:
Merge2Compare (Sec. 3.1), which converts information from human quality
descriptions on single images, and the ability of single-modal LLMs on
comparing and analyzing texts; and Teach2Compare (Sec. 3.2), which leverages
the verified ability of GPT-4V on comparing images. Finally, we discuss how
the two subsets complement each other (Sec. 3.3) under a co-instruct scheme.

###  3.1 Merge2Compare.

In this part, we define the construction process for Merge2Compare includes
three steps: 1) pair/group matching; 2) top-similarity pair removal; and 3)
LLM merging. An examplar illustration of the process is shown in Fig. 4.

Figure 4: The pipeline of constructing Merge2Compare: images are first matched
into groups (a), and then filtered via top-similarity removal (b). After
filtering, the single image quality descriptions are merged (c) into
comparisons by the LLM [16].

#### Step 1: Pair/Group Matching (Fig. 4(a)).

To best utilize information from existing single image quality descriptions,
following the empirical rule to sample $O(n\log n)$ pairwise combinations to
effectively rank among all individual items in a set [37, 45], we randomly
sample 81K image pairs from all 19K images in the Q-Pathway. Despite pairs, we
further sample 27K groups with three images and 18K groups with four images to
cover the scenarios of more images.

#### Step 2: Top-Similarity Pair Removal (Fig. 4(b)).

The effectiveness of the merge comes from the differences among descriptions,
e.g. between The quality is acceptable for <img${}_{\texttt{0}}$> and The
quality is poor for <img${}_{\texttt{1}}$>. However, if descriptions in a
pair/group contains almost the same information (_e.g_. both images with The
clarity is good, but lighting is dark), the merged comparisons will lack
enough information or even with false predictions. Henceforth, we use
E5-Mistral [44] text embedding model to compute similarities among
descriptions, and remove if any high-similarity description pairs exist in the
group. After removal, 70K image pairs (86% of initial samples), 20K groups of
three (74% of initial) and 10K groups of four (55% of initial) are preserved
and fed into the LLM for merging.

#### Step 3: LLM Merging (Fig. 4(c)).

The key step for the Merge2Compare is to prompt LLMs to convert the single
image evaluations to comparative texts. Specifically, following many existing
practices [26, 68, 58], we put the descriptions as alternates of images in the
context. Denote the description for image <img${}_{\texttt{i}}$> as
<desc${}_{\texttt{i}}$>, the user query for LLM merging is formulated as
follows:
(Pairs) The first image: <desc${}_{\texttt{0}}$> The second image:
<desc${}_{\texttt{1}}$>
Which image has better quality, and why?
(Groups of Three/Four) $\\{$The $K_{i+1}$ image: <desc${}_{\texttt{i}}$>
$|_{i=0}^{N-1}\\}$
Please rank the quality of the images and justify your rankings.
where $K_{i}$ represents the ordinal form of $i+1$, _e.g_. $K_{1}$ is first,
$K_{2}$ is second.

The merged comparisons are overall comparisons with reasonings (Fig. 4 right).
To validate their reliability, we conducted a human examination on 250 random
samples from Merge2Compare mixed with merged comparisons from 250 random
removed groups. The correctness rate in Merge2Compare is 96%, while it is only
72% in the removed groups, demonstrating the effects of the top-similarity
removal process in ensuring the quality of training data.

Figure 5: The pipeline of constructing Teach2Compare: 9K diverse images are
collected and matched into 30K groups (a). The groups are then fed to GPT-4V
to obtain general quality comparisons (b) and question-answering (c) related
to quality comparisons.

###  3.2 Teach2Compare.

Given existing evaluations [69, 67] suggesting that GPT-4V is decent at
comparing visual quality (Tab. 2/3/4), we propose to collect GPT-4V responses
as pseudo labels for quality comparison. As shown in Fig. 5, we collect
diverse unlabeled images and feed them to GPT-4V with different prompts to
obtain Teach2Compare-general (overall quality comparison) and
Teach2Compare-Q&A (question-answer pairs related to quality comparison).
Details as follows.

#### Collection of Images (Fig. 5(a)).

For Teach2Compare, we collect 9K images from various sources to cover
different quality concerns and visual appearances: 1) 5.4K in-the-wild images
from YFCC-100M [41] database; 2) 1.8K images artificially-distorted images
from COCO [3] (with 15 types of distortions via ImageCorruptions [32]) and
KADIS-700K [24] (25 types of distortions); 3) 1.8K AI-generated images from
ImageRewardDB [53]. These 9K diverse unlabeled images are further grouped into
18K pairs, 6K groups of three, and 6K groups of four, for GPT-4V to provide
pseudo labels under two formats, as follows.

####  _Teach2Compare_ -general (Fig. 5(b)).

Similar to Merge2Compare, the general subset also consists of overall
comparison with reasonings. Specifically, we substitute the
<desc${}_{\texttt{i}}$> in the Merge2Compare prompt template to respective
real images <img${}_{\texttt{i}}$> to feed to GPT-4V. After collection, we
also conduct a similar 250-sample spot check on the output pseudo labels,
which reports around 94% correctness. Though slightly less accurate than
Merge2Compare (96%), examiners also report that GPT-4V labels contain more
content information which has been observed to enhance quality understandings
of models [47, 19, 20]. The two subsets are expected to complement each other
for better learning outcomes.

####  _Teach2Compare_ -Q&A (Fig. 5(c)).

Despite general comparisons, for GPT-4V, we also collect a specific subset to
improve LMM ability on responding to open-range questions. To achieve this, we
first list reference aspects (clarity, lighting, color, etc) and then ask
GPT-4V to generate questions (and respective correct answers, and false
answers for the questions) on comparing these aspects among the images. After
removing failed generations, we obtain 230K question-answers from 30K image
groups. These question-answers are converted to both direct question-answering
and multi-choice questions (as A-OKVQA[38]) for training.

Figure 6: The structure of Co-Instruct. (a) Images are encoded by visual
embedding layers and then passsed through an abstractor module to reduce token
numbers, and then (c) fused with text embeddings into under the image-text
interleaved format.

###  3.3 Rationale of Combinations.

As discussed in principle 3, our motivation is to collect subsets that can
complement each other. This complementarity is reflected in the following two
aspects. Firstly, in terms of general comparisons, Merge2Compare has higher
accuracy but lacks fine-grained comparison (excluded by Top-similarity Pair
Removal), while Teach2Compare-general, although slightly less accurate, offers
more diverse scenarios and includes content information as background. Joint
training of both contributes to a more comprehensive quality comparison by our
model. Additionally, Teach2Compare includes a unique Q&A subset, which
significantly enhances the model’s ability to answer open-range questions.

##  4 The Co-Instruct Model

In this section, we discuss the proposed model, Co-Instruct. Specifically, we
have made two non-trivial adaptations for the multi-image comparative setting:

#### Visual Token Reduction (Fig. 6 (a)).

Most state-of-the-art LMMs [25, 39, 5] have adopted the simple projector that
keeps a large number of tokens (e.g. 1,025). This structure is not friendly to
multi-image scenarios: passing only two images will exceed the max length of
LLaVA [26] (2,048), and four images will exceed the context window of LLaMA-2
[42] (4,096). Thus, we adopt another widely-used abstractor [4, 55, 56]
structure to reduce token numbers before feeding the visual embeddings to LLM,
so as to easily adapt to multi-image scenarios.

#### Image-text Interleaved Format (Fig. 6 (c)).

Typical single-image instruction tuning usually does not care about “position
of images”. Most approaches [68, 25, 4] directly pile all images before texts
(<img${}_{\texttt{0}}$>(<img${}_{\texttt{1}}$>$\dots$)<text>). Under this
piling, multiple images are not separates and LMMs might confuse the
information from different images and fail to compare well (see baseline
result in Fig. 8). To solve this, we propose an image-text interleaved format
for multi-image training, that each image is started with explicit text to
identify its nominal:

User: The first image: <img${}_{\texttt{0}}$> The second image:
<img${}_{\texttt{1}}$> ($\dots$) <query>
Assistant: <response>

In our experiments, we demonstrated that this interleaved format significantly
enhances the performance of Co-Instruct (Tab. 8), notably better than using
learnable special tokens (<img_st> and <img_end>) to divide images.

Figure 7: Dataset Card of MICBench, made up of (a) Which questions (60%), (b)
Yes-or-No questions (22%), and (c) Other types of questions (18%) on
three/four images.

##  5 The MICBench

In this section, we discuss the proposed MICBench to cover the open-ended
evaluation settings on groups of three or four images, as a complementary of
existing evaluation settings (Sec. 6.3). It contains 2,000 groups of open-
range questions equipped with multiple candidates, with details elaborated as
follows:

#### Sourcing Diverse Image Groups.

To improve the diversity of the benchmark, in the MICBench, we sample image
groups from two sources: (1) 400 groups of three and 400 groups of four from
the images in LLVisionQA [49], which are originally sourced from 9 datasets
[18, 66, 6, 12, 59, 7, 24, 15, 3]; (2) 600 groups of three and 600 groups of
four on 1,000 random images sampled from unlabeled databases [41, 24, 3, 53]
(zero overlap with training-set images). With in-total 2,000 groups, the
MICBench contains a wide variety of quality issues and low-level appearances,
providing a non-biased evaluation on quality comparison.

#### Evaluation Form: Multi-choice Questions (MCQs).

As the most popular evaluation form for LLM/LMM benchmarks [11, 60, 29, 49],
multi-choice question (MCQ) is adopted as the evaluation form of MICBench. As
is shown in Fig. 7, each image group is associated with a expert-crafted
question that compare quality or related attributes among the images. Despite
common question types (Yes-or-No/What/How, etc), the MICBench also introduces
a special type of question, the Which questions (Fig. 7(a)), to cover this
common type of human query on comparison. In total 10 human experts
participate in annotating the MICBench, and the answer of each MCQ is cross-
examined by another expert. Similar as existing benchmarks [29, 49], MICBench
is further divided into a dev set (1,004) for method development (answers will
be public), and a test set (996) to evaluate performance of LMMs (answers will
be hidden from public).

##  6 Evaluation

###  6.1 Implementation Details

The Co-Instruct is fine-tuned after the released checkpoint of mPLUG-Owl2
[56], with LLaMA-2 [42] as LLM and CLIP-ViT-L14 [36] as visual embedding
module. Images are padded to square and then resized to $448\times 448$ before
fed into the model. The learning rate is set as $2e$-$5$, with two epochs
under batch size $192$. The final checkpoint is used for evaluation. To avoid
over-fitting, only dev subsets of evaluation datasets are used to choose best
training hyper-parameters, where the final reported results are from non-
overlapped test subsets. All parameters are updated during training, costing
in total 25 hours on 8*NVIDIA A100 GPUs.

###  6.2 Baseline Models

We choose five open-source recent state-of-the-art LMMs that supports multi-
image inputs to compare with: LLaVA-v1.5-13B [25], InternLM-XComposer2 [5],
BakLLaVA [39], EMU2-Chat [40], mPLUG-Owl2 [56] (baseline of Co-Instruct).
Additionally, we also compare with three well-recognized proprietary close-
source models: Qwen-VL-Max, Gemini-Pro, and GPT-4V (teacher of Co-Instruct).

###  6.3 Results on Existing Evaluation Settings

Despite the MICBench (Sec. 5), we also evaluate the proposed Co-Instruct
against baseline models on several existing visual quality
evaluation/comparison benchmarks for LMMs. The evaluation settings and results
are as follows.

Table 2: Results on Q-Bench${}^{\text{\tt PAIR}}$-A1. Co-Instruct is
remarkably 51% better than the variant without comparative data, and the only
LMM that surpasses human capability.

Sub-categories | Question Types | Low-level Concerns | Pairwise Settings | Overall$\uparrow$
---|---|---|---|---
1-8 Model | Yes-or-No$\uparrow$ | What$\uparrow$ | How$\uparrow$ | Distortion$\uparrow$ | Other$\uparrow$ | Compare$\uparrow$ | Joint$\uparrow$
random guess accuracy | 50.00% | 32.03% | 33.16% | 38.95% | 41.95% | 38.69% | 43.70% | 39.82%
1-9 (Sep/2023) LLaVA-v1.5-13B | 57.34% | 47.45% | 49.13% | 49.01% | 59.51% | 52.06% | 52.00% | 52.05%
(Oct/2023) BakLLava | 60.09% | 45.42% | 50.86% | 53.09% | 58.82% | 54.52% | 55.55% | 52.75%
(Nov/2023) mPLUG-Owl2 (baseline of Co-Instruct) | 58.07% | 36.61% | 48.44% | 47.74% | 51.90% | 45.73% | 60.00% | 48.94%
(Dec/2023) Emu2-Chat | 51.94% | 29.78% | 53.84% | 42.01% | 55.71% | 46.26% | 49.09% | 47.08%
(Feb/2024) InternLM-XComposer2-VL | 71.81% | 58.64% | 62.28% | 65.77% | 63.67% | 64.34% | 68.00% | 65.16%
Qwen-VL-Max (Proprietary) | 67.65% | 67.56% | 65.35% | 69.09% | 61.18% | 68.65% | 61.29% | 66.99%
Gemini-Pro (Proprietary) | 65.78% | 56.61% | 56.74% | 60.42% | 60.55% | 60.46% | 60.44% | 60.46%
GPT-4V (Proprietary, teacher of Co-Instruct) | 79.75% | 69.49% | 84.42% | 77.32% | 79.93% | 81.00% | 68.00% | 78.07%
Non-expert Human | 78.11% | 77.04% | 82.33% | 78.17% | 77.22% | 80.26% | 76.39% | 80.12%
without Multi-image Comparative Data | 60.24% | 47.46% | 48.78% | 52.81% | 53.97% | 51.42% | 59.11% | 53.15%
Co-Instruct (Ours) | 86.50% | 72.20% | 79.23% | 80.00% | 80.62% | 81.91% | 74.22% | 80.18%

Q-Bench${}^{\text{\tt PAIR}}$-A1[67] is a benchmark for visual quality
comparison with 1,999 expert-crafted open-range quality-related MCQs on image
pairs. In Tab. 2, we compare Co-Instruct against existing open-source and
proprietary models on this benchmark. Co-Instruct shows far superior accuracy
than open-source LMMs: it is 64% better than its baseline (mPLUG-Owl2), 51%
better than the variant without our multi-image subsets (Merge2Compare and
Teach2Compare), and also 23% better than the best of them. It also outperforms
Qwen-VL-Max and Gemini-Pro by a large margin (21%/33%). Additionally, though
its all MCQ training data are from GPT-4V, the student (Co-Instruct) still
outperforms its teacher on this MCQ evaluation set by notable 2.7%, suggesting
the effectiveness of the collaborative teaching strategy. Our model is also
the only LMM that surpasses the accuracy of a non-expert human (esp. on
Compare subset) in this benchmark, strongly supporting the meaningful vision
of using models to relieve human labors on real-world visual quality
comparisons in the future.

Table 3: Results on Q-Bench${}^{\text{\tt PAIR}}$-A2. $P_{i}$ denotes
frequency for score $i$ (score in $[0,2]$). While slightly inferior to GPT-4V,
Co-Instruct has significantly improved over both the variant without
comparative data (+31%), especially for Precision metric (+59%).

Dimensions | Completeness | Precision | Relevance | Sum.$\uparrow$
---|---|---|---|---
1-13 Model | $P_{0}$ | $P_{1}$ | $P_{2}$ | score$\uparrow$ | $P_{0}$ | $P_{1}$ | $P_{2}$ | score$\uparrow$ | $P_{0}$ | $P_{1}$ | $P_{2}$ | score$\uparrow$
(Sep/2023) LLaVA-v1.5-13B | 18.77% | 73.44% | 7.79% | 0.89 | 34.66% | 38.72% | 26.62% | 0.92 | 1.02% | 34.59% | 64.39% | 1.63 | 3.44
(Oct/2023) BakLLava | 29.46% | 59.77% | 10.57% | 0.80 | 40.0% | 38.08% | 21.33% | 0.80 | 2.26% | 15.06% | 82.04% | 1.79 | 3.40
(Nov/2023) mPLUG-Owl2 (baseline) | 19.43% | 65.54% | 14.45% | 0.94 | 30.94% | 43.71% | 24.63% | 0.92 | 3.79% | 26.94% | 68.28% | 1.63 | 3.50
(Dec/2023) Emu2-Chat | 41.25% | 54.33% | 4.42% | 0.63 | 38.11% | 36.41% | 25.48% | 0.87 | 4.12% | 38.61% | 57.27% | 1.53 | 3.03
(Feb/2024) InternLM-XComposer2-VL | 13.20% | 72.17% | 14.13% | 1.00 | 31.28% | 42.13% | 25.77% | 0.93 | 1.60% | 24.17% | 72.93% | 1.70 | 3.64
Qwen-VL-Max (Proprietary) | 11.64% | 54.08% | 34.08% | 1.22 | 24.26% | 39.15% | 36.22% | 1.11 | 2.533% | 10.97% | 85.64% | 1.82 | 4.16
Gemini-Pro (Proprietary) | 18.22% | 44.48% | 36.84% | 1.18 | 34.13% | 37.95% | 27.02% | 0.92 | 0.67% | 5.91% | 92.22% | 1.90 | 4.00
GPT-4V (Proprietary, teacher of Ours) | 4.09% | 31.82% | 64.09% | 1.60 | 10.44% | 45.12% | 44.44% | 1.34 | 0.18% | 1.69% | 96.35% | 1.94 | 4.89
w/o Multi-Image Comparative Data | 15.25% | 65.76% | 18.32% | 1.02 | 39.44% | 40.18% | 19.62% | 0.79 | 0.09% | 9.86% | 89.02% | 1.87 | 3.69
Co-Instruct (Ours) | 4.04% | 31.55% | 63.55% | 1.58 | 13.68% | 43.68% | 41.37% | 1.26 | 0.0% | 0.44% | 98.22% | 1.96 | 4.82

Figure 8: Qualitative Visualization on Q-Bench${}^{\text{\tt PAIR}}$-A2.
GPT-4V gives longest outputs and achieves high precision score even if it
includes incorrect information.

Q-Bench${}^{\text{\tt PAIR}}$-A2 is a benchmark setting for general and
detailed visual quality comparison with detailed reasonings on image pairs.
Consisting of 499 image pairs, it employs GPT to evaluate LMM responses
against the golden expert-labeled comparisons on Completeness, Precision, and
Relevance. As listed in Tab. 3, the Co-Instruct can remarkably improve the
Completeness (+57%) and Precision (+59%) of the comparative outputs than the
w/o comparative data version, but still falls a little bit behind GPT-4V on
the quantitative metrics. This might be because outputs from GPT-4V are more
than twice as long as Co-Instruct outputs, while GPT evaluation used here is
observed [22] to be in favor of longer text outputs. To further analyze this
potential bias, we qualitatively visualize the result of different LMMs in
Fig. 8. As shown in the figure, the baseline open-source LMM even confuses the
information from the two images, and Gemini-Pro makes rather poor detailed
reasonings. For GPT-4V, it generates the longest outputs among all LMMs, which
might be the reason that it gets a relatively high precision score even its
outputs are not totally correct. In short, the capability of Co-Instruct in
reasoning-related comparisons can match that of GPT-4V, while significantly
surpassing other existing LMMs.

Table 4: Results on 2AFC-LMM. $\kappa$ denotes binary judgment consistency
while swapping first image and second image; $\rho$ denotes Pearson’s linear
correlation.

Dataset | CSIQ | MM21 | KADID-10k | LIVEC | KonIQ-10k | SPAQ | weighted avg.
---|---|---|---|---|---|---|---
Model | $\kappa$ | $\rho$ | $\kappa$ | $\rho$ | $\kappa$ | $\rho$ | $\kappa$ | $\rho$ | $\kappa$ | $\rho$ | $\kappa$ | $\rho$ | $\kappa$ | $\rho$
(Aug/2023) IDEFICS-Instruct-9B | 0.206 | 0.570 | 0.337 | 0.338 | 0.202 | 0.552 | 0.323 | 0.492 | 0.251 | 0.479 | 0.330 | 0.474 | 0.286 | 0.470
(Sep/2023) LLaVA-v1.5-13B | 0.483 | 0.423 | 0.356 | 0.149 | 0.310 | 0.137 | 0.273 | 0.162 | 0.262 | 0.403 | 0.291 | 0.156 | 0.302 | 0.224
(Oct/2023) BakLLava | 0.356 | 0.235 | 0.337 | 0.244 | 0.245 | 0.166 | 0.296 | 0.159 | 0.185 | 0.217 | 0.274 | 0.146 | 0.261 | 0.185
(Nov/2023) mPLUG-Owl2 (baseline) | 0.435 | 0.627 | 0.378 | 0.306 | 0.402 | 0.443 | 0.375 | 0.441 | 0.386 | 0.417 | 0.362 | 0.356 | 0.460 | 0.397
(Feb/2024) InternLM-XComposer2-VL | 0.800 | 0.527 | 0.688 | 0.377 | 0.600 | 0.552 | 0.600 | 0.516 | 0.825 | 0.581 | 0.700 | 0.755 | 0.705 | 0.567
Qwen-VL-Max (Proprietary) | 0.540 | 0.418 | 0.497 | 0.304 | 0.625 | 0.406 | 0.578 | 0.544 | 0.631 | 0.610 | 0.592 | 0.718 | 0.592 | 0.540
Gemini-Pro (Proprietary) | 0.672 | 0.527 | 0.604 | 0.377 | 0.790 | 0.552 | 0.650 | 0.516 | 0.652 | 0.581 | 0.671 | 0.755 | 0.678 | 0.622
GPT-4V (Proprietary, teacher of Ours) | 0.778 | 0.764 | 0.792 | 0.474 | 0.763 | 0.560 | 0.837 | 0.685 | 0.835 | 0.800 | 0.871 | 0.876 | 0.823 | 0.721
w/o Multi-Image Comparative Data | 0.117 | 0.650 | 0.480 | 0.392 | 0.397 | 0.466 | 0.327 | 0.432 | 0.489 | 0.512 | 0.485 | 0.397 | 0.432 | 0.449
Co-Instruct (Ours) | 0.800 | 0.779 | 0.852 | 0.325 | 0.829 | 0.685 | 0.872 | 0.797 | 0.883 | 0.927 | 0.881 | 0.931 | 0.864 | 0.754

2AFC-LMM[69] is a benchmark setting for general quality comparison on image
pairs. It prompts LMMs to make a two-alternative forced choice (2AFC) on a
pair of images. The maximum a posterior estimation is utilized to aggregate
comparative preferences to single-image quality scores [43]. Then, it computes
Peason’s linear correlation ($\rho$) between regressed scores and ground truth
MOS. As shown in Tab. 4, Co-Instruct outperforms all existing models in 2AFC-
LMM, including GPT-4V. Co-Instruct also shows very high consistency $\kappa$
while swapping two images. Among all datasets, the proposed model is only
inferior on the MM21 [23]. Nonetheless, we observe that the Co-Instruct has
higher direct comparison accuracy than GPT-4V (Co-Instruct: 55.2%, GPT-4V:
54.4%, see supplementary for full results) on it, yet the dataset contains a
large proportion of extremely similar pairs, for which Co-Instruct responds a
forced choice but GPT-4V will answer “two images have similar quality” (tie),
which impacts the aggregated single-image quality scores. We hope this
observation can inspire further research to design better evaluation settings
for fine-grained comparisons.

Table 5: Results on Q-Bench${}^{\text{\tt SINGLE}}$-A1, proving that the
comparative data (Sec. 3) can also effectively boost the capability of LMMs on
single image quality evaluation.

Sub-categories | Question Types | Quadrants of Low-level Concerns | Overall$\uparrow$
---|---|---|---
1-8 Model | Yes-or-No$\uparrow$ | What$\uparrow$ | How$\uparrow$ | Distortion$\uparrow$ | Other$\uparrow$ | In-context | In-context
Distortion$\uparrow$ | Other$\uparrow$
random guess accuracy | 50.00% | 28.48% | 33.30% | 37.24% | 38.50% | 39.13% | 37.10% | 37.94%
1-9 (Sep/2023) LLaVA-v1.5-13B | 64.96% | 64.86% | 54.12% | 53.55% | 66.59% | 58.90% | 71.48% | 61.40%
(Oct/2023) BakLLava | 66.46% | 61.48% | 54.83% | 51.33% | 63.76% | 56.52% | 78.16% | 61.02%
(Nov/2023) mPLUG-Owl2 (baseline of Co-Instruct) | 72.26% | 55.53% | 58.64% | 52.59% | 71.36% | 58.90% | 73.00% | 62.68%
(Dec/2023) Emu2-Chat | 70.09% | 65.12% | 54.11% | 66.22% | 62.96% | 63.47% | 73.21% | 64.32%
(Feb/2024) InternLM-XComposer2-VL | 72.44% | 78.13% | 67.28% | 68.00% | 75.65% | 68.15% | 81.36% | 72.52%
Qwen-VL-Max (Proprietary) | 73.20% | 81.02% | 68.39% | 70.84% | 74.57% | 73.11% | 80.44% | 73.90%
Gemini-Pro (Proprietary) | 71.26% | 71.39% | 65.59% | 67.30% | 73.04% | 65.88% | 73.60% | 69.46%
GPT-4V (Proprietary, teacher of Co-Instruct) | 77.72% | 78.39% | 66.45% | 71.01% | 71.07% | 79.36% | 78.91% | 74.10%
Non-expert Human | 82.48% | 79.39% | 60.29% | 75.62% | 72.08% | 76.37% | 73.00% | 74.31%
without Multi-image Comparative Data | 79.38% | 72.23% | 67.70% | 68.71% | 72.32% | 73.97% | 83.65% | 73.38%
Co-Instruct (Ours) | 81.93% | 78.74% | 70.16% | 74.28% | 76.37% | 76.71% | 84.41% | 77.12%

Q-Bench${}^{\text{\tt SINGLE}}$-A1. Despite the comparative benchmarks above,
we also evaluate Co-Instruct on single image MCQs from Q-Bench${}^{\text{\tt
SINGLE}}$-A1 to verify the influences of comparative tuning on single-image
quality perception. As shown in Tab. 5, Co-Instruct shows 5% improvement over
the variant trained with single images only, leading GPT-4V by 4%, and marks
the only LLM that surpasses non-expert human. These results have demonstrated
the contribution of comparative training on general quality-related
understanding of LMMs, and suggested that single-image quality evaluation does
not conflict with multi-image quality comparison for LMMs and can be improved
together under a unified model.

###  6.4 Results on MICBench

Table 6: Results on MICBench. Co-Instruct is 60% better than the variant
without comparative data, and also notably better than GPT-4V (+5.7%) and
human (+6.4%).

Sub-categories | Question Types | Number of Images | Overall$\uparrow$
---|---|---|---
1-6 Model | Yes-or-No$\uparrow$ | Which$\uparrow$ | Others$\uparrow$ | Three$\uparrow$ | Four$\uparrow$
#questions | 220 | 594 | 182 | 503 | 493 | 996
random guess accuracy | 49.55% | 28.59% | 28.31% | 34.10% | 29.17% | 31.47%
(Sep/2023) LLaVA-v1.5-13B (length: 2048$\to$2560) | 47.51% | 40.74% | 52.49% | 46.81% | 41.90%* | 44.38%
(Oct/2023) BakLLava (length: 2048$\to$2560) | 68.35% | 35.01% | 52.78% | 48.51% | 42.54%* | 45.56%
(Nov/2023) mPLUG-Owl2 (baseline of Co-Instruct) | 62.25% | 35.70% | 53.71% | 44.19% | 45.42% | 44.80%
(Feb/2024) InternLM-XComposer2-VL (length: 4096$\to$5120) | 62.95% | 47.29% | 52.02% | 55.70% | 46.51%* | 51.76%
Qwen-VL-Max (Proprietary) | 62.33% | 70.00% | 81.54% | 72.35% | 68.79% | 70.55%
Gemini-Pro (Proprietary) | 75.00% | 67.37% | 66.92% | 68.71% | 70.87% | 69.79%
GPT-4V (Proprietary, teacher of Co-Instruct) | 80.32% | 77.28% | 78.82% | 80.32% | 77.28% | 78.82%
Non-expert Human | 82.27% | 78.15% | 74.31% | 77.18% | 79.55% | 78.35%
without Multi-image Comparative Data | 62.72% | 37.54% | 53.30% | 45.33% | 46.65% | 45.98%
Co-Instruct (Ours) | 79.55% | 85.35% | 81.32% | 84.69% | 81.94% | 83.33%

As is shown in Tab. 6, Co-Instruct provides very competitive accuracy on open-
question quality comparison among three/four images, 5.7% better than GPT-4V
(best existing) and 6.4% more accurate than non-expert human; open-source LMMs
even struggle to obtain 50% accuracy on this setting. It is also noteworthy
that LLaVA series and XComposer2-VL’s original context lengths are not enough
for four images as they have not reduced visual token numbers, so we have to
extend their context windows to evaluate them for MICBench. Consequentially,
all these models have experienced notably worse accuracy on groups of four (on
extended context length) than groups of three (within original context
length), as noted in * in Tab. 6. This degradation highlights the importance
to reduce visual tokens (Sec. 4) to adapt to multi-image scenarios.

###  6.5 Ablation Studies

Ablation on Training Data. As the proposed dataset is composed of three parts,
we discuss the effects on different subsets of data on the six evaluation
scenarios in Tab. 8. Through the results, we have arrived at several important
conclusions: A) Even with all information from Q-Instruct-200K, the
incorporation of Merge2Compare (variant (2)) still notably enhances the
capability across various settings; B) Only involving the Teach2Compare
(_i.e_. GPT-4V labels, variant (5)) cannot outperform its teacher GPT-4V; C)
instead, the superiority towards GPT-4V on question-answering benchmarks is
benefited from the co-instruction on hetero-sourced training subsets (variants
(6) and (7)).

Table 7: Ablation Study on data. Abbreviation: Merge: Merge2Compare;
Teach${}^{\textit{G}}$: Teach2Compare-general; Teach${}^{\textit{QA}}$:
Teach2Compare-Q&A; QIn: Q-Instruct-200K.

Training Subset | Evaluation Scenario
---|---
Variant No. | QIn | Merge | Teach${}^{\textit{G}}$ | Teach${}^{\textit{QA}}$ | Q-Bench${}^{\texttt{PAIR}}$-A1 | Q-Bench${}^{\texttt{PAIR}}$-A2 | Q-Bench${}^{\texttt{SINGLE}}$-A1 | 2AFC-LMM | MICBench
Reference Results of GPT-4V | 78.07 | 4.89 | 74.10 | 0.721 | 78.82
(1) | ✓ |  |  |  | 53.15 | 3.69 | 73.38 | 0.449 | 45.98
(2) | ✓ | ✓ |  |  | 68.67 | 4.67 | 75.12 | 0.701 | 60.34
(3) | ✓ |  | ✓ |  | 65.44 | 4.64 | 74.38 | 0.647 | 64.82
(4) | ✓ | ✓ | ✓ |  | 69.87 | 4.82 | 76.52 | 0.749 | 66.37
(5) | ✓ |  | ✓ | ✓ | 78.28 | 4.65 | 75.72 | 0.676 | 76.41
(6) | ✓ | ✓ |  | ✓ | 80.08 | 4.68 | 75.65 | 0.728 | 81.82
(7, full) | ✓ | ✓ | ✓ | ✓ | 80.18 | 4.82 | 77.31 | 0.754 | 83.33

Table 8: Ablation Study on the text-image interleaved format for the Co-
Instruct.

Format | Q-Bench${}^{\texttt{PAIR}}$-A1 | Q-Bench${}^{\texttt{PAIR}}$-A2 | Q-Bench${}^{\texttt{SINGLE}}$-A1 | 2AFC-LMM | MICBench
---|---|---|---|---|---
<img${}_{\texttt{0}}$>(<img${}_{\texttt{1}}$> ...) (baseline, popular strategy [4, 25, 68]) | 76.37 | 4.73 | 74.11 | 0.729 | 78.92
<img_st><img${}_{\texttt{0}}$><img_end> (<img_st><img${}_{\texttt{1}}$><img_end>$\dots$) | 76.77 | 4.79 | 73.85 | 0.736 | 80.12
The input image: <img${}_{\texttt{0}}$> (The input image: <img${}_{\texttt{1}}$>$\dots$) | 78.28 | 4.80 | 75.12 | 0.749 | 82.22
The first image: <img${}_{\texttt{0}}$> (The second image: <img${}_{\texttt{1}}$>$\dots$) | 80.18 | 4.82 | 77.31 | 0.754 | 83.33

Figure 9: Effects of Data Scaling for (a) detailed reasoning and (b) question
answering.

Besides the composition of three subsets, we also explore the effects of data
scale: in Fig. 9(a), we confirm that more general comparison data contributes
to an increase in detailed reasoning capabilities, and mixed data has a better
effect than homogenous data at the same scale; in Fig. 9(b), we also validate
that scaling up the Q&A subset helps to improve multiple MCQ metrics.

Ablation on Interleaved Format. As shown in Tab. 8, the proposed text-image
interleaved format has proved non-negligible advantages than the baseline, as
well as other variants without explicitly noting image orders. The results
suggest the rationale of the format on multi-image comparative settings.

##  7 Conclusion

In this work, we investigate the open-ended visual quality comparison problem,
with the aim of a model that provides answers and detailed reasonings on open-
range questions that compares quality among multiple images. To achieve this,
we collect the first instruction-tuning dataset to fine-tune large multi-
modality models (LMMs) for comparison, the Co-Instruct-562K, with two subsets
from human annotations on single images (merged by LLMs), and GPT-4V
responses. With the dataset, we propose the Co-Instruct, which not only
outperforms all existing LMMs (including its teacher, GPT-4V) on visual
quality comparison, but also marks the first LMM with the capability to
surpass human accuracy on related settings. We further construct the MICBench,
the first benchmark that evaluates multi-image quality comparison for LMMs on
three and four images. We expect our work to motivate future studies on visual
quality comparison.

## References

  * [1] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: VQA: Visual Question Answering. In: IEEE ICCV. pp. 2425–2433 (2015)
  * [2] Chen, L., Li, J., Dong, X., Zhang, P., He, C., Wang, J., Zhao, F., Lin, D.: ShareGPT4V: Improving large multi-modal models with better captions. CoRR abs/2311.12793 (2023)
  * [3] Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: Microsoft COCO captions: Data collection and evaluation server. CoRR abs/1504.00325 (2015)
  * [4] Dai, W., Li, J., Li, D., Tiong, A.M.H., Zhao, J., Wang, W., Li, B., Fung, P., Hoi, S.: InstructBLIP: Towards general-purpose vision-language models with instruction tuning. CoRR abs/2305.06500 (2023)
  * [5] Dong, X., Zhang, P., Zang, Y., Cao, Y., Wang, B., Ouyang, L., Wei, X., Zhang, S., Duan, H., Cao, M., Zhang, W., Li, Y., Yan, H., Gao, Y., Zhang, X., Li, W., Li, J., Chen, K., He, C., Zhang, X., Qiao, Y., Lin, D., Wang, J.: InternLM-XComposer2: Mastering free-form text-image composition and comprehension in vision-language large model. CoRR abs/2401.16420 (2024)
  * [6] Fang, Y., Zhu, H., Zeng, Y., Ma, K., Wang, Z.: Perceptual quality assessment of smartphone photography. In: IEEE CVPR. pp. 3677–3686 (2020)
  * [7] Ghadiyaram, D., Bovik, A.C.: Massive online crowdsourced study of subjective and objective picture quality. IEEE TIP 25(1), 372–387 (2016)
  * [8] Golestaneh, S.A., Dadsetan, S., Kitani, K.M.: No-reference image quality assessment via transformers, relative ranking, and self-consistency. In: IEEE WACV. pp. 3209–3218 (2022)
  * [9] Google: Gemini Pro (2023), https://deepmind.google/technologies/gemini
  * [10] Gu, J., Cai, H., Chen, H., Ye, X., Ren, J., Dong, C.: PIPAL: A large-scale image quality assessment dataset for perceptual image restoration. In: ECCV. pp. 633–651 (2020)
  * [11] Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., Steinhardt, J.: Measuring massive multitask language understanding. In: ICLR. pp. 1–10 (2021)
  * [12] Hosu, V., Lin, H., Sziranyi, T., Saupe, D.: Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment. IEEE TIP 29, 4041–4056 (2020)
  * [13] Hudson, D.A., Manning, C.D.: GQA: A new dataset for real-world visual reasoning and compositional question answering. In: IEEE CVPR. pp. 6700–6709 (2019)
  * [14] ITU-R, B.T.: Methodology for the subjective assessment of the quality of television pictures. https://www.itu.int/rec/R-REC-BT.500 (2002)
  * [15] Jayaraman, D., Mittal, A., Moorthy, A.K., Bovik, A.C.: Objective quality assessment of multiply distorted images. In: ASILOMAR. pp. 1693–1697 (2012)
  * [16] Jiang, A.Q., Sablayrolles, A., Roux, A., Mensch, A., Savary, B., Bamford, C., Chaplot, D.S., de las Casas, D., Hanna, E.B., Bressand, F., Lengyel, G., Bour, G., Lample, G., Lavaud, L.R., Saulnier, L., Lachaux, M.A., Stock, P., Subramanian, S., Yang, S., Antoniak, S., Scao, T.L., Gervet, T., Lavril, T., Wang, T., Lacroix, T., Sayed, W.E.: Mixtral of experts. CoRR abs/2401.04088 (2024)
  * [17] LAION: LAION GPT-4V dataset. https://huggingface.co/datasets/laion/gpt4v-dataset (2023)
  * [18] Li, C., Zhang, Z., Wu, H., Sun, W., Min, X., Liu, X., Zhai, G., Lin, W.: AGIQA-3K: An open database for ai-generated image quality assessment. CoRR 2306.04717 (2023)
  * [19] Li, D., Jiang, T., Jiang, M.: Quality assessment of in-the-wild videos. In: ACM MM. pp. 2351–2359 (2019)
  * [20] Li, D., Jiang, T., Lin, W., Jiang, M.: Which has better visual quality: The clear blue sky or a blurry animal? IEEE TMM 21(5), 1221–1234 (2019)
  * [21] Li, J., Mantiuk, R., Wang, J., Ling, S., Le Callet, P.: Hybrid-MST: A hybrid active sampling strategy for pairwise preference aggregation. In: NeurIPS. pp. 1–11 (2018)
  * [22] Li, X., Zhang, T., Dubois, Y., Taori, R., Gulrajani, I., Guestrin, C., Liang, P., Hashimoto, T.B.: AlpacaEval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval (2023)
  * [23] Li, Y., Wang, S., Zhang, X., Wang, S., Ma, S., Wang, Y.: Quality assessment of end-to-end learned image compression: The benchmark and objective measure. In: ACM MM. pp. 4297–4305 (2021)
  * [24] Lin, H., Hosu, V., Saupe, D.: KADID-10k: A large-scale artificially distorted iqa database. In: QoMEX. pp. 1–3 (2019)
  * [25] Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. CoRR abs/2310.03744 (2023)
  * [26] Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. CoRR abs/2304.08485 (2023)
  * [27] Liu, X., Van De Weijer, J., Bagdanov, A.D.: RankIQA: Learning from rankings for no-reference image quality assessment. In: IEEE ICCV. pp. 1040–1049 (2017)
  * [28] Liu, X., Van De Weijer, J., Bagdanov, A.D.: Exploiting unlabeled data in cnns by self-supervised learning to rank. IEEE TPAMI 41(8), 1862–1878 (2019)
  * [29] Liu, Y., Duan, H., Zhang, Y., Li, B., Zhang, S., Zhao, W., Yuan, Y., Wang, J., He, C., Liu, Z., Chen, K., Lin, D.: MMBench: Is your multi-modal model an all-around player? CoRR abs/2307.06281 (2023)
  * [30] Ma, K., Liu, W., Liu, T., Wang, Z., Tao, D.: dipIQ: Blind image quality assessment by learning-to-rank discriminable image pairs. IEEE TIP 26(8), 3951–3964 (2017)
  * [31] Mantiuk, R.K., Tomaszewska, A., Mantiuk, R.: Comparison of four subjective methods for image quality assessment. In: Computer Graphics Forum. vol. 31, pp. 2478–2491 (2012)
  * [32] Michaelis, C., Mitzkus, B., Geirhos, R., Rusak, E., Bringmann, O., Ecker, A.S., Bethge, M., Brendel, W.: Benchmarking robustness in object detection: Autonomous driving when winter is coming. CoRR abs/1907.07484 (2019)
  * [33] Mikhailiuk, A., Wilmot, C., Perez-Ortiz, M., Yue, D., Mantiuk, R.K.: Active sampling for pairwise comparisons via approximate message passing and information gain maximization. In: IEEE ICPR. pp. 2559–2566 (2021)
  * [34] OpenAI: Gpt-4 technical report (2023)
  * [35] Prashnani, E., Cai, H., Mostofi, Y., Sen, P.: PieAPP: Perceptual image-error assessment through pairwise preference. In: IEEE CVPR. pp. 1808–1817 (2018)
  * [36] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763 (2021)
  * [37] Rajkumar, A., Agarwal, S.: When can we rank well from comparisons of o (nlog (n)) non-actively chosen pairs? In: Conference on Learning Theory. pp. 1376–1401 (2016)
  * [38] Schwenk, D., Khandelwal, A., Clark, C., Marino, K., Mottaghi, R.: A-OKVQA: A benchmark for visual question answering using world knowledge. In: ECCV. pp. 146–162 (2022)
  * [39] SkunkworksAI: BakLLaVA (2024), https://github.com/SkunkworksAI/BakLLaVA
  * [40] Sun, Q., Cui, Y., Zhang, X., Zhang, F., Yu, Q., Luo, Z., Wang, Y., Rao, Y., Liu, J., Huang, T., et al.: Generative multimodal models are in-context learners. CoRR abs/2312.13286 (2023)
  * [41] Thomee, B., Shamma, D.A., Friedland, G., Elizalde, B., Ni, K., Poland, D., Borth, D., Li, L.J.: YFCC100M: The new data in multimedia research. Commun. ACM 59(2), 64–73 (2016)
  * [42] Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C.C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P.S., Lachaux, M.A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E.M., Subramanian, R., Tan, X.E., Tang, B., Taylor, R., Williams, A., Kuan, J.X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., Scialom, T.: Llama 2: Open foundation and fine-tuned chat models. CoRR abs/2307.09288 (2023)
  * [43] Tsukida, K., Gupta, M.R.: How to analyze paired comparison data (Technical Report UWEETR-2011-0004, University of Washington, 2011), https://api.semanticscholar.org/CorpusID:15425240
  * [44] Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., Wei, F.: Improving text embeddings with large language models. CoRR abs/2401.00368 (2024)
  * [45] Wauthier, F., Jordan, M., Jojic, N.: Efficient ranking from pairwise comparisons. In: ICML. pp. 109–117 (2013)
  * [46] Wu, H., Chen, C., Hou, J., Liao, L., Wang, A., Sun, W., Yan, Q., Lin, W.: FAST-VQA: Efficient end-to-end video quality assessment with fragment sampling. In: ECCV. pp. 538–554 (2022)
  * [47] Wu, H., Chen, C., Liao, L., Hou, J., Sun, W., Yan, Q., Gu, J., Lin, W.: Neighbourhood representative sampling for efficient end-to-end video quality assessment. IEEE TPAMI (2023)
  * [48] Wu, H., Zhang, E., Liao, L., Chen, C., Hou, J., Wang, A., Sun, W., Yan, Q., Lin, W.: Exploring video quality assessment on user generated contents from aesthetic and technical perspectives. In: IEEE ICCV (2023)
  * [49] Wu, H., Zhang, Z., Zhang, E., Chen, C., Liao, L., Wang, A., Li, C., Sun, W., Yan, Q., Zhai, G., Lin, W.: Q-Bench: A benchmark for general-purpose foundation models on low-level vision. In: ICLR (2024)
  * [50] Wu, H., Zhang, Z., Zhang, E., Chen, C., Liao, L., Wang, A., Xu, K., Li, C., Hou, J., Zhai, G., et al.: Q-instruct: Improving low-level visual abilities for multi-modality foundation models. CoRR abs/2311.06783 (2023)
  * [51] Wu, H., Zhang, Z., Zhang, W., Chen, C., Liao, L., Li, C., Gao, Y., Wang, A., Zhang, E., Sun, W., et al.: Q-Align: Teaching lmms for visual scoring via discrete text-defined levels. CoRR abs/2312.17090 (2023)
  * [52] Wu, X., Sun, K., Zhu, F., Zhao, R., Li, H.: Better aligning text-to-image models with human preference. CoRR abs/2303.14420 (2023)
  * [53] Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., Dong, Y.: ImageReward: Learning and evaluating human preferences for text-to-image generation. CoRR abs/2304.05977 (2023)
  * [54] Ye, P., Doermann, D.: Active sampling for subjective image quality assessment. In: IEEE CVPR. pp. 4249–4256 (2014)
  * [55] Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., Jiang, C., Li, C., Xu, Y., Chen, H., Tian, J., Qi, Q., Zhang, J., Huang, F.: mPLUG-Owl: Modularization empowers large language models with multimodality. CoRR abs/2304.14178 (2023)
  * [56] Ye, Q., Xu, H., Ye, J., Yan, M., Liu, H., Qian, Q., Zhang, J., Huang, F., Zhou, J.: mPLUG-Owl2: Revolutionizing multi-modal large language model with modality collaboration. CoRR abs/2311.04257 (2023)
  * [57] Yim, J.G., Wang, Y., Birkbeck, N., Adsumilli, B.: Subjective quality assessment for youtube UGC dataset. In: IEEE ICIP. pp. 1–5 (2020)
  * [58] Yin, Z., Wang, J., Cao, J., Shi, Z., Liu, D., Li, M., Sheng, L., Bai, L., Huang, X., Wang, Z., et al.: LAMM: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. CoRR abs/2306.06687 (2023)
  * [59] Ying, Z., Niu, H., Gupta, P., Mahajan, D., Ghadiyaram, D., Bovik, A.: From patches to pictures (PaQ-2-PiQ): Mapping the perceptual space of picture quality. In: IEEE CVPR. pp. 3575–3585 (2020)
  * [60] Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., et al.: MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. CoRR abs/2311.16502 (2023)
  * [61] Zhang, C., Su, S., Zhu, Y., Yan, Q., Sun, J., Zhang, Y.: Exploring and evaluating image restoration potential in dynamic scenes. In: IEEE CVPR. pp. 2057–2066 (2022)
  * [62] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: IEEE CVPR. pp. 586–595 (2018)
  * [63] Zhang, W., Ma, K., Zhai, G., Yang, X.: Uncertainty-aware blind image quality assessment in the laboratory and wild. IEEE TIP 30, 3474–3486 (2021)
  * [64] Zhang, W., Zhai, G., Wei, Y., Yang, X., Ma, K.: Blind image quality assessment via vision-language correspondence: A multitask learning perspective. In: IEEE CVPR. pp. 14071–14081 (2023)
  * [65] Zhang, W., Liu, Y., Dong, C., Qiao, Y.: RankSRGan: Generative adversarial networks with ranker for image super-resolution. In: ICCV. pp. 3096–3105 (2019)
  * [66] Zhang, Z., Sun, W., Wang, T., Lu, W., Zhou, Q., Wang, Q., Min, X., Zhai, G., et al.: Subjective and objective quality assessment for in-the-wild computer graphics images. ACM TOMM 20(4), 1–22 (2023)
  * [67] Zhang, Z., Wu, H., Zhang, E., Zhai, G., Lin, W.: A benchmark for multi-modal foundation models on low-level vision: from single images to pairs. CoRR abs/2402.07116 (2024)
  * [68] Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M.: MiniGPT-4: Enhancing vision-language understanding with advanced large language models. CoRR abs/2304.10592 (2023)
  * [69] Zhu, H., Sui, X., Chen, B., Liu, X., Chen, P., Fang, Y., Wang, S.: 2AFC prompting of large multimodal models for image quality assessment. CoRR abs/2402.01162 (2024)


# On the Limits of Minimal Pairs in Contrastive Evaluation

Jannis Vamvas1 Rico Sennrich1,2
1Department of Computational Linguistics, University of Zurich
2School of Informatics, University of Edinburgh
<EMAIL_ADDRESS>
###### Abstract

Minimal sentence pairs are frequently used to analyze the behavior of language
models. It is often assumed that model behavior on contrastive pairs is
predictive of model behavior at large. We argue that two conditions are
necessary for this assumption to hold: First, a tested hypothesis should be
well-motivated, since experiments show that contrastive evaluation can lead to
false positives. Secondly, test data should be chosen such as to minimize
distributional discrepancy between evaluation time and deployment time. For a
good approximation of deployment-time decoding, we recommend that minimal
pairs are created based on machine-generated text, as opposed to human-written
references. We present a contrastive evaluation suite for English–German MT
that implements this recommendation.111https://github.com/ZurichNLP/distil-
lingeval

##  1 Introduction

Contrastive evaluation is one of the most widely used evaluation techniques
for generative language models, both for causal models Linzen et al. (2016)
and sequence-to-sequence models Sennrich (2017). Various phenomena have been
analyzed using this technique, including syntax (Marvin and Linzen 2018; among
others), word sense disambiguation (Rios et al. 2017; among others), document
coherence (Bawden et al. 2018; Beyer et al. 2021; among others), and
grammatical acceptability in general Warstadt et al. (2020); Xiang et al.
(2021).

Contrastive evaluation allows for a targeted, automated evaluation of
generative models but is restricted to a specific behavioral interface, namely
the ranking of pre-defined minimal pairs. However, most models in application
areas such as translation or conversation are deployed to produce 1-best
sequences, exposing a different behavioral interface to users. While this
limitation of contrastive evaluation is well known, its practical relevance
has been unclear.

We show that under certain conditions, the gap between evaluation and
deployment can indeed cause misleading results. As a main factor we identify
the distributional discrepancy of contrastive evaluation datasets: Minimal
pairs are usually derived from human-written references, but when deployed, a
model is conditioned on its own output.

To measure the effect of this factor on evaluation, we focus on neural machine
translation (NMT) systems. Our approach is to test implausible research
hypotheses in addition to plausible ones. We find that distributional
discrepancy increases the number of false positives regarding implausible
hypotheses. They particularly occur when evaluating distilled NMT models Kim
and Rush (2016), indicating that in such models, ranking behavior on noisy
sequences diverges from generative behavior.

We also propose a way to reduce the distributional discrepancy of minimal
pairs. Our experiments show that false positives can be largely avoided by
using machine-generated text instead of human-written text. This inspires us
to release DistilLingEval, a variant of the LingEval97 English–German MT
evaluation suite Sennrich (2017) that uses MT-generated references.

We recommend that future efforts to create contrastive datasets for the
evaluation of language generation models minimize distributional discrepancy
between evaluation and deployment. Due to the possibility of false positives,
linguistic conclusions about knowledge or abilities of models should be
corroborated by additional evidence from a more natural setting.

##  2 Background and Related Work

###  2.1 Contrastive Evaluation

Contrastive evaluation compares the probability scores that a model assigns to
two minimally different sequences. For example, the sentences “The cats sleep”
and “The cats sleeps” differ in verb number only; if a model assigns higher
scores to sentences of the first kind than to sentences of the second kind, it
is said to prefer verb forms in agreement with the noun Linzen et al. (2016).

An established method for scoring is to compute the score for a full sentence
$X=x_{0},x_{1},...,x_{n}$ as the sum of token log-probabilities predicted by
the model $\theta$ Marvin and Linzen (2018):

$\textrm{score}(X)=\sum_{i=0}^{n}\log p_{\theta}(x_{i}|x_{<i})$

When contrastive evaluation is applied to sequence-to-sequence models, two
target sequences are scored given the same source sequence $X$ Sennrich
(2017). We follow previous work and normalize sequence-to-sequence scores by
the length of the target sequence $Y$:

$\textrm{score}(Y|X)=\frac{1}{|Y|}\sum_{i=0}^{|Y|}\log
p_{\theta}(y_{i}|X,y_{<i})$

###  2.2 Limitations of Forced Choice

Since a limited set of variants is scored, contrastive evaluation presents the
model with a forced choice. In fact, scoring a pre-defined sequence is related
to teacher forcing, i.e., the conditioning of a model on a ground truth prefix
during training. Whenever an application involves unconstrained generation, a
discrepancy between evaluation and deployment arises that is comparable to the
exposure bias of cross-entropy training Ranzato et al. (2016).

With regard to syntactic evaluation of language models, Newman et al. (2021)
point out that contrastive evaluation and evaluation of systematicity across a
paradigm do not necessarily describe a model’s likely behavior. They propose
to analyze the complete search space, which, however, is difficult to
implement in many use cases. We pursue a different strategy and create minimal
pairs that are more similar to sequences the model will likely generate at
deployment time.

##  3 Experiments

In previous work, contrastive evaluation has commonly been used to test
plausible research hypotheses, for example the hypothesis that RNNs can
predict long-distance number agreement Gulordava et al. (2018), or the
hypothesis that word dropout improves pronoun resolution in translation
Fernandes et al. (2021). In this paper, we are interested in implausible
hypotheses and in how the testing of such hypotheses is affected by the
limitations described in the previous section.

We formulate two implausible hypotheses about NMT systems, which we mark with
an asterisk (*):

  1. 1.

*Vague language:
NMT systems make liberal use of vague placeholder words. Specifically,
English–German models use the German placeholder noun Ding (‘thing’)
ubiquitously.

  2. 2.

*Hypercorrection:
NMT systems have a tendency for hypercorrect language. Specifically,
English–German models tend to use genitive case with prepositions that require
dative case.

Examples are given in the next section. The two hypotheses are chosen because
they seem implausible both theoretically and empirically. From a theoretical
standpoint, both linguistic phenomena rarely occur in the training data and
the model is unlikely to adopt them broadly. Furthermore, the cognitive and
social factors that cause the phenomena in human speech do not apply to neural
language models. Empirically, we find that both phenomena are indeed very rare
in neural machine translations, independent of model quality.

For comparison, we also test two plausible hypotheses about NMT systems:

  1. 3.

Polarity affix deletion:
NMT systems sometimes omit negation affixes, changing the polarity of a word
Hossain et al. (2020). Specifically, English–German models sometimes omit the
negation prefix un- from German words Sennrich (2017).

  2. 4.

Clause omission:
NMT systems sometimes omit a clause from the translated sentence Tu et al.
(2016).

###  3.1 Test Set Creation

For each of the four hypotheses, we create an English–German contrastive test
set. For vague language, polarity affix deletion and clause omission, we use
the newstest datasets 2009–2016 as a data source. For hypercorrection, we
combine five data sources: newstest 2009–2019 as well as OpenSubtitles2016
Lison and Tiedemann (2016), TED2020 Reimers and Gurevych (2020), QED Abdelali
et al. (2014) and JW300 Agić and Vulić (2019), which are provided by OPUS
Tiedemann (2012).

#### Vague language

Contrastive variants are created by replacing a random noun in each reference
with an uninflected Ding ‘thing’, which is a common replacement noun in spoken
German Vogel (2020):

  * English: Prague Stock Market falls to minus by the end of the trading day

  * German (correct): Die Prager Börse stürzt gegen Geschäftsschluss ins Minus.

  * German (contrastive): Die Prager Börse stürzt gegen Ding ins Minus.

#### Hypercorrection

To create contrastive variants for hypercorrect genitives, we select
references containing German propositions that require dative in Standard
German, but are sometimes used hypercorrectly with a genitive case Hentschel
and Weydt (2013).222We use the prepositions entgegen, entsprechend, gegenüber,
gemäß, nahe, nebst, (mit)samt and seit. We construct contrastive variants by
converting the dative case into genitive case:

  * English: I’ve loved you ever since that day in the rose garden.

  * German (correct): Ich liebe dich seit dem Tag im Rosengarten.

  * German (contrastive): Ich liebe dich seit des Tags im Rosengarten.

#### Polarity affix deletion

Contrastive variants are created by deleting the prefix un- from adjectives,
adverbs and nouns in the German references in cases where this changes the
polarity of the word, similar to the test set created by Sennrich (2017):

  * English: The probes unexpectedly become faster or slower.

  * German (correct): Die Sonden werden unerwartet schneller oder langsamer.

  * German (contrastive): Die Sonden werden erwartet schneller oder langsamer.

#### Clause omission

Contrastive variants are created by deleting a clause from the reference. As
clauses we treat token sequences segmented by the Stanza sentence splitter Qi
et al. (2020):

  * English: And even if it could be proved for humans - how would one want to prove it for rats?

  * German (correct): Und selbst wenn man das für den Menschen beweisen könnte: Wie wollte man es bei Ratten nachweisen?

  * German (contrastive): Und selbst wenn man das für den Menschen beweisen könnte:

###  3.2 Human-Written References

The above test sets are derived from naturally occurring parallel text, which
is common practice when creating contrastive datasets for MT Sennrich (2017);
Rios et al. (2017); Bawden et al. (2018); Müller et al. (2018); Voita et al.
(2019); Raganato et al. (2019); Sugiyama and Yoshinaga (2019); Nagata and
Morishita (2020); Shimazu et al. (2020); Lopes et al. (2020); He et al.
(2020); Stojanovski et al. (2020). However, comparisons have shown that human-
written references are different from machine translations in that they
contain more noise and have more linguistic diversity Zhang et al. (2018);
Vanmassenhove et al. (2019).

We propose to measure the “distance” between a pre-defined target sequence and
the 1-best translation $\hat{Y}$ generated by an MT system as the difference
in log-scores (according to Equation 2.1) that the system assigns to the two
sequences. Furthermore, we define the distributional discrepancy of a
contrastive evaluation dataset as the mean difference in scores between the
1-best translation and the preferred variant:

$\displaystyle\textrm{score\\_preferred}={}$
$\displaystyle\max(\textrm{score}(Y^{correct}),\textrm{score}(Y^{contrast.}))$
$\displaystyle\textrm{discrepancy}={}$
$\displaystyle\frac{1}{n}\sum_{i=0}^{n}\textrm{score}(\hat{Y}_{i})-\textrm{score\\_preferred}_{i}$

It should be noted that this definition of distributional discrepancy is
mainly useful for comparing multiple test sets with respect to a single model.
It is less useful for assessing a single test set with respect to multiple
models, because score differences are not necessarily comparable between
models.

Hypothesis |  Test set type | Discrepancy of test set | Reported accuracy
---|---|---|---
|  | Transformer | Distilled | Transformer | Distilled
*Vague language |  human references | $1.2\pm 0.0$ | $2.5\pm 0.1$ | $99.1\pm 0.1$ | $94.7\pm 0.4$
|  machine references | $0.3\pm 0.0$ | $0.7\pm 0.0$ | $99.9\pm 0.0$ | $98.7\pm 0.2$
*Hypercorrection |  human references | $1.3\pm 0.0$ | $2.7\pm 0.1$ | $95.4\pm 0.3$ | $91.2\pm 0.5$
|  machine references | $0.4\pm 0.0$ | $1.1\pm 0.1$ | $99.9\pm 0.1$ | $99.6\pm 0.4$
Polarity affix del. |  human references | $1.3\pm 0.0$ | $2.7\pm 0.1$ | $94.0\pm 1.1$ | $78.3\pm 0.9$
|  machine references | $0.3\pm 0.0$ | $0.7\pm 0.1$ | $96.7\pm 1.5$ | $93.9\pm 1.1$
Clause omission |  human references | $1.3\pm 0.0$ | $2.8\pm 0.1$ | $75.5\pm 3.7$ | $71.3\pm 0.7$
|  machine references | $0.3\pm 0.0$ | $0.7\pm 0.0$ | $87.7\pm 2.7$ | $86.3\pm 2.7$
Table 1: Results for four different hypotheses about English–German NMT
systems. An asterisk (*) marks hypotheses that are a priori implausible. The
table reports distributional discrepancies of different test set types, as
well as the accuracy scores achieved by non-distilled and distilled systems
when evaluated with the test sets. We report averages and standard deviations
across three models trained independently with different random seeds.

###  3.3 Machine-Generated References

With the goal of reducing distributional discrepancy, we create versions of
our test sets that use machine-generated references. First, we re-translate
the sources from our test sets using commercial NMT systems.333We used Amazon
Translate, DeepL Translator, Google Translate, and Microsoft Translator for
25% sentences each. We then repeat the steps described in Section 3.1 to
create contrastive variants.

#### Validation

Since some machine-generated references contain errors, a validation step is
needed. The validation should ensure that (a) the machine references are
correct with respect to the linguistic phenomenon at hand, and that (b) no
undesired bias is introduced into the evaluation.

We use a semi-automatic approach and look for lexical overlap with the human
references regarding the phenomenon. For example, in the case of polarity
affix deletion, we label the machine reference as correct if it contains the
same polarity word as the human reference. Otherwise we manually check whether
the machine reference might be incorrect, but only if it contains the same
polarity word as the human contrastive variant. This occurs rarely, and most
of the time we find that it is the original human reference that is incorrect
while the machine reference is correct. In the rare cases where the machine
reference is verifiably incorrect with regard to the phenomenon, we use it as
the contrastive variant and derive the correct variant manually.

Machine references that have no phenomenon-specific lexical overlap to the
human references are dropped from the test set because they cannot be
automatically validated. This raises the question whether test sets created in
such a way contain undesired bias.

#### Dataset Bias

We discuss two kinds of bias that might be introduced. First of all, by only
including machine references that can be classified automatically as either
correct or incorrect based on the human references, the distribution of the
machine-generated test set could become more similar to the human-written test
set. However, our experiments show that the difference in distributional
discrepancy between the two test sets is sufficiently large. Future work could
avoid this bias by employing human annotators to validate machine references.

Secondly, it might be that machine references only use the phenomenon in
unambiguous contexts. This would cut off the long tail of human-written test
samples that is especially challenging for NLP models. While such a bias is
likely to be introduced to a degree, we see it is a desired bias, since our
goal is to reduce distributional discrepancy between a test set and the
generative behavior of an evaluated system.

###  3.4 Experimental Setting

We evaluate two types of NMT systems:

  1. 1.

Transformer: Transformer models of size ‘big’ Vaswani et al. (2017).

  2. 2.

Distilled: Transformer models of size ‘small’ distilled from (1) using
sequence-level knowledge distillation Kim and Rush (2016).

#### Training

For both types, we trained three models with different random seeds. To train
the Transformer models, we used similar data and configuration as Ng et al.
(2019), using Fairseq Ott et al. (2019). We used the English–German parallel
training data from the WMT19 news translation task Barrault et al. (2019).
Sentences longer than 250 tokens and pairs with a length ratio larger than 1.5
were filtered, resulting in 42.9M sentence pairs used for training and
distillation. We selected the best checkpoint with respect to BLEU based on
the newstest sets from the preceding years.

#### Distillation

We then used each of the three Transformer models as a teacher to train an
individual student model. A comparison of hyperparameters is provided in
Appendix A.

For decoding we always use beam search with size 5.

#### Model Quality

The models of type Transformer achieve an average BLEU score of $37.3\pm 0.3$,
while the Distilled models achieve $35.7\pm 0.4$ BLEU when evaluated on
newstest19.

###  3.5 Results

The left-hand side of Table 1 shows the distributional discrepancies of the
test sets. As expected, the test sets derived from human-written references
have a higher discrepancy, while those derived from machine-generated
references are closer to what the evaluated model would generate.

The right-hand side of Table 1 shows the reported accuracy of the evaluated
models, i.e. the ratio of test instances where the model prefers the correct
variant over the contrastive variant. While all the accuracies are much better
than random, the results for implausible hypotheses seem to indicate that
models do occasionally generate the implausible phenomena, and that distilled
models generate them more often than other models. Since this is not reflected
by the actual generative behavior, the testing of implausible hypotheses shows
the danger of false positives.

The test sets with machine-generated references produce far fewer false
positives. The reported accuracy is higher with machine references also for
the plausible hypotheses, but a gap to 100% accuracy remains, which is in line
with previous work on these types of NMT errors Hossain et al. (2020); Tang et
al. (2021); Tu et al. (2016).

##  4 Dataset Release

Given the improved specificity of test sets with machine-generated references,
we release corresponding test sets for other phenomena in the LingEval97 test
suite Sennrich (2017), terming our dataset variant DistilLingEval.

LingEval97, the original test suite, is a collection of 97k contrastive
translation pairs for 13 different error types in English–German translation.
Building on LingEval97, we create test sets with machine-generated references
for the following error types, in addition to the ones discussed in the
previous section: noun phrase agreement, subject-verb agreement and other
polarity deletion phenomena involving the German negation lexemes kein and
nicht. Results for these test sets are reported in Table 3, and further
results for a state-of-the-art NMT system are provided in Appendix C. Table 2
provides an overview of the test set sizes per error type in DistilLingEval.

Error type | Human Ref. | MT Ref.
---|---|---
clause_omission | 1104 | 1025
hypercorrect_genitive | 3404 | 635
np_agreement | 24 055 | 10 595
placeholder_ding | 18 647 | 18 659
polarity_affix_del | 408 | 180
polarity_particle_kein_del | 554 | 201
polarity_particle_nicht_del | 2561 | 888
subj_verb_agreement | 31 978 | 6701
Table 2: Number of samples per DistilLingEval error type. Error types with machine-generated references tend to have fewer samples, which is discussed in Section 3.3. Error Type |  Test set type | Discrepancy of test set | Reported accuracy
---|---|---|---
|  | Transf. | Distilled | Transf. | Distilled
np_agreement |  human references | $2.0\pm 0.6$ | $2.6\pm 0.1$ | $95.9\pm 2.5$ | $94.4\pm 0.8$
|  machine references | $0.5\pm 0.2$ | $0.7\pm 0.1$ | $98.8\pm 0.8$ | $99.0\pm 0.4$
polarity_particle_kein_del |  human references | $1.3\pm 0.0$ | $2.7\pm 0.1$ | $95.3\pm 0.8$ | $90.7\pm 0.9$
|  machine references | $0.2\pm 0.0$ | $0.6\pm 0.1$ | $99.8\pm 0.3$ | $99.8\pm 0.3$
polarity_particle_nicht_del |  human references | $1.3\pm 0.0$ | $2.6\pm 0.1$ | $95.5\pm 0.2$ | $87.9\pm 0.7$
|  machine references | $0.3\pm 0.0$ | $0.7\pm 0.0$ | $99.7\pm 0.3$ | $98.8\pm 0.1$
subj_verb_agreement |  human references | $1.2\pm 0.0$ | $2.6\pm 0.1$ | $97.1\pm 0.3$ | $91.4\pm 0.3$
|  machine references | $0.3\pm 0.0$ | $0.7\pm 0.0$ | $99.2\pm 0.1$ | $97.9\pm 0.3$
Table 3: Test set discrepancies and model accuracies for the other four error
types included in DistilLingEval (in addition to the four error types in Table
1).

##  5 Discussion

By testing implausible hypotheses, we demonstrate the risk of drawing wrong
inferences about generative behavior of (conditional) language models,
especially if there is a large distributional discrepancy between minimal
pairs and generated sequences.

This problem is especially apparent for distilled NMT models, which perform
poorly on human-written minimal pairs because they were never exposed to such
a distribution during training. While this indicates that distilled NMT models
are less robust against improbable contexts, human-crafted minimal pairs also
become less useful to predict their unconstrained generative behavior.

The danger of false positives from minimal pairs highlights the fact that
behaviorist approaches to measuring knowledge are limited to the behavioral
interface that is observed. Systematic assessments of linguistic knowledge or
syntactic abilities of neural models should be qualified accordingly, in case
minimal pairs are the primary analysis method. We suspect that whenever a
broad range of hypotheses is tested, including phenomena that are rarely
observed in actual machine-generated text, the risk of false positives is
increased.

We thus recommend that minimal pairs be constructed from machine-generated
text in evaluation settings where unconstrained generation is the behavior of
interest. This is relatively straightforward for sequence-to-sequence
evaluation, as we demonstrated in our experiments. For other settings, e.g.
the evaluation of dialogue models, obtaining useful machine-generated text
might require more elaborate techniques, such as round-trip translation.

However, human-crafted minimal pairs remain valuable in other use cases. While
machine-generated pairs may be more appropriate when the main interest is to
study the model’s behavior close to its mode, e.g. in a sequence-to-sequence
task, human-written pairs (or pairs that are machine-generated to be different
from the training distribution on purpose) may tell us more about the
robustness of models outside the mode. For example, terminology-constrained or
interactive applications depend on robustness against improbable contexts, and
contrastive evaluation indicates that current NMT systems lack such robustness
Stojanovski et al. (2020). Similarly, syntactic evaluation of language models
using randomly generated or nonsensical sentences Gulordava et al. (2018);
Warstadt et al. (2020) can be seen as method to assess the robustness of a
model under improbable input, rather than as an assessment of generative
capabilities in general.

##  6 Conclusion

We show that there are conditions where contrastive evaluation leads to false
positives if generative behavior is inferred from behavior under forced
choice. Experiments with English–German NMT indicate that the gap between the
two behavioral interfaces is especially high when human-written text is used
to create minimal pairs. Using machine-generated text largely reduces the gap.
We recommend that human-written minimal pairs are mainly used for assessing
the robustness of models, but that for predicting the generative behavior of
language models, machine-generated minimal pairs are used.

### Broader Impact

For language generation systems to be deployed, they should behave according
to specified principles in a robust way. Typical requirements are linguistic
acceptability, avoidance of undesirable societal biases Sheng et al. (2021),
and the avoidance of harmful speech acts. Contrastive evaluation is one of
several methods that can help predict the behavior of language generation
systems. However, to our knowledge the method has been mainly used to evaluate
linguistic acceptability, and less to evaluate ethically sensitive aspects of
generation.

It is crucial that evaluation methods have a high predictiveness regarding the
behavior of a deployed system. On the one hand, lack of sensitivity can lead
to unforeseen negative impact. On the other hand, lack of specificity – which
we address in this paper – reduces the usefulness of comparisons between
systems.

## Acknowledgments

This work was funded by the Swiss National Science Foundation (project
MUTAMUR; no. 176727) and made use of infrastructure services provided by S3IT,
the Service and Support for Science IT team at the University of Zurich. We
would like to thank Anne Beyer and the anonymous reviewers for helpful
feedback.

## References

  * Abdelali et al. (2014) Ahmed Abdelali, Francisco Guzman, Hassan Sajjad, and Stephan Vogel. 2014.  The AMARA corpus: Building parallel language resources for the educational domain.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 1856–1862, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Agić and Vulić (2019) Željko Agić and Ivan Vulić. 2019.  JW300: A wide-coverage parallel corpus for low-resource languages.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3204–3210, Florence, Italy. Association for Computational Linguistics.
  * Barrault et al. (2019) Loïc Barrault, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. 2019.  Findings of the 2019 conference on machine translation (WMT19).  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 1–61, Florence, Italy. Association for Computational Linguistics.
  * Bawden et al. (2018) Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018.  Evaluating discourse phenomena in neural machine translation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1304–1313, New Orleans, Louisiana. Association for Computational Linguistics.
  * Beyer et al. (2021) Anne Beyer, Sharid Loáiciga, and David Schlangen. 2021.  Is incoherence surprising? Targeted evaluation of coherence prediction from language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4164–4173, Online. Association for Computational Linguistics.
  * Fernandes et al. (2021) Patrick Fernandes, Kayo Yin, Graham Neubig, and André F. T. Martins. 2021.  Measuring and increasing context usage in context-aware machine translation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6467–6478, Online. Association for Computational Linguistics.
  * Gulordava et al. (2018) Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018.  Colorless green recurrent networks dream hierarchically.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1195–1205, New Orleans, Louisiana. Association for Computational Linguistics.
  * He et al. (2020) Jie He, Tao Wang, Deyi Xiong, and Qun Liu. 2020.  The box is in the pen: Evaluating commonsense reasoning in neural machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3662–3672, Online. Association for Computational Linguistics.
  * Hentschel and Weydt (2013) Elke Hentschel and Harald Weydt. 2013.  _Handbuch der deutschen Grammatik_.  De Gruyter.
  * Hossain et al. (2020) Md Mosharaf Hossain, Antonios Anastasopoulos, Eduardo Blanco, and Alexis Palmer. 2020.  It’s not a non-issue: Negation as a source of error in machine translation.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3869–3885, Online. Association for Computational Linguistics.
  * Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016.  Sequence-level knowledge distillation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
  * Linzen et al. (2016) Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016.  Assessing the ability of LSTMs to learn syntax-sensitive dependencies.  _Transactions of the Association for Computational Linguistics_ , 4:521–535.
  * Lison and Tiedemann (2016) Pierre Lison and Jörg Tiedemann. 2016.  OpenSubtitles2016: Extracting large parallel corpora from movie and TV subtitles.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_ , pages 923–929, Portorož, Slovenia. European Language Resources Association (ELRA).
  * Lopes et al. (2020) António Lopes, M. Amin Farajian, Rachel Bawden, Michael Zhang, and André F. T. Martins. 2020.  Document-level neural MT: A systematic comparison.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 225–234, Lisboa, Portugal. European Association for Machine Translation.
  * Marvin and Linzen (2018) Rebecca Marvin and Tal Linzen. 2018.  Targeted syntactic evaluation of language models.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics.
  * Müller et al. (2018) Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018.  A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation.  In _Proceedings of the Third Conference on Machine Translation: Research Papers_ , pages 61–72, Brussels, Belgium. Association for Computational Linguistics.
  * Nagata and Morishita (2020) Masaaki Nagata and Makoto Morishita. 2020.  A test set for discourse translation from Japanese to English.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 3704–3709, Marseille, France. European Language Resources Association.
  * Newman et al. (2021) Benjamin Newman, Kai-Siang Ang, Julia Gong, and John Hewitt. 2021.  Refining targeted syntactic evaluation of language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3710–3723, Online. Association for Computational Linguistics.
  * Ng et al. (2019) Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019\.  Facebook FAIR’s WMT19 news translation task submission.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 314–319, Florence, Italy. Association for Computational Linguistics.
  * Ott et al. (2019) Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019.  fairseq: A fast, extensible toolkit for sequence modeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)_ , pages 48–53, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Qi et al. (2020) Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020\.  Stanza: A python natural language processing toolkit for many human languages.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 101–108, Online. Association for Computational Linguistics.
  * Raganato et al. (2019) Alessandro Raganato, Yves Scherrer, and Jörg Tiedemann. 2019.  The MuCoW test suite at WMT 2019: Automatically harvested multilingual contrastive word sense disambiguation test sets for machine translation.  In _Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)_ , pages 470–480, Florence, Italy. Association for Computational Linguistics.
  * Ranzato et al. (2016) Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016.  Sequence level training with recurrent neural networks.  In _4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Rios et al. (2017) Annette Rios, Laura Mascarell, and Rico Sennrich. 2017.  Improving word sense disambiguation in neural machine translation with sense embeddings.  In _Proceedings of the Second Conference on Machine Translation_ , pages 11–19, Copenhagen, Denmark. Association for Computational Linguistics.
  * Sennrich (2017) Rico Sennrich. 2017.  How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 376–382, Valencia, Spain. Association for Computational Linguistics.
  * Sheng et al. (2021) Emily Sheng, Kai-Wei Chang, Prem Natarajan, and Nanyun Peng. 2021.  Societal biases in language generation: Progress and challenges.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 4275–4293, Online. Association for Computational Linguistics.
  * Shimazu et al. (2020) Sho Shimazu, Sho Takase, Toshiaki Nakazawa, and Naoaki Okazaki. 2020.  Evaluation dataset for zero pronoun in Japanese to English translation.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 3630–3634, Marseille, France. European Language Resources Association.
  * Stojanovski et al. (2020) Dario Stojanovski, Benno Krojer, Denis Peskov, and Alexander Fraser. 2020.  ContraCAT: Contrastive coreference analytical templates for machine translation.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4732–4749, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Sugiyama and Yoshinaga (2019) Amane Sugiyama and Naoki Yoshinaga. 2019.  Data augmentation using back-translation for context-aware neural machine translation.  In _Proceedings of the Fourth Workshop on Discourse in Machine Translation (DiscoMT 2019)_ , pages 35–44, Hong Kong, China. Association for Computational Linguistics.
  * Tang et al. (2021) Gongbo Tang, Philipp Rönchen, Rico Sennrich, and Joakim Nivre. 2021.  Revisiting Negation in Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ , 9:740–755.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_ , pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA).
  * Tu et al. (2016) Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016.  Modeling coverage for neural machine translation.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 76–85, Berlin, Germany. Association for Computational Linguistics.
  * Vanmassenhove et al. (2019) Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. 2019.  Lost in translation: Loss and decay of linguistic richness in machine translation.  In _Proceedings of Machine Translation Summit XVII Volume 1: Research Track_ , pages 222–232, Dublin, Ireland. European Association for Machine Translation.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Vogel (2020) Petra M. Vogel. 2020.  _Dingsbums and Thingy: Placeholders for Names in German and Other Languages_, page 362–383. Cambridge University Press.
  * Voita et al. (2019) Elena Voita, Rico Sennrich, and Ivan Titov. 2019.  When a good translation is wrong in context: Context-aware machine translation improves on deixis, ellipsis, and lexical cohesion.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1198–1212, Florence, Italy. Association for Computational Linguistics.
  * Warstadt et al. (2020) Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020.  BLiMP: The benchmark of linguistic minimal pairs for English.  _Transactions of the Association for Computational Linguistics_ , 8:377–392.
  * Xiang et al. (2021) Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021.  CLiMP: A benchmark for Chinese language model evaluation.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 2784–2790, Online. Association for Computational Linguistics.
  * Zhang et al. (2018) Dakun Zhang, Josep Crego, and Jean Senellart. 2018.  Analyzing knowledge distillation in neural machine translation.  In _Proceedings of the 15th International Workshop on Spoken Language Translation_ , pages 23–30.

##  Appendix A Hyperparameters

Name | $N$ | $d_{\text{model}}$ | $d_{\text{ffn}}$ | $h$ | Parameters
---|---|---|---|---|---
Transformer (big) | 6 | 1024 | 8192 | 16 | 269.7M
Distilled (small) | 4 | 512 | 2048 | 4 | 50.9M
Table 4: Hyperparameters of the Transformer variants used for the experiments

##  Appendix B Examples

Example Inputs (English–German) | Score Assigned by Model
---|---
Source: Yesterday evening, the committee wanted to vote on the appointment. |
1-best translation by the evaluated system: |
Gestern Abend wollte der Ausschuss über die Ernennung abstimmen. | -0.09
Minimal pair based on a human-written reference: |
– Correct: Gestern Abend wollte das Gremium über die Personalie abstimmen. | -3.61
– Incorrect: Gestern Abend wollte das Gremium über die Ding abstimmen. | -2.34
Minimal pair based on a machine-generated reference (Amazon Translate): |
– Correct: Gestern Abend wollte der Ausschuss über die Ernennung abstimmen. | -0.09
– Incorrect: Gestern Abend wollte der Ausschuss über die Ding abstimmen. | -1.25
Source: Why did Judah lose its land and temple? |
1-best translation by the evaluated system: |
Warum hat Juda sein Land und seinen Tempel verloren? | -0.11
Minimal pair based on a human-written reference: |
– Correct: Warum verlor Juda sein Land mitsamt dem Tempel? | -2.58
– Incorrect: Warum verlor Juda sein Land mitsamt des Tempels? | -2.55
Minimal pair based on a machine-generated reference (DeepL): |
– Correct: Warum hat Juda sein Land und seinen Tempel verloren? | -0.11
– Incorrect: N/A |
Table 5: Examples of human-written and machine-generated minimal pairs for the
*Vague language hypothesis (top) and the *Hypercorrection hypothesis (bottom).
The log-scores are computed by an NMT model of type Distilled.
The first example demonstrates that a model often assigns a lower score to the
correct human reference than to the incorrect machine reference. The human
reference differs from the machine reference only in how the words committee
and appointment are translated. The human word choice is fluent but has a
lower probability under the model.
The second example shows that machine references often avoid the phenomenon
altogether. Here, a simple conjunction is used instead of the more prestigious
preposition mitsamt ‘along with’ in the human reference. This removes any risk
of inserting a hypercorrect genitive. Since a contrastive variant cannot be
derived from the machine reference, the sample is excluded from the machine-
generated test set.

##  Appendix C State-of-the-art Accuracies for DistilLingEval

Error Type |  Test set type | Discrepancy of test set | Reported accuracy
---|---|---|---
clause_omission |  human references | $0.91$ | $78.1$
|  machine references | $0.19$ | $87.1$
hypercorrect_genitive |  human references | $1.13$ | $94.2$
|  machine references | $0.18$ | $100.0$
np_agreement |  human references | $0.84$ | $99.7$
|  machine references | $0.15$ | $100.0$
placeholder_ding |  human references | $0.79$ | $99.8$
|  machine references | $0.16$ | $100.0$
polarity_affix_del |  human references | $0.87$ | $98.8$
|  machine references | $0.13$ | $100.0$
polarity_particle_kein_del |  human references | $0.88$ | $97.5$
|  machine references | $0.12$ | $100.0$
polarity_particle_nicht_del |  human references | $0.84$ | $97.9$
|  machine references | $0.14$ | $99.9$
subj_verb_agreement |  human references | $0.83$ | $98.7$
|  machine references | $0.14$ | $99.8$
Table 6: DistilLingEval results of a state-of-the-art NMT ensemble Ng et al.
(2019). The accuracies on machine references suggest that clause omission is
an error type that still occurs with state-of-the-art NMT systems.