#  Dolomites: Domain-Specific Long-Form Methodical Tasks

Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev,
Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar,
Dipanjan Das, Mirella Lapata, Chris Alberti
University of Pennsylvania Google DeepMind

<EMAIL_ADDRESS><EMAIL_ADDRESS>
Work done at Google DeepMind.

###### Abstract

Experts in various fields routinely perform methodical writing tasks to plan,
organize, and report their work. From a clinician writing a differential
diagnosis for a patient, to a teacher writing a lesson plan for students,
these tasks are pervasive, requiring to _methodically_ generate structured
long-form output for a given input. We develop a typology of methodical tasks
structured in the form of a task objective, procedure, input, and output, and
introduce DoLoMiTes, a novel benchmark with specifications for 519 such tasks
elicited from hundreds of experts from across 25 fields. Our benchmark further
contains specific instantiations of methodical tasks with concrete input and
output examples (1,857 in total) which we obtain by collecting expert
revisions of up to 10 model-generated examples of each task. We use these
examples to evaluate contemporary language models highlighting that automating
methodical tasks is a challenging long-form generation problem, as it requires
performing complex inferences, while drawing upon the given context as well as
domain knowledge. Our dataset is available at https://dolomites-
benchmark.github.io/.

##  1 Introduction

Figure 1: Dolomites contains descriptions of 519 methodical tasks elicited
from domain experts across various fields. We instantiate these tasks with
examples that contain plausible inputs and outputs, formulating a challenging
long-form generation problem that requires domain expertise and structured
problem-solving.

Experts in various fields regularly use writing as a means for planning,
organizing, and sharing their work. For instance, a teacher might draft a
lesson plan for what they would like to teach in their next class, and a
lawyer might draft a patent application for an invention. Experts generally
follow a consistent and methodical approach to conduct these writing tasks. In
the lesson plan example, a teacher would know the lesson objectives, format
and profile of the class, and would produce a plan with the topics to be
covered and activities to improve learning. Importantly, the teacher follows a
systematic procedure to write this lesson plan, using their expertise and what
they know about the current context (e.g., the class profile).

Figure 2: A sample of methodical tasks from law, biology and medicine in
Dolomites.

Across fields, from law to visual arts and engineering, experts accomplish on
a regular basis such methodical tasks, i.e., writing tasks which loosely
follow a standard template for what is usually given as input and what is
required from the output. These tasks often follow a structured and consistent
procedure as they are performed regularly and tend to be fairly time-
consuming, taking from a few hours to several days (see Figure 1). As large
language models become more capable and widely accessible to a more
sophisticated set of users (Owens, 2023; Mollick and Mollick, 2023; Lee et
al., 2023; Birhane et al., 2023; Mollick and Mollick, 2023; Frankenreiter and
Nyarko, 2022; Demszky et al., 2023; Wang et al., 2023a; Bommasani et al.,
2021), they hold great potential for assisting experts with methodical writing
tasks, increasing their efficiency and allowing them to focus on complex
problem-solving activities (Noy and Zhang, 2023).

Given their potential for assisting experts, it would be beneficial to
evaluate language models on a realistic set of methodical writing tasks.
However, we currently do not have benchmarks that contain a typology of such
tasks. The most natural source for such data would be query logs (Nguyen et
al., 2016; Kwiatkowski et al., 2019) or chat histories (Zhao et al., 2023).
However, these data sources do not specifically reflect domain-specific use
cases and do not allow us to study specific use cases in a controlled manner.

In this work, we bridge this gap by eliciting 519 methodical task descriptions
(see a few examples in Figure 2) from 266 experts across 25 different fields
(Section 3.1). These writing tasks are formatted in a standard way, with a
task procedure, input, and output. Further analysis with an independent group
of experts reveals that they are indeed plausible ($\sim$76% of them are
likely to be conducted by an expert on a regular basis) and most experts
($\sim$63%) would find it useful if they could use a capable AI model as a
writing assistant (Section 3.3). Our tasks serve as the first collection of
realistic use cases of experts spanning multiple domains.

To evaluate the ability of existing models to assist experts with these tasks,
we collect examples (see Figure 1), where we instantiate each task with
plausible inputs and outputs (Section 3.4). Examples are created semi-
automatically: we first retrieve web documents that could potentially serve as
samples of the task, and then generate an example using a language model based
on the retrieved web documents. These examples are then significantly post-
edited by the same experts (who contributed the task) for improving
consistency, factual correctness, and level of detail.

We use our benchmark, called Dolomites (short for Domain-Specific Long-Form
Methodical Tasks), to evaluate current models in their ability to generate
accurate and detailed outputs (Section 4). We formulate the modeling problem
as long-form generation, where models are provided the task description and
the example input and asked to generate the example output. Our experiments
reveal that there is significant headroom in improving performance on
methodical tasks (Section 5) which are inherently difficult (requiring
reasoning skills and domain-knowledge), and in terms of improving automatic
evaluation of long-form text. In addition to well-known shortcomings
(Schluter, 2017; Krishna et al., 2021), conventional metrics are not designed
to capture expert knowledge.

We hope that Dolomites can serve as a reference for common domain-specific use
cases of language models and as a means for evaluating future models. We
release our dataset at https://dolomites-benchmark.github.io.

##  2 Problem Formulation

In this section, we first describe the types of writing tasks considered in
this work. We refer to these tasks as methodical writing tasks due to two
properties that are common to their execution. Firstly, each task requires
structured problem-solving, where the task follows a specific order where each
step logically flows from the previous ones. For instance, in the Medicine
task in Figure 2, the task requires producing an assessment of the patient,
then a plan of care and finally a full evaluation of the patient. Secondly,
every task usually follows a consistent execution across inputs, where there
is a standard specification of the input, the output and the procedure for the
task. In the same Medicine task, given a patient’s subjective and objective
data, the task structure and procedure would mostly stay consistent across
patients.

To elicit descriptions of tasks from experts, we operationalize our definition
of a methodical task into a standard template (see Figure 2 for examples). We
require that every task contains a brief task objective, a task procedure
walking a beginner through how this task is conducted, input and output
sections, which include information that is typically given, and information
that needs to be generated. Both input and output sections are formatted in
the form of section titles and section descriptions. Finally, we collect
additional notes about the task, which can include best practices or common
mistakes, and missing context that is important when conducting the task.

We further expect our tasks to meet the following criteria: (1) they are
purely textual and do not involve other modalities in the input or output; (2)
they require domain expertise and can only be completed by an expert; (3) they
do not require use of specific equipment or software, with the exception of
searching the web; (4) they are frequent, routinely performed by an expert at
least once every few months; and (5) time-consuming, taking a significant but
not indefinite amount of time to complete (e.g., from a half hour to a few
days, but not several months).

Aside from task descriptions, our dataset contains specific instantiations of
methodical tasks (see Figure 1). We create examples by populating descriptions
like those shown in Figure 2 with plausible input and output sections (see
Section 3.4).

##  3 Dolomites: Data Curation

Field | Sample Task Objective
---|---
Anthropology (8) |  A survey to examine specific cultural practices, rituals, and societal norms within a cultural group or community
Architecture (20) |  Developing a construction phasing plan for a building project
Biology (21) |  Developing a protocol for a toxicity assay
Business (26) |  Write a section of a non-financial report for a client, focusing on a company’s environmental and social activities
Chemistry (21) |  To write a retrosynthesis scheme/plan for a specific target molecule
Economics (17) |  Reviewing investment options for advising companies
Education (23) |  To create a lesson plan for a school class
Engineering (22) |  To write the instructions for conducting a radioactive experiment.
Environmental Sci (23) |  Writing the life cycle assessment of a system, product or process
Geography (20) |  Analyzing the environmental and social impacts of illegal mining activities in a specific region
History (22) |  Summarize and analyze a specific medieval legal code
Hospitality (21) |  Adapt existing recipes to cater to various dietary preferences
Journalism (20) |  Write a news story based on an interview
Law (38) |  Drafting a petition to challenge a decision
Linguistics (20) |  Carry out a short literature review of a given problem in linguistics
Literature (20) |  To write a research proposal for a presentation at a literary research conference
Mathematics (15) |  Writing an experimental setup suitable for testing a research hypothesis in applied mathematics
Medicine (24) |  Writing a list of potential radiotherapy regimens for a cancer patient
Music (23) |  Writing lyrics for a game’s soundtrack
Philosophy (13) |  Provide ethical recommendations for patient/doctor cases
Physics (21) |  Design specifications for a pump or turbine system
Political Sci (20) |  Redline a management measure / legislative policy
Psychology (21) |  Writing a study protocol of a neuroimaging research project
Sociology (20) |  Analyzing responses from sociological interviews to identify themes relevant to the research question
Visual Arts (20) |  The objective of this task is to write a catalog entry for an art exhibition

Table 1: Fields represented in Dolomites, with number of tasks in parentheses
and a sample task from each field.

###  3.1 Task Collection

In our data curation process, we first collect a typology of realistic tasks
that span multiple fields. These tasks are not meant to be exhaustive, but
instead represent realistic use cases across fields.

##### Participants.

We recruit 266 participants from the Prolific crowdsourcing platform. We
recruit experts from 25 different fields, shown in Table 1, aiming for a broad
coverage across disciplines. Participants qualify as experts if they have
formal education in the field, and at least 3 years of work experience.
Additional details about the participants’ backgrounds are provided in
Appendix A.

##### Annotation Task.

We ask each annotator to provide descriptions of two writing tasks they
routinely perform in their profession subject to the criteria listed in
Section 2. For each task, annotators are asked to fill in predefined fields
(task objective, procedure, input and output sections, additional notes), the
same way as shown in Figure 2. We ask annotators to give thorough descriptions
as if they are teaching a novice how to perform each task. Instructions and
interface screenshots are in Appendix A.

###  3.2 Task Analysis

After collecting the initial set of tasks, and filtering them manually to
ensure they meet the criteria outlined in Section 2, we obtain a total of 519
tasks. We find that there are very few tasks from a field that are highly
similar. Tasks which are somewhat similar still vary based on precisely what
is expected in the input and output sections.

Table 1 provides the number of tasks across fields and the task objective of a
sample task from each field. Most fields have at least 20 tasks, with some
exceptions, where we were not able to recruit as many experts. Across tasks,
there are an average of $\sim$2.78 sections in the input and $\sim$2.82
sections in the output. Collectively, tasks in Dolomites are cognitively
demanding and versatile in the types of reasoning they require. For instance,
a diagnostic task in medicine requires _inductive reasoning_ to go from
particular symptoms to a general diagnosis. Whereas, in legal analysis,
_deductive reasoning_ is required to reason about how laws are interpreted in
a specific case and _analogical reasoning_ is needed as lawyers compare
current cases with precedents. Similarly, in software application design,
_abstract_ reasoning is important for creating modular application designs
while creativity is necessary for certain tasks in the visual arts. While it
is hard to exhaustively describe the type of reasoning required for all tasks,
every methodical task essentially involves analyzing the input, making
inferences based on the input and domain-specific knowledge, and finally,
providing a justification in writing.

Figure 3: Validation of methodical tasks in the Dolomites task collection.

###  3.3 Task Validation and Societal Implications

We validate our collection of tasks by consulting an _independent_ group of
experts. Specifically, we collect Likert ratings for each task from three
experts on the following axes (the precise description for each item on the
scale is provided in Appendix A):

  * $\bullet$

Representativeness: How likely is this task to be conducted by an expert in
your field?

  * $\bullet$

Complexity: How would you rate the complexity of this task?

  * $\bullet$

Time Required: How much time is typically required to complete this task?

  * $\bullet$

Usefulness: Would you or other experts find it useful if an AI system were to
propose initial outputs for this task (which may be lacking), that can be
validated and improved by experts?

The question above is motivated by prior work showing that AI writing
assistants could significantly benefit the productivity of experts (Eloundou
et al., 2023; Noy and Zhang, 2023; Dell’Acqua et al., 2023). Beyond
productivity, we were additionally interested in expert opinions about the
_societal implications_ of using language models as writing assistants. Hence,
for each task, we elicit answers to the following questions which require a
free-text response in addition to a Likert rating.

  * $\bullet$

Anonymity Required: Is it important to ensure anonymity of any individuals or
organizations if an AI system is used for conducting this task?

  * $\bullet$

Biased Outcomes: Could relying on automatically generated outputs for this
task result in biased or potentially harmful decisions for certain groups of
people?

  * $\bullet$

Ethical Considerations: Are there ethical considerations (e.g., privacy,
copyright issues) associated with the use of AI systems for this task?

  * $\bullet$

Workforce Impact: Could partial automation of this task have an impact on the
workforce in the short term?

  * $\bullet$

Accessibility Requirements: Would the use of AI tools for this task require
making exceptional considerations to ensure accessibility?

The main outcomes of our validation study are presented in Figure 3. We find
that the tasks collected in Dolomites are ecologically valid, i.e., they are
likely ($\sim$76%) to be conducted by field experts. Most of them are of
medium or high complexity, requiring moderate (few years of experience) or
substantial (several years of experience) expertise. While they are complex
tasks, judgements about time taken reveal that most ($\sim$61%) would take an
expert from 1 hour to 7 days to complete. This degree of difficulty suggests
that it is conceivable for language models to be useful assistants for these
tasks. Finally, an overwhelming majority of experts would be interested
($\sim$62%) or open to trying ($\sim$32%) to use a language model that
proposes initial outputs for the task.

With regard to the societal implications of language model use, the need for
anonymity emerged as a concern for a significant number of tasks ($\sim$58%).
In fields like medicine, psychology, and law, experts emphasized the
importance of protecting patient/client confidentiality. Similarly, experts
felt strongly that proprietary information and trade secrets should be kept
private in fields like business. Experts further thought that using language
model responses without careful perusal can result in biased outcomes
($\sim$56% of tasks) which could affect marginalized or underrepresented
groups. They also raised various ethical concerns relating to copyright
issues, privacy issues and stifling of human creativity due to over-reliance
on AI.

Many experts ($\sim$43%) recognized that partial automation of writing tasks
is likely to impact the workforce in the short term, potentially leading to
changes in job roles or skill requirements. At the same time, they were
optimistic that this would improve productivity and bring positive changes to
the nature of the work. It is important to ensure that users of all
backgrounds and capabilities have equal access to language models. A
significant number of tasks ($\sim 40\%$) were rated as requiring exceptional
considerations to be made for ensuring accessibility to all users. Across
fields, experts highlighted that while language models as writing assistants
can improve productivity, human oversight is important for responsible use of
these technologies.

Figure 4: Method for constructing examples of tasks in Dolomites.

###  3.4 Example Collection

To evaluate language model capabilities in assisting experts with their tasks,
we create examples of input and output sections with concrete details. We
adopt a human-in-the-loop methodology, where initial examples are generated by
a model, which are then post-edited by the same expert who provided the task.
We describe this process below.

####  3.4.1 Retrieval-Augmented Generation

We believe that samples of methodical tasks are partially available in
documents on the web. Hence, we retrieve relevant documents for each task and
generate examples by prompting models with passages from these documents as
context. This process is depicted in Figure 4 and explained below.

##### Query Formulation.

Given a task description, we first need to find more specific queries that
could potentially result in relevant web documents. For example, for writing a
catalog entry for an art exhibition, search queries like “renaissance art
exhibition catalog entry” or “picasso guernica catalog entry” are likely to
result in documents that contain examples of the task. To generate search
queries, we prompt Bard (Manyika and Hsiao, 2023) (with 1 exemplar) with the
task objective and instruct it to generate more specific queries that can help
find web documents which contain demonstrations of the task (Table 11 in
Appendix B). We generate 10 search queries for each task and restrict search
to reliable and authoritative sources. These are collected by prompting Bard
(with 1 exemplar) with the task objective to generate URLs to domain names
which will be useful to find real examples for the given task (Table 12 in
Appendix B).

##### Evidence Collection.

Using each search query, we gather the top-10 documents from Google search
restricted to relevant domains with the site: operator in the query. Documents
are then split into passages of 4,000 characters with a 100 character sliding
window.

##### Conditional Generation.

Having gathered evidence which may contain task demonstrations, we generate
examples by prompting models with this evidence. We explore a _multi-document_
setting, where passages are sampled from multiple documents and a _single
document setting_ , where passages are sampled from a single document as we
found that the appropriate choice depends on the task.111For instance, a task
that requires drafting a legal opinion might benefit from multiple relevant
documents whereas a single document might be sufficient for writing the
catalog entry for an artwork. Passages are reranked using an in-house reranker
and the top-5 passages are provided as context, along with the task
description, to a large language model (Gemini-Ultra (Team et al., 2023) or
Bard in our case), which is asked to generate an example of the task (Table 13
in Appendix B). Not all information mentioned in the task description is
required to be present in the context and the model is allowed to infill
content to construct an example. We generate up to 10 examples for each task.

####  3.4.2 Expert Post-Editing

Figure 5: Expert judgements of original examples along three dimensions:
adherence to task structure, level of detail and factual correctness.

Even though they are based on retrieved web documents, model-generated
examples could have many issues. They may not adhere to the structure
specified in the task description, they may contain factual inconsistencies or
inaccuracies, lack in depth, or be vague. To remedy these issues, we present
the examples to the same experts who wrote the tasks for post-editing. They
are asked to choose the most plausible example out of four variants, and use
single or multi-document evidence.

Prior to post-editing, experts are asked to label examples according to three
criteria on a Likert scale: adherence to task structure, factual correctness,
and level of detail. They are also shown 1) the evidence passages for the
example and 2) a critique (generated using Gemini-Ultra with the prompt in
Appendix B, Table 14) that may not be comprehensive, to aid them in
identifying issues with the example. The critique is provided to make post-
editing more efficient. They are required to fix any valid issues they
recognize in the example as well as any valid issues identified by the
critique.

####  3.4.3 Example Quality Analysis

##### Automatic Analysis.

Expert judgements of automatically generated examples are shown in Figure 5.
We find that the majority of examples already follow the task structure
($\sim$85%) and most of them are probably or definitely correct. However, a
large number of examples are lacking in depth and detail. We show a histogram
of the word-level edit distance between the original and edited example in
Figure 6a and relevant statistics of the original and post-edited examples in
Table 2. The histogram suggests that on average, there are significant changes
made to the original examples during post-editing. Since most experts judge
that examples are lacking in depth, the edited examples are expectedly much
longer on average. The edited examples also adhere better to the task
description (on average, 97.67% sections in the task description are found in
the edited examples compared to 92.81% in the original examples). Finally, we
compute readability scores of the examples as a noisy approximation of the
complexity and level of detail in the text, using the Flesch-Kincaid Grade
Level test (Kincaid et al., 1975). Higher readability values indicate that a
piece of text requires more formal education and expertise to understand. We
find that post-edited examples have higher scores, possibly indicating higher
level of technical depth.

| Avg Length |  Avg section presence %($\uparrow$) |  Flesch-Kincaid Grade Level ($\uparrow$)
---|---|---|---
original | 388.98 | 92.81 | 11.69
post-edited | 590.24 | 97.67 | 13.46

Table 2: Statistics of the original and post-edited examples in Dolomites.

##### Memorization.

Examples in Dolomites are created by conditioning on passages from web
documents. We do not require these documents to contain complete examples of
the task, and models are allowed to infill information to create an example.
However, if these documents are seen by models during large-scale pretraining,
it might be difficult to conduct a clean evaluation. We examine whether this
is the case by computing the similarity of the post-edited examples to the
evidence passages provided during generation. Figure 6b plots the cosine
similarity between the n-grams found in the post-edited examples and evidence
used as context. As can be seen, similarity to retrieved passages is fairly
low in most examples, which suggests a low risk of memorization due to
pretraining.

a. | b.
---|---
|
Figure 6:  Histogram of (a) word-level edit distance between original and
post-edited example and (b) cosine similarities based on n-grams in post-
edited examples and evidence used as context.

##  4 Experiments

Model |  Avg Length |  Avg Section Presence % | BLEURT | ROUGE-L |  nli (forward) |  nli (reverse) |  nli (h-mean)
---|---|---|---|---|---|---|---
Claude-3 Opus | 417.41 | 91.53 | 0.4156 | 0.2395 |  0.3584[0.34,0.38] |  0.3769[0.36,0.40] | 0.3674
Command-R-Plus | 440.44 | 92.92 | 0.4068 | 0.2134 |  0.3926[0.37,0.41] |  0.3623[0.34,0.38] | 0.3768
Gemini-1.5-Pro | 349.27 | 95.25 | 0.4136 | 0.2371 |  0.4065[0.39,0.42] |  0.3846[0.37,0.40] | 0.3952
Gemini-1.5-0409 | 361.87 | 95.74 | 0.4068 | 0.2361 |  0.3994[0.38,0.42] |  0.3984[0.38,0.42] | 0.3989
Gemini-Pro | 269.68 | 93.61 | 0.4124 | 0.2280 |  0.3415[0.32,0.36] |  0.3090[0.29,0.33] | 0.3244
GPT-3.5-Turbo | 240.89 | 88.98 | 0.4276 | 0.2309 |  0.3854[0.37,0.40] |  0.2949[0.28,0.31] | 0.3341
GPT-4 | 407.35 | 95.46 | 0.4155 | 0.2271 |  0.3934[0.38,0.41] |  0.3993[0.38,0.42] | 0.3963
Mistral-Large | 327.12 | 92.61 | 0.4158 | 0.2390 |  0.3524[0.33,0.37] |  0.3523[0.33,0.37] | 0.3523
Mixtral-8x22B | 339.16 | 95.16 | 0.4212 | 0.2450 |  0.3951[0.38,0.41] |  0.3583[0.34,0.37] | 0.3758
Mixtral-8x7B | 386.61 | 88.39 | 0.4098 | 0.2266 |  0.3290[0.31,0.35] |  0.3097[0.29,0.33] | 0.3191
OLMo-7B-Instruct | 784.22 | 74.02 | 0.3905 | 0.1752 |  0.1929[0.18,0.21] |  0.1721[0.16,0.19] | 0.1819

Table 3: Results on the Dolomites test set with standard metrics and factual
consistency using NLI models. We report 95% confidence intervals along with
the average NLI scores.

###  4.1 Setup and Models

We create a development-test split for Dolomites, with 820 examples in the
development (dev) set and 1,037 examples in the test set. There are 172
_seen_ tasks with examples in both dev and test and 99 _unseen_ tasks with
examples only in the test set.

For evaluation, we considered multiple performant models from various
companies as well as open-source models. In all cases, we favored instruction-
tuned variants because of their better performance on other benchmarks.
Specifically, we report experiments with Claude-3 Opus (Anthropic, ), Command-
R-Plus (Cohere, 2024), Gemini-1.5-Pro and Gemini-1.5-0409 (Reid et al., 2024)
and Gemini-Pro (Team et al., 2023), GPT-3.5-Turbo and GPT-4 (OpenAI, 2023),
Mixtral-8x7B and Mixtral-8x22B (Jiang et al., 2024) and Mistral-Large
(Mistral, 2024), and OLMo-7B-Instruct (Groeneveld et al., 2024). In all cases,
we prompt models with the task description, the input sections corresponding
to an example, and instruct them to generate the output sections for the
example, in a zero-shot manner. Hyperparameters, prompts and model identifiers
are in Appendix B.

###  4.2 Automatic Evaluation

##### Conventional Metrics.

Prior work has recognized that conventional metrics for text generation are
lacking in various ways Liu et al. (2016); Novikova et al. (2017); Krishna et
al. (2021). Nevertheless, for completeness, we report results with ROUGE-L Lin
(2004) and BLEURT Sellam et al. (2020). In addition, we report the average
output length and average section presence (i.e., the percentage of output
sections specified in the task that are present in the generated output,
averaged across all examples) as a measure of instruction-following
capabilities. Note that the average length of reference outputs is 341.42
tokens.

##### Round-Trip Factual Consistency.

We also measure the extent to which statements in the model output are
consistent with statements in the reference output. We compute 1) _forward_
entailment considering a reference section as the premise and the
corresponding model section as the hypothesis and 2) _reverse_ entailment
considering a model output section as the premise and the corresponding
reference section as the hypothesis. Scores are aggregated over all sections
and examples. These metrics loosely capture the notions of precision and
recall, we also report the harmonic mean of the two. We use the TRUE model
Honovich et al. (2022) to predict entailment scores (ranging from 0 to 1) and
report 95% confidence intervals.

Model | GPT-4 | Claude-3 Opus | Gemini-1.5-0409
---|---|---|---
Claude-3 Opus | 48.1 | 52.7 | 49.6
Command-R-Plus | 34.4 | 45.8 | 38.7
Gemini-1.5-Pro | 41.1 | 46.1 | 51.0
Gemini-1.5-0409 | 42.9 | 55.4 | 60.9
Gemini-Pro | 17.6 | 21.0 | 22.1
GPT-3.5-Turbo | 12.2 | 11.5 | 12.4
GPT-4 | 50.0 | 50.0 | 50.0
Mistral-Large | 27.2 | 28.8 | 26.7
Mixtral-8x22B | 21.6 | 25.1 | 17.6
Mixtral-8x7B | 17.8 | 23.5 | 15.6
OLMo-7B-Instruct |  4.2 |  5.5 |  3.3

Table 4: Model win rates ($\pm 3$) against GPT-4 on the Dolomites test set
using three LM-based autoraters (GPT-4, Claude-3 Opus, and Gemini-1.5-PP).
GPT-4’s win rate is 50% since it is the base comparison model.

##### LM-based Evaluation.

Language models are being increasingly used as evaluators (Chiang and Lee,
2023), despite concerns that they can be misleading and biased (Shen et al.,
2023; Wang et al., 2023b; Zheng et al., 2023; Panickssery et al., 2024). We
use _multiple_ language models as judges to give preferences for a pair of
model outputs. While this does not alleviate the problem of biased LM
judgements, we believe it is slightly more reliable since we are not biased by
a single model’s judgements. In all comparisons, we use one of the strongest
models, GPT-4, as the base comparison model. We sample outputs on the test set
from a candidate model and GPT-4 (randomizing their order in the prompt) and
ask the evaluator model to judge which output is better and provide a
justification. We consider three models as evaluators: GPT-4, Claude-3 Opus
and Gemini-1.5-0409. The win rate is computed by summing up the number of wins
for a candidate model plus half the number of ties.

##  5 Results and Discussion

##### Main Results.

Our results on the test set using automatic metrics are reported in Table 3.
Based on the average section presence, we note that models are largely
effective at generating almost all relevant output sections from the task
description. A few models (GPT-4 and Gemini-1.5-0409) excel more at following
this instruction. Based on the NLI scores, we note that the nli (reverse)
scores are on average lower than nli (forward), which suggests that generated
outputs contain statements not entailed by the reference, e.g., because they
are inaccurate or irrelevant. We observe that Gemini-1.5-0409 and GPT-4
produce more information that is factually consistent with the reference,
while Claude-3 and Command-R-Plus are also performant. We show the win rates
according to different LM evaluators in Table 4. Based on these win rates, we
note that a few models such as Claude-3 Opus, Gemini-1.5-Pro and
Gemini-1.5-0409 prove to be comparable to GPT-4. We also report win rates with
a length penalty for longer outputs in Table 6 and the overall rankings do not
change.

##### Human Evaluation.

To understand how well automatic evaluation measures correlate with human
judgements, we sample 200 pairs of model outputs, where each pair comes from
two randomly chosen models. We (the authors) then label which model output is
better (or if they are tied) according to their task adherence, factual
correctness and depth. On these 200 pairs, we also get automatic preference
judgements from all the evaluation measures discussed in Section 4.2 (we
convert float scores for two outputs into binary judgements). The percentage
agreements between human labels (with and without pairs with ties) and all
evaluation measures are in Figure 7. In summary, we find that LM-based
evaluation measures are most helpful in discriminating between models,
followed by the NLI measures and tnen ROUGE-L and BLEURT. We note that an
evaluator that always picks the longer output also has reasonable agreement
rates.

Figure 7: Percentage agreement of automatic evaluation measures with human
labels. Figure 8: Heatmap of scores from GPT-4, Gemini-1.5-0409 and Claude-3
Opus, aggregated by field.

##### Results by Field.

Figure 8 illustrates how model performance fluctuates across fields; we show
results for GPT-4, Gemini-1.5-0409, and Claude-3 Opus, normalized across
fields. Across models, we find that a few fields have significantly lower NLI
scores: Hospitality, Education, and Visual Arts. Tasks from these fields are
sometimes subjective (e.g., Create recipes with special dietary requirements,
Create a user persona for an application), which might justify the lower NLI
scores. On the other hand, tasks in fields such as Literature, History, and
Journalism have higher NLI scores. Some of these tasks focus more on factual
reporting and narratives that may be easier to reason about.

Figure 9: Heatmap of GPT-4 scores aggregated by length (in tokens) of
reference output or length in terms of the number of output sections. Figure
10: Heatmap of GPT-4 scores aggregated by task complexity labels from
annotators.

##### Results by Length.

Next, we evaluate whether output length is correlated with performance.
Specifically, we show scores for GPT-4 split into bins by 1) length of the
reference outputs and 2) number of output sections expected for the task.
Scores normalized by bins are shown in Figure 9. We find that examples which
require longer outputs are significantly harder for models, supported by the
fact that examples with longer outputs have lower NLI scores.

##### Results by Task Complexity.

Figure 10 groups results by task complexity labels collected during our task
validation study. We find that tasks with High complexity or tasks that take
longer for experts, are harder. Nonetheless, lower or medium complexity tasks
are also challenging.

##### Error Analysis.

We analyze generated outputs from 3 high-performing models (Gemini-1.5-0409,
GPT-4 and Claude-3 Opus) for examples where the NLI scores are lowest.
Broadly, we observe the following patterns:

  * $\bullet$

Lacking depth (Table 7): Writing technical documents requires depth and focus,
which was sometimes found to be lacking. For example, concrete statistical
results and method details were absent from a task on writing up a report on
an experimental study in clinical psychology.

  * $\bullet$

Verbosity (Table 8): A common characteristic of some model outputs was their
verbosity. Common patterns included defining jargon when not necessary, and
generating many filler statements that do not introduce new information.

  * $\bullet$

Missing information (Table 9): There were a few cases where a single output
section required multiple pieces of information, but the model entirely missed
producing a subset of them.

  * $\bullet$

Subjectivity (Table 10): Finally, we identified tasks that are subjective in
nature (e.g., Creating recipes with special dietary requirements or Writing a
plan for a photography shoot). While reference-based evaluation measures used
in this work do not handle this subjectivity, future work can explore ways to
evaluate such outputs.

##  6 Related Work

##### Domain-Specific NLP Benchmarks.

The use of language technologies in domain-specific scenarios has the
potential to help experts. Prior work has evaluated models through domain-
specific benchmarks for standard tasks like QA (Hendrycks et al., 2021;
Malaviya et al., 2024) and summarization Hayashi et al. (2021). Many
benchmarks have been proposed for specific fields (Rein et al., 2023; Xia et
al., 2024), including law (Shen et al., 2022; Niklaus et al., 2023; Guha et
al., 2024) and medicine (Tsatsaronis et al., 2015; Pampari et al., 2018; Jin
et al., 2019, 2020; Fleming et al., 2023).

A notable difference between this line of work and Dolomites is the task
formulation (i.e., QA vs methodical tasks). QA involves addressing a specific
information need in response to a query while conducting methodical tasks
requires following a structured and consistent procedure, involving multiple
steps, to complete a goal-oriented task.

##### Naturalistic Evaluation.

Evaluation that is grounded in realistic use cases, is important for reliable
benchmarking Rolnick et al. (2024). Prior efforts on creating NLP benchmarks
which are representative of real user needs use query logs (Nguyen et al.,
2016; Kwiatkowski et al., 2019) and chat histories (Lin et al., 2024). While
these benchmarks are useful for evaluating responses to generic user queries,
they do not allow us to study their abilities in assisting with domain-
specific tasks in an isolated manner. For instance, Ouyang et al. (2023) find
that user requests in chat histories often involve "planning" and "design",
but these are largely ignored in benchmarks. Our work fills this gap by
presenting a typology of domain-specific tasks grounded in realistic
scenarios.

##### Language Models as Writing Assistants.

Recent work has investigated the potential of language models to act as
writing assistants for domain experts (Calderwood et al., 2020; Lee et al.,
2022; Gero et al., 2022; Li et al., 2024). While there is favorable evidence
that they can improve productivity of experts (Noy and Zhang, 2023), their
usage has broader societal consequences, including potential impact on the
workforce (Eloundou et al., 2023). We analyze a subset of these societal
implications relevant for our methodical tasks in Section 3.

##  7 Conclusion

We introduce Dolomites, a benchmark that is closely tied to realistic use
cases of domain experts. The generalization of these use cases as methodical
tasks provides a way to study capabilities of language models across tasks and
domains. We consider a scenario where AI systems can act as tools for experts
to amplify their problem-solving capabilities (Engelbart, 2023) and perform
their tasks more efficiently. We verify that our tasks are representative
across fields and that human oversight is necessary if language models propose
initial outputs for these tasks. Evaluation of a broad range of contemporary
language models suggests that there is a large room for models to improve on
generating outputs for our tasks.

Future directions are many and varied. The methodical tasks in our collection
constitute a mere sample from 25 fields in English language. We hope to
further expand the set of tasks to cover a wider range of scenarios and
languages. We could also consider tasks that involve modalities other than
text in input or output, and multi-turn settings, where models continually
improve their outputs through feedback and revision. Our experimental results
revealed that automatic evaluation of generated text is particularly
challenging. Our data contains a single reference output for an example input
and does not model the potentially diverse perspectives of field experts and
the innate subjectivity of tasks (Ganguli et al., 2023). While factual
consistency metrics do not account for this subjectivity, it is unclear if
language model judgements innately capture this subjectivity. More research is
needed to ensure language model responses are given credit for alternative,
but valid responses.

## Acknowledgments

We thank the 266 annotators who participated in our data collection studies
and the following people for their help with facilitating data collection:
Shereen Ashraf, Michelle Chen Huebscher and Michael Sharman. We also thank the
following people for helpful discussions and feedback: Sumit Asthana, Tuhin
Chakrabarty, Michael Collins, Sunipa Dev, Jacob Eisenstein, Sebastian
Gehrmann, Kalpesh Krishna, Tom Kwiatkowski, Matthew Lamm, Kenton Lee, Joshua
Maynez, Slav Petrov, Hannah Rashkin, David Reitter, Marco Tulio Ribeiro,
Elizabeth Sieber and Kristina Toutanova.

## References

  * (1) Anthropic.  The claude 3 model family: Opus, sonnet, haiku.
  * Birhane et al. (2023) Abeba Birhane, Atoosa Kasirzadeh, David Leslie, and Sandra Wachter. 2023.  Science in the age of large language models.  _Nature Reviews Physics_ , 5(5):277–280.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Calderwood et al. (2020) Alex Calderwood, Vivian Qiu, Katy Ilonka Gero, and Lydia B Chilton. 2020.  How novelists use generative language models: An exploratory user study. In _HAI-GEN+ user2agent@ IUI_.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can large language models be an alternative to human evaluations? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15607–15631, Toronto, Canada. Association for Computational Linguistics.
  * Cohere (2024) Cohere. 2024.  Introducing Command R+: A Scalable LLM Built for Business.  [Accessed 16-04-2024].
  * Dell’Acqua et al. (2023) Fabrizio Dell’Acqua, Edward McFowland, Ethan R Mollick, Hila Lifshitz-Assaf, Katherine Kellogg, Saran Rajendran, Lisa Krayer, François Candelon, and Karim R Lakhani. 2023.  Navigating the jagged technological frontier: Field experimental evidence of the effects of ai on knowledge worker productivity and quality.  _Harvard Business School Technology & Operations Mgt. Unit Working Paper_, (24-013).
  * Demszky et al. (2023) Dorottya Demszky, Diyi Yang, David S Yeager, Christopher J Bryan, Margarett Clapper, Susannah Chandhok, Johannes C Eichstaedt, Cameron Hecht, Jeremy Jamieson, Meghann Johnson, et al. 2023.  Using large language models in psychology.  _Nature Reviews Psychology_ , 2(11):688–701.
  * Eloundou et al. (2023) Tyna Eloundou, Sam Manning, Pamela Mishkin, and Daniel Rock. 2023.  Gpts are gpts: An early look at the labor market impact potential of large language models.  _arXiv preprint arXiv:2303.10130_.
  * Engelbart (2023) Douglas C Engelbart. 2023.  Augmenting human intellect: A conceptual framework.  In _Augmented Education in the Global Age_ , pages 13–29. Routledge.
  * Fleming et al. (2023) Scott L Fleming, Alejandro Lozano, William J Haberkorn, Jenelle A Jindal, Eduardo P Reis, Rahul Thapa, Louis Blankemeier, Julian Z Genkins, Ethan Steinberg, Ashwin Nayak, et al. 2023.  Medalign: A clinician-generated dataset for instruction following with electronic medical records.  _arXiv preprint arXiv:2308.14089_.
  * Frankenreiter and Nyarko (2022) Jens Frankenreiter and Julian Nyarko. 2022.  Natural language processing in legal tech.  _Legal Tech and the Future of Civil Justice (David Engstrom ed.) Forthcoming_.
  * Ganguli et al. (2023) Deep Ganguli, Nicholas Schiefer, Marina Favaro, and Jack Clark. 2023.  Challenges in evaluating AI systems.
  * Gero et al. (2022) Katy Ilonka Gero, Vivian Liu, and Lydia Chilton. 2022.  Sparks: Inspiration for science writing using language models.  In _Proceedings of the 2022 ACM Designing Interactive Systems Conference_ , pages 1002–1019.
  * Groeneveld et al. (2024) Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, et al. 2024.  Olmo: Accelerating the science of language models.  _arXiv preprint arXiv:2402.00838_.
  * Guha et al. (2024) Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2024.  Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 36.
  * Hayashi et al. (2021) Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021.  Wikiasp: A dataset for multi-domain aspect-based summarization.  _Transactions of the Association for Computational Linguistics_ , 9:211–225.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_.
  * Honovich et al. (2022) Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022.  TRUE: Re-evaluating factual consistency evaluation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3905–3920, Seattle, United States. Association for Computational Linguistics.
  * Jiang et al. (2024) Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024.  Mixtral of experts.  _arXiv preprint arXiv:2401.04088_.
  * Jin et al. (2020) Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020.  What disease does this patient have? a large-scale open domain question answering dataset from medical exams.  _arXiv preprint arXiv:2009.13081_.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019.  Pubmedqa: A dataset for biomedical research question answering.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2567–2577.
  * Kincaid et al. (1975) J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975.  Derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to progress in long-form question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4940–4957, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lee et al. (2022) Mina Lee, Percy Liang, and Qian Yang. 2022.  Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities.  In _Proceedings of the 2022 CHI conference on human factors in computing systems_ , pages 1–19.
  * Lee et al. (2023) Peter Lee, Sebastien Bubeck, and Joseph Petro. 2023.  Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine.  _New England Journal of Medicine_ , 388(13):1233–1239.
  * Li et al. (2024) Zhuoyan Li, Chen Liang, Jing Peng, and Ming Yin. 2024.  The value, benefits, and concerns of generative ai-powered assistance in writing.
  * Lin et al. (2024) Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. 2024.  Wildbench: Benchmarking llms with challenging tasks from real users in the wild.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. 2016.  How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2122–2132, Austin, Texas. Association for Computational Linguistics.
  * Malaviya et al. (2024) Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. 2024.  ExpertQA: Expert-curated questions and attributed answers.  In _2024 Annual Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Manyika and Hsiao (2023) James Manyika and Sissie Hsiao. 2023.  An overview of bard: an early experiment with generative ai.  _AI. Google Static Documents_ , 2.
  * Mistral (2024) Mistral. 2024.  Au large.  Accessed on March 26, 2024.
  * Mollick and Mollick (2023) Ethan R Mollick and Lilach Mollick. 2023.  Using ai to implement effective teaching strategies in classrooms: Five strategies, including prompts.  _Including Prompts (March 17, 2023)_.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _choice_ , 2640:660.
  * Niklaus et al. (2023) Joel Niklaus, Veton Matoshi, Pooja Rani, Andrea Galassi, Matthias Stürmer, and Ilias Chalkidis. 2023.  LEXTREME: A multi-lingual and multi-task benchmark for the legal domain.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 3016–3054, Singapore. Association for Computational Linguistics.
  * Novikova et al. (2017) Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017.  Why we need new evaluation metrics for NLG.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2241–2252, Copenhagen, Denmark. Association for Computational Linguistics.
  * Noy and Zhang (2023) Shakked Noy and Whitney Zhang. 2023.  Experimental evidence on the productivity effects of generative artificial intelligence.  _Science_ , 381(6654):187–192.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Ouyang et al. (2023) Siru Ouyang, Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023.  The shifted and the overlooked: A task-oriented investigation of user-GPT interactions.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 2375–2393, Singapore. Association for Computational Linguistics.
  * Owens (2023) Brian Owens. 2023.  How nature readers are using chatgpt.  _Nature_.
  * Pampari et al. (2018) Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng. 2018.  emrQA: A large corpus for question answering on electronic medical records.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2357–2368, Brussels, Belgium. Association for Computational Linguistics.
  * Panickssery et al. (2024) Arjun Panickssery, Samuel R Bowman, and Shi Feng. 2024.  Llm evaluators recognize and favor their own generations.  _arXiv preprint arXiv:2404.13076_.
  * Reid et al. (2024) Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  _arXiv preprint arXiv:2403.05530_.
  * Rein et al. (2023) David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023.  Gpqa: A graduate-level google-proof q&a benchmark.  _ArXiv_ , abs/2311.12022.
  * Rolnick et al. (2024) David Rolnick, Alan Aspuru-Guzik, Sara Beery, Bistra Dilkina, Priya L Donti, Marzyeh Ghassemi, Hannah Kerner, Claire Monteleoni, Esther Rolf, Milind Tambe, et al. 2024.  Application-driven innovation in machine learning.  _arXiv preprint arXiv:2403.17381_.
  * Schluter (2017) Natalie Schluter. 2017.  The limits of automatic summarisation according to ROUGE.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 41–45, Valencia, Spain. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shen et al. (2023) Chenhui Shen, Liying Cheng, Xuan-Phi Nguyen, Yang You, and Lidong Bing. 2023.  Large language models are not yet human-level evaluators for abstractive summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 4215–4233, Singapore. Association for Computational Linguistics.
  * Shen et al. (2022) Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022.  Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities.  _Advances in Neural Information Processing Systems_ , 35:13158–13173.
  * Singhal et al. (2023) Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023.  A long way to go: Investigating length correlations in rlhf.  _arXiv preprint arXiv:2310.03716_.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_.
  * Tsatsaronis et al. (2015) George Tsatsaronis, Georgios Balikas, Prodromos Malakasiotis, Ioannis Partalas, Matthias Zschunke, Michael R Alvers, Dirk Weissenborn, Anastasia Krithara, Sergios Petridis, Dimitris Polychronopoulos, et al. 2015.  An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.  _BMC bioinformatics_ , 16(1):1–28.
  * Wang et al. (2023a) Hanchen Wang, Tianfan Fu, Yuanqi Du, Wenhao Gao, Kexin Huang, Ziming Liu, Payal Chandak, Shengchao Liu, Peter Van Katwyk, Andreea Deac, et al. 2023a.  Scientific discovery in the age of artificial intelligence.  _Nature_ , 620(7972):47–60.
  * Wang et al. (2023b) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023b.  Large language models are not fair evaluators.  _arXiv preprint arXiv:2305.17926_.
  * Xia et al. (2024) Congying Xia, Chen Xing, Jiangshu Du, Xinyi Yang, Yihao Feng, Ran Xu, Wenpeng Yin, and Caiming Xiong. 2024.  Fofo: A benchmark to evaluate llms’ format-following capability.  _arXiv preprint arXiv:2402.18667_.
  * Zhao et al. (2023) Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2023.  (inthe) wildchat: 570k chatgpt interaction logs in the wild.  In _The Twelfth International Conference on Learning Representations_.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.

##  Appendix A Annotation Details

##### Participants.

We recruited 266 participants for our study from Prolific. Participants were
required to be fluent in English, and came from 25 different countries, across
Africa, Europe, North and South America. In terms of their background,
participants were required to have an undergraduate degree and 3 years of work
experience in their respective field. These requirements were first enforced
through Prolific’s audience filters, followed by a screening where
participants were asked to self-report their educational qualifications and
work experience. They each provided two tasks, so for each field, we recruited
half the number of the participants as the number of tasks reported in Table
1. Lastly, they were required to have at least 50 prior approved submissions
and an approval rate of over 99%. Participants were informed that their
provided data will be used to evaluate large language models in realistic
scenarios. We obtained prior consent from all annotators before recruiting
them for all studies.

##### Setup.

Annotators were paid $20 per hour for their work. For task collection, we
allocated 40 minutes to write two tasks and for task validation, we allocated
15 minutes per task. For post-editing examples, we allocated 20 minutes per
example.

Figure 11: Interface shown to annotators for task validation. Figure 12:
Interface shown to annotators for example post-editing.

##### Annotation Interface Screenshots.

We show screenshots of the annotation interfaces presented to annotators for
task validation and example post-editing in Figures 11 and 12 respectively.

##  Appendix B Experimental Details

##### Models.

The specific identifiers for the models evaluated in this work are given in
Table 5. Open-source models were obtained from the HuggingFace model hub,
while proprietary models were obtained through the organization’s official
APIs.

##### Generation Configurations.

In all generation tasks, we set the temperature for generation to be 0.1. For
both example generation and model evaluation, we sampled a maximum of 4,096
tokens (or the maximum sequence length of the model).

##### Prompts.

We provide the prompts used for various components of our work. The prompts
used for example creation are given in Tables 11–13. The prompt used to
generate the critique shown to annotators is shown in Table 14. The prompt
used to generate outputs from candidate models is shown in Table 15. Finally,
the prompt used for generating LM-based judgements for evaluation is shown in
Table 16.

Model Name | Identifier
---|---
Claude-3 Opus | claude-3-opus-20240229
Command-R-Plus | command-r-plus
Gemini-1.5-Pro222Accessed from https://aistudio.google.com/app | gemini-1.5-pro-latest
Gemini-1.5-0409333Accessed from https://console.cloud.google.com/vertex-ai/generative | gemini-1.5-pro-preview-0409
Gemini-Pro | gemini-pro
GPT-3.5-Turbo | gpt-3.5-turbo
GPT-4 | gpt-4-turbo-preview
Mistral-Large | mistral-large-latest
Mixtral-8x22B | Mixtral-8x22B-Instruct-v0.1
Mixtral-8x7B | Mixtral-8x7B-v0.1
OLMo-7B-Instruct | OLMo-7B-Instruct

Table 5: List of models used in our experiments and their identifiers.

##  Appendix C Additional Results

We report additional results for LM-based evaluation of models. Prior work has
reported that language models show a bias for longer outputs Singhal et al.
(2023). To correct for this bias, we report length-adjusted win rates in Table
6. The length adjustment requires applying a length penalty on the points
awarded for each win. This penalty is 1% for every 10 words in the average
length difference between the two models.

Model | GPT-4 | Claude-3 Opus | Gemini-1.5-0409
---|---|---|---
Claude-3 Opus | 47.6 | 52.1 | 49.1
Command-R-Plus | 33.4 | 44.4 | 37.5
Gemini-1.5-Pro | 43.2 | 48.4 | 53.6
Gemini-1.5-0409 | 44.6 | 57.6 | 63.4
Gemini-Pro | 19.8 | 23.6 | 25.0
GPT-3.5-Turbo | 14.1 | 13.3 | 14.4
GPT-4 | 50.0 | 50.0 | 50.0
Mistral-Large | 29.4 | 31.1 | 28.8
Mixtral-8x22B | 22.9 | 26.6 | 18.7
Mixtral-8x7B | 18.2 | 24.0 | 15.9
OLMo-7B-Instruct |  2.7 |  3.5 |  3.0

Table 6: Model win rate ($\pm 3$) against GPT-4 on the Dolomites benchmark
using three LM-based autoraters (GPT-4, Claude-3 Opus, and Gemini-1.5-PP),
with a length penalty.

Pattern: Lack of Detail
---
TASK DESCRIPTION Task Objective: Designing an observation plan for different
celestial bodies and objects using an infrared telescope Task Procedure:
Observing space can be quite a complicated task to achieve, space contains a
lot of different celestial bodies and a variety of objects. Some of these
objects emits "special" kind of electromagnetic radiation - radiation that we
humans cannot see with our eyes. So in this task we’re focusing on the
invisible glow these objects emit in the infrared part of the spectrum. Our
task is to basically decide which celestial bodies (star, planet, galaxies,
nebulas and more) we want to study and investigate considering and taking into
account their unique infrared features. We also need to plan what kind of
telescope is going to be used in order to successfully achieve that mission
and literally "see" what we want to see. Additional Notes: Planning it
correctly can save a lot of time and frustrations, taking in account different
information like I presented and learning from mistakes can lead to a
successful observation. Input Sections: * Scientific/main Goal AND target
object: 1 paragraph, 3-4 sentences. To begin with the planning, we mostly need
to understand what is our main objective - so we have to outline the
scientific goals we aim to gather from the observation such as understanding
atmospheric composition of different bodies, stars/planets life cycle and
formation process and more. * Kind of telescope and wavelength range: 2
paragraphs, 7-8 sentences. The user should provide detailed information
regarding what instrument is being used, which includes technical
specifications such as focal length of the telescope and eyepieces, apertures,
focal ratios, the type of telescope and also what kind of additional items are
being used like filters and cameras to detect that special electromagnetic
spectrum. Providing what wavelength the observation is going to be in can
surely help, infrared radiation varies in different nano-metrica (NIR, MIR,
FIR), each wavelength is good for a certain type of observation. Output
Sections: * Observing conditions: 1 paragraph, 3-4 sentences. We need to know
what conditions the observation is taking place and what location, mostly
because this plays a real key factor in terms of how the observation would be
carried out. Providing information about things like light pollution (Bortle
scale), density of air, outside temperature and more. * Instrument
sensitivity: 1 paragraph, 2-3 sentences. We need to know what are the
specifications of sensitivity of the instruments being used. Sensors or
cameras can detect so many details in comparison to the human eye, as well as
filters of eyepieces.
EXAMPLE INPUT * Scientific/main Goal AND target object: Our scientific goal is
to study the formation and evolution of young stars within the Orion Nebula.
The Orion Nebula is a vast region of star formation, and by observing it in
the infrared, we can penetrate the dust and gas that obscure these young stars
in visible light. * Kind of telescope and wavelength range: We will use the
Stratospheric Observatory for Infrared Astronomy (SOFIA), a 2.7-meter infrared
telescope mounted on a Boeing 747SP aircraft. SOFIA is capable of observing in
the mid-infrared and far-infrared wavelengths, from 5.4 to 25.5 microns. We
will specifically target the mid-infrared range (8-13 microns), which is
optimal for detecting the warm dust and gas associated with young stellar
objects. Additionally, we will employ a medium-resolution spectrograph to
obtain detailed information about the composition and physical properties of
the observed targets.
REFERENCE OUTPUT * Observing conditions: The observations will be conducted from an altitude of approximately 45,000 feet, above the majority of Earth’s water vapor, which absorbs infrared radiation. The flight will take place during a moonless night to minimize light pollution. The observing site will be chosen to have a low Bortle scale rating, indicating minimal light pollution from artificial sources. * Instrument sensitivity: SOFIA’s mid-infrared camera, FORCAST, has a sensitivity of approximately 0.1 Jy for a point source in a 10-minute integration time. The spectrograph has a spectral resolution of R 1000, allowing us to resolve individual spectral lines and determine the chemical composition of the target objects. |  Gemini-1.5-0409 OUTPUT * Observing conditions: SOFIA operates at altitudes of 38,000-45,000 feet, above 99% of the Earth’s water vapor, which allows for clear observations in the infrared spectrum. The observations will be conducted during nighttime flights to minimize background radiation from the Sun. Additionally, SOFIA’s mobility allows it to choose observing locations with minimal light pollution, typically over remote ocean areas or sparsely populated regions, corresponding to Bortle Class 1 or 2. * Instrument sensitivity: SOFIA’s mid-infrared instruments, such as the FORCAST camera and the FLITECAM spectrograph, are highly sensitive to faint infrared emissions. They can detect temperature variations as small as a few degrees Kelvin and resolve spatial details down to a few arcseconds. This sensitivity enables the detection and characterization of young stellar objects and their surrounding environments within the Orion Nebula.
Table 7: Sample response showcasing lack of detail in the generated output.
Pattern: Verbosity
---
TASK DESCRIPTION Task Objective: Write a review of a novel. Task Procedure:
You need to write an extensive review of a novel, you need to summarize and
give basic facts of the writer and novel, but also give value judgement.
Additional Notes: nan Input Sections: * Title: only one or several words, this
is the title of the book that you will review. * Writer: only a name, this is
the author’s name. * Pages: only a number, the number of pages in the novel. *
Introduction: 1 paragraph, a general introduction to both the author and their
work (the title that will be reviewed in the article) * Summary: 1 paragraph,
a specific summary of the book which gives the reader an insight into the
story without spoiling major events or the ending; it needs to stay
exciting/enticing. Output Sections:* Analysis: 2-3 paragraphs, varying
analyses of several aspects of the story, this includes content analysis, but
also linguistic and semantical analysis; what is meant by certain passages?
What are the cultural/societal connotations? In what way does the writer use
language and form? * Verdict: 1 paragraph, what do we think about the things
we just analysed? Is it of high quality or low quality? And in what ways? Why?
* Conclusion: 1 sentence, summarize everything said above and give a final
review in only one sentence.
EXAMPLE INPUT * Title: Dead in Long Beach, California * Writer: Venita
Blackburn * Pages: 157 * Introduction: A rising literary star, Venita
Blackburn, tackles the complexities of grief and unconventional coping
mechanisms in her captivating novel, "Dead in Long Beach, California." This
poignant tale explores the depths of human sorrow through the story of a young
woman grappling with loss. Blackburn delves into the lengths individuals go to
find solace, weaving a narrative that is both heartbreaking and thought-
provoking. * Summary: The novel follows the journey of a young woman who,
after the sudden death of her brother, stumbles upon an unexpected and
ethically dubious method of coping. Posing as her deceased sibling, she
immerses herself in his online world, seeking comfort in the digital memories
and connections he left behind. As she delves deeper into her brother’s
virtual existence, she uncovers hidden secrets and confronts the consequences
of her actions.
REFERENCE OUTPUT * Analysis: Blackburn’s novel is a profound exploration of the grieving process. Through the protagonist’s unconventional coping mechanism, she examines the ways in which technology can both connect and isolate us in times of loss. The author skillfully intertwines the protagonist’s emotional turmoil with the digital landscape, creating a thought-provoking commentary on the nature of identity and the boundaries between the living and the dead. Furthermore, Blackburn’s prose is both lyrical and incisive. For instance, her description of the protagonist staring at her brother’s empty online profile, "a hollow echo of a life once lived," perfectly captures the profound sense of emptiness and longing she experiences. The novel’s exploration of ethical dilemmas, such as the protagonist’s manipulation of online identities, is handled with sensitivity and nuance. It leaves readers to ponder the complexities of right and wrong in the face of overwhelming grief, sparking valuable discussions about the boundaries of acceptable coping mechanisms. * Verdict: "Dead in Long Beach, California" is a powerful and deeply moving novel. Blackburn’s ability to capture the raw emotions of grief and the complexities of our digital lives is commendable. Her exploration of unconventional coping mechanisms and the ethical implications they raise is both thought-provoking and timely. * Conclusion: Venita Blackburn’s "Dead in Long Beach, California" is a haunting and unforgettable tale that lingers long after the last page is turned. It serves as a powerful reminder of the enduring strength of human connection and the profound bond that transcends even death. |  GPT-4 OUTPUT * Analysis: Venita Blackburn’s "Dead in Long Beach, California" is a masterful exploration of grief and identity in the digital age. Through her concise yet powerful prose, Blackburn challenges the reader to consider the boundaries between the self and the other, the living and the dead. The novel’s central theme of coping with loss through digital impersonation raises poignant questions about the authenticity of online relationships and the ethics of assuming another’s identity posthumously. Blackburn’s use of language is both evocative and precise, capturing the protagonist’s turmoil and desperation with striking clarity. The narrative structure, alternating between the protagonist’s real-life struggles and her online escapades, effectively mirrors the fragmented nature of grief, enhancing the reader’s empathy for the protagonist’s plight. Furthermore, Blackburn subtly incorporates societal critiques, particularly concerning the internet’s role in shaping our perceptions of death and mourning. The novel’s setting in Long Beach, California, with its juxtaposition of urbanity and the vastness of the Pacific, further underscores the protagonist’s feelings of isolation and her quest for connection in an increasingly disconnected world. * Verdict: "Dead in Long Beach, California" is a compelling and thought-provoking novel that showcases Venita Blackburn’s talent for delving into complex emotional landscapes with nuance and sensitivity. The quality of Blackburn’s writing is evident in her ability to weave a deeply personal story with broader societal implications, making the novel both intimate and universally relevant. The ethical dilemmas and emotional depth explored in the book are handled with a deftness that invites readers to reflect on their own experiences of loss and the ways in which technology intersects with our most human experiences. Blackburn’s linguistic prowess and her skillful narrative construction make this novel a high-quality literary work that is both engaging and enlightening. It is a poignant reminder of the power of fiction to explore the nuances of the human condition. * Conclusion: Venita Blackburn’s "Dead in Long Beach, California" is a beautifully crafted novel that offers a profound and nuanced exploration of grief in the digital age, making it a must-read for those who appreciate literature that delves into the complexities of the human heart.
Table 8: Sample response showcasing verbosity in the generated output (note
the Analysis and Verdict sections). Pattern: Missing Information
---
TASK DESCRIPTION Task Objective: Writing a funding application for a medical
research project. Task Procedure: To write a funding application for a
research project, you need to start with a clear and concise idea for the
research project, identify matching funding sources, understand the funder’s
priorities. Then, write a detailed project description and prepare a specific
budget. Complete the application form provided by the funder and follow all
guidelines. Review the application, submit it, and await feedback. Additional
Notes: It is important to clearly define the research idea, understand the
funder’s priorities, write a detailed project description, prepare a specific
budget, and review before submitting. Common mistakes include overlooking
eligibility, using jargon, poor structure of the application, and an
overwhelming page layout. Input Sections: * Project Idea: 1 paragraph. A clear
and concise idea for the research project is essential. * Potential Funding
Sources: 1 paragraph. Estimated cost and identification of potential funding
sources that align with the research topic. Output Sections: * Funding
application: 2 paragraphs. The funding application will include a summary, an
introductory section including background and motivation, detailed project
description, and detailed budget. * Timeline: In addition, a detailed timeline
with key milestones is essential to demonstrate preparedness and careful
planning for the project.
EXAMPLE INPUT * Project Idea: We aim to develop a novel risk prediction model
for asthma exacerbations in children aged 1-6 years. By integrating clinical
data, environmental factors, and genetic markers, we will create a
comprehensive tool to identify children at high risk of exacerbations,
enabling early intervention and improved disease management. * Potential
Funding Sources: The National Heart, Lung, and Blood Institute (NHLBI) offers
funding opportunities for research on respiratory diseases, including asthma.
We estimate that the project will require a budget of $250,000 over five
years.
REFERENCE OUTPUT * Funding Application: Summary: This application proposes a comprehensive study to develop a risk prediction model for asthma exacerbations in children. The model will integrate clinical, environmental, and genetic data to identify high-risk individuals, facilitating early intervention and improved disease management. Introduction: Asthma is a common chronic disease that affects millions of children worldwide. Exacerbations can be life-threatening and lead to significant healthcare costs. Current prediction methods are inadequate, and there is an urgent need for more accurate tools. This project will address this gap by developing a novel risk prediction model that combines multiple factors to provide a personalized assessment of exacerbation risk. Project Description: We will recruit 500 children aged 1-6 years with asthma from the Aspen Medical Group and St. Paul Children’s Hospital. Data collection will include clinical assessments, environmental monitoring, and genetic testing. Baseline clinical assessments will include use of short-acting beta agonists, previous asthma-related emergency department attendances and hospitalizations, history of lower respiratory tract infections, environmental tobacco exposure. Environmental monitoring will be performed using indoor air-quality sensors. Genetic testing will include the analysis of specific genes linked to asthma. Outcome events will include asthma-related emergency hospitalization. We will use survival models to estimate the hazard ratio associated with key predictors. The model will be validated using an independent cohort of children. Budget: The total budget for the project is $250,000 over five years. This includes costs for participant recruitment, data collection, analysis, and dissemination. The budget includes:* $150,000: Staff salaries * $50,000: Patient recruitment data, collection and analysis * $30,000: Equipment and supplies * $20,000: Travel expenses * Timeline: * Year 1: Participant recruitment and data collection: This include three months for participant recruitment and nine months for data collection. * Year 2: Model development and validation: This includes nine months for model development and three months for validation. * Year 3: Model refinement, implementation, and real-world testing. * Year 4: Dissemination of results and exploration of clinical applications: This include three months for preparing and publishing findings, and nine months for exploring clinical applications. * Year 5: Project evaluation and future directions: The entire year will be dedicated to project evaluation and planning for future research. |  GPT-4 OUTPUT * Funding Application: Our project proposes the development of an innovative risk prediction model for asthma exacerbations in children aged 1-6 years, a critical period for asthma management and intervention. By leveraging a multidisciplinary approach that combines clinical data, environmental factors, and genetic markers, our model aims to revolutionize how healthcare providers identify children at high risk of asthma exacerbations. This early identification will facilitate timely interventions, potentially reducing the frequency and severity of exacerbations and improving overall disease management and patient outcomes. The project will unfold in several phases, beginning with the collection and analysis of clinical data from pediatric asthma patients. Concurrently, we will conduct a comprehensive review of environmental factors known to influence asthma exacerbations, such as air quality and allergen exposure, and perform genetic analyses to identify markers associated with increased risk. This integrated data will form the basis of our predictive model, which will be rigorously tested and validated in clinical settings. To support this ambitious project, we are seeking funding of $250,000 from the National Heart, Lung, and Blood Institute (NHLBI), an organization that prioritizes research on respiratory diseases, including asthma. This budget will cover costs associated with data collection, analysis, personnel, and dissemination of findings, ensuring a thorough and impactful study. * Timeline: Year 1: - Q1-Q2: Project initiation, including team assembly and training, and development of data collection protocols. - Q3-Q4: Begin data collection on clinical data and environmental factors, and initiate genetic marker identification. Year 2: - Q1-Q2: Continue data collection and start preliminary data analysis. - Q3-Q4: Development of the initial model based on collected data and initial testing. Year 3: - Q1-Q2: Refinement of the model based on initial testing feedback and integration of additional data as necessary. - Q3-Q4: Begin large-scale validation of the model in clinical settings. Year 4: - Q1-Q4: Continued validation and refinement of the model, including adjustments based on ongoing feedback from clinical applications. Year 5: - Q1-Q2: Finalization of the model and preparation of materials for dissemination. - Q3-Q4: Dissemination of findings through publications, presentations at conferences, and direct outreach to healthcare providers. This timeline demonstrates our commitment to careful planning and preparedness, ensuring that each phase of the project builds upon the last towards the successful development and implementation of our novel asthma risk prediction model.
Table 9: Sample response showcasing missing information in the generated
output (note the Funding application section). Pattern: Subjectivity
---
TASK DESCRIPTION Task Objective: Writing a plan for a photography shoot Task
Procedure: I would start by considering my inspirations and previous research.
I will then plan out the photography shoot by writing a description of each
shot that I expect to get, the layout and the lighting techniques I will use.
Additional Notes: Make sure not to overly complicate the plan but stay clear
and concise. Input Sections: * Goals: 1 paragraph. The goal of the photoshoot
based on the event being photographed, the requirements from the photographs
and how many final photographs are needed for release. * Target Audience: 1-2
sentences. Who will the photographs be released to. Output Sections: *
Location: 1 paragraph, 2 sentences. Choose Location: Select an appropriate
setting that aligns with the objective. * Lighting: 1 paragraph, 2-3
sentences. Plan for natural or artificial lighting, as needed, decide on
lighting techniques and angles, and how many of each will aim for. * Layout:
1-2 Sentences. Choose layout of props/people etc. and detail exact positions
for each shot..
EXAMPLE INPUT * Goals: The goal of this photoshoot is to capture the essence
of the serene and rugged beauty of the Scottish Highlands. We aim to showcase
the rolling hills, dramatic cliffs, and crystal-clear lochs that make this
region so unique. We will need to capture a variety of images, including wide-
angle landscape shots, intimate portraits of the local flora and fauna, and
detailed shots of the region’s unique geological features. We will need to
deliver a total of 30 final photographs for release. * Target Audience: The
target audience for these photographs is potential tourists and travel
enthusiasts interested in exploring the Scottish Highlands. These photographs
will be used in a variety of marketing materials, including brochures, website
banners, and social media posts.
REFERENCE OUTPUT * Location: To achieve the desired effect, we will be shooting on location in the Cairngorms National Park. We will visit the Highland Wildlife Park, Rothiemurchus Estate, and Balmoral Castle. These location offers a wide variety of landscapes, from towering mountains and windswept moors to tranquil forests, castles and shimmering lochs. * Lighting: We will a primarily use natural lighting for this photoshoot, taking advantage of the soft morning light, dramatic evening skies, the golden hour and sunsets. We will hope to visit in different weathers such as misty rain or blue skies to get a variety of shots. We may also use artificial lighting such as a flash if absolutely necessary for certain shots, such as portraits or close-ups of specific details. * Layout: For the landscape shots, we will use the rule of thirds in some shots (as to not get repetitive) to create balanced and dynamic compositions . We will also use leading lines, such as paths and streams, to draw the viewer’s eye into the scene. For the close-up shots, we will focus on capturing the intricate details of the subject, such as the veins in a leaf or the texture of a rock. We hope to include wildlife such as red deer, birds or insects in the photos. We will bring no props, and use only what nature and the landscape provides. We will photograph woodlands, streams, lakes, forests, castles and mountains. We will aim for 10 close up and wildlife shots and 20 landscape photos. |  Claude-3 Opus OUTPUT * Location: The photoshoot will take place in the Scottish Highlands, specifically focusing on the areas around Glencoe and the Isle of Skye. These locations offer a diverse range of landscapes, from the rugged peaks of the Three Sisters in Glencoe to the stunning coastline and rock formations of the Quiraing on the Isle of Skye. * Lighting: Given the unpredictable weather conditions in the Scottish Highlands, we will primarily rely on natural lighting for our shots. We will aim to shoot during the golden hours of sunrise and sunset to capture the warm, soft light that enhances the natural beauty of the landscape. Additionally, we will experiment with backlighting and side-lighting techniques to add depth and dimension to our shots, particularly when photographing the local flora and fauna. * Layout: For the wide-angle landscape shots, we will position our cameras to capture the vastness and scale of the Scottish Highlands, emphasizing the dramatic cliffs, rolling hills, and expansive lochs. When photographing the local flora and fauna, we will use a shallow depth of field to isolate our subjects and create intimate, detailed portraits that showcase the unique characteristics of each species.
Table 10: Sample response showcasing subjectivity in the task which is not
accounted for during automatic evaluation. Query Formulation Prompt
---
Generate 10 search queries for finding specific examples of the given task
from the specified field. The search queries should be brief and request
documents in more specific contexts than the given task. We would like the
documents to contain real examples of the task. List the queries and nothing
else. FIELD: Visual Arts (Graphic design) TASK: The objective of this task is
to write a catalog entry for an art exhibition. QUERIES: 1) Example of catalog
entry for art exhibition 2) Catalog entry art exhibition Dali 3) Notable art
catalog entries 2023 4) memorable art catalog entries 2000s 5) catalog entry
for jackson pollock painting 6) frida kahlo painting catalog entry 7) picasso
guernica catalog entry 8) da vinci mona lisa catalog entry 9) The Great Wave
off Kanagawa catalog entry 10) renaissance art exhibition catalog entry FIELD:
[FIELD] TASK: [TASK] QUERIES:
Table 11: Prompt used for generating specific search queries for a task.
Domain Name Prompt
---
List 20-30 URLs to domain names which will be useful to find real examples for
the given task. These websites should be reliable, trustworthy and
authoritative sources for an expert in the field. They should be ranked by
their likely usefulness. FIELD: Engineering and Technology (NLP research)
TASK:Summarizing related work on an NLP subproblem. URLs: 1) arxiv.org 2)
aclweb.org 3) ldc.upenn.edu 4) nlp.stanford.edu 5) aclanthology.org 6)
towardsdatascience.com 7) semanticscholar.org 8) openreview.net 9) medium.com
10) nature.com 11) transacl.org 12) cambridge.org 13) iclr.cc 14) aaai.org 15)
academic.microsoft.com 16) nips.cc 17) onlinelibrary.wiley.com 18)
link.springer.com 19) naacl.org 20) plos.org FIELD: [FIELD] TASK: [TASK] URLs:
Table 12: Prompt used for searching for authoritative domain names for a task.
Example Generation Prompt
---
You are given a description of a task from the field of [FIELD] by an expert.
Generate a concrete example of all the Input Sections and Output Sections
listed for the given TASK DESCRIPTION. The example should resemble a real
example that is written by an expert in the field, and should be highly
technical and detailed. Further instructions:- You are also given CONTEXT in
the form of Passages from web documents and will need to generate an example
based on this CONTEXT. Make sure to generate the example based on the provided
CONTEXT. If the CONTEXT is insufficient, you can say "The context is
insufficient". - Make sure the length of each section matches the required
length and the section headers are exactly the same. - The example should be
highly detailed, and not be generic and vague.====CONTEXT====[CONTEXT]====TASK
DESCRIPTION====[TASK DESCRIPTION]====EXAMPLE====
Table 13: Prompt used for generating initial examples for a task. Critique
Generation Prompt
---
You are an expert in the field of [FIELD]. You are given a task description of
a writing task from your field and an imperfect example for this task, where
an example is a concrete sample of the task. You need to describe what is
lacking in the example for the task. You are given a list of properties based
on which you should critique the example:* Inconsistencies: Are there any
inconsistencies in the information provided across the input and output?*
Factual Inaccuracies: Are there any factual inaccuracies in the information
presented in the input or how the output is inferred? * Structure: Are there
any issues with how closely the example follows the instructions specified in
the task description? This includes information requested in the task but
missing in the example, or mismatch in the length required for a section.*
Depth: How could the example benefit from more detail? Note that the example
should resemble what an expert might write and so it should not be vague with
details.====TASK DESCRIPTION==== [TASK DESCRIPTION] ====EXAMPLE==== [EXAMPLE]
====Critique====
Table 14: Prompt used for generating critiques for model-generated examples.
Output Generation Prompt
---
You need to perform a writing task from the field of [FIELD]. You are given
(1) a task description which contains input and output sections, and (2) an
example input for this task, which is a sample of the input sections of the
task with concrete details. You need to generate the output sections for the
given example input. - Make sure the length of each output section matches the
required length and the section headers are exactly the same. - Make sure the
output follows the structure of the output sections in the task description,
is factually accurate and detailed.====TASK DESCRIPTION====[TASK
DESCRIPTION]====EXAMPLE INPUT====[EXAMPLE INPUT]====EXAMPLE OUTPUT====
Table 15: Prompt used for generating outputs from candidate models for
evaluation. LM-based Evaluation Prompt
---
You are an expert in the field of [FIELD]. You are given a task description of
a writing task from your field. For this task description, you are given an
input example, which is a concrete sample of the input sections of this task,
as well as the reference output, which is the gold standard output for this
input. You will be given two candidate outputs for the input example and you
need to judge which output is better by comparing it to the reference output.
First, you should say "**output 1**" if output 1 is better, "**output 2**" if
output 2 is better and "**same**", if the two outputs are equivalent in
quality (note the stars). Then you should explain why you picked this output.
**Important: Keep in mind that longer outputs are not necessarily better
quality outputs. Being concise is a good quality for outputs.** ====TASK
DESCRIPTION====[TASK DESCRIPTION]====INPUT EXAMPLE====[EXAMPLE
INPUT]====REFERENCE OUTPUT====[REFERENCE OUTPUT]====EXAMPLE OUTPUT
1====[EXAMPLE OUTPUT 1]====EXAMPLE OUTPUT 2====[EXAMPLE OUTPUT
2]====Decision====
Table 16: Prompt used for generating LM-based judgements.