judgements [2, 4] has looked at research questions related to how to best
evaluate IR systems by asking human assessors which of two results is the
better given an information need. The recent availability of LLMs has opened
the possibility to use them to automatically generate relevance assessments in
the form of preference judgements. While the idea of automatically generated
judgements has been looked at before [1], new-generation LLMs drive us to re-
ask the question of whether human assessors are still necessary.

New models tend to fail in a different and more diverse way compared to
traditional approaches. Failure points for old models were more uniform and
clear, with new systems it is harder to predict in which ways the model will
fail. In most cases, LLMs (especially for what concerns generative aspects)
focus on entertainment tasks. Models tend to report false facts in such a
convincing way that they need to be carefully read by some expert to identify
lacking factuality (e.g., Michel Foucault
simulation212121https://www.youtube.com/watch?v=L6c0xeAqEz4E).

Figure 4: The three most relevant components in our system: the human
assessor, the Large Language Model (LLM) that can help humans or replace them
in annotating documents for relevance, and the system that we want to evaluate
using the newly produced relevance judgements.

Our motivation to investigate the possibility of using LLMs in order to
provide automatic annotations stems from some fundamental research questions
that can be summarized as follows.

  * •

RQ1: In which way automatic approaches, and in particular LLMs, can help
assessors with the assessment task to yield the most reliable annotations
while improving the efficiency of the annotation process? This question raises
other interesting related inquiries. For example, if we were to build such a
mixed human-machine annotation paradigm, which held out (not provided to the
IR system) supporting information about the topic would yield the best and
fastest annotations? What weighting between human and LLMs and AI-assisted
annotations is ideal?

  * •

RQ2: Can machines (either in the form of LLMs or in general as Artificial
Intelligence (AI) models) replace humans in assessing and annotating? This
question raises also concerns about what annotation target (e.g., relevance
labeling, summarization, paragraph highlighting, exam questions [5]) would
yield the best and fastest annotations.

  * •

RQ3: What are the conditions under which human assessors cannot be replaced by
machines? Alternatively, in which role can the Human assessor most
productively provide relevance assessments?

Answering the questions mentioned would also require finding viable solutions
for a set of additional questions and open issues that touch a number of IR
evaluation process steps.

  * •

Assessors And Collections:

    * –

How to use LLM to help assessors: some examples of possible usages include,
summarising text, associating keywords and identifying the content of long
podcasts to help assessors annotate the documents, for example by highlighting
relevant fragments of text/podcast or segments with correct answers.

    * –

What is the effective role of the human assessor in annotating material for
generative models? Should the annotator provide input at the beginning of the
pipeline, by annotating the original documents, or are they more useful
downstream, after the task has been carried out?

    * –

Generative models can be used to create new collections: corpora,
conversations, queries, abstracts and so on.

  * •

LLM and generative models to retrieve information in a broader sense:

    * –

IR tasks that employ LLMs have the means to provide more details: often a
single answer is not satisfactory for the user. How to support the user in
exploring the results further (for example via links and connected pages).
Generative models can help, but is this helpful when the model simply
generates the response without knowing where it comes from? In many cases, the
user is not interested in receiving only the direct/short answer, but rather
in seeing which documents contain it and related pieces of information to
expand their knowledge.

  * •

LLMs as an evaluation tool:

    * –

The model is biased: how can we use it to evaluate itself? If a model has been
trained on biased data, then also the evaluation is prone to the same biases.
How to detect and account for such biases?

  * •

Evaluating LLMs and their trustworthiness:

    * –

Can we find a way to understand and measure to what level we can trust the
results of a generative model?

    * –

How to carry out fact-checking, for example by identifying the source of
information of a generative model and verifying that it is presented
accurately.

    * –

Distinguish between human and machine-generated data: Important for many
tasks, such as journalism, where it is of uttermost importance to verify the
information. Human-generated data is more trusted.

We argue that the collaboration between humans and ML, especially under the
form of LLMs, could be abstracted in the form of a spectrum. On the two
extremes of this spectrum, we have either the human or the machine entirely
tasked to annotate documents for relevance with respect to a query. Within the
spectrum, humans and LLMs interact to a different extent. Theoretically, such
a spectrum corresponds also to moving from highly expansive annotations in
terms of human effort, cost and time, but with high-quality annotations, to a
much less expensive annotation procedure with also a decreased annotation
quality. We also argue that something exists beyond the spectrum; it
corresponds to the scenario in which the machine overcomes the human, by
producing relevance judgments without any form of bias. We observed this
phenomenon happening already in several tasks and scenarios, and therefore we
can aspect this to happen also with respect to the construction of the
relevance judgments.

The remainder of this chapter is organized as follows: Subsection 3.0.10
reports details on the current state of the art and limitations associated
with the current usage of LLMs ad AI in annotating documents. Subsection
3.0.11 illustrates our proposal of a spectrum of possible interactions between
the human and the machine, to provide more efficient and effective annotations
and relevance judgments. Subsection 3.0.12 outlines a possible experimental
protocol that would allow us to verify at what point modern LLMs and whether
they can be used to produce automatically relevance judgements.

####  3.0.10 State of the Art, Idea, and Gaps

##### Using LLMs to Generate Annotations and Label Automatically

Potential uses of LLMs to annotate documents, extract snippets, summarize and,
in the end, annotate documents for relevance. If this can be made to work
reliably, it opens up many opportunities for evaluation. For example, the LLM
can be used directly to evaluate the output of other large language models
(for example in summarization).

Assessments can arise from different sources, with different levels of quality
and collection costs as follows.

  * •

Human assessors or, in the enterprise scenario, final users. This scenario, at
the current time, is the most expansive, but also likely to produce high-
quality annotations.

  * •

Human assessors aided by mild automatic support systems (e.g. remove
redundancy, encourage consistency)

  * •

Half of the judgments are produced by human assessors and half of the
judgments are produced automatically.

  * •

Automatic annotation of a collection, which is verified and corrected by human
intervention.

  * •

At some point even a fully automatic assessment.

An additional axis describes the type of annotations. Typically an annotation
is a graded relevance judgment, but for example in EXAM [5], humans are used
for generating questions instead. This can be generalized by asking human
assessors for something different than traditional annotation while some
Machine Learning (ML) converts the human responses into relevance assessments.
This follows the paradigm of Competence Partitioning of Human-Machine-
Collaboration where humans and machines are performing tasks they are best at
(not vice versa).

One concern is that fully automatic assessment with LLMs can be very
expensive, which is also the reason why we consider the application of LLMs as
part of the retrieval process. In such a case, we could reduce the cost by
considering a teacher-student training paradigm (knowledge distillation) in
which a large and expensive LLM is used to train a smaller model that is less
expensive to run.

Not all IR tasks focus on topics. For example, one may want to search for
podcasts where two or more people interact or with a particular style. Another
issue is regarding truth. For example, finding a podcast for the query “does
lemon cure cancer?” that talks about healing cancer with lemon might be on
topic. Nevertheless, it is unlikely to be factually correct, and therefore not
relevant to correctly answering the information needs. To overcome this issue,
assessors have to access external information to determine the trustworthiness
of a source, or the truthfulness of a document. In a similar way, we can
assume our LLM is used as an oracle that accesses external facts, verified by
humans. To properly support different tasks, human intervention can be plugged
into the collection and annotation of additional facts, to define relevance.

There are open questions for the special case of 100%-machine/0%human. How is
this ranking evaluation different from being an approach that produces a
ranking? (circularity problem). We can use multiple LLMs, possibly based on
different rationales, such that it is possible to define an inter-annotation
systems agreement, in which different systems are used to verify if there is
an agreement between each other. An alternative approach is to endow the
evaluation with additional information about relevant facts/questions/nuggets
that the system (under evaluation) does not have access to.

It is yet to be understood what the risks associated with such technology are:
it is likely that in the next few years, we will assist in a substantial
increase in the usage of LLMs to replace human annotators. Nevertheless, a
similar change in terms of data collection paradigm was observed with the
increased use of crowd assessor. Up to that moment, annotations were typically
made by in-house experts. Then, such annotation tasks were delegated to crowd
workers, with a substantial decrease in terms of quality of the annotation,
compensated by a huge increase in annotated data. It is a concern that
machine-annotated assessments might degrade the quality, while dramatically
increasing the number of annotations available.

The Cranfield paradigm [6] is based on simplifying assumptions that make
manual evaluation feasible: 1) independence of queries; 2) independence of
relevance of documents; 3) Relevance is static (and not changing in time).
Recently, the field is diverging from this paradigm, for example with TREC CAR
and TREC CAsT/iKAT where the information needs are developing as the user
learns more about the domain. The TREC Evaluation of CAST describes a tree of
connected information needs, where one conversation takes a path through the
tree. The Human-Machine evaluation paradigm might make it feasible to assess
more connected (and hence, realistic) definitions of relevance.

####  3.0.11 Collaborative Human-Machine Relevance Judgments

We can describe a spectrum of Collaborative-Human-Machine paradigms to create
relevance judgments, where the weighting of human contributions vs machine
contributions changes along the spectrum.

Figure 5: A spectrum of Collaborative-Human-Machine paradigms to create
relevance judgments.

  * •

Only Human (100%H / 0%M): On one extreme, the human will do all assessments
manually without any kind of support.

  * •

Human with assessment system (99%H / 1%M): This is a more realistic case for
how TREC assessment is conducted, where humans have full control of what is
relevant but are supported in the following ways: Humans can define “scan
terms” that will be highlighted in the text, can limit view the pool that is
already judged, ordering documents so that similar documents are near one
another, produce readable presentations of retrieve content.

  * •

Human with document summaries (80%H/ 20%M): A text summarization model
produces a generative summary representation of the document to be judged. The
human assessor judges the representation, which is more efficient to do.

  * •

EXAM (60%H / 30% M): For each query, the human defines information nuggets
that are relevant (e.g. exam questions). The machine is trained to
automatically determine how many test nuggets are contained in the retrieved
results (e.g. via a Q/A system).

  * •

Equal contribution (50%H / 50%M): A theoretic midpoint in the collaborative
spectrum. Humans perform tasks that humans are good at. Machines perform the
tasks that machines are good at. It is yet to be concretely defined what this
might be.

  * •

3-Brain Setup (32%H / 58%M): Two machines each generate an assessment, and a
human will select the best of the two assessments (+verification). Human
decision trumps machines’.

  * •

LLM for first pass + human verification (30%H / 60%M): A first-pass assessment
of the LLM is automatically produced as a suggestion. This can also be an
assessment-supporting surrogate prediction like a rationale. The human
assessment is based on this suggestion, but the human will have the final say.

  * •

LLM replaces humans completely (0%H / 100%M): We explore the possibility that
a fully automatic assessment system might be as good as a human in producing
high-quality relevance judgments.

  * •

LLM is beyond human (0%H / 100%M): Given known biases in human assessments, we
contemplate the possibility that the automatic assessments might even surpass
the human in terms of quality. While not feasible at the current time, this is
an important case to consider when we evaluate the HMC evaluation.

##### Use LLMs to Help Humans in Annotating Documents

LLMs could be successfully applied in helping human assessors with annotating
data. For example, LLMs might be particularly useful in recognizing near
duplicates and using them to verify if the two near duplicates share the same
relevance annotation – with the human entering the loop only in those cases
where the system has a high degree of uncertainty.

Related to the case of (100%H / 0%M), we have the human-in-the-loop, helping
the system in realizing its annotation goal. Such help might include providing
annotated facts or verifying the annotation after a first pass from the
system. In the 50%/50% case, equal contributions, we have a substantial
equilibrium between both the human and the machine. We refer to this scenario
as competence partitioning: the task is assigned to either the human or the
machine, depending on who is currently better at the current moment. On the
other side of the spectrum (%M > %H), the scenario is called model-in-the-
loop: the model offers its contribution in organizing the data, where the
human is used as a verification step. The concern is that any bias in the LLM
might be affecting the relevance assessments, as the human will not be able to
correct for information it will not see.

An alternative approach to the collaborative one is a complementary one, where
the human and the machine both produce judgments, but different ones. This
then becomes a task allocation problem where the aim is to predict who among
the human and the machine assessor is best suited for any given judgment.

##### Beyond Human Performance

We could expect that, at a certain point in the future, the LLMs will overcome
humans in a number of tasks that can be reconducted to annotate the documents.
Humans are likely to make mistakes when annotating documents and are limited
in the time dedicated to the annotation. In contrast, LLMs are likely to be
more self-consistent and potentially capable of annotating all the documents
perfectly. Machines can also annotate a much larger number of data points.

Furthermore, we have a series of assumptions, such as the fact that relevance
does not change through time, that are enforced to make evaluation tractable.
These assumptions can be relaxed if the machine annotates automatically.

It is an open issue in recognizing when the human is failing. All the above
strategies assume human annotations are the gold standard without errors. This
assumption is strong: the LLM, having access to more information, might be
able to correct human mistakes.

We are likely to reach the limit of measurement: we will not be able to use
differences between the current evaluation paradigms to evaluate such models.
A problem is that if we surpass the quality of only human-annotated data, we
will not be able to detect this if we use only human-annotated data as a gold
standard. will not suffice and will fail in providing a gold standard.

Another research question is to identify optimal competence partitioning. One
idea is to use the LLM to generate rationales for explaining the relevance.
While humans are often considered experts for rational generation, recent
advancements, including chatGPT, suggest that we are on the verge of a shift
of paradigm, with LLMs constantly improving in identifying why a document is
(non)-relevant, either considering information with the document, or other
relevant external pieces of information.

##### Trust, Correctness, and Inter-annotator Agreement

One important difference between humans and automatic assessors concerns the
assessment sample size. While it is possible to hire multiple assessors to
annotate the documents and, possibly, resolve disagreements between
annotators, this is not that trivial in the automatic assessor case. We can
expect that LLMs which are trained on similar corpora will likely produce
correlated answers — but we don’t know whether these are correct. A possible
solution to this would include the usage of different subcorpora based on
different sets of documents. This, in turn, could lead to personalized LLMs,
fine-tuned on data from different types of users, which would allow to auto-
annotate documents directly according to the user’s subjective point of view,
while also helping with increasing the pool of annotations collected. While
this technology is not available yet, mostly due to computational reasons, we
expect it to be available in a few years.

A related idea that can be implemented today is to allow LLMs to learn by
observing human annotators performing the task or following an active learning
paradigm. The LLM starts with mild suggestions to the user on how to annotate
the documents, then it continues to learn by considering actual decisions made
by the annotator and finally improving the quality of the suggestions
provided.

####  3.0.12 Next Steps

Tables 1 and 2 report two examples of document annotation done with two well-
known LLMs: YouChat222222https://you.com/ and
ChatGPT232323https://chat.openai.com/. It is interesting to notice that, in
both cases, both models provided the correct answer, correctly identifying the
passage which was annotated as more relevant. It is possible to observe that,
while ChatGPT simply repeats the relevant passage, YouChat is capable of
correctly identifying the reason why a passage is more relevant than the
other.

Table 1: Response of two well-known LLMs in comparing two passages as answer for the query 54_8 from CAsT ’19: “What are anemia’s possible causes?” Prompt |  Select the passage that answers the question better.
\- If both answers are similar, select the one with the least extraneous
information.
\- If both answers are still similar, select the one with the best formatting.

Question:

What are anemia’s possible causes?

Passage 1:

If you have anemia, your blood does not carry enough oxygen to the rest of
your body. The most common cause of anemia is not having enough iron. Your
body needs iron to make hemoglobin. Hemoglobin is an iron-rich protein that
gives the red color to blood. It carries oxygen from the lungs to the rest of
the body. Anemia has three main causes: blood loss, lack of red blood cell
production, and high rates of red blood cell destruction

passage 2:

Broadly, causes of anemia may be classified as impaired red blood cell (RBC)
production, increased RBC destruction (hemolytic anemias), blood loss and
fluid overload (hypervolemia
---|---
ChatGPT |  Passage 1:

If you have anemia, your blood does not carry enough oxygen to the rest of
your body. The most common cause of anemia is not having enough iron. Your
body needs iron to make hemoglobin. Hemoglobin is an iron-rich protein that
gives the red color to blood. It carries oxygen from the lungs to the rest of
the body. Anemia has three main causes: blood loss, lack of red blood cell
production, and high rates of red blood cell destruction.
YouChat |  Passage 1 is the better answer, as it provides an explanation of what anemia is, and then lists the three main causes in detail, which is the most relevant information for the question. Passage 2, while providing some good background information, does not provide the same level of specificity as Passage 1, and includes extraneous information that is not directly relevant to the question.
Table 2: Response of two well-known LLMs in comparing two passages as answer for the query 67_8 from CAsT ’19: “What is a DC half smoke?” Prompt |  Select the passage that answers the question better.
\- If both answers are similar, select the one with the least extraneous
information.
\- If both answers are still similar, select the one with the best formatting.

Question:

What is a DC half smoke?

Passage 1:

However, celebrities at Ben’s are not limited to American citizens. In 1998,
former DC mayor Marion Barry described having traveled to Ghana and meeting
the Mayor of Accra, an alumnus of Howard University, whose greeting was, “Glad
to have you in Accra. Is Ben’s Chili Bowl still there?” And when French
President Nicolas Sarkozy and his wife Carla Bruni-Sarkozy visited Washington,
D.C. in March 2010, they reportedly each had two of Ben’s half smokes during
their visit to the restaurant.

Passage 2:

A half-smoke is a “local sausage delicacy” found in Washington, D.C. and the
surrounding region. Similar to a hot dog, but usually larger, spicier, and
with more coarsely-ground meat, the sausage is often half-pork and half-beef,
smoked, and served with herbs, onion, and chili sauce.
---|---
ChatGPT |  Passage 2:

A half-smoke is a local sausage delicacy found in Washington, D.C. and the
surrounding region. Similar to a hot dog, but usually larger, spicier, and
with more coarsely-ground meat, the sausage is often half-pork and half-beef,
smoked, and served with herbs, onion, and chili sauce.
YouChat |  Passage 2 is the better answer as it more directly answers the question and does not include any extraneous information.

To assess the feasibility of the proposed approaches, next steps would include
an experimental comparison of the different Collaborative-Human-Machine
paradigms. This should include multiple test collections (e.g., TREC-8 and
TREC Deep Learning), multiple types of judgments (e.g., binary, graded,
preference), and multiple models (e.g., GPT-2, GPT-3, chatGPT, etc.).
Comparison between human-generated judgments and machine-generated judgments
may be performed both using inter-assessor agreement metrics as well as IR
system ranking correlation methods.

## References

  * [1] Stefan Büttcher, Charles L. A. Clarke, Peter C. K. Yeung, and Ian Soboroff.  Reliable information retrieval evaluation with incomplete and biased judgements.  In Wessel Kraaij, Arjen P. de Vries, Charles L. A. Clarke, Norbert Fuhr, and Noriko Kando, editors, SIGIR 2007: Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands, July 23-27, 2007, pages 63–70. ACM, 2007.
  * [2] Charles L. A. Clarke, Alexandra Vtyurina, and Mark D. Smucker.  Assessing top-k preferences.  ACM Trans. Inf. Syst., 39(3):33:1–33:21, 2021.
  * [3] Donna Harman.  Information retrieval evaluation.  2011\.
  * [4] Martin Potthast, Lukas Gienapp, Florian Euchner, Nick Heilenkötter, Nico Weidmann, Henning Wachsmuth, Benno Stein, and Matthias Hagen.  Argument search: Assessing argument relevance.  In Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer, editors, Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1117–1120. ACM, 2019.
  * [5] David P. Sander and Laura Dietz.  EXAM: how to evaluate retrieve-and-generate systems for users who do not (yet) know what they want.  In Omar Alonso, Stefano Marchesin, Marc Najork, and Gianmaria Silvello, editors, Proceedings of the Second International Conference on Design of Experimental Search & Information REtrieval Systems, Padova, Italy, September 15-18, 2021, volume 2950 of CEUR Workshop Proceedings, pages 136–146. CEUR-WS.org, 2021.
  * [6] Ellen M. Voorhees.  The philosophy of information retrieval evaluation.  In Carol Peters, Martin Braschler, Julio Gonzalo, and Michael Kluck, editors, Evaluation of Cross-Language Information Retrieval Systems, Second Workshop of the Cross-Language Evaluation Forum, CLEF 2001, Darmstadt, Germany, September 3-4, 2001, Revised Papers, volume 2406 of Lecture Notes in Computer Science, pages 355–370. Springer, 2001.

Overcoming Methodological Challenges in Information Retrieval and Recommender
Systems through Awareness and Education [Christine Bauer, Maik Fröbe, Dietmar
Jannach, Udo Kruschwitz, Paolo Rosso, Damiano Spina, Nava Tintarev]Christine
Bauer (Utrecht University, NL<EMAIL_ADDRESS>
Maik Fröbe (Friedrich-Schiller-Universität Jena, DE<EMAIL_ADDRESS>
Dietmar Jannach (University of Klagenfurt, AT<EMAIL_ADDRESS>
Udo Kruschwitz (University of Regensburg, DE<EMAIL_ADDRESS>
Paolo Rosso (Technical University of Valencia, ES<EMAIL_ADDRESS>
Damiano Spina (RMIT University, AU<EMAIL_ADDRESS>
Nava Tintarev (Maastricht University, NL<EMAIL_ADDRESS>
####  3.0.13 Background & Motivation

In recent years, we have observed a substantial increase in research in IR and
RS. To a large extent, this increase is fueled by progress in ML (deep
learning) technology. As a result, countless papers are nowadays published
each year which report that they improved the state-of-the-art when adopting
common experimental procedures to evaluate ML based systems. However, a number
of issues were identified in the past few years regarding these reported
findings and their interpretation. For example, both in IR and RS, studies
point to methodological issues in _offline_ experiments, where researchers for
example compare their models against weak or non-optimized baselines or where
researchers optimize their models on test data rather than on held-out
validation data [4, 13, 48, 53].

Besides these issues in offline experiments, questions concerning the
_ecological validity_ of the reported findings are raised increasingly.
Ecological validity measures how generalizable experimental findings are to
the real world. An example of this problem in information retrieval is the
known problem of mismatch between offline effectiveness measurement and user
satisfaction measured with online experimentation [10, 5, 40, 46, 56] or when
the definition of relevance does not consider the effect on a searcher and
their decision-making. For example, the order of search results, and the
viewpoints represented therein, can shift undecided voters toward any
particular candidate if high-ranking search results support that candidate
[19]. This phenomenon—often referred to as the  Search Engine Manipulation
Effect (SEME) —has been demonstrated for both politics [19, 20] and health [2,
43]. By being aware of the phenomena, methods have been adapted to measure its
presence [14, 15], and studies to evaluate when and how it affects human
decision-makers [16]. Similar questions of ecological validity were also
raised in the RS field regarding the suitability of commonly used
computational accuracy metrics as predictors of the impact and value such
systems have on users in the real world. Several studies indeed indicate that
the outcomes of offline experiments are often _not_ good proxies of real-world
performance indicators such as user satisfaction, engagement, or revenue [7,
25, 30].

Overall, these observations point to a number of open challenges in how
experimentation is predominantly done in the field of information access
systems. Ultimately, this leads to the questions of _(i)_ how much progress we
really make despite the large number of research works that are published
every year [4, 35, 57] and _(ii)_ how effective we are in sharing and
translating the knowledge we currently have for doing IR and RS
experimentation [23, 45]. One major cause for the mentioned issues, for
example, seems to lie in the somewhat narrow way we tend to evaluate
information retrieval and recommender systems: primarily based on various
computational effectiveness measures. In reality, information access systems
are interactive systems used over longer periods of time, i.e., they may only
be assessed holistically if the user’s perspective (task and context) is taken
into account, cf. [36, 51, 55]. Studies on long-term impact furthermore need
to consider the wider scope of stakeholders [6, 30]. Moreover, for several
types of information access systems, the specific and potentially competing
interests of multiple stakeholders have to be taken into account [6]. Typical
stakeholders in a recommendation scenario include not only the consumers who
receive recommendations but also recommendation service providers who for
example want to maximize their revenue through the recommendations [29, 30].

Various factors contribute to our somewhat limited view of such systems, e.g.,
the difficulties of getting access to real systems and real-world data for
evaluation purposes. Unfortunately, the IR and RS research communities to a
certain extent seem to have accepted to live with the limitations of the
predominant evaluation practices of today. Even more worryingly, the described
narrow evaluation approach has become more or less a standard in the
scientific literature, and there is not much debate and—as we
believe—sometimes even limited awareness of the various limitations of our
evaluation practices.

There seems to be no easy and quick way out of this situation, even though
some of the problems are known for many years now [17, 5, 32, 46]. However, we
argue that improved _education_ of the various actors in the research
ecosystem (including students, educators, and scholars) is one key approach to
improve our experimentation practices and ensure real-world impact in the
future. As will be discussed in the next sections, better training in
experimentation practices is not only important for students, but also for
academic teachers, research scholars, practitioners and different types of
decision-makers in academia, business, and other organizations. This will, in
fact, help address the much broader problem of
reproducibility242424https://www.wired.com/story/machine-learning-
reproducibility-crisis/ and replicability
252525https://cacm.acm.org/magazines/2020/8/246369-threats-of-a-replication-
crisis-in-empirical-computer-science/abstract we face in Computer Science [12,
1] in general and in AI in particular [26].

This chapter is organized as follows: Next, in Section 3.0.14 we briefly
review which kinds of actors may benefit from better education in information
access system experimentation. Afterwards, in Section 3.0.15, we provide
concrete examples of what we can do in terms of concrete resources and
initiatives to increase the awareness and knowledge level of the different
actors. Finally, in Section 3.0.16, we sketch the main challenges that we may
need to be aware of when implementing some of the described educational
initiatives.

####  3.0.14 Actors

As in any process related to the advancement, communication, and sharing of
knowledge, knowing how to properly design and carry out correct and robust
experimentation concerns people with various different roles. This covers a
broad spectrum including academia, industry, and public organizations, e.g.,
from a lecturer in IR and RS introducing evaluation paradigms to undergrad
students and data scientists—not necessarily experienced in IR and RS—choosing
metrics aligned to business Key Performance Indicators by looking at textbooks
and Wikipedia pages. We have identified a number of actors that are involved
in the education to experimentation in information access, who are listed
below. Note that this categorization is not exhaustive nor exclusive, as
actors may have multiple roles.

##### Students

This category embraces the different stages of academic training. Starting
from students enrolled in IR & RS courses [41], including, for instance,
undergraduate students in Computer Science degrees and Master’s students in
Data Science, AI, and Human-Computer Interaction. It also includes students
enrolled in a doctoral degree, i.e., PhD students, including those jointly co-
supervised with industry.

##### Educators

Academic roles related to education, such as course coordinators, lecturers,
teaching assistants, as well as research student supervisors.

##### Scholars

Researchers and academics involved in academic services, including reviewers,
journal editors, program chairs, grant writers, etc.

##### Practitioners

Data scientists, developers, User Experience (UX) designers, and other
practitioners outside academia, that may need support in their lifelong
learning.

##### Decision-makers

People that make strategic decisions in processes, policies, products and/or
human resources (e.g., managers in industry or policy-makers) that may benefit
from having a better understanding of IR and RS core concepts in evaluation
and experimentation.

Figure 6: Interaction among actors involved in IR and RS experimental
education.

Figure 6 shows the interaction among the identified actors. In academia,
students, educators, and scholars are in continuous interaction through
learning, teaching, and supervision processes, which are overseen and/or led
by decision-makers such as deans, heads of departments, etc. In industry,
decision-makers such as product and team managers, as well as practitioners,
make use of training and education resources and initiatives to support
experimentation in real-world domains. The cyclic arrows represent the active
participation in the creation and development of those resources and
initiatives. Decision-makers in public organizations, such as policy-makers,
are also key actors in the definition of curricula, which has a direct impact
on how and to which extent experimentation in IR and RS is included in Data
Science, Computer Science, Computer-Human-Interaction (CHI), and AI programs.

####  3.0.15 What can we do?

In this section, we first provide examples of helpful _resources_ to improve
education in IR and RS evaluation. Then, we outline several possible
_initiatives_ that contribute to increasing awareness about current
methodological issues and to disseminate knowledge about experimentation
approaches.

##### Resources

The resources with which the actors interact are a way to share, maintain, and
promote best practices while ensuring a low barrier of entry to the field.
Given that those resources might be widely used in education, research
(experimentation, etc.), and even production systems, resources have great
potential to continuously grow the knowledge of future generations of
scholars, practitioners, and decision-makers.

General Teaching Material. Textbooks quickly may become outdated,262626In
contrast to that, the main textbook in the area of natural language processing
has for years only been available as an online draft and is continuously being
updated: https://web.stanford.edu/~jurafsky/slp3/ but have the advantage that
these typically reach a wide audience, whereas slides and tutorials that cover
evaluation methodology in more depth might only reach smaller audiences.
Often, today’s online lectures primarily report on ‘mainstream’ information
retrieval (e.g., offline studies, common metrics), but foster reflection and
discussion only to a very limited extent. More comprehensive resources should
be made publicly available and shared across universities, summer schools, and
meetups.272727For instance, Sebastian Hofstätter released Open-Source
Information Retrieval Courses: https://github.com/sebastian-
hofstaetter/teaching. Finally, having the IR and RS community actively
contribute to the curation of material in sources that are widely used by the
general public—and, thus, also by students—as a starting point to get a basic
understanding of a topic (e.g., Wikipedia) is advisable. Further, contributing
to the documentation of software such as Apache
Solr,282828https://solr.apache.org/
Elasticsearch,292929https://www.elastic.co/es/elasticsearch/
Surprise,303030https://surpriselib.com/
Implicit,313131https://implicit.readthedocs.io etc. (see the report by Ferro
et al. [22] for more that are widely used in practice), can help to make non-
experts more aware of the best practices in IR and RS experimentation.

Apart from introducing modern information retrieval systems, teaching material
should give more attention to a wider set of application fields of IR,
including recommender systems and topics related to query and interaction
mining and understanding, and online learning to rank [41]. To date, also
online evaluation falls short in such resources although it is essential in
the spectrum of evaluation types [41]. Students need to be introduced to
concepts such as reproducibility and replicability, and it is essential that
students understand what makes a research work impactful in practice. To lower
the entry barrier to the field, students should be taught how to use available
tools and environments that enable quick prototyping, and that have real-world
relevance. Teaching fairness, privacy, and ethical aspects, both in designing
experiments and also in how to evaluate them, is also important.323232 Cyprus
Center for Algorithmic Transparency (CyCAT) project:
https://sites.google.com/view/biasvisualizationactivity/home

Moreover, the participation in shared tasks (challenges or competitions) of
evaluation campaigns in IR (e.g., TREC,333333https://trec.nist.gov/
CLEF,343434https://www.clef-initiative.eu/
NTCIR,353535https://research.nii.ac.jp/ntcir/ or
FIRE363636https://fire.irsi.res.in/fire/) and RecSys (e.g., the yearly ACM
RecSys challenges373737https://recsys.acm.org/challenges/) should be fostered.
To facilitate the participation of students, it is worthwhile to make the
timelines of such challenges and competitions compatible with the academic
(teaching) schedules (e.g., in terms of semesters). Students will be provided
with the datasets used in the benchmarks and will be able to learn more on
evaluation methodologies (for instance, students from Padua, Leipzig, and
Halle participated in Touché [8, 9] hosted at CLEF). At the same time, it is
important to critically reflect with students on the limitations and dangers
of competitions [11] and encourage them to go beyond leaderboard State Of The
Art (SOTA) chasing culture—e.g., only optimizing on one metric or a limited
set of metrics without reflection of the suitability of these metrics in a
given application context [50, 30]. Hence, it is important that a student’s
(or student group’s) grade does not depend on their rank in the leaderboard
but to a large degree on their approach, reasoning, and reflection to
counteract SOTA chasing and help students to focus on insights. Inspired by
result-blind reviewing in Section References, we might refer to this as
‘result-blind grading’.

Test collections383838In IR, an offline test collection is typically composed
of a set of topics, a document collection, and a set of relevance judgments.
and runs/submissions—typically combined with novel evaluation
methodologies—are the main resources resulting from shared tasks or evaluation
campaigns. Integrating the resulting test collections into tools such as
Hugging Face datasets [34], ir_datasets [38] or EvALL [3] allows for unified
access to a wide range of datasets. Furthermore, some software components such
as Anserini [52], Capreolus [54], PyTerrier [39], OpenNIR [37], etc., can
directly load test collections integrated into ir_datasets which substantially
simplifies data wrangling for scholars of all levels. For instance, PyTerrier
allows for defining end-to-end experiments, including significance tests and
multiple-test correction, using a declarative pipeline and is already used in
research and teaching alike (e.g., in a master course with 240 students [39]).
Other resources for performance modeling and prediction in RS, IR, and NLP can
also be found in the manifesto of a previous Dagstuhl Perspectives Workshop
[22]. The broad availability of such resources makes it tremendously easier to
replicate and reproduce approaches that were submitted to a shared task
(challenge) before. Further, it lowers the entry barrier to experiment with a
wider set of datasets and approaches across domains as switching between
collections will be easy. New test collections can be added with limited
effort. Still, further promoting the practice of sharing code and
documentation,393939https://www.go-fair.org/fair-principles/ or using software
submissions with tools such as TIRA [24, 44] in shared tasks is important.

Combining and integrating the resources listed above in novel ways has the
potential to reduce or even remove barriers between research and education,
ultimately enabling Humboldt’s ideal to combine teaching and research.
Students who participate in shared tasks as part of their curriculum already
go in this direction [18]. Continuously maintaining and promoting the
integration of test collections and up-to-date best practices for shared tasks
into a shared resource might further foster student participants because it
becomes easier to “stand on the shoulders of giants” yielding to the cycle of
education, research, and evaluation that is streamlined by ECIR, CLEF, and
ESSIR (see Section 2.0.4).

##### Initiatives

We have identified a range of actors, and we argue that addressing the
problems around education requires a number of different initiatives some of
which target one particular type of actor but more commonly offer benefits for
different groups. These initiatives should not be seen in isolation as our
vision is in line with what has been proposed in Section 2.0.4 which calls for
coordinated action around education, evaluation, and research. Here we will
discuss instruments we consider to be essential on that path. There is no
particular order in this discussion other than starting with well-established
popular concepts.

Summer schools are a key instrument primarily aimed at graduate students.
ESSIR404040https://www.essir.eu is a prime example of a summer school focusing
on delivering up-to-date educational content in the field of IR; the
Recommender Systems Summer School is organized in a similar manner focusing on
RS. Beyond the technical content, summer schools do also serve the purpose of
community-building involving different actors, namely students and scholars.
Annually organized summer schools appear most effective as they make planning
easier by integrating them into the annual timeline of IR\- and RS-related
events. This is in line with the _flow-wise_ vision discussed earlier in
Section 2.0.4.

Summer schools also provide a good setting to embed (research-focused)
Mentoring programs and Doctoral Consortia. This allows PhD students as well as
early-career researchers to learn from experts in the field outside their own
institutions. Both instruments are well-established in the field. However,
even though the established summer schools are repeatedly organized, these
often happen on an irregular basis (sometimes yearly, sometimes with longer
breaks) and using different formats. This irregular setting makes it difficult
to integrate it into a PhD student’s journey from the outset. Currently,
Mentoring is often merely a by-product of other initiatives such as Summer
Schools and Doctoral Consortia. It may be a fruitful path to see mentoring
programs as an independent (yet, not isolated) initiative. For instance, the “
Women in Music Information Retrieval (WiMIR) Mentoring
program”414141https://wimir.wordpress.com/mentoring-program/ sets an example
of a sustainable initiative that is organized independently of other
initiatives and on yearly basis. A similar format seems a fruitful path to
follow in the IR and RS communities, where it is advisable to facilitate
exchange across (sub-)disciplines and open up the initiative to the entire
community. We note that—similar to the WiMIR—mentoring may not only address
PhD students but is well suited also for later-career stages.

While the IR and RS communities have a tradition of research-topic-driven
Tutorials as part of the main conferences, Courses that address skills and
practices beyond research topics (similar to courses hosted by the CHI
conference424242https://chi2023.acm.org/for-authors/courses/accepted-courses/)
would be an additional fruitful path to follow. Such courses may, for
instance, address specific research and evaluation methods on an operational
level434343See, e.g., CHI 2023’s C12: Empirical Research Methods for Human-
Computer Interaction https://chi2023.acm.org/for-authors/courses/accepted-
courses/#C12, C18: Statistics for CHI https://chi2023.acm.org/for-
authors/courses/accepted-courses/#C18 or how to write better research papers
for a specific outlet or community444444See, e.g., CHI 2021’s C02: How to
Write CHI Papers [42]. With regard to support in writing better papers, see
also Section References. In Bachelor and Master education, more resources in
the form of Formal Educational Materials could be developed. For example,
students could benefit from The Black Mirror Writers’ Room
exercise454545https://discourse.mozilla.org/t/the-black-mirror-writers-
room/46666 which helps convey ethical thinking around the use of technology.
Participants choose current technologies that they find ethically troubling
and speculate about what the next stage of that technology might be. They work
collaboratively as if they were science fiction writers, and use a combination
of creative writing and ethical speculation to consider what protagonist and
plot would be best suited to showcase the potential negative consequences of
this technology. They plot episodes, but then also consider what steps they
might take now (in regulation, technology design, social change) that might
result in _not_ getting to this negative future. More experienced Bachelor
students and Master students could have assessments similar to paper reviews
as part of their curriculum to practice critical thinking.

Topically relevant Meetups ranging from informal one-off meetings to more
regular thematically structured events offer a much more flexible and informal
way to learn about the field. Unlike summer schools they bring together the
community for an evening and cater for a much more diverse audience involving
_all_ actors with speakers as well as attendees from industry, academia and
beyond. Talks range from specific use cases of IR in the industry (e.g.,
search at Bloomberg), to the latest developments in well-established tools
(such as Elasticsearch) to user studies in realistic settings. There is a
growing number of information-retrieval-related and recommender-systems-
related Meetups464646See, e.g., https://opensourceconnections.com/search-
meetups-map/, https://recommender-systems.com/community/meetups/ and many of
which have become more accessible recently as they offer virtual or hybrid
events. Meetups offer a low entry barrier in particular for students at all
levels of education and they help participants obtain a more holistic view of
the challenges of building and evaluating IR and RS applications. Loosely
incorporating Meetups in the curriculum, in particular when there is alignment
with teaching content (e.g., joint seminars), has been demonstrated to be
effective in our own experience. These joint initiatives may go beyond the
dissemination of content, but also involve practitioners as well as decision-
makers in terms of facilitating (or hindering) strategic alliances or setting
strategic themes.

Knowledge Transfer through collaboration between industry and academia is
another instrument offering a mutually beneficial collaboration between three
key actors: PhD students, academic scholars, and practitioners in the
industry. By tackling real-world problems (as defined by the industrial
partner) using state-of-the-art research approaches in the fields of IR and RS
(as provided by the academic partner) knowledge does not just flow in one
direction but both ways. In the context of our discussion, this is an
opportunity to gain insights into evaluation methods and concerns in the
industry. There are well-established frameworks to foster knowledge transfer
such as Knowledge Transfer Partnerships474747http://ktp.innovateuk.org in the
UK with demonstrated impact in
IR484848https://www.gov.uk/government/news/media-tracking-firm-wins-knowledge-
transfer-partnership-2015 and beyond.

Knowledge transfer should also be facilitated and supported at a higher level
at conferences and workshops. This is where the RS community is particularly
successful in attracting industry contributions to the RecSys conference
series. In IR, there is still an observable gap between key academic
conferences such as SIGIR and practitioners’ events like Haystack (_“the
conference for improving search relevance”_ 494949https://haystackconf.com).
The annual Search Solutions conference is an example of a successful forum to
exchange ideas between all different
actors.505050https://www.bcs.org/membership-and-registrations/member-
communities/information-retrieval-specialist-group/conferences-and-
events/search-solutions/

With a view to improving evaluation practices in the long-term, the reviewing
process and practices play an important role. Hence, addressing reviewers and
editors is essential. Reviewers are important actors in shaping what papers
will be published and which not. And it is essential that good evaluation is
acknowledged and understood while poorly evaluated papers are not let through.
Similarly, it is crucial to have reviewers who acknowledge and understand
information retrieval and recommendation problems in their broader context
(e.g., tasks, users, organizational value, user interface, societal impact)
and review papers accordingly. Hence, it is essential to develop educational
initiatives concerning evaluation that address current and future reviewers
(and editors) accordingly. Promising initiatives include the following:

  * •

Clear reviewer guidelines acknowledging the wide spectrum of evaluation
methodology and the holistic view on information retrieval and recommendation
problems. For example, CHI515151ACM CHI Conference on Human Factors in
Computing Systems and Association for Computational Linguistics
(ACL)525252Association for Computational Linguistics provide detailed
descriptions of what needs to be addressed and considered in a review and what
steps to take.535353CHI 2023 Guide to reviewing papers
https://chi2023.acm.org/submission-guides/guide-to-reviewing-papers/; ACL’s
How to Review for ACL Rolling Review
https://aclrollingreview.org/reviewertutorial; Ken Hinckley’s comment on what
excellent reviewing is [28]. Care has to be taken, though, that such
guidelines are kept concise to not overwhelm people before even starting to
read. Further suggestions on results-blind reviewing and guidance for authors
can be found in Sections References and Section References respectively.

  * •

Next to reviewers, meta-reviewers and editors is another entity to address,
which can be done in a similar manner as addressing reviewers. These senior
roles can have strong momentum in inducing change—but have a strong power
position in preventing it. Stronger resistance might be expected on that
(hierarchical) level. Seemingly, only a few conferences and journals—for
instance, ACL545454ACL’s Action Editor Guide to Meta-Reviewing
https://aclrollingreview.org/aetutorial—seem to offer clear guidelines for the
meta-reviewing activity.

  * •

Similar to courses on research methods or addressing paper-writing skills, it
is advisable to provide courses that specifically address how to peer
review.555555https://chi2023.acm.org/for-authors/courses/accepted-courses/#C16

  * •

Mentored reviewing is another promising initiative to have better reviews
that, on the one hand, better assess submitted papers and, on the other hand,
are more constructive to induce better evaluation practices for future
research. Mentored reviewing programs are, for instance, established in
Psychology565656https://www.apa.org/pubs/journals/cpp/reviewer-mentoring-
program. The MIR community575757https://www.ismir.net has a New-to-ISMIR
mentoring program585858https://ismir2022.ismir.net/diversity/mentoring that
mainly addresses paper-writing for people who are new to the community but
will likely also have an impact on reviewing practices. Similar programs could
be established in the IR and RS communities with a particular focus on
evaluation aspects. It is worthwhile to note that a recent study (in ML and
AI) indicates that novice reviewers provide valuable contributions in the
reviewing process [47].

  * •

Summer schools mainly address (advanced) students and are also a good
opportunity to include initiatives addressing reviewing.

General Public Dissemination is another important aspect that needs to be
addressed. Communication in the lay language of our field is very important.
Editing and curating better relevant Wikipedia pages on evaluation measures
for information
retrieval595959https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)
[Accessed: 20-Jan-2023] and recommender
systems606060https://en.wikipedia.org/wiki/Recommender_system#Evaluation
[Accessed: 20-Jan-2023] will increase the potential of reaching a wider
audience, including potential future students. Other actions can concern
publishing papers in magazines with a wider and differentiated audience, such
as _Communications of the ACM_ 616161https://cacm.acm.org/, _ACM Inroads_
626262https://inroads.acm.org/, _ACM XRDS: Crossroads_
636363https://xrds.acm.org/, _IEEE Spectrum_ 646464https://spectrum.ieee.org/.
One of the final goals is to make IR and RS more popular to both attract
students to the field and grow a healthy ecosystem of professionals at various
levels.

We have described actors, resources, and initiatives that we think are worth
considering in moving forward as a community towards creating more awareness,
as well as sharing and transferring knowledge on experimental evaluation for
IR and RS. We summarize the participation (either primary or secondary actors)
in generating and consuming these resources and initiatives in Table 3. This
is not intended as a definitive list but aimed to represent the primary and
secondary actors which are involved.

Table 3: Actors generating or consuming resources and initiatives related to
education in evaluation for IR and RS. ✓and (✓) indicate primary and secondary
actors, respectively.

Actors: Students Educators Scholars Practitioners Decision-makers Resources
Teaching Materials ✓ ✓ (✓) Shared tasks/challenges/competitions ✓ ✓ ✓ ✓ Test
collections & runs/submissions ✓ ✓ ✓ ✓ Software (components) ✓ ✓ ✓ ✓
Initiatives Mentoring: Summer schools and Doctoral Consortia ✓ ✓ (✓) Tutorials
and courses ✓ ✓ ✓ Meetups (✓) (✓) ✓ ✓ ✓ Joint seminars ✓ ✓ ✓ (✓) Collaboration
between industry and academia ✓ ✓ ✓ Reviewing (✓) ✓ General public
dissemination (✓) (✓) ✓ ✓ ✓

####  3.0.16 Challenges & Outlook

Given the importance of reliable and ecologically valid results, one may ask
oneself which obstacles occur in the path of developing better education for
experimentation and evaluation of information access systems. We see different
potential barriers (and possibilities) for the different actors: students,
educators, scholars, practitioners, and decision-makers. We will investigate
each actor in turn.

Scholars. As has also been identified in a previous Dagstuhl seminar [22], it
is significantly harder to test the importance of assumptions in user-facing
aspects of the system, such as the presentation of results or the task model,
as it is prohibitively expensive to simulate arbitrarily many versions of a
system and put them before users. User studies are therefore also at higher
risk of resulting in hypotheses that cannot be clearly rejected (non-
significant results), leading to fear of criticism and rejection from paper
reviewers. There are some proponents of Equivalence Testing [33]656565See also
https://cran.r-project.org/web/packages/TOSTER/TOSTER.pdf and Bayesian
Analysis [49] in Psychology which may also be useful in Computer Science.

As LLMs are becoming a commodity, policies to educate and guide authors and
reviewers in how different AI tools can (or cannot) be used for writing
assistance should be discussed and defined.666666For instance, see the ACL
2023 Policy on AI Writing Assistance:
https://2023.aclweb.org/blog/ACL-2023-policy/. These guidelines may inspire
educators on how to characterize the role of these tools in learning &
teaching environments, including assessment design and plagiarism
policies676767https://www.theatlantic.com/technology/archive/2022/12/chatgpt-
ai-writing-college-student-essays/672371/.

In addition, a current culture of ‘publish or perish’ incentivizes short-term
and incremental findings686868https://harzing.com/resources/publish-or-perish,
over more holistic thinking and thoughtful comparative analysis. The problem
of ‘SOTA-chasing’ has also been discussed in other research areas, e.g., in
NLP [11]. Change in academic incentive systems both within institutions and
for conferences and journals change slowly but they do evolve.

Students and Educators. Thankfully, institutions are increasingly recognizing
the need for reviewing studies before they are performed, such as Ethics and
Data Management plan696969Further proposals for methodological review are also
under discussion in Psychology, but will likely take longer to reach Computer
Science: https://www.nature.com/articles/d41586-022-04504-8. In Bachelor and
Master education, in particular, this means that instructors may require
training in writing such documents, and institutions appreciate and are
equipped for timely review. Therefore, planning of education would benefit
from allowing sufficient time for submission, review, and revision.

In that context, teaching evaluation methodologies may require some colleagues
to retrain, in which case some resistance can be expected. Improving access to
training initiatives and materials at post-graduate level can support
colleagues who are willing but need additional support. Various forms of
informal or even organized exchange between teachers may be a helpful
instrument to grow the competency of educators.

Furthermore, certain evaluation concepts and methodologies cannot be taught
before certain topics are covered in the curriculum. A student in recommender
systems may need to understand the difference between a classification and
regression problem; or the difference between precision and recall (for a
given task and user it may be more important to retrieve accurate results, or
to retrieve a wider range of results) before they can start thinking about the
social implications.

Moreover, some students are prone to satisfice, thinking that “good enough is
good enough”: there are many methodologies available for evaluation, and the
options are difficult to digest in a cost-effective way at entry-
level—highlighting the need for availability of tutorials and low-entry level
materials as indicated earlier in Section 3.0.15. Embedding participation to
shared tasks and competitions (e.g., CLEF labs or TREC tracks) which provide a
common framework for robust experimentation may help overcome this
challenge—although the synchronization between the semester and participation
timelines may not be straightforward.

Finally, there is a growing number of experiments in developing multi-
disciplinary curricula – with the appreciation that different disciplines
bring to such a program. Successful initiatives include group projects
consisting of students in both Social Sciences and Humanities (SSH) and
Computer Science. In fact, one of the underlying principles of the
continuously growing _iSchools consortium_ 707070https://www.ischools.org is
to foster such interdisciplinarity. The challenge here is not only the design
of the content but also accreditation and support from the strategic level of
institutions.

Practitioners. Maintenance of resources used to translate knowledge about
models and methodologies for evaluation is challenging given the fast pace of
the field. This can make it hard to compare results across studies and to keep
up with the SOTA of best practices in experimentation. In this regard lowering
the entry barrier to participating in initiatives such as shared
tasks/challenges [21, 27] and maintaining documentation of resources commonly
used by non-experts are increasingly helpful.

Another issue is the homogeneity of actors. Often there is no active
involvement of actors outside a narrow academic Computer Science sphere, who
otherwise might have indicated assumptions or limitations early on. It can be
challenging to set up productive collaborations between industry and academia,
as well as across disciplines. Typical issues include, for instance, common
terminology used in a different way, or different levels of knowledge of key
performance indicators. Co-design in labs has set a good precedent in this
regard. Examples are ICAI in the Netherlands717171https://icai.ai/, its
extension in the new 10-year ROBUST initiative727272https://icai.ai/ltp-
robust/, and the Australian Centre of Excellence for Automated Decision-Making
and Society (ADM+S)737373https://www.admscentre.org.au/, where PhDs in
multiple disciplines (Social Sciences & Humanities, Computer Science, Law,
etc.) are jointly being trained in shared projects.

Research Advisory Boards are another effective instrument to draw in
practitioners but here the challenge is to make the most of the little time
that is usually available for the exchange of ideas between practitioners and
academics.

Decision-makers. The output of evaluation and experimentation in IR and RS may
be used to inform decision-making on the societal level. Consequently, if the
evaluation is poorly done, or the results incorrectly generalized, the
implications may also be poor decision-making with far-reaching impacts on
society, e.g. [31, Ch. 10].

The ability of the other actors to support education on evaluation is
constrained and shaped by decision-makers. Policy-makers in public
organizations and program managers or deans in academia play a crucial role in
curriculum design. Scholars and educators will have to communicate effectively
the importance of experimental evaluation in information access in order to
inform the decision-making process. The challenge here is to initiate change
in the first place and to drive such changes. Any new initiative will
necessarily involve not just a single decision-maker but more stakeholders and
committees making this a more effortful but possibly also more impactful
process than many of the other initiatives we have identified.

Additionally, decision-makers within academic institutions, namely libraries
and career development centres, can play an important role towards developing
the competency of students and educators. Making best practices in evaluation
available as a commodity through these channels will require making resources
more accessible for non-experts in IR and RS.

####  3.0.17 Concluding Remarks

Education and dissemination represent key pillars to overcoming methodological
challenges in Information Retrieval and Recommender Systems. What we have
sketched here can be interpreted as a general roadmap to create more awareness
among and beyond the IR and RS communities. We hope the recommendations—and
the identified challenges to consider—on what we can do will help to support
education for better evaluation in the different stages of the lifelong
learning journey. We acknowledge that facets such as incentive mechanisms and
processes in institutions are often slow-moving. The vision proposed in this
section is therefore also aimed at a longer-term (5–10 years) perspective.

## References

  * [1] Reproducibility of Data-Oriented Experiments in e-Science (Dagstuhl Seminar 16041), volume 6, 2016.
  * [2] Ahmed Allam, Peter Johannes Schulz, and Kent Nakamoto.  The impact of search engine selection and sorting criteria on vaccination beliefs and attitudes: Two experiments manipulating google output.  Journal of Medical Internet Research, 16(4):e100, 2014.
  * [3] Enrique Amigó, Jorge Carrillo de Albornoz, Mario Almagro-Cádiz, Julio Gonzalo, Javier Rodríguez-Vidal, and Felisa Verdejo.  Evall: Open access evaluation for information access systems.  In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, editors, Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 1301–1304. ACM, 2017.
  * [4] Timothy G. Armstrong, Alistair Moffat, William Webber, and Justin Zobel.  Improvements that don’t add up: ad-hoc retrieval results since 1998.  In David Wai-Lok Cheung, Il-Yeol Song, Wesley W. Chu, Xiaohua Hu, and Jimmy Lin, editors, Proceedings of the 18th ACM Conference on Information and Knowledge Management, CIKM 2009, Hong Kong, China, November 2-6, 2009, pages 601–610. ACM, 2009.
  * [5] Ahmed Hassan Awadallah, Rosie Jones, and Kristina Lisa Klinkner.  Beyond DCG: user behavior as a predictor of a successful search.  In Brian D. Davison, Torsten Suel, Nick Craswell, and Bing Liu, editors, Proceedings of the Third International Conference on Web Search and Web Data Mining, WSDM 2010, New York, NY, USA, February 4-6, 2010, pages 221–230. ACM, 2010.
  * [6] Christine Bauer and Eva Zangerle.  Leveraging multi-method evaluation for multi-stakeholder settings.  In Oren Sar Shalom, Dietmar Jannach, and Ido Guy, editors, Proceedings of the 1st Workshop on the Impact of Recommender Systems co-located with 13th ACM Conference on Recommender Systems, ImpactRS@RecSys 2019), Copenhagen, Denmark, September 19, 2019, volume 2462 of CEUR Workshop Proceedings. CEUR-WS.org, 2019.
  * [7] Jöran Beel and Stefan Langer.  A comparison of offline evaluations, online evaluations, and user studies in the context of research-paper recommender systems.  In Sarantos Kapidakis, Cezary Mazurek, and Marcin Werla, editors, Research and Advanced Technology for Digital Libraries - 19th International Conference on Theory and Practice of Digital Libraries, TPDL 2015, Poznań, Poland, September 14-18, 2015. Proceedings, volume 9316 of Lecture Notes in Computer Science, pages 153–168. Springer, 2015.
  * [8] Alexander Bondarenko, Maik Fröbe, Johannes Kiesel, Shahbaz Syed, Timon Gurcke, Meriem Beloucif, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, and Matthias Hagen.  Overview of touché 2022: Argument retrieval.  In Alberto Barrón-Cedeño, Giovanni Da San Martino, Mirko Degli Esposti, Fabrizio Sebastiani, Craig Macdonald, Gabriella Pasi, Allan Hanbury, Martin Potthast, Guglielmo Faggioli, and Nicola Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction - 13th International Conference of the CLEF Association, CLEF 2022, Bologna, Italy, September 5-8, 2022, Proceedings, volume 13390 of Lecture Notes in Computer Science, pages 311–336. Springer, 2022.
  * [9] Alexander Bondarenko, Lukas Gienapp, Maik Fröbe, Meriem Beloucif, Yamen Ajjour, Alexander Panchenko, Chris Biemann, Benno Stein, Henning Wachsmuth, Martin Potthast, and Matthias Hagen.  Overview of touché 2021: Argument retrieval.  In K. Selçuk Candan, Bogdan Ionescu, Lorraine Goeuriot, Birger Larsen, Henning Müller, Alexis Joly, Maria Maistro, Florina Piroi, Guglielmo Faggioli, and Nicola Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction - 12th International Conference of the CLEF Association, CLEF 2021, Virtual Event, September 21-24, 2021, Proceedings, volume 12880 of Lecture Notes in Computer Science, pages 450–467. Springer, 2021.
  * [10] Ye Chen, Ke Zhou, Yiqun Liu, Min Zhang, and Shaoping Ma.  Meta-evaluation of online and offline web search evaluation metrics.  In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, editors, Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 15–24. ACM, 2017.
  * [11] Kenneth Ward Church and Valia Kordoni.  Emerging trends: Sota-chasing.  Nat. Lang. Eng., 28(2):249–269, 2022.
  * [12] Andy Cockburn, Pierre Dragicevic, Lonni Besançon, and Carl Gutwin.  Threats of a replication crisis in empirical computer science.  Commun. ACM, 63(8):70–79, 2020.
  * [13] Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach.  Are we really making much progress? A worrying analysis of recent neural recommendation approaches.  In Toine Bogers, Alan Said, Peter Brusilovsky, and Domonkos Tikk, editors, Proceedings of the 13th ACM Conference on Recommender Systems, RecSys 2019, Copenhagen, Denmark, September 16-20, 2019, pages 101–109. ACM, 2019.
  * [14] Tim Draws, Nirmal Roy, Oana Inel, Alisa Rieger, Rishav Hada, Mehmet Orcun Yalcin, Benjamin Timmermans, and Nava Tintarev.  Viewpoint diversity in search results.  In ECIR, 2023.
  * [15] Tim Draws, Nava Tintarev, and Ujwal Gadiraju.  Assessing viewpoint diversity in search results using ranking fairness metrics.  SIGKDD Explor., 23(1):50–58, 2021.
  * [16] Tim Draws, Nava Tintarev, Ujwal Gadiraju, Alessandro Bozzon, and Benjamin Timmermans.  This is not what we ordered: Exploring why biased search result rankings affect user attitudes on debated topics.  In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 295–305. ACM, 2021\.
  * [17] Michael D. Ekstrand, Michael Ludwig, Joseph A. Konstan, and John Riedl.  Rethinking the recommender research ecosystem: reproducibility, openness, and lenskit.  In Bamshad Mobasher, Robin D. Burke, Dietmar Jannach, and Gediminas Adomavicius, editors, Proceedings of the 2011 ACM Conference on Recommender Systems, RecSys 2011, Chicago, IL, USA, October 23-27, 2011, pages 133–140. ACM, 2011.
  * [18] Theresa Elstner, Frank Loebe, Yamen Ajjour, Christopher Akiki, Alexander Bondarenko, Maik Fröbe, Lukas Gienapp, Nikolay Kolyada, Janis Mohr, Stephan Sandfuchs, Matti Wiegmann, Jörg Frochte, Nicola Ferro, Sven Hofmann, Benno Stein, Matthias Hagen, and Martin Potthast.  Shared Tasks as Tutorials: A Methodical Approach.  In 37th AAAI Conference on Artificial Intelligence (AAAI 2023). AAAI, 2023.
  * [19] Robert Epstein and Ronald E. Robertson.  The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections.  Proceedings of the National Academy of Sciences, 112(33):E4512–E4521, 2015.
  * [20] Robert Epstein, Ronald E. Robertson, David Lazer, and Christo Wilson.  Suppressing the search engine manipulation effect (SEME).  Proc. ACM Hum. Comput. Interact., 1(CSCW):42:1–42:22, 2017\.
  * [21] Nicola Ferro.  What happened in CLEF \ldots for a while?  In Fabio Crestani, Martin Braschler, Jacques Savoy, Andreas Rauber, Henning Müller, David E. Losada, Gundula Heinatz Bürki, Linda Cappellato, and Nicola Ferro, editors, Experimental IR Meets Multilinguality, Multimodality, and Interaction - 10th International Conference of the CLEF Association, CLEF 2019, Lugano, Switzerland, September 9-12, 2019, Proceedings, volume 11696 of Lecture Notes in Computer Science, pages 3–45. Springer, 2019.
  * [22] Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, and Justin Zobel.  From evaluating to forecasting performance: How to turn information retrieval, natural language processing and recommender systems into predictive sciences (dagstuhl perspectives workshop 17442).  Dagstuhl Manifestos, 7(1):96–139, 2018.
  * [23] Nicola Ferro and Mark Sanderson.  How do you test a test?: A multifaceted examination of significance tests.  In K. Selcuk Candan, Huan Liu, Leman Akoglu, Xin Luna Dong, and Jiliang Tang, editors, WSDM ’22: The Fifteenth ACM International Conference on Web Search and Data Mining, Virtual Event / Tempe, AZ, USA, February 21 - 25, 2022, pages 280–288. ACM, 2022.
  * [24] Maik Fröbe, Matti Wiegmann, Nikolay Kolyada, Bastian Grahm, Theresa Elstner, Frank Loebe, Matthias Hagen, Benno Stein, and Martin Potthast.  Continuous Integration for Reproducible Shared Tasks with TIRA.io.  In Advances in Information Retrieval. 45th European Conference on IR Research (ECIR 2023), Lecture Notes in Computer Science, Berlin Heidelberg New York, 2023. Springer.
  * [25] Carlos Alberto Gomez-Uribe and Neil Hunt.  The netflix recommender system: Algorithms, business value, and innovation.  ACM Trans. Manag. Inf. Syst., 6(4):13:1–13:19, 2016.
  * [26] Odd Erik Gundersen and Sigbjørn Kjensmo.  State of the art: Reproducibility in artificial intelligence.  In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1644–1651. AAAI Press, 2018.
  * [27] D. K. Harman and E. M. Voorhees, editors.  TREC. Experiment and Evaluation in Information Retrieval. MIT Press, Cambridge (MA), USA, 2005.
  * [28] Ken Hinckley.  So you’re a program committee member now: On excellence in reviews and meta-reviews and championing submitted work that has merit, 2016.
  * [29] Dietmar Jannach and Gediminas Adomavicius.  Price and profit awareness in recommender systems.  In Proceedings of the ACM RecSys 2017 Workshop on Value-Aware and Multi-Stakeholder Recommendation, Como, Italy, 2017.
  * [30] Dietmar Jannach and Christine Bauer.  Escaping the mcnamara fallacy: Towards more impactful recommender systems research.  AI Mag., 41(4):79–95, 2020.
  * [31] Daniel Kahneman.  Thinking, fast and slow.  Penguin, 2011.
  * [32] Joseph A. Konstan and Gediminas Adomavicius.  Toward identification and adoption of best practices in algorithmic recommender systems research.  In Alejandro Bellogín, Pablo Castells, Alan Said, and Domonkos Tikk, editors, Proceedings of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, RepSys 2013, Hong Kong, China, October 12, 2013, pages 23–28. ACM, 2013.
  * [33] Daniël Lakens.  Equivalence tests: A practical primer for t tests, correlations, and meta-analyses.  Social psychological and personality science, 8(4):355–362, 2017\.
  * [34] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander M. Rush, and Thomas Wolf.  Datasets: A community library for natural language processing.  In Heike Adel and Shuming Shi, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, EMNLP 2021, Online and Punta Cana, Dominican Republic, 7-11 November, 2021, pages 175–184. Association for Computational Linguistics, 2021\.
  * [35] Jimmy Lin, Daniel Campos, Nick Craswell, Bhaskar Mitra, and Emine Yilmaz.  Significant improvements over the state of the art? A case study of the MS MARCO document ranking leaderboard.  In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2283–2287. ACM, 2021\.
  * [36] Marianne Lykke, Ann Bygholm, Louise Bak Søndergaard, and Katriina Byström.  The role of historical and contextual knowledge in enterprise search.  J. Documentation, 78(5):1053–1074, 2022.
  * [37] Sean MacAvaney.  Opennir: A complete neural ad-hoc ranking pipeline.  In James Caverlee, Xia (Ben) Hu, Mounia Lalmas, and Wei Wang, editors, WSDM ’20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020, pages 845–848. ACM, 2020.
  * [38] Sean MacAvaney, Andrew Yates, Sergey Feldman, Doug Downey, Arman Cohan, and Nazli Goharian.  Simplified data wrangling with ir_datasets.  In Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, Rosie Jones, and Tetsuya Sakai, editors, SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2429–2436. ACM, 2021\.
  * [39] Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis.  Pyterrier: Declarative experimentation in python from BM25 to dense retrieval.  In Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong, editors, CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021, pages 4526–4533. ACM, 2021.
  * [40] Jiaxin Mao, Yiqun Liu, Ke Zhou, Jian-Yun Nie, Jingtao Song, Min Zhang, Shaoping Ma, Jiashen Sun, and Hengliang Luo.  When does relevance mean usefulness and user satisfaction in web search?  In Raffaele Perego, Fabrizio Sebastiani, Javed A. Aslam, Ian Ruthven, and Justin Zobel, editors, Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval, SIGIR 2016, Pisa, Italy, July 17-21, 2016, pages 463–472. ACM, 2016.
  * [41] Ilya Markov and Maarten de Rijke.  What should we teach in information retrieval?  SIGIR Forum, 52(2):19–39, 2018.
  * [42] Lennart E. Nacke.  How to write CHI papers, online edition.  In Yoshifumi Kitamura, Aaron Quigley, Katherine Isbister, and Takeo Igarashi, editors, CHI ’21: CHI Conference on Human Factors in Computing Systems, Virtual Event / Yokohama Japan, May 8-13, 2021, Extended Abstracts, pages 126:1–126:3. ACM, 2021.
  * [43] Frances A. Pogacar, Amira Ghenai, Mark D. Smucker, and Charles L. A. Clarke.  The positive and negative influence of search results on people’s decisions about the efficacy of medical treatments.  In Jaap Kamps, Evangelos Kanoulas, Maarten de Rijke, Hui Fang, and Emine Yilmaz, editors, Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR 2017, Amsterdam, The Netherlands, October 1-4, 2017, pages 209–216. ACM, 2017.
  * [44] Martin Potthast, Tim Gollub, Matti Wiegmann, and Benno Stein.  TIRA integrated research architecture.  In Nicola Ferro and Carol Peters, editors, Information Retrieval Evaluation in a Changing World - Lessons Learned from 20 Years of CLEF, volume 41 of The Information Retrieval Series, pages 123–160. Springer, 2019.
  * [45] Tetsuya Sakai.  Laboratory experiments in information retrieval - sample sizes, effect sizes, and statistical power.  40, 2018.
  * [46] Mark Sanderson, Monica Lestari Paramita, Paul D. Clough, and Evangelos Kanoulas.  Do user preferences and evaluation measures line up?  In Fabio Crestani, Stéphane Marchand-Maillet, Hsin-Hsi Chen, Efthimis N. Efthimiadis, and Jacques Savoy, editors, Proceeding of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2010, Geneva, Switzerland, July 19-23, 2010, pages 555–562. ACM, 2010.
  * [47] Ivan Stelmakh, Nihar B. Shah, Aarti Singh, and Hal Daumé III.  Prior and prejudice: The novice reviewers’ bias against resubmissions in conference peer review.  CoRR, abs/2011.14646, 2020.
  * [48] Zhu Sun, Di Yu, Hui Fang, Jie Yang, Xinghua Qu, Jie Zhang, and Cong Geng.  Are we evaluating rigorously? benchmarking recommendation for reproducible evaluation and fair comparison.  In Rodrygo L. T. Santos, Leandro Balby Marinho, Elizabeth M. Daly, Li Chen, Kim Falk, Noam Koenigstein, and Edleno Silva de Moura, editors, RecSys 2020: Fourteenth ACM Conference on Recommender Systems, Virtual Event, Brazil, September 22-26, 2020, pages 23–32. ACM, 2020.
  * [49] Johnny van Doorn, Don van den Bergh, Udo Böhm, Fabian Dablander, Koen Derks, Tim Draws, Alexander Etz, Nathan J Evans, Quentin F Gronau, Julia M Haaf, et al.  The jasp guidelines for conducting and reporting a bayesian analysis.  Psychonomic Bulletin & Review, 28(3):813–826, 2021.
  * [50] Ellen M. Voorhees.  Coopetition in IR research.  In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, page 3. ACM, 2020.
  * [51] Ryen W. White.  Interactions with Search Systems.  Cambridge University Press, 2016.
  * [52] Peilin Yang, Hui Fang, and Jimmy Lin.  Anserini: Enabling the use of lucene for information retrieval research.  In Noriko Kando, Tetsuya Sakai, Hideo Joho, Hang Li, Arjen P. de Vries, and Ryen W. White, editors, Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017, pages 1253–1256. ACM, 2017.
  * [53] Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin.  Critically examining the "neural hype": Weak baselines and the additivity of effectiveness gains from neural ranking models.  In Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer, editors, Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1129–1132. ACM, 2019.
  * [54] Andrew Yates, Siddhant Arora, Xinyu Zhang, Wei Yang, Kevin Martin Jose, and Jimmy Lin.  Capreolus: A toolkit for end-to-end neural ad hoc retrieval.  In James Caverlee, Xia (Ben) Hu, Mounia Lalmas, and Wei Wang, editors, WSDM ’20: The Thirteenth ACM International Conference on Web Search and Data Mining, Houston, TX, USA, February 3-7, 2020, pages 861–864. ACM, 2020.
  * [55] Eva Zangerle and Christine Bauer.  Evaluating recommender systems: Survey and framework.  ACM Comput. Surv., 55(8):170:1–170:38, 2023.
  * [56] Fan Zhang, Jiaxin Mao, Yiqun Liu, Xiaohui Xie, Weizhi Ma, Min Zhang, and Shaoping Ma.  Models versus satisfaction: Towards a better understanding of evaluation metrics.  In Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu, editors, Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 379–388. ACM, 2020.
  * [57] J. Zobel.  When measurement misleads: The limits of batch assessment of retrieval systems.  SIGIR Forum, 56(1), 2022.

Results-blind Reviewing [Joeran Beel, Timo Breuer, Anita Crescenzi, Norbert
Fuhr, Meijie Li]Joeran Beel (University of Siegen, DE, joeran.beel@uni-
siegen.de)
Timo Breuer (Technische Hochschule Köln, DE<EMAIL_ADDRESS>
Anita Crescenzi (University of North Carolina at Chapel Hill, US,
<EMAIL_ADDRESS>
Norbert Fuhr (University of Duisburg-Essen, DE<EMAIL_ADDRESS>
Meijie Li (University of Duisburg-Essen, DE<EMAIL_ADDRESS>
####  3.0.18 Motivation

Campbell and Stanley defined experiments as “that portion of research in which
variables are manipulated and their effects upon other variables observed” (p.
1 in [1]).” Scientific experiments are used in confirmatory research to test a
priori hypotheses as well as in exploratory research to gain new insights and
help to generate hypotheses for future research [7]. In information access
research, the ultimate goal is to gain insights into cause and effect.
Unfortunately, many reviewers of information access experiments place undue
emphasis on performance, rejecting papers that contain insights if they fail
to show improvements in performance. The focus on performance numbers not only
leads to publication bias. It also puts additional pressure on early-career
researchers who must publish or perish, thus being tempted to cheat if their
proposed method does not yield the desired results. Moreover, reviewers pay
little attention to the experimental methodology and analysis [4] in case the
results are impressive. Focusing primarily on performance (and in particular
aggregated performance) can lead to a neglect of insights; gaining insights is
critical to move the information access field forward and essential to be able
to make performance predictions [2].

We think that one important step to change the situation is if we alter the
review process such that there is more emphasis on the theoretical background,
the hypotheses, the methodological plan and the analysis plan of an
experiment, while improvement or decline of performance should play less of a
role when deciding about the quality of a paper. It is hoped that this will
lead to a higher scientific quality of publications, more insights, and
improved reproducibility (as there is less incentive for beautifying results).
As Woznyj et al. [8] note in their survey of editorial board members, overall
there are positive attitudes towards results-blind reviewing and advantages
for the scientific community outweigh concerns.

In order to move the review focus away from performance improvement, appealing
to reviewers alone will not be sufficient. A more drastic measure is the
change of the review process such that reviewers decide about acceptance vs.
rejection of a paper without knowing the outcome of the experiments described.

####  3.0.19 Current Situation and Gaps

As part of IR or RS conferences, the peer-reviewing process usually involves
the review of the full paper using double-blinded reviewing, i.e., both
authors and reviewers remain anonymous to each other. Before submission,
authors are informed about possible reviewing criteria and areas of interest
in the Call for Papers (CfP) that can be found on the conference website. Upon
submission, the paper should contain all of the relevant information regarding
the motivation, the research methodology or study design, the experimental
results, and finally, a discussion that puts the results into context.

For each submission, usually, a group of three reviewers is assigned. All of
them should align their reviews to those criteria mentioned in the CfP and,
depending on the submission system, express their opinion in written text or
by pre-defined answers regarding particular aspects. In addition, they can
assign (overall) scores. The final decision is based on a discussion among
reviewers, which is governed by an additional meta-reviewer, and consolidation
with the program chairs.

Even though this traditional review model has been established for several
years, it can imply negative impacts on the stakeholders or the scientific
community as a whole. Under the assumption that reviewers overemphasize
positive outcomes, the authors might be inclined to “search for” performance
gains in system-oriented experiments at the cost of scientific rigor and
reasoning. Even more, there is the danger of fraud or selecting positive
outcomes, considering the need to publish in order to proceed in an academic
career.

Table 4: Comparison of traditional and emerging approaches to peer review: results-blind, preregistered reports, and registered reports. |  Traditional |  Results-Blind |  Preregistered |  Registered Report
---|---|---|---|---
protocol preregistration |  optional |  optional |  yes (in journal repository) |  no
protocol publication (separate from research article) |  no |  no |  no |  yes
peer review of research protocol before data collection |  no |  no |  yes |  yes
peer review of paper with blinded results |  no |  yes |  no |  no
peer review of full paper |  yes |  yes (if in-principle acceptance) |  yes with focus on results (if in-principle acceptance) |  yes (if in-principle acceptance)
Example publication(s) |  ACM SIGIR, ACM CHIIR |  BMC Psychology |  PLOS Biology |  PLOS ONE

Alternatives to the traditional review process have emerged with an initial
round of peer review of a manuscript with the results blinded or a study
protocol and a subsequent round of peer review of the full paper including
results. Table 4 shows the traditional peer review model with our recommended
results-blind reviewing and two other variants, each of which we describe
below. The Center for Open Science notes that, as of January 2023, over 300
journals have adopted one or more variants of this
approach.747474https://www.cos.io/initiatives/registered-reports In addition,
several preliminary analyses of their implementation have been conducted and
published (e.g., [3, 5, 8]).

A results-blind review involves an in-principle acceptance or rejection
decision based on peer review of the paper _with the results blinded_ from the
reviewers (see the third column of Table 4). The reviewers can put more
emphasis on judging the merits of the general motivation, the study design,
and what kinds of scientific insights could be gained from the experiments. If
the paper is accepted in-principle, it proceeds to a second stage of peer
review of the _paper with the results_ included for reviewers. The final
decision about the acceptance is based on the second stage of the review in
which the reviewers have access to the experimental outcomes.

Other peer-reviewing models have emerged in recent years as part of the
growing awareness of
preregistration757575https://www.cos.io/initiatives/prereg,767676https://plos.org/open-
science/preregistration/ and its adoption [6]. One such approach to peer
review involves the review and in-principle acceptance of the study protocol
including the methods and analysis plan before data is collected or analysis
begins. Variants of this approach include preregistered research articles and
registered reports for confirmatory research 777777For examples of how
preregistered research articles and research reports have been implemented,
see the summary provided by PLOS. https://plos.org/open-
science/preregistration/. Although preregistered reports and registered
reports are typically used for confirmatory research, there are variants for
exploratory research and some journals also use a separate approach for
exploratory research projects which do not have a confirmatory component
(e.g., an Exploratory Report article type in journal _Cortex_).

Preregistered research articles involve researchers submitting a research
study protocol including the rationale and hypotheses, methodology including
analysis plan, and materials to a journal for review and simultaneous
depositing into a repository often associated with the journal (see the fourth
column of Table 4). The preregistered protocol is peer-reviewed with a focus
on methods and the analytic approach, and a provisional in-principle
acceptance conditional upon the execution of the study as designed. The
researchers execute the study, analyze the results, and submit a full
manuscript. After peer review of the new sections, the completed manuscript is
published.

Registered Reports also involve submission and peer review of a study protocol
(see the third column of Table 4). A key difference from preregistered
articles is that accepted protocols are published immediately and a future
article with the results of the study is given an in-principle acceptance.
After the study execution, the full manuscript is submitted and reviewed.

####  3.0.20 Next Steps

We propose several changes to the reviewing processes for information access
papers to reduce publication biases. Our recommendations are that information
access scholarly community:

  1. 1.

adopts a pilot test of results-blind reviewing for a conference or journal,

  2. 2.

considers starting from our initial process recommendation for results-blind
reviewing,

  3. 3.

ask authors, conference organizers, and reviewers to place more emphasis
within papers on the insights that can be gained from their research,

  4. 4.

considers allowing additional space for additional details about study
methodology, and

  5. 5.

considers whether to implement a two-stage review process in which research
proposals and/or preregistered research reports are reviewed with a tentative
acceptance decision before data collection and analysis are conducted.

Each of these is described in more detail below.

##### Recommendation 1: Pilot test of results-blind reviewing in conference(s)
or journal(s)

Our first and most important recommendation is that the information access
research communities (i.e., IR and RS communities) adopt a results-blind
approach to peer reviewing for conference(s) and/or journal(s). We recommend
that the community start with a pilot test of results-blind reviewing in an
established conference track, perhaps with a new paper track with an earlier
deadline to allow for a two-stage review process. In results-blind reviewing,
the authors submit two versions of their manuscript: one version of the paper
with the full results, and one version with the results blinded. The two
submitted versions are the basis of a results-blind reviewing process with two
major stages (see Figure 79).

Stage 1 consists of the Results-Blind Review. The results-blind version of the
manuscript is reviewed and an in-principle acceptance (or rejection) is made.
During Stage 1, as in the traditional reviewing process, the paper is reviewed
by multiple reviewers who also make acceptance recommendations. In the case of
conferences, the in-principle acceptance (or rejection) decision is made after
discussion with the Senior Program Committee (SPC)/meta reviewer and in the
Program Committee (PC) meeting. Papers that receive an in-principle acceptance
proceed to Stage 2.

Stage 2 consists of the Results Review. The paper containing the results is
reviewed by the same set of reviewers with a focus on the results. In the case
of a conference, the final acceptance (or rejection) decision is made after a
discussion period with the SPC and in the PC meeting.

Figure 7: Proposed two-stage process for results-blind reviewing (figure
adapted from BMC797979https://www.biomedcentral.com/collections/RFPR)

##### Recommendation 2: Initial process recommendation for a results-blind
reviewing pilot

Below, we recommend a high-level process for how a results-blind reviewing
process pilot might be implemented and important considerations for conference
organizers and reviewers as well as authors.

###### Conference organizers

Once the decision for results-blind reviewing has been made, conference
organizers would have to take the following steps:

  * •

First, the CfP for the new track should be written. As the proposed results-
blind reviewing process with two stages of review will take longer to
complete, an earlier deadline for this track should be set.

  * •

Criteria for both stages of the review (blinded and with results) should be
defined. Special attention should be given to the criteria for changing an
initial acceptance recommendation into a rejection.

  * •

Author instructions for the results-blind reviewing track have to be
formulated, describing not only the new reviewing criteria and process but
also specific instructions on how to prepare the blinded version of an
article. For the results-blind version of the paper, the authors will need to
blind all mentions of the results (e.g., in the abstract, introduction,
discussion, and conclusion in addition to in a results section) in a way that
it is not technically possible to recover the blinded text. There should be a
way for reviewers to easily determine the differences between the results-
blind version of the paper and the one with the results.

  * •

Reviewers for the results-blind reviewing track have to be recruited. In the
beginning, additional or different expertise will be required for this track.
A special introduction of training for the reviewers might be necessary in
order to make them familiar with the new process and criteria.

  * •

The reviewing software will need to be configured for multiple stages of
review for the results-blind reviewing. In the first stage of reviewing, only
the blinded version of the papers should be distributed to reviewers (see
below for the process for reviewers).

  * •

After the final decision by the PC, the authors will be provided with the
review and informed about the final accept or reject decision. In the case of
a rejection decision, authors should also be notified at which stage the paper
was rejected.

  * •

The organizers should give special recognition to the PC member of the track
(on the conference Web site and in the proceedings)

  * •

The success of the new track and the process should be evaluated.

###### Reviewers

Once the reviewers are provided with instructions about the general process
and received additional training, we recommend the following process:

  * •

In the first stage, the reviewers are provided with the results-blind version
of the submission and complete their review including a recommendation about
the in-principle acceptance.

  * •

Once the reviews are complete, a discussion phase with the SPC follows,
leading to a recommendation for each paper.

  * •

The PC for the track meets and makes an initial decision (in-principle
acceptance or rejection) for each paper.

  * •

For the second reviewing stage, only in-principle accepted papers are
considered. Reviewers get the full versions of the papers they reviewed
before. They add an additional part to their review focusing on the results
which were previously blinded. Also, they make a second recommendation about
acceptance.

  * •

As for the first phase, a discussion phase with the SPC follows leading to a
recommendation for each paper.

  * •

The track PC meets for the second time and makes the final decision for each
paper.

###### Authors

Authors will have to understand the new reviewing scheme, and possibly be
trained/educated for preparing manuscripts that satisfy the new reviewing
criteria. They will have to prepare and submit two versions of a paper, a
version with the results as in the traditional model as well as one in which
the results are blinded.

##### Recommendation 3: Emphasize insights in papers

We recommend that authors, conference organizers, and reviewers place
additional emphasis on communicating expected insights to be gained from
experiments. Guidelines (and review forms) should ask the reviewers to comment
on the theoretical background, the hypotheses, the methodological plan and the
analysis plan of the experiment(s) described. Special attention should be
given to the expected insights to be gained from experiments, i.e. regarding
cause and effect.

##### Recommendation 4: Extra space for methods information

Another recommendation is for the community to consider explicitly allowing
methodological appendices for authors to provide additional methodological
details outside of page and/or word limits and to include these appendices
with the text of the paper and not as supplementary materials. While not
needed for all publications, this would be very beneficial for some types of
studies so that the authors can include all study materials. For example, in
user studies, researchers may administer multiple questionnaires, conduct a
semi-structured interview, and read from a script. It is not uncommon for
researchers to administer multiple questionnaires and conduct a semi-
structured interview.

This would be especially important if adopting a results-blind reviewing
process as careful scrutiny of the study design and all study materials is
needed to ascertain whether the authors will be able to answer the research
questions. For example, due to page limits, it is common for authors to
describe the topics of an interview but uncommon to include the full text of
an interview guide due to page limits.

In addition, this would have an additional benefit for other researchers who
wish to replicate the study. While, for example, authors can currently make
supplementary materials available in ACM Digital Library (ACM DL), these
materials are not included in the downloadable version of the article or when
reading online in the ACM DL in the eReader or HTML formats.

##### Recommendation 5: Consider a two-stage review process adapted from
preregistered or registered reports

Although our primary recommendation is for conference organizers or journal
editors to embrace a results-blind reviewing approach, we also recommend that
they consider piloting a conference track or article type in which the study
protocol undergoes peer review and is accepted in-principle before data
collection or analysis begins. This may be more appropriate for certain types
of research (e.g., user studies).

####  3.0.21 Conclusion

At first glance, the new result-blind reviewing scheme might seem to be only
attractive for papers describing failed experiments, while authors with
successful results would go to the established tracks. In order to avoid this
impression, it is essential that the new scheme is piloted as a highly visible
and prestigious track in an established conference. Furthermore, it should be
clearly communicated that the results-blind reviewing scheme aims at
establishing high standards for the design, execution and analysis of
experiments while shielding the reviewers from being blinded by shiny
experimental results. Thus, it is our hope that papers published in this track
will be regarded as high-quality publications which thoroughly address
research questions and clearly demonstrate the insights that may be gained
from the research.

## References

  * [1] Donald T. Campbell and Julian C. Stanley.  Experimental and quasi-experimental designs for research.  Houghton Mifflin Company, Boston, 1963.
  * [2] Nicola Ferro, Norbert Fuhr, Gregory Grefenstette, Joseph A. Konstan, Pablo Castells, Elizabeth M. Daly, Thierry Declerck, Michael D. Ekstrand, Werner Geyer, Julio Gonzalo, Tsvi Kuflik, Krister Lindén, Bernardo Magnini, Jian-Yun Nie, Raffaele Perego, Bracha Shapira, Ian Soboroff, Nava Tintarev, Karin Verspoor, Martijn C. Willemsen, and Justin Zobel.  From evaluating to forecasting performance: How to turn information retrieval, natural language processing and recommender systems into predictive sciences (dagstuhl perspectives workshop 17442).  Dagstuhl Manifestos, 7(1):96–139, 2018.
  * [3] Michael G. Findley, Nathan M. Jensen, Edmund J. Malesky, and Thomas B. Pepinsky.  Can Results-Free Review Reduce Publication Bias? The Results and Implications of a Pilot Study.  Comparative Political Studies, 49(13):1667–1703, 2016.  Publisher: SAGE Publications Inc.
  * [4] Norbert Fuhr.  Some common mistakes in IR evaluation, and how they can be avoided.  SIGIR Forum, 51(3):32–41, 2017.
  * [5] Daniel M. Maggin, Rachel E. Robertson, and Bryan G. Cook.  Introduction to the special series on results-blind peer review: An experimental analysis on editorial recommendations and manuscript evaluations.  Behavioral Disorders, 45(4):195–206, 2020.
  * [6] Brian A. Nosek, Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor.  The preregistration revolution.  Proceedings of the National Academy of Sciences, 115(11):2600–2606, 2018.
  * [7] William R Shadish, Thomas D Cook, and Donald T. Campbell.  Experimental and quasi-experimental designs for generalized causal inference.  Houghton, Mifflin and Company, New York, 2002.
  * [8] Haley M. Woznyj, Kelcie Grenier, Roxanne Ross, George C. Banks, and Steven G. Rogelberg.  Results-blind review: a masked crusader for science.  European Journal of Work and Organizational Psychology, 27(5):561–576, 2018.

Guidance for Authors [Giorgio Maria Di Nunzio, Maria Maistro, Christin
Seifert, Julián Urbano, Justin Zobel]Giorgio Maria Di Nunzio (University of
Padova, IT<EMAIL_ADDRESS>
Maria Maistro (University of Copenhagen, DK<EMAIL_ADDRESS>
Christin Seifert (University of Duisburg-Essen, DE, christin.seifert@uni-
due.de)
Julián Urbano (Delft University of Technology, NL<EMAIL_ADDRESS>
Justin Zobel (University of Melbourne, AU<EMAIL_ADDRESS>
####  3.0.22 Motivation

The IR community has over time developed a strong shared culture of
expectations of published papers, particularly in our leading venues. However,
these expectations are not explicit and the evidence of submitted papers is
that many authors are not aware of what elements, or omissions, are likely to
be of concern to reviewers. While accepted papers do provide an indication of
what an author should do, they are, of course, uneven, and the small set of
papers that an author is consulting in their new work could easily be
unrepresentative of the best IR work as a whole.

In this section, our aim is to provide a basis for general guidance for
authors and reviewers, with a focus on people who are new to the community. It
should communicate to authors and reviewers a range of factors that the
community regards as significant. Such guidance, if well designed, should help
authors to lift the standard of their work and provide context should it not
be accepted; for reviewers, especially those new to the task, it can provide
checklists and (at a high level) advice about the field from beyond their
immediate research environment.

Some elements in papers have attracted specific criticism in publications;
this is particularly true of effectiveness measurement, where a long history
of research on method has argued for and against a range of measures, forms of
evidence for statistical validity, treatment of test collections, and so on.
Such literature is critical to improving the quality of our research but does
not necessarily represent a settled, shared view of best practice.

In our view, it is essential that general advice be constructive, readily
understandable by new IR authors and reviewers, and—to the extent that is
possible—not the subject of active debate. In the following, we have sought to
follow this principle. We first explain the basis of the draft guidance for
authors that we have developed and then present that guidance. How this work
might develop over time is considered under “next steps”.

####  3.0.23 Flaws in Submitted IR Papers

For our goal of developing draft guidelines for authors for the community, we
have multiple sources of inspiration. As a first step, it is valuable to
understand and list the kinds of issues that lead experienced reviewers to
criticize papers, that is, to collect the opinions from the community based on
their experience from different roles as scientists: authors, readers,
reviewers and meta-reviewers. Another valuable source of information consists
in existing guidelines in adjacent research fields, as they reflect a common
agreement of what constitutes a good scientific paper in that community and
point out commonly agreed issues that may lead to rejection.

By collecting, consolidating, and harmonising the collected information, we
aim to establish a strong foundation for the synthesis of a new set of draft
guidelines that comprehensively capture the community-agreed strengths aspects
of good scientific papers as well as issues that commonly lead to rejection;
and separately to identify significant emerging aspects that are not yet
captured by existing guidelines.808080As an example, ACL 2023 includes a
“Policy on AI Writing Assistance” in their call for
papershttps://2023.aclweb.org/blog/ACL-2023-policy/. To obtain concise,
comprehensive, understandable, and actionable guidelines for early-career
researchers, we translated the identified issues, points of criticism, and
guideline items, which have been described at varying levels of detail, into
observations on elements that papers should include and on elements that can
lead to rejection.

We designed the following approach to create our guidelines: (1) search of
existing guidelines; (2) brainstorming to identify common pitfalls; (3)
categorization of the outcomes from the brainstorming exercise and comparison
of these with existing guidelines; and (4) consolidation and integration with
existing SIGIR guidelines.818181https://sigir.org/sigir2023/submit/call-for-
full-papers/checklist-to-strengthen-an-ir-paper/ Throughout each step of the
process, we adhere to the principle of keeping only issues that we believe to
be widely agreed upon within the community.

We now describe our approach.

##### Identifying existing guidelines

We started by searching for existing guidelines for authors and reviewers that
have been proposed in adjacent research communities. In our search for
existing guidelines, we considered the following sources.

  * •

The ACM Special Interest Group on Information Retrieval (SIGIR) developed
recommendations to strengthen IR papers. These are rather general suggestions
concerning presentation and experimentation. We used them as the initial stage
and extend them to design our list of recommendations for authors (see Section
3.0.24).

  * •

Empirical Evaluation Guidelines from the ACM Special Interest Group on
Programming Languages
(SIGPLAN).828282https://www.sigplan.org/Resources/EmpiricalEvaluation/ This is
a checklist that presents best practices meant to support both authors and
reviewers within the community. The checklist includes some broad categories
(e.g., appropriate presentation of results) and examples of violations for
each subcategory (e.g., a misleading summary of results). These are reported
in Appendix 5.1.

  * •

The Special Interest Group on CHI SIGCHI838383https://chi2022.acm.org/for-
authors/presenting/papers/guide-to-reviewing-papers/ published a guide for
reviewing papers submitted to the CHI conference. This is a general overview
of both quality considerations (e.g., whether the paper contribution is
sufficiently original), and more practical considerations related to the paper
length and the review process. SIGCHI also suggested the Equitable Reviewing
Guide,848484https://chi2022.acm.org/for-authors/presenting/papers/equitable-
reviewing-guide/ which is a list of recommendations to help reviewers write
fair reviews. Some of their points include reflecting on personal bias or
considering that many authors are not native English speakers, thus being
lenient on writing style and typos.

  * •

The ACL presented an online tutorial to instruct reviewers on the ACL Rolling
Review process.858585https://aclrollingreview.org/reviewertutorial This
tutorial presents some practical suggestions (e.g., planning the reading and
reviewing time to avoid rushed reviews), as well as suggestions to evaluate
the quality of the paper and a list of common reasons for rejection, which
often lead to author complaints because such reasons are not actual weaknesses
but rather easy, unreasonable grounds for rejection.

  * •

Ulmer et al. [1] present a list of best practices and guidelines for
experimental standards within NLP. These guidelines contain some broad
categories, (e.g., data), and minimal requirements and recommendations for
each category (e.g., publish the dataset accessibly and indicate changes).
These are reported in Appendix 5.2.

##### Brainstorming to identify common issues

After our search for guidelines, we ran a brainstorming exercise among
contributors of the working group. The goal of this exercise was to identify
concerns and flaws that we, as reviewers, would not want to find in IR papers
and can very likely lead to rejection. This list of reflections is included in
Appendix 5.3.

We extended the brainstorming exercise to all participants in the Dagstuhl
seminar through an online survey. We asked participants to list “things we
don’t like to see in papers”, and provided some examples for guidance and the
full list of SIGPLAN categories for inspiration. We received $35$ items.
Comments concerning strategic issues, such as “I prefer to have a new paper
category” were omitted from further analysis; others were integrated into our
findings. As mentioned above, we adhere to the principle of keeping only
issues that we did not regard as controversial issues or the subject of
debate, with the aim of omitting points that might lead to disagreement in the
community.

##### Integration and categorization

Inspired by the SIGPLAN and NLP guidelines, we developed an initial set of
broad categories to organize the issues we identified above. We then mapped
each item in our list of reflections to the corresponding category. We did the
same for the suggestions collected from the participant survey, as well as for
the pertinent points identified in the SIGPLAN and NLP guidelines and the
SIGIR guidance. In this process, we focused on issues that specifically relate
to IR papers and set aside more general issues such as “captions of tables
should be clear”.

There were several rounds of review to clarify and consolidate similar items,
with minor re-categorizations when needed. The final result of this process is
a list of what we believe are recognised as common flaws in IR papers. The
final list consists of $57$ items organized in the following $9$ categories
(see Appendix 5.4): (1) Design, motivation and hypothesis; (2) Literature; (3)
Model and method; (4) Data, data gathering and datasets; (5) Metrics; (6)
Experiments; (7) Analysis of results and presentation; (8) Repeatability,
reproducibility, and replicability; and (9) Conclusions and claims.

Finally, we used this list of concerns to propose an update to the existing
SIGIR guidelines. This is described in the next section.

####  3.0.24 Draft Guidance for Authors

Some years ago, SIGIR introduced brief guidance for authors as “Things that
strengthen an IR paper”.868686The earliest mention we are aware of is from
SIGIR 2021, https://sigir.org/sigir2021/checklist-to-strengthen-an-ir-paper/.
One of us (Zobel) recently updated this guidance for SIGIR-AP’23, in
consultation with the other Program Chairs, but we note that it represented
the views of just a couple of individuals. The SIGIR guidance proposed, at a
high level, aspects to consider in presentation and experiments. The SIGIR-AP
revision primarily addressed some aspects—omissions, oversights, and
shortcomings—that are offered as grounds for rejection.

Here, we took the SIGIR-AP draft guidance as a starting point and reviewed it
against the list of concerns that we set out in Section 3.0.22. We also took
note of generic writing advice that is widely available and decided to omit
elements that we regarded as pertinent to computer science research in
general. This led to the following, which we propose as a basis for the advice
provided by venues that publish IR work.

We have sought to make the advice broad, understandable, and constructive; but
it is of necessity brief and some readers may seek more detail. For that
reason, when the advice (or a revision of it) is used, it might also be
helpful to link to a version of the lists of concerns in Appendix 5.4.

Our proposed draft guidance is as follows.

##### Motivation and claims

• The problem is well characterised and motivated, and the potential impact is
discussed. • The proposed application of the work is contextualised by
pertinent knowledge from that domain, including potential ethical, social, or
environmental impacts. • The research goals and original contributions (that
is, the elements that are a contrast to the prior art) are stated and are
clearly distinguished from prior work. • The claims are properly scoped and
supported. • There are explicit statements of what was done and what was not.

##### Presentation

• The literature review considers competitive previous solutions for the
problem, that is, it is not limited to consideration of other work on the same
technology as that explored in the submission. • There is a reasoned
justification for each of the choices made in each step of the research and
each element of the method. • Results are presented in keeping with the norms
in the field as exemplified in strong prior work. • A substantive, focused,
and insightful discussion accompanies the results taking into account
limitations and scope of the work.

##### Experiments

• The experimental design and its scale are appropriate to the problem. • In
comparative studies, appropriate baselines are used; they are deployed and
optimized in ways comparable to those used for the proposed method. • The
experimental results are reliable and generalizable, and preferably show
illustrative individual cases as well as aggregated results. • Where
appropriate, a diversity of data sets are used, including public-domain data
sets used in prior work. • Sufficient details (with data and code where
appropriate) are provided to enable other researchers to assess and reproduce
the experiments; this includes the nature, source, and collection process for
the data, and the data preparation steps.

##### Results and analysis

• The evaluation methods and measures address the research questions; the use
of redundant or highly correlated measures should be avoided. • Statistical
analysis is used and reported appropriately. • Development data, training
data, and test data are distinguished from each other. • User studies are
based on adequately sized, representative cohorts; data is gathered in ways
that meet ethical norms, or where appropriate in keeping with prescribed
ethics practices. • Final results were obtained after all development was
complete, that is, not selected because they are the best outcomes amongst a
larger set of experiments or hand-fitted to the data.

##### Common problems that lead to rejection

Issues with papers in relation to the recommendations above can lead to
rejection. Other problems that can lead to rejection are as follows. •
Literature reviews that lack critical analysis of prior work or that largely
consist of lists of papers, that is, do not have an insightful discussion. •
Contributions that consist of small modifications to established techniques,
particularly where the contribution is a straightforward variation of the
established technique or where there are numerous prior papers exploring
similar variations. • Methods that appear to be developed and hand-tuned on a
specific data set without discussion or demonstration of their lessons for
future work or of how the methods would be more generally applicable. •
Justification of a method solely by its score in experiments, lacking an a
priori rationale for why the method is worth exploring. • Experiments where
the data volumes are too small to support the conclusions. • Any form of
academic fraud, misrepresentation, or dishonesty.

####  3.0.25 Next Steps

Guidance and lists of issues should be living documents that reflect a current
and uncontroversial agreement in the community. Therefore, they should be open
to change because there can always be some disagreements and expectations of
authors can change over time, in some cases quite quickly, especially as the
subjects of research shift to focus on new topics. For that reason, no set of
advice should be regarded as fixed, but revision should be undertaken
consultatively and with a spectrum of colleagues.

We suggest that the detailed list of issues of concern in Appendix 5.4 be made
available in some form as educative for reviewers. We stress here that it is
not our intention that reviewers simply reject papers because of these issues.
It could also provide a resource at forums such as doctoral consortia.

We thus believe that it would be valuable for the community to:

  * •

Ensure that the guidelines are prominent in the calls-for-papers at our major
conferences and journals, or otherwise disseminated.

  * •

Encourage the SIGIR executive committee to take ownership of the guidelines
and to occasionally convene a panel to produce an update.

  * •

Use these resources educatively for new members of the community and for new
reviewers.

In this exercise, we have not produced guidance for reviewers, which in other
disciplines tends to consist of two parts: general advice on how to approach
the task and specifics for the field. An example that we found was produced by
the ACL, as discussed above; a particular strength of these guidelines in our
view is the enumeration of unfair grounds for rejection. We believe that such
guidance would be of value to our community, and could make use of the
materials we have presented here.

## References

  * [1] Dennis Ulmer, Elisa Bassignana, Max Müller-Eberstein, Daniel Varab, Mike Zhang, Rob van der Goot, Christian Hardmeier, and Barbara Plank.  Experimental standards for deep learning in natural language processing research, 2022.

Christine Bauer
Utrecht University, NL Joeran Beel
University of Siegen, DE Timo Breuer
Technische Hochschule Köln, DE
Charles L. A. Clarke
University of Waterloo, CA
Anita Crescenzi
University of North Carolina at Chapel Hill, US Gianluca Demartini
The University of Queensland, AU Giorgio Maria Di Nunzio
University of Padua, IT Laura Dietz
University of New Hampshire, US Guglielmo Faggioli
University of Padua, IT Nicola Ferro
University of Padua, IT Bruce Ferwerda
Jönköping University, SE Maik Fröbe
Friedrich-Schiller-Universität Jena, DE Norbert Fuhr
University of Duisburg-Essen, DE Matthias Hagen
Friedrich-Schiller-Universität Jena, DE Allan Hanbury
TU Wien, AT Claudia Hauff
Spotify, NL Dietmar Jannach
University of Klagenfurt, AT Noriko Kando
National Institute of Informatics, JP Evangelos Kanoulas
University of Amsterdam, NL Bart P. Knijnenburg
Clemson University, US Udo Kruschwitz
University of Regensburg, DE Meijie Li
University of Duisburg-Essen, DE Maria Maistro
University of Copenhagen, DK Lien Michiels
University of Antwerp, BE Andrea Papenmeier
University of Duisburg-Essen, DE Martin Potthast
Leipzig University, DE Paolo Rosso
Technical University of Valencia, ES Alan Said
University of Gothenburg, SE Philipp Schaer
Technische Hochschule Köln, DE
Christin Seifert
University of Duisburg-Essen, DE Ian Soboroff
National Institute of Standards and Technology, US Damiano Spina
RMIT University, AU Benno Stein
Bauhaus-Universität Weimar, DE Nava Tintarev
Maastricht University, NL Julián Urbano
Delft University of Technology, NL Henning Wachsmuth
Leibniz Universität Hannover, DE Martijn Willemsen
Eindhoven University of Technology & JADS, NL Justin Zobel
University of Melbourne, AU

##  4 List of Acronyms

ACL

    Association for Computational Linguistics
ACM DL

    ACM Digital Library
ADM+S

    Automated Decision-Making and Society
AI

    Artificial Intelligence
ASSIA

    Asian Summer School in Information Access
CfP

    Call for Papers
CHI

    Computer-Human-Interaction
CLEF

    Conference and Labs of the Evaluation Forum
CyCAT

    Cyprus Center for Algorithmic Transparency
ECIR

    European Conference on Information Retrieval
ESS

    Experiment Support System
ESSIR

    European Summer School on Information Retrieval
FAccT

    Fairness, Accountability, and Transparency
FDIA

    Future Directions in Information Access
FIRE

    Forum for Information Retrieval Evaluation
GDPR

    General Data Protection Regulation
IR

    Information Retrieval
KPI

    Key Performance Indicator
LLM

    Large Language Model
ML

    Machine Learning
MRR

    Mean Reciprocal Rank
NLP

    Natural Language Processing
nDCG

    normalized Discounted Cumulative Gain
NTCIR

    NII Testbeds and Community for Information access Research
PC

    Program Committee
PyIRE

    Python Interactive Information Retrieval Evaluation
QA

    Question Answering
RS

    Recommender Systems
RecSys

    ACM Conference on Recommender Systems
SSH

    Social Sciences and Humanities
SPC

    Senior Program Committee
SEME

    Search Engine Manipulation Effect
SIGIR

    ACM SIGIR Conference on Research and Development in Information Retrieval
SIGIR-AP

    Information Retrieval in the Asia Pacific
SIGPLAN

    ACM Special Interest Group on Programming Languages
SOTA

    State Of The Art
TREC

    Text REtrieval Conference
UX

    User Experience
WiMIR

    Women in Music Information Retrieval

##  5 Author Guidance Appendix

This Appendix consists of annotated materials that helped to inform the list
of concerns described in Section References, and the list of concerns itself.
As explained in the main text, our aim was to gather suggestions of guidance
and issues from a range of sources and consolidate them into a resource for
IR.

###  5.1 SIGPLAN Empirical Evaluation Guidelines with Annotations

The following is our annotation of the SIGPLAN
Guidelines.878787https://www.sigplan.org/Resources/EmpiricalEvaluation/ In
this annotation, we underlined aspects deemed particularly worth reflecting in
guidelines for IR. Strikethrough was used for aspects that we felt did not
translate to our community well, and greyed text for aspects, we felt to be
valuable but needing adaptation for an IR context.

#### Clearly stated claims

S1: Claims not explicit

  * •

Claims must be explicit in order for the reader to assess whether the
empirical evaluation supports them. Missing claims cannot possibly be
assessed. Claims should also aim to state not just what is achieved but how.

S2: Claims not appropriately scoped

  * •

The truth of a claim should clearly follow from the evidence provided. Claims
that are not fully supported mislead readers. ’Works for all Java’ is over-
broad when based on a subset of Java. Other examples are ’works on real
hardware’ when evaluating only with (unrealistic) simulation, and ’automatic