# Auctions and Peer Prediction
for Academic Peer Review

Siddarth Srinivasan
School of Computer Science and Engineering
University of Washington
Seattle, WA 98195
<EMAIL_ADDRESS>
&Jamie Morgenstern
School of Computer Science and Engineering
University of Washington
Seattle, WA 98195
<EMAIL_ADDRESS>

###### Abstract

Peer reviewed publications are considered the gold standard in certifying and
disseminating ideas that a research community considers valuable. However, we
identify two major drawbacks of the current system: (1) the overwhelming
demand for reviewers due to a large volume of submissions, and (2) the lack of
incentives for reviewers to participate and expend the necessary effort to
provide high-quality reviews. In this work, we adopt a mechanism-design
approach to propose improvements to the peer review process, tying together
the paper submission and review processes and simultaneously incentivizing
high-quality submissions and reviews. In the submission stage, authors
participate in a VCG auction for review slots by submitting their papers along
with a bid that represents their expected value for having their paper
reviewed. For the reviewing stage, we propose a novel peer prediction
mechanism (H-DIPP) building on recent work in the information elicitation
literature, which incentivizes participating reviewers to provide honest and
effortful reviews. The revenue raised in the submission stage auction is used
to pay reviewers based on the quality of their reviews in the reviewing stage.

##  1 Introduction

Scientific publishing through peer review is widely regarded as the gold
standard for disseminating scientific research. Yet, despite its widespread
use and acceptance, it is plagued by a number of well-documented problems:
willing reviewers are often in short supply and reviews themselves are often
of poor quality, giving rise to noisy acceptance decisions (Cortes and
Lawrence,, 2021; Shah et al.,, 2018). Studying the history of peer review in
sociology, Merriman, (2020) writes that while peer review is ‘a practice
intended to assess the merits of pieces of scholarship, its features did not
arise because they were believed to be an especially apt means of identifying
the best work.’ The issues with peer review have become especially salient in
machine learning as well as computing more broadly. Computing relies largely
on a conference model for publishing and disseminating research, and while the
number of submissions to major conferences has exploded, the pool of qualified
reviewers has grown more slowly (Sculley et al., 2018b, ; Shah,, 2019;
Stelmakh et al.,, 2020). As these issues are only likely to persist, a
solution addressing the problem of low-quality reviews is needed.

In this work, we adopt a mechanism-design approach to tackle two drawbacks of
the current system of conference-based publishing in machine learning: (1) a
large number of submissions, leading to poor and often arbitrary criteria for
desk rejection, and (2) poor incentives for reviewers that make it challenging
to find qualified reviewers willing to provide effortful and honest reviews.
Our proposed mechanism tackles (1) by requiring authors to participate in an
auction for review slots, and tackles (2) via a peer prediction mechanism
(involving predicting peers’ review scores) that rewards reviewers for honest
and effortful reviews. The mechanism involves minimal modification to the
current peer review system; it requires authors to submit a bid accompanying
each paper, and reviewers to participate in a simple prediction game after
submitting their reviews. The paper-reviewer matching process and
accept/reject decision process may be independently improved or remain
unchanged. Our mechanism would adopt a novel currency usable within a
consortium of conferences, with the goal of incentivizing authors to provide
the much needed service of peer review in order to earn credits exchangeable
for having their own papers reviewed. We outline our proposed process below:

  1. 1.

Reviewer Signup: The conference announces a reviewer signup period open until
a few days prior to the submission deadline.

  2. 2.

Review Slot Announcement: Based on the number of reviewers signed up, the
conference estimates and announces the number of review slots $P$ that can be
supported without overburdening reviewers and compromising review quality.

  3. 3.

Review Slot Auction: Each submitted paper is accompanied by an author’s bid
for a review slot. A VCG auction determines which papers get review slots and
how much the bidder pays. Authors are not charged for papers that don’t
receive review slots.

  4. 4.

Reviewer Matching (_unchanged_): The conference assigns reviewers to papers
accepted for review per the usual approach, ensuring at least 3 well-matched
reviewers to a paper.

  5. 5.

Reviewing: Each reviewer scores their assigned paper on several criteria and
submits their review per the usual process. _Additionally_ , they play a
simple asynchronous game after all reviews are submitted, involving predicting
another reviewer’s score. Reviewers are compensated in artificial currency
according to their performance in this game.

  6. 6.

Accept/Reject Decisions (_unchanged_): Decisions are made per the standard
process.

While our proposal is generally applicable to peer review, our perspective is
informed by the conference publication model in machine learning. There remain
many questions to be answered before our proposed mechanism can be deployed in
practice, and we primarily intend for this work to lay the theoretical
foundations for a plausible (but by no means comprehensive) approach to be
iterated upon in designing a robust scientific publication process. Our paper
is structured as follows: in section 2, we provide some background on academic
peer review and motivate the problem; in section 3, we describe our proposed
mechanism; in section 4, we present related work; and in Section 5, we
conclude with a discussion of some important considerations.

##### Main Contributions

We make two main contributions: (1) we propose a peer review mechanism
consisting of a submission stage and a reviewing stage, where the revenue
raised by the former is used to pay for the latter, and (2) we design the
reviewing stage by building on previous work in the information elicitation
literature to develop a novel peer prediction mechanism that incentivizes
reviewers to provide honest and effortful reviews.

Figure 1: Diagram of proposed mechanism for academic peer review

##  2 Background

Although there are some minor variations, the process of peer review can be
summarized as follows: (1) authors submit their work to a conference, (2) some
papers receive desk rejection, (3) a paper that is not desk rejected is
matched with a small number (3-5) of willing reviewers (who do not know the
identity of the authors and vice-versa), (4) the reviewers independently score
the paper’s quality for publication based on several criteria such as
technical correctness, interest to the community, and quality of exposition
(the reviewers may or may not know each others’ identities), (5) a meta-
reviewer aggregates the reviewers’ scores and makes an accept/reject decision,
incorporating the conference’s targeted acceptance rate into their decision.
With this context, we now discuss the often cited primary motivations for and
criticisms of peer review below. Merriman, (2020) argues that these
‘intellectual’ motivations tend to be post-hoc rationalizations, and the
current process of peer review is ‘best understood as the product of
continuous efforts to steward editors’ scarce attention while preserving an
open submission policy that favors authors’ interests’. Part of our goal then
is to develop a peer review mechanism that lives up to the stated
‘intellectual’ motivations for the process.

###  2.1 Motivations for Peer Review

  1. M1

Screening Results for Validity: Peer review serves as a community-issued
certificate that the approach and results in a publication seem plausible,
replicable, and not fraudulent.

  2. M2

Screening Results for Importance to the Community: The publication record
serves as a central repository of curated results that the field considers
important or impactful and wishes to focus on; the conference itself serves to
build consensus and host debates on the cutting-edge issues in the field.

  3. M3

Reputation Building/Signaling: Publication at selective venues also serves to
build reputation and advance careers of authors; the idea is that a selective
acceptance process that identifies valid and interesting results also
identifies individuals who do good work.

  4. M4

Feedback and Networking: Peer review serves as a systematic approach to
obtaining feedback on both ongoing and completed work, as well as finding
collaborators interested in tackling similar problems.

M1 and M2 are often cited as the primary motivations for peer review, and are
arguably the most foundational. While M3 is not often cited explicitly as a
motivation, it is well-known that publishing frequently at top-tier venues
improves career advancement prospects; peer review clearly serves as a
mechanism to identify individuals who should be rewarded. M3 derives its
legitimacy from M1 and M2. Meanwhile, M4 is an auxiliary benefit derived from
M1 and M2. We argue that the peer review system should be judged on its
ability to satisfy M1 and M2 while cognizant of its social functions M3 and M4
so as to avoid unintended consequences.

###  2.2 Criticisms of Peer Review

The general criticism of peer review is that the reviews are often sub-par and
the accept/reject decisions are too noisy. Various NeurIPS experiments (Cortes
and Lawrence,, 2021; Shah et al.,, 2018; NeurIPS,, 2019) provide some
empirical basis for this criticism; the 2014 NeurIPS experiment in particular
(see Appendix B) found that if the conference was re-run, about a quarter of
papers would have their accept/reject decisions swapped, and only about half
of accepted papers would be re-accepted. There is also a perception that
reviewers are often misled by unsubstantiated claims (Lipton and Steinhardt,,
2018) or work that _looks_ important but isn’t (Vazire,, 2017). This may
incentivize the submission of premature, lower quality work which if rejected,
may be resubmitted at no cost to future conferences with minor tweaks with the
goal of getting through a noisy review process (Bengio,, 2020). In this
context, we discuss some of the explanations for why the current system of
peer review within machine learning may be sub-par, although our discussion
may apply to peer review in general.

  1. C1

The Extensive Margin of Reviewing: There are too many submissions and not
enough reviewers, so the current system struggles to match submissions with
qualified and interested reviewers. There are no explicit incentives for
reviewers, so reviewers participate entirely for pro-social reasons. The
shortage of reviewers has been exacerbated by the recent explosion in
submissions to top conference venues in machine learning over the past decade.
The large number of submissions can also lead to desk-rejection policies that
can be somewhat opaque and arbitrary. Poor matching due to a scarcity of
reviewers could explain why reviews are often sub-par, so a mechanism that
incentivizes reviewers to participate could be valuable.

  2. C2

The Intensive Margin of Reviewing: Even when reviewers participate
voluntarily, they may _not be willing to exert enough effort_ to provide a
high-quality assessment of the paper, and there is currently no explicit
incentive to do so. In an attempt to redress the shortage of reviewers, some
conferences require authors to also participate in review. While this would
increase the supply of reviewers, it could potentially crowd-out pro-social
motivations and worsen the average quality of reviews without proper
incentives. Thus, we may need to provide the appropriate incentives to elicit
effortful reviews, especially when requiring participation in the review
process.

  3. C3

Dishonest Reviews: Some have noted that even when reviewers exert enough
effort to understand a paper, they may not report their honest opinion for
various reasons. Thus, it is desirable to design the review process to remove
obstacles to truthful reviewing.

  4. C4

Ability of Reviewers: Another strain of criticism is that peer review is
fundamentally flawed and _it cannot work_ even when reviewers provide honest
and effortful reviews on well-matched papers. There are two versions of this
criticism: the first is that reviewing is too subjective for reviewers to
agree on results’ validity and importance to the community, and the second is
that even when reviewers agree, their reviews do not track any meaningful
measure of importance to the community. Empirical evidence from the 2014
NeurIPS experiment (where all the aforementioned incentive issues persist)
suggests that reviewers’ scores correlate moderately well with each other as
well as with log-citation counts; it may be quite possible to improve upon
these results by remedying incentive issues.

Our focus in this paper is to address the first three criticisms head on with
a mechanism-design approach. On the paper submission side of the process, our
proposed mechanism disincentivizes lower-quality submissions aimed at getting
through a noisy review process. On the paper reviewing side, our proposed
mechanism incentivizes reviewers to participate in the peer review process,
exert effort, and report their assessments truthfully. Our mechanism is
agnostic to the reviewer-paper matching process, and we discuss this in
Appendix G. The key challenge we face is the lack of a ground-truth
verification for the submitted reviews, which makes it difficult to verify
whether reviewers’ assessments are truthful and effortful. We tackle this
issue by building on recently proposed mechanisms from the information
elicitation without verification (IEWV) literature.

##  3 A Mechanism for Academic Peer Review

We now describe our peer review mechanism that incentivizes authors to submit
papers they believe are likely to be accepted, and reviewers to participate,
exert effort, and honestly review papers. Stage I deals with the paper
submission process, where authors bid for review slots, and Stage II deals
with the paper review process, where reviewers are paid based on the quality
of their reviews. The paper-reviewer matching process and accept/reject
decision process may remain unchanged. We envision the mechanism using its own
currency (henceforth referred to simply as ‘credits’) that can be utilized
within a consortium of conferences.

##### Summary

In the Stage I submission mechanism, authors participate in a VCG auction for
review slots, with one author on each paper $\omega_{i}$ also submitting an
accompanying bid $b_{i}$ (in mechanism-specific credits). The winners of the
auction pay the highest losing bid and have their papers _accepted for
review_. In the Stage II reviewing mechanism (which we call H-DIPP), every
paper $\omega_{i}$ accepted for review is assigned at least 3 reviewers.
Reviewers proceed per the usual peer review process, supplying reviews and
scoring the paper on a common, fixed set of $T$ criteria (e.g. novelty,
correctness, interest to the community, quality of writing). The reviewers
will then also participate in a simple peer prediction game to report
_predictions_ of the scores of one other peer, twice for each criterion;
reviewers first predict the peer’s review score on a given criterion, are
shown the third peer’s score on that criterion, and are asked to update their
prediction. This proceeds sequentially for each criterion in a pre-determined
order. Reviewers’ predictions of a peer’s review scores are rewarded based on
their accuracy (with the log proper scoring rule), and reviewers’ review
scores on each criterion are rewarded based on how much it aids another peer
in their prediction game. We prove that under such a reward scheme, reviewers
maximize their expected payoff when they provide honest and effortful reviews
(when everyone else also does so). We emphasize that the prediction game
involves predicting other reviewers’ review scores, and not something like the
paper’s expected impact. The revenue raised from the submission stage auction
can be used to pay reviewers for their reviews in the reviewing stage.
Decisions on which papers are _accepted for publication_ can be made per the
usual process. Authors who submitted bids must ensure that they have (or can
earn) enough credits to cover their auction bids, or risk having their reviews
and potential acceptance withheld. In general, authors face the budget
constraint of submitting bids no larger than the value of credits they are
willing and able to earn through reviewing. Authors who provide the valuable
and scarce service of reviewing will be able to earn and bid more credits,
providing a positive incentive to participate in the review process.

######  Remark 1.

We note that there are two simple ways to incentivize authors to earn
sufficient credits via reviewing to cover their auction bid. The first is to
refuse to accept papers through the auction without sufficient store of
reviewing credit; the second is to model the conference submission process as
a repeated game where an author can either save credits or be indebted credits
from round to round. We do not assume that every author is a reviewer for the
same conference, or vice-versa.

###  3.1 Setup

##### Notation

Suppose we have $N$ papers submitted to the conference peer review mechanism
denoted $\omega_{i}$ (for $i\in[N]$ ) with $K$ participating agents. We use
the index $i$ when referring to papers in general and the index $k$ in
referring to agents in general. We represent the papers submitted by agent $k$
as $\\{\omega_{i_{s}}^{k}\\}$ and the papers reviewed by an agent $k$ as
$\\{\omega_{i_{r}}^{k}\\}$ with the natural condition that
$\\{\omega_{i_{s}}^{k}\\}\cap\\{\omega_{i_{r}}^{k}\\}=\emptyset$. Author $k$’s
Stage I bid accompanying paper $\omega_{i}$ is $b_{i}$. Reviewer $k$’s scores
of paper $\omega_{i}$ on the $T$ criteria are written as $({\bf
x}_{k}^{[T]})_{i}$, their first predictions of peer $k^{\prime}$’s reported
scores as $(\hat{\bf p}_{k^{\prime[T]}\leftarrow k})_{i}$, and their second
predictions of peer $k^{\prime}$’s reported scores as $(\hat{\bf
p}^{+}_{k^{\prime[T]}\leftarrow k})_{i}$. When reviewer $k$ exerts enough
effort to receive informed private signals on $t_{k}$ of $T$ criteria, the
cost of the effort is $e_{k}(t_{k})$. A complete table of notation is
presented in Appendix A.

##### Mechanism Design Background

We define an agent $k$’s _strategy_ as a map from their private information to
a distribution over reports elicited by a mechanism. An agent’s _submission
strategy_ consists of their bids for all their submitted papers, denoted
$s_{k}^{(1)}=\left(b_{i}\right)_{i\in\\{\omega_{i_{s}}^{k}\\}}$. An agent’s
_reviewing strategy_ consists of the effort they exerted to arrive at an
informed review score on $t_{k}$ criteria, reported scores on all $T$ criteria
and their two predictions of a peer $k^{\prime}$s score on every criterion for
every paper they review, denoted
$s_{k}^{(2)}=\left(\left\\{e_{k}(t_{k}),\hat{\bf x}_{k}^{[T]},\hat{\bf
p}_{k^{\prime[T]}\leftarrow k},\hat{\bf p}_{k^{\prime[T]}\leftarrow
k}\right\\}_{i}\right)_{i\in\\{\omega_{i_{r}}^{k}\\}}$. A _strategy profile_
is a tuple of every agent’s strategy ${\bf s}=(s_{1},s_{2},\ldots,s_{K})$. The
strategy profile of all agents excluding agent $k$ is ${\bf s}_{-k}$.

######  Definition 1 (Dominant Strategy).

A strategy $s_{k}$ is dominant if the strategy profile $(s_{k},{\bf s}_{-k})$
maximizes agent $k$’s payoff for any strategy profile ${\bf s}_{-k}$.

######  Definition 2 ((Strict) Bayesian Nash Equilibrium (BNE)).

Suppose player $k$ observes private signal $x_{k}$. A strategy profile ${\bf
s}=(s_{1}(x_{1}),s_{2}(x_{2}),\ldots,s_{K}(x_{K}))$ is a Bayesian Nash
Equilibrium if no agent $k$ can increase their expected payoff by deviating
from strategy $s_{k}(\cdot)$ when the other agents play ${\bf s}_{-k}$, for
any private signal $x_{k}$. The BNE is strict if deviating from $s_{k}$
decreases agent $k$’s expected payoff. If the strategy profile of the BNE
consists of identical strategies for all agents, the BNE is symmetric.

##### Utility Function

We specify the utility function $U^{(1)}_{k}$ of an author agent $k$
participating in the submission mechanism (Stage I) as follows:

$U_{k}^{(1)}=\begin{cases}\sum_{i\in\\{\omega_{i_{s}}^{k}\\}}(v_{k}\delta_{i}-c_{i})&\text{accepted
for review}\\\ 0&\text{reject without review}\end{cases}$ (1)

where $\delta_{i}$ is a random variable indicating whether or not the paper is
accepted _for publication_ , $v_{k}$ is the agent’s value in credits for a
paper accepted for publication and $c_{i}$ is the price charged by the VCG
auction (Stage I) for a review slot. We can similarly specify the utility
function $U_{k}^{(2)}$ of a reviewing agent $k$ participating in the H-DIPP
reviewing mechanism (Stage II) as follows:

$U_{k}^{(2)}=\sum_{i\in\\{\omega_{i_{r}}^{k}\\}}r_{k}\left(\left[s_{k}^{(2)}\right]_{i},\left[{\bf
s}_{-k}^{(2)}\right]_{i}\right)-\left({e}_{k}(t_{k})\right)_{i}$ (2)

where $r_{k}$ is the reward credits paid to reviewer $k$ by the H-DIPP
mechanism for a given strategy $[s_{k}^{(2)}]_{i}$ on paper $\omega_{i}$. We
analyze the incentives and strategy profiles of the two stages separately.
With this setup, we now present the submission and reviewing stages of the
mechanism.

###  3.2 Stage I: Paper Submission

Our Stage I mechanism uses a VCG auction to allocate review slots and charge
winners the threshold bid necessary for their paper to be reviewed. We assume
that the number of review slots $P$ is fixed and announced by the conference
before paper submission deadline, based on how many reviews it expects to be
able to pay for (perhaps with some predictive model using historical data,
with some leeway to run a deficit). We defer a discussion of how the number of
review slots $P$ may be determined dynamically to Appendix G, along with the
associated considerations and challenges. The auction asks authors to submit a
bid $b_{i}$ along with every paper $\omega_{i}$ indicating how much they are
willing to pay for a review slot for that paper. Authors can observe the
quality of their paper $\omega_{i}$ and arrive at a belief that the paper will
be accepted for publication with probability $\eta_{i}$ conditional on review
under the equilibrium of the submission mechanism111In other words, authors
should reason about their paper’s acceptance probability in a pool with a
particular combination of $P$ of the ‘the best papers’ and papers from
prolific reviewers who can bid higher since they review more. This would be
the equilibrium of the submission mechanism; if the pool had some other
combination of papers, a stronger paper or a prolific reviewer with larger
store of credits would be willing to bid higher to displace a paper in the
pool with their paper. We are assuming that by attending conferences and
participating in review, authors have some notion of how competitive their
paper will be in relative terms. (_bids_ have no bearing on the chance of
acceptance for publication). Hence, denoting reviewer $k$’s value of a paper
accepted for publication as $v_{k}$, the expected value of a review slot is
$\mathds{E}[v_{k}\delta_{i}]=v_{k}\eta_{i}$. We also have the following
natural conditions: (1) the review slots are identical222Even though from the
reviewing side, the review slots are heterogeneous (papers will differ in
terms of cost of finding well-matched reviewers), we treat the slots as
identical from the author’s side in the interest of fairness. This could be
relaxed by splitting the conference into ‘tracks’ within which this treatment
is more justifiable., (2) each paper may only receive a single review slot,
and (3) package bids are not allowed, i.e., authors cannot make a single bid
for multiple review slots for multiple papers, but must instead make a
separate bid for each slot for each paper. In this setting, it is easy and
practical to identify the winners of the auction when a single author can
submit multiple papers for review.

After collecting bids, the VCG auction allocates the $P$ review slots to the
$P$ papers with the highest bids, denoted
$\\{\omega^{*}_{1},\ldots,\omega^{*}_{P}\\}$, with associated bids
$\\{b_{p}^{*}\\}_{p\in[P]}$. Denoting the paper with the $P+1$-highest bid
${b}^{*}_{{P+1}}$ as ${\omega}^{*}_{P+1}$, the mechanism charges all authors
of the papers with winning bids a price of $c_{i}=\sum_{\ell=1,\ell\neq
i}^{P+1}{b}^{*}_{\ell}-\sum_{\ell=1,\ell\neq
i}^{P}{b}^{*}_{\ell}={b}^{*}_{P+1}$. Thus, all papers accepted for review are
charged the highest losing bid333This is correct when viewing papers-as-
bidders, but a potential concern when viewing authors-as-bidders is that
authors who submit multiple papers may try to bid strategically. Strictly
speaking, VCG would require charging author $k$ who gets $p_{k}$ slots the sum
of the $p_{k}$ highest losing bids excluding their own. In practice, this is
unlikely to matter since the number of review slots vastly outnumbers an
author’s submissions., and accepting these papers maximizes author welfare for
a given number of review slots $P$.

Truthful bidding is a _weakly dominant strategy_ for VCG, and we expect
authors’ bids to equal to their value for having their paper reviewed. The
natural assumption of identical review slots and prohibition of package bids
precludes the need to solve a computationally challenging knapsack problem to
identify winners. Thus, the _Stage I strategy profile_ that maximizes
aggregate author welfare under a given number of review slots $P$ is ${\bf
s}^{(1)}=\left(\left(\\{v_{k}\eta_{i}\\}_{i\in\\{\omega_{i_{s}}^{k}\\}}\right)_{k=1,\ldots,K}\right)$.
The mechanism allocates review slots to the highest bidding papers, which will
generally be papers that are likely to be accepted for publication and/or have
authors who generate a lot of value as reviewers. Thus, the papers accepted
for review should also ideally be of higher quality if the review process
screens well for quality. This should also eliminate the need for desk
rejection.

###  3.3 Stage II: Peer Review

We now present the second stage of the mechanism which focuses on
incentivizing honest and effortful reviews of the $P$ papers accepted by the
VCG auction. The context here is as follows: the bid requirement for the
submission process in Stage I auction creates demand for credits, and since
this can only be earned by reviewing, it incentivizes researchers to join the
reviewer pool. While this is desirable, we must structure the payment for
reviews carefully. A flat fee to reviewers could result in reviewers putting
in minimal effort and getting paid anyway; if credits could be earned with
little effort, bids could be made arbitrarily high, defeating the point of the
mechanism. Instead, we need to structure the payment such that honest and
effortful reviews are paid more than low-effort or false reviews. This is a
challenging problem in the absence of ground truth verification; and our
proposed H-DIPP mechanism is a novel solution to this problem, drawing on
insights from the Target Differential Peer Prediction (TDPP) (Schoenebeck and
Yu,, 2020) and Hierarchical Mutual Information Paradigm (HMIP) (Kong and
Schoenebeck,, 2018) (see Appendix D). The main technical challenge here
involves determining what information to reveal during the prediction game, in
what order, and how to structure the payments. To our knowledge, our design is
the only approach that provides the desired equilibrium results. Note that the
H-DIPP mechanism stage describes only the reviewing process for a single
paper; we are agnostic about the paper-reviewer matching process, and meta-
reviewers can make final accept/reject decisions as usual. We present some
fundamentals of the Mutual Information Paradigm (Kong and Schoenebeck,, 2019)
used to develop this theory in Appendix C. Proofs are presented in Appendix I.
We now give a more detailed discussion of our reviewing mechanism H-DIPP
(Mechanism 1).

####  3.3.1 Model Preliminaries

Each reviewer rates the paper $\omega_{i}$ on a common set of $T$ criteria.
Each criterion $t$ is rated on a $D_{t}$-point scale, i.e., reviewers’
reported scores can take on any value in a
$\Sigma_{t}=\\{1,2,\ldots,D_{t}\\}$. The same H-DIPP mechanism will be used
determine payments for every paper’s reviewers, so we drop the index $i$
referring to a specific paper going forward. Now, suppose we have a sample
space $\Omega$ of all the possible realizations of papers (‘world states’) a
reviewer may see. A reviewer $k$’s _private review score_ on criterion $t$ is
the random variable ${X}^{t}_{k}:\Omega\rightarrow\Sigma_{t}$ describing their
honest private assessment (or ‘signal’) of the paper on that criterion, _when
they exert the necessary effort to arrive at an informed opinion_. A
reviewer’s joint distribution of such ‘effort-informed review scores’ for all
criteria is represented by the multivariate random variable ${\bf
X}_{k}^{[T]}=(X_{k}^{1},X_{k}^{2},\ldots,X_{k}^{T})\in\Sigma_{1}\times\Sigma_{2}\times\ldots\times\Sigma_{T}$,
and the full joint distribution of effort-informed review scores over all
criteria for all reviewers is $\mathds{P}({\bf X}_{A}^{[T]},{\bf
X}_{B}^{[T]},{\bf X}_{C}^{[T]})$. We assume that this joint distribution is
common knowledge and has full support, i.e., every assignment of outcomes has
non-zero probability. This is a standard basic assumption in our setting, and
necessary in order to make any claims about agents’ beliefs about the papers
they’re reviewing. We discuss how this assumption can be made more realistic
in Section 5, and other assumptions in Appendix E. We state this assumption
formally below:

######  Assumption 1 (Common Prior with Full Support).

Reviewers’ effort-informed signals on all criteria
$\\{{\textbf{x}}_{A}^{[T]},{\textbf{x}}_{B}^{[T]},{\textbf{x}}_{C}^{[T]}\\}$
are drawn from the common prior $\mathds{P}({\bf X}_{A}^{[T]},{\bf
X}_{B}^{[T]},{\bf X}_{C}^{[T]})$, and the common prior is common knowledge.
The common prior also has full support.

##### Criteria Hierarchy

Next, we construct a common _hierarchy of criteria_ that each reviewer will
rate the paper on. The idea is structure the hierarchy so such that reviewers
require less effort to come to an informed private assessment on ‘lower’ level
criteria than ‘higher’ level criteria, i.e., the total effort required to
arrive at informed private assessments increases up the hierarchy. Thus, we
say $t_{i}>t_{j}$ if scoring the paper on criterion $t_{i}$ requires strictly
more effort than providing a score on criterion $t_{j}$. When a reviewer has
exerted enough effort to arrive at an informed review score $x_{k}^{t}$ on
criterion $t$, we say that reviewer has _completed_ criteria
$[t_{k}]=\\{1,\ldots,t_{k}\\}$. The idea is to explicitly construct lower
level criteria to collect the cheap signals and pay agents more for the
information _added_ by valuable higher level criteria. We can define the
positive, increasing function $e_{k}:[T]\rightarrow\mathds{R}^{+}$ specifying
reviewer $k$’s _total effort cost_ (in the mechanism-specific credits) to
obtain effort-informed signals up to criterion $t$, where we assume
$e_{k}(t)>e_{k}({t^{\prime}})$ if $t>t^{\prime}$.

In the current peer review setting at machine learning conferences, reviewers
are already required to report scores on a variety of criteria, so we simply
require a particular structuring of existing criteria. As an example, one
possible hierarchy of criteria could be the following (where responses fall on
a 5-point scale): 1) _(presentation)_ “How clear is the writing/exposition?”;
2) _(completeness)_ “How easy would it be to reproduce major results?”; 3)
_(correctness)_ “How accurate are the technical claims and methodology?”; 4)
_(contribution)_ “How interesting and valuable of a contribution is the work?
”; 5) _(overall quality)_ “In which quintile of submitted papers would this
work fall?"

Figure 2: An illustration of the H-DIPP mechanism from reviewer $A$’s
perspective, when reviewer $B$ is their assigned target and reviewer $C$ is
their assigned source.

####  3.3.2 Mechanism and Payments

Having detailed our assumptions, we now describe exactly how the mechanism
collects reports and computes the payment. The mechanism is summarized as
Mechanism 1 and illustrated in Figure 2. We give a step-by-step example in
Appendix F.

##### Reports

First, each reviewer reports their review scores to the mechanism per the
usual process. Reviewers arrive at private review scores ${\bf
x}^{[t_{k}]}_{k}$ on $t_{k}$ criteria by exerting $e_{k}(t_{k})$ effort, and
report this to the mechanism (potentially strategically) as $\hat{\bf
x}^{[T]}_{k}\sim\theta_{k}(\cdot|{\bf x}^{[t_{k}]}_{k})$.

Once all the reviews are in, every reviewer is matched with two others and
(independently) plays a simple prediction game. Suppose reviewer $A$ plays the
game and is matched with reviewers $B,C$. The mechanism will first elicit the
reviewer $A$’s prediction $\hat{\bf p}_{B^{1}\leftarrow
A}=P\left(\hat{X}_{B}^{1}|{\bf x}_{A}^{[t_{A}]}\right)$ of the reviewer $B$’s
score on criterion $1$. Then, it reveals reviewer $C$’s score
$\hat{x}_{C}^{1}$ on that criterion and asks for an updated prediction from
reviewer $A$: $\hat{\bf p}^{+}_{B^{1}\leftarrow A}=P\left(\hat{X}_{B}^{1}|{\bf
x}_{A}^{[t_{A}]},\hat{\bf x}_{C}^{1}\right)$. The mechanism repeats this for
criterion $2$, $3$, $\ldots$, and so on until the $T$th criterion. The
mechanism thus requests $3T$ reports: reviewers’ $T$ review scores, their
first predictions of their assigned target’s scores on $T$ criteria, and
updated second predictions of that target peer’s scores on $T$ criteria given
the source peer’s scores. A reviewer’s strategy on the reviewing task is
$s_{k}=\left(e_{k}(t_{k}),\hat{\bf x}_{k}^{[T]},\hat{\bf p}_{k^{\prime
t}\leftarrow k},\hat{\bf p}^{+}_{k^{\prime t}\leftarrow k}\right)$.

Input: $\omega_{i}$ (paper under review), Reviewer $A$ matched with reviewers
$B$ and $C$, $\\{t_{1},t_{2},\ldots,t_{T}\\}$ (hierarchical criteria),
$\\{\boldsymbol{\alpha}_{A}^{[T]},\boldsymbol{\beta}_{A}^{[T]}\\}$ (mechanism
hyperparameters)

Output: $r_{A}$ (payments to reviewer $A$)

1

2Elicit reviewer $A$’s scores on each criterion $\\{\hat{\bf x}_{A}^{[T]}\\}$
per the usual peer review process

3 for  _each criterion $t$ ascending the criteria hierarchy_ do

4

5 Elicit reviewer $A$’s predictions of reviewer $B$’s score on criterion $t$
($\hat{\bf p}_{B^{t}\leftarrow A}$).

6

7 Reveal reviewer $C$’s score $\hat{x}_{C}^{t}$ on criterion $t$ to reviewer
$A$.

8

9 Elicit reviewer $A$’s _updated_ prediction of reviewer $B$’s score on
criterion $t$ ($\hat{\bf p}^{+}_{B^{t}\leftarrow A}$).

10

11
$r_{A^{t}}^{\text{expert}}\leftarrow{S_{\text{log}}(\hat{x}_{B}^{t},\hat{\mathbf{p}}_{B^{t}\leftarrow
A})+S_{\text{log}}(\hat{x}_{B}^{t},\hat{\mathbf{p}}^{+}_{B^{t}\leftarrow A})}$

12

13 $r_{A^{t}}^{\text{target}}\leftarrow
S_{\text{log}}(\hat{x}_{A}^{t},\hat{\mathbf{p}}_{A^{t}\leftarrow
C}^{+})-S_{\text{log}}(\hat{x}_{A}^{t},\hat{\mathbf{p}}_{A^{t}\leftarrow C})$

$r_{A}\leftarrow\sum_{t=1}^{T}\alpha_{A}^{t}r_{A^{t}}^{\text{expert}}+\beta_{A}^{t}r_{A^{t}}^{\text{target}}$

Mechanism 1 H-DIPP

##### Payments

The mechanism will pay each reviewer for their roles as expert and target
(source payment is 0). Reviewers receive ‘expert payment’ based on their
performance in the prediction game, and receive ‘target payment’ based on how
reliably their scores relate to a peer’s (Schoenebeck and Yu,, 2020). With
mechanism hyperparameters
$\\{\boldsymbol{\alpha}_{A}^{[T]},\boldsymbol{\beta}_{A}^{[T]}\\}$, the
criterion-$t$ specific payment to reviewer $A$ using the log scoring rule
$S_{\text{log}}$ is:

$\small
r_{A^{t}}=\alpha_{A}^{t}\left(\underbrace{S_{\text{log}}(\hat{x}_{B}^{t},\hat{\bf
p}_{B^{t}\leftarrow A})+S_{\text{log}}(\hat{x}_{B}^{t},\hat{\bf
p}^{+}_{B^{t}\leftarrow A})}_{\text{expert
payment}}\right)+\beta_{A}^{t}\left(\underbrace{S_{\text{log}}(\hat{x}_{A}^{t},\hat{\bf
p}_{A^{t}\leftarrow C}^{+})-S_{\text{log}}(\hat{x}_{A}^{t},\hat{\bf
p}_{A^{t}\leftarrow C})}_{\text{target payment}}\right)$ (3)

A reviewer’s _expert payments_ depend on how good their predictions were
during the prediction game (incentivizing truthful predictions). A reviewer’s
_target payments_ depend how much better a peer expert’s second prediction of
their reported review score was over their first; the intuition is that their
peer’s second prediction can only update reliably towards the target
reviewer’s true review score (thus incentivizing truthful reviews), given a
third reviewer’s score (see Proof of Theorem 1 in Appendix I). The mechanism
hyper-parameters are used to appropriately weight the criterion-specific
payments to cover the marginal effort costs of completing higher level
criteria (see Theorem 3 in Appendix F). To compute reviewer $A$’s total
payment, we simply sum the individual criterion-specific payments
$r_{A}=\sum_{t}r_{A^{t}}$.

####  3.3.3 Equilibria

We now analyze the equilibria of our mechanism444We ignore permutation
equilibria, where the labels for signals are permuted. Such equilibria exist
in our mechanism, but they are not of serious concern; see Kong and
Schoenebeck, (2019) for details.. First, we give the strict truthfulness of
our mechanism when all reviewers complete a certain number of criteria.

######  Theorem 1 (Strict Truthfulness).

Suppose reviewers $A,B,C$ complete $[t_{A}],[t_{B}]$, and $[t_{C}]$ criteria
respectively. Then, on shared complete criteria
$[t^{\prime}]=[\min(t_{A},t_{B},t_{C})]$, the H-DIPP mechanism is strictly
truthful, i.e., all reviewers honestly reporting their true predictions and
private review scores on shared complete criteria $[t^{\prime}]$ is a strict
Bayesian Nash equilibrium.

However, mere truthfulness is not enough; the mechanism must also incentivize
reviewers to exert the necessary effort to complete all criteria. After all,
we care about eliciting reviewers’ _effort-informed signals_ on all criteria.
To achieve this, we reason about reviewers’ marginal payments and costs of
effort to determine mechanism hyperparameters in Appendix F.

Note that a reviewer’s expected expert payment is always non-positive (since
the Shannon entropy is non-negative), and a reviewer’s expected target payment
is always non-negative. We must ensure that a reviewer’s expected payment is
greater than their cost in the fully informative equilibrium, so that it is
individually rational for reviewers to participate in the mechanism. We can
achieve this by scaling the hyperparameters such that the target payment pays
more than the expert payment subtracts. This also grants that the fully-
informative equilibrium pays more than an uninformative equilibrium where no
reviewers exert effort. We state this formally below.

######  Theorem 2 (Uninformative Equilibria and Individual Rationality).

=

  1. (a)

In the highest paying uninformative equilibrium (where all reviewers exert
zero effort), every reviewer’s expected payment (and hence aggregate reviewer
welfare) is zero.

  2. (b)

Suppose we set mechanism hyperparameters so reviewers’ expected utility in the
fully-informative equilibrium is positive (see Theorem 3 in Appendix F). Then,
it is individually rational to participate in the mechanism and complete all
criteria and report truthfully, given peers are doing the same. Additionally,
the fully-informative equilibrium has higher individual and aggregate utility
than the uninformed equilibrium.

##  4 Related Work

There is an immense amount of scholarship on improving the peer review
process. Here, we discuss three bodies of related work: 1) proposals to
improve scientific peer review from computer science, 2) general proposals to
reform or replace scientific peer review, and 3) the information elicitation
without verification literature (Faltings and Radanovic,, 2017; Waggoner and
Chen,, 2013). We refer readers to Shah, (2022) for a comprehensive overview of
the challenges facing peer review today. Readers may also be interested in a
broader historical overview of the development of peer review (focused
primarily on the social sciences) presented by Merriman, (2020).

###  4.1 Proposals from Computer Science

A number of proposals to handle the rapid rise in submissions and reviewing
load have been put forward by the machine learning community. One idea is to
allow some editorial screening, where Area Chairs can reject submissions
without full reviews; this would eliminate the need for $\sim 4\%$ of reviews
but some papers may still be screened out unfairly. Another proposal is to
introduce submission caps, which limit the number of submissions by a single
author, but a cap of 10 submissions would again only eliminate $\sim 4\%$ of
the reviewing load. An analysis of NeurIPS 2019 data (NeurIPS,, 2019)
concluded that it was unclear how best to rapidly filter papers prior to full
review; our proposed submission mechanism uses a VCG auction to determine
which papers are accepted for review, where authors signal how ready their
work is and the mechanism accepts as many papers for review as it can find
quality reviewers for.

Criticizing the slow pace of peer review, LeCun, (2012) and Zhao, (2012)
propose an ‘open-market’, where all papers are uploaded to an online
repository as soon as they are ready for viewing. ‘Reviewing entities’ may
then identify papers they wish to review, although authors may request reviews
as well. However, data from past NeurIPS conferences (Shah et al.,, 2018;
NeurIPS,, 2019) suggest that under the current implementation, reviewers do
not make many eager bids to review papers, and eager bids are not predictive
of eventual acceptance. Thus, mechanisms that rely on reviewers to identify
and review papers may need to provide explicit incentives. Identifying the
tension between rapid dissemination of ideas and the need to carefully screen
results, Bengio, (2020) proposes a hybrid arXiv-journal model, where arXiv
addresses the former need and journals take on the role of slow, careful
screening of results to promote higher-quality publications. In these
aforementioned approaches, reviewing incentive issues would still persist (as
noted in fields with journal publication models); indeed, the reviewing burden
may move from conferences to the open-market system or journals, as authors
from less well-known backgrounds seeking community-issued certificates of the
quality of their work for reputation-building purposes begin to substitute
conference submissions for journal submissions. Our submission mechanism
disincentivizes weak submissions that add to the reviewing load while being
unlikely to be accepted, and our reviewing mechanism provides direct
incentives for honest and effortful reviews.

Another particular area of focus has been on improving the reviewer-paper
matching process (Stelmakh et al.,, 2023). Several recent works in this space
include a market-inspired approach assigning budgets and prices to incentivize
reviewers to bid for their favourite papers (Meir et al.,, 2021), optimizing
reviewer selection or assignment to obtain less noisy reviews (Shah,, 2019) or
better matching when the assignment is over multiple rounds (Jecmen et al.,,
2021), mitigating conflicts of interest (Kurokawa et al.,, 2015; Xu et al.,,
2018; Aziz et al.,, 2019; Jecmen et al.,, 2020), and determining the optimal
way to display papers to reviewers to elicit meaningful bids (Fiez et al.,,
2020). Other proposals involve improvements to various procedural aspects of
peer review, such as a recruitment and mentorship pipeline for junior
academics to alleviate the scarcity of reviewers (Stelmakh et al.,, 2020) and
better aggregation of subjective opinions on criteria scores to accept/reject
recommendations (Noothigattu et al.,, 2021). Such ideas are complementary to
our reviewing mechanism; they focus on improving paper-reviewer matching or
the acceptance decision process, while we focus on incentivizing honest and
effortful reviews after reviewers are assigned and before scores are
aggregated into accept/reject decisions.

Lastly, we mention relevant empirical evidence from the NeurIPS 2021 review
process (Rastogi et al.,, 2022) which finds that authors systematically
overestimate their paper’s probability of acceptance, and suggests genuine
uncalibration, non-response bias, and a potential desire to signal confidence
to reviewers as possible explanations. Verifying if these results hold even
when authors have explicit incentives to provide accurate forecasts would
inform the viability of our proposed submission stage.

###  4.2 General Proposals to Reform or Replace Peer Review

Most relevant to this paper is the empirical work by Chetty et al., (2014)
studying interventions to promote pro-social behavior in peer review setting.
They found that shorter deadlines, cash incentives, and social incentives all
effectively reduced review times, and notably, cash incentives did not
undermine intrinsic motivation to review. These interventions incurred minimal
costs and did not impact review acceptance rates, report quality, or review
times at other journals.

Another strain of research proposes modest changes to the system, such as
allowing reviewers to re-purpose reviews from past submissions (Alberts et
al.,, 2008) to reduce the demand for reviews, or improving the publication
culture so authors submit more complete and relevant work to the appropriate
venues instead of simply targeting the most prestigious venues. Others (Arns,,
2014) propose reducing the reviewing burden by allowing some papers (e.g. null
results) to be accepted without peer review. However, these proposals do not
systematically tackle the previously discussed core incentive issues.

Fox and Petchey, (2010) propose an approach similar to ours, where authors
must pay in an artificial currency to have their paper reviewed, and reviewers
are paid for their completed reviews. However, reviewers receive flat payment
for their participation, and so may not provide honest or effortful reviews.
Sculley et al., 2018a  propose pecuniary and non-pecuniary compensation for
reviewers based on review quality (as judged by a rubric), but rating reviews
would increase the workload on area chairs.

In a different approach, de la Rosa and Szymanski, 2007b ; de la Rosa and
Szymanski, 2007a  propose entirely _replacing the existing peer review system_
with market-based citation auctions, i.e., where earned citations are treated
as currency, every submission comes with a “citation bid", and the papers with
the highest bids are accepted for publication. The citations that the paper
goes on to earn are used to recoup the bet, with the idea being that the bid
will reveal how impactful (as measured by number of citations) a researcher
expects their work to be and the mechanism will publish papers expected to be
highly cited. However, even as this fixes the shortage of reviewers by getting
rid of the need for them in the first place, predicting citations is likely
extremely high variance, can lead to distorted citation practices, and poses
equity issues if highly-cited authors can publish any paper they want by
simply outbidding everyone else regardless of the quality of their work.

Prüfer and Zetland, (2010) also propose a citation-based paper submission
system for journals, although their proposal does not abolish academic peer
review. Instead, they propose a new currency of ‘academic dollars’ used by
_editors_ of various journals to bid for submitted papers in an auction; if
accepted for review, the dollar value is paid to the authors, referees, and
editors of _cited_ works. This is essentially a citation betting market for
journal editors; a paper’s bid will be commensurate with its future expected
citation count. Referees will exert effort to provide better feedback to
authors to improve the quality of their work in order to earn dollars from
future citations, and authors have a marginally higher incentive to write
better to earn citations. While an intriguing proposal, we suspect the task of
predicting citations is challenging and the singular focus on optimizing for
citations may distort citation practices.

Frijters and Torgler, (2019) also propose a modified peer review incorporating
market-based ideas, with a focus on mitigating ‘gate-keeping’ by insiders and
improving transparency. Their proposal includes allowing authors to bid for
reviews of varying quality, and opening up the reviews themselves to be rated
thereby allowing for ‘professionalization’ of the process. However, the system
is somewhat complicated and the market for rating reviews may not be very
liquid, especially in technical fields. Xiao et al., (2018) propose a peer
review mechanism to elicit effortful reviews by assigning papers to reviewers
based on the quality of the authors’ past reviews, but it computes this review
quality by asking authors to rate the reviews they received and assuming these
ratings to be honest. This is likely too strong an assumption in our setting
where authors may simply report how favourable a review was; this would
incentivize flattering and likely dishonest reviews, undermining the
mechanism.

###  4.3 Information Elicitation without Verification

The IEWV literature is focused on the problem of eliciting truthful private
‘signals’ when the mechanism designer is unable to verify the responses; this
may be because the ground truth is difficult or impossible for the mechanism
to access or there is simply no ground truth since the responses are
subjective. The key application domains of interest include tasks like
crowdsourcing data, product ratings, community sensing, and peer grading (Cai
et al.,, 2015; Radanovic et al.,, 2016; Dasgupta and Ghosh,, 2013; Kong and
Schoenebeck,, 2018). More recent work is also interested in ensuring that
private signals are obtained with effort. The typical setting in this
literature consists of risk-neutral, utility-maximizing agents who maintain
some prior distribution over ‘signals’ they can receive from a mechanism. When
the agents participate in the mechanism, the world reveals to them their
private information or ‘signal’, and they subsequently form posterior beliefs
over the distribution of signals observed by the other agents. In our setting,
this would be analogous to reviewers having some prior belief over the quality
of papers they may see, and when presented with a specific paper, the reviewer
forms their opinion of that particular paper and develops a posterior belief
over other reviewers’ assessments of the paper. The task of the mechanism
designer is to elicit agents’ private signals when verification is not
possible by providing incentives for truthful reports.

Two of the foundational approaches to this problem are the peer prediction
framework (Miller et al.,, 2005) and the Bayesian Truth Serum (BTS) (Prelec,,
2004). The fundamental idea is to design payments by reasoning about agents’
reports and posteriors over other agents’ reports in a way that makes truth-
telling a Bayesian Nash equilibrium. However, these mechanisms make strong
assumptions that are unlikely to hold in practice; subsequent works have
relaxed these assumptions (Witkowski and Parkes, 2012a, ; Witkowski and
Parkes,, 2013; Witkowski and Parkes, 2012b, ; Radanovic and Faltings,, 2013,
2014). There are also _output agreement_ mechanisms (Waggoner and Chen,, 2014)
with weaker assumptions, but these eschew truthfulness and seek to elicit
common knowledge. A directly relevant proposal to our work is by Carvalho et
al., (2013), which proposes a peer prediction-like mechanism for the same task
of academic peer review. They assume that reviewers’ signals on a given paper
are drawn from a multinomial distribution, and model reviewers’ prior over the
multinomials with the conjugate (Dirichlet) prior so inference is tractable,
but this a somewhat restrictive assumption. They further test their mechanism
on an experiment with Amazon Mechanical Turk workers, and find that their
mechanism does incentivize more accurate reviews and such reviews also produce
higher payments for the participating agents. The truthful mechanism
applicable in the most general setting is the recently proposed prediction
market-style Differential Peer Prediction (DPP) mechanism (Schoenebeck and
Yu,, 2020). The truth-telling equilibrium maximizes agent welfare, and the
mechanism is applicable to a small group of heterogeneous agents. Our proposed
reviewing mechanism is a direct extension of this work.

These aforementioned works do not incorporate any model of agent effort, and
instead assume that nature or some black box simply reveals the private signal
to the agents. The early literature incorporating effort models (Dasgupta and
Ghosh,, 2013; Witkowski et al.,, 2014; Radanovic et al.,, 2016; Liu and Chen,,
2016) is limited to the case of binary effort: either agents exert low/no
effort, or they exert high/full effort. A binary effort model may be
appropriate for quick tasks like those on crowdsourcing platforms, but
reviewing technical papers is a time-intensive process, so a more
sophisticated effort model is needed. Cai et al., (2015) propose a continuous
effort model for data elicitation, where exerting effort gives agents a lower
variance signal. However, in our setting, lower effort reviews may not only be
noisier, they may also be systematically biased (Kong and Schoenebeck,, 2018).
Gao et al., (2016) caution that in practice (and as highly relevant in our
case), agents have a variety of low-effort signals such as paper length,
topic, etc., potentially leading to low-effort coordination into uninformative
equilibria. Indeed, Gao et al., (2014) present empirical information showing
that agents can successfully coordinate into uninformative equilibria,
especially if these equilibria pay more than truthful equilibria.

With these considerations, Kong and Schoenebeck, (2018) propose a Hierarchical
Mutual Information Paradigm (HMIP) that constructs a hierarchy of criteria
that capture a range of cheap to expensive signals, and pays agents the
_information gained_ from more expensive signals. This work builds on the
Mutual Information Paradigm (Kong and Schoenebeck,, 2019) which shows that
mechanisms incentivize truth-telling when the payment is based on the mutual
information between agents’ responses, and this is what peer prediction and
BTS are attempting to do. We integrate the very general differential peer
prediction (DPP) mechanism into the hierarchical mutual information paradigm
(HMIP) to develop a strictly truthful prediction market-style mechanism that
incentivizes effortful reviews, with the desired equilibria paying more than
uninformative equilibria.

##  5 Discussion

We tackle incentive issues in scientific peer review by proposing a two-stage
mechanism that allocates review slots to papers via a VCG auction and uses the
revenue raised to pay participating reviewers in our novel H-DIPP mechanism.
The H-DIPP mechanism aims to incentivize reviewers to obtain and report a
clearer signal of a paper’s quality in order to reduce the noise in the
accept/reject decisions. We showed that the H-DIPP mechanism is strictly
truthful, and payoffs can be designed so that exerting enough effort to score
the paper honestly on all criteria is a strict Bayesian Nash equilibrium. We
also showed that such a fully informative equilibrium pays more in expectation
than any uninformative equilibrium, and described how the mechanism
hyperparameters can be further tweaked so participating in the H-DIPP
reviewing mechanism is individually rational. However, there are many
important considerations for a practical implementation of our mechanism, some
of which we discuss below (see Appendix H for concrete suggestions for future
work).

##### Fairness and Access Considerations

Our mechanism requires that each submitted paper is accompanied by a bid from
a single author. In practice, papers are largely multi-author efforts, so the
question of who pays the bid naturally arises. Not every co-author is equally
qualified to participate in the review process; research and review may
require different skills. Furthermore, new or under-resourced authors may not
be able to afford strong bids555A paper should only have a single bid sponsor;
allowing multiple sponsors to contribute to a bid could unfairly privilege
papers with more authors or those from larger research groups. For the same
reason, credits should not be freely giftable to others. as they may not have
been able to build up a store of credits from reviewing. Thus, we will need a
few interventions to ensure the proposed mechanism does not become an undue
barrier to publishing academic research. One suggestion is to set a norm that
the bid sponsor typically be a senior author who has banked enough credits by
reviewing. We may also wish to allow non-author researchers to sponsor the
credits for a paper’s accompanying bid and be acknowledged in their capacity
as sponsor. Additionally, we suggest granting a lump-sum of credits to first-
time or junior authors to give them the right to submit some number of papers
in a given time period while they build the necessary expertise for review.

##### Common Prior Assumption

We assumed that reviewers’ review scores are drawn from a (potentially
asymmetric) common prior that is common knowledge, a standard assumption in
the IEWV literature. This assumption is important to obtain the equilibrium
results, but the extent to which it holds in practice needs to be answered
empirically. This assumption can perhaps be made more realistic by making
examples or summary statistics of a reviewer’s past reviews available for
reference (so any reviewer can gauge how their opinion correlates with their
peers’) .

##### Post-processing Review Scores

The H-DIPP mechanism elicits honest and effortful opinions from
_heterogeneous_ reviewers, with the goal of reducing the noise in estimates of
a paper’s quality. However, this also means that reviewers’ honest and
effortful opinions may be miscalibrated with respect to the rating scale (Shah
et al.,, 2018). Thus, some post-processing of these scores (Wang and Shah,,
2018; Cortes and Lawrence,, 2021) to correct reviewer biases may produce a
better picture of a paper’s quality.

##### Conflicts of Interest

We note that our results on incentivizing effortful and truthful reviews hold
under the assumption that there are no other incentives in play. However, if
the matched reviewers have a conflict of interest, this would weaken the
incentive to report honest and effortful reviews. Thus, related work on
optimizing paper-reviewer matches (Xu et al.,, 2018; Jecmen et al.,, 2020) to
avoid conflicts of interest is complementary and important to the success of
our mechanism.

#####  A Secondary Market

We have assumed that our mechanism operates with its own currency, but we
could further allow the sale and purchase of credits on a secondary market.
This has the potential to allow individuals to earn real-world currency as
reviewers, and could even encourage specialization in the review process:
researchers who excel at reviewing in H-DIPP mechanism could allocate more of
their time providing high quality reviews and be appropriately compensated for
their work.

##### Behavioural and Social Considerations

Our focus in this paper has been to develop a mechanism for paper submission
and reviewing with desirable theoretical properties. However, our fundamental
assumptions about human behaviour may or may not hold in practice. A primary
concern is that introducing payments for reviewers could crowd-out pro-social
motivations to review (Squazzoni et al.,, 2013; Gneezy and Rustichini,, 2000);
Chetty et al., (2014) empirically study precisely this question and find that
“cash payments do not crowd out intrinsic motivation" and have “little or no
effect on rates of agreement to review, quality of reports, or review times at
other journals". Further empirical evidence for when such peer prediction
mechanisms in particular succeed (Debmalya et al.,, 2020) and fail (Gao et
al.,, 2014) can help refine our mechanism and similar ideas.

Such issues merit further in-depth discussion and analysis. Our proposed
mechanism is intended to serve as a theoretically grounded approach to
tackling incentive issues in academic peer review, and we expect that a
practical implementation of our proposed mechanism will require ad-hoc
changes. Ultimately, experimental validation will be the final arbiter of the
plausibility of our approach.

## Acknowledgments and Disclosure of Funding

We thank Sandesh Adhikary for assistance with figures in the paper.

## References

  * Agarwal et al., (2017) Agarwal, A., Mandal, D., Parkes, D. C., and Shah, N. (2017).  Peer prediction with heterogeneous users.  In Proceedings of the 2017 ACM Conference on Economics and Computation, pages 81–98.
  * Alberts et al., (2008) Alberts, B., Hanson, B., and Kelner, K. L. (2008).  Reviewing peer review.
  * Arns, (2014) Arns, M. (2014).  Open access is tiring out peer reviewers.  Nature, 515(7528):467–467.
  * Aziz et al., (2019) Aziz, H., Lev, O., Mattei, N., Rosenschein, J. S., and Walsh, T. (2019).  Strategyproof peer selection using randomization, partitioning, and apportionment.  Artificial Intelligence, 275:295–309.
  * Bengio, (2020) Bengio, Y. (2020).  Time to rethink the publication process in machine learning.
  * Cai et al., (2015) Cai, Y., Daskalakis, C., and Papadimitriou, C. (2015).  Optimum statistical estimation with strategic data sources.  In Conference on Learning Theory, pages 280–296.
  * Carvalho et al., (2013) Carvalho, A., Dimitrov, S., and Larson, K. (2013).  Inducing honest reporting without observing outcomes: An application to the peer-review process.  arXiv preprint arXiv:1309.3197.
  * Charlin and Zemel, (2013) Charlin, L. and Zemel, R. (2013).  The toronto paper matching system: an automated paper-reviewer assignment system.
  * Chetty et al., (2014) Chetty, R., Saez, E., and Sándor, L. (2014).  What policies increase prosocial behavior? an experiment with referees at the journal of public economics.  Journal of Economic Perspectives, 28(3):169–188.
  * Cortes and Lawrence, (2021) Cortes, C. and Lawrence, N. D. (2021).  Inconsistency in conference peer review: Revisiting the 2014 neurips experiment.  arXiv preprint arXiv:2109.09774.
  * Dasgupta and Ghosh, (2013) Dasgupta, A. and Ghosh, A. (2013).  Crowdsourced judgement elicitation with endogenous proficiency.  In Proceedings of the 22nd international conference on World Wide Web, pages 319–330.
  * (12) de la Rosa, J. L. and Szymanski, B. K. (2007a).  Citation auctions as a method to improve selection of scientific papers.  In 2007 2nd International Conference on Digital Information Management, volume 1, pages 479–486. IEEE.
  * (13) de la Rosa, J. L. and Szymanski, B. K. (2007b).  Selecting scientific papers for publication via citation auctions.  IEEE Intelligent Systems, 22(6):16–20.
  * Debmalya et al., (2020) Debmalya, M., Goran, R., and David, P. (2020).  The effectiveness of peer prediction in long-term forecasting.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2160–2167.
  * Faltings and Radanovic, (2017) Faltings, B. and Radanovic, G. (2017).  Game theory for data science: Eliciting truthful information.  Synthesis Lectures on Artificial Intelligence and Machine Learning, 11(2):1–151.
  * Fiez et al., (2020) Fiez, T., Shah, N., and Ratliff, L. (2020).  A super* algorithm to optimize paper bidding in peer review.  In Conference on Uncertainty in Artificial Intelligence, pages 580–589. PMLR.
  * Fox and Petchey, (2010) Fox, J. and Petchey, O. L. (2010).  Pubcreds: fixing the peer review process by “privatizing” the reviewer commons.  The Bulletin of the Ecological Society of America, 91(3):325–333.
  * Frank, (2016) Frank, R. H. (2016).  Success and Luck.  Princeton University Press.
  * Frijters and Torgler, (2019) Frijters, P. and Torgler, B. (2019).  Improving the peer review process: a proposed market system.  Scientometrics, 119(2):1285–1288.
  * Gao et al., (2016) Gao, A., Wright, J. R., and Leyton-Brown, K. (2016).  Incentivizing evaluation via limited access to ground truth: Peer-prediction makes things worse.  arXiv preprint arXiv:1606.07042.
  * Gao et al., (2014) Gao, X. A., Mao, A., Chen, Y., and Adams, R. P. (2014).  Trick or treat: putting peer prediction to the test.  In Proceedings of the fifteenth ACM conference on Economics and computation, pages 507–524.
  * Gneezy and Rustichini, (2000) Gneezy, U. and Rustichini, A. (2000).  Pay enough or don’t pay at all.  The Quarterly journal of economics, 115(3):791–810.
  * Jecmen et al., (2021) Jecmen, S., Zhang, H., Liu, R., Fang, F., Conitzer, V., and Shah, N. B. (2021).  Near-optimal reviewer splitting in two-phase paper reviewing and conference experiment design.  arXiv preprint arXiv:2108.06371.
  * Jecmen et al., (2020) Jecmen, S., Zhang, H., Liu, R., Shah, N., Conitzer, V., and Fang, F. (2020).  Mitigating manipulation in peer review via randomized reviewer assignments.  Advances in Neural Information Processing Systems, 33:12533–12545.
  * Kong and Schoenebeck, (2018) Kong, Y. and Schoenebeck, G. (2018).  Eliciting expertise without verification.  In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 195–212.
  * Kong and Schoenebeck, (2019) Kong, Y. and Schoenebeck, G. (2019).  An information theoretic framework for designing information elicitation mechanisms that reward truth-telling.  ACM Transactions on Economics and Computation (TEAC), 7(1):1–33.
  * Kurokawa et al., (2015) Kurokawa, D., Lev, O., Morgenstern, J., and Procaccia, A. D. (2015).  Impartial peer review.  Citeseer.
  * LeCun, (2012) LeCun, Y. (2012).  A new publishing model in computer science.
  * Lipton and Steinhardt, (2018) Lipton, Z. C. and Steinhardt, J. (2018).  Troubling trends in machine learning scholarship.  arXiv preprint arXiv:1807.03341.
  * Liu and Chen, (2016) Liu, Y. and Chen, Y. (2016).  Learning to incentivize: Eliciting effort via output agreement.  arXiv preprint arXiv:1604.04928.
  * Meir et al., (2021) Meir, R., Lang, J., Lesca, J., Mattei, N., and Kaminsky, N. (2021).  A market-inspired bidding scheme for peer review paper assignment.  In 35th AAAI Conference on Artificial Intelligence.
  * Merriman, (2020) Merriman, B. (2020).  Peer review as an evolving response to organizational constraint: Evidence from sociology journals, 1952–2018.  The American Sociologist, pages 1–26.
  * Miller et al., (2005) Miller, N., Resnick, P., and Zeckhauser, R. (2005).  Eliciting informative feedback: The peer-prediction method.  Management Science, 51(9):1359–1373.
  * NeurIPS, (2019) NeurIPS (2019).  What we learned from neurips 2019 data.  Medium.
  * Noothigattu et al., (2021) Noothigattu, R., Shah, N., and Procaccia, A. (2021).  Loss functions, axioms, and peer review.  Journal of Artificial Intelligence Research, 70:1481–1515.
  * Prelec, (2004) Prelec, D. (2004).  A bayesian truth serum for subjective data.  science, 306(5695):462–466.
  * Prüfer and Zetland, (2010) Prüfer, J. and Zetland, D. (2010).  An auction market for journal articles.  Public Choice, 145(3-4):379–403.
  * Radanovic and Faltings, (2013) Radanovic, G. and Faltings, B. (2013).  A robust bayesian truth serum for non-binary signals.  In Proceedings of the 27th AAAI Conference on Artificial Intelligence (AAAI’13), number CONF, pages 833–839.
  * Radanovic and Faltings, (2014) Radanovic, G. and Faltings, B. (2014).  Incentives for truthful information elicitation of continuous signals.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 28.
  * Radanovic et al., (2016) Radanovic, G., Faltings, B., and Jurca, R. (2016).  Incentives for effort in crowdsourcing using the peer truth serum.  ACM Transactions on Intelligent Systems and Technology (TIST), 7(4):1–28.
  * Rastogi et al., (2022) Rastogi, C., Stelmakh, I., Beygelzimer, A., Dauphin, Y. N., Liang, P., Vaughan, J. W., Xue, Z., Daumé III, H., Pierson, E., and Shah, N. B. (2022).  How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions?  arXiv preprint arXiv:2211.12966.
  * Schoenebeck and Yu, (2020) Schoenebeck, G. and Yu, F.-Y. (2020).  Two strongly truthful mechanisms for three heterogeneous agents answering one question.  In International Conference on Web and Internet Economics, pages 119–132. Springer.
  * (43) Sculley, D., Snoek, J., and Wiltschko, A. (2018a).  Avoiding a tragedy of the commons in the peer review process.  arXiv preprint arXiv:1901.06246.
  * (44) Sculley, D., Snoek, J., Wiltschko, A., and Rahimi, A. (2018b).  Winner’s curse? on pace, progress, and empirical rigor.
  * Shah, (2019) Shah, N. B. (2019).  Principled methods to improve peer review.  Retrieved May, 27:2020.
  * Shah, (2022) Shah, N. B. (2022).  Challenges, experiments, and computational solutions in peer review.  Communications of the ACM, 65(6):76–87.
  * Shah et al., (2018) Shah, N. B., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018).  Design and analysis of the nips 2016 review process.  Journal of machine learning research.
  * Squazzoni et al., (2013) Squazzoni, F., Bravo, G., and Takács, K. (2013).  Does incentive provision increase the quality of peer review? an experimental study.  Research Policy, 42(1):287–294.
  * Stelmakh et al., (2020) Stelmakh, I., Shah, N. B., Singh, A., and Daumé III, H. (2020).  A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences.  arXiv preprint arXiv:2011.15050.
  * Stelmakh et al., (2023) Stelmakh, I., Wieting, J., Neubig, G., and Shah, N. B. (2023).  A gold standard dataset for the reviewer assignment problem.  arXiv preprint arXiv:2303.16750.
  * Vazire, (2017) Vazire, S. (2017).  Quality uncertainty erodes trust in science.  Collabra: Psychology, 3(1).
  * Waggoner and Chen, (2013) Waggoner, B. and Chen, Y. (2013).  Information elicitation sans verification.  In Proceedings of the 3rd Workshop on Social Computing and User Generated Content (SC13).
  * Waggoner and Chen, (2014) Waggoner, B. and Chen, Y. (2014).  Output agreement mechanisms and common knowledge.  In HCOMP.
  * Wang and Shah, (2018) Wang, J. and Shah, N. B. (2018).  Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings.  arXiv preprint arXiv:1806.05085.
  * Witkowski et al., (2014) Witkowski, J., Bachrach, Y., Key, P., and Parkes, D. C. (2014).  Dwelling on the negative: Incentivizing effort in peer prediction.
  * (56) Witkowski, J. and Parkes, D. C. (2012a).  Peer prediction without a common prior.  In Proceedings of the 13th ACM Conference on Electronic Commerce, pages 964–981.
  * (57) Witkowski, J. and Parkes, D. C. (2012b).  A robust bayesian truth serum for small populations.
  * Witkowski and Parkes, (2013) Witkowski, J. and Parkes, D. C. (2013).  Learning the prior in minimal peer prediction.  In Proceedings of the 3rd Workshop on Social Computing and User Generated Content at the ACM Conference on Electronic Commerce, volume 14.
  * Xiao et al., (2018) Xiao, Y., Dörfler, F., and Van Der Schaar, M. (2018).  Incentive design in peer review: Rating and repeated endogenous matching.  IEEE Transactions on Network Science and Engineering, 6(4):898–908.
  * Xu et al., (2018) Xu, Y., Zhao, H., Shi, X., Zhang, J., and Shah, N. B. (2018).  On strategyproof conference peer review.  arXiv preprint arXiv:1806.06266.
  * Zhao, (2012) Zhao, S. (2012).  Two proposals for a new publishing and reviewing system in computer science.

##  Appendix A Table of Notation

Symbol | Description
---|---
$N$ | Number of papers submitted to the mechanism
$K$ | Number of agents participating in the mechanism
$T$ | Number of criteria reviewers must evaluate a paper on
$i$ | Index for a paper
$k$ | Index for a reviewer
$t$ | Index for a criterion
$\omega_{i}$ | The $i$-th paper submitted to the mechanism
$s_{k}^{(\cdot)}$ | Agent $k$’s Stage I/II strategy
$s_{k}$ | Agent $k$’s full Stage I and Stage II strategy
${\bf s}$ | Full strategy profile of all agents on Stage I and Stage II
${\bf s}_{-k}$ | Strategy profile of all agents except agent $k$
$U_{k}^{(\cdot)}$ | Agent $k$’s Stage I/II utility
$U_{k}$ | Agent $k$’s total utility
$b_{i}$ | The bid accompanying paper $\omega_{i}$ for a review slot
$c_{i}$ | The price charged for accepting paper $\omega_{i}$ for review
$b^{*}_{p}$ | The $p$-th highest bid
$\omega^{*}_{p}$ | The paper with the $p$-th highest bid
$\delta_{i}$ | Boolean denoting whether paper $\omega_{i}$ is accepted _for publication_ after review
$\eta_{i}$ | Probability that paper $\omega_{i}$ is accepted _for publication_ conditional on review
$v_{k}$ | Agent $k$’s value for a paper accepted for publication
$P$ | The number of papers accepted for review
$A,B,C$ | The reviewer triplet assigned to a given a paper
$\Sigma_{t}$ | Set of allowed review scores on criterion $t$
$e_{k}(t)$ | Reviewer $k$’s cost for completing up to $t$ criteria
$\mathds{P}$ | A reviewer triplet’s common prior over effort-informed scores
$\mathds{Q}$ | A reviewer triplet’s joint distribution over reported scores and effort-informed scores
${\bf X}_{k}^{[t]}$ | Reviewer $k$’s true review scores on criteria $1,\ldots,t$
$g_{k}^{t^{\prime}|t}({\bf x}_{k}^{[t]})$ | Reviewer $k$’s ‘best guess’ of their signal on criterion $t^{\prime}$ given signals ${\bf x}_{k}^{[t]}$ (where $t^{\prime}>t$)
$\hat{\bf X}_{k}^{[T]}$ | Reviewer $k$’s (potentially strategically) reported review scores on criteria $1,\ldots,T$
$\theta_{k}$ | Transition matrix representing reviewer $k$’s signal reporting strategy
$\hat{\bf p}_{k^{\prime[T]}\leftarrow k}$ | Probability distribution of reviewer $k$’ first prediction of $k^{\prime}$’s scores on criteria $1,\ldots,T$
$\hat{\bf p}^{+}_{k^{\prime[T]}\leftarrow k}$ | Probability distribution of reviewer $k$’s second prediction of $k^{\prime}$’s scores on criteria $1,\ldots,T$
$r_{k}$ | Reviewer $k$’s payment for reviewing
$R_{k}(t)$ | Reviewer $k$’s expected payment for completing $t$ criteria when peers complete $T$ criteria and report truthfully
$R_{k}^{\text{role}}(t)$ | Reviewer $k$’s expected expert/target payment for completing $t$ criteria when peers complete $T$ criteria and report truthfully
$R_{k^{t^{\prime}}}^{\text{role}}(t)$ | Reviewer $k$’s expected expert/target payment _on criterion $t^{\prime}$_ for completing $t$ criteria when peers complete $T$ criteria and report truthfully
$\Delta R_{k}(t)$ | Reviewer $k$’s marginal expected payment for completing criterion $t$ when peers complete $T$ criteria and report truthfully
$\Delta R_{k}^{\text{role}}(t)$ | Reviewer $k$’s marginal expected expert/target payment for completing criterion $t$ when peers complete $T$ criteria and report truthfully
$\Delta R_{k^{t^{\prime}}}^{\text{role}}(t)$ | Reviewer $k$’s marginal expected expert/target payment _on criterion $t^{\prime}$_ for completing criterion $t$ when peers complete $T$ criteria and report truthfully
$S_{\text{log}}(\cdot,\cdot)$ | Log proper scoring rule
$I(\cdot)$ | Mutual information

##  Appendix B The 2014 NeurIPS Experiment

In 2014, program chairs of the NeurIPS conference sought to study the
consistency of the peer review process (Cortes and Lawrence,, 2021), and set
up an experiment where 10% of submitted papers underwent review by two
independent committees. We describe the major findings below:

  * •

Inconsistency of Reviews: About 26% of papers received different outcomes from
the two committees (compared to 37.5% expected from random acceptance).

  * •

Accept Precision: Of the accepted papers, only about 50% of papers would have
survived re-review by an independent committee (compared to 25% expected from
random acceptance).

  * •

Reject Precision: Of the rejected papers, about 80% would have also been
rejected in a re-review by an independent committee (compared to 75% expected
from random rejection).

These findings showed that the _accept/reject_ decisions were moderately
better than pure chance, although still a fair bit noisy. Some of this is
unavoidable in any highly selective mechanism seeking to allocate a limited
number of spots to a large number of candidates on the basis of some noisily
observed measure of ‘quality’ (Frank,, 2016). When a mechanism seeks to accept
some small percentage of top quality candidates, even a small amount of noise
in observing the quality of a candidate can add a great deal of randomness to
which candidates make the cut. Yet it will be the case that the bulk of the
randomness will be faced by the candidates clustered around the cutoff (‘messy
middle’); candidates well above the cutoff will likely be consistently
accepted, and candidates well below the cutoff will likely be consistently
rejected. Thus, the noise in observations of paper quality is an important
factor affecting the randomness faced by authors in a highly selective
acceptance process. Shah et al., (2018) find that about $45\%$ of NeurIPS 2015
papers and $30\%$ of NeurIPS 2016 papers are in ‘messy middle’ where ‘inter-
reviewer agreements are near random’. Back in the 2014 NeurIPS experiment, an
analysis of the independent committees’ ratings of paper quality, finds the
following:

  * •

Correlation of Quality Scores: The correlation between calibrated ‘quality’
scores assigned by the two independent committees (adjusted for a reviewers’
systematic biases) was 0.546.

  * •

Citation Impact: For accepted papers, there was little correlation (0.051)
between reviewers’ assessments of paper quality and papers’ log-citation count
(likely due to the restricted range effect). For rejected papers, there was a
positive correlation (0.22) between reviewers’ assessments of paper quality
and papers’ log citation count.

The finding that about half the variance in scores is intrinsic to the paper
(‘objective’) and half is idiosyncratic to the reviewer (‘subjective’) (Cortes
and Lawrence,, 2021) provides more information on the noise in observations.
This subjectivity can arise from more readily identifiable sources like
differences in effort or expertise, or from truly subjective differences of
perspective. Naturally, if reviewers exert different levels of effort or have
differing levels of expertise, their reviews may be systematically different
from each other and would weaken the ‘objective’ proportion of review scores.
The correlation between quality scores and impact also suggests that the
objective proportion of review scores has some better-than-random ability to
predict the long-term impact of papers as measured by citations (relevant for
criticism C4). In sum, the above results on accept/reject decisions as well as
reviewers’ quality scores paint a picture of a peer review system that
performs noticeably better than random, but with a lot of room for
improvement.

These results provide a baseline against which to judge the performance of our
proposed mechanism. The goal then is for our proposed mechanism to incentivize
reviewers to coordinate into a high-effort equilibrium where the
‘subjectivity’ of reviewers’ scores is lower, the correlation between
independent reviewers’ quality scores is higher, and the messy middle is
smaller. The hypothesis is that the provision of incentives for effort _during
the review process_ may help reviewers better observe the quality of papers,
and reduce the variance of decisions under the current _selective acceptance
process_.

Finally, we note that conference selectiveness serves the dual purposes of
focusing the community’s attention on major results (motivation M2), as well
as building reputation for authors who do high-quality work (motivation M3).
While Cortes and Lawrence, (2021) argues against using conference publications
as a proxy for research quality or career advancement, decision-makers looking
to make hiring decisions or determine what research to focus on will
implicitly or explicitly be seeking relevant signals; in the absence of an
alternative signal, we argue that it is important to ensure a peer-reviewed
publication gives an appropriate signal of quality.

##  Appendix C Key elements of the Mutual Information Paradigm (Kong and
Schoenebeck,, 2019)

A scoring rule $S:\Sigma\times\Delta_{\Sigma}\rightarrow\mathds{R}$ is a
function that accepts some outcome $\sigma$ (from some signal space $\Sigma$)
and a probability distribution $\delta$ (representing a forecast over that
signal space $\Delta_{\Sigma}$) and scores the forecast with a real number.
The scoring rule is _proper_ if whenever the outcome $\sigma$ is drawn from a
distribution $\delta\in\Delta_{\Sigma}$, the expected score
$\mathds{E}_{\sigma}[S(\sigma,\cdot)]$ is maximized when the forecast is
$\delta$. The scoring rule is _strictly proper_ if this maximum is unique. Our
mechanism will use the log scoring rule
$S_{\log}(\sigma,\delta)=\ln\delta(\sigma)$, which is a strictly proper
scoring rule. Next, recall that the Shannon Mutual Information of two random
variables is defined as the KL divergence of the product of their marginal
distributions from the joint distribution:
$I(X;Y)=D_{\text{KL}}(P(X,Y)||P(X)\otimes P(Y))$. The conditional Shannon
mutual information is given as
$I(X;Y|Z)=\sum_{z}P(z)D_{\text{KL}}(P(X,Y|z)||P(X|z)\otimes
P(Y|z))=\sum_{z}P(z)I(X;Y|z)$. Kong and Schoenebeck, (2019) show an
interesting connection between conditional Shannon mutual information and the
log scoring rule:

$I(X;Y|Z)=\mathds{E}_{X,Y,Z}[S_{\text{log}}(Y,P(Y|X,Z))-S_{\text{log}}(Y,P(Y|Z))]$
(4)

Finally, we present the important data processing inequality, typically useful
in proving that deviating from truth-telling decreases payment. The inequality
states that whenever a signal $Y$ is manipulated in a way independent of
another signal $X$, the manipulated signal can only have lower mutual
information with $X$.

######  Lemma 1 (Strict Data Processing Inequality (Schoenebeck and Yu,,
2020)).

If $(X,Y)$ on a finite space $\mathcal{X}\times\mathcal{Y}$ is stochastic
relevant (i.e., for any distinct $x,x^{\prime}\in\mathcal{X}$, $P(Y|x)\neq
P(Y|x^{\prime})$) and has full support, then for any random function
$\theta:\mathcal{Y}\rightarrow\mathcal{Y}$ where the randomness is independent
of (X, Y), we have that $I(X;\theta(Y))\leq I(X;Y)$. We have equality if and
only if $\theta$ is an injective map.

######  Lemma 2 (Strict Conditional Data Processing Inequality (Schoenebeck
and Yu,, 2020)).

If $(W,X,Y)$ on a finite space $\mathcal{W}\times\mathcal{X}\times\mathcal{Y}$
is second-order stochastic relevant (i.e., for any distinct
$x,x^{\prime}\in\mathcal{X}$, there is some $w\in\mathcal{W}$ such that
$P(Y|w,x)\neq P(Y|w,x^{\prime})$) and has full support, then for any random
function $\theta:\mathcal{Y}\rightarrow\mathcal{Y}$ where the randomness is
independent of (W, X, Y), we have that $I(X;\theta(Y)|W)\leq I(X;Y|W)$. We
have equality if and only if $\theta$ is an injective map.

These lemmas are the cornerstone of the mutual information paradigm, as they
assert that whenever a mechanism pays an individual proportional to the mutual
information between their signal and a peer’s, the individual is incentivized
to be truthful. Any strategy that involves lying, i.e., post-processing the
signal, will only result in lower reward, and is hence disincentivized. In
this work, our result on truthfulness of the review mechanism under our
payment scheme (Theorem 1) relies on a similar argument. We also note that a
reviewer’s review-based payment (i.e., their ‘target’ payment) is a
conditional mutual information _in every equilibrium_.

##  Appendix D Relationship to HMIM and TDPP

Our two main considerations are incentivizing reviewers to exert effort, and
to report their true reviews. We achieve both by drawing on ideas from the
Target Differential Peer Prediction (TDPP) (Schoenebeck and Yu,, 2020) and
Hierarchical Mutual Information Paradigm (HMIP) (Kong and Schoenebeck,, 2018).
The Differential Peer Prediction mechanism is a recently proposed prediction
market-style mechanism applicable to our small-group, heterogeneous reviewer
setting and determines payments for reviews of each paper independently. They
present their mechanism for a group of 3 agents, ideal for our peer review
setting. The mechanism works as follows: first, it assigns the 3 agents the
roles of _expert_ , _target_ , and _source_. Then, it asks the _expert_ for a
probability distribution representing their prediction of the _target’s_
signal. Finally, it reveals the _source’s_ signal and asks the expert for an
updated prediction of the target’s signal. Schoenebeck and Yu, (2020) show
that the total expected payment to agents is the mutual information of the
source and target’s signals conditioned on the expert’s signals, and this
yields a strong truthfulness guarantee (Kong and Schoenebeck,, 2019).

However, truthfulness is not enough; reporting low-effort signals can still be
truthful. Additionally, since these truthful signals are paid their mutual
information, when low-effort signals correlate more than high-effort signals,
it becomes natural to coordinate into less informative equilibria. HMIP
proposes a framework to get around this issue by constructing a hierarchy of
criteria that range from requiring low-effort to high-effort signals, and
paying agents the _gain_ in information on higher level criteria. Single-HMIM
is an instantiation of the HMIP framework to incentivize effortful reviews on
a single task, but uninformative equilibria can pay more than truthful
equilibria and we seek to overcome this.

We combine the insights from TDPP and HMIP to propose a Hierarchical-
Differential Peer Prediction (H-DIPP) mechanism.

###  D.1 Relationship to HMIM

Here we discuss how our mechanism differs from the single-HMIM proposal under
the HMIP framework (Kong and Schoenebeck,, 2018).

  * •

Compulsory Reports: Unlike single-HMIM, reviewers must provide scores for all
criteria, not just the ones they have completed. This is currently the norm in
peer review, and we would like to design the mechanism around this
expectation. We model reviewers’ alternative to effortful reports as ‘best
guess’ reports, as opposed to HMIM where a reviewer who does not complete a
criterion does not report any signal and simply forgoes any payment on it.
Realistically, a reviewer’s signals on completed criteria may enable them to
make reasonable guesses on higher level criteria, and we account for this in
computing reviewers’ marginal payments and setting hyperparameters.

  * •

Partial Vs Total Order: Kong and Schoenebeck, (2018) only impose a partial
order on their criteria hierarchy to keep it general; we impose a total order
for ease of exposition.

  * •

Different Payment Scheme: While we adopt the hierarchical criteria structure
from HMIP, the actual payments we provide are adopted from TDPP. However, at
equilibrium, a reviewer’s expected target payment is almost the mutual
information payment scheme proposed in HMIP, under our conditional
independence assumption (Assumption 2). The difference is that while HMIP
(under the Conditional Independence assumption) would compute the mutual
information of the target’s and source’s signals on criterion $t$ conditioned
on the source’s signals on lower level criteria, our mechanism also conditions
this on the expert’s private information.

  * •

Strategy and Equilibria: Our mechanism, like single-HMIM, is strictly
truthful. The ‘idealized’ HMIP has the desirable property that truthful
reporting is a dominant strategy, but the practical single-HMIM algorithm does
not have this property. Truthful reports are not a dominant strategy in our
H-DIPP mechanism either, since target payments depend on the prediction
strategy of experts. However, it is the case that a reviewer is paid greater
target payments in truth-telling equilibria, and the use of proper scoring
rules incentivizes experts to predict truthfully if targets deviate to truth-
telling.

###  D.2 Relationship to TDPP

Here we discuss how our mechanism differs from the TDPP proposal (Schoenebeck
and Yu,, 2020).

  * •

Playing All Roles: TDPP assigns each agent a single role (expert, target, or
source). In our mechanism, every agent plays all three roles.

  * •

Effort Model: TDPP assumes that signals are revealed to agents at no cost. We
assume that the effort level determines the signals that are revealed to
agents, so we adopt the hierarchical approach of HMIP to incentivize the
effort needed to respond to higher level criteria.

  * •

Knowledge of Common Prior: TDPP and other ‘detail-free’ mechanisms assume no
knowledge of the common prior. While we do not strictly need to know a
reviewer triplet’s common-prior to run the mechanism, we would like to know
certain properties (the marginal payments from Lemma 3) of the common prior
(as in HMIP) so that we can set the mechanism hyperparameters to incentivize
the desired fully informative equilibrium.

  * •

Multi-criteria Predictions and No Source Payment: TDPP only elicits source and
target responses on a single question, while H-DIPP elicits responses to a
hierarchy of criteria. H-DIPP reveals the source’s signal after each criterion
(as opposed to revealing it all at once, or revealing target signals, etc.) as
this brings the expected target payment close to the HMIP recommendation.
However, this does mean that unlike TDPP, we do not provide any source payment
for the following reason. The TDPP expected source payment was based on the
expert $k$’s first prediction of target $k^{\prime}$’s signals on a criterion
$t$ (made without the source’s information). This term did not change the
source’s incentives in the original TDPP (as their actions could not affect
the expert’s first prediction), but it was a convenient accounting trick to
make the ex ante agent welfare equal to the conditional mutual information of
the source and target’s reports. However, we cannot incorporate such a source
payment in H-DIPP: the expert makes predictions with the source’s signals on
lower level criteria, so _the source can influence the expert’s first
prediction on every criterion $t>1$_. Thus, including such a source payment
could lead the source to strategically misreport so that the expert’s first
predictions are poor. Naturally, leaving out source payments preserves
source’s incentives to report truthfully, but the ex ante reviewer welfare is
no longer the conditional mutual information. This is why there are ‘extra’
entropy terms in Lemma 4.

  * •

Strategy and Equilibria: Like TDPP, our H-DIPP mechanism is strictly truthful.
However, TDPP has the additional very desirable property of being _strongly
truthful_ , i.e., the truth-telling equilibrium pays more than all other
equilibria, so agents have no reason to coordinate into other equilibria.
Owing to the multi-round structure of H-DIPP, we cannot quite make the same
guarantee; it _may_ be possible to have equilibria where reviewers
strategically misreport to increase a peer’s target payment by more than the
loss of their target payment thereby increasing aggregate payment, although
there is no individual incentive to do so. It may be possible for all
reviewers to do this so their individual and aggregate payments are higher
than the fully informative equilibrium, although accepting the increased
target payment bestowed by a source’s strategic misreports and _not
reciprocating_ by reporting truthfully would lead to a higher-paying
equilibrium for the target. Investigating such equilibria is an important
direction for future work. Nevertheless, we can make the weaker claim that the
fully informative equilibrium pays more than an effortless uninformative
equilibrium (Theorem 2).

##  Appendix E Other Assumptions

We discuss some of the other assumptions underlying our H-DIPP review
mechanism (Stage II) here.

##### Strategic and Uninformed Reports

Our mechanism will elicit reviewers’ private review scores $X_{k}^{t}$ on all
criteria, and we denote the (potentially strategic) _reported review score_ as
$\hat{X}_{k}^{t}$. With $\theta_{k}$ as the transition matrix describing
reviewer $k$’s reporting strategy, the joint distribution of true signals and
reported signals under a signal strategy profile
$(\theta_{A},\theta_{B},\theta_{C})$ is $\mathds{Q}(\hat{\bf
X}_{A}^{[T]},\hat{\bf X}_{B}^{[T]},\hat{\bf X}_{C}^{[T]},{\bf
X}_{A}^{[T]},{\bf X}_{B}^{[T]},{\bf X}_{C}^{[T]})=\theta_{A}(\hat{\bf
X}_{A}^{[T]}|{\bf X}_{A}^{[T]})\theta_{B}(\hat{\bf X}_{B}^{[T]}|{\bf
X}_{B}^{[T]})\theta_{C}(\hat{\bf X}_{C}^{[T]}|{\bf
X}_{C}^{[T]})\mathds{P}({\bf X}_{A}^{[T]},{\bf X}_{B}^{[T]},{\bf
X}_{C}^{[T]})$. When a reviewer plays a truthful reporting strategy (i.e.,
$\theta_{k}$ is identity), we denote it is as $\tau_{k}$.

Now, if a reviewer only completes up to $t$ criteria, they will not arrive at
an informed private score on criteria at higher levels. However, their signals
on lower level criteria do provide some information, allowing them to make
reasonable guesses of the scores on higher level criteria666This is an
important difference between our approach and the original HMIP, where
reporting on incomplete criteria is optional. See Appendix D for more
details.. Indeed, we assume that on incomplete higher level criteria, a
reviewer’s default private signal is their ‘best guess’ of the signal that
other agents would expect of them given the information they already have, and
this may depend on how much effort the other agents exert. This is given by
some function
$g_{k}^{t^{\prime}|t}:\Sigma_{1}\times\ldots\times\Sigma_{t}\rightarrow\Sigma_{t^{\prime}}$
where $x_{k}^{t^{\prime}}=g_{k}^{t^{\prime}|t}({\bf x}_{k}^{[t]})$ represents
reviewer $k$’s best guess of their private assessment on criterion
$t^{\prime}$ when they have completed only criteria at level $t$ (where
${t^{\prime}}>t)$. Note that $x_{k}^{t}$ thus either refers to an effort-
informed signal drawn from the common prior or a best-guess, depending on the
reviewer’s effort level. We use the notation ${\bf x}_{k}^{[t_{k}]}$ if we
wish to emphasize that we are only referring to the effort-informed signals on
criteria $[t_{k}]$ (when $t_{k}<T$).

##### Conditional Independence and Stochastic Relevance

First, we adopt a conditional independence assumption from Kong and
Schoenebeck, (2018), which states that a reviewer $k^{\prime}$’s private score
on some criterion $t$ contains all the information needed to predict their
peer $k$’s score on that criterion, given that peer’s scores on lower level
criteria. We write $I(\cdot)$ to represent the mutual information (see
Appendix C for more details).

######  Assumption 2 (Conditional Independence).

We make the following conditional independence assumption on the common prior:

$I({\bf X}_{k^{\prime}}^{[T]},X_{k}^{t}|{\bf
X}_{k}^{[t-1]})=I(X_{k^{\prime}}^{t},X_{k}^{t}|{\bf X}_{k}^{[t-1]})$ (5)

or equivalently,

$I({\bf X}_{k^{\prime}}^{[T]}\backslash X_{k^{\prime}}^{t},X_{k}^{t}|{\bf
X}_{k}^{[t-1]},X_{k^{\prime}}^{t})=0$ (6)

Finally, we make an assumption on the informativeness of the signals a
reviewer might observe. When a reviewer $k$ arrives at review scores ${\bf
x}_{k}^{[t]}$ on $[t]$ criteria, they also arrive at a posterior belief
$\mathds{P}({\bf X}_{-k}^{[T]}|{\bf x}_{k}^{[t]})$ about the review scores of
other reviewers. Informally, we assume there is some signal a reviewer could
observe, which when paired with two different signals from a second reviewer,
would induce two different beliefs about the signals of the third reviewer.

######  Assumption 3 (Second Order Stochastic Relevance).

Suppose that for any distinct signals
$x_{B}^{t},\tilde{x}_{B}^{t}\in\Sigma_{t}$ on any criterion $t$, there is
$x_{A}^{t}\in\Sigma_{t}$ such that:

$\mathds{P}(X_{C}^{t}|x_{A}^{t},x_{B}^{t})\neq\mathds{P}(X_{C}^{t}|x_{A}^{t},\tilde{x}_{B}^{t})$

We assume the above holds for any permutation of the reviewers $\\{A,B,C\\}$,
so the common prior $\mathds{P}$ is second order stochastic relevant.

##  Appendix F Additional Results

Here, we give some additional results characterizing the marginal payments to
reviewers, optimal setting of mechanism hyperparameters, and aggregate
reviewer payments. We also give a sample walkthrough for our H-DIPP reviewing
mechanism.

Going forward, we use $R$ to denote expected payment. A reviewer’s _marginal
expected payment_ is the additional payment they gain for privately exerting
enough effort to complete an additional criterion, under a fixed strategy by
the other agents. We give the following lemma on reviewers’ marginal expected
payments for completing an additional criterion and show that it is always
non-negative (when peers have completed all criteria and are reporting
truthfully). We break up the total marginal expected payment $\Delta R_{k}$
into the marginal expected expert and target payments $\Delta
R_{k}^{\text{expert}}$ and $\Delta R_{k}^{\text{target}}$, which in turn are
each broken up into marginal payments $\Delta R_{k^{t}}^{\text{expert}}$ and
$\Delta R_{k^{t}}^{\text{target}}$ on each criterion.

######  Lemma 3 (Non-negative Marginal Payments).

Suppose reviewers $B,C$ complete all criteria and truthfully report their
private review scores and predictions. Then, reviewer $A$’s marginal expected
payment for completing criterion $t_{A}$ and honestly reporting private review
scores and predictions is $\Delta
R_{A}({t_{A}}):=R_{A}(t_{A})-R_{A}(t_{A}-1)=\Delta{R}_{A}^{\text{expert}}(t_{A})+\Delta{R}_{A}^{\text{target}}(t_{A})$,
where:

  1. 1.

$\Delta
R_{A}^{\text{expert}}(t_{A}):={R}_{A}^{\text{expert}}(t_{A})-{R}_{A}^{\text{expert}}(t_{A}-1)=\sum_{t=1}^{T}\alpha_{A}^{t}\Delta
R_{A^{t}}^{{\text{ expert}}}(t_{A})$

  2. 2.

$\Delta
R_{A}^{\text{target}}(t_{A}):={R}_{A}^{\text{target}}(t_{A})-{R}_{A}^{\text{target}}(t_{A}-1)=\sum_{t=1}^{T}\beta_{A}^{t}\Delta
R_{A^{t}}^{\text{target}}(t_{A})$

  3. 3.

$\Delta R_{A^{t}}^{\text{expert}}(t_{A}):=I\left(X_{B}^{t};X_{A}^{t_{A}}|{\bf
X}_{A}^{[t_{A}-1]},{\bf
X}_{C}^{[t]}\right)+I\left(X_{B}^{t};X_{A}^{t_{A}}|{\bf
X}_{A}^{[t_{A}-1]},{\bf X}_{C}^{[t-1]}\right)$

  4. 4.

$\Delta R_{A^{t}}^{\text{target}}(t_{A}):=\begin{cases}0,&t<t_{A}\\\
I\left({X}_{B}^{t_{A}};{X}_{A}^{t_{A}}|{\bf X}_{C}^{[T]},{\bf
X}_{B}^{[t_{A}-1]}\right)-I\left({X}_{B}^{t_{A}};g_{A}^{t_{A}|t_{A}-1}({\bf
X}_{A}^{[t_{A}-1]})|{\bf X}_{C}^{[T]},{\bf
X}_{B}^{[t_{A}-1]}\right),&t=t_{A}\\\
\mathds{E}\left[\text{KL}\left({\mathds{P}({X}_{B}^{t}|g_{A}^{t|t_{A}}({\bf
X}_{A}^{[t_{A}]}),{\bf X}_{C}^{[T]},{\bf
X}_{B}^{[t-1]})||\mathds{P}({X}_{B}^{t}|g_{A}^{t|t_{A}-1}({\bf
X}_{A}^{[t_{A}-1]}),{\bf X}_{C}^{[T]},{\bf
X}_{B}^{[t-1]})}\right)\right],&t>t_{A}\end{cases}$

Furthermore, the marginal expected expert and target payments on each
criterion are always non-negative, i.e., $\Delta
R_{A^{t}}^{\text{expert}}(t_{A}),\Delta R_{A^{t}}^{\text{target}}(t_{A})\geq
0$ for all $t,t_{A}\in[T]$ and the inequality is strict when $t=t_{A}$.
Similar results hold for reviewers $B$ and $C$ when peers complete all
criteria and report truthfully.

With this, we can now specify how to set the mechanism hyperparameters to
incentivize a reviewer to complete all criteria, such that all reviewers
completing all criteria and reporting truthfully is a strict BNE (equivalent
to the _potent_ property of hyperparameters in Kong and Schoenebeck, (2018)).

######  Theorem 3 (Optimal Mechanism Hyperparameters).

Suppose mechanism hyperparameters
$\\{\boldsymbol{\alpha}_{A}^{[T]},\boldsymbol{\beta}_{A}^{[T]},\boldsymbol{\alpha}_{B}^{[T]},\boldsymbol{\beta}_{B}^{[T]},\boldsymbol{\alpha}_{C}^{[T]},\boldsymbol{\beta}_{C}^{[T]}\\}$
are solutions to the following linear program:

$\small\begin{split}&\min\sum_{k\in\\{A,B,C\\}}\sum_{t=1}^{T}\alpha_{k}^{t}R_{k^{t}}^{\text{expert}}(T)+\beta_{k}^{t}R_{k^{t}}^{\text{target}}(T)\\\
&\text{s.t. }\sum_{t=1}^{T}\alpha_{A}^{t}\Delta
R_{A^{t}}^{\text{expert}}(t_{A})+\beta_{A}^{t}\Delta
R_{A^{t}}^{\text{target}}(t_{A})>\Delta e_{A}(t_{A})\text{ for }1\leq
t_{A}\leq T\\\ &\phantom{\text{s.t. }}\sum_{t=1}^{T}\alpha_{B}^{t}\Delta
R_{B^{t}}^{\text{expert}}(t_{B})+\beta_{B}^{t}\Delta
R_{B^{t}}^{\text{target}}(t_{B})>\Delta e_{B}(t_{B})\text{ for }1\leq
t_{B}\leq T\\\ &\phantom{\text{s.t. }}\sum_{t=1}^{T}\alpha_{C}^{t}\Delta
R_{C^{t}}^{\text{expert}}(t_{C})+\beta_{C}^{t}\Delta
R_{C^{t}}^{\text{target}}(t_{C})>\Delta e_{C}(t_{C})\text{ for }1\leq
t_{C}\leq T\\\ &\phantom{\text{s.t. }}\alpha_{k}^{t},\beta_{k}^{t}\geq
0\end{split}$ (7)

Then, for any reviewer $k$, if their peers complete all criteria and report
predictions and review scores truthfully, reviewer $k$ strictly maximizes
their expected utility by also completing all criteria and reporting
predictions and review scores truthfully. In other words, completing all
criteria and reporting truthfully is a strict Bayesian-Nash equilibrium. We
refer to this equilibrium as the fully-informative equilibrium.

Having previously shown that our desired strategy profile (all reviewers
completing all criteria and reporting truthfully) is a strict BNE, we now
discuss how this compares to other equilibria and strategy profiles. A common
challenge is that truthful equilibria pay less than effortless uninformative
equilibria (like in single-HMIM), so the truthful equilibria are not realized
in practice. However, this is not the case with H-DIPP; as we stated in
Theorem 2 an uninformative equilibrium pays nothing in expectation and the
fully informative equilibrium pays strictly more. To show this, we need the
following lemma on the total payment to reviewers in any equilibrium:

######  Lemma 4 (Aggregate Reviewer Payments).

Let $s_{k}=\left(e_{k}(t_{k}),\hat{\bf x}_{k}^{[T]},\hat{\bf
p}^{+}_{k^{\prime[T]}\leftarrow k},\hat{\bf p}_{k^{\prime[T]}\leftarrow
k}\right)$ be reviewer $k$’s strategy, where $k^{\prime}$ is reviewer $k$’s
target. If ${\bf s}=(s_{A},s_{B},s_{C})$ is an equilibrium strategy profile,
reviewer $A$’s expected payment is:

$\small\mathds{E}[r_{A}]=\sum_{t=1}^{T}\alpha_{A}^{t}\left(-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{B}^{t}|{\bf
x}_{A}^{[t_{A}]},\hat{\bf
x}_{C}^{[t]}}\right)\right)\right]-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{B}^{t}|{\bf
x}_{A}^{[t_{A}]},\hat{\bf
x}_{C}^{[t-1]}}\right)\right)\right]\right)+\beta_{A}^{t}I\left(\hat{X}_{B}^{t};\hat{X}_{A}^{t}|{\bf
X}_{C}^{[t_{C}]},\hat{\bf X}_{B}^{[t-1]}\right)$ (8)

(and expected payments for peers $B,C$ computed analogously). Reviewers’ total
expected payment is:

$\small\begin{split}\mathds{E}[r_{A}+r_{B}+r_{C}]&=\sum_{t=1}^{T}\alpha_{A}^{t}\left(-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{B}^{t}|{\bf
x}_{A}^{[t_{A}]},\hat{\bf
x}_{C}^{[t]}}\right)\right)\right]-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{B}^{t}|{\bf
x}_{A}^{[t_{A}]},\hat{\bf
x}_{C}^{[t-1]}}\right)\right)\right]\right)+\beta_{A}^{t}I\left(\hat{X}_{B}^{t};\hat{X}_{A}^{t}|{\bf
X}_{C}^{[t_{C}]},\hat{\bf X}_{B}^{[t-1]}\right)\\\
&\phantom{=}+\alpha_{B}^{t}\left(-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{C}^{t}|{\bf
x}_{B}^{[t_{B}]},\hat{\bf
x}_{A}^{[t]}}\right)\right)\right]-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{C}^{t}|{\bf
x}_{B}^{[t_{B}]},\hat{\bf
x}_{A}^{[t-1]}}\right)\right)\right]\right)+\beta_{B}^{t}I\left(\hat{X}_{C}^{t};\hat{X}_{B}^{t}|{\bf
X}_{A}^{[t_{A}]},\hat{\bf X}_{C}^{[t-1]}\right)\\\
&\phantom{=}+\alpha_{C}^{t}\left(-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{A}^{t}|{\bf
x}_{C}^{[t_{C}]},\hat{\bf
x}_{B}^{[t]}}\right)\right)\right]-\mathds{E}\left[H\left(\mathds{Q}\left({\hat{X}_{A}^{t}|{\bf
x}_{C}^{[t_{A}]},\hat{\bf
x}_{B}^{[t-1]}}\right)\right)\right]\right)+\beta_{C}^{t}I\left(\hat{X}_{A}^{t};\hat{X}_{C}^{t}|{\bf
X}_{B}^{[t_{B}]},\hat{\bf X}_{A}^{[t-1]}\right)\end{split}$ (9)

We make several observations. First, unlike the ideal-HMIP proposal (and like
the concrete single-HMIM proposal) in Kong and Schoenebeck, (2018), we cannot
claim that truthful reporting of private signals is a dominant strategy, since
a reviewer’s target payment for reporting private scores will depend on their
peer’s expert prediction strategy. However, it is the case that when we
compare equilibria fixing (truthful or untruthful) peer strategies, a reviewer
earns higher target rewards in the equilibria where they report truthfully,
compared to ones where they strategically manipulate the score, since
$I\left(\hat{X}_{B}^{t};\hat{X}_{A}^{t}|{\bf X}_{C}^{[t_{C}]},\hat{\bf
X}_{B}^{[t-1]}\right)\leq I\left(\hat{X}_{B}^{t};{X}_{A}^{t}|{\bf
X}_{C}^{[t_{C}]},\hat{\bf X}_{B}^{[t-1]}\right)$ for all $t\in[T]$ (and
similarly for other reviewers) due to the data processing inequality (Kong and
Schoenebeck,, 2019).

Second, unlike TDPP (which does not account for effort), we cannot guarantee
the strong truthfulness of the fully informative truth-telling equilibrium,
i.e., we cannot guarantee that the fully informative equilibrium has higher
aggregate utility than any other equilibrium. It may be possible for a
reviewer to strategically manipulate their private signals so that when used
as a source for some expert’s prediction, the target receives higher payment;
e.g., it may be possible that reviewer $B$’s strategic reports provides a
higher target payment for reviewer $A$ with
$I\left({X}_{B}^{t};\hat{X}_{A}^{t}|{\bf X}_{C}^{[t_{C}]},{\bf
X}_{B}^{[t-1]}\right)\leq I\left(\hat{X}_{B}^{t};\hat{X}_{A}^{t}|{\bf
X}_{C}^{[t_{C}]},\hat{\bf X}_{B}^{[t-1]}\right)$. Reviewer $B$’s strategic
reports could then increase reviewer $A$’s target payment by more than it
hurts their own target payment, thereby raising aggregate utility (costs are
the same). However, reviewers do not receive payments for their role as
source, and further must accept lower target payment for such strategic
manipulation, so reviewers are disincentivized from pursuing such equilibria
unilaterally. Nevertheless, we cannot rule out the possibility of viable
collusive equilibria.

###  F.1 Sample Mechanism Flow

Here, we walkthrough an example of how the H-DIPP reviewing mechanism would
work according to Algorithm 1. We emphasize that this is just an example; the
number of questions, the response scale, etc. may be modified in practice.

Suppose reviewers Alice, Bob, and Charlie are assigned to review paper
$\omega$. Each reviewer reads and scores the paper on the same five pre-
defined criteria on a five-point scale: 1) _(presentation)_ “How clear is the
writing/exposition?”; 2) _(completeness)_ “How easy would it be to reproduce
major results?”; 3) _(correctness)_ “How accurate are the technical claims and
methodology?”; 4) _(contribution)_ “How interesting and valuable of a
contribution is the work? ”; 5) _(overall quality)_ “In which quintile of
submitted papers would this work fall?"

Once all the reviews are in, each of the reviewers logs into the review
website and plays an asynchronous prediction game. We consider the game from
Alice’s perspective, who is tasked with predicting Bob’s review, and shown
Charlie’s scores for assistance. Bob will be tasked with predicting Charlie’s
scores, with Alice’s scores for assistance, and Charlie will be tasked with
predicting Alice’s scores, with Bob’s scores for assistance. The game from
Alice’s perspective proceeds as follows:

  1. 1.

Q1: Presentation

     * •

Alice makes a prediction of Bob’s score on Q1: presentation.

     * •

Alice is shown Charlie’s score on Q1: presentation.

     * •

Alice makes an updated prediction of Bob’s score on Q1: presentation.

  2. 2.

Q2: Completeness

     * •

Alice makes a prediction of Bob’s score on Q2: completeness.

     * •

Alice is shown Charlie’s score on Q2: completeness.

     * •

Alice makes an updated prediction of Bob’s score on Q2: completeness.

  3. 3.

Q3: Correctness

     * •

Alice makes a prediction of Bob’s score on Q3: correctness.

     * •

Alice is shown Charlie’s score on Q3: correctness.

     * •

Alice makes an updated prediction of Bob’s score on Q3: correctness.

  4. 4.

Q4: Contribution

     * •

Alice makes a prediction of Bob’s score on Q4: contribution.

     * •

Alice is shown Charlie’s score on Q4: contribution.

     * •

Alice makes an updated prediction of Bob’s score on Q4: contribution.

  5. 5.

Q5: Overall Quality

     * •

Alice makes a prediction of Bob’s score on Q5: overall quality.

     * •

Alice is shown Charlie’s score on Q5: overall quality.

     * •

Alice makes an updated prediction of Bob’s score on Q5: overall quality.

The game then concludes for Alice. Once Bob and Charlie have also completed
the game, the mechanism may determine payments as per the H-DIPP payment
scheme.

##  Appendix G Demand and Supply

In Section 3.2, we assumed the VCG auction had a fixed number of review slots
$P$ to be auctioned. Here, we present the considerations and challenges in
designing a mechanism that sets $P$ dynamically _after_ paper submissions
based on the supply of reviewers, but a comprehensive solution is a task for
future work. The foremost considerations in setting $P$ based on the
availability of reviewers are:

  1. 1.

Individual Rationality and Truthfulness: Authors and reviewers should not
expect to lose credits by participating the mechanism, and authors should not
be able to game the number of review slots by bidding something other than
their expected value for having their paper reviewed.

  2. 2.

Balanced Budget: We aim to set $P$ based on the number of reviews that can be
paid for under the H-DIPP payment scheme using revenue from the auction.
Recall that the cost to the mechanism to have one paper reviewed by _a given
reviewer triplet_ in the fully-informative equilibrium (if this is indeed the
observed equilibrium) is given by Lemma 4, where hyperparameters have been set
using Theorem 3.

  3. 3.

Maximize $P$: We would like to accept as many papers for review as feasible.

  4. 4.

Fair Expertise-based Matching: Reviewers must be well-qualified to review
their assigned papers and have the appropriate domain expertise. Additionally,
all papers accepted for review should have equal consideration when being
assigned reviewers; the bid accompanying a paper should have no bearing on the
quality of reviewer matches777Aside from the motivation of fairness, if higher
bidding papers received priority in matching reviewers, reviewers could infer
a paper’s bid from the quality of their match, giving them a low-effort signal
of paper quality.. Thus, although the review slots are identical from authors’
perspective (for fairness), the mechanism must match them to reviewers
appropriately. We assume access to a black-box algorithm that can score a
reviewer’s match with a paper, and assign reviewers to papers based on this
score (e.g. Toronto Paper Matching System (Charlin and Zemel,, 2013)).

The final consideration comes from the observation that the mechanism should
not blindly seek to minimize its total cost over all papers in an attempt to
squeeze in as many review acceptances $P$ as possible. The cost to the
mechanism for reviewing a single paper depends both on the hyperparameters
(optimally set via Theorem 3) as well as the mutual information in reviewers’
signals, so total costs can be made abnormally low by badly matching reviewers
and paying them little since their signals will have naturally low mutual
information. Heuristically, we expect that well-matched expert reviewers hit
the sweet spot of having low effort costs to review a paper and more mutually
informative review scores with expert peers, so the mechanism pays a fair
price for such expert reviews.

Nevertheless, we face a trade-off where accepting more papers for review can
decrease the average quality of the paper-reviewer matches. Formally, let
$S_{i_{k}}^{*}$ denote the match score for the $k$-th reviewer of paper
$\omega_{i}$ and $R_{i_{k}}^{*}$ denote the expected payment to the $k$-th
reviewer of paper $\omega_{i}$ under the optimal assignment of reviewers (as
determined by the black-box algorithm) for some given number of papers $P$
accepted for review. Then, a naive mechanism designer’s objective is to pick
the number of review slots to be the largest $P$ with an acceptable average
reviewer match score that also satisfies the balanced budget constraint (where
slots are priced as in the VCG auction):

$\small\begin{split}&\underset{P}{\text{argmax}}\,\,\,\,\left(\frac{1}{P}\sum_{i=1}^{P}S_{i_{A}}^{*}+S_{i_{B}}^{*}+S_{i_{C}}^{*}\right)+\lambda
P\\\ &\text{s.t. }\,\,\,\,b^{*}_{P+1}\cdot
P\geq\sum_{i=1}^{P}R_{i_{A}}^{*}+R_{i_{B}}^{*}+R_{i_{C}}^{*}\end{split}$ (10)

The parameter $\lambda$ controls the tradeoff between match quality and number
of papers accepted for review, and is a design decision to be made by
individual conferences. The objective can be modified if desired; substituting
the mean reviewer match score for the median or an additional term penalizing
variance in reviewer match scores are alternative choices. In a naive
algorithm, we can find the optimal number of review slots by first computing
all the valid number of review slots under the balanced budget constraint, and
picking the largest one with an acceptable average reviewer match score
(summarized in Algorithm 2).

Input: $N$ papers submitted to the conference, a black-box algorithm matching
reviewers to papers based on expertise (e.g. Toronto Paper Matching System)

Output: Optimal Number of Review Slots $P$

1 for  _$n=1:N$_ do

2 Obtain optimal matching of reviewers to $n$ papers from black-box algorithm

3 If accepting $n$ papers satisfies the budget constraint $b^{*}_{P+1}\cdot
P\geq\sum_{i=1}^{P}R_{i_{A}}^{*}+R_{i_{B}}^{*}+R_{i_{C}}^{*}$, add $p$ to the
list of allowed number of review slots
$\mathcal{P}\leftarrow\mathcal{P}\cup\\{n\\}$

Select the largest $P\in\mathcal{P}$ with an acceptable average reviewer match
score, i.e., largest $P$ that maximizes the objective
$\left(\frac{1}{P}\sum_{i=1}^{P}S_{i_{A}}^{*}+S_{i_{B}}^{*}+S_{i_{C}}^{*}\right)+\lambda
P$

Algorithm 2 Naive Algorithm to Determine Number of Review Slots $P$

However, such a mechanism faces several challenges. Firstly, it is not obvious
that it is truthful; authors may be able to affect the number of review slots
by changing their bids. Secondly, we need to identify the mechanism
hyperparameters by solving the linear program in Theorem 3 and computing the
expected payment in the fully informative equilibrium (Lemma 4); this requires
estimating or eliciting information about the common prior $\mathds{P}$ and
the reviewer triplet’s effort cost functions $e_{k}(\cdot)$. Furthermore, in
practice we may need a single set of mechanism hyperparameters
$\\{\boldsymbol{\alpha}^{[T]},\boldsymbol{\beta}^{[T]}\\}$ that determine
payments to all reviewers. Lastly, when the number of review slots $P$ is
fixed, authors’ bids will not affect their utilities in the reviewing stage of
the same conference since the hyperparameters will have been pre-determined;
this allows us to analyze the two stages of the mechanism independently.
However, endogenizing $P$ would mean the setting of hyperparameters (and hence
payments to reviewers) is sensitive to the revenue raised (and hence bids made
by authors). This coupling of the two stages complicates the analysis. These
challenges are important directions for future work.

##  Appendix H Directions for Future Work

We present some important directions for future work below:

  1. 1.

Criteria Hierarchy: What is the best construction of the criteria hierarchy?

  2. 2.

Estimating Hyperparameters and Costs: Setting mechanism hyperparameters and
the number of review slots $P$ requires information about reviewers’ marginal
payments and costs, so identifying ways to learn (Liu and Chen,, 2016) or
elicit this information would be valuable.

  3. 3.

Endogenizing the Number of Review Slots $P$: How can we allow $P$ to be set
based on the availability of reviewers, while guaranteeing an individually
rational, truthful, and balanced budget mechanism that accepts as many well-
matched papers as possible?

  4. 4.

Variance in Payments: Is the variance in payments so large that honest and
effortful reviews get negative payments with unacceptable regularity? If so,
can the addition of a few more reviewers reduce this variance?

  5. 5.

Improving Equilibria: Can we leverage information across reviewing tasks
(Dasgupta and Ghosh,, 2013; Agarwal et al.,, 2017) in an efficient way to
further increase total payoffs in the truth-telling equilibrium? How does the
utility of untruthful equilibria compare to the fully-informative equilibrium?
How robust is the mechanism to collusion?

  6. 6.

Truthful, but Useful?: H-DIPP is strictly _truthful_ , but whether or not this
is useful is a different empirical question. If H-DIPP-incentivized review
scores vary widely even with post-processing (i.e., truthful reviews end up
being quite idiosyncratically subjective), or do not track other measures of
paper quality, it may not be possible to extract a meaningful signal
(criticism C4). Running small scale experiments with real money could shed
light on whether the benefits of our proposed mechanism warrant the overhead.

##  Appendix I Proofs

##### Theorem 1

(Strictly Truthful) _Suppose reviewers $A,B,C$ complete $[t_{A}],[t_{B}]$, and
$[t_{C}]$ criteria respectively. Then, on shared complete criteria
$[t^{\prime}]=[\min(t_{A},t_{B},t_{C})]$, the H-DIPP mechanism is strictly
truthful, i.e., all reviewers honestly reporting their true predictions and
private review scores on shared complete criteria $[t^{\prime}]$ is a strict
Bayesian Nash equilibrium._

###### Proof of Theorem 1.

We adapt the proof of strict truthfulness of TDPP by Schoenebeck and Yu,
(2020) for H-DIPP. We show that agents reporting their honest predictions and
scores on shared complete criteria is a strict Bayesian Nash equilibrium;
importantly, payments for each criterion are independent and can be
strategized about independently, hence we can reason strictly about strategies
on the shared complete criteria ${\bf x}_{k}^{[t^{\prime}]}\subseteq{\bf
x}_{k}^{[t_{k}]}$. Reviewer $A$’s private signals on incomplete criteria
$t>t_{A}$ are ${\bf x}_{A}^{t}=g_{A}^{t|t_{A}}({\bf x}_{A}^{[t_{A}]})$ for
some best guess function $g$, and similarly for reviewers $B$ and $C$. Since
all reviewers play the role of expert, target, and source, we only analyze
reviewer $A$’s incentives when peers $B$ and $C$ report truthfully; these
incentives are symmetrical for all reviewers. Additionally, we hold the number
of completed criteria and hence effort cost $e_{k}(t_{k})$ constant, and look
to optimize payoff under this condition.

Suppose reviewers $B$ and $C$ report their private signals truthfully, i.e.,
$\hat{x}_{B}^{t}=x_{B}^{t}$ and $\hat{x}_{C}^{t}=x_{C}^{t}$, and reviewer $C$
as expert truthfully predicts reviewer $A$’s score as the posterior from
updating the common prior $\hat{\bf p}_{A^{t}\leftarrow
C}=\mathds{P}(X_{A}^{t}|{\bf x}_{C}^{[t^{\prime}]},{\bf x}_{B}^{[t-1]})$, and
$\hat{\bf p}^{+}_{A^{t}\leftarrow C}=\mathds{P}(X_{A}^{t}|{\bf
x}_{C}^{[t^{\prime}]},{\bf x}_{B}^{[t]})$.

Now, reviewer $A$’s expected payment (taken with respect to the joint
distribution $\mathds{Q}(\hat{\bf X}^{[T]}_{A},{\bf X}^{[t^{\prime}]}_{A},{\bf
X}^{[t^{\prime}]}_{B},{\bf X}^{[t^{\prime}]}_{C})$ of true signals and $A$’s
potentially strategic reports) on the shared complete criteria can be written
as:

$\small\begin{split}\mathds{E}_{\mathds{Q}}[r_{A^{t}}]&=\alpha_{A}^{t}\mathds{E}\left[\underbrace{[S_{\text{log}}({x}_{B}^{t},\hat{\bf
p}_{B^{t}\leftarrow A})+S_{\text{log}}({x}_{B}^{t},\hat{\bf
p}^{+}_{B^{t}\leftarrow A})}_{\text{expert
payment}}\right]+\beta_{A}^{t}\mathds{E}\left[\underbrace{S_{\text{log}}(\hat{x}_{A}^{t},\mathds{P}(X_{A}^{t}|{\bf
x}_{C}^{[t^{\prime}]},{\bf
x}_{B}^{[t]}))-S_{\text{log}}(\hat{x}_{A}^{t},\mathds{P}(X_{A}^{t}|{\bf
x}_{C}^{[t^{\prime}]},{\bf x}_{B}^{[t-1]}))}_{\text{target
payment}}\right]\end{split}$ (11)

Now, reviewer $A$’s payment on a given criterion $t$ is independent of their
reported signals and predictions on other criteria, so the sum of payments on
complete criteria can be maximized by maximizing the expert payment and target
payment on each criterion:

_Expert Payment_ The expected expert payment consists of reviewer $A$’s
predictions of their target $B$’s signal on each criterion $t$ first with the
source $C$’s signals on criteria less than $t$ and then with $C$’s signal on
criterion $t$ as well. Predictions on each criterion are scored independently,
so a strategic prediction on one criterion cannot impact future payments on
other criteria. Since we use a strictly proper scoring rule to score both
predictions on each criterion, reviewer $A$’s expected payoff for each
criterion is strictly maximized when they report their true posterior belief
over their target $B$’s signals under the common prior, i.e., $\hat{\bf
p}^{+}_{B^{t}\leftarrow A}=\mathds{P}(X_{B}^{t}|{\bf x}_{A}^{[t_{A}]},{\bf
x}_{C}^{[t]})$ and $\hat{\bf p}_{B^{t}\leftarrow A}=\mathds{P}(X_{B}^{t}|{\bf
x}_{A}^{[t_{A}]},{\bf x}_{C}^{[t-1]})$.

_Target Payment_ Now, consider the reviewer $A$’s expected target payment on
criteria $t\leq t^{\prime}$. We observe that we can model the strategies for
responses to different criteria independently since the target payments are
also independent for different criteria. Let
$\hat{x}_{A}^{t}=\theta_{A^{t}}({\bf x}_{A}^{[t_{A}]})$ be reviewer $A$’s
_deterministic best response_ report (on criterion $t$) to reviewer $B$’s
honest signal reports $\hat{\bf x}_{B}^{[T]}$ and reviewer $C$’s honest
(posterior) predictions $\hat{\bf p}^{+}_{A^{t}\leftarrow
C}=\mathds{P}({X}_{A}^{t}|{\bf x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t]})$, $\hat{\bf
p}_{A^{t}\leftarrow C}=\mathds{P}({X}_{A}^{t}|{\bf x}_{C}^{[t_{C}]},{\bf
x}_{B}^{[t-1]})$.

For $t\leq t^{\prime}$, reviewer $A$’s expected target payment for playing
this strategy $\hat{x}_{A}^{t}=\theta_{A^{t}}({\bf x}_{A}^{[t^{\prime}]})$ is:

$\small\begin{split}u_{t}(\theta):&=\mathds{E}\left[S_{\text{log}}(\hat{x}_{A}^{t},\mathds{P}(X_{A}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf
x}_{B}^{[t]}))-S_{\text{log}}(\hat{x}_{A}^{t},\mathds{P}(X_{A}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]}))\right]\\\
&=\mathds{E}_{\mathds{Q}}\left[\log\frac{\mathds{P}(\hat{x}_{A}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t]})}{\mathds{P}(\hat{x}_{A}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]})}\right]\\\
&=\mathds{E}_{\mathds{Q}}\left[\log\frac{\mathds{P}(\hat{x}_{A}^{t},x_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]})}{\mathds{P}(\hat{x}_{A}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]})\mathds{P}(x_{B}^{t}|{\bf
x}_{C}^{[t^{\prime}]},{\bf x}_{B}^{[t-1]})}\right]\\\
&=\mathds{E}_{\mathds{P}}\left[\log\frac{\mathds{P}({x}_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]},\theta_{A^{t}}({\bf
x}_{A}^{[t_{A}]}))}{\mathds{P}(x_{B}^{t}|{\bf x}_{C}^{[t_{C}]},{\bf
x}_{B}^{[t-1]})}\right]\end{split}$ (12)

By a similar calculation, we have that reviewer $A$’s expected payment under a
truth-telling strategy is
$u_{t}(\tau):=\mathds{E}_{\mathds{P}}\left[\log\frac{\mathds{P}({x}_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]},{x}_{A}^{t})}{\mathds{P}(x_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]})}\right]$. Then, the difference between
truth-telling payment and the best response payment is:

$\begin{split}u_{t}(\tau)-u_{t}(\theta)&=\mathds{E}_{\mathds{P}}\left[\log\frac{\mathds{P}({x}_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]},{x}_{A}^{t})}{\mathds{P}({x}_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]},\theta_{A^{t}}({\bf
x}_{A}^{[t_{A}]}))}\right]\\\ &=\mathds{E}_{{\bf X}_{A}^{[t_{A}]},{\bf
X}_{B}^{[t-1]},{\bf
X}_{C}^{[t_{C}]}}\left[\mathds{E}_{X^{t}}\left[\log\frac{\mathds{P}({x}_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]},{x}_{A}^{t})}{\mathds{P}({\bf
x}_{B}^{t}|{\bf x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]},\theta_{A^{t}}({\bf
x}_{A}^{[t_{A}]}))}\right]\left|{\bf X}_{A}^{[t_{A}]}={\bf
x}_{A}^{[t_{A}]},{\bf X}_{B}^{[t-1]}={\bf x}_{B}^{[t-1]},{\bf
X}_{C}^{[t_{C}]}={\bf x}_{C}^{[t_{C}]}\right.\right]\\\ &=\mathds{E}_{{\bf
X}_{A}^{[t_{A}]},{\bf X}_{B}^{[t-1]},{\bf
X}_{C}^{[t_{C}]}}\left[\text{KL}\left({\mathds{P}({x}_{B}^{t}|x_{C}^{[t_{C}]},x_{B}^{[t-1]},{x}_{A}^{t})}||{\mathds{P}({x}_{B}^{t}|x_{C}^{[t_{C}]},x_{B}^{[t-1]},\theta_{A^{t}}({\bf
x}_{A}^{[t_{A}]}))}\right)\right]\end{split}$ (13)

Since KL is a divergence, it is greater than or equal to zero, so we must have
$u_{t}(\tau)-u_{t}(\theta)\geq 0$. Finally, observe that if the strategy
$\theta$ is not truthful, i.e., $\theta_{A^{t}}({\bf
x}_{A}^{[t_{A}]})\neq{x}_{A}^{t}$, then by the second-order stochastic
relevance there is some $x_{C}^{t}\subset{\bf x}_{C}^{[t_{C}]}$ for which
${\mathds{P}({x}_{B}^{t}|{\bf x}_{C}^{[t_{C}]},{\bf
x}_{B}^{[t-1]},{x}_{A}^{t})}\neq{\mathds{P}({x}_{B}^{t}|{\bf
x}_{C}^{[t_{C}]},{\bf x}_{B}^{[t-1]},\hat{x}_{A}^{t})}$, so the KL divergence
is not identically zero over all signals in the expectation. Consequently, the
inequality is strict, implying $u(\tau)>u(\theta)$. Thus, the best response
strategy is truthtelling ($\theta=\tau$), and reviewer $A$ strictly maximizes
their target payment by truthfully reporting their signal
$\hat{x}_{A}^{t}=x_{A}^{t}$ on all complete criteria.

We have shown that when peers report their predictions and private review
scores truthfully, a reviewer should truthfully report their prediction to
strictly maximize their expert payment and truthfully report their private
review score to strictly maximize their target payment. These incentives are
symmetrical for reviewers $B$ and $C$, so truthful reporting is a strict
Bayesian Nash Equilibrium. ∎

##### Lemma 3

_Suppose reviewers $B,C$ complete all criteria and truthfully report their
private review scores and predictions. Then, reviewer $A$’s marginal expected
payment for completing criterion $t_{A}$ and honestly reporting private review
scores and predictions is $\Delta
R_{A}({t_{A}}):=R_{A}(t_{A})-R_{A}(t_{A}-1)=\Delta{R}_{A}^{\text{expert}}(t_{A})+\Delta{R}_{A}^{\text{target}}(t_{A})$,
where:_

  1. 1.

$\Delta
R_{A}^{\text{expert}}(t_{A}):={R}_{A}^{\text{expert}}(t_{A})-{R}_{A}^{\text{expert}}(t_{A}-1)=\sum_{t=1}^{T}\alpha_{A}^{t}\Delta
R_{A^{t}}^{{\text{ expert}}}(t_{A})$

  2. 2.

$\Delta
R_{A}^{\text{target}}(t_{A}):={R}_{A}^{\text{target}}(t_{A})-{R}_{A}^{\text{target}}(t_{A}-1)=\sum_{t=1}^{T}\beta_{A}^{t}\Delta
R_{A^{t}}^{\text{target}}(t_{A})$

  3. 3.

$\Delta R_{A^{t}}^{\text{expert}}(t_{A}):=I\left(X_{B}^{t};X_{A}^{t_{A}}|{\bf
X}_{A}^{[t_{A}-1]},{\bf
X}_{C}^{[t]}\right)+I\left(X_{B}^{t};X_{A}^{t_{A}}|{\bf
X}_{A}^{[t_{A}-1]},{\bf X}_{C}^{[t-1]}\right)$

  4. 4.

$\Delta R_{A^{t}}^{\text{target}}(t_{A}):=\begin{cases}0,&t<t_{A}\\\
I\left({X}_{B}^{t_{A}};{X}_{A}^{t_{A}}|{\bf X}_{C}^{[T]},{\bf
X}_{B}^{[t_{A}-1]}\right)-I\left({X}_{B}^{t_{A}};g_{A}^{t_{A}|t_{A}-1}({\bf
X}_{A}^{[t_{A}-1]})|{\bf X}_{C}^{[T]},{\bf
X}_{B}^{[t_{A}-1]}\right),&t=t_{A}\\\
\mathds{E}\left[\text{KL}\left({\mathds{P}({X}_{B}^{t}|g_{A}^{t|t_{A}}({\bf
X}_{A}^{[t_{A}]}),{\bf X}_{C}^{[T]},{\bf
X}_{B}^{[t-1]})||\mathds{P}({X}_{B}^{t}|g_{A}^{t|t_{A}-1}({\bf