|  PoseBusters: AI-based docking methods fail to generate physically valid
poses or generalise to novel sequences†
---|---
|  Martin Buttenschoen, Garrett M. Morris, and Charlotte M. Deane‡
|  The last few years have seen the development of numerous deep learning-
based protein-ligand docking methods. They offer huge promise in terms of
speed and accuracy. However, despite claims of state-of-the-art performance in
terms of crystallographic root-mean-square deviation (RMSD), upon closer
inspection, it has become apparent that they often produce physically
implausible molecular structures. It is therefore not sufficient to evaluate
these methods solely by RMSD to a native binding mode. It is vital,
particularly for deep learning-based methods, that they are also evaluated on
steric and energetic criteria. We present PoseBusters, a Python package that
performs a series of standard quality checks using the well-established
cheminformatics toolkit RDKit. The PoseBusters test suite validates chemical
and geometric consistency of a ligand including its stereochemistry, and the
physical plausibility of intra- and intermolecular measurements such as the
planarity of aromatic rings, standard bond lengths, and protein-ligand
clashes. Only methods that both pass these checks and predict native-like
binding modes should be classed as having “state-of-the-art” performance. We
use PoseBusters to compare five deep learning-based docking methods (DeepDock,
DiffDock, EquiBind, TankBind, and Uni-Mol) and two well-established standard
docking methods (AutoDock Vina and CCDC Gold) with and without an additional
post-prediction energy minimisation step using a molecular mechanics force
field. We show that both in terms of physical plausibility and the ability to
generalise to examples that are distinct from the training data, no deep
learning-based method yet outperforms classical docking tools. In addition, we
find that molecular mechanics force fields contain docking-relevant physics
missing from deep-learning methods. PoseBusters allows practitioners to assess
docking and molecular generation methods and may inspire new inductive biases
still required to improve deep learning-based methods, which will help drive
the development of more accurate and more realistic predictions.

##

††footnotetext: Department of Statistics, 24-29 St Giles’, Oxford OX1 3LB,
United Kingdom.††footnotetext: † Electronic Supplementary Information (ESI)
available. See DOI: 00.0000/00000000.††footnotetext: ‡ Corresponding author:
<EMAIL_ADDRESS>
##  1 Introduction

Docking, an essential step in structure-based drug discovery 1, is the task of
predicting the predominant binding modes of a protein-ligand complex given an
experimentally solved or computationally modelled protein structure and a
ligand structure 2. The predicted complexes are often used in a virtual
screening workflow to help select molecules from a large library of possible
candidates 3; or directly by medicinal chemists to understand the binding mode
and to decide whether a small molecule is a suitable drug candidate 4.

Docking methods are designed with the understanding that binding is enabled by
interactions between target and ligand structures but due to the complexity of
this property methods tend to strike a balance between fast calculation and
accuracy 5. Deep learning (DL) promises to disrupt the dominant design
principle of classical docking software, and DL-based docking methods promise
to unlock fast and accurate virtual screening for drug discovery. To this end,
a handful of different DL-based docking methods have already been proposed 6,
7, 8, 9, 10.

Classical non-DL-based docking methods include within their search and scoring
functions terms that help ensure chemical consistency and physical
plausibility; for example limiting the degrees of movement in the ligand to
only the rotatable bonds in the ligand and including penalties if the protein
and ligand clash 11, 12. Some current DL-based docking methods, as we will
show, still lack such key “inductive biases” resulting in the creation of
unrealistic poses despite obtaining root-mean-squared deviation (RMSD) values
from the experimental binding mode that are less than the widely-used 2
threshold 13. To assess such docking methods, an independent test suite is
necessary to check the chemical consistency and physical plausibility
alongside established metrics, such as the binding mode RMSD. Such a test
suite would help the field to identify missing inductive biases required to
improve DL-based docking methods, driving the development of more accurate and
realistic docking predictions.

The problem of assessing the physical plausibility of docking predictions is
akin to the structure validation of ligand data in the Protein Data Bank (PDB)
14, 15. Structure validation assesses the agreement of the ligands bond
lengths and angles with those observed in related chemical structures and the
presence of steric clashes both within the ligand and between it and its
surroundings 15. While these tests were designed for users to select those
ligand crystal structures which are likely to be correct 15, docking methods
are evaluated on their ability to recover crystal structures so their output
should pass the same physical plausibility tests.

Physical plausibility checks are also part of some workflows for conformation
generation 16, 17. Friedrich _et al._ use geometry checks performed by NAOMI
18 which measures—like the PDB tests mention above—the deviation from known
optimal values for bond lengths and bond angles, and also tests for
divergences from the planarity of aromatic rings 17.

In addition to physical checks, chemical checks are also needed 19. Chemical
checks proposed for checking PDB structures include the identification of
mislabelled stereo assignment, inconsistent bonding patterns, missing
functional groups, and unlikely ionisation states 19. The problem of checking
chemical plausibility has also come up in _de novo_ molecule generation, where
Brown _et al._ proposed a test suite including checks for the chemical
validity of any proposed molecule 20. For docking, the focus is less on
stability and synthetic accessibility of a molecular structure as it is hoped
that these have been tested prior to attempting docking, but more on chemical
consistency and physical realism of the predicted bound conformation.

Some comparisons of docking methods have included additional metrics based on
volume overlap 21 or protein-ligand interactions 22 to supplement pose
accuracy-based metrics such as RMSD of atomic positions and run time
measurements, but the majority of comparisons of docking methods are
predominantly based on binding mode RMSD 13, 23, 24, 25.

The current standard practice of comparing docking methods based on RMSD-based
metrics alone also extends to the introduction papers of recent new methods.
The five DL-based docking methods we test in this paper 6, 7, 8, 9, 10 all
claim better performance than standard docking methods but these claims rest
entirely on RMSD. None of these methods test their outputs for physical
plausibility.

In this paper we present PoseBusters, a test suite that is designed to
identify implausible conformations and ligand poses. We used PoseBusters to
evaluate the predicted ligand poses generated by the five DL-based docking
methods (DeepDock6, DiffDock7, EquiBind8, TankBind9, and Uni-Mol10) and two
standard non-DL-based docking methods (AutoDock Vina12 and Gold26). These
poses were generated by re-docking the cognate ligands of the 81 protein-
ligand crystal complexes in the Astex Diverse set27 and 428 ligands of the
protein-ligand crystal complexes in the PoseBusters Benchmark set, a new set
of complexes released from 2021 onwards, into their cognate receptor crystal
structures. On the commonly-used Astex Diverse set, the DL-based docking
method DiffDock appears to perform best in terms of RMSD alone but when taking
physical plausibility into account, Gold and AutoDock Vina perform best. On
the PoseBusters Benchmark set, a test set that is harder because it contains
only complexes that the DL methods have not been trained on, Gold and AutoDock
Vina are the best methods in terms of RMSD alone and when taking physical
plausibility into account or when proteins with novel sequences are
considered. The DL-based methods make few valid predictions for the unseen
complexes. Overall, we show that no DL-based method yet outperforms standard
docking methods when consideration of physical plausibility is taken into
account. The PoseBusters test suite will enable DL method developers to better
understand the limitations of current methods, ultimately resulting in more
accurate and realistic predictions.

##  2 Methods

Five DL-based and two classical docking methods were used to re-dock known
ligands into their respective proteins and the predicted ligand poses were
evaluated with the PoseBusters test suite. The following section describes the
docking methods, the data sets, and the PoseBusters test suite for checking
physicochemical consistency and structural plausibility of the generated
poses.

###  2.1 Docking methods

The selected five DL-based docking methods 6, 7, 8, 9, 10 cover a wide range
of DL-based approaches for pose prediction. Table 1 lists the methods and
their publications. In order to examine the ability of standard non-DL-based
methods to predict accurate chemically and physically valid poses, we also
included the well-established docking methods AutoDock Vina 28 and Gold 29.

Table 1: Selected DL-based docking methods. The selection includes five methodologically different DL-based docking methods published over the last two years. Method | Authors | Date | Search space
---|---|---|---
DeepDock 6 | Méndez-Lucio _et al._ | Dec 2021 | pocket
DiffDock 7 | Corso _et al._ | Feb 2023 | blind
EquiBind 8 | Stärk _et al._ | Feb 2022 | blind
TANKBind 9 | Lu _et al._ | Oct 2022 | blind
Uni-Mol 10 | Zhou _et al._ | Feb 2023 | pocket

The five DL-based docking methods can be summarised as follows. Full details
of each can be found in their respective references. DeepDock 6 learns a
statistical potential based on the distance likelihood between ligand heavy
atoms and points of the mesh of the surface of the binding pocket. DiffDock 7
uses equivariant graph neural networks in a diffusion process for blind
docking. EquiBind 8 applies equivariant graph neural networks for blind
docking. TankBind 9 is a blind docking method that uses a trigonometry-aware
neural network for docking in each pocket predicted by a binding pocket
prediction method. Uni-Mol 10 carries out docking with SE3-equivariant
transformers. All five DL-based docking methods are trained on subsets of the
PDBbind General Set 30 as detailed in Table 2. DeepDock is trained on v2019
and the other four are trained on v2020.

Table 2: Data sets used to train the selected five machine learning-based docking methods. All five DL-based methods were trained on subsets of the PDBBind 2020 General Set. Method | Training and validation set
---|---
DeepDock | PDBBind 2019 General Set without complexes included in CASF-2016 or those that fail pre-processing—$16\,367$ complexes
DiffDock, |
EquiBind | PDBbind 2020 General Set keeping complexes published before 2019 and without those with ligands found in test set—$17\,347$ complexes
TankBind | PDBbind 2020 General Set keeping complexes published before 2019 and without those failing pre-processing—$18\,755$ complexes
Uni-Mol | PDBBind 2020 General Set without complexes where protein sequence identity (MMSeq2) with CASF-2016 is above 40% and ligand fingerprint similarity is above 80%—$18\,404$ complexes

The docking protocols that were used to generate predictions with each method
and the software versions used are given in section S1 of the Supplementary
Information. Table 3 lists the search space definitions that we used for each
method. DeepDock and Uni-Mol require the definition of a binding site while
DiffDock, EquiBind, and TankBind are ‘blind’ docking methods that search over
the entire protein. We used the default search spaces for the DL-based methods
but larger than default search spaces for AutoDock Vina and Gold such that
they are more comparable with the blind docking DL-based methods. SI Figure S1
shows the search spaces for one example protein-ligand complex.

Table 3: Search spaces of the docking methods used.  Method | Search space
---|---
_Classical docking methods_ |
Gold | Sphere of radius 25 centered on the geometric centre of the crystal ligand heavy atoms
Vina | Cube with side length 25 centered on the geometric centre of crystal ligand heavy atoms
_DL-based docking methods_ |
DeepDock | Protein surface mesh nodes within 10 of any crystal ligand atom
Uni-Mol | Protein residues within 6 of any crystal ligand heavy atom
_DL-based blind docking methods_ |
DiffDock | Entire crystal protein
EquiBind | Chains of crystal protein which are within 10 of any crystal ligand heavy atom
TankBind | Pockets identified by P2Rank31

###  2.2 The PoseBusters test suite

The PoseBusters test suite is organised into three groups of tests. The first
checks chemical validity and contains tests for the chemical validity and
consistency relative to the input. The second group checks intramolecular
properties and tests for the ligand geometry and the ligand conformation’s
energy computed using the universal force field (UFF)32. The third group
considers intermolecular interactions and checks for protein-ligand and
ligand-cofactor clashes. Descriptions of all the tests PoseBusters performs in
the three sections are listed in Table 4. Molecule poses which pass all tests
in PoseBusters are ‘PB-valid’.

For evaluating docking predictions, PoseBusters requires three input files: an
SDF file containing the re-docked ligands, an SDF file containing the true
ligand(s), and a PDB file containing the protein with any cofactors. The three
files are loaded into RDKit molecule objects with the sanitisation option
turned off.

Table 4: Description of the checks used in the PoseBusters test suite. Test name | Description
---|---
Chemical validity and consistency |
File loads | The input molecule can be loaded into a molecule object by RDKit.
Sanitisation | The input molecule passes RDKit’s chemical sanitisation checks.
Molecular formula | The molecular formula of the input molecule is the same as that of the true molecule.
Bonds | The bonds in the input molecule are the same as in the true molecule.
Tetrahedral chirality | The specified tetrahedral chirality in the input molecule is the same as in the true molecule.
Double bond stereochemistry | The specified double bond stereochemistry in the input molecule is the same as in the true molecule.
Intramolecular validity |
Bond lengths | The bond lengths in the input molecule are within $0.75$ of the lower and $1.25$ of the upper bounds determined by distance geometry.
Bond angles | The angles in the input molecule are within $0.75$ of the lower and $1.25$ of the upper bounds determined by distance geometry.
Planar aromatic rings | All atoms in aromatic rings with 5 or 6 members are within 0.25 of the closest shared plane.
Planar double bonds | The two carbons of aliphatic carbon-carbon double bonds and their four neighbours are within 0.25 of the closest shared plane.
Internal steric clash | The interatomic distance between pairs of non-covalently bound atoms is above $0.8$ of the lower bound determined by distance geometry.
Energy ratio | The calculated energy of the input molecule is no more than $100$ times the average energy of an ensemble of $50$ conformations generated for the input molecule. The energy is calculated using the UFF32 in RDKit and the conformations are generated with ETKDGv3 followed by force field relaxation using the UFF with up to $200$ iterations.
Intermolecular validity |
Minimum protein-ligand distance | The distance between protein-ligand atom pairs is larger than $0.75$ times the sum of the pairs van der Waals radii.
Minimum distance to organic cofactors | The distance between ligand and organic cofactor atoms is larger than $0.75$ times the sum of the pairs van der Waals radii.
Minimum distance to inorganic cofactors | The distance between ligand and inorganic cofactor atoms is larger than $0.75$ times the sum of the pairs covalent radii.
Volume overlap with protein | The share of ligand volume that intersects with the protein is less than 7.5. The volumes are defined by the van der Waals radii around the heavy atoms scaled by $0.8$.
Volume overlap with organic cofactors | The share of ligand volume that intersects with organic cofactors is less than 7.5. The volumes are defined by the van der Waals radii around the heavy atoms scaled by $0.8$.
Volume overlap with inorganic cofactors | The share of ligand volume that intersects with inorganic cofactors is less than 7.5. The volumes are defined by the van der Waals radii around the heavy atoms scaled by $0.5$.

####  2.2.1 Chemical validity and consistency

The first test in PoseBusters checks whether the ligand passes RDKit’s
sanitisation. RDKit’s sanitisation processes information on the valency,
aromaticity, radicals, conjugation, hydridization, chirality tags, and
protonation to check whether a molecule can be represented as an octet-
complete Lewis dot structure 33. Passing RDKit’s sanitisation is a commonly-
used test for chemical validity in cheminformatics, for example in _de novo_
molecular generation 20.

The next test in PoseBusters checks for docking-relevant chemical consistency
between the predicted and the true ligands by generating ‘standard InChI’
strings 34 for the input and output ligands after removing isotopic
information and neutralising charges by adding or removing hydrogens where
possible. InChI is the _de facto_ standard for molecular comparison 35, and
the ‘standard InChI’ strings generated include the layers for the molecular
formula (/), molecular bonds (/c), hydrogens (/h), net charge (/q), protons
(/p), tetrahedral chirality (/t), and double bond stereochemistry (/b).
Standardisation of the ligand’s protonation and charge state is needed because
the stereochemistry layer is dependent on the hydrogen (/h), net charge (/q)
and proton (/p) layers. These can unexpectedly change during docking even
though most docking software considers the charge distribution and protonation
state of a ligand as fixed 12, 36. The normalisation protocol also removes the
stereochemistry information of double bonds in primary ketimines which only
depends on the hydrogen atom’s ambiguous location.

####  2.2.2 Intramolecular validity

The first set of physical plausibility tests in the PoseBusters test suite
validates bond lengths, bond angles, and internal distances between non-
covalently bound pairs of atoms in the docked ligand against the corresponding
limits in the distance bounds matrix obtained from RDKit’s Distance Geometry
module. To pass the tests, all molecular measurements must lie within the
user-specified tolerances. The tolerance used throughout this manuscript is 25
for bond lengths and bond angles and 30 for non-covalently bound pairs of
atoms e.g.: if a bond is less than 75 of the Distance Geometry bond length
lower bound, it is treated as anomalous. This was selected as all but one of
the crystal ligands in the Astex Diverse set and all of those in the
PoseBuster Benchmark set pass at this threshold.

The PoseBusters test for flatness checks that groups of atoms lie in a plane
by calculating the closest plane to the atoms and checking that all atoms are
within a user-defined distance from this plane. This test is performed for 5-
and 6-membered aromatic rings and non-ring non-aromatic carbon-carbon double
bonds. The chosen threshold of 0.25 admits all Astex Diverse and PoseBusters
Benchmark set crystal structures by a wide margin and as with all other
thresholds can be adjusted by the user.

The final test for intramolecular physicochemical plausibility carried out by
PoseBusters is an energy calculation to detect unlikely conformations. Our
metric for this is the ratio of the energy of the docked ligand conformation
to the mean of the energies of a set of 50 generated unconstrained
conformations as in Wills _et al._ 37. The conformations are generated using
RDKit’s ETKDGv3 conformation generator 38 followed by a force field relaxation
using the UFF 32 and up to $200$ iterations. The test suite rejects
conformations for which this ratio is larger than a user-specified threshold.
Wills _et al._ set a ratio of 7 based on the value where 95 of the crystal
ligands in the PDBbind data set are considered plausible 37. We selected a
less strict ratio of $100$ where only one structure each from the Astex
Diverse and PoseBuster Benchmark set is rejected.

####  2.2.3 Intermolecular validity

Intermolecular interactions are evaluated by two sets of tests in the
PoseBusters test suite. The first set checks the minimum distance between
molecules and the second checks the share of overlapping volume. Both sets of
tests report on intermolecular interactions of the ligand with four types of
molecules: the protein, organic cofactors, and inorganic cofactors.

For the distance-based intermolecular tests PoseBusters calculates the ratio
of the pairwise distance between pairs of heavy atoms of two molecules and the
sum of the two atoms’ van der Waals radii. If this ratio is smaller than a
user-defined threshold then the test fails. The default threshold is $0.75$
for all pairings. For inorganic cofactor-ligand pairings the covalent radii
are used. All crystal structures in the Astex Diverse set and all but one in
the PoseBuster Benchmark set pass at this threshold.

For the second set of intermolecular checks, PoseBusters calculates the share
of the van der Waals volume of the heavy atoms of the ligand that overlaps
with the van der Waals volume of the heavy atoms of the protein using RDKit’s
`ShapeTverskyIndex` function. The tests have a configurable scaling factor for
the volume-defining van der Waals radii and a threshold that defines how much
overlap constitutes a clash. A threshold is necessary because many crystal
structures already contain clashes. For example, Verdonk _et al._ found that
$81$ out of $305$ selected high-quality protein-ligand complexes from the PDB
contain steric clashes 26. The overlap threshold is 7.5 for all molecule
pairings and the scaling factor is $0.8$ for protein-ligand and organic
cofactor-ligand pairings and $0.5$ for inorganic cofactor-ligand pairings.

###  2.3 Quality of fit

PoseBusters calculates the minimum heavy-atom symmetry-aware root-mean-square
deviation (RMSD) between the predicted ligand binding mode and the closest
crystallographic ligand. Coverage, a metric often used for testing docking
methods, is the share of predictions that are within a user adjustable
threshold which by default is 2 RMSD. This value is arbitrary but commonly-
used and recommended for regular-size ligands 13.

###  2.4 Sequence identity

In this paper, sequence identity between two amino acid chains is the number
of exact residue matches after sequence alignment divided by the number of
residues of the query sequence. The sequence alignment used is the
Smith–Waterman algorithm 39 implemented in Biopython 40 using an open gap
score of $-11$ and an extension gap score of $-1$ and the `BLOSUM62`
substitution matrix. Unknown amino acid residues are counted as mismatches.

###  2.5 Molecular mechanics energy minimisation

Post-docking energy minimisation of the ligand structure in the binding pocket
was performed using the AMBER ff14sb force field 41 and the Sage small
molecule force field 42 in OpenMM 43. The protein files were prepared using
PDBfixer 43 and all protein atom positions were fixed in space only allowing
updates to the ligand atoms positions. Minimisation was performed until energy
convergence within 0.01.

###  2.6 Data

####  2.6.1 Astex Diverse set

The Astex Diverse set 27 published in 2007 is a set of hand-picked, relevant,
diverse, and high-quality protein-ligand complexes from the PDB 14. The
complexes were downloaded from the PDB as MMTF files 44 and PyMOL 45 was used
to remove solvents and all occurrences of the ligand of interest from the
complexes before saving the proteins with the cofactors in PDB files and the
ligands in SDF files.

####  2.6.2 PoseBusters Benchmark set

The PoseBusters Benchmark set is a new set of carefully-selected publicly-
available crystal complexes from the PDB. It is a diverse set of recent high-
quality protein-ligand complexes which contain drug-like molecules. It only
contains complexes released since 2021 and therefore does not contain any
complexes present in the PDBbind General Set v2020 used to train many of the
methods. Table S2 lists the steps used to select the 428 unique proteins and
428 unique ligands in the PoseBuster Benchmark set. The complexes were
downloaded from the PDB as MMTF files and PyMOL was used to remove solvents
and all occurrences of the ligand of interest before saving the proteins with
the cofactors in PDB files and the ligands in SDF files.

##  3 Results

The following section presents the analysis of the PoseBusters test suite on
the re-docked ligands of five DL-based docking methods and two standard non-
DL-based docking methods on the $85$ ligands of the Astex Diverse set and the
$428$ ligands of the PoseBusters Benchmark set into the receptors crystal
structures.

###  3.1 Results on the Astex Diverse set

Fig. 1:  Comparative performance of the docking methods. The Astex Diverse set
($85$ cases) was chosen as an easy test set containing many complexes the five
DL-based methods were trained on while the PoseBusters Benchmark set ($428$
cases) was chosen to be a difficult test set containing complexes none of the
methods was trained on. The striped bars show the share of predictions of each
method that have an RMSD within 2 and the solid bars show the subset that in
addition have valid geometries and energies, i.e., pass all PoseBuster tests
and are therefore ‘PB-Valid’. DiffDock appears to outperform the classical
methods on the Astex Diverse set when only binding mode RMSD is considered
(striped teal bars). However, when physical plausibility is also considered
(solid teal bars) _or_ when presented with the PoseBusters Benchmark set
(coral bars), AutoDock Vina and Gold outperform all DL-based methods.

Figure 1 shows the overall results of the seven (AutoDock Vina 12, Gold 26,
DeepDock 6, DiffDock 7, EquiBind 8, TankBind 9, Uni-Mol 10) docking methods on
the Astex Diverse set in ocean green. The striped bars show the performance
only in terms of RMSD coverage ($\text{RMSD}\leqslant\qty{2}{}$) and the solid
bars show the performance after also considering physical plausibility, i.e.,
only predictions which in addition pass all tests in PoseBusters and are
therefore PB-valid.

Fig. 2: Waterfall plot showing the PoseBusters tests as filters for the
TankBind predictions on the Astex Diverse data set. The tests in the
PoseBuster test suits are described in Table 4. The leftmost (dotted) bar
shows the number of complexes in the test set. The red bars show the number of
predictions that fail with each additional test going from left to right. The
rightmost (solid) bar indicates the number of predictions that pass all tests,
i.e. those that are ‘PB-Valid’. For the $85$ test cases in the Astex Diverse
set $50$ (59) predictions have RMSD within 2 RMSD and $5$ (5.9) pass all
tests. Figures S5 and S6 in the Supplementary Information show waterfall plots
for all methods and both data sets.

The Astex Diverse set is a well-established and commonly-used benchmark for
evaluating docking methods. Good performance on this set is expected because
the five DL-based methods evaluated here have been trained on most of these
complexes. $47$ of the $81$ complexes in the Astex Diverse set are in the
PDBbind 2020 General Set and $67$ out of the $81$ of the Astex Diverse set
proteins have more than 95 sequence identity with proteins found in PDBbind
2020 General Set. AutoDock Vina may also perform well on this data set because
the linear regression model behind the scoring function was trained on an
earlier version of PDBbind 12 which already included most of the Astex Diverse
set.

The RMSD criterion alone (striped green bars in Figure 1) gives the impression
that DiffDock (72) performs better than TankBind (59), Gold (67), AutoDock
Vina (58) and Uni-Mol (45). However, when we look closer, accepting only
ligand binding modes that are physically sensible, i.e., those predictions
that pass all PoseBusters tests and are therefore PB-valid (solid green bars
in Figure 1), many of the apparently impressive DL predictions are removed.
The best three methods when considering RMSD and physical plausibility are
Gold (64), AutoDock Vina (56), and DiffDock (47) followed by Uni-Mol (12),
DeepDock (11) and TankBind (5.9). DiffDock is therefore the only DL-based
method that has comparable performance to the standard methods on the Astex
Diverse set when considering physical plausibility of the predicted poses.

All five DL-based docking methods struggle with physical plausibility, but
even the poses produced by the classical methods Gold and AutoDock Vina do not
always pass all the checks. Figure 2 shows a waterfall plot that indicates how
many predicted binding modes fail each test. The waterfall plots for the
remaining methods are shown in SI Figure S5. The DL-based methods fail on
different tests. TankBind habitually overlooks stereochemistry, Uni-Mol very
often fails to predict valid bond lengths, and EquiBind tends to produce
protein-ligand clashes. The classical methods Gold and AutoDock Vina pass most
tests but also generated a few protein-ligand clashes. Figure 3 shows examples
of poses generated by the methods illustrating various failure modes.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

Fig. 3: Examples of failure modes that PoseBusters is able to detect.
Predictions are shown on the left with white carbons and the crystal
structures on the right have cyan carbons. Oxygen atoms are red, nitrogen
atoms are dark blue, chlorine atoms are green. Most of the shown predictions
have a RMSD within 2 but all are physically invalid.

The results on the Astex Diverse set suggest that despite what the
$\text{RMSD}\leqslant\qty{2}{}$ criterion would indicate, no DL-based method
outperforms classical docking methods when the physical plausibility of the
ligand binding mode is taken into account. However, DiffDock in particular is
capable of making a large number of useful predictions.

###  3.2 Results on the PoseBuster Benchmark set

The results of the seven (AutoDock Vina, Gold, DeepDock, DiffDock, EquiBind,
TankBind, Uni-Mol) docking methods on the PoseBuster Benchmark set are shown
in coral in Figure 1. The striped bars show the performance only in terms of
coverage (RMSD within 2Å) and the solid bars show the performance in terms of
PB-validity (passing all of PoseBusters tests).

The PoseBuster Benchmark set was designed to contain no complexes that are in
the training data for any of the methods. Performing well on this data set
requires a method to be able to generalise well.

All methods perform worse on the PoseBusters Benchmark set than the Astex
Diverse set. Gold (48) and AutoDock Vina (51) perform the best out of the
seven methods with (solid coral bars) and without (striped coral bars)
considering PB-validity. On the PoseBusters Benchmark set, the best performing
DL method, DiffDock (14), does not compete with the two standard docking
methods. Gold and AutoDock Vina again pass the most tests but for a few
protein-ligand clashes.

The waterfall plots in SI Figure S6 show which tests fail for each method on
the PoseBusters Benchmark set. Again, the methods have different merits and
shortcomings. Out of the five DL-based methods, DiffDock still produces the
most physically valid poses but few predictions lie within the 2 RMSD
threshold. EquiBind, Uni-Mol and TankBind generate almost no physically valid
poses that pass all tests. Uni-Mol has a relatively good RMSD score (23) but
struggles to predict planar aromatic rings and correct bond lengths.

Figure 4 shows the results of the docking methods on the PoseBusters Benchmark
set but stratified by the target protein receptor’s maximum sequence identity
with the proteins in the PDBbind 2020 General Set 30. As the DL-based methods
were all trained on subsets of the PDBbind 2020 General Set, this roughly
quantifies how different the test set protein targets are from those that the
methods were trained on. We bin the test cases into three categories low [$0$,
30], medium (30, 90], and high (90, 100] maximum percentage sequence identity.
Without considering physical plausibility (striped bars), the classical
methods appear to perform as well on the three protein similarity bins while
the DL-based methods perform worse on the proteins with lower sequence
identity. This suggests that the DL-based methods are overfitting to the
protein targets in their training sets.

Fig. 4:  Comparative performance of docking methods on the PoseBusters
Benchmark set stratified by sequence identity relative to the PDBBind General
Set v2020. The sequence identity is the maximum sequence identity between all
chains in the PoseBuster test protein and all chains in the PDBBind General
Set v2020. The striped bars show the share of predictions of each method that
have an RMSD within 2 and the solid bars show those predictions which in
addition pass all PoseBuster tests and are therefore PB-valid. The DL-based
methods perform far better on proteins that they are similar to those they
were trained on.

We also compared the performance of the docking methods on the PoseBusters
Benchmark set stratified by whether protein-ligand complexes contain cofactors
(SI Figure S3). Here, we loosely define cofactors as non-protein non-ligand
compounds such as metal ions, iron-sulfur clusters, and organic small
molecules in the crystal complex within 4.0 of any ligand heavy atom. About
45% of protein-ligand complexes in the PoseBusters Benchmark set have a
cofactor (SI Figure S2). The classical methods perform slightly better when a
cofactor is present while the DL-based docking methods perform worse on those
systems.

###  3.3 Results with pose-docking energy minimisation

In order to examine whether the outputs of the DL-based methods can be made
physically plausible we performed an additional post-docking energy
minimisation of the ligand structures in the binding pocket for the
PoseBusters Benchmark set (Figure 5). Again, striped bars indicate predictions
with $\text{RMSD}\leqslant\qty{2}{}$ while solid bars indicate which of those
are also PB-valid i.e., pass all PoseBusters tests. The figure shows that
post-docking energy minimisation significantly increases the number of
physically plausible structures of the DL-based methods DiffDock, DeepDock,
TankBind, and Uni-Mol but does not improve the poses predicted by AutoDock
Vina and Gold. The fact that energy minimisation is able to repair many of the
DL methods predicted poses and increase coverage shows that at least some
force field physics is missing from DL-based docking methods. However, even
with the energy minimisation step, the best DL-based docking method DiffDock
still performs worse than the classical methods Gold and AutoDock Vina.

Fig. 5:  Comparative performance of docking methods with post-pocking energy
minimisation of the ligand (while keeping the protein fixed) on the
PoseBusters Benchmark set. The striped bars show the share of predictions of
each method that have an RMSD within 2 of the crystal pose and the solid bars
show those predictions which in addition pass all PoseBuster tests and are
therefore PB-valid. Post-docking energy minimisation significantly improves
the relative physical plausibility of the DL-based methods’ predictions. This
indicates that force fields contain docking-relevant physics which is missing
from DL-based methods.

##  4 Discussion

We present PoseBusters, a test suite designed and built to identify chemically
inconsistent and physically implausible ligand poses predicted by protein-
ligand docking and molecular generation methods. We show the results of
applying the PoseBusters test suite to the output of seven different docking
methods, five current DL-based docking methods (DeepDock, DiffDock, EquiBind,
TankBind, and Uni-Mol) and two standard methods (AutoDock Vina and Gold).

We find that no DL-based docking method yet outperforms standard docking
methods when _both_ physical plausibility and binding mode RMSD is taken into
account. Our work demonstrates the need for physical plausibility to be taken
into account when assessing docking tools because it is possible to perform
well on an RMSD-based metric while predicting physically implausible ligand
poses (Figure 3). Using the tests in the PoseBusters test suite as an
additional criterion when developing DL-based docking methods will help
improve methods and the development of more accurate and realistic
predictions.

In addition, the individual tests in the PoseBusters test suite highlight
docking-relevant failure modes. The results show that Uni-Mol for example
predicts non-standard bond lengths and TankBind creates internal ligand
clashes. The ability to identify such failure modes in predicted ligand poses
makes PoseBusters a helpful tool for developers to identify inductive biases
that could improve their binding mode prediction methods.

Our results also show that, unlike classical docking methods, DL-based docking
methods do not generalise well to novel data. The performance of the DL-based
methods on the PoseBusters Benchmark set overall was poor and the subset of
the PoseBusters Benchmark set with low sequence identity to PDBbind 2020
revealed that DL-based methods are prone to overfitting to the proteins they
were trained on. Our analysis of the targets with sequence identity lower than
30 to any member of PDBbind General Set v2020 revealed that across all of the
DL-based docking methods almost no physically valid poses were generated
within the 2 threshold.

The most commonly-used train-test approach for building DL-based docking
models is time-based, e.g., complexes released before a certain date are used
for training and complexes released later for testing. Based on our results,
we argue that this is insufficient for testing generalisation to novel targets
and the sequence identity between the proteins in the training and test must
be reported on.

Post-docking energy minimisation of the ligand using force fields can
considerably improve the docking poses generated by DL-based methods. However,
even with an energy minimisation step, the best DL-based method, DiffDock,
does not outperform classical docking methods like Gold and AutoDock Vina.
This shows that at least some key aspects of chemistry and physics encoded in
force fields are missing from deep learning models.

The PoseBusters test suite provides a new criterion, PB-validity, beyond the
traditional “$\text{RMSD}\leqslant\qty{2}{}$” rule to evaluate the predictions
of new DL-based methods, and hopefully will help to identify inductive biases
needed for the field to improve docking and molecular generation methods,
ultimately resulting in more accurate and realistic predictions. PoseBusters
is made available as a pip-installable Python package and as open source code
under the BSD-3-Clause license at github.com/maabuu/posebusters. The next
generation of DL-based docking methods should aim to outperform standard
docking tools on both RMSD criteria and in terms of chemical consistency,
physical plausibility, and generalisability.

## Conflicts of interest

There are no conflicts to declare.

## References

  * De Ruyck _et al._ 2016 J. De Ruyck, G. Brysbaert, R. Blossey and M. Lensink, _Advances and Applications in Bioinformatics and Chemistry_ , 2016, Volume 9, 1–11.
  * Morris and Lim-Wilby 2008 G. M. Morris and M. Lim-Wilby, in _Molecular Modeling of Proteins_ , Humana Press, Totowa, NJ, 2008, pp. 365–382.
  * Lionta _et al._ 2014 E. Lionta, G. Spyrou, D. Vassilatis and Z. Cournia, _Current Topics in Medicinal Chemistry_ , 2014, 14, 1923–1938.
  * Patrick 2017 G. Patrick, in _An Introduction to Medicinal Chemistry_ , Oxford University Press, 2017, pp. 223–255.
  * Patrick 2017 G. Patrick, in _An Introduction to Medicinal Chemistry_ , Oxford University Press, 2017, pp. 349–394.
  * Méndez-Lucio _et al._ 2021 O. Méndez-Lucio, M. Ahmad, E. A. del Rio-Chanona and J. K. Wegner, _Nature Machine Intelligence_ , 2021, 3, 1033–1039.
  * Corso _et al._ 2023 G. Corso, H. Stärk, B. Jing, R. Barzilay and T. Jaakkola, International Conference on Learning Representations, 2023.
  * Stärk _et al._ 2022 H. Stärk, O. Ganea, L. Pattanaik, D. Barzilay and T. Jaakkola, International Conference on Machine Learning, 2022, pp. 20503–20521.
  * Lu _et al._ 2022 W. Lu, Q. Wu, J. Zhang, J. Rao, C. Li and S. Zheng, Advances in Neural Information Processing Systems, 2022, pp. 7236–7249.
  * Zhou _et al._ 2023 G. Zhou, Z. Gao, Q. Ding, H. Zheng, H. Xu, Z. Wei, L. Zhang and G. Ke, International Conference on Learning Representations, 2023.
  * Jones _et al._ 1995 G. Jones, P. Willett and R. C. Glen, _Journal of Molecular Biology_ , 1995, 245, 43–53.
  * Trott and Olson 2009 O. Trott and A. J. Olson, _Journal of Computational Chemistry_ , 2009, 455–461.
  * Cole _et al._ 2005 J. C. Cole, C. W. Murray, J. W. M. Nissink, R. D. Taylor and R. Taylor, _Proteins: Structure, Function, and Bioinformatics_ , 2005, 60, 325–332.
  * Berman 2000 H. M. Berman, _Nucleic Acids Research_ , 2000, 28, 235–242.
  * Shao _et al._ 2022 C. Shao, J. D. Westbrook, C. Lu, C. Bhikadiya, E. Peisach, J. Y. Young, J. M. Duarte, R. Lowe, S. Wang, Y. Rose, Z. Feng and S. K. Burley, _Structure_ , 2022, 30, 252–262.e4.
  * Hawkins _et al._ 2010 P. C. D. Hawkins, A. G. Skillman, G. L. Warren, B. A. Ellingson and M. T. Stahl, _Journal of Chemical Information and Modeling_ , 2010, 50, 572–584.
  * Friedrich _et al._ 2017 N.-O. Friedrich, C. de Bruyn Kops, F. Flachsenberg, K. Sommer, M. Rarey and J. Kirchmair, _Journal of Chemical Information and Modeling_ , 2017, 57, 2719–2728.
  * Urbaczek _et al._ 2011 S. Urbaczek, A. Kolodzik, J. R. Fischer, T. Lippert, S. Heuser, I. Groth, T. Schulz-Gasch and M. Rarey, _Journal of Chemical Information and Modeling_ , 2011, 51, 3199–3207.
  * Warren _et al._ 2012 G. L. Warren, T. D. Do, B. P. Kelley, A. Nicholls and S. D. Warren, _Drug Discovery Today_ , 2012, 17, 1270–1281.
  * Brown _et al._ 2019 N. Brown, M. Fiscato, M. H. Segler and A. C. Vaucher, _Journal of Chemical Information and Modeling_ , 2019, 59, 1096–1108.
  * Warren _et al._ 2006 G. L. Warren, C. W. Andrews, A.-M. Capelli, B. Clarke, J. LaLonde, M. H. Lambert, M. Lindvall, N. Nevins, S. F. Semus, S. Senger, G. Tedesco, I. D. Wall, J. M. Woolven, C. E. Peishoff and M. S. Head, _Journal of Medicinal Chemistry_ , 2006, 49, 5912–5931.
  * Ciancetta _et al._ 2014 A. Ciancetta, A. Cuzzolin and S. Moro, _Journal of Chemical Information and Modeling_ , 2014, 54, 2243–2254.
  * Onodera _et al._ 2007 K. Onodera, K. Satou and H. Hirota, _Journal of Chemical Information and Modeling_ , 2007, 47, 1609–1618.
  * Plewczynski _et al._ 2011 D. Plewczynski, M. Łaźniewski, R. Augustyniak and K. Ginalski, _Journal of Computational Chemistry_ , 2011, 32, 742–755.
  * Bolcato _et al._ 2019 G. Bolcato, A. Cuzzolin, M. Bissaro, S. Moro and M. Sturlese, _International Journal of Molecular Sciences_ , 2019, 20, 3558\.
  * Verdonk _et al._ 2003 M. L. Verdonk, J. C. Cole, M. J. Hartshorn, C. W. Murray and R. D. Taylor, _Proteins: Structure, Function, and Bioinformatics_ , 2003, 52, 609–623.
  * Hartshorn _et al._ 2007 M. J. Hartshorn, M. L. Verdonk, G. Chessari, S. C. Brewerton, W. T. M. Mooij, P. N. Mortenson and C. W. Murray, _Journal of Medicinal Chemistry_ , 2007, 50, 726–741.
  * Forli _et al._ 2016 S. Forli, R. Huey, M. E. Pique, M. F. Sanner, D. S. Goodsell and A. J. Olson, _Nature Protocols_ , 2016, 11, 905–919.
  * Jones _et al._ 1997 G. Jones, P. Willett, R. C. Glen, A. R. Leach and R. Taylor, _Journal of Molecular Biology_ , 1997, 267, 727–748.
  * Liu _et al._ 2017 Z. Liu, M. Su, L. Han, J. Liu, Q. Yang, Y. Li and R. Wang, _Accounts of Chemical Research_ , 2017, 50, 302–309.
  * Krivák and Hoksza 2018 R. Krivák and D. Hoksza, _Journal of Cheminformatics_ , 2018, 10, 39.
  * Rappe _et al._ 1992 A. K. Rappe, C. J. Casewit, K. S. Colwell, W. A. Goddard and W. M. Skiff, _Journal of the American Chemical Society_ , 1992, 114, 10024–10035.
  * Landrum _et al._ 2023 G. Landrum, P. Tosco, B. Kelley, Ric, Sriniker, Gedeck, D. Cosgrove, R. Vianello, NadineSchneider, E. Kawashima, D. N, A. Dalke, G. Jones, B. Cole, M. Swain, S. Turk, AlexanderSavelyev, A. Vaucher, M. Wójcikowski, I. Take, D. Probst, V. F. Scalfani, K. Ujihara, G. Godin, A. Pahl, F. Berenger, JLVarjo, Jasondbiggs, Strets123 and JP, _RDKit Q3 2022 Release_ , Zenodo, 2023.
  * Heller _et al._ 2015 S. R. Heller, A. McNaught, I. Pletnev, S. Stein and D. Tchekhovskoi, _Journal of Cheminformatics_ , 2015, 7, 1–34.
  * Goodman _et al._ 2021 J. M. Goodman, I. Pletnev, P. Thiessen, E. Bolton and S. R. Heller, _Journal of Cheminformatics_ , 2021, 13, 40.
  * Torres _et al._ 2019 P. H. M. Torres, A. C. R. Sodero, P. Jofily and F. P. Silva-Jr, _International Journal of Molecular Sciences_ , 2019, 20, 4574\.
  * Wills _et al._ 2022 S. Wills, R. Sanchez-Garcia, S. D. Roughley, A. Merritt, R. E. Hubbard, T. Dudgeon, J. Davidson, F. von Delft and C. M. Deane, _The Use of a Graph Database Is a Complementary Approach to a Classical Similarity Search for Identifying Commercially Available Fragment Merges_ , 2022.
  * Riniker and Landrum 2015 S. Riniker and G. A. Landrum, _Journal of Chemical Information and Modeling_ , 2015, 55, 2562–2574.
  * Smith and Waterman 1981 T. Smith and M. Waterman, _Journal of Molecular Biology_ , 1981, 147, 195–197.
  * Cock _et al._ 2009 P. J. A. Cock, T. Antao, J. T. Chang, B. A. Chapman, C. J. Cox, A. Dalke, I. Friedberg, T. Hamelryck, F. Kauff, B. Wilczynski and M. J. L. De Hoon, _Bioinformatics_ , 2009, 25, 1422–1423.
  * Maier _et al._ 2015 J. A. Maier, C. Martinez, K. Kasavajhala, L. Wickstrom, K. E. Hauser and C. Simmerling, _Journal of Chemical Theory and Computation_ , 2015, 11, 3696–3713.
  * Boothroyd _et al._ 2023 S. Boothroyd, P. K. Behara, O. Madin, D. Hahn, H. Jang, V. Gapsys, J. Wagner, J. Horton, D. Dotson, M. Thompson, J. Maat, T. Gokey, L.-P. Wang, D. Cole, M. Gilson, J. Chodera, C. Bayly, M. Shirts and D. Mobley, _Development and Benchmarking of Open Force Field 2.0.0 — the Sage Small Molecule Force Field_ , 2023.
  * Eastman _et al._ 2017 P. Eastman, J. Swails, J. D. Chodera, R. T. McGibbon, Y. Zhao, K. A. Beauchamp, L.-P. Wang, A. C. Simmonett, M. P. Harrigan, C. D. Stern, R. P. Wiewiora, B. R. Brooks and V. S. Pande, _PLOS Computational Biology_ , 2017, 13, e1005659.
  * Bradley _et al._ 2017 A. R. Bradley, A. S. Rose, A. Pavelka, Y. Valasatava, J. M. Duarte, A. Prlić and P. W. Rose, _PLOS Computational Biology_ , 2017, 13, e1005575.
  * Schrödinger, LLC 2015 Schrödinger, LLC, _The PyMOL Molecular Graphics System_ , 2015\.


* Taylor et al. (2018) Taylor, P. L., Kitching, T. D., & McEwen, J. D. 2018, Physical Review D, 98, doi: 10.1103/PhysRevD.98.043532
  * Valentino et al. (2021) Valentino, E. D., Mena, O., Pan, S., et al. 2021, Classical and Quantum Gravity, 38, doi: 10.1088/1361-6382/ac086d
  * Valogiannis et al. (2019) Valogiannis, G., Bean, R., & Aviles, A. 2019, Journal of Cosmology and Astroparticle Physics, 2020, doi: 10.1088/1475-7516/2020/01/055
  * Verde et al. (2019) Verde, L., Treu, T., & Riess, A. G. 2019, Nature Astronomy, 3, 891, doi: 10.1038/s41550-019-0902-0
  * Wang et al. (2023) Wang, Z., Mirpoorian, S. H., Pogosian, L., Silvestri, A., & Zhao, G.-B. 2023, Journal of Cosmology and Astroparticle Physics, 2023, doi: 10.1088/1475-7516/2023/08/038
  * Will (2014) Will, C. M. 2014, Living Reviews in Relativity 2014 17:1, 17, 1, doi: 10.12942/LRR-2014-4
  * Winther et al. (2015) Winther, H. A., Schmidt, F., Barreira, A., et al. 2015, Monthly Notices of the Royal Astronomical Society, 454, 4208, doi: 10.1093/mnras/stv2253
  * Ye & Silvestri (2024) Ye, G., & Silvestri, A. 2024.  https://arxiv.org/abs/2407.02471v1
  * Zhao et al. (2010) Zhao, G. B., Giannantonio, T., Pogosian, L., et al. 2010, Phys.Rev.D, 81, doi: 10.1103/PHYSREVD.81.103510
  * Zhao et al. (2008) Zhao, G.-B., Pogosian, L., Silvestri, A., & Zylberberg, J. 2008, Physical Review D - Particles, Fields, Gravitation and Cosmology, 79, doi: 10.1103/PhysRevD.79.083513
  * Zhao et al. (2009) —. 2009, Physical Review Letters, 103, doi: 10.1103/PhysRevLett.103.241301
  * Zucca et al. (2019a) Zucca, A., Pogosian, L., Silvestri, A., Wang, Y., & Zhao, G.-B. 2019a, Physical Review D, 101, doi: 10.1103/PhysRevD.101.043518
  * Zucca et al. (2019b) Zucca, A., Pogosian, L., Silvestri, A., & Zhao, G.-B. 2019b, Journal of Cosmology and Astroparticle Physics, 2019, doi: 10.1088/1475-7516/2019/05/001
  * Zumalacarregui et al. (2016) Zumalacarregui, M., Bellini, E., Sawicki, I., Lesgourgues, J., & Ferreira, P. G. 2016, Journal of Cosmology and Astroparticle Physics, 2017, doi: 10.1088/1475-7516/2017/08/019
  * Zumalacárregui (2020) Zumalacárregui, M. 2020, Phys.Rev.D, 102, doi: 10.1103/PHYSREVD.102.023523


#  BotEval: Facilitating Interactive Human Evaluation

Hyundong Cho1 Thamme Gowda2 Yuyang Huang1 Zixun Lu1 Tianli Tong1 Jonathan May1

1University of Southern California, Information Sciences Institute
2Microsoft Translator
<EMAIL_ADDRESS>
###### Abstract

Following the rapid progress in natural language processing (NLP) models,
language models are applied to increasingly more complex interactive tasks
such as negotiations and conversation moderations. Having human evaluators
directly interact with these NLP models is essential for adequately evaluating
the performance on such interactive tasks. We develop BotEval, an easily
customizable, open-source, evaluation toolkit that focuses on enabling human-
bot interactions as part of the evaluation process, as opposed to human
evaluators making judgements for a static input. BotEval balances flexibility
for customization and user-friendliness by providing templates for common use
cases that span various degrees of complexity and built-in compatibility with
popular crowdsourcing platforms. We showcase the numerous useful features of
BotEval through a study that evaluates the performance of various chatbots on
their effectiveness for conversational moderation and discuss how BotEval
differs from other annotation tools.

BotEval: Facilitating Interactive Human Evaluation

Hyundong Cho1 Thamme Gowda2 Yuyang Huang1

Zixun Lu1 Tianli Tong1 Jonathan May1 1University of Southern California,
Information Sciences Institute 2Microsoft Translator<EMAIL_ADDRESS>

##  1 Introduction

As natural language processing (NLP) models become more versatile with the
recent advances of language models and their instruction-tuned counterparts
Ouyang et al. (2022), it is becoming more common to create language agents
Sumers et al. (2023) and apply them to complex interactive tasks, such as
negotiations Chawla et al. (2021a), conversational moderation Cho et al.
(2023), reasoning-guided response generation Zhou et al. (2022), and
personalized response generation Liu et al. (2023).

As noted by Smith et al. (2022), the evaluation methodology plays a critical
role in accurately comparing models. For example, rankings between dialogue
models can change depending on whether they are evaluated based on single-turn
responses or full conversations. In addition, Cho et al. (2023) found the
evaluators point of view when evaluating a model is also an important factor.
They showed that human evaluators perceived conversational moderators as more
effective in making the evaluators become more cooperative and respectful when
the evaluators directly interacted with the moderators while acting as the
moderated user (first person point of view) compared to when they evaluated a
completed interaction between a moderator and a moderated user as a bystander
(third person point of view). However, these factors are overlooked in
previous approaches that have focused on a simplified evaluation, such as
comparing two complete conversations or individual responses Smith et al.
(2022); Li et al. (2019), or specific dialogue applications such as task-
oriented dialogue Cucurnia et al. (2021); Collins et al. (2019). Therefore, it
is important to develop evaluation tools that enable an environment that
evaluates models in a setting that best encapsulates how humans actually
interact with models.

To facilitate accurate human evaluations of complex interactive tasks, we
developed BotEval,111Source code and documentation for BotEval can be found at
https://github.com/isi-nlp/boteval. We make the demo video of BotEval
available at https://justin-cho.com/boteval. In addition, a live demo of
BotEval is also available at https://spolin.isi.edu/boteval-dev1 where
reviewers can complete a sample human evaluation task. a comprehensive
evaluation toolkit that focuses on enabling human - bot222We use bot loosely
to describe any AI system that a human being interacts with. interactions as
part of the human evaluation process. For flexibility, it is dynamically
configurable to accommodate as many human agents and model agents to interact
with each other simultaneously with a custom dialogue manager. It is also
designed with modular components, such as the interaction interface,
instructions, and survey, so that they can be individually adapted to
accommodate various use cases. While maintaining generalizability, BotEval
strives to maximize user-friendliness by providing templates for frequent use
cases that involve human evaluation where a human evaluator must interact with
a NLP model, multiple models, or another human being to measure human
performance. In addition, it is integrated with Amazon Mechanical Turk
(AMT)333https://www.mturk.com for crowdsourcing. It can also be deployed
independently of these platforms so that it can be used for internal
annotations or used with survey tools that allow for external links, such as
Qualtrics444https://www.qualtrics.com and Prolific.555https://www.prolific.com

To showcase the usefulness of BotEval and demonstrate its key features, we
share a case study that uses BotEval for evaluating models on their
performance on conversational moderation Cho et al. (2023). In this study,
BotEval is used to conduct various evaluations: (i) human-bot interactions to
compare models; (ii) human-human interactions to measure a human performance
threshold; and (iii) completed human-bot interactions by another evaluator to
measure evaluator consistency and third-person point of view (POV) results.

In summary, BotEval’s main contributions are:

  * •

An open-source and customizable evaluation tool for interactive NLP tasks that
incorporates human-bot and human-human interactions into the evaluation
process.

  * •

Detailed documentation and templates for various use cases to make
modifications easy.

  * •

Flexible deployment options with built-in integration with popular
crowdsourcing platforms such as AMT and Prolific.

  * •

Evaluation task management features that facilitate task monitoring and
managing crowdsource workers.

  * •

Dynamically configurable interaction logic with custom dialogue manager and
multi-human and multi-bot evaluation settings.

Figure 1: A snapshot of the admin point of view of an evaluation interface
with a completed evaluation example. The interface is identical for the
evaluator except for the text that shows the evaluator’s worker ID (hidden
with asterisks in the figure for privacy). The three main components of the
user-facing interface are the 1 conversation pane, 2 simple instruction pane,
and the 3 survey pane.

##  2 BotEval System Overview

BotEval is a web application that provides an evaluation interface, what the
human evaluators (i.e., crowdsource workers) see (Section 2.1), and an
administrator dashboard, what the administrator uses to manage the evaluation
task and evaluators (Section 2.2). We recommend that the bots that evaluators
interact with are provided as separate APIs that BotEval can make queries to,
as this isolates the management of the bot deployment and BotEval (Section
2.3). Human evaluators can be flexibly set to crowdsource workers from AMT or
Prolific or any other evaluators with internet access by having them create an
account directly for a deployment of BotEval using a public external link. An
evaluation task is configured with a central YAML config file that identifies
the frontend components to use, the deployment environment, and the
crowdsourcing platform to use.

###  2.1 Evaluation interface

A sample evaluation interface for the case study later described in Section
4.1 is shown in Figure 1.

The evaluation interface consists of three main components: 1 Conversation
pane: a section where the interaction between the human and the bot takes
place. This pane can be easily customized to contain seed conversations to
serve as initial starting points for interactions to continue off of or it can
instead contain any piece of text or completed conversation without requiring
any interactions from the evaluators, making BotEval also suitable for simpler
annotation tasks. 2 Instruction pane: this is an optional section that shows
the main directions. Evaluators can see detailed instructions by clicking on
the detailed instructions button. Administrators can choose to show detailed
instructions as part of the consent form if one is needed to make sure that
evaluators have read them. 3 Survey pane: this is where the human evaluators
provide their evaluations. In the given example, it is configured to only be
shown after the human evaluators have interacted with the bot for a set number
of turns.

The conversation pane and instruction pane is configurable by providing custom
HTML scripts, while the survey pane is even more easily customizable by
configuring a YAML config file. An example of the YAML config file is shown in
Appendix A.1. An optional consent form can be shown to evaluators as well,
which is also managed with a separate HTML file. Further detail on how the
consent form can be configured is in Appendix A.2.

###  2.2 Administrator dashboard

BotEval’s administrator dashboard provides numerous features for managing
evaluation tasks and evaluators. Its main benefit is a GUI that enables a non-
technical user to easily become an administrator for human evaluation tasks.
The topics page, shown in Figure 2, is one view that allows the management of
launching and deleting tasks. A topic refers to any predetermined context,
such as seed conversation or external information relevant for the evaluation
task. These topics are provided to BotEval as a JSON file. If a user is
interested in general open-domain dialogue evaluation, measuring a model’s
general conversational capabilities, they can use a dummy topic file that
contains an empty dictionary. This will launch an evaluation task that starts
a conversation from scratch, with the human evaluator initiating the first
turn.

After launching tasks, users can use the administrator dashboard to
conveniently examine tasks that are completed or in progress with the same
interface that the evaluators used to complete the task to easily visualize
their work rather than examining a database or JSON file, as shown in Figure
1. The user can also directly export individual JSON files of the collected
data if needed. Also, tasks can be deleted in batches using the parallel
management tool shown in 1 of Figure 2.

In addition to these features, we provide convenient AMT-specific features for
managing workers and tasks known as human intelligence tasks (HITs). One of
the most convenient features is being able to directly assign and remove
qualifications for workers after examining their work without having to leave
the administrator dashboard. This is an important convenience feature for
ensuring the quality of work for human evaluations are kept to the desired
standard by blocking unreliable workers. Another is being able to make bonus
payments directly after examining the completed task, which is useful when
each task is expected to involve variable rewards, such as to account for each
HIT taking a different amount of time to complete.

Figure 2: A snapshot of the topics page of the admin dashboard. 1 is a
parallel management tool that enables setting global configurations such as
how many tasks each evaluator is allowed to complete and launching or deleting
multiple tasks at once. 2 is a topics table that shares more information about
each topic, such as its name, how many tasks have been created, and when they
were created. 3 is a list of parameters that can be chosen for launching a
task, which includes parameters that can be passed on to API queries for the
bots.

###  2.3 Bot customization

Users are given multiple options to choose how they will service the bot that
they want to evaluate, but the recommended setup is to set up a separate
RESTful API and defining a logic within BotEval to interface with this API. As
shown in 3 in Figure 2, users can define task-specific parameters for bots
that get passed on to the API if the API allows for it. This is useful if you
are using the same model but adjusting the instruction prompt (e.g., using
OpenAI endpoints). While BotEval users have the option to launch bots
simultaneously on the same server with BotEval’s process, it is more efficient
to separately manage human evaluation tasks and the processes that load and
query NLP models because most NLP models are better served with GPUs for
reducing latency.

###  2.4 Sourcing human evaluators

BotEval can be customized to use with any crowdsourcing platform, and it is
designed to be directly used with many popular ones such as AMT, Prolific, and
Qualtrics. If the goal is to do internal annotations, the setup is even
simpler as the user only has to configure BotEval to not use any. Then the
user can share their custom URL with the evaluators, where they can sign up
and directly work on tasks that are made available to them without going
through any other platform.

##  3 System Architecture

An overview of BotEval’s system architecture is shown in Figure 3. BotEval is
a web application (i.e., a client-server model). We describe the front- and
back-end technology stacks in the following sections.

###  3.1 Frontend

The frontend is a simple web interface (i.e., HTML) created with Bootstrap
stylesheet. While the majority of the HTML structure is constructed on the
server side using Jinja2, some dynamic updates such as responses coming from
bots or other participants in the interaction are achieved using AJAX and
RESTful APIs.

Figure 3: BotEval system architecture. We use popular frameworks that are well
documented and easy to use.

###  3.2 Backend

The backend is implemented in Python language using Flask framework, following
a model-view-controller architecture pattern. Models are implemented using
Python classes and stored in a relational database, specifically SQLite. In
addition, we use SQLAlchemy, an object-relational mapper, to abstract the
mapping between Python classes and database tables. For views, Flask uses
Jinja2 for server side templating of HTML pages. Controllers are based on
Flask’s builtin URL routers and RESTful API constructs.

While internally our server is an HTTP server, crowdsourcing platforms such as
AMT require annotation interface be served via secure connections (HTTPS).
HTTPS can be enabled by obtaining and installing an SSL/TLS certificate. We
use free certificates from Certbot,666https://certbot.eff.org/ and configure
Nginx777https://nginx.org/en/ as a reverse proxy server for HTTPS requests.

Some scenarios may require several simultaneous instances of BotEval to
facilitate multiple annotation tasks, and obtaining SSL certificate for each
instance maybe cumbersome. We address this problem by using a different TCP
port for each instance, and configuring a single Nginx (with SSL certificate)
route requests for all instances.

##  4 Case Study and Use Cases

###  4.1 Case study: conversational moderation evaluation

To showcase the usefulness of BotEval, we share a case study that uses BotEval
to conduct a study on how effective various zero-shot instruction-tuned
language models (ITLM) and dialogue model are in performing conversational
moderation (CM) Cho et al. (2023).888The BotEval template for this work is
available at https://github.com/isi-nlp/isi_darma/tree/main/boteval-darma-
task. Instead of iron-fisted approaches to moderation such as deleting
comments or banning users, which may exacerbate societal polarization as these
users find refuge in echo chambers, CM seeks to have moderators interact with
users exhibiting problematic behavior to guide them back to more constructive
and respectful conversations.

This study makes full use of BotEval as it requires evaluating multiple bots
by interacting with them for a preset number of turns (in this case 3),
starting with a variety of conversation stubs. The evaluations were conducted
with all desired configurations simultaneously to get the most representative
and fair results that is not affected by any confounding factors such as
recency bias. The evalution was conducted with AMT, and being able to easily
monitor evaluations enabled rapid iterations of updating the instructions and
giving feedback to the evaluators.

Therefore, BotEval was integral in being able to refine the evaluation study
efficiently and ultimately collect statistically meaningful results for an
interactive evaluation setup. The study showed that prompt-engineered ITLMs
outperformed prosocial dialogue models and that a conflict resolution prompt
based on the Socratic method was the best performing prompt. In addition, one
of this work’s central findings was discovering that there are differences
between evaluation results when the models were evaluated from a first person
point of view (POV) and a third person POV. With BotEval, collecting human
evaluations in these two different settings was a simple change of updating
the topics file such that the conversation stubs were the completed
conversations, rewording the questions such that it is in third person POV,
and setting the number of turns required for human evaluators to interact with
the bots to zero.

Name Human-bot interaction-focused Crowdsource integration Multi-human & bot
support Language BotEval (Ours) ✓ ✓ ✓ Python Mephisto Urbanek and Ringshia
(2023) ✗ ✓ ✗ Python Pigeon11 ✗ ✗ ✗ Python MATILDA Cucurnia et al. (2021) ✗ ✓ ✗
Python LIDA Collins et al. (2019) ✗ ✓ ✗ Python INCEpTION Klie et al. (2018) ✗
✗ ✗ Java GATE Cunningham (2002) ✗ ✗ ✗ Java BRAT Stenetorp et al. (2012) ✗ ✗ ✗
Python doccano Nakayama et al. (2018) ✗ ✗ ✗ Python Potato Pei et al. (2022) ✗
✗ ✗ Python Argilla10 ✗ ✗ ✗ Python Prodigy12 ✗ ✗ ✗ Python DialogueView Yang and
Heeman (2005) ✗ ✗ ✗ TcK/TK DART Weisser (2016) ✗ ✗ ✗ Perl Anvil Kipp (2001) ✗
✗ ✗ Java EZCAT Guibon et al. (2022) ✗ ✗ ✗ Javascript

Table 1: Comparison overview with other annotation tools. BotEval innately
supports evaluations that require human-bot interactions and allow for
multiple human agents or bot agents to be involved in each evaluation sample.

###  4.2 Main use cases

BotEval’s main differentiation with previous annotation tools and frameworks
is that it is focused on, but not limited to, interactive use cases. In other
words, it is useful when the annotated data is not static, e.g., bot responses
over multiple turns or other dynamic outputs that can change based on user
interaction. Therefore, BotEval is appealing for evaluating or collecting data
for conversational tasks that usually require multi-turn interactions for
fulfilling the goal, rather than a single generated output. Many real-life
tasks go through multi-turn interactions, such as negotiations Chawla et al.
(2021b), counseling Mehta et al. (2022), and improvisational theater Cho and
May (2020). As artificial systems become more capable, more will be applied to
completing these complex multi-turn tasks, and BotEval will serve as a handy
starting point for facilitating their evaluation.

Although BotEval was designed for interactive tasks, BotEval can also be
easily adapt for simple static annotation tasks by simplifying the
conversation pane in Figure 1. This pane can contain any other modality such
as images, video, and audio, and adjusting the instruction and survey panes
accordingly can make BotEval also suitable for text classification or
conversation-level or turn-level comparisons, similar to Smith et al. (2022).
As BotEval gets actively used for more research studies, we will be able to
provide a variety of templates that accommodate a comprehensive set of use
cases, further lowering the effort required to conduct effective human
evaluation for new studies.

##  5 Related Work

In Table 1, we compare BotEval with other related annotation tools and discuss
differences further here.

###  5.1 General text annotation tools

A popular general annotation tool is Mephisto Urbanek and Ringshia (2023),
which started by isolating the crowdsourcing features from ParlAI Miller et
al. (2017). Mephisto provides a general annotation framework that interfaces
with Amazon Mechanical Turk and Prolific and includes basic templates for
simple annotation tasks. BotEval adapted many of its AMT integration features,
but Mephisto is not customized for common interactive data annotation and
evaluation use cases, and thus requires nontrivial effort to create a human
evaluation environment for interactive NLP tasks where a human evaluator needs
to interact with a bot or another human and then evaluate their performance.
ParlAI still provides templates for Mephisto for human-bot
interactions999https://parl.ai/docs/tutorial_crowdsourcing.html, but it is not
easy to use with a dialogue model that is not developed with ParlAI. With
BotEval, we also provide a GUI administrator dashboard for task and worker
management, which is absent in ParlAI and Mephisto.

GATE Cunningham (2002) and INCEpTION Klie et al. (2018) are annotation tools
that provide many predefined features, but they are also not designed for
interactive human evaluations. Other simpler general text annotation tools
that share similar limitations are Doccano Nakayama et al. (2018), brat
Stenetorp et al. (2012), Argilla101010https://argilla.io, Potato Pei et al.
(2022) and Pigeon111111https://github.com/agermanidis/pigeon, which are web-
based annotation tools that enable rapid annotations for text classification
and machine translation. Prodigy121212https://prodi.gy is a commercial
annotation tool for text annotations that provides similar features.

###  5.2 Dialogue annotation tools and evaluation methodologies

A prominent set of annotation tools specific to dialogue are centered around
task-oriented dialogue Budzianowski et al. (2018). LIDA Collins et al. (2019)
is an annotation tool that provides useful features for efficiently making
turn-level annotations, incorporating model-provided label recommendations to
speed up annotations, and resolving inter-annotation disagreements. MATILDA
Cucurnia et al. (2021) builds on LIDA for multilingual support and improved
management of crowdsourcing tasks among multiple workers. However, they do not
have built-in compatibility with popular crowdsourcing platforms and do not
support human-bot interactions to take place within the crowdsourcing task. A
lightweight option for dialogue annotations is EZCAT Guibon et al. (2022),
which provides a web-based serverless annotation framework that focuses on
enhanced accessibility for conversation-level and turn-level annotations.

Other work have created tools for multimodal annotations or speech-based
annotations. Anvil Kipp (2001) provides a multi-modal dialogue annotation tool
that enables annotation of audiovisual content. DialogueView Yang and Heeman
(2005) is an annotation tool that is focused on segmenting audio
conversations. DART Weisser (2016) focuses on enabling efficient annotations
of speech acts and linguistic criteria to facilitate corpus-based research
into pragmatics. Text is still the primary focus of BotEval and the templates
we provide, but BotEval remains general enough to be adapted to such cases as
well by modifying the templates we provide.

##  6 Conclusion

We presented BotEval and its usefulness in collecting human evaluations for
interactive tasks that require live human-bot interactions through a case
study of evaluating various language models on their ability to
conversationally moderate online discussions. BotEval provides a customizable
interface that can be adapted for various evaluation and annotation use cases
while also providing integration with popular crowdsourcing platforms and task
management features. We hope that this work will serve as an important
foundation for setting up custom interactive human evaluation tasks that
facilitate our understanding of more complex NLP systems as they become
increasingly sophisticated and capable.

## Limitations

We designed BotEval to be modular such that customizing existing templates and
modifying the dialogue manager’s logic is simple, but it is yet not configured
so that the task management process, shown in Figure 2 is independent of the
process that serves the evaluation, shown in Figure 1. This means that any
updates to BotEval that help with task management cannot be applied without
restarting evaluation tasks that were launched already, which will interfere
with any concurrent tasks that evaluators are working on. While inconvenient,
this has not been a major issue as restarting can be done quickly such that it
does not interfere the work of many evaluators and this doesn’t mean that
existing crowdsourcing tasks, as in the case of AMT, will be deleted and need
to be relaunched again.

Another challenge for using BotEval may arise from the difficulty of managing
a separate process that serves the bots that the human evaluators will
interact with. However, if the BotEval user is able to launch a bot as part of
BotEval, refactoring the code for that bot such that its responses are
accessed through an API instead is a simple modification with plenty of online
tutorials and tools, such as FastAPI.131313https://github.com/tiangolo/fastapi

## Acknowledgements

This material is based upon work supported by the Defense Advanced Research
Projects Agency (DARPA) under Agreement Nos. HR00112290025 and HR00112490374.
Approved for public release; distribution is unlimited.

## References

  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Chawla et al. (2021a) Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. 2021a.  Casino: A corpus of campsite negotiation dialogues for automatic negotiation systems.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3167–3185.
  * Chawla et al. (2021b) Kushal Chawla, Jaysa Ramirez, Rene Clever, Gale Lucas, Jonathan May, and Jonathan Gratch. 2021b.  CaSiNo: A corpus of campsite negotiation dialogues for automatic negotiation systems.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3167–3185, Online. Association for Computational Linguistics.
  * Cho et al. (2023) Hyundong Cho, Shuai Liu, Taiwei Shi, Darpan Jain, Basem Rizk, Yuyang Huang, Zixun Lu, Nuan Wen, Jonathan Gratch, Emilio Ferrara, and Jonathan May. 2023.  Can language model moderators improve the health of online discourse?
  * Cho and May (2020) Hyundong Cho and Jonathan May. 2020.  Grounding conversations with improvised dialogues.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2398–2413, Online. Association for Computational Linguistics.
  * Collins et al. (2019) Edward Collins, Nikolai Rozanov, and Bingbing Zhang. 2019.  LIDA: Lightweight interactive dialogue annotator.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations_ , pages 121–126, Hong Kong, China. Association for Computational Linguistics.
  * Cucurnia et al. (2021) Davide Cucurnia, Nikolai Rozanov, Irene Sucameli, Augusto Ciuffoletti, and Maria Simi. 2021.  MATILDA - multi-AnnoTator multi-language InteractiveLight-weight dialogue annotator.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_ , pages 32–39, Online. Association for Computational Linguistics.
  * Cunningham (2002) Hamish Cunningham. 2002.  Gate, a general architecture for text engineering.  _Computers and the Humanities_ , 36:223–254.
  * Guibon et al. (2022) Gaël Guibon, Matthieu Labeau, Luce Lefeuvre, and Chloé Clavel. 2022.  Ezcat: an easy conversation annotation tool.  In _Proceedings of the 13th Language Resources and Evaluation Conference_ , Marseille, France. European Language Resources Association.
  * Kipp (2001) Michael Kipp. 2001.  Anvil-a generic annotation tool for multimodal dialogue.  In _Seventh European conference on speech communication and technology_. Citeseer.
  * Klie et al. (2018) Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart De Castilho, and Iryna Gurevych. 2018.  The inception platform: Machine-assisted and knowledge-oriented interactive annotation.  In _Proceedings of the 27th international conference on computational linguistics: system demonstrations_ , pages 5–9.
  * Li et al. (2019) Margaret Li, Jason Weston, and Stephen Roller. 2019.  Acute-eval: Improved dialogue evaluation with optimized questions and multi-turn comparisons.  _arXiv preprint arXiv:1909.03087_.
  * Liu et al. (2023) Shuai Liu, Hyundong Cho, Marjorie Freedman, Xuezhe Ma, and Jonathan May. 2023.  Recap: Retrieval-enhanced context-aware prefix encoder for personalized dialogue response generation.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8404–8419.
  * Mehta et al. (2022) Maitrey Mehta, Derek Caperton, Katherine Axford, Lauren Weitzman, David Atkins, Vivek Srikumar, and Zac Imel. 2022.  Psychotherapy is not one thing: Simultaneous modeling of different therapeutic approaches.  In _Proceedings of the Eighth Workshop on Computational Linguistics and Clinical Psychology_ , pages 47–58, Seattle, USA. Association for Computational Linguistics.
  * Miller et al. (2017) A. H. Miller, W. Feng, A. Fisch, J. Lu, D. Batra, A. Bordes, D. Parikh, and J. Weston. 2017.  Parlai: A dialog research software platform.  _arXiv preprint arXiv:1705.06476_.
  * Nakayama et al. (2018) Hiroki Nakayama, Takahiro Kubo, Junya Kamura, Yasufumi Taniguchi, and Xu Liang. 2018.  doccano: Text annotation tool for human.  Software available from https://github.com/doccano/doccano.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_.
  * Pei et al. (2022) Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Apostolos Dedeloudis, Jackson Sargent, and David Jurgens. 2022.  Potato: The portable text annotation tool.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
  * Smith et al. (2022) Eric Smith, Orion Hsu, Rebecca Qian, Stephen Roller, Y-Lan Boureau, and Jason Weston. 2022.  Human evaluation of conversations is an open problem: comparing the sensitivity of various methods for evaluating dialogue agents.  In _Proceedings of the 4th Workshop on NLP for Conversational AI_ , pages 77–97, Dublin, Ireland. Association for Computational Linguistics.
  * Stenetorp et al. (2012) Pontus Stenetorp, Sampo Pyysalo, Goran Topić, Tomoko Ohta, Sophia Ananiadou, and Jun’ichi Tsujii. 2012.  Brat: a web-based tool for nlp-assisted text annotation.  In _Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 102–107.
  * Sumers et al. (2023) Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. 2023.  Cognitive architectures for language agents.  _Transactions on Machine Learning Research_.
  * Urbanek and Ringshia (2023) Jack Urbanek and Pratik Ringshia. 2023.  Mephisto: A framework for portable, reproducible, and iterative crowdsourcing.
  * Weisser (2016) Martin Weisser. 2016.  Dart–the dialogue annotation and research tool.  _Corpus Linguistics and Linguistic Theory_ , 12(2):355–388.
  * Yang and Heeman (2005) Fan Yang and Peter A Heeman. 2005.  Dialogueview: an annotation tool for dialogue.  In _Proceedings of HLT/EMNLP 2005 Interactive Demonstrations_ , pages 20–21.
  * Zhou et al. (2022) Pei Zhou, Hyundong Cho, Pegah Jandaghi, Dong-Ho Lee, Bill Yuchen Lin, Jay Pujara, and Xiang Ren. 2022.  Reflect, not reflex: Inference-based common ground improves dialogue response quality.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10450–10468, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

## Appendix

##  Appendix A Evaluation interface configurations

###  A.1 Sample survey configuration

An example survey configuration is shown in Figure 4. Survey components are
also easily dynamically configurable with common input options such as radio
buttons, Likert scales, freeform text, etc. Users can create their own survey
with HTML as well, but the customization options we provide through the YAML
file covers most conversation-level evaluation use cases.

###  A.2 Consent form configuration

An example of configuring the consent form is shown in Figure 5. When deployed
without a crowdsourcing platform, these appear as checkboxes in the sign up
process. Within AMT, we automatically sign up the workers with their worker ID
and do not require a password, but they have to check the same checkboxes in
order to move on to the task if they are doing the task for the first time.

Figure 4: An example of the survey pane configuration that contains a custom
Likert scale and freeform text input fields. This configuration corresponds to
the survey pane partially shown in Figure 1.  Figure 5: An example of
configuring the consent form. The agreement_file parameter should point to the
HTML file that shows the content of the consent form.